From patchwork Fri Jan 17 05:24:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 211249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FF1DC33CB1 for ; Fri, 17 Jan 2020 05:25:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5BE0B2082F for ; Fri, 17 Jan 2020 05:25:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kNNKlqw0" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729112AbgAQFZP (ORCPT ); Fri, 17 Jan 2020 00:25:15 -0500 Received: from mail-pg1-f195.google.com ([209.85.215.195]:40383 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726889AbgAQFZP (ORCPT ); Fri, 17 Jan 2020 00:25:15 -0500 Received: by mail-pg1-f195.google.com with SMTP id k25so11081930pgt.7; Thu, 16 Jan 2020 21:25:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :in-reply-to:references; bh=9hD4gGqAw0F2IWW7/hfAo40ccL6ZEiDWdc/VVvCImBQ=; b=kNNKlqw0lkhwYBdH1URfgVP79ku9pFovcRny9rqAHb9dUli7eodWan37zXpd+AZpLI ExJeJDzBj9MUB6SzWhv0hO/HeFxIUwMaLqx6/fs4Y5ZxyWrYkfYL7QdVReLXQStXMEEh ypl3Z7WiN17iyGUZw72rI0eRjjgaeboKx1k3PAUMOiGWCdmQzNryKfwg+UFQs9wAB275 GPeOKWAJkbjFNsFYPayof35AhczX6PMBa0wnOYpJU+OkNdR6c0Zhin+tsv6fUrhCVRTh Qy7TMckiiGEIk443BtiB3ksZRc41nCfAzJdtScXHOBg8lLw8zwJ1YgoH3KKVRrBCZAo+ i83A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:in-reply-to:references; bh=9hD4gGqAw0F2IWW7/hfAo40ccL6ZEiDWdc/VVvCImBQ=; b=hOmnGO/aZ9EsArOfjPFfwPfbVQ94aMIRZOR/xiGYArDgz/saugMWJteQfeFApq0YoB pKAkx38DE1PsJYYCr6SLoO/4m46mlv05V46K4mrv5SWQtFqRI1qrVlvGkoan3+f90xT2 OfbYzYBf4RH4D31rrcK2e05U8CLyvWQoO+lL8hkm0dRViHV9EUUrLwupZBKz/BwXuo17 et5SnC4b/tszUnn/qxJxgw691hwqWh9uiDcjyEbkEQvHeYwfXV0g4dpwDe8LsS9LOxOc yG8xSa5opTsjl4kek2J6bQEZ3ITwGswUVNtNCC704qv2LHM1CfvhJKw0bey8PvkWo/00 QlXA== X-Gm-Message-State: APjAAAXkuRQcw0Fun6H/+xDA1cBe8awp8VknTdWK+Jfqs1DykKkslYz/ ey6QDgV1z7sLUFzjZkvI5VM= X-Google-Smtp-Source: APXvYqyEMRVZwkt+pzXBDoNL2mnZWvJpMT+GhWy8Ji9BnNWZq1bIwPN/radTDPhX+sZh4t9p7axFnA== X-Received: by 2002:aa7:8a99:: with SMTP id a25mr1278869pfc.42.1579238714157; Thu, 16 Jan 2020 21:25:14 -0800 (PST) Received: from baolinwangubtpc.spreadtrum.com ([117.18.48.82]) by smtp.gmail.com with ESMTPSA id c26sm28844756pfj.101.2020.01.16.21.25.10 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 16 Jan 2020 21:25:13 -0800 (PST) From: Baolin Wang To: axboe@kernel.dk, paolo.valente@linaro.org, adrian.hunter@intel.com, ulf.hansson@linaro.org Cc: zhang.lyra@gmail.com, orsonzhai@gmail.com, arnd@arndb.de, linus.walleij@linaro.org, baolin.wang7@gmail.com, linux-mmc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org Subject: [RFC PATCH 1/8] block: Change the dispatch_request() API to support batch requests Date: Fri, 17 Jan 2020 13:24:20 +0800 Message-Id: <2fbdb13a61d0db6615b8fd11ddca9106e5417dae.1579164455.git.baolin.wang7@gmail.com> X-Mailer: git-send-email 1.7.9.5 In-Reply-To: References: In-Reply-To: References: Sender: linux-mmc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-mmc@vger.kernel.org Now some SD/MMC host controllers can support packed command or packed request, that means we can package several requests to host controller at one time to improve performence. But the blk-mq always takes one request from the scheduler and dispatch it to the device, regardless of the driver or the scheduler, so there should only ever be one request in the local list in blk_mq_dispatch_rq_list(), that means the bd.last is always true and the driver can not use bd.last to decide if there are requests are pending now in hardware queue to help to package requests. Thus this is a preparation patch, which tries to change the dispatch_request() API to allow dispatching more than one request from the scheduler. Suggested-by: Arnd Bergmann Signed-off-by: Baolin Wang --- block/bfq-iosched.c | 12 +++++++++--- block/blk-mq-sched.c | 15 ++++----------- block/kyber-iosched.c | 20 +++++++++++++------- block/mq-deadline.c | 12 +++++++++--- include/linux/elevator.h | 2 +- 5 files changed, 36 insertions(+), 25 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index ad4af4a..decabc4 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -4774,7 +4774,8 @@ static inline void bfq_update_dispatch_stats(struct request_queue *q, bool idle_timer_disabled) {} #endif /* CONFIG_BFQ_CGROUP_DEBUG */ -static struct request *bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) +static int bfq_dispatch_requests(struct blk_mq_hw_ctx *hctx, + struct list_head *list) { struct bfq_data *bfqd = hctx->queue->elevator->elevator_data; struct request *rq; @@ -4796,7 +4797,12 @@ static struct request *bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) bfq_update_dispatch_stats(hctx->queue, rq, in_serv_queue, idle_timer_disabled); - return rq; + if (!rq) + return 0; + + list_add(&rq->queuelist, list); + + return 1; } /* @@ -6772,7 +6778,7 @@ static ssize_t bfq_low_latency_store(struct elevator_queue *e, .finish_request = bfq_finish_requeue_request, .exit_icq = bfq_exit_icq, .insert_requests = bfq_insert_requests, - .dispatch_request = bfq_dispatch_request, + .dispatch_requests = bfq_dispatch_requests, .next_request = elv_rb_latter_request, .former_request = elv_rb_former_request, .allow_merge = bfq_allow_bio_merge, diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index ca22afd..f49f9d9 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -90,28 +90,21 @@ static void blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx) struct request_queue *q = hctx->queue; struct elevator_queue *e = q->elevator; LIST_HEAD(rq_list); + int ret; do { - struct request *rq; - if (e->type->ops.has_work && !e->type->ops.has_work(hctx)) break; if (!blk_mq_get_dispatch_budget(hctx)) break; - rq = e->type->ops.dispatch_request(hctx); - if (!rq) { + ret = e->type->ops.dispatch_requests(hctx, &rq_list); + if (ret == 0) { blk_mq_put_dispatch_budget(hctx); break; } - /* - * Now this rq owns the budget which has to be released - * if this rq won't be queued to driver via .queue_rq() - * in blk_mq_dispatch_rq_list(). - */ - list_add(&rq->queuelist, &rq_list); } while (blk_mq_dispatch_rq_list(q, &rq_list, true)); } @@ -171,7 +164,7 @@ void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx) { struct request_queue *q = hctx->queue; struct elevator_queue *e = q->elevator; - const bool has_sched_dispatch = e && e->type->ops.dispatch_request; + const bool has_sched_dispatch = e && e->type->ops.dispatch_requests; LIST_HEAD(rq_list); /* RCU or SRCU read lock is needed before checking quiesced flag */ diff --git a/block/kyber-iosched.c b/block/kyber-iosched.c index 34dcea0..8f58434 100644 --- a/block/kyber-iosched.c +++ b/block/kyber-iosched.c @@ -796,12 +796,13 @@ static int kyber_get_domain_token(struct kyber_queue_data *kqd, return NULL; } -static struct request *kyber_dispatch_request(struct blk_mq_hw_ctx *hctx) +static int kyber_dispatch_requests(struct blk_mq_hw_ctx *hctx, + struct list_head *list) { struct kyber_queue_data *kqd = hctx->queue->elevator->elevator_data; struct kyber_hctx_data *khd = hctx->sched_data; struct request *rq; - int i; + int i, ret = 0; spin_lock(&khd->lock); @@ -811,8 +812,11 @@ static struct request *kyber_dispatch_request(struct blk_mq_hw_ctx *hctx) */ if (khd->batching < kyber_batch_size[khd->cur_domain]) { rq = kyber_dispatch_cur_domain(kqd, khd, hctx); - if (rq) + if (rq) { + list_add(&rq->queuelist, list); + ret = 1; goto out; + } } /* @@ -832,14 +836,16 @@ static struct request *kyber_dispatch_request(struct blk_mq_hw_ctx *hctx) khd->cur_domain++; rq = kyber_dispatch_cur_domain(kqd, khd, hctx); - if (rq) + if (rq) { + list_add(&rq->queuelist, list); + ret = 1; goto out; + } } - rq = NULL; out: spin_unlock(&khd->lock); - return rq; + return ret; } static bool kyber_has_work(struct blk_mq_hw_ctx *hctx) @@ -1020,7 +1026,7 @@ static int kyber_batching_show(void *data, struct seq_file *m) .finish_request = kyber_finish_request, .requeue_request = kyber_finish_request, .completed_request = kyber_completed_request, - .dispatch_request = kyber_dispatch_request, + .dispatch_requests = kyber_dispatch_requests, .has_work = kyber_has_work, }, #ifdef CONFIG_BLK_DEBUG_FS diff --git a/block/mq-deadline.c b/block/mq-deadline.c index b490f47..9fbffba 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -378,7 +378,8 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) * different hardware queue. This is because mq-deadline has shared * state for all hardware queues, in terms of sorting, FIFOs, etc. */ -static struct request *dd_dispatch_request(struct blk_mq_hw_ctx *hctx) +static int dd_dispatch_requests(struct blk_mq_hw_ctx *hctx, + struct list_head *list) { struct deadline_data *dd = hctx->queue->elevator->elevator_data; struct request *rq; @@ -387,7 +388,12 @@ static struct request *dd_dispatch_request(struct blk_mq_hw_ctx *hctx) rq = __dd_dispatch_request(dd); spin_unlock(&dd->lock); - return rq; + if (!rq) + return 0; + + list_add(&rq->queuelist, list); + + return 1; } static void dd_exit_queue(struct elevator_queue *e) @@ -774,7 +780,7 @@ static void deadline_dispatch_stop(struct seq_file *m, void *v) static struct elevator_type mq_deadline = { .ops = { .insert_requests = dd_insert_requests, - .dispatch_request = dd_dispatch_request, + .dispatch_requests = dd_dispatch_requests, .prepare_request = dd_prepare_request, .finish_request = dd_finish_request, .next_request = elv_rb_latter_request, diff --git a/include/linux/elevator.h b/include/linux/elevator.h index 901bda3..a65bf5d 100644 --- a/include/linux/elevator.h +++ b/include/linux/elevator.h @@ -42,7 +42,7 @@ struct elevator_mq_ops { void (*prepare_request)(struct request *, struct bio *bio); void (*finish_request)(struct request *); void (*insert_requests)(struct blk_mq_hw_ctx *, struct list_head *, bool); - struct request *(*dispatch_request)(struct blk_mq_hw_ctx *); + int (*dispatch_requests)(struct blk_mq_hw_ctx *, struct list_head *); bool (*has_work)(struct blk_mq_hw_ctx *); void (*completed_request)(struct request *, u64); void (*requeue_request)(struct request *); From patchwork Fri Jan 17 05:24:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 211248 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CEFAC3F68F for ; Fri, 17 Jan 2020 05:25:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5468121582 for ; Fri, 17 Jan 2020 05:25:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MmoRabuB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729252AbgAQFZ0 (ORCPT ); Fri, 17 Jan 2020 00:25:26 -0500 Received: from mail-pf1-f195.google.com ([209.85.210.195]:43357 "EHLO mail-pf1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729247AbgAQFZZ (ORCPT ); Fri, 17 Jan 2020 00:25:25 -0500 Received: by mail-pf1-f195.google.com with SMTP id x6so11379303pfo.10; Thu, 16 Jan 2020 21:25:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :in-reply-to:references; bh=/WG2iQgLHGShIMmm9+FFgsfln8TiyvQKnoRvhMx5wMQ=; b=MmoRabuBuA3+ELwShUAWka+szte9HUdod3jk9G1BuqHWzjbp5zXR9XdLCYojr1C2UY nAklWhCb48BZy8OAuIXDBJq1oo0Rqqb6BiMMTIW6mRFyhkqbhQLr86OVyVAe/IeAcBrW cAdZ0ktAHhj3eleco6skM+F4WIR4AwkL/3+f4RUkJZViShbePV3ecDglds1HmZz8E0rP nk6Bi961FOim1bo7DRqY+aQp98IY2r78CNxIBOU3Q4fMZVEncpa5eWiJ72sOxu6d3oZF kvg2lF/ggoNcfeJjtm+4BHl1iB7uVYmDqn+Y0wvB8rYQDTr/Z9UYt9wtCfEMofg+tIjt qOuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:in-reply-to:references; bh=/WG2iQgLHGShIMmm9+FFgsfln8TiyvQKnoRvhMx5wMQ=; b=MGF7mm1OcAYYkNv05fxoIOQROY19ddV4LqnHDp2eZFLqgOguBmKOLnFnyLOSEf86Zz jSx02UBNg304U+zQRpeflz6A/zdflxGYqSPzErG546Htn9lgv9EHBn6ZPD+wCiAHWTw+ E54hVr4EtyCM0PcZW2BhIwMcJ9vJseePdkWXtZpDUU1kaz6Nohayvl4AgisSplBGfSss KwjLptRi7TdjxjaIH8B7uxi2r7IV95ZS0NwvShWt9n5/wvEb+fLZ1pIrakEH9x3Zgnyy Iz/xKWFtnttg9d7s1uv+B/xtPw9G1Y4bw2NUCMxCRWCNaCa4H2gR96YiQQRcwYJ20G7N ofUA== X-Gm-Message-State: APjAAAXIhHcfvjA8Tr5FIf8FluVRW6q9gDbRw1QtQ+59Eki1hyRq5lh/ YhB2qnnQJHYlALUhGAJukrA= X-Google-Smtp-Source: APXvYqyHWGV15eskLKsXEKjmH1OjZ9ypbfB3mYpzeXIuF60ja1JmBcbtbecxNtK+A4u/DWtZ2BV6Vw== X-Received: by 2002:a63:5f91:: with SMTP id t139mr42520741pgb.185.1579238722975; Thu, 16 Jan 2020 21:25:22 -0800 (PST) Received: from baolinwangubtpc.spreadtrum.com ([117.18.48.82]) by smtp.gmail.com with ESMTPSA id c26sm28844756pfj.101.2020.01.16.21.25.19 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 16 Jan 2020 21:25:22 -0800 (PST) From: Baolin Wang To: axboe@kernel.dk, paolo.valente@linaro.org, adrian.hunter@intel.com, ulf.hansson@linaro.org Cc: zhang.lyra@gmail.com, orsonzhai@gmail.com, arnd@arndb.de, linus.walleij@linaro.org, baolin.wang7@gmail.com, linux-mmc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org Subject: [RFC PATCH 3/8] mmc: Add MMC packed request support for MMC software queue Date: Fri, 17 Jan 2020 13:24:22 +0800 Message-Id: <53da23898ab36c45ad5298e508ef7d81ecd7e8b8.1579164456.git.baolin.wang7@gmail.com> X-Mailer: git-send-email 1.7.9.5 In-Reply-To: References: In-Reply-To: References: Sender: linux-mmc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-mmc@vger.kernel.org Some SD controllers can support packed command or packed request, that means it can package several requests to host controller to be handled at one time, which can reduce interrutps and improve the DMA transfer. As a result, the I/O performence can be improved. Thus this patch adds MMC packed function to support packed requests or packed command based on the MMC software queue. The basic concept of this function is that, we try to collect more requests from block layer as much as possible to be linked into MMC packed queue by mmc_blk_cqe_issue_rw_rq(). When the last request of the hardware queue comes, or the collected request numbers are larger than 16, or a larger request comes, then we can start to pakage a packed request to host controller. The MMC packed function also supplies packed algorithm operations to help to package qualified requests. After finishing the packed request, the MMC packed function will help to complete each request, at the same time, the MMC packed queue will allow to collect more requests from block layer. After completing each request, the MMC packed function can try to package another packed request to host controller directly in the complete path, if there are enough requests in MMC packed queue or the request pending flag is not set. If the pending flag was set, we should let the mmc_blk_cqe_issue_rw_rq() collect more request as much as possible.i Signed-off-by: Baolin Wang --- drivers/mmc/core/block.c | 22 ++- drivers/mmc/core/block.h | 3 +- drivers/mmc/core/core.c | 26 ++++ drivers/mmc/core/core.h | 2 + drivers/mmc/core/queue.c | 22 ++- drivers/mmc/host/mmc_hsq.c | 310 ++++++++++++++++++++++++++++++++++++----- drivers/mmc/host/mmc_hsq.h | 26 +++- drivers/mmc/host/sdhci-sprd.c | 2 +- include/linux/mmc/core.h | 6 + include/linux/mmc/host.h | 9 ++ 10 files changed, 375 insertions(+), 53 deletions(-) diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c index eb5f15f..0715ad0 100644 --- a/drivers/mmc/core/block.c +++ b/drivers/mmc/core/block.c @@ -1537,7 +1537,8 @@ static int mmc_blk_cqe_issue_flush(struct mmc_queue *mq, struct request *req) return mmc_blk_cqe_start_req(mq->card->host, mrq); } -static int mmc_blk_swq_issue_rw_rq(struct mmc_queue *mq, struct request *req) +static int mmc_blk_swq_issue_rw_rq(struct mmc_queue *mq, struct request *req, + bool last) { struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req); struct mmc_host *host = mq->card->host; @@ -1548,19 +1549,25 @@ static int mmc_blk_swq_issue_rw_rq(struct mmc_queue *mq, struct request *req) mmc_pre_req(host, &mqrq->brq.mrq); err = mmc_cqe_start_req(host, &mqrq->brq.mrq); - if (err) + if (err) { mmc_post_req(host, &mqrq->brq.mrq, err); + return err; + } - return err; + if (last) + mmc_cqe_commit_rqs(host, true); + + return 0; } -static int mmc_blk_cqe_issue_rw_rq(struct mmc_queue *mq, struct request *req) +static int mmc_blk_cqe_issue_rw_rq(struct mmc_queue *mq, struct request *req, + bool last) { struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req); struct mmc_host *host = mq->card->host; if (host->swq_enabled) - return mmc_blk_swq_issue_rw_rq(mq, req); + return mmc_blk_swq_issue_rw_rq(mq, req, last); mmc_blk_data_prep(mq, mqrq, 0, NULL, NULL); @@ -2236,7 +2243,8 @@ static int mmc_blk_wait_for_idle(struct mmc_queue *mq, struct mmc_host *host) return mmc_blk_rw_wait(mq, NULL); } -enum mmc_issued mmc_blk_mq_issue_rq(struct mmc_queue *mq, struct request *req) +enum mmc_issued mmc_blk_mq_issue_rq(struct mmc_queue *mq, struct request *req, + bool last) { struct mmc_blk_data *md = mq->blkdata; struct mmc_card *card = md->queue.card; @@ -2280,7 +2288,7 @@ enum mmc_issued mmc_blk_mq_issue_rq(struct mmc_queue *mq, struct request *req) case REQ_OP_READ: case REQ_OP_WRITE: if (mq->use_cqe) - ret = mmc_blk_cqe_issue_rw_rq(mq, req); + ret = mmc_blk_cqe_issue_rw_rq(mq, req, last); else ret = mmc_blk_mq_issue_rw_rq(mq, req); break; diff --git a/drivers/mmc/core/block.h b/drivers/mmc/core/block.h index 31153f6..8bfb89f 100644 --- a/drivers/mmc/core/block.h +++ b/drivers/mmc/core/block.h @@ -9,7 +9,8 @@ enum mmc_issued; -enum mmc_issued mmc_blk_mq_issue_rq(struct mmc_queue *mq, struct request *req); +enum mmc_issued mmc_blk_mq_issue_rq(struct mmc_queue *mq, struct request *req, + bool last); void mmc_blk_mq_complete(struct request *req); void mmc_blk_mq_recovery(struct mmc_queue *mq); diff --git a/drivers/mmc/core/core.c b/drivers/mmc/core/core.c index abf8f5e..987aa17 100644 --- a/drivers/mmc/core/core.c +++ b/drivers/mmc/core/core.c @@ -329,6 +329,7 @@ static int mmc_mrq_prep(struct mmc_host *host, struct mmc_request *mrq) } } + INIT_LIST_HEAD(&mrq->list); return 0; } @@ -536,6 +537,31 @@ void mmc_cqe_post_req(struct mmc_host *host, struct mmc_request *mrq) } EXPORT_SYMBOL(mmc_cqe_post_req); +/** + * mmc_cqe_commit_rqs - Commit requests pending in CQE + * @host: MMC host + * @last: Indicate if the last request from block layer + */ +void mmc_cqe_commit_rqs(struct mmc_host *host, bool last) +{ + if (host->cqe_ops->cqe_commit_rqs) + host->cqe_ops->cqe_commit_rqs(host, last); +} +EXPORT_SYMBOL(mmc_cqe_commit_rqs); + +/** + * mmc_cqe_is_busy - If CQE is busy or not + * @host: MMC host + */ +bool mmc_cqe_is_busy(struct mmc_host *host) +{ + if (host->cqe_ops->cqe_is_busy) + return host->cqe_ops->cqe_is_busy(host); + + return false; +} +EXPORT_SYMBOL(mmc_cqe_is_busy); + /* Arbitrary 1 second timeout */ #define MMC_CQE_RECOVERY_TIMEOUT 1000 diff --git a/drivers/mmc/core/core.h b/drivers/mmc/core/core.h index 575ac02..e72e6f1 100644 --- a/drivers/mmc/core/core.h +++ b/drivers/mmc/core/core.h @@ -139,6 +139,8 @@ static inline void mmc_claim_host(struct mmc_host *host) int mmc_cqe_start_req(struct mmc_host *host, struct mmc_request *mrq); void mmc_cqe_post_req(struct mmc_host *host, struct mmc_request *mrq); int mmc_cqe_recovery(struct mmc_host *host); +void mmc_cqe_commit_rqs(struct mmc_host *host, bool last); +bool mmc_cqe_is_busy(struct mmc_host *host); /** * mmc_pre_req - Prepare for a new request diff --git a/drivers/mmc/core/queue.c b/drivers/mmc/core/queue.c index d9086c1..e9967bf 100644 --- a/drivers/mmc/core/queue.c +++ b/drivers/mmc/core/queue.c @@ -285,11 +285,7 @@ static blk_status_t mmc_mq_queue_rq(struct blk_mq_hw_ctx *hctx, } break; case MMC_ISSUE_ASYNC: - /* - * For MMC host software queue, we only allow 2 requests in - * flight to avoid a long latency. - */ - if (host->swq_enabled && mq->in_flight[issue_type] > 2) { + if (mq->use_cqe && mmc_cqe_is_busy(host)) { spin_unlock_irq(&mq->lock); return BLK_STS_RESOURCE; } @@ -330,7 +326,7 @@ static blk_status_t mmc_mq_queue_rq(struct blk_mq_hw_ctx *hctx, blk_mq_start_request(req); - issued = mmc_blk_mq_issue_rq(mq, req); + issued = mmc_blk_mq_issue_rq(mq, req, bd->last); switch (issued) { case MMC_REQ_BUSY: @@ -362,8 +358,19 @@ static blk_status_t mmc_mq_queue_rq(struct blk_mq_hw_ctx *hctx, return ret; } +static void mmc_mq_commit_rqs(struct blk_mq_hw_ctx *hctx) +{ + struct mmc_queue *mq = hctx->queue->queuedata; + struct mmc_card *card = mq->card; + struct mmc_host *host = card->host; + + if (mq->use_cqe) + mmc_cqe_commit_rqs(host, false); +} + static const struct blk_mq_ops mmc_mq_ops = { .queue_rq = mmc_mq_queue_rq, + .commit_rqs = mmc_mq_commit_rqs, .init_request = mmc_mq_init_request, .exit_request = mmc_mq_exit_request, .complete = mmc_blk_mq_complete, @@ -390,6 +397,9 @@ static void mmc_setup_queue(struct mmc_queue *mq, struct mmc_card *card) "merging was advertised but not possible"); blk_queue_max_segments(mq->queue, mmc_get_max_segments(host)); + if (host->max_packed_reqs != 0) + blk_queue_max_batch_requests(mq->queue, host->max_packed_reqs); + if (mmc_card_mmc(card)) block_size = card->ext_csd.data_sector_size; diff --git a/drivers/mmc/host/mmc_hsq.c b/drivers/mmc/host/mmc_hsq.c index 6a6bdd0..1eef0f8 100644 --- a/drivers/mmc/host/mmc_hsq.c +++ b/drivers/mmc/host/mmc_hsq.c @@ -8,22 +8,54 @@ #include #include +#include #include "mmc_hsq.h" #define HSQ_NUM_SLOTS 32 #define HSQ_INVALID_TAG HSQ_NUM_SLOTS +#define HSQ_PACKED_FLUSH_BLOCKS 256 +#define HSQ_PACKED_QUEUE_DEPTH 64 + +/** + * mmc_hsq_packed_algo_rw - the algorithm to package read or write requests + * @mmc: the host controller + * + * TODO: we can add more condition to decide if we can package this + * request or not. + */ +void mmc_hsq_packed_algo_rw(struct mmc_host *mmc) +{ + struct mmc_hsq *hsq = mmc->cqe_private; + struct hsq_packed *packed = hsq->packed; + struct mmc_packed_request *prq = &packed->prq; + struct mmc_request *mrq, *t; + u32 i = 0; + + list_for_each_entry_safe(mrq, t, &packed->list, list) { + if (++i > packed->max_entries) + break; + + list_move_tail(&mrq->list, &prq->list); + prq->nr_reqs++; + } +} +EXPORT_SYMBOL_GPL(mmc_hsq_packed_algo_rw); + static void mmc_hsq_pump_requests(struct mmc_hsq *hsq) { + struct hsq_packed *packed = hsq->packed; struct mmc_host *mmc = hsq->mmc; struct hsq_slot *slot; + struct mmc_request *mrq; unsigned long flags; + int ret; spin_lock_irqsave(&hsq->lock, flags); /* Make sure we are not already running a request now */ - if (hsq->mrq) { + if (hsq->mrq || (packed && packed->prq.nr_reqs)) { spin_unlock_irqrestore(&hsq->lock, flags); return; } @@ -34,16 +66,58 @@ static void mmc_hsq_pump_requests(struct mmc_hsq *hsq) return; } - slot = &hsq->slot[hsq->next_tag]; - hsq->mrq = slot->mrq; - hsq->qcnt--; + if (packed) { + /* Try to package requests */ + packed->ops->packed_algo(mmc); + + packed->busy = true; + hsq->qcnt -= packed->prq.nr_reqs; + } else { + slot = &hsq->slot[hsq->next_tag]; + hsq->mrq = slot->mrq; + hsq->qcnt--; + } spin_unlock_irqrestore(&hsq->lock, flags); - if (mmc->ops->request_atomic) - mmc->ops->request_atomic(mmc, hsq->mrq); - else - mmc->ops->request(mmc, hsq->mrq); + if (!packed) { + if (mmc->ops->request_atomic) + mmc->ops->request_atomic(mmc, hsq->mrq); + else + mmc->ops->request(mmc, hsq->mrq); + + return; + } + + if (packed->ops->prepare_hardware) { + ret = packed->ops->prepare_hardware(mmc); + if (ret) { + pr_err("failed to prepare hardware\n"); + goto error; + } + } + + ret = packed->ops->packed_request(mmc, &packed->prq); + if (ret) { + pr_err("failed to packed requests\n"); + goto error; + } + + return; + +error: + spin_lock_irqsave(&hsq->lock, flags); + + list_for_each_entry(mrq, &packed->prq.list, list) { + struct mmc_data *data = mrq->data; + + data->error = ret; + data->bytes_xfered = 0; + } + + spin_unlock_irqrestore(&hsq->lock, flags); + + mmc_hsq_finalize_packed_request(mmc, &packed->prq); } static void mmc_hsq_update_next_tag(struct mmc_hsq *hsq, int remains) @@ -85,16 +159,24 @@ static void mmc_hsq_update_next_tag(struct mmc_hsq *hsq, int remains) static void mmc_hsq_post_request(struct mmc_hsq *hsq) { + struct hsq_packed *packed = hsq->packed; + bool need_pump; unsigned long flags; int remains; spin_lock_irqsave(&hsq->lock, flags); remains = hsq->qcnt; - hsq->mrq = NULL; - - /* Update the next available tag to be queued. */ - mmc_hsq_update_next_tag(hsq, remains); + if (packed) { + need_pump = !packed->rqs_pending; + packed->prq.nr_reqs = 0; + } else { + need_pump = true; + hsq->mrq = NULL; + + /* Update the next available tag to be queued. */ + mmc_hsq_update_next_tag(hsq, remains); + } if (hsq->waiting_for_idle && !remains) { hsq->waiting_for_idle = false; @@ -110,10 +192,17 @@ static void mmc_hsq_post_request(struct mmc_hsq *hsq) spin_unlock_irqrestore(&hsq->lock, flags); /* - * Try to pump new request to host controller as fast as possible, - * after completing previous request. - */ - if (remains > 0) + * For non-packed request mode, we should try to pump new request to + * host controller as fast as possible if there are pending requests, + * after completing previous request. + * + * For packed request mode, If there are not enough requests in queue + * and the request pending flag was set, then do not pump requests here, + * instead we should let the mmc_blk_swq_issue_rw_rq() combine more + * requests and pump them. + */ + if ((need_pump && remains > 0) || + (packed && remains >= packed->max_entries)) mmc_hsq_pump_requests(hsq); } @@ -152,6 +241,104 @@ bool mmc_hsq_finalize_request(struct mmc_host *mmc, struct mmc_request *mrq) } EXPORT_SYMBOL_GPL(mmc_hsq_finalize_request); +/** + * mmc_hsq_finalize_packed_request - finalize one packed request + * @mmc: the host controller + * @prq: the packed request need to be finalized + */ +void mmc_hsq_finalize_packed_request(struct mmc_host *mmc, + struct mmc_packed_request *prq) +{ + struct mmc_hsq *hsq = mmc->cqe_private; + struct hsq_packed *packed = hsq->packed; + struct mmc_request *mrq, *t; + LIST_HEAD(head); + unsigned long flags; + + if (!packed || !prq) + return; + + if (packed->ops->unprepare_hardware && + packed->ops->unprepare_hardware(mmc)) + pr_err("failed to unprepare hardware\n"); + + /* + * Clear busy flag to allow collecting more requests into command + * queue, but now we can not pump them to controller, we should wait + * for all requests are completed. During the period of completing + * requests, we should collect more requests from block layer as much + * as possible. + */ + spin_lock_irqsave(&hsq->lock, flags); + list_splice_tail_init(&prq->list, &head); + packed->busy = false; + spin_unlock_irqrestore(&hsq->lock, flags); + + list_for_each_entry_safe(mrq, t, &head, list) { + list_del(&mrq->list); + + mmc_cqe_request_done(mmc, mrq); + } + + mmc_hsq_post_request(hsq); +} +EXPORT_SYMBOL_GPL(mmc_hsq_finalize_packed_request); + +static void mmc_hsq_commit_rqs(struct mmc_host *mmc, bool last) +{ + struct mmc_hsq *hsq = mmc->cqe_private; + struct hsq_packed *packed = hsq->packed; + + if (!packed) + return; + + spin_lock_irq(&hsq->lock); + + /* Set pending flag which indicates more request will be coming */ + if (!last && !packed->rqs_pending) + packed->rqs_pending = true; + + spin_unlock_irq(&hsq->lock); + + /* + * If the last request is coming in hardware queue, then pump requests + * to controller as fast as possible. + */ + if (last) + mmc_hsq_pump_requests(hsq); +} + +static bool mmc_hsq_is_busy(struct mmc_host *mmc) +{ + struct mmc_hsq *hsq = mmc->cqe_private; + struct hsq_packed *packed = hsq->packed; + unsigned long flags; + bool busy; + + spin_lock_irqsave(&hsq->lock, flags); + + /* + * For packed mode, when hardware is busy, we can only allow maximum + * packed number requests to be ready in software queue to be queued + * after previous packed request is completed, which avoiding long + * latency. + * + * For non-packed mode, we can only allow 2 requests in flight to avoid + * long latency. + * + * Otherwise return BLK_STS_RESOURCE to tell block layer to dispatch + * requests later. + */ + if (packed) + busy = packed->busy && hsq->qcnt >= packed->max_entries; + else + busy = hsq->qcnt > 1; + + spin_unlock_irqrestore(&hsq->lock, flags); + + return busy; +} + static void mmc_hsq_recovery_start(struct mmc_host *mmc) { struct mmc_hsq *hsq = mmc->cqe_private; @@ -187,7 +374,8 @@ static void mmc_hsq_recovery_finish(struct mmc_host *mmc) static int mmc_hsq_request(struct mmc_host *mmc, struct mmc_request *mrq) { struct mmc_hsq *hsq = mmc->cqe_private; - int tag = mrq->tag; + struct hsq_packed *packed = hsq->packed; + int nr_rqs = 0, tag = mrq->tag; spin_lock_irq(&hsq->lock); @@ -202,20 +390,41 @@ static int mmc_hsq_request(struct mmc_host *mmc, struct mmc_request *mrq) return -EBUSY; } - hsq->slot[tag].mrq = mrq; + hsq->qcnt++; - /* - * Set the next tag as current request tag if no available - * next tag. - */ - if (hsq->next_tag == HSQ_INVALID_TAG) - hsq->next_tag = tag; + if (packed) { + list_add_tail(&mrq->list, &packed->list); - hsq->qcnt++; + /* New request comes, then clear pending flag */ + if (packed->rqs_pending) + packed->rqs_pending = false; + + nr_rqs = hsq->qcnt; + } else { + hsq->slot[tag].mrq = mrq; + + /* + * Set the next tag as current request tag if no available + * next tag. + */ + if (hsq->next_tag == HSQ_INVALID_TAG) + hsq->next_tag = tag; + } spin_unlock_irq(&hsq->lock); - mmc_hsq_pump_requests(hsq); + /* + * For non-packed request mode, we should pump requests as soon as + * possible. + * + * For the packed request mode, if it is a larger request or the + * request count is larger than the maximum packed number, we + * should pump requests to controller. Otherwise we should try to + * combine requests as much as we can. + */ + if (!packed || mrq->data->blocks > HSQ_PACKED_FLUSH_BLOCKS || + nr_rqs >= packed->max_entries) + mmc_hsq_pump_requests(hsq); return 0; } @@ -228,12 +437,17 @@ static void mmc_hsq_post_req(struct mmc_host *mmc, struct mmc_request *mrq) static bool mmc_hsq_queue_is_idle(struct mmc_hsq *hsq, int *ret) { + struct hsq_packed *packed = hsq->packed; bool is_idle; spin_lock_irq(&hsq->lock); - is_idle = (!hsq->mrq && !hsq->qcnt) || - hsq->recovery_halt; + if (packed) + is_idle = (!packed->prq.nr_reqs && !hsq->qcnt) || + hsq->recovery_halt; + else + is_idle = (!hsq->mrq && !hsq->qcnt) || + hsq->recovery_halt; *ret = hsq->recovery_halt ? -EBUSY : 0; hsq->waiting_for_idle = !is_idle; @@ -310,18 +524,40 @@ static int mmc_hsq_enable(struct mmc_host *mmc, struct mmc_card *card) .cqe_wait_for_idle = mmc_hsq_wait_for_idle, .cqe_recovery_start = mmc_hsq_recovery_start, .cqe_recovery_finish = mmc_hsq_recovery_finish, + .cqe_is_busy = mmc_hsq_is_busy, + .cqe_commit_rqs = mmc_hsq_commit_rqs, }; -int mmc_hsq_init(struct mmc_hsq *hsq, struct mmc_host *mmc) +int mmc_hsq_init(struct mmc_hsq *hsq, struct mmc_host *mmc, + const struct hsq_packed_ops *ops, int max_packed) { - hsq->num_slots = HSQ_NUM_SLOTS; - hsq->next_tag = HSQ_INVALID_TAG; - mmc->cqe_qdepth = HSQ_NUM_SLOTS; - - hsq->slot = devm_kcalloc(mmc_dev(mmc), hsq->num_slots, - sizeof(struct hsq_slot), GFP_KERNEL); - if (!hsq->slot) - return -ENOMEM; + if (ops && max_packed > 1) { + struct hsq_packed *packed; + + packed = devm_kzalloc(mmc_dev(mmc), sizeof(struct hsq_packed), + GFP_KERNEL); + if (!packed) + return -ENOMEM; + + packed->ops = ops; + packed->max_entries = max_packed; + INIT_LIST_HEAD(&packed->list); + INIT_LIST_HEAD(&packed->prq.list); + + hsq->packed = packed; + mmc->cqe_qdepth = HSQ_PACKED_QUEUE_DEPTH; + mmc->max_packed_reqs = max_packed; + } else { + hsq->num_slots = HSQ_NUM_SLOTS; + hsq->next_tag = HSQ_INVALID_TAG; + mmc->cqe_qdepth = HSQ_NUM_SLOTS; + + hsq->slot = devm_kcalloc(mmc_dev(mmc), hsq->num_slots, + sizeof(struct hsq_slot), + GFP_KERNEL); + if (!hsq->slot) + return -ENOMEM; + } hsq->mmc = mmc; hsq->mmc->cqe_private = hsq; diff --git a/drivers/mmc/host/mmc_hsq.h b/drivers/mmc/host/mmc_hsq.h index d51beb7..c3d3bc7 100644 --- a/drivers/mmc/host/mmc_hsq.h +++ b/drivers/mmc/host/mmc_hsq.h @@ -2,6 +2,24 @@ #ifndef LINUX_MMC_HSQ_H #define LINUX_MMC_HSQ_H +struct hsq_packed_ops { + void (*packed_algo)(struct mmc_host *mmc); + int (*prepare_hardware)(struct mmc_host *mmc); + int (*unprepare_hardware)(struct mmc_host *mmc); + int (*packed_request)(struct mmc_host *mmc, + struct mmc_packed_request *prq); +}; + +struct hsq_packed { + bool busy; + bool rqs_pending; + int max_entries; + + struct list_head list; + struct mmc_packed_request prq; + const struct hsq_packed_ops *ops; +}; + struct hsq_slot { struct mmc_request *mrq; }; @@ -20,11 +38,17 @@ struct mmc_hsq { bool enabled; bool waiting_for_idle; bool recovery_halt; + + struct hsq_packed *packed; }; -int mmc_hsq_init(struct mmc_hsq *hsq, struct mmc_host *mmc); +int mmc_hsq_init(struct mmc_hsq *hsq, struct mmc_host *mmc, + const struct hsq_packed_ops *ops, int max_packed); void mmc_hsq_suspend(struct mmc_host *mmc); int mmc_hsq_resume(struct mmc_host *mmc); bool mmc_hsq_finalize_request(struct mmc_host *mmc, struct mmc_request *mrq); +void mmc_hsq_finalize_packed_request(struct mmc_host *mmc, + struct mmc_packed_request *prq); +void mmc_hsq_packed_algo_rw(struct mmc_host *mmc); #endif diff --git a/drivers/mmc/host/sdhci-sprd.c b/drivers/mmc/host/sdhci-sprd.c index ae9acb8..49afe1c 100644 --- a/drivers/mmc/host/sdhci-sprd.c +++ b/drivers/mmc/host/sdhci-sprd.c @@ -676,7 +676,7 @@ static int sdhci_sprd_probe(struct platform_device *pdev) goto err_cleanup_host; } - ret = mmc_hsq_init(hsq, host->mmc); + ret = mmc_hsq_init(hsq, host->mmc, NULL, 0); if (ret) goto err_cleanup_host; diff --git a/include/linux/mmc/core.h b/include/linux/mmc/core.h index b7ba881..b1be983 100644 --- a/include/linux/mmc/core.h +++ b/include/linux/mmc/core.h @@ -165,6 +165,12 @@ struct mmc_request { bool cap_cmd_during_tfr; int tag; + struct list_head list; +}; + +struct mmc_packed_request { + struct list_head list; + u32 nr_reqs; }; struct mmc_card; diff --git a/include/linux/mmc/host.h b/include/linux/mmc/host.h index cc1a79b..7ac5be1 100644 --- a/include/linux/mmc/host.h +++ b/include/linux/mmc/host.h @@ -216,6 +216,14 @@ struct mmc_cqe_ops { * will have zero data bytes transferred. */ void (*cqe_recovery_finish)(struct mmc_host *host); + + /* If CQE is busy or not. */ + bool (*cqe_is_busy)(struct mmc_host *host); + /* + * Serve the purpose of kicking the hardware to handle pending + * requests. + */ + void (*cqe_commit_rqs)(struct mmc_host *host, bool last); }; struct mmc_async_req { @@ -385,6 +393,7 @@ struct mmc_host { unsigned int max_blk_size; /* maximum size of one mmc block */ unsigned int max_blk_count; /* maximum number of blocks in one req */ unsigned int max_busy_timeout; /* max busy timeout in ms */ + unsigned int max_packed_reqs; /* max number of requests can be packed */ /* private data */ spinlock_t lock; /* lock for claim and bus ops */ From patchwork Fri Jan 17 05:24:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 211247 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA662C33CB8 for ; Fri, 17 Jan 2020 05:25:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 817CB2087E for ; Fri, 17 Jan 2020 05:25:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hvYxXIn0" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729346AbgAQFZd (ORCPT ); Fri, 17 Jan 2020 00:25:33 -0500 Received: from mail-pj1-f66.google.com ([209.85.216.66]:55989 "EHLO mail-pj1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729327AbgAQFZb (ORCPT ); Fri, 17 Jan 2020 00:25:31 -0500 Received: by mail-pj1-f66.google.com with SMTP id d5so2629139pjz.5; Thu, 16 Jan 2020 21:25:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :in-reply-to:references; bh=nxaLy9F6yE2CecepgHfL+z5cQMSlQrayHjxpSI6bCwI=; b=hvYxXIn0tdSfCKR32UMVyUwVIAzSQtbnsUQpvvqwhuWpsLLnDF+XXOmFTIV864g6BL CSLX7lJeC/1KAEbUB6xTjFUNBlgkm3Q/v53Z+vcq8cyhwQtLZ+N2WUaaxcFqb0Xv6gU8 DEoLFk3D5LPK7yro5VgeB8+sVJOZco9K3A94a/qQmB3HIAsbQlqmeCjGpztncyogCwkc V2ENT6lnBicFjrqzYrasrC6jHzBfmPk/PB/GVozchwVYzviD1s0wFF0sZzFyoF7hYiM6 71hyk8fCiaCNaLvxOLE0xFZ8Kh+wHpoIcitziS13112Cd71iUiNfZJ9apkD3nDZw5ukr Z7bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:in-reply-to:references; bh=nxaLy9F6yE2CecepgHfL+z5cQMSlQrayHjxpSI6bCwI=; b=Vz83whDeE6J4MjIbu3X9QshBndVInLJ/BWjHHwAeIbkjJS5y0NJd5u1mAq+z6UUsnC IoVVYVlqpTHPqC25j2hVxVoS93q82uf3xVOaYjNoAg8B/T2wNSqkuoTpPN5kua8yJbYu AWYMt9bd/nsclmg5mnX9uUYvVDIlmz4dLVKQj3i0aeQo/FHgDmR0ybLTtNGikaa7MfMo 1qRH501vkDH6dFNgxHX0W/xujIOQKxLodMQcoUbgw9gEzV0MwxlxVNHjuJY2v0NvGj/G 4EIeEjCPTjHM2jOV1XtgLc9HO59Pay0tSLct74PXC86+JPsCq+P1qNi40fM7K5z2gBNU zplQ== X-Gm-Message-State: APjAAAUW0AT/gAX2GpRDG/n5U+ZmXICfBztNnzBtfgm7BYJOTFWOkLD7 fr0JVEpGlZLtQHS5idFmuO8= X-Google-Smtp-Source: APXvYqz05UCb1S5G9rNU+MhgC5WiZHFPMWCPRi3Z2lwqjuMLGHgq7B5AM7Qo8cir8gPMBgmTINFHDg== X-Received: by 2002:a17:902:be06:: with SMTP id r6mr23548273pls.99.1579238730834; Thu, 16 Jan 2020 21:25:30 -0800 (PST) Received: from baolinwangubtpc.spreadtrum.com ([117.18.48.82]) by smtp.gmail.com with ESMTPSA id c26sm28844756pfj.101.2020.01.16.21.25.27 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 16 Jan 2020 21:25:30 -0800 (PST) From: Baolin Wang To: axboe@kernel.dk, paolo.valente@linaro.org, adrian.hunter@intel.com, ulf.hansson@linaro.org Cc: zhang.lyra@gmail.com, orsonzhai@gmail.com, arnd@arndb.de, linus.walleij@linaro.org, baolin.wang7@gmail.com, linux-mmc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org Subject: [RFC PATCH 5/8] mmc: host: sdhci: Factor out the command configuration Date: Fri, 17 Jan 2020 13:24:24 +0800 Message-Id: <47c4217fd274b77e2fb5c86812cce883c8ed9bd2.1579164456.git.baolin.wang7@gmail.com> X-Mailer: git-send-email 1.7.9.5 In-Reply-To: References: In-Reply-To: References: Sender: linux-mmc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-mmc@vger.kernel.org Move the SD command configuration into one separate function to simplify the sdhci_send_command(). Moreover this function can be used to support ADMA3 transfer mode in following patches. Signed-off-by: Baolin Wang --- drivers/mmc/host/sdhci.c | 65 +++++++++++++++++++++++++++------------------- 1 file changed, 38 insertions(+), 27 deletions(-) diff --git a/drivers/mmc/host/sdhci.c b/drivers/mmc/host/sdhci.c index 907a35e..f517121 100644 --- a/drivers/mmc/host/sdhci.c +++ b/drivers/mmc/host/sdhci.c @@ -1350,9 +1350,43 @@ static void sdhci_finish_data(struct sdhci_host *host) } } -void sdhci_send_command(struct sdhci_host *host, struct mmc_command *cmd) +static int sdhci_get_command(struct sdhci_host *host, struct mmc_command *cmd) { int flags; + + if ((cmd->flags & MMC_RSP_136) && (cmd->flags & MMC_RSP_BUSY)) { + pr_err("%s: Unsupported response type!\n", + mmc_hostname(host->mmc)); + cmd->error = -EINVAL; + sdhci_finish_mrq(host, cmd->mrq); + return -EINVAL; + } + + if (!(cmd->flags & MMC_RSP_PRESENT)) + flags = SDHCI_CMD_RESP_NONE; + else if (cmd->flags & MMC_RSP_136) + flags = SDHCI_CMD_RESP_LONG; + else if (cmd->flags & MMC_RSP_BUSY) + flags = SDHCI_CMD_RESP_SHORT_BUSY; + else + flags = SDHCI_CMD_RESP_SHORT; + + if (cmd->flags & MMC_RSP_CRC) + flags |= SDHCI_CMD_CRC; + if (cmd->flags & MMC_RSP_OPCODE) + flags |= SDHCI_CMD_INDEX; + + /* CMD19 is special in that the Data Present Select should be set */ + if (cmd->data || cmd->opcode == MMC_SEND_TUNING_BLOCK || + cmd->opcode == MMC_SEND_TUNING_BLOCK_HS200) + flags |= SDHCI_CMD_DATA; + + return SDHCI_MAKE_CMD(cmd->opcode, flags); +} + +void sdhci_send_command(struct sdhci_host *host, struct mmc_command *cmd) +{ + int command; u32 mask; unsigned long timeout; @@ -1402,32 +1436,9 @@ void sdhci_send_command(struct sdhci_host *host, struct mmc_command *cmd) sdhci_set_transfer_mode(host, cmd); - if ((cmd->flags & MMC_RSP_136) && (cmd->flags & MMC_RSP_BUSY)) { - pr_err("%s: Unsupported response type!\n", - mmc_hostname(host->mmc)); - cmd->error = -EINVAL; - sdhci_finish_mrq(host, cmd->mrq); + command = sdhci_get_command(host, cmd); + if (command < 0) return; - } - - if (!(cmd->flags & MMC_RSP_PRESENT)) - flags = SDHCI_CMD_RESP_NONE; - else if (cmd->flags & MMC_RSP_136) - flags = SDHCI_CMD_RESP_LONG; - else if (cmd->flags & MMC_RSP_BUSY) - flags = SDHCI_CMD_RESP_SHORT_BUSY; - else - flags = SDHCI_CMD_RESP_SHORT; - - if (cmd->flags & MMC_RSP_CRC) - flags |= SDHCI_CMD_CRC; - if (cmd->flags & MMC_RSP_OPCODE) - flags |= SDHCI_CMD_INDEX; - - /* CMD19 is special in that the Data Present Select should be set */ - if (cmd->data || cmd->opcode == MMC_SEND_TUNING_BLOCK || - cmd->opcode == MMC_SEND_TUNING_BLOCK_HS200) - flags |= SDHCI_CMD_DATA; timeout = jiffies; if (host->data_timeout) @@ -1438,7 +1449,7 @@ void sdhci_send_command(struct sdhci_host *host, struct mmc_command *cmd) timeout += 10 * HZ; sdhci_mod_timer(host, cmd->mrq, timeout); - sdhci_writew(host, SDHCI_MAKE_CMD(cmd->opcode, flags), SDHCI_COMMAND); + sdhci_writew(host, command, SDHCI_COMMAND); } EXPORT_SYMBOL_GPL(sdhci_send_command); From patchwork Fri Jan 17 05:24:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 211246 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94B80C33CB1 for ; Fri, 17 Jan 2020 05:25:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 65E1A2087E for ; Fri, 17 Jan 2020 05:25:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KaZz1WJ9" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729551AbgAQFZn (ORCPT ); Fri, 17 Jan 2020 00:25:43 -0500 Received: from mail-pl1-f193.google.com ([209.85.214.193]:42794 "EHLO mail-pl1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729537AbgAQFZm (ORCPT ); Fri, 17 Jan 2020 00:25:42 -0500 Received: by mail-pl1-f193.google.com with SMTP id p9so9367587plk.9; Thu, 16 Jan 2020 21:25:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :in-reply-to:references; bh=VFz24DYx+AJEk47BONRgaGsS8LjnX5pSIY7Tnm7mCJo=; b=KaZz1WJ9hDeSXfTdPY7q2phDBHVx1zx6FJU5x0Ubzj8AzyPoxeeJiz/Urrw17TAlKI Gq79vwsd16J/wD9/LemeFpLRrXcXd6XJomy5sjrc5LWb97jYVScFWabJZUuIa+blgnSH PXQjvfDKqyN+eN6+JOake8O4SJRgByeedVV/HxpqfEBb2IhGBkaukEWnb/IZfpRqRSrL njAyXEEoZ4jII4GskFjGjxH3md5eF4OpDIdd+SW/cQ4qGUM5AEzNj4LeShxCMBSSaIPn BLDyN4C3nebqdFrjyhJ6UpUBrkFbU1sLSrFWwCDEGSPjVgG3+pRYoJSWwWbLXtGAasXx vu5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:in-reply-to:references; bh=VFz24DYx+AJEk47BONRgaGsS8LjnX5pSIY7Tnm7mCJo=; b=fS93deDR51PliemLJREXfux1PBkPPAE/fH6veK9fhEB9rcsA33f3vgJC1jfw3bk9Vw Kn6DwLwR7OHR8JEg0L+ZYVpuxLapMWSsfxmts7wgnuzob4FgsSmcRcGQTRCBkOgfZGeW VGyE/3Y27LgKGCa2kThU0e73GOxGL+PwVXc4WR7YQHaQZkLma9jB7aBuGLhAPD+Fi0uO aO0HGrTdjt2+adIhRVAIGbQpw6E6MyL6WBRp3og6xnw7T6prgZzhWeKqiT0d1JtneSq2 Q8ynPCCC3LB1Qr2PZhxeP1LN4Yy5EkrPacFN+v8IZGLUqvoRj+ZVg96R7b+Rv0bUaa+6 hKEg== X-Gm-Message-State: APjAAAU2pUIXHWOHwkXGTxAhGV6HzTVx+9pfTg2JMDOlcngYBmd+dUC6 EwzllqhM/vejn3qSFfghQ7Q= X-Google-Smtp-Source: APXvYqyMgiGlD9tGbi7+t2vq6IqNQC4YNjC/LY/EYxbj4BpulT+k0yA4i62+ylI7hYpLwFvlbpGCog== X-Received: by 2002:a17:90a:c211:: with SMTP id e17mr3767593pjt.14.1579238742145; Thu, 16 Jan 2020 21:25:42 -0800 (PST) Received: from baolinwangubtpc.spreadtrum.com ([117.18.48.82]) by smtp.gmail.com with ESMTPSA id c26sm28844756pfj.101.2020.01.16.21.25.38 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 16 Jan 2020 21:25:41 -0800 (PST) From: Baolin Wang To: axboe@kernel.dk, paolo.valente@linaro.org, adrian.hunter@intel.com, ulf.hansson@linaro.org Cc: zhang.lyra@gmail.com, orsonzhai@gmail.com, arnd@arndb.de, linus.walleij@linaro.org, baolin.wang7@gmail.com, linux-mmc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org Subject: [RFC PATCH 8/8] mmc: host: sdhci-sprd: Add MMC packed request support Date: Fri, 17 Jan 2020 13:24:27 +0800 Message-Id: <96e3ad74f2717029e5705fbf00bf3e61db90211d.1579164456.git.baolin.wang7@gmail.com> X-Mailer: git-send-email 1.7.9.5 In-Reply-To: References: In-Reply-To: References: Sender: linux-mmc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-mmc@vger.kernel.org Enable the ADMA3 transfer mode as well as adding packed operations to support MMC packed requests to improve IO performance. Signed-off-by: Baolin Wang --- drivers/mmc/host/sdhci-sprd.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/drivers/mmc/host/sdhci-sprd.c b/drivers/mmc/host/sdhci-sprd.c index 49afe1c..daa38ed 100644 --- a/drivers/mmc/host/sdhci-sprd.c +++ b/drivers/mmc/host/sdhci-sprd.c @@ -390,6 +390,12 @@ static void sdhci_sprd_request_done(struct sdhci_host *host, mmc_request_done(host->mmc, mrq); } +static void sdhci_sprd_packed_request_done(struct sdhci_host *host, + struct mmc_packed_request *prq) +{ + mmc_hsq_finalize_packed_request(host->mmc, prq); +} + static struct sdhci_ops sdhci_sprd_ops = { .read_l = sdhci_sprd_readl, .write_l = sdhci_sprd_writel, @@ -404,6 +410,7 @@ static void sdhci_sprd_request_done(struct sdhci_host *host, .get_max_timeout_count = sdhci_sprd_get_max_timeout_count, .get_ro = sdhci_sprd_get_ro, .request_done = sdhci_sprd_request_done, + .packed_request_done = sdhci_sprd_packed_request_done, }; static void sdhci_sprd_request(struct mmc_host *mmc, struct mmc_request *mrq) @@ -546,10 +553,18 @@ static void sdhci_sprd_phy_param_parse(struct sdhci_sprd_host *sprd_host, SDHCI_QUIRK_MISSING_CAPS, .quirks2 = SDHCI_QUIRK2_BROKEN_HS200 | SDHCI_QUIRK2_USE_32BIT_BLK_CNT | - SDHCI_QUIRK2_PRESET_VALUE_BROKEN, + SDHCI_QUIRK2_PRESET_VALUE_BROKEN | + SDHCI_QUIRK2_USE_ADMA3_SUPPORT, .ops = &sdhci_sprd_ops, }; +static const struct hsq_packed_ops packed_ops = { + .packed_algo = mmc_hsq_packed_algo_rw, + .prepare_hardware = sdhci_prepare_packed, + .unprepare_hardware = sdhci_unprepare_packed, + .packed_request = sdhci_packed_request, +}; + static int sdhci_sprd_probe(struct platform_device *pdev) { struct sdhci_host *host; @@ -676,7 +691,18 @@ static int sdhci_sprd_probe(struct platform_device *pdev) goto err_cleanup_host; } - ret = mmc_hsq_init(hsq, host->mmc, NULL, 0); + /* + * If the host controller can support ADMA3 mode, we can enable the + * packed request mode to improve the read/write performance. + * + * Considering the maximum ADMA3 entries (default is 16) and the request + * latency, we set the default maximum packed requests number is 8. + */ + if (host->flags & SDHCI_USE_ADMA3) + ret = mmc_hsq_init(hsq, host->mmc, &packed_ops, + SDHCI_MAX_ADMA3_ENTRIES / 2); + else + ret = mmc_hsq_init(hsq, host->mmc, NULL, 0); if (ret) goto err_cleanup_host;