From patchwork Wed Jul 14 15:06:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 476770 Delivered-To: patch@linaro.org Received: by 2002:a02:c94a:0:0:0:0:0 with SMTP id u10csp659674jao; Wed, 14 Jul 2021 08:11:36 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyFULH2QG+pIkvhYfvCREIBKxhJxRcHJQbpujKeUyFkYDxv/5huBW7B1CrFFphuUC07XZC7 X-Received: by 2002:a05:651c:b0c:: with SMTP id b12mr9878003ljr.190.1626275496661; Wed, 14 Jul 2021 08:11:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1626275496; cv=none; d=google.com; s=arc-20160816; b=dPS1GmFDNV2UGrHxroQhGvGKoez5TzJBF+TP8w8lZYi6i7jTU5J0coSRuVoSPR2kEv eGeVi7OEJK4xvqEswEmkfbdk1pwt9xE4LrM1aqakgiehn8or2/G7eeVmmzoV9VOW/+hM kTBAAGaTBj02X8C6/mmaaVOTuAtTFeVD0q7Dce34QgskAetrdbAt6rR2CmWcLVtwIasN qHYm9a6mP1BRnAgaX9BNu2PS6k2xQx9vFfU5137qf/G1SPBs6AQkLm/auvEXwSCvj+CB DvRsaVFEg6HlNFVfZNRUmN815lx/LUTYsfgiPz0lN14lZVgaqzYqJyejoSi5VolpRhhN a58A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from; bh=xlalhxttNcak4nLqOSjxLB6UZN5k0AGIGJxjo9j++dU=; b=RvdDwCe4r8sHENp6se/Dz2Vulg7St84szwTOp1IwAV4Bm7s1vaBPX6ozxETOv661Z+ iWG6jsYEqHOWz+ZO1OE5jRdNuR5w4J8vdtw5LYA9rqWSj9DO/SqROaOMajJr3vr6d2N7 K/YUXq8M/Opsn0dO+Xbt0+j9VUTyrZc+zU9l0ZUFo4MUm9jeIf+2Cq5oJhgQV97BGNSr gtKNj2TfxXjZkwRA+pUHOrAMYPojV4Aiyutmlmt7ZGSpO5cy3W+6giSXVwndGOCHVl3e K73AnOn3HsrFOzuUFRktpjXBB5Si52SX9261sV7C9w0/2Jnr0h601rnr4NK6Zw5l9hjK klsw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id f21si2888186ejj.567.2021.07.14.08.11.36 for ; Wed, 14 Jul 2021 08:11:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239673AbhGNPOX (ORCPT ); Wed, 14 Jul 2021 11:14:23 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:3410 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239653AbhGNPOV (ORCPT ); Wed, 14 Jul 2021 11:14:21 -0400 Received: from fraeml742-chm.china.huawei.com (unknown [172.18.147.207]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4GQ12n6kyjz6K648; Wed, 14 Jul 2021 23:02:57 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml742-chm.china.huawei.com (10.206.15.223) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Wed, 14 Jul 2021 17:11:28 +0200 Received: from localhost.localdomain (10.69.192.58) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Wed, 14 Jul 2021 16:11:25 +0100 From: John Garry To: CC: , , , , , , John Garry Subject: [PATCH RFC 7/9] blk-mq: Allocate per tag set static rqs for shared sbitmap Date: Wed, 14 Jul 2021 23:06:33 +0800 Message-ID: <1626275195-215652-8-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1626275195-215652-1-git-send-email-john.garry@huawei.com> References: <1626275195-215652-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org It is wasteful for memory that a full set of static rqs is allocated per hw queue for shared sbitmap, considering that the total number of requests usable at any given time across all hctx is limited by the shared sbitmap depth. As such, it is considerably more memory efficient in the case of shared sbitmap to allocate a set of static rqs per tag set, and not per hw queue. Make this change, and ensure that the hctx_idx argument to blk_mq_init_request() and __blk_mq_free_rqs() is == 0 for case of shared sbitmap. If a driver needs to init the static rq with a hctx idx, then it cannot now use shared sbitmap - only SCSI drivers and null block use shared sbitmap today, and neither use hctx_idx for static request init or free. Signed-off-by: John Garry --- block/blk-mq.c | 66 +++++++++++++++++++++++++++++++++++++----- include/linux/blk-mq.h | 4 +++ 2 files changed, 62 insertions(+), 8 deletions(-) -- 2.26.2 diff --git a/block/blk-mq.c b/block/blk-mq.c index 801af0993044..764c601376c6 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2350,6 +2350,9 @@ void __blk_mq_free_rqs(struct blk_mq_tag_set *set, unsigned int hctx_idx, { struct page *page; + if (WARN_ON((hctx_idx > 0) && blk_mq_is_sbitmap_shared(set->flags))) + return; + if (static_rqs && set->ops->exit_request) { int i; @@ -2455,6 +2458,9 @@ int __blk_mq_alloc_rqs(struct blk_mq_tag_set *set, unsigned int hctx_idx, size_t rq_size, left; int node; + if (WARN_ON((hctx_idx > 0) && blk_mq_is_sbitmap_shared(set->flags))) + return -EINVAL; + node = blk_mq_hw_queue_to_node(&set->map[HCTX_TYPE_DEFAULT], hctx_idx); if (node == NUMA_NO_NODE) node = set->numa_node; @@ -2875,15 +2881,30 @@ static bool __blk_mq_alloc_map_and_request(struct blk_mq_tag_set *set, set->tags[hctx_idx] = blk_mq_alloc_rq_map(set, hctx_idx, set->queue_depth, - set->reserved_tags, flags); + set->reserved_tags, + flags); if (!set->tags[hctx_idx]) return false; - ret = blk_mq_alloc_rqs(set, set->tags[hctx_idx], hctx_idx, - set->queue_depth); + if (blk_mq_is_sbitmap_shared(flags)) { + int i; + + if (!set->static_rqs) + goto err_out; + + for (i = 0; i < set->queue_depth; i++) + set->tags[hctx_idx]->static_rqs[i] = + set->static_rqs[i]; + + return true; + } else { + ret = blk_mq_alloc_rqs(set, set->tags[hctx_idx], hctx_idx, + set->queue_depth); + } if (!ret) return true; +err_out: blk_mq_free_rq_map(set->tags[hctx_idx], flags); set->tags[hctx_idx] = NULL; return false; @@ -2895,7 +2916,8 @@ static void blk_mq_free_map_and_requests(struct blk_mq_tag_set *set, unsigned int flags = set->flags; if (set->tags && set->tags[hctx_idx]) { - blk_mq_free_rqs(set, set->tags[hctx_idx], hctx_idx); + if (!blk_mq_is_sbitmap_shared(set->flags)) + blk_mq_free_rqs(set, set->tags[hctx_idx], hctx_idx); blk_mq_free_rq_map(set->tags[hctx_idx], flags); set->tags[hctx_idx] = NULL; } @@ -3363,6 +3385,21 @@ static int __blk_mq_alloc_rq_maps(struct blk_mq_tag_set *set) { int i; + if (blk_mq_is_sbitmap_shared(set->flags)) { + gfp_t flags = GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY; + int ret; + + set->static_rqs = kcalloc_node(set->queue_depth, + sizeof(struct request *), + flags, set->numa_node); + if (!set->static_rqs) + return -ENOMEM; + ret = __blk_mq_alloc_rqs(set, 0, set->queue_depth, + &set->page_list, set->static_rqs); + if (ret < 0) + goto out_free_static_rqs; + } + for (i = 0; i < set->nr_hw_queues; i++) { if (!__blk_mq_alloc_map_and_request(set, i)) goto out_unwind; @@ -3374,7 +3411,15 @@ static int __blk_mq_alloc_rq_maps(struct blk_mq_tag_set *set) out_unwind: while (--i >= 0) blk_mq_free_map_and_requests(set, i); - + if (blk_mq_is_sbitmap_shared(set->flags)) { + __blk_mq_free_rqs(set, 0, set->queue_depth, &set->page_list, + set->static_rqs); + } +out_free_static_rqs: + if (blk_mq_is_sbitmap_shared(set->flags)) { + kfree(set->static_rqs); + set->static_rqs = NULL; + } return -ENOMEM; } @@ -3601,12 +3646,17 @@ void blk_mq_free_tag_set(struct blk_mq_tag_set *set) { int i, j; + if (blk_mq_is_sbitmap_shared(set->flags)) { + blk_mq_exit_shared_sbitmap(set); + __blk_mq_free_rqs(set, 0, set->queue_depth, &set->page_list, + set->static_rqs); + kfree(set->static_rqs); + set->static_rqs = NULL; + } + for (i = 0; i < set->nr_hw_queues; i++) blk_mq_free_map_and_requests(set, i); - if (blk_mq_is_sbitmap_shared(set->flags)) - blk_mq_exit_shared_sbitmap(set); - for (j = 0; j < set->nr_maps; j++) { kfree(set->map[j].mq_map); set->map[j].mq_map = NULL; diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 1d18447ebebc..68b648ab5435 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -261,6 +261,10 @@ struct blk_mq_tag_set { struct sbitmap_queue __breserved_tags; struct blk_mq_tags **tags; + /* for shared sbitmap */ + struct request **static_rqs; + struct list_head page_list; + struct mutex tag_list_lock; struct list_head tag_list; };