From patchwork Tue Mar 2 17:39:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Orr X-Patchwork-Id: 393449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.2 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A951C28D14 for ; Wed, 3 Mar 2021 01:02:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 598EF64FA0 for ; Wed, 3 Mar 2021 01:02:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241191AbhCCAwZ (ORCPT ); Tue, 2 Mar 2021 19:52:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343978AbhCBRkL (ORCPT ); Tue, 2 Mar 2021 12:40:11 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5953AC06121D for ; Tue, 2 Mar 2021 09:39:30 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id q77so23415111ybq.0 for ; Tue, 02 Mar 2021 09:39:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:message-id:mime-version:subject:from:to:cc; bh=2kGfAvKyVkw+dh3SbBOor4fSKz60nTkeOupR2omHaQY=; b=H/pVjHNX6DHeY2l7ihNS7/gAAV93fdgM9k+ezvRFKDASCvfKexsfMS5E0FRRWp8Lvc Bv2udFF3eMCd3O0VrTVmO4qh3mkjqIdZJ+vRj3rmpL/R5D7IWUOYTldxjnyPbTQaNgQH ZWsdZm5cMS6EtBMwQMAOt8FrOzYOhiMOnxanXDlfh5X1br45KDUOYyyZyX84IQkn2zxW iLJSBYktAWcbPYJFNN6Eu8OnDLts+sOb8YCXJK6c/tE30v4SE14korlXFd2zdv4e1Frb o03GxlVD0MiVImdnIp7FXn+Tv27/YnyesZqfo3X1WT7mxLmKAa9MmdJz/WPWVWrLRyss HAdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:message-id:mime-version:subject:from :to:cc; bh=2kGfAvKyVkw+dh3SbBOor4fSKz60nTkeOupR2omHaQY=; b=VVCWMJrGuHlrxe5FGktvifpHnGOedFPnAfyO2fcVty10cTVnCOP0lftQD9T3AvABm8 sUKFx0APJG4Cd6z+SYkkR3BuG3wn1ihPPeJqR0WeurngUDnXkfW9rG+u6uS1Ef/AeXLl 0VFZZKeD+Pu1QpeooSKNG9RVdWnwgawHk/2ZC0JhrpEC8tNyZdsnlHhzGMOWJV5OHU9H bevsw/OKjUGwCpUPLJX+OCk4QG0TFBNphYEmx8VYqEq4hG3lgqO+e/+SC6C0oJUcR/N4 C1nqxlJ+V416ymy0SXZ1Cg2G18G720EWcrs4nyNN+i6mdIh7SihpJQ+6pHoetz99DtvH Oxkw== X-Gm-Message-State: AOAM532gTFlsXcUeegOJpz2gm1oTYnbPlvvdGgzhquslHVIYyudk3uzv eD27mjrhxF5OweW9S25ntvKmLT/4P6fUmVcjGzrUFBJRfKPtt7vm5pQ8+LxHVhgrgv4nGfa1J7V ta+m9vPKaPrkZQZXBLefaaYJFfylXkCIBhuFZ9KyYkJah4wvkns71YybKQFOjY57u X-Google-Smtp-Source: ABdhPJw09GA87fJB5L1me8I8jtPf4Zdlt3PDHmKOl3t3WpzqxEnow1ol/HmWDOBEcsrRpTnWB1alziK3CzOi Sender: "marcorr via sendgmr" X-Received: from marcorr-sp.c.googlers.com ([fda3:e722:ac3:10:7f:e700:c0a8:e50]) (user=marcorr job=sendgmr) by 2002:a25:34d2:: with SMTP id b201mr34717490yba.149.1614706769546; Tue, 02 Mar 2021 09:39:29 -0800 (PST) Date: Tue, 2 Mar 2021 17:39:10 +0000 Message-Id: <20210302173911.12044-1-marcorr@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 5.4 1/2] nvme-pci: refactor nvme_unmap_data From: Marc Orr To: stable@vger.kernel.org Cc: Christoph Hellwig , Keith Busch , Marc Orr Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Christoph Hellwig commit 9275c206f88e5c49cb3e71932c81c8561083db9e upstream. Split out three helpers from nvme_unmap_data that will allow finer grained unwinding from nvme_map_data. Signed-off-by: Christoph Hellwig Reviewed-by: Keith Busch Reviewed-by: Marc Orr Signed-off-by: Marc Orr --- drivers/nvme/host/pci.c | 77 ++++++++++++++++++++++++++--------------- 1 file changed, 49 insertions(+), 28 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 19e375b59f40..90dc86132007 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -528,50 +528,71 @@ static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req) return true; } -static void nvme_unmap_data(struct nvme_dev *dev, struct request *req) +static void nvme_free_prps(struct nvme_dev *dev, struct request *req) { - struct nvme_iod *iod = blk_mq_rq_to_pdu(req); const int last_prp = dev->ctrl.page_size / sizeof(__le64) - 1; - dma_addr_t dma_addr = iod->first_dma, next_dma_addr; + struct nvme_iod *iod = blk_mq_rq_to_pdu(req); + dma_addr_t dma_addr = iod->first_dma; int i; - if (iod->dma_len) { - dma_unmap_page(dev->dev, dma_addr, iod->dma_len, - rq_dma_dir(req)); - return; + for (i = 0; i < iod->npages; i++) { + __le64 *prp_list = nvme_pci_iod_list(req)[i]; + dma_addr_t next_dma_addr = le64_to_cpu(prp_list[last_prp]); + + dma_pool_free(dev->prp_page_pool, prp_list, dma_addr); + dma_addr = next_dma_addr; } - WARN_ON_ONCE(!iod->nents); +} - if (is_pci_p2pdma_page(sg_page(iod->sg))) - pci_p2pdma_unmap_sg(dev->dev, iod->sg, iod->nents, - rq_dma_dir(req)); - else - dma_unmap_sg(dev->dev, iod->sg, iod->nents, rq_dma_dir(req)); +static void nvme_free_sgls(struct nvme_dev *dev, struct request *req) +{ + const int last_sg = SGES_PER_PAGE - 1; + struct nvme_iod *iod = blk_mq_rq_to_pdu(req); + dma_addr_t dma_addr = iod->first_dma; + int i; + for (i = 0; i < iod->npages; i++) { + struct nvme_sgl_desc *sg_list = nvme_pci_iod_list(req)[i]; + dma_addr_t next_dma_addr = le64_to_cpu((sg_list[last_sg]).addr); - if (iod->npages == 0) - dma_pool_free(dev->prp_small_pool, nvme_pci_iod_list(req)[0], - dma_addr); + dma_pool_free(dev->prp_page_pool, sg_list, dma_addr); + dma_addr = next_dma_addr; + } - for (i = 0; i < iod->npages; i++) { - void *addr = nvme_pci_iod_list(req)[i]; +} - if (iod->use_sgl) { - struct nvme_sgl_desc *sg_list = addr; +static void nvme_unmap_sg(struct nvme_dev *dev, struct request *req) +{ + struct nvme_iod *iod = blk_mq_rq_to_pdu(req); - next_dma_addr = - le64_to_cpu((sg_list[SGES_PER_PAGE - 1]).addr); - } else { - __le64 *prp_list = addr; + if (is_pci_p2pdma_page(sg_page(iod->sg))) + pci_p2pdma_unmap_sg(dev->dev, iod->sg, iod->nents, + rq_dma_dir(req)); + else + dma_unmap_sg(dev->dev, iod->sg, iod->nents, rq_dma_dir(req)); +} - next_dma_addr = le64_to_cpu(prp_list[last_prp]); - } +static void nvme_unmap_data(struct nvme_dev *dev, struct request *req) +{ + struct nvme_iod *iod = blk_mq_rq_to_pdu(req); - dma_pool_free(dev->prp_page_pool, addr, dma_addr); - dma_addr = next_dma_addr; + if (iod->dma_len) { + dma_unmap_page(dev->dev, iod->first_dma, iod->dma_len, + rq_dma_dir(req)); + return; } + WARN_ON_ONCE(!iod->nents); + + nvme_unmap_sg(dev, req); + if (iod->npages == 0) + dma_pool_free(dev->prp_small_pool, nvme_pci_iod_list(req)[0], + iod->first_dma); + else if (iod->use_sgl) + nvme_free_sgls(dev, req); + else + nvme_free_prps(dev, req); mempool_free(iod->sg, dev->iod_mempool); } From patchwork Tue Mar 2 17:39:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Orr X-Patchwork-Id: 392608 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.2 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAA99C28D15 for ; Wed, 3 Mar 2021 01:02:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 92EC964FA0 for ; Wed, 3 Mar 2021 01:02:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241306AbhCCAwd (ORCPT ); Tue, 2 Mar 2021 19:52:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51018 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344052AbhCBRkx (ORCPT ); Tue, 2 Mar 2021 12:40:53 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4AB7C06121E for ; Tue, 2 Mar 2021 09:39:33 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id 6so23403336ybq.7 for ; Tue, 02 Mar 2021 09:39:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=IvcU72kyzl+dTsNEe1dtDRtwfa+sANJ7uN/5rgWi+VE=; b=tanWmZ0daUkhnlFd1Wz2mvpSifJBQ77dqOX267qv7NxfpALcFmrdmSeVXidrLCJcIk vLxIIOagBZoCOlbyGDAoD+dwDiB8SOhHjh7qhQkvpTQZ657I8EbmgKQP7aqPQdsy46YQ 1AJprK1Nr51MMBdgHTTtThlte/1gHHwV+sz4zTEsj8b8petSG/gF7RE6q3OqsMu+lAwo KMQ2abixt7x8DN6JLiW6HCYXkATqcekTL8sOfX14UGOsou08l/x8l7cFn+pQBFISCKFW XPd+ojktOuW8polM4u8Crnq+KwEyjsgyYANao8G9lkvOzolQjLoIR1AZjLLc7t3EqOzl 8KLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=IvcU72kyzl+dTsNEe1dtDRtwfa+sANJ7uN/5rgWi+VE=; b=hiIoCBhaXLal5Ru8gVq1L4+Rw/yMmjZ1P5mc3Y3tbz3MTPd4YhUPbOSol/fi2Ctn6E F5U19dfNxgE5ctCyzhrcnVMCZXLV8UIrizx4Lsvj0p5oI3V5SWZtwecXi+NZoVveHEgf avzuaSZJxil1LbNHftsictf/lvZGDoV8gX62rRItg8/D2dUAe30IZcWBtVeNIn4IMXcK HxrPaL08uRoCnuXj6QG5zcp5+Ib6WBrYk660m7VQDM0fYPKoN7mGcePsryBcmN3YlKoD 4mhYZWRla1cnqqDYeJQ2Y0/GmYkzkPNA3xp1MZR/5j1t9iwShQ+o01hdmiVVnRF7EAiu FHqw== X-Gm-Message-State: AOAM533I+aUI9+BD2QMOr72s4TUWkROa5R5sDPbCWknZeUJlG4C3RJS0 CEJg5RJBc2exMoENDm8M7dpic+fGBF+mJujAIm4JUf0c85vaWGcPqX+yETmdbRXoZQdQbHZd2og HKQZLKXPq2Dx6Xnl9OCJeD4GlabocXENZmhIgeWts350xaHDVSHNsy540cAHCujkg X-Google-Smtp-Source: ABdhPJzwr2ny8gEk95WMzLEYwQflj/RX4f9B06UXOqJ/UGy0UpLBMfD+8vwi6SEhfBWbS6lTCk1pJqdbKMc0 Sender: "marcorr via sendgmr" X-Received: from marcorr-sp.c.googlers.com ([fda3:e722:ac3:10:7f:e700:c0a8:e50]) (user=marcorr job=sendgmr) by 2002:a25:2389:: with SMTP id j131mr33642338ybj.67.1614706773044; Tue, 02 Mar 2021 09:39:33 -0800 (PST) Date: Tue, 2 Mar 2021 17:39:11 +0000 In-Reply-To: <20210302173911.12044-1-marcorr@google.com> Message-Id: <20210302173911.12044-2-marcorr@google.com> Mime-Version: 1.0 References: <20210302173911.12044-1-marcorr@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 5.4 2/2] nvme-pci: fix error unwind in nvme_map_data From: Marc Orr To: stable@vger.kernel.org Cc: Christoph Hellwig , Marc Orr , Keith Busch Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Christoph Hellwig commit fa0732168fa1369dd089e5b06d6158a68229f7b7 upstream. Properly unwind step by step using refactored helpers from nvme_unmap_data to avoid a potential double dma_unmap on a mapping failure. Fixes: 7fe07d14f71f ("nvme-pci: merge nvme_free_iod into nvme_unmap_data") Reported-by: Marc Orr Signed-off-by: Christoph Hellwig Reviewed-by: Keith Busch Reviewed-by: Marc Orr Signed-off-by: Marc Orr --- drivers/nvme/host/pci.c | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 90dc86132007..abc342db3b33 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -669,7 +669,7 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev, __le64 *old_prp_list = prp_list; prp_list = dma_pool_alloc(pool, GFP_ATOMIC, &prp_dma); if (!prp_list) - return BLK_STS_RESOURCE; + goto free_prps; list[iod->npages++] = prp_list; prp_list[0] = old_prp_list[i - 1]; old_prp_list[i - 1] = cpu_to_le64(prp_dma); @@ -689,14 +689,14 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev, dma_addr = sg_dma_address(sg); dma_len = sg_dma_len(sg); } - done: cmnd->dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg)); cmnd->dptr.prp2 = cpu_to_le64(iod->first_dma); - return BLK_STS_OK; - - bad_sgl: +free_prps: + nvme_free_prps(dev, req); + return BLK_STS_RESOURCE; +bad_sgl: WARN(DO_ONCE(nvme_print_sgl, iod->sg, iod->nents), "Invalid SGL for payload:%d nents:%d\n", blk_rq_payload_bytes(req), iod->nents); @@ -768,7 +768,7 @@ static blk_status_t nvme_pci_setup_sgls(struct nvme_dev *dev, sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma); if (!sg_list) - return BLK_STS_RESOURCE; + goto free_sgls; i = 0; nvme_pci_iod_list(req)[iod->npages++] = sg_list; @@ -781,6 +781,9 @@ static blk_status_t nvme_pci_setup_sgls(struct nvme_dev *dev, } while (--entries > 0); return BLK_STS_OK; +free_sgls: + nvme_free_sgls(dev, req); + return BLK_STS_RESOURCE; } static blk_status_t nvme_setup_prp_simple(struct nvme_dev *dev, @@ -849,7 +852,7 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req, sg_init_table(iod->sg, blk_rq_nr_phys_segments(req)); iod->nents = blk_rq_map_sg(req->q, req, iod->sg); if (!iod->nents) - goto out; + goto out_free_sg; if (is_pci_p2pdma_page(sg_page(iod->sg))) nr_mapped = pci_p2pdma_map_sg_attrs(dev->dev, iod->sg, @@ -858,16 +861,21 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req, nr_mapped = dma_map_sg_attrs(dev->dev, iod->sg, iod->nents, rq_dma_dir(req), DMA_ATTR_NO_WARN); if (!nr_mapped) - goto out; + goto out_free_sg; iod->use_sgl = nvme_pci_use_sgls(dev, req); if (iod->use_sgl) ret = nvme_pci_setup_sgls(dev, req, &cmnd->rw, nr_mapped); else ret = nvme_pci_setup_prps(dev, req, &cmnd->rw); -out: if (ret != BLK_STS_OK) - nvme_unmap_data(dev, req); + goto out_unmap_sg; + return BLK_STS_OK; + +out_unmap_sg: + nvme_unmap_sg(dev, req); +out_free_sg: + mempool_free(iod->sg, dev->iod_mempool); return ret; }