From patchwork Thu Apr 29 17:33:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jianxiong Gao X-Patchwork-Id: 429717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.2 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88695C433ED for ; Thu, 29 Apr 2021 17:34:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5B1B861457 for ; Thu, 29 Apr 2021 17:34:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233706AbhD2Rfa (ORCPT ); Thu, 29 Apr 2021 13:35:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53464 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232991AbhD2Rfa (ORCPT ); Thu, 29 Apr 2021 13:35:30 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E43EC06138B for ; Thu, 29 Apr 2021 10:34:42 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id w9-20020a17090a4609b0290150d07df879so15334431pjg.9 for ; Thu, 29 Apr 2021 10:34:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=gUujUgkguLuTeeJSnMWCbPSuzJnLLvExvqc94TrwfjA=; b=ZxHHriHCwdMbqE0BtqCzyqooUbIpuy88X3ppS3mPgzpZ7i2tdo/VQIrnB/HgkT8XS/ xwjjZK5c12AwvWysw1AAVXZIYTk4fqUy+xPdAVF60tI4DkvChnvM9FukBONHVQ0yX/sN 2lFT4CTeGvirWJoLfrh7WBfGitmHBuIBVGu4qj4XruxTibsd5hXbBMCwgYKwo1vcMLuZ Z6U+KwfoQ6ToZpWHLVrhX17Ie9daXA7KQfjgQ6CH06qdTWaFAepjbEZW5B8TVDgChlrK JxzZcher/cehQnjvldxwg62TQQU0PjkJ8TPVSdb9QuD1CKHBth+ZPwI7Ll5EeHw+cmpv ukxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=gUujUgkguLuTeeJSnMWCbPSuzJnLLvExvqc94TrwfjA=; b=pbZ0OEceRitA/ejZLRGnLmoPsFuY7U4vBtmg8GMABataW91wU1MU9831zBeJSOICHr qhB7GEJc1y+fQQW5lcvWJWUmZn5P9f2RbQYEOPlh2a5uS+qj5SsXMpdIVJGPz2Fx9IF9 R31gNqmgQKqu3HZWzDPTV0PnCisZcrlB2yq9r8yHanBsPfXIN1+vUEECGHAxTkWjliJC Y9yFOeQPf/0QaWC3JFDWJb/TsOIyvxxvwC+FpgoxcqungFg+QZo96cFHsRGPzKVzW6Qs tzI3q7zY0qG1+Mv9eVCJYUMZFjf6nxGzeMrBh0COkcHdJFckOjxV4sIanoXAiqyWuf/O MtFA== X-Gm-Message-State: AOAM531M4fck62t5Yy7zkJOYq2MA5PWLUBoQwt7kT02RAXMQ2WabacXw lt88i7+7js4pAD4HKPWMijJjPbXSTXylGOipmMX+gjGpK7fsy+5fA6FV4+Vx/6yL6GTHWohqp7z vGnKWcSKvB60eEd6gr6RY4cBzrCLFcFhtfnntfk4oTuOxoinxZmyEK+q1MlA= X-Google-Smtp-Source: ABdhPJzZ+Pf1fRJJ0k7/6FxkwEAju+10bQgVyTr6eOAQxX4Vc6G8Ybb/N3DMX7V0UMzwNpVrvdtpIheR9Q== X-Received: from jxgao-snp.c.googlers.com ([fda3:e722:ac3:10:7f:e700:c0a8:1373]) (user=jxgao job=sendgmr) by 2002:a17:90a:174c:: with SMTP id 12mr812800pjm.1.1619717681057; Thu, 29 Apr 2021 10:34:41 -0700 (PDT) Date: Thu, 29 Apr 2021 17:33:08 +0000 In-Reply-To: <20210429173315.1252465-1-jxgao@google.com> Message-Id: <20210429173315.1252465-3-jxgao@google.com> Mime-Version: 1.0 References: <20210429173315.1252465-1-jxgao@google.com> X-Mailer: git-send-email 2.31.1.498.g6c1eba8ee3d-goog Subject: [PATCH v2 5.10 2/9] swiotlb: add a IO_TLB_SIZE define From: Jianxiong Gao To: stable@vger.kernel.org, hch@lst.de, marcorr@google.com, sashal@kernel.org Cc: Jianxiong Gao , Konrad Rzeszutek Wilk Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org commit: b5d7ccb7aac3895c2138fe0980a109116ce15eff Add a new IO_TLB_SIZE define instead open coding it using IO_TLB_SHIFT all over. Signed-off-by: Christoph Hellwig Acked-by: Jianxiong Gao Tested-by: Jianxiong Gao Signed-off-by: Konrad Rzeszutek Wilk Signed-off-by: Jianxiong Gao --- include/linux/swiotlb.h | 1 + kernel/dma/swiotlb.c | 12 ++++++------ 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index fbdc65782195..5d2dbe7e04c3 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -29,6 +29,7 @@ enum swiotlb_force { * controllable. */ #define IO_TLB_SHIFT 11 +#define IO_TLB_SIZE (1 << IO_TLB_SHIFT) extern void swiotlb_init(int verbose); int swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose); diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 781b9dca197c..fb88c2982645 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -475,20 +475,20 @@ phys_addr_t swiotlb_tbl_map_single(struct device *hwdev, phys_addr_t orig_addr, tbl_dma_addr &= mask; - offset_slots = ALIGN(tbl_dma_addr, 1 << IO_TLB_SHIFT) >> IO_TLB_SHIFT; + offset_slots = ALIGN(tbl_dma_addr, IO_TLB_SIZE) >> IO_TLB_SHIFT; /* * Carefully handle integer overflow which can occur when mask == ~0UL. */ max_slots = mask + 1 - ? ALIGN(mask + 1, 1 << IO_TLB_SHIFT) >> IO_TLB_SHIFT + ? ALIGN(mask + 1, IO_TLB_SIZE) >> IO_TLB_SHIFT : 1UL << (BITS_PER_LONG - IO_TLB_SHIFT); /* * For mappings greater than or equal to a page, we limit the stride * (and hence alignment) to a page size. */ - nslots = ALIGN(alloc_size, 1 << IO_TLB_SHIFT) >> IO_TLB_SHIFT; + nslots = ALIGN(alloc_size, IO_TLB_SIZE) >> IO_TLB_SHIFT; if (alloc_size >= PAGE_SIZE) stride = (1 << (PAGE_SHIFT - IO_TLB_SHIFT)); else @@ -582,7 +582,7 @@ void swiotlb_tbl_unmap_single(struct device *hwdev, phys_addr_t tlb_addr, enum dma_data_direction dir, unsigned long attrs) { unsigned long flags; - int i, count, nslots = ALIGN(alloc_size, 1 << IO_TLB_SHIFT) >> IO_TLB_SHIFT; + int i, count, nslots = ALIGN(alloc_size, IO_TLB_SIZE) >> IO_TLB_SHIFT; int index = (tlb_addr - io_tlb_start) >> IO_TLB_SHIFT; phys_addr_t orig_addr = io_tlb_orig_addr[index]; @@ -633,7 +633,7 @@ void swiotlb_tbl_sync_single(struct device *hwdev, phys_addr_t tlb_addr, if (orig_addr == INVALID_PHYS_ADDR) return; - orig_addr += (unsigned long)tlb_addr & ((1 << IO_TLB_SHIFT) - 1); + orig_addr += (unsigned long)tlb_addr & (IO_TLB_SIZE - 1); switch (target) { case SYNC_FOR_CPU: @@ -691,7 +691,7 @@ dma_addr_t swiotlb_map(struct device *dev, phys_addr_t paddr, size_t size, size_t swiotlb_max_mapping_size(struct device *dev) { - return ((size_t)1 << IO_TLB_SHIFT) * IO_TLB_SEGSIZE; + return ((size_t)IO_TLB_SIZE) * IO_TLB_SEGSIZE; } bool is_swiotlb_active(void) From patchwork Thu Apr 29 17:33:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jianxiong Gao X-Patchwork-Id: 429716 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.2 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05740C433B4 for ; Thu, 29 Apr 2021 17:34:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C7CD06144E for ; Thu, 29 Apr 2021 17:34:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237244AbhD2Rfd (ORCPT ); Thu, 29 Apr 2021 13:35:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53482 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232991AbhD2Rfc (ORCPT ); Thu, 29 Apr 2021 13:35:32 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96633C06138B for ; Thu, 29 Apr 2021 10:34:45 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id n129-20020a2527870000b02904ed02e1aab5so34989839ybn.21 for ; Thu, 29 Apr 2021 10:34:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=ADgLt44c5UjxWVqp7l4awTjeHsx68ICN3sgpsWTgLCM=; b=fmFkVaaTEj9oqCMFYeK3hEkaOggf9AohCYtKb5BFA0nTXhopx9nsvN/tFtlWcCPfOU BRSkM5/EzVkCK9P7slYto0Kb85fjgOQ/ZgYZXo5dCViL3zUZp+kBjY8NR+y9G/OPiSWX DQpnPrWrOHp2OsSinTIEdKv+ON/8y+zvkiyE1lhqpou9ZiL4sLRbMoHyApR9KHrMsm2x Qv10gq1R1/sFkUrkMyfmg8M/DsuwEn51tiTDCxPfprrhFSUARF3PhRXRCjW7GCFovpSn JGaZflgKq2TO49mMuW2Yg7d56axyHHXjE5fdANVRL4sl27H6cmCD9CkYJRgM1gYVQ32+ b9nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ADgLt44c5UjxWVqp7l4awTjeHsx68ICN3sgpsWTgLCM=; b=YwQxfDDecIsig3UD4f2GXDd+nrvzhGhZRNZjtd5NBDiCjsj0Q311kqwq1ICrwK06+A ltFGhumbR0XmUrGFo0JWZpFodZMWjc3AzuyL7egL+JZ/TkmDRa9eAPmo+M58jM6XehbR fFgI9qLqCNzwT/uZGDTW+qQoFguBXHaF9LoyGn5a40ZpDREsxna3cbnvZceEvriSV52D UDpyJ/P9TgkXzJY+w4Hn9a//KKS88FpztnN8xroktqRxvcmW9VLrftJo+BSWZ7EBGDNc gC7kbu/mnili/nKFYngLz4smCF7uRXNqd43R73+dBB9RL1SC6ntN3+Y2sn/OGkEXjQT8 rj9w== X-Gm-Message-State: AOAM532vfd7+q9kWiLln234FTZ+xuSBdX1APvWh7GYsVWarn6L3Nr03Q Ygs9x65WtkvRYHC7AtQ64zJEHEvYMT+D/g72FoJ8WHG9Rb8LJboFM5HHHz6wUYgTXWFx/NfojB1 1X7ouM8KLHy/Q5a4J9jmdELaXCRLBLKfBYrZ5X/SFZ8FNSD00S08Vz1gynF0= X-Google-Smtp-Source: ABdhPJxXvGOXvp1D1l4PNaSv33kPclxOBWr7ZpskG7x5zsgZVBAEmqzu18NMvDi3tvMXwQ4UCZVAeHFDwg== X-Received: from jxgao-snp.c.googlers.com ([fda3:e722:ac3:10:7f:e700:c0a8:1373]) (user=jxgao job=sendgmr) by 2002:a5b:191:: with SMTP id r17mr892679ybl.297.1619717684666; Thu, 29 Apr 2021 10:34:44 -0700 (PDT) Date: Thu, 29 Apr 2021 17:33:10 +0000 In-Reply-To: <20210429173315.1252465-1-jxgao@google.com> Message-Id: <20210429173315.1252465-5-jxgao@google.com> Mime-Version: 1.0 References: <20210429173315.1252465-1-jxgao@google.com> X-Mailer: git-send-email 2.31.1.498.g6c1eba8ee3d-goog Subject: [PATCH v2 5.10 4/9] swiotlb: factor out a nr_slots helper From: Jianxiong Gao To: stable@vger.kernel.org, hch@lst.de, marcorr@google.com, sashal@kernel.org Cc: Jianxiong Gao , Konrad Rzeszutek Wilk Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org commit: c32a77fd18780a5192dfb6eec69f239faebf28fd Factor out a helper to find the number of slots for a given size. Signed-off-by: Christoph Hellwig Acked-by: Jianxiong Gao Tested-by: Jianxiong Gao Signed-off-by: Konrad Rzeszutek Wilk Signed-off-by: Jianxiong Gao --- kernel/dma/swiotlb.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index f55248f16366..f0be199da527 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -178,6 +178,11 @@ static inline unsigned long io_tlb_offset(unsigned long val) return val & (IO_TLB_SEGSIZE - 1); } +static inline unsigned long nr_slots(u64 val) +{ + return DIV_ROUND_UP(val, IO_TLB_SIZE); +} + /* * Early SWIOTLB allocation may be too early to allow an architecture to * perform the desired operations. This function allows the architecture to @@ -477,20 +482,20 @@ phys_addr_t swiotlb_tbl_map_single(struct device *hwdev, phys_addr_t orig_addr, tbl_dma_addr &= mask; - offset_slots = ALIGN(tbl_dma_addr, IO_TLB_SIZE) >> IO_TLB_SHIFT; + offset_slots = nr_slots(tbl_dma_addr); /* * Carefully handle integer overflow which can occur when mask == ~0UL. */ max_slots = mask + 1 - ? ALIGN(mask + 1, IO_TLB_SIZE) >> IO_TLB_SHIFT + ? nr_slots(mask + 1) : 1UL << (BITS_PER_LONG - IO_TLB_SHIFT); /* * For mappings greater than or equal to a page, we limit the stride * (and hence alignment) to a page size. */ - nslots = ALIGN(alloc_size, IO_TLB_SIZE) >> IO_TLB_SHIFT; + nslots = nr_slots(alloc_size); if (alloc_size >= PAGE_SIZE) stride = (1 << (PAGE_SHIFT - IO_TLB_SHIFT)); else @@ -586,7 +591,7 @@ void swiotlb_tbl_unmap_single(struct device *hwdev, phys_addr_t tlb_addr, enum dma_data_direction dir, unsigned long attrs) { unsigned long flags; - int i, count, nslots = ALIGN(alloc_size, IO_TLB_SIZE) >> IO_TLB_SHIFT; + int i, count, nslots = nr_slots(alloc_size); int index = (tlb_addr - io_tlb_start) >> IO_TLB_SHIFT; phys_addr_t orig_addr = io_tlb_orig_addr[index]; From patchwork Thu Apr 29 17:33:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jianxiong Gao X-Patchwork-Id: 429715 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.2 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7516C433B4 for ; Thu, 29 Apr 2021 17:34:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B5CD561457 for ; Thu, 29 Apr 2021 17:34:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240832AbhD2Rfg (ORCPT ); Thu, 29 Apr 2021 13:35:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236036AbhD2Rfg (ORCPT ); Thu, 29 Apr 2021 13:35:36 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A7E9EC06138B for ; Thu, 29 Apr 2021 10:34:48 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id t6-20020a170902a5c6b02900e8dfb1a5d7so29191368plq.3 for ; Thu, 29 Apr 2021 10:34:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=0GNrkxT81z5oKYj/AtWk+Mm49meVhMgZxhQBU3MwaSs=; b=ahVR4fKlkdXsEmaOs6tBkSbUHE3zQ3WUCNWc4Fy2Sm/nFfmFGg7js72OtIlcu/yDwc BV86gkdY1b9E/tiSL1PIzFIl1j8w7FMGXQdj0sP+fl3wn1Nj0jgww4WRZJrZV+m0+P3U DY7QUuhf4D3Qn+rcWTiMn8lpbDQyPcuDEyaRRjY+14k+pafW3gt5vo8lSYYTS97PcHdx rB97GrCEfSoOjqJm/wSTLT/s/t4I1Ca4LxBZelGrPWHrL2qvRvRpUs143z2MP7rJ99J8 6+gUEJlS7X63PH0T3Hq8lm+ksYVEecf0QAxZfhv7WvZb+qFSlwUAtLKjPvuVThG1GCqx zIHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=0GNrkxT81z5oKYj/AtWk+Mm49meVhMgZxhQBU3MwaSs=; b=j3JTtnD8CxdIdrAj7rQOgeEd4HR2po/NTNoaCRoZVEPBzM+sGemmmTpWQcrIPt4LXM tU2Eup9ssIdLLT+oDN70dlsOh8CyR4ehegQiD5F4pWETrohWa95dqpciXnyRREevXU7W Cfb4Fs1F9LhuwO44px0YkRVGcLFB5ggXbCeM0RoE7qMvDCkUlY0aSqwpHWO/anHyHQar e4e8rgjwfYzhkstwKZguYKENReNww6+6O/PTFjAvA65w0LuJebfOKQK7a6LD/tBBN8Sq BGg1iUjTRQCoQPv9jZjL8QoujenFu9UozPZ2xKU9c2R6ukUYlfPfxRiIDz/hUZ5QLaQy WHjw== X-Gm-Message-State: AOAM530YEnVbRmBAeUGomOlJzvUzIqJqjZsuBskVwk5/RmyJlnRD0N0A voBkIRm3XI3b+7DtBCTkV5+0UQc7t29Q0WW5oWqL131vGCElRpFIFv88SJFsufu4NgREHKXFUp+ YDleSDvBFOvnBjocTSimiVZ+D7Ye1FEmWVFCtLBb34pfZge0pv3DtEgIcit4= X-Google-Smtp-Source: ABdhPJz/srN3TWvb+hyGXA/lB9FX2/FgP/e6U7LdXoD29aXvO3Hlrf+SoWRpbn+YFux5kRV/mZwMbbhwoA== X-Received: from jxgao-snp.c.googlers.com ([fda3:e722:ac3:10:7f:e700:c0a8:1373]) (user=jxgao job=sendgmr) by 2002:aa7:80c9:0:b029:249:cac5:e368 with SMTP id a9-20020aa780c90000b0290249cac5e368mr792703pfn.12.1619717688101; Thu, 29 Apr 2021 10:34:48 -0700 (PDT) Date: Thu, 29 Apr 2021 17:33:12 +0000 In-Reply-To: <20210429173315.1252465-1-jxgao@google.com> Message-Id: <20210429173315.1252465-7-jxgao@google.com> Mime-Version: 1.0 References: <20210429173315.1252465-1-jxgao@google.com> X-Mailer: git-send-email 2.31.1.498.g6c1eba8ee3d-goog Subject: [PATCH v2 5.10 6/9] swiotlb: refactor swiotlb_tbl_map_single From: Jianxiong Gao To: stable@vger.kernel.org, hch@lst.de, marcorr@google.com, sashal@kernel.org Cc: Jianxiong Gao , Konrad Rzeszutek Wilk Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org commit: 26a7e094783d482f3e125f09945a5bb1d867b2e6 Split out a bunch of a self-contained helpers to make the function easier to follow. Signed-off-by: Christoph Hellwig Acked-by: Jianxiong Gao Tested-by: Jianxiong Gao Signed-off-by: Konrad Rzeszutek Wilk Signed-off-by: Jianxiong Gao --- kernel/dma/swiotlb.c | 179 +++++++++++++++++++++---------------------- 1 file changed, 89 insertions(+), 90 deletions(-) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index f5530336d7ca..243750453b3d 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -452,134 +452,133 @@ static void swiotlb_bounce(phys_addr_t orig_addr, phys_addr_t tlb_addr, } } -phys_addr_t swiotlb_tbl_map_single(struct device *hwdev, phys_addr_t orig_addr, - size_t mapping_size, size_t alloc_size, - enum dma_data_direction dir, unsigned long attrs) -{ - dma_addr_t tbl_dma_addr = phys_to_dma_unencrypted(hwdev, io_tlb_start); - unsigned long flags; - phys_addr_t tlb_addr; - unsigned int nslots, stride, index, wrap; - int i; - unsigned long mask; - unsigned long offset_slots; - unsigned long max_slots; - unsigned long tmp_io_tlb_used; - - if (no_iotlb_memory) - panic("Can not allocate SWIOTLB buffer earlier and can't now provide you with the DMA bounce buffer"); - - if (mem_encrypt_active()) - pr_warn_once("Memory encryption is active and system is using DMA bounce buffers\n"); +#define slot_addr(start, idx) ((start) + ((idx) << IO_TLB_SHIFT)) - if (mapping_size > alloc_size) { - dev_warn_once(hwdev, "Invalid sizes (mapping: %zd bytes, alloc: %zd bytes)", - mapping_size, alloc_size); - return (phys_addr_t)DMA_MAPPING_ERROR; - } - - mask = dma_get_seg_boundary(hwdev); +/* + * Carefully handle integer overflow which can occur when boundary_mask == ~0UL. + */ +static inline unsigned long get_max_slots(unsigned long boundary_mask) +{ + if (boundary_mask == ~0UL) + return 1UL << (BITS_PER_LONG - IO_TLB_SHIFT); + return nr_slots(boundary_mask + 1); +} - tbl_dma_addr &= mask; +static unsigned int wrap_index(unsigned int index) +{ + if (index >= io_tlb_nslabs) + return 0; + return index; +} - offset_slots = nr_slots(tbl_dma_addr); +/* + * Find a suitable number of IO TLB entries size that will fit this request and + * allocate a buffer from that IO TLB pool. + */ +static int find_slots(struct device *dev, size_t alloc_size) +{ + unsigned long boundary_mask = dma_get_seg_boundary(dev); + dma_addr_t tbl_dma_addr = + phys_to_dma_unencrypted(dev, io_tlb_start) & boundary_mask; + unsigned long max_slots = get_max_slots(boundary_mask); + unsigned int nslots = nr_slots(alloc_size), stride = 1; + unsigned int index, wrap, count = 0, i; + unsigned long flags; - /* - * Carefully handle integer overflow which can occur when mask == ~0UL. - */ - max_slots = mask + 1 - ? nr_slots(mask + 1) - : 1UL << (BITS_PER_LONG - IO_TLB_SHIFT); + BUG_ON(!nslots); /* * For mappings greater than or equal to a page, we limit the stride * (and hence alignment) to a page size. */ - nslots = nr_slots(alloc_size); if (alloc_size >= PAGE_SIZE) - stride = (1 << (PAGE_SHIFT - IO_TLB_SHIFT)); - else - stride = 1; + stride <<= (PAGE_SHIFT - IO_TLB_SHIFT); - BUG_ON(!nslots); - - /* - * Find suitable number of IO TLB entries size that will fit this - * request and allocate a buffer from that IO TLB pool. - */ spin_lock_irqsave(&io_tlb_lock, flags); - if (unlikely(nslots > io_tlb_nslabs - io_tlb_used)) goto not_found; - index = ALIGN(io_tlb_index, stride); - if (index >= io_tlb_nslabs) - index = 0; - wrap = index; - + index = wrap = wrap_index(ALIGN(io_tlb_index, stride)); do { - while (iommu_is_span_boundary(index, nslots, offset_slots, - max_slots)) { - index += stride; - if (index >= io_tlb_nslabs) - index = 0; - if (index == wrap) - goto not_found; - } - /* * If we find a slot that indicates we have 'nslots' number of * contiguous buffers, we allocate the buffers from that slot * and mark the entries as '0' indicating unavailable. */ - if (io_tlb_list[index] >= nslots) { - int count = 0; - - for (i = index; i < (int) (index + nslots); i++) - io_tlb_list[i] = 0; - for (i = index - 1; - io_tlb_offset(i) != IO_TLB_SEGSIZE - 1 && - io_tlb_list[i]; i--) - io_tlb_list[i] = ++count; - tlb_addr = io_tlb_start + (index << IO_TLB_SHIFT); - - /* - * Update the indices to avoid searching in the next - * round. - */ - io_tlb_index = ((index + nslots) < io_tlb_nslabs - ? (index + nslots) : 0); - - goto found; + if (!iommu_is_span_boundary(index, nslots, + nr_slots(tbl_dma_addr), + max_slots)) { + if (io_tlb_list[index] >= nslots) + goto found; } - index += stride; - if (index >= io_tlb_nslabs) - index = 0; + index = wrap_index(index + stride); } while (index != wrap); not_found: - tmp_io_tlb_used = io_tlb_used; - spin_unlock_irqrestore(&io_tlb_lock, flags); - if (!(attrs & DMA_ATTR_NO_WARN) && printk_ratelimit()) - dev_warn(hwdev, "swiotlb buffer is full (sz: %zd bytes), total %lu (slots), used %lu (slots)\n", - alloc_size, io_tlb_nslabs, tmp_io_tlb_used); - return (phys_addr_t)DMA_MAPPING_ERROR; + return -1; + found: + for (i = index; i < index + nslots; i++) + io_tlb_list[i] = 0; + for (i = index - 1; + io_tlb_offset(i) != IO_TLB_SEGSIZE - 1 && + io_tlb_list[i]; i--) + io_tlb_list[i] = ++count; + + /* + * Update the indices to avoid searching in the next round. + */ + if (index + nslots < io_tlb_nslabs) + io_tlb_index = index + nslots; + else + io_tlb_index = 0; io_tlb_used += nslots; + spin_unlock_irqrestore(&io_tlb_lock, flags); + return index; +} + +phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr, + size_t mapping_size, size_t alloc_size, + enum dma_data_direction dir, unsigned long attrs) +{ + unsigned int index, i; + phys_addr_t tlb_addr; + + if (no_iotlb_memory) + panic("Can not allocate SWIOTLB buffer earlier and can't now provide you with the DMA bounce buffer"); + + if (mem_encrypt_active()) + pr_warn_once("Memory encryption is active and system is using DMA bounce buffers\n"); + + if (mapping_size > alloc_size) { + dev_warn_once(dev, "Invalid sizes (mapping: %zd bytes, alloc: %zd bytes)", + mapping_size, alloc_size); + return (phys_addr_t)DMA_MAPPING_ERROR; + } + + index = find_slots(dev, alloc_size); + if (index == -1) { + if (!(attrs & DMA_ATTR_NO_WARN)) + dev_warn_ratelimited(dev, + "swiotlb buffer is full (sz: %zd bytes), total %lu (slots), used %lu (slots)\n", + alloc_size, io_tlb_nslabs, io_tlb_used); + return (phys_addr_t)DMA_MAPPING_ERROR; + } /* * Save away the mapping from the original address to the DMA address. * This is needed when we sync the memory. Then we sync the buffer if * needed. */ - for (i = 0; i < nslots; i++) - io_tlb_orig_addr[index+i] = orig_addr + (i << IO_TLB_SHIFT); + for (i = 0; i < nr_slots(alloc_size); i++) + io_tlb_orig_addr[index + i] = slot_addr(orig_addr, i); + + tlb_addr = slot_addr(io_tlb_start, index); if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC) && (dir == DMA_TO_DEVICE || dir == DMA_BIDIRECTIONAL)) swiotlb_bounce(orig_addr, tlb_addr, mapping_size, DMA_TO_DEVICE); - return tlb_addr; } From patchwork Thu Apr 29 17:33:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jianxiong Gao X-Patchwork-Id: 429714 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.2 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30DD9C433B4 for ; Thu, 29 Apr 2021 17:34:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 047C161457 for ; Thu, 29 Apr 2021 17:34:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240856AbhD2Rfk (ORCPT ); Thu, 29 Apr 2021 13:35:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53510 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236036AbhD2Rfj (ORCPT ); Thu, 29 Apr 2021 13:35:39 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B64C0C06138B for ; Thu, 29 Apr 2021 10:34:51 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id c2-20020a1709027242b02900e9636a90acso29231450pll.12 for ; Thu, 29 Apr 2021 10:34:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=P+y0Cx6YlUcuB23DD5kQr2szixapkEODz5+veEbGVLo=; b=e+X87rCeApjEWuD9YWo5Ejv5aehOiYGuDUYK/lxddBwshBY9xPI/yUVMxtuf80ve7+ 8ronD5vd5KLQm3MXck5yERca901kOHUqN+cwyOpIaKubkE+KDTamNM0RmFXbq29FoqI7 Ci9M4hsZcotbgoTrc050L3ZIRTvTFh84LHXFT0IIYQ7cdaKXFyOOrpjHPiVjClT4EFYS aRm0TupS0a04pNfAlqYugMxugsSmgiKg8uk1U8H7JRkOVTRkfdt6xOoSu1oUYMhDYM5d GRdEM9pQbBEe6M4WP3stYstFg7bvC1KlZNqXNUnOdoiTSuz6KRGrcTtWxhw/etM+Kjbz pZUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=P+y0Cx6YlUcuB23DD5kQr2szixapkEODz5+veEbGVLo=; b=RbprEJcr3fcIkSLEyM9lIr+HwHoKAwKAGp7CVRysfkMg+ksQLJEIU+GMzulXUhUMvJ yArz7uxemdgO/1CAQgAx3x6jqZGQ5w+rPUVk2uhyQYxIyDHazSvm0sUjXZ4n0fmTgUp2 FiT83GxDVCOAifjuyvo3y0Ljp6UPGUrz3fqbC7EwetKOe4zLGUVhd5GrBdVgWdBdjCMV LubdVHMIJqXr3m6q0isr7OhWlSdnLYf7dz2Rb40SKuzQocunb6a/wVuU7r0L+WGJWM8I ICNONDJKiJu24Ybe0COjP00pIGWGgaAXnINII28RG6ZSemBYpYLQW2o9kdWqWzoHaPGR KoFg== X-Gm-Message-State: AOAM533ZWBcNI3gpYiywWN62BLOooVtrp6obW2GQmPAhRi9px2ozNfu2 kFpRTE9H6LtSqKTbrWpYUP5qktNRpvh1n4zAafVpZ/4Nanzl6hbVSsAdBRhdE/Tqbr/1plPYJfn /pmqr7vOAXhi9A2pYz8VfQNz7HP/gb2vOkE7breGygQJjSvprz9Sh0cAzbkg= X-Google-Smtp-Source: ABdhPJwHsjs4p1YVOF5dY0oTz8EVuQDNh/HjTXZrHAA+T4+yXU+c4HX9+Fmo9WJxM/WZJA4kH1IfUbsBZQ== X-Received: from jxgao-snp.c.googlers.com ([fda3:e722:ac3:10:7f:e700:c0a8:1373]) (user=jxgao job=sendgmr) by 2002:a17:902:e892:b029:ec:d257:b8a2 with SMTP id w18-20020a170902e892b02900ecd257b8a2mr749353plg.15.1619717691204; Thu, 29 Apr 2021 10:34:51 -0700 (PDT) Date: Thu, 29 Apr 2021 17:33:14 +0000 In-Reply-To: <20210429173315.1252465-1-jxgao@google.com> Message-Id: <20210429173315.1252465-9-jxgao@google.com> Mime-Version: 1.0 References: <20210429173315.1252465-1-jxgao@google.com> X-Mailer: git-send-email 2.31.1.498.g6c1eba8ee3d-goog Subject: [PATCH v2 5.10 8/9] swiotlb: respect min_align_mask From: Jianxiong Gao To: stable@vger.kernel.org, hch@lst.de, marcorr@google.com, sashal@kernel.org Cc: Jianxiong Gao , Konrad Rzeszutek Wilk Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org commit: 1f221a0d0dbf0e48ef3a9c62871281d6a7819f05 swiotlb: respect min_align_mask Respect the min_align_mask in struct device_dma_parameters in swiotlb. There are two parts to it: 1) for the lower bits of the alignment inside the io tlb slot, just extent the size of the allocation and leave the start of the slot empty 2) for the high bits ensure we find a slot that matches the high bits of the alignment to avoid wasting too much memory Based on an earlier patch from Jianxiong Gao . Signed-off-by: Christoph Hellwig Acked-by: Jianxiong Gao Tested-by: Jianxiong Gao Signed-off-by: Konrad Rzeszutek Wilk Signed-off-by: Jianxiong Gao --- kernel/dma/swiotlb.c | 41 +++++++++++++++++++++++++++++++---------- 1 file changed, 31 insertions(+), 10 deletions(-) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index b8f82f96c4c2..ba4055a192e4 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -454,6 +454,14 @@ static void swiotlb_bounce(phys_addr_t orig_addr, phys_addr_t tlb_addr, #define slot_addr(start, idx) ((start) + ((idx) << IO_TLB_SHIFT)) +/* + * Return the offset into a iotlb slot required to keep the device happy. + */ +static unsigned int swiotlb_align_offset(struct device *dev, u64 addr) +{ + return addr & dma_get_min_align_mask(dev) & (IO_TLB_SIZE - 1); +} + /* * Carefully handle integer overflow which can occur when boundary_mask == ~0UL. */ @@ -475,24 +483,29 @@ static unsigned int wrap_index(unsigned int index) * Find a suitable number of IO TLB entries size that will fit this request and * allocate a buffer from that IO TLB pool. */ -static int find_slots(struct device *dev, size_t alloc_size) +static int find_slots(struct device *dev, phys_addr_t orig_addr, + size_t alloc_size) { unsigned long boundary_mask = dma_get_seg_boundary(dev); dma_addr_t tbl_dma_addr = phys_to_dma_unencrypted(dev, io_tlb_start) & boundary_mask; unsigned long max_slots = get_max_slots(boundary_mask); - unsigned int nslots = nr_slots(alloc_size), stride = 1; + unsigned int iotlb_align_mask = + dma_get_min_align_mask(dev) & ~(IO_TLB_SIZE - 1); + unsigned int nslots = nr_slots(alloc_size), stride; unsigned int index, wrap, count = 0, i; unsigned long flags; BUG_ON(!nslots); /* - * For mappings greater than or equal to a page, we limit the stride - * (and hence alignment) to a page size. + * For mappings with an alignment requirement don't bother looping to + * unaligned slots once we found an aligned one. For allocations of + * PAGE_SIZE or larger only look for page aligned allocations. */ + stride = (iotlb_align_mask >> IO_TLB_SHIFT) + 1; if (alloc_size >= PAGE_SIZE) - stride <<= (PAGE_SHIFT - IO_TLB_SHIFT); + stride = max(stride, stride << (PAGE_SHIFT - IO_TLB_SHIFT)); spin_lock_irqsave(&io_tlb_lock, flags); if (unlikely(nslots > io_tlb_nslabs - io_tlb_used)) @@ -500,6 +513,12 @@ static int find_slots(struct device *dev, size_t alloc_size) index = wrap = wrap_index(ALIGN(io_tlb_index, stride)); do { + if ((slot_addr(tbl_dma_addr, index) & iotlb_align_mask) != + (orig_addr & iotlb_align_mask)) { + index = wrap_index(index + 1); + continue; + } + /* * If we find a slot that indicates we have 'nslots' number of * contiguous buffers, we allocate the buffers from that slot @@ -543,6 +562,7 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr, size_t mapping_size, size_t alloc_size, enum dma_data_direction dir, unsigned long attrs) { + unsigned int offset = swiotlb_align_offset(dev, orig_addr); unsigned int index, i; phys_addr_t tlb_addr; @@ -558,7 +578,7 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr, return (phys_addr_t)DMA_MAPPING_ERROR; } - index = find_slots(dev, alloc_size); + index = find_slots(dev, orig_addr, alloc_size + offset); if (index == -1) { if (!(attrs & DMA_ATTR_NO_WARN)) dev_warn_ratelimited(dev, @@ -572,10 +592,10 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr, * This is needed when we sync the memory. Then we sync the buffer if * needed. */ - for (i = 0; i < nr_slots(alloc_size); i++) + for (i = 0; i < nr_slots(alloc_size + offset); i++) io_tlb_orig_addr[index + i] = slot_addr(orig_addr, i); - tlb_addr = slot_addr(io_tlb_start, index); + tlb_addr = slot_addr(io_tlb_start, index) + offset; if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC) && (dir == DMA_TO_DEVICE || dir == DMA_BIDIRECTIONAL)) swiotlb_bounce(orig_addr, tlb_addr, mapping_size, DMA_TO_DEVICE); @@ -590,8 +610,9 @@ void swiotlb_tbl_unmap_single(struct device *hwdev, phys_addr_t tlb_addr, enum dma_data_direction dir, unsigned long attrs) { unsigned long flags; - int i, count, nslots = nr_slots(alloc_size); - int index = (tlb_addr - io_tlb_start) >> IO_TLB_SHIFT; + unsigned int offset = swiotlb_align_offset(hwdev, tlb_addr); + int i, count, nslots = nr_slots(alloc_size + offset); + int index = (tlb_addr - offset - io_tlb_start) >> IO_TLB_SHIFT; phys_addr_t orig_addr = io_tlb_orig_addr[index]; /*