From patchwork Thu Aug 15 12:11:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Leizhen \(ThunderTown\)" X-Patchwork-Id: 171431 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp2103371ily; Thu, 15 Aug 2019 05:11:53 -0700 (PDT) X-Google-Smtp-Source: APXvYqwqqyaJHKuwNYnMzaMR/je6oKIctPK0pzT8ePo6K5UNcb0LS6lURniIeRO7SslFrns5V5g3 X-Received: by 2002:a17:902:1e6:: with SMTP id b93mr3948609plb.295.1565871113598; Thu, 15 Aug 2019 05:11:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1565871113; cv=none; d=google.com; s=arc-20160816; b=HU3WJVV5pdYvLREnMY+uIg1fj8mTZ9DVzHv072fpwedC9NlgxHPo5Ffi2B7q6sKSQ/ j7aW5yF7Z2+28DG8o5NXk6/XzOG55jNTwP4JV9av8nF79/LA3e5fMyB71zbD3LjhYPlQ LJuBAyjQWmF9OwE4wQ6sTr6+1JlzSD+VrdnKd2PHnCNQrUREL1i+UCBjSQq5wnPo8YCJ Klm2aoy1atzTkXMChq8vuQoL8YOKq7QjVxV8vmnQQHc7dJQjLbCseLMELusEVjX3guzT 2YUuolaKdZKfDwVvKU62emL4n1mJwOA0qQj95BKKjz6EPLznf5A8VOMrJE4pLkTUe1Ch zdQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=2RR27ccmOXJMfdagP4Tod2X5aHvvEUUykC8v55+NLXA=; b=FQTBh4IUB2VvezwEAv9lSOFLhuj2bHCVeiIQJWtpRT/woFBAPir+6vrDVHY8pVl3SP vPue0C8UDB9ioqHk/+6cDy4hDVv2lVCsjAKeLV5CvQ4AOgx1KTdnSVwxmjskERB60bRx 3/yWLL6R6BDbwvL+4ExWbDZnALm0LmzbvpyA3OtehEkrOB4X7tPhkHHrd+1KzbADRkL3 LP2mYeO/ovgj+c+AvbLSm5h8XVN9XKtXrY9ITYtxoNbJ4HBG+5npQbiYClkEd8koGvpe q/500idEYWDkXRaeYU+pIqYTLE5TwlO3Gv+4QDEMb0ZM8+X40MGzk/WHcaAXtyAaxAdN VfnQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 131si1737811pgf.597.2019.08.15.05.11.53; Thu, 15 Aug 2019 05:11:53 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731444AbfHOMLw (ORCPT + 28 others); Thu, 15 Aug 2019 08:11:52 -0400 Received: from szxga07-in.huawei.com ([45.249.212.35]:50936 "EHLO huawei.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1730032AbfHOMLv (ORCPT ); Thu, 15 Aug 2019 08:11:51 -0400 Received: from DGGEMS404-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 71EE8288D19D6B69B9DB; Thu, 15 Aug 2019 20:11:40 +0800 (CST) Received: from HGHY4L002753561.china.huawei.com (10.133.215.186) by DGGEMS404-HUB.china.huawei.com (10.3.19.204) with Microsoft SMTP Server id 14.3.439.0; Thu, 15 Aug 2019 20:11:33 +0800 From: Zhen Lei To: Jean-Philippe Brucker , "Jean-Philippe Brucker" , John Garry , "Robin Murphy" , Will Deacon , Joerg Roedel , iommu , Omer Peleg , Adam Morrison , Shaohua Li , Ben Serebrin , David Woodhouse , linux-arm-kernel , linux-kernel CC: Zhen Lei Subject: [PATCH v2 1/2] iommu/iova: introduce iova_magazine_compact_pfns() Date: Thu, 15 Aug 2019 20:11:03 +0800 Message-ID: <20190815121104.29140-2-thunder.leizhen@huawei.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20190815121104.29140-1-thunder.leizhen@huawei.com> References: <20190815121104.29140-1-thunder.leizhen@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.133.215.186] X-CFilter-Loop: Reflected Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org iova_magazine_free_pfns() can only free the whole magazine buffer, add iova_magazine_compact_pfns() to support free part of it. Signed-off-by: Zhen Lei --- drivers/iommu/iova.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) -- 1.8.3 diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c index 3e1a8a6755723a9..4b7a9efa0ef40af 100644 --- a/drivers/iommu/iova.c +++ b/drivers/iommu/iova.c @@ -795,18 +795,19 @@ static void iova_magazine_free(struct iova_magazine *mag) kfree(mag); } -static void -iova_magazine_free_pfns(struct iova_magazine *mag, struct iova_domain *iovad) +static void iova_magazine_compact_pfns(struct iova_magazine *mag, + struct iova_domain *iovad, + unsigned long newsize) { unsigned long flags; int i; - if (!mag) + if (!mag || mag->size <= newsize) return; spin_lock_irqsave(&iovad->iova_rbtree_lock, flags); - for (i = 0 ; i < mag->size; ++i) { + for (i = newsize; i < mag->size; ++i) { struct iova *iova = private_find_iova(iovad, mag->pfns[i]); BUG_ON(!iova); @@ -815,7 +816,13 @@ static void iova_magazine_free(struct iova_magazine *mag) spin_unlock_irqrestore(&iovad->iova_rbtree_lock, flags); - mag->size = 0; + mag->size = newsize; +} + +static void +iova_magazine_free_pfns(struct iova_magazine *mag, struct iova_domain *iovad) +{ + iova_magazine_compact_pfns(mag, iovad, 0); } static bool iova_magazine_full(struct iova_magazine *mag) From patchwork Thu Aug 15 12:11:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Leizhen \(ThunderTown\)" X-Patchwork-Id: 171432 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp2103412ily; Thu, 15 Aug 2019 05:11:56 -0700 (PDT) X-Google-Smtp-Source: APXvYqx0RYOVaYid54iSAekEHppuQnJd0liHZzkPjXctmVWJ5mSFBFoeJCgaYGv2YDM6w1I/KmRK X-Received: by 2002:aa7:9903:: with SMTP id z3mr4903424pff.200.1565871116104; Thu, 15 Aug 2019 05:11:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1565871116; cv=none; d=google.com; s=arc-20160816; b=L6nMpEbJG52Zpat43hkx5EAkW/fheYpnOU7+Q/74Rt6ZP99wWQduXTKE5E/LxOhKH8 04xvMzl6pq43o4pkWgM59f3BM56nNHfL36+qrwgLLrO90Rs6m17yUFMPtZh6B6WCFgUt Smk3bPn5+4UvpiZskw3Mx6+kabqPnImqH0bCf0FK4pJO17Egg+noH07iwV0YGtaNsaL1 fXFB2mke8llzb6rbkxh//YKsAW4bza5dOsgTT1hPyxEIux6fGQMSqMhbNIp3sGQTULmt TG0NZPcbmDqlZJGKVXbFC8DpX6eHZEIFqadkKAr/QBtvARQXalxUURGxSp6Tzwi2mNHD KFHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=y1TLLNS0QZOer/hWVZKuRzBigK1roVklGOm1nyXB1L4=; b=vgrKwslOM5hcapWWlWMhSdIkdnm+oVNHnY5VSMqYaurJ6IFRnsjbpy3LBJq5eqFB1d zr9BhqJCDibXamuEsa43KcV59VaKZRKNxdog2zgA7H6gIkz/ShCFy/FYJ/OMOj7BkKQX FZbywIDG3KNAmf28egj0dMrBqZHC+pZ8exnT69rs7g6hhW1UVbrdBCf4NAqRg7MtyLUz 96nhKsf/A1IIXb97SJKJOMaZ/7TkGy2lVfooMuygL0oA1Yc98+biRgDdAJVm3EuB4hLw zBoP14LObUUCe82FyXrsUg2u9IRBck/iLklzQ6eSem1kyKoC+lUaYUz05L5ctCIvpVHw 7haw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 131si1737811pgf.597.2019.08.15.05.11.55; Thu, 15 Aug 2019 05:11:56 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731534AbfHOMLz (ORCPT + 28 others); Thu, 15 Aug 2019 08:11:55 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:4714 "EHLO huawei.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729900AbfHOMLx (ORCPT ); Thu, 15 Aug 2019 08:11:53 -0400 Received: from DGGEMS404-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 55BB1E228C6DE30AA7D5; Thu, 15 Aug 2019 20:11:45 +0800 (CST) Received: from HGHY4L002753561.china.huawei.com (10.133.215.186) by DGGEMS404-HUB.china.huawei.com (10.3.19.204) with Microsoft SMTP Server id 14.3.439.0; Thu, 15 Aug 2019 20:11:34 +0800 From: Zhen Lei To: Jean-Philippe Brucker , "Jean-Philippe Brucker" , John Garry , "Robin Murphy" , Will Deacon , Joerg Roedel , iommu , Omer Peleg , Adam Morrison , Shaohua Li , Ben Serebrin , David Woodhouse , linux-arm-kernel , linux-kernel CC: Zhen Lei Subject: [PATCH v2 2/2] iommu/iova: enhance the rcache optimization Date: Thu, 15 Aug 2019 20:11:04 +0800 Message-ID: <20190815121104.29140-3-thunder.leizhen@huawei.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20190815121104.29140-1-thunder.leizhen@huawei.com> References: <20190815121104.29140-1-thunder.leizhen@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.133.215.186] X-CFilter-Loop: Reflected Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The rcache method caches the freed IOVAs, to improve the performance of IOVAs allocation and release. This is usually okay, but it maybe declined in some special scenarios. For example, currently the IOVA_RANGE_CACHE_MAX_SIZE is 6, and for ecch size, contains: MAX_GLOBAL_MAGS=32 shareable depot magazines, each vcpu has two magazines(cpu_rcaches->loaded and cpu_rcaches->prev). In an extreme case, it can max cache ((num_possible_cpus() * 2 + 32) * 128 * 6) IOVAs, it's very large. The worst case happens when the depot magazines of a certain size(usually 4K) is full, further free_iova_fast() invoking will cause iova_magazine_free_pfns() to be called. As the above saied, too many IOVAs buffered, so that the RB tree is very large, the iova_magazine_free_pfns()-->private_find_iova(), and the missed iova allocation: alloc_iova()-->__alloc_and_insert_iova_range() will spend too much time. And that the current rcache method have no cleanup operation, the buffered IOVAs will only increase but not decrease. For my FIO stress test scenario, the performance drop about 35%, and can not recover even if re-execute the test cases. Jobs: 21 (f=21): [2.3% done] [8887M/0K /s] [2170K/0 iops] Jobs: 21 (f=21): [2.3% done] [8902M/0K /s] [2173K/0 iops] Jobs: 21 (f=21): [2.3% done] [6010M/0K /s] [1467K/0 iops] Jobs: 21 (f=21): [2.3% done] [5397M/0K /s] [1318K/0 iops] So that, I add the statistic about the rcache, when the above case happened, release the IOVAs which are not hit. Jobs: 21 (f=21): [100.0% done] [10324M/0K /s] [2520K/0 iops] Jobs: 21 (f=21): [100.0% done] [10290M/0K /s] [2512K/0 iops] Jobs: 21 (f=21): [100.0% done] [10035M/0K /s] [2450K/0 iops] Jobs: 21 (f=21): [100.0% done] [10214M/0K /s] [2494K/0 iops] Signed-off-by: Zhen Lei --- drivers/iommu/iova.c | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++- include/linux/iova.h | 1 + 2 files changed, 83 insertions(+), 1 deletion(-) -- 1.8.3 diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c index 4b7a9efa0ef40af..f3828f4add25375 100644 --- a/drivers/iommu/iova.c +++ b/drivers/iommu/iova.c @@ -23,6 +23,8 @@ static unsigned long iova_rcache_get(struct iova_domain *iovad, unsigned long limit_pfn); static void init_iova_rcaches(struct iova_domain *iovad); static void free_iova_rcaches(struct iova_domain *iovad); +static void iova_compact_rcache(struct iova_domain *iovad, + struct iova_rcache *curr_rcache); static void fq_destroy_all_entries(struct iova_domain *iovad); static void fq_flush_timeout(struct timer_list *t); @@ -781,6 +783,8 @@ struct iova_magazine { struct iova_cpu_rcache { spinlock_t lock; + bool prev_mag_hit; + unsigned long nr_hit; struct iova_magazine *loaded; struct iova_magazine *prev; }; @@ -934,6 +938,7 @@ static bool __iova_rcache_insert(struct iova_domain *iovad, if (mag_to_free) { iova_magazine_free_pfns(mag_to_free, iovad); iova_magazine_free(mag_to_free); + iova_compact_rcache(iovad, rcache); } return can_insert; @@ -971,18 +976,22 @@ static unsigned long __iova_rcache_get(struct iova_rcache *rcache, } else if (!iova_magazine_empty(cpu_rcache->prev)) { swap(cpu_rcache->prev, cpu_rcache->loaded); has_pfn = true; + cpu_rcache->prev_mag_hit = true; } else { spin_lock(&rcache->lock); if (rcache->depot_size > 0) { iova_magazine_free(cpu_rcache->loaded); cpu_rcache->loaded = rcache->depot[--rcache->depot_size]; has_pfn = true; + rcache->depot_mags_hit = true; } spin_unlock(&rcache->lock); } - if (has_pfn) + if (has_pfn) { + cpu_rcache->nr_hit++; iova_pfn = iova_magazine_pop(cpu_rcache->loaded, limit_pfn); + } spin_unlock_irqrestore(&cpu_rcache->lock, flags); @@ -1049,5 +1058,77 @@ void free_cpu_cached_iovas(unsigned int cpu, struct iova_domain *iovad) } } +static void iova_compact_percpu_mags(struct iova_domain *iovad, + struct iova_rcache *rcache) +{ + unsigned int cpu; + + for_each_possible_cpu(cpu) { + unsigned long flags; + struct iova_cpu_rcache *cpu_rcache; + + cpu_rcache = per_cpu_ptr(rcache->cpu_rcaches, cpu); + + spin_lock_irqsave(&cpu_rcache->lock, flags); + if (!cpu_rcache->prev_mag_hit) + iova_magazine_free_pfns(cpu_rcache->prev, iovad); + + if (cpu_rcache->nr_hit < IOVA_MAG_SIZE) + iova_magazine_compact_pfns(cpu_rcache->loaded, + iovad, + cpu_rcache->nr_hit); + + cpu_rcache->nr_hit = 0; + cpu_rcache->prev_mag_hit = false; + spin_unlock_irqrestore(&cpu_rcache->lock, flags); + } +} + +static void iova_compact_depot_mags(struct iova_domain *iovad, + struct iova_rcache *rcache) +{ + int i; + unsigned long depot_size; + struct iova_magazine *depot[MAX_GLOBAL_MAGS]; + + spin_lock(&rcache->lock); + if (!rcache->depot_size || rcache->depot_mags_hit) { + spin_unlock(&rcache->lock); + return; + } + + depot_size = rcache->depot_size; + for (i = 0; i < depot_size; i++) + depot[i] = rcache->depot[i]; + rcache->depot_size = 0; + rcache->depot_mags_hit = false; + spin_unlock(&rcache->lock); + + for (i = 0; i < depot_size; i++) { + iova_magazine_free_pfns(depot[i], iovad); + iova_magazine_free(depot[i]); + } +} + +static void iova_compact_rcache(struct iova_domain *iovad, + struct iova_rcache *curr_rcache) +{ + int i; + struct iova_rcache *rcache; + + for (i = 0; i < IOVA_RANGE_CACHE_MAX_SIZE; i++) { + rcache = &iovad->rcaches[i]; + + /* + * Don's compact current rcache, that maybe reused immediately + */ + if (rcache == curr_rcache) + continue; + + iova_compact_percpu_mags(iovad, rcache); + iova_compact_depot_mags(iovad, rcache); + } +} + MODULE_AUTHOR("Anil S Keshavamurthy "); MODULE_LICENSE("GPL"); diff --git a/include/linux/iova.h b/include/linux/iova.h index a0637abffee88b0..44f35b2641b736c 100644 --- a/include/linux/iova.h +++ b/include/linux/iova.h @@ -30,6 +30,7 @@ struct iova { struct iova_rcache { spinlock_t lock; + bool depot_mags_hit; unsigned long depot_size; struct iova_magazine *depot[MAX_GLOBAL_MAGS]; struct iova_cpu_rcache __percpu *cpu_rcaches;