From patchwork Fri Sep 14 14:30:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 146703 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp780164ljw; Fri, 14 Sep 2018 07:30:37 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZulDxpYWyUqfdraRQxIZt+QUu7CdI1C7ezd2MadQI/mkuydkjgvTeh467VT0h0fajnCSlJ X-Received: by 2002:a17:902:3a2:: with SMTP id d31-v6mr12513234pld.287.1536935437765; Fri, 14 Sep 2018 07:30:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536935437; cv=none; d=google.com; s=arc-20160816; b=Vj8yBld+RmnqXthnSoZwLdvBEmC+dRveCxRjSMfPwxzB/PYBOYz2uM4zzLEbPlBuTa BMnpmNXFtnoAjPNVMs/gxYmItygL4fUW7PzRhWymQW2j95Gjk84xxL+aEcmSqhBQ7JVq 0NKJAV4mHUeBAxBd2AVuR+UbLmVGzPO4/Dni+RPNDnFV+R0Y0KzxR9UxN62W1HmWhKqh mCFad3VufZXzdR6w2Cq1SUKxkI9Yag5UONIncw9kcClyo+5E1waCak7m0VO4Vs53MlMT V9TUGoel7iLZFxlKID7ksiC71hTtdXrTmCC7txVzz96QZMPCnGjT+macoguRNfteS6v2 fuBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=TxGcoem7bN74tOOynqUeMCYbvFifD3l7WEsIfvxmpXs=; b=RphL7GZIR7HU9QK77at1r8HAP3//rSWfmLsSPKQyf3XLAFM5gv5CQZOYMLZyVwJN0U DwB66Iu8Na86hmj3aKSGKqyBRiwQrlmtrpQfT+Z05fR2wOUMuOp+NBLue3OJCNoPstXc G9TS8JOUjHPA5dZDmaVZYamAePikLAiCUxQBbren1UlUyhZaGdqLq2139cw+tK+Gyhiy PRY8aSQ5TUI//yNC0geNkI9Yc7MIJ25OOYb+qVX3efDZW1MKEGgRJ/vPay8nSrVNW8OA Y7gzeqmLTp4dDziyqCSIXF92OCMzbLAeBu32scoGSXEB8wtrkIWDXvGUtV2UwCxDFUu2 f0IQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m2-v6si7448863pfi.351.2018.09.14.07.30.37; Fri, 14 Sep 2018 07:30:37 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728284AbeINTpV (ORCPT + 32 others); Fri, 14 Sep 2018 15:45:21 -0400 Received: from foss.arm.com ([217.140.101.70]:34202 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726902AbeINTpV (ORCPT ); Fri, 14 Sep 2018 15:45:21 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 03FF5ED1; Fri, 14 Sep 2018 07:30:35 -0700 (PDT) Received: from e110467-lin.cambridge.arm.com (e110467-lin.emea.arm.com [10.4.12.131]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 39A5F3F575; Fri, 14 Sep 2018 07:30:33 -0700 (PDT) From: Robin Murphy To: joro@8bytes.org, will.deacon@arm.com, thunder.leizhen@huawei.com, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: linuxarm@huawei.com, guohanjun@huawei.com, huawei.libin@huawei.com, john.garry@huawei.com Subject: [PATCH v7 1/6] iommu/arm-smmu-v3: Implement flush_iotlb_all hook Date: Fri, 14 Sep 2018 15:30:19 +0100 Message-Id: X-Mailer: git-send-email 2.19.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Zhen Lei .flush_iotlb_all is currently stubbed to arm_smmu_iotlb_sync() since the only time it would ever need to actually do anything is for callers doing their own explicit batching, e.g.: iommu_unmap_fast(domain, ...); iommu_unmap_fast(domain, ...); iommu_iotlb_flush_all(domain, ...); where since io-pgtable still issues the TLBI commands implicitly in the unmap instead of implementing .iotlb_range_add, the "flush" only needs to ensure completion of those already-in-flight invalidations. However, we're about to start using it in anger with flush queues, so let's get a proper implementation wired up. Signed-off-by: Zhen Lei Reviewed-by: Robin Murphy [rm: expand commit message] Signed-off-by: Robin Murphy --- drivers/iommu/arm-smmu-v3.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) -- 2.19.0.dirty diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index e395f1ff3f81..f10c852479fc 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -1781,6 +1781,14 @@ arm_smmu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size) return ops->unmap(ops, iova, size); } +static void arm_smmu_flush_iotlb_all(struct iommu_domain *domain) +{ + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + + if (smmu_domain->smmu) + arm_smmu_tlb_inv_context(smmu_domain); +} + static void arm_smmu_iotlb_sync(struct iommu_domain *domain) { struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu; @@ -2008,7 +2016,7 @@ static struct iommu_ops arm_smmu_ops = { .attach_dev = arm_smmu_attach_dev, .map = arm_smmu_map, .unmap = arm_smmu_unmap, - .flush_iotlb_all = arm_smmu_iotlb_sync, + .flush_iotlb_all = arm_smmu_flush_iotlb_all, .iotlb_sync = arm_smmu_iotlb_sync, .iova_to_phys = arm_smmu_iova_to_phys, .add_device = arm_smmu_add_device, From patchwork Fri Sep 14 14:30:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 146704 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp780208ljw; Fri, 14 Sep 2018 07:30:40 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdax3S8MU0EC+0VHaWpeSGTeg15u1ZehxB85EkXcTwf9+y2c9341m3HAFLWXiDjyHbfgO2ZQ X-Received: by 2002:a17:902:b595:: with SMTP id a21-v6mr12616032pls.23.1536935440340; Fri, 14 Sep 2018 07:30:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536935440; cv=none; d=google.com; s=arc-20160816; b=EV+08igLAvEdF23EFIrlaLC3ReRQHpNJo7b8rpJJyPuAKcSLEbdjkuEj0nq73byWEq y7RjriwC1Ib+FfGy6qtGrGWw22YWUpj5xEzLHvxZ6R05gCiFBNPzsNZDhyz5icBXdcWf Wi7al/gH4dlyWoNww4CamkCOU708jihm4rKmnliRI3hosB3w6jAaxMvjAqnwhm1yP2ii e65ICcs5BPgYp4jK7gAL0iXmP2lPIbVaTjymbPD0PbHjadIesouunpIRO8/yYQ1+J34X g1KHFdSIKTdC/XTG1bLI9f3I/C6T3qqnS2Dn9WdJcHmLSHHO/ImB0vvVwsN77BBBM/JZ 5EmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=vjnifCg7P6wq+QD1LIaWbjWBjY+9QPgVvA2efY1JjBc=; b=JClldUV4V/9KtI8DKYdiBaeknZ8hlwDDD7KMy7SqG8p+pjebiekgKkWYssqHr/nyNO JNF9m6rQWJgAsOwC2KUCoF3RGmSZXILnESpZtVwOI+N49tWM4JzK/dnYdxAHgpNi3emr 7EqE1f56bU6i/wfFX1udFfEXtwD72aT5Drjda3ucQ6huIXluUoN2sYdYA1KjpOrTWw3R cVeharN4XzksDFpodnL3J5WtO4wpWbwqjKyhOBQen/7iLlg3YV1f0xvAxGrbYSHPzbWA z2LRaThXpqqN8mp9V7wOWEtoApQSOwYYxvAYrIJGldYOW08OWqTwfwhTLlEwPrPCJmaQ 35eg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m2-v6si7448863pfi.351.2018.09.14.07.30.39; Fri, 14 Sep 2018 07:30:40 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728296AbeINTpX (ORCPT + 32 others); Fri, 14 Sep 2018 15:45:23 -0400 Received: from foss.arm.com ([217.140.101.70]:34218 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726902AbeINTpW (ORCPT ); Fri, 14 Sep 2018 15:45:22 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0CA8315B2; Fri, 14 Sep 2018 07:30:37 -0700 (PDT) Received: from e110467-lin.cambridge.arm.com (e110467-lin.emea.arm.com [10.4.12.131]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 42C8B3F575; Fri, 14 Sep 2018 07:30:35 -0700 (PDT) From: Robin Murphy To: joro@8bytes.org, will.deacon@arm.com, thunder.leizhen@huawei.com, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: linuxarm@huawei.com, guohanjun@huawei.com, huawei.libin@huawei.com, john.garry@huawei.com Subject: [PATCH v7 2/6] iommu/dma: Add support for non-strict mode Date: Fri, 14 Sep 2018 15:30:20 +0100 Message-Id: <0a891cec4bb164f8cf2f57c753791a7d1f5f1d81.1536935328.git.robin.murphy@arm.com> X-Mailer: git-send-email 2.19.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Zhen Lei 1. Save the related domain pointer in struct iommu_dma_cookie, make iovad capable call domain->ops->flush_iotlb_all to flush TLB. 2. During the iommu domain initialization phase, base on domain->non_strict field to check whether non-strict mode is supported or not. If so, call init_iova_flush_queue to register iovad->flush_cb callback. 3. All unmap(contains iova-free) APIs will finally invoke __iommu_dma_unmap -->iommu_dma_free_iova. If the domain is non-strict, call queue_iova to put off iova freeing, and omit iommu_tlb_sync operation. Signed-off-by: Zhen Lei [rm: convert raw boolean to domain attribute] Signed-off-by: Robin Murphy --- drivers/iommu/dma-iommu.c | 29 ++++++++++++++++++++++++++++- include/linux/iommu.h | 1 + 2 files changed, 29 insertions(+), 1 deletion(-) -- 2.19.0.dirty diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 511ff9a1d6d9..092e6926dc3c 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -55,6 +55,9 @@ struct iommu_dma_cookie { }; struct list_head msi_page_list; spinlock_t msi_lock; + + /* Only be assigned in non-strict mode, otherwise it's NULL */ + struct iommu_domain *domain; }; static inline size_t cookie_msi_granule(struct iommu_dma_cookie *cookie) @@ -257,6 +260,17 @@ static int iova_reserve_iommu_regions(struct device *dev, return ret; } +static void iommu_dma_flush_iotlb_all(struct iova_domain *iovad) +{ + struct iommu_dma_cookie *cookie; + struct iommu_domain *domain; + + cookie = container_of(iovad, struct iommu_dma_cookie, iovad); + domain = cookie->domain; + + domain->ops->flush_iotlb_all(domain); +} + /** * iommu_dma_init_domain - Initialise a DMA mapping domain * @domain: IOMMU domain previously prepared by iommu_get_dma_cookie() @@ -275,6 +289,7 @@ int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, struct iommu_dma_cookie *cookie = domain->iova_cookie; struct iova_domain *iovad = &cookie->iovad; unsigned long order, base_pfn, end_pfn; + int attr = 1; if (!cookie || cookie->type != IOMMU_DMA_IOVA_COOKIE) return -EINVAL; @@ -308,6 +323,13 @@ int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, } init_iova_domain(iovad, 1UL << order, base_pfn); + + if (!iommu_domain_get_attr(domain, DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE, + &attr) && attr) { + cookie->domain = domain; + init_iova_flush_queue(iovad, iommu_dma_flush_iotlb_all, NULL); + } + if (!dev) return 0; @@ -393,6 +415,9 @@ static void iommu_dma_free_iova(struct iommu_dma_cookie *cookie, /* The MSI case is only ever cleaning up its most recent allocation */ if (cookie->type == IOMMU_DMA_MSI_COOKIE) cookie->msi_iova -= size; + else if (cookie->domain) /* non-strict mode */ + queue_iova(iovad, iova_pfn(iovad, iova), + size >> iova_shift(iovad), 0); else free_iova_fast(iovad, iova_pfn(iovad, iova), size >> iova_shift(iovad)); @@ -408,7 +433,9 @@ static void __iommu_dma_unmap(struct iommu_domain *domain, dma_addr_t dma_addr, dma_addr -= iova_off; size = iova_align(iovad, size + iova_off); - WARN_ON(iommu_unmap(domain, dma_addr, size) != size); + WARN_ON(iommu_unmap_fast(domain, dma_addr, size) != size); + if (!cookie->domain) + iommu_tlb_sync(domain); iommu_dma_free_iova(cookie, dma_addr, size); } diff --git a/include/linux/iommu.h b/include/linux/iommu.h index 87994c265bf5..decabe8e8dbe 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -124,6 +124,7 @@ enum iommu_attr { DOMAIN_ATTR_FSL_PAMU_ENABLE, DOMAIN_ATTR_FSL_PAMUV1, DOMAIN_ATTR_NESTING, /* two stages of translation */ + DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE, DOMAIN_ATTR_MAX, }; From patchwork Fri Sep 14 14:30:21 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 146705 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp780244ljw; Fri, 14 Sep 2018 07:30:42 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdbp463+DcK/fCx3gy7Ofh9fn/rc0JTflFKtHR+d0EFLFl1u/f5+RFKOw4B2bcNLOIC2PRNR X-Received: by 2002:a62:5ec3:: with SMTP id s186-v6mr13014758pfb.146.1536935442333; Fri, 14 Sep 2018 07:30:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536935442; cv=none; d=google.com; s=arc-20160816; b=Uh+r18hUjt1Tbbr6SeJU48JfBsFMjUhnzjOtlTEF8JVR2ZiIbWUQCXUmVVjmiF7KrN QHnjQLUHXzjTbCorpRU6/BLWfQsFDzxT3icOLRwFQ4tSF8ikarV6oisfm/htYLsYGnMN i3sJDzndx/gr7q8+e5PYiv1VGSWsCdyLHVoD5NCMY6eXwvswNAdwKJq+OxssmtZsPttu NLfiq+VeZoCq3Q0ok8lTKnvd8WSrugQMA4gwewaHvTYttWtTEsNgRNg4FZo7O3yQHGl1 6eCs83yEX75vz9fPA4ByNZdqngz5lUcvykTT+fnuc+HRbNufuSTg+vKm5090Am2Oi2B9 dDDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=dLiTFHPE9aU4Hn6YgQH8nqBIpwAxz2ElmjiuA4HNLw4=; b=WFEP70GZbEEHtOzb0Y2bP6YvhcYJqU09L/SMoR8xdX05R7/etJZ4UtSXNWeOB6L/tA mI/wms/Qk3iOtozTMSKaZ2sFl6FFp9uFd6uKPV547NSbxkmZqZiw30BSurYPJss8kKth GJ4H/1+tG35/5sH+7urzI3XbbBAZc2TVpUV9B3wL4EALYG8TWWs3XAxRr0QMD5VDPjgU K2R/+yQrHGZRuwS9DHK5sky125di7sKwd7RUavzv5L5NVjR8hjGMkVzkfoczVlvWFTm4 4NSksFJfvRU0YOEDUnrHo/FQ2JtccBZK0jVIA4B+2zV332d1z+Kbsx5kyLW/NOkajW8h oFng== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q66-v6si7342064pfk.268.2018.09.14.07.30.42; Fri, 14 Sep 2018 07:30:42 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728318AbeINTp0 (ORCPT + 32 others); Fri, 14 Sep 2018 15:45:26 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:34224 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726902AbeINTpZ (ORCPT ); Fri, 14 Sep 2018 15:45:25 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 159B915BE; Fri, 14 Sep 2018 07:30:39 -0700 (PDT) Received: from e110467-lin.cambridge.arm.com (e110467-lin.emea.arm.com [10.4.12.131]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4B94E3F575; Fri, 14 Sep 2018 07:30:37 -0700 (PDT) From: Robin Murphy To: joro@8bytes.org, will.deacon@arm.com, thunder.leizhen@huawei.com, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: linuxarm@huawei.com, guohanjun@huawei.com, huawei.libin@huawei.com, john.garry@huawei.com Subject: [PATCH v7 3/6] iommu/io-pgtable-arm: Add support for non-strict mode Date: Fri, 14 Sep 2018 15:30:21 +0100 Message-Id: <75c1c1f9e8bc2fb9f199be5d3aef041a92d30160.1536935328.git.robin.murphy@arm.com> X-Mailer: git-send-email 2.19.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Zhen Lei To support non-strict mode, now we only TLBI and sync for strict mode, except for non-leaf invalidations since page table updates themselves must always be synchronous. To save having to reason about it too much, make sure the invalidation in arm_lpae_split_blk_unmap() just performs its own unconditional sync to minimise the window in which we're technically violating the break- before-make requirement on a live mapping. This might work out redundant with an outer-level sync for strict unmaps, but we'll never be splitting blocks on a DMA fastpath anyway. Signed-off-by: Zhen Lei [rm: tweak comment, commit message, and split_blk_unmap logic] Signed-off-by: Robin Murphy --- drivers/iommu/io-pgtable-arm.c | 9 ++++++--- drivers/iommu/io-pgtable.h | 5 +++++ 2 files changed, 11 insertions(+), 3 deletions(-) -- 2.19.0.dirty diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c index 2f79efd16a05..5b915aab7fd3 100644 --- a/drivers/iommu/io-pgtable-arm.c +++ b/drivers/iommu/io-pgtable-arm.c @@ -576,6 +576,7 @@ static size_t arm_lpae_split_blk_unmap(struct arm_lpae_io_pgtable *data, tablep = iopte_deref(pte, data); } else if (unmap_idx >= 0) { io_pgtable_tlb_add_flush(&data->iop, iova, size, size, true); + io_pgtable_tlb_sync(&data->iop); return size; } @@ -609,7 +610,7 @@ static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data, io_pgtable_tlb_sync(iop); ptep = iopte_deref(pte, data); __arm_lpae_free_pgtable(data, lvl + 1, ptep); - } else { + } else if (!(iop->cfg.quirks & IO_PGTABLE_QUIRK_NON_STRICT)) { io_pgtable_tlb_add_flush(iop, iova, size, size, true); } @@ -771,7 +772,8 @@ arm_64_lpae_alloc_pgtable_s1(struct io_pgtable_cfg *cfg, void *cookie) u64 reg; struct arm_lpae_io_pgtable *data; - if (cfg->quirks & ~(IO_PGTABLE_QUIRK_ARM_NS | IO_PGTABLE_QUIRK_NO_DMA)) + if (cfg->quirks & ~(IO_PGTABLE_QUIRK_ARM_NS | IO_PGTABLE_QUIRK_NO_DMA | + IO_PGTABLE_QUIRK_NON_STRICT)) return NULL; data = arm_lpae_alloc_pgtable(cfg); @@ -863,7 +865,8 @@ arm_64_lpae_alloc_pgtable_s2(struct io_pgtable_cfg *cfg, void *cookie) struct arm_lpae_io_pgtable *data; /* The NS quirk doesn't apply at stage 2 */ - if (cfg->quirks & ~IO_PGTABLE_QUIRK_NO_DMA) + if (cfg->quirks & ~(IO_PGTABLE_QUIRK_NO_DMA | + IO_PGTABLE_QUIRK_NON_STRICT)) return NULL; data = arm_lpae_alloc_pgtable(cfg); diff --git a/drivers/iommu/io-pgtable.h b/drivers/iommu/io-pgtable.h index 2df79093cad9..47d5ae559329 100644 --- a/drivers/iommu/io-pgtable.h +++ b/drivers/iommu/io-pgtable.h @@ -71,12 +71,17 @@ struct io_pgtable_cfg { * be accessed by a fully cache-coherent IOMMU or CPU (e.g. for a * software-emulated IOMMU), such that pagetable updates need not * be treated as explicit DMA data. + * + * IO_PGTABLE_QUIRK_NON_STRICT: Skip issuing synchronous leaf TLBIs + * on unmap, for DMA domains using the flush queue mechanism for + * delayed invalidation. */ #define IO_PGTABLE_QUIRK_ARM_NS BIT(0) #define IO_PGTABLE_QUIRK_NO_PERMS BIT(1) #define IO_PGTABLE_QUIRK_TLBI_ON_MAP BIT(2) #define IO_PGTABLE_QUIRK_ARM_MTK_4GB BIT(3) #define IO_PGTABLE_QUIRK_NO_DMA BIT(4) + #define IO_PGTABLE_QUIRK_NON_STRICT BIT(5) unsigned long quirks; unsigned long pgsize_bitmap; unsigned int ias; From patchwork Fri Sep 14 14:30:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 146706 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp780278ljw; Fri, 14 Sep 2018 07:30:44 -0700 (PDT) X-Google-Smtp-Source: ANB0VdY5POI1NSNV8z9ShBVsQ0G9wJ0I69ohlRbtQaMo0N6m5RmLg6toekl7V6A/GmyBNCcK6JRS X-Received: by 2002:a62:4bc6:: with SMTP id d67-v6mr12908383pfj.175.1536935444678; Fri, 14 Sep 2018 07:30:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536935444; cv=none; d=google.com; s=arc-20160816; b=AfseUb+SVmTxsZmf6H2GbRl101Si9d92GGD31gChOH0K88Ew1vP/nwhBN/I84qnRa8 suZjLZPVogob25xnMq0mUvvnpz34t5hACNUg4+E0RaGZqmmh7DJF+bY0lweKqLJzCGsC xDfe4ULr9dnhn7kuH+GINQPS5KcrPJA1ez6OeEYaiLeOng8ca60ywtasM9htNTrV2OzN 2+iTfnEkuOAFxP3e1zXogjvd5XYibZ/zwH9Uuz4EL8BDuluvE0n+3c/nYlgNA7kYKhk3 lqHP7ItlB47fMhZWfCbgg7AlK6LkGubRDA6XUPowEvHuVu+uAPmB/Ho7sHcDP3jexgSw SBkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=CxhohGZZRSZLez5qmBG5B0gm6SqMT8QtZzQHSdPgqN8=; b=AjgWJ68mCslTnu843/sL8YTY2uZyTECV3gBm0jmkcgrIKyXdai47wwIbucSYiFmD/K Neisz20FJG6PVon63CMcVfdpDRCXDsAm9jvdonwuVg1yMNF1iy6DI3H+ZV+lpqOr1gs9 zl0vTwK64UPTZikT0jUnVpaTdDllcAN729hqOCv3TXzXItICNhsXcbmwyl/Sa9doNGNp cdjaDWpsNH8FOfVyp4TB6eYNhfoJWacumxa/3sxaPJX8viP5BE+nvsVVA/94OpxxAiPx TjS3TUEIf8seBQsdYgVtb7dab3z2GHohzP+1yFBfQ+GB8c1gdWZIeZaBon9ixhJWAH5E 4+jw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id y13-v6si7532929pgl.555.2018.09.14.07.30.44; Fri, 14 Sep 2018 07:30:44 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728332AbeINTp2 (ORCPT + 32 others); Fri, 14 Sep 2018 15:45:28 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:34240 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726902AbeINTp0 (ORCPT ); Fri, 14 Sep 2018 15:45:26 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1EA8E15BF; Fri, 14 Sep 2018 07:30:41 -0700 (PDT) Received: from e110467-lin.cambridge.arm.com (e110467-lin.emea.arm.com [10.4.12.131]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 549373F575; Fri, 14 Sep 2018 07:30:39 -0700 (PDT) From: Robin Murphy To: joro@8bytes.org, will.deacon@arm.com, thunder.leizhen@huawei.com, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: linuxarm@huawei.com, guohanjun@huawei.com, huawei.libin@huawei.com, john.garry@huawei.com Subject: [PATCH v7 4/6] iommu: Add bootup option "iommu.non_strict" Date: Fri, 14 Sep 2018 15:30:22 +0100 Message-Id: <41f81c46348db4acd9a542184f10e7ca24f6c219.1536935328.git.robin.murphy@arm.com> X-Mailer: git-send-email 2.19.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Zhen Lei Add a bootup option to make the system manager can choose which mode to be used. The default mode is strict. Signed-off-by: Zhen Lei [rm: move handling out of SMMUv3 driver] Signed-off-by: Robin Murphy --- .../admin-guide/kernel-parameters.txt | 13 ++++++++++ drivers/iommu/iommu.c | 26 +++++++++++++++++++ 2 files changed, 39 insertions(+) -- 2.19.0.dirty diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 9871e649ffef..406b91759b62 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1749,6 +1749,19 @@ nobypass [PPC/POWERNV] Disable IOMMU bypass, using IOMMU for PCI devices. + iommu.non_strict= [ARM64] + Format: { "0" | "1" } + 0 - strict mode, default. + Release IOVAs after the related TLBs are invalid + completely. + 1 - non-strict mode. + Put off TLBs invalidation and release memory first. + It's good for scatter-gather performance but lacks + full isolation, an untrusted device can access the + reused memory because the TLBs may still valid. + Please take full consideration before choosing this + mode. Note that, VFIO will always use strict mode. + iommu.passthrough= [ARM64] Configure DMA to bypass the IOMMU by default. Format: { "0" | "1" } diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 8c15c5980299..2cabd0c0a4f3 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -41,6 +41,7 @@ static unsigned int iommu_def_domain_type = IOMMU_DOMAIN_IDENTITY; #else static unsigned int iommu_def_domain_type = IOMMU_DOMAIN_DMA; #endif +static bool iommu_dma_non_strict __read_mostly; struct iommu_callback_data { const struct iommu_ops *ops; @@ -131,6 +132,24 @@ static int __init iommu_set_def_domain_type(char *str) } early_param("iommu.passthrough", iommu_set_def_domain_type); +static int __init iommu_dma_setup(char *str) +{ + int ret; + + ret = kstrtobool(str, &iommu_dma_non_strict); + if (ret) + return ret; + + if (iommu_dma_non_strict) { + pr_warn("WARNING: iommu non-strict mode is chosen.\n" + "It's good for scatter-gather performance but lacks full isolation\n"); + add_taint(TAINT_WARN, LOCKDEP_STILL_OK); + } + + return 0; +} +early_param("iommu.non_strict", iommu_dma_setup); + static ssize_t iommu_group_attr_show(struct kobject *kobj, struct attribute *__attr, char *buf) { @@ -1072,6 +1091,13 @@ struct iommu_group *iommu_group_get_for_dev(struct device *dev) group->default_domain = dom; if (!group->domain) group->domain = dom; + + if (dom && iommu_dma_non_strict) { + int attr = 1; + iommu_domain_set_attr(dom, + DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE, + &attr); + } } ret = iommu_group_add_device(group, dev); From patchwork Fri Sep 14 14:30:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 146707 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp780321ljw; Fri, 14 Sep 2018 07:30:46 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZNxhlnXq3Od0m4Ul+U6eoTyXuOxmYa/fsGiECFqncYVeJnDLhOPRalK1yS+RYassKA/WXA X-Received: by 2002:a17:902:4d45:: with SMTP id o5-v6mr12629273plh.78.1536935446613; Fri, 14 Sep 2018 07:30:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536935446; cv=none; d=google.com; s=arc-20160816; b=sMiKS6TCiy6LN26VjrxBJkrZCHJX43X4jfcOOeGIoGL9wX5iz7FyPATAE0vSPFFoUM 8YUh625BSpJrPP+2YCUf7RpsYoyajJF4GwRlRzQ46I0mRxVbNJs5oEtXo83X/je19Jnu eW1Oibt1rHikqpnZPfc3Mp5AO3Iih+bRsXlVAoRBsiDsCSmrJFdvVOPhz9F3Um5239L+ NkwH6Trd8MavBCot91h33MAG8ta/5fwaYmy+iNBcEwlGPgyF6QC8bYa6LmDYAJf5kDiG aHe9sVXxH/JU0Uu/JQeYVnDVnty66EakGrRkAjUafxmFZScD0+OKHPtwM1ZGgMDNCv9O +wZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=+4Q4y4nXIVpXOHQZ8EUOiPgBXMMYeLBvHy0quzyKhu0=; b=Hb5yNNyzm5rEQaY/RYwhv3RFUBhdfqoWn7GN4JKk2y8TA8nmytgRXsvu8nT1qxVXlA iPAruiMa0HFP7TKmbdrn0XDOALfBnsYI93h3Ws7ETq263XnV0XgmVGdR5omarNehR1jw 9cjv1r9B1RNG7USB+m0z/kCmnNwIqRWCQA3c0rELTxa+TdmIYJHJi0I+wdNyVKt98KaH FGtLP811N5C1bMjzzmGrf04UAee5Q27V2y3oYIy6hpyWtnrYuJVbYCszQY1UFnbJfhXD CkorEAF0vYuCign/QkuBRzj9EZDgA+k7tzF64h74kHlJdaIXi4/rorguBctU5Yt63yYu 4EZQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m10-v6si7711260pgc.105.2018.09.14.07.30.46; Fri, 14 Sep 2018 07:30:46 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728349AbeINTpa (ORCPT + 32 others); Fri, 14 Sep 2018 15:45:30 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:34248 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726902AbeINTp2 (ORCPT ); Fri, 14 Sep 2018 15:45:28 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 27DC01650; Fri, 14 Sep 2018 07:30:43 -0700 (PDT) Received: from e110467-lin.cambridge.arm.com (e110467-lin.emea.arm.com [10.4.12.131]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 5DE173F575; Fri, 14 Sep 2018 07:30:41 -0700 (PDT) From: Robin Murphy To: joro@8bytes.org, will.deacon@arm.com, thunder.leizhen@huawei.com, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: linuxarm@huawei.com, guohanjun@huawei.com, huawei.libin@huawei.com, john.garry@huawei.com Subject: [PATCH v7 5/6] iommu/arm-smmu-v3: Add support for non-strict mode Date: Fri, 14 Sep 2018 15:30:23 +0100 Message-Id: <4f9444c7224ea0e6e4cef8ddbf77bb5292a383e3.1536935328.git.robin.murphy@arm.com> X-Mailer: git-send-email 2.19.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Zhen Lei Dynamically choose strict or non-strict mode for page table config based on the iommu domain type. Signed-off-by: Zhen Lei [rm: convert to domain attribute] Signed-off-by: Robin Murphy --- drivers/iommu/arm-smmu-v3.c | 30 ++++++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-) -- 2.19.0.dirty diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index f10c852479fc..7bbfa5f7ce8e 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -612,6 +612,7 @@ struct arm_smmu_domain { struct mutex init_mutex; /* Protects smmu pointer */ struct io_pgtable_ops *pgtbl_ops; + bool non_strict; enum arm_smmu_domain_stage stage; union { @@ -1633,6 +1634,9 @@ static int arm_smmu_domain_finalise(struct iommu_domain *domain) if (smmu->features & ARM_SMMU_FEAT_COHERENCY) pgtbl_cfg.quirks = IO_PGTABLE_QUIRK_NO_DMA; + if (smmu_domain->non_strict) + pgtbl_cfg.quirks |= IO_PGTABLE_QUIRK_NON_STRICT; + pgtbl_ops = alloc_io_pgtable_ops(fmt, &pgtbl_cfg, smmu_domain); if (!pgtbl_ops) return -ENOMEM; @@ -1934,13 +1938,17 @@ static int arm_smmu_domain_get_attr(struct iommu_domain *domain, { struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); - if (domain->type != IOMMU_DOMAIN_UNMANAGED) - return -EINVAL; - switch (attr) { case DOMAIN_ATTR_NESTING: + if (domain->type != IOMMU_DOMAIN_UNMANAGED) + return -EINVAL; *(int *)data = (smmu_domain->stage == ARM_SMMU_DOMAIN_NESTED); return 0; + case DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE: + if (domain->type != IOMMU_DOMAIN_DMA) + return -EINVAL; + *(int *)data = smmu_domain->non_strict; + return 0; default: return -ENODEV; } @@ -1952,13 +1960,15 @@ static int arm_smmu_domain_set_attr(struct iommu_domain *domain, int ret = 0; struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); - if (domain->type != IOMMU_DOMAIN_UNMANAGED) - return -EINVAL; - mutex_lock(&smmu_domain->init_mutex); switch (attr) { case DOMAIN_ATTR_NESTING: + if (domain->type != IOMMU_DOMAIN_UNMANAGED) { + ret = -EINVAL; + goto out_unlock; + } + if (smmu_domain->smmu) { ret = -EPERM; goto out_unlock; @@ -1969,6 +1979,14 @@ static int arm_smmu_domain_set_attr(struct iommu_domain *domain, else smmu_domain->stage = ARM_SMMU_DOMAIN_S1; + break; + case DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE: + if (domain->type != IOMMU_DOMAIN_DMA) { + ret = -EINVAL; + goto out_unlock; + } + + smmu_domain->non_strict = *(int *)data; break; default: ret = -ENODEV;