From patchwork Thu Aug 20 15:08:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 250591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D4C2C433E5 for ; Thu, 20 Aug 2020 15:12:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 42B2A208C7 for ; Thu, 20 Aug 2020 15:12:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728602AbgHTPMy (ORCPT ); Thu, 20 Aug 2020 11:12:54 -0400 Received: from foss.arm.com ([217.140.110.172]:40754 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727817AbgHTPJC (ORCPT ); Thu, 20 Aug 2020 11:09:02 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6BF761063; Thu, 20 Aug 2020 08:09:01 -0700 (PDT) Received: from e121345-lin.cambridge.arm.com (e121345-lin.cambridge.arm.com [10.1.196.37]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B842E3F6CF; Thu, 20 Aug 2020 08:08:57 -0700 (PDT) From: Robin Murphy To: hch@lst.de, joro@8bytes.org, linux@armlinux.org.uk Cc: will@kernel.org, inki.dae@samsung.com, sw0312.kim@samsung.com, kyungmin.park@samsung.com, m.szyprowski@samsung.com, agross@kernel.org, bjorn.andersson@linaro.org, thierry.reding@gmail.com, jonathanh@nvidia.com, vdumpa@nvidia.com, digetx@gmail.com, matthias.bgg@gmail.com, yong.wu@mediatek.com, geert+renesas@glider.be, magnus.damm@gmail.com, t-kristo@ti.com, s-anna@ti.com, laurent.pinchart@ideasonboard.com, linux-arm-kernel@lists.infradead.org, iommu@lists.linux-foundation.org, linux-samsung-soc@vger.kernel.org, linux-tegra@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-mediatek@lists.infradead.org, dri-devel@lists.freedesktop.org, linux-media@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 02/18] ARM/dma-mapping: Consolidate IOMMU ops callbacks Date: Thu, 20 Aug 2020 16:08:21 +0100 Message-Id: X-Mailer: git-send-email 2.28.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-arm-msm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Merge the coherent and non-coherent callbacks down to a single implementation each, relying on the generic dev->dma_coherent flag at the points where the difference matters. Signed-off-by: Robin Murphy --- arch/arm/Kconfig | 4 +- arch/arm/mm/dma-mapping.c | 281 +++++++++++--------------------------- 2 files changed, 79 insertions(+), 206 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index e00d94b16658..b91273f9fd43 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -19,8 +19,8 @@ config ARM select ARCH_HAS_SET_MEMORY select ARCH_HAS_STRICT_KERNEL_RWX if MMU && !XIP_KERNEL select ARCH_HAS_STRICT_MODULE_RWX if MMU - select ARCH_HAS_SYNC_DMA_FOR_DEVICE if SWIOTLB - select ARCH_HAS_SYNC_DMA_FOR_CPU if SWIOTLB + select ARCH_HAS_SYNC_DMA_FOR_DEVICE if SWIOTLB || ARM_DMA_USE_IOMMU + select ARCH_HAS_SYNC_DMA_FOR_CPU if SWIOTLB || ARM_DMA_USE_IOMMU select ARCH_HAS_TEARDOWN_DMA_OPS if MMU select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST select ARCH_HAVE_CUSTOM_GPIO_H diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index ffa387f343dc..1bb7e9608f75 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -1418,13 +1418,13 @@ static void __iommu_free_atomic(struct device *dev, void *cpu_addr, __free_from_pool(cpu_addr, size); } -static void *__arm_iommu_alloc_attrs(struct device *dev, size_t size, - dma_addr_t *handle, gfp_t gfp, unsigned long attrs, - int coherent_flag) +static void *arm_iommu_alloc_attrs(struct device *dev, size_t size, + dma_addr_t *handle, gfp_t gfp, unsigned long attrs) { pgprot_t prot = __get_dma_pgprot(attrs, PAGE_KERNEL); struct page **pages; void *addr = NULL; + int coherent_flag = dev->dma_coherent ? COHERENT : NORMAL; *handle = DMA_MAPPING_ERROR; size = PAGE_ALIGN(size); @@ -1467,19 +1467,7 @@ static void *__arm_iommu_alloc_attrs(struct device *dev, size_t size, return NULL; } -static void *arm_iommu_alloc_attrs(struct device *dev, size_t size, - dma_addr_t *handle, gfp_t gfp, unsigned long attrs) -{ - return __arm_iommu_alloc_attrs(dev, size, handle, gfp, attrs, NORMAL); -} - -static void *arm_coherent_iommu_alloc_attrs(struct device *dev, size_t size, - dma_addr_t *handle, gfp_t gfp, unsigned long attrs) -{ - return __arm_iommu_alloc_attrs(dev, size, handle, gfp, attrs, COHERENT); -} - -static int __arm_iommu_mmap_attrs(struct device *dev, struct vm_area_struct *vma, +static int arm_iommu_mmap_attrs(struct device *dev, struct vm_area_struct *vma, void *cpu_addr, dma_addr_t dma_addr, size_t size, unsigned long attrs) { @@ -1493,35 +1481,24 @@ static int __arm_iommu_mmap_attrs(struct device *dev, struct vm_area_struct *vma if (vma->vm_pgoff >= nr_pages) return -ENXIO; + if (!dev->dma_coherent) + vma->vm_page_prot = __get_dma_pgprot(attrs, vma->vm_page_prot); + err = vm_map_pages(vma, pages, nr_pages); if (err) pr_err("Remapping memory failed: %d\n", err); return err; } -static int arm_iommu_mmap_attrs(struct device *dev, - struct vm_area_struct *vma, void *cpu_addr, - dma_addr_t dma_addr, size_t size, unsigned long attrs) -{ - vma->vm_page_prot = __get_dma_pgprot(attrs, vma->vm_page_prot); - - return __arm_iommu_mmap_attrs(dev, vma, cpu_addr, dma_addr, size, attrs); -} - -static int arm_coherent_iommu_mmap_attrs(struct device *dev, - struct vm_area_struct *vma, void *cpu_addr, - dma_addr_t dma_addr, size_t size, unsigned long attrs) -{ - return __arm_iommu_mmap_attrs(dev, vma, cpu_addr, dma_addr, size, attrs); -} /* * free a page as defined by the above mapping. * Must not be called with IRQs disabled. */ -static void __arm_iommu_free_attrs(struct device *dev, size_t size, void *cpu_addr, - dma_addr_t handle, unsigned long attrs, int coherent_flag) +static void arm_iommu_free_attrs(struct device *dev, size_t size, void *cpu_addr, + dma_addr_t handle, unsigned long attrs) { + int coherent_flag = dev->dma_coherent ? COHERENT : NORMAL; struct page **pages; size = PAGE_ALIGN(size); @@ -1543,19 +1520,6 @@ static void __arm_iommu_free_attrs(struct device *dev, size_t size, void *cpu_ad __iommu_free_buffer(dev, pages, size, attrs); } -static void arm_iommu_free_attrs(struct device *dev, size_t size, - void *cpu_addr, dma_addr_t handle, - unsigned long attrs) -{ - __arm_iommu_free_attrs(dev, size, cpu_addr, handle, attrs, NORMAL); -} - -static void arm_coherent_iommu_free_attrs(struct device *dev, size_t size, - void *cpu_addr, dma_addr_t handle, unsigned long attrs) -{ - __arm_iommu_free_attrs(dev, size, cpu_addr, handle, attrs, COHERENT); -} - static int arm_iommu_get_sgtable(struct device *dev, struct sg_table *sgt, void *cpu_addr, dma_addr_t dma_addr, size_t size, unsigned long attrs) @@ -1575,8 +1539,7 @@ static int arm_iommu_get_sgtable(struct device *dev, struct sg_table *sgt, */ static int __map_sg_chunk(struct device *dev, struct scatterlist *sg, size_t size, dma_addr_t *handle, - enum dma_data_direction dir, unsigned long attrs, - bool is_coherent) + enum dma_data_direction dir, unsigned long attrs) { struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev); dma_addr_t iova, iova_base; @@ -1596,7 +1559,7 @@ static int __map_sg_chunk(struct device *dev, struct scatterlist *sg, phys_addr_t phys = page_to_phys(sg_page(s)); unsigned int len = PAGE_ALIGN(s->offset + s->length); - if (!is_coherent && (attrs & DMA_ATTR_SKIP_CPU_SYNC) == 0) + if (!dev->dma_coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) __dma_page_cpu_to_dev(sg_page(s), s->offset, s->length, dir); prot = __dma_info_to_prot(dir, attrs); @@ -1616,70 +1579,6 @@ static int __map_sg_chunk(struct device *dev, struct scatterlist *sg, return ret; } -static int __iommu_map_sg(struct device *dev, struct scatterlist *sg, int nents, - enum dma_data_direction dir, unsigned long attrs, - bool is_coherent) -{ - struct scatterlist *s = sg, *dma = sg, *start = sg; - int i, count = 0; - unsigned int offset = s->offset; - unsigned int size = s->offset + s->length; - unsigned int max = dma_get_max_seg_size(dev); - - for (i = 1; i < nents; i++) { - s = sg_next(s); - - s->dma_address = DMA_MAPPING_ERROR; - s->dma_length = 0; - - if (s->offset || (size & ~PAGE_MASK) || size + s->length > max) { - if (__map_sg_chunk(dev, start, size, &dma->dma_address, - dir, attrs, is_coherent) < 0) - goto bad_mapping; - - dma->dma_address += offset; - dma->dma_length = size - offset; - - size = offset = s->offset; - start = s; - dma = sg_next(dma); - count += 1; - } - size += s->length; - } - if (__map_sg_chunk(dev, start, size, &dma->dma_address, dir, attrs, - is_coherent) < 0) - goto bad_mapping; - - dma->dma_address += offset; - dma->dma_length = size - offset; - - return count+1; - -bad_mapping: - for_each_sg(sg, s, count, i) - __iommu_remove_mapping(dev, sg_dma_address(s), sg_dma_len(s)); - return 0; -} - -/** - * arm_coherent_iommu_map_sg - map a set of SG buffers for streaming mode DMA - * @dev: valid struct device pointer - * @sg: list of buffers - * @nents: number of buffers to map - * @dir: DMA transfer direction - * - * Map a set of i/o coherent buffers described by scatterlist in streaming - * mode for DMA. The scatter gather list elements are merged together (if - * possible) and tagged with the appropriate dma address and length. They are - * obtained via sg_dma_{address,length}. - */ -static int arm_coherent_iommu_map_sg(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir, unsigned long attrs) -{ - return __iommu_map_sg(dev, sg, nents, dir, attrs, true); -} - /** * arm_iommu_map_sg - map a set of SG buffers for streaming mode DMA * @dev: valid struct device pointer @@ -1695,41 +1594,45 @@ static int arm_coherent_iommu_map_sg(struct device *dev, struct scatterlist *sg, static int arm_iommu_map_sg(struct device *dev, struct scatterlist *sg, int nents, enum dma_data_direction dir, unsigned long attrs) { - return __iommu_map_sg(dev, sg, nents, dir, attrs, false); -} + struct scatterlist *s = sg, *dma = sg, *start = sg; + int i, count = 0; + unsigned int offset = s->offset; + unsigned int size = s->offset + s->length; + unsigned int max = dma_get_max_seg_size(dev); -static void __iommu_unmap_sg(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir, - unsigned long attrs, bool is_coherent) -{ - struct scatterlist *s; - int i; + for (i = 1; i < nents; i++) { + s = sg_next(s); - for_each_sg(sg, s, nents, i) { - if (sg_dma_len(s)) - __iommu_remove_mapping(dev, sg_dma_address(s), - sg_dma_len(s)); - if (!is_coherent && (attrs & DMA_ATTR_SKIP_CPU_SYNC) == 0) - __dma_page_dev_to_cpu(sg_page(s), s->offset, - s->length, dir); + s->dma_address = DMA_MAPPING_ERROR; + s->dma_length = 0; + + if (s->offset || (size & ~PAGE_MASK) || size + s->length > max) { + if (__map_sg_chunk(dev, start, size, &dma->dma_address, + dir, attrs) < 0) + goto bad_mapping; + + dma->dma_address += offset; + dma->dma_length = size - offset; + + size = offset = s->offset; + start = s; + dma = sg_next(dma); + count += 1; + } + size += s->length; } -} + if (__map_sg_chunk(dev, start, size, &dma->dma_address, dir, attrs) < 0) + goto bad_mapping; -/** - * arm_coherent_iommu_unmap_sg - unmap a set of SG buffers mapped by dma_map_sg - * @dev: valid struct device pointer - * @sg: list of buffers - * @nents: number of buffers to unmap (same as was passed to dma_map_sg) - * @dir: DMA transfer direction (same as was passed to dma_map_sg) - * - * Unmap a set of streaming mode DMA translations. Again, CPU access - * rules concerning calls here are the same as for dma_unmap_single(). - */ -static void arm_coherent_iommu_unmap_sg(struct device *dev, - struct scatterlist *sg, int nents, enum dma_data_direction dir, - unsigned long attrs) -{ - __iommu_unmap_sg(dev, sg, nents, dir, attrs, true); + dma->dma_address += offset; + dma->dma_length = size - offset; + + return count+1; + +bad_mapping: + for_each_sg(sg, s, count, i) + __iommu_remove_mapping(dev, sg_dma_address(s), sg_dma_len(s)); + return 0; } /** @@ -1747,7 +1650,17 @@ static void arm_iommu_unmap_sg(struct device *dev, enum dma_data_direction dir, unsigned long attrs) { - __iommu_unmap_sg(dev, sg, nents, dir, attrs, false); + struct scatterlist *s; + int i; + + for_each_sg(sg, s, nents, i) { + if (sg_dma_len(s)) + __iommu_remove_mapping(dev, sg_dma_address(s), + sg_dma_len(s)); + if (!dev->dma_coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) + __dma_page_dev_to_cpu(sg_page(s), s->offset, + s->length, dir); + } } /** @@ -1787,18 +1700,17 @@ static void arm_iommu_sync_sg_for_device(struct device *dev, __dma_page_cpu_to_dev(sg_page(s), s->offset, s->length, dir); } - /** - * arm_coherent_iommu_map_page + * arm_iommu_map_page * @dev: valid struct device pointer * @page: page that buffer resides in * @offset: offset into page for start of buffer * @size: size of buffer to map * @dir: DMA transfer direction * - * Coherent IOMMU aware version of arm_dma_map_page() + * IOMMU aware version of arm_dma_map_page() */ -static dma_addr_t arm_coherent_iommu_map_page(struct device *dev, struct page *page, +static dma_addr_t arm_iommu_map_page(struct device *dev, struct page *page, unsigned long offset, size_t size, enum dma_data_direction dir, unsigned long attrs) { @@ -1806,6 +1718,9 @@ static dma_addr_t arm_coherent_iommu_map_page(struct device *dev, struct page *p dma_addr_t dma_addr; int ret, prot, len = PAGE_ALIGN(size + offset); + if (!dev->dma_coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) + __dma_page_cpu_to_dev(page, offset, size, dir); + dma_addr = __alloc_iova(mapping, len); if (dma_addr == DMA_MAPPING_ERROR) return dma_addr; @@ -1822,50 +1737,6 @@ static dma_addr_t arm_coherent_iommu_map_page(struct device *dev, struct page *p return DMA_MAPPING_ERROR; } -/** - * arm_iommu_map_page - * @dev: valid struct device pointer - * @page: page that buffer resides in - * @offset: offset into page for start of buffer - * @size: size of buffer to map - * @dir: DMA transfer direction - * - * IOMMU aware version of arm_dma_map_page() - */ -static dma_addr_t arm_iommu_map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, enum dma_data_direction dir, - unsigned long attrs) -{ - if ((attrs & DMA_ATTR_SKIP_CPU_SYNC) == 0) - __dma_page_cpu_to_dev(page, offset, size, dir); - - return arm_coherent_iommu_map_page(dev, page, offset, size, dir, attrs); -} - -/** - * arm_coherent_iommu_unmap_page - * @dev: valid struct device pointer - * @handle: DMA address of buffer - * @size: size of buffer (same as passed to dma_map_page) - * @dir: DMA transfer direction (same as passed to dma_map_page) - * - * Coherent IOMMU aware version of arm_dma_unmap_page() - */ -static void arm_coherent_iommu_unmap_page(struct device *dev, dma_addr_t handle, - size_t size, enum dma_data_direction dir, unsigned long attrs) -{ - struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev); - dma_addr_t iova = handle & PAGE_MASK; - int offset = handle & ~PAGE_MASK; - int len = PAGE_ALIGN(size + offset); - - if (!iova) - return; - - iommu_unmap(mapping->domain, iova, len); - __free_iova(mapping, iova, len); -} - /** * arm_iommu_unmap_page * @dev: valid struct device pointer @@ -1880,15 +1751,17 @@ static void arm_iommu_unmap_page(struct device *dev, dma_addr_t handle, { struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev); dma_addr_t iova = handle & PAGE_MASK; - struct page *page = phys_to_page(iommu_iova_to_phys(mapping->domain, iova)); + struct page *page; int offset = handle & ~PAGE_MASK; int len = PAGE_ALIGN(size + offset); if (!iova) return; - if ((attrs & DMA_ATTR_SKIP_CPU_SYNC) == 0) + if (!dev->dma_coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) { + page = phys_to_page(iommu_iova_to_phys(mapping->domain, iova)); __dma_page_dev_to_cpu(page, offset, size, dir); + } iommu_unmap(mapping->domain, iova, len); __free_iova(mapping, iova, len); @@ -2000,16 +1873,16 @@ static const struct dma_map_ops iommu_ops = { }; static const struct dma_map_ops iommu_coherent_ops = { - .alloc = arm_coherent_iommu_alloc_attrs, - .free = arm_coherent_iommu_free_attrs, - .mmap = arm_coherent_iommu_mmap_attrs, + .alloc = arm_iommu_alloc_attrs, + .free = arm_iommu_free_attrs, + .mmap = arm_iommu_mmap_attrs, .get_sgtable = arm_iommu_get_sgtable, - .map_page = arm_coherent_iommu_map_page, - .unmap_page = arm_coherent_iommu_unmap_page, + .map_page = arm_iommu_map_page, + .unmap_page = arm_iommu_unmap_page, - .map_sg = arm_coherent_iommu_map_sg, - .unmap_sg = arm_coherent_iommu_unmap_sg, + .map_sg = arm_iommu_map_sg, + .unmap_sg = arm_iommu_unmap_sg, .map_resource = arm_iommu_map_resource, .unmap_resource = arm_iommu_unmap_resource, @@ -2291,7 +2164,7 @@ void arch_teardown_dma_ops(struct device *dev) set_dma_ops(dev, NULL); } -#ifdef CONFIG_SWIOTLB +#if defined(CONFIG_SWIOTLB) || defined(CONFIG_ARM_DMA_USE_IOMMU) void arch_sync_dma_for_device(phys_addr_t paddr, size_t size, enum dma_data_direction dir) { @@ -2319,4 +2192,4 @@ void arch_dma_free(struct device *dev, size_t size, void *cpu_addr, { __arm_dma_free(dev, size, cpu_addr, dma_handle, attrs, false); } -#endif /* CONFIG_SWIOTLB */ +#endif /* CONFIG_SWIOTLB || CONFIG_ARM_DMA_USE_IOMMU */ From patchwork Thu Aug 20 15:08:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 250592 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC81BC433DF for ; Thu, 20 Aug 2020 15:12:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B31AE208C7 for ; Thu, 20 Aug 2020 15:12:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729253AbgHTPMl (ORCPT ); Thu, 20 Aug 2020 11:12:41 -0400 Received: from foss.arm.com ([217.140.110.172]:40830 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728615AbgHTPJK (ORCPT ); Thu, 20 Aug 2020 11:09:10 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 09BE1106F; Thu, 20 Aug 2020 08:09:09 -0700 (PDT) Received: from e121345-lin.cambridge.arm.com (e121345-lin.cambridge.arm.com [10.1.196.37]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 6C78F3F6CF; Thu, 20 Aug 2020 08:09:05 -0700 (PDT) From: Robin Murphy To: hch@lst.de, joro@8bytes.org, linux@armlinux.org.uk Cc: will@kernel.org, inki.dae@samsung.com, sw0312.kim@samsung.com, kyungmin.park@samsung.com, m.szyprowski@samsung.com, agross@kernel.org, bjorn.andersson@linaro.org, thierry.reding@gmail.com, jonathanh@nvidia.com, vdumpa@nvidia.com, digetx@gmail.com, matthias.bgg@gmail.com, yong.wu@mediatek.com, geert+renesas@glider.be, magnus.damm@gmail.com, t-kristo@ti.com, s-anna@ti.com, laurent.pinchart@ideasonboard.com, linux-arm-kernel@lists.infradead.org, iommu@lists.linux-foundation.org, linux-samsung-soc@vger.kernel.org, linux-tegra@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-mediatek@lists.infradead.org, dri-devel@lists.freedesktop.org, linux-media@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 04/18] iommu/dma: Add temporary hacks for arch/arm Date: Thu, 20 Aug 2020 16:08:23 +0100 Message-Id: X-Mailer: git-send-email 2.28.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-arm-msm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org In order to wrangle arch/arm over to iommu_dma_ops, we first need to convert all its associated IOMMU drivers over to default domains, and deal with users of its public dma_iommu_mappinng API. Since that can't reasonably be done in a single patch, we've no choice but to go through an ugly transitional phase. That starts with exposing some hooks into iommu-dma's internals so that it can start to do most of the heavy lifting. Before you start thinking about how horrible that is, here's a zebra: , c@ `||||)\ < / Signed-off-by: Robin Murphy --- drivers/iommu/dma-iommu.c | 38 +++++++++++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 4959f5df21bd..ab157d155bf7 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -25,6 +25,19 @@ #include #include +#ifdef CONFIG_ARM +#include +#endif +static struct iommu_domain *__iommu_get_dma_domain(struct device *dev) +{ +#ifdef CONFIG_ARM + struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev); + if (mapping) + return mapping->domain; +#endif + return iommu_get_dma_domain(dev); +} + struct iommu_dma_msi_page { struct list_head list; dma_addr_t iova; @@ -298,8 +311,11 @@ static void iommu_dma_flush_iotlb_all(struct iova_domain *iovad) * avoid rounding surprises. If necessary, we reserve the page at address 0 * to ensure it is an invalid IOVA. It is safe to reinitialise a domain, but * any change which could make prior IOVAs invalid will fail. + * + * XXX: Not formally exported, but needs to be referenced + * from arch/arm/mm/dma-mapping.c temporarily */ -static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, +int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, u64 size, struct device *dev) { struct iommu_dma_cookie *cookie = domain->iova_cookie; @@ -456,7 +472,7 @@ static void iommu_dma_free_iova(struct iommu_dma_cookie *cookie, static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, size_t size) { - struct iommu_domain *domain = iommu_get_dma_domain(dev); + struct iommu_domain *domain = __iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; struct iova_domain *iovad = &cookie->iovad; size_t iova_off = iova_offset(iovad, dma_addr); @@ -478,7 +494,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, size_t size, int prot, u64 dma_mask) { - struct iommu_domain *domain = iommu_get_dma_domain(dev); + struct iommu_domain *domain = __iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; struct iova_domain *iovad = &cookie->iovad; size_t iova_off = iova_offset(iovad, phys); @@ -582,7 +598,7 @@ static struct page **__iommu_dma_alloc_pages(struct device *dev, static void *iommu_dma_alloc_remap(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs) { - struct iommu_domain *domain = iommu_get_dma_domain(dev); + struct iommu_domain *domain = __iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; struct iova_domain *iovad = &cookie->iovad; bool coherent = dev_is_dma_coherent(dev); @@ -678,7 +694,7 @@ static void iommu_dma_sync_single_for_cpu(struct device *dev, if (dev_is_dma_coherent(dev)) return; - phys = iommu_iova_to_phys(iommu_get_dma_domain(dev), dma_handle); + phys = iommu_iova_to_phys(__iommu_get_dma_domain(dev), dma_handle); arch_sync_dma_for_cpu(phys, size, dir); } @@ -690,7 +706,7 @@ static void iommu_dma_sync_single_for_device(struct device *dev, if (dev_is_dma_coherent(dev)) return; - phys = iommu_iova_to_phys(iommu_get_dma_domain(dev), dma_handle); + phys = iommu_iova_to_phys(__iommu_get_dma_domain(dev), dma_handle); arch_sync_dma_for_device(phys, size, dir); } @@ -831,7 +847,7 @@ static void __invalidate_sg(struct scatterlist *sg, int nents) static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, int nents, enum dma_data_direction dir, unsigned long attrs) { - struct iommu_domain *domain = iommu_get_dma_domain(dev); + struct iommu_domain *domain = __iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; struct iova_domain *iovad = &cookie->iovad; struct scatterlist *s, *prev = NULL; @@ -1112,12 +1128,16 @@ static int iommu_dma_get_sgtable(struct device *dev, struct sg_table *sgt, static unsigned long iommu_dma_get_merge_boundary(struct device *dev) { - struct iommu_domain *domain = iommu_get_dma_domain(dev); + struct iommu_domain *domain = __iommu_get_dma_domain(dev); return (1UL << __ffs(domain->pgsize_bitmap)) - 1; } -static const struct dma_map_ops iommu_dma_ops = { +/* + * XXX: Not formally exported, but needs to be referenced + * from arch/arm/mm/dma-mapping.c temporarily + */ +const struct dma_map_ops iommu_dma_ops = { .alloc = iommu_dma_alloc, .free = iommu_dma_free, .mmap = iommu_dma_mmap, From patchwork Thu Aug 20 15:08:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 250599 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A085C433E1 for ; Thu, 20 Aug 2020 15:09:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3F8D621734 for ; Thu, 20 Aug 2020 15:09:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728666AbgHTPJY (ORCPT ); Thu, 20 Aug 2020 11:09:24 -0400 Received: from foss.arm.com ([217.140.110.172]:40872 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727053AbgHTPJP (ORCPT ); Thu, 20 Aug 2020 11:09:15 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E01F2113E; Thu, 20 Aug 2020 08:09:12 -0700 (PDT) Received: from e121345-lin.cambridge.arm.com (e121345-lin.cambridge.arm.com [10.1.196.37]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 39E1D3F6CF; Thu, 20 Aug 2020 08:09:09 -0700 (PDT) From: Robin Murphy To: hch@lst.de, joro@8bytes.org, linux@armlinux.org.uk Cc: will@kernel.org, inki.dae@samsung.com, sw0312.kim@samsung.com, kyungmin.park@samsung.com, m.szyprowski@samsung.com, agross@kernel.org, bjorn.andersson@linaro.org, thierry.reding@gmail.com, jonathanh@nvidia.com, vdumpa@nvidia.com, digetx@gmail.com, matthias.bgg@gmail.com, yong.wu@mediatek.com, geert+renesas@glider.be, magnus.damm@gmail.com, t-kristo@ti.com, s-anna@ti.com, laurent.pinchart@ideasonboard.com, linux-arm-kernel@lists.infradead.org, iommu@lists.linux-foundation.org, linux-samsung-soc@vger.kernel.org, linux-tegra@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-mediatek@lists.infradead.org, dri-devel@lists.freedesktop.org, linux-media@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 05/18] ARM/dma-mapping: Switch to iommu_dma_ops Date: Thu, 20 Aug 2020 16:08:24 +0100 Message-Id: <4b51f1685a7ff88b673bf013ca6c27501e52f9b4.1597931876.git.robin.murphy@arm.com> X-Mailer: git-send-email 2.28.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-arm-msm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org With the IOMMU ops now looking much the same shape as iommu_dma_ops, switch them out in favour of the iommu-dma library, currently enhanced with temporary workarounds that allow it to also sit underneath the arch-specific API. With that in place, we can now start converting the remaining IOMMU drivers and consumers to work with IOMMU API default domains instead. Signed-off-by: Robin Murphy --- arch/arm/Kconfig | 24 +- arch/arm/include/asm/dma-iommu.h | 8 - arch/arm/mm/dma-mapping.c | 887 +------------------------------ drivers/iommu/Kconfig | 8 - drivers/media/platform/Kconfig | 1 - 5 files changed, 22 insertions(+), 906 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index b91273f9fd43..79406fe5cd6b 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -133,31 +133,11 @@ config ARM_HAS_SG_CHAIN bool config ARM_DMA_USE_IOMMU - bool + def_bool IOMMU_SUPPORT select ARM_HAS_SG_CHAIN + select IOMMU_DMA select NEED_SG_DMA_LENGTH -if ARM_DMA_USE_IOMMU - -config ARM_DMA_IOMMU_ALIGNMENT - int "Maximum PAGE_SIZE order of alignment for DMA IOMMU buffers" - range 4 9 - default 8 - help - DMA mapping framework by default aligns all buffers to the smallest - PAGE_SIZE order which is greater than or equal to the requested buffer - size. This works well for buffers up to a few hundreds kilobytes, but - for larger buffers it just a waste of address space. Drivers which has - relatively small addressing window (like 64Mib) might run out of - virtual space with just a few allocations. - - With this parameter you can specify the maximum PAGE_SIZE order for - DMA IOMMU buffers. Larger buffers will be aligned only to this - specified order. The order is expressed as a power of two multiplied - by the PAGE_SIZE. - -endif - config SYS_SUPPORTS_APM_EMULATION bool diff --git a/arch/arm/include/asm/dma-iommu.h b/arch/arm/include/asm/dma-iommu.h index 86405cc81385..f39cfa509fe4 100644 --- a/arch/arm/include/asm/dma-iommu.h +++ b/arch/arm/include/asm/dma-iommu.h @@ -13,14 +13,6 @@ struct dma_iommu_mapping { /* iommu specific data */ struct iommu_domain *domain; - unsigned long **bitmaps; /* array of bitmaps */ - unsigned int nr_bitmaps; /* nr of elements in array */ - unsigned int extensions; - size_t bitmap_size; /* size of a single bitmap */ - size_t bits; /* per bitmap */ - dma_addr_t base; - - spinlock_t lock; struct kref kref; }; diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index 0537c97cebe1..0f69ede44cd7 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include #include @@ -1074,812 +1075,9 @@ static const struct dma_map_ops *arm_get_dma_map_ops(bool coherent) #ifdef CONFIG_ARM_DMA_USE_IOMMU -static int __dma_info_to_prot(enum dma_data_direction dir, unsigned long attrs) -{ - int prot = 0; - - if (attrs & DMA_ATTR_PRIVILEGED) - prot |= IOMMU_PRIV; - - switch (dir) { - case DMA_BIDIRECTIONAL: - return prot | IOMMU_READ | IOMMU_WRITE; - case DMA_TO_DEVICE: - return prot | IOMMU_READ; - case DMA_FROM_DEVICE: - return prot | IOMMU_WRITE; - default: - return prot; - } -} - -/* IOMMU */ - -static int extend_iommu_mapping(struct dma_iommu_mapping *mapping); - -static inline dma_addr_t __alloc_iova(struct dma_iommu_mapping *mapping, - size_t size) -{ - unsigned int order = get_order(size); - unsigned int align = 0; - unsigned int count, start; - size_t mapping_size = mapping->bits << PAGE_SHIFT; - unsigned long flags; - dma_addr_t iova; - int i; - - if (order > CONFIG_ARM_DMA_IOMMU_ALIGNMENT) - order = CONFIG_ARM_DMA_IOMMU_ALIGNMENT; - - count = PAGE_ALIGN(size) >> PAGE_SHIFT; - align = (1 << order) - 1; - - spin_lock_irqsave(&mapping->lock, flags); - for (i = 0; i < mapping->nr_bitmaps; i++) { - start = bitmap_find_next_zero_area(mapping->bitmaps[i], - mapping->bits, 0, count, align); - - if (start > mapping->bits) - continue; - - bitmap_set(mapping->bitmaps[i], start, count); - break; - } - - /* - * No unused range found. Try to extend the existing mapping - * and perform a second attempt to reserve an IO virtual - * address range of size bytes. - */ - if (i == mapping->nr_bitmaps) { - if (extend_iommu_mapping(mapping)) { - spin_unlock_irqrestore(&mapping->lock, flags); - return DMA_MAPPING_ERROR; - } - - start = bitmap_find_next_zero_area(mapping->bitmaps[i], - mapping->bits, 0, count, align); - - if (start > mapping->bits) { - spin_unlock_irqrestore(&mapping->lock, flags); - return DMA_MAPPING_ERROR; - } - - bitmap_set(mapping->bitmaps[i], start, count); - } - spin_unlock_irqrestore(&mapping->lock, flags); - - iova = mapping->base + (mapping_size * i); - iova += start << PAGE_SHIFT; - - return iova; -} - -static inline void __free_iova(struct dma_iommu_mapping *mapping, - dma_addr_t addr, size_t size) -{ - unsigned int start, count; - size_t mapping_size = mapping->bits << PAGE_SHIFT; - unsigned long flags; - dma_addr_t bitmap_base; - u32 bitmap_index; - - if (!size) - return; - - bitmap_index = (u32) (addr - mapping->base) / (u32) mapping_size; - BUG_ON(addr < mapping->base || bitmap_index > mapping->extensions); - - bitmap_base = mapping->base + mapping_size * bitmap_index; - - start = (addr - bitmap_base) >> PAGE_SHIFT; - - if (addr + size > bitmap_base + mapping_size) { - /* - * The address range to be freed reaches into the iova - * range of the next bitmap. This should not happen as - * we don't allow this in __alloc_iova (at the - * moment). - */ - BUG(); - } else - count = size >> PAGE_SHIFT; - - spin_lock_irqsave(&mapping->lock, flags); - bitmap_clear(mapping->bitmaps[bitmap_index], start, count); - spin_unlock_irqrestore(&mapping->lock, flags); -} - -/* We'll try 2M, 1M, 64K, and finally 4K; array must end with 0! */ -static const int iommu_order_array[] = { 9, 8, 4, 0 }; - -static struct page **__iommu_alloc_buffer(struct device *dev, size_t size, - gfp_t gfp, unsigned long attrs, - int coherent_flag) -{ - struct page **pages; - int count = size >> PAGE_SHIFT; - int array_size = count * sizeof(struct page *); - int i = 0; - int order_idx = 0; - - if (array_size <= PAGE_SIZE) - pages = kzalloc(array_size, GFP_KERNEL); - else - pages = vzalloc(array_size); - if (!pages) - return NULL; - - if (attrs & DMA_ATTR_FORCE_CONTIGUOUS) - { - unsigned long order = get_order(size); - struct page *page; - - page = dma_alloc_from_contiguous(dev, count, order, - gfp & __GFP_NOWARN); - if (!page) - goto error; - - __dma_clear_buffer(page, size, coherent_flag); - - for (i = 0; i < count; i++) - pages[i] = page + i; - - return pages; - } - - /* Go straight to 4K chunks if caller says it's OK. */ - if (attrs & DMA_ATTR_ALLOC_SINGLE_PAGES) - order_idx = ARRAY_SIZE(iommu_order_array) - 1; - - /* - * IOMMU can map any pages, so himem can also be used here - */ - gfp |= __GFP_NOWARN | __GFP_HIGHMEM; - - while (count) { - int j, order; - - order = iommu_order_array[order_idx]; - - /* Drop down when we get small */ - if (__fls(count) < order) { - order_idx++; - continue; - } - - if (order) { - /* See if it's easy to allocate a high-order chunk */ - pages[i] = alloc_pages(gfp | __GFP_NORETRY, order); - - /* Go down a notch at first sign of pressure */ - if (!pages[i]) { - order_idx++; - continue; - } - } else { - pages[i] = alloc_pages(gfp, 0); - if (!pages[i]) - goto error; - } - - if (order) { - split_page(pages[i], order); - j = 1 << order; - while (--j) - pages[i + j] = pages[i] + j; - } - - __dma_clear_buffer(pages[i], PAGE_SIZE << order, coherent_flag); - i += 1 << order; - count -= 1 << order; - } - - return pages; -error: - while (i--) - if (pages[i]) - __free_pages(pages[i], 0); - kvfree(pages); - return NULL; -} - -static int __iommu_free_buffer(struct device *dev, struct page **pages, - size_t size, unsigned long attrs) -{ - int count = size >> PAGE_SHIFT; - int i; - - if (attrs & DMA_ATTR_FORCE_CONTIGUOUS) { - dma_release_from_contiguous(dev, pages[0], count); - } else { - for (i = 0; i < count; i++) - if (pages[i]) - __free_pages(pages[i], 0); - } - - kvfree(pages); - return 0; -} - -/* - * Create a mapping in device IO address space for specified pages - */ -static dma_addr_t -__iommu_create_mapping(struct device *dev, struct page **pages, size_t size, - unsigned long attrs) -{ - struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev); - unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT; - dma_addr_t dma_addr, iova; - int i; - - dma_addr = __alloc_iova(mapping, size); - if (dma_addr == DMA_MAPPING_ERROR) - return dma_addr; - - iova = dma_addr; - for (i = 0; i < count; ) { - int ret; - - unsigned int next_pfn = page_to_pfn(pages[i]) + 1; - phys_addr_t phys = page_to_phys(pages[i]); - unsigned int len, j; - - for (j = i + 1; j < count; j++, next_pfn++) - if (page_to_pfn(pages[j]) != next_pfn) - break; - - len = (j - i) << PAGE_SHIFT; - ret = iommu_map(mapping->domain, iova, phys, len, - __dma_info_to_prot(DMA_BIDIRECTIONAL, attrs)); - if (ret < 0) - goto fail; - iova += len; - i = j; - } - return dma_addr; -fail: - iommu_unmap(mapping->domain, dma_addr, iova-dma_addr); - __free_iova(mapping, dma_addr, size); - return DMA_MAPPING_ERROR; -} - -static int __iommu_remove_mapping(struct device *dev, dma_addr_t iova, size_t size) -{ - struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev); - - /* - * add optional in-page offset from iova to size and align - * result to page size - */ - size = PAGE_ALIGN((iova & ~PAGE_MASK) + size); - iova &= PAGE_MASK; - - iommu_unmap(mapping->domain, iova, size); - __free_iova(mapping, iova, size); - return 0; -} - -static struct page **__atomic_get_pages(void *addr) -{ - struct page *page; - phys_addr_t phys; - - phys = gen_pool_virt_to_phys(atomic_pool, (unsigned long)addr); - page = phys_to_page(phys); - - return (struct page **)page; -} - -static struct page **__iommu_get_pages(void *cpu_addr, unsigned long attrs) -{ - if (__in_atomic_pool(cpu_addr, PAGE_SIZE)) - return __atomic_get_pages(cpu_addr); - - if (attrs & DMA_ATTR_NO_KERNEL_MAPPING) - return cpu_addr; - - return dma_common_find_pages(cpu_addr); -} - -static void *__iommu_alloc_simple(struct device *dev, size_t size, gfp_t gfp, - dma_addr_t *handle, int coherent_flag, - unsigned long attrs) -{ - struct page *page; - void *addr; - - if (coherent_flag == COHERENT) - addr = __alloc_simple_buffer(dev, size, gfp, &page); - else - addr = __alloc_from_pool(size, &page); - if (!addr) - return NULL; - - *handle = __iommu_create_mapping(dev, &page, size, attrs); - if (*handle == DMA_MAPPING_ERROR) - goto err_mapping; - - return addr; - -err_mapping: - __free_from_pool(addr, size); - return NULL; -} - -static void __iommu_free_atomic(struct device *dev, void *cpu_addr, - dma_addr_t handle, size_t size, int coherent_flag) -{ - __iommu_remove_mapping(dev, handle, size); - if (coherent_flag == COHERENT) - __dma_free_buffer(virt_to_page(cpu_addr), size); - else - __free_from_pool(cpu_addr, size); -} - -static void *arm_iommu_alloc_attrs(struct device *dev, size_t size, - dma_addr_t *handle, gfp_t gfp, unsigned long attrs) -{ - pgprot_t prot = __get_dma_pgprot(attrs, PAGE_KERNEL); - struct page **pages; - void *addr = NULL; - int coherent_flag = dev->dma_coherent ? COHERENT : NORMAL; - - *handle = DMA_MAPPING_ERROR; - size = PAGE_ALIGN(size); - - if (coherent_flag == COHERENT || !gfpflags_allow_blocking(gfp)) - return __iommu_alloc_simple(dev, size, gfp, handle, - coherent_flag, attrs); - - /* - * Following is a work-around (a.k.a. hack) to prevent pages - * with __GFP_COMP being passed to split_page() which cannot - * handle them. The real problem is that this flag probably - * should be 0 on ARM as it is not supported on this - * platform; see CONFIG_HUGETLBFS. - */ - gfp &= ~(__GFP_COMP); - - pages = __iommu_alloc_buffer(dev, size, gfp, attrs, coherent_flag); - if (!pages) - return NULL; - - *handle = __iommu_create_mapping(dev, pages, size, attrs); - if (*handle == DMA_MAPPING_ERROR) - goto err_buffer; - - if (attrs & DMA_ATTR_NO_KERNEL_MAPPING) - return pages; - - addr = dma_common_pages_remap(pages, size, prot, - __builtin_return_address(0)); - if (!addr) - goto err_mapping; - - return addr; - -err_mapping: - __iommu_remove_mapping(dev, *handle, size); -err_buffer: - __iommu_free_buffer(dev, pages, size, attrs); - return NULL; -} - -static int arm_iommu_mmap_attrs(struct device *dev, struct vm_area_struct *vma, - void *cpu_addr, dma_addr_t dma_addr, size_t size, - unsigned long attrs) -{ - struct page **pages = __iommu_get_pages(cpu_addr, attrs); - unsigned long nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT; - int err; - - if (!pages) - return -ENXIO; - - if (vma->vm_pgoff >= nr_pages) - return -ENXIO; - - if (!dev->dma_coherent) - vma->vm_page_prot = __get_dma_pgprot(attrs, vma->vm_page_prot); - - err = vm_map_pages(vma, pages, nr_pages); - if (err) - pr_err("Remapping memory failed: %d\n", err); - - return err; -} - -/* - * free a page as defined by the above mapping. - * Must not be called with IRQs disabled. - */ -static void arm_iommu_free_attrs(struct device *dev, size_t size, void *cpu_addr, - dma_addr_t handle, unsigned long attrs) -{ - int coherent_flag = dev->dma_coherent ? COHERENT : NORMAL; - struct page **pages; - size = PAGE_ALIGN(size); - - if (coherent_flag == COHERENT || __in_atomic_pool(cpu_addr, size)) { - __iommu_free_atomic(dev, cpu_addr, handle, size, coherent_flag); - return; - } - - pages = __iommu_get_pages(cpu_addr, attrs); - if (!pages) { - WARN(1, "trying to free invalid coherent area: %p\n", cpu_addr); - return; - } - - if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) == 0) - dma_common_free_remap(cpu_addr, size); - - __iommu_remove_mapping(dev, handle, size); - __iommu_free_buffer(dev, pages, size, attrs); -} - -static int arm_iommu_get_sgtable(struct device *dev, struct sg_table *sgt, - void *cpu_addr, dma_addr_t dma_addr, - size_t size, unsigned long attrs) -{ - unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT; - struct page **pages = __iommu_get_pages(cpu_addr, attrs); - - if (!pages) - return -ENXIO; - - return sg_alloc_table_from_pages(sgt, pages, count, 0, size, - GFP_KERNEL); -} - -/* - * Map a part of the scatter-gather list into contiguous io address space - */ -static int __map_sg_chunk(struct device *dev, struct scatterlist *sg, - size_t size, dma_addr_t *handle, - enum dma_data_direction dir, unsigned long attrs) -{ - struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev); - dma_addr_t iova, iova_base; - int ret = 0; - unsigned int count; - struct scatterlist *s; - int prot; - - size = PAGE_ALIGN(size); - *handle = DMA_MAPPING_ERROR; - - iova_base = iova = __alloc_iova(mapping, size); - if (iova == DMA_MAPPING_ERROR) - return -ENOMEM; - - for (count = 0, s = sg; count < (size >> PAGE_SHIFT); s = sg_next(s)) { - phys_addr_t phys = page_to_phys(sg_page(s)); - unsigned int len = PAGE_ALIGN(s->offset + s->length); - - if (!dev->dma_coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - __dma_page_cpu_to_dev(sg_page(s), s->offset, s->length, dir); - - prot = __dma_info_to_prot(dir, attrs); - - ret = iommu_map(mapping->domain, iova, phys, len, prot); - if (ret < 0) - goto fail; - count += len >> PAGE_SHIFT; - iova += len; - } - *handle = iova_base; - - return 0; -fail: - iommu_unmap(mapping->domain, iova_base, count * PAGE_SIZE); - __free_iova(mapping, iova_base, size); - return ret; -} - -/** - * arm_iommu_map_sg - map a set of SG buffers for streaming mode DMA - * @dev: valid struct device pointer - * @sg: list of buffers - * @nents: number of buffers to map - * @dir: DMA transfer direction - * - * Map a set of buffers described by scatterlist in streaming mode for DMA. - * The scatter gather list elements are merged together (if possible) and - * tagged with the appropriate dma address and length. They are obtained via - * sg_dma_{address,length}. - */ -static int arm_iommu_map_sg(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir, unsigned long attrs) -{ - struct scatterlist *s = sg, *dma = sg, *start = sg; - int i, count = 0; - unsigned int offset = s->offset; - unsigned int size = s->offset + s->length; - unsigned int max = dma_get_max_seg_size(dev); - - for (i = 1; i < nents; i++) { - s = sg_next(s); - - s->dma_address = DMA_MAPPING_ERROR; - s->dma_length = 0; - - if (s->offset || (size & ~PAGE_MASK) || size + s->length > max) { - if (__map_sg_chunk(dev, start, size, &dma->dma_address, - dir, attrs) < 0) - goto bad_mapping; - - dma->dma_address += offset; - dma->dma_length = size - offset; - - size = offset = s->offset; - start = s; - dma = sg_next(dma); - count += 1; - } - size += s->length; - } - if (__map_sg_chunk(dev, start, size, &dma->dma_address, dir, attrs) < 0) - goto bad_mapping; - - dma->dma_address += offset; - dma->dma_length = size - offset; - - return count+1; - -bad_mapping: - for_each_sg(sg, s, count, i) - __iommu_remove_mapping(dev, sg_dma_address(s), sg_dma_len(s)); - return 0; -} - -/** - * arm_iommu_unmap_sg - unmap a set of SG buffers mapped by dma_map_sg - * @dev: valid struct device pointer - * @sg: list of buffers - * @nents: number of buffers to unmap (same as was passed to dma_map_sg) - * @dir: DMA transfer direction (same as was passed to dma_map_sg) - * - * Unmap a set of streaming mode DMA translations. Again, CPU access - * rules concerning calls here are the same as for dma_unmap_single(). - */ -static void arm_iommu_unmap_sg(struct device *dev, - struct scatterlist *sg, int nents, - enum dma_data_direction dir, - unsigned long attrs) -{ - struct scatterlist *s; - int i; - - for_each_sg(sg, s, nents, i) { - if (sg_dma_len(s)) - __iommu_remove_mapping(dev, sg_dma_address(s), - sg_dma_len(s)); - if (!dev->dma_coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - __dma_page_dev_to_cpu(sg_page(s), s->offset, - s->length, dir); - } -} - -/** - * arm_iommu_sync_sg_for_cpu - * @dev: valid struct device pointer - * @sg: list of buffers - * @nents: number of buffers to map (returned from dma_map_sg) - * @dir: DMA transfer direction (same as was passed to dma_map_sg) - */ -static void arm_iommu_sync_sg_for_cpu(struct device *dev, - struct scatterlist *sg, - int nents, enum dma_data_direction dir) -{ - struct scatterlist *s; - int i; - - if (dev->dma_coherent) - return; - - for_each_sg(sg, s, nents, i) - __dma_page_dev_to_cpu(sg_page(s), s->offset, s->length, dir); - -} - -/** - * arm_iommu_sync_sg_for_device - * @dev: valid struct device pointer - * @sg: list of buffers - * @nents: number of buffers to map (returned from dma_map_sg) - * @dir: DMA transfer direction (same as was passed to dma_map_sg) - */ -static void arm_iommu_sync_sg_for_device(struct device *dev, - struct scatterlist *sg, - int nents, enum dma_data_direction dir) -{ - struct scatterlist *s; - int i; - - if (dev->dma_coherent) - return; - - for_each_sg(sg, s, nents, i) - __dma_page_cpu_to_dev(sg_page(s), s->offset, s->length, dir); -} - -/** - * arm_iommu_map_page - * @dev: valid struct device pointer - * @page: page that buffer resides in - * @offset: offset into page for start of buffer - * @size: size of buffer to map - * @dir: DMA transfer direction - * - * IOMMU aware version of arm_dma_map_page() - */ -static dma_addr_t arm_iommu_map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, enum dma_data_direction dir, - unsigned long attrs) -{ - struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev); - dma_addr_t dma_addr; - int ret, prot, len = PAGE_ALIGN(size + offset); - - if (!dev->dma_coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - __dma_page_cpu_to_dev(page, offset, size, dir); - - dma_addr = __alloc_iova(mapping, len); - if (dma_addr == DMA_MAPPING_ERROR) - return dma_addr; - - prot = __dma_info_to_prot(dir, attrs); - - ret = iommu_map(mapping->domain, dma_addr, page_to_phys(page), len, prot); - if (ret < 0) - goto fail; - - return dma_addr + offset; -fail: - __free_iova(mapping, dma_addr, len); - return DMA_MAPPING_ERROR; -} - -/** - * arm_iommu_unmap_page - * @dev: valid struct device pointer - * @handle: DMA address of buffer - * @size: size of buffer (same as passed to dma_map_page) - * @dir: DMA transfer direction (same as passed to dma_map_page) - * - * IOMMU aware version of arm_dma_unmap_page() - */ -static void arm_iommu_unmap_page(struct device *dev, dma_addr_t handle, - size_t size, enum dma_data_direction dir, unsigned long attrs) -{ - struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev); - dma_addr_t iova = handle & PAGE_MASK; - struct page *page; - int offset = handle & ~PAGE_MASK; - int len = PAGE_ALIGN(size + offset); - - if (!iova) - return; - - if (!dev->dma_coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) { - page = phys_to_page(iommu_iova_to_phys(mapping->domain, iova)); - __dma_page_dev_to_cpu(page, offset, size, dir); - } - - iommu_unmap(mapping->domain, iova, len); - __free_iova(mapping, iova, len); -} - -/** - * arm_iommu_map_resource - map a device resource for DMA - * @dev: valid struct device pointer - * @phys_addr: physical address of resource - * @size: size of resource to map - * @dir: DMA transfer direction - */ -static dma_addr_t arm_iommu_map_resource(struct device *dev, - phys_addr_t phys_addr, size_t size, - enum dma_data_direction dir, unsigned long attrs) -{ - struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev); - dma_addr_t dma_addr; - int ret, prot; - phys_addr_t addr = phys_addr & PAGE_MASK; - unsigned int offset = phys_addr & ~PAGE_MASK; - size_t len = PAGE_ALIGN(size + offset); - - dma_addr = __alloc_iova(mapping, len); - if (dma_addr == DMA_MAPPING_ERROR) - return dma_addr; - - prot = __dma_info_to_prot(dir, attrs) | IOMMU_MMIO; - - ret = iommu_map(mapping->domain, dma_addr, addr, len, prot); - if (ret < 0) - goto fail; - - return dma_addr + offset; -fail: - __free_iova(mapping, dma_addr, len); - return DMA_MAPPING_ERROR; -} - -/** - * arm_iommu_unmap_resource - unmap a device DMA resource - * @dev: valid struct device pointer - * @dma_handle: DMA address to resource - * @size: size of resource to map - * @dir: DMA transfer direction - */ -static void arm_iommu_unmap_resource(struct device *dev, dma_addr_t dma_handle, - size_t size, enum dma_data_direction dir, - unsigned long attrs) -{ - struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev); - dma_addr_t iova = dma_handle & PAGE_MASK; - unsigned int offset = dma_handle & ~PAGE_MASK; - size_t len = PAGE_ALIGN(size + offset); - - if (!iova) - return; - - iommu_unmap(mapping->domain, iova, len); - __free_iova(mapping, iova, len); -} - -static void arm_iommu_sync_single_for_cpu(struct device *dev, - dma_addr_t handle, size_t size, enum dma_data_direction dir) -{ - struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev); - dma_addr_t iova = handle & PAGE_MASK; - struct page *page; - unsigned int offset = handle & ~PAGE_MASK; - - if (dev->dma_coherent || !iova) - return; - - page = phys_to_page(iommu_iova_to_phys(mapping->domain, iova)); - __dma_page_dev_to_cpu(page, offset, size, dir); -} - -static void arm_iommu_sync_single_for_device(struct device *dev, - dma_addr_t handle, size_t size, enum dma_data_direction dir) -{ - struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev); - dma_addr_t iova = handle & PAGE_MASK; - struct page *page; - unsigned int offset = handle & ~PAGE_MASK; - - if (dev->dma_coherent || !iova) - return; - - page = phys_to_page(iommu_iova_to_phys(mapping->domain, iova)); - __dma_page_cpu_to_dev(page, offset, size, dir); -} - -static const struct dma_map_ops iommu_ops = { - .alloc = arm_iommu_alloc_attrs, - .free = arm_iommu_free_attrs, - .mmap = arm_iommu_mmap_attrs, - .get_sgtable = arm_iommu_get_sgtable, - - .map_page = arm_iommu_map_page, - .unmap_page = arm_iommu_unmap_page, - .sync_single_for_cpu = arm_iommu_sync_single_for_cpu, - .sync_single_for_device = arm_iommu_sync_single_for_device, - - .map_sg = arm_iommu_map_sg, - .unmap_sg = arm_iommu_unmap_sg, - .sync_sg_for_cpu = arm_iommu_sync_sg_for_cpu, - .sync_sg_for_device = arm_iommu_sync_sg_for_device, - - .map_resource = arm_iommu_map_resource, - .unmap_resource = arm_iommu_unmap_resource, -}; - +extern const struct dma_map_ops iommu_dma_ops; +extern int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, + u64 size, struct device *dev); /** * arm_iommu_create_mapping * @bus: pointer to the bus holding the client device (for IOMMU calls) @@ -1896,55 +1094,31 @@ static const struct dma_map_ops iommu_ops = { struct dma_iommu_mapping * arm_iommu_create_mapping(struct bus_type *bus, dma_addr_t base, u64 size) { - unsigned int bits = size >> PAGE_SHIFT; - unsigned int bitmap_size = BITS_TO_LONGS(bits) * sizeof(long); struct dma_iommu_mapping *mapping; - int extensions = 1; int err = -ENOMEM; - /* currently only 32-bit DMA address space is supported */ - if (size > DMA_BIT_MASK(32) + 1) - return ERR_PTR(-ERANGE); - - if (!bitmap_size) - return ERR_PTR(-EINVAL); - - if (bitmap_size > PAGE_SIZE) { - extensions = bitmap_size / PAGE_SIZE; - bitmap_size = PAGE_SIZE; - } - - mapping = kzalloc(sizeof(struct dma_iommu_mapping), GFP_KERNEL); + mapping = kzalloc(sizeof(*mapping), GFP_KERNEL); if (!mapping) goto err; - mapping->bitmap_size = bitmap_size; - mapping->bitmaps = kcalloc(extensions, sizeof(unsigned long *), - GFP_KERNEL); - if (!mapping->bitmaps) - goto err2; - - mapping->bitmaps[0] = kzalloc(bitmap_size, GFP_KERNEL); - if (!mapping->bitmaps[0]) - goto err3; - - mapping->nr_bitmaps = 1; - mapping->extensions = extensions; - mapping->base = base; - mapping->bits = BITS_PER_BYTE * bitmap_size; - - spin_lock_init(&mapping->lock); - mapping->domain = iommu_domain_alloc(bus); if (!mapping->domain) + goto err2; + + err = iommu_get_dma_cookie(mapping->domain); + if (err) + goto err3; + + err = iommu_dma_init_domain(mapping->domain, base, size, NULL); + if (err) goto err4; kref_init(&mapping->kref); return mapping; err4: - kfree(mapping->bitmaps[0]); + iommu_put_dma_cookie(mapping->domain); err3: - kfree(mapping->bitmaps); + iommu_domain_free(mapping->domain); err2: kfree(mapping); err: @@ -1954,35 +1128,14 @@ EXPORT_SYMBOL_GPL(arm_iommu_create_mapping); static void release_iommu_mapping(struct kref *kref) { - int i; struct dma_iommu_mapping *mapping = container_of(kref, struct dma_iommu_mapping, kref); + iommu_put_dma_cookie(mapping->domain); iommu_domain_free(mapping->domain); - for (i = 0; i < mapping->nr_bitmaps; i++) - kfree(mapping->bitmaps[i]); - kfree(mapping->bitmaps); kfree(mapping); } -static int extend_iommu_mapping(struct dma_iommu_mapping *mapping) -{ - int next_bitmap; - - if (mapping->nr_bitmaps >= mapping->extensions) - return -EINVAL; - - next_bitmap = mapping->nr_bitmaps; - mapping->bitmaps[next_bitmap] = kzalloc(mapping->bitmap_size, - GFP_ATOMIC); - if (!mapping->bitmaps[next_bitmap]) - return -ENOMEM; - - mapping->nr_bitmaps++; - - return 0; -} - void arm_iommu_release_mapping(struct dma_iommu_mapping *mapping) { if (mapping) @@ -2028,7 +1181,7 @@ int arm_iommu_attach_device(struct device *dev, if (err) return err; - set_dma_ops(dev, &iommu_ops); + set_dma_ops(dev, &iommu_dma_ops); return 0; } EXPORT_SYMBOL_GPL(arm_iommu_attach_device); @@ -2126,7 +1279,7 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size, return; if (arm_setup_iommu_dma_ops(dev, dma_base, size, iommu)) - dma_ops = &iommu_ops; + dma_ops = &iommu_dma_ops; else dma_ops = arm_get_dma_map_ops(coherent); @@ -2149,7 +1302,7 @@ void arch_teardown_dma_ops(struct device *dev) set_dma_ops(dev, NULL); } -#if defined(CONFIG_SWIOTLB) || defined(CONFIG_ARM_DMA_USE_IOMMU) +#if defined(CONFIG_SWIOTLB) || defined(CONFIG_IOMMU_DMA) void arch_sync_dma_for_device(phys_addr_t paddr, size_t size, enum dma_data_direction dir) { @@ -2177,4 +1330,4 @@ void arch_dma_free(struct device *dev, size_t size, void *cpu_addr, { __arm_dma_free(dev, size, cpu_addr, dma_handle, attrs, false); } -#endif /* CONFIG_SWIOTLB || CONFIG_ARM_DMA_USE_IOMMU */ +#endif /* CONFIG_SWIOTLB || CONFIG_IOMMU_DMA */ diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index bef5d75e306b..ca0bdf826e9d 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -163,7 +163,6 @@ config ROCKCHIP_IOMMU bool "Rockchip IOMMU Support" depends on ARCH_ROCKCHIP || COMPILE_TEST select IOMMU_API - select ARM_DMA_USE_IOMMU help Support for IOMMUs found on Rockchip rk32xx SOCs. These IOMMUs allow virtualization of the address space used by most @@ -175,7 +174,6 @@ config SUN50I_IOMMU bool "Allwinner H6 IOMMU Support" depends on HAS_DMA depends on ARCH_SUNXI || COMPILE_TEST - select ARM_DMA_USE_IOMMU select IOMMU_API help Support for the IOMMU introduced in the Allwinner H6 SoCs. @@ -206,7 +204,6 @@ config EXYNOS_IOMMU depends on ARCH_EXYNOS || COMPILE_TEST depends on !CPU_BIG_ENDIAN # revisit driver if we can enable big-endian ptes select IOMMU_API - select ARM_DMA_USE_IOMMU help Support for the IOMMU (System MMU) of Samsung Exynos application processor family. This enables H/W multimedia accelerators to see @@ -229,7 +226,6 @@ config IPMMU_VMSA depends on ARCH_RENESAS || (COMPILE_TEST && !GENERIC_ATOMIC64) select IOMMU_API select IOMMU_IO_PGTABLE_LPAE - select ARM_DMA_USE_IOMMU help Support for the Renesas VMSA-compatible IPMMU found in the R-Mobile APE6, R-Car Gen2, and R-Car Gen3 SoCs. @@ -250,7 +246,6 @@ config ARM_SMMU depends on ARM64 || ARM || (COMPILE_TEST && !GENERIC_ATOMIC64) select IOMMU_API select IOMMU_IO_PGTABLE_LPAE - select ARM_DMA_USE_IOMMU if ARM help Support for implementations of the ARM System MMU architecture versions 1 and 2. @@ -334,7 +329,6 @@ config S390_AP_IOMMU config MTK_IOMMU bool "MTK IOMMU Support" depends on ARCH_MEDIATEK || COMPILE_TEST - select ARM_DMA_USE_IOMMU select IOMMU_API select IOMMU_IO_PGTABLE_ARMV7S select MEMORY @@ -350,7 +344,6 @@ config MTK_IOMMU_V1 bool "MTK IOMMU Version 1 (M4U gen1) Support" depends on ARM depends on ARCH_MEDIATEK || COMPILE_TEST - select ARM_DMA_USE_IOMMU select IOMMU_API select MEMORY select MTK_SMI @@ -367,7 +360,6 @@ config QCOM_IOMMU depends on ARCH_QCOM || (COMPILE_TEST && !GENERIC_ATOMIC64) select IOMMU_API select IOMMU_IO_PGTABLE_LPAE - select ARM_DMA_USE_IOMMU help Support for IOMMU on certain Qualcomm SoCs. diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig index c57ee78fa99d..9d6ff7b5f7f1 100644 --- a/drivers/media/platform/Kconfig +++ b/drivers/media/platform/Kconfig @@ -76,7 +76,6 @@ config VIDEO_OMAP3 depends on VIDEO_V4L2 && I2C depends on (ARCH_OMAP3 && OMAP_IOMMU) || COMPILE_TEST depends on COMMON_CLK && OF - select ARM_DMA_USE_IOMMU if OMAP_IOMMU select MEDIA_CONTROLLER select VIDEO_V4L2_SUBDEV_API select VIDEOBUF2_DMA_CONTIG From patchwork Thu Aug 20 15:08:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 250593 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C703C433E1 for ; Thu, 20 Aug 2020 15:12:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 233BA204EA for ; Thu, 20 Aug 2020 15:12:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729214AbgHTPMS (ORCPT ); Thu, 20 Aug 2020 11:12:18 -0400 Received: from foss.arm.com ([217.140.110.172]:40906 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728638AbgHTPJR (ORCPT ); Thu, 20 Aug 2020 11:09:17 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C34B713A1; Thu, 20 Aug 2020 08:09:16 -0700 (PDT) Received: from e121345-lin.cambridge.arm.com (e121345-lin.cambridge.arm.com [10.1.196.37]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 20A2C3F6CF; Thu, 20 Aug 2020 08:09:13 -0700 (PDT) From: Robin Murphy To: hch@lst.de, joro@8bytes.org, linux@armlinux.org.uk Cc: will@kernel.org, inki.dae@samsung.com, sw0312.kim@samsung.com, kyungmin.park@samsung.com, m.szyprowski@samsung.com, agross@kernel.org, bjorn.andersson@linaro.org, thierry.reding@gmail.com, jonathanh@nvidia.com, vdumpa@nvidia.com, digetx@gmail.com, matthias.bgg@gmail.com, yong.wu@mediatek.com, geert+renesas@glider.be, magnus.damm@gmail.com, t-kristo@ti.com, s-anna@ti.com, laurent.pinchart@ideasonboard.com, linux-arm-kernel@lists.infradead.org, iommu@lists.linux-foundation.org, linux-samsung-soc@vger.kernel.org, linux-tegra@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-mediatek@lists.infradead.org, dri-devel@lists.freedesktop.org, linux-media@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 06/18] ARM/dma-mapping: Support IOMMU default domains Date: Thu, 20 Aug 2020 16:08:25 +0100 Message-Id: <2144220ecbc88e9c1f1311c56db32f96dea05396.1597931876.git.robin.murphy@arm.com> X-Mailer: git-send-email 2.28.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-arm-msm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Now that iommu-dma is wired up, we can let it work as normal without the dma_iommu_mapping hacks if the IOMMU driver already supports default domains. Signed-off-by: Robin Murphy --- arch/arm/mm/dma-mapping.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index 0f69ede44cd7..2ef0afc17645 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -1220,6 +1220,13 @@ static bool arm_setup_iommu_dma_ops(struct device *dev, u64 dma_base, u64 size, if (!iommu) return false; + /* If a default domain exists, just let iommu-dma work normally */ + if (iommu_get_domain_for_dev(dev)) { + iommu_setup_dma_ops(dev, dma_base, size); + return true; + } + + /* Otherwise, use the workaround until the IOMMU driver is updated */ mapping = arm_iommu_create_mapping(dev->bus, dma_base, size); if (IS_ERR(mapping)) { pr_warn("Failed to create %llu-byte IOMMU mapping for device %s\n", @@ -1234,6 +1241,7 @@ static bool arm_setup_iommu_dma_ops(struct device *dev, u64 dma_base, u64 size, return false; } + set_dma_ops(dev, &iommu_dma_ops); return true; } @@ -1263,8 +1271,6 @@ static void arm_teardown_iommu_dma_ops(struct device *dev) { } void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size, const struct iommu_ops *iommu, bool coherent) { - const struct dma_map_ops *dma_ops; - dev->archdata.dma_coherent = coherent; #ifdef CONFIG_SWIOTLB dev->dma_coherent = coherent; @@ -1278,12 +1284,9 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size, if (dev->dma_ops) return; - if (arm_setup_iommu_dma_ops(dev, dma_base, size, iommu)) - dma_ops = &iommu_dma_ops; - else - dma_ops = arm_get_dma_map_ops(coherent); + set_dma_ops(dev, arm_get_dma_map_ops(coherent)); - set_dma_ops(dev, dma_ops); + arm_setup_iommu_dma_ops(dev, dma_base, size, iommu); #ifdef CONFIG_XEN if (xen_initial_domain()) From patchwork Thu Aug 20 15:08:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 250594 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51217C433E5 for ; Thu, 20 Aug 2020 15:12:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 31DD021775 for ; Thu, 20 Aug 2020 15:12:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729187AbgHTPMD (ORCPT ); Thu, 20 Aug 2020 11:12:03 -0400 Received: from foss.arm.com ([217.140.110.172]:41020 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727053AbgHTPJ3 (ORCPT ); Thu, 20 Aug 2020 11:09:29 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 370C8143B; Thu, 20 Aug 2020 08:09:28 -0700 (PDT) Received: from e121345-lin.cambridge.arm.com (e121345-lin.cambridge.arm.com [10.1.196.37]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 863493F6CF; Thu, 20 Aug 2020 08:09:24 -0700 (PDT) From: Robin Murphy To: hch@lst.de, joro@8bytes.org, linux@armlinux.org.uk Cc: will@kernel.org, inki.dae@samsung.com, sw0312.kim@samsung.com, kyungmin.park@samsung.com, m.szyprowski@samsung.com, agross@kernel.org, bjorn.andersson@linaro.org, thierry.reding@gmail.com, jonathanh@nvidia.com, vdumpa@nvidia.com, digetx@gmail.com, matthias.bgg@gmail.com, yong.wu@mediatek.com, geert+renesas@glider.be, magnus.damm@gmail.com, t-kristo@ti.com, s-anna@ti.com, laurent.pinchart@ideasonboard.com, linux-arm-kernel@lists.infradead.org, iommu@lists.linux-foundation.org, linux-samsung-soc@vger.kernel.org, linux-tegra@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-mediatek@lists.infradead.org, dri-devel@lists.freedesktop.org, linux-media@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 09/18] iommu/mediatek-v1: Add IOMMU_DOMAIN_DMA support Date: Thu, 20 Aug 2020 16:08:28 +0100 Message-Id: X-Mailer: git-send-email 2.28.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-arm-msm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Now that arch/arm is wired up for default domains and iommu-dma, implement the corresponding driver-side support for groups and DMA domains to replace the shared mapping workaround. Signed-off-by: Robin Murphy Tested-by: Yong Wu --- drivers/iommu/mtk_iommu.h | 2 - drivers/iommu/mtk_iommu_v1.c | 153 +++++++++++------------------------ 2 files changed, 48 insertions(+), 107 deletions(-) diff --git a/drivers/iommu/mtk_iommu.h b/drivers/iommu/mtk_iommu.h index 122925dbe547..6253e98d810c 100644 --- a/drivers/iommu/mtk_iommu.h +++ b/drivers/iommu/mtk_iommu.h @@ -67,8 +67,6 @@ struct mtk_iommu_data { struct iommu_device iommu; const struct mtk_iommu_plat_data *plat_data; - struct dma_iommu_mapping *mapping; /* For mtk_iommu_v1.c */ - struct list_head list; struct mtk_smi_larb_iommu larb_imu[MTK_LARB_NR_MAX]; }; diff --git a/drivers/iommu/mtk_iommu_v1.c b/drivers/iommu/mtk_iommu_v1.c index 82ddfe9170d4..40c89b8d3ac4 100644 --- a/drivers/iommu/mtk_iommu_v1.c +++ b/drivers/iommu/mtk_iommu_v1.c @@ -28,7 +28,6 @@ #include #include #include -#include #include #include #include @@ -240,13 +239,18 @@ static struct iommu_domain *mtk_iommu_domain_alloc(unsigned type) { struct mtk_iommu_domain *dom; - if (type != IOMMU_DOMAIN_UNMANAGED) + if (type != IOMMU_DOMAIN_UNMANAGED && type != IOMMU_DOMAIN_DMA) return NULL; dom = kzalloc(sizeof(*dom), GFP_KERNEL); if (!dom) return NULL; + if (type == IOMMU_DOMAIN_DMA && iommu_get_dma_cookie(&dom->domain)) { + kfree(dom); + return NULL; + } + return &dom->domain; } @@ -257,6 +261,7 @@ static void mtk_iommu_domain_free(struct iommu_domain *domain) dma_free_coherent(data->dev, M2701_IOMMU_PGT_SIZE, dom->pgt_va, dom->pgt_pa); + iommu_put_dma_cookie(domain); kfree(to_mtk_domain(domain)); } @@ -265,14 +270,8 @@ static int mtk_iommu_attach_device(struct iommu_domain *domain, { struct mtk_iommu_data *data = dev_iommu_priv_get(dev); struct mtk_iommu_domain *dom = to_mtk_domain(domain); - struct dma_iommu_mapping *mtk_mapping; int ret; - /* Only allow the domain created internally. */ - mtk_mapping = data->mapping; - if (mtk_mapping->domain != domain) - return 0; - if (!data->m4u_dom) { data->m4u_dom = dom; ret = mtk_iommu_domain_finalise(data); @@ -358,18 +357,39 @@ static phys_addr_t mtk_iommu_iova_to_phys(struct iommu_domain *domain, static const struct iommu_ops mtk_iommu_ops; -/* - * MTK generation one iommu HW only support one iommu domain, and all the client - * sharing the same iova address space. - */ -static int mtk_iommu_create_mapping(struct device *dev, - struct of_phandle_args *args) +static struct iommu_device *mtk_iommu_probe_device(struct device *dev) { struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); struct mtk_iommu_data *data; + + if (!fwspec || fwspec->ops != &mtk_iommu_ops) + return ERR_PTR(-ENODEV); /* Not a iommu client device */ + + data = dev_iommu_priv_get(dev); + + return &data->iommu; +} + +static void mtk_iommu_release_device(struct device *dev) +{ + struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); + + if (!fwspec || fwspec->ops != &mtk_iommu_ops) + return; + + iommu_fwspec_free(dev); +} + +static struct iommu_group *mtk_iommu_device_group(struct device *dev) +{ + struct mtk_iommu_data *data = dev_iommu_priv_get(dev); + + return iommu_group_ref_get(data->m4u_group); +} + +static int mtk_iommu_of_xlate(struct device *dev, struct of_phandle_args *args) +{ struct platform_device *m4updev; - struct dma_iommu_mapping *mtk_mapping; - int ret; if (args->args_count != 1) { dev_err(dev, "invalid #iommu-cells(%d) property for IOMMU\n", @@ -377,15 +397,6 @@ static int mtk_iommu_create_mapping(struct device *dev, return -EINVAL; } - if (!fwspec) { - ret = iommu_fwspec_init(dev, &args->np->fwnode, &mtk_iommu_ops); - if (ret) - return ret; - fwspec = dev_iommu_fwspec_get(dev); - } else if (dev_iommu_fwspec_get(dev)->ops != &mtk_iommu_ops) { - return -EINVAL; - } - if (!dev_iommu_priv_get(dev)) { /* Get the m4u device */ m4updev = of_find_device_by_node(args->np); @@ -395,83 +406,7 @@ static int mtk_iommu_create_mapping(struct device *dev, dev_iommu_priv_set(dev, platform_get_drvdata(m4updev)); } - ret = iommu_fwspec_add_ids(dev, args->args, 1); - if (ret) - return ret; - - data = dev_iommu_priv_get(dev); - mtk_mapping = data->mapping; - if (!mtk_mapping) { - /* MTK iommu support 4GB iova address space. */ - mtk_mapping = arm_iommu_create_mapping(&platform_bus_type, - 0, 1ULL << 32); - if (IS_ERR(mtk_mapping)) - return PTR_ERR(mtk_mapping); - - data->mapping = mtk_mapping; - } - - return 0; -} - -static int mtk_iommu_def_domain_type(struct device *dev) -{ - return IOMMU_DOMAIN_UNMANAGED; -} - -static struct iommu_device *mtk_iommu_probe_device(struct device *dev) -{ - struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); - struct of_phandle_args iommu_spec; - struct of_phandle_iterator it; - struct mtk_iommu_data *data; - int err; - - of_for_each_phandle(&it, err, dev->of_node, "iommus", - "#iommu-cells", -1) { - int count = of_phandle_iterator_args(&it, iommu_spec.args, - MAX_PHANDLE_ARGS); - iommu_spec.np = of_node_get(it.node); - iommu_spec.args_count = count; - - mtk_iommu_create_mapping(dev, &iommu_spec); - - /* dev->iommu_fwspec might have changed */ - fwspec = dev_iommu_fwspec_get(dev); - - of_node_put(iommu_spec.np); - } - - if (!fwspec || fwspec->ops != &mtk_iommu_ops) - return ERR_PTR(-ENODEV); /* Not a iommu client device */ - - data = dev_iommu_priv_get(dev); - - return &data->iommu; -} - -static void mtk_iommu_probe_finalize(struct device *dev) -{ - struct dma_iommu_mapping *mtk_mapping; - struct mtk_iommu_data *data; - int err; - - data = dev_iommu_priv_get(dev); - mtk_mapping = data->mapping; - - err = arm_iommu_attach_device(dev, mtk_mapping); - if (err) - dev_err(dev, "Can't create IOMMU mapping - DMA-OPS will not work\n"); -} - -static void mtk_iommu_release_device(struct device *dev) -{ - struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); - - if (!fwspec || fwspec->ops != &mtk_iommu_ops) - return; - - iommu_fwspec_free(dev); + return iommu_fwspec_add_ids(dev, args->args, 1); } static int mtk_iommu_hw_init(const struct mtk_iommu_data *data) @@ -524,10 +459,9 @@ static const struct iommu_ops mtk_iommu_ops = { .unmap = mtk_iommu_unmap, .iova_to_phys = mtk_iommu_iova_to_phys, .probe_device = mtk_iommu_probe_device, - .probe_finalize = mtk_iommu_probe_finalize, .release_device = mtk_iommu_release_device, - .def_domain_type = mtk_iommu_def_domain_type, - .device_group = generic_device_group, + .device_group = mtk_iommu_device_group, + .of_xlate = mtk_iommu_of_xlate, .pgsize_bitmap = ~0UL << MT2701_IOMMU_PAGE_SHIFT, }; @@ -626,6 +560,14 @@ static int mtk_iommu_probe(struct platform_device *pdev) if (ret) return ret; + /* + * MTK generation one iommu HW only support one iommu domain, + * and all the client sharing the same iova address space. + */ + data->m4u_group = iommu_group_alloc(); + if (IS_ERR(data->m4u_group)) + return PTR_ERR(data->m4u_group); + if (!iommu_present(&platform_bus_type)) bus_set_iommu(&platform_bus_type, &mtk_iommu_ops); @@ -636,6 +578,7 @@ static int mtk_iommu_remove(struct platform_device *pdev) { struct mtk_iommu_data *data = platform_get_drvdata(pdev); + iommu_group_put(data->m4u_group); iommu_device_sysfs_remove(&data->iommu); iommu_device_unregister(&data->iommu); From patchwork Thu Aug 20 15:08:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 250595 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8654EC433E3 for ; Thu, 20 Aug 2020 15:11:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6C7B5208DB for ; Thu, 20 Aug 2020 15:11:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729136AbgHTPLx (ORCPT ); Thu, 20 Aug 2020 11:11:53 -0400 Received: from foss.arm.com ([217.140.110.172]:41100 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728722AbgHTPJg (ORCPT ); Thu, 20 Aug 2020 11:09:36 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A704E1477; Thu, 20 Aug 2020 08:09:35 -0700 (PDT) Received: from e121345-lin.cambridge.arm.com (e121345-lin.cambridge.arm.com [10.1.196.37]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 226C33F6CF; Thu, 20 Aug 2020 08:09:32 -0700 (PDT) From: Robin Murphy To: hch@lst.de, joro@8bytes.org, linux@armlinux.org.uk Cc: will@kernel.org, inki.dae@samsung.com, sw0312.kim@samsung.com, kyungmin.park@samsung.com, m.szyprowski@samsung.com, agross@kernel.org, bjorn.andersson@linaro.org, thierry.reding@gmail.com, jonathanh@nvidia.com, vdumpa@nvidia.com, digetx@gmail.com, matthias.bgg@gmail.com, yong.wu@mediatek.com, geert+renesas@glider.be, magnus.damm@gmail.com, t-kristo@ti.com, s-anna@ti.com, laurent.pinchart@ideasonboard.com, linux-arm-kernel@lists.infradead.org, iommu@lists.linux-foundation.org, linux-samsung-soc@vger.kernel.org, linux-tegra@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-mediatek@lists.infradead.org, dri-devel@lists.freedesktop.org, linux-media@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 11/18] iommu/omap: Add IOMMU_DOMAIN_DMA support Date: Thu, 20 Aug 2020 16:08:30 +0100 Message-Id: <5ac3788f9f61f7698cfa9c5924d62714e230f678.1597931876.git.robin.murphy@arm.com> X-Mailer: git-send-email 2.28.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-arm-msm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Now that arch/arm is wired up for default domains and iommu-dma, implement the corresponding driver-side support for DMA domains. Signed-off-by: Robin Murphy --- drivers/iommu/omap-iommu.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/omap-iommu.c b/drivers/iommu/omap-iommu.c index 71f29c0927fc..ea25c2fe0418 100644 --- a/drivers/iommu/omap-iommu.c +++ b/drivers/iommu/omap-iommu.c @@ -9,6 +9,7 @@ * Paul Mundt and Toshihiro Kobayashi */ +#include #include #include #include @@ -1574,13 +1575,19 @@ static struct iommu_domain *omap_iommu_domain_alloc(unsigned type) { struct omap_iommu_domain *omap_domain; - if (type != IOMMU_DOMAIN_UNMANAGED) + if (type != IOMMU_DOMAIN_UNMANAGED && type != IOMMU_DOMAIN_DMA) return NULL; omap_domain = kzalloc(sizeof(*omap_domain), GFP_KERNEL); if (!omap_domain) return NULL; + if (type == IOMMU_DOMAIN_DMA && + iommu_get_dma_cookie(&omap_domain->domain)) { + kfree(omap_domain); + return NULL; + } + spin_lock_init(&omap_domain->lock); omap_domain->domain.geometry.aperture_start = 0; @@ -1601,6 +1608,7 @@ static void omap_iommu_domain_free(struct iommu_domain *domain) if (omap_domain->dev) _omap_iommu_detach_dev(omap_domain, omap_domain->dev); + iommu_put_dma_cookie(&omap_domain->domain); kfree(omap_domain); } @@ -1736,6 +1744,17 @@ static struct iommu_group *omap_iommu_device_group(struct device *dev) return group; } +static int omap_iommu_of_xlate(struct device *dev, + struct of_phandle_args *args) +{ + /* + * Logically, some of the housekeeping from _omap_iommu_add_device() + * should probably move here, but the minimum we *need* is simply to + * cooperate with of_iommu at all to let iommu-dma work. + */ + return 0; +} + static const struct iommu_ops omap_iommu_ops = { .domain_alloc = omap_iommu_domain_alloc, .domain_free = omap_iommu_domain_free, @@ -1747,6 +1766,7 @@ static const struct iommu_ops omap_iommu_ops = { .probe_device = omap_iommu_probe_device, .release_device = omap_iommu_release_device, .device_group = omap_iommu_device_group, + .of_xlate = omap_iommu_of_xlate, .pgsize_bitmap = OMAP_IOMMU_PGSIZES, }; From patchwork Thu Aug 20 15:08:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 250596 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF6E1C433DF for ; Thu, 20 Aug 2020 15:11:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A2B59208DB for ; Thu, 20 Aug 2020 15:11:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729092AbgHTPL3 (ORCPT ); Thu, 20 Aug 2020 11:11:29 -0400 Received: from foss.arm.com ([217.140.110.172]:41190 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728765AbgHTPJo (ORCPT ); Thu, 20 Aug 2020 11:09:44 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4B97B1509; Thu, 20 Aug 2020 08:09:43 -0700 (PDT) Received: from e121345-lin.cambridge.arm.com (e121345-lin.cambridge.arm.com [10.1.196.37]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id AAB5A3F6CF; Thu, 20 Aug 2020 08:09:39 -0700 (PDT) From: Robin Murphy To: hch@lst.de, joro@8bytes.org, linux@armlinux.org.uk Cc: will@kernel.org, inki.dae@samsung.com, sw0312.kim@samsung.com, kyungmin.park@samsung.com, m.szyprowski@samsung.com, agross@kernel.org, bjorn.andersson@linaro.org, thierry.reding@gmail.com, jonathanh@nvidia.com, vdumpa@nvidia.com, digetx@gmail.com, matthias.bgg@gmail.com, yong.wu@mediatek.com, geert+renesas@glider.be, magnus.damm@gmail.com, t-kristo@ti.com, s-anna@ti.com, laurent.pinchart@ideasonboard.com, linux-arm-kernel@lists.infradead.org, iommu@lists.linux-foundation.org, linux-samsung-soc@vger.kernel.org, linux-tegra@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-mediatek@lists.infradead.org, dri-devel@lists.freedesktop.org, linux-media@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 13/18] iommu/tegra: Add IOMMU_DOMAIN_DMA support Date: Thu, 20 Aug 2020 16:08:32 +0100 Message-Id: X-Mailer: git-send-email 2.28.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-arm-msm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Now that arch/arm is wired up for default domains and iommu-dma, implement the corresponding driver-side support for DMA domains. Signed-off-by: Robin Murphy --- drivers/iommu/tegra-smmu.c | 37 +++++++++++++++++++++---------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/drivers/iommu/tegra-smmu.c b/drivers/iommu/tegra-smmu.c index 124c8848ab7e..8e276eac84d9 100644 --- a/drivers/iommu/tegra-smmu.c +++ b/drivers/iommu/tegra-smmu.c @@ -5,6 +5,7 @@ #include #include +#include #include #include #include @@ -278,7 +279,7 @@ static struct iommu_domain *tegra_smmu_domain_alloc(unsigned type) { struct tegra_smmu_as *as; - if (type != IOMMU_DOMAIN_UNMANAGED) + if (type != IOMMU_DOMAIN_UNMANAGED && type != IOMMU_DOMAIN_DMA) return NULL; as = kzalloc(sizeof(*as), GFP_KERNEL); @@ -288,25 +289,19 @@ static struct iommu_domain *tegra_smmu_domain_alloc(unsigned type) as->attr = SMMU_PD_READABLE | SMMU_PD_WRITABLE | SMMU_PD_NONSECURE; as->pd = alloc_page(GFP_KERNEL | __GFP_DMA | __GFP_ZERO); - if (!as->pd) { - kfree(as); - return NULL; - } + if (!as->pd) + goto out_free_as; as->count = kcalloc(SMMU_NUM_PDE, sizeof(u32), GFP_KERNEL); - if (!as->count) { - __free_page(as->pd); - kfree(as); - return NULL; - } + if (!as->count) + goto out_free_all; as->pts = kcalloc(SMMU_NUM_PDE, sizeof(*as->pts), GFP_KERNEL); - if (!as->pts) { - kfree(as->count); - __free_page(as->pd); - kfree(as); - return NULL; - } + if (!as->pts) + goto out_free_all; + + if (type == IOMMU_DOMAIN_DMA && iommu_get_dma_cookie(&as->domain)) + goto out_free_all; /* setup aperture */ as->domain.geometry.aperture_start = 0; @@ -314,12 +309,22 @@ static struct iommu_domain *tegra_smmu_domain_alloc(unsigned type) as->domain.geometry.force_aperture = true; return &as->domain; + +out_free_all: + kfree(as->pts); + kfree(as->count); + __free_page(as->pd); +out_free_as: + kfree(as); + return NULL; } static void tegra_smmu_domain_free(struct iommu_domain *domain) { struct tegra_smmu_as *as = to_smmu_as(domain); + iommu_put_dma_cookie(domain); + /* TODO: free page directory and page tables */ WARN_ON_ONCE(as->use_count); From patchwork Thu Aug 20 15:08:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 250597 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 669E3C433E3 for ; Thu, 20 Aug 2020 15:11:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 465B7208DB for ; Thu, 20 Aug 2020 15:11:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729020AbgHTPLF (ORCPT ); Thu, 20 Aug 2020 11:11:05 -0400 Received: from foss.arm.com ([217.140.110.172]:41342 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728824AbgHTPJ7 (ORCPT ); Thu, 20 Aug 2020 11:09:59 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 84CA2152B; Thu, 20 Aug 2020 08:09:58 -0700 (PDT) Received: from e121345-lin.cambridge.arm.com (e121345-lin.cambridge.arm.com [10.1.196.37]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id EA9CB3F6CF; Thu, 20 Aug 2020 08:09:54 -0700 (PDT) From: Robin Murphy To: hch@lst.de, joro@8bytes.org, linux@armlinux.org.uk Cc: will@kernel.org, inki.dae@samsung.com, sw0312.kim@samsung.com, kyungmin.park@samsung.com, m.szyprowski@samsung.com, agross@kernel.org, bjorn.andersson@linaro.org, thierry.reding@gmail.com, jonathanh@nvidia.com, vdumpa@nvidia.com, digetx@gmail.com, matthias.bgg@gmail.com, yong.wu@mediatek.com, geert+renesas@glider.be, magnus.damm@gmail.com, t-kristo@ti.com, s-anna@ti.com, laurent.pinchart@ideasonboard.com, linux-arm-kernel@lists.infradead.org, iommu@lists.linux-foundation.org, linux-samsung-soc@vger.kernel.org, linux-tegra@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-mediatek@lists.infradead.org, dri-devel@lists.freedesktop.org, linux-media@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 17/18] media/omap3isp: Clean up IOMMU workaround Date: Thu, 20 Aug 2020 16:08:36 +0100 Message-Id: <11d8419744e4e744a9448180801b0c4683328afd.1597931876.git.robin.murphy@arm.com> X-Mailer: git-send-email 2.28.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-arm-msm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Now that arch/arm is wired up for default domains and iommu-dma, devices behind IOMMUs will get mappings set up automatically as appropriate, so there is no need for drivers to do so manually. Signed-off-by: Robin Murphy --- drivers/media/platform/omap3isp/isp.c | 68 ++------------------------- drivers/media/platform/omap3isp/isp.h | 3 -- 2 files changed, 3 insertions(+), 68 deletions(-) diff --git a/drivers/media/platform/omap3isp/isp.c b/drivers/media/platform/omap3isp/isp.c index b91e472ee764..196522883231 100644 --- a/drivers/media/platform/omap3isp/isp.c +++ b/drivers/media/platform/omap3isp/isp.c @@ -56,10 +56,6 @@ #include #include -#ifdef CONFIG_ARM_DMA_USE_IOMMU -#include -#endif - #include #include #include @@ -1942,51 +1938,6 @@ static int isp_initialize_modules(struct isp_device *isp) return ret; } -static void isp_detach_iommu(struct isp_device *isp) -{ -#ifdef CONFIG_ARM_DMA_USE_IOMMU - arm_iommu_detach_device(isp->dev); - arm_iommu_release_mapping(isp->mapping); - isp->mapping = NULL; -#endif -} - -static int isp_attach_iommu(struct isp_device *isp) -{ -#ifdef CONFIG_ARM_DMA_USE_IOMMU - struct dma_iommu_mapping *mapping; - int ret; - - /* - * Create the ARM mapping, used by the ARM DMA mapping core to allocate - * VAs. This will allocate a corresponding IOMMU domain. - */ - mapping = arm_iommu_create_mapping(&platform_bus_type, SZ_1G, SZ_2G); - if (IS_ERR(mapping)) { - dev_err(isp->dev, "failed to create ARM IOMMU mapping\n"); - return PTR_ERR(mapping); - } - - isp->mapping = mapping; - - /* Attach the ARM VA mapping to the device. */ - ret = arm_iommu_attach_device(isp->dev, mapping); - if (ret < 0) { - dev_err(isp->dev, "failed to attach device to VA mapping\n"); - goto error; - } - - return 0; - -error: - arm_iommu_release_mapping(isp->mapping); - isp->mapping = NULL; - return ret; -#else - return -ENODEV; -#endif -} - /* * isp_remove - Remove ISP platform device * @pdev: Pointer to ISP platform device @@ -2002,10 +1953,6 @@ static int isp_remove(struct platform_device *pdev) isp_cleanup_modules(isp); isp_xclk_cleanup(isp); - __omap3isp_get(isp, false); - isp_detach_iommu(isp); - __omap3isp_put(isp, false); - media_entity_enum_cleanup(&isp->crashed); v4l2_async_notifier_cleanup(&isp->notifier); @@ -2383,18 +2330,11 @@ static int isp_probe(struct platform_device *pdev) isp->mmio_hist_base_phys = mem->start + isp_res_maps[m].offset[OMAP3_ISP_IOMEM_HIST]; - /* IOMMU */ - ret = isp_attach_iommu(isp); - if (ret < 0) { - dev_err(&pdev->dev, "unable to attach to IOMMU\n"); - goto error_isp; - } - /* Interrupt */ ret = platform_get_irq(pdev, 0); if (ret <= 0) { ret = -ENODEV; - goto error_iommu; + goto error_isp; } isp->irq_num = ret; @@ -2402,13 +2342,13 @@ static int isp_probe(struct platform_device *pdev) "OMAP3 ISP", isp)) { dev_err(isp->dev, "Unable to request IRQ\n"); ret = -EINVAL; - goto error_iommu; + goto error_isp; } /* Entities */ ret = isp_initialize_modules(isp); if (ret < 0) - goto error_iommu; + goto error_isp; ret = isp_register_entities(isp); if (ret < 0) @@ -2433,8 +2373,6 @@ static int isp_probe(struct platform_device *pdev) isp_unregister_entities(isp); error_modules: isp_cleanup_modules(isp); -error_iommu: - isp_detach_iommu(isp); error_isp: isp_xclk_cleanup(isp); __omap3isp_put(isp, false); diff --git a/drivers/media/platform/omap3isp/isp.h b/drivers/media/platform/omap3isp/isp.h index a9d760fbf349..b50459106d89 100644 --- a/drivers/media/platform/omap3isp/isp.h +++ b/drivers/media/platform/omap3isp/isp.h @@ -145,7 +145,6 @@ struct isp_xclk { * @syscon: Regmap for the syscon register space * @syscon_offset: Offset of the CSIPHY control register in syscon * @phy_type: ISP_PHY_TYPE_{3430,3630} - * @mapping: IOMMU mapping * @stat_lock: Spinlock for handling statistics * @isp_mutex: Mutex for serializing requests to ISP. * @stop_failure: Indicates that an entity failed to stop. @@ -185,8 +184,6 @@ struct isp_device { u32 syscon_offset; u32 phy_type; - struct dma_iommu_mapping *mapping; - /* ISP Obj */ spinlock_t stat_lock; /* common lock for statistic drivers */ struct mutex isp_mutex; /* For handling ref_count field */ From patchwork Thu Aug 20 15:08:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 250598 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01383C433E8 for ; Thu, 20 Aug 2020 15:10:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D4170204EA for ; Thu, 20 Aug 2020 15:10:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728916AbgHTPKq (ORCPT ); Thu, 20 Aug 2020 11:10:46 -0400 Received: from foss.arm.com ([217.140.110.172]:41298 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728838AbgHTPKE (ORCPT ); Thu, 20 Aug 2020 11:10:04 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 51BFC1597; Thu, 20 Aug 2020 08:10:02 -0700 (PDT) Received: from e121345-lin.cambridge.arm.com (e121345-lin.cambridge.arm.com [10.1.196.37]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B6D8F3F6CF; Thu, 20 Aug 2020 08:09:58 -0700 (PDT) From: Robin Murphy To: hch@lst.de, joro@8bytes.org, linux@armlinux.org.uk Cc: will@kernel.org, inki.dae@samsung.com, sw0312.kim@samsung.com, kyungmin.park@samsung.com, m.szyprowski@samsung.com, agross@kernel.org, bjorn.andersson@linaro.org, thierry.reding@gmail.com, jonathanh@nvidia.com, vdumpa@nvidia.com, digetx@gmail.com, matthias.bgg@gmail.com, yong.wu@mediatek.com, geert+renesas@glider.be, magnus.damm@gmail.com, t-kristo@ti.com, s-anna@ti.com, laurent.pinchart@ideasonboard.com, linux-arm-kernel@lists.infradead.org, iommu@lists.linux-foundation.org, linux-samsung-soc@vger.kernel.org, linux-tegra@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-mediatek@lists.infradead.org, dri-devel@lists.freedesktop.org, linux-media@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 18/18] ARM/dma-mapping: Remove legacy dma-iommu API Date: Thu, 20 Aug 2020 16:08:37 +0100 Message-Id: X-Mailer: git-send-email 2.28.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-arm-msm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org With no users left and generic iommu-dma now doing all the work, clean up the last traces of the arch-specific API, plus the temporary workarounds that you'd forgotten about because you were thinking about zebras instead. Signed-off-by: Robin Murphy --- arch/arm/common/dmabounce.c | 1 - arch/arm/include/asm/device.h | 9 -- arch/arm/include/asm/dma-iommu.h | 29 ----- arch/arm/mm/dma-mapping.c | 200 +------------------------------ drivers/iommu/dma-iommu.c | 38 ++---- 5 files changed, 11 insertions(+), 266 deletions(-) delete mode 100644 arch/arm/include/asm/dma-iommu.h diff --git a/arch/arm/common/dmabounce.c b/arch/arm/common/dmabounce.c index f4b719bde763..064349df7bbf 100644 --- a/arch/arm/common/dmabounce.c +++ b/arch/arm/common/dmabounce.c @@ -30,7 +30,6 @@ #include #include -#include #undef STATS diff --git a/arch/arm/include/asm/device.h b/arch/arm/include/asm/device.h index be666f58bf7a..db33f389c94e 100644 --- a/arch/arm/include/asm/device.h +++ b/arch/arm/include/asm/device.h @@ -8,9 +8,6 @@ struct dev_archdata { #ifdef CONFIG_DMABOUNCE struct dmabounce_device_info *dmabounce; -#endif -#ifdef CONFIG_ARM_DMA_USE_IOMMU - struct dma_iommu_mapping *mapping; #endif unsigned int dma_coherent:1; unsigned int dma_ops_setup:1; @@ -24,10 +21,4 @@ struct pdev_archdata { #endif }; -#ifdef CONFIG_ARM_DMA_USE_IOMMU -#define to_dma_iommu_mapping(dev) ((dev)->archdata.mapping) -#else -#define to_dma_iommu_mapping(dev) NULL -#endif - #endif diff --git a/arch/arm/include/asm/dma-iommu.h b/arch/arm/include/asm/dma-iommu.h deleted file mode 100644 index f39cfa509fe4..000000000000 --- a/arch/arm/include/asm/dma-iommu.h +++ /dev/null @@ -1,29 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef ASMARM_DMA_IOMMU_H -#define ASMARM_DMA_IOMMU_H - -#ifdef __KERNEL__ - -#include -#include -#include -#include - -struct dma_iommu_mapping { - /* iommu specific data */ - struct iommu_domain *domain; - - struct kref kref; -}; - -struct dma_iommu_mapping * -arm_iommu_create_mapping(struct bus_type *bus, dma_addr_t base, u64 size); - -void arm_iommu_release_mapping(struct dma_iommu_mapping *mapping); - -int arm_iommu_attach_device(struct device *dev, - struct dma_iommu_mapping *mapping); -void arm_iommu_detach_device(struct device *dev); - -#endif /* __KERNEL__ */ -#endif diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index 2ef0afc17645..ff6c4962161a 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -33,7 +33,6 @@ #include #include #include -#include #include #include #include @@ -1073,201 +1072,6 @@ static const struct dma_map_ops *arm_get_dma_map_ops(bool coherent) return coherent ? &arm_coherent_dma_ops : &arm_dma_ops; } -#ifdef CONFIG_ARM_DMA_USE_IOMMU - -extern const struct dma_map_ops iommu_dma_ops; -extern int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, - u64 size, struct device *dev); -/** - * arm_iommu_create_mapping - * @bus: pointer to the bus holding the client device (for IOMMU calls) - * @base: start address of the valid IO address space - * @size: maximum size of the valid IO address space - * - * Creates a mapping structure which holds information about used/unused - * IO address ranges, which is required to perform memory allocation and - * mapping with IOMMU aware functions. - * - * The client device need to be attached to the mapping with - * arm_iommu_attach_device function. - */ -struct dma_iommu_mapping * -arm_iommu_create_mapping(struct bus_type *bus, dma_addr_t base, u64 size) -{ - struct dma_iommu_mapping *mapping; - int err = -ENOMEM; - - mapping = kzalloc(sizeof(*mapping), GFP_KERNEL); - if (!mapping) - goto err; - - mapping->domain = iommu_domain_alloc(bus); - if (!mapping->domain) - goto err2; - - err = iommu_get_dma_cookie(mapping->domain); - if (err) - goto err3; - - err = iommu_dma_init_domain(mapping->domain, base, size, NULL); - if (err) - goto err4; - - kref_init(&mapping->kref); - return mapping; -err4: - iommu_put_dma_cookie(mapping->domain); -err3: - iommu_domain_free(mapping->domain); -err2: - kfree(mapping); -err: - return ERR_PTR(err); -} -EXPORT_SYMBOL_GPL(arm_iommu_create_mapping); - -static void release_iommu_mapping(struct kref *kref) -{ - struct dma_iommu_mapping *mapping = - container_of(kref, struct dma_iommu_mapping, kref); - - iommu_put_dma_cookie(mapping->domain); - iommu_domain_free(mapping->domain); - kfree(mapping); -} - -void arm_iommu_release_mapping(struct dma_iommu_mapping *mapping) -{ - if (mapping) - kref_put(&mapping->kref, release_iommu_mapping); -} -EXPORT_SYMBOL_GPL(arm_iommu_release_mapping); - -static int __arm_iommu_attach_device(struct device *dev, - struct dma_iommu_mapping *mapping) -{ - int err; - - err = iommu_attach_device(mapping->domain, dev); - if (err) - return err; - - kref_get(&mapping->kref); - to_dma_iommu_mapping(dev) = mapping; - - pr_debug("Attached IOMMU controller to %s device.\n", dev_name(dev)); - return 0; -} - -/** - * arm_iommu_attach_device - * @dev: valid struct device pointer - * @mapping: io address space mapping structure (returned from - * arm_iommu_create_mapping) - * - * Attaches specified io address space mapping to the provided device. - * This replaces the dma operations (dma_map_ops pointer) with the - * IOMMU aware version. - * - * More than one client might be attached to the same io address space - * mapping. - */ -int arm_iommu_attach_device(struct device *dev, - struct dma_iommu_mapping *mapping) -{ - int err; - - err = __arm_iommu_attach_device(dev, mapping); - if (err) - return err; - - set_dma_ops(dev, &iommu_dma_ops); - return 0; -} -EXPORT_SYMBOL_GPL(arm_iommu_attach_device); - -/** - * arm_iommu_detach_device - * @dev: valid struct device pointer - * - * Detaches the provided device from a previously attached map. - * This overwrites the dma_ops pointer with appropriate non-IOMMU ops. - */ -void arm_iommu_detach_device(struct device *dev) -{ - struct dma_iommu_mapping *mapping; - - mapping = to_dma_iommu_mapping(dev); - if (!mapping) { - dev_warn(dev, "Not attached\n"); - return; - } - - iommu_detach_device(mapping->domain, dev); - kref_put(&mapping->kref, release_iommu_mapping); - to_dma_iommu_mapping(dev) = NULL; - set_dma_ops(dev, arm_get_dma_map_ops(dev->archdata.dma_coherent)); - - pr_debug("Detached IOMMU controller from %s device.\n", dev_name(dev)); -} -EXPORT_SYMBOL_GPL(arm_iommu_detach_device); - -static bool arm_setup_iommu_dma_ops(struct device *dev, u64 dma_base, u64 size, - const struct iommu_ops *iommu) -{ - struct dma_iommu_mapping *mapping; - - if (!iommu) - return false; - - /* If a default domain exists, just let iommu-dma work normally */ - if (iommu_get_domain_for_dev(dev)) { - iommu_setup_dma_ops(dev, dma_base, size); - return true; - } - - /* Otherwise, use the workaround until the IOMMU driver is updated */ - mapping = arm_iommu_create_mapping(dev->bus, dma_base, size); - if (IS_ERR(mapping)) { - pr_warn("Failed to create %llu-byte IOMMU mapping for device %s\n", - size, dev_name(dev)); - return false; - } - - if (__arm_iommu_attach_device(dev, mapping)) { - pr_warn("Failed to attached device %s to IOMMU_mapping\n", - dev_name(dev)); - arm_iommu_release_mapping(mapping); - return false; - } - - set_dma_ops(dev, &iommu_dma_ops); - return true; -} - -static void arm_teardown_iommu_dma_ops(struct device *dev) -{ - struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev); - - if (!mapping) - return; - - arm_iommu_detach_device(dev); - arm_iommu_release_mapping(mapping); -} - -#else - -static bool arm_setup_iommu_dma_ops(struct device *dev, u64 dma_base, u64 size, - const struct iommu_ops *iommu) -{ - return false; -} - -static void arm_teardown_iommu_dma_ops(struct device *dev) { } - -#endif /* CONFIG_ARM_DMA_USE_IOMMU */ - void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size, const struct iommu_ops *iommu, bool coherent) { @@ -1286,7 +1090,8 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size, set_dma_ops(dev, arm_get_dma_map_ops(coherent)); - arm_setup_iommu_dma_ops(dev, dma_base, size, iommu); + if (iommu) + iommu_setup_dma_ops(dev, dma_base, size); #ifdef CONFIG_XEN if (xen_initial_domain()) @@ -1300,7 +1105,6 @@ void arch_teardown_dma_ops(struct device *dev) if (!dev->archdata.dma_ops_setup) return; - arm_teardown_iommu_dma_ops(dev); /* Let arch_setup_dma_ops() start again from scratch upon re-probe */ set_dma_ops(dev, NULL); } diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index ab157d155bf7..4959f5df21bd 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -25,19 +25,6 @@ #include #include -#ifdef CONFIG_ARM -#include -#endif -static struct iommu_domain *__iommu_get_dma_domain(struct device *dev) -{ -#ifdef CONFIG_ARM - struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev); - if (mapping) - return mapping->domain; -#endif - return iommu_get_dma_domain(dev); -} - struct iommu_dma_msi_page { struct list_head list; dma_addr_t iova; @@ -311,11 +298,8 @@ static void iommu_dma_flush_iotlb_all(struct iova_domain *iovad) * avoid rounding surprises. If necessary, we reserve the page at address 0 * to ensure it is an invalid IOVA. It is safe to reinitialise a domain, but * any change which could make prior IOVAs invalid will fail. - * - * XXX: Not formally exported, but needs to be referenced - * from arch/arm/mm/dma-mapping.c temporarily */ -int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, +static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, u64 size, struct device *dev) { struct iommu_dma_cookie *cookie = domain->iova_cookie; @@ -472,7 +456,7 @@ static void iommu_dma_free_iova(struct iommu_dma_cookie *cookie, static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, size_t size) { - struct iommu_domain *domain = __iommu_get_dma_domain(dev); + struct iommu_domain *domain = iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; struct iova_domain *iovad = &cookie->iovad; size_t iova_off = iova_offset(iovad, dma_addr); @@ -494,7 +478,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, size_t size, int prot, u64 dma_mask) { - struct iommu_domain *domain = __iommu_get_dma_domain(dev); + struct iommu_domain *domain = iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; struct iova_domain *iovad = &cookie->iovad; size_t iova_off = iova_offset(iovad, phys); @@ -598,7 +582,7 @@ static struct page **__iommu_dma_alloc_pages(struct device *dev, static void *iommu_dma_alloc_remap(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs) { - struct iommu_domain *domain = __iommu_get_dma_domain(dev); + struct iommu_domain *domain = iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; struct iova_domain *iovad = &cookie->iovad; bool coherent = dev_is_dma_coherent(dev); @@ -694,7 +678,7 @@ static void iommu_dma_sync_single_for_cpu(struct device *dev, if (dev_is_dma_coherent(dev)) return; - phys = iommu_iova_to_phys(__iommu_get_dma_domain(dev), dma_handle); + phys = iommu_iova_to_phys(iommu_get_dma_domain(dev), dma_handle); arch_sync_dma_for_cpu(phys, size, dir); } @@ -706,7 +690,7 @@ static void iommu_dma_sync_single_for_device(struct device *dev, if (dev_is_dma_coherent(dev)) return; - phys = iommu_iova_to_phys(__iommu_get_dma_domain(dev), dma_handle); + phys = iommu_iova_to_phys(iommu_get_dma_domain(dev), dma_handle); arch_sync_dma_for_device(phys, size, dir); } @@ -847,7 +831,7 @@ static void __invalidate_sg(struct scatterlist *sg, int nents) static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, int nents, enum dma_data_direction dir, unsigned long attrs) { - struct iommu_domain *domain = __iommu_get_dma_domain(dev); + struct iommu_domain *domain = iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; struct iova_domain *iovad = &cookie->iovad; struct scatterlist *s, *prev = NULL; @@ -1128,16 +1112,12 @@ static int iommu_dma_get_sgtable(struct device *dev, struct sg_table *sgt, static unsigned long iommu_dma_get_merge_boundary(struct device *dev) { - struct iommu_domain *domain = __iommu_get_dma_domain(dev); + struct iommu_domain *domain = iommu_get_dma_domain(dev); return (1UL << __ffs(domain->pgsize_bitmap)) - 1; } -/* - * XXX: Not formally exported, but needs to be referenced - * from arch/arm/mm/dma-mapping.c temporarily - */ -const struct dma_map_ops iommu_dma_ops = { +static const struct dma_map_ops iommu_dma_ops = { .alloc = iommu_dma_alloc, .free = iommu_dma_free, .mmap = iommu_dma_mmap,