From patchwork Fri Aug 1 14:56:23 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefano Stabellini X-Patchwork-Id: 34711 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-ie0-f198.google.com (mail-ie0-f198.google.com [209.85.223.198]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id B8F12202A1 for ; Fri, 1 Aug 2014 14:59:23 +0000 (UTC) Received: by mail-ie0-f198.google.com with SMTP id rl12sf21245950iec.9 for ; Fri, 01 Aug 2014 07:59:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:subject:date:message-id :in-reply-to:references:mime-version:cc:precedence:list-id :list-unsubscribe:list-archive:list-post:list-help:list-subscribe :sender:errors-to:x-original-sender :x-original-authentication-results:mailing-list:content-type :content-transfer-encoding; bh=mrURCuSgwAKL68VDPAHsISwBRH5NXB872wyHbHODneU=; b=PAn0BLuXZ5K91Vsur6rOLnQL6bOx52M7vlgLEk9ojQ5N4QSOMlY/HaPIODVaaJ/u7T 5eFXtL3JWZngyr0j3wfZgO+WDGVIty6KvZZYL3zgyZvjjIzpUzA36Abg5KHFn23FWxYa 2FWX4ILxQKFEnDKbtzatwqm6LvZNwxxjevhB28iqT/NrmtvO6C4cklRmzk9ssk+KmuLn a+PJ5Wr6l832BtnT9JxdMq6tv2xym1m6qgx7sBrvjwHHdN+z04tAG05DJfUD68ct9ahB 0Ltyj5Od2ym7f7eHR1sh2Y/YCM5oP1U3p4lkhQU/T5nf1aiQfkyEugF3tJXBbrIdQyZG sJTw== X-Gm-Message-State: ALoCoQnwskqUUj4j8Q6+9dipWcejaOq9SzijSNmbhdx9wGhOmTSKO2WScDf8jLB7gHuPNtYEvuqt X-Received: by 10.182.125.37 with SMTP id mn5mr2558239obb.49.1406905163261; Fri, 01 Aug 2014 07:59:23 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.38.51 with SMTP id s48ls1428324qgs.64.gmail; Fri, 01 Aug 2014 07:59:23 -0700 (PDT) X-Received: by 10.220.199.193 with SMTP id et1mr3292761vcb.58.1406905163095; Fri, 01 Aug 2014 07:59:23 -0700 (PDT) Received: from mail-vc0-f171.google.com (mail-vc0-f171.google.com [209.85.220.171]) by mx.google.com with ESMTPS id jc2si3427034vec.2.2014.08.01.07.59.23 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 01 Aug 2014 07:59:23 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.171 as permitted sender) client-ip=209.85.220.171; Received: by mail-vc0-f171.google.com with SMTP id hq11so6832246vcb.2 for ; Fri, 01 Aug 2014 07:59:23 -0700 (PDT) X-Received: by 10.220.15.8 with SMTP id i8mr6861440vca.45.1406905163013; Fri, 01 Aug 2014 07:59:23 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.221.37.5 with SMTP id tc5csp117450vcb; Fri, 1 Aug 2014 07:59:22 -0700 (PDT) X-Received: by 10.67.22.65 with SMTP id hq1mr6753054pad.137.1406905161875; Fri, 01 Aug 2014 07:59:21 -0700 (PDT) Received: from bombadil.infradead.org (bombadil.infradead.org. [2001:1868:205::9]) by mx.google.com with ESMTPS id i5si5034096pdj.428.2014.08.01.07.59.21 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 01 Aug 2014 07:59:21 -0700 (PDT) Received-SPF: none (google.com: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org does not designate permitted sender hosts) client-ip=2001:1868:205::9; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1XDEHo-000056-8f; Fri, 01 Aug 2014 14:58:24 +0000 Received: from smtp.citrix.com ([66.165.176.89]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1XDEHk-0008FJ-I5 for linux-arm-kernel@lists.infradead.org; Fri, 01 Aug 2014 14:58:21 +0000 X-IronPort-AV: E=Sophos;i="5.01,780,1400025600"; d="scan'208";a="157911336" Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.3.181.6; Fri, 1 Aug 2014 10:57:37 -0400 Received: from kaball.uk.xensource.com ([10.80.2.59]) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1XDEGy-00053m-9P; Fri, 01 Aug 2014 15:57:32 +0100 From: Stefano Stabellini To: Subject: [PATCH v3 2/3] xen/arm: reimplement xen_dma_unmap_page & friends Date: Fri, 1 Aug 2014 15:56:23 +0100 Message-ID: <1406904984-16068-2-git-send-email-stefano.stabellini@eu.citrix.com> X-Mailer: git-send-email 1.7.9.5 In-Reply-To: References: MIME-Version: 1.0 X-DLP: MIA1 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20140801_075820_837574_CD5D02B6 X-CRM114-Status: GOOD ( 19.39 ) X-Spam-Score: -5.7 (-----) X-Spam-Report: SpamAssassin version 3.4.0 on bombadil.infradead.org summary: Content analysis details: (-5.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -5.0 RCVD_IN_DNSWL_HI RBL: Sender listed at http://www.dnswl.org/, high trust [66.165.176.89 listed in list.dnswl.org] -0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3) [66.165.176.89 listed in wl.mailspike.net] -0.7 RP_MATCHES_RCVD Envelope sender domain matches handover relay domain -0.0 SPF_PASS SPF: sender matches SPF record -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 RCVD_IN_MSPIKE_WL Mailspike good senders Cc: Ian.Campbell@citrix.com, Stefano Stabellini , julien.grall@citrix.com, david.vrabel@citrix.com, v1ne2go@gmail.com, linux-arm-kernel@lists.infradead.org X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: stefano.stabellini@eu.citrix.com X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.171 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 xen_dma_unmap_page, xen_dma_sync_single_for_cpu and xen_dma_sync_single_for_device are currently implemented by calling into the corresponding generic ARM implementation of these functions. In order to do this, firstly the dma_addr_t handle, that on Xen is a machine address, needs to be translated into a physical address. The operation is expensive and inaccurate, given that a single machine address can correspond to multiple physical addresses in one domain, because the same page can be granted multiple times by the frontend. To avoid this problem, we introduce a Xen specific implementation of xen_dma_unmap_page, xen_dma_sync_single_for_cpu and xen_dma_sync_single_for_device, that can operate on machine addresses directly. The new implementation relies on the fact that the hypervisor creates a second p2m mapping of any grant pages at physical address == machine address of the page for dom0. Therefore we can access memory at physical address == dma_addr_r handle and perform the cache flushing there. Some cache maintenance operations require a virtual address. Instead of using ioremap_cache, that is not safe in interrupt context, we allocate a per-cpu PAGE_KERNEL scratch page and we manually update the pte for it. arm64 doesn't need cache maintenance operations on unmap for now. Signed-off-by: Stefano Stabellini Tested-by: Denis Schneider --- Changes in v3: - rename XENFEAT_grant_map_11 to XENFEAT_grant_map_identity. Changes in v2: - check for XENFEAT_grant_map_11; - remeber the ptep corresponding to scratch pages so that we don't need to calculate it again every time; - do not acutally unmap the page on xen_mm32_unmap; - properly account preempt_enable/disable. --- arch/arm/include/asm/xen/page-coherent.h | 25 ++-- arch/arm/xen/Makefile | 2 +- arch/arm/xen/mm32.c | 202 ++++++++++++++++++++++++++++++ 3 files changed, 210 insertions(+), 19 deletions(-) create mode 100644 arch/arm/xen/mm32.c diff --git a/arch/arm/include/asm/xen/page-coherent.h b/arch/arm/include/asm/xen/page-coherent.h index 1109017..e8275ea 100644 --- a/arch/arm/include/asm/xen/page-coherent.h +++ b/arch/arm/include/asm/xen/page-coherent.h @@ -26,25 +26,14 @@ static inline void xen_dma_map_page(struct device *hwdev, struct page *page, __generic_dma_ops(hwdev)->map_page(hwdev, page, offset, size, dir, attrs); } -static inline void xen_dma_unmap_page(struct device *hwdev, dma_addr_t handle, +void xen_dma_unmap_page(struct device *hwdev, dma_addr_t handle, size_t size, enum dma_data_direction dir, - struct dma_attrs *attrs) -{ - if (__generic_dma_ops(hwdev)->unmap_page) - __generic_dma_ops(hwdev)->unmap_page(hwdev, handle, size, dir, attrs); -} + struct dma_attrs *attrs); -static inline void xen_dma_sync_single_for_cpu(struct device *hwdev, - dma_addr_t handle, size_t size, enum dma_data_direction dir) -{ - if (__generic_dma_ops(hwdev)->sync_single_for_cpu) - __generic_dma_ops(hwdev)->sync_single_for_cpu(hwdev, handle, size, dir); -} +void xen_dma_sync_single_for_cpu(struct device *hwdev, + dma_addr_t handle, size_t size, enum dma_data_direction dir); + +void xen_dma_sync_single_for_device(struct device *hwdev, + dma_addr_t handle, size_t size, enum dma_data_direction dir); -static inline void xen_dma_sync_single_for_device(struct device *hwdev, - dma_addr_t handle, size_t size, enum dma_data_direction dir) -{ - if (__generic_dma_ops(hwdev)->sync_single_for_device) - __generic_dma_ops(hwdev)->sync_single_for_device(hwdev, handle, size, dir); -} #endif /* _ASM_ARM_XEN_PAGE_COHERENT_H */ diff --git a/arch/arm/xen/Makefile b/arch/arm/xen/Makefile index 1296952..1f85bfe 100644 --- a/arch/arm/xen/Makefile +++ b/arch/arm/xen/Makefile @@ -1 +1 @@ -obj-y := enlighten.o hypercall.o grant-table.o p2m.o mm.o +obj-y := enlighten.o hypercall.o grant-table.o p2m.o mm.o mm32.o diff --git a/arch/arm/xen/mm32.c b/arch/arm/xen/mm32.c new file mode 100644 index 0000000..534f362 --- /dev/null +++ b/arch/arm/xen/mm32.c @@ -0,0 +1,202 @@ +#include +#include +#include +#include + +#include + +static DEFINE_PER_CPU(unsigned long, xen_mm32_scratch_virt); +static DEFINE_PER_CPU(pte_t *, xen_mm32_scratch_ptep); + +static int alloc_xen_mm32_scratch_page(int cpu) +{ + struct page *page; + unsigned long virt; + pmd_t *pmdp; + pte_t *ptep; + + if (per_cpu(xen_mm32_scratch_ptep, cpu) != NULL) + return 0; + + page = alloc_page(GFP_KERNEL); + if (page == NULL) { + pr_warn("Failed to allocate xen_mm32_scratch_page for cpu %d\n", cpu); + return -ENOMEM; + } + + virt = (unsigned long)__va(page_to_phys(page)); + pmdp = pmd_offset(pud_offset(pgd_offset_k(virt), virt), virt); + ptep = pte_offset_kernel(pmdp, virt); + + per_cpu(xen_mm32_scratch_virt, cpu) = virt; + per_cpu(xen_mm32_scratch_ptep, cpu) = ptep; + + return 0; +} + +static int xen_mm32_cpu_notify(struct notifier_block *self, + unsigned long action, void *hcpu) +{ + int cpu = (long)hcpu; + switch (action) { + case CPU_UP_PREPARE: + if (alloc_xen_mm32_scratch_page(cpu)) + return NOTIFY_BAD; + break; + default: + break; + } + return NOTIFY_OK; +} + +static struct notifier_block xen_mm32_cpu_notifier = { + .notifier_call = xen_mm32_cpu_notify, +}; + +static void* xen_mm32_remap_page(dma_addr_t handle) +{ + unsigned long virt = get_cpu_var(xen_mm32_scratch_virt); + pte_t *ptep = __get_cpu_var(xen_mm32_scratch_ptep); + + *ptep = pfn_pte(handle >> PAGE_SHIFT, PAGE_KERNEL); + local_flush_tlb_kernel_page(virt); + + return (void*)virt; +} + +static void xen_mm32_unmap(void *vaddr) +{ + put_cpu_var(xen_mm32_scratch_virt); +} + + +/* functions called by SWIOTLB */ + +static void dma_cache_maint(dma_addr_t handle, unsigned long offset, + size_t size, enum dma_data_direction dir, + void (*op)(const void *, size_t, int)) +{ + unsigned long pfn; + size_t left = size; + + pfn = (handle >> PAGE_SHIFT) + offset / PAGE_SIZE; + offset %= PAGE_SIZE; + + do { + size_t len = left; + void *vaddr; + + if (!pfn_valid(pfn)) + { + /* Cannot map the page, we don't know its physical address. + * Return and hope for the best */ + if (!xen_feature(XENFEAT_grant_map_identity)) + return; + vaddr = xen_mm32_remap_page(handle) + offset; + op(vaddr, len, dir); + xen_mm32_unmap(vaddr - offset); + } else { + struct page *page = pfn_to_page(pfn); + + if (PageHighMem(page)) { + if (len + offset > PAGE_SIZE) + len = PAGE_SIZE - offset; + + if (cache_is_vipt_nonaliasing()) { + vaddr = kmap_atomic(page); + op(vaddr + offset, len, dir); + kunmap_atomic(vaddr); + } else { + vaddr = kmap_high_get(page); + if (vaddr) { + op(vaddr + offset, len, dir); + kunmap_high(page); + } + } + } else { + vaddr = page_address(page) + offset; + op(vaddr, len, dir); + } + } + + offset = 0; + pfn++; + left -= len; + } while (left); +} + +static void __xen_dma_page_dev_to_cpu(struct device *hwdev, dma_addr_t handle, + size_t size, enum dma_data_direction dir) +{ + /* Cannot use __dma_page_dev_to_cpu because we don't have a + * struct page for handle */ + + if (dir == DMA_TO_DEVICE) + outer_inv_range(handle, handle + size); + + dma_cache_maint(handle & PAGE_MASK, handle & ~PAGE_MASK, size, dir, dmac_unmap_area); +} + +static void __xen_dma_page_cpu_to_dev(struct device *hwdev, dma_addr_t handle, + size_t size, enum dma_data_direction dir) +{ + + dma_cache_maint(handle & PAGE_MASK, handle & ~PAGE_MASK, size, dir, dmac_map_area); + + if (dir == DMA_FROM_DEVICE) { + outer_inv_range(handle, handle + size); + } else { + outer_clean_range(handle, handle + size); + } +} + +void xen_dma_unmap_page(struct device *hwdev, dma_addr_t handle, + size_t size, enum dma_data_direction dir, + struct dma_attrs *attrs) + +{ + if (!__generic_dma_ops(hwdev)->unmap_page) + return; + if (dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs)) + return; + + __xen_dma_page_dev_to_cpu(hwdev, handle, size, dir); +} + +void xen_dma_sync_single_for_cpu(struct device *hwdev, + dma_addr_t handle, size_t size, enum dma_data_direction dir) +{ + if (!__generic_dma_ops(hwdev)->sync_single_for_cpu) + return; + __xen_dma_page_dev_to_cpu(hwdev, handle, size, dir); +} + +void xen_dma_sync_single_for_device(struct device *hwdev, + dma_addr_t handle, size_t size, enum dma_data_direction dir) +{ + if (!__generic_dma_ops(hwdev)->sync_single_for_device) + return; + __xen_dma_page_cpu_to_dev(hwdev, handle, size, dir); +} + +int __init xen_mm32_init(void) +{ + int cpu; + + if (!xen_initial_domain()) + return 0; + + register_cpu_notifier(&xen_mm32_cpu_notifier); + get_online_cpus(); + for_each_online_cpu(cpu) { + if (alloc_xen_mm32_scratch_page(cpu)) { + put_online_cpus(); + unregister_cpu_notifier(&xen_mm32_cpu_notifier); + return -ENOMEM; + } + } + put_online_cpus(); + + return 0; +} +arch_initcall(xen_mm32_init);