From patchwork Tue Sep 24 09:34:53 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Capper X-Patchwork-Id: 20551 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-qa0-f72.google.com (mail-qa0-f72.google.com [209.85.216.72]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id DF3D325E03 for ; Tue, 24 Sep 2013 09:35:04 +0000 (UTC) Received: by mail-qa0-f72.google.com with SMTP id j7sf3512409qaq.7 for ; Tue, 24 Sep 2013 02:35:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:delivered-to:from:to:cc:subject :date:message-id:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-unsubscribe; bh=+2lFE5xVe/vsmm4l2NbtVgHedn84cGtcZ0qgaZ5gqPU=; b=gzMp18IHR/GcvmhH2ssJtAiIpYwEN8fB6rWaw69S4QeqlC6SMbTcseqntkZKdt5DWb cZzT0dJNsogxPTS+OhUj+6Ib9BNTNki5n2lH351yVBW4MpeTU3YyzhQBW4PUfny0+POs p85693Fce6oJtfhjd1Pq7MfPu4uQBAqRrY+h74J6v/8KzOFOI8KVYrGYmZE758nuM4LB 4c3vI4AcLu29rVzmW7CkbRbHDZsTBsdfY84oMZ/bK7xTyH1RYzahCydEL3bqgEHt8pND bo4do1bZMhO0aFgyBpAykBQCfMMYLIaHdFS3cKNGYL5g19wiSAZO29vKHf50qQ+auCtP lpFQ== X-Gm-Message-State: ALoCoQkg6Qh9VITN71FnZGSv/C0IP8csh7Z4pApntJa+DLMXrRzlzrwMjefCe5+6JiIAF+khHHyp X-Received: by 10.224.97.1 with SMTP id j1mr6484382qan.6.1380015304262; Tue, 24 Sep 2013 02:35:04 -0700 (PDT) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.49.99.106 with SMTP id ep10ls46501qeb.59.gmail; Tue, 24 Sep 2013 02:35:04 -0700 (PDT) X-Received: by 10.52.187.5 with SMTP id fo5mr2962876vdc.47.1380015304153; Tue, 24 Sep 2013 02:35:04 -0700 (PDT) Received: from mail-vb0-f52.google.com (mail-vb0-f52.google.com [209.85.212.52]) by mx.google.com with ESMTPS id zw10si8167034vdb.83.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 24 Sep 2013 02:35:03 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.212.52 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.212.52; Received: by mail-vb0-f52.google.com with SMTP id f12so3211236vbg.11 for ; Tue, 24 Sep 2013 02:35:03 -0700 (PDT) X-Received: by 10.220.2.138 with SMTP id 10mr125907vcj.77.1380015303769; Tue, 24 Sep 2013 02:35:03 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patches@linaro.org Received: by 10.220.174.196 with SMTP id u4csp231845vcz; Tue, 24 Sep 2013 02:35:03 -0700 (PDT) X-Received: by 10.180.149.179 with SMTP id ub19mr16815959wib.43.1380015302688; Tue, 24 Sep 2013 02:35:02 -0700 (PDT) Received: from mail-wi0-f170.google.com (mail-wi0-f170.google.com [209.85.212.170]) by mx.google.com with ESMTPS id wg1si12273592wjb.115.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 24 Sep 2013 02:35:02 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.212.170 is neither permitted nor denied by best guess record for domain of steve.capper@linaro.org) client-ip=209.85.212.170; Received: by mail-wi0-f170.google.com with SMTP id cb5so3423364wib.5 for ; Tue, 24 Sep 2013 02:35:02 -0700 (PDT) X-Received: by 10.180.185.101 with SMTP id fb5mr17295191wic.11.1380015302161; Tue, 24 Sep 2013 02:35:02 -0700 (PDT) Received: from marmot.wormnet.eu (marmot.wormnet.eu. [188.246.204.87]) by mx.google.com with ESMTPSA id l9sm5621191wif.10.1969.12.31.16.00.00 (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 24 Sep 2013 02:35:01 -0700 (PDT) From: Steve Capper To: linux-arm-kernel@lists.infradead.org Cc: linux@arm.linux.org.uk, nico@linaro.org, linaro-kernel@lists.linaro.org, patches@linaro.org, Steve Capper Subject: [PATCH] ARM: mm: make UACCESS_WITH_MEMCPY huge page aware Date: Tue, 24 Sep 2013 10:34:53 +0100 Message-Id: <1380015293-9590-1-git-send-email-steve.capper@linaro.org> X-Mailer: git-send-email 1.7.10.4 X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: steve.capper@linaro.org X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.212.52 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Precedence: list Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org List-ID: X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , The memory pinning code in uaccess_with_memcpy.c does not check for HugeTLB or THP pmds, and will enter an infinite loop should a __copy_to_user or __clear_user occur against a huge page. This patch adds detection code for huge pages to pin_page_for_write. As this code can be executed in a fast path it refers to the actual pmds rather than the vma. If a HugeTLB or THP is found (they have the same pmd representation on ARM), the page table spinlock is taken to prevent modification whilst the page is pinned. On ARM, huge pages are only represented as pmds, thus no huge pud checks are performed. (For huge puds one would lock the page table in a similar manner as in the pmd case). Two helper functions are introduced; pmd_thp_or_huge will check whether or not a page is huge or transparent huge (which have the same pmd layout on ARM), and pmd_hugewillfault will detect whether or not a page fault will occur on write to the page. Reviewed-by: nicolas Pitre --- Changes since second RFC: * After some more thought about the potential impact on the overall latency, I have decided not to widen the page mask for huge pages. Changes since first RFC: * The page mask is widened for hugepages to reduce the number of potential locks/unlocks. (A knobbled /dev/zero with its latency reduction chunks removed shows a 2x data rate boost with hugepages backing: dd if=/dev/zero of=/dev/null bs=10M count=1024 ) --- Signed-off-by: Steve Capper --- arch/arm/include/asm/pgtable-3level.h | 3 +++ arch/arm/lib/uaccess_with_memcpy.c | 41 ++++++++++++++++++++++++++++++++--- 2 files changed, 41 insertions(+), 3 deletions(-) diff --git a/arch/arm/include/asm/pgtable-3level.h b/arch/arm/include/asm/pgtable-3level.h index 5689c18..39c54cf 100644 --- a/arch/arm/include/asm/pgtable-3level.h +++ b/arch/arm/include/asm/pgtable-3level.h @@ -206,6 +206,9 @@ static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr) #define __HAVE_ARCH_PMD_WRITE #define pmd_write(pmd) (!(pmd_val(pmd) & PMD_SECT_RDONLY)) +#define pmd_hugewillfault(pmd) (!pmd_young(pmd) || !pmd_write(pmd)) +#define pmd_thp_or_huge(pmd) (pmd_huge(pmd) || pmd_trans_huge(pmd)) + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define pmd_trans_huge(pmd) (pmd_val(pmd) && !(pmd_val(pmd) & PMD_TABLE_BIT)) #define pmd_trans_splitting(pmd) (pmd_val(pmd) & PMD_SECT_SPLITTING) diff --git a/arch/arm/lib/uaccess_with_memcpy.c b/arch/arm/lib/uaccess_with_memcpy.c index 025f742..3e58d71 100644 --- a/arch/arm/lib/uaccess_with_memcpy.c +++ b/arch/arm/lib/uaccess_with_memcpy.c @@ -18,6 +18,7 @@ #include /* for in_atomic() */ #include #include +#include #include #include @@ -40,7 +41,35 @@ pin_page_for_write(const void __user *_addr, pte_t **ptep, spinlock_t **ptlp) return 0; pmd = pmd_offset(pud, addr); - if (unlikely(pmd_none(*pmd) || pmd_bad(*pmd))) + if (unlikely(pmd_none(*pmd))) + return 0; + + /* + * A pmd can be bad if it refers to a HugeTLB or THP page. + * + * Both THP and HugeTLB pages have the same pmd layout + * and should not be manipulated by the pte functions. + * + * Lock the page table for the destination and check + * to see that it's still huge and whether or not we will + * need to fault on write, or if we have a splitting THP. + */ + if (unlikely(pmd_thp_or_huge(*pmd))) { + ptl = ¤t->mm->page_table_lock; + spin_lock(ptl); + if (unlikely(!pmd_thp_or_huge(*pmd) + || pmd_hugewillfault(*pmd) + || pmd_trans_splitting(*pmd))) { + spin_unlock(ptl); + return 0; + } + + *ptep = NULL; + *ptlp = ptl; + return 1; + } + + if (unlikely(pmd_bad(*pmd))) return 0; pte = pte_offset_map_lock(current->mm, pmd, addr, &ptl); @@ -94,7 +123,10 @@ __copy_to_user_memcpy(void __user *to, const void *from, unsigned long n) from += tocopy; n -= tocopy; - pte_unmap_unlock(pte, ptl); + if (pte) + pte_unmap_unlock(pte, ptl); + else + spin_unlock(ptl); } if (!atomic) up_read(¤t->mm->mmap_sem); @@ -147,7 +179,10 @@ __clear_user_memset(void __user *addr, unsigned long n) addr += tocopy; n -= tocopy; - pte_unmap_unlock(pte, ptl); + if (pte) + pte_unmap_unlock(pte, ptl); + else + spin_unlock(ptl); } up_read(¤t->mm->mmap_sem);