From patchwork Wed Apr 1 23:15:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 228271 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE731C43331 for ; Wed, 1 Apr 2020 23:15:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9E19A2077D for ; Wed, 1 Apr 2020 23:15:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1585782938; bh=WGgZ62ijNh9fi+fcWG0IWjCbUlEAWxjkT5rD32fRC3w=; h=Date:From:To:Subject:List-ID:From; b=EDUeJ7KtHEx9c5fAS7GUjRABHRNcTgND9UaVosgw+G/Eg/lhjeCkeJHmMkDxzL2xd ApnCUFo2InOeDMxCsaVMTUePMZGr2THRFnij31xhV2AOHLQZ1tgHf/qqEY4ZBv0yn+ x8ou/T/jvaZMNSDr5PKLy/uzeJ2hcng7VuiXzqfc= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732732AbgDAXPi (ORCPT ); Wed, 1 Apr 2020 19:15:38 -0400 Received: from mail.kernel.org ([198.145.29.99]:58784 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732537AbgDAXPi (ORCPT ); Wed, 1 Apr 2020 19:15:38 -0400 Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 3816520719; Wed, 1 Apr 2020 23:15:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1585782935; bh=WGgZ62ijNh9fi+fcWG0IWjCbUlEAWxjkT5rD32fRC3w=; h=Date:From:To:Subject:From; b=VjstFfAFeFuoXDJlV7xEjlKL3I0dE1/w4PmSaYbn0mHAHeNZw7NPmT2LCk4ntMesF 5lil7cAbwn+vSrEUAlwB75Q3BCqSbdOoqKc+OpwttiwdnqBUAqb7EpPSAqyQfZYmdN T2qR+4K84fpZTZc5/rJi7fueE9BZmgpnfY3VuHFg= Date: Wed, 01 Apr 2020 16:15:34 -0700 From: akpm@linux-foundation.org To: akpm@linux-foundation.org, jgg@mellanox.com, jgg@ziepe.ca, longpeng2@huawei.com, mike.kravetz@oracle.com, mm-commits@vger.kernel.org, sean.j.christopherson@intel.com, stable@vger.kernel.org, willy@infradead.org Subject: [to-be-updated] mm-hugetlb-fix-a-addressing-exception-caused-by-huge_pte_offset.patch removed from -mm tree Message-ID: <20200401231534.2A7YCkw00%akpm@linux-foundation.org> User-Agent: s-nail v14.8.16 Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org The patch titled Subject: mm/hugetlb: fix a addressing exception caused by huge_pte_offset has been removed from the -mm tree. Its filename was mm-hugetlb-fix-a-addressing-exception-caused-by-huge_pte_offset.patch This patch was dropped because an updated version will be merged ------------------------------------------------------ From: Longpeng Subject: mm/hugetlb: fix a addressing exception caused by huge_pte_offset Our machine encountered a panic(addressing exception) after run for a long time and the calltrace is: RIP: 0010:[] [] hugetlb_fault+0x307/0xbe0 RSP: 0018:ffff9567fc27f808 EFLAGS: 00010286 RAX: e800c03ff1258d48 RBX: ffffd3bb003b69c0 RCX: e800c03ff1258d48 RDX: 17ff3fc00eda72b7 RSI: 00003ffffffff000 RDI: e800c03ff1258d48 RBP: ffff9567fc27f8c8 R08: e800c03ff1258d48 R09: 0000000000000080 R10: ffffaba0704c22a8 R11: 0000000000000001 R12: ffff95c87b4b60d8 R13: 00005fff00000000 R14: 0000000000000000 R15: ffff9567face8074 FS: 00007fe2d9ffb700(0000) GS:ffff956900e40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffd3bb003b69c0 CR3: 000000be67374000 CR4: 00000000003627e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: [] ? unlock_page+0x2b/0x30 [] ? hugetlb_fault+0x222/0xbe0 [] follow_hugetlb_page+0x175/0x540 [] ? cpumask_next_and+0x35/0x50 [] __get_user_pages+0x2a0/0x7e0 [] __get_user_pages_unlocked+0x15d/0x210 [] __gfn_to_pfn_memslot+0x3c5/0x460 [kvm] [] try_async_pf+0x6e/0x2a0 [kvm] [] tdp_page_fault+0x151/0x2d0 [kvm] ... [] kvm_arch_vcpu_ioctl_run+0x330/0x490 [kvm] [] kvm_vcpu_ioctl+0x309/0x6d0 [kvm] [] ? dequeue_signal+0x32/0x180 [] ? do_sigtimedwait+0xcd/0x230 [] do_vfs_ioctl+0x3f0/0x540 [] SyS_ioctl+0xa1/0xc0 [] system_call_fastpath+0x22/0x27 For 1G hugepages, huge_pte_offset() wants to return NULL or pudp, but it may return a wrong 'pmdp' if there is a race. Please look at the following code snippet: ... pud = pud_offset(p4d, addr); if (sz != PUD_SIZE && pud_none(*pud)) return NULL; /* hugepage or swap? */ if (pud_huge(*pud) || !pud_present(*pud)) return (pte_t *)pud; pmd = pmd_offset(pud, addr); if (sz != PMD_SIZE && pmd_none(*pmd)) return NULL; /* hugepage or swap? */ if (pmd_huge(*pmd) || !pmd_present(*pmd)) return (pte_t *)pmd; ... The following sequence would trigger this bug: 1. CPU0: sz = PUD_SIZE and *pud = 0 , continue 1. CPU0: "pud_huge(*pud)" is false 2. CPU1: calling hugetlb_no_page and set *pud to xxxx8e7(PRESENT) 3. CPU0: "!pud_present(*pud)" is false, continue 4. CPU0: pmd = pmd_offset(pud, addr) and maybe return a wrong pmdp However, we want CPU0 to return NULL or pudp in this case. Also, according to the section 'COMPILER BARRIER' of memory-barriers.txt: ''' (*) The compiler is within its rights to reorder loads and stores to the same variable, and in some cases, the CPU is within its rights to reorder loads to the same variable. This means that the following code: a[0] = x; a[1] = x; Might result in an older value of x stored in a[1] than in a[0]. ''' there're several other data races in huge_pte_offset, for example: ''' p4d = p4d_offset(pgd, addr) if (!p4d_present(*p4d)) return NULL; pud = pud_offset(p4d, addr) <-- will be unwinded as: pud = (pud_t *)p4d_page_vaddr(*p4d) + pud_index(address); ''' which is free for the compiler/CPU to execute as: ''' p4d = p4d_offset(pgd, addr) p4d_for_vaddr = *p4d; if (!p4d_present(*p4d)) return NULL; pud = (pud_t *)p4d_page_vaddr(p4d_for_vaddr) + pud_index(address); ''' so in the case where *p4d goes from '!present' to 'present': p4d_present(*p4d) == true and p4d_for_vaddr == none, meaning the p4d_page_vaddr() will crash. For these reasons, we must make sure there is exactly one dereference of p4d, pud and pmd. Link: http://lkml.kernel.org/r/20200327235748.2048-1-longpeng2@huawei.com Signed-off-by: Longpeng Suggested-by: Jason Gunthorpe Reviewed-by: Jason Gunthorpe Cc: Mike Kravetz Cc: Jason Gunthorpe Cc: Matthew Wilcox Cc: Sean Christopherson Cc: Signed-off-by: Andrew Morton --- mm/hugetlb.c | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) --- a/mm/hugetlb.c~mm-hugetlb-fix-a-addressing-exception-caused-by-huge_pte_offset +++ a/mm/hugetlb.c @@ -4909,29 +4909,33 @@ pte_t *huge_pte_offset(struct mm_struct unsigned long addr, unsigned long sz) { pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; + p4d_t *p4d, p4d_entry; + pud_t *pud, pud_entry; + pmd_t *pmd, pmd_entry; pgd = pgd_offset(mm, addr); if (!pgd_present(*pgd)) return NULL; + p4d = p4d_offset(pgd, addr); - if (!p4d_present(*p4d)) + p4d_entry = READ_ONCE(*p4d); + if (!p4d_present(p4d_entry)) return NULL; - pud = pud_offset(p4d, addr); - if (sz != PUD_SIZE && pud_none(*pud)) + pud = pud_offset(&p4d_entry, addr); + pud_entry = READ_ONCE(*pud); + if (sz != PUD_SIZE && pud_none(pud_entry)) return NULL; /* hugepage or swap? */ - if (pud_huge(*pud) || !pud_present(*pud)) + if (pud_huge(pud_entry) || !pud_present(pud_entry)) return (pte_t *)pud; - pmd = pmd_offset(pud, addr); - if (sz != PMD_SIZE && pmd_none(*pmd)) + pmd = pmd_offset(&pud_entry, addr); + pmd_entry = READ_ONCE(*pmd); + if (sz != PMD_SIZE && pmd_none(pmd_entry)) return NULL; /* hugepage or swap? */ - if (pmd_huge(*pmd) || !pmd_present(*pmd)) + if (pmd_huge(pmd_entry) || !pmd_present(pmd_entry)) return (pte_t *)pmd; return NULL;