From patchwork Wed Dec 6 12:35:20 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120822 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp6999509qgn; Wed, 6 Dec 2017 04:36:16 -0800 (PST) X-Google-Smtp-Source: AGs4zMaoKv6MB6eVjZxxmoxkHTL455Y6pzJeEcNn4XeFPQpTBuLwXfz5J/6JcL7X91DkKAM6xNgL X-Received: by 10.98.138.149 with SMTP id o21mr2639272pfk.225.1512563776189; Wed, 06 Dec 2017 04:36:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512563776; cv=none; d=google.com; s=arc-20160816; b=v+mQVZ16b9UDNRv0hKSqFBSPPdo7a90TUNMpn1LR1Eyl2DE+U72qeuGlO23j17Z/pe nmrGCNzhGeXjzB6WHuU/eWHAvQEky4LC88mm4wlzxdtFI9AZHw7jH25Q1Cjp8J9b/f3C Ej9RTt6R+Gguhr12AXPe7k7NhRjSEcO2+wnEjxOdlYynPDWQF+LT3mp7gc9XMlBb8VUs COBsB28+662Wj4Ry4KAx+DL5ZAT58MUxN4AQmXaH+kSnF6pxOMSoVblYbwISlBA8oQ1t uqmCTGjmOTKgzThozf0o8uiajxxmYO1vRgzBNzRjdFsb8P83ylt2wZDRgwEz6ZKMVitX L0Tw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=LZ5PykpbdgBjtU3S+WGyoAMot1+7KgJI6DgyEcz3VRo=; b=p+xKe3YUlw9/+mXmNbpB0WY/j1deYzshWyFTgwouP9rRNAhOLvDKV4hxhHC2xsmUin 7jSP4acMv+4sFpeBMgu2yy6JhaNl+wMHRjAyOyqtIjDAa3kc02gIqFqdKZzo6/duiVQw iUe/vprn7V98zGXmGGjLyFGZEtO/3U3LQsXxwNNDy9M126Ms8JYkz8su4AtnYZA94aGd HQc77Mkpns5Q2rsK2V6I3K6RMIs1TSnsqXhsLvhsuT9py7yppVZu9N7tqXYFaAWVcnJ7 j5yfl9YXKMv/rk+m+G4VT/A36u9a6XfKydzmN2idWWgN58BmFKSmK1Q6wESkZ/j9RQnM 2ewg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id k64si2005767pfj.347.2017.12.06.04.36.15; Wed, 06 Dec 2017 04:36:16 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752219AbdLFMgO (ORCPT + 28 others); Wed, 6 Dec 2017 07:36:14 -0500 Received: from foss.arm.com ([217.140.101.70]:34598 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751550AbdLFMgK (ORCPT ); Wed, 6 Dec 2017 07:36:10 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6D0D215AD; Wed, 6 Dec 2017 04:36:10 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 3DF8D3F53E; Wed, 6 Dec 2017 04:36:10 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id D2F2B1AE3546; Wed, 6 Dec 2017 12:36:15 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 01/20] arm64: mm: Use non-global mappings for kernel space Date: Wed, 6 Dec 2017 12:35:20 +0000 Message-Id: <1512563739-25239-2-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In preparation for unmapping the kernel whilst running in userspace, make the kernel mappings non-global so we can avoid expensive TLB invalidation on kernel exit to userspace. Reviewed-by: Mark Rutland Signed-off-by: Will Deacon --- arch/arm64/include/asm/kernel-pgtable.h | 12 ++++++++++-- arch/arm64/include/asm/pgtable-prot.h | 21 +++++++++++++++------ 2 files changed, 25 insertions(+), 8 deletions(-) -- 2.1.4 diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 7803343e5881..77a27af01371 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -78,8 +78,16 @@ /* * Initial memory map attributes. */ -#define SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) -#define SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) +#define _SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) +#define _SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) + +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 +#define SWAPPER_PTE_FLAGS (_SWAPPER_PTE_FLAGS | PTE_NG) +#define SWAPPER_PMD_FLAGS (_SWAPPER_PMD_FLAGS | PMD_SECT_NG) +#else +#define SWAPPER_PTE_FLAGS _SWAPPER_PTE_FLAGS +#define SWAPPER_PMD_FLAGS _SWAPPER_PMD_FLAGS +#endif #if ARM64_SWAPPER_USES_SECTION_MAPS #define SWAPPER_MM_MMUFLAGS (PMD_ATTRINDX(MT_NORMAL) | SWAPPER_PMD_FLAGS) diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h index 0a5635fb0ef9..22a926825e3f 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -34,8 +34,16 @@ #include -#define PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) -#define PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) +#define _PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) +#define _PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) + +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 +#define PROT_DEFAULT (_PROT_DEFAULT | PTE_NG) +#define PROT_SECT_DEFAULT (_PROT_SECT_DEFAULT | PMD_SECT_NG) +#else +#define PROT_DEFAULT _PROT_DEFAULT +#define PROT_SECT_DEFAULT _PROT_SECT_DEFAULT +#endif /* CONFIG_UNMAP_KERNEL_AT_EL0 */ #define PROT_DEVICE_nGnRnE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRnE)) #define PROT_DEVICE_nGnRE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRE)) @@ -48,6 +56,7 @@ #define PROT_SECT_NORMAL_EXEC (PROT_SECT_DEFAULT | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL)) #define _PAGE_DEFAULT (PROT_DEFAULT | PTE_ATTRINDX(MT_NORMAL)) +#define _HYP_PAGE_DEFAULT (_PAGE_DEFAULT & ~PTE_NG) #define PAGE_KERNEL __pgprot(_PAGE_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE) #define PAGE_KERNEL_RO __pgprot(_PAGE_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_RDONLY) @@ -55,15 +64,15 @@ #define PAGE_KERNEL_EXEC __pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_WRITE) #define PAGE_KERNEL_EXEC_CONT __pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_WRITE | PTE_CONT) -#define PAGE_HYP __pgprot(_PAGE_DEFAULT | PTE_HYP | PTE_HYP_XN) -#define PAGE_HYP_EXEC __pgprot(_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY) -#define PAGE_HYP_RO __pgprot(_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY | PTE_HYP_XN) +#define PAGE_HYP __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_HYP_XN) +#define PAGE_HYP_EXEC __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY) +#define PAGE_HYP_RO __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY | PTE_HYP_XN) #define PAGE_HYP_DEVICE __pgprot(PROT_DEVICE_nGnRE | PTE_HYP) #define PAGE_S2 __pgprot(PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_NORMAL) | PTE_S2_RDONLY) #define PAGE_S2_DEVICE __pgprot(PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_DEVICE_nGnRE) | PTE_S2_RDONLY | PTE_UXN) -#define PAGE_NONE __pgprot(((_PAGE_DEFAULT) & ~PTE_VALID) | PTE_PROT_NONE | PTE_RDONLY | PTE_PXN | PTE_UXN) +#define PAGE_NONE __pgprot(((_PAGE_DEFAULT) & ~PTE_VALID) | PTE_PROT_NONE | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN) #define PAGE_SHARED __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN | PTE_UXN | PTE_WRITE) #define PAGE_SHARED_EXEC __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN | PTE_WRITE) #define PAGE_READONLY __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN) From patchwork Wed Dec 6 12:35:21 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120823 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp6999576qgn; Wed, 6 Dec 2017 04:36:20 -0800 (PST) X-Google-Smtp-Source: AGs4zMY2o/EMUqCxgkm2/oTrMbTjcgdL5TwtfAZuagxWMHulDfTS4pI635pyftrgR7HzQrXueang X-Received: by 10.84.215.207 with SMTP id g15mr21459709plj.369.1512563780056; Wed, 06 Dec 2017 04:36:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512563780; cv=none; d=google.com; s=arc-20160816; b=XhGDjnYGIkGFS+zmGV/t55TrS8XFy5j8MWHY9cHvgTM2mlgR6AMlA+f6nWLCoJjFtT rxMspQeKYExIX2AhoCNOcVY+3tRV8sI8L2Jpqg1SIuTfcI91F3jZMSAHutxkKhwfq/iA jLaVR4KP2VN8kL3nd4+AsY3qHaYj0+kSNnqBQlfPUsflZQ94CXBSX685nSyH1v8diSwP PLMB23+RHVjAylygPtnbMY293YNFZjTkI2SJa1f0YErFU/ztqcby/uSv72Ki3zbmtTFq 5r4+t5gcAvxZy3IJf4Rr9k+oANWGIm6oDYNQgzkrv+cytzfrcI4jR7cpTH3Rz+x5G50E OKSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=gVdqTyu+83Gd3I1MtBMtUEKRWL0d5Zxwn9cWMXcQWNU=; b=lkhmB44a3TOJDvJMGa/Sk0nN6fxwOMCsXAz9bv2UNFga3hKyUbrGee0ibNphczIT54 YwumOjywP7qKBVlgIneFDQ7dPSmprashWeIh5tbEK2T7M1T4LH7eL1BSdwRpSKISzqMj MH6Qyh5FLM/+ZK+L1If39dQz6cZEE9pfRgj+ml9OhZnYUz15ydNTdISH30Oc5G2VsNAh sVzVhA4SRp38I+/dCJdckbN5lQ6glzXyQ0l32IvQHHBqIxrNrOgOc+4EzG3H2ZoWs/O+ Q/lYCm8agZCv+Z0d2aX7RSeZ+3Yt5PCKQIHPA1ubIczixY6BySoGpMP1b/VnGiWouKOz EgbA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f8si1924185pli.451.2017.12.06.04.36.19; Wed, 06 Dec 2017 04:36:20 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752274AbdLFMgQ (ORCPT + 28 others); Wed, 6 Dec 2017 07:36:16 -0500 Received: from foss.arm.com ([217.140.101.70]:34608 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751748AbdLFMgK (ORCPT ); Wed, 6 Dec 2017 07:36:10 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 763C115BF; Wed, 6 Dec 2017 04:36:10 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 46D813F5BD; Wed, 6 Dec 2017 04:36:10 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id DF3B81AE35FB; Wed, 6 Dec 2017 12:36:15 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 02/20] arm64: mm: Temporarily disable ARM64_SW_TTBR0_PAN Date: Wed, 6 Dec 2017 12:35:21 +0000 Message-Id: <1512563739-25239-3-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org We're about to rework the way ASIDs are allocated, switch_mm is implemented and low-level kernel entry/exit is handled, so keep the ARM64_SW_TTBR0_PAN code out of the way whilst we do the heavy lifting. It will be re-enabled in a subsequent patch. Reviewed-by: Mark Rutland Signed-off-by: Will Deacon --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) -- 2.1.4 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a93339f5178f..7e7d7fd152c4 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -910,6 +910,7 @@ endif config ARM64_SW_TTBR0_PAN bool "Emulate Privileged Access Never using TTBR0_EL1 switching" + depends on BROKEN # Temporary while switch_mm is reworked help Enabling this option prevents the kernel from accessing user-space memory directly by pointing TTBR0_EL1 to a reserved From patchwork Wed Dec 6 12:35:22 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120824 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp6999605qgn; Wed, 6 Dec 2017 04:36:22 -0800 (PST) X-Google-Smtp-Source: AGs4zMZ52joY6gFumGvb3HpMOyyEw+kOE4PRlVT1VYKGMjpvPnPitIXTedmm3gSmT5DmmAWhHPMQ X-Received: by 10.159.205.139 with SMTP id v11mr176402plo.233.1512563782348; Wed, 06 Dec 2017 04:36:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512563782; cv=none; d=google.com; s=arc-20160816; b=WlOhQIXBEOSUiYTR6I9bXd4cLbajoKn/b2d/dFNv+yQfF5tot//kMHJ/G8mPBbVVCZ w/NzkDG9wcz35iqgjPJCPU6lsGbe5e87vdRWSVDtBHx6jL+qjgUMgcALOLdgJRl8dbId tluWL4TrYVrkx6h5rIa3Ilu4xHWd/99IVDTlWOyXBlZ+UyIyfkx6nih/cWLRGxH24N7C hjfzTbx4J8Q59qD7ZBtj+884sjusdStl9QqmeHiOM7Bz/t+TvUAeriLqKEFmjZ9bfQ+w HS/lR/WLJBhTXjkIfu4wTfUgP9k6cLQuADNyciZzyxSKxXW6S5yZdjqAmbA/Nzl0+UY4 X9jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=zI2qYtDRi6Q9X+xGgw4GyZcMFvHTlFZq4mxjaPEOJLk=; b=s7NMS2OyaEy7y3vzqZieCre3/7Ns+BQ/PGR02b2ZGu5rcG+n4PwyVICstcM0/pKNvr u8eXBwEPhH2VW1v2nLUDiSzZC3iFaCCnojCA5xyUMfUfj4fXEF8ZafHXJCu3rv9G1x4+ Z3dP7iaJdfUyBYynaxM77dC7i5lxj0IMuzfYamtLmLjJ4ImwlgKiMZuIA88p36QsHzIu rdcq+tcf2TT+oWVaJPTQjqbkzKAZnXW3lT8gI1ydEXJ0TrrYsuKb4Vtar7K2sTMFhWtD Lp+ROmUNPOGI4Fq79vyaAS0BBQIRCvNp/MolL98qYg73rEvpe/woldWe4ggdUYeKPQer oQ8g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f8si1924185pli.451.2017.12.06.04.36.22; Wed, 06 Dec 2017 04:36:22 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752312AbdLFMgU (ORCPT + 28 others); Wed, 6 Dec 2017 07:36:20 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:34614 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751847AbdLFMgK (ORCPT ); Wed, 6 Dec 2017 07:36:10 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 842F91610; Wed, 6 Dec 2017 04:36:10 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 5589E3F627; Wed, 6 Dec 2017 04:36:10 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id EB9B91AE3603; Wed, 6 Dec 2017 12:36:15 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 03/20] arm64: mm: Move ASID from TTBR0 to TTBR1 Date: Wed, 6 Dec 2017 12:35:22 +0000 Message-Id: <1512563739-25239-4-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In preparation for mapping kernelspace and userspace with different ASIDs, move the ASID to TTBR1 and update switch_mm to context-switch TTBR0 via an invalid mapping (the zero page). Reviewed-by: Mark Rutland Signed-off-by: Will Deacon --- arch/arm64/include/asm/mmu_context.h | 7 +++++++ arch/arm64/include/asm/pgtable-hwdef.h | 1 + arch/arm64/include/asm/proc-fns.h | 6 ------ arch/arm64/mm/proc.S | 9 ++++++--- 4 files changed, 14 insertions(+), 9 deletions(-) -- 2.1.4 diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 3257895a9b5e..2d63611e4311 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -57,6 +57,13 @@ static inline void cpu_set_reserved_ttbr0(void) isb(); } +static inline void cpu_switch_mm(pgd_t *pgd, struct mm_struct *mm) +{ + BUG_ON(pgd == swapper_pg_dir); + cpu_set_reserved_ttbr0(); + cpu_do_switch_mm(virt_to_phys(pgd),mm); +} + /* * TCR.T0SZ value to use when the ID map is active. Usually equals * TCR_T0SZ(VA_BITS), unless system RAM is positioned very high in diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index eb0c2bd90de9..8df4cb6ac6f7 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -272,6 +272,7 @@ #define TCR_TG1_4K (UL(2) << TCR_TG1_SHIFT) #define TCR_TG1_64K (UL(3) << TCR_TG1_SHIFT) +#define TCR_A1 (UL(1) << 22) #define TCR_ASID16 (UL(1) << 36) #define TCR_TBI0 (UL(1) << 37) #define TCR_HA (UL(1) << 39) diff --git a/arch/arm64/include/asm/proc-fns.h b/arch/arm64/include/asm/proc-fns.h index 14ad6e4e87d1..16cef2e8449e 100644 --- a/arch/arm64/include/asm/proc-fns.h +++ b/arch/arm64/include/asm/proc-fns.h @@ -35,12 +35,6 @@ extern u64 cpu_do_resume(phys_addr_t ptr, u64 idmap_ttbr); #include -#define cpu_switch_mm(pgd,mm) \ -do { \ - BUG_ON(pgd == swapper_pg_dir); \ - cpu_do_switch_mm(virt_to_phys(pgd),mm); \ -} while (0) - #endif /* __ASSEMBLY__ */ #endif /* __KERNEL__ */ #endif /* __ASM_PROCFNS_H */ diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 95233dfc4c39..a8a64898a2aa 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -139,9 +139,12 @@ ENDPROC(cpu_do_resume) */ ENTRY(cpu_do_switch_mm) pre_ttbr0_update_workaround x0, x2, x3 + mrs x2, ttbr1_el1 mmid x1, x1 // get mm->context.id - bfi x0, x1, #48, #16 // set the ASID - msr ttbr0_el1, x0 // set TTBR0 + bfi x2, x1, #48, #16 // set the ASID + msr ttbr1_el1, x2 // in TTBR1 (since TCR.A1 is set) + isb + msr ttbr0_el1, x0 // now update TTBR0 isb post_ttbr0_update_workaround ret @@ -224,7 +227,7 @@ ENTRY(__cpu_setup) * both user and kernel. */ ldr x10, =TCR_TxSZ(VA_BITS) | TCR_CACHE_FLAGS | TCR_SMP_FLAGS | \ - TCR_TG_FLAGS | TCR_ASID16 | TCR_TBI0 + TCR_TG_FLAGS | TCR_ASID16 | TCR_TBI0 | TCR_A1 tcr_set_idmap_t0sz x10, x9 /* From patchwork Wed Dec 6 12:35:23 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120841 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7003667qgn; Wed, 6 Dec 2017 04:40:40 -0800 (PST) X-Google-Smtp-Source: AGs4zMZ+T6vWA46tUNBFD25+Fg17zkfIO8ymn4Jj+pKgwnqozyjnyV0vW6Q7pbEQEQdldYVJekop X-Received: by 10.101.90.138 with SMTP id c10mr20915472pgt.441.1512564040361; Wed, 06 Dec 2017 04:40:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512564040; cv=none; d=google.com; s=arc-20160816; b=WS8xXZULngG6KM0ns1FAmIL2Hbo6PQdJ1jxUJZqUjnzqYYpSoJeT/LyA9i6vZqwOtW pVDxqUxpIwmnSbzaSg43QiRgqpBx0kdpajAwgvp7VC5O52KLrcNBHOLuUjxKixUe+qVu SskmDqpHuwix1smjmLy6FYfcTL1a5v21My4XaIvuXPbt0YNwZ4EdnyGy5QO4GWZqYv02 vjxxNJ0wNCRUbmxgQn2Mv1DrVsOOfsMFqPeiXlJ8LEdEdp0WiCG4Jnemi/bThkK8KQN7 RUkL14Rw991doH5usUAmAtf2p9ZqwNpR1v+4qR9t9pI9SihOwwAu+N57DUFihUrcooVs AdgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=QoyhVXlf7ELmqfAEOG/clCTloI3eCK0ovFi1xt4PMSs=; b=u4nW+Bp2WIuctlFQgiaG7eKJFpRzk49YFIiKSbZ/rWhsTKxRoL45EgXVr6nQ6Bh0gH Ig9JN4FyYURwbAStaFeDBiKmpswD6cAwnpqLSFb4mDttGc4z9ToeZtBOXagTc+eqhufF 0/s4lrKahLW/vf6V3urJnjOOM9p9JVNBGt+DjI+t3QzZZylMaw5cS1S2O5WBxMY7Lwp6 R7p/86lLSGudTSK5v+Bch0E8mSluZvukHUfaDHrOH/RG4WSgw8dfosoabbYaYCU3N9aT x+ohxHPAtTFchaNT3ODfm3K/EusLddt3wbw1FlfxC4WdI/ASUIsy/Xnar7y4w/2M5WS0 aH/g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a6si1914876pll.466.2017.12.06.04.40.40; Wed, 06 Dec 2017 04:40:40 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752609AbdLFMkh (ORCPT + 28 others); Wed, 6 Dec 2017 07:40:37 -0500 Received: from foss.arm.com ([217.140.101.70]:34620 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751864AbdLFMgK (ORCPT ); Wed, 6 Dec 2017 07:36:10 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9274D164F; Wed, 6 Dec 2017 04:36:10 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 63E643F7A6; Wed, 6 Dec 2017 04:36:10 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 077E41AE3632; Wed, 6 Dec 2017 12:36:16 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 04/20] arm64: mm: Remove pre_ttbr0_update_workaround for Falkor erratum #E1003 Date: Wed, 6 Dec 2017 12:35:23 +0000 Message-Id: <1512563739-25239-5-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The pre_ttbr0_update_workaround hook is called prior to context-switching TTBR0 because Falkor erratum E1003 can cause TLB allocation with the wrong ASID if both the ASID and the base address of the TTBR are updated at the same time. With the ASID sitting safely in TTBR1, we no longer update things atomically, so we can remove the pre_ttbr0_update_workaround macro as it's no longer required. The erratum infrastructure and documentation is left around for #E1003, as it will be required by the entry trampoline code in a future patch. Reviewed-by: Mark Rutland Signed-off-by: Will Deacon --- arch/arm64/include/asm/assembler.h | 22 ---------------------- arch/arm64/include/asm/mmu_context.h | 2 -- arch/arm64/mm/context.c | 11 ----------- arch/arm64/mm/proc.S | 1 - 4 files changed, 36 deletions(-) -- 2.1.4 diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index aef72d886677..e1fa5db858b7 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -26,7 +26,6 @@ #include #include #include -#include #include #include #include @@ -478,27 +477,6 @@ alternative_endif .endm /* - * Errata workaround prior to TTBR0_EL1 update - * - * val: TTBR value with new BADDR, preserved - * tmp0: temporary register, clobbered - * tmp1: other temporary register, clobbered - */ - .macro pre_ttbr0_update_workaround, val, tmp0, tmp1 -#ifdef CONFIG_QCOM_FALKOR_ERRATUM_1003 -alternative_if ARM64_WORKAROUND_QCOM_FALKOR_E1003 - mrs \tmp0, ttbr0_el1 - mov \tmp1, #FALKOR_RESERVED_ASID - bfi \tmp0, \tmp1, #48, #16 // reserved ASID + old BADDR - msr ttbr0_el1, \tmp0 - isb - bfi \tmp0, \val, #0, #48 // reserved ASID + new BADDR - msr ttbr0_el1, \tmp0 - isb -alternative_else_nop_endif -#endif - .endm - /* * Errata workaround post TTBR0_EL1 update. */ diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 2d63611e4311..f9744944cf12 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -19,8 +19,6 @@ #ifndef __ASM_MMU_CONTEXT_H #define __ASM_MMU_CONTEXT_H -#define FALKOR_RESERVED_ASID 1 - #ifndef __ASSEMBLY__ #include diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index 6f4017046323..78a2dc596fee 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -79,13 +79,6 @@ void verify_cpu_asid_bits(void) } } -static void set_reserved_asid_bits(void) -{ - if (IS_ENABLED(CONFIG_QCOM_FALKOR_ERRATUM_1003) && - cpus_have_const_cap(ARM64_WORKAROUND_QCOM_FALKOR_E1003)) - __set_bit(FALKOR_RESERVED_ASID, asid_map); -} - static void flush_context(unsigned int cpu) { int i; @@ -94,8 +87,6 @@ static void flush_context(unsigned int cpu) /* Update the list of reserved ASIDs and the ASID bitmap. */ bitmap_clear(asid_map, 0, NUM_USER_ASIDS); - set_reserved_asid_bits(); - for_each_possible_cpu(i) { asid = atomic64_xchg_relaxed(&per_cpu(active_asids, i), 0); /* @@ -254,8 +245,6 @@ static int asids_init(void) panic("Failed to allocate bitmap for %lu ASIDs\n", NUM_USER_ASIDS); - set_reserved_asid_bits(); - pr_info("ASID allocator initialised with %lu entries\n", NUM_USER_ASIDS); return 0; } diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index a8a64898a2aa..f2ff0837577c 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -138,7 +138,6 @@ ENDPROC(cpu_do_resume) * - pgd_phys - physical address of new TTB */ ENTRY(cpu_do_switch_mm) - pre_ttbr0_update_workaround x0, x2, x3 mrs x2, ttbr1_el1 mmid x1, x1 // get mm->context.id bfi x2, x1, #48, #16 // set the ASID From patchwork Wed Dec 6 12:35:24 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120840 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7003573qgn; Wed, 6 Dec 2017 04:40:34 -0800 (PST) X-Google-Smtp-Source: AGs4zMaBkt7cg8iIDwbB5f/YK+5HBI3J8aF/MGEzx6d+MhZ/BHbaRiTdz102i9SBnTi2+IJPoyhq X-Received: by 10.84.178.37 with SMTP id y34mr22085723plb.260.1512564034223; Wed, 06 Dec 2017 04:40:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512564034; cv=none; d=google.com; s=arc-20160816; b=sJF/mDz5bNlZ/K9qZtkGQXLvlU009ClXsXiJLjNkA9uEkfa7NZ8sqSY3bNpbIIEjoR iebOaMmIYvYb8fWxVjJ7qMXOWapatPaDO+tOGhA9mSuwsx+GhvGKOweNc48A86CF3Gvb qeWqudAOe2rUdAU8AsCKTmO1aoTuyy2b84z6OI8B3SbC2a2jFbwnDkwvOTw7wmiLmwVM YJFvcYWBlVrIU8+gggCHyIRcudgVIywfvdlHnJyNRW2OW/7amUcEwxbGkkfnnLAh6NWs fgg+OhqQBBvG/cGg4GqbVwPR9cvmf28Ap/SQLl6qetDWWzNLhd5/+bnacQ1KD5dDFlyP MTig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=xsYaxVEvIt1tMVeZ/68raCM1OvJcwsghUJ9N018ZgRU=; b=nx35OP0rj5sHy8Zt/jMLWPFS5DFTfrkDKMrS/Vx3J6LZDwbjOyUOGWNitS3cJyH34z cqNhYjQ6JN13Ys31t0nZbjo5bEoMpU3SPi074poz12tEJmM7Wd2JNXs5SfSgoutJ50dc mhnned2MhD8WeoSWuF4ziXTA4K5AyzSIUJncC9+zmijGzIc+H7hEYSJn2lg3AmIbeIOx No9y4frGKJVNJR+xQk6iOPGcvIRMSJRw1GntdzDzH1rIfKpf9kyJf3cX+T8e3LrMij03 CYiTFu5sWb4pq8vJrGIDfD0ZsVIkhlE9A+VLa/wCK65Wkwl+psp4DC3dW1MCnmQaibNu g3Rg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h8si1999757pfi.0.2017.12.06.04.40.33; Wed, 06 Dec 2017 04:40:34 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752601AbdLFMkc (ORCPT + 28 others); Wed, 6 Dec 2017 07:40:32 -0500 Received: from foss.arm.com ([217.140.101.70]:34694 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752141AbdLFMgL (ORCPT ); Wed, 6 Dec 2017 07:36:11 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 60543165C; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 3211C3F53E; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 1644D1AE3719; Wed, 6 Dec 2017 12:36:16 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 05/20] arm64: mm: Rename post_ttbr0_update_workaround Date: Wed, 6 Dec 2017 12:35:24 +0000 Message-Id: <1512563739-25239-6-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The post_ttbr0_update_workaround hook applies to any change to TTBRx_EL1. Since we're using TTBR1 for the ASID, rename the hook to make it clearer as to what it's doing. Reviewed-by: Mark Rutland Signed-off-by: Will Deacon --- arch/arm64/include/asm/assembler.h | 5 ++--- arch/arm64/kernel/entry.S | 2 +- arch/arm64/mm/proc.S | 2 +- 3 files changed, 4 insertions(+), 5 deletions(-) -- 2.1.4 diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index e1fa5db858b7..c45bc94f15d0 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -477,10 +477,9 @@ alternative_endif .endm /* -/* - * Errata workaround post TTBR0_EL1 update. + * Errata workaround post TTBRx_EL1 update. */ - .macro post_ttbr0_update_workaround + .macro post_ttbr_update_workaround #ifdef CONFIG_CAVIUM_ERRATUM_27456 alternative_if ARM64_WORKAROUND_CAVIUM_27456 ic iallu diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 6d14b8f29b5f..804e43c9cb0b 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -257,7 +257,7 @@ alternative_else_nop_endif * Cavium erratum 27456 (broadcast TLBI instructions may cause I-cache * corruption). */ - post_ttbr0_update_workaround + post_ttbr_update_workaround .endif 1: .if \el != 0 diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index f2ff0837577c..3146dc96f05b 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -145,7 +145,7 @@ ENTRY(cpu_do_switch_mm) isb msr ttbr0_el1, x0 // now update TTBR0 isb - post_ttbr0_update_workaround + post_ttbr_update_workaround ret ENDPROC(cpu_do_switch_mm) From patchwork Wed Dec 6 12:35:25 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120825 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp6999652qgn; Wed, 6 Dec 2017 04:36:24 -0800 (PST) X-Google-Smtp-Source: AGs4zMbqPfX9945m1jVknmtcrVdEzICzWNFjRXFtm6xVG2dH+id4La026Xwm8mvy91Wp1QWjr/Bf X-Received: by 10.159.253.144 with SMTP id q16mr4585533pls.104.1512563784712; Wed, 06 Dec 2017 04:36:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512563784; cv=none; d=google.com; s=arc-20160816; b=Z+S2kRvejDDfUvx9y07UKjJHIV9Qg5t0MC+EFfPbMltfjSm/72n2eSMNAs7/4jpEy4 s3y78SwCxpSjK4fBA9QOFp8SYoQhDyl6BxwwjjWiyTfD5Ye9NMxO5VqxXOaXvp37QAMw O/t4JuSXORSD+pH6M8iXzyo/jLEXT/c1UmBXPdn0F5go/BpvidNZq3X2ZNyqh31PQaKB haa173xYIgRsqje2jGqhwIU+nMtpCH11sajDvQsNjA+Xv9mMn+TqFDeaa7QLhveZEryZ xIaAL0gBI6U8nK6DwyNv3iSkCwWtnFibH6N6CHiXGdSI+XtNrw+x7zamhQSKFWyFHMPX SaYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=5/fj0owyk81rdZMNGWL2cLmodNEJtaOE4msn490M5sk=; b=u3T3VQFaNsF0SOe3PyMhA3Am07aEQEQSd1liBxGB+IITb0/ueOSMUJsvYOiMgFTzTz KPVVKKgYQrdWcnYIh39Q55+7vG54vV7YYWMlIPivzPEpjwaqyo/E+T+e4rxbM292wM+3 np8j2akqgVDbxXB1BOOyUi99zJ38mAmnu5Lwuuqc1ClV440AAgBJqkUJpfkHG8R/JvQJ sa5Tjje6oRWXb59Xr01W3FrkkQQiWZmmCc5LFdRqu+wJA6GOlKVmQ9/sPCGglXP7Fokb A6ifqKqzLet5FJannLmD6PEuzz5B4nTADzqtaDsmPHrkBQBetwViCoQsi25rvPV0I4Yd gnkw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f8si1924185pli.451.2017.12.06.04.36.24; Wed, 06 Dec 2017 04:36:24 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752361AbdLFMgX (ORCPT + 28 others); Wed, 6 Dec 2017 07:36:23 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:34686 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752047AbdLFMgL (ORCPT ); Wed, 6 Dec 2017 07:36:11 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 579391650; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 292BE3F236; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 296BD1AE3744; Wed, 6 Dec 2017 12:36:16 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 06/20] arm64: mm: Fix and re-enable ARM64_SW_TTBR0_PAN Date: Wed, 6 Dec 2017 12:35:25 +0000 Message-Id: <1512563739-25239-7-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org With the ASID now installed in TTBR1, we can re-enable ARM64_SW_TTBR0_PAN by ensuring that we switch to a reserved ASID of zero when disabling user access and restore the active user ASID on the uaccess enable path. Reviewed-by: Mark Rutland Signed-off-by: Will Deacon --- arch/arm64/Kconfig | 1 - arch/arm64/include/asm/asm-uaccess.h | 25 +++++++++++++++++-------- arch/arm64/include/asm/uaccess.h | 21 +++++++++++++++++---- arch/arm64/kernel/entry.S | 4 ++-- arch/arm64/lib/clear_user.S | 2 +- arch/arm64/lib/copy_from_user.S | 2 +- arch/arm64/lib/copy_in_user.S | 2 +- arch/arm64/lib/copy_to_user.S | 2 +- arch/arm64/mm/cache.S | 2 +- arch/arm64/xen/hypercall.S | 2 +- 10 files changed, 42 insertions(+), 21 deletions(-) -- 2.1.4 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 7e7d7fd152c4..a93339f5178f 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -910,7 +910,6 @@ endif config ARM64_SW_TTBR0_PAN bool "Emulate Privileged Access Never using TTBR0_EL1 switching" - depends on BROKEN # Temporary while switch_mm is reworked help Enabling this option prevents the kernel from accessing user-space memory directly by pointing TTBR0_EL1 to a reserved diff --git a/arch/arm64/include/asm/asm-uaccess.h b/arch/arm64/include/asm/asm-uaccess.h index b3da6c886835..21b8cf304028 100644 --- a/arch/arm64/include/asm/asm-uaccess.h +++ b/arch/arm64/include/asm/asm-uaccess.h @@ -16,11 +16,20 @@ add \tmp1, \tmp1, #SWAPPER_DIR_SIZE // reserved_ttbr0 at the end of swapper_pg_dir msr ttbr0_el1, \tmp1 // set reserved TTBR0_EL1 isb + sub \tmp1, \tmp1, #SWAPPER_DIR_SIZE + bic \tmp1, \tmp1, #(0xffff << 48) + msr ttbr1_el1, \tmp1 // set reserved ASID + isb .endm - .macro __uaccess_ttbr0_enable, tmp1 + .macro __uaccess_ttbr0_enable, tmp1, tmp2 get_thread_info \tmp1 ldr \tmp1, [\tmp1, #TSK_TI_TTBR0] // load saved TTBR0_EL1 + mrs \tmp2, ttbr1_el1 + extr \tmp2, \tmp2, \tmp1, #48 + ror \tmp2, \tmp2, #16 + msr ttbr1_el1, \tmp2 // set the active ASID + isb msr ttbr0_el1, \tmp1 // set the non-PAN TTBR0_EL1 isb .endm @@ -31,18 +40,18 @@ alternative_if_not ARM64_HAS_PAN alternative_else_nop_endif .endm - .macro uaccess_ttbr0_enable, tmp1, tmp2 + .macro uaccess_ttbr0_enable, tmp1, tmp2, tmp3 alternative_if_not ARM64_HAS_PAN - save_and_disable_irq \tmp2 // avoid preemption - __uaccess_ttbr0_enable \tmp1 - restore_irq \tmp2 + save_and_disable_irq \tmp3 // avoid preemption + __uaccess_ttbr0_enable \tmp1, \tmp2 + restore_irq \tmp3 alternative_else_nop_endif .endm #else .macro uaccess_ttbr0_disable, tmp1 .endm - .macro uaccess_ttbr0_enable, tmp1, tmp2 + .macro uaccess_ttbr0_enable, tmp1, tmp2, tmp3 .endm #endif @@ -56,8 +65,8 @@ alternative_if ARM64_ALT_PAN_NOT_UAO alternative_else_nop_endif .endm - .macro uaccess_enable_not_uao, tmp1, tmp2 - uaccess_ttbr0_enable \tmp1, \tmp2 + .macro uaccess_enable_not_uao, tmp1, tmp2, tmp3 + uaccess_ttbr0_enable \tmp1, \tmp2, \tmp3 alternative_if ARM64_ALT_PAN_NOT_UAO SET_PSTATE_PAN(0) alternative_else_nop_endif diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h index fc0f9eb66039..750a3b76a01c 100644 --- a/arch/arm64/include/asm/uaccess.h +++ b/arch/arm64/include/asm/uaccess.h @@ -107,15 +107,19 @@ static inline void __uaccess_ttbr0_disable(void) { unsigned long ttbr; + ttbr = read_sysreg(ttbr1_el1); /* reserved_ttbr0 placed at the end of swapper_pg_dir */ - ttbr = read_sysreg(ttbr1_el1) + SWAPPER_DIR_SIZE; - write_sysreg(ttbr, ttbr0_el1); + write_sysreg(ttbr + SWAPPER_DIR_SIZE, ttbr0_el1); + isb(); + /* Set reserved ASID */ + ttbr &= ~(0xffffUL << 48); + write_sysreg(ttbr, ttbr1_el1); isb(); } static inline void __uaccess_ttbr0_enable(void) { - unsigned long flags; + unsigned long flags, ttbr0, ttbr1; /* * Disable interrupts to avoid preemption between reading the 'ttbr0' @@ -123,7 +127,16 @@ static inline void __uaccess_ttbr0_enable(void) * roll-over and an update of 'ttbr0'. */ local_irq_save(flags); - write_sysreg(current_thread_info()->ttbr0, ttbr0_el1); + ttbr0 = current_thread_info()->ttbr0; + + /* Restore active ASID */ + ttbr1 = read_sysreg(ttbr1_el1); + ttbr1 |= ttbr0 & (0xffffUL << 48); + write_sysreg(ttbr1, ttbr1_el1); + isb(); + + /* Restore user page table */ + write_sysreg(ttbr0, ttbr0_el1); isb(); local_irq_restore(flags); } diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 804e43c9cb0b..d454d8ed45e4 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -184,7 +184,7 @@ alternative_if ARM64_HAS_PAN alternative_else_nop_endif .if \el != 0 - mrs x21, ttbr0_el1 + mrs x21, ttbr1_el1 tst x21, #0xffff << 48 // Check for the reserved ASID orr x23, x23, #PSR_PAN_BIT // Set the emulated PAN in the saved SPSR b.eq 1f // TTBR0 access already disabled @@ -248,7 +248,7 @@ alternative_else_nop_endif tbnz x22, #22, 1f // Skip re-enabling TTBR0 access if the PSR_PAN_BIT is set .endif - __uaccess_ttbr0_enable x0 + __uaccess_ttbr0_enable x0, x1 .if \el == 0 /* diff --git a/arch/arm64/lib/clear_user.S b/arch/arm64/lib/clear_user.S index e88fb99c1561..8f9c4641e706 100644 --- a/arch/arm64/lib/clear_user.S +++ b/arch/arm64/lib/clear_user.S @@ -30,7 +30,7 @@ * Alignment fixed up by hardware. */ ENTRY(__clear_user) - uaccess_enable_not_uao x2, x3 + uaccess_enable_not_uao x2, x3, x4 mov x2, x1 // save the size for fixup return subs x1, x1, #8 b.mi 2f diff --git a/arch/arm64/lib/copy_from_user.S b/arch/arm64/lib/copy_from_user.S index 4b5d826895ff..69d86a80f3e2 100644 --- a/arch/arm64/lib/copy_from_user.S +++ b/arch/arm64/lib/copy_from_user.S @@ -64,7 +64,7 @@ end .req x5 ENTRY(__arch_copy_from_user) - uaccess_enable_not_uao x3, x4 + uaccess_enable_not_uao x3, x4, x5 add end, x0, x2 #include "copy_template.S" uaccess_disable_not_uao x3 diff --git a/arch/arm64/lib/copy_in_user.S b/arch/arm64/lib/copy_in_user.S index b24a830419ad..e442b531252a 100644 --- a/arch/arm64/lib/copy_in_user.S +++ b/arch/arm64/lib/copy_in_user.S @@ -65,7 +65,7 @@ end .req x5 ENTRY(raw_copy_in_user) - uaccess_enable_not_uao x3, x4 + uaccess_enable_not_uao x3, x4, x5 add end, x0, x2 #include "copy_template.S" uaccess_disable_not_uao x3 diff --git a/arch/arm64/lib/copy_to_user.S b/arch/arm64/lib/copy_to_user.S index 351f0766f7a6..318f15d5c336 100644 --- a/arch/arm64/lib/copy_to_user.S +++ b/arch/arm64/lib/copy_to_user.S @@ -63,7 +63,7 @@ end .req x5 ENTRY(__arch_copy_to_user) - uaccess_enable_not_uao x3, x4 + uaccess_enable_not_uao x3, x4, x5 add end, x0, x2 #include "copy_template.S" uaccess_disable_not_uao x3 diff --git a/arch/arm64/mm/cache.S b/arch/arm64/mm/cache.S index 7f1dbe962cf5..6cd20a8c0952 100644 --- a/arch/arm64/mm/cache.S +++ b/arch/arm64/mm/cache.S @@ -49,7 +49,7 @@ ENTRY(flush_icache_range) * - end - virtual end address of region */ ENTRY(__flush_cache_user_range) - uaccess_ttbr0_enable x2, x3 + uaccess_ttbr0_enable x2, x3, x4 dcache_line_size x2, x3 sub x3, x2, #1 bic x4, x0, x3 diff --git a/arch/arm64/xen/hypercall.S b/arch/arm64/xen/hypercall.S index 401ceb71540c..acdbd2c9e899 100644 --- a/arch/arm64/xen/hypercall.S +++ b/arch/arm64/xen/hypercall.S @@ -101,7 +101,7 @@ ENTRY(privcmd_call) * need the explicit uaccess_enable/disable if the TTBR0 PAN emulation * is enabled (it implies that hardware UAO and PAN disabled). */ - uaccess_ttbr0_enable x6, x7 + uaccess_ttbr0_enable x6, x7, x8 hvc XEN_IMM /* From patchwork Wed Dec 6 12:35:26 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120838 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7003473qgn; Wed, 6 Dec 2017 04:40:27 -0800 (PST) X-Google-Smtp-Source: AGs4zMbJcOkGJZi0bNFC1HnoO53Hlnmg43EDeAPAUYulZH3tkbT/6n9iNg7RUEJ8PpDe4nJXZhUV X-Received: by 10.101.100.65 with SMTP id s1mr21070516pgv.185.1512564027180; Wed, 06 Dec 2017 04:40:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512564027; cv=none; d=google.com; s=arc-20160816; b=CwzQEqoggwxBRLLZAC+549ikEM6tFaXBmJS8aJqEvblz1Q2Ues/INSBKk4fxjxARLz eF+L5+PWhSYRbln8dzsGDteMkql5Z9WVPjWqClJU7Y6P3LsgEPD6c5UtrNhVe3B7dZVG N5oa1aZEBZR9tF9ne4RzzWrXotzpN21W+duKsIgzFDnNGcMjDnJJPINBox83CL8pjR04 u0fnKqjXNJGvkvEYHMfRInrMJF9UjrldOHYRWiRwOXS32Rdye4leWSLf1ajRYn2GC/B6 2LBzYRISHnYg76i8t3SWy6eiEq++wJKNexW313OLyJnl0YomhF19UWx98wJT0myI6hTm JUUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=rWa85yXB3p8CJfg0UnnXrX+2QJNQTHxDIns8qwvgeg4=; b=pcbGWiiFhg9l/UXYPCuV/wSRqOAcnMaBElkwyGNsEFgPOWcNC+m7EK9W8DcC2m9C7c u0vXzxm0EvCRNTGLA3CLyo1o/7U7Mczj1rDnqAKZEkoH1vYTlLXAH4VeOc86ifsER7gZ Fxg4fPbO/79Cecgf9fD+3bgtaDsDDddv37dMMk+pY3rLz6kC6c2GjyRY5IkYUs8EkWXM pNbEQzXsPBaSDko11sP71vLBgE+D2CSKjcTF/+a0JkhvX3r5yORmGfAn9Yw5jCM/fEBQ qHWwXdvLnoZNHjTi3GENGCM4VY8nmDrcPFQ+yDLWpe7NmDpxOwBWBrgP9FDq4KAIni19 LfDA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m32si1893838pld.592.2017.12.06.04.40.26; Wed, 06 Dec 2017 04:40:27 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752512AbdLFMkY (ORCPT + 28 others); Wed, 6 Dec 2017 07:40:24 -0500 Received: from foss.arm.com ([217.140.101.70]:34712 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752145AbdLFMgL (ORCPT ); Wed, 6 Dec 2017 07:36:11 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 69A4F1682; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 3B0033F5BD; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 3859F1AE37A6; Wed, 6 Dec 2017 12:36:16 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 07/20] arm64: mm: Allocate ASIDs in pairs Date: Wed, 6 Dec 2017 12:35:26 +0000 Message-Id: <1512563739-25239-8-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In preparation for separate kernel/user ASIDs, allocate them in pairs for each mm_struct. The bottom bit distinguishes the two: if it is set, then the ASID will map only userspace. Reviewed-by: Mark Rutland Signed-off-by: Will Deacon --- arch/arm64/include/asm/mmu.h | 1 + arch/arm64/mm/context.c | 25 +++++++++++++++++-------- 2 files changed, 18 insertions(+), 8 deletions(-) -- 2.1.4 diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 0d34bf0a89c7..01bfb184f2a8 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -17,6 +17,7 @@ #define __ASM_MMU_H #define MMCF_AARCH32 0x1 /* mm context flag for AArch32 executables */ +#define USER_ASID_FLAG (UL(1) << 48) typedef struct { atomic64_t id; diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index 78a2dc596fee..1cb3bc92ae5c 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -39,7 +39,16 @@ static cpumask_t tlb_flush_pending; #define ASID_MASK (~GENMASK(asid_bits - 1, 0)) #define ASID_FIRST_VERSION (1UL << asid_bits) -#define NUM_USER_ASIDS ASID_FIRST_VERSION + +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 +#define NUM_USER_ASIDS (ASID_FIRST_VERSION >> 1) +#define asid2idx(asid) (((asid) & ~ASID_MASK) >> 1) +#define idx2asid(idx) (((idx) << 1) & ~ASID_MASK) +#else +#define NUM_USER_ASIDS (ASID_FIRST_VERSION) +#define asid2idx(asid) ((asid) & ~ASID_MASK) +#define idx2asid(idx) asid2idx(idx) +#endif /* Get the ASIDBits supported by the current CPU */ static u32 get_cpu_asid_bits(void) @@ -98,7 +107,7 @@ static void flush_context(unsigned int cpu) */ if (asid == 0) asid = per_cpu(reserved_asids, i); - __set_bit(asid & ~ASID_MASK, asid_map); + __set_bit(asid2idx(asid), asid_map); per_cpu(reserved_asids, i) = asid; } @@ -153,16 +162,16 @@ static u64 new_context(struct mm_struct *mm, unsigned int cpu) * We had a valid ASID in a previous life, so try to re-use * it if possible. */ - asid &= ~ASID_MASK; - if (!__test_and_set_bit(asid, asid_map)) + if (!__test_and_set_bit(asid2idx(asid), asid_map)) return newasid; } /* * Allocate a free ASID. If we can't find one, take a note of the - * currently active ASIDs and mark the TLBs as requiring flushes. - * We always count from ASID #1, as we use ASID #0 when setting a - * reserved TTBR0 for the init_mm. + * currently active ASIDs and mark the TLBs as requiring flushes. We + * always count from ASID #2 (index 1), as we use ASID #0 when setting + * a reserved TTBR0 for the init_mm and we allocate ASIDs in even/odd + * pairs. */ asid = find_next_zero_bit(asid_map, NUM_USER_ASIDS, cur_idx); if (asid != NUM_USER_ASIDS) @@ -179,7 +188,7 @@ static u64 new_context(struct mm_struct *mm, unsigned int cpu) set_asid: __set_bit(asid, asid_map); cur_idx = asid; - return asid | generation; + return idx2asid(asid) | generation; } void check_and_switch_context(struct mm_struct *mm, unsigned int cpu) From patchwork Wed Dec 6 12:35:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120839 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7003543qgn; Wed, 6 Dec 2017 04:40:32 -0800 (PST) X-Google-Smtp-Source: AGs4zMZGwXIljjJhHgCDdav/Ju1n+AO4J5lMRXdJTMtvSBoF1352a8eGBEs3pcU0sPFUTJyyBpKO X-Received: by 10.98.153.221 with SMTP id t90mr2661646pfk.210.1512564032289; Wed, 06 Dec 2017 04:40:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512564032; cv=none; d=google.com; s=arc-20160816; b=qFgJKnO14HSN42zh3xS58UKR2rH2/8sxLdJ1p0dhlZ4w+wTMseGiIDH4AVrl+nBdVU DCDEEu7DbJn5/gHfZPmr4xzr0xEM7G0NdzdmshvKKV+XvgNpyQngK4Q6217h0VPlMEmX U81+r/b6fAN4GeLN87kD0YWCt1cQSXjF2b9bDhTqo2JomborlFloXhYRtcOlxSDjcRSg SwKvIj64lGq4w6EfYxgJ8u6CY0NVppIc9vOCeGwKO4DZamRKDSuWhQWyUyEIW8IpXx+h RPkF4plTQtGQ+bg5pCq/+rLXwEcGkwg+sBtgl2F4CmSvyZNZxHmupwDuS/KxBPmxXDPb Pkdg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=SH81J+ySp5Q8ZwdDnG6GDnzY+iTGLqepGx9CviqoXLI=; b=cPs7zgjTSFMCJjfYW2tmrmE1GDr2QrTcZqmpDt3WY/2mu0PjjSzUPBcASdqVJU3sCD 2+/MfEaWl5IKNW/wRUzI024sfbpyCGa/6fkxTd3OucXYMT/+bCpIeMrSwSzqxvUbzVyH qvzzhmNRxmuWnMrL9q3aFXUwnHLRnLJ9CethRsbv71/OyN4eenfhiBCZbozD84gWqQty TIZSmaTSwwOw7k86qV639dMWGXN0HdTwjmfdYcp+eDpouBuhtaQgGoOjXob/6AyDngS1 VXusjObcPwrVC0JlLDPYWmo56vfd5NbI8WfDgqBc2mOBBEMg/foVkqDk3H1KjgdBkfHS tiog== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g8si133054plt.245.2017.12.06.04.40.32; Wed, 06 Dec 2017 04:40:32 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752564AbdLFMka (ORCPT + 28 others); Wed, 6 Dec 2017 07:40:30 -0500 Received: from foss.arm.com ([217.140.101.70]:34704 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752143AbdLFMgL (ORCPT ); Wed, 6 Dec 2017 07:36:11 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 721F81688; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 43F153F627; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 489811AE37AD; Wed, 6 Dec 2017 12:36:16 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 08/20] arm64: mm: Add arm64_kernel_unmapped_at_el0 helper Date: Wed, 6 Dec 2017 12:35:27 +0000 Message-Id: <1512563739-25239-9-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In order for code such as TLB invalidation to operate efficiently when the decision to map the kernel at EL0 is determined at runtime, this patch introduces a helper function, arm64_kernel_unmapped_at_el0, to determine whether or not the kernel is mapped whilst running in userspace. Currently, this just reports the value of CONFIG_UNMAP_KERNEL_AT_EL0, but will later be hooked up to a fake CPU capability using a static key. Reviewed-by: Mark Rutland Signed-off-by: Will Deacon --- arch/arm64/include/asm/mmu.h | 8 ++++++++ 1 file changed, 8 insertions(+) -- 2.1.4 diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 01bfb184f2a8..c07954638658 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -19,6 +19,8 @@ #define MMCF_AARCH32 0x1 /* mm context flag for AArch32 executables */ #define USER_ASID_FLAG (UL(1) << 48) +#ifndef __ASSEMBLY__ + typedef struct { atomic64_t id; void *vdso; @@ -32,6 +34,11 @@ typedef struct { */ #define ASID(mm) ((mm)->context.id.counter & 0xffff) +static inline bool arm64_kernel_unmapped_at_el0(void) +{ + return IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0); +} + extern void paging_init(void); extern void bootmem_init(void); extern void __iomem *early_io_map(phys_addr_t phys, unsigned long virt); @@ -42,4 +49,5 @@ extern void create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, extern void *fixmap_remap_fdt(phys_addr_t dt_phys); extern void mark_linear_text_alias_ro(void); +#endif /* !__ASSEMBLY__ */ #endif From patchwork Wed Dec 6 12:35:28 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120837 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7002192qgn; Wed, 6 Dec 2017 04:39:00 -0800 (PST) X-Google-Smtp-Source: AGs4zMa5Dmu/fZMDVoTwWuQXt3/XbGYEuN1w/tqPvEv6Q8Qhh6yE9H12nguTuSBPgFDkBTfrSNwM X-Received: by 10.98.206.133 with SMTP id y127mr2617234pfg.152.1512563940621; Wed, 06 Dec 2017 04:39:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512563940; cv=none; d=google.com; s=arc-20160816; b=n/zp30xiSMADfAGI5Kyq02Z+9gou3nJdht9qd9vnxt295nD89Xmw652NLS335vgGRo pq/2eKrRz0x9wWzl1+CRX7UDLpPYRdxnms0+rtr7C5uoHyn3KT60Dd+/M65AiYVvhvFK yLQLc8/kINbfeDmEZaJJxrbsTcIOic0oSpdBlu/wS7ZrodOTt7wpmUbyhnukxtMR80qS W8zQX8hT0RRaj398twg+4obB9RB3IK9+UXW0xFgwIRx14CTY0q0h9wWZVle0dlxColoI V/qZIo1tI6jkDc43Fy2SisTCs87JcKD0FGRyRIXUsb7Twk4lLMg5qdb/6ljXXEAnmrL2 6Fbg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=8q4P6pK8i5ZhwdriSm1KrtzGkdE6ze/BH2aPvOP04oE=; b=kK0KlF+bkiITgIZ+r1hiiSfksUQ0wIcJje72M+czudBniDZAijvDHR4vHw5AVa3FuR mcWlYL/68MKIJVp8BsFnlX1VZ+Os9/rJqK2IFdddnMVE8kax1LwAYumoSS+bCsI0j8jp X84JbDtl5vyLpp6K3NqMBHgyR6//oRKsZCpevexWl3W1h40J1GWjl3K4/FYsz7dwXH+2 dgoK5rZmjbbhLubzg3dLYrFnZD15eHADMw1kMPrfApLJi0KPX+k1osbHmedcrs9iHkfc 9Gyy1JnSWmXFi1eVvBk+/20LealhB0mQzTfFyEiWdu2uWW3A4MvIk5ZG8vctq4V/PDFD OhUw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w9si431875plp.783.2017.12.06.04.39.00; Wed, 06 Dec 2017 04:39:00 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752676AbdLFMi5 (ORCPT + 28 others); Wed, 6 Dec 2017 07:38:57 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:34728 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752151AbdLFMgL (ORCPT ); Wed, 6 Dec 2017 07:36:11 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7B57D169E; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4D95F3F7A6; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 5E54E1AE37CA; Wed, 6 Dec 2017 12:36:16 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 09/20] arm64: mm: Invalidate both kernel and user ASIDs when performing TLBI Date: Wed, 6 Dec 2017 12:35:28 +0000 Message-Id: <1512563739-25239-10-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Since an mm has both a kernel and a user ASID, we need to ensure that broadcast TLB maintenance targets both address spaces so that things like CoW continue to work with the uaccess primitives in the kernel. Reviewed-by: Mark Rutland Signed-off-by: Will Deacon --- arch/arm64/include/asm/tlbflush.h | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) -- 2.1.4 diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index af1c76981911..9e82dd79c7db 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -23,6 +23,7 @@ #include #include +#include /* * Raw TLBI operations. @@ -54,6 +55,11 @@ #define __tlbi(op, ...) __TLBI_N(op, ##__VA_ARGS__, 1, 0) +#define __tlbi_user(op, arg) do { \ + if (arm64_kernel_unmapped_at_el0()) \ + __tlbi(op, (arg) | USER_ASID_FLAG); \ +} while (0) + /* * TLB Management * ============== @@ -115,6 +121,7 @@ static inline void flush_tlb_mm(struct mm_struct *mm) dsb(ishst); __tlbi(aside1is, asid); + __tlbi_user(aside1is, asid); dsb(ish); } @@ -125,6 +132,7 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, dsb(ishst); __tlbi(vale1is, addr); + __tlbi_user(vale1is, addr); dsb(ish); } @@ -151,10 +159,13 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, dsb(ishst); for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12)) { - if (last_level) + if (last_level) { __tlbi(vale1is, addr); - else + __tlbi_user(vale1is, addr); + } else { __tlbi(vae1is, addr); + __tlbi_user(vae1is, addr); + } } dsb(ish); } @@ -194,6 +205,7 @@ static inline void __flush_tlb_pgtable(struct mm_struct *mm, unsigned long addr = uaddr >> 12 | (ASID(mm) << 48); __tlbi(vae1is, addr); + __tlbi_user(vae1is, addr); dsb(ish); } From patchwork Wed Dec 6 12:35:29 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120836 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7002126qgn; Wed, 6 Dec 2017 04:38:56 -0800 (PST) X-Google-Smtp-Source: AGs4zMZw1b2DabPiGhzEBpueGVMUKKe5Q+h38tv2m9/XMq1ve1GidI8n1uuska6hdzwQY2jHv3/+ X-Received: by 10.99.149.76 with SMTP id t12mr10003384pgn.382.1512563936068; Wed, 06 Dec 2017 04:38:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512563936; cv=none; d=google.com; s=arc-20160816; b=alBGccFxVa22/czjZGVQngNmYo7+pECeb1wcbs+WL1/pIXbdVr6sdLnp7pUIOfdA59 JnMe68/2rfN+rgEMXB0+k6H14LsKUmT/TrWJ8VRuVZ36sg3BI108MF/otjoQmywRpKUs 1/LjzJFtUYep7mWdjamTSNsiray3uqWIO73szgWyr484IxWmpIhMkxivmD8uAmeC6Plx bEV+SuQy1PIfozoDmaVUsUoomnK7JFy8y6q4X0jX1M3CJmpHcH4hCGs6U2KsnqZfiwto CB3AeIYYYYYx14ZZxo9hScbC5M+gCSGssNo1QEbLENK7uXS3yanqCisMOutpx9mMigLD AiqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=6mhciA1XoSLQntlqD5BQeKulxiBT7CjnFoQJAhxFrOo=; b=rntvl1EiqwQt4F2XzzbwFXGCbfo9mJDEUez41lVMUw49F4AHtPT2h/YbYygDg3iv/V p8vLVpOCrOzSvDptJ07lYtxWdTP+fR7Vo00dg6tajDNokN3Z0H/+HXdUR0yRV2475vOn 1TUAUCu/kyLFZcDDM9ImR2FXIbFqF/rgFs5A671MnjW7A27JTjim4rzK1PDhK1BXhDiM PDANMnfp/W7Ixz1Do+mhWtwm1iygOJDTQgPUZFs9cdxaf9pTz5MAVmcMnEvzrgNCr90t jYR4lysSESSdZpvLHatqTG4RhMTqeBfGVyqlWlIP6IBPyaVTWU++u3s2Gel6U6seiUtw FQGA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h5si1932853pls.722.2017.12.06.04.38.55; Wed, 06 Dec 2017 04:38:56 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752653AbdLFMix (ORCPT + 28 others); Wed, 6 Dec 2017 07:38:53 -0500 Received: from foss.arm.com ([217.140.101.70]:34730 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752155AbdLFMgL (ORCPT ); Wed, 6 Dec 2017 07:36:11 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 84382169F; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 55FD43F7C4; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 74B6D1AE37F8; Wed, 6 Dec 2017 12:36:16 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 10/20] arm64: entry: Add exception trampoline page for exceptions from EL0 Date: Wed, 6 Dec 2017 12:35:29 +0000 Message-Id: <1512563739-25239-11-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org To allow unmapping of the kernel whilst running at EL0, we need to point the exception vectors at an entry trampoline that can map/unmap the kernel on entry/exit respectively. This patch adds the trampoline page, although it is not yet plugged into the vector table and is therefore unused. Reviewed-by: Mark Rutland Signed-off-by: Will Deacon --- arch/arm64/kernel/entry.S | 86 +++++++++++++++++++++++++++++++++++++++++ arch/arm64/kernel/vmlinux.lds.S | 17 ++++++++ 2 files changed, 103 insertions(+) -- 2.1.4 diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index d454d8ed45e4..716b5ef42e29 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -28,6 +28,8 @@ #include #include #include +#include +#include #include #include #include @@ -943,6 +945,90 @@ __ni_sys_trace: .popsection // .entry.text +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 +/* + * Exception vectors trampoline. + */ + .pushsection ".entry.tramp.text", "ax" + + .macro tramp_map_kernel, tmp + mrs \tmp, ttbr1_el1 + sub \tmp, \tmp, #(SWAPPER_DIR_SIZE + RESERVED_TTBR0_SIZE) + bic \tmp, \tmp, #USER_ASID_FLAG + msr ttbr1_el1, \tmp + .endm + + .macro tramp_unmap_kernel, tmp + mrs \tmp, ttbr1_el1 + add \tmp, \tmp, #(SWAPPER_DIR_SIZE + RESERVED_TTBR0_SIZE) + orr \tmp, \tmp, #USER_ASID_FLAG + msr ttbr1_el1, \tmp + /* + * We avoid running the post_ttbr_update_workaround here because the + * user and kernel ASIDs don't have conflicting mappings, so any + * "blessing" as described in: + * + * http://lkml.kernel.org/r/56BB848A.6060603@caviumnetworks.com + * + * will not hurt correctness. Whilst this may partially defeat the + * point of using split ASIDs in the first place, it avoids + * the hit of invalidating the entire I-cache on every return to + * userspace. + */ + .endm + + .macro tramp_ventry, regsize = 64 + .align 7 +1: + .if \regsize == 64 + msr tpidrro_el0, x30 // Restored in kernel_ventry + .endif + tramp_map_kernel x30 + ldr x30, =vectors + prfm plil1strm, [x30, #(1b - tramp_vectors)] + msr vbar_el1, x30 + add x30, x30, #(1b - tramp_vectors) + isb + br x30 + .endm + + .macro tramp_exit, regsize = 64 + adr x30, tramp_vectors + msr vbar_el1, x30 + tramp_unmap_kernel x30 + .if \regsize == 64 + mrs x30, far_el1 + .endif + eret + .endm + + .align 11 +ENTRY(tramp_vectors) + .space 0x400 + + tramp_ventry + tramp_ventry + tramp_ventry + tramp_ventry + + tramp_ventry 32 + tramp_ventry 32 + tramp_ventry 32 + tramp_ventry 32 +END(tramp_vectors) + +ENTRY(tramp_exit_native) + tramp_exit +END(tramp_exit_native) + +ENTRY(tramp_exit_compat) + tramp_exit 32 +END(tramp_exit_compat) + + .ltorg + .popsection // .entry.tramp.text +#endif /* CONFIG_UNMAP_KERNEL_AT_EL0 */ + /* * Special system call wrappers. */ diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 7da3e5c366a0..6b4260f22aab 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -57,6 +57,17 @@ jiffies = jiffies_64; #define HIBERNATE_TEXT #endif +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 +#define TRAMP_TEXT \ + . = ALIGN(PAGE_SIZE); \ + VMLINUX_SYMBOL(__entry_tramp_text_start) = .; \ + *(.entry.tramp.text) \ + . = ALIGN(PAGE_SIZE); \ + VMLINUX_SYMBOL(__entry_tramp_text_end) = .; +#else +#define TRAMP_TEXT +#endif + /* * The size of the PE/COFF section that covers the kernel image, which * runs from stext to _edata, must be a round multiple of the PE/COFF @@ -113,6 +124,7 @@ SECTIONS HYPERVISOR_TEXT IDMAP_TEXT HIBERNATE_TEXT + TRAMP_TEXT *(.fixup) *(.gnu.warning) . = ALIGN(16); @@ -214,6 +226,11 @@ SECTIONS . += RESERVED_TTBR0_SIZE; #endif +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 + tramp_pg_dir = .; + . += PAGE_SIZE; +#endif + __pecoff_data_size = ABSOLUTE(. - __initdata_begin); _end = .; From patchwork Wed Dec 6 12:35:30 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120834 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7001961qgn; Wed, 6 Dec 2017 04:38:47 -0800 (PST) X-Google-Smtp-Source: AGs4zMaQ/TdAA7pnvR4E6bUbuFdmeYURnBje0YktMRHQ9Yy3zixA7FGM4wPfv10OfzttxCUiCe/1 X-Received: by 10.99.95.13 with SMTP id t13mr20692052pgb.448.1512563927264; Wed, 06 Dec 2017 04:38:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512563927; cv=none; d=google.com; s=arc-20160816; b=pl01iKSWpu8+dbPztYHoHG24lCDA6WOfhqS4ffMRCzSjKyN5m6ZuFTUojuKQIKHSVE uLKXCBSEtRBkd86jS52Y/TpTkzUCgTkCjJ4oYZTC8r0KzBLUKYrQ3aSUSpdIzWP4GzH4 Mg3BVacgEdqzA6bZxfnCyheCmNkPVgigHPES9cEpb/d1VayKv/+5Gm3FOxW+XOcu7bWz jatrF5dUOkovdHlsylf1ufHF10yCvYdgc3vMIPlRz4rylHe3aehEm2ROvR6N4Z/2mf3G e9MO7Bjz7+DHj5iy5pC4YX48WVjwVNOMSUaDE7k2/qVdzJqsrjcBwu+EY3ovUpwkgGBA GahQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=K3fiSOHHPNieVNUHblnbGKc3qxCMObM3xKfvPIZCgI8=; b=G4hP2KEpU5RHA9ZkJJp+bsKkCOAP9D6OKvSa40QES/pobZQcRhBV+CWkOYiOuw91Qv 8JJxG2pIzPBJZPN9HxvIgUCkQmspl0VQnLtGJYbFArPtialn1vn+jZWq0ECTUjem2D5K u3aL6lGtqSssjIOMpucvAla7if01x+qjtgoL/R6b+aDrHNeXufNAWMKV4bGiWdAjiqB4 YWNGFYbSqSXl3seuwj1fq2upiHw2iRu1PpFAZHUgfbV4ki44FOm2q2uDZ+RRDZEL0pwP HhM+zAD8MszAxUwSpbYNQ4gyXPEo66UtD96HE+O8jG8y3h0CQIZHxUte1HotNJlmMilP OWLg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 89si1949624plf.200.2017.12.06.04.38.47; Wed, 06 Dec 2017 04:38:47 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752624AbdLFMip (ORCPT + 28 others); Wed, 6 Dec 2017 07:38:45 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:34784 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752165AbdLFMgM (ORCPT ); Wed, 6 Dec 2017 07:36:12 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 91EDF16A0; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 5EEA93F236; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 8B1D91AE382D; Wed, 6 Dec 2017 12:36:16 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 11/20] arm64: mm: Map entry trampoline into trampoline and kernel page tables Date: Wed, 6 Dec 2017 12:35:30 +0000 Message-Id: <1512563739-25239-12-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The exception entry trampoline needs to be mapped at the same virtual address in both the trampoline page table (which maps nothing else) and also the kernel page table, so that we can swizzle TTBR1_EL1 on exceptions from and return to EL0. This patch maps the trampoline at a fixed virtual address in the fixmap area of the kernel virtual address space, which allows the kernel proper to be randomized with respect to the trampoline when KASLR is enabled. Signed-off-by: Will Deacon --- arch/arm64/include/asm/fixmap.h | 4 ++++ arch/arm64/include/asm/pgtable.h | 1 + arch/arm64/kernel/asm-offsets.c | 6 +++++- arch/arm64/mm/mmu.c | 23 +++++++++++++++++++++++ 4 files changed, 33 insertions(+), 1 deletion(-) -- 2.1.4 Reviewed-by: Mark Rutland diff --git a/arch/arm64/include/asm/fixmap.h b/arch/arm64/include/asm/fixmap.h index 4052ec39e8db..8119b49be98d 100644 --- a/arch/arm64/include/asm/fixmap.h +++ b/arch/arm64/include/asm/fixmap.h @@ -58,6 +58,10 @@ enum fixed_addresses { FIX_APEI_GHES_NMI, #endif /* CONFIG_ACPI_APEI_GHES */ +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 + FIX_ENTRY_TRAMP_TEXT, +#define TRAMP_VALIAS (__fix_to_virt(FIX_ENTRY_TRAMP_TEXT)) +#endif /* CONFIG_UNMAP_KERNEL_AT_EL0 */ __end_of_permanent_fixed_addresses, /* diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 149d05fb9421..774003b247ad 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -680,6 +680,7 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, extern pgd_t swapper_pg_dir[PTRS_PER_PGD]; extern pgd_t idmap_pg_dir[PTRS_PER_PGD]; +extern pgd_t tramp_pg_dir[PTRS_PER_PGD]; /* * Encode and decode a swap entry: diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 71bf088f1e4b..af247d10252f 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -148,11 +149,14 @@ int main(void) DEFINE(ARM_SMCCC_RES_X2_OFFS, offsetof(struct arm_smccc_res, a2)); DEFINE(ARM_SMCCC_QUIRK_ID_OFFS, offsetof(struct arm_smccc_quirk, id)); DEFINE(ARM_SMCCC_QUIRK_STATE_OFFS, offsetof(struct arm_smccc_quirk, state)); - BLANK(); DEFINE(HIBERN_PBE_ORIG, offsetof(struct pbe, orig_address)); DEFINE(HIBERN_PBE_ADDR, offsetof(struct pbe, address)); DEFINE(HIBERN_PBE_NEXT, offsetof(struct pbe, next)); DEFINE(ARM64_FTR_SYSVAL, offsetof(struct arm64_ftr_reg, sys_val)); + BLANK(); +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 + DEFINE(TRAMP_VALIAS, TRAMP_VALIAS); +#endif return 0; } diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 267d2b79d52d..fe68a48c64cb 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -525,6 +525,29 @@ static int __init parse_rodata(char *arg) } early_param("rodata", parse_rodata); +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 +static int __init map_entry_trampoline(void) +{ + extern char __entry_tramp_text_start[]; + + pgprot_t prot = rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC; + phys_addr_t pa_start = __pa_symbol(__entry_tramp_text_start); + + /* The trampoline is always mapped and can therefore be global */ + pgprot_val(prot) &= ~PTE_NG; + + /* Map only the text into the trampoline page table */ + memset(tramp_pg_dir, 0, PGD_SIZE); + __create_pgd_mapping(tramp_pg_dir, pa_start, TRAMP_VALIAS, PAGE_SIZE, + prot, pgd_pgtable_alloc, 0); + + /* ...as well as the kernel page table */ + __set_fixmap(FIX_ENTRY_TRAMP_TEXT, pa_start, prot); + return 0; +} +core_initcall(map_entry_trampoline); +#endif + /* * Create fine-grained mappings for the kernel. */ From patchwork Wed Dec 6 12:35:31 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120833 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7001896qgn; Wed, 6 Dec 2017 04:38:43 -0800 (PST) X-Google-Smtp-Source: AGs4zMa1h3Eb4UUtjaf1KtnkA4SZo4SGg121IznVn3eN2mD4FJYSMTBufC3C0U0bt5PEO1sTbXB2 X-Received: by 10.101.96.74 with SMTP id b10mr20642448pgv.155.1512563923046; Wed, 06 Dec 2017 04:38:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512563923; cv=none; d=google.com; s=arc-20160816; b=Y51N2Jxv8sgnGMKzKxaz3T/+j/mnH8lUwFNxEM8IcIvroWVHrgRibicJPx67vyx1+G aGEKjbp9RScW43gRVxtTbasZvKWB4WQqza1cv0yNoIJQMdF4Qp1lPBZGvME6f8cCMpVo lTQfnLjLgkf4MZbVGiWa1rWziLqbkhCTgdddKcZ9tFbcE/86kOu3wtX6GyMr856JOmsO LiIWLJ8R19Z6597YXAzva2zvDc4p+49ppdo5LcxuYeqWWERmWiNJsD6CQTdp6JvGlObB otJ3sLmSUeFhVnC8yHRux+yU1nkAx2rCdRTlkZPYhuS3WkTTwX5BZx9Dh6Gu7wZhb/NY CAZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=IRQ45YUsWv7W+pCi24am1sTzJNtzkF71bnIeBsXf+pg=; b=uE5gAhCzkrEen3ASmc7275oAfjVJMDw4zrm3q0/OzPxFGggg8T+TDE+sA6191XMN9k OP+7GGeVjvpTU+vla/WE/Sjp6e/dZaHeEcWuFwbaDfdiYOEMp5bpQwpqxUGZdxBucixl ShCVlWpDoAOZpSBTW3H8t5tNMGRbGygYlxlit+IGs5cdRHTKqD4qBQEJzu4Idvvb8/Ff pD+z/cMxhqPg1/Z2ZeKXkbUj4GnoRnYKPK62eB2dX9EjK0kcWuzm/chsRrjNgmS2nDYq lzTVGEiuN03NGZcpIacY+om1YGf4cW56yU6w6wBYhUIXzeFfch388ts+riN5H8lSxqja so1w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e10si1803731pgo.537.2017.12.06.04.38.42; Wed, 06 Dec 2017 04:38:43 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752422AbdLFMik (ORCPT + 28 others); Wed, 6 Dec 2017 07:38:40 -0500 Received: from foss.arm.com ([217.140.101.70]:34802 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752166AbdLFMgM (ORCPT ); Wed, 6 Dec 2017 07:36:12 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 979C416A3; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 67D4C3F53E; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 9F7871AE385E; Wed, 6 Dec 2017 12:36:16 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 12/20] arm64: entry: Explicitly pass exception level to kernel_ventry macro Date: Wed, 6 Dec 2017 12:35:31 +0000 Message-Id: <1512563739-25239-13-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org We will need to treat exceptions from EL0 differently in kernel_ventry, so rework the macro to take the exception level as an argument and construct the branch target using that. Reviewed-by: Mark Rutland Signed-off-by: Will Deacon --- arch/arm64/kernel/entry.S | 46 +++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 23 deletions(-) -- 2.1.4 diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 716b5ef42e29..b99fc928119c 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -71,7 +71,7 @@ #define BAD_FIQ 2 #define BAD_ERROR 3 - .macro kernel_ventry label + .macro kernel_ventry, el, label, regsize = 64 .align 7 sub sp, sp, #S_FRAME_SIZE #ifdef CONFIG_VMAP_STACK @@ -84,7 +84,7 @@ tbnz x0, #THREAD_SHIFT, 0f sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x0 sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp - b \label + b el\()\el\()_\label 0: /* @@ -116,7 +116,7 @@ sub sp, sp, x0 mrs x0, tpidrro_el0 #endif - b \label + b el\()\el\()_\label .endm .macro kernel_entry, el, regsize = 64 @@ -369,31 +369,31 @@ tsk .req x28 // current thread_info .align 11 ENTRY(vectors) - kernel_ventry el1_sync_invalid // Synchronous EL1t - kernel_ventry el1_irq_invalid // IRQ EL1t - kernel_ventry el1_fiq_invalid // FIQ EL1t - kernel_ventry el1_error_invalid // Error EL1t + kernel_ventry 1, sync_invalid // Synchronous EL1t + kernel_ventry 1, irq_invalid // IRQ EL1t + kernel_ventry 1, fiq_invalid // FIQ EL1t + kernel_ventry 1, error_invalid // Error EL1t - kernel_ventry el1_sync // Synchronous EL1h - kernel_ventry el1_irq // IRQ EL1h - kernel_ventry el1_fiq_invalid // FIQ EL1h - kernel_ventry el1_error // Error EL1h + kernel_ventry 1, sync // Synchronous EL1h + kernel_ventry 1, irq // IRQ EL1h + kernel_ventry 1, fiq_invalid // FIQ EL1h + kernel_ventry 1, error // Error EL1h - kernel_ventry el0_sync // Synchronous 64-bit EL0 - kernel_ventry el0_irq // IRQ 64-bit EL0 - kernel_ventry el0_fiq_invalid // FIQ 64-bit EL0 - kernel_ventry el0_error // Error 64-bit EL0 + kernel_ventry 0, sync // Synchronous 64-bit EL0 + kernel_ventry 0, irq // IRQ 64-bit EL0 + kernel_ventry 0, fiq_invalid // FIQ 64-bit EL0 + kernel_ventry 0, error // Error 64-bit EL0 #ifdef CONFIG_COMPAT - kernel_ventry el0_sync_compat // Synchronous 32-bit EL0 - kernel_ventry el0_irq_compat // IRQ 32-bit EL0 - kernel_ventry el0_fiq_invalid_compat // FIQ 32-bit EL0 - kernel_ventry el0_error_compat // Error 32-bit EL0 + kernel_ventry 0, sync_compat, 32 // Synchronous 32-bit EL0 + kernel_ventry 0, irq_compat, 32 // IRQ 32-bit EL0 + kernel_ventry 0, fiq_invalid_compat, 32 // FIQ 32-bit EL0 + kernel_ventry 0, error_compat, 32 // Error 32-bit EL0 #else - kernel_ventry el0_sync_invalid // Synchronous 32-bit EL0 - kernel_ventry el0_irq_invalid // IRQ 32-bit EL0 - kernel_ventry el0_fiq_invalid // FIQ 32-bit EL0 - kernel_ventry el0_error_invalid // Error 32-bit EL0 + kernel_ventry 0, sync_invalid, 32 // Synchronous 32-bit EL0 + kernel_ventry 0, irq_invalid, 32 // IRQ 32-bit EL0 + kernel_ventry 0, fiq_invalid, 32 // FIQ 32-bit EL0 + kernel_ventry 0, error_invalid, 32 // Error 32-bit EL0 #endif END(vectors) From patchwork Wed Dec 6 12:35:32 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120835 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7002003qgn; Wed, 6 Dec 2017 04:38:49 -0800 (PST) X-Google-Smtp-Source: AGs4zMbOwasM90X6QPJ0sFDC9DjE3bYRNZSN/B8n0uEpju1XZaYEX3NUgLReAtwd+O8HmXp+gVXj X-Received: by 10.84.131.68 with SMTP id 62mr6673764pld.185.1512563929370; Wed, 06 Dec 2017 04:38:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512563929; cv=none; d=google.com; s=arc-20160816; b=whQBZu2J9zp95Zcp/EXBhp7vLDlZe0772uI9vUCNu9KPbU4QHOQGIJjH7qaH7QUTPV 9bzlZsDWJYIH1+KPGmT+qBTUmkFAxHGR0/47tsossI+TatX4lMXDjGP+GSGW96UEiPt5 hVDgrIZwI2++WO8ugPRzRxA7GQcoGgD8W37trMUXmUrva5ABhjHcicQWMHY5iSJDvp3p 9EouT1Lj/WfoxMsr6Nvq9ia2lWFd67D5fZI0BRV4zvVErLm01CD1/QwWsQna3kWX+qJ6 8Xm4SH+CZlru2gwgS8cYfoRnAtvqJMlq1xUFl2SS4UbBDnENUwTqFTt3AdhKPc69TuOf Z3Vw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=c4AjNLTQvdi08vPHUO6iM1ed2QLNFvT2kvw7HDZFCfc=; b=wKTsC44kb0LYcEdefx0jRq8fjR9EFzEfhp4GmB2e8ge0KpcZhgd2r9bYvh6EC/UCg/ PKO0i0OIPWg11Pbd6QjJNzJdwQk4jGVpTaOZRltXxeddSXOwn3WimJYEzgpXnWHMruqY Q6/1pE9Z7T30dlZVvhnW/IC4ImrKjDjEvJFLcvNqMa31RwBb+5gbCCUGPvwXRXEbCgPg GKJT+RjfelpNDdPLzsb4kR3e/0kmPD5c6QMBnrfmJlrFUuR1xz40LrZp2/LOvunrGE+Q pq1N7HXUjkvNPwo1B25bXeMEMyvx5cN5vaduf3ycMFOhVOSxDpCcMfQALlH/NxR5Q0cP GgmQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 89si1949624plf.200.2017.12.06.04.38.48; Wed, 06 Dec 2017 04:38:49 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752642AbdLFMir (ORCPT + 28 others); Wed, 6 Dec 2017 07:38:47 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:34782 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752162AbdLFMgM (ORCPT ); Wed, 6 Dec 2017 07:36:12 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9F11C16BA; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 70C8E3F7C5; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id B3D5B1AE38AD; Wed, 6 Dec 2017 12:36:16 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 13/20] arm64: entry: Hook up entry trampoline to exception vectors Date: Wed, 6 Dec 2017 12:35:32 +0000 Message-Id: <1512563739-25239-14-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hook up the entry trampoline to our exception vectors so that all exceptions from and returns to EL0 go via the trampoline, which swizzles the vector base register accordingly. Transitioning to and from the kernel clobbers x30, so we use tpidrro_el0 and far_el1 as scratch registers for native tasks. Reviewed-by: Mark Rutland Signed-off-by: Will Deacon --- arch/arm64/kernel/entry.S | 39 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-) -- 2.1.4 diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index b99fc928119c..39e3873b8d5a 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -73,6 +73,17 @@ .macro kernel_ventry, el, label, regsize = 64 .align 7 +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 + .if \el == 0 + .if \regsize == 64 + mrs x30, tpidrro_el0 + msr tpidrro_el0, xzr + .else + mov x30, xzr + .endif + .endif +#endif + sub sp, sp, #S_FRAME_SIZE #ifdef CONFIG_VMAP_STACK /* @@ -119,6 +130,11 @@ b el\()\el\()_\label .endm + .macro tramp_alias, dst, sym + mov_q \dst, TRAMP_VALIAS + add \dst, \dst, #(\sym - .entry.tramp.text) + .endm + .macro kernel_entry, el, regsize = 64 .if \regsize == 32 mov w0, w0 // zero upper 32 bits of x0 @@ -271,18 +287,20 @@ alternative_else_nop_endif .if \el == 0 ldr x23, [sp, #S_SP] // load return stack pointer msr sp_el0, x23 + tst x22, #PSR_MODE32_BIT // native task? + b.eq 3f + #ifdef CONFIG_ARM64_ERRATUM_845719 alternative_if ARM64_WORKAROUND_845719 - tbz x22, #4, 1f #ifdef CONFIG_PID_IN_CONTEXTIDR mrs x29, contextidr_el1 msr contextidr_el1, x29 #else msr contextidr_el1, xzr #endif -1: alternative_else_nop_endif #endif +3: .endif msr elr_el1, x21 // set up the return data @@ -304,7 +322,22 @@ alternative_else_nop_endif ldp x28, x29, [sp, #16 * 14] ldr lr, [sp, #S_LR] add sp, sp, #S_FRAME_SIZE // restore sp - eret // return to kernel + +#ifndef CONFIG_UNMAP_KERNEL_AT_EL0 + eret +#else + .if \el == 0 + bne 4f + msr far_el1, x30 + tramp_alias x30, tramp_exit_native + br x30 +4: + tramp_alias x30, tramp_exit_compat + br x30 + .else + eret + .endif +#endif .endm .macro irq_stack_entry From patchwork Wed Dec 6 12:35:33 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120832 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7001871qgn; Wed, 6 Dec 2017 04:38:40 -0800 (PST) X-Google-Smtp-Source: AGs4zMZJA+qfmU09utuM/yko1L7SP28OdKXtYk/k7gq4cdyovM8KAyUdaH+CeM46kIFdnMg1Fq2f X-Received: by 10.98.150.26 with SMTP id c26mr2670961pfe.224.1512563920528; Wed, 06 Dec 2017 04:38:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512563920; cv=none; d=google.com; s=arc-20160816; b=VyveP6n+fBzHgCUifKqeRq9BjiAm295waGfWGXsz9fdhFYE9m+eFj7B2B6R84limqG FbDp7HQkpZ+VrWGvpcvxfkh4IYG7URa8cY/FobfWTBDtPFjs1lINNu2R8ItI2WUUzoD7 cOn0NA0eq1bZjR6hvbxAECgNy12o5P3VIFA+ED4NkJCPm2vPYus5yuzf5GAEjmocMbkM 4KtA3H0zjq2NFRFI/PtIzBv+99wrCvIa8BqgrMRpK23hm2dRTXf72xNLA2PfLWFrLaqQ Tkbl57lrJMczPTLlzenYJVsOxxhNlshWeJ40n5pIhJh39obCRkZoLOI9mbR5vB9Vfvuo vuHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=53iazDMkYmHFGPow82x5uWrLUNt4geEQkPni+qrZJrM=; b=JATfNQKfdIq8Oi2F9B0oQ4lqVoebTUkKA5bpBUWiJ/2y0a7BHBfOltA6iF6qJomJzK e13hFJFBls0UzP9wr+tdFz34gaAbGBzd1XGOTvYzrYyb+5HonKdL3QTvJeVl13/VLBLs FTpFQTI1BUKmlsbkWfHZ+wpRgHr5nMAggSDB3K/05ou+U6C0qdVt+Rh57kjPzh0B4m84 rjaKhMwmvRenpK5mcSr1GB7F0YF3u1zBea1rCLXEjXtXdXcv6Na1uTTlU5vA+NDXY0tB VeaewQi4G3Qt7vaQRtOoqJOd4MXRZ6jQ8qdJ6qrGzDGlODI5olL+wTkY7BFefzkG+yhs SVEA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e10si1803731pgo.537.2017.12.06.04.38.40; Wed, 06 Dec 2017 04:38:40 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752361AbdLFMii (ORCPT + 28 others); Wed, 6 Dec 2017 07:38:38 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:34804 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752172AbdLFMgM (ORCPT ); Wed, 6 Dec 2017 07:36:12 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A809C16EA; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 79C153F5BD; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id C83111AE38BF; Wed, 6 Dec 2017 12:36:16 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 14/20] arm64: erratum: Work around Falkor erratum #E1003 in trampoline code Date: Wed, 6 Dec 2017 12:35:33 +0000 Message-Id: <1512563739-25239-15-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org We rely on an atomic swizzling of TTBR1 when transitioning from the entry trampoline to the kernel proper on an exception. We can't rely on this atomicity in the face of Falkor erratum #E1003, so on affected cores we can issue a TLB invalidation to invalidate the walk cache prior to jumping into the kernel. There is still the possibility of a TLB conflict here due to conflicting walk cache entries prior to the invalidation, but this doesn't appear to be the case on these CPUs in practice. Reviewed-by: Mark Rutland Signed-off-by: Will Deacon --- arch/arm64/Kconfig | 17 +++++------------ arch/arm64/kernel/entry.S | 12 ++++++++++++ 2 files changed, 17 insertions(+), 12 deletions(-) -- 2.1.4 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a93339f5178f..fdcc7b9bb15d 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -522,20 +522,13 @@ config CAVIUM_ERRATUM_30115 config QCOM_FALKOR_ERRATUM_1003 bool "Falkor E1003: Incorrect translation due to ASID change" default y - select ARM64_PAN if ARM64_SW_TTBR0_PAN help On Falkor v1, an incorrect ASID may be cached in the TLB when ASID - and BADDR are changed together in TTBRx_EL1. The workaround for this - issue is to use a reserved ASID in cpu_do_switch_mm() before - switching to the new ASID. Saying Y here selects ARM64_PAN if - ARM64_SW_TTBR0_PAN is selected. This is done because implementing and - maintaining the E1003 workaround in the software PAN emulation code - would be an unnecessary complication. The affected Falkor v1 CPU - implements ARMv8.1 hardware PAN support and using hardware PAN - support versus software PAN emulation is mutually exclusive at - runtime. - - If unsure, say Y. + and BADDR are changed together in TTBRx_EL1. Since we keep the ASID + in TTBR1_EL1, this situation only occurs in the entry trampoline and + then only for entries in the walk cache, since the leaf translation + is unchanged. Work around the erratum by invalidating the walk cache + entries for the trampoline before entering the kernel proper. config QCOM_FALKOR_ERRATUM_1009 bool "Falkor E1009: Prematurely complete a DSB after a TLBI" diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 39e3873b8d5a..ce56592b5f70 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -989,6 +989,18 @@ __ni_sys_trace: sub \tmp, \tmp, #(SWAPPER_DIR_SIZE + RESERVED_TTBR0_SIZE) bic \tmp, \tmp, #USER_ASID_FLAG msr ttbr1_el1, \tmp +#ifdef CONFIG_QCOM_FALKOR_ERRATUM_1003 +alternative_if ARM64_WORKAROUND_QCOM_FALKOR_E1003 + /* ASID already in \tmp[63:48] */ + movk \tmp, #:abs_g2_nc:(TRAMP_VALIAS >> 12) + movk \tmp, #:abs_g1_nc:(TRAMP_VALIAS >> 12) + /* 2MB boundary containing the vectors, so we nobble the walk cache */ + movk \tmp, #:abs_g0_nc:((TRAMP_VALIAS & ~(SZ_2M - 1)) >> 12) + isb + tlbi vae1, \tmp + dsb nsh +alternative_else_nop_endif +#endif /* CONFIG_QCOM_FALKOR_ERRATUM_1003 */ .endm .macro tramp_unmap_kernel, tmp From patchwork Wed Dec 6 12:35:34 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120831 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7001801qgn; Wed, 6 Dec 2017 04:38:36 -0800 (PST) X-Google-Smtp-Source: AGs4zMbnzuDc3YURcw8SVfyP4JW8d7yjItFUVBT2+fGVSmqwz0TNEG0/+KGHtbEFTOY6q0X7TGOf X-Received: by 10.99.109.73 with SMTP id i70mr5810859pgc.134.1512563916539; Wed, 06 Dec 2017 04:38:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512563916; cv=none; d=google.com; s=arc-20160816; b=Wlhdick5L4ydQ44dwXPa34Yhc9DCvYNB9XpchjnmR1j5INJnSphTsdaX1A31Xxg8Je TQFU84LRQYPJ/JDQejW/tFrE1iwPgg42Yc2XCQVVQx1rXDtCScxs3q9ZodQB0nsg+Iq9 c0PWmgp5g393zvx1fZl2f3n26RSRh0uNNnKyRAwvRp9EmFz3eXoA5N30yiCm7guoh2le sonYM4OFR38S7Z145XN3o/Iq1HxlGeUb0X9inXjMt265inAL97JDUcf+jKkJ3czUAxP+ 3zEXpiegWDwpbFzTEhgRyaogO+V0TuAieQtl+V4gz3rHdCyjA/ZJBTkjM2bPmTjy/aYF 6sNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=d6TeP91CE/95IFaqza+mFJrdaUY+71l0IehiCKcx9w8=; b=aVy/XqAAUItGXqPY0e7RrJBshSiJdzE4USTyAziRSOZGtUhm5ZxsRakvGoSSO+M3uj C88dz586v6JNMJgsBUZgTtluTRiUBM7EiWkgq3wcQ3NzXrn/jBcgX+wgEgqYs/VH4+qZ vq2DfNfK9PEQoFYCZkdfz9kiw06R3Y13lHc6LLylzn8LZOCEvHb/GZBzce6/Tt8xVew7 Q3/L9B4v/iCEx5qnoqiBYtMyWLOm2yvkVdUAn7/oXDAe8pvNKk2s6rMFduk8d8jxkXyd o121gZjyNN+/tNOdkN3bh2eKRqIN9WSuixiW4LoMzFTPlm/cj2gejH4BGciO0KG/0m7C Up/Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id s59si1910014plb.232.2017.12.06.04.38.36; Wed, 06 Dec 2017 04:38:36 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752298AbdLFMif (ORCPT + 28 others); Wed, 6 Dec 2017 07:38:35 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:34686 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752177AbdLFMgM (ORCPT ); Wed, 6 Dec 2017 07:36:12 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C34781713; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 952BB3F627; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id DF1D31AE3564; Wed, 6 Dec 2017 12:36:16 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 15/20] arm64: tls: Avoid unconditional zeroing of tpidrro_el0 for native tasks Date: Wed, 6 Dec 2017 12:35:34 +0000 Message-Id: <1512563739-25239-16-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When unmapping the kernel at EL0, we use tpidrro_el0 as a scratch register during exception entry from native tasks and subsequently zero it in the kernel_ventry macro. We can therefore avoid zeroing tpidrro_el0 in the context-switch path for native tasks using the entry trampoline. Reviewed-by: Mark Rutland Signed-off-by: Will Deacon --- arch/arm64/kernel/process.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) -- 2.1.4 diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index b2adcce7bc18..aba3a1fb492d 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -361,16 +361,14 @@ void tls_preserve_current_state(void) static void tls_thread_switch(struct task_struct *next) { - unsigned long tpidr, tpidrro; - tls_preserve_current_state(); - tpidr = *task_user_tls(next); - tpidrro = is_compat_thread(task_thread_info(next)) ? - next->thread.tp_value : 0; + if (is_compat_thread(task_thread_info(next))) + write_sysreg(next->thread.tp_value, tpidrro_el0); + else if (!arm64_kernel_unmapped_at_el0()) + write_sysreg(0, tpidrro_el0); - write_sysreg(tpidr, tpidr_el0); - write_sysreg(tpidrro, tpidrro_el0); + write_sysreg(*task_user_tls(next), tpidr_el0); } /* Restore the UAO state depending on next's addr_limit */ From patchwork Wed Dec 6 12:35:35 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120827 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7000867qgn; Wed, 6 Dec 2017 04:37:41 -0800 (PST) X-Google-Smtp-Source: AGs4zMaMjCfxCXoeeqSn+g0T1P0dq001U5m8Tg1oLN5KuWQJuV78iHuz79JaMRv6m+wAbSThz9lE X-Received: by 10.99.124.85 with SMTP id l21mr21008045pgn.85.1512563861761; Wed, 06 Dec 2017 04:37:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512563861; cv=none; d=google.com; s=arc-20160816; b=w3UO11QBQAI9z+E8wRmi/qSXpylaKZaue3FDt9AmIoh8+FUOpdP9BgIUznuj7Nl7oQ u6adVSR/A9zwPl1xljnXbfENI7pDoAbSaHtrStWt0hndOGLZQ2w0cRO7Hv00ZN2Jqbt7 hx0C2xNBVIz3TLGgzO6gJKyxgY4i2O+bcdFEZ/f2lPuLkKhdYq52zxNmymjMdhJCb2gd Ek6/DnerXFfBlC8yZIpqOAqQRWmD/bWUl5DVYIgzvvL6xM+Gew1qH3si8hIjBM+YbJ+z JqOIC5fIABLD9jzINJtCsEDXxXxJscu9J4SFKzeBqXIfCP9WKng/vjTGW3ejjDTYngkX nt0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=kP3a5nQSi1U6cgvHB69oVr/7HJTt8Vr1kigjMu0NVCY=; b=rU8SSN4yh+DsTBZQPd+rm3aZWgAYlIK4jjfo5UZnGy7DDkjALKNnpIP0KKkF48rvhw svzCTTjlGDFFf4lFztZTNLSgRd/f+TtLP7+6X203573XUUlWnPqy0XKmXkwk/n2NwGBB jmpPkvfMeMk2zjru1C8bJmbfOQkqfmBVRLbdwLDiSaiwhTqyBrO9YvL2zLpnMntlrX28 Q5FOR/e3+A2AUfJLfBZ3jNSPA1hrmk8sA7KWW+KW1OXQvPCwHvG7bIG4fm16ndCYJcKL Ekd08PlpKFZ8RtmOE5Pds8R8xw9NrJf2Dx3ffmoi+oHoY/a7qlP2EXyrr88nGP45TAqb +2Mg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w3si1841951pge.245.2017.12.06.04.37.41; Wed, 06 Dec 2017 04:37:41 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752570AbdLFMhj (ORCPT + 28 others); Wed, 6 Dec 2017 07:37:39 -0500 Received: from foss.arm.com ([217.140.101.70]:34704 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752209AbdLFMgN (ORCPT ); Wed, 6 Dec 2017 07:36:13 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 44E1419F6; Wed, 6 Dec 2017 04:36:12 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 165673F236; Wed, 6 Dec 2017 04:36:12 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id EEDD51AE3546; Wed, 6 Dec 2017 12:36:16 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 16/20] arm64: entry: Add fake CPU feature for unmapping the kernel at EL0 Date: Wed, 6 Dec 2017 12:35:35 +0000 Message-Id: <1512563739-25239-17-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Allow explicit disabling of the entry trampoline on the kernel command line (kpti=off) by adding a fake CPU feature (ARM64_UNMAP_KERNEL_AT_EL0) that can be used to toggle the alternative sequences in our entry code and avoid use of the trampoline altogether if desired. This also allows us to make use of a static key in arm64_kernel_unmapped_at_el0(). Signed-off-by: Will Deacon --- arch/arm64/include/asm/cpucaps.h | 3 ++- arch/arm64/include/asm/mmu.h | 3 ++- arch/arm64/kernel/cpufeature.c | 41 ++++++++++++++++++++++++++++++++++++++++ arch/arm64/kernel/entry.S | 9 +++++---- 4 files changed, 50 insertions(+), 6 deletions(-) -- 2.1.4 Reviewed-by: Mark Rutland diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index 2ff7c5e8efab..b4537ffd1018 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -41,7 +41,8 @@ #define ARM64_WORKAROUND_CAVIUM_30115 20 #define ARM64_HAS_DCPOP 21 #define ARM64_SVE 22 +#define ARM64_UNMAP_KERNEL_AT_EL0 23 -#define ARM64_NCAPS 23 +#define ARM64_NCAPS 24 #endif /* __ASM_CPUCAPS_H */ diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index c07954638658..da6f12e40714 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -36,7 +36,8 @@ typedef struct { static inline bool arm64_kernel_unmapped_at_el0(void) { - return IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0); + return IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0) && + cpus_have_const_cap(ARM64_UNMAP_KERNEL_AT_EL0); } extern void paging_init(void); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index c5ba0097887f..98e6563015a4 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -845,6 +845,40 @@ static bool has_no_fpsimd(const struct arm64_cpu_capabilities *entry, int __unus ID_AA64PFR0_FP_SHIFT) < 0; } +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 +static int __kpti_forced; /* 0: not forced, >0: forced on, <0: forced off */ + +static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, + int __unused) +{ + /* Forced on command line? */ + if (__kpti_forced) { + pr_info("kernel page table isolation forced %s by command line option\n", + __kpti_forced > 0 ? "ON" : "OFF"); + return __kpti_forced > 0; + } + + /* Useful for KASLR robustness */ + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) + return true; + + return false; +} + +static int __init parse_kpti(char *str) +{ + bool enabled; + int ret = strtobool(str, &enabled); + + if (ret) + return ret; + + __kpti_forced = enabled ? 1 : -1; + return 0; +} +__setup("kpti=", parse_kpti); +#endif /* CONFIG_UNMAP_KERNEL_AT_EL0 */ + static const struct arm64_cpu_capabilities arm64_features[] = { { .desc = "GIC system register CPU interface", @@ -931,6 +965,13 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .def_scope = SCOPE_SYSTEM, .matches = hyp_offset_low, }, +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 + { + .capability = ARM64_UNMAP_KERNEL_AT_EL0, + .def_scope = SCOPE_SYSTEM, + .matches = unmap_kernel_at_el0, + }, +#endif { /* FP/SIMD is not implemented */ .capability = ARM64_HAS_NO_FPSIMD, diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index ce56592b5f70..5d51bdbb2131 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -74,6 +74,7 @@ .macro kernel_ventry, el, label, regsize = 64 .align 7 #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 +alternative_if ARM64_UNMAP_KERNEL_AT_EL0 .if \el == 0 .if \regsize == 64 mrs x30, tpidrro_el0 @@ -82,6 +83,7 @@ mov x30, xzr .endif .endif +alternative_else_nop_endif #endif sub sp, sp, #S_FRAME_SIZE @@ -323,10 +325,9 @@ alternative_else_nop_endif ldr lr, [sp, #S_LR] add sp, sp, #S_FRAME_SIZE // restore sp -#ifndef CONFIG_UNMAP_KERNEL_AT_EL0 - eret -#else .if \el == 0 +alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 bne 4f msr far_el1, x30 tramp_alias x30, tramp_exit_native @@ -334,10 +335,10 @@ alternative_else_nop_endif 4: tramp_alias x30, tramp_exit_compat br x30 +#endif .else eret .endif -#endif .endm .macro irq_stack_entry From patchwork Wed Dec 6 12:35:36 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120830 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7001052qgn; Wed, 6 Dec 2017 04:37:52 -0800 (PST) X-Google-Smtp-Source: AGs4zMZQZlMRKOQM5TlBvG0WXew1nEn4Gki3IKWycWiz1qQdUA7zh+6ATfpE+eeKHQmUtg/xsY4I X-Received: by 10.101.100.199 with SMTP id t7mr21687960pgv.316.1512563871946; Wed, 06 Dec 2017 04:37:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512563871; cv=none; d=google.com; s=arc-20160816; b=qBApizCNKvaamG2OgOcsfn7QwYo88N5/LZw7siW75M3ioysnSRpDPvXk83aNTi1nKv kLmRbifXH4vIOR4SFv/iQ26yDxhjcQ7ieJIAGAybTF+F7gDJmoCUdwskAsbWFnM7bXly DXajAbgHd4GeIOVsMNiJztRJygJaEbaI3QaEoDE7QU9oTjvWLzStMryUlqi4/uTkj50+ D2kWMb25aZLXi3JUABT5ana2q1hpHfqX1xATj32EkQFTTbE5H3iWwIMwXORxFTt5XN6l TbaiyIDZObfllTrrjCAl+b+ZIW1muO9VzCmKbhiUs/F1bzVhWafT7/iA/jqJGWYfU+EO En6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=alp05x0m5Mlc2GcL6RyLGvmSzq+O1tmXi7Fc/XnMFog=; b=g3+CNMNYJC4ulfNNvQaT7QnuClhgscjkB99gXo/Kt4RvhuCusLc1Zsppu/Gqw6v/p2 Tif/iXqfKCBif0Rp5fSE4S+0p9G9lYNgFn/S6ZHPg8g+Zwk+FanKja1sHr9htPRaF+L5 LCdQ4wgAdEgGv044TYZG0roeME+NDJpriIyNUB3+T5IvKAXbQgyKv+PEkY6/kmKNAJPM 7+k9nOPV77WPRMRvNVvtlYJUZjZpKyqGH1SQdLpd+YHVM5h+Ew49nFUV2RpeqsgRMBs4 zl9njNPCkQO8h6p9nDe8+O842B4IGfr0Sk5mPlE+wTDDIh1OVG997eeoIZQZALLJco0B BLMw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id s59si1910014plb.232.2017.12.06.04.37.51; Wed, 06 Dec 2017 04:37:51 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752624AbdLFMhu (ORCPT + 28 others); Wed, 6 Dec 2017 07:37:50 -0500 Received: from foss.arm.com ([217.140.101.70]:34816 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752183AbdLFMgN (ORCPT ); Wed, 6 Dec 2017 07:36:13 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CBDBA174E; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9DE653F236; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 111891AE35FB; Wed, 6 Dec 2017 12:36:17 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 17/20] arm64: Kconfig: Add CONFIG_UNMAP_KERNEL_AT_EL0 Date: Wed, 6 Dec 2017 12:35:36 +0000 Message-Id: <1512563739-25239-18-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Add a Kconfig entry to control use of the entry trampoline, which allows us to unmap the kernel whilst running in userspace and improve the robustness of KASLR. Reviewed-by: Mark Rutland Signed-off-by: Will Deacon --- arch/arm64/Kconfig | 13 +++++++++++++ 1 file changed, 13 insertions(+) -- 2.1.4 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index fdcc7b9bb15d..3af1657fcac3 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -833,6 +833,19 @@ config FORCE_MAX_ZONEORDER However for 4K, we choose a higher default value, 11 as opposed to 10, giving us 4M allocations matching the default size used by generic code. +config UNMAP_KERNEL_AT_EL0 + bool "Unmap kernel when running in userspace (aka \"KAISER\")" + default y + help + Some attacks against KASLR make use of the timing difference between + a permission fault which could arise from a page table entry that is + present in the TLB, and a translation fault which always requires a + page table walk. This option defends against these attacks by unmapping + the kernel whilst running in userspace, therefore forcing translation + faults for all of kernel space. + + If unsure, say Y. + menuconfig ARMV8_DEPRECATED bool "Emulate deprecated/obsolete ARMv8 instructions" depends on COMPAT From patchwork Wed Dec 6 12:35:37 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120828 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7000936qgn; Wed, 6 Dec 2017 04:37:45 -0800 (PST) X-Google-Smtp-Source: AGs4zMZC9Ym+yrjYgkHvGRMnQZyk4XcWw0/rF+II6WTWrD7Edp8oAQXxTWhKY8Y/EYfLu4/prpqs X-Received: by 10.99.112.89 with SMTP id a25mr20833280pgn.2.1512563865564; Wed, 06 Dec 2017 04:37:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512563865; cv=none; d=google.com; s=arc-20160816; b=NhuYBIWeFG0yIya/K/zcbH6F3QUSogGD93uwCBMCoxsCPlElv0DDMWQpUOkokh7Yr4 t3ofQlNOoYUijFdMmH25MjLDUWNbeBBVA2+FyYv5BJPVEudtIkCV6vjIhHUWJO3oB/9Y 2T/s91MFdWPyNFM5Y6sm/rjYAVWXrmAsK9oblihaojT8rei3THrkVECyH/36ntqo9Ixj 7Z+0RxwtwbUaLkL4GDSjTp+u2+mMohEcY1qED4K3okOF00sUb3UPJn0GJMb/OdgqBfh2 BP5iaH2ONWPMy5caheFTNZ1UD8swU0HMMhTbDMnGQHeVXXJIal6JBQ+ifrENkcnfsPWB IGcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=0qMMbJH30m15OASqcqOi/r8r8+WR8yt+DuXw63sGUJY=; b=P1Os6tZ5/bNpf0rDZ2YJY4GRqizv4y/6XXNYuK8Wv8CfA2LYfC1i7z/arhFTRhB3ol Sxu3WQ0EtGeD67gvSNC6ra0ZB4d3uMZpy+JyiSIdaoS9Spmlo4tLth652DEccHIE7tDX dyfvEsPWvD4oyaCj3xwQiLba47Jd7KC+9IaSqQp5kt5qZX9MgPareoRyYOlgdsrxSHHP Z0ku0C6krA1/BSpalz9KwwKaKLAYz9JwFHnO48Iwv1lmqyO8GDQqQvQlRM1vXCZ/l/ZW 8wb4C7JY+pcaeokORdQzL9a6fELcNJWiEhTmgZSOd9TL40aVnU6JHVVqqz7VDkHhabZ8 QkVg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w3si1841951pge.245.2017.12.06.04.37.45; Wed, 06 Dec 2017 04:37:45 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752595AbdLFMho (ORCPT + 28 others); Wed, 6 Dec 2017 07:37:44 -0500 Received: from foss.arm.com ([217.140.101.70]:34830 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752189AbdLFMgN (ORCPT ); Wed, 6 Dec 2017 07:36:13 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4D9401A09; Wed, 6 Dec 2017 04:36:12 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 1F3B63F53E; Wed, 6 Dec 2017 04:36:12 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 21F6F1AE3603; Wed, 6 Dec 2017 12:36:17 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 18/20] perf: arm_spe: Fail device probe when arm64_kernel_unmapped_at_el0() Date: Wed, 6 Dec 2017 12:35:37 +0000 Message-Id: <1512563739-25239-19-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When running with the kernel unmapped whilst at EL0, the virtually-addressed SPE buffer is also unmapped, which can lead to buffer faults if userspace profiling is enabled and potentially also when writing back kernel samples unless an expensive drain operation is performed on exception return. For now, fail the SPE driver probe when arm64_kernel_unmapped_at_el0(). Signed-off-by: Will Deacon --- drivers/perf/arm_spe_pmu.c | 9 +++++++++ 1 file changed, 9 insertions(+) -- 2.1.4 Reviewed-by: Mark Rutland diff --git a/drivers/perf/arm_spe_pmu.c b/drivers/perf/arm_spe_pmu.c index 8ce262fc2561..51b40aecb776 100644 --- a/drivers/perf/arm_spe_pmu.c +++ b/drivers/perf/arm_spe_pmu.c @@ -1164,6 +1164,15 @@ static int arm_spe_pmu_device_dt_probe(struct platform_device *pdev) struct arm_spe_pmu *spe_pmu; struct device *dev = &pdev->dev; + /* + * If kernelspace is unmapped when running at EL0, then the SPE + * buffer will fault and prematurely terminate the AUX session. + */ + if (arm64_kernel_unmapped_at_el0()) { + dev_warn_once(dev, "profiling buffer inaccessible. Try passing \"kpti=off\" on the kernel command line\n"); + return -EPERM; + } + spe_pmu = devm_kzalloc(dev, sizeof(*spe_pmu), GFP_KERNEL); if (!spe_pmu) { dev_err(dev, "failed to allocate spe_pmu\n"); From patchwork Wed Dec 6 12:35:38 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120829 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7001012qgn; Wed, 6 Dec 2017 04:37:49 -0800 (PST) X-Google-Smtp-Source: AGs4zMZ3yq+byTqhnt/unzOwhlUe6L203zAto0qCkwX0FcZEfxC1AwK4NfpwKn3Y0kxxJC19QokB X-Received: by 10.99.150.9 with SMTP id c9mr20887449pge.386.1512563869732; Wed, 06 Dec 2017 04:37:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512563869; cv=none; d=google.com; s=arc-20160816; b=VHA8bw2Y1hnipqbpI251Qot7zu0g/QfvwJ38wBWs0FfUNMgXHeiS3zJ6ZoGXdOxwSF Vkpmb6ypV0YKNBDBKLyC+XxruieuTe6hgUER7f73wftPUkWhYNkIm8ZU9ASsrk4ypl2p ezkovq8x7WI+VbjwDuwhEapFEvTlCKEcwhUVmzphfx+4zXBUVJmkdEnXcUcQL3Gbyxd8 5sWdIhk3IAYoBMf7QStUjUFO6FE2+wXWvYA+s4bEH8/mlGm8WXzMAdlvYxlfnKybClu3 JPtoK+uSed2nOs1kKhM8MtnnKoYfTffpourXwVS21v+6xJgFSdFA63EanenIImlg3C1I ynFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=MWK0I+38g9PduFq3oPJBrEnulhT3e5DtfUfQUVA9k3Y=; b=DsZYB0VSBTXU1P3nwC0FHxZRn9O8weLGs/mM1npV5Ar0V5bvLS3XDHUROWgWXz+zue cSS+PUXHg7RpYUSP2OzFk4sGJdBmJrVeZ98hi/DxDGScXqCBy7P3PaqGwOdKQJ9BiXu8 GGFTOeDe7es8xMscxgUz5otnvRKR+5igHVX+LulVhGDFM+DnH7MWVYrxyE8aDcT5Bq1j 29YzTg6i7QRENZncHzRG64m60Te519VnZ70LfcsAdXVVYsVwuFnPM3uyWbF6UjNwtybU T+Rhq7FYGipo713QsL+bHBpbsWQ5YueqDExChdpIXw6cX/WEK9M9xIayIz7P/ERt8Dzk Roaw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id s59si1910014plb.232.2017.12.06.04.37.49; Wed, 06 Dec 2017 04:37:49 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752606AbdLFMhq (ORCPT + 28 others); Wed, 6 Dec 2017 07:37:46 -0500 Received: from foss.arm.com ([217.140.101.70]:34694 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751709AbdLFMgN (ORCPT ); Wed, 6 Dec 2017 07:36:13 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D80541993; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A9F753F53E; Wed, 6 Dec 2017 04:36:11 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 322CC1AE3632; Wed, 6 Dec 2017 12:36:17 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 19/20] arm64: mm: Introduce TTBR_ASID_MASK for getting at the ASID in the TTBR Date: Wed, 6 Dec 2017 12:35:38 +0000 Message-Id: <1512563739-25239-20-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org There are now a handful of open-coded masks to extract the ASID from a TTBR value, so introduce a TTBR_ASID_MASK and use that instead. Suggested-by: Mark Rutland Signed-off-by: Will Deacon --- arch/arm64/include/asm/asm-uaccess.h | 3 ++- arch/arm64/include/asm/mmu.h | 1 + arch/arm64/include/asm/uaccess.h | 4 ++-- arch/arm64/kernel/entry.S | 2 +- 4 files changed, 6 insertions(+), 4 deletions(-) -- 2.1.4 Reviewed-by: Mark Rutland diff --git a/arch/arm64/include/asm/asm-uaccess.h b/arch/arm64/include/asm/asm-uaccess.h index 21b8cf304028..f4f234b6155e 100644 --- a/arch/arm64/include/asm/asm-uaccess.h +++ b/arch/arm64/include/asm/asm-uaccess.h @@ -4,6 +4,7 @@ #include #include +#include #include #include @@ -17,7 +18,7 @@ msr ttbr0_el1, \tmp1 // set reserved TTBR0_EL1 isb sub \tmp1, \tmp1, #SWAPPER_DIR_SIZE - bic \tmp1, \tmp1, #(0xffff << 48) + bic \tmp1, \tmp1, #TTBR_ASID_MASK msr ttbr1_el1, \tmp1 // set reserved ASID isb .endm diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index da6f12e40714..6f7bdb89817f 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -18,6 +18,7 @@ #define MMCF_AARCH32 0x1 /* mm context flag for AArch32 executables */ #define USER_ASID_FLAG (UL(1) << 48) +#define TTBR_ASID_MASK (UL(0xffff) << 48) #ifndef __ASSEMBLY__ diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h index 750a3b76a01c..6eadf55ebaf0 100644 --- a/arch/arm64/include/asm/uaccess.h +++ b/arch/arm64/include/asm/uaccess.h @@ -112,7 +112,7 @@ static inline void __uaccess_ttbr0_disable(void) write_sysreg(ttbr + SWAPPER_DIR_SIZE, ttbr0_el1); isb(); /* Set reserved ASID */ - ttbr &= ~(0xffffUL << 48); + ttbr &= ~TTBR_ASID_MASK; write_sysreg(ttbr, ttbr1_el1); isb(); } @@ -131,7 +131,7 @@ static inline void __uaccess_ttbr0_enable(void) /* Restore active ASID */ ttbr1 = read_sysreg(ttbr1_el1); - ttbr1 |= ttbr0 & (0xffffUL << 48); + ttbr1 |= ttbr0 & TTBR_ASID_MASK; write_sysreg(ttbr1, ttbr1_el1); isb(); diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 5d51bdbb2131..3eabcb194c87 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -205,7 +205,7 @@ alternative_else_nop_endif .if \el != 0 mrs x21, ttbr1_el1 - tst x21, #0xffff << 48 // Check for the reserved ASID + tst x21, #TTBR_ASID_MASK // Check for the reserved ASID orr x23, x23, #PSR_PAN_BIT // Set the emulated PAN in the saved SPSR b.eq 1f // TTBR0 access already disabled and x23, x23, #~PSR_PAN_BIT // Clear the emulated PAN in the saved SPSR From patchwork Wed Dec 6 12:35:39 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 120826 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7000665qgn; Wed, 6 Dec 2017 04:37:27 -0800 (PST) X-Google-Smtp-Source: AGs4zMa00MslHuHr5fz4dF/zET2Nl8Dt3ku6f8aMC3aGb/4jWhNhgBksZdMDnQA6Yy798oc3SHhJ X-Received: by 10.84.128.76 with SMTP id 70mr21633326pla.171.1512563847820; Wed, 06 Dec 2017 04:37:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512563847; cv=none; d=google.com; s=arc-20160816; b=SuEf/5Cej6CaTwYwAD8LF1T0yYcavwWPJXTJPpcvMLI4kWln5OssmWgE6ISJvqSAg0 vfAQKSSvXQBqUGVMC8nk0zHBKHpxy6BJuM1hEJMVY8PNiT5euyVuUVW4LQ8B1PdRe3Id 9JAvyS5LF/hR8A8keS1XEt+UzU4CPB0L33UaQKoBukzaSerH1nzTKeYXZFUl4DkSKwgf +bF690DYbrouvRtuEcTmgOxi4rpiTZVTArUKOy7vQCOVrwMw2ijHaSQSAO/XDAxf7Pz4 AQtEUVeHJBDLaa7t0hzKvWeeO7/o6oi3IcF9Et2wX2BgN4XmDY+KdYCKZB0OWAaRUemM Ekdg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=vkivRbjPJxZ6U5AiBhzxetZ3KZKwpfh7wwCUoC1F3uQ=; b=OZPznLShZ7+O9LjTsewN3i8r8sIP1RzSysmR9gVxdKH6r0EANVH67yJqbktcWktu/E G7I6HEt/3Xe3DWIa8w8+Wp1WxWaIvi9SzL3MYbd7MsEYQwnV+BKPtVOTp7ujtjB06IqS UlXbZi6U9S3n4MSwhRz5OOyzfG5T0dDyqIQDYtE50GIvUWuj7mepiQ9E19W1IF6q+yXR iRzx4fwZGILDqs0MWRrpIPPwFjRTRd/zB+Hv3n9BH3jmutLrS6aTwL84R8RP5NBZGjQH AtM1krlehLcBLrET81g9HymHuL9v9MQAHz5brKVhqNJ3eks1/5NwqaW/8LuHAvADF/59 mnaA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w3si1841951pge.245.2017.12.06.04.37.27; Wed, 06 Dec 2017 04:37:27 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752545AbdLFMhX (ORCPT + 28 others); Wed, 6 Dec 2017 07:37:23 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:34838 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752212AbdLFMgO (ORCPT ); Wed, 6 Dec 2017 07:36:14 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 569711AD7; Wed, 6 Dec 2017 04:36:12 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 282C83F5BD; Wed, 6 Dec 2017 04:36:12 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 4274F1AE3A0E; Wed, 6 Dec 2017 12:36:17 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, ard.biesheuvel@linaro.org, sboyd@codeaurora.org, dave.hansen@linux.intel.com, keescook@chromium.org, msalter@redhat.com, labbott@redhat.com, tglx@linutronix.de, Will Deacon Subject: [PATCH v3 20/20] arm64: kaslr: Put kernel vectors address in separate data page Date: Wed, 6 Dec 2017 12:35:39 +0000 Message-Id: <1512563739-25239-21-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1512563739-25239-1-git-send-email-will.deacon@arm.com> References: <1512563739-25239-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The literal pool entry for identifying the vectors base is the only piece of information in the trampoline page that identifies the true location of the kernel. This patch moves it into its own page, which is only mapped by the full kernel page table, which protects against any accidental leakage of the trampoline contents. Suggested-by: Ard Biesheuvel Signed-off-by: Will Deacon --- arch/arm64/include/asm/fixmap.h | 1 + arch/arm64/kernel/entry.S | 11 +++++++++++ arch/arm64/kernel/vmlinux.lds.S | 35 ++++++++++++++++++++++++++++------- arch/arm64/mm/mmu.c | 10 +++++++++- 4 files changed, 49 insertions(+), 8 deletions(-) -- 2.1.4 diff --git a/arch/arm64/include/asm/fixmap.h b/arch/arm64/include/asm/fixmap.h index 8119b49be98d..ec1e6d6fa14c 100644 --- a/arch/arm64/include/asm/fixmap.h +++ b/arch/arm64/include/asm/fixmap.h @@ -59,6 +59,7 @@ enum fixed_addresses { #endif /* CONFIG_ACPI_APEI_GHES */ #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 + FIX_ENTRY_TRAMP_DATA, FIX_ENTRY_TRAMP_TEXT, #define TRAMP_VALIAS (__fix_to_virt(FIX_ENTRY_TRAMP_TEXT)) #endif /* CONFIG_UNMAP_KERNEL_AT_EL0 */ diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 3eabcb194c87..a70c6dd2cc19 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -1030,7 +1030,13 @@ alternative_else_nop_endif msr tpidrro_el0, x30 // Restored in kernel_ventry .endif tramp_map_kernel x30 +#ifdef CONFIG_RANDOMIZE_BASE + adr x30, tramp_vectors + PAGE_SIZE +alternative_insn isb, nop, ARM64_WORKAROUND_QCOM_FALKOR_E1003 + ldr x30, [x30] +#else ldr x30, =vectors +#endif prfm plil1strm, [x30, #(1b - tramp_vectors)] msr vbar_el1, x30 add x30, x30, #(1b - tramp_vectors) @@ -1073,6 +1079,11 @@ END(tramp_exit_compat) .ltorg .popsection // .entry.tramp.text +#ifdef CONFIG_RANDOMIZE_BASE + .pushsection ".entry.tramp.data", "a" // .entry.tramp.data + .quad vectors + .popsection // .entry.tramp.data +#endif /* CONFIG_RANDOMIZE_BASE */ #endif /* CONFIG_UNMAP_KERNEL_AT_EL0 */ /* diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 6b4260f22aab..976109b3ae51 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -58,15 +58,28 @@ jiffies = jiffies_64; #endif #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 -#define TRAMP_TEXT \ - . = ALIGN(PAGE_SIZE); \ - VMLINUX_SYMBOL(__entry_tramp_text_start) = .; \ - *(.entry.tramp.text) \ - . = ALIGN(PAGE_SIZE); \ +#define TRAMP_TEXT \ + . = ALIGN(PAGE_SIZE); \ + VMLINUX_SYMBOL(__entry_tramp_text_start) = .; \ + *(.entry.tramp.text) \ + . = ALIGN(PAGE_SIZE); \ VMLINUX_SYMBOL(__entry_tramp_text_end) = .; +#ifdef CONFIG_RANDOMIZE_BASE +#define TRAMP_DATA \ + .entry.tramp.data : { \ + . = ALIGN(PAGE_SIZE); \ + VMLINUX_SYMBOL(__entry_tramp_data_start) = .; \ + *(.entry.tramp.data) \ + . = ALIGN(PAGE_SIZE); \ + VMLINUX_SYMBOL(__entry_tramp_data_end) = .; \ + } +#else +#define TRAMP_DATA +#endif /* CONFIG_RANDOMIZE_BASE */ #else #define TRAMP_TEXT -#endif +#define TRAMP_DATA +#endif /* CONFIG_UNMAP_KERNEL_AT_EL0 */ /* * The size of the PE/COFF section that covers the kernel image, which @@ -137,6 +150,7 @@ SECTIONS RO_DATA(PAGE_SIZE) /* everything from this point to */ EXCEPTION_TABLE(8) /* __init_begin will be marked RO NX */ NOTES + TRAMP_DATA . = ALIGN(SEGMENT_ALIGN); __init_begin = .; @@ -251,7 +265,14 @@ ASSERT(__idmap_text_end - (__idmap_text_start & ~(SZ_4K - 1)) <= SZ_4K, ASSERT(__hibernate_exit_text_end - (__hibernate_exit_text_start & ~(SZ_4K - 1)) <= SZ_4K, "Hibernate exit text too big or misaligned") #endif - +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 +ASSERT((__entry_tramp_text_end - __entry_tramp_text_start) == PAGE_SIZE, + "Entry trampoline text too big") +#ifdef CONFIG_RANDOMIZE_BASE +ASSERT((__entry_tramp_data_end - __entry_tramp_data_start) == PAGE_SIZE, + "Entry trampoline data too big") +#endif +#endif /* * If padding is applied before .head.text, virt<->phys conversions will fail. */ diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index fe68a48c64cb..916d9ced1c3f 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -541,8 +541,16 @@ static int __init map_entry_trampoline(void) __create_pgd_mapping(tramp_pg_dir, pa_start, TRAMP_VALIAS, PAGE_SIZE, prot, pgd_pgtable_alloc, 0); - /* ...as well as the kernel page table */ + /* Map both the text and data into the kernel page table */ __set_fixmap(FIX_ENTRY_TRAMP_TEXT, pa_start, prot); + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) { + extern char __entry_tramp_data_start[]; + + __set_fixmap(FIX_ENTRY_TRAMP_DATA, + __pa_symbol(__entry_tramp_data_start), + PAGE_KERNEL_RO); + } + return 0; } core_initcall(map_entry_trampoline);