From patchwork Thu Nov 14 14:59:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suzuki K Poulose X-Patchwork-Id: 179448 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp11170000ilf; Thu, 14 Nov 2019 07:00:04 -0800 (PST) X-Google-Smtp-Source: APXvYqzfWHekIQg6Dn/K7CtB+wuvNP0pXYdAZzXClCld2xQNSI/is7uJNVnRnBZ7ZAmRNwWIwYVj X-Received: by 2002:a50:c191:: with SMTP id m17mr1714646edf.259.1573743604098; Thu, 14 Nov 2019 07:00:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573743604; cv=none; d=google.com; s=arc-20160816; b=NQWj+gxMdz7zkeTURHivveiR9yQcjj5Z9efOPH8fR2/5jJpigxoGBGwtQEFDT0V1Wy Mqr1isFINZ5osbd1sGbashtg9A9cTqQN0jmSaNJM6Us/bj76UGYla5vXRCGv2z3J1MJm ySrSgCyyZa5OrSjuYmZupo2UYGq/dTbUYVw/2jNjn0G5+x6SMnGW1CxiQZ/mt/7WIa0P VULXBK7IxsyPJ+rGMo43sZlrGbyqjRi/izIKJbStcgUSNtFV018zuND0P9mT1bkpPYE/ NSdCyFGxjuQ9kMyM9SeZ+zeux3hK0/ZWfBCbfSuEFm7netb+qJwfnzBNWP9g/rhL/1i3 sPmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=sbaFBaoSi/IGg3KP39hiSZ2i4vS/njnwPXoIJaWCCTA=; b=VeJybQJjmCEXJ+94h8mBUTpBhcbXMpDGDcsGnaT6B9DKlInAKo2V7+dZXR+uPUrXhe BSSQFnpD0oyYHP5oqlnkubBghX5LS7LSkd7X4v5fPJ4n2Uaj96m3K6BGYQ8ldMdRdNzE 5CD9YL1zFpaAaBT/pzmT91vlbhRaARcv5KZgY7gFrVxjnUSIJ6pl9a8YKwQB0xsnCxBS Joj2KEpqOpK5uVVHdlfTTwAaLmWpiJ5Qdq9hISyR2dlWpRymb3PExHvMKG489jkSeGH3 Gm6mWfDXshpdrfp014Vm1m1SomSQL+rKER40JqAgU5/p10hg40vTapKxMsJ+S2T/Suyj viaA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i14si4010117edr.68.2019.11.14.07.00.03; Thu, 14 Nov 2019 07:00:04 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726605AbfKNPAC (ORCPT + 26 others); Thu, 14 Nov 2019 10:00:02 -0500 Received: from foss.arm.com ([217.140.110.172]:44614 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726251AbfKNO77 (ORCPT ); Thu, 14 Nov 2019 09:59:59 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2FC1F55D; Thu, 14 Nov 2019 06:59:59 -0800 (PST) Received: from ewhatever.cambridge.arm.com (ewhatever.cambridge.arm.com [10.1.197.1]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0E4063F52E; Thu, 14 Nov 2019 06:59:57 -0800 (PST) From: Suzuki K Poulose To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, james.morse@arm.com, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, maz@kernel.org, suzuki.poulose@arm.com Subject: [PATCH 1/5] arm64: Add MIDR encoding for Arm Cortex-A77 Date: Thu, 14 Nov 2019 14:59:14 +0000 Message-Id: <20191114145918.235339-2-suzuki.poulose@arm.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191114145918.235339-1-suzuki.poulose@arm.com> References: <20191114145918.235339-1-suzuki.poulose@arm.com> MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: James Morse Add Arm Cortex-A77's part-number so we can match against its MIDR_EL1. Signed-off-by: James Morse Signed-off-by: Suzuki K Poulose --- arch/arm64/include/asm/cputype.h | 2 ++ 1 file changed, 2 insertions(+) -- 2.23.0 diff --git a/arch/arm64/include/asm/cputype.h b/arch/arm64/include/asm/cputype.h index aca07c2f6e6e..3c8c1580527d 100644 --- a/arch/arm64/include/asm/cputype.h +++ b/arch/arm64/include/asm/cputype.h @@ -71,6 +71,7 @@ #define ARM_CPU_PART_CORTEX_A55 0xD05 #define ARM_CPU_PART_CORTEX_A76 0xD0B #define ARM_CPU_PART_NEOVERSE_N1 0xD0C +#define ARM_CPU_PART_CORTEX_A77 0xD0D #define APM_CPU_PART_POTENZA 0x000 @@ -101,6 +102,7 @@ #define MIDR_CORTEX_A35 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A35) #define MIDR_CORTEX_A55 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A55) #define MIDR_CORTEX_A76 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A76) +#define MIDR_CORTEX_A77 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A77) #define MIDR_NEOVERSE_N1 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_NEOVERSE_N1) #define MIDR_THUNDERX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX) #define MIDR_THUNDERX_81XX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX_81XX) From patchwork Thu Nov 14 14:59:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suzuki K Poulose X-Patchwork-Id: 179452 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp11170313ilf; Thu, 14 Nov 2019 07:00:18 -0800 (PST) X-Google-Smtp-Source: APXvYqw2QN7Boq2sO6DeUhG+15VXo2YEdkEeulXX6aT4sT9MZwOsbIwlvZOFRszEwZHCjEWoCmLG X-Received: by 2002:a50:b6cb:: with SMTP id f11mr1710477ede.299.1573743617903; Thu, 14 Nov 2019 07:00:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573743617; cv=none; d=google.com; s=arc-20160816; b=e1F/nMDTagiZC1dcOvx4NqbSLNPgfvNfth8E/4fwzPuDxwDtiug1SzY2m5ESSSKhR0 OlYB1nRwoiRw54IOejwN1qr/tqkRsPpqh9N1IK4S44Z11hGpMSMS4U9TLJh6wBhsdZb4 Hd3Tvtt4WKJTppdybQLILMkLJ3r7bfN5UZIJJ+bXgt3TJe1zxm2XmderS0T8u9663exu 99S0a0+8KQnXC8XRR9WRZ+5NoBvMClfzyOVO+wmcCvvbML/13qijJuNDi+LKYt8iWoKP nKJy2uCOJ1UWHfPlyfJ0M7adiba1wE/WuR1xywCn/P57+ujJ1HpHiZneHvaGiAHIVlRm WYbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=bHyEXIqD7MnAqEbnF+WXkNYZ0sYBWjK+b4SMbFMahrM=; b=VqUSCFjwbz9uI9NkU8Vh9hgplcqNAA8YXDA1qh0Rwum+PH3UhFpETwuB+TAvGmZeHB 4NjPb+QPnP7lLCoiaEhYY6gUlP3OsiUnc2tOSzrlUP1gV5vuW1E2UdI30xa2L12LF0h4 Za+K0fYAN8Gl7MWdQQNBSjfDTbYEN6GqVMFVqSHYrvu6PprtDaFxQzZY/3uxohsx7R0s XrO0Yq1rmIW+uKrP6uC9DnZCSzyhlarHA0YRGZgFZK7pYBbCaBUEaIUJjO+Trp+eRqip fkqp3UnFO2FwsFrrGIeQwC4/c2Bd00OhfjhXA4/UmXFawUk4lapc8Ipgkdes/PtJ1KAv DOAA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id gs14si3604239ejb.347.2019.11.14.07.00.17; Thu, 14 Nov 2019 07:00:17 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726983AbfKNPAQ (ORCPT + 26 others); Thu, 14 Nov 2019 10:00:16 -0500 Received: from foss.arm.com ([217.140.110.172]:44624 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726482AbfKNPAB (ORCPT ); Thu, 14 Nov 2019 10:00:01 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8219BC8F; Thu, 14 Nov 2019 07:00:00 -0800 (PST) Received: from ewhatever.cambridge.arm.com (ewhatever.cambridge.arm.com [10.1.197.1]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 63B813F52E; Thu, 14 Nov 2019 06:59:59 -0800 (PST) From: Suzuki K Poulose To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, james.morse@arm.com, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, maz@kernel.org, suzuki.poulose@arm.com Subject: [PATCH 2/5] arm64: mm: Workaround Cortex-A77 erratum 1542418 on ASID rollover Date: Thu, 14 Nov 2019 14:59:15 +0000 Message-Id: <20191114145918.235339-3-suzuki.poulose@arm.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191114145918.235339-1-suzuki.poulose@arm.com> References: <20191114145918.235339-1-suzuki.poulose@arm.com> MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: James Morse On affected Cortex-A77 cores, software relying on the prefetch-speculation-protection instead of explicit synchronisation may fetch a stale instruction from a CPU-specific cache. This violates the ordering rules for instruction fetches. This can only happen when the CPU correctly predicts the modified branch based on a previous ASID/VMID. The workaround is to prevent these predictions by selecting 60 ASIDs before an ASID is reused. Add this logic as a workaround in the asid-alloctor's per-cpu rollover path. When the first asid of the new generation is about to be used, select 60 different ASIDs before we do the TLB maintenance. Signed-off-by: James Morse [ Added/modified commentary ] Signed-off-by: Suzuki K Poulose --- Documentation/arm64/silicon-errata.rst | 2 + arch/arm64/Kconfig | 16 ++++++++ arch/arm64/include/asm/cpucaps.h | 3 +- arch/arm64/kernel/cpu_errata.c | 7 ++++ arch/arm64/mm/context.c | 56 +++++++++++++++++++++++++- 5 files changed, 82 insertions(+), 2 deletions(-) -- 2.23.0 diff --git a/Documentation/arm64/silicon-errata.rst b/Documentation/arm64/silicon-errata.rst index 5a09661330fc..a6a5ece00392 100644 --- a/Documentation/arm64/silicon-errata.rst +++ b/Documentation/arm64/silicon-errata.rst @@ -84,6 +84,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A76 | #1463225 | ARM64_ERRATUM_1463225 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A77 | #1542418 | ARM64_ERRATUM_1542418 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N1 | #1188873,1418040| ARM64_ERRATUM_1418040 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N1 | #1349291 | N/A | diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 3f047afb982c..f0fc570ce05f 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -558,6 +558,22 @@ config ARM64_ERRATUM_1463225 If unsure, say Y. +config ARM64_ERRATUM_1542418 + bool "Cortex-A77: The core might fetch a stale instuction, violating the ordering of instruction fetches" + default y + help + This option adds a workaround for Arm Cortex-A77 erratum 1542418. + + On the affected Cortex-A77 cores (r0p0 and r1p0), software relying + on the prefetch-speculation-protection instead of explicit + synchronisation may fetch a stale instruction from a CPU-specific + cache. This violates the ordering rules for instruction fetches. + + Work around the erratum by ensuring that 60 ASIDs are selected + before any ASID is reused. + + If unsure, say Y. + config CAVIUM_ERRATUM_22375 bool "Cavium erratum 22375, 24313" default y diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index ac1dbca3d0cd..1f90084e8a59 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -54,7 +54,8 @@ #define ARM64_WORKAROUND_1463225 44 #define ARM64_WORKAROUND_CAVIUM_TX2_219_TVM 45 #define ARM64_WORKAROUND_CAVIUM_TX2_219_PRFM 46 +#define ARM64_WORKAROUND_1542418 47 -#define ARM64_NCAPS 47 +#define ARM64_NCAPS 48 #endif /* __ASM_CPUCAPS_H */ diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c index 93f34b4eca25..a66d433d0113 100644 --- a/arch/arm64/kernel/cpu_errata.c +++ b/arch/arm64/kernel/cpu_errata.c @@ -926,6 +926,13 @@ const struct arm64_cpu_capabilities arm64_errata[] = { .capability = ARM64_WORKAROUND_CAVIUM_TX2_219_PRFM, ERRATA_MIDR_RANGE_LIST(tx2_family_cpus), }, +#endif +#ifdef CONFIG_ARM64_ERRATUM_1542418 + { + .desc = "ARM erratum 1542418", + .capability = ARM64_WORKAROUND_1542418, + ERRATA_MIDR_RANGE(MIDR_CORTEX_A77, 0, 0, 1, 0), + }, #endif { } diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index b5e329fde2dd..ae3ee8e101d6 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -77,6 +77,58 @@ void verify_cpu_asid_bits(void) } } + +/* + * When the CnP is active, the caller must have set the ttbr0 to reserved + * before calling this function. + * Upon completion, the caller must ensure to: + * - restore the ttbr0 + * - execute isb() to synchronize the change. + */ +static void __arm64_workaround_1542418_asid_rollover(void) +{ + phys_addr_t ttbr1_baddr; + u64 idx, ttbr1; /* ASID is in ttbr1 due to TCR_EL1.A1 */ + + if (!IS_ENABLED(CONFIG_ARM64_ERRATUM_1542418) || + !cpus_have_const_cap(ARM64_WORKAROUND_1542418) || + !this_cpu_has_cap(ARM64_WORKAROUND_1542418)) + return; + + /* + * We're about to use an arbitrary set of ASIDs, which may have + * live entries in the TLB (and on other CPUs with CnP). Ensure + * that we can't allocate conflicting entries using this task's + * TTBR0. + */ + if (!system_supports_cnp()) + cpu_set_reserved_ttbr0(); + /* else: the caller must have already set this */ + + ttbr1 = read_sysreg(ttbr1_el1); + ttbr1_baddr = ttbr1 & ~TTBR_ASID_MASK; + + /* + * Select 60 asids to invalidate the branch history for this generation. + * If kpti is in use we avoid selecting a user asid as + * __sdei_asm_entry_trampoline() uses USER_ASID_FLAG to determine if + * the NMI interrupted the kpti trampoline. Avoid using the reserved + * asid 0. + */ + for (idx = 1; idx <= 61; idx++) { + write_sysreg((idx2asid(idx) << 48) | ttbr1_baddr, ttbr1_el1); + isb(); + } + + /* restore the current ASID */ + write_sysreg(ttbr1, ttbr1_el1); + + /* + * Rely on local_flush_tlb_all()'s isb to complete the ASID restore. + * check_and_switch_context() will call cpu_switch_mm() to (re)set ttbr0_el1. + */ +} + static void flush_context(void) { int i; @@ -219,8 +271,10 @@ void check_and_switch_context(struct mm_struct *mm, unsigned int cpu) atomic64_set(&mm->context.id, asid); } - if (cpumask_test_and_clear_cpu(cpu, &tlb_flush_pending)) + if (cpumask_test_and_clear_cpu(cpu, &tlb_flush_pending)) { + __arm64_workaround_1542418_asid_rollover(); local_flush_tlb_all(); + } atomic64_set(&per_cpu(active_asids, cpu), asid); raw_spin_unlock_irqrestore(&cpu_asid_lock, flags); From patchwork Thu Nov 14 14:59:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suzuki K Poulose X-Patchwork-Id: 179449 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp11170039ilf; Thu, 14 Nov 2019 07:00:06 -0800 (PST) X-Google-Smtp-Source: APXvYqwPFEataDMdpR1Rqz3oNYTQT7hnhDU068ZyPFF8yzvhfI8RHzAiKQnGcTFMEJO7Lz8I6x5l X-Received: by 2002:a17:906:234e:: with SMTP id m14mr8585811eja.94.1573743606523; Thu, 14 Nov 2019 07:00:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573743606; cv=none; d=google.com; s=arc-20160816; b=usyBc53qgx2iL5rWkH2P+6VRwopaT8fJthFbv0tpfHMp1vFUqTjo8hpxtIO5r0/+xv 1ft1rLp9td//Up9CpPBAVy9Zrt48i5+Y+ZPvsRxblCHeiNv5q8LIQX9opcfeeH/ZplvL llcCsMrp1Z3g49U+4MWwhG/aBEpe4aMCGVfwIJ8liLFhndxoR8ECQT5uvbO58D6pfXP1 KRkIU8XdAYYT/QbLyxhB+TofvwALrgNl/x5WEIO8057QDIGo5n12RX7YwZ8MM1Ujc1WB bnJGE/DBxiIvxBh1V9UXHISx2ie/4F84cEk5A2Hbm47aN4yorBT5ah3Q2feRvTMIFn93 j2Cg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=VN9jOFV7Frwuk54WF1nfP09q1zCyX3bHMUG50K0sQtk=; b=NAk8mH+aAcfy5l4CdLFF9AEHM3zv4Hywc7SCxl/F1036D2mqkU4qxrte7Yus0zVkeo /D9uYQHiod7m5lKRPkpBBAeFrGmiUEC3Gw+rJaUJxVc1wofU2YvvZExpXZ6Th3I6s9/s y14+Qe2SFaguYDlYRn5Q+xqQeKhlJqBLDUs3CmuniK9hsDitdPK7ZHI8yuDXNQoddnTy VTHNk3FUuX/VgmLU396d4ufSPvIBU7pEsGZ7Vj5W8H39Ur5xZr0Dle2JdSk3K6EWxs/a 2CeVhzCrUZbwa/IbeP/VuQ462PZuVvE8rE1f1ERPf2tZMOLaa+w7yebyKI9q8yYXr3BL 3mKQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i14si4010117edr.68.2019.11.14.07.00.06; Thu, 14 Nov 2019 07:00:06 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726812AbfKNPAF (ORCPT + 26 others); Thu, 14 Nov 2019 10:00:05 -0500 Received: from foss.arm.com ([217.140.110.172]:44630 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726599AbfKNPAC (ORCPT ); Thu, 14 Nov 2019 10:00:02 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D4107328; Thu, 14 Nov 2019 07:00:01 -0800 (PST) Received: from ewhatever.cambridge.arm.com (ewhatever.cambridge.arm.com [10.1.197.1]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B5F243F52E; Thu, 14 Nov 2019 07:00:00 -0800 (PST) From: Suzuki K Poulose To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, james.morse@arm.com, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, maz@kernel.org, suzuki.poulose@arm.com Subject: [PATCH 3/5] arm64: Workaround Cortex-A77 erratum 1542418 on boot due to kexec Date: Thu, 14 Nov 2019 14:59:16 +0000 Message-Id: <20191114145918.235339-4-suzuki.poulose@arm.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191114145918.235339-1-suzuki.poulose@arm.com> References: <20191114145918.235339-1-suzuki.poulose@arm.com> MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: James Morse Kexec allows us to inherit dirty ASIDs from a previous kernel. We can't wait until the next ASID rollover to cleanup, do it early as part of the cpu-errata's enable callback. This extends __arm64_workaround_1542418_asid_rollover() to put everything back as it was. Signed-off-by: James Morse [ skip CPUs not affected, refactor cpu_enable callback ] Signed-off-by: Suzuki K Poulose --- arch/arm64/include/asm/mmu_context.h | 1 + arch/arm64/kernel/cpu_errata.c | 14 ++++++++++++++ arch/arm64/mm/context.c | 17 +++++++++++++++++ 3 files changed, 32 insertions(+) -- 2.23.0 diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 3827ff4040a3..434a5c661d78 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -247,6 +247,7 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next, void verify_cpu_asid_bits(void); void post_ttbr_update_workaround(void); +void arm64_workaround_1542418_asid_rollover(void); #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c index a66d433d0113..4656157ffa36 100644 --- a/arch/arm64/kernel/cpu_errata.c +++ b/arch/arm64/kernel/cpu_errata.c @@ -12,6 +12,7 @@ #include #include #include +#include #include static bool __maybe_unused @@ -650,6 +651,18 @@ needs_tx2_tvm_workaround(const struct arm64_cpu_capabilities *entry, return false; } +#ifdef CONFIG_ARM64_ERRATUM_1542418 +static void run_workaround_1542418_asid_rollover(const struct arm64_cpu_capabilities *c) +{ + /* + * If this CPU is affected by the erratum, run the workaround + * to protect us in case we are running on a kexec'ed kernel. + */ + if (c->matches(c, SCOPE_LOCAL_CPU)) + arm64_workaround_1542418_asid_rollover(); +} +#endif + #ifdef CONFIG_HARDEN_EL2_VECTORS static const struct midr_range arm64_harden_el2_vectors[] = { @@ -932,6 +945,7 @@ const struct arm64_cpu_capabilities arm64_errata[] = { .desc = "ARM erratum 1542418", .capability = ARM64_WORKAROUND_1542418, ERRATA_MIDR_RANGE(MIDR_CORTEX_A77, 0, 0, 1, 0), + .cpu_enable = run_workaround_1542418_asid_rollover, }, #endif { diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index ae3ee8e101d6..ad4e78bb68ed 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -129,6 +129,23 @@ static void __arm64_workaround_1542418_asid_rollover(void) */ } +void arm64_workaround_1542418_asid_rollover(void) +{ + u64 ttbr0 = read_sysreg(ttbr0_el1); + + lockdep_assert_irqs_disabled(); + + /* Mirror check_and_switch_context() */ + if (system_supports_cnp()) + cpu_set_reserved_ttbr0(); + + __arm64_workaround_1542418_asid_rollover(); + isb(); + + write_sysreg(ttbr0, ttbr0_el1); + isb(); +} + static void flush_context(void) { int i; From patchwork Thu Nov 14 14:59:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suzuki K Poulose X-Patchwork-Id: 179450 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp11170152ilf; Thu, 14 Nov 2019 07:00:12 -0800 (PST) X-Google-Smtp-Source: APXvYqw5yhWHPwdN50oprkTPbNrsuAMNSSv05TbRrGn4rBzNUEotRhYBwezT4SJyq20oDbCvdEtR X-Received: by 2002:a17:907:1102:: with SMTP id qu2mr8687345ejb.300.1573743612369; Thu, 14 Nov 2019 07:00:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573743612; cv=none; d=google.com; s=arc-20160816; b=FRLchgACLxpPdZ60TkmcAfvAvcnlpsqQL32BbWUk8DgtWcYh6R7QSLPIpmXtHNH6Fh G6CRe+AUuS9xuTc0iflSSFgF6w+jLsCmKo6WHalj49bu9FwrsJCpnaRnE5lU1HyJf1u8 pF+/5ZnmgU+t1fuK7weEWQJIOQYg/WxFq8ekx3cAzYOwq+6U/sIXlQL9rMi/WmHgV9YG NZO0Pj9C+Z/NWqMEmm4TI9xBW8t5KQiyhr35DhvI8bgvn+B3fgE0z24S2cZfIou2lqgP AK2gEKJaXcJOeKtHSyPgGUyAgXfxCrjMBSxec0pZ9fR6bg6XsGyX7f5Jk/11RGauoBtH mHTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=N9ZsZNQnkf+r6xWJcUA9oRfajB+AWFilTGu9OZJTs/c=; b=kunnusv9lpso6c/0LhCX89IHC0KtU8pJNzwD8wo/aJMa9M1T/tI/VF5ekydKEKQbYI 6TLiw63rxgy/zDVXxuybmt7Inj5a2jvIVPL3tatzDaYd0w6KCTauaDdlUpDjkdSt34gF CgeKlytof8uXTFdTL9Fb+gosYbG/0JxhI++fXIw2m5R8mzFkCtaPZtDAJBYsyZ+T2zQe fkNC/kZo+zoUuT1/O1NByaXXub4+xkfgYcc3nWKPlMdIASqk91QmOIMDQ07MkJ/wT037 Y4nmhpEaoKRGe2nw9yEWzZ8XNmeGHPaYteurRCcuDnZh4vi7nmoFuE2K1/QSrZWEWRif 5Zig== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id gs14si3604239ejb.347.2019.11.14.07.00.12; Thu, 14 Nov 2019 07:00:12 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726909AbfKNPAH (ORCPT + 26 others); Thu, 14 Nov 2019 10:00:07 -0500 Received: from foss.arm.com ([217.140.110.172]:44642 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726717AbfKNPAD (ORCPT ); Thu, 14 Nov 2019 10:00:03 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3268DDA7; Thu, 14 Nov 2019 07:00:03 -0800 (PST) Received: from ewhatever.cambridge.arm.com (ewhatever.cambridge.arm.com [10.1.197.1]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 1418F3F52E; Thu, 14 Nov 2019 07:00:01 -0800 (PST) From: Suzuki K Poulose To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, james.morse@arm.com, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, maz@kernel.org, suzuki.poulose@arm.com Subject: [PATCH 4/5] KVM: arm64: Workaround Cortex-A77 erratum 1542418 on VMID rollover Date: Thu, 14 Nov 2019 14:59:17 +0000 Message-Id: <20191114145918.235339-5-suzuki.poulose@arm.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191114145918.235339-1-suzuki.poulose@arm.com> References: <20191114145918.235339-1-suzuki.poulose@arm.com> MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: James Morse Cortex-A77's erratum 1542418 workaround needs to be applied for VMID re-use too. This prevents the CPU correctly predicting a modified branch based on a previous user of the VMID and ASID. KVM doesn't use force_vm_exit or exit_vm_noop for anything other than vmid rollover. Rename them, and use this to invoke the VMID workaround on each CPU. Another case where VMID and ASID may get reused is if the system is over-provisioned and two vCPUs of the same VMID are scheduled on one physical CPU. KVM invalidates the TLB to prevent ASID sharing in this case, invoke the asid-rollover workaround too so we avoid the ASID sharing tripping the erratum. Signed-off-by: James Morse Signed-off-by: Suzuki K Poulose --- arch/arm/include/asm/kvm_mmu.h | 5 +++++ arch/arm64/include/asm/kvm_mmu.h | 15 +++++++++++++++ virt/kvm/arm/arm.c | 20 ++++++++++++++------ 3 files changed, 34 insertions(+), 6 deletions(-) -- 2.23.0 diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index 0d84d50bf9ba..8a5702e0c3f8 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -430,6 +430,11 @@ static __always_inline u64 kvm_get_vttbr(struct kvm *kvm) return kvm_phys_to_vttbr(baddr) | vmid_field; } +static inline void kvm_workaround_1542418_vmid_rollover(void) +{ + /* not affected */ +} + #endif /* !__ASSEMBLY__ */ #endif /* __ARM_KVM_MMU_H__ */ diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index befe37d4bc0e..5776e53c296d 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -9,6 +9,7 @@ #include #include +#include #include /* @@ -603,5 +604,19 @@ static __always_inline u64 kvm_get_vttbr(struct kvm *kvm) return kvm_phys_to_vttbr(baddr) | vmid_field | cnp; } +static inline void kvm_workaround_1542418_vmid_rollover(void) +{ + unsigned long flags; + + if (!IS_ENABLED(CONFIG_ARM64_ERRATUM_1542418) || + !cpus_have_const_cap(ARM64_WORKAROUND_1542418)) + return; + + local_irq_save(flags); + arm64_workaround_1542418_asid_rollover(); + local_irq_restore(flags); + +} + #endif /* __ASSEMBLY__ */ #endif /* __ARM64_KVM_MMU_H__ */ diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index 86c6aa1cb58e..ac9e017df7c9 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -368,6 +368,13 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) */ if (*last_ran != vcpu->vcpu_id) { kvm_call_hyp(__kvm_tlb_flush_local_vmid, vcpu); + + /* + * 'last_ran' and this vcpu may share an ASID and hit the + * conditions for Cortex-A77 erratum 1542418. + */ + kvm_workaround_1542418_vmid_rollover(); + *last_ran = vcpu->vcpu_id; } @@ -458,15 +465,16 @@ bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu) return vcpu_mode_priv(vcpu); } -/* Just ensure a guest exit from a particular CPU */ -static void exit_vm_noop(void *info) +static void exit_vmid_rollover(void *info) { + kvm_workaround_1542418_vmid_rollover(); } -void force_vm_exit(const cpumask_t *mask) +static void force_vmid_rollover_exit(const cpumask_t *mask) { preempt_disable(); - smp_call_function_many(mask, exit_vm_noop, NULL, true); + smp_call_function_many(mask, exit_vmid_rollover, NULL, true); + kvm_workaround_1542418_vmid_rollover(); preempt_enable(); } @@ -518,10 +526,10 @@ static void update_vmid(struct kvm_vmid *vmid) /* * On SMP we know no other CPUs can use this CPU's or each - * other's VMID after force_vm_exit returns since the + * other's VMID after force_vmid_rollover_exit returns since the * kvm_vmid_lock blocks them from reentry to the guest. */ - force_vm_exit(cpu_all_mask); + force_vmid_rollover_exit(cpu_all_mask); /* * Now broadcast TLB + ICACHE invalidation over the inner * shareable domain to make sure all data structures are From patchwork Thu Nov 14 14:59:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suzuki K Poulose X-Patchwork-Id: 179451 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp11170169ilf; Thu, 14 Nov 2019 07:00:13 -0800 (PST) X-Google-Smtp-Source: APXvYqwccPEPoX8abwUZA7uyiY0CWy/TJCOMAIB+4ORYJVVMIPi1su7sobkfy0j6L1o1S2jrVECQ X-Received: by 2002:a05:6402:883:: with SMTP id e3mr1767221edy.32.1573743612882; Thu, 14 Nov 2019 07:00:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573743612; cv=none; d=google.com; s=arc-20160816; b=XtIisEHv1FN16G826aPGyzP6TjudG2EhJOpesUn4G6XJKT+yaU1E/igqfD6oJQxDdb u8CtjbqIA3D+ShhugbQn3Uwl40/1NhCWaglDGRJLZF+p4gevosxPh1z5b5RaeI02Q8Fm 65PUM5OVDxay6OECHJZ4AUINl1b++3yHZij89NOYBi7YHE6jKdIhm8F9bvJ/eFtDIrPf Cn7Jj/+m9plYbr4qrWLLnUAdunC7ksIBkyoj2YgZP8TDl5yjf4SdgU4TNVG1Lc0y9F1k 1FnT3IANI7KtPwIZbz2oFYfKyPnI1DUnj7y5ivm1THELuehwj6kaQWB3lLrwWhw82eaE YSnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=ll/JgHjCtv3FaR0mZfjhrg0KpztudlfPt00tIZAFMXQ=; b=aazR7auIGWmJzysfNOZe1TssOK2er/kY77buZ3NiTkLDr/rIZCVjR0T2L6TAyQxVWW DMqxryJlE8r95EmPHxkbozQtkSZHzyNCbfTcMcW9GHWmkVUGaeHxA93+X0Zo3wjJi1rj 6bUTWrSAOIz/QrON9iKB/ioEwq28jbOtzVBBVV6sxq+d8tUXBhcpj7qCtshXdURF4Kuw wRc9rJFfqARILf13yqX9jttDdIKItMNmR++pMShRmuxtWqGKhxWDNX6/r3QcGBFo+gqQ bNopsXLi+htJT3QbF0i1diQWtDZGCUGDcOLTV3fNr7DRBpjbA6XFvjQuU8O3a8Re7AxC KkIQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id gs14si3604239ejb.347.2019.11.14.07.00.12; Thu, 14 Nov 2019 07:00:12 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726953AbfKNPAL (ORCPT + 26 others); Thu, 14 Nov 2019 10:00:11 -0500 Received: from foss.arm.com ([217.140.110.172]:44650 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726766AbfKNPAF (ORCPT ); Thu, 14 Nov 2019 10:00:05 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 84A00328; Thu, 14 Nov 2019 07:00:04 -0800 (PST) Received: from ewhatever.cambridge.arm.com (ewhatever.cambridge.arm.com [10.1.197.1]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 663033F52E; Thu, 14 Nov 2019 07:00:03 -0800 (PST) From: Suzuki K Poulose To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, james.morse@arm.com, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, maz@kernel.org, suzuki.poulose@arm.com Subject: [PATCH 5/5] KVM: arm/arm64: Don't invoke defacto-CnP on first run Date: Thu, 14 Nov 2019 14:59:18 +0000 Message-Id: <20191114145918.235339-6-suzuki.poulose@arm.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191114145918.235339-1-suzuki.poulose@arm.com> References: <20191114145918.235339-1-suzuki.poulose@arm.com> MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: James Morse When KVM finds itself switching between two vCPUs of the same VM on one physical CPU it has to invalidate the TLB for this VMID to avoid unintended sharing of TLB entries between vCPU. This is done by tracking the 'last_vcpu_ran' as a percpu variable for each vm. kvm_arch_init_vm() is careful to initialise these to an impossible vcpu id, but we never check for this. The first time vm_arch_vcpu_load() is called on a new physical CPU, we will fail the last_ran check and invalidate the TLB. Now that we have an errata workaround in this path, it means we trigger the workaround whenever a guest is migrated to a new CPU. Check for the impossible vcpu id, and skip defacto-CnP. Signed-off-by: James Morse Signed-off-by: Suzuki K Poulose --- virt/kvm/arm/arm.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) -- 2.23.0 diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index ac9e017df7c9..6f729739cf6f 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -366,7 +366,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) * We might get preempted before the vCPU actually runs, but * over-invalidation doesn't affect correctness. */ - if (*last_ran != vcpu->vcpu_id) { + if (*last_ran != -1 && *last_ran != vcpu->vcpu_id) { kvm_call_hyp(__kvm_tlb_flush_local_vmid, vcpu); /* @@ -374,9 +374,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) * conditions for Cortex-A77 erratum 1542418. */ kvm_workaround_1542418_vmid_rollover(); - - *last_ran = vcpu->vcpu_id; } + *last_ran = vcpu->vcpu_id; vcpu->cpu = cpu; vcpu->arch.host_cpu_context = &cpu_data->host_ctxt;