From patchwork Wed Jan 17 11:54:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Thierry X-Patchwork-Id: 124802 Delivered-To: patch@linaro.org Received: by 10.46.62.1 with SMTP id l1csp11326lja; Wed, 17 Jan 2018 03:55:20 -0800 (PST) X-Google-Smtp-Source: ACJfBouI6YCx+3jo7m4+rl+h07EHonH12nvx9NNEhdUxaYqCdvdNUyjbwheagkCpDmK/22z4JjNp X-Received: by 10.84.228.214 with SMTP id y22mr36947577pli.444.1516190120697; Wed, 17 Jan 2018 03:55:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1516190120; cv=none; d=google.com; s=arc-20160816; b=yh7s3qcSAJvoW7NiRyXDVREeEoyUe81dUNMQ77IdBl3mYfmXwRR0MTSKtafY0qxOoN rDRRv71o2/tNoLccRN+N39DXS6qWuWr8BBS0eujnvyUQnWkKxbAZoeupoAbsm7VYR8Zl xnCS7hGZvg8nub8PilcMtzTjKU9TLbzahVuLfgM/6BgTw68lD+coZC1VwveGD715VgEL 2VrK6rUhjtH7y1+VtZlfx/IPtewwf1IUcVNGfbjQ6NhO5Z6gxd6ao1czAqxt4Q95kBJr ePtLduIX4FyTbqn8lScGFemBQAt77RbsswK8qbYApsfvmz2w/CB3bPAnuzoWbTxLJTug +3oQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=S10gRvI67aT4sQTz4nNnVk2Mu87O3vwZf1yv8q0/kL8=; b=W7UKJf+zfu7shQcXuoMJhzLIOgGlwQjqvJhi9SgTelXTupHMrf0DwE4xaZzw/csg5u jwZIVH9bMQrURf1bGm5nnYwbhY+uu8LzlP0BRZ0hOcYC7bsgVGuxZDFv65meQUEK6WIZ NpAiT2YoEwXo/s+LGXuNHALgFT2Z1SUXBwsRb+dkCkXt+HRPa2szDetRMXfTCFXqAyBN T8E5taRSiSy0urmvBfDfxpMkpjCDf0C7FH7gATS7FzLWIlfcZANKAr0jljwoeGGKIwMD jW/wv0vxh/lFTHXax1rrDEQZ9bDPoymsYjIewVctKOshN5rI4QRJ+ZKAqu9psHLw9G18 MhbA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id v4si3731573pgf.103.2018.01.17.03.55.20; Wed, 17 Jan 2018 03:55:20 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753153AbeAQLzQ (ORCPT + 28 others); Wed, 17 Jan 2018 06:55:16 -0500 Received: from foss.arm.com ([217.140.101.70]:39136 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752664AbeAQLzN (ORCPT ); Wed, 17 Jan 2018 06:55:13 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2C68815A2; Wed, 17 Jan 2018 03:55:13 -0800 (PST) Received: from e112298-lin.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 83BE93F53D; Wed, 17 Jan 2018 03:55:11 -0800 (PST) From: Julien Thierry To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: mark.rutland@arm.com, marc.zyngier@arm.com, james.morse@arm.com, daniel.thompson@linaro.org, Julien Thierry , Catalin Marinas , Will Deacon , Suzuki K Poulose Subject: [PATCH v2 1/6] arm64: cpufeature: Allow early detect of specific features Date: Wed, 17 Jan 2018 11:54:39 +0000 Message-Id: <1516190084-18978-2-git-send-email-julien.thierry@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1516190084-18978-1-git-send-email-julien.thierry@arm.com> References: <1516190084-18978-1-git-send-email-julien.thierry@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Daniel Thompson Currently it is not possible to detect features of the boot CPU until the other CPUs have been brought up. This prevents us from reacting to features of the boot CPU until fairly late in the boot process. To solve this we allow a subset of features (that are likely to be common to all clusters) to be detected based on the boot CPU alone. Signed-off-by: Daniel Thompson [julien.thierry@arm.com: check non-boot cpu missing early features, avoid duplicates between early features and normal features] Signed-off-by: Julien Thierry Cc: Catalin Marinas Cc: Will Deacon Cc: Suzuki K Poulose --- arch/arm64/kernel/cpufeature.c | 69 ++++++++++++++++++++++++++++-------------- 1 file changed, 47 insertions(+), 22 deletions(-) -- 1.9.1 diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index a73a592..6698404 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -52,6 +52,8 @@ DECLARE_BITMAP(cpu_hwcaps, ARM64_NCAPS); EXPORT_SYMBOL(cpu_hwcaps); +static void __init setup_early_feature_capabilities(void); + /* * Flag to indicate if we have computed the system wide * capabilities based on the boot time active CPUs. This @@ -542,6 +544,8 @@ void __init init_cpu_features(struct cpuinfo_arm64 *info) init_cpu_ftr_reg(SYS_ZCR_EL1, info->reg_zcr); sve_init_vq_map(); } + + setup_early_feature_capabilities(); } static void update_cpu_ftr_reg(struct arm64_ftr_reg *reg, u64 new) @@ -846,7 +850,7 @@ static bool has_no_fpsimd(const struct arm64_cpu_capabilities *entry, int __unus ID_AA64PFR0_FP_SHIFT) < 0; } -static const struct arm64_cpu_capabilities arm64_features[] = { +static const struct arm64_cpu_capabilities arm64_early_features[] = { { .desc = "GIC system register CPU interface", .capability = ARM64_HAS_SYSREG_GIC_CPUIF, @@ -857,6 +861,10 @@ static bool has_no_fpsimd(const struct arm64_cpu_capabilities *entry, int __unus .sign = FTR_UNSIGNED, .min_field_value = 1, }, + {} +}; + +static const struct arm64_cpu_capabilities arm64_features[] = { #ifdef CONFIG_ARM64_PAN { .desc = "Privileged Access Never", @@ -1111,6 +1119,29 @@ void __init enable_cpu_capabilities(const struct arm64_cpu_capabilities *caps) } } +/* Returns false on a capability mismatch */ +static bool +verify_local_cpu_features(const struct arm64_cpu_capabilities *caps) +{ + for (; caps->matches; caps++) { + if (!cpus_have_cap(caps->capability)) + continue; + /* + * If the new CPU misses an advertised feature, we cannot + * proceed further, park the cpu. + */ + if (!caps->matches(caps, SCOPE_LOCAL_CPU)) { + pr_crit("CPU%d: missing feature: %s\n", + smp_processor_id(), caps->desc); + return false; + } + if (caps->enable) + caps->enable(NULL); + } + + return true; +} + /* * Check for CPU features that are used in early boot * based on the Boot CPU value. @@ -1119,6 +1150,9 @@ static void check_early_cpu_features(void) { verify_cpu_run_el(); verify_cpu_asid_bits(); + + if (!verify_local_cpu_features(arm64_early_features)) + cpu_panic_kernel(); } static void @@ -1133,26 +1167,6 @@ static void check_early_cpu_features(void) } } -static void -verify_local_cpu_features(const struct arm64_cpu_capabilities *caps) -{ - for (; caps->matches; caps++) { - if (!cpus_have_cap(caps->capability)) - continue; - /* - * If the new CPU misses an advertised feature, we cannot proceed - * further, park the cpu. - */ - if (!caps->matches(caps, SCOPE_LOCAL_CPU)) { - pr_crit("CPU%d: missing feature: %s\n", - smp_processor_id(), caps->desc); - cpu_die_early(); - } - if (caps->enable) - caps->enable(NULL); - } -} - static void verify_sve_features(void) { u64 safe_zcr = read_sanitised_ftr_reg(SYS_ZCR_EL1); @@ -1181,7 +1195,10 @@ static void verify_sve_features(void) static void verify_local_cpu_capabilities(void) { verify_local_cpu_errata_workarounds(); - verify_local_cpu_features(arm64_features); + + if (!verify_local_cpu_features(arm64_features)) + cpu_die_early(); + verify_local_elf_hwcaps(arm64_elf_hwcaps); if (system_supports_32bit_el0()) @@ -1211,6 +1228,13 @@ void check_local_cpu_capabilities(void) verify_local_cpu_capabilities(); } +static void __init setup_early_feature_capabilities(void) +{ + update_cpu_capabilities(arm64_early_features, + "early detected feature:"); + enable_cpu_capabilities(arm64_early_features); +} + static void __init setup_feature_capabilities(void) { update_cpu_capabilities(arm64_features, "detected feature:"); @@ -1249,6 +1273,7 @@ static bool __this_cpu_has_cap(const struct arm64_cpu_capabilities *cap_array, bool this_cpu_has_cap(unsigned int cap) { return (__this_cpu_has_cap(arm64_features, cap) || + __this_cpu_has_cap(arm64_early_features, cap) || __this_cpu_has_cap(arm64_errata, cap)); } From patchwork Wed Jan 17 11:54:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Thierry X-Patchwork-Id: 124804 Delivered-To: patch@linaro.org Received: by 10.46.62.1 with SMTP id l1csp11724lja; Wed, 17 Jan 2018 03:56:29 -0800 (PST) X-Google-Smtp-Source: ACJfBoudI/oSPVccuB0COtb6pEBOxEzUS852oDJc3YVgBrB0WO8W5Kd7BI4bpam7MKNE0H/o40T9 X-Received: by 10.99.165.87 with SMTP id r23mr12428815pgu.93.1516190189218; Wed, 17 Jan 2018 03:56:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1516190189; cv=none; d=google.com; s=arc-20160816; b=V58p3KGGVI7P6K/cCpcpptXS+WHldmu8aaW1vN6+0BdSll+0tIXEUtxFfheJYR8FNX 7E3oby3D/DlX4R+UUN9W/PpH3i70HQhS2fWKWbxKR4t11IQKU2LbERHe/9zGj2ln5fHS x6npa/qe61mVpbyzKeLoLr08xZCTjQOqv80NrkFowIDKJjzg/z7KCZ/jb7kPre7K8aMW ERAl3vYv6C/2OfOM4qs5yqnL0zrz8QLiBE1f5CNdscib/6QzTvDD11kDSmnAlQppKBTO gOcblCeUOsVUORaSurAT77vGUAYKc/Ndtap/wpWCHFfPleI4hlC9HCHev38DN5Nl+Qel Ejsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=JBaDJUY/aBUFpCEhsd0a5rur5UCWwAfixcIuIfZWQOQ=; b=bLIRj6Z/1Zi5CSleI+EgnX1ZmIayVweVHdFLMLTFBlAVtnY8ECoNdK1b03eMXdg/cf SiKkRiiYb/yuHfUKR+T5EeB3WOrj7vsE7xQ9DVOGpcpjjViCA7Mh8+BMiVwmPEUiabP8 GASTqv2q6Xmay8zloCKGAmfCfyERvGusHYyShJvV5Z4FVSiv2P6rRuZOz1kcXuB8vKU4 29BLZYTt1ExAJdgRcQ0x6xBozrRmH5OwBI2QmLSXtkAp2otZ5ByITJxgC6Y0767P805Q AMRUXSDtPZq9XPpsRYyu/2/AJ7PFBPPTOvs2s499prDSj+VZyt62Y5hTpr+DQnrHZFdq jLoQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id z6si3773894plo.200.2018.01.17.03.56.28; Wed, 17 Jan 2018 03:56:29 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753238AbeAQL40 (ORCPT + 28 others); Wed, 17 Jan 2018 06:56:26 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:39142 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753009AbeAQLzP (ORCPT ); Wed, 17 Jan 2018 06:55:15 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E515215AD; Wed, 17 Jan 2018 03:55:14 -0800 (PST) Received: from e112298-lin.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 6BD653F53D; Wed, 17 Jan 2018 03:55:13 -0800 (PST) From: Julien Thierry To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: mark.rutland@arm.com, marc.zyngier@arm.com, james.morse@arm.com, daniel.thompson@linaro.org, Julien Thierry , Catalin Marinas , Will Deacon Subject: [PATCH v2 2/6] arm64: alternative: Apply alternatives early in boot process Date: Wed, 17 Jan 2018 11:54:40 +0000 Message-Id: <1516190084-18978-3-git-send-email-julien.thierry@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1516190084-18978-1-git-send-email-julien.thierry@arm.com> References: <1516190084-18978-1-git-send-email-julien.thierry@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Daniel Thompson Currently alternatives are applied very late in the boot process (and a long time after we enable scheduling). Some alternative sequences, such as those that alter the way CPU context is stored, must be applied much earlier in the boot sequence. Introduce apply_alternatives_early() to allow some alternatives to be applied immediately after we detect the CPU features of the boot CPU. Signed-off-by: Daniel Thompson Signed-off-by: Julien Thierry Cc: Catalin Marinas Cc: Will Deacon --- arch/arm64/include/asm/alternative.h | 1 + arch/arm64/kernel/alternative.c | 39 +++++++++++++++++++++++++++++++++--- arch/arm64/kernel/smp.c | 6 ++++++ 3 files changed, 43 insertions(+), 3 deletions(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/alternative.h b/arch/arm64/include/asm/alternative.h index 4a85c69..1fc1cdb 100644 --- a/arch/arm64/include/asm/alternative.h +++ b/arch/arm64/include/asm/alternative.h @@ -20,6 +20,7 @@ struct alt_instr { u8 alt_len; /* size of new instruction(s), <= orig_len */ }; +void __init apply_alternatives_early(void); void __init apply_alternatives_all(void); void apply_alternatives(void *start, size_t length); diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c index 6dd0a3a3..78051d4 100644 --- a/arch/arm64/kernel/alternative.c +++ b/arch/arm64/kernel/alternative.c @@ -28,6 +28,18 @@ #include #include +/* + * early-apply features are detected using only the boot CPU and checked on + * secondary CPUs startup, even then, + * These early-apply features should only include features where we must + * patch the kernel very early in the boot process. + * + * Note that the cpufeature logic *must* be made aware of early-apply + * features to ensure they are reported as enabled without waiting + * for other CPUs to boot. + */ +#define EARLY_APPLY_FEATURE_MASK BIT(ARM64_HAS_SYSREG_GIC_CPUIF) + #define __ALT_PTR(a,f) ((void *)&(a)->f + (a)->f) #define ALT_ORIG_PTR(a) __ALT_PTR(a, orig_offset) #define ALT_REPL_PTR(a) __ALT_PTR(a, alt_offset) @@ -105,7 +117,8 @@ static u32 get_alt_insn(struct alt_instr *alt, __le32 *insnptr, __le32 *altinsnp return insn; } -static void __apply_alternatives(void *alt_region, bool use_linear_alias) +static void __apply_alternatives(void *alt_region, bool use_linear_alias, + unsigned long feature_mask) { struct alt_instr *alt; struct alt_region *region = alt_region; @@ -115,6 +128,9 @@ static void __apply_alternatives(void *alt_region, bool use_linear_alias) u32 insn; int i, nr_inst; + if ((BIT(alt->cpufeature) & feature_mask) == 0) + continue; + if (!cpus_have_cap(alt->cpufeature)) continue; @@ -138,6 +154,21 @@ static void __apply_alternatives(void *alt_region, bool use_linear_alias) } /* + * This is called very early in the boot process (directly after we run + * a feature detect on the boot CPU). No need to worry about other CPUs + * here. + */ +void apply_alternatives_early(void) +{ + struct alt_region region = { + .begin = (struct alt_instr *)__alt_instructions, + .end = (struct alt_instr *)__alt_instructions_end, + }; + + __apply_alternatives(®ion, true, EARLY_APPLY_FEATURE_MASK); +} + +/* * We might be patching the stop_machine state machine, so implement a * really simple polling protocol here. */ @@ -156,7 +187,9 @@ static int __apply_alternatives_multi_stop(void *unused) isb(); } else { BUG_ON(patched); - __apply_alternatives(®ion, true); + + __apply_alternatives(®ion, true, ~EARLY_APPLY_FEATURE_MASK); + /* Barriers provided by the cache flushing */ WRITE_ONCE(patched, 1); } @@ -177,5 +210,5 @@ void apply_alternatives(void *start, size_t length) .end = start + length, }; - __apply_alternatives(®ion, false); + __apply_alternatives(®ion, false, -1); } diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 551eb07..37361b5 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -453,6 +453,12 @@ void __init smp_prepare_boot_cpu(void) * cpuinfo_store_boot_cpu() above. */ update_cpu_errata_workarounds(); + /* + * We now know enough about the boot CPU to apply the + * alternatives that cannot wait until interrupt handling + * and/or scheduling is enabled. + */ + apply_alternatives_early(); } static u64 __init of_get_cpu_mpidr(struct device_node *dn) From patchwork Wed Jan 17 11:54:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Thierry X-Patchwork-Id: 124803 Delivered-To: patch@linaro.org Received: by 10.46.62.1 with SMTP id l1csp11390lja; Wed, 17 Jan 2018 03:55:29 -0800 (PST) X-Google-Smtp-Source: ACJfBotwFk05D1JvjYbynG5ZtoZfafVLE9mT/my+RYuucrk/6IusRDdl/8T3JIOD4XjgsE4KZM3s X-Received: by 10.84.214.130 with SMTP id j2mr31178665pli.339.1516190129614; Wed, 17 Jan 2018 03:55:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1516190129; cv=none; d=google.com; s=arc-20160816; b=Q+io0DBsU146Rm+MKYwaBojSQ/4922mujHG2Od6I8CgaEl34KtmDs5G455pR6AkyO/ YkC+IhlytCXh2LCbe7ZbgaDV62BMGeU3kYdfVI1DmTyyCNkF+y/xoNVzDU00ewiQO7xd HX1T21XtlyACr6Ir3Uk1UGes5Ch3hWgzupg3Ph2yJp8E1BhQ0nzgUEAgtB+itPETqimD 4srDH75+2io4FXob1BN2xQx1aG2kz5FeiQnOmXdrn62X8Vp8on5bZJrJ+S2vs2mAwXu/ vhetobSRhmvLbmhSnEOKbA3UaY2FBUL9hV7Re8o268TUjKM9yBGn7i7uUx7bKXxxhrUc Xhog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=Z5pa2MZ5VqPNU6mx9pXHg8AWl0NeWIdjvygbpZa2ldk=; b=jd25g/MMka8zrkBdQ3nOUUy+e3VoMotw1N86K0mdyjJWYMHY4cpQQ8/t8OoE8XHpUo lFPFQCwB/0FLolPtq92te3CaUH3SPxwt3u/1rQQw4xfQZnxO2rnXnZZDglPKiyvJ+Q7r eBvV+MtuJhuC2u4nlOrxXi8e7AZ4J1RQPDl3H5+5a0lV/sqW1KHSMhcLuTRblsDMBq0P wA65YD+MMna1ekad8fKQ7ZCs+YQOUL93VNTAUL18GsTjT3hXvcwZl8LQZxd2uXBSWgZD yV8Wc3ayJJHJQlRZMwFb9FZg9smLFy3tBPRcRBRpE3SzMMp6+t0KugtcZ7WoXVoURchk iYeQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m2si3737109pgd.498.2018.01.17.03.55.29; Wed, 17 Jan 2018 03:55:29 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753194AbeAQLz0 (ORCPT + 28 others); Wed, 17 Jan 2018 06:55:26 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:39156 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753035AbeAQLzS (ORCPT ); Wed, 17 Jan 2018 06:55:18 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 45E5A1610; Wed, 17 Jan 2018 03:55:17 -0800 (PST) Received: from e112298-lin.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 312CD3F53D; Wed, 17 Jan 2018 03:55:15 -0800 (PST) From: Julien Thierry To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: mark.rutland@arm.com, marc.zyngier@arm.com, james.morse@arm.com, daniel.thompson@linaro.org, Julien Thierry , Catalin Marinas , Will Deacon , Christoffer Dall , Thomas Gleixner , Jason Cooper Subject: [PATCH v2 3/6] arm64: irqflags: Use ICC sysregs to implement IRQ masking Date: Wed, 17 Jan 2018 11:54:41 +0000 Message-Id: <1516190084-18978-4-git-send-email-julien.thierry@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1516190084-18978-1-git-send-email-julien.thierry@arm.com> References: <1516190084-18978-1-git-send-email-julien.thierry@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Daniel Thompson Currently irqflags is implemented using the PSR's I bit. It is possible to implement irqflags by using the co-processor interface to the GIC. Using the co-processor interface makes it feasible to simulate NMIs using GIC interrupt prioritization. This patch changes the irqflags macros to modify, save and restore ICC_PMR_EL1. This has a substantial knock on effect for the rest of the kernel. There are four reasons for this: 1. The state of the PMR becomes part of the interrupt context and must be saved and restored during exceptions. It is saved on the stack as part of the saved context when an interrupt/exception is taken. 2. The hardware automatically masks the I bit (at boot, during traps, etc). I bit status is inherited in the different kernel entry types and PMR value is unaffected by exception. So once the I bit is inherited, IRQ flags are back to the same state as before the exception. In the interrupt entry, however, daif flags are not inherited. Switching from I bit masking to PMR masking is done after acknowledging the interrupt (otherwise PMR would prevent the IRQ ack). 3. Some instructions, such as wfi, require that the PMR not be used for interrupt masking. Before calling these instructions we must switch from PMR masking to I bit masking. This is also the case when KVM runs a guest, if the CPU receives an interrupt from the host, interrupts must not be masked in PMR otherwise the GIC will not signal it to the CPU. 4. We use the alternatives system to allow a single kernel to boot and be switched to the alternative masking approach at runtime. Signed-off-by: Daniel Thompson [julien.thierry@arm.com: changes reflected in commit, message, fixes, renaming] Signed-off-by: Julien Thierry Cc: Catalin Marinas Cc: Will Deacon Cc: Christoffer Dall Cc: Marc Zyngier Cc: Thomas Gleixner Cc: Jason Cooper Cc: James Morse --- arch/arm64/Kconfig | 15 ++++ arch/arm64/include/asm/arch_gicv3.h | 37 ++++++++++ arch/arm64/include/asm/assembler.h | 23 +++++- arch/arm64/include/asm/daifflags.h | 36 +++++++--- arch/arm64/include/asm/efi.h | 5 ++ arch/arm64/include/asm/irqflags.h | 125 +++++++++++++++++++++++++++++++++ arch/arm64/include/asm/processor.h | 4 ++ arch/arm64/include/asm/ptrace.h | 14 +++- arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/entry.S | 28 +++++++- arch/arm64/kernel/head.S | 38 ++++++++++ arch/arm64/kernel/process.c | 6 ++ arch/arm64/kernel/smp.c | 8 +++ arch/arm64/kvm/hyp/hyp-entry.S | 20 ++++++ arch/arm64/kvm/hyp/switch.c | 21 ++++++ arch/arm64/mm/proc.S | 23 ++++++ drivers/irqchip/irq-gic-v3-its.c | 2 +- drivers/irqchip/irq-gic-v3.c | 82 +++++++++++---------- include/linux/irqchip/arm-gic-common.h | 6 ++ include/linux/irqchip/arm-gic.h | 5 -- 20 files changed, 439 insertions(+), 60 deletions(-) -- 1.9.1 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index c9a7e9e..9834ff4 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -850,6 +850,21 @@ config FORCE_MAX_ZONEORDER However for 4K, we choose a higher default value, 11 as opposed to 10, giving us 4M allocations matching the default size used by generic code. +config USE_ICC_SYSREGS_FOR_IRQFLAGS + bool "Use ICC system registers for IRQ masking" + select CONFIG_ARM_GIC_V3 + help + Using the ICC system registers for IRQ masking makes it possible + to simulate NMI on ARM64 systems. This allows several interesting + features (especially debug features) to be used on these systems. + + Say Y here to implement IRQ masking using ICC system + registers when the GIC System Registers are available. The changes + are applied dynamically using the alternatives system so it is safe + to enable this option on systems with older interrupt controllers. + + If unsure, say N + menuconfig ARMV8_DEPRECATED bool "Emulate deprecated/obsolete ARMv8 instructions" depends on COMPAT diff --git a/arch/arm64/include/asm/arch_gicv3.h b/arch/arm64/include/asm/arch_gicv3.h index 9becba9..490bb3a 100644 --- a/arch/arm64/include/asm/arch_gicv3.h +++ b/arch/arm64/include/asm/arch_gicv3.h @@ -76,6 +76,11 @@ static inline u64 gic_read_iar_cavium_thunderx(void) return irqstat; } +static inline u32 gic_read_pmr(void) +{ + return read_sysreg_s(SYS_ICC_PMR_EL1); +} + static inline void gic_write_pmr(u32 val) { write_sysreg_s(val, SYS_ICC_PMR_EL1); @@ -145,5 +150,37 @@ static inline void gic_write_bpr1(u32 val) #define gits_write_vpendbaser(v, c) writeq_relaxed(v, c) #define gits_read_vpendbaser(c) readq_relaxed(c) +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS +static inline void gic_start_pmr_masking(void) +{ + if (cpus_have_const_cap(ARM64_HAS_SYSREG_GIC_CPUIF)) { + gic_write_pmr(ICC_PMR_EL1_MASKED); + asm volatile ("msr daifclr, #2" : : : "memory"); + } +} + +static inline u32 gic_pmr_save_and_unmask(void) +{ + if (cpus_have_const_cap(ARM64_HAS_SYSREG_GIC_CPUIF)) { + u32 old; + + old = gic_read_pmr(); + gic_write_pmr(ICC_PMR_EL1_UNMASKED); + dsb(sy); + + return old; + } else { + /* Idle priority, no masking */ + return ICC_PMR_EL1_UNMASKED; + } +} + +static inline void gic_pmr_restore(u32 pmr) +{ + if (cpus_have_const_cap(ARM64_HAS_SYSREG_GIC_CPUIF)) + gic_write_pmr(pmr); +} +#endif + #endif /* __ASSEMBLY__ */ #endif /* __ASM_ARCH_GICV3_H */ diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 8b16828..d320bd6 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -23,6 +23,7 @@ #ifndef __ASM_ASSEMBLER_H #define __ASM_ASSEMBLER_H +#include #include #include #include @@ -63,12 +64,30 @@ /* * Enable and disable interrupts. */ - .macro disable_irq + .macro disable_irq, tmp +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS + mov \tmp, #ICC_PMR_EL1_MASKED +alternative_if_not ARM64_HAS_SYSREG_GIC_CPUIF msr daifset, #2 +alternative_else + msr_s SYS_ICC_PMR_EL1, \tmp +alternative_endif +#else + msr daifset, #2 +#endif .endm - .macro enable_irq + .macro enable_irq, tmp +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS + mov \tmp, #ICC_PMR_EL1_UNMASKED +alternative_if_not ARM64_HAS_SYSREG_GIC_CPUIF msr daifclr, #2 +alternative_else + msr_s SYS_ICC_PMR_EL1, \tmp +alternative_endif +#else + msr daifclr, #2 +#endif .endm .macro save_and_disable_irq, flags diff --git a/arch/arm64/include/asm/daifflags.h b/arch/arm64/include/asm/daifflags.h index 22e4c83..ba85822 100644 --- a/arch/arm64/include/asm/daifflags.h +++ b/arch/arm64/include/asm/daifflags.h @@ -18,9 +18,24 @@ #include +#ifndef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS + #define DAIF_PROCCTX 0 #define DAIF_PROCCTX_NOIRQ PSR_I_BIT +#else + +#define DAIF_PROCCTX \ + (cpus_have_const_cap(ARM64_HAS_SYSREG_GIC_CPUIF) ? \ + MAKE_ARCH_FLAGS(0, ICC_PMR_EL1_UNMASKED) : \ + 0) + +#define DAIF_PROCCTX_NOIRQ \ + (cpus_have_const_cap(ARM64_HAS_SYSREG_GIC_CPUIF) ? \ + MAKE_ARCH_FLAGS(0, ICC_PMR_EL1_MASKED) : \ + PSR_I_BIT) +#endif + /* mask/save/unmask/restore all exceptions, including interrupts. */ static inline void local_daif_mask(void) { @@ -36,11 +51,8 @@ static inline unsigned long local_daif_save(void) { unsigned long flags; - asm volatile( - "mrs %0, daif // local_daif_save\n" - : "=r" (flags) - : - : "memory"); + flags = arch_local_save_flags(); + local_daif_mask(); return flags; @@ -54,17 +66,21 @@ static inline void local_daif_unmask(void) : : : "memory"); + +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS + /* Unmask IRQs in PMR if needed */ + if (cpus_have_const_cap(ARM64_HAS_SYSREG_GIC_CPUIF)) + arch_local_irq_enable(); +#endif } static inline void local_daif_restore(unsigned long flags) { if (!arch_irqs_disabled_flags(flags)) trace_hardirqs_on(); - asm volatile( - "msr daif, %0 // local_daif_restore" - : - : "r" (flags) - : "memory"); + + arch_local_irq_restore(flags); + if (arch_irqs_disabled_flags(flags)) trace_hardirqs_off(); } diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h index c4cd508..421525f 100644 --- a/arch/arm64/include/asm/efi.h +++ b/arch/arm64/include/asm/efi.h @@ -40,7 +40,12 @@ efi_virtmap_unload(); \ }) +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS +#define ARCH_EFI_IRQ_FLAGS_MASK \ + (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT | ARCH_FLAG_PMR_EN) +#else #define ARCH_EFI_IRQ_FLAGS_MASK (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT) +#endif /* arch specific definitions used by the stub code */ diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h index 24692ed..3d5d443 100644 --- a/arch/arm64/include/asm/irqflags.h +++ b/arch/arm64/include/asm/irqflags.h @@ -18,7 +18,10 @@ #ifdef __KERNEL__ +#include +#include #include +#include /* * Aarch64 has flags for masking: Debug, Asynchronous (serror), Interrupts and @@ -33,6 +36,7 @@ * unmask it at all other times. */ +#ifndef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS /* * CPU interrupt mask handling. */ @@ -96,5 +100,126 @@ static inline int arch_irqs_disabled_flags(unsigned long flags) { return flags & PSR_I_BIT; } + +static inline void maybe_switch_to_sysreg_gic_cpuif(void) {} + +#else /* CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS */ + +#define ARCH_FLAG_PMR_EN 0x1 + +#define MAKE_ARCH_FLAGS(daif, pmr) \ + ((daif) | (((pmr) >> ICC_PMR_EL1_EN_SHIFT) & ARCH_FLAG_PMR_EN)) + +#define ARCH_FLAGS_GET_PMR(flags) \ + ((((flags) & ARCH_FLAG_PMR_EN) << ICC_PMR_EL1_EN_SHIFT) \ + | ICC_PMR_EL1_MASKED) + +#define ARCH_FLAGS_GET_DAIF(flags) ((flags) & ~ARCH_FLAG_PMR_EN) + +/* + * CPU interrupt mask handling. + */ +static inline unsigned long arch_local_irq_save(void) +{ + unsigned long flags, masked = ICC_PMR_EL1_MASKED; + unsigned long pmr = 0; + + asm volatile(ALTERNATIVE( + "mrs %0, daif // arch_local_irq_save\n" + "msr daifset, #2\n" + "mov %1, #" __stringify(ICC_PMR_EL1_UNMASKED), + /* --- */ + "mrs %0, daif\n" + "mrs_s %1, " __stringify(SYS_ICC_PMR_EL1) "\n" + "msr_s " __stringify(SYS_ICC_PMR_EL1) ", %2", + ARM64_HAS_SYSREG_GIC_CPUIF) + : "=&r" (flags), "=&r" (pmr) + : "r" (masked) + : "memory"); + + return MAKE_ARCH_FLAGS(flags, pmr); +} + +static inline void arch_local_irq_enable(void) +{ + unsigned long unmasked = ICC_PMR_EL1_UNMASKED; + + asm volatile(ALTERNATIVE( + "msr daifclr, #2 // arch_local_irq_enable\n" + "nop", + "msr_s " __stringify(SYS_ICC_PMR_EL1) ",%0\n" + "dsb sy", + ARM64_HAS_SYSREG_GIC_CPUIF) + : + : "r" (unmasked) + : "memory"); +} + +static inline void arch_local_irq_disable(void) +{ + unsigned long masked = ICC_PMR_EL1_MASKED; + + asm volatile(ALTERNATIVE( + "msr daifset, #2 // arch_local_irq_disable", + "msr_s " __stringify(SYS_ICC_PMR_EL1) ",%0", + ARM64_HAS_SYSREG_GIC_CPUIF) + : + : "r" (masked) + : "memory"); +} + +/* + * Save the current interrupt enable state. + */ +static inline unsigned long arch_local_save_flags(void) +{ + unsigned long flags; + unsigned long pmr = 0; + + asm volatile(ALTERNATIVE( + "mrs %0, daif // arch_local_save_flags\n" + "mov %1, #" __stringify(ICC_PMR_EL1_UNMASKED), + "mrs %0, daif\n" + "mrs_s %1, " __stringify(SYS_ICC_PMR_EL1), + ARM64_HAS_SYSREG_GIC_CPUIF) + : "=r" (flags), "=r" (pmr) + : + : "memory"); + + return MAKE_ARCH_FLAGS(flags, pmr); +} + +/* + * restore saved IRQ state + */ +static inline void arch_local_irq_restore(unsigned long flags) +{ + unsigned long pmr = ARCH_FLAGS_GET_PMR(flags); + + flags = ARCH_FLAGS_GET_DAIF(flags); + + asm volatile(ALTERNATIVE( + "msr daif, %0 // arch_local_irq_restore\n" + "nop\n" + "nop", + "msr daif, %0\n" + "msr_s " __stringify(SYS_ICC_PMR_EL1) ",%1\n" + "dsb sy", + ARM64_HAS_SYSREG_GIC_CPUIF) + : + : "r" (flags), "r" (pmr) + : "memory"); +} + +static inline int arch_irqs_disabled_flags(unsigned long flags) +{ + return (ARCH_FLAGS_GET_DAIF(flags) & (PSR_I_BIT)) | + !(ARCH_FLAGS_GET_PMR(flags) & ICC_PMR_EL1_EN_BIT); +} + +void maybe_switch_to_sysreg_gic_cpuif(void); + +#endif /* CONFIG_IRQFLAGS_GIC_MASKING */ + #endif #endif diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 023cacb..d569dee 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -137,6 +137,10 @@ static inline void start_thread_common(struct pt_regs *regs, unsigned long pc) memset(regs, 0, sizeof(*regs)); forget_syscall(regs); regs->pc = pc; +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS + /* Have IRQs enabled by default */ + regs->pmr_save = ICC_PMR_EL1_UNMASKED; +#endif } static inline void start_thread(struct pt_regs *regs, unsigned long pc, diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h index 6069d66..aa1e948 100644 --- a/arch/arm64/include/asm/ptrace.h +++ b/arch/arm64/include/asm/ptrace.h @@ -25,6 +25,12 @@ #define CurrentEL_EL1 (1 << 2) #define CurrentEL_EL2 (2 << 2) +/* PMR values used to mask/unmask interrupts */ +#define ICC_PMR_EL1_EN_SHIFT 6 +#define ICC_PMR_EL1_EN_BIT (1 << ICC_PMR_EL1_EN_SHIFT) // PMR IRQ enable +#define ICC_PMR_EL1_UNMASKED 0xf0 +#define ICC_PMR_EL1_MASKED (ICC_PMR_EL1_UNMASKED ^ ICC_PMR_EL1_EN_BIT) + /* AArch32-specific ptrace requests */ #define COMPAT_PTRACE_GETREGS 12 #define COMPAT_PTRACE_SETREGS 13 @@ -136,7 +142,7 @@ struct pt_regs { #endif u64 orig_addr_limit; - u64 unused; // maintain 16 byte alignment + u64 pmr_save; u64 stackframe[2]; }; @@ -171,8 +177,14 @@ static inline void forget_syscall(struct pt_regs *regs) #define processor_mode(regs) \ ((regs)->pstate & PSR_MODE_MASK) +#ifndef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS #define interrupts_enabled(regs) \ (!((regs)->pstate & PSR_I_BIT)) +#else +#define interrupts_enabled(regs) \ + ((!((regs)->pstate & PSR_I_BIT)) && \ + ((regs)->pmr_save & ICC_PMR_EL1_EN_BIT)) +#endif #define fast_interrupts_enabled(regs) \ (!((regs)->pstate & PSR_F_BIT)) diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 71bf088..6b00b0d 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -75,6 +75,7 @@ int main(void) DEFINE(S_ORIG_X0, offsetof(struct pt_regs, orig_x0)); DEFINE(S_SYSCALLNO, offsetof(struct pt_regs, syscallno)); DEFINE(S_ORIG_ADDR_LIMIT, offsetof(struct pt_regs, orig_addr_limit)); + DEFINE(S_PMR_SAVE, offsetof(struct pt_regs, pmr_save)); DEFINE(S_STACKFRAME, offsetof(struct pt_regs, stackframe)); DEFINE(S_FRAME_SIZE, sizeof(struct pt_regs)); BLANK(); diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 6d14b8f..8209b45 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -20,6 +20,7 @@ #include #include +#include #include #include @@ -210,6 +211,16 @@ alternative_else_nop_endif msr sp_el0, tsk .endif +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS + /* Save pmr */ +alternative_if ARM64_HAS_SYSREG_GIC_CPUIF + mrs_s x20, SYS_ICC_PMR_EL1 +alternative_else + mov x20, #ICC_PMR_EL1_UNMASKED +alternative_endif + str x20, [sp, #S_PMR_SAVE] +#endif + /* * Registers that may be useful after this macro is invoked: * @@ -230,6 +241,15 @@ alternative_else_nop_endif /* No need to restore UAO, it will be restored from SPSR_EL1 */ .endif +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS + /* Restore pmr, ensuring IRQs are off before restoring context. */ +alternative_if ARM64_HAS_SYSREG_GIC_CPUIF + ldr x20, [sp, #S_PMR_SAVE] + msr_s SYS_ICC_PMR_EL1, x20 + dsb sy +alternative_else_nop_endif +#endif + ldp x21, x22, [sp, #S_PC] // load ELR, SPSR .if \el == 0 ct_user_enter @@ -820,17 +840,18 @@ ENDPROC(el0_error) * and this includes saving x0 back into the kernel stack. */ ret_fast_syscall: - disable_daif + disable_irq x21 // disable interrupts str x0, [sp, #S_X0] // returned x0 ldr x1, [tsk, #TSK_TI_FLAGS] // re-check for syscall tracing and x2, x1, #_TIF_SYSCALL_WORK cbnz x2, ret_fast_syscall_trace and x2, x1, #_TIF_WORK_MASK cbnz x2, work_pending + disable_daif enable_step_tsk x1, x2 kernel_exit 0 ret_fast_syscall_trace: - enable_daif + enable_daif // enable interrupts b __sys_trace_return_skipped // we already saved x0 /* @@ -848,11 +869,12 @@ work_pending: * "slow" syscall return path. */ ret_to_user: - disable_daif + disable_irq x21 // disable interrupts ldr x1, [tsk, #TSK_TI_FLAGS] and x2, x1, #_TIF_WORK_MASK cbnz x2, work_pending finish_ret_to_user: + disable_daif enable_step_tsk x1, x2 kernel_exit 0 ENDPROC(ret_to_user) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index e3cb9fb..ec2eb4a 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -563,6 +563,44 @@ set_cpu_boot_mode_flag: ret ENDPROC(set_cpu_boot_mode_flag) +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS +/* + * void maybe_switch_to_sysreg_gic_cpuif(void) + * + * Enable interrupt controller system register access if this feature + * has been detected by the alternatives system. + * + * Before we jump into generic code we must enable interrupt controller system + * register access because this is required by the irqflags macros. We must + * also mask interrupts at the PMR and unmask them within the PSR. That leaves + * us set up and ready for the kernel to make its first call to + * arch_local_irq_enable(). + + * + */ +ENTRY(maybe_switch_to_sysreg_gic_cpuif) +alternative_if_not ARM64_HAS_SYSREG_GIC_CPUIF + b 1f +alternative_else + mrs_s x0, SYS_ICC_SRE_EL1 +alternative_endif + orr x0, x0, #1 + msr_s SYS_ICC_SRE_EL1, x0 // Set ICC_SRE_EL1.SRE==1 + isb // Make sure SRE is now set + mrs x0, daif + tbz x0, #7, no_mask_pmr // Are interrupts on? + mov x0, ICC_PMR_EL1_MASKED + msr_s SYS_ICC_PMR_EL1, x0 // Prepare for unmask of I bit + msr daifclr, #2 // Clear the I bit + b 1f +no_mask_pmr: + mov x0, ICC_PMR_EL1_UNMASKED + msr_s SYS_ICC_PMR_EL1, x0 +1: + ret +ENDPROC(maybe_switch_to_sysreg_gic_cpuif) +#endif + /* * These values are written with the MMU off, but read with the MMU on. * Writers will invalidate the corresponding address, discarding up to a diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 6b7dcf4..56871f2 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -66,6 +66,8 @@ EXPORT_SYMBOL(__stack_chk_guard); #endif +#include + /* * Function pointers to optional machine specific functions */ @@ -224,6 +226,7 @@ void __show_regs(struct pt_regs *regs) print_symbol("pc : %s\n", regs->pc); print_symbol("lr : %s\n", lr); printk("sp : %016llx\n", sp); + printk("pmr_save: %08llx\n", regs->pmr_save); i = top_reg; @@ -349,6 +352,9 @@ int copy_thread(unsigned long clone_flags, unsigned long stack_start, } else { memset(childregs, 0, sizeof(struct pt_regs)); childregs->pstate = PSR_MODE_EL1h; +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS + childregs->pmr_save = ICC_PMR_EL1_UNMASKED; +#endif if (IS_ENABLED(CONFIG_ARM64_UAO) && cpus_have_const_cap(ARM64_HAS_UAO)) childregs->pstate |= PSR_UAO_BIT; diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 37361b5..ec56ee1 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -221,6 +221,8 @@ asmlinkage void secondary_start_kernel(void) struct mm_struct *mm = &init_mm; unsigned int cpu; + maybe_switch_to_sysreg_gic_cpuif(); + cpu = task_cpu(current); set_my_cpu_offset(per_cpu_offset(cpu)); @@ -459,6 +461,12 @@ void __init smp_prepare_boot_cpu(void) * and/or scheduling is enabled. */ apply_alternatives_early(); + + /* + * Conditionally switch to GIC PMR for interrupt masking (this + * will be a nop if we are using normal interrupt masking) + */ + maybe_switch_to_sysreg_gic_cpuif(); } static u64 __init of_get_cpu_mpidr(struct device_node *dn) diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S index 5170ce1..e5e97e8 100644 --- a/arch/arm64/kvm/hyp/hyp-entry.S +++ b/arch/arm64/kvm/hyp/hyp-entry.S @@ -42,7 +42,27 @@ .endm ENTRY(__vhe_hyp_call) +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS +alternative_if ARM64_HAS_SYSREG_GIC_CPUIF + /* + * In non-VHE, trapping to EL2 will set the PSR.I bit. + * Force it here whenever we are playing with PMR. + */ + str x19, [sp, #-16]! + mrs x19, daif + msr daifset, #2 +alternative_else_nop_endif +#endif + do_el2_call + +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS +alternative_if ARM64_HAS_SYSREG_GIC_CPUIF + msr daif, x19 + ldr x19, [sp], #16 +alternative_else_nop_endif +#endif + /* * We used to rely on having an exception return to get * an implicit isb. In the E2H case, we don't have it anymore. diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index f7c651f..4fac70d 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -18,6 +18,9 @@ #include #include +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS +#include +#endif #include #include #include @@ -303,6 +306,19 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu) struct kvm_cpu_context *guest_ctxt; bool fp_enabled; u64 exit_code; +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS + u32 pmr_save; +#endif + +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS + /* + * Having IRQs masked via PMR when entering the guest means the GIC + * will not signal the CPU of interrupts of lower priority, and the + * only way to get out will be via guest exceptions. + * Naturally, we want to avoid this. + */ + pmr_save = gic_pmr_save_and_unmask(); +#endif vcpu = kern_hyp_va(vcpu); write_sysreg(vcpu, tpidr_el2); @@ -417,6 +433,11 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu) */ __debug_cond_restore_host_state(vcpu); +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS + /* PMR was unmasked, no need for dsb */ + gic_pmr_restore(pmr_save); +#endif + return exit_code; } diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 95233df..8b91661 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -20,6 +20,7 @@ #include #include +#include #include #include #include @@ -47,11 +48,33 @@ * cpu_do_idle() * * Idle the processor (wait for interrupt). + * + * If CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS is set we must do additional + * work to ensure that interrupts are not masked at the PMR (because the + * core will not wake up if we block the wake up signal in the interrupt + * controller). */ ENTRY(cpu_do_idle) +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS +alternative_if_not ARM64_HAS_SYSREG_GIC_CPUIF +#endif + dsb sy // WFI may enter a low-power mode + wfi + ret +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS +alternative_else + mrs x0, daif // save I bit + msr daifset, #2 // set I bit + mrs_s x1, SYS_ICC_PMR_EL1 // save PMR +alternative_endif + mov x2, #ICC_PMR_EL1_UNMASKED + msr_s SYS_ICC_PMR_EL1, x2 // unmask at PMR dsb sy // WFI may enter a low-power mode wfi + msr_s SYS_ICC_PMR_EL1, x1 // restore PMR + msr daif, x0 // restore I bit ret +#endif ENDPROC(cpu_do_idle) #ifdef CONFIG_CPU_PM diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c index 06f025f..35e5f45 100644 --- a/drivers/irqchip/irq-gic-v3-its.c +++ b/drivers/irqchip/irq-gic-v3-its.c @@ -60,7 +60,7 @@ #define LPI_PROPBASE_SZ ALIGN(BIT(LPI_NRBITS), SZ_64K) #define LPI_PENDBASE_SZ ALIGN(BIT(LPI_NRBITS) / 8, SZ_64K) -#define LPI_PROP_DEFAULT_PRIO 0xa0 +#define LPI_PROP_DEFAULT_PRIO GICD_INT_DEF_PRI /* * Collection structure - just an ID, and a redistributor address to diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c index b56c3e2..df51d96 100644 --- a/drivers/irqchip/irq-gic-v3.c +++ b/drivers/irqchip/irq-gic-v3.c @@ -71,9 +71,6 @@ struct gic_chip_data { #define gic_data_rdist_rd_base() (gic_data_rdist()->rd_base) #define gic_data_rdist_sgi_base() (gic_data_rdist_rd_base() + SZ_64K) -/* Our default, arbitrary priority value. Linux only uses one anyway. */ -#define DEFAULT_PMR_VALUE 0xf0 - static inline unsigned int gic_irq(struct irq_data *d) { return d->hwirq; @@ -348,48 +345,55 @@ static asmlinkage void __exception_irq_entry gic_handle_irq(struct pt_regs *regs { u32 irqnr; - do { - irqnr = gic_read_iar(); + irqnr = gic_read_iar(); - if (likely(irqnr > 15 && irqnr < 1020) || irqnr >= 8192) { - int err; +#ifdef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS + isb(); + /* Masking IRQs earlier would prevent to ack the current interrupt */ + gic_start_pmr_masking(); +#endif - if (static_key_true(&supports_deactivate)) + if (likely(irqnr > 15 && irqnr < 1020) || irqnr >= 8192) { + int err; + + if (static_key_true(&supports_deactivate)) + gic_write_eoir(irqnr); + else { +#ifndef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS + isb(); +#endif + } + + err = handle_domain_irq(gic_data.domain, irqnr, regs); + if (err) { + WARN_ONCE(true, "Unexpected interrupt received!\n"); + if (static_key_true(&supports_deactivate)) { + if (irqnr < 8192) + gic_write_dir(irqnr); + } else { gic_write_eoir(irqnr); - else - isb(); - - err = handle_domain_irq(gic_data.domain, irqnr, regs); - if (err) { - WARN_ONCE(true, "Unexpected interrupt received!\n"); - if (static_key_true(&supports_deactivate)) { - if (irqnr < 8192) - gic_write_dir(irqnr); - } else { - gic_write_eoir(irqnr); - } } - continue; } - if (irqnr < 16) { - gic_write_eoir(irqnr); - if (static_key_true(&supports_deactivate)) - gic_write_dir(irqnr); + return; + } + if (irqnr < 16) { + gic_write_eoir(irqnr); + if (static_key_true(&supports_deactivate)) + gic_write_dir(irqnr); #ifdef CONFIG_SMP - /* - * Unlike GICv2, we don't need an smp_rmb() here. - * The control dependency from gic_read_iar to - * the ISB in gic_write_eoir is enough to ensure - * that any shared data read by handle_IPI will - * be read after the ACK. - */ - handle_IPI(irqnr, regs); + /* + * Unlike GICv2, we don't need an smp_rmb() here. + * The control dependency from gic_read_iar to + * the ISB in gic_write_eoir is enough to ensure + * that any shared data read by handle_IPI will + * be read after the ACK. + */ + handle_IPI(irqnr, regs); #else - WARN_ONCE(true, "Unexpected SGI received!\n"); + WARN_ONCE(true, "Unexpected SGI received!\n"); #endif - continue; - } - } while (irqnr != ICC_IAR1_EL1_SPURIOUS); + return; + } } static void __init gic_dist_init(void) @@ -543,8 +547,10 @@ static void gic_cpu_sys_reg_init(void) if (!gic_enable_sre()) pr_err("GIC: unable to set SRE (disabled at EL2), panic ahead\n"); +#ifndef CONFIG_USE_ICC_SYSREGS_FOR_IRQFLAGS /* Set priority mask register */ - gic_write_pmr(DEFAULT_PMR_VALUE); + gic_write_pmr(ICC_PMR_EL1_UNMASKED); +#endif /* * Some firmwares hand over to the kernel with the BPR changed from diff --git a/include/linux/irqchip/arm-gic-common.h b/include/linux/irqchip/arm-gic-common.h index 0a83b43..2c9a4b3 100644 --- a/include/linux/irqchip/arm-gic-common.h +++ b/include/linux/irqchip/arm-gic-common.h @@ -13,6 +13,12 @@ #include #include +#define GICD_INT_DEF_PRI 0xc0 +#define GICD_INT_DEF_PRI_X4 ((GICD_INT_DEF_PRI << 24) |\ + (GICD_INT_DEF_PRI << 16) |\ + (GICD_INT_DEF_PRI << 8) |\ + GICD_INT_DEF_PRI) + enum gic_type { GIC_V2, GIC_V3, diff --git a/include/linux/irqchip/arm-gic.h b/include/linux/irqchip/arm-gic.h index d3453ee..47f5a8c 100644 --- a/include/linux/irqchip/arm-gic.h +++ b/include/linux/irqchip/arm-gic.h @@ -65,11 +65,6 @@ #define GICD_INT_EN_CLR_X32 0xffffffff #define GICD_INT_EN_SET_SGI 0x0000ffff #define GICD_INT_EN_CLR_PPI 0xffff0000 -#define GICD_INT_DEF_PRI 0xa0 -#define GICD_INT_DEF_PRI_X4 ((GICD_INT_DEF_PRI << 24) |\ - (GICD_INT_DEF_PRI << 16) |\ - (GICD_INT_DEF_PRI << 8) |\ - GICD_INT_DEF_PRI) #define GICH_HCR 0x0 #define GICH_VTR 0x4