From patchwork Fri Jun 1 17:12:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 137571 Delivered-To: patch@linaro.org Received: by 2002:a2e:9706:0:0:0:0:0 with SMTP id r6-v6csp1198587lji; Fri, 1 Jun 2018 10:12:21 -0700 (PDT) X-Google-Smtp-Source: ADUXVKJXxnaHL8t8MtNnncMFMiFGQw8dqOHLp3Kk9P3zJqf9/FRy8gZvicTXU7LmlG5ke1QpRWCl X-Received: by 2002:a65:43c9:: with SMTP id n9-v6mr9376295pgp.399.1527873141311; Fri, 01 Jun 2018 10:12:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1527873141; cv=none; d=google.com; s=arc-20160816; b=r9jdhWZr8tbVAyWEWNfbU7rwHJKkErFuyvEuqfJ5SKY7ufgJimhlDOPHwgjbNY9jcE BnVqNajXR9LzhZMVH/swjpKkKwa5jke4vuC4vXpkGR+E41BBnws18h54eDycjQf9Zd7M 40ey9uFr9A3RlLtvfUn4DhtxsJQS5RRcRm/Y82YoDsA6TkACWMMO8hJ4R1WM0C2dXWQD nt1pH3JtOg5t2JUVTcKejsA85DTpwefpNptH0SvJ5PXNH1ddYDfcUVusWVlSDBtlzo0T e/70QGBhbWSah2+tcUEaDEFFbPwxy/DQxw2/L4QgDN35qm9F+mtr+Awo7XNFg5gUPP1K c9TQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=8qwjRnOByH6vACvfgtkxNb9HxAPwMT43L6ogreFWJVE=; b=ALqRFvMVmt2o3Wx4pZpTuInmLmNmRtyjyubG5YY8rnZnce96GQIqPoanjrsLK1Uw54 BSWwUEk81V1eAYRFEWKIYvwY7r4d4v8URrpACrPxssR93eIo3/T2gHrKxUPEbXRNnWw8 2LJbuMeG821yyi51dk724YwYqk42cpUpHVvYthrKZKw1fpwpkQY/rHP2K8F6GJGrfGgY 3hKpTLjmFPZo7iHFLI73Z3YQn1qZxJuaek4wNznIhVV1Ix6xx5XFRHjRvh6UoYhJxBYA SjdfdpNN0PWm/zkeOW+otjm04m2gldXIUPCJfItdcIqWtmiWyQIXbCpSl/+qAmUjXW7g WK4Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 84-v6si4135003pfj.6.2018.06.01.10.12.21; Fri, 01 Jun 2018 10:12:21 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752203AbeFARMU (ORCPT + 13 others); Fri, 1 Jun 2018 13:12:20 -0400 Received: from foss.arm.com ([217.140.101.70]:56096 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751964AbeFARMT (ORCPT ); Fri, 1 Jun 2018 13:12:19 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7209515AB; Fri, 1 Jun 2018 10:12:19 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id AB08E3F53D; Fri, 1 Jun 2018 10:12:18 -0700 (PDT) From: Mark Rutland To: stable@vger.kernel.org Cc: marc.zyngier@arm.com, mark.rutland@arm.com Subject: [PATCH v4.9.y 1/2] arm64: Add hypervisor safe helper for checking constant capabilities Date: Fri, 1 Jun 2018 18:12:09 +0100 Message-Id: <20180601171210.16870-2-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180601171210.16870-1-mark.rutland@arm.com> References: <20180601171210.16870-1-mark.rutland@arm.com> Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Suzuki K Poulose commit a4023f682739439b434165b54af7cb3676a4766e upstream. The hypervisor may not have full access to the kernel data structures and hence cannot safely use cpus_have_cap() helper for checking the system capability. Add a safe helper for hypervisors to check a constant system capability, which *doesn't* fall back to checking the bitmap maintained by the kernel. With this, make the cpus_have_cap() only check the bitmask and force constant cap checks to use the new API for quicker checks. Cc: Robert Ritcher Cc: Tirumalesh Chalamarla Signed-off-by: Suzuki K Poulose Reviewed-by: Will Deacon Reviewed-by: Marc Zyngier Signed-off-by: Catalin Marinas [4.9: restore cpus_have_const_cap() to previously-backported code] Signed-off-by: Mark Rutland [v4.9 backport] --- arch/arm64/include/asm/cpufeature.h | 19 ++++++++++++------- arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/include/asm/kvm_mmu.h | 2 +- arch/arm64/include/asm/mmu.h | 2 +- arch/arm64/kernel/cpufeature.c | 5 +++-- arch/arm64/kernel/process.c | 2 +- drivers/irqchip/irq-gic-v3.c | 13 +------------ 7 files changed, 20 insertions(+), 25 deletions(-) -- 2.11.0 diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 0bc0b1de90c4..9890d20f2356 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -9,8 +9,6 @@ #ifndef __ASM_CPUFEATURE_H #define __ASM_CPUFEATURE_H -#include - #include #include #include @@ -27,6 +25,8 @@ #ifndef __ASSEMBLY__ +#include +#include #include /* CPU feature register tracking */ @@ -104,14 +104,19 @@ static inline bool cpu_have_feature(unsigned int num) return elf_hwcap & (1UL << num); } +/* System capability check for constant caps */ +static inline bool cpus_have_const_cap(int num) +{ + if (num >= ARM64_NCAPS) + return false; + return static_branch_unlikely(&cpu_hwcap_keys[num]); +} + static inline bool cpus_have_cap(unsigned int num) { if (num >= ARM64_NCAPS) return false; - if (__builtin_constant_p(num)) - return static_branch_unlikely(&cpu_hwcap_keys[num]); - else - return test_bit(num, cpu_hwcaps); + return test_bit(num, cpu_hwcaps); } static inline void cpus_set_cap(unsigned int num) @@ -200,7 +205,7 @@ static inline bool cpu_supports_mixed_endian_el0(void) static inline bool system_supports_32bit_el0(void) { - return cpus_have_cap(ARM64_HAS_32BIT_EL0); + return cpus_have_const_cap(ARM64_HAS_32BIT_EL0); } static inline bool system_supports_mixed_endian_el0(void) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 0a33ea304e63..7f402f64c744 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -398,7 +398,7 @@ static inline void __cpu_init_stage2(void) static inline bool kvm_arm_harden_branch_predictor(void) { - return cpus_have_cap(ARM64_HARDEN_BRANCH_PREDICTOR); + return cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR); } #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index eac73a640ea7..824c83db9b47 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -341,7 +341,7 @@ static inline void *kvm_get_hyp_vector(void) vect = __bp_harden_hyp_vecs_start + data->hyp_vectors_slot * SZ_2K; - if (!cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN)) + if (!cpus_have_const_cap(ARM64_HAS_VIRT_HOST_EXTN)) vect = lm_alias(vect); } diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index d51158a61892..6ac34c75f4e1 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -37,7 +37,7 @@ typedef struct { static inline bool arm64_kernel_unmapped_at_el0(void) { return IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0) && - cpus_have_cap(ARM64_UNMAP_KERNEL_AT_EL0); + cpus_have_const_cap(ARM64_UNMAP_KERNEL_AT_EL0); } typedef void (*bp_hardening_cb_t)(void); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index a0ee01202503..e28665411bd1 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -47,6 +47,7 @@ unsigned int compat_elf_hwcap2 __read_mostly; #endif DECLARE_BITMAP(cpu_hwcaps, ARM64_NCAPS); +EXPORT_SYMBOL(cpu_hwcaps); DEFINE_STATIC_KEY_ARRAY_FALSE(cpu_hwcap_keys, ARM64_NCAPS); EXPORT_SYMBOL(cpu_hwcap_keys); @@ -762,7 +763,7 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, * ThunderX leads to apparent I-cache corruption of kernel text, which * ends as well as you might imagine. Don't even try. */ - if (cpus_have_cap(ARM64_WORKAROUND_CAVIUM_27456)) { + if (cpus_have_const_cap(ARM64_WORKAROUND_CAVIUM_27456)) { str = "ARM64_WORKAROUND_CAVIUM_27456"; __kpti_forced = -1; } @@ -1203,5 +1204,5 @@ void __init setup_cpu_features(void) static bool __maybe_unused cpufeature_pan_not_uao(const struct arm64_cpu_capabilities *entry, int __unused) { - return (cpus_have_cap(ARM64_HAS_PAN) && !cpus_have_cap(ARM64_HAS_UAO)); + return (cpus_have_const_cap(ARM64_HAS_PAN) && !cpus_have_const_cap(ARM64_HAS_UAO)); } diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 0972ce58316d..e917d119490c 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -291,7 +291,7 @@ int copy_thread(unsigned long clone_flags, unsigned long stack_start, memset(childregs, 0, sizeof(struct pt_regs)); childregs->pstate = PSR_MODE_EL1h; if (IS_ENABLED(CONFIG_ARM64_UAO) && - cpus_have_cap(ARM64_HAS_UAO)) + cpus_have_const_cap(ARM64_HAS_UAO)) childregs->pstate |= PSR_UAO_BIT; p->thread.cpu_context.x19 = stack_start; p->thread.cpu_context.x20 = stk_sz; diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c index 0b1d5bdd0862..f7b8681aed3f 100644 --- a/drivers/irqchip/irq-gic-v3.c +++ b/drivers/irqchip/irq-gic-v3.c @@ -120,11 +120,10 @@ static void gic_redist_wait_for_rwp(void) } #ifdef CONFIG_ARM64 -static DEFINE_STATIC_KEY_FALSE(is_cavium_thunderx); static u64 __maybe_unused gic_read_iar(void) { - if (static_branch_unlikely(&is_cavium_thunderx)) + if (cpus_have_const_cap(ARM64_WORKAROUND_CAVIUM_23154)) return gic_read_iar_cavium_thunderx(); else return gic_read_iar_common(); @@ -908,14 +907,6 @@ static const struct irq_domain_ops partition_domain_ops = { .select = gic_irq_domain_select, }; -static void gicv3_enable_quirks(void) -{ -#ifdef CONFIG_ARM64 - if (cpus_have_cap(ARM64_WORKAROUND_CAVIUM_23154)) - static_branch_enable(&is_cavium_thunderx); -#endif -} - static int __init gic_init_bases(void __iomem *dist_base, struct redist_region *rdist_regs, u32 nr_redist_regions, @@ -938,8 +929,6 @@ static int __init gic_init_bases(void __iomem *dist_base, gic_data.nr_redist_regions = nr_redist_regions; gic_data.redist_stride = redist_stride; - gicv3_enable_quirks(); - /* * Find out how many interrupts are supported. * The GIC only supports up to 1020 interrupt sources (SGI+PPI+SPI) From patchwork Fri Jun 1 17:12:10 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 137572 Delivered-To: patch@linaro.org Received: by 2002:a2e:9706:0:0:0:0:0 with SMTP id r6-v6csp1198676lji; Fri, 1 Jun 2018 10:12:26 -0700 (PDT) X-Google-Smtp-Source: ADUXVKK/TQo82pI3rz/CFn12yTGTye9A2KGlUTdIIQefOActU1mq4wgksgdGakblHMFdfroOjiq0 X-Received: by 2002:a62:c987:: with SMTP id l7-v6mr11475248pfk.221.1527873146183; Fri, 01 Jun 2018 10:12:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1527873146; cv=none; d=google.com; s=arc-20160816; b=MpeVyOXWnebe/0QgRaN+bx+q5noWP27J1JBOppeZQoSgAdHX2XfiDo2YtEOG/9xQma e7scUsDyljYdR3Yc4ibN7rwnp8hH+cUnQ4u2oT01SQggk9B9lVEUZyZro5HMG2hsrT7K 3gIsvhO8ZAmURdj9KNZSh3R/V5HFxzk1R+MmHulPOTpiCiR/eAu0RqSDiPDrwQwIwIhd aBUa0+a4v/NZfyoW7brUnnetGLZCkc31f+eZepLCyw2i39poUTZG0iteNbglcqoEcCi/ 1+zjpngiXeqhV3EMYuMl4OGkgBL3Oizxeeg7PAWM+mW1/KUW+uEGeo6ODYV7YYH7wRv8 sewA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=HIW4rJzZzWYE1CYVt38UEYzn+W9r46d4IqC5EmdM2gY=; b=Y5ohquqY8BTnqx30+av2XmIZjneRAnqmqnq+hJM1hIH/aeIfXDOsLLtvxOJilzAxvW bDprczofXfPaiPVdBxsGi69Q+jpm43wkE5ZvKCyOID0RCFcL+9P8hU0FEwP/9U9WtYPv S2Vsc/TmVJwjRs0E2hMTrgkQm/Fev8G1k8iZ6df/aFv+4LSReOaB0IvqRAVH6uKl/iIu lrcbJCDkho5n5Q3RzgXoOjy1EcQekharjlnqoKE3lehBQi/PF2kG0U4qcna+gKVBhbaW FpP6nH5paz2kJpLfy+VaTbPXeuL3tGckPr8EVqYvhVy4Pts+ErHaz98fyFHfkn2qu+Vy hOng== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id bd4-v6si21210085plb.338.2018.06.01.10.12.25; Fri, 01 Jun 2018 10:12:26 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753185AbeFARMW (ORCPT + 13 others); Fri, 1 Jun 2018 13:12:22 -0400 Received: from foss.arm.com ([217.140.101.70]:56098 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752262AbeFARMV (ORCPT ); Fri, 1 Jun 2018 13:12:21 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D946C1529; Fri, 1 Jun 2018 10:12:20 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 1E6483F53D; Fri, 1 Jun 2018 10:12:19 -0700 (PDT) From: Mark Rutland To: stable@vger.kernel.org Cc: marc.zyngier@arm.com, mark.rutland@arm.com Subject: [PATCH v4.9.y 2/2] arm64/cpufeature: don't use mutex in bringup path Date: Fri, 1 Jun 2018 18:12:10 +0100 Message-Id: <20180601171210.16870-3-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180601171210.16870-1-mark.rutland@arm.com> References: <20180601171210.16870-1-mark.rutland@arm.com> Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org commit 63a1e1c95e60e798fa09ab3c536fb555aa5bbf2b upstream. Currently, cpus_set_cap() calls static_branch_enable_cpuslocked(), which must take the jump_label mutex. We call cpus_set_cap() in the secondary bringup path, from the idle thread where interrupts are disabled. Taking a mutex in this path "is a NONO" regardless of whether it's contended, and something we must avoid. We didn't spot this until recently, as ___might_sleep() won't warn for this case until all CPUs have been brought up. This patch avoids taking the mutex in the secondary bringup path. The poking of static keys is deferred until enable_cpu_capabilities(), which runs in a suitable context on the boot CPU. To account for the static keys being set later, cpus_have_const_cap() is updated to use another static key to check whether the const cap keys have been initialised, falling back to the caps bitmap until this is the case. This means that users of cpus_have_const_cap() gain should only gain a single additional NOP in the fast path once the const caps are initialised, but should always see the current cap value. The hyp code should never dereference the caps array, since the caps are initialized before we run the module initcall to initialise hyp. A check is added to the hyp init code to document this requirement. This change will sidestep a number of issues when the upcoming hotplug locking rework is merged. Signed-off-by: Mark Rutland Reviewed-by: Marc Zyniger Reviewed-by: Suzuki Poulose Acked-by: Will Deacon Cc: Christoffer Dall Cc: Peter Zijlstra Cc: Sebastian Sewior Cc: Thomas Gleixner Signed-off-by: Catalin Marinas [4.9: this avoids an IPI before GICv3 is up, preventing a boot time crash] Signed-off-by: Mark Rutland [v4.9 backport] --- arch/arm64/include/asm/cpufeature.h | 12 ++++++++++-- arch/arm64/include/asm/kvm_host.h | 8 ++++++-- arch/arm64/kernel/cpufeature.c | 23 +++++++++++++++++++++-- 3 files changed, 37 insertions(+), 6 deletions(-) -- 2.11.0 diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 9890d20f2356..4ea85ebdf4df 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -96,6 +96,7 @@ struct arm64_cpu_capabilities { extern DECLARE_BITMAP(cpu_hwcaps, ARM64_NCAPS); extern struct static_key_false cpu_hwcap_keys[ARM64_NCAPS]; +extern struct static_key_false arm64_const_caps_ready; bool this_cpu_has_cap(unsigned int cap); @@ -105,7 +106,7 @@ static inline bool cpu_have_feature(unsigned int num) } /* System capability check for constant caps */ -static inline bool cpus_have_const_cap(int num) +static inline bool __cpus_have_const_cap(int num) { if (num >= ARM64_NCAPS) return false; @@ -119,6 +120,14 @@ static inline bool cpus_have_cap(unsigned int num) return test_bit(num, cpu_hwcaps); } +static inline bool cpus_have_const_cap(int num) +{ + if (static_branch_likely(&arm64_const_caps_ready)) + return __cpus_have_const_cap(num); + else + return cpus_have_cap(num); +} + static inline void cpus_set_cap(unsigned int num) { if (num >= ARM64_NCAPS) { @@ -126,7 +135,6 @@ static inline void cpus_set_cap(unsigned int num) num, ARM64_NCAPS); } else { __set_bit(num, cpu_hwcaps); - static_branch_enable(&cpu_hwcap_keys[num]); } } diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 7f402f64c744..2abb4493f4f6 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -24,6 +24,7 @@ #include #include +#include #include #include #include @@ -358,9 +359,12 @@ static inline void __cpu_init_hyp_mode(phys_addr_t pgd_ptr, unsigned long vector_ptr) { /* - * Call initialization code, and switch to the full blown - * HYP code. + * Call initialization code, and switch to the full blown HYP code. + * If the cpucaps haven't been finalized yet, something has gone very + * wrong, and hyp will crash and burn when it uses any + * cpus_have_const_cap() wrapper. */ + BUG_ON(!static_branch_likely(&arm64_const_caps_ready)); __kvm_call_hyp((void *)pgd_ptr, hyp_stack_ptr, vector_ptr); } diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index e28665411bd1..7959d2c92010 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1052,8 +1052,16 @@ void update_cpu_capabilities(const struct arm64_cpu_capabilities *caps, */ void __init enable_cpu_capabilities(const struct arm64_cpu_capabilities *caps) { - for (; caps->matches; caps++) - if (caps->enable && cpus_have_cap(caps->capability)) + for (; caps->matches; caps++) { + unsigned int num = caps->capability; + + if (!cpus_have_cap(num)) + continue; + + /* Ensure cpus_have_const_cap(num) works */ + static_branch_enable(&cpu_hwcap_keys[num]); + + if (caps->enable) { /* * Use stop_machine() as it schedules the work allowing * us to modify PSTATE, instead of on_each_cpu() which @@ -1061,6 +1069,8 @@ void __init enable_cpu_capabilities(const struct arm64_cpu_capabilities *caps) * we return. */ stop_machine(caps->enable, (void *)caps, cpu_online_mask); + } + } } /* @@ -1164,6 +1174,14 @@ static void __init setup_feature_capabilities(void) enable_cpu_capabilities(arm64_features); } +DEFINE_STATIC_KEY_FALSE(arm64_const_caps_ready); +EXPORT_SYMBOL(arm64_const_caps_ready); + +static void __init mark_const_caps_ready(void) +{ + static_branch_enable(&arm64_const_caps_ready); +} + extern const struct arm64_cpu_capabilities arm64_errata[]; bool this_cpu_has_cap(unsigned int cap) @@ -1180,6 +1198,7 @@ void __init setup_cpu_features(void) /* Set the CPU feature capabilies */ setup_feature_capabilities(); enable_errata_workarounds(); + mark_const_caps_ready(); setup_elf_hwcaps(arm64_elf_hwcaps); if (system_supports_32bit_el0())