From patchwork Mon May 21 11:35:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Thierry X-Patchwork-Id: 136455 Delivered-To: patch@linaro.org Received: by 2002:a2e:9706:0:0:0:0:0 with SMTP id r6-v6csp200238lji; Mon, 21 May 2018 04:38:45 -0700 (PDT) X-Google-Smtp-Source: AB8JxZrtscmk4dhvY32m7ZPzxaXchSO1Iek6X009mZKi7WzhWWhuUncs4JBLwyKQZOOHoJHwcbxK X-Received: by 2002:a63:2783:: with SMTP id n125-v6mr15207810pgn.377.1526902725825; Mon, 21 May 2018 04:38:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1526902725; cv=none; d=google.com; s=arc-20160816; b=FkX7YlAcvmmiXvzXcMCifqCE59mYPGDvJJiXJ+nRmxMt1sZdDxVgbZs7tfXWL/cTcj eo+RZx5QK76VyLtdoa32fbyxUYhRyMnFuMcBbLbGY2fWZK/gWOByQwHyojsdGJmdkN2C GvYY6t/CDTt93cXiJiIUTRayzkM6sbkBo9w+uTX1Dc+Q6CzF8RGPfwrAp2R7i/e9+hy2 3z+ZhiDtwF8KQm9xUN/L02ASI+QnOAtjuIEYcxbLJH6WDTWEIFBGzcsbzPnuI6e8ESkf A1ps3JNhlMVNPk6K5krOi3Ziv+PJpmdcVf1FG77g5CGzmLsHA+6gNSMl21txHeatKqNn g+0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=XPnxfTgynqql5cYbuokTPWkRsMhY2BYomHzK0DFnMng=; b=Eg6GJ/mBEfXeRQfDat/t+UfZaI26NcsnrMq0bJUk4n3h9lZ3NSqZMQ7VJksN7pJYGY dkBjHWcq4tpyST5RioKHfXeKWyhhAAZjxN5vQbtvaPXOCN15wDGf+pWVrM40UGVCpBfE SIm194VioabpOlJtaVFYlKK4RfMTWK6b8Yx94ImGbNBYqBP9nFgVIubnF7WF8Z8t6ZZj b9NErCLdGKfni2Z5z+Mk0208zI4xOGv+5PdRdyAxkhusO8ffyKWZsiZI7yVba+ZnB6+q bewBjKfttUXDJ9bR/PVTGmb5Xo/dyi91tKcFUkFdLcxi4WeyWXOBz2xBeGJz544vb+Pn zJJw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h186-v6si10941128pge.324.2018.05.21.04.38.45; Mon, 21 May 2018 04:38:45 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753207AbeEULio (ORCPT + 29 others); Mon, 21 May 2018 07:38:44 -0400 Received: from foss.arm.com ([217.140.101.70]:47562 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752807AbeEULfm (ORCPT ); Mon, 21 May 2018 07:35:42 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1C16A165C; Mon, 21 May 2018 04:35:42 -0700 (PDT) Received: from e112298-lin.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0BCAA3F25D; Mon, 21 May 2018 04:35:39 -0700 (PDT) From: Julien Thierry To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, marc.zyngier@arm.com, mark.rutland@arm.com, christoffer.dall@arm.com, james.morse@arm.com, joelaf@google.com, joel.opensrc@gmail.com, daniel.thompson@linaro.org, catalin.marinas@arm.com, will.deacon@arm.com, Julien Thierry , Suzuki K Poulose Subject: [PATCH v3 2/6] arm64: alternative: Apply alternatives early in boot process Date: Mon, 21 May 2018 12:35:11 +0100 Message-Id: <1526902515-13769-3-git-send-email-julien.thierry@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1526902515-13769-1-git-send-email-julien.thierry@arm.com> References: <1526902515-13769-1-git-send-email-julien.thierry@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Daniel Thompson Currently alternatives are applied very late in the boot process (and a long time after we enable scheduling). Some alternative sequences, such as those that alter the way CPU context is stored, must be applied much earlier in the boot sequence. Introduce apply_boot_alternatives() to allow some alternatives to be applied immediately after we detect the CPU features of the boot CPU. Since alternatives are now applied at different times, provide function to check whether alternatives are applied per feature. Signed-off-by: Daniel Thompson [julien.thierry@arm.com: rename to fit new cpufeature framework better, apply BOOT_SCOPE feature early in boot, add per feature alternative checking] Signed-off-by: Julien Thierry Cc: Catalin Marinas Cc: Will Deacon Cc: Christoffer Dall Cc: Marc Zyngier Cc: Suzuki K Poulose --- arch/arm64/include/asm/alternative.h | 5 +++-- arch/arm64/include/asm/cpufeature.h | 2 ++ arch/arm64/kernel/alternative.c | 39 +++++++++++++++++++++++++++++++++--- arch/arm64/kernel/cpufeature.c | 7 ++++++- arch/arm64/kernel/smp.c | 7 +++++++ 5 files changed, 54 insertions(+), 6 deletions(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/alternative.h b/arch/arm64/include/asm/alternative.h index a91933b..36b0703 100644 --- a/arch/arm64/include/asm/alternative.h +++ b/arch/arm64/include/asm/alternative.h @@ -14,8 +14,6 @@ #include #include -extern int alternatives_applied; - struct alt_instr { s32 orig_offset; /* offset to original instruction */ s32 alt_offset; /* offset to replacement instruction */ @@ -27,9 +25,12 @@ struct alt_instr { typedef void (*alternative_cb_t)(struct alt_instr *alt, __le32 *origptr, __le32 *updptr, int nr_inst); +void __init apply_boot_alternatives(void); void __init apply_alternatives_all(void); void apply_alternatives(void *start, size_t length); +bool feature_alternatives_applied(u16 cpufeature); + #define ALTINSTR_ENTRY(feature,cb) \ " .word 661b - .\n" /* label */ \ " .if " __stringify(cb) " == 0\n" \ diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 09b0f2a..19efe4e 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -359,6 +359,8 @@ static inline int cpucap_default_scope(const struct arm64_cpu_capabilities *cap) extern struct static_key_false cpu_hwcap_keys[ARM64_NCAPS]; extern struct static_key_false arm64_const_caps_ready; +extern unsigned long boot_capabilities; + bool this_cpu_has_cap(unsigned int cap); static inline bool cpu_have_feature(unsigned int num) diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c index 5c4bce4..29885eb 100644 --- a/arch/arm64/kernel/alternative.c +++ b/arch/arm64/kernel/alternative.c @@ -34,6 +34,8 @@ int alternatives_applied; +DECLARE_BITMAP(alternatives_status, ARM64_NCAPS); + struct alt_region { struct alt_instr *begin; struct alt_instr *end; @@ -122,7 +124,8 @@ static void patch_alternative(struct alt_instr *alt, } } -static void __apply_alternatives(void *alt_region, bool use_linear_alias) +static void __apply_alternatives(void *alt_region, bool use_linear_alias, + unsigned long feature_mask) { struct alt_instr *alt; struct alt_region *region = alt_region; @@ -132,6 +135,9 @@ static void __apply_alternatives(void *alt_region, bool use_linear_alias) for (alt = region->begin; alt < region->end; alt++) { int nr_inst; + if ((BIT(alt->cpufeature) & feature_mask) == 0) + continue; + /* Use ARM64_CB_PATCH as an unconditional patch */ if (alt->cpufeature < ARM64_CB_PATCH && !cpus_have_cap(alt->cpufeature)) @@ -142,6 +148,8 @@ static void __apply_alternatives(void *alt_region, bool use_linear_alias) else BUG_ON(alt->alt_len != alt->orig_len); + __set_bit(alt->cpufeature, alternatives_status); + pr_info_once("patching kernel code\n"); origptr = ALT_ORIG_PTR(alt); @@ -178,7 +186,9 @@ static int __apply_alternatives_multi_stop(void *unused) isb(); } else { BUG_ON(alternatives_applied); - __apply_alternatives(®ion, true); + + __apply_alternatives(®ion, true, ~boot_capabilities); + /* Barriers provided by the cache flushing */ WRITE_ONCE(alternatives_applied, 1); } @@ -192,6 +202,24 @@ void __init apply_alternatives_all(void) stop_machine(__apply_alternatives_multi_stop, NULL, cpu_online_mask); } +/* + * This is called very early in the boot process (directly after we run + * a feature detect on the boot CPU). No need to worry about other CPUs + * here. + */ +void __init apply_boot_alternatives(void) +{ + struct alt_region region = { + .begin = (struct alt_instr *)__alt_instructions, + .end = (struct alt_instr *)__alt_instructions_end, + }; + + /* If called on non-boot cpu things could go wrong */ + WARN_ON(smp_processor_id() != 0); + + __apply_alternatives(®ion, true, boot_capabilities); +} + void apply_alternatives(void *start, size_t length) { struct alt_region region = { @@ -199,5 +227,10 @@ void apply_alternatives(void *start, size_t length) .end = start + length, }; - __apply_alternatives(®ion, false); + __apply_alternatives(®ion, false, -1); +} + +bool feature_alternatives_applied(u16 cpufeature) +{ + return test_bit(cpufeature, alternatives_status); } diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index e03e897..021ae87 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -52,6 +52,8 @@ DECLARE_BITMAP(cpu_hwcaps, ARM64_NCAPS); EXPORT_SYMBOL(cpu_hwcaps); +unsigned long boot_capabilities; + /* * Flag to indicate if we have computed the system wide * capabilities based on the boot time active CPUs. This @@ -1021,7 +1023,7 @@ static void cpu_copy_el2regs(const struct arm64_cpu_capabilities *__unused) * that, freshly-onlined CPUs will set tpidr_el2, so we don't need to * do anything here. */ - if (!alternatives_applied) + if (!feature_alternatives_applied(ARM64_HAS_VIRT_HOST_EXTN)) write_sysreg(read_sysreg(tpidr_el1), tpidr_el2); } #endif @@ -1346,6 +1348,9 @@ static void __update_cpu_capabilities(const struct arm64_cpu_capabilities *caps, if (!cpus_have_cap(caps->capability) && caps->desc) pr_info("%s %s\n", info, caps->desc); cpus_set_cap(caps->capability); + + if (scope_mask & SCOPE_BOOT_CPU) + __set_bit(caps->capability, &boot_capabilities); } } diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index f3e2e3a..b7fb909 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -410,6 +410,13 @@ void __init smp_prepare_boot_cpu(void) */ jump_label_init(); cpuinfo_store_boot_cpu(); + + /* + * We now know enough about the boot CPU to apply the + * alternatives that cannot wait until interrupt handling + * and/or scheduling is enabled. + */ + apply_boot_alternatives(); } static u64 __init of_get_cpu_mpidr(struct device_node *dn)