From patchwork Thu Feb 14 14:49:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Long X-Patchwork-Id: 158408 Delivered-To: patches@linaro.org Received: by 2002:a02:48:0:0:0:0:0 with SMTP id 69csp1462408jaa; Thu, 14 Feb 2019 06:49:52 -0800 (PST) X-Received: by 2002:a0c:9c89:: with SMTP id i9mr3163939qvf.153.1550155792626; Thu, 14 Feb 2019 06:49:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550155792; cv=none; d=google.com; s=arc-20160816; b=k3uyL7fbrC65vRSNqt8Zz8XbnP4oXqUSbYR4p+ZKOoSAlPj7bsIsVmKOwU1jPjqYgj QH75jv+oavwGD0W5RyzpjoGgZzMM/2RC0PO6d0YKjzZZrZSIU7IqaNEaPvPrjWqPsMMk z/JQxyg4N0kocLo14jZxVuVrD/nAIEsbnK8IeOExaxO504s5F7i6BaBAGX+6kuyxNrEu 0HXQAd+0cco7AzbN50daEzmDYaef4bG71jU+G3vSgFL7dspJdKzJsQPASOcs+ZlQs7er 9iBEoEjKeXMSUGdDkkeEm+9PdGgd+Avjs8EVXKmUD1hjp3NYLh/XEWOwx04Tu3dh5MUg 2jvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=nqpC+x7OojB5e6AT6Ko7lpgIEeP3+R6qKzQcfmayAMM=; b=JqNNtC66Cb1Mkz4l2FcWqh0Lk4CiykKC6iWenSiKJWPDB7N4LrpZWKWWibBcnImhMD uJJqguK9KAzqgGC855zjUWuvkPNWy3jBo8Zye5WxfBM09PVi8bVSW/GNzZv4+am9Ht5b rCVLAJP9pZS8z8P+PwHiER09WBpdKXO5nNhle6Xr8/fbBAZYLA9uy8zxm5UPTxwrpLSb evM2RraT9ODhoLbCG6NyNarhMN2uhMTekcFmt6U16tKVafClB4fmrbufQwshPw3+a3Ss lbCYZM1o4mjRe3iQuWNQgpFQhfMMcrK9eQ96ghS5ATyU47oIWh9mtXAfbLXKQ/VyFkWz xZsA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=meYdUgvQ; spf=pass (google.com: domain of dave.long@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=dave.long@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id u41sor2921178qvf.42.2019.02.14.06.49.52 for (Google Transport Security); Thu, 14 Feb 2019 06:49:52 -0800 (PST) Received-SPF: pass (google.com: domain of dave.long@linaro.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=meYdUgvQ; spf=pass (google.com: domain of dave.long@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=dave.long@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=nqpC+x7OojB5e6AT6Ko7lpgIEeP3+R6qKzQcfmayAMM=; b=meYdUgvQh35QkE7UMg173Kn3m6ZmtUcD+hKsyAXGMHsvzuNMAa7X7SaN/B1RYTg1Og +5Cpz4ayI8VABKaDQa6i5Nw86AYJZkMELL1slCfj3JFws1/R/noHNzEWAgk5qyzuSGzU OvMSrj+mbPIAcjBbst8SxHWqOywr8AlRRpHl7ZANZlR5suo3zxoVbgIZwe96YOHWrbs5 H2KcacmsFeNt7TFOZt1CofU4g0PoCpgEFsFYzUr0JrSNPJMk0mzTHSn/H+bm1Yxj64PW 0PbK6b4YGDs/ywwaI2qZUnZIHGsVxN2/cKIWzPaq4pkOhPLDVrxsvbKqyzQuN3A0PQS2 psJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=nqpC+x7OojB5e6AT6Ko7lpgIEeP3+R6qKzQcfmayAMM=; b=oOpVDv7nEcIn7pNfPq1Pj9EPC5S/UhCazE3nRb6PfgUxtzL3ccT/JEGgIdyHlnTfwa 5pUxJdPV5gns30/d5ZvZ107a3E87nfPuvWSztH5tGAXhDrUB259rwuWH3M5MyyzO3/aL tZQzFpuMjNgT5xP6LxZIpsRpUUrZ95VdcsdmhB0QAb1GnHLarqh2flhalHL7XuSiZEL8 3uNOARU7klOs411EarL5PPQvvAhIy3rVrI3UcuSTF4Ks9V6YqQsOfg+XADtDOkVcUQK0 tn2xVrLAQS6pUCuzgO8PQsb06MYdVWITHZfQ1RAlzVXAIIfs7kzWLazFpEE9PLpuZh0/ oj4A== X-Gm-Message-State: AHQUAub6HFcjW7SKTVgzcVTB1A38LlNIoPrDPxqlzlFcA8npAkBOsICB jUTj4+WIIPJfwDKrJ0Aqn8P53exXGOabiw== X-Google-Smtp-Source: AHgI3IahuzIaHiJUFBVk8WUV+O5N6w8lyQUfNkWVFf8GozAuuaTLSwzi6GBnC6eRxH906LFmswMkdw== X-Received: by 2002:a0c:8542:: with SMTP id n60mr3175888qva.205.1550155792303; Thu, 14 Feb 2019 06:49:52 -0800 (PST) Return-Path: Received: from localhost.localdomain (pool-72-71-243-63.cncdnh.fast00.myfairpoint.net. [72.71.243.63]) by smtp.googlemail.com with ESMTPSA id k66sm1498919qkc.25.2019.02.14.06.49.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Feb 2019 06:49:51 -0800 (PST) From: David Long To: stable@vger.kernel.org, Russell King - ARM Linux , Florian Fainelli , Julien Thierry , Tony Lindgren , Marc Zyngier , Greg KH , Mark Rutland Cc: Will Deacon , Mark Brown , linux-kernel@vger.kernel.org Subject: [PATCH 4.9 14/16] ARM: spectre-v2: per-CPU vtables to work around big.Little systems Date: Thu, 14 Feb 2019 09:49:28 -0500 Message-Id: <20190214144930.27539-15-dave.long@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190214144930.27539-1-dave.long@linaro.org> References: <20190214144930.27539-1-dave.long@linaro.org> From: Russell King Commit 383fb3ee8024d596f488d2dbaf45e572897acbdb upstream. In big.Little systems, some CPUs require the Spectre workarounds in paths such as the context switch, but other CPUs do not. In order to handle these differences, we need per-CPU vtables. We are unable to use the kernel's per-CPU variables to support this as per-CPU is not initialised at times when we need access to the vtables, so we have to use an array indexed by logical CPU number. We use an array-of-pointers to avoid having function pointers in the kernel's read/write .data section. Note: Added include of linux/slab.h in arch/arm/smp.c. Reviewed-by: Julien Thierry Signed-off-by: Russell King Signed-off-by: David A. Long --- arch/arm/include/asm/proc-fns.h | 23 +++++++++++++++++++++++ arch/arm/kernel/setup.c | 5 +++++ arch/arm/kernel/smp.c | 32 ++++++++++++++++++++++++++++++++ arch/arm/mm/proc-v7-bugs.c | 17 ++--------------- 4 files changed, 62 insertions(+), 15 deletions(-) -- 2.17.1 diff --git a/arch/arm/include/asm/proc-fns.h b/arch/arm/include/asm/proc-fns.h index a1a71b068edc..1bfcc3bcfc6d 100644 --- a/arch/arm/include/asm/proc-fns.h +++ b/arch/arm/include/asm/proc-fns.h @@ -104,12 +104,35 @@ extern void cpu_do_resume(void *); #else extern struct processor processor; +#if defined(CONFIG_BIG_LITTLE) && defined(CONFIG_HARDEN_BRANCH_PREDICTOR) +#include +/* + * This can't be a per-cpu variable because we need to access it before + * per-cpu has been initialised. We have a couple of functions that are + * called in a pre-emptible context, and so can't use smp_processor_id() + * there, hence PROC_TABLE(). We insist in init_proc_vtable() that the + * function pointers for these are identical across all CPUs. + */ +extern struct processor *cpu_vtable[]; +#define PROC_VTABLE(f) cpu_vtable[smp_processor_id()]->f +#define PROC_TABLE(f) cpu_vtable[0]->f +static inline void init_proc_vtable(const struct processor *p) +{ + unsigned int cpu = smp_processor_id(); + *cpu_vtable[cpu] = *p; + WARN_ON_ONCE(cpu_vtable[cpu]->dcache_clean_area != + cpu_vtable[0]->dcache_clean_area); + WARN_ON_ONCE(cpu_vtable[cpu]->set_pte_ext != + cpu_vtable[0]->set_pte_ext); +} +#else #define PROC_VTABLE(f) processor.f #define PROC_TABLE(f) processor.f static inline void init_proc_vtable(const struct processor *p) { processor = *p; } +#endif #define cpu_proc_init PROC_VTABLE(_proc_init) #define cpu_check_bugs PROC_VTABLE(check_bugs) diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c index 2eebb67fa08b..4764742db7b0 100644 --- a/arch/arm/kernel/setup.c +++ b/arch/arm/kernel/setup.c @@ -115,6 +115,11 @@ EXPORT_SYMBOL(elf_hwcap2); #ifdef MULTI_CPU struct processor processor __ro_after_init; +#if defined(CONFIG_BIG_LITTLE) && defined(CONFIG_HARDEN_BRANCH_PREDICTOR) +struct processor *cpu_vtable[NR_CPUS] = { + [0] = &processor, +}; +#endif #endif #ifdef MULTI_TLB struct cpu_tlb_fns cpu_tlb __ro_after_init; diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c index 4b129aac7233..8faf869e9fb2 100644 --- a/arch/arm/kernel/smp.c +++ b/arch/arm/kernel/smp.c @@ -27,6 +27,7 @@ #include #include #include +#include #include #include @@ -40,6 +41,7 @@ #include #include #include +#include #include #include #include @@ -100,6 +102,30 @@ static unsigned long get_arch_pgd(pgd_t *pgd) #endif } +#if defined(CONFIG_BIG_LITTLE) && defined(CONFIG_HARDEN_BRANCH_PREDICTOR) +static int secondary_biglittle_prepare(unsigned int cpu) +{ + if (!cpu_vtable[cpu]) + cpu_vtable[cpu] = kzalloc(sizeof(*cpu_vtable[cpu]), GFP_KERNEL); + + return cpu_vtable[cpu] ? 0 : -ENOMEM; +} + +static void secondary_biglittle_init(void) +{ + init_proc_vtable(lookup_processor(read_cpuid_id())->proc); +} +#else +static int secondary_biglittle_prepare(unsigned int cpu) +{ + return 0; +} + +static void secondary_biglittle_init(void) +{ +} +#endif + int __cpu_up(unsigned int cpu, struct task_struct *idle) { int ret; @@ -107,6 +133,10 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle) if (!smp_ops.smp_boot_secondary) return -ENOSYS; + ret = secondary_biglittle_prepare(cpu); + if (ret) + return ret; + /* * We need to tell the secondary core where to find * its stack and the page tables. @@ -358,6 +388,8 @@ asmlinkage void secondary_start_kernel(void) struct mm_struct *mm = &init_mm; unsigned int cpu; + secondary_biglittle_init(); + /* * The identity mapping is uncached (strongly ordered), so * switch away from it before attempting any exclusive accesses. diff --git a/arch/arm/mm/proc-v7-bugs.c b/arch/arm/mm/proc-v7-bugs.c index 5544b82a2e7a..9a07916af8dd 100644 --- a/arch/arm/mm/proc-v7-bugs.c +++ b/arch/arm/mm/proc-v7-bugs.c @@ -52,8 +52,6 @@ static void cpu_v7_spectre_init(void) case ARM_CPU_PART_CORTEX_A17: case ARM_CPU_PART_CORTEX_A73: case ARM_CPU_PART_CORTEX_A75: - if (processor.switch_mm != cpu_v7_bpiall_switch_mm) - goto bl_error; per_cpu(harden_branch_predictor_fn, cpu) = harden_branch_predictor_bpiall; spectre_v2_method = "BPIALL"; @@ -61,8 +59,6 @@ static void cpu_v7_spectre_init(void) case ARM_CPU_PART_CORTEX_A15: case ARM_CPU_PART_BRAHMA_B15: - if (processor.switch_mm != cpu_v7_iciallu_switch_mm) - goto bl_error; per_cpu(harden_branch_predictor_fn, cpu) = harden_branch_predictor_iciallu; spectre_v2_method = "ICIALLU"; @@ -88,11 +84,9 @@ static void cpu_v7_spectre_init(void) ARM_SMCCC_ARCH_WORKAROUND_1, &res); if ((int)res.a0 != 0) break; - if (processor.switch_mm != cpu_v7_hvc_switch_mm && cpu) - goto bl_error; per_cpu(harden_branch_predictor_fn, cpu) = call_hvc_arch_workaround_1; - processor.switch_mm = cpu_v7_hvc_switch_mm; + cpu_do_switch_mm = cpu_v7_hvc_switch_mm; spectre_v2_method = "hypervisor"; break; @@ -101,11 +95,9 @@ static void cpu_v7_spectre_init(void) ARM_SMCCC_ARCH_WORKAROUND_1, &res); if ((int)res.a0 != 0) break; - if (processor.switch_mm != cpu_v7_smc_switch_mm && cpu) - goto bl_error; per_cpu(harden_branch_predictor_fn, cpu) = call_smc_arch_workaround_1; - processor.switch_mm = cpu_v7_smc_switch_mm; + cpu_do_switch_mm = cpu_v7_smc_switch_mm; spectre_v2_method = "firmware"; break; @@ -119,11 +111,6 @@ static void cpu_v7_spectre_init(void) if (spectre_v2_method) pr_info("CPU%u: Spectre v2: using %s workaround\n", smp_processor_id(), spectre_v2_method); - return; - -bl_error: - pr_err("CPU%u: Spectre v2: incorrect context switching function, system vulnerable\n", - cpu); } #else static void cpu_v7_spectre_init(void)