From patchwork Sun May 25 18:19:01 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoffer Dall X-Patchwork-Id: 30911 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-qc0-f200.google.com (mail-qc0-f200.google.com [209.85.216.200]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id B2155203AC for ; Sun, 25 May 2014 18:24:11 +0000 (UTC) Received: by mail-qc0-f200.google.com with SMTP id c9sf9588995qcz.11 for ; Sun, 25 May 2014 11:24:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:subject:date:message-id :in-reply-to:references:cc:precedence:list-id:list-unsubscribe :list-archive:list-post:list-help:list-subscribe:mime-version:sender :errors-to:x-original-sender:x-original-authentication-results :mailing-list:content-type:content-transfer-encoding; bh=jstdX9CY96oxa6A4C8EFG+6GZfCqlngnHj+edlz2FsU=; b=k3txRBxlNfj6q5JEDWfZTprGGYyOhLLJNyAtww7oXPdNarXmA0qvTHw7ERvxZ+0AMX ZnomrkLxqEte/7GX/iGMKNLBwcTGLB4aaU66YzhMTVso4ph0bVUGPSbJy3VtdTZJDEjg nrQHa/XrbPOENQoVM38UzpdLlw6hKe/Y4nxGoOuSs6UrGcV4dIWS2EqaawkRLCUItjYo pGLPDz4jpR/1ZxRdUeF7pai6piKvJWiVPM8hoxfEuSkykrN6y54wW16FIrOWlAbFN4Yn IirnZLP05+9DCdy+IcfJttsLyAN7s2CTdxn09C1eLoJJ054E0UjUy1d/W1JY+Z3tmE+W ZJbQ== X-Gm-Message-State: ALoCoQlWFzKC8zPt90u//CH0nLvvEl+mM6i+OHrKtevoTvYfgor4GX83u857x1zu8ly+UTG2oHyJ X-Received: by 10.224.120.67 with SMTP id c3mr8557721qar.4.1401042251456; Sun, 25 May 2014 11:24:11 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.28.195 with SMTP id 61ls1312382qgz.77.gmail; Sun, 25 May 2014 11:24:11 -0700 (PDT) X-Received: by 10.58.85.3 with SMTP id d3mr2692665vez.34.1401042251373; Sun, 25 May 2014 11:24:11 -0700 (PDT) Received: from mail-vc0-f171.google.com (mail-vc0-f171.google.com [209.85.220.171]) by mx.google.com with ESMTPS id uv5si5077910vec.96.2014.05.25.11.24.11 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 25 May 2014 11:24:11 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.171 as permitted sender) client-ip=209.85.220.171; Received: by mail-vc0-f171.google.com with SMTP id lc6so8285863vcb.16 for ; Sun, 25 May 2014 11:24:11 -0700 (PDT) X-Received: by 10.221.29.137 with SMTP id ry9mr16588666vcb.6.1401042251249; Sun, 25 May 2014 11:24:11 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.221.72 with SMTP id ib8csp52581vcb; Sun, 25 May 2014 11:24:10 -0700 (PDT) X-Received: by 10.140.35.212 with SMTP id n78mr24461967qgn.87.1401042250512; Sun, 25 May 2014 11:24:10 -0700 (PDT) Received: from bombadil.infradead.org (bombadil.infradead.org. [2001:1868:205::9]) by mx.google.com with ESMTPS id c32si10621544qgd.64.2014.05.25.11.24.10 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 25 May 2014 11:24:10 -0700 (PDT) Received-SPF: none (google.com: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org does not designate permitted sender hosts) client-ip=2001:1868:205::9; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1Wod45-0008Ot-8X; Sun, 25 May 2014 18:22:33 +0000 Received: from mail-la0-f47.google.com ([209.85.215.47]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1Wod1h-0004sF-FY for linux-arm-kernel@lists.infradead.org; Sun, 25 May 2014 18:20:06 +0000 Received: by mail-la0-f47.google.com with SMTP id pn19so5322883lab.34 for ; Sun, 25 May 2014 11:19:42 -0700 (PDT) X-Received: by 10.152.10.2 with SMTP id e2mr1863382lab.76.1401041982640; Sun, 25 May 2014 11:19:42 -0700 (PDT) Received: from localhost.localdomain (x1-6-b8-c7-5d-cb-5a-ca.cpe.webspeed.dk. [2.104.6.253]) by mx.google.com with ESMTPSA id o1sm9683895lbw.27.2014.05.25.11.19.40 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 25 May 2014 11:19:41 -0700 (PDT) From: Christoffer Dall To: Paolo Bonzini , Gleb Natapov Subject: [PATCH 15/16] ARM: Check if a CPU has gone offline Date: Sun, 25 May 2014 20:19:01 +0200 Message-Id: <1401041942-23471-16-git-send-email-christoffer.dall@linaro.org> X-Mailer: git-send-email 1.8.5.2 In-Reply-To: <1401041942-23471-1-git-send-email-christoffer.dall@linaro.org> References: <1401041942-23471-1-git-send-email-christoffer.dall@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20140525_112005_957169_D3B73263 X-CRM114-Status: GOOD ( 20.80 ) X-Spam-Score: -0.7 (/) X-Spam-Report: SpamAssassin version 3.3.2 on bombadil.infradead.org summary: Content analysis details: (-0.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low trust [209.85.215.47 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record Cc: Mark Rutland , Peter Maydell , Anup Patel , kvm@vger.kernel.org, Marc Zyngier , Ashwin Chaugule , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: christoffer.dall@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.171 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 From: Ashwin Chaugule PSCIv0.2 adds a new function called AFFINITY_INFO, which can be used to query if a specified CPU has actually gone offline. Calling this function via cpu_kill ensures that a CPU has quiesced after a call to cpu_die. This helps prevent the CPU from doing arbitrary bad things when data or instructions are clobbered (as happens with kexec) in the window between a CPU announcing that it is dead and said CPU leaving the kernel. Signed-off-by: Ashwin Chaugule Signed-off-by: Mark Rutland Reviewed-by: Rob Herring Acked-by: Catalin Marinas --- arch/arm/kernel/psci_smp.c | 33 +++++++++++++++++++++++++++++++++ arch/arm64/include/asm/cpu_ops.h | 2 ++ arch/arm64/kernel/psci.c | 31 +++++++++++++++++++++++++++++++ arch/arm64/kernel/smp.c | 22 ++++++++++++++++++++++ 4 files changed, 88 insertions(+) diff --git a/arch/arm/kernel/psci_smp.c b/arch/arm/kernel/psci_smp.c index 570a48c..28a1db4 100644 --- a/arch/arm/kernel/psci_smp.c +++ b/arch/arm/kernel/psci_smp.c @@ -16,6 +16,8 @@ #include #include #include +#include +#include #include #include @@ -66,6 +68,36 @@ void __ref psci_cpu_die(unsigned int cpu) /* We should never return */ panic("psci: cpu %d failed to shutdown\n", cpu); } + +int __ref psci_cpu_kill(unsigned int cpu) +{ + int err, i; + + if (!psci_ops.affinity_info) + return 1; + /* + * cpu_kill could race with cpu_die and we can + * potentially end up declaring this cpu undead + * while it is dying. So, try again a few times. + */ + + for (i = 0; i < 10; i++) { + err = psci_ops.affinity_info(cpu_logical_map(cpu), 0); + if (err == PSCI_0_2_AFFINITY_LEVEL_OFF) { + pr_info("CPU%d killed.\n", cpu); + return 1; + } + + msleep(10); + pr_info("Retrying again to check for CPU kill\n"); + } + + pr_warn("CPU%d may not have shut down cleanly (AFFINITY_INFO reports %d)\n", + cpu, err); + /* Make platform_cpu_kill() fail. */ + return 0; +} + #endif bool __init psci_smp_available(void) @@ -78,5 +110,6 @@ struct smp_operations __initdata psci_smp_ops = { .smp_boot_secondary = psci_boot_secondary, #ifdef CONFIG_HOTPLUG_CPU .cpu_die = psci_cpu_die, + .cpu_kill = psci_cpu_kill, #endif }; diff --git a/arch/arm64/include/asm/cpu_ops.h b/arch/arm64/include/asm/cpu_ops.h index 1524130..d7b4b38 100644 --- a/arch/arm64/include/asm/cpu_ops.h +++ b/arch/arm64/include/asm/cpu_ops.h @@ -39,6 +39,7 @@ struct device_node; * from the cpu to be killed. * @cpu_die: Makes a cpu leave the kernel. Must not fail. Called from the * cpu being killed. + * @cpu_kill: Ensures a cpu has left the kernel. Called from another cpu. * @cpu_suspend: Suspends a cpu and saves the required context. May fail owing * to wrong parameters or error conditions. Called from the * CPU being suspended. Must be called with IRQs disabled. @@ -52,6 +53,7 @@ struct cpu_operations { #ifdef CONFIG_HOTPLUG_CPU int (*cpu_disable)(unsigned int cpu); void (*cpu_die)(unsigned int cpu); + int (*cpu_kill)(unsigned int cpu); #endif #ifdef CONFIG_ARM64_CPU_SUSPEND int (*cpu_suspend)(unsigned long); diff --git a/arch/arm64/kernel/psci.c b/arch/arm64/kernel/psci.c index 90df6e6..9e9798f 100644 --- a/arch/arm64/kernel/psci.c +++ b/arch/arm64/kernel/psci.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include @@ -403,6 +404,35 @@ static void cpu_psci_cpu_die(unsigned int cpu) pr_crit("unable to power off CPU%u (%d)\n", cpu, ret); } + +static int cpu_psci_cpu_kill(unsigned int cpu) +{ + int err, i; + + if (!psci_ops.affinity_info) + return 1; + /* + * cpu_kill could race with cpu_die and we can + * potentially end up declaring this cpu undead + * while it is dying. So, try again a few times. + */ + + for (i = 0; i < 10; i++) { + err = psci_ops.affinity_info(cpu_logical_map(cpu), 0); + if (err == PSCI_0_2_AFFINITY_LEVEL_OFF) { + pr_info("CPU%d killed.\n", cpu); + return 1; + } + + msleep(10); + pr_info("Retrying again to check for CPU kill\n"); + } + + pr_warn("CPU%d may not have shut down cleanly (AFFINITY_INFO reports %d)\n", + cpu, err); + /* Make op_cpu_kill() fail. */ + return 0; +} #endif const struct cpu_operations cpu_psci_ops = { @@ -413,6 +443,7 @@ const struct cpu_operations cpu_psci_ops = { #ifdef CONFIG_HOTPLUG_CPU .cpu_disable = cpu_psci_cpu_disable, .cpu_die = cpu_psci_cpu_die, + .cpu_kill = cpu_psci_cpu_kill, #endif }; diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index f0a141d..c3cb160 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -228,6 +228,19 @@ int __cpu_disable(void) return 0; } +static int op_cpu_kill(unsigned int cpu) +{ + /* + * If we have no means of synchronising with the dying CPU, then assume + * that it is really dead. We can only wait for an arbitrary length of + * time and hope that it's dead, so let's skip the wait and just hope. + */ + if (!cpu_ops[cpu]->cpu_kill) + return 1; + + return cpu_ops[cpu]->cpu_kill(cpu); +} + static DECLARE_COMPLETION(cpu_died); /* @@ -241,6 +254,15 @@ void __cpu_die(unsigned int cpu) return; } pr_notice("CPU%u: shutdown\n", cpu); + + /* + * Now that the dying CPU is beyond the point of no return w.r.t. + * in-kernel synchronisation, try to get the firwmare to help us to + * verify that it has really left the kernel before we consider + * clobbering anything it might still be using. + */ + if (!op_cpu_kill(cpu)) + pr_warn("CPU%d may not have shut down cleanly\n", cpu); } /*