From patchwork Tue Sep 26 11:41:52 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 114247 Delivered-To: patch@linaro.org Received: by 10.140.106.117 with SMTP id d108csp3724048qgf; Tue, 26 Sep 2017 04:43:26 -0700 (PDT) X-Google-Smtp-Source: AOwi7QCyROFj1K2H6e72rznSuu1D9ed2FF/aSNG7fa/iPlc9MmlnByJhsSvCIjeU9YM4vKM4CUkq X-Received: by 10.101.74.207 with SMTP id c15mr10647355pgu.259.1506426206517; Tue, 26 Sep 2017 04:43:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1506426206; cv=none; d=google.com; s=arc-20160816; b=t0jdbH3sbFUtWEr+leBLdQJZSyl7PPSYb9bXXEuFc1Taf83yN7I8AMc4JoE4kqLKNq xV/v5KOTLJKCxtYgQoAhMz0N99RQpptOnfm9olLusXRCAsvjTuW8AvRYp78Lpl+TJXcH gH/roUk/FrXKKPRCa0MPCsXGNNGgLevXY053OQfbL75jQGZk26dAbQQ5pMS3zi/ZdZzQ 7/w+h2SiiLdObknZGir+lB/obnhPNs/wEuywOXrmPlIdJsnE2KhF4LP3qaoYkNVmHPqm nFt1WmIx+CglFC1vM5wQSWkms5Tpu2ZvvfjLM55DpFG1pkSpc3cP5oIRxwapUie5bo41 rQAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:cc:to:from :arc-authentication-results; bh=XRB634ASRhuVv8EDovcAy4gHiH4op/KJnenAtdgeBs8=; b=ZgsvbCWJ4evkeIrhJ/8hS0xDBWwa+d9tRuJXgqqieDX3L1GfQwiW5GJ7byFe4QVAtf RSXbrysBObK1z3ONEeFn1ZmZchgYFuLmX2bWo/sGT9QQYMQyi+z7zt5IKUYKXxXXF9cU fB4ZBWgIPp6biEVqzO6dvfY2WZWrucTQQBe2Ajt+QZ0CTZVDULUwP9AJDESTfAqA5yrk E203hZ3OPBAM8Z5sOzC7rurDods1OvWpXs4YhbG1tuSrk5XHj/kbY5GgXtCFV2wiXhkK E4NOcOoukJiBhLNXG0fIxWLfhAUPukYIYXOfcTh0PAmpjOGQStt3DfP/Hx+iroXDMHKs OpYQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id y12si5517554pfg.315.2017.09.26.04.43.25; Tue, 26 Sep 2017 04:43:26 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754492AbdIZLnY (ORCPT + 26 others); Tue, 26 Sep 2017 07:43:24 -0400 Received: from foss.arm.com ([217.140.101.70]:32874 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751450AbdIZLnW (ORCPT ); Tue, 26 Sep 2017 07:43:22 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4B7E31435; Tue, 26 Sep 2017 04:43:22 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 997353F483; Tue, 26 Sep 2017 04:43:20 -0700 (PDT) From: Mark Rutland To: linux-kernel@vger.kernel.org Cc: Mark Rutland , Arnd Bergmann , Christoph Lameter , Peter Zijlstra , Pranith Kumar , Tejun Heo , Thomas Gleixner , linux-arch@vger.kernel.org Subject: [PATCHv2] percpu: make this_cpu_generic_read() atomic w.r.t. interrupts Date: Tue, 26 Sep 2017 12:41:52 +0100 Message-Id: <1506426112-19826-1-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org As raw_cpu_generic_read() is a plain read from a raw_cpu_ptr() address, it's possible (albeit unlikely) that the compiler will split the access across multiple instructions. In this_cpu_generic_read() we disable preemption but not interrupts before calling raw_cpu_generic_read(). Thus, an interrupt could be taken in the middle of the split load instructions. If a this_cpu_write() or RMW this_cpu_*() op is made to the same variable in the interrupt handling path, this_cpu_read() will return a torn value. For native word types, we can avoid tearing using READ_ONCE(), but this won't work in all cases (e.g. 64-bit types on most 32-bit platforms). This patch reworks this_cpu_generic_read() to use READ_ONCE() where possible, otherwise falling back to disabling interrupts. Signed-off-by: Mark Rutland Cc: Arnd Bergmann Cc: Christoph Lameter Cc: Peter Zijlstra Cc: Pranith Kumar Cc: Tejun Heo Cc: Thomas Gleixner Cc: linux-arch@vger.kernel.org --- include/asm-generic/percpu.h | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) Since v1 [1]: * Use READ_ONCE() only for native word types * Disable IRQs for non-native word types [1] https://lkml.kernel.org/r/1506345872-30559-1-git-send-email-mark.rutland@arm.com -- 1.9.1 diff --git a/include/asm-generic/percpu.h b/include/asm-generic/percpu.h index 0504ef8..976f8ac 100644 --- a/include/asm-generic/percpu.h +++ b/include/asm-generic/percpu.h @@ -115,15 +115,35 @@ (__ret); \ }) -#define this_cpu_generic_read(pcp) \ +#define __this_cpu_generic_read_nopreempt(pcp) \ ({ \ typeof(pcp) __ret; \ preempt_disable_notrace(); \ - __ret = raw_cpu_generic_read(pcp); \ + __ret = READ_ONCE(*raw_cpu_ptr(&(pcp))); \ preempt_enable_notrace(); \ __ret; \ }) +#define __this_cpu_generic_read_noirq(pcp) \ +({ \ + typeof(pcp) __ret; \ + unsigned long __flags; \ + raw_local_irq_save(__flags); \ + __ret = raw_cpu_generic_read(pcp); \ + raw_local_irq_restore(__flags); \ + __ret; \ +}) + +#define this_cpu_generic_read(pcp) \ +({ \ + typeof(pcp) __ret; \ + if (__native_word(pcp)) \ + __ret = __this_cpu_generic_read_nopreempt(pcp); \ + else \ + __ret = __this_cpu_generic_read_noirq(pcp); \ + __ret; \ +}) + #define this_cpu_generic_to_op(pcp, val, op) \ do { \ unsigned long __flags; \