From patchwork Thu Mar 19 14:52:31 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Capper X-Patchwork-Id: 46036 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-wg0-f71.google.com (mail-wg0-f71.google.com [74.125.82.71]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 01C8621515 for ; Thu, 19 Mar 2015 14:55:04 +0000 (UTC) Received: by wghl18 with SMTP id l18sf12900116wgh.1 for ; Thu, 19 Mar 2015 07:55:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:subject:date:message-id:cc :precedence:list-id:list-unsubscribe:list-archive:list-post :list-help:list-subscribe:mime-version:content-type :content-transfer-encoding:sender:errors-to:x-original-sender :x-original-authentication-results:mailing-list; bh=OtcnM6O0A7D6C0JZ4di5cmGQeDxviDvohVYhb/0dGTA=; b=dUZbNgi4hT23Qq5q3ZlGXRnvgbDrQHAGgv7rGQ94cim4vkp9AUVVlSKmwHHVYTUJhV TEIpZ01KZl6ai/SCTQ+Bnix/OxftpLNepCibGoh39qtLQvBxx3In9cLE/JB5BIY6Ncge 097RQ+yTvjMVeCogQLcj8vXSpnuL7y5aO/a3rqWaKogxvRKwhpFuT3Y9D1icVVyRehvD FRITZfBwUX1hxjyja/GUl1DYzzxsSAYdsEkcSbHbAqAuENzrOqnAkyJPeyJ3F6ruGdqD uXDiF7QcTnoVppj/LuVtjSi8FqkpA+vMu66ZZWW/kr79m1rdUSJY+Rl/UGHgf/Qwz4SH JrcA== X-Gm-Message-State: ALoCoQkcfhWUXLUpLrX1dgOWucmQu0Se38fIj8lox8q1WjbzvmMz90lpmr7TyWxh6B5lBvnsp//V X-Received: by 10.194.53.225 with SMTP id e1mr12180370wjp.4.1426776903167; Thu, 19 Mar 2015 07:55:03 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.228.169 with SMTP id sj9ls321880lac.35.gmail; Thu, 19 Mar 2015 07:55:02 -0700 (PDT) X-Received: by 10.112.85.165 with SMTP id i5mr70930901lbz.7.1426776902734; Thu, 19 Mar 2015 07:55:02 -0700 (PDT) Received: from mail-lb0-f180.google.com (mail-lb0-f180.google.com. [209.85.217.180]) by mx.google.com with ESMTPS id p9si1104427lap.178.2015.03.19.07.55.02 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 19 Mar 2015 07:55:02 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.180 as permitted sender) client-ip=209.85.217.180; Received: by lbcgn8 with SMTP id gn8so54583015lbc.2 for ; Thu, 19 Mar 2015 07:55:02 -0700 (PDT) X-Received: by 10.152.120.134 with SMTP id lc6mr68082138lab.72.1426776902618; Thu, 19 Mar 2015 07:55:02 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.35.133 with SMTP id h5csp439401lbj; Thu, 19 Mar 2015 07:55:01 -0700 (PDT) X-Received: by 10.70.56.34 with SMTP id x2mr175213053pdp.127.1426776900487; Thu, 19 Mar 2015 07:55:00 -0700 (PDT) Received: from bombadil.infradead.org (bombadil.infradead.org. [2001:1868:205::9]) by mx.google.com with ESMTPS id i4si3348265pdd.147.2015.03.19.07.54.59 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 19 Mar 2015 07:55:00 -0700 (PDT) Received-SPF: none (google.com: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org does not designate permitted sender hosts) client-ip=2001:1868:205::9; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1YYbos-0000oK-DV; Thu, 19 Mar 2015 14:53:10 +0000 Received: from mail-wg0-f46.google.com ([74.125.82.46]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1YYbol-0000bY-6d for linux-arm-kernel@lists.infradead.org; Thu, 19 Mar 2015 14:53:04 +0000 Received: by wgra20 with SMTP id a20so64613300wgr.3 for ; Thu, 19 Mar 2015 07:52:40 -0700 (PDT) X-Received: by 10.180.214.99 with SMTP id nz3mr17374934wic.82.1426776760390; Thu, 19 Mar 2015 07:52:40 -0700 (PDT) Received: from marmot.wormnet.eu (marmot.wormnet.eu. [188.246.204.87]) by mx.google.com with ESMTPSA id k1sm2296949wjn.9.2015.03.19.07.52.39 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 19 Mar 2015 07:52:39 -0700 (PDT) From: Steve Capper To: linux-arm-kernel@lists.infradead.org Subject: [PATCH] arm64: percpu: Make this_cpu accessors pre-empt safe Date: Thu, 19 Mar 2015 14:52:31 +0000 Message-Id: <1426776751-20526-1-git-send-email-steve.capper@linaro.org> X-Mailer: git-send-email 1.7.10.4 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20150319_075303_559697_A0F78F91 X-CRM114-Status: GOOD ( 11.27 ) X-Spam-Score: -0.7 (/) X-Spam-Report: SpamAssassin version 3.4.0 on bombadil.infradead.org summary: Content analysis details: (-0.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low trust [74.125.82.46 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record -0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3) [74.125.82.46 listed in wl.mailspike.net] -0.0 RCVD_IN_MSPIKE_WL Mailspike good senders Cc: mark.rutland@arm.com, catalin.marinas@arm.com, will.deacon@arm.com, Steve Capper X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: steve.capper@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.180 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 this_cpu operations were implemented for arm64 in: 5284e1b arm64: xchg: Implement cmpxchg_double f97fc81 arm64: percpu: Implement this_cpu operations Unfortunately, it is possible for pre-emption to take place between address generation and data access. This can lead to cases where data is being manipulated by this_cpu for a different CPU than it was called on. Which effectively breaks the spec. This patch disables pre-emption for the this_cpu operations guaranteeing that address generation and data manipulation. Fixes: 5284e1b: arm64: xchg: Implement cmpxchg_double Fixes: f97fc81: arm64: percpu: Implement this_cpu operations Reported-by: Mark Rutland Signed-off-by: Steve Capper --- arch/arm64/include/asm/cmpxchg.h | 32 ++++++++++++++++------ arch/arm64/include/asm/percpu.h | 57 ++++++++++++++++++++++++++++++++-------- 2 files changed, 70 insertions(+), 19 deletions(-) diff --git a/arch/arm64/include/asm/cmpxchg.h b/arch/arm64/include/asm/cmpxchg.h index cb95930..0ff749b 100644 --- a/arch/arm64/include/asm/cmpxchg.h +++ b/arch/arm64/include/asm/cmpxchg.h @@ -246,14 +246,30 @@ static inline unsigned long __cmpxchg_mb(volatile void *ptr, unsigned long old, __ret; \ }) -#define this_cpu_cmpxchg_1(ptr, o, n) cmpxchg_local(raw_cpu_ptr(&(ptr)), o, n) -#define this_cpu_cmpxchg_2(ptr, o, n) cmpxchg_local(raw_cpu_ptr(&(ptr)), o, n) -#define this_cpu_cmpxchg_4(ptr, o, n) cmpxchg_local(raw_cpu_ptr(&(ptr)), o, n) -#define this_cpu_cmpxchg_8(ptr, o, n) cmpxchg_local(raw_cpu_ptr(&(ptr)), o, n) - -#define this_cpu_cmpxchg_double_8(ptr1, ptr2, o1, o2, n1, n2) \ - cmpxchg_double_local(raw_cpu_ptr(&(ptr1)), raw_cpu_ptr(&(ptr2)), \ - o1, o2, n1, n2) +#define _protect_cmpxchg_local(pcp, o, n) \ +({ \ + typeof(*raw_cpu_ptr(&(pcp))) __ret; \ + __pcp_preempt_disable(); \ + __ret = cmpxchg_local(raw_cpu_ptr(&(pcp)), o, n); \ + __pcp_preempt_enable(); \ + __ret; \ +}) + +#define this_cpu_cmpxchg_1(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_2(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_4(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_8(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) + +#define this_cpu_cmpxchg_double_8(ptr1, ptr2, o1, o2, n1, n2) \ +({ \ + int __ret; \ + __pcp_preempt_disable(); \ + __ret = cmpxchg_double_local( raw_cpu_ptr(&(ptr1)), \ + raw_cpu_ptr(&(ptr2)), \ + o1, o2, n1, n2); \ + __pcp_preempt_enable(); \ + __ret; \ +}) #define cmpxchg64(ptr,o,n) cmpxchg((ptr),(o),(n)) #define cmpxchg64_local(ptr,o,n) cmpxchg_local((ptr),(o),(n)) diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h index 09da25b..94f118e 100644 --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -204,25 +204,60 @@ static inline unsigned long __percpu_xchg(void *ptr, unsigned long val, return ret; } +/* + * Modules aren't allowed to use preempt_enable_no_resched, and it is + * undef'ed. If we are unable to use preempt_enable_no_resched, then + * fallback to the standard preempt_enable. + */ +#ifdef preempt_enable_no_resched +#define __pcp_preempt_enable() preempt_enable_no_resched() +#else +#define __pcp_preempt_enable() preempt_enable() +#endif /* preempt_enable_no_resched */ + +#define __pcp_preempt_disable() preempt_disable() + +#define _percpu_read(pcp) \ +({ \ + typeof(pcp) __retval; \ + __pcp_preempt_disable(); \ + __retval = (typeof(pcp)) __percpu_read(raw_cpu_ptr(&(pcp)), \ + sizeof(pcp)); \ + __pcp_preempt_enable(); \ + __retval; \ +}) + +#define _percpu_write(pcp, val) \ +do { \ + __pcp_preempt_disable(); \ + __percpu_write(raw_cpu_ptr(&(pcp)), (unsigned long) (val), \ + sizeof(pcp)); \ + __pcp_preempt_enable(); \ +} while(0) \ + +#define _pcp_protect(operation, pcp, val) \ +({ \ + typeof(pcp) __retval; \ + __pcp_preempt_disable(); \ + __retval = (typeof(pcp)) operation(raw_cpu_ptr(&(pcp)), \ + (val), sizeof(pcp)); \ + __pcp_preempt_enable(); \ + __retval; \ +}) + #define _percpu_add(pcp, val) \ - __percpu_add(raw_cpu_ptr(&(pcp)), val, sizeof(pcp)) + _pcp_protect(__percpu_add, pcp, val) -#define _percpu_add_return(pcp, val) (typeof(pcp)) (_percpu_add(pcp, val)) +#define _percpu_add_return(pcp, val) _percpu_add(pcp, val) #define _percpu_and(pcp, val) \ - __percpu_and(raw_cpu_ptr(&(pcp)), val, sizeof(pcp)) + _pcp_protect(__percpu_and, pcp, val) #define _percpu_or(pcp, val) \ - __percpu_or(raw_cpu_ptr(&(pcp)), val, sizeof(pcp)) - -#define _percpu_read(pcp) (typeof(pcp)) \ - (__percpu_read(raw_cpu_ptr(&(pcp)), sizeof(pcp))) - -#define _percpu_write(pcp, val) \ - __percpu_write(raw_cpu_ptr(&(pcp)), (unsigned long)(val), sizeof(pcp)) + _pcp_protect(__percpu_or, pcp, val) #define _percpu_xchg(pcp, val) (typeof(pcp)) \ - (__percpu_xchg(raw_cpu_ptr(&(pcp)), (unsigned long)(val), sizeof(pcp))) + _pcp_protect(__percpu_xchg, pcp, (unsigned long)(val)) #define this_cpu_add_1(pcp, val) _percpu_add(pcp, val) #define this_cpu_add_2(pcp, val) _percpu_add(pcp, val)