From patchwork Mon Jul 16 11:30:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 142002 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp2305893ljj; Mon, 16 Jul 2018 04:30:59 -0700 (PDT) X-Google-Smtp-Source: AAOMgpc70AYU/1MCxDeOyB3QLxD/fnJcTRhzKxHHKlS7HfqyrWqCl0V3AMZY3bAIDSKQn+vx/1fP X-Received: by 2002:a63:cf10:: with SMTP id j16-v6mr15206121pgg.238.1531740659401; Mon, 16 Jul 2018 04:30:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1531740659; cv=none; d=google.com; s=arc-20160816; b=cGvt6X6IfJwO/KwMuVrjJC859N+wbE8TrTS7gR7XXLhpSQ+ROpMB7geO/5JRiG5tLC g6yTNNQJQ/VMj0yo+f0Y0lTPgr3gRf0Ypk7Q4QWF5XK0KANaU2jTRll4fX0S71Wtn4zU 2rA3XMPgbQKen5gz4wDQFPJUV14uhEsLIP5MYHn6KkFHC34wNhT33Fnd9t4/4DzTvFgi KKzaK9l1/ftPphdeQfi1p0LEvi+hlNJHQokXEPe4gmFnNY6hZovl7XAq90ohLBS0jVmt 2JadStbCr/UZR8/wvztaZ6iBbnLF2gya3O1BBSbis/DmE4/o5QaXVBP8kXkO1raeELga Lumg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=DgOhWE7VJTD4++7V0+tpv4bb9oe00InAO1fp5UySGKY=; b=VK7Ikb+nl4yDm5Txstpze/clq9W+oYyQ6lgWSehofsVgdB0/BG57WMgZSVfGXR2zvd nT3ibC6kCt8JcFN3Ir/ELABBkuXikRSIvKwq7S+XS/w/GROgy9AWMsHh0hLHUjiHgSuh kvO/9UboDYHB+gcnI6wX63g+8/PkG3EAGL2ptnoglDSJfjqUD2lQFqHCo6cdmX8l1g9l XGABzBnwizGf5nBOUMhlw+YgBfnRqPzi1IcPLQCrUtLsmOgmW4cHZG+R8qvv1yX3Go4m 9+o2mZQafOsgYOMR0aUeImfYs6nx/4lhv+ctolLnSEyCrqaeXQq6Z1TFj4KLSkna7u4v or/g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m21-v6si2487736pgh.664.2018.07.16.04.30.59; Mon, 16 Jul 2018 04:30:59 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731324AbeGPL54 (ORCPT + 31 others); Mon, 16 Jul 2018 07:57:56 -0400 Received: from foss.arm.com ([217.140.101.70]:57728 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728588AbeGPL54 (ORCPT ); Mon, 16 Jul 2018 07:57:56 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A6D851684; Mon, 16 Jul 2018 04:30:56 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B6D533F589; Mon, 16 Jul 2018 04:30:53 -0700 (PDT) From: Mark Rutland To: mingo@kernel.org, mingo@redhat.com Cc: andy.shevchenko@gmail.com, arnd@arndb.de, aryabinin@virtuozzo.com, boqun.feng@gmail.com, catalin.marinas@arm.com, dvyukov@google.com, glider@google.com, hpa@zytor.com, linux-kernel@vger.kernel.org, mark.rutland@arm.com, parri.andrea@gmail.com, peter@hurleysoftware.com, peterz@infradead.org, tglx@linutronix.de, will.deacon@arm.com, linux-arm-kernel@lists.infradead.org Subject: [PATCHv4 06/12] atomics/treewide: rework ordering barriers Date: Mon, 16 Jul 2018 12:30:11 +0100 Message-Id: <20180716113017.3909-7-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180716113017.3909-1-mark.rutland@arm.com> References: <20180716113017.3909-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Currently architectures can override __atomic_op_*() to define the barriers used before/after a relaxed atomic when used to build acquire/release/fence variants. This has the unfortunate property of requiring the architecture to define the full wrapper for the atomics, rather than just the barriers they care about, and gets in the way of generating atomics which can be easily read. Instead, this patch has architectures define an optional set of barriers: * __atomic_acquire_fence() * __atomic_release_fence() * __atomic_pre_full_fence() * __atomic_post_full_fence() ... which uses to build the wrappers. It would be nice if we could undef these, along with the __atomic_op_*() wrappers, but that would break the cmpxchg() wrappers, which are written in preprocessor. Undefs would have been nice, but alas. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Acked-by: Peter Zijlstra (Intel) Acked-by: Will Deacon Cc: Andrea Parri Cc: Boqun Feng Cc: Ingo Molnar --- arch/alpha/include/asm/atomic.h | 8 ++++---- arch/powerpc/include/asm/atomic.h | 17 +++++------------ arch/riscv/include/asm/atomic.h | 17 +++++------------ include/linux/atomic.h | 38 ++++++++++++++++++++++---------------- 4 files changed, 36 insertions(+), 44 deletions(-) -- 2.11.0 diff --git a/arch/alpha/include/asm/atomic.h b/arch/alpha/include/asm/atomic.h index 4a6a8f58c9c9..150a1c5d6a2c 100644 --- a/arch/alpha/include/asm/atomic.h +++ b/arch/alpha/include/asm/atomic.h @@ -18,11 +18,11 @@ * To ensure dependency ordering is preserved for the _relaxed and * _release atomics, an smp_read_barrier_depends() is unconditionally * inserted into the _relaxed variants, which are used to build the - * barriered versions. To avoid redundant back-to-back fences, we can - * define the _acquire and _fence versions explicitly. + * barriered versions. Avoid redundant back-to-back fences in the + * _acquire and _fence versions. */ -#define __atomic_op_acquire(op, args...) op##_relaxed(args) -#define __atomic_op_fence __atomic_op_release +#define __atomic_acquire_fence() +#define __atomic_post_full_fence() #define ATOMIC_INIT(i) { (i) } #define ATOMIC64_INIT(i) { (i) } diff --git a/arch/powerpc/include/asm/atomic.h b/arch/powerpc/include/asm/atomic.h index a0156cb43d1f..963abf8bf1c0 100644 --- a/arch/powerpc/include/asm/atomic.h +++ b/arch/powerpc/include/asm/atomic.h @@ -18,18 +18,11 @@ * a "bne-" instruction at the end, so an isync is enough as a acquire barrier * on the platform without lwsync. */ -#define __atomic_op_acquire(op, args...) \ -({ \ - typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \ - __asm__ __volatile__(PPC_ACQUIRE_BARRIER "" : : : "memory"); \ - __ret; \ -}) - -#define __atomic_op_release(op, args...) \ -({ \ - __asm__ __volatile__(PPC_RELEASE_BARRIER "" : : : "memory"); \ - op##_relaxed(args); \ -}) +#define __atomic_acquire_fence() \ + __asm__ __volatile__(PPC_ACQUIRE_BARRIER "" : : : "memory") + +#define __atomic_release_fence() \ + __asm__ __volatile__(PPC_RELEASE_BARRIER "" : : : "memory") static __inline__ int atomic_read(const atomic_t *v) { diff --git a/arch/riscv/include/asm/atomic.h b/arch/riscv/include/asm/atomic.h index 512b89485790..c452359c9cb8 100644 --- a/arch/riscv/include/asm/atomic.h +++ b/arch/riscv/include/asm/atomic.h @@ -25,18 +25,11 @@ #define ATOMIC_INIT(i) { (i) } -#define __atomic_op_acquire(op, args...) \ -({ \ - typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \ - __asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory"); \ - __ret; \ -}) - -#define __atomic_op_release(op, args...) \ -({ \ - __asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory"); \ - op##_relaxed(args); \ -}) +#define __atomic_acquire_fence() \ + __asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory") + +#define __atomic_release_fence() \ + __asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory"); static __always_inline int atomic_read(const atomic_t *v) { diff --git a/include/linux/atomic.h b/include/linux/atomic.h index 8e04f1f69bd9..1e8e88bdaf09 100644 --- a/include/linux/atomic.h +++ b/include/linux/atomic.h @@ -38,40 +38,46 @@ * barriers on top of the relaxed variant. In the case where the relaxed * variant is already fully ordered, no additional barriers are needed. * - * Besides, if an arch has a special barrier for acquire/release, it could - * implement its own __atomic_op_* and use the same framework for building - * variants - * - * If an architecture overrides __atomic_op_acquire() it will probably want - * to define smp_mb__after_spinlock(). + * If an architecture overrides __atomic_acquire_fence() it will probably + * want to define smp_mb__after_spinlock(). */ -#ifndef __atomic_op_acquire +#ifndef __atomic_acquire_fence +#define __atomic_acquire_fence smp_mb__after_atomic +#endif + +#ifndef __atomic_release_fence +#define __atomic_release_fence smp_mb__before_atomic +#endif + +#ifndef __atomic_pre_full_fence +#define __atomic_pre_full_fence smp_mb__before_atomic +#endif + +#ifndef __atomic_post_full_fence +#define __atomic_post_full_fence smp_mb__after_atomic +#endif + #define __atomic_op_acquire(op, args...) \ ({ \ typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \ - smp_mb__after_atomic(); \ + __atomic_acquire_fence(); \ __ret; \ }) -#endif -#ifndef __atomic_op_release #define __atomic_op_release(op, args...) \ ({ \ - smp_mb__before_atomic(); \ + __atomic_release_fence(); \ op##_relaxed(args); \ }) -#endif -#ifndef __atomic_op_fence #define __atomic_op_fence(op, args...) \ ({ \ typeof(op##_relaxed(args)) __ret; \ - smp_mb__before_atomic(); \ + __atomic_pre_full_fence(); \ __ret = op##_relaxed(args); \ - smp_mb__after_atomic(); \ + __atomic_post_full_fence(); \ __ret; \ }) -#endif /* atomic_add_return_relaxed */ #ifndef atomic_add_return_relaxed