From patchwork Mon Jun 25 10:59:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 139824 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp3839326lji; Mon, 25 Jun 2018 04:01:26 -0700 (PDT) X-Google-Smtp-Source: ADUXVKIzhK2t4OT1t/1/jEKNWs/NmhFmvddmNTGiQOLZkrnV+CIMYjf8C3rehzy81qnM4728Dg43 X-Received: by 2002:a63:7b1b:: with SMTP id w27-v6mr9921380pgc.199.1529924486640; Mon, 25 Jun 2018 04:01:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529924486; cv=none; d=google.com; s=arc-20160816; b=O/hXJLzG0j4viXuWV9DLpqngpnbxmWC+uGbYpZ1y4IBVvNhu1AK0fi88DerhQFycyo XSpmkp9PPxkSfbpfl/juF6mm3XqvqAkX7zMr9gEGf5w6iWw6p07JaA5PGH6cb/YyJxBi e086kJlYPuJkJplQwHyekpSMk7cc/3iFN8fwCzuZUVwNeCUjhbDP7q5VoOcXTLLVwt8x FU3LtVS2M4fSJapekuIpfNuBSMx2e6zMzQdlkQGz6VfrjxL4qjF3DG3XD4Ja4xu5l2w4 H2XUT8WCtudPW3E+wtBdQf/keQ2uW4KP1qj9XypU8qPN7HYb/BihtdE85AsdJEoDQIL0 TI2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=T/scNnJ4L9PdoLU4V9nn0hIZaeiLAaOyJKdkQLyXukA=; b=KFSQEYfG/v6JJbxwVLoSX7Q6nKbBDDM0AY8lwAWVQGPIRBZjxoDDE2evnQyJ1wxwQ8 bbNsL1HVAQ4Xd2slgy6ky1wAl1Xw38n+lZAMZyjlb8Oe8WpzQo+bXllXWNSgYDpwPsHc /awXElA1n5gSrFRGirssm0g70LGKdPNwdwxA3aMUThK7QKr0ZTAf8u2bOKQEcVK6niei S8Gy2KAyah1yo4BV8ny0tSWfI4bB/qbQsfX6WosKbnfIDEHKo5f/y+RVescR8jzB589E OIx/wXMBCK7isv5AQJVK0KuOctYETr4h5Gvws4CFlbVdjqCRRyNVXejjozjyrPrAbOFc uJeQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id v16-v6si14169294plo.186.2018.06.25.04.01.26; Mon, 25 Jun 2018 04:01:26 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932944AbeFYLBZ (ORCPT + 31 others); Mon, 25 Jun 2018 07:01:25 -0400 Received: from foss.arm.com ([217.140.101.70]:54600 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932353AbeFYLAN (ORCPT ); Mon, 25 Jun 2018 07:00:13 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C72EA22D9; Mon, 25 Jun 2018 04:00:12 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9682D3F73D; Mon, 25 Jun 2018 04:00:11 -0700 (PDT) From: Mark Rutland To: linux-kernel@vger.kernel.org, will.deacon@arm.com, peterz@infradead.org, boqun.feng@gmail.com Cc: Mark Rutland , Andrea Parri Subject: [PATCHv2 06/11] atomics/treewide: rework ordering barriers Date: Mon, 25 Jun 2018 11:59:47 +0100 Message-Id: <20180625105952.3756-7-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180625105952.3756-1-mark.rutland@arm.com> References: <20180625105952.3756-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Currently architectures can override __atomic_op_*() to define the barriers used before/after a relaxed atomic when used to build acquire/release/fence variants. This has the unfortunate property of requiring the architecture to define the full wrapper for the atomics, rather than just the barriers they care about, and gets in the way of generating atomics which can be easily read. Instead, this patch has architectures define an optional set of barriers, __atomic_mb_{before,after}_{acquire,release,fence}(), which uses to build the wrappers. At the same time, let's avoid any potential reliance on these elsewhere by ensuring each is undef'd at the end of . There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Boqun Feng Cc: Peter Zijlstra Cc: Will Deacon Cc: Andrea Parri --- arch/alpha/include/asm/atomic.h | 8 +++---- arch/powerpc/include/asm/atomic.h | 17 +++++--------- arch/riscv/include/asm/atomic.h | 17 +++++--------- include/linux/atomic.h | 47 ++++++++++++++++++++++++++------------- 4 files changed, 45 insertions(+), 44 deletions(-) -- 2.11.0 diff --git a/arch/alpha/include/asm/atomic.h b/arch/alpha/include/asm/atomic.h index 4a6a8f58c9c9..df8d9aa2a47e 100644 --- a/arch/alpha/include/asm/atomic.h +++ b/arch/alpha/include/asm/atomic.h @@ -18,11 +18,11 @@ * To ensure dependency ordering is preserved for the _relaxed and * _release atomics, an smp_read_barrier_depends() is unconditionally * inserted into the _relaxed variants, which are used to build the - * barriered versions. To avoid redundant back-to-back fences, we can - * define the _acquire and _fence versions explicitly. + * barriered versions. Avoid redundant back-to-back fences in the + * _acquire and _fence versions. */ -#define __atomic_op_acquire(op, args...) op##_relaxed(args) -#define __atomic_op_fence __atomic_op_release +#define __atomic_acquire_fence() +#define __atomic_post_fence() #define ATOMIC_INIT(i) { (i) } #define ATOMIC64_INIT(i) { (i) } diff --git a/arch/powerpc/include/asm/atomic.h b/arch/powerpc/include/asm/atomic.h index a0156cb43d1f..963abf8bf1c0 100644 --- a/arch/powerpc/include/asm/atomic.h +++ b/arch/powerpc/include/asm/atomic.h @@ -18,18 +18,11 @@ * a "bne-" instruction at the end, so an isync is enough as a acquire barrier * on the platform without lwsync. */ -#define __atomic_op_acquire(op, args...) \ -({ \ - typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \ - __asm__ __volatile__(PPC_ACQUIRE_BARRIER "" : : : "memory"); \ - __ret; \ -}) - -#define __atomic_op_release(op, args...) \ -({ \ - __asm__ __volatile__(PPC_RELEASE_BARRIER "" : : : "memory"); \ - op##_relaxed(args); \ -}) +#define __atomic_acquire_fence() \ + __asm__ __volatile__(PPC_ACQUIRE_BARRIER "" : : : "memory") + +#define __atomic_release_fence() \ + __asm__ __volatile__(PPC_RELEASE_BARRIER "" : : : "memory") static __inline__ int atomic_read(const atomic_t *v) { diff --git a/arch/riscv/include/asm/atomic.h b/arch/riscv/include/asm/atomic.h index 512b89485790..c452359c9cb8 100644 --- a/arch/riscv/include/asm/atomic.h +++ b/arch/riscv/include/asm/atomic.h @@ -25,18 +25,11 @@ #define ATOMIC_INIT(i) { (i) } -#define __atomic_op_acquire(op, args...) \ -({ \ - typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \ - __asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory"); \ - __ret; \ -}) - -#define __atomic_op_release(op, args...) \ -({ \ - __asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory"); \ - op##_relaxed(args); \ -}) +#define __atomic_acquire_fence() \ + __asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory") + +#define __atomic_release_fence() \ + __asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory"); static __always_inline int atomic_read(const atomic_t *v) { diff --git a/include/linux/atomic.h b/include/linux/atomic.h index 8e04f1f69bd9..fc39e4bead30 100644 --- a/include/linux/atomic.h +++ b/include/linux/atomic.h @@ -38,40 +38,46 @@ * barriers on top of the relaxed variant. In the case where the relaxed * variant is already fully ordered, no additional barriers are needed. * - * Besides, if an arch has a special barrier for acquire/release, it could - * implement its own __atomic_op_* and use the same framework for building - * variants - * - * If an architecture overrides __atomic_op_acquire() it will probably want - * to define smp_mb__after_spinlock(). + * If an architecture overrides __atomic_acquire_fence() it will probably + * want to define smp_mb__after_spinlock(). */ -#ifndef __atomic_op_acquire +#ifndef __atomic_acquire_fence +#define __atomic_acquire_fence smp_mb__after_atomic +#endif + +#ifndef __atomic_release_fence +#define __atomic_release_fence smp_mb__before_atomic +#endif + +#ifndef __atomic_pre_fence +#define __atomic_pre_fence smp_mb__before_atomic +#endif + +#ifndef __atomic_post_fence +#define __atomic_post_fence smp_mb__after_atomic +#endif + #define __atomic_op_acquire(op, args...) \ ({ \ typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \ - smp_mb__after_atomic(); \ + __atomic_acquire_fence(); \ __ret; \ }) -#endif -#ifndef __atomic_op_release #define __atomic_op_release(op, args...) \ ({ \ - smp_mb__before_atomic(); \ + __atomic_release_fence(); \ op##_relaxed(args); \ }) -#endif -#ifndef __atomic_op_fence #define __atomic_op_fence(op, args...) \ ({ \ typeof(op##_relaxed(args)) __ret; \ - smp_mb__before_atomic(); \ + __atomic_pre_fence(); \ __ret = op##_relaxed(args); \ - smp_mb__after_atomic(); \ + __atomic_post_fence(); \ __ret; \ }) -#endif /* atomic_add_return_relaxed */ #ifndef atomic_add_return_relaxed @@ -1308,4 +1314,13 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v) #include +#undef __atomic_op_acquire +#undef __atomic_op_release +#undef __atomic_op_fence + +#undef __atomic_acquire_fence +#undef __atomic_release_fence +#undef __atomic_pre_fence +#undef __atomic_post_fence + #endif /* _LINUX_ATOMIC_H */