From patchwork Thu Apr 26 10:34:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 134467 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp2078891lji; Thu, 26 Apr 2018 03:34:20 -0700 (PDT) X-Google-Smtp-Source: AIpwx4/3n52IPUiLs6Buv5+tG8XJONr5SnZE1GRlECqg713vdOnoTQ2R6duE7AfSU21t+qwxuzNj X-Received: by 2002:a17:902:8c91:: with SMTP id t17-v6mr14500355plo.182.1524738859923; Thu, 26 Apr 2018 03:34:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524738859; cv=none; d=google.com; s=arc-20160816; b=J+ByiZn45J74RtdiFQBNHW3OknNRO48qp0eP8hepX3850vX+S9aa12PsEmd4zcbpMz UkF46erbItSx50d7UxikKJ4LMjSxkQuwHpvg9WBCgXZhyHttYunxrQd8W6BaM396SIvp k9rmQLCRQP8zVSsl4bgFR4jnMsm8IrctblJYJ3gbstHehNR2WnYY8fQno7i7jxtt+hvM 2X1Okm6ls/b2JSd6tL7rMa5I2dfZVphdgxj4tp8KGrilHUILJNW+h36DVG3r4/VomQpi 7YgfZ7PZXArtJr1NQdGXHFNE43TaGt8QGRRNisSi4H5iNlnUtUck+lkSt6fALnEXpryV PF2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=tXA4VxgZFfNm2hYPnLeFRGZpvv7qavJLuOV5r/1iomk=; b=g42C3mcYhveltgK6Mact5WgivoLgViXdeb99iulXiAdwnnw7UYrlOiBFW+TZQ3i3dP KeUaHliBhvfny154NSdE40HRz9X/xm3f31ESsEonanm4EuUWPm7+RHsvYq3kdeq8MoML SWcDf5LUDrHnfhK66fWuIssmxkRfhW/odPGXiVgBd2Tu30BqxUyQu25NYslCAgpFyqvj xN+acaMrYdBAnIDTovloEgOdTX7IJURSk7OmBEuJZ/Pr+TYX872XbxCO8PrStQPHNzcH J0iZ/YEN3IeREkUygO3zQ8JhZf5Up7Gb5ULAK7jQDzMf8Rnt+jbCKMjVC32r8dBEcThr PMqA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id az2-v6si4315596plb.555.2018.04.26.03.34.19; Thu, 26 Apr 2018 03:34:19 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755468AbeDZKeR (ORCPT + 29 others); Thu, 26 Apr 2018 06:34:17 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:50982 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754428AbeDZKeJ (ORCPT ); Thu, 26 Apr 2018 06:34:09 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5DC53165D; Thu, 26 Apr 2018 03:34:09 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 2F8BA3F5AF; Thu, 26 Apr 2018 03:34:09 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id D42241AE4F9F; Thu, 26 Apr 2018 11:34:28 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, will.deacon@arm.com Subject: [PATCH v3 01/14] barriers: Introduce smp_cond_load_relaxed and atomic_cond_read_relaxed Date: Thu, 26 Apr 2018 11:34:15 +0100 Message-Id: <1524738868-31318-2-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1524738868-31318-1-git-send-email-will.deacon@arm.com> References: <1524738868-31318-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Whilst we currently provide smp_cond_load_acquire and atomic_cond_read_acquire, there are cases where the ACQUIRE semantics are not required because of a subsequent fence or release operation once the conditional loop has exited. This patch adds relaxed versions of the conditional spinning primitives to avoid unnecessary barrier overhead on architectures such as arm64. Signed-off-by: Will Deacon --- include/asm-generic/atomic-long.h | 2 ++ include/asm-generic/barrier.h | 27 +++++++++++++++++++++------ include/linux/atomic.h | 2 ++ 3 files changed, 25 insertions(+), 6 deletions(-) -- 2.1.4 diff --git a/include/asm-generic/atomic-long.h b/include/asm-generic/atomic-long.h index 34a028a7bcc5..5b2b0b5ea06d 100644 --- a/include/asm-generic/atomic-long.h +++ b/include/asm-generic/atomic-long.h @@ -244,6 +244,8 @@ static inline long atomic_long_add_unless(atomic_long_t *l, long a, long u) #define atomic_long_inc_not_zero(l) \ ATOMIC_LONG_PFX(_inc_not_zero)((ATOMIC_LONG_PFX(_t) *)(l)) +#define atomic_long_cond_read_relaxed(v, c) \ + ATOMIC_LONG_PFX(_cond_read_relaxed)((ATOMIC_LONG_PFX(_t) *)(v), (c)) #define atomic_long_cond_read_acquire(v, c) \ ATOMIC_LONG_PFX(_cond_read_acquire)((ATOMIC_LONG_PFX(_t) *)(v), (c)) diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h index 29458bbb2fa0..2cafdbb9ae4c 100644 --- a/include/asm-generic/barrier.h +++ b/include/asm-generic/barrier.h @@ -221,18 +221,17 @@ do { \ #endif /** - * smp_cond_load_acquire() - (Spin) wait for cond with ACQUIRE ordering + * smp_cond_load_relaxed() - (Spin) wait for cond with no ordering guarantees * @ptr: pointer to the variable to wait on * @cond: boolean expression to wait for * - * Equivalent to using smp_load_acquire() on the condition variable but employs - * the control dependency of the wait to reduce the barrier on many platforms. + * Equivalent to using READ_ONCE() on the condition variable. * * Due to C lacking lambda expressions we load the value of *ptr into a * pre-named variable @VAL to be used in @cond. */ -#ifndef smp_cond_load_acquire -#define smp_cond_load_acquire(ptr, cond_expr) ({ \ +#ifndef smp_cond_load_relaxed +#define smp_cond_load_relaxed(ptr, cond_expr) ({ \ typeof(ptr) __PTR = (ptr); \ typeof(*ptr) VAL; \ for (;;) { \ @@ -241,10 +240,26 @@ do { \ break; \ cpu_relax(); \ } \ - smp_acquire__after_ctrl_dep(); \ VAL; \ }) #endif +/** + * smp_cond_load_acquire() - (Spin) wait for cond with ACQUIRE ordering + * @ptr: pointer to the variable to wait on + * @cond: boolean expression to wait for + * + * Equivalent to using smp_load_acquire() on the condition variable but employs + * the control dependency of the wait to reduce the barrier on many platforms. + */ +#ifndef smp_cond_load_acquire +#define smp_cond_load_acquire(ptr, cond_expr) ({ \ + typeof(*ptr) _val; \ + _val = smp_cond_load_relaxed(ptr, cond_expr); \ + smp_acquire__after_ctrl_dep(); \ + _val; \ +}) +#endif + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_GENERIC_BARRIER_H */ diff --git a/include/linux/atomic.h b/include/linux/atomic.h index 8b276fd9a127..01ce3997cb42 100644 --- a/include/linux/atomic.h +++ b/include/linux/atomic.h @@ -654,6 +654,7 @@ static inline int atomic_dec_if_positive(atomic_t *v) } #endif +#define atomic_cond_read_relaxed(v, c) smp_cond_load_relaxed(&(v)->counter, (c)) #define atomic_cond_read_acquire(v, c) smp_cond_load_acquire(&(v)->counter, (c)) #ifdef CONFIG_GENERIC_ATOMIC64 @@ -1075,6 +1076,7 @@ static inline long long atomic64_fetch_andnot_release(long long i, atomic64_t *v } #endif +#define atomic64_cond_read_relaxed(v, c) smp_cond_load_relaxed(&(v)->counter, (c)) #define atomic64_cond_read_acquire(v, c) smp_cond_load_acquire(&(v)->counter, (c)) #include From patchwork Thu Apr 26 10:34:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 134468 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp2079029lji; Thu, 26 Apr 2018 03:34:26 -0700 (PDT) X-Google-Smtp-Source: AIpwx49pzQ37ZZK4Ty4448OqoZhvQNLiHHhyQ55Mh1jLmsRql4eGqxvKeBwtrJQyaLRclcxv47fS X-Received: by 2002:a17:902:22a:: with SMTP id 39-v6mr33512142plc.128.1524738866569; Thu, 26 Apr 2018 03:34:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524738866; cv=none; d=google.com; s=arc-20160816; b=WvhFo4AdRPCPhb0XiTH9/vIY5yh3xvyfBRDLBxC+zB7tN9qX3E3bnCNBXQgWyWNXpR jOMB9V0Tqz6ZXjg2L0Z3fV97LQ3fSpPv84o9+gmr3bcdhKB7G8uupRPtRlUdq+4/e2FB 6S1gyutRQnuXR62JY6T8t1yKZhJFyqlrGH6m3L311QMW7VpMqe3v2c4mJ+jUKgflp4iB QtQgrgJ+9MGgV7g6o8FieQjTEXm31K4JhqlpmwJPvkD5PbcJYundUpn992dnAdJeB4Pt 56EkO6LYllr9qEH95+Pu7uH7K7/9yw0fWXpQ0pP8znK7By1xwTfTWP20Nf6GHrWwHVWw shGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=o2BbQNHFiM7I44f0UrGiiNEL5ds+SkJXCc/viN5yII0=; b=CsqFUMM5jDSs+gLb/8Gfe8yOiR/9GV87gD/wKw8vk/rQajJYt1+rVTZOHoFM6LmpaM YoUhdY9WNwEVNSr9oX+gPDgiiPiti99iZ/OY2Va75terTJSZcKaGgyVjjkUNKoAxS7KX HrfKj+o5lxJKW0GjiwXmaWlIrUA9CthyPsP02WQi0F0aOP5ik/a1NvusAQzFNj8zBUdn mxWjBTHE6U5XNN5KMBHeU4jvxVzQk+fbQBArWU9SY24mmiuXXWns7OJEYyiMj2/nxHCS uPH+23FO1TCQnLqHJLGjeSuszu9XVwl0nhIwoD5dhiPsYlKou/USDPiD3Kzzxv8+mvDq FqOQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e15si14928706pga.573.2018.04.26.03.34.26; Thu, 26 Apr 2018 03:34:26 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755551AbeDZKeU (ORCPT + 29 others); Thu, 26 Apr 2018 06:34:20 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51020 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754805AbeDZKeJ (ORCPT ); Thu, 26 Apr 2018 06:34:09 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 94915168F; Thu, 26 Apr 2018 03:34:09 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4342C3F5B7; Thu, 26 Apr 2018 03:34:09 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id E41A71AE509C; Thu, 26 Apr 2018 11:34:28 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, will.deacon@arm.com Subject: [PATCH v3 02/14] locking/qspinlock: Merge struct __qspinlock into struct qspinlock Date: Thu, 26 Apr 2018 11:34:16 +0100 Message-Id: <1524738868-31318-3-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1524738868-31318-1-git-send-email-will.deacon@arm.com> References: <1524738868-31318-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org struct __qspinlock provides a handy union of fields so that subcomponents of the lockword can be accessed by name, without having to manage shifts and masks explicitly and take endianness into account. This is useful in qspinlock.h and also potentially in arch headers, so move the struct __qspinlock into struct qspinlock and kill the extra definition. Cc: Peter Zijlstra Cc: Ingo Molnar Acked-by: Boqun Feng Signed-off-by: Will Deacon --- arch/x86/include/asm/qspinlock.h | 2 +- arch/x86/include/asm/qspinlock_paravirt.h | 3 +- include/asm-generic/qspinlock_types.h | 32 +++++++++++++++++++-- kernel/locking/qspinlock.c | 46 ++----------------------------- kernel/locking/qspinlock_paravirt.h | 34 ++++++++--------------- 5 files changed, 46 insertions(+), 71 deletions(-) -- 2.1.4 diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h index 5e16b5d40d32..90b0b0ed8161 100644 --- a/arch/x86/include/asm/qspinlock.h +++ b/arch/x86/include/asm/qspinlock.h @@ -16,7 +16,7 @@ */ static inline void native_queued_spin_unlock(struct qspinlock *lock) { - smp_store_release((u8 *)lock, 0); + smp_store_release(&lock->locked, 0); } #ifdef CONFIG_PARAVIRT_SPINLOCKS diff --git a/arch/x86/include/asm/qspinlock_paravirt.h b/arch/x86/include/asm/qspinlock_paravirt.h index 923307ea11c7..9ef5ee03d2d7 100644 --- a/arch/x86/include/asm/qspinlock_paravirt.h +++ b/arch/x86/include/asm/qspinlock_paravirt.h @@ -22,8 +22,7 @@ PV_CALLEE_SAVE_REGS_THUNK(__pv_queued_spin_unlock_slowpath); * * void __pv_queued_spin_unlock(struct qspinlock *lock) * { - * struct __qspinlock *l = (void *)lock; - * u8 lockval = cmpxchg(&l->locked, _Q_LOCKED_VAL, 0); + * u8 lockval = cmpxchg(&lock->locked, _Q_LOCKED_VAL, 0); * * if (likely(lockval == _Q_LOCKED_VAL)) * return; diff --git a/include/asm-generic/qspinlock_types.h b/include/asm-generic/qspinlock_types.h index 034acd0c4956..0763f065b975 100644 --- a/include/asm-generic/qspinlock_types.h +++ b/include/asm-generic/qspinlock_types.h @@ -29,13 +29,41 @@ #endif typedef struct qspinlock { - atomic_t val; + union { + atomic_t val; + + /* + * By using the whole 2nd least significant byte for the + * pending bit, we can allow better optimization of the lock + * acquisition for the pending bit holder. + */ +#ifdef __LITTLE_ENDIAN + struct { + u8 locked; + u8 pending; + }; + struct { + u16 locked_pending; + u16 tail; + }; +#else + struct { + u16 tail; + u16 locked_pending; + }; + struct { + u8 reserved[2]; + u8 pending; + u8 locked; + }; +#endif + }; } arch_spinlock_t; /* * Initializier */ -#define __ARCH_SPIN_LOCK_UNLOCKED { ATOMIC_INIT(0) } +#define __ARCH_SPIN_LOCK_UNLOCKED { .val = ATOMIC_INIT(0) } /* * Bitfields in the atomic value: diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index d880296245c5..f5b0e59f6d14 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -114,40 +114,6 @@ static inline __pure struct mcs_spinlock *decode_tail(u32 tail) #define _Q_LOCKED_PENDING_MASK (_Q_LOCKED_MASK | _Q_PENDING_MASK) -/* - * By using the whole 2nd least significant byte for the pending bit, we - * can allow better optimization of the lock acquisition for the pending - * bit holder. - * - * This internal structure is also used by the set_locked function which - * is not restricted to _Q_PENDING_BITS == 8. - */ -struct __qspinlock { - union { - atomic_t val; -#ifdef __LITTLE_ENDIAN - struct { - u8 locked; - u8 pending; - }; - struct { - u16 locked_pending; - u16 tail; - }; -#else - struct { - u16 tail; - u16 locked_pending; - }; - struct { - u8 reserved[2]; - u8 pending; - u8 locked; - }; -#endif - }; -}; - #if _Q_PENDING_BITS == 8 /** * clear_pending_set_locked - take ownership and clear the pending bit. @@ -159,9 +125,7 @@ struct __qspinlock { */ static __always_inline void clear_pending_set_locked(struct qspinlock *lock) { - struct __qspinlock *l = (void *)lock; - - WRITE_ONCE(l->locked_pending, _Q_LOCKED_VAL); + WRITE_ONCE(lock->locked_pending, _Q_LOCKED_VAL); } /* @@ -176,13 +140,11 @@ static __always_inline void clear_pending_set_locked(struct qspinlock *lock) */ static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail) { - struct __qspinlock *l = (void *)lock; - /* * Use release semantics to make sure that the MCS node is properly * initialized before changing the tail code. */ - return (u32)xchg_release(&l->tail, + return (u32)xchg_release(&lock->tail, tail >> _Q_TAIL_OFFSET) << _Q_TAIL_OFFSET; } @@ -237,9 +199,7 @@ static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail) */ static __always_inline void set_locked(struct qspinlock *lock) { - struct __qspinlock *l = (void *)lock; - - WRITE_ONCE(l->locked, _Q_LOCKED_VAL); + WRITE_ONCE(lock->locked, _Q_LOCKED_VAL); } diff --git a/kernel/locking/qspinlock_paravirt.h b/kernel/locking/qspinlock_paravirt.h index 6ee477765e6c..2711940429f5 100644 --- a/kernel/locking/qspinlock_paravirt.h +++ b/kernel/locking/qspinlock_paravirt.h @@ -87,8 +87,6 @@ struct pv_node { #define queued_spin_trylock(l) pv_hybrid_queued_unfair_trylock(l) static inline bool pv_hybrid_queued_unfair_trylock(struct qspinlock *lock) { - struct __qspinlock *l = (void *)lock; - /* * Stay in unfair lock mode as long as queued mode waiters are * present in the MCS wait queue but the pending bit isn't set. @@ -97,7 +95,7 @@ static inline bool pv_hybrid_queued_unfair_trylock(struct qspinlock *lock) int val = atomic_read(&lock->val); if (!(val & _Q_LOCKED_PENDING_MASK) && - (cmpxchg_acquire(&l->locked, 0, _Q_LOCKED_VAL) == 0)) { + (cmpxchg_acquire(&lock->locked, 0, _Q_LOCKED_VAL) == 0)) { qstat_inc(qstat_pv_lock_stealing, true); return true; } @@ -117,16 +115,12 @@ static inline bool pv_hybrid_queued_unfair_trylock(struct qspinlock *lock) #if _Q_PENDING_BITS == 8 static __always_inline void set_pending(struct qspinlock *lock) { - struct __qspinlock *l = (void *)lock; - - WRITE_ONCE(l->pending, 1); + WRITE_ONCE(lock->pending, 1); } static __always_inline void clear_pending(struct qspinlock *lock) { - struct __qspinlock *l = (void *)lock; - - WRITE_ONCE(l->pending, 0); + WRITE_ONCE(lock->pending, 0); } /* @@ -136,10 +130,8 @@ static __always_inline void clear_pending(struct qspinlock *lock) */ static __always_inline int trylock_clear_pending(struct qspinlock *lock) { - struct __qspinlock *l = (void *)lock; - - return !READ_ONCE(l->locked) && - (cmpxchg_acquire(&l->locked_pending, _Q_PENDING_VAL, + return !READ_ONCE(lock->locked) && + (cmpxchg_acquire(&lock->locked_pending, _Q_PENDING_VAL, _Q_LOCKED_VAL) == _Q_PENDING_VAL); } #else /* _Q_PENDING_BITS == 8 */ @@ -384,7 +376,6 @@ static void pv_wait_node(struct mcs_spinlock *node, struct mcs_spinlock *prev) static void pv_kick_node(struct qspinlock *lock, struct mcs_spinlock *node) { struct pv_node *pn = (struct pv_node *)node; - struct __qspinlock *l = (void *)lock; /* * If the vCPU is indeed halted, advance its state to match that of @@ -413,7 +404,7 @@ static void pv_kick_node(struct qspinlock *lock, struct mcs_spinlock *node) * the hash table later on at unlock time, no atomic instruction is * needed. */ - WRITE_ONCE(l->locked, _Q_SLOW_VAL); + WRITE_ONCE(lock->locked, _Q_SLOW_VAL); (void)pv_hash(lock, pn); } @@ -428,7 +419,6 @@ static u32 pv_wait_head_or_lock(struct qspinlock *lock, struct mcs_spinlock *node) { struct pv_node *pn = (struct pv_node *)node; - struct __qspinlock *l = (void *)lock; struct qspinlock **lp = NULL; int waitcnt = 0; int loop; @@ -479,13 +469,13 @@ pv_wait_head_or_lock(struct qspinlock *lock, struct mcs_spinlock *node) * * Matches the smp_rmb() in __pv_queued_spin_unlock(). */ - if (xchg(&l->locked, _Q_SLOW_VAL) == 0) { + if (xchg(&lock->locked, _Q_SLOW_VAL) == 0) { /* * The lock was free and now we own the lock. * Change the lock value back to _Q_LOCKED_VAL * and unhash the table. */ - WRITE_ONCE(l->locked, _Q_LOCKED_VAL); + WRITE_ONCE(lock->locked, _Q_LOCKED_VAL); WRITE_ONCE(*lp, NULL); goto gotlock; } @@ -493,7 +483,7 @@ pv_wait_head_or_lock(struct qspinlock *lock, struct mcs_spinlock *node) WRITE_ONCE(pn->state, vcpu_hashed); qstat_inc(qstat_pv_wait_head, true); qstat_inc(qstat_pv_wait_again, waitcnt); - pv_wait(&l->locked, _Q_SLOW_VAL); + pv_wait(&lock->locked, _Q_SLOW_VAL); /* * Because of lock stealing, the queue head vCPU may not be @@ -518,7 +508,6 @@ pv_wait_head_or_lock(struct qspinlock *lock, struct mcs_spinlock *node) __visible void __pv_queued_spin_unlock_slowpath(struct qspinlock *lock, u8 locked) { - struct __qspinlock *l = (void *)lock; struct pv_node *node; if (unlikely(locked != _Q_SLOW_VAL)) { @@ -547,7 +536,7 @@ __pv_queued_spin_unlock_slowpath(struct qspinlock *lock, u8 locked) * Now that we have a reference to the (likely) blocked pv_node, * release the lock. */ - smp_store_release(&l->locked, 0); + smp_store_release(&lock->locked, 0); /* * At this point the memory pointed at by lock can be freed/reused, @@ -573,7 +562,6 @@ __pv_queued_spin_unlock_slowpath(struct qspinlock *lock, u8 locked) #ifndef __pv_queued_spin_unlock __visible void __pv_queued_spin_unlock(struct qspinlock *lock) { - struct __qspinlock *l = (void *)lock; u8 locked; /* @@ -581,7 +569,7 @@ __visible void __pv_queued_spin_unlock(struct qspinlock *lock) * unhash. Otherwise it would be possible to have multiple @lock * entries, which would be BAD. */ - locked = cmpxchg_release(&l->locked, _Q_LOCKED_VAL, 0); + locked = cmpxchg_release(&lock->locked, _Q_LOCKED_VAL, 0); if (likely(locked == _Q_LOCKED_VAL)) return; From patchwork Thu Apr 26 10:34:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 134481 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp2083757lji; Thu, 26 Apr 2018 03:38:05 -0700 (PDT) X-Google-Smtp-Source: AIpwx4+LSElvSpsfJdQycZl1BF5QTQ2TGjJi6Ywgs6pZTX3x+/5oQlK0k5mmLRl+kV6GUlFvlFUb X-Received: by 10.98.72.74 with SMTP id v71mr31724026pfa.241.1524739085633; Thu, 26 Apr 2018 03:38:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524739085; cv=none; d=google.com; s=arc-20160816; b=fMK38+kVABcqfqZTyTBxYcE27iOCd0LupEM+NvHximA8tFRktkUep3ZLBUeDocGtIZ SnS7nJEYIfNRsixqwX57gfmQWXkn5WRv2oEGXwFbqi8RqsBwRGAQqlfEBXG1CAFYAXEs lsp5QINli4swIw2NdMBhjoCOUXjwCs85lyp5NeuV8poOgWc1y+rUwJs4hXywzEAKSPvV 9MTTFw9z4+qQn9cKJeluy9K49OyGJeRn0HPoUQljz9gXSMbzM6Vnc4mtQXnI5THAgu6W vykQAUOwgJAmol61OX5UhGPJ8adxI2WAbOx7zap+plDz9zBqZ0zXp6x0zc1DuLYu3Yvt K8dA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=xc6s9HS5ti7pmWSBFcBDmDyJIxQZQ6uQpM22zxIDbSk=; b=rHbQKUa54j/fbNPUR9277pt952SRG4uzSHAWkIo9HqG6QV9odr39vUm6Pi7wsS1m9l ENXPoqF3WJGdwVJ0V7RjaSPz3umhVH/6Ydgza60fcnC8zu/TTH1iiQ81NH4uAubAw1FP PojAGP03T2CvCU8uFKv4l3MMzzQhf3OoquvM+pvjKkzGWBOnqIvZycC4K2o+TPVB0I6N VyxDZRrzIXfs+NJtnz8qmdChnMQ8aoi8St6F54QWNSGTLXCnLjL6Tqw0EmRsrQTy186N 1pVzlNEaR9Ek5DZav75MPjvlX1ZqHXATE6P7FdXAeapLhZqhLGj6rKD2r9/dFNOcLb5r zX3g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b2si15721607pgc.275.2018.04.26.03.38.05; Thu, 26 Apr 2018 03:38:05 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755967AbeDZKiD (ORCPT + 29 others); Thu, 26 Apr 2018 06:38:03 -0400 Received: from foss.arm.com ([217.140.101.70]:51004 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754678AbeDZKeJ (ORCPT ); Thu, 26 Apr 2018 06:34:09 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 83D7B1688; Thu, 26 Apr 2018 03:34:09 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 559D93F487; Thu, 26 Apr 2018 03:34:09 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 0450C1AE50BE; Thu, 26 Apr 2018 11:34:28 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, will.deacon@arm.com Subject: [PATCH v3 03/14] locking/qspinlock: Bound spinning on pending->locked transition in slowpath Date: Thu, 26 Apr 2018 11:34:17 +0100 Message-Id: <1524738868-31318-4-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1524738868-31318-1-git-send-email-will.deacon@arm.com> References: <1524738868-31318-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org If a locker taking the qspinlock slowpath reads a lock value indicating that only the pending bit is set, then it will spin whilst the concurrent pending->locked transition takes effect. Unfortunately, there is no guarantee that such a transition will ever be observed since concurrent lockers could continuously set pending and hand over the lock amongst themselves, leading to starvation. Whilst this would probably resolve in practice, it means that it is not possible to prove liveness properties about the lock and means that lock acquisition time is unbounded. Rather than removing the pending->locked spinning from the slowpath altogether (which has been shown to heavily penalise a 2-threaded locking stress test on x86), this patch replaces the explicit spinning with a call to atomic_cond_read_relaxed and allows the architecture to provide a bound on the number of spins. For architectures that can respond to changes in cacheline state in their smp_cond_load implementation, it should be sufficient to use the default bound of 1. Cc: Peter Zijlstra Cc: Ingo Molnar Suggested-by: Waiman Long Signed-off-by: Will Deacon --- kernel/locking/qspinlock.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index f5b0e59f6d14..a0f7976348f8 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -77,6 +77,18 @@ #endif /* + * The pending bit spinning loop count. + * This heuristic is used to limit the number of lockword accesses + * made by atomic_cond_read_relaxed when waiting for the lock to + * transition out of the "== _Q_PENDING_VAL" state. We don't spin + * indefinitely because there's no guarantee that we'll make forward + * progress. + */ +#ifndef _Q_PENDING_LOOPS +#define _Q_PENDING_LOOPS 1 +#endif + +/* * Per-CPU queue node structures; we can never have more than 4 nested * contexts: task, softirq, hardirq, nmi. * @@ -266,13 +278,15 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) return; /* - * wait for in-progress pending->locked hand-overs + * Wait for in-progress pending->locked hand-overs with a bounded + * number of spins so that we guarantee forward progress. * * 0,1,0 -> 0,0,1 */ if (val == _Q_PENDING_VAL) { - while ((val = atomic_read(&lock->val)) == _Q_PENDING_VAL) - cpu_relax(); + int cnt = _Q_PENDING_LOOPS; + val = atomic_cond_read_relaxed(&lock->val, + (VAL != _Q_PENDING_VAL) || !cnt--); } /* From patchwork Thu Apr 26 10:34:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 134469 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp2079083lji; Thu, 26 Apr 2018 03:34:29 -0700 (PDT) X-Google-Smtp-Source: AIpwx4+q87GhkOfAS4x+RbDXuiAxVJVk/SN6LWia/xhWe8Yx/Yjuf+UbLHsubcJQqm3b+RE5Mo0A X-Received: by 10.167.133.12 with SMTP id v12mr20842909pfn.90.1524738869426; Thu, 26 Apr 2018 03:34:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524738869; cv=none; d=google.com; s=arc-20160816; b=oXErxNh+7odT+vrLQ8S1gWzc1cJUWAZDBC6/bw7THq3WZF0me9gaclIYG4xjAhkUAl Suzygb9C//2J9NENGw1StVYIZMtHUchR5yaIkB44+vsRbFrrFLq4upfpfwlyFf6Q/R40 9UoD2yndpJRnlGgQdcODfn/7H5rDHhTOtHOHgHtKrwlkdBEnTgCiWn+i4N6YLe0+QvJo 1tRRqgQSM9VcjlkI6HEnQM2CjhKkysc8H4xIvPyjUYMfxn/zv/nx+hOsfD1f+Ba7SrJ4 TTOGFW5UfpARbRpbRHhorYFKcR6EpBsLhaE46XQ7Ta/KEtvGMA35UOU6elP24bflk+EM Wurw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=N6DvhPmlmCuRy8VmGQHWq4TvcqHsUTaCg374LC/OgXE=; b=AE7iv3lYfVi5TRHdQhtWwsG7wkrTZ6kC2TdxhzeqkNr66qiQon4Nh/XCG8zJsoIDF7 dD8kP2JkgGU9dh1pt8lHh7Qnj4k7vJfJMkDWnSabO8WdSOD798489D7CtWZ/ia1D7WHm cFR97g6cbCXW4JNlT1ztHQDi1iDV5Sb0E/9NBcpaXDCZ0VdmYLC+CuJk+bb2yNU5V2dq LhYpTWQVvEXUrIYHj/OBEHT7qcQdrkOoNxWPqoVE6U19PVdNfxWc+pChh0AFKwwI5MRh DBpuVtyB047p/e/+IZp/GzenlnlKw4OFaVXwCIzXiXgvlUi2aNGBR/NuRXDQpHXbx+ER HHQw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e15si14928706pga.573.2018.04.26.03.34.29; Thu, 26 Apr 2018 03:34:29 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755583AbeDZKeZ (ORCPT + 29 others); Thu, 26 Apr 2018 06:34:25 -0400 Received: from foss.arm.com ([217.140.101.70]:51038 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754862AbeDZKeJ (ORCPT ); Thu, 26 Apr 2018 06:34:09 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 95BEC169F; Thu, 26 Apr 2018 03:34:09 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 6640D3F77A; Thu, 26 Apr 2018 03:34:09 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 16A001AE50EB; Thu, 26 Apr 2018 11:34:29 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, will.deacon@arm.com Subject: [PATCH v3 04/14] locking/qspinlock/x86: Increase _Q_PENDING_LOOPS upper bound Date: Thu, 26 Apr 2018 11:34:18 +0100 Message-Id: <1524738868-31318-5-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1524738868-31318-1-git-send-email-will.deacon@arm.com> References: <1524738868-31318-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On x86, atomic_cond_read_relaxed will busy-wait with a cpu_relax() loop, so it is desirable to increase the number of times we spin on the qspinlock lockword when it is found to be transitioning from pending to locked. According to Waiman Long: | Ideally, the spinning times should be at least a few times the typical | cacheline load time from memory which I think can be down to 100ns or | so for each cacheline load with the newest systems or up to several | hundreds ns for older systems. which in his benchmarking corresponded to 512 iterations. Cc: Peter Zijlstra Cc: Ingo Molnar Suggested-by: Waiman Long Signed-off-by: Will Deacon --- arch/x86/include/asm/qspinlock.h | 2 ++ 1 file changed, 2 insertions(+) -- 2.1.4 diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h index 90b0b0ed8161..da1370ad206d 100644 --- a/arch/x86/include/asm/qspinlock.h +++ b/arch/x86/include/asm/qspinlock.h @@ -7,6 +7,8 @@ #include #include +#define _Q_PENDING_LOOPS (1 << 9) + #define queued_spin_unlock queued_spin_unlock /** * queued_spin_unlock - release a queued spinlock From patchwork Thu Apr 26 10:34:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 134479 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp2083548lji; Thu, 26 Apr 2018 03:37:57 -0700 (PDT) X-Google-Smtp-Source: AIpwx4/lIo0TgzABUBU1HVXGo21qXrJ/+Lg2KeOJ4e9J5thXAwBC9n6meGwuaCDGt2ItL/N+D0UV X-Received: by 10.99.137.74 with SMTP id v71mr27607584pgd.423.1524739077778; Thu, 26 Apr 2018 03:37:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524739077; cv=none; d=google.com; s=arc-20160816; b=jDoH8zY3336GXgY14fYjSjlB7k7wJ4+KeL5v+LosZPoDo/A09GQ/8CHsR68t0PGS12 DtFE6CZZLLiuf8azr+pvdrdTcMAZjhNd2j7im3G5fNQBpH5PywbLR3ZN/cmEUPd/JDHj gZij3mnQuPuBDRD99LvI/VL7LTTMqskn8BmY0FDhzvxn6m9RzHdjziI7CSPVp8+g+SN6 XUZSaEGPx6/hHJkzxO2lQQ3XwK6yVixzz0ZmkLyLaBnv/KgOkJcZY3X1R3VS7b5Uy6YF /fnx4+0wYMfdHB8JdWqCLFpJCedvXWxKziz1rlD1ZDBD3vpQP7Jtz47+z9FYyJrpXLtB zMaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=dF0rME0tKGup+o+u6kRe02oXhUsbeknlmqSWIZbZ9+s=; b=Bc/OPvbNIPYOetZePpYET9LFkAVbVd4FVbJ+JZS/1n4QhE0wDi741Cp8S01VwDYx+C DgpR5hMHY8O1GLbXq3G6UArDAUTWHeajirRafveCV/f4ow9Sd0RbCx52g00bFWzn3qTR q7U/ZZmnr+T0bI5i3BjmSXiVgakWAHE2A0/DDKCoIIrDnITTK/gh/D3pfEtb6nJ3EOUf teY4YmNTSa9LtViulXJ9XeSkVMqNbLdkU6CxqPoUXYm0uEHkHehc6gv1H8cd86T2DGgV Dq0G89csz2DuQGsLxOVacrBCce43HhP6EFBfmON+Y8/bzTK8MxQVvr26axCBj4KEi91J 5pbA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 59-v6si18771898plc.103.2018.04.26.03.37.57; Thu, 26 Apr 2018 03:37:57 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755772AbeDZKhz (ORCPT + 29 others); Thu, 26 Apr 2018 06:37:55 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51046 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754932AbeDZKeK (ORCPT ); Thu, 26 Apr 2018 06:34:10 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4339F16A3; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 14D6C3F487; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 26E601AE5107; Thu, 26 Apr 2018 11:34:29 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, will.deacon@arm.com Subject: [PATCH v3 05/14] locking/qspinlock: Remove unbounded cmpxchg loop from locking slowpath Date: Thu, 26 Apr 2018 11:34:19 +0100 Message-Id: <1524738868-31318-6-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1524738868-31318-1-git-send-email-will.deacon@arm.com> References: <1524738868-31318-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The qspinlock locking slowpath utilises a "pending" bit as a simple form of an embedded test-and-set lock that can avoid the overhead of explicit queuing in cases where the lock is held but uncontended. This bit is managed using a cmpxchg loop which tries to transition the uncontended lock word from (0,0,0) -> (0,0,1) or (0,0,1) -> (0,1,1). Unfortunately, the cmpxchg loop is unbounded and lockers can be starved indefinitely if the lock word is seen to oscillate between unlocked (0,0,0) and locked (0,0,1). This could happen if concurrent lockers are able to take the lock in the cmpxchg loop without queuing and pass it around amongst themselves. This patch fixes the problem by unconditionally setting _Q_PENDING_VAL using atomic_fetch_or, and then inspecting the old value to see whether we need to spin on the current lock owner, or whether we now effectively hold the lock. The tricky scenario is when concurrent lockers end up queuing on the lock and the lock becomes available, causing us to see a lockword of (n,0,0). With pending now set, simply queuing could lead to deadlock as the head of the queue may not have observed the pending flag being cleared. Conversely, if the head of the queue did observe pending being cleared, then it could transition the lock from (n,0,0) -> (0,0,1) meaning that any attempt to "undo" our setting of the pending bit could race with a concurrent locker trying to set it. We handle this race by preserving the pending bit when taking the lock after reaching the head of the queue and leaving the tail entry intact if we saw pending set, because we know that the tail is going to be updated shortly. Cc: Peter Zijlstra Cc: Ingo Molnar Signed-off-by: Will Deacon --- kernel/locking/qspinlock.c | 102 ++++++++++++++++++++---------------- kernel/locking/qspinlock_paravirt.h | 5 -- 2 files changed, 58 insertions(+), 49 deletions(-) -- 2.1.4 Reported-by: Waiman Long Signed-off-by: Will Deacon diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index a0f7976348f8..ad94b7def005 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -128,6 +128,17 @@ static inline __pure struct mcs_spinlock *decode_tail(u32 tail) #if _Q_PENDING_BITS == 8 /** + * clear_pending - clear the pending bit. + * @lock: Pointer to queued spinlock structure + * + * *,1,* -> *,0,* + */ +static __always_inline void clear_pending(struct qspinlock *lock) +{ + WRITE_ONCE(lock->pending, 0); +} + +/** * clear_pending_set_locked - take ownership and clear the pending bit. * @lock: Pointer to queued spinlock structure * @@ -163,6 +174,17 @@ static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail) #else /* _Q_PENDING_BITS == 8 */ /** + * clear_pending - clear the pending bit. + * @lock: Pointer to queued spinlock structure + * + * *,1,* -> *,0,* + */ +static __always_inline void clear_pending(struct qspinlock *lock) +{ + atomic_andnot(_Q_PENDING_VAL, &lock->val); +} + +/** * clear_pending_set_locked - take ownership and clear the pending bit. * @lock: Pointer to queued spinlock structure * @@ -266,7 +288,7 @@ static __always_inline u32 __pv_wait_head_or_lock(struct qspinlock *lock, void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) { struct mcs_spinlock *prev, *next, *node; - u32 new, old, tail; + u32 old, tail; int idx; BUILD_BUG_ON(CONFIG_NR_CPUS >= (1U << _Q_TAIL_CPU_BITS)); @@ -290,58 +312,50 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) } /* + * If we observe any contention; queue. + */ + if (val & ~_Q_LOCKED_MASK) + goto queue; + + /* * trylock || pending * * 0,0,0 -> 0,0,1 ; trylock * 0,0,1 -> 0,1,1 ; pending */ - for (;;) { + val = atomic_fetch_or_acquire(_Q_PENDING_VAL, &lock->val); + if (!(val & ~_Q_LOCKED_MASK)) { /* - * If we observe any contention; queue. + * we're pending, wait for the owner to go away. + * + * *,1,1 -> *,1,0 + * + * this wait loop must be a load-acquire such that we match the + * store-release that clears the locked bit and create lock + * sequentiality; this is because not all + * clear_pending_set_locked() implementations imply full + * barriers. */ - if (val & ~_Q_LOCKED_MASK) - goto queue; - - new = _Q_LOCKED_VAL; - if (val == new) - new |= _Q_PENDING_VAL; + if (val & _Q_LOCKED_MASK) { + smp_cond_load_acquire(&lock->val.counter, + !(VAL & _Q_LOCKED_MASK)); + } /* - * Acquire semantic is required here as the function may - * return immediately if the lock was free. + * take ownership and clear the pending bit. + * + * *,1,0 -> *,0,1 */ - old = atomic_cmpxchg_acquire(&lock->val, val, new); - if (old == val) - break; - - val = old; - } - - /* - * we won the trylock - */ - if (new == _Q_LOCKED_VAL) + clear_pending_set_locked(lock); return; + } /* - * we're pending, wait for the owner to go away. - * - * *,1,1 -> *,1,0 - * - * this wait loop must be a load-acquire such that we match the - * store-release that clears the locked bit and create lock - * sequentiality; this is because not all clear_pending_set_locked() - * implementations imply full barriers. - */ - smp_cond_load_acquire(&lock->val.counter, !(VAL & _Q_LOCKED_MASK)); - - /* - * take ownership and clear the pending bit. - * - * *,1,0 -> *,0,1 + * If pending was clear but there are waiters in the queue, then + * we need to undo our setting of pending before we queue ourselves. */ - clear_pending_set_locked(lock); - return; + if (!(val & _Q_PENDING_MASK)) + clear_pending(lock); /* * End of pending bit optimistic spinning and beginning of MCS @@ -445,15 +459,15 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) * claim the lock: * * n,0,0 -> 0,0,1 : lock, uncontended - * *,0,0 -> *,0,1 : lock, contended + * *,*,0 -> *,*,1 : lock, contended * - * If the queue head is the only one in the queue (lock value == tail), - * clear the tail code and grab the lock. Otherwise, we only need - * to grab the lock. + * If the queue head is the only one in the queue (lock value == tail) + * and nobody is pending, clear the tail code and grab the lock. + * Otherwise, we only need to grab the lock. */ for (;;) { /* In the PV case we might already have _Q_LOCKED_VAL set */ - if ((val & _Q_TAIL_MASK) != tail) { + if ((val & _Q_TAIL_MASK) != tail || (val & _Q_PENDING_MASK)) { set_locked(lock); break; } diff --git a/kernel/locking/qspinlock_paravirt.h b/kernel/locking/qspinlock_paravirt.h index 2711940429f5..2dbad2f25480 100644 --- a/kernel/locking/qspinlock_paravirt.h +++ b/kernel/locking/qspinlock_paravirt.h @@ -118,11 +118,6 @@ static __always_inline void set_pending(struct qspinlock *lock) WRITE_ONCE(lock->pending, 1); } -static __always_inline void clear_pending(struct qspinlock *lock) -{ - WRITE_ONCE(lock->pending, 0); -} - /* * The pending bit check in pv_queued_spin_steal_lock() isn't a memory * barrier. Therefore, an atomic cmpxchg_acquire() is used to acquire the From patchwork Thu Apr 26 10:34:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 134480 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp2083627lji; Thu, 26 Apr 2018 03:38:01 -0700 (PDT) X-Google-Smtp-Source: AIpwx4+Nd2NIb/L0ViMkCk9tTN60IoNzYxQUdZKrg/nWCldTQVm1+r1RZTn+xEAhY1ZTkpS3dUOc X-Received: by 10.98.214.218 with SMTP id a87mr27459733pfl.200.1524739080931; Thu, 26 Apr 2018 03:38:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524739080; cv=none; d=google.com; s=arc-20160816; b=KIBHtNjUchBpfqLfG9Ra/RS9K0n0Uqdsk0c/fz1Ky8YIgTcfk7chGh0eC+6cm/gz0Y jczNPKhb+gPMxc5vjMENdJgPVZk74bAn2ptMzRzf69Kkj0epaXIODs8yq/7tNSolIViV BJ+N2LH0z47JmXIasvgkrzvxCaEuGBjhBXxw10S1Ej2ZrTfH69SWKD6txicCqerIhTX9 42S0+qWhmoEQtkEGHyB/SV8TroxcfUCcjHMxL9XHNKFLm76a7ByawMJFaa9M+FEwYh5E qe6kGD4DOvH3sl0DDygTCDWgYUCVSbrhI+bkffhgUbNqkVnnENbFtX+wb8a+HyyRVo3n 8qJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=kux5n1VJT9ByJWEJc8PHGsikEDL9iGj2a/IhBM5/xMI=; b=Kiop29NISf0k9bWxxJV0mQNBtxXsSDV0hvk6LrWOMALMStMsmB9/BgWe1mnozTjOCz 4lhuRAdjjGGsDuYkTXT+VQes/43wik6kZCw6ze9pno3cjNgbwQaKxTso6Vgbg4mmmMZp JD1bCbuSJ9mdMzpfNgn8/TPMQOgZJKGO4AwwzIPWkOAa0d/G9FDVGLjLV5/yL7XQtuWa fXD7CWDQGdy5HXbWJC8yC5rj8P8jw5qaaxAjmMHMWLdwt4WSmYww03jeMTw+4wPF4qnG UB8wuJmsweL5pbvPcNV5ekcOxghxHIye96I80hb1qzbLvEjfMeXuBj6F0HLBY5vv+cto gx0A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b2si15721607pgc.275.2018.04.26.03.38.00; Thu, 26 Apr 2018 03:38:00 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754903AbeDZKh6 (ORCPT + 29 others); Thu, 26 Apr 2018 06:37:58 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51054 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755017AbeDZKeK (ORCPT ); Thu, 26 Apr 2018 06:34:10 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4BC6F16BA; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 1DE023F5AF; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 393801AE5133; Thu, 26 Apr 2018 11:34:29 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, will.deacon@arm.com Subject: [PATCH v3 06/14] locking/qspinlock: Kill cmpxchg loop when claiming lock from head of queue Date: Thu, 26 Apr 2018 11:34:20 +0100 Message-Id: <1524738868-31318-7-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1524738868-31318-1-git-send-email-will.deacon@arm.com> References: <1524738868-31318-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When a queued locker reaches the head of the queue, it claims the lock by setting _Q_LOCKED_VAL in the lockword. If there isn't contention, it must also clear the tail as part of this operation so that subsequent lockers can avoid taking the slowpath altogether. Currently this is expressed as a cmpxchg loop that practically only runs up to two iterations. This is confusing to the reader and unhelpful to the compiler. Rewrite the cmpxchg loop without the loop, so that a failed cmpxchg implies that there is contention and we just need to write to _Q_LOCKED_VAL without considering the rest of the lockword. Cc: Peter Zijlstra Cc: Ingo Molnar Signed-off-by: Will Deacon --- kernel/locking/qspinlock.c | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index ad94b7def005..e42f50d69ed6 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -465,24 +465,21 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) * and nobody is pending, clear the tail code and grab the lock. * Otherwise, we only need to grab the lock. */ - for (;;) { - /* In the PV case we might already have _Q_LOCKED_VAL set */ - if ((val & _Q_TAIL_MASK) != tail || (val & _Q_PENDING_MASK)) { - set_locked(lock); - break; - } + + /* In the PV case we might already have _Q_LOCKED_VAL set */ + if ((val & _Q_TAIL_MASK) == tail) { /* * The smp_cond_load_acquire() call above has provided the - * necessary acquire semantics required for locking. At most - * two iterations of this loop may be ran. + * necessary acquire semantics required for locking. */ old = atomic_cmpxchg_relaxed(&lock->val, val, _Q_LOCKED_VAL); if (old == val) - goto release; /* No contention */ - - val = old; + goto release; /* No contention */ } + /* Either somebody is queued behind us or _Q_PENDING_VAL is set */ + set_locked(lock); + /* * contended path; wait for next if not observed yet, release. */ From patchwork Thu Apr 26 10:34:21 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 134477 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp2080745lji; Thu, 26 Apr 2018 03:35:47 -0700 (PDT) X-Google-Smtp-Source: AIpwx49V+WWinZgo5BFPky4lZruosQ3je+omfEvcET1otPC14OYH7NRC7fCyez9Ma7mVYSQRDRLJ X-Received: by 10.98.33.28 with SMTP id h28mr31006508pfh.249.1524738947677; Thu, 26 Apr 2018 03:35:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524738947; cv=none; d=google.com; s=arc-20160816; b=J4Vulr9JdQnSfntv03hXi4ehnDjctt6jJMNUzV/Zj2Paz0WmKoYYZvwTsZuGaeRbh2 fFDy80qYh02BYwiUdkT0ghRjXyeFRWszMZxulyycrFQ9Wm/t5RMhIr+M4joVbW7o/Y1C qa0duVCRWciRcPq9vt97kOWhPn9qLSqmGXlj5eVqALirfPiWdRdcXPypn1Yqo9f/wP10 lN5tuubNnTaJ1FXBaFYpAFsiMF1EbmgfGoUCcjeqdPiN4qX4h7ZsVPQBY9T4kc8VYHWS c49/J6vyXpgxDgTHSvgDzDmnjvOgrcunmAXGckoY74/PFcSjnlphmjOhDvkjrMW/URiu XnJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=m/V2x5qi+RwKCLq34kHqvJu5qr8/9ix8BbKKox06C48=; b=FDqwN043sXJkkJlEt7AISK+6csUio70Ojd6xyVgoBptIDlPqcB+RN/KzuyHDYwEdLV 88OqdYBXm5QzF5XzZt6R1UgWrlbcYbHwZL8Zhbleq95jx1TGHNFTiH+N80gA3BEI19Rw kErK+wDhBRJQEl6MgAq7YQL3v6BVXWIqwzN5prVZFmSh+z7vRMamc22irN1riI9T8dIe +k6uLizIL6u0iopy6FLCaroHWoy4OQo1CeDjXkoHOITx/Z7DLfx7RiHJ6EDXw1b2dkVn ixu3uKgAoUqyM8Gz5WUHbE9mQG3DHCofzZz4yQaGoCF8R0xeywSFmZISIBWdD1PHCANT HSzQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id z188si15700603pgb.134.2018.04.26.03.35.47; Thu, 26 Apr 2018 03:35:47 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755810AbeDZKfp (ORCPT + 29 others); Thu, 26 Apr 2018 06:35:45 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51078 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755108AbeDZKeK (ORCPT ); Thu, 26 Apr 2018 06:34:10 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 56FF416EA; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 26DDD3F5B7; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 4F9DF1AE5152; Thu, 26 Apr 2018 11:34:29 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, will.deacon@arm.com Subject: [PATCH v3 07/14] locking/qspinlock: Use atomic_cond_read_acquire Date: Thu, 26 Apr 2018 11:34:21 +0100 Message-Id: <1524738868-31318-8-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1524738868-31318-1-git-send-email-will.deacon@arm.com> References: <1524738868-31318-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Rather than dig into the counter field of the atomic_t inside the qspinlock structure so that we can call smp_cond_load_acquire, use atomic_cond_read_acquire instead, which operates on the atomic_t directly. Cc: Peter Zijlstra Cc: Ingo Molnar Signed-off-by: Will Deacon --- kernel/locking/qspinlock.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index e42f50d69ed6..5e383cfe4cce 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -337,8 +337,8 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) * barriers. */ if (val & _Q_LOCKED_MASK) { - smp_cond_load_acquire(&lock->val.counter, - !(VAL & _Q_LOCKED_MASK)); + atomic_cond_read_acquire(&lock->val, + !(VAL & _Q_LOCKED_MASK)); } /* @@ -441,8 +441,8 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) * * The PV pv_wait_head_or_lock function, if active, will acquire * the lock and return a non-zero value. So we have to skip the - * smp_cond_load_acquire() call. As the next PV queue head hasn't been - * designated yet, there is no way for the locked value to become + * atomic_cond_read_acquire() call. As the next PV queue head hasn't + * been designated yet, there is no way for the locked value to become * _Q_SLOW_VAL. So both the set_locked() and the * atomic_cmpxchg_relaxed() calls will be safe. * @@ -452,7 +452,7 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) if ((val = pv_wait_head_or_lock(lock, node))) goto locked; - val = smp_cond_load_acquire(&lock->val.counter, !(VAL & _Q_LOCKED_PENDING_MASK)); + val = atomic_cond_read_acquire(&lock->val, !(VAL & _Q_LOCKED_PENDING_MASK)); locked: /* @@ -469,7 +469,7 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) /* In the PV case we might already have _Q_LOCKED_VAL set */ if ((val & _Q_TAIL_MASK) == tail) { /* - * The smp_cond_load_acquire() call above has provided the + * The atomic_cond_read_acquire() call above has provided the * necessary acquire semantics required for locking. */ old = atomic_cmpxchg_relaxed(&lock->val, val, _Q_LOCKED_VAL); From patchwork Thu Apr 26 10:34:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 134476 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp2080654lji; Thu, 26 Apr 2018 03:35:44 -0700 (PDT) X-Google-Smtp-Source: AB8JxZoS4j8A+aSL1xvAe1qdiFdgX+MfEk5MoHv5R9QXQpg9XX40BB3+T4H1AiGckuS3oyrQMVB9 X-Received: by 2002:a17:902:5902:: with SMTP id o2-v6mr10910009pli.79.1524738943916; Thu, 26 Apr 2018 03:35:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524738943; cv=none; d=google.com; s=arc-20160816; b=bwiUnUhoJNF7frQIxMcK7lU6bMbAL8o0n6RCHCtdPo3ITQl5eQiGNEy+tm7GK4eLVs b/3LHJBYsqIU+TGwqIv+887qYtgBV9yBsMmWVbJe5JrI1IFw5eqUTOmTmjak0XbCFuOJ IIoyJbIAY9fRLq54gwlGwozdSoBAVaStmlQyjc4OGdGSv9MKcDmRxbFuIMs9/pAzdOKG ghP/kwvAwcv7Ubbddy3GeoOeSceTcpLcKI2PQr66IwLewhLqBRscW/XMNf7Gfc6xY7rN pu6gy2it4lbwLDIAiQq5eJGqsMDAb4EaPYTpZiNYppvhMvIJFxk2u2w0ZalvlumkE/ZX Z6rA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=irkULNHmtNPgNAdekth1s68Hkl75EPNE3ReOorX3J4o=; b=gxykXyRAdJxCsl4yvcGC8y+xQ3djyTV3PlHcvb9BjwIdOe7zAcOkH58WXCi3bFL3y6 J3j3Jf6IiTPz3FJwBGgEpMOP10njEiIrZw23gST5aAWCE9exJXEcosvHIZxpEPSr/9Y5 SWvJ6RpTkLobHpTTl8kwYZsPT4XPHOtfhv3lsbPxh5ijo+MbTMMKNEA4eTvfTDk6iKnT 9dQg/ETFvrZS4ag6MGMIr0LRyTjZ1xb2KVkTnW3sgwVLulEv83I5PCK/EGOEv5oAs+eH 1dY7fZbWihSywyJK3/JC0AvTjnODvpH7DhUW0dpIuEl9S+ubvRw4r5VG5wkUIpRhCvvc EedA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id z188si15700603pgb.134.2018.04.26.03.35.43; Thu, 26 Apr 2018 03:35:43 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755780AbeDZKfk (ORCPT + 29 others); Thu, 26 Apr 2018 06:35:40 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51080 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755074AbeDZKeK (ORCPT ); Thu, 26 Apr 2018 06:34:10 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5E03A1713; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 3074E3F77A; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 5FE561AE515F; Thu, 26 Apr 2018 11:34:29 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, will.deacon@arm.com Subject: [PATCH v3 08/14] locking/mcs: Use smp_cond_load_acquire() in mcs spin loop Date: Thu, 26 Apr 2018 11:34:22 +0100 Message-Id: <1524738868-31318-9-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1524738868-31318-1-git-send-email-will.deacon@arm.com> References: <1524738868-31318-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Jason Low For qspinlocks on ARM64, we would like to use WFE instead of purely spinning. Qspinlocks internally have lock contenders spin on an MCS lock. Update arch_mcs_spin_lock_contended() such that it uses the new smp_cond_load_acquire() so that ARM64 can also override this spin loop with its own implementation using WFE. On x86, this can also be cheaper than spinning on smp_load_acquire(). Signed-off-by: Jason Low Signed-off-by: Will Deacon --- kernel/locking/mcs_spinlock.h | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) -- 2.1.4 diff --git a/kernel/locking/mcs_spinlock.h b/kernel/locking/mcs_spinlock.h index f046b7ce9dd6..5e10153b4d3c 100644 --- a/kernel/locking/mcs_spinlock.h +++ b/kernel/locking/mcs_spinlock.h @@ -23,13 +23,15 @@ struct mcs_spinlock { #ifndef arch_mcs_spin_lock_contended /* - * Using smp_load_acquire() provides a memory barrier that ensures - * subsequent operations happen after the lock is acquired. + * Using smp_cond_load_acquire() provides the acquire semantics + * required so that subsequent operations happen after the + * lock is acquired. Additionally, some architectures such as + * ARM64 would like to do spin-waiting instead of purely + * spinning, and smp_cond_load_acquire() provides that behavior. */ #define arch_mcs_spin_lock_contended(l) \ do { \ - while (!(smp_load_acquire(l))) \ - cpu_relax(); \ + smp_cond_load_acquire(l, VAL); \ } while (0) #endif From patchwork Thu Apr 26 10:34:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 134475 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp2080581lji; Thu, 26 Apr 2018 03:35:40 -0700 (PDT) X-Google-Smtp-Source: AIpwx4/jgnd0K1pvtO2YTEIkl3nR5JC/hC2+r954DbldB4qCLZpt17B36/oRdx84DeVBw3+NLdAO X-Received: by 2002:a17:902:5602:: with SMTP id h2-v6mr30607970pli.115.1524738940236; Thu, 26 Apr 2018 03:35:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524738940; cv=none; d=google.com; s=arc-20160816; b=XCGP+vNCMRfYMOg0V4oJhJTvfgwu/t7oIPsMJEtwfNaMIA47Q/0eKKPHr1h3rJORH2 wI3QJfzYEhp4aJ5ZdBRQCHbeGTdfFcDA5a4+gc1toRNxbRIJZ2QNvJRginOFvnW4bDsC l8jSE77hykxQN5+UWNZD1OD0fVUc0+I0eUwVzfOlB5vdS6/Fff+j+hnb6nuWqcuURWrq fnbxoNiXhFgu6mj6og2IDOPXVYQ2H/6smMN85wT0SdsK/0Noy8NGajvXyBvWhekc1cdh BY1YwcsWOBgu2H0rxcTxsnFZBctIjT/TtdkawlOrMETZsIdwBHvl0Fl4USGRq9XxWIQh bO0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=4MQhOO1+AA8rmTSqhaSD7GxCnAmqykwMxorLeKJlh9g=; b=zKfR1XdrtdmDuPOPZ9gCcWmZcAGW5OcAUjam67xNFAa77aHF2wHcDS5aw4uyJufcQe xyiZfBI9SQ6Sb8zjtyDcfQzj4XPFcsSRAkjCjryRHIXpiNQlpA3Hu/APQFRA8J3kScXh gzF9f6ymkrdWi2jGogIOl5oGFAhMkGuQXcdCjN76DP+9uLz4wbvntKlTh+zijk/n05jS gbj3/5GDUwBFur1QsYwyz/A0JmTx1Hq5Q+qXzOy5jC8Tk54hteA8ORjSnaYvNeNf2v7X QjigP59U8vyaDax7IuzylBkQmmI1tczRmwDJj8xlltFYgqtNEi0uMV8OXpc5S/10L7IQ H3Ww== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id v6si460746pgs.399.2018.04.26.03.35.39; Thu, 26 Apr 2018 03:35:40 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755755AbeDZKfg (ORCPT + 29 others); Thu, 26 Apr 2018 06:35:36 -0400 Received: from foss.arm.com ([217.140.101.70]:51126 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755164AbeDZKeL (ORCPT ); Thu, 26 Apr 2018 06:34:11 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6D9F8174E; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 3FCEC3F7BE; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 7C6201AE516A; Thu, 26 Apr 2018 11:34:29 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, will.deacon@arm.com Subject: [PATCH v3 09/14] locking/qspinlock: Use smp_cond_load_relaxed to wait for next node Date: Thu, 26 Apr 2018 11:34:23 +0100 Message-Id: <1524738868-31318-10-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1524738868-31318-1-git-send-email-will.deacon@arm.com> References: <1524738868-31318-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When a locker reaches the head of the queue and takes the lock, a concurrent locker may enqueue and force the lock holder to spin whilst its node->next field is initialised. Rather than open-code a READ_ONCE/cpu_relax() loop, this can be implemented using smp_cond_load_relaxed instead. Cc: Peter Zijlstra Cc: Ingo Molnar Signed-off-by: Will Deacon --- kernel/locking/qspinlock.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index 5e383cfe4cce..7b8c81ebb15e 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -483,10 +483,8 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) /* * contended path; wait for next if not observed yet, release. */ - if (!next) { - while (!(next = READ_ONCE(node->next))) - cpu_relax(); - } + if (!next) + next = smp_cond_load_relaxed(&node->next, (VAL)); arch_mcs_spin_unlock_contended(&next->locked); pv_kick_node(lock, next); From patchwork Thu Apr 26 10:34:24 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 134474 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp2080553lji; Thu, 26 Apr 2018 03:35:39 -0700 (PDT) X-Google-Smtp-Source: AIpwx48DpS3BUZVgFIyh1/J/2DzTpyL5LqxPI8UtPdI12HgI8CXoDotpy+MqU9UV//iiaShibIW5 X-Received: by 2002:a17:902:6f16:: with SMTP id w22-v6mr32748923plk.216.1524738939084; Thu, 26 Apr 2018 03:35:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524738939; cv=none; d=google.com; s=arc-20160816; b=CqOBgiUy9Ao5e5WAs8p1ZLclZ5kzaK5huBNusZFRuOhjpji9l6rRpBsrp/yT+R20nl AQtgFuotK73I/j/4qx1mWOwvoNp579UMhVcNkpACUaooQYrm5G4/tgMz7F56L4o8PIwc tt5Ue+vyTiCz5MehED/TIJr6rW15EUqhuVg0BI+9cXbPMDE65FwHaMJ5r9HhtqJP79hS qQSk+uhUZFiDWueU+demRXk9BiOSSYORvnY4zNybD2MhCnaV9a00ON4ZI4CW149f56Ch pmCbxPsPE1m8l7rKpUSjb9v+99tBzx2KirhODD7x+a+K8DEgm7fc9rJtbdWB+kw1lQUI HDcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=wkblKAmBxnaQe+cTwroykfXYTv1d1hqWOBC1dh053Dg=; b=YpmCUlDV7bOY2+5276l9vwC4QE04wEkrfH9lBOfrYZmznlTy9c4APO9iqINE6u06jB Vu0opcQscBrimaCh/+my+AgIWp5ljGECPRUL+SAXkhHY/aTiCkzN6dVO39b44D6mJhPe IrDfU6rvCou/9KzXw4WvBEBgudxf0K8/H56UGbYVqkImBzgZqO2HbfhnHZbAc30DfCvF MwyreC/1ibnnjHceNXiatqmWk4p0ig8rdAm1ZoZNY7VXiWbm/WFYOsJ/Hbzu4twVDBuN THQnMIKDU1o1chuhwIVcgRvmjPDS/hUuxR2aY+2/gkqQmjSk2IfrSE6DcyG94620sn4p O/qA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b62si10585531pgc.505.2018.04.26.03.35.38; Thu, 26 Apr 2018 03:35:39 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755725AbeDZKfc (ORCPT + 29 others); Thu, 26 Apr 2018 06:35:32 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51124 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755172AbeDZKeL (ORCPT ); Thu, 26 Apr 2018 06:34:11 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 770F51993; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 48F203F487; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 90BAE1AE5194; Thu, 26 Apr 2018 11:34:29 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, will.deacon@arm.com Subject: [PATCH v3 10/14] locking/qspinlock: Make queued_spin_unlock use smp_store_release Date: Thu, 26 Apr 2018 11:34:24 +0100 Message-Id: <1524738868-31318-11-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1524738868-31318-1-git-send-email-will.deacon@arm.com> References: <1524738868-31318-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org A qspinlock can be unlocked simply by writing zero to the locked byte. This can be implemented in the generic code, so do that and remove the arch-specific override for x86 in the !PV case. Cc: Peter Zijlstra Cc: Ingo Molnar Signed-off-by: Will Deacon --- arch/x86/include/asm/qspinlock.h | 17 ++++++----------- include/asm-generic/qspinlock.h | 2 +- 2 files changed, 7 insertions(+), 12 deletions(-) -- 2.1.4 diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h index da1370ad206d..3e70bed8a978 100644 --- a/arch/x86/include/asm/qspinlock.h +++ b/arch/x86/include/asm/qspinlock.h @@ -9,6 +9,12 @@ #define _Q_PENDING_LOOPS (1 << 9) +#ifdef CONFIG_PARAVIRT_SPINLOCKS +extern void native_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); +extern void __pv_init_lock_hash(void); +extern void __pv_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); +extern void __raw_callee_save___pv_queued_spin_unlock(struct qspinlock *lock); + #define queued_spin_unlock queued_spin_unlock /** * queued_spin_unlock - release a queued spinlock @@ -21,12 +27,6 @@ static inline void native_queued_spin_unlock(struct qspinlock *lock) smp_store_release(&lock->locked, 0); } -#ifdef CONFIG_PARAVIRT_SPINLOCKS -extern void native_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); -extern void __pv_init_lock_hash(void); -extern void __pv_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); -extern void __raw_callee_save___pv_queued_spin_unlock(struct qspinlock *lock); - static inline void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) { pv_queued_spin_lock_slowpath(lock, val); @@ -42,11 +42,6 @@ static inline bool vcpu_is_preempted(long cpu) { return pv_vcpu_is_preempted(cpu); } -#else -static inline void queued_spin_unlock(struct qspinlock *lock) -{ - native_queued_spin_unlock(lock); -} #endif #ifdef CONFIG_PARAVIRT diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinlock.h index b37b4ad7eb94..a8ed0a352d75 100644 --- a/include/asm-generic/qspinlock.h +++ b/include/asm-generic/qspinlock.h @@ -100,7 +100,7 @@ static __always_inline void queued_spin_unlock(struct qspinlock *lock) /* * unlock() needs release semantics: */ - (void)atomic_sub_return_release(_Q_LOCKED_VAL, &lock->val); + smp_store_release(&lock->locked, 0); } #endif From patchwork Thu Apr 26 10:34:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 134472 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp2080358lji; Thu, 26 Apr 2018 03:35:28 -0700 (PDT) X-Google-Smtp-Source: AIpwx4+aoEPS60VkNQmRXPEz/vD1LmmhPypHDImLSB5Kbfut6kpTvkajzbqg0bG1zavXgKlYSNxh X-Received: by 2002:a17:902:bf41:: with SMTP id u1-v6mr29082719pls.257.1524738928568; Thu, 26 Apr 2018 03:35:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524738928; cv=none; d=google.com; s=arc-20160816; b=n/oWBoH2jX6ziBl5QoTxr1KLJ2Dpl6uxgeBxnpe+mDFAVRR6gZrFcQK/ya2AOaN9Gj Lhfb9FudH7x2izoxCgsuIX59bYkMJIqgxmzi5sZGRN1eZcPyr9veD98R0Cfe+mvChVEG adV4rzPRo1S9ss787LP49mKAj02g9pM6GtcqTvqMfc7OGJO6PybyHv24bwtrDYKHPyfk ObqbFjy0JsauRAPGMeWrUNjwM86XNq4O30A7YzdfWMSHQ4EjW6Gav85NtcSEK/z/oYJo kMNlupOsYmLKPf8LCXiofPa4qIdO/tWVnL0dS1Mc4WktHlARtd066LkPVXYGJycxC8EM COIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=2tA+Zl24djFBobDnFPgPbgE1iwOO4fg0RFP1P3nWloM=; b=V1kJt/MFTvuMbonxHvZvxNTWtVGpChMMvF8t6YaZB0TkhccjaZVsSujJCm9JqLIMgT EaSUPflM71uyZJMMa6LqE+2HLE+1bDAp2qsuRs7Fn45vrf9F9ZDbbsJzaz4t3uyXuLui 0pt40WGrRN3Z6AGqO4tDbNvV4fOQBmBDuh4946IvrXsWGcgkob0EIQcsEuqP1Ck3HcMp wrijKMHzFAKqmwItxgIc+Uq2uXOrPHA/E++H1H5dbPh/gmhhEGAwZyvaqYxuNtp3qI+9 EHIo3+OlioKA/srWdl2ve5oxi47o+TpZzcsRqWbas24hOL4WbXUnAyy6eeAbEc2415IJ 1WwA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f9si12380800pge.639.2018.04.26.03.35.28; Thu, 26 Apr 2018 03:35:28 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755663AbeDZKf0 (ORCPT + 29 others); Thu, 26 Apr 2018 06:35:26 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51150 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755235AbeDZKeL (ORCPT ); Thu, 26 Apr 2018 06:34:11 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7FE1819CC; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 51F8B3F5AF; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id A92981AE519F; Thu, 26 Apr 2018 11:34:29 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, will.deacon@arm.com Subject: [PATCH v3 11/14] locking/qspinlock: Elide back-to-back RELEASE operations with smp_wmb() Date: Thu, 26 Apr 2018 11:34:25 +0100 Message-Id: <1524738868-31318-12-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1524738868-31318-1-git-send-email-will.deacon@arm.com> References: <1524738868-31318-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The qspinlock slowpath must ensure that the MCS node is fully initialised before it can be reached by another other CPU. This is currently enforced by using a RELEASE operation when updating the tail and also when linking the node into the waitqueue (since the control dependency off xchg_tail is insufficient to enforce sufficient ordering -- see 95bcade33a8a ("locking/qspinlock: Ensure node is initialised before updating prev->next")). Back-to-back RELEASE operations may be expensive on some architectures, particularly those that implement them using fences under the hood. We can replace the two RELEASE operations with a single smp_wmb() fence and use RELAXED operations for the subsequent publishing of the node. Cc: Peter Zijlstra Cc: Ingo Molnar Signed-off-by: Will Deacon --- kernel/locking/qspinlock.c | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index 7b8c81ebb15e..fa5d2ab369f9 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -164,10 +164,10 @@ static __always_inline void clear_pending_set_locked(struct qspinlock *lock) static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail) { /* - * Use release semantics to make sure that the MCS node is properly - * initialized before changing the tail code. + * We can use relaxed semantics since the caller ensures that the + * MCS node is properly initialized before updating the tail. */ - return (u32)xchg_release(&lock->tail, + return (u32)xchg_relaxed(&lock->tail, tail >> _Q_TAIL_OFFSET) << _Q_TAIL_OFFSET; } @@ -212,10 +212,11 @@ static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail) for (;;) { new = (val & _Q_LOCKED_PENDING_MASK) | tail; /* - * Use release semantics to make sure that the MCS node is - * properly initialized before changing the tail code. + * We can use relaxed semantics since the caller ensures that + * the MCS node is properly initialized before updating the + * tail. */ - old = atomic_cmpxchg_release(&lock->val, val, new); + old = atomic_cmpxchg_relaxed(&lock->val, val, new); if (old == val) break; @@ -388,12 +389,18 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) goto release; /* + * Ensure that the initialisation of @node is complete before we + * publish the updated tail via xchg_tail() and potentially link + * @node into the waitqueue via WRITE_ONCE(prev->next, node) below. + */ + smp_wmb(); + + /* + * Publish the updated tail. * We have already touched the queueing cacheline; don't bother with * pending stuff. * * p,*,* -> n,*,* - * - * RELEASE, such that the stores to @node must be complete. */ old = xchg_tail(lock, tail); next = NULL; @@ -405,14 +412,8 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) if (old & _Q_TAIL_MASK) { prev = decode_tail(old); - /* - * We must ensure that the stores to @node are observed before - * the write to prev->next. The address dependency from - * xchg_tail is not sufficient to ensure this because the read - * component of xchg_tail is unordered with respect to the - * initialisation of @node. - */ - smp_store_release(&prev->next, node); + /* Link @node into the waitqueue. */ + WRITE_ONCE(prev->next, node); pv_wait_node(node, prev); arch_mcs_spin_lock_contended(&node->locked); From patchwork Thu Apr 26 10:34:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 134473 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp2080408lji; Thu, 26 Apr 2018 03:35:31 -0700 (PDT) X-Google-Smtp-Source: AIpwx4/uf7R0Tk+Qnvru6riYY/bVRC30zsP9nuIf2z1Gr05d8KXgqd1Qu+VxPHsKgIiP4Flri1oH X-Received: by 2002:a17:902:9a0c:: with SMTP id v12-v6mr32459367plp.162.1524738931604; Thu, 26 Apr 2018 03:35:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524738931; cv=none; d=google.com; s=arc-20160816; b=vVWdXUXrFUQzuS8lNxqYZ4IhRHtq6DTG9e47hqBRqKdTkxLOIx7DIjU7H2dRIz0Bjf r2JSUKdSBbDvoo0GqE8MbxlbLNlUNbM3MLAuLRteiZvYuJIgHfrHsiG/FciR9InbEd3c ouwhFianHI3q79FzuyxAYey+mwa9PFpfx5wxeSDKGxJx8DrGCIfzauofK1s4/b0vVTFN /qD2r10Uzd9i3hE1mRyWjSdPElkXHxsiUyi2j4ZyStGgJyzhiTazwWJPJsAiRbjcJP0L fS3Kgi/vmxJWiZtyrzD15TK1Zc2qin4/CjX3eVdBSi0rJAob6qdytFGqlBKtBFNUVSih e2bg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=Gd79BK1RjZEjjy2T6jUzYMNwBJjAvxevnUzM+pM9VQo=; b=tAbY84MA0I7h1gw7i4uMtieGZwInxPGWg8Tv4Se6hbRPJ4+y4JOPlsPbWa9Qu00jEy b32GJ5hKNdeu8vs43GbDflRzz81ZV84vhs7c8CqaFtTfc0eDa6cLFZUIZIN146KUQ8ky 2R2tm0SjbRdB+FpqcP0hshPvMNaJGQ8z6BEp7aOdEUyo7yNxECMH3bb3UJu8AU3GNUK+ ysNhNLli7I1dJrJhdssfZAt5WfcVIXCcBQfZEA7awB8ERa7SSjhBm+zsBKQ6HhEqXVVs tFAz1YFMMJd6mJNNDIAho25PA2sTvkSD1M4CJgQbgPlTm4g1cz/gYl/VS7f4uP35dsBY h3ew== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f9si12380800pge.639.2018.04.26.03.35.31; Thu, 26 Apr 2018 03:35:31 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755694AbeDZKf2 (ORCPT + 29 others); Thu, 26 Apr 2018 06:35:28 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51148 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755227AbeDZKeL (ORCPT ); Thu, 26 Apr 2018 06:34:11 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 890E919F6; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 5AF143F5B7; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id BD8361AE51DF; Thu, 26 Apr 2018 11:34:29 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, will.deacon@arm.com Subject: [PATCH v3 12/14] locking/qspinlock: Use try_cmpxchg instead of cmpxchg when locking Date: Thu, 26 Apr 2018 11:34:26 +0100 Message-Id: <1524738868-31318-13-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1524738868-31318-1-git-send-email-will.deacon@arm.com> References: <1524738868-31318-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When reaching the head of an uncontended queue on the qspinlock slow-path, using a try_cmpxchg instead of a cmpxchg operation to transition the lock work to _Q_LOCKED_VAL generates slightly better code for x86 and pretty much identical code for arm64. Cc: Ingo Molnar Reported-by: Peter Zijlstra Signed-off-by: Will Deacon --- kernel/locking/qspinlock.c | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index fa5d2ab369f9..1e3ddc42135e 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -467,16 +467,15 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) * Otherwise, we only need to grab the lock. */ - /* In the PV case we might already have _Q_LOCKED_VAL set */ - if ((val & _Q_TAIL_MASK) == tail) { - /* - * The atomic_cond_read_acquire() call above has provided the - * necessary acquire semantics required for locking. - */ - old = atomic_cmpxchg_relaxed(&lock->val, val, _Q_LOCKED_VAL); - if (old == val) - goto release; /* No contention */ - } + /* + * In the PV case we might already have _Q_LOCKED_VAL set. + * + * The atomic_cond_read_acquire() call above has provided the + * necessary acquire semantics required for locking. + */ + if (((val & _Q_TAIL_MASK) == tail) && + atomic_try_cmpxchg_relaxed(&lock->val, &val, _Q_LOCKED_VAL)) + goto release; /* No contention */ /* Either somebody is queued behind us or _Q_PENDING_VAL is set */ set_locked(lock); From patchwork Thu Apr 26 10:34:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 134470 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp2079700lji; Thu, 26 Apr 2018 03:34:58 -0700 (PDT) X-Google-Smtp-Source: AIpwx4+0FTIGcm3SxKCph//1QFV0iOlxUpv7LyWVc8vQCELANgVlOpiml+cc5AdXQH0yCi8oAWZ9 X-Received: by 10.99.127.80 with SMTP id p16mr26593807pgn.79.1524738898204; Thu, 26 Apr 2018 03:34:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524738898; cv=none; d=google.com; s=arc-20160816; b=rjGwxBVNXEfKzxZY8avkj6CBCtJzQE7JLuX6wBgrXbvPB9ezhp/qUwAppLMYiTKxkv 5cjabMpowwZtIAgAojCk5W/UwSsFzXB3gEI66BVOqeqOs+T3BssCKxOPp0CO0A7RakFk N/QR6bdTcd1OLsP1/BqGLQFZrOVCEE5/owiVEIxQ4lsY0gxIgah7yXk/3xKkZZmAPAiP Hc/pOZE3J8AYj8S14/wUMAddQjGgmmHkT2A5kKfeLvNbHVt0/ERZmm4txoAnmkXR3PDh Y6En6d3uajXa4Dr5vx5xB4I/b6K7kKpUiwx2kvzPNdL0zV59vLjlKiOTGtMtCzsQxZXc LzAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=Fo+tpocf6rtszTk920yWZ/brvMN099xxiJ87YT1ESVI=; b=uJVks6R/uXNgEX5NEhI5zzyZ4Zc0qGH5vzeLc4RYe7fIK9x0F91Mt4pgmA/jUBn21x zK3KsPYxyH1TYpW+1TiIAvxSYNCRT4sqautxB5YQSaVkOvL2GH7FYQsT+6pDo1ZymB+j yGbRp8X0A84xQ+hxD43f7fF9d3jNGhjSNNvJSYfEODzt7U2DwY6hIQFa9uXptkKccKxw LspI76JbrYsp8eMi07JEJfaVuEK4QCsqw4HLTgndSTXlYBfdfbAb3ACATPNgwQGGY2jb mHxh1gwuG9/mv81h4KqVGGU8zqUiOs2b3B9HX3ghQro0Kq4NeW5c6g5OaIYKiiy7Kj6y sw7w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j20-v6si2831688pll.223.2018.04.26.03.34.57; Thu, 26 Apr 2018 03:34:58 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755505AbeDZKez (ORCPT + 29 others); Thu, 26 Apr 2018 06:34:55 -0400 Received: from foss.arm.com ([217.140.101.70]:51152 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755379AbeDZKeM (ORCPT ); Thu, 26 Apr 2018 06:34:12 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 929981A09; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 63F0A3F77A; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id D7F681AE4F9F; Thu, 26 Apr 2018 11:34:29 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, will.deacon@arm.com Subject: [PATCH v3 13/14] locking/qspinlock: Add stat tracking for pending vs slowpath Date: Thu, 26 Apr 2018 11:34:27 +0100 Message-Id: <1524738868-31318-14-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1524738868-31318-1-git-send-email-will.deacon@arm.com> References: <1524738868-31318-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Waiman Long Currently, the qspinlock_stat code tracks only statistical counts in the PV qspinlock code. However, it may also be useful to track the number of locking operations done via the pending code vs. the MCS lock queue slowpath for the non-PV case. The qspinlock stat code is modified to do that. The stat counter pv_lock_slowpath is renamed to lock_slowpath so that it can be used by both the PV and non-PV cases. Signed-off-by: Waiman Long --- kernel/locking/qspinlock.c | 14 +++++++++++--- kernel/locking/qspinlock_paravirt.h | 7 +------ kernel/locking/qspinlock_stat.h | 9 ++++++--- 3 files changed, 18 insertions(+), 12 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index 1e3ddc42135e..b39f341d831b 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -12,11 +12,11 @@ * GNU General Public License for more details. * * (C) Copyright 2013-2015 Hewlett-Packard Development Company, L.P. - * (C) Copyright 2013-2014 Red Hat, Inc. + * (C) Copyright 2013-2014,2018 Red Hat, Inc. * (C) Copyright 2015 Intel Corp. * (C) Copyright 2015 Hewlett-Packard Enterprise Development LP * - * Authors: Waiman Long + * Authors: Waiman Long * Peter Zijlstra */ @@ -33,6 +33,11 @@ #include /* + * Include queued spinlock statistics code + */ +#include "qspinlock_stat.h" + +/* * The basic principle of a queue-based spinlock can best be understood * by studying a classic queue-based spinlock implementation called the * MCS lock. The paper below provides a good description for this kind @@ -295,7 +300,7 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) BUILD_BUG_ON(CONFIG_NR_CPUS >= (1U << _Q_TAIL_CPU_BITS)); if (pv_enabled()) - goto queue; + goto pv_queue; if (virt_spin_lock(lock)) return; @@ -348,6 +353,7 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) * *,1,0 -> *,0,1 */ clear_pending_set_locked(lock); + qstat_inc(qstat_lock_pending, true); return; } @@ -363,6 +369,8 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) * queuing. */ queue: + qstat_inc(qstat_lock_slowpath, true); +pv_queue: node = this_cpu_ptr(&mcs_nodes[0]); idx = node->count++; tail = encode_tail(smp_processor_id(), idx); diff --git a/kernel/locking/qspinlock_paravirt.h b/kernel/locking/qspinlock_paravirt.h index 2dbad2f25480..25730b2ac022 100644 --- a/kernel/locking/qspinlock_paravirt.h +++ b/kernel/locking/qspinlock_paravirt.h @@ -56,11 +56,6 @@ struct pv_node { }; /* - * Include queued spinlock statistics code - */ -#include "qspinlock_stat.h" - -/* * Hybrid PV queued/unfair lock * * By replacing the regular queued_spin_trylock() with the function below, @@ -428,7 +423,7 @@ pv_wait_head_or_lock(struct qspinlock *lock, struct mcs_spinlock *node) /* * Tracking # of slowpath locking operations */ - qstat_inc(qstat_pv_lock_slowpath, true); + qstat_inc(qstat_lock_slowpath, true); for (;; waitcnt++) { /* diff --git a/kernel/locking/qspinlock_stat.h b/kernel/locking/qspinlock_stat.h index 4a30ef63c607..6bd78c0740fc 100644 --- a/kernel/locking/qspinlock_stat.h +++ b/kernel/locking/qspinlock_stat.h @@ -22,13 +22,14 @@ * pv_kick_wake - # of vCPU kicks used for computing pv_latency_wake * pv_latency_kick - average latency (ns) of vCPU kick operation * pv_latency_wake - average latency (ns) from vCPU kick to wakeup - * pv_lock_slowpath - # of locking operations via the slowpath * pv_lock_stealing - # of lock stealing operations * pv_spurious_wakeup - # of spurious wakeups in non-head vCPUs * pv_wait_again - # of wait's after a queue head vCPU kick * pv_wait_early - # of early vCPU wait's * pv_wait_head - # of vCPU wait's at the queue head * pv_wait_node - # of vCPU wait's at a non-head queue node + * lock_pending - # of locking operations via pending code + * lock_slowpath - # of locking operations via MCS lock queue * * Writing to the "reset_counters" file will reset all the above counter * values. @@ -46,13 +47,14 @@ enum qlock_stats { qstat_pv_kick_wake, qstat_pv_latency_kick, qstat_pv_latency_wake, - qstat_pv_lock_slowpath, qstat_pv_lock_stealing, qstat_pv_spurious_wakeup, qstat_pv_wait_again, qstat_pv_wait_early, qstat_pv_wait_head, qstat_pv_wait_node, + qstat_lock_pending, + qstat_lock_slowpath, qstat_num, /* Total number of statistical counters */ qstat_reset_cnts = qstat_num, }; @@ -73,12 +75,13 @@ static const char * const qstat_names[qstat_num + 1] = { [qstat_pv_spurious_wakeup] = "pv_spurious_wakeup", [qstat_pv_latency_kick] = "pv_latency_kick", [qstat_pv_latency_wake] = "pv_latency_wake", - [qstat_pv_lock_slowpath] = "pv_lock_slowpath", [qstat_pv_lock_stealing] = "pv_lock_stealing", [qstat_pv_wait_again] = "pv_wait_again", [qstat_pv_wait_early] = "pv_wait_early", [qstat_pv_wait_head] = "pv_wait_head", [qstat_pv_wait_node] = "pv_wait_node", + [qstat_lock_pending] = "lock_pending", + [qstat_lock_slowpath] = "lock_slowpath", [qstat_reset_cnts] = "reset_counters", }; From patchwork Thu Apr 26 10:34:28 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 134471 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp2079765lji; Thu, 26 Apr 2018 03:35:01 -0700 (PDT) X-Google-Smtp-Source: AB8JxZqI4atOmzqifQ1OtCfzlGKuQnJNobiiCMbN2Vk2cXx/p90e31imSrJ7PRz++ttVZsJTNCRa X-Received: by 10.99.67.132 with SMTP id q126mr3325627pga.294.1524738900877; Thu, 26 Apr 2018 03:35:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524738900; cv=none; d=google.com; s=arc-20160816; b=AtodXLenRF8gMnDfh3JpDW9G+b8H+h9CK8xRhAhURG0yP77EfPdDiN/jHaF2eKBfPe HkReQ7YegDavuxYrIkoIjMl128SGpb9lmLddI+XdfFOotvgHa3k/6PhCP7nl7HES6bL1 W/xZwFZD8Xar+VUeMYvHiKSQo7mldUp87MydTLtt7Nuyn/fx3Hk7UkTMLZZxiveu88/r 6fY9ZE7SnfG5Tuii9vaykaf5vE2cwQq8c0FALbwTIoTFrlcg2gpwOp5OMuCV8Mhn84SW z+izJq7SaACzKIOWDME3O/TPckQoAf8sz2I0hS6EIyNQ2KVsRB/+7jd6EdmXqdrseEv+ L/cw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=7IzF29N4ryxRV5KI8xnq6m4chRY2JaXK2iBgvNbi2GI=; b=j3VTGAIFeMcsMbqYbmxcf524JonDB4riGhtQUAE19jZ5BugTZJ9ZVyVqwj9gkMPE4M suA8xcHYhsr01FMUrRRGpBVLadqL8DD7wxpOe0eaWXQ23sTTKQLMaqfhJTebz1kadpQS ixQ60QyvxdpG/e2LSuij4z6e7ZTZp3uP4Q+oGxKXJ/CN6lTVDgIiHDDQQ49ZQFPjK2Iv lJVK6w8F50KIhW814mMedi7nO9qhkYAxBiLPBpmEf3Bo1MIMewbe0YFTKz0C+6biqHEG e1V8CQZB5N4Qja7cZfECl+bZQFAIToFB122AN7ED4P+pf3vzLVzEtQ9oz92tsjdb7CgW joNA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j20-v6si2831688pll.223.2018.04.26.03.35.00; Thu, 26 Apr 2018 03:35:00 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755619AbeDZKe6 (ORCPT + 29 others); Thu, 26 Apr 2018 06:34:58 -0400 Received: from foss.arm.com ([217.140.101.70]:51153 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755314AbeDZKeM (ORCPT ); Thu, 26 Apr 2018 06:34:12 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9B0841AD7; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 6D34F3F7C3; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id F26461AE5098; Thu, 26 Apr 2018 11:34:29 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, will.deacon@arm.com Subject: [PATCH v3 14/14] MAINTAINERS: Add myself as a co-maintainer for LOCKING PRIMITIVES Date: Thu, 26 Apr 2018 11:34:28 +0100 Message-Id: <1524738868-31318-15-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1524738868-31318-1-git-send-email-will.deacon@arm.com> References: <1524738868-31318-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org I've been heavily involved with concurrency and memory ordering stuff (see ATOMIC INFRASTRUCTURE and LINUX KERNEL MEMORY CONSISTENCY MODEL) and with arm64 now using qrwlock with a view to using qspinlock in the near future, I'm going to continue being involved with the core locking primitives. Reflect this by adding myself as a co-maintainer alongside Ingo and Peter. Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Will Deacon --- MAINTAINERS | 1 + 1 file changed, 1 insertion(+) -- 2.1.4 diff --git a/MAINTAINERS b/MAINTAINERS index 92be777d060a..2509c20a979e 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -8328,6 +8328,7 @@ F: Documentation/admin-guide/LSM/LoadPin.rst LOCKING PRIMITIVES M: Peter Zijlstra M: Ingo Molnar +M: Will Deacon L: linux-kernel@vger.kernel.org T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git locking/core S: Maintained