From patchwork Wed Apr 11 18:01:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 133162 Delivered-To: patch@linaro.org Received: by 10.46.84.29 with SMTP id i29csp896649ljb; Wed, 11 Apr 2018 11:02:44 -0700 (PDT) X-Google-Smtp-Source: AIpwx4+IQS1zdew2cFVY4+C80dIREXc2ZLgO2n2ePCAM760tz3Hk76hRabfG6VLwGheXbyE5Yrfb X-Received: by 10.101.98.90 with SMTP id q26mr4248438pgv.113.1523469763981; Wed, 11 Apr 2018 11:02:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523469763; cv=none; d=google.com; s=arc-20160816; b=FDp83JPJbCYjjkeupfb5i/nUVK0BltqSjUp2+mRgMmlW3s/O1EdgHbHpPEw504/tVv 8nSb/Xt+oD13uy30nO9oM/SBni4wPPtjToVdNVkK+lsYYCWLaI53RcLVGKWN18C5XgB/ qNGr5wfgSNz9kkUNAk6Sy6s0iHvkMZlWqBgaTzjcLaehiPMYxITfFIHWMbCqyG+dMTDW QYlBs+lrJPHcT64NSBXnAQIf4BfgO50A6A6OmYS4HSEabT/NNtWgXYT33X+TTIi/XpM9 iKqn01TGjzvjukBAD3FCpfpHXWXwxjA6NaIXTWyTchQjlsRIDe31lWTVv3rlADCG/NFx rqfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=sIzpkZ7Pn1Br270bb+bGBwhsobFp3VlAlWO4r+9vlko=; b=r7S8QIv/J38f9rb6oFTTnW0BBOP69J34apSGSXqoepBecHLymjMLC/RcLTXWfXXRUs OmP/8NHqMCaeZEz/JrlfwbBiLGsTBgOmoHCifhfNCc84meH3+tHf3Le/SJQy3JMGsfVH WmboSM+TieHaY3lwTOLAju+UNtN+Kpy/eZRKv2UqRcDRHYYfvZIv8vDCX4mditOChhVn wEw2y0gnzplcX5ZFa+KswyeC05Z/tZf1ip2+ROvctaN7oKLbAOu/tBDL/5eg/cMEypx/ Z7Y0Jn0ik+gyhZfj3ZBSyYTqGDVoqIqhR31snhBYONinWnq+tkgHwW9mzdL/+5Qf0dTs vKfg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id d10si1231452pfn.84.2018.04.11.11.02.43; Wed, 11 Apr 2018 11:02:43 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752901AbeDKSBI (ORCPT + 29 others); Wed, 11 Apr 2018 14:01:08 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:52034 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752512AbeDKSBG (ORCPT ); Wed, 11 Apr 2018 14:01:06 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2186615B2; Wed, 11 Apr 2018 11:01:06 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E60D93F5B1; Wed, 11 Apr 2018 11:01:05 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 5AEF61AE553A; Wed, 11 Apr 2018 19:01:21 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, Will Deacon Subject: [PATCH v2 01/13] barriers: Introduce smp_cond_load_relaxed and atomic_cond_read_relaxed Date: Wed, 11 Apr 2018 19:01:08 +0100 Message-Id: <1523469680-17699-2-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1523469680-17699-1-git-send-email-will.deacon@arm.com> References: <1523469680-17699-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Whilst we currently provide smp_cond_load_acquire and atomic_cond_read_acquire, there are cases where the ACQUIRE semantics are not required because of a subsequent fence or release operation once the conditional loop has exited. This patch adds relaxed versions of the conditional spinning primitives to avoid unnecessary barrier overhead on architectures such as arm64. Signed-off-by: Will Deacon --- include/asm-generic/atomic-long.h | 2 ++ include/asm-generic/barrier.h | 27 +++++++++++++++++++++------ include/linux/atomic.h | 2 ++ 3 files changed, 25 insertions(+), 6 deletions(-) -- 2.1.4 diff --git a/include/asm-generic/atomic-long.h b/include/asm-generic/atomic-long.h index 34a028a7bcc5..5b2b0b5ea06d 100644 --- a/include/asm-generic/atomic-long.h +++ b/include/asm-generic/atomic-long.h @@ -244,6 +244,8 @@ static inline long atomic_long_add_unless(atomic_long_t *l, long a, long u) #define atomic_long_inc_not_zero(l) \ ATOMIC_LONG_PFX(_inc_not_zero)((ATOMIC_LONG_PFX(_t) *)(l)) +#define atomic_long_cond_read_relaxed(v, c) \ + ATOMIC_LONG_PFX(_cond_read_relaxed)((ATOMIC_LONG_PFX(_t) *)(v), (c)) #define atomic_long_cond_read_acquire(v, c) \ ATOMIC_LONG_PFX(_cond_read_acquire)((ATOMIC_LONG_PFX(_t) *)(v), (c)) diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h index fe297b599b0a..305e03b19a26 100644 --- a/include/asm-generic/barrier.h +++ b/include/asm-generic/barrier.h @@ -221,18 +221,17 @@ do { \ #endif /** - * smp_cond_load_acquire() - (Spin) wait for cond with ACQUIRE ordering + * smp_cond_load_relaxed() - (Spin) wait for cond with no ordering guarantees * @ptr: pointer to the variable to wait on * @cond: boolean expression to wait for * - * Equivalent to using smp_load_acquire() on the condition variable but employs - * the control dependency of the wait to reduce the barrier on many platforms. + * Equivalent to using READ_ONCE() on the condition variable. * * Due to C lacking lambda expressions we load the value of *ptr into a * pre-named variable @VAL to be used in @cond. */ -#ifndef smp_cond_load_acquire -#define smp_cond_load_acquire(ptr, cond_expr) ({ \ +#ifndef smp_cond_load_relaxed +#define smp_cond_load_relaxed(ptr, cond_expr) ({ \ typeof(ptr) __PTR = (ptr); \ typeof(*ptr) VAL; \ for (;;) { \ @@ -241,10 +240,26 @@ do { \ break; \ cpu_relax(); \ } \ - smp_acquire__after_ctrl_dep(); \ VAL; \ }) #endif +/** + * smp_cond_load_acquire() - (Spin) wait for cond with ACQUIRE ordering + * @ptr: pointer to the variable to wait on + * @cond: boolean expression to wait for + * + * Equivalent to using smp_load_acquire() on the condition variable but employs + * the control dependency of the wait to reduce the barrier on many platforms. + */ +#ifndef smp_cond_load_acquire +#define smp_cond_load_acquire(ptr, cond_expr) ({ \ + typeof(*ptr) _val; \ + _val = smp_cond_load_relaxed(ptr, cond_expr); \ + smp_acquire__after_ctrl_dep(); \ + _val; \ +}) +#endif + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_GENERIC_BARRIER_H */ diff --git a/include/linux/atomic.h b/include/linux/atomic.h index 8b276fd9a127..01ce3997cb42 100644 --- a/include/linux/atomic.h +++ b/include/linux/atomic.h @@ -654,6 +654,7 @@ static inline int atomic_dec_if_positive(atomic_t *v) } #endif +#define atomic_cond_read_relaxed(v, c) smp_cond_load_relaxed(&(v)->counter, (c)) #define atomic_cond_read_acquire(v, c) smp_cond_load_acquire(&(v)->counter, (c)) #ifdef CONFIG_GENERIC_ATOMIC64 @@ -1075,6 +1076,7 @@ static inline long long atomic64_fetch_andnot_release(long long i, atomic64_t *v } #endif +#define atomic64_cond_read_relaxed(v, c) smp_cond_load_relaxed(&(v)->counter, (c)) #define atomic64_cond_read_acquire(v, c) smp_cond_load_acquire(&(v)->counter, (c)) #include From patchwork Wed Apr 11 18:01:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 133161 Delivered-To: patch@linaro.org Received: by 10.46.84.29 with SMTP id i29csp896493ljb; Wed, 11 Apr 2018 11:02:34 -0700 (PDT) X-Google-Smtp-Source: AIpwx49+zo3J+UjF+4b38+lBqHICXEBC0Fn9AzUdsxAOogaioV1OkLChmsZL7mkvMbfRr5Jv7wmw X-Received: by 2002:a17:902:864c:: with SMTP id y12-v6mr6050462plt.293.1523469754582; Wed, 11 Apr 2018 11:02:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523469754; cv=none; d=google.com; s=arc-20160816; b=LTb9jjwRvneS6xQH7+oOcqXzo5NB4HJsXzPvVpiE8y0y0myRCVNJeoCTwuGl9uHd6T 2QNewt/0vZmRh3dNU77FRfHB/fIpEaNhG90jzXg4pNmWM+JkgiX583NFMqnFGigzXgCQ fugLzS3u22sSFwptzZZ8pDi9pnuBLxpGaqXZoUomuFbx6zb96w5HZY6BbZQMhtfHYBIr ImOajBzAusDnydOcRWk0GA3jhp3U9flpNRGYVhKc6DxLlZvS9w3+naaNvZ/l+p/5H7lS uHHSG0ALHYoGvGx8UnRsdCwC/9lfKiIHedW6Ed7vvnpu8mi+/qj4nwPQlhuBKDfNESi8 DKFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=GziclgBxCVLrqFHp0zydwd1uHOP18mZDQqFYJx+XHeM=; b=LIGSpVToXJqlx8HrxdftZKc7vHg3O7T0GRGAPzpjOi2oatBhlW6Tiata9LijORmfnX 1Oo5f2xmoceEL4VVI6FqFfYMYtFdecOww7cNApXVkXSWlxwVIBBrrFIGAVu7UhgKHPNN n93lgP+26vHDcuA4p/ZHa2P7N23Wr5QW1vPbhLGYvr0H9u8jOad0F6lKFcSYMqeipvsm pHrUk8PQujroI32iV48mQHPyZRa8/xxtxxFABf9cGb+e5IzT2aL4AYCqNoGHSuxjKuBG xOC55Lda2dG0E/ZxxSydRKko8dc2C8h/toFG4QfGe2+jK5Zbskc+O/7KDYxGx9Tz5w5m rzAQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id d10si1231452pfn.84.2018.04.11.11.02.34; Wed, 11 Apr 2018 11:02:34 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752956AbeDKSBJ (ORCPT + 29 others); Wed, 11 Apr 2018 14:01:09 -0400 Received: from foss.arm.com ([217.140.101.70]:52088 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752527AbeDKSBG (ORCPT ); Wed, 11 Apr 2018 14:01:06 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5DC5A164F; Wed, 11 Apr 2018 11:01:06 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id F382E3F614; Wed, 11 Apr 2018 11:01:05 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 683351AE558F; Wed, 11 Apr 2018 19:01:21 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, Will Deacon Subject: [PATCH v2 02/13] locking/qspinlock: Bound spinning on pending->locked transition in slowpath Date: Wed, 11 Apr 2018 19:01:09 +0100 Message-Id: <1523469680-17699-3-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1523469680-17699-1-git-send-email-will.deacon@arm.com> References: <1523469680-17699-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org If a locker taking the qspinlock slowpath reads a lock value indicating that only the pending bit is set, then it will spin whilst the concurrent pending->locked transition takes effect. Unfortunately, there is no guarantee that such a transition will ever be observed since concurrent lockers could continuously set pending and hand over the lock amongst themselves, leading to starvation. Whilst this would probably resolve in practice, it means that it is not possible to prove liveness properties about the lock and means that lock acquisition time is unbounded. Rather than removing the pending->locked spinning from the slowpath altogether (which has been shown to heavily penalise a 2-threaded locking stress test on x86), this patch replaces the explicit spinning with a call to atomic_cond_read_relaxed and allows the architecture to provide a bound on the number of spins. For architectures that can respond to changes in cacheline state in their smp_cond_load implementation, it should be sufficient to use the default bound of 1. Cc: Peter Zijlstra Cc: Ingo Molnar Suggested-by: Waiman Long Signed-off-by: Will Deacon --- kernel/locking/qspinlock.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index d880296245c5..396701e8c62d 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -77,6 +77,18 @@ #endif /* + * The pending bit spinning loop count. + * This heuristic is used to limit the number of lockword accesses + * made by atomic_cond_read_relaxed when waiting for the lock to + * transition out of the "== _Q_PENDING_VAL" state. We don't spin + * indefinitely because there's no guarantee that we'll make forward + * progress. + */ +#ifndef _Q_PENDING_LOOPS +#define _Q_PENDING_LOOPS 1 +#endif + +/* * Per-CPU queue node structures; we can never have more than 4 nested * contexts: task, softirq, hardirq, nmi. * @@ -306,13 +318,15 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) return; /* - * wait for in-progress pending->locked hand-overs + * Wait for in-progress pending->locked hand-overs with a bounded + * number of spins so that we guarantee forward progress. * * 0,1,0 -> 0,0,1 */ if (val == _Q_PENDING_VAL) { - while ((val = atomic_read(&lock->val)) == _Q_PENDING_VAL) - cpu_relax(); + int cnt = _Q_PENDING_LOOPS; + val = atomic_cond_read_relaxed(&lock->val, + (VAL != _Q_PENDING_VAL) || !cnt--); } /* From patchwork Wed Apr 11 18:01:10 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 133155 Delivered-To: patch@linaro.org Received: by 10.46.84.29 with SMTP id i29csp894966ljb; Wed, 11 Apr 2018 11:01:16 -0700 (PDT) X-Google-Smtp-Source: AIpwx4/ZFXR+G/OMCQhndPKNVf8chnQLcii+hLkj53IsHtfCfIX3d/fjdcPn5llH2TQGaA4v+ndd X-Received: by 2002:a17:902:14cb:: with SMTP id y11-v6mr6247029plg.23.1523469676843; Wed, 11 Apr 2018 11:01:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523469676; cv=none; d=google.com; s=arc-20160816; b=vSg7l0TDVLmoBCi+kWKQjtAlaHduzkJEp9qzXaFv7jgeQebT8JntJ6rZDVrq9XJAdq /mzWPNfQr7rNgTOp4o+ohwTMq74c2ZZLqTG10ibsPb2sfSXKzBoP607aKXRxG0p64x3X AYw+he4lwW1YiO6YBbbu/6jOiX0DuqbHkqgISKw0MV503JE8bMJk9o6nbsrN7PPm1RDP oA2D3e7B7556uDANXv4C7xMRTphAERArfJgAqPj2DOQATITQAH+Cr3rExxb7KyqZGP0o QYUnVg4RW6yVSQ7603A1Ixb1SYPkm9/IgwL9Q4JA95Hnzsk1NBt2kDz/hMeR9FrlBozn M1VQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=UpprWzWKCZ0TgY+oDKG2nWb+NaAp1aw4+EHl+/QuLWw=; b=fzc8YDGq/lP2ugt4EOY8qL8EJ07/fFeUjstawiOyvfN2WJojk2Z9LH70B+4WWJhOfA Dty+YGpby5yks1C6WTh5CSH/2bkFWFNIivXHr9aYenSnt5cdooUu5VO+YnDlA97KOuVW a8ccU5yEJBBoWzYY0KeMJTt0ojG2NoqGrgQBsWU6Y0VM1helWpxS+k8mz2hvB7CRitFR XYWxlMp80MomM/Wb3vWKfdzGrTduMInGgLn7a6r07swB8p/q6F5U8zWh4Z9uOTESMmfD GkcOLx6UL1nwTFvt6lBBh69/6BmdfjyF2Eqs/x/MWG0UmWok/iV4GvhdjHgJ8kzP7H8X adqA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id o12-v6si1543676plg.715.2018.04.11.11.01.16; Wed, 11 Apr 2018 11:01:16 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753047AbeDKSBK (ORCPT + 29 others); Wed, 11 Apr 2018 14:01:10 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:52054 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752584AbeDKSBG (ORCPT ); Wed, 11 Apr 2018 14:01:06 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3FCA115BE; Wed, 11 Apr 2018 11:01:06 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 104753F77D; Wed, 11 Apr 2018 11:01:06 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 767411AE5591; Wed, 11 Apr 2018 19:01:21 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, Will Deacon Subject: [PATCH v2 03/13] locking/qspinlock/x86: Increase _Q_PENDING_LOOPS upper bound Date: Wed, 11 Apr 2018 19:01:10 +0100 Message-Id: <1523469680-17699-4-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1523469680-17699-1-git-send-email-will.deacon@arm.com> References: <1523469680-17699-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On x86, atomic_cond_read_relaxed will busy-wait with a cpu_relax() loop, so it is desirable to increase the number of times we spin on the qspinlock lockword when it is found to be transitioning from pending to locked. According to Waiman Long: | Ideally, the spinning times should be at least a few times the typical | cacheline load time from memory which I think can be down to 100ns or | so for each cacheline load with the newest systems or up to several | hundreds ns for older systems. which in his benchmarking corresponded to 512 iterations. Cc: Peter Zijlstra Cc: Ingo Molnar Suggested-by: Waiman Long Signed-off-by: Will Deacon --- arch/x86/include/asm/qspinlock.h | 2 ++ 1 file changed, 2 insertions(+) -- 2.1.4 diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h index 5e16b5d40d32..2f09915f4aa4 100644 --- a/arch/x86/include/asm/qspinlock.h +++ b/arch/x86/include/asm/qspinlock.h @@ -7,6 +7,8 @@ #include #include +#define _Q_PENDING_LOOPS (1 << 9) + #define queued_spin_unlock queued_spin_unlock /** * queued_spin_unlock - release a queued spinlock From patchwork Wed Apr 11 18:01:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 133159 Delivered-To: patch@linaro.org Received: by 10.46.84.29 with SMTP id i29csp895919ljb; Wed, 11 Apr 2018 11:02:02 -0700 (PDT) X-Google-Smtp-Source: AIpwx482wezWySqPczGNplFGC4KyplxslarIlY7l56bsbf4D86NTUxla4Q4hdVVCIlgR8ujnl/eK X-Received: by 10.99.111.136 with SMTP id k130mr4122069pgc.378.1523469722314; Wed, 11 Apr 2018 11:02:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523469722; cv=none; d=google.com; s=arc-20160816; b=ugg7ecuYmiiiy7atKnhd3w1P3OGlI6oGzwfpdfeGSuPZaIkHAyEWj2moVbWJCCJIms VWzCSntKbPD6XS38zi56gcXHTl62RT6Y/GaJHgaoMuv4YSx4GHFp99q803SBR+thtva+ K8r+QeF6oZVyen1CW/K+jVEWcEBsXkU5k2kOd2+B0haaRkeqvL+7Ql1+SyOwbbhkmeHN CK3GDH1kQGWixlGGxsXbhtlPZGkvpAeSlp7cB92rgfB5sYu1RWPkc7KhdtvRWIk1UyNl hmn0fSio/Fjlanw7JwGHo/ynW67vpqK/XLiACCwXgVn1kayiG400oXPay5OOzkNZZBt1 X2nw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=jogq1PSvaE9goeMt6+g3G+VezE+MccDQKZEajYCIqzU=; b=MuJCVYSQOSJvRPAPMEBhh3qSvYxDeF53uOrytdqONnhJkoZxxFNKh1Xd/OIaLXPy9l E4DJ2y30vNpO0rgXcRt+GLvppQcDdAqxyiBE+wMRVI9u+56qBqsc42hcME1A7inTWM29 wrom7vIOU5Hf1hKii4bqHS4FdkL8LZzJEXj9yvW7RGwWnbW0QK7PI88yIOJmQJovbErW NSfOHsOSX6fHtrH0NDfv/GpjtoeCIivAKQ1vsban/5ykpyWdIJAlHgaQGcuHUnHpsm05 5y+nWvmwVSj0e0WUkRmjO4Q3PqPSM/cF8BaSMHobrcHp+pivszWN9kP648yIhL3KwYLR hgTQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f2-v6si1587087plo.434.2018.04.11.11.02.02; Wed, 11 Apr 2018 11:02:02 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753317AbeDKSBN (ORCPT + 29 others); Wed, 11 Apr 2018 14:01:13 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:52066 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752574AbeDKSBG (ORCPT ); Wed, 11 Apr 2018 14:01:06 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4C18415BF; Wed, 11 Apr 2018 11:01:06 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 1C1E63F487; Wed, 11 Apr 2018 11:01:06 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 8654F1AE5592; Wed, 11 Apr 2018 19:01:21 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, Will Deacon Subject: [PATCH v2 04/13] locking/qspinlock: Remove unbounded cmpxchg loop from locking slowpath Date: Wed, 11 Apr 2018 19:01:11 +0100 Message-Id: <1523469680-17699-5-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1523469680-17699-1-git-send-email-will.deacon@arm.com> References: <1523469680-17699-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The qspinlock locking slowpath utilises a "pending" bit as a simple form of an embedded test-and-set lock that can avoid the overhead of explicit queuing in cases where the lock is held but uncontended. This bit is managed using a cmpxchg loop which tries to transition the uncontended lock word from (0,0,0) -> (0,0,1) or (0,0,1) -> (0,1,1). Unfortunately, the cmpxchg loop is unbounded and lockers can be starved indefinitely if the lock word is seen to oscillate between unlocked (0,0,0) and locked (0,0,1). This could happen if concurrent lockers are able to take the lock in the cmpxchg loop without queuing and pass it around amongst themselves. This patch fixes the problem by unconditionally setting _Q_PENDING_VAL using atomic_fetch_or, and then inspecting the old value to see whether we need to spin on the current lock owner, or whether we now effectively hold the lock. The tricky scenario is when concurrent lockers end up queuing on the lock and the lock becomes available, causing us to see a lockword of (n,0,0). With pending now set, simply queuing could lead to deadlock as the head of the queue may not have observed the pending flag being cleared. Conversely, if the head of the queue did observe pending being cleared, then it could transition the lock from (n,0,0) -> (0,0,1) meaning that any attempt to "undo" our setting of the pending bit could race with a concurrent locker trying to set it. We handle this race by preserving the pending bit when taking the lock after reaching the head of the queue and leaving the tail entry intact if we saw pending set, because we know that the tail is going to be updated shortly. Cc: Peter Zijlstra Cc: Ingo Molnar Signed-off-by: Will Deacon --- kernel/locking/qspinlock.c | 102 ++++++++++++++++++++++++++------------------- 1 file changed, 58 insertions(+), 44 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index 396701e8c62d..a8fc402b3f3a 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -162,6 +162,17 @@ struct __qspinlock { #if _Q_PENDING_BITS == 8 /** + * clear_pending - clear the pending bit. + * @lock: Pointer to queued spinlock structure + * + * *,1,* -> *,0,* + */ +static __always_inline void clear_pending(struct qspinlock *lock) +{ + WRITE_ONCE(lock->pending, 0); +} + +/** * clear_pending_set_locked - take ownership and clear the pending bit. * @lock: Pointer to queued spinlock structure * @@ -201,6 +212,17 @@ static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail) #else /* _Q_PENDING_BITS == 8 */ /** + * clear_pending - clear the pending bit. + * @lock: Pointer to queued spinlock structure + * + * *,1,* -> *,0,* + */ +static __always_inline void clear_pending(struct qspinlock *lock) +{ + atomic_andnot(_Q_PENDING_VAL, &lock->val); +} + +/** * clear_pending_set_locked - take ownership and clear the pending bit. * @lock: Pointer to queued spinlock structure * @@ -306,7 +328,7 @@ static __always_inline u32 __pv_wait_head_or_lock(struct qspinlock *lock, void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) { struct mcs_spinlock *prev, *next, *node; - u32 new, old, tail; + u32 old, tail; int idx; BUILD_BUG_ON(CONFIG_NR_CPUS >= (1U << _Q_TAIL_CPU_BITS)); @@ -330,58 +352,50 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) } /* + * If we observe any contention; queue. + */ + if (val & ~_Q_LOCKED_MASK) + goto queue; + + /* * trylock || pending * * 0,0,0 -> 0,0,1 ; trylock * 0,0,1 -> 0,1,1 ; pending */ - for (;;) { + val = atomic_fetch_or_acquire(_Q_PENDING_VAL, &lock->val); + if (!(val & ~_Q_LOCKED_MASK)) { /* - * If we observe any contention; queue. + * we're pending, wait for the owner to go away. + * + * *,1,1 -> *,1,0 + * + * this wait loop must be a load-acquire such that we match the + * store-release that clears the locked bit and create lock + * sequentiality; this is because not all + * clear_pending_set_locked() implementations imply full + * barriers. */ - if (val & ~_Q_LOCKED_MASK) - goto queue; - - new = _Q_LOCKED_VAL; - if (val == new) - new |= _Q_PENDING_VAL; + if (val & _Q_LOCKED_MASK) { + smp_cond_load_acquire(&lock->val.counter, + !(VAL & _Q_LOCKED_MASK)); + } /* - * Acquire semantic is required here as the function may - * return immediately if the lock was free. + * take ownership and clear the pending bit. + * + * *,1,0 -> *,0,1 */ - old = atomic_cmpxchg_acquire(&lock->val, val, new); - if (old == val) - break; - - val = old; - } - - /* - * we won the trylock - */ - if (new == _Q_LOCKED_VAL) + clear_pending_set_locked(lock); return; + } /* - * we're pending, wait for the owner to go away. - * - * *,1,1 -> *,1,0 - * - * this wait loop must be a load-acquire such that we match the - * store-release that clears the locked bit and create lock - * sequentiality; this is because not all clear_pending_set_locked() - * implementations imply full barriers. - */ - smp_cond_load_acquire(&lock->val.counter, !(VAL & _Q_LOCKED_MASK)); - - /* - * take ownership and clear the pending bit. - * - * *,1,0 -> *,0,1 + * If pending was clear but there are waiters in the queue, then + * we need to undo our setting of pending before we queue ourselves. */ - clear_pending_set_locked(lock); - return; + if (!(val & _Q_PENDING_MASK)) + clear_pending(lock); /* * End of pending bit optimistic spinning and beginning of MCS @@ -485,15 +499,15 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) * claim the lock: * * n,0,0 -> 0,0,1 : lock, uncontended - * *,0,0 -> *,0,1 : lock, contended + * *,*,0 -> *,*,1 : lock, contended * - * If the queue head is the only one in the queue (lock value == tail), - * clear the tail code and grab the lock. Otherwise, we only need - * to grab the lock. + * If the queue head is the only one in the queue (lock value == tail) + * and nobody is pending, clear the tail code and grab the lock. + * Otherwise, we only need to grab the lock. */ for (;;) { /* In the PV case we might already have _Q_LOCKED_VAL set */ - if ((val & _Q_TAIL_MASK) != tail) { + if ((val & _Q_TAIL_MASK) != tail || (val & _Q_PENDING_MASK)) { set_locked(lock); break; } From patchwork Wed Apr 11 18:01:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 133158 Delivered-To: patch@linaro.org Received: by 10.46.84.29 with SMTP id i29csp895693ljb; Wed, 11 Apr 2018 11:01:51 -0700 (PDT) X-Google-Smtp-Source: AIpwx4+3zeRsRWL1rXk0fP+hTt3PXNBrXLiuMSuQxMkG2oUhYFvuBwS2AlscIQ8OfVBUN6QR0m9l X-Received: by 10.98.189.14 with SMTP id a14mr4947759pff.30.1523469711131; Wed, 11 Apr 2018 11:01:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523469711; cv=none; d=google.com; s=arc-20160816; b=kxS/LqDz/gmFB4bwiZJ+wJ0JeLNGa2I95me94Kfww6olgCka0X2BPR9A4GtmzBHrGm XIvx7AON1K0+9F+b4M95cgH+SEMO9yojq2aJILn6U3QcrMhyX/bfU1dvwbpD4Uwy5uuO 7YFHFxqrIeLKlUQ8w5U6kXkzanx3sbq+tpN8d1G6y1fg4FKF9w3GxN2dBEzaB3dEtlkx Zs/ktWLyeSLCbVg6UMA1FBNQ8t/V8IFLpGWGNJeq2c7wXIohDiokyxat8zz7X4wGJMDQ Ue2La3nXGOLwagCXwNJ972Lvrj+Ax7RBdsdUNa1UYFWvEFGFfJ8TdP4v6YaPMHASo4LX 24Lg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=xCwX6s17JMme4myPU+G2D7lG238iBzymvDdYZhpI09E=; b=RqJZDtspApeFr1pfjl+28LyBasNulqr1tiRjwHd2dLJuHgop1PeR+vxB/YGB7sth0d lnsAYM8HWYEk0G5uAGq5XqhqTLKQwJD79NgWGq1hppubE9hiF/NYgkKVsEXX0/qy5BUU f6A4YGQwpHiA89iUNYazHrx0djN5JTQDKeR7jhzhSrenofCe3lsnGTl97xOi8TU3RfnM 0p7XppTBP1SSYU9E3WrqDSFq5Uz6d0aasr6WdsA5Ob4uWWp/izQyoWKn4wQYUb864ID5 rdqfwhgVAQ/3W5O8rSQLjGpi/8QqhMgigi8AIdTYZSyRLJ2Gvm7jkJCR7hZazoclNdDj zG7A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p20si1258214pfi.345.2018.04.11.11.01.50; Wed, 11 Apr 2018 11:01:51 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753584AbeDKSBP (ORCPT + 29 others); Wed, 11 Apr 2018 14:01:15 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:52128 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752588AbeDKSBH (ORCPT ); Wed, 11 Apr 2018 14:01:07 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 054B21650; Wed, 11 Apr 2018 11:01:07 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id CB88C3F487; Wed, 11 Apr 2018 11:01:06 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 928FC1AE55BA; Wed, 11 Apr 2018 19:01:21 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, Will Deacon Subject: [PATCH v2 05/13] locking/qspinlock: Kill cmpxchg loop when claiming lock from head of queue Date: Wed, 11 Apr 2018 19:01:12 +0100 Message-Id: <1523469680-17699-6-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1523469680-17699-1-git-send-email-will.deacon@arm.com> References: <1523469680-17699-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When a queued locker reaches the head of the queue, it claims the lock by setting _Q_LOCKED_VAL in the lockword. If there isn't contention, it must also clear the tail as part of this operation so that subsequent lockers can avoid taking the slowpath altogether. Currently this is expressed as a cmpxchg loop that practically only runs up to two iterations. This is confusing to the reader and unhelpful to the compiler. Rewrite the cmpxchg loop without the loop, so that a failed cmpxchg implies that there is contention and we just need to write to _Q_LOCKED_VAL without considering the rest of the lockword. Cc: Peter Zijlstra Cc: Ingo Molnar Signed-off-by: Will Deacon --- kernel/locking/qspinlock.c | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index a8fc402b3f3a..01b660442d87 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -505,24 +505,21 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) * and nobody is pending, clear the tail code and grab the lock. * Otherwise, we only need to grab the lock. */ - for (;;) { - /* In the PV case we might already have _Q_LOCKED_VAL set */ - if ((val & _Q_TAIL_MASK) != tail || (val & _Q_PENDING_MASK)) { - set_locked(lock); - break; - } + + /* In the PV case we might already have _Q_LOCKED_VAL set */ + if ((val & _Q_TAIL_MASK) == tail) { /* * The smp_cond_load_acquire() call above has provided the - * necessary acquire semantics required for locking. At most - * two iterations of this loop may be ran. + * necessary acquire semantics required for locking. */ old = atomic_cmpxchg_relaxed(&lock->val, val, _Q_LOCKED_VAL); if (old == val) - goto release; /* No contention */ - - val = old; + goto release; /* No contention */ } + /* Either somebody is queued behind us or _Q_PENDING_VAL is set */ + set_locked(lock); + /* * contended path; wait for next if not observed yet, release. */ From patchwork Wed Apr 11 18:01:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 133163 Delivered-To: patch@linaro.org Received: by 10.46.84.29 with SMTP id i29csp896915ljb; Wed, 11 Apr 2018 11:03:00 -0700 (PDT) X-Google-Smtp-Source: AIpwx49vB+mvFYxOlw4ylDZXdMjN36HU7dzAVZbUMN+PM5XqYEzasKsL5k2eDpELn0xd8z0igF7T X-Received: by 2002:a17:902:2b84:: with SMTP id l4-v6mr6290449plb.65.1523469780048; Wed, 11 Apr 2018 11:03:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523469780; cv=none; d=google.com; s=arc-20160816; b=1K/8A3qj4xuD/Q/y1UhwhnsnY0U1PUQI8nM7WzDzqzp+nXfWnqS1dCPkHTvFlitZUB xFOtmw8PWP2GNxHp0ZE4sxhl9dk0R3QxMGiyf+JEzP2nqMDQTuY9P1aXPyl3aHKnAeiS Er7jyziTRuyALIB93RBXRYwg8bWSJJEp1ukJy34Wp/mh7JxlcNslM6frRowrL3EDqqnV qWZoooKbTDy9TVlAmB3ST7WWkQthC1UG4VlzQ0zlM2JMEXdT2/mgCWl2rADwOdMtuwzH jMlLnIarmLbpejl5ojvwypC/5DqxfVPlh4Puaiq9CUtBvl+tjb3INC7ZqToCaS0GyQd9 P2Rg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=8XpX1C6Q4iwZGgJB7HSRDSsXSUOB7Uj4yn14t9Dkl7o=; b=0TBr+uyX/OM243GUve/XhJiUg+bBudrOBuB4etB+NjBxWwPEcIjwE/CkC8Rc68+iPb 6PqxBH4tQgh1sQBWaoXT1iqdgJT2jaG0NBrmUTaG0DLx6HDtFUszeknx+W38DagzD0rs VaPstKDsqEbevOhJc7dYtFhbW3SRh6SmUmQMgUv4ZY9PGNHgnAGfuUGKuMgE1n81Fdg+ JcxTO9TH2pp6tesLwfH+a4Pt/62T6AQHUA1sT1wWLcHCKZZtS0QPV7tbbtqvRMbFziFg xhBGmvQuoP4CBe/SZhv4WKKxJHzl8YlFpBH6kkBdLaKu/qNGNTuEMm+/dL9Ax8J+Moil Ooxg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p12-v6si1686221plr.131.2018.04.11.11.02.59; Wed, 11 Apr 2018 11:03:00 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754122AbeDKSC5 (ORCPT + 29 others); Wed, 11 Apr 2018 14:02:57 -0400 Received: from foss.arm.com ([217.140.101.70]:52130 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752591AbeDKSBH (ORCPT ); Wed, 11 Apr 2018 14:01:07 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0E3CD165C; Wed, 11 Apr 2018 11:01:07 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id D3E7D3F5B1; Wed, 11 Apr 2018 11:01:06 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id A47E91AE55BF; Wed, 11 Apr 2018 19:01:21 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, Will Deacon Subject: [PATCH v2 06/13] locking/qspinlock: Use atomic_cond_read_acquire Date: Wed, 11 Apr 2018 19:01:13 +0100 Message-Id: <1523469680-17699-7-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1523469680-17699-1-git-send-email-will.deacon@arm.com> References: <1523469680-17699-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Rather than dig into the counter field of the atomic_t inside the qspinlock structure so that we can call smp_cond_load_acquire, use atomic_cond_read_acquire instead, which operates on the atomic_t directly. Cc: Peter Zijlstra Cc: Ingo Molnar Signed-off-by: Will Deacon --- kernel/locking/qspinlock.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index 01b660442d87..648a16a2cd23 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -377,8 +377,8 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) * barriers. */ if (val & _Q_LOCKED_MASK) { - smp_cond_load_acquire(&lock->val.counter, - !(VAL & _Q_LOCKED_MASK)); + atomic_cond_read_acquire(&lock->val, + !(VAL & _Q_LOCKED_MASK)); } /* @@ -481,8 +481,8 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) * * The PV pv_wait_head_or_lock function, if active, will acquire * the lock and return a non-zero value. So we have to skip the - * smp_cond_load_acquire() call. As the next PV queue head hasn't been - * designated yet, there is no way for the locked value to become + * atomic_cond_read_acquire() call. As the next PV queue head hasn't + * been designated yet, there is no way for the locked value to become * _Q_SLOW_VAL. So both the set_locked() and the * atomic_cmpxchg_relaxed() calls will be safe. * @@ -492,7 +492,7 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) if ((val = pv_wait_head_or_lock(lock, node))) goto locked; - val = smp_cond_load_acquire(&lock->val.counter, !(VAL & _Q_LOCKED_PENDING_MASK)); + val = atomic_cond_read_acquire(&lock->val, !(VAL & _Q_LOCKED_PENDING_MASK)); locked: /* @@ -509,7 +509,7 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) /* In the PV case we might already have _Q_LOCKED_VAL set */ if ((val & _Q_TAIL_MASK) == tail) { /* - * The smp_cond_load_acquire() call above has provided the + * The atomic_cond_read_acquire() call above has provided the * necessary acquire semantics required for locking. */ old = atomic_cmpxchg_relaxed(&lock->val, val, _Q_LOCKED_VAL); From patchwork Wed Apr 11 18:01:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 133164 Delivered-To: patch@linaro.org Received: by 10.46.84.29 with SMTP id i29csp896966ljb; Wed, 11 Apr 2018 11:03:03 -0700 (PDT) X-Google-Smtp-Source: AIpwx494MaYX2XIFGgxR15NRHEWanaU29pjBMOytDx1G4jc1m/brJ4Na/ZeEIRzgGavYU6uwiATT X-Received: by 10.99.43.80 with SMTP id r77mr4136822pgr.193.1523469783406; Wed, 11 Apr 2018 11:03:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523469783; cv=none; d=google.com; s=arc-20160816; b=YzTK8Ey5HVjc8xeMAele4ebTcDhaYu9Pw8Jk4BmdI5WKHKRFmCBnVk5gW5Rcv3dTl9 RiVzK1ZjhLW4CTrisdShesInqSwfH6nl8eGuie7VRRwuj/7OUmoFxGA3muzaf/dtrNrN BXwF9sijjsf0EuUd10OThqcIWIvKHqDfRh9VWLvow6TxCaQcQNF5gWBnaswWs6Ox9WRL ghNUAr0fEIpbc8DQjGzVGkVg9pLtwjEeGjyDSqWFJlaE3H0aBF97XaZam8Bl7p+vegWo nKm3x8TuBmGQnaItUBRpHMn3iP5ovd1SEuJ2fumDKYm7PJs6JWxStYxecjfzviS0usLS IsCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=irkULNHmtNPgNAdekth1s68Hkl75EPNE3ReOorX3J4o=; b=e3lgye4TlPa/zQRXcfUukVCOMSrEnieUu0gFmzi32K1FxvXSdwBwgNw2i9X3GhuMVb f8pBmajPl2ywi8IDovOTPFukIdc+tcqCJLeQsCf2JttryII0rM65ZRAIBGzBUTO9MfnT y/oiRHp58u+RPWdKl0/IY5ZXpVeW4oO37lK8P+Nqu6S7MUhGmWEe7g/NkLEe5et+RKyC YCz6T4KAf4OkYnrjB/WlfCMeDUZ5sfk3cOPoZ+LR7D9ZOCh97C2CD2rd8o58nV108A0V Vzh4S4vUVkHocOXnTvI+7xdNF3kgb8SqDK1svFr49dBzQlMGNnL8rUcD+XmSxK89Ryye wDOQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p3si1042816pgc.463.2018.04.11.11.03.03; Wed, 11 Apr 2018 11:03:03 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754138AbeDKSC6 (ORCPT + 29 others); Wed, 11 Apr 2018 14:02:58 -0400 Received: from foss.arm.com ([217.140.101.70]:52132 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752639AbeDKSBH (ORCPT ); Wed, 11 Apr 2018 14:01:07 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 17B451682; Wed, 11 Apr 2018 11:01:07 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id DCDB13F614; Wed, 11 Apr 2018 11:01:06 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id B6D031AE55C3; Wed, 11 Apr 2018 19:01:21 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, Jason Low , Will Deacon Subject: [PATCH v2 07/13] locking/mcs: Use smp_cond_load_acquire() in mcs spin loop Date: Wed, 11 Apr 2018 19:01:14 +0100 Message-Id: <1523469680-17699-8-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1523469680-17699-1-git-send-email-will.deacon@arm.com> References: <1523469680-17699-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Jason Low For qspinlocks on ARM64, we would like to use WFE instead of purely spinning. Qspinlocks internally have lock contenders spin on an MCS lock. Update arch_mcs_spin_lock_contended() such that it uses the new smp_cond_load_acquire() so that ARM64 can also override this spin loop with its own implementation using WFE. On x86, this can also be cheaper than spinning on smp_load_acquire(). Signed-off-by: Jason Low Signed-off-by: Will Deacon --- kernel/locking/mcs_spinlock.h | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) -- 2.1.4 diff --git a/kernel/locking/mcs_spinlock.h b/kernel/locking/mcs_spinlock.h index f046b7ce9dd6..5e10153b4d3c 100644 --- a/kernel/locking/mcs_spinlock.h +++ b/kernel/locking/mcs_spinlock.h @@ -23,13 +23,15 @@ struct mcs_spinlock { #ifndef arch_mcs_spin_lock_contended /* - * Using smp_load_acquire() provides a memory barrier that ensures - * subsequent operations happen after the lock is acquired. + * Using smp_cond_load_acquire() provides the acquire semantics + * required so that subsequent operations happen after the + * lock is acquired. Additionally, some architectures such as + * ARM64 would like to do spin-waiting instead of purely + * spinning, and smp_cond_load_acquire() provides that behavior. */ #define arch_mcs_spin_lock_contended(l) \ do { \ - while (!(smp_load_acquire(l))) \ - cpu_relax(); \ + smp_cond_load_acquire(l, VAL); \ } while (0) #endif From patchwork Wed Apr 11 18:01:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 133157 Delivered-To: patch@linaro.org Received: by 10.46.84.29 with SMTP id i29csp895326ljb; Wed, 11 Apr 2018 11:01:34 -0700 (PDT) X-Google-Smtp-Source: AIpwx4/6BovW9c9pOrWaMnMpg6tJbY+8m/+ITbbb9X0/Y3aVie/KKsEjfj4LrXKmToBln0zHzsPr X-Received: by 10.99.138.202 with SMTP id y193mr4171831pgd.224.1523469694052; Wed, 11 Apr 2018 11:01:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523469694; cv=none; d=google.com; s=arc-20160816; b=g6DmXDST+j3gosVErpgqmLWfsemlTndZ0sY4OpB5edPGO+qZKSS1xvRxnl7yJ2Gij6 Io9/uLU9iqhJyfzRvXs1wCUjvM4qSIcqhw+wOZo3XLTrUSld3IHkmoVkG17qQQtaWbAN x5fkjXW4bZJ9bBnpeMeLF1S14j9hz0bSbQvAZFyAUJz5EHaFhK5D9jMJSty/37tYjBbN ASi4r/UCj4NHsmv7gTptc2ycaWRKjIafctUzZzduuwQw/e3wtvFf6tYPLtXAgEu9KOVZ J3XYjVHcYEC3s4UMxWfSc99KFVRznRjCfOMpYUTi6EDmMDoQIFGbRZ4HT+62vnKlCRY3 avMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=cDKKlZei8NAUemnDDyM83M5fTxXVCZkFOWVvrfMVSzY=; b=rVttk4mdUKJasq8r9nJwKlO/lfCzbb4GgOQlydTB02oCV9/RDF+OrUxYYXV23mxTda BMer0tYk9uECyTffnfQ3JZ5Ea882StBO10oVzQFmXh8qQHmjyNhRyfQopoRLQZ5IxHph oy+WLW3bCxqP5SSNj4L9gVM49sUhJbHtTsivUdkF4XANBHD4ooneSrnshch9KmKyU8Yb HFirBdyhmjL5T+aDEsmXMOK3V+EVzwNPciUprxJkEiR0hvr4fCsYXIb3t17wDcvHZYnU cnBZb7CddldJOz+UNDloBGcPVJzvIm5VmsSfzOWBdmSf633iheTIA/EpmmUvVHfNFTYT ffZw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p20si1258214pfi.345.2018.04.11.11.01.33; Wed, 11 Apr 2018 11:01:34 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753697AbeDKSBR (ORCPT + 29 others); Wed, 11 Apr 2018 14:01:17 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:52134 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752638AbeDKSBH (ORCPT ); Wed, 11 Apr 2018 14:01:07 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 210541684; Wed, 11 Apr 2018 11:01:07 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E69BB3F77D; Wed, 11 Apr 2018 11:01:06 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id C5E191AE55C6; Wed, 11 Apr 2018 19:01:21 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, Will Deacon Subject: [PATCH v2 08/13] locking/qspinlock: Use smp_cond_load_relaxed to wait for next node Date: Wed, 11 Apr 2018 19:01:15 +0100 Message-Id: <1523469680-17699-9-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1523469680-17699-1-git-send-email-will.deacon@arm.com> References: <1523469680-17699-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When a locker reaches the head of the queue and takes the lock, a concurrent locker may enqueue and force the lock holder to spin whilst its node->next field is initialised. Rather than open-code a READ_ONCE/cpu_relax() loop, this can be implemented using smp_cond_load_relaxed instead. Cc: Peter Zijlstra Cc: Ingo Molnar Signed-off-by: Will Deacon --- kernel/locking/qspinlock.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index 648a16a2cd23..c781ddbe59a6 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -523,10 +523,8 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) /* * contended path; wait for next if not observed yet, release. */ - if (!next) { - while (!(next = READ_ONCE(node->next))) - cpu_relax(); - } + if (!next) + next = smp_cond_load_relaxed(&node->next, (VAL)); arch_mcs_spin_unlock_contended(&next->locked); pv_kick_node(lock, next); From patchwork Wed Apr 11 18:01:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 133166 Delivered-To: patch@linaro.org Received: by 10.46.84.29 with SMTP id i29csp897444ljb; Wed, 11 Apr 2018 11:03:35 -0700 (PDT) X-Google-Smtp-Source: AIpwx4+kkOGTUGwmr75RVT1Cviad7eUX/H4CPjnBqPJEZ1AbIbmHGDKP1xn4xB2RBIxxx0pFxoxY X-Received: by 2002:a17:902:b187:: with SMTP id s7-v6mr6175069plr.170.1523469815675; Wed, 11 Apr 2018 11:03:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523469815; cv=none; d=google.com; s=arc-20160816; b=ugwjpn/g0Q0wCTrZPbA0/H+fjXsxHddhX2QJeuNwh/4gkZjTC8INciHhcgjR0xkjOv NFGV7OItGfRTyN+I9rUqfB+PS6dIiaf5UJdLrM3iCAM3Ux3w4pnJ33O6P6t+cnd/huKN FaRLy4y3AWSNIGN0MpgUQ34fPCjbZ1a9VcZQA5/Cb1Wc1TJ/CypREKJTFhveL83vcJfA gRr0AdMJIAYWLuQjSZE8/TvFI93LlgnyaZyAxyKHERI2Tze2A6NZ1GIFAkwm261zKPJc hjweiXOoCfRA5oNQD9Ni+WGDnWV2zdGSF9MsyaF7xuPtscivQSyUwcxWfI/W3VFFnH9q SMYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=h8YOBGT1/U3cQukz3A44aBEzoq9+ZqvRdhIvLOE1AQs=; b=upQE1si8LM3Y2TggW5NQR8YUh+xkML7mqeTr7s26t+P8Lg9xEHQaecoXCSAxvduN2S lFJHf6Eek3quhIUAFJvI/OujX1sZClbJpScKAkWAC2tJqLFgZyH2MBxwt+c0mT7jqeM+ 4ZxBPBwX9WPC93kV0tAGF4th/0gkEdqbA/ctNdACT1H/Ylyt4+YmtgvGJCzi5fAQT+Hw wf+Oaft6JDf0updp+Bn5YiNKbzOTAsJEbk8kxP36ckRXPWAYgtuVp5vgWwpMTyROAc5o xM+WijVl0d4mC4oSbD5HRvmH/px0MkG1EJVhSjUfwtqBEdy2vz+ctibp1cUZAoOqDnqV E4Vw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h11-v6si186034plk.706.2018.04.11.11.03.35; Wed, 11 Apr 2018 11:03:35 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754091AbeDKSCz (ORCPT + 29 others); Wed, 11 Apr 2018 14:02:55 -0400 Received: from foss.arm.com ([217.140.101.70]:52150 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752646AbeDKSBH (ORCPT ); Wed, 11 Apr 2018 14:01:07 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 318251688; Wed, 11 Apr 2018 11:01:07 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 026DE3F7DB; Wed, 11 Apr 2018 11:01:07 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id D5BA21AE55C7; Wed, 11 Apr 2018 19:01:21 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, Will Deacon Subject: [PATCH v2 09/13] locking/qspinlock: Merge struct __qspinlock into struct qspinlock Date: Wed, 11 Apr 2018 19:01:16 +0100 Message-Id: <1523469680-17699-10-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1523469680-17699-1-git-send-email-will.deacon@arm.com> References: <1523469680-17699-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org struct __qspinlock provides a handy union of fields so that subcomponents of the lockword can be accessed by name, without having to manage shifts and masks explicitly and take endianness into account. This is useful in qspinlock.h and also potentially in arch headers, so move the struct __qspinlock into struct qspinlock and kill the extra definition. Cc: Peter Zijlstra Cc: Ingo Molnar Acked-by: Boqun Feng Signed-off-by: Will Deacon --- arch/x86/include/asm/qspinlock.h | 2 +- arch/x86/include/asm/qspinlock_paravirt.h | 3 +- include/asm-generic/qspinlock_types.h | 32 +++++++++++++++++++-- kernel/locking/qspinlock.c | 46 ++----------------------------- kernel/locking/qspinlock_paravirt.h | 34 ++++++++--------------- 5 files changed, 46 insertions(+), 71 deletions(-) -- 2.1.4 diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h index 2f09915f4aa4..da1370ad206d 100644 --- a/arch/x86/include/asm/qspinlock.h +++ b/arch/x86/include/asm/qspinlock.h @@ -18,7 +18,7 @@ */ static inline void native_queued_spin_unlock(struct qspinlock *lock) { - smp_store_release((u8 *)lock, 0); + smp_store_release(&lock->locked, 0); } #ifdef CONFIG_PARAVIRT_SPINLOCKS diff --git a/arch/x86/include/asm/qspinlock_paravirt.h b/arch/x86/include/asm/qspinlock_paravirt.h index 923307ea11c7..9ef5ee03d2d7 100644 --- a/arch/x86/include/asm/qspinlock_paravirt.h +++ b/arch/x86/include/asm/qspinlock_paravirt.h @@ -22,8 +22,7 @@ PV_CALLEE_SAVE_REGS_THUNK(__pv_queued_spin_unlock_slowpath); * * void __pv_queued_spin_unlock(struct qspinlock *lock) * { - * struct __qspinlock *l = (void *)lock; - * u8 lockval = cmpxchg(&l->locked, _Q_LOCKED_VAL, 0); + * u8 lockval = cmpxchg(&lock->locked, _Q_LOCKED_VAL, 0); * * if (likely(lockval == _Q_LOCKED_VAL)) * return; diff --git a/include/asm-generic/qspinlock_types.h b/include/asm-generic/qspinlock_types.h index 034acd0c4956..0763f065b975 100644 --- a/include/asm-generic/qspinlock_types.h +++ b/include/asm-generic/qspinlock_types.h @@ -29,13 +29,41 @@ #endif typedef struct qspinlock { - atomic_t val; + union { + atomic_t val; + + /* + * By using the whole 2nd least significant byte for the + * pending bit, we can allow better optimization of the lock + * acquisition for the pending bit holder. + */ +#ifdef __LITTLE_ENDIAN + struct { + u8 locked; + u8 pending; + }; + struct { + u16 locked_pending; + u16 tail; + }; +#else + struct { + u16 tail; + u16 locked_pending; + }; + struct { + u8 reserved[2]; + u8 pending; + u8 locked; + }; +#endif + }; } arch_spinlock_t; /* * Initializier */ -#define __ARCH_SPIN_LOCK_UNLOCKED { ATOMIC_INIT(0) } +#define __ARCH_SPIN_LOCK_UNLOCKED { .val = ATOMIC_INIT(0) } /* * Bitfields in the atomic value: diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index c781ddbe59a6..7b8c81ebb15e 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -126,40 +126,6 @@ static inline __pure struct mcs_spinlock *decode_tail(u32 tail) #define _Q_LOCKED_PENDING_MASK (_Q_LOCKED_MASK | _Q_PENDING_MASK) -/* - * By using the whole 2nd least significant byte for the pending bit, we - * can allow better optimization of the lock acquisition for the pending - * bit holder. - * - * This internal structure is also used by the set_locked function which - * is not restricted to _Q_PENDING_BITS == 8. - */ -struct __qspinlock { - union { - atomic_t val; -#ifdef __LITTLE_ENDIAN - struct { - u8 locked; - u8 pending; - }; - struct { - u16 locked_pending; - u16 tail; - }; -#else - struct { - u16 tail; - u16 locked_pending; - }; - struct { - u8 reserved[2]; - u8 pending; - u8 locked; - }; -#endif - }; -}; - #if _Q_PENDING_BITS == 8 /** * clear_pending - clear the pending bit. @@ -182,9 +148,7 @@ static __always_inline void clear_pending(struct qspinlock *lock) */ static __always_inline void clear_pending_set_locked(struct qspinlock *lock) { - struct __qspinlock *l = (void *)lock; - - WRITE_ONCE(l->locked_pending, _Q_LOCKED_VAL); + WRITE_ONCE(lock->locked_pending, _Q_LOCKED_VAL); } /* @@ -199,13 +163,11 @@ static __always_inline void clear_pending_set_locked(struct qspinlock *lock) */ static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail) { - struct __qspinlock *l = (void *)lock; - /* * Use release semantics to make sure that the MCS node is properly * initialized before changing the tail code. */ - return (u32)xchg_release(&l->tail, + return (u32)xchg_release(&lock->tail, tail >> _Q_TAIL_OFFSET) << _Q_TAIL_OFFSET; } @@ -271,9 +233,7 @@ static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail) */ static __always_inline void set_locked(struct qspinlock *lock) { - struct __qspinlock *l = (void *)lock; - - WRITE_ONCE(l->locked, _Q_LOCKED_VAL); + WRITE_ONCE(lock->locked, _Q_LOCKED_VAL); } diff --git a/kernel/locking/qspinlock_paravirt.h b/kernel/locking/qspinlock_paravirt.h index 6ee477765e6c..2711940429f5 100644 --- a/kernel/locking/qspinlock_paravirt.h +++ b/kernel/locking/qspinlock_paravirt.h @@ -87,8 +87,6 @@ struct pv_node { #define queued_spin_trylock(l) pv_hybrid_queued_unfair_trylock(l) static inline bool pv_hybrid_queued_unfair_trylock(struct qspinlock *lock) { - struct __qspinlock *l = (void *)lock; - /* * Stay in unfair lock mode as long as queued mode waiters are * present in the MCS wait queue but the pending bit isn't set. @@ -97,7 +95,7 @@ static inline bool pv_hybrid_queued_unfair_trylock(struct qspinlock *lock) int val = atomic_read(&lock->val); if (!(val & _Q_LOCKED_PENDING_MASK) && - (cmpxchg_acquire(&l->locked, 0, _Q_LOCKED_VAL) == 0)) { + (cmpxchg_acquire(&lock->locked, 0, _Q_LOCKED_VAL) == 0)) { qstat_inc(qstat_pv_lock_stealing, true); return true; } @@ -117,16 +115,12 @@ static inline bool pv_hybrid_queued_unfair_trylock(struct qspinlock *lock) #if _Q_PENDING_BITS == 8 static __always_inline void set_pending(struct qspinlock *lock) { - struct __qspinlock *l = (void *)lock; - - WRITE_ONCE(l->pending, 1); + WRITE_ONCE(lock->pending, 1); } static __always_inline void clear_pending(struct qspinlock *lock) { - struct __qspinlock *l = (void *)lock; - - WRITE_ONCE(l->pending, 0); + WRITE_ONCE(lock->pending, 0); } /* @@ -136,10 +130,8 @@ static __always_inline void clear_pending(struct qspinlock *lock) */ static __always_inline int trylock_clear_pending(struct qspinlock *lock) { - struct __qspinlock *l = (void *)lock; - - return !READ_ONCE(l->locked) && - (cmpxchg_acquire(&l->locked_pending, _Q_PENDING_VAL, + return !READ_ONCE(lock->locked) && + (cmpxchg_acquire(&lock->locked_pending, _Q_PENDING_VAL, _Q_LOCKED_VAL) == _Q_PENDING_VAL); } #else /* _Q_PENDING_BITS == 8 */ @@ -384,7 +376,6 @@ static void pv_wait_node(struct mcs_spinlock *node, struct mcs_spinlock *prev) static void pv_kick_node(struct qspinlock *lock, struct mcs_spinlock *node) { struct pv_node *pn = (struct pv_node *)node; - struct __qspinlock *l = (void *)lock; /* * If the vCPU is indeed halted, advance its state to match that of @@ -413,7 +404,7 @@ static void pv_kick_node(struct qspinlock *lock, struct mcs_spinlock *node) * the hash table later on at unlock time, no atomic instruction is * needed. */ - WRITE_ONCE(l->locked, _Q_SLOW_VAL); + WRITE_ONCE(lock->locked, _Q_SLOW_VAL); (void)pv_hash(lock, pn); } @@ -428,7 +419,6 @@ static u32 pv_wait_head_or_lock(struct qspinlock *lock, struct mcs_spinlock *node) { struct pv_node *pn = (struct pv_node *)node; - struct __qspinlock *l = (void *)lock; struct qspinlock **lp = NULL; int waitcnt = 0; int loop; @@ -479,13 +469,13 @@ pv_wait_head_or_lock(struct qspinlock *lock, struct mcs_spinlock *node) * * Matches the smp_rmb() in __pv_queued_spin_unlock(). */ - if (xchg(&l->locked, _Q_SLOW_VAL) == 0) { + if (xchg(&lock->locked, _Q_SLOW_VAL) == 0) { /* * The lock was free and now we own the lock. * Change the lock value back to _Q_LOCKED_VAL * and unhash the table. */ - WRITE_ONCE(l->locked, _Q_LOCKED_VAL); + WRITE_ONCE(lock->locked, _Q_LOCKED_VAL); WRITE_ONCE(*lp, NULL); goto gotlock; } @@ -493,7 +483,7 @@ pv_wait_head_or_lock(struct qspinlock *lock, struct mcs_spinlock *node) WRITE_ONCE(pn->state, vcpu_hashed); qstat_inc(qstat_pv_wait_head, true); qstat_inc(qstat_pv_wait_again, waitcnt); - pv_wait(&l->locked, _Q_SLOW_VAL); + pv_wait(&lock->locked, _Q_SLOW_VAL); /* * Because of lock stealing, the queue head vCPU may not be @@ -518,7 +508,6 @@ pv_wait_head_or_lock(struct qspinlock *lock, struct mcs_spinlock *node) __visible void __pv_queued_spin_unlock_slowpath(struct qspinlock *lock, u8 locked) { - struct __qspinlock *l = (void *)lock; struct pv_node *node; if (unlikely(locked != _Q_SLOW_VAL)) { @@ -547,7 +536,7 @@ __pv_queued_spin_unlock_slowpath(struct qspinlock *lock, u8 locked) * Now that we have a reference to the (likely) blocked pv_node, * release the lock. */ - smp_store_release(&l->locked, 0); + smp_store_release(&lock->locked, 0); /* * At this point the memory pointed at by lock can be freed/reused, @@ -573,7 +562,6 @@ __pv_queued_spin_unlock_slowpath(struct qspinlock *lock, u8 locked) #ifndef __pv_queued_spin_unlock __visible void __pv_queued_spin_unlock(struct qspinlock *lock) { - struct __qspinlock *l = (void *)lock; u8 locked; /* @@ -581,7 +569,7 @@ __visible void __pv_queued_spin_unlock(struct qspinlock *lock) * unhash. Otherwise it would be possible to have multiple @lock * entries, which would be BAD. */ - locked = cmpxchg_release(&l->locked, _Q_LOCKED_VAL, 0); + locked = cmpxchg_release(&lock->locked, _Q_LOCKED_VAL, 0); if (likely(locked == _Q_LOCKED_VAL)) return; From patchwork Wed Apr 11 18:01:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 133165 Delivered-To: patch@linaro.org Received: by 10.46.84.29 with SMTP id i29csp897272ljb; Wed, 11 Apr 2018 11:03:22 -0700 (PDT) X-Google-Smtp-Source: AIpwx49X2Sb3qOinrfK1umOH+R7Cpfpeg2xr/UmrNkA/NA7VpYu1uTioEua2zxQGuDcoT3GGHZwp X-Received: by 10.98.127.144 with SMTP id a138mr3338347pfd.239.1523469802835; Wed, 11 Apr 2018 11:03:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523469802; cv=none; d=google.com; s=arc-20160816; b=B24m0kYaj6seHtwFR0xiVoVMlcV/0GbIfQp8T+cBH8hDwK258YWmz/NiF8TR44QTKp t3w1D47mU10ceG7tErlS0NojWgXB9/zqeuejy5BfcmqpcSZbbQ+l8HxW3UY3y7mooOEQ EqkMH+jBSBpRghMO+39XKyWW6krVlCwrVNsr8u3Zcm81Ut8VL6L1nyhjYYdRE4stONXA 3PrOLqaGaCPn7Mub3WkMC0HlzCvmBZhe1ApZRxkp/q7SiPnysPKYBMSKCQ2TlUeayqGX /yhCvuyLthPksMvrLFYGFTlLc5WT+LZLIFFKRw/oQVx8LM3IaYlc8pv2bkcmApAmkmCi TUMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=wkblKAmBxnaQe+cTwroykfXYTv1d1hqWOBC1dh053Dg=; b=C1+20e1r8V4fPxc2GQRqrLza6UnB5c9AkYjayOwPGokLopJG3Z1TXDyE6kl1I+ehqd f8/ezb8JdMFkmvK4tDMhGtIjJBxWf0V5kI3jKg75NB4FfOXaZBQgHj0+btRc6so3/H6E uIIK/6n7UeBz87d7QbKYD6xFKaA6S+luki+xc/abrM3lQlsdmcpIwr8xpMuw/yYZUaYe 6bFoHCbOsg2IRbm8uSrt/C/Awmo/BPlR8RsBOYOUwpTwsLexSjEqlmcJGqt27iK/7Ctn dUMKIXXcEqiobHCIxJYnnKmbEcXrQSZG3+OmCExs9sk+JgbbTo6L2dNet8cW+GuLKhQ7 BRBw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p3si1042816pgc.463.2018.04.11.11.03.22; Wed, 11 Apr 2018 11:03:22 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754104AbeDKSC4 (ORCPT + 29 others); Wed, 11 Apr 2018 14:02:56 -0400 Received: from foss.arm.com ([217.140.101.70]:52152 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752662AbeDKSBH (ORCPT ); Wed, 11 Apr 2018 14:01:07 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3B823168F; Wed, 11 Apr 2018 11:01:07 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0C4623F487; Wed, 11 Apr 2018 11:01:07 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id E60141AE55CF; Wed, 11 Apr 2018 19:01:21 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, Will Deacon Subject: [PATCH v2 10/13] locking/qspinlock: Make queued_spin_unlock use smp_store_release Date: Wed, 11 Apr 2018 19:01:17 +0100 Message-Id: <1523469680-17699-11-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1523469680-17699-1-git-send-email-will.deacon@arm.com> References: <1523469680-17699-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org A qspinlock can be unlocked simply by writing zero to the locked byte. This can be implemented in the generic code, so do that and remove the arch-specific override for x86 in the !PV case. Cc: Peter Zijlstra Cc: Ingo Molnar Signed-off-by: Will Deacon --- arch/x86/include/asm/qspinlock.h | 17 ++++++----------- include/asm-generic/qspinlock.h | 2 +- 2 files changed, 7 insertions(+), 12 deletions(-) -- 2.1.4 diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h index da1370ad206d..3e70bed8a978 100644 --- a/arch/x86/include/asm/qspinlock.h +++ b/arch/x86/include/asm/qspinlock.h @@ -9,6 +9,12 @@ #define _Q_PENDING_LOOPS (1 << 9) +#ifdef CONFIG_PARAVIRT_SPINLOCKS +extern void native_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); +extern void __pv_init_lock_hash(void); +extern void __pv_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); +extern void __raw_callee_save___pv_queued_spin_unlock(struct qspinlock *lock); + #define queued_spin_unlock queued_spin_unlock /** * queued_spin_unlock - release a queued spinlock @@ -21,12 +27,6 @@ static inline void native_queued_spin_unlock(struct qspinlock *lock) smp_store_release(&lock->locked, 0); } -#ifdef CONFIG_PARAVIRT_SPINLOCKS -extern void native_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); -extern void __pv_init_lock_hash(void); -extern void __pv_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); -extern void __raw_callee_save___pv_queued_spin_unlock(struct qspinlock *lock); - static inline void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) { pv_queued_spin_lock_slowpath(lock, val); @@ -42,11 +42,6 @@ static inline bool vcpu_is_preempted(long cpu) { return pv_vcpu_is_preempted(cpu); } -#else -static inline void queued_spin_unlock(struct qspinlock *lock) -{ - native_queued_spin_unlock(lock); -} #endif #ifdef CONFIG_PARAVIRT diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinlock.h index b37b4ad7eb94..a8ed0a352d75 100644 --- a/include/asm-generic/qspinlock.h +++ b/include/asm-generic/qspinlock.h @@ -100,7 +100,7 @@ static __always_inline void queued_spin_unlock(struct qspinlock *lock) /* * unlock() needs release semantics: */ - (void)atomic_sub_return_release(_Q_LOCKED_VAL, &lock->val); + smp_store_release(&lock->locked, 0); } #endif From patchwork Wed Apr 11 18:01:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 133169 Delivered-To: patch@linaro.org Received: by 10.46.84.29 with SMTP id i29csp897959ljb; Wed, 11 Apr 2018 11:04:10 -0700 (PDT) X-Google-Smtp-Source: AIpwx4+3uBlBdhYyvKLwL1OAlR8m9E7LpU+c5zhN6ODrthLDQjpP94zl1Cd0iA+Bap1ocTJPxPoH X-Received: by 2002:a17:902:3381:: with SMTP id b1-v6mr6144659plc.214.1523469850177; Wed, 11 Apr 2018 11:04:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523469850; cv=none; d=google.com; s=arc-20160816; b=eOd9vnMzoF2AjL9YasPz8MTpWyZT9fwTyEPHQJ8QYwyzbZKaaAD3myn7NntyWr7pGa CQumlzzHkh4XfxZCLYTs+LZ14xwzs5MtpBpUAW3cC/vxpQh9+ZJVYC5lYo2lxo+2tISP w4T8GzC8suoyjCmaZb6JBEXHAbmFcJlf4Rt8b8ttbjvEsb8ctTU9ZlCdlY044VzMoeRs 2ceq5h7jI3XW9apzAeF2dHfayndyBJRIxsL34pOdg008B7AIO3EOJqdxtWnQ7Eq+0Xv2 vNv4nsk7yrXb/DNNtzjMFzJiaUBOYHRitZDCoyIQOO8lP4GiiJhQ/l3gJRE25JiwHcHx rNXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=2tA+Zl24djFBobDnFPgPbgE1iwOO4fg0RFP1P3nWloM=; b=mTcHhuS/MZ10wR2RdYXmkD2C8KPfVSOHURy7gPUAGxtH8ABENq5x0yBC7yBnaBvhWe DKtJeE2poVoJ5IFIjspnri0lBCdEofRlvFBPldASiOQGQxvjTNsjKn67Mzz67w3wBmNd DjwOSsNtnosXT2gl3/SYEXSjqvtGMmD9cuRHbse8iEuq+QHmV/o81+a0qIKujSD7jlv3 yESJAPoawCyZuRp6V2KqBhcrCw/XjMyRDLwkPiULbWHPt+0JOAtiLBh5vv/DtrzBimD2 MyTe0gFKNirE/RWMoIAFoShI1EMb6Zhfevnhi2x52X2C4Fy7FV2EWXTPfwtkhkA0Lyo+ 6CzA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g5si1074288pgr.411.2018.04.11.11.04.09; Wed, 11 Apr 2018 11:04:10 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754055AbeDKSCx (ORCPT + 29 others); Wed, 11 Apr 2018 14:02:53 -0400 Received: from foss.arm.com ([217.140.101.70]:52154 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752664AbeDKSBH (ORCPT ); Wed, 11 Apr 2018 14:01:07 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4548C16A3; Wed, 11 Apr 2018 11:01:07 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 158DE3F5B1; Wed, 11 Apr 2018 11:01:07 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 03C581AE55D7; Wed, 11 Apr 2018 19:01:21 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, Will Deacon Subject: [PATCH v2 11/13] locking/qspinlock: Elide back-to-back RELEASE operations with smp_wmb() Date: Wed, 11 Apr 2018 19:01:18 +0100 Message-Id: <1523469680-17699-12-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1523469680-17699-1-git-send-email-will.deacon@arm.com> References: <1523469680-17699-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The qspinlock slowpath must ensure that the MCS node is fully initialised before it can be reached by another other CPU. This is currently enforced by using a RELEASE operation when updating the tail and also when linking the node into the waitqueue (since the control dependency off xchg_tail is insufficient to enforce sufficient ordering -- see 95bcade33a8a ("locking/qspinlock: Ensure node is initialised before updating prev->next")). Back-to-back RELEASE operations may be expensive on some architectures, particularly those that implement them using fences under the hood. We can replace the two RELEASE operations with a single smp_wmb() fence and use RELAXED operations for the subsequent publishing of the node. Cc: Peter Zijlstra Cc: Ingo Molnar Signed-off-by: Will Deacon --- kernel/locking/qspinlock.c | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index 7b8c81ebb15e..fa5d2ab369f9 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -164,10 +164,10 @@ static __always_inline void clear_pending_set_locked(struct qspinlock *lock) static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail) { /* - * Use release semantics to make sure that the MCS node is properly - * initialized before changing the tail code. + * We can use relaxed semantics since the caller ensures that the + * MCS node is properly initialized before updating the tail. */ - return (u32)xchg_release(&lock->tail, + return (u32)xchg_relaxed(&lock->tail, tail >> _Q_TAIL_OFFSET) << _Q_TAIL_OFFSET; } @@ -212,10 +212,11 @@ static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail) for (;;) { new = (val & _Q_LOCKED_PENDING_MASK) | tail; /* - * Use release semantics to make sure that the MCS node is - * properly initialized before changing the tail code. + * We can use relaxed semantics since the caller ensures that + * the MCS node is properly initialized before updating the + * tail. */ - old = atomic_cmpxchg_release(&lock->val, val, new); + old = atomic_cmpxchg_relaxed(&lock->val, val, new); if (old == val) break; @@ -388,12 +389,18 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) goto release; /* + * Ensure that the initialisation of @node is complete before we + * publish the updated tail via xchg_tail() and potentially link + * @node into the waitqueue via WRITE_ONCE(prev->next, node) below. + */ + smp_wmb(); + + /* + * Publish the updated tail. * We have already touched the queueing cacheline; don't bother with * pending stuff. * * p,*,* -> n,*,* - * - * RELEASE, such that the stores to @node must be complete. */ old = xchg_tail(lock, tail); next = NULL; @@ -405,14 +412,8 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) if (old & _Q_TAIL_MASK) { prev = decode_tail(old); - /* - * We must ensure that the stores to @node are observed before - * the write to prev->next. The address dependency from - * xchg_tail is not sufficient to ensure this because the read - * component of xchg_tail is unordered with respect to the - * initialisation of @node. - */ - smp_store_release(&prev->next, node); + /* Link @node into the waitqueue. */ + WRITE_ONCE(prev->next, node); pv_wait_node(node, prev); arch_mcs_spin_lock_contended(&node->locked); From patchwork Wed Apr 11 18:01:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 133168 Delivered-To: patch@linaro.org Received: by 10.46.84.29 with SMTP id i29csp897769ljb; Wed, 11 Apr 2018 11:03:58 -0700 (PDT) X-Google-Smtp-Source: AIpwx4/KLZm/MgtgT5iWFsbs/WGGlSaN0+STag0YunvP+2KAB2jRz66nn/Bd/AQUM12TL4Z5ssmH X-Received: by 2002:a17:902:7e42:: with SMTP id a2-v6mr6137535pln.13.1523469838774; Wed, 11 Apr 2018 11:03:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523469838; cv=none; d=google.com; s=arc-20160816; b=NpOxlM8YiQka/SgVtlo4iInFf+Z/45kre4li4/jeabd3NvwrXrAUNgj1KkZLr1wjSi CiZ+SzPLI0vglvpdGDLa6cNqwu6oEbH/JYVvYLFmKHt6jP33nMiu/qAVExkU0dcZy+P+ BWT1UJW77Pkw0e/yC1W6wpcOzrVZ4Z9TCh37AXeS6kAmJal26ikrlxTYzqwkziISqcl9 ddm16CENfVAlvWywgIT/nbqewpq7hR/8wUCLGyhc+tgouPQ2ASo24n3S14dB5t+BDeDR R2+4Ja3Rwa83aWbwq1a++oXzT0CxYPhMavSOUa/RctGZQTLz1AdP41GPRRvPF7rW4MUs LfEA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=Gd79BK1RjZEjjy2T6jUzYMNwBJjAvxevnUzM+pM9VQo=; b=weYMuXWVPw9NpvlJ4px6E2uwMxiEK4EITYOscjFdPlePn2B9/OcRPipI0gXlzzcdnG 3KO/f3XorXSJYXfFUyVQUTkiuC/INZ96pq9zcC0bTZbysMm3SvpqXgM5mBxdBCuMGO07 cLXdg+8GYuYAbTTXloFuM8Tl0aGqYfDGgXvToeaxtfVHDYWKdamPW/JCrYgaQIWVEnVA KtKKZfvfYpgdS70jI6iKcsuA4ayl/YX6JbQ++3LL8CURVeBlSkvbAbUNmBQZq0Mf8eIQ 0eBqnQ8MZc1yo6Du9hmNJd1BHK/VpPp9LO8ruAb4mB01GPPBAEnVho+aNMAKYAaDdON0 ry+w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g5si1074288pgr.411.2018.04.11.11.03.58; Wed, 11 Apr 2018 11:03:58 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754075AbeDKSCy (ORCPT + 29 others); Wed, 11 Apr 2018 14:02:54 -0400 Received: from foss.arm.com ([217.140.101.70]:52156 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752669AbeDKSBH (ORCPT ); Wed, 11 Apr 2018 14:01:07 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4E33416EA; Wed, 11 Apr 2018 11:01:07 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 1EB353F614; Wed, 11 Apr 2018 11:01:07 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 1202B1AE55D9; Wed, 11 Apr 2018 19:01:22 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, Will Deacon Subject: [PATCH v2 12/13] locking/qspinlock: Use try_cmpxchg instead of cmpxchg when locking Date: Wed, 11 Apr 2018 19:01:19 +0100 Message-Id: <1523469680-17699-13-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1523469680-17699-1-git-send-email-will.deacon@arm.com> References: <1523469680-17699-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When reaching the head of an uncontended queue on the qspinlock slow-path, using a try_cmpxchg instead of a cmpxchg operation to transition the lock work to _Q_LOCKED_VAL generates slightly better code for x86 and pretty much identical code for arm64. Cc: Ingo Molnar Reported-by: Peter Zijlstra Signed-off-by: Will Deacon --- kernel/locking/qspinlock.c | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index fa5d2ab369f9..1e3ddc42135e 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -467,16 +467,15 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) * Otherwise, we only need to grab the lock. */ - /* In the PV case we might already have _Q_LOCKED_VAL set */ - if ((val & _Q_TAIL_MASK) == tail) { - /* - * The atomic_cond_read_acquire() call above has provided the - * necessary acquire semantics required for locking. - */ - old = atomic_cmpxchg_relaxed(&lock->val, val, _Q_LOCKED_VAL); - if (old == val) - goto release; /* No contention */ - } + /* + * In the PV case we might already have _Q_LOCKED_VAL set. + * + * The atomic_cond_read_acquire() call above has provided the + * necessary acquire semantics required for locking. + */ + if (((val & _Q_TAIL_MASK) == tail) && + atomic_try_cmpxchg_relaxed(&lock->val, &val, _Q_LOCKED_VAL)) + goto release; /* No contention */ /* Either somebody is queued behind us or _Q_PENDING_VAL is set */ set_locked(lock); From patchwork Wed Apr 11 18:01:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 133156 Delivered-To: patch@linaro.org Received: by 10.46.84.29 with SMTP id i29csp895139ljb; Wed, 11 Apr 2018 11:01:25 -0700 (PDT) X-Google-Smtp-Source: AIpwx4+0PlZJuCEEFqAXnOu75ZZo0ou/RYe4n55I5XgH5El5jhzi0Zj2HYTD4/k045yBFETDZoMC X-Received: by 10.101.81.11 with SMTP id f11mr4206052pgq.150.1523469685271; Wed, 11 Apr 2018 11:01:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523469685; cv=none; d=google.com; s=arc-20160816; b=msqwHIgwk2xlwXisrMFHc0TkNBOFYazcYrIfIkcgYBzf1LVybpVs+Q7nQNJZFXbIpy zeas/lPt9uDbMmd0COaJT94I2XMzMgBnXSWQadh0ayuiM+hVigWZYmFmb1Tc3ulwkH6U 8oT8yhABlQK0tb0AbB8zQj7ZkW+n4kI/dfi9ujgh4VqMK5zqW/zJlsLklIS0eyqMe89J mSPtwfcHxIkaiI+16XSxQNx+jl1S7sBMsDH/+aFvZjYnLNAXPa0AQH0mj8LufWE5ydlz tkzRnbElsjneNOWZjY4B3SbS9PXjLIJABd7HfGOlnkSBRPTUcyOqpsi7MMnuEqIa9/TZ Ovgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=FynOxYFKxmol1hx5KiUFRtNAR98he4fUWuWi0/cUaj8=; b=aA3r66oNx2Z3NQCxjXlZ2fBL3qLbTgidKGwi/mgdLyJ1meGgOyvKxpMTkJ5QmL1PcT yyYh049rrByD9sE6inzylrt7mKxyHzmJdAm1kpo7PTQJg/HhPXfjkshQ9XkIkhDmnkUE Kloi3eDgYFGBjTdkY6RjllNO25QZLvXDdbqawgQ7ZqJf0RnjgnjdAs9fWAcjtTY6nton yHw32r3ZNL8Qqx1W1TIczosBeDACTMF0IPoDEkaPBRkswNUUGEeW9R16F+hd9KxU0+ts hOXCYhgMFCJZAEMsZ+RlAPRfb/0W6DVFPKtR8ocGMZgZJ3z5YvSg+0TfQ7GVnExeNEeP ovvQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a85si1232213pfe.186.2018.04.11.11.01.24; Wed, 11 Apr 2018 11:01:25 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753833AbeDKSBS (ORCPT + 29 others); Wed, 11 Apr 2018 14:01:18 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:52128 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752735AbeDKSBI (ORCPT ); Wed, 11 Apr 2018 14:01:08 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DE3E11713; Wed, 11 Apr 2018 11:01:07 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 27F583F77D; Wed, 11 Apr 2018 11:01:07 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 204641AE55DA; Wed, 11 Apr 2018 19:01:22 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com Subject: [PATCH v2 13/13] locking/qspinlock: Add stat tracking for pending vs slowpath Date: Wed, 11 Apr 2018 19:01:20 +0100 Message-Id: <1523469680-17699-14-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1523469680-17699-1-git-send-email-will.deacon@arm.com> References: <1523469680-17699-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Waiman Long Currently, the qspinlock_stat code tracks only statistical counts in the PV qspinlock code. However, it may also be useful to track the number of locking operations done via the pending code vs. the MCS lock queue slowpath for the non-PV case. The qspinlock stat code is modified to do that. The stat counter pv_lock_slowpath is renamed to lock_slowpath so that it can be used by both the PV and non-PV cases. Signed-off-by: Waiman Long --- kernel/locking/qspinlock.c | 14 +++++++++++--- kernel/locking/qspinlock_paravirt.h | 7 +------ kernel/locking/qspinlock_stat.h | 9 ++++++--- 3 files changed, 18 insertions(+), 12 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index 1e3ddc42135e..b39f341d831b 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -12,11 +12,11 @@ * GNU General Public License for more details. * * (C) Copyright 2013-2015 Hewlett-Packard Development Company, L.P. - * (C) Copyright 2013-2014 Red Hat, Inc. + * (C) Copyright 2013-2014,2018 Red Hat, Inc. * (C) Copyright 2015 Intel Corp. * (C) Copyright 2015 Hewlett-Packard Enterprise Development LP * - * Authors: Waiman Long + * Authors: Waiman Long * Peter Zijlstra */ @@ -33,6 +33,11 @@ #include /* + * Include queued spinlock statistics code + */ +#include "qspinlock_stat.h" + +/* * The basic principle of a queue-based spinlock can best be understood * by studying a classic queue-based spinlock implementation called the * MCS lock. The paper below provides a good description for this kind @@ -295,7 +300,7 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) BUILD_BUG_ON(CONFIG_NR_CPUS >= (1U << _Q_TAIL_CPU_BITS)); if (pv_enabled()) - goto queue; + goto pv_queue; if (virt_spin_lock(lock)) return; @@ -348,6 +353,7 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) * *,1,0 -> *,0,1 */ clear_pending_set_locked(lock); + qstat_inc(qstat_lock_pending, true); return; } @@ -363,6 +369,8 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) * queuing. */ queue: + qstat_inc(qstat_lock_slowpath, true); +pv_queue: node = this_cpu_ptr(&mcs_nodes[0]); idx = node->count++; tail = encode_tail(smp_processor_id(), idx); diff --git a/kernel/locking/qspinlock_paravirt.h b/kernel/locking/qspinlock_paravirt.h index 2711940429f5..970f47632b6f 100644 --- a/kernel/locking/qspinlock_paravirt.h +++ b/kernel/locking/qspinlock_paravirt.h @@ -56,11 +56,6 @@ struct pv_node { }; /* - * Include queued spinlock statistics code - */ -#include "qspinlock_stat.h" - -/* * Hybrid PV queued/unfair lock * * By replacing the regular queued_spin_trylock() with the function below, @@ -433,7 +428,7 @@ pv_wait_head_or_lock(struct qspinlock *lock, struct mcs_spinlock *node) /* * Tracking # of slowpath locking operations */ - qstat_inc(qstat_pv_lock_slowpath, true); + qstat_inc(qstat_lock_slowpath, true); for (;; waitcnt++) { /* diff --git a/kernel/locking/qspinlock_stat.h b/kernel/locking/qspinlock_stat.h index 4a30ef63c607..6bd78c0740fc 100644 --- a/kernel/locking/qspinlock_stat.h +++ b/kernel/locking/qspinlock_stat.h @@ -22,13 +22,14 @@ * pv_kick_wake - # of vCPU kicks used for computing pv_latency_wake * pv_latency_kick - average latency (ns) of vCPU kick operation * pv_latency_wake - average latency (ns) from vCPU kick to wakeup - * pv_lock_slowpath - # of locking operations via the slowpath * pv_lock_stealing - # of lock stealing operations * pv_spurious_wakeup - # of spurious wakeups in non-head vCPUs * pv_wait_again - # of wait's after a queue head vCPU kick * pv_wait_early - # of early vCPU wait's * pv_wait_head - # of vCPU wait's at the queue head * pv_wait_node - # of vCPU wait's at a non-head queue node + * lock_pending - # of locking operations via pending code + * lock_slowpath - # of locking operations via MCS lock queue * * Writing to the "reset_counters" file will reset all the above counter * values. @@ -46,13 +47,14 @@ enum qlock_stats { qstat_pv_kick_wake, qstat_pv_latency_kick, qstat_pv_latency_wake, - qstat_pv_lock_slowpath, qstat_pv_lock_stealing, qstat_pv_spurious_wakeup, qstat_pv_wait_again, qstat_pv_wait_early, qstat_pv_wait_head, qstat_pv_wait_node, + qstat_lock_pending, + qstat_lock_slowpath, qstat_num, /* Total number of statistical counters */ qstat_reset_cnts = qstat_num, }; @@ -73,12 +75,13 @@ static const char * const qstat_names[qstat_num + 1] = { [qstat_pv_spurious_wakeup] = "pv_spurious_wakeup", [qstat_pv_latency_kick] = "pv_latency_kick", [qstat_pv_latency_wake] = "pv_latency_wake", - [qstat_pv_lock_slowpath] = "pv_lock_slowpath", [qstat_pv_lock_stealing] = "pv_lock_stealing", [qstat_pv_wait_again] = "pv_wait_again", [qstat_pv_wait_early] = "pv_wait_early", [qstat_pv_wait_head] = "pv_wait_head", [qstat_pv_wait_node] = "pv_wait_node", + [qstat_lock_pending] = "lock_pending", + [qstat_lock_slowpath] = "lock_slowpath", [qstat_reset_cnts] = "reset_counters", };