From patchwork Thu Oct 5 12:54:52 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 114932 Delivered-To: patch@linaro.org Received: by 10.80.163.170 with SMTP id s39csp193421edb; Thu, 5 Oct 2017 05:55:03 -0700 (PDT) X-Google-Smtp-Source: AOwi7QBWcptXFjS2TUKDS0v7HuWVFW3a95doS19gql5Bxu8LaQNs3izgwRXqIXP9nKGjB0gEmwIO X-Received: by 10.84.216.26 with SMTP id m26mr10089193pli.26.1507208102936; Thu, 05 Oct 2017 05:55:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507208102; cv=none; d=google.com; s=arc-20160816; b=rkQRbeBxqCQxndSIbZampTXfH+V5HZg8Ac/5TYOoCprkoLl5h8sAsH0w/Gw2ThHtSV bPP7b7DiwSAR+rB/OB+D5iHq5YJWalAK17+T06wiC0qdVmXCBV2Vmo0k5PLvZX4f10Oo M6Fak8DGfxTYz6e4qgI9+hBO9b4/hn0Gz8dXzn0QGBagy2SBPXF3T8yzlcHPwW1jgTp9 9L95kouI0aJKF5oX80GuQfOnPOaVLzovSLK0fpYLRB603SXJv7I7H6L9NHgNmBScT+3w t/qZurtfoUyZkwpAii+x/pUsWWo0z9LHQkgoJDlYNjaPLwZLTaV50qojiKm7xG1ou6qM Lz6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=w7TWWYNyuCNcNGH1mif+F9tnZIJtaXKDG3iGHaIC/Rg=; b=DOetdMaF8BdF7+rPPM7LHf3y/8jVTrZTM0X/sd4g+mlddBECz6FzUgmYTgZuTy61o1 BUS+1aqLtFNaKzLUL3QRKOM7WbLvlhIelOjY+eTijREWKEG3qinUplP1U6V8lVzHa4LI dhQQkVZLA9q53oiHIT0Ri5Nwb3F9EIVi+1vpNynpUHGDyz5pzaIvh+u7RJfFZ99g5w7q rNSK/i9UwvbsVBIpnZFfOB/tc2/iz8RjfiSMUHmWPGyvm8u8d4fvr3jqiFezbP/FiNQZ XXn5J5I4l44jX71t8w+4w1bhBft9LPxEO7Qq4vd6t4R8E1rh2AjzoXGEQnXZpBOiEjsn VXvQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 3si13927964pls.167.2017.10.05.05.55.02; Thu, 05 Oct 2017 05:55:02 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751515AbdJEMzA (ORCPT + 26 others); Thu, 5 Oct 2017 08:55:00 -0400 Received: from foss.arm.com ([217.140.101.70]:44980 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751317AbdJEMy5 (ORCPT ); Thu, 5 Oct 2017 08:54:57 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 98BA01596; Thu, 5 Oct 2017 05:54:57 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 6A9CC3F59E; Thu, 5 Oct 2017 05:54:57 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id B29D21AE0EF7; Thu, 5 Oct 2017 13:54:58 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Jeremy.Linton@arm.com, peterz@infradead.org, mingo@redhat.com, longman@redhat.com, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, Will Deacon Subject: [PATCH 1/6] kernel/locking: Use struct qrwlock instead of struct __qrwlock Date: Thu, 5 Oct 2017 13:54:52 +0100 Message-Id: <1507208097-825-2-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1507208097-825-1-git-send-email-will.deacon@arm.com> References: <1507208097-825-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org There's no good reason to keep the internal structure of struct qrwlock hidden from qrwlock.h, particularly as it's actually needed for unlock and ends up being abstracted independently behind the __qrwlock_write_byte function. Stop pretending we can hide this stuff, and move the __qrwlock definition into qrwlock, removing the __qrwlock_write_byte nastiness and using the same struct definition everywhere instead. Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Signed-off-by: Will Deacon --- include/asm-generic/qrwlock.h | 12 +----------- include/asm-generic/qrwlock_types.h | 15 +++++++++++++-- kernel/locking/qrwlock.c | 26 ++------------------------ 3 files changed, 16 insertions(+), 37 deletions(-) -- 2.1.4 diff --git a/include/asm-generic/qrwlock.h b/include/asm-generic/qrwlock.h index 50925327b0a8..02c0a768e6b0 100644 --- a/include/asm-generic/qrwlock.h +++ b/include/asm-generic/qrwlock.h @@ -129,22 +129,12 @@ static inline void queued_read_unlock(struct qrwlock *lock) } /** - * __qrwlock_write_byte - retrieve the write byte address of a queue rwlock - * @lock : Pointer to queue rwlock structure - * Return: the write byte address of a queue rwlock - */ -static inline u8 *__qrwlock_write_byte(struct qrwlock *lock) -{ - return (u8 *)lock + 3 * IS_BUILTIN(CONFIG_CPU_BIG_ENDIAN); -} - -/** * queued_write_unlock - release write lock of a queue rwlock * @lock : Pointer to queue rwlock structure */ static inline void queued_write_unlock(struct qrwlock *lock) { - smp_store_release(__qrwlock_write_byte(lock), 0); + smp_store_release(&lock->wmode, 0); } /* diff --git a/include/asm-generic/qrwlock_types.h b/include/asm-generic/qrwlock_types.h index 0abc6b6062fb..507f2dc51bba 100644 --- a/include/asm-generic/qrwlock_types.h +++ b/include/asm-generic/qrwlock_types.h @@ -9,12 +9,23 @@ */ typedef struct qrwlock { - atomic_t cnts; + union { + atomic_t cnts; + struct { +#ifdef __LITTLE_ENDIAN + u8 wmode; /* Writer mode */ + u8 rcnts[3]; /* Reader counts */ +#else + u8 rcnts[3]; /* Reader counts */ + u8 wmode; /* Writer mode */ +#endif + }; + }; arch_spinlock_t wait_lock; } arch_rwlock_t; #define __ARCH_RW_LOCK_UNLOCKED { \ - .cnts = ATOMIC_INIT(0), \ + { .cnts = ATOMIC_INIT(0), }, \ .wait_lock = __ARCH_SPIN_LOCK_UNLOCKED, \ } diff --git a/kernel/locking/qrwlock.c b/kernel/locking/qrwlock.c index 2655f26ec882..1af791e37348 100644 --- a/kernel/locking/qrwlock.c +++ b/kernel/locking/qrwlock.c @@ -23,26 +23,6 @@ #include #include -/* - * This internal data structure is used for optimizing access to some of - * the subfields within the atomic_t cnts. - */ -struct __qrwlock { - union { - atomic_t cnts; - struct { -#ifdef __LITTLE_ENDIAN - u8 wmode; /* Writer mode */ - u8 rcnts[3]; /* Reader counts */ -#else - u8 rcnts[3]; /* Reader counts */ - u8 wmode; /* Writer mode */ -#endif - }; - }; - arch_spinlock_t lock; -}; - /** * rspin_until_writer_unlock - inc reader count & spin until writer is gone * @lock : Pointer to queue rwlock structure @@ -125,10 +105,8 @@ void queued_write_lock_slowpath(struct qrwlock *lock) * or wait for a previous writer to go away. */ for (;;) { - struct __qrwlock *l = (struct __qrwlock *)lock; - - if (!READ_ONCE(l->wmode) && - (cmpxchg_relaxed(&l->wmode, 0, _QW_WAITING) == 0)) + if (!READ_ONCE(lock->wmode) && + (cmpxchg_relaxed(&lock->wmode, 0, _QW_WAITING) == 0)) break; cpu_relax(); From patchwork Thu Oct 5 12:54:53 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 114937 Delivered-To: patch@linaro.org Received: by 10.80.163.170 with SMTP id s39csp194446edb; Thu, 5 Oct 2017 05:56:20 -0700 (PDT) X-Google-Smtp-Source: AOwi7QCD+G1Lco3+Hf67ZpBeTrlZf2nzQdl3MSZpihCD1cRx1UVLkT8rB9A9A9Gbin+R1DKucnd0 X-Received: by 10.99.176.14 with SMTP id h14mr21482833pgf.358.1507208180523; Thu, 05 Oct 2017 05:56:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507208180; cv=none; d=google.com; s=arc-20160816; b=bKucdNhTCE7FYPiv5cUbJCp8Qwg+SFczvm0ydeWsLxcN/VKfktsKxXvkwOfAOS9ILW hdHNB4AhqHvIcA11JgkFZhjtmTUhsUW69tlopQixX4wlbKVUeZnKjTgB31w8aDrltM0y Uw0NHfxDVr61t6bSyq1W8ItjyXgBdQYGlmr5LoRqyCDOlqVh1rBc+hWUuVXAaCyLAOss UUkiBfbp6qXQBJs6vxtaj5LjC6PPlwSLcZYc66bveye+n19MQColuRTh4NG/OsVpBk56 I40Cn+vVGkyEhIOyYTb11ObrQK2SlWp/eGZkK85rFfYJe6q94bpGJ6qVGBKJo3mnYFZT JEbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=Lsbu1jjDUivTkhhOKC6THwxDtyqrfTUgDs8peL1zrsc=; b=BQuvjM4CDAt4fOd4Ge/ciUpvvElTL+vA/u0M85KpLh3RM2dS+KDAXMGqI11ECOJFqB 41h7f1Ugh+O26P9CfKuJMYGBSnPfKhqsSDeKdbqBJ2ONaNYUk4P4z0gIeQoUSBD7LhnK jcm6V4zc+I/X0v7aexKTTEswwQ5tPIhM1DfoWRh1/J00gjlOA2q14Sqv0jnn8sJDE3++ wjzYKC627WfNF5woy3fJKFIHvEqjqgiPo38fa+JQmhO4iPldNh+H0dnQU7YrxhzbBh0w pD6K7tvqjJ68mky3ASAoKqfDk8E8mejKZEbTrU0KDHlIjB+Bng1JFnyRcLWcxVkNtVm/ 4vNg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q14si13757542pll.260.2017.10.05.05.56.20; Thu, 05 Oct 2017 05:56:20 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751917AbdJEM4R (ORCPT + 26 others); Thu, 5 Oct 2017 08:56:17 -0400 Received: from foss.arm.com ([217.140.101.70]:44982 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751318AbdJEMy5 (ORCPT ); Thu, 5 Oct 2017 08:54:57 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A771915AD; Thu, 5 Oct 2017 05:54:57 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 78BB53F5CA; Thu, 5 Oct 2017 05:54:57 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id BF08B1AE2E09; Thu, 5 Oct 2017 13:54:58 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Jeremy.Linton@arm.com, peterz@infradead.org, mingo@redhat.com, longman@redhat.com, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, Will Deacon Subject: [PATCH 2/6] locking/atomic: Add atomic_cond_read_acquire Date: Thu, 5 Oct 2017 13:54:53 +0100 Message-Id: <1507208097-825-3-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1507208097-825-1-git-send-email-will.deacon@arm.com> References: <1507208097-825-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org smp_cond_load_acquire provides a way to spin on a variable with acquire semantics until some conditional expression involing the variable is satisfied. Architectures such as arm64 can potentially enter a low-power state, waking up only when the value of the variable changes, which reduces the system impact of tight polling loops. This patch makes the same interface available to users of atomic_t, atomic64_t and atomic_long_t, rather than require messy accesses to the structure internals. Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Signed-off-by: Will Deacon --- include/asm-generic/atomic-long.h | 3 +++ include/linux/atomic.h | 4 ++++ 2 files changed, 7 insertions(+) -- 2.1.4 diff --git a/include/asm-generic/atomic-long.h b/include/asm-generic/atomic-long.h index 288cc9e96395..f2d97b782031 100644 --- a/include/asm-generic/atomic-long.h +++ b/include/asm-generic/atomic-long.h @@ -243,4 +243,7 @@ static inline long atomic_long_add_unless(atomic_long_t *l, long a, long u) #define atomic_long_inc_not_zero(l) \ ATOMIC_LONG_PFX(_inc_not_zero)((ATOMIC_LONG_PFX(_t) *)(l)) +#define atomic_long_cond_read_acquire(v, c) \ + ATOMIC_LONG_PFX(_cond_read_acquire)((ATOMIC_LONG_PFX(_t) *)(v), (c)) + #endif /* _ASM_GENERIC_ATOMIC_LONG_H */ diff --git a/include/linux/atomic.h b/include/linux/atomic.h index 40d6bfec0e0d..0aeb2b3f4578 100644 --- a/include/linux/atomic.h +++ b/include/linux/atomic.h @@ -653,6 +653,8 @@ static inline int atomic_dec_if_positive(atomic_t *v) } #endif +#define atomic_cond_read_acquire(v, c) smp_cond_load_acquire(&(v)->counter, (c)) + #ifdef CONFIG_GENERIC_ATOMIC64 #include #endif @@ -1072,6 +1074,8 @@ static inline long long atomic64_fetch_andnot_release(long long i, atomic64_t *v } #endif +#define atomic64_cond_read_acquire(v, c) smp_cond_load_acquire(&(v)->counter, (c)) + #include #endif /* _LINUX_ATOMIC_H */ From patchwork Thu Oct 5 12:54:54 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 114938 Delivered-To: patch@linaro.org Received: by 10.80.163.170 with SMTP id s39csp194637edb; Thu, 5 Oct 2017 05:56:33 -0700 (PDT) X-Google-Smtp-Source: AOwi7QB2kFhBGFn9RL+/GdVVG/tiydil3F+RGyPau1jEf7VLzArY8zCryWLTZWJP97YbS6e2LQ73 X-Received: by 10.99.177.74 with SMTP id g10mr21568114pgp.326.1507208192949; Thu, 05 Oct 2017 05:56:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507208192; cv=none; d=google.com; s=arc-20160816; b=l/qQFADfIl/Q67pJ3TtyxwyDGV+kPf4qyyYbTJmeUGbFJM8BMGfoPWsJ2lRPAN7ez0 9jp3UpusLF1VLmYl4/TDqgg7wARigTNba1gn6tVWJumAE7zSJ9w4sZ8egEwBZLSRa7D9 f4Pmd4p4GreMi74tI7eKvJbUKE2i7kjtfdXOd9TwdU+DZI++Sqgm9BtrIEGnlu+dtTSF PnwNJlfB0EBe2LkDmEmTZXzeBE4soyEHhEnA3mjIlkmwpLcu0zVUqrr2hZxQObwInUim znat1fszn5SkvYsO+eV58LAuQ43HoBm5nlTa3806AQDgE6UiNfkAd3E4M0Bfv50Ip8VJ UsMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=Z10243CUOT6we7NbHVCeLszLWlJSMZdsQMovmOORf9M=; b=VUO4vJDDbZaOW7Oskg6dDjKjqL0oaECxTEmmfma8FY4LZL7SLxwryzSuL3cC7NhvWi McADRGsie4OemCye6ysu95NTJixtr8a6i05BqEpq1T6lZ5rAZaFxqWWvxmOOfyTYahNZ Z/Yuloatw8zzNnsIPKxtLp1SuSNMzH0WX69eYveAxYiz4sArTSqe0i/wK6n9+5BMOFiL HqNSyAltYf77grzh88wogL0Sufq8jBLZ3vXd7GjnOh4yIDY2kQ4ZslikFM09yaXNC0Pu ZgQlDtzFbl2ZtPO3BHUDcFM2HD8U6Xn+Pgprl91EppPtw8fHW3IcWc7LdMw4m4gBWp69 7ysA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q14si13757542pll.260.2017.10.05.05.56.32; Thu, 05 Oct 2017 05:56:32 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751888AbdJEM4R (ORCPT + 26 others); Thu, 5 Oct 2017 08:56:17 -0400 Received: from foss.arm.com ([217.140.101.70]:45000 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751357AbdJEMy6 (ORCPT ); Thu, 5 Oct 2017 08:54:58 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B4E4D15BE; Thu, 5 Oct 2017 05:54:57 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 86D573F7C6; Thu, 5 Oct 2017 05:54:57 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id CD4FF1AE2E15; Thu, 5 Oct 2017 13:54:58 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Jeremy.Linton@arm.com, peterz@infradead.org, mingo@redhat.com, longman@redhat.com, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, Will Deacon Subject: [PATCH 3/6] kernel/locking: Use atomic_cond_read_acquire when spinning in qrwlock Date: Thu, 5 Oct 2017 13:54:54 +0100 Message-Id: <1507208097-825-4-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1507208097-825-1-git-send-email-will.deacon@arm.com> References: <1507208097-825-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The qrwlock slowpaths involve spinning when either a prospective reader is waiting for a concurrent writer to drain, or a prospective writer is waiting for concurrent readers to drain. In both of these situations, atomic_cond_read_acquire can be used to avoid busy-waiting and make use of any backoff functionality provided by the architecture. This patch replaces the open-code loops and rspin_until_writer_unlock implementation with atomic_cond_read_acquire. The write mode transition zero to _QW_WAITING is left alone, since (a) this doesn't need acquire semantics and (b) should be fast. Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Signed-off-by: Will Deacon --- kernel/locking/qrwlock.c | 47 +++++++++++------------------------------------ 1 file changed, 11 insertions(+), 36 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qrwlock.c b/kernel/locking/qrwlock.c index 1af791e37348..b7ea4647c74d 100644 --- a/kernel/locking/qrwlock.c +++ b/kernel/locking/qrwlock.c @@ -24,23 +24,6 @@ #include /** - * rspin_until_writer_unlock - inc reader count & spin until writer is gone - * @lock : Pointer to queue rwlock structure - * @writer: Current queue rwlock writer status byte - * - * In interrupt context or at the head of the queue, the reader will just - * increment the reader count & wait until the writer releases the lock. - */ -static __always_inline void -rspin_until_writer_unlock(struct qrwlock *lock, u32 cnts) -{ - while ((cnts & _QW_WMASK) == _QW_LOCKED) { - cpu_relax(); - cnts = atomic_read_acquire(&lock->cnts); - } -} - -/** * queued_read_lock_slowpath - acquire read lock of a queue rwlock * @lock: Pointer to queue rwlock structure * @cnts: Current qrwlock lock value @@ -53,13 +36,12 @@ void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts) if (unlikely(in_interrupt())) { /* * Readers in interrupt context will get the lock immediately - * if the writer is just waiting (not holding the lock yet). - * The rspin_until_writer_unlock() function returns immediately - * in this case. Otherwise, they will spin (with ACQUIRE - * semantics) until the lock is available without waiting in - * the queue. + * if the writer is just waiting (not holding the lock yet), + * so spin with ACQUIRE semantics until the lock is available + * without waiting in the queue. */ - rspin_until_writer_unlock(lock, cnts); + atomic_cond_read_acquire(&lock->cnts, (VAL & _QW_WMASK) + != _QW_LOCKED); return; } atomic_sub(_QR_BIAS, &lock->cnts); @@ -68,14 +50,14 @@ void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts) * Put the reader into the wait queue */ arch_spin_lock(&lock->wait_lock); + atomic_add(_QR_BIAS, &lock->cnts); /* * The ACQUIRE semantics of the following spinning code ensure * that accesses can't leak upwards out of our subsequent critical * section in the case that the lock is currently held for write. */ - cnts = atomic_fetch_add_acquire(_QR_BIAS, &lock->cnts); - rspin_until_writer_unlock(lock, cnts); + atomic_cond_read_acquire(&lock->cnts, (VAL & _QW_WMASK) != _QW_LOCKED); /* * Signal the next one in queue to become queue head @@ -90,8 +72,6 @@ EXPORT_SYMBOL(queued_read_lock_slowpath); */ void queued_write_lock_slowpath(struct qrwlock *lock) { - u32 cnts; - /* Put the writer into the wait queue */ arch_spin_lock(&lock->wait_lock); @@ -113,15 +93,10 @@ void queued_write_lock_slowpath(struct qrwlock *lock) } /* When no more readers, set the locked flag */ - for (;;) { - cnts = atomic_read(&lock->cnts); - if ((cnts == _QW_WAITING) && - (atomic_cmpxchg_acquire(&lock->cnts, _QW_WAITING, - _QW_LOCKED) == _QW_WAITING)) - break; - - cpu_relax(); - } + do { + atomic_cond_read_acquire(&lock->cnts, VAL == _QW_WAITING); + } while (atomic_cmpxchg_relaxed(&lock->cnts, _QW_WAITING, + _QW_LOCKED) != _QW_WAITING); unlock: arch_spin_unlock(&lock->wait_lock); } From patchwork Thu Oct 5 12:54:55 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 114934 Delivered-To: patch@linaro.org Received: by 10.80.163.170 with SMTP id s39csp193736edb; Thu, 5 Oct 2017 05:55:29 -0700 (PDT) X-Google-Smtp-Source: AOwi7QAmuYzl3yaCCJMBCb4zZtioaSmgosWePn5R5uxO3C935Sdcd7C8VZ3/nnl7qV0E0+xva8v3 X-Received: by 10.99.3.213 with SMTP id 204mr12896469pgd.407.1507208129272; Thu, 05 Oct 2017 05:55:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507208129; cv=none; d=google.com; s=arc-20160816; b=Dkgy3fUj9KP2WMtlaK98jvTwEHMgz3tLkhXwr5QA3kpfdzr+gZtLyY5n1CGD63gHu9 NfvpXycSSLTirYuW1CdmfuM7M/CVnpgPtv9tkItvn8+YF1AsBtGAt/uk2TrnBLfyIZef N7MavzO/J+Bcv5GBduC2CQzZ9ryglTlhlWIUPzkwy+KoNMeR+5tVtBnbdo8ca4+ys1Gw tImtr+SFdFEQCyCgEHeLQKtwNmepln1XiC7dOkNk8JZtmG24e4tyX+qm/z2RbDVDewpn 6PO5awx8KDU3GU959GUJ6jid5oMq31pJlUlaB39HscVzFpl8xgtiL+u0tajasdyGvK0s Z2JQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=Bby+xnszLGR7+D52U2zXV/aPXMAujDDwFfOHl4tvid0=; b=nNl7R9eXMzVahXkkMkLoWJCUn4A00ohwyuIzymnoi/XBVq+vjofPsAF4xmttUgY790 gOolxZP83/kY2X3CHYY0bQc3Bh2hcXmUhawwZgEW1QSaR2LwSFpyJ7kDnaa0wpeS5JxF gU0FwOmQjXxk9h2QokDU3LkTnXDiZ80+giX6AtD9V7vD9YIdc4HNY3kDoPwl5M0uL6ms vjvnlJiW0+rjeqY76Kv+0buaW3ViEeX5yJKDEPd7dC6SGTtBRqGOQgBwIktuaqn08oAG Mnil/TqvG16X79n1ipPeqjo3IM5xLW9zvZz2+mrby/fmxUq8eoLIEwEcW8/+aotQ/nYM ruOQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id o8si2568411pgd.251.2017.10.05.05.55.29; Thu, 05 Oct 2017 05:55:29 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751717AbdJEMzZ (ORCPT + 26 others); Thu, 5 Oct 2017 08:55:25 -0400 Received: from foss.arm.com ([217.140.101.70]:45026 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751366AbdJEMy6 (ORCPT ); Thu, 5 Oct 2017 08:54:58 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C55ED15BF; Thu, 5 Oct 2017 05:54:57 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 975563F483; Thu, 5 Oct 2017 05:54:57 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id DBF481AE2F39; Thu, 5 Oct 2017 13:54:58 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Jeremy.Linton@arm.com, peterz@infradead.org, mingo@redhat.com, longman@redhat.com, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, Will Deacon Subject: [PATCH 4/6] arm64: locking: Move rwlock implementation over to qrwlocks Date: Thu, 5 Oct 2017 13:54:55 +0100 Message-Id: <1507208097-825-5-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1507208097-825-1-git-send-email-will.deacon@arm.com> References: <1507208097-825-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Now that the qrwlock can make use of WFE, remove our homebrew rwlock code in favour of the generic queued implementation. Signed-off-by: Will Deacon --- arch/arm64/Kconfig | 17 ++++ arch/arm64/include/asm/Kbuild | 1 + arch/arm64/include/asm/spinlock.h | 164 +------------------------------- arch/arm64/include/asm/spinlock_types.h | 6 +- 4 files changed, 20 insertions(+), 168 deletions(-) -- 2.1.4 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 0df64a6a56d4..6d32c9b0d4bb 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -22,7 +22,24 @@ config ARM64 select ARCH_HAS_STRICT_MODULE_RWX select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST select ARCH_HAVE_NMI_SAFE_CMPXCHG if ACPI_APEI_SEA + select ARCH_INLINE_READ_LOCK if !PREEMPT + select ARCH_INLINE_READ_LOCK_BH if !PREEMPT + select ARCH_INLINE_READ_LOCK_IRQ if !PREEMPT + select ARCH_INLINE_READ_LOCK_IRQSAVE if !PREEMPT + select ARCH_INLINE_READ_UNLOCK if !PREEMPT + select ARCH_INLINE_READ_UNLOCK_BH if !PREEMPT + select ARCH_INLINE_READ_UNLOCK_IRQ if !PREEMPT + select ARCH_INLINE_READ_UNLOCK_IRQSAVE if !PREEMPT + select ARCH_INLINE_WRITE_LOCK if !PREEMPT + select ARCH_INLINE_WRITE_LOCK_BH if !PREEMPT + select ARCH_INLINE_WRITE_LOCK_IRQ if !PREEMPT + select ARCH_INLINE_WRITE_LOCK_IRQSAVE if !PREEMPT + select ARCH_INLINE_WRITE_UNLOCK if !PREEMPT + select ARCH_INLINE_WRITE_UNLOCK_BH if !PREEMPT + select ARCH_INLINE_WRITE_UNLOCK_IRQ if !PREEMPT + select ARCH_INLINE_WRITE_UNLOCK_IRQSAVE if !PREEMPT select ARCH_USE_CMPXCHG_LOCKREF + select ARCH_USE_QUEUED_RWLOCKS select ARCH_SUPPORTS_MEMORY_FAILURE select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_NUMA_BALANCING diff --git a/arch/arm64/include/asm/Kbuild b/arch/arm64/include/asm/Kbuild index 2326e39d5892..e63d0a8312de 100644 --- a/arch/arm64/include/asm/Kbuild +++ b/arch/arm64/include/asm/Kbuild @@ -16,6 +16,7 @@ generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += msi.h generic-y += preempt.h +generic-y += qrwlock.h generic-y += rwsem.h generic-y += segment.h generic-y += serial.h diff --git a/arch/arm64/include/asm/spinlock.h b/arch/arm64/include/asm/spinlock.h index aa51a38e46e4..fdb827c7832f 100644 --- a/arch/arm64/include/asm/spinlock.h +++ b/arch/arm64/include/asm/spinlock.h @@ -137,169 +137,7 @@ static inline int arch_spin_is_contended(arch_spinlock_t *lock) } #define arch_spin_is_contended arch_spin_is_contended -/* - * Write lock implementation. - * - * Write locks set bit 31. Unlocking, is done by writing 0 since the lock is - * exclusively held. - * - * The memory barriers are implicit with the load-acquire and store-release - * instructions. - */ - -static inline void arch_write_lock(arch_rwlock_t *rw) -{ - unsigned int tmp; - - asm volatile(ARM64_LSE_ATOMIC_INSN( - /* LL/SC */ - " sevl\n" - "1: wfe\n" - "2: ldaxr %w0, %1\n" - " cbnz %w0, 1b\n" - " stxr %w0, %w2, %1\n" - " cbnz %w0, 2b\n" - __nops(1), - /* LSE atomics */ - "1: mov %w0, wzr\n" - "2: casa %w0, %w2, %1\n" - " cbz %w0, 3f\n" - " ldxr %w0, %1\n" - " cbz %w0, 2b\n" - " wfe\n" - " b 1b\n" - "3:") - : "=&r" (tmp), "+Q" (rw->lock) - : "r" (0x80000000) - : "memory"); -} - -static inline int arch_write_trylock(arch_rwlock_t *rw) -{ - unsigned int tmp; - - asm volatile(ARM64_LSE_ATOMIC_INSN( - /* LL/SC */ - "1: ldaxr %w0, %1\n" - " cbnz %w0, 2f\n" - " stxr %w0, %w2, %1\n" - " cbnz %w0, 1b\n" - "2:", - /* LSE atomics */ - " mov %w0, wzr\n" - " casa %w0, %w2, %1\n" - __nops(2)) - : "=&r" (tmp), "+Q" (rw->lock) - : "r" (0x80000000) - : "memory"); - - return !tmp; -} - -static inline void arch_write_unlock(arch_rwlock_t *rw) -{ - asm volatile(ARM64_LSE_ATOMIC_INSN( - " stlr wzr, %0", - " swpl wzr, wzr, %0") - : "=Q" (rw->lock) :: "memory"); -} - -/* write_can_lock - would write_trylock() succeed? */ -#define arch_write_can_lock(x) ((x)->lock == 0) - -/* - * Read lock implementation. - * - * It exclusively loads the lock value, increments it and stores the new value - * back if positive and the CPU still exclusively owns the location. If the - * value is negative, the lock is already held. - * - * During unlocking there may be multiple active read locks but no write lock. - * - * The memory barriers are implicit with the load-acquire and store-release - * instructions. - * - * Note that in UNDEFINED cases, such as unlocking a lock twice, the LL/SC - * and LSE implementations may exhibit different behaviour (although this - * will have no effect on lockdep). - */ -static inline void arch_read_lock(arch_rwlock_t *rw) -{ - unsigned int tmp, tmp2; - - asm volatile( - " sevl\n" - ARM64_LSE_ATOMIC_INSN( - /* LL/SC */ - "1: wfe\n" - "2: ldaxr %w0, %2\n" - " add %w0, %w0, #1\n" - " tbnz %w0, #31, 1b\n" - " stxr %w1, %w0, %2\n" - " cbnz %w1, 2b\n" - __nops(1), - /* LSE atomics */ - "1: wfe\n" - "2: ldxr %w0, %2\n" - " adds %w1, %w0, #1\n" - " tbnz %w1, #31, 1b\n" - " casa %w0, %w1, %2\n" - " sbc %w0, %w1, %w0\n" - " cbnz %w0, 2b") - : "=&r" (tmp), "=&r" (tmp2), "+Q" (rw->lock) - : - : "cc", "memory"); -} - -static inline void arch_read_unlock(arch_rwlock_t *rw) -{ - unsigned int tmp, tmp2; - - asm volatile(ARM64_LSE_ATOMIC_INSN( - /* LL/SC */ - "1: ldxr %w0, %2\n" - " sub %w0, %w0, #1\n" - " stlxr %w1, %w0, %2\n" - " cbnz %w1, 1b", - /* LSE atomics */ - " movn %w0, #0\n" - " staddl %w0, %2\n" - __nops(2)) - : "=&r" (tmp), "=&r" (tmp2), "+Q" (rw->lock) - : - : "memory"); -} - -static inline int arch_read_trylock(arch_rwlock_t *rw) -{ - unsigned int tmp, tmp2; - - asm volatile(ARM64_LSE_ATOMIC_INSN( - /* LL/SC */ - " mov %w1, #1\n" - "1: ldaxr %w0, %2\n" - " add %w0, %w0, #1\n" - " tbnz %w0, #31, 2f\n" - " stxr %w1, %w0, %2\n" - " cbnz %w1, 1b\n" - "2:", - /* LSE atomics */ - " ldr %w0, %2\n" - " adds %w1, %w0, #1\n" - " tbnz %w1, #31, 1f\n" - " casa %w0, %w1, %2\n" - " sbc %w1, %w1, %w0\n" - __nops(1) - "1:") - : "=&r" (tmp), "=&r" (tmp2), "+Q" (rw->lock) - : - : "cc", "memory"); - - return !tmp2; -} - -/* read_can_lock - would read_trylock() succeed? */ -#define arch_read_can_lock(x) ((x)->lock < 0x80000000) +#include /* See include/linux/spinlock.h */ #define smp_mb__after_spinlock() smp_mb() diff --git a/arch/arm64/include/asm/spinlock_types.h b/arch/arm64/include/asm/spinlock_types.h index 55be59a35e3f..6b856012c51b 100644 --- a/arch/arm64/include/asm/spinlock_types.h +++ b/arch/arm64/include/asm/spinlock_types.h @@ -36,10 +36,6 @@ typedef struct { #define __ARCH_SPIN_LOCK_UNLOCKED { 0 , 0 } -typedef struct { - volatile unsigned int lock; -} arch_rwlock_t; - -#define __ARCH_RW_LOCK_UNLOCKED { 0 } +#include #endif From patchwork Thu Oct 5 12:54:56 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 114935 Delivered-To: patch@linaro.org Received: by 10.80.163.170 with SMTP id s39csp194221edb; Thu, 5 Oct 2017 05:56:03 -0700 (PDT) X-Google-Smtp-Source: AOwi7QBJwUnfReNv1t0LDoRYb0qTGMsvL7ev1jU/bgw01P5qJFm62aXh9Z8qH/HqaJIMF+/UpdlX X-Received: by 10.84.204.136 with SMTP id b8mr22571585ple.361.1507208163759; Thu, 05 Oct 2017 05:56:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507208163; cv=none; d=google.com; s=arc-20160816; b=dL50AHaJpmBdIblN/8fShsWXYt6lTXS0/0gVaZGERGRJJd/NjZVkg8DCJd6X2BXdV3 IuRw6VZrSIEUmnImtPp2rgwnQqyVnw/TWBXk7XB3U4ChiNJv/a3OKg1GmwG5wvqIhArJ ize9/tBlWlLMT3xhaKKZtRSlSvifqPKnm/kzqf8NUeX/oQSZ5c8zzmdQx06wgzvHdsIr pVoy9scGjX4/nz97V7rdYTIYvaZ1zF4q24OJnN7cZQmBvBtgvpcpEtkJYh8AbyZXCGt/ h0+VJ0olmxN6V6k9qYFRkf5R49iJcgkcLwH1VNWS7hxr0a7NmP+/QzkY/orimasqC/hk LoRQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=xL0PXOokHQYRwtvAEzIrntNGZ+jje0YZWEDBJNfdUDo=; b=sQVc3T6AKqztrHdBKKFj+wAJ+74PFqKJG3ZsLMyahAyZKm8FeU3GlSIBWjes6tPBSC w27AjP/zRixKTxD2fYMhzp5S7lE17SKUr18lgFygBt74rFTeO9P00tbwwjskEcCfdrbR rbff0aOG9MT5I5H1ngrNquFN+Ibl5+CZmVQWl6iyRp/BeW+G16YRJgv6HI7BVoOOZVkP krRc6ZpLimWXfFNkOv7bAwiiHOL5czx3FpjLApeCaBf/JxN6xk1Jg45p6d6x2rXcO2Vx mSZJdvf24ojYksl/T+vJPOCNUHnFvymQlID1+ePvd93WpjMXNVrf+jUiPaXIRPqwEWxT HrBA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i9si3855234plt.396.2017.10.05.05.55.51; Thu, 05 Oct 2017 05:56:03 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751659AbdJEMzY (ORCPT + 26 others); Thu, 5 Oct 2017 08:55:24 -0400 Received: from foss.arm.com ([217.140.101.70]:45048 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751415AbdJEMy6 (ORCPT ); Thu, 5 Oct 2017 08:54:58 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7E03A1650; Thu, 5 Oct 2017 05:54:58 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 501923F483; Thu, 5 Oct 2017 05:54:58 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id EC4021AE2F43; Thu, 5 Oct 2017 13:54:58 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Jeremy.Linton@arm.com, peterz@infradead.org, mingo@redhat.com, longman@redhat.com, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, Will Deacon Subject: [PATCH 5/6] kernel/locking: Prevent slowpath writers getting held up by fastpath Date: Thu, 5 Oct 2017 13:54:56 +0100 Message-Id: <1507208097-825-6-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1507208097-825-1-git-send-email-will.deacon@arm.com> References: <1507208097-825-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When a prospective writer takes the qrwlock locking slowpath due to the lock being held, it attempts to cmpxchg the wmode field from 0 to _QW_WAITING so that concurrent lockers also take the slowpath and queue on the spinlock accordingly, allowing the lockers to drain. Unfortunately, this isn't fair, because a fastpath writer that comes in after the lock is made available but before the _QW_WAITING flag is set can effectively jump the queue. If there is a steady stream of prospective writers, then the waiter will be held off indefinitely. This patch restores fairness by separating _QW_WAITING and _QW_LOCKED into two bits in the wmode byte and having the waiter set _QW_WAITING unconditionally. This then forces the slow-path for concurrent lockers, but requires that a writer unlock operation performs an atomic_sub_release instead of a store_release so that the waiting status is preserved. Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Signed-off-by: Will Deacon --- include/asm-generic/qrwlock.h | 4 ++-- kernel/locking/qrwlock.c | 20 +++++--------------- 2 files changed, 7 insertions(+), 17 deletions(-) -- 2.1.4 Acked-by: Waiman Long diff --git a/include/asm-generic/qrwlock.h b/include/asm-generic/qrwlock.h index 02c0a768e6b0..8b7edef500e5 100644 --- a/include/asm-generic/qrwlock.h +++ b/include/asm-generic/qrwlock.h @@ -41,7 +41,7 @@ * +----+----+----+----+ */ #define _QW_WAITING 1 /* A writer is waiting */ -#define _QW_LOCKED 0xff /* A writer holds the lock */ +#define _QW_LOCKED 2 /* A writer holds the lock */ #define _QW_WMASK 0xff /* Writer mask */ #define _QR_SHIFT 8 /* Reader count shift */ #define _QR_BIAS (1U << _QR_SHIFT) @@ -134,7 +134,7 @@ static inline void queued_read_unlock(struct qrwlock *lock) */ static inline void queued_write_unlock(struct qrwlock *lock) { - smp_store_release(&lock->wmode, 0); + (void)atomic_sub_return_release(_QW_LOCKED, &lock->cnts); } /* diff --git a/kernel/locking/qrwlock.c b/kernel/locking/qrwlock.c index b7ea4647c74d..e940f2c2b4f2 100644 --- a/kernel/locking/qrwlock.c +++ b/kernel/locking/qrwlock.c @@ -40,8 +40,7 @@ void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts) * so spin with ACQUIRE semantics until the lock is available * without waiting in the queue. */ - atomic_cond_read_acquire(&lock->cnts, (VAL & _QW_WMASK) - != _QW_LOCKED); + atomic_cond_read_acquire(&lock->cnts, !(VAL & _QW_LOCKED)); return; } atomic_sub(_QR_BIAS, &lock->cnts); @@ -57,7 +56,7 @@ void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts) * that accesses can't leak upwards out of our subsequent critical * section in the case that the lock is currently held for write. */ - atomic_cond_read_acquire(&lock->cnts, (VAL & _QW_WMASK) != _QW_LOCKED); + atomic_cond_read_acquire(&lock->cnts, !(VAL & _QW_LOCKED)); /* * Signal the next one in queue to become queue head @@ -80,19 +79,10 @@ void queued_write_lock_slowpath(struct qrwlock *lock) (atomic_cmpxchg_acquire(&lock->cnts, 0, _QW_LOCKED) == 0)) goto unlock; - /* - * Set the waiting flag to notify readers that a writer is pending, - * or wait for a previous writer to go away. - */ - for (;;) { - if (!READ_ONCE(lock->wmode) && - (cmpxchg_relaxed(&lock->wmode, 0, _QW_WAITING) == 0)) - break; - - cpu_relax(); - } + /* Set the waiting flag to notify readers that a writer is pending */ + atomic_add(_QW_WAITING, &lock->cnts); - /* When no more readers, set the locked flag */ + /* When no more readers or writers, set the locked flag */ do { atomic_cond_read_acquire(&lock->cnts, VAL == _QW_WAITING); } while (atomic_cmpxchg_relaxed(&lock->cnts, _QW_WAITING, From patchwork Thu Oct 5 12:54:57 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 114936 Delivered-To: patch@linaro.org Received: by 10.80.163.170 with SMTP id s39csp194227edb; Thu, 5 Oct 2017 05:56:04 -0700 (PDT) X-Google-Smtp-Source: AOwi7QBYBU0PUrS9f2G7+Dj9+/augQY7pvpfv0u5lHewiV1nD2/9MeI4g/QV6wQ1jh/RipGbhQaN X-Received: by 10.159.233.133 with SMTP id bh5mr22004191plb.28.1507208164214; Thu, 05 Oct 2017 05:56:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507208164; cv=none; d=google.com; s=arc-20160816; b=dA7nWnHxjz7Ly8OktGO1gLh2JLF++B7Bob93nacjyvdNcEDS36NJwd9ETjmhYsRU1E duja9mflRCFo6byV29IlHIJH9d94OziJmHhzngdQYFZ8fQ8kD0uHa5iWsblYOqUVeI0A EOhgaNcdPjUXzIab/40PYMJSToGkJuuqgSR075+/lnFpQUETZmbyEBEf2j2Fsn4Tayry 0CgRamglUEsj6PsBgVv0Cl0JclGxktBNWSu59d31bE28MJuhGrKtaV3zC42kTArZ+eIy 0udZxhhUxlUFPBp1nHHeRukYzsl8jc7bf5ekzbYY1v7a9cUaROz+HtYCLs715BSQ+dFt f6sA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=hZHKDKlS4yGLvU7mZr4fOasa9F0cTxybm/zHh07l518=; b=xqR90QxGokiWF+nrgtzp39W5xd2um3egjqEDjNyNQNWtSKlwcjUVPcWpWLdpn1KhEJ hwvjS8zA6ngGz4nmVEgqSD4GlWoYBCayhuIc0coMk4kDhzLV8HHpH/HrULm9NNktzZXZ bgW5XgbkmSGpOfawEnakiqc+RiMHfkeFGv69RezSD2iuBI6IjVyOB754h9H9gf4iACtW /2hIkDbjuSwprhUIiqYstlcGc+m2WkKHgsq69Tx/MO5UclFgn8l/3SEbNQwGhO4vrpWy mZ571nrd37px87iwyP7ITUhxWFPy8SyrhapTXELx4z2oeeVBsGuXgqi+zug5SzxDkA2W 4EWA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i9si3855234plt.396.2017.10.05.05.56.03; Thu, 05 Oct 2017 05:56:04 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751587AbdJEMzX (ORCPT + 26 others); Thu, 5 Oct 2017 08:55:23 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:45052 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751419AbdJEMy6 (ORCPT ); Thu, 5 Oct 2017 08:54:58 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 87F75165C; Thu, 5 Oct 2017 05:54:58 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 590B93F59E; Thu, 5 Oct 2017 05:54:58 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 086641AE2F5E; Thu, 5 Oct 2017 13:54:59 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Jeremy.Linton@arm.com, peterz@infradead.org, mingo@redhat.com, longman@redhat.com, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, Will Deacon Subject: [PATCH 6/6] kernel/locking: Remove unused union members from struct qrwlock Date: Thu, 5 Oct 2017 13:54:57 +0100 Message-Id: <1507208097-825-7-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1507208097-825-1-git-send-email-will.deacon@arm.com> References: <1507208097-825-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Now that all atomic operations are performed on the full lock word of the qrwlock, there's no need to define a union type exposing the reader and writer subcomponents. Remove them. Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Signed-off-by: Will Deacon --- include/asm-generic/qrwlock_types.h | 15 ++------------- 1 file changed, 2 insertions(+), 13 deletions(-) -- 2.1.4 diff --git a/include/asm-generic/qrwlock_types.h b/include/asm-generic/qrwlock_types.h index 507f2dc51bba..7f53be31359c 100644 --- a/include/asm-generic/qrwlock_types.h +++ b/include/asm-generic/qrwlock_types.h @@ -9,23 +9,12 @@ */ typedef struct qrwlock { - union { - atomic_t cnts; - struct { -#ifdef __LITTLE_ENDIAN - u8 wmode; /* Writer mode */ - u8 rcnts[3]; /* Reader counts */ -#else - u8 rcnts[3]; /* Reader counts */ - u8 wmode; /* Writer mode */ -#endif - }; - }; + atomic_t cnts; arch_spinlock_t wait_lock; } arch_rwlock_t; #define __ARCH_RW_LOCK_UNLOCKED { \ - { .cnts = ATOMIC_INIT(0), }, \ + .cnts = ATOMIC_INIT(0), \ .wait_lock = __ARCH_SPIN_LOCK_UNLOCKED, \ }