From patchwork Thu Oct 12 12:20:47 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 115612 Delivered-To: patch@linaro.org Received: by 10.140.22.163 with SMTP id 32csp1875367qgn; Thu, 12 Oct 2017 05:21:10 -0700 (PDT) X-Google-Smtp-Source: AOwi7QAv4Q/0BXwvFW2sw+RwdualHcYH/3GWWosaIwU8k/ghJAtTx+ehkJuHdPFkIYYujJeSRB74 X-Received: by 10.99.164.81 with SMTP id c17mr70041pgp.445.1507810869901; Thu, 12 Oct 2017 05:21:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507810869; cv=none; d=google.com; s=arc-20160816; b=cWtTv8zhL0d4e9ivj7e0sPZnva+DHnGkFXD+s+OrCeaaY/3P94yTXN+sfg0kbd27my 4HCVhr5/KhQd9rN9n5iiLdS+U9PbGb7Z2ELDcyhxRpLvw5xGApU3XZsb901v9cmjAV2r tPawjIfHVyvmNpiLMC3DAInbKb/XNBPXPM94qx3aMplg0hqs7XxSjna1q9gRgZfmDt4m R/7tkozPIsWK77Z4KIX1Mwu7TB5UwQkmedV3+8Cv3XTmfmU+wRpGdLpSmT5gfshgGQmd AjVcxIwVdm15JFnco4hBk0h6WF0Tr2Jo1GO7SZp9MTt2V9atcjWnnmCNpVNiakMtOzHn PIhw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=w7TWWYNyuCNcNGH1mif+F9tnZIJtaXKDG3iGHaIC/Rg=; b=Kent5uaoLXR0ff7mRrfeiSMOSD2dU1fTULzszAs8woSwcDF6xCw1Gll+XYY0pTm/tX +2DXOXbqUoXWiBlYVOL6J42Q+HHeiObXvdBhy4pCaPNArZD2G8f8o3L6Sy9QiOXp1782 bxK68xUY0J25jDVOytmx+ixiiH+bd8IUPsqrce08V00sV35QrEJdO5HLoEoYETcgPNW5 k8faPVnokDgNFqyhjylsWw/CpHMmvISKpCVMPO7Uu8gDPQHJLy33BQ57lhsVsm0Rb+rp oIW7ght7G4f6ABcQpvMLPU4vUFl2wCJuIfrNarxNtToHAQ2HTJQVz9BvPs9ugO54cuMl hhXg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id r145si6892595pgr.753.2017.10.12.05.21.09; Thu, 12 Oct 2017 05:21:09 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757419AbdJLMUz (ORCPT + 27 others); Thu, 12 Oct 2017 08:20:55 -0400 Received: from foss.arm.com ([217.140.101.70]:45514 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757008AbdJLMUt (ORCPT ); Thu, 12 Oct 2017 08:20:49 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9647A15BE; Thu, 12 Oct 2017 05:20:48 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 682DB3F59C; Thu, 12 Oct 2017 05:20:48 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 184241AE1952; Thu, 12 Oct 2017 13:20:52 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Jeremy.Linton@arm.com, peterz@infradead.org, mingo@redhat.com, longman@redhat.com, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, Will Deacon Subject: [PATCH v3 1/5] kernel/locking: Use struct qrwlock instead of struct __qrwlock Date: Thu, 12 Oct 2017 13:20:47 +0100 Message-Id: <1507810851-306-2-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1507810851-306-1-git-send-email-will.deacon@arm.com> References: <1507810851-306-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org There's no good reason to keep the internal structure of struct qrwlock hidden from qrwlock.h, particularly as it's actually needed for unlock and ends up being abstracted independently behind the __qrwlock_write_byte function. Stop pretending we can hide this stuff, and move the __qrwlock definition into qrwlock, removing the __qrwlock_write_byte nastiness and using the same struct definition everywhere instead. Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Signed-off-by: Will Deacon --- include/asm-generic/qrwlock.h | 12 +----------- include/asm-generic/qrwlock_types.h | 15 +++++++++++++-- kernel/locking/qrwlock.c | 26 ++------------------------ 3 files changed, 16 insertions(+), 37 deletions(-) -- 2.1.4 diff --git a/include/asm-generic/qrwlock.h b/include/asm-generic/qrwlock.h index 50925327b0a8..02c0a768e6b0 100644 --- a/include/asm-generic/qrwlock.h +++ b/include/asm-generic/qrwlock.h @@ -129,22 +129,12 @@ static inline void queued_read_unlock(struct qrwlock *lock) } /** - * __qrwlock_write_byte - retrieve the write byte address of a queue rwlock - * @lock : Pointer to queue rwlock structure - * Return: the write byte address of a queue rwlock - */ -static inline u8 *__qrwlock_write_byte(struct qrwlock *lock) -{ - return (u8 *)lock + 3 * IS_BUILTIN(CONFIG_CPU_BIG_ENDIAN); -} - -/** * queued_write_unlock - release write lock of a queue rwlock * @lock : Pointer to queue rwlock structure */ static inline void queued_write_unlock(struct qrwlock *lock) { - smp_store_release(__qrwlock_write_byte(lock), 0); + smp_store_release(&lock->wmode, 0); } /* diff --git a/include/asm-generic/qrwlock_types.h b/include/asm-generic/qrwlock_types.h index 0abc6b6062fb..507f2dc51bba 100644 --- a/include/asm-generic/qrwlock_types.h +++ b/include/asm-generic/qrwlock_types.h @@ -9,12 +9,23 @@ */ typedef struct qrwlock { - atomic_t cnts; + union { + atomic_t cnts; + struct { +#ifdef __LITTLE_ENDIAN + u8 wmode; /* Writer mode */ + u8 rcnts[3]; /* Reader counts */ +#else + u8 rcnts[3]; /* Reader counts */ + u8 wmode; /* Writer mode */ +#endif + }; + }; arch_spinlock_t wait_lock; } arch_rwlock_t; #define __ARCH_RW_LOCK_UNLOCKED { \ - .cnts = ATOMIC_INIT(0), \ + { .cnts = ATOMIC_INIT(0), }, \ .wait_lock = __ARCH_SPIN_LOCK_UNLOCKED, \ } diff --git a/kernel/locking/qrwlock.c b/kernel/locking/qrwlock.c index 2655f26ec882..1af791e37348 100644 --- a/kernel/locking/qrwlock.c +++ b/kernel/locking/qrwlock.c @@ -23,26 +23,6 @@ #include #include -/* - * This internal data structure is used for optimizing access to some of - * the subfields within the atomic_t cnts. - */ -struct __qrwlock { - union { - atomic_t cnts; - struct { -#ifdef __LITTLE_ENDIAN - u8 wmode; /* Writer mode */ - u8 rcnts[3]; /* Reader counts */ -#else - u8 rcnts[3]; /* Reader counts */ - u8 wmode; /* Writer mode */ -#endif - }; - }; - arch_spinlock_t lock; -}; - /** * rspin_until_writer_unlock - inc reader count & spin until writer is gone * @lock : Pointer to queue rwlock structure @@ -125,10 +105,8 @@ void queued_write_lock_slowpath(struct qrwlock *lock) * or wait for a previous writer to go away. */ for (;;) { - struct __qrwlock *l = (struct __qrwlock *)lock; - - if (!READ_ONCE(l->wmode) && - (cmpxchg_relaxed(&l->wmode, 0, _QW_WAITING) == 0)) + if (!READ_ONCE(lock->wmode) && + (cmpxchg_relaxed(&lock->wmode, 0, _QW_WAITING) == 0)) break; cpu_relax(); From patchwork Thu Oct 12 12:20:48 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 115609 Delivered-To: patch@linaro.org Received: by 10.140.22.163 with SMTP id 32csp1875117qgn; Thu, 12 Oct 2017 05:20:54 -0700 (PDT) X-Google-Smtp-Source: AOwi7QBNcfURSa5vRmfJAx+iznhoMSBRpuk7ZETW68KfzNm5ZgS+degBPxJkn5Qv0eMeJDVEoxyn X-Received: by 10.98.196.209 with SMTP id h78mr2063470pfk.249.1507810854243; Thu, 12 Oct 2017 05:20:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507810854; cv=none; d=google.com; s=arc-20160816; b=VyKGG0Ef55gJXssBMlV2XZvjMWt6fN7vtHQpoJ6Qs87XsCZqHzGwUN94GmfSdw9XM9 mbOFzoMcME+uck/YxPQMdtbU8BXPz3mZqzpkXrRr6AWoRCcKPXqtjXjohkr8anruktak t7LuId3mfHo1ZsugwEjjvZmkwp1Jvj+iB4NauTFhnqStRAICDDaVf0zzb7ARHmrJqUwW K+wUEAgFEg97UYZslNpAPHWjlTD2UTPtSDUAeMwtrLWB2vCYlPjY+pFKvHJ4Jc6cfjuN CTDyO8t5Vr94tVsfeBgAk99H8B2nzHk0U97pKx6IIWHbrvf/CejjfpM8nMYYadPgROXG VCFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=Lsbu1jjDUivTkhhOKC6THwxDtyqrfTUgDs8peL1zrsc=; b=vkWcZS8JAGhvdpXAeY8od6oBXcDswbiCB5EIt467MT1Qy1R79ipQjhGRNH6dtLaGi2 NsMGdgDdQSGSXE7slCRgpRbTxp4cN8jTYHcL1ai3HcFvfht4Z74Z0HazOkWqqNjlLg7t ObY/NBeukTVw5PInMb/d+CHjfi1uBaCIZey8/rVC1bjTdca37Msq2hHMFQ8Th5MXbRZE 7xtz8exo69aFsu2M3rWgvE3B8Zyf4eRB1tAp6aJFLNQ94wTz64twoFWHMFek2mVdX1pJ kC2/TefnZoo9YLpvDHdrEA/KlhSIFvESrngqhBLObSHUXJy2xV6TbHM/H5y3ioyllFeh q6Fw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id r145si6892595pgr.753.2017.10.12.05.20.53; Thu, 12 Oct 2017 05:20:54 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757297AbdJLMUw (ORCPT + 27 others); Thu, 12 Oct 2017 08:20:52 -0400 Received: from foss.arm.com ([217.140.101.70]:45518 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756918AbdJLMUt (ORCPT ); Thu, 12 Oct 2017 08:20:49 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A4F5B15BF; Thu, 12 Oct 2017 05:20:48 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 769183F59E; Thu, 12 Oct 2017 05:20:48 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 268611AE2E1C; Thu, 12 Oct 2017 13:20:52 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Jeremy.Linton@arm.com, peterz@infradead.org, mingo@redhat.com, longman@redhat.com, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, Will Deacon Subject: [PATCH v3 2/5] locking/atomic: Add atomic_cond_read_acquire Date: Thu, 12 Oct 2017 13:20:48 +0100 Message-Id: <1507810851-306-3-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1507810851-306-1-git-send-email-will.deacon@arm.com> References: <1507810851-306-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org smp_cond_load_acquire provides a way to spin on a variable with acquire semantics until some conditional expression involing the variable is satisfied. Architectures such as arm64 can potentially enter a low-power state, waking up only when the value of the variable changes, which reduces the system impact of tight polling loops. This patch makes the same interface available to users of atomic_t, atomic64_t and atomic_long_t, rather than require messy accesses to the structure internals. Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Signed-off-by: Will Deacon --- include/asm-generic/atomic-long.h | 3 +++ include/linux/atomic.h | 4 ++++ 2 files changed, 7 insertions(+) -- 2.1.4 diff --git a/include/asm-generic/atomic-long.h b/include/asm-generic/atomic-long.h index 288cc9e96395..f2d97b782031 100644 --- a/include/asm-generic/atomic-long.h +++ b/include/asm-generic/atomic-long.h @@ -243,4 +243,7 @@ static inline long atomic_long_add_unless(atomic_long_t *l, long a, long u) #define atomic_long_inc_not_zero(l) \ ATOMIC_LONG_PFX(_inc_not_zero)((ATOMIC_LONG_PFX(_t) *)(l)) +#define atomic_long_cond_read_acquire(v, c) \ + ATOMIC_LONG_PFX(_cond_read_acquire)((ATOMIC_LONG_PFX(_t) *)(v), (c)) + #endif /* _ASM_GENERIC_ATOMIC_LONG_H */ diff --git a/include/linux/atomic.h b/include/linux/atomic.h index 40d6bfec0e0d..0aeb2b3f4578 100644 --- a/include/linux/atomic.h +++ b/include/linux/atomic.h @@ -653,6 +653,8 @@ static inline int atomic_dec_if_positive(atomic_t *v) } #endif +#define atomic_cond_read_acquire(v, c) smp_cond_load_acquire(&(v)->counter, (c)) + #ifdef CONFIG_GENERIC_ATOMIC64 #include #endif @@ -1072,6 +1074,8 @@ static inline long long atomic64_fetch_andnot_release(long long i, atomic64_t *v } #endif +#define atomic64_cond_read_acquire(v, c) smp_cond_load_acquire(&(v)->counter, (c)) + #include #endif /* _LINUX_ATOMIC_H */ From patchwork Thu Oct 12 12:20:49 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 115613 Delivered-To: patch@linaro.org Received: by 10.140.22.163 with SMTP id 32csp1875975qgn; Thu, 12 Oct 2017 05:21:50 -0700 (PDT) X-Google-Smtp-Source: AOwi7QDuHmQ67Ot6AQ6PjAHVix+h6dUK1XT2vq//892by7qJZTXzzsc9WcT2Tozlay93WCxSuWj5 X-Received: by 10.99.117.10 with SMTP id q10mr91753pgc.288.1507810910012; Thu, 12 Oct 2017 05:21:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507810910; cv=none; d=google.com; s=arc-20160816; b=sHjFJOcM6CDA5zKF7Tx9jLKT1mPYIqiVgVMKJfZ7+44mUDctQQbJUfTCUITgEoNMSV VcTmWRdrLSzb/0Nqgb9c9a1gZO2ynKf6EhyGoSHYNXWdXWF1648aQ8ovZKsd9JU8wkH9 Sl9/+PC4t56WrPiFEVrzQJXnXYIZ2Xd8SPZDbzw5cPnFTsNJp76Mt/OgFb8kUWgc1kYM gmnfx9Y/pUBq3GXvigAApbQ7iPnyRBaIK+WiD48kJCHX7I+PHTR5CuLpvRnYvIBqO23Y Ik+tvhvp9Xge7jVeSwXe58KuEcsIEDkZuUJ1aGQyOzLwV9nkX/l7ouGqYmcNNMRHPYph dLKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=dP8UKkJ8/psIsQC4OFpMTRWJY4CNFC479IfnQlyNZus=; b=t+flYBBb+Fwuh9sGpHGXGF4HpBkPuVxQ/q1muX6dmtKXxcRGpJ4+4yXJQ19GH+k0Y9 S3qLrX/6vmPIjy5my5VYTEjWawpgyofXfymtt2hKwloxg+pZqcZWyB6Y/rrMt48e3Xon s6OVt8BuA36G3K7+GAWLGUtngTdMVfKssTRVSP7on7qJfsx25W6bAHY1gNn35Np2G3v6 M0xqsLZRWyuTOUd4hZuvfOIr03JdS4Ccs+tt6OSDk9FJ6BPIFOcR9SHdFGix9Oer783p cZomjo6CgsKs2BKAjLMe8NAY/0hk//oOFlqks+uTIZUMX242CupvXd1oZGaPeP3i0+Yb gyug== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id z7si7418476pgp.424.2017.10.12.05.21.49; Thu, 12 Oct 2017 05:21:50 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757197AbdJLMUv (ORCPT + 27 others); Thu, 12 Oct 2017 08:20:51 -0400 Received: from foss.arm.com ([217.140.101.70]:45522 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756914AbdJLMUt (ORCPT ); Thu, 12 Oct 2017 08:20:49 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B22C51610; Thu, 12 Oct 2017 05:20:48 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8421E3F673; Thu, 12 Oct 2017 05:20:48 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 34ECF1AE2E48; Thu, 12 Oct 2017 13:20:52 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Jeremy.Linton@arm.com, peterz@infradead.org, mingo@redhat.com, longman@redhat.com, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, Will Deacon Subject: [PATCH v3 3/5] kernel/locking: Use atomic_cond_read_acquire when spinning in qrwlock Date: Thu, 12 Oct 2017 13:20:49 +0100 Message-Id: <1507810851-306-4-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1507810851-306-1-git-send-email-will.deacon@arm.com> References: <1507810851-306-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The qrwlock slowpaths involve spinning when either a prospective reader is waiting for a concurrent writer to drain, or a prospective writer is waiting for concurrent readers to drain. In both of these situations, atomic_cond_read_acquire can be used to avoid busy-waiting and make use of any backoff functionality provided by the architecture. This patch replaces the open-code loops and rspin_until_writer_unlock implementation with atomic_cond_read_acquire. The write mode transition zero to _QW_WAITING is left alone, since (a) this doesn't need acquire semantics and (b) should be fast. Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Tested-by: Waiman Long Tested-by: Jeremy Linton Tested-by: Adam Wallis Tested-by: Jan Glauber Signed-off-by: Will Deacon --- include/asm-generic/qrwlock.h | 4 ++-- kernel/locking/qrwlock.c | 50 +++++++++++-------------------------------- 2 files changed, 14 insertions(+), 40 deletions(-) -- 2.1.4 diff --git a/include/asm-generic/qrwlock.h b/include/asm-generic/qrwlock.h index 02c0a768e6b0..c716b02e8fd7 100644 --- a/include/asm-generic/qrwlock.h +++ b/include/asm-generic/qrwlock.h @@ -49,7 +49,7 @@ /* * External function declarations */ -extern void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts); +extern void queued_read_lock_slowpath(struct qrwlock *lock); extern void queued_write_lock_slowpath(struct qrwlock *lock); /** @@ -100,7 +100,7 @@ static inline void queued_read_lock(struct qrwlock *lock) return; /* The slowpath will decrement the reader count, if necessary. */ - queued_read_lock_slowpath(lock, cnts); + queued_read_lock_slowpath(lock); } /** diff --git a/kernel/locking/qrwlock.c b/kernel/locking/qrwlock.c index 1af791e37348..5825e0fc1a8e 100644 --- a/kernel/locking/qrwlock.c +++ b/kernel/locking/qrwlock.c @@ -24,28 +24,10 @@ #include /** - * rspin_until_writer_unlock - inc reader count & spin until writer is gone - * @lock : Pointer to queue rwlock structure - * @writer: Current queue rwlock writer status byte - * - * In interrupt context or at the head of the queue, the reader will just - * increment the reader count & wait until the writer releases the lock. - */ -static __always_inline void -rspin_until_writer_unlock(struct qrwlock *lock, u32 cnts) -{ - while ((cnts & _QW_WMASK) == _QW_LOCKED) { - cpu_relax(); - cnts = atomic_read_acquire(&lock->cnts); - } -} - -/** * queued_read_lock_slowpath - acquire read lock of a queue rwlock * @lock: Pointer to queue rwlock structure - * @cnts: Current qrwlock lock value */ -void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts) +void queued_read_lock_slowpath(struct qrwlock *lock) { /* * Readers come here when they cannot get the lock without waiting @@ -53,13 +35,12 @@ void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts) if (unlikely(in_interrupt())) { /* * Readers in interrupt context will get the lock immediately - * if the writer is just waiting (not holding the lock yet). - * The rspin_until_writer_unlock() function returns immediately - * in this case. Otherwise, they will spin (with ACQUIRE - * semantics) until the lock is available without waiting in - * the queue. + * if the writer is just waiting (not holding the lock yet), + * so spin with ACQUIRE semantics until the lock is available + * without waiting in the queue. */ - rspin_until_writer_unlock(lock, cnts); + atomic_cond_read_acquire(&lock->cnts, (VAL & _QW_WMASK) + != _QW_LOCKED); return; } atomic_sub(_QR_BIAS, &lock->cnts); @@ -68,14 +49,14 @@ void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts) * Put the reader into the wait queue */ arch_spin_lock(&lock->wait_lock); + atomic_add(_QR_BIAS, &lock->cnts); /* * The ACQUIRE semantics of the following spinning code ensure * that accesses can't leak upwards out of our subsequent critical * section in the case that the lock is currently held for write. */ - cnts = atomic_fetch_add_acquire(_QR_BIAS, &lock->cnts); - rspin_until_writer_unlock(lock, cnts); + atomic_cond_read_acquire(&lock->cnts, (VAL & _QW_WMASK) != _QW_LOCKED); /* * Signal the next one in queue to become queue head @@ -90,8 +71,6 @@ EXPORT_SYMBOL(queued_read_lock_slowpath); */ void queued_write_lock_slowpath(struct qrwlock *lock) { - u32 cnts; - /* Put the writer into the wait queue */ arch_spin_lock(&lock->wait_lock); @@ -113,15 +92,10 @@ void queued_write_lock_slowpath(struct qrwlock *lock) } /* When no more readers, set the locked flag */ - for (;;) { - cnts = atomic_read(&lock->cnts); - if ((cnts == _QW_WAITING) && - (atomic_cmpxchg_acquire(&lock->cnts, _QW_WAITING, - _QW_LOCKED) == _QW_WAITING)) - break; - - cpu_relax(); - } + do { + atomic_cond_read_acquire(&lock->cnts, VAL == _QW_WAITING); + } while (atomic_cmpxchg_relaxed(&lock->cnts, _QW_WAITING, + _QW_LOCKED) != _QW_WAITING); unlock: arch_spin_unlock(&lock->wait_lock); } From patchwork Thu Oct 12 12:20:50 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 115610 Delivered-To: patch@linaro.org Received: by 10.140.22.163 with SMTP id 32csp1875138qgn; Thu, 12 Oct 2017 05:20:55 -0700 (PDT) X-Google-Smtp-Source: AOwi7QC0YfRdatkEgLl/8V1jabpvmk58Fh/s6yM383WBYMhxStO5L7nasY3EUYWHRw3Y+ElP9VMF X-Received: by 10.99.108.132 with SMTP id h126mr86936pgc.434.1507810855326; Thu, 12 Oct 2017 05:20:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507810855; cv=none; d=google.com; s=arc-20160816; b=yg3YeUFiElQzSNvJ4xlnRyCbFI7hxtJDj73thb3lmVTj+/E12FXibSzZ/AQNCS1EF3 MqzkEes7n6/EtPTk6ZgTpCjfj9Gh7pRp6pIClNsoyho9ZFRXVytjfARl8gto9WsN95wH wKoZacuBT8LuUoK0Zz0ZPoHhNUuaGYB+TYDoFjeth4jYoj/ro81JjWOb4mXTLuXhJk5/ 0Y8KiUKsXULp2BYO8QotrPqlQWnwTpDbUgur87L1OT219vL6cBARRs13EA7iLBcTrR8p 3Gpi/coBp0KHyPSHKiQx7RHp3gdDWkDcuPBBWNA6S9gBVJikm339da5DX85O+HRVIFc5 VR6w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=MEEmjVs62p8KMG1MC+HGkPmC9pLEt4UdJVXQfFdaWCs=; b=n1v//9e3EPU/lYtf1y8kGlScO2WHtmXjGbyo1WScqJpbmL/aL+22VmySvMmwP9bOaY C4gTRrJ9WIai85bGt+GYoPf+9eRcHJ47fK8mCMDy7h3nTUARFQnC7u7NWnigK6Cl5IkH wACiWda8IYbw5paMMCzWYBhcB6qxkZ5LfVyo0+MErpIZeitiIsh7RyPgapOOS6mTYpw/ Ai+JxYnEQp9um1Ol85R87ruGFeKOGS6DGSGf0y4KSwNO9tkRhxfdIRqcYvpSyY5sJmmk 0cCIHAjHkA2DEpRevzi0/1qnckMyxV98HeN1uAiQMeVkSt49WcMV0MIqV4kGxm11HwHI /iBQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id r145si6892595pgr.753.2017.10.12.05.20.55; Thu, 12 Oct 2017 05:20:55 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757378AbdJLMUx (ORCPT + 27 others); Thu, 12 Oct 2017 08:20:53 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:45566 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756998AbdJLMUt (ORCPT ); Thu, 12 Oct 2017 08:20:49 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C0EC2164F; Thu, 12 Oct 2017 05:20:48 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 92DE53F599; Thu, 12 Oct 2017 05:20:48 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 430331AE2F43; Thu, 12 Oct 2017 13:20:52 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Jeremy.Linton@arm.com, peterz@infradead.org, mingo@redhat.com, longman@redhat.com, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, Will Deacon Subject: [PATCH v3 4/5] arm64: locking: Move rwlock implementation over to qrwlocks Date: Thu, 12 Oct 2017 13:20:50 +0100 Message-Id: <1507810851-306-5-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1507810851-306-1-git-send-email-will.deacon@arm.com> References: <1507810851-306-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Now that the qrwlock can make use of WFE, remove our homebrew rwlock code in favour of the generic queued implementation. Tested-by: Waiman Long Tested-by: Jeremy Linton Tested-by: Adam Wallis Tested-by: Jan Glauber Signed-off-by: Will Deacon --- arch/arm64/Kconfig | 17 ++++ arch/arm64/include/asm/Kbuild | 1 + arch/arm64/include/asm/spinlock.h | 164 +------------------------------- arch/arm64/include/asm/spinlock_types.h | 6 +- 4 files changed, 20 insertions(+), 168 deletions(-) -- 2.1.4 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 0df64a6a56d4..df02ad932020 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -22,7 +22,24 @@ config ARM64 select ARCH_HAS_STRICT_MODULE_RWX select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST select ARCH_HAVE_NMI_SAFE_CMPXCHG if ACPI_APEI_SEA + select ARCH_INLINE_READ_LOCK if !PREEMPT + select ARCH_INLINE_READ_LOCK_BH if !PREEMPT + select ARCH_INLINE_READ_LOCK_IRQ if !PREEMPT + select ARCH_INLINE_READ_LOCK_IRQSAVE if !PREEMPT + select ARCH_INLINE_READ_UNLOCK if !PREEMPT + select ARCH_INLINE_READ_UNLOCK_BH if !PREEMPT + select ARCH_INLINE_READ_UNLOCK_IRQ if !PREEMPT + select ARCH_INLINE_READ_UNLOCK_IRQRESTORE if !PREEMPT + select ARCH_INLINE_WRITE_LOCK if !PREEMPT + select ARCH_INLINE_WRITE_LOCK_BH if !PREEMPT + select ARCH_INLINE_WRITE_LOCK_IRQ if !PREEMPT + select ARCH_INLINE_WRITE_LOCK_IRQSAVE if !PREEMPT + select ARCH_INLINE_WRITE_UNLOCK if !PREEMPT + select ARCH_INLINE_WRITE_UNLOCK_BH if !PREEMPT + select ARCH_INLINE_WRITE_UNLOCK_IRQ if !PREEMPT + select ARCH_INLINE_WRITE_UNLOCK_IRQRESTORE if !PREEMPT select ARCH_USE_CMPXCHG_LOCKREF + select ARCH_USE_QUEUED_RWLOCKS select ARCH_SUPPORTS_MEMORY_FAILURE select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_NUMA_BALANCING diff --git a/arch/arm64/include/asm/Kbuild b/arch/arm64/include/asm/Kbuild index 2326e39d5892..e63d0a8312de 100644 --- a/arch/arm64/include/asm/Kbuild +++ b/arch/arm64/include/asm/Kbuild @@ -16,6 +16,7 @@ generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += msi.h generic-y += preempt.h +generic-y += qrwlock.h generic-y += rwsem.h generic-y += segment.h generic-y += serial.h diff --git a/arch/arm64/include/asm/spinlock.h b/arch/arm64/include/asm/spinlock.h index aa51a38e46e4..fdb827c7832f 100644 --- a/arch/arm64/include/asm/spinlock.h +++ b/arch/arm64/include/asm/spinlock.h @@ -137,169 +137,7 @@ static inline int arch_spin_is_contended(arch_spinlock_t *lock) } #define arch_spin_is_contended arch_spin_is_contended -/* - * Write lock implementation. - * - * Write locks set bit 31. Unlocking, is done by writing 0 since the lock is - * exclusively held. - * - * The memory barriers are implicit with the load-acquire and store-release - * instructions. - */ - -static inline void arch_write_lock(arch_rwlock_t *rw) -{ - unsigned int tmp; - - asm volatile(ARM64_LSE_ATOMIC_INSN( - /* LL/SC */ - " sevl\n" - "1: wfe\n" - "2: ldaxr %w0, %1\n" - " cbnz %w0, 1b\n" - " stxr %w0, %w2, %1\n" - " cbnz %w0, 2b\n" - __nops(1), - /* LSE atomics */ - "1: mov %w0, wzr\n" - "2: casa %w0, %w2, %1\n" - " cbz %w0, 3f\n" - " ldxr %w0, %1\n" - " cbz %w0, 2b\n" - " wfe\n" - " b 1b\n" - "3:") - : "=&r" (tmp), "+Q" (rw->lock) - : "r" (0x80000000) - : "memory"); -} - -static inline int arch_write_trylock(arch_rwlock_t *rw) -{ - unsigned int tmp; - - asm volatile(ARM64_LSE_ATOMIC_INSN( - /* LL/SC */ - "1: ldaxr %w0, %1\n" - " cbnz %w0, 2f\n" - " stxr %w0, %w2, %1\n" - " cbnz %w0, 1b\n" - "2:", - /* LSE atomics */ - " mov %w0, wzr\n" - " casa %w0, %w2, %1\n" - __nops(2)) - : "=&r" (tmp), "+Q" (rw->lock) - : "r" (0x80000000) - : "memory"); - - return !tmp; -} - -static inline void arch_write_unlock(arch_rwlock_t *rw) -{ - asm volatile(ARM64_LSE_ATOMIC_INSN( - " stlr wzr, %0", - " swpl wzr, wzr, %0") - : "=Q" (rw->lock) :: "memory"); -} - -/* write_can_lock - would write_trylock() succeed? */ -#define arch_write_can_lock(x) ((x)->lock == 0) - -/* - * Read lock implementation. - * - * It exclusively loads the lock value, increments it and stores the new value - * back if positive and the CPU still exclusively owns the location. If the - * value is negative, the lock is already held. - * - * During unlocking there may be multiple active read locks but no write lock. - * - * The memory barriers are implicit with the load-acquire and store-release - * instructions. - * - * Note that in UNDEFINED cases, such as unlocking a lock twice, the LL/SC - * and LSE implementations may exhibit different behaviour (although this - * will have no effect on lockdep). - */ -static inline void arch_read_lock(arch_rwlock_t *rw) -{ - unsigned int tmp, tmp2; - - asm volatile( - " sevl\n" - ARM64_LSE_ATOMIC_INSN( - /* LL/SC */ - "1: wfe\n" - "2: ldaxr %w0, %2\n" - " add %w0, %w0, #1\n" - " tbnz %w0, #31, 1b\n" - " stxr %w1, %w0, %2\n" - " cbnz %w1, 2b\n" - __nops(1), - /* LSE atomics */ - "1: wfe\n" - "2: ldxr %w0, %2\n" - " adds %w1, %w0, #1\n" - " tbnz %w1, #31, 1b\n" - " casa %w0, %w1, %2\n" - " sbc %w0, %w1, %w0\n" - " cbnz %w0, 2b") - : "=&r" (tmp), "=&r" (tmp2), "+Q" (rw->lock) - : - : "cc", "memory"); -} - -static inline void arch_read_unlock(arch_rwlock_t *rw) -{ - unsigned int tmp, tmp2; - - asm volatile(ARM64_LSE_ATOMIC_INSN( - /* LL/SC */ - "1: ldxr %w0, %2\n" - " sub %w0, %w0, #1\n" - " stlxr %w1, %w0, %2\n" - " cbnz %w1, 1b", - /* LSE atomics */ - " movn %w0, #0\n" - " staddl %w0, %2\n" - __nops(2)) - : "=&r" (tmp), "=&r" (tmp2), "+Q" (rw->lock) - : - : "memory"); -} - -static inline int arch_read_trylock(arch_rwlock_t *rw) -{ - unsigned int tmp, tmp2; - - asm volatile(ARM64_LSE_ATOMIC_INSN( - /* LL/SC */ - " mov %w1, #1\n" - "1: ldaxr %w0, %2\n" - " add %w0, %w0, #1\n" - " tbnz %w0, #31, 2f\n" - " stxr %w1, %w0, %2\n" - " cbnz %w1, 1b\n" - "2:", - /* LSE atomics */ - " ldr %w0, %2\n" - " adds %w1, %w0, #1\n" - " tbnz %w1, #31, 1f\n" - " casa %w0, %w1, %2\n" - " sbc %w1, %w1, %w0\n" - __nops(1) - "1:") - : "=&r" (tmp), "=&r" (tmp2), "+Q" (rw->lock) - : - : "cc", "memory"); - - return !tmp2; -} - -/* read_can_lock - would read_trylock() succeed? */ -#define arch_read_can_lock(x) ((x)->lock < 0x80000000) +#include /* See include/linux/spinlock.h */ #define smp_mb__after_spinlock() smp_mb() diff --git a/arch/arm64/include/asm/spinlock_types.h b/arch/arm64/include/asm/spinlock_types.h index 55be59a35e3f..6b856012c51b 100644 --- a/arch/arm64/include/asm/spinlock_types.h +++ b/arch/arm64/include/asm/spinlock_types.h @@ -36,10 +36,6 @@ typedef struct { #define __ARCH_SPIN_LOCK_UNLOCKED { 0 , 0 } -typedef struct { - volatile unsigned int lock; -} arch_rwlock_t; - -#define __ARCH_RW_LOCK_UNLOCKED { 0 } +#include #endif From patchwork Thu Oct 12 12:20:51 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 115611 Delivered-To: patch@linaro.org Received: by 10.140.22.163 with SMTP id 32csp1875173qgn; Thu, 12 Oct 2017 05:20:57 -0700 (PDT) X-Google-Smtp-Source: AOwi7QBt61mTVFuR97Kw7ama+6eIhFm9FQxUSYthwg3me1XUzU+sn4r9fvzyD8sxJtYi3++gB6W/ X-Received: by 10.159.195.67 with SMTP id z3mr82881pln.9.1507810857554; Thu, 12 Oct 2017 05:20:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507810857; cv=none; d=google.com; s=arc-20160816; b=cpsJP4aawSi1IxZUPWgPl0SCZOi315AFf3rHUL+pl+nmBNLKCk6vTGH2O9et6XDqw+ XcM/WH6zaAdfrQvG4fLJLsfPBUdZifI+MEEM7bgaoQMAMYnk0y4549gbMEKzcaekbMsJ 1upvrdIeFX/SlalX2hZeE2La4kKfJG75CzKmFt2DVLG8jWKXZVQmc1An63QSogJFcG3e IAP8SF2QLKS+D0EtEBioKTBQek2sRAgdakBPkz8rmNlnmNQEvTjzCcXxBB7erBwNCUyt 5mbgzJV/XQBy0A/F+urjnkjmusKI2Bp3ejkWSKJ7TuIHbrlg9GYzuFN+5QCOIZPHtV2A jZPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=TMBhTbI13xT+r3I3HBrqmGkoTIqkc/VVd4Q8j6zisic=; b=wLoI0EXxZnh6THFre5cAA5tSoIAy95DN/8WxZ8933gzuYXyoP4UslBknikZ8CtMEu0 QVBuNbc3OqU/KNOIWPBgKV3xSWWXaOIbEkXw5TrUehjBWxRHOIyaIhSjxQkxxNxO69vU cxpxHnoq4Z42Rnirv+lFScIZIfy81ze6WoPf0RR9u2sStEcfcnhUQHqafdZlSyUgdE7a mHjg7Y2iouNWjbv7b7lbH5Yp9e4YiRY9C6UTHzxDZ99JW5uNSrGJFIVTYtUR5XqXQDGO /MBuO3ZWvIQ9khBd/tpM63AXbyJHJZGQKEIj4ywd/kMaI/o+6A7pcZlDKJcdbSAP76Ty FJRA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id r145si6892595pgr.753.2017.10.12.05.20.57; Thu, 12 Oct 2017 05:20:57 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757470AbdJLMU4 (ORCPT + 27 others); Thu, 12 Oct 2017 08:20:56 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:45578 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751458AbdJLMUt (ORCPT ); Thu, 12 Oct 2017 08:20:49 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 73BF01650; Thu, 12 Oct 2017 05:20:49 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 45CAD3F599; Thu, 12 Oct 2017 05:20:49 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 519491AE2F6D; Thu, 12 Oct 2017 13:20:52 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Jeremy.Linton@arm.com, peterz@infradead.org, mingo@redhat.com, longman@redhat.com, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, Will Deacon Subject: [PATCH v3 5/5] kernel/locking: Prevent slowpath writers getting held up by fastpath Date: Thu, 12 Oct 2017 13:20:51 +0100 Message-Id: <1507810851-306-6-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1507810851-306-1-git-send-email-will.deacon@arm.com> References: <1507810851-306-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When a prospective writer takes the qrwlock locking slowpath due to the lock being held, it attempts to cmpxchg the wmode field from 0 to _QW_WAITING so that concurrent lockers also take the slowpath and queue on the spinlock accordingly, allowing the lockers to drain. Unfortunately, this isn't fair, because a fastpath writer that comes in after the lock is made available but before the _QW_WAITING flag is set can effectively jump the queue. If there is a steady stream of prospective writers, then the waiter will be held off indefinitely. This patch restores fairness by separating _QW_WAITING and _QW_LOCKED into two distinct fields: _QW_LOCKED continues to occupy the bottom byte of the lockword so that it can be cleared unconditionally when unlocking, but _QW_WAITING now occupies what used to be the bottom bit of the reader count. This then forces the slow-path for concurrent lockers. Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Tested-by: Waiman Long Tested-by: Jeremy Linton Tested-by: Adam Wallis Tested-by: Jan Glauber Signed-off-by: Will Deacon --- include/asm-generic/qrwlock.h | 23 +++++------------------ include/asm-generic/qrwlock_types.h | 8 ++++---- kernel/locking/qrwlock.c | 20 +++++--------------- 3 files changed, 14 insertions(+), 37 deletions(-) -- 2.1.4 diff --git a/include/asm-generic/qrwlock.h b/include/asm-generic/qrwlock.h index c716b02e8fd7..0f7062bd55e5 100644 --- a/include/asm-generic/qrwlock.h +++ b/include/asm-generic/qrwlock.h @@ -26,24 +26,11 @@ /* * Writer states & reader shift and bias. - * - * | +0 | +1 | +2 | +3 | - * ----+----+----+----+----+ - * LE | 78 | 56 | 34 | 12 | 0x12345678 - * ----+----+----+----+----+ - * | wr | rd | - * +----+----+----+----+ - * - * ----+----+----+----+----+ - * BE | 12 | 34 | 56 | 78 | 0x12345678 - * ----+----+----+----+----+ - * | rd | wr | - * +----+----+----+----+ */ -#define _QW_WAITING 1 /* A writer is waiting */ -#define _QW_LOCKED 0xff /* A writer holds the lock */ -#define _QW_WMASK 0xff /* Writer mask */ -#define _QR_SHIFT 8 /* Reader count shift */ +#define _QW_WAITING 0x100 /* A writer is waiting */ +#define _QW_LOCKED 0x0ff /* A writer holds the lock */ +#define _QW_WMASK 0x1ff /* Writer mask */ +#define _QR_SHIFT 9 /* Reader count shift */ #define _QR_BIAS (1U << _QR_SHIFT) /* @@ -134,7 +121,7 @@ static inline void queued_read_unlock(struct qrwlock *lock) */ static inline void queued_write_unlock(struct qrwlock *lock) { - smp_store_release(&lock->wmode, 0); + smp_store_release(&lock->wlocked, 0); } /* diff --git a/include/asm-generic/qrwlock_types.h b/include/asm-generic/qrwlock_types.h index 507f2dc51bba..8af752acbdc0 100644 --- a/include/asm-generic/qrwlock_types.h +++ b/include/asm-generic/qrwlock_types.h @@ -13,11 +13,11 @@ typedef struct qrwlock { atomic_t cnts; struct { #ifdef __LITTLE_ENDIAN - u8 wmode; /* Writer mode */ - u8 rcnts[3]; /* Reader counts */ + u8 wlocked; /* Locked for write? */ + u8 __lstate[3]; #else - u8 rcnts[3]; /* Reader counts */ - u8 wmode; /* Writer mode */ + u8 __lstate[3]; + u8 wlocked; /* Locked for write? */ #endif }; }; diff --git a/kernel/locking/qrwlock.c b/kernel/locking/qrwlock.c index 5825e0fc1a8e..c7471c3fb798 100644 --- a/kernel/locking/qrwlock.c +++ b/kernel/locking/qrwlock.c @@ -39,8 +39,7 @@ void queued_read_lock_slowpath(struct qrwlock *lock) * so spin with ACQUIRE semantics until the lock is available * without waiting in the queue. */ - atomic_cond_read_acquire(&lock->cnts, (VAL & _QW_WMASK) - != _QW_LOCKED); + atomic_cond_read_acquire(&lock->cnts, !(VAL & _QW_LOCKED)); return; } atomic_sub(_QR_BIAS, &lock->cnts); @@ -56,7 +55,7 @@ void queued_read_lock_slowpath(struct qrwlock *lock) * that accesses can't leak upwards out of our subsequent critical * section in the case that the lock is currently held for write. */ - atomic_cond_read_acquire(&lock->cnts, (VAL & _QW_WMASK) != _QW_LOCKED); + atomic_cond_read_acquire(&lock->cnts, !(VAL & _QW_LOCKED)); /* * Signal the next one in queue to become queue head @@ -79,19 +78,10 @@ void queued_write_lock_slowpath(struct qrwlock *lock) (atomic_cmpxchg_acquire(&lock->cnts, 0, _QW_LOCKED) == 0)) goto unlock; - /* - * Set the waiting flag to notify readers that a writer is pending, - * or wait for a previous writer to go away. - */ - for (;;) { - if (!READ_ONCE(lock->wmode) && - (cmpxchg_relaxed(&lock->wmode, 0, _QW_WAITING) == 0)) - break; - - cpu_relax(); - } + /* Set the waiting flag to notify readers that a writer is pending */ + atomic_add(_QW_WAITING, &lock->cnts); - /* When no more readers, set the locked flag */ + /* When no more readers or writers, set the locked flag */ do { atomic_cond_read_acquire(&lock->cnts, VAL == _QW_WAITING); } while (atomic_cmpxchg_relaxed(&lock->cnts, _QW_WAITING,