From patchwork Thu Oct 5 12:54:54 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 114938 Delivered-To: patch@linaro.org Received: by 10.80.163.170 with SMTP id s39csp194637edb; Thu, 5 Oct 2017 05:56:33 -0700 (PDT) X-Google-Smtp-Source: AOwi7QB2kFhBGFn9RL+/GdVVG/tiydil3F+RGyPau1jEf7VLzArY8zCryWLTZWJP97YbS6e2LQ73 X-Received: by 10.99.177.74 with SMTP id g10mr21568114pgp.326.1507208192949; Thu, 05 Oct 2017 05:56:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507208192; cv=none; d=google.com; s=arc-20160816; b=l/qQFADfIl/Q67pJ3TtyxwyDGV+kPf4qyyYbTJmeUGbFJM8BMGfoPWsJ2lRPAN7ez0 9jp3UpusLF1VLmYl4/TDqgg7wARigTNba1gn6tVWJumAE7zSJ9w4sZ8egEwBZLSRa7D9 f4Pmd4p4GreMi74tI7eKvJbUKE2i7kjtfdXOd9TwdU+DZI++Sqgm9BtrIEGnlu+dtTSF PnwNJlfB0EBe2LkDmEmTZXzeBE4soyEHhEnA3mjIlkmwpLcu0zVUqrr2hZxQObwInUim znat1fszn5SkvYsO+eV58LAuQ43HoBm5nlTa3806AQDgE6UiNfkAd3E4M0Bfv50Ip8VJ UsMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=Z10243CUOT6we7NbHVCeLszLWlJSMZdsQMovmOORf9M=; b=VUO4vJDDbZaOW7Oskg6dDjKjqL0oaECxTEmmfma8FY4LZL7SLxwryzSuL3cC7NhvWi McADRGsie4OemCye6ysu95NTJixtr8a6i05BqEpq1T6lZ5rAZaFxqWWvxmOOfyTYahNZ Z/Yuloatw8zzNnsIPKxtLp1SuSNMzH0WX69eYveAxYiz4sArTSqe0i/wK6n9+5BMOFiL HqNSyAltYf77grzh88wogL0Sufq8jBLZ3vXd7GjnOh4yIDY2kQ4ZslikFM09yaXNC0Pu ZgQlDtzFbl2ZtPO3BHUDcFM2HD8U6Xn+Pgprl91EppPtw8fHW3IcWc7LdMw4m4gBWp69 7ysA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q14si13757542pll.260.2017.10.05.05.56.32; Thu, 05 Oct 2017 05:56:32 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751888AbdJEM4R (ORCPT + 26 others); Thu, 5 Oct 2017 08:56:17 -0400 Received: from foss.arm.com ([217.140.101.70]:45000 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751357AbdJEMy6 (ORCPT ); Thu, 5 Oct 2017 08:54:58 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B4E4D15BE; Thu, 5 Oct 2017 05:54:57 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 86D573F7C6; Thu, 5 Oct 2017 05:54:57 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id CD4FF1AE2E15; Thu, 5 Oct 2017 13:54:58 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Jeremy.Linton@arm.com, peterz@infradead.org, mingo@redhat.com, longman@redhat.com, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, Will Deacon Subject: [PATCH 3/6] kernel/locking: Use atomic_cond_read_acquire when spinning in qrwlock Date: Thu, 5 Oct 2017 13:54:54 +0100 Message-Id: <1507208097-825-4-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1507208097-825-1-git-send-email-will.deacon@arm.com> References: <1507208097-825-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The qrwlock slowpaths involve spinning when either a prospective reader is waiting for a concurrent writer to drain, or a prospective writer is waiting for concurrent readers to drain. In both of these situations, atomic_cond_read_acquire can be used to avoid busy-waiting and make use of any backoff functionality provided by the architecture. This patch replaces the open-code loops and rspin_until_writer_unlock implementation with atomic_cond_read_acquire. The write mode transition zero to _QW_WAITING is left alone, since (a) this doesn't need acquire semantics and (b) should be fast. Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Signed-off-by: Will Deacon --- kernel/locking/qrwlock.c | 47 +++++++++++------------------------------------ 1 file changed, 11 insertions(+), 36 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qrwlock.c b/kernel/locking/qrwlock.c index 1af791e37348..b7ea4647c74d 100644 --- a/kernel/locking/qrwlock.c +++ b/kernel/locking/qrwlock.c @@ -24,23 +24,6 @@ #include /** - * rspin_until_writer_unlock - inc reader count & spin until writer is gone - * @lock : Pointer to queue rwlock structure - * @writer: Current queue rwlock writer status byte - * - * In interrupt context or at the head of the queue, the reader will just - * increment the reader count & wait until the writer releases the lock. - */ -static __always_inline void -rspin_until_writer_unlock(struct qrwlock *lock, u32 cnts) -{ - while ((cnts & _QW_WMASK) == _QW_LOCKED) { - cpu_relax(); - cnts = atomic_read_acquire(&lock->cnts); - } -} - -/** * queued_read_lock_slowpath - acquire read lock of a queue rwlock * @lock: Pointer to queue rwlock structure * @cnts: Current qrwlock lock value @@ -53,13 +36,12 @@ void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts) if (unlikely(in_interrupt())) { /* * Readers in interrupt context will get the lock immediately - * if the writer is just waiting (not holding the lock yet). - * The rspin_until_writer_unlock() function returns immediately - * in this case. Otherwise, they will spin (with ACQUIRE - * semantics) until the lock is available without waiting in - * the queue. + * if the writer is just waiting (not holding the lock yet), + * so spin with ACQUIRE semantics until the lock is available + * without waiting in the queue. */ - rspin_until_writer_unlock(lock, cnts); + atomic_cond_read_acquire(&lock->cnts, (VAL & _QW_WMASK) + != _QW_LOCKED); return; } atomic_sub(_QR_BIAS, &lock->cnts); @@ -68,14 +50,14 @@ void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts) * Put the reader into the wait queue */ arch_spin_lock(&lock->wait_lock); + atomic_add(_QR_BIAS, &lock->cnts); /* * The ACQUIRE semantics of the following spinning code ensure * that accesses can't leak upwards out of our subsequent critical * section in the case that the lock is currently held for write. */ - cnts = atomic_fetch_add_acquire(_QR_BIAS, &lock->cnts); - rspin_until_writer_unlock(lock, cnts); + atomic_cond_read_acquire(&lock->cnts, (VAL & _QW_WMASK) != _QW_LOCKED); /* * Signal the next one in queue to become queue head @@ -90,8 +72,6 @@ EXPORT_SYMBOL(queued_read_lock_slowpath); */ void queued_write_lock_slowpath(struct qrwlock *lock) { - u32 cnts; - /* Put the writer into the wait queue */ arch_spin_lock(&lock->wait_lock); @@ -113,15 +93,10 @@ void queued_write_lock_slowpath(struct qrwlock *lock) } /* When no more readers, set the locked flag */ - for (;;) { - cnts = atomic_read(&lock->cnts); - if ((cnts == _QW_WAITING) && - (atomic_cmpxchg_acquire(&lock->cnts, _QW_WAITING, - _QW_LOCKED) == _QW_WAITING)) - break; - - cpu_relax(); - } + do { + atomic_cond_read_acquire(&lock->cnts, VAL == _QW_WAITING); + } while (atomic_cmpxchg_relaxed(&lock->cnts, _QW_WAITING, + _QW_LOCKED) != _QW_WAITING); unlock: arch_spin_unlock(&lock->wait_lock); }