From patchwork Fri Oct 6 13:34:40 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 115062 Delivered-To: patch@linaro.org Received: by 10.80.163.170 with SMTP id s39csp1375831edb; Fri, 6 Oct 2017 06:34:48 -0700 (PDT) X-Google-Smtp-Source: AOwi7QChEV+4G3WMHoPKymKpSoMJNs6xnIQvT6U2W+Km40kFY3gTlkPlc+4EsGpFCbM3cqz2q3xA X-Received: by 10.99.110.199 with SMTP id j190mr2063658pgc.330.1507296888509; Fri, 06 Oct 2017 06:34:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507296888; cv=none; d=google.com; s=arc-20160816; b=HO0H56fLNLqSCUzW6MWS+4stYrhK6v1yx173Aftn9zf3x+zK5AOuurkWddyN9Q3/Hi VCSrwSTtS20iQEO5yOG4zIbGPzE9d8vFQQLqjAoGu5EFro31ukBJlhkMSxlECYz6Izro kkS1EalIaAi3eVwa6G/2U1XPcavTdOH3tWZlmo/FTgfjnsxOysUn56MZUCtFHMtHvHGJ RkEXQ8eGktybLJzn2jRYWljN6p11fwkDSwLJVubkqX4Cy5EI8k8AsryWssA55NYJ8EgD 3qQTSNX0nhIoOS+1zde1+guEWhl5cOPtyaSYBxjrV/aCiCPpnFjKpCFXMzJM81cnh42v KocA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=Z10243CUOT6we7NbHVCeLszLWlJSMZdsQMovmOORf9M=; b=b4RSupoM46mPQzqZkdWaxabGxNvh55a8DuucN5m2wcbuGapPC+SKfr1H3k5lIerwc2 q1wOjjoVHQLIk5FBbfU3KObPAegz+K6tZTtnnYZdj5N64uPYxNnJobinjwxjSdZu1/Q1 HXK7U/ArFJp2DT1hTjV0p0KoWzsj0rnNGFvz0IRGy6zK5lvqU+yOMFxHEuYgQ8Ad7Uc6 Fx7k419io2l4P/402CdDj+eLXZBv9erV+8G16dVTHAgmdQXajci0OLA9N943FmoVugPZ Nx4TKxU30DCi1Am42RdQE4Di+47rIa550ICRwg3z3xrEYpzya0w0ZhVrpiF28be/A4Q+ RTyw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id t24si84665pfj.580.2017.10.06.06.34.48; Fri, 06 Oct 2017 06:34:48 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752587AbdJFNep (ORCPT + 26 others); Fri, 6 Oct 2017 09:34:45 -0400 Received: from foss.arm.com ([217.140.101.70]:33600 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752431AbdJFNem (ORCPT ); Fri, 6 Oct 2017 09:34:42 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id F00521610; Fri, 6 Oct 2017 06:34:41 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C1E863F7BE; Fri, 6 Oct 2017 06:34:41 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 6C6C61AE2E15; Fri, 6 Oct 2017 14:34:43 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Jeremy.Linton@arm.com, peterz@infradead.org, mingo@redhat.com, longman@redhat.com, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, Will Deacon Subject: [PATCH v2 3/5] kernel/locking: Use atomic_cond_read_acquire when spinning in qrwlock Date: Fri, 6 Oct 2017 14:34:40 +0100 Message-Id: <1507296882-18721-4-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1507296882-18721-1-git-send-email-will.deacon@arm.com> References: <1507296882-18721-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The qrwlock slowpaths involve spinning when either a prospective reader is waiting for a concurrent writer to drain, or a prospective writer is waiting for concurrent readers to drain. In both of these situations, atomic_cond_read_acquire can be used to avoid busy-waiting and make use of any backoff functionality provided by the architecture. This patch replaces the open-code loops and rspin_until_writer_unlock implementation with atomic_cond_read_acquire. The write mode transition zero to _QW_WAITING is left alone, since (a) this doesn't need acquire semantics and (b) should be fast. Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Signed-off-by: Will Deacon --- kernel/locking/qrwlock.c | 47 +++++++++++------------------------------------ 1 file changed, 11 insertions(+), 36 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qrwlock.c b/kernel/locking/qrwlock.c index 1af791e37348..b7ea4647c74d 100644 --- a/kernel/locking/qrwlock.c +++ b/kernel/locking/qrwlock.c @@ -24,23 +24,6 @@ #include /** - * rspin_until_writer_unlock - inc reader count & spin until writer is gone - * @lock : Pointer to queue rwlock structure - * @writer: Current queue rwlock writer status byte - * - * In interrupt context or at the head of the queue, the reader will just - * increment the reader count & wait until the writer releases the lock. - */ -static __always_inline void -rspin_until_writer_unlock(struct qrwlock *lock, u32 cnts) -{ - while ((cnts & _QW_WMASK) == _QW_LOCKED) { - cpu_relax(); - cnts = atomic_read_acquire(&lock->cnts); - } -} - -/** * queued_read_lock_slowpath - acquire read lock of a queue rwlock * @lock: Pointer to queue rwlock structure * @cnts: Current qrwlock lock value @@ -53,13 +36,12 @@ void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts) if (unlikely(in_interrupt())) { /* * Readers in interrupt context will get the lock immediately - * if the writer is just waiting (not holding the lock yet). - * The rspin_until_writer_unlock() function returns immediately - * in this case. Otherwise, they will spin (with ACQUIRE - * semantics) until the lock is available without waiting in - * the queue. + * if the writer is just waiting (not holding the lock yet), + * so spin with ACQUIRE semantics until the lock is available + * without waiting in the queue. */ - rspin_until_writer_unlock(lock, cnts); + atomic_cond_read_acquire(&lock->cnts, (VAL & _QW_WMASK) + != _QW_LOCKED); return; } atomic_sub(_QR_BIAS, &lock->cnts); @@ -68,14 +50,14 @@ void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts) * Put the reader into the wait queue */ arch_spin_lock(&lock->wait_lock); + atomic_add(_QR_BIAS, &lock->cnts); /* * The ACQUIRE semantics of the following spinning code ensure * that accesses can't leak upwards out of our subsequent critical * section in the case that the lock is currently held for write. */ - cnts = atomic_fetch_add_acquire(_QR_BIAS, &lock->cnts); - rspin_until_writer_unlock(lock, cnts); + atomic_cond_read_acquire(&lock->cnts, (VAL & _QW_WMASK) != _QW_LOCKED); /* * Signal the next one in queue to become queue head @@ -90,8 +72,6 @@ EXPORT_SYMBOL(queued_read_lock_slowpath); */ void queued_write_lock_slowpath(struct qrwlock *lock) { - u32 cnts; - /* Put the writer into the wait queue */ arch_spin_lock(&lock->wait_lock); @@ -113,15 +93,10 @@ void queued_write_lock_slowpath(struct qrwlock *lock) } /* When no more readers, set the locked flag */ - for (;;) { - cnts = atomic_read(&lock->cnts); - if ((cnts == _QW_WAITING) && - (atomic_cmpxchg_acquire(&lock->cnts, _QW_WAITING, - _QW_LOCKED) == _QW_WAITING)) - break; - - cpu_relax(); - } + do { + atomic_cond_read_acquire(&lock->cnts, VAL == _QW_WAITING); + } while (atomic_cmpxchg_relaxed(&lock->cnts, _QW_WAITING, + _QW_LOCKED) != _QW_WAITING); unlock: arch_spin_unlock(&lock->wait_lock); }