From patchwork Thu Oct 5 12:54:56 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 114935 Delivered-To: patch@linaro.org Received: by 10.80.163.170 with SMTP id s39csp194221edb; Thu, 5 Oct 2017 05:56:03 -0700 (PDT) X-Google-Smtp-Source: AOwi7QBJwUnfReNv1t0LDoRYb0qTGMsvL7ev1jU/bgw01P5qJFm62aXh9Z8qH/HqaJIMF+/UpdlX X-Received: by 10.84.204.136 with SMTP id b8mr22571585ple.361.1507208163759; Thu, 05 Oct 2017 05:56:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507208163; cv=none; d=google.com; s=arc-20160816; b=dL50AHaJpmBdIblN/8fShsWXYt6lTXS0/0gVaZGERGRJJd/NjZVkg8DCJd6X2BXdV3 IuRw6VZrSIEUmnImtPp2rgwnQqyVnw/TWBXk7XB3U4ChiNJv/a3OKg1GmwG5wvqIhArJ ize9/tBlWlLMT3xhaKKZtRSlSvifqPKnm/kzqf8NUeX/oQSZ5c8zzmdQx06wgzvHdsIr pVoy9scGjX4/nz97V7rdYTIYvaZ1zF4q24OJnN7cZQmBvBtgvpcpEtkJYh8AbyZXCGt/ h0+VJ0olmxN6V6k9qYFRkf5R49iJcgkcLwH1VNWS7hxr0a7NmP+/QzkY/orimasqC/hk LoRQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=xL0PXOokHQYRwtvAEzIrntNGZ+jje0YZWEDBJNfdUDo=; b=sQVc3T6AKqztrHdBKKFj+wAJ+74PFqKJG3ZsLMyahAyZKm8FeU3GlSIBWjes6tPBSC w27AjP/zRixKTxD2fYMhzp5S7lE17SKUr18lgFygBt74rFTeO9P00tbwwjskEcCfdrbR rbff0aOG9MT5I5H1ngrNquFN+Ibl5+CZmVQWl6iyRp/BeW+G16YRJgv6HI7BVoOOZVkP krRc6ZpLimWXfFNkOv7bAwiiHOL5czx3FpjLApeCaBf/JxN6xk1Jg45p6d6x2rXcO2Vx mSZJdvf24ojYksl/T+vJPOCNUHnFvymQlID1+ePvd93WpjMXNVrf+jUiPaXIRPqwEWxT HrBA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i9si3855234plt.396.2017.10.05.05.55.51; Thu, 05 Oct 2017 05:56:03 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751659AbdJEMzY (ORCPT + 26 others); Thu, 5 Oct 2017 08:55:24 -0400 Received: from foss.arm.com ([217.140.101.70]:45048 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751415AbdJEMy6 (ORCPT ); Thu, 5 Oct 2017 08:54:58 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7E03A1650; Thu, 5 Oct 2017 05:54:58 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 501923F483; Thu, 5 Oct 2017 05:54:58 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id EC4021AE2F43; Thu, 5 Oct 2017 13:54:58 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Jeremy.Linton@arm.com, peterz@infradead.org, mingo@redhat.com, longman@redhat.com, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, Will Deacon Subject: [PATCH 5/6] kernel/locking: Prevent slowpath writers getting held up by fastpath Date: Thu, 5 Oct 2017 13:54:56 +0100 Message-Id: <1507208097-825-6-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1507208097-825-1-git-send-email-will.deacon@arm.com> References: <1507208097-825-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When a prospective writer takes the qrwlock locking slowpath due to the lock being held, it attempts to cmpxchg the wmode field from 0 to _QW_WAITING so that concurrent lockers also take the slowpath and queue on the spinlock accordingly, allowing the lockers to drain. Unfortunately, this isn't fair, because a fastpath writer that comes in after the lock is made available but before the _QW_WAITING flag is set can effectively jump the queue. If there is a steady stream of prospective writers, then the waiter will be held off indefinitely. This patch restores fairness by separating _QW_WAITING and _QW_LOCKED into two bits in the wmode byte and having the waiter set _QW_WAITING unconditionally. This then forces the slow-path for concurrent lockers, but requires that a writer unlock operation performs an atomic_sub_release instead of a store_release so that the waiting status is preserved. Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Signed-off-by: Will Deacon --- include/asm-generic/qrwlock.h | 4 ++-- kernel/locking/qrwlock.c | 20 +++++--------------- 2 files changed, 7 insertions(+), 17 deletions(-) -- 2.1.4 Acked-by: Waiman Long diff --git a/include/asm-generic/qrwlock.h b/include/asm-generic/qrwlock.h index 02c0a768e6b0..8b7edef500e5 100644 --- a/include/asm-generic/qrwlock.h +++ b/include/asm-generic/qrwlock.h @@ -41,7 +41,7 @@ * +----+----+----+----+ */ #define _QW_WAITING 1 /* A writer is waiting */ -#define _QW_LOCKED 0xff /* A writer holds the lock */ +#define _QW_LOCKED 2 /* A writer holds the lock */ #define _QW_WMASK 0xff /* Writer mask */ #define _QR_SHIFT 8 /* Reader count shift */ #define _QR_BIAS (1U << _QR_SHIFT) @@ -134,7 +134,7 @@ static inline void queued_read_unlock(struct qrwlock *lock) */ static inline void queued_write_unlock(struct qrwlock *lock) { - smp_store_release(&lock->wmode, 0); + (void)atomic_sub_return_release(_QW_LOCKED, &lock->cnts); } /* diff --git a/kernel/locking/qrwlock.c b/kernel/locking/qrwlock.c index b7ea4647c74d..e940f2c2b4f2 100644 --- a/kernel/locking/qrwlock.c +++ b/kernel/locking/qrwlock.c @@ -40,8 +40,7 @@ void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts) * so spin with ACQUIRE semantics until the lock is available * without waiting in the queue. */ - atomic_cond_read_acquire(&lock->cnts, (VAL & _QW_WMASK) - != _QW_LOCKED); + atomic_cond_read_acquire(&lock->cnts, !(VAL & _QW_LOCKED)); return; } atomic_sub(_QR_BIAS, &lock->cnts); @@ -57,7 +56,7 @@ void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts) * that accesses can't leak upwards out of our subsequent critical * section in the case that the lock is currently held for write. */ - atomic_cond_read_acquire(&lock->cnts, (VAL & _QW_WMASK) != _QW_LOCKED); + atomic_cond_read_acquire(&lock->cnts, !(VAL & _QW_LOCKED)); /* * Signal the next one in queue to become queue head @@ -80,19 +79,10 @@ void queued_write_lock_slowpath(struct qrwlock *lock) (atomic_cmpxchg_acquire(&lock->cnts, 0, _QW_LOCKED) == 0)) goto unlock; - /* - * Set the waiting flag to notify readers that a writer is pending, - * or wait for a previous writer to go away. - */ - for (;;) { - if (!READ_ONCE(lock->wmode) && - (cmpxchg_relaxed(&lock->wmode, 0, _QW_WAITING) == 0)) - break; - - cpu_relax(); - } + /* Set the waiting flag to notify readers that a writer is pending */ + atomic_add(_QW_WAITING, &lock->cnts); - /* When no more readers, set the locked flag */ + /* When no more readers or writers, set the locked flag */ do { atomic_cond_read_acquire(&lock->cnts, VAL == _QW_WAITING); } while (atomic_cmpxchg_relaxed(&lock->cnts, _QW_WAITING,