From patchwork Thu Apr 26 10:34:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 134481 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp2083757lji; Thu, 26 Apr 2018 03:38:05 -0700 (PDT) X-Google-Smtp-Source: AIpwx4+LSElvSpsfJdQycZl1BF5QTQ2TGjJi6Ywgs6pZTX3x+/5oQlK0k5mmLRl+kV6GUlFvlFUb X-Received: by 10.98.72.74 with SMTP id v71mr31724026pfa.241.1524739085633; Thu, 26 Apr 2018 03:38:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524739085; cv=none; d=google.com; s=arc-20160816; b=fMK38+kVABcqfqZTyTBxYcE27iOCd0LupEM+NvHximA8tFRktkUep3ZLBUeDocGtIZ SnS7nJEYIfNRsixqwX57gfmQWXkn5WRv2oEGXwFbqi8RqsBwRGAQqlfEBXG1CAFYAXEs lsp5QINli4swIw2NdMBhjoCOUXjwCs85lyp5NeuV8poOgWc1y+rUwJs4hXywzEAKSPvV 9MTTFw9z4+qQn9cKJeluy9K49OyGJeRn0HPoUQljz9gXSMbzM6Vnc4mtQXnI5THAgu6W vykQAUOwgJAmol61OX5UhGPJ8adxI2WAbOx7zap+plDz9zBqZ0zXp6x0zc1DuLYu3Yvt K8dA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=xc6s9HS5ti7pmWSBFcBDmDyJIxQZQ6uQpM22zxIDbSk=; b=rHbQKUa54j/fbNPUR9277pt952SRG4uzSHAWkIo9HqG6QV9odr39vUm6Pi7wsS1m9l ENXPoqF3WJGdwVJ0V7RjaSPz3umhVH/6Ydgza60fcnC8zu/TTH1iiQ81NH4uAubAw1FP PojAGP03T2CvCU8uFKv4l3MMzzQhf3OoquvM+pvjKkzGWBOnqIvZycC4K2o+TPVB0I6N VyxDZRrzIXfs+NJtnz8qmdChnMQ8aoi8St6F54QWNSGTLXCnLjL6Tqw0EmRsrQTy186N 1pVzlNEaR9Ek5DZav75MPjvlX1ZqHXATE6P7FdXAeapLhZqhLGj6rKD2r9/dFNOcLb5r zX3g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b2si15721607pgc.275.2018.04.26.03.38.05; Thu, 26 Apr 2018 03:38:05 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755967AbeDZKiD (ORCPT + 29 others); Thu, 26 Apr 2018 06:38:03 -0400 Received: from foss.arm.com ([217.140.101.70]:51004 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754678AbeDZKeJ (ORCPT ); Thu, 26 Apr 2018 06:34:09 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 83D7B1688; Thu, 26 Apr 2018 03:34:09 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 559D93F487; Thu, 26 Apr 2018 03:34:09 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 0450C1AE50BE; Thu, 26 Apr 2018 11:34:28 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, will.deacon@arm.com Subject: [PATCH v3 03/14] locking/qspinlock: Bound spinning on pending->locked transition in slowpath Date: Thu, 26 Apr 2018 11:34:17 +0100 Message-Id: <1524738868-31318-4-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1524738868-31318-1-git-send-email-will.deacon@arm.com> References: <1524738868-31318-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org If a locker taking the qspinlock slowpath reads a lock value indicating that only the pending bit is set, then it will spin whilst the concurrent pending->locked transition takes effect. Unfortunately, there is no guarantee that such a transition will ever be observed since concurrent lockers could continuously set pending and hand over the lock amongst themselves, leading to starvation. Whilst this would probably resolve in practice, it means that it is not possible to prove liveness properties about the lock and means that lock acquisition time is unbounded. Rather than removing the pending->locked spinning from the slowpath altogether (which has been shown to heavily penalise a 2-threaded locking stress test on x86), this patch replaces the explicit spinning with a call to atomic_cond_read_relaxed and allows the architecture to provide a bound on the number of spins. For architectures that can respond to changes in cacheline state in their smp_cond_load implementation, it should be sufficient to use the default bound of 1. Cc: Peter Zijlstra Cc: Ingo Molnar Suggested-by: Waiman Long Signed-off-by: Will Deacon --- kernel/locking/qspinlock.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index f5b0e59f6d14..a0f7976348f8 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -77,6 +77,18 @@ #endif /* + * The pending bit spinning loop count. + * This heuristic is used to limit the number of lockword accesses + * made by atomic_cond_read_relaxed when waiting for the lock to + * transition out of the "== _Q_PENDING_VAL" state. We don't spin + * indefinitely because there's no guarantee that we'll make forward + * progress. + */ +#ifndef _Q_PENDING_LOOPS +#define _Q_PENDING_LOOPS 1 +#endif + +/* * Per-CPU queue node structures; we can never have more than 4 nested * contexts: task, softirq, hardirq, nmi. * @@ -266,13 +278,15 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) return; /* - * wait for in-progress pending->locked hand-overs + * Wait for in-progress pending->locked hand-overs with a bounded + * number of spins so that we guarantee forward progress. * * 0,1,0 -> 0,0,1 */ if (val == _Q_PENDING_VAL) { - while ((val = atomic_read(&lock->val)) == _Q_PENDING_VAL) - cpu_relax(); + int cnt = _Q_PENDING_LOOPS; + val = atomic_cond_read_relaxed(&lock->val, + (VAL != _Q_PENDING_VAL) || !cnt--); } /*