From patchwork Thu Dec 13 14:09:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 153644 Delivered-To: patch@linaro.org Received: by 2002:a2e:299d:0:0:0:0:0 with SMTP id p29-v6csp846161ljp; Thu, 13 Dec 2018 06:10:03 -0800 (PST) X-Google-Smtp-Source: AFSGD/Vh3L8hHjZG9lq15rd8xq4EDpopfI2dITtBWwyNGkT/+AM3d6Ixqb7279Bd1HB2LBRGZA7/ X-Received: by 2002:a17:902:6502:: with SMTP id b2mr23409299plk.44.1544710203133; Thu, 13 Dec 2018 06:10:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544710203; cv=none; d=google.com; s=arc-20160816; b=LO0Uy1Z02Z1NufARSWOv4e+cNxY5+oUHWr2BMtO2dN09+KAWBZgIdjJmOpYmauVNKe tSc3MHAn75f9ghnKD0NFJya8g1q0P12awD730bxbtOYYB3jdxUW3znMYHEY+Eu6WvBgE 5U/R+Ub/h9QXd9I2xioonHbdIexJd6DH6mX/rLryrrfCEihJzep8IlhZ6rsB8NGiNrzw h+froO3i6SFx3c8JHgiAT7IkC+/iDZIKSyF9tt/XgOJqvjqY+/OCxJJTqSxiTQFHWl9U NTJAjfZMRIdqXnNzbLKEN618cJfXXVJz/psSs5oIVtEaz1rcT0HXz7IhpWhKjnkAGWs0 FpxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :illegal-object:reply-to:references:in-reply-to:message-id:date :subject:cc:to:from; bh=7f+cbSfiLLXJ81KnA6/y7WrHOafjTUa1VHuPl7Vf25s=; b=a9Iis8RbGWdjMEwsQ2Q4Q/0wvg2FfSSlZveFuCyXkY/yENa+uyUH2zrHZ8NzNif7d7 MbUnkwY4FYJS413HKSm/+Jlz+YbdMByASsCG6DbllMgNQGbBnLmxV+OBdbHT9NuWiREL nba2OWwXWmlXd98/bLqiwhaYXLkX/vyPlI0DE4WfYdhDZh1Q85phmOkiweu4slrKIVoG f4PlwOSa97UcxoF3X9AK5321IVORRD4ieI5fUJKMnCg2AroodHvhhkxE22KqUC4+rLPw 80PSedZ5XA+Zxn7P2jfPYHgc2qRcZ2XMfojCtMbpJSVEXnvkiCXY/gWqVYZXryVlvXUs OlWw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id t16si1817482pfk.139.2018.12.13.06.10.02; Thu, 13 Dec 2018 06:10:03 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727757AbeLMOJn (ORCPT + 15 others); Thu, 13 Dec 2018 09:09:43 -0500 Received: from Galois.linutronix.de ([146.0.238.70]:46519 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728596AbeLMOJn (ORCPT ); Thu, 13 Dec 2018 09:09:43 -0500 Received: from localhost ([127.0.0.1] helo=bazinga.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1gXRgJ-0001G3-5h; Thu, 13 Dec 2018 15:09:39 +0100 From: Sebastian Andrzej Siewior To: stable@vger.kernel.org Cc: Peter Zijlstra , Will Deacon , Thomas Gleixner , Daniel Wagner , Waiman Long , Linus Torvalds , boqun.feng@gmail.com, linux-arm-kernel@lists.infradead.org, paulmck@linux.vnet.ibm.com, Ingo Molnar , Sebastian Andrzej Siewior Subject: [PATCH STABLE v4.9 07/10] locking/qspinlock: Kill cmpxchg() loop when claiming lock from head of queue Date: Thu, 13 Dec 2018 15:09:22 +0100 Message-Id: <20181213140925.6179-8-bigeasy@linutronix.de> X-Mailer: git-send-email 2.20.0 In-Reply-To: <20181213140925.6179-1-bigeasy@linutronix.de> References: <20181213140925.6179-1-bigeasy@linutronix.de> Reply-To: [PATCH STABLE v4.9 00/10], Backport, for, cache line starvation on x86 Illegal-Object: Syntax error in Reply-To: address found on vger.kernel.org: Reply-To: cache line starvation on x86 ^-extraneous tokens in address MIME-Version: 1.0 Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Will Deacon commit c61da58d8a9ba9238250a548f00826eaf44af0f7 upstream. When a queued locker reaches the head of the queue, it claims the lock by setting _Q_LOCKED_VAL in the lockword. If there isn't contention, it must also clear the tail as part of this operation so that subsequent lockers can avoid taking the slowpath altogether. Currently this is expressed as a cmpxchg() loop that practically only runs up to two iterations. This is confusing to the reader and unhelpful to the compiler. Rewrite the cmpxchg() loop without the loop, so that a failed cmpxchg() implies that there is contention and we just need to write to _Q_LOCKED_VAL without considering the rest of the lockword. Signed-off-by: Will Deacon Acked-by: Peter Zijlstra (Intel) Acked-by: Waiman Long Cc: Linus Torvalds Cc: Thomas Gleixner Cc: boqun.feng@gmail.com Cc: linux-arm-kernel@lists.infradead.org Cc: paulmck@linux.vnet.ibm.com Link: http://lkml.kernel.org/r/1524738868-31318-7-git-send-email-will.deacon@arm.com Signed-off-by: Ingo Molnar Signed-off-by: Sebastian Andrzej Siewior --- kernel/locking/qspinlock.c | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) -- 2.20.0 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index e7ab99a1f4387..ba5dc86a4d831 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -581,24 +581,21 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) * and nobody is pending, clear the tail code and grab the lock. * Otherwise, we only need to grab the lock. */ - for (;;) { - /* In the PV case we might already have _Q_LOCKED_VAL set */ - if ((val & _Q_TAIL_MASK) != tail || (val & _Q_PENDING_MASK)) { - set_locked(lock); - break; - } + + /* In the PV case we might already have _Q_LOCKED_VAL set */ + if ((val & _Q_TAIL_MASK) == tail) { /* * The smp_cond_load_acquire() call above has provided the - * necessary acquire semantics required for locking. At most - * two iterations of this loop may be ran. + * necessary acquire semantics required for locking. */ old = atomic_cmpxchg_relaxed(&lock->val, val, _Q_LOCKED_VAL); if (old == val) - goto release; /* No contention */ - - val = old; + goto release; /* No contention */ } + /* Either somebody is queued behind us or _Q_PENDING_VAL is set */ + set_locked(lock); + /* * contended path; wait for next if not observed yet, release. */