From patchwork Tue Oct 30 16:27:50 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 12588 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 4CE4023EF7 for ; Tue, 30 Oct 2012 16:40:01 +0000 (UTC) Received: from mail-ie0-f180.google.com (mail-ie0-f180.google.com [209.85.223.180]) by fiordland.canonical.com (Postfix) with ESMTP id E0BCCA18B1C for ; Tue, 30 Oct 2012 16:40:00 +0000 (UTC) Received: by mail-ie0-f180.google.com with SMTP id e10so671091iej.11 for ; Tue, 30 Oct 2012 09:40:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf:from:to:cc :subject:date:message-id:x-mailer:in-reply-to:references :x-content-scanned:x-cbid:x-gm-message-state; bh=viql0g7887YmlXS4ktA1FzWjDJB+RF4GQ5PqH/3N10E=; b=Otb20UkAdzQOfXnwAmXUyC4doCN2HUb3LEDu4MUWbgKOnnoc3w+TbtvTDZehJUrq6Z 9qF3Uv1yWr6mCokHpSrsc8Hi5mehLvCaTyZ4te1ECyn+nP3/An/vWKhi7SFJPrlbtVhf IEYIrdEfDVAFre9oWno7yb8rcZgX/cO8iuP2JdnxzTBTnd91cjxR0vi8GJOALajYvSTO RWVffixtopFhR54/YZAgEFKFl/j9/lW3f7nuBJ4WGnkkEUbYH58bSqggA0FSwXVEiO1X q6qlcjatGiNKYrC6/l7a4EhRdWoxMHuUOyqBdiyY7AjwH0KFgo5JEv1zPB3a7lx62WPX 0VDw== Received: by 10.50.161.169 with SMTP id xt9mr2002845igb.62.1351615200674; Tue, 30 Oct 2012 09:40:00 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.50.67.148 with SMTP id n20csp446837igt; Tue, 30 Oct 2012 09:40:00 -0700 (PDT) Received: by 10.42.132.194 with SMTP id e2mr28777693ict.21.1351615200385; Tue, 30 Oct 2012 09:40:00 -0700 (PDT) Received: from e34.co.us.ibm.com (e34.co.us.ibm.com. [32.97.110.152]) by mx.google.com with ESMTPS id nz6si9884522igc.24.2012.10.30.09.40.00 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 30 Oct 2012 09:40:00 -0700 (PDT) Received-SPF: pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.110.152 as permitted sender) client-ip=32.97.110.152; Authentication-Results: mx.google.com; spf=pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.110.152 as permitted sender) smtp.mail=paulmck@linux.vnet.ibm.com Received: from /spool/local by e34.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 30 Oct 2012 10:39:57 -0600 Received: from d03dlp02.boulder.ibm.com (9.17.202.178) by e34.co.us.ibm.com (192.168.1.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 30 Oct 2012 10:39:19 -0600 Received: from d03relay03.boulder.ibm.com (d03relay03.boulder.ibm.com [9.17.195.228]) by d03dlp02.boulder.ibm.com (Postfix) with ESMTP id A61BD3E40026 for ; Tue, 30 Oct 2012 10:39:14 -0600 (MDT) Received: from d03av01.boulder.ibm.com (d03av01.boulder.ibm.com [9.17.195.167]) by d03relay03.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q9UGdBwX167798 for ; Tue, 30 Oct 2012 10:39:12 -0600 Received: from d03av01.boulder.ibm.com (loopback [127.0.0.1]) by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q9UGd9Ti010812 for ; Tue, 30 Oct 2012 10:39:11 -0600 Received: from paulmck-ThinkPad-W500 (sig-9-65-77-17.mts.ibm.com [9.65.77.17]) by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id q9UGd7EU010506; Tue, 30 Oct 2012 10:39:09 -0600 Received: by paulmck-ThinkPad-W500 (Postfix, from userid 1000) id E7BD1EBE99; Tue, 30 Oct 2012 09:27:56 -0700 (PDT) From: "Paul E. McKenney" To: linux-kernel@vger.kernel.org Cc: mingo@elte.hu, laijs@cn.fujitsu.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@polymtl.ca, josh@joshtriplett.org, niv@us.ibm.com, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, Valdis.Kletnieks@vt.edu, dhowells@redhat.com, edumazet@google.com, darren@dvhart.com, fweisbec@gmail.com, sbw@mit.edu, patches@linaro.org, "Paul E. McKenney" Subject: [PATCH tip/core/rcu 1/6] rcu: Accelerate callbacks for CPU initiating a grace period Date: Tue, 30 Oct 2012 09:27:50 -0700 Message-Id: <1351614475-22895-1-git-send-email-paulmck@linux.vnet.ibm.com> X-Mailer: git-send-email 1.7.8 In-Reply-To: <20121030162728.GA22648@linux.vnet.ibm.com> References: <20121030162728.GA22648@linux.vnet.ibm.com> X-Content-Scanned: Fidelis XPS MAILER x-cbid: 12103016-2876-0000-0000-000001817151 X-Gm-Message-State: ALoCoQkPvN/L/0pM8Eqv35ztqAZ2WbiMXMAe1lROthId/oK8ZysnAGXGNYpFHN8RFCvOZSZ5T0Z3 From: "Paul E. McKenney" Because grace-period initialization is carried out by a separate kthread, it might happen on a different CPU than the one that had the callback needing a grace period -- which is where the callback acceleration needs to happen. Fortunately, rcu_start_gp() holds the root rcu_node structure's ->lock, which prevents a new grace period from starting. This allows this function to safely determine that a grace period has not yet started, which in turn allows it to fully accelerate any callbacks that it has pending. This commit adds this acceleration. Signed-off-by: Paul E. McKenney --- kernel/rcutree.c | 26 ++++++++++++++++++++++++-- 1 files changed, 24 insertions(+), 2 deletions(-) diff --git a/kernel/rcutree.c b/kernel/rcutree.c index 74df86b..93d6871 100644 --- a/kernel/rcutree.c +++ b/kernel/rcutree.c @@ -1404,15 +1404,37 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags) !cpu_needs_another_gp(rsp, rdp)) { /* * Either we have not yet spawned the grace-period - * task or this CPU does not need another grace period. + * task, this CPU does not need another grace period, + * or a grace period is already in progress. * Either way, don't start a new grace period. */ raw_spin_unlock_irqrestore(&rnp->lock, flags); return; } + /* + * Because there is no grace period in progress right now, + * any callbacks we have up to this point will be satisfied + * by the next grace period. So promote all callbacks to be + * handled after the end of the next grace period. If the + * CPU is not yet aware of the end of the previous grace period, + * we need to allow for the callback advancement that will + * occur when it does become aware. Deadlock prevents us from + * making it aware at this point: We cannot acquire a leaf + * rcu_node ->lock while holding the root rcu_node ->lock. + */ + rdp->nxttail[RCU_NEXT_READY_TAIL] = rdp->nxttail[RCU_NEXT_TAIL]; + if (rdp->completed == rsp->completed) + rdp->nxttail[RCU_WAIT_TAIL] = rdp->nxttail[RCU_NEXT_TAIL]; + rsp->gp_flags = RCU_GP_FLAG_INIT; - raw_spin_unlock_irqrestore(&rnp->lock, flags); + raw_spin_unlock(&rnp->lock); /* Interrupts remain disabled. */ + + /* Ensure that CPU is aware of completion of last grace period. */ + rcu_process_gp_end(rsp, rdp); + local_irq_restore(flags); + + /* Wake up rcu_gp_kthread() to start the grace period. */ wake_up(&rsp->gp_wq); }