From patchwork Sat Jan 5 17:48:57 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13828 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 3C3D523E21 for ; Sat, 5 Jan 2013 17:49:21 +0000 (UTC) Received: from mail-vc0-f173.google.com (mail-vc0-f173.google.com [209.85.220.173]) by fiordland.canonical.com (Postfix) with ESMTP id D5BA2A191C4 for ; Sat, 5 Jan 2013 17:49:20 +0000 (UTC) Received: by mail-vc0-f173.google.com with SMTP id f13so17609925vcb.4 for ; Sat, 05 Jan 2013 09:49:20 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:x-forwarded-to:x-forwarded-for:delivered-to:x-received :received-spf:from:to:cc:subject:date:message-id:x-mailer :in-reply-to:references:x-content-scanned:x-cbid:x-gm-message-state; bh=8ny2p9VFVKaF8Npe1Mn1jS0wds+bTdG1kSZphwpznmk=; b=W4h4m2vS1HhJjoO6m8FqZ3BKexBov4oqskekDLOznMg6aJt8T6AN/wS0DoZ56ZDlwt mw1xL7NUwQlvIc66GPPTlD4R5JdGtVXSZgePKvJFsrgH5wg7y/WRaFTqWMvASgCcgY3A nIpAT9nxpzkVY9YJVOALO9M+4xvHe/dPbyjqFtWozG/KMXAch9GKl/YYDg4eXUR+Ef6r qAC634GOTUpYAN0Fo0AJnqFvuRA1ryhr4pFYdlRMUyIYFWLYv2xAD7reB3N9tw2CPqOI NW3OnO/enR8n/WROACIco22jwnR5mlY+tkJZv48AoxzhEAWxP5P868UfD9bIoe5YveMi 91Lw== X-Received: by 10.220.157.9 with SMTP id z9mr79704582vcw.45.1357408160348; Sat, 05 Jan 2013 09:49:20 -0800 (PST) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.58.145.101 with SMTP id st5csp17429veb; Sat, 5 Jan 2013 09:49:19 -0800 (PST) X-Received: by 10.43.114.135 with SMTP id fa7mr42080451icc.21.1357408157831; Sat, 05 Jan 2013 09:49:17 -0800 (PST) Received: from e31.co.us.ibm.com (e31.co.us.ibm.com. [32.97.110.149]) by mx.google.com with ESMTPS id p8si23727100icj.107.2013.01.05.09.49.17 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 05 Jan 2013 09:49:17 -0800 (PST) Received-SPF: pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.110.149 as permitted sender) client-ip=32.97.110.149; Authentication-Results: mx.google.com; spf=pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.110.149 as permitted sender) smtp.mail=paulmck@linux.vnet.ibm.com Received: from /spool/local by e31.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Sat, 5 Jan 2013 10:49:17 -0700 Received: from d03dlp03.boulder.ibm.com (9.17.202.179) by e31.co.us.ibm.com (192.168.1.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Sat, 5 Jan 2013 10:49:15 -0700 Received: from d03relay02.boulder.ibm.com (d03relay02.boulder.ibm.com [9.17.195.227]) by d03dlp03.boulder.ibm.com (Postfix) with ESMTP id 84F2619D803F; Sat, 5 Jan 2013 10:49:14 -0700 (MST) Received: from d03av02.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.195.168]) by d03relay02.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r05HnEn4185648; Sat, 5 Jan 2013 10:49:14 -0700 Received: from d03av02.boulder.ibm.com (loopback [127.0.0.1]) by d03av02.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r05HnBeo032266; Sat, 5 Jan 2013 10:49:14 -0700 Received: from paulmck-ThinkPad-W500 ([9.80.23.97]) by d03av02.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id r05HnAbK032181; Sat, 5 Jan 2013 10:49:10 -0700 Received: by paulmck-ThinkPad-W500 (Postfix, from userid 1000) id 1D6BEE4DB3; Sat, 5 Jan 2013 09:49:06 -0800 (PST) From: "Paul E. McKenney" To: linux-kernel@vger.kernel.org Cc: mingo@elte.hu, laijs@cn.fujitsu.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@polymtl.ca, josh@joshtriplett.org, niv@us.ibm.com, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, Valdis.Kletnieks@vt.edu, dhowells@redhat.com, edumazet@google.com, darren@dvhart.com, fweisbec@gmail.com, sbw@mit.edu, patches@linaro.org, "Paul E. McKenney" , "Paul E. McKenney" Subject: [PATCH tip/core/rcu 07/14] rcu: Accelerate RCU callbacks at grace-period end Date: Sat, 5 Jan 2013 09:48:57 -0800 Message-Id: <1357408144-15830-7-git-send-email-paulmck@linux.vnet.ibm.com> X-Mailer: git-send-email 1.7.8 In-Reply-To: <1357408144-15830-1-git-send-email-paulmck@linux.vnet.ibm.com> References: <20130105174844.GA14172@linux.vnet.ibm.com> <1357408144-15830-1-git-send-email-paulmck@linux.vnet.ibm.com> X-Content-Scanned: Fidelis XPS MAILER x-cbid: 13010517-7282-0000-0000-000012CCA920 X-Gm-Message-State: ALoCoQmbDPqxBXHoCclupIKzTson/VeBhqARaKhK84Bqn///ijs4H43Cwtz85vqHqfCVgd4fl/Jx From: "Paul E. McKenney" Now that callback acceleration is idempotent, it is safe to accelerate callbacks during grace-period cleanup on any CPUs that the kthread happens to be running on. This commit therefore propagates the completion of the grace period to the per-CPU data structures, and also adds an rcu_advance_cbs() just before the cpu_needs_another_gp() check in order to reduce false-positive grace periods. Signed-off-by: Paul E. McKenney --- kernel/rcutree.c | 21 +++++++++++++-------- 1 files changed, 13 insertions(+), 8 deletions(-) diff --git a/kernel/rcutree.c b/kernel/rcutree.c index 4ec797e..392c977 100644 --- a/kernel/rcutree.c +++ b/kernel/rcutree.c @@ -1434,6 +1434,9 @@ static void rcu_gp_cleanup(struct rcu_state *rsp) rcu_for_each_node_breadth_first(rsp, rnp) { raw_spin_lock_irq(&rnp->lock); rnp->completed = rsp->gpnum; + rdp = this_cpu_ptr(rsp->rda); + if (rnp == rdp->mynode) + __rcu_process_gp_end(rsp, rnp, rdp); nocb += rcu_nocb_gp_cleanup(rsp, rnp); raw_spin_unlock_irq(&rnp->lock); cond_resched(); @@ -1446,6 +1449,7 @@ static void rcu_gp_cleanup(struct rcu_state *rsp) trace_rcu_grace_period(rsp->name, rsp->completed, "end"); rsp->fqs_state = RCU_GP_IDLE; rdp = this_cpu_ptr(rsp->rda); + rcu_advance_cbs(rsp, rnp, rdp); /* Reduce false positives below. */ if (cpu_needs_another_gp(rsp, rdp)) rsp->gp_flags = 1; raw_spin_unlock_irq(&rnp->lock); @@ -1535,6 +1539,15 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags) struct rcu_data *rdp = this_cpu_ptr(rsp->rda); struct rcu_node *rnp = rcu_get_root(rsp); + /* + * If there is no grace period in progress right now, any + * callbacks we have up to this point will be satisfied by the + * next grace period. Also, advancing the callbacks reduces the + * probability of false positives from cpu_needs_another_gp() + * resulting in pointless grace periods. So, advance callbacks! + */ + rcu_advance_cbs(rsp, rnp, rdp); + if (!rsp->gp_kthread || !cpu_needs_another_gp(rsp, rdp)) { /* @@ -1547,14 +1560,6 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags) return; } - /* - * Because there is no grace period in progress right now, - * any callbacks we have up to this point will be satisfied - * by the next grace period. So this is a good place to - * assign a grace period number to recently posted callbacks. - */ - rcu_accelerate_cbs(rsp, rnp, rdp); - rsp->gp_flags = RCU_GP_FLAG_INIT; raw_spin_unlock(&rnp->lock); /* Interrupts remain disabled. */