From patchwork Tue Oct 30 17:19:21 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 12625 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 7005723F6F for ; Tue, 30 Oct 2012 17:37:11 +0000 (UTC) Received: from mail-ia0-f180.google.com (mail-ia0-f180.google.com [209.85.210.180]) by fiordland.canonical.com (Postfix) with ESMTP id F0876A18543 for ; Tue, 30 Oct 2012 17:37:10 +0000 (UTC) Received: by mail-ia0-f180.google.com with SMTP id f6so354805iag.11 for ; Tue, 30 Oct 2012 10:37:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf:from:to:cc :subject:date:message-id:x-mailer:in-reply-to:references :x-content-scanned:x-cbid:x-gm-message-state; bh=dccJjRQJZF9FRd4gGIyx5M1NskynkkCueGvKRsQCyNk=; b=DpQobh70bTK8ZiFzJ2CIPh3vDjq2nEKlyBLEVjEA9UwWtLkQYPdaUuNPwulHMyv9yS huowj8H9+iklCWW7DJKLPEW/7ULHYGZPoGwLuNTaozB8pHNvj3sZSoME7d8iRRF3FBvX ikw1wC02e7nLVKmKWqRsl4mi1zI4cjhzJ/GMLAAve+5qFrTuPGFiJVGfwgVjbTYjju0Z ZxlxU98623Y6sbvZrQDscVq1oFYavyMWvkIUDZuFJ+RCGW2H8RtR/QXMSUSCse47WVQ7 MVWYV1jo/t5r3TJRRAN2u5MahLab5vvv1i49qIQpGLUYASuRdepdYV/0ZgyMa2GRkhY6 a9eA== Received: by 10.50.161.169 with SMTP id xt9mr2180735igb.62.1351618630282; Tue, 30 Oct 2012 10:37:10 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.50.67.148 with SMTP id n20csp458818igt; Tue, 30 Oct 2012 10:37:10 -0700 (PDT) Received: by 10.42.57.10 with SMTP id b10mr29348909ich.54.1351618629824; Tue, 30 Oct 2012 10:37:09 -0700 (PDT) Received: from e6.ny.us.ibm.com (e6.ny.us.ibm.com. [32.97.182.146]) by mx.google.com with ESMTPS id l7si1364560igb.8.2012.10.30.10.37.09 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 30 Oct 2012 10:37:09 -0700 (PDT) Received-SPF: pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.182.146 as permitted sender) client-ip=32.97.182.146; Authentication-Results: mx.google.com; spf=pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.182.146 as permitted sender) smtp.mail=paulmck@linux.vnet.ibm.com Received: from /spool/local by e6.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 30 Oct 2012 13:37:07 -0400 Received: from d01dlp03.pok.ibm.com (9.56.250.168) by e6.ny.us.ibm.com (192.168.1.106) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 30 Oct 2012 13:32:11 -0400 Received: from d01relay04.pok.ibm.com (d01relay04.pok.ibm.com [9.56.227.236]) by d01dlp03.pok.ibm.com (Postfix) with ESMTP id 9D744C9003E for ; Tue, 30 Oct 2012 13:32:03 -0400 (EDT) Received: from d03av01.boulder.ibm.com (d03av01.boulder.ibm.com [9.17.195.167]) by d01relay04.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q9UHW1W1267992 for ; Tue, 30 Oct 2012 13:32:02 -0400 Received: from d03av01.boulder.ibm.com (loopback [127.0.0.1]) by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q9UHVcbR027890 for ; Tue, 30 Oct 2012 11:31:39 -0600 Received: from paulmck-ThinkPad-W500 (sig-9-49-139-14.mts.ibm.com [9.49.139.14]) by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id q9UHVZbG027383; Tue, 30 Oct 2012 11:31:37 -0600 Received: by paulmck-ThinkPad-W500 (Postfix, from userid 1000) id 1F042EBEEB; Tue, 30 Oct 2012 10:19:25 -0700 (PDT) From: "Paul E. McKenney" To: linux-kernel@vger.kernel.org Cc: mingo@elte.hu, laijs@cn.fujitsu.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@polymtl.ca, josh@joshtriplett.org, niv@us.ibm.com, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, Valdis.Kletnieks@vt.edu, dhowells@redhat.com, edumazet@google.com, darren@dvhart.com, fweisbec@gmail.com, sbw@mit.edu, patches@linaro.org, Antti P Miettinen , Antti P Miettinen , "Paul E. McKenney" Subject: [PATCH tip/core/rcu 3/5] rcu: Add a module parameter to force use of expedited RCU primitives Date: Tue, 30 Oct 2012 10:19:21 -0700 Message-Id: <1351617563-28506-3-git-send-email-paulmck@linux.vnet.ibm.com> X-Mailer: git-send-email 1.7.8 In-Reply-To: <1351617563-28506-1-git-send-email-paulmck@linux.vnet.ibm.com> References: <20121030171906.GA27431@linux.vnet.ibm.com> <1351617563-28506-1-git-send-email-paulmck@linux.vnet.ibm.com> X-Content-Scanned: Fidelis XPS MAILER x-cbid: 12103017-1976-0000-0000-000012DBA90D X-Gm-Message-State: ALoCoQnCHJNOpfiQfTYXTVTS244iPdU79iDLSnolj/mqVR/cFNDWUk93yd580LxwsKPXfSYa65Nu From: Antti P Miettinen There have been some embedded applications that would benefit from use of expedited grace-period primitives. In some ways, this is similar to synchronize_net() doing either a normal or an expedited grace period depending on lock state, but with control outside of the kernel. This commit therefore adds rcu_expedited boot and sysfs parameters that cause the kernel to substitute expedited primitives for the normal grace-period primitives. [ paulmck: Add trace/event/rcu.h to kernel/srcu.c to avoid build error. Get rid of infinite loop through contention path.] Signed-off-by: Antti P Miettinen Signed-off-by: Paul E. McKenney --- kernel/ksysfs.c | 18 ++++++++++++++++++ kernel/rcu.h | 2 ++ kernel/rcupdate.c | 3 +++ kernel/rcutiny_plugin.h | 5 ++++- kernel/rcutree.c | 12 +++++++++--- kernel/rcutree_plugin.h | 7 +++++-- kernel/srcu.c | 8 +++++++- 7 files changed, 48 insertions(+), 7 deletions(-) diff --git a/kernel/ksysfs.c b/kernel/ksysfs.c index 4e316e1..8715a79 100644 --- a/kernel/ksysfs.c +++ b/kernel/ksysfs.c @@ -141,6 +141,23 @@ static ssize_t fscaps_show(struct kobject *kobj, } KERNEL_ATTR_RO(fscaps); +int rcu_expedited; +static ssize_t rcu_expedited_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sprintf(buf, "%d\n", rcu_expedited); +} +static ssize_t rcu_expedited_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + if (kstrtoint(buf, 0, &rcu_expedited)) + return -EINVAL; + + return count; +} +KERNEL_ATTR_RW(rcu_expedited); + /* * Make /sys/kernel/notes give the raw contents of our kernel .notes section. */ @@ -182,6 +199,7 @@ static struct attribute * kernel_attrs[] = { &kexec_crash_size_attr.attr, &vmcoreinfo_attr.attr, #endif + &rcu_expedited_attr.attr, NULL }; diff --git a/kernel/rcu.h b/kernel/rcu.h index 8ba99cd..20dfba5 100644 --- a/kernel/rcu.h +++ b/kernel/rcu.h @@ -109,4 +109,6 @@ static inline bool __rcu_reclaim(char *rn, struct rcu_head *head) } } +extern int rcu_expedited; + #endif /* __LINUX_RCU_H */ diff --git a/kernel/rcupdate.c b/kernel/rcupdate.c index 29ca1c6..a2cf761 100644 --- a/kernel/rcupdate.c +++ b/kernel/rcupdate.c @@ -46,12 +46,15 @@ #include #include #include +#include #define CREATE_TRACE_POINTS #include #include "rcu.h" +module_param(rcu_expedited, int, 0); + #ifdef CONFIG_PREEMPT_RCU /* diff --git a/kernel/rcutiny_plugin.h b/kernel/rcutiny_plugin.h index 3d01902..f85016a 100644 --- a/kernel/rcutiny_plugin.h +++ b/kernel/rcutiny_plugin.h @@ -706,7 +706,10 @@ void synchronize_rcu(void) return; /* Once we get past the fastpath checks, same code as rcu_barrier(). */ - rcu_barrier(); + if (rcu_expedited) + synchronize_rcu_expedited(); + else + rcu_barrier(); } EXPORT_SYMBOL_GPL(synchronize_rcu); diff --git a/kernel/rcutree.c b/kernel/rcutree.c index 74df86b..f9c17c3 100644 --- a/kernel/rcutree.c +++ b/kernel/rcutree.c @@ -2224,7 +2224,10 @@ void synchronize_sched(void) "Illegal synchronize_sched() in RCU-sched read-side critical section"); if (rcu_blocking_is_gp()) return; - wait_rcu_gp(call_rcu_sched); + if (rcu_expedited) + synchronize_sched_expedited(); + else + wait_rcu_gp(call_rcu_sched); } EXPORT_SYMBOL_GPL(synchronize_sched); @@ -2245,7 +2248,10 @@ void synchronize_rcu_bh(void) "Illegal synchronize_rcu_bh() in RCU-bh read-side critical section"); if (rcu_blocking_is_gp()) return; - wait_rcu_gp(call_rcu_bh); + if (rcu_expedited) + synchronize_rcu_bh_expedited(); + else + wait_rcu_gp(call_rcu_bh); } EXPORT_SYMBOL_GPL(synchronize_rcu_bh); @@ -2328,7 +2334,7 @@ void synchronize_sched_expedited(void) if (trycount++ < 10) { udelay(trycount * num_online_cpus()); } else { - synchronize_sched(); + wait_rcu_gp(call_rcu_sched); return; } diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h index f921154..c177ba0 100644 --- a/kernel/rcutree_plugin.h +++ b/kernel/rcutree_plugin.h @@ -679,7 +679,10 @@ void synchronize_rcu(void) "Illegal synchronize_rcu() in RCU read-side critical section"); if (!rcu_scheduler_active) return; - wait_rcu_gp(call_rcu); + if (rcu_expedited) + synchronize_rcu_expedited(); + else + wait_rcu_gp(call_rcu); } EXPORT_SYMBOL_GPL(synchronize_rcu); @@ -831,7 +834,7 @@ void synchronize_rcu_expedited(void) udelay(trycount * num_online_cpus()); } else { put_online_cpus(); - synchronize_rcu(); + wait_rcu_gp(call_rcu); return; } } diff --git a/kernel/srcu.c b/kernel/srcu.c index 97c465e..de90740 100644 --- a/kernel/srcu.c +++ b/kernel/srcu.c @@ -34,6 +34,10 @@ #include #include +#include + +#include "rcu.h" + /* * Initialize an rcu_batch structure to empty. */ @@ -464,7 +468,9 @@ static void __synchronize_srcu(struct srcu_struct *sp, int trycount) */ void synchronize_srcu(struct srcu_struct *sp) { - __synchronize_srcu(sp, SYNCHRONIZE_SRCU_TRYCOUNT); + __synchronize_srcu(sp, rcu_expedited + ? SYNCHRONIZE_SRCU_EXP_TRYCOUNT + : SYNCHRONIZE_SRCU_TRYCOUNT); } EXPORT_SYMBOL_GPL(synchronize_srcu);