From patchwork Sat Apr 14 16:20:33 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 7819 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id E701723E42 for ; Sat, 14 Apr 2012 16:21:26 +0000 (UTC) Received: from mail-iy0-f180.google.com (mail-iy0-f180.google.com [209.85.210.180]) by fiordland.canonical.com (Postfix) with ESMTP id AF9AAA185AE for ; Sat, 14 Apr 2012 16:21:26 +0000 (UTC) Received: by mail-iy0-f180.google.com with SMTP id e36so7572362iag.11 for ; Sat, 14 Apr 2012 09:21:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf:from:to:cc :subject:date:message-id:x-mailer:in-reply-to:references :x-content-scanned:x-cbid:x-gm-message-state; bh=pMJ6Wa6ZkM1dPutXEV1kFvwOlIkPcUMOTeWNIUTK3Jk=; b=ShgSMuTXRqFR5p0RhIAOFGQpYMJjI+GxbGn/4ZWS2jNgVDz/nhmzWkqBVCJEoFo+An Azpx+sjz8r+nAJl1wVIi8IBa4xSsWA2nsSkCJS/ZEY2OaZosbdB8ZeCCboaCtrFMFuuu g1Ta89JWQ1ThMOCSmA7GgrBqvDDFvbL2UbkMZFTLvxju4FD719UfeOuPCiM3i+RKTcsv 5r5lCUb2RHzrx2oRQTgYgn+tipR0fIF/SKcLN339kRW7EBVvKI7GSOHqcY6wENDdH40X hSkA8S/COA8gbbNPaEJ/zJaLbqczTaJK2jl7TULJew7xhcpYX1422MmnecJ7xXsrK0lY YYJg== Received: by 10.42.141.72 with SMTP id n8mr3522015icu.47.1334420486481; Sat, 14 Apr 2012 09:21:26 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.231.70.69 with SMTP id c5csp89605ibj; Sat, 14 Apr 2012 09:21:26 -0700 (PDT) Received: by 10.60.2.6 with SMTP id 6mr7751010oeq.21.1334420485768; Sat, 14 Apr 2012 09:21:25 -0700 (PDT) Received: from e37.co.us.ibm.com (e37.co.us.ibm.com. [32.97.110.158]) by mx.google.com with ESMTPS id i9si6476975oba.96.2012.04.14.09.21.25 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 14 Apr 2012 09:21:25 -0700 (PDT) Received-SPF: pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.110.158 as permitted sender) client-ip=32.97.110.158; Authentication-Results: mx.google.com; spf=pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.110.158 as permitted sender) smtp.mail=paulmck@linux.vnet.ibm.com Received: from /spool/local by e37.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Sat, 14 Apr 2012 10:21:25 -0600 Received: from d03dlp02.boulder.ibm.com (9.17.202.178) by e37.co.us.ibm.com (192.168.1.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Sat, 14 Apr 2012 10:20:48 -0600 Received: from d03relay05.boulder.ibm.com (d03relay05.boulder.ibm.com [9.17.195.107]) by d03dlp02.boulder.ibm.com (Postfix) with ESMTP id B785C3E40049; Sat, 14 Apr 2012 10:20:46 -0600 (MDT) Received: from d03av01.boulder.ibm.com (d03av01.boulder.ibm.com [9.17.195.167]) by d03relay05.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q3EGKiv7220852; Sat, 14 Apr 2012 10:20:45 -0600 Received: from d03av01.boulder.ibm.com (loopback [127.0.0.1]) by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q3EGKghG025758; Sat, 14 Apr 2012 10:20:43 -0600 Received: from paulmck-ThinkPad-W500 ([9.49.223.21]) by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id q3EGKeUB025671; Sat, 14 Apr 2012 10:20:41 -0600 Received: by paulmck-ThinkPad-W500 (Postfix, from userid 1000) id CDC17E5268; Sat, 14 Apr 2012 09:20:39 -0700 (PDT) From: "Paul E. McKenney" To: linux-kernel@vger.kernel.org Cc: mingo@elte.hu, laijs@cn.fujitsu.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@polymtl.ca, josh@joshtriplett.org, niv@us.ibm.com, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, Valdis.Kletnieks@vt.edu, dhowells@redhat.com, eric.dumazet@gmail.com, darren@dvhart.com, fweisbec@gmail.com, patches@linaro.org, torvalds@linux-foundation.org, "Paul E. McKenney" , "Paul E. McKenney" Subject: [PATCH RFC 3/7] rcu: Make exit_rcu() more precise and consolidate Date: Sat, 14 Apr 2012 09:20:33 -0700 Message-Id: <1334420437-19264-3-git-send-email-paulmck@linux.vnet.ibm.com> X-Mailer: git-send-email 1.7.8 In-Reply-To: <1334420437-19264-1-git-send-email-paulmck@linux.vnet.ibm.com> References: <20120414161953.GA18140@linux.vnet.ibm.com> <1334420437-19264-1-git-send-email-paulmck@linux.vnet.ibm.com> X-Content-Scanned: Fidelis XPS MAILER x-cbid: 12041416-7408-0000-0000-0000043343AB X-Gm-Message-State: ALoCoQlYIAAWCoNCKwbTjKYtxS2ATXnqFfduY6Yh8LKCN2LcNLqj2zqZvKB+0P7dXKmJF6IpfNe8 From: "Paul E. McKenney" When running preemptible RCU, if a task exits in an RCU read-side critical section having blocked within that same RCU read-side critical section, the task must be removed from the list of tasks blocking a grace period (perhaps the current grace period, perhaps the next grace period, depending on timing). The exit() path invokes exit_rcu() to do this cleanup. However, the current implementation of exit_rcu() needlessly does the cleanup even if the task did not block within the current RCU read-side critical section, which wastes time and needlessly increases the size of the state space. Fix this by only doing the cleanup if the current task is actually on the list of tasks blocking some grace period. While we are at it, consolidate the two identical exit_rcu() functions into a single function. Signed-off-by: Paul E. McKenney Signed-off-by: Paul E. McKenney --- include/linux/rcupdate.h | 1 + include/linux/rcutiny.h | 5 ----- include/linux/rcutree.h | 12 ------------ kernel/rcupdate.c | 27 ++++++++++++++++++++++++++- kernel/rcutiny_plugin.h | 16 ---------------- kernel/rcutree_plugin.h | 16 ---------------- 6 files changed, 27 insertions(+), 50 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 1cf19ef..5a94dcf 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -193,6 +193,7 @@ extern void rcu_idle_enter(void); extern void rcu_idle_exit(void); extern void rcu_irq_enter(void); extern void rcu_irq_exit(void); +extern void exit_rcu(void); /** * RCU_NONIDLE - Indicate idle-loop code that needs RCU readers diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h index 080b5bd..adb5e5a 100644 --- a/include/linux/rcutiny.h +++ b/include/linux/rcutiny.h @@ -87,10 +87,6 @@ static inline void kfree_call_rcu(struct rcu_head *head, #ifdef CONFIG_TINY_RCU -static inline void exit_rcu(void) -{ -} - static inline int rcu_needs_cpu(int cpu) { return 0; @@ -98,7 +94,6 @@ static inline int rcu_needs_cpu(int cpu) #else /* #ifdef CONFIG_TINY_RCU */ -extern void exit_rcu(void); int rcu_preempt_needs_cpu(void); static inline int rcu_needs_cpu(int cpu) diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h index e8ee5dd..782a8ab 100644 --- a/include/linux/rcutree.h +++ b/include/linux/rcutree.h @@ -45,18 +45,6 @@ static inline void rcu_virt_note_context_switch(int cpu) rcu_note_context_switch(cpu); } -#ifdef CONFIG_TREE_PREEMPT_RCU - -extern void exit_rcu(void); - -#else /* #ifdef CONFIG_TREE_PREEMPT_RCU */ - -static inline void exit_rcu(void) -{ -} - -#endif /* #else #ifdef CONFIG_TREE_PREEMPT_RCU */ - extern void synchronize_rcu_bh(void); extern void synchronize_sched_expedited(void); extern void synchronize_rcu_expedited(void); diff --git a/kernel/rcupdate.c b/kernel/rcupdate.c index eb5d160..1acf0da 100644 --- a/kernel/rcupdate.c +++ b/kernel/rcupdate.c @@ -54,7 +54,32 @@ #ifdef CONFIG_PREEMPT_RCU DEFINE_PER_CPU(int, rcu_read_lock_nesting); DEFINE_PER_CPU(int, rcu_read_unlock_special); -#endif /* #ifdef CONFIG_PREEMPT_RCU */ + +/* + * Check for a task exiting while in a preemptible-RCU read-side + * critical section, clean up if so. No need to issue warnings, + * as debug_check_no_locks_held() already does this if lockdep + * is enabled. + */ +void exit_rcu(void) +{ + struct task_struct *t = current; + + if (likely(list_empty(¤t->rcu_node_entry))) + return; + t->rcu_read_lock_nesting = 1; + barrier(); + t->rcu_read_unlock_special = RCU_READ_UNLOCK_BLOCKED; + __rcu_read_unlock(); +} + +#else /* #ifdef CONFIG_PREEMPT_RCU */ + +void exit_rcu(void) +{ +} + +#endif /* #else #ifdef CONFIG_PREEMPT_RCU */ #ifdef CONFIG_DEBUG_LOCK_ALLOC static struct lock_class_key rcu_lock_key; diff --git a/kernel/rcutiny_plugin.h b/kernel/rcutiny_plugin.h index ff7ec65..66a542a 100644 --- a/kernel/rcutiny_plugin.h +++ b/kernel/rcutiny_plugin.h @@ -851,22 +851,6 @@ int rcu_preempt_needs_cpu(void) return rcu_preempt_ctrlblk.rcb.rcucblist != NULL; } -/* - * Check for a task exiting while in a preemptible -RCU read-side - * critical section, clean up if so. No need to issue warnings, - * as debug_check_no_locks_held() already does this if lockdep - * is enabled. - */ -void exit_rcu(void) -{ - struct task_struct *t = current; - - if (t->rcu_read_lock_nesting == 0) - return; - t->rcu_read_lock_nesting = 1; - __rcu_read_unlock(); -} - #else /* #ifdef CONFIG_TINY_PREEMPT_RCU */ #ifdef CONFIG_RCU_TRACE diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h index f60b315..76a1ba9 100644 --- a/kernel/rcutree_plugin.h +++ b/kernel/rcutree_plugin.h @@ -969,22 +969,6 @@ static void __init __rcu_init_preempt(void) rcu_init_one(&rcu_preempt_state, &rcu_preempt_data); } -/* - * Check for a task exiting while in a preemptible-RCU read-side - * critical section, clean up if so. No need to issue warnings, - * as debug_check_no_locks_held() already does this if lockdep - * is enabled. - */ -void exit_rcu(void) -{ - struct task_struct *t = current; - - if (t->rcu_read_lock_nesting == 0) - return; - t->rcu_read_lock_nesting = 1; - __rcu_read_unlock(); -} - #else /* #ifdef CONFIG_TREE_PREEMPT_RCU */ static struct rcu_state *rcu_state = &rcu_sched_state;