From patchwork Sat Jan 9 02:05:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 360427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0986DC433E0 for ; Sat, 9 Jan 2021 02:07:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C92AA23A3C for ; Sat, 9 Jan 2021 02:07:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726427AbhAICGZ (ORCPT ); Fri, 8 Jan 2021 21:06:25 -0500 Received: from mail.kernel.org ([198.145.29.99]:41438 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725835AbhAICGY (ORCPT ); Fri, 8 Jan 2021 21:06:24 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id CBE5523AA3; Sat, 9 Jan 2021 02:05:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1610157944; bh=v791WB2v2utt682YfDHd4SulRJUpxAa2zQeuc72C4kg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GkPX/mOR3gDRm7NSey1D4Du9WsvRK6JUlmC/slaUymT+g/benNfPlygpMEK5WD3MO 7l4m9qm6UN+/iCupRdK4R5+e+yZZ1bJBKCF84LuaxmXmhxIhFn6zOSBe63W/OYYkPj 3BCrwKkiX8wAU82qJx+3q1Q8BXRMfqm+Amyr9zWc4gN/WbuYI3t2WWPul8I6ZoXD/V k5+NtL3JiiauD16DZssaed2jTOOymRat9jY/p5189TPTWFYwjDQKIn/k8mQOL1+dqP fGOpKPKI6llBSvy5htk9FR9YNVPbQ/d7PdygMl85b8pgUMoA6r289rA4SJjgrMf5+l hi3wGawtZDoUw== From: Frederic Weisbecker To: Peter Zijlstra , "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , "Rafael J . Wysocki" , Ingo Molnar , Thomas Gleixner , stable@vger.kernel.org Subject: [RFC PATCH 1/8] rcu: Remove superfluous rdp fetch Date: Sat, 9 Jan 2021 03:05:29 +0100 Message-Id: <20210109020536.127953-2-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210109020536.127953-1-frederic@kernel.org> References: <20210109020536.127953-1-frederic@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org Signed-off-by: Frederic Weisbecker Cc: Paul E. McKenney Cc: Rafael J. Wysocki Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar --- kernel/rcu/tree.c | 1 - 1 file changed, 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 40e5e3dd253e..fef90c467670 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -643,7 +643,6 @@ static noinstr void rcu_eqs_enter(bool user) instrumentation_begin(); trace_rcu_dyntick(TPS("Start"), rdp->dynticks_nesting, 0, atomic_read(&rdp->dynticks)); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current)); - rdp = this_cpu_ptr(&rcu_data); do_nocb_deferred_wakeup(rdp); rcu_prepare_for_idle(); rcu_preempt_deferred_qs(current); From patchwork Sat Jan 9 02:05:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 360428 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44EC6C43331 for ; Sat, 9 Jan 2021 02:06:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2073523B02 for ; Sat, 9 Jan 2021 02:06:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726500AbhAICG1 (ORCPT ); Fri, 8 Jan 2021 21:06:27 -0500 Received: from mail.kernel.org ([198.145.29.99]:41464 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725835AbhAICG1 (ORCPT ); Fri, 8 Jan 2021 21:06:27 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id EDFDC23AA8; Sat, 9 Jan 2021 02:05:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1610157946; bh=1SrBTlbPzwqTruhS60yTCVOH5SjTUATV2RzaG5lWSgg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=td22Phc4eRAflXnP8TT3AyoVYzQq7GiKGreDRwkxOaEMscBkDWrkwDOM79Lor+6oS e9kc/bsLYoapFPi5yCnaYSTG69IFjJYpmpPmITqCH8oPuSnwfp42rLddHTemL5jECF iM7anToMEedQpVqNK+rPEZnHZzix32DV3gyOK+9Qb8i2qFltnPfxH26sG/nuNfytH9 t9DcRgeGyBRVcblXAnquvpqGu0qwpsLB27r7VH77+Pvn43LRDi1tRxmEanFf1NIGB5 KWqcCTLqE+Knmxj4wTOpjYmxpvUC9icfgkbiExLdD3/arta4/V9BpuYrqy0mjXJhAA Mp7hAzBHnIqhA== From: Frederic Weisbecker To: Peter Zijlstra , "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , "Rafael J . Wysocki" , Ingo Molnar , Thomas Gleixner , stable@vger.kernel.org Subject: [RFC PATCH 2/8] rcu: Pull deferred rcuog wake up to rcu_eqs_enter() callers Date: Sat, 9 Jan 2021 03:05:30 +0100 Message-Id: <20210109020536.127953-3-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210109020536.127953-1-frederic@kernel.org> References: <20210109020536.127953-1-frederic@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org Deferred wakeup of rcuog kthreads upon RCU idle mode entry is going to be handled differently whether initiated by idle, user or guest. Prepare with pulling that control up to rcu_eqs_enter() callers. Signed-off-by: Frederic Weisbecker Cc: Paul E. McKenney Cc: Rafael J. Wysocki Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar --- kernel/rcu/tree.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index fef90c467670..b9fff18d14d9 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -643,7 +643,6 @@ static noinstr void rcu_eqs_enter(bool user) instrumentation_begin(); trace_rcu_dyntick(TPS("Start"), rdp->dynticks_nesting, 0, atomic_read(&rdp->dynticks)); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current)); - do_nocb_deferred_wakeup(rdp); rcu_prepare_for_idle(); rcu_preempt_deferred_qs(current); @@ -671,7 +670,10 @@ static noinstr void rcu_eqs_enter(bool user) */ void rcu_idle_enter(void) { + struct rcu_data *rdp = this_cpu_ptr(&rcu_data); + lockdep_assert_irqs_disabled(); + do_nocb_deferred_wakeup(rdp); rcu_eqs_enter(false); } EXPORT_SYMBOL_GPL(rcu_idle_enter); @@ -690,7 +692,10 @@ EXPORT_SYMBOL_GPL(rcu_idle_enter); */ noinstr void rcu_user_enter(void) { + struct rcu_data *rdp = this_cpu_ptr(&rcu_data); + lockdep_assert_irqs_disabled(); + do_nocb_deferred_wakeup(rdp); rcu_eqs_enter(true); } #endif /* CONFIG_NO_HZ_FULL */ From patchwork Sat Jan 9 02:05:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 360168 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F9FEC43332 for ; Sat, 9 Jan 2021 02:06:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 07F5723A3C for ; Sat, 9 Jan 2021 02:06:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726578AbhAICGa (ORCPT ); Fri, 8 Jan 2021 21:06:30 -0500 Received: from mail.kernel.org ([198.145.29.99]:41478 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726571AbhAICG3 (ORCPT ); Fri, 8 Jan 2021 21:06:29 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id 185C023AC1; Sat, 9 Jan 2021 02:05:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1610157948; bh=DmtAyxnXnAsrifI9+yfmddVBH+zgcl5C6QweDwHY7cc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UWmX/ewLt97KeYAkf1nu+jZeVUHO/YqX78CIF8XLQakwwC1JwJ1k3U8HkTPGoXjqP EWLVeYpMQW/aFDH6uz3Q8oRKR+sKYHCYrVY3q7jC5NsSMS/t/B8dSLQmLWhq6ec0od EbERWFzb0PCS6IFxzJrcQecV8l1hcM/WXytIXrveUpctxEgo5YGLaO+sPEUg/ix8l0 +gfCkEvjew3swNFk6laFX3NdgbsBeQ7wa110m3yVTO2Va+a2GoPPwguJQgl7QnBnIA GRW5tsW8IpOphG3CBjiwtHfH/Fov2FK3AiNZGRfxzzRVFLaQZQWhxUR6VjndP9ZIGo El4sG3WD10R1Q== From: Frederic Weisbecker To: Peter Zijlstra , "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , "Rafael J . Wysocki" , Ingo Molnar , Thomas Gleixner , stable@vger.kernel.org Subject: [RFC PATCH 3/8] rcu/nocb: Perform deferred wake up before last idle's need_resched() check Date: Sat, 9 Jan 2021 03:05:31 +0100 Message-Id: <20210109020536.127953-4-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210109020536.127953-1-frederic@kernel.org> References: <20210109020536.127953-1-frederic@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org Entering RCU idle mode may cause a deferred wake up of an RCU NOCB_GP kthread (rcuog) to be serviced. Usually a local wake up happening while running the idle task is handled in one of the need_resched() checks carefully placed within the idle loop that can break to the scheduler. Unfortunately the call to rcu_idle_enter() is already beyond the last generic need_resched() check and we may halt the CPU with a resched request unhandled, leaving the task hanging. Fix this with splitting the rcuog wakeup handling from rcu_idle_enter() and place it before the last generic need_resched() check in the idle loop. It is then assumed that no call to call_rcu() will be performed after that in the idle loop until the CPU is put in low power mode. Further debug code will help spotting the offenders. Reported-by: Paul E. McKenney Fixes: 96d3fd0d315a (rcu: Break call_rcu() deadlock involving scheduler and perf) Cc: stable@vger.kernel.org Cc: Rafael J. Wysocki Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Signed-off-by: Frederic Weisbecker --- include/linux/rcupdate.h | 2 ++ kernel/rcu/tree.c | 3 --- kernel/rcu/tree_plugin.h | 5 +++++ kernel/sched/idle.c | 3 +++ 4 files changed, 10 insertions(+), 3 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index de0826411311..4068234fb303 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -104,8 +104,10 @@ static inline void rcu_user_exit(void) { } #ifdef CONFIG_RCU_NOCB_CPU void rcu_init_nohz(void); +void rcu_nocb_flush_deferred_wakeup(void); #else /* #ifdef CONFIG_RCU_NOCB_CPU */ static inline void rcu_init_nohz(void) { } +static inline void rcu_nocb_flush_deferred_wakeup(void) { } #endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */ /** diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index b9fff18d14d9..b6e1377774e3 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -670,10 +670,7 @@ static noinstr void rcu_eqs_enter(bool user) */ void rcu_idle_enter(void) { - struct rcu_data *rdp = this_cpu_ptr(&rcu_data); - lockdep_assert_irqs_disabled(); - do_nocb_deferred_wakeup(rdp); rcu_eqs_enter(false); } EXPORT_SYMBOL_GPL(rcu_idle_enter); diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 7e291ce0a1d6..d5b38c28abd1 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -2187,6 +2187,11 @@ static void do_nocb_deferred_wakeup(struct rcu_data *rdp) do_nocb_deferred_wakeup_common(rdp); } +void rcu_nocb_flush_deferred_wakeup(void) +{ + do_nocb_deferred_wakeup(this_cpu_ptr(&rcu_data)); +} + void __init rcu_init_nohz(void) { int cpu; diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c index 305727ea0677..b601a3aa2152 100644 --- a/kernel/sched/idle.c +++ b/kernel/sched/idle.c @@ -55,6 +55,7 @@ __setup("hlt", cpu_idle_nopoll_setup); static noinline int __cpuidle cpu_idle_poll(void) { trace_cpu_idle(0, smp_processor_id()); + rcu_nocb_flush_deferred_wakeup(); stop_critical_timings(); rcu_idle_enter(); local_irq_enable(); @@ -173,6 +174,8 @@ static void cpuidle_idle_call(void) struct cpuidle_driver *drv = cpuidle_get_cpu_driver(dev); int next_state, entered_state; + rcu_nocb_flush_deferred_wakeup(); + /* * Check if the idle task must be rescheduled. If it is the * case, exit the function after re-enabling the local irq. From patchwork Sat Jan 9 02:05:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 360167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A9AAC4332D for ; Sat, 9 Jan 2021 02:06:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E6C1523B09 for ; Sat, 9 Jan 2021 02:06:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726642AbhAICGc (ORCPT ); Fri, 8 Jan 2021 21:06:32 -0500 Received: from mail.kernel.org ([198.145.29.99]:41500 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726571AbhAICGb (ORCPT ); Fri, 8 Jan 2021 21:06:31 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id 37FCA23AC2; Sat, 9 Jan 2021 02:05:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1610157950; bh=QHfGc37y+RJrvHoN5PghJVWpKd0a96DMtNk0V+pDWzU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WMc+KZAtrsmNDDcqxe+XSFxq5ypEOt1JbOY2t9CfEO4fZHof9U1yifzQJe74DvKjv gApJGnRVxYcwYV5Cn7UFLyoD3KYySEjvrs1VYl3ptFcJH+YWDkqlu/pRd0eWSBKeM7 dTiCLVhsvBJl9LU7haVmjIITosHYDHYt05kC9VXtSIJ7piZ0EK6at4cMeIMHLJGOBJ r0ERgkx07SXSjI9SEO96S99VNIPUhxYmQPZIfHQh7W9/JmoNHX3OheTa20UCOUTIP5 9ShxNJMWn65me9H7vWDJrseKcbNuyQvHup+NcSz/D+vmk3eCDaTSNmt/WPiKFmsvxo tE3rI+wA4Aayg== From: Frederic Weisbecker To: Peter Zijlstra , "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , "Rafael J . Wysocki" , Ingo Molnar , Thomas Gleixner , stable@vger.kernel.org Subject: [RFC PATCH 4/8] rcu/nocb: Trigger self-IPI on late deferred wake up before user resume Date: Sat, 9 Jan 2021 03:05:32 +0100 Message-Id: <20210109020536.127953-5-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210109020536.127953-1-frederic@kernel.org> References: <20210109020536.127953-1-frederic@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org Entering RCU idle mode may cause a deferred wake up of an RCU NOCB_GP kthread (rcuog) to be serviced. Unfortunately the call to rcu_user_enter() is already past the last rescheduling opportunity before we resume to userspace or to guest mode. We may escape there with the woken task ignored. The ultimate resort to fix every callsites is to trigger a self-IPI (nohz_full depends on IRQ_WORK) that will trigger a reschedule on IRQ tail or guest exit. Eventually every site that want a saner treatment will need to carefully place a call to rcu_nocb_flush_deferred_wakeup() before the last explicit need_resched() check upon resume. Reported-by: Paul E. McKenney Fixes: 96d3fd0d315a (rcu: Break call_rcu() deadlock involving scheduler and perf) Cc: stable@vger.kernel.org Cc: Rafael J. Wysocki Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree.c | 22 +++++++++++++++++++++- kernel/rcu/tree.h | 2 +- kernel/rcu/tree_plugin.h | 25 ++++++++++++++++--------- 3 files changed, 38 insertions(+), 11 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index b6e1377774e3..2920dfc9f58c 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -676,6 +676,18 @@ void rcu_idle_enter(void) EXPORT_SYMBOL_GPL(rcu_idle_enter); #ifdef CONFIG_NO_HZ_FULL + +/* + * An empty function that will trigger a reschedule on + * IRQ tail once IRQs get re-enabled on userspace resume. + */ +static void late_wakeup_func(struct irq_work *work) +{ +} + +static DEFINE_PER_CPU(struct irq_work, late_wakeup_work) = + IRQ_WORK_INIT(late_wakeup_func); + /** * rcu_user_enter - inform RCU that we are resuming userspace. * @@ -692,9 +704,17 @@ noinstr void rcu_user_enter(void) struct rcu_data *rdp = this_cpu_ptr(&rcu_data); lockdep_assert_irqs_disabled(); - do_nocb_deferred_wakeup(rdp); + /* + * We may be past the last rescheduling opportunity in the entry code. + * Trigger a self IPI that will fire and reschedule once we resume to + * user/guest mode. + */ + if (do_nocb_deferred_wakeup(rdp) && need_resched()) + irq_work_queue(this_cpu_ptr(&late_wakeup_work)); + rcu_eqs_enter(true); } + #endif /* CONFIG_NO_HZ_FULL */ /** diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 7708ed161f4a..9226f4021a36 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -433,7 +433,7 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty, unsigned long flags); static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp); -static void do_nocb_deferred_wakeup(struct rcu_data *rdp); +static bool do_nocb_deferred_wakeup(struct rcu_data *rdp); static void rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp); static void rcu_spawn_cpu_nocb_kthread(int cpu); static void __init rcu_spawn_nocb_kthreads(void); diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index d5b38c28abd1..384856e4d13e 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -1631,8 +1631,8 @@ bool rcu_is_nocb_cpu(int cpu) * Kick the GP kthread for this NOCB group. Caller holds ->nocb_lock * and this function releases it. */ -static void wake_nocb_gp(struct rcu_data *rdp, bool force, - unsigned long flags) +static bool wake_nocb_gp(struct rcu_data *rdp, bool force, + unsigned long flags) __releases(rdp->nocb_lock) { bool needwake = false; @@ -1643,7 +1643,7 @@ static void wake_nocb_gp(struct rcu_data *rdp, bool force, trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("AlreadyAwake")); rcu_nocb_unlock_irqrestore(rdp, flags); - return; + return false; } del_timer(&rdp->nocb_timer); rcu_nocb_unlock_irqrestore(rdp, flags); @@ -1656,6 +1656,8 @@ static void wake_nocb_gp(struct rcu_data *rdp, bool force, raw_spin_unlock_irqrestore(&rdp_gp->nocb_gp_lock, flags); if (needwake) wake_up_process(rdp_gp->nocb_gp_kthread); + + return needwake; } /* @@ -2152,20 +2154,23 @@ static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp) } /* Do a deferred wakeup of rcu_nocb_kthread(). */ -static void do_nocb_deferred_wakeup_common(struct rcu_data *rdp) +static bool do_nocb_deferred_wakeup_common(struct rcu_data *rdp) { unsigned long flags; int ndw; + int ret; rcu_nocb_lock_irqsave(rdp, flags); if (!rcu_nocb_need_deferred_wakeup(rdp)) { rcu_nocb_unlock_irqrestore(rdp, flags); - return; + return false; } ndw = READ_ONCE(rdp->nocb_defer_wakeup); WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOCB_WAKE_NOT); - wake_nocb_gp(rdp, ndw == RCU_NOCB_WAKE_FORCE, flags); + ret = wake_nocb_gp(rdp, ndw == RCU_NOCB_WAKE_FORCE, flags); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("DeferredWake")); + + return ret; } /* Do a deferred wakeup of rcu_nocb_kthread() from a timer handler. */ @@ -2181,10 +2186,11 @@ static void do_nocb_deferred_wakeup_timer(struct timer_list *t) * This means we do an inexact common-case check. Note that if * we miss, ->nocb_timer will eventually clean things up. */ -static void do_nocb_deferred_wakeup(struct rcu_data *rdp) +static bool do_nocb_deferred_wakeup(struct rcu_data *rdp) { if (rcu_nocb_need_deferred_wakeup(rdp)) - do_nocb_deferred_wakeup_common(rdp); + return do_nocb_deferred_wakeup_common(rdp); + return false; } void rcu_nocb_flush_deferred_wakeup(void) @@ -2523,8 +2529,9 @@ static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp) return false; } -static void do_nocb_deferred_wakeup(struct rcu_data *rdp) +static bool do_nocb_deferred_wakeup(struct rcu_data *rdp) { + return false; } static void rcu_spawn_cpu_nocb_kthread(int cpu) From patchwork Sat Jan 9 02:05:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 360166 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22958C433DB for ; Sat, 9 Jan 2021 02:07:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ECBCD23A9B for ; Sat, 9 Jan 2021 02:07:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726346AbhAICHE (ORCPT ); Fri, 8 Jan 2021 21:07:04 -0500 Received: from mail.kernel.org ([198.145.29.99]:41576 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725835AbhAICHD (ORCPT ); Fri, 8 Jan 2021 21:07:03 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id 5449B23A9B; Sat, 9 Jan 2021 02:05:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1610157953; bh=JV1534qKdNdXiG675FUGgE+Nz8wK9zsXkCHXh1nZizc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=drl43+zVX0KqpUq9BrF1GpSdmkuWQMlFTKvcGqHkgii0KRulkbRdjzebacM9Miyl6 FThUSOKZeANzDjugEfkods3Xf5dkLBchzWYJW/18kOvu0llCnYuhdn6MxF998D45nN zvB+WN0eDwe3Jv8aspSBSUovfkhSb128/5Rvj+83UnNCcURPW/wgiJ6emsGfdthCQ7 FlqH3yJm46O43uzA0XSOCQtH0fJHWhSXQEeVJtJLJyBR6Xz4r/HAfmHkP2JxL1H8k1 0LdMFMl0V6QUcQXkjBYRxuJ2YsLB3ugMVDqgGxQorRoIJNEaGOAB4paI5Z5srdTHCe 1KAhk2pA/vtHQ== From: Frederic Weisbecker To: Peter Zijlstra , "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , "Rafael J . Wysocki" , Ingo Molnar , Thomas Gleixner , stable@vger.kernel.org Subject: [RFC PATCH 5/8] entry: Explicitly flush pending rcuog wakeup before last rescheduling points Date: Sat, 9 Jan 2021 03:05:33 +0100 Message-Id: <20210109020536.127953-6-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210109020536.127953-1-frederic@kernel.org> References: <20210109020536.127953-1-frederic@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org Following the idle loop model, cleanly check for pending rcuog wakeup before the last rescheduling point on resuming to user mode. This way we can avoid to do it from rcu_user_enter() with the last resort self-IPI hack that enforces rescheduling. Signed-off-by: Frederic Weisbecker Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Paul E. McKenney Cc: Rafael J. Wysocki --- kernel/entry/common.c | 6 ++++++ kernel/rcu/tree.c | 12 +++++++----- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/kernel/entry/common.c b/kernel/entry/common.c index 378341642f94..8f3292b5f9b7 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -178,6 +178,9 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs *regs, /* Architecture specific TIF work */ arch_exit_to_user_mode_work(regs, ti_work); + /* Check if any of the above work has queued a deferred wakeup */ + rcu_nocb_flush_deferred_wakeup(); + /* * Disable interrupts and reevaluate the work flags as they * might have changed while interrupts and preemption was @@ -197,6 +200,9 @@ static void exit_to_user_mode_prepare(struct pt_regs *regs) lockdep_assert_irqs_disabled(); + /* Flush pending rcuog wakeup before the last need_resched() check */ + rcu_nocb_flush_deferred_wakeup(); + if (unlikely(ti_work & EXIT_TO_USER_MODE_WORK)) ti_work = exit_to_user_mode_loop(regs, ti_work); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 2920dfc9f58c..3c4c0d5cea65 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -705,12 +705,14 @@ noinstr void rcu_user_enter(void) lockdep_assert_irqs_disabled(); /* - * We may be past the last rescheduling opportunity in the entry code. - * Trigger a self IPI that will fire and reschedule once we resume to - * user/guest mode. + * Other than generic entry implementation, we may be past the last + * rescheduling opportunity in the entry code. Trigger a self IPI + * that will fire and reschedule once we resume in user/guest mode. */ - if (do_nocb_deferred_wakeup(rdp) && need_resched()) - irq_work_queue(this_cpu_ptr(&late_wakeup_work)); + if (!IS_ENABLED(CONFIG_GENERIC_ENTRY) || (current->flags & PF_VCPU)) { + if (do_nocb_deferred_wakeup(rdp) && need_resched()) + irq_work_queue(this_cpu_ptr(&late_wakeup_work)); + } rcu_eqs_enter(true); } From patchwork Sat Jan 9 02:05:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 360426 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE10BC433DB for ; Sat, 9 Jan 2021 02:07:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8DC9723A9B for ; Sat, 9 Jan 2021 02:07:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726771AbhAICHF (ORCPT ); Fri, 8 Jan 2021 21:07:05 -0500 Received: from mail.kernel.org ([198.145.29.99]:41578 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725916AbhAICHE (ORCPT ); Fri, 8 Jan 2021 21:07:04 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id 765BD23AC0; Sat, 9 Jan 2021 02:05:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1610157955; bh=IV3Fs0Tk44Jn/JKmGxOFZ56+bWFBTn/p6qGPe5NQbU0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uZt+mIZCXCyB4A5z7v/wHBhDSeAecI+Zk0qMUEmwOzSFTeH5LykuoWLhyLcjJY6fc 3PeKwVK5wGFLGoyBni84W7zlCifL8EHHr3qqUzZr6wmQ3Mm6h064lICfkHwJTD/hCm 6kmyOjcd5YHHGPSkGuwYdi0rtWBRqisKmy6DONhQu9yhxcNdZSXg9GE3wurIRWQLy5 67kWIroDOCgbLXUDvJPWlpx0Du6oQuTTKZtNkk5jZXKqIp0zfdd3bhzYvYUvlWrGAs 2JLgadTjAhxIMI75PIeAfrtZivOyNDbEE9940i7heJxjOkL4/zyDUfv23um+ipCYjr Ro8JCbPpZ/Iig== From: Frederic Weisbecker To: Peter Zijlstra , "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , "Rafael J . Wysocki" , Ingo Molnar , Thomas Gleixner , stable@vger.kernel.org Subject: [RFC PATCH 6/8] sched: Report local wake up on resched blind zone within idle loop Date: Sat, 9 Jan 2021 03:05:34 +0100 Message-Id: <20210109020536.127953-7-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210109020536.127953-1-frederic@kernel.org> References: <20210109020536.127953-1-frederic@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org The idle loop has several need_resched() checks that make sure we don't miss a rescheduling request. This means that any wake up performed on the local runqueue after the last generic need_resched() check is going to have its rescheduling silently ignored. This has happened in the past with rcu kthreads awaken from rcu_idle_enter() for example. Perform sanity checks to report these situations. Signed-off-by: Frederic Weisbecker Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Paul E. McKenney Cc: Rafael J. Wysocki Reported-by: kernel test robot --- include/linux/sched.h | 11 +++++++++++ kernel/sched/core.c | 42 ++++++++++++++++++++++++++++++++++++++++++ kernel/sched/idle.c | 3 +++ kernel/sched/sched.h | 3 +++ 4 files changed, 59 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index 6e3a5eeec509..83fedda54943 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1917,6 +1917,17 @@ static __always_inline bool need_resched(void) return unlikely(tif_need_resched()); } +#ifdef CONFIG_SCHED_DEBUG +extern void sched_resched_local_allow(void); +extern void sched_resched_local_forbid(void); +extern void sched_resched_local_assert_allowed(void); +#else +static inline void sched_resched_local_allow(void) { } +static inline void sched_resched_local_forbid(void) { } +static inline void sched_resched_local_assert_allowed(void) { } +#endif + + /* * Wrappers for p->thread_info->cpu access. No-op on UP. */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 15d2562118d1..6056f0374674 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -591,6 +591,44 @@ void wake_up_q(struct wake_q_head *head) } } +#ifdef CONFIG_SCHED_DEBUG +void noinstr sched_resched_local_allow(void) +{ + this_rq()->resched_local_allow = 1; +} + +void noinstr sched_resched_local_forbid(void) +{ + this_rq()->resched_local_allow = 0; +} + +void noinstr sched_resched_local_assert_allowed(void) +{ + if (this_rq()->resched_local_allow) + return; + + /* + * Idle interrupts break the CPU from its pause and + * rescheduling happens on idle loop exit. + */ + if (in_hardirq()) + return; + + /* + * What applies to hardirq also applies to softirq as + * we assume they execute on hardirq tail. Ksoftirqd + * shouldn't have resched_local_allow == 0. + * We also assume that no local_bh_enable() call may + * execute softirqs inline on fragile idle/entry + * path... + */ + if (in_serving_softirq()) + return; + + WARN_ONCE(1, "Late current task rescheduling may be lost\n"); +} +#endif + /* * resched_curr - mark rq's current task 'to be rescheduled now'. * @@ -613,6 +651,7 @@ void resched_curr(struct rq *rq) if (cpu == smp_processor_id()) { set_tsk_need_resched(curr); set_preempt_need_resched(); + sched_resched_local_assert_allowed(); return; } @@ -7796,6 +7835,9 @@ void __init sched_init(void) #endif /* CONFIG_SMP */ hrtick_rq_init(rq); atomic_set(&rq->nr_iowait, 0); +#ifdef CONFIG_SCHED_DEBUG + rq->resched_local_allow = 1; +#endif } set_load_weight(&init_task, false); diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c index b601a3aa2152..cdffd32812bd 100644 --- a/kernel/sched/idle.c +++ b/kernel/sched/idle.c @@ -185,6 +185,8 @@ static void cpuidle_idle_call(void) return; } + sched_resched_local_forbid(); + /* * The RCU framework needs to be told that we are entering an idle * section, so no more rcu read side critical sections and one more @@ -247,6 +249,7 @@ static void cpuidle_idle_call(void) } exit_idle: + sched_resched_local_allow(); __current_set_polling(); /* diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 12ada79d40f3..a9416c383451 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1060,6 +1060,9 @@ struct rq { #endif unsigned int push_busy; struct cpu_stop_work push_work; +#ifdef CONFIG_SCHED_DEBUG + unsigned int resched_local_allow; +#endif }; #ifdef CONFIG_FAIR_GROUP_SCHED From patchwork Sat Jan 9 02:05:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 360425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC292C433E6 for ; Sat, 9 Jan 2021 02:07:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A41E423AA3 for ; Sat, 9 Jan 2021 02:07:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726826AbhAICHG (ORCPT ); Fri, 8 Jan 2021 21:07:06 -0500 Received: from mail.kernel.org ([198.145.29.99]:41602 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726787AbhAICHF (ORCPT ); Fri, 8 Jan 2021 21:07:05 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id A50BA23AAC; Sat, 9 Jan 2021 02:05:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1610157957; bh=Fzo/QSm35F0Z7LELOpY7VgizteZ3qNLeNfRMdCSQd0M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KnLp8klBGeRrcBUdXeUNkxoySoLM9jFuUgz4No1rubfYcuQE+O6x7CTwUKL5IddD0 CCS5ncNzXNfERbKR9ClrKbfWIP3wx5G/vTaNQTNt2lugxkHBZrFIhea3doEvOA3WVH HsR5920fqt6Ks+RAGRXJ4A67/mh3fiQfjgIxC+H2v6JWVpVp605nUt+YrmBOUnzC4F 6/G0N64MKOa4yLHnsjGiMmr18faEjsr4kz4Yp/lU4l0pa4datekO+z6I8pHaiBhuFF vY95wYvTlmU0bPlMXoiyKWm0naa61Q63MIZBSH+fhkCLV00l6rThL4yjJyZ91WDEOm QXUkO6LzHlJWg== From: Frederic Weisbecker To: Peter Zijlstra , "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , "Rafael J . Wysocki" , Ingo Molnar , Thomas Gleixner , stable@vger.kernel.org Subject: [RFC PATCH 7/8] entry: Report local wake up on resched blind zone while resuming to user Date: Sat, 9 Jan 2021 03:05:35 +0100 Message-Id: <20210109020536.127953-8-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210109020536.127953-1-frederic@kernel.org> References: <20210109020536.127953-1-frederic@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org The last rescheduling opportunity while resuming to user is in exit_to_user_mode_loop(). This means that any wake up performed on the local runqueue after this point is going to have its rescheduling silently ignored. Perform sanity checks to report these situations. Signed-off-by: Frederic Weisbecker Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Paul E. McKenney Cc: Rafael J. Wysocki Reported-by: kernel test robot --- kernel/entry/common.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/entry/common.c b/kernel/entry/common.c index 8f3292b5f9b7..1dfb97762336 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -5,6 +5,7 @@ #include #include #include +#include #include "common.h" @@ -23,6 +24,8 @@ static __always_inline void __enter_from_user_mode(struct pt_regs *regs) instrumentation_begin(); trace_hardirqs_off_finish(); instrumentation_end(); + + sched_resched_local_allow(); } void noinstr enter_from_user_mode(struct pt_regs *regs) @@ -206,6 +209,7 @@ static void exit_to_user_mode_prepare(struct pt_regs *regs) if (unlikely(ti_work & EXIT_TO_USER_MODE_WORK)) ti_work = exit_to_user_mode_loop(regs, ti_work); + sched_resched_local_forbid(); arch_exit_to_user_mode_prepare(regs, ti_work); /* Ensure that the address limit is intact and no locks are held */ From patchwork Sat Jan 9 02:05:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 360165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E91AEC43381 for ; Sat, 9 Jan 2021 02:07:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C368523A3C for ; Sat, 9 Jan 2021 02:07:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726820AbhAICHG (ORCPT ); Fri, 8 Jan 2021 21:07:06 -0500 Received: from mail.kernel.org ([198.145.29.99]:41604 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726784AbhAICHF (ORCPT ); Fri, 8 Jan 2021 21:07:05 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id C860623AC4; Sat, 9 Jan 2021 02:05:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1610157959; bh=FruuVTRbSSmFOTUvQqJ2CO8k/I1uHvJl6EZqrve6RFA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aC10n/IUNawtRE92wVvr68B5rdZLVdR1L9PuWxnwZZX0xx6GiyVTgOYMHLuBy/AgL dx1mlRuQNDmkbd6skznXFvrMqcVdxv0kfgYVHH/W4GJLj3Zqi0cScFpp9CGIqCHzg+ eFeuSiDBn8HzP70aCllSyE+vHIkTBHdLqpGRkAhEXiw/ymu6rwKuYAOJ7pzvx0lcn8 3vou37KCqPlnHItNHWHmf/d4amypl7lWKqrG5B7gKnZtoTC/ulcKka3xfs9CI6FW0m 0YvL0YkudWdWw7wRbSKzRhI2YzhP1j4ooFLaLEeshcP/LSKh8zmsrj1lJiv5gODKpZ PMUB00CHuXEdg== From: Frederic Weisbecker To: Peter Zijlstra , "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , "Rafael J . Wysocki" , Ingo Molnar , Thomas Gleixner , stable@vger.kernel.org Subject: [RFC PATCH 8/8] timer: Report ignored local enqueue in nohz mode Date: Sat, 9 Jan 2021 03:05:36 +0100 Message-Id: <20210109020536.127953-9-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210109020536.127953-1-frederic@kernel.org> References: <20210109020536.127953-1-frederic@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org Enqueuing a local timer after the tick has been stopped will result in the timer being ignored until the next random interrupt. Perform sanity checks to report these situations. Signed-off-by: Frederic Weisbecker Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Paul E. McKenney Cc: Rafael J. Wysocki --- kernel/sched/core.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 6056f0374674..6c8b04272a9a 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -715,6 +715,26 @@ int get_nohz_timer_target(void) return cpu; } +static void wake_idle_assert_possible(void) +{ +#ifdef CONFIG_SCHED_DEBUG + /* Timers are re-evaluated after idle IRQs */ + if (in_hardirq()) + return; + /* + * Same as hardirqs, assuming they are executing + * on IRQ tail. Ksoftirqd shouldn't reach here + * as the timer base wouldn't be idle. And inline + * softirq processing after a call to local_bh_enable() + * within idle loop sound too fun to be considered here. + */ + if (in_serving_softirq()) + return; + + WARN_ON_ONCE("Late timer enqueue may be ignored\n"); +#endif +} + /* * When add_timer_on() enqueues a timer into the timer wheel of an * idle CPU then this timer might expire before the next timer event @@ -729,8 +749,10 @@ static void wake_up_idle_cpu(int cpu) { struct rq *rq = cpu_rq(cpu); - if (cpu == smp_processor_id()) + if (cpu == smp_processor_id()) { + wake_idle_assert_possible(); return; + } if (set_nr_and_not_polling(rq->idle)) smp_send_reschedule(cpu);