From patchwork Wed Aug 2 13:24:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 710568 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4AFEC001E0 for ; Wed, 2 Aug 2023 13:31:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234051AbjHBNbf (ORCPT ); Wed, 2 Aug 2023 09:31:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59580 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233444AbjHBNbc (ORCPT ); Wed, 2 Aug 2023 09:31:32 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B25D1BFD; Wed, 2 Aug 2023 06:31:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=JAEdkNYRa3AXqsFdc832avh6PD2FjtrGNGx2xJmd/wI=; b=fU6G3YXwQ34Cb1/wQwfYvcumtt LdxhBQ2iLVc8CMCboG1kI/Le/16V4MPrdeVKmPfbuURe5qbMWj5RjiOyNU+Nq4b8NQetB6dPC/SVx 0X/DZFYsbtW4wMTqAdyzw/kYkZpsK5CgMufWNDUJJgcUSYP7tdpOLysCSaq6mBykBA+mtFdznhE1U eYmf+IZ5Tjg5P2CQdVYTBGWwf9qDFsVZtjZ9rHW4cDBosVOdTLhEebk5G0ceo9nIDkSUXOSr9ixQ1 tJhwIjp4FFEyr8Y0BxmXi07xsYgMy/mVdfLiYz8L2Htb4dQxwMFUTu6VUQ4Zrq2K7ChqThiWnJ0gE ZhsV3++Q==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qRBwZ-00FxfC-21; Wed, 02 Aug 2023 13:31:17 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 12B0A300301; Wed, 2 Aug 2023 15:31:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id ECE022107D7E6; Wed, 2 Aug 2023 15:31:14 +0200 (CEST) Message-ID: <20230802132925.549643116@infradead.org> User-Agent: quilt/0.66 Date: Wed, 02 Aug 2023 15:24:32 +0200 From: Peter Zijlstra To: anna-maria@linutronix.de, rafael@kernel.org, tglx@linutronix.de, frederic@kernel.org, gautham.shenoy@amd.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, daniel.lezcano@linaro.org, linux-pm@vger.kernel.org, mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, kajetan.puchalski@arm.com Subject: [RFC][PATCH v2 1/5] tick/nohz: Introduce tick_get_sleep_length() References: <20230802132431.254614419@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Add a variant of tick_nohz_get_sleep_length() that conditionally does the NOHZ part. tick_get_sleep_length(false) returns the delta_next return value of tick_nohz_get_sleep_length(), while tick_get_sleep_length(true) returns the regular return of tick_nohz_get_sleep_length(). This allows eliding tick_nohz_next_event() -- which is going to be expensive with timer-pull. Signed-off-by: Peter Zijlstra (Intel) --- include/linux/tick.h | 5 +++++ kernel/time/tick-sched.c | 35 +++++++++++++++++++++++------------ 2 files changed, 28 insertions(+), 12 deletions(-) --- a/include/linux/tick.h +++ b/include/linux/tick.h @@ -136,6 +136,7 @@ extern void tick_nohz_irq_exit(void); extern bool tick_nohz_idle_got_tick(void); extern ktime_t tick_nohz_get_next_hrtimer(void); extern ktime_t tick_nohz_get_sleep_length(ktime_t *delta_next); +extern ktime_t tick_get_sleep_length(bool nohz); extern unsigned long tick_nohz_get_idle_calls(void); extern unsigned long tick_nohz_get_idle_calls_cpu(int cpu); extern u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time); @@ -168,6 +169,10 @@ static inline ktime_t tick_nohz_get_slee *delta_next = TICK_NSEC; return *delta_next; } +static inline ktime_t tick_get_sleep_length(bool nohz) +{ + return TICK_NSEC; +} static inline u64 get_cpu_idle_time_us(int cpu, u64 *unused) { return -1; } static inline u64 get_cpu_iowait_time_us(int cpu, u64 *unused) { return -1; } --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -1218,17 +1218,7 @@ ktime_t tick_nohz_get_next_hrtimer(void) return __this_cpu_read(tick_cpu_device.evtdev)->next_event; } -/** - * tick_nohz_get_sleep_length - return the expected length of the current sleep - * @delta_next: duration until the next event if the tick cannot be stopped - * - * Called from power state control code with interrupts disabled. - * - * The return value of this function and/or the value returned by it through the - * @delta_next pointer can be negative which must be taken into account by its - * callers. - */ -ktime_t tick_nohz_get_sleep_length(ktime_t *delta_next) +static ktime_t __tick_nohz_get_sleep_length(ktime_t *delta_next, bool nohz) { struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev); struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched); @@ -1244,7 +1234,7 @@ ktime_t tick_nohz_get_sleep_length(ktime *delta_next = ktime_sub(dev->next_event, now); - if (!can_stop_idle_tick(cpu, ts)) + if (!nohz || !can_stop_idle_tick(cpu, ts)) return *delta_next; next_event = tick_nohz_next_event(ts, cpu); @@ -1262,6 +1252,27 @@ ktime_t tick_nohz_get_sleep_length(ktime } /** + * tick_nohz_get_sleep_length - return the expected length of the current sleep + * @delta_next: duration until the next event if the tick cannot be stopped + * + * Called from power state control code with interrupts disabled. + * + * The return value of this function and/or the value returned by it through the + * @delta_next pointer can be negative which must be taken into account by its + * callers. + */ +ktime_t tick_nohz_get_sleep_length(ktime_t *delta_next) +{ + return __tick_nohz_get_sleep_length(delta_next, true); +} + +ktime_t tick_get_sleep_length(bool nohz) +{ + ktime_t delta; + return __tick_nohz_get_sleep_length(&delta, nohz); +} + +/** * tick_nohz_get_idle_calls_cpu - return the current idle calls counter value * for a particular CPU. * From patchwork Wed Aug 2 13:24:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 709474 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E15FC001E0 for ; Wed, 2 Aug 2023 13:31:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231743AbjHBNba (ORCPT ); Wed, 2 Aug 2023 09:31:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231612AbjHBNb3 (ORCPT ); Wed, 2 Aug 2023 09:31:29 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70F8A1BF9; Wed, 2 Aug 2023 06:31:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=/jk8cjAzNp5umM9o/8ZP9pvl8gboqgF2w9reMo9nXqk=; b=rP7UMCWAI6sRygyJ9fEfu2wtlg +1g4wOf8fa5r/s/Y6W9qw52cgpojG2KcQPpmfY/7P78768ajegVsRX48aFbqVG20QhnxuvYG9SJ0S uZf/6cSOgfQCWCQ3N0CZloMXdrKLxmAYwO2Al3+k0mRJtPGI5NTnUYBW0RttPTn663iDq5P1yoKUU 0fogOAMmncYeQNU6LEeoJTCWMIMFp215RTWiG2ItW1PFQxTMOwVLppVf7dFkw2qEnMh2PX5iW+OQS D3qujBk2eh30GOHf0epVr1bMReR4azwigRCde6XkGsqHHf5bz0lfxd+grG8x3iIuJi7O2/PuE6Q8c Ory2eoGA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1qRBwZ-00FBnW-JB; Wed, 02 Aug 2023 13:31:15 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 173A03006E2; Wed, 2 Aug 2023 15:31:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id EF6922066B201; Wed, 2 Aug 2023 15:31:14 +0200 (CEST) Message-ID: <20230802132925.618923837@infradead.org> User-Agent: quilt/0.66 Date: Wed, 02 Aug 2023 15:24:33 +0200 From: Peter Zijlstra To: anna-maria@linutronix.de, rafael@kernel.org, tglx@linutronix.de, frederic@kernel.org, gautham.shenoy@amd.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, daniel.lezcano@linaro.org, linux-pm@vger.kernel.org, mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, kajetan.puchalski@arm.com Subject: [RFC][PATCH v2 2/5] cpuidle: Inject tick boundary state References: <20230802132431.254614419@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org In order to facilitate governors that track history in idle-state buckets (TEO) making a useful decision about NOHZ, make sure we have a bucket that counts tick-and-longer. In order to be inclusive of the tick itself -- after all, if we do not disable NOHZ we'll sleep for a full tick, the actual boundary should be just short of a full tick. IOW, when registering the idle-states, add one that is always disabled, just to have a bucket. Signed-off-by: Peter Zijlstra (Intel) --- drivers/cpuidle/cpuidle.h | 2 + drivers/cpuidle/driver.c | 48 +++++++++++++++++++++++++++++++++++++++++++++- include/linux/cpuidle.h | 2 - 3 files changed, 50 insertions(+), 2 deletions(-) --- a/drivers/cpuidle/cpuidle.h +++ b/drivers/cpuidle/cpuidle.h @@ -72,4 +72,6 @@ static inline void cpuidle_coupled_unreg } #endif +#define SHORT_TICK_NSEC (TICK_NSEC - TICK_NSEC/32) + #endif /* __DRIVER_CPUIDLE_H */ --- a/drivers/cpuidle/driver.c +++ b/drivers/cpuidle/driver.c @@ -147,13 +147,37 @@ static void cpuidle_setup_broadcast_time tick_broadcast_disable(); } +static int tick_enter(struct cpuidle_device *dev, + struct cpuidle_driver *drv, + int index) +{ + return -ENODEV; +} + +static void __cpuidle_state_init_tick(struct cpuidle_state *s) +{ + strcpy(s->name, "TICK"); + strcpy(s->desc, "(no-op)"); + + s->target_residency_ns = SHORT_TICK_NSEC; + s->target_residency = div_u64(SHORT_TICK_NSEC, NSEC_PER_USEC); + + s->exit_latency_ns = 0; + s->exit_latency = 0; + + s->flags |= CPUIDLE_FLAG_UNUSABLE; + + s->enter = tick_enter; + s->enter_s2idle = tick_enter; +} + /** * __cpuidle_driver_init - initialize the driver's internal data * @drv: a valid pointer to a struct cpuidle_driver */ static void __cpuidle_driver_init(struct cpuidle_driver *drv) { - int i; + int tick = 0, i; /* * Use all possible CPUs as the default, because if the kernel boots @@ -163,6 +187,9 @@ static void __cpuidle_driver_init(struct if (!drv->cpumask) drv->cpumask = (struct cpumask *)cpu_possible_mask; + if (WARN_ON_ONCE(drv->state_count >= CPUIDLE_STATE_MAX-2)) + tick = 1; + for (i = 0; i < drv->state_count; i++) { struct cpuidle_state *s = &drv->states[i]; @@ -192,6 +219,25 @@ static void __cpuidle_driver_init(struct s->exit_latency_ns = 0; else s->exit_latency = div_u64(s->exit_latency_ns, NSEC_PER_USEC); + + if (!tick && s->target_residency_ns >= SHORT_TICK_NSEC) { + tick = 1; + + if (s->target_residency_ns == SHORT_TICK_NSEC) + continue; + + memmove(&drv->states[i+1], &drv->states[i], + sizeof(struct cpuidle_state) * (CPUIDLE_STATE_MAX - i - 1)); + __cpuidle_state_init_tick(s); + drv->state_count++; + i++; + } + } + + if (!tick) { + struct cpuidle_state *s = &drv->states[i]; + __cpuidle_state_init_tick(s); + drv->state_count++; } } --- a/include/linux/cpuidle.h +++ b/include/linux/cpuidle.h @@ -16,7 +16,7 @@ #include #include -#define CPUIDLE_STATE_MAX 10 +#define CPUIDLE_STATE_MAX 16 #define CPUIDLE_NAME_LEN 16 #define CPUIDLE_DESC_LEN 32 From patchwork Wed Aug 2 13:24:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 709472 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B79D2C04A6A for ; Wed, 2 Aug 2023 13:31:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233127AbjHBNbi (ORCPT ); Wed, 2 Aug 2023 09:31:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59590 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233772AbjHBNbe (ORCPT ); Wed, 2 Aug 2023 09:31:34 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 97402269A; Wed, 2 Aug 2023 06:31:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=eQsShqCZQhSRZELiV+wvARE7XUysNmI7Pn/FkVFS3UY=; b=CoKaaQvwg2eF7NA5/aiaXSbUSO NKjRN2+A9m+yrIzlnelbVshYusq0wWHEJjIfqOmc2tEHNYIyC3pFm+zlhPbFKFi5aDDI6j5bNAYHl Xo6lGXsqowLKdsPT9y0eWuX9IwL3ppBs+zRNi5rpFJDe9+ZmJftsDpEHfLyaL8tUvu84bDkoWshP2 cFxLMrucsBCZWPIBecpJM7evZxCMfgK2ivTEQo5eFLWUAnviBmdijWpZZzw3iTKIlJ6CM/eSQ1GiF 4InJSRVopZBFp8pMJPyCZDxdhm9KupCLJIcqPpAz3K+V8uzGQBXu1J5kCVMS135vXTCrH0Hwim2tC OALx8RAA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qRBwZ-00Fxf9-1s; Wed, 02 Aug 2023 13:31:16 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 1942E300768; Wed, 2 Aug 2023 15:31:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 00BDA2107D7E7; Wed, 2 Aug 2023 15:31:14 +0200 (CEST) Message-ID: <20230802132925.686745535@infradead.org> User-Agent: quilt/0.66 Date: Wed, 02 Aug 2023 15:24:34 +0200 From: Peter Zijlstra To: anna-maria@linutronix.de, rafael@kernel.org, tglx@linutronix.de, frederic@kernel.org, gautham.shenoy@amd.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, daniel.lezcano@linaro.org, linux-pm@vger.kernel.org, mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, kajetan.puchalski@arm.com Subject: [RFC][PATCH v2 3/5] cpuidle/teo: Simplify a little References: <20230802132431.254614419@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Remove some of the early exit cases that rely on state_count, since we have the additional tick state. Declutters some of the next patches, can possibly be re-instated later if desired. Signed-off-by: Peter Zijlstra (Intel) --- drivers/cpuidle/governors/teo.c | 31 +------------------------------ 1 file changed, 1 insertion(+), 30 deletions(-) --- a/drivers/cpuidle/governors/teo.c +++ b/drivers/cpuidle/governors/teo.c @@ -187,7 +187,6 @@ struct teo_bin { * @next_recent_idx: Index of the next @recent_idx entry to update. * @recent_idx: Indices of bins corresponding to recent "intercepts". * @util_threshold: Threshold above which the CPU is considered utilized - * @utilized: Whether the last sleep on the CPU happened while utilized */ struct teo_cpu { s64 time_span_ns; @@ -197,7 +196,6 @@ struct teo_cpu { int next_recent_idx; int recent_idx[NR_RECENT]; unsigned long util_threshold; - bool utilized; }; static DEFINE_PER_CPU(struct teo_cpu, teo_cpus); @@ -379,33 +377,6 @@ static int teo_select(struct cpuidle_dri duration_ns = tick_nohz_get_sleep_length(&delta_tick); cpu_data->sleep_length_ns = duration_ns; - /* Check if there is any choice in the first place. */ - if (drv->state_count < 2) { - idx = 0; - goto end; - } - if (!dev->states_usage[0].disable) { - idx = 0; - if (drv->states[1].target_residency_ns > duration_ns) - goto end; - } - - cpu_data->utilized = teo_cpu_is_utilized(dev->cpu, cpu_data); - /* - * If the CPU is being utilized over the threshold and there are only 2 - * states to choose from, the metrics need not be considered, so choose - * the shallowest non-polling state and exit. - */ - if (drv->state_count < 3 && cpu_data->utilized) { - for (i = 0; i < drv->state_count; ++i) { - if (!dev->states_usage[i].disable && - !(drv->states[i].flags & CPUIDLE_FLAG_POLLING)) { - idx = i; - goto end; - } - } - } - /* * Find the deepest idle state whose target residency does not exceed * the current sleep length and the deepest idle state not deeper than @@ -541,7 +512,7 @@ static int teo_select(struct cpuidle_dri * If the CPU is being utilized over the threshold, choose a shallower * non-polling state to improve latency */ - if (cpu_data->utilized) + if (teo_cpu_is_utilized(dev->cpu, cpu_data)) idx = teo_find_shallower_state(drv, dev, idx, duration_ns, true); end: From patchwork Wed Aug 2 13:24:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 710569 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39C8CC04A6A for ; Wed, 2 Aug 2023 13:31:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231612AbjHBNbd (ORCPT ); Wed, 2 Aug 2023 09:31:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59570 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233127AbjHBNbc (ORCPT ); Wed, 2 Aug 2023 09:31:32 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B2B02690; Wed, 2 Aug 2023 06:31:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=6U0TKuKXrlOoIUDQ5shVBrCtBNLJq3+WFZcswjHGIas=; b=jF3QA3R+FU2BVbcTtrdX7BmbsT kQjgCqDuI7Ao572IhkkUlGo37pjSWJJhXhZQYhD0b0U0HjS+O8JirD0sPl9cyOHe8DJwiYFELAHQM 8Nv67xq9fV9gEhVxs5RutYYfcqujldqUK0jJn2KPHlO8cx44u/OOSP+PIMNL6p/5IszceRN5PSqXb gzZlKs7dpWYG7faIBj/w7W7Y1CeF+mkRAYaUNxpo6dOVKi+cyu6ea/HslbuD8Oq9R2IiylaC0i54D cyLvPWpCEk7VIL9rqH3tXCJRvgV8x5hvHBxtwHPGTF7edGAfxH45cEbj8vB25BeHNPe06W1DWU5zs B5iW5y3w==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qRBwZ-00FxfA-1v; Wed, 02 Aug 2023 13:31:16 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 1C0AB3008C6; Wed, 2 Aug 2023 15:31:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 0527D2107D7E5; Wed, 2 Aug 2023 15:31:15 +0200 (CEST) Message-ID: <20230802132925.755164086@infradead.org> User-Agent: quilt/0.66 Date: Wed, 02 Aug 2023 15:24:35 +0200 From: Peter Zijlstra To: anna-maria@linutronix.de, rafael@kernel.org, tglx@linutronix.de, frederic@kernel.org, gautham.shenoy@amd.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, daniel.lezcano@linaro.org, linux-pm@vger.kernel.org, mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, kajetan.puchalski@arm.com Subject: [RFC][PATCH v2 4/5] cpuidle/teo: Avoid tick_nohz_next_event() References: <20230802132431.254614419@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Use the new tick_get_sleep_length() call in conjunction with the new TICK state to elide tick_nohz_next_event() when possible. Specifically, start the state selection using the existing next timer (tick or earlier). And only when state selection lands on the TICK state ask for the NOHZ next timer. Signed-off-by: Peter Zijlstra (Intel) --- drivers/cpuidle/governors/teo.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) --- a/drivers/cpuidle/governors/teo.c +++ b/drivers/cpuidle/governors/teo.c @@ -139,6 +139,7 @@ #include #include #include +#include "../cpuidle.h" /* * The number of bits to shift the CPU's capacity by in order to determine @@ -363,8 +364,7 @@ static int teo_select(struct cpuidle_dri int constraint_idx = 0; int idx0 = 0, idx = -1; bool alt_intercepts, alt_recent; - ktime_t delta_tick; - s64 duration_ns; + s64 duration_ns, tick_ns; int i; if (dev->last_state_idx >= 0) { @@ -374,8 +374,7 @@ static int teo_select(struct cpuidle_dri cpu_data->time_span_ns = local_clock(); - duration_ns = tick_nohz_get_sleep_length(&delta_tick); - cpu_data->sleep_length_ns = duration_ns; + duration_ns = tick_ns = tick_get_sleep_length(false); /* * Find the deepest idle state whose target residency does not exceed @@ -407,6 +406,14 @@ static int teo_select(struct cpuidle_dri if (s->target_residency_ns > duration_ns) break; + if (s->target_residency_ns == SHORT_TICK_NSEC) { + /* + * We hit the tick state, see if it makes sense to + * disable the tick and go deeper still. + */ + duration_ns = tick_get_sleep_length(true); + } + idx = i; if (s->exit_latency_ns <= latency_req) @@ -417,6 +424,8 @@ static int teo_select(struct cpuidle_dri idx_recent_sum = recent_sum; } + cpu_data->sleep_length_ns = duration_ns; + /* Avoid unnecessary overhead. */ if (idx < 0) { idx = 0; /* No states enabled, must use 0. */ @@ -531,8 +540,8 @@ static int teo_select(struct cpuidle_dri * that. */ if (idx > idx0 && - drv->states[idx].target_residency_ns > delta_tick) - idx = teo_find_shallower_state(drv, dev, idx, delta_tick, false); + drv->states[idx].target_residency_ns > tick_ns) + idx = teo_find_shallower_state(drv, dev, idx, tick_ns, false); } return idx; From patchwork Wed Aug 2 13:24:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 710567 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51CF5C04FE0 for ; Wed, 2 Aug 2023 13:31:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234164AbjHBNbj (ORCPT ); Wed, 2 Aug 2023 09:31:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234161AbjHBNbf (ORCPT ); Wed, 2 Aug 2023 09:31:35 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D86302698; Wed, 2 Aug 2023 06:31:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=FEv2iT5iG0jeFDcJt9mW5bYGJa6fvQIcftXsRiIKsDM=; b=OJeHWtCn/qCA8UDeTUprU4JcoE 77B5rES8YJQG+Q34u94zXR/7WTH2Or1bRj2FBQDgfqyoVbbhzgSlM58+STeDCDdBKRDhygd1kseue CZmWUYiCbUxqCXDdVvTAD+m4In1W14yavTbJ/LOI8YuWgbV6XlFA0xDIfjfKpZ3Dxdtv19mss6loJ rRfiEacR0MiXzF5PPxbZcZ6dd8IHuh8Wz8s6n8CA2TXE3+BorgkSgKhIEEyOEO3leo4VBS6/hKCJ+ 9PGvGM7SgNwVSXAGA3lcMmmvP64s3JDMdBMzXPLpXRggMhrWL8KyTOyuHAUGThBTh0GonGYJgAuwO DjZ5gJrw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1qRBwa-00FBnY-5G; Wed, 02 Aug 2023 13:31:16 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 56827300DCF; Wed, 2 Aug 2023 15:31:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 0788F2107D7EA; Wed, 2 Aug 2023 15:31:15 +0200 (CEST) Message-ID: <20230802132925.824393002@infradead.org> User-Agent: quilt/0.66 Date: Wed, 02 Aug 2023 15:24:36 +0200 From: Peter Zijlstra To: anna-maria@linutronix.de, rafael@kernel.org, tglx@linutronix.de, frederic@kernel.org, gautham.shenoy@amd.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, daniel.lezcano@linaro.org, linux-pm@vger.kernel.org, mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, kajetan.puchalski@arm.com Subject: [RFC][PATCH v2 5/5] cpuidle,teo: Improve state selection References: <20230802132431.254614419@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org When selecting a state, stop when history tells us 66% of recent idles were at or below our current state. Signed-off-by: Peter Zijlstra (Intel) --- drivers/cpuidle/governors/teo.c | 6 ++++++ 1 file changed, 6 insertions(+) --- a/drivers/cpuidle/governors/teo.c +++ b/drivers/cpuidle/governors/teo.c @@ -361,6 +361,7 @@ static int teo_select(struct cpuidle_dri unsigned int recent_sum = 0; unsigned int idx_hit_sum = 0; unsigned int hit_sum = 0; + unsigned int thresh_sum = 0; int constraint_idx = 0; int idx0 = 0, idx = -1; bool alt_intercepts, alt_recent; @@ -376,6 +377,8 @@ static int teo_select(struct cpuidle_dri duration_ns = tick_ns = tick_get_sleep_length(false); + thresh_sum = 2 * cpu_data->total / 3; /* 66% */ + /* * Find the deepest idle state whose target residency does not exceed * the current sleep length and the deepest idle state not deeper than @@ -406,6 +409,9 @@ static int teo_select(struct cpuidle_dri if (s->target_residency_ns > duration_ns) break; + if (intercept_sum + hit_sum > thresh_sum) + break; + if (s->target_residency_ns == SHORT_TICK_NSEC) { /* * We hit the tick state, see if it makes sense to