From patchwork Wed Jan 22 17:35:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Douglas RAILLARD X-Patchwork-Id: 212983 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98893C2D0DB for ; Wed, 22 Jan 2020 17:36:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 70B3D21835 for ; Wed, 22 Jan 2020 17:36:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728984AbgAVRgO (ORCPT ); Wed, 22 Jan 2020 12:36:14 -0500 Received: from foss.arm.com ([217.140.110.172]:59002 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725802AbgAVRgO (ORCPT ); Wed, 22 Jan 2020 12:36:14 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DE8B8106F; Wed, 22 Jan 2020 09:36:13 -0800 (PST) Received: from e107049-lin.arm.com (e107049-lin.cambridge.arm.com [10.1.195.43]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8FAE43F6C4; Wed, 22 Jan 2020 09:36:12 -0800 (PST) From: Douglas RAILLARD To: linux-kernel@vger.kernel.org, rjw@rjwysocki.net, viresh.kumar@linaro.org, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org Cc: douglas.raillard@arm.com, dietmar.eggemann@arm.com, qperret@google.com, linux-pm@vger.kernel.org Subject: [RFC PATCH v4 2/6] sched/cpufreq: Attach perf domain to sugov policy Date: Wed, 22 Jan 2020 17:35:34 +0000 Message-Id: <20200122173538.1142069-3-douglas.raillard@arm.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200122173538.1142069-1-douglas.raillard@arm.com> References: <20200122173538.1142069-1-douglas.raillard@arm.com> MIME-Version: 1.0 Sender: linux-pm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Attach an Energy Model perf_domain to each sugov_policy to prepare the ground for energy-aware schedutil. Signed-off-by: Douglas RAILLARD --- kernel/sched/cpufreq_schedutil.c | 41 ++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c index 9b8916fd00a2..09e284dc918a 100644 --- a/kernel/sched/cpufreq_schedutil.c +++ b/kernel/sched/cpufreq_schedutil.c @@ -42,6 +42,10 @@ struct sugov_policy { bool limits_changed; bool need_freq_update; + +#ifdef CONFIG_ENERGY_MODEL + struct em_perf_domain *pd; +#endif }; struct sugov_cpu { @@ -66,6 +70,40 @@ static DEFINE_PER_CPU(struct sugov_cpu, sugov_cpu); /************************ Governor internals ***********************/ +#ifdef CONFIG_ENERGY_MODEL +static void sugov_policy_attach_pd(struct sugov_policy *sg_policy) +{ + struct cpufreq_policy *policy = sg_policy->policy; + struct em_perf_domain *pd; + + sg_policy->pd = NULL; + pd = em_cpu_get(policy->cpu); + if (!pd) + return; + + if (cpumask_equal(policy->related_cpus, to_cpumask(pd->cpus))) { + sg_policy->pd = pd; + } else { + pr_warn("%s: Not all CPUs in schedutil policy %u share the same perf domain, no perf domain for that policy will be registered\n", + __func__, policy->cpu); + } +} + +static struct em_perf_domain * +sugov_policy_get_pd(struct sugov_policy *sg_policy) +{ + return sg_policy->pd; +} +#else /* CONFIG_ENERGY_MODEL */ +static void sugov_policy_attach_pd(struct sugov_policy *sg_policy) {} + +static struct em_perf_domain * +sugov_policy_get_pd(struct sugov_policy *sg_policy) +{ + return NULL; +} +#endif /* CONFIG_ENERGY_MODEL */ + static bool sugov_should_update_freq(struct sugov_policy *sg_policy, u64 time) { s64 delta_ns; @@ -859,6 +897,9 @@ static int sugov_start(struct cpufreq_policy *policy) sugov_update_shared : sugov_update_single); } + + sugov_policy_attach_pd(sg_policy); + return 0; } From patchwork Wed Jan 22 17:35:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Douglas RAILLARD X-Patchwork-Id: 212981 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A02DFC33CAF for ; Wed, 22 Jan 2020 17:36:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 84CE62465B for ; Wed, 22 Jan 2020 17:36:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725972AbgAVRgS (ORCPT ); Wed, 22 Jan 2020 12:36:18 -0500 Received: from foss.arm.com ([217.140.110.172]:59034 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726004AbgAVRgR (ORCPT ); Wed, 22 Jan 2020 12:36:17 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id ED08A11D4; Wed, 22 Jan 2020 09:36:16 -0800 (PST) Received: from e107049-lin.arm.com (e107049-lin.cambridge.arm.com [10.1.195.43]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9E3BC3F6C4; Wed, 22 Jan 2020 09:36:15 -0800 (PST) From: Douglas RAILLARD To: linux-kernel@vger.kernel.org, rjw@rjwysocki.net, viresh.kumar@linaro.org, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org Cc: douglas.raillard@arm.com, dietmar.eggemann@arm.com, qperret@google.com, linux-pm@vger.kernel.org Subject: [RFC PATCH v4 4/6] sched/cpufreq: Introduce sugov_cpu_ramp_boost Date: Wed, 22 Jan 2020 17:35:36 +0000 Message-Id: <20200122173538.1142069-5-douglas.raillard@arm.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200122173538.1142069-1-douglas.raillard@arm.com> References: <20200122173538.1142069-1-douglas.raillard@arm.com> MIME-Version: 1.0 Sender: linux-pm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Use the utilization signals dynamic to detect when the utilization of a set of tasks starts increasing because of a change in tasks' behavior. This allows detecting when spending extra power for faster frequency ramp up response would be beneficial to the reactivity of the system. This ramp boost is computed as the difference between util_avg and util_est_enqueued. This number somehow represents a lower bound of how much extra utilization this tasks is actually using, compared to our best current stable knowledge of it (which is util_est_enqueued). When the set of runnable tasks changes, the boost is disabled as the impact of blocked utilization on util_avg will make the delta with util_est_enqueued not very informative. Signed-off-by: Douglas RAILLARD --- kernel/sched/cpufreq_schedutil.c | 43 ++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c index 608963da4916..25a410a1ff6a 100644 --- a/kernel/sched/cpufreq_schedutil.c +++ b/kernel/sched/cpufreq_schedutil.c @@ -61,6 +61,10 @@ struct sugov_cpu { unsigned long bw_dl; unsigned long max; + unsigned long ramp_boost; + unsigned long util_est_enqueued; + unsigned long util_avg; + /* The field below is for single-CPU policies only: */ #ifdef CONFIG_NO_HZ_COMMON unsigned long saved_idle_calls; @@ -183,6 +187,42 @@ static void sugov_deferred_update(struct sugov_policy *sg_policy, u64 time, } } +static unsigned long sugov_cpu_ramp_boost(struct sugov_cpu *sg_cpu) +{ + return READ_ONCE(sg_cpu->ramp_boost); +} + +static unsigned long sugov_cpu_ramp_boost_update(struct sugov_cpu *sg_cpu) +{ + struct rq *rq = cpu_rq(sg_cpu->cpu); + unsigned long util_est_enqueued; + unsigned long util_avg; + unsigned long boost = 0; + + util_est_enqueued = READ_ONCE(rq->cfs.avg.util_est.enqueued); + util_avg = READ_ONCE(rq->cfs.avg.util_avg); + + /* + * Boost when util_avg becomes higher than the previous stable + * knowledge of the enqueued tasks' set util, which is CPU's + * util_est_enqueued. + * + * We try to spot changes in the workload itself, so we want to + * avoid the noise of tasks being enqueued/dequeued. To do that, + * we only trigger boosting when the "amount of work" enqueued + * is stable. + */ + if (util_est_enqueued == sg_cpu->util_est_enqueued && + util_avg >= sg_cpu->util_avg && + util_avg > util_est_enqueued) + boost = util_avg - util_est_enqueued; + + sg_cpu->util_est_enqueued = util_est_enqueued; + sg_cpu->util_avg = util_avg; + WRITE_ONCE(sg_cpu->ramp_boost, boost); + return boost; +} + /** * get_next_freq - Compute a new frequency for a given cpufreq policy. * @sg_policy: schedutil policy object to compute the new frequency for. @@ -514,6 +554,7 @@ static void sugov_update_single(struct update_util_data *hook, u64 time, busy = !sg_policy->need_freq_update && sugov_cpu_is_busy(sg_cpu); util = sugov_get_util(sg_cpu); + sugov_cpu_ramp_boost_update(sg_cpu); max = sg_cpu->max; util = sugov_iowait_apply(sg_cpu, time, util, max); next_f = get_next_freq(sg_policy, util, max); @@ -554,6 +595,8 @@ static unsigned int sugov_next_freq_shared(struct sugov_cpu *sg_cpu, u64 time) unsigned long j_util, j_max; j_util = sugov_get_util(j_sg_cpu); + if (j_sg_cpu == sg_cpu) + sugov_cpu_ramp_boost_update(sg_cpu); j_max = j_sg_cpu->max; j_util = sugov_iowait_apply(j_sg_cpu, time, j_util, j_max); From patchwork Wed Jan 22 17:35:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Douglas RAILLARD X-Patchwork-Id: 212982 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F36EC2D0DB for ; Wed, 22 Jan 2020 17:36:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4E7DC21835 for ; Wed, 22 Jan 2020 17:36:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726968AbgAVRgY (ORCPT ); Wed, 22 Jan 2020 12:36:24 -0500 Received: from foss.arm.com ([217.140.110.172]:59050 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729239AbgAVRgT (ORCPT ); Wed, 22 Jan 2020 12:36:19 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7ABA011FB; Wed, 22 Jan 2020 09:36:18 -0800 (PST) Received: from e107049-lin.arm.com (e107049-lin.cambridge.arm.com [10.1.195.43]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 2B9493F6C4; Wed, 22 Jan 2020 09:36:17 -0800 (PST) From: Douglas RAILLARD To: linux-kernel@vger.kernel.org, rjw@rjwysocki.net, viresh.kumar@linaro.org, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org Cc: douglas.raillard@arm.com, dietmar.eggemann@arm.com, qperret@google.com, linux-pm@vger.kernel.org Subject: [RFC PATCH v4 5/6] sched/cpufreq: Boost schedutil frequency ramp up Date: Wed, 22 Jan 2020 17:35:37 +0000 Message-Id: <20200122173538.1142069-6-douglas.raillard@arm.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200122173538.1142069-1-douglas.raillard@arm.com> References: <20200122173538.1142069-1-douglas.raillard@arm.com> MIME-Version: 1.0 Sender: linux-pm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org In some situations, it can be interesting to spend temporarily more power if that can give a useful frequency boost. Use the new sugov_cpu_ramp_boost() function to drive an energy-aware boost, on top of the minimal required frequency. As that boost number is not accurate (and cannot be without a crystal ball), we only use it in a way that allows direct control over the power it is going to cost. This allows keeping a platform-independent level of control over the average power, while allowing for frequency bursts when we know a (set of) tasks can make use of it. In shared policies, the maximum of all CPU's boost is used. Since the extra power expenditure is bounded, it cannot skyrocket even on platforms with a large number of cores in the same frequency domain and/or very high ratio between lowest and highest OPP cost. Signed-off-by: Douglas RAILLARD --- kernel/sched/cpufreq_schedutil.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c index 25a410a1ff6a..9a7617ea7bf4 100644 --- a/kernel/sched/cpufreq_schedutil.c +++ b/kernel/sched/cpufreq_schedutil.c @@ -228,6 +228,9 @@ static unsigned long sugov_cpu_ramp_boost_update(struct sugov_cpu *sg_cpu) * @sg_policy: schedutil policy object to compute the new frequency for. * @util: Current CPU utilization. * @max: CPU capacity. + * @boost: Extra power that can be spent on top of the minimum amount of power + * required to meet capacity requirements, as a percentage between 0 and + * EM_COST_MARGIN_SCALE. * * If the utilization is frequency-invariant, choose the new frequency to be * proportional to it, that is @@ -246,7 +249,8 @@ static unsigned long sugov_cpu_ramp_boost_update(struct sugov_cpu *sg_cpu) * cpufreq driver limitations. */ static unsigned int get_next_freq(struct sugov_policy *sg_policy, - unsigned long util, unsigned long max) + unsigned long util, unsigned long max, + unsigned long boost) { struct cpufreq_policy *policy = sg_policy->policy; unsigned int freq = arch_scale_freq_invariant() ? @@ -259,7 +263,7 @@ static unsigned int get_next_freq(struct sugov_policy *sg_policy, * Try to get a higher frequency if one is available, given the extra * power we are ready to spend. */ - freq = em_pd_get_higher_freq(pd, freq, 0); + freq = em_pd_get_higher_freq(pd, freq, boost); if (freq == sg_policy->cached_raw_freq && !sg_policy->need_freq_update) return sg_policy->next_freq; @@ -541,6 +545,7 @@ static void sugov_update_single(struct update_util_data *hook, u64 time, unsigned long util, max; unsigned int next_f; bool busy; + unsigned long ramp_boost = 0; sugov_iowait_boost(sg_cpu, time, flags); sg_cpu->last_update = time; @@ -554,10 +559,10 @@ static void sugov_update_single(struct update_util_data *hook, u64 time, busy = !sg_policy->need_freq_update && sugov_cpu_is_busy(sg_cpu); util = sugov_get_util(sg_cpu); - sugov_cpu_ramp_boost_update(sg_cpu); + ramp_boost = sugov_cpu_ramp_boost_update(sg_cpu); max = sg_cpu->max; util = sugov_iowait_apply(sg_cpu, time, util, max); - next_f = get_next_freq(sg_policy, util, max); + next_f = get_next_freq(sg_policy, util, max, ramp_boost); /* * Do not reduce the frequency if the CPU has not been idle * recently, as the reduction is likely to be premature then. @@ -589,14 +594,19 @@ static unsigned int sugov_next_freq_shared(struct sugov_cpu *sg_cpu, u64 time) struct cpufreq_policy *policy = sg_policy->policy; unsigned long util = 0, max = 1; unsigned int j; + unsigned long ramp_boost = 0; for_each_cpu(j, policy->cpus) { struct sugov_cpu *j_sg_cpu = &per_cpu(sugov_cpu, j); - unsigned long j_util, j_max; + unsigned long j_util, j_max, j_ramp_boost; j_util = sugov_get_util(j_sg_cpu); if (j_sg_cpu == sg_cpu) - sugov_cpu_ramp_boost_update(sg_cpu); + j_ramp_boost = sugov_cpu_ramp_boost_update(sg_cpu); + else + j_ramp_boost = sugov_cpu_ramp_boost(j_sg_cpu); + ramp_boost = max(ramp_boost, j_ramp_boost); + j_max = j_sg_cpu->max; j_util = sugov_iowait_apply(j_sg_cpu, time, j_util, j_max); @@ -606,7 +616,7 @@ static unsigned int sugov_next_freq_shared(struct sugov_cpu *sg_cpu, u64 time) } } - return get_next_freq(sg_policy, util, max); + return get_next_freq(sg_policy, util, max, ramp_boost); } static void