From patchwork Thu Jul 13 06:44:37 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 107550 Delivered-To: patch@linaro.org Received: by 10.140.101.44 with SMTP id t41csp1830618qge; Wed, 12 Jul 2017 23:45:12 -0700 (PDT) X-Received: by 10.99.7.72 with SMTP id 69mr7599769pgh.136.1499928312180; Wed, 12 Jul 2017 23:45:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1499928312; cv=none; d=google.com; s=arc-20160816; b=pCRKfml47Xsnm+zQDZI9sf4dulOHzPV4xuK6AevqArtXrtC5HOOZfdLwxA+IGB9FOY +CnRxmjBu9XmJ4FVc4WpSk2Ub5nHajyx+w+u0dDoJJxnrBp2mfpFmRcy7roTqdSK+O8L KLZcuMNouD8XqYJnI+UxI0AZ4z20+H7kZOqByKgWANa7mpHl3aMnbYRnAHbMps4/1w2B 1vJAcM+3uWdLYW1rOLgf8/qpWzKa5Qy2VhDpdGkYpWqZlFUYe5zr0MNWooNm5KI3pmcU qEg3mfExBEXghv4NMTF/srXC6d39CbDcs4symUfbb2/VZpNIp7VlohlQNR/0Ivo/KOUi qNrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature :arc-authentication-results; bh=EDXeQCw+DWGELAZPxq0cYtE6Xd0/q913WUXfhGpIIwE=; b=PgW+GHfiiTyRU0cI8B6Zjc90IwnvOwdHt0mJwNStctumcHJFyULdaGBjbqx131/p0e fqOIrNotRL2BK/XAJgkLhYHC63iJcxlzGnMG6YsuYNvLMukYMVmmzICYEAh/CCyhaNTU zssTaChehdLXrQlnUdN/FB52k5/uc5OC0IfieShXwrg1hoxnCEc8X8StUBNq8SgAzDa5 UMPBJhPGemSmmtdo8IXfRe3w6X0iPWeN6PDTiXZtzOyD+WXwkXmLkSh1XIz0Ei43N4cE OdCyiTXSSfKKbfS9JQUHOmcNjfdU91brHyqAOb7oJFr+9pjFt2VchSKHED6EF3XebN2C k9hg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.b=iZGDjH0v; spf=pass (google.com: best guess record for domain of linux-pm-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e127si3566133pgc.44.2017.07.12.23.45.11; Wed, 12 Jul 2017 23:45:12 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-pm-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.b=iZGDjH0v; spf=pass (google.com: best guess record for domain of linux-pm-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751199AbdGMGpK (ORCPT + 14 others); Thu, 13 Jul 2017 02:45:10 -0400 Received: from mail-pg0-f45.google.com ([74.125.83.45]:34161 "EHLO mail-pg0-f45.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751153AbdGMGpI (ORCPT ); Thu, 13 Jul 2017 02:45:08 -0400 Received: by mail-pg0-f45.google.com with SMTP id t186so24937019pgb.1 for ; Wed, 12 Jul 2017 23:45:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :in-reply-to:references; bh=EDXeQCw+DWGELAZPxq0cYtE6Xd0/q913WUXfhGpIIwE=; b=iZGDjH0vXckbXnzaPlE7Fbbgkujp6nEr/YBvBXT1NG9eW40BghxDk1DPAZd5DOsZtr DyAy4gY8I+D9vYqqNeiJxyJ2+wocHAgJRr9+LX4wuJPDHIkJk5A6kDxtYG3inns/IeHs GvXgjXY2bz19w5c0LKs75rtXJvrV0O9ohyzK4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:in-reply-to:references; bh=EDXeQCw+DWGELAZPxq0cYtE6Xd0/q913WUXfhGpIIwE=; b=WKjAWobP6+fa1a4B/KU34BpMOPXZjVnWowComXGA8kCb9zICvjTsupn0vM3KgunMnv Epdlgzsc7mO8t4ygxpbX9FxhditwE3/C6Pnq0xJ1cnrWxvR2nuHaaKZ9k+a3WQPmF1Zr uTfPgGd4FZdHbpRTYbFQKGt2gDhFeF5czq4blG2wIJn2Pbyp8gof2UJcNtoaHmB2L7q1 fjhNrXYuMDdOoAWFlobtWk0sgcRW/IRZj76wm0qCa60seAsMoWEuSgmD46AKhBkRGzH2 CtSSyHVvNK5F0hwOcqelFtzSebQ7CgKLNiB2CjDnnTaezQCQa1BICsI5J1idIuvf7Dek 2/RQ== X-Gm-Message-State: AIVw110UhYf0TDSsBpJgHqrNMJtFP7ruLHkmZn9cyA8XSrlW1t0MuZon d8Sugi86ufPJkjg2 X-Received: by 10.98.78.88 with SMTP id c85mr60501654pfb.17.1499928307209; Wed, 12 Jul 2017 23:45:07 -0700 (PDT) Received: from localhost ([122.171.81.230]) by smtp.gmail.com with ESMTPSA id 63sm8477168pgd.57.2017.07.12.23.45.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Jul 2017 23:45:06 -0700 (PDT) From: Viresh Kumar To: Rafael Wysocki , Viresh Kumar , Srinivas Pandruvada , Len Brown , Ingo Molnar , Peter Zijlstra Cc: linux-pm@vger.kernel.org, Vincent Guittot , smuckle.linux@gmail.com, juri.lelli@arm.com, Morten.Rasmussen@arm.com, patrick.bellasi@arm.com, eas-dev@lists.linaro.org, linux-kernel@vger.kernel.org Subject: [PATCH V3 1/3] sched: cpufreq: Allow remote cpufreq callbacks Date: Thu, 13 Jul 2017 12:14:37 +0530 Message-Id: <0f950529a63fb95e87944644c4854be4fcfaea38.1499927699.git.viresh.kumar@linaro.org> X-Mailer: git-send-email 2.13.0.71.gd7076ec9c9cb In-Reply-To: References: In-Reply-To: References: Sender: linux-pm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org We do not call cpufreq callbacks from scheduler core for remote (non-local) CPUs currently. But there are cases where such remote callbacks are useful, specially in the case of shared cpufreq policies. This patch updates the scheduler core to call the cpufreq callbacks for remote CPUs as well. For now, all the registered utilization update callbacks are updated to return early if remote callback is detected. That is, this patch just moves the decision making down in the hierarchy. Later patches would enable remote callbacks for shared policies. Based on initial work from Steve Muckle. Signed-off-by: Steve Muckle Signed-off-by: Viresh Kumar --- drivers/cpufreq/cpufreq_governor.c | 4 ++++ drivers/cpufreq/intel_pstate.c | 8 ++++++++ include/linux/sched/cpufreq.h | 1 + kernel/sched/cpufreq.c | 1 + kernel/sched/cpufreq_schedutil.c | 8 ++++++++ kernel/sched/deadline.c | 2 +- kernel/sched/fair.c | 8 +++++--- kernel/sched/rt.c | 2 +- kernel/sched/sched.h | 10 ++-------- 9 files changed, 31 insertions(+), 13 deletions(-) -- 2.13.0.71.gd7076ec9c9cb diff --git a/drivers/cpufreq/cpufreq_governor.c b/drivers/cpufreq/cpufreq_governor.c index 47e24b5384b3..606b1a37a1af 100644 --- a/drivers/cpufreq/cpufreq_governor.c +++ b/drivers/cpufreq/cpufreq_governor.c @@ -275,6 +275,10 @@ static void dbs_update_util_handler(struct update_util_data *data, u64 time, struct policy_dbs_info *policy_dbs = cdbs->policy_dbs; u64 delta_ns, lst; + /* Don't allow remote callbacks */ + if (smp_processor_id() != data->cpu) + return; + /* * The work may not be allowed to be queued up right now. * Possible reasons: diff --git a/drivers/cpufreq/intel_pstate.c b/drivers/cpufreq/intel_pstate.c index b7fb8b7c980d..4bee2f4cbc28 100644 --- a/drivers/cpufreq/intel_pstate.c +++ b/drivers/cpufreq/intel_pstate.c @@ -1732,6 +1732,10 @@ static void intel_pstate_update_util_pid(struct update_util_data *data, struct cpudata *cpu = container_of(data, struct cpudata, update_util); u64 delta_ns = time - cpu->sample.time; + /* Don't allow remote callbacks */ + if (smp_processor_id() != data->cpu) + return; + if ((s64)delta_ns < pid_params.sample_rate_ns) return; @@ -1749,6 +1753,10 @@ static void intel_pstate_update_util(struct update_util_data *data, u64 time, struct cpudata *cpu = container_of(data, struct cpudata, update_util); u64 delta_ns; + /* Don't allow remote callbacks */ + if (smp_processor_id() != data->cpu) + return; + if (flags & SCHED_CPUFREQ_IOWAIT) { cpu->iowait_boost = int_tofp(1); } else if (cpu->iowait_boost) { diff --git a/include/linux/sched/cpufreq.h b/include/linux/sched/cpufreq.h index d2be2ccbb372..8256a8f35f22 100644 --- a/include/linux/sched/cpufreq.h +++ b/include/linux/sched/cpufreq.h @@ -16,6 +16,7 @@ #ifdef CONFIG_CPU_FREQ struct update_util_data { void (*func)(struct update_util_data *data, u64 time, unsigned int flags); + unsigned int cpu; }; void cpufreq_add_update_util_hook(int cpu, struct update_util_data *data, diff --git a/kernel/sched/cpufreq.c b/kernel/sched/cpufreq.c index dbc51442ecbc..ee4c596b71b4 100644 --- a/kernel/sched/cpufreq.c +++ b/kernel/sched/cpufreq.c @@ -42,6 +42,7 @@ void cpufreq_add_update_util_hook(int cpu, struct update_util_data *data, return; data->func = func; + data->cpu = cpu; rcu_assign_pointer(per_cpu(cpufreq_update_util_data, cpu), data); } EXPORT_SYMBOL_GPL(cpufreq_add_update_util_hook); diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c index 29a397067ffa..ed9c589e5386 100644 --- a/kernel/sched/cpufreq_schedutil.c +++ b/kernel/sched/cpufreq_schedutil.c @@ -218,6 +218,10 @@ static void sugov_update_single(struct update_util_data *hook, u64 time, unsigned int next_f; bool busy; + /* Remote callbacks aren't allowed for policies which aren't shared */ + if (smp_processor_id() != hook->cpu) + return; + sugov_set_iowait_boost(sg_cpu, time, flags); sg_cpu->last_update = time; @@ -290,6 +294,10 @@ static void sugov_update_shared(struct update_util_data *hook, u64 time, unsigned long util, max; unsigned int next_f; + /* Don't allow remote callbacks */ + if (smp_processor_id() != hook->cpu) + return; + sugov_get_util(&util, &max); raw_spin_lock(&sg_policy->update_lock); diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index a84299f44b5d..7fcfaee39d19 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -1136,7 +1136,7 @@ static void update_curr_dl(struct rq *rq) } /* kick cpufreq (see the comment in kernel/sched/sched.h). */ - cpufreq_update_this_cpu(rq, SCHED_CPUFREQ_DL); + cpufreq_update_util(rq, SCHED_CPUFREQ_DL); schedstat_set(curr->se.statistics.exec_max, max(curr->se.statistics.exec_max, delta_exec)); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index c95880e216f6..d378d02fdfcb 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -3278,7 +3278,9 @@ static inline void set_tg_cfs_propagate(struct cfs_rq *cfs_rq) {} static inline void cfs_rq_util_change(struct cfs_rq *cfs_rq) { - if (&this_rq()->cfs == cfs_rq) { + struct rq *rq = rq_of(cfs_rq); + + if (&rq->cfs == cfs_rq) { /* * There are a few boundary cases this might miss but it should * get called often enough that that should (hopefully) not be @@ -3295,7 +3297,7 @@ static inline void cfs_rq_util_change(struct cfs_rq *cfs_rq) * * See cpu_util(). */ - cpufreq_update_util(rq_of(cfs_rq), 0); + cpufreq_update_util(rq, 0); } } @@ -4875,7 +4877,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) * passed. */ if (p->in_iowait) - cpufreq_update_this_cpu(rq, SCHED_CPUFREQ_IOWAIT); + cpufreq_update_util(rq, SCHED_CPUFREQ_IOWAIT); for_each_sched_entity(se) { if (se->on_rq) diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index 45caf937ef90..0af5ca9e3e3f 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -970,7 +970,7 @@ static void update_curr_rt(struct rq *rq) return; /* Kick cpufreq (see the comment in kernel/sched/sched.h). */ - cpufreq_update_this_cpu(rq, SCHED_CPUFREQ_RT); + cpufreq_update_util(rq, SCHED_CPUFREQ_RT); schedstat_set(curr->se.statistics.exec_max, max(curr->se.statistics.exec_max, delta_exec)); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index eeef1a3086d1..aa9d5b87b4f8 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2070,19 +2070,13 @@ static inline void cpufreq_update_util(struct rq *rq, unsigned int flags) { struct update_util_data *data; - data = rcu_dereference_sched(*this_cpu_ptr(&cpufreq_update_util_data)); + data = rcu_dereference_sched(*per_cpu_ptr(&cpufreq_update_util_data, + cpu_of(rq))); if (data) data->func(data, rq_clock(rq), flags); } - -static inline void cpufreq_update_this_cpu(struct rq *rq, unsigned int flags) -{ - if (cpu_of(rq) == smp_processor_id()) - cpufreq_update_util(rq, flags); -} #else static inline void cpufreq_update_util(struct rq *rq, unsigned int flags) {} -static inline void cpufreq_update_this_cpu(struct rq *rq, unsigned int flags) {} #endif /* CONFIG_CPU_FREQ */ #ifdef arch_scale_freq_capacity