From patchwork Fri Sep 1 13:21:09 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 111487 Delivered-To: patch@linaro.org Received: by 10.37.128.210 with SMTP id c18csp941531ybm; Fri, 1 Sep 2017 06:35:48 -0700 (PDT) X-Google-Smtp-Source: ADKCNb6e45LJrTOw8arfFx7qCI0IMpNPv+xVqn7op3EMHrd2AB3FsxlxT0Q+jTYR5rMegtbe9tAB X-Received: by 10.84.225.6 with SMTP id t6mr2453935plj.431.1504272948668; Fri, 01 Sep 2017 06:35:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504272948; cv=none; d=google.com; s=arc-20160816; b=pmUirpSzYZvqQO2sTjBzZdmZK6Jmp7hVHqsa2WonqaF8kFCanFMwteCF4mWigLnIpu Pa25L8aeu5jsCy+HrWdCixUn40M1r/VKTSe3nhKCUhoHd5fD9ho13byp/DtFlYinXiEy BbeE7Sm7rwcJ7O2lgvv8jug627/Kn3x1/C+o0oPe1X+0Ucrv+VJMR/uL/uAbefYIFGxw iavFzWnGRxTaTgzri/PRNLHmQl9whlFC8kMrgfKoRsyo69o2jYwljxcBxPKxnwpfPqtu fs9bHTSzaNMZB1xjzIk0VWUVJODCqhNc+GMIjnFnT8coBLhclUkWd2ZKFYUhev0cbTce XW/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-disposition:mime-version :references:subject:cc:to:from:date:user-agent:message-id :dkim-signature:arc-authentication-results; bh=reXW2muz6MTIqIPE1WnkUcdVILnWZoWDQqWqnWF71MU=; b=j9uKUnJDoJwCkV6Cc5INTGl044odeYKgJmeom41YA+Z4hLXRtZkpWSZKkxfyc90mUD RnCz+wWkYfFg2Bzw0atgNXewnfZp0mkUodozBYr08m2wS1gkFauCTqugiIsZxy5g3gnp YNRklLLxR1fIMYuaFMg/j0Y2cir4yL0IoA2wWhRy1qwUwnew3CQIIYbS8H2Z+sOUs86V FAigzuvSaHAW/8/jJx/TRglMhFwIc3T8oJ6PeQAsaSNVYjQKFvpSj+TXIHY3Fy1KFaak rWkvyDowFt6Ow8kKiCHYyqnFuCXJoeqcdQ30zN/SwwsdcUHLFVDA1KjUpiAm9ST6o29f HG4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@infradead.org header.s=merlin.20170209 header.b=t14icx2u; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w82si120016pfj.463.2017.09.01.06.35.48; Fri, 01 Sep 2017 06:35:48 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=fail header.i=@infradead.org header.s=merlin.20170209 header.b=t14icx2u; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752569AbdIANfa (ORCPT + 26 others); Fri, 1 Sep 2017 09:35:30 -0400 Received: from merlin.infradead.org ([205.233.59.134]:54946 "EHLO merlin.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752359AbdIANfX (ORCPT ); Fri, 1 Sep 2017 09:35:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=reXW2muz6MTIqIPE1WnkUcdVILnWZoWDQqWqnWF71MU=; b=t14icx2usIjjJyq/yWDkusOZlW ZJf5vj/mQszqe4G9cdNSYUGgMvSOYpndhzcV+fUvvMpXn52Ol8gOXhmCeT/g+/Iglx1FG2gjw7Z9A LRpOGOBG/b/z5gxxsUlseSNaVwn3x9U2o5rouOaf8+J51scYQoewbWGS3gxobMbrTG12OxJm4OC6V EyJe5opqCUohnQoXyI83dBVxEwNiRFM6CT5PhgNmlkLUEnPOUXEGx1qE7aL4LqgxbCLLKj6fiO9qo LUytoJ24oBklh+sls40kNhAGmLPg/XUrLfkByCkc3jxCV2TJEzVwq84oFKCdg9WV4tlo6ibdVjrJp npLiYuMQ==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=hirez.programming.kicks-ass.net) by merlin.infradead.org with esmtpsa (Exim 4.87 #1 (Red Hat Linux)) id 1dnm6G-0005EY-Km; Fri, 01 Sep 2017 13:35:09 +0000 Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 3014320292D0F; Fri, 1 Sep 2017 15:35:04 +0200 (CEST) Message-Id: <20170901132748.483072146@infradead.org> User-Agent: quilt/0.63-1 Date: Fri, 01 Sep 2017 15:21:09 +0200 From: Peter Zijlstra To: mingo@kernel.org, linux-kernel@vger.kernel.org, tj@kernel.org, josef@toxicpanda.com Cc: torvalds@linux-foundation.org, vincent.guittot@linaro.org, efault@gmx.de, pjt@google.com, clm@fb.com, dietmar.eggemann@arm.com, morten.rasmussen@arm.com, bsegall@google.com, yuyang.du@intel.com, peterz@infradead.org Subject: [PATCH -v2 10/18] sched/fair: Use reweight_entity() for set_user_nice() References: <20170901132059.342024223@infradead.org> MIME-Version: 1.0 Content-Disposition: inline; filename=vincent_guittot-sched_fair-remove_se-_load_weight_from_se-_avg_load_sum.patch Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Vincent Guittot Now that we directly change load_avg and propagate that change into the sums, sys_nice() and co should do the same, otherwise its possible to confuse load accounting when we migrate near the weight change. [peterz: Changelog, call condition] [josef: fixed runnable and !SMP compilation] Signed-off-by: Peter Zijlstra (Intel) Link: http://lkml.kernel.org/r/20170517095045.GA8420@linaro.org --- kernel/sched/core.c | 22 ++++++++++++----- kernel/sched/fair.c | 63 +++++++++++++++++++++++++++++---------------------- kernel/sched/sched.h | 2 + 3 files changed, 54 insertions(+), 33 deletions(-) --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -733,7 +733,7 @@ int tg_nop(struct task_group *tg, void * } #endif -static void set_load_weight(struct task_struct *p) +static void set_load_weight(struct task_struct *p, bool update_load) { int prio = p->static_prio - MAX_RT_PRIO; struct load_weight *load = &p->se.load; @@ -747,8 +747,16 @@ static void set_load_weight(struct task_ return; } - load->weight = scale_load(sched_prio_to_weight[prio]); - load->inv_weight = sched_prio_to_wmult[prio]; + /* + * SCHED_OTHER tasks have to update their load when changing their + * weight + */ + if (update_load && p->sched_class == &fair_sched_class) { + reweight_task(p, prio); + } else { + load->weight = scale_load(sched_prio_to_weight[prio]); + load->inv_weight = sched_prio_to_wmult[prio]; + } } static inline void enqueue_task(struct rq *rq, struct task_struct *p, int flags) @@ -2356,7 +2364,7 @@ int sched_fork(unsigned long clone_flags p->static_prio = NICE_TO_PRIO(0); p->prio = p->normal_prio = __normal_prio(p); - set_load_weight(p); + set_load_weight(p, false); /* * We don't need the reset flag anymore after the fork. It has @@ -3803,7 +3811,7 @@ void set_user_nice(struct task_struct *p put_prev_task(rq, p); p->static_prio = NICE_TO_PRIO(nice); - set_load_weight(p); + set_load_weight(p, true); old_prio = p->prio; p->prio = effective_prio(p); delta = p->prio - old_prio; @@ -3960,7 +3968,7 @@ static void __setscheduler_params(struct */ p->rt_priority = attr->sched_priority; p->normal_prio = normal_prio(p); - set_load_weight(p); + set_load_weight(p, true); } /* Actually do priority change: must hold pi & rq lock. */ @@ -5910,7 +5918,7 @@ void __init sched_init(void) atomic_set(&rq->nr_iowait, 0); } - set_load_weight(&init_task); + set_load_weight(&init_task, false); /* * The boot idle thread does lazy MMU switching as well: --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -2789,6 +2789,43 @@ static inline void dequeue_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) { } #endif +static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, + unsigned long weight) +{ + if (se->on_rq) { + /* commit outstanding execution time */ + if (cfs_rq->curr == se) + update_curr(cfs_rq); + account_entity_dequeue(cfs_rq, se); + dequeue_runnable_load_avg(cfs_rq, se); + } + dequeue_load_avg(cfs_rq, se); + + update_load_set(&se->load, weight); + +#ifdef CONFIG_SMP + se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, + LOAD_AVG_MAX - 1024 + se->avg.period_contrib); +#endif + + enqueue_load_avg(cfs_rq, se); + if (se->on_rq) { + account_entity_enqueue(cfs_rq, se); + enqueue_runnable_load_avg(cfs_rq, se); + } +} + +void reweight_task(struct task_struct *p, int prio) +{ + struct sched_entity *se = &p->se; + struct cfs_rq *cfs_rq = cfs_rq_of(se); + struct load_weight *load = &se->load; + unsigned long weight = scale_load(sched_prio_to_weight[prio]); + + reweight_entity(cfs_rq, se, weight); + load->inv_weight = sched_prio_to_wmult[prio]; +} + #ifdef CONFIG_FAIR_GROUP_SCHED # ifdef CONFIG_SMP /* @@ -2892,32 +2929,6 @@ static long calc_cfs_shares(struct cfs_r } # endif /* CONFIG_SMP */ -static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, - unsigned long weight) -{ - if (se->on_rq) { - /* commit outstanding execution time */ - if (cfs_rq->curr == se) - update_curr(cfs_rq); - account_entity_dequeue(cfs_rq, se); - dequeue_runnable_load_avg(cfs_rq, se); - } - dequeue_load_avg(cfs_rq, se); - - update_load_set(&se->load, weight); - -#ifdef CONFIG_SMP - se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, - LOAD_AVG_MAX - 1024 + se->avg.period_contrib); -#endif - - enqueue_load_avg(cfs_rq, se); - if (se->on_rq) { - account_entity_enqueue(cfs_rq, se); - enqueue_runnable_load_avg(cfs_rq, se); - } -} - static inline int throttled_hierarchy(struct cfs_rq *cfs_rq); static void update_cfs_shares(struct sched_entity *se) --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1532,6 +1532,8 @@ extern void init_sched_dl_class(void); extern void init_sched_rt_class(void); extern void init_sched_fair_class(void); +extern void reweight_task(struct task_struct *p, int prio); + extern void resched_curr(struct rq *rq); extern void resched_cpu(int cpu);