From patchwork Wed Feb 4 18:30:47 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Morten Rasmussen X-Patchwork-Id: 44380 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-lb0-f197.google.com (mail-lb0-f197.google.com [209.85.217.197]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 4DDAD21173 for ; Wed, 4 Feb 2015 18:43:11 +0000 (UTC) Received: by mail-lb0-f197.google.com with SMTP id b6sf2526502lbj.0 for ; Wed, 04 Feb 2015 10:43:10 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:delivered-to:from:to:cc:subject :date:message-id:in-reply-to:references:sender:precedence:list-id :x-original-sender:x-original-authentication-results:mailing-list :list-post:list-help:list-archive:list-unsubscribe; bh=Atsj9SNcPV2zEG6joLKDyjHAkV10WNm9N9UcOjdhy/s=; b=MuDWz0T4Qs9gGoW3lWgVjejNou+rqe8o1bzqPoKOy0yJHywoZooM+ZYviyUL61NKWM j4/L+qAyifK5E4HuAz67FtPCOjTZkcK/tqVXefYc8o6FAeeDrDGw43Mpv+UZwMkWG7dN r40MSbxutkba5X/+HQCHD7HDhGFGgmWdpv3468gtiP/rmsRFz3nKVyqvbZbNQbQxuh/S DkIaon0zcz9221n/JvPFDq3ksk71WElfvLqeCaNFaf00kLVtsz/3Mo5bmMad4BB00ltn S8zs4ZCLJaisj/4C/GjFO+e9owVER9HCwJmfH+3ePpB8QjOezeBIeikaqadOza+CjdHG TUhg== X-Gm-Message-State: ALoCoQnDuTZ5psf3871DKu6InZ4cB2luc3htmFqLgtyWwQDaE7Ekcc37mS/K402B20oxBW1NPcSL X-Received: by 10.180.12.146 with SMTP id y18mr565680wib.6.1423075390279; Wed, 04 Feb 2015 10:43:10 -0800 (PST) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.6.42 with SMTP id x10ls69615lax.96.gmail; Wed, 04 Feb 2015 10:43:10 -0800 (PST) X-Received: by 10.152.23.98 with SMTP id l2mr32302889laf.46.1423075389982; Wed, 04 Feb 2015 10:43:09 -0800 (PST) Received: from mail-lb0-f169.google.com (mail-lb0-f169.google.com. [209.85.217.169]) by mx.google.com with ESMTPS id n3si2109228lah.45.2015.02.04.10.43.09 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Feb 2015 10:43:09 -0800 (PST) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.169 as permitted sender) client-ip=209.85.217.169; Received: by mail-lb0-f169.google.com with SMTP id f15so3131693lbj.0 for ; Wed, 04 Feb 2015 10:43:09 -0800 (PST) X-Received: by 10.152.2.41 with SMTP id 9mr11811161lar.59.1423075389774; Wed, 04 Feb 2015 10:43:09 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.35.133 with SMTP id h5csp670528lbj; Wed, 4 Feb 2015 10:43:08 -0800 (PST) X-Received: by 10.70.140.44 with SMTP id rd12mr7804147pdb.64.1423075387120; Wed, 04 Feb 2015 10:43:07 -0800 (PST) Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id z13si2981222pdi.249.2015.02.04.10.43.06; Wed, 04 Feb 2015 10:43:07 -0800 (PST) Received-SPF: none (google.com: linux-kernel-owner@vger.kernel.org does not designate permitted sender hosts) client-ip=209.132.180.67; Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1162219AbbBDSnB (ORCPT + 29 others); Wed, 4 Feb 2015 13:43:01 -0500 Received: from foss-mx-na.foss.arm.com ([217.140.108.86]:41535 "EHLO foss-mx-na.foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1161585AbbBDSbB (ORCPT ); Wed, 4 Feb 2015 13:31:01 -0500 Received: from foss-smtp-na-1.foss.arm.com (unknown [10.80.61.8]) by foss-mx-na.foss.arm.com (Postfix) with ESMTP id 3BDDB50C; Wed, 4 Feb 2015 12:30:56 -0600 (CST) Received: from collaborate-mta1.arm.com (highbank-bc01-b06.austin.arm.com [10.112.81.134]) by foss-smtp-na-1.foss.arm.com (Postfix) with ESMTP id 0299B5FAC1; Wed, 4 Feb 2015 12:30:49 -0600 (CST) Received: from e105550-lin.cambridge.arm.com (e105550-lin.cambridge.arm.com [10.2.131.193]) by collaborate-mta1.arm.com (Postfix) with ESMTP id 6AA0F13F6EE; Wed, 4 Feb 2015 12:30:47 -0600 (CST) From: Morten Rasmussen To: peterz@infradead.org, mingo@redhat.com Cc: vincent.guittot@linaro.org, dietmar.eggemann@arm.com, yuyang.du@intel.com, preeti@linux.vnet.ibm.com, mturquette@linaro.org, nico@linaro.org, rjw@rjwysocki.net, juri.lelli@arm.com, linux-kernel@vger.kernel.org Subject: [RFCv3 PATCH 10/48] sched: move cfs task on a CPU with higher capacity Date: Wed, 4 Feb 2015 18:30:47 +0000 Message-Id: <1423074685-6336-11-git-send-email-morten.rasmussen@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1423074685-6336-1-git-send-email-morten.rasmussen@arm.com> References: <1423074685-6336-1-git-send-email-morten.rasmussen@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: list List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: morten.rasmussen@arm.com X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.169 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , From: Vincent Guittot When a CPU is used to handle a lot of IRQs or some RT tasks, the remaining capacity for CFS tasks can be significantly reduced. Once we detect such situation by comparing cpu_capacity_orig and cpu_capacity, we trig an idle load balance to check if it's worth moving its tasks on an idle CPU. Once the idle load_balance has selected the busiest CPU, it will look for an active load balance for only two cases : - there is only 1 task on the busiest CPU. - we haven't been able to move a task of the busiest rq. A CPU with a reduced capacity is included in the 1st case, and it's worth to actively migrate its task if the idle CPU has got full capacity. This test has been added in need_active_balance. As a sidenote, this will note generate more spurious ilb because we already trig an ilb if there is more than 1 busy cpu. If this cpu is the only one that has a task, we will trig the ilb once for migrating the task. The nohz_kick_needed function has been cleaned up a bit while adding the new test env.src_cpu and env.src_rq must be set unconditionnally because they are used in need_active_balance which is called even if busiest->nr_running equals 1 Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 74 ++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 53 insertions(+), 21 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 52c494f..bd73f26 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6841,6 +6841,28 @@ static int need_active_balance(struct lb_env *env) return 1; } + /* + * The dst_cpu is idle and the src_cpu CPU has only 1 CFS task. + * It's worth migrating the task if the src_cpu's capacity is reduced + * because of other sched_class or IRQs whereas capacity stays + * available on dst_cpu. + */ + if ((env->idle != CPU_NOT_IDLE) && + (env->src_rq->cfs.h_nr_running == 1)) { + unsigned long src_eff_capacity, dst_eff_capacity; + + dst_eff_capacity = 100; + dst_eff_capacity *= capacity_of(env->dst_cpu); + dst_eff_capacity *= capacity_orig_of(env->src_cpu); + + src_eff_capacity = sd->imbalance_pct; + src_eff_capacity *= capacity_of(env->src_cpu); + src_eff_capacity *= capacity_orig_of(env->dst_cpu); + + if (src_eff_capacity < dst_eff_capacity) + return 1; + } + return unlikely(sd->nr_balance_failed > sd->cache_nice_tries+2); } @@ -6940,6 +6962,9 @@ static int load_balance(int this_cpu, struct rq *this_rq, schedstat_add(sd, lb_imbalance[idle], env.imbalance); + env.src_cpu = busiest->cpu; + env.src_rq = busiest; + ld_moved = 0; if (busiest->nr_running > 1) { /* @@ -6949,8 +6974,6 @@ static int load_balance(int this_cpu, struct rq *this_rq, * correctly treated as an imbalance. */ env.flags |= LBF_ALL_PINNED; - env.src_cpu = busiest->cpu; - env.src_rq = busiest; env.loop_max = min(sysctl_sched_nr_migrate, busiest->nr_running); more_balance: @@ -7650,22 +7673,25 @@ static void nohz_idle_balance(struct rq *this_rq, enum cpu_idle_type idle) /* * Current heuristic for kicking the idle load balancer in the presence - * of an idle cpu is the system. + * of an idle cpu in the system. * - This rq has more than one task. - * - At any scheduler domain level, this cpu's scheduler group has multiple - * busy cpu's exceeding the group's capacity. + * - This rq has at least one CFS task and the capacity of the CPU is + * significantly reduced because of RT tasks or IRQs. + * - At parent of LLC scheduler domain level, this cpu's scheduler group has + * multiple busy cpu. * - For SD_ASYM_PACKING, if the lower numbered cpu's in the scheduler * domain span are idle. */ -static inline int nohz_kick_needed(struct rq *rq) +static inline bool nohz_kick_needed(struct rq *rq) { unsigned long now = jiffies; struct sched_domain *sd; struct sched_group_capacity *sgc; int nr_busy, cpu = rq->cpu; + bool kick = false; if (unlikely(rq->idle_balance)) - return 0; + return false; /* * We may be recently in ticked or tickless idle mode. At the first @@ -7679,38 +7705,44 @@ static inline int nohz_kick_needed(struct rq *rq) * balancing. */ if (likely(!atomic_read(&nohz.nr_cpus))) - return 0; + return false; if (time_before(now, nohz.next_balance)) - return 0; + return false; if (rq->nr_running >= 2) - goto need_kick; + return true; rcu_read_lock(); sd = rcu_dereference(per_cpu(sd_busy, cpu)); - if (sd) { sgc = sd->groups->sgc; nr_busy = atomic_read(&sgc->nr_busy_cpus); - if (nr_busy > 1) - goto need_kick_unlock; + if (nr_busy > 1) { + kick = true; + goto unlock; + } + } - sd = rcu_dereference(per_cpu(sd_asym, cpu)); + sd = rcu_dereference(rq->sd); + if (sd) { + if ((rq->cfs.h_nr_running >= 1) && + check_cpu_capacity(rq, sd)) { + kick = true; + goto unlock; + } + } + sd = rcu_dereference(per_cpu(sd_asym, cpu)); if (sd && (cpumask_first_and(nohz.idle_cpus_mask, sched_domain_span(sd)) < cpu)) - goto need_kick_unlock; + kick = true; +unlock: rcu_read_unlock(); - return 0; - -need_kick_unlock: - rcu_read_unlock(); -need_kick: - return 1; + return kick; } #else static void nohz_idle_balance(struct rq *this_rq, enum cpu_idle_type idle) { }