From patchwork Thu Apr 25 09:37:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 162836 Delivered-To: patch@linaro.org Received: by 2002:a02:c6d8:0:0:0:0:0 with SMTP id r24csp1701182jan; Thu, 25 Apr 2019 02:37:53 -0700 (PDT) X-Google-Smtp-Source: APXvYqy748vyC1yfe7ezo74RWF5mnmU5sv2TM0QhIkl+6L1WR3wMZwJdqHqP2jHK01uWdBpPfbed X-Received: by 2002:a63:5041:: with SMTP id q1mr13072253pgl.386.1556185073775; Thu, 25 Apr 2019 02:37:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556185073; cv=none; d=google.com; s=arc-20160816; b=ynY4e1CtLXrVxzMgGTFU+KkWCGcfyAcLJmwaufCPrA0Wi+oCmcO535x7VlAUhc/tns WqevfMSykOrRO8ZhiEFWV+zXpRQ01OSIU+OOniaQLUj/546bi1iVFZzgGCHErknceO2M FSlBC4urO74Aa+e32PUTij35zC9VYJIMWdAIaoRWs9ycq+N6cFeKviCl7Vo+J0vYvYN/ Exiu1Vu78aYAANfutWgzabLMXz2tS7GNgOuxBfxSLzAGKHxXNc6zQp783RE8U52TM12N tPp7IMzfY0ubQmwcwgT9KXX1XmFKsjXzn0Fvu2lRRjC8WftpkzQpif3X9o4mhouMzLpT /M8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=iq/bivqDNw4UWEitcfvy3I+JHGmEQEUsypk4F+POh50=; b=OtXWPj/O5nB7ouvRKbB+p6n3q6/gifsKXEPLwO2YO1Iz+rpj6OpFSAzSutJ076sS/K jUb3rnji7OM40pciL0fW51YPIwYbzdw0I2CD8RsFf+5SrIKlwVCW69pabCmv0BqyxobW lm9J9lBBzo+CdSOkt524WrHsYPjeb1KQlMtBjUU0lkAGhBNP0YeyJibu+t1LE3F4DKIg 6g8i/zNSBxhH7+5PiMuLPJrRGSdjXu1dMnhxd/7QhpLxTamUbN6VyqnPxMbLnTxO1Hcz w+AIhEpDXnDP7MOgB4YVSDuAwV907VozyBBiwi9sCXRpaIChKrH4sOwjByH0KKhlqnet DB4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=pM9GmN8F; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p2si20120987pgk.326.2019.04.25.02.37.53; Thu, 25 Apr 2019 02:37:53 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=pM9GmN8F; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728650AbfDYJhw (ORCPT + 30 others); Thu, 25 Apr 2019 05:37:52 -0400 Received: from mail-pl1-f194.google.com ([209.85.214.194]:45948 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728258AbfDYJht (ORCPT ); Thu, 25 Apr 2019 05:37:49 -0400 Received: by mail-pl1-f194.google.com with SMTP id o5so6483089pls.12 for ; Thu, 25 Apr 2019 02:37:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=iq/bivqDNw4UWEitcfvy3I+JHGmEQEUsypk4F+POh50=; b=pM9GmN8F0uzTCNoL074eI2mPsgbkwh0i9EM7iZ2buiBA+82fygsTV1YSb4zel73rAU HxPKdULUwLWNm4yfdqHOX4+8Db3s4TOe9CTNtBmYnZcVg5DvBN/3MjWkwLSmLOI0WT/y 917900Gz178TyLvXmj5nK1JGTeLBW4dFQv0JOceOIefGDLv9VSn0YB53GM3ZJ5qDraDs 1tQ7A5+jCTqbE6XbSSNhd4AEjcx3ieY4tqs/d6ylrO86xTzL5rwrm94+MO3yMo0sRN/y uWsk94MGfHqF2H5/bKTIJEtyLuZvMMd0iHt44jrJTjHmEamOc61+MYdsFsrbU23wqj5c R7Ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iq/bivqDNw4UWEitcfvy3I+JHGmEQEUsypk4F+POh50=; b=OsKKwQy9h9/GQFOQ0ffCTkirk3KmSbOSqTTCWefaiI1ytRZsOm2UUUZrMrLua7aZDn DdNuvA2X2QVIJlSOluZS4g7YZaAy8vi/y5+JjLGAxrYtRJb7BJkfoF3t7IA6nk+aHY4+ L6xEtSoYD8AS2jK31vbkSrVizI37K6UI4tRSjM3vnW5HSxohiLcE1VJ0bwALWAKtU5op Dh2kvnwzNB7rfz1fH/KidlIrFmqLgm1WK5Ffp32HM+j78vWEk9+HiQPlXzxI/sC4ugdr +EyuJgE+PfxjAyLSpDcMqBy/wpy9GdCgXQZtFuw2Foc2zuTPtd0JMIywR+XAl2Q0kw// Vhaw== X-Gm-Message-State: APjAAAUVLCw0qvM0nuuUrWOFTKwq78M0w0ImxHgHfIN4DZ9couQrZNNV pYxNfDEKiA1vq84gIyFkBYMq5VnrIcI= X-Received: by 2002:a17:902:8d83:: with SMTP id v3mr38598391plo.283.1556185067744; Thu, 25 Apr 2019 02:37:47 -0700 (PDT) Received: from localhost ([122.166.139.136]) by smtp.gmail.com with ESMTPSA id h4sm29206595pfo.119.2019.04.25.02.37.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 25 Apr 2019 02:37:47 -0700 (PDT) From: Viresh Kumar To: Ingo Molnar , Peter Zijlstra Cc: Viresh Kumar , Vincent Guittot , tkjos@google.com, Daniel Lezcano , quentin.perret@linaro.org, chris.redpath@arm.com, Dietmar.Eggemann@arm.com, linux-kernel@vger.kernel.org Subject: [RFC V2 1/2] sched: Start tracking SCHED_IDLE tasks count in cfs_rq Date: Thu, 25 Apr 2019 15:07:39 +0530 Message-Id: X-Mailer: git-send-email 2.21.0.rc0.269.g1a574e7a288b In-Reply-To: References: MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Start tracking how many tasks with SCHED_IDLE policy are present in each cfs_rq. This will be used by later commits. Signed-off-by: Viresh Kumar --- kernel/sched/fair.c | 14 ++++++++++++-- kernel/sched/sched.h | 2 ++ 2 files changed, 14 insertions(+), 2 deletions(-) -- 2.21.0.rc0.269.g1a574e7a288b diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 4275eb07c0b2..6511cb57acdd 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4475,7 +4475,7 @@ static void throttle_cfs_rq(struct cfs_rq *cfs_rq) struct rq *rq = rq_of(cfs_rq); struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg); struct sched_entity *se; - long task_delta, dequeue = 1; + long task_delta, idle_task_delta, dequeue = 1; bool empty; se = cfs_rq->tg->se[cpu_of(rq_of(cfs_rq))]; @@ -4486,6 +4486,7 @@ static void throttle_cfs_rq(struct cfs_rq *cfs_rq) rcu_read_unlock(); task_delta = cfs_rq->h_nr_running; + idle_task_delta = cfs_rq->idle_h_nr_running; for_each_sched_entity(se) { struct cfs_rq *qcfs_rq = cfs_rq_of(se); /* throttled entity or throttle-on-deactivate */ @@ -4495,6 +4496,7 @@ static void throttle_cfs_rq(struct cfs_rq *cfs_rq) if (dequeue) dequeue_entity(qcfs_rq, se, DEQUEUE_SLEEP); qcfs_rq->h_nr_running -= task_delta; + qcfs_rq->idle_h_nr_running -= idle_task_delta; if (qcfs_rq->load.weight) dequeue = 0; @@ -4534,7 +4536,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg); struct sched_entity *se; int enqueue = 1; - long task_delta; + long task_delta, idle_task_delta; se = cfs_rq->tg->se[cpu_of(rq)]; @@ -4554,6 +4556,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) return; task_delta = cfs_rq->h_nr_running; + idle_task_delta = cfs_rq->idle_h_nr_running; for_each_sched_entity(se) { if (se->on_rq) enqueue = 0; @@ -4562,6 +4565,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) if (enqueue) enqueue_entity(cfs_rq, se, ENQUEUE_WAKEUP); cfs_rq->h_nr_running += task_delta; + cfs_rq->idle_h_nr_running += idle_task_delta; if (cfs_rq_throttled(cfs_rq)) break; @@ -5166,6 +5170,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) { struct cfs_rq *cfs_rq; struct sched_entity *se = &p->se; + int idle_h_nr_running = unlikely(task_has_idle_policy(p)) ? 1 : 0; /* * The code below (indirectly) updates schedutil which looks at @@ -5198,6 +5203,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (cfs_rq_throttled(cfs_rq)) break; cfs_rq->h_nr_running++; + cfs_rq->idle_h_nr_running += idle_h_nr_running; flags = ENQUEUE_WAKEUP; } @@ -5205,6 +5211,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) for_each_sched_entity(se) { cfs_rq = cfs_rq_of(se); cfs_rq->h_nr_running++; + cfs_rq->idle_h_nr_running += idle_h_nr_running; if (cfs_rq_throttled(cfs_rq)) break; @@ -5266,6 +5273,7 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags) struct cfs_rq *cfs_rq; struct sched_entity *se = &p->se; int task_sleep = flags & DEQUEUE_SLEEP; + int idle_h_nr_running = unlikely(task_has_idle_policy(p)) ? 1 : 0; for_each_sched_entity(se) { cfs_rq = cfs_rq_of(se); @@ -5280,6 +5288,7 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (cfs_rq_throttled(cfs_rq)) break; cfs_rq->h_nr_running--; + cfs_rq->idle_h_nr_running -= idle_h_nr_running; /* Don't dequeue parent if it has other entities besides us */ if (cfs_rq->load.weight) { @@ -5299,6 +5308,7 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags) for_each_sched_entity(se) { cfs_rq = cfs_rq_of(se); cfs_rq->h_nr_running--; + cfs_rq->idle_h_nr_running -= idle_h_nr_running; if (cfs_rq_throttled(cfs_rq)) break; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index b52ed1ada0be..0f07d9b55a79 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -490,6 +490,8 @@ struct cfs_rq { unsigned long runnable_weight; unsigned int nr_running; unsigned int h_nr_running; + /* h_nr_running for SCHED_IDLE tasks */ + unsigned int idle_h_nr_running; u64 exec_clock; u64 min_vruntime; From patchwork Thu Apr 25 09:37:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 162837 Delivered-To: patch@linaro.org Received: by 2002:a02:c6d8:0:0:0:0:0 with SMTP id r24csp1701234jan; Thu, 25 Apr 2019 02:37:56 -0700 (PDT) X-Google-Smtp-Source: APXvYqw0EYIzW/hdYEOeqkIIZ/LahDNlNxLrH5tik3A6AACTK5D1IIt3mhqMY3Sklv7L/70c7Atb X-Received: by 2002:aa7:9a99:: with SMTP id w25mr16959993pfi.240.1556185076112; Thu, 25 Apr 2019 02:37:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556185076; cv=none; d=google.com; s=arc-20160816; b=VlwcNMJUG/APbvSvma8Pf6pyGyt0opaPiAxujgCgY5BIGZyfuaUjlyZ18ZSoioCMie 6xnFyr9uoVwf06mF8kG1YZVa5xmrwlPZ92zN2PB8YRS/VfnArUvPDOyzb3UjnGSw2AYX fh3GH1jEutj+anwrFnNgUJUZbpYTsRtCbhCvz5Homrb23gkl8phd7yxF4bj3MkVGhHvo OZs4QA+SU++R9r77CQKxbSXOcVVGtKGgakTTTT/tWazpVYt0le/Hmjxzay19k4FlYHml PR414c86uZzeyQlIM7xHbwTfDA98TjmNoQM/ykNCufYL2p4qpkIO7efqYd5idEaTo8U2 2+0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=VxJHwMzwUC98NetKMBaSVmux2SmEBHzy7mtAhsKX81g=; b=zfets2zaH2KXzZypVOAn13LD1sDgvnh7nwPWMB3UJVscX2rFZ9qx8ylUKCUC5a5rxX wXB3fLXsaeer/ASPMbvgNZOn+dUa5Nfw5Xph7AnYJ8gXItS6LI0GGJv+6Cdm5TUyvjuf 2RMuoDCrSq4zLUKrkn7xQCvK3OuBJWWqmAtBFgjKkEbEnHCd92Qmz3TMS8Pjl8icebZh czc1t1kJAJVdlkU9GCFOCjnYgdAru6ieTMD0DiFlRrBf/4MKTpD7t1QVjJapIJickFwK TRVcYNDdL06gHY4dmJ/I2EMgGFyFuLHKBQ2do03dxrHCuNzjgEly0gzUIr7lr+GDSsRq h3AQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=D7k0W7bi; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id c12si20614179pgj.461.2019.04.25.02.37.55; Thu, 25 Apr 2019 02:37:56 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=D7k0W7bi; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728683AbfDYJhy (ORCPT + 30 others); Thu, 25 Apr 2019 05:37:54 -0400 Received: from mail-pl1-f194.google.com ([209.85.214.194]:34051 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728102AbfDYJhw (ORCPT ); Thu, 25 Apr 2019 05:37:52 -0400 Received: by mail-pl1-f194.google.com with SMTP id y6so10831791plt.1 for ; Thu, 25 Apr 2019 02:37:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VxJHwMzwUC98NetKMBaSVmux2SmEBHzy7mtAhsKX81g=; b=D7k0W7bi+xA2D+e+ltNJAPSgTGiJWE2lFoihqveBXLl44pk3niJa9NTdPbh2WrMMig T/k4eOG/q3M4oMs7VflyUURz0JTx5d0NQckbSwOtEv6wexIA9U0EbGYyMI9Oc22je+ef 46yHHl0CJY95v7ISduXRs8mupNH/G0uYl9paXlwasBw4MNij9AenhXPWfPFno/jnNlmV +PXbap3M7CvxWx06poiISMFME5Whg803sNTXS6S583CsgTDRxVAzzYxE5IRmcu6kIAi5 Jp5FGEfwNq5xhnnDI5Ms3fWJaedUTiOyi72k/FintRuXdkoW6wZRo5nO7hjCJs8EwFZN heHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VxJHwMzwUC98NetKMBaSVmux2SmEBHzy7mtAhsKX81g=; b=Rp/vyTnPmmGhMMFB1ik1iXq6VR/mlGxZra5wasWjvqWLXYscDgZGdgGntZyTOYOwJF R/7Hzi6UnO+fYhMr4BKD9RpLwPUKKNCRht8a17d8ArO/+ZJb4f81iroGlgYX43JwUUqo kOAKXjEynSHI58629qm8AiLkT1NeCXeKP0x0PsfEYF1nxXjbQkj/H8V3mlomHw3t886x bEYKhfRAVcAUw0Mfjy6KqwDDumMfUhpey80y9oRGagzm2Qi6PyRrOV4l9BEMoUKWWZwm oek21ZZnS3vA1mcPY7FYx+S3j6HOfIckGB/s4FkwQ6Xw7D66Dm1W9/+F1IIXQ1+MUQT8 f6Ww== X-Gm-Message-State: APjAAAUEjY2wqv9BkwtJkV/Najplok1GsRUaxYRVm3zvvquX6snB2v52 dC2pvhN3HLT96FzeKjE7hrfnXw== X-Received: by 2002:a17:902:7d8f:: with SMTP id a15mr37534600plm.3.1556185070478; Thu, 25 Apr 2019 02:37:50 -0700 (PDT) Received: from localhost ([122.166.139.136]) by smtp.gmail.com with ESMTPSA id b13sm30552400pfd.12.2019.04.25.02.37.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 25 Apr 2019 02:37:49 -0700 (PDT) From: Viresh Kumar To: Ingo Molnar , Peter Zijlstra Cc: Viresh Kumar , Vincent Guittot , tkjos@google.com, Daniel Lezcano , quentin.perret@linaro.org, chris.redpath@arm.com, Dietmar.Eggemann@arm.com, linux-kernel@vger.kernel.org Subject: [RFC V2 2/2] sched/fair: Fallback to sched-idle CPU if idle CPU isn't found Date: Thu, 25 Apr 2019 15:07:40 +0530 Message-Id: <59b37c56b8fcb834f7d3234e776eaeff74ad117f.1556182965.git.viresh.kumar@linaro.org> X-Mailer: git-send-email 2.21.0.rc0.269.g1a574e7a288b In-Reply-To: References: MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org We target for an idle CPU in select_idle_sibling() to run the next task, but in case we don't find idle CPUs it is better to pick a CPU which will run the task the soonest, for performance reason. A CPU which isn't idle but has only SCHED_IDLE activity queued on it should be a good target based on this criteria as any normal fair task will most likely preempt the currently running SCHED_IDLE task immediately. In fact, choosing a SCHED_IDLE CPU shall give better results as it should be able to run the task sooner than an idle CPU (which requires to be woken up from an idle state). This patch updates the fast path to fallback to a sched-idle CPU if the idle CPU isn't found, the slow path can be updated separately later. Following is the order in which select_idle_sibling() picks up next CPU to run the task now: 1. idle_cpu(target) OR sched_idle_cpu(target) 2. idle_cpu(prev) OR sched_idle_cpu(prev) 3. idle_cpu(recent_used_cpu) OR sched_idle_cpu(recent_used_cpu) 4. idle core(sd) 5. idle_cpu(sd) 6. sched_idle_cpu(sd) 7. idle_cpu(p) - smt 8. sched_idle_cpu(p)- smt Though the policy can be tweaked a bit if we want to have different priorities. Signed-off-by: Viresh Kumar --- kernel/sched/fair.c | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) -- 2.21.0.rc0.269.g1a574e7a288b diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 6511cb57acdd..fbaefb9a9296 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6057,6 +6057,15 @@ static inline int find_idlest_cpu(struct sched_domain *sd, struct task_struct *p return new_cpu; } +/* CPU only has SCHED_IDLE tasks enqueued */ +static int sched_idle_cpu(int cpu) +{ + struct rq *rq = cpu_rq(cpu); + + return unlikely(rq->nr_running == rq->cfs.idle_h_nr_running && + rq->nr_running); +} + #ifdef CONFIG_SCHED_SMT DEFINE_STATIC_KEY_FALSE(sched_smt_present); EXPORT_SYMBOL_GPL(sched_smt_present); @@ -6154,7 +6163,7 @@ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, int */ static int select_idle_smt(struct task_struct *p, int target) { - int cpu; + int cpu, si_cpu = -1; if (!static_branch_likely(&sched_smt_present)) return -1; @@ -6164,9 +6173,11 @@ static int select_idle_smt(struct task_struct *p, int target) continue; if (available_idle_cpu(cpu)) return cpu; + if (si_cpu == -1 && sched_idle_cpu(cpu)) + si_cpu = cpu; } - return -1; + return si_cpu; } #else /* CONFIG_SCHED_SMT */ @@ -6194,7 +6205,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t u64 avg_cost, avg_idle; u64 time, cost; s64 delta; - int cpu, nr = INT_MAX; + int cpu, nr = INT_MAX, si_cpu = -1; this_sd = rcu_dereference(*this_cpu_ptr(&sd_llc)); if (!this_sd) @@ -6222,11 +6233,13 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t for_each_cpu_wrap(cpu, sched_domain_span(sd), target) { if (!--nr) - return -1; + return si_cpu; if (!cpumask_test_cpu(cpu, &p->cpus_allowed)) continue; if (available_idle_cpu(cpu)) break; + if (si_cpu == -1 && sched_idle_cpu(cpu)) + si_cpu = cpu; } time = local_clock() - time; @@ -6245,13 +6258,14 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) struct sched_domain *sd; int i, recent_used_cpu; - if (available_idle_cpu(target)) + if (available_idle_cpu(target) || sched_idle_cpu(target)) return target; /* * If the previous CPU is cache affine and idle, don't be stupid: */ - if (prev != target && cpus_share_cache(prev, target) && available_idle_cpu(prev)) + if (prev != target && cpus_share_cache(prev, target) && + (available_idle_cpu(prev) || sched_idle_cpu(prev))) return prev; /* Check a recently used CPU as a potential idle candidate: */ @@ -6259,7 +6273,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) if (recent_used_cpu != prev && recent_used_cpu != target && cpus_share_cache(recent_used_cpu, target) && - available_idle_cpu(recent_used_cpu) && + (available_idle_cpu(recent_used_cpu) || sched_idle_cpu(recent_used_cpu)) && cpumask_test_cpu(p->recent_used_cpu, &p->cpus_allowed)) { /* * Replace recent_used_cpu with prev as it is a potential