From patchwork Fri Mar 31 01:43:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Dai X-Patchwork-Id: 669181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85693C761AF for ; Fri, 31 Mar 2023 01:44:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229697AbjCaBod (ORCPT ); Thu, 30 Mar 2023 21:44:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47578 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229664AbjCaBob (ORCPT ); Thu, 30 Mar 2023 21:44:31 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 698AA11E9E for ; Thu, 30 Mar 2023 18:44:23 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id y144-20020a253296000000b00b69ce0e6f2dso20617054yby.18 for ; Thu, 30 Mar 2023 18:44:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680227062; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=bQHMl9pwJydZAQbrAoDkSnvavcJ/jzjaFAvojHFhORk=; b=cE6t4ZFZ1GrVpjJSvVne/IoBs3nTvhJx2/x7tbZXEgJWa1Mk0XQ0npmPY8f/9VQI/r K75GmTXHd4epQTJXabgReN4I22zYPK7rPDu8IZR6IGBgbKDvzB9+b6UtiKkohpPFxoPX b2L7y9HVxegUECa/hEkNEht7jAlxcJ1tJvwL9ceHPnOFEWz/igzM3p3WSjHHgg7mxyp8 sRMKMOEZaZQ61niu5ApADy/l4UYir9EHW+X+ghjI8PIwO+2S4AVDQ9XYGpCJ//hJsLPh NDSDv2w67CQ/pth2hxiZBg6Ks/QMp3as8Oqh8rgVpxqdYgDkYf0aXake8sqaJq31zq8n /tIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680227062; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bQHMl9pwJydZAQbrAoDkSnvavcJ/jzjaFAvojHFhORk=; b=23ZIZJxGCBoFEKiI2yC+YDDR2l5MMDpkcIMxOjsuFMSPZdeCBThDyTgml/FrC6G8hM QsWrKK2dmN0/3AFCsUNxtYVSrt0xMfFMZwlQ3y/gcTNsgtO4+k16kbu6O7rutPuWSl76 Z0LB00dqgyz7CYZFod32Aj2MrW3hDTTZnUrXV0r/7DZRx0UIQfGXrG4SRyCkWwzP/yXn Yq2pUChIElKJ7NCQh1ftFFIqkpsHLm8UiI1mqbQmRqqI8kAI6xPVSRCc/QqMEk212vwe P34fw6iOgbFI+1mXDG3hXoHqC9nrLCIrapNp606ajeGDLe/VqhIEd+FACLeTcMHohLJm xcGw== X-Gm-Message-State: AAQBX9dB/100rDCpzY8hwRDPab7XfdFQDuSgd29388SoQsMiJ/KL7uSW IlL9tnApVm/hWHHspI/N3QR34eDRFEQP X-Google-Smtp-Source: AKy350ZivNvxJZ18U+W1XPsWsfvpSQ+cLGN8Aj2+BZDUohIE+m0rmZgZiCfr6NdVfa3nqdRvfK2U2A2N5cMv X-Received: from davidai2.mtv.corp.google.com ([2620:15c:211:201:c162:24e8:ec5e:d520]) (user=davidai job=sendgmr) by 2002:a25:ab11:0:b0:a5f:0:bf12 with SMTP id u17-20020a25ab11000000b00a5f0000bf12mr12526269ybi.13.1680227062710; Thu, 30 Mar 2023 18:44:22 -0700 (PDT) Date: Thu, 30 Mar 2023 18:43:45 -0700 In-Reply-To: <20230331014356.1033759-1-davidai@google.com> Mime-Version: 1.0 References: <20230331014356.1033759-1-davidai@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230331014356.1033759-2-davidai@google.com> Subject: [RFC PATCH v2 1/6] sched/fair: Add util_guest for tasks From: David Dai To: "Rafael J. Wysocki" , Viresh Kumar , Rob Herring , Krzysztof Kozlowski , Paolo Bonzini , Jonathan Corbet , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Mark Rutland , Lorenzo Pieralisi , Sudeep Holla , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , David Dai Cc: Saravana Kannan , kernel-team@android.com, linux-pm@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org For virtualization usecases, util_est and util_avg currently tracked on the host aren't sufficient to accurately represent the workload on vCPU threads, which results in poor frequency selection and performance. For example, when a large workload migrates from a busy vCPU thread to an idle vCPU thread, it incurs additional DVFS ramp-up latencies as util accumulates. Introduce a new "util_guest" member as an additional PELT signal that's independently updated by the guest. When used, it's max aggregated to provide a boost to both task_util and task_util_est. Updating task_util and task_util_est will ensure: -Better task placement decisions for vCPU threads on the host -Correctly updating util_est.ewma during dequeue -Additive util with other threads on the same runqueue for more accurate frequency responses Co-developed-by: Saravana Kannan Signed-off-by: Saravana Kannan Signed-off-by: David Dai --- include/linux/sched.h | 11 +++++++++++ kernel/sched/core.c | 18 +++++++++++++++++- kernel/sched/fair.c | 15 +++++++++++++-- 3 files changed, 41 insertions(+), 3 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 63d242164b1a..d8c346fcdf52 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -445,6 +445,16 @@ struct util_est { #define UTIL_AVG_UNCHANGED 0x80000000 } __attribute__((__aligned__(sizeof(u64)))); +/* + * For sched_setattr_nocheck() (kernel) only + * + * Allow vCPU threads to use UTIL_GUEST as a way to hint the scheduler with more + * accurate utilization info. This is useful when guest kernels have some way of + * tracking its own runqueue's utilization. + * + */ +#define SCHED_FLAG_UTIL_GUEST 0x20000000 + /* * The load/runnable/util_avg accumulates an infinite geometric series * (see __update_load_avg_cfs_rq() in kernel/sched/pelt.c). @@ -499,6 +509,7 @@ struct sched_avg { unsigned long load_avg; unsigned long runnable_avg; unsigned long util_avg; + unsigned long util_guest; struct util_est util_est; } ____cacheline_aligned; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 0d18c3969f90..7700ef5610c1 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2024,6 +2024,16 @@ static inline void uclamp_post_fork(struct task_struct *p) { } static inline void init_uclamp(void) { } #endif /* CONFIG_UCLAMP_TASK */ +static void __setscheduler_task_util(struct task_struct *p, + const struct sched_attr *attr) +{ + + if (likely(!(attr->sched_flags & SCHED_FLAG_UTIL_GUEST))) + return; + + p->se.avg.util_guest = attr->sched_util_min; +} + bool sched_task_on_rq(struct task_struct *p) { return task_on_rq_queued(p); @@ -7561,7 +7571,7 @@ static int __sched_setscheduler(struct task_struct *p, return -EINVAL; } - if (attr->sched_flags & ~(SCHED_FLAG_ALL | SCHED_FLAG_SUGOV)) + if (attr->sched_flags & ~(SCHED_FLAG_ALL | SCHED_FLAG_SUGOV | SCHED_FLAG_UTIL_GUEST)) return -EINVAL; /* @@ -7583,6 +7593,9 @@ static int __sched_setscheduler(struct task_struct *p, if (attr->sched_flags & SCHED_FLAG_SUGOV) return -EINVAL; + if (attr->sched_flags & SCHED_FLAG_UTIL_GUEST) + return -EINVAL; + retval = security_task_setscheduler(p); if (retval) return retval; @@ -7629,6 +7642,8 @@ static int __sched_setscheduler(struct task_struct *p, goto change; if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP) goto change; + if (attr->sched_flags & SCHED_FLAG_UTIL_GUEST) + goto change; p->sched_reset_on_fork = reset_on_fork; retval = 0; @@ -7718,6 +7733,7 @@ static int __sched_setscheduler(struct task_struct *p, __setscheduler_prio(p, newprio); } __setscheduler_uclamp(p, attr); + __setscheduler_task_util(p, attr); if (queued) { /* diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 6986ea31c984..998649554344 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4276,14 +4276,16 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf); static inline unsigned long task_util(struct task_struct *p) { - return READ_ONCE(p->se.avg.util_avg); + return max(READ_ONCE(p->se.avg.util_avg), + READ_ONCE(p->se.avg.util_guest)); } static inline unsigned long _task_util_est(struct task_struct *p) { struct util_est ue = READ_ONCE(p->se.avg.util_est); - return max(ue.ewma, (ue.enqueued & ~UTIL_AVG_UNCHANGED)); + return max_t(unsigned long, READ_ONCE(p->se.avg.util_guest), + max(ue.ewma, (ue.enqueued & ~UTIL_AVG_UNCHANGED))); } static inline unsigned long task_util_est(struct task_struct *p) @@ -6242,6 +6244,15 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) */ util_est_enqueue(&rq->cfs, p); + /* + * The normal code path for host thread enqueue doesn't take into + * account guest task migrations when updating cpufreq util. + * So, always update the cpufreq when a vCPU thread has a + * non-zero util_guest value. + */ + if (READ_ONCE(p->se.avg.util_guest)) + cpufreq_update_util(rq, 0); + /* * If in_iowait is set, the code below may not trigger any cpufreq * utilization updates, so do it here explicitly with the IOWAIT flag