From patchwork Wed Mar 10 05:23:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 396669 Delivered-To: patch@linaro.org Received: by 2002:a02:8562:0:0:0:0:0 with SMTP id g89csp92311jai; Tue, 9 Mar 2021 21:24:42 -0800 (PST) X-Google-Smtp-Source: ABdhPJzlzpjcG6pCH8pJM+n+pVG6mNwsL2hoM/Mn9r4nmfE13zwUS6Cgzon6skiKqJHRXyTlfIS4 X-Received: by 2002:a17:906:7015:: with SMTP id n21mr1659065ejj.391.1615353882439; Tue, 09 Mar 2021 21:24:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1615353882; cv=none; d=google.com; s=arc-20160816; b=ypNnieiizKd184RZ/aRedCEYRERcxoikjZeoOl4cWscFQo/CBs/pJsZxEhBwfcSHHE Q+PABullW4riCn+MmiiNVDcx3HrZcCszZbNpGJXzZuMs3DCbfcizPLIZP/wfHt2O3yMk UxH3WEEC0PkdCMo6aXfqZbuHcXUMNtBqcXwf6iEGl/tyghlzO6vtU+srYdOyVQhveOgD 2uTQV1CHhweTFtOFKryEqYmsGFG1inYU2pTcYep9RxorLjoW+aH/TDBG5i/SIxVgg65f OgFnaj3k7TNJEwEVtx6rcuWA/wBQbOz7tLiDGIBbcMgg8c40Ho7132PKOzMMtqdWfNXB nTSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=gXPftVm0Zz9RXWAlvb9njkCnptfxOnMIFmu/ocdebw0=; b=UpyrICgZ30P/LZ7RTrZul+L7iGJ7PKw52Pd+Tempht8fY1H98wf9GhlmewffqprOTW dk5C+DomcwkLUS6FzN7BnnQjz5UWiDdW1+BcE2DVpknLU8lfgg8ssPrKYqUNEiDz9n7c MTpyu7cn7dtde8ImdxsB8fbGAJbSttdbjRrStv6vZmk9UKiNVrEpDN6/BS9pKyYV5lfA XT0SIJ7eSQSsr2+Xip2TQJS/FAo+VEx2nOoWxWwLsCLRGNQFmpDWrqpHgUiXDxXkxACG y/eg7+yuTCpOf7FTN5GsG6ygdCxxVzJ7ElREvzJpCN9NmjUZhOa8wEOH7FGxwIjXiYsX h5kg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=h5+1uQxe; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id u19si10720380edq.33.2021.03.09.21.24.42; Tue, 09 Mar 2021 21:24:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=h5+1uQxe; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229483AbhCJFYG (ORCPT + 7 others); Wed, 10 Mar 2021 00:24:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54308 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229557AbhCJFXo (ORCPT ); Wed, 10 Mar 2021 00:23:44 -0500 Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 62E9AC061760 for ; Tue, 9 Mar 2021 21:23:44 -0800 (PST) Received: by mail-pl1-x629.google.com with SMTP id d23so4689094plq.2 for ; Tue, 09 Mar 2021 21:23:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gXPftVm0Zz9RXWAlvb9njkCnptfxOnMIFmu/ocdebw0=; b=h5+1uQxetPSjw8PACHK4t2Hd7vuH90JC6wRZq6oSAr8QycfGdMjzlfDxYz7UPNs3fE aDipwaizdLvcrXJmgIwgVsPbDXRHBrjM2Ffxw7BTeYHyahm/g90kElvQ3d9rj4KC7NhV BuNDfIjuhXTcKFLFyB3fQV7Sfaaq6AEOIWiZ/ExmDnVp6FdYreeMALKRdw0GJ379k504 Ln6D2d+eN45oJSVdvHjS70ZNp1uAmrJ5IijFqhJ/jluNKdWGPelt/+E+MyN7U9lz6X5R AGVcCgiYFeBhgNSUgcXaphlisYiaf6y2AyeoRYehUEEnZtySl24rGgSo21c3iSNRu5cR 2Qzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gXPftVm0Zz9RXWAlvb9njkCnptfxOnMIFmu/ocdebw0=; b=VbP6Q9Jt7JYxFtFk5ZxwDvceyhd0S8U3odl48x3LFd4X79vFpWUg5J46sQLCdfprc6 kjJsblQDgwZNXgf/yjr7iriXiDFEmvi6PoYUKaDIvMSaxwQmuszoowUjUGTQeKwUU3aa ceV4P+mCmQEmgXgWAClfpxFXWgzujP8OUBrk9Y3ly09wSMFBmo3ZzXjHGQW0LpCZV8DA rM3bija7Or7ixS1v4ph0BuhhkOFV46U5Ttb3LzfhZ+eYucOWiMRrJc5vYS72COw8KSqG Sd5YB4o+cBc6iLseypEM2xaxLTDJE4NjUup+GnZUw0wpybMDI3mTRiS+k3RLJ8Y8rueo EA5Q== X-Gm-Message-State: AOAM533HqdDaHQadIjhd9pUSby/zemLozwHvXTZQIt4fkHu16KUUKClp W0Z3Ts1bbgY69CGFwWMrCufRJw== X-Received: by 2002:a17:90b:2307:: with SMTP id mt7mr1736715pjb.37.1615353823991; Tue, 09 Mar 2021 21:23:43 -0800 (PST) Received: from localhost ([122.171.124.15]) by smtp.gmail.com with ESMTPSA id u15sm14361469pfk.128.2021.03.09.21.23.43 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Mar 2021 21:23:43 -0800 (PST) From: Viresh Kumar To: Rafael Wysocki , Catalin Marinas , Will Deacon , Sudeep Holla , Greg Kroah-Hartman , "Rafael J. Wysocki" Cc: Viresh Kumar , linux-pm@vger.kernel.org, Vincent Guittot , Ionela Voinescu , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH V6 1/4] arch_topology: Rename freq_scale as arch_freq_scale Date: Wed, 10 Mar 2021 10:53:23 +0530 Message-Id: <2dd2c60c0268550f566c3c83d06b651b4f0e9e4e.1615351622.git.viresh.kumar@linaro.org> X-Mailer: git-send-email 2.25.0.rc1.19.g042ed3e048af In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Rename freq_scale to a less generic name, as it will get exported soon for modules. Since x86 already names its own implementation of this as arch_freq_scale, lets stick to that. Suggested-by: Will Deacon Signed-off-by: Viresh Kumar --- arch/arm64/kernel/topology.c | 6 +++--- drivers/base/arch_topology.c | 4 ++-- include/linux/arch_topology.h | 4 ++-- 3 files changed, 7 insertions(+), 7 deletions(-) -- 2.25.0.rc1.19.g042ed3e048af Acked-by: Will Deacon diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c index e08a4126453a..aa8d893619ed 100644 --- a/arch/arm64/kernel/topology.c +++ b/arch/arm64/kernel/topology.c @@ -256,8 +256,8 @@ static int init_amu_fie_callback(struct notifier_block *nb, unsigned long val, * initialized AMU support and enabled invariance. The AMU counters will * keep on working just fine in the absence of the cpufreq driver, and * for the CPUs for which there are no counters available, the last set - * value of freq_scale will remain valid as that is the frequency those - * CPUs are running at. + * value of arch_freq_scale will remain valid as that is the frequency + * those CPUs are running at. */ return 0; @@ -327,7 +327,7 @@ void topology_scale_freq_tick(void) const_cnt - prev_const_cnt); scale = min_t(unsigned long, scale, SCHED_CAPACITY_SCALE); - this_cpu_write(freq_scale, (unsigned long)scale); + this_cpu_write(arch_freq_scale, (unsigned long)scale); } #ifdef CONFIG_ACPI_CPPC_LIB diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c index de8587cc119e..2a1cecbde0a4 100644 --- a/drivers/base/arch_topology.c +++ b/drivers/base/arch_topology.c @@ -31,7 +31,7 @@ __weak bool arch_freq_counters_available(const struct cpumask *cpus) { return false; } -DEFINE_PER_CPU(unsigned long, freq_scale) = SCHED_CAPACITY_SCALE; +DEFINE_PER_CPU(unsigned long, arch_freq_scale) = SCHED_CAPACITY_SCALE; void topology_set_freq_scale(const struct cpumask *cpus, unsigned long cur_freq, unsigned long max_freq) @@ -53,7 +53,7 @@ void topology_set_freq_scale(const struct cpumask *cpus, unsigned long cur_freq, scale = (cur_freq << SCHED_CAPACITY_SHIFT) / max_freq; for_each_cpu(i, cpus) - per_cpu(freq_scale, i) = scale; + per_cpu(arch_freq_scale, i) = scale; } DEFINE_PER_CPU(unsigned long, cpu_scale) = SCHED_CAPACITY_SCALE; diff --git a/include/linux/arch_topology.h b/include/linux/arch_topology.h index 0f6cd6b73a61..583af517f123 100644 --- a/include/linux/arch_topology.h +++ b/include/linux/arch_topology.h @@ -23,11 +23,11 @@ static inline unsigned long topology_get_cpu_scale(int cpu) void topology_set_cpu_scale(unsigned int cpu, unsigned long capacity); -DECLARE_PER_CPU(unsigned long, freq_scale); +DECLARE_PER_CPU(unsigned long, arch_freq_scale); static inline unsigned long topology_get_freq_scale(int cpu) { - return per_cpu(freq_scale, cpu); + return per_cpu(arch_freq_scale, cpu); } void topology_set_freq_scale(const struct cpumask *cpus, unsigned long cur_freq, From patchwork Wed Mar 10 05:23:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 396668 Delivered-To: patch@linaro.org Received: by 2002:a02:8562:0:0:0:0:0 with SMTP id g89csp92296jai; Tue, 9 Mar 2021 21:24:41 -0800 (PST) X-Google-Smtp-Source: ABdhPJwBIB22iKFqEZL21mJ0mo2jxIEeJPCjIpJSto4+omq8YHTpnc/iQpBpwXoNWrBtmUYZLopF X-Received: by 2002:a17:906:fa04:: with SMTP id lo4mr1724781ejb.44.1615353881687; Tue, 09 Mar 2021 21:24:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1615353881; cv=none; d=google.com; s=arc-20160816; b=S6+MdfnwE76vDRljmKns2E8Yf9OzUj5p9YZhej0JAMzk4shsDfOwVeryZ1QsFkadmk 0GfiWUzWCI3y24nTtZY2W0q9IhTIVQrZRzF3zEUocS7WmJutyALFKQMH5rdYz1pCoSfv 1gg+dbOkT1pCstOxe40ReJPv07WOFsDdTLoeRi3HbX/TW0ddNKI+HjtZa99WfHlo+VjC mSgNdqw1TtZDUBEQS2zHBjNykWT9mP0Ftbxl9OqcUGrsd7bpd6ogS1YKQckgn6e2J0qz mH+rLweq2hy+s/Kdd6WD9tgMVhxdFI4l7j1FWFidVwgtvWT9SnQNhkXTwR/H6+ygx981 Tfzg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=MEAXzUrWlVVEIkd6/enraEtoxzINjvaaB6cgDN5URsY=; b=uI/UUDV9m96z9kcvkKzwXdMY9xPOcuNMDB/OqoP8uj1QM+w0jFzZ59CAUpzlLbTTQY RHJ+uL0+ly5CxBuJE77r979zf0LP4ZTEsNqTlggY4Ogu7MEu58IWav748FgLeEeMCCVm E60bZmNinn3VT6bWh85gsmTnb7FbapdcAuN8WAWGMw1zNKyfr18mtqOL2VzXiJuSo+dZ Am4J5IyPynyRXVojh9WvKuYYAehqcygn9pVc/H4DTUnhuoUbMjdoJaAfzcsx0HYK+08d nLC6aosPobTAAowoaO6cba61F6lnfeIKOXLFjjB+QBgq8Bcfi3BdMBX4crf+7ov6seEw sTVg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="UYF/XgzB"; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id u19si10720380edq.33.2021.03.09.21.24.41; Tue, 09 Mar 2021 21:24:41 -0800 (PST) Received-SPF: pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="UYF/XgzB"; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229810AbhCJFYG (ORCPT + 7 others); Wed, 10 Mar 2021 00:24:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54324 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229624AbhCJFXr (ORCPT ); Wed, 10 Mar 2021 00:23:47 -0500 Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 94E40C06174A for ; Tue, 9 Mar 2021 21:23:47 -0800 (PST) Received: by mail-pf1-x432.google.com with SMTP id l7so11289884pfd.3 for ; Tue, 09 Mar 2021 21:23:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MEAXzUrWlVVEIkd6/enraEtoxzINjvaaB6cgDN5URsY=; b=UYF/XgzB8+EImKu+A2bembjT2GS9ALv2830P/N09gDxWWu9Masz7a0S5cFBeevpkUg oXn2yc3u4vVJEEIfu4+xSYyVxf7JSPSWE921YjpC3pJaEfD6iXq9qBw49Og+Q2e8j8+h aGUQsgIZ5nU0jtvxWYMnbiXuzFY9MdC7i11IaZjmkOBH7jvQVjEOcKmHi98IEYs/xFkR m5aahpLhhQS8FZ3haE+DhxiG2UwoS7YdTiviFq2qgDWu1AOnNHpAHnQZY5eIkpABdWFl zRTHI+Xk56RJIeOW5/XrQ5TVikrOUXnlLIaGBZnhp86GeBDSkSkvKms0sKg4tJMCQ7fh 0eUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MEAXzUrWlVVEIkd6/enraEtoxzINjvaaB6cgDN5URsY=; b=eHYaZtyQbD+SFrFBZVo7VdQwJOzZltnnyDaL0yVMD56ZC5OhtTtPHY94NToCtWF1sP 18dT86ADjgkEgTBUoaiXU0aIQFSXHoWB4Tca9UnKtv/OFB8c/tDWc9TYwyk0/Lxklgeg pWNfWOllCGx34IpBltjkJkxMTwjsdohqNGESIya5KagtfkrxD+ljzq8GpDB5Y7FEZXcg faMC2Z76jtJ8Hf1VBzEqZTMNWYgnZ7VHY6i51fV5XONYQobj8m/W1B5iosR3rB4K2grY Kfa+WhOMQ3GpaTjuIPwptBgqM5lqZKupSfAtGusEECBRK44aAfwHzN1eXG/DvtMpVeFk wouA== X-Gm-Message-State: AOAM530XNUCeuUUW2O+fSXi23SXdA/N2qqSTN2QbLg+WcvkG1YqNcF/i qmRN1SMmzsUOpJOqcoY8cJ+NOA== X-Received: by 2002:a63:ce18:: with SMTP id y24mr1321967pgf.246.1615353827115; Tue, 09 Mar 2021 21:23:47 -0800 (PST) Received: from localhost ([122.171.124.15]) by smtp.gmail.com with ESMTPSA id u7sm14268026pfh.150.2021.03.09.21.23.46 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Mar 2021 21:23:46 -0800 (PST) From: Viresh Kumar To: Rafael Wysocki , Catalin Marinas , Will Deacon , Sudeep Holla , Greg Kroah-Hartman , "Rafael J. Wysocki" Cc: Viresh Kumar , linux-pm@vger.kernel.org, Vincent Guittot , Ionela Voinescu , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH V6 2/4] arch_topology: Allow multiple entities to provide sched_freq_tick() callback Date: Wed, 10 Mar 2021 10:53:24 +0530 Message-Id: X-Mailer: git-send-email 2.25.0.rc1.19.g042ed3e048af In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org This patch attempts to make it generic enough so other parts of the kernel can also provide their own implementation of scale_freq_tick() callback, which is called by the scheduler periodically to update the per-cpu arch_freq_scale variable. The implementations now need to provide 'struct scale_freq_data' for the CPUs for which they have hardware counters available, and a callback gets registered for each possible CPU in a per-cpu variable. The arch specific (or ARM AMU) counters are updated to adapt to this and they take the highest priority if they are available, i.e. they will be used instead of CPPC based counters for example. The special code to rebuild the sched domains, in case invariance status change for the system, is moved out of arm64 specific code and is added to arch_topology.c. Note that this also defines SCALE_FREQ_SOURCE_CPUFREQ but doesn't use it and it is added to show that cpufreq is also acts as source of information for FIE and will be used by default if no other counters are supported for a platform. Reviewed-by: Ionela Voinescu Tested-by: Ionela Voinescu Acked-by: Will Deacon # for arm64 Tested-by: Vincent Guittot Signed-off-by: Viresh Kumar --- arch/arm64/include/asm/topology.h | 10 +-- arch/arm64/kernel/topology.c | 105 +++++++++++------------------- drivers/base/arch_topology.c | 82 +++++++++++++++++++++-- include/linux/arch_topology.h | 14 +++- 4 files changed, 131 insertions(+), 80 deletions(-) -- 2.25.0.rc1.19.g042ed3e048af diff --git a/arch/arm64/include/asm/topology.h b/arch/arm64/include/asm/topology.h index 3b8dca4eb08d..ec2db3419c41 100644 --- a/arch/arm64/include/asm/topology.h +++ b/arch/arm64/include/asm/topology.h @@ -17,17 +17,9 @@ int pcibus_to_node(struct pci_bus *bus); #include void update_freq_counters_refs(void); -void topology_scale_freq_tick(void); - -#ifdef CONFIG_ARM64_AMU_EXTN -/* - * Replace task scheduler's default counter-based - * frequency-invariance scale factor setting. - */ -#define arch_scale_freq_tick topology_scale_freq_tick -#endif /* CONFIG_ARM64_AMU_EXTN */ /* Replace task scheduler's default frequency-invariant accounting */ +#define arch_scale_freq_tick topology_scale_freq_tick #define arch_set_freq_scale topology_set_freq_scale #define arch_scale_freq_capacity topology_get_freq_scale #define arch_scale_freq_invariant topology_scale_freq_invariant diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c index aa8d893619ed..4dd14a6620c1 100644 --- a/arch/arm64/kernel/topology.c +++ b/arch/arm64/kernel/topology.c @@ -199,12 +199,47 @@ static int freq_inv_set_max_ratio(int cpu, u64 max_rate, u64 ref_rate) return 0; } -static DEFINE_STATIC_KEY_FALSE(amu_fie_key); -#define amu_freq_invariant() static_branch_unlikely(&amu_fie_key) +static void amu_scale_freq_tick(void) +{ + u64 prev_core_cnt, prev_const_cnt; + u64 core_cnt, const_cnt, scale; + + prev_const_cnt = this_cpu_read(arch_const_cycles_prev); + prev_core_cnt = this_cpu_read(arch_core_cycles_prev); + + update_freq_counters_refs(); + + const_cnt = this_cpu_read(arch_const_cycles_prev); + core_cnt = this_cpu_read(arch_core_cycles_prev); + + if (unlikely(core_cnt <= prev_core_cnt || + const_cnt <= prev_const_cnt)) + return; + + /* + * /\core arch_max_freq_scale + * scale = ------- * -------------------- + * /\const SCHED_CAPACITY_SCALE + * + * See validate_cpu_freq_invariance_counters() for details on + * arch_max_freq_scale and the use of SCHED_CAPACITY_SHIFT. + */ + scale = core_cnt - prev_core_cnt; + scale *= this_cpu_read(arch_max_freq_scale); + scale = div64_u64(scale >> SCHED_CAPACITY_SHIFT, + const_cnt - prev_const_cnt); + + scale = min_t(unsigned long, scale, SCHED_CAPACITY_SCALE); + this_cpu_write(arch_freq_scale, (unsigned long)scale); +} + +static struct scale_freq_data amu_sfd = { + .source = SCALE_FREQ_SOURCE_ARCH, + .set_freq_scale = amu_scale_freq_tick, +}; static void amu_fie_setup(const struct cpumask *cpus) { - bool invariant; int cpu; /* We are already set since the last insmod of cpufreq driver */ @@ -221,25 +256,10 @@ static void amu_fie_setup(const struct cpumask *cpus) cpumask_or(amu_fie_cpus, amu_fie_cpus, cpus); - invariant = topology_scale_freq_invariant(); - - /* We aren't fully invariant yet */ - if (!invariant && !cpumask_equal(amu_fie_cpus, cpu_present_mask)) - return; - - static_branch_enable(&amu_fie_key); + topology_set_scale_freq_source(&amu_sfd, amu_fie_cpus); pr_debug("CPUs[%*pbl]: counters will be used for FIE.", cpumask_pr_args(cpus)); - - /* - * Task scheduler behavior depends on frequency invariance support, - * either cpufreq or counter driven. If the support status changes as - * a result of counter initialisation and use, retrigger the build of - * scheduling domains to ensure the information is propagated properly. - */ - if (!invariant) - rebuild_sched_domains_energy(); } static int init_amu_fie_callback(struct notifier_block *nb, unsigned long val, @@ -283,53 +303,6 @@ static int __init init_amu_fie(void) } core_initcall(init_amu_fie); -bool arch_freq_counters_available(const struct cpumask *cpus) -{ - return amu_freq_invariant() && - cpumask_subset(cpus, amu_fie_cpus); -} - -void topology_scale_freq_tick(void) -{ - u64 prev_core_cnt, prev_const_cnt; - u64 core_cnt, const_cnt, scale; - int cpu = smp_processor_id(); - - if (!amu_freq_invariant()) - return; - - if (!cpumask_test_cpu(cpu, amu_fie_cpus)) - return; - - prev_const_cnt = this_cpu_read(arch_const_cycles_prev); - prev_core_cnt = this_cpu_read(arch_core_cycles_prev); - - update_freq_counters_refs(); - - const_cnt = this_cpu_read(arch_const_cycles_prev); - core_cnt = this_cpu_read(arch_core_cycles_prev); - - if (unlikely(core_cnt <= prev_core_cnt || - const_cnt <= prev_const_cnt)) - return; - - /* - * /\core arch_max_freq_scale - * scale = ------- * -------------------- - * /\const SCHED_CAPACITY_SCALE - * - * See validate_cpu_freq_invariance_counters() for details on - * arch_max_freq_scale and the use of SCHED_CAPACITY_SHIFT. - */ - scale = core_cnt - prev_core_cnt; - scale *= this_cpu_read(arch_max_freq_scale); - scale = div64_u64(scale >> SCHED_CAPACITY_SHIFT, - const_cnt - prev_const_cnt); - - scale = min_t(unsigned long, scale, SCHED_CAPACITY_SCALE); - this_cpu_write(arch_freq_scale, (unsigned long)scale); -} - #ifdef CONFIG_ACPI_CPPC_LIB #include diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c index 2a1cecbde0a4..ebcd2ea3091f 100644 --- a/drivers/base/arch_topology.c +++ b/drivers/base/arch_topology.c @@ -21,16 +21,90 @@ #include #include +static DEFINE_PER_CPU(struct scale_freq_data *, sft_data); +static struct cpumask scale_freq_counters_mask; +static bool scale_freq_invariant; + +static bool supports_scale_freq_counters(const struct cpumask *cpus) +{ + return cpumask_subset(cpus, &scale_freq_counters_mask); +} + bool topology_scale_freq_invariant(void) { return cpufreq_supports_freq_invariance() || - arch_freq_counters_available(cpu_online_mask); + supports_scale_freq_counters(cpu_online_mask); } -__weak bool arch_freq_counters_available(const struct cpumask *cpus) +static void update_scale_freq_invariant(bool status) { - return false; + if (scale_freq_invariant == status) + return; + + /* + * Task scheduler behavior depends on frequency invariance support, + * either cpufreq or counter driven. If the support status changes as + * a result of counter initialisation and use, retrigger the build of + * scheduling domains to ensure the information is propagated properly. + */ + if (topology_scale_freq_invariant() == status) { + scale_freq_invariant = status; + rebuild_sched_domains_energy(); + } } + +void topology_set_scale_freq_source(struct scale_freq_data *data, + const struct cpumask *cpus) +{ + struct scale_freq_data *sfd; + int cpu; + + /* + * Avoid calling rebuild_sched_domains() unnecessarily if FIE is + * supported by cpufreq. + */ + if (cpumask_empty(&scale_freq_counters_mask)) + scale_freq_invariant = topology_scale_freq_invariant(); + + for_each_cpu(cpu, cpus) { + sfd = per_cpu(sft_data, cpu); + + /* Use ARCH provided counters whenever possible */ + if (!sfd || sfd->source != SCALE_FREQ_SOURCE_ARCH) { + per_cpu(sft_data, cpu) = data; + cpumask_set_cpu(cpu, &scale_freq_counters_mask); + } + } + + update_scale_freq_invariant(true); +} + +void topology_clear_scale_freq_source(enum scale_freq_source source, + const struct cpumask *cpus) +{ + struct scale_freq_data *sfd; + int cpu; + + for_each_cpu(cpu, cpus) { + sfd = per_cpu(sft_data, cpu); + + if (sfd && sfd->source == source) { + per_cpu(sft_data, cpu) = NULL; + cpumask_clear_cpu(cpu, &scale_freq_counters_mask); + } + } + + update_scale_freq_invariant(false); +} + +void topology_scale_freq_tick(void) +{ + struct scale_freq_data *sfd = *this_cpu_ptr(&sft_data); + + if (sfd) + sfd->set_freq_scale(); +} + DEFINE_PER_CPU(unsigned long, arch_freq_scale) = SCHED_CAPACITY_SCALE; void topology_set_freq_scale(const struct cpumask *cpus, unsigned long cur_freq, @@ -47,7 +121,7 @@ void topology_set_freq_scale(const struct cpumask *cpus, unsigned long cur_freq, * want to update the scale factor with information from CPUFREQ. * Instead the scale factor will be updated from arch_scale_freq_tick. */ - if (arch_freq_counters_available(cpus)) + if (supports_scale_freq_counters(cpus)) return; scale = (cur_freq << SCHED_CAPACITY_SHIFT) / max_freq; diff --git a/include/linux/arch_topology.h b/include/linux/arch_topology.h index 583af517f123..11e555cfaecb 100644 --- a/include/linux/arch_topology.h +++ b/include/linux/arch_topology.h @@ -34,7 +34,19 @@ void topology_set_freq_scale(const struct cpumask *cpus, unsigned long cur_freq, unsigned long max_freq); bool topology_scale_freq_invariant(void); -bool arch_freq_counters_available(const struct cpumask *cpus); +enum scale_freq_source { + SCALE_FREQ_SOURCE_CPUFREQ = 0, + SCALE_FREQ_SOURCE_ARCH, +}; + +struct scale_freq_data { + enum scale_freq_source source; + void (*set_freq_scale)(void); +}; + +void topology_scale_freq_tick(void); +void topology_set_scale_freq_source(struct scale_freq_data *data, const struct cpumask *cpus); +void topology_clear_scale_freq_source(enum scale_freq_source source, const struct cpumask *cpus); DECLARE_PER_CPU(unsigned long, thermal_pressure); From patchwork Wed Mar 10 05:23:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 396667 Delivered-To: patch@linaro.org Received: by 2002:a02:8562:0:0:0:0:0 with SMTP id g89csp92293jai; Tue, 9 Mar 2021 21:24:41 -0800 (PST) X-Google-Smtp-Source: ABdhPJwiaCmjX9RCrE9HrgKA23JjNNZHGb930/Dfzmj5RNmCSapF+0Xt6dSeQ2asYq3qyt63ftl1 X-Received: by 2002:a17:907:1b06:: with SMTP id mp6mr1694135ejc.408.1615353881276; Tue, 09 Mar 2021 21:24:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1615353881; cv=none; d=google.com; s=arc-20160816; b=GtigGEPBoQbmvsANO6ORiDXnl8sroBBrD2ISf9l9NCfAcM5diJd4j8SAGKuVsxIVfT YvfHwRZlt1ZTCMXYZiZnaQ90Mtr86oyOF9XKGHI2vwt7/Yzf0tn75gCe2IjkWzCkmI7H cGV9ri7OfoUO7c6gNAfqgPUmrxdVk6qtWWyckzEu5YzyWsANE41Jurvmv+CgakHZrVmY COvIb6jM9y9tzvJv2vANb0XIN0t+apLPaGdDfRKx2kHqc3c/saTqbF5eGMVFcmzHXsSh jhNgixV90g8irivlMwQG2RIRqztgVNZJyUiSkdzqXo8keFL3CvfQQwM4oi2qPSpPrPvM E5JQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=w6wCPUb8jc+iccj9hgb1LHjwvC5J45cqXfco5MhHDUE=; b=AI/vCW/IU9aDpw9ZB5oLf4FXW3Y9In6p3wuym1YiaoAi2Sd+rxTjpZsOZZaL8d0GJV 1e+tQaftoj8r4ijKQ4BRtrJ+XZcJFiksADyO0fCy9o5eqVLEH+TrZQJb84w+zuiQPwT3 0SYsf4m+QZNbZ/GltndfXeLzmvNgnAq6bsPkatdCP+2SW6z5HLBjzFj1ew8BUU4JahfB 0TMWwJHg7Ju1BeH6r/jv5n3joImnSCtVADPTQ4nP/X0HtRG60rvMfyqAWYBdsx2umo0b hjU1a7JVxnRrN74YbWFGKXnUlzm0DBVtceXx20rmABAhbbXtlRKC+m1gSU6RV8We7rbI K4CA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GcDUVEmO; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id u19si10720380edq.33.2021.03.09.21.24.41; Tue, 09 Mar 2021 21:24:41 -0800 (PST) Received-SPF: pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GcDUVEmO; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229624AbhCJFYH (ORCPT + 7 others); Wed, 10 Mar 2021 00:24:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229712AbhCJFXu (ORCPT ); Wed, 10 Mar 2021 00:23:50 -0500 Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 42C09C061760 for ; Tue, 9 Mar 2021 21:23:50 -0800 (PST) Received: by mail-pj1-x1033.google.com with SMTP id q6-20020a17090a4306b02900c42a012202so6829258pjg.5 for ; Tue, 09 Mar 2021 21:23:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=w6wCPUb8jc+iccj9hgb1LHjwvC5J45cqXfco5MhHDUE=; b=GcDUVEmOuOLJi1qjvT3xY70Q/VduglDQg/3OhVeOvPvJ9qfFdveFwVIIfQjIT/XdBW tM9lMLOWZDnsVZvmUUx9+A6PgmSp3B7LdL4z6GqjsUcM4UBcdeL8q1Tl0/2gEUoemmG4 ECwBW+gzKTcWu4h17TLRON6jDRvwVTmkEB3NmVxBxq986TvJFpebyjMs7PisM1mtablQ y1TSdRSZ2m9URtC8qbht97oXOdmfmNwma0+CC0moeRV6dRc0rXpcJSogODML5hwMfW4j l9WayPudva/pFfzQlUf563yEkNeBgSjUn0+TcsKy/liKV6Z11cYKRAMT2KtqhdtjFM5J rojQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=w6wCPUb8jc+iccj9hgb1LHjwvC5J45cqXfco5MhHDUE=; b=TPcQigxBi7flNtqRmmLUwfIH/0UO5vM5bUSwc/HtqaHQLPPwVWWVcmsobsk37MwMpT zGJf91GDiv82xnpSpJKjimhUlu5o3O5knug7lwyg7J9npuFaKzclWJGVFob1iRC82AyS XAa7FbjpJ7LkKFT8qqLZeBkawdy8B4w0wf0Dsigkbx0Wy+ylArlyQd5S2x++S4ggpvv0 mDwU5ojmc4f6OcElIjy8wBP3eg2Kms3LgfTzLrb15hagtAluQhr3C7Q80fA394ihKKa7 6XaiwlRw4ebxSoZbJl4hi70H9iZX7F0kFf2WxbXmySbklMnoY9gclN2o9sYAZfM0FGFC b33A== X-Gm-Message-State: AOAM532zFg5RnQfcEhYuh/kcmcvKYnllQbw7kMlXY5W7RKX39HNDsQpm 1YbQhQGmyDlNlZbuODn3pO7GNg== X-Received: by 2002:a17:90a:d78e:: with SMTP id z14mr1644954pju.221.1615353829821; Tue, 09 Mar 2021 21:23:49 -0800 (PST) Received: from localhost ([122.171.124.15]) by smtp.gmail.com with ESMTPSA id z9sm8755996pgs.32.2021.03.09.21.23.49 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Mar 2021 21:23:49 -0800 (PST) From: Viresh Kumar To: Rafael Wysocki , Sudeep Holla , Greg Kroah-Hartman , "Rafael J. Wysocki" Cc: Viresh Kumar , linux-pm@vger.kernel.org, Vincent Guittot , Ionela Voinescu , linux-kernel@vger.kernel.org Subject: [PATCH V6 3/4] arch_topology: Export arch_freq_scale and helpers Date: Wed, 10 Mar 2021 10:53:25 +0530 Message-Id: <1b5c83549720da480b1e23e542f44b1edb2e4157.1615351622.git.viresh.kumar@linaro.org> X-Mailer: git-send-email 2.25.0.rc1.19.g042ed3e048af In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org It is possible now for other parts of the kernel to provide their own implementation of sched_freq_tick() and they can very well be modules themselves (like CPPC cpufreq driver, which is going to use these in a later commit). Export arch_freq_scale and topology_{set|clear}_scale_freq_source(). Reviewed-by: Ionela Voinescu Tested-by: Ionela Voinescu Tested-by: Vincent Guittot Signed-off-by: Viresh Kumar --- drivers/base/arch_topology.c | 3 +++ 1 file changed, 3 insertions(+) -- 2.25.0.rc1.19.g042ed3e048af diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c index ebcd2ea3091f..995e52b9eca4 100644 --- a/drivers/base/arch_topology.c +++ b/drivers/base/arch_topology.c @@ -78,6 +78,7 @@ void topology_set_scale_freq_source(struct scale_freq_data *data, update_scale_freq_invariant(true); } +EXPORT_SYMBOL_GPL(topology_set_scale_freq_source); void topology_clear_scale_freq_source(enum scale_freq_source source, const struct cpumask *cpus) @@ -96,6 +97,7 @@ void topology_clear_scale_freq_source(enum scale_freq_source source, update_scale_freq_invariant(false); } +EXPORT_SYMBOL_GPL(topology_clear_scale_freq_source); void topology_scale_freq_tick(void) { @@ -106,6 +108,7 @@ void topology_scale_freq_tick(void) } DEFINE_PER_CPU(unsigned long, arch_freq_scale) = SCHED_CAPACITY_SCALE; +EXPORT_SYMBOL_GPL(arch_freq_scale); void topology_set_freq_scale(const struct cpumask *cpus, unsigned long cur_freq, unsigned long max_freq) From patchwork Wed Mar 10 05:23:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 397091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16DCBC4332E for ; Wed, 10 Mar 2021 05:24:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ECF1164FF7 for ; Wed, 10 Mar 2021 05:24:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229920AbhCJFYH (ORCPT ); Wed, 10 Mar 2021 00:24:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229734AbhCJFXx (ORCPT ); Wed, 10 Mar 2021 00:23:53 -0500 Received: from mail-pg1-x533.google.com (mail-pg1-x533.google.com [IPv6:2607:f8b0:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C58CC06174A for ; Tue, 9 Mar 2021 21:23:53 -0800 (PST) Received: by mail-pg1-x533.google.com with SMTP id g4so10616821pgj.0 for ; Tue, 09 Mar 2021 21:23:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MkVUxgwyPS3aEheKZYSBvapF/1MfUHgnW/8MUkfzrCE=; b=o41+CqQkG9FctEKF9oKuahffqzu9TZE/9oxqYUhxniwB3TFnsWlD9AQQAOgDadbjE/ eQZJhsx7IoMJqbME2V37BWBoV7gF43PFep+bRFmABnnBkCIpHb7IJpXs0eTRHBnXagAG HBc+oRWTxoLQdQn8B33suXFnLTjy1RUIcsrQlMlK9wYIYlzUC8kfg03uts5cYWPaEfjg 7kCJTipkfATqVXd73RDiQHb40sInsdBNngusbPHQVHhE/XoW+BSsdfAD8EBrqlXJN/RO iZvw/q73G9npqraJ/yPgCNKTA0N6xum30lHoXe99mDVQL4A5MbW+kNYWV6N8Ba11Y+hE 5EqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MkVUxgwyPS3aEheKZYSBvapF/1MfUHgnW/8MUkfzrCE=; b=Q6ag1pbe5WZKj+rMRUmLnNjso3lV4PJPjFIlGoZ6k8VLSFbHJobnJnkkClN/K7Eh5f pZS+w69jrCHSVMxiL+hFGIc7s+HQfuwNSZ95E7CmId2rmvFIkLrpQFrw/PAt6j3vX0XA +s4hEdLakbmeC33jAu/FoAZDqP/MxrYdOS+k3A9E+GGbYJznLRZS+Wgqp9zOvC+TWyua nzo1rKMeV3Z/Ov7roRu046ITbv6uO616hWRWqTUw8B+S/ziTrqKqrowTv6JY1iNZbKrO 2EUJ+fvovfSf9IgIaNO008E7kILHYMq5l4emfmc+6SSks1AGCdyJz6QqKcUkpxV8lzvt KGtw== X-Gm-Message-State: AOAM532jGXLnWdW8RYwGAdMJhjPeuPp1QMeRNI/IaBsm/tKTH806p/T/ NR4hLagZe6rMbU+OiBR3vdSHAQ== X-Google-Smtp-Source: ABdhPJy6xnfsJrkUj3uOk6RhMOfN24ZWOtCJb02Yz1JiSld9zo6DEIm3wY/Vx5iSksicOWD+yOq52g== X-Received: by 2002:aa7:8d92:0:b029:1ee:75d1:c87 with SMTP id i18-20020aa78d920000b02901ee75d10c87mr1573717pfr.9.1615353832983; Tue, 09 Mar 2021 21:23:52 -0800 (PST) Received: from localhost ([122.171.124.15]) by smtp.gmail.com with ESMTPSA id p184sm14818427pgp.13.2021.03.09.21.23.52 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Mar 2021 21:23:52 -0800 (PST) From: Viresh Kumar To: Rafael Wysocki , Viresh Kumar , Sudeep Holla , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira Cc: linux-pm@vger.kernel.org, Ionela Voinescu , linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V6 4/4] cpufreq: CPPC: Add support for frequency invariance Date: Wed, 10 Mar 2021 10:53:26 +0530 Message-Id: <19fbb10acaaceaa25671c973b9eb6f170015de00.1615351622.git.viresh.kumar@linaro.org> X-Mailer: git-send-email 2.25.0.rc1.19.g042ed3e048af In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org The Frequency Invariance Engine (FIE) is providing a frequency scaling correction factor that helps achieve more accurate load-tracking. Normally, this scaling factor can be obtained directly with the help of the cpufreq drivers as they know the exact frequency the hardware is running at. But that isn't the case for CPPC cpufreq driver. Another way of obtaining that is using the arch specific counter support, which is already present in kernel, but that hardware is optional for platforms. This patch updates the CPPC driver to register itself with the topology core to provide its own implementation (cppc_scale_freq_tick()) of topology_scale_freq_tick() which gets called by the scheduler on every tick. Note that the arch specific counters have higher priority than CPPC counters, if available, though the CPPC driver doesn't need to have any special handling for that. On an invocation of cppc_scale_freq_tick(), we schedule an irq work (since we reach here from hard-irq context), which then schedules a normal work item and cppc_scale_freq_workfn() updates the per_cpu arch_freq_scale variable based on the counter updates since the last tick. To allow platforms to disable this CPPC counter-based frequency invariance support, this is all done under CONFIG_ACPI_CPPC_CPUFREQ_FIE, which is enabled by default. This also exports sched_setattr_nocheck() as the CPPC driver can be built as a module. Cc: linux-acpi@vger.kernel.org Reviewed-by: Ionela Voinescu Tested-by: Ionela Voinescu Tested-by: Vincent Guittot Signed-off-by: Viresh Kumar Acked-by: Rafael J. Wysocki --- drivers/cpufreq/Kconfig.arm | 10 ++ drivers/cpufreq/cppc_cpufreq.c | 245 +++++++++++++++++++++++++++++++-- include/linux/arch_topology.h | 1 + kernel/sched/core.c | 1 + 4 files changed, 245 insertions(+), 12 deletions(-) diff --git a/drivers/cpufreq/Kconfig.arm b/drivers/cpufreq/Kconfig.arm index e65e0a43be64..a5c5f70acfc9 100644 --- a/drivers/cpufreq/Kconfig.arm +++ b/drivers/cpufreq/Kconfig.arm @@ -19,6 +19,16 @@ config ACPI_CPPC_CPUFREQ If in doubt, say N. +config ACPI_CPPC_CPUFREQ_FIE + bool "Frequency Invariance support for CPPC cpufreq driver" + depends on ACPI_CPPC_CPUFREQ && GENERIC_ARCH_TOPOLOGY + default y + help + This extends frequency invariance support in the CPPC cpufreq driver, + by using CPPC delivered and reference performance counters. + + If in doubt, say N. + config ARM_ALLWINNER_SUN50I_CPUFREQ_NVMEM tristate "Allwinner nvmem based SUN50I CPUFreq driver" depends on ARCH_SUNXI diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index 8a482c434ea6..b8e1b8ea628c 100644 --- a/drivers/cpufreq/cppc_cpufreq.c +++ b/drivers/cpufreq/cppc_cpufreq.c @@ -10,14 +10,18 @@ #define pr_fmt(fmt) "CPPC Cpufreq:" fmt +#include #include #include #include #include #include #include +#include +#include #include #include +#include #include @@ -57,6 +61,204 @@ static struct cppc_workaround_oem_info wa_info[] = { } }; +#ifdef CONFIG_ACPI_CPPC_CPUFREQ_FIE + +/* Frequency invariance support */ +struct cppc_freq_invariance { + int cpu; + struct irq_work irq_work; + struct kthread_work work; + struct cppc_perf_fb_ctrs prev_perf_fb_ctrs; + struct cppc_cpudata *cpu_data; +}; + +static DEFINE_PER_CPU(struct cppc_freq_invariance, cppc_freq_inv); +static struct kthread_worker *kworker_fie; +static bool fie_disabled; + +static struct cpufreq_driver cppc_cpufreq_driver; +static unsigned int hisi_cppc_cpufreq_get_rate(unsigned int cpu); +static int cppc_perf_from_fbctrs(struct cppc_cpudata *cpu_data, + struct cppc_perf_fb_ctrs fb_ctrs_t0, + struct cppc_perf_fb_ctrs fb_ctrs_t1); + +/** + * cppc_scale_freq_workfn - CPPC arch_freq_scale updater for frequency invariance + * @work: The work item. + * + * The CPPC driver register itself with the topology core to provide its own + * implementation (cppc_scale_freq_tick()) of topology_scale_freq_tick() which + * gets called by the scheduler on every tick. + * + * Note that the arch specific counters have higher priority than CPPC counters, + * if available, though the CPPC driver doesn't need to have any special + * handling for that. + * + * On an invocation of cppc_scale_freq_tick(), we schedule an irq work (since we + * reach here from hard-irq context), which then schedules a normal work item + * and cppc_scale_freq_workfn() updates the per_cpu arch_freq_scale variable + * based on the counter updates since the last tick. + */ +static void cppc_scale_freq_workfn(struct kthread_work *work) +{ + struct cppc_freq_invariance *cppc_fi; + struct cppc_perf_fb_ctrs fb_ctrs = {0}; + struct cppc_cpudata *cpu_data; + unsigned long local_freq_scale; + u64 perf; + + cppc_fi = container_of(work, struct cppc_freq_invariance, work); + cpu_data = cppc_fi->cpu_data; + + if (cppc_get_perf_ctrs(cppc_fi->cpu, &fb_ctrs)) { + pr_warn("%s: failed to read perf counters\n", __func__); + return; + } + + cppc_fi->prev_perf_fb_ctrs = fb_ctrs; + perf = cppc_perf_from_fbctrs(cpu_data, cppc_fi->prev_perf_fb_ctrs, + fb_ctrs); + + perf <<= SCHED_CAPACITY_SHIFT; + local_freq_scale = div64_u64(perf, cpu_data->perf_caps.highest_perf); + if (WARN_ON(local_freq_scale > 1024)) + local_freq_scale = 1024; + + per_cpu(arch_freq_scale, cppc_fi->cpu) = local_freq_scale; +} + +static void cppc_irq_work(struct irq_work *irq_work) +{ + struct cppc_freq_invariance *cppc_fi; + + cppc_fi = container_of(irq_work, struct cppc_freq_invariance, irq_work); + kthread_queue_work(kworker_fie, &cppc_fi->work); +} + +static void cppc_scale_freq_tick(void) +{ + struct cppc_freq_invariance *cppc_fi = &per_cpu(cppc_freq_inv, smp_processor_id()); + + /* + * cppc_get_perf_ctrs() can potentially sleep, call that from the right + * context. + */ + irq_work_queue(&cppc_fi->irq_work); +} + +static struct scale_freq_data cppc_sftd = { + .source = SCALE_FREQ_SOURCE_CPPC, + .set_freq_scale = cppc_scale_freq_tick, +}; + +static void cppc_freq_invariance_policy_init(struct cpufreq_policy *policy, + struct cppc_cpudata *cpu_data) +{ + struct cppc_perf_fb_ctrs fb_ctrs = {0}; + struct cppc_freq_invariance *cppc_fi; + int i, ret; + + if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate) + return; + + if (fie_disabled) + return; + + for_each_cpu(i, policy->cpus) { + cppc_fi = &per_cpu(cppc_freq_inv, i); + cppc_fi->cpu = i; + cppc_fi->cpu_data = cpu_data; + kthread_init_work(&cppc_fi->work, cppc_scale_freq_workfn); + init_irq_work(&cppc_fi->irq_work, cppc_irq_work); + + ret = cppc_get_perf_ctrs(i, &fb_ctrs); + if (ret) { + pr_warn("%s: failed to read perf counters: %d\n", + __func__, ret); + fie_disabled = true; + } else { + cppc_fi->prev_perf_fb_ctrs = fb_ctrs; + } + } +} + +static void __init cppc_freq_invariance_init(void) +{ + struct sched_attr attr = { + .size = sizeof(struct sched_attr), + .sched_policy = SCHED_DEADLINE, + .sched_nice = 0, + .sched_priority = 0, + /* + * Fake (unused) bandwidth; workaround to "fix" + * priority inheritance. + */ + .sched_runtime = 1000000, + .sched_deadline = 10000000, + .sched_period = 10000000, + }; + int ret; + + if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate) + return; + + if (fie_disabled) + return; + + kworker_fie = kthread_create_worker(0, "cppc_fie"); + if (IS_ERR(kworker_fie)) + return; + + ret = sched_setattr_nocheck(kworker_fie->task, &attr); + if (ret) { + pr_warn("%s: failed to set SCHED_DEADLINE: %d\n", __func__, + ret); + kthread_destroy_worker(kworker_fie); + return; + } + + /* Register for freq-invariance */ + topology_set_scale_freq_source(&cppc_sftd, cpu_present_mask); +} + +static void cppc_freq_invariance_exit(void) +{ + struct cppc_freq_invariance *cppc_fi; + int i; + + if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate) + return; + + if (fie_disabled) + return; + + topology_clear_scale_freq_source(SCALE_FREQ_SOURCE_CPPC, cpu_present_mask); + + for_each_possible_cpu(i) { + cppc_fi = &per_cpu(cppc_freq_inv, i); + irq_work_sync(&cppc_fi->irq_work); + } + + kthread_destroy_worker(kworker_fie); + kworker_fie = NULL; +} + +#else +static inline void +cppc_freq_invariance_policy_init(struct cpufreq_policy *policy, + struct cppc_cpudata *cpu_data) +{ +} + +static inline void cppc_freq_invariance_init(void) +{ +} + +static inline void cppc_freq_invariance_exit(void) +{ +} +#endif /* CONFIG_ACPI_CPPC_CPUFREQ_FIE */ + /* Callback function used to retrieve the max frequency from DMI */ static void cppc_find_dmi_mhz(const struct dmi_header *dm, void *private) { @@ -355,9 +557,12 @@ static int cppc_cpufreq_cpu_init(struct cpufreq_policy *policy) cpu_data->perf_ctrls.desired_perf = caps->highest_perf; ret = cppc_set_perf(cpu, &cpu_data->perf_ctrls); - if (ret) + if (ret) { pr_debug("Err setting perf value:%d on CPU:%d. ret:%d\n", caps->highest_perf, cpu, ret); + } else { + cppc_freq_invariance_policy_init(policy, cpu_data); + } return ret; } @@ -370,12 +575,12 @@ static inline u64 get_delta(u64 t1, u64 t0) return (u32)t1 - (u32)t0; } -static int cppc_get_rate_from_fbctrs(struct cppc_cpudata *cpu_data, - struct cppc_perf_fb_ctrs fb_ctrs_t0, - struct cppc_perf_fb_ctrs fb_ctrs_t1) +static int cppc_perf_from_fbctrs(struct cppc_cpudata *cpu_data, + struct cppc_perf_fb_ctrs fb_ctrs_t0, + struct cppc_perf_fb_ctrs fb_ctrs_t1) { u64 delta_reference, delta_delivered; - u64 reference_perf, delivered_perf; + u64 reference_perf; reference_perf = fb_ctrs_t0.reference_perf; @@ -384,12 +589,21 @@ static int cppc_get_rate_from_fbctrs(struct cppc_cpudata *cpu_data, delta_delivered = get_delta(fb_ctrs_t1.delivered, fb_ctrs_t0.delivered); - /* Check to avoid divide-by zero */ - if (delta_reference || delta_delivered) - delivered_perf = (reference_perf * delta_delivered) / - delta_reference; - else - delivered_perf = cpu_data->perf_ctrls.desired_perf; + /* Check to avoid divide-by zero and invalid delivered_perf */ + if (!delta_reference || !delta_delivered) + return cpu_data->perf_ctrls.desired_perf; + + return (reference_perf * delta_delivered) / delta_reference; +} + +static int cppc_get_rate_from_fbctrs(struct cppc_cpudata *cpu_data, + struct cppc_perf_fb_ctrs fb_ctrs_t0, + struct cppc_perf_fb_ctrs fb_ctrs_t1) +{ + u64 delivered_perf; + + delivered_perf = cppc_perf_from_fbctrs(cpu_data, fb_ctrs_t0, + fb_ctrs_t1); return cppc_cpufreq_perf_to_khz(cpu_data, delivered_perf); } @@ -514,6 +728,8 @@ static void cppc_check_hisi_workaround(void) static int __init cppc_cpufreq_init(void) { + int ret; + if ((acpi_disabled) || !acpi_cpc_valid()) return -ENODEV; @@ -521,7 +737,11 @@ static int __init cppc_cpufreq_init(void) cppc_check_hisi_workaround(); - return cpufreq_register_driver(&cppc_cpufreq_driver); + ret = cpufreq_register_driver(&cppc_cpufreq_driver); + if (!ret) + cppc_freq_invariance_init(); + + return ret; } static inline void free_cpu_data(void) @@ -538,6 +758,7 @@ static inline void free_cpu_data(void) static void __exit cppc_cpufreq_exit(void) { + cppc_freq_invariance_exit(); cpufreq_unregister_driver(&cppc_cpufreq_driver); free_cpu_data(); diff --git a/include/linux/arch_topology.h b/include/linux/arch_topology.h index 11e555cfaecb..f180240dc95f 100644 --- a/include/linux/arch_topology.h +++ b/include/linux/arch_topology.h @@ -37,6 +37,7 @@ bool topology_scale_freq_invariant(void); enum scale_freq_source { SCALE_FREQ_SOURCE_CPUFREQ = 0, SCALE_FREQ_SOURCE_ARCH, + SCALE_FREQ_SOURCE_CPPC, }; struct scale_freq_data { diff --git a/kernel/sched/core.c b/kernel/sched/core.c index ca2bb629595f..3adedc7b1725 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6386,6 +6386,7 @@ int sched_setattr_nocheck(struct task_struct *p, const struct sched_attr *attr) { return __sched_setscheduler(p, attr, false, true); } +EXPORT_SYMBOL_GPL(sched_setattr_nocheck); /** * sched_setscheduler_nocheck - change the scheduling policy and/or RT priority of a thread from kernelspace.