From patchwork Mon Jun 21 09:19:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 464492 Delivered-To: patch@linaro.org Received: by 2002:a05:6638:102:0:0:0:0 with SMTP id x2csp2377302jao; Mon, 21 Jun 2021 02:20:18 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy+hUSZ/OxRF8ZurLLxiRWxPZYfWMlRqHEv1Fx3r1J65i5rMDT3AyDnzdcJirmXYaZXd6pT X-Received: by 2002:a05:6638:24d0:: with SMTP id y16mr16604577jat.41.1624267218585; Mon, 21 Jun 2021 02:20:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1624267218; cv=none; d=google.com; s=arc-20160816; b=zN+G01LdHfHE5j7+i1ADGYSr7mx+c3Bh64r4q482KbqaiIcGsJoiknUiJ7ak6OcqzR ssL3LNTzDKqMxexxCexZxobxu10mNiOJzlNZIrBydmqtCbv/ljbC+tJI+9bcs6Rt2zjH /l5vf8dbW+QVSwF9QqPxqa37lromkrewY+6Rxl9h8YXxIfh9qu58H/1y3Kn8CtPuvXcG utu2FhiahsNv4VzihC8TAAyvdZmlfMrKhi/h1jkG0Fn6rDiMZp7unfrDYvyPGZiVwKg0 NC7u2GHmnSxuxe3pCroTmnXyV4IRIumYTweqInmuyLA8LnMNBMcHQrNBVuSGtPrvmsx2 yIiw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=h+Pr0hB3aQxiWpb5eCV/aiPVZfCXjZdbK/uF+fA2suE=; b=tqJ3G5fj1TKpAwVBV/DyysWifaJNCAWsG4D78uhGsK/WFcGmQrONo7TOPUURQhviGc G/mOt7THwLHE2ifzIbw0FnUFS7b5JpzG0hLGyr04w0++9/uNqJ35kn0IkjU4gPvAfn9K 5cgiBzID+aruFKyYU7bj11qowFQQr+dFObLa56DhybJyFGeo73LFMCtA+6CTwqKSMliD hC1XTqRHFAXJJSP3Iz/NpQJoo9rp8ghAT/OqLGrz4Gv8rrY1SxqPDA5iyXsMioEq3JTV lgTJErKkNBzxWmPk3bYdCsjQszxpx+HcdrIFAzZJMjEPgaJFihSyoXuKqmnceCpi1LJt eiag== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GzDORQUQ; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id v126si1153344iof.68.2021.06.21.02.20.18; Mon, 21 Jun 2021 02:20:18 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GzDORQUQ; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230379AbhFUJWb (ORCPT + 8 others); Mon, 21 Jun 2021 05:22:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44526 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230330AbhFUJWa (ORCPT ); Mon, 21 Jun 2021 05:22:30 -0400 Received: from mail-pg1-x529.google.com (mail-pg1-x529.google.com [IPv6:2607:f8b0:4864:20::529]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9AD40C061756 for ; Mon, 21 Jun 2021 02:20:15 -0700 (PDT) Received: by mail-pg1-x529.google.com with SMTP id i34so8027287pgl.9 for ; Mon, 21 Jun 2021 02:20:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=h+Pr0hB3aQxiWpb5eCV/aiPVZfCXjZdbK/uF+fA2suE=; b=GzDORQUQfU4EyaIz84r9ofxl7S2qiMUmnL8adgMAMRlSCje4XuoLkb9AZt5+OZ/uYc bdmFmBbJfOOzTSdfwKiThKNfnYfOjUzcgK0E1v1AHbdJxj8yDUAmT/WcHTgZF5V86eaW IOmErKVtiXIbnK72LGtHfyBwiLysLsLxXc0MfVqxq9qm/mlS5bUsh24zUD08THTH36H0 7lBFyItmFb6sudEMOE4c8N6ae5aX1P0TbHDDmPPZdn5KYMBx5RczSegJQAcVOZyMWfGg vGNdwbIlfXD/vcmiDSuJsLaxrDRF+VXQsFrVUwIzFWJfHeRnc3YnWU0PCpSzn6OM8NwF B5ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=h+Pr0hB3aQxiWpb5eCV/aiPVZfCXjZdbK/uF+fA2suE=; b=YGOEe24yLkeCtAU5tWCqEEKsjqoo8NEFCSvRTb0Q2vmQS1np7TS3DelADeV8RLC4N4 EQfSRL9KbrsfWQigSaVmwWGEt5BpGsyJJNi6D1Kz7Ds4nxTY4qJ5rcZy1uxcr8xqQfLy w5XWeELiXxZkixOu7M+Es55mKKRgDgXYhTdMOuOOeyvfxRn4VqT+2PuW4QkUXP+eA5w5 NmVFoPH0f9Dw74y1dUpPAIDlXuQREFicwzMMu2JUxqh9XUeM2MLFc2gzELvgjWE753bz 1p0ZehZ9TnNrS+XX1HdB0Z128VuPZl5xbs1P1Wf0Fo0hJBzQhLUvyJwaiO97EqUU9ZSi TlpQ== X-Gm-Message-State: AOAM532Uj7GjOatC3ayAKfv9jhsPirZk9ybQd6nqFd5w2GTb2GQ0V4QA 6FGl1tvzy5L3zEDT9GNIufs5bA== X-Received: by 2002:a63:36c1:: with SMTP id d184mr23133582pga.47.1624267215148; Mon, 21 Jun 2021 02:20:15 -0700 (PDT) Received: from localhost ([136.185.134.182]) by smtp.gmail.com with ESMTPSA id c18sm1521186pfo.143.2021.06.21.02.20.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Jun 2021 02:20:14 -0700 (PDT) From: Viresh Kumar To: Rafael Wysocki , Ionela Voinescu , Viresh Kumar Cc: linux-pm@vger.kernel.org, Vincent Guittot , Qian Cai , "Rafael J. Wysocki" , linux-kernel@vger.kernel.org Subject: [PATCH V3 1/4] cpufreq: cppc: Fix potential memleak in cppc_cpufreq_cpu_init Date: Mon, 21 Jun 2021 14:49:34 +0530 Message-Id: <579689469ed8a7dfd68dcbb41e9191472799a326.1624266901.git.viresh.kumar@linaro.org> X-Mailer: git-send-email 2.31.1.272.g89b43f80a514 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org It's a classic example of memleak, we allocate something, we fail and never free the resources. Make sure we free all resources on policy ->init() failures. Fixes: a28b2bfc099c ("cppc_cpufreq: replace per-cpu data array with a list") Tested-by: Vincent Guittot Signed-off-by: Viresh Kumar --- drivers/cpufreq/cppc_cpufreq.c | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) -- 2.31.1.272.g89b43f80a514 diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index be4f62e2c5f1..35b8ae66d1fb 100644 --- a/drivers/cpufreq/cppc_cpufreq.c +++ b/drivers/cpufreq/cppc_cpufreq.c @@ -256,6 +256,16 @@ static struct cppc_cpudata *cppc_cpufreq_get_cpu_data(unsigned int cpu) return NULL; } +static void cppc_cpufreq_put_cpu_data(struct cpufreq_policy *policy) +{ + struct cppc_cpudata *cpu_data = policy->driver_data; + + list_del(&cpu_data->node); + free_cpumask_var(cpu_data->shared_cpu_map); + kfree(cpu_data); + policy->driver_data = NULL; +} + static int cppc_cpufreq_cpu_init(struct cpufreq_policy *policy) { unsigned int cpu = policy->cpu; @@ -309,7 +319,8 @@ static int cppc_cpufreq_cpu_init(struct cpufreq_policy *policy) default: pr_debug("Unsupported CPU co-ord type: %d\n", policy->shared_type); - return -EFAULT; + ret = -EFAULT; + goto out; } /* @@ -324,10 +335,14 @@ static int cppc_cpufreq_cpu_init(struct cpufreq_policy *policy) cpu_data->perf_ctrls.desired_perf = caps->highest_perf; ret = cppc_set_perf(cpu, &cpu_data->perf_ctrls); - if (ret) - pr_debug("Err setting perf value:%d on CPU:%d. ret:%d\n", - caps->highest_perf, cpu, ret); + if (!ret) + return 0; + pr_debug("Err setting perf value:%d on CPU:%d. ret:%d\n", + caps->highest_perf, cpu, ret); + +out: + cppc_cpufreq_put_cpu_data(policy); return ret; } @@ -345,12 +360,7 @@ static int cppc_cpufreq_cpu_exit(struct cpufreq_policy *policy) pr_debug("Err setting perf value:%d on CPU:%d. ret:%d\n", caps->lowest_perf, cpu, ret); - /* Remove CPU node from list and free driver data for policy */ - free_cpumask_var(cpu_data->shared_cpu_map); - list_del(&cpu_data->node); - kfree(policy->driver_data); - policy->driver_data = NULL; - + cppc_cpufreq_put_cpu_data(policy); return 0; } From patchwork Mon Jun 21 09:19:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 464493 Delivered-To: patch@linaro.org Received: by 2002:a05:6638:102:0:0:0:0 with SMTP id x2csp2377339jao; Mon, 21 Jun 2021 02:20:21 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwPGARUNBgc0FAa6gyHWSqOxGYAphjTZXfGazDmeKUKdrtxctk8oykHdoodqGTpeDUy/J7g X-Received: by 2002:a02:cba8:: with SMTP id v8mr2174238jap.111.1624267221470; Mon, 21 Jun 2021 02:20:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1624267221; cv=none; d=google.com; s=arc-20160816; b=kOP/jakvJ9a06RiErDRpAa1uMF1lHk6PgavTxbtKY49pNwdKt341YJiopHqYKdpqJ2 YPutNNxbn9EN117JvAMdksx4mj4CSYvE0IBVz4n47/pBdl8IpH2UB6eEXnoW4+QmpbUN r05bzqBg9t64xA47bs3Fa2Z6vElI5rZEXqa94k8gpjczB8XQb+TDa0Jqmms2NW95KWQd cMVczoiAVcWFDHn86chuBxweXwHNzm94tV6K8eHnR50b3qQisfaIdM25EO+lT+/FoWOA Q2RH9Mmr3V5guhHkg6R0AFt0Gf3q2MXluas9r9l/UQ9KwdXfcSGqxhmOujOa0VgXzmBU 4ONA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=bMx8BP4MqYPHp6/XglTQi8htgyQDCyPm2g5UMRM0MZY=; b=FGnfazkppoNDPCFQTAOKR3N+sgeO9B5bRk2LEeISP6acNWU0HDkx3hH0WuPoYQp7mY 4Gr3cnNRzsK7sEvQnEtw2q3B6AVhO8zUqsZ/5p/spM5OP4D1LtpHZuY0Hw2YF5v6Xxd6 hwo7tiXiKMoNPIzfwvFIwPreo+vnh808ZkIJvcBtZxieG9PBmddZTXRMCDjcH83yIY0D GYigoaatIY2cZWVt03xlKHsYBI6pbG9YiDo+nnr1R6A6uRSznfKTHOL4kKXPeHEEQmQF knSQl8SkJ8FNmKHmbww1Ti4usqqq3hjo3sYGtqYLxllNxnFY54WqgfNucKXO+LE30I+T rQLg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=iCa0Ce9t; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id v126si1153344iof.68.2021.06.21.02.20.21; Mon, 21 Jun 2021 02:20:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=iCa0Ce9t; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230384AbhFUJWe (ORCPT + 8 others); Mon, 21 Jun 2021 05:22:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44540 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230330AbhFUJWc (ORCPT ); Mon, 21 Jun 2021 05:22:32 -0400 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 68C1EC06175F for ; Mon, 21 Jun 2021 02:20:18 -0700 (PDT) Received: by mail-pl1-x632.google.com with SMTP id y21so2343410plb.4 for ; Mon, 21 Jun 2021 02:20:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bMx8BP4MqYPHp6/XglTQi8htgyQDCyPm2g5UMRM0MZY=; b=iCa0Ce9ts0oLs1Mc9I319xqPvIhFbThuzm0+AGoxvFz9bQyewJwTrt4krfZCsdkkqx JO+qWXwVx0iN4KfJKWFJybLmNSQNxsfTE+nQwBiYEs8I2arWPeh+G5Gib7c3gvT4q/5K 904OQ/JWYo+pzpGkZdotKSxGhLMhlnSsiDyQr+0vcT7wtpHgV7RU6AS+Dig6V+09zkYp z2FAbjA8K73lHkH9LrHCID0QW2Na37AMrEhJP7x1eD9poBMPDD+/AK9/SVXopCgYYZy5 Bz5Vq8iwJLT3MQYDRGod0IDUKhR2LiHYeU/ej8Au/8vu04VKFsNBmMHVinY6Qf46yNX9 gwlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bMx8BP4MqYPHp6/XglTQi8htgyQDCyPm2g5UMRM0MZY=; b=LhjRxIeWqsA/yD6aUrEx3XD31iACcPf5MDZ6bvD0/5S91iy2qcka7K3mOW1qxemVW0 wul+mS6FXMourhK6V+yWt8i4rXqlbPWQaw4u2Nmpr0EPuP99j6YUY4zahNk3AXUxt+Cw bMBxAv7E4zJm6AHrINEumS1akr5AAJUr5YgjyowjB1T7rsHvqIUIYCKqFaVcjIPjcNYn Z+vWfAWhb1PWSsxdOau34pbBZ6u66KGUHt1aByvyfvOsKom3JpsnGQMLMeznlvUz195z lffghhn0TVO08Mk3wlLR7gPCT5KFyKQuad54V+VoJekR5MY/TRmjru1uoRUOJC4Z6gGE BesQ== X-Gm-Message-State: AOAM533i7PvWIK3CESghDyjww4mgVATR21qkPj1P5CuTSbKCIJzizTYX 3Gsqy1IhhmT2r+rWZU+3StMzzA== X-Received: by 2002:a17:90b:1881:: with SMTP id mn1mr2569667pjb.225.1624267217999; Mon, 21 Jun 2021 02:20:17 -0700 (PDT) Received: from localhost ([136.185.134.182]) by smtp.gmail.com with ESMTPSA id u24sm15712153pfm.200.2021.06.21.02.20.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Jun 2021 02:20:17 -0700 (PDT) From: Viresh Kumar To: Rafael Wysocki , Ionela Voinescu , Viresh Kumar Cc: linux-pm@vger.kernel.org, Vincent Guittot , Qian Cai , linux-kernel@vger.kernel.org Subject: [PATCH V3 2/4] cpufreq: cppc: Pass structure instance by reference Date: Mon, 21 Jun 2021 14:49:35 +0530 Message-Id: X-Mailer: git-send-email 2.31.1.272.g89b43f80a514 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Don't pass structure instance by value, pass it by reference instead. Tested-by: Vincent Guittot Signed-off-by: Viresh Kumar --- drivers/cpufreq/cppc_cpufreq.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) -- 2.31.1.272.g89b43f80a514 Reviewed-by: Ionela Voinescu diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index 35b8ae66d1fb..490175d65082 100644 --- a/drivers/cpufreq/cppc_cpufreq.c +++ b/drivers/cpufreq/cppc_cpufreq.c @@ -373,18 +373,18 @@ static inline u64 get_delta(u64 t1, u64 t0) } static int cppc_get_rate_from_fbctrs(struct cppc_cpudata *cpu_data, - struct cppc_perf_fb_ctrs fb_ctrs_t0, - struct cppc_perf_fb_ctrs fb_ctrs_t1) + struct cppc_perf_fb_ctrs *fb_ctrs_t0, + struct cppc_perf_fb_ctrs *fb_ctrs_t1) { u64 delta_reference, delta_delivered; u64 reference_perf, delivered_perf; - reference_perf = fb_ctrs_t0.reference_perf; + reference_perf = fb_ctrs_t0->reference_perf; - delta_reference = get_delta(fb_ctrs_t1.reference, - fb_ctrs_t0.reference); - delta_delivered = get_delta(fb_ctrs_t1.delivered, - fb_ctrs_t0.delivered); + delta_reference = get_delta(fb_ctrs_t1->reference, + fb_ctrs_t0->reference); + delta_delivered = get_delta(fb_ctrs_t1->delivered, + fb_ctrs_t0->delivered); /* Check to avoid divide-by zero */ if (delta_reference || delta_delivered) @@ -415,7 +415,7 @@ static unsigned int cppc_cpufreq_get_rate(unsigned int cpu) if (ret) return ret; - return cppc_get_rate_from_fbctrs(cpu_data, fb_ctrs_t0, fb_ctrs_t1); + return cppc_get_rate_from_fbctrs(cpu_data, &fb_ctrs_t0, &fb_ctrs_t1); } static int cppc_cpufreq_set_boost(struct cpufreq_policy *policy, int state) From patchwork Mon Jun 21 09:19:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 464494 Delivered-To: patch@linaro.org Received: by 2002:a05:6638:102:0:0:0:0 with SMTP id x2csp2377449jao; Mon, 21 Jun 2021 02:20:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzgp1zSc6MinyimmivbbePDOoHMnEe8LjvBLEY6qsFLBagCq/x2zBphE4BuEf9HP+7fm9ce X-Received: by 2002:a6b:7714:: with SMTP id n20mr18352336iom.126.1624267232147; Mon, 21 Jun 2021 02:20:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1624267232; cv=none; d=google.com; s=arc-20160816; b=0REjJu/DPh6x9NfKNBbXKI71Mi1RtMW7bKtqTz8InP7kcJlZnxPFnYb+JAdFBpCErK Hzg44jr7Syhph/wLt4UVaZN9pH8lKc712agocFtqf3hXDyIAMw+BQzfIZELRvoo/jQx0 BT5yml5z5KqE69It8dUe803QHIPZwFVfImGDeswfBTvdrRY8FOwBDbar06/cCzQBlH00 00cwv5hpTVavMizlaHOo4xM5UferpWzAq5Xf0cq/VEyOvOUAPRPWLePMIeN6Nau/Qn5i /Fnr+ekSug5N6LGf5HMLJLEGVA3vOx0Bs6/ZUJkcXkSngwZk/3wqCU2NDP1kAIeln79t cyHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=BmIDZ3cAKPLMmD3sDAHfD3r1HwdBZo6Uo9FfuobBvbk=; b=Bb7opZgukD2HWEdrKH5MIrbLLvnuhl8GwaEH68Rq6BLDzGIe6I0dc+iahS/XNK8dWf bXx5P8eWnvFSst7TFRoVH9HEryfsHisQ/0/aIac5t3qtNn+sDCgVakL4tm55srdy8mpe jykMbw2gZbPDp53ZV3WGIAX0I4gV+a8oMGS2NhmSN1UlSHtq+B/+3KJLxh843qPnjFqR Ka2ZlKkt1B3CbUw7ArJh56dzOdEjOHpKXmzB4KAWQrOnq7E5WaeAdD2YZgQ4db2r/Uaq j8ZVMJ8Ok1SK/e1uTnJI6vZoliogcbtBFGZBmNcykN2YNlK50GspP3kI2ezH+fpluE3l aT+g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="Y+dxkC/p"; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id z10si11008128ioq.37.2021.06.21.02.20.31; Mon, 21 Jun 2021 02:20:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="Y+dxkC/p"; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230381AbhFUJWk (ORCPT + 8 others); Mon, 21 Jun 2021 05:22:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44558 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230402AbhFUJWf (ORCPT ); Mon, 21 Jun 2021 05:22:35 -0400 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2A248C06175F for ; Mon, 21 Jun 2021 02:20:21 -0700 (PDT) Received: by mail-pj1-x1029.google.com with SMTP id k5so9588687pjj.1 for ; Mon, 21 Jun 2021 02:20:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BmIDZ3cAKPLMmD3sDAHfD3r1HwdBZo6Uo9FfuobBvbk=; b=Y+dxkC/pF6ZmWU24hO2eIFxfT96wCod+04pXPhOSRopF5srtkgkiC1mSwfxcQdpV/o 9goLQgwz+WDXK9mwVCNp0F896JJ799cVV3gZo6+oqAVDxyzZZG8Norle7LBNo4e3kpV7 2m1aav/bdFzUyxxqJSWLYaQ4DX5AxwElPm12TxIZ9q5arteGbGfH4mW0A+uybg5cNsXv DfvuB84/7MGqeV7TN/TSM8Mj81W88wbB2irt1snZW1ibV6MDPnXM9d56QPmO3EtVKEKr c30n523ed3hKNl0H/vLjusqBO1aYtW2GjDTFEUeHxpQLx1LZMTqKii9wZt/yx0fvUgl9 TjTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BmIDZ3cAKPLMmD3sDAHfD3r1HwdBZo6Uo9FfuobBvbk=; b=RQsk3ITtKBh+NBXlIBWLsogMZLzPZctTtDps0CHownec4SBIjCXFkBzourQqjTjfx9 iwxkkZ1NQyiCB3okJcregQtP+/1dCIPl6AkeZNEGrrmQxGdU1on4LIBeLEnwof8NzIRJ AWecdcujy/ZiimGugqplfhZb/lp/uT14MEwssUsAXoyx+aZYD68cT0EnrvUxKCthifIr YIDFXdU+Y26pD5jboktUZoHiSUrZBdPLyItrZQRAS4ywQJcQUEWKiimuBvY9RRvBnith NT+qfCOiIJ54UgeQz2M40IPa9fboWbPfA4lmAO2+xc9DIaE+brWwDWE7GUjbMdHyBjL5 m4QA== X-Gm-Message-State: AOAM531APH3Wh5br9j/DXTyyNV9y01qQAPBio9ow6H1nEdTOAT9eha/+ hVWtcnqbPjQlcdOssSc8iHpWHg== X-Received: by 2002:a17:90a:1382:: with SMTP id i2mr30426790pja.221.1624267220762; Mon, 21 Jun 2021 02:20:20 -0700 (PDT) Received: from localhost ([136.185.134.182]) by smtp.gmail.com with ESMTPSA id x1sm3015078pfn.198.2021.06.21.02.20.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Jun 2021 02:20:20 -0700 (PDT) From: Viresh Kumar To: Rafael Wysocki , Ionela Voinescu , Sudeep Holla , Greg Kroah-Hartman , "Rafael J. Wysocki" , Viresh Kumar , Will Deacon Cc: linux-pm@vger.kernel.org, Vincent Guittot , Qian Cai , "Paul E . McKenney" , linux-kernel@vger.kernel.org Subject: [PATCH V3 3/4] arch_topology: Avoid use-after-free for scale_freq_data Date: Mon, 21 Jun 2021 14:49:36 +0530 Message-Id: X-Mailer: git-send-email 2.31.1.272.g89b43f80a514 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Currently topology_scale_freq_tick() (which gets called from scheduler_tick()) may end up using a pointer to "struct scale_freq_data", which was previously cleared by topology_clear_scale_freq_source(), as there is no protection in place here. The users of topology_clear_scale_freq_source() though needs a guarantee that the previously cleared scale_freq_data isn't used anymore, so they can free the related resources. Since topology_scale_freq_tick() is called from scheduler tick, we don't want to add locking in there. Use the RCU update mechanism instead (which is already used by the scheduler's utilization update path) to guarantee race free updates here. synchronize_rcu() makes sure that all RCU critical sections that started before it is called, will finish before it returns. And so the callers of topology_clear_scale_freq_source() don't need to worry about their callback getting called anymore. Cc: Paul E. McKenney Fixes: 01e055c120a4 ("arch_topology: Allow multiple entities to provide sched_freq_tick() callback") Tested-by: Vincent Guittot Signed-off-by: Viresh Kumar --- drivers/base/arch_topology.c | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) -- 2.31.1.272.g89b43f80a514 Reviewed-by: Ionela Voinescu diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c index c1179edc0f3b..921312a8d957 100644 --- a/drivers/base/arch_topology.c +++ b/drivers/base/arch_topology.c @@ -18,10 +18,11 @@ #include #include #include +#include #include #include -static DEFINE_PER_CPU(struct scale_freq_data *, sft_data); +static DEFINE_PER_CPU(struct scale_freq_data __rcu *, sft_data); static struct cpumask scale_freq_counters_mask; static bool scale_freq_invariant; @@ -66,16 +67,20 @@ void topology_set_scale_freq_source(struct scale_freq_data *data, if (cpumask_empty(&scale_freq_counters_mask)) scale_freq_invariant = topology_scale_freq_invariant(); + rcu_read_lock(); + for_each_cpu(cpu, cpus) { - sfd = per_cpu(sft_data, cpu); + sfd = rcu_dereference(*per_cpu_ptr(&sft_data, cpu)); /* Use ARCH provided counters whenever possible */ if (!sfd || sfd->source != SCALE_FREQ_SOURCE_ARCH) { - per_cpu(sft_data, cpu) = data; + rcu_assign_pointer(per_cpu(sft_data, cpu), data); cpumask_set_cpu(cpu, &scale_freq_counters_mask); } } + rcu_read_unlock(); + update_scale_freq_invariant(true); } EXPORT_SYMBOL_GPL(topology_set_scale_freq_source); @@ -86,22 +91,32 @@ void topology_clear_scale_freq_source(enum scale_freq_source source, struct scale_freq_data *sfd; int cpu; + rcu_read_lock(); + for_each_cpu(cpu, cpus) { - sfd = per_cpu(sft_data, cpu); + sfd = rcu_dereference(*per_cpu_ptr(&sft_data, cpu)); if (sfd && sfd->source == source) { - per_cpu(sft_data, cpu) = NULL; + rcu_assign_pointer(per_cpu(sft_data, cpu), NULL); cpumask_clear_cpu(cpu, &scale_freq_counters_mask); } } + rcu_read_unlock(); + + /* + * Make sure all references to previous sft_data are dropped to avoid + * use-after-free races. + */ + synchronize_rcu(); + update_scale_freq_invariant(false); } EXPORT_SYMBOL_GPL(topology_clear_scale_freq_source); void topology_scale_freq_tick(void) { - struct scale_freq_data *sfd = *this_cpu_ptr(&sft_data); + struct scale_freq_data *sfd = rcu_dereference_sched(*this_cpu_ptr(&sft_data)); if (sfd) sfd->set_freq_scale(); From patchwork Mon Jun 21 09:19:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 464495 Delivered-To: patch@linaro.org Received: by 2002:a05:6638:102:0:0:0:0 with SMTP id x2csp2377454jao; Mon, 21 Jun 2021 02:20:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwjSL7gHsAySNud6q4xM+ti9thotbulCkG3K8ybb1wcBObBfxwYWIajPboLHQQNWRx8c7B7 X-Received: by 2002:a05:6e02:219d:: with SMTP id j29mr17385872ila.64.1624267232424; Mon, 21 Jun 2021 02:20:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1624267232; cv=none; d=google.com; s=arc-20160816; b=oF0T504vLc8/smibAM/9Z2YYBur+pclSFOzobjT1VpR9bI+KxeHqk8upZe6QjINATP cnoWuDYYQh4rIzd5oZ5+aL2USyLKnUZJe+QvQcuVufaRmvXHSyycxDvi4Dj5DN2PjX9H IMWhzb3Dcn8L/NaX4y9qrIArp2GX1lkjLXtZtKsFcCh7iE55a0bQgNjb6T5eZKnZ+pu3 TkQhB8GzWdWfOU9P8E83uJmp3co1W2QLfoljXe3w+pdtZ4/sw+tqik9plku/6NGipKV/ YxAgclgCYYRLmN4lmdKiPBgqqepQteLNt4WgfBUQRTTTnDBdTi3LzrUwfgOV5yflmwch jGAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=v9CpQJBHn1mD56CNWpr1Tpf2MsYJr9cLOW/B9ZtLcrQ=; b=VU1TBxsoHuC35ipNcN9u6kLWVzm2wh+mRoCA0XTivE2bn24wbtnRcNJzfO7fFRjoqj ptB184RRgdwpiTBee9rXpv599SKhqFpAR5ipc9YsY/IfEL2wEEcFCcqpR9m4xv885ckJ U9T4fszRRxnzoXWArU0v+FFp1XkyKfpg9PD/fnFpCzPbKAzj10P/f2tRLlj0Yq+tM0hg CHRZAzRjklmUUU5ZNK6cGq1sS5JSZMKyGO15Jwp+wao+m+4lhGjRBQojvJycKkgetx+q KSiZB2OGaVfotbfK3YhRHd3pluLHC1mVAbcK0ZabJrYHypcQuO77B3wvuLJYq9RgGlsL 8/sA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=mylYZcsE; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id z10si11008128ioq.37.2021.06.21.02.20.32; Mon, 21 Jun 2021 02:20:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=mylYZcsE; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230515AbhFUJWo (ORCPT + 8 others); Mon, 21 Jun 2021 05:22:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44586 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230415AbhFUJWj (ORCPT ); Mon, 21 Jun 2021 05:22:39 -0400 Received: from mail-pg1-x52a.google.com (mail-pg1-x52a.google.com [IPv6:2607:f8b0:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A4B5C061767 for ; Mon, 21 Jun 2021 02:20:25 -0700 (PDT) Received: by mail-pg1-x52a.google.com with SMTP id v7so13666841pgl.2 for ; Mon, 21 Jun 2021 02:20:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=v9CpQJBHn1mD56CNWpr1Tpf2MsYJr9cLOW/B9ZtLcrQ=; b=mylYZcsEi4JnjamhB4tmmfsBtgKYnQyvHtWiVfCfxeptprLWITq2TTxuIEIVQ+WFJU KLqOmEqSUbFRr+3cuDl1pHtIpZh8NUVAL/vBxpK7qLghZ5gYZit8v47wR6ZkTqfkxsV9 JuxTTJn9jp6mh31pkrqf8CrbsOh/2cPmMWuhpCtEf+9xUNzXpVrQHhnY4zS1WnviP8HW QbqdWbjjv6pWKwTsXxo/Y+3bPd5l3H3Hl7S+wAtXBKWuRXI0xe3gBpBm3k8UrVyR5Fay D85ZByOC+nl29sKhIfhBluSPdrOEkeGG6mrPJn2dKN3GL0dC3awRYMTso2Qs617wuQvJ bGog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=v9CpQJBHn1mD56CNWpr1Tpf2MsYJr9cLOW/B9ZtLcrQ=; b=H8ci/RZd8XKPi5wLum2nCh02SZgvtjENYmpi02Imt/v6mQGv3ksh+wY7ncSb1VsgVE 00xD0N6VkaBbLLHkVblaAxY5NVCA0ml34X3E9+REY5HBfcgt9zkOgZsV3HXE6nMgzcrn XCBJxxz/g02mV0mNvYrWM0T8MHwhn9G3F/zWx8E9DuYglZDHLV5aJx+BSNGVYcMmhGx+ cSblXhTDoqLUINJsHJrN/YgPlBLafdAqC2s8siZ3Wrik+/lJ1Bc6762w0dmfyDVrFQzV ZAGiJkA8vE03H1PXG1Dj9q/vbaI+pOomNwS28zbXfLQo7yA65HC836XeW1iz3+FzCgan bFEg== X-Gm-Message-State: AOAM532kIqj83FtNMcFP3bEcig8KixdjyS+O/4KsQlQ99NRBVzJH2e8z tD2fyz3AJvTDKLPbJyl8RQDEqw== X-Received: by 2002:a63:5d52:: with SMTP id o18mr23136666pgm.440.1624267224517; Mon, 21 Jun 2021 02:20:24 -0700 (PDT) Received: from localhost ([136.185.134.182]) by smtp.gmail.com with ESMTPSA id c18sm15641423pgf.66.2021.06.21.02.20.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Jun 2021 02:20:24 -0700 (PDT) From: Viresh Kumar To: Rafael Wysocki , Ionela Voinescu , Viresh Kumar , Sudeep Holla , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira Cc: linux-pm@vger.kernel.org, Qian Cai , linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V3 4/4] cpufreq: CPPC: Add support for frequency invariance Date: Mon, 21 Jun 2021 14:49:37 +0530 Message-Id: X-Mailer: git-send-email 2.31.1.272.g89b43f80a514 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org The Frequency Invariance Engine (FIE) is providing a frequency scaling correction factor that helps achieve more accurate load-tracking. Normally, this scaling factor can be obtained directly with the help of the cpufreq drivers as they know the exact frequency the hardware is running at. But that isn't the case for CPPC cpufreq driver. Another way of obtaining that is using the arch specific counter support, which is already present in kernel, but that hardware is optional for platforms. This patch updates the CPPC driver to register itself with the topology core to provide its own implementation (cppc_scale_freq_tick()) of topology_scale_freq_tick() which gets called by the scheduler on every tick. Note that the arch specific counters have higher priority than CPPC counters, if available, though the CPPC driver doesn't need to have any special handling for that. On an invocation of cppc_scale_freq_tick(), we schedule an irq work (since we reach here from hard-irq context), which then schedules a normal work item and cppc_scale_freq_workfn() updates the per_cpu arch_freq_scale variable based on the counter updates since the last tick. To allow platforms to disable this CPPC counter-based frequency invariance support, this is all done under CONFIG_ACPI_CPPC_CPUFREQ_FIE, which is enabled by default. This also exports sched_setattr_nocheck() as the CPPC driver can be built as a module. Cc: linux-acpi@vger.kernel.org Tested-by: Vincent Guittot Signed-off-by: Viresh Kumar --- drivers/cpufreq/Kconfig.arm | 10 ++ drivers/cpufreq/cppc_cpufreq.c | 249 +++++++++++++++++++++++++++++++-- include/linux/arch_topology.h | 1 + kernel/sched/core.c | 1 + 4 files changed, 247 insertions(+), 14 deletions(-) -- 2.31.1.272.g89b43f80a514 Reviewed-by: Ionela Voinescu diff --git a/drivers/cpufreq/Kconfig.arm b/drivers/cpufreq/Kconfig.arm index e65e0a43be64..a5c5f70acfc9 100644 --- a/drivers/cpufreq/Kconfig.arm +++ b/drivers/cpufreq/Kconfig.arm @@ -19,6 +19,16 @@ config ACPI_CPPC_CPUFREQ If in doubt, say N. +config ACPI_CPPC_CPUFREQ_FIE + bool "Frequency Invariance support for CPPC cpufreq driver" + depends on ACPI_CPPC_CPUFREQ && GENERIC_ARCH_TOPOLOGY + default y + help + This extends frequency invariance support in the CPPC cpufreq driver, + by using CPPC delivered and reference performance counters. + + If in doubt, say N. + config ARM_ALLWINNER_SUN50I_CPUFREQ_NVMEM tristate "Allwinner nvmem based SUN50I CPUFreq driver" depends on ARCH_SUNXI diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index 490175d65082..db550fc92931 100644 --- a/drivers/cpufreq/cppc_cpufreq.c +++ b/drivers/cpufreq/cppc_cpufreq.c @@ -10,14 +10,18 @@ #define pr_fmt(fmt) "CPPC Cpufreq:" fmt +#include #include #include #include #include #include #include +#include +#include #include #include +#include #include @@ -57,6 +61,210 @@ static struct cppc_workaround_oem_info wa_info[] = { } }; +#ifdef CONFIG_ACPI_CPPC_CPUFREQ_FIE + +/* Frequency invariance support */ +struct cppc_freq_invariance { + int cpu; + struct irq_work irq_work; + struct kthread_work work; + struct cppc_perf_fb_ctrs prev_perf_fb_ctrs; + struct cppc_cpudata *cpu_data; +}; + +static DEFINE_PER_CPU(struct cppc_freq_invariance, cppc_freq_inv); +static struct kthread_worker *kworker_fie; + +static struct cpufreq_driver cppc_cpufreq_driver; +static unsigned int hisi_cppc_cpufreq_get_rate(unsigned int cpu); +static int cppc_perf_from_fbctrs(struct cppc_cpudata *cpu_data, + struct cppc_perf_fb_ctrs *fb_ctrs_t0, + struct cppc_perf_fb_ctrs *fb_ctrs_t1); + +/** + * cppc_scale_freq_workfn - CPPC arch_freq_scale updater for frequency invariance + * @work: The work item. + * + * The CPPC driver register itself with the topology core to provide its own + * implementation (cppc_scale_freq_tick()) of topology_scale_freq_tick() which + * gets called by the scheduler on every tick. + * + * Note that the arch specific counters have higher priority than CPPC counters, + * if available, though the CPPC driver doesn't need to have any special + * handling for that. + * + * On an invocation of cppc_scale_freq_tick(), we schedule an irq work (since we + * reach here from hard-irq context), which then schedules a normal work item + * and cppc_scale_freq_workfn() updates the per_cpu arch_freq_scale variable + * based on the counter updates since the last tick. + */ +static void cppc_scale_freq_workfn(struct kthread_work *work) +{ + struct cppc_freq_invariance *cppc_fi; + struct cppc_perf_fb_ctrs fb_ctrs = {0}; + struct cppc_cpudata *cpu_data; + unsigned long local_freq_scale; + u64 perf; + + cppc_fi = container_of(work, struct cppc_freq_invariance, work); + cpu_data = cppc_fi->cpu_data; + + if (cppc_get_perf_ctrs(cppc_fi->cpu, &fb_ctrs)) { + pr_warn("%s: failed to read perf counters\n", __func__); + return; + } + + perf = cppc_perf_from_fbctrs(cpu_data, &cppc_fi->prev_perf_fb_ctrs, + &fb_ctrs); + cppc_fi->prev_perf_fb_ctrs = fb_ctrs; + + perf <<= SCHED_CAPACITY_SHIFT; + local_freq_scale = div64_u64(perf, cpu_data->perf_caps.highest_perf); + + /* This can happen due to counter's overflow */ + if (unlikely(local_freq_scale > 1024)) + local_freq_scale = 1024; + + per_cpu(arch_freq_scale, cppc_fi->cpu) = local_freq_scale; +} + +static void cppc_irq_work(struct irq_work *irq_work) +{ + struct cppc_freq_invariance *cppc_fi; + + cppc_fi = container_of(irq_work, struct cppc_freq_invariance, irq_work); + kthread_queue_work(kworker_fie, &cppc_fi->work); +} + +static void cppc_scale_freq_tick(void) +{ + struct cppc_freq_invariance *cppc_fi = &per_cpu(cppc_freq_inv, smp_processor_id()); + + /* + * cppc_get_perf_ctrs() can potentially sleep, call that from the right + * context. + */ + irq_work_queue(&cppc_fi->irq_work); +} + +static struct scale_freq_data cppc_sftd = { + .source = SCALE_FREQ_SOURCE_CPPC, + .set_freq_scale = cppc_scale_freq_tick, +}; + +static void cppc_cpufreq_cpu_fie_init(struct cpufreq_policy *policy) +{ + struct cppc_freq_invariance *cppc_fi; + int cpu, ret; + + if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate) + return; + + for_each_cpu(cpu, policy->cpus) { + cppc_fi = &per_cpu(cppc_freq_inv, cpu); + cppc_fi->cpu = cpu; + cppc_fi->cpu_data = policy->driver_data; + kthread_init_work(&cppc_fi->work, cppc_scale_freq_workfn); + init_irq_work(&cppc_fi->irq_work, cppc_irq_work); + + ret = cppc_get_perf_ctrs(cpu, &cppc_fi->prev_perf_fb_ctrs); + if (ret) { + pr_warn("%s: failed to read perf counters for cpu:%d: %d\n", + __func__, cpu, ret); + return; + } + } + + /* Register for freq-invariance */ + topology_set_scale_freq_source(&cppc_sftd, policy->cpus); +} + +/* + * We free all the resources on policy's removal and not on CPU removal as the + * irq-work are per-cpu and the hotplug core takes care of flushing the pending + * irq-works (hint: smpcfd_dying_cpu()) on CPU hotplug. Even if the kthread-work + * fires on another CPU after the concerned CPU is removed, it won't harm. + * + * We just need to make sure to remove them all on policy->exit(). + */ +static void cppc_cpufreq_cpu_fie_exit(struct cpufreq_policy *policy) +{ + struct cppc_freq_invariance *cppc_fi; + int cpu; + + if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate) + return; + + /* policy->cpus will be empty here, use related_cpus instead */ + topology_clear_scale_freq_source(SCALE_FREQ_SOURCE_CPPC, policy->related_cpus); + + for_each_cpu(cpu, policy->related_cpus) { + cppc_fi = &per_cpu(cppc_freq_inv, cpu); + irq_work_sync(&cppc_fi->irq_work); + kthread_cancel_work_sync(&cppc_fi->work); + } +} + +static void __init cppc_freq_invariance_init(void) +{ + struct sched_attr attr = { + .size = sizeof(struct sched_attr), + .sched_policy = SCHED_DEADLINE, + .sched_nice = 0, + .sched_priority = 0, + /* + * Fake (unused) bandwidth; workaround to "fix" + * priority inheritance. + */ + .sched_runtime = 1000000, + .sched_deadline = 10000000, + .sched_period = 10000000, + }; + int ret; + + if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate) + return; + + kworker_fie = kthread_create_worker(0, "cppc_fie"); + if (IS_ERR(kworker_fie)) + return; + + ret = sched_setattr_nocheck(kworker_fie->task, &attr); + if (ret) { + pr_warn("%s: failed to set SCHED_DEADLINE: %d\n", __func__, + ret); + kthread_destroy_worker(kworker_fie); + return; + } +} + +static void cppc_freq_invariance_exit(void) +{ + if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate) + return; + + kthread_destroy_worker(kworker_fie); + kworker_fie = NULL; +} + +#else +static inline void cppc_cpufreq_cpu_fie_init(struct cpufreq_policy *policy) +{ +} + +static inline void cppc_cpufreq_cpu_fie_exit(struct cpufreq_policy *policy) +{ +} + +static inline void cppc_freq_invariance_init(void) +{ +} + +static inline void cppc_freq_invariance_exit(void) +{ +} +#endif /* CONFIG_ACPI_CPPC_CPUFREQ_FIE */ + /* Callback function used to retrieve the max frequency from DMI */ static void cppc_find_dmi_mhz(const struct dmi_header *dm, void *private) { @@ -335,8 +543,10 @@ static int cppc_cpufreq_cpu_init(struct cpufreq_policy *policy) cpu_data->perf_ctrls.desired_perf = caps->highest_perf; ret = cppc_set_perf(cpu, &cpu_data->perf_ctrls); - if (!ret) + if (!ret) { + cppc_cpufreq_cpu_fie_init(policy); return 0; + } pr_debug("Err setting perf value:%d on CPU:%d. ret:%d\n", caps->highest_perf, cpu, ret); @@ -353,6 +563,8 @@ static int cppc_cpufreq_cpu_exit(struct cpufreq_policy *policy) unsigned int cpu = policy->cpu; int ret; + cppc_cpufreq_cpu_fie_exit(policy); + cpu_data->perf_ctrls.desired_perf = caps->lowest_perf; ret = cppc_set_perf(cpu, &cpu_data->perf_ctrls); @@ -372,12 +584,12 @@ static inline u64 get_delta(u64 t1, u64 t0) return (u32)t1 - (u32)t0; } -static int cppc_get_rate_from_fbctrs(struct cppc_cpudata *cpu_data, - struct cppc_perf_fb_ctrs *fb_ctrs_t0, - struct cppc_perf_fb_ctrs *fb_ctrs_t1) +static int cppc_perf_from_fbctrs(struct cppc_cpudata *cpu_data, + struct cppc_perf_fb_ctrs *fb_ctrs_t0, + struct cppc_perf_fb_ctrs *fb_ctrs_t1) { u64 delta_reference, delta_delivered; - u64 reference_perf, delivered_perf; + u64 reference_perf; reference_perf = fb_ctrs_t0->reference_perf; @@ -386,14 +598,11 @@ static int cppc_get_rate_from_fbctrs(struct cppc_cpudata *cpu_data, delta_delivered = get_delta(fb_ctrs_t1->delivered, fb_ctrs_t0->delivered); - /* Check to avoid divide-by zero */ - if (delta_reference || delta_delivered) - delivered_perf = (reference_perf * delta_delivered) / - delta_reference; - else - delivered_perf = cpu_data->perf_ctrls.desired_perf; + /* Check to avoid divide-by zero and invalid delivered_perf */ + if (!delta_reference || !delta_delivered) + return cpu_data->perf_ctrls.desired_perf; - return cppc_cpufreq_perf_to_khz(cpu_data, delivered_perf); + return (reference_perf * delta_delivered) / delta_reference; } static unsigned int cppc_cpufreq_get_rate(unsigned int cpu) @@ -401,6 +610,7 @@ static unsigned int cppc_cpufreq_get_rate(unsigned int cpu) struct cppc_perf_fb_ctrs fb_ctrs_t0 = {0}, fb_ctrs_t1 = {0}; struct cpufreq_policy *policy = cpufreq_cpu_get(cpu); struct cppc_cpudata *cpu_data = policy->driver_data; + u64 delivered_perf; int ret; cpufreq_cpu_put(policy); @@ -415,7 +625,10 @@ static unsigned int cppc_cpufreq_get_rate(unsigned int cpu) if (ret) return ret; - return cppc_get_rate_from_fbctrs(cpu_data, &fb_ctrs_t0, &fb_ctrs_t1); + delivered_perf = cppc_perf_from_fbctrs(cpu_data, &fb_ctrs_t0, + &fb_ctrs_t1); + + return cppc_cpufreq_perf_to_khz(cpu_data, delivered_perf); } static int cppc_cpufreq_set_boost(struct cpufreq_policy *policy, int state) @@ -516,14 +729,21 @@ static void cppc_check_hisi_workaround(void) static int __init cppc_cpufreq_init(void) { + int ret; + if ((acpi_disabled) || !acpi_cpc_valid()) return -ENODEV; INIT_LIST_HEAD(&cpu_data_list); cppc_check_hisi_workaround(); + cppc_freq_invariance_init(); - return cpufreq_register_driver(&cppc_cpufreq_driver); + ret = cpufreq_register_driver(&cppc_cpufreq_driver); + if (ret) + cppc_freq_invariance_exit(); + + return ret; } static inline void free_cpu_data(void) @@ -541,6 +761,7 @@ static inline void free_cpu_data(void) static void __exit cppc_cpufreq_exit(void) { cpufreq_unregister_driver(&cppc_cpufreq_driver); + cppc_freq_invariance_exit(); free_cpu_data(); } diff --git a/include/linux/arch_topology.h b/include/linux/arch_topology.h index 11e555cfaecb..f180240dc95f 100644 --- a/include/linux/arch_topology.h +++ b/include/linux/arch_topology.h @@ -37,6 +37,7 @@ bool topology_scale_freq_invariant(void); enum scale_freq_source { SCALE_FREQ_SOURCE_CPUFREQ = 0, SCALE_FREQ_SOURCE_ARCH, + SCALE_FREQ_SOURCE_CPPC, }; struct scale_freq_data { diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 4ca80df205ce..5226cc26a095 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6389,6 +6389,7 @@ int sched_setattr_nocheck(struct task_struct *p, const struct sched_attr *attr) { return __sched_setscheduler(p, attr, false, true); } +EXPORT_SYMBOL_GPL(sched_setattr_nocheck); /** * sched_setscheduler_nocheck - change the scheduling policy and/or RT priority of a thread from kernelspace.