From patchwork Wed Jun 16 06:48:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 461378 Delivered-To: patch@linaro.org Received: by 2002:a02:735a:0:0:0:0:0 with SMTP id a26csp321440jae; Tue, 15 Jun 2021 23:48:22 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzZ2AtumkX2J5DT8Y+WZe4czbO/Y6NeJLyyxsbYMUviwCgX6Kun5VjKvh0M2TJmCYUuT7J5 X-Received: by 2002:a02:8241:: with SMTP id q1mr2779161jag.134.1623826102179; Tue, 15 Jun 2021 23:48:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1623826102; cv=none; d=google.com; s=arc-20160816; b=Fk535qFec/RSuHEMRLnANnC1ri5ZGAiMY0viUNwi5oUWyTwbJVTDpwYI75KXn1uXVr W5ZMkvk5vTfch5ncLYM62LI9t68FkdrsBOJgwONyZYXnGjDFp84QB/L9kRrRaktmfkvq TZGBHzM3SQnWfcjOnRo990QDb1tgltVT2sfwnv8WKQTXxdd4PFYFOzWQ4m333QZroPPI mp/Jxca/EOSXEpRNGBW/OkfzYvTxz9DY61/lg+zntK5wx057v8D1QxQknQHSU1o83YDQ Ov7A4O7jvh3kCkE+8Bi4ZB0nK9db3BeXxrnS+rR5d0v6/y/gyMNAEhsD+pEekhgmgLFo OrXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=G2o9MLJnJsw+6A1+/Voo6W0MsEPUjxURc0icFdbia4A=; b=gNonxO5mWZ2/LwmzAGkTIW80FVhnyOxpj8XmXhjHrv63PWlOZSFH/eqcDATIDyA4jv giLZ7E80QiLIUD1pl78OfD55Cl+yX3r0Tq/BhZEgkf+TqYUI/tFO5ryjg58gS9RjU6l+ vC/MFvY9ujIy7zEqF8kggGV9PGCkX+StcDrth0u7m55k6a8TjR2QNvt05KJv4DBiZVA8 drRcEIN2L/9DAxB2dLIY7bpzd5g+yQph+Pwe0Tq+79x2odVNcl0ihGOs36fEqaCEd0PE xcNn/NcDRmzYBz+aprIZjI8RERG9Mz5Ffi13VixOkfKOjk8nEePoeRymQMDUQHmpSg7U 6bxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=o3ZGRm+0; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id z21si1477145ioq.36.2021.06.15.23.48.22; Tue, 15 Jun 2021 23:48:22 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=o3ZGRm+0; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231589AbhFPGuZ (ORCPT + 8 others); Wed, 16 Jun 2021 02:50:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33096 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231548AbhFPGuY (ORCPT ); Wed, 16 Jun 2021 02:50:24 -0400 Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 714E5C061574 for ; Tue, 15 Jun 2021 23:48:18 -0700 (PDT) Received: by mail-pj1-x1034.google.com with SMTP id s17-20020a17090a8811b029016e89654f93so3380692pjn.1 for ; Tue, 15 Jun 2021 23:48:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=G2o9MLJnJsw+6A1+/Voo6W0MsEPUjxURc0icFdbia4A=; b=o3ZGRm+0PhR57FZeE69PXo3qtAMRcOMhbEPnUNPCmO6a3bopp2N37v+kmO19XhecJk d3fAwvpax0VF0gdRK99Gg4WeuW8tMXK7i/gICzuT6F+LczfGHLkIDT840edL0ZLQBuDq YCpzFBNxvRYhinwex9s/GmHopX+OQ7YH2HlsKvPIbmpp0ICcVQm/fzAo3DUcy3JkN0TV MwKlKRz8wqfI4Vv82WBDRXlMTO93AKiAYh5Bx2YPL/ValHsZQ9U7fINE4lqsxaIczaDo tcjkTMqK0LBuaT+RcZkqhbcpFfEFCbYlDiBOf2q1jktpKcLMXiGdXDUy0yezAqW15yO+ QgJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=G2o9MLJnJsw+6A1+/Voo6W0MsEPUjxURc0icFdbia4A=; b=qxujnsPwcVxl6OTV0iXAobblnbbr5nR4VoKkMKdHPkeW/gfYP4ZXuOT6+wb3V7j93H B0gjIQoI1RP3QTXKC2yqL6u6hgpMstlHYgFO5RVVIApoRqqN214Hw8iYz8yFGRVhL+4N KgYfW9huPw1zr2erS08XWcEYwRawZHAE4586jcQ7aMVJKB2RBe1bWiUjTfnsFpSMXifW NNFrZvKEWgMrxBgs83JGuk7zCdTFdr/ey9qFDMtgQ9MbHZl+/vrJ7ZAv8KgCN6NIJo4x rH366PDdPpB3WTrWivxkQ+cOJvzdvpjM+leminJ2py/+R7xQ1sbRwe4blXpUHQIqzvQb As1w== X-Gm-Message-State: AOAM532W/sIaPZBW1B5Fq4GI3qBbZxELxTWtSiw4Rx2Gzla6oHuFzQf2 A0BMcsIm5kmKdqxJxSVNjhU1Yw== X-Received: by 2002:a17:902:b609:b029:118:8a66:6963 with SMTP id b9-20020a170902b609b02901188a666963mr7854149pls.65.1623826097975; Tue, 15 Jun 2021 23:48:17 -0700 (PDT) Received: from localhost ([136.185.134.182]) by smtp.gmail.com with ESMTPSA id d8sm1127995pfq.198.2021.06.15.23.48.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Jun 2021 23:48:17 -0700 (PDT) From: Viresh Kumar To: Rafael Wysocki , Ionela Voinescu , Viresh Kumar , Jonathan Corbet Cc: linux-pm@vger.kernel.org, Vincent Guittot , Qian Cai , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 1/3] cpufreq: Add start_cpu() and stop_cpu() callbacks Date: Wed, 16 Jun 2021 12:18:07 +0530 Message-Id: <2ffbaf079a21c2810c402cb5bba4e9c14c4a0ff4.1623825725.git.viresh.kumar@linaro.org> X-Mailer: git-send-email 2.31.1.272.g89b43f80a514 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org On CPU hot-unplug, the cpufreq core doesn't call any driver specific callback unless all the CPUs of a policy went away, in which case we call stop_cpu() callback. For the CPPC cpufreq driver, we need to perform per-cpu init/exit work which that can't be performed from policy specific init()/exit() callbacks. This patch adds a new callback, start_cpu() and modifies the stop_cpu() callback, to perform such CPU specific work. These routines are called whenever a CPU is added or removed from a policy. Note that this also moves the setting of policy->cpus to online CPUs only, outside of rwsem as we needed to call start_cpu() for online CPUs only. This shouldn't have any side effects. Signed-off-by: Viresh Kumar --- Documentation/cpu-freq/cpu-drivers.rst | 7 +++++-- drivers/cpufreq/cpufreq.c | 19 +++++++++++++++---- include/linux/cpufreq.h | 5 ++++- 3 files changed, 24 insertions(+), 7 deletions(-) -- 2.31.1.272.g89b43f80a514 diff --git a/Documentation/cpu-freq/cpu-drivers.rst b/Documentation/cpu-freq/cpu-drivers.rst index a697278ce190..15cfe42b4075 100644 --- a/Documentation/cpu-freq/cpu-drivers.rst +++ b/Documentation/cpu-freq/cpu-drivers.rst @@ -71,8 +71,11 @@ And optionally .exit - A pointer to a per-policy cleanup function called during CPU_POST_DEAD phase of cpu hotplug process. - .stop_cpu - A pointer to a per-policy stop function called during - CPU_DOWN_PREPARE phase of cpu hotplug process. + .start_cpu - A pointer to a per-policy per-cpu start function called + during CPU online phase. + + .stop_cpu - A pointer to a per-policy per-cpu stop function called + during CPU offline phase. .suspend - A pointer to a per-policy suspend function which is called with interrupts disabled and _after_ the governor is stopped for the diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index 802abc925b2a..128dfb1b0cdf 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -1119,6 +1119,10 @@ static int cpufreq_add_policy_cpu(struct cpufreq_policy *policy, unsigned int cp cpumask_set_cpu(cpu, policy->cpus); + /* Do CPU specific initialization if required */ + if (cpufreq_driver->start_cpu) + cpufreq_driver->start_cpu(policy, cpu); + if (has_target()) { ret = cpufreq_start_governor(policy); if (ret) @@ -1375,13 +1379,19 @@ static int cpufreq_online(unsigned int cpu) cpumask_copy(policy->related_cpus, policy->cpus); } - down_write(&policy->rwsem); /* * affected cpus must always be the one, which are online. We aren't * managing offline cpus here. */ cpumask_and(policy->cpus, policy->cpus, cpu_online_mask); + /* Do CPU specific initialization if required */ + if (cpufreq_driver->start_cpu) { + for_each_cpu(j, policy->cpus) + cpufreq_driver->start_cpu(policy, j); + } + + down_write(&policy->rwsem); if (new_policy) { for_each_cpu(j, policy->related_cpus) { per_cpu(cpufreq_cpu_data, j) = policy; @@ -1581,6 +1591,10 @@ static int cpufreq_offline(unsigned int cpu) policy->cpu = cpumask_any(policy->cpus); } + /* Do CPU specific de-initialization if required */ + if (cpufreq_driver->stop_cpu) + cpufreq_driver->stop_cpu(policy, cpu); + /* Start governor again for active policy */ if (!policy_is_inactive(policy)) { if (has_target()) { @@ -1597,9 +1611,6 @@ static int cpufreq_offline(unsigned int cpu) policy->cdev = NULL; } - if (cpufreq_driver->stop_cpu) - cpufreq_driver->stop_cpu(policy); - if (has_target()) cpufreq_exit_governor(policy); diff --git a/include/linux/cpufreq.h b/include/linux/cpufreq.h index 353969c7acd3..c281b3df4e2f 100644 --- a/include/linux/cpufreq.h +++ b/include/linux/cpufreq.h @@ -371,7 +371,10 @@ struct cpufreq_driver { int (*online)(struct cpufreq_policy *policy); int (*offline)(struct cpufreq_policy *policy); int (*exit)(struct cpufreq_policy *policy); - void (*stop_cpu)(struct cpufreq_policy *policy); + + /* CPU specific start/stop */ + void (*start_cpu)(struct cpufreq_policy *policy, unsigned int cpu); + void (*stop_cpu)(struct cpufreq_policy *policy, unsigned int cpu); int (*suspend)(struct cpufreq_policy *policy); int (*resume)(struct cpufreq_policy *policy); From patchwork Wed Jun 16 06:48:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 461379 Delivered-To: patch@linaro.org Received: by 2002:a02:735a:0:0:0:0:0 with SMTP id a26csp321493jae; Tue, 15 Jun 2021 23:48:25 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwaMllv8jIvP0IFYXc9r302C1pkGZTEooHGT7pNedCAnm4U/IZvWV/PdtD1oWmWfCNtnosQ X-Received: by 2002:a92:5404:: with SMTP id i4mr2445250ilb.226.1623826105171; Tue, 15 Jun 2021 23:48:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1623826105; cv=none; d=google.com; s=arc-20160816; b=IW2HO2BN0WLiQkubQBFNA5x+ccpq2O35N2SIwzC/GeAJ2pul5HkBgzZ3/9Wix50hqf YCMsHmj1J8LTnBMpDDv+eNyvRMn2FF2ZpA7TZVlz9YRyw/jXymTYbD+H3EL0QWaOnIJ3 OgPR1HNX+XOK4qOdF9gugzbyUCB6RhDEns5Uz8mz37VS5itwJsvV4HmQWpcd4o4ohj5t q1OpZVBkQ9L8rwx0e13qYAvaDtDMegb822XfSvqvzMXW3azuAbixB8IAOMC4xwyHRtGf yLVjRo+qjgsjIGzLb0Bh+sRZhVhRwsy6aaHeKTKpfGZ/XUou8gCHzPWNBQGWMCxNHxZL J3qw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5RkESd0HK4MR77nDjYWPFbKW7m8spyK8zJmc8uVKNCY=; b=D+9tCEN3qISj4JQA+v/8Kl0XLrkJ+Yxqkz5uLeM+wAa3Auy4Gm7vh3PZN06zbyo6i3 6sfqjIErRiBUe31yJyBT9gtUkLEM7+J24ZERm0mtu4sejwpfDetQ0+M08szeDYoolRan GMJkAycXyCh2NzducNgNrI3ZIK+mhju3ujtAep8xMevfaCF4pdyAqgMxuHbxo85boSu+ qWhE5ePsYp7n6vPARegjBhMvAkpQF8o3hZO4mP/i/Z4G6CX7p5552DnIaHDfWazQg93Y TYFUtM2ahVeoJhr/JgShhKzxpgO9tKdw38Rj52bZVX/a+lov7njWQf/r1X4GAkZ9ZNve khiA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=qfJXKeo4; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id k7si1975943jad.43.2021.06.15.23.48.25; Tue, 15 Jun 2021 23:48:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=qfJXKeo4; spf=pass (google.com: domain of linux-pm-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231628AbhFPGu2 (ORCPT + 8 others); Wed, 16 Jun 2021 02:50:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33120 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231617AbhFPGu1 (ORCPT ); Wed, 16 Jun 2021 02:50:27 -0400 Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9BA60C061767 for ; Tue, 15 Jun 2021 23:48:21 -0700 (PDT) Received: by mail-pf1-x42d.google.com with SMTP id c12so1400917pfl.3 for ; Tue, 15 Jun 2021 23:48:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5RkESd0HK4MR77nDjYWPFbKW7m8spyK8zJmc8uVKNCY=; b=qfJXKeo4h64kbUFW1dLRqCxx1NM3VzPw5VPItHIh652oS+xP0v5eQrBk8wC0asRupc urD7ajq67fiNMNrNOlN1GjrYkMdyDTjZHOzfL1ePfvPW7HHsRVq6kC3yqGHk4A1JOeNg hpcjD5zn0aUprDFarQsQaL7ztHMKjXVqLPziBk//OG9fQsm1rCNlWtNJlkv8oYacUMeG uz+DArLn43Rz92Ak/Fo7+p0G5k+Te1J19SJkX3TnHA80JCoBGpx57wafMSo8jUjyKm+z QFgjAlsK52MSIfX5nkVwR6D6JHdboS/xB5+O6LFmU9+v4SNzjy2QgF2pvBqZRWPDQQ7x y2uQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5RkESd0HK4MR77nDjYWPFbKW7m8spyK8zJmc8uVKNCY=; b=LMYMA5fI+Qiz87dxRgMcJZtJUTS16gPvXBdXnoOHbL4zwdnqNitxTBhcNoa2yYdsRy VYVmSu4ubJKjvIlsyzHmKNkAgYaf89Ru9eyCMO7M59L4EB1PpuXdK2Jv6XOK2PAfbuSi qOgXs2JHSYRl2/nlArHmZYyoFxScoGdHksKUYT5JrDhiC0yJfSdkH0dhbbLFqygAXhG7 4mndh3ccvURrpu1JuS8OVuMKgcw7RpuEsY3EmUE51CyBVr5KrdFceUg+0PMIeR95XToY Li4owve5BaUFuEg2w8rqNEbDaZY7xsooWodvq8Ae6zSTDmBoEKzMKco6mHLwL2f+VkC/ Vldw== X-Gm-Message-State: AOAM530U8xuneizca22he8y9zB9ylfP8KqLhepYX8pMRPjiYNnb9dWiW Itb0tk9MO2/QRB42Il1eyADY/Q== X-Received: by 2002:a65:6642:: with SMTP id z2mr3507720pgv.388.1623826101221; Tue, 15 Jun 2021 23:48:21 -0700 (PDT) Received: from localhost ([136.185.134.182]) by smtp.gmail.com with ESMTPSA id p6sm4434629pjk.34.2021.06.15.23.48.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Jun 2021 23:48:20 -0700 (PDT) From: Viresh Kumar To: Rafael Wysocki , Ionela Voinescu , Sudeep Holla , Greg Kroah-Hartman , "Rafael J. Wysocki" Cc: Viresh Kumar , linux-pm@vger.kernel.org, Vincent Guittot , Qian Cai , "Paul E . McKenney" , linux-kernel@vger.kernel.org Subject: [PATCH V2 2/3] arch_topology: Avoid use-after-free for scale_freq_data Date: Wed, 16 Jun 2021 12:18:08 +0530 Message-Id: <9dba462b4d09a1a8a9fbb75740b74bf91a09a3e1.1623825725.git.viresh.kumar@linaro.org> X-Mailer: git-send-email 2.31.1.272.g89b43f80a514 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Currently topology_scale_freq_tick() may end up using a pointer to struct scale_freq_data, which was previously cleared by topology_clear_scale_freq_source(), as there is no protection in place here. The users of topology_clear_scale_freq_source() though needs a guarantee that the previous scale_freq_data isn't used anymore. Since topology_scale_freq_tick() is called from scheduler tick, we don't want to add locking in there. Use the RCU update mechanism instead (which is already used by the scheduler's utilization update path) to guarantee race free updates here. Cc: Paul E. McKenney Signed-off-by: Viresh Kumar --- drivers/base/arch_topology.c | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) -- 2.31.1.272.g89b43f80a514 diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c index c1179edc0f3b..921312a8d957 100644 --- a/drivers/base/arch_topology.c +++ b/drivers/base/arch_topology.c @@ -18,10 +18,11 @@ #include #include #include +#include #include #include -static DEFINE_PER_CPU(struct scale_freq_data *, sft_data); +static DEFINE_PER_CPU(struct scale_freq_data __rcu *, sft_data); static struct cpumask scale_freq_counters_mask; static bool scale_freq_invariant; @@ -66,16 +67,20 @@ void topology_set_scale_freq_source(struct scale_freq_data *data, if (cpumask_empty(&scale_freq_counters_mask)) scale_freq_invariant = topology_scale_freq_invariant(); + rcu_read_lock(); + for_each_cpu(cpu, cpus) { - sfd = per_cpu(sft_data, cpu); + sfd = rcu_dereference(*per_cpu_ptr(&sft_data, cpu)); /* Use ARCH provided counters whenever possible */ if (!sfd || sfd->source != SCALE_FREQ_SOURCE_ARCH) { - per_cpu(sft_data, cpu) = data; + rcu_assign_pointer(per_cpu(sft_data, cpu), data); cpumask_set_cpu(cpu, &scale_freq_counters_mask); } } + rcu_read_unlock(); + update_scale_freq_invariant(true); } EXPORT_SYMBOL_GPL(topology_set_scale_freq_source); @@ -86,22 +91,32 @@ void topology_clear_scale_freq_source(enum scale_freq_source source, struct scale_freq_data *sfd; int cpu; + rcu_read_lock(); + for_each_cpu(cpu, cpus) { - sfd = per_cpu(sft_data, cpu); + sfd = rcu_dereference(*per_cpu_ptr(&sft_data, cpu)); if (sfd && sfd->source == source) { - per_cpu(sft_data, cpu) = NULL; + rcu_assign_pointer(per_cpu(sft_data, cpu), NULL); cpumask_clear_cpu(cpu, &scale_freq_counters_mask); } } + rcu_read_unlock(); + + /* + * Make sure all references to previous sft_data are dropped to avoid + * use-after-free races. + */ + synchronize_rcu(); + update_scale_freq_invariant(false); } EXPORT_SYMBOL_GPL(topology_clear_scale_freq_source); void topology_scale_freq_tick(void) { - struct scale_freq_data *sfd = *this_cpu_ptr(&sft_data); + struct scale_freq_data *sfd = rcu_dereference_sched(*this_cpu_ptr(&sft_data)); if (sfd) sfd->set_freq_scale(); From patchwork Wed Jun 16 06:48:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 462591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85C77C48BE8 for ; Wed, 16 Jun 2021 06:48:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6F857611CE for ; Wed, 16 Jun 2021 06:48:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231744AbhFPGue (ORCPT ); Wed, 16 Jun 2021 02:50:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33146 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231713AbhFPGub (ORCPT ); Wed, 16 Jun 2021 02:50:31 -0400 Received: from mail-pf1-x42f.google.com (mail-pf1-x42f.google.com [IPv6:2607:f8b0:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0FA39C0617A6 for ; Tue, 15 Jun 2021 23:48:25 -0700 (PDT) Received: by mail-pf1-x42f.google.com with SMTP id u18so1366740pfk.11 for ; Tue, 15 Jun 2021 23:48:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wHWamenmOHlNmoXLuZe9DO++QMOAk9UZVcjrT6HRY+4=; b=EdvUml/9a05FhcQpLX9ydI3EUXlz01ykjhPBZ2zCA+1baNB1swTocErpbKAZJe7MjR Y5LRLcoCWr7tOyZh8mgiFJlTlW1ymyZm2DVNqLTHOIgOIcmlxvjV3up2w4sHzowOaChJ 65TUeeaXW6QyKvM/Xmeq9hzgyW638pAmKf1JKZ32hY7XTT7yWel65Vww6oq6rIYFVr/r GXWKooueFwYupxYgkkzvFuJ01oYzlo/ctnXVckbnfSc7zD47amf9tp9830YJSWBuddK9 ZCgWHf1FMZB2f2cBvqcOGmTZNcAKfpjxtHyMUgQ/wwX8YHsblUub3d0NrC7wm4XdQZOH FDNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wHWamenmOHlNmoXLuZe9DO++QMOAk9UZVcjrT6HRY+4=; b=bY2vG3CO/C2US1i+JhsQ96jqGfhyp3Vj7tsLRMLdHzV43rVp8CDAzPT6GL9FcCpFbX uv8UE17/ABz9qcSvuQdeUd0FB8NDcSYFdXoLF5yK/Zhtc18M0Jlmj0QsqMYDMFU+uTwR +joMmANs+emIlIknC70i+Cy8gCHXvqFHRM9wyg2LwEVcefwILDAX6QIas5JUP+7DFDeP h/VQ11mSY9F4uNd8O0CG+IboukRQ78caS6L95T34U0TxMhGnd4mGp7fAq1AySNa7j384 qygGP5HgW98rXU2gpaPYAB9X6YNmD5QYD4c9yrl5NIFpWNe98EZA5R0cd8c0yiepVZcn cToA== X-Gm-Message-State: AOAM5309LbUYjz4asomtf9VvW9Tdd0E/Rl4dJjpouYWYeDoqEhvQ2MqS 7CYC27My7v0iJnFgrESGMsVQzQ== X-Google-Smtp-Source: ABdhPJxLDGSfqj9vCMeL5YWhlJPxptsn+YM5ydskdhxt3tKHMtWqat1WJRzV3wzECvggtBtizwpr7A== X-Received: by 2002:a62:1d0e:0:b029:2d8:30a3:687f with SMTP id d14-20020a621d0e0000b02902d830a3687fmr7996691pfd.17.1623826104569; Tue, 15 Jun 2021 23:48:24 -0700 (PDT) Received: from localhost ([136.185.134.182]) by smtp.gmail.com with ESMTPSA id ig1sm1176056pjb.27.2021.06.15.23.48.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Jun 2021 23:48:24 -0700 (PDT) From: Viresh Kumar To: Rafael Wysocki , Ionela Voinescu , Viresh Kumar , Sudeep Holla , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira Cc: linux-pm@vger.kernel.org, Qian Cai , linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 3/3] cpufreq: CPPC: Add support for frequency invariance Date: Wed, 16 Jun 2021 12:18:09 +0530 Message-Id: X-Mailer: git-send-email 2.31.1.272.g89b43f80a514 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org The Frequency Invariance Engine (FIE) is providing a frequency scaling correction factor that helps achieve more accurate load-tracking. Normally, this scaling factor can be obtained directly with the help of the cpufreq drivers as they know the exact frequency the hardware is running at. But that isn't the case for CPPC cpufreq driver. Another way of obtaining that is using the arch specific counter support, which is already present in kernel, but that hardware is optional for platforms. This patch updates the CPPC driver to register itself with the topology core to provide its own implementation (cppc_scale_freq_tick()) of topology_scale_freq_tick() which gets called by the scheduler on every tick. Note that the arch specific counters have higher priority than CPPC counters, if available, though the CPPC driver doesn't need to have any special handling for that. On an invocation of cppc_scale_freq_tick(), we schedule an irq work (since we reach here from hard-irq context), which then schedules a normal work item and cppc_scale_freq_workfn() updates the per_cpu arch_freq_scale variable based on the counter updates since the last tick. To allow platforms to disable this CPPC counter-based frequency invariance support, this is all done under CONFIG_ACPI_CPPC_CPUFREQ_FIE, which is enabled by default. This also exports sched_setattr_nocheck() as the CPPC driver can be built as a module. Cc: linux-acpi@vger.kernel.org Signed-off-by: Viresh Kumar --- drivers/cpufreq/Kconfig.arm | 10 ++ drivers/cpufreq/cppc_cpufreq.c | 232 ++++++++++++++++++++++++++++++--- include/linux/arch_topology.h | 1 + kernel/sched/core.c | 1 + 4 files changed, 227 insertions(+), 17 deletions(-) diff --git a/drivers/cpufreq/Kconfig.arm b/drivers/cpufreq/Kconfig.arm index e65e0a43be64..a5c5f70acfc9 100644 --- a/drivers/cpufreq/Kconfig.arm +++ b/drivers/cpufreq/Kconfig.arm @@ -19,6 +19,16 @@ config ACPI_CPPC_CPUFREQ If in doubt, say N. +config ACPI_CPPC_CPUFREQ_FIE + bool "Frequency Invariance support for CPPC cpufreq driver" + depends on ACPI_CPPC_CPUFREQ && GENERIC_ARCH_TOPOLOGY + default y + help + This extends frequency invariance support in the CPPC cpufreq driver, + by using CPPC delivered and reference performance counters. + + If in doubt, say N. + config ARM_ALLWINNER_SUN50I_CPUFREQ_NVMEM tristate "Allwinner nvmem based SUN50I CPUFreq driver" depends on ARCH_SUNXI diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index be4f62e2c5f1..63e4cff46f95 100644 --- a/drivers/cpufreq/cppc_cpufreq.c +++ b/drivers/cpufreq/cppc_cpufreq.c @@ -10,14 +10,18 @@ #define pr_fmt(fmt) "CPPC Cpufreq:" fmt +#include #include #include #include #include #include #include +#include +#include #include #include +#include #include @@ -57,6 +61,183 @@ static struct cppc_workaround_oem_info wa_info[] = { } }; +#ifdef CONFIG_ACPI_CPPC_CPUFREQ_FIE + +/* Frequency invariance support */ +struct cppc_freq_invariance { + int cpu; + struct irq_work irq_work; + struct kthread_work work; + struct cppc_perf_fb_ctrs prev_perf_fb_ctrs; + struct cppc_cpudata *cpu_data; +}; + +static DEFINE_PER_CPU(struct cppc_freq_invariance, cppc_freq_inv); +static struct kthread_worker *kworker_fie; + +static struct cpufreq_driver cppc_cpufreq_driver; +static unsigned int hisi_cppc_cpufreq_get_rate(unsigned int cpu); +static int cppc_perf_from_fbctrs(struct cppc_cpudata *cpu_data, + struct cppc_perf_fb_ctrs *fb_ctrs_t0, + struct cppc_perf_fb_ctrs *fb_ctrs_t1); + +/** + * cppc_scale_freq_workfn - CPPC arch_freq_scale updater for frequency invariance + * @work: The work item. + * + * The CPPC driver register itself with the topology core to provide its own + * implementation (cppc_scale_freq_tick()) of topology_scale_freq_tick() which + * gets called by the scheduler on every tick. + * + * Note that the arch specific counters have higher priority than CPPC counters, + * if available, though the CPPC driver doesn't need to have any special + * handling for that. + * + * On an invocation of cppc_scale_freq_tick(), we schedule an irq work (since we + * reach here from hard-irq context), which then schedules a normal work item + * and cppc_scale_freq_workfn() updates the per_cpu arch_freq_scale variable + * based on the counter updates since the last tick. + */ +static void cppc_scale_freq_workfn(struct kthread_work *work) +{ + struct cppc_freq_invariance *cppc_fi; + struct cppc_perf_fb_ctrs fb_ctrs = {0}; + struct cppc_cpudata *cpu_data; + unsigned long local_freq_scale; + u64 perf; + + cppc_fi = container_of(work, struct cppc_freq_invariance, work); + cpu_data = cppc_fi->cpu_data; + + if (cppc_get_perf_ctrs(cppc_fi->cpu, &fb_ctrs)) { + pr_warn("%s: failed to read perf counters\n", __func__); + return; + } + + perf = cppc_perf_from_fbctrs(cpu_data, &cppc_fi->prev_perf_fb_ctrs, + &fb_ctrs); + cppc_fi->prev_perf_fb_ctrs = fb_ctrs; + + perf <<= SCHED_CAPACITY_SHIFT; + local_freq_scale = div64_u64(perf, cpu_data->perf_caps.highest_perf); + if (WARN_ON(local_freq_scale > 1024)) + local_freq_scale = 1024; + + per_cpu(arch_freq_scale, cppc_fi->cpu) = local_freq_scale; +} + +static void cppc_irq_work(struct irq_work *irq_work) +{ + struct cppc_freq_invariance *cppc_fi; + + cppc_fi = container_of(irq_work, struct cppc_freq_invariance, irq_work); + kthread_queue_work(kworker_fie, &cppc_fi->work); +} + +static void cppc_scale_freq_tick(void) +{ + struct cppc_freq_invariance *cppc_fi = &per_cpu(cppc_freq_inv, smp_processor_id()); + + /* + * cppc_get_perf_ctrs() can potentially sleep, call that from the right + * context. + */ + irq_work_queue(&cppc_fi->irq_work); +} + +static struct scale_freq_data cppc_sftd = { + .source = SCALE_FREQ_SOURCE_CPPC, + .set_freq_scale = cppc_scale_freq_tick, +}; + +static void cppc_cpufreq_start_cpu(struct cpufreq_policy *policy, + unsigned int cpu) +{ + struct cppc_freq_invariance *cppc_fi = &per_cpu(cppc_freq_inv, cpu); + int ret; + + cppc_fi->cpu = cpu; + cppc_fi->cpu_data = policy->driver_data; + kthread_init_work(&cppc_fi->work, cppc_scale_freq_workfn); + init_irq_work(&cppc_fi->irq_work, cppc_irq_work); + + ret = cppc_get_perf_ctrs(cpu, &cppc_fi->prev_perf_fb_ctrs); + if (ret) { + pr_warn("%s: failed to read perf counters: %d\n", __func__, + ret); + return; + } + + /* Register for freq-invariance */ + topology_set_scale_freq_source(&cppc_sftd, cpumask_of(cpu)); +} + +static void cppc_cpufreq_stop_cpu(struct cpufreq_policy *policy, + unsigned int cpu) +{ + struct cppc_freq_invariance *cppc_fi = &per_cpu(cppc_freq_inv, cpu); + + topology_clear_scale_freq_source(SCALE_FREQ_SOURCE_CPPC, cpumask_of(cpu)); + + irq_work_sync(&cppc_fi->irq_work); + kthread_cancel_work_sync(&cppc_fi->work); +} + +static void __init cppc_freq_invariance_init(void) +{ + struct sched_attr attr = { + .size = sizeof(struct sched_attr), + .sched_policy = SCHED_DEADLINE, + .sched_nice = 0, + .sched_priority = 0, + /* + * Fake (unused) bandwidth; workaround to "fix" + * priority inheritance. + */ + .sched_runtime = 1000000, + .sched_deadline = 10000000, + .sched_period = 10000000, + }; + int ret; + + if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate) + return; + + kworker_fie = kthread_create_worker(0, "cppc_fie"); + if (IS_ERR(kworker_fie)) + return; + + ret = sched_setattr_nocheck(kworker_fie->task, &attr); + if (ret) { + pr_warn("%s: failed to set SCHED_DEADLINE: %d\n", __func__, + ret); + kthread_destroy_worker(kworker_fie); + return; + } + + cppc_cpufreq_driver.start_cpu = cppc_cpufreq_start_cpu; + cppc_cpufreq_driver.stop_cpu = cppc_cpufreq_stop_cpu; +} + +static void cppc_freq_invariance_exit(void) +{ + if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate) + return; + + kthread_destroy_worker(kworker_fie); + kworker_fie = NULL; +} + +#else +static inline void cppc_freq_invariance_init(void) +{ +} + +static inline void cppc_freq_invariance_exit(void) +{ +} +#endif /* CONFIG_ACPI_CPPC_CPUFREQ_FIE */ + /* Callback function used to retrieve the max frequency from DMI */ static void cppc_find_dmi_mhz(const struct dmi_header *dm, void *private) { @@ -362,26 +543,35 @@ static inline u64 get_delta(u64 t1, u64 t0) return (u32)t1 - (u32)t0; } -static int cppc_get_rate_from_fbctrs(struct cppc_cpudata *cpu_data, - struct cppc_perf_fb_ctrs fb_ctrs_t0, - struct cppc_perf_fb_ctrs fb_ctrs_t1) +static int cppc_perf_from_fbctrs(struct cppc_cpudata *cpu_data, + struct cppc_perf_fb_ctrs *fb_ctrs_t0, + struct cppc_perf_fb_ctrs *fb_ctrs_t1) { u64 delta_reference, delta_delivered; - u64 reference_perf, delivered_perf; + u64 reference_perf; - reference_perf = fb_ctrs_t0.reference_perf; + reference_perf = fb_ctrs_t0->reference_perf; - delta_reference = get_delta(fb_ctrs_t1.reference, - fb_ctrs_t0.reference); - delta_delivered = get_delta(fb_ctrs_t1.delivered, - fb_ctrs_t0.delivered); + delta_reference = get_delta(fb_ctrs_t1->reference, + fb_ctrs_t0->reference); + delta_delivered = get_delta(fb_ctrs_t1->delivered, + fb_ctrs_t0->delivered); - /* Check to avoid divide-by zero */ - if (delta_reference || delta_delivered) - delivered_perf = (reference_perf * delta_delivered) / - delta_reference; - else - delivered_perf = cpu_data->perf_ctrls.desired_perf; + /* Check to avoid divide-by zero and invalid delivered_perf */ + if (!delta_reference || !delta_delivered) + return cpu_data->perf_ctrls.desired_perf; + + return (reference_perf * delta_delivered) / delta_reference; +} + +static int cppc_get_rate_from_fbctrs(struct cppc_cpudata *cpu_data, + struct cppc_perf_fb_ctrs *fb_ctrs_t0, + struct cppc_perf_fb_ctrs *fb_ctrs_t1) +{ + u64 delivered_perf; + + delivered_perf = cppc_perf_from_fbctrs(cpu_data, fb_ctrs_t0, + fb_ctrs_t1); return cppc_cpufreq_perf_to_khz(cpu_data, delivered_perf); } @@ -405,7 +595,7 @@ static unsigned int cppc_cpufreq_get_rate(unsigned int cpu) if (ret) return ret; - return cppc_get_rate_from_fbctrs(cpu_data, fb_ctrs_t0, fb_ctrs_t1); + return cppc_get_rate_from_fbctrs(cpu_data, &fb_ctrs_t0, &fb_ctrs_t1); } static int cppc_cpufreq_set_boost(struct cpufreq_policy *policy, int state) @@ -506,14 +696,21 @@ static void cppc_check_hisi_workaround(void) static int __init cppc_cpufreq_init(void) { + int ret; + if ((acpi_disabled) || !acpi_cpc_valid()) return -ENODEV; INIT_LIST_HEAD(&cpu_data_list); cppc_check_hisi_workaround(); + cppc_freq_invariance_init(); - return cpufreq_register_driver(&cppc_cpufreq_driver); + ret = cpufreq_register_driver(&cppc_cpufreq_driver); + if (ret) + cppc_freq_invariance_exit(); + + return ret; } static inline void free_cpu_data(void) @@ -531,6 +728,7 @@ static inline void free_cpu_data(void) static void __exit cppc_cpufreq_exit(void) { cpufreq_unregister_driver(&cppc_cpufreq_driver); + cppc_freq_invariance_exit(); free_cpu_data(); } diff --git a/include/linux/arch_topology.h b/include/linux/arch_topology.h index 11e555cfaecb..f180240dc95f 100644 --- a/include/linux/arch_topology.h +++ b/include/linux/arch_topology.h @@ -37,6 +37,7 @@ bool topology_scale_freq_invariant(void); enum scale_freq_source { SCALE_FREQ_SOURCE_CPUFREQ = 0, SCALE_FREQ_SOURCE_ARCH, + SCALE_FREQ_SOURCE_CPPC, }; struct scale_freq_data { diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 4ca80df205ce..5226cc26a095 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6389,6 +6389,7 @@ int sched_setattr_nocheck(struct task_struct *p, const struct sched_attr *attr) { return __sched_setscheduler(p, attr, false, true); } +EXPORT_SYMBOL_GPL(sched_setattr_nocheck); /** * sched_setscheduler_nocheck - change the scheduling policy and/or RT priority of a thread from kernelspace.