From patchwork Tue May 14 17:55:59 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 16904 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-we0-f199.google.com (mail-we0-f199.google.com [74.125.82.199]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id DEAB425D00 for ; Tue, 14 May 2013 17:56:43 +0000 (UTC) Received: by mail-we0-f199.google.com with SMTP id p57sf738923wes.2 for ; Tue, 14 May 2013 10:56:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:x-beenthere:x-received:received-spf:x-received :x-forwarded-to:x-forwarded-for:delivered-to:x-received:received-spf :x-received:sender:date:from:to:cc:subject:message-id:references :mime-version:in-reply-to:user-agent:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-google-group-id:list-post:list-help:list-archive:list-unsubscribe :content-type:content-disposition; bh=ZAQ69n9A3//SReqqCxEVjxHrL7y4J0kNqfyN/q+PQRI=; b=I6C68ovsuR6mqpxSWpf4HYV5JXwJeo+BL5CiY9VdZhHcU9l6yWUV1XHmhp0k7kfR1+ 4tln/gPutEFk4QZOgFbtNiUMXGDnqyA5DRZ64wWZQ+ZlDWTNDBPi6VfwhBFNljTSKKaN m5Ofh8n5qTxXk2H3avFwR9lNC80CjofYH8QJh2c9xngzUFgzHS9/x/qMgfn2kYC9kv0d 4t4Ess2rb79VoOSMmkn8nWEpXrPDrHjHZU8J6V9uPR/dKq5ZdFrP0Wfw0b8e062M/ObE HbWwAq78kHc075n9HILQnoUjTpQSm5fjpF+z3RPpEcivfkS2+cfGFDA/prAJPHXlNKQU mi5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:x-beenthere:x-received:received-spf:x-received :x-forwarded-to:x-forwarded-for:delivered-to:x-received:received-spf :x-received:sender:date:from:to:cc:subject:message-id:references :mime-version:in-reply-to:user-agent:x-gm-message-state :x-original-sender:x-original-authentication-results:precedence :mailing-list:list-id:x-google-group-id:list-post:list-help :list-archive:list-unsubscribe:content-type:content-disposition; bh=ZAQ69n9A3//SReqqCxEVjxHrL7y4J0kNqfyN/q+PQRI=; b=eg1dMBigJ9KofC2p4ztGq9yAmGcEzPlB9d61Qfyl1cHhjxalSNak0nSwbsjRK8j+Ii +TiU5nvVLsSVFTuCUS/me4bwz53QMnpVHlVqHiduOsIbKCykjW/aiI7kAZcHyj7Ylkr3 apbgDbxeS44Sg/+ajY8PJljbLog6TWk23TBRJeoSBh8z301RPZu8LIYkvUqCuVZsPj6r oxbuSxn1t939pHZJiOqQpmJ8Yzcuk4u5fUbENFP08P9vXszgLD3hGZPFvwIFT7lLvcrK doqkLlX5gSHxHXNzDB6QP1DAzilLI9lhASrvdB13HtdImZmXhS4eHJXGBn5yj98GuJoU Tr7Q== X-Received: by 10.180.20.105 with SMTP id m9mr1258065wie.5.1368554168084; Tue, 14 May 2013 10:56:08 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.180.185.205 with SMTP id fe13ls287433wic.16.gmail; Tue, 14 May 2013 10:56:08 -0700 (PDT) X-Received: by 10.180.183.210 with SMTP id eo18mr8819235wic.17.1368554167974; Tue, 14 May 2013 10:56:07 -0700 (PDT) Received: from mail-ve0-x22c.google.com (mail-ve0-x22c.google.com [2607:f8b0:400c:c01::22c]) by mx.google.com with ESMTPS id b12si3701972wik.89.2013.05.14.10.56.07 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 14 May 2013 10:56:07 -0700 (PDT) Received-SPF: neutral (google.com: 2607:f8b0:400c:c01::22c is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=2607:f8b0:400c:c01::22c; Received: by mail-ve0-f172.google.com with SMTP id b10so997160vea.17 for ; Tue, 14 May 2013 10:56:06 -0700 (PDT) X-Received: by 10.52.175.200 with SMTP id cc8mr19184676vdc.94.1368554166782; Tue, 14 May 2013 10:56:06 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patches@linaro.org Received: by 10.220.217.15 with SMTP id hk15csp73219vcb; Tue, 14 May 2013 10:56:06 -0700 (PDT) X-Received: by 10.236.2.164 with SMTP id 24mr18282190yhf.27.1368554165851; Tue, 14 May 2013 10:56:05 -0700 (PDT) Received: from mail-ye0-f180.google.com (mail-ye0-f180.google.com [209.85.213.180]) by mx.google.com with ESMTPS id x23si1455532yhx.239.2013.05.14.10.56.05 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 14 May 2013 10:56:05 -0700 (PDT) Received-SPF: pass (google.com: domain of htejun@gmail.com designates 209.85.213.180 as permitted sender) client-ip=209.85.213.180; Received: by mail-ye0-f180.google.com with SMTP id r11so144835yen.39 for ; Tue, 14 May 2013 10:56:05 -0700 (PDT) X-Received: by 10.236.13.39 with SMTP id a27mr18132874yha.41.1368554165399; Tue, 14 May 2013 10:56:05 -0700 (PDT) Received: from mtj.dyndns.org (nat-pool-3-rdu.redhat.com. [66.187.233.203]) by mx.google.com with ESMTPSA id e64sm27946968yhm.4.2013.05.14.10.56.01 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 14 May 2013 10:56:04 -0700 (PDT) Sender: Tejun Heo Date: Tue, 14 May 2013 10:55:59 -0700 From: Tejun Heo To: Viresh Kumar Cc: Amit Kucheria , davem@davemloft.net, linux-rt-users@vger.kernel.org, linux-kernel@vger.kernel.org, Robin Randhawa , Charles Garcia-Tobin , Steve Bannister , Peter Zijlstra , Steven Rostedt , Arvind Chauhan , Patch Tracking , airlied@redhat.com, mingo@redhat.com, Jens Axboe , Liviu Dudau , Lists linaro-kernel Subject: Re: [PATCH V5 1/5] workqueues: Introduce new flag WQ_POWER_EFFICIENT for power oriented workqueues Message-ID: <20130514175559.GH6795@mtj.dyndns.org> References: <480f5d6da920e2f2b868f311228c81635727799e.1366803121.git.viresh.kumar@linaro.org> <20130424161208.GA16991@mtj.dyndns.org> <20130426191118.GD1433@mtj.dyndns.org> MIME-Version: 1.0 In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Gm-Message-State: ALoCoQktXChsD7u2njOJsDsoepLI/IdUQ4BPVWYvwyAILlphgIOghUc+d5vcg+LjZEH8zuSqyb8r X-Original-Sender: tj@kernel.org X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 2607:f8b0:400c:c01::22c is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org; dkim=pass header.i=@gmail.com Precedence: list Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org List-ID: X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , Content-Disposition: inline >From cee22a15052faa817e3ec8985a28154d3fabc7aa Mon Sep 17 00:00:00 2001 From: Viresh Kumar Date: Mon, 8 Apr 2013 16:45:40 +0530 Subject: workqueues: Introduce new flag WQ_POWER_EFFICIENT for power oriented workqueues Workqueues can be performance or power-oriented. Currently, most workqueues are bound to the CPU they were created on. This gives good performance (due to cache effects) at the cost of potentially waking up otherwise idle cores (Idle from scheduler's perspective. Which may or may not be physically idle) just to process some work. To save power, we can allow the work to be rescheduled on a core that is already awake. Workqueues created with the WQ_UNBOUND flag will allow some power savings. However, we don't change the default behaviour of the system. To enable power-saving behaviour, a new config option CONFIG_WQ_POWER_EFFICIENT needs to be turned on. This option can also be overridden by the workqueue.power_efficient boot parameter. tj: Updated config description and comments. Renamed CONFIG_WQ_POWER_EFFICIENT to CONFIG_WQ_POWER_EFFICIENT_DEFAULT. Signed-off-by: Viresh Kumar Reviewed-by: Amit Kucheria Signed-off-by: Tejun Heo --- Documentation/kernel-parameters.txt | 15 +++++++++++++++ include/linux/workqueue.h | 27 +++++++++++++++++++++++++++ kernel/power/Kconfig | 20 ++++++++++++++++++++ kernel/workqueue.c | 13 +++++++++++++ 4 files changed, 75 insertions(+) diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt index c3bfacb..37dfd72 100644 --- a/Documentation/kernel-parameters.txt +++ b/Documentation/kernel-parameters.txt @@ -3320,6 +3320,21 @@ bytes respectively. Such letter suffixes can also be entirely omitted. that this also can be controlled per-workqueue for workqueues visible under /sys/bus/workqueue/. + workqueue.power_efficient + Per-cpu workqueues are generally preferred because + they show better performance thanks to cache + locality; unfortunately, per-cpu workqueues tend to + be more power hungry than unbound workqueues. + + Enabling this makes the per-cpu workqueues which + were observed to contribute significantly to power + consumption unbound, leading to measurably lower + power usage at the cost of small performance + overhead. + + The default value of this parameter is determined by + the config option CONFIG_WQ_POWER_EFFICIENT_DEFAULT. + x2apic_phys [X86-64,APIC] Use x2apic physical mode instead of default x2apic cluster mode on platforms supporting x2apic. diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h index 623488f..fc0136b 100644 --- a/include/linux/workqueue.h +++ b/include/linux/workqueue.h @@ -303,6 +303,33 @@ enum { WQ_CPU_INTENSIVE = 1 << 5, /* cpu instensive workqueue */ WQ_SYSFS = 1 << 6, /* visible in sysfs, see wq_sysfs_register() */ + /* + * Per-cpu workqueues are generally preferred because they tend to + * show better performance thanks to cache locality. Per-cpu + * workqueues exclude the scheduler from choosing the CPU to + * execute the worker threads, which has an unfortunate side effect + * of increasing power consumption. + * + * The scheduler considers a CPU idle if it doesn't have any task + * to execute and tries to keep idle cores idle to conserve power; + * however, for example, a per-cpu work item scheduled from an + * interrupt handler on an idle CPU will force the scheduler to + * excute the work item on that CPU breaking the idleness, which in + * turn may lead to more scheduling choices which are sub-optimal + * in terms of power consumption. + * + * Workqueues marked with WQ_POWER_EFFICIENT are per-cpu by default + * but become unbound if workqueue.power_efficient kernel param is + * specified. Per-cpu workqueues which are identified to + * contribute significantly to power-consumption are identified and + * marked with this flag and enabling the power_efficient mode + * leads to noticeable power saving at the cost of small + * performance disadvantage. + * + * http://thread.gmane.org/gmane.linux.kernel/1480396 + */ + WQ_POWER_EFFICIENT = 1 << 7, + __WQ_DRAINING = 1 << 16, /* internal: workqueue is draining */ __WQ_ORDERED = 1 << 17, /* internal: workqueue is ordered */ diff --git a/kernel/power/Kconfig b/kernel/power/Kconfig index 5dfdc9e..4645596 100644 --- a/kernel/power/Kconfig +++ b/kernel/power/Kconfig @@ -263,6 +263,26 @@ config PM_GENERIC_DOMAINS bool depends on PM +config WQ_POWER_EFFICIENT_DEFAULT + bool "Enable workqueue power-efficient mode by default" + depends on PM + default n + help + Per-cpu workqueues are generally preferred because they show + better performance thanks to cache locality; unfortunately, + per-cpu workqueues tend to be more power hungry than unbound + workqueues. + + Enabling workqueue.power_efficient kernel parameter makes the + per-cpu workqueues which were observed to contribute + significantly to power consumption unbound, leading to measurably + lower power usage at the cost of small performance overhead. + + This config option determines whether workqueue.power_efficient + is enabled by default. + + If in doubt, say N. + config PM_GENERIC_DOMAINS_SLEEP def_bool y depends on PM_SLEEP && PM_GENERIC_DOMAINS diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 4aa9f5b..8068d97 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -272,6 +272,15 @@ static cpumask_var_t *wq_numa_possible_cpumask; static bool wq_disable_numa; module_param_named(disable_numa, wq_disable_numa, bool, 0444); +/* see the comment above the definition of WQ_POWER_EFFICIENT */ +#ifdef CONFIG_WQ_POWER_EFFICIENT_DEFAULT +static bool wq_power_efficient = true; +#else +static bool wq_power_efficient; +#endif + +module_param_named(power_efficient, wq_power_efficient, bool, 0444); + static bool wq_numa_enabled; /* unbound NUMA affinity enabled */ /* buf for wq_update_unbound_numa_attrs(), protected by CPU hotplug exclusion */ @@ -4085,6 +4094,10 @@ struct workqueue_struct *__alloc_workqueue_key(const char *fmt, struct workqueue_struct *wq; struct pool_workqueue *pwq; + /* see the comment above the definition of WQ_POWER_EFFICIENT */ + if ((flags & WQ_POWER_EFFICIENT) && wq_power_efficient) + flags |= WQ_UNBOUND; + /* allocate wq and format name */ if (flags & WQ_UNBOUND) tbl_size = wq_numa_tbl_len * sizeof(wq->numa_pwq_tbl[0]);