diff mbox series

[v10,7/8] cgroup/cpuset: Update description of cpuset.cpus.partition in cgroup-v2.rst

Message ID 20220503162149.1764245-8-longman@redhat.com
State New
Headers show
Series cgroup/cpuset: Major cpu partition code restructuring | expand

Commit Message

Waiman Long May 3, 2022, 4:21 p.m. UTC 
Update Documentation/admin-guide/cgroup-v2.rst on the newly introduced
"isolated" cpuset partition type as well as other changes made in other
cpuset patches.

Signed-off-by: Waiman Long <longman@redhat.com>
---
 Documentation/admin-guide/cgroup-v2.rst | 145 +++++++++++++-----------
 1 file changed, 79 insertions(+), 66 deletions(-)

Comments

Michal Koutný May 4, 2022, 11:25 a.m. UTC | #1 
Hello.

On Tue, May 03, 2022 at 12:21:48PM -0400, Waiman Long <longman@redhat.com> wrote:
>  Documentation/admin-guide/cgroup-v2.rst | 145 +++++++++++++-----------
>  1 file changed, 79 insertions(+), 66 deletions(-)

A note across various lines -- it seems your new text accidentally mixes
both spaces and tabs for indentation.

> diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
> index 69d7a6983f78..94e1e3771830 100644
> --- a/Documentation/admin-guide/cgroup-v2.rst
> +++ b/Documentation/admin-guide/cgroup-v2.rst
> [...]
> +	The value shown in "cpuset.cpus.effective" of a partition root is
> +	the CPUs that the parent partition root can dedicate to the new
> +	partition root.  They are subtracted from "cpuset.cpus.effective"
> +	of the parent and may be different from "cpuset.cpus"

I find this paragraph a bit hard to comprehend (I read it as it talks
about three levels of cgroups (parent, child, grandparent). It is
correct but I'd suggect following formulation (where I additionally
simplifed it by talking about "available" cpus):

> The value shown in "cpuset.cpus.effective" of a partition root is
> the CPUs that the partition root can dedicate to a potential new child
> partition root. The new child subtracts available CPUs from its parent
> "cpuset.cpus.effective".


> +	For a partition root to become valid, the following conditions
> +	must be met.
> +
> +	1) The "cpuset.cpus" is exclusive, i.e. they are not shared by
> +	   any of its siblings (exclusivity rule).
> +	2) The parent cgroup is a valid partition root.
> +	3) The "cpuset.cpus" is not empty and must contain at least
> +	   one of the CPUs from parent's "cpuset.cpus", i.e. they overlap.
> +        4) The "cpuset.cpus.effective" must be a subset of "cpuset.cpus"
> +           and cannot be empty unless there is no task associated with
> +           this partition.

This sounds good to me.

> +        Care must be taken to change a valid partition root to "member"
> +        as all its child partitions, if present, will become invalid.

This does not talk about recovering. Is it intentional? (I.e. to left
implementation defined)

Except the remarks above, I find the concepts described here good. I'll
reply to implementation separately & later.

Regards,
Michal
Waiman Long May 4, 2022, 4:02 p.m. UTC | #2 
On 5/4/22 07:25, Michal Koutný wrote:
> Hello.
>
> On Tue, May 03, 2022 at 12:21:48PM -0400, Waiman Long <longman@redhat.com> wrote:
>>   Documentation/admin-guide/cgroup-v2.rst | 145 +++++++++++++-----------
>>   1 file changed, 79 insertions(+), 66 deletions(-)
> A note across various lines -- it seems your new text accidentally mixes
> both spaces and tabs for indentation.

You are right. I will fix that.

>
>> diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
>> index 69d7a6983f78..94e1e3771830 100644
>> --- a/Documentation/admin-guide/cgroup-v2.rst
>> +++ b/Documentation/admin-guide/cgroup-v2.rst
>> [...]
>> +	The value shown in "cpuset.cpus.effective" of a partition root is
>> +	the CPUs that the parent partition root can dedicate to the new
>> +	partition root.  They are subtracted from "cpuset.cpus.effective"
>> +	of the parent and may be different from "cpuset.cpus"
> I find this paragraph a bit hard to comprehend (I read it as it talks
> about three levels of cgroups (parent, child, grandparent). It is
> correct but I'd suggect following formulation (where I additionally
> simplifed it by talking about "available" cpus):
>
>> The value shown in "cpuset.cpus.effective" of a partition root is
>> the CPUs that the partition root can dedicate to a potential new child
>> partition root. The new child subtracts available CPUs from its parent
>> "cpuset.cpus.effective".


Thanks for the suggestion, will modify the text as suggested.


>
>> +	For a partition root to become valid, the following conditions
>> +	must be met.
>> +
>> +	1) The "cpuset.cpus" is exclusive, i.e. they are not shared by
>> +	   any of its siblings (exclusivity rule).
>> +	2) The parent cgroup is a valid partition root.
>> +	3) The "cpuset.cpus" is not empty and must contain at least
>> +	   one of the CPUs from parent's "cpuset.cpus", i.e. they overlap.
>> +        4) The "cpuset.cpus.effective" must be a subset of "cpuset.cpus"
>> +           and cannot be empty unless there is no task associated with
>> +           this partition.
> This sounds good to me.
>
>> +        Care must be taken to change a valid partition root to "member"
>> +        as all its child partitions, if present, will become invalid.
> This does not talk about recovering. Is it intentional? (I.e. to left
> implementation defined)

This new patch series does have the ability to recover now.  I am just 
not emphasizing the recovery aspect of it in the doc file. I will add a 
sentence about it.

>
> Except the remarks above, I find the concepts described here good. I'll
> reply to implementation separately & later.

Thanks,
Longman
diff mbox series

Patch

diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 69d7a6983f78..94e1e3771830 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -2110,74 +2110,87 @@  Cpuset Interface Files
 	It accepts only the following input values when written to.
 
 	  ========	================================
-	  "root"	a partition root
-	  "member"	a non-root member of a partition
+	  "member"	Non-root member of a partition
+	  "root"	Partition root
+	  "isolated"	Partition root without load balancing
 	  ========	================================
 
-	When set to be a partition root, the current cgroup is the
-	root of a new partition or scheduling domain that comprises
-	itself and all its descendants except those that are separate
-	partition roots themselves and their descendants.  The root
-	cgroup is always a partition root.
-
-	There are constraints on where a partition root can be set.
-	It can only be set in a cgroup if all the following conditions
-	are true.
-
-	1) The "cpuset.cpus" is not empty and the list of CPUs are
-	   exclusive, i.e. they are not shared by any of its siblings.
-	2) The parent cgroup is a partition root.
-	3) The "cpuset.cpus" is also a proper subset of the parent's
-	   "cpuset.cpus.effective".
-	4) There is no child cgroups with cpuset enabled.  This is for
-	   eliminating corner cases that have to be handled if such a
-	   condition is allowed.
-
-	Setting it to partition root will take the CPUs away from the
-	effective CPUs of the parent cgroup.  Once it is set, this
-	file cannot be reverted back to "member" if there are any child
-	cgroups with cpuset enabled.
-
-	A parent partition cannot distribute all its CPUs to its
-	child partitions.  There must be at least one cpu left in the
-	parent partition.
-
-	Once becoming a partition root, changes to "cpuset.cpus" is
-	generally allowed as long as the first condition above is true,
-	the change will not take away all the CPUs from the parent
-	partition and the new "cpuset.cpus" value is a superset of its
-	children's "cpuset.cpus" values.
-
-	Sometimes, external factors like changes to ancestors'
-	"cpuset.cpus" or cpu hotplug can cause the state of the partition
-	root to change.  On read, the "cpuset.sched.partition" file
-	can show the following values.
-
-	  ==============	==============================
-	  "member"		Non-root member of a partition
-	  "root"		Partition root
-	  "root invalid"	Invalid partition root
-	  ==============	==============================
-
-	It is a partition root if the first 2 partition root conditions
-	above are true and at least one CPU from "cpuset.cpus" is
-	granted by the parent cgroup.
-
-	A partition root can become invalid if none of CPUs requested
-	in "cpuset.cpus" can be granted by the parent cgroup or the
-	parent cgroup is no longer a partition root itself.  In this
-	case, it is not a real partition even though the restriction
-	of the first partition root condition above will still apply.
-	The cpu affinity of all the tasks in the cgroup will then be
-	associated with CPUs in the nearest ancestor partition.
-
-	An invalid partition root can be transitioned back to a
-	real partition root if at least one of the requested CPUs
-	can now be granted by its parent.  In this case, the cpu
-	affinity of all the tasks in the formerly invalid partition
-	will be associated to the CPUs of the newly formed partition.
-	Changing the partition state of an invalid partition root to
-	"member" is always allowed even if child cpusets are present.
+	The root cgroup is always a partition root and its state
+	cannot be changed.  All other non-root cgroups start out as
+	"member".
+
+	When set to "root", the current cgroup is the root of a new
+	partition or scheduling domain that comprises itself and all
+	its descendants except those that are separate partition roots
+	themselves and their descendants.
+
+	When set to "isolated", the CPUs in that partition root will
+	be in an isolated state without any load balancing from the
+	scheduler.  Tasks placed in such a partition with multiple
+	CPUs should be carefully distributed and bound to each of the
+	individual CPUs for optimal performance.
+
+	The value shown in "cpuset.cpus.effective" of a partition root is
+	the CPUs that the parent partition root can dedicate to the new
+	partition root.  They are subtracted from "cpuset.cpus.effective"
+	of the parent and may be different from "cpuset.cpus"
+
+	A partition root ("root" or "isolated") can be in one of the
+	two possible states - valid or invalid.  An invalid partition
+	root is in a degraded state where some state information may
+	be retained, but behaves more like a "member".
+
+	All possible state transitions among "member", "root" and
+	"isolated" are allowed.
+
+	On read, the "cpuset.cpus.partition" file can show the following
+	values.
+
+	  ======================	==============================
+	  "member"			Non-root member of a partition
+	  "root"			Partition root
+	  "isolated"			Partition root without load balancing
+	  "root invalid (<reason>)"	Invalid partition root
+	  "isolated invalid (<reason>)"	Invalid isolated partition root
+	  ======================	==============================
+
+	In the case of an invalid partition root, a descriptive string on
+	why the partition is invalid is included within parentheses.
+
+	For a partition root to become valid, the following conditions
+	must be met.
+
+	1) The "cpuset.cpus" is exclusive, i.e. they are not shared by
+	   any of its siblings (exclusivity rule).
+	2) The parent cgroup is a valid partition root.
+	3) The "cpuset.cpus" is not empty and must contain at least
+	   one of the CPUs from parent's "cpuset.cpus", i.e. they overlap.
+        4) The "cpuset.cpus.effective" must be a subset of "cpuset.cpus"
+           and cannot be empty unless there is no task associated with
+           this partition.
+
+	External events like hotplug or changes to "cpuset.cpus" can
+	cause a valid partition root to become invalid and vice versa.
+	Note that a task cannot be moved to a cgroup with empty
+	"cpuset.cpus.effective".
+
+        For a valid partition root or an invalid partition root with
+        the exclusivity rule enabled, changes made to "cpuset.cpus"
+        that violate the exclusivity rule will not be allowed.
+
+	A valid non-root parent partition may distribute out all its CPUs
+	to its child partitions when there is no task associated with it.
+
+        Care must be taken to change a valid partition root to "member"
+        as all its child partitions, if present, will become invalid.
+
+	Poll and inotify events are triggered whenever the state of
+	"cpuset.cpus.partition" changes.  That includes changes caused
+	by write to "cpuset.cpus.partition", cpu hotplug or other
+	changes that modify the validity status of the partition.
+	This will allow user space agents to monitor unexpected changes
+	to "cpuset.cpus.partition" without the need to do continuous
+	polling.
 
 
 Device controller