| Message ID | 20250424-isolcpus-io-queues-v6-6-9a53a870ca1f@kernel.org |
|---|---|
| State | New |
| Headers | show |
| Series | blk: honor isolcpus configuration | expand |
On Fri, Apr 25, 2025 at 09:32:16AM +0200, Daniel Wagner wrote: > On Fri, Apr 25, 2025 at 08:26:22AM +0200, Hannes Reinecke wrote: > > On 4/24/25 20:19, Daniel Wagner wrote: > > > Multiqueue drivers spreading IO queues on all CPUs for optimal > > > performance. The drivers are not aware of the CPU isolated requirement > > > and will spread all queues ignoring the isolcpus configuration. > > > > > > Introduce a new isolcpus mask which allows the user to define on which > > > CPUs IO queues should be placed. This is similar to the managed_irq but > > > for drivers which do not use the managed IRQ infrastructure. > > > > > > Signed-off-by: Daniel Wagner <wagi@kernel.org> > > > --- > > > include/linux/sched/isolation.h | 1 + > > > kernel/sched/isolation.c | 7 +++++++ > > > 2 files changed, 8 insertions(+) > > > > > Reviewed-by: Hannes Reinecke <hare@suse.de> > > Just realized I forgot to also add some document on this new argument: > > io_queue > Isolate from IO queue work caused by multiqueue > device drivers. Restrict the placement of > queues to housekeeping CPUs only, ensuring that > all IO work is processed by a housekeeping CPU. > > Note: When an isolated CPU issues an IO, it is > forwarded to a housekeeping CPU. This will > trigger a software interrupt on the completion > path. > > Note: It is not possible to offline housekeeping > CPUs that serve isolated CPUs. This patch adds kernel parameter only, but not apply it at all, the above words just confuses everyone, so I'd suggest to not expose the kernel command line & document until the whole mechanism is supported. Especially 'irqaffinity=0 isolcpus=io_queue' requires the application to offline CPU in order, which has to be documented: https://lore.kernel.org/all/cc5e44dd-e1dc-4f24-88d9-ce45a8b0794f@flourine.local/ Thanks, Ming
On Fri, May 09, 2025 at 10:04:18AM +0800, Ming Lei wrote: > > io_queue > > Isolate from IO queue work caused by multiqueue > > device drivers. Restrict the placement of > > queues to housekeeping CPUs only, ensuring that > > all IO work is processed by a housekeeping CPU. > > > > Note: When an isolated CPU issues an IO, it is > > forwarded to a housekeeping CPU. This will > > trigger a software interrupt on the completion > > path. > > > > Note: It is not possible to offline housekeeping > > CPUs that serve isolated CPUs. > > This patch adds kernel parameter only, but not apply it at all, the above > words just confuses everyone, so I'd suggest to not expose the kernel > command line & document until the whole mechanism is supported. I'll add this doc update as last patch. > Especially 'irqaffinity=0 isolcpus=io_queue' requires the application > to offline CPU in order, which has to be documented: > > https://lore.kernel.org/all/cc5e44dd-e1dc-4f24-88d9-ce45a8b0794f@flourine.local/ Okay, so you want me to extend the above second note in this case. I'll give it a go.
diff --git a/include/linux/sched/isolation.h b/include/linux/sched/isolation.h index d8501f4709b583b8a1c91574446382f093bccdb1..6b6ae9c5b2f61a93c649a98ea27482b932627fca 100644 --- a/include/linux/sched/isolation.h +++ b/include/linux/sched/isolation.h @@ -9,6 +9,7 @@ enum hk_type { HK_TYPE_DOMAIN, HK_TYPE_MANAGED_IRQ, + HK_TYPE_IO_QUEUE, HK_TYPE_KERNEL_NOISE, HK_TYPE_MAX, diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c index 81bc8b329ef17cd3a3f5ae0a20ca02af3a1a69bc..687b11f900e31ab656e25cae263f15f6d8f46a9a 100644 --- a/kernel/sched/isolation.c +++ b/kernel/sched/isolation.c @@ -11,6 +11,7 @@ enum hk_flags { HK_FLAG_DOMAIN = BIT(HK_TYPE_DOMAIN), HK_FLAG_MANAGED_IRQ = BIT(HK_TYPE_MANAGED_IRQ), + HK_FLAG_IO_QUEUE = BIT(HK_TYPE_IO_QUEUE), HK_FLAG_KERNEL_NOISE = BIT(HK_TYPE_KERNEL_NOISE), }; @@ -224,6 +225,12 @@ static int __init housekeeping_isolcpus_setup(char *str) continue; } + if (!strncmp(str, "io_queue,", 9)) { + str += 9; + flags |= HK_FLAG_IO_QUEUE; + continue; + } + /* * Skip unknown sub-parameter and validate that it is not * containing an invalid character.
Multiqueue drivers spreading IO queues on all CPUs for optimal performance. The drivers are not aware of the CPU isolated requirement and will spread all queues ignoring the isolcpus configuration. Introduce a new isolcpus mask which allows the user to define on which CPUs IO queues should be placed. This is similar to the managed_irq but for drivers which do not use the managed IRQ infrastructure. Signed-off-by: Daniel Wagner <wagi@kernel.org> --- include/linux/sched/isolation.h | 1 + kernel/sched/isolation.c | 7 +++++++ 2 files changed, 8 insertions(+)