| Message ID | 20220713075014.411739-1-juri.lelli@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | sched/deadline: Fix BUG_ON condition for deboosted tasks | expand |
On Wed, Jul 13, 2022 at 09:50:14AM +0200, Juri Lelli wrote: > Tasks the are being deboosted from SCHED_DEADLINE might enter > enqueue_task_dl() one last time and hit an erroneous BUG_ON condition: > since they are not boosted anymore, the if (is_dl_boosted()) branch is > not taken, but the else if (!dl_prio) is and inside this one we > BUG_ON(!is_dl_boosted), which is of course false (BUG_ON triggered) > otherwise we had entered the if branch above. Long story short, the > current condition doesn't make sense and always leads to triggering of a > BUG. > > Fix this by only checking enqueue flags, properly: ENQUEUE_REPLENISH has > to be present, but additional flags are not a problem. > > Fixes: 2279f540ea7d ("sched/deadline: Fix priority inheritance with multiple scheduling classes") > Signed-off-by: Juri Lelli <juri.lelli@redhat.com> > --- > kernel/sched/deadline.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c > index 5867e186c39a..0447d46f4718 100644 > --- a/kernel/sched/deadline.c > +++ b/kernel/sched/deadline.c > @@ -1703,7 +1703,7 @@ static void enqueue_task_dl(struct rq *rq, struct task_struct *p, int flags) > * the throttle. > */ > p->dl.dl_throttled = 0; > - BUG_ON(!is_dl_boosted(&p->dl) || flags != ENQUEUE_REPLENISH); > + BUG_ON(!(flags & ENQUEUE_REPLENISH)); While there, can we perhaps make it less fatal?
Hi Juri, On 7/13/22 12:50 AM, Juri Lelli wrote: > Tasks the are being deboosted from SCHED_DEADLINE might enter > enqueue_task_dl() one last time and hit an erroneous BUG_ON condition: > since they are not boosted anymore, the if (is_dl_boosted()) branch is > not taken, but the else if (!dl_prio) is and inside this one we > BUG_ON(!is_dl_boosted), which is of course false (BUG_ON triggered) > otherwise we had entered the if branch above. Long story short, the > current condition doesn't make sense and always leads to triggering of a > BUG. > > Fix this by only checking enqueue flags, properly: ENQUEUE_REPLENISH has > to be present, but additional flags are not a problem. > > Fixes: 2279f540ea7d ("sched/deadline: Fix priority inheritance with multiple scheduling classes") It looks like this problem goes further back than the above commit (which was merged in v5.10). Even the oldest LTS kernel (4.9) has code like this: if (... && p->dl.dl_boosted && ...)) { /* code */ } else if (!dl_prio(p->normal_prio)) { BUG_ON(!p->dl.dl_boosted || flags != ENQUEUE_REPLENISH); return; } And we have observed crashes in the 4.19 kernel series too (CC'ed Ankit Jain and Him Kalyan who have reproduced this issue). I believe commit 64be6f1f5f71 ("sched/deadline: Don't replenish from a !SCHED_DEADLINE entity") introduced the problem, which dates back to v3.18. Would you mind updating the Fixes: tag and adding a CC: stable tag as well, when you respin the patch, please? Thank you! > Signed-off-by: Juri Lelli <juri.lelli@redhat.com> > --- > kernel/sched/deadline.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c > index 5867e186c39a..0447d46f4718 100644 > --- a/kernel/sched/deadline.c > +++ b/kernel/sched/deadline.c > @@ -1703,7 +1703,7 @@ static void enqueue_task_dl(struct rq *rq, struct task_struct *p, int flags) > * the throttle. > */ > p->dl.dl_throttled = 0; > - BUG_ON(!is_dl_boosted(&p->dl) || flags != ENQUEUE_REPLENISH); > + BUG_ON(!(flags & ENQUEUE_REPLENISH)); > return; > } > > Regards, Srivatsa VMware Photon OS
Hi, On 13/07/22 14:31, Srivatsa S. Bhat wrote: > > Hi Juri, > > On 7/13/22 12:50 AM, Juri Lelli wrote: > > Tasks the are being deboosted from SCHED_DEADLINE might enter > > enqueue_task_dl() one last time and hit an erroneous BUG_ON condition: > > since they are not boosted anymore, the if (is_dl_boosted()) branch is > > not taken, but the else if (!dl_prio) is and inside this one we > > BUG_ON(!is_dl_boosted), which is of course false (BUG_ON triggered) > > otherwise we had entered the if branch above. Long story short, the > > current condition doesn't make sense and always leads to triggering of a > > BUG. > > > > Fix this by only checking enqueue flags, properly: ENQUEUE_REPLENISH has > > to be present, but additional flags are not a problem. > > > > Fixes: 2279f540ea7d ("sched/deadline: Fix priority inheritance with multiple scheduling classes") > > It looks like this problem goes further back than the above commit > (which was merged in v5.10). > > Even the oldest LTS kernel (4.9) has code like this: > > if (... && p->dl.dl_boosted && ...)) { > /* code */ > > } else if (!dl_prio(p->normal_prio)) { > > BUG_ON(!p->dl.dl_boosted || flags != ENQUEUE_REPLENISH); > return; > } > > And we have observed crashes in the 4.19 kernel series too (CC'ed > Ankit Jain and Him Kalyan who have reproduced this issue). > > I believe commit 64be6f1f5f71 ("sched/deadline: Don't replenish from a > !SCHED_DEADLINE entity") introduced the problem, which dates back to > v3.18. > > Would you mind updating the Fixes: tag and adding a CC: stable tag as > well, when you respin the patch, please? I think you are right. Will do. Thanks for taking a look! Best, Juri
[Resending my previous email in plaintext] > On 14-Jul-2022, at 12:58 PM, Juri Lelli <juri.lelli@redhat.com> wrote: > > ⚠ External Email > > Hi, > > On 13/07/22 14:31, Srivatsa S. Bhat wrote: >> >> Hi Juri, >> >> On 7/13/22 12:50 AM, Juri Lelli wrote: >>> Tasks the are being deboosted from SCHED_DEADLINE might enter >>> enqueue_task_dl() one last time and hit an erroneous BUG_ON condition: >>> since they are not boosted anymore, the if (is_dl_boosted()) branch is >>> not taken, but the else if (!dl_prio) is and inside this one we >>> BUG_ON(!is_dl_boosted), which is of course false (BUG_ON triggered) >>> otherwise we had entered the if branch above. Long story short, the >>> current condition doesn't make sense and always leads to triggering of a >>> BUG. >>> >>> Fix this by only checking enqueue flags, properly: ENQUEUE_REPLENISH has >>> to be present, but additional flags are not a problem. >>> >>> Fixes: 2279f540ea7d ("sched/deadline: Fix priority inheritance with multiple scheduling classes") >> >> It looks like this problem goes further back than the above commit >> (which was merged in v5.10). >> >> Even the oldest LTS kernel (4.9) has code like this: >> >> if (... && p->dl.dl_boosted && ...)) { >> /* code */ >> >> } else if (!dl_prio(p->normal_prio)) { >> >> BUG_ON(!p->dl.dl_boosted || flags != ENQUEUE_REPLENISH); >> return; >> } >> >> And we have observed crashes in the 4.19 kernel series too (CC'ed >> Ankit Jain and Him Kalyan who have reproduced this issue). >> >> I believe commit 64be6f1f5f71 ("sched/deadline: Don't replenish from a >> !SCHED_DEADLINE entity") introduced the problem, which dates back to >> v3.18. >> >> Would you mind updating the Fixes: tag and adding a CC: stable tag as >> well, when you respin the patch, please? > > I think you are right. Will do. > > Thanks for taking a look! > > Best, > Juri > Hi Juri, I tried the patch but it still hit the BUG_ON. [ 163.094094] ------------[ cut here ]------------ [ 163.094095] kernel BUG at kernel/sched/deadline.c:1525! [ 163.094103] invalid opcode: 0000 [#1] PREEMPT SMP PTI [ 163.094105] CPU: 0 PID: 5494 Comm: stalld/34 Not tainted 4.19.247-rt108-10.ph3-rt #1-photon [ 163.094107] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018 [ 163.094113] RIP: 0010:enqueue_task_dl+0x35d/0x9d0 [ 163.094115] Code: 83 c4 10 5b 41 5c 41 5d 41 5e 41 5f 5d c3 8b 56 74 85 d2 0f 88 91 fd ff ff 80 a6 0c 03 00 00 fe 41 83 e5 20 0f 85 [ 163.094116] RSP: 0018:ffff9b9286537e40 EFLAGS: 00010046 [ 163.094118] RAX: ffffffff840bded0 RBX: ffff8dda07c48000 RCX: 0000000000002000 [ 163.094119] RDX: 0000000000000078 RSI: ffff8dda07c48000 RDI: ffff8dddb79a87c0 [ 163.094120] RBP: ffff9b9286537e78 R08: 0000000000000000 R09: 000000000000007f [ 163.094121] R10: ffff9b9286537e68 R11: 0000000000000000 R12: ffff9b9286537ef0 [ 163.094121] R13: 0000000000000000 R14: ffff8dddb79a87c0 R15: ffff8dda07c482b8 [ 163.094123] FS: 00007f81a27e4700(0000) GS:ffff8ddbb7600000(0000) knlGS:0000000000000000 [ 163.094124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 163.094125] CR2: 00007f85e6cfc4c0 CR3: 0000000233744004 CR4: 00000000007606b0 [ 163.094176] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 163.094177] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 163.094177] PKRU: 55555554 [ 163.094178] Call Trace: [ 163.094183] ? dequeue_task_dl+0x38/0x1d0 [ 163.094188] __sched_setscheduler+0x2e2/0x8e0 [ 163.094191] __x64_sys_sched_setattr+0x74/0xb0 [ 163.094194] do_syscall_64+0x60/0x1b0 [ 163.094200] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 163.094201] RIP: 0033:0x7f81b6ffe319 [ 163.094202] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c [ 163.094203] RSP: 002b:00007f81a27e3e28 EFLAGS: 00000206 ORIG_RAX: 000000000000013a [ 163.094205] RAX: ffffffffffffffda RBX: 000000000000150a RCX: 00007f81b6ffe319 [ 163.094205] RDX: 0000000000000000 RSI: 00007f81a27e3e50 RDI: 000000000000150a [ 163.094206] RBP: 000000000000150a R08: 0000000000000000 R09: 0000000000000030 [ 163.094207] R10: 00007f80f0002090 R11: 0000000000000206 R12: 0000556306de7a20 [ 163.094208] R13: 0000000000000002 R14: 0000556306ba3570 R15: 00007f81a27e4700 [ 163.094210] Modules linked in: ipt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter bridge stp [ 163.099512] ---[ end trace 0000000000000002 ]--- In enqueue_task_dl(): } else if (!dl_prio(p->normal_prio)) { … BUG_ON(!(flags & ENQUEUE_REPLENISH)); return; } I observe flags value as (ENQUEUE_RESTORE | ENQUEUE_NOCLOCK) Thanks, Ankit
Hi, On 15/07/22 04:49, Ankit Jain wrote: [...] > Hi Juri, > > I tried the patch but it still hit the BUG_ON. > > [ 163.094094] ------------[ cut here ]------------ > [ 163.094095] kernel BUG at kernel/sched/deadline.c:1525! > [ 163.094103] invalid opcode: 0000 [#1] PREEMPT SMP PTI > [ 163.094105] CPU: 0 PID: 5494 Comm: stalld/34 Not tainted 4.19.247-rt108-10.ph3-rt #1-photon > [ 163.094107] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018 > [ 163.094113] RIP: 0010:enqueue_task_dl+0x35d/0x9d0 > [ 163.094115] Code: 83 c4 10 5b 41 5c 41 5d 41 5e 41 5f 5d c3 8b 56 74 85 d2 0f 88 91 fd ff ff 80 a6 0c 03 00 00 fe 41 83 e5 20 0f 85 > [ 163.094116] RSP: 0018:ffff9b9286537e40 EFLAGS: 00010046 > [ 163.094118] RAX: ffffffff840bded0 RBX: ffff8dda07c48000 RCX: 0000000000002000 > [ 163.094119] RDX: 0000000000000078 RSI: ffff8dda07c48000 RDI: ffff8dddb79a87c0 > [ 163.094120] RBP: ffff9b9286537e78 R08: 0000000000000000 R09: 000000000000007f > [ 163.094121] R10: ffff9b9286537e68 R11: 0000000000000000 R12: ffff9b9286537ef0 > [ 163.094121] R13: 0000000000000000 R14: ffff8dddb79a87c0 R15: ffff8dda07c482b8 > [ 163.094123] FS: 00007f81a27e4700(0000) GS:ffff8ddbb7600000(0000) knlGS:0000000000000000 > [ 163.094124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 163.094125] CR2: 00007f85e6cfc4c0 CR3: 0000000233744004 CR4: 00000000007606b0 > [ 163.094176] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 163.094177] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > [ 163.094177] PKRU: 55555554 > [ 163.094178] Call Trace: > [ 163.094183] ? dequeue_task_dl+0x38/0x1d0 > [ 163.094188] __sched_setscheduler+0x2e2/0x8e0 > [ 163.094191] __x64_sys_sched_setattr+0x74/0xb0 > [ 163.094194] do_syscall_64+0x60/0x1b0 > [ 163.094200] entry_SYSCALL_64_after_hwframe+0x44/0xa9 > [ 163.094201] RIP: 0033:0x7f81b6ffe319 > [ 163.094202] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c > [ 163.094203] RSP: 002b:00007f81a27e3e28 EFLAGS: 00000206 ORIG_RAX: 000000000000013a > [ 163.094205] RAX: ffffffffffffffda RBX: 000000000000150a RCX: 00007f81b6ffe319 > [ 163.094205] RDX: 0000000000000000 RSI: 00007f81a27e3e50 RDI: 000000000000150a > [ 163.094206] RBP: 000000000000150a R08: 0000000000000000 R09: 0000000000000030 > [ 163.094207] R10: 00007f80f0002090 R11: 0000000000000206 R12: 0000556306de7a20 > [ 163.094208] R13: 0000000000000002 R14: 0000556306ba3570 R15: 00007f81a27e4700 > [ 163.094210] Modules linked in: ipt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter bridge stp > [ 163.099512] ---[ end trace 0000000000000002 ]--- > > > In enqueue_task_dl(): > > } else if (!dl_prio(p->normal_prio)) { > … > BUG_ON(!(flags & ENQUEUE_REPLENISH)); > return; > } > > I observe flags value as (ENQUEUE_RESTORE | ENQUEUE_NOCLOCK) Thanks for testing! However, it looks like 4.19-rt is at least missing commit 46fcc4b00c3cc ("sched/deadline: Fix stale throttling on de-/boosted tasks") and commit 2279f540ea7d0 ("sched/deadline: Fix priority inheritance with multiple scheduling classes") that might be playing also a role here. Best, Juri
> On 15-Jul-2022, at 1:17 PM, Juri Lelli <juri.lelli@redhat.com> wrote: > > ⚠ External Email > > Hi, > > On 15/07/22 04:49, Ankit Jain wrote: > > [...] > >> Hi Juri, >> >> I tried the patch but it still hit the BUG_ON. >> >> [ 163.094094] ------------[ cut here ]------------ >> [ 163.094095] kernel BUG at kernel/sched/deadline.c:1525! >> [ 163.094103] invalid opcode: 0000 [#1] PREEMPT SMP PTI >> [ 163.094105] CPU: 0 PID: 5494 Comm: stalld/34 Not tainted 4.19.247-rt108-10.ph3-rt #1-photon >> [ 163.094107] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018 >> [ 163.094113] RIP: 0010:enqueue_task_dl+0x35d/0x9d0 >> [ 163.094115] Code: 83 c4 10 5b 41 5c 41 5d 41 5e 41 5f 5d c3 8b 56 74 85 d2 0f 88 91 fd ff ff 80 a6 0c 03 00 00 fe 41 83 e5 20 0f 85 >> [ 163.094116] RSP: 0018:ffff9b9286537e40 EFLAGS: 00010046 >> [ 163.094118] RAX: ffffffff840bded0 RBX: ffff8dda07c48000 RCX: 0000000000002000 >> [ 163.094119] RDX: 0000000000000078 RSI: ffff8dda07c48000 RDI: ffff8dddb79a87c0 >> [ 163.094120] RBP: ffff9b9286537e78 R08: 0000000000000000 R09: 000000000000007f >> [ 163.094121] R10: ffff9b9286537e68 R11: 0000000000000000 R12: ffff9b9286537ef0 >> [ 163.094121] R13: 0000000000000000 R14: ffff8dddb79a87c0 R15: ffff8dda07c482b8 >> [ 163.094123] FS: 00007f81a27e4700(0000) GS:ffff8ddbb7600000(0000) knlGS:0000000000000000 >> [ 163.094124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 163.094125] CR2: 00007f85e6cfc4c0 CR3: 0000000233744004 CR4: 00000000007606b0 >> [ 163.094176] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> [ 163.094177] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 >> [ 163.094177] PKRU: 55555554 >> [ 163.094178] Call Trace: >> [ 163.094183] ? dequeue_task_dl+0x38/0x1d0 >> [ 163.094188] __sched_setscheduler+0x2e2/0x8e0 >> [ 163.094191] __x64_sys_sched_setattr+0x74/0xb0 >> [ 163.094194] do_syscall_64+0x60/0x1b0 >> [ 163.094200] entry_SYSCALL_64_after_hwframe+0x44/0xa9 >> [ 163.094201] RIP: 0033:0x7f81b6ffe319 >> [ 163.094202] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c >> [ 163.094203] RSP: 002b:00007f81a27e3e28 EFLAGS: 00000206 ORIG_RAX: 000000000000013a >> [ 163.094205] RAX: ffffffffffffffda RBX: 000000000000150a RCX: 00007f81b6ffe319 >> [ 163.094205] RDX: 0000000000000000 RSI: 00007f81a27e3e50 RDI: 000000000000150a >> [ 163.094206] RBP: 000000000000150a R08: 0000000000000000 R09: 0000000000000030 >> [ 163.094207] R10: 00007f80f0002090 R11: 0000000000000206 R12: 0000556306de7a20 >> [ 163.094208] R13: 0000000000000002 R14: 0000556306ba3570 R15: 00007f81a27e4700 >> [ 163.094210] Modules linked in: ipt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter bridge stp >> [ 163.099512] ---[ end trace 0000000000000002 ]--- >> >> >> In enqueue_task_dl(): >> >> } else if (!dl_prio(p->normal_prio)) { >> … >> BUG_ON(!(flags & ENQUEUE_REPLENISH)); >> return; >> } >> >> I observe flags value as (ENQUEUE_RESTORE | ENQUEUE_NOCLOCK) > > Thanks for testing! > > However, it looks like 4.19-rt is at least missing commit 46fcc4b00c3cc > ("sched/deadline: Fix stale throttling on de-/boosted tasks") and commit > 2279f540ea7d0 ("sched/deadline: Fix priority inheritance with multiple > scheduling classes") that might be playing also a role here. > > Best, > Juri > Hi Juri, Actually, while testing I already included below commits in 4.19-rt : feff2e65efd8d84cf831668e182b2ce73c604bbb (sched/deadline: Unthrottle PI boosted threads while enqueuing) 46fcc4b00c3cca8adb9b7c9afdd499f64e427135 (sched/deadline: Fix stale throttling on de-/boosted tasks) 2279f540ea7d05f22d2f0c4224319330228586bc (sched/deadline: Fix priority inheritance with multiple) 0e3872499de1a1230cef5221607d71aa09264bd5 (kernel/sched: Remove dl_boosted flag comment) Thanks, Ankit
On 18/07/22 07:46, Ankit Jain wrote: ... > Hi Juri, > > Actually, while testing I already included below commits in 4.19-rt : > > feff2e65efd8d84cf831668e182b2ce73c604bbb (sched/deadline: Unthrottle PI boosted threads while > enqueuing) > 46fcc4b00c3cca8adb9b7c9afdd499f64e427135 (sched/deadline: Fix stale throttling on de-/boosted tasks) > 2279f540ea7d05f22d2f0c4224319330228586bc (sched/deadline: Fix priority inheritance with multiple) > 0e3872499de1a1230cef5221607d71aa09264bd5 (kernel/sched: Remove dl_boosted flag comment) Interesting. Is the workload you are using to test this easily reproducible? I'd like to try that out on my end to check if I see the same (of course the issue I was working on goes away with my fix :). Best, Juri
> On 18-Jul-2022, at 6:31 PM, Juri Lelli <juri.lelli@redhat.com> wrote: > > ⚠ External Email > > On 18/07/22 07:46, Ankit Jain wrote: > > ... > >> Hi Juri, >> >> Actually, while testing I already included below commits in 4.19-rt : >> >> feff2e65efd8d84cf831668e182b2ce73c604bbb (sched/deadline: Unthrottle PI boosted threads while >> enqueuing) >> 46fcc4b00c3cca8adb9b7c9afdd499f64e427135 (sched/deadline: Fix stale throttling on de-/boosted tasks) >> 2279f540ea7d05f22d2f0c4224319330228586bc (sched/deadline: Fix priority inheritance with multiple) >> 0e3872499de1a1230cef5221607d71aa09264bd5 (kernel/sched: Remove dl_boosted flag comment) > > Interesting. > > Is the workload you are using to test this easily reproducible? I'd like > to try that out on my end to check if I see the same (of course the > issue I was working on goes away with my fix :). > > Best, > Juri > Hi Juri, The test with which i am able to hit the issue is as follows: • Schedule SCHED_FIFO/55 (sched_priority = 55) tasks running infinite loop on all isolated cores. • spwan 30-40 docker containers in loop (docker load , docker run) • Immediately after that schedule SCHED_FIFO/55 (sched_priority = 55) tasks running infinite loop on all isolated cores again. • BUG_ON gets hit almost every time. System config as follows: • 4.19-rt kernel • 40 cpu (0-1 housekeeping, 2-39 isol cpus) • stalld-1.3.0 with the fixes from latest version (for task starvation avoidance), "tuned" with real-time profile Thanks.
On 19/07/22 05:30, Ankit Jain wrote: > > > > On 18-Jul-2022, at 6:31 PM, Juri Lelli <juri.lelli@redhat.com> wrote: > > > > ⚠ External Email > > > > On 18/07/22 07:46, Ankit Jain wrote: > > > > ... > > > >> Hi Juri, > >> > >> Actually, while testing I already included below commits in 4.19-rt : > >> > >> feff2e65efd8d84cf831668e182b2ce73c604bbb (sched/deadline: Unthrottle PI boosted threads while > >> enqueuing) > >> 46fcc4b00c3cca8adb9b7c9afdd499f64e427135 (sched/deadline: Fix stale throttling on de-/boosted tasks) > >> 2279f540ea7d05f22d2f0c4224319330228586bc (sched/deadline: Fix priority inheritance with multiple) > >> 0e3872499de1a1230cef5221607d71aa09264bd5 (kernel/sched: Remove dl_boosted flag comment) > > > > Interesting. > > > > Is the workload you are using to test this easily reproducible? I'd like > > to try that out on my end to check if I see the same (of course the > > issue I was working on goes away with my fix :). > > > > Best, > > Juri > > > > Hi Juri, > > The test with which i am able to hit the issue is as follows: > • Schedule SCHED_FIFO/55 (sched_priority = 55) tasks running infinite loop on all isolated cores. > • spwan 30-40 docker containers in loop (docker load , docker run) > • Immediately after that schedule SCHED_FIFO/55 (sched_priority = 55) tasks running infinite loop on all isolated cores again. > • BUG_ON gets hit almost every time. > System config as follows: > • 4.19-rt kernel > • 40 cpu (0-1 housekeeping, 2-39 isol cpus) > • stalld-1.3.0 with the fixes from latest version (for task starvation avoidance), "tuned" with real-time profile Thanks for the details. Yeah, I should be able to set this up on my end. It might just take a bit though, as I have some pto planned coming up. But, I'll get to it eventually. Thanks again, Juri
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 5867e186c39a..0447d46f4718 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -1703,7 +1703,7 @@ static void enqueue_task_dl(struct rq *rq, struct task_struct *p, int flags) * the throttle. */ p->dl.dl_throttled = 0; - BUG_ON(!is_dl_boosted(&p->dl) || flags != ENQUEUE_REPLENISH); + BUG_ON(!(flags & ENQUEUE_REPLENISH)); return; }
Tasks the are being deboosted from SCHED_DEADLINE might enter enqueue_task_dl() one last time and hit an erroneous BUG_ON condition: since they are not boosted anymore, the if (is_dl_boosted()) branch is not taken, but the else if (!dl_prio) is and inside this one we BUG_ON(!is_dl_boosted), which is of course false (BUG_ON triggered) otherwise we had entered the if branch above. Long story short, the current condition doesn't make sense and always leads to triggering of a BUG. Fix this by only checking enqueue flags, properly: ENQUEUE_REPLENISH has to be present, but additional flags are not a problem. Fixes: 2279f540ea7d ("sched/deadline: Fix priority inheritance with multiple scheduling classes") Signed-off-by: Juri Lelli <juri.lelli@redhat.com> --- kernel/sched/deadline.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)