| Message ID | 56BDFC86.5010705@arm.com |
|---|---|
| State | New |
| Headers | show |
On Fri, Feb 12, 2016 at 03:38:46PM +0000, Sudeep Holla wrote: > > On 12/02/16 15:26, Catalin Marinas wrote: > >On Fri, Feb 12, 2016 at 04:17:09PM +0100, Ard Biesheuvel wrote: > >>On 12 February 2016 at 16:10, Catalin Marinas <catalin.marinas@arm.com> wrote: > >>>On Fri, Feb 12, 2016 at 04:02:58PM +0100, Ard Biesheuvel wrote: > >>>>On 12 February 2016 at 15:58, Catalin Marinas <catalin.marinas@arm.com> wrote: > >>>>>On Mon, Feb 01, 2016 at 11:54:52AM +0100, Ard Biesheuvel wrote: > >>>>>>This moves the module area to right before the vmalloc area, and > >>>>>>moves the kernel image to the base of the vmalloc area. This is > >>>>>>an intermediate step towards implementing KASLR, which allows the > >>>>>>kernel image to be located anywhere in the vmalloc area. > >>>>>> > >>>>>>Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> > >>>>> > >>>>>This patch is causing lots of KASAN warnings on Juno (interestingly, it > >>>>>doesn't seem to trigger on Seattle, though we only tried for-next/core). > >>>>>I pushed the branch that I'm currently using here: > >>>>> > >>>>>git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux for-next/kernmap > >>>>> > >>>>> > >>>>>A typical error (though its place varies based on the config options, > >>>>>kernel layout): > >>>>> > >>>>>BUG: KASAN: stack-out-of-bounds in clockevents_program_event+0x28/0x1b0 at addr ffffffc936257cc8 > >>>> > >>>>Can you confirm that these are stack accesses? I was having similar > >>>>errors before, and I ended up creating the kasan zero page patch > >>>>because it turned out the kasan shadow page in question was aliased > >>>>and the stack writes were occurring elsewhere. > >>> > >>>It's possible, we are looking into this. Is there any other patch I miss on > >>>the above branch? > >> > >>I don't think so but I will check > > > >Commit 7b1af9795773 ("arm64: kasan: ensure that the KASAN zero page is > >mapped read-only") was merged in -rc2 while the branch above is based on > >-rc1. Anyway, I merged it into -rc2 and the errors are similar. > > > > Sorry to add more confusion, but I observed similar KASAN warning > with latest mainline(v4.5-rc3+, commit c05235d50f68) with below diff. I can reproduce this with UBSAN enabled (log below for the record). So far, we have: KASAN+for-next/kernmap goes wrong KASAN+UBSAN goes wrong Enabled individually, KASAN, UBSAN and for-next/kernmap seem fine. I may have to trim for-next/core down until we figure out where the problem is. BUG: KASAN: stack-out-of-bounds in find_busiest_group+0x164/0x16a0 at addr ffffffc93665bc8c Read of size 4 by task swapper/3/0 page:ffffffbde6d996c0 count:0 mapcount:0 mapping: (null) index:0x0 flags: 0x4000000000000000() page dumped because: kasan: bad access detected CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.5.0-rc3+ #134 Hardware name: Juno (DT) Call trace: [<ffffffc00008f8f0>] dump_backtrace+0x0/0x358 [<ffffffc00008fc5c>] show_stack+0x14/0x20 [<ffffffc00069d0a8>] dump_stack+0x108/0x150 [<ffffffc0003077f8>] kasan_report_error+0x690/0x970 [<ffffffc0003082c0>] kasan_report+0x60/0xc0 [<ffffffc00030634c>] __asan_load4+0x64/0x80 [<ffffffc00015f714>] find_busiest_group+0x164/0x16a0 [<ffffffc000160ea0>] load_balance+0x250/0x1450 [<ffffffc0001630c0>] pick_next_task_fair+0x5d0/0xb40 [<ffffffc000f08090>] __schedule+0x460/0xbc8 [<ffffffc000f08870>] schedule+0x78/0x208 [<ffffffc000f092d4>] schedule_preempt_disabled+0x3c/0xd8 [<ffffffc000172208>] cpu_startup_entry+0x160/0x4c8 [<ffffffc0000985b8>] secondary_start_kernel+0x280/0x428 [<0000000080082e2c>] 0x80082e2c Memory state around the buggy address: ffffffc93665bb80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ffffffc93665bc00: f1 f1 f1 f1 00 f4 f4 f4 f2 f2 f2 f2 00 00 f1 f1 >ffffffc93665bc80: f1 f1 00 00 00 00 f3 f3 00 f4 f4 f4 f3 f3 f3 f3 ^ ffffffc93665bd00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ffffffc93665bd80: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 04 f4 f4 f4 -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On 12 February 2016 at 17:06, Catalin Marinas <catalin.marinas@arm.com> wrote: > On Fri, Feb 12, 2016 at 03:38:46PM +0000, Sudeep Holla wrote: >> >> On 12/02/16 15:26, Catalin Marinas wrote: >> >On Fri, Feb 12, 2016 at 04:17:09PM +0100, Ard Biesheuvel wrote: >> >>On 12 February 2016 at 16:10, Catalin Marinas <catalin.marinas@arm.com> wrote: >> >>>On Fri, Feb 12, 2016 at 04:02:58PM +0100, Ard Biesheuvel wrote: >> >>>>On 12 February 2016 at 15:58, Catalin Marinas <catalin.marinas@arm.com> wrote: >> >>>>>On Mon, Feb 01, 2016 at 11:54:52AM +0100, Ard Biesheuvel wrote: >> >>>>>>This moves the module area to right before the vmalloc area, and >> >>>>>>moves the kernel image to the base of the vmalloc area. This is >> >>>>>>an intermediate step towards implementing KASLR, which allows the >> >>>>>>kernel image to be located anywhere in the vmalloc area. >> >>>>>> >> >>>>>>Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> >> >>>>> >> >>>>>This patch is causing lots of KASAN warnings on Juno (interestingly, it >> >>>>>doesn't seem to trigger on Seattle, though we only tried for-next/core). >> >>>>>I pushed the branch that I'm currently using here: >> >>>>> >> >>>>>git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux for-next/kernmap >> >>>>> >> >>>>> >> >>>>>A typical error (though its place varies based on the config options, >> >>>>>kernel layout): >> >>>>> >> >>>>>BUG: KASAN: stack-out-of-bounds in clockevents_program_event+0x28/0x1b0 at addr ffffffc936257cc8 >> >>>> >> >>>>Can you confirm that these are stack accesses? I was having similar >> >>>>errors before, and I ended up creating the kasan zero page patch >> >>>>because it turned out the kasan shadow page in question was aliased >> >>>>and the stack writes were occurring elsewhere. >> >>> >> >>>It's possible, we are looking into this. Is there any other patch I miss on >> >>>the above branch? >> >> >> >>I don't think so but I will check >> > >> >Commit 7b1af9795773 ("arm64: kasan: ensure that the KASAN zero page is >> >mapped read-only") was merged in -rc2 while the branch above is based on >> >-rc1. Anyway, I merged it into -rc2 and the errors are similar. >> > >> >> Sorry to add more confusion, but I observed similar KASAN warning >> with latest mainline(v4.5-rc3+, commit c05235d50f68) with below diff. > > I can reproduce this with UBSAN enabled (log below for the record). > > So far, we have: > > KASAN+for-next/kernmap goes wrong > KASAN+UBSAN goes wrong > > Enabled individually, KASAN, UBSAN and for-next/kernmap seem fine. I may > have to trim for-next/core down until we figure out where the problem > is. > I haven't managed to reproduce this yet on QEMU, Seattle or FVP, but I did notice something that may or may not be related: without my changes the memory map show this: kasan : 0xffffff8000000000 - 0xffffff9000000000 ( 64 GB) vmalloc : 0xffffff9000010000 - 0xffffffbdbfff0000 ( 182 GB) i.e., there is a 64 KB guard region between the shadow region and the vmalloc region. I am not sure what it is for, but I realize now that I accidentally removed it in my patch: kasan : 0xffffff8000000000 - 0xffffff9000000000 ( 64 GB) modules : 0xffffff9000000000 - 0xffffff9004000000 ( 64 MB) vmalloc : 0xffffff9004000000 - 0xffffffbdbfff0000 ( 182 GB) > > BUG: KASAN: stack-out-of-bounds in find_busiest_group+0x164/0x16a0 at addr ffffffc93665bc8c > Read of size 4 by task swapper/3/0 > page:ffffffbde6d996c0 count:0 mapcount:0 mapping: (null) index:0x0 > flags: 0x4000000000000000() > page dumped because: kasan: bad access detected > CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.5.0-rc3+ #134 > Hardware name: Juno (DT) > Call trace: > [<ffffffc00008f8f0>] dump_backtrace+0x0/0x358 > [<ffffffc00008fc5c>] show_stack+0x14/0x20 > [<ffffffc00069d0a8>] dump_stack+0x108/0x150 > [<ffffffc0003077f8>] kasan_report_error+0x690/0x970 > [<ffffffc0003082c0>] kasan_report+0x60/0xc0 > [<ffffffc00030634c>] __asan_load4+0x64/0x80 > [<ffffffc00015f714>] find_busiest_group+0x164/0x16a0 > [<ffffffc000160ea0>] load_balance+0x250/0x1450 > [<ffffffc0001630c0>] pick_next_task_fair+0x5d0/0xb40 > [<ffffffc000f08090>] __schedule+0x460/0xbc8 > [<ffffffc000f08870>] schedule+0x78/0x208 > [<ffffffc000f092d4>] schedule_preempt_disabled+0x3c/0xd8 > [<ffffffc000172208>] cpu_startup_entry+0x160/0x4c8 > [<ffffffc0000985b8>] secondary_start_kernel+0x280/0x428 > [<0000000080082e2c>] 0x80082e2c > Memory state around the buggy address: > ffffffc93665bb80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > ffffffc93665bc00: f1 f1 f1 f1 00 f4 f4 f4 f2 f2 f2 f2 00 00 f1 f1 >>ffffffc93665bc80: f1 f1 00 00 00 00 f3 f3 00 f4 f4 f4 f3 f3 f3 f3 > ^ > ffffffc93665bd00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > ffffffc93665bd80: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 04 f4 f4 f4 > > -- > Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Mon, Feb 15, 2016 at 05:28:02PM +0300, Andrey Ryabinin wrote: > > > On 02/12/2016 07:06 PM, Catalin Marinas wrote: > > So far, we have: > > > > KASAN+for-next/kernmap goes wrong > > KASAN+UBSAN goes wrong > > > > Enabled individually, KASAN, UBSAN and for-next/kernmap seem fine. I may > > have to trim for-next/core down until we figure out where the problem > > is. > > > > BUG: KASAN: stack-out-of-bounds in find_busiest_group+0x164/0x16a0 at addr ffffffc93665bc8c > > Can it be related to TLB conflicts, which supposed to be fixed in "arm64: kasan: avoid TLB conflicts" patch > from "arm64: mm: rework page table creation" series ? Currently I don't believe this is a TLB issue. We've been seeing issues even with those patches in for-next/core with that patch included. It's also incredibly reliable to trigger. It seems that issues are more likely the larger the kernel image, so my suspicion is that at some boundary condition we create the page tables for the shadow region incorrectly. I'm only able to trigger this on a particular machine, so the physical memory layout may also matter. I'm currently looking into that. Mark. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Mon, Feb 15, 2016 at 05:28:02PM +0300, Andrey Ryabinin wrote: > On 02/12/2016 07:06 PM, Catalin Marinas wrote: > > So far, we have: > > > > KASAN+for-next/kernmap goes wrong > > KASAN+UBSAN goes wrong > > > > Enabled individually, KASAN, UBSAN and for-next/kernmap seem fine. I may > > have to trim for-next/core down until we figure out where the problem > > is. > > > > BUG: KASAN: stack-out-of-bounds in find_busiest_group+0x164/0x16a0 at addr ffffffc93665bc8c > > Can it be related to TLB conflicts, which supposed to be fixed in > "arm64: kasan: avoid TLB conflicts" patch from "arm64: mm: rework page > table creation" series ? I can very easily reproduce this with a vanilla 4.5-rc1 series by enabling inline instrumentation (maybe Mark's theory is true w.r.t. image size). Some information, maybe you can shed some light on this. It seems to happen only for secondary CPUs on the swapper stack (I think allocated via fork_idle()). The code generated looks sane to me, so KASAN should not complain but maybe there is some uninitialised shadow, hence the error. The report: -------------->8--------------- BUG: KASAN: stack-out-of-bounds in clockevents_program_event+0x354/0x368 at addr ffffffc93651bca8 Read of size 8 by task swapper/1/0 page:ffffffbde6d946c0 count:0 mapcount:0 mapping: (null) index:0x0 flags: 0x4000000000000000() page dumped because: kasan: bad access detected CPU: 1 PID: 0 Comm: swapper/1 Tainted: G B 4.5.0-rc1+ #163 Hardware name: Juno (DT) Call trace: [<ffffffc00008f130>] dump_backtrace+0x0/0x358 [<ffffffc00008f49c>] show_stack+0x14/0x20 [<ffffffc000785dc0>] dump_stack+0xf8/0x188 [<ffffffc000343c0c>] kasan_report_error+0x524/0x550 [<ffffffc000343d50>] __asan_report_load8_noabort+0x40/0x48 [<ffffffc0001f2bc4>] clockevents_program_event+0x354/0x368 [<ffffffc0001f73d4>] tick_program_event+0xac/0x108 [<ffffffc0001d85c8>] hrtimer_start_range_ns+0x8a0/0xb20 [<ffffffc0001f8ba8>] __tick_nohz_idle_enter+0x970/0xca8 [<ffffffc0001f9368>] tick_nohz_idle_enter+0x60/0x98 [<ffffffc0001933ec>] cpu_startup_entry+0x14c/0x448 [<ffffffc000098654>] secondary_start_kernel+0x264/0x2e0 [<0000000080082ecc>] 0x80082ecc Memory state around the buggy address: ffffffc93651bb80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ffffffc93651bc00: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1 >ffffffc93651bc80: 00 00 00 00 f3 f3 f3 f3 00 00 00 00 00 00 00 00 ^ ffffffc93651bd00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ffffffc93651bd80: 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1 00 f4 -------------->8--------------- I put some printks in clockevents_program_event() and the SP is 0xffffffc93651bc70, so it matches the above. Disassembling the code: -------------->8--------------- ffffffc0001f2870 <clockevents_program_event>: ffffffc0001f2870: a9bc7bfd stp x29, x30, [sp,#-64]! ffffffc0001f2874: d2dff204 mov x4, #0xff9000000000 // #280993940373504 ffffffc0001f2878: 910003fd mov x29, sp ffffffc0001f287c: 910103a3 add x3, x29, #0x40 ffffffc0001f2880: f2fbffe4 movk x4, #0xdfff, lsl #48 ffffffc0001f2884: a90153f3 stp x19, x20, [sp,#16] ffffffc0001f2888: a9025bf5 stp x21, x22, [sp,#32] ffffffc0001f288c: f81f8c61 str x1, [x3,#-8]! ffffffc0001f2890: aa0003f3 mov x19, x0 ffffffc0001f2894: 53001c55 uxtb w21, w2 ffffffc0001f2898: d343fc60 lsr x0, x3, #3 ffffffc0001f289c: 38e46800 ldrsb w0, [x0,x4] ffffffc0001f28a0: 350018e0 cbnz w0, ffffffc0001f2bbc <clockevents_program_event+0x34c> [...] ffffffc0001f2bbc: aa0303e0 mov x0, x3 ffffffc0001f2bc0: 94054454 bl ffffffc000343d10 <__asan_report_load8_noabort> -------------->8--------------- To me, line ffffffc0001f288c looks like a normal store to a stack variable and the stack boundaries look fine. The ffffffc0001f289c line checks shadow and reads non-zero, hence the report. But I don't get what's wrong with this function, other than corrupt KASAN shadow. -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Tue, Feb 16, 2016 at 03:59:09PM +0300, Andrey Ryabinin wrote: > > On 02/15/2016 09:59 PM, Catalin Marinas wrote: > > On Mon, Feb 15, 2016 at 05:28:02PM +0300, Andrey Ryabinin wrote: > >> On 02/12/2016 07:06 PM, Catalin Marinas wrote: > >>> So far, we have: > >>> > >>> KASAN+for-next/kernmap goes wrong > >>> KASAN+UBSAN goes wrong > >>> > >>> Enabled individually, KASAN, UBSAN and for-next/kernmap seem fine. I may > >>> have to trim for-next/core down until we figure out where the problem > >>> is. > >>> > >>> BUG: KASAN: stack-out-of-bounds in find_busiest_group+0x164/0x16a0 at addr ffffffc93665bc8c > >> > >> Can it be related to TLB conflicts, which supposed to be fixed in > >> "arm64: kasan: avoid TLB conflicts" patch from "arm64: mm: rework page > >> table creation" series ? > > > > I can very easily reproduce this with a vanilla 4.5-rc1 series by > > enabling inline instrumentation (maybe Mark's theory is true w.r.t. > > image size). > > > > Some information, maybe you can shed some light on this. It seems to > > happen only for secondary CPUs on the swapper stack (I think allocated > > via fork_idle()). The code generated looks sane to me, so KASAN should > > not complain but maybe there is some uninitialised shadow, hence the > > error. > > > > The report: > > > > Actually, the first report is a bit more useful. It shows that shadow memory was corrupted: > > ffffffc93665bc00: f1 f1 f1 f1 00 f4 f4 f4 f2 f2 f2 f2 00 00 f1 f1 > > ffffffc93665bc80: f1 f1 00 00 00 00 f3 f3 00 f4 f4 f4 f3 f3 f3 f3 > ^ > F1 - left redzone, it indicates start of stack frame > F3 - right redzone, it should be the end of stack frame. > > But here we have the second set of F1s without F3s which should close the first set of F1s. > Also those two F3s in the middle cannot be right. > > So shadow is corrupted. > Some hypotheses: > > 1) We share stack between several tasks (e.g. stack overflow, somehow corrupted SP). > But this probably should cause kernel crash later, after kasan reports. > > 2) Shadow memory wasn't cleared. GCC poison memory on function entrance and unpoisons it before return. > If we use some tricky way to exit from function this could cause false-positives like that. > E.g. some hand-written assembly return code. > > 3) Screwed shadow mapping. I think the patch below should uncover such problem. > It boot-tested on qemu and didn't show any problem With that path applied I get: [ 0.000000] kasan: screwed shadow mapping 62184, 62182 [ 0.000000] kasan: KernelAddressSanitizer initialized I'm using v4.5-rc1 with KASAN_INLINE, and a random collection of debug options to bloat the kernel per prior theory that the text size had somethign to do with the issue. Later in the boot process I see lots of failures like: [ 13.292190] ================================================================== [ 13.299543] BUG: KASAN: stack-out-of-bounds in find_busiest_group+0x1950/0x19b8 at addr ffffffc936ad3c8c [ 13.309090] Read of size 4 by task swapper/3/0 [ 13.313575] page:ffffffbde6dab4c0 count:0 mapcount:0 mapping: (null) index:0x0 [ 13.321657] flags: 0x4000000000000000() [ 13.325539] page dumped because: kasan: bad access detected [ 13.331150] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.5.0-rc1+ #19 [ 13.337528] Hardware name: ARM Juno development board (r1) (DT) [ 13.343471] Call trace: [ 13.345978] [<ffffffc000091400>] dump_backtrace+0x0/0x3c0 [ 13.351416] [<ffffffc0000917e4>] show_stack+0x24/0x30 [ 13.356507] [<ffffffc0008c3a64>] dump_stack+0xc4/0x150 [ 13.361685] [<ffffffc0004032bc>] kasan_report_error+0x52c/0x558 [ 13.367640] [<ffffffc0004033fc>] __asan_report_load4_noabort+0x54/0x60 [ 13.374200] [<ffffffc0001a46e8>] find_busiest_group+0x1950/0x19b8 [ 13.380327] [<ffffffc0001a49ec>] load_balance+0x29c/0x19e0 [ 13.385851] [<ffffffc0001a67c0>] pick_next_task_fair+0x690/0xd88 [ 13.391896] [<ffffffc001213cf4>] __schedule+0x85c/0x13c8 [ 13.397248] [<ffffffc001214d7c>] schedule+0xe4/0x228 [ 13.402256] [<ffffffc00121549c>] schedule_preempt_disabled+0x24/0xb8 [ 13.408642] [<ffffffc0001b97f8>] cpu_startup_entry+0x188/0x738 [ 13.414511] [<ffffffc00009bcfc>] secondary_start_kernel+0x244/0x2b8 [ 13.420806] [<0000000080082efc>] 0x80082efc [ 13.425023] Memory state around the buggy address: [ 13.429854] ffffffc936ad3b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 13.437153] ffffffc936ad3c00: 00 00 00 00 00 00 f1 f1 f1 f1 f1 f1 00 00 f3 f3 [ 13.444451] >ffffffc936ad3c80: f3 f3 00 00 00 00 00 00 00 f4 f4 f4 f3 f3 f3 f3 [ 13.451742] ^ [ 13.455274] ffffffc936ad3d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 13.462572] ffffffc936ad3d80: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1 [ 13.469863] ================================================================== I guess memroy layout has something to do with this. FWIW on this board my memory map comes from EFI: [ 0.000000] Processing EFI memory map: [ 0.000000] 0x000008000000-0x00000bffffff [Memory Mapped I/O |RUN| |XP| | | | | | | |UC] [ 0.000000] 0x00001c170000-0x00001c170fff [Memory Mapped I/O |RUN| |XP| | | | | | | |UC] [ 0.000000] 0x000080000000-0x00008000ffff [Loader Data | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x000080010000-0x00008007ffff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x000080080000-0x000081dbffff [Loader Data | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x000081dc0000-0x00009fdfffff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x00009fe00000-0x00009fe0ffff [Loader Data | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x00009fe10000-0x0000dfffffff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000e00f0000-0x0000f5a58fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000f5a59000-0x0000f7793fff [Loader Code | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000f7794000-0x0000f9431fff [Loader Data | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000f9432000-0x0000f944ffff [Loader Code | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000f9450000-0x0000f945ffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000f9460000-0x0000f94dffff [ACPI Reclaim Memory| | | | | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000f94e0000-0x0000f94effff [ACPI Memory NVS | | | | | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000f94f0000-0x0000f94fffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000f9500000-0x0000f950ffff [Runtime Code |RUN| | | | |RO| |WB|WT|WC|UC]* [ 0.000000] 0x0000f9510000-0x0000f953ffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000f9540000-0x0000f954ffff [Runtime Code |RUN| | | | |RO| |WB|WT|WC|UC]* [ 0.000000] 0x0000f9550000-0x0000f956ffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000f9570000-0x0000f958ffff [ACPI Reclaim Memory| | | | | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000f9590000-0x0000f960ffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000f9610000-0x0000f961ffff [Runtime Code |RUN| | | | |RO| |WB|WT|WC|UC]* [ 0.000000] 0x0000f9620000-0x0000f96effff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000f96f0000-0x0000f96fffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000f9700000-0x0000f970ffff [Runtime Code |RUN| | | | |RO| |WB|WT|WC|UC]* [ 0.000000] 0x0000f9710000-0x0000f974ffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000f9750000-0x0000f975ffff [Runtime Code |RUN| | | | |RO| |WB|WT|WC|UC]* [ 0.000000] 0x0000f9760000-0x0000f97cffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000f97d0000-0x0000f97dffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000f97e0000-0x0000f97effff [Runtime Code |RUN| | | | |RO| |WB|WT|WC|UC]* [ 0.000000] 0x0000f97f0000-0x0000f981ffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000f9820000-0x0000f9820fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000f9821000-0x0000f9827fff [Loader Data | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000f9828000-0x0000f982bfff [Reserved | | | | | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000f982c000-0x0000fdaedfff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fdaee000-0x0000fdfbefff [Boot Data | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fdfbf000-0x0000fdfbffff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fdfc0000-0x0000fdffbfff [Boot Data | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fdffc000-0x0000fe018fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe019000-0x0000fe020fff [Boot Data | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe021000-0x0000fe022fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe023000-0x0000fe02bfff [Boot Data | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe02c000-0x0000fe03afff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe03b000-0x0000fe03dfff [Boot Data | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe03e000-0x0000fe04efff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe04f000-0x0000fe057fff [Boot Data | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe058000-0x0000fe073fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe074000-0x0000fe074fff [Boot Data | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe075000-0x0000fe078fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe079000-0x0000fe07bfff [Boot Data | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe07c000-0x0000fe07dfff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe07e000-0x0000fe085fff [Boot Data | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe086000-0x0000fe087fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe088000-0x0000fe171fff [Boot Data | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe172000-0x0000fe198fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe199000-0x0000fe65ffff [Boot Data | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe660000-0x0000fe6a2fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe6a3000-0x0000fe7effff [Boot Code | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe7f0000-0x0000fe7fffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000fe800000-0x0000fe80ffff [Runtime Code |RUN| | | | |RO| |WB|WT|WC|UC]* [ 0.000000] 0x0000fe810000-0x0000fe82ffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000fe830000-0x0000fe83ffff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe840000-0x0000fe88ffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* [ 0.000000] 0x0000fe890000-0x0000fe891fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x0000fe892000-0x0000feffffff [Boot Data | | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x000880000000-0x00099bffffff [Conventional Memory| | | | | | | |WB|WT|WC|UC] [ 0.000000] 0x00099c000000-0x0009ffffffff [Loader Data | | | | | | | |WB|WT|WC|UC] Thanks, Mark. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Tue, Feb 16, 2016 at 02:12:59PM +0000, Mark Rutland wrote: > On Tue, Feb 16, 2016 at 03:59:09PM +0300, Andrey Ryabinin wrote: > > So shadow is corrupted. > > Some hypotheses: > > > > 1) We share stack between several tasks (e.g. stack overflow, somehow corrupted SP). > > But this probably should cause kernel crash later, after kasan reports. > > > > 2) Shadow memory wasn't cleared. GCC poison memory on function entrance and unpoisons it before return. > > If we use some tricky way to exit from function this could cause false-positives like that. > > E.g. some hand-written assembly return code. > > > > 3) Screwed shadow mapping. I think the patch below should uncover such problem. > > It boot-tested on qemu and didn't show any problem > > With that path applied I get: > > [ 0.000000] kasan: screwed shadow mapping 62184, 62182 > [ 0.000000] kasan: KernelAddressSanitizer initialized > > I'm using v4.5-rc1 with KASAN_INLINE, and a random collection of debug options > to bloat the kernel per prior theory that the text size had somethign to do > with the issue. I hacked kasan_init to dump info as it created each shadow region: [ 0.000000] kasan_init shadowing [ffffffc000000000-ffffffc060000000] @ [ffffff8800000000-ffffff880c000001] nid 0 [ 0.000000] kasan_init shadowing [ffffffc0600f0000-ffffffc079450000] @ [ffffff880c01e000-ffffff880f28a001] nid 0 [ 0.000000] kasan_init shadowing [ffffffc079450000-ffffffc079820000] @ [ffffff880f28a000-ffffff880f304001] nid 0 [ 0.000000] kasan_init shadowing [ffffffc079820000-ffffffc079821000] @ [ffffff880f304000-ffffff880f304201] nid 0 [ 0.000000] kasan_init shadowing [ffffffc079821000-ffffffc079822000] @ [ffffff880f304200-ffffff880f304401] nid 0 [ 0.000000] kasan_init shadowing [ffffffc079822000-ffffffc079828000] @ [ffffff880f304400-ffffff880f305001] nid 0 [ 0.000000] kasan_init shadowing [ffffffc079828000-ffffffc07982c000] @ [ffffff880f305000-ffffff880f305801] nid 0 [ 0.000000] kasan_init shadowing [ffffffc07982c000-ffffffc07e7f0000] @ [ffffff880f305800-ffffff880fcfe001] nid 0 [ 0.000000] kasan_init shadowing [ffffffc07e7f0000-ffffffc07e830000] @ [ffffff880fcfe000-ffffff880fd06001] nid 0 [ 0.000000] kasan_init shadowing [ffffffc07e830000-ffffffc07e840000] @ [ffffff880fd06000-ffffff880fd08001] nid 0 [ 0.000000] kasan_init shadowing [ffffffc07e840000-ffffffc07e890000] @ [ffffff880fd08000-ffffff880fd12001] nid 0 [ 0.000000] kasan_init shadowing [ffffffc07e890000-ffffffc07f000000] @ [ffffff880fd12000-ffffff880fe00001] nid 0 [ 0.000000] kasan_init shadowing [ffffffc800000000-ffffffc980000000] @ [ffffff8900000000-ffffff8930000001] nid 0 [ 0.000000] kasan: screwed shadow mapping 62184, 62182 [ 0.000000] kasan: KernelAddressSanitizer initialized I note the the end of each shadow region overlaps the beginning of the next due to the intentional end+1... Other than the waste of memory (and the TLB conflict that gets solved by my pgtable rework), I'm not sure though I'm not sure that's a problem, though. Mark. > I guess memroy layout has something to do with this. FWIW on this board my > memory map comes from EFI: > > [ 0.000000] Processing EFI memory map: > [ 0.000000] 0x000008000000-0x00000bffffff [Memory Mapped I/O |RUN| |XP| | | | | | | |UC] > [ 0.000000] 0x00001c170000-0x00001c170fff [Memory Mapped I/O |RUN| |XP| | | | | | | |UC] > [ 0.000000] 0x000080000000-0x00008000ffff [Loader Data | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x000080010000-0x00008007ffff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x000080080000-0x000081dbffff [Loader Data | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x000081dc0000-0x00009fdfffff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x00009fe00000-0x00009fe0ffff [Loader Data | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x00009fe10000-0x0000dfffffff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000e00f0000-0x0000f5a58fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000f5a59000-0x0000f7793fff [Loader Code | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000f7794000-0x0000f9431fff [Loader Data | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000f9432000-0x0000f944ffff [Loader Code | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000f9450000-0x0000f945ffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000f9460000-0x0000f94dffff [ACPI Reclaim Memory| | | | | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000f94e0000-0x0000f94effff [ACPI Memory NVS | | | | | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000f94f0000-0x0000f94fffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000f9500000-0x0000f950ffff [Runtime Code |RUN| | | | |RO| |WB|WT|WC|UC]* > [ 0.000000] 0x0000f9510000-0x0000f953ffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000f9540000-0x0000f954ffff [Runtime Code |RUN| | | | |RO| |WB|WT|WC|UC]* > [ 0.000000] 0x0000f9550000-0x0000f956ffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000f9570000-0x0000f958ffff [ACPI Reclaim Memory| | | | | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000f9590000-0x0000f960ffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000f9610000-0x0000f961ffff [Runtime Code |RUN| | | | |RO| |WB|WT|WC|UC]* > [ 0.000000] 0x0000f9620000-0x0000f96effff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000f96f0000-0x0000f96fffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000f9700000-0x0000f970ffff [Runtime Code |RUN| | | | |RO| |WB|WT|WC|UC]* > [ 0.000000] 0x0000f9710000-0x0000f974ffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000f9750000-0x0000f975ffff [Runtime Code |RUN| | | | |RO| |WB|WT|WC|UC]* > [ 0.000000] 0x0000f9760000-0x0000f97cffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000f97d0000-0x0000f97dffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000f97e0000-0x0000f97effff [Runtime Code |RUN| | | | |RO| |WB|WT|WC|UC]* > [ 0.000000] 0x0000f97f0000-0x0000f981ffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000f9820000-0x0000f9820fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000f9821000-0x0000f9827fff [Loader Data | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000f9828000-0x0000f982bfff [Reserved | | | | | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000f982c000-0x0000fdaedfff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fdaee000-0x0000fdfbefff [Boot Data | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fdfbf000-0x0000fdfbffff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fdfc0000-0x0000fdffbfff [Boot Data | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fdffc000-0x0000fe018fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe019000-0x0000fe020fff [Boot Data | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe021000-0x0000fe022fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe023000-0x0000fe02bfff [Boot Data | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe02c000-0x0000fe03afff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe03b000-0x0000fe03dfff [Boot Data | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe03e000-0x0000fe04efff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe04f000-0x0000fe057fff [Boot Data | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe058000-0x0000fe073fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe074000-0x0000fe074fff [Boot Data | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe075000-0x0000fe078fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe079000-0x0000fe07bfff [Boot Data | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe07c000-0x0000fe07dfff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe07e000-0x0000fe085fff [Boot Data | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe086000-0x0000fe087fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe088000-0x0000fe171fff [Boot Data | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe172000-0x0000fe198fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe199000-0x0000fe65ffff [Boot Data | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe660000-0x0000fe6a2fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe6a3000-0x0000fe7effff [Boot Code | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe7f0000-0x0000fe7fffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000fe800000-0x0000fe80ffff [Runtime Code |RUN| | | | |RO| |WB|WT|WC|UC]* > [ 0.000000] 0x0000fe810000-0x0000fe82ffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000fe830000-0x0000fe83ffff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe840000-0x0000fe88ffff [Runtime Data |RUN| |XP| | | | |WB|WT|WC|UC]* > [ 0.000000] 0x0000fe890000-0x0000fe891fff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x0000fe892000-0x0000feffffff [Boot Data | | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x000880000000-0x00099bffffff [Conventional Memory| | | | | | | |WB|WT|WC|UC] > [ 0.000000] 0x00099c000000-0x0009ffffffff [Loader Data | | | | | | | |WB|WT|WC|UC] > > Thanks, > Mark. > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On 16 February 2016 at 13:59, Andrey Ryabinin <aryabinin@virtuozzo.com> wrote: > > > On 02/15/2016 09:59 PM, Catalin Marinas wrote: >> On Mon, Feb 15, 2016 at 05:28:02PM +0300, Andrey Ryabinin wrote: >>> On 02/12/2016 07:06 PM, Catalin Marinas wrote: >>>> So far, we have: >>>> >>>> KASAN+for-next/kernmap goes wrong >>>> KASAN+UBSAN goes wrong >>>> >>>> Enabled individually, KASAN, UBSAN and for-next/kernmap seem fine. I may >>>> have to trim for-next/core down until we figure out where the problem >>>> is. >>>> >>>> BUG: KASAN: stack-out-of-bounds in find_busiest_group+0x164/0x16a0 at addr ffffffc93665bc8c >>> >>> Can it be related to TLB conflicts, which supposed to be fixed in >>> "arm64: kasan: avoid TLB conflicts" patch from "arm64: mm: rework page >>> table creation" series ? >> >> I can very easily reproduce this with a vanilla 4.5-rc1 series by >> enabling inline instrumentation (maybe Mark's theory is true w.r.t. >> image size). >> >> Some information, maybe you can shed some light on this. It seems to >> happen only for secondary CPUs on the swapper stack (I think allocated >> via fork_idle()). The code generated looks sane to me, so KASAN should >> not complain but maybe there is some uninitialised shadow, hence the >> error. >> >> The report: >> > > Actually, the first report is a bit more useful. It shows that shadow memory was corrupted: > > ffffffc93665bc00: f1 f1 f1 f1 00 f4 f4 f4 f2 f2 f2 f2 00 00 f1 f1 >> ffffffc93665bc80: f1 f1 00 00 00 00 f3 f3 00 f4 f4 f4 f3 f3 f3 f3 > ^ > F1 - left redzone, it indicates start of stack frame > F3 - right redzone, it should be the end of stack frame. > > But here we have the second set of F1s without F3s which should close the first set of F1s. > Also those two F3s in the middle cannot be right. > > So shadow is corrupted. > Some hypotheses: > > 1) We share stack between several tasks (e.g. stack overflow, somehow corrupted SP). > But this probably should cause kernel crash later, after kasan reports. > > 2) Shadow memory wasn't cleared. GCC poison memory on function entrance and unpoisons it before return. > If we use some tricky way to exit from function this could cause false-positives like that. > E.g. some hand-written assembly return code. > > 3) Screwed shadow mapping. I think the patch below should uncover such problem. > It boot-tested on qemu and didn't show any problem > I think this patch gives false positive warnings in some cases: > > --- > arch/arm64/mm/kasan_init.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 55 insertions(+) > > diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c > index cf038c7..25d685c 100644 > --- a/arch/arm64/mm/kasan_init.c > +++ b/arch/arm64/mm/kasan_init.c > @@ -117,6 +117,59 @@ static void __init cpu_set_ttbr1(unsigned long ttbr1) > : "r" (ttbr1)); > } > > +static void verify_shadow(void) > +{ > + struct memblock_region *reg; > + int i = 0; > + > + for_each_memblock(memory, reg) { > + void *start = (void *)__phys_to_virt(reg->base); > + void *end = (void *)__phys_to_virt(reg->base + reg->size); > + int *shadow_start, *shadow_end; > + > + if (start >= end) > + break; > + shadow_start = (int *)((unsigned long)kasan_mem_to_shadow(start) & ~(PAGE_SIZE - 1)); > + shadow_end = (int *)kasan_mem_to_shadow(end); shadow_start and shadow_end can refer to the same page as in the previous iteration. For instance, I have these two regions 0x00006e090000-0x00006e0adfff [Conventional Memory| | | | | | | |WB|WT|WC|UC] 0x00006e0ae000-0x00006e0affff [Loader Data | | | | | | | |WB|WT|WC|UC] which are covered by different memblocks since the second one is marked as MEMBLOCK_NOMAP, due to the fact that it contains the UEFI memory map. I get the following output kasan: screwed shadow mapping 23575, 23573 which I think is simply a result from the fact the shadow_start refers to the same page as in the previous iteration(s) > + for (; shadow_start < shadow_end; shadow_start += PAGE_SIZE/sizeof(int)) { > + *shadow_start = i; > + i++; > + } > + } > + > + i = 0; > + for_each_memblock(memory, reg) { > + void *start = (void *)__phys_to_virt(reg->base); > + void *end = (void *)__phys_to_virt(reg->base + reg->size); > + int *shadow_start, *shadow_end; > + > + if (start >= end) > + break; > + shadow_start = (int *)((unsigned long)kasan_mem_to_shadow(start) & ~(PAGE_SIZE - 1)); > + shadow_end = (int *)kasan_mem_to_shadow(end); > + for (; shadow_start < shadow_end; shadow_start += PAGE_SIZE/sizeof(int)) { > + if (*shadow_start != i) { > + pr_err("screwed shadow mapping %d, %d\n", *shadow_start, i); > + goto clear; > + } > + i++; > + } > + } > +clear: > + for_each_memblock(memory, reg) { > + void *start = (void *)__phys_to_virt(reg->base); > + void *end = (void *)__phys_to_virt(reg->base + reg->size); > + unsigned long shadow_start, shadow_end; > + > + if (start >= end) > + break; > + shadow_start = ((unsigned long)kasan_mem_to_shadow(start) & ~(PAGE_SIZE - 1)); > + shadow_end = (unsigned long)kasan_mem_to_shadow(end); > + memset((void *)shadow_start, 0, shadow_end - shadow_start); > + } > + > +} > + > void __init kasan_init(void) > { > struct memblock_region *reg; > @@ -159,6 +212,8 @@ void __init kasan_init(void) > cpu_set_ttbr1(__pa(swapper_pg_dir)); > flush_tlb_all(); > > + verify_shadow(); > + > /* At this point kasan is fully initialized. Enable error messages */ > init_task.kasan_depth = 0; > pr_info("KernelAddressSanitizer initialized\n"); > -- > > > > > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Tue, Feb 16, 2016 at 06:36:36PM +0300, Andrey Ryabinin wrote: > > On 02/16/2016 06:17 PM, Ard Biesheuvel wrote: > > On 16 February 2016 at 13:59, Andrey Ryabinin <aryabinin@virtuozzo.com> wrote: > >> +static void verify_shadow(void) > >> +{ > >> + struct memblock_region *reg; > >> + int i = 0; > >> + > >> + for_each_memblock(memory, reg) { > >> + void *start = (void *)__phys_to_virt(reg->base); > >> + void *end = (void *)__phys_to_virt(reg->base + reg->size); > >> + int *shadow_start, *shadow_end; > >> + > >> + if (start >= end) > >> + break; > >> + shadow_start = (int *)((unsigned long)kasan_mem_to_shadow(start) & ~(PAGE_SIZE - 1)); > >> + shadow_end = (int *)kasan_mem_to_shadow(end); > > > > shadow_start and shadow_end can refer to the same page as in the > > previous iteration. For instance, I have these two regions > > > > 0x00006e090000-0x00006e0adfff [Conventional Memory| | | | | | > > | |WB|WT|WC|UC] > > 0x00006e0ae000-0x00006e0affff [Loader Data | | | | | | > > | |WB|WT|WC|UC] > > > > which are covered by different memblocks since the second one is > > marked as MEMBLOCK_NOMAP, due to the fact that it contains the UEFI > > memory map. > > > > I get the following output > > > > kasan: screwed shadow mapping 23575, 23573 > > > > which I think is simply a result from the fact the shadow_start refers > > to the same page as in the previous iteration(s) > > > > You are right. > So we should write 'shadow_start' instead of 'i'. FWIW with the below patch I don't see any "screwed shadow mapping" warnings on my board, and still later see a tonne of KASAN splats in the scheduler. Mark. > --- > arch/arm64/mm/kasan_init.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 51 insertions(+) > > diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c > index cf038c7..ee035c2 100644 > --- a/arch/arm64/mm/kasan_init.c > +++ b/arch/arm64/mm/kasan_init.c > @@ -117,6 +117,55 @@ static void __init cpu_set_ttbr1(unsigned long ttbr1) > : "r" (ttbr1)); > } > > +static void verify_shadow(void) > +{ > + struct memblock_region *reg; > + > + for_each_memblock(memory, reg) { > + void *start = (void *)__phys_to_virt(reg->base); > + void *end = (void *)__phys_to_virt(reg->base + reg->size); > + unsigned long *shadow_start, *shadow_end; > + > + if (start >= end) > + break; > + shadow_start = (unsigned long *)kasan_mem_to_shadow(start); > + shadow_end = (unsigned long *)kasan_mem_to_shadow(end); > + for (; shadow_start < shadow_end; shadow_start += PAGE_SIZE/sizeof(unsigned long)) { > + *shadow_start = (unsigned long)shadow_start; > + } > + } > + > + for_each_memblock(memory, reg) { > + void *start = (void *)__phys_to_virt(reg->base); > + void *end = (void *)__phys_to_virt(reg->base + reg->size); > + unsigned long *shadow_start, *shadow_end; > + > + if (start >= end) > + break; > + shadow_start = (unsigned long *)kasan_mem_to_shadow(start); > + shadow_end = (unsigned long *)kasan_mem_to_shadow(end); > + for (; shadow_start < shadow_end; shadow_start += PAGE_SIZE/sizeof(unsigned long)) { > + if (*shadow_start != (unsigned long)shadow_start) { > + pr_err("screwed shadow mapping %lx, %lx\n", *shadow_start, (unsigned long)shadow_start); > + goto clear; > + } > + } > + } > +clear: > + for_each_memblock(memory, reg) { > + void *start = (void *)__phys_to_virt(reg->base); > + void *end = (void *)__phys_to_virt(reg->base + reg->size); > + unsigned long shadow_start, shadow_end; > + > + if (start >= end) > + break; > + shadow_start = (unsigned long)kasan_mem_to_shadow(start); > + shadow_end = (unsigned long)kasan_mem_to_shadow(end); > + memset((void *)shadow_start, 0, shadow_end - shadow_start); > + } > + > +} > + > void __init kasan_init(void) > { > struct memblock_region *reg; > @@ -159,6 +208,8 @@ void __init kasan_init(void) > cpu_set_ttbr1(__pa(swapper_pg_dir)); > flush_tlb_all(); > > + verify_shadow(); > + > /* At this point kasan is fully initialized. Enable error messages */ > init_task.kasan_depth = 0; > pr_info("KernelAddressSanitizer initialized\n"); > -- > > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Wed, Feb 17, 2016 at 12:15:15PM +0300, Andrey Ryabinin wrote: > On 02/16/2016 07:42 PM, Mark Rutland wrote: > > FWIW with the below patch I don't see any "screwed shadow mapping" > > warnings on my board, and still later see a tonne of KASAN splats in the > > scheduler. > > It is possible that I missed something, but I think it means that > shadow is alright. > > I wonder whether this happens on 4.4. If not, than something in > 4.5-rc1 caused this, and the obvious suspect here is irq stack. It doesn't seem to happen on 4.4, it starts somewhere before 4.5-rc1. I tested the arm64 branch that we pushed upstream with the irq stack changes but it didn't trigger. It could as well be a combination of multiple change or just something else. We'll do some bisecting, though it's not that fun going through the merging window commits, especially since many are based on 4.4-rcX (we could try on merges only first, there are fewer). -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Wed, Feb 17, 2016 at 10:10:00AM +0000, James Morse wrote: > On 17/02/16 09:15, Andrey Ryabinin wrote: > > On 02/16/2016 07:42 PM, Mark Rutland wrote: > >> On Tue, Feb 16, 2016 at 06:36:36PM +0300, Andrey Ryabinin wrote: > >>> You are right. > >>> So we should write 'shadow_start' instead of 'i'. > >> > >> FWIW with the below patch I don't see any "screwed shadow mapping" > >> warnings on my board, and still later see a tonne of KASAN splats in the > >> scheduler. > >> > > > > It is possible that I missed something, but I think it means that shadow is alright. > > > > I wonder whether this happens on 4.4. If not, than something in 4.5-rc1 caused this, and the obvious suspect > > here is irq stack. > > This quick hack will prevent ever switching to the irq stack: > > ---------------------------%<--------------------------- > diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S > index 1f7f5a2b61bf..83ae736429b6 100644 > --- a/arch/arm64/kernel/entry.S > +++ b/arch/arm64/kernel/entry.S > @@ -188,7 +188,7 @@ alternative_endif > */ > and x25, x19, #~(THREAD_SIZE - 1) > cmp x25, tsk > - b.ne 9998f > + b 9998f > > this_cpu_ptr irq_stack, x25, x26 > mov x26, #IRQ_STACK_START_SP Thanks James. I'll give it a try. -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Wed, Feb 17, 2016 at 10:19:41AM +0000, Catalin Marinas wrote: > On Wed, Feb 17, 2016 at 10:10:00AM +0000, James Morse wrote: > > On 17/02/16 09:15, Andrey Ryabinin wrote: > > > On 02/16/2016 07:42 PM, Mark Rutland wrote: > > >> On Tue, Feb 16, 2016 at 06:36:36PM +0300, Andrey Ryabinin wrote: > > >>> You are right. > > >>> So we should write 'shadow_start' instead of 'i'. > > >> > > >> FWIW with the below patch I don't see any "screwed shadow mapping" > > >> warnings on my board, and still later see a tonne of KASAN splats in the > > >> scheduler. > > >> > > > > > > It is possible that I missed something, but I think it means that shadow is alright. > > > > > > I wonder whether this happens on 4.4. If not, than something in 4.5-rc1 caused this, and the obvious suspect > > > here is irq stack. > > > > This quick hack will prevent ever switching to the irq stack: > > > > ---------------------------%<--------------------------- > > diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S > > index 1f7f5a2b61bf..83ae736429b6 100644 > > --- a/arch/arm64/kernel/entry.S > > +++ b/arch/arm64/kernel/entry.S > > @@ -188,7 +188,7 @@ alternative_endif > > */ > > and x25, x19, #~(THREAD_SIZE - 1) > > cmp x25, tsk > > - b.ne 9998f > > + b 9998f > > > > this_cpu_ptr irq_stack, x25, x26 > > mov x26, #IRQ_STACK_START_SP > > Thanks James. I'll give it a try. And it didn't make any difference (on top of 4.5-rc1), still the same KASAN warnings. -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Wed, Feb 17, 2016 at 10:18:03AM +0000, Catalin Marinas wrote: > On Wed, Feb 17, 2016 at 12:15:15PM +0300, Andrey Ryabinin wrote: > > On 02/16/2016 07:42 PM, Mark Rutland wrote: > > > FWIW with the below patch I don't see any "screwed shadow mapping" > > > warnings on my board, and still later see a tonne of KASAN splats in the > > > scheduler. > > > > It is possible that I missed something, but I think it means that > > shadow is alright. > > > > I wonder whether this happens on 4.4. If not, than something in > > 4.5-rc1 caused this, and the obvious suspect here is irq stack. > > It doesn't seem to happen on 4.4, it starts somewhere before 4.5-rc1. I > tested the arm64 branch that we pushed upstream with the irq stack > changes but it didn't trigger. It could as well be a combination of > multiple change or just something else. > > We'll do some bisecting, though it's not that fun going through the > merging window commits, especially since many are based on 4.4-rcX (we > could try on merges only first, there are fewer). FWIW I did that bisect last night, and that fingered commit f11aef69b235bc30 ("Merge branch 'pm-cpuidle'") as the first bad commit. Either there's some subtle interaction, or it's less reproducible than I thought and the "good" commits are only "possibly-good". I'll try to dig into that. FWIW, my bisect log is: git bisect start # bad: [92e963f50fc74041b5e9e744c330dca48e04f08d] Linux 4.5-rc1 git bisect bad 92e963f50fc74041b5e9e744c330dca48e04f08d # good: [afd2ff9b7e1b367172f18ba7f693dfb62bdcb2dc] Linux 4.4 git bisect good afd2ff9b7e1b367172f18ba7f693dfb62bdcb2dc # good: [4dffbfc48d65e5d8157a634fd670065d237a9377] arm64/efi: mark UEFI reserved regions as MEMBLOCK_NOMAP git bisect good 4dffbfc48d65e5d8157a634fd670065d237a9377 # good: [1289ace5b4f70f1e68ce785735b82c7e483de863] Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi git bisect good 1289ace5b4f70f1e68ce785735b82c7e483de863 # good: [984065055e6e39f8dd812529e11922374bd39352] Merge branch 'drm-next' of git://people.freedesktop.org/~airlied/linux git bisect good 984065055e6e39f8dd812529e11922374bd39352 # good: [6d1c244803f2c013fb9c31b0904c01f1830b73ab] Merge tag 'armsoc-dt' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc git bisect good 6d1c244803f2c013fb9c31b0904c01f1830b73ab # bad: [0a13daedf7ffc71b0c374a036355da7fddb20d6d] Merge branch 'for-4.5/lightnvm' of git://git.kernel.dk/linux-block git bisect bad 0a13daedf7ffc71b0c374a036355da7fddb20d6d # bad: [278e5acae1321978686e85ca92906054a36aa19b] Merge tag 'for-4.5' of git://git.osdn.jp/gitroot/uclinux-h8/linux git bisect bad 278e5acae1321978686e85ca92906054a36aa19b # good: [f9cd69fe5eb6347b4de56458d0378bc0fa44bce9] Merge tag 'armsoc-defconfig' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc git bisect good f9cd69fe5eb6347b4de56458d0378bc0fa44bce9 # good: [9638685e32af961943b679fcb72d4ddd458eb18f] Merge tag 'armsoc-drivers' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc git bisect good 9638685e32af961943b679fcb72d4ddd458eb18f # good: [fa8bb4518771b19460a318fbab3eb36c81db3a50] Merge branch 'pm-devfreq' git bisect good fa8bb4518771b19460a318fbab3eb36c81db3a50 # bad: [30f05309bde49295e02e45c7e615f73aa4e0ccc2] Merge tag 'pm+acpi-4.5-rc1-2' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm git bisect bad 30f05309bde49295e02e45c7e615f73aa4e0ccc2 # good: [5bb1729cbdfbe974ad6385be94b14afbac97e19f] cpuidle: menu: Avoid pointless checks in menu_select() git bisect good 5bb1729cbdfbe974ad6385be94b14afbac97e19f # bad: [db2b52f75250c88ee3c6ba3d91bef38f3f1a1e8c] Merge branch 'pm-tools' git bisect bad db2b52f75250c88ee3c6ba3d91bef38f3f1a1e8c # good: [38cb76a307821f76c7f9dff7449f73aeb014d5cc] cpupower: Fix build error in cpufreq-info git bisect good 38cb76a307821f76c7f9dff7449f73aeb014d5cc # bad: [f11aef69b235bc30c323776d75ac23b43aac45bb] Merge branch 'pm-cpuidle' git bisect bad f11aef69b235bc30c323776d75ac23b43aac45bb # first bad commit: [f11aef69b235bc30c323776d75ac23b43aac45bb] Merge branch 'pm-cpuidle' Thanks, Mark. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Tue, Feb 16, 2016 at 03:59:09PM +0300, Andrey Ryabinin wrote: > Actually, the first report is a bit more useful. It shows that shadow memory was corrupted: > > ffffffc93665bc00: f1 f1 f1 f1 00 f4 f4 f4 f2 f2 f2 f2 00 00 f1 f1 > > ffffffc93665bc80: f1 f1 00 00 00 00 f3 f3 00 f4 f4 f4 f3 f3 f3 f3 > ^ > F1 - left redzone, it indicates start of stack frame > F3 - right redzone, it should be the end of stack frame. > > But here we have the second set of F1s without F3s which should close the first set of F1s. > Also those two F3s in the middle cannot be right. > > So shadow is corrupted. > Some hypotheses: > 2) Shadow memory wasn't cleared. GCC poison memory on function entrance and unpoisons it before return. > If we use some tricky way to exit from function this could cause false-positives like that. > E.g. some hand-written assembly return code. I think this is what's happenening, at least for the idle case. A second attempt at bisecting led me to commit e679660dbb8347f2 ("ARM: 8481/2: drivers: psci: replace psci firmware calls"). Reverting that makes v4.5-rc1 boot without KASAN splats. That patch turned __invoke_psci_fn_{smc,hvc} into (ASAN-instrumented) C functions. Prior to that commit, __invoke_psci_fn_{smc,hvc} were pure assembly functions which used no stack. When we go down for idle, in __cpu_suspend_enter we stash some context to the stack (in assembly). The CPU may return from a cold state via cpu_resume, where we restore context from the stack. However, after storing the context we call psci_suspend_finisher, which calls psci_cpu_suspend, which calls invoke_psci_fn_*. As psci_cpu_suspend and invoke_psci_fn_* are instrumented, they poison memory on function entrance, but we never perform the unpoisoning. That was always the case for psci_suspend_finisher, so there was a latent issue that we were somehow avoiding. Perhaps we got luck with stack layout and never hit the poison. I'm not sure how we fix that, as invoke_psci_fn_* may or may not return for arbitrary reasons (e.g. a CPU_SUSPEND_CALL may or may not return depending on whether an interrupt comes in at the right time). Perhaps the simplest option is to not instrument invoke_psci_fn_* and psci_suspend_finisher. Do we have a per-function annotation to avoid KASAN instrumentation, like notrace? I need to investigate, but we may also need notrace for similar reasons. Andrey, on a tangential note, what do we do around hotplug? I assume that we must unpooison the shadow region for the stack of a dead CPU, but I wasn't able to figure out where we do that. Hopefuly we're not just getting lucky? Thanks, Mark. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Wed, Feb 17, 2016 at 07:31:43PM +0300, Andrey Ryabinin wrote: > On 02/17/2016 05:39 PM, Mark Rutland wrote: > > Andrey, on a tangential note, what do we do around hotplug? I assume > > that we must unpooison the shadow region for the stack of a dead CPU, > > but I wasn't able to figure out where we do that. Hopefuly we're not > > just getting lucky? > > We do nothing about it. AFAIU we need to clear swapper's stack, > somewhere in secondary_start_kernel() perhaps. Oh, joy... Surely other architectures (e.g. x86) will need to do something similar? Do they do anything currently? I can't see that they do... Mark. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
diff --git i/arch/arm64/Kconfig w/arch/arm64/Kconfig index 8cc62289a63e..fdd1d75f5bad 100644 --- i/arch/arm64/Kconfig +++ w/arch/arm64/Kconfig @@ -9,6 +9,7 @@ config ARM64 select ARCH_HAS_GCOV_PROFILE_ALL select ARCH_HAS_SG_CHAIN select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST + select ARCH_HAS_UBSAN_SANITIZE_ALL select ARCH_USE_CMPXCHG_LOCKREF select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_WANT_OPTIONAL_GPIOLIB diff --git i/arch/arm64/configs/defconfig w/arch/arm64/configs/defconfig index 86581f793e39..0006b0204b97 100644 --- i/arch/arm64/configs/defconfig +++ w/arch/arm64/configs/defconfig @@ -240,11 +240,14 @@ CONFIG_DEBUG_INFO=y CONFIG_DEBUG_FS=y CONFIG_MAGIC_SYSRQ=y CONFIG_DEBUG_KERNEL=y +CONFIG_KASAN=y +CONFIG_TEST_KASAN=m CONFIG_LOCKUP_DETECTOR=y # CONFIG_SCHED_DEBUG is not set # CONFIG_DEBUG_PREEMPT is not set # CONFIG_FTRACE is not set CONFIG_MEMTEST=y +CONFIG_UBSAN=y CONFIG_SECURITY=y CONFIG_CRYPTO_ECHAINIV=y CONFIG_CRYPTO_ANSI_CPRNG=y