| Message ID | 43dfaba0646d498fe94c1a8479b812346133f438.1646765290.git.darren@os.amperecomputing.com |
|---|---|
| State | Accepted |
| Commit | 3f8dec116210ca649163574ed5f8df1e3b837d07 |
| Headers | show |
| Series | ACPI/APEI: Limit printable size of BERT table data | expand |
On Tue, Mar 8, 2022 at 7:51 PM Darren Hart <darren@os.amperecomputing.com> wrote: > > Platforms with large BERT table data can trigger soft lockup errors > while attempting to print the entire BERT table data to the console at > boot: > > watchdog: BUG: soft lockup - CPU#160 stuck for 23s! [swapper/0:1] > > Observed on Ampere Altra systems with a single BERT record of ~250KB. > > The original bert driver appears to have assumed relatively small table > data. Since it is impractical to reassemble large table data from > interwoven console messages, and the table data is available in > > /sys/firmware/acpi/tables/data/BERT > > limit the size for tables printed to the console to 1024 (for no reason > other than it seemed like a good place to kick off the discussion, would > appreciate feedback from existing users in terms of what size would > maintain their current usage model). > > Alternatively, we could make printing a CONFIG option, use the > bert_disable boot arg (or something similar), or use a debug log level. > However, all those solutions require extra steps or change the existing > behavior for small table data. Limiting the size preserves existing > behavior on existing platforms with small table data, and eliminates the > soft lockups for platforms with large table data, while still making it > available. > > Cc: "Rafael J. Wysocki" <rafael@kernel.org> > Cc: Len Brown <lenb@kernel.org> > Cc: James Morse <james.morse@arm.com> > Cc: Tony Luck <tony.luck@intel.com> > Cc: Borislav Petkov <bp@alien8.de> > Cc: Doug Rady <dcrady@os.amperecomputing.com> > Signed-off-by: Darren Hart <darren@os.amperecomputing.com> Not that I have a particularly strong opinion here, but this looks reasonable to me, so I've queued it up for 5.18. APEI reviewers, please chime in if you disagree with the above. > --- > drivers/acpi/apei/bert.c | 8 ++++++-- > 1 file changed, 6 insertions(+), 2 deletions(-) > > diff --git a/drivers/acpi/apei/bert.c b/drivers/acpi/apei/bert.c > index 19e50fcbf4d6..ad8ab3f12cf3 100644 > --- a/drivers/acpi/apei/bert.c > +++ b/drivers/acpi/apei/bert.c > @@ -29,6 +29,7 @@ > > #undef pr_fmt > #define pr_fmt(fmt) "BERT: " fmt > +#define ACPI_BERT_PRINT_MAX_LEN 1024 > > static int bert_disable; > > @@ -58,8 +59,11 @@ static void __init bert_print_all(struct acpi_bert_region *region, > } > > pr_info_once("Error records from previous boot:\n"); > - > - cper_estatus_print(KERN_INFO HW_ERR, estatus); > + if (region_len < ACPI_BERT_PRINT_MAX_LEN) > + cper_estatus_print(KERN_INFO HW_ERR, estatus); > + else > + pr_info_once("Max print length exceeded, table data is available at:\n" > + "/sys/firmware/acpi/tables/data/BERT"); > > /* > * Because the boot error source is "one-time polled" type, > -- > 2.31.1 >
On Wed, Mar 09, 2022 at 07:42:26PM +0100, Rafael J. Wysocki wrote: > On Tue, Mar 8, 2022 at 7:51 PM Darren Hart > Not that I have a particularly strong opinion here, but this looks > reasonable to me, so I've queued it up for 5.18. > > APEI reviewers, please chime in if you disagree with the above. It looked reasonable to me when I skimmed it in March. But the reality check now needs cashing because some validation team here is complaining that they don't see any errors printed from their BERT tests. :-( So I looked again. This test inside the loop seems bogus: if (region_len < ACPI_BERT_PRINT_MAX_LEN) { because "region_len" isn't updated inside the loop. If it is too big then it will prevent Linux from printing any/all of the records in the BERT table (and the test could have been done before the loop). Maybe below patch is better? It avoids printing individual CPER records that are too large (checking estatus_len instead of region_len). I also added a limit to how many records to print (I randomly picked "5" as the limit ... the specific failing test only want to print one). -Tony [I will write up a proper commit message and add a Signed-off-by if this looks to be a reasonable direction] diff --git a/drivers/acpi/apei/bert.c b/drivers/acpi/apei/bert.c index 598fd19b65fa..4e894a728c02 100644 --- a/drivers/acpi/apei/bert.c +++ b/drivers/acpi/apei/bert.c @@ -29,6 +29,8 @@ #undef pr_fmt #define pr_fmt(fmt) "BERT: " fmt + +#define ACPI_BERT_PRINT_MAX_RECORDS 5 #define ACPI_BERT_PRINT_MAX_LEN 1024 static int bert_disable; @@ -39,6 +41,7 @@ static void __init bert_print_all(struct acpi_bert_region *region, struct acpi_hest_generic_status *estatus = (struct acpi_hest_generic_status *)region; int remain = region_len; + int ncper = 0, skipped = 0; u32 estatus_len; while (remain >= sizeof(struct acpi_bert_region)) { @@ -46,24 +49,23 @@ static void __init bert_print_all(struct acpi_bert_region *region, if (remain < estatus_len) { pr_err(FW_BUG "Truncated status block (length: %u).\n", estatus_len); - return; + break; } /* No more error records. */ if (!estatus->block_status) - return; + break; if (cper_estatus_check(estatus)) { pr_err(FW_BUG "Invalid error record.\n"); - return; + break; } pr_info_once("Error records from previous boot:\n"); - if (region_len < ACPI_BERT_PRINT_MAX_LEN) + if (ncper++ < ACPI_BERT_PRINT_MAX_RECORDS && estatus_len < ACPI_BERT_PRINT_MAX_LEN) cper_estatus_print(KERN_INFO HW_ERR, estatus); else - pr_info_once("Max print length exceeded, table data is available at:\n" - "/sys/firmware/acpi/tables/data/BERT"); + skipped++; /* * Because the boot error source is "one-time polled" type, @@ -75,6 +77,9 @@ static void __init bert_print_all(struct acpi_bert_region *region, estatus = (void *)estatus + estatus_len; remain -= estatus_len; } + if (skipped) + pr_info("Skipped %d error records, full table data is available at:\n" + "/sys/firmware/acpi/tables/data/BERT", skipped); } static int __init setup_bert_disable(char *str)
diff --git a/drivers/acpi/apei/bert.c b/drivers/acpi/apei/bert.c index 19e50fcbf4d6..ad8ab3f12cf3 100644 --- a/drivers/acpi/apei/bert.c +++ b/drivers/acpi/apei/bert.c @@ -29,6 +29,7 @@ #undef pr_fmt #define pr_fmt(fmt) "BERT: " fmt +#define ACPI_BERT_PRINT_MAX_LEN 1024 static int bert_disable; @@ -58,8 +59,11 @@ static void __init bert_print_all(struct acpi_bert_region *region, } pr_info_once("Error records from previous boot:\n"); - - cper_estatus_print(KERN_INFO HW_ERR, estatus); + if (region_len < ACPI_BERT_PRINT_MAX_LEN) + cper_estatus_print(KERN_INFO HW_ERR, estatus); + else + pr_info_once("Max print length exceeded, table data is available at:\n" + "/sys/firmware/acpi/tables/data/BERT"); /* * Because the boot error source is "one-time polled" type,
Platforms with large BERT table data can trigger soft lockup errors while attempting to print the entire BERT table data to the console at boot: watchdog: BUG: soft lockup - CPU#160 stuck for 23s! [swapper/0:1] Observed on Ampere Altra systems with a single BERT record of ~250KB. The original bert driver appears to have assumed relatively small table data. Since it is impractical to reassemble large table data from interwoven console messages, and the table data is available in /sys/firmware/acpi/tables/data/BERT limit the size for tables printed to the console to 1024 (for no reason other than it seemed like a good place to kick off the discussion, would appreciate feedback from existing users in terms of what size would maintain their current usage model). Alternatively, we could make printing a CONFIG option, use the bert_disable boot arg (or something similar), or use a debug log level. However, all those solutions require extra steps or change the existing behavior for small table data. Limiting the size preserves existing behavior on existing platforms with small table data, and eliminates the soft lockups for platforms with large table data, while still making it available. Cc: "Rafael J. Wysocki" <rafael@kernel.org> Cc: Len Brown <lenb@kernel.org> Cc: James Morse <james.morse@arm.com> Cc: Tony Luck <tony.luck@intel.com> Cc: Borislav Petkov <bp@alien8.de> Cc: Doug Rady <dcrady@os.amperecomputing.com> Signed-off-by: Darren Hart <darren@os.amperecomputing.com> --- drivers/acpi/apei/bert.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)