From patchwork Mon Dec 5 20:12:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 631182 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 525B5C4321E for ; Mon, 5 Dec 2022 20:12:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232724AbiLEUM0 (ORCPT ); Mon, 5 Dec 2022 15:12:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52972 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232788AbiLEUMY (ORCPT ); Mon, 5 Dec 2022 15:12:24 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9883F275E8 for ; Mon, 5 Dec 2022 12:12:23 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 50D60B81212 for ; Mon, 5 Dec 2022 20:12:22 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6C4C4C433D7; Mon, 5 Dec 2022 20:12:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1670271141; bh=YmexWb2wA2VRGsLRvQxNTHg2UZC1lXNmFZOrLUB5d4E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Dd50h/1iawL+iuPA6ZwAzWdr/7HqiDAWWwRuDCtUq3+XmupQJQchpExGHgtd9qRcA 517T5kzcgGEu+h8vuaR7m5IdXxbWmfFhpKk/b6dSClEEvksq1FUxGxeyCawW1oV84y HrMv6EuXL7otNd2Gr7q9Wx6W9ScEdZxl0dC0YraglkLBjvRwlD+EOFFVR0WPPz4GLv 18yfNtFZi4bmVNL6zGg6E/UpmcaZMwiOvGii3PnwATJdRB9qMNEniq6BKMm1Wo0kXi 36NESD9h6LYxlY4NIB80mNt/xsIdSb1bD38KemYv/a6xHgR4XxTSqOYVwwRxFQQo6a 8v5DsBF0WvkCA== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, Ard Biesheuvel , Sami Tolvanen , Kees Cook Subject: [PATCH 1/2] arm64: efi: Execute runtime services from a dedicated stack Date: Mon, 5 Dec 2022 21:12:09 +0100 Message-Id: <20221205201210.463781-2-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221205201210.463781-1-ardb@kernel.org> References: <20221205201210.463781-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4194; i=ardb@kernel.org; h=from:subject; bh=YmexWb2wA2VRGsLRvQxNTHg2UZC1lXNmFZOrLUB5d4E=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjjlCY1HWUzLnlv1k1WgzcWLtByrBOxwVGaGSKc3jy YHObtZGJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY45QmAAKCRDDTyI5ktmPJItwC/ 9zrnYKpLdJRUJ2wAfoKuizXY17XfFCrE+7N2OpM4xp60KHwK0mYEXHX8cMfZtbyKfuonoRsaQjTbDf gpwop6siIzgdl5/k/w3KcCiQvaTlFLPSkZjq67UgP1JPi2bUhU2cUVO34NaQlX0TNKarkqazfG9yWS g9DHjeLbL4ELm58g+VbLmZmc+eK1uHGrqSRxCr6KqSVHRF44yVU0EpUgueKHJjT+XRjmLlCst9958f Zup4SWlK/4+ybaLMvzIGeM1iNTZQjgutDyslF6CvCFPlxhib3oj5KGzi2UE1Daf/nh2P3d4pXtbm8p HPcNESya6K7+9E7hpf3v1TkirtcgzcK7MEnmHRelXC0l999qfIuSPb5H07jzSYK9M7E4DtYqDfOh0I iAnCwfXFALtESRq5yvoqmUC4LM3P3bIfm6kDUvN3p8eBKp1pJtCURISg7tsKkC9+yKg2zOxGYPcxaT dTLEt2vQRmEAmcIQHbmcLtcUx1Y1TNbrsmj845tPJC6+A= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org With the introduction of PRMT in the ACPI subsystem, the EFI rts workqueue is no longer the only caller of efi_call_virt_pointer() in the kernel. This means the EFI runtime services lock is no longer sufficient to manage concurrent calls into firmware, but also that firmware calls may occur that are not marshalled via the workqueue mechanism, but originate directly from the caller context. For added robustness, and to ensure that the runtime services have 8 KiB of stack space available as per the EFI spec, introduce a spinlock protected EFI runtime stack of 8 KiB, where the spinlock also ensures serialization between the EFI rts workqueue (which itself serializes EFI runtime calls) and other callers of efi_call_virt_pointer(). While at it, use the stack pivot to avoid reloading the shadow call stack pointer from the ordinary stack, as doing so could produce a gadget to defeat it. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/efi.h | 3 +++ arch/arm64/kernel/efi-rt-wrapper.S | 13 +++++++++- arch/arm64/kernel/efi.c | 25 ++++++++++++++++++++ 3 files changed, 40 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h index 7c12e01c2b312e7b..1c408ec3c8b3a883 100644 --- a/arch/arm64/include/asm/efi.h +++ b/arch/arm64/include/asm/efi.h @@ -25,6 +25,7 @@ int efi_set_mapping_permissions(struct mm_struct *mm, efi_memory_desc_t *md); ({ \ efi_virtmap_load(); \ __efi_fpsimd_begin(); \ + spin_lock(&efi_rt_lock); \ }) #undef arch_efi_call_virt @@ -33,10 +34,12 @@ int efi_set_mapping_permissions(struct mm_struct *mm, efi_memory_desc_t *md); #define arch_efi_call_virt_teardown() \ ({ \ + spin_unlock(&efi_rt_lock); \ __efi_fpsimd_end(); \ efi_virtmap_unload(); \ }) +extern spinlock_t efi_rt_lock; efi_status_t __efi_rt_asm_wrapper(void *, const char *, ...); #define ARCH_EFI_IRQ_FLAGS_MASK (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT) diff --git a/arch/arm64/kernel/efi-rt-wrapper.S b/arch/arm64/kernel/efi-rt-wrapper.S index 75691a2641c1c0f8..b2786b968fee68dd 100644 --- a/arch/arm64/kernel/efi-rt-wrapper.S +++ b/arch/arm64/kernel/efi-rt-wrapper.S @@ -16,6 +16,12 @@ SYM_FUNC_START(__efi_rt_asm_wrapper) */ stp x1, x18, [sp, #16] + ldr_l x16, efi_rt_stack_top + mov sp, x16 +#ifdef CONFIG_SHADOW_CALL_STACK + str x18, [sp, #-16]! +#endif + /* * We are lucky enough that no EFI runtime services take more than * 5 arguments, so all are passed in registers rather than via the @@ -29,6 +35,7 @@ SYM_FUNC_START(__efi_rt_asm_wrapper) mov x4, x6 blr x8 + mov sp, x29 ldp x1, x2, [sp, #16] cmp x2, x18 ldp x29, x30, [sp], #32 @@ -42,6 +49,10 @@ SYM_FUNC_START(__efi_rt_asm_wrapper) * called with preemption disabled and a separate shadow stack is used * for interrupts. */ - mov x18, x2 +#ifdef CONFIG_SHADOW_CALL_STACK + ldr_l x18, efi_rt_stack_top + ldr x18, [x18, #-16] +#endif + b efi_handle_corrupted_x18 // tail call SYM_FUNC_END(__efi_rt_asm_wrapper) diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index a908a37f03678b6b..8cb2e005f8aca589 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -144,3 +144,28 @@ asmlinkage efi_status_t efi_handle_corrupted_x18(efi_status_t s, const char *f) pr_err_ratelimited(FW_BUG "register x18 corrupted by EFI %s\n", f); return s; } + +DEFINE_SPINLOCK(efi_rt_lock); + +asmlinkage u64 *efi_rt_stack_top __ro_after_init; + +/* required by the EFI spec */ +static_assert(THREAD_SIZE >= SZ_8K); + +int __init arm64_efi_rt_init(void) +{ + void *p = __vmalloc_node_range(THREAD_SIZE, THREAD_ALIGN, + VMALLOC_START, VMALLOC_END, GFP_KERNEL, + PAGE_KERNEL, 0, NUMA_NO_NODE, + __builtin_return_address(0)); + + if (!p) { + pr_warn("Failed to allocate EFI runtime stack\n"); + clear_bit(EFI_RUNTIME_SERVICES, &efi.flags); + return -ENOMEM; + } + + efi_rt_stack_top = p + THREAD_SIZE; + return 0; +} +core_initcall(arm64_efi_rt_init); From patchwork Mon Dec 5 20:12:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 631428 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13B29C47089 for ; Mon, 5 Dec 2022 20:12:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232272AbiLEUM0 (ORCPT ); Mon, 5 Dec 2022 15:12:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232792AbiLEUMZ (ORCPT ); Mon, 5 Dec 2022 15:12:25 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2B78127173 for ; Mon, 5 Dec 2022 12:12:24 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id AD0EB61307 for ; Mon, 5 Dec 2022 20:12:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 78E55C433D6; Mon, 5 Dec 2022 20:12:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1670271143; bh=goSU823T86J6kjDXvEC7WqXZhLUmNmBC0euU4f+9sqs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pabG0PJoEcvA1GNYvmeWie0JzlyOvvYvjlMpOv1O+MZx+Fcj0h5TkDOVLsZbJLJlF bmUD7/Ll51YvT7DtXdlBLm6bIvTZfeloy/iLOZVoHm0O3rtzL6hzqMrvsP/JP6Rexx LMisBcF0NiGJvzG2PEMN6yrcDmMwsm4RPRGLOK0zdB5rI4rnguy1DMyNjI/xlpLIZG VmaygdjHbnho7Ig/rcofsV6CDmsma7lFPjmLQcTZ2gZmsBF/4JVpsT7eWWyEWOnws7 c8BXJc1bghpSk7UGQfk2G3keMyby4JUamptIgkTQUuH/gRfU6JFJcqTv62fH8oyvNP PkNSIbNTVZIdw== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, Ard Biesheuvel , Sami Tolvanen , Kees Cook Subject: [PATCH 2/2] arm64: efi: Recover from synchronous exceptions occurring in firmware Date: Mon, 5 Dec 2022 21:12:10 +0100 Message-Id: <20221205201210.463781-3-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221205201210.463781-1-ardb@kernel.org> References: <20221205201210.463781-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6103; i=ardb@kernel.org; h=from:subject; bh=goSU823T86J6kjDXvEC7WqXZhLUmNmBC0euU4f+9sqs=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjjlCZNXpWHY902aRBpsDx0WD587MJ0qaBcvGASIqi s6a+AGyJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY45QmQAKCRDDTyI5ktmPJEXjDA Cxjw9Hj4lduImg48FLjV21xlNgYnwOgZqEaZ8XrxM5aQqiNus3mMMI0IjaYlc9+rv3EAhcK57e9AFJ YY8WiIHJAufFtFnAPDBIqOdNJ+Vnqs4ZLFdhpzSj++bMI+MAQkTSlixKtSQKTWuoTQ2HE6MpNfYQX1 XZyH89ZaPvHx1kQYfMacsDO6lE3smrdMKvz1AR40vf9faTQ+97h6yh0arqLJM7/H31MYu7NxhqWmm2 0/QUPwvYe5z9ABhEqAMq9ktPFB7UJsCD8Qr1tU5Ei9UPLQ8b4q7SG6ogwxyNoyvW2CtC0uyuS8eVjV fXUWel0d69VjCApES2O3OhZcHj1Cdf935AOAA3AezZ6K/ADN2Jca5Ppm2GjjzB5I/NtZ5OMZ47DnHz 3j+G9EwM0eNMOlBmFFabjXImNobST9kVMUbrKBoT0oTDJdkCUdivXU4LkSkHa3NNgyPzokMLkqPKCT uCjOKgnC5W9hVsAhw0VHPOR2rZdXAoIEloyjPfhYRlYwQ= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org Unlike x86, which has machinery to deal with page faults that occur during the execution of EFI runtime services, arm64 has nothing like that, and a synchronous exception raised by firmware code brings down the whole system. With more EFI based systems appearing that were not built to run Linux (such as the Windows-on-ARM laptops based on Qualcomm SOCs), as well as the introduction of PRM (platform specific firmware routines that are callable just like EFI runtime services), we are more likely to run into issues of this sort, and it is much more likely that we can identify and work around such issues if they don't bring down the system entirely. Since we already use a EFI runtime services call wrapper in assembler, we can quite easily add some code that captures the execution state at the point where the call is made, allowing us to revert to this state and proceed execution if the call triggered a synchronous exception. Given that the kernel and the firmware don't share any data structures that could end up in an indeterminate state, we can happily continue running, as long as we mark the EFI runtime services as unavailable from that point on. Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- arch/arm64/include/asm/efi.h | 8 +++++ arch/arm64/kernel/efi-rt-wrapper.S | 31 ++++++++++++++++---- arch/arm64/kernel/efi.c | 22 ++++++++++++++ arch/arm64/mm/fault.c | 4 +++ drivers/firmware/efi/runtime-wrappers.c | 1 + 5 files changed, 61 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h index 1c408ec3c8b3a883..31d13a6001df49c4 100644 --- a/arch/arm64/include/asm/efi.h +++ b/arch/arm64/include/asm/efi.h @@ -14,8 +14,16 @@ #ifdef CONFIG_EFI extern void efi_init(void); + +bool efi_runtime_fixup_exception(struct pt_regs *regs, const char *msg); #else #define efi_init() + +static inline +bool efi_runtime_fixup_exception(struct pt_regs *regs, const char *msg) +{ + return false; +} #endif int efi_create_mapping(struct mm_struct *mm, efi_memory_desc_t *md); diff --git a/arch/arm64/kernel/efi-rt-wrapper.S b/arch/arm64/kernel/efi-rt-wrapper.S index b2786b968fee68dd..ee0c6c719ab573b7 100644 --- a/arch/arm64/kernel/efi-rt-wrapper.S +++ b/arch/arm64/kernel/efi-rt-wrapper.S @@ -6,7 +6,7 @@ #include SYM_FUNC_START(__efi_rt_asm_wrapper) - stp x29, x30, [sp, #-32]! + stp x29, x30, [sp, #-112]! mov x29, sp /* @@ -16,11 +16,20 @@ SYM_FUNC_START(__efi_rt_asm_wrapper) */ stp x1, x18, [sp, #16] + /* + * Preserve all callee saved registers and record the stack pointer + * value in a per-CPU variable so we can recover from synchronous + * exceptions occurring while running the firmware routines. + */ + stp x19, x20, [sp, #32] + stp x21, x22, [sp, #48] + stp x23, x24, [sp, #64] + stp x25, x26, [sp, #80] + stp x27, x28, [sp, #96] + ldr_l x16, efi_rt_stack_top mov sp, x16 -#ifdef CONFIG_SHADOW_CALL_STACK - str x18, [sp, #-16]! -#endif + stp x18, x29, [sp, #-16]! /* * We are lucky enough that no EFI runtime services take more than @@ -38,7 +47,7 @@ SYM_FUNC_START(__efi_rt_asm_wrapper) mov sp, x29 ldp x1, x2, [sp, #16] cmp x2, x18 - ldp x29, x30, [sp], #32 + ldp x29, x30, [sp], #112 b.ne 0f ret 0: @@ -56,3 +65,15 @@ SYM_FUNC_START(__efi_rt_asm_wrapper) b efi_handle_corrupted_x18 // tail call SYM_FUNC_END(__efi_rt_asm_wrapper) + +SYM_CODE_START(__efi_rt_asm_recover) + mov sp, x30 + + ldp x19, x20, [sp, #32] + ldp x21, x22, [sp, #48] + ldp x23, x24, [sp, #64] + ldp x25, x26, [sp, #80] + ldp x27, x28, [sp, #96] + ldp x29, x30, [sp], #112 + ret +SYM_CODE_END(__efi_rt_asm_recover) diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index 8cb2e005f8aca589..0169a669fde7544f 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -149,6 +149,28 @@ DEFINE_SPINLOCK(efi_rt_lock); asmlinkage u64 *efi_rt_stack_top __ro_after_init; +asmlinkage efi_status_t __efi_rt_asm_recover(void); + +bool efi_runtime_fixup_exception(struct pt_regs *regs, const char *msg) +{ + /* Check whether the exception occurred while running the firmware */ + if (current_work() != &efi_rts_work.work || regs->pc >= TASK_SIZE_64) + return false; + + pr_err(FW_BUG "Unable to handle %s in EFI runtime service\n", msg); + add_taint(TAINT_FIRMWARE_WORKAROUND, LOCKDEP_STILL_OK); + clear_bit(EFI_RUNTIME_SERVICES, &efi.flags); + + regs->regs[0] = EFI_ABORTED; + regs->regs[30] = efi_rt_stack_top[-1]; + regs->pc = (u64)__efi_rt_asm_recover; + + if (IS_ENABLED(CONFIG_SHADOW_CALL_STACK)) + regs->regs[18] = efi_rt_stack_top[-2]; + + return true; +} + /* required by the EFI spec */ static_assert(THREAD_SIZE >= SZ_8K); diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 5b391490e045be91..3e9cf9826417a434 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include #include @@ -391,6 +392,9 @@ static void __do_kernel_fault(unsigned long addr, unsigned long esr, msg = "paging request"; } + if (efi_runtime_fixup_exception(regs, msg)) + return; + die_kernel_fault(msg, addr, esr, regs); } diff --git a/drivers/firmware/efi/runtime-wrappers.c b/drivers/firmware/efi/runtime-wrappers.c index f3e54f6616f02475..7feee3d9c2bfbeec 100644 --- a/drivers/firmware/efi/runtime-wrappers.c +++ b/drivers/firmware/efi/runtime-wrappers.c @@ -83,6 +83,7 @@ struct efi_runtime_work efi_rts_work; else \ pr_err("Failed to queue work to efi_rts_wq.\n"); \ \ + WARN_ON_ONCE(efi_rts_work.status == EFI_ABORTED); \ exit: \ efi_rts_work.efi_rts_id = EFI_NONE; \ efi_rts_work.status; \