From patchwork Sun May 4 09:52:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887284 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0432019DF66 for ; Sun, 4 May 2025 09:53:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352389; cv=none; b=P8dncO3cdnFMAovlrcHtuN3qKeofsrrvxO5N4Lfy6U+gLr6Ha49rkg0LJdvrOIBvoP8rUY639H06c6LpACNdsRf7kSHsZUiIanrnGoOY/6g6ZHVkz51vW7Mb2h7Hzxj6tLuyH4AKHIWvk1FjTlWmAFz5HjTEkK4qgWaEHy2g9xo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352389; c=relaxed/simple; bh=+FyIvsPsWVXVLkDD32Yax8szrjhp6N3YAfdKHzYClww=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ixX83UEBUM1HMH4TeYeqMb0bXM4WfIyODGtOf8rAdN5coPsHTOYBMX5HUnmGmG/JkccdVHqBHk3XXhMIL1b8ofQoML6R/DRrNHY7UTcd+iS4fy6pI8voRTVQ8+ciN24AgoCKYosxzBRcnUjKnlNqjKqEW03lVNurRM6LslwMOb0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=0d611bLO; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="0d611bLO" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-43d0830c3f7so24969325e9.2 for ; Sun, 04 May 2025 02:53:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352386; x=1746957186; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=edyQoTvfRG8x1DOua7oEfmK1XYQ+Ovhdt7C5cZkBAXA=; b=0d611bLOwIGwb+rkyLn67HBGwaRaCShLUW/TxHloM0So4v4VlG+ux+8D2HtSDwTfxH 8hxvQ0j6f5itAy2CwDnlE6SPNPDvx5X8g86FZBwwdkzipRsn9CLTGHyg96rAj6DkbhRa Ye9sdOlajN0XjTe4Di/BStTjJyoiUkVcqFy55fHMKs386Rx715aR8rMM66GeKP1thx86 KVYdkUL3+3G4MqcuoG8PYTiNRBYuVeTXE1k0HQtSdWQS94WNh1z3VJ5Wal90Pl7r9h3A g1E9nkZM9y92rQe7Z5rIUOaMC2PUXhVNrImVNorCmmh93OYy25x88vO12SdbutRo6uyo UbrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352386; x=1746957186; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=edyQoTvfRG8x1DOua7oEfmK1XYQ+Ovhdt7C5cZkBAXA=; b=Z4jwvxNR6rkJpPtNYupOib1Z+x9qbC757gaT+pmxKcdBjxH0bKc9BGb547vg4XBQEj 16WJJFqW6hHLC0LwhDvAxJ7WlnPvwXI6z5q0obk1SXcgUlzoicTsgk4XyTrzj8OM4Orb gOoWNxGrQc8lqfcGWTEG9r7E01QdrxZIjXF9bSmkO0ojg66Uuw85z5nXCL6xkeAwxiA0 QazkjDaL5JrLXEA9CKcg6tJffhdD67GbvjldyxQOCkgc+GJ3cHu8jq0mCrHiiEWqMSLo 0WMrgd1jeSkUMLvjAjmEG2i76GEaC/gzhxLitf/Yp4V+HzmsgjTKJyKf7QJnmKAasd55 fuJw== X-Gm-Message-State: AOJu0Yxif/v9lF93gWZy06YxZBZQBq3jf0VFrtFzH3Pfv0zQkl3BdQ1s VzJeHqSNkQAZCP6ZpTdqsLDWIq1Ce5lvudhYDxkx+uHpcjqCGIpZtkGhWEc1Bm0zwLnAwA== X-Google-Smtp-Source: AGHT+IGUfyOfLV6G6UR2fsX+wLERnuEA+YabEo3i5wYRA1IwhxBQ8DRNG3LYswYRLLnc5hLaEzfYvnCv X-Received: from wmbbi7.prod.google.com ([2002:a05:600c:3d87:b0:441:b3f0:c46]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:5027:b0:441:b698:3431 with SMTP id 5b1f17b1804b1-441c4948315mr24664205e9.28.1746352386480; Sun, 04 May 2025 02:53:06 -0700 (PDT) Date: Sun, 4 May 2025 11:52:31 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2866; i=ardb@kernel.org; h=from:subject; bh=RC+wmdw3k8C0pDFZiD1ezLCeDtzkHNknaU3PXMx+JFM=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4sH3NckcFxmUDsnu3X5/w/TPj30UpGdFM3gEvDD+2 HZQkf12RykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjIKhVGhv4XX1anOf22nWb5 JvE6285joe1FHdwnymx9LiX/9lh2JI7hv6+I3Ean3bMyV3IVdv6Z5G00jzPYmqln412xPu3XbaY fOQA= X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-26-ardb+git@google.com> Subject: [RFT PATCH v2 01/23] x86/boot: Move early_setup_gdt() back into head64.c From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel Move early_setup_gdt() out of the startup code that is callable from the 1:1 mapping - this is not needed, and instead, it is better to expose the helper that does reside in __head directly. This reduces the amount of code that needs special checks for 1:1 execution suitability. In particular, it avoids dealing with the GHCB page (and its physical address) in startup code, which runs from the 1:1 mapping, making physical to virtual translations ambiguous. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/startup/gdt_idt.c | 15 +-------------- arch/x86/include/asm/setup.h | 1 + arch/x86/kernel/head64.c | 12 ++++++++++++ 3 files changed, 14 insertions(+), 14 deletions(-) diff --git a/arch/x86/boot/startup/gdt_idt.c b/arch/x86/boot/startup/gdt_idt.c index 7e34d0b426b1..a3112a69b06a 100644 --- a/arch/x86/boot/startup/gdt_idt.c +++ b/arch/x86/boot/startup/gdt_idt.c @@ -24,7 +24,7 @@ static gate_desc bringup_idt_table[NUM_EXCEPTION_VECTORS] __page_aligned_data; /* This may run while still in the direct mapping */ -static void __head startup_64_load_idt(void *vc_handler) +void __head startup_64_load_idt(void *vc_handler) { struct desc_ptr desc = { .address = (unsigned long)rip_rel_ptr(bringup_idt_table), @@ -43,19 +43,6 @@ static void __head startup_64_load_idt(void *vc_handler) native_load_idt(&desc); } -/* This is used when running on kernel addresses */ -void early_setup_idt(void) -{ - void *handler = NULL; - - if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) { - setup_ghcb(); - handler = vc_boot_ghcb; - } - - startup_64_load_idt(handler); -} - /* * Setup boot CPU state needed before kernel switches to virtual addresses. */ diff --git a/arch/x86/include/asm/setup.h b/arch/x86/include/asm/setup.h index ad9212df0ec0..6324f4c6c545 100644 --- a/arch/x86/include/asm/setup.h +++ b/arch/x86/include/asm/setup.h @@ -52,6 +52,7 @@ extern void reserve_standard_io_resources(void); extern void i386_reserve_resources(void); extern unsigned long __startup_64(unsigned long p2v_offset, struct boot_params *bp); extern void startup_64_setup_gdt_idt(void); +extern void startup_64_load_idt(void *vc_handler); extern void early_setup_idt(void); extern void __init do_early_exception(struct pt_regs *regs, int trapnr); diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 6b68a206fa7f..29226f3ac064 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -303,3 +303,15 @@ void __init __noreturn x86_64_start_reservations(char *real_mode_data) start_kernel(); } + +void early_setup_idt(void) +{ + void *handler = NULL; + + if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) { + setup_ghcb(); + handler = vc_boot_ghcb; + } + + startup_64_load_idt(handler); +} From patchwork Sun May 4 09:52:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887283 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B15A1A4F0A for ; Sun, 4 May 2025 09:53:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352394; cv=none; b=rEw2A/ULSm5WaNfC7G+0TLZGTA9bZBHmHVmQyLFXRLh92lxUhQh+4HGWYt4bPdSp2I5xtLcXe28+8hop9lF9QGaC1k5bxJGR7Zdm07f0o6GIClM3E7HXoBHwsAQPGiPxAejwBmLHmIkHJMMuvxKlMWqHqnriObgLCzTccW7+TO8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352394; c=relaxed/simple; bh=gNZGHqyOZ8kqPKqESXMKN4wrhEpSByrTvzHlmUl4fE4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=hsUpMfYW0D7P6CetP9NXWLe3df1LovTXIravtIG/Xx2N7DFOJCnI7unfW33k4xV4Dobg+FAuctSoRZEfKzYCP7KLKRbtabFCU1SbPMVWhdlmW3MNzAMu7pO8AYMpSMgbu18IjDebUag9EMFOw/BnfNcQzXY56LLvDYSnRBFeUo8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=4x2XJdLL; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="4x2XJdLL" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-43cf44b66f7so17998975e9.1 for ; Sun, 04 May 2025 02:53:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352390; x=1746957190; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=V3KvQ4IaPUIPHwhX5gNyqgIsIykUHwLmOqlwVneJyU0=; b=4x2XJdLLlgQlVdFT/JwccfgjuVI9esZmr9f0LPWzrp7tC0rbk672T/TbZnXnUWysEq OMnV5b4S+8lvzGLaXlhTMufYQJapNOJyp6wkyvnbWUoNYn0taOrXEwYVlPHFO+Gf2mgN rlHwMm6eBZtwyrQIU6CC511PwT6FIMvfL0qsC2PXoMCqYdvaQLb6kl8bO7SSH4x+F9B8 PXtDzPASsmNlL9iPDaD8EOILgcxAr5pmSUfQzz6l7fPGhVcgHXAZvjiGnrwiOMJdZ0RO bdBYDnvu2xdyx9QQS7otswBkmCgNUKZPV4k1FONZsT7MHUhsQ3ZQeScr4TeeNY1h7Xhi sO2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352390; x=1746957190; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=V3KvQ4IaPUIPHwhX5gNyqgIsIykUHwLmOqlwVneJyU0=; b=PeIP/5OdLmP5ShRxH2vhQUfzGq2nPmHNs+YltiHKf69YGM6LsP90vsTZOAto3yk75g 7udi3hVQzcFGwigth3/sdVNVVRbdNvqdk6MYDcPPyZ7A8gBqA4n1vlzk3KnXLkBRpCjQ ioCxbNnsghW61jLGG84pub6Bu7lDhmOgEvIFzAOFllBpFRo6rDpCPDqioXkihlSe+faE SseuDr1EYDyKMyEhYeRC4YG66qQua2nwr4IsQWY+5nmCArmG7eNjvua+OCpE/q2CQPDu 9YQ6hef/+1W0zzbbEWdyNQA7BqJb8IPRW8DMOYq7n1hHZ0BohLC/xhG2ZhjAwfqzw+rk XhiA== X-Gm-Message-State: AOJu0YzUM+HaamliAtCnQPWBOOcDbHzJJ0IOE0g7O30irPZQ0lvjlz+e 1VMHEATMk46R5Nz4HQXYmPiyunOLXCkGf89JwCd+dvjt/2jU3pIa1abfLZ8DpWpBU7xPOA== X-Google-Smtp-Source: AGHT+IH4o59y+hsvxaJrLfYuHjis35NQIrge7NC7Dy2cR0sdpFlyVLpjG9qzhqRzqll7hgTgzHEF5e5a X-Received: from wmbbe8.prod.google.com ([2002:a05:600c:1e88:b0:441:b661:2d94]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:a40e:b0:441:ac58:ead5 with SMTP id 5b1f17b1804b1-441c1d739a8mr34930995e9.31.1746352390621; Sun, 04 May 2025 02:53:10 -0700 (PDT) Date: Sun, 4 May 2025 11:52:33 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=9953; i=ardb@kernel.org; h=from:subject; bh=koYlPPxJjdcJPSpPZ44TX3JNDHpVDK2BDJVMPMG1Oms=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4lGbyNEnR/yenLmgVeqk7bGFu3zpC101UQGb6K9iT Fv/y0zvKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABNJv8HwT6Vs2kvuBv3ns88U /17rEGDvsKNk1TXnrL8x+75yX8j+P4+RYfasJc4uapq792hnuE8xvvDzy4yG34s+XZ0Qoy41bZL bPlYA X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-28-ardb+git@google.com> Subject: [RFT PATCH v2 03/23] x86/boot: Drop global variables keeping track of LA57 state From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel On x86_64, the core kernel is entered in long mode, which implies that paging is enabled. This means that the CR4.LA57 control bit is guaranteed to be in sync with the number of paging levels used by the kernel, and there is no need to store this in a variable. There is also no need to use variables for storing the calculations of pgdir_shift and ptrs_per_p4d, as they are easily determined on the fly. This removes the need for two different sources of truth (i.e., early and late) for determining whether 5-level paging is in use: CR4.LA57 always reflects the actual state, and never changes from the point of view of the 64-bit core kernel. It also removes the need for exposing the associated variables to the startup code. The only potential concern is the cost of CR4 accesses, which can be mitigated using alternatives patching based on feature detection. Note that even the decompressor does not manipulate any page tables before updating CR4.LA57, so it can also avoid the associated global variables entirely. However, as it does not implement alternatives patching, the associated ELF sections need to be discarded. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/misc.h | 4 -- arch/x86/boot/compressed/pgtable_64.c | 12 ------ arch/x86/boot/compressed/vmlinux.lds.S | 1 + arch/x86/boot/startup/map_kernel.c | 12 +----- arch/x86/boot/startup/sme.c | 9 ---- arch/x86/include/asm/pgtable_64_types.h | 43 ++++++++++---------- arch/x86/kernel/cpu/common.c | 2 - arch/x86/kernel/head64.c | 11 ----- arch/x86/mm/kasan_init_64.c | 3 -- 9 files changed, 24 insertions(+), 73 deletions(-) diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h index dd8d1a85f671..450d27d0f449 100644 --- a/arch/x86/boot/compressed/misc.h +++ b/arch/x86/boot/compressed/misc.h @@ -16,9 +16,6 @@ #define __NO_FORTIFY -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 - /* * Boot stub deals with identity mappings, physical and virtual addresses are * the same, so override these defines. @@ -181,7 +178,6 @@ static inline int count_immovable_mem_regions(void) { return 0; } #endif /* ident_map_64.c */ -extern unsigned int __pgtable_l5_enabled, pgdir_shift, ptrs_per_p4d; extern void kernel_add_identity_map(unsigned long start, unsigned long end); /* Used by PAGE_KERN* macros: */ diff --git a/arch/x86/boot/compressed/pgtable_64.c b/arch/x86/boot/compressed/pgtable_64.c index 5a6c7a190e5b..591d28f2feb6 100644 --- a/arch/x86/boot/compressed/pgtable_64.c +++ b/arch/x86/boot/compressed/pgtable_64.c @@ -10,13 +10,6 @@ #define BIOS_START_MIN 0x20000U /* 128K, less than this is insane */ #define BIOS_START_MAX 0x9f000U /* 640K, absolute maximum */ -#ifdef CONFIG_X86_5LEVEL -/* __pgtable_l5_enabled needs to be in .data to avoid being cleared along with .bss */ -unsigned int __section(".data") __pgtable_l5_enabled; -unsigned int __section(".data") pgdir_shift = 39; -unsigned int __section(".data") ptrs_per_p4d = 1; -#endif - /* Buffer to preserve trampoline memory */ static char trampoline_save[TRAMPOLINE_32BIT_SIZE]; @@ -127,11 +120,6 @@ asmlinkage void configure_5level_paging(struct boot_params *bp, void *pgtable) native_cpuid_eax(0) >= 7 && (native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) { l5_required = true; - - /* Initialize variables for 5-level paging */ - __pgtable_l5_enabled = 1; - pgdir_shift = 48; - ptrs_per_p4d = 512; } /* diff --git a/arch/x86/boot/compressed/vmlinux.lds.S b/arch/x86/boot/compressed/vmlinux.lds.S index 3b2bc61c9408..d3e786ff7dcb 100644 --- a/arch/x86/boot/compressed/vmlinux.lds.S +++ b/arch/x86/boot/compressed/vmlinux.lds.S @@ -81,6 +81,7 @@ SECTIONS *(.dynamic) *(.dynsym) *(.dynstr) *(.dynbss) *(.hash) *(.gnu.hash) *(.note.*) + *(.altinstructions .altinstr_replacement) } .got.plt (INFO) : { diff --git a/arch/x86/boot/startup/map_kernel.c b/arch/x86/boot/startup/map_kernel.c index 099ae2559336..11f99d907f76 100644 --- a/arch/x86/boot/startup/map_kernel.c +++ b/arch/x86/boot/startup/map_kernel.c @@ -16,19 +16,9 @@ extern unsigned int next_early_pgt; static inline bool check_la57_support(void) { - if (!IS_ENABLED(CONFIG_X86_5LEVEL)) + if (!pgtable_l5_enabled()) return false; - /* - * 5-level paging is detected and enabled at kernel decompression - * stage. Only check if it has been enabled there. - */ - if (!(native_read_cr4() & X86_CR4_LA57)) - return false; - - __pgtable_l5_enabled = 1; - pgdir_shift = 48; - ptrs_per_p4d = 512; page_offset_base = __PAGE_OFFSET_BASE_L5; vmalloc_base = __VMALLOC_BASE_L5; vmemmap_base = __VMEMMAP_BASE_L5; diff --git a/arch/x86/boot/startup/sme.c b/arch/x86/boot/startup/sme.c index 5738b31c8e60..ade5ad5060e9 100644 --- a/arch/x86/boot/startup/sme.c +++ b/arch/x86/boot/startup/sme.c @@ -25,15 +25,6 @@ #undef CONFIG_PARAVIRT_XXL #undef CONFIG_PARAVIRT_SPINLOCKS -/* - * This code runs before CPU feature bits are set. By default, the - * pgtable_l5_enabled() function uses bit X86_FEATURE_LA57 to determine if - * 5-level paging is active, so that won't work here. USE_EARLY_PGTABLE_L5 - * is provided to handle this situation and, instead, use a variable that - * has been set by the early boot code. - */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm/pgtable_64_types.h index 5bb782d856f2..40dceb7d80f5 100644 --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -6,7 +6,10 @@ #ifndef __ASSEMBLER__ #include +#include +#include #include +#include /* * These are used to make use of C type-checking.. @@ -21,28 +24,24 @@ typedef unsigned long pgprotval_t; typedef struct { pteval_t pte; } pte_t; typedef struct { pmdval_t pmd; } pmd_t; -extern unsigned int __pgtable_l5_enabled; - -#ifdef CONFIG_X86_5LEVEL -#ifdef USE_EARLY_PGTABLE_L5 -/* - * cpu_feature_enabled() is not available in early boot code. - * Use variable instead. - */ -static inline bool pgtable_l5_enabled(void) +static __always_inline __pure bool pgtable_l5_enabled(void) { - return __pgtable_l5_enabled; -} -#else -#define pgtable_l5_enabled() cpu_feature_enabled(X86_FEATURE_LA57) -#endif /* USE_EARLY_PGTABLE_L5 */ + unsigned long r; + bool ret; -#else -#define pgtable_l5_enabled() 0 -#endif /* CONFIG_X86_5LEVEL */ + if (!IS_ENABLED(CONFIG_X86_5LEVEL)) + return false; -extern unsigned int pgdir_shift; -extern unsigned int ptrs_per_p4d; + asm(ALTERNATIVE_TERNARY( + "movq %%cr4, %[reg] \n\t btl %[la57], %k[reg]" CC_SET(c), + %P[feat], "stc", "clc") + : [reg] "=&r" (r), CC_OUT(c) (ret) + : [feat] "i" (X86_FEATURE_LA57), + [la57] "i" (X86_CR4_LA57_BIT) + : "cc"); + + return ret; +} #endif /* !__ASSEMBLER__ */ @@ -53,7 +52,7 @@ extern unsigned int ptrs_per_p4d; /* * PGDIR_SHIFT determines what a top-level page table entry can map */ -#define PGDIR_SHIFT pgdir_shift +#define PGDIR_SHIFT (pgtable_l5_enabled() ? 48 : 39) #define PTRS_PER_PGD 512 /* @@ -61,7 +60,7 @@ extern unsigned int ptrs_per_p4d; */ #define P4D_SHIFT 39 #define MAX_PTRS_PER_P4D 512 -#define PTRS_PER_P4D ptrs_per_p4d +#define PTRS_PER_P4D (pgtable_l5_enabled() ? 512 : 1) #define P4D_SIZE (_AC(1, UL) << P4D_SHIFT) #define P4D_MASK (~(P4D_SIZE - 1)) @@ -76,6 +75,8 @@ extern unsigned int ptrs_per_p4d; #define PTRS_PER_PGD 512 #define MAX_PTRS_PER_P4D 1 +#define MAX_POSSIBLE_PHYSMEM_BITS 46 + #endif /* CONFIG_X86_5LEVEL */ /* diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 12126adbc3a9..eb6a7f6e20c4 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1,6 +1,4 @@ // SPDX-License-Identifier: GPL-2.0-only -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 #include #include diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 29226f3ac064..3d49abb1bb3a 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -5,9 +5,6 @@ * Copyright (C) 2000 Andrea Arcangeli SuSE */ -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include @@ -50,14 +47,6 @@ extern pmd_t early_dynamic_pgts[EARLY_DYNAMIC_PAGE_TABLES][PTRS_PER_PMD]; unsigned int __initdata next_early_pgt; pmdval_t early_pmd_flags = __PAGE_KERNEL_LARGE & ~(_PAGE_GLOBAL | _PAGE_NX); -#ifdef CONFIG_X86_5LEVEL -unsigned int __pgtable_l5_enabled __ro_after_init; -unsigned int pgdir_shift __ro_after_init = 39; -EXPORT_SYMBOL(pgdir_shift); -unsigned int ptrs_per_p4d __ro_after_init = 1; -EXPORT_SYMBOL(ptrs_per_p4d); -#endif - #ifdef CONFIG_DYNAMIC_MEMORY_LAYOUT unsigned long page_offset_base __ro_after_init = __PAGE_OFFSET_BASE_L4; EXPORT_SYMBOL(page_offset_base); diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c index 0539efd0d216..7c4fafbd52cc 100644 --- a/arch/x86/mm/kasan_init_64.c +++ b/arch/x86/mm/kasan_init_64.c @@ -1,9 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 #define pr_fmt(fmt) "kasan: " fmt -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include From patchwork Sun May 4 09:52:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887281 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EEC571AED5C for ; Sun, 4 May 2025 09:53:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352403; cv=none; b=H7jure+MjskJ2LvlCCSnnJqqEan48NYhZ7CdVSsaETMNThdk9cE43I0bD1Ng3k2WwDiu6rDgS1rrTvhVarhuBHgoXNf8Qk0lhs9tx2u2LziqDsbGtFUM0Ze8/uuiva7Lb4vzM/IcVAmVlUdljxSvgSih9NY1jw1jalXHG1qhfUo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352403; c=relaxed/simple; bh=uHtXWXjI/fi30+pkywdogAKaVVkPSPO0JZPHE7tRVPk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ni5X5IXYA81Tl0FtEgBLFaLWAA+KYMLZrxPlC5EcTxkPISYJ+n3UTgUw2Y4w5ZCUScv+mJ4Py82E1hJPDV23JRC4rfriO9dCo17m7jlncooRiUcJXSazUgcHL06v1ixKDcbtfK1nkUsQh5SP/3zoHoee54BrkJDglIRW+mVC3Co= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=MgyciTJK; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MgyciTJK" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-43f405810b4so17120315e9.1 for ; Sun, 04 May 2025 02:53:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352396; x=1746957196; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=c/Ree8BoBNU+3RrEqP1oFbuP8noOOfP40c2KOku3IVE=; b=MgyciTJKJoPH6/TYJA3EuxlkZMnu370vZVeuxIxnESZ/tN3ZgWbwXVmdhzyNO64TMi OaRLjHTFijssblsmkr3c9Atv+jUkw7c19wPuxIZqabzf5+Wm0aVDdf5bk2qB6h551fNx d4AMOxh9SAwadf/0F3yC/sZvN0uxyttmfUN1we8n+IE9aNr59XRswUH0lXKakaHXLHEx zC0C7JbWrV3vMtS5r08uNZeBNjEJUl8uw4GnDTCBKFYY6orDbIu0sEmavptto6DTNDSr Z79b8Ko+7txGUExRnCVg9ewFNUb3cQTzsALWHRJI56VM/Z2JzeLYa5r2x5gHf+77SbPu uktA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352396; x=1746957196; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=c/Ree8BoBNU+3RrEqP1oFbuP8noOOfP40c2KOku3IVE=; b=ERxX31DXsBXg795LiG6BgrW/LfjBlfHvc4+VOeZRAzxt8VA5Ou+NVdTIMJMmLHwN/U ho03+sR2l0qWrUHeUgkHlZHPgXW3uBZXHRmNnJKGhPBR71dMRCf1vm2cFo2PUZktRrvX Cl/u/e8rzbbHNn8vUWDY9Yy7Tsok1UI3hHMEJo5xlppkV9Y7KPxJxUxbgOyZ3NwYa7BU uxBfkSxVOxBghjLBkj13/Vt+Gc375YTFNCfz1Cl3FCB7mxi7DuuoVhvp6s/Nue4z5WWd a1SJdN97vadzMffVbWKh7JMYbFPHVjTUv4b2P0NokYKq8f3DPelplFvrlPJ/XZHub0z9 riOg== X-Gm-Message-State: AOJu0YxCNo86kn8s1I2WCq6qGz3pgkDlZCzVXydEsXlkd3YDW6z9Y+UO FCSnJQGIIWDeaBE4W5h0GYg4UqSIF1bOE4Ydibi/QWCCwnaQReeCuGt/nLEsITS+x2MmWQ== X-Google-Smtp-Source: AGHT+IHmn9rA1yNSHzEyqV5/C9FAwA1xPI4lksw+RZRSNDoH1wi2MuuBbs66rALvHeFWBDykaGTholhe X-Received: from wmcn18-n1.prod.google.com ([2002:a05:600c:c0d2:10b0:441:c197:9856]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:8706:b0:43c:eeee:b706 with SMTP id 5b1f17b1804b1-441c49375femr24410705e9.24.1746352396707; Sun, 04 May 2025 02:53:16 -0700 (PDT) Date: Sun, 4 May 2025 11:52:36 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=94102; i=ardb@kernel.org; h=from:subject; bh=opAvTz72cU/OKowlnsxsdhWM207AigxqFfI7wxA47h0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4llj3MVpGrU/4m8aSr6/uIHtv6boHe2Sv3L/Dm01n vX8RZtCRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZhIkwIjw7PJq37W7nfY//u/ oNvbX3uE93tPv1Np5WO7W+a3rtRvxpUM/8wZqnOOmfMGazzNSLM50XLi4PTLt7UY+XkshEIkPWP WMAEA X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-31-ardb+git@google.com> Subject: [RFT PATCH v2 06/23] x86/sev: Disentangle #VC handling code from startup code From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel Most of the SEV support code used to reside in a single C source file that was included in two places: the core kernel, and the decompressor. The code that is actually shared with the decompressor was moved into a separate, shared source file under startup/, on the basis that the decompressor also executes from the early 1:1 mapping of memory. However, while the elaborate #VC handling and instruction decoding that it involves is also performed by the decompressor, it does not actually occur in the core kernel at early boot, and therefore, does not need to be part of the confined early startup code. So split off the #VC handling code and move it back into arch/x86/coco where it came from, into another C source file that is included from both the decompressor and the core kernel. Code movement only - no functional change intended. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/misc.h | 1 + arch/x86/boot/compressed/sev-handle-vc.c | 83 ++ arch/x86/boot/compressed/sev.c | 96 +- arch/x86/boot/compressed/sev.h | 19 + arch/x86/boot/startup/sev-shared.c | 519 +--------- arch/x86/boot/startup/sev-startup.c | 1022 ------------------- arch/x86/coco/sev/Makefile | 2 +- arch/x86/coco/sev/vc-handle.c | 1061 ++++++++++++++++++++ arch/x86/coco/sev/vc-shared.c | 504 ++++++++++ arch/x86/include/asm/sev-internal.h | 8 - arch/x86/include/asm/sev.h | 22 + 11 files changed, 1694 insertions(+), 1643 deletions(-) diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h index ccd3f4257bcd..b33f1c5a486c 100644 --- a/arch/x86/boot/compressed/misc.h +++ b/arch/x86/boot/compressed/misc.h @@ -147,6 +147,7 @@ void sev_prep_identity_maps(unsigned long top_level_pgt); enum es_result vc_decode_insn(struct es_em_ctxt *ctxt); bool insn_has_rep_prefix(struct insn *insn); void sev_insn_decode_init(void); +bool early_setup_ghcb(void); #else static inline void sev_enable(struct boot_params *bp) { diff --git a/arch/x86/boot/compressed/sev-handle-vc.c b/arch/x86/boot/compressed/sev-handle-vc.c index b1aa073b732c..89dd02de2a0f 100644 --- a/arch/x86/boot/compressed/sev-handle-vc.c +++ b/arch/x86/boot/compressed/sev-handle-vc.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 #include "misc.h" +#include "sev.h" #include #include @@ -8,6 +9,9 @@ #include #include #include +#include +#include +#include #define __BOOT_COMPRESSED @@ -49,3 +53,82 @@ enum es_result vc_decode_insn(struct es_em_ctxt *ctxt) } extern void sev_insn_decode_init(void) __alias(inat_init_tables); + +/* + * Only a dummy for insn_get_seg_base() - Early boot-code is 64bit only and + * doesn't use segments. + */ +static unsigned long insn_get_seg_base(struct pt_regs *regs, int seg_reg_idx) +{ + return 0UL; +} + +static enum es_result vc_write_mem(struct es_em_ctxt *ctxt, + void *dst, char *buf, size_t size) +{ + memcpy(dst, buf, size); + + return ES_OK; +} + +static enum es_result vc_read_mem(struct es_em_ctxt *ctxt, + void *src, char *buf, size_t size) +{ + memcpy(buf, src, size); + + return ES_OK; +} + +static enum es_result vc_ioio_check(struct es_em_ctxt *ctxt, u16 port, size_t size) +{ + return ES_OK; +} + +static bool fault_in_kernel_space(unsigned long address) +{ + return false; +} + +#define sev_printk(fmt, ...) + +#include "../../coco/sev/vc-shared.c" + +void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code) +{ + struct es_em_ctxt ctxt; + enum es_result result; + + if (!boot_ghcb && !early_setup_ghcb()) + sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SEV_ES_GEN_REQ); + + vc_ghcb_invalidate(boot_ghcb); + result = vc_init_em_ctxt(&ctxt, regs, exit_code); + if (result != ES_OK) + goto finish; + + result = vc_check_opcode_bytes(&ctxt, exit_code); + if (result != ES_OK) + goto finish; + + switch (exit_code) { + case SVM_EXIT_RDTSC: + case SVM_EXIT_RDTSCP: + result = vc_handle_rdtsc(boot_ghcb, &ctxt, exit_code); + break; + case SVM_EXIT_IOIO: + result = vc_handle_ioio(boot_ghcb, &ctxt); + break; + case SVM_EXIT_CPUID: + result = vc_handle_cpuid(boot_ghcb, &ctxt); + break; + default: + result = ES_UNSUPPORTED; + break; + } + +finish: + if (result == ES_OK) + vc_finish_insn(&ctxt); + else if (result != ES_RETRY) + sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SEV_ES_GEN_REQ); +} diff --git a/arch/x86/boot/compressed/sev.c b/arch/x86/boot/compressed/sev.c index 5cd029c4f36d..804166457022 100644 --- a/arch/x86/boot/compressed/sev.c +++ b/arch/x86/boot/compressed/sev.c @@ -24,63 +24,11 @@ #include #include "error.h" -#include "../msr.h" +#include "sev.h" static struct ghcb boot_ghcb_page __aligned(PAGE_SIZE); struct ghcb *boot_ghcb; -/* - * Only a dummy for insn_get_seg_base() - Early boot-code is 64bit only and - * doesn't use segments. - */ -static unsigned long insn_get_seg_base(struct pt_regs *regs, int seg_reg_idx) -{ - return 0UL; -} - -static inline u64 sev_es_rd_ghcb_msr(void) -{ - struct msr m; - - boot_rdmsr(MSR_AMD64_SEV_ES_GHCB, &m); - - return m.q; -} - -static inline void sev_es_wr_ghcb_msr(u64 val) -{ - struct msr m; - - m.q = val; - boot_wrmsr(MSR_AMD64_SEV_ES_GHCB, &m); -} - -static enum es_result vc_write_mem(struct es_em_ctxt *ctxt, - void *dst, char *buf, size_t size) -{ - memcpy(dst, buf, size); - - return ES_OK; -} - -static enum es_result vc_read_mem(struct es_em_ctxt *ctxt, - void *src, char *buf, size_t size) -{ - memcpy(buf, src, size); - - return ES_OK; -} - -static enum es_result vc_ioio_check(struct es_em_ctxt *ctxt, u16 port, size_t size) -{ - return ES_OK; -} - -static bool fault_in_kernel_space(unsigned long address) -{ - return false; -} - #undef __init #define __init @@ -182,7 +130,7 @@ void snp_set_page_shared(unsigned long paddr) __page_state_change(paddr, SNP_PAGE_STATE_SHARED); } -static bool early_setup_ghcb(void) +bool early_setup_ghcb(void) { if (set_page_decrypted((unsigned long)&boot_ghcb_page)) return false; @@ -266,46 +214,6 @@ bool sev_es_check_ghcb_fault(unsigned long address) return ((address & PAGE_MASK) == (unsigned long)&boot_ghcb_page); } -void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code) -{ - struct es_em_ctxt ctxt; - enum es_result result; - - if (!boot_ghcb && !early_setup_ghcb()) - sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SEV_ES_GEN_REQ); - - vc_ghcb_invalidate(boot_ghcb); - result = vc_init_em_ctxt(&ctxt, regs, exit_code); - if (result != ES_OK) - goto finish; - - result = vc_check_opcode_bytes(&ctxt, exit_code); - if (result != ES_OK) - goto finish; - - switch (exit_code) { - case SVM_EXIT_RDTSC: - case SVM_EXIT_RDTSCP: - result = vc_handle_rdtsc(boot_ghcb, &ctxt, exit_code); - break; - case SVM_EXIT_IOIO: - result = vc_handle_ioio(boot_ghcb, &ctxt); - break; - case SVM_EXIT_CPUID: - result = vc_handle_cpuid(boot_ghcb, &ctxt); - break; - default: - result = ES_UNSUPPORTED; - break; - } - -finish: - if (result == ES_OK) - vc_finish_insn(&ctxt); - else if (result != ES_RETRY) - sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SEV_ES_GEN_REQ); -} - /* * SNP_FEATURES_IMPL_REQ is the mask of SNP features that will need * guest side implementation for proper functioning of the guest. If any diff --git a/arch/x86/boot/compressed/sev.h b/arch/x86/boot/compressed/sev.h index 9d21af3a220d..3ed7fb27bd4d 100644 --- a/arch/x86/boot/compressed/sev.h +++ b/arch/x86/boot/compressed/sev.h @@ -10,9 +10,28 @@ #ifdef CONFIG_AMD_MEM_ENCRYPT +#include "../msr.h" + void snp_accept_memory(phys_addr_t start, phys_addr_t end); u64 sev_get_status(void); +static inline u64 sev_es_rd_ghcb_msr(void) +{ + struct msr m; + + boot_rdmsr(MSR_AMD64_SEV_ES_GHCB, &m); + + return m.q; +} + +static inline void sev_es_wr_ghcb_msr(u64 val) +{ + struct msr m; + + m.q = val; + boot_wrmsr(MSR_AMD64_SEV_ES_GHCB, &m); +} + #else static inline void snp_accept_memory(phys_addr_t start, phys_addr_t end) { } diff --git a/arch/x86/boot/startup/sev-shared.c b/arch/x86/boot/startup/sev-shared.c index 173f3d1f777a..7a706db87b93 100644 --- a/arch/x86/boot/startup/sev-shared.c +++ b/arch/x86/boot/startup/sev-shared.c @@ -14,13 +14,9 @@ #ifndef __BOOT_COMPRESSED #define error(v) pr_err(v) #define has_cpuflag(f) boot_cpu_has(f) -#define sev_printk(fmt, ...) printk(fmt, ##__VA_ARGS__) -#define sev_printk_rtl(fmt, ...) printk_ratelimited(fmt, ##__VA_ARGS__) #else #undef WARN #define WARN(condition, format...) (!!(condition)) -#define sev_printk(fmt, ...) -#define sev_printk_rtl(fmt, ...) #undef vc_forward_exception #define vc_forward_exception(c) panic("SNP: Hypervisor requested exception\n") #endif @@ -36,16 +32,6 @@ struct svsm_ca *boot_svsm_caa __ro_after_init; u64 boot_svsm_caa_pa __ro_after_init; -/* I/O parameters for CPUID-related helpers */ -struct cpuid_leaf { - u32 fn; - u32 subfn; - u32 eax; - u32 ebx; - u32 ecx; - u32 edx; -}; - /* * Since feature negotiation related variables are set early in the boot * process they must reside in the .data section so as not to be zeroed @@ -151,33 +137,6 @@ bool sev_es_negotiate_protocol(void) return true; } -static bool vc_decoding_needed(unsigned long exit_code) -{ - /* Exceptions don't require to decode the instruction */ - return !(exit_code >= SVM_EXIT_EXCP_BASE && - exit_code <= SVM_EXIT_LAST_EXCP); -} - -static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt, - struct pt_regs *regs, - unsigned long exit_code) -{ - enum es_result ret = ES_OK; - - memset(ctxt, 0, sizeof(*ctxt)); - ctxt->regs = regs; - - if (vc_decoding_needed(exit_code)) - ret = vc_decode_insn(ctxt); - - return ret; -} - -static void vc_finish_insn(struct es_em_ctxt *ctxt) -{ - ctxt->regs->ip += ctxt->insn.length; -} - static enum es_result verify_exception_info(struct ghcb *ghcb, struct es_em_ctxt *ctxt) { u32 ret; @@ -627,7 +586,7 @@ snp_cpuid_postprocess(struct ghcb *ghcb, struct es_em_ctxt *ctxt, * Returns -EOPNOTSUPP if feature not enabled. Any other non-zero return value * should be treated as fatal by caller. */ -static int __head +int __head snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf) { const struct snp_cpuid_table *cpuid_table = snp_cpuid_get_table(); @@ -737,391 +696,6 @@ void __head do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code) sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SEV_ES_GEN_REQ); } -static enum es_result vc_insn_string_check(struct es_em_ctxt *ctxt, - unsigned long address, - bool write) -{ - if (user_mode(ctxt->regs) && fault_in_kernel_space(address)) { - ctxt->fi.vector = X86_TRAP_PF; - ctxt->fi.error_code = X86_PF_USER; - ctxt->fi.cr2 = address; - if (write) - ctxt->fi.error_code |= X86_PF_WRITE; - - return ES_EXCEPTION; - } - - return ES_OK; -} - -static enum es_result vc_insn_string_read(struct es_em_ctxt *ctxt, - void *src, char *buf, - unsigned int data_size, - unsigned int count, - bool backwards) -{ - int i, b = backwards ? -1 : 1; - unsigned long address = (unsigned long)src; - enum es_result ret; - - ret = vc_insn_string_check(ctxt, address, false); - if (ret != ES_OK) - return ret; - - for (i = 0; i < count; i++) { - void *s = src + (i * data_size * b); - char *d = buf + (i * data_size); - - ret = vc_read_mem(ctxt, s, d, data_size); - if (ret != ES_OK) - break; - } - - return ret; -} - -static enum es_result vc_insn_string_write(struct es_em_ctxt *ctxt, - void *dst, char *buf, - unsigned int data_size, - unsigned int count, - bool backwards) -{ - int i, s = backwards ? -1 : 1; - unsigned long address = (unsigned long)dst; - enum es_result ret; - - ret = vc_insn_string_check(ctxt, address, true); - if (ret != ES_OK) - return ret; - - for (i = 0; i < count; i++) { - void *d = dst + (i * data_size * s); - char *b = buf + (i * data_size); - - ret = vc_write_mem(ctxt, d, b, data_size); - if (ret != ES_OK) - break; - } - - return ret; -} - -#define IOIO_TYPE_STR BIT(2) -#define IOIO_TYPE_IN 1 -#define IOIO_TYPE_INS (IOIO_TYPE_IN | IOIO_TYPE_STR) -#define IOIO_TYPE_OUT 0 -#define IOIO_TYPE_OUTS (IOIO_TYPE_OUT | IOIO_TYPE_STR) - -#define IOIO_REP BIT(3) - -#define IOIO_ADDR_64 BIT(9) -#define IOIO_ADDR_32 BIT(8) -#define IOIO_ADDR_16 BIT(7) - -#define IOIO_DATA_32 BIT(6) -#define IOIO_DATA_16 BIT(5) -#define IOIO_DATA_8 BIT(4) - -#define IOIO_SEG_ES (0 << 10) -#define IOIO_SEG_DS (3 << 10) - -static enum es_result vc_ioio_exitinfo(struct es_em_ctxt *ctxt, u64 *exitinfo) -{ - struct insn *insn = &ctxt->insn; - size_t size; - u64 port; - - *exitinfo = 0; - - switch (insn->opcode.bytes[0]) { - /* INS opcodes */ - case 0x6c: - case 0x6d: - *exitinfo |= IOIO_TYPE_INS; - *exitinfo |= IOIO_SEG_ES; - port = ctxt->regs->dx & 0xffff; - break; - - /* OUTS opcodes */ - case 0x6e: - case 0x6f: - *exitinfo |= IOIO_TYPE_OUTS; - *exitinfo |= IOIO_SEG_DS; - port = ctxt->regs->dx & 0xffff; - break; - - /* IN immediate opcodes */ - case 0xe4: - case 0xe5: - *exitinfo |= IOIO_TYPE_IN; - port = (u8)insn->immediate.value & 0xffff; - break; - - /* OUT immediate opcodes */ - case 0xe6: - case 0xe7: - *exitinfo |= IOIO_TYPE_OUT; - port = (u8)insn->immediate.value & 0xffff; - break; - - /* IN register opcodes */ - case 0xec: - case 0xed: - *exitinfo |= IOIO_TYPE_IN; - port = ctxt->regs->dx & 0xffff; - break; - - /* OUT register opcodes */ - case 0xee: - case 0xef: - *exitinfo |= IOIO_TYPE_OUT; - port = ctxt->regs->dx & 0xffff; - break; - - default: - return ES_DECODE_FAILED; - } - - *exitinfo |= port << 16; - - switch (insn->opcode.bytes[0]) { - case 0x6c: - case 0x6e: - case 0xe4: - case 0xe6: - case 0xec: - case 0xee: - /* Single byte opcodes */ - *exitinfo |= IOIO_DATA_8; - size = 1; - break; - default: - /* Length determined by instruction parsing */ - *exitinfo |= (insn->opnd_bytes == 2) ? IOIO_DATA_16 - : IOIO_DATA_32; - size = (insn->opnd_bytes == 2) ? 2 : 4; - } - - switch (insn->addr_bytes) { - case 2: - *exitinfo |= IOIO_ADDR_16; - break; - case 4: - *exitinfo |= IOIO_ADDR_32; - break; - case 8: - *exitinfo |= IOIO_ADDR_64; - break; - } - - if (insn_has_rep_prefix(insn)) - *exitinfo |= IOIO_REP; - - return vc_ioio_check(ctxt, (u16)port, size); -} - -static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt) -{ - struct pt_regs *regs = ctxt->regs; - u64 exit_info_1, exit_info_2; - enum es_result ret; - - ret = vc_ioio_exitinfo(ctxt, &exit_info_1); - if (ret != ES_OK) - return ret; - - if (exit_info_1 & IOIO_TYPE_STR) { - - /* (REP) INS/OUTS */ - - bool df = ((regs->flags & X86_EFLAGS_DF) == X86_EFLAGS_DF); - unsigned int io_bytes, exit_bytes; - unsigned int ghcb_count, op_count; - unsigned long es_base; - u64 sw_scratch; - - /* - * For the string variants with rep prefix the amount of in/out - * operations per #VC exception is limited so that the kernel - * has a chance to take interrupts and re-schedule while the - * instruction is emulated. - */ - io_bytes = (exit_info_1 >> 4) & 0x7; - ghcb_count = sizeof(ghcb->shared_buffer) / io_bytes; - - op_count = (exit_info_1 & IOIO_REP) ? regs->cx : 1; - exit_info_2 = min(op_count, ghcb_count); - exit_bytes = exit_info_2 * io_bytes; - - es_base = insn_get_seg_base(ctxt->regs, INAT_SEG_REG_ES); - - /* Read bytes of OUTS into the shared buffer */ - if (!(exit_info_1 & IOIO_TYPE_IN)) { - ret = vc_insn_string_read(ctxt, - (void *)(es_base + regs->si), - ghcb->shared_buffer, io_bytes, - exit_info_2, df); - if (ret) - return ret; - } - - /* - * Issue an VMGEXIT to the HV to consume the bytes from the - * shared buffer or to have it write them into the shared buffer - * depending on the instruction: OUTS or INS. - */ - sw_scratch = __pa(ghcb) + offsetof(struct ghcb, shared_buffer); - ghcb_set_sw_scratch(ghcb, sw_scratch); - ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_IOIO, - exit_info_1, exit_info_2); - if (ret != ES_OK) - return ret; - - /* Read bytes from shared buffer into the guest's destination. */ - if (exit_info_1 & IOIO_TYPE_IN) { - ret = vc_insn_string_write(ctxt, - (void *)(es_base + regs->di), - ghcb->shared_buffer, io_bytes, - exit_info_2, df); - if (ret) - return ret; - - if (df) - regs->di -= exit_bytes; - else - regs->di += exit_bytes; - } else { - if (df) - regs->si -= exit_bytes; - else - regs->si += exit_bytes; - } - - if (exit_info_1 & IOIO_REP) - regs->cx -= exit_info_2; - - ret = regs->cx ? ES_RETRY : ES_OK; - - } else { - - /* IN/OUT into/from rAX */ - - int bits = (exit_info_1 & 0x70) >> 1; - u64 rax = 0; - - if (!(exit_info_1 & IOIO_TYPE_IN)) - rax = lower_bits(regs->ax, bits); - - ghcb_set_rax(ghcb, rax); - - ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_IOIO, exit_info_1, 0); - if (ret != ES_OK) - return ret; - - if (exit_info_1 & IOIO_TYPE_IN) { - if (!ghcb_rax_is_valid(ghcb)) - return ES_VMM_ERROR; - regs->ax = lower_bits(ghcb->save.rax, bits); - } - } - - return ret; -} - -static int vc_handle_cpuid_snp(struct ghcb *ghcb, struct es_em_ctxt *ctxt) -{ - struct pt_regs *regs = ctxt->regs; - struct cpuid_leaf leaf; - int ret; - - leaf.fn = regs->ax; - leaf.subfn = regs->cx; - ret = snp_cpuid(ghcb, ctxt, &leaf); - if (!ret) { - regs->ax = leaf.eax; - regs->bx = leaf.ebx; - regs->cx = leaf.ecx; - regs->dx = leaf.edx; - } - - return ret; -} - -static enum es_result vc_handle_cpuid(struct ghcb *ghcb, - struct es_em_ctxt *ctxt) -{ - struct pt_regs *regs = ctxt->regs; - u32 cr4 = native_read_cr4(); - enum es_result ret; - int snp_cpuid_ret; - - snp_cpuid_ret = vc_handle_cpuid_snp(ghcb, ctxt); - if (!snp_cpuid_ret) - return ES_OK; - if (snp_cpuid_ret != -EOPNOTSUPP) - return ES_VMM_ERROR; - - ghcb_set_rax(ghcb, regs->ax); - ghcb_set_rcx(ghcb, regs->cx); - - if (cr4 & X86_CR4_OSXSAVE) - /* Safe to read xcr0 */ - ghcb_set_xcr0(ghcb, xgetbv(XCR_XFEATURE_ENABLED_MASK)); - else - /* xgetbv will cause #GP - use reset value for xcr0 */ - ghcb_set_xcr0(ghcb, 1); - - ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_CPUID, 0, 0); - if (ret != ES_OK) - return ret; - - if (!(ghcb_rax_is_valid(ghcb) && - ghcb_rbx_is_valid(ghcb) && - ghcb_rcx_is_valid(ghcb) && - ghcb_rdx_is_valid(ghcb))) - return ES_VMM_ERROR; - - regs->ax = ghcb->save.rax; - regs->bx = ghcb->save.rbx; - regs->cx = ghcb->save.rcx; - regs->dx = ghcb->save.rdx; - - return ES_OK; -} - -static enum es_result vc_handle_rdtsc(struct ghcb *ghcb, - struct es_em_ctxt *ctxt, - unsigned long exit_code) -{ - bool rdtscp = (exit_code == SVM_EXIT_RDTSCP); - enum es_result ret; - - /* - * The hypervisor should not be intercepting RDTSC/RDTSCP when Secure - * TSC is enabled. A #VC exception will be generated if the RDTSC/RDTSCP - * instructions are being intercepted. If this should occur and Secure - * TSC is enabled, guest execution should be terminated as the guest - * cannot rely on the TSC value provided by the hypervisor. - */ - if (sev_status & MSR_AMD64_SNP_SECURE_TSC) - return ES_VMM_ERROR; - - ret = sev_es_ghcb_hv_call(ghcb, ctxt, exit_code, 0, 0); - if (ret != ES_OK) - return ret; - - if (!(ghcb_rax_is_valid(ghcb) && ghcb_rdx_is_valid(ghcb) && - (!rdtscp || ghcb_rcx_is_valid(ghcb)))) - return ES_VMM_ERROR; - - ctxt->regs->ax = ghcb->save.rax; - ctxt->regs->dx = ghcb->save.rdx; - if (rdtscp) - ctxt->regs->cx = ghcb->save.rcx; - - return ES_OK; -} - struct cc_setup_data { struct setup_data header; u32 cc_blob_address; @@ -1238,97 +812,6 @@ static void __head pvalidate_4k_page(unsigned long vaddr, unsigned long paddr, } } -static enum es_result vc_check_opcode_bytes(struct es_em_ctxt *ctxt, - unsigned long exit_code) -{ - unsigned int opcode = (unsigned int)ctxt->insn.opcode.value; - u8 modrm = ctxt->insn.modrm.value; - - switch (exit_code) { - - case SVM_EXIT_IOIO: - case SVM_EXIT_NPF: - /* handled separately */ - return ES_OK; - - case SVM_EXIT_CPUID: - if (opcode == 0xa20f) - return ES_OK; - break; - - case SVM_EXIT_INVD: - if (opcode == 0x080f) - return ES_OK; - break; - - case SVM_EXIT_MONITOR: - /* MONITOR and MONITORX instructions generate the same error code */ - if (opcode == 0x010f && (modrm == 0xc8 || modrm == 0xfa)) - return ES_OK; - break; - - case SVM_EXIT_MWAIT: - /* MWAIT and MWAITX instructions generate the same error code */ - if (opcode == 0x010f && (modrm == 0xc9 || modrm == 0xfb)) - return ES_OK; - break; - - case SVM_EXIT_MSR: - /* RDMSR */ - if (opcode == 0x320f || - /* WRMSR */ - opcode == 0x300f) - return ES_OK; - break; - - case SVM_EXIT_RDPMC: - if (opcode == 0x330f) - return ES_OK; - break; - - case SVM_EXIT_RDTSC: - if (opcode == 0x310f) - return ES_OK; - break; - - case SVM_EXIT_RDTSCP: - if (opcode == 0x010f && modrm == 0xf9) - return ES_OK; - break; - - case SVM_EXIT_READ_DR7: - if (opcode == 0x210f && - X86_MODRM_REG(ctxt->insn.modrm.value) == 7) - return ES_OK; - break; - - case SVM_EXIT_VMMCALL: - if (opcode == 0x010f && modrm == 0xd9) - return ES_OK; - - break; - - case SVM_EXIT_WRITE_DR7: - if (opcode == 0x230f && - X86_MODRM_REG(ctxt->insn.modrm.value) == 7) - return ES_OK; - break; - - case SVM_EXIT_WBINVD: - if (opcode == 0x90f) - return ES_OK; - break; - - default: - break; - } - - sev_printk(KERN_ERR "Wrong/unhandled opcode bytes: 0x%x, exit_code: 0x%lx, rIP: 0x%lx\n", - opcode, exit_code, ctxt->regs->ip); - - return ES_UNSUPPORTED; -} - /* * Maintain the GPA of the SVSM Calling Area (CA) in order to utilize the SVSM * services needed when not running in VMPL0. diff --git a/arch/x86/boot/startup/sev-startup.c b/arch/x86/boot/startup/sev-startup.c index f901ce9680e6..435853a55768 100644 --- a/arch/x86/boot/startup/sev-startup.c +++ b/arch/x86/boot/startup/sev-startup.c @@ -9,7 +9,6 @@ #define pr_fmt(fmt) "SEV: " fmt -#include /* For show_regs() */ #include #include #include @@ -112,315 +111,6 @@ noinstr struct ghcb *__sev_get_ghcb(struct ghcb_state *state) return ghcb; } -static int vc_fetch_insn_kernel(struct es_em_ctxt *ctxt, - unsigned char *buffer) -{ - return copy_from_kernel_nofault(buffer, (unsigned char *)ctxt->regs->ip, MAX_INSN_SIZE); -} - -static enum es_result __vc_decode_user_insn(struct es_em_ctxt *ctxt) -{ - char buffer[MAX_INSN_SIZE]; - int insn_bytes; - - insn_bytes = insn_fetch_from_user_inatomic(ctxt->regs, buffer); - if (insn_bytes == 0) { - /* Nothing could be copied */ - ctxt->fi.vector = X86_TRAP_PF; - ctxt->fi.error_code = X86_PF_INSTR | X86_PF_USER; - ctxt->fi.cr2 = ctxt->regs->ip; - return ES_EXCEPTION; - } else if (insn_bytes == -EINVAL) { - /* Effective RIP could not be calculated */ - ctxt->fi.vector = X86_TRAP_GP; - ctxt->fi.error_code = 0; - ctxt->fi.cr2 = 0; - return ES_EXCEPTION; - } - - if (!insn_decode_from_regs(&ctxt->insn, ctxt->regs, buffer, insn_bytes)) - return ES_DECODE_FAILED; - - if (ctxt->insn.immediate.got) - return ES_OK; - else - return ES_DECODE_FAILED; -} - -static enum es_result __vc_decode_kern_insn(struct es_em_ctxt *ctxt) -{ - char buffer[MAX_INSN_SIZE]; - int res, ret; - - res = vc_fetch_insn_kernel(ctxt, buffer); - if (res) { - ctxt->fi.vector = X86_TRAP_PF; - ctxt->fi.error_code = X86_PF_INSTR; - ctxt->fi.cr2 = ctxt->regs->ip; - return ES_EXCEPTION; - } - - ret = insn_decode(&ctxt->insn, buffer, MAX_INSN_SIZE, INSN_MODE_64); - if (ret < 0) - return ES_DECODE_FAILED; - else - return ES_OK; -} - -static enum es_result vc_decode_insn(struct es_em_ctxt *ctxt) -{ - if (user_mode(ctxt->regs)) - return __vc_decode_user_insn(ctxt); - else - return __vc_decode_kern_insn(ctxt); -} - -static enum es_result vc_write_mem(struct es_em_ctxt *ctxt, - char *dst, char *buf, size_t size) -{ - unsigned long error_code = X86_PF_PROT | X86_PF_WRITE; - - /* - * This function uses __put_user() independent of whether kernel or user - * memory is accessed. This works fine because __put_user() does no - * sanity checks of the pointer being accessed. All that it does is - * to report when the access failed. - * - * Also, this function runs in atomic context, so __put_user() is not - * allowed to sleep. The page-fault handler detects that it is running - * in atomic context and will not try to take mmap_sem and handle the - * fault, so additional pagefault_enable()/disable() calls are not - * needed. - * - * The access can't be done via copy_to_user() here because - * vc_write_mem() must not use string instructions to access unsafe - * memory. The reason is that MOVS is emulated by the #VC handler by - * splitting the move up into a read and a write and taking a nested #VC - * exception on whatever of them is the MMIO access. Using string - * instructions here would cause infinite nesting. - */ - switch (size) { - case 1: { - u8 d1; - u8 __user *target = (u8 __user *)dst; - - memcpy(&d1, buf, 1); - if (__put_user(d1, target)) - goto fault; - break; - } - case 2: { - u16 d2; - u16 __user *target = (u16 __user *)dst; - - memcpy(&d2, buf, 2); - if (__put_user(d2, target)) - goto fault; - break; - } - case 4: { - u32 d4; - u32 __user *target = (u32 __user *)dst; - - memcpy(&d4, buf, 4); - if (__put_user(d4, target)) - goto fault; - break; - } - case 8: { - u64 d8; - u64 __user *target = (u64 __user *)dst; - - memcpy(&d8, buf, 8); - if (__put_user(d8, target)) - goto fault; - break; - } - default: - WARN_ONCE(1, "%s: Invalid size: %zu\n", __func__, size); - return ES_UNSUPPORTED; - } - - return ES_OK; - -fault: - if (user_mode(ctxt->regs)) - error_code |= X86_PF_USER; - - ctxt->fi.vector = X86_TRAP_PF; - ctxt->fi.error_code = error_code; - ctxt->fi.cr2 = (unsigned long)dst; - - return ES_EXCEPTION; -} - -static enum es_result vc_read_mem(struct es_em_ctxt *ctxt, - char *src, char *buf, size_t size) -{ - unsigned long error_code = X86_PF_PROT; - - /* - * This function uses __get_user() independent of whether kernel or user - * memory is accessed. This works fine because __get_user() does no - * sanity checks of the pointer being accessed. All that it does is - * to report when the access failed. - * - * Also, this function runs in atomic context, so __get_user() is not - * allowed to sleep. The page-fault handler detects that it is running - * in atomic context and will not try to take mmap_sem and handle the - * fault, so additional pagefault_enable()/disable() calls are not - * needed. - * - * The access can't be done via copy_from_user() here because - * vc_read_mem() must not use string instructions to access unsafe - * memory. The reason is that MOVS is emulated by the #VC handler by - * splitting the move up into a read and a write and taking a nested #VC - * exception on whatever of them is the MMIO access. Using string - * instructions here would cause infinite nesting. - */ - switch (size) { - case 1: { - u8 d1; - u8 __user *s = (u8 __user *)src; - - if (__get_user(d1, s)) - goto fault; - memcpy(buf, &d1, 1); - break; - } - case 2: { - u16 d2; - u16 __user *s = (u16 __user *)src; - - if (__get_user(d2, s)) - goto fault; - memcpy(buf, &d2, 2); - break; - } - case 4: { - u32 d4; - u32 __user *s = (u32 __user *)src; - - if (__get_user(d4, s)) - goto fault; - memcpy(buf, &d4, 4); - break; - } - case 8: { - u64 d8; - u64 __user *s = (u64 __user *)src; - if (__get_user(d8, s)) - goto fault; - memcpy(buf, &d8, 8); - break; - } - default: - WARN_ONCE(1, "%s: Invalid size: %zu\n", __func__, size); - return ES_UNSUPPORTED; - } - - return ES_OK; - -fault: - if (user_mode(ctxt->regs)) - error_code |= X86_PF_USER; - - ctxt->fi.vector = X86_TRAP_PF; - ctxt->fi.error_code = error_code; - ctxt->fi.cr2 = (unsigned long)src; - - return ES_EXCEPTION; -} - -static enum es_result vc_slow_virt_to_phys(struct ghcb *ghcb, struct es_em_ctxt *ctxt, - unsigned long vaddr, phys_addr_t *paddr) -{ - unsigned long va = (unsigned long)vaddr; - unsigned int level; - phys_addr_t pa; - pgd_t *pgd; - pte_t *pte; - - pgd = __va(read_cr3_pa()); - pgd = &pgd[pgd_index(va)]; - pte = lookup_address_in_pgd(pgd, va, &level); - if (!pte) { - ctxt->fi.vector = X86_TRAP_PF; - ctxt->fi.cr2 = vaddr; - ctxt->fi.error_code = 0; - - if (user_mode(ctxt->regs)) - ctxt->fi.error_code |= X86_PF_USER; - - return ES_EXCEPTION; - } - - if (WARN_ON_ONCE(pte_val(*pte) & _PAGE_ENC)) - /* Emulated MMIO to/from encrypted memory not supported */ - return ES_UNSUPPORTED; - - pa = (phys_addr_t)pte_pfn(*pte) << PAGE_SHIFT; - pa |= va & ~page_level_mask(level); - - *paddr = pa; - - return ES_OK; -} - -static enum es_result vc_ioio_check(struct es_em_ctxt *ctxt, u16 port, size_t size) -{ - BUG_ON(size > 4); - - if (user_mode(ctxt->regs)) { - struct thread_struct *t = ¤t->thread; - struct io_bitmap *iobm = t->io_bitmap; - size_t idx; - - if (!iobm) - goto fault; - - for (idx = port; idx < port + size; ++idx) { - if (test_bit(idx, iobm->bitmap)) - goto fault; - } - } - - return ES_OK; - -fault: - ctxt->fi.vector = X86_TRAP_GP; - ctxt->fi.error_code = 0; - - return ES_EXCEPTION; -} - -static __always_inline void vc_forward_exception(struct es_em_ctxt *ctxt) -{ - long error_code = ctxt->fi.error_code; - int trapnr = ctxt->fi.vector; - - ctxt->regs->orig_ax = ctxt->fi.error_code; - - switch (trapnr) { - case X86_TRAP_GP: - exc_general_protection(ctxt->regs, error_code); - break; - case X86_TRAP_UD: - exc_invalid_op(ctxt->regs); - break; - case X86_TRAP_PF: - write_cr2(ctxt->fi.cr2); - exc_page_fault(ctxt->regs, error_code); - break; - case X86_TRAP_AC: - exc_alignment_check(ctxt->regs, error_code); - break; - default: - pr_emerg("Unsupported exception in #VC instruction emulation - can't continue\n"); - BUG(); - } -} - /* Include code shared with pre-decompression boot stage */ #include "sev-shared.c" @@ -563,718 +253,6 @@ void __head early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr early_set_pages_state(vaddr, paddr, npages, SNP_PAGE_STATE_SHARED); } -/* Writes to the SVSM CAA MSR are ignored */ -static enum es_result __vc_handle_msr_caa(struct pt_regs *regs, bool write) -{ - if (write) - return ES_OK; - - regs->ax = lower_32_bits(this_cpu_read(svsm_caa_pa)); - regs->dx = upper_32_bits(this_cpu_read(svsm_caa_pa)); - - return ES_OK; -} - -/* - * TSC related accesses should not exit to the hypervisor when a guest is - * executing with Secure TSC enabled, so special handling is required for - * accesses of MSR_IA32_TSC and MSR_AMD64_GUEST_TSC_FREQ. - */ -static enum es_result __vc_handle_secure_tsc_msrs(struct pt_regs *regs, bool write) -{ - u64 tsc; - - /* - * GUEST_TSC_FREQ should not be intercepted when Secure TSC is enabled. - * Terminate the SNP guest when the interception is enabled. - */ - if (regs->cx == MSR_AMD64_GUEST_TSC_FREQ) - return ES_VMM_ERROR; - - /* - * Writes: Writing to MSR_IA32_TSC can cause subsequent reads of the TSC - * to return undefined values, so ignore all writes. - * - * Reads: Reads of MSR_IA32_TSC should return the current TSC value, use - * the value returned by rdtsc_ordered(). - */ - if (write) { - WARN_ONCE(1, "TSC MSR writes are verboten!\n"); - return ES_OK; - } - - tsc = rdtsc_ordered(); - regs->ax = lower_32_bits(tsc); - regs->dx = upper_32_bits(tsc); - - return ES_OK; -} - -static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt) -{ - struct pt_regs *regs = ctxt->regs; - enum es_result ret; - bool write; - - /* Is it a WRMSR? */ - write = ctxt->insn.opcode.bytes[1] == 0x30; - - switch (regs->cx) { - case MSR_SVSM_CAA: - return __vc_handle_msr_caa(regs, write); - case MSR_IA32_TSC: - case MSR_AMD64_GUEST_TSC_FREQ: - if (sev_status & MSR_AMD64_SNP_SECURE_TSC) - return __vc_handle_secure_tsc_msrs(regs, write); - break; - default: - break; - } - - ghcb_set_rcx(ghcb, regs->cx); - if (write) { - ghcb_set_rax(ghcb, regs->ax); - ghcb_set_rdx(ghcb, regs->dx); - } - - ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_MSR, write, 0); - - if ((ret == ES_OK) && !write) { - regs->ax = ghcb->save.rax; - regs->dx = ghcb->save.rdx; - } - - return ret; -} - -static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt) -{ - int trapnr = ctxt->fi.vector; - - if (trapnr == X86_TRAP_PF) - native_write_cr2(ctxt->fi.cr2); - - ctxt->regs->orig_ax = ctxt->fi.error_code; - do_early_exception(ctxt->regs, trapnr); -} - -static long *vc_insn_get_rm(struct es_em_ctxt *ctxt) -{ - long *reg_array; - int offset; - - reg_array = (long *)ctxt->regs; - offset = insn_get_modrm_rm_off(&ctxt->insn, ctxt->regs); - - if (offset < 0) - return NULL; - - offset /= sizeof(long); - - return reg_array + offset; -} -static enum es_result vc_do_mmio(struct ghcb *ghcb, struct es_em_ctxt *ctxt, - unsigned int bytes, bool read) -{ - u64 exit_code, exit_info_1, exit_info_2; - unsigned long ghcb_pa = __pa(ghcb); - enum es_result res; - phys_addr_t paddr; - void __user *ref; - - ref = insn_get_addr_ref(&ctxt->insn, ctxt->regs); - if (ref == (void __user *)-1L) - return ES_UNSUPPORTED; - - exit_code = read ? SVM_VMGEXIT_MMIO_READ : SVM_VMGEXIT_MMIO_WRITE; - - res = vc_slow_virt_to_phys(ghcb, ctxt, (unsigned long)ref, &paddr); - if (res != ES_OK) { - if (res == ES_EXCEPTION && !read) - ctxt->fi.error_code |= X86_PF_WRITE; - - return res; - } - - exit_info_1 = paddr; - /* Can never be greater than 8 */ - exit_info_2 = bytes; - - ghcb_set_sw_scratch(ghcb, ghcb_pa + offsetof(struct ghcb, shared_buffer)); - - return sev_es_ghcb_hv_call(ghcb, ctxt, exit_code, exit_info_1, exit_info_2); -} - -/* - * The MOVS instruction has two memory operands, which raises the - * problem that it is not known whether the access to the source or the - * destination caused the #VC exception (and hence whether an MMIO read - * or write operation needs to be emulated). - * - * Instead of playing games with walking page-tables and trying to guess - * whether the source or destination is an MMIO range, split the move - * into two operations, a read and a write with only one memory operand. - * This will cause a nested #VC exception on the MMIO address which can - * then be handled. - * - * This implementation has the benefit that it also supports MOVS where - * source _and_ destination are MMIO regions. - * - * It will slow MOVS on MMIO down a lot, but in SEV-ES guests it is a - * rare operation. If it turns out to be a performance problem the split - * operations can be moved to memcpy_fromio() and memcpy_toio(). - */ -static enum es_result vc_handle_mmio_movs(struct es_em_ctxt *ctxt, - unsigned int bytes) -{ - unsigned long ds_base, es_base; - unsigned char *src, *dst; - unsigned char buffer[8]; - enum es_result ret; - bool rep; - int off; - - ds_base = insn_get_seg_base(ctxt->regs, INAT_SEG_REG_DS); - es_base = insn_get_seg_base(ctxt->regs, INAT_SEG_REG_ES); - - if (ds_base == -1L || es_base == -1L) { - ctxt->fi.vector = X86_TRAP_GP; - ctxt->fi.error_code = 0; - return ES_EXCEPTION; - } - - src = ds_base + (unsigned char *)ctxt->regs->si; - dst = es_base + (unsigned char *)ctxt->regs->di; - - ret = vc_read_mem(ctxt, src, buffer, bytes); - if (ret != ES_OK) - return ret; - - ret = vc_write_mem(ctxt, dst, buffer, bytes); - if (ret != ES_OK) - return ret; - - if (ctxt->regs->flags & X86_EFLAGS_DF) - off = -bytes; - else - off = bytes; - - ctxt->regs->si += off; - ctxt->regs->di += off; - - rep = insn_has_rep_prefix(&ctxt->insn); - if (rep) - ctxt->regs->cx -= 1; - - if (!rep || ctxt->regs->cx == 0) - return ES_OK; - else - return ES_RETRY; -} - -static enum es_result vc_handle_mmio(struct ghcb *ghcb, struct es_em_ctxt *ctxt) -{ - struct insn *insn = &ctxt->insn; - enum insn_mmio_type mmio; - unsigned int bytes = 0; - enum es_result ret; - u8 sign_byte; - long *reg_data; - - mmio = insn_decode_mmio(insn, &bytes); - if (mmio == INSN_MMIO_DECODE_FAILED) - return ES_DECODE_FAILED; - - if (mmio != INSN_MMIO_WRITE_IMM && mmio != INSN_MMIO_MOVS) { - reg_data = insn_get_modrm_reg_ptr(insn, ctxt->regs); - if (!reg_data) - return ES_DECODE_FAILED; - } - - if (user_mode(ctxt->regs)) - return ES_UNSUPPORTED; - - switch (mmio) { - case INSN_MMIO_WRITE: - memcpy(ghcb->shared_buffer, reg_data, bytes); - ret = vc_do_mmio(ghcb, ctxt, bytes, false); - break; - case INSN_MMIO_WRITE_IMM: - memcpy(ghcb->shared_buffer, insn->immediate1.bytes, bytes); - ret = vc_do_mmio(ghcb, ctxt, bytes, false); - break; - case INSN_MMIO_READ: - ret = vc_do_mmio(ghcb, ctxt, bytes, true); - if (ret) - break; - - /* Zero-extend for 32-bit operation */ - if (bytes == 4) - *reg_data = 0; - - memcpy(reg_data, ghcb->shared_buffer, bytes); - break; - case INSN_MMIO_READ_ZERO_EXTEND: - ret = vc_do_mmio(ghcb, ctxt, bytes, true); - if (ret) - break; - - /* Zero extend based on operand size */ - memset(reg_data, 0, insn->opnd_bytes); - memcpy(reg_data, ghcb->shared_buffer, bytes); - break; - case INSN_MMIO_READ_SIGN_EXTEND: - ret = vc_do_mmio(ghcb, ctxt, bytes, true); - if (ret) - break; - - if (bytes == 1) { - u8 *val = (u8 *)ghcb->shared_buffer; - - sign_byte = (*val & 0x80) ? 0xff : 0x00; - } else { - u16 *val = (u16 *)ghcb->shared_buffer; - - sign_byte = (*val & 0x8000) ? 0xff : 0x00; - } - - /* Sign extend based on operand size */ - memset(reg_data, sign_byte, insn->opnd_bytes); - memcpy(reg_data, ghcb->shared_buffer, bytes); - break; - case INSN_MMIO_MOVS: - ret = vc_handle_mmio_movs(ctxt, bytes); - break; - default: - ret = ES_UNSUPPORTED; - break; - } - - return ret; -} - -static enum es_result vc_handle_dr7_write(struct ghcb *ghcb, - struct es_em_ctxt *ctxt) -{ - struct sev_es_runtime_data *data = this_cpu_read(runtime_data); - long val, *reg = vc_insn_get_rm(ctxt); - enum es_result ret; - - if (sev_status & MSR_AMD64_SNP_DEBUG_SWAP) - return ES_VMM_ERROR; - - if (!reg) - return ES_DECODE_FAILED; - - val = *reg; - - /* Upper 32 bits must be written as zeroes */ - if (val >> 32) { - ctxt->fi.vector = X86_TRAP_GP; - ctxt->fi.error_code = 0; - return ES_EXCEPTION; - } - - /* Clear out other reserved bits and set bit 10 */ - val = (val & 0xffff23ffL) | BIT(10); - - /* Early non-zero writes to DR7 are not supported */ - if (!data && (val & ~DR7_RESET_VALUE)) - return ES_UNSUPPORTED; - - /* Using a value of 0 for ExitInfo1 means RAX holds the value */ - ghcb_set_rax(ghcb, val); - ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_WRITE_DR7, 0, 0); - if (ret != ES_OK) - return ret; - - if (data) - data->dr7 = val; - - return ES_OK; -} - -static enum es_result vc_handle_dr7_read(struct ghcb *ghcb, - struct es_em_ctxt *ctxt) -{ - struct sev_es_runtime_data *data = this_cpu_read(runtime_data); - long *reg = vc_insn_get_rm(ctxt); - - if (sev_status & MSR_AMD64_SNP_DEBUG_SWAP) - return ES_VMM_ERROR; - - if (!reg) - return ES_DECODE_FAILED; - - if (data) - *reg = data->dr7; - else - *reg = DR7_RESET_VALUE; - - return ES_OK; -} - -static enum es_result vc_handle_wbinvd(struct ghcb *ghcb, - struct es_em_ctxt *ctxt) -{ - return sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_WBINVD, 0, 0); -} - -static enum es_result vc_handle_rdpmc(struct ghcb *ghcb, struct es_em_ctxt *ctxt) -{ - enum es_result ret; - - ghcb_set_rcx(ghcb, ctxt->regs->cx); - - ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_RDPMC, 0, 0); - if (ret != ES_OK) - return ret; - - if (!(ghcb_rax_is_valid(ghcb) && ghcb_rdx_is_valid(ghcb))) - return ES_VMM_ERROR; - - ctxt->regs->ax = ghcb->save.rax; - ctxt->regs->dx = ghcb->save.rdx; - - return ES_OK; -} - -static enum es_result vc_handle_monitor(struct ghcb *ghcb, - struct es_em_ctxt *ctxt) -{ - /* - * Treat it as a NOP and do not leak a physical address to the - * hypervisor. - */ - return ES_OK; -} - -static enum es_result vc_handle_mwait(struct ghcb *ghcb, - struct es_em_ctxt *ctxt) -{ - /* Treat the same as MONITOR/MONITORX */ - return ES_OK; -} - -static enum es_result vc_handle_vmmcall(struct ghcb *ghcb, - struct es_em_ctxt *ctxt) -{ - enum es_result ret; - - ghcb_set_rax(ghcb, ctxt->regs->ax); - ghcb_set_cpl(ghcb, user_mode(ctxt->regs) ? 3 : 0); - - if (x86_platform.hyper.sev_es_hcall_prepare) - x86_platform.hyper.sev_es_hcall_prepare(ghcb, ctxt->regs); - - ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_VMMCALL, 0, 0); - if (ret != ES_OK) - return ret; - - if (!ghcb_rax_is_valid(ghcb)) - return ES_VMM_ERROR; - - ctxt->regs->ax = ghcb->save.rax; - - /* - * Call sev_es_hcall_finish() after regs->ax is already set. - * This allows the hypervisor handler to overwrite it again if - * necessary. - */ - if (x86_platform.hyper.sev_es_hcall_finish && - !x86_platform.hyper.sev_es_hcall_finish(ghcb, ctxt->regs)) - return ES_VMM_ERROR; - - return ES_OK; -} - -static enum es_result vc_handle_trap_ac(struct ghcb *ghcb, - struct es_em_ctxt *ctxt) -{ - /* - * Calling ecx_alignment_check() directly does not work, because it - * enables IRQs and the GHCB is active. Forward the exception and call - * it later from vc_forward_exception(). - */ - ctxt->fi.vector = X86_TRAP_AC; - ctxt->fi.error_code = 0; - return ES_EXCEPTION; -} - -static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt, - struct ghcb *ghcb, - unsigned long exit_code) -{ - enum es_result result = vc_check_opcode_bytes(ctxt, exit_code); - - if (result != ES_OK) - return result; - - switch (exit_code) { - case SVM_EXIT_READ_DR7: - result = vc_handle_dr7_read(ghcb, ctxt); - break; - case SVM_EXIT_WRITE_DR7: - result = vc_handle_dr7_write(ghcb, ctxt); - break; - case SVM_EXIT_EXCP_BASE + X86_TRAP_AC: - result = vc_handle_trap_ac(ghcb, ctxt); - break; - case SVM_EXIT_RDTSC: - case SVM_EXIT_RDTSCP: - result = vc_handle_rdtsc(ghcb, ctxt, exit_code); - break; - case SVM_EXIT_RDPMC: - result = vc_handle_rdpmc(ghcb, ctxt); - break; - case SVM_EXIT_INVD: - pr_err_ratelimited("#VC exception for INVD??? Seriously???\n"); - result = ES_UNSUPPORTED; - break; - case SVM_EXIT_CPUID: - result = vc_handle_cpuid(ghcb, ctxt); - break; - case SVM_EXIT_IOIO: - result = vc_handle_ioio(ghcb, ctxt); - break; - case SVM_EXIT_MSR: - result = vc_handle_msr(ghcb, ctxt); - break; - case SVM_EXIT_VMMCALL: - result = vc_handle_vmmcall(ghcb, ctxt); - break; - case SVM_EXIT_WBINVD: - result = vc_handle_wbinvd(ghcb, ctxt); - break; - case SVM_EXIT_MONITOR: - result = vc_handle_monitor(ghcb, ctxt); - break; - case SVM_EXIT_MWAIT: - result = vc_handle_mwait(ghcb, ctxt); - break; - case SVM_EXIT_NPF: - result = vc_handle_mmio(ghcb, ctxt); - break; - default: - /* - * Unexpected #VC exception - */ - result = ES_UNSUPPORTED; - } - - return result; -} - -static __always_inline bool is_vc2_stack(unsigned long sp) -{ - return (sp >= __this_cpu_ist_bottom_va(VC2) && sp < __this_cpu_ist_top_va(VC2)); -} - -static __always_inline bool vc_from_invalid_context(struct pt_regs *regs) -{ - unsigned long sp, prev_sp; - - sp = (unsigned long)regs; - prev_sp = regs->sp; - - /* - * If the code was already executing on the VC2 stack when the #VC - * happened, let it proceed to the normal handling routine. This way the - * code executing on the VC2 stack can cause #VC exceptions to get handled. - */ - return is_vc2_stack(sp) && !is_vc2_stack(prev_sp); -} - -static bool vc_raw_handle_exception(struct pt_regs *regs, unsigned long error_code) -{ - struct ghcb_state state; - struct es_em_ctxt ctxt; - enum es_result result; - struct ghcb *ghcb; - bool ret = true; - - ghcb = __sev_get_ghcb(&state); - - vc_ghcb_invalidate(ghcb); - result = vc_init_em_ctxt(&ctxt, regs, error_code); - - if (result == ES_OK) - result = vc_handle_exitcode(&ctxt, ghcb, error_code); - - __sev_put_ghcb(&state); - - /* Done - now check the result */ - switch (result) { - case ES_OK: - vc_finish_insn(&ctxt); - break; - case ES_UNSUPPORTED: - pr_err_ratelimited("Unsupported exit-code 0x%02lx in #VC exception (IP: 0x%lx)\n", - error_code, regs->ip); - ret = false; - break; - case ES_VMM_ERROR: - pr_err_ratelimited("Failure in communication with VMM (exit-code 0x%02lx IP: 0x%lx)\n", - error_code, regs->ip); - ret = false; - break; - case ES_DECODE_FAILED: - pr_err_ratelimited("Failed to decode instruction (exit-code 0x%02lx IP: 0x%lx)\n", - error_code, regs->ip); - ret = false; - break; - case ES_EXCEPTION: - vc_forward_exception(&ctxt); - break; - case ES_RETRY: - /* Nothing to do */ - break; - default: - pr_emerg("Unknown result in %s():%d\n", __func__, result); - /* - * Emulating the instruction which caused the #VC exception - * failed - can't continue so print debug information - */ - BUG(); - } - - return ret; -} - -static __always_inline bool vc_is_db(unsigned long error_code) -{ - return error_code == SVM_EXIT_EXCP_BASE + X86_TRAP_DB; -} - -/* - * Runtime #VC exception handler when raised from kernel mode. Runs in NMI mode - * and will panic when an error happens. - */ -DEFINE_IDTENTRY_VC_KERNEL(exc_vmm_communication) -{ - irqentry_state_t irq_state; - - /* - * With the current implementation it is always possible to switch to a - * safe stack because #VC exceptions only happen at known places, like - * intercepted instructions or accesses to MMIO areas/IO ports. They can - * also happen with code instrumentation when the hypervisor intercepts - * #DB, but the critical paths are forbidden to be instrumented, so #DB - * exceptions currently also only happen in safe places. - * - * But keep this here in case the noinstr annotations are violated due - * to bug elsewhere. - */ - if (unlikely(vc_from_invalid_context(regs))) { - instrumentation_begin(); - panic("Can't handle #VC exception from unsupported context\n"); - instrumentation_end(); - } - - /* - * Handle #DB before calling into !noinstr code to avoid recursive #DB. - */ - if (vc_is_db(error_code)) { - exc_debug(regs); - return; - } - - irq_state = irqentry_nmi_enter(regs); - - instrumentation_begin(); - - if (!vc_raw_handle_exception(regs, error_code)) { - /* Show some debug info */ - show_regs(regs); - - /* Ask hypervisor to sev_es_terminate */ - sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SEV_ES_GEN_REQ); - - /* If that fails and we get here - just panic */ - panic("Returned from Terminate-Request to Hypervisor\n"); - } - - instrumentation_end(); - irqentry_nmi_exit(regs, irq_state); -} - -/* - * Runtime #VC exception handler when raised from user mode. Runs in IRQ mode - * and will kill the current task with SIGBUS when an error happens. - */ -DEFINE_IDTENTRY_VC_USER(exc_vmm_communication) -{ - /* - * Handle #DB before calling into !noinstr code to avoid recursive #DB. - */ - if (vc_is_db(error_code)) { - noist_exc_debug(regs); - return; - } - - irqentry_enter_from_user_mode(regs); - instrumentation_begin(); - - if (!vc_raw_handle_exception(regs, error_code)) { - /* - * Do not kill the machine if user-space triggered the - * exception. Send SIGBUS instead and let user-space deal with - * it. - */ - force_sig_fault(SIGBUS, BUS_OBJERR, (void __user *)0); - } - - instrumentation_end(); - irqentry_exit_to_user_mode(regs); -} - -bool __init handle_vc_boot_ghcb(struct pt_regs *regs) -{ - unsigned long exit_code = regs->orig_ax; - struct es_em_ctxt ctxt; - enum es_result result; - - vc_ghcb_invalidate(boot_ghcb); - - result = vc_init_em_ctxt(&ctxt, regs, exit_code); - if (result == ES_OK) - result = vc_handle_exitcode(&ctxt, boot_ghcb, exit_code); - - /* Done - now check the result */ - switch (result) { - case ES_OK: - vc_finish_insn(&ctxt); - break; - case ES_UNSUPPORTED: - early_printk("PANIC: Unsupported exit-code 0x%02lx in early #VC exception (IP: 0x%lx)\n", - exit_code, regs->ip); - goto fail; - case ES_VMM_ERROR: - early_printk("PANIC: Failure in communication with VMM (exit-code 0x%02lx IP: 0x%lx)\n", - exit_code, regs->ip); - goto fail; - case ES_DECODE_FAILED: - early_printk("PANIC: Failed to decode instruction (exit-code 0x%02lx IP: 0x%lx)\n", - exit_code, regs->ip); - goto fail; - case ES_EXCEPTION: - vc_early_forward_exception(&ctxt); - break; - case ES_RETRY: - /* Nothing to do */ - break; - default: - BUG(); - } - - return true; - -fail: - show_regs(regs); - - sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SEV_ES_GEN_REQ); -} - /* * Initial set up of SNP relies on information provided by the * Confidential Computing blob, which can be passed to the kernel diff --git a/arch/x86/coco/sev/Makefile b/arch/x86/coco/sev/Makefile index 2919dcfc4b02..db3255b979bd 100644 --- a/arch/x86/coco/sev/Makefile +++ b/arch/x86/coco/sev/Makefile @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += core.o sev-nmi.o +obj-y += core.o sev-nmi.o vc-handle.o # Clang 14 and older may fail to respect __no_sanitize_undefined when inlining UBSAN_SANITIZE_sev-nmi.o := n diff --git a/arch/x86/coco/sev/vc-handle.c b/arch/x86/coco/sev/vc-handle.c new file mode 100644 index 000000000000..b4895c648024 --- /dev/null +++ b/arch/x86/coco/sev/vc-handle.c @@ -0,0 +1,1061 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * AMD Memory Encryption Support + * + * Copyright (C) 2019 SUSE + * + * Author: Joerg Roedel + */ + +#define pr_fmt(fmt) "SEV: " fmt + +#include /* For show_regs() */ +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static enum es_result vc_slow_virt_to_phys(struct ghcb *ghcb, struct es_em_ctxt *ctxt, + unsigned long vaddr, phys_addr_t *paddr) +{ + unsigned long va = (unsigned long)vaddr; + unsigned int level; + phys_addr_t pa; + pgd_t *pgd; + pte_t *pte; + + pgd = __va(read_cr3_pa()); + pgd = &pgd[pgd_index(va)]; + pte = lookup_address_in_pgd(pgd, va, &level); + if (!pte) { + ctxt->fi.vector = X86_TRAP_PF; + ctxt->fi.cr2 = vaddr; + ctxt->fi.error_code = 0; + + if (user_mode(ctxt->regs)) + ctxt->fi.error_code |= X86_PF_USER; + + return ES_EXCEPTION; + } + + if (WARN_ON_ONCE(pte_val(*pte) & _PAGE_ENC)) + /* Emulated MMIO to/from encrypted memory not supported */ + return ES_UNSUPPORTED; + + pa = (phys_addr_t)pte_pfn(*pte) << PAGE_SHIFT; + pa |= va & ~page_level_mask(level); + + *paddr = pa; + + return ES_OK; +} + +static enum es_result vc_ioio_check(struct es_em_ctxt *ctxt, u16 port, size_t size) +{ + BUG_ON(size > 4); + + if (user_mode(ctxt->regs)) { + struct thread_struct *t = ¤t->thread; + struct io_bitmap *iobm = t->io_bitmap; + size_t idx; + + if (!iobm) + goto fault; + + for (idx = port; idx < port + size; ++idx) { + if (test_bit(idx, iobm->bitmap)) + goto fault; + } + } + + return ES_OK; + +fault: + ctxt->fi.vector = X86_TRAP_GP; + ctxt->fi.error_code = 0; + + return ES_EXCEPTION; +} + +void vc_forward_exception(struct es_em_ctxt *ctxt) +{ + long error_code = ctxt->fi.error_code; + int trapnr = ctxt->fi.vector; + + ctxt->regs->orig_ax = ctxt->fi.error_code; + + switch (trapnr) { + case X86_TRAP_GP: + exc_general_protection(ctxt->regs, error_code); + break; + case X86_TRAP_UD: + exc_invalid_op(ctxt->regs); + break; + case X86_TRAP_PF: + write_cr2(ctxt->fi.cr2); + exc_page_fault(ctxt->regs, error_code); + break; + case X86_TRAP_AC: + exc_alignment_check(ctxt->regs, error_code); + break; + default: + pr_emerg("Unsupported exception in #VC instruction emulation - can't continue\n"); + BUG(); + } +} + +static int vc_fetch_insn_kernel(struct es_em_ctxt *ctxt, + unsigned char *buffer) +{ + return copy_from_kernel_nofault(buffer, (unsigned char *)ctxt->regs->ip, MAX_INSN_SIZE); +} + +static enum es_result __vc_decode_user_insn(struct es_em_ctxt *ctxt) +{ + char buffer[MAX_INSN_SIZE]; + int insn_bytes; + + insn_bytes = insn_fetch_from_user_inatomic(ctxt->regs, buffer); + if (insn_bytes == 0) { + /* Nothing could be copied */ + ctxt->fi.vector = X86_TRAP_PF; + ctxt->fi.error_code = X86_PF_INSTR | X86_PF_USER; + ctxt->fi.cr2 = ctxt->regs->ip; + return ES_EXCEPTION; + } else if (insn_bytes == -EINVAL) { + /* Effective RIP could not be calculated */ + ctxt->fi.vector = X86_TRAP_GP; + ctxt->fi.error_code = 0; + ctxt->fi.cr2 = 0; + return ES_EXCEPTION; + } + + if (!insn_decode_from_regs(&ctxt->insn, ctxt->regs, buffer, insn_bytes)) + return ES_DECODE_FAILED; + + if (ctxt->insn.immediate.got) + return ES_OK; + else + return ES_DECODE_FAILED; +} + +static enum es_result __vc_decode_kern_insn(struct es_em_ctxt *ctxt) +{ + char buffer[MAX_INSN_SIZE]; + int res, ret; + + res = vc_fetch_insn_kernel(ctxt, buffer); + if (res) { + ctxt->fi.vector = X86_TRAP_PF; + ctxt->fi.error_code = X86_PF_INSTR; + ctxt->fi.cr2 = ctxt->regs->ip; + return ES_EXCEPTION; + } + + ret = insn_decode(&ctxt->insn, buffer, MAX_INSN_SIZE, INSN_MODE_64); + if (ret < 0) + return ES_DECODE_FAILED; + else + return ES_OK; +} + +static enum es_result vc_decode_insn(struct es_em_ctxt *ctxt) +{ + if (user_mode(ctxt->regs)) + return __vc_decode_user_insn(ctxt); + else + return __vc_decode_kern_insn(ctxt); +} + +static enum es_result vc_write_mem(struct es_em_ctxt *ctxt, + char *dst, char *buf, size_t size) +{ + unsigned long error_code = X86_PF_PROT | X86_PF_WRITE; + + /* + * This function uses __put_user() independent of whether kernel or user + * memory is accessed. This works fine because __put_user() does no + * sanity checks of the pointer being accessed. All that it does is + * to report when the access failed. + * + * Also, this function runs in atomic context, so __put_user() is not + * allowed to sleep. The page-fault handler detects that it is running + * in atomic context and will not try to take mmap_sem and handle the + * fault, so additional pagefault_enable()/disable() calls are not + * needed. + * + * The access can't be done via copy_to_user() here because + * vc_write_mem() must not use string instructions to access unsafe + * memory. The reason is that MOVS is emulated by the #VC handler by + * splitting the move up into a read and a write and taking a nested #VC + * exception on whatever of them is the MMIO access. Using string + * instructions here would cause infinite nesting. + */ + switch (size) { + case 1: { + u8 d1; + u8 __user *target = (u8 __user *)dst; + + memcpy(&d1, buf, 1); + if (__put_user(d1, target)) + goto fault; + break; + } + case 2: { + u16 d2; + u16 __user *target = (u16 __user *)dst; + + memcpy(&d2, buf, 2); + if (__put_user(d2, target)) + goto fault; + break; + } + case 4: { + u32 d4; + u32 __user *target = (u32 __user *)dst; + + memcpy(&d4, buf, 4); + if (__put_user(d4, target)) + goto fault; + break; + } + case 8: { + u64 d8; + u64 __user *target = (u64 __user *)dst; + + memcpy(&d8, buf, 8); + if (__put_user(d8, target)) + goto fault; + break; + } + default: + WARN_ONCE(1, "%s: Invalid size: %zu\n", __func__, size); + return ES_UNSUPPORTED; + } + + return ES_OK; + +fault: + if (user_mode(ctxt->regs)) + error_code |= X86_PF_USER; + + ctxt->fi.vector = X86_TRAP_PF; + ctxt->fi.error_code = error_code; + ctxt->fi.cr2 = (unsigned long)dst; + + return ES_EXCEPTION; +} + +static enum es_result vc_read_mem(struct es_em_ctxt *ctxt, + char *src, char *buf, size_t size) +{ + unsigned long error_code = X86_PF_PROT; + + /* + * This function uses __get_user() independent of whether kernel or user + * memory is accessed. This works fine because __get_user() does no + * sanity checks of the pointer being accessed. All that it does is + * to report when the access failed. + * + * Also, this function runs in atomic context, so __get_user() is not + * allowed to sleep. The page-fault handler detects that it is running + * in atomic context and will not try to take mmap_sem and handle the + * fault, so additional pagefault_enable()/disable() calls are not + * needed. + * + * The access can't be done via copy_from_user() here because + * vc_read_mem() must not use string instructions to access unsafe + * memory. The reason is that MOVS is emulated by the #VC handler by + * splitting the move up into a read and a write and taking a nested #VC + * exception on whatever of them is the MMIO access. Using string + * instructions here would cause infinite nesting. + */ + switch (size) { + case 1: { + u8 d1; + u8 __user *s = (u8 __user *)src; + + if (__get_user(d1, s)) + goto fault; + memcpy(buf, &d1, 1); + break; + } + case 2: { + u16 d2; + u16 __user *s = (u16 __user *)src; + + if (__get_user(d2, s)) + goto fault; + memcpy(buf, &d2, 2); + break; + } + case 4: { + u32 d4; + u32 __user *s = (u32 __user *)src; + + if (__get_user(d4, s)) + goto fault; + memcpy(buf, &d4, 4); + break; + } + case 8: { + u64 d8; + u64 __user *s = (u64 __user *)src; + if (__get_user(d8, s)) + goto fault; + memcpy(buf, &d8, 8); + break; + } + default: + WARN_ONCE(1, "%s: Invalid size: %zu\n", __func__, size); + return ES_UNSUPPORTED; + } + + return ES_OK; + +fault: + if (user_mode(ctxt->regs)) + error_code |= X86_PF_USER; + + ctxt->fi.vector = X86_TRAP_PF; + ctxt->fi.error_code = error_code; + ctxt->fi.cr2 = (unsigned long)src; + + return ES_EXCEPTION; +} + +#define sev_printk(fmt, ...) printk(fmt, ##__VA_ARGS__) + +#include "vc-shared.c" + +/* Writes to the SVSM CAA MSR are ignored */ +static enum es_result __vc_handle_msr_caa(struct pt_regs *regs, bool write) +{ + if (write) + return ES_OK; + + regs->ax = lower_32_bits(this_cpu_read(svsm_caa_pa)); + regs->dx = upper_32_bits(this_cpu_read(svsm_caa_pa)); + + return ES_OK; +} + +/* + * TSC related accesses should not exit to the hypervisor when a guest is + * executing with Secure TSC enabled, so special handling is required for + * accesses of MSR_IA32_TSC and MSR_AMD64_GUEST_TSC_FREQ. + */ +static enum es_result __vc_handle_secure_tsc_msrs(struct pt_regs *regs, bool write) +{ + u64 tsc; + + /* + * GUEST_TSC_FREQ should not be intercepted when Secure TSC is enabled. + * Terminate the SNP guest when the interception is enabled. + */ + if (regs->cx == MSR_AMD64_GUEST_TSC_FREQ) + return ES_VMM_ERROR; + + /* + * Writes: Writing to MSR_IA32_TSC can cause subsequent reads of the TSC + * to return undefined values, so ignore all writes. + * + * Reads: Reads of MSR_IA32_TSC should return the current TSC value, use + * the value returned by rdtsc_ordered(). + */ + if (write) { + WARN_ONCE(1, "TSC MSR writes are verboten!\n"); + return ES_OK; + } + + tsc = rdtsc_ordered(); + regs->ax = lower_32_bits(tsc); + regs->dx = upper_32_bits(tsc); + + return ES_OK; +} + +static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt) +{ + struct pt_regs *regs = ctxt->regs; + enum es_result ret; + bool write; + + /* Is it a WRMSR? */ + write = ctxt->insn.opcode.bytes[1] == 0x30; + + switch (regs->cx) { + case MSR_SVSM_CAA: + return __vc_handle_msr_caa(regs, write); + case MSR_IA32_TSC: + case MSR_AMD64_GUEST_TSC_FREQ: + if (sev_status & MSR_AMD64_SNP_SECURE_TSC) + return __vc_handle_secure_tsc_msrs(regs, write); + break; + default: + break; + } + + ghcb_set_rcx(ghcb, regs->cx); + if (write) { + ghcb_set_rax(ghcb, regs->ax); + ghcb_set_rdx(ghcb, regs->dx); + } + + ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_MSR, write, 0); + + if ((ret == ES_OK) && !write) { + regs->ax = ghcb->save.rax; + regs->dx = ghcb->save.rdx; + } + + return ret; +} + +static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt) +{ + int trapnr = ctxt->fi.vector; + + if (trapnr == X86_TRAP_PF) + native_write_cr2(ctxt->fi.cr2); + + ctxt->regs->orig_ax = ctxt->fi.error_code; + do_early_exception(ctxt->regs, trapnr); +} + +static long *vc_insn_get_rm(struct es_em_ctxt *ctxt) +{ + long *reg_array; + int offset; + + reg_array = (long *)ctxt->regs; + offset = insn_get_modrm_rm_off(&ctxt->insn, ctxt->regs); + + if (offset < 0) + return NULL; + + offset /= sizeof(long); + + return reg_array + offset; +} +static enum es_result vc_do_mmio(struct ghcb *ghcb, struct es_em_ctxt *ctxt, + unsigned int bytes, bool read) +{ + u64 exit_code, exit_info_1, exit_info_2; + unsigned long ghcb_pa = __pa(ghcb); + enum es_result res; + phys_addr_t paddr; + void __user *ref; + + ref = insn_get_addr_ref(&ctxt->insn, ctxt->regs); + if (ref == (void __user *)-1L) + return ES_UNSUPPORTED; + + exit_code = read ? SVM_VMGEXIT_MMIO_READ : SVM_VMGEXIT_MMIO_WRITE; + + res = vc_slow_virt_to_phys(ghcb, ctxt, (unsigned long)ref, &paddr); + if (res != ES_OK) { + if (res == ES_EXCEPTION && !read) + ctxt->fi.error_code |= X86_PF_WRITE; + + return res; + } + + exit_info_1 = paddr; + /* Can never be greater than 8 */ + exit_info_2 = bytes; + + ghcb_set_sw_scratch(ghcb, ghcb_pa + offsetof(struct ghcb, shared_buffer)); + + return sev_es_ghcb_hv_call(ghcb, ctxt, exit_code, exit_info_1, exit_info_2); +} + +/* + * The MOVS instruction has two memory operands, which raises the + * problem that it is not known whether the access to the source or the + * destination caused the #VC exception (and hence whether an MMIO read + * or write operation needs to be emulated). + * + * Instead of playing games with walking page-tables and trying to guess + * whether the source or destination is an MMIO range, split the move + * into two operations, a read and a write with only one memory operand. + * This will cause a nested #VC exception on the MMIO address which can + * then be handled. + * + * This implementation has the benefit that it also supports MOVS where + * source _and_ destination are MMIO regions. + * + * It will slow MOVS on MMIO down a lot, but in SEV-ES guests it is a + * rare operation. If it turns out to be a performance problem the split + * operations can be moved to memcpy_fromio() and memcpy_toio(). + */ +static enum es_result vc_handle_mmio_movs(struct es_em_ctxt *ctxt, + unsigned int bytes) +{ + unsigned long ds_base, es_base; + unsigned char *src, *dst; + unsigned char buffer[8]; + enum es_result ret; + bool rep; + int off; + + ds_base = insn_get_seg_base(ctxt->regs, INAT_SEG_REG_DS); + es_base = insn_get_seg_base(ctxt->regs, INAT_SEG_REG_ES); + + if (ds_base == -1L || es_base == -1L) { + ctxt->fi.vector = X86_TRAP_GP; + ctxt->fi.error_code = 0; + return ES_EXCEPTION; + } + + src = ds_base + (unsigned char *)ctxt->regs->si; + dst = es_base + (unsigned char *)ctxt->regs->di; + + ret = vc_read_mem(ctxt, src, buffer, bytes); + if (ret != ES_OK) + return ret; + + ret = vc_write_mem(ctxt, dst, buffer, bytes); + if (ret != ES_OK) + return ret; + + if (ctxt->regs->flags & X86_EFLAGS_DF) + off = -bytes; + else + off = bytes; + + ctxt->regs->si += off; + ctxt->regs->di += off; + + rep = insn_has_rep_prefix(&ctxt->insn); + if (rep) + ctxt->regs->cx -= 1; + + if (!rep || ctxt->regs->cx == 0) + return ES_OK; + else + return ES_RETRY; +} + +static enum es_result vc_handle_mmio(struct ghcb *ghcb, struct es_em_ctxt *ctxt) +{ + struct insn *insn = &ctxt->insn; + enum insn_mmio_type mmio; + unsigned int bytes = 0; + enum es_result ret; + u8 sign_byte; + long *reg_data; + + mmio = insn_decode_mmio(insn, &bytes); + if (mmio == INSN_MMIO_DECODE_FAILED) + return ES_DECODE_FAILED; + + if (mmio != INSN_MMIO_WRITE_IMM && mmio != INSN_MMIO_MOVS) { + reg_data = insn_get_modrm_reg_ptr(insn, ctxt->regs); + if (!reg_data) + return ES_DECODE_FAILED; + } + + if (user_mode(ctxt->regs)) + return ES_UNSUPPORTED; + + switch (mmio) { + case INSN_MMIO_WRITE: + memcpy(ghcb->shared_buffer, reg_data, bytes); + ret = vc_do_mmio(ghcb, ctxt, bytes, false); + break; + case INSN_MMIO_WRITE_IMM: + memcpy(ghcb->shared_buffer, insn->immediate1.bytes, bytes); + ret = vc_do_mmio(ghcb, ctxt, bytes, false); + break; + case INSN_MMIO_READ: + ret = vc_do_mmio(ghcb, ctxt, bytes, true); + if (ret) + break; + + /* Zero-extend for 32-bit operation */ + if (bytes == 4) + *reg_data = 0; + + memcpy(reg_data, ghcb->shared_buffer, bytes); + break; + case INSN_MMIO_READ_ZERO_EXTEND: + ret = vc_do_mmio(ghcb, ctxt, bytes, true); + if (ret) + break; + + /* Zero extend based on operand size */ + memset(reg_data, 0, insn->opnd_bytes); + memcpy(reg_data, ghcb->shared_buffer, bytes); + break; + case INSN_MMIO_READ_SIGN_EXTEND: + ret = vc_do_mmio(ghcb, ctxt, bytes, true); + if (ret) + break; + + if (bytes == 1) { + u8 *val = (u8 *)ghcb->shared_buffer; + + sign_byte = (*val & 0x80) ? 0xff : 0x00; + } else { + u16 *val = (u16 *)ghcb->shared_buffer; + + sign_byte = (*val & 0x8000) ? 0xff : 0x00; + } + + /* Sign extend based on operand size */ + memset(reg_data, sign_byte, insn->opnd_bytes); + memcpy(reg_data, ghcb->shared_buffer, bytes); + break; + case INSN_MMIO_MOVS: + ret = vc_handle_mmio_movs(ctxt, bytes); + break; + default: + ret = ES_UNSUPPORTED; + break; + } + + return ret; +} + +static enum es_result vc_handle_dr7_write(struct ghcb *ghcb, + struct es_em_ctxt *ctxt) +{ + struct sev_es_runtime_data *data = this_cpu_read(runtime_data); + long val, *reg = vc_insn_get_rm(ctxt); + enum es_result ret; + + if (sev_status & MSR_AMD64_SNP_DEBUG_SWAP) + return ES_VMM_ERROR; + + if (!reg) + return ES_DECODE_FAILED; + + val = *reg; + + /* Upper 32 bits must be written as zeroes */ + if (val >> 32) { + ctxt->fi.vector = X86_TRAP_GP; + ctxt->fi.error_code = 0; + return ES_EXCEPTION; + } + + /* Clear out other reserved bits and set bit 10 */ + val = (val & 0xffff23ffL) | BIT(10); + + /* Early non-zero writes to DR7 are not supported */ + if (!data && (val & ~DR7_RESET_VALUE)) + return ES_UNSUPPORTED; + + /* Using a value of 0 for ExitInfo1 means RAX holds the value */ + ghcb_set_rax(ghcb, val); + ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_WRITE_DR7, 0, 0); + if (ret != ES_OK) + return ret; + + if (data) + data->dr7 = val; + + return ES_OK; +} + +static enum es_result vc_handle_dr7_read(struct ghcb *ghcb, + struct es_em_ctxt *ctxt) +{ + struct sev_es_runtime_data *data = this_cpu_read(runtime_data); + long *reg = vc_insn_get_rm(ctxt); + + if (sev_status & MSR_AMD64_SNP_DEBUG_SWAP) + return ES_VMM_ERROR; + + if (!reg) + return ES_DECODE_FAILED; + + if (data) + *reg = data->dr7; + else + *reg = DR7_RESET_VALUE; + + return ES_OK; +} + +static enum es_result vc_handle_wbinvd(struct ghcb *ghcb, + struct es_em_ctxt *ctxt) +{ + return sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_WBINVD, 0, 0); +} + +static enum es_result vc_handle_rdpmc(struct ghcb *ghcb, struct es_em_ctxt *ctxt) +{ + enum es_result ret; + + ghcb_set_rcx(ghcb, ctxt->regs->cx); + + ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_RDPMC, 0, 0); + if (ret != ES_OK) + return ret; + + if (!(ghcb_rax_is_valid(ghcb) && ghcb_rdx_is_valid(ghcb))) + return ES_VMM_ERROR; + + ctxt->regs->ax = ghcb->save.rax; + ctxt->regs->dx = ghcb->save.rdx; + + return ES_OK; +} + +static enum es_result vc_handle_monitor(struct ghcb *ghcb, + struct es_em_ctxt *ctxt) +{ + /* + * Treat it as a NOP and do not leak a physical address to the + * hypervisor. + */ + return ES_OK; +} + +static enum es_result vc_handle_mwait(struct ghcb *ghcb, + struct es_em_ctxt *ctxt) +{ + /* Treat the same as MONITOR/MONITORX */ + return ES_OK; +} + +static enum es_result vc_handle_vmmcall(struct ghcb *ghcb, + struct es_em_ctxt *ctxt) +{ + enum es_result ret; + + ghcb_set_rax(ghcb, ctxt->regs->ax); + ghcb_set_cpl(ghcb, user_mode(ctxt->regs) ? 3 : 0); + + if (x86_platform.hyper.sev_es_hcall_prepare) + x86_platform.hyper.sev_es_hcall_prepare(ghcb, ctxt->regs); + + ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_VMMCALL, 0, 0); + if (ret != ES_OK) + return ret; + + if (!ghcb_rax_is_valid(ghcb)) + return ES_VMM_ERROR; + + ctxt->regs->ax = ghcb->save.rax; + + /* + * Call sev_es_hcall_finish() after regs->ax is already set. + * This allows the hypervisor handler to overwrite it again if + * necessary. + */ + if (x86_platform.hyper.sev_es_hcall_finish && + !x86_platform.hyper.sev_es_hcall_finish(ghcb, ctxt->regs)) + return ES_VMM_ERROR; + + return ES_OK; +} + +static enum es_result vc_handle_trap_ac(struct ghcb *ghcb, + struct es_em_ctxt *ctxt) +{ + /* + * Calling ecx_alignment_check() directly does not work, because it + * enables IRQs and the GHCB is active. Forward the exception and call + * it later from vc_forward_exception(). + */ + ctxt->fi.vector = X86_TRAP_AC; + ctxt->fi.error_code = 0; + return ES_EXCEPTION; +} + +static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt, + struct ghcb *ghcb, + unsigned long exit_code) +{ + enum es_result result = vc_check_opcode_bytes(ctxt, exit_code); + + if (result != ES_OK) + return result; + + switch (exit_code) { + case SVM_EXIT_READ_DR7: + result = vc_handle_dr7_read(ghcb, ctxt); + break; + case SVM_EXIT_WRITE_DR7: + result = vc_handle_dr7_write(ghcb, ctxt); + break; + case SVM_EXIT_EXCP_BASE + X86_TRAP_AC: + result = vc_handle_trap_ac(ghcb, ctxt); + break; + case SVM_EXIT_RDTSC: + case SVM_EXIT_RDTSCP: + result = vc_handle_rdtsc(ghcb, ctxt, exit_code); + break; + case SVM_EXIT_RDPMC: + result = vc_handle_rdpmc(ghcb, ctxt); + break; + case SVM_EXIT_INVD: + pr_err_ratelimited("#VC exception for INVD??? Seriously???\n"); + result = ES_UNSUPPORTED; + break; + case SVM_EXIT_CPUID: + result = vc_handle_cpuid(ghcb, ctxt); + break; + case SVM_EXIT_IOIO: + result = vc_handle_ioio(ghcb, ctxt); + break; + case SVM_EXIT_MSR: + result = vc_handle_msr(ghcb, ctxt); + break; + case SVM_EXIT_VMMCALL: + result = vc_handle_vmmcall(ghcb, ctxt); + break; + case SVM_EXIT_WBINVD: + result = vc_handle_wbinvd(ghcb, ctxt); + break; + case SVM_EXIT_MONITOR: + result = vc_handle_monitor(ghcb, ctxt); + break; + case SVM_EXIT_MWAIT: + result = vc_handle_mwait(ghcb, ctxt); + break; + case SVM_EXIT_NPF: + result = vc_handle_mmio(ghcb, ctxt); + break; + default: + /* + * Unexpected #VC exception + */ + result = ES_UNSUPPORTED; + } + + return result; +} + +static __always_inline bool is_vc2_stack(unsigned long sp) +{ + return (sp >= __this_cpu_ist_bottom_va(VC2) && sp < __this_cpu_ist_top_va(VC2)); +} + +static __always_inline bool vc_from_invalid_context(struct pt_regs *regs) +{ + unsigned long sp, prev_sp; + + sp = (unsigned long)regs; + prev_sp = regs->sp; + + /* + * If the code was already executing on the VC2 stack when the #VC + * happened, let it proceed to the normal handling routine. This way the + * code executing on the VC2 stack can cause #VC exceptions to get handled. + */ + return is_vc2_stack(sp) && !is_vc2_stack(prev_sp); +} + +static bool vc_raw_handle_exception(struct pt_regs *regs, unsigned long error_code) +{ + struct ghcb_state state; + struct es_em_ctxt ctxt; + enum es_result result; + struct ghcb *ghcb; + bool ret = true; + + ghcb = __sev_get_ghcb(&state); + + vc_ghcb_invalidate(ghcb); + result = vc_init_em_ctxt(&ctxt, regs, error_code); + + if (result == ES_OK) + result = vc_handle_exitcode(&ctxt, ghcb, error_code); + + __sev_put_ghcb(&state); + + /* Done - now check the result */ + switch (result) { + case ES_OK: + vc_finish_insn(&ctxt); + break; + case ES_UNSUPPORTED: + pr_err_ratelimited("Unsupported exit-code 0x%02lx in #VC exception (IP: 0x%lx)\n", + error_code, regs->ip); + ret = false; + break; + case ES_VMM_ERROR: + pr_err_ratelimited("Failure in communication with VMM (exit-code 0x%02lx IP: 0x%lx)\n", + error_code, regs->ip); + ret = false; + break; + case ES_DECODE_FAILED: + pr_err_ratelimited("Failed to decode instruction (exit-code 0x%02lx IP: 0x%lx)\n", + error_code, regs->ip); + ret = false; + break; + case ES_EXCEPTION: + vc_forward_exception(&ctxt); + break; + case ES_RETRY: + /* Nothing to do */ + break; + default: + pr_emerg("Unknown result in %s():%d\n", __func__, result); + /* + * Emulating the instruction which caused the #VC exception + * failed - can't continue so print debug information + */ + BUG(); + } + + return ret; +} + +static __always_inline bool vc_is_db(unsigned long error_code) +{ + return error_code == SVM_EXIT_EXCP_BASE + X86_TRAP_DB; +} + +/* + * Runtime #VC exception handler when raised from kernel mode. Runs in NMI mode + * and will panic when an error happens. + */ +DEFINE_IDTENTRY_VC_KERNEL(exc_vmm_communication) +{ + irqentry_state_t irq_state; + + /* + * With the current implementation it is always possible to switch to a + * safe stack because #VC exceptions only happen at known places, like + * intercepted instructions or accesses to MMIO areas/IO ports. They can + * also happen with code instrumentation when the hypervisor intercepts + * #DB, but the critical paths are forbidden to be instrumented, so #DB + * exceptions currently also only happen in safe places. + * + * But keep this here in case the noinstr annotations are violated due + * to bug elsewhere. + */ + if (unlikely(vc_from_invalid_context(regs))) { + instrumentation_begin(); + panic("Can't handle #VC exception from unsupported context\n"); + instrumentation_end(); + } + + /* + * Handle #DB before calling into !noinstr code to avoid recursive #DB. + */ + if (vc_is_db(error_code)) { + exc_debug(regs); + return; + } + + irq_state = irqentry_nmi_enter(regs); + + instrumentation_begin(); + + if (!vc_raw_handle_exception(regs, error_code)) { + /* Show some debug info */ + show_regs(regs); + + /* Ask hypervisor to sev_es_terminate */ + sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SEV_ES_GEN_REQ); + + /* If that fails and we get here - just panic */ + panic("Returned from Terminate-Request to Hypervisor\n"); + } + + instrumentation_end(); + irqentry_nmi_exit(regs, irq_state); +} + +/* + * Runtime #VC exception handler when raised from user mode. Runs in IRQ mode + * and will kill the current task with SIGBUS when an error happens. + */ +DEFINE_IDTENTRY_VC_USER(exc_vmm_communication) +{ + /* + * Handle #DB before calling into !noinstr code to avoid recursive #DB. + */ + if (vc_is_db(error_code)) { + noist_exc_debug(regs); + return; + } + + irqentry_enter_from_user_mode(regs); + instrumentation_begin(); + + if (!vc_raw_handle_exception(regs, error_code)) { + /* + * Do not kill the machine if user-space triggered the + * exception. Send SIGBUS instead and let user-space deal with + * it. + */ + force_sig_fault(SIGBUS, BUS_OBJERR, (void __user *)0); + } + + instrumentation_end(); + irqentry_exit_to_user_mode(regs); +} + +bool __init handle_vc_boot_ghcb(struct pt_regs *regs) +{ + unsigned long exit_code = regs->orig_ax; + struct es_em_ctxt ctxt; + enum es_result result; + + vc_ghcb_invalidate(boot_ghcb); + + result = vc_init_em_ctxt(&ctxt, regs, exit_code); + if (result == ES_OK) + result = vc_handle_exitcode(&ctxt, boot_ghcb, exit_code); + + /* Done - now check the result */ + switch (result) { + case ES_OK: + vc_finish_insn(&ctxt); + break; + case ES_UNSUPPORTED: + early_printk("PANIC: Unsupported exit-code 0x%02lx in early #VC exception (IP: 0x%lx)\n", + exit_code, regs->ip); + goto fail; + case ES_VMM_ERROR: + early_printk("PANIC: Failure in communication with VMM (exit-code 0x%02lx IP: 0x%lx)\n", + exit_code, regs->ip); + goto fail; + case ES_DECODE_FAILED: + early_printk("PANIC: Failed to decode instruction (exit-code 0x%02lx IP: 0x%lx)\n", + exit_code, regs->ip); + goto fail; + case ES_EXCEPTION: + vc_early_forward_exception(&ctxt); + break; + case ES_RETRY: + /* Nothing to do */ + break; + default: + BUG(); + } + + return true; + +fail: + show_regs(regs); + + sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SEV_ES_GEN_REQ); +} + diff --git a/arch/x86/coco/sev/vc-shared.c b/arch/x86/coco/sev/vc-shared.c new file mode 100644 index 000000000000..2c0ab0fdc060 --- /dev/null +++ b/arch/x86/coco/sev/vc-shared.c @@ -0,0 +1,504 @@ +// SPDX-License-Identifier: GPL-2.0 + +static enum es_result vc_check_opcode_bytes(struct es_em_ctxt *ctxt, + unsigned long exit_code) +{ + unsigned int opcode = (unsigned int)ctxt->insn.opcode.value; + u8 modrm = ctxt->insn.modrm.value; + + switch (exit_code) { + + case SVM_EXIT_IOIO: + case SVM_EXIT_NPF: + /* handled separately */ + return ES_OK; + + case SVM_EXIT_CPUID: + if (opcode == 0xa20f) + return ES_OK; + break; + + case SVM_EXIT_INVD: + if (opcode == 0x080f) + return ES_OK; + break; + + case SVM_EXIT_MONITOR: + /* MONITOR and MONITORX instructions generate the same error code */ + if (opcode == 0x010f && (modrm == 0xc8 || modrm == 0xfa)) + return ES_OK; + break; + + case SVM_EXIT_MWAIT: + /* MWAIT and MWAITX instructions generate the same error code */ + if (opcode == 0x010f && (modrm == 0xc9 || modrm == 0xfb)) + return ES_OK; + break; + + case SVM_EXIT_MSR: + /* RDMSR */ + if (opcode == 0x320f || + /* WRMSR */ + opcode == 0x300f) + return ES_OK; + break; + + case SVM_EXIT_RDPMC: + if (opcode == 0x330f) + return ES_OK; + break; + + case SVM_EXIT_RDTSC: + if (opcode == 0x310f) + return ES_OK; + break; + + case SVM_EXIT_RDTSCP: + if (opcode == 0x010f && modrm == 0xf9) + return ES_OK; + break; + + case SVM_EXIT_READ_DR7: + if (opcode == 0x210f && + X86_MODRM_REG(ctxt->insn.modrm.value) == 7) + return ES_OK; + break; + + case SVM_EXIT_VMMCALL: + if (opcode == 0x010f && modrm == 0xd9) + return ES_OK; + + break; + + case SVM_EXIT_WRITE_DR7: + if (opcode == 0x230f && + X86_MODRM_REG(ctxt->insn.modrm.value) == 7) + return ES_OK; + break; + + case SVM_EXIT_WBINVD: + if (opcode == 0x90f) + return ES_OK; + break; + + default: + break; + } + + sev_printk(KERN_ERR "Wrong/unhandled opcode bytes: 0x%x, exit_code: 0x%lx, rIP: 0x%lx\n", + opcode, exit_code, ctxt->regs->ip); + + return ES_UNSUPPORTED; +} + +static bool vc_decoding_needed(unsigned long exit_code) +{ + /* Exceptions don't require to decode the instruction */ + return !(exit_code >= SVM_EXIT_EXCP_BASE && + exit_code <= SVM_EXIT_LAST_EXCP); +} + +static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt, + struct pt_regs *regs, + unsigned long exit_code) +{ + enum es_result ret = ES_OK; + + memset(ctxt, 0, sizeof(*ctxt)); + ctxt->regs = regs; + + if (vc_decoding_needed(exit_code)) + ret = vc_decode_insn(ctxt); + + return ret; +} + +static void vc_finish_insn(struct es_em_ctxt *ctxt) +{ + ctxt->regs->ip += ctxt->insn.length; +} + +static enum es_result vc_insn_string_check(struct es_em_ctxt *ctxt, + unsigned long address, + bool write) +{ + if (user_mode(ctxt->regs) && fault_in_kernel_space(address)) { + ctxt->fi.vector = X86_TRAP_PF; + ctxt->fi.error_code = X86_PF_USER; + ctxt->fi.cr2 = address; + if (write) + ctxt->fi.error_code |= X86_PF_WRITE; + + return ES_EXCEPTION; + } + + return ES_OK; +} + +static enum es_result vc_insn_string_read(struct es_em_ctxt *ctxt, + void *src, char *buf, + unsigned int data_size, + unsigned int count, + bool backwards) +{ + int i, b = backwards ? -1 : 1; + unsigned long address = (unsigned long)src; + enum es_result ret; + + ret = vc_insn_string_check(ctxt, address, false); + if (ret != ES_OK) + return ret; + + for (i = 0; i < count; i++) { + void *s = src + (i * data_size * b); + char *d = buf + (i * data_size); + + ret = vc_read_mem(ctxt, s, d, data_size); + if (ret != ES_OK) + break; + } + + return ret; +} + +static enum es_result vc_insn_string_write(struct es_em_ctxt *ctxt, + void *dst, char *buf, + unsigned int data_size, + unsigned int count, + bool backwards) +{ + int i, s = backwards ? -1 : 1; + unsigned long address = (unsigned long)dst; + enum es_result ret; + + ret = vc_insn_string_check(ctxt, address, true); + if (ret != ES_OK) + return ret; + + for (i = 0; i < count; i++) { + void *d = dst + (i * data_size * s); + char *b = buf + (i * data_size); + + ret = vc_write_mem(ctxt, d, b, data_size); + if (ret != ES_OK) + break; + } + + return ret; +} + +#define IOIO_TYPE_STR BIT(2) +#define IOIO_TYPE_IN 1 +#define IOIO_TYPE_INS (IOIO_TYPE_IN | IOIO_TYPE_STR) +#define IOIO_TYPE_OUT 0 +#define IOIO_TYPE_OUTS (IOIO_TYPE_OUT | IOIO_TYPE_STR) + +#define IOIO_REP BIT(3) + +#define IOIO_ADDR_64 BIT(9) +#define IOIO_ADDR_32 BIT(8) +#define IOIO_ADDR_16 BIT(7) + +#define IOIO_DATA_32 BIT(6) +#define IOIO_DATA_16 BIT(5) +#define IOIO_DATA_8 BIT(4) + +#define IOIO_SEG_ES (0 << 10) +#define IOIO_SEG_DS (3 << 10) + +static enum es_result vc_ioio_exitinfo(struct es_em_ctxt *ctxt, u64 *exitinfo) +{ + struct insn *insn = &ctxt->insn; + size_t size; + u64 port; + + *exitinfo = 0; + + switch (insn->opcode.bytes[0]) { + /* INS opcodes */ + case 0x6c: + case 0x6d: + *exitinfo |= IOIO_TYPE_INS; + *exitinfo |= IOIO_SEG_ES; + port = ctxt->regs->dx & 0xffff; + break; + + /* OUTS opcodes */ + case 0x6e: + case 0x6f: + *exitinfo |= IOIO_TYPE_OUTS; + *exitinfo |= IOIO_SEG_DS; + port = ctxt->regs->dx & 0xffff; + break; + + /* IN immediate opcodes */ + case 0xe4: + case 0xe5: + *exitinfo |= IOIO_TYPE_IN; + port = (u8)insn->immediate.value & 0xffff; + break; + + /* OUT immediate opcodes */ + case 0xe6: + case 0xe7: + *exitinfo |= IOIO_TYPE_OUT; + port = (u8)insn->immediate.value & 0xffff; + break; + + /* IN register opcodes */ + case 0xec: + case 0xed: + *exitinfo |= IOIO_TYPE_IN; + port = ctxt->regs->dx & 0xffff; + break; + + /* OUT register opcodes */ + case 0xee: + case 0xef: + *exitinfo |= IOIO_TYPE_OUT; + port = ctxt->regs->dx & 0xffff; + break; + + default: + return ES_DECODE_FAILED; + } + + *exitinfo |= port << 16; + + switch (insn->opcode.bytes[0]) { + case 0x6c: + case 0x6e: + case 0xe4: + case 0xe6: + case 0xec: + case 0xee: + /* Single byte opcodes */ + *exitinfo |= IOIO_DATA_8; + size = 1; + break; + default: + /* Length determined by instruction parsing */ + *exitinfo |= (insn->opnd_bytes == 2) ? IOIO_DATA_16 + : IOIO_DATA_32; + size = (insn->opnd_bytes == 2) ? 2 : 4; + } + + switch (insn->addr_bytes) { + case 2: + *exitinfo |= IOIO_ADDR_16; + break; + case 4: + *exitinfo |= IOIO_ADDR_32; + break; + case 8: + *exitinfo |= IOIO_ADDR_64; + break; + } + + if (insn_has_rep_prefix(insn)) + *exitinfo |= IOIO_REP; + + return vc_ioio_check(ctxt, (u16)port, size); +} + +static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt) +{ + struct pt_regs *regs = ctxt->regs; + u64 exit_info_1, exit_info_2; + enum es_result ret; + + ret = vc_ioio_exitinfo(ctxt, &exit_info_1); + if (ret != ES_OK) + return ret; + + if (exit_info_1 & IOIO_TYPE_STR) { + + /* (REP) INS/OUTS */ + + bool df = ((regs->flags & X86_EFLAGS_DF) == X86_EFLAGS_DF); + unsigned int io_bytes, exit_bytes; + unsigned int ghcb_count, op_count; + unsigned long es_base; + u64 sw_scratch; + + /* + * For the string variants with rep prefix the amount of in/out + * operations per #VC exception is limited so that the kernel + * has a chance to take interrupts and re-schedule while the + * instruction is emulated. + */ + io_bytes = (exit_info_1 >> 4) & 0x7; + ghcb_count = sizeof(ghcb->shared_buffer) / io_bytes; + + op_count = (exit_info_1 & IOIO_REP) ? regs->cx : 1; + exit_info_2 = min(op_count, ghcb_count); + exit_bytes = exit_info_2 * io_bytes; + + es_base = insn_get_seg_base(ctxt->regs, INAT_SEG_REG_ES); + + /* Read bytes of OUTS into the shared buffer */ + if (!(exit_info_1 & IOIO_TYPE_IN)) { + ret = vc_insn_string_read(ctxt, + (void *)(es_base + regs->si), + ghcb->shared_buffer, io_bytes, + exit_info_2, df); + if (ret) + return ret; + } + + /* + * Issue an VMGEXIT to the HV to consume the bytes from the + * shared buffer or to have it write them into the shared buffer + * depending on the instruction: OUTS or INS. + */ + sw_scratch = __pa(ghcb) + offsetof(struct ghcb, shared_buffer); + ghcb_set_sw_scratch(ghcb, sw_scratch); + ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_IOIO, + exit_info_1, exit_info_2); + if (ret != ES_OK) + return ret; + + /* Read bytes from shared buffer into the guest's destination. */ + if (exit_info_1 & IOIO_TYPE_IN) { + ret = vc_insn_string_write(ctxt, + (void *)(es_base + regs->di), + ghcb->shared_buffer, io_bytes, + exit_info_2, df); + if (ret) + return ret; + + if (df) + regs->di -= exit_bytes; + else + regs->di += exit_bytes; + } else { + if (df) + regs->si -= exit_bytes; + else + regs->si += exit_bytes; + } + + if (exit_info_1 & IOIO_REP) + regs->cx -= exit_info_2; + + ret = regs->cx ? ES_RETRY : ES_OK; + + } else { + + /* IN/OUT into/from rAX */ + + int bits = (exit_info_1 & 0x70) >> 1; + u64 rax = 0; + + if (!(exit_info_1 & IOIO_TYPE_IN)) + rax = lower_bits(regs->ax, bits); + + ghcb_set_rax(ghcb, rax); + + ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_IOIO, exit_info_1, 0); + if (ret != ES_OK) + return ret; + + if (exit_info_1 & IOIO_TYPE_IN) { + if (!ghcb_rax_is_valid(ghcb)) + return ES_VMM_ERROR; + regs->ax = lower_bits(ghcb->save.rax, bits); + } + } + + return ret; +} + +static int vc_handle_cpuid_snp(struct ghcb *ghcb, struct es_em_ctxt *ctxt) +{ + struct pt_regs *regs = ctxt->regs; + struct cpuid_leaf leaf; + int ret; + + leaf.fn = regs->ax; + leaf.subfn = regs->cx; + ret = snp_cpuid(ghcb, ctxt, &leaf); + if (!ret) { + regs->ax = leaf.eax; + regs->bx = leaf.ebx; + regs->cx = leaf.ecx; + regs->dx = leaf.edx; + } + + return ret; +} + +static enum es_result vc_handle_cpuid(struct ghcb *ghcb, + struct es_em_ctxt *ctxt) +{ + struct pt_regs *regs = ctxt->regs; + u32 cr4 = native_read_cr4(); + enum es_result ret; + int snp_cpuid_ret; + + snp_cpuid_ret = vc_handle_cpuid_snp(ghcb, ctxt); + if (!snp_cpuid_ret) + return ES_OK; + if (snp_cpuid_ret != -EOPNOTSUPP) + return ES_VMM_ERROR; + + ghcb_set_rax(ghcb, regs->ax); + ghcb_set_rcx(ghcb, regs->cx); + + if (cr4 & X86_CR4_OSXSAVE) + /* Safe to read xcr0 */ + ghcb_set_xcr0(ghcb, xgetbv(XCR_XFEATURE_ENABLED_MASK)); + else + /* xgetbv will cause #GP - use reset value for xcr0 */ + ghcb_set_xcr0(ghcb, 1); + + ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_CPUID, 0, 0); + if (ret != ES_OK) + return ret; + + if (!(ghcb_rax_is_valid(ghcb) && + ghcb_rbx_is_valid(ghcb) && + ghcb_rcx_is_valid(ghcb) && + ghcb_rdx_is_valid(ghcb))) + return ES_VMM_ERROR; + + regs->ax = ghcb->save.rax; + regs->bx = ghcb->save.rbx; + regs->cx = ghcb->save.rcx; + regs->dx = ghcb->save.rdx; + + return ES_OK; +} + +static enum es_result vc_handle_rdtsc(struct ghcb *ghcb, + struct es_em_ctxt *ctxt, + unsigned long exit_code) +{ + bool rdtscp = (exit_code == SVM_EXIT_RDTSCP); + enum es_result ret; + + /* + * The hypervisor should not be intercepting RDTSC/RDTSCP when Secure + * TSC is enabled. A #VC exception will be generated if the RDTSC/RDTSCP + * instructions are being intercepted. If this should occur and Secure + * TSC is enabled, guest execution should be terminated as the guest + * cannot rely on the TSC value provided by the hypervisor. + */ + if (sev_status & MSR_AMD64_SNP_SECURE_TSC) + return ES_VMM_ERROR; + + ret = sev_es_ghcb_hv_call(ghcb, ctxt, exit_code, 0, 0); + if (ret != ES_OK) + return ret; + + if (!(ghcb_rax_is_valid(ghcb) && ghcb_rdx_is_valid(ghcb) && + (!rdtscp || ghcb_rcx_is_valid(ghcb)))) + return ES_VMM_ERROR; + + ctxt->regs->ax = ghcb->save.rax; + ctxt->regs->dx = ghcb->save.rdx; + if (rdtscp) + ctxt->regs->cx = ghcb->save.rcx; + + return ES_OK; +} diff --git a/arch/x86/include/asm/sev-internal.h b/arch/x86/include/asm/sev-internal.h index a78f97208a39..b7232081f8f7 100644 --- a/arch/x86/include/asm/sev-internal.h +++ b/arch/x86/include/asm/sev-internal.h @@ -3,7 +3,6 @@ #define DR7_RESET_VALUE 0x400 extern struct ghcb boot_ghcb_page; -extern struct ghcb *boot_ghcb; extern u64 sev_hv_features; extern u64 sev_secrets_pa; @@ -59,8 +58,6 @@ DECLARE_PER_CPU(struct sev_es_save_area *, sev_vmsa); void early_set_pages_state(unsigned long vaddr, unsigned long paddr, unsigned long npages, enum psc_op op); -void __noreturn sev_es_terminate(unsigned int set, unsigned int reason); - DECLARE_PER_CPU(struct svsm_ca *, svsm_caa); DECLARE_PER_CPU(u64, svsm_caa_pa); @@ -100,11 +97,6 @@ static __always_inline void sev_es_wr_ghcb_msr(u64 val) native_wrmsr(MSR_AMD64_SEV_ES_GHCB, low, high); } -enum es_result sev_es_ghcb_hv_call(struct ghcb *ghcb, - struct es_em_ctxt *ctxt, - u64 exit_code, u64 exit_info_1, - u64 exit_info_2); - void snp_register_ghcb_early(unsigned long paddr); bool sev_es_negotiate_protocol(void); bool sev_es_check_cpu_features(void); diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h index a8661dfc9a9a..6158893786d6 100644 --- a/arch/x86/include/asm/sev.h +++ b/arch/x86/include/asm/sev.h @@ -521,6 +521,28 @@ static __always_inline void vc_ghcb_invalidate(struct ghcb *ghcb) __builtin_memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap)); } +void vc_forward_exception(struct es_em_ctxt *ctxt); + +/* I/O parameters for CPUID-related helpers */ +struct cpuid_leaf { + u32 fn; + u32 subfn; + u32 eax; + u32 ebx; + u32 ecx; + u32 edx; +}; + +int snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf); + +void __noreturn sev_es_terminate(unsigned int set, unsigned int reason); +enum es_result sev_es_ghcb_hv_call(struct ghcb *ghcb, + struct es_em_ctxt *ctxt, + u64 exit_code, u64 exit_info_1, + u64 exit_info_2); + +extern struct ghcb *boot_ghcb; + #else /* !CONFIG_AMD_MEM_ENCRYPT */ #define snp_vmpl 0 From patchwork Sun May 4 09:52:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887904 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 815261B3952 for ; Sun, 4 May 2025 09:53:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352402; cv=none; b=TeV7IzRVpHItRG0KDFLBn5Cqi8KC0GXvJPq7BGrGOVxL5hSXGTbeM91ceB+3ErKZKGYDZ1DUvhii9VhQsju+X8wfNCs9oNbOppPrxysGmVX2cFOz5j0FQnrNnC8gROk9eA9hOewsLAEUEdiZ5+ovbxXb0QsBJPHgCmwvBnI8Fcg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352402; c=relaxed/simple; bh=0XHRubWlvxhGoOPlMmYL4p0PTmPlw/mD7RVIaahL8sY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=QfHqhE7mvpSJZYA5AFPoNr8vt5Qg6pr445RLH6Rxvt7/DCk71YEWMJxrnJ6WoYdFRCnnZ+Q2MF3lQsuNVCJxfaKqB63ncTvem1LScyqx9ySufXS6dKOuzrDiLpTzdWfqVPHCs1yOgumQwB4QJRlr5wsZ+jaAn+x78DMmugPfOhU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ZZXwJrJH; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ZZXwJrJH" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-43d01024089so22445285e9.1 for ; Sun, 04 May 2025 02:53:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352399; x=1746957199; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3kJ+KMNddTv+kjQ55dJFzUMVOm9H3ODRuwtb5qBjuuQ=; b=ZZXwJrJHjW3E2BMvr0YFJXDRQRVLjUc8AtKZq3Nxzu5MZo0XetbwZzEfkiRXt7y761 ESrXijpt/DeLW9/9GQCltJh2da+PqTrWvBouLjX4A9Z1MLC74D32q37gtEyqMczyFPci 5lIR6m9/dJpLdhDmifbZKMEg32JlEvhPiZTdrds6Q+Jm0ZflcZiwEmYGNibXhSfsYhWb FUrKOArV6QDilsOznCNHJx9aYIrqVo8FW7iIRKWsp5Qnog4w+NlAgSF98UCdC8Y7JSK2 zHsaAX67op6rUjDF//csrWB4MZd5N9dBkVr5VVSl4HcEvwN1uTKu1HB67pP7nKjUeh4E ixRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352399; x=1746957199; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3kJ+KMNddTv+kjQ55dJFzUMVOm9H3ODRuwtb5qBjuuQ=; b=jvAx1Z5COEpBHZhSsoXEtPGrqEbQ2reI178k3iHP4q0sVmoIdS1IRa7A8sCPdCcUHS zk0vhEfZA6MBjmP50wXVJxygYw65XWEtZuAUDmlgNZYaOehZ1Rvca7V7OiuevAt8XWQy FgCgxN52J8t0nz1Gjr+T8xoHDsJEM8UdeFGBtgtCvbwGTnBsV6MspxXtT4Bu7CkL5mSj Jaglk/LnMUS1Ew22By9LJkyTUqe9KRjbt/MLFkaxefx9IxSd4FNPOYwfSPJLWN80kQvO 6iwJQdxMXyaPd+91B1yWqlfjUmZ2qCwsZ9XP7gfN152eKLLCV+mpZSYa5kvJOch5A4+5 ik6w== X-Gm-Message-State: AOJu0Yy+Qw4Z7Ixo3DRWwhDVtk1HkPYtvVnjI2JVa2gA+/XdswhYVyrW u2UV8shMItzBo7eQVlnup3AUSZnNidi+Gw+1PQl8t2opgwPlHKM0rUlYu/sUPWcDWtKQug== X-Google-Smtp-Source: AGHT+IEzXD80Qu7nqtQLSpTQz6noqKOtcFrzINt/dHyOuOPEcLbKdNREk9XewiF1CRE8vF7zu9fi4Br1 X-Received: from wmsp15.prod.google.com ([2002:a05:600c:1d8f:b0:43d:56fa:9b95]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:c07:b0:43c:fe9f:ab90 with SMTP id 5b1f17b1804b1-441c4937630mr25859795e9.28.1746352398955; Sun, 04 May 2025 02:53:18 -0700 (PDT) Date: Sun, 4 May 2025 11:52:37 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=7268; i=ardb@kernel.org; h=from:subject; bh=Zd4Hy2l6+2Xwx0i5ZHR0ISwcKyYHYyYNXAs7wzR0PHY=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4sWDGrsSH3W+mysdvJcufZzdVGtYGau+JCJz+o71s WdKZDk6SlkYxDgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwETuf2X4Z8Kk/9rS2ZJ1akj7 yai+JlNP/Wfmy2Pr3orkbImtLEmxZWT4duht98lvxR/uJR+7nPOht3TrrX1Vfx+82irEm+VX+02 TEQA= X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-32-ardb+git@google.com> Subject: [RFT PATCH v2 07/23] x86/sev: Separate MSR and GHCB based snp_cpuid() via a callback From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel There are two distinct callers of snp_cpuid(): one where the MSR protocol is always used, and one where the GHCB page based interface is always used. The snp_cpuid() logic does not care about the distinction, which only matters at a lower level. But the fact that it supports both interfaces means that the GHCB page based logic is pulled into the early startup code where PA to VA conversions are problematic, given that it runs from the 1:1 mapping of memory. So keep snp_cpuid() itself in the startup code, but factor out the hypervisor calls via a callback, so that the GHCB page handling can be moved out. Code refactoring only - no functional change intended. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/startup/sev-shared.c | 58 ++++---------------- arch/x86/coco/sev/vc-shared.c | 49 ++++++++++++++++- arch/x86/include/asm/sev.h | 3 +- 3 files changed, 61 insertions(+), 49 deletions(-) diff --git a/arch/x86/boot/startup/sev-shared.c b/arch/x86/boot/startup/sev-shared.c index 7a706db87b93..408e064a80d9 100644 --- a/arch/x86/boot/startup/sev-shared.c +++ b/arch/x86/boot/startup/sev-shared.c @@ -342,44 +342,7 @@ static int __sev_cpuid_hv_msr(struct cpuid_leaf *leaf) return ret; } -static int __sev_cpuid_hv_ghcb(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf) -{ - u32 cr4 = native_read_cr4(); - int ret; - - ghcb_set_rax(ghcb, leaf->fn); - ghcb_set_rcx(ghcb, leaf->subfn); - - if (cr4 & X86_CR4_OSXSAVE) - /* Safe to read xcr0 */ - ghcb_set_xcr0(ghcb, xgetbv(XCR_XFEATURE_ENABLED_MASK)); - else - /* xgetbv will cause #UD - use reset value for xcr0 */ - ghcb_set_xcr0(ghcb, 1); - - ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_CPUID, 0, 0); - if (ret != ES_OK) - return ret; - - if (!(ghcb_rax_is_valid(ghcb) && - ghcb_rbx_is_valid(ghcb) && - ghcb_rcx_is_valid(ghcb) && - ghcb_rdx_is_valid(ghcb))) - return ES_VMM_ERROR; - leaf->eax = ghcb->save.rax; - leaf->ebx = ghcb->save.rbx; - leaf->ecx = ghcb->save.rcx; - leaf->edx = ghcb->save.rdx; - - return ES_OK; -} - -static int sev_cpuid_hv(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf) -{ - return ghcb ? __sev_cpuid_hv_ghcb(ghcb, ctxt, leaf) - : __sev_cpuid_hv_msr(leaf); -} /* * This may be called early while still running on the initial identity @@ -484,21 +447,21 @@ snp_cpuid_get_validated_func(struct cpuid_leaf *leaf) return false; } -static void snp_cpuid_hv(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf) +static void snp_cpuid_hv_no_ghcb(void *ctx, struct cpuid_leaf *leaf) { - if (sev_cpuid_hv(ghcb, ctxt, leaf)) + if (__sev_cpuid_hv_msr(leaf)) sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_CPUID_HV); } static int __head -snp_cpuid_postprocess(struct ghcb *ghcb, struct es_em_ctxt *ctxt, - struct cpuid_leaf *leaf) +snp_cpuid_postprocess(void (*cpuid_hv)(void *ctx, struct cpuid_leaf *), + void *ctx, struct cpuid_leaf *leaf) { struct cpuid_leaf leaf_hv = *leaf; switch (leaf->fn) { case 0x1: - snp_cpuid_hv(ghcb, ctxt, &leaf_hv); + cpuid_hv(ctx, &leaf_hv); /* initial APIC ID */ leaf->ebx = (leaf_hv.ebx & GENMASK(31, 24)) | (leaf->ebx & GENMASK(23, 0)); @@ -517,7 +480,7 @@ snp_cpuid_postprocess(struct ghcb *ghcb, struct es_em_ctxt *ctxt, break; case 0xB: leaf_hv.subfn = 0; - snp_cpuid_hv(ghcb, ctxt, &leaf_hv); + cpuid_hv(ctx, &leaf_hv); /* extended APIC ID */ leaf->edx = leaf_hv.edx; @@ -565,7 +528,7 @@ snp_cpuid_postprocess(struct ghcb *ghcb, struct es_em_ctxt *ctxt, } break; case 0x8000001E: - snp_cpuid_hv(ghcb, ctxt, &leaf_hv); + cpuid_hv(ctx, &leaf_hv); /* extended APIC ID */ leaf->eax = leaf_hv.eax; @@ -587,7 +550,8 @@ snp_cpuid_postprocess(struct ghcb *ghcb, struct es_em_ctxt *ctxt, * should be treated as fatal by caller. */ int __head -snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf) +snp_cpuid(void (*cpuid_hv)(void *ctx, struct cpuid_leaf *), + void *ctx, struct cpuid_leaf *leaf) { const struct snp_cpuid_table *cpuid_table = snp_cpuid_get_table(); @@ -621,7 +585,7 @@ snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf) return 0; } - return snp_cpuid_postprocess(ghcb, ctxt, leaf); + return snp_cpuid_postprocess(cpuid_hv, ctx, leaf); } /* @@ -648,7 +612,7 @@ void __head do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code) leaf.fn = fn; leaf.subfn = subfn; - ret = snp_cpuid(NULL, NULL, &leaf); + ret = snp_cpuid(snp_cpuid_hv_no_ghcb, NULL, &leaf); if (!ret) goto cpuid_done; diff --git a/arch/x86/coco/sev/vc-shared.c b/arch/x86/coco/sev/vc-shared.c index 2c0ab0fdc060..b4688f69102e 100644 --- a/arch/x86/coco/sev/vc-shared.c +++ b/arch/x86/coco/sev/vc-shared.c @@ -409,15 +409,62 @@ static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt) return ret; } +static int __sev_cpuid_hv_ghcb(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf) +{ + u32 cr4 = native_read_cr4(); + int ret; + + ghcb_set_rax(ghcb, leaf->fn); + ghcb_set_rcx(ghcb, leaf->subfn); + + if (cr4 & X86_CR4_OSXSAVE) + /* Safe to read xcr0 */ + ghcb_set_xcr0(ghcb, xgetbv(XCR_XFEATURE_ENABLED_MASK)); + else + /* xgetbv will cause #UD - use reset value for xcr0 */ + ghcb_set_xcr0(ghcb, 1); + + ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_CPUID, 0, 0); + if (ret != ES_OK) + return ret; + + if (!(ghcb_rax_is_valid(ghcb) && + ghcb_rbx_is_valid(ghcb) && + ghcb_rcx_is_valid(ghcb) && + ghcb_rdx_is_valid(ghcb))) + return ES_VMM_ERROR; + + leaf->eax = ghcb->save.rax; + leaf->ebx = ghcb->save.rbx; + leaf->ecx = ghcb->save.rcx; + leaf->edx = ghcb->save.rdx; + + return ES_OK; +} + +struct cpuid_ctx { + struct ghcb *ghcb; + struct es_em_ctxt *ctxt; +}; + +static void snp_cpuid_hv_ghcb(void *p, struct cpuid_leaf *leaf) +{ + struct cpuid_ctx *ctx = p; + + if (__sev_cpuid_hv_ghcb(ctx->ghcb, ctx->ctxt, leaf)) + sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_CPUID_HV); +} + static int vc_handle_cpuid_snp(struct ghcb *ghcb, struct es_em_ctxt *ctxt) { + struct cpuid_ctx ctx = { ghcb, ctxt }; struct pt_regs *regs = ctxt->regs; struct cpuid_leaf leaf; int ret; leaf.fn = regs->ax; leaf.subfn = regs->cx; - ret = snp_cpuid(ghcb, ctxt, &leaf); + ret = snp_cpuid(snp_cpuid_hv_ghcb, &ctx, &leaf); if (!ret) { regs->ax = leaf.eax; regs->bx = leaf.ebx; diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h index 6158893786d6..f50a606be4f1 100644 --- a/arch/x86/include/asm/sev.h +++ b/arch/x86/include/asm/sev.h @@ -533,7 +533,8 @@ struct cpuid_leaf { u32 edx; }; -int snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf); +int snp_cpuid(void (*cpuid_hv)(void *ctx, struct cpuid_leaf *), + void *ctx, struct cpuid_leaf *leaf); void __noreturn sev_es_terminate(unsigned int set, unsigned int reason); enum es_result sev_es_ghcb_hv_call(struct ghcb *ghcb, From patchwork Sun May 4 09:52:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887903 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 738C41B423D for ; Sun, 4 May 2025 09:53:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352404; cv=none; b=i+fmfIf3bbyO1nijEiD44NYmQB8r313gc9z4YSWu51ldjTSH0kGTbOgtws0ejAhq5KeNT2VrGqkRZTnvbZ6X/K2MiN35yyjyjDXBr4HNdE5w0sf5K87dxxWvj+WfeHUHKKN0tQVZNO6csDaT5ynJeuRxiDyCUtv21n/MKJrZ6Qs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352404; c=relaxed/simple; bh=W1ANImcDpd+uTiUUIN+N3oqFLIGNXO0Quh9RJ0+Di5M=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=jIqbtyZJCsTaI+ZTpMuVCHlg8X8vOVQc7rgqIRgwbBgk1OJKMj/t2IUCjO4i0F+AIx41+J4lCuE9kxAAUOoULaa0QmDSr2eHYY/Vu7SbA0osJNszYuTKzu6snhMVxlr5oEZkTSsTW82d3A+cF8+kl9NKqBA7+rIZ9KQFoqfVOG0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=n/D4vNMy; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="n/D4vNMy" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-43d5ca7c86aso19949445e9.0 for ; Sun, 04 May 2025 02:53:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352401; x=1746957201; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UMlO4D1Qq+GHqs6fBTMVEjatUIBm7+dezuV4a9YlkXI=; b=n/D4vNMyvhgy9yLwEVIE1c1l29IGwSs4hYBssaQyoTI1gY8eNU1tAWDrAcCqwj9QO4 te6BLpLzwY2ZZGS14rAfZPiD8wx6Q2347MgnofXhGF7UgjN53ja88/G8se0fMvpNLTGp HWnSSz+AEMrpQZ7407wbAJHr6irano95+NXNNYHqFIvkIQNUIkDZpiG0c7ZQoI2RKSMI zFJXRIJuh5Yf5ntCbMYoOcFueg94YxZ45N3Erxkx9RZml55PYKkR8furz8KHG/82dDDb EJEgjxuoLbkaaJUB0rsILe9eM9G8zX7fuRBxp2hxoBdE0aDTgy7CerGvYNxN+ub5hz6O g2xQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352401; x=1746957201; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UMlO4D1Qq+GHqs6fBTMVEjatUIBm7+dezuV4a9YlkXI=; b=azt/LoMunGVL+04DfJ97Q5ZACO7TrEPxbL8l9k9jnaIqv191YB0rKkB1eNbAJGDty9 BvmboWXJm4NRkaVp9oZtboVS48+Eb6Vwdsw2xrhdJvNJNjTmqcC2D8HgVjC1hYMkMDVz 4A4ESEupbTxq27ncPyi3WoobNhK2eryuUy8So/10I/QJH0nwh7TP+QjMkU2pGHScqcMy L0aeZX1eXPji52jM7YQgZizU3XjxnSjxQTvrPuDJQhKjKcssrwfr0rlxun3ybsUG3wQj 2PD524GCEGHqfKaecZQFXuvPp9BhwcLZ6A1On722ooNrClRa8zKwwqIFUIn7CntoLSUa AeHw== X-Gm-Message-State: AOJu0YxGCeg+j778rEMpn86iW0f7vRXZurlabUdIVwdGpa8LyT+KNpAa xpp7JhafhK/5PKMTxN8u0frQX4J1/e923fw79lzenyXRi+QgUnooVGRuCV1mdFAJNDLY/w== X-Google-Smtp-Source: AGHT+IE1Daw2zJJ96MU5LWVcGK7VzrYPVT92nrQxCcKknFvAnfrQZ+3dSIOn7g9+yUY0RB5OD0d9jfW0 X-Received: from wrbhb13.prod.google.com ([2002:a05:6000:490d:b0:39a:be15:381b]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a5d:59a4:0:b0:3a0:88c7:47d with SMTP id ffacd0b85a97d-3a099ae91c7mr6832921f8f.38.1746352400875; Sun, 04 May 2025 02:53:20 -0700 (PDT) Date: Sun, 4 May 2025 11:52:38 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1805; i=ardb@kernel.org; h=from:subject; bh=5yrr0SwSIItxka+ivl8Mm8JUpubwLhO1mQq4KuDXpLg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4mXj77CKK0zTbogpXfTujfo0O7/Z6mWvpMRcr7yf9 o6PrQ52lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgIlMu83I0LpBcHaPidfcxbl/ y8LubmP+wPMuLTp53trCUsOnbw49t2Zk2J6zyEV2b3NREN/JnXenrVU6HJHoUxUb+qx++XWhgkm PWQE= X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-33-ardb+git@google.com> Subject: [RFT PATCH v2 08/23] x86/sev: Fall back to early page state change code only during boot From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel The core SEV-SNP page state change code will fall back to the early page state change code if no boot_ghcb has been assigned yet. This is unlikely to occur in practice, and it prevents the early code from being emitted as __init, making it harder to reason about whether or not startup code may be called at runtime. So ensure that early_page_state_change() is only called before freeing initmem, and move the call into a separate function annotated as __ref, so that the callee can be annotated as __init in a subsequent patch. Signed-off-by: Ard Biesheuvel --- arch/x86/coco/sev/core.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/arch/x86/coco/sev/core.c b/arch/x86/coco/sev/core.c index ac400525de73..fd5cd7d9b5f2 100644 --- a/arch/x86/coco/sev/core.c +++ b/arch/x86/coco/sev/core.c @@ -501,14 +501,24 @@ static unsigned long __set_pages_state(struct snp_psc_desc *data, unsigned long return vaddr; } +static bool __ref __early_set_pages_state(unsigned long vaddr, + unsigned long npages, int op) +{ + /* Use the MSR protocol when a GHCB is not yet available. */ + if (boot_ghcb || WARN_ON(system_state >= SYSTEM_FREEING_INITMEM)) + return false; + + early_set_pages_state(vaddr, __pa(vaddr), npages, op); + return true; +} + static void set_pages_state(unsigned long vaddr, unsigned long npages, int op) { struct snp_psc_desc desc; unsigned long vaddr_end; - /* Use the MSR protocol when a GHCB is not available. */ - if (!boot_ghcb) - return early_set_pages_state(vaddr, __pa(vaddr), npages, op); + if (__early_set_pages_state(vaddr, npages, op)) + return; vaddr = vaddr & PAGE_MASK; vaddr_end = vaddr + (npages << PAGE_SHIFT); From patchwork Sun May 4 09:52:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887280 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6BF741B423D for ; Sun, 4 May 2025 09:53:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352406; cv=none; b=lwllhwt403povgvqYWM2THgdtzeDJIRJliyaCp8eVsk/cTElTRXfgnCGr+HCWPuvGmm9tGm7dvcBPWXeM5ojmeW+3UicQo++XG7sIcx5CQFQM+M912k/91JgszLa5LWL8tXBtLYT9TwHH/QeiUZyjrnw72N1qFoOWsdyG3NKwl8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352406; c=relaxed/simple; bh=y89axQVBWipouQHDSui3OGxlzl+xhdZc/Zk8JNOfNdM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=cNsx/7Kj3VhMLWvTCGyy2yQW0LveBMuP3jZ2Alm9nnKuEc9pQL7HB/2+8KZuBejUsIhVrH4kzUieDWt9L+kerl3x+7JOyIV/pJsmIiW9pnQzucrMmYWFYgHURAXdfTULJTssQq8f20TGh8tIVpvMNbwtOjA4SMMfzFOGR5WnhSs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=sqHFUAWg; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="sqHFUAWg" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-43cf3168b87so14045635e9.2 for ; Sun, 04 May 2025 02:53:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352403; x=1746957203; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=z1QH716ojaXjFcqqp07sVPR7HrgO2wCXBEAvpYPhtoI=; b=sqHFUAWg3qHBIt9CIBqYcFpBVNLC7c4OnrpmxNe6UMQTE6XWqI9AKw+QYiakZBD6nh EBddY29/p89sqcJ53gZg0OJAoyQa9RziRH79T/Go70f8pWV3SuzFFMjNLGKoYxOjkDXe Y9gOAgrPF3SgDdF/Mpo8CWu6cyCarQKfjAmN3bjcS8yFIg+9jlIaUgVWPv2c/+9JntNs zDCa5fTwP4XqPQSNPrn7jmM94TWt6bWz14raLg7L6bAwTVHNROFQQ5Z1/dnG0WFCL8JL 2J8A7j36sUtmXVAbNMGXbbEjoUbOprWaWbc96tNOlSGp6GoKAbGanY8+VEKWZnXDSbw6 XUnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352403; x=1746957203; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=z1QH716ojaXjFcqqp07sVPR7HrgO2wCXBEAvpYPhtoI=; b=tMDnhguw9QRCoqugebc8iroOQBJ9N1vv+IJOZCgyM2rzX5UCanHI941x+IHw55XAz6 OvXq2CmxbV8fZ2+xPe14oBewKkPxGF1q4T0rWZl9Oee37clRTNRwV21ybIBj2FpaeoFy +9FkEyr4ZcQxLgE8MmVOnTX9DyphD/OTmHugDItScV8UwhYRZfz1oeB4/HowpR/xV9ms ESVeKZiC2aWbgtCzWPM3CPCtMgsLMMJ+DfvIX13JNBeFkVOKpMcRHmB+Kz0LkISgWHUH B6CIi8ossFSoYHsCTX3/6ROWXz6hC3j4CSExnyLoBaZFiC9oahtD9Bt2DWqQtqIt1W0J JG7w== X-Gm-Message-State: AOJu0YxExaAus3JFBbp/UzYcQyVEbCUF+SJm7QoHoqL7m6JcIClILd7W FDtsAMaVjuRdlK8mP/N1iYgHCMPmVzlx/HTof2Oq7iCNBIOvZfX4Th0o94nphhiWBWeLaw== X-Google-Smtp-Source: AGHT+IHiGEF2ESmBkDldONt9tAIuqdZIas0i+OaCNwgRPOhI5Ltffdmqvhw/PGG0XlGutukjhLU6tHYy X-Received: from wmbes9.prod.google.com ([2002:a05:600c:8109:b0:43b:d523:6ad8]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3507:b0:43c:f8fc:f687 with SMTP id 5b1f17b1804b1-441bbf3bd94mr57367955e9.27.1746352402830; Sun, 04 May 2025 02:53:22 -0700 (PDT) Date: Sun, 4 May 2025 11:52:39 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=18216; i=ardb@kernel.org; h=from:subject; bh=t8Vlwi5e8jQExbTgowtVYDZST8xqzSWPWGIcCPnvdy0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4lUdO3NU5NebL/42ze51msFQ3et+p+P073NXLKUNp MvC50/rKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABP5tZuRYU3yYqaq2ZYi5yWd UvP3Wqxp416eYn/0+Ir29NCYh0ceszAyNG9+Yx685RpnUvep/Xmztq1h68jdUsYTqfpunouRYMV ZJgA= X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-34-ardb+git@google.com> Subject: [RFT PATCH v2 09/23] x86/sev: Move GHCB page based HV communication out of startup code From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel Both the decompressor and the core kernel implement an early #VC handler, which only deals with CPUID instructions, and full featured one, which can handle any #VC exception. The former communicates with the hypervisor using the MSR based protocol, whereas the latter uses a shared GHCB page, which is configured a bit later. The SVSM PVALIDATE code can use either interface, but will prefer the GHCB page based one once available. Accessing this shared GHCB page from the core kernel's startup code is problematic, because it involves converting the GHCB address provided by the caller to a physical address. In the startup code, which may execute from a 1:1 mapping of memory, virtual to physical address translations are therefore ambiguous, and should be avoided. This means that exposing startup code dealing with the GHCB to callers that execute from the ordinary kernel virtual mapping should be avoided too. So move all GHCB page based communication out of the startup code, and rely on the MSR protocol only for all communication occurring before the kernel virtual mapping is up, as well as all SVSM PVALIDATE calls made via the 'early' page state change API (which may be called a bit later as well) Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/sev.c | 20 --- arch/x86/boot/startup/sev-shared.c | 160 +------------------- arch/x86/boot/startup/sev-startup.c | 35 +---- arch/x86/coco/sev/core.c | 67 ++++++++ arch/x86/coco/sev/vc-shared.c | 63 ++++++++ arch/x86/include/asm/sev-internal.h | 4 +- arch/x86/include/asm/sev.h | 56 ++++++- 7 files changed, 190 insertions(+), 215 deletions(-) diff --git a/arch/x86/boot/compressed/sev.c b/arch/x86/boot/compressed/sev.c index 804166457022..9b6eebc24e78 100644 --- a/arch/x86/boot/compressed/sev.c +++ b/arch/x86/boot/compressed/sev.c @@ -50,31 +50,11 @@ u64 svsm_get_caa_pa(void) return boot_svsm_caa_pa; } -int svsm_perform_call_protocol(struct svsm_call *call); - u8 snp_vmpl; /* Include code for early handlers */ #include "../../boot/startup/sev-shared.c" -int svsm_perform_call_protocol(struct svsm_call *call) -{ - struct ghcb *ghcb; - int ret; - - if (boot_ghcb) - ghcb = boot_ghcb; - else - ghcb = NULL; - - do { - ret = ghcb ? svsm_perform_ghcb_protocol(ghcb, call) - : svsm_perform_msr_protocol(call); - } while (ret == -EAGAIN); - - return ret; -} - static bool sev_snp_enabled(void) { return sev_status & MSR_AMD64_SEV_SNP_ENABLED; diff --git a/arch/x86/boot/startup/sev-shared.c b/arch/x86/boot/startup/sev-shared.c index 408e064a80d9..0709c8a8655a 100644 --- a/arch/x86/boot/startup/sev-shared.c +++ b/arch/x86/boot/startup/sev-shared.c @@ -17,8 +17,6 @@ #else #undef WARN #define WARN(condition, format...) (!!(condition)) -#undef vc_forward_exception -#define vc_forward_exception(c) panic("SNP: Hypervisor requested exception\n") #endif /* @@ -39,7 +37,7 @@ u64 boot_svsm_caa_pa __ro_after_init; * * GHCB protocol version negotiated with the hypervisor. */ -static u16 ghcb_version __ro_after_init; +u16 ghcb_version __ro_after_init; /* Copy of the SNP firmware's CPUID page. */ static struct snp_cpuid_table cpuid_table_copy __ro_after_init; @@ -100,22 +98,6 @@ u64 get_hv_features(void) return GHCB_MSR_HV_FT_RESP_VAL(val); } -void snp_register_ghcb_early(unsigned long paddr) -{ - unsigned long pfn = paddr >> PAGE_SHIFT; - u64 val; - - sev_es_wr_ghcb_msr(GHCB_MSR_REG_GPA_REQ_VAL(pfn)); - VMGEXIT(); - - val = sev_es_rd_ghcb_msr(); - - /* If the response GPA is not ours then abort the guest */ - if ((GHCB_RESP_CODE(val) != GHCB_MSR_REG_GPA_RESP) || - (GHCB_MSR_REG_GPA_RESP_VAL(val) != pfn)) - sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_REGISTER); -} - bool sev_es_negotiate_protocol(void) { u64 val; @@ -137,86 +119,7 @@ bool sev_es_negotiate_protocol(void) return true; } -static enum es_result verify_exception_info(struct ghcb *ghcb, struct es_em_ctxt *ctxt) -{ - u32 ret; - - ret = ghcb->save.sw_exit_info_1 & GENMASK_ULL(31, 0); - if (!ret) - return ES_OK; - - if (ret == 1) { - u64 info = ghcb->save.sw_exit_info_2; - unsigned long v = info & SVM_EVTINJ_VEC_MASK; - - /* Check if exception information from hypervisor is sane. */ - if ((info & SVM_EVTINJ_VALID) && - ((v == X86_TRAP_GP) || (v == X86_TRAP_UD)) && - ((info & SVM_EVTINJ_TYPE_MASK) == SVM_EVTINJ_TYPE_EXEPT)) { - ctxt->fi.vector = v; - - if (info & SVM_EVTINJ_VALID_ERR) - ctxt->fi.error_code = info >> 32; - - return ES_EXCEPTION; - } - } - - return ES_VMM_ERROR; -} - -static inline int svsm_process_result_codes(struct svsm_call *call) -{ - switch (call->rax_out) { - case SVSM_SUCCESS: - return 0; - case SVSM_ERR_INCOMPLETE: - case SVSM_ERR_BUSY: - return -EAGAIN; - default: - return -EINVAL; - } -} - -/* - * Issue a VMGEXIT to call the SVSM: - * - Load the SVSM register state (RAX, RCX, RDX, R8 and R9) - * - Set the CA call pending field to 1 - * - Issue VMGEXIT - * - Save the SVSM return register state (RAX, RCX, RDX, R8 and R9) - * - Perform atomic exchange of the CA call pending field - * - * - See the "Secure VM Service Module for SEV-SNP Guests" specification for - * details on the calling convention. - * - The calling convention loosely follows the Microsoft X64 calling - * convention by putting arguments in RCX, RDX, R8 and R9. - * - RAX specifies the SVSM protocol/callid as input and the return code - * as output. - */ -static __always_inline void svsm_issue_call(struct svsm_call *call, u8 *pending) -{ - register unsigned long rax asm("rax") = call->rax; - register unsigned long rcx asm("rcx") = call->rcx; - register unsigned long rdx asm("rdx") = call->rdx; - register unsigned long r8 asm("r8") = call->r8; - register unsigned long r9 asm("r9") = call->r9; - - call->caa->call_pending = 1; - - asm volatile("rep; vmmcall\n\t" - : "+r" (rax), "+r" (rcx), "+r" (rdx), "+r" (r8), "+r" (r9) - : : "memory"); - - *pending = xchg(&call->caa->call_pending, *pending); - - call->rax_out = rax; - call->rcx_out = rcx; - call->rdx_out = rdx; - call->r8_out = r8; - call->r9_out = r9; -} - -static int svsm_perform_msr_protocol(struct svsm_call *call) +int svsm_perform_msr_protocol(struct svsm_call *call) { u8 pending = 0; u64 val, resp; @@ -247,63 +150,6 @@ static int svsm_perform_msr_protocol(struct svsm_call *call) return svsm_process_result_codes(call); } -static int svsm_perform_ghcb_protocol(struct ghcb *ghcb, struct svsm_call *call) -{ - struct es_em_ctxt ctxt; - u8 pending = 0; - - vc_ghcb_invalidate(ghcb); - - /* - * Fill in protocol and format specifiers. This can be called very early - * in the boot, so use rip-relative references as needed. - */ - ghcb->protocol_version = ghcb_version; - ghcb->ghcb_usage = GHCB_DEFAULT_USAGE; - - ghcb_set_sw_exit_code(ghcb, SVM_VMGEXIT_SNP_RUN_VMPL); - ghcb_set_sw_exit_info_1(ghcb, 0); - ghcb_set_sw_exit_info_2(ghcb, 0); - - sev_es_wr_ghcb_msr(__pa(ghcb)); - - svsm_issue_call(call, &pending); - - if (pending) - return -EINVAL; - - switch (verify_exception_info(ghcb, &ctxt)) { - case ES_OK: - break; - case ES_EXCEPTION: - vc_forward_exception(&ctxt); - fallthrough; - default: - return -EINVAL; - } - - return svsm_process_result_codes(call); -} - -enum es_result sev_es_ghcb_hv_call(struct ghcb *ghcb, - struct es_em_ctxt *ctxt, - u64 exit_code, u64 exit_info_1, - u64 exit_info_2) -{ - /* Fill in protocol and format specifiers */ - ghcb->protocol_version = ghcb_version; - ghcb->ghcb_usage = GHCB_DEFAULT_USAGE; - - ghcb_set_sw_exit_code(ghcb, exit_code); - ghcb_set_sw_exit_info_1(ghcb, exit_info_1); - ghcb_set_sw_exit_info_2(ghcb, exit_info_2); - - sev_es_wr_ghcb_msr(__pa(ghcb)); - VMGEXIT(); - - return verify_exception_info(ghcb, ctxt); -} - static int __sev_cpuid_hv(u32 fn, int reg_idx, u32 *reg) { u64 val; @@ -755,7 +601,7 @@ static void __head svsm_pval_4k_page(unsigned long paddr, bool validate) call.rax = SVSM_CORE_CALL(SVSM_CORE_PVALIDATE); call.rcx = pc_pa; - ret = svsm_perform_call_protocol(&call); + ret = svsm_perform_msr_protocol(&call); if (ret) sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_PVALIDATE); diff --git a/arch/x86/boot/startup/sev-startup.c b/arch/x86/boot/startup/sev-startup.c index 435853a55768..797ca3e29b12 100644 --- a/arch/x86/boot/startup/sev-startup.c +++ b/arch/x86/boot/startup/sev-startup.c @@ -139,39 +139,6 @@ noinstr void __sev_put_ghcb(struct ghcb_state *state) } } -int svsm_perform_call_protocol(struct svsm_call *call) -{ - struct ghcb_state state; - unsigned long flags; - struct ghcb *ghcb; - int ret; - - /* - * This can be called very early in the boot, use native functions in - * order to avoid paravirt issues. - */ - flags = native_local_irq_save(); - - if (sev_cfg.ghcbs_initialized) - ghcb = __sev_get_ghcb(&state); - else if (boot_ghcb) - ghcb = boot_ghcb; - else - ghcb = NULL; - - do { - ret = ghcb ? svsm_perform_ghcb_protocol(ghcb, call) - : svsm_perform_msr_protocol(call); - } while (ret == -EAGAIN); - - if (sev_cfg.ghcbs_initialized) - __sev_put_ghcb(&state); - - native_local_irq_restore(flags); - - return ret; -} - void __head early_set_pages_state(unsigned long vaddr, unsigned long paddr, unsigned long npages, enum psc_op op) @@ -325,7 +292,7 @@ static __head void svsm_setup(struct cc_blob_sev_info *cc_info) call.caa = svsm_get_caa(); call.rax = SVSM_CORE_CALL(SVSM_CORE_REMAP_CA); call.rcx = pa; - ret = svsm_perform_call_protocol(&call); + ret = svsm_perform_msr_protocol(&call); if (ret) sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_SVSM_CA_REMAP_FAIL); diff --git a/arch/x86/coco/sev/core.c b/arch/x86/coco/sev/core.c index fd5cd7d9b5f2..883b2719986d 100644 --- a/arch/x86/coco/sev/core.c +++ b/arch/x86/coco/sev/core.c @@ -153,6 +153,73 @@ static u64 __init get_jump_table_addr(void) return ret; } +static int svsm_perform_ghcb_protocol(struct ghcb *ghcb, struct svsm_call *call) +{ + struct es_em_ctxt ctxt; + u8 pending = 0; + + vc_ghcb_invalidate(ghcb); + + /* + * Fill in protocol and format specifiers. This can be called very early + * in the boot, so use rip-relative references as needed. + */ + ghcb->protocol_version = ghcb_version; + ghcb->ghcb_usage = GHCB_DEFAULT_USAGE; + + ghcb_set_sw_exit_code(ghcb, SVM_VMGEXIT_SNP_RUN_VMPL); + ghcb_set_sw_exit_info_1(ghcb, 0); + ghcb_set_sw_exit_info_2(ghcb, 0); + + sev_es_wr_ghcb_msr(__pa(ghcb)); + + svsm_issue_call(call, &pending); + + if (pending) + return -EINVAL; + + switch (verify_exception_info(ghcb, &ctxt)) { + case ES_OK: + break; + case ES_EXCEPTION: + vc_forward_exception(&ctxt); + fallthrough; + default: + return -EINVAL; + } + + return svsm_process_result_codes(call); +} + +static int svsm_perform_call_protocol(struct svsm_call *call) +{ + struct ghcb_state state; + unsigned long flags; + struct ghcb *ghcb; + int ret; + + flags = native_local_irq_save(); + + if (sev_cfg.ghcbs_initialized) + ghcb = __sev_get_ghcb(&state); + else if (boot_ghcb) + ghcb = boot_ghcb; + else + ghcb = NULL; + + do { + ret = ghcb ? svsm_perform_ghcb_protocol(ghcb, call) + : svsm_perform_msr_protocol(call); + } while (ret == -EAGAIN); + + if (sev_cfg.ghcbs_initialized) + __sev_put_ghcb(&state); + + native_local_irq_restore(flags); + + return ret; +} + static inline void __pval_terminate(u64 pfn, bool action, unsigned int page_size, int ret, u64 svsm_ret) { diff --git a/arch/x86/coco/sev/vc-shared.c b/arch/x86/coco/sev/vc-shared.c index b4688f69102e..6df758560803 100644 --- a/arch/x86/coco/sev/vc-shared.c +++ b/arch/x86/coco/sev/vc-shared.c @@ -409,6 +409,53 @@ static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt) return ret; } +enum es_result verify_exception_info(struct ghcb *ghcb, struct es_em_ctxt *ctxt) +{ + u32 ret; + + ret = ghcb->save.sw_exit_info_1 & GENMASK_ULL(31, 0); + if (!ret) + return ES_OK; + + if (ret == 1) { + u64 info = ghcb->save.sw_exit_info_2; + unsigned long v = info & SVM_EVTINJ_VEC_MASK; + + /* Check if exception information from hypervisor is sane. */ + if ((info & SVM_EVTINJ_VALID) && + ((v == X86_TRAP_GP) || (v == X86_TRAP_UD)) && + ((info & SVM_EVTINJ_TYPE_MASK) == SVM_EVTINJ_TYPE_EXEPT)) { + ctxt->fi.vector = v; + + if (info & SVM_EVTINJ_VALID_ERR) + ctxt->fi.error_code = info >> 32; + + return ES_EXCEPTION; + } + } + + return ES_VMM_ERROR; +} + +enum es_result sev_es_ghcb_hv_call(struct ghcb *ghcb, + struct es_em_ctxt *ctxt, + u64 exit_code, u64 exit_info_1, + u64 exit_info_2) +{ + /* Fill in protocol and format specifiers */ + ghcb->protocol_version = ghcb_version; + ghcb->ghcb_usage = GHCB_DEFAULT_USAGE; + + ghcb_set_sw_exit_code(ghcb, exit_code); + ghcb_set_sw_exit_info_1(ghcb, exit_info_1); + ghcb_set_sw_exit_info_2(ghcb, exit_info_2); + + sev_es_wr_ghcb_msr(__pa(ghcb)); + VMGEXIT(); + + return verify_exception_info(ghcb, ctxt); +} + static int __sev_cpuid_hv_ghcb(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf) { u32 cr4 = native_read_cr4(); @@ -549,3 +596,19 @@ static enum es_result vc_handle_rdtsc(struct ghcb *ghcb, return ES_OK; } + +void snp_register_ghcb_early(unsigned long paddr) +{ + unsigned long pfn = paddr >> PAGE_SHIFT; + u64 val; + + sev_es_wr_ghcb_msr(GHCB_MSR_REG_GPA_REQ_VAL(pfn)); + VMGEXIT(); + + val = sev_es_rd_ghcb_msr(); + + /* If the response GPA is not ours then abort the guest */ + if ((GHCB_RESP_CODE(val) != GHCB_MSR_REG_GPA_RESP) || + (GHCB_MSR_REG_GPA_RESP_VAL(val) != pfn)) + sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_REGISTER); +} diff --git a/arch/x86/include/asm/sev-internal.h b/arch/x86/include/asm/sev-internal.h index b7232081f8f7..0d02e780beb3 100644 --- a/arch/x86/include/asm/sev-internal.h +++ b/arch/x86/include/asm/sev-internal.h @@ -80,7 +80,8 @@ static __always_inline u64 svsm_get_caa_pa(void) return boot_svsm_caa_pa; } -int svsm_perform_call_protocol(struct svsm_call *call); +enum es_result verify_exception_info(struct ghcb *ghcb, struct es_em_ctxt *ctxt); +void vc_forward_exception(struct es_em_ctxt *ctxt); static inline u64 sev_es_rd_ghcb_msr(void) { @@ -97,7 +98,6 @@ static __always_inline void sev_es_wr_ghcb_msr(u64 val) native_wrmsr(MSR_AMD64_SEV_ES_GHCB, low, high); } -void snp_register_ghcb_early(unsigned long paddr); bool sev_es_negotiate_protocol(void); bool sev_es_check_cpu_features(void); u64 get_hv_features(void); diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h index f50a606be4f1..ca7168cc2118 100644 --- a/arch/x86/include/asm/sev.h +++ b/arch/x86/include/asm/sev.h @@ -485,6 +485,7 @@ static inline int pvalidate(unsigned long vaddr, bool rmp_psize, bool validate) struct snp_guest_request_ioctl; void setup_ghcb(void); +void snp_register_ghcb_early(unsigned long paddr); void early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, unsigned long npages); void early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr, @@ -521,8 +522,6 @@ static __always_inline void vc_ghcb_invalidate(struct ghcb *ghcb) __builtin_memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap)); } -void vc_forward_exception(struct es_em_ctxt *ctxt); - /* I/O parameters for CPUID-related helpers */ struct cpuid_leaf { u32 fn; @@ -533,15 +532,68 @@ struct cpuid_leaf { u32 edx; }; +int svsm_perform_msr_protocol(struct svsm_call *call); int snp_cpuid(void (*cpuid_hv)(void *ctx, struct cpuid_leaf *), void *ctx, struct cpuid_leaf *leaf); +/* + * Issue a VMGEXIT to call the SVSM: + * - Load the SVSM register state (RAX, RCX, RDX, R8 and R9) + * - Set the CA call pending field to 1 + * - Issue VMGEXIT + * - Save the SVSM return register state (RAX, RCX, RDX, R8 and R9) + * - Perform atomic exchange of the CA call pending field + * + * - See the "Secure VM Service Module for SEV-SNP Guests" specification for + * details on the calling convention. + * - The calling convention loosely follows the Microsoft X64 calling + * convention by putting arguments in RCX, RDX, R8 and R9. + * - RAX specifies the SVSM protocol/callid as input and the return code + * as output. + */ +static __always_inline void svsm_issue_call(struct svsm_call *call, u8 *pending) +{ + register unsigned long rax asm("rax") = call->rax; + register unsigned long rcx asm("rcx") = call->rcx; + register unsigned long rdx asm("rdx") = call->rdx; + register unsigned long r8 asm("r8") = call->r8; + register unsigned long r9 asm("r9") = call->r9; + + call->caa->call_pending = 1; + + asm volatile("rep; vmmcall\n\t" + : "+r" (rax), "+r" (rcx), "+r" (rdx), "+r" (r8), "+r" (r9) + : : "memory"); + + *pending = xchg(&call->caa->call_pending, *pending); + + call->rax_out = rax; + call->rcx_out = rcx; + call->rdx_out = rdx; + call->r8_out = r8; + call->r9_out = r9; +} + +static inline int svsm_process_result_codes(struct svsm_call *call) +{ + switch (call->rax_out) { + case SVSM_SUCCESS: + return 0; + case SVSM_ERR_INCOMPLETE: + case SVSM_ERR_BUSY: + return -EAGAIN; + default: + return -EINVAL; + } +} + void __noreturn sev_es_terminate(unsigned int set, unsigned int reason); enum es_result sev_es_ghcb_hv_call(struct ghcb *ghcb, struct es_em_ctxt *ctxt, u64 exit_code, u64 exit_info_1, u64 exit_info_2); +extern u16 ghcb_version; extern struct ghcb *boot_ghcb; #else /* !CONFIG_AMD_MEM_ENCRYPT */ From patchwork Sun May 4 09:52:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887902 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AAF6F1C6FF6 for ; Sun, 4 May 2025 09:53:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352408; cv=none; b=KPwzBmrLNZv35bXvSdNqU/QfdiO+iWxE3ORzs619f6Krjo/jppyL8c57uIP2FG1+AAjLB2ZJmkRJqG/cBxF8h6yFjBdBlzA57cFL6EFly5WParwTbz8ZEVzXq9hWgKkdkjrrUVitniBOfyKXM2E03oLEAzuDV1vVgW4xyRuB5YU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352408; c=relaxed/simple; bh=Da8gefPc/MkvKcwMBiYhIH1VS9MQxWYpqqsM/kBWj2k=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=rHZnQdmm0QAKDEfyJeTvrKdPk5rBGQnBd3eKqXrQgESaROGG0OzzUPQ+L4Ib7WDJG1AyIfrlW5AACVTFD5v9hWv/omoaTaiSF8rDl0loJHR6uyLjwzPo+4kqWY6BLBQIWeNZYPL8sAFkGvtYw9yxk2AbJcRO5dXNeMH26I3Xb/g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=BuSWy7FD; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="BuSWy7FD" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-43d4d15058dso25195625e9.0 for ; Sun, 04 May 2025 02:53:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352405; x=1746957205; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wX4/+1PkJw6Jnyy4g/DPm+Tnx+UM3RROZEvunHorr8w=; b=BuSWy7FDncgofHjWoRwDC6CE6TfX0rM0sSqhhTgOWasCHGF5vfUpoW2keBrQhAk54I LIAGVpTK6agyN/80oP+5NaC7KJWKg2/LEoUHZaHQU7c7x0I2mYTUjBwGHfKX68ujlGvN 1tAs3IAXg7+O/jr7NXS2bcq/izZ0xrF8NicJGKj1mY2QyAoHzAAlXB1CWhGhSD5AEEWx oYv/5xQCmGpwJzUqte7IIMWoAvFBN6vKuDczp04Uf40TABhm5qjaHbVlEdvC+5VOtjD2 uhSRR35gEB238FUeR1iG7xK5zPN0QYkHGa61lD4MwfucKHQrpCz7gdpXqqVir8XnvhiY +xXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352405; x=1746957205; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wX4/+1PkJw6Jnyy4g/DPm+Tnx+UM3RROZEvunHorr8w=; b=gtq1siQwfUstXgWIkeJgRt08fB/CbG16dHDGBShrh9ZtiVKyA5UBoeoYqmQMgjyU1X 071bwfe59V3oBmHwSOc+Omdzr5Pkr6ruNDg8YFkZrqz7LtPKhpeCcNQiTvL04FvdrvOz MuzFQ5HCdU6LXaEUMlZxUT4Er6W/UBdee+K01UxhsMh4mv3qukOGkSsWV47aijRDmdJI iIV9jNNSpwFmJncLZnox5gyvvkojCDMGUX6ytTpCGIx59gWnDHEWWiY2kQvfsjzq+t5/ WBYQ4VvG2Fa6SHMuEaJXNMP/90UqdHhqzQqZa7NFzIsUk4SoQCrdko4B6ekBIAazREkz fN0w== X-Gm-Message-State: AOJu0YzhVHt6HJKSpjtU2/9ZeZlMifurGCC4tReRoTi4xtSXka1vinyJ a5mqlZODetXdx2SW30nJMqTpxrcByJuEEpu8FTgULDRfAEJwgGjewQhdgsz6mKbjs38CgA== X-Google-Smtp-Source: AGHT+IFjslrWfCH+uAS9ty2elKHdVUzc19rUudtpTn5fWKEmJz4qLk7mVsC2Sil+38bd4AnXTSXFQj40 X-Received: from wmbfl15.prod.google.com ([2002:a05:600c:b8f:b0:441:b38b:9cc3]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1d8d:b0:440:66a4:8d1a with SMTP id 5b1f17b1804b1-441c48aa302mr27049065e9.7.1746352405226; Sun, 04 May 2025 02:53:25 -0700 (PDT) Date: Sun, 4 May 2025 11:52:40 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6031; i=ardb@kernel.org; h=from:subject; bh=xk86JG/R04sMdF0/gJrtMthpOxFG4UyGc1tpigbLlgU=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4nXQ9wjJM+pm/4+6GIS8y/m44eIDBXutlHcub7IXH l8sKfOxo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzkeCkjwxPBGXFRQZWr93Ou K6rOal3zYdcLX4buK8Vc0ywtby55YMbIMLM5YiYPz5uIVVcC3mlr3HloMslq0/TtAZ777abPy8m /wwwA X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-35-ardb+git@google.com> Subject: [RFT PATCH v2 10/23] x86/sev: Use boot SVSM CA for all startup and init code From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel To avoid having to reason about whether or not to use the per-CPU SVSM calling area when running startup and init code on the boot CPU, reuse the boot SVSM calling area as the per-CPU area for CPU #0. This removes the need to make the per-CPU variables and associated state in sev_cfg accessible to the startup code once confined. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/sev.c | 13 ------ arch/x86/boot/startup/sev-shared.c | 4 +- arch/x86/boot/startup/sev-startup.c | 6 +-- arch/x86/coco/sev/core.c | 47 +++++++++----------- arch/x86/include/asm/sev-internal.h | 16 ------- 5 files changed, 26 insertions(+), 60 deletions(-) diff --git a/arch/x86/boot/compressed/sev.c b/arch/x86/boot/compressed/sev.c index 9b6eebc24e78..91e8140250f6 100644 --- a/arch/x86/boot/compressed/sev.c +++ b/arch/x86/boot/compressed/sev.c @@ -37,19 +37,6 @@ struct ghcb *boot_ghcb; #define __BOOT_COMPRESSED -extern struct svsm_ca *boot_svsm_caa; -extern u64 boot_svsm_caa_pa; - -struct svsm_ca *svsm_get_caa(void) -{ - return boot_svsm_caa; -} - -u64 svsm_get_caa_pa(void) -{ - return boot_svsm_caa_pa; -} - u8 snp_vmpl; /* Include code for early handlers */ diff --git a/arch/x86/boot/startup/sev-shared.c b/arch/x86/boot/startup/sev-shared.c index 0709c8a8655a..b1f4b9b15045 100644 --- a/arch/x86/boot/startup/sev-shared.c +++ b/arch/x86/boot/startup/sev-shared.c @@ -585,10 +585,10 @@ static void __head svsm_pval_4k_page(unsigned long paddr, bool validate) */ flags = native_local_irq_save(); - call.caa = svsm_get_caa(); + call.caa = boot_svsm_caa; pc = (struct svsm_pvalidate_call *)call.caa->svsm_buffer; - pc_pa = svsm_get_caa_pa() + offsetof(struct svsm_ca, svsm_buffer); + pc_pa = boot_svsm_caa_pa + offsetof(struct svsm_ca, svsm_buffer); pc->num_entries = 1; pc->cur_index = 0; diff --git a/arch/x86/boot/startup/sev-startup.c b/arch/x86/boot/startup/sev-startup.c index 797ca3e29b12..ca6a9863ffab 100644 --- a/arch/x86/boot/startup/sev-startup.c +++ b/arch/x86/boot/startup/sev-startup.c @@ -59,9 +59,6 @@ u64 sev_secrets_pa __ro_after_init; /* For early boot SVSM communication */ struct svsm_ca boot_svsm_ca_page __aligned(PAGE_SIZE); -DEFINE_PER_CPU(struct svsm_ca *, svsm_caa); -DEFINE_PER_CPU(u64, svsm_caa_pa); - /* * Nothing shall interrupt this code path while holding the per-CPU * GHCB. The backup GHCB is only for NMIs interrupting this path. @@ -261,6 +258,7 @@ static __head struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp) static __head void svsm_setup(struct cc_blob_sev_info *cc_info) { + struct snp_secrets_page *secrets = (void *)cc_info->secrets_phys; struct svsm_call call = {}; int ret; u64 pa; @@ -289,7 +287,7 @@ static __head void svsm_setup(struct cc_blob_sev_info *cc_info) * RAX = 0 (Protocol=0, CallID=0) * RCX = New CA GPA */ - call.caa = svsm_get_caa(); + call.caa = (struct svsm_ca *)secrets->svsm_caa; call.rax = SVSM_CORE_CALL(SVSM_CORE_REMAP_CA); call.rcx = pa; ret = svsm_perform_msr_protocol(&call); diff --git a/arch/x86/coco/sev/core.c b/arch/x86/coco/sev/core.c index 883b2719986d..36edf670ff19 100644 --- a/arch/x86/coco/sev/core.c +++ b/arch/x86/coco/sev/core.c @@ -45,6 +45,25 @@ #include #include +DEFINE_PER_CPU(struct svsm_ca *, svsm_caa); +DEFINE_PER_CPU(u64, svsm_caa_pa); + +static inline struct svsm_ca *svsm_get_caa(void) +{ + if (sev_cfg.use_cas) + return this_cpu_read(svsm_caa); + else + return boot_svsm_caa; +} + +static inline u64 svsm_get_caa_pa(void) +{ + if (sev_cfg.use_cas) + return this_cpu_read(svsm_caa_pa); + else + return boot_svsm_caa_pa; +} + /* AP INIT values as documented in the APM2 section "Processor Initialization State" */ #define AP_INIT_CS_LIMIT 0xffff #define AP_INIT_DS_LIMIT 0xffff @@ -1207,7 +1226,8 @@ static void __init alloc_runtime_data(int cpu) struct svsm_ca *caa; /* Allocate the SVSM CA page if an SVSM is present */ - caa = memblock_alloc_or_panic(sizeof(*caa), PAGE_SIZE); + caa = cpu ? memblock_alloc_or_panic(sizeof(*caa), PAGE_SIZE) + : boot_svsm_caa; per_cpu(svsm_caa, cpu) = caa; per_cpu(svsm_caa_pa, cpu) = __pa(caa); @@ -1261,32 +1281,9 @@ void __init sev_es_init_vc_handling(void) init_ghcb(cpu); } - /* If running under an SVSM, switch to the per-cpu CA */ - if (snp_vmpl) { - struct svsm_call call = {}; - unsigned long flags; - int ret; - - local_irq_save(flags); - - /* - * SVSM_CORE_REMAP_CA call: - * RAX = 0 (Protocol=0, CallID=0) - * RCX = New CA GPA - */ - call.caa = svsm_get_caa(); - call.rax = SVSM_CORE_CALL(SVSM_CORE_REMAP_CA); - call.rcx = this_cpu_read(svsm_caa_pa); - ret = svsm_perform_call_protocol(&call); - if (ret) - panic("Can't remap the SVSM CA, ret=%d, rax_out=0x%llx\n", - ret, call.rax_out); - + if (snp_vmpl) sev_cfg.use_cas = true; - local_irq_restore(flags); - } - sev_es_setup_play_dead(); /* Secondary CPUs use the runtime #VC handler */ diff --git a/arch/x86/include/asm/sev-internal.h b/arch/x86/include/asm/sev-internal.h index 0d02e780beb3..4335711274e3 100644 --- a/arch/x86/include/asm/sev-internal.h +++ b/arch/x86/include/asm/sev-internal.h @@ -64,22 +64,6 @@ DECLARE_PER_CPU(u64, svsm_caa_pa); extern struct svsm_ca *boot_svsm_caa; extern u64 boot_svsm_caa_pa; -static __always_inline struct svsm_ca *svsm_get_caa(void) -{ - if (sev_cfg.use_cas) - return this_cpu_read(svsm_caa); - else - return boot_svsm_caa; -} - -static __always_inline u64 svsm_get_caa_pa(void) -{ - if (sev_cfg.use_cas) - return this_cpu_read(svsm_caa_pa); - else - return boot_svsm_caa_pa; -} - enum es_result verify_exception_info(struct ghcb *ghcb, struct es_em_ctxt *ctxt); void vc_forward_exception(struct es_em_ctxt *ctxt); From patchwork Sun May 4 09:52:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887279 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C4EAD1C863D for ; Sun, 4 May 2025 09:53:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352410; cv=none; b=ZmQr11kJX+lwazDL4eBDEihMofwUNllbjSuwTYj9tGlGSWapA0lFaIiLys5YjppP5LJgmZVABoXN2GYewmk95JjTL8fOc+pX/Jd9DXJ41vvpgJZHxWQS7Z5LGcDWgJTOGYvEqHVvjsiYGl2n47Obp3OribyiTmffr/4EC959iVQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352410; c=relaxed/simple; bh=+Z8FJSCGbwwgj1awTIcuTLRAEXnP5VkhJvt9KubUTbs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=g9Gmnv5dvescas12QMuSF3ZD2xwcyPguCwMjrX1a7u51mLSTfYUWIJG15I4HO/u1YZvUWiDsvDtkSbSp+qQjuOjHM6ShqA+rIMZQizM68VxNnQrV5WdbAnR1XzcTBpi8pD2YVRyb8p439kmAtMttiLFXK8yP3oP79sPrnAq+rAg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=2Nt1fAkw; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="2Nt1fAkw" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-39135d31ca4so1325524f8f.1 for ; Sun, 04 May 2025 02:53:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352407; x=1746957207; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3lTLYqTHmduCIUn2kSdkAOOr4P+q4Sd3ebjwsj9/7+o=; b=2Nt1fAkwLotd/TygoNrhN9CYIAZ9gAuAOMNt7CzRNvZrijR/OldO1YCxbO9lpGXcnn 1OPQOTuHiWnUa/YKolhz0tj6jByimdOVVPJ3gDPAJKaj9v96kuTnC90JkhUCwcSyLnUz A6EUkVgpseorWlB8+96W3FEochud3u36+4lBlXy+P3QLucsDoZYA4yrt2qBjoYChRbZ3 4Z/f2zDaiIVz8KZjl5pTZeSrYhEGVKvlQd/+4fyDwSYg6viDCeyxueoijQyRK6gWdB6C ZPbLT0qbln9/QuAf5Pu+uCvRVKH7cK/t9aQTZWjZdx/26o/jkCXRzAuz7MbVwbr35oUZ 5O9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352407; x=1746957207; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3lTLYqTHmduCIUn2kSdkAOOr4P+q4Sd3ebjwsj9/7+o=; b=Q/BfOzitM1k4+3v0xtGS2Lv5+xoka/7SdLjz7oppepn2AtzU3pziBksadfl/gmNw9X vQ7DAEX0+DOfVIzDNp5PBcVv7zxBKMncSDPAvWFl8TK92OSXIG06b6rseW6v9MQezcgZ y3XW5aijt7MvWqvrnr3Hqjv6XIca1yTVp+dAmmWU/tDq3zkgNyf/N3mcgT9xkZlNMEzz rZYxFys55Kc/bNPiv8psWu5Zl/t3YFcko8le9gd2hMrE3e8DMowIMwKeR1oZLYknWPUI qEk4ERy4bDAlXtBAAuFBeD2i0Vr+5ONL/Xr7Io3D7Wex0uGZUPcfM8yaPD50JO9pLuT3 45vg== X-Gm-Message-State: AOJu0YzP1k4rHtfrU+yA21TXqka5T0Yl6nyJ90auUZRHmRwkT1XugNDA bIJ3FgJAh23WiWF8gBQIuBPbSK3WSIJnIsUbJHbENT07rkmILOOswaGzo57xsV5087t4aA== X-Google-Smtp-Source: AGHT+IFJ8HW78R93DSDeWNBG9slWAdIJ5rmrSaYG1wr1R/vQNOIhV5y3ZcE3vwvhBOseCVNt759f3z8K X-Received: from wmbel14.prod.google.com ([2002:a05:600c:3e0e:b0:440:5f8a:667c]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:240c:b0:3a0:8a19:db31 with SMTP id ffacd0b85a97d-3a094034de9mr10207309f8f.9.1746352407154; Sun, 04 May 2025 02:53:27 -0700 (PDT) Date: Sun, 4 May 2025 11:52:41 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2029; i=ardb@kernel.org; h=from:subject; bh=xt5xyfXf/gi2XPcOGjFts1TusoE6wHjvuVonSZNWdN4=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4u3Kl/0b1vi2eN46zx2nanNmi+LnTQLLDft/pb32m ii3zG1bRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjIIm1Ghg9G13Vf7Pr248Bu 86bQz6t/5jILP0q/PktS9F5j2KEX6tcYGXrkBAqtJzKqds7ZsL5z56OnqYLV8zb5L3F9o9J+SqO jlwMA X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-36-ardb+git@google.com> Subject: [RFT PATCH v2 11/23] x86/boot: Drop redundant RMPADJUST in SEV SVSM presence check From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel snp_vmpl will be assigned a non-zero value when executing at a VMPL other than 0, and this is inferred from a call to RMPADJUST, which only works when running at VMPL0. This means that testing snp_vmpl is sufficient, and there is no need to perform the same check again. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/sev.c | 20 +++----------------- 1 file changed, 3 insertions(+), 17 deletions(-) diff --git a/arch/x86/boot/compressed/sev.c b/arch/x86/boot/compressed/sev.c index 91e8140250f6..18ba81c0b0ed 100644 --- a/arch/x86/boot/compressed/sev.c +++ b/arch/x86/boot/compressed/sev.c @@ -420,30 +420,16 @@ void sev_enable(struct boot_params *bp) */ if (sev_status & MSR_AMD64_SEV_SNP_ENABLED) { u64 hv_features; - int ret; hv_features = get_hv_features(); if (!(hv_features & GHCB_HV_FT_SNP)) sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SNP_UNSUPPORTED); /* - * Enforce running at VMPL0 or with an SVSM. - * - * Use RMPADJUST (see the rmpadjust() function for a description of - * what the instruction does) to update the VMPL1 permissions of a - * page. If the guest is running at VMPL0, this will succeed. If the - * guest is running at any other VMPL, this will fail. Linux SNP guests - * only ever run at a single VMPL level so permission mask changes of a - * lesser-privileged VMPL are a don't-care. + * Running at VMPL0 is required unless an SVSM is present and + * the hypervisor supports the required SVSM GHCB events. */ - ret = rmpadjust((unsigned long)&boot_ghcb_page, RMP_PG_SIZE_4K, 1); - - /* - * Running at VMPL0 is not required if an SVSM is present and the hypervisor - * supports the required SVSM GHCB events. - */ - if (ret && - !(snp_vmpl && (hv_features & GHCB_HV_FT_SNP_MULTI_VMPL))) + if (snp_vmpl > 0 && !(hv_features & GHCB_HV_FT_SNP_MULTI_VMPL)) sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_NOT_VMPL0); } From patchwork Sun May 4 09:52:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887901 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F2A711D6DB4 for ; Sun, 4 May 2025 09:53:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352412; cv=none; b=THSXmxASJEx7FhShzKuu8K2oPAGocm74cuj8zOyl2EwUBApGbVd/JZk3dclXrr6Avo72OpFx61nyV0qqWnn7w65sKi4zK3qQWBPWpMk0reG0Vyjdfct8DNlzB1SfyYrH6M63SDk803aOsswB32jFskYFvYRb78fWHwV0uo5kTkk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352412; c=relaxed/simple; bh=5ULEMcpNxvFS8qCRTlhFI13Km44zrVehn0FrJFS36Hs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=AMLbEbDTIABAkLj4SqZ8AljpVf63t6C92P+nn/4VqpZFLoXtaDEq1OuvLUrwjJO5Uf/+sWNRa6h0Jl1K4TN+YnONbgaaBERMP2XSzzSmycO0TgoyPxbT2P89qyc+nRl30HqltAZEigWNZb8Voa0H9O2sFwrx7MJ74dOJtwdB4RM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=YRFPdL91; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="YRFPdL91" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-43efa869b0aso22371285e9.3 for ; Sun, 04 May 2025 02:53:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352409; x=1746957209; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=e/d0BPzGs3MdNKHAWj2sgSYd18qYPnubQKVmpcEvDb4=; b=YRFPdL91nC86pov+Yobrkfo6TuhJkGCwVCUZEdXBlLElFf9MpA61axYu5icLujoAic CfEnM/P1TuqbZGwzW9l0mGmFXdSFmVV2uQ59a19zH3JFPD+1sJWZyhyexN19nBDrqsSP 7E18Sf9abJwClrZ578sPRHkyIdseY18MTcXOEIQuR6XvXDwqgT9HXhiFUAgay/3BWjEH qOB3LIXdLRh+gf7RWPFNoBc4I0il/zp+xct31MvFCAFPlsZWPEoHq6bY0PlYypiSjLXe uVlAWuL1ikqH3AnmZ8ZAAoHPedFdcyldnccxprnmPdIqHKc4+9WG3xrKnUX+q/z1LXca HSFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352409; x=1746957209; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=e/d0BPzGs3MdNKHAWj2sgSYd18qYPnubQKVmpcEvDb4=; b=tTb7T6caDyF/tf1y3Kvcg59EZ73pZFmed8du9QTiLEz/DnsrgryJ5K6ysJhobSMqhd WYGOlCvmq6OfMUcG9aeGAgXFvSgKYCCFMlLXFv1aKrh/Y/ziKWvZRlL5SqKWl3CUeody lAunARX1U/VO2gINlkNF03+mhLPOeyEJP44VgNrmGKmYzZnBjhaZH2DiK2EPCo6SnCAn D07/yW+qIC+eNcuyjVM7epxvEsMGPKvshJnUumx0tD+LuXn4wPuSuQF9qEkSfHu0pUhZ 68i0N5WsjMGWyWLU50uI9qSfsTbG1DvLLNKiuROl32APYPnMhStszKWzRlT2FtSjJTbq AwHA== X-Gm-Message-State: AOJu0YxhD5EKQQLlLLYjp2KgQi7VV6KXYWI3DlSwU4NYlS7PUth3+a5l iQ8ytc1+dwnioJEgIcrkAWORyPycsgLC/QymskkkNeQGe/DyRsOu0sexiv90KnbJUoHa4Q== X-Google-Smtp-Source: AGHT+IHIinamTewPRChs9JThiYfWnsnrrjk8ws/eEYLogc6R+jucO0qeiQ8XF/VTmlh3M6rTNXmq/YUh X-Received: from wmbes9.prod.google.com ([2002:a05:600c:8109:b0:43b:d523:6ad8]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3508:b0:43d:fa59:bced with SMTP id 5b1f17b1804b1-441c493401bmr18743425e9.32.1746352409312; Sun, 04 May 2025 02:53:29 -0700 (PDT) Date: Sun, 4 May 2025 11:52:42 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6173; i=ardb@kernel.org; h=from:subject; bh=/+cX53OtqhrxbtCxNs51oh/f3aA1xTFZJXd9z8j8Auw=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4h2b7eGfSkcmx3j/ne8wMfvZx1qhF8/ZNwQ6a77P0 xO0sP/WUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACZSJM7w31f67yZmU3ZZ5cy9 TL1K5191vYrV2V0vmpf4497f8I/l9xkZbu+dJHyDK2Jl6QzFnfsTY37a37x8/JJb+XJWDebV3w+ b8AIA X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-37-ardb+git@google.com> Subject: [RFT PATCH v2 12/23] x86/sev: Unify SEV-SNP hypervisor feature check From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel The decompressor and the core kernel both check the hypervisor feature mask exposed by the hypervisor, but test it in slightly different ways. This disparity seems unintentional, and simply a result of the fact that the decompressor and the core kernel evolve differently over time when it comes to setting up the SEV-SNP execution context. So move the HV feature check into a helper function and call that instead. For the core kernel, move the check to an earlier boot stage, right after the point where it is established that the guest is executing in SEV-SNP mode. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/sev.c | 19 +---------- arch/x86/boot/startup/sev-shared.c | 33 +++++++++++++++----- arch/x86/boot/startup/sme.c | 2 ++ arch/x86/coco/sev/core.c | 11 ------- arch/x86/include/asm/sev-internal.h | 2 +- arch/x86/include/asm/sev.h | 2 ++ 6 files changed, 32 insertions(+), 37 deletions(-) diff --git a/arch/x86/boot/compressed/sev.c b/arch/x86/boot/compressed/sev.c index 18ba81c0b0ed..79550736ad2a 100644 --- a/arch/x86/boot/compressed/sev.c +++ b/arch/x86/boot/compressed/sev.c @@ -414,24 +414,7 @@ void sev_enable(struct boot_params *bp) sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SEV_ES_PROT_UNSUPPORTED); } - /* - * SNP is supported in v2 of the GHCB spec which mandates support for HV - * features. - */ - if (sev_status & MSR_AMD64_SEV_SNP_ENABLED) { - u64 hv_features; - - hv_features = get_hv_features(); - if (!(hv_features & GHCB_HV_FT_SNP)) - sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SNP_UNSUPPORTED); - - /* - * Running at VMPL0 is required unless an SVSM is present and - * the hypervisor supports the required SVSM GHCB events. - */ - if (snp_vmpl > 0 && !(hv_features & GHCB_HV_FT_SNP_MULTI_VMPL)) - sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_NOT_VMPL0); - } + snp_check_hv_features(); if (snp && !(sev_status & MSR_AMD64_SEV_SNP_ENABLED)) error("SEV-SNP supported indicated by CC blob, but not SEV status MSR."); diff --git a/arch/x86/boot/startup/sev-shared.c b/arch/x86/boot/startup/sev-shared.c index b1f4b9b15045..b72c2b9c40c3 100644 --- a/arch/x86/boot/startup/sev-shared.c +++ b/arch/x86/boot/startup/sev-shared.c @@ -78,16 +78,10 @@ sev_es_terminate(unsigned int set, unsigned int reason) asm volatile("hlt\n" : : : "memory"); } -/* - * The hypervisor features are available from GHCB version 2 onward. - */ -u64 get_hv_features(void) +static u64 __head get_hv_features(void) { u64 val; - if (ghcb_version < 2) - return 0; - sev_es_wr_ghcb_msr(GHCB_MSR_HV_FT_REQ); VMGEXIT(); @@ -98,6 +92,31 @@ u64 get_hv_features(void) return GHCB_MSR_HV_FT_RESP_VAL(val); } +u64 __head snp_check_hv_features(void) +{ + /* + * SNP is supported in v2 of the GHCB spec which mandates support for HV + * features. + */ + if (sev_status & MSR_AMD64_SEV_SNP_ENABLED) { + u64 hv_features; + + hv_features = get_hv_features(); + if (!(hv_features & GHCB_HV_FT_SNP)) + sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SNP_UNSUPPORTED); + + /* + * Running at VMPL0 is required unless an SVSM is present and + * the hypervisor supports the required SVSM GHCB events. + */ + if (snp_vmpl > 0 && !(hv_features & GHCB_HV_FT_SNP_MULTI_VMPL)) + sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_NOT_VMPL0); + + return hv_features; + } + return 0; +} + bool sev_es_negotiate_protocol(void) { u64 val; diff --git a/arch/x86/boot/startup/sme.c b/arch/x86/boot/startup/sme.c index ade5ad5060e9..7fc6a689cefe 100644 --- a/arch/x86/boot/startup/sme.c +++ b/arch/x86/boot/startup/sme.c @@ -524,6 +524,8 @@ void __head sme_enable(struct boot_params *bp) if (snp_en ^ !!(msr & MSR_AMD64_SEV_SNP_ENABLED)) snp_abort(); + sev_hv_features = snp_check_hv_features(); + /* Check if memory encryption is enabled */ if (feature_mask == AMD_SME_BIT) { if (!(bp->hdr.xloadflags & XLF_MEM_ENCRYPTION)) diff --git a/arch/x86/coco/sev/core.c b/arch/x86/coco/sev/core.c index 36edf670ff19..106c231d8ded 100644 --- a/arch/x86/coco/sev/core.c +++ b/arch/x86/coco/sev/core.c @@ -1264,17 +1264,6 @@ void __init sev_es_init_vc_handling(void) if (!sev_es_check_cpu_features()) panic("SEV-ES CPU Features missing"); - /* - * SNP is supported in v2 of the GHCB spec which mandates support for HV - * features. - */ - if (cc_platform_has(CC_ATTR_GUEST_SEV_SNP)) { - sev_hv_features = get_hv_features(); - - if (!(sev_hv_features & GHCB_HV_FT_SNP)) - sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SNP_UNSUPPORTED); - } - /* Initialize per-cpu GHCB pages */ for_each_possible_cpu(cpu) { alloc_runtime_data(cpu); diff --git a/arch/x86/include/asm/sev-internal.h b/arch/x86/include/asm/sev-internal.h index 4335711274e3..3aa59ad288ce 100644 --- a/arch/x86/include/asm/sev-internal.h +++ b/arch/x86/include/asm/sev-internal.h @@ -84,6 +84,6 @@ static __always_inline void sev_es_wr_ghcb_msr(u64 val) bool sev_es_negotiate_protocol(void); bool sev_es_check_cpu_features(void); -u64 get_hv_features(void); +void check_hv_features(void); const struct snp_cpuid_table *snp_cpuid_get_table(void); diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h index ca7168cc2118..ca914f7384c6 100644 --- a/arch/x86/include/asm/sev.h +++ b/arch/x86/include/asm/sev.h @@ -418,6 +418,7 @@ struct svsm_call { #ifdef CONFIG_AMD_MEM_ENCRYPT extern u8 snp_vmpl; +extern u64 sev_hv_features; extern void __sev_es_ist_enter(struct pt_regs *regs); extern void __sev_es_ist_exit(void); @@ -495,6 +496,7 @@ void snp_set_memory_private(unsigned long vaddr, unsigned long npages); void snp_set_wakeup_secondary_cpu(void); bool snp_init(struct boot_params *bp); void __noreturn snp_abort(void); +u64 snp_check_hv_features(void); void snp_dmi_setup(void); int snp_issue_svsm_attest_req(u64 call_id, struct svsm_call *call, struct svsm_attest_call *input); void snp_accept_memory(phys_addr_t start, phys_addr_t end); From patchwork Sun May 4 09:52:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887278 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF9BB1DED60 for ; Sun, 4 May 2025 09:53:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352414; cv=none; b=rK1A2albrZ1dR7W9thgNg+KEKT6KHNkWLRMlBGnFfrsmFhUBvAYZBXXiuByJTtLMK8z7IYg1sZUPqcZ3+lOE5enG4rFYqxFUu1fBFbHMuCUaZGMktgHoZ7ayF5oeRBq7BTnYH5ykkUkpl0WZ2BwbTL96lUtjEllCxXPougPL7ec= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352414; c=relaxed/simple; bh=hjTn1iC1IqN51EiMAtjjKytIMtEs0cf/aZrcEIcCD5M=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=dFgawRTaceSsZNXgaP6arPciM4QPgFbUPIh6UivR9TZ3AKEq1iu55ySkbQNBbWjd/P7JNhPxVZrns7KOfeeuxnNXRJKpXJ0sWc3cIPvNd1fzrrzlOoDXGKIiGEHgbux3Sga33/6vSJSb0wzCMZIsDF4kpjiJvw0bVB2NK+9eZT4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Kjaw9ZQW; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Kjaw9ZQW" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-39d917b105bso1654700f8f.2 for ; Sun, 04 May 2025 02:53:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352411; x=1746957211; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=V6n8Ej6Vfc5zcFVzwKiIyzsJGi4r0jcCeV0Mu4Nt7NI=; b=Kjaw9ZQWeCmI6JjRWhfAEAIreR8hJDpOClrsHESC0jSJ2NB8KWm27vyZ2Ph4XrZNam qW/OaS+o3Z/rKoEmjTOafyPFhRReR02F5QiDZRnI2C8pnlQ+oyF1Kp0a/riDP7abtlGm uxTDprT0QNMa3LlCWRmpsA0vW0wS8DUnTWcAMDbFotCxI9C2soAjfh7dA7A8QuKLUte5 4iNKR+np3suISLKEztDLbMPQwgPpLa25HsvXNgsJW0iRNK90cmKrG++65nxgzlSYcG0n xZDpC7wSfZNg/oywnq61f13nvMRu4kSdjKGI263BqVgGLBhnrolaiyWOx4tNV3BVbP4W jDLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352411; x=1746957211; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=V6n8Ej6Vfc5zcFVzwKiIyzsJGi4r0jcCeV0Mu4Nt7NI=; b=n3nZk1Ll7syOIZvaAkmC/DCOzdMNkGYnIIh6X+pMgVHWu/bLdd3cSL7C6vviLZRx8K 7PHfRbdLLWkz/7EfFtu8wMATlLmURNYRMDGoFCpcoL/ijLELp4r2vNxowbWzgSyLCJ86 ylEv4vUKANSFMqXJsBbH88XiMr1DgEHD85Gtacoiqave2VwHoCS4kEqZgMZBoX49Cc88 R/0S6lwOvYtTzR1T/p5KR34/si3b8C2m1xXMlGvwDpInTWdTNK3hF4jyLscdP4icRQui Gt0eo3n2fb6FqE1MAeFtEN8mmJfgoVaKV7G1/EM9dgsj8ib+UW0JpYFHtRBUUNsK34ci nztw== X-Gm-Message-State: AOJu0Yyc1FwzHySXzNLLx1FQCKJIOvweN/FxGtTLnfVD1x+japl9TOKJ gbV56VCd/Ak1sp80t1gW7M0cz9g5108VWh1WR0/8FiDRxpaDGxWnB1hciqBIeT91HGjI5A== X-Google-Smtp-Source: AGHT+IEAQx+886S9LMNHFdxflvzeiaW5o4RQOjeHMaXqTly8oAOoMo9psvzGvfzUDK7SOeb4Nxqg5m98 X-Received: from wmbgw7.prod.google.com ([2002:a05:600c:8507:b0:43d:8244:7f6d]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a5d:5f86:0:b0:3a0:8712:5391 with SMTP id ffacd0b85a97d-3a09fd6dcecmr2392107f8f.12.1746352411407; Sun, 04 May 2025 02:53:31 -0700 (PDT) Date: Sun, 4 May 2025 11:52:43 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1174; i=ardb@kernel.org; h=from:subject; bh=sXN2aX5uRXQRstOSIZ62Xtcz/h/qKJPn4/i4S0dGRas=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4v2HyGnev0N6anm+JM26ULtJNz6z/53I56jNaqUaB skH4mI7SlkYxDgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwER4xBgZlgW/DL9mcOsZ479S 10fru2/8u3Hs58yL9/7/V/v59HPMzGJGhl8rNp3LLzVKW3XTTHZl3564w4espBaeltqndmjhLub fQjwA X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-38-ardb+git@google.com> Subject: [RFT PATCH v2 13/23] x86/linkage: Add SYM_PIC_ALIAS() macro helper to emit symbol aliases From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel Startup code that may execute from the early 1:1 mapping of memory will be confined into its own address space, and only be permitted to access ordinary kernel symbols if this is known to be safe. Introduce a macro helper SYM_PIC_ALIAS() that emits a __pi_ prefixed alias for a symbol, which allows startup code to access it. Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/linkage.h | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/arch/x86/include/asm/linkage.h b/arch/x86/include/asm/linkage.h index b51d8a4673f5..9d38ae744a2e 100644 --- a/arch/x86/include/asm/linkage.h +++ b/arch/x86/include/asm/linkage.h @@ -141,5 +141,15 @@ #define SYM_FUNC_START_WEAK_NOALIGN(name) \ SYM_START(name, SYM_L_WEAK, SYM_A_NONE) +/* + * Expose 'sym' to the startup code in arch/x86/boot/startup/, by emitting an + * alias prefixed with __pi_ + */ +#ifdef __ASSEMBLER__ +#define SYM_PIC_ALIAS(sym) SYM_ALIAS(__pi_ ## sym, sym, SYM_L_GLOBAL) +#else +#define SYM_PIC_ALIAS(sym) extern typeof(sym) __PASTE(__pi_, sym) __alias(sym) +#endif + #endif /* _ASM_X86_LINKAGE_H */ From patchwork Sun May 4 09:52:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887277 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A7CF1E5B7C for ; Sun, 4 May 2025 09:53:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352419; cv=none; b=cu4eUKCU7aWFsfv3Ed9M3jtcUG+UkfMlckeDPb+ijT/YDKRmAnibUtLgOkblQ27oEh76VChPWajtyw2lnY6YBQUMDRWaCGIvDlbKGahCFealf20bPY0/zTjkyolJ9ep/3MHmS276PJk5CjG6bwUSKO6w5Vy27imeosBlMICL1i8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352419; c=relaxed/simple; bh=ZBSTtuc/SHZrCynYqyN5o0yl/kE4GBgwDpGzNrdh3Pg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=eWfLTSky1S/a5TAtzTLCA5ggNYZQgvzJXHTaa24XD+aXwrZXY5infCx2qgIE6BLZdYhbPpmnXj5W8+saxhw5rN2W34LuN7eo/oxujfMFXNGQl7bhzTpZ34c8dKoDxpeffGwZnhDMIfgujqbFraJ02rJ3MHOgrKpL9+QKmTiyhUk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=k/RUd3+N; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="k/RUd3+N" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-39131f2bbe5so1040627f8f.3 for ; Sun, 04 May 2025 02:53:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352416; x=1746957216; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=vtMDjkMYmNedwHhZa0cXr3HfTQY0/WG8XydWhWIUIFo=; b=k/RUd3+NhBz1j0jvxXZIyfL0pYq3mcMWW6llztqWOHN1RLyic6mWSg+ZygvoqUhYgG Tb7EzAaTQYTih8IXXHvSYNkX57TLF349t+9I90ILke9grERwg0ogqR78/1rAU6lmPdQk AW8a56CCKd3HK1OlguyW/9jh0ilEzE+soVuJG07hTFqxx9J59G0G3AANrsHs+jKFATaa YQHAbEGNAdrOd5r2DI5PAuFLRLUp7KnSV7LOFq8sj08X81kbYJUY2UIHq/20NuLMkSX5 pxEYx3Q7meleZbacl2S4pTYZYlzzvHtITRdLhHkFt5dMKj8QSesvbfhFYfviIJHE7JJn CBkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352416; x=1746957216; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vtMDjkMYmNedwHhZa0cXr3HfTQY0/WG8XydWhWIUIFo=; b=eZd3sWjBHZrIUa4YOg4ZDL75EtL4FnJlGulc8YZUPhrz4XkSMBpRFoz97QvMGqBbR5 tUGXZmhpeqB5gu8lYuQoHR/LTyXxMDDjon9lDg4OAoYK23rPU/u9bX7qy2ECAtABc4H8 tG77V94lrs2/rjwL5MsAUBzNgDubU4WkS85cPeOaBYiNu5hwJ+DG+2KUge/u2EmRjpOD 2DD2BoRKbg7JLWsMvY0Fzq5wwK4WAKIEs3SGPSLPkXzBCPhXUMxNCHmjXMIlQMoNx+FW HLy/IfEJSe1diVrXE3bGJfLWRESdQjUauXPiRw4YvZ/FVEqmBJwzrtNJuMcX2HW1xPVL /Jnw== X-Gm-Message-State: AOJu0YxThxJ4XqRf+dEaaUcVEvGqGPkmi49xiAoH8Yjmpqk8ecrEfdHX /oW7/M15kT7EBYCygXik0BEyXiBpKAgQ8jAk+H6oflpI8KZvWimsWbLUdvJsMnFMcwuKDA== X-Google-Smtp-Source: AGHT+IHzM9sbmgfFD5bqalsjiElI995bcq0K3rL9cQY0mQCUn5LiV79+CyQPKzABCIynM/3ZCgU5uMRJ X-Received: from wmbfl15.prod.google.com ([2002:a05:600c:b8f:b0:441:b38b:9cc3]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:40e1:b0:3a0:9dd8:4ba7 with SMTP id ffacd0b85a97d-3a09fdd8438mr2913867f8f.52.1746352415647; Sun, 04 May 2025 02:53:35 -0700 (PDT) Date: Sun, 4 May 2025 11:52:45 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1393; i=ardb@kernel.org; h=from:subject; bh=jNeHtJDyRJuUYx7C5oayW7yyMKmGvkftnluPCF2j4+Q=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4lOksVvdH1ZJ/7h68fyVr9wDQudmxb6pm5Vi17/r8 p80u1kdpSwMYhwMsmKKLAKz/77beXqiVK3zLFmYOaxMIEMYuDgFYCKSnAz/rC/c3Xm/NK0kNDep kpc7yjN026x34Umi9g26j2q/RpzZAVQhxp5Zmf8hMvDjDn6e5yr5grVLeXlqlTe86X6guuvTQ2Y A X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-40-ardb+git@google.com> Subject: [RFT PATCH v2 15/23] x86/boot: Provide __pti_set_user_pgtbl() to startup code From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel The SME encryption startup code populates page tables using the ordinary set_pXX() helpers, and in a PTI build, these will call out to __pti_set_user_pgtbl() to manipulate the shadow copy of the page tables for user space. This is unneeded for the startup code, which only manipulates the swapper page tables, and so this call could be avoided in this particular case. So instead of exposing the ordinary __pti_set_user_pgtblt() to the startup code after its gets confined into its own symbol space, provide an alternative which just returns pgd, which is always correct in the startup context. Annotate it as __weak for now, this will be dropped in a subsequent patch. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/startup/sme.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/boot/startup/sme.c b/arch/x86/boot/startup/sme.c index 7fc6a689cefe..1e9f1f5a753c 100644 --- a/arch/x86/boot/startup/sme.c +++ b/arch/x86/boot/startup/sme.c @@ -557,3 +557,12 @@ void __head sme_enable(struct boot_params *bp) cc_vendor = CC_VENDOR_AMD; cc_set_mask(me_mask); } + +#ifdef CONFIG_MITIGATION_PAGE_TABLE_ISOLATION +/* Local version for startup code, which never operates on user page tables */ +__weak +pgd_t __pti_set_user_pgtbl(pgd_t *pgdp, pgd_t pgd) +{ + return pgd; +} +#endif From patchwork Sun May 4 09:52:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887899 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 632721E9907 for ; Sun, 4 May 2025 09:53:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352421; cv=none; b=boz4EhRYPPvp7jG4KUHjaUt9vddxol0kXM022OraZXgsoAUK8BjCf2SSLbWt7/koHXX3/A+4Q4OhMyN0FjAe8N6MNRcCfhhZ320VRuYdHPc1Qckw3aTWPDypFlwW0u+fsIYI7cwfdRsVKvNwgpgkFwuETD9lHB864CjIfD1O8S0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352421; c=relaxed/simple; bh=aHloFHnPa0bpo40NNxCXch1Gt5dJk47uA3Hl9cxt5Dc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=OavxfCYyNhmreHg3wKY/0O99UEk+4eSQxoyM/fkSn6ZOgKHShlTLC/OauKwt3N4intvsV+XzPq2xEPHOyETe5jAs/AINlqex0wtxVZJ3NwDRRqEZ3/c2T/4M3bkxPIOUkibXJDibN4Wgyh2SEqcXu3PRtsfzZM4wuLmp0FkDFTs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=y152u7Ck; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="y152u7Ck" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-39141ffa913so1624424f8f.2 for ; Sun, 04 May 2025 02:53:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352418; x=1746957218; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0qjh3aA4OXGqpKzUr6s6nXvvtZjVv7X5+NLywnovwT8=; b=y152u7CktdNDld+IA4sf/+2eZxYYWvegL6/4BJ3Z+5TRjjVdQ0znyLHPn4JolOvzS3 JT2Kz+cywj9oIsbwmK1GrrfFE7FEBW03rE8n9sz4h7Tk9vFPf61P40a0k3LoRWCyzx6q ZIfdFNIjSHYK+tTXzzU0ncdWAY9JsgoZDAJU3k0OWgq+uWxoEDcECNHeFe2S3LQmFof3 hO6kXi7SjOZ/YtR/u8uHZJnoe29Ae9YWTjase5qc9sdjYZfWNxGcl3xvUaLqRFgkHirO gcpmRyXWKkBHVDNRdF1AsswhR35y4z0geGB3GaKHEe7iRhe2PHEJAYmeV1vvpD8qLA2R HaiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352418; x=1746957218; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0qjh3aA4OXGqpKzUr6s6nXvvtZjVv7X5+NLywnovwT8=; b=fkCRqC670WFyQLDOCCQOmchPjEDaxOefpUkrFJTj4RXYoCJ+ILHxYVWdjhozpoIwlQ PMlriwVLTO97vmrha8/6TVueok2B/5cyXVXomGrF7DO21K4SkR1VDD2qH+1tdiwyfpNy d3WE4ZkGeyEGREG2kibIFI0XrWEvr/dcA9wa1PerueIN2DKwp6iIk4YWQe4oPNLrgs1x /Mlb3vPIRxvUSpMyx4RFmqjF1HEPmW39Gpx16t4D7FF9qsDS5bcwEoQrjg3R5aQpzJYd 01iDsr8u8Hml4jR9VjDsvwBsKwOWBR+FtbWm1/YD2aY/xHwjDTBpUrDN+XJh6LFAZWAN bdUw== X-Gm-Message-State: AOJu0Yyj94URRx8X0deUtwl9Pdm5SliuDBhpvTfB7B/7/3CxWPeTpPBT +JABPIL9XzPYgXlIWgMUn5faopLpPpoeyChZwdI+wC09n3kI1M0oEla1udZfO8xp1LwU+g== X-Google-Smtp-Source: AGHT+IFyEvpKlKi4oAJU+SLd/0+ezNebN25q32YXRpUDQMUSoQruZndAojpWGPc582aK3iHaU0tPRNKJ X-Received: from wrbfo4.prod.google.com ([2002:a05:6000:2904:b0:39a:be42:4b14]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:2481:b0:3a0:90c3:dd90 with SMTP id ffacd0b85a97d-3a09fd6d9ddmr3209750f8f.11.1746352417796; Sun, 04 May 2025 02:53:37 -0700 (PDT) Date: Sun, 4 May 2025 11:52:46 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5163; i=ardb@kernel.org; h=from:subject; bh=k2TTUi4nLqQXdxGniT4a97pQU/HXQmCwOe1Xt6uAJe0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4rN2uW9WZE+NM5/SipvKpouSkqLaV/fdDWEze5yt8 3bH6TcdpSwMYhwMsmKKLAKz/77beXqiVK3zLFmYOaxMIEMYuDgFYCJq1YwM1yZvL2tMmLdA2f3I nJhHmrJdoWeaG9mmWnyJMdHX/cp5h+F/7pa6pjM2m5ckS6zguFBhfbOX+0VGQNWflg2r9wct+B/ BBAA= X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-41-ardb+git@google.com> Subject: [RFT PATCH v2 16/23] x86/sev: Provide PIC aliases for SEV related data objects From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/sev.c | 4 ++ arch/x86/boot/startup/sev-shared.c | 20 -------- arch/x86/boot/startup/sev-startup.c | 17 ------- arch/x86/coco/sev/core.c | 48 ++++++++++++++++++++ 4 files changed, 52 insertions(+), 37 deletions(-) diff --git a/arch/x86/boot/compressed/sev.c b/arch/x86/boot/compressed/sev.c index 79550736ad2a..8118a270485f 100644 --- a/arch/x86/boot/compressed/sev.c +++ b/arch/x86/boot/compressed/sev.c @@ -38,6 +38,10 @@ struct ghcb *boot_ghcb; #define __BOOT_COMPRESSED u8 snp_vmpl; +u16 ghcb_version; + +struct svsm_ca *boot_svsm_caa; +u64 boot_svsm_caa_pa; /* Include code for early handlers */ #include "../../boot/startup/sev-shared.c" diff --git a/arch/x86/boot/startup/sev-shared.c b/arch/x86/boot/startup/sev-shared.c index b72c2b9c40c3..9ab132bed77b 100644 --- a/arch/x86/boot/startup/sev-shared.c +++ b/arch/x86/boot/startup/sev-shared.c @@ -19,26 +19,6 @@ #define WARN(condition, format...) (!!(condition)) #endif -/* - * SVSM related information: - * During boot, the page tables are set up as identity mapped and later - * changed to use kernel virtual addresses. Maintain separate virtual and - * physical addresses for the CAA to allow SVSM functions to be used during - * early boot, both with identity mapped virtual addresses and proper kernel - * virtual addresses. - */ -struct svsm_ca *boot_svsm_caa __ro_after_init; -u64 boot_svsm_caa_pa __ro_after_init; - -/* - * Since feature negotiation related variables are set early in the boot - * process they must reside in the .data section so as not to be zeroed - * out when the .bss section is later cleared. - * - * GHCB protocol version negotiated with the hypervisor. - */ -u16 ghcb_version __ro_after_init; - /* Copy of the SNP firmware's CPUID page. */ static struct snp_cpuid_table cpuid_table_copy __ro_after_init; diff --git a/arch/x86/boot/startup/sev-startup.c b/arch/x86/boot/startup/sev-startup.c index ca6a9863ffab..2c2a5a043f18 100644 --- a/arch/x86/boot/startup/sev-startup.c +++ b/arch/x86/boot/startup/sev-startup.c @@ -41,23 +41,6 @@ #include #include -/* For early boot hypervisor communication in SEV-ES enabled guests */ -struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE); - -/* - * Needs to be in the .data section because we need it NULL before bss is - * cleared - */ -struct ghcb *boot_ghcb __section(".data"); - -/* Bitmap of SEV features supported by the hypervisor */ -u64 sev_hv_features __ro_after_init; - -/* Secrets page physical address from the CC blob */ -u64 sev_secrets_pa __ro_after_init; - -/* For early boot SVSM communication */ -struct svsm_ca boot_svsm_ca_page __aligned(PAGE_SIZE); /* * Nothing shall interrupt this code path while holding the per-CPU diff --git a/arch/x86/coco/sev/core.c b/arch/x86/coco/sev/core.c index 106c231d8ded..33332c4299b9 100644 --- a/arch/x86/coco/sev/core.c +++ b/arch/x86/coco/sev/core.c @@ -45,6 +45,43 @@ #include #include +/* For early boot hypervisor communication in SEV-ES enabled guests */ +struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE); +SYM_PIC_ALIAS(boot_ghcb_page); + +/* + * Needs to be in the .data section because we need it NULL before bss is + * cleared + */ +struct ghcb *boot_ghcb __section(".data"); +SYM_PIC_ALIAS(boot_ghcb); + +/* Bitmap of SEV features supported by the hypervisor */ +u64 sev_hv_features __ro_after_init; +SYM_PIC_ALIAS(sev_hv_features); + +/* Secrets page physical address from the CC blob */ +u64 sev_secrets_pa __ro_after_init; +SYM_PIC_ALIAS(sev_secrets_pa); + +/* For early boot SVSM communication */ +struct svsm_ca boot_svsm_ca_page __aligned(PAGE_SIZE); +SYM_PIC_ALIAS(boot_svsm_ca_page); + +/* + * SVSM related information: + * During boot, the page tables are set up as identity mapped and later + * changed to use kernel virtual addresses. Maintain separate virtual and + * physical addresses for the CAA to allow SVSM functions to be used during + * early boot, both with identity mapped virtual addresses and proper kernel + * virtual addresses. + */ +struct svsm_ca *boot_svsm_caa __ro_after_init; +SYM_PIC_ALIAS(boot_svsm_caa); + +u64 boot_svsm_caa_pa __ro_after_init; +SYM_PIC_ALIAS(boot_svsm_caa_pa); + DEFINE_PER_CPU(struct svsm_ca *, svsm_caa); DEFINE_PER_CPU(u64, svsm_caa_pa); @@ -118,6 +155,17 @@ DEFINE_PER_CPU(struct sev_es_save_area *, sev_vmsa); */ u8 snp_vmpl __ro_after_init; EXPORT_SYMBOL_GPL(snp_vmpl); +SYM_PIC_ALIAS(snp_vmpl); + +/* + * Since feature negotiation related variables are set early in the boot + * process they must reside in the .data section so as not to be zeroed + * out when the .bss section is later cleared. + * + * GHCB protocol version negotiated with the hypervisor. + */ +u16 ghcb_version __ro_after_init; +SYM_PIC_ALIAS(ghcb_version); static u64 __init get_snp_jump_table_addr(void) { From patchwork Sun May 4 09:52:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887276 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 970541EDA34 for ; Sun, 4 May 2025 09:53:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352423; cv=none; b=MguHqO2h1jExqtjBe9zJJAbwavncB5Fdr+xtHSgp5tFB+9AFPxY86Ls37ij1IuCoWsV0panfnpQsIbFkRxPNV2UeyKEr2etIJMNOfmLRMtWw1yyDpxY/DZ7Mjo5HvaC3xh59jI1JjgV3U188yQA6CoIRseKHFmDIVEUZeYnoV0M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352423; c=relaxed/simple; bh=ySFC3DvBrtMkqxLkZqKoMPHnxZg5PFYGfuyRwinvm3I=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=AOevtIEeVZ04zjdixdckpxOQxlM5KfcDMZTltcADUaPYQwd1TT0MK4lIoaeDTCNiDEScnk47rO2dGnfL0Ra9Rsnul/YRHQOV6Qf0688C5vH2syZJT/YhgvPZVc94DBiEMoEZw7yCJV/POaC2qaQM3Um74QbYl/X7AUfp2w7TWog= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=daSkEe+U; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="daSkEe+U" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-43cfda30a3cso15895605e9.3 for ; Sun, 04 May 2025 02:53:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352420; x=1746957220; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gVvnJaxsL7lB6x7Rus/Rk5JP6sPC9wQuUZ5nrdcRGng=; b=daSkEe+UPwmo3f+BhS/Zck+4+efV8sJ4XlteVDHF+Cke1oS5xN2sTi+qmBdhid0c8T v5e8rp9qgUvo+fD0G4xRtjNxuRiz/ieMv5yMIujdhJIFlfYLDci48Bo/rM7rd2pUkfOC rRz5snQzn/N6pZA84Nj6wuvvoitcO3pVhKuLaEjtdti52utDZZP8bdORwSMWnm0pS2xC uGwc+AYjLHyltkA8oMkrmqQKDXn3pk5EbceCwkv3UZRfmvoUKuGMSksiwHWrMa760Ni2 chB51zO8/1HmN/PG9YN5TgriK3FyyeUBWqWXqCyBlvMxg5quD6YgTmZCsrjxkMcH0vRJ sj0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352420; x=1746957220; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gVvnJaxsL7lB6x7Rus/Rk5JP6sPC9wQuUZ5nrdcRGng=; b=iIo+vWDctMwMbxsJPAKU92D9z4nVSTJ6WMYmj4w4lWLriJxAgZhjgvFTqubUaubv6l T8Q5JCKcjYKncrxr/0NyI0Die/doqAsYYPE90RvqVivW9MlTPOBZ99Ud907tAvBxHHO0 CmToUFjHMoYf6ADBLFRSjii4a6laI8bqensK5Y9j2QwRkWpxJ+NXx2KNJ4LdqOGLr2jb 6LJ6008nRZrb7OZnqja5+47ikGwl+b84wICcqD78xIXxW6huvgICrlX/Os62Mrh5uMfc 4UY24i3asPbwfzYMftqCYFl/iskjwaBREcrSACoaj5u6/UvcFHz+uGWdqjuq+mPrLsLK 4oyg== X-Gm-Message-State: AOJu0YyCStjDegj/awAZ69OhIyDkNd90NRC9r4AEHhmYwiG3ZXdayPvH Wm2xhhw0+pRblyuFtf4no7Pf7/qU0z8aauftSZMxXHE7PfmjvfUbm1JAjcy8FbS/JIT30g== X-Google-Smtp-Source: AGHT+IGC9HB79SjiFADEJMdvUMIsjyiGMqsclow0OUbWPcWQeEIwslxVQavWgsvcq85J+GxQSaX3XDDt X-Received: from wmbhj22.prod.google.com ([2002:a05:600c:5296:b0:43d:5828:13ee]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1914:b0:43d:47e:3205 with SMTP id 5b1f17b1804b1-441c48bca55mr21296145e9.11.1746352420214; Sun, 04 May 2025 02:53:40 -0700 (PDT) Date: Sun, 4 May 2025 11:52:47 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5759; i=ardb@kernel.org; h=from:subject; bh=S+tSbPFyOP+LxITxmH/ySCnhoV8I/NLanALIjXIGmXk=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4quoq/sMs6Mtz/qY7tUtbbg4ZcsiiY4jLe68EaHfp rzkbP7QUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACZisoXhf+aCD3M6nzT+k6yu OCdeXHLm9DLVW6xX/DKDMpfbbHyQOImR4dqmV7XnlnWxPNgiNOtpWY1WqsCaU0G/ZmzS2/x+la6 ALC8A X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-42-ardb+git@google.com> Subject: [RFT PATCH v2 17/23] x86/sev: Move __sev_[get|put]_ghcb() into separate noinstr object From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel Rename sev-nmi.c to sev-noinstr.c, and move the get/put GHCB routines into it too. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/startup/sev-startup.c | 75 -------------------- arch/x86/coco/sev/Makefile | 6 +- arch/x86/coco/sev/{sev-nmi.c => sev-noinstr.c} | 74 +++++++++++++++++++ 3 files changed, 77 insertions(+), 78 deletions(-) diff --git a/arch/x86/boot/startup/sev-startup.c b/arch/x86/boot/startup/sev-startup.c index 2c2a5a043f18..6ba50723d546 100644 --- a/arch/x86/boot/startup/sev-startup.c +++ b/arch/x86/boot/startup/sev-startup.c @@ -41,84 +41,9 @@ #include #include - -/* - * Nothing shall interrupt this code path while holding the per-CPU - * GHCB. The backup GHCB is only for NMIs interrupting this path. - * - * Callers must disable local interrupts around it. - */ -noinstr struct ghcb *__sev_get_ghcb(struct ghcb_state *state) -{ - struct sev_es_runtime_data *data; - struct ghcb *ghcb; - - WARN_ON(!irqs_disabled()); - - data = this_cpu_read(runtime_data); - ghcb = &data->ghcb_page; - - if (unlikely(data->ghcb_active)) { - /* GHCB is already in use - save its contents */ - - if (unlikely(data->backup_ghcb_active)) { - /* - * Backup-GHCB is also already in use. There is no way - * to continue here so just kill the machine. To make - * panic() work, mark GHCBs inactive so that messages - * can be printed out. - */ - data->ghcb_active = false; - data->backup_ghcb_active = false; - - instrumentation_begin(); - panic("Unable to handle #VC exception! GHCB and Backup GHCB are already in use"); - instrumentation_end(); - } - - /* Mark backup_ghcb active before writing to it */ - data->backup_ghcb_active = true; - - state->ghcb = &data->backup_ghcb; - - /* Backup GHCB content */ - *state->ghcb = *ghcb; - } else { - state->ghcb = NULL; - data->ghcb_active = true; - } - - return ghcb; -} - /* Include code shared with pre-decompression boot stage */ #include "sev-shared.c" -noinstr void __sev_put_ghcb(struct ghcb_state *state) -{ - struct sev_es_runtime_data *data; - struct ghcb *ghcb; - - WARN_ON(!irqs_disabled()); - - data = this_cpu_read(runtime_data); - ghcb = &data->ghcb_page; - - if (state->ghcb) { - /* Restore GHCB from Backup */ - *ghcb = *state->ghcb; - data->backup_ghcb_active = false; - state->ghcb = NULL; - } else { - /* - * Invalidate the GHCB so a VMGEXIT instruction issued - * from userspace won't appear to be valid. - */ - vc_ghcb_invalidate(ghcb); - data->ghcb_active = false; - } -} - void __head early_set_pages_state(unsigned long vaddr, unsigned long paddr, unsigned long npages, enum psc_op op) diff --git a/arch/x86/coco/sev/Makefile b/arch/x86/coco/sev/Makefile index db3255b979bd..b348c9486315 100644 --- a/arch/x86/coco/sev/Makefile +++ b/arch/x86/coco/sev/Makefile @@ -1,9 +1,9 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += core.o sev-nmi.o vc-handle.o +obj-y += core.o sev-noinstr.o vc-handle.o # Clang 14 and older may fail to respect __no_sanitize_undefined when inlining -UBSAN_SANITIZE_sev-nmi.o := n +UBSAN_SANITIZE_sev-noinstr.o := n # GCC may fail to respect __no_sanitize_address when inlining -KASAN_SANITIZE_sev-nmi.o := n +KASAN_SANITIZE_sev-noinstr.o := n diff --git a/arch/x86/coco/sev/sev-nmi.c b/arch/x86/coco/sev/sev-noinstr.c similarity index 61% rename from arch/x86/coco/sev/sev-nmi.c rename to arch/x86/coco/sev/sev-noinstr.c index d8dfaddfb367..b527eafb6312 100644 --- a/arch/x86/coco/sev/sev-nmi.c +++ b/arch/x86/coco/sev/sev-noinstr.c @@ -106,3 +106,77 @@ void noinstr __sev_es_nmi_complete(void) __sev_put_ghcb(&state); } + +/* + * Nothing shall interrupt this code path while holding the per-CPU + * GHCB. The backup GHCB is only for NMIs interrupting this path. + * + * Callers must disable local interrupts around it. + */ +noinstr struct ghcb *__sev_get_ghcb(struct ghcb_state *state) +{ + struct sev_es_runtime_data *data; + struct ghcb *ghcb; + + WARN_ON(!irqs_disabled()); + + data = this_cpu_read(runtime_data); + ghcb = &data->ghcb_page; + + if (unlikely(data->ghcb_active)) { + /* GHCB is already in use - save its contents */ + + if (unlikely(data->backup_ghcb_active)) { + /* + * Backup-GHCB is also already in use. There is no way + * to continue here so just kill the machine. To make + * panic() work, mark GHCBs inactive so that messages + * can be printed out. + */ + data->ghcb_active = false; + data->backup_ghcb_active = false; + + instrumentation_begin(); + panic("Unable to handle #VC exception! GHCB and Backup GHCB are already in use"); + instrumentation_end(); + } + + /* Mark backup_ghcb active before writing to it */ + data->backup_ghcb_active = true; + + state->ghcb = &data->backup_ghcb; + + /* Backup GHCB content */ + *state->ghcb = *ghcb; + } else { + state->ghcb = NULL; + data->ghcb_active = true; + } + + return ghcb; +} + +noinstr void __sev_put_ghcb(struct ghcb_state *state) +{ + struct sev_es_runtime_data *data; + struct ghcb *ghcb; + + WARN_ON(!irqs_disabled()); + + data = this_cpu_read(runtime_data); + ghcb = &data->ghcb_page; + + if (state->ghcb) { + /* Restore GHCB from Backup */ + *ghcb = *state->ghcb; + data->backup_ghcb_active = false; + state->ghcb = NULL; + } else { + /* + * Invalidate the GHCB so a VMGEXIT instruction issued + * from userspace won't appear to be valid. + */ + vc_ghcb_invalidate(ghcb); + data->ghcb_active = false; + } +} From patchwork Sun May 4 09:52:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887898 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 946DF1EF388 for ; Sun, 4 May 2025 09:53:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352425; cv=none; b=DkId7NS7IgN2bixM3q7nPj+nSOdTPKzIXYdI/Zah92MsJXqppB4R8qWgsgRfiAH2zH1KVK1Y2ZwPZ+lTAsUho5jfl6GRuo8bPqBN969s6OhwSbK7dikpNj+cVN7BMyiX9IHtB1ErUfbzQahW1bOo8DJW75PPtPitrhM0uLoJ6ug= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352425; c=relaxed/simple; bh=iwQiKCrnRw6e5t0cbBba0LK388tT/jjrdqhKVRDej5Y=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=EYhTk5X0CmvOJSz4yRPtUh9BpTrspMZKsc/mwtgSucDsa8rnEYbdAFqugwcSm24Lo1Fs9etC9pWLh1B63GAAVwh8ckCKUmLcXizLe3d+0FkhervYl05ZJxnLLMe/bQ9ORjVYZZtw0VT7NKJUFT0kr5DC05wQqvXSnuMpl6j/PzE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=qYqZ4xwb; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="qYqZ4xwb" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-43d733063cdso25243885e9.0 for ; Sun, 04 May 2025 02:53:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352422; x=1746957222; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=09I89rcrSx0lGt2Hht5eDxGNoY/1U5TFE9afFv+NSJg=; b=qYqZ4xwbGYbWZyLbms3wb67/1uPTOjphrcgsmow96gpinHW7MJVOcqy2sqaBX3D/kP UTU03PP7LaHT71R8BHnglrLXGpI9x3LFCsL9XSsr4iQENXPlcRMpEROknFvT2RK0f+ft FgsSRcuMIWFabqDjeABN4acuQhtaOT8C6dlraYM4Fmza0gJrLdhI2m3MLXaRbg25nTfd T4pvpSSTB/60FRmwJBTVpCaE/C++fKIdNml+WATAT7ZM8p4blFLbEd6zo00otzokXfM2 2NJnjCJ0A1mijAHH9PYJm6louk/nOWw0Feu46MbgIuLmOislMknHyG82zcJuzXI/iXeA oTzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352422; x=1746957222; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=09I89rcrSx0lGt2Hht5eDxGNoY/1U5TFE9afFv+NSJg=; b=QKKkjLdKjeE+ddezLy+/5HQORl0rPWya6gg/5ebBgSAXyLsxdpQTKhSvTcClJdbpqX 47w4YjFJIRt2p1wLXSVGoZvvX9g6n0hklAUp8L4dFAngxL055vBLcDB4A4MEmPgTxodF voxO7Ilk0KFmwY4cfFcGsSWzymcTVYsORQgRBcK6a/XG1guOTmLYx0CT8OduWNyC+C46 AxG3pNphJMY5KjTa8O3TFywdeCXQ4v1Hv6TxTM5O/a0Qme0l/p08POYNPnDHAq+/lIa7 r+UbCyhkTYwjbQJWCE7tfk9M2oHfS3fl1jLyuHL1TcRqHtc7HiWKDF2L/c3EIppbC9RH WyIg== X-Gm-Message-State: AOJu0Yzjrlj5ZlABGpE2cs5BLYB04PActu6oDPQ39md0Z926YJOBjiBG n+fWPF7In/YHwVSupQGT9bO3AX29ZfnhVObDx+xMebZpYW/x4JdUj9NVe0T/VXfyGYe9sg== X-Google-Smtp-Source: AGHT+IEHBRZOthoNCWHQ+QuYQRNm81nOsUZ5ai5bBhDx4UgSm07jqGJS8LMx1Ev6Aa64AbUsy2b67Hgg X-Received: from wmbbe3.prod.google.com ([2002:a05:600c:1e83:b0:43d:2235:bd45]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:510f:b0:43c:ec0a:ddfd with SMTP id 5b1f17b1804b1-441c1cc4826mr41965635e9.6.1746352422181; Sun, 04 May 2025 02:53:42 -0700 (PDT) Date: Sun, 4 May 2025 11:52:48 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1672; i=ardb@kernel.org; h=from:subject; bh=Tkjl/5QM7fobSnvXL0Qey90uzdX00dsND7VqTMhk7jY=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4pvD32nLKrTrLjJ3Zcm1/TnFXHT3i8aMvVtNXUri9 mU6bJvfUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACbyeC/D/4ofnQc+LPupJH3P yf1ASoLTi4Dtq3JCtvk86uBnObP6nxHDP61Nms2R7MeXzFid71bEVrCoc2b24aSVhw2k1F3rF1y 4zQoA X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-43-ardb+git@google.com> Subject: [RFT PATCH v2 18/23] x86/sev: Export startup routines for ordinary use From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel Create aliases that expose routines that are part of the startup code to other code in the core kernel. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/startup/exports.h | 14 ++++++++++++++ arch/x86/kernel/vmlinux.lds.S | 2 ++ 2 files changed, 16 insertions(+) diff --git a/arch/x86/boot/startup/exports.h b/arch/x86/boot/startup/exports.h new file mode 100644 index 000000000000..5cbc14c64d61 --- /dev/null +++ b/arch/x86/boot/startup/exports.h @@ -0,0 +1,14 @@ + +/* + * The symbols below are functions that are implemented by the startup code, + * but called at runtime by the SEV code residing in the core kernel. + */ +PROVIDE(early_set_pages_state = __pi_early_set_pages_state); +PROVIDE(early_snp_set_memory_private = __pi_early_snp_set_memory_private); +PROVIDE(early_snp_set_memory_shared = __pi_early_snp_set_memory_shared); +PROVIDE(get_hv_features = __pi_get_hv_features); +PROVIDE(sev_es_check_cpu_features = __pi_sev_es_check_cpu_features); +PROVIDE(sev_es_negotiate_protocol = __pi_sev_es_negotiate_protocol); +PROVIDE(sev_es_terminate = __pi_sev_es_terminate); +PROVIDE(snp_cpuid = __pi_snp_cpuid); +PROVIDE(snp_cpuid_get_table = __pi_snp_cpuid_get_table); diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 9340c74b680d..4aaa1693b262 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -517,3 +517,5 @@ xen_elfnote_entry_value = xen_elfnote_phys32_entry_value = ABSOLUTE(xen_elfnote_phys32_entry) + ABSOLUTE(pvh_start_xen - LOAD_OFFSET); #endif + +#include "../boot/startup/exports.h" From patchwork Sun May 4 09:52:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887275 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 97E951F1506 for ; Sun, 4 May 2025 09:53:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352427; cv=none; b=JT665GtAHvA9uI18b9nCGpgwlEqtxZVDwwfIyF5mNN4uNgXBpANDerJweUoWIK/gtHD1VlzzFdwXu8sTxSuFi1X8mO1SVk3KBSJ/OEBV21/SjHjw+ABTSrmfJsZG8e4Pa9a4K7kZ94oCdm3j3Jr8jmo2+itJSHjgdXTAC25z2/4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352427; c=relaxed/simple; bh=0kv5PyQQgZGwr5MoEfydVYsoz61xoJGS3+ClS7AvY1E=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=BgbdUlGlNO01r2semuT4DS99FmWjOnRP4NOClsp7lVjW1aQ5eavj0oNNm94ilSSVoJj9j45fBKpxoIdjtTOHsyWgyiXv9tl9AICxo0XzcVZtJ1SEB0hq0SE36jBzeYQzF7p41ZmGDv8S1F6CK6f/FFnkYEhNTGUGa3TfbGeVPTs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=TrrLupAG; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="TrrLupAG" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-43f405810b4so17121155e9.1 for ; Sun, 04 May 2025 02:53:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352424; x=1746957224; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=am/u+C2vtdImB5wlXl0RNWC/asTWgVRv3T2s5E1iSBI=; b=TrrLupAGHm29o9oo5PByFCFqQe2m7KZWndQJp84IEmHvk01oxz9WjbkBzHr4+rg2i+ yUTFvS5m7kStAdtaRne4NWO0wl7SiJe+E0JYGjY50GokKlPgrrpfvo+T2kFKxiZVvxEX qNmELI1uVdZfUaQSOabKkCOOh3oz8qGzEo2mkEkdflLTnCjJcU3GCwX8AUUchSYErNcR Li3uRlb6xll1s6y9WSB/TyGobxPXBKbMrmBgh2d+Ikt3wixreurIdwpWzkkqSztvDP4b 9ENK9Bg8hfclZgCihiWo8PUOPDTQE/PH2X0Y9sFTpAGwWTLK2Ohv/2LL+Tnz+EwDuLKv frwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352424; x=1746957224; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=am/u+C2vtdImB5wlXl0RNWC/asTWgVRv3T2s5E1iSBI=; b=u+tDcjaLJLgMIcbdkF3hnDmYQLLSrUdCP3L5Nzwgz7EForCpnJ0ot48I/11f4eKky1 9/JBR4bif/I9EPVtklFMWYLXEQSW3q+vkNfWsS0N4SfEmPEjeyD2uEvzpNJJ7rDmdIOE wUAUbIolIQmyo4Dp+g9A++n3uvXvi43/OyjpueBE88CRLDeBbe3B90xqUpwhRC7urb+G aI646OEoGC6Ul7+gu3nC9ZuUNDSLTZ3c5pQ2z90Z2jN/F52pPZzKUB8wLFcMF/E0Hiok jnPjCWl1huEWxR6YAwey/fP+7S02t6laMjcp49W3Cb+Omui9UGoxfcztVIAwgJuZ1qdz q19Q== X-Gm-Message-State: AOJu0Ywee74bafJLskgvee43AonR3MFTjdELXT5vHVlV5Fr2rPb5oV5+ 670yYEOa2O55QVv72wfjVevjWmDdvUaidgXW5xNBKPzBySw/9l0cRLEGnfkaejdEEhOU6A== X-Google-Smtp-Source: AGHT+IE5qTd3zmGwdZAGiec0nXHYb95d5lMEBRus8RIXHken0LirX+y0hfAo2/LHIhLvc8/uIRrFEs+P X-Received: from wmrm10.prod.google.com ([2002:a05:600c:37ca:b0:43b:bd03:5d2]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:8012:b0:43d:26e3:f2f6 with SMTP id 5b1f17b1804b1-441c48b069emr30474665e9.5.1746352424068; Sun, 04 May 2025 02:53:44 -0700 (PDT) Date: Sun, 4 May 2025 11:52:49 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6615; i=ardb@kernel.org; h=from:subject; bh=K7Qy6HmBAvo+4sm++BxtoNxO67AhO3gRrFNSlDBcNSA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4vthZ5ZDkhcDJp/6u9JMJeeCBa84r2XCsa43/9S2v itZ49bQUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACbyU5fhr+SE8pKJTWn9d5QY F92anPf45ucHDdL753887TbXV7PzeAwjQ3vGOt1Tq5n3eJmaKeyUOLBus22qz4n7r/mXXpDNu6y yjQkA X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-44-ardb+git@google.com> Subject: [RFT PATCH v2 19/23] x86/boot: Created a confined code area for startup code From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel In order to be able to have tight control over which code may execute from the early 1:1 mapping of memory, but still link vmlinux as a single executable, prefix all symbol references in startup code with __pi_, and invoke it from outside using the __pi_ prefix. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/startup/Makefile | 13 +++++++++++++ arch/x86/boot/startup/sev-shared.c | 2 +- arch/x86/boot/startup/sme.c | 1 - arch/x86/coco/sev/core.c | 2 +- arch/x86/coco/sev/vc-handle.c | 1 - arch/x86/include/asm/setup.h | 1 + arch/x86/include/asm/sev.h | 1 + arch/x86/kernel/head64.c | 2 +- arch/x86/kernel/head_64.S | 8 ++++---- arch/x86/mm/mem_encrypt_boot.S | 6 +++--- 10 files changed, 25 insertions(+), 12 deletions(-) diff --git a/arch/x86/boot/startup/Makefile b/arch/x86/boot/startup/Makefile index b514f7e81332..c7e690091f81 100644 --- a/arch/x86/boot/startup/Makefile +++ b/arch/x86/boot/startup/Makefile @@ -28,3 +28,16 @@ lib-$(CONFIG_EFI_MIXED) += efi-mixed.o # to be linked into the decompressor or the EFI stub but not vmlinux # $(patsubst %.o,$(obj)/%.o,$(lib-y)): OBJECT_FILES_NON_STANDARD := y + +# +# Confine the startup code by prefixing all symbols with __pi_ (for position +# independent). This ensures that startup code can only call other startup +# code, or code that has explicitly been made accessible to it via a symbol +# alias. +# +$(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ +$(obj)/%.pi.o: $(obj)/%.o FORCE + $(call if_changed,objcopy) + +extra-y := $(obj-y) +obj-y := $(patsubst %.o,%.pi.o,$(obj-y)) diff --git a/arch/x86/boot/startup/sev-shared.c b/arch/x86/boot/startup/sev-shared.c index 9ab132bed77b..0215f2355dfa 100644 --- a/arch/x86/boot/startup/sev-shared.c +++ b/arch/x86/boot/startup/sev-shared.c @@ -12,7 +12,7 @@ #include #ifndef __BOOT_COMPRESSED -#define error(v) pr_err(v) +#define error(v) #define has_cpuflag(f) boot_cpu_has(f) #else #undef WARN diff --git a/arch/x86/boot/startup/sme.c b/arch/x86/boot/startup/sme.c index 1e9f1f5a753c..19dce2304636 100644 --- a/arch/x86/boot/startup/sme.c +++ b/arch/x86/boot/startup/sme.c @@ -560,7 +560,6 @@ void __head sme_enable(struct boot_params *bp) #ifdef CONFIG_MITIGATION_PAGE_TABLE_ISOLATION /* Local version for startup code, which never operates on user page tables */ -__weak pgd_t __pti_set_user_pgtbl(pgd_t *pgdp, pgd_t pgd) { return pgd; diff --git a/arch/x86/coco/sev/core.c b/arch/x86/coco/sev/core.c index 33332c4299b9..98921cde217e 100644 --- a/arch/x86/coco/sev/core.c +++ b/arch/x86/coco/sev/core.c @@ -276,7 +276,7 @@ static int svsm_perform_call_protocol(struct svsm_call *call) do { ret = ghcb ? svsm_perform_ghcb_protocol(ghcb, call) - : svsm_perform_msr_protocol(call); + : __pi_svsm_perform_msr_protocol(call); } while (ret == -EAGAIN); if (sev_cfg.ghcbs_initialized) diff --git a/arch/x86/coco/sev/vc-handle.c b/arch/x86/coco/sev/vc-handle.c index b4895c648024..0820accb9e80 100644 --- a/arch/x86/coco/sev/vc-handle.c +++ b/arch/x86/coco/sev/vc-handle.c @@ -1058,4 +1058,3 @@ bool __init handle_vc_boot_ghcb(struct pt_regs *regs) sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SEV_ES_GEN_REQ); } - diff --git a/arch/x86/include/asm/setup.h b/arch/x86/include/asm/setup.h index 6324f4c6c545..895d09faaf83 100644 --- a/arch/x86/include/asm/setup.h +++ b/arch/x86/include/asm/setup.h @@ -53,6 +53,7 @@ extern void i386_reserve_resources(void); extern unsigned long __startup_64(unsigned long p2v_offset, struct boot_params *bp); extern void startup_64_setup_gdt_idt(void); extern void startup_64_load_idt(void *vc_handler); +extern void __pi_startup_64_load_idt(void *vc_handler); extern void early_setup_idt(void); extern void __init do_early_exception(struct pt_regs *regs, int trapnr); diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h index ca914f7384c6..cfec803a9b6c 100644 --- a/arch/x86/include/asm/sev.h +++ b/arch/x86/include/asm/sev.h @@ -535,6 +535,7 @@ struct cpuid_leaf { }; int svsm_perform_msr_protocol(struct svsm_call *call); +int __pi_svsm_perform_msr_protocol(struct svsm_call *call); int snp_cpuid(void (*cpuid_hv)(void *ctx, struct cpuid_leaf *), void *ctx, struct cpuid_leaf *leaf); diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index b7da8b45b6d8..23dc505bf609 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -306,5 +306,5 @@ void early_setup_idt(void) handler = vc_boot_ghcb; } - startup_64_load_idt(handler); + __pi_startup_64_load_idt(handler); } diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 069420853304..88cbf932f7c2 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -71,7 +71,7 @@ SYM_CODE_START_NOALIGN(startup_64) xorl %edx, %edx wrmsr - call startup_64_setup_gdt_idt + call __pi_startup_64_setup_gdt_idt /* Now switch to __KERNEL_CS so IRET works reliably */ pushq $__KERNEL_CS @@ -91,7 +91,7 @@ SYM_CODE_START_NOALIGN(startup_64) * subsequent code. Pass the boot_params pointer as the first argument. */ movq %r15, %rdi - call sme_enable + call __pi_sme_enable #endif /* Sanitize CPU configuration */ @@ -111,7 +111,7 @@ SYM_CODE_START_NOALIGN(startup_64) * programmed into CR3. */ movq %r15, %rsi - call __startup_64 + call __pi___startup_64 /* Form the CR3 value being sure to include the CR3 modifier */ leaq early_top_pgt(%rip), %rcx @@ -562,7 +562,7 @@ SYM_CODE_START_NOALIGN(vc_no_ghcb) /* Call C handler */ movq %rsp, %rdi movq ORIG_RAX(%rsp), %rsi - call do_vc_no_ghcb + call __pi_do_vc_no_ghcb /* Unwind pt_regs */ POP_REGS diff --git a/arch/x86/mm/mem_encrypt_boot.S b/arch/x86/mm/mem_encrypt_boot.S index f8a33b25ae86..edbf9c998848 100644 --- a/arch/x86/mm/mem_encrypt_boot.S +++ b/arch/x86/mm/mem_encrypt_boot.S @@ -16,7 +16,7 @@ .text .code64 -SYM_FUNC_START(sme_encrypt_execute) +SYM_FUNC_START(__pi_sme_encrypt_execute) /* * Entry parameters: @@ -69,9 +69,9 @@ SYM_FUNC_START(sme_encrypt_execute) ANNOTATE_UNRET_SAFE ret int3 -SYM_FUNC_END(sme_encrypt_execute) +SYM_FUNC_END(__pi_sme_encrypt_execute) -SYM_FUNC_START(__enc_copy) +SYM_FUNC_START_LOCAL(__enc_copy) ANNOTATE_NOENDBR /* * Routine used to encrypt memory in place. From patchwork Sun May 4 09:52:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887897 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B1E071F2BAE for ; Sun, 4 May 2025 09:53:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352430; cv=none; b=bR3/B62Kqd0IC9xR4ogoRRDWHsQWuTX4v6XhX5eWB5W9qOL47dajfWyyauZohzeMr8RCAYHwL/3nNCajg4k0rmmCjs/AJS4ZViZ+6M88w/bhLxuQUtIY1MuUMSRfwzL8/p92q3VsY66cnta3L2ccMj7b4H3z3BJ/Jo7lXeQj304= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352430; c=relaxed/simple; bh=DWhIjtxAtA0gJ/IGitaMt1L8fiM9pvnFlSR46io8Hl0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=VeCieD94LKNQYZ8jbhbmJq+clszTj6bv+qK6KRGB/s/ieTRlNkD/o3mWGyefa654VIdBgO8b5CHrkN2JVwYBzy8O1rfpgwjVQeYGoefljh0XvySlHhP9Md3tZDGrAxqNVlgA90LrLMbHd3Y/beysbokKwyX+T06KHIlITG6Momg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=KmEjijVw; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="KmEjijVw" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-39134c762ebso961182f8f.0 for ; Sun, 04 May 2025 02:53:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352426; x=1746957226; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lxoQJBZ6aNwZCSOgRMHT0PvNdXWyaayDZ7A+o/XdNKg=; b=KmEjijVwclJuCc+qaTz+QRycc0qQeyCGvS4aMu4QqIMXplU9OhUlyGOY9AxFwQr6Jf /3jSCw69K/VmpZj4yWbY2yipripSkGikJH33jfJk90UDuNVvvbTTPJC1tVMpFtxQsf1m b2FxYm8VJ2x5ivRHQwet+hwD93gB/cSD2VMu22PlNjHAAyO3M/+VmKKSbRf/Cg9JLrlr LtSeU2g4vT6KoRVQyQGeN+VljI3QoA+ZeY65O95m/XSwRWzdejUxR2dvpxrEfzsQAkxM FIYFFU4RcGvLfHyBwP2BxmngUz/NNjnTkYlK639v3VqJOVy3u0PZO4JZeqv8VDVFtBQ4 XhwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352426; x=1746957226; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lxoQJBZ6aNwZCSOgRMHT0PvNdXWyaayDZ7A+o/XdNKg=; b=mvUtdUPu+/NGVX2kPScRcTZze8t8BmXLb3PkcohrmWkASRPfJtfPKDjM+lqHKvS6oh gyAfRu7bJmNgbr+32oXRPKXip3K9ieF+SdOv8sL6Engd22+fnQDwD3eXmUjfmRJglWrq gnpK020W2sh24Rh26ZEwlCQUnfYwfatwK/QIguX9dvaK51tbFD8MaiA7bvpwAlZacpVE Ekpx0Fpe/yGlwwLUVBH/K0aGNOvFbJvOlP02xS90iYC3GFA/TGjFYES+F4FWjM9X/Jkq rawGGpZ9uZre37BXGusCEglAMh0bgsaH9zwVb7e6TY6m6Fvg7YYCu8XXSBxdNaWDN4zv RsfA== X-Gm-Message-State: AOJu0YwnWxuVYDQc+2wRqbKwhf3hej3gKV5RZWa2YKOp6hbz3NQHkexO yGAK3/C+60diFRYd7XAzRuhW/m6hurMWePiIINOdKCsaZYogyhmcFe6x9aOsZ8SegauNkw== X-Google-Smtp-Source: AGHT+IGodw1fYWyjf9jevNKZkEapYHXEdHuU90CJwL0V2cAiFrF3MOLDe3NV5WyWPPBTwdhFAQfutT/u X-Received: from wrfu2.prod.google.com ([2002:a05:6000:382:b0:39a:be1a:5df9]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:18a5:b0:39c:dfa:c3de with SMTP id ffacd0b85a97d-3a09cf1e0fbmr4099547f8f.47.1746352426032; Sun, 04 May 2025 02:53:46 -0700 (PDT) Date: Sun, 4 May 2025 11:52:50 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=16149; i=ardb@kernel.org; h=from:subject; bh=PPOIAUxlSEO7gDjvZAFjN+tzvlJhEVfDWefDDPpbB2E=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4iezyuaLD7LYLhQYeNh/NvnS8G7CrRoJoYbwLI/H6 xtKreo6SlkYxDgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwERefWT4K7P6nrxA2LX257/P VbQ333rNqP+s/+wy/e7zJw7mf2Xkj2dk2PnIKNOlNbBJ9VyxRuT8OzUXN53QOD23l53dge1T+/l LnAA= X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-45-ardb+git@google.com> Subject: [RFT PATCH v2 20/23] x86/boot: Move startup code out of __head section From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel Move startup code out of the __head section, now that this no longer has a special significance. Move everything into .text or .init.text as appropriate, so that startup code is not kept around unnecessarily. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/sev.c | 3 --- arch/x86/boot/startup/gdt_idt.c | 4 +-- arch/x86/boot/startup/map_kernel.c | 4 +-- arch/x86/boot/startup/sev-shared.c | 28 ++++++++++---------- arch/x86/boot/startup/sev-startup.c | 14 +++++----- arch/x86/boot/startup/sme.c | 26 +++++++++--------- arch/x86/include/asm/init.h | 6 ----- arch/x86/kernel/head_32.S | 2 +- arch/x86/kernel/head_64.S | 2 +- arch/x86/platform/pvh/head.S | 2 +- 10 files changed, 41 insertions(+), 50 deletions(-) diff --git a/arch/x86/boot/compressed/sev.c b/arch/x86/boot/compressed/sev.c index 8118a270485f..065525e7fb5e 100644 --- a/arch/x86/boot/compressed/sev.c +++ b/arch/x86/boot/compressed/sev.c @@ -32,9 +32,6 @@ struct ghcb *boot_ghcb; #undef __init #define __init -#undef __head -#define __head - #define __BOOT_COMPRESSED u8 snp_vmpl; diff --git a/arch/x86/boot/startup/gdt_idt.c b/arch/x86/boot/startup/gdt_idt.c index a3112a69b06a..d16102abdaec 100644 --- a/arch/x86/boot/startup/gdt_idt.c +++ b/arch/x86/boot/startup/gdt_idt.c @@ -24,7 +24,7 @@ static gate_desc bringup_idt_table[NUM_EXCEPTION_VECTORS] __page_aligned_data; /* This may run while still in the direct mapping */ -void __head startup_64_load_idt(void *vc_handler) +void startup_64_load_idt(void *vc_handler) { struct desc_ptr desc = { .address = (unsigned long)rip_rel_ptr(bringup_idt_table), @@ -46,7 +46,7 @@ void __head startup_64_load_idt(void *vc_handler) /* * Setup boot CPU state needed before kernel switches to virtual addresses. */ -void __head startup_64_setup_gdt_idt(void) +void __init startup_64_setup_gdt_idt(void) { struct gdt_page *gp = rip_rel_ptr((void *)(__force unsigned long)&gdt_page); void *handler = NULL; diff --git a/arch/x86/boot/startup/map_kernel.c b/arch/x86/boot/startup/map_kernel.c index 11f99d907f76..e38f57b81745 100644 --- a/arch/x86/boot/startup/map_kernel.c +++ b/arch/x86/boot/startup/map_kernel.c @@ -26,7 +26,7 @@ static inline bool check_la57_support(void) return true; } -static unsigned long __head sme_postprocess_startup(struct boot_params *bp, +static unsigned long __init sme_postprocess_startup(struct boot_params *bp, pmdval_t *pmd, unsigned long p2v_offset) { @@ -80,7 +80,7 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp, * the 1:1 mapping of memory. Kernel virtual addresses can be determined by * subtracting p2v_offset from the RIP-relative address. */ -unsigned long __head __startup_64(unsigned long p2v_offset, +unsigned long __init __startup_64(unsigned long p2v_offset, struct boot_params *bp) { pmd_t (*early_pgts)[PTRS_PER_PMD] = rip_rel_ptr(early_dynamic_pgts); diff --git a/arch/x86/boot/startup/sev-shared.c b/arch/x86/boot/startup/sev-shared.c index 0215f2355dfa..5d700565b5f1 100644 --- a/arch/x86/boot/startup/sev-shared.c +++ b/arch/x86/boot/startup/sev-shared.c @@ -42,7 +42,7 @@ bool __init sev_es_check_cpu_features(void) return true; } -void __head __noreturn +void __noreturn sev_es_terminate(unsigned int set, unsigned int reason) { u64 val = GHCB_MSR_TERM_REQ; @@ -58,7 +58,7 @@ sev_es_terminate(unsigned int set, unsigned int reason) asm volatile("hlt\n" : : : "memory"); } -static u64 __head get_hv_features(void) +static u64 __init get_hv_features(void) { u64 val; @@ -72,7 +72,7 @@ static u64 __head get_hv_features(void) return GHCB_MSR_HV_FT_RESP_VAL(val); } -u64 __head snp_check_hv_features(void) +u64 __init snp_check_hv_features(void) { /* * SNP is supported in v2 of the GHCB spec which mandates support for HV @@ -220,7 +220,7 @@ const struct snp_cpuid_table *snp_cpuid_get_table(void) * * Return: XSAVE area size on success, 0 otherwise. */ -static u32 __head snp_cpuid_calc_xsave_size(u64 xfeatures_en, bool compacted) +static u32 snp_cpuid_calc_xsave_size(u64 xfeatures_en, bool compacted) { const struct snp_cpuid_table *cpuid_table = snp_cpuid_get_table(); u64 xfeatures_found = 0; @@ -256,7 +256,7 @@ static u32 __head snp_cpuid_calc_xsave_size(u64 xfeatures_en, bool compacted) return xsave_size; } -static bool __head +static bool snp_cpuid_get_validated_func(struct cpuid_leaf *leaf) { const struct snp_cpuid_table *cpuid_table = snp_cpuid_get_table(); @@ -298,7 +298,7 @@ static void snp_cpuid_hv_no_ghcb(void *ctx, struct cpuid_leaf *leaf) sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_CPUID_HV); } -static int __head +static int snp_cpuid_postprocess(void (*cpuid_hv)(void *ctx, struct cpuid_leaf *), void *ctx, struct cpuid_leaf *leaf) { @@ -394,7 +394,7 @@ snp_cpuid_postprocess(void (*cpuid_hv)(void *ctx, struct cpuid_leaf *), * Returns -EOPNOTSUPP if feature not enabled. Any other non-zero return value * should be treated as fatal by caller. */ -int __head +int snp_cpuid(void (*cpuid_hv)(void *ctx, struct cpuid_leaf *), void *ctx, struct cpuid_leaf *leaf) { @@ -438,7 +438,7 @@ snp_cpuid(void (*cpuid_hv)(void *ctx, struct cpuid_leaf *), * page yet, so it only supports the MSR based communication with the * hypervisor and only the CPUID exit-code. */ -void __head do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code) +void do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code) { unsigned int subfn = lower_bits(regs->cx, 32); unsigned int fn = lower_bits(regs->ax, 32); @@ -514,7 +514,7 @@ struct cc_setup_data { * Search for a Confidential Computing blob passed in as a setup_data entry * via the Linux Boot Protocol. */ -static __head +static __init struct cc_blob_sev_info *find_cc_blob_setup_data(struct boot_params *bp) { struct cc_setup_data *sd = NULL; @@ -542,7 +542,7 @@ struct cc_blob_sev_info *find_cc_blob_setup_data(struct boot_params *bp) * mapping needs to be updated in sync with all the changes to virtual memory * layout and related mapping facilities throughout the boot process. */ -static void __head setup_cpuid_table(const struct cc_blob_sev_info *cc_info) +static void __init setup_cpuid_table(const struct cc_blob_sev_info *cc_info) { const struct snp_cpuid_table *cpuid_table_fw, *cpuid_table; int i; @@ -570,7 +570,7 @@ static void __head setup_cpuid_table(const struct cc_blob_sev_info *cc_info) } } -static void __head svsm_pval_4k_page(unsigned long paddr, bool validate) +static void svsm_pval_4k_page(unsigned long paddr, bool validate) { struct svsm_pvalidate_call *pc; struct svsm_call call = {}; @@ -607,8 +607,8 @@ static void __head svsm_pval_4k_page(unsigned long paddr, bool validate) native_local_irq_restore(flags); } -static void __head pvalidate_4k_page(unsigned long vaddr, unsigned long paddr, - bool validate) +static void pvalidate_4k_page(unsigned long vaddr, unsigned long paddr, + bool validate) { int ret; @@ -625,7 +625,7 @@ static void __head pvalidate_4k_page(unsigned long vaddr, unsigned long paddr, * Maintain the GPA of the SVSM Calling Area (CA) in order to utilize the SVSM * services needed when not running in VMPL0. */ -static bool __head svsm_setup_ca(const struct cc_blob_sev_info *cc_info) +static bool __init svsm_setup_ca(const struct cc_blob_sev_info *cc_info) { struct snp_secrets_page *secrets_page; struct snp_cpuid_table *cpuid_table; diff --git a/arch/x86/boot/startup/sev-startup.c b/arch/x86/boot/startup/sev-startup.c index 6ba50723d546..7c1532ca1e87 100644 --- a/arch/x86/boot/startup/sev-startup.c +++ b/arch/x86/boot/startup/sev-startup.c @@ -44,7 +44,7 @@ /* Include code shared with pre-decompression boot stage */ #include "sev-shared.c" -void __head +void __init early_set_pages_state(unsigned long vaddr, unsigned long paddr, unsigned long npages, enum psc_op op) { @@ -90,7 +90,7 @@ early_set_pages_state(unsigned long vaddr, unsigned long paddr, sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_PSC); } -void __head early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, +void __init early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, unsigned long npages) { /* @@ -109,7 +109,7 @@ void __head early_snp_set_memory_private(unsigned long vaddr, unsigned long padd early_set_pages_state(vaddr, paddr, npages, SNP_PAGE_STATE_PRIVATE); } -void __head early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr, +void __init early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr, unsigned long npages) { /* @@ -138,7 +138,7 @@ void __head early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr * * Scan for the blob in that order. */ -static __head struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp) +static struct cc_blob_sev_info *__init find_cc_blob(struct boot_params *bp) { struct cc_blob_sev_info *cc_info; @@ -164,7 +164,7 @@ static __head struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp) return cc_info; } -static __head void svsm_setup(struct cc_blob_sev_info *cc_info) +static void __init svsm_setup(struct cc_blob_sev_info *cc_info) { struct snp_secrets_page *secrets = (void *)cc_info->secrets_phys; struct svsm_call call = {}; @@ -206,7 +206,7 @@ static __head void svsm_setup(struct cc_blob_sev_info *cc_info) boot_svsm_caa_pa = pa; } -bool __head snp_init(struct boot_params *bp) +bool __init snp_init(struct boot_params *bp) { struct cc_blob_sev_info *cc_info; @@ -235,7 +235,7 @@ bool __head snp_init(struct boot_params *bp) return true; } -void __head __noreturn snp_abort(void) +void __init __noreturn snp_abort(void) { sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SNP_UNSUPPORTED); } diff --git a/arch/x86/boot/startup/sme.c b/arch/x86/boot/startup/sme.c index 19dce2304636..e8f48d8bb59a 100644 --- a/arch/x86/boot/startup/sme.c +++ b/arch/x86/boot/startup/sme.c @@ -82,7 +82,7 @@ struct sme_populate_pgd_data { */ static char sme_workarea[2 * PMD_SIZE] __section(".init.scratch"); -static void __head sme_clear_pgd(struct sme_populate_pgd_data *ppd) +static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) { unsigned long pgd_start, pgd_end, pgd_size; pgd_t *pgd_p; @@ -97,7 +97,7 @@ static void __head sme_clear_pgd(struct sme_populate_pgd_data *ppd) memset(pgd_p, 0, pgd_size); } -static pud_t __head *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) +static pud_t __init *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) { pgd_t *pgd; p4d_t *p4d; @@ -134,7 +134,7 @@ static pud_t __head *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) return pud; } -static void __head sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) +static void __init sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) { pud_t *pud; pmd_t *pmd; @@ -150,7 +150,7 @@ static void __head sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) set_pmd(pmd, __pmd(ppd->paddr | ppd->pmd_flags)); } -static void __head sme_populate_pgd(struct sme_populate_pgd_data *ppd) +static void __init sme_populate_pgd(struct sme_populate_pgd_data *ppd) { pud_t *pud; pmd_t *pmd; @@ -176,7 +176,7 @@ static void __head sme_populate_pgd(struct sme_populate_pgd_data *ppd) set_pte(pte, __pte(ppd->paddr | ppd->pte_flags)); } -static void __head __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) +static void __init __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) { while (ppd->vaddr < ppd->vaddr_end) { sme_populate_pgd_large(ppd); @@ -186,7 +186,7 @@ static void __head __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) } } -static void __head __sme_map_range_pte(struct sme_populate_pgd_data *ppd) +static void __init __sme_map_range_pte(struct sme_populate_pgd_data *ppd) { while (ppd->vaddr < ppd->vaddr_end) { sme_populate_pgd(ppd); @@ -196,7 +196,7 @@ static void __head __sme_map_range_pte(struct sme_populate_pgd_data *ppd) } } -static void __head __sme_map_range(struct sme_populate_pgd_data *ppd, +static void __init __sme_map_range(struct sme_populate_pgd_data *ppd, pmdval_t pmd_flags, pteval_t pte_flags) { unsigned long vaddr_end; @@ -220,22 +220,22 @@ static void __head __sme_map_range(struct sme_populate_pgd_data *ppd, __sme_map_range_pte(ppd); } -static void __head sme_map_range_encrypted(struct sme_populate_pgd_data *ppd) +static void __init sme_map_range_encrypted(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_ENC, PTE_FLAGS_ENC); } -static void __head sme_map_range_decrypted(struct sme_populate_pgd_data *ppd) +static void __init sme_map_range_decrypted(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_DEC, PTE_FLAGS_DEC); } -static void __head sme_map_range_decrypted_wp(struct sme_populate_pgd_data *ppd) +static void __init sme_map_range_decrypted_wp(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_DEC_WP, PTE_FLAGS_DEC_WP); } -static unsigned long __head sme_pgtable_calc(unsigned long len) +static unsigned long __init sme_pgtable_calc(unsigned long len) { unsigned long entries = 0, tables = 0; @@ -272,7 +272,7 @@ static unsigned long __head sme_pgtable_calc(unsigned long len) return entries + tables; } -void __head sme_encrypt_kernel(struct boot_params *bp) +void __init sme_encrypt_kernel(struct boot_params *bp) { unsigned long workarea_start, workarea_end, workarea_len; unsigned long execute_start, execute_end, execute_len; @@ -476,7 +476,7 @@ void __head sme_encrypt_kernel(struct boot_params *bp) native_write_cr3(__native_read_cr3()); } -void __head sme_enable(struct boot_params *bp) +void __init sme_enable(struct boot_params *bp) { unsigned int eax, ebx, ecx, edx; unsigned long feature_mask; diff --git a/arch/x86/include/asm/init.h b/arch/x86/include/asm/init.h index 8b1b1abcef15..01ccdd168df0 100644 --- a/arch/x86/include/asm/init.h +++ b/arch/x86/include/asm/init.h @@ -2,12 +2,6 @@ #ifndef _ASM_X86_INIT_H #define _ASM_X86_INIT_H -#if defined(CONFIG_CC_IS_CLANG) && CONFIG_CLANG_VERSION < 170000 -#define __head __section(".head.text") __no_sanitize_undefined __no_stack_protector -#else -#define __head __section(".head.text") __no_sanitize_undefined -#endif - struct x86_mapping_info { void *(*alloc_pgt_page)(void *); /* allocate buf for page table */ void (*free_pgt_page)(void *, void *); /* free buf for page table */ diff --git a/arch/x86/kernel/head_32.S b/arch/x86/kernel/head_32.S index 2e42056d2306..5962ff2a189a 100644 --- a/arch/x86/kernel/head_32.S +++ b/arch/x86/kernel/head_32.S @@ -61,7 +61,7 @@ RESERVE_BRK(pagetables, INIT_MAP_SIZE) * any particular GDT layout, because we load our own as soon as we * can. */ -__HEAD + __INIT SYM_CODE_START(startup_32) movl pa(initial_stack),%ecx diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 88cbf932f7c2..ac1116f623d3 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -33,7 +33,7 @@ * because we need identity-mapped pages. */ - __HEAD + __INIT .code64 SYM_CODE_START_NOALIGN(startup_64) UNWIND_HINT_END_OF_STACK diff --git a/arch/x86/platform/pvh/head.S b/arch/x86/platform/pvh/head.S index cfa18ec7d55f..16aa1f018b80 100644 --- a/arch/x86/platform/pvh/head.S +++ b/arch/x86/platform/pvh/head.S @@ -24,7 +24,7 @@ #include #include - __HEAD + __INIT /* * Entry point for PVH guests. From patchwork Sun May 4 09:52:51 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887274 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B07711EF388 for ; Sun, 4 May 2025 09:53:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352431; cv=none; b=JWD4hLMfVnn9aj0kfglWg4cU6NjsLzHEsiISY3FbBUo4v9ROCabqq6YCjw8sn8VGI7/T4FOanbiGRZEVqWizrsQ/jwtcu9iuGr0JgjXPMQQpnBeQlL/+6G1ExhQGsYtlDf74ZizNpKH1a8/Sev6ppCpbFf/7I5vFdKW5xdmaQCk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352431; c=relaxed/simple; bh=kR6rswlwwIYujb+EJ0yOVEFScg9Jru2OEjE3Kvh/Q84=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=sPGw6VWw4m8aCHb3fTCCsTkvrwbQDlI7rUEeHX0uaqKqo+U6e62lGia5Wt3u8sE0IYBMElt3nBWBFTjTHoSh4Vetkg0eG3P386nu6tvXIEeR5+ShZbQxav8FY66oZFXGaJiWM2VuI5yYL+QAuiWblU0j/APKGeX9hBOi+L3bJtQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=fkR05YwZ; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="fkR05YwZ" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-43cf446681cso19721405e9.1 for ; Sun, 04 May 2025 02:53:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352428; x=1746957228; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hLjo2Urmr03naf19fIolYJZFVTpPiE8779NTKQlY+0c=; b=fkR05YwZ/J2BzRZj4WKzzWGD/iwRFofk1cIXjY+lNaRYqL6UEpObAs/ws85ReQZVA9 RKBn2hpRofCOg5kWFQTedJ71w6FtDKvQbI2nSq2gGBWzft91PH/A9NxzQlyjOB5AThsw By1ZuefjgZcT/uarMcj3k6uRbT1KOUb/KPX43JuuL8wYOsMRHF2TFMYsSofooLOLOImf TX1ycZm9je7gzxIOg1jzQW9cmm/DhA1DvI6LaA+HfzH0R6RW9FtdcGzkuq4T4l9FtD+x d5SgNpe2fIvR3Zrp53kGVH0pH3SsGsWGvCVeRbDdFKzQCaZiFv8S+GvZbKXsIRFgLLjD uJhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352428; x=1746957228; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hLjo2Urmr03naf19fIolYJZFVTpPiE8779NTKQlY+0c=; b=b942vtbdPiLYdg4ItarQkoGNcQPQ/rsUyK/YKs9f9B2SvqO1waPQSSU2K1fgijjLPQ zY6vcjZhLigTYMkettLO3yU1W9xrej9FVyGJwfG1ovVP3GGSaZI4WPHIhWQh0LrZ95Ls KAmFOzrAiGIi5geqVYY9bH4m/6rwvFE7p8llxmbzs31RnwRyfTIou9HscE1WjeS+3chj cpFDHi+DK7KM17IqCI4ka2Hc0BBAOnBGR5f6/lZ9XCHbELAnlAPwobNBrXh7i8FkqT7y g6xsMwLira8Y2GC4mg8QPDqZeHvxdz8lXiAYVV41hc0jJiwUVOVoRm4b4BmeT/469i1k 65AQ== X-Gm-Message-State: AOJu0Yzj275xYJzQfq2o40/a8ehoeCTFmOJQWrE+NTEuBiW87TcliE4i SLp60ZjqhzJ2KYcBH80iQlkb6MezyiTlsFvdJOa7GH1UE5y9jm09XzQCzdrZmLjwbY5FOw== X-Google-Smtp-Source: AGHT+IH90gqSPJxs8k9nVTdFHRkCC/+sHj+b3LeroAd+gdqmLY7ywJfO2ETmybdwDECaqma2vA+qbFP+ X-Received: from wmoo8-n2.prod.google.com ([2002:a05:600d:108:20b0:441:c147:991e]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3e05:b0:43e:bdf7:7975 with SMTP id 5b1f17b1804b1-441c49486cbmr22764355e9.32.1746352428071; Sun, 04 May 2025 02:53:48 -0700 (PDT) Date: Sun, 4 May 2025 11:52:51 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1669; i=ardb@kernel.org; h=from:subject; bh=4KgoJzYmdouJK1RzBO038uMdZjYS6ocYk6yPrPHnQMw=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4tfjUuFv7w+wzBWI0676+dkxPddmnZ5R7o+0AyKP/ da/2zq5o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzk9Q2G/xUdPIviW/av19Cz urCmLastaxvf2kv31AOfM1UlpFmpmzMy7HknzW7xI2HzRfk7izkz7Tdw7rX+cv/6MVbJExlHk++ EMgAA X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-46-ardb+git@google.com> Subject: [RFT PATCH v2 21/23] x86/boot: Disallow absolute symbol references in startup code From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel Check that the objects built under arch/x86/boot/startup do not contain any absolute symbol reference. Given that the code is built with -fPIC, such references can only be emitted using R_X86_64_64 relocations, so checking that those are absent is sufficient. Note that debug sections and __patchable_funtion_entries section may contain such relocations nonetheless, but these are unnecessary in the startup code, so they can be dropped first. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/startup/Makefile | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/arch/x86/boot/startup/Makefile b/arch/x86/boot/startup/Makefile index c7e690091f81..7b83c8f597ef 100644 --- a/arch/x86/boot/startup/Makefile +++ b/arch/x86/boot/startup/Makefile @@ -35,9 +35,17 @@ $(patsubst %.o,$(obj)/%.o,$(lib-y)): OBJECT_FILES_NON_STANDARD := y # code, or code that has explicitly been made accessible to it via a symbol # alias. # -$(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ +$(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ --strip-debug \ + --remove-section=.rela__patchable_function_entries $(obj)/%.pi.o: $(obj)/%.o FORCE - $(call if_changed,objcopy) + $(call if_changed,piobjcopy) + +quiet_cmd_piobjcopy = $(quiet_cmd_objcopy) + cmd_piobjcopy = $(cmd_objcopy); \ + if $(READELF) -r $(@) | grep R_X86_64_64; then \ + echo "$@: R_X86_64_64 references not allowed in startup code" >&2; \ + /bin/false; \ + fi extra-y := $(obj-y) obj-y := $(patsubst %.o,%.pi.o,$(obj-y)) From patchwork Sun May 4 09:52:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887896 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A46421F3BAA for ; Sun, 4 May 2025 09:53:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352433; cv=none; b=MX272QIBo3k3gv3ULE6zCYgL7Pl1n7BmiSyGPFDff0hU8Wa3Jj9ID0wfX7REasibOpR4u3teNQpXNcKI1aPQq8ttGsspW8BV+YLcD/DMkFH3W+ySZVWwWvZwLwz6SlIe/ISqZ6nqcJGaCxVZ+bGd8Q+XW5iwnhgtpP+KBbgHAGU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352433; c=relaxed/simple; bh=wuHAfmatSUs0eQGLjdRIUylP9w5pVwSp6jih2u1d2Nc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=boyN8sKZcksAdS3vYbAssoWg1OGaobXNCyvoPArX5NibHjH2j8pRztJfcwyJhyjzYtuT0sR6DvX5IME1tBd+1lJejTRaAXhRPa8SyrUAhnHbYmoOmIWg2ekmP/EOrRX3vKy8hfHeQ+i0fZYA5KEslda/GtseJTk+oboKF00jRNs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=q4Pt2ZI0; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="q4Pt2ZI0" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-43cf3168b87so14046685e9.2 for ; Sun, 04 May 2025 02:53:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352430; x=1746957230; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BngAQLfFrYEWXhjcTmuThO4xgzttYUC9luJvEjHMWdQ=; b=q4Pt2ZI0VWKKVj3cSxoegX4K7yR1vgrwHtcGEnIEOZ3W24aa/rrGjZXyDBYJjeQZnz XjHDGoT7n+yR398RT5ib/aKuEWglOcqE2U0eYqOi7pjLbOERGrPD1b8nVt5/pb6T4e42 fGrttmcy9OVDVXJmkGFLeVdnq50Fr78IHUyM7REfEyzN7MNrfzCQLCbXZUv6HJvXpQhP XDqKtoelu4GqqryQqwPq9ES70zfIz6Vkuxo58BbNGv/qa+rdeP6psNrOw5jAhgl7Lt4J CrGXsy59vO+AVxoqpi2ZzJ7dKcNNPnXh3htn9KCfSmAGr2YpXWSkypilVfmIyKwrZb+7 ZDog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352430; x=1746957230; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BngAQLfFrYEWXhjcTmuThO4xgzttYUC9luJvEjHMWdQ=; b=CEzX+jjrhaIWvxxWfkYOvqPqlBp0yppKiFHW9rUjydUY2Df+5NuKP8/hv5PVDQmOLE vYKZZaiZWKdiprk7MCqV3l6pBzN5oMX8wpg378z7lv00gs0VSzf/ZJfV/OIgW/rkHoG/ xBXspaE5BdreIE/AmFMoo0CWT4XEzW0QmjdVWkYpSJyTJENky5dMz9rVO4LSXbsF4ZUp eH4/QoRVy2whIDVo+xDRKVeEzbPnU3iqMBsUVRsoms8WlHZNThEAYpt46xQzvJcYCIEO OegpWa7OZLzktJkzz8P0UUe6CoFoCL6gGc+RyFwwGVsB146MzywI3998+wpucEwsF+ng y83w== X-Gm-Message-State: AOJu0YwR6Cq4IKTz/jbeipet9gb3unjw+VVs3ZqUcvVsrl1yaKtXmvWV V5h3CrrqVDxjIqTNUj7EyFlcNuEcuUTaMVoFreHKSdRg1NE+G85RufAXUGSQ+DMBWVoNew== X-Google-Smtp-Source: AGHT+IHMOiV5Mdn5v77/gifWvnDjJAZiAxp/513QM5JcQVs+myLHjDeNQH95LNNEsbP9E0pI95c69lwy X-Received: from wmbbi15.prod.google.com ([2002:a05:600c:3d8f:b0:43b:bf16:d6be]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:5246:b0:43c:ec4c:25b4 with SMTP id 5b1f17b1804b1-441bbec23e8mr88386825e9.10.1746352430181; Sun, 04 May 2025 02:53:50 -0700 (PDT) Date: Sun, 4 May 2025 11:52:52 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1438; i=ardb@kernel.org; h=from:subject; bh=SfyGJVp/cmZ3yoM/3fyJKQWjWHpwNBvXWOMO7V8FZ4I=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4vfX/MYf5f1zwwzKZ2wNCdpqLG86K/P1QdbQNdfVj j6zefy3o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEyk3JXhv3/GviO9DDaO89Yc za93yHo0S/d7aVCOr1rJasXgqWKrpzIyPJh+5g3Tl11p3FZGRQkctopygW+aDr0Uf3ok7kSB66E qNgA= X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-47-ardb+git@google.com> Subject: [RFT PATCH v2 22/23] x86/boot: Revert "Reject absolute references in .head.text" From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel This reverts commit faf0ed487415f76fe4acf7980ce360901f5e1698. The startup code is checked directly for the absence of absolute symbol references, so checking the .head.text section in the relocs tool is no longer needed. Signed-off-by: Ard Biesheuvel --- arch/x86/tools/relocs.c | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/arch/x86/tools/relocs.c b/arch/x86/tools/relocs.c index 5778bc498415..e5a2b9a912d1 100644 --- a/arch/x86/tools/relocs.c +++ b/arch/x86/tools/relocs.c @@ -740,10 +740,10 @@ static void walk_relocs(int (*process)(struct section *sec, Elf_Rel *rel, static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym, const char *symname) { - int headtext = !strcmp(sec_name(sec->shdr.sh_info), ".head.text"); unsigned r_type = ELF64_R_TYPE(rel->r_info); ElfW(Addr) offset = rel->r_offset; int shn_abs = (sym->st_shndx == SHN_ABS) && !is_reloc(S_REL, symname); + if (sym->st_shndx == SHN_UNDEF) return 0; @@ -783,12 +783,6 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym, break; } - if (headtext) { - die("Absolute reference to symbol '%s' not permitted in .head.text\n", - symname); - break; - } - /* * Relocation offsets for 64 bit kernels are output * as 32 bits and sign extended back to 64 bits when From patchwork Sun May 4 09:52:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 887273 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5891B1F3B8D for ; Sun, 4 May 2025 09:53:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352435; cv=none; b=r8aqbvvKBWjQgWn7KkEm/H0AAZwZintSbzX1WbkInbc6kYh1Z6oy5I1CVuKAhuHOOSx2LB0Uw1ydmcFamF5HlLY+CSAHiLcRF+16z4MRSstAmAd9zoSVLaw1LMDVZSKlQoziOQcnfLe/xYuTn6XSx65gqveSFzCiAtKK6SKdAGM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352435; c=relaxed/simple; bh=C6gzvknBtFypmZWra6mOhD5B9I5YTmOiy1ngK8W8VIA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=e1IbWD+kpT2aEmvEBK43AIkgYsPmkbVmKc3BMvgyk6GHsU7rBZPM9aoBCA6RkVg3UIqC9kSCM8yGcuHD8wKIUpq6ceiwcu80/Pc3s8RzLnQlPDaxl6ideHOtSy7RzY2YH/kgzJN+4BiTou3RBB6oe44Zq5Zo/xto/O6ygDUk7YM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=pnHGt8mq; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="pnHGt8mq" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-43f251dc364so19247655e9.2 for ; Sun, 04 May 2025 02:53:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746352432; x=1746957232; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lC4kTjWNtYVJtNr/R34O8MumjOtUSCO/boDQXdKZavw=; b=pnHGt8mqiuF+GMqLCoQhhPgIaOlJo3iwzdQ3DI+jOcmt9vFeK+wL4nmgq4B28DqPqL dfzu3epn+rmr++Xzp1iYGnZWO2A1KN4bvWB5GeuGE0VsQJZD+jN0hPZWJHhzSg922k+Q nhKgVwiLBBnEv+WBCyvCDD/1eTJJetUyHOI7UjW7XhxiLu3MIA5T4TeOIZsa9CC+MeIZ 462zrWDY06o8XspYTwbA/iiqjbvZ94faR2L++BsFHDDlPCO6VoFdm5V/l+1ReNUG9MnL k9y39SItd74ylX2A8aqF+hiDr8Pa61m+7vxe/CZgb52TwuB7f2qi+kXa+jO78WbV0B5f XS/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352432; x=1746957232; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lC4kTjWNtYVJtNr/R34O8MumjOtUSCO/boDQXdKZavw=; b=F3PNg3t335Zqe106no+5QdAcm7kmnZg+1EVVfM4dI+gswYvt4TkvPj/Ww/KDt1d48q Amg3sr/VdBHwiiE1RII1w0bhgvqfleLaeI+s51pEIp1GE21ZIdwN2jgkm2FUtFSZ2oVB LxanC9yyWIcOALot721koQvm6OXDdvhQH11UhKU3mNx7jlVc2rn6L96fG3LbtbIderrg r2V5QoprxXQiOUyadM87JqvwlqEPVOglLAm6PB9OouNRBFEyuk0D7lLIR2XT3MJ8r/Ev qWWosxFaUOMGQYokIrlQBEo8g+Ii1ur9hQHjplEB2zVNW1VDqDUADEi7K6OQwKEnvlpo BMPw== X-Gm-Message-State: AOJu0YytrtbG2hIK96Pi9CMiNnzournUaCyT56Fna4ZOkXWICj1Q69oH zLsmjKe5MlL+UR9mUl5jodzODEBNOgzJEmPqnddGIbHpJF3yAa9lPHu0+sbejxG4OvWrPQ== X-Google-Smtp-Source: AGHT+IEmLbsKySGDhi+UwQYPdC4xgeVngLAICWAnBPmQkT2h+F73uzVvKde7aylFSfpCInOSZFE5iYR3 X-Received: from wmbjh5.prod.google.com ([2002:a05:600c:a085:b0:43b:c450:ea70]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:8711:b0:43d:47b7:b32d with SMTP id 5b1f17b1804b1-441c4919fb3mr23588375e9.25.1746352432075; Sun, 04 May 2025 02:53:52 -0700 (PDT) Date: Sun, 4 May 2025 11:52:53 +0200 In-Reply-To: <20250504095230.2932860-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250504095230.2932860-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=737; i=ardb@kernel.org; h=from:subject; bh=zbymTFtFVf98uj8Itu8GL+XtbjDBIM1MUB1YpDi01GQ=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIUPc4o9J9pr5v31sQhY7Xrs6r2nfuyWPri+dwcCjtsW5b kmdOndMRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjItumMDP23rh9evG3K30dz u2Qy193e2NR6a3/IR5mGV76N//dd8W9gZGg4XCoufDDz/3Wmlbzbni2s0n4rcPeOgw/3n5hfdV+ WbGQHAA== X-Mailer: git-send-email 2.49.0.906.g1f30a19c02-goog Message-ID: <20250504095230.2932860-48-ardb+git@google.com> Subject: [RFT PATCH v2 23/23] x86/boot: Get rid of the .head.text section From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-efi@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , Borislav Petkov , Ingo Molnar , Dionna Amalie Glaze , Kevin Loughlin , Tom Lendacky From: Ard Biesheuvel The .head.text section is now empty, so it can be dropped from the linker script. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/vmlinux.lds.S | 5 ----- 1 file changed, 5 deletions(-) diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 4aaa1693b262..af8bc4a6f6c3 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -160,11 +160,6 @@ SECTIONS } :text = 0xcccccccc - /* bootstrapping code */ - .head.text : AT(ADDR(.head.text) - LOAD_OFFSET) { - HEAD_TEXT - } :text = 0xcccccccc - /* End of text section, which should occupy whole number of pages */ _etext = .; . = ALIGN(PAGE_SIZE);