From patchwork Mon Oct 14 10:59:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 835582 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2EDE31DD860; Mon, 14 Oct 2024 11:02:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728903738; cv=none; b=XkAKuOneSGzFlkapxH1SG2TwCEvCvA02vhqYwKzO8ISkn5IctbAVmfJ24/O5PRLvt2XBvcIVAIdvFr6bpkx5PSz7jt0LSTDbyM9mvzXeBBBnKf1aL1guFE4vHrGurKwFfeKlQwEDqTES8izSjgr6k7/cTQnRTF+vvssxfG5XXAo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728903738; c=relaxed/simple; bh=wXcWDfoIbzEAXadEzvyVMkkfOTcBg0Jj+Po0Slo/R4g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DYRUugAfWfEP9XLBDF9Hg5diLQyo6Nk1NklV6tSTgREqSVBDCLiCCOw1YM6iIJabyMey7ZkcO1tfhdVUjax6f5gkPMBu7B7Ex9d7ZhvsKthn7nHBFICRyPLvr/w+WnkEDk0jvKq0lALs0iBUcO2E+ID4sW4q8ryVVqimM/AuzAY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6C0D71756; Mon, 14 Oct 2024 04:02:46 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0FB423F51B; Mon, 14 Oct 2024 04:02:13 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Oliver Upton , Will Deacon Cc: Ryan Roberts , kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 56/57] arm64: Determine THREAD_SIZE at boot-time Date: Mon, 14 Oct 2024 11:59:03 +0100 Message-ID: <20241014105912.3207374-56-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Since THREAD_SIZE depends on PAGE_SIZE when stacks are vmapped, we must defer the decision on THREAD_SIZE until we have selected PAGE_SIZE at boot. The one wrinkle is entry.S's requirement to have THREAD_SHIFT as an immediate in order to check that the stack has not overflowed without clobbering any registers, early in the exception handler. Solve this by patching alternatives. During early boot, all 3 options are NOPs until the alternative is patched in. So we forgo overflow checking until boot-cpu patching is complete. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/include/asm/assembler.h | 5 +++ arch/arm64/include/asm/efi.h | 2 +- arch/arm64/include/asm/memory.h | 51 +++++++++++++++++++++++++----- arch/arm64/kernel/efi.c | 2 +- arch/arm64/kernel/entry.S | 10 +++++- arch/arm64/kernel/head.S | 3 +- arch/arm64/kernel/vmlinux.lds.S | 4 +-- arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 2 +- 8 files changed, 64 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 0cfa7c3efd214..745328e7768b7 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -928,4 +928,9 @@ alternative_cb_end get_tg0 \val value_for_page_size \val, \val, (~(SZ_4K-1)), (~(SZ_16K-1)), (~(SZ_64K-1)) .endm + + .macro get_task_size, val + get_tg0 \val + value_for_page_size \val, \val, (1 << THREAD_SHIFT_4K), (1 << THREAD_SHIFT_16K), (1 << THREAD_SHIFT_64K) + .endm #endif /* __ASM_ASSEMBLER_H */ diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h index bcd5622aa0968..913f599c14e40 100644 --- a/arch/arm64/include/asm/efi.h +++ b/arch/arm64/include/asm/efi.h @@ -68,7 +68,7 @@ void arch_efi_call_virt_teardown(void); * kernel need greater alignment than we require the segments to be padded to. */ #define EFI_KIMG_ALIGN \ - (SEGMENT_ALIGN > THREAD_ALIGN ? SEGMENT_ALIGN : THREAD_ALIGN) + (SEGMENT_ALIGN > THREAD_ALIGN_MAX ? SEGMENT_ALIGN : THREAD_ALIGN_MAX) /* * On arm64, we have to ensure that the initrd ends up in the linear region, diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 5393a859183f7..e28f5700ef022 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -110,23 +110,56 @@ #define PAGE_END (_PAGE_END(VA_BITS_MIN)) #endif /* CONFIG_KASAN */ -#define MIN_THREAD_SHIFT (14 + KASAN_THREAD_SHIFT) +#define IDEAL_THREAD_SHIFT (14 + KASAN_THREAD_SHIFT) /* * VMAP'd stacks are allocated at page granularity, so we must ensure that such * stacks are a multiple of page size. */ -#if defined(CONFIG_VMAP_STACK) && (MIN_THREAD_SHIFT < PAGE_SHIFT) -#define THREAD_SHIFT PAGE_SHIFT + +#if defined(CONFIG_VMAP_STACK) +#define THREAD_SHIFT \ + (IDEAL_THREAD_SHIFT < PAGE_SHIFT ? PAGE_SHIFT : IDEAL_THREAD_SHIFT) +#if (IDEAL_THREAD_SHIFT < PAGE_SHIFT_MIN) +#define THREAD_SHIFT_MIN PAGE_SHIFT_MIN #else -#define THREAD_SHIFT MIN_THREAD_SHIFT +#define THREAD_SHIFT_MIN IDEAL_THREAD_SHIFT #endif - -#if THREAD_SHIFT >= PAGE_SHIFT -#define THREAD_SIZE_ORDER (THREAD_SHIFT - PAGE_SHIFT) +#if (IDEAL_THREAD_SHIFT < PAGE_SHIFT_MAX) +#define THREAD_SHIFT_MAX PAGE_SHIFT_MAX +#else +#define THREAD_SHIFT_MAX IDEAL_THREAD_SHIFT +#endif +#if (IDEAL_THREAD_SHIFT < ARM64_PAGE_SHIFT_4K) +#define THREAD_SHIFT_4K ARM64_PAGE_SHIFT_4K +#else +#define THREAD_SHIFT_4K IDEAL_THREAD_SHIFT +#endif +#if (IDEAL_THREAD_SHIFT < ARM64_PAGE_SHIFT_16K) +#define THREAD_SHIFT_16K ARM64_PAGE_SHIFT_16K +#else +#define THREAD_SHIFT_16K IDEAL_THREAD_SHIFT +#endif +#if (IDEAL_THREAD_SHIFT < ARM64_PAGE_SHIFT_64K) +#define THREAD_SHIFT_64K ARM64_PAGE_SHIFT_64K +#else +#define THREAD_SHIFT_64K IDEAL_THREAD_SHIFT #endif +#else +#define THREAD_SHIFT IDEAL_THREAD_SHIFT +#define THREAD_SHIFT_MIN IDEAL_THREAD_SHIFT +#define THREAD_SHIFT_MAX IDEAL_THREAD_SHIFT +#define THREAD_SHIFT_4K IDEAL_THREAD_SHIFT +#define THREAD_SHIFT_16K IDEAL_THREAD_SHIFT +#define THREAD_SHIFT_64K IDEAL_THREAD_SHIFT +#endif + +#define THREAD_SIZE_ORDER \ + (PAGE_SHIFT < THREAD_SHIFT ? THREAD_SHIFT - PAGE_SHIFT : 0) #define THREAD_SIZE (UL(1) << THREAD_SHIFT) +#define THREAD_SIZE_MIN (UL(1) << THREAD_SHIFT_MIN) +#define THREAD_SIZE_MAX (UL(1) << THREAD_SHIFT_MAX) /* * By aligning VMAP'd stacks to 2 * THREAD_SIZE, we can detect overflow by @@ -135,11 +168,13 @@ */ #ifdef CONFIG_VMAP_STACK #define THREAD_ALIGN (2 * THREAD_SIZE) +#define THREAD_ALIGN_MAX (2 * THREAD_SIZE_MAX) #else #define THREAD_ALIGN THREAD_SIZE +#define THREAD_ALIGN_MAX THREAD_SIZE_MAX #endif -#define IRQ_STACK_SIZE THREAD_SIZE +#define IRQ_STACK_SIZE THREAD_SIZE_MIN #define OVERFLOW_STACK_SIZE SZ_4K diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index 712718aed5dd9..ebc44b7e83199 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -197,7 +197,7 @@ bool efi_runtime_fixup_exception(struct pt_regs *regs, const char *msg) } /* EFI requires 8 KiB of stack space for runtime services */ -static_assert(THREAD_SIZE >= SZ_8K); +static_assert(THREAD_SIZE_MIN >= SZ_8K); static int __init arm64_efi_rt_init(void) { diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index ba47dc8672c04..1ab65e406b62e 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -62,7 +62,15 @@ */ add sp, sp, x0 // sp' = sp + x0 sub x0, sp, x0 // x0' = sp' - x0 = (sp + x0) - x0 = sp - tbnz x0, #THREAD_SHIFT, 0f +alternative_if ARM64_USE_PAGE_SIZE_4K + tbnz x0, #THREAD_SHIFT_4K, 0f +alternative_else_nop_endif +alternative_if ARM64_USE_PAGE_SIZE_16K + tbnz x0, #THREAD_SHIFT_16K, 0f +alternative_else_nop_endif +alternative_if ARM64_USE_PAGE_SIZE_64K + tbnz x0, #THREAD_SHIFT_64K, 0f +alternative_else_nop_endif sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x0 sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp b el\el\ht\()_\regsize\()_\label diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 761b7f5633e15..2530ee5cee548 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -198,7 +198,8 @@ SYM_CODE_END(preserve_boot_args) msr sp_el0, \tsk ldr \tmp1, [\tsk, #TSK_STACK] - add sp, \tmp1, #THREAD_SIZE + get_task_size \tmp2 + add sp, \tmp1, \tmp2 sub sp, sp, #PT_REGS_SIZE stp xzr, xzr, [sp, #S_STACKFRAME] diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 09fcc234c0f77..937900a458a89 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -60,11 +60,11 @@ #define RO_EXCEPTION_TABLE_ALIGN 4 #define RUNTIME_DISCARD_EXIT +#include #include #include #include #include -#include #include #include "image.h" @@ -292,7 +292,7 @@ SECTIONS _data = .; _sdata = .; - RW_DATA(L1_CACHE_BYTES, PAGE_SIZE_MAX, THREAD_ALIGN) + RW_DATA(L1_CACHE_BYTES, PAGE_SIZE_MAX, THREAD_ALIGN_MAX) /* * Data written with the MMU off but read with the MMU on requires diff --git a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S index 74c7c21626270..fe1fbfa8f8f05 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S @@ -7,9 +7,9 @@ */ #include +#include #include #include -#include SECTIONS { HYP_SECTION(.idmap.text) From patchwork Mon Oct 14 10:59:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 835258 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 3E46D1DD9D6; Mon, 14 Oct 2024 11:02:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728903742; cv=none; b=SrLAKnCoYZ5zfQK3FnpEAt7uL/s7R3N0ZHwIfofot6a1PDxSjsNa4qxjQOe1/nVptZupFbBozJk7Munw1P4ypia4jybLGx63qKq2LXcZLRAmSX/OFLuqAFL/pqDoRQe+sDtfiYjP64KhSi7WzDt8WEgt8Ti7465Xh36WvgoG0UQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728903742; c=relaxed/simple; bh=j6rv1RFgs7FwfIXsLLHTHGBzQWTRdMzhDAi4Jd+RDZo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gT6ScA+/aOhdNBnd9EaTSzjy6lPrLfqH99zGkF8ybuQUdgtRMUBGZK0w8JxYRU9W0te/CX0hLgfSTtMVmLT2uAnL76A1MW+aaZXDxOYQsMtvGtSGNsYR9Jt+Zg6izBCJP19exQOE5cuYCpIs++ZXk+A4nczNGlebUDAhWabPyT0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7ACB21758; Mon, 14 Oct 2024 04:02:49 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 1E2F93F51B; Mon, 14 Oct 2024 04:02:16 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Oliver Upton , Will Deacon Cc: Ryan Roberts , kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 57/57] arm64: Enable boot-time page size selection Date: Mon, 14 Oct 2024 11:59:04 +0100 Message-ID: <20241014105912.3207374-57-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Introduce a new Kconfig, ARM64_BOOT_TIME_PAGE_SIZE, which can be selected instead of a page size. When selected, the resulting kernel's page size can be configured at boot via the command line. For now, boot-time page size kernels are limited to 48-bit VA, since more work is required to support LPA2. Additionally MMAP_RND_BITS and SECTION_SIZE_BITS are configured for the worst case (64K pages). Future work could be implemented to be able to configure these at boot time for optimial page size-specific values. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/Kconfig | 26 ++++++++++--- arch/arm64/include/asm/kvm_hyp.h | 11 ++++++ arch/arm64/include/asm/pgtable-geometry.h | 22 ++++++++++- arch/arm64/include/asm/pgtable-hwdef.h | 6 +-- arch/arm64/include/asm/pgtable.h | 10 ++++- arch/arm64/include/asm/sparsemem.h | 4 ++ arch/arm64/kernel/image-vars.h | 11 ++++++ arch/arm64/kernel/image.h | 4 ++ arch/arm64/kernel/pi/map_kernel.c | 45 ++++++++++++++++++++++ arch/arm64/kvm/arm.c | 10 +++++ arch/arm64/kvm/hyp/nvhe/Makefile | 1 + arch/arm64/kvm/hyp/nvhe/pgtable-geometry.c | 16 ++++++++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/pgd.c | 10 +++-- arch/arm64/mm/pgtable-geometry.c | 24 ++++++++++++ drivers/firmware/efi/libstub/arm64.c | 3 +- 16 files changed, 187 insertions(+), 17 deletions(-) create mode 100644 arch/arm64/kvm/hyp/nvhe/pgtable-geometry.c create mode 100644 arch/arm64/mm/pgtable-geometry.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a2f8ff354ca67..573d308741169 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -121,6 +121,7 @@ config ARM64 select BUILDTIME_TABLE_SORT select CLONE_BACKWARDS select COMMON_CLK + select CONSTRUCTORS if ARM64_BOOT_TIME_PAGE_SIZE select CPU_PM if (SUSPEND || CPU_IDLE) select CPUMASK_OFFSTACK if NR_CPUS > 256 select CRC32 @@ -284,18 +285,20 @@ config MMU config ARM64_CONT_PTE_SHIFT int + depends on !ARM64_BOOT_TIME_PAGE_SIZE default 5 if PAGE_SIZE_64KB default 7 if PAGE_SIZE_16KB default 4 config ARM64_CONT_PMD_SHIFT int + depends on !ARM64_BOOT_TIME_PAGE_SIZE default 5 if PAGE_SIZE_64KB default 5 if PAGE_SIZE_16KB default 4 config ARCH_MMAP_RND_BITS_MIN - default 14 if PAGE_SIZE_64KB + default 14 if ARM64_64K_PAGES || ARM64_BOOT_TIME_PAGE_SIZE default 16 if PAGE_SIZE_16KB default 18 @@ -306,15 +309,15 @@ config ARCH_MMAP_RND_BITS_MAX default 24 if ARM64_VA_BITS=39 default 27 if ARM64_VA_BITS=42 default 30 if ARM64_VA_BITS=47 - default 29 if ARM64_VA_BITS=48 && ARM64_64K_PAGES + default 29 if ARM64_VA_BITS=48 && (ARM64_64K_PAGES || ARM64_BOOT_TIME_PAGE_SIZE) default 31 if ARM64_VA_BITS=48 && ARM64_16K_PAGES default 33 if ARM64_VA_BITS=48 - default 14 if ARM64_64K_PAGES + default 14 if ARM64_64K_PAGES || ARM64_BOOT_TIME_PAGE_SIZE default 16 if ARM64_16K_PAGES default 18 config ARCH_MMAP_RND_COMPAT_BITS_MIN - default 7 if ARM64_64K_PAGES + default 7 if ARM64_64K_PAGES || ARM64_BOOT_TIME_PAGE_SIZE default 9 if ARM64_16K_PAGES default 11 @@ -362,6 +365,7 @@ config FIX_EARLYCON_MEM config PGTABLE_LEVELS int + default 4 if ARM64_BOOT_TIME_PAGE_SIZE # Advertise max supported levels default 2 if ARM64_16K_PAGES && ARM64_VA_BITS_36 default 2 if ARM64_64K_PAGES && ARM64_VA_BITS_42 default 3 if ARM64_64K_PAGES && (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) @@ -1316,6 +1320,14 @@ config ARM64_64K_PAGES look-up. AArch32 emulation requires applications compiled with 64K aligned segments. +config ARM64_BOOT_TIME_PAGE_SIZE + bool "Boot-time selection" + select HAVE_PAGE_SIZE_64KB # Advertise largest page size to core + help + Select desired page size (4KB, 16KB or 64KB) at boot-time via the + kernel command line option "arm64.pagesize=4k", "arm64.pagesize=16k" + or "arm64.pagesize=64k". + endchoice choice @@ -1348,6 +1360,7 @@ config ARM64_VA_BITS_48 config ARM64_VA_BITS_52 bool "52-bit" depends on ARM64_PAN || !ARM64_SW_TTBR0_PAN + depends on !ARM64_BOOT_TIME_PAGE_SIZE help Enable 52-bit virtual addressing for userspace when explicitly requested via a hint to mmap(). The kernel will also use 52-bit @@ -1588,9 +1601,10 @@ config XEN # 4K | 27 | 12 | 15 | 10 | # 16K | 27 | 14 | 13 | 11 | # 64K | 29 | 16 | 13 | 13 | +# BOOT| 29 | 16 (max) | 13 | 13 | config ARCH_FORCE_MAX_ORDER int - default "13" if ARM64_64K_PAGES + default "13" if ARM64_64K_PAGES || ARM64_BOOT_TIME_PAGE_SIZE default "11" if ARM64_16K_PAGES default "10" help @@ -1663,7 +1677,7 @@ config ARM64_TAGGED_ADDR_ABI menuconfig COMPAT bool "Kernel support for 32-bit EL0" - depends on ARM64_4K_PAGES || EXPERT + depends on ARM64_4K_PAGES || ARM64_BOOT_TIME_PAGE_SIZE || EXPERT select HAVE_UID16 select OLD_SIGSUSPEND3 select COMPAT_OLD_SIGACTION diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index c838309e4ec47..9397a14642afa 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -145,4 +145,15 @@ extern unsigned long kvm_nvhe_sym(__icache_flags); extern unsigned int kvm_nvhe_sym(kvm_arm_vmid_bits); extern unsigned int kvm_nvhe_sym(kvm_host_sve_max_vl); +#ifdef CONFIG_ARM64_BOOT_TIME_PAGE_SIZE +extern int kvm_nvhe_sym(ptg_page_shift); +extern int kvm_nvhe_sym(ptg_pmd_shift); +extern int kvm_nvhe_sym(ptg_pud_shift); +extern int kvm_nvhe_sym(ptg_p4d_shift); +extern int kvm_nvhe_sym(ptg_pgdir_shift); +extern int kvm_nvhe_sym(ptg_cont_pte_shift); +extern int kvm_nvhe_sym(ptg_cont_pmd_shift); +extern int kvm_nvhe_sym(ptg_pgtable_levels); +#endif /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ + #endif /* __ARM64_KVM_HYP_H__ */ diff --git a/arch/arm64/include/asm/pgtable-geometry.h b/arch/arm64/include/asm/pgtable-geometry.h index 62fe125909c08..18a8c8d499ecc 100644 --- a/arch/arm64/include/asm/pgtable-geometry.h +++ b/arch/arm64/include/asm/pgtable-geometry.h @@ -6,16 +6,33 @@ #define ARM64_PAGE_SHIFT_16K 14 #define ARM64_PAGE_SHIFT_64K 16 +#ifdef CONFIG_ARM64_BOOT_TIME_PAGE_SIZE +#define PAGE_SHIFT_MIN ARM64_PAGE_SHIFT_4K +#define PAGE_SHIFT_MAX ARM64_PAGE_SHIFT_64K +#else /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ #define PAGE_SHIFT_MIN CONFIG_PAGE_SHIFT +#define PAGE_SHIFT_MAX CONFIG_PAGE_SHIFT +#endif /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ + #define PAGE_SIZE_MIN (_AC(1, UL) << PAGE_SHIFT_MIN) #define PAGE_MASK_MIN (~(PAGE_SIZE_MIN-1)) - -#define PAGE_SHIFT_MAX CONFIG_PAGE_SHIFT #define PAGE_SIZE_MAX (_AC(1, UL) << PAGE_SHIFT_MAX) #define PAGE_MASK_MAX (~(PAGE_SIZE_MAX-1)) #include +#ifdef CONFIG_ARM64_BOOT_TIME_PAGE_SIZE +#ifndef __ASSEMBLY__ +extern int ptg_page_shift; +extern int ptg_pmd_shift; +extern int ptg_pud_shift; +extern int ptg_p4d_shift; +extern int ptg_pgdir_shift; +extern int ptg_cont_pte_shift; +extern int ptg_cont_pmd_shift; +extern int ptg_pgtable_levels; +#endif /* __ASSEMBLY__ */ +#else /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ #define ptg_page_shift CONFIG_PAGE_SHIFT #define ptg_pmd_shift ARM64_HW_PGTABLE_LEVEL_SHIFT(2) #define ptg_pud_shift ARM64_HW_PGTABLE_LEVEL_SHIFT(1) @@ -24,5 +41,6 @@ #define ptg_cont_pte_shift (CONFIG_ARM64_CONT_PTE_SHIFT + PAGE_SHIFT) #define ptg_cont_pmd_shift (CONFIG_ARM64_CONT_PMD_SHIFT + PMD_SHIFT) #define ptg_pgtable_levels CONFIG_PGTABLE_LEVELS +#endif /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ #endif /* ASM_PGTABLE_GEOMETRY_H */ diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index ca8bcbc1fe220..da5404617acbf 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -52,7 +52,7 @@ #define PMD_SHIFT ptg_pmd_shift #define PMD_SIZE (_AC(1, UL) << PMD_SHIFT) #define PMD_MASK (~(PMD_SIZE-1)) -#define PTRS_PER_PMD (1 << (PAGE_SHIFT - 3)) +#define PTRS_PER_PMD (ptg_pgtable_levels > 2 ? (1 << (PAGE_SHIFT - 3)) : 1) #define MAX_PTRS_PER_PMD (1 << (PAGE_SHIFT_MAX - 3)) #endif @@ -63,7 +63,7 @@ #define PUD_SHIFT ptg_pud_shift #define PUD_SIZE (_AC(1, UL) << PUD_SHIFT) #define PUD_MASK (~(PUD_SIZE-1)) -#define PTRS_PER_PUD (1 << (PAGE_SHIFT - 3)) +#define PTRS_PER_PUD (ptg_pgtable_levels > 3 ? (1 << (PAGE_SHIFT - 3)) : 1) #define MAX_PTRS_PER_PUD (1 << (PAGE_SHIFT_MAX - 3)) #endif @@ -71,7 +71,7 @@ #define P4D_SHIFT ptg_p4d_shift #define P4D_SIZE (_AC(1, UL) << P4D_SHIFT) #define P4D_MASK (~(P4D_SIZE-1)) -#define PTRS_PER_P4D (1 << (PAGE_SHIFT - 3)) +#define PTRS_PER_P4D (ptg_pgtable_levels > 4 ? (1 << (PAGE_SHIFT - 3)) : 1) #define MAX_PTRS_PER_P4D (1 << (PAGE_SHIFT_MAX - 3)) #endif diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 8ead41da715b0..ad9f75f5cc29a 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -755,7 +755,7 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd) static __always_inline bool pgtable_l3_enabled(void) { - return true; + return ptg_pgtable_levels > 2; } static inline bool mm_pmd_folded(const struct mm_struct *mm) @@ -888,6 +888,8 @@ static inline bool pgtable_l3_enabled(void) { return false; } static __always_inline bool pgtable_l4_enabled(void) { + if (IS_ENABLED(CONFIG_ARM64_BOOT_TIME_PAGE_SIZE)) + return ptg_pgtable_levels > 3; if (CONFIG_PGTABLE_LEVELS > 4 || !IS_ENABLED(CONFIG_ARM64_LPA2)) return true; if (!alternative_has_cap_likely(ARM64_ALWAYS_BOOT)) @@ -935,6 +937,8 @@ static inline phys_addr_t p4d_page_paddr(p4d_t p4d) static inline pud_t *p4d_to_folded_pud(p4d_t *p4dp, unsigned long addr) { + if (IS_ENABLED(CONFIG_ARM64_BOOT_TIME_PAGE_SIZE)) + return (pud_t *)p4dp; return (pud_t *)PTR_ALIGN_DOWN(p4dp, PAGE_SIZE) + pud_index(addr); } @@ -1014,6 +1018,8 @@ static inline bool pgtable_l4_enabled(void) { return false; } static __always_inline bool pgtable_l5_enabled(void) { + if (IS_ENABLED(CONFIG_ARM64_BOOT_TIME_PAGE_SIZE)) + return ptg_pgtable_levels > 4; if (!alternative_has_cap_likely(ARM64_ALWAYS_BOOT)) return vabits_actual == VA_BITS; return alternative_has_cap_unlikely(ARM64_HAS_VA52); @@ -1059,6 +1065,8 @@ static inline phys_addr_t pgd_page_paddr(pgd_t pgd) static inline p4d_t *pgd_to_folded_p4d(pgd_t *pgdp, unsigned long addr) { + if (IS_ENABLED(CONFIG_ARM64_BOOT_TIME_PAGE_SIZE)) + return (p4d_t *)pgdp; return (p4d_t *)PTR_ALIGN_DOWN(pgdp, PAGE_SIZE) + p4d_index(addr); } diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h index a05fdd54014f7..2daf1263ba638 100644 --- a/arch/arm64/include/asm/sparsemem.h +++ b/arch/arm64/include/asm/sparsemem.h @@ -17,6 +17,10 @@ * entries could not be created for vmemmap mappings. * 16K follows 4K for simplicity. */ +#ifdef CONFIG_ARM64_BOOT_TIME_PAGE_SIZE +#define SECTION_SIZE_BITS 29 +#else #define SECTION_SIZE_BITS (PAGE_SIZE == SZ_64K ? 29 : 27) +#endif #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index a168f3337446f..9968320f83bc4 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -36,6 +36,17 @@ PROVIDE(__pi___memcpy = __pi_memcpy); PROVIDE(__pi___memmove = __pi_memmove); PROVIDE(__pi___memset = __pi_memset); +#ifdef CONFIG_ARM64_BOOT_TIME_PAGE_SIZE +PROVIDE(__pi_ptg_page_shift = ptg_page_shift); +PROVIDE(__pi_ptg_pmd_shift = ptg_pmd_shift); +PROVIDE(__pi_ptg_pud_shift = ptg_pud_shift); +PROVIDE(__pi_ptg_p4d_shift = ptg_p4d_shift); +PROVIDE(__pi_ptg_pgdir_shift = ptg_pgdir_shift); +PROVIDE(__pi_ptg_cont_pte_shift = ptg_cont_pte_shift); +PROVIDE(__pi_ptg_cont_pmd_shift = ptg_cont_pmd_shift); +PROVIDE(__pi_ptg_pgtable_levels = ptg_pgtable_levels); +#endif /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ + PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); PROVIDE(__pi_id_aa64mmfr0_override = id_aa64mmfr0_override); diff --git a/arch/arm64/kernel/image.h b/arch/arm64/kernel/image.h index 7bc3ba8979019..01502fc3b891b 100644 --- a/arch/arm64/kernel/image.h +++ b/arch/arm64/kernel/image.h @@ -47,7 +47,11 @@ #define __HEAD_FLAG_BE ARM64_IMAGE_FLAG_LE #endif +#ifdef CONFIG_ARM64_BOOT_TIME_PAGE_SIZE +#define __HEAD_FLAG_PAGE_SIZE 0 +#else #define __HEAD_FLAG_PAGE_SIZE ((PAGE_SHIFT - 10) / 2) +#endif #define __HEAD_FLAG_PHYS_BASE 1 diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index deb8cd50b0b0c..22b3c70e04f9c 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -221,6 +221,49 @@ static void __init map_fdt(u64 fdt, int page_shift) dsb(ishst); } +#ifdef CONFIG_ARM64_BOOT_TIME_PAGE_SIZE +static void __init ptg_init(int page_shift) +{ + ptg_pgtable_levels = + __ARM64_HW_PGTABLE_LEVELS(page_shift, CONFIG_ARM64_VA_BITS); + + ptg_pgdir_shift = __ARM64_HW_PGTABLE_LEVEL_SHIFT(page_shift, + 4 - ptg_pgtable_levels); + + ptg_p4d_shift = ptg_pgtable_levels >= 5 ? + __ARM64_HW_PGTABLE_LEVEL_SHIFT(page_shift, 0) : + ptg_pgdir_shift; + + ptg_pud_shift = ptg_pgtable_levels >= 4 ? + __ARM64_HW_PGTABLE_LEVEL_SHIFT(page_shift, 1) : + ptg_pgdir_shift; + + ptg_pmd_shift = ptg_pgtable_levels >= 3 ? + __ARM64_HW_PGTABLE_LEVEL_SHIFT(page_shift, 2) : + ptg_pgdir_shift; + + ptg_page_shift = page_shift; + + switch (page_shift) { + case ARM64_PAGE_SHIFT_64K: + ptg_cont_pte_shift = ptg_page_shift + 5; + ptg_cont_pmd_shift = ptg_pmd_shift + 5; + break; + case ARM64_PAGE_SHIFT_16K: + ptg_cont_pte_shift = ptg_page_shift + 7; + ptg_cont_pmd_shift = ptg_pmd_shift + 5; + break; + default: /* ARM64_PAGE_SHIFT_4K */ + ptg_cont_pte_shift = ptg_page_shift + 4; + ptg_cont_pmd_shift = ptg_pmd_shift + 4; + } +} +#else /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ +static inline void ptg_init(int page_shift) +{ +} +#endif /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ + asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) { static char const chosen_str[] __initconst = "/chosen"; @@ -247,6 +290,8 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) if (!page_shift) page_shift = early_page_shift; + ptg_init(page_shift); + if (va_bits > 48) { u64 page_size = early_page_size(page_shift); diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 9bef7638342ef..c835a50b8b768 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -2424,6 +2424,16 @@ static void kvm_hyp_init_symbols(void) kvm_nvhe_sym(id_aa64smfr0_el1_sys_val) = read_sanitised_ftr_reg(SYS_ID_AA64SMFR0_EL1); kvm_nvhe_sym(__icache_flags) = __icache_flags; kvm_nvhe_sym(kvm_arm_vmid_bits) = kvm_arm_vmid_bits; +#ifdef CONFIG_ARM64_BOOT_TIME_PAGE_SIZE + kvm_nvhe_sym(ptg_page_shift) = ptg_page_shift; + kvm_nvhe_sym(ptg_pmd_shift) = ptg_pmd_shift; + kvm_nvhe_sym(ptg_pud_shift) = ptg_pud_shift; + kvm_nvhe_sym(ptg_p4d_shift) = ptg_p4d_shift; + kvm_nvhe_sym(ptg_pgdir_shift) = ptg_pgdir_shift; + kvm_nvhe_sym(ptg_cont_pte_shift) = ptg_cont_pte_shift; + kvm_nvhe_sym(ptg_cont_pmd_shift) = ptg_cont_pmd_shift; + kvm_nvhe_sym(ptg_pgtable_levels) = ptg_pgtable_levels; +#endif /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ } static int __init kvm_hyp_init_protection(u32 hyp_va_bits) diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index b43426a493df5..a8fcbb84c7996 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -27,6 +27,7 @@ hyp-obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o hyp-obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o +hyp-obj-$(CONFIG_ARM64_BOOT_TIME_PAGE_SIZE) += pgtable-geometry.o hyp-obj-$(CONFIG_LIST_HARDENED) += list_debug.o hyp-obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/pgtable-geometry.c b/arch/arm64/kvm/hyp/nvhe/pgtable-geometry.c new file mode 100644 index 0000000000000..17f807450a31a --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/pgtable-geometry.c @@ -0,0 +1,16 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2024 ARM Ltd. + */ + +#include +#include + +int ptg_page_shift __ro_after_init; +int ptg_pmd_shift __ro_after_init; +int ptg_pud_shift __ro_after_init; +int ptg_p4d_shift __ro_after_init; +int ptg_pgdir_shift __ro_after_init; +int ptg_cont_pte_shift __ro_after_init; +int ptg_cont_pmd_shift __ro_after_init; +int ptg_pgtable_levels __ro_after_init; diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 60454256945b8..2ba30d06b35fe 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -3,6 +3,7 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ cache.o copypage.o flush.o \ ioremap.o mmap.o pgd.o mmu.o \ context.o proc.o pageattr.o fixmap.o +obj-$(CONFIG_ARM64_BOOT_TIME_PAGE_SIZE) += pgtable-geometry.o obj-$(CONFIG_ARM64_CONTPTE) += contpte.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c index 4b106510358b1..c052d0dcb0c69 100644 --- a/arch/arm64/mm/pgd.c +++ b/arch/arm64/mm/pgd.c @@ -21,10 +21,12 @@ static bool pgdir_is_page_size(void) { if (PGD_SIZE == PAGE_SIZE) return true; - if (CONFIG_PGTABLE_LEVELS == 4) - return !pgtable_l4_enabled(); - if (CONFIG_PGTABLE_LEVELS == 5) - return !pgtable_l5_enabled(); + if (!IS_ENABLED(CONFIG_ARM64_BOOT_TIME_PAGE_SIZE)) { + if (CONFIG_PGTABLE_LEVELS == 4) + return !pgtable_l4_enabled(); + if (CONFIG_PGTABLE_LEVELS == 5) + return !pgtable_l5_enabled(); + } return false; } diff --git a/arch/arm64/mm/pgtable-geometry.c b/arch/arm64/mm/pgtable-geometry.c new file mode 100644 index 0000000000000..ba50637f1e9d0 --- /dev/null +++ b/arch/arm64/mm/pgtable-geometry.c @@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2024 ARM Ltd. + */ + +#include +#include + +/* + * TODO: These should be __ro_after_init, but we need to write to them from the + * pi code where they are mapped in the early page table as read-only. + * __ro_after_init doesn't become writable until later when the swapper pgtable + * is fully set up. We should update the early page table to map __ro_after_init + * as read-write. + */ + +int ptg_page_shift __read_mostly; +int ptg_pmd_shift __read_mostly; +int ptg_pud_shift __read_mostly; +int ptg_p4d_shift __read_mostly; +int ptg_pgdir_shift __read_mostly; +int ptg_cont_pte_shift __read_mostly; +int ptg_cont_pmd_shift __read_mostly; +int ptg_pgtable_levels __read_mostly; diff --git a/drivers/firmware/efi/libstub/arm64.c b/drivers/firmware/efi/libstub/arm64.c index e57cd3de0a00f..8db9dba7d5423 100644 --- a/drivers/firmware/efi/libstub/arm64.c +++ b/drivers/firmware/efi/libstub/arm64.c @@ -68,7 +68,8 @@ efi_status_t check_platform_features(void) efi_novamap = true; /* UEFI mandates support for 4 KB granularity, no need to check */ - if (IS_ENABLED(CONFIG_ARM64_4K_PAGES)) + if (IS_ENABLED(CONFIG_ARM64_4K_PAGES) || + IS_ENABLED(CONFIG_ARM64_BOOT_TIME_PAGE_SIZE)) return EFI_SUCCESS; tg = (read_cpuid(ID_AA64MMFR0_EL1) >> ID_AA64MMFR0_EL1_TGRAN_SHIFT) & 0xf;