From patchwork Tue Feb 7 16:20:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Liu X-Patchwork-Id: 651264 Delivered-To: patch@linaro.org Received: by 2002:a17:522:d8c:b0:4be:c3dc:14d8 with SMTP id d12csp2965724pva; Tue, 7 Feb 2023 08:20:49 -0800 (PST) X-Google-Smtp-Source: AK7set8S35IrFiHlTNI5jpVSCIrmiujhNBve5HgW4DjwUXCTERWtNsm5LqI0Ph9h/4XEWlJ37BPf X-Received: by 2002:aca:2b14:0:b0:364:f951:1a5a with SMTP id i20-20020aca2b14000000b00364f9511a5amr1483342oik.31.1675786849162; Tue, 07 Feb 2023 08:20:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675786849; cv=none; d=google.com; s=arc-20160816; b=Xrk8+DgirKFnd/JgN71ea4bPN7qDM+f4ZjolvDdA4PZQG8axPcGTx/skW8PxeIul4i YNn42//czy/3nc4QIpbTasw0OYGLrP1UfVZDZx9HGz7cloy5kYBRiZ7nvdL3mDPkTKwB BXIXXgloBZ36h30CTboqtxNZtHVnAst6kIbqetP5W2g0K769il9rKZ9MIMVr95swmsXy vPQcybmIB+OzJtcNZTpxL7zCJNR/pPY+Akz3VghBV+HvG7My+PPI84metbcN9nNhREL0 MysUM5X6VYskQ5EbQANNBv+9++YygYfY0P44pxj6xDzEcIRgDy6x+OLUU9iBp3tq8ylZ v6hw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=xDzrZj0QYSJnrHx2zyrglbazzr8is6uNmuej/81MAE4=; b=hH/eLl3o3LaUJ5hx2dZyKqbLlVkhsnZU0NUIbE9G13qJ1HXBIycdUpJFmoPENZg4Uz yged2Sio1JWwGZFTETkq+lPuJCAkFuH548FKYeTcL8v9jfrWZVI8Guhb/+tEVD+pyb07 48C/QylljcpCj22S+ck8WWwmzeXfab1pgCUhreCNvaug/iHznHgr6ke12Wr+ogG5uz+d OFMZv1nJbKfjOINGl3jX2wpDI9W4+mSQFlx0kRlGT6D3qY4crEAogmJLM+sqX8LTKzr1 C/X8Tr4IACJvEiw2RE+QPp18Ouw4zx2zrOQGOLIEMHMU0EyCm6uO0sRdhOg2eKitRfYR 0uSg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Ywszl8bU; spf=pass (google.com: domain of u-boot-bounces@lists.denx.de designates 85.214.62.61 as permitted sender) smtp.mailfrom=u-boot-bounces@lists.denx.de; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from phobos.denx.de (phobos.denx.de. [85.214.62.61]) by mx.google.com with ESMTPS id p83-20020aca5b56000000b0036d3ac5e43fsi13589453oib.86.2023.02.07.08.20.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Feb 2023 08:20:49 -0800 (PST) Received-SPF: pass (google.com: domain of u-boot-bounces@lists.denx.de designates 85.214.62.61 as permitted sender) client-ip=85.214.62.61; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Ywszl8bU; spf=pass (google.com: domain of u-boot-bounces@lists.denx.de designates 85.214.62.61 as permitted sender) smtp.mailfrom=u-boot-bounces@lists.denx.de; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id AA46085E08; Tue, 7 Feb 2023 17:20:36 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="Ywszl8bU"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 45F5485E39; Tue, 7 Feb 2023 17:20:34 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 8096285D9E for ; Tue, 7 Feb 2023 17:20:28 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=paul.liu@linaro.org Received: by mail-pl1-x635.google.com with SMTP id e19so8232753plc.9 for ; Tue, 07 Feb 2023 08:20:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xDzrZj0QYSJnrHx2zyrglbazzr8is6uNmuej/81MAE4=; b=Ywszl8bUelRYKCF0hGjLAZxDbG7dROUjtj54YqDSdQR+BdMsBW4izYzp2TompNRbOE jzFiM/+XdrvaPE7eVDnffEtWVNmUyKjcUymwLV3xgsQcDLcktdCszC4KgObNQeMJ97Gy Agzl1pfleGPGXogWo2Y2usny7HnU8zuXLwQtx9+yYiU7gUUMinisUKtIGzw4VFP25vze whNBHL6oelC6kRAlvz+TQ2oz9yE3SLM+xceuGL58AO/y7B3XZhiyEH8J5MfPL0Ckjkk6 oI6fqTxNllnMuywomMFsN+Nemk9sdfp/G5jPG3yyBF2HiX/yeB2hiD1YHGVifkoKL9hP e5DA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xDzrZj0QYSJnrHx2zyrglbazzr8is6uNmuej/81MAE4=; b=gDnlZSwBq8HL0fz7spu4l3qg9bNOSWTVjaPKs1MMiOH0ZLcLKV5BgHfTOMwU13Lx1m Fh2k7OcPg0zF5KDc9g/jBjWFwrd6uhm62Afyk9G2Dn4tEEsbQCgURh338xDUJsUmhc+k Bc3aOH5vCDw9wYWuE55Ev5RF1jzDErhwHBzny3EXPNDLlSqmfySgN2+PiwTY18zE0dzO 1WUjOSQZy56OelGe5Acg822U+sF9Owu5gwraCyiinexhsJLC1msFJSJSrJr+vs1QtiOf i2lGGLyWivVGQ7kv9drw5/86I4GZEYSNbzHYl8iBwjmfXKqx07DffOcguFFCM2bsw87n 0gUw== X-Gm-Message-State: AO0yUKUolFnp91LI2jmJSKLsnzJSwz5/u3qTuKlfYVt8d/eXZlcv8hB9 zqkvisfGer8n4t1gm+ifjKK7ysiayD8v9n8P X-Received: by 2002:a17:90b:3a8b:b0:230:9b7b:20fd with SMTP id om11-20020a17090b3a8b00b002309b7b20fdmr4814353pjb.5.1675786826385; Tue, 07 Feb 2023 08:20:26 -0800 (PST) Received: from localhost ([111.184.129.17]) by smtp.gmail.com with ESMTPSA id jm24-20020a17090304d800b00198fb25d09bsm6399372plb.237.2023.02.07.08.20.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Feb 2023 08:20:26 -0800 (PST) From: "Ying-Chun Liu (PaulLiu)" To: u-boot@lists.denx.de Cc: "Ying-Chun Liu (PaulLiu)" , Marc Zyngier , Will Deacon , Tom Rini Subject: [PATCH 1/2] arm: cpu: Add optional CMOs by VA Date: Wed, 8 Feb 2023 00:20:13 +0800 Message-Id: <20230207162014.58664-2-paul.liu@linaro.org> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230207162014.58664-1-paul.liu@linaro.org> References: <20230207162014.58664-1-paul.liu@linaro.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean Exposing set/way cache maintenance to a virtual machine is unsafe, not least because the instructions are not permission-checked but also because they are not broadcast between CPUs. Consequently, KVM traps and emulates such maintenance in the host kernel using by-VA operations and looping over the stage-2 page-tables. However, when running under protected KVM, these instructions are not able to be emulated and will instead result in an exception being delivered to the guest. Introduce CONFIG_CMO_BY_VA_ONLY so that virtual platforms can select this option and perform by-VA cache maintenance instead of using the set/way instructions. Signed-off-by: Ying-Chun Liu (PaulLiu) Signed-off-by: Marc Zyngier Signed-off-by: Will Deacon Cc: Tom Rini Signed-off-by: Random Developer Signed-off-by: Random Committer Signed-off-by: Paul Liu --- arch/arm/cpu/armv8/Kconfig | 4 ++ arch/arm/cpu/armv8/cache.S | 50 +++++++++++++----- arch/arm/cpu/armv8/cache_v8.c | 97 ++++++++++++++++++++++++++++++++++- arch/arm/cpu/armv8/cpu.c | 30 +++++++---- 4 files changed, 155 insertions(+), 26 deletions(-) diff --git a/arch/arm/cpu/armv8/Kconfig b/arch/arm/cpu/armv8/Kconfig index 1305238c9d..7d5cf1594d 100644 --- a/arch/arm/cpu/armv8/Kconfig +++ b/arch/arm/cpu/armv8/Kconfig @@ -1,5 +1,9 @@ if ARM64 +config CMO_BY_VA_ONLY + bool "Force cache maintenance to be exclusively by VA" + depends on !SYS_DISABLE_DCACHE_OPS + config ARMV8_SPL_EXCEPTION_VECTORS bool "Install crash dump exception vectors" depends on SPL diff --git a/arch/arm/cpu/armv8/cache.S b/arch/arm/cpu/armv8/cache.S index d1cee23437..3fe935cf28 100644 --- a/arch/arm/cpu/armv8/cache.S +++ b/arch/arm/cpu/armv8/cache.S @@ -12,6 +12,7 @@ #include #include +#ifndef CONFIG_CMO_BY_VA_ONLY /* * void __asm_dcache_level(level) * @@ -116,6 +117,41 @@ ENTRY(__asm_invalidate_dcache_all) ENDPROC(__asm_invalidate_dcache_all) .popsection +.pushsection .text.__asm_flush_l3_dcache, "ax" +WEAK(__asm_flush_l3_dcache) + mov x0, #0 /* return status as success */ + ret +ENDPROC(__asm_flush_l3_dcache) +.popsection + +.pushsection .text.__asm_invalidate_l3_icache, "ax" +WEAK(__asm_invalidate_l3_icache) + mov x0, #0 /* return status as success */ + ret +ENDPROC(__asm_invalidate_l3_icache) +.popsection + +#else /* CONFIG_CMO_BY_VA */ + +/* + * Define these so that they actively clash with in implementation + * accidentally selecting CONFIG_CMO_BY_VA + */ + +.pushsection .text.__asm_invalidate_l3_icache, "ax" +ENTRY(__asm_invalidate_l3_icache) + mov x0, xzr + ret +ENDPROC(__asm_invalidate_l3_icache) +.popsection +.pushsection .text.__asm_flush_l3_dcache, "ax" +ENTRY(__asm_flush_l3_dcache) + mov x0, xzr + ret +ENDPROC(__asm_flush_l3_dcache) +.popsection +#endif /* CONFIG_CMO_BY_VA */ + /* * void __asm_flush_dcache_range(start, end) * @@ -189,20 +225,6 @@ WEAK(__asm_invalidate_l3_dcache) ENDPROC(__asm_invalidate_l3_dcache) .popsection -.pushsection .text.__asm_flush_l3_dcache, "ax" -WEAK(__asm_flush_l3_dcache) - mov x0, #0 /* return status as success */ - ret -ENDPROC(__asm_flush_l3_dcache) -.popsection - -.pushsection .text.__asm_invalidate_l3_icache, "ax" -WEAK(__asm_invalidate_l3_icache) - mov x0, #0 /* return status as success */ - ret -ENDPROC(__asm_invalidate_l3_icache) -.popsection - /* * void __asm_switch_ttbr(ulong new_ttbr) * diff --git a/arch/arm/cpu/armv8/cache_v8.c b/arch/arm/cpu/armv8/cache_v8.c index 2a226fd063..f333ad8889 100644 --- a/arch/arm/cpu/armv8/cache_v8.c +++ b/arch/arm/cpu/armv8/cache_v8.c @@ -163,6 +163,83 @@ static u64 *find_pte(u64 addr, int level) return NULL; } +#ifdef CONFIG_CMO_BY_VA_ONLY +static void __cmo_on_leaves(void (*cmo_fn)(unsigned long, unsigned long), + u64 pte, int level, u64 base) +{ + u64 *ptep; + int i; + + ptep = (u64 *)(pte & GENMASK_ULL(47, PAGE_SHIFT)); + for (i = 0; i < PAGE_SIZE / sizeof(u64); i++) { + u64 end, va = base + i * BIT(level2shift(level)); + u64 type, attrs; + + pte = ptep[i]; + type = pte & PTE_TYPE_MASK; + attrs = pte & PMD_ATTRINDX_MASK; + debug("PTE %llx at level %d VA %llx\n", pte, level, va); + + /* Not valid? next! */ + if (!(type & PTE_TYPE_VALID)) + continue; + + /* Not a leaf? Recurse on the next level */ + if (!(type == PTE_TYPE_BLOCK || + (level == 3 && type == PTE_TYPE_PAGE))) { + __cmo_on_leaves(cmo_fn, pte, level + 1, va); + continue; + } + + /* + * From this point, this must be a leaf. + * + * Start excluding non memory mappings + */ + if (attrs != PTE_BLOCK_MEMTYPE(MT_NORMAL) && + attrs != PTE_BLOCK_MEMTYPE(MT_NORMAL_NC)) + continue; + + end = va + BIT(level2shift(level)) - 1; + + /* No intersection with RAM? */ + if (end < gd->ram_base || + va >= (gd->ram_base + gd->ram_size)) + continue; + + /* + * OK, we have a partial RAM mapping. However, this + * can cover *more* than the RAM. Yes, u-boot is + * *that* braindead. Compute the intersection we care + * about, and not a byte more. + */ + va = max(va, (u64)gd->ram_base); + end = min(end, gd->ram_base + gd->ram_size); + + debug("Flush PTE %llx at level %d: %llx-%llx\n", + pte, level, va, end); + cmo_fn(va, end); + } +} + +static void apply_cmo_to_mappings(void (*cmo_fn)(unsigned long, unsigned long)) +{ + u64 va_bits; + int sl = 0; + + if (!gd->arch.tlb_addr) + return; + + get_tcr(NULL, &va_bits); + if (va_bits < 39) + sl = 1; + + __cmo_on_leaves(cmo_fn, gd->arch.tlb_addr, sl, 0); +} +#else +static inline void apply_cmo_to_mappings(void *dummy) {} +#endif + /* Returns and creates a new full table (512 entries) */ static u64 *create_table(void) { @@ -447,8 +524,12 @@ __weak void mmu_setup(void) */ void invalidate_dcache_all(void) { +#ifndef CONFIG_CMO_BY_VA_ONLY __asm_invalidate_dcache_all(); __asm_invalidate_l3_dcache(); +#else + apply_cmo_to_mappings(invalidate_dcache_range); +#endif } /* @@ -458,6 +539,7 @@ void invalidate_dcache_all(void) */ inline void flush_dcache_all(void) { +#ifndef CONFIG_CMO_BY_VA_ONLY int ret; __asm_flush_dcache_all(); @@ -466,6 +548,9 @@ inline void flush_dcache_all(void) debug("flushing dcache returns 0x%x\n", ret); else debug("flushing dcache successfully.\n"); +#else + apply_cmo_to_mappings(flush_dcache_range); +#endif } #ifndef CONFIG_SYS_DISABLE_DCACHE_OPS @@ -520,9 +605,19 @@ void dcache_disable(void) if (!(sctlr & CR_C)) return; + if (IS_ENABLED(CONFIG_CMO_BY_VA_ONLY)) { + /* + * When invalidating by VA, do it *before* turning the MMU + * off, so that at least our stack is coherent. + */ + flush_dcache_all(); + } + set_sctlr(sctlr & ~(CR_C|CR_M)); - flush_dcache_all(); + if (!IS_ENABLED(CONFIG_CMO_BY_VA_ONLY)) + flush_dcache_all(); + __asm_invalidate_tlb_all(); } diff --git a/arch/arm/cpu/armv8/cpu.c b/arch/arm/cpu/armv8/cpu.c index db5d460eb4..3c7f36ad8d 100644 --- a/arch/arm/cpu/armv8/cpu.c +++ b/arch/arm/cpu/armv8/cpu.c @@ -48,18 +48,26 @@ int cleanup_before_linux(void) disable_interrupts(); - /* - * Turn off I-cache and invalidate it - */ - icache_disable(); - invalidate_icache_all(); + if (IS_ENABLED(CONFIG_CMO_BY_VA_ONLY)) { + /* + * Disable D-cache. + */ + dcache_disable(); + } else { + /* + * Turn off I-cache and invalidate it + */ + icache_disable(); + invalidate_icache_all(); - /* - * turn off D-cache - * dcache_disable() in turn flushes the d-cache and disables MMU - */ - dcache_disable(); - invalidate_dcache_all(); + /* + * turn off D-cache + * dcache_disable() in turn flushes the d-cache and disables + * MMU + */ + dcache_disable(); + invalidate_dcache_all(); + } return 0; }