From patchwork Tue Jan 5 17:19:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 356905 Delivered-To: patch@linaro.org Received: by 2002:a02:85a7:0:0:0:0:0 with SMTP id d36csp187626jai; Tue, 5 Jan 2021 09:45:10 -0800 (PST) X-Google-Smtp-Source: ABdhPJxdJt2mLyXN0/5G2Pqa7AlNvPykesZZWsQSNJJixPUY3fG7kz7Vv28GniUESz3AphN/8Qv6 X-Received: by 2002:a25:250b:: with SMTP id l11mr849288ybl.196.1609868710300; Tue, 05 Jan 2021 09:45:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1609868710; cv=none; d=google.com; s=arc-20160816; b=dsu0mDfHwsYAXr1wTYpH0sluODuDttZst5p1my0Zp/kUHPls8dbKr8LqJydv5jWSRi 0b4/d1k8lLviW87Utxvl1AZrpmfYr7h1ma2x1uf7ZqKXzX0dEf0MyGdZgcktvpvyuzN8 vRNaAz1+0xtQfe0Sosyy7hqSDnlWIR0PZkIIXtpkEZgjdNj2TRRul6JnzPvFQFt4iAzG mT9+446v/IV2l8d5TNUIoA/bIWDpBeLS12LhkRzz8+zijMC5NIRNe7HfScOgzP+T8tLe vcieMLjXwPCVrXLyKvVMBIMb3s5dIRSo6JmPV8LF/YpTVAnSpcXKH+hIqbWelQ3pJUNF hGaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=NqDvAjSsLSimLcFmAjOdfLypVhA8Qg70u4hgiCmTSw4=; b=C2dhOygABZYJoKAMB88kTKVu4B8syTNs5xeYFL6Zdu+pQ4IbDCK8KsNtG4ckXp0S/M JnayhBBdInJReFfkSWUzlQhr+GrBNqRXq7/3MbqGEvUWKCz20YbcS//TCWv+cVCyrC2D iU2pLiY/Rol6xDf894LamuJPSFoj4kM3O1asz0GcyIiqo/iuAIoAiIAPHQaXaVP2gHUg 9Al+ggSj+bUeQtciq+MZboldCEjUFYnmsnhrDfnd5VV3MHrUrjdHIPngniSELjEAHOtm lbkK/9VVlO0JA+0XvZZEfunYDVcMotyb4nHQvemj4cY7vH3ZccIbte/SwzwaqBN+5nJi Ra6w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="x/ESSped"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id h13si507827ybq.232.2021.01.05.09.45.10 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 05 Jan 2021 09:45:10 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="x/ESSped"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:49476 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kwqOL-0002C7-QS for patch@linaro.org; Tue, 05 Jan 2021 12:45:09 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:51274) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kwq0H-0002Is-4V for qemu-devel@nongnu.org; Tue, 05 Jan 2021 12:20:17 -0500 Received: from mail-pf1-x436.google.com ([2607:f8b0:4864:20::436]:45543) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kwq07-0006RI-A3 for qemu-devel@nongnu.org; Tue, 05 Jan 2021 12:20:12 -0500 Received: by mail-pf1-x436.google.com with SMTP id q22so101296pfk.12 for ; Tue, 05 Jan 2021 09:20:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NqDvAjSsLSimLcFmAjOdfLypVhA8Qg70u4hgiCmTSw4=; b=x/ESSpedXBeFlnRFsPIKceBYtW9mD5NB57qPwmlqel24EloiJsL9LBAaaxslOKF7/k m8ETtSHJqF55iMCeFbx5uSrmOCsdH1lS+AgBws4JYEZTSTjTYpcmMFFLdoI4DB0sJhGd wbpoPjUF6gzCtoqNrHb4lOZSnJd1kPw0ajNTyJocxBVtZfXs+HZqrAKB16o1J7uQZeD+ 8AxR9pFh2LkI0f2UYvehpfRMC7nvQ5qFXBc1IeLo+HSFXrmuCgDx0sLOCamFl/PNvkBr kMZE1zsExc0dz7gvCNUwjz3JFxV52rNtPlbPnJI1a0Ha+6yazlb7xZS54Y4KQUMtCamQ C6LA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NqDvAjSsLSimLcFmAjOdfLypVhA8Qg70u4hgiCmTSw4=; b=N0byVBZZce/dmBJY5EAKResj2o984SNbkcKqtZhUy+3y4PCd3/vHNnpLy1Xo6Fr9UO JSa8YHTxAmNyhY2ocDHTb84bn+vo7pUhMaYF1Xzn2YbqMZcvzaheEwsirl8Y8dKxiVid hac849ns1jQOADkgTVFePH+uJu2fYHM5GPnNRjOew3/+LIznE0SUvYkcL2phUMTIe42k P0ny7KN6EwKaTIeFpxD8fGWFAnDXdphp0VOZjx9aLIfQKuTjZpg78ZumELaY2NZH7Oja illDMs3R2lOXMsgX2+5L3RroAjD1A7WbkkivBXcptM6f35d7K6Gy1r6pCiIZ1HoD3SfL M9uA== X-Gm-Message-State: AOAM5322sjAVJ3NWfrxkBT4nIAq7IPvcThxs8G22WEF8gyM9fDZsGCMP xJhWFJ9/p6ZZ0M3HsW32g8ksq128UNy1Gw== X-Received: by 2002:a63:4d59:: with SMTP id n25mr362531pgl.122.1609867204698; Tue, 05 Jan 2021 09:20:04 -0800 (PST) Received: from localhost.localdomain (rrcs-173-197-107-21.west.biz.rr.com. [173.197.107.21]) by smtp.gmail.com with ESMTPSA id u12sm2010pfh.98.2021.01.05.09.20.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Jan 2021 09:20:04 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v5 04/43] util: Specialize flush_idcache_range for aarch64 Date: Tue, 5 Jan 2021 07:19:11 -1000 Message-Id: <20210105171950.415486-5-richard.henderson@linaro.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210105171950.415486-1-richard.henderson@linaro.org> References: <20210105171950.415486-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::436; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x436.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Joelle van Dyne Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" For darwin, the CTR_EL0 register is not accessible, but there are system routines that we can use. For other hosts, copy the single pointer implementation from libgcc and modify it to support the double pointer interface we require. This halves the number of cache operations required when split-rwx is enabled. Reviewed-by: Joelle van Dyne Signed-off-by: Richard Henderson --- util/cacheflush.c | 69 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) -- 2.25.1 diff --git a/util/cacheflush.c b/util/cacheflush.c index 92805efe49..6a20723902 100644 --- a/util/cacheflush.c +++ b/util/cacheflush.c @@ -7,12 +7,81 @@ #include "qemu/osdep.h" #include "qemu/cacheflush.h" +#include "qemu/bitops.h" #if defined(__i386__) || defined(__x86_64__) || defined(__s390__) /* Caches are coherent and do not require flushing; symbol inline. */ +#elif defined(__aarch64__) + +#ifdef CONFIG_DARWIN +/* Apple does not expose CTR_EL0, so we must use system interfaces. */ +extern void sys_icache_invalidate(void *start, size_t len); +extern void sys_dcache_flush(void *start, size_t len); +void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len) +{ + sys_dcache_flush((void *)rw, len); + sys_icache_invalidate((void *)rx, len); +} +#else + +/* + * TODO: unify this with cacheinfo.c. + * We want to save the whole contents of CTR_EL0, so that we + * have more than the linesize, but also IDC and DIC. + */ +static unsigned int save_ctr_el0; +static void __attribute__((constructor)) init_ctr_el0(void) +{ + asm volatile("mrs\t%0, ctr_el0" : "=r"(save_ctr_el0)); +} + +/* + * This is a copy of gcc's __aarch64_sync_cache_range, modified + * to fit this three-operand interface. + */ +void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len) +{ + const unsigned CTR_IDC = 1u << 28; + const unsigned CTR_DIC = 1u << 29; + const unsigned int ctr_el0 = save_ctr_el0; + const uintptr_t icache_lsize = 4 << extract32(ctr_el0, 0, 4); + const uintptr_t dcache_lsize = 4 << extract32(ctr_el0, 16, 4); + uintptr_t p; + + /* + * If CTR_EL0.IDC is enabled, Data cache clean to the Point of Unification + * is not required for instruction to data coherence. + */ + if (!(ctr_el0 & CTR_IDC)) { + /* + * Loop over the address range, clearing one cache line at once. + * Data cache must be flushed to unification first to make sure + * the instruction cache fetches the updated data. + */ + for (p = rw & -dcache_lsize; p < rw + len; p += dcache_lsize) { + asm volatile("dc\tcvau, %0" : : "r" (p) : "memory"); + } + asm volatile("dsb\tish" : : : "memory"); + } + + /* + * If CTR_EL0.DIC is enabled, Instruction cache cleaning to the Point + * of Unification is not required for instruction to data coherence. + */ + if (!(ctr_el0 & CTR_DIC)) { + for (p = rx & -icache_lsize; p < rx + len; p += icache_lsize) { + asm volatile("ic\tivau, %0" : : "r"(p) : "memory"); + } + asm volatile ("dsb\tish" : : : "memory"); + } + + asm volatile("isb" : : : "memory"); +} +#endif /* CONFIG_DARWIN */ + #elif defined(__mips__) #ifdef __OpenBSD__