From patchwork Fri Aug 12 18:08:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 596832 Delivered-To: patch@linaro.org Received: by 2002:a05:7000:b345:0:0:0:0 with SMTP id w5csp1797472maz; Fri, 12 Aug 2022 11:44:02 -0700 (PDT) X-Google-Smtp-Source: AA6agR7zHRiPhBAdmZar20NY/H3UEmoJcjwAGpdJrrzNrReigLdWJbcN6bT8zALz/XnTXfNFjffu X-Received: by 2002:a05:620a:269a:b0:6b5:b76c:11c9 with SMTP id c26-20020a05620a269a00b006b5b76c11c9mr3903863qkp.100.1660329842543; Fri, 12 Aug 2022 11:44:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1660329842; cv=none; d=google.com; s=arc-20160816; b=ozQecc+5TZzurbmpQyRNbNa6vl5XzFVBXN2wJ1gVUFk2nFN9eirM2mf4umFj+LzxY5 /eplOGPAGkEdoa1l2dZ8PTeNA7GiqepWiRY68U+enNntTK991Nz3PK/BaB8Q7Ph/VZSS DB4BAUCV61KbOKUGMMR1WzaG/OoKD5ARQvPtvkr/+Db/Nyre+mcFmh7OYgwX6mbyj6Uc FJr93BUA/oq58yzJTp/714Flv7mPDBR6UAS/DkyT22uHELbv6TudURX+2IfXXLrVjDuz tqoLy04y5b+e0stw24uR0csWgMAyqVxEE3jRZhutQ7BteYD/BNEUywLskGjw2Gzjpjgk 0jCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=ZQFsOm0NhHDSCJf8al04ZyX95hDikXGZM+mnODcfSLg=; b=z8duWD30dtR3Vaz3YWdVlF0K57YM5go1j60c4BDGOGHBmrNZIWZRBVUnphWLC4GTlG AzHnzSEDpknHH1DuWzDvkHTFLMkD/W/o+ACY6dJtWSKZSl27501hGIpNzYJS+4cDXgED DXd+T44jXgsOCPPL+GUWka4TucPRGd3tQB7sivgDaIoYAQmRQz//VsgLdGH3vCxWqa2o fpl/5DXpP8YFSB/YKozGBmy4Xks7HVBitprXB/1axOQzv0mjzDxEBb38cPZwO5ppaIwx SxJP3Y6n27vvFA1VuRPEMQWQbHMbV4GjI39MdnuAm/09ybFPbcZoM4XgGYjYHcIQSiKy th7g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=edpLKjsm; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id n145-20020a372797000000b006b6099bce73si1511392qkn.264.2022.08.12.11.44.02 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 12 Aug 2022 11:44:02 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=edpLKjsm; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:38220 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oMZda-0005TG-3c for patch@linaro.org; Fri, 12 Aug 2022 14:44:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54492) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oMZ5I-0003rl-5D for qemu-devel@nongnu.org; Fri, 12 Aug 2022 14:08:36 -0400 Received: from mail-pl1-x633.google.com ([2607:f8b0:4864:20::633]:42803) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oMZ5G-0006Zm-1C for qemu-devel@nongnu.org; Fri, 12 Aug 2022 14:08:35 -0400 Received: by mail-pl1-x633.google.com with SMTP id w14so1389902plp.9 for ; Fri, 12 Aug 2022 11:08:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=ZQFsOm0NhHDSCJf8al04ZyX95hDikXGZM+mnODcfSLg=; b=edpLKjsmIQbqCIHLZVBaBLdHcx5tTiJwVn1aY763uv96C43fizQ2wcUfhUcPbbaEQe R2KSsu71Vrd8OHRz3iO9lH4XWoBVQyQVDvz5cYz0gKp4wP7LIZrvZB2PCXjfH/ajKjPy jgQ/+8zTlverY2podN5yc3QB36ZYnhHXzz7yK04k3L3Jwt6DCKXmGEghwcABtjqpbme+ wTY1uinURui5yu0Dj/UCxlEQ5kfJQG3Tf6AwdRgHrQigQcNRqpunKWtg9P8PcprcHPmx hhu55dlPJ91Uo2m8zor3wNljkepoYHiq3gRiobP7AhpCNSvXb8NOv0vFE3ySiti3T/AD voAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=ZQFsOm0NhHDSCJf8al04ZyX95hDikXGZM+mnODcfSLg=; b=6CurGiRYfD66uP/Ske5MnxxLfmnG2K39MP6TontOfN665NuefkVMgJptBy0ady5wGZ 37p4odLYj1TXEvIWI+FiSHhNpch5kvNPBBSV/O0Nm6cJ+FeyXhBWcSB5sIJIYYCyQw5w P2ciuPZ9JmFJNM5DfPqvmKGrEhlYs5gDTHXDnMJ6u/S3vHp73HOKJA7uEXkx+6mCDqDD G0tRjmSOvzC95DTBvIRHqZ3sEhrchKhgotvaC7kcEBzZ8FrcGeJW7/fwEG8PUXo/HUov 21PJ6J9/AoCYnSzeKG1vXkFCyc3e+vI2B6PphPr8gfkWhE6f9AZOe4YQCYxh2WAQaW39 HGEA== X-Gm-Message-State: ACgBeo2Dlq6xI/NZAp9o2OlBybpf1WFm9vrs3AkzF0njTP9g7BcFWUcH 3ElR1dqZHxOs40TJSmhIxrv2RYusDP7GQw== X-Received: by 2002:a17:90b:3c4c:b0:1f3:3d62:39e2 with SMTP id pm12-20020a17090b3c4c00b001f33d6239e2mr14906322pjb.88.1660327712621; Fri, 12 Aug 2022 11:08:32 -0700 (PDT) Received: from stoup.. ([2602:ae:154e:e201:a7aa:1d1d:c857:5500]) by smtp.gmail.com with ESMTPSA id h9-20020a056a00000900b0052dee21fecdsm1914761pfk.77.2022.08.12.11.08.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 Aug 2022 11:08:32 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: laurent@vivier.eu, iii@linux.ibm.com, alex.bennee@linaro.org Subject: [PATCH for-7.2 20/21] accel/tcg: Add fast path for translator_ld* Date: Fri, 12 Aug 2022 11:08:05 -0700 Message-Id: <20220812180806.2128593-21-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220812180806.2128593-1-richard.henderson@linaro.org> References: <20220812180806.2128593-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::633; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x633.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Cache the translation from guest to host address, so we may use direct loads when we hit on the primary translation page. Look up the second translation page only once, during translation. This obviates another lookup of the second page within tb_gen_code after translation. Fixes a bug in that plugin_insn_append should be passed the bytes in the original memory order, not bswapped by pieces. Signed-off-by: Richard Henderson --- include/exec/translator.h | 52 ++++++++++++------ accel/tcg/translate-all.c | 22 +++----- accel/tcg/translator.c | 111 +++++++++++++++++++++++++++++++------- 3 files changed, 135 insertions(+), 50 deletions(-) diff --git a/include/exec/translator.h b/include/exec/translator.h index 69db0f5c21..177a001698 100644 --- a/include/exec/translator.h +++ b/include/exec/translator.h @@ -81,13 +81,14 @@ typedef enum DisasJumpType { * Architecture-agnostic disassembly context. */ typedef struct DisasContextBase { - const TranslationBlock *tb; + TranslationBlock *tb; target_ulong pc_first; target_ulong pc_next; DisasJumpType is_jmp; int num_insns; int max_insns; bool singlestep_enabled; + void *host_addr[2]; #ifdef CONFIG_USER_ONLY /* * Guest address of the last byte of the last protected page. @@ -183,24 +184,43 @@ bool translator_use_goto_tb(DisasContextBase *db, target_ulong dest); * the relevant information at translation time. */ -#define GEN_TRANSLATOR_LD(fullname, type, load_fn, swap_fn) \ - type fullname ## _swap(CPUArchState *env, DisasContextBase *dcbase, \ - abi_ptr pc, bool do_swap); \ - static inline type fullname(CPUArchState *env, \ - DisasContextBase *dcbase, abi_ptr pc) \ - { \ - return fullname ## _swap(env, dcbase, pc, false); \ +uint8_t translator_ldub(CPUArchState *env, DisasContextBase *db, abi_ptr pc); +uint16_t translator_lduw(CPUArchState *env, DisasContextBase *db, abi_ptr pc); +uint32_t translator_ldl(CPUArchState *env, DisasContextBase *db, abi_ptr pc); +uint64_t translator_ldq(CPUArchState *env, DisasContextBase *db, abi_ptr pc); + +static inline uint16_t +translator_lduw_swap(CPUArchState *env, DisasContextBase *db, + abi_ptr pc, bool do_swap) +{ + uint16_t ret = translator_lduw(env, db, pc); + if (do_swap) { + ret = bswap16(ret); } + return ret; +} -#define FOR_EACH_TRANSLATOR_LD(F) \ - F(translator_ldub, uint8_t, cpu_ldub_code, /* no swap */) \ - F(translator_lduw, uint16_t, cpu_lduw_code, bswap16) \ - F(translator_ldl, uint32_t, cpu_ldl_code, bswap32) \ - F(translator_ldq, uint64_t, cpu_ldq_code, bswap64) +static inline uint32_t +translator_ldl_swap(CPUArchState *env, DisasContextBase *db, + abi_ptr pc, bool do_swap) +{ + uint32_t ret = translator_ldl(env, db, pc); + if (do_swap) { + ret = bswap32(ret); + } + return ret; +} -FOR_EACH_TRANSLATOR_LD(GEN_TRANSLATOR_LD) - -#undef GEN_TRANSLATOR_LD +static inline uint64_t +translator_ldq_swap(CPUArchState *env, DisasContextBase *db, + abi_ptr pc, bool do_swap) +{ + uint64_t ret = translator_ldq_swap(env, db, pc, false); + if (do_swap) { + ret = bswap64(ret); + } + return ret; +} /* * Return whether addr is on the same page as where disassembly started. diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index d52097ab2d..299b068f9c 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -1333,8 +1333,6 @@ TranslationBlock *tb_gen_code(CPUState *cpu, { CPUArchState *env = cpu->env_ptr; TranslationBlock *tb, *existing_tb; - tb_page_addr_t phys_page2; - target_ulong virt_page2; tcg_insn_unit *gen_code_buf; int gen_code_size, search_size, max_insns; #ifdef CONFIG_PROFILER @@ -1374,6 +1372,8 @@ TranslationBlock *tb_gen_code(CPUState *cpu, tb->flags = flags; tb->cflags = cflags; tb->trace_vcpu_dstate = *cpu->trace_dstate; + tb->page_addr[0] = phys_pc; + tb->page_addr[1] = -1; tcg_ctx->tb_cflags = cflags; tb_overflow: @@ -1567,13 +1567,11 @@ TranslationBlock *tb_gen_code(CPUState *cpu, } /* - * If the TB is not associated with a physical RAM page then - * it must be a temporary one-insn TB, and we have nothing to do - * except fill in the page_addr[] fields. Return early before - * attempting to link to other TBs or add to the lookup table. + * If the TB is not associated with a physical RAM page then it must be + * a temporary one-insn TB, and we have nothing left to do. Return early + * before attempting to link to other TBs or add to the lookup table. */ - if (phys_pc == -1) { - tb->page_addr[0] = tb->page_addr[1] = -1; + if (tb->page_addr[0] == -1) { return tb; } @@ -1584,17 +1582,11 @@ TranslationBlock *tb_gen_code(CPUState *cpu, */ tcg_tb_insert(tb); - /* check next page if needed */ - virt_page2 = (pc + tb->size - 1) & TARGET_PAGE_MASK; - phys_page2 = -1; - if ((pc & TARGET_PAGE_MASK) != virt_page2) { - phys_page2 = get_page_addr_code(env, virt_page2); - } /* * No explicit memory barrier is required -- tb_link_page() makes the * TB visible in a consistent state. */ - existing_tb = tb_link_page(tb, phys_pc, phys_page2); + existing_tb = tb_link_page(tb, tb->page_addr[0], tb->page_addr[1]); /* if the TB already exists, discard what we just translated */ if (unlikely(existing_tb != tb)) { uintptr_t orig_aligned = (uintptr_t)gen_code_buf; diff --git a/accel/tcg/translator.c b/accel/tcg/translator.c index 3eef30d93a..a693c17259 100644 --- a/accel/tcg/translator.c +++ b/accel/tcg/translator.c @@ -66,6 +66,8 @@ void translator_loop(CPUState *cpu, TranslationBlock *tb, int max_insns, db->num_insns = 0; db->max_insns = max_insns; db->singlestep_enabled = cflags & CF_SINGLE_STEP; + db->host_addr[0] = host_pc; + db->host_addr[1] = NULL; translator_page_protect(db, db->pc_next); ops->init_disas_context(db, cpu); @@ -151,31 +153,102 @@ void translator_loop(CPUState *cpu, TranslationBlock *tb, int max_insns, #endif } -static inline void translator_maybe_page_protect(DisasContextBase *dcbase, - target_ulong pc, size_t len) +static void *translator_access(CPUArchState *env, DisasContextBase *db, + target_ulong pc, size_t len) { + void *host; + target_ulong base; + TranslationBlock *tb; + #ifdef CONFIG_USER_ONLY target_ulong end = pc + len - 1; - - if (end > dcbase->page_protect_end) { - translator_page_protect(dcbase, end); + if (end > db->page_protect_end) { + translator_page_protect(db, end); } #endif -} -#define GEN_TRANSLATOR_LD(fullname, type, load_fn, swap_fn) \ - type fullname ## _swap(CPUArchState *env, DisasContextBase *dcbase, \ - abi_ptr pc, bool do_swap) \ - { \ - translator_maybe_page_protect(dcbase, pc, sizeof(type)); \ - type ret = load_fn(env, pc); \ - if (do_swap) { \ - ret = swap_fn(ret); \ - } \ - plugin_insn_append(pc, &ret, sizeof(ret)); \ - return ret; \ + tb = db->tb; + if (unlikely(tb->page_addr[0] == -1)) { + /* Use slow path if first page is MMIO. */ + return NULL; + } else if (likely(is_same_page(db, pc + len - 1))) { + host = db->host_addr[0]; + base = db->pc_first; + } else if (is_same_page(db, pc)) { + /* Use slow path when crossing pages. */ + return NULL; + } else { + host = db->host_addr[1]; + base = TARGET_PAGE_ALIGN(db->pc_first); + if (host == NULL) { + tb->page_addr[1] = + get_page_addr_code_hostp(env, base, false, + &db->host_addr[1]); + /* We cannot handle MMIO as second page. */ + assert(tb->page_addr[1] != -1); + host = db->host_addr[1]; + } } -FOR_EACH_TRANSLATOR_LD(GEN_TRANSLATOR_LD) + tcg_debug_assert(pc >= base); + return host + (pc - base); +} -#undef GEN_TRANSLATOR_LD +uint8_t translator_ldub(CPUArchState *env, DisasContextBase *db, abi_ptr pc) +{ + uint8_t ret; + void *p = translator_access(env, db, pc, sizeof(ret)); + + if (p) { + plugin_insn_append(pc, p, sizeof(ret)); + return ldub_p(p); + } + ret = cpu_ldub_code(env, pc); + plugin_insn_append(pc, &ret, sizeof(ret)); + return ret; +} + +uint16_t translator_lduw(CPUArchState *env, DisasContextBase *db, abi_ptr pc) +{ + uint16_t ret, plug; + void *p = translator_access(env, db, pc, sizeof(ret)); + + if (p) { + plugin_insn_append(pc, p, sizeof(ret)); + return lduw_p(p); + } + ret = cpu_lduw_code(env, pc); + plug = tswap16(ret); + plugin_insn_append(pc, &plug, sizeof(ret)); + return ret; +} + +uint32_t translator_ldl(CPUArchState *env, DisasContextBase *db, abi_ptr pc) +{ + uint32_t ret, plug; + void *p = translator_access(env, db, pc, sizeof(ret)); + + if (p) { + plugin_insn_append(pc, p, sizeof(ret)); + return ldl_p(p); + } + ret = cpu_ldl_code(env, pc); + plug = tswap32(ret); + plugin_insn_append(pc, &plug, sizeof(ret)); + return ret; +} + +uint64_t translator_ldq(CPUArchState *env, DisasContextBase *db, abi_ptr pc) +{ + uint64_t ret, plug; + void *p = translator_access(env, db, pc, sizeof(ret)); + + if (p) { + plugin_insn_append(pc, p, sizeof(ret)); + return ldq_p(p); + } + ret = cpu_ldq_code(env, pc); + plug = tswap64(ret); + plugin_insn_append(pc, &plug, sizeof(ret)); + return ret; +}