From patchwork Wed May 3 07:22:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 678785 Delivered-To: patch@linaro.org Received: by 2002:a5d:4a41:0:0:0:0:0 with SMTP id v1csp916849wrs; Wed, 3 May 2023 00:50:13 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5uc9obxLyLEOOGeAzU0cI5AcGxsCx0tHaDuqaKwPj7aX7rALGIQAUIB/Tm+4hsTTtSARU8 X-Received: by 2002:ad4:5f0c:0:b0:5fe:dffc:fefa with SMTP id fo12-20020ad45f0c000000b005fedffcfefamr8851696qvb.41.1683100213409; Wed, 03 May 2023 00:50:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683100213; cv=none; d=google.com; s=arc-20160816; b=bpiiFTTNUeLDIYHwLPiczUZv9elfwswuQ6hWt2om9ROymBYFoEY4ZathkWmp9hPKQa WTtPvwcjPo8iOF/K2Spk7xsEBOw6mzZrMU0WNoPIhQZspaqLdJzIeHEsnv9SKDKiWoLC YOi6rg29SjwgAKxv9qlBcjBGcCP3G2Nft7AgaZaEHyUOye/vcT23ohp3GF4D0h4sQuHb ENiEiubXIsuwflpD2uGOYKsiZoQvpkXyMaI7aF2ql0BSAb0Yjai71siaXw/ZoMLpKAQe yvrnGEEYFVYYAyF9mDoN0n6nbmZebpUxZ1IuL0JFW+m7BPW0iQB9VLUaMCqMmdTf3I2A UXNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=N7rypmxHlO/iT+tYBIrpiiKKuEYMdJljO+I5NLCaqnQ=; b=Hr5Gk4GToi3mjj+bghb//8GlmFuRdwv1VR+ewb2asLme8FgJNUrhy0fsYlmto+PqmY xPoRti8syS/g8Q90hVWk0VpK+FgKm7zgb823X7k17J98DfnhIWxrFKwNjqAWgxAr2A7s G7ZpGyG7wG5gGfozBI2vgWO3hUT3x70bEPmntyGN1Y+6RxAgCIbYs/7gTkeFa5Ko6gOP wR96tNqEjPjEOgk4wdbr8zrh8jV+U2PvP/dWkqPr89+yydvSfLrZLPlpPzMv6Jogytkr vcTI40Fil4puXC1P58Ots7uRevTKlRT+KOIVRRTDwhVtSUIIkeyOIDsHURIMXsEwN6z1 J1zA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=rcsGClPe; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id fq2-20020a056214258200b005f15c06c800si6085474qvb.286.2023.05.03.00.50.13 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 03 May 2023 00:50:13 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=rcsGClPe; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pu6qI-0002S7-Vi; Wed, 03 May 2023 03:24:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pu6q6-0001ks-97 for qemu-devel@nongnu.org; Wed, 03 May 2023 03:23:50 -0400 Received: from mail-wm1-x32d.google.com ([2a00:1450:4864:20::32d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pu6q1-0001fI-Oa for qemu-devel@nongnu.org; Wed, 03 May 2023 03:23:49 -0400 Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-3f315735514so11240855e9.1 for ; Wed, 03 May 2023 00:23:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1683098624; x=1685690624; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=N7rypmxHlO/iT+tYBIrpiiKKuEYMdJljO+I5NLCaqnQ=; b=rcsGClPeIp0GF2hiik7gjGxAprrYKCDRmcJ1aCfP3Yjzjb3xW5cUs+hBwUc2XLKLmo DhY6btGMGSlcl8zdOBN50NpA/xuJuzBT8g7RtI6WJF4cXiJaVvYblIxHFnxK0EzoSCHb JaI82qt6KrBfpajgN3L60SDd14nT4uQbMK0VOnfpYVPzKfC7kTECv2ni2L8XDi5F0qX3 aMzyd1at7cZFvHJhOlJYi4tgomesB7lsDpguA8F8yPtBKKSEYZqB82MFhW1dLBKgx0PG mMJCikgF03Eo0TXlCxoDWjURf1Iq86FlEjt90Ub95khPIftlIxOovMlQAVKdLl2wJUtS Dyog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683098624; x=1685690624; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=N7rypmxHlO/iT+tYBIrpiiKKuEYMdJljO+I5NLCaqnQ=; b=OVeWf2MwkiU3PUVKMaN4XoJ+R0tCXlcLcc0sk4QwV3YwTQP4aA70VgwnWuOLTw5klp es2iC44VJUI6pcPCmqR6Tno3U78TE8t4eMTyjgUhz36WNB9hJGbnKqTAk9NRHGCcY1++ oWt0Nm2mW+0kSs/VNNSiYXN66nRzW8PKZxrU491mzSpO5dwM74P1Wa54yPqlbEMBmF9/ +AbyXDPBkwunmLNv2Mlw7Ev+lS39kVGxZl0GzpvrJO2X736gLb939Dbx1sJjr9O5wvXC r+SVZerHSyBIyaPwuZa8ymsms+ycYMQnS9VNHgV8HefjYlVnbk9hQudmArBttZZcawA9 eLpA== X-Gm-Message-State: AC+VfDzGug2prH8/N8GCUuo5jPvIkXpcKTBoWmhRXF4Bb34+vXWrYgeb aQeXZszck3BmhOL37+XZe+abLh35XF+5fWQRO+67FA== X-Received: by 2002:adf:ea82:0:b0:306:2ff1:5227 with SMTP id s2-20020adfea82000000b003062ff15227mr724633wrm.23.1683098624026; Wed, 03 May 2023 00:23:44 -0700 (PDT) Received: from stoup.Home ([2a02:c7c:74db:8d00:c01d:9d74:b630:9087]) by smtp.gmail.com with ESMTPSA id b14-20020a05600010ce00b0030627f58325sm9586745wrx.25.2023.05.03.00.23.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 May 2023 00:23:43 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: ale@rev.ng, philmd@linaro.org, marcel.apfelbaum@gmail.com, wangyanan55@huawei.com, anjo@rev.ng Subject: [PATCH 12/84] tcg: Remove TCGv from tcg_gen_qemu_{ld,st}_* Date: Wed, 3 May 2023 08:22:19 +0100 Message-Id: <20230503072331.1747057-13-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230503072331.1747057-1-richard.henderson@linaro.org> References: <20230503072331.1747057-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::32d; envelope-from=richard.henderson@linaro.org; helo=mail-wm1-x32d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Expand from TCGv to TCGTemp inline in the translators, and validate that the size matches tcg_ctx->addr_type. These inlines will eventually be seen only by target-specific code. Signed-off-by: Richard Henderson Reviewed-by: Alex Bennée --- include/tcg/tcg-op.h | 50 ++++++- tcg/tcg-op-ldst.c | 343 ++++++++++++++++++++++++++----------------- 2 files changed, 251 insertions(+), 142 deletions(-) diff --git a/include/tcg/tcg-op.h b/include/tcg/tcg-op.h index de3b70aa84..e556450ba9 100644 --- a/include/tcg/tcg-op.h +++ b/include/tcg/tcg-op.h @@ -803,22 +803,60 @@ static inline void tcg_gen_plugin_cb_end(void) #define tcg_temp_new() tcg_temp_new_i32() #define tcg_global_mem_new tcg_global_mem_new_i32 #define tcg_temp_free tcg_temp_free_i32 +#define tcgv_tl_temp tcgv_i32_temp #define tcg_gen_qemu_ld_tl tcg_gen_qemu_ld_i32 #define tcg_gen_qemu_st_tl tcg_gen_qemu_st_i32 #else #define tcg_temp_new() tcg_temp_new_i64() #define tcg_global_mem_new tcg_global_mem_new_i64 #define tcg_temp_free tcg_temp_free_i64 +#define tcgv_tl_temp tcgv_i64_temp #define tcg_gen_qemu_ld_tl tcg_gen_qemu_ld_i64 #define tcg_gen_qemu_st_tl tcg_gen_qemu_st_i64 #endif -void tcg_gen_qemu_ld_i32(TCGv_i32, TCGv, TCGArg, MemOp); -void tcg_gen_qemu_st_i32(TCGv_i32, TCGv, TCGArg, MemOp); -void tcg_gen_qemu_ld_i64(TCGv_i64, TCGv, TCGArg, MemOp); -void tcg_gen_qemu_st_i64(TCGv_i64, TCGv, TCGArg, MemOp); -void tcg_gen_qemu_ld_i128(TCGv_i128, TCGv, TCGArg, MemOp); -void tcg_gen_qemu_st_i128(TCGv_i128, TCGv, TCGArg, MemOp); +void tcg_gen_qemu_ld_i32_chk(TCGv_i32, TCGTemp *, TCGArg, MemOp, TCGType); +void tcg_gen_qemu_st_i32_chk(TCGv_i32, TCGTemp *, TCGArg, MemOp, TCGType); +void tcg_gen_qemu_ld_i64_chk(TCGv_i64, TCGTemp *, TCGArg, MemOp, TCGType); +void tcg_gen_qemu_st_i64_chk(TCGv_i64, TCGTemp *, TCGArg, MemOp, TCGType); +void tcg_gen_qemu_ld_i128_chk(TCGv_i128, TCGTemp *, TCGArg, MemOp, TCGType); +void tcg_gen_qemu_st_i128_chk(TCGv_i128, TCGTemp *, TCGArg, MemOp, TCGType); + +static inline void +tcg_gen_qemu_ld_i32(TCGv_i32 v, TCGv a, TCGArg i, MemOp m) +{ + tcg_gen_qemu_ld_i32_chk(v, tcgv_tl_temp(a), i, m, TCG_TYPE_TL); +} + +static inline void +tcg_gen_qemu_st_i32(TCGv_i32 v, TCGv a, TCGArg i, MemOp m) +{ + tcg_gen_qemu_st_i32_chk(v, tcgv_tl_temp(a), i, m, TCG_TYPE_TL); +} + +static inline void +tcg_gen_qemu_ld_i64(TCGv_i64 v, TCGv a, TCGArg i, MemOp m) +{ + tcg_gen_qemu_ld_i64_chk(v, tcgv_tl_temp(a), i, m, TCG_TYPE_TL); +} + +static inline void +tcg_gen_qemu_st_i64(TCGv_i64 v, TCGv a, TCGArg i, MemOp m) +{ + tcg_gen_qemu_st_i64_chk(v, tcgv_tl_temp(a), i, m, TCG_TYPE_TL); +} + +static inline void +tcg_gen_qemu_ld_i128(TCGv_i128 v, TCGv a, TCGArg i, MemOp m) +{ + tcg_gen_qemu_ld_i128_chk(v, tcgv_tl_temp(a), i, m, TCG_TYPE_TL); +} + +static inline void +tcg_gen_qemu_st_i128(TCGv_i128 v, TCGv a, TCGArg i, MemOp m) +{ + tcg_gen_qemu_st_i128_chk(v, tcgv_tl_temp(a), i, m, TCG_TYPE_TL); +} void tcg_gen_atomic_cmpxchg_i32(TCGv_i32, TCGv, TCGv_i32, TCGv_i32, TCGArg, MemOp); diff --git a/tcg/tcg-op-ldst.c b/tcg/tcg-op-ldst.c index 0530dcf00a..a94a70e8c4 100644 --- a/tcg/tcg-op-ldst.c +++ b/tcg/tcg-op-ldst.c @@ -68,39 +68,38 @@ static inline MemOp tcg_canonicalize_memop(MemOp op, bool is64, bool st) return op; } -static void gen_ldst_i32(TCGOpcode opc, TCGv_i32 val, TCGv addr, - MemOp memop, TCGArg idx) +static void gen_ldst(TCGOpcode opc, TCGTemp *vl, TCGTemp *vh, + TCGTemp *addr, MemOpIdx oi) { - MemOpIdx oi = make_memop_idx(memop, idx); -#if TARGET_LONG_BITS == 32 - tcg_gen_op3i_i32(opc, val, addr, oi); -#else - if (TCG_TARGET_REG_BITS == 32) { - tcg_gen_op4i_i32(opc, val, TCGV_LOW(addr), TCGV_HIGH(addr), oi); + if (TCG_TARGET_REG_BITS == 64 || tcg_ctx->addr_type == TCG_TYPE_I32) { + if (vh) { + tcg_gen_op4(opc, temp_arg(vl), temp_arg(vh), temp_arg(addr), oi); + } else { + tcg_gen_op3(opc, temp_arg(vl), temp_arg(addr), oi); + } } else { - tcg_gen_op3(opc, tcgv_i32_arg(val), tcgv_i64_arg(addr), oi); + /* See TCGV_LOW/HIGH. */ + TCGTemp *al = addr + HOST_BIG_ENDIAN; + TCGTemp *ah = addr + !HOST_BIG_ENDIAN; + + if (vh) { + tcg_gen_op5(opc, temp_arg(vl), temp_arg(vh), + temp_arg(al), temp_arg(ah), oi); + } else { + tcg_gen_op4(opc, temp_arg(vl), temp_arg(al), temp_arg(ah), oi); + } } -#endif } -static void gen_ldst_i64(TCGOpcode opc, TCGv_i64 val, TCGv addr, - MemOp memop, TCGArg idx) +static void gen_ldst_i64(TCGOpcode opc, TCGv_i64 v, TCGTemp *addr, MemOpIdx oi) { - MemOpIdx oi = make_memop_idx(memop, idx); -#if TARGET_LONG_BITS == 32 if (TCG_TARGET_REG_BITS == 32) { - tcg_gen_op4i_i32(opc, TCGV_LOW(val), TCGV_HIGH(val), addr, oi); + TCGTemp *vl = tcgv_i32_temp(TCGV_LOW(v)); + TCGTemp *vh = tcgv_i32_temp(TCGV_HIGH(v)); + gen_ldst(opc, vl, vh, addr, oi); } else { - tcg_gen_op3(opc, tcgv_i64_arg(val), tcgv_i32_arg(addr), oi); + gen_ldst(opc, tcgv_i64_temp(v), NULL, addr, oi); } -#else - if (TCG_TARGET_REG_BITS == 32) { - tcg_gen_op5i_i32(opc, TCGV_LOW(val), TCGV_HIGH(val), - TCGV_LOW(addr), TCGV_HIGH(addr), oi); - } else { - tcg_gen_op3i_i64(opc, val, addr, oi); - } -#endif } static void tcg_gen_req_mo(TCGBar type) @@ -115,13 +114,17 @@ static void tcg_gen_req_mo(TCGBar type) } /* Only required for loads, where value might overlap addr. */ -static TCGv_i64 plugin_maybe_preserve_addr(TCGv vaddr) +static TCGv_i64 plugin_maybe_preserve_addr(TCGTemp *addr) { #ifdef CONFIG_PLUGIN if (tcg_ctx->plugin_insn != NULL) { /* Save a copy of the vaddr for use after a load. */ TCGv_i64 temp = tcg_temp_ebb_new_i64(); - tcg_gen_extu_tl_i64(temp, vaddr); + if (tcg_ctx->addr_type == TCG_TYPE_I32) { + tcg_gen_extu_i32_i64(temp, temp_tcgv_i32(addr)); + } else { + tcg_gen_mov_i64(temp, temp_tcgv_i64(addr)); + } return temp; } #endif @@ -129,54 +132,55 @@ static TCGv_i64 plugin_maybe_preserve_addr(TCGv vaddr) } static void -plugin_gen_mem_callbacks(TCGv_i64 copy_addr, TCGv orig_addr, MemOpIdx oi, +plugin_gen_mem_callbacks(TCGv_i64 copy_addr, TCGTemp *orig_addr, MemOpIdx oi, enum qemu_plugin_mem_rw rw) { #ifdef CONFIG_PLUGIN if (tcg_ctx->plugin_insn != NULL) { qemu_plugin_meminfo_t info = make_plugin_meminfo(oi, rw); -#if TARGET_LONG_BITS == 64 - if (copy_addr) { + if (tcg_ctx->addr_type == TCG_TYPE_I32) { + if (!copy_addr) { + copy_addr = tcg_temp_ebb_new_i64(); + tcg_gen_extu_i32_i64(copy_addr, temp_tcgv_i32(orig_addr)); + } plugin_gen_empty_mem_callback(copy_addr, info); tcg_temp_free_i64(copy_addr); } else { - plugin_gen_empty_mem_callback(orig_addr, info); + if (copy_addr) { + plugin_gen_empty_mem_callback(copy_addr, info); + tcg_temp_free_i64(copy_addr); + } else { + plugin_gen_empty_mem_callback(temp_tcgv_i64(orig_addr), info); + } } -#else - if (!copy_addr) { - copy_addr = tcg_temp_ebb_new_i64(); - tcg_gen_extu_tl_i64(copy_addr, orig_addr); - } - plugin_gen_empty_mem_callback(copy_addr, info); - tcg_temp_free_i64(copy_addr); -#endif } #endif } -void tcg_gen_qemu_ld_i32(TCGv_i32 val, TCGv addr, TCGArg idx, MemOp memop) +static void tcg_gen_qemu_ld_i32_int(TCGv_i32 val, TCGTemp *addr, + TCGArg idx, MemOp memop) { MemOp orig_memop; - MemOpIdx oi; + MemOpIdx orig_oi, oi; TCGv_i64 copy_addr; tcg_gen_req_mo(TCG_MO_LD_LD | TCG_MO_ST_LD); - memop = tcg_canonicalize_memop(memop, 0, 0); - oi = make_memop_idx(memop, idx); + orig_memop = memop = tcg_canonicalize_memop(memop, 0, 0); + orig_oi = oi = make_memop_idx(memop, idx); - orig_memop = memop; if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) { memop &= ~MO_BSWAP; /* The bswap primitive benefits from zero-extended input. */ if ((memop & MO_SSIZE) == MO_SW) { memop &= ~MO_SIGN; } + oi = make_memop_idx(memop, idx); } copy_addr = plugin_maybe_preserve_addr(addr); - gen_ldst_i32(INDEX_op_qemu_ld_i32, val, addr, memop, idx); - plugin_gen_mem_callbacks(copy_addr, addr, oi, QEMU_PLUGIN_MEM_R); + gen_ldst(INDEX_op_qemu_ld_i32, tcgv_i32_temp(val), NULL, addr, oi); + plugin_gen_mem_callbacks(copy_addr, addr, orig_oi, QEMU_PLUGIN_MEM_R); if ((orig_memop ^ memop) & MO_BSWAP) { switch (orig_memop & MO_SIZE) { @@ -194,14 +198,24 @@ void tcg_gen_qemu_ld_i32(TCGv_i32 val, TCGv addr, TCGArg idx, MemOp memop) } } -void tcg_gen_qemu_st_i32(TCGv_i32 val, TCGv addr, TCGArg idx, MemOp memop) +void tcg_gen_qemu_ld_i32_chk(TCGv_i32 val, TCGTemp *addr, TCGArg idx, + MemOp memop, TCGType addr_type) +{ + tcg_debug_assert(addr_type == tcg_ctx->addr_type); + tcg_debug_assert((memop & MO_SIZE) <= MO_32); + tcg_gen_qemu_ld_i32_int(val, addr, idx, memop); +} + +static void tcg_gen_qemu_st_i32_int(TCGv_i32 val, TCGTemp *addr, + TCGArg idx, MemOp memop) { TCGv_i32 swap = NULL; - MemOpIdx oi; + MemOpIdx orig_oi, oi; + TCGOpcode opc; tcg_gen_req_mo(TCG_MO_LD_ST | TCG_MO_ST_ST); memop = tcg_canonicalize_memop(memop, 0, 1); - oi = make_memop_idx(memop, idx); + orig_oi = oi = make_memop_idx(memop, idx); if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) { swap = tcg_temp_ebb_new_i32(); @@ -217,28 +231,39 @@ void tcg_gen_qemu_st_i32(TCGv_i32 val, TCGv addr, TCGArg idx, MemOp memop) } val = swap; memop &= ~MO_BSWAP; + oi = make_memop_idx(memop, idx); } if (TCG_TARGET_HAS_qemu_st8_i32 && (memop & MO_SIZE) == MO_8) { - gen_ldst_i32(INDEX_op_qemu_st8_i32, val, addr, memop, idx); + opc = INDEX_op_qemu_st8_i32; } else { - gen_ldst_i32(INDEX_op_qemu_st_i32, val, addr, memop, idx); + opc = INDEX_op_qemu_st_i32; } - plugin_gen_mem_callbacks(NULL, addr, oi, QEMU_PLUGIN_MEM_W); + gen_ldst(opc, tcgv_i32_temp(val), NULL, addr, oi); + plugin_gen_mem_callbacks(NULL, addr, orig_oi, QEMU_PLUGIN_MEM_W); if (swap) { tcg_temp_free_i32(swap); } } -void tcg_gen_qemu_ld_i64(TCGv_i64 val, TCGv addr, TCGArg idx, MemOp memop) +void tcg_gen_qemu_st_i32_chk(TCGv_i32 val, TCGTemp *addr, TCGArg idx, + MemOp memop, TCGType addr_type) +{ + tcg_debug_assert(addr_type == tcg_ctx->addr_type); + tcg_debug_assert((memop & MO_SIZE) <= MO_32); + tcg_gen_qemu_st_i32_int(val, addr, idx, memop); +} + +static void tcg_gen_qemu_ld_i64_int(TCGv_i64 val, TCGTemp *addr, + TCGArg idx, MemOp memop) { MemOp orig_memop; - MemOpIdx oi; + MemOpIdx orig_oi, oi; TCGv_i64 copy_addr; if (TCG_TARGET_REG_BITS == 32 && (memop & MO_SIZE) < MO_64) { - tcg_gen_qemu_ld_i32(TCGV_LOW(val), addr, idx, memop); + tcg_gen_qemu_ld_i32_int(TCGV_LOW(val), addr, idx, memop); if (memop & MO_SIGN) { tcg_gen_sari_i32(TCGV_HIGH(val), TCGV_LOW(val), 31); } else { @@ -248,21 +273,21 @@ void tcg_gen_qemu_ld_i64(TCGv_i64 val, TCGv addr, TCGArg idx, MemOp memop) } tcg_gen_req_mo(TCG_MO_LD_LD | TCG_MO_ST_LD); - memop = tcg_canonicalize_memop(memop, 1, 0); - oi = make_memop_idx(memop, idx); + orig_memop = memop = tcg_canonicalize_memop(memop, 1, 0); + orig_oi = oi = make_memop_idx(memop, idx); - orig_memop = memop; if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) { memop &= ~MO_BSWAP; /* The bswap primitive benefits from zero-extended input. */ if ((memop & MO_SIGN) && (memop & MO_SIZE) < MO_64) { memop &= ~MO_SIGN; } + oi = make_memop_idx(memop, idx); } copy_addr = plugin_maybe_preserve_addr(addr); - gen_ldst_i64(INDEX_op_qemu_ld_i64, val, addr, memop, idx); - plugin_gen_mem_callbacks(copy_addr, addr, oi, QEMU_PLUGIN_MEM_R); + gen_ldst_i64(INDEX_op_qemu_ld_i64, val, addr, oi); + plugin_gen_mem_callbacks(copy_addr, addr, orig_oi, QEMU_PLUGIN_MEM_R); if ((orig_memop ^ memop) & MO_BSWAP) { int flags = (orig_memop & MO_SIGN @@ -284,19 +309,28 @@ void tcg_gen_qemu_ld_i64(TCGv_i64 val, TCGv addr, TCGArg idx, MemOp memop) } } -void tcg_gen_qemu_st_i64(TCGv_i64 val, TCGv addr, TCGArg idx, MemOp memop) +void tcg_gen_qemu_ld_i64_chk(TCGv_i64 val, TCGTemp *addr, TCGArg idx, + MemOp memop, TCGType addr_type) +{ + tcg_debug_assert(addr_type == tcg_ctx->addr_type); + tcg_debug_assert((memop & MO_SIZE) <= MO_64); + tcg_gen_qemu_ld_i64_int(val, addr, idx, memop); +} + +static void tcg_gen_qemu_st_i64_int(TCGv_i64 val, TCGTemp *addr, + TCGArg idx, MemOp memop) { TCGv_i64 swap = NULL; - MemOpIdx oi; + MemOpIdx orig_oi, oi; if (TCG_TARGET_REG_BITS == 32 && (memop & MO_SIZE) < MO_64) { - tcg_gen_qemu_st_i32(TCGV_LOW(val), addr, idx, memop); + tcg_gen_qemu_st_i32_int(TCGV_LOW(val), addr, idx, memop); return; } tcg_gen_req_mo(TCG_MO_LD_ST | TCG_MO_ST_ST); memop = tcg_canonicalize_memop(memop, 1, 1); - oi = make_memop_idx(memop, idx); + orig_oi = oi = make_memop_idx(memop, idx); if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) { swap = tcg_temp_ebb_new_i64(); @@ -315,16 +349,25 @@ void tcg_gen_qemu_st_i64(TCGv_i64 val, TCGv addr, TCGArg idx, MemOp memop) } val = swap; memop &= ~MO_BSWAP; + oi = make_memop_idx(memop, idx); } - gen_ldst_i64(INDEX_op_qemu_st_i64, val, addr, memop, idx); - plugin_gen_mem_callbacks(NULL, addr, oi, QEMU_PLUGIN_MEM_W); + gen_ldst_i64(INDEX_op_qemu_st_i64, val, addr, oi); + plugin_gen_mem_callbacks(NULL, addr, orig_oi, QEMU_PLUGIN_MEM_W); if (swap) { tcg_temp_free_i64(swap); } } +void tcg_gen_qemu_st_i64_chk(TCGv_i64 val, TCGTemp *addr, TCGArg idx, + MemOp memop, TCGType addr_type) +{ + tcg_debug_assert(addr_type == tcg_ctx->addr_type); + tcg_debug_assert((memop & MO_SIZE) <= MO_64); + tcg_gen_qemu_st_i64_int(val, addr, idx, memop); +} + /* * Return true if @mop, without knowledge of the pointer alignment, * does not require 16-byte atomicity, and it would be adventagous @@ -371,9 +414,6 @@ static void canonicalize_memop_i128_as_i64(MemOp ret[2], MemOp orig) { MemOp mop_1 = orig, mop_2; - tcg_debug_assert((orig & MO_SIZE) == MO_128); - tcg_debug_assert((orig & MO_SIGN) == 0); - /* Reduce the size to 64-bit. */ mop_1 = (mop_1 & ~MO_SIZE) | MO_64; @@ -414,12 +454,6 @@ static void canonicalize_memop_i128_as_i64(MemOp ret[2], MemOp orig) ret[1] = mop_2; } -#if TARGET_LONG_BITS == 64 -#define tcg_temp_ebb_new tcg_temp_ebb_new_i64 -#else -#define tcg_temp_ebb_new tcg_temp_ebb_new_i32 -#endif - static TCGv_i64 maybe_extend_addr64(TCGv addr) { #if TARGET_LONG_BITS == 32 @@ -438,39 +472,32 @@ static void maybe_free_addr64(TCGv_i64 a64) #endif } -void tcg_gen_qemu_ld_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop) +static void tcg_gen_qemu_ld_i128_int(TCGv_i128 val, TCGTemp *addr, + TCGArg idx, MemOp memop) { - const MemOpIdx oi = make_memop_idx(memop, idx); - - tcg_debug_assert((memop & MO_SIZE) == MO_128); - tcg_debug_assert((memop & MO_SIGN) == 0); + const MemOpIdx orig_oi = make_memop_idx(memop, idx); + TCGv_i64 ext_addr = NULL; tcg_gen_req_mo(TCG_MO_LD_LD | TCG_MO_ST_LD); /* TODO: For now, force 32-bit hosts to use the helper. */ if (TCG_TARGET_HAS_qemu_ldst_i128 && TCG_TARGET_REG_BITS == 64) { TCGv_i64 lo, hi; - TCGArg addr_arg; - MemOpIdx adj_oi; bool need_bswap = false; + MemOpIdx oi = orig_oi; if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) { lo = TCGV128_HIGH(val); hi = TCGV128_LOW(val); - adj_oi = make_memop_idx(memop & ~MO_BSWAP, idx); + oi = make_memop_idx(memop & ~MO_BSWAP, idx); need_bswap = true; } else { lo = TCGV128_LOW(val); hi = TCGV128_HIGH(val); - adj_oi = oi; } -#if TARGET_LONG_BITS == 32 - addr_arg = tcgv_i32_arg(addr); -#else - addr_arg = tcgv_i64_arg(addr); -#endif - tcg_gen_op4ii_i64(INDEX_op_qemu_ld_i128, lo, hi, addr_arg, adj_oi); + gen_ldst(INDEX_op_qemu_ld_i128, tcgv_i64_temp(lo), + tcgv_i64_temp(hi), addr, oi); if (need_bswap) { tcg_gen_bswap64_i64(lo, lo); @@ -478,10 +505,13 @@ void tcg_gen_qemu_ld_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop) } } else if (use_two_i64_for_i128(memop)) { MemOp mop[2]; - TCGv addr_p8; + TCGTemp *addr_p8; TCGv_i64 x, y; + MemOpIdx oi; + bool need_bswap; canonicalize_memop_i128_as_i64(mop, memop); + need_bswap = (mop[0] ^ memop) & MO_BSWAP; /* * Since there are no global TCGv_i128, there is no visible state @@ -496,35 +526,56 @@ void tcg_gen_qemu_ld_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop) y = TCGV128_LOW(val); } - gen_ldst_i64(INDEX_op_qemu_ld_i64, x, addr, mop[0], idx); + oi = make_memop_idx(mop[0], idx); + gen_ldst_i64(INDEX_op_qemu_ld_i64, x, addr, oi); - if ((mop[0] ^ memop) & MO_BSWAP) { + if (need_bswap) { tcg_gen_bswap64_i64(x, x); } - addr_p8 = tcg_temp_ebb_new(); - tcg_gen_addi_tl(addr_p8, addr, 8); - gen_ldst_i64(INDEX_op_qemu_ld_i64, y, addr_p8, mop[1], idx); - tcg_temp_free(addr_p8); + if (tcg_ctx->addr_type == TCG_TYPE_I32) { + TCGv_i32 t = tcg_temp_ebb_new_i32(); + tcg_gen_addi_i32(t, temp_tcgv_i32(addr), 8); + addr_p8 = tcgv_i32_temp(t); + } else { + TCGv_i64 t = tcg_temp_ebb_new_i64(); + tcg_gen_addi_i64(t, temp_tcgv_i64(addr), 8); + addr_p8 = tcgv_i64_temp(t); + } - if ((mop[0] ^ memop) & MO_BSWAP) { + gen_ldst_i64(INDEX_op_qemu_ld_i64, y, addr_p8, oi); + tcg_temp_free_internal(addr_p8); + + if (need_bswap) { tcg_gen_bswap64_i64(y, y); } } else { - TCGv_i64 a64 = maybe_extend_addr64(addr); - gen_helper_ld_i128(val, cpu_env, a64, tcg_constant_i32(oi)); - maybe_free_addr64(a64); + if (tcg_ctx->addr_type == TCG_TYPE_I32) { + ext_addr = tcg_temp_ebb_new_i64(); + tcg_gen_extu_i32_i64(ext_addr, temp_tcgv_i32(addr)); + addr = tcgv_i64_temp(ext_addr); + } + gen_helper_ld_i128(val, cpu_env, temp_tcgv_i64(addr), + tcg_constant_i32(orig_oi)); } - plugin_gen_mem_callbacks(NULL, addr, oi, QEMU_PLUGIN_MEM_R); + plugin_gen_mem_callbacks(ext_addr, addr, orig_oi, QEMU_PLUGIN_MEM_R); } -void tcg_gen_qemu_st_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop) +void tcg_gen_qemu_ld_i128_chk(TCGv_i128 val, TCGTemp *addr, TCGArg idx, + MemOp memop, TCGType addr_type) { - const MemOpIdx oi = make_memop_idx(memop, idx); - + tcg_debug_assert(addr_type == tcg_ctx->addr_type); tcg_debug_assert((memop & MO_SIZE) == MO_128); tcg_debug_assert((memop & MO_SIGN) == 0); + tcg_gen_qemu_ld_i128_int(val, addr, idx, memop); +} + +static void tcg_gen_qemu_st_i128_int(TCGv_i128 val, TCGTemp *addr, + TCGArg idx, MemOp memop) +{ + const MemOpIdx orig_oi = make_memop_idx(memop, idx); + TCGv_i64 ext_addr = NULL; tcg_gen_req_mo(TCG_MO_ST_LD | TCG_MO_ST_ST); @@ -532,29 +583,23 @@ void tcg_gen_qemu_st_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop) if (TCG_TARGET_HAS_qemu_ldst_i128 && TCG_TARGET_REG_BITS == 64) { TCGv_i64 lo, hi; - TCGArg addr_arg; - MemOpIdx adj_oi; + MemOpIdx oi = orig_oi; bool need_bswap = false; if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) { - lo = tcg_temp_new_i64(); - hi = tcg_temp_new_i64(); + lo = tcg_temp_ebb_new_i64(); + hi = tcg_temp_ebb_new_i64(); tcg_gen_bswap64_i64(lo, TCGV128_HIGH(val)); tcg_gen_bswap64_i64(hi, TCGV128_LOW(val)); - adj_oi = make_memop_idx(memop & ~MO_BSWAP, idx); + oi = make_memop_idx(memop & ~MO_BSWAP, idx); need_bswap = true; } else { lo = TCGV128_LOW(val); hi = TCGV128_HIGH(val); - adj_oi = oi; } -#if TARGET_LONG_BITS == 32 - addr_arg = tcgv_i32_arg(addr); -#else - addr_arg = tcgv_i64_arg(addr); -#endif - tcg_gen_op4ii_i64(INDEX_op_qemu_st_i128, lo, hi, addr_arg, adj_oi); + gen_ldst(INDEX_op_qemu_st_i128, tcgv_i64_temp(lo), + tcgv_i64_temp(hi), addr, oi); if (need_bswap) { tcg_temp_free_i64(lo); @@ -562,8 +607,8 @@ void tcg_gen_qemu_st_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop) } } else if (use_two_i64_for_i128(memop)) { MemOp mop[2]; - TCGv addr_p8; - TCGv_i64 x, y; + TCGTemp *addr_p8; + TCGv_i64 x, y, b = NULL; canonicalize_memop_i128_as_i64(mop, memop); @@ -575,29 +620,55 @@ void tcg_gen_qemu_st_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop) y = TCGV128_LOW(val); } - addr_p8 = tcg_temp_ebb_new(); if ((mop[0] ^ memop) & MO_BSWAP) { - TCGv_i64 t = tcg_temp_ebb_new_i64(); - - tcg_gen_bswap64_i64(t, x); - gen_ldst_i64(INDEX_op_qemu_st_i64, t, addr, mop[0], idx); - tcg_gen_bswap64_i64(t, y); - tcg_gen_addi_tl(addr_p8, addr, 8); - gen_ldst_i64(INDEX_op_qemu_st_i64, t, addr_p8, mop[1], idx); - tcg_temp_free_i64(t); - } else { - gen_ldst_i64(INDEX_op_qemu_st_i64, x, addr, mop[0], idx); - tcg_gen_addi_tl(addr_p8, addr, 8); - gen_ldst_i64(INDEX_op_qemu_st_i64, y, addr_p8, mop[1], idx); + b = tcg_temp_ebb_new_i64(); + tcg_gen_bswap64_i64(b, x); + x = b; } - tcg_temp_free(addr_p8); + gen_ldst_i64(INDEX_op_qemu_st_i64, x, addr, + make_memop_idx(mop[0], idx)); + + if (tcg_ctx->addr_type == TCG_TYPE_I32) { + TCGv_i32 t = tcg_temp_ebb_new_i32(); + tcg_gen_addi_i32(t, temp_tcgv_i32(addr), 8); + addr_p8 = tcgv_i32_temp(t); + } else { + TCGv_i64 t = tcg_temp_ebb_new_i64(); + tcg_gen_addi_i64(t, temp_tcgv_i64(addr), 8); + addr_p8 = tcgv_i64_temp(t); + } + + if (b) { + tcg_gen_bswap64_i64(b, y); + y = b; + } + gen_ldst_i64(INDEX_op_qemu_st_i64, y, addr_p8, + make_memop_idx(mop[1], idx)); + + if (b) { + tcg_temp_free_i64(b); + } + tcg_temp_free_internal(addr_p8); } else { - TCGv_i64 a64 = maybe_extend_addr64(addr); - gen_helper_st_i128(cpu_env, a64, val, tcg_constant_i32(oi)); - maybe_free_addr64(a64); + if (tcg_ctx->addr_type == TCG_TYPE_I32) { + ext_addr = tcg_temp_ebb_new_i64(); + tcg_gen_extu_i32_i64(ext_addr, temp_tcgv_i32(addr)); + addr = tcgv_i64_temp(ext_addr); + } + gen_helper_st_i128(cpu_env, temp_tcgv_i64(addr), val, + tcg_constant_i32(orig_oi)); } - plugin_gen_mem_callbacks(NULL, addr, oi, QEMU_PLUGIN_MEM_W); + plugin_gen_mem_callbacks(ext_addr, addr, orig_oi, QEMU_PLUGIN_MEM_W); +} + +void tcg_gen_qemu_st_i128_chk(TCGv_i128 val, TCGTemp *addr, TCGArg idx, + MemOp memop, TCGType addr_type) +{ + tcg_debug_assert(addr_type == tcg_ctx->addr_type); + tcg_debug_assert((memop & MO_SIZE) == MO_128); + tcg_debug_assert((memop & MO_SIGN) == 0); + tcg_gen_qemu_st_i128_int(val, addr, idx, memop); } static void tcg_gen_ext_i32(TCGv_i32 ret, TCGv_i32 val, MemOp opc)