From patchwork Mon May 15 14:32:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681934 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1148138wrr; Mon, 15 May 2023 07:34:41 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5znZIh+i3ly9bFRKzZgaw0lF+hm2udbTsX+VpYtwZmha04D1QdaZqYpWN3RP/APZYnUq4c X-Received: by 2002:a05:622a:120b:b0:3e6:55b2:35e with SMTP id y11-20020a05622a120b00b003e655b2035emr52517467qtx.26.1684161280923; Mon, 15 May 2023 07:34:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161280; cv=none; d=google.com; s=arc-20160816; b=lKGVM3ckZ6NwOkeJBmwziTlUATgq5hu9KhpLDFET+kv4T/EOwHwwBl0R/MpO4EB6LE pvLLvaK1vRkNsIMWaf+4bVDAi17EZDHVZxtHVrO+ngazFSr+jfYUrr1K7byDinA8gIPx oXyMn5xyegbG1mKMRF7jhEAcZFeivCWN6xkI7O8idV0nFl33Ih9Hq2iJDCY9P4Szkxf/ Bz9IrEIwtqUws1rbn2FzCyVBLGLKyvVMm16Y4vkky66BVIggzTW0mXqURQCQsF5ZnN3y qDibbyxCfUHUKAIylUSpNWfAa3Xf7shCUdPY0/AvFd9v1svNGQTpw4LJqdwo6FqHdnhr xVCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=Bi+7zXX3SkE4OXmw+QtVLrvmyt1/y9mkItKgRCmaxGs=; b=bFiOTyRzeLeyMQRiujZui1kZkEGCsxM/nmPpbOIXlY8V719AnhqccIh5Ni6tBA9g35 35pajktGFkYtCf/FqCzOZUgA+X7pQBpdEVPIt3Tc7tySa88ynyX5WgssX2PePQ6auhbw opukWcyZm59WzKdj86Lz/fqrbzy8oE8O/AUg9WfFvUmv/RDWZKeGpY3VUS3njwE5KsiX BXIDXhP0NRPTyUy+QQt+pqNDihhDO2DBJ+aebVukD13Rxxqm98ZKMBNGC+QJJRV3GphZ lG9XGCiaIlErPW2J3TlwljtGJCSg6BlB2Z6u5dVbaL7dVkQF+6Bbok61iHAssW4Uatkj zDsg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=XLwuMQtn; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id s20-20020a05622a179400b003ef5d34757fsi9876780qtk.747.2023.05.15.07.34.40 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:34:40 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=XLwuMQtn; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZGX-0006lI-P0; Mon, 15 May 2023 10:34:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZGK-0006gR-NX for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:21 -0400 Received: from mail-pl1-x62a.google.com ([2607:f8b0:4864:20::62a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGH-0004qA-AG for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:20 -0400 Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-1aaf2ede38fso124252995ad.2 for ; Mon, 15 May 2023 07:33:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161196; x=1686753196; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Bi+7zXX3SkE4OXmw+QtVLrvmyt1/y9mkItKgRCmaxGs=; b=XLwuMQtnYBpwgEHJkTvUHakQ85jtbPvCc1zbaZLotBlJ3FItgfHJMKcZq+zhfkwrTU wjm6zBDqUGVuowwev3/3Ta28ICwknQ9qCNCjstTGtbjWMiSxNOF5ruvvbF2UUJWD32Hg N71mQ3XsFd0ujfl30QeVnbc5fbEF512bKggeZgwh5sFhVNivQtQ8VJCwBXTLvFJsjx3V GYqndc0kRFhRoXABR+YjFICby1Witqu+HPKEtkOgUSE5fjLCz5CMc7qZJlBvnHanq/IA djSdNKpg8tTv7rbwF+ANLa1fqHmiozfdVM3nmr6sspalNoyddES69DKe/8htPL5QfAXy flSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161196; x=1686753196; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Bi+7zXX3SkE4OXmw+QtVLrvmyt1/y9mkItKgRCmaxGs=; b=M3SUIO1j+3BJhMJuogkREnuReDroVSXCWHIK5qMMPJH46CJT1G3kRGb5EeFA3LoECo XQNgnMj/zLkjUJ1Gqehxs8FdBdAX5JLWfP8qXsWQ8TrtNiuqZuVk8yH8b+3elFk50wRx 1RKA/83zlltgyWRNW638sq2nO0Ya7FaNi6+BjUZ9Jbrd6A/+VtkAX9u6gKHX/JfuAeNd QpTfUXkKTrzsq7zWXhKQGXo0mTPNPMamQfzMsdWwGJB6DiFKwBBIike1h4NoFgMjiigG E3amfvZy9JYDfl6WQaytZnXX7VRDgyWgROP9tPieAsRw82aOlACGZglVFCbDOpZsgt90 fwhA== X-Gm-Message-State: AC+VfDxZj4y5av47RijaulOC3iCeTM8bVr7z6PzNv7nstQ8QWukMaCgr dp7B5S63EtyoTtB1QMV7YPTO/2Q9orlcke5Xru8= X-Received: by 2002:a17:902:ee8a:b0:1ac:6c26:c32f with SMTP id a10-20020a170902ee8a00b001ac6c26c32fmr29497059pld.46.1684161195895; Mon, 15 May 2023 07:33:15 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:15 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 01/54] include/exec/memop: Add MO_ATOM_* Date: Mon, 15 May 2023 07:32:20 -0700 Message-Id: <20230515143313.734053-2-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62a; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org This field may be used to describe the precise atomicity requirements of the guest, which may then be used to constrain the methods by which it may be emulated by the host. For instance, the AArch64 LDP (32-bit) instruction changes semantics with ARMv8.4 LSE2, from MO_64 | MO_ATOM_IFALIGN_PAIR (64-bits, single-copy atomic only on 4 byte units, nonatomic if not aligned by 4), to MO_64 | MO_ATOM_WITHIN16 (64-bits, single-copy atomic within a 16 byte block) The former may be implemented with two 4 byte loads, or a single 8 byte load if that happens to be efficient on the host. The latter may not be implemented with two 4 byte loads and may also require a helper when misaligned. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- v5: Fold MO_ATMAX_* into MO_ATOM_* to make two MO_ATOM_*_PAIR cases. This avoids the creation of combinations that will not be implemented. Dump the new field. Reviews dropped. --- include/exec/memop.h | 37 +++++++++++++++++++++++++++++++++++++ tcg/tcg.c | 27 +++++++++++++++++++++------ 2 files changed, 58 insertions(+), 6 deletions(-) diff --git a/include/exec/memop.h b/include/exec/memop.h index 07f5f88188..a86dc6743a 100644 --- a/include/exec/memop.h +++ b/include/exec/memop.h @@ -72,6 +72,43 @@ typedef enum MemOp { MO_ALIGN_64 = 6 << MO_ASHIFT, MO_ALIGN = MO_AMASK, + /* + * MO_ATOM_* describes the atomicity requirements of the operation: + * MO_ATOM_IFALIGN: the operation must be single-copy atomic if it + * is aligned; if unaligned there is no atomicity. + * MO_ATOM_IFALIGN_PAIR: the entire operation may be considered to + * be a pair of half-sized operations which are packed together + * for convenience, with single-copy atomicity on each half if + * the half is aligned. + * This is the atomicity e.g. of Arm pre-FEAT_LSE2 LDP. + * MO_ATOM_WITHIN16: the operation is single-copy atomic, even if it + * is unaligned, so long as it does not cross a 16-byte boundary; + * if it crosses a 16-byte boundary there is no atomicity. + * This is the atomicity e.g. of Arm FEAT_LSE2 LDR. + * MO_ATOM_WITHIN16_PAIR: the entire operation is single-copy atomic, + * if it happens to be within a 16-byte boundary, otherwise it + * devolves to a pair of half-sized MO_ATOM_WITHIN16 operations. + * Depending on alignment, one or both will be single-copy atomic. + * This is the atomicity e.g. of Arm FEAT_LSE2 LDP. + * MO_ATOM_SUBALIGN: the operation is single-copy atomic by parts + * by the alignment. E.g. if the address is 0 mod 4, then each + * 4-byte subobject is single-copy atomic. + * This is the atomicity e.g. of IBM Power. + * MO_ATOM_NONE: the operation has no atomicity requirements. + * + * Note the default (i.e. 0) value is single-copy atomic to the + * size of the operation, if aligned. This retains the behaviour + * from before this field was introduced. + */ + MO_ATOM_SHIFT = 8, + MO_ATOM_IFALIGN = 0 << MO_ATOM_SHIFT, + MO_ATOM_IFALIGN_PAIR = 1 << MO_ATOM_SHIFT, + MO_ATOM_WITHIN16 = 2 << MO_ATOM_SHIFT, + MO_ATOM_WITHIN16_PAIR = 3 << MO_ATOM_SHIFT, + MO_ATOM_SUBALIGN = 4 << MO_ATOM_SHIFT, + MO_ATOM_NONE = 5 << MO_ATOM_SHIFT, + MO_ATOM_MASK = 7 << MO_ATOM_SHIFT, + /* Combinations of the above, for ease of use. */ MO_UB = MO_8, MO_UW = MO_16, diff --git a/tcg/tcg.c b/tcg/tcg.c index 1231c8ab4c..ea5975805e 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -2195,6 +2195,15 @@ static const char * const alignment_name[(MO_AMASK >> MO_ASHIFT) + 1] = { [MO_ALIGN_64 >> MO_ASHIFT] = "al64+", }; +static const char * const atom_name[(MO_ATOM_MASK >> MO_ATOM_SHIFT) + 1] = { + [MO_ATOM_IFALIGN >> MO_ATOM_SHIFT] = "", + [MO_ATOM_IFALIGN_PAIR >> MO_ATOM_SHIFT] = "pair+", + [MO_ATOM_WITHIN16 >> MO_ATOM_SHIFT] = "w16+", + [MO_ATOM_WITHIN16_PAIR >> MO_ATOM_SHIFT] = "w16p+", + [MO_ATOM_SUBALIGN >> MO_ATOM_SHIFT] = "sub+", + [MO_ATOM_NONE >> MO_ATOM_SHIFT] = "noat+", +}; + static const char bswap_flag_name[][6] = { [TCG_BSWAP_IZ] = "iz", [TCG_BSWAP_OZ] = "oz", @@ -2330,17 +2339,23 @@ static void tcg_dump_ops(TCGContext *s, FILE *f, bool have_prefs) case INDEX_op_qemu_ld_i64: case INDEX_op_qemu_st_i64: { + const char *s_al, *s_op, *s_at = NULL; MemOpIdx oi = op->args[k++]; MemOp op = get_memop(oi); unsigned ix = get_mmuidx(oi); - if (op & ~(MO_AMASK | MO_BSWAP | MO_SSIZE)) { - col += ne_fprintf(f, ",$0x%x,%u", op, ix); + s_al = alignment_name[(op & MO_AMASK) >> MO_ASHIFT]; + s_op = ldst_name[op & (MO_BSWAP | MO_SSIZE)]; + s_at = atom_name[(op & MO_ATOM_MASK) >> MO_ATOM_SHIFT]; + op &= ~(MO_AMASK | MO_BSWAP | MO_SSIZE | MO_ATOM_MASK); + + /* If all fields are accounted for, print symbolically. */ + if (!op && s_al && s_op && s_at) { + col += ne_fprintf(f, ",%s%s%s,%u", + s_at, s_al, s_op, ix); } else { - const char *s_al, *s_op; - s_al = alignment_name[(op & MO_AMASK) >> MO_ASHIFT]; - s_op = ldst_name[op & (MO_BSWAP | MO_SSIZE)]; - col += ne_fprintf(f, ",%s%s,%u", s_al, s_op, ix); + op = get_memop(oi); + col += ne_fprintf(f, ",$0x%x,%u", op, ix); } i = 1; } From patchwork Mon May 15 14:32:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681935 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1148618wrr; Mon, 15 May 2023 07:35:44 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7ejGWN2qZyQ1WoQa4KIkYiiQ7fzVZCAnOZmOXYgt6ZLsX2x0HmWrx/Lwl9WYjZOycgK32h X-Received: by 2002:a05:622a:510:b0:3ef:2eab:3451 with SMTP id l16-20020a05622a051000b003ef2eab3451mr64624068qtx.20.1684161344382; Mon, 15 May 2023 07:35:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161344; cv=none; d=google.com; s=arc-20160816; b=bepYi9/w1P7/MKWUciFvWv7+SbOEC3KwGQDyhcujCyElaKlIaGNYQx66FGQW9Fk6Yu iTiHXEZkalRlpQbbTHaghBST0zEcEv4gtCmg8kbXaMhIjUrN6DXznfqEEK/Pr51PMEAs KjAs2J4revHN2y14kZy72vKFYZtrIfM42PYkWhprcFC6XUFwQx3M+yjDlyFaG611FVdT 7iP8A5wkYi/1Qq/hbaEUkd2Ro09ZrTWohWMnjEM1NWWDJkBM5ySC6xW+5dNnZkl5gEXY ky+Omgbngo6+CtgH3vB/smYrhwQb6b9tHEghBU2wSMKNZar8md7usW9olnv2rsB1xLYm byVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=lqkFItzss73JSGsigJ9bxvDmiX30Xrz62lzTSn0mvUw=; b=pA6xFbVijygG4rbDFT3S87kJvQhMNCmhMIc2rzNDqJFUK0VNGmSNFWseCBL2QwtA40 d4E41+SejDhZixul4lMv2V74pPYWMHxPRhIog7WqzMH+mObLqoRq/Nh9qtpc4tWuP+5k ipGZ/tXjHPmQhbmWxzbgLz+PDkeQhVwNgXxWvRP9KS2cO1HQyhA4Cx2vlRRsZa4BiZg7 5nDytCDBbS8HZigeBNcsIjqpNTajcRYsOGleT+Mzj8zK7+to2+h7Chg/q9ci6WdccECY /acoD6tk8IpSGYydk620fN13VpUlK6ApSKb9UQ3oWCVg3xzVESuxmbfiQ7AfoBoHnPuC tP2A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=RLy1hi0j; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id l20-20020ae9f014000000b0074606147007si12730185qkg.679.2023.05.15.07.35.44 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:35:44 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=RLy1hi0j; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZHy-0007ND-Ja; Mon, 15 May 2023 10:35:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZGN-0006hC-KQ for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:25 -0400 Received: from mail-pf1-x42d.google.com ([2607:f8b0:4864:20::42d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGJ-0004qU-10 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:22 -0400 Received: by mail-pf1-x42d.google.com with SMTP id d2e1a72fcca58-64ab2a37812so20414056b3a.1 for ; Mon, 15 May 2023 07:33:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161197; x=1686753197; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lqkFItzss73JSGsigJ9bxvDmiX30Xrz62lzTSn0mvUw=; b=RLy1hi0jzUn8bkAvpXar6Ot4t+q6FPFw3oi0jFo+h2mUCKBnLJggQL2dXD+mcHEHMy djGebVaYlsThYU7X0/ItOYujUNwsLRFkHRZKEWVtDyWL75IkdJuZq3Mi6/+w4id6yLiE IVP1cn5b6hDZ3oXnew8ayNNuwE4tur9EfoFUr1Qg0OKyJZyhM8oQ7qjMCrgq/2ja0d3J NCZThW3Jivog/Gb7FdymvzjPB71jnJRgba8zans6XvrNBWbU+C6RnUI2cagXO6/KcdnX kMhcsILptc5wTZApDWLIAkIer9syj10lD1tuYFTYGuKSElAWa/jYEFRqPF3v0hPAO/g9 8UmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161197; x=1686753197; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lqkFItzss73JSGsigJ9bxvDmiX30Xrz62lzTSn0mvUw=; b=GbJzKOfdBjbnFzO4TgNTamP3CcZhwoiXGB2D3N64/cgK57nljj6sTxsIEiLO6NxkGc crGsD9YHoLw88rcOfwWSxn8IjVjABxqaR/WCbDEslBhIFZrtMTTmICi3D30ATD7DurwO o59dL6yrmjcvAx+s+DopQOxK8MzjWiJCIOfTRiS/ZfJoqauTjFbeZghGO9JJQd/xUwGJ huTsazooxfxqFTMWV/biyvdGcWPCjL7SO2YnAkmJhaKlEcEkwD9CBjL+Ndg+MNo9u56t N0ZfsXjXEjvmohoZ9nrCeroNNz+EtViM2WU8+1x92HU5XYSYUZe3flshRM1ui0ewyeS4 +c6w== X-Gm-Message-State: AC+VfDzUvziyK1Gl6m5syNzABBSaPvNpR+/vZHPM25+K4iUstHZyLoV5 ZtGzk8IzpvF30M9f+v9Cmoyb0P1p4CLZx/BW40k= X-Received: by 2002:a17:903:1252:b0:1ae:10a5:8349 with SMTP id u18-20020a170903125200b001ae10a58349mr5143967plh.23.1684161196724; Mon, 15 May 2023 07:33:16 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:16 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 02/54] accel/tcg: Honor atomicity of loads Date: Mon, 15 May 2023 07:32:21 -0700 Message-Id: <20230515143313.734053-3-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::42d; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x42d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Create ldst_atomicity.c.inc. Not required for user-only code loads, because we've ensured that the page is read-only before beginning to translate code. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- v5: Dropped r-b because of MO_ATOM_* reorg --- accel/tcg/cputlb.c | 175 +++++++--- accel/tcg/user-exec.c | 26 +- accel/tcg/ldst_atomicity.c.inc | 566 +++++++++++++++++++++++++++++++++ 3 files changed, 716 insertions(+), 51 deletions(-) create mode 100644 accel/tcg/ldst_atomicity.c.inc diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index 617777055a..33e75ae962 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -1668,6 +1668,9 @@ tb_page_addr_t get_page_addr_code_hostp(CPUArchState *env, target_ulong addr, return qemu_ram_addr_from_host_nofail(p); } +/* Load/store with atomicity primitives. */ +#include "ldst_atomicity.c.inc" + #ifdef CONFIG_PLUGIN /* * Perform a TLB lookup and populate the qemu_plugin_hwaddr structure. @@ -2035,35 +2038,7 @@ static void validate_memop(MemOpIdx oi, MemOp expected) * specifically for reading instructions from system memory. It is * called by the translation loop and in some helpers where the code * is disassembled. It shouldn't be called directly by guest code. - */ - -typedef uint64_t FullLoadHelper(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr); - -static inline uint64_t QEMU_ALWAYS_INLINE -load_memop(const void *haddr, MemOp op) -{ - switch (op) { - case MO_UB: - return ldub_p(haddr); - case MO_BEUW: - return lduw_be_p(haddr); - case MO_LEUW: - return lduw_le_p(haddr); - case MO_BEUL: - return (uint32_t)ldl_be_p(haddr); - case MO_LEUL: - return (uint32_t)ldl_le_p(haddr); - case MO_BEUQ: - return ldq_be_p(haddr); - case MO_LEUQ: - return ldq_le_p(haddr); - default: - qemu_build_not_reached(); - } -} - -/* + * * For the benefit of TCG generated code, we want to avoid the * complication of ABI-specific return type promotion and always * return a value extended to the register size of the host. This is @@ -2119,17 +2094,139 @@ static uint64_t do_ld_bytes_beN(MMULookupPageData *p, uint64_t ret_be) return ret_be; } +/** + * do_ld_parts_beN + * @p: translation parameters + * @ret_be: accumulated data + * + * As do_ld_bytes_beN, but atomically on each aligned part. + */ +static uint64_t do_ld_parts_beN(MMULookupPageData *p, uint64_t ret_be) +{ + void *haddr = p->haddr; + int size = p->size; + + do { + uint64_t x; + int n; + + /* + * Find minimum of alignment and size. + * This is slightly stronger than required by MO_ATOM_SUBALIGN, which + * would have only checked the low bits of addr|size once at the start, + * but is just as easy. + */ + switch (((uintptr_t)haddr | size) & 7) { + case 4: + x = cpu_to_be32(load_atomic4(haddr)); + ret_be = (ret_be << 32) | x; + n = 4; + break; + case 2: + case 6: + x = cpu_to_be16(load_atomic2(haddr)); + ret_be = (ret_be << 16) | x; + n = 2; + break; + default: + x = *(uint8_t *)haddr; + ret_be = (ret_be << 8) | x; + n = 1; + break; + case 0: + g_assert_not_reached(); + } + haddr += n; + size -= n; + } while (size != 0); + return ret_be; +} + +/** + * do_ld_parts_be4 + * @p: translation parameters + * @ret_be: accumulated data + * + * As do_ld_bytes_beN, but with one atomic load. + * Four aligned bytes are guaranteed to cover the load. + */ +static uint64_t do_ld_whole_be4(MMULookupPageData *p, uint64_t ret_be) +{ + int o = p->addr & 3; + uint32_t x = load_atomic4(p->haddr - o); + + x = cpu_to_be32(x); + x <<= o * 8; + x >>= (4 - p->size) * 8; + return (ret_be << (p->size * 8)) | x; +} + +/** + * do_ld_parts_be8 + * @p: translation parameters + * @ret_be: accumulated data + * + * As do_ld_bytes_beN, but with one atomic load. + * Eight aligned bytes are guaranteed to cover the load. + */ +static uint64_t do_ld_whole_be8(CPUArchState *env, uintptr_t ra, + MMULookupPageData *p, uint64_t ret_be) +{ + int o = p->addr & 7; + uint64_t x = load_atomic8_or_exit(env, ra, p->haddr - o); + + x = cpu_to_be64(x); + x <<= o * 8; + x >>= (8 - p->size) * 8; + return (ret_be << (p->size * 8)) | x; +} + /* * Wrapper for the above. */ static uint64_t do_ld_beN(CPUArchState *env, MMULookupPageData *p, - uint64_t ret_be, int mmu_idx, - MMUAccessType type, uintptr_t ra) + uint64_t ret_be, int mmu_idx, MMUAccessType type, + MemOp mop, uintptr_t ra) { + MemOp atom; + unsigned tmp, half_size; + if (unlikely(p->flags & TLB_MMIO)) { return do_ld_mmio_beN(env, p, ret_be, mmu_idx, type, ra); - } else { + } + + /* + * It is a given that we cross a page and therefore there is no + * atomicity for the load as a whole, but subobjects may need attention. + */ + atom = mop & MO_ATOM_MASK; + switch (atom) { + case MO_ATOM_SUBALIGN: + return do_ld_parts_beN(p, ret_be); + + case MO_ATOM_IFALIGN_PAIR: + case MO_ATOM_WITHIN16_PAIR: + tmp = mop & MO_SIZE; + tmp = tmp ? tmp - 1 : 0; + half_size = 1 << tmp; + if (atom == MO_ATOM_IFALIGN_PAIR + ? p->size == half_size + : p->size >= half_size) { + if (!HAVE_al8_fast && p->size < 4) { + return do_ld_whole_be4(p, ret_be); + } else { + return do_ld_whole_be8(env, ra, p, ret_be); + } + } + /* fall through */ + + case MO_ATOM_IFALIGN: + case MO_ATOM_WITHIN16: + case MO_ATOM_NONE: return do_ld_bytes_beN(p, ret_be); + + default: + g_assert_not_reached(); } } @@ -2153,7 +2250,7 @@ static uint16_t do_ld_2(CPUArchState *env, MMULookupPageData *p, int mmu_idx, } /* Perform the load host endian, then swap if necessary. */ - ret = load_memop(p->haddr, MO_UW); + ret = load_atom_2(env, ra, p->haddr, memop); if (memop & MO_BSWAP) { ret = bswap16(ret); } @@ -2170,7 +2267,7 @@ static uint32_t do_ld_4(CPUArchState *env, MMULookupPageData *p, int mmu_idx, } /* Perform the load host endian. */ - ret = load_memop(p->haddr, MO_UL); + ret = load_atom_4(env, ra, p->haddr, memop); if (memop & MO_BSWAP) { ret = bswap32(ret); } @@ -2187,7 +2284,7 @@ static uint64_t do_ld_8(CPUArchState *env, MMULookupPageData *p, int mmu_idx, } /* Perform the load host endian. */ - ret = load_memop(p->haddr, MO_UQ); + ret = load_atom_8(env, ra, p->haddr, memop); if (memop & MO_BSWAP) { ret = bswap64(ret); } @@ -2263,8 +2360,8 @@ static uint32_t do_ld4_mmu(CPUArchState *env, target_ulong addr, MemOpIdx oi, return do_ld_4(env, &l.page[0], l.mmu_idx, access_type, l.memop, ra); } - ret = do_ld_beN(env, &l.page[0], 0, l.mmu_idx, access_type, ra); - ret = do_ld_beN(env, &l.page[1], ret, l.mmu_idx, access_type, ra); + ret = do_ld_beN(env, &l.page[0], 0, l.mmu_idx, access_type, l.memop, ra); + ret = do_ld_beN(env, &l.page[1], ret, l.mmu_idx, access_type, l.memop, ra); if ((l.memop & MO_BSWAP) == MO_LE) { ret = bswap32(ret); } @@ -2297,8 +2394,8 @@ static uint64_t do_ld8_mmu(CPUArchState *env, target_ulong addr, MemOpIdx oi, return do_ld_8(env, &l.page[0], l.mmu_idx, access_type, l.memop, ra); } - ret = do_ld_beN(env, &l.page[0], 0, l.mmu_idx, access_type, ra); - ret = do_ld_beN(env, &l.page[1], ret, l.mmu_idx, access_type, ra); + ret = do_ld_beN(env, &l.page[0], 0, l.mmu_idx, access_type, l.memop, ra); + ret = do_ld_beN(env, &l.page[1], ret, l.mmu_idx, access_type, l.memop, ra); if ((l.memop & MO_BSWAP) == MO_LE) { ret = bswap64(ret); } diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c index fc597a010d..fefc83cc8c 100644 --- a/accel/tcg/user-exec.c +++ b/accel/tcg/user-exec.c @@ -931,6 +931,8 @@ static void *cpu_mmu_lookup(CPUArchState *env, target_ulong addr, return ret; } +#include "ldst_atomicity.c.inc" + uint8_t cpu_ldb_mmu(CPUArchState *env, abi_ptr addr, MemOpIdx oi, uintptr_t ra) { @@ -953,10 +955,10 @@ uint16_t cpu_ldw_be_mmu(CPUArchState *env, abi_ptr addr, validate_memop(oi, MO_BEUW); haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); - ret = lduw_be_p(haddr); + ret = load_atom_2(env, ra, haddr, get_memop(oi)); clear_helper_retaddr(); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); - return ret; + return cpu_to_be16(ret); } uint32_t cpu_ldl_be_mmu(CPUArchState *env, abi_ptr addr, @@ -967,10 +969,10 @@ uint32_t cpu_ldl_be_mmu(CPUArchState *env, abi_ptr addr, validate_memop(oi, MO_BEUL); haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); - ret = ldl_be_p(haddr); + ret = load_atom_4(env, ra, haddr, get_memop(oi)); clear_helper_retaddr(); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); - return ret; + return cpu_to_be32(ret); } uint64_t cpu_ldq_be_mmu(CPUArchState *env, abi_ptr addr, @@ -981,10 +983,10 @@ uint64_t cpu_ldq_be_mmu(CPUArchState *env, abi_ptr addr, validate_memop(oi, MO_BEUQ); haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); - ret = ldq_be_p(haddr); + ret = load_atom_8(env, ra, haddr, get_memop(oi)); clear_helper_retaddr(); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); - return ret; + return cpu_to_be64(ret); } uint16_t cpu_ldw_le_mmu(CPUArchState *env, abi_ptr addr, @@ -995,10 +997,10 @@ uint16_t cpu_ldw_le_mmu(CPUArchState *env, abi_ptr addr, validate_memop(oi, MO_LEUW); haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); - ret = lduw_le_p(haddr); + ret = load_atom_2(env, ra, haddr, get_memop(oi)); clear_helper_retaddr(); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); - return ret; + return cpu_to_le16(ret); } uint32_t cpu_ldl_le_mmu(CPUArchState *env, abi_ptr addr, @@ -1009,10 +1011,10 @@ uint32_t cpu_ldl_le_mmu(CPUArchState *env, abi_ptr addr, validate_memop(oi, MO_LEUL); haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); - ret = ldl_le_p(haddr); + ret = load_atom_4(env, ra, haddr, get_memop(oi)); clear_helper_retaddr(); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); - return ret; + return cpu_to_le32(ret); } uint64_t cpu_ldq_le_mmu(CPUArchState *env, abi_ptr addr, @@ -1023,10 +1025,10 @@ uint64_t cpu_ldq_le_mmu(CPUArchState *env, abi_ptr addr, validate_memop(oi, MO_LEUQ); haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); - ret = ldq_le_p(haddr); + ret = load_atom_8(env, ra, haddr, get_memop(oi)); clear_helper_retaddr(); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); - return ret; + return cpu_to_le64(ret); } Int128 cpu_ld16_be_mmu(CPUArchState *env, abi_ptr addr, diff --git a/accel/tcg/ldst_atomicity.c.inc b/accel/tcg/ldst_atomicity.c.inc new file mode 100644 index 0000000000..c5790b2ec9 --- /dev/null +++ b/accel/tcg/ldst_atomicity.c.inc @@ -0,0 +1,566 @@ +/* + * Routines common to user and system emulation of load/store. + * + * Copyright (c) 2022 Linaro, Ltd. + * + * SPDX-License-Identifier: GPL-2.0-or-later + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + */ + +#ifdef CONFIG_ATOMIC64 +# define HAVE_al8 true +#else +# define HAVE_al8 false +#endif +#define HAVE_al8_fast (ATOMIC_REG_SIZE >= 8) + +#if defined(CONFIG_ATOMIC128) +# define HAVE_al16_fast true +#else +# define HAVE_al16_fast false +#endif + +/** + * required_atomicity: + * + * Return the lg2 bytes of atomicity required by @memop for @p. + * If the operation must be split into two operations to be + * examined separately for atomicity, return -lg2. + */ +static int required_atomicity(CPUArchState *env, uintptr_t p, MemOp memop) +{ + MemOp atom = memop & MO_ATOM_MASK; + MemOp size = memop & MO_SIZE; + MemOp half = size ? size - 1 : 0; + unsigned tmp; + int atmax; + + switch (atom) { + case MO_ATOM_NONE: + atmax = MO_8; + break; + + case MO_ATOM_IFALIGN_PAIR: + size = half; + /* fall through */ + + case MO_ATOM_IFALIGN: + tmp = (1 << size) - 1; + atmax = p & tmp ? MO_8 : size; + break; + + case MO_ATOM_WITHIN16: + tmp = p & 15; + atmax = (tmp + (1 << size) <= 16 ? size : MO_8); + break; + + case MO_ATOM_WITHIN16_PAIR: + tmp = p & 15; + if (tmp + (1 << size) <= 16) { + atmax = size; + } else if (tmp + (1 << half) == 16) { + /* + * The pair exactly straddles the boundary. + * Both halves are naturally aligned and atomic. + */ + atmax = half; + } else { + /* + * One of the pair crosses the boundary, and is non-atomic. + * The other of the pair does not cross, and is atomic. + */ + atmax = -half; + } + break; + + case MO_ATOM_SUBALIGN: + /* + * Examine the alignment of p to determine if there are subobjects + * that must be aligned. Note that we only really need ctz4() -- + * any more sigificant bits are discarded by the immediately + * following comparison. + */ + tmp = ctz32(p); + atmax = MIN(size, tmp); + break; + + default: + g_assert_not_reached(); + } + + /* + * Here we have the architectural atomicity of the operation. + * However, when executing in a serial context, we need no extra + * host atomicity in order to avoid racing. This reduction + * avoids looping with cpu_loop_exit_atomic. + */ + if (cpu_in_serial_context(env_cpu(env))) { + return MO_8; + } + return atmax; +} + +/** + * load_atomic2: + * @pv: host address + * + * Atomically load 2 aligned bytes from @pv. + */ +static inline uint16_t load_atomic2(void *pv) +{ + uint16_t *p = __builtin_assume_aligned(pv, 2); + return qatomic_read(p); +} + +/** + * load_atomic4: + * @pv: host address + * + * Atomically load 4 aligned bytes from @pv. + */ +static inline uint32_t load_atomic4(void *pv) +{ + uint32_t *p = __builtin_assume_aligned(pv, 4); + return qatomic_read(p); +} + +/** + * load_atomic8: + * @pv: host address + * + * Atomically load 8 aligned bytes from @pv. + */ +static inline uint64_t load_atomic8(void *pv) +{ + uint64_t *p = __builtin_assume_aligned(pv, 8); + + qemu_build_assert(HAVE_al8); + return qatomic_read__nocheck(p); +} + +/** + * load_atomic16: + * @pv: host address + * + * Atomically load 16 aligned bytes from @pv. + */ +static inline Int128 load_atomic16(void *pv) +{ +#ifdef CONFIG_ATOMIC128 + __uint128_t *p = __builtin_assume_aligned(pv, 16); + Int128Alias r; + + r.u = qatomic_read__nocheck(p); + return r.s; +#else + qemu_build_not_reached(); +#endif +} + +/** + * load_atomic8_or_exit: + * @env: cpu context + * @ra: host unwind address + * @pv: host address + * + * Atomically load 8 aligned bytes from @pv. + * If this is not possible, longjmp out to restart serially. + */ +static uint64_t load_atomic8_or_exit(CPUArchState *env, uintptr_t ra, void *pv) +{ + if (HAVE_al8) { + return load_atomic8(pv); + } + +#ifdef CONFIG_USER_ONLY + /* + * If the page is not writable, then assume the value is immutable + * and requires no locking. This ignores the case of MAP_SHARED with + * another process, because the fallback start_exclusive solution + * provides no protection across processes. + */ + if (!page_check_range(h2g(pv), 8, PAGE_WRITE)) { + uint64_t *p = __builtin_assume_aligned(pv, 8); + return *p; + } +#endif + + /* Ultimate fallback: re-execute in serial context. */ + cpu_loop_exit_atomic(env_cpu(env), ra); +} + +/** + * load_atomic16_or_exit: + * @env: cpu context + * @ra: host unwind address + * @pv: host address + * + * Atomically load 16 aligned bytes from @pv. + * If this is not possible, longjmp out to restart serially. + */ +static Int128 load_atomic16_or_exit(CPUArchState *env, uintptr_t ra, void *pv) +{ + Int128 *p = __builtin_assume_aligned(pv, 16); + + if (HAVE_al16_fast) { + return load_atomic16(p); + } + +#ifdef CONFIG_USER_ONLY + /* + * We can only use cmpxchg to emulate a load if the page is writable. + * If the page is not writable, then assume the value is immutable + * and requires no locking. This ignores the case of MAP_SHARED with + * another process, because the fallback start_exclusive solution + * provides no protection across processes. + */ + if (!page_check_range(h2g(p), 16, PAGE_WRITE)) { + return *p; + } +#endif + + /* + * In system mode all guest pages are writable, and for user-only + * we have just checked writability. Try cmpxchg. + */ +#if defined(CONFIG_CMPXCHG128) + /* Swap 0 with 0, with the side-effect of returning the old value. */ + { + Int128Alias r; + r.u = __sync_val_compare_and_swap_16((__uint128_t *)p, 0, 0); + return r.s; + } +#endif + + /* Ultimate fallback: re-execute in serial context. */ + cpu_loop_exit_atomic(env_cpu(env), ra); +} + +/** + * load_atom_extract_al4x2: + * @pv: host address + * + * Load 4 bytes from @p, from two sequential atomic 4-byte loads. + */ +static uint32_t load_atom_extract_al4x2(void *pv) +{ + uintptr_t pi = (uintptr_t)pv; + int sh = (pi & 3) * 8; + uint32_t a, b; + + pv = (void *)(pi & ~3); + a = load_atomic4(pv); + b = load_atomic4(pv + 4); + + if (HOST_BIG_ENDIAN) { + return (a << sh) | (b >> (-sh & 31)); + } else { + return (a >> sh) | (b << (-sh & 31)); + } +} + +/** + * load_atom_extract_al8x2: + * @pv: host address + * + * Load 8 bytes from @p, from two sequential atomic 8-byte loads. + */ +static uint64_t load_atom_extract_al8x2(void *pv) +{ + uintptr_t pi = (uintptr_t)pv; + int sh = (pi & 7) * 8; + uint64_t a, b; + + pv = (void *)(pi & ~7); + a = load_atomic8(pv); + b = load_atomic8(pv + 8); + + if (HOST_BIG_ENDIAN) { + return (a << sh) | (b >> (-sh & 63)); + } else { + return (a >> sh) | (b << (-sh & 63)); + } +} + +/** + * load_atom_extract_al8_or_exit: + * @env: cpu context + * @ra: host unwind address + * @pv: host address + * @s: object size in bytes, @s <= 4. + * + * Atomically load @s bytes from @p, when p % s != 0, and [p, p+s-1] does + * not cross an 8-byte boundary. This means that we can perform an atomic + * 8-byte load and extract. + * The value is returned in the low bits of a uint32_t. + */ +static uint32_t load_atom_extract_al8_or_exit(CPUArchState *env, uintptr_t ra, + void *pv, int s) +{ + uintptr_t pi = (uintptr_t)pv; + int o = pi & 7; + int shr = (HOST_BIG_ENDIAN ? 8 - s - o : o) * 8; + + pv = (void *)(pi & ~7); + return load_atomic8_or_exit(env, ra, pv) >> shr; +} + +/** + * load_atom_extract_al16_or_exit: + * @env: cpu context + * @ra: host unwind address + * @p: host address + * @s: object size in bytes, @s <= 8. + * + * Atomically load @s bytes from @p, when p % 16 < 8 + * and p % 16 + s > 8. I.e. does not cross a 16-byte + * boundary, but *does* cross an 8-byte boundary. + * This is the slow version, so we must have eliminated + * any faster load_atom_extract_al8_or_exit case. + * + * If this is not possible, longjmp out to restart serially. + */ +static uint64_t load_atom_extract_al16_or_exit(CPUArchState *env, uintptr_t ra, + void *pv, int s) +{ + uintptr_t pi = (uintptr_t)pv; + int o = pi & 7; + int shr = (HOST_BIG_ENDIAN ? 16 - s - o : o) * 8; + Int128 r; + + /* + * Note constraints above: p & 8 must be clear. + * Provoke SIGBUS if possible otherwise. + */ + pv = (void *)(pi & ~7); + r = load_atomic16_or_exit(env, ra, pv); + + r = int128_urshift(r, shr); + return int128_getlo(r); +} + +/** + * load_atom_extract_al16_or_al8: + * @p: host address + * @s: object size in bytes, @s <= 8. + * + * Load @s bytes from @p, when p % s != 0. If [p, p+s-1] does not + * cross an 16-byte boundary then the access must be 16-byte atomic, + * otherwise the access must be 8-byte atomic. + */ +static inline uint64_t load_atom_extract_al16_or_al8(void *pv, int s) +{ +#if defined(CONFIG_ATOMIC128) + uintptr_t pi = (uintptr_t)pv; + int o = pi & 7; + int shr = (HOST_BIG_ENDIAN ? 16 - s - o : o) * 8; + __uint128_t r; + + pv = (void *)(pi & ~7); + if (pi & 8) { + uint64_t *p8 = __builtin_assume_aligned(pv, 16, 8); + uint64_t a = qatomic_read__nocheck(p8); + uint64_t b = qatomic_read__nocheck(p8 + 1); + + if (HOST_BIG_ENDIAN) { + r = ((__uint128_t)a << 64) | b; + } else { + r = ((__uint128_t)b << 64) | a; + } + } else { + __uint128_t *p16 = __builtin_assume_aligned(pv, 16, 0); + r = qatomic_read__nocheck(p16); + } + return r >> shr; +#else + qemu_build_not_reached(); +#endif +} + +/** + * load_atom_4_by_2: + * @pv: host address + * + * Load 4 bytes from @pv, with two 2-byte atomic loads. + */ +static inline uint32_t load_atom_4_by_2(void *pv) +{ + uint32_t a = load_atomic2(pv); + uint32_t b = load_atomic2(pv + 2); + + if (HOST_BIG_ENDIAN) { + return (a << 16) | b; + } else { + return (b << 16) | a; + } +} + +/** + * load_atom_8_by_2: + * @pv: host address + * + * Load 8 bytes from @pv, with four 2-byte atomic loads. + */ +static inline uint64_t load_atom_8_by_2(void *pv) +{ + uint32_t a = load_atom_4_by_2(pv); + uint32_t b = load_atom_4_by_2(pv + 4); + + if (HOST_BIG_ENDIAN) { + return ((uint64_t)a << 32) | b; + } else { + return ((uint64_t)b << 32) | a; + } +} + +/** + * load_atom_8_by_4: + * @pv: host address + * + * Load 8 bytes from @pv, with two 4-byte atomic loads. + */ +static inline uint64_t load_atom_8_by_4(void *pv) +{ + uint32_t a = load_atomic4(pv); + uint32_t b = load_atomic4(pv + 4); + + if (HOST_BIG_ENDIAN) { + return ((uint64_t)a << 32) | b; + } else { + return ((uint64_t)b << 32) | a; + } +} + +/** + * load_atom_2: + * @p: host address + * @memop: the full memory op + * + * Load 2 bytes from @p, honoring the atomicity of @memop. + */ +static uint16_t load_atom_2(CPUArchState *env, uintptr_t ra, + void *pv, MemOp memop) +{ + uintptr_t pi = (uintptr_t)pv; + int atmax; + + if (likely((pi & 1) == 0)) { + return load_atomic2(pv); + } + if (HAVE_al16_fast) { + return load_atom_extract_al16_or_al8(pv, 2); + } + + atmax = required_atomicity(env, pi, memop); + switch (atmax) { + case MO_8: + return lduw_he_p(pv); + case MO_16: + /* The only case remaining is MO_ATOM_WITHIN16. */ + if (!HAVE_al8_fast && (pi & 3) == 1) { + /* Big or little endian, we want the middle two bytes. */ + return load_atomic4(pv - 1) >> 8; + } + if (unlikely((pi & 15) != 7)) { + return load_atom_extract_al8_or_exit(env, ra, pv, 2); + } + return load_atom_extract_al16_or_exit(env, ra, pv, 2); + default: + g_assert_not_reached(); + } +} + +/** + * load_atom_4: + * @p: host address + * @memop: the full memory op + * + * Load 4 bytes from @p, honoring the atomicity of @memop. + */ +static uint32_t load_atom_4(CPUArchState *env, uintptr_t ra, + void *pv, MemOp memop) +{ + uintptr_t pi = (uintptr_t)pv; + int atmax; + + if (likely((pi & 3) == 0)) { + return load_atomic4(pv); + } + if (HAVE_al16_fast) { + return load_atom_extract_al16_or_al8(pv, 4); + } + + atmax = required_atomicity(env, pi, memop); + switch (atmax) { + case MO_8: + case MO_16: + case -MO_16: + /* + * For MO_ATOM_IFALIGN, this is more atomicity than required, + * but it's trivially supported on all hosts, better than 4 + * individual byte loads (when the host requires alignment), + * and overlaps with the MO_ATOM_SUBALIGN case of p % 2 == 0. + */ + return load_atom_extract_al4x2(pv); + case MO_32: + if (!(pi & 4)) { + return load_atom_extract_al8_or_exit(env, ra, pv, 4); + } + return load_atom_extract_al16_or_exit(env, ra, pv, 4); + default: + g_assert_not_reached(); + } +} + +/** + * load_atom_8: + * @p: host address + * @memop: the full memory op + * + * Load 8 bytes from @p, honoring the atomicity of @memop. + */ +static uint64_t load_atom_8(CPUArchState *env, uintptr_t ra, + void *pv, MemOp memop) +{ + uintptr_t pi = (uintptr_t)pv; + int atmax; + + /* + * If the host does not support 8-byte atomics, wait until we have + * examined the atomicity parameters below. + */ + if (HAVE_al8 && likely((pi & 7) == 0)) { + return load_atomic8(pv); + } + if (HAVE_al16_fast) { + return load_atom_extract_al16_or_al8(pv, 8); + } + + atmax = required_atomicity(env, pi, memop); + if (atmax == MO_64) { + if (!HAVE_al8 && (pi & 7) == 0) { + load_atomic8_or_exit(env, ra, pv); + } + return load_atom_extract_al16_or_exit(env, ra, pv, 8); + } + if (HAVE_al8_fast) { + return load_atom_extract_al8x2(pv); + } + switch (atmax) { + case MO_8: + return ldq_he_p(pv); + case MO_16: + return load_atom_8_by_2(pv); + case MO_32: + return load_atom_8_by_4(pv); + case -MO_32: + if (HAVE_al8) { + return load_atom_extract_al8x2(pv); + } + cpu_loop_exit_atomic(env_cpu(env), ra); + default: + g_assert_not_reached(); + } +} From patchwork Mon May 15 14:32:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681987 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1154886wrr; Mon, 15 May 2023 07:49:34 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5aqUqFXEpkSEImSaSN4cWvC1fnp6venlyjkFniApAVM3/CM7+Zjh3rMRu8Fv9mpWw9RQlH X-Received: by 2002:a17:906:da8c:b0:94a:653b:ba41 with SMTP id xh12-20020a170906da8c00b0094a653bba41mr17400689ejb.15.1684162174542; Mon, 15 May 2023 07:49:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684162174; cv=none; d=google.com; s=arc-20160816; b=Enr2S/pdoM8rDvQeNk7d5BuoLT0DhlfhN8uXx8yenP/g5O6/ujbkz7cVYObdK/MfQI JZIN6LKA8etr3UAibODK9CCNjMNl2jTB5R32gsyj3KJDm3eW35SgNMhQW5Bc0nTvK8LH JZsZIcEKCgt+uSglEDjJDaNNGuxAouj4pSIc4cIKnKiDIqbN6rS6yqjAh1KtRkHoLuyR hyNvNyYToK80atc88ncUngjzKURqNUw8aIfZQf1/rx82IV76iZz+FLWpRWDP1a5pgR0f VnjEEhr1xY6C97jG71XUp7LKiHBBVRb9idewkNqBOxZpKPO4CtEp+wm8A0KB9F/kDOTQ EhIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=nk1HxWVRmifEGCFmHU7ZbzfTYOGxETf7RvWHo+ttDAk=; b=XSPFIDV18wLTfOxcPeCvFPbBfSuKSBf311AEFAReE9qdz9DAhpK6kLwRL2kEjmsKty IC5Gp/dgZ8ZFePcoUsy5yF0YAlFwvcf8rlkm3xj82JFpFh0gGetwOE3y1Bzg2gxTU37S FlYALyBzCiCqT3Zov0pFqBriftGL3xbFBxBABJROJFH6CUAE7PCTXZDJ+n2cBHV4y4wc emDqcDR9FFBaoFdhF/OuELK/v0sgWRSkfIkJMi2MYSX+goXZsbUQCtc6uW68UMUikEfQ sf9iXJJWcIpVQwR28G3JDXyYhKuSZIsuNdwDvXUQSarZxc3tEDr8IWXVuchY/+fxvcj+ Dmiw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=XmpHYKiK; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id ja3-20020a170907988300b0094ee039da42si13533904ejc.751.2023.05.15.07.49.34 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:49:34 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=XmpHYKiK; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZHx-0007KI-0m; Mon, 15 May 2023 10:35:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZGN-0006hE-L0 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:25 -0400 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGJ-0004qg-Mw for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:23 -0400 Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-1aaea3909d1so120067035ad.2 for ; Mon, 15 May 2023 07:33:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161198; x=1686753198; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nk1HxWVRmifEGCFmHU7ZbzfTYOGxETf7RvWHo+ttDAk=; b=XmpHYKiKOicF9Qu04+X+xZlu3QUAW/1l66CU0cXyuRjtvx3LdpRKY/QVXnDVbMOf/x riMGmoMR/dNVCCU9vw3GLp08PkuDQnwcVMdM1GIC9oCvpCk+cLE1mMWo2+dCrrzbzk5k llAWa8LPSOfKXLQc3sjRsmZcX2vZG10DTR0RkntuuZnhtf419/onVr9jBE1yHDGxIB8D qTVc9kPlPnw7cUaqQPKpYOvfgOvOAvota2H2qO4OHUQxdYucEarjljbjrq1+VHJ9ua5m l5e1l16b8uR/96dh/e/EZ2ofe9ydSvDIrg4ibmENhGLAUakrIrdwVc/Y+ZProtj8p8m/ BUjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161198; x=1686753198; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nk1HxWVRmifEGCFmHU7ZbzfTYOGxETf7RvWHo+ttDAk=; b=f//aCGuW32uCJfeK5gU8lAY4SVWmi+ky7jghSwoOXxtggO0aANza3X9NDrrXZqh9cj PRvB0cQjf2hnG4nXCZlD0ApcFFSlDiOlLiLPF7vOFxz/ElI9DdMEH88UznVeHV9BC2Ib rHcNfVcHZKofXqCcNj3XUr/5ZBsHxKsPYKbk83I0/Bs+pYzEmeYjtrZdyHE9bdU0005C 5BBvJU0VHprAlFXRV2d3n15qEjKWQSxYUZRSF/ttfSu3LbqnWzkC35w1y9lkOisPlp/x p9sWREVzacqd470fQGIU1KawAZXhHjDkSfBU4MclQR6ehwQ+80KH+nyAZxVHzSipOWMA J1gQ== X-Gm-Message-State: AC+VfDwzvehdfaFhP++OmtLTtK+KYuwCEeBT5evMwrBNcAH/qY2TiapC r71rcg33dlJaNpf+qJ6ubyE374ZbqSbkyLz62Po= X-Received: by 2002:a17:903:190:b0:1ad:bccc:af6a with SMTP id z16-20020a170903019000b001adbcccaf6amr21864589plg.56.1684161197725; Mon, 15 May 2023 07:33:17 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:17 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 03/54] accel/tcg: Honor atomicity of stores Date: Mon, 15 May 2023 07:32:22 -0700 Message-Id: <20230515143313.734053-4-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::635; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x635.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- v5: Dropped r-b because of MO_ATOM_* reorg --- accel/tcg/cputlb.c | 108 ++++---- accel/tcg/user-exec.c | 12 +- accel/tcg/ldst_atomicity.c.inc | 491 +++++++++++++++++++++++++++++++++ 3 files changed, 545 insertions(+), 66 deletions(-) diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index 33e75ae962..d910464c36 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -2599,36 +2599,6 @@ Int128 cpu_ld16_le_mmu(CPUArchState *env, abi_ptr addr, * Store Helpers */ -static inline void QEMU_ALWAYS_INLINE -store_memop(void *haddr, uint64_t val, MemOp op) -{ - switch (op) { - case MO_UB: - stb_p(haddr, val); - break; - case MO_BEUW: - stw_be_p(haddr, val); - break; - case MO_LEUW: - stw_le_p(haddr, val); - break; - case MO_BEUL: - stl_be_p(haddr, val); - break; - case MO_LEUL: - stl_le_p(haddr, val); - break; - case MO_BEUQ: - stq_be_p(haddr, val); - break; - case MO_LEUQ: - stq_le_p(haddr, val); - break; - default: - qemu_build_not_reached(); - } -} - /** * do_st_mmio_leN: * @env: cpu context @@ -2655,38 +2625,56 @@ static uint64_t do_st_mmio_leN(CPUArchState *env, MMULookupPageData *p, return val_le; } -/** - * do_st_bytes_leN: - * @p: translation parameters - * @val_le: data to store - * - * Store @p->size bytes at @p->haddr, which is RAM. - * The bytes to store are extracted in little-endian order from @val_le; - * return the bytes of @val_le beyond @p->size that have not been stored. - */ -static uint64_t do_st_bytes_leN(MMULookupPageData *p, uint64_t val_le) -{ - uint8_t *haddr = p->haddr; - int i, size = p->size; - - for (i = 0; i < size; i++, val_le >>= 8) { - haddr[i] = val_le; - } - return val_le; -} - /* * Wrapper for the above. */ static uint64_t do_st_leN(CPUArchState *env, MMULookupPageData *p, - uint64_t val_le, int mmu_idx, uintptr_t ra) + uint64_t val_le, int mmu_idx, + MemOp mop, uintptr_t ra) { + MemOp atom; + unsigned tmp, half_size; + if (unlikely(p->flags & TLB_MMIO)) { return do_st_mmio_leN(env, p, val_le, mmu_idx, ra); } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) { return val_le >> (p->size * 8); - } else { - return do_st_bytes_leN(p, val_le); + } + + /* + * It is a given that we cross a page and therefore there is no atomicity + * for the store as a whole, but subobjects may need attention. + */ + atom = mop & MO_ATOM_MASK; + switch (atom) { + case MO_ATOM_SUBALIGN: + return store_parts_leN(p->haddr, p->size, val_le); + + case MO_ATOM_IFALIGN_PAIR: + case MO_ATOM_WITHIN16_PAIR: + tmp = mop & MO_SIZE; + tmp = tmp ? tmp - 1 : 0; + half_size = 1 << tmp; + if (atom == MO_ATOM_IFALIGN_PAIR + ? p->size == half_size + : p->size >= half_size) { + if (!HAVE_al8_fast && p->size <= 4) { + return store_whole_le4(p->haddr, p->size, val_le); + } else if (HAVE_al8) { + return store_whole_le8(p->haddr, p->size, val_le); + } else { + cpu_loop_exit_atomic(env_cpu(env), ra); + } + } + /* fall through */ + + case MO_ATOM_IFALIGN: + case MO_ATOM_WITHIN16: + case MO_ATOM_NONE: + return store_bytes_leN(p->haddr, p->size, val_le); + + default: + g_assert_not_reached(); } } @@ -2714,7 +2702,7 @@ static void do_st_2(CPUArchState *env, MMULookupPageData *p, uint16_t val, if (memop & MO_BSWAP) { val = bswap16(val); } - store_memop(p->haddr, val, MO_UW); + store_atom_2(env, ra, p->haddr, memop, val); } } @@ -2730,7 +2718,7 @@ static void do_st_4(CPUArchState *env, MMULookupPageData *p, uint32_t val, if (memop & MO_BSWAP) { val = bswap32(val); } - store_memop(p->haddr, val, MO_UL); + store_atom_4(env, ra, p->haddr, memop, val); } } @@ -2746,7 +2734,7 @@ static void do_st_8(CPUArchState *env, MMULookupPageData *p, uint64_t val, if (memop & MO_BSWAP) { val = bswap64(val); } - store_memop(p->haddr, val, MO_UQ); + store_atom_8(env, ra, p->haddr, memop, val); } } @@ -2815,8 +2803,8 @@ static void do_st4_mmu(CPUArchState *env, target_ulong addr, uint32_t val, if ((l.memop & MO_BSWAP) != MO_LE) { val = bswap32(val); } - val = do_st_leN(env, &l.page[0], val, l.mmu_idx, ra); - (void) do_st_leN(env, &l.page[1], val, l.mmu_idx, ra); + val = do_st_leN(env, &l.page[0], val, l.mmu_idx, l.memop, ra); + (void) do_st_leN(env, &l.page[1], val, l.mmu_idx, l.memop, ra); } void helper_le_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, @@ -2849,8 +2837,8 @@ static void do_st8_mmu(CPUArchState *env, target_ulong addr, uint64_t val, if ((l.memop & MO_BSWAP) != MO_LE) { val = bswap64(val); } - val = do_st_leN(env, &l.page[0], val, l.mmu_idx, ra); - (void) do_st_leN(env, &l.page[1], val, l.mmu_idx, ra); + val = do_st_leN(env, &l.page[0], val, l.mmu_idx, l.memop, ra); + (void) do_st_leN(env, &l.page[1], val, l.mmu_idx, l.memop, ra); } void helper_le_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c index fefc83cc8c..b89fa35a83 100644 --- a/accel/tcg/user-exec.c +++ b/accel/tcg/user-exec.c @@ -1086,7 +1086,7 @@ void cpu_stw_be_mmu(CPUArchState *env, abi_ptr addr, uint16_t val, validate_memop(oi, MO_BEUW); haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE); - stw_be_p(haddr, val); + store_atom_2(env, ra, haddr, get_memop(oi), be16_to_cpu(val)); clear_helper_retaddr(); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); } @@ -1098,7 +1098,7 @@ void cpu_stl_be_mmu(CPUArchState *env, abi_ptr addr, uint32_t val, validate_memop(oi, MO_BEUL); haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE); - stl_be_p(haddr, val); + store_atom_4(env, ra, haddr, get_memop(oi), be32_to_cpu(val)); clear_helper_retaddr(); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); } @@ -1110,7 +1110,7 @@ void cpu_stq_be_mmu(CPUArchState *env, abi_ptr addr, uint64_t val, validate_memop(oi, MO_BEUQ); haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE); - stq_be_p(haddr, val); + store_atom_8(env, ra, haddr, get_memop(oi), be64_to_cpu(val)); clear_helper_retaddr(); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); } @@ -1122,7 +1122,7 @@ void cpu_stw_le_mmu(CPUArchState *env, abi_ptr addr, uint16_t val, validate_memop(oi, MO_LEUW); haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE); - stw_le_p(haddr, val); + store_atom_2(env, ra, haddr, get_memop(oi), le16_to_cpu(val)); clear_helper_retaddr(); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); } @@ -1134,7 +1134,7 @@ void cpu_stl_le_mmu(CPUArchState *env, abi_ptr addr, uint32_t val, validate_memop(oi, MO_LEUL); haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE); - stl_le_p(haddr, val); + store_atom_4(env, ra, haddr, get_memop(oi), le32_to_cpu(val)); clear_helper_retaddr(); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); } @@ -1146,7 +1146,7 @@ void cpu_stq_le_mmu(CPUArchState *env, abi_ptr addr, uint64_t val, validate_memop(oi, MO_LEUQ); haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE); - stq_le_p(haddr, val); + store_atom_8(env, ra, haddr, get_memop(oi), le64_to_cpu(val)); clear_helper_retaddr(); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); } diff --git a/accel/tcg/ldst_atomicity.c.inc b/accel/tcg/ldst_atomicity.c.inc index c5790b2ec9..61567f5d0b 100644 --- a/accel/tcg/ldst_atomicity.c.inc +++ b/accel/tcg/ldst_atomicity.c.inc @@ -21,6 +21,12 @@ #else # define HAVE_al16_fast false #endif +#if defined(CONFIG_ATOMIC128) || defined(CONFIG_CMPXCHG128) +# define HAVE_al16 true +#else +# define HAVE_al16 false +#endif + /** * required_atomicity: @@ -564,3 +570,488 @@ static uint64_t load_atom_8(CPUArchState *env, uintptr_t ra, g_assert_not_reached(); } } + +/** + * store_atomic2: + * @pv: host address + * @val: value to store + * + * Atomically store 2 aligned bytes to @pv. + */ +static inline void store_atomic2(void *pv, uint16_t val) +{ + uint16_t *p = __builtin_assume_aligned(pv, 2); + qatomic_set(p, val); +} + +/** + * store_atomic4: + * @pv: host address + * @val: value to store + * + * Atomically store 4 aligned bytes to @pv. + */ +static inline void store_atomic4(void *pv, uint32_t val) +{ + uint32_t *p = __builtin_assume_aligned(pv, 4); + qatomic_set(p, val); +} + +/** + * store_atomic8: + * @pv: host address + * @val: value to store + * + * Atomically store 8 aligned bytes to @pv. + */ +static inline void store_atomic8(void *pv, uint64_t val) +{ + uint64_t *p = __builtin_assume_aligned(pv, 8); + + qemu_build_assert(HAVE_al8); + qatomic_set__nocheck(p, val); +} + +/** + * store_atom_4x2 + */ +static inline void store_atom_4_by_2(void *pv, uint32_t val) +{ + store_atomic2(pv, val >> (HOST_BIG_ENDIAN ? 16 : 0)); + store_atomic2(pv + 2, val >> (HOST_BIG_ENDIAN ? 0 : 16)); +} + +/** + * store_atom_8_by_2 + */ +static inline void store_atom_8_by_2(void *pv, uint64_t val) +{ + store_atom_4_by_2(pv, val >> (HOST_BIG_ENDIAN ? 32 : 0)); + store_atom_4_by_2(pv + 4, val >> (HOST_BIG_ENDIAN ? 0 : 32)); +} + +/** + * store_atom_8_by_4 + */ +static inline void store_atom_8_by_4(void *pv, uint64_t val) +{ + store_atomic4(pv, val >> (HOST_BIG_ENDIAN ? 32 : 0)); + store_atomic4(pv + 4, val >> (HOST_BIG_ENDIAN ? 0 : 32)); +} + +/** + * store_atom_insert_al4: + * @p: host address + * @val: shifted value to store + * @msk: mask for value to store + * + * Atomically store @val to @p, masked by @msk. + */ +static void store_atom_insert_al4(uint32_t *p, uint32_t val, uint32_t msk) +{ + uint32_t old, new; + + p = __builtin_assume_aligned(p, 4); + old = qatomic_read(p); + do { + new = (old & ~msk) | val; + } while (!__atomic_compare_exchange_n(p, &old, new, true, + __ATOMIC_RELAXED, __ATOMIC_RELAXED)); +} + +/** + * store_atom_insert_al8: + * @p: host address + * @val: shifted value to store + * @msk: mask for value to store + * + * Atomically store @val to @p masked by @msk. + */ +static void store_atom_insert_al8(uint64_t *p, uint64_t val, uint64_t msk) +{ + uint64_t old, new; + + qemu_build_assert(HAVE_al8); + p = __builtin_assume_aligned(p, 8); + old = qatomic_read__nocheck(p); + do { + new = (old & ~msk) | val; + } while (!__atomic_compare_exchange_n(p, &old, new, true, + __ATOMIC_RELAXED, __ATOMIC_RELAXED)); +} + +/** + * store_atom_insert_al16: + * @p: host address + * @val: shifted value to store + * @msk: mask for value to store + * + * Atomically store @val to @p masked by @msk. + */ +static void store_atom_insert_al16(Int128 *ps, Int128Alias val, Int128Alias msk) +{ +#if defined(CONFIG_ATOMIC128) + __uint128_t *pu, old, new; + + /* With CONFIG_ATOMIC128, we can avoid the memory barriers. */ + pu = __builtin_assume_aligned(ps, 16); + old = *pu; + do { + new = (old & ~msk.u) | val.u; + } while (!__atomic_compare_exchange_n(pu, &old, new, true, + __ATOMIC_RELAXED, __ATOMIC_RELAXED)); +#elif defined(CONFIG_CMPXCHG128) + __uint128_t *pu, old, new; + + /* + * Without CONFIG_ATOMIC128, __atomic_compare_exchange_n will always + * defer to libatomic, so we must use __sync_*_compare_and_swap_16 + * and accept the sequential consistency that comes with it. + */ + pu = __builtin_assume_aligned(ps, 16); + do { + old = *pu; + new = (old & ~msk.u) | val.u; + } while (!__sync_bool_compare_and_swap_16(pu, old, new)); +#else + qemu_build_not_reached(); +#endif +} + +/** + * store_bytes_leN: + * @pv: host address + * @size: number of bytes to store + * @val_le: data to store + * + * Store @size bytes at @p. The bytes to store are extracted in little-endian order + * from @val_le; return the bytes of @val_le beyond @size that have not been stored. + */ +static uint64_t store_bytes_leN(void *pv, int size, uint64_t val_le) +{ + uint8_t *p = pv; + for (int i = 0; i < size; i++, val_le >>= 8) { + p[i] = val_le; + } + return val_le; +} + +/** + * store_parts_leN + * @pv: host address + * @size: number of bytes to store + * @val_le: data to store + * + * As store_bytes_leN, but atomically on each aligned part. + */ +G_GNUC_UNUSED +static uint64_t store_parts_leN(void *pv, int size, uint64_t val_le) +{ + do { + int n; + + /* Find minimum of alignment and size */ + switch (((uintptr_t)pv | size) & 7) { + case 4: + store_atomic4(pv, le32_to_cpu(val_le)); + val_le >>= 32; + n = 4; + break; + case 2: + case 6: + store_atomic2(pv, le16_to_cpu(val_le)); + val_le >>= 16; + n = 2; + break; + default: + *(uint8_t *)pv = val_le; + val_le >>= 8; + n = 1; + break; + case 0: + g_assert_not_reached(); + } + pv += n; + size -= n; + } while (size != 0); + + return val_le; +} + +/** + * store_whole_le4 + * @pv: host address + * @size: number of bytes to store + * @val_le: data to store + * + * As store_bytes_leN, but atomically as a whole. + * Four aligned bytes are guaranteed to cover the store. + */ +static uint64_t store_whole_le4(void *pv, int size, uint64_t val_le) +{ + int sz = size * 8; + int o = (uintptr_t)pv & 3; + int sh = o * 8; + uint32_t m = MAKE_64BIT_MASK(0, sz); + uint32_t v; + + if (HOST_BIG_ENDIAN) { + v = bswap32(val_le) >> sh; + m = bswap32(m) >> sh; + } else { + v = val_le << sh; + m <<= sh; + } + store_atom_insert_al4(pv - o, v, m); + return val_le >> sz; +} + +/** + * store_whole_le8 + * @pv: host address + * @size: number of bytes to store + * @val_le: data to store + * + * As store_bytes_leN, but atomically as a whole. + * Eight aligned bytes are guaranteed to cover the store. + */ +static uint64_t store_whole_le8(void *pv, int size, uint64_t val_le) +{ + int sz = size * 8; + int o = (uintptr_t)pv & 7; + int sh = o * 8; + uint64_t m = MAKE_64BIT_MASK(0, sz); + uint64_t v; + + qemu_build_assert(HAVE_al8); + if (HOST_BIG_ENDIAN) { + v = bswap64(val_le) >> sh; + m = bswap64(m) >> sh; + } else { + v = val_le << sh; + m <<= sh; + } + store_atom_insert_al8(pv - o, v, m); + return val_le >> sz; +} + +/** + * store_whole_le16 + * @pv: host address + * @size: number of bytes to store + * @val_le: data to store + * + * As store_bytes_leN, but atomically as a whole. + * 16 aligned bytes are guaranteed to cover the store. + */ +static uint64_t store_whole_le16(void *pv, int size, Int128 val_le) +{ + int sz = size * 8; + int o = (uintptr_t)pv & 15; + int sh = o * 8; + Int128 m, v; + + qemu_build_assert(HAVE_al16); + + /* Like MAKE_64BIT_MASK(0, sz), but larger. */ + if (sz <= 64) { + m = int128_make64(MAKE_64BIT_MASK(0, sz)); + } else { + m = int128_make128(-1, MAKE_64BIT_MASK(0, sz - 64)); + } + + if (HOST_BIG_ENDIAN) { + v = int128_urshift(bswap128(val_le), sh); + m = int128_urshift(bswap128(m), sh); + } else { + v = int128_lshift(val_le, sh); + m = int128_lshift(m, sh); + } + store_atom_insert_al16(pv - o, v, m); + + /* Unused if sz <= 64. */ + return int128_gethi(val_le) >> (sz - 64); +} + +/** + * store_atom_2: + * @p: host address + * @val: the value to store + * @memop: the full memory op + * + * Store 2 bytes to @p, honoring the atomicity of @memop. + */ +static void store_atom_2(CPUArchState *env, uintptr_t ra, + void *pv, MemOp memop, uint16_t val) +{ + uintptr_t pi = (uintptr_t)pv; + int atmax; + + if (likely((pi & 1) == 0)) { + store_atomic2(pv, val); + return; + } + + atmax = required_atomicity(env, pi, memop); + if (atmax == MO_8) { + stw_he_p(pv, val); + return; + } + + /* + * The only case remaining is MO_ATOM_WITHIN16. + * Big or little endian, we want the middle two bytes in each test. + */ + if ((pi & 3) == 1) { + store_atom_insert_al4(pv - 1, (uint32_t)val << 8, MAKE_64BIT_MASK(8, 16)); + return; + } else if ((pi & 7) == 3) { + if (HAVE_al8) { + store_atom_insert_al8(pv - 3, (uint64_t)val << 24, MAKE_64BIT_MASK(24, 16)); + return; + } + } else if ((pi & 15) == 7) { + if (HAVE_al16) { + Int128 v = int128_lshift(int128_make64(val), 56); + Int128 m = int128_lshift(int128_make64(0xffff), 56); + store_atom_insert_al16(pv - 7, v, m); + return; + } + } else { + g_assert_not_reached(); + } + + cpu_loop_exit_atomic(env_cpu(env), ra); +} + +/** + * store_atom_4: + * @p: host address + * @val: the value to store + * @memop: the full memory op + * + * Store 4 bytes to @p, honoring the atomicity of @memop. + */ +static void store_atom_4(CPUArchState *env, uintptr_t ra, + void *pv, MemOp memop, uint32_t val) +{ + uintptr_t pi = (uintptr_t)pv; + int atmax; + + if (likely((pi & 3) == 0)) { + store_atomic4(pv, val); + return; + } + + atmax = required_atomicity(env, pi, memop); + switch (atmax) { + case MO_8: + stl_he_p(pv, val); + return; + case MO_16: + store_atom_4_by_2(pv, val); + return; + case -MO_16: + { + uint32_t val_le = cpu_to_le32(val); + int s2 = pi & 3; + int s1 = 4 - s2; + + switch (s2) { + case 1: + val_le = store_whole_le4(pv, s1, val_le); + *(uint8_t *)(pv + 3) = val_le; + break; + case 3: + *(uint8_t *)pv = val_le; + store_whole_le4(pv + 1, s2, val_le >> 8); + break; + case 0: /* aligned */ + case 2: /* atmax MO_16 */ + default: + g_assert_not_reached(); + } + } + return; + case MO_32: + if ((pi & 7) < 4) { + if (HAVE_al8) { + store_whole_le8(pv, 4, cpu_to_le32(val)); + return; + } + } else { + if (HAVE_al16) { + store_whole_le16(pv, 4, int128_make64(cpu_to_le32(val))); + return; + } + } + cpu_loop_exit_atomic(env_cpu(env), ra); + default: + g_assert_not_reached(); + } +} + +/** + * store_atom_8: + * @p: host address + * @val: the value to store + * @memop: the full memory op + * + * Store 8 bytes to @p, honoring the atomicity of @memop. + */ +static void store_atom_8(CPUArchState *env, uintptr_t ra, + void *pv, MemOp memop, uint64_t val) +{ + uintptr_t pi = (uintptr_t)pv; + int atmax; + + if (HAVE_al8 && likely((pi & 7) == 0)) { + store_atomic8(pv, val); + return; + } + + atmax = required_atomicity(env, pi, memop); + switch (atmax) { + case MO_8: + stq_he_p(pv, val); + return; + case MO_16: + store_atom_8_by_2(pv, val); + return; + case MO_32: + store_atom_8_by_4(pv, val); + return; + case -MO_32: + if (HAVE_al8) { + uint64_t val_le = cpu_to_le64(val); + int s2 = pi & 7; + int s1 = 8 - s2; + + switch (s2) { + case 1 ... 3: + val_le = store_whole_le8(pv, s1, val_le); + store_bytes_leN(pv + s1, s2, val_le); + break; + case 5 ... 7: + val_le = store_bytes_leN(pv, s1, val_le); + store_whole_le8(pv + s1, s2, val_le); + break; + case 0: /* aligned */ + case 4: /* atmax MO_32 */ + default: + g_assert_not_reached(); + } + return; + } + break; + case MO_64: + if (HAVE_al16) { + store_whole_le16(pv, 8, int128_make64(cpu_to_le64(val))); + return; + } + break; + default: + g_assert_not_reached(); + } + cpu_loop_exit_atomic(env_cpu(env), ra); +} From patchwork Mon May 15 14:32:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681948 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1150006wrr; Mon, 15 May 2023 07:38:31 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ505CXTzfJOX0JzjQWDF+d/6F2E3o/WwXa986Axt5xI15nFDpayWrZ6stZa23fKlh+nBbvc X-Received: by 2002:a5d:4485:0:b0:306:2d32:8ec with SMTP id j5-20020a5d4485000000b003062d3208ecmr26418398wrq.6.1684161511222; Mon, 15 May 2023 07:38:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161511; cv=none; d=google.com; s=arc-20160816; b=KEeTmMoLempYlrr0zsLz7mbggXAhlcW2wrvoGcl70SlnyCC5EWIJM2v8/zRAh4AH9I lyRwQ4Oa7dNnl9OAt+31PjMzsg+FzBXT+ckuqW2LhTL/9PI3PW9onqiQcvNwJ+2hrS7v fV7h+MD5bK2aR3WaUhhbwzAJay4gCgB6CLg+XpWKcEwtdZcqRd2fbezzKH5o6E8Y8iMc uTXsth0EazYqLrPpaXITwXvIwlIeOtGrT2lbx2AgI59toD5vQ/OomQ413bjZaEv4hLWQ B17HxYN/RFnSDuUCk6Qu/GOW0lhnnkWlTDYE+6bXBRYetfo0nY/SpjrgwN7Nv0vTVQcW iJAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=5OmysGu52+AjJDrP8N0KqUn7tp8gT2MbDLmdHlT9nEM=; b=DCSyofo809zlonAB7zxUvnTybhOHRx0BpPHUO9GB1IFWu6vvNfRcziak0RUTP4Qamc F9Zgg7GhE0pxtmmV0SEzhUabMeydk/87Y5eHxNCykZFXE3zJpUkERfMPV1BtsoQbPyc4 Vkl62ndtQeRdAr8rfYwtfbP4yqAm2BTr5alxnr5pInGfrR7B64oewd++i5n1dXDYqbj9 xaNeryKbOK+Nq6jkxJmbiZW78NcbYY542m1OdbWjOHsKJg9PJACBhhcIKOaMuXg1Ubtp Hcw3/iE2Dj8bmWJLxdJFFt6oPoiimdNLRMqJkm9RCnzXY5SdcX/X/9E2I+M40JBIItve e8UA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ug9rsvzr; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id s8-20020adff808000000b003062fc00e21si14889390wrp.524.2023.05.15.07.38.30 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:38:31 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ug9rsvzr; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZI3-0007V3-O2; Mon, 15 May 2023 10:35:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZGR-0006ir-Nj for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:29 -0400 Received: from mail-pf1-x42f.google.com ([2607:f8b0:4864:20::42f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGK-0004rC-DH for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:27 -0400 Received: by mail-pf1-x42f.google.com with SMTP id d2e1a72fcca58-64a9335a8e7so21246512b3a.0 for ; Mon, 15 May 2023 07:33:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161199; x=1686753199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5OmysGu52+AjJDrP8N0KqUn7tp8gT2MbDLmdHlT9nEM=; b=ug9rsvzra5KMsG7EAV+/yiu9jMLVKk+jwRE5hLS0zQs+ZJX+65B0fJaYokNLyJxEGl 41WtazHgTfiJWNM7aI2PC2SWFzOobSc+knmOD6O9BBI6nORGQsQmk+1jTf423nVscRc1 lmVEEMMvZGH/kXXASx2kgarNmx2AzsigFaKPUfzQ0VgXm5mmmp6tRuRCPT/GN+N/TWIR yFRv/VE4WFlxy2DZztcVSBo+T7q24Ydm4HeUvNixmfP2bXcFX0DkDcj0WkD3cqxqYIW3 jAwUN1XW0qRTVpTOcnqFJNs0l1QCR48eDwleCJaZU6DbbQhelwV9WmCYUPK09TY3KQyX 6ELA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161199; x=1686753199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5OmysGu52+AjJDrP8N0KqUn7tp8gT2MbDLmdHlT9nEM=; b=ROIZ3nQ2kR9tPDzKkO0o9tNo1kVjsV+BBz7RdVWb1t2adNOyWXxmS3mobm3pdVdwAX 4MyYd6ZTVWYoG/J7akvjchcH/26z2spRiysiX2Wa8j/LTlupptdezSb7O71z9BMbbzAi VK03db4E/fXGIkZGU74JgGeG3Scwc27qFevtJOJLy6DICaVfTI/jvmrivHl0JXIWtnEP su35dxa5fH2EnGo+5R/DVRrj1PF4Kga+seZr1NyhB6ZW7vRUBANzGaLgdSnCUxiFKzrT 0WzARvNZd+T3K89uSt/Z+0PmaqqHdrRoXtGE0N5wJBrhizyarMZumCSrsruMPbR+nWBY XczQ== X-Gm-Message-State: AC+VfDwvn1fBf1uStxJG4nDxHJkpmnJfvuN/tyDYru3iUyhldEVVP1tX F40lNu6ir5lbeHkUMwLHzbVEOMdC//k1864/ZvY= X-Received: by 2002:a17:902:e805:b0:1ac:820e:c34a with SMTP id u5-20020a170902e80500b001ac820ec34amr29768463plg.0.1684161198614; Mon, 15 May 2023 07:33:18 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:18 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, =?utf-8?q?Philippe_Mathieu-D?= =?utf-8?q?aud=C3=A9?= , Peter Maydell Subject: [PATCH v5 04/54] tcg: Unify helper_{be,le}_{ld,st}* Date: Mon, 15 May 2023 07:32:23 -0700 Message-Id: <20230515143313.734053-5-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::42f; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x42f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org With the current structure of cputlb.c, there is no difference between the little-endian and big-endian entry points, aside from the assert. Unify the pairs of functions. Hoist the qemu_{ld,st}_helpers arrays to tcg.c. Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- docs/devel/loads-stores.rst | 36 ++---- include/tcg/tcg-ldst.h | 60 ++++------ accel/tcg/cputlb.c | 190 ++++++++++--------------------- tcg/tcg.c | 21 ++++ tcg/tci.c | 61 ++++------ tcg/aarch64/tcg-target.c.inc | 33 ------ tcg/arm/tcg-target.c.inc | 37 ------ tcg/i386/tcg-target.c.inc | 30 +---- tcg/loongarch64/tcg-target.c.inc | 23 ---- tcg/mips/tcg-target.c.inc | 31 ----- tcg/ppc/tcg-target.c.inc | 30 +---- tcg/riscv/tcg-target.c.inc | 42 ------- tcg/s390x/tcg-target.c.inc | 31 +---- tcg/sparc64/tcg-target.c.inc | 32 +----- 14 files changed, 146 insertions(+), 511 deletions(-) diff --git a/docs/devel/loads-stores.rst b/docs/devel/loads-stores.rst index ad5dfe133e..d2cefc77a2 100644 --- a/docs/devel/loads-stores.rst +++ b/docs/devel/loads-stores.rst @@ -297,31 +297,20 @@ swap: ``translator_ld{sign}{size}_swap(env, ptr, swap)`` Regexes for git grep - ``\`` -``helper_*_{ld,st}*_mmu`` +``helper_{ld,st}*_mmu`` ~~~~~~~~~~~~~~~~~~~~~~~~~ These functions are intended primarily to be called by the code -generated by the TCG backend. They may also be called by target -CPU helper function code. Like the ``cpu_{ld,st}_mmuidx_ra`` functions -they perform accesses by guest virtual address, with a given ``mmuidx``. +generated by the TCG backend. Like the ``cpu_{ld,st}_mmu`` functions +they perform accesses by guest virtual address, with a given ``MemOpIdx``. -These functions specify an ``opindex`` parameter which encodes -(among other things) the mmu index to use for the access. This parameter -should be created by calling ``make_memop_idx()``. +They differ from ``cpu_{ld,st}_mmu`` in that they take the endianness +of the operation only from the MemOpIdx, and loads extend the return +value to the size of a host general register (``tcg_target_ulong``). -The ``retaddr`` parameter should be the result of GETPC() called directly -from the top level HELPER(foo) function (or 0 if no guest CPU state -unwinding is required). +load: ``helper_ld{sign}{size}_mmu(env, addr, opindex, retaddr)`` -**TODO** The names of these functions are a bit odd for historical -reasons because they were originally expected to be called only from -within generated code. We should rename them to bring them more in -line with the other memory access functions. The explicit endianness -is the only feature they have beyond ``*_mmuidx_ra``. - -load: ``helper_{endian}_ld{sign}{size}_mmu(env, addr, opindex, retaddr)`` - -store: ``helper_{endian}_st{size}_mmu(env, addr, val, opindex, retaddr)`` +store: ``helper_{size}_mmu(env, addr, val, opindex, retaddr)`` ``sign`` - (empty) : for 32 or 64 bit sizes @@ -334,14 +323,9 @@ store: ``helper_{endian}_st{size}_mmu(env, addr, val, opindex, retaddr)`` - ``l`` : 32 bits - ``q`` : 64 bits -``endian`` - - ``le`` : little endian - - ``be`` : big endian - - ``ret`` : target endianness - Regexes for git grep - - ``\`` - - ``\`` + - ``\`` + - ``\`` ``address_space_*`` ~~~~~~~~~~~~~~~~~~~ diff --git a/include/tcg/tcg-ldst.h b/include/tcg/tcg-ldst.h index 684e394b06..3d897ca942 100644 --- a/include/tcg/tcg-ldst.h +++ b/include/tcg/tcg-ldst.h @@ -28,51 +28,35 @@ #ifdef CONFIG_SOFTMMU /* Value zero-extended to tcg register size. */ -tcg_target_ulong helper_ret_ldub_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr); -tcg_target_ulong helper_le_lduw_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr); -tcg_target_ulong helper_le_ldul_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr); -uint64_t helper_le_ldq_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr); -tcg_target_ulong helper_be_lduw_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr); -tcg_target_ulong helper_be_ldul_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr); -uint64_t helper_be_ldq_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr); +tcg_target_ulong helper_ldub_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t retaddr); +tcg_target_ulong helper_lduw_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t retaddr); +tcg_target_ulong helper_ldul_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t retaddr); +uint64_t helper_ldq_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t retaddr); /* Value sign-extended to tcg register size. */ -tcg_target_ulong helper_ret_ldsb_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr); -tcg_target_ulong helper_le_ldsw_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr); -tcg_target_ulong helper_le_ldsl_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr); -tcg_target_ulong helper_be_ldsw_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr); -tcg_target_ulong helper_be_ldsl_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr); +tcg_target_ulong helper_ldsb_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t retaddr); +tcg_target_ulong helper_ldsw_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t retaddr); +tcg_target_ulong helper_ldsl_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t retaddr); /* * Value extended to at least uint32_t, so that some ABIs do not require * zero-extension from uint8_t or uint16_t. */ -void helper_ret_stb_mmu(CPUArchState *env, target_ulong addr, uint32_t val, - MemOpIdx oi, uintptr_t retaddr); -void helper_le_stw_mmu(CPUArchState *env, target_ulong addr, uint32_t val, - MemOpIdx oi, uintptr_t retaddr); -void helper_le_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, - MemOpIdx oi, uintptr_t retaddr); -void helper_le_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, - MemOpIdx oi, uintptr_t retaddr); -void helper_be_stw_mmu(CPUArchState *env, target_ulong addr, uint32_t val, - MemOpIdx oi, uintptr_t retaddr); -void helper_be_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, - MemOpIdx oi, uintptr_t retaddr); -void helper_be_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, - MemOpIdx oi, uintptr_t retaddr); +void helper_stb_mmu(CPUArchState *env, target_ulong addr, uint32_t val, + MemOpIdx oi, uintptr_t retaddr); +void helper_stw_mmu(CPUArchState *env, target_ulong addr, uint32_t val, + MemOpIdx oi, uintptr_t retaddr); +void helper_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, + MemOpIdx oi, uintptr_t retaddr); +void helper_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, + MemOpIdx oi, uintptr_t retaddr); #else diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index d910464c36..34796ef568 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -2012,25 +2012,6 @@ static void *atomic_mmu_lookup(CPUArchState *env, target_ulong addr, cpu_loop_exit_atomic(env_cpu(env), retaddr); } -/* - * Verify that we have passed the correct MemOp to the correct function. - * - * In the case of the helper_*_mmu functions, we will have done this by - * using the MemOp to look up the helper during code generation. - * - * In the case of the cpu_*_mmu functions, this is up to the caller. - * We could present one function to target code, and dispatch based on - * the MemOp, but so far we have worked hard to avoid an indirect function - * call along the memory path. - */ -static void validate_memop(MemOpIdx oi, MemOp expected) -{ -#ifdef CONFIG_DEBUG_TCG - MemOp have = get_memop(oi) & (MO_SIZE | MO_BSWAP); - assert(have == expected); -#endif -} - /* * Load Helpers * @@ -2303,10 +2284,10 @@ static uint8_t do_ld1_mmu(CPUArchState *env, target_ulong addr, MemOpIdx oi, return do_ld_1(env, &l.page[0], l.mmu_idx, access_type, ra); } -tcg_target_ulong helper_ret_ldub_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr) +tcg_target_ulong helper_ldub_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t retaddr) { - validate_memop(oi, MO_UB); + tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_8); return do_ld1_mmu(env, addr, oi, retaddr, MMU_DATA_LOAD); } @@ -2334,17 +2315,10 @@ static uint16_t do_ld2_mmu(CPUArchState *env, target_ulong addr, MemOpIdx oi, return ret; } -tcg_target_ulong helper_le_lduw_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr) +tcg_target_ulong helper_lduw_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t retaddr) { - validate_memop(oi, MO_LEUW); - return do_ld2_mmu(env, addr, oi, retaddr, MMU_DATA_LOAD); -} - -tcg_target_ulong helper_be_lduw_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr) -{ - validate_memop(oi, MO_BEUW); + tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_16); return do_ld2_mmu(env, addr, oi, retaddr, MMU_DATA_LOAD); } @@ -2368,17 +2342,10 @@ static uint32_t do_ld4_mmu(CPUArchState *env, target_ulong addr, MemOpIdx oi, return ret; } -tcg_target_ulong helper_le_ldul_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr) +tcg_target_ulong helper_ldul_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t retaddr) { - validate_memop(oi, MO_LEUL); - return do_ld4_mmu(env, addr, oi, retaddr, MMU_DATA_LOAD); -} - -tcg_target_ulong helper_be_ldul_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr) -{ - validate_memop(oi, MO_BEUL); + tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_32); return do_ld4_mmu(env, addr, oi, retaddr, MMU_DATA_LOAD); } @@ -2402,17 +2369,10 @@ static uint64_t do_ld8_mmu(CPUArchState *env, target_ulong addr, MemOpIdx oi, return ret; } -uint64_t helper_le_ldq_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr) +uint64_t helper_ldq_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t retaddr) { - validate_memop(oi, MO_LEUQ); - return do_ld8_mmu(env, addr, oi, retaddr, MMU_DATA_LOAD); -} - -uint64_t helper_be_ldq_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr) -{ - validate_memop(oi, MO_BEUQ); + tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_64); return do_ld8_mmu(env, addr, oi, retaddr, MMU_DATA_LOAD); } @@ -2421,35 +2381,22 @@ uint64_t helper_be_ldq_mmu(CPUArchState *env, target_ulong addr, * avoid this for 64-bit data, or for 32-bit data on 32-bit host. */ - -tcg_target_ulong helper_ret_ldsb_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr) +tcg_target_ulong helper_ldsb_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t retaddr) { - return (int8_t)helper_ret_ldub_mmu(env, addr, oi, retaddr); + return (int8_t)helper_ldub_mmu(env, addr, oi, retaddr); } -tcg_target_ulong helper_le_ldsw_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr) +tcg_target_ulong helper_ldsw_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t retaddr) { - return (int16_t)helper_le_lduw_mmu(env, addr, oi, retaddr); + return (int16_t)helper_lduw_mmu(env, addr, oi, retaddr); } -tcg_target_ulong helper_be_ldsw_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr) +tcg_target_ulong helper_ldsl_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t retaddr) { - return (int16_t)helper_be_lduw_mmu(env, addr, oi, retaddr); -} - -tcg_target_ulong helper_le_ldsl_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr) -{ - return (int32_t)helper_le_ldul_mmu(env, addr, oi, retaddr); -} - -tcg_target_ulong helper_be_ldsl_mmu(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t retaddr) -{ - return (int32_t)helper_be_ldul_mmu(env, addr, oi, retaddr); + return (int32_t)helper_ldul_mmu(env, addr, oi, retaddr); } /* @@ -2465,7 +2412,7 @@ uint8_t cpu_ldb_mmu(CPUArchState *env, abi_ptr addr, MemOpIdx oi, uintptr_t ra) { uint8_t ret; - validate_memop(oi, MO_UB); + tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_UB); ret = do_ld1_mmu(env, addr, oi, ra, MMU_DATA_LOAD); plugin_load_cb(env, addr, oi); return ret; @@ -2476,7 +2423,7 @@ uint16_t cpu_ldw_be_mmu(CPUArchState *env, abi_ptr addr, { uint16_t ret; - validate_memop(oi, MO_BEUW); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_BEUW); ret = do_ld2_mmu(env, addr, oi, ra, MMU_DATA_LOAD); plugin_load_cb(env, addr, oi); return ret; @@ -2487,7 +2434,7 @@ uint32_t cpu_ldl_be_mmu(CPUArchState *env, abi_ptr addr, { uint32_t ret; - validate_memop(oi, MO_BEUL); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_BEUL); ret = do_ld4_mmu(env, addr, oi, ra, MMU_DATA_LOAD); plugin_load_cb(env, addr, oi); return ret; @@ -2498,7 +2445,7 @@ uint64_t cpu_ldq_be_mmu(CPUArchState *env, abi_ptr addr, { uint64_t ret; - validate_memop(oi, MO_BEUQ); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_BEUQ); ret = do_ld8_mmu(env, addr, oi, ra, MMU_DATA_LOAD); plugin_load_cb(env, addr, oi); return ret; @@ -2509,7 +2456,7 @@ uint16_t cpu_ldw_le_mmu(CPUArchState *env, abi_ptr addr, { uint16_t ret; - validate_memop(oi, MO_LEUW); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_LEUW); ret = do_ld2_mmu(env, addr, oi, ra, MMU_DATA_LOAD); plugin_load_cb(env, addr, oi); return ret; @@ -2520,7 +2467,7 @@ uint32_t cpu_ldl_le_mmu(CPUArchState *env, abi_ptr addr, { uint32_t ret; - validate_memop(oi, MO_LEUL); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_LEUL); ret = do_ld4_mmu(env, addr, oi, ra, MMU_DATA_LOAD); plugin_load_cb(env, addr, oi); return ret; @@ -2531,7 +2478,7 @@ uint64_t cpu_ldq_le_mmu(CPUArchState *env, abi_ptr addr, { uint64_t ret; - validate_memop(oi, MO_LEUQ); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_LEUQ); ret = do_ld8_mmu(env, addr, oi, ra, MMU_DATA_LOAD); plugin_load_cb(env, addr, oi); return ret; @@ -2559,8 +2506,8 @@ Int128 cpu_ld16_be_mmu(CPUArchState *env, abi_ptr addr, mop = (mop & ~(MO_SIZE | MO_AMASK)) | MO_64 | MO_UNALN; new_oi = make_memop_idx(mop, mmu_idx); - h = helper_be_ldq_mmu(env, addr, new_oi, ra); - l = helper_be_ldq_mmu(env, addr + 8, new_oi, ra); + h = helper_ldq_mmu(env, addr, new_oi, ra); + l = helper_ldq_mmu(env, addr + 8, new_oi, ra); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); return int128_make128(l, h); @@ -2588,8 +2535,8 @@ Int128 cpu_ld16_le_mmu(CPUArchState *env, abi_ptr addr, mop = (mop & ~(MO_SIZE | MO_AMASK)) | MO_64 | MO_UNALN; new_oi = make_memop_idx(mop, mmu_idx); - l = helper_le_ldq_mmu(env, addr, new_oi, ra); - h = helper_le_ldq_mmu(env, addr + 8, new_oi, ra); + l = helper_ldq_mmu(env, addr, new_oi, ra); + h = helper_ldq_mmu(env, addr + 8, new_oi, ra); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); return int128_make128(l, h); @@ -2738,13 +2685,13 @@ static void do_st_8(CPUArchState *env, MMULookupPageData *p, uint64_t val, } } -void helper_ret_stb_mmu(CPUArchState *env, target_ulong addr, uint32_t val, - MemOpIdx oi, uintptr_t ra) +void helper_stb_mmu(CPUArchState *env, target_ulong addr, uint32_t val, + MemOpIdx oi, uintptr_t ra) { MMULookupLocals l; bool crosspage; - validate_memop(oi, MO_UB); + tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_8); crosspage = mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE, &l); tcg_debug_assert(!crosspage); @@ -2773,17 +2720,10 @@ static void do_st2_mmu(CPUArchState *env, target_ulong addr, uint16_t val, do_st_1(env, &l.page[1], b, l.mmu_idx, ra); } -void helper_le_stw_mmu(CPUArchState *env, target_ulong addr, uint32_t val, - MemOpIdx oi, uintptr_t retaddr) +void helper_stw_mmu(CPUArchState *env, target_ulong addr, uint32_t val, + MemOpIdx oi, uintptr_t retaddr) { - validate_memop(oi, MO_LEUW); - do_st2_mmu(env, addr, val, oi, retaddr); -} - -void helper_be_stw_mmu(CPUArchState *env, target_ulong addr, uint32_t val, - MemOpIdx oi, uintptr_t retaddr) -{ - validate_memop(oi, MO_BEUW); + tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_16); do_st2_mmu(env, addr, val, oi, retaddr); } @@ -2807,17 +2747,10 @@ static void do_st4_mmu(CPUArchState *env, target_ulong addr, uint32_t val, (void) do_st_leN(env, &l.page[1], val, l.mmu_idx, l.memop, ra); } -void helper_le_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, - MemOpIdx oi, uintptr_t retaddr) +void helper_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, + MemOpIdx oi, uintptr_t retaddr) { - validate_memop(oi, MO_LEUL); - do_st4_mmu(env, addr, val, oi, retaddr); -} - -void helper_be_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, - MemOpIdx oi, uintptr_t retaddr) -{ - validate_memop(oi, MO_BEUL); + tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_32); do_st4_mmu(env, addr, val, oi, retaddr); } @@ -2841,17 +2774,10 @@ static void do_st8_mmu(CPUArchState *env, target_ulong addr, uint64_t val, (void) do_st_leN(env, &l.page[1], val, l.mmu_idx, l.memop, ra); } -void helper_le_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, - MemOpIdx oi, uintptr_t retaddr) +void helper_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, + MemOpIdx oi, uintptr_t retaddr) { - validate_memop(oi, MO_LEUQ); - do_st8_mmu(env, addr, val, oi, retaddr); -} - -void helper_be_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, - MemOpIdx oi, uintptr_t retaddr) -{ - validate_memop(oi, MO_BEUQ); + tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_64); do_st8_mmu(env, addr, val, oi, retaddr); } @@ -2867,49 +2793,55 @@ static void plugin_store_cb(CPUArchState *env, abi_ptr addr, MemOpIdx oi) void cpu_stb_mmu(CPUArchState *env, target_ulong addr, uint8_t val, MemOpIdx oi, uintptr_t retaddr) { - helper_ret_stb_mmu(env, addr, val, oi, retaddr); + helper_stb_mmu(env, addr, val, oi, retaddr); plugin_store_cb(env, addr, oi); } void cpu_stw_be_mmu(CPUArchState *env, target_ulong addr, uint16_t val, MemOpIdx oi, uintptr_t retaddr) { - helper_be_stw_mmu(env, addr, val, oi, retaddr); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_BEUW); + do_st2_mmu(env, addr, val, oi, retaddr); plugin_store_cb(env, addr, oi); } void cpu_stl_be_mmu(CPUArchState *env, target_ulong addr, uint32_t val, MemOpIdx oi, uintptr_t retaddr) { - helper_be_stl_mmu(env, addr, val, oi, retaddr); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_BEUL); + do_st4_mmu(env, addr, val, oi, retaddr); plugin_store_cb(env, addr, oi); } void cpu_stq_be_mmu(CPUArchState *env, target_ulong addr, uint64_t val, MemOpIdx oi, uintptr_t retaddr) { - helper_be_stq_mmu(env, addr, val, oi, retaddr); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_BEUQ); + do_st8_mmu(env, addr, val, oi, retaddr); plugin_store_cb(env, addr, oi); } void cpu_stw_le_mmu(CPUArchState *env, target_ulong addr, uint16_t val, MemOpIdx oi, uintptr_t retaddr) { - helper_le_stw_mmu(env, addr, val, oi, retaddr); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_LEUW); + do_st2_mmu(env, addr, val, oi, retaddr); plugin_store_cb(env, addr, oi); } void cpu_stl_le_mmu(CPUArchState *env, target_ulong addr, uint32_t val, MemOpIdx oi, uintptr_t retaddr) { - helper_le_stl_mmu(env, addr, val, oi, retaddr); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_LEUL); + do_st4_mmu(env, addr, val, oi, retaddr); plugin_store_cb(env, addr, oi); } void cpu_stq_le_mmu(CPUArchState *env, target_ulong addr, uint64_t val, MemOpIdx oi, uintptr_t retaddr) { - helper_le_stq_mmu(env, addr, val, oi, retaddr); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_LEUQ); + do_st8_mmu(env, addr, val, oi, retaddr); plugin_store_cb(env, addr, oi); } @@ -2934,8 +2866,8 @@ void cpu_st16_be_mmu(CPUArchState *env, abi_ptr addr, Int128 val, mop = (mop & ~(MO_SIZE | MO_AMASK)) | MO_64 | MO_UNALN; new_oi = make_memop_idx(mop, mmu_idx); - helper_be_stq_mmu(env, addr, int128_gethi(val), new_oi, ra); - helper_be_stq_mmu(env, addr + 8, int128_getlo(val), new_oi, ra); + helper_stq_mmu(env, addr, int128_gethi(val), new_oi, ra); + helper_stq_mmu(env, addr + 8, int128_getlo(val), new_oi, ra); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); } @@ -2961,8 +2893,8 @@ void cpu_st16_le_mmu(CPUArchState *env, abi_ptr addr, Int128 val, mop = (mop & ~(MO_SIZE | MO_AMASK)) | MO_64 | MO_UNALN; new_oi = make_memop_idx(mop, mmu_idx); - helper_le_stq_mmu(env, addr, int128_getlo(val), new_oi, ra); - helper_le_stq_mmu(env, addr + 8, int128_gethi(val), new_oi, ra); + helper_stq_mmu(env, addr, int128_getlo(val), new_oi, ra); + helper_stq_mmu(env, addr + 8, int128_gethi(val), new_oi, ra); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); } diff --git a/tcg/tcg.c b/tcg/tcg.c index ea5975805e..a26410fb68 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -197,6 +197,27 @@ static void tcg_out_st_helper_args(TCGContext *s, const TCGLabelQemuLdst *l, const TCGLdstHelperParam *p) __attribute__((unused)); +#ifdef CONFIG_SOFTMMU +static void * const qemu_ld_helpers[MO_SSIZE + 1] = { + [MO_UB] = helper_ldub_mmu, + [MO_SB] = helper_ldsb_mmu, + [MO_UW] = helper_lduw_mmu, + [MO_SW] = helper_ldsw_mmu, + [MO_UL] = helper_ldul_mmu, + [MO_UQ] = helper_ldq_mmu, +#if TCG_TARGET_REG_BITS == 64 + [MO_SL] = helper_ldsl_mmu, +#endif +}; + +static void * const qemu_st_helpers[MO_SIZE + 1] = { + [MO_8] = helper_stb_mmu, + [MO_16] = helper_stw_mmu, + [MO_32] = helper_stl_mmu, + [MO_64] = helper_stq_mmu, +}; +#endif + TCGContext tcg_init_ctx; __thread TCGContext *tcg_ctx; diff --git a/tcg/tci.c b/tcg/tci.c index fc67e7e767..5bde2e1f2e 100644 --- a/tcg/tci.c +++ b/tcg/tci.c @@ -293,31 +293,21 @@ static uint64_t tci_qemu_ld(CPUArchState *env, target_ulong taddr, uintptr_t ra = (uintptr_t)tb_ptr; #ifdef CONFIG_SOFTMMU - switch (mop & (MO_BSWAP | MO_SSIZE)) { + switch (mop & MO_SSIZE) { case MO_UB: - return helper_ret_ldub_mmu(env, taddr, oi, ra); + return helper_ldub_mmu(env, taddr, oi, ra); case MO_SB: - return helper_ret_ldsb_mmu(env, taddr, oi, ra); - case MO_LEUW: - return helper_le_lduw_mmu(env, taddr, oi, ra); - case MO_LESW: - return helper_le_ldsw_mmu(env, taddr, oi, ra); - case MO_LEUL: - return helper_le_ldul_mmu(env, taddr, oi, ra); - case MO_LESL: - return helper_le_ldsl_mmu(env, taddr, oi, ra); - case MO_LEUQ: - return helper_le_ldq_mmu(env, taddr, oi, ra); - case MO_BEUW: - return helper_be_lduw_mmu(env, taddr, oi, ra); - case MO_BESW: - return helper_be_ldsw_mmu(env, taddr, oi, ra); - case MO_BEUL: - return helper_be_ldul_mmu(env, taddr, oi, ra); - case MO_BESL: - return helper_be_ldsl_mmu(env, taddr, oi, ra); - case MO_BEUQ: - return helper_be_ldq_mmu(env, taddr, oi, ra); + return helper_ldsb_mmu(env, taddr, oi, ra); + case MO_UW: + return helper_lduw_mmu(env, taddr, oi, ra); + case MO_SW: + return helper_ldsw_mmu(env, taddr, oi, ra); + case MO_UL: + return helper_ldul_mmu(env, taddr, oi, ra); + case MO_SL: + return helper_ldsl_mmu(env, taddr, oi, ra); + case MO_UQ: + return helper_ldq_mmu(env, taddr, oi, ra); default: g_assert_not_reached(); } @@ -382,27 +372,18 @@ static void tci_qemu_st(CPUArchState *env, target_ulong taddr, uint64_t val, uintptr_t ra = (uintptr_t)tb_ptr; #ifdef CONFIG_SOFTMMU - switch (mop & (MO_BSWAP | MO_SIZE)) { + switch (mop & MO_SIZE) { case MO_UB: - helper_ret_stb_mmu(env, taddr, val, oi, ra); + helper_stb_mmu(env, taddr, val, oi, ra); break; - case MO_LEUW: - helper_le_stw_mmu(env, taddr, val, oi, ra); + case MO_UW: + helper_stw_mmu(env, taddr, val, oi, ra); break; - case MO_LEUL: - helper_le_stl_mmu(env, taddr, val, oi, ra); + case MO_UL: + helper_stl_mmu(env, taddr, val, oi, ra); break; - case MO_LEUQ: - helper_le_stq_mmu(env, taddr, val, oi, ra); - break; - case MO_BEUW: - helper_be_stw_mmu(env, taddr, val, oi, ra); - break; - case MO_BEUL: - helper_be_stl_mmu(env, taddr, val, oi, ra); - break; - case MO_BEUQ: - helper_be_stq_mmu(env, taddr, val, oi, ra); + case MO_UQ: + helper_stq_mmu(env, taddr, val, oi, ra); break; default: g_assert_not_reached(); diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc index 62dd22d73c..e6636c1f8b 100644 --- a/tcg/aarch64/tcg-target.c.inc +++ b/tcg/aarch64/tcg-target.c.inc @@ -1587,39 +1587,6 @@ typedef struct { } HostAddress; #ifdef CONFIG_SOFTMMU -/* helper signature: helper_ret_ld_mmu(CPUState *env, target_ulong addr, - * MemOpIdx oi, uintptr_t ra) - */ -static void * const qemu_ld_helpers[MO_SIZE + 1] = { - [MO_8] = helper_ret_ldub_mmu, -#if HOST_BIG_ENDIAN - [MO_16] = helper_be_lduw_mmu, - [MO_32] = helper_be_ldul_mmu, - [MO_64] = helper_be_ldq_mmu, -#else - [MO_16] = helper_le_lduw_mmu, - [MO_32] = helper_le_ldul_mmu, - [MO_64] = helper_le_ldq_mmu, -#endif -}; - -/* helper signature: helper_ret_st_mmu(CPUState *env, target_ulong addr, - * uintxx_t val, MemOpIdx oi, - * uintptr_t ra) - */ -static void * const qemu_st_helpers[MO_SIZE + 1] = { - [MO_8] = helper_ret_stb_mmu, -#if HOST_BIG_ENDIAN - [MO_16] = helper_be_stw_mmu, - [MO_32] = helper_be_stl_mmu, - [MO_64] = helper_be_stq_mmu, -#else - [MO_16] = helper_le_stw_mmu, - [MO_32] = helper_le_stl_mmu, - [MO_64] = helper_le_stq_mmu, -#endif -}; - static const TCGLdstHelperParam ldst_helper_param = { .ntmp = 1, .tmp = { TCG_REG_TMP } }; diff --git a/tcg/arm/tcg-target.c.inc b/tcg/arm/tcg-target.c.inc index df514e56fc..8b0d526659 100644 --- a/tcg/arm/tcg-target.c.inc +++ b/tcg/arm/tcg-target.c.inc @@ -1333,43 +1333,6 @@ typedef struct { } HostAddress; #ifdef CONFIG_SOFTMMU -/* helper signature: helper_ret_ld_mmu(CPUState *env, target_ulong addr, - * int mmu_idx, uintptr_t ra) - */ -static void * const qemu_ld_helpers[MO_SSIZE + 1] = { - [MO_UB] = helper_ret_ldub_mmu, - [MO_SB] = helper_ret_ldsb_mmu, -#if HOST_BIG_ENDIAN - [MO_UW] = helper_be_lduw_mmu, - [MO_UL] = helper_be_ldul_mmu, - [MO_UQ] = helper_be_ldq_mmu, - [MO_SW] = helper_be_ldsw_mmu, - [MO_SL] = helper_be_ldul_mmu, -#else - [MO_UW] = helper_le_lduw_mmu, - [MO_UL] = helper_le_ldul_mmu, - [MO_UQ] = helper_le_ldq_mmu, - [MO_SW] = helper_le_ldsw_mmu, - [MO_SL] = helper_le_ldul_mmu, -#endif -}; - -/* helper signature: helper_ret_st_mmu(CPUState *env, target_ulong addr, - * uintxx_t val, int mmu_idx, uintptr_t ra) - */ -static void * const qemu_st_helpers[MO_SIZE + 1] = { - [MO_8] = helper_ret_stb_mmu, -#if HOST_BIG_ENDIAN - [MO_16] = helper_be_stw_mmu, - [MO_32] = helper_be_stl_mmu, - [MO_64] = helper_be_stq_mmu, -#else - [MO_16] = helper_le_stw_mmu, - [MO_32] = helper_le_stl_mmu, - [MO_64] = helper_le_stq_mmu, -#endif -}; - static TCGReg ldst_ra_gen(TCGContext *s, const TCGLabelQemuLdst *l, int arg) { /* We arrive at the slow path via "BLNE", so R14 contains l->raddr. */ diff --git a/tcg/i386/tcg-target.c.inc b/tcg/i386/tcg-target.c.inc index 9fc5592f5d..826f7764c9 100644 --- a/tcg/i386/tcg-target.c.inc +++ b/tcg/i386/tcg-target.c.inc @@ -1776,32 +1776,6 @@ typedef struct { } HostAddress; #if defined(CONFIG_SOFTMMU) -/* helper signature: helper_ret_ld_mmu(CPUState *env, target_ulong addr, - * int mmu_idx, uintptr_t ra) - */ -static void * const qemu_ld_helpers[(MO_SIZE | MO_BSWAP) + 1] = { - [MO_UB] = helper_ret_ldub_mmu, - [MO_LEUW] = helper_le_lduw_mmu, - [MO_LEUL] = helper_le_ldul_mmu, - [MO_LEUQ] = helper_le_ldq_mmu, - [MO_BEUW] = helper_be_lduw_mmu, - [MO_BEUL] = helper_be_ldul_mmu, - [MO_BEUQ] = helper_be_ldq_mmu, -}; - -/* helper signature: helper_ret_st_mmu(CPUState *env, target_ulong addr, - * uintxx_t val, int mmu_idx, uintptr_t ra) - */ -static void * const qemu_st_helpers[(MO_SIZE | MO_BSWAP) + 1] = { - [MO_UB] = helper_ret_stb_mmu, - [MO_LEUW] = helper_le_stw_mmu, - [MO_LEUL] = helper_le_stl_mmu, - [MO_LEUQ] = helper_le_stq_mmu, - [MO_BEUW] = helper_be_stw_mmu, - [MO_BEUL] = helper_be_stl_mmu, - [MO_BEUQ] = helper_be_stq_mmu, -}; - /* * Because i686 has no register parameters and because x86_64 has xchg * to handle addr/data register overlap, we have placed all input arguments @@ -1842,7 +1816,7 @@ static bool tcg_out_qemu_ld_slow_path(TCGContext *s, TCGLabelQemuLdst *l) } tcg_out_ld_helper_args(s, l, &ldst_helper_param); - tcg_out_branch(s, 1, qemu_ld_helpers[opc & (MO_BSWAP | MO_SIZE)]); + tcg_out_branch(s, 1, qemu_ld_helpers[opc & MO_SIZE]); tcg_out_ld_helper_ret(s, l, false, &ldst_helper_param); tcg_out_jmp(s, l->raddr); @@ -1864,7 +1838,7 @@ static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *l) } tcg_out_st_helper_args(s, l, &ldst_helper_param); - tcg_out_branch(s, 1, qemu_st_helpers[opc & (MO_BSWAP | MO_SIZE)]); + tcg_out_branch(s, 1, qemu_st_helpers[opc & MO_SIZE]); tcg_out_jmp(s, l->raddr); return true; diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index 83fa45c802..d1bc29826f 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -784,29 +784,6 @@ static bool tcg_out_sti(TCGContext *s, TCGType type, TCGArg val, */ #if defined(CONFIG_SOFTMMU) -/* - * helper signature: helper_ret_ld_mmu(CPUState *env, target_ulong addr, - * MemOpIdx oi, uintptr_t ra) - */ -static void * const qemu_ld_helpers[4] = { - [MO_8] = helper_ret_ldub_mmu, - [MO_16] = helper_le_lduw_mmu, - [MO_32] = helper_le_ldul_mmu, - [MO_64] = helper_le_ldq_mmu, -}; - -/* - * helper signature: helper_ret_st_mmu(CPUState *env, target_ulong addr, - * uintxx_t val, MemOpIdx oi, - * uintptr_t ra) - */ -static void * const qemu_st_helpers[4] = { - [MO_8] = helper_ret_stb_mmu, - [MO_16] = helper_le_stw_mmu, - [MO_32] = helper_le_stl_mmu, - [MO_64] = helper_le_stq_mmu, -}; - static bool tcg_out_goto(TCGContext *s, const tcg_insn_unit *target) { tcg_out_opc_b(s, 0); diff --git a/tcg/mips/tcg-target.c.inc b/tcg/mips/tcg-target.c.inc index 5ad9867882..7770ef46bd 100644 --- a/tcg/mips/tcg-target.c.inc +++ b/tcg/mips/tcg-target.c.inc @@ -1076,37 +1076,6 @@ static void tcg_out_call(TCGContext *s, const tcg_insn_unit *arg, } #if defined(CONFIG_SOFTMMU) -static void * const qemu_ld_helpers[MO_SSIZE + 1] = { - [MO_UB] = helper_ret_ldub_mmu, - [MO_SB] = helper_ret_ldsb_mmu, -#if HOST_BIG_ENDIAN - [MO_UW] = helper_be_lduw_mmu, - [MO_SW] = helper_be_ldsw_mmu, - [MO_UL] = helper_be_ldul_mmu, - [MO_SL] = helper_be_ldsl_mmu, - [MO_UQ] = helper_be_ldq_mmu, -#else - [MO_UW] = helper_le_lduw_mmu, - [MO_SW] = helper_le_ldsw_mmu, - [MO_UL] = helper_le_ldul_mmu, - [MO_UQ] = helper_le_ldq_mmu, - [MO_SL] = helper_le_ldsl_mmu, -#endif -}; - -static void * const qemu_st_helpers[MO_SIZE + 1] = { - [MO_UB] = helper_ret_stb_mmu, -#if HOST_BIG_ENDIAN - [MO_UW] = helper_be_stw_mmu, - [MO_UL] = helper_be_stl_mmu, - [MO_UQ] = helper_be_stq_mmu, -#else - [MO_UW] = helper_le_stw_mmu, - [MO_UL] = helper_le_stl_mmu, - [MO_UQ] = helper_le_stq_mmu, -#endif -}; - /* We have four temps, we might as well expose three of them. */ static const TCGLdstHelperParam ldst_helper_param = { .ntmp = 3, .tmp = { TCG_TMP0, TCG_TMP1, TCG_TMP2 } diff --git a/tcg/ppc/tcg-target.c.inc b/tcg/ppc/tcg-target.c.inc index 29bfbfcc61..6a81916e64 100644 --- a/tcg/ppc/tcg-target.c.inc +++ b/tcg/ppc/tcg-target.c.inc @@ -1963,32 +1963,6 @@ static const uint32_t qemu_stx_opc[(MO_SIZE + MO_BSWAP) + 1] = { }; #if defined (CONFIG_SOFTMMU) -/* helper signature: helper_ld_mmu(CPUState *env, target_ulong addr, - * int mmu_idx, uintptr_t ra) - */ -static void * const qemu_ld_helpers[(MO_SIZE | MO_BSWAP) + 1] = { - [MO_UB] = helper_ret_ldub_mmu, - [MO_LEUW] = helper_le_lduw_mmu, - [MO_LEUL] = helper_le_ldul_mmu, - [MO_LEUQ] = helper_le_ldq_mmu, - [MO_BEUW] = helper_be_lduw_mmu, - [MO_BEUL] = helper_be_ldul_mmu, - [MO_BEUQ] = helper_be_ldq_mmu, -}; - -/* helper signature: helper_st_mmu(CPUState *env, target_ulong addr, - * uintxx_t val, int mmu_idx, uintptr_t ra) - */ -static void * const qemu_st_helpers[(MO_SIZE | MO_BSWAP) + 1] = { - [MO_UB] = helper_ret_stb_mmu, - [MO_LEUW] = helper_le_stw_mmu, - [MO_LEUL] = helper_le_stl_mmu, - [MO_LEUQ] = helper_le_stq_mmu, - [MO_BEUW] = helper_be_stw_mmu, - [MO_BEUL] = helper_be_stl_mmu, - [MO_BEUQ] = helper_be_stq_mmu, -}; - static TCGReg ldst_ra_gen(TCGContext *s, const TCGLabelQemuLdst *l, int arg) { if (arg < 0) { @@ -2017,7 +1991,7 @@ static bool tcg_out_qemu_ld_slow_path(TCGContext *s, TCGLabelQemuLdst *lb) } tcg_out_ld_helper_args(s, lb, &ldst_helper_param); - tcg_out_call_int(s, LK, qemu_ld_helpers[opc & (MO_BSWAP | MO_SIZE)]); + tcg_out_call_int(s, LK, qemu_ld_helpers[opc & MO_SIZE]); tcg_out_ld_helper_ret(s, lb, false, &ldst_helper_param); tcg_out_b(s, 0, lb->raddr); @@ -2033,7 +2007,7 @@ static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *lb) } tcg_out_st_helper_args(s, lb, &ldst_helper_param); - tcg_out_call_int(s, LK, qemu_st_helpers[opc & (MO_BSWAP | MO_SIZE)]); + tcg_out_call_int(s, LK, qemu_st_helpers[opc & MO_SIZE]); tcg_out_b(s, 0, lb->raddr); return true; diff --git a/tcg/riscv/tcg-target.c.inc b/tcg/riscv/tcg-target.c.inc index d12b824d8c..8ed0e2f210 100644 --- a/tcg/riscv/tcg-target.c.inc +++ b/tcg/riscv/tcg-target.c.inc @@ -847,48 +847,6 @@ static void tcg_out_mb(TCGContext *s, TCGArg a0) */ #if defined(CONFIG_SOFTMMU) -/* helper signature: helper_ret_ld_mmu(CPUState *env, target_ulong addr, - * MemOpIdx oi, uintptr_t ra) - */ -static void * const qemu_ld_helpers[MO_SSIZE + 1] = { - [MO_UB] = helper_ret_ldub_mmu, - [MO_SB] = helper_ret_ldsb_mmu, -#if HOST_BIG_ENDIAN - [MO_UW] = helper_be_lduw_mmu, - [MO_SW] = helper_be_ldsw_mmu, - [MO_UL] = helper_be_ldul_mmu, -#if TCG_TARGET_REG_BITS == 64 - [MO_SL] = helper_be_ldsl_mmu, -#endif - [MO_UQ] = helper_be_ldq_mmu, -#else - [MO_UW] = helper_le_lduw_mmu, - [MO_SW] = helper_le_ldsw_mmu, - [MO_UL] = helper_le_ldul_mmu, -#if TCG_TARGET_REG_BITS == 64 - [MO_SL] = helper_le_ldsl_mmu, -#endif - [MO_UQ] = helper_le_ldq_mmu, -#endif -}; - -/* helper signature: helper_ret_st_mmu(CPUState *env, target_ulong addr, - * uintxx_t val, MemOpIdx oi, - * uintptr_t ra) - */ -static void * const qemu_st_helpers[MO_SIZE + 1] = { - [MO_8] = helper_ret_stb_mmu, -#if HOST_BIG_ENDIAN - [MO_16] = helper_be_stw_mmu, - [MO_32] = helper_be_stl_mmu, - [MO_64] = helper_be_stq_mmu, -#else - [MO_16] = helper_le_stw_mmu, - [MO_32] = helper_le_stl_mmu, - [MO_64] = helper_le_stq_mmu, -#endif -}; - static void tcg_out_goto(TCGContext *s, const tcg_insn_unit *target) { tcg_out_opc_jump(s, OPC_JAL, TCG_REG_ZERO, 0); diff --git a/tcg/s390x/tcg-target.c.inc b/tcg/s390x/tcg-target.c.inc index aacbaf21d5..968977be98 100644 --- a/tcg/s390x/tcg-target.c.inc +++ b/tcg/s390x/tcg-target.c.inc @@ -438,33 +438,6 @@ static const uint8_t tcg_cond_to_ltr_cond[] = { [TCG_COND_GEU] = S390_CC_ALWAYS, }; -#ifdef CONFIG_SOFTMMU -static void * const qemu_ld_helpers[(MO_SSIZE | MO_BSWAP) + 1] = { - [MO_UB] = helper_ret_ldub_mmu, - [MO_SB] = helper_ret_ldsb_mmu, - [MO_LEUW] = helper_le_lduw_mmu, - [MO_LESW] = helper_le_ldsw_mmu, - [MO_LEUL] = helper_le_ldul_mmu, - [MO_LESL] = helper_le_ldsl_mmu, - [MO_LEUQ] = helper_le_ldq_mmu, - [MO_BEUW] = helper_be_lduw_mmu, - [MO_BESW] = helper_be_ldsw_mmu, - [MO_BEUL] = helper_be_ldul_mmu, - [MO_BESL] = helper_be_ldsl_mmu, - [MO_BEUQ] = helper_be_ldq_mmu, -}; - -static void * const qemu_st_helpers[(MO_SIZE | MO_BSWAP) + 1] = { - [MO_UB] = helper_ret_stb_mmu, - [MO_LEUW] = helper_le_stw_mmu, - [MO_LEUL] = helper_le_stl_mmu, - [MO_LEUQ] = helper_le_stq_mmu, - [MO_BEUW] = helper_be_stw_mmu, - [MO_BEUL] = helper_be_stl_mmu, - [MO_BEUQ] = helper_be_stq_mmu, -}; -#endif - static const tcg_insn_unit *tb_ret_addr; uint64_t s390_facilities[3]; @@ -1721,7 +1694,7 @@ static bool tcg_out_qemu_ld_slow_path(TCGContext *s, TCGLabelQemuLdst *lb) } tcg_out_ld_helper_args(s, lb, &ldst_helper_param); - tcg_out_call_int(s, qemu_ld_helpers[opc & (MO_BSWAP | MO_SIZE)]); + tcg_out_call_int(s, qemu_ld_helpers[opc & MO_SIZE]); tcg_out_ld_helper_ret(s, lb, false, &ldst_helper_param); tgen_gotoi(s, S390_CC_ALWAYS, lb->raddr); @@ -1738,7 +1711,7 @@ static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *lb) } tcg_out_st_helper_args(s, lb, &ldst_helper_param); - tcg_out_call_int(s, qemu_st_helpers[opc & (MO_BSWAP | MO_SIZE)]); + tcg_out_call_int(s, qemu_st_helpers[opc & MO_SIZE]); tgen_gotoi(s, S390_CC_ALWAYS, lb->raddr); return true; diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc index 7e6466d3b6..e997db2645 100644 --- a/tcg/sparc64/tcg-target.c.inc +++ b/tcg/sparc64/tcg-target.c.inc @@ -919,33 +919,11 @@ static void tcg_out_mb(TCGContext *s, TCGArg a0) } #ifdef CONFIG_SOFTMMU -static const tcg_insn_unit *qemu_ld_trampoline[(MO_SSIZE | MO_BSWAP) + 1]; -static const tcg_insn_unit *qemu_st_trampoline[(MO_SIZE | MO_BSWAP) + 1]; +static const tcg_insn_unit *qemu_ld_trampoline[MO_SSIZE + 1]; +static const tcg_insn_unit *qemu_st_trampoline[MO_SIZE + 1]; static void build_trampolines(TCGContext *s) { - static void * const qemu_ld_helpers[] = { - [MO_UB] = helper_ret_ldub_mmu, - [MO_SB] = helper_ret_ldsb_mmu, - [MO_LEUW] = helper_le_lduw_mmu, - [MO_LESW] = helper_le_ldsw_mmu, - [MO_LEUL] = helper_le_ldul_mmu, - [MO_LEUQ] = helper_le_ldq_mmu, - [MO_BEUW] = helper_be_lduw_mmu, - [MO_BESW] = helper_be_ldsw_mmu, - [MO_BEUL] = helper_be_ldul_mmu, - [MO_BEUQ] = helper_be_ldq_mmu, - }; - static void * const qemu_st_helpers[] = { - [MO_UB] = helper_ret_stb_mmu, - [MO_LEUW] = helper_le_stw_mmu, - [MO_LEUL] = helper_le_stl_mmu, - [MO_LEUQ] = helper_le_stq_mmu, - [MO_BEUW] = helper_be_stw_mmu, - [MO_BEUL] = helper_be_stl_mmu, - [MO_BEUQ] = helper_be_stq_mmu, - }; - int i; for (i = 0; i < ARRAY_SIZE(qemu_ld_helpers); ++i) { @@ -1210,9 +1188,9 @@ static void tcg_out_qemu_ld(TCGContext *s, TCGReg data, TCGReg addr, /* We use the helpers to extend SB and SW data, leaving the case of SL needing explicit extending below. */ if ((memop & MO_SSIZE) == MO_SL) { - func = qemu_ld_trampoline[memop & (MO_BSWAP | MO_SIZE)]; + func = qemu_ld_trampoline[MO_UL]; } else { - func = qemu_ld_trampoline[memop & (MO_BSWAP | MO_SSIZE)]; + func = qemu_ld_trampoline[memop & MO_SSIZE]; } tcg_debug_assert(func != NULL); tcg_out_call_nodelay(s, func, false); @@ -1353,7 +1331,7 @@ static void tcg_out_qemu_st(TCGContext *s, TCGReg data, TCGReg addr, tcg_out_movext(s, (memop & MO_SIZE) == MO_64 ? TCG_TYPE_I64 : TCG_TYPE_I32, TCG_REG_O2, data_type, memop & MO_SIZE, data); - func = qemu_st_trampoline[memop & (MO_BSWAP | MO_SIZE)]; + func = qemu_st_trampoline[memop & MO_SIZE]; tcg_debug_assert(func != NULL); tcg_out_call_nodelay(s, func, false); /* delay slot */ From patchwork Mon May 15 14:32:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681945 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1149770wrr; Mon, 15 May 2023 07:38:01 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4e5pC1tUosLDXCmqLmz18yXZsGPl6ibSpqbeojn6y5VeHKbwU+Hyx/C/Kr0h+OGAVl5XNE X-Received: by 2002:a5d:510a:0:b0:304:79c1:725d with SMTP id s10-20020a5d510a000000b0030479c1725dmr23705230wrt.45.1684161481202; Mon, 15 May 2023 07:38:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161481; cv=none; d=google.com; s=arc-20160816; b=o6AVZ1nUObnjJGJqDLww/L6mjUWaUT/2JRwpUB6bfeF4e+ii/LEY5pIKSTwA7rNRXD W87pV75ONNhHFGIGAoqqXt8CYc6pugWpuYOz5zYTyC7UBQHW3DtRUxnLAnlI4HXFor/j ++Yjxgj0+v0TSajy/hp2HUguKoXitiKe5DqxjRDs45uCNsA2/zlaL0yBFyS0GETFFCjd 9kM2sjkBZ1bsaXfL69LkaolHz0WSgCS5PObtkjCUTCKHgY3sbPlgaI0FJhtNLRSB05+q ILVj9x3a+dFC46CFTgoais9Jp+HHt41q70vFESWDgDPAt+To+9fjX+pK9Bbn0WVnRBMN F3gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=W/061ZE7ccFdh+QezoQH0JEIcD0VyHiX4WZjQj5eRxI=; b=y6jEqzYHhoMcSV+ryuOu2v8xCx7sruxTuqg2DzhvpzEWnyCB5rYf3+/qbVa7NjESP1 5n4NoSv4gRGwqqcY+y/Ev2zVSZigYP3aSpdSxRozJyJoMabguI3SKP22mx6A+1Vfsx99 TLHvkkvz6jS6pTUFDo5YxDEck5Z513WFuXPcA6YvXHAM/iE43kNmS1JXgBjofn61Ublr ZIomztr2detgI+mHPl5w7cVyZ0D3UhIjP+QlygltDQAD6+b6iwPwgwOjgTcz4h/Sem0p lbJc78TCMp3aMUWAtMPob7DQ6WkTsw4K88HwaoKw8EPd/NKyiHBYYat6X9x2GHlOm61S c4Uw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b="RT/N3c2v"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id j6-20020adfe506000000b003077f3a97cdsi6416wrm.27.2023.05.15.07.38.00 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:38:01 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b="RT/N3c2v"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZI2-0007Tx-6J; Mon, 15 May 2023 10:35:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZGT-0006jt-Lp for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:30 -0400 Received: from mail-pl1-x631.google.com ([2607:f8b0:4864:20::631]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGL-0004rX-BI for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:27 -0400 Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-1ab01bf474aso97650565ad.1 for ; Mon, 15 May 2023 07:33:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161199; x=1686753199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=W/061ZE7ccFdh+QezoQH0JEIcD0VyHiX4WZjQj5eRxI=; b=RT/N3c2vTmNgcoB81oZuieoiDVmokMUmMnsIrhz/T2POKNkGWYobLow58qn2uy2EQB g8bIPmdCCUs0ib0LZPnSQsDOEaAbTECFHliHjtclw4JSBBZsL8169DpLKYP9KAmIKwqN 16UNXPS1hizdHrBQy2KdZiudP0qPJE8sICZ/Tcm03Bnz7LKoF5H9m1BHCK7KwKD/+l7W c+5oTSkL7vP+854/P2heiSvrgaDlpRBJYlehOj3CylcgXyhtwDIdmld3ShrGZBlD9uaB z1L9plM3iJVgvrgD+CeQwtHm6VAu4+u1eY81YldhO9q6i6Zec7oFiwBZMx45hRzyPpRW zglg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161199; x=1686753199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W/061ZE7ccFdh+QezoQH0JEIcD0VyHiX4WZjQj5eRxI=; b=X9gDKgMw5uBjg70qsKJfrt+J2mA/vrOxE04FibRvIGa/6fUFpHHubtFIyezRY6xGkJ v1Ymsn57ttQ3/fnugwmAAXWEmP+1GeGSk8nJTNa2S9pICM/FLmWwDUM8vlxEbnlWNOcs znNCutwOBHlC8GpuWSx8+5gS3GF2r6WOro7iuIZ2D6q+smLyo2NKsg9Iq+QXyjgPjYBf /xIpDa/DqgCR9oUK86c11oz1BiFZWmiPGmsivcQCk8HiXVaTyjbvIoia4Y/vG+hnNij1 vkLBj8aZFSiWU5ruIN5U2Khk1HeK6iyaq8khyFGsTYhUFVDv6p6ruDD1XDZLUzeGrYij UPCg== X-Gm-Message-State: AC+VfDwzqbs1ZfMTvHwVodneVd2b7t0X6eEKjVhkl1Js3PYVWi7QOyGS 6dJLHDKA3DI9jzEtzl2dSy5oOLj1RfKmIL5y+/g= X-Received: by 2002:a17:903:1247:b0:1ab:afd:903a with SMTP id u7-20020a170903124700b001ab0afd903amr44832805plh.24.1684161199510; Mon, 15 May 2023 07:33:19 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:19 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 05/54] accel/tcg: Implement helper_{ld, st}*_mmu for user-only Date: Mon, 15 May 2023 07:32:24 -0700 Message-Id: <20230515143313.734053-6-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::631; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x631.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org TCG backends may need to defer to a helper to implement the atomicity required by a given operation. Mirror the interface used in system mode. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- include/tcg/tcg-ldst.h | 6 +- accel/tcg/user-exec.c | 393 ++++++++++++++++++++++++++++------------- tcg/tcg.c | 6 +- 3 files changed, 278 insertions(+), 127 deletions(-) diff --git a/include/tcg/tcg-ldst.h b/include/tcg/tcg-ldst.h index 3d897ca942..57fafa14b1 100644 --- a/include/tcg/tcg-ldst.h +++ b/include/tcg/tcg-ldst.h @@ -25,8 +25,6 @@ #ifndef TCG_LDST_H #define TCG_LDST_H -#ifdef CONFIG_SOFTMMU - /* Value zero-extended to tcg register size. */ tcg_target_ulong helper_ldub_mmu(CPUArchState *env, target_ulong addr, MemOpIdx oi, uintptr_t retaddr); @@ -58,10 +56,10 @@ void helper_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, void helper_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, MemOpIdx oi, uintptr_t retaddr); -#else +#ifdef CONFIG_USER_ONLY G_NORETURN void helper_unaligned_ld(CPUArchState *env, target_ulong addr); G_NORETURN void helper_unaligned_st(CPUArchState *env, target_ulong addr); -#endif /* CONFIG_SOFTMMU */ +#endif /* CONFIG_USER_ONLY */ #endif /* TCG_LDST_H */ diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c index b89fa35a83..d9f9766b7f 100644 --- a/accel/tcg/user-exec.c +++ b/accel/tcg/user-exec.c @@ -889,21 +889,6 @@ void page_reset_target_data(target_ulong start, target_ulong last) { } /* The softmmu versions of these helpers are in cputlb.c. */ -/* - * Verify that we have passed the correct MemOp to the correct function. - * - * We could present one function to target code, and dispatch based on - * the MemOp, but so far we have worked hard to avoid an indirect function - * call along the memory path. - */ -static void validate_memop(MemOpIdx oi, MemOp expected) -{ -#ifdef CONFIG_DEBUG_TCG - MemOp have = get_memop(oi) & (MO_SIZE | MO_BSWAP); - assert(have == expected); -#endif -} - void helper_unaligned_ld(CPUArchState *env, target_ulong addr) { cpu_loop_exit_sigbus(env_cpu(env), addr, MMU_DATA_LOAD, GETPC()); @@ -914,10 +899,9 @@ void helper_unaligned_st(CPUArchState *env, target_ulong addr) cpu_loop_exit_sigbus(env_cpu(env), addr, MMU_DATA_STORE, GETPC()); } -static void *cpu_mmu_lookup(CPUArchState *env, target_ulong addr, - MemOpIdx oi, uintptr_t ra, MMUAccessType type) +static void *cpu_mmu_lookup(CPUArchState *env, abi_ptr addr, + MemOp mop, uintptr_t ra, MMUAccessType type) { - MemOp mop = get_memop(oi); int a_bits = get_alignment_bits(mop); void *ret; @@ -933,100 +917,206 @@ static void *cpu_mmu_lookup(CPUArchState *env, target_ulong addr, #include "ldst_atomicity.c.inc" -uint8_t cpu_ldb_mmu(CPUArchState *env, abi_ptr addr, - MemOpIdx oi, uintptr_t ra) +static uint8_t do_ld1_mmu(CPUArchState *env, abi_ptr addr, + MemOp mop, uintptr_t ra) { void *haddr; uint8_t ret; - validate_memop(oi, MO_UB); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); + tcg_debug_assert((mop & MO_SIZE) == MO_8); + haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_LOAD); ret = ldub_p(haddr); clear_helper_retaddr(); + return ret; +} + +tcg_target_ulong helper_ldub_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t ra) +{ + return do_ld1_mmu(env, addr, get_memop(oi), ra); +} + +tcg_target_ulong helper_ldsb_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t ra) +{ + return (int8_t)do_ld1_mmu(env, addr, get_memop(oi), ra); +} + +uint8_t cpu_ldb_mmu(CPUArchState *env, abi_ptr addr, + MemOpIdx oi, uintptr_t ra) +{ + uint8_t ret = do_ld1_mmu(env, addr, get_memop(oi), ra); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); return ret; } +static uint16_t do_ld2_he_mmu(CPUArchState *env, abi_ptr addr, + MemOp mop, uintptr_t ra) +{ + void *haddr; + uint16_t ret; + + tcg_debug_assert((mop & MO_SIZE) == MO_16); + haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_LOAD); + ret = load_atom_2(env, ra, haddr, mop); + clear_helper_retaddr(); + return ret; +} + +tcg_target_ulong helper_lduw_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + uint16_t ret = do_ld2_he_mmu(env, addr, mop, ra); + + if (mop & MO_BSWAP) { + ret = bswap16(ret); + } + return ret; +} + +tcg_target_ulong helper_ldsw_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + int16_t ret = do_ld2_he_mmu(env, addr, mop, ra); + + if (mop & MO_BSWAP) { + ret = bswap16(ret); + } + return ret; +} + uint16_t cpu_ldw_be_mmu(CPUArchState *env, abi_ptr addr, MemOpIdx oi, uintptr_t ra) { - void *haddr; + MemOp mop = get_memop(oi); uint16_t ret; - validate_memop(oi, MO_BEUW); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); - ret = load_atom_2(env, ra, haddr, get_memop(oi)); - clear_helper_retaddr(); + tcg_debug_assert((mop & MO_BSWAP) == MO_BE); + ret = do_ld2_he_mmu(env, addr, mop, ra); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); return cpu_to_be16(ret); } -uint32_t cpu_ldl_be_mmu(CPUArchState *env, abi_ptr addr, - MemOpIdx oi, uintptr_t ra) -{ - void *haddr; - uint32_t ret; - - validate_memop(oi, MO_BEUL); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); - ret = load_atom_4(env, ra, haddr, get_memop(oi)); - clear_helper_retaddr(); - qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); - return cpu_to_be32(ret); -} - -uint64_t cpu_ldq_be_mmu(CPUArchState *env, abi_ptr addr, - MemOpIdx oi, uintptr_t ra) -{ - void *haddr; - uint64_t ret; - - validate_memop(oi, MO_BEUQ); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); - ret = load_atom_8(env, ra, haddr, get_memop(oi)); - clear_helper_retaddr(); - qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); - return cpu_to_be64(ret); -} - uint16_t cpu_ldw_le_mmu(CPUArchState *env, abi_ptr addr, MemOpIdx oi, uintptr_t ra) { - void *haddr; + MemOp mop = get_memop(oi); uint16_t ret; - validate_memop(oi, MO_LEUW); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); - ret = load_atom_2(env, ra, haddr, get_memop(oi)); - clear_helper_retaddr(); + tcg_debug_assert((mop & MO_BSWAP) == MO_LE); + ret = do_ld2_he_mmu(env, addr, mop, ra); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); return cpu_to_le16(ret); } +static uint32_t do_ld4_he_mmu(CPUArchState *env, abi_ptr addr, + MemOp mop, uintptr_t ra) +{ + void *haddr; + uint32_t ret; + + tcg_debug_assert((mop & MO_SIZE) == MO_32); + haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_LOAD); + ret = load_atom_4(env, ra, haddr, mop); + clear_helper_retaddr(); + return ret; +} + +tcg_target_ulong helper_ldul_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + uint32_t ret = do_ld4_he_mmu(env, addr, mop, ra); + + if (mop & MO_BSWAP) { + ret = bswap32(ret); + } + return ret; +} + +tcg_target_ulong helper_ldsl_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + int32_t ret = do_ld4_he_mmu(env, addr, mop, ra); + + if (mop & MO_BSWAP) { + ret = bswap32(ret); + } + return ret; +} + +uint32_t cpu_ldl_be_mmu(CPUArchState *env, abi_ptr addr, + MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + uint32_t ret; + + tcg_debug_assert((mop & MO_BSWAP) == MO_BE); + ret = do_ld4_he_mmu(env, addr, mop, ra); + qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); + return cpu_to_be32(ret); +} + uint32_t cpu_ldl_le_mmu(CPUArchState *env, abi_ptr addr, MemOpIdx oi, uintptr_t ra) { - void *haddr; + MemOp mop = get_memop(oi); uint32_t ret; - validate_memop(oi, MO_LEUL); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); - ret = load_atom_4(env, ra, haddr, get_memop(oi)); - clear_helper_retaddr(); + tcg_debug_assert((mop & MO_BSWAP) == MO_LE); + ret = do_ld4_he_mmu(env, addr, mop, ra); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); return cpu_to_le32(ret); } +static uint64_t do_ld8_he_mmu(CPUArchState *env, abi_ptr addr, + MemOp mop, uintptr_t ra) +{ + void *haddr; + uint64_t ret; + + tcg_debug_assert((mop & MO_SIZE) == MO_64); + haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_LOAD); + ret = load_atom_8(env, ra, haddr, mop); + clear_helper_retaddr(); + return ret; +} + +uint64_t helper_ldq_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + uint64_t ret = do_ld8_he_mmu(env, addr, mop, ra); + + if (mop & MO_BSWAP) { + ret = bswap64(ret); + } + return ret; +} + +uint64_t cpu_ldq_be_mmu(CPUArchState *env, abi_ptr addr, + MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + uint64_t ret; + + tcg_debug_assert((mop & MO_BSWAP) == MO_BE); + ret = do_ld8_he_mmu(env, addr, mop, ra); + qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); + return cpu_to_be64(ret); +} + uint64_t cpu_ldq_le_mmu(CPUArchState *env, abi_ptr addr, MemOpIdx oi, uintptr_t ra) { - void *haddr; + MemOp mop = get_memop(oi); uint64_t ret; - validate_memop(oi, MO_LEUQ); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); - ret = load_atom_8(env, ra, haddr, get_memop(oi)); - clear_helper_retaddr(); + tcg_debug_assert((mop & MO_BSWAP) == MO_LE); + ret = do_ld8_he_mmu(env, addr, mop, ra); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); return cpu_to_le64(ret); } @@ -1037,7 +1127,7 @@ Int128 cpu_ld16_be_mmu(CPUArchState *env, abi_ptr addr, void *haddr; Int128 ret; - validate_memop(oi, MO_128 | MO_BE); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == (MO_128 | MO_BE)); haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); memcpy(&ret, haddr, 16); clear_helper_retaddr(); @@ -1055,7 +1145,7 @@ Int128 cpu_ld16_le_mmu(CPUArchState *env, abi_ptr addr, void *haddr; Int128 ret; - validate_memop(oi, MO_128 | MO_LE); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == (MO_128 | MO_LE)); haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); memcpy(&ret, haddr, 16); clear_helper_retaddr(); @@ -1067,87 +1157,153 @@ Int128 cpu_ld16_le_mmu(CPUArchState *env, abi_ptr addr, return ret; } -void cpu_stb_mmu(CPUArchState *env, abi_ptr addr, uint8_t val, - MemOpIdx oi, uintptr_t ra) +static void do_st1_mmu(CPUArchState *env, abi_ptr addr, uint8_t val, + MemOp mop, uintptr_t ra) { void *haddr; - validate_memop(oi, MO_UB); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE); + tcg_debug_assert((mop & MO_SIZE) == MO_8); + haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_STORE); stb_p(haddr, val); clear_helper_retaddr(); +} + +void helper_stb_mmu(CPUArchState *env, target_ulong addr, uint32_t val, + MemOpIdx oi, uintptr_t ra) +{ + do_st1_mmu(env, addr, val, get_memop(oi), ra); +} + +void cpu_stb_mmu(CPUArchState *env, abi_ptr addr, uint8_t val, + MemOpIdx oi, uintptr_t ra) +{ + do_st1_mmu(env, addr, val, get_memop(oi), ra); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); } +static void do_st2_he_mmu(CPUArchState *env, abi_ptr addr, uint16_t val, + MemOp mop, uintptr_t ra) +{ + void *haddr; + + tcg_debug_assert((mop & MO_SIZE) == MO_16); + haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_STORE); + store_atom_2(env, ra, haddr, mop, val); + clear_helper_retaddr(); +} + +void helper_stw_mmu(CPUArchState *env, target_ulong addr, uint32_t val, + MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + + if (mop & MO_BSWAP) { + val = bswap16(val); + } + do_st2_he_mmu(env, addr, val, mop, ra); +} + void cpu_stw_be_mmu(CPUArchState *env, abi_ptr addr, uint16_t val, MemOpIdx oi, uintptr_t ra) { - void *haddr; + MemOp mop = get_memop(oi); - validate_memop(oi, MO_BEUW); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE); - store_atom_2(env, ra, haddr, get_memop(oi), be16_to_cpu(val)); - clear_helper_retaddr(); - qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); -} - -void cpu_stl_be_mmu(CPUArchState *env, abi_ptr addr, uint32_t val, - MemOpIdx oi, uintptr_t ra) -{ - void *haddr; - - validate_memop(oi, MO_BEUL); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE); - store_atom_4(env, ra, haddr, get_memop(oi), be32_to_cpu(val)); - clear_helper_retaddr(); - qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); -} - -void cpu_stq_be_mmu(CPUArchState *env, abi_ptr addr, uint64_t val, - MemOpIdx oi, uintptr_t ra) -{ - void *haddr; - - validate_memop(oi, MO_BEUQ); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE); - store_atom_8(env, ra, haddr, get_memop(oi), be64_to_cpu(val)); - clear_helper_retaddr(); + tcg_debug_assert((mop & MO_BSWAP) == MO_BE); + do_st2_he_mmu(env, addr, be16_to_cpu(val), mop, ra); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); } void cpu_stw_le_mmu(CPUArchState *env, abi_ptr addr, uint16_t val, MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + + tcg_debug_assert((mop & MO_BSWAP) == MO_LE); + do_st2_he_mmu(env, addr, le16_to_cpu(val), mop, ra); + qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); +} + +static void do_st4_he_mmu(CPUArchState *env, abi_ptr addr, uint32_t val, + MemOp mop, uintptr_t ra) { void *haddr; - validate_memop(oi, MO_LEUW); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE); - store_atom_2(env, ra, haddr, get_memop(oi), le16_to_cpu(val)); + tcg_debug_assert((mop & MO_SIZE) == MO_32); + haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_STORE); + store_atom_4(env, ra, haddr, mop, val); clear_helper_retaddr(); +} + +void helper_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, + MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + + if (mop & MO_BSWAP) { + val = bswap32(val); + } + do_st4_he_mmu(env, addr, val, mop, ra); +} + +void cpu_stl_be_mmu(CPUArchState *env, abi_ptr addr, uint32_t val, + MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + + tcg_debug_assert((mop & MO_BSWAP) == MO_BE); + do_st4_he_mmu(env, addr, be32_to_cpu(val), mop, ra); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); } void cpu_stl_le_mmu(CPUArchState *env, abi_ptr addr, uint32_t val, MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + + tcg_debug_assert((mop & MO_BSWAP) == MO_LE); + do_st4_he_mmu(env, addr, le32_to_cpu(val), mop, ra); + qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); +} + +static void do_st8_he_mmu(CPUArchState *env, abi_ptr addr, uint64_t val, + MemOp mop, uintptr_t ra) { void *haddr; - validate_memop(oi, MO_LEUL); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE); - store_atom_4(env, ra, haddr, get_memop(oi), le32_to_cpu(val)); + tcg_debug_assert((mop & MO_SIZE) == MO_64); + haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_STORE); + store_atom_8(env, ra, haddr, mop, val); clear_helper_retaddr(); +} + +void helper_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, + MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + + if (mop & MO_BSWAP) { + val = bswap64(val); + } + do_st8_he_mmu(env, addr, val, mop, ra); +} + +void cpu_stq_be_mmu(CPUArchState *env, abi_ptr addr, uint64_t val, + MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + + tcg_debug_assert((mop & MO_BSWAP) == MO_BE); + do_st8_he_mmu(env, addr, cpu_to_be64(val), mop, ra); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); } void cpu_stq_le_mmu(CPUArchState *env, abi_ptr addr, uint64_t val, MemOpIdx oi, uintptr_t ra) { - void *haddr; + MemOp mop = get_memop(oi); - validate_memop(oi, MO_LEUQ); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE); - store_atom_8(env, ra, haddr, get_memop(oi), le64_to_cpu(val)); - clear_helper_retaddr(); + tcg_debug_assert((mop & MO_BSWAP) == MO_LE); + do_st8_he_mmu(env, addr, cpu_to_le64(val), mop, ra); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); } @@ -1156,7 +1312,7 @@ void cpu_st16_be_mmu(CPUArchState *env, abi_ptr addr, { void *haddr; - validate_memop(oi, MO_128 | MO_BE); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == (MO_128 | MO_BE)); haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE); if (!HOST_BIG_ENDIAN) { val = bswap128(val); @@ -1171,7 +1327,7 @@ void cpu_st16_le_mmu(CPUArchState *env, abi_ptr addr, { void *haddr; - validate_memop(oi, MO_128 | MO_LE); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == (MO_128 | MO_LE)); haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE); if (HOST_BIG_ENDIAN) { val = bswap128(val); @@ -1269,7 +1425,6 @@ uint64_t cpu_ldq_code_mmu(CPUArchState *env, abi_ptr addr, void *haddr; uint64_t ret; - validate_memop(oi, MO_BEUQ); haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); ret = ldq_p(haddr); clear_helper_retaddr(); diff --git a/tcg/tcg.c b/tcg/tcg.c index a26410fb68..dc1c0fdf2b 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -197,8 +197,7 @@ static void tcg_out_st_helper_args(TCGContext *s, const TCGLabelQemuLdst *l, const TCGLdstHelperParam *p) __attribute__((unused)); -#ifdef CONFIG_SOFTMMU -static void * const qemu_ld_helpers[MO_SSIZE + 1] = { +static void * const qemu_ld_helpers[MO_SSIZE + 1] __attribute__((unused)) = { [MO_UB] = helper_ldub_mmu, [MO_SB] = helper_ldsb_mmu, [MO_UW] = helper_lduw_mmu, @@ -210,13 +209,12 @@ static void * const qemu_ld_helpers[MO_SSIZE + 1] = { #endif }; -static void * const qemu_st_helpers[MO_SIZE + 1] = { +static void * const qemu_st_helpers[MO_SIZE + 1] __attribute__((unused)) = { [MO_8] = helper_stb_mmu, [MO_16] = helper_stw_mmu, [MO_32] = helper_stl_mmu, [MO_64] = helper_stq_mmu, }; -#endif TCGContext tcg_init_ctx; __thread TCGContext *tcg_ctx; From patchwork Mon May 15 14:32:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681939 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1149091wrr; Mon, 15 May 2023 07:36:36 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5onN9Xw15OPX9pPbJdCMBi0PSVGiTbcQc9g5ODsBZlu9aw7ISC+XXm37S6cQRlBa6q6HB3 X-Received: by 2002:ac8:580e:0:b0:3ef:52ac:10d2 with SMTP id g14-20020ac8580e000000b003ef52ac10d2mr56323958qtg.43.1684161396404; Mon, 15 May 2023 07:36:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161396; cv=none; d=google.com; s=arc-20160816; b=kSQWHXuB7VoTRDL0MuyoR2NJ76p+KAEFkyAA6HwOnU5QOA0py9ofBoTm2RIUztAVg7 +XtAvtsM7ZtOk2UkQjhp/MVeAffDy7PW3thiR2h6nJmJjnD1+t5UzyDxKzLcEmHPF6A4 8jmTN8ShQMn7s0mOKZksOUPgXFNQtQmjHZ8mWgJYybDlTOUob0w/o2faJHjQBsYG3gt/ Dn0sVVUm2TmANIwvGPer63I6tUvA2Rr5vVMjLG/DFcR4rqopcbpx+tTcYENQYeqpbqJK 9s3XU+TzT1ytvZR8dM4Sczw+Ynde4Nh/8P4XQdFlV1M4dUY8K1TVbhql95MNanKGbKLL Y8Uw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=aZUd62xGtvRPa1FRa9Mi8zYQ92RuZVwVBZs3Zj8eEbg=; b=dLcJgAodkaeZw2386xqP2l4BkIeLT38uKhHkkRbedWvk9XxSsOZSg4fm+OMjUq2/6T BsKoO39HDv9OGMaES3OmPeOKYL96GerJMwwAJNn5unjs8AHls5pE5BetuP4dZb2hVcYi jDDLuAt431qp7P8jLXOSROGb3ZsFX1F3NCTyClbEiUEJFF1Ox4PAuOI4Lzm1Oxqfio1J Xooj2QWTYKaN4PbQPylf7i3Tr34/HzQUCGy2XkY6cVtvGrgbamjxYHT1g/hydKpoDOXx PAf2F+EwXhNNuKWPuza/KMMdlmlx9xM2q4BgjD7bcyEbDsHi9rTVgLmAZNyc6qAzEr6F 1xMQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Ea8lcdlR; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id h21-20020ac87d55000000b003f4e31d32fcsi7456323qtb.52.2023.05.15.07.36.36 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:36:36 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Ea8lcdlR; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZHz-0007P6-Pi; Mon, 15 May 2023 10:35:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZGT-0006jv-Kn for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:30 -0400 Received: from mail-pl1-x634.google.com ([2607:f8b0:4864:20::634]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGM-0004rv-W5 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:28 -0400 Received: by mail-pl1-x634.google.com with SMTP id d9443c01a7336-1aaf91ae451so120241875ad.1 for ; Mon, 15 May 2023 07:33:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161200; x=1686753200; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aZUd62xGtvRPa1FRa9Mi8zYQ92RuZVwVBZs3Zj8eEbg=; b=Ea8lcdlRv2hSkVaF/8DUszGBYxhDv6xHJp0xPkq3fSEgXjxuIa0QWJSh1ySSeSjVSr Ofkvf94pvgSODcPNNaT72VClfKVRjZYyWr4j2nQiQwware4w8uDK20fPbd4sk1FSac3X mvMLdTOWH7yZlZGKH1ikgNZoUUZxh4mLMas1IHQ69+QyzW+zciBegyPpo4LKOBMMwXVO cX9wYmxd2TgBoziBqnlXtJUyvIugzKFMc1/QXkXD2GuSKGxFHqzY7FUR7zGGttRSjVp+ 3AHpzfnhrWcqr3gJ2kabUeian7v5Wmuzn2FigljlrPLULMYl4O7tPaaUlTKFAz0Apdyg 1iKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161200; x=1686753200; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aZUd62xGtvRPa1FRa9Mi8zYQ92RuZVwVBZs3Zj8eEbg=; b=Xiw7iJQFzVGG1zff16scfsV/E1vBxjmIbhT4du7u06JQEP/1G1uyNmtMFk4k3eIL+8 n+G2CFfxLZaYU6U2e0sZq7ky3Ty0UzDzHpz0ntaORDxT/nTCTVSqtFx4uYb6N+h12CvD M9IYWhtBh3MjV8gUiF/Bp4Msl2dzH8oB9vEHzQaU4JRERnSDvW/z838ZSEj+i6XKNz72 8zyG7ivnmEcWo9WuBba/4SNn3p/872q1ZCUgsaUMxq83944UtaTDGyqxV28Qp/lvtP1N ZZ2YhD1fe2iL0I9PpOKE6MRc4mRWiuXpkru4K2rUFwhElTh0chNS9nb59MwB0GzYzrZ0 3E4A== X-Gm-Message-State: AC+VfDzvNbSsakzzQvce/ldktsK7sQhuMBs83L/QhPTtiIYaqUg3i3TV M599nb6NNoOrOpWjSxxxOQ/iiuFw8AQs9pen8VM= X-Received: by 2002:a17:902:ce82:b0:1a9:9a18:345c with SMTP id f2-20020a170902ce8200b001a99a18345cmr42150632plg.44.1684161200358; Mon, 15 May 2023 07:33:20 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:19 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 06/54] tcg/tci: Use helper_{ld,st}*_mmu for user-only Date: Mon, 15 May 2023 07:32:25 -0700 Message-Id: <20230515143313.734053-7-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::634; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x634.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org We can now fold these two pieces of code. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/tci.c | 89 ------------------------------------------------------- 1 file changed, 89 deletions(-) diff --git a/tcg/tci.c b/tcg/tci.c index 5bde2e1f2e..15f2f8c463 100644 --- a/tcg/tci.c +++ b/tcg/tci.c @@ -292,7 +292,6 @@ static uint64_t tci_qemu_ld(CPUArchState *env, target_ulong taddr, MemOp mop = get_memop(oi); uintptr_t ra = (uintptr_t)tb_ptr; -#ifdef CONFIG_SOFTMMU switch (mop & MO_SSIZE) { case MO_UB: return helper_ldub_mmu(env, taddr, oi, ra); @@ -311,58 +310,6 @@ static uint64_t tci_qemu_ld(CPUArchState *env, target_ulong taddr, default: g_assert_not_reached(); } -#else - void *haddr = g2h(env_cpu(env), taddr); - unsigned a_mask = (1u << get_alignment_bits(mop)) - 1; - uint64_t ret; - - set_helper_retaddr(ra); - if (taddr & a_mask) { - helper_unaligned_ld(env, taddr); - } - switch (mop & (MO_BSWAP | MO_SSIZE)) { - case MO_UB: - ret = ldub_p(haddr); - break; - case MO_SB: - ret = ldsb_p(haddr); - break; - case MO_LEUW: - ret = lduw_le_p(haddr); - break; - case MO_LESW: - ret = ldsw_le_p(haddr); - break; - case MO_LEUL: - ret = (uint32_t)ldl_le_p(haddr); - break; - case MO_LESL: - ret = (int32_t)ldl_le_p(haddr); - break; - case MO_LEUQ: - ret = ldq_le_p(haddr); - break; - case MO_BEUW: - ret = lduw_be_p(haddr); - break; - case MO_BESW: - ret = ldsw_be_p(haddr); - break; - case MO_BEUL: - ret = (uint32_t)ldl_be_p(haddr); - break; - case MO_BESL: - ret = (int32_t)ldl_be_p(haddr); - break; - case MO_BEUQ: - ret = ldq_be_p(haddr); - break; - default: - g_assert_not_reached(); - } - clear_helper_retaddr(); - return ret; -#endif } static void tci_qemu_st(CPUArchState *env, target_ulong taddr, uint64_t val, @@ -371,7 +318,6 @@ static void tci_qemu_st(CPUArchState *env, target_ulong taddr, uint64_t val, MemOp mop = get_memop(oi); uintptr_t ra = (uintptr_t)tb_ptr; -#ifdef CONFIG_SOFTMMU switch (mop & MO_SIZE) { case MO_UB: helper_stb_mmu(env, taddr, val, oi, ra); @@ -388,41 +334,6 @@ static void tci_qemu_st(CPUArchState *env, target_ulong taddr, uint64_t val, default: g_assert_not_reached(); } -#else - void *haddr = g2h(env_cpu(env), taddr); - unsigned a_mask = (1u << get_alignment_bits(mop)) - 1; - - set_helper_retaddr(ra); - if (taddr & a_mask) { - helper_unaligned_st(env, taddr); - } - switch (mop & (MO_BSWAP | MO_SIZE)) { - case MO_UB: - stb_p(haddr, val); - break; - case MO_LEUW: - stw_le_p(haddr, val); - break; - case MO_LEUL: - stl_le_p(haddr, val); - break; - case MO_LEUQ: - stq_le_p(haddr, val); - break; - case MO_BEUW: - stw_be_p(haddr, val); - break; - case MO_BEUL: - stl_be_p(haddr, val); - break; - case MO_BEUQ: - stq_be_p(haddr, val); - break; - default: - g_assert_not_reached(); - } - clear_helper_retaddr(); -#endif } #if TCG_TARGET_REG_BITS == 64 From patchwork Mon May 15 14:32:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681936 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1148742wrr; Mon, 15 May 2023 07:35:59 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6JoZp16W/pgkCEynITsZhEFeB1uAIHD9mHujpXU7TIHtBjhxDolHBbOlVK6PXs6K0hFMOT X-Received: by 2002:a05:622a:145:b0:3f5:15bf:94e5 with SMTP id v5-20020a05622a014500b003f515bf94e5mr12994398qtw.17.1684161358832; Mon, 15 May 2023 07:35:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161358; cv=none; d=google.com; s=arc-20160816; b=NH2xCBlIUQPgJEUN6WUW85OEueCPPob8S9iQSXFTt00pmwoq8mJYcofS5PVkIAELIy FoMA9sjSpM83J2jADI0rN1WSdYHALSiTcv0v6JQ3OIXZi2O7KpFzhXiHjsucGk13OsRR ka3xw4IFJNfaCLFjgbHDZYGDQMkIBc4VJ/NbwJ+KQX1S0mbpdD3jE2vObo7U0aKnCGG3 ZJsLY8RzrolHM8FKrTr4r7SJ3wADCNwXY22tyV7rEpAcI/qx0Ez2EzdPNUaClDl05My8 W+xyjrr2y/DOnVFlLY0iVFCaEUvOyL+c1uD14ylojmUHKzYnOQU3pJ7oldJg9h75XxNI s+tw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=/r4SsAkFdgG4apSOLHQZbRPQcgMVnRVI69gDf5G5KZ0=; b=idIy6K+w9RSYSNZhQH+XSJtRjOzeTfANzVhzYZw5PW6lPW0KlwdAekbbMhY1ss1QcR czhTUJsudZLm1T4fwYuHyihyePxosKEfmbXGUoTGZUglM4vMSAfOKGJTK0Q5gfOU0eel t3ZF4wRKWuCVQ8vCkrUZ26x4R/ApryOfzzCvFadaSrrL03m1GKekyHs4iEwylwMyWAiS h9tXNpXM+Q3vWXyXRlgUEN+1sL1ORMm2xNB9onFzgEGC27l055e8tnTOkz1q4nLNEZW8 4icCtWjbP0XcKoWkZ3E32s9YSQWX46foHEmtssIhfJc7P6atO2F0KiOKWjxfWVaYol5J mDcw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=VrW8XJdr; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id y20-20020ac85f54000000b003f51ff7db7asi2477951qta.193.2023.05.15.07.35.58 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:35:58 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=VrW8XJdr; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZI6-0007WE-OO; Mon, 15 May 2023 10:35:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZGX-0006lk-BO for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:33 -0400 Received: from mail-pl1-x62a.google.com ([2607:f8b0:4864:20::62a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGN-0004sJ-0C for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:32 -0400 Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-1ab05018381so118660655ad.2 for ; Mon, 15 May 2023 07:33:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161201; x=1686753201; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/r4SsAkFdgG4apSOLHQZbRPQcgMVnRVI69gDf5G5KZ0=; b=VrW8XJdrnfiaK6Gt46+ojp4CjCgSR+022hcGm+sziLEYpXB462M8hh+CSoPV7pChOT Z1luyTn1uRePF3SLQc3vZXycxKG7Z8GSDv7LKpskZZHW7iW3/3QjBjVgGYNCWzTMmq9+ gTN9Q6RL8XDJOLF/8m1ifjep4bP8RnWcM5pC38VO8BhNqWz0qHkC+Uh2uGekxrrw2sW8 NUEYRgKovm/0GmX6VuIb5NFc7ugCxTuLHTJ9py1zb0mj9gKpPxRhGFwx5zBEHwsIarBy 3UApUnzHcUyp8TnbEs2aXQzhCzCsoI1wxOol98xw/pEOKs0bqf3VI9MKTxU5NEvAmCxX LIrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161201; x=1686753201; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/r4SsAkFdgG4apSOLHQZbRPQcgMVnRVI69gDf5G5KZ0=; b=dKwn+PQciZ8XAFjiyeWAfEor/jLnKGIBtqRdWMBzpPp1NMc6vmEKC+xRMdKx3UmUsy M+d4i4U7UH8r2n/nTyg9HgNorNDwZ4GwaThARR1dKnBwyKtnIDmSl9ENOchWvyIShZxN yCj8mDI95AP00sTH790tR1XBtTaDjPtFI7jvtMuCakFn1Ty7WCY9L5ej8XyVXZuXlR/Y pe0+YsZQ6RT3s/FbdOc9GQx5R5u+YOp0sZSRXgaJS/nrozWHbMmhw9g/7FGxQgSIm0D0 Rvr1LygorENXtFzgTm7PgzfYkKHpRhFG6SdLCMtjBKH1QiM+qPwdhr3flhuzEvsSXUIE s7Eg== X-Gm-Message-State: AC+VfDxq8zzcgVLX1sWHqOlT+stZjXonGxbVeZfPPU+xfiPG/ZuMBNFL yuMzCasMh1ekpiiMM0pudhifZy1/3cP8Da9NOCs= X-Received: by 2002:a17:903:41ca:b0:1aa:d545:462e with SMTP id u10-20020a17090341ca00b001aad545462emr47055023ple.13.1684161201100; Mon, 15 May 2023 07:33:21 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:20 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 07/54] tcg: Add 128-bit guest memory primitives Date: Mon, 15 May 2023 07:32:26 -0700 Message-Id: <20230515143313.734053-8-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62a; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- v5: Dropped r-b because of MO_ATOM_* reorg --- accel/tcg/tcg-runtime.h | 3 + include/tcg/tcg-ldst.h | 4 + accel/tcg/cputlb.c | 399 +++++++++++++++++++++++++-------- accel/tcg/user-exec.c | 94 ++++++-- tcg/tcg-op.c | 173 +++++++++----- accel/tcg/ldst_atomicity.c.inc | 184 +++++++++++++++ 6 files changed, 679 insertions(+), 178 deletions(-) diff --git a/accel/tcg/tcg-runtime.h b/accel/tcg/tcg-runtime.h index b8e6421c8a..d9adc646c1 100644 --- a/accel/tcg/tcg-runtime.h +++ b/accel/tcg/tcg-runtime.h @@ -39,6 +39,9 @@ DEF_HELPER_FLAGS_1(exit_atomic, TCG_CALL_NO_WG, noreturn, env) DEF_HELPER_FLAGS_3(memset, TCG_CALL_NO_RWG, ptr, ptr, int, ptr) #endif /* IN_HELPER_PROTO */ +DEF_HELPER_FLAGS_3(ld_i128, TCG_CALL_NO_WG, i128, env, tl, i32) +DEF_HELPER_FLAGS_4(st_i128, TCG_CALL_NO_WG, void, env, tl, i128, i32) + DEF_HELPER_FLAGS_5(atomic_cmpxchgb, TCG_CALL_NO_WG, i32, env, tl, i32, i32, i32) DEF_HELPER_FLAGS_5(atomic_cmpxchgw_be, TCG_CALL_NO_WG, diff --git a/include/tcg/tcg-ldst.h b/include/tcg/tcg-ldst.h index 57fafa14b1..64f48e6990 100644 --- a/include/tcg/tcg-ldst.h +++ b/include/tcg/tcg-ldst.h @@ -34,6 +34,8 @@ tcg_target_ulong helper_ldul_mmu(CPUArchState *env, target_ulong addr, MemOpIdx oi, uintptr_t retaddr); uint64_t helper_ldq_mmu(CPUArchState *env, target_ulong addr, MemOpIdx oi, uintptr_t retaddr); +Int128 helper_ld16_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t retaddr); /* Value sign-extended to tcg register size. */ tcg_target_ulong helper_ldsb_mmu(CPUArchState *env, target_ulong addr, @@ -55,6 +57,8 @@ void helper_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, MemOpIdx oi, uintptr_t retaddr); void helper_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, MemOpIdx oi, uintptr_t retaddr); +void helper_st16_mmu(CPUArchState *env, target_ulong addr, Int128 val, + MemOpIdx oi, uintptr_t retaddr); #ifdef CONFIG_USER_ONLY diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index 34796ef568..49e49f75a4 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -40,6 +40,7 @@ #include "qemu/plugin-memory.h" #endif #include "tcg/tcg-ldst.h" +#include "exec/helper-proto.h" /* DEBUG defines, enable DEBUG_TLB_LOG to log to the CPU_LOG_MMU target */ /* #define DEBUG_TLB */ @@ -2162,6 +2163,31 @@ static uint64_t do_ld_whole_be8(CPUArchState *env, uintptr_t ra, return (ret_be << (p->size * 8)) | x; } +/** + * do_ld_parts_be16 + * @p: translation parameters + * @ret_be: accumulated data + * + * As do_ld_bytes_beN, but with one atomic load. + * 16 aligned bytes are guaranteed to cover the load. + */ +static Int128 do_ld_whole_be16(CPUArchState *env, uintptr_t ra, + MMULookupPageData *p, uint64_t ret_be) +{ + int o = p->addr & 15; + Int128 x, y = load_atomic16_or_exit(env, ra, p->haddr - o); + int size = p->size; + + if (!HOST_BIG_ENDIAN) { + y = bswap128(y); + } + y = int128_lshift(y, o * 8); + y = int128_urshift(y, (16 - size) * 8); + x = int128_make64(ret_be); + x = int128_lshift(x, size * 8); + return int128_or(x, y); +} + /* * Wrapper for the above. */ @@ -2211,6 +2237,63 @@ static uint64_t do_ld_beN(CPUArchState *env, MMULookupPageData *p, } } +/* + * Wrapper for the above, for 8 < size < 16. + */ +static Int128 do_ld16_beN(CPUArchState *env, MMULookupPageData *p, + uint64_t a, int mmu_idx, MemOp mop, uintptr_t ra) +{ + int size = p->size; + uint64_t b; + MemOp atom; + + if (unlikely(p->flags & TLB_MMIO)) { + p->size = size - 8; + a = do_ld_mmio_beN(env, p, a, mmu_idx, MMU_DATA_LOAD, ra); + p->addr += p->size; + p->size = 8; + b = do_ld_mmio_beN(env, p, 0, mmu_idx, MMU_DATA_LOAD, ra); + return int128_make128(b, a); + } + + /* + * It is a given that we cross a page and therefore there is no + * atomicity for the load as a whole, but subobjects may need attention. + */ + atom = mop & MO_ATOM_MASK; + switch (atom) { + case MO_ATOM_SUBALIGN: + p->size = size - 8; + a = do_ld_parts_beN(p, a); + p->haddr += size - 8; + p->size = 8; + b = do_ld_parts_beN(p, 0); + break; + + case MO_ATOM_WITHIN16_PAIR: + /* Since size > 8, this is the half that must be atomic. */ + return do_ld_whole_be16(env, ra, p, a); + + case MO_ATOM_IFALIGN_PAIR: + /* + * Since size > 8, both halves are misaligned, + * and so neither is atomic. + */ + case MO_ATOM_IFALIGN: + case MO_ATOM_WITHIN16: + case MO_ATOM_NONE: + p->size = size - 8; + a = do_ld_bytes_beN(p, a); + b = ldq_be_p(p->haddr + size - 8); + break; + + default: + g_assert_not_reached(); + } + + return int128_make128(b, a); +} + static uint8_t do_ld_1(CPUArchState *env, MMULookupPageData *p, int mmu_idx, MMUAccessType type, uintptr_t ra) { @@ -2399,6 +2482,80 @@ tcg_target_ulong helper_ldsl_mmu(CPUArchState *env, target_ulong addr, return (int32_t)helper_ldul_mmu(env, addr, oi, retaddr); } +static Int128 do_ld16_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t ra) +{ + MMULookupLocals l; + bool crosspage; + uint64_t a, b; + Int128 ret; + int first; + + crosspage = mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD, &l); + if (likely(!crosspage)) { + /* Perform the load host endian. */ + if (unlikely(l.page[0].flags & TLB_MMIO)) { + QEMU_IOTHREAD_LOCK_GUARD(); + a = io_readx(env, l.page[0].full, l.mmu_idx, addr, + ra, MMU_DATA_LOAD, MO_64); + b = io_readx(env, l.page[0].full, l.mmu_idx, addr + 8, + ra, MMU_DATA_LOAD, MO_64); + ret = int128_make128(HOST_BIG_ENDIAN ? b : a, + HOST_BIG_ENDIAN ? a : b); + } else { + ret = load_atom_16(env, ra, l.page[0].haddr, l.memop); + } + if (l.memop & MO_BSWAP) { + ret = bswap128(ret); + } + return ret; + } + + first = l.page[0].size; + if (first == 8) { + MemOp mop8 = (l.memop & ~MO_SIZE) | MO_64; + + a = do_ld_8(env, &l.page[0], l.mmu_idx, MMU_DATA_LOAD, mop8, ra); + b = do_ld_8(env, &l.page[1], l.mmu_idx, MMU_DATA_LOAD, mop8, ra); + if ((mop8 & MO_BSWAP) == MO_LE) { + ret = int128_make128(a, b); + } else { + ret = int128_make128(b, a); + } + return ret; + } + + if (first < 8) { + a = do_ld_beN(env, &l.page[0], 0, l.mmu_idx, + MMU_DATA_LOAD, l.memop, ra); + ret = do_ld16_beN(env, &l.page[1], a, l.mmu_idx, l.memop, ra); + } else { + ret = do_ld16_beN(env, &l.page[0], 0, l.mmu_idx, l.memop, ra); + b = int128_getlo(ret); + ret = int128_lshift(ret, l.page[1].size * 8); + a = int128_gethi(ret); + b = do_ld_beN(env, &l.page[1], b, l.mmu_idx, + MMU_DATA_LOAD, l.memop, ra); + ret = int128_make128(b, a); + } + if ((l.memop & MO_BSWAP) == MO_LE) { + ret = bswap128(ret); + } + return ret; +} + +Int128 helper_ld16_mmu(CPUArchState *env, target_ulong addr, + uint32_t oi, uintptr_t retaddr) +{ + tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_128); + return do_ld16_mmu(env, addr, oi, retaddr); +} + +Int128 helper_ld_i128(CPUArchState *env, target_ulong addr, uint32_t oi) +{ + return helper_ld16_mmu(env, addr, oi, GETPC()); +} + /* * Load helpers for cpu_ldst.h. */ @@ -2487,59 +2644,23 @@ uint64_t cpu_ldq_le_mmu(CPUArchState *env, abi_ptr addr, Int128 cpu_ld16_be_mmu(CPUArchState *env, abi_ptr addr, MemOpIdx oi, uintptr_t ra) { - MemOp mop = get_memop(oi); - int mmu_idx = get_mmuidx(oi); - MemOpIdx new_oi; - unsigned a_bits; - uint64_t h, l; + Int128 ret; - tcg_debug_assert((mop & (MO_BSWAP|MO_SSIZE)) == (MO_BE|MO_128)); - a_bits = get_alignment_bits(mop); - - /* Handle CPU specific unaligned behaviour */ - if (addr & ((1 << a_bits) - 1)) { - cpu_unaligned_access(env_cpu(env), addr, MMU_DATA_LOAD, - mmu_idx, ra); - } - - /* Construct an unaligned 64-bit replacement MemOpIdx. */ - mop = (mop & ~(MO_SIZE | MO_AMASK)) | MO_64 | MO_UNALN; - new_oi = make_memop_idx(mop, mmu_idx); - - h = helper_ldq_mmu(env, addr, new_oi, ra); - l = helper_ldq_mmu(env, addr + 8, new_oi, ra); - - qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); - return int128_make128(l, h); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP|MO_SIZE)) == (MO_BE|MO_128)); + ret = do_ld16_mmu(env, addr, oi, ra); + plugin_load_cb(env, addr, oi); + return ret; } Int128 cpu_ld16_le_mmu(CPUArchState *env, abi_ptr addr, MemOpIdx oi, uintptr_t ra) { - MemOp mop = get_memop(oi); - int mmu_idx = get_mmuidx(oi); - MemOpIdx new_oi; - unsigned a_bits; - uint64_t h, l; + Int128 ret; - tcg_debug_assert((mop & (MO_BSWAP|MO_SSIZE)) == (MO_LE|MO_128)); - a_bits = get_alignment_bits(mop); - - /* Handle CPU specific unaligned behaviour */ - if (addr & ((1 << a_bits) - 1)) { - cpu_unaligned_access(env_cpu(env), addr, MMU_DATA_LOAD, - mmu_idx, ra); - } - - /* Construct an unaligned 64-bit replacement MemOpIdx. */ - mop = (mop & ~(MO_SIZE | MO_AMASK)) | MO_64 | MO_UNALN; - new_oi = make_memop_idx(mop, mmu_idx); - - l = helper_ldq_mmu(env, addr, new_oi, ra); - h = helper_ldq_mmu(env, addr + 8, new_oi, ra); - - qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); - return int128_make128(l, h); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP|MO_SIZE)) == (MO_LE|MO_128)); + ret = do_ld16_mmu(env, addr, oi, ra); + plugin_load_cb(env, addr, oi); + return ret; } /* @@ -2625,6 +2746,60 @@ static uint64_t do_st_leN(CPUArchState *env, MMULookupPageData *p, } } +/* + * Wrapper for the above, for 8 < size < 16. + */ +static uint64_t do_st16_leN(CPUArchState *env, MMULookupPageData *p, + Int128 val_le, int mmu_idx, + MemOp mop, uintptr_t ra) +{ + int size = p->size; + MemOp atom; + + if (unlikely(p->flags & TLB_MMIO)) { + p->size = 8; + do_st_mmio_leN(env, p, int128_getlo(val_le), mmu_idx, ra); + p->size = size - 8; + p->addr += 8; + return do_st_mmio_leN(env, p, int128_gethi(val_le), mmu_idx, ra); + } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) { + return int128_gethi(val_le) >> ((size - 8) * 8); + } + + /* + * It is a given that we cross a page and therefore there is no atomicity + * for the store as a whole, but subobjects may need attention. + */ + atom = mop & MO_ATOM_MASK; + switch (atom) { + case MO_ATOM_SUBALIGN: + store_parts_leN(p->haddr, 8, int128_getlo(val_le)); + return store_parts_leN(p->haddr + 8, p->size - 8, + int128_gethi(val_le)); + + case MO_ATOM_WITHIN16_PAIR: + /* Since size > 8, this is the half that must be atomic. */ + if (!HAVE_al16) { + cpu_loop_exit_atomic(env_cpu(env), ra); + } + return store_whole_le16(p->haddr, p->size, val_le); + + case MO_ATOM_IFALIGN_PAIR: + /* + * Since size > 8, both halves are misaligned, + * and so neither is atomic. + */ + case MO_ATOM_IFALIGN: + case MO_ATOM_NONE: + stq_le_p(p->haddr, int128_getlo(val_le)); + return store_bytes_leN(p->haddr + 8, p->size - 8, + int128_gethi(val_le)); + + default: + g_assert_not_reached(); + } +} + static void do_st_1(CPUArchState *env, MMULookupPageData *p, uint8_t val, int mmu_idx, uintptr_t ra) { @@ -2781,6 +2956,80 @@ void helper_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, do_st8_mmu(env, addr, val, oi, retaddr); } +static void do_st16_mmu(CPUArchState *env, target_ulong addr, Int128 val, + MemOpIdx oi, uintptr_t ra) +{ + MMULookupLocals l; + bool crosspage; + uint64_t a, b; + int first; + + crosspage = mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE, &l); + if (likely(!crosspage)) { + /* Swap to host endian if necessary, then store. */ + if (l.memop & MO_BSWAP) { + val = bswap128(val); + } + if (unlikely(l.page[0].flags & TLB_MMIO)) { + QEMU_IOTHREAD_LOCK_GUARD(); + if (HOST_BIG_ENDIAN) { + b = int128_getlo(val), a = int128_gethi(val); + } else { + a = int128_getlo(val), b = int128_gethi(val); + } + io_writex(env, l.page[0].full, l.mmu_idx, a, addr, ra, MO_64); + io_writex(env, l.page[0].full, l.mmu_idx, b, addr + 8, ra, MO_64); + } else if (unlikely(l.page[0].flags & TLB_DISCARD_WRITE)) { + /* nothing */ + } else { + store_atom_16(env, ra, l.page[0].haddr, l.memop, val); + } + return; + } + + first = l.page[0].size; + if (first == 8) { + MemOp mop8 = (l.memop & ~(MO_SIZE | MO_BSWAP)) | MO_64; + + if (l.memop & MO_BSWAP) { + val = bswap128(val); + } + if (HOST_BIG_ENDIAN) { + b = int128_getlo(val), a = int128_gethi(val); + } else { + a = int128_getlo(val), b = int128_gethi(val); + } + do_st_8(env, &l.page[0], a, l.mmu_idx, mop8, ra); + do_st_8(env, &l.page[1], b, l.mmu_idx, mop8, ra); + return; + } + + if ((l.memop & MO_BSWAP) != MO_LE) { + val = bswap128(val); + } + if (first < 8) { + do_st_leN(env, &l.page[0], int128_getlo(val), l.mmu_idx, l.memop, ra); + val = int128_urshift(val, first * 8); + do_st16_leN(env, &l.page[1], val, l.mmu_idx, l.memop, ra); + } else { + b = do_st16_leN(env, &l.page[0], val, l.mmu_idx, l.memop, ra); + do_st_leN(env, &l.page[1], b, l.mmu_idx, l.memop, ra); + } +} + +void helper_st16_mmu(CPUArchState *env, target_ulong addr, Int128 val, + MemOpIdx oi, uintptr_t retaddr) +{ + tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_128); + do_st16_mmu(env, addr, val, oi, retaddr); +} + +void helper_st_i128(CPUArchState *env, target_ulong addr, Int128 val, + MemOpIdx oi) +{ + helper_st16_mmu(env, addr, val, oi, GETPC()); +} + /* * Store Helpers for cpu_ldst.h */ @@ -2845,58 +3094,20 @@ void cpu_stq_le_mmu(CPUArchState *env, target_ulong addr, uint64_t val, plugin_store_cb(env, addr, oi); } -void cpu_st16_be_mmu(CPUArchState *env, abi_ptr addr, Int128 val, - MemOpIdx oi, uintptr_t ra) +void cpu_st16_be_mmu(CPUArchState *env, target_ulong addr, Int128 val, + MemOpIdx oi, uintptr_t retaddr) { - MemOp mop = get_memop(oi); - int mmu_idx = get_mmuidx(oi); - MemOpIdx new_oi; - unsigned a_bits; - - tcg_debug_assert((mop & (MO_BSWAP|MO_SSIZE)) == (MO_BE|MO_128)); - a_bits = get_alignment_bits(mop); - - /* Handle CPU specific unaligned behaviour */ - if (addr & ((1 << a_bits) - 1)) { - cpu_unaligned_access(env_cpu(env), addr, MMU_DATA_STORE, - mmu_idx, ra); - } - - /* Construct an unaligned 64-bit replacement MemOpIdx. */ - mop = (mop & ~(MO_SIZE | MO_AMASK)) | MO_64 | MO_UNALN; - new_oi = make_memop_idx(mop, mmu_idx); - - helper_stq_mmu(env, addr, int128_gethi(val), new_oi, ra); - helper_stq_mmu(env, addr + 8, int128_getlo(val), new_oi, ra); - - qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP|MO_SIZE)) == (MO_BE|MO_128)); + do_st16_mmu(env, addr, val, oi, retaddr); + plugin_store_cb(env, addr, oi); } -void cpu_st16_le_mmu(CPUArchState *env, abi_ptr addr, Int128 val, - MemOpIdx oi, uintptr_t ra) +void cpu_st16_le_mmu(CPUArchState *env, target_ulong addr, Int128 val, + MemOpIdx oi, uintptr_t retaddr) { - MemOp mop = get_memop(oi); - int mmu_idx = get_mmuidx(oi); - MemOpIdx new_oi; - unsigned a_bits; - - tcg_debug_assert((mop & (MO_BSWAP|MO_SSIZE)) == (MO_LE|MO_128)); - a_bits = get_alignment_bits(mop); - - /* Handle CPU specific unaligned behaviour */ - if (addr & ((1 << a_bits) - 1)) { - cpu_unaligned_access(env_cpu(env), addr, MMU_DATA_STORE, - mmu_idx, ra); - } - - /* Construct an unaligned 64-bit replacement MemOpIdx. */ - mop = (mop & ~(MO_SIZE | MO_AMASK)) | MO_64 | MO_UNALN; - new_oi = make_memop_idx(mop, mmu_idx); - - helper_stq_mmu(env, addr, int128_getlo(val), new_oi, ra); - helper_stq_mmu(env, addr + 8, int128_gethi(val), new_oi, ra); - - qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); + tcg_debug_assert((get_memop(oi) & (MO_BSWAP|MO_SIZE)) == (MO_LE|MO_128)); + do_st16_mmu(env, addr, val, oi, retaddr); + plugin_store_cb(env, addr, oi); } #include "ldst_common.c.inc" diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c index d9f9766b7f..8f86254eb4 100644 --- a/accel/tcg/user-exec.c +++ b/accel/tcg/user-exec.c @@ -1121,18 +1121,45 @@ uint64_t cpu_ldq_le_mmu(CPUArchState *env, abi_ptr addr, return cpu_to_le64(ret); } -Int128 cpu_ld16_be_mmu(CPUArchState *env, abi_ptr addr, - MemOpIdx oi, uintptr_t ra) +static Int128 do_ld16_he_mmu(CPUArchState *env, abi_ptr addr, + MemOp mop, uintptr_t ra) { void *haddr; Int128 ret; - tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == (MO_128 | MO_BE)); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); - memcpy(&ret, haddr, 16); + tcg_debug_assert((mop & MO_SIZE) == MO_128); + haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_LOAD); + ret = load_atom_16(env, ra, haddr, mop); clear_helper_retaddr(); - qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); + return ret; +} +Int128 helper_ld16_mmu(CPUArchState *env, target_ulong addr, + MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + Int128 ret = do_ld16_he_mmu(env, addr, mop, ra); + + if (mop & MO_BSWAP) { + ret = bswap128(ret); + } + return ret; +} + +Int128 helper_ld_i128(CPUArchState *env, target_ulong addr, MemOpIdx oi) +{ + return helper_ld16_mmu(env, addr, oi, GETPC()); +} + +Int128 cpu_ld16_be_mmu(CPUArchState *env, abi_ptr addr, + MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + Int128 ret; + + tcg_debug_assert((mop & MO_BSWAP) == MO_BE); + ret = do_ld16_he_mmu(env, addr, mop, ra); + qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); if (!HOST_BIG_ENDIAN) { ret = bswap128(ret); } @@ -1142,15 +1169,12 @@ Int128 cpu_ld16_be_mmu(CPUArchState *env, abi_ptr addr, Int128 cpu_ld16_le_mmu(CPUArchState *env, abi_ptr addr, MemOpIdx oi, uintptr_t ra) { - void *haddr; + MemOp mop = get_memop(oi); Int128 ret; - tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == (MO_128 | MO_LE)); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD); - memcpy(&ret, haddr, 16); - clear_helper_retaddr(); + tcg_debug_assert((mop & MO_BSWAP) == MO_LE); + ret = do_ld16_he_mmu(env, addr, mop, ra); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R); - if (HOST_BIG_ENDIAN) { ret = bswap128(ret); } @@ -1307,33 +1331,57 @@ void cpu_stq_le_mmu(CPUArchState *env, abi_ptr addr, uint64_t val, qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); } -void cpu_st16_be_mmu(CPUArchState *env, abi_ptr addr, - Int128 val, MemOpIdx oi, uintptr_t ra) +static void do_st16_he_mmu(CPUArchState *env, abi_ptr addr, Int128 val, + MemOp mop, uintptr_t ra) { void *haddr; - tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == (MO_128 | MO_BE)); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE); + tcg_debug_assert((mop & MO_SIZE) == MO_128); + haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_STORE); + store_atom_16(env, ra, haddr, mop, val); + clear_helper_retaddr(); +} + +void helper_st16_mmu(CPUArchState *env, target_ulong addr, Int128 val, + MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + + if (mop & MO_BSWAP) { + val = bswap128(val); + } + do_st16_he_mmu(env, addr, val, mop, ra); +} + +void helper_st_i128(CPUArchState *env, target_ulong addr, + Int128 val, MemOpIdx oi) +{ + helper_st16_mmu(env, addr, val, oi, GETPC()); +} + +void cpu_st16_be_mmu(CPUArchState *env, abi_ptr addr, + Int128 val, MemOpIdx oi, uintptr_t ra) +{ + MemOp mop = get_memop(oi); + + tcg_debug_assert((mop & MO_BSWAP) == MO_BE); if (!HOST_BIG_ENDIAN) { val = bswap128(val); } - memcpy(haddr, &val, 16); - clear_helper_retaddr(); + do_st16_he_mmu(env, addr, val, mop, ra); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); } void cpu_st16_le_mmu(CPUArchState *env, abi_ptr addr, Int128 val, MemOpIdx oi, uintptr_t ra) { - void *haddr; + MemOp mop = get_memop(oi); - tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == (MO_128 | MO_LE)); - haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_STORE); + tcg_debug_assert((mop & MO_BSWAP) == MO_LE); if (HOST_BIG_ENDIAN) { val = bswap128(val); } - memcpy(haddr, &val, 16); - clear_helper_retaddr(); + do_st16_he_mmu(env, addr, val, mop, ra); qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W); } diff --git a/tcg/tcg-op.c b/tcg/tcg-op.c index 3136cef81a..22481a344c 100644 --- a/tcg/tcg-op.c +++ b/tcg/tcg-op.c @@ -3119,6 +3119,37 @@ void tcg_gen_qemu_st_i64(TCGv_i64 val, TCGv addr, TCGArg idx, MemOp memop) } } +/* + * Return true if @mop, without knowledge of the pointer alignment, + * does not require 16-byte atomicity, and it would be adventagous + * to avoid a call to a helper function. + */ +static bool use_two_i64_for_i128(MemOp mop) +{ +#ifdef CONFIG_SOFTMMU + /* Two softmmu tlb lookups is larger than one function call. */ + return false; +#else + /* + * For user-only, two 64-bit operations may well be smaller than a call. + * Determine if that would be legal for the requested atomicity. + */ + switch (mop & MO_ATOM_MASK) { + case MO_ATOM_NONE: + case MO_ATOM_IFALIGN_PAIR: + return true; + case MO_ATOM_IFALIGN: + case MO_ATOM_SUBALIGN: + case MO_ATOM_WITHIN16: + case MO_ATOM_WITHIN16_PAIR: + /* In a serialized context, no atomicity is required. */ + return !(tcg_ctx->gen_tb->cflags & CF_PARALLEL); + default: + g_assert_not_reached(); + } +#endif +} + static void canonicalize_memop_i128_as_i64(MemOp ret[2], MemOp orig) { MemOp mop_1 = orig, mop_2; @@ -3164,93 +3195,113 @@ static void canonicalize_memop_i128_as_i64(MemOp ret[2], MemOp orig) ret[1] = mop_2; } +#if TARGET_LONG_BITS == 64 +#define tcg_temp_ebb_new tcg_temp_ebb_new_i64 +#else +#define tcg_temp_ebb_new tcg_temp_ebb_new_i32 +#endif + void tcg_gen_qemu_ld_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop) { - MemOp mop[2]; - TCGv addr_p8; - TCGv_i64 x, y; + MemOpIdx oi = make_memop_idx(memop, idx); - canonicalize_memop_i128_as_i64(mop, memop); + tcg_debug_assert((memop & MO_SIZE) == MO_128); + tcg_debug_assert((memop & MO_SIGN) == 0); tcg_gen_req_mo(TCG_MO_LD_LD | TCG_MO_ST_LD); addr = plugin_prep_mem_callbacks(addr); - /* TODO: respect atomicity of the operation. */ /* TODO: allow the tcg backend to see the whole operation. */ - /* - * Since there are no global TCGv_i128, there is no visible state - * changed if the second load faults. Load directly into the two - * subwords. - */ - if ((memop & MO_BSWAP) == MO_LE) { - x = TCGV128_LOW(val); - y = TCGV128_HIGH(val); + if (use_two_i64_for_i128(memop)) { + MemOp mop[2]; + TCGv addr_p8; + TCGv_i64 x, y; + + canonicalize_memop_i128_as_i64(mop, memop); + + /* + * Since there are no global TCGv_i128, there is no visible state + * changed if the second load faults. Load directly into the two + * subwords. + */ + if ((memop & MO_BSWAP) == MO_LE) { + x = TCGV128_LOW(val); + y = TCGV128_HIGH(val); + } else { + x = TCGV128_HIGH(val); + y = TCGV128_LOW(val); + } + + gen_ldst_i64(INDEX_op_qemu_ld_i64, x, addr, mop[0], idx); + + if ((mop[0] ^ memop) & MO_BSWAP) { + tcg_gen_bswap64_i64(x, x); + } + + addr_p8 = tcg_temp_ebb_new(); + tcg_gen_addi_tl(addr_p8, addr, 8); + gen_ldst_i64(INDEX_op_qemu_ld_i64, y, addr_p8, mop[1], idx); + tcg_temp_free(addr_p8); + + if ((mop[0] ^ memop) & MO_BSWAP) { + tcg_gen_bswap64_i64(y, y); + } } else { - x = TCGV128_HIGH(val); - y = TCGV128_LOW(val); + gen_helper_ld_i128(val, cpu_env, addr, tcg_constant_i32(oi)); } - gen_ldst_i64(INDEX_op_qemu_ld_i64, x, addr, mop[0], idx); - - if ((mop[0] ^ memop) & MO_BSWAP) { - tcg_gen_bswap64_i64(x, x); - } - - addr_p8 = tcg_temp_new(); - tcg_gen_addi_tl(addr_p8, addr, 8); - gen_ldst_i64(INDEX_op_qemu_ld_i64, y, addr_p8, mop[1], idx); - tcg_temp_free(addr_p8); - - if ((mop[0] ^ memop) & MO_BSWAP) { - tcg_gen_bswap64_i64(y, y); - } - - plugin_gen_mem_callbacks(addr, make_memop_idx(memop, idx), - QEMU_PLUGIN_MEM_R); + plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_R); } void tcg_gen_qemu_st_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop) { - MemOp mop[2]; - TCGv addr_p8; - TCGv_i64 x, y; + MemOpIdx oi = make_memop_idx(memop, idx); - canonicalize_memop_i128_as_i64(mop, memop); + tcg_debug_assert((memop & MO_SIZE) == MO_128); + tcg_debug_assert((memop & MO_SIGN) == 0); tcg_gen_req_mo(TCG_MO_ST_LD | TCG_MO_ST_ST); addr = plugin_prep_mem_callbacks(addr); - /* TODO: respect atomicity of the operation. */ /* TODO: allow the tcg backend to see the whole operation. */ - if ((memop & MO_BSWAP) == MO_LE) { - x = TCGV128_LOW(val); - y = TCGV128_HIGH(val); + if (use_two_i64_for_i128(memop)) { + MemOp mop[2]; + TCGv addr_p8; + TCGv_i64 x, y; + + canonicalize_memop_i128_as_i64(mop, memop); + + if ((memop & MO_BSWAP) == MO_LE) { + x = TCGV128_LOW(val); + y = TCGV128_HIGH(val); + } else { + x = TCGV128_HIGH(val); + y = TCGV128_LOW(val); + } + + addr_p8 = tcg_temp_ebb_new(); + if ((mop[0] ^ memop) & MO_BSWAP) { + TCGv_i64 t = tcg_temp_ebb_new_i64(); + + tcg_gen_bswap64_i64(t, x); + gen_ldst_i64(INDEX_op_qemu_st_i64, t, addr, mop[0], idx); + tcg_gen_bswap64_i64(t, y); + tcg_gen_addi_tl(addr_p8, addr, 8); + gen_ldst_i64(INDEX_op_qemu_st_i64, t, addr_p8, mop[1], idx); + tcg_temp_free_i64(t); + } else { + gen_ldst_i64(INDEX_op_qemu_st_i64, x, addr, mop[0], idx); + tcg_gen_addi_tl(addr_p8, addr, 8); + gen_ldst_i64(INDEX_op_qemu_st_i64, y, addr_p8, mop[1], idx); + } + tcg_temp_free(addr_p8); } else { - x = TCGV128_HIGH(val); - y = TCGV128_LOW(val); + gen_helper_st_i128(cpu_env, addr, val, tcg_constant_i32(oi)); } - addr_p8 = tcg_temp_new(); - if ((mop[0] ^ memop) & MO_BSWAP) { - TCGv_i64 t = tcg_temp_ebb_new_i64(); - - tcg_gen_bswap64_i64(t, x); - gen_ldst_i64(INDEX_op_qemu_st_i64, t, addr, mop[0], idx); - tcg_gen_bswap64_i64(t, y); - tcg_gen_addi_tl(addr_p8, addr, 8); - gen_ldst_i64(INDEX_op_qemu_st_i64, t, addr_p8, mop[1], idx); - tcg_temp_free_i64(t); - } else { - gen_ldst_i64(INDEX_op_qemu_st_i64, x, addr, mop[0], idx); - tcg_gen_addi_tl(addr_p8, addr, 8); - gen_ldst_i64(INDEX_op_qemu_st_i64, y, addr_p8, mop[1], idx); - } - tcg_temp_free(addr_p8); - - plugin_gen_mem_callbacks(addr, make_memop_idx(memop, idx), - QEMU_PLUGIN_MEM_W); + plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_W); } static void tcg_gen_ext_i32(TCGv_i32 ret, TCGv_i32 val, MemOp opc) diff --git a/accel/tcg/ldst_atomicity.c.inc b/accel/tcg/ldst_atomicity.c.inc index 61567f5d0b..5cad4f4857 100644 --- a/accel/tcg/ldst_atomicity.c.inc +++ b/accel/tcg/ldst_atomicity.c.inc @@ -439,6 +439,21 @@ static inline uint64_t load_atom_8_by_4(void *pv) } } +/** + * load_atom_8_by_8_or_4: + * @pv: host address + * + * Load 8 bytes from aligned @pv, with at least 4-byte atomicity. + */ +static inline uint64_t load_atom_8_by_8_or_4(void *pv) +{ + if (HAVE_al8_fast) { + return load_atomic8(pv); + } else { + return load_atom_8_by_4(pv); + } +} + /** * load_atom_2: * @p: host address @@ -571,6 +586,64 @@ static uint64_t load_atom_8(CPUArchState *env, uintptr_t ra, } } +/** + * load_atom_16: + * @p: host address + * @memop: the full memory op + * + * Load 16 bytes from @p, honoring the atomicity of @memop. + */ +static Int128 load_atom_16(CPUArchState *env, uintptr_t ra, + void *pv, MemOp memop) +{ + uintptr_t pi = (uintptr_t)pv; + int atmax; + Int128 r; + uint64_t a, b; + + /* + * If the host does not support 16-byte atomics, wait until we have + * examined the atomicity parameters below. + */ + if (HAVE_al16_fast && likely((pi & 15) == 0)) { + return load_atomic16(pv); + } + + atmax = required_atomicity(env, pi, memop); + switch (atmax) { + case MO_8: + memcpy(&r, pv, 16); + return r; + case MO_16: + a = load_atom_8_by_2(pv); + b = load_atom_8_by_2(pv + 8); + break; + case MO_32: + a = load_atom_8_by_4(pv); + b = load_atom_8_by_4(pv + 8); + break; + case MO_64: + if (!HAVE_al8) { + cpu_loop_exit_atomic(env_cpu(env), ra); + } + a = load_atomic8(pv); + b = load_atomic8(pv + 8); + break; + case -MO_64: + if (!HAVE_al8) { + cpu_loop_exit_atomic(env_cpu(env), ra); + } + a = load_atom_extract_al8x2(pv); + b = load_atom_extract_al8x2(pv + 8); + break; + case MO_128: + return load_atomic16_or_exit(env, ra, pv); + default: + g_assert_not_reached(); + } + return int128_make128(HOST_BIG_ENDIAN ? b : a, HOST_BIG_ENDIAN ? a : b); +} + /** * store_atomic2: * @pv: host address @@ -612,6 +685,35 @@ static inline void store_atomic8(void *pv, uint64_t val) qatomic_set__nocheck(p, val); } +/** + * store_atomic16: + * @pv: host address + * @val: value to store + * + * Atomically store 16 aligned bytes to @pv. + */ +static inline void store_atomic16(void *pv, Int128Alias val) +{ +#if defined(CONFIG_ATOMIC128) + __uint128_t *pu = __builtin_assume_aligned(pv, 16); + qatomic_set__nocheck(pu, val.u); +#elif defined(CONFIG_CMPXCHG128) + __uint128_t *pu = __builtin_assume_aligned(pv, 16); + __uint128_t o; + + /* + * Without CONFIG_ATOMIC128, __atomic_compare_exchange_n will always + * defer to libatomic, so we must use __sync_*_compare_and_swap_16 + * and accept the sequential consistency that comes with it. + */ + do { + o = *pu; + } while (!__sync_bool_compare_and_swap_16(pu, o, val.u)); +#else + qemu_build_not_reached(); +#endif +} + /** * store_atom_4x2 */ @@ -1055,3 +1157,85 @@ static void store_atom_8(CPUArchState *env, uintptr_t ra, } cpu_loop_exit_atomic(env_cpu(env), ra); } + +/** + * store_atom_16: + * @p: host address + * @val: the value to store + * @memop: the full memory op + * + * Store 16 bytes to @p, honoring the atomicity of @memop. + */ +static void store_atom_16(CPUArchState *env, uintptr_t ra, + void *pv, MemOp memop, Int128 val) +{ + uintptr_t pi = (uintptr_t)pv; + uint64_t a, b; + int atmax; + + if (HAVE_al16_fast && likely((pi & 15) == 0)) { + store_atomic16(pv, val); + return; + } + + atmax = required_atomicity(env, pi, memop); + + a = HOST_BIG_ENDIAN ? int128_gethi(val) : int128_getlo(val); + b = HOST_BIG_ENDIAN ? int128_getlo(val) : int128_gethi(val); + switch (atmax) { + case MO_8: + memcpy(pv, &val, 16); + return; + case MO_16: + store_atom_8_by_2(pv, a); + store_atom_8_by_2(pv + 8, b); + return; + case MO_32: + store_atom_8_by_4(pv, a); + store_atom_8_by_4(pv + 8, b); + return; + case MO_64: + if (HAVE_al8) { + store_atomic8(pv, a); + store_atomic8(pv + 8, b); + return; + } + break; + case -MO_64: + if (HAVE_al16) { + uint64_t val_le; + int s2 = pi & 15; + int s1 = 16 - s2; + + if (HOST_BIG_ENDIAN) { + val = bswap128(val); + } + switch (s2) { + case 1 ... 7: + val_le = store_whole_le16(pv, s1, val); + store_bytes_leN(pv + s1, s2, val_le); + break; + case 9 ... 15: + store_bytes_leN(pv, s1, int128_getlo(val)); + val = int128_urshift(val, s1 * 8); + store_whole_le16(pv + s1, s2, val); + break; + case 0: /* aligned */ + case 8: /* atmax MO_64 */ + default: + g_assert_not_reached(); + } + return; + } + break; + case MO_128: + if (HAVE_al16) { + store_atomic16(pv, val); + return; + } + break; + default: + g_assert_not_reached(); + } + cpu_loop_exit_atomic(env_cpu(env), ra); +} From patchwork Mon May 15 14:32:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681965 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1152402wrr; Mon, 15 May 2023 07:43:49 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7iruZRFOfq2FAGv7YOV6xlcnNBXePIcNdF4y2TSCJ9O9EEii5/Kt92/hPI/717q3an2ZB+ X-Received: by 2002:a05:600c:21c8:b0:3f4:e9f6:1c3b with SMTP id x8-20020a05600c21c800b003f4e9f61c3bmr10620899wmj.12.1684161829849; Mon, 15 May 2023 07:43:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161829; cv=none; d=google.com; s=arc-20160816; b=aulYwDADSQ/3x3FgncymYUQfilkpf4MQ+JONJ0zSxaiGkeMQrOr+8QsWKvYJXkZiFw iwM+CrQH8DQXSodyQ0qyLNU1Bt8vMKYjh50N0MHPKLUD8Q8ghcXDOqmOARl7eZTBEovB 56SJcUcxoxX+mR3V056htZGuOsG22w1ms0JAc/uYaYbAivO5+2AtSZiqp24aZgvi7xAN LS6hgiQuSv2l8ABhDsUZM5klbYcCz7r3xB6rMotI+4Wm5s2ppEEesPnS0b6vT5bWEfOo +vNBE2F9HMlurLxNS3SYFlMiTvpjedPKJKgggnvJpizBXtdrrsoKeHzrs8mkwjCcZoAG YT3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=07Ill1IzvM80FPBTnlvOPncBZCfgn9gnqlrTz0L90+A=; b=BAaqpXnzkgjk2WfAtjpXfQvIhmOjIi0oTZS0UHbOnlE3JuFC07DUhLBwK03G9gX9eQ ZLfJpBmY4Qph8FiCVGec/waK4wb4ZPGR94hnhFzdwXZ90Zj1EXwVqfJcln044V7lsT1K OF5HctBk8+GZxynnWZl2iJj9UFh8UOsmqgBtDqN1jfvGtBGh/bE2BoCAuKdABBhVSyR3 P9KnmVixQJEq87iVTREQM90ad9hEGwgdnrm9/7NVGhqbXTze8a4M7nxZP0NH6PPwlnyY /0BMqZgqqs4gw/oU5QeePd/lOH/GWIBQ31Uhe5zRTCUldPHo9pGmpwdtEXv9I1tpLom0 kE8g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Qq8090uf; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id u7-20020adfed47000000b003063ae38fc2si14951894wro.1051.2023.05.15.07.43.49 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:43:49 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Qq8090uf; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZI4-0007V6-4D; Mon, 15 May 2023 10:35:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZGZ-0006n3-59 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:10 -0400 Received: from mail-pl1-x62d.google.com ([2607:f8b0:4864:20::62d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGN-0004sR-T9 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:34 -0400 Received: by mail-pl1-x62d.google.com with SMTP id d9443c01a7336-1aaea3909d1so120067835ad.2 for ; Mon, 15 May 2023 07:33:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161202; x=1686753202; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=07Ill1IzvM80FPBTnlvOPncBZCfgn9gnqlrTz0L90+A=; b=Qq8090ufVGkup5LwZzppBbGEJsaZeB66Lw1NE7FvhLMF7qlIVuIp7fbLoYardC24cL DYc86EhaL55AO+tGAxUnYU0+KoXQlo8BBupufiJcfDCxabtvlT7Kwzcvz2C5LdM7UVDf 5RKqGb0CMrN9EgKKgJFTGxXSxOGLWDzj7K31mqxv3V2mEhKhvZq7f9/cEoYt5vYuH3NI h7uw5B0vZSVv4E4PCP/gFwO5MrWAOOmJcE7kedOoaFV1NkCJ9L25GcOXgs6F/b1ELQeJ tz35NHU1etXUZtc8jxvOvb061Y7tfY4KsNT9GNJ5qUpodPbObOFoMydXQOppmJAZunr3 KVJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161202; x=1686753202; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=07Ill1IzvM80FPBTnlvOPncBZCfgn9gnqlrTz0L90+A=; b=iD9E4BivMuIkFY9GPqu60zZoki+18P+PO3lB4LuUCCeFPXaWkT7OQpO66uXvBVoD2G t+1qucBssXKuhJKDWjVIIBhvkpYRYtJy7y196pwh9CVAAGmpV1xjvUE0pxEJudzSsnN5 GKmoxhD6M2CfeeXJXgpOxOkcN9Bhm2GgqaSzKmVsIl88fSY0QfM7imWamx4rGRfZipNR 1w3vDOIYYWDTuGjH2N9Q8Yci0p+pLgiZfO17X/IUUgIJ54nkOBuIrpNO6f3GSMdyox7n L8o254uLKTFgN1l3WDVT0neGato3hfubdq/8GACGIO6TQE00XM256nsiVlIkIlDtg44p RhDg== X-Gm-Message-State: AC+VfDwaME1sL52fLFdiHQHLmGIEDJ9sYqtXl/iT5E4NhXz0Bt7RpWyI QpRVIUzAiJw9opahPzcek5ePPr0H6QbiJgMp9Ns= X-Received: by 2002:a17:902:eccc:b0:1ac:b449:3528 with SMTP id a12-20020a170902eccc00b001acb4493528mr23307832plh.46.1684161201916; Mon, 15 May 2023 07:33:21 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:21 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 08/54] meson: Detect atomic128 support with optimization Date: Mon, 15 May 2023 07:32:27 -0700 Message-Id: <20230515143313.734053-9-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62d; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org There is an edge condition prior to gcc13 for which optimization is required to generate 16-byte atomic sequences. Detect this. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- meson.build | 52 ++++++++++++++++++++++------------ accel/tcg/ldst_atomicity.c.inc | 29 ++++++++++++++++--- 2 files changed, 59 insertions(+), 22 deletions(-) diff --git a/meson.build b/meson.build index d3cf48960b..61de8199cb 100644 --- a/meson.build +++ b/meson.build @@ -2249,23 +2249,21 @@ config_host_data.set('HAVE_BROKEN_SIZE_MAX', not cc.compiles(''' return printf("%zu", SIZE_MAX); }''', args: ['-Werror'])) -atomic_test = ''' +# See if 64-bit atomic operations are supported. +# Note that without __atomic builtins, we can only +# assume atomic loads/stores max at pointer size. +config_host_data.set('CONFIG_ATOMIC64', cc.links(''' #include int main(void) { - @0@ x = 0, y = 0; + uint64_t x = 0, y = 0; y = __atomic_load_n(&x, __ATOMIC_RELAXED); __atomic_store_n(&x, y, __ATOMIC_RELAXED); __atomic_compare_exchange_n(&x, &y, x, 0, __ATOMIC_RELAXED, __ATOMIC_RELAXED); __atomic_exchange_n(&x, y, __ATOMIC_RELAXED); __atomic_fetch_add(&x, y, __ATOMIC_RELAXED); return 0; - }''' - -# See if 64-bit atomic operations are supported. -# Note that without __atomic builtins, we can only -# assume atomic loads/stores max at pointer size. -config_host_data.set('CONFIG_ATOMIC64', cc.links(atomic_test.format('uint64_t'))) + }''')) has_int128 = cc.links(''' __int128_t a; @@ -2283,21 +2281,39 @@ if has_int128 # "do we have 128-bit atomics which are handled inline and specifically not # via libatomic". The reason we can't use libatomic is documented in the # comment starting "GCC is a house divided" in include/qemu/atomic128.h. - has_atomic128 = cc.links(atomic_test.format('unsigned __int128')) + # We only care about these operations on 16-byte aligned pointers, so + # force 16-byte alignment of the pointer, which may be greater than + # __alignof(unsigned __int128) for the host. + atomic_test_128 = ''' + int main(int ac, char **av) { + unsigned __int128 *p = __builtin_assume_aligned(av[ac - 1], sizeof(16)); + p[1] = __atomic_load_n(&p[0], __ATOMIC_RELAXED); + __atomic_store_n(&p[2], p[3], __ATOMIC_RELAXED); + __atomic_compare_exchange_n(&p[4], &p[5], p[6], 0, __ATOMIC_RELAXED, __ATOMIC_RELAXED); + return 0; + }''' + has_atomic128 = cc.links(atomic_test_128) config_host_data.set('CONFIG_ATOMIC128', has_atomic128) if not has_atomic128 - has_cmpxchg128 = cc.links(''' - int main(void) - { - unsigned __int128 x = 0, y = 0; - __sync_val_compare_and_swap_16(&x, y, x); - return 0; - } - ''') + # Even with __builtin_assume_aligned, the above test may have failed + # without optimization enabled. Try again with optimizations locally + # enabled for the function. See + # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107389 + has_atomic128_opt = cc.links('__attribute__((optimize("O1")))' + atomic_test_128) + config_host_data.set('CONFIG_ATOMIC128_OPT', has_atomic128_opt) - config_host_data.set('CONFIG_CMPXCHG128', has_cmpxchg128) + if not has_atomic128_opt + config_host_data.set('CONFIG_CMPXCHG128', cc.links(''' + int main(void) + { + unsigned __int128 x = 0, y = 0; + __sync_val_compare_and_swap_16(&x, y, x); + return 0; + } + ''')) + endif endif endif diff --git a/accel/tcg/ldst_atomicity.c.inc b/accel/tcg/ldst_atomicity.c.inc index 5cad4f4857..dd387c9bdd 100644 --- a/accel/tcg/ldst_atomicity.c.inc +++ b/accel/tcg/ldst_atomicity.c.inc @@ -16,6 +16,23 @@ #endif #define HAVE_al8_fast (ATOMIC_REG_SIZE >= 8) +/* + * If __alignof(unsigned __int128) < 16, GCC may refuse to inline atomics + * that are supported by the host, e.g. s390x. We can force the pointer to + * have our known alignment with __builtin_assume_aligned, however prior to + * GCC 13 that was only reliable with optimization enabled. See + * https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107389 + */ +#if defined(CONFIG_ATOMIC128_OPT) +# if !defined(__OPTIMIZE__) +# define ATTRIBUTE_ATOMIC128_OPT __attribute__((optimize("O1"))) +# endif +# define CONFIG_ATOMIC128 +#endif +#ifndef ATTRIBUTE_ATOMIC128_OPT +# define ATTRIBUTE_ATOMIC128_OPT +#endif + #if defined(CONFIG_ATOMIC128) # define HAVE_al16_fast true #else @@ -152,7 +169,8 @@ static inline uint64_t load_atomic8(void *pv) * * Atomically load 16 aligned bytes from @pv. */ -static inline Int128 load_atomic16(void *pv) +static inline Int128 ATTRIBUTE_ATOMIC128_OPT +load_atomic16(void *pv) { #ifdef CONFIG_ATOMIC128 __uint128_t *p = __builtin_assume_aligned(pv, 16); @@ -356,7 +374,8 @@ static uint64_t load_atom_extract_al16_or_exit(CPUArchState *env, uintptr_t ra, * cross an 16-byte boundary then the access must be 16-byte atomic, * otherwise the access must be 8-byte atomic. */ -static inline uint64_t load_atom_extract_al16_or_al8(void *pv, int s) +static inline uint64_t ATTRIBUTE_ATOMIC128_OPT +load_atom_extract_al16_or_al8(void *pv, int s) { #if defined(CONFIG_ATOMIC128) uintptr_t pi = (uintptr_t)pv; @@ -692,7 +711,8 @@ static inline void store_atomic8(void *pv, uint64_t val) * * Atomically store 16 aligned bytes to @pv. */ -static inline void store_atomic16(void *pv, Int128Alias val) +static inline void ATTRIBUTE_ATOMIC128_OPT +store_atomic16(void *pv, Int128Alias val) { #if defined(CONFIG_ATOMIC128) __uint128_t *pu = __builtin_assume_aligned(pv, 16); @@ -790,7 +810,8 @@ static void store_atom_insert_al8(uint64_t *p, uint64_t val, uint64_t msk) * * Atomically store @val to @p masked by @msk. */ -static void store_atom_insert_al16(Int128 *ps, Int128Alias val, Int128Alias msk) +static void ATTRIBUTE_ATOMIC128_OPT +store_atom_insert_al16(Int128 *ps, Int128Alias val, Int128Alias msk) { #if defined(CONFIG_ATOMIC128) __uint128_t *pu, old, new; From patchwork Mon May 15 14:32:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681937 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1148774wrr; Mon, 15 May 2023 07:36:03 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6xa+gYF4780QpDXENFwgD/hyOPKlM0JpmGW01V44VanoscImGB73GEViOA3ZmF4YU87DRP X-Received: by 2002:a05:622a:11ca:b0:3f4:f522:dd8c with SMTP id n10-20020a05622a11ca00b003f4f522dd8cmr20547314qtk.18.1684161362959; Mon, 15 May 2023 07:36:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161362; cv=none; d=google.com; s=arc-20160816; b=L31Qc0kDzXho//piMaLBg/r1qVLhau1cL/C2EsA/cPX8Gw45QzI+y0/ArFgM/QXTki edI2bK69vSEJr0ww2cKu1tb7nW/vsHHgxszs77lsmz6hScmKadEQ426NnTn7or7eO0Um E/CW867D84ZQe4zekT3HjQFKzVR8Eg/UQZ6/ia+Rp7O/rlc+/BLTMMNmEFFJkZHsL2rz rEzvMQyOfspObPfCmo9qK1dOS94z5g2p2B2vzHAkLfzpdVNqez/pNN1jqL4pmMuvPhUi 1AJnBTF/zVoNFBYbnVWj43Tw/rf+Kz71tDRYVRJaDWsQqn/OV4X7nE81CXFmjq1AwO3u kk1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=433Q0BDA2EvHaZur3cVyCHd6hqrDDnFwM+Ls6LBVsKw=; b=Uy+GLANUO4ZMZXZ6mwvInbJ1B4yvvdWiuzliAOE+WwWnHUdfWMKt2FijBqdAZQQ0wJ AHpetlMrW2rFemQydPJU/+mr8O6y7bmQoblMe1ZIwPcZNKKWZQq1hkOv1+MKMPKeMhoV Il7jltGmrjYdrUEB2P0xm1ivORODDPKDInn/GgtFbv0MZ4Pgsj3gVeRf503FisYl0d+H ML457NMfAitapf5g6cjXjgr/hLMSCkVgGLXxGAdbojGgcRnxF4hY1L/9frfQ7HcI9d2x jOB4X9zx2MCn7vEyVv2TvZ1NlHVNs23lMpe0pM6c9YQa90MjVP5uqh268Bbq6Pg63yL1 MYOg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=LVnjoRBd; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id p17-20020ae9f311000000b0075784a8f12asi10579185qkg.277.2023.05.15.07.36.02 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:36:02 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=LVnjoRBd; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZI8-0007Wv-Rd; Mon, 15 May 2023 10:35:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZGZ-0006n5-5q for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:10 -0400 Received: from mail-pf1-x42b.google.com ([2607:f8b0:4864:20::42b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGO-0004ss-BR for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:33 -0400 Received: by mail-pf1-x42b.google.com with SMTP id d2e1a72fcca58-64a9335a8e7so21247276b3a.0 for ; Mon, 15 May 2023 07:33:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161202; x=1686753202; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=433Q0BDA2EvHaZur3cVyCHd6hqrDDnFwM+Ls6LBVsKw=; b=LVnjoRBdBbFvh1iQjJDeNvT4j+ThvKpwxT85Kr5VQGH+uwQdjwmo7C9q/dsTyQTBlv EcHwRenQCgeNMxW3NfvdjHm10PRvhRfunreQX+i6D98F2R3yHITdppnHaIkO2tXf5qcI RF3RbNT0fGy2I2u7gXzUcmLEfIjjUL2nYByPsl8HNV629qLOmk/V1V9ac+KaN04ZOjw8 dqEXa4o+4kgdgn0Qdd4wXnDcCOt6Yx0RdmuoihmraKJ3UgmXZ/CWqz9LsUTOc78MHGcO MFdWYrT3SXkLYi3uw7AIB7jNNtOUpk/M16+eyj339OQHVQzaJ2UXuTsmgXB5IZorHb2i vmfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161202; x=1686753202; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=433Q0BDA2EvHaZur3cVyCHd6hqrDDnFwM+Ls6LBVsKw=; b=b3UJ0DrHgpq83BPm+UozRnxmt0ZqGpqae4JF1F9SDn8g4M56cedECrHDHwl2CvtFGX WNYtH1TnRrl785KhVQTJW4lJOj1h0cvSVBz1W3PGvMJgBe9Sp3lpswQexQfYH/UJebKz ROVtzg3ebJwAn+Q3QTwdS33FK5HFYz9/0Y8N7L5+cGJPb8sgp4TfVieOdIr80rob2JEx XuAiwoe0KhidaPBlX+OrZcM0FR5lqti7+9QYwSHLtK/sZTs3AskmzODWujrzo0vV25Uf 3XNBA/M6t7JBpswIsYsWhl2DRtINizuWA+UWdWiU6urQbVKZkjBtVcLQn8iyTZka5qMf TxzQ== X-Gm-Message-State: AC+VfDxnmeW9r9c0I6CUutWRnJszO9VXHfMi6R9cq+Z4FOn7t0DSwLd7 2KEgBoCwotaoRqFl3Y6HRLYV0W53OQkDvTAS+MI= X-Received: by 2002:a17:903:1252:b0:1ae:10a5:8349 with SMTP id u18-20020a170903125200b001ae10a58349mr5144291plh.23.1684161202697; Mon, 15 May 2023 07:33:22 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:22 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 09/54] tcg/i386: Add have_atomic16 Date: Mon, 15 May 2023 07:32:28 -0700 Message-Id: <20230515143313.734053-10-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::42b; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x42b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Notice when Intel or AMD have guaranteed that vmovdqa is atomic. The new variable will also be used in generated code. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- include/qemu/cpuid.h | 18 ++++++++++++++++++ tcg/i386/tcg-target.h | 1 + tcg/i386/tcg-target.c.inc | 27 +++++++++++++++++++++++++++ 3 files changed, 46 insertions(+) diff --git a/include/qemu/cpuid.h b/include/qemu/cpuid.h index 1451e8ef2f..35325f1995 100644 --- a/include/qemu/cpuid.h +++ b/include/qemu/cpuid.h @@ -71,6 +71,24 @@ #define bit_LZCNT (1 << 5) #endif +/* + * Signatures for different CPU implementations as returned from Leaf 0. + */ + +#ifndef signature_INTEL_ecx +/* "Genu" "ineI" "ntel" */ +#define signature_INTEL_ebx 0x756e6547 +#define signature_INTEL_edx 0x49656e69 +#define signature_INTEL_ecx 0x6c65746e +#endif + +#ifndef signature_AMD_ecx +/* "Auth" "enti" "cAMD" */ +#define signature_AMD_ebx 0x68747541 +#define signature_AMD_edx 0x69746e65 +#define signature_AMD_ecx 0x444d4163 +#endif + static inline unsigned xgetbv_low(unsigned c) { unsigned a, d; diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h index d4f2a6f8c2..0421776cb8 100644 --- a/tcg/i386/tcg-target.h +++ b/tcg/i386/tcg-target.h @@ -120,6 +120,7 @@ extern bool have_avx512dq; extern bool have_avx512vbmi2; extern bool have_avx512vl; extern bool have_movbe; +extern bool have_atomic16; /* optional instructions */ #define TCG_TARGET_HAS_div2_i32 1 diff --git a/tcg/i386/tcg-target.c.inc b/tcg/i386/tcg-target.c.inc index 826f7764c9..911123cfa8 100644 --- a/tcg/i386/tcg-target.c.inc +++ b/tcg/i386/tcg-target.c.inc @@ -185,6 +185,7 @@ bool have_avx512dq; bool have_avx512vbmi2; bool have_avx512vl; bool have_movbe; +bool have_atomic16; #ifdef CONFIG_CPUID_H static bool have_bmi2; @@ -4026,6 +4027,32 @@ static void tcg_target_init(TCGContext *s) have_avx512dq = (b7 & bit_AVX512DQ) != 0; have_avx512vbmi2 = (c7 & bit_AVX512VBMI2) != 0; } + + /* + * The Intel SDM has added: + * Processors that enumerate support for Intel® AVX + * (by setting the feature flag CPUID.01H:ECX.AVX[bit 28]) + * guarantee that the 16-byte memory operations performed + * by the following instructions will always be carried + * out atomically: + * - MOVAPD, MOVAPS, and MOVDQA. + * - VMOVAPD, VMOVAPS, and VMOVDQA when encoded with VEX.128. + * - VMOVAPD, VMOVAPS, VMOVDQA32, and VMOVDQA64 when encoded + * with EVEX.128 and k0 (masking disabled). + * Note that these instructions require the linear addresses + * of their memory operands to be 16-byte aligned. + * + * AMD has provided an even stronger guarantee that processors + * with AVX provide 16-byte atomicity for all cachable, + * naturally aligned single loads and stores, e.g. MOVDQU. + * + * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104688 + */ + if (have_avx1) { + __cpuid(0, a, b, c, d); + have_atomic16 = (c == signature_INTEL_ecx || + c == signature_AMD_ecx); + } } } } From patchwork Mon May 15 14:32:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681942 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1149560wrr; Mon, 15 May 2023 07:37:35 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4c1fLCXAuIQxhmfAwkUVeb08VdpSVcooNzJGtgy6J4XlMR7Z2eiaLoCDSGHnFof95He00j X-Received: by 2002:a05:600c:c5:b0:3f4:2d85:53f6 with SMTP id u5-20020a05600c00c500b003f42d8553f6mr14748584wmm.34.1684161454949; Mon, 15 May 2023 07:37:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161454; cv=none; d=google.com; s=arc-20160816; b=b1gVz0gxYTx7HfKWJ8Xtt4HNx7R1XXQQjuT2itf1OiH50kCfNMhCIWY8Z57GgTQECD /bKsQFw2eeztiG0ShujuNenpoP53zjQxriucDYLTJ2TEK3UP1X262gCN/DsBy4Ly3qnr QWNrNvPguOmVfJTZHDHCfkSzChfJtkZ42CJptFpoKPtfh/9i4C/tsMevG/mOfHJlwHXl jYchvAjbI0DyeSrbiQb5kcpx+FZeOAlRq9hFx55gLa0AlrzM9+3x5hpPHtvGVJsjmv/T MZrbwWJOHHxOOtq+3UyQru6aBbGPtwwYzHazH34daiNCMgP64tYe6CKerrKQ/EjgjsOc 9DvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=qq/bfx/cnVfgddia2Z24C2Fe0/LSiH7rE9uM4DdN6u4=; b=q+jX3f6RD1j6PNMgRnJK6O9qyAJ2r370QUb1gaYvvSJ+BbzHdRKliHJ1V3FmnylCf0 6qIzgTLI9bmGSq6x33cH3Go/oQ1e1CaUsr3VFJ+okNVXTtS/NxJ7QVOzThyZifpCkwZt 9BNTAX6dd40qX3cEImm52dbg9tCHQRVNfw4WCDypnGxfgIwV9ZDWqhgi0OOXr7NOVu70 A+hD79XsGDLnzXbBvtHOUh1aa3IDfSpIyMiGPdOau+rcxuCR3r+z+Br65WlSNEyDiBYj 4d7fu3LVliU2k8dyAIUhFfVcRn2U8Y5PkDGT7N8mv9LTgoNdvsoH646tDiKk0yeCE4BT weEg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=DePA+PM+; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id x9-20020a05600c21c900b003f426c38dd4si11159898wmj.123.2023.05.15.07.37.34 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:37:34 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=DePA+PM+; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZI4-0007VP-SC; Mon, 15 May 2023 10:35:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZH5-0006qp-1w for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:15 -0400 Received: from mail-pl1-x632.google.com ([2607:f8b0:4864:20::632]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGP-0004td-4p for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:35 -0400 Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-1adc913094aso45799925ad.0 for ; Mon, 15 May 2023 07:33:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161203; x=1686753203; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qq/bfx/cnVfgddia2Z24C2Fe0/LSiH7rE9uM4DdN6u4=; b=DePA+PM+BOtrodLW+PFFvwWQ3nNBMcmf5p9EyBz8Zq+xK+VlTZ3UK1mYbU8XW6+D5L a4ncrrydjYt+iYtVOTRXAsn4jTIvx0kU52BAyZasInblt11bg5zjyniNSdJQbESxS89l B7ZI8Coro054XOok1JWAomt/OL/y8Z6FGPtoIdJFzawBRmoGrqNW3yCU9wijOEcfUpO1 Ctp56jdc5xi6Trvne3PxLi22YbQmG4GhqNpqV3v+hwXGfAMbosiS6JeY73GxYgYOOE85 o29VMq8s48zsJBjTcfFwRj0xvR0dwbDt9hdRuTl3SowZxblpEOUdop/L4kN07YzaYBPF m4ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161203; x=1686753203; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qq/bfx/cnVfgddia2Z24C2Fe0/LSiH7rE9uM4DdN6u4=; b=CeoJQgdQ8Ia0ZHOMLGd3e8R5n7kQJ3dJsZNuYq/M6z1oBuqZLVIEe4Iyj9XHoRLqH6 /abkTfJFCGh+qOukljc5TfpqwxUj/Wyu96o3h2xHdUUYivufPc5z81xdnPcPFODITInD YlGAwzc4087xYng4InmHBJTlNiATx9z0EG8qdeBukFgsd7hKlWxN49XOVoaR8AAujrd3 FOpTNPy/ZROrJwSGjuZSMKkYZGjeCxtSiVbESwGMQdjX+xjgtoHEntBaCPJfxxbu0b2W YNKvOtSu3iyTPDMD1EnaY8YWqkJjCpISovfP1NvJEE23w7Al7TzJRvnhHEKpmdlIxYUl t10w== X-Gm-Message-State: AC+VfDxyonFPNR3sLRIlZYksN0qsgglr2xZj+IJsnEVAbPJMt2T4RQiJ v8ZAu6x//KltJPne1FurEJV4Ye8Av77uJwJmy1E= X-Received: by 2002:a17:902:b707:b0:1ac:aaf6:ee3a with SMTP id d7-20020a170902b70700b001acaaf6ee3amr19980240pls.32.1684161203683; Mon, 15 May 2023 07:33:23 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:23 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 10/54] accel/tcg: Use have_atomic16 in ldst_atomicity.c.inc Date: Mon, 15 May 2023 07:32:29 -0700 Message-Id: <20230515143313.734053-11-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::632; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x632.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Hosts using Intel and AMD AVX cpus are quite common. Add fast paths through ldst_atomicity using this. Only enable with CONFIG_INT128; some older clang versions do not support __int128_t, and the inline assembly won't work on structures. Signed-off-by: Richard Henderson --- accel/tcg/ldst_atomicity.c.inc | 76 +++++++++++++++++++++++++++------- 1 file changed, 60 insertions(+), 16 deletions(-) diff --git a/accel/tcg/ldst_atomicity.c.inc b/accel/tcg/ldst_atomicity.c.inc index dd387c9bdd..69c1c61997 100644 --- a/accel/tcg/ldst_atomicity.c.inc +++ b/accel/tcg/ldst_atomicity.c.inc @@ -35,6 +35,14 @@ #if defined(CONFIG_ATOMIC128) # define HAVE_al16_fast true +#elif defined(CONFIG_TCG_INTERPRETER) +/* + * FIXME: host specific detection for this is in tcg/$host/, + * but we're using tcg/tci/ instead. + */ +# define HAVE_al16_fast false +#elif defined(__x86_64__) && defined(CONFIG_INT128) +# define HAVE_al16_fast likely(have_atomic16) #else # define HAVE_al16_fast false #endif @@ -178,6 +186,12 @@ load_atomic16(void *pv) r.u = qatomic_read__nocheck(p); return r.s; +#elif defined(__x86_64__) && defined(CONFIG_INT128) + Int128Alias r; + + /* Via HAVE_al16_fast, have_atomic16 is true. */ + asm("vmovdqa %1, %0" : "=x" (r.u) : "m" (*(Int128 *)pv)); + return r.s; #else qemu_build_not_reached(); #endif @@ -399,6 +413,24 @@ load_atom_extract_al16_or_al8(void *pv, int s) r = qatomic_read__nocheck(p16); } return r >> shr; +#elif defined(__x86_64__) && defined(CONFIG_INT128) + uintptr_t pi = (uintptr_t)pv; + int shr = (pi & 7) * 8; + uint64_t a, b; + + /* Via HAVE_al16_fast, have_atomic16 is true. */ + pv = (void *)(pi & ~7); + if (pi & 8) { + uint64_t *p8 = __builtin_assume_aligned(pv, 16, 8); + a = qatomic_read__nocheck(p8); + b = qatomic_read__nocheck(p8 + 1); + } else { + asm("vmovdqa %2, %0\n\tvpextrq $1, %0, %1" + : "=x"(a), "=r"(b) : "m" (*(__uint128_t *)pv)); + } + asm("shrd %b2, %1, %0" : "+r"(a) : "r"(b), "c"(shr)); + + return a; #else qemu_build_not_reached(); #endif @@ -715,23 +747,35 @@ static inline void ATTRIBUTE_ATOMIC128_OPT store_atomic16(void *pv, Int128Alias val) { #if defined(CONFIG_ATOMIC128) - __uint128_t *pu = __builtin_assume_aligned(pv, 16); - qatomic_set__nocheck(pu, val.u); -#elif defined(CONFIG_CMPXCHG128) - __uint128_t *pu = __builtin_assume_aligned(pv, 16); - __uint128_t o; - - /* - * Without CONFIG_ATOMIC128, __atomic_compare_exchange_n will always - * defer to libatomic, so we must use __sync_*_compare_and_swap_16 - * and accept the sequential consistency that comes with it. - */ - do { - o = *pu; - } while (!__sync_bool_compare_and_swap_16(pu, o, val.u)); -#else - qemu_build_not_reached(); + { + __uint128_t *pu = __builtin_assume_aligned(pv, 16); + qatomic_set__nocheck(pu, val.u); + return; + } #endif +#if defined(__x86_64__) && defined(CONFIG_INT128) + if (HAVE_al16_fast) { + asm("vmovdqa %1, %0" : "=m"(*(__uint128_t *)pv) : "x" (val.u)); + return; + } +#endif +#if defined(CONFIG_CMPXCHG128) + { + __uint128_t *pu = __builtin_assume_aligned(pv, 16); + __uint128_t o; + + /* + * Without CONFIG_ATOMIC128, __atomic_compare_exchange_n will always + * defer to libatomic, so we must use __sync_*_compare_and_swap_16 + * and accept the sequential consistency that comes with it. + */ + do { + o = *pu; + } while (!__sync_bool_compare_and_swap_16(pu, o, val.u)); + return; + } +#endif + qemu_build_not_reached(); } /** From patchwork Mon May 15 14:32:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681947 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1150003wrr; Mon, 15 May 2023 07:38:31 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5q0VBothsFiN5C9JuL79G3P9fWwwMDvFJqO6MkQRzgFFjAVqdREJiHdkH68ad97FAbNX/i X-Received: by 2002:a5d:4052:0:b0:2f7:cfe2:6507 with SMTP id w18-20020a5d4052000000b002f7cfe26507mr25292209wrp.20.1684161510928; Mon, 15 May 2023 07:38:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161510; cv=none; d=google.com; s=arc-20160816; b=Vou8vOnKdQmvHcjkFz0RWyhqHsIOjJ3XCGqy3IO+ukOXgg80cikjg44ojNT5lN1SW/ l2r4Q5fLVntN9kYwXlCiSXjxtKTb2yB+WVlQvuO5asHZhFO9ISeOzqV/c+nhdJ2M+JsB BI/C5/aJTfO2Lso0K3OPDfYoduuRTIonktZj7fiJFqWhksyFc6O2IQUNgaBgDfBzRH0I C0/k0s8fR5JidccMCKVl5RcrC0VCIDilRawBdeaTiOO3fhnr2FAA/zAOkzg2pQ4dkFXC 8EvQ9BGBussXHUy7LKZMauQzsouJzLSDxMU0xnVtmZo8Q/x2/8Qg9IgwTkNH7lFBCOIf 89+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=vwB32Lu8jReXcA/wvC/qx9nAhkYXsChfxxD2grqRfag=; b=hdvDRcB8PcxTPEYZzPtKT7cjJV9wLWT/nmBUrk+xUIySaZgliRHmcx1kvpZmrZqkTE oj86IKjnQ6Q9TaWtkOZz4/+GMBiaHso+88NuuYGtDxi2VcBR9Raersx18RYU+vLcieXA DZ/MpB/GU+JO2QHoBiDm2Iqbu2PXe8mDU9W547quJMxtsp6pJ6+v7pjG0MIVY8eC5eKM 3FMLNLouGfIoUAge2PjTDGeuwK63IqQejDUaU64iwyTuP1SU8Vik4yX34XPCYB0LsLNi z1bu3ypzZp2gds+YryvSPBlmQUaH3ws4IR/Y2P90vuFsdK5P5iohT+ZhesKGz5RM5TTl 0zrA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=BvisT1N3; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id v27-20020a5d591b000000b003063b7224bdsi15998968wrd.502.2023.05.15.07.38.30 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:38:30 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=BvisT1N3; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZI6-0007Vl-D8; Mon, 15 May 2023 10:35:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHQ-0006wr-Oo for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:33 -0400 Received: from mail-pl1-x633.google.com ([2607:f8b0:4864:20::633]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGP-0004q9-SC for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:36 -0400 Received: by mail-pl1-x633.google.com with SMTP id d9443c01a7336-1ab13da70a3so124359735ad.1 for ; Mon, 15 May 2023 07:33:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161205; x=1686753205; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vwB32Lu8jReXcA/wvC/qx9nAhkYXsChfxxD2grqRfag=; b=BvisT1N3QcyJn+HIcURGy7vgTVeVGmWfGpUbBahF7IYPxhfkg2FVuq5+hi3EsMLJB2 Lswr1bhQT90XOdsuO9YtoEwXnES88HJW547919/UvgVKKrGn7MK8uzsgDz/wH7o6EQo9 k1HuIS0v9dHuktoHpkVC9Ba+mXlO/2sX7aYzuC3aGjYLmMg2pMf/KNj1cdQhZrwoMu6Y w2Ft0Dxf6WZ57SxpjAA3snn+YD8UBU3gOYxsBRyMt8gdM2GpxyCTgIE2Xo8tlFfoPsBm J1IPEYypC+Gdi72T+fBqBB4IHdb2ISLA1FrbxH55fOLdY7wldvFy9c2D9WnYoGL5yWMD SuAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161205; x=1686753205; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vwB32Lu8jReXcA/wvC/qx9nAhkYXsChfxxD2grqRfag=; b=f2efdyJoVEVyeSat1QeOOvegMAqBHd0EESKqweL3bxL5cgkih0Gs1WQdPUP86BhaEy 9MIKPzXnCzRhyRsnZFt85Zx4ruaKgvwBwNjSzMWVApZQVI75+CcFc+luKAICkMfA+H9b ntmWYCYHP/6OxQx3/9qmHXRSGowUVhMfDEhOv9h81hxzGO87LsFpNR+XOkILz4uW1S5q HDFDzrQFm1oTqh1ixAWw0yak/7v8EeRA3CPvWuROjJsFNJRwCdhullxHJvtk6k1WZo3A FP2+8c4mS8deDQBlD+c1CpK5H4zpr3tix10m+96FwAYCipc5DgwNhQdy+uwN7TouMiSy 4D/A== X-Gm-Message-State: AC+VfDz0VHVa6jbD9Vi/hKLKh8ZK+uQ66WPo4+JnTLwN3I55iO7GOQNT 61PW14tk/lB748LLYzQjLEJXeZuvtlJMOQzwvEI= X-Received: by 2002:a17:902:ee8a:b0:1ac:6c26:c32f with SMTP id a10-20020a170902ee8a00b001ac6c26c32fmr29497439pld.46.1684161204522; Mon, 15 May 2023 07:33:24 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:24 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 11/54] accel/tcg: Add aarch64 specific support in ldst_atomicity Date: Mon, 15 May 2023 07:32:30 -0700 Message-Id: <20230515143313.734053-12-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::633; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x633.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org We have code in atomic128.h noting that through GCC 8, there was no support for atomic operations on __uint128. This has been fixed in GCC 10. But we can still improve over any basic compare-and-swap loop using the ldxp/stxp instructions. Signed-off-by: Richard Henderson --- accel/tcg/ldst_atomicity.c.inc | 60 ++++++++++++++++++++++++++++++++-- 1 file changed, 57 insertions(+), 3 deletions(-) diff --git a/accel/tcg/ldst_atomicity.c.inc b/accel/tcg/ldst_atomicity.c.inc index 69c1c61997..c3b2b35823 100644 --- a/accel/tcg/ldst_atomicity.c.inc +++ b/accel/tcg/ldst_atomicity.c.inc @@ -263,7 +263,22 @@ static Int128 load_atomic16_or_exit(CPUArchState *env, uintptr_t ra, void *pv) * In system mode all guest pages are writable, and for user-only * we have just checked writability. Try cmpxchg. */ -#if defined(CONFIG_CMPXCHG128) +#if defined(__aarch64__) + /* We can do better than cmpxchg for AArch64. */ + { + uint64_t l, h; + uint32_t fail; + + /* The load must be paired with the store to guarantee not tearing. */ + asm("0: ldxp %0, %1, %3\n\t" + "stxp %w2, %0, %1, %3\n\t" + "cbnz %w2, 0b" + : "=&r"(l), "=&r"(h), "=&r"(fail) : "Q"(*p)); + + qemu_build_assert(!HOST_BIG_ENDIAN); + return int128_make128(l, h); + } +#elif defined(CONFIG_CMPXCHG128) /* Swap 0 with 0, with the side-effect of returning the old value. */ { Int128Alias r; @@ -759,7 +774,22 @@ store_atomic16(void *pv, Int128Alias val) return; } #endif -#if defined(CONFIG_CMPXCHG128) +#if defined(__aarch64__) + /* We can do better than cmpxchg for AArch64. */ + { + uint64_t l, h, t; + + qemu_build_assert(!HOST_BIG_ENDIAN); + l = int128_getlo(val.s); + h = int128_gethi(val.s); + + asm("0: ldxp %0, xzr, %1\n\t" + "stxp %w0, %2, %3, %1\n\t" + "cbnz %w0, 0b" + : "=&r"(t), "=Q"(*(__uint128_t *)pv) : "r"(l), "r"(h)); + return; + } +#elif defined(CONFIG_CMPXCHG128) { __uint128_t *pu = __builtin_assume_aligned(pv, 16); __uint128_t o; @@ -857,7 +887,31 @@ static void store_atom_insert_al8(uint64_t *p, uint64_t val, uint64_t msk) static void ATTRIBUTE_ATOMIC128_OPT store_atom_insert_al16(Int128 *ps, Int128Alias val, Int128Alias msk) { -#if defined(CONFIG_ATOMIC128) +#if defined(__aarch64__) + /* + * GCC only implements __sync* primitives for int128 on aarch64. + * We can do better without the barriers, and integrating the + * arithmetic into the load-exclusive/store-conditional pair. + */ + uint64_t tl, th, vl, vh, ml, mh; + uint32_t fail; + + qemu_build_assert(!HOST_BIG_ENDIAN); + vl = int128_getlo(val.s); + vh = int128_gethi(val.s); + ml = int128_getlo(msk.s); + mh = int128_gethi(msk.s); + + asm("0: ldxp %[l], %[h], %[mem]\n\t" + "bic %[l], %[l], %[ml]\n\t" + "bic %[h], %[h], %[mh]\n\t" + "orr %[l], %[l], %[vl]\n\t" + "orr %[h], %[h], %[vh]\n\t" + "stxp %w[f], %[l], %[h], %[mem]\n\t" + "cbnz %w[f], 0b\n" + : [mem] "+Q"(*ps), [f] "=&r"(fail), [l] "=&r"(tl), [h] "=&r"(th) + : [vl] "r"(vl), [vh] "r"(vh), [ml] "r"(ml), [mh] "r"(mh)); +#elif defined(CONFIG_ATOMIC128) __uint128_t *pu, old, new; /* With CONFIG_ATOMIC128, we can avoid the memory barriers. */ From patchwork Mon May 15 14:32:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681974 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1154148wrr; Mon, 15 May 2023 07:47:53 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6owEpt97p0fFHiYEdmpnAdlqyVdzbqUDIIopjqzWpRfJDm3BWwnyaG+7wyVRrYYyhfsdVv X-Received: by 2002:a5d:558d:0:b0:306:3604:255b with SMTP id i13-20020a5d558d000000b003063604255bmr24458990wrv.5.1684162072853; Mon, 15 May 2023 07:47:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684162072; cv=none; d=google.com; s=arc-20160816; b=HDNrwrXSQPnR5wsUwNJdhw/Y5LNmWzZ7CBOYIR1vbYPz1y6W7SA/yYytxipDUlmH14 nUhuS50Qqk9iFDnFp2YUCUb1J/wIHBEBG3s+nVZUp9bbkt6BY7oiUvTeOOIGGSd4BK1N iJUWiVcWFHW820wQvQG+w47d7ZFCPpBAksTzi6Jp2ndKGa8tKnW4eo6GgUJD1JA4x2mg A9UoEypcyrZpjsPMK86DTesSSHtG3O/vztm7IIe2SnFgOe2kvMDk1jbiOfYD7JECMCqN wUZJw3fCfLWarVQR4QXL9KPEuNLNWbI+RXbOphoAHE9nAUVRi+oXPkhTlO0zQhe+dJPp JZgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=WjHj7SEbww3DXTRNkJQIziiBJKKRW1RlTjQnuos+9yc=; b=jnD/DIH0VVl7aiBNVhnP9e0FHXnuZ6vw7mJN8R1wFWSbXTfL8D4148HC2rtHJfXWGe NDZyrCnTOiqsoDrdnMyabutnd4HMTJZO/ZGFI0dbkJV2+Ss28vghLDuPxRiKlea2LBQ3 eu+Gm1uzEChprt+EfCHhTDlb+iYUiA00GdJ8I9YH0EshU/3fpEQQtICtKDoW7AnHWYdp EiDpkCUTEpp59js7nixgdnw0W6Adl6clvcSnT6T5JGTM5JJdxiZMzjqqdX8N6gQ+aWkA S0zHUxetx1cQ3pk9CDqNUvFLN5syUuv75QsEsizsO/L0hK+n4f6DOPw1yDDpOIjE0wP+ dQbQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ahEt4rWQ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id z14-20020adff74e000000b00304638e6f04si16188218wrp.540.2023.05.15.07.47.52 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:47:52 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ahEt4rWQ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIW-00084j-05; Mon, 15 May 2023 10:35:37 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHK-0006vn-UA for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:28 -0400 Received: from mail-pl1-x634.google.com ([2607:f8b0:4864:20::634]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGR-0004uA-59 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:37 -0400 Received: by mail-pl1-x634.google.com with SMTP id d9443c01a7336-1aafa41116fso87159795ad.1 for ; Mon, 15 May 2023 07:33:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161205; x=1686753205; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WjHj7SEbww3DXTRNkJQIziiBJKKRW1RlTjQnuos+9yc=; b=ahEt4rWQeXhEzDbAU6Ui5eLXYz1GIjQ7FlyH69ElqkXdM0GhYfGVf+eevcN8PfxAjj hNbNHcBCHjhNUvqJ/cSllWZd/vCxteA21LGj9hAKlbbeLqC3BvY6XOUAxn5vKM87yv8Z MxCd+mPeM56HV0n5P8DY+ugBXoQPjxezx4hpcMC1vmQmJW5L8U59mKx7qF0ZD0wtH7h6 noI7l94N3Lg4T2pwezIfPSAV/io8kCD2yckQhuT6qFTvSuXbDIOFSEzUNcvv7rEtj8za g/tQssgcVipSDaUzyNuNAkg6Uv/80Xc8x+Au+SUbK3wbrljomqx0ACyoTl1TJXHIMDMf sHwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161205; x=1686753205; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WjHj7SEbww3DXTRNkJQIziiBJKKRW1RlTjQnuos+9yc=; b=KJTAMSzqhlVYrN3VvLQDgfHefOZajnhvDl0vSx7h/KVWiRiB4ia2CHakymyNNmbjtf 2Le88ioX1v+Babm5hRG6dwG8sNh3l4oc1CINTO7OGxVwIfaDl0WDRBc93DOngF4lxxSc JE1IqmxNTPw8/x1XCKvdF7ET2jka1VMODkecV80BaP1A7oCs7ZsEXdbCjGkxxNUWkLKa LXuY89Pp48jHchW9vseT0dNfVnJsFlImx8/4m5aiKBODM0vsl5eOXkwdkWnWak7LjKQD TVS72fNDkis5k2+tnbtzbtZI7nZvSIdzOxUUKhX8OsQY01r1y/yGF6rt12oDxj4S6pMN 2C8Q== X-Gm-Message-State: AC+VfDxOhxxysqbzmBCwMKPFXnBmbP9TYqCmM5aCSaQOiGlWzVLcNOPJ 7hEN5fG2kJRdBsxgkW7l5IFv0Fe3tlAFBYH4pWw= X-Received: by 2002:a17:902:c085:b0:1ab:d16:3c8a with SMTP id j5-20020a170902c08500b001ab0d163c8amr30257003pld.6.1684161205301; Mon, 15 May 2023 07:33:25 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:24 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell , =?utf-8?q?Philippe_Mathieu-Daud?= =?utf-8?q?=C3=A9?= Subject: [PATCH v5 12/54] tcg/aarch64: Detect have_lse, have_lse2 for linux Date: Mon, 15 May 2023 07:32:31 -0700 Message-Id: <20230515143313.734053-13-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::634; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x634.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Notice when the host has additional atomic instructions. The new variables will also be used in generated code. Reviewed-by: Peter Maydell Reviewed-by: Philippe Mathieu-Daudé Signed-off-by: Richard Henderson --- tcg/aarch64/tcg-target.h | 3 +++ tcg/aarch64/tcg-target.c.inc | 12 ++++++++++++ 2 files changed, 15 insertions(+) diff --git a/tcg/aarch64/tcg-target.h b/tcg/aarch64/tcg-target.h index c0b0f614ba..3c0b0d312d 100644 --- a/tcg/aarch64/tcg-target.h +++ b/tcg/aarch64/tcg-target.h @@ -57,6 +57,9 @@ typedef enum { #define TCG_TARGET_CALL_ARG_I128 TCG_CALL_ARG_EVEN #define TCG_TARGET_CALL_RET_I128 TCG_CALL_RET_NORMAL +extern bool have_lse; +extern bool have_lse2; + /* optional instructions */ #define TCG_TARGET_HAS_div_i32 1 #define TCG_TARGET_HAS_rem_i32 1 diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc index e6636c1f8b..fc551a3d10 100644 --- a/tcg/aarch64/tcg-target.c.inc +++ b/tcg/aarch64/tcg-target.c.inc @@ -13,6 +13,9 @@ #include "../tcg-ldst.c.inc" #include "../tcg-pool.c.inc" #include "qemu/bitops.h" +#ifdef __linux__ +#include +#endif /* We're going to re-use TCGType in setting of the SF bit, which controls the size of the operation performed. If we know the values match, it @@ -71,6 +74,9 @@ static TCGReg tcg_target_call_oarg_reg(TCGCallReturnKind kind, int slot) return TCG_REG_X0 + slot; } +bool have_lse; +bool have_lse2; + #define TCG_REG_TMP TCG_REG_X30 #define TCG_VEC_TMP TCG_REG_V31 @@ -2899,6 +2905,12 @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) static void tcg_target_init(TCGContext *s) { +#ifdef __linux__ + unsigned long hwcap = qemu_getauxval(AT_HWCAP); + have_lse = hwcap & HWCAP_ATOMICS; + have_lse2 = hwcap & HWCAP_USCAT; +#endif + tcg_target_available_regs[TCG_TYPE_I32] = 0xffffffffu; tcg_target_available_regs[TCG_TYPE_I64] = 0xffffffffu; tcg_target_available_regs[TCG_TYPE_V64] = 0xffffffff00000000ull; From patchwork Mon May 15 14:32:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681980 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1154569wrr; Mon, 15 May 2023 07:48:54 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7EkTyKLDDINWMpcptaVNMHTEpyIOxhGwXAEDGhyKIAjAFuRWvKKaqma5zm9oLehTSO1Jky X-Received: by 2002:a05:6402:1255:b0:50b:cadd:21e6 with SMTP id l21-20020a056402125500b0050bcadd21e6mr30203208edw.8.1684162134740; Mon, 15 May 2023 07:48:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684162134; cv=none; d=google.com; s=arc-20160816; b=y5YBRwDiSdi7+BSL8Bm2GAcMQqKMc8cwjD2SI9z/4mmMhMVe260mvRloju3UyIAr1r QAoudqPMmB+5zRSWws7rLt4kQ2VYM8tQkv7yXJymFGuMJqXPC5q+bC6lP4x+2z9u0O/M TQUYDpZRxYccQXIjilsekmsE0rLDSrv+IenVyV3FhrVK+yNVVrkP3W9RpLKEIkL1ATSq PRulECihRjzBLbTTP7R72bLnP2hj/E/TztikLlQxBhbaeyUJml6q+Y4Z5yW+rvJ3/bfs 5l4Lfc9pJrXZTmfrsN9hVsfBSQkTR1FJqmaXIsB+W5FtcxF8eHzRLqS+SYg2jf+cWZ0/ miPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=nP6uwN475NI4kEPgZuacjyw/HnU7r6xvn2JKlD8Wx/Y=; b=APCVGp7OIcQCTw68NA7wWmsLmBxcoBOoV7Lf2bVRJ/IfAfIwxMZo3Kgpv/f5PPeKMh 50t9TgRm92Ybx5oNEX5HrfSYDnM51YDNApKZKvyDT5IKpsGgdAyLsbjLkHC6tQZRI76t wjrJdxyH1kiyF/BENUSiK61CIwGR7Ox/k/6AtARaOe91ZayUMHgw8mfgvZM9VuSFl3YC UodJcBpxdYoNUG9HBFMPcT164u5CtjtcMiIdROEJQmy4XgGMIy9PdQ/S4FZYO9wnFUPk iQquCUQ+nSJR1Zyo+SufWk50ZzYZXXUPq2l+k+E1jrOgdjl3fRqTKW9qnaJDJEIK3CUs YUwQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=eR6hW5d2; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id l5-20020a056402344500b0050dba8a2bd0si12654561edc.164.2023.05.15.07.48.54 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:48:54 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=eR6hW5d2; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIv-0001wU-De; Mon, 15 May 2023 10:36:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHS-0006y8-U7 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:36 -0400 Received: from mail-pl1-x62b.google.com ([2607:f8b0:4864:20::62b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGT-0004uP-3r for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:39 -0400 Received: by mail-pl1-x62b.google.com with SMTP id d9443c01a7336-1aaf91ae451so120243185ad.1 for ; Mon, 15 May 2023 07:33:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161206; x=1686753206; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nP6uwN475NI4kEPgZuacjyw/HnU7r6xvn2JKlD8Wx/Y=; b=eR6hW5d2c5jgJx0SalWGMezSFCdPzutuO6EzIUQlbOJLobzQ5YkxfmoobEaOXTXtci i36RiMD8PMHD4eqBDK0KTi/3nMGCJOXxsmoLVNnL759l2WY/jFLPbWzXojJa8S0otzWR tnU2B3jv1kmA+yrbbBWOi9ngiY+P4z0E/H6Cmz6Serl4LYjBZNeyOw0QIyf9jS+bUKM5 y7juefpleZmc/K3ruLul8K49999Rwfbz8bBm+8GCfQtEsiDMjhINNvnQBA0zn1XIlSbA K/lk91bXgYvv2SSX9Np+v5kr/bt2/Vzox5wO4nUZVCeR6jf9cpcoAjin2UeAnLSafcIB WiEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161206; x=1686753206; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nP6uwN475NI4kEPgZuacjyw/HnU7r6xvn2JKlD8Wx/Y=; b=RPai/wBVEGMOeF9hqIm0koVo/PjlSsAvfHA827bemg0yifzazjcCncpsi9OzF9+SBi 0KCulgS0YU92r+98U6BAz+QT4bdV1MmtMt9LfrJuSbBhcQg4ktOaDIgAvaCGP2w+EOqD NdPK5cAoArbaZDtfauVNZ7dpTCXCn2T4gb9dcwndezRiQcEn9cGSHSbkWhshi1wF+6Zb lf1WdZonTXkBmSHJ7aReVN//8J10LLbiQJ7ujsnwMSKZBHKwlHGLguS4cr+ylXS/BNog /2KhVkuOlZyLy0Vme9lKkWN+qjPBUteElHIFBeCJOgHsVSKfqsJefuHNSxhWTWRAR2FG LjbQ== X-Gm-Message-State: AC+VfDxENNgGLhEHzdXuXcDfqSOSQpfHw6iqBnwBqDfUmMGnTgRV2jZO FyO3/MbFXR/iWN0ZwCU2I5hDWW5uyFIeNnnxKEs= X-Received: by 2002:a17:902:db10:b0:1ac:5c90:23e with SMTP id m16-20020a170902db1000b001ac5c90023emr40924197plx.7.1684161206122; Mon, 15 May 2023 07:33:26 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:25 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, =?utf-8?q?Philippe_Mathieu-D?= =?utf-8?q?aud=C3=A9?= , Peter Maydell Subject: [PATCH v5 13/54] tcg/aarch64: Detect have_lse, have_lse2 for darwin Date: Mon, 15 May 2023 07:32:32 -0700 Message-Id: <20230515143313.734053-14-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62b; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org These features are present for Apple M1. Tested-by: Philippe Mathieu-Daudé Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/aarch64/tcg-target.c.inc | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc index fc551a3d10..c64606af5b 100644 --- a/tcg/aarch64/tcg-target.c.inc +++ b/tcg/aarch64/tcg-target.c.inc @@ -16,6 +16,9 @@ #ifdef __linux__ #include #endif +#ifdef CONFIG_DARWIN +#include +#endif /* We're going to re-use TCGType in setting of the SF bit, which controls the size of the operation performed. If we know the values match, it @@ -2903,6 +2906,27 @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) } } +#ifdef CONFIG_DARWIN +static bool sysctl_for_bool(const char *name) +{ + int val = 0; + size_t len = sizeof(val); + + if (sysctlbyname(name, &val, &len, NULL, 0) == 0) { + return val != 0; + } + + /* + * We might in the future ask for properties not present in older kernels, + * but we're only asking about static properties, all of which should be + * 'int'. So we shouln't see ENOMEM (val too small), or any of the other + * more exotic errors. + */ + assert(errno == ENOENT); + return false; +} +#endif + static void tcg_target_init(TCGContext *s) { #ifdef __linux__ @@ -2910,6 +2934,10 @@ static void tcg_target_init(TCGContext *s) have_lse = hwcap & HWCAP_ATOMICS; have_lse2 = hwcap & HWCAP_USCAT; #endif +#ifdef CONFIG_DARWIN + have_lse = sysctl_for_bool("hw.optional.arm.FEAT_LSE"); + have_lse2 = sysctl_for_bool("hw.optional.arm.FEAT_LSE2"); +#endif tcg_target_available_regs[TCG_TYPE_I32] = 0xffffffffu; tcg_target_available_regs[TCG_TYPE_I64] = 0xffffffffu; From patchwork Mon May 15 14:32:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681951 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1150140wrr; Mon, 15 May 2023 07:38:48 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4FxGIUiPJhWhmTe1b8WVFHORDSt9dpjmXBtvSfeTTr91yXC7Bk1Ako9Z5M1IlCroG7f0vh X-Received: by 2002:a7b:ca51:0:b0:3f4:2148:e8e5 with SMTP id m17-20020a7bca51000000b003f42148e8e5mr20534015wml.1.1684161528511; Mon, 15 May 2023 07:38:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161528; cv=none; d=google.com; s=arc-20160816; b=bVDi7QOTP2QdDARRyEm/IpL5y20KKlyQWLOx4xGbOpVNOpe7Q4Fakh6mKlx8VhN2V+ TLYY2eGGTaaQlUZwEcdrg7YvpDegeOQPP3CWcqBWAaUEyC6HpC+GWVvIebuEbi10JOhK kJRSn53ri+eKY5oraxLylW4Q3PBLXE11tJb/OoaUcz9erNjXbYS7VJFDEhuFM6McEf2U gXa/vePby6/qWPXlofvImZDR29XeJFo3h1ijUNiSWYX2ZobK1ck5gfYu2pfhvBzO2E/d QOKP/kn48VAPDcvGVqIp5jdsTZEtMnCyeZbQpLVfq80o6K/FrK5L4IveUoTGOaqlvvze Ispg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=DMoWoQ65Io3QGIn4AARUwI/lV3SXmcaQgibucuZllj0=; b=Yb8iS24ow51Sb0K9AdKJP31yYytzB4HZLSAbX+9JXWA17C4eFExLvRQXJDbgtjOXzo PNbV74PfqvbD37beLocxgyD6eS3Gc/HZWiybnwUI88ybDEKGoaHCam0ANO/xjz49RlHa fZbQIEf8NJsrj+WZNxeLhZ6OfAJWaX5MgvVAh7T5og1dhZE71jEdiXhR2CRdegg8E0IH A4gQgGA/Sd8X6AsHTwgIpRpkqZbdGHAbmf/61nGTxfrVfZy0FCsCWi1WgfM9nrGGZoYO 6KZH9AJwwcm3Cm5p70u7FJwMRTgs9R0ywsh7Ny5ZMXBLUNLOxc+Zz/wtpGphztfjbCMV bfnA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=iFPd+TIF; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id i2-20020a05600c354200b003f16fc56e81si10219464wmq.4.2023.05.15.07.38.48 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:38:48 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=iFPd+TIF; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIc-0000Rs-Nu; Mon, 15 May 2023 10:35:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHS-0006xb-Ft for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:36 -0400 Received: from mail-pl1-x62a.google.com ([2607:f8b0:4864:20::62a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGT-0004ur-5e for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:40 -0400 Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-1ab1ce53ca6so91622325ad.0 for ; Mon, 15 May 2023 07:33:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161207; x=1686753207; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DMoWoQ65Io3QGIn4AARUwI/lV3SXmcaQgibucuZllj0=; b=iFPd+TIF2ky4pI0KR/xAtEw8XBlXrQnDOhq7Rpm2BD88C6AnIOBLbMOconYCQ5s9f9 OYUt2UxfEGWceiTNmOhFe80Ql/Ws7Mo5uONbYVyz6cRYOUXCPCoIYZayCtz5mbl8KqJs YSwQQXKuW9YLIgw0Pf/YRBDhAXpDOI9bky8Q6m3icKjGhRPYoToxEEtT7/wWoJRR6NEJ MwftKv/dzEU0o0/jxqekloG+9zkPWjpxwVVFkpgTUTDpv8/sdxOwoWFeSE9MQ60ExIYR 1wt/jHvPJoQDfal8AryeOOIr9kQYTHIk81AEUf5SWDO8OKzYj6Ra6LrcllYv1w4/Dxlz 3mXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161207; x=1686753207; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DMoWoQ65Io3QGIn4AARUwI/lV3SXmcaQgibucuZllj0=; b=bu15FZ0c0qnuPRJWbIBy+lbCYkYJB4gjnqGAyfa/vtcrBB74CyhovHgtYdMgcEPcTG jCPrdT6P7B/HY71SP2iAR3xFOq6AFXiDjSBvEn44/dBZIx5FW/6WbtH2z8JXtNLtxl7V ipueH6FKC96JQfdRcvFzziMCF78EatRSSIVPBczz4tfdF/+ye7FMuYtZHMN68u6v+RBK uPpgCiIyDCaxAa7+CFwbZZuW/eemCHXNAN2R+nkRDe4aAVbyYkSk/1a2XO8uQcpqgUPr +Nk2dJz5hmxDdDmHC04fKA6EXJGDlq9V9pcY05+umpykpnc5JKvf5thqATlXHwi1x/DA 7zCQ== X-Gm-Message-State: AC+VfDyY+WIcLiMi4IgA1y40itQQdgOqKhS/5Grg/lVIAM/eXzs0k3yr wgxiQNFAYEEUg5dH5p52dBc+uLhiFdh+Ml0/aFg= X-Received: by 2002:a17:903:22cd:b0:1a2:19c1:a974 with SMTP id y13-20020a17090322cd00b001a219c1a974mr41100139plg.68.1684161206843; Mon, 15 May 2023 07:33:26 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:26 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 14/54] accel/tcg: Add have_lse2 support in ldst_atomicity Date: Mon, 15 May 2023 07:32:33 -0700 Message-Id: <20230515143313.734053-15-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62a; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Add fast paths for FEAT_LSE2, using the detection in tcg. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- accel/tcg/ldst_atomicity.c.inc | 37 ++++++++++++++++++++++++++++++---- 1 file changed, 33 insertions(+), 4 deletions(-) diff --git a/accel/tcg/ldst_atomicity.c.inc b/accel/tcg/ldst_atomicity.c.inc index c3b2b35823..baa935ab8b 100644 --- a/accel/tcg/ldst_atomicity.c.inc +++ b/accel/tcg/ldst_atomicity.c.inc @@ -41,6 +41,8 @@ * but we're using tcg/tci/ instead. */ # define HAVE_al16_fast false +#elif defined(__aarch64__) +# define HAVE_al16_fast likely(have_lse2) #elif defined(__x86_64__) && defined(CONFIG_INT128) # define HAVE_al16_fast likely(have_atomic16) #else @@ -48,6 +50,8 @@ #endif #if defined(CONFIG_ATOMIC128) || defined(CONFIG_CMPXCHG128) # define HAVE_al16 true +#elif defined(__aarch64__) +# define HAVE_al16 true #else # define HAVE_al16 false #endif @@ -186,6 +190,14 @@ load_atomic16(void *pv) r.u = qatomic_read__nocheck(p); return r.s; +#elif defined(__aarch64__) + uint64_t l, h; + + /* Via HAVE_al16_fast, FEAT_LSE2 is present: LDP becomes atomic. */ + asm("ldp %0, %1, %2" : "=r"(l), "=r"(h) : "m"(*(__uint128_t *)pv)); + + qemu_build_assert(!HOST_BIG_ENDIAN); + return int128_make128(l, h); #elif defined(__x86_64__) && defined(CONFIG_INT128) Int128Alias r; @@ -428,6 +440,18 @@ load_atom_extract_al16_or_al8(void *pv, int s) r = qatomic_read__nocheck(p16); } return r >> shr; +#elif defined(__aarch64__) + /* + * Via HAVE_al16_fast, FEAT_LSE2 is present. + * LDP becomes single-copy atomic if 16-byte aligned, and + * single-copy atomic on the parts if 8-byte aligned. + */ + uintptr_t pi = (uintptr_t)pv; + int shr = (pi & 7) * 8; + uint64_t l, h; + + asm("ldp %0, %1, %2" : "=r"(l), "=r"(h) : "m"(*(__uint128_t *)(pi & ~7))); + return (l >> shr) | (h << (-shr & 63)); #elif defined(__x86_64__) && defined(CONFIG_INT128) uintptr_t pi = (uintptr_t)pv; int shr = (pi & 7) * 8; @@ -783,10 +807,15 @@ store_atomic16(void *pv, Int128Alias val) l = int128_getlo(val.s); h = int128_gethi(val.s); - asm("0: ldxp %0, xzr, %1\n\t" - "stxp %w0, %2, %3, %1\n\t" - "cbnz %w0, 0b" - : "=&r"(t), "=Q"(*(__uint128_t *)pv) : "r"(l), "r"(h)); + if (HAVE_al16_fast) { + /* Via HAVE_al16_fast, FEAT_LSE2 is present: STP becomes atomic. */ + asm("stp %1, %2, %0" : "=Q"(*(__uint128_t *)pv) : "r"(l), "r"(h)); + } else { + asm("0: ldxp %0, xzr, %1\n\t" + "stxp %w0, %2, %3, %1\n\t" + "cbnz %w0, 0b" + : "=&r"(t), "=Q"(*(__uint128_t *)pv) : "r"(l), "r"(h)); + } return; } #elif defined(CONFIG_CMPXCHG128) From patchwork Mon May 15 14:32:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681971 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1152783wrr; Mon, 15 May 2023 07:44:36 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5GKuglst53R0RrF8cDgBmXH7nlQWdpFt7CZMWIXt3A/XdPJ0ysbCaROydKqHCrsfW9ebbr X-Received: by 2002:a7b:c3cf:0:b0:3f4:21cf:b4a4 with SMTP id t15-20020a7bc3cf000000b003f421cfb4a4mr19037797wmj.20.1684161876242; Mon, 15 May 2023 07:44:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161876; cv=none; d=google.com; s=arc-20160816; b=aQWrCtTN4+s6kXtBrN1KrCvSRgvyAE/z2Eg5J1L6NlKHNs/QGlr8HdcjAHVwab/FPn 2Ww8njkIqYCjdGZ7IN7/X+ghBXiinr2I+mvKhexKe/MAtyD4UYuIGf7hoZWDbFedYRgf +IaLEPEAYYLwPPUCozbCQGfUU75FGJUSmd50u+9S7orcMDzWkcoC3Q0wCXcQmJC3bpqL KOzsV8iJVcTFFD+uA8Tq2irUYaXXqI421k+z6/Qbak/W3T6qOTBIbWiaX41kg2I0Qx1A /e9nihLxCYv7tlCDuYPJZmSkrAVUR4qV4Y1def4+clqjNGRhVVn7v6dCUju0skyAL+QG Kmbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=0BFtx9KYq0Th2HWZb/BY+/P+rVcQ7G1nOi7Bio8e4oI=; b=pvG2YIJ4rAphKF8bcNR8pIcQRPfuHFHZfWsP2Ab8ODkdXQqOuwpHlyUxZbG6iMIULr 99/HPhccQCeA9ni6pdm9ajpHJ4/BhQKQu555dL2zjfhPBkjNy3pERbv7MBb40Bi5wMFY 7jMQIqwJjNpLX/OlNZef/5boibDan+N7J9YS6+uWXdkjstTnjBjWCqv1DCKpuozpBsq3 Hzt8F9XUqqsV7ApisTyJnmlXhhS8it2i6kNCwE7ga4e31yC9/pIQpIvRmBczxCbwpQqg sa26Am7LJv8kkh6WDiBKU0G3YSqIvjsHDKS24SbR2XK2O+0ppG7i4+y0bGSjw9pkYkkK 21XQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=D0mLsNBO; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id z1-20020a05600c0a0100b003f315cf2214si8896949wmp.76.2023.05.15.07.44.35 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:44:36 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=D0mLsNBO; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIq-0001gv-RP; Mon, 15 May 2023 10:35:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHS-0006y0-Ov for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:36 -0400 Received: from mail-pl1-x62a.google.com ([2607:f8b0:4864:20::62a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGT-0004vD-UB for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:41 -0400 Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-1aaef97652fso88356225ad.0 for ; Mon, 15 May 2023 07:33:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161208; x=1686753208; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0BFtx9KYq0Th2HWZb/BY+/P+rVcQ7G1nOi7Bio8e4oI=; b=D0mLsNBOlB4E1d/QXUJ3CBVX0Ojzxk1EkB2fw7iVbK3DU/z78M4krW//IrJ+lEcd6n fss0ZJd4kuYXC8lNa9LV9aaSsfJtqyrJFAaUajrPSJPQSkr6AhZGy3HBH8QO7JOoCbIJ BA2Ek+fEGtlXV+zJKIPl2cuqwWk80EXh45c8vwjpNN8QbmbMpV/X2QOP1ZVCC7xPrPIV NyNL0Inu5lMZopsp/MhXry2d4DHxogEykgyN/bYmwfb5NbDLxUBARlq/rkWOGcQ6k07Y 7lzouWjfz2hiFnT1Vsoa16i2byGf+k3IJM8R4LAMMZK03+aV2pOHMJSOBPqRJyBYZ8bB 12uQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161208; x=1686753208; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0BFtx9KYq0Th2HWZb/BY+/P+rVcQ7G1nOi7Bio8e4oI=; b=EvqkVzlelTEjqgqVAhgCAoOEfZfS4XdxczSkVA/Ah5arzuTIrqSIUoPhjry94difb0 ThcteOOS1cY61jwz/niLOplmZQM59kjX0OvDjOZhBVQlPPOVtoWlTT/caPcvgbdSt4QS opIqx2g56zz00SI5QDbQf8/inhxowJuDTu9dLaGpuIpIr0BzXIwJPdtQ6OgNyjhCqfDI 3Vrqc6VElxvxJmBK/1uVx3zXKnJYm6CgiMyAFmHCOMToDOq2hx1gPonLU7QzunpwVc7L BZTtDRraQgvlMYcuEvnfYh27jWim62qgPaL6Qf8FztxLjhpB3Y5otOGTeRlJZ20tosyr J1Cw== X-Gm-Message-State: AC+VfDz+rEZPGuvRidiiERhpdOkpaGKa/zi+uHUKqVPXMpAKqgOhSqaz cicbVO6IalOUFj7a3TrYWmVELjTcTcDDO7Shqik= X-Received: by 2002:a17:903:120d:b0:1aa:f53a:5e47 with SMTP id l13-20020a170903120d00b001aaf53a5e47mr40736600plh.16.1684161207962; Mon, 15 May 2023 07:33:27 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:27 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 15/54] tcg/i386: Use full load/store helpers in user-only mode Date: Mon, 15 May 2023 07:32:34 -0700 Message-Id: <20230515143313.734053-16-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62a; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Instead of using helper_unaligned_{ld,st}, use the full load/store helpers. This will allow the fast path to increase alignment to implement atomicity while not immediately raising an alignment exception. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/i386/tcg-target.c.inc | 52 +++------------------------------------ 1 file changed, 4 insertions(+), 48 deletions(-) diff --git a/tcg/i386/tcg-target.c.inc b/tcg/i386/tcg-target.c.inc index 911123cfa8..21553f3c39 100644 --- a/tcg/i386/tcg-target.c.inc +++ b/tcg/i386/tcg-target.c.inc @@ -1776,7 +1776,6 @@ typedef struct { int seg; } HostAddress; -#if defined(CONFIG_SOFTMMU) /* * Because i686 has no register parameters and because x86_64 has xchg * to handle addr/data register overlap, we have placed all input arguments @@ -1812,7 +1811,7 @@ static bool tcg_out_qemu_ld_slow_path(TCGContext *s, TCGLabelQemuLdst *l) /* resolve label address */ tcg_patch32(label_ptr[0], s->code_ptr - label_ptr[0] - 4); - if (TARGET_LONG_BITS > TCG_TARGET_REG_BITS) { + if (label_ptr[1]) { tcg_patch32(label_ptr[1], s->code_ptr - label_ptr[1] - 4); } @@ -1834,7 +1833,7 @@ static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *l) /* resolve label address */ tcg_patch32(label_ptr[0], s->code_ptr - label_ptr[0] - 4); - if (TARGET_LONG_BITS > TCG_TARGET_REG_BITS) { + if (label_ptr[1]) { tcg_patch32(label_ptr[1], s->code_ptr - label_ptr[1] - 4); } @@ -1844,51 +1843,8 @@ static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *l) tcg_out_jmp(s, l->raddr); return true; } -#else -static bool tcg_out_fail_alignment(TCGContext *s, TCGLabelQemuLdst *l) -{ - /* resolve label address */ - tcg_patch32(l->label_ptr[0], s->code_ptr - l->label_ptr[0] - 4); - - if (TCG_TARGET_REG_BITS == 32) { - int ofs = 0; - - tcg_out_st(s, TCG_TYPE_PTR, TCG_AREG0, TCG_REG_ESP, ofs); - ofs += 4; - - tcg_out_st(s, TCG_TYPE_I32, l->addrlo_reg, TCG_REG_ESP, ofs); - ofs += 4; - if (TARGET_LONG_BITS == 64) { - tcg_out_st(s, TCG_TYPE_I32, l->addrhi_reg, TCG_REG_ESP, ofs); - ofs += 4; - } - - tcg_out_pushi(s, (uintptr_t)l->raddr); - } else { - tcg_out_mov(s, TCG_TYPE_TL, tcg_target_call_iarg_regs[1], - l->addrlo_reg); - tcg_out_mov(s, TCG_TYPE_PTR, tcg_target_call_iarg_regs[0], TCG_AREG0); - - tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_RAX, (uintptr_t)l->raddr); - tcg_out_push(s, TCG_REG_RAX); - } - - /* "Tail call" to the helper, with the return address back inline. */ - tcg_out_jmp(s, (const void *)(l->is_ld ? helper_unaligned_ld - : helper_unaligned_st)); - return true; -} - -static bool tcg_out_qemu_ld_slow_path(TCGContext *s, TCGLabelQemuLdst *l) -{ - return tcg_out_fail_alignment(s, l); -} - -static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *l) -{ - return tcg_out_fail_alignment(s, l); -} +#ifndef CONFIG_SOFTMMU static HostAddress x86_guest_base = { .index = -1 }; @@ -1920,7 +1876,7 @@ static inline int setup_guest_base_seg(void) return 0; } #endif /* setup_guest_base_seg */ -#endif /* SOFTMMU */ +#endif /* !SOFTMMU */ /* * For softmmu, perform the TLB load and compare. From patchwork Mon May 15 14:32:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681978 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1154306wrr; Mon, 15 May 2023 07:48:17 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5VCEvk9E7Xd2YLxgKNMu5fqjBAcmu8hv2xlBviRy6SXqSciA8LAQFTVaP+zN5CA4HiJw6z X-Received: by 2002:a5d:604e:0:b0:2f6:9bc4:76f1 with SMTP id j14-20020a5d604e000000b002f69bc476f1mr24896740wrt.1.1684162097549; Mon, 15 May 2023 07:48:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684162097; cv=none; d=google.com; s=arc-20160816; b=nME951n3u3qdQDr9tR7OW+oQImUFKzPXd9vlGUnuv8QN7f6L7XZbCj3tKAWYExwv0e yrjS4KvooxWIoJdwtUhoFBfCQU9oT4v1klBcs6K4HSpIfvFiXKzg6sZrwCc5t2gXNwHJ jcbt8DjOZ1ua+uQ5UJMJ1YYyCH00W8AGtz6fB99bijT8TbGLiW0Evykfed9nxrfaCB5A +W+yoBkXMAtE11R7kaKuZ41nXkUR/nsKAI8Gv4m55B5/pb/EpLPyRdr4WHhtty2W2VmJ rDCVmYxJvN5GnC2xi2eKcu28RbM0wx//x2mPeVaBW1oZ+04VGrmN8L7Rk8N1MIn7ScpP HYVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=jxeAK+0nBaqYYTLaoVSvaQrIxJ38Tit6zinqkGACFIA=; b=d0iUeDXsHmM2JPJWm0DSfqoPqNIiTfJm7fxNod4vGelu+4iJGlriSVvXuSZpxd2GBK 3px/U87dxIaCO4tRMdNC3UtF3jCC2PU18uAfe2qp0riuyVxs0qxlG7LjhwDMVvB9ZGKk h6hylfmuyU0l8BdsY1AOBSXlkesIZP87oMHJVipf8fm2+GuiWIqw2n9nbWcozlgxHjo7 A/XE6uWmvnuKMitEB3p50ZirWLguoABgW1KO8v/GEDujVeGKQ3mLhxu2TUYLMoUA99pu V+bvsKJck9hjKqlSTljD5DgRxXj9h4amU4tJZDWtnhhQRnpdOUr6pR0P72kleR9fIW20 VYGw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="P/gzS0Vk"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id n10-20020a5d6b8a000000b00307945980a1si11678380wrx.1032.2023.05.15.07.48.17 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:48:17 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="P/gzS0Vk"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIp-0001VU-Gi; Mon, 15 May 2023 10:35:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHT-0006yB-59 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:36 -0400 Received: from mail-pl1-x62a.google.com ([2607:f8b0:4864:20::62a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGU-0004vV-Aj for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:41 -0400 Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-1aaebed5bd6so90781835ad.1 for ; Mon, 15 May 2023 07:33:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161209; x=1686753209; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jxeAK+0nBaqYYTLaoVSvaQrIxJ38Tit6zinqkGACFIA=; b=P/gzS0VkxIZ7eGFXLCopzy44T1r+4+EnpyMnmvd5Fb51AVs+ozKrNUS+wVPH5m/aQj 4wan5D8gXim2b/eXpq9CwTT0w50OU2QAE4lntAcW9FgaJ/SxBO1m9rQR9f4zGbDRe6sg JYW6PTw5ed1t5//gYnuZh1eJ8eXpKjDPbZTn8syBdddLpbnOqVFwkCWroRauxKhShh4F n2C/fU4QXrsmIVGUIbcEcJctSig0nlgXHl3ur3YcyV+C/gKJpZ13Cp3VtmTfgAjR/i0g 33lJ5i5olmwD3mmnbcnx4pe5wcccDMNSH5s1i3Ok0IktmouQHWNUwEzzWg3CemqMkfTP z/tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161209; x=1686753209; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jxeAK+0nBaqYYTLaoVSvaQrIxJ38Tit6zinqkGACFIA=; b=P97lOyTFkgGlyVtfJcid+2stCcSFelu1R9Mn0wQFYoRMeSdVYGC9wr0c8lqk+m4MP9 rkCIR9VTzGcr6YQndz4G4WfwMuMM7Ur6FYGU72F3HohTJWNGrX6qLVwExD35yyDmcKNZ ueNpQiSo9frSObbabB3qRVzxwQVfkoWpGFYpkGBoX0l6Mk4HQpDBQprfrSatOHf2Kbja ECytPMu3KnkIr3drUkEYSx3Khu9wJhcJGLZaaC4iULbWSgF+VI2oCfRqAAmBz/WLXEqs Wga/ouqR83v4Wd53/jR87BNmuShB1jzRkcPc0TshBovNbrNA3I2tAwts8S7JLxmutoMa 8Yxg== X-Gm-Message-State: AC+VfDzJmh/dsw8Ldd7V63eAWq5neVdoRCNRMBMv9n5dVBJG2t9ISZSl 6ZtcjMAIi0mZLfDqtrd37v0iun4gIZvwTWFw9qA= X-Received: by 2002:a17:902:d2c4:b0:1ac:5717:fd2 with SMTP id n4-20020a170902d2c400b001ac57170fd2mr34765471plc.47.1684161208766; Mon, 15 May 2023 07:33:28 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:28 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 16/54] tcg/aarch64: Use full load/store helpers in user-only mode Date: Mon, 15 May 2023 07:32:35 -0700 Message-Id: <20230515143313.734053-17-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62a; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Instead of using helper_unaligned_{ld,st}, use the full load/store helpers. This will allow the fast path to increase alignment to implement atomicity while not immediately raising an alignment exception. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/aarch64/tcg-target.c.inc | 35 ----------------------------------- 1 file changed, 35 deletions(-) diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc index c64606af5b..36d8798bca 100644 --- a/tcg/aarch64/tcg-target.c.inc +++ b/tcg/aarch64/tcg-target.c.inc @@ -1595,7 +1595,6 @@ typedef struct { TCGType index_ext; } HostAddress; -#ifdef CONFIG_SOFTMMU static const TCGLdstHelperParam ldst_helper_param = { .ntmp = 1, .tmp = { TCG_REG_TMP } }; @@ -1628,40 +1627,6 @@ static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *lb) tcg_out_goto(s, lb->raddr); return true; } -#else -static void tcg_out_adr(TCGContext *s, TCGReg rd, const void *target) -{ - ptrdiff_t offset = tcg_pcrel_diff(s, target); - tcg_debug_assert(offset == sextract64(offset, 0, 21)); - tcg_out_insn(s, 3406, ADR, rd, offset); -} - -static bool tcg_out_fail_alignment(TCGContext *s, TCGLabelQemuLdst *l) -{ - if (!reloc_pc19(l->label_ptr[0], tcg_splitwx_to_rx(s->code_ptr))) { - return false; - } - - tcg_out_mov(s, TCG_TYPE_TL, TCG_REG_X1, l->addrlo_reg); - tcg_out_mov(s, TCG_TYPE_PTR, TCG_REG_X0, TCG_AREG0); - - /* "Tail call" to the helper, with the return address back inline. */ - tcg_out_adr(s, TCG_REG_LR, l->raddr); - tcg_out_goto_long(s, (const void *)(l->is_ld ? helper_unaligned_ld - : helper_unaligned_st)); - return true; -} - -static bool tcg_out_qemu_ld_slow_path(TCGContext *s, TCGLabelQemuLdst *l) -{ - return tcg_out_fail_alignment(s, l); -} - -static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *l) -{ - return tcg_out_fail_alignment(s, l); -} -#endif /* CONFIG_SOFTMMU */ /* * For softmmu, perform the TLB load and compare. From patchwork Mon May 15 14:32:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681958 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1151205wrr; Mon, 15 May 2023 07:41:08 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4NFeDybb+zuktiVO5caxI/tQn1r1DXzeMSGD10JP594cYVjRAyNdpQ1aKIMekHBaxyN3yQ X-Received: by 2002:a7b:c381:0:b0:3f1:819d:d046 with SMTP id s1-20020a7bc381000000b003f1819dd046mr28050422wmj.5.1684161668134; Mon, 15 May 2023 07:41:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161668; cv=none; d=google.com; s=arc-20160816; b=tIOvXB+6hmQA6QUiwK0+WaiUQmRK74vCDI8balImIQg6BXt9mJLGPOS+WbjRTAD3kp yREZ6Nc17z/0vm0+zgq7lqz9pMuXmfF1ETUJZgXJRoDCY28v2Us2NdP++mAz6RwfH6xe d2PtCranp5ymhDZm28YpNYMMDD/Wg6sTF4JBA42QagrUu5BRd96XxneW0/QBkBi820SK 5wrGZubyBnP7Zbkxqmvnstb9/nLNPjl0tLI0lSPoa6huzwLI3iPP5dQ/opuVJ+FRoBLM rpNHLoAOlQNzW6TvMGZxmahs7fLq0nL+zce3RQXM1o4IPIG3EZ7SDJuq0Yv/T0RNQSc3 29mw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=EAy5XbfXYPrm/J1tRB5DUCT9JP3KNW8vRC1FRgDgvtM=; b=axvY0qrf5Tuyw0lkqfyTnrocsyhCBrx7cFfoiodd0b6eimem2L3bn2fcscg2ryOeD9 v66SDPn6X0bdM2IklCyGbP1Xnz4vAEKWfCw87Vck/WW/lsAj49tfbkc/CZnw/Sc+Qinl vsQvKSibMZAJlLD74ytgB9WAo/jC/bME/47+n2CcQ69XaNKx01pLYYZHLo+lRQ09xVkn 42/gLJhUBrNs+EuoVINcjuyrW0sw+bkR9kNOxhxVNtcucBgOt8KOeWQUzCGapc2OmsFR U8clvvnnKRy0ytI3OEFoBUnaaF+TZb9p8iTVyQiGhTKJeQODCSOG2YKO3WcYNpOFAWBG smMQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=epYmYlRP; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTP id f13-20020a05600c154d00b003f4fd0e1520si49388wmg.215.2023.05.15.07.41.08 for ; Mon, 15 May 2023 07:41:08 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=epYmYlRP; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIm-0001LT-MC; Mon, 15 May 2023 10:35:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHT-0006y9-49 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:36 -0400 Received: from mail-pf1-x42f.google.com ([2607:f8b0:4864:20::42f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGU-0004rC-Jg for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:42 -0400 Received: by mail-pf1-x42f.google.com with SMTP id d2e1a72fcca58-64a9335a8e7so21248168b3a.0 for ; Mon, 15 May 2023 07:33:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161209; x=1686753209; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EAy5XbfXYPrm/J1tRB5DUCT9JP3KNW8vRC1FRgDgvtM=; b=epYmYlRPfdH9RCCiKJE1xL0m8noeFmpUuvnf9UR3vVNfbQqYDeMOO+4ifo04ejyMFv H6Hq7aZjZWz6PVH0zn9ToEb8XLIYPR5zc6wFUlKWpgDfi7iu1HEK4aLHsNoEqOWA64Zr 0cfZUWhP2QDmgbVXZGJ+WvNAr6MO6KNIZjX8Fq9YG7DB708bu7A3xr/uHS5SwOAzdoV3 duEdBo38WTfJ2xGNLuP4Epj9SyVw0MO4uBBlnKfgpEPKndT3o7QV5pmNfC19hvwkXEdU +2XG3hBbbTog0Y/JraOWlV/XdH9wOkPJc0hXFJclMgi5dm8/H8fVBYCWwlrtbenxUwDj hcyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161209; x=1686753209; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EAy5XbfXYPrm/J1tRB5DUCT9JP3KNW8vRC1FRgDgvtM=; b=DW6BxnQdRC+5o0ZjWsw7s9ICWrh7IrDFTbWbPYyTlANzI3Y5Gn/BxiZIc5f4fW3h8X B9lI6+ZO9m+zOQCAYGMD1q+r6c0eInALBiFIrS3mHiMJuu60UXwOzoIXqNjpjhuf+NLs UXsPgBUzx0eVkyz+/v3IVJOwGp/JRaKhTp69QzyNYSlI98HDLU0LumBKvC8dPSoKUAvg nKvZBu4ajqZcC6pPio5PCpZh7ho8D/RKGm5K2tthTj0Oh04QdxkomQxvqPbB++jy90gh +khPAHJ/tNV2fZ+tOtyvh1GJNsERmI5VdEqxjcS4xW3AoQ0klnc73Vuu7y721caQ1kvu I5jA== X-Gm-Message-State: AC+VfDxtHzoNFjJ+d6n+n96H0Sf85nJXytzDLsPFdvR7CxUvWE4vE+fl b1+CQx7XuivsPht8GLwg6jaM78fNzn02Zxk4UlY= X-Received: by 2002:a17:902:d488:b0:1ad:d2b1:3fa9 with SMTP id c8-20020a170902d48800b001add2b13fa9mr17380961plg.2.1684161209536; Mon, 15 May 2023 07:33:29 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:29 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 17/54] tcg/ppc: Use full load/store helpers in user-only mode Date: Mon, 15 May 2023 07:32:36 -0700 Message-Id: <20230515143313.734053-18-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::42f; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x42f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Instead of using helper_unaligned_{ld,st}, use the full load/store helpers. This will allow the fast path to increase alignment to implement atomicity while not immediately raising an alignment exception. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/ppc/tcg-target.c.inc | 44 ---------------------------------------- 1 file changed, 44 deletions(-) diff --git a/tcg/ppc/tcg-target.c.inc b/tcg/ppc/tcg-target.c.inc index 6a81916e64..218602c10c 100644 --- a/tcg/ppc/tcg-target.c.inc +++ b/tcg/ppc/tcg-target.c.inc @@ -1962,7 +1962,6 @@ static const uint32_t qemu_stx_opc[(MO_SIZE + MO_BSWAP) + 1] = { [MO_BSWAP | MO_UQ] = STDBRX, }; -#if defined (CONFIG_SOFTMMU) static TCGReg ldst_ra_gen(TCGContext *s, const TCGLabelQemuLdst *l, int arg) { if (arg < 0) { @@ -2012,49 +2011,6 @@ static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *lb) tcg_out_b(s, 0, lb->raddr); return true; } -#else -static bool tcg_out_fail_alignment(TCGContext *s, TCGLabelQemuLdst *l) -{ - if (!reloc_pc14(l->label_ptr[0], tcg_splitwx_to_rx(s->code_ptr))) { - return false; - } - - if (TCG_TARGET_REG_BITS < TARGET_LONG_BITS) { - TCGReg arg = TCG_REG_R4; - - arg |= (TCG_TARGET_CALL_ARG_I64 == TCG_CALL_ARG_EVEN); - if (l->addrlo_reg != arg) { - tcg_out_mov(s, TCG_TYPE_I32, arg, l->addrhi_reg); - tcg_out_mov(s, TCG_TYPE_I32, arg + 1, l->addrlo_reg); - } else if (l->addrhi_reg != arg + 1) { - tcg_out_mov(s, TCG_TYPE_I32, arg + 1, l->addrlo_reg); - tcg_out_mov(s, TCG_TYPE_I32, arg, l->addrhi_reg); - } else { - tcg_out_mov(s, TCG_TYPE_I32, TCG_REG_R0, arg); - tcg_out_mov(s, TCG_TYPE_I32, arg, arg + 1); - tcg_out_mov(s, TCG_TYPE_I32, arg + 1, TCG_REG_R0); - } - } else { - tcg_out_mov(s, TCG_TYPE_TL, TCG_REG_R4, l->addrlo_reg); - } - tcg_out_mov(s, TCG_TYPE_TL, TCG_REG_R3, TCG_AREG0); - - /* "Tail call" to the helper, with the return address back inline. */ - tcg_out_call_int(s, 0, (const void *)(l->is_ld ? helper_unaligned_ld - : helper_unaligned_st)); - return true; -} - -static bool tcg_out_qemu_ld_slow_path(TCGContext *s, TCGLabelQemuLdst *l) -{ - return tcg_out_fail_alignment(s, l); -} - -static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *l) -{ - return tcg_out_fail_alignment(s, l); -} -#endif /* SOFTMMU */ typedef struct { TCGReg base; From patchwork Mon May 15 14:32:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681953 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1150382wrr; Mon, 15 May 2023 07:39:21 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5teSg3pXDomCGOyYs1dz4IaiX7OZlqgTm5J1qgSo/V5xnhlQWKUyIzYA1e0NwGUrop5BZp X-Received: by 2002:a1c:7708:0:b0:3f4:278a:bb5a with SMTP id t8-20020a1c7708000000b003f4278abb5amr16817089wmi.14.1684161560889; Mon, 15 May 2023 07:39:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161560; cv=none; d=google.com; s=arc-20160816; b=02QiMV/JkWxs8ag/FSRaxi3SQvjL/qgSHih8oLlZtfKBK9iiNezZ3wj1YXvpmpHxrN UELyPlR0k8r97IYAhLL9OPyhIiHgcwPDxho96yvU72zYrR3jPbFujI1arVcaYJELy8tB +N5M/f+Ot4OWKxv2eGkkeZorjp3TahDckMGFqcwW+mj5LX7KYO3VcLcfIly9A2lvS7OX +zPcp/FbGTXQhPJ8E91IwT8KUwABfgICOM+aFL8qMUzSJIWnXVfS9hHReRIp+oHkmdXD tYOKZ4LD0Cgd5IHv8Iv7tSKLAntYD3hj4vIAi2sLyMNBAe8GvtJ8R47qPFSPkCt+Ghpi x6Gw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=c2X0vEktAU9CIzOCMmP9cn2Oif9GXs/gZcvny2bfDX8=; b=rwmjgEvQekogBjCFR9x9M0blVGVM/NqGm5k6jvy/WTBwqIFi2MVu2jT2yAMk6o2UCs 8mTkCfgfbROoqpGhUmobUoGoLFmQC3IERNQ+zBl5ANi5j3C+hBU8TM+XXK+ITgcc5J7l nAXm2j8bOdQcTioHEx5oBsS6eIZUxentjhowdpM8RYer4R6rdU+WGglqmE7ETufIbJ6p 4tDwWngWOfFbAcNt3wPp7DgAMBZ8oPNRLVhO7+EVZNXaE2ndYHKgcKBI7s+HgtCVY1Px hu21q0e9q548ra5oYYbk5V4Ge0d+46Tn5WnZx5f7z/AY7TNXCmimlqgvTrycI4NNDbxV fqsw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=S3dWbWCL; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id w12-20020a5d404c000000b00306331189d2si14737280wrp.947.2023.05.15.07.39.20 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:39:20 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=S3dWbWCL; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZI8-0007Wp-PD; Mon, 15 May 2023 10:35:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHS-0006xi-Iy for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:36 -0400 Received: from mail-pl1-x62d.google.com ([2607:f8b0:4864:20::62d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGV-0004wG-NG for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:43 -0400 Received: by mail-pl1-x62d.google.com with SMTP id d9443c01a7336-1aaf706768cso98044685ad.0 for ; Mon, 15 May 2023 07:33:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161210; x=1686753210; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=c2X0vEktAU9CIzOCMmP9cn2Oif9GXs/gZcvny2bfDX8=; b=S3dWbWCLjrZXoFH7472Pjl+571El8373lbF7dk4WJUBI0rTU4i/Txyf1n6UtCmu9VI QJ8yRupr+THchXWcWJxfW4Hr3+ZwVWxXsNI+iv6mztqxf+0kh/YWpUB2cfmJH4ec/bCy tL8thNR7atdDRtk/mSwY7biLgi3k341i/iY3tATsOw/6z+2NAr9geDxnqeX1bFZPahrR Hu+swDy7z+63Ep017ZIVZF1YzWG8tTCiMZ0y3wlP2EZLWlbvlqnuOZnqAdQXXHqMXNUP IcLys6D2LzbECBqyPVRRcEqgzai1tlNB8s2K4sgp/mImJIb67VFe39FFepjLujY39ZOL 85yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161210; x=1686753210; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=c2X0vEktAU9CIzOCMmP9cn2Oif9GXs/gZcvny2bfDX8=; b=Uy4mqyXgg5VXWxoRXGl+B84DFkpzKXHbjY99dZQGsu/yGsZ3j6emqa2ToYzKALM4M4 leT+53joPCRSbTbtTrmuV8YW7THQYCk1A7ezWq8ZxarHLBrXAYQtnyxG9iNBrFaoNYmy lHMV5+6q4wgyMmf7v35RR4B0BckDad4qzll3xND1J4HfslxonULFmOLvti7xnPKY39P+ wpY/rpUMzzJreTVulvikAwF7JmR1vjoXyXdpkiGSNruUQvlK53MGsRc+J/ap0iNcxqwl 6ZIU0BRsjAVmvnL2ZkHgj2YsD3klmglCjoSfhI6kX45aEFVahdf1mI4/aaUsbVn0MrOI aE9Q== X-Gm-Message-State: AC+VfDy0VGt1IRI+iePO6/fr1F/Iwf2KNR34GoEFVLs2m5GTpfe/J6QH L9ZB8ITq7a8UzcAPRtiCmrjEM8g/ZgJGE6w0ACU= X-Received: by 2002:a17:902:da8f:b0:1aa:e739:4092 with SMTP id j15-20020a170902da8f00b001aae7394092mr47032971plx.52.1684161210351; Mon, 15 May 2023 07:33:30 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:29 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 18/54] tcg/loongarch64: Use full load/store helpers in user-only mode Date: Mon, 15 May 2023 07:32:37 -0700 Message-Id: <20230515143313.734053-19-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62d; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Instead of using helper_unaligned_{ld,st}, use the full load/store helpers. This will allow the fast path to increase alignment to implement atomicity while not immediately raising an alignment exception. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/loongarch64/tcg-target.c.inc | 30 ------------------------------ 1 file changed, 30 deletions(-) diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index d1bc29826f..e651ec5c71 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -783,7 +783,6 @@ static bool tcg_out_sti(TCGContext *s, TCGType type, TCGArg val, * Load/store helpers for SoftMMU, and qemu_ld/st implementations */ -#if defined(CONFIG_SOFTMMU) static bool tcg_out_goto(TCGContext *s, const tcg_insn_unit *target) { tcg_out_opc_b(s, 0); @@ -822,35 +821,6 @@ static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *l) tcg_out_call_int(s, qemu_st_helpers[opc & MO_SIZE], false); return tcg_out_goto(s, l->raddr); } -#else -static bool tcg_out_fail_alignment(TCGContext *s, TCGLabelQemuLdst *l) -{ - /* resolve label address */ - if (!reloc_br_sk16(l->label_ptr[0], tcg_splitwx_to_rx(s->code_ptr))) { - return false; - } - - tcg_out_mov(s, TCG_TYPE_TL, TCG_REG_A1, l->addrlo_reg); - tcg_out_mov(s, TCG_TYPE_PTR, TCG_REG_A0, TCG_AREG0); - - /* tail call, with the return address back inline. */ - tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_RA, (uintptr_t)l->raddr); - tcg_out_call_int(s, (const void *)(l->is_ld ? helper_unaligned_ld - : helper_unaligned_st), true); - return true; -} - -static bool tcg_out_qemu_ld_slow_path(TCGContext *s, TCGLabelQemuLdst *l) -{ - return tcg_out_fail_alignment(s, l); -} - -static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *l) -{ - return tcg_out_fail_alignment(s, l); -} - -#endif /* CONFIG_SOFTMMU */ typedef struct { TCGReg base; From patchwork Mon May 15 14:32:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681946 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1149900wrr; Mon, 15 May 2023 07:38:18 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5uTx71gvpthrgJ4Y5VD2F27EBMg2gYy0bR3DqjNSi5quGka7HE/DWW2Qxj9dIEVd3jD61n X-Received: by 2002:a5d:428b:0:b0:307:82e3:70cd with SMTP id k11-20020a5d428b000000b0030782e370cdmr26511382wrq.14.1684161498493; Mon, 15 May 2023 07:38:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161498; cv=none; d=google.com; s=arc-20160816; b=XpBuJ8iNAVC3a4NRnq+glTUYghALJiE8f48Y+43KOdAR6HwzuorCVcNmLBSw2+2+NW f369b0VL7SrrcVKgorw5dfKjY3ndfXYvFRuVyOgK2yuAwSC181EaXE6ZGEy/6/GPdX+R m5QkXwiym4rWWD6khPD2Bi7fy8KRPr6uX1eD+RQudHU1RVJRu97d0KiyIna4ht+kQE8C 54ajkW3zoWOtpn4CpMmJJ11bkf9A/rGP5vQA8FGlDDh3zAxZbY0w+7IToojiNUuudlXT FPUIfxhuNw2LKJX/edPz10CsOFrZs2MKjR17Kny3oOZceT5slR+LfTVatvKp7W5RlLO0 PNAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=W19ruQQxWDDIeEDzZGJivenEm5AmU9KSchMTkoB+WqU=; b=d9rM6JR6NIee9pc8MSGDnprS9fXLfL8vdyGDgGGFLWQniZb2PhdCvGAUl9no+chlQ9 Pb6m113gOHlQmjY8mVo0OSWd/mn8LowBe3oCVM0KHfRlMfDHJmVK+Bam7vKMO2kWxmaJ msZ8kmHl8A6oga3bu6hCWv0PZSPZWcyc9w1F/5YFdwe5pgbrWz5f760Iw4mpUtUjAiTl d6E/e4vBf14GKtw3m3jfKNWYJ63hoxVseGhftwFRU/Ll4iDZ4skOXEVi/6MfWgY3dpnB pVs+xjPNyXloMhDC5+19FA+h3lGd/2D+6m5xbG2C991rL63I3uVIvp5d2tQsgFwT5BGX vIXw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=KnuDKmix; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id z17-20020adff1d1000000b003078e433e0csi12824301wro.442.2023.05.15.07.38.18 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:38:18 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=KnuDKmix; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIa-00009E-Es; Mon, 15 May 2023 10:35:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHS-0006xp-Rp for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:36 -0400 Received: from mail-pl1-x631.google.com ([2607:f8b0:4864:20::631]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGW-0004wh-RJ for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:44 -0400 Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-1ab14cb3aaeso91646425ad.2 for ; Mon, 15 May 2023 07:33:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161211; x=1686753211; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=W19ruQQxWDDIeEDzZGJivenEm5AmU9KSchMTkoB+WqU=; b=KnuDKmixTWNf1+WQ9EELsmyOgKu1k2cdwzF0UTOzCEeUro89rK/YZHJRdDHwGP5Hh1 lC+dJ/59V0kQHlOitXxQvEz2nFYylxkVoR26MHaHb4UUsVdf2+AVnUPzfodGrSgC0JQF 8fendN+RfOn5MqehSKRLayGeU683TPWEzPfrnCm97u0uoVjR6qlG924NExUjvPajZ17S qO4vlpHE/6qqARS/bMaFm8zMhDlG+AcfiG6tXMksYWmx3GqAT5sTDu64jBgyGJsJSvHV 12MjiJHR3MDuZoyTYWAuOAztBN6MxEia82EhuswI0IxYxtApoSMAquAAtEiXJyqLxodI wOcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161211; x=1686753211; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W19ruQQxWDDIeEDzZGJivenEm5AmU9KSchMTkoB+WqU=; b=e6GMs9lX/1v/TUs74gL8ChRH+67UPbiDo3gOY27lNWYcMO8RAiSGYFao6UZrxh6d4b kumSgCZStcPUB24n14VfQj8Hl5AXiSVXoOO/RNVXxHok2AmXqs69Q9y37pK+CW6lVkiw TFUnzYHABgmGX2yPqcrX14a59Dh4WXC6eObgckyD/aDa8SN2wdxgDmZBfSVDbqtPeSps sEl8+z1a8eYeEMJ82DnTyrhG1uvC+e/3Ni/rb75kJRUxCsxG2UubdwUNfPsnJYLmTK+T QXH47Q9Ab7lz6IMZI7+CTSx63embjoSScUzQvDR0Es66ZTELGiXaPlR7GwIioohVBBIZ wbHw== X-Gm-Message-State: AC+VfDznZknXksb/ggOluOfrVN7QlegaZ7iLABC0HjV0V2atA6kJcpty AEdtnZTDb5xbIGz3DnEPSGuWCCAc+MIA8ebxTAU= X-Received: by 2002:a17:903:2292:b0:1a2:3108:5cc9 with SMTP id b18-20020a170903229200b001a231085cc9mr46161994plh.40.1684161211318; Mon, 15 May 2023 07:33:31 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:30 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 19/54] tcg/riscv: Use full load/store helpers in user-only mode Date: Mon, 15 May 2023 07:32:38 -0700 Message-Id: <20230515143313.734053-20-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::631; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x631.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Instead of using helper_unaligned_{ld,st}, use the full load/store helpers. This will allow the fast path to increase alignment to implement atomicity while not immediately raising an alignment exception. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/riscv/tcg-target.c.inc | 29 ----------------------------- 1 file changed, 29 deletions(-) diff --git a/tcg/riscv/tcg-target.c.inc b/tcg/riscv/tcg-target.c.inc index 8ed0e2f210..19cd4507fb 100644 --- a/tcg/riscv/tcg-target.c.inc +++ b/tcg/riscv/tcg-target.c.inc @@ -846,7 +846,6 @@ static void tcg_out_mb(TCGContext *s, TCGArg a0) * Load/store and TLB */ -#if defined(CONFIG_SOFTMMU) static void tcg_out_goto(TCGContext *s, const tcg_insn_unit *target) { tcg_out_opc_jump(s, OPC_JAL, TCG_REG_ZERO, 0); @@ -893,34 +892,6 @@ static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *l) tcg_out_goto(s, l->raddr); return true; } -#else -static bool tcg_out_fail_alignment(TCGContext *s, TCGLabelQemuLdst *l) -{ - /* resolve label address */ - if (!reloc_sbimm12(l->label_ptr[0], tcg_splitwx_to_rx(s->code_ptr))) { - return false; - } - - tcg_out_mov(s, TCG_TYPE_TL, TCG_REG_A1, l->addrlo_reg); - tcg_out_mov(s, TCG_TYPE_PTR, TCG_REG_A0, TCG_AREG0); - - /* tail call, with the return address back inline. */ - tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_RA, (uintptr_t)l->raddr); - tcg_out_call_int(s, (const void *)(l->is_ld ? helper_unaligned_ld - : helper_unaligned_st), true); - return true; -} - -static bool tcg_out_qemu_ld_slow_path(TCGContext *s, TCGLabelQemuLdst *l) -{ - return tcg_out_fail_alignment(s, l); -} - -static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *l) -{ - return tcg_out_fail_alignment(s, l); -} -#endif /* CONFIG_SOFTMMU */ /* * For softmmu, perform the TLB load and compare. From patchwork Mon May 15 14:32:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681940 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1149358wrr; Mon, 15 May 2023 07:37:05 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7xvr7toWXRTWj9h/QTZYewWbM2YcZooMIvfZpbkC3nCLeY8nLPYiEMSzEwDZApJT3vrzTP X-Received: by 2002:a5d:514e:0:b0:307:91a0:ede1 with SMTP id u14-20020a5d514e000000b0030791a0ede1mr20747552wrt.7.1684161425062; Mon, 15 May 2023 07:37:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161425; cv=none; d=google.com; s=arc-20160816; b=Noq1skQlfswV8L95PDpOY0JcVnfAFp7thPC2Xejbhqvew/lzltiAQGgifI0OfHWrCW fSAPL+u2HlOxnENCEOmvFRQIPmMFYhqImtU9pJKwsYflABnqoTn2UeiFiRWGCrzUyFxN 0EbNL1yTg3leZP8SRlsGOHA9hwG6DXGyZmI18125yKtazldNf4eFnqSHgUcLMAOrtXC7 vWvxLpNVxX791LlrXcz5u5o7kiZ7+1eQ5EXdM9moTa8RVXGzxShhvh0HRnBhHS2nBq3c nNjJ/dEhECvvRCRdh3geqwwuO2QMhOsgKiQCUq49zquU0zmeZoVRp9pjBThW80ZPD3XD ZAOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=GZ7SPbKoQkb/kckt0ZzABkZ84Z1c924yY+RMLRaPnSA=; b=cFd+rcoPXnPZCBRdsJCsg3FFfVhWsKYWKqf0OboRm5PxKoWFYK9qtrYeQoNW7aAYbN X714w0t0FdJPUbmR1CJbbR43/0Mly1rYELakvGXyHzsWqOBRhFzrVMgpw8MLp4p/KX0b bQ+Su9XQ/oQ+FovMt3WUJCIGj56uLu/Z7QNlce9G2+w9CHFdzU++HKRiP8g3mv5MmNL+ J9FoUCKUfbaz7nWv2+s1W43sYhQCPLXHwyVH0U1F+CkOCPsJxHLgKkGLIfknnDYrYYUV 0ub/McbLheNdnsaOt5sYP4nSMiu99zbfpw4zO3XxIImXlzCXWrqXrWYvbeUnM47K4Uyk v2og== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=AHp8Lh2Q; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id i2-20020a5d4382000000b00307967c55e6si11203978wrq.525.2023.05.15.07.37.04 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:37:05 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=AHp8Lh2Q; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIk-00012w-Gd; Mon, 15 May 2023 10:35:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHS-0006xf-HQ for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:36 -0400 Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGY-0004yE-Pb for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:45 -0400 Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-1ab1ce53ca6so91623425ad.0 for ; Mon, 15 May 2023 07:33:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161212; x=1686753212; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GZ7SPbKoQkb/kckt0ZzABkZ84Z1c924yY+RMLRaPnSA=; b=AHp8Lh2QaBGILnU1XfUsuhB4iv9F8Pz9ErzbaRzFGTHkvlI78NEAbKHx3OLOCA8wLA Es5N18bQvKvuJL8kbP3/cSmgoV9mWvqseXFgNxp0cF2lE7wfGpX4Jlg1iJUfUI5M7wjT boMjogLS2nKXhCMMt+V9oAZ7nknPJtAFd4ufPO/XYUzB++5ppGLdE9ih3/XltWPwX/HM iz5cj0EIEsdrIePGiZf5/O+WGxrdeeia4ylrIwpBPd3CGf9NQEvjKoMa+sZwjee61l+W raEGgXgg0o+YaY7WuUaqUCGmr2Jp+k6TaBM41Dblgz147PRZcmc2dubdJfUIzpqUb5hD ZC1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161212; x=1686753212; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GZ7SPbKoQkb/kckt0ZzABkZ84Z1c924yY+RMLRaPnSA=; b=e8+xwGGL8Ubu4ASUTnmM8rUCia87Fnvt7ih+kPPZ3OEsRF5pjnrNmD2cB/ex/gI+Zv m+yKVgazGZ9LQno3/12v5fKbiYF1PXkkVuFMn7Hq4JvVv5zSsKuvGrYNaCBVW218KkQ3 OY7rE2uV6ikb8z21XJNLxKMjcLUxiO8tocBTHr4HhPnBOgxUSClmlxHAle8Sq6QU56At lskh6ip0a/FH1ec+2wIjoxa7hKfuCC3HxsqBYb/XxP3Ao36Kw8vA6ENryydf0CnZ9NZ4 xUzGca7aTlCHNeCRiWhIzBwRuL6wz0rGb3cCbDYXFjyEqyfzUM46TXYYSpwmZLy/4o+U u4gQ== X-Gm-Message-State: AC+VfDyEsMl+DskYz36SDQlHgM9A7DCyvpLiYROjUwd06kE7Mp71JWXK 3zTsE0Atq+hYit/RpA6iQ90jdyVF4K9dv4Bc1Xw= X-Received: by 2002:a17:902:f9cd:b0:1a5:1e7:86d7 with SMTP id kz13-20020a170902f9cd00b001a501e786d7mr31373272plb.52.1684161212221; Mon, 15 May 2023 07:33:32 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:31 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 20/54] tcg/arm: Adjust constraints on qemu_ld/st Date: Mon, 15 May 2023 07:32:39 -0700 Message-Id: <20230515143313.734053-21-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62f; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Always reserve r3 for tlb softmmu lookup. Fix a bug in user-only ALL_QLDST_REGS, in that r14 is clobbered by the BLNE that leads to the misaligned trap. Remove r0+r1 from user-only ALL_QLDST_REGS; I believe these had been reserved for bswap, which we no longer perform during qemu_st. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- tcg/arm/tcg-target-con-set.h | 16 ++++++++-------- tcg/arm/tcg-target-con-str.h | 5 ++--- tcg/arm/tcg-target.c.inc | 23 ++++++++--------------- 3 files changed, 18 insertions(+), 26 deletions(-) diff --git a/tcg/arm/tcg-target-con-set.h b/tcg/arm/tcg-target-con-set.h index b8849b2478..229ae258ac 100644 --- a/tcg/arm/tcg-target-con-set.h +++ b/tcg/arm/tcg-target-con-set.h @@ -12,19 +12,19 @@ C_O0_I1(r) C_O0_I2(r, r) C_O0_I2(r, rIN) -C_O0_I2(s, s) +C_O0_I2(q, q) C_O0_I2(w, r) -C_O0_I3(s, s, s) -C_O0_I3(S, p, s) +C_O0_I3(q, q, q) +C_O0_I3(Q, p, q) C_O0_I4(r, r, rI, rI) -C_O0_I4(S, p, s, s) -C_O1_I1(r, l) +C_O0_I4(Q, p, q, q) +C_O1_I1(r, q) C_O1_I1(r, r) C_O1_I1(w, r) C_O1_I1(w, w) C_O1_I1(w, wr) C_O1_I2(r, 0, rZ) -C_O1_I2(r, l, l) +C_O1_I2(r, q, q) C_O1_I2(r, r, r) C_O1_I2(r, r, rI) C_O1_I2(r, r, rIK) @@ -39,8 +39,8 @@ C_O1_I2(w, w, wZ) C_O1_I3(w, w, w, w) C_O1_I4(r, r, r, rI, rI) C_O1_I4(r, r, rIN, rIK, 0) -C_O2_I1(e, p, l) -C_O2_I2(e, p, l, l) +C_O2_I1(e, p, q) +C_O2_I2(e, p, q, q) C_O2_I2(r, r, r, r) C_O2_I4(r, r, r, r, rIN, rIK) C_O2_I4(r, r, rI, rI, rIN, rIK) diff --git a/tcg/arm/tcg-target-con-str.h b/tcg/arm/tcg-target-con-str.h index 24b4b59feb..f83f1d3919 100644 --- a/tcg/arm/tcg-target-con-str.h +++ b/tcg/arm/tcg-target-con-str.h @@ -10,9 +10,8 @@ */ REGS('e', ALL_GENERAL_REGS & 0x5555) /* even regs */ REGS('r', ALL_GENERAL_REGS) -REGS('l', ALL_QLOAD_REGS) -REGS('s', ALL_QSTORE_REGS) -REGS('S', ALL_QSTORE_REGS & 0x5555) /* even qstore */ +REGS('q', ALL_QLDST_REGS) +REGS('Q', ALL_QLDST_REGS & 0x5555) /* even qldst */ REGS('w', ALL_VECTOR_REGS) /* diff --git a/tcg/arm/tcg-target.c.inc b/tcg/arm/tcg-target.c.inc index 8b0d526659..a02804dd69 100644 --- a/tcg/arm/tcg-target.c.inc +++ b/tcg/arm/tcg-target.c.inc @@ -353,23 +353,16 @@ static bool patch_reloc(tcg_insn_unit *code_ptr, int type, #define ALL_VECTOR_REGS 0xffff0000u /* - * r0-r2 will be overwritten when reading the tlb entry (softmmu only) - * and r0-r1 doing the byte swapping, so don't use these. - * r3 is removed for softmmu to avoid clashes with helper arguments. + * r0-r3 will be overwritten when reading the tlb entry (softmmu only); + * r14 will be overwritten by the BLNE branching to the slow path. */ #ifdef CONFIG_SOFTMMU -#define ALL_QLOAD_REGS \ +#define ALL_QLDST_REGS \ (ALL_GENERAL_REGS & ~((1 << TCG_REG_R0) | (1 << TCG_REG_R1) | \ (1 << TCG_REG_R2) | (1 << TCG_REG_R3) | \ (1 << TCG_REG_R14))) -#define ALL_QSTORE_REGS \ - (ALL_GENERAL_REGS & ~((1 << TCG_REG_R0) | (1 << TCG_REG_R1) | \ - (1 << TCG_REG_R2) | (1 << TCG_REG_R14) | \ - ((TARGET_LONG_BITS == 64) << TCG_REG_R3))) #else -#define ALL_QLOAD_REGS ALL_GENERAL_REGS -#define ALL_QSTORE_REGS \ - (ALL_GENERAL_REGS & ~((1 << TCG_REG_R0) | (1 << TCG_REG_R1))) +#define ALL_QLDST_REGS (ALL_GENERAL_REGS & ~(1 << TCG_REG_R14)) #endif /* @@ -2203,13 +2196,13 @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) return C_O1_I4(r, r, r, rI, rI); case INDEX_op_qemu_ld_i32: - return TARGET_LONG_BITS == 32 ? C_O1_I1(r, l) : C_O1_I2(r, l, l); + return TARGET_LONG_BITS == 32 ? C_O1_I1(r, q) : C_O1_I2(r, q, q); case INDEX_op_qemu_ld_i64: - return TARGET_LONG_BITS == 32 ? C_O2_I1(e, p, l) : C_O2_I2(e, p, l, l); + return TARGET_LONG_BITS == 32 ? C_O2_I1(e, p, q) : C_O2_I2(e, p, q, q); case INDEX_op_qemu_st_i32: - return TARGET_LONG_BITS == 32 ? C_O0_I2(s, s) : C_O0_I3(s, s, s); + return TARGET_LONG_BITS == 32 ? C_O0_I2(q, q) : C_O0_I3(q, q, q); case INDEX_op_qemu_st_i64: - return TARGET_LONG_BITS == 32 ? C_O0_I3(S, p, s) : C_O0_I4(S, p, s, s); + return TARGET_LONG_BITS == 32 ? C_O0_I3(Q, p, q) : C_O0_I4(Q, p, q, q); case INDEX_op_st_vec: return C_O0_I2(w, r); From patchwork Mon May 15 14:32:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681957 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1150851wrr; Mon, 15 May 2023 07:40:21 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6Z1ecdQPx5Ro+CoiU99GZHR25Li7Me+YMT8GyiOyumIs5zWTrFsD3MS/iqx0ZKoMdpmvZV X-Received: by 2002:a1c:e90c:0:b0:3f1:952a:4bf0 with SMTP id q12-20020a1ce90c000000b003f1952a4bf0mr23504201wmc.33.1684161620966; Mon, 15 May 2023 07:40:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161620; cv=none; d=google.com; s=arc-20160816; b=ACNilfPVcg53xQVHfqvGclUSN2Y06udlpDxiChDgqZiEI6p81Sx4H5AUwNq/h0l3fS ZhmUo0zkwriqXESy38FfvEo9uRw+DQhKK1tS3xCo8SjnK46EE96UOIO4CMy4pZaxtnpT Tewe3+eHqUrRao/pHp2COA8ujjnspj6QMrACIW5kYVkl7czjFxmjZKtYTAl8cCH4aC82 x7K0cIaYsd/JFUHHKZgx2bYayAe6o7n1cBP5xE9vHA6Zc8CAxul/kGZYC+KYX53cd+2b yHKsEZaV/4ecnkf1wpjvEjaEA4kPreBbDxAB7wuyRFHClpwzmUJx+BwIFPCNUW6mDXTm LxIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=gXvb5M+QQ4GlJC4+AWTZXZCLTvkRCJufT4wwLa/Umbo=; b=jQdIdzNFFQiroSNNpOxrmBpae2FKDQjp0EbFd+Q4Tfpd133xtKItorJyLwzfsjQPai VzOvsvwc6mTGAxKcWqYhMnDBvjGN7f5BtrtJ5+iNyWJSXAVv20u/so0JFu3z1X8X8iqM 6HSOBn8l0KhLfGbwlj5QEDhlLOqHnP4J40j/+v2YYASaBlwMpn49h4yzQAhxUWXFz7L4 HmqF+USQ28qiqHLaBS9XpzpfvE2ZIC+/JgJ9skSHN5hPqtld+4Gp89Pw2IGKpO22jH9a St7DsSWQ9qYT5ax/8pAzGfldu+0rpMIzuZXEs0WZY/2aYarPuNvt/P20Rbeo1nWoO9GN D96A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=s4NhjvT2; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id w21-20020a1cf615000000b003f4feb2bbe4si3394202wmc.8.2023.05.15.07.40.20 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:40:20 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=s4NhjvT2; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIb-0000F7-16; Mon, 15 May 2023 10:35:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHS-0006xy-Na for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:36 -0400 Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGZ-0004yl-GR for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:46 -0400 Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-1aaf706768cso98045135ad.0 for ; Mon, 15 May 2023 07:33:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161213; x=1686753213; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gXvb5M+QQ4GlJC4+AWTZXZCLTvkRCJufT4wwLa/Umbo=; b=s4NhjvT2n7X+pja39Pi4BDc1sNK7U0CQVYP46Ulru9k1GoWMfgt59ssGik9AVDTNWg vbfm0HVMdgLD6Yvghnb8VABMDYUed5x9qeu4mMimczP/Q/Cw43VuPYlwejpOwwhdIrQQ xa0Xnxh//jtRJgKj1rGnYMFjYG2KKbbMZYZJhSW6p/i3DJho7xqIjvNex9O8MLXgd2cL Inw0IkGguUeaxzjxUAExJiaX94/lelPAudR1q2bkXDvzLEoJ30BNsTHn0k7r/qeNLmGY RtFzvUKxIbO2IWH2NS/wQ9BSiBhcnuU+kiRBkmY3sptzTgbmTn5P9nJwy8uv2LiBpEPQ UVZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161213; x=1686753213; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gXvb5M+QQ4GlJC4+AWTZXZCLTvkRCJufT4wwLa/Umbo=; b=GvAFvVvrpPbkdvY3UEYIPHZCz0svVa4djv3HZsyGZAnT7iO3YJR6Q/6YIqucePE00y FycUE5Zx08rHOtfIIQbfJ6Le/2N8no5ngjV+OTu3q3mEaTrFIdZ1OLxh/zTHIIwIibrf 5FLNai4Sq4m23TCt6RQkBZAtJJO6QJnOrayMy6wmH/ey1WeWR+bkiCclw2KQR+w+/hhh Vgn+iZ4iGibuTzWVvPt6oFn+eEKAURJZRGCHXiujEaOols+6O112up29+ey+0XagnPN6 FafCLPUkOMAghtZWEuB8qN+pZejiK6CK3ap1vsI/Jq4aTe0bjFUZoy1EyVjIyLoiiOIV Npfw== X-Gm-Message-State: AC+VfDzBtOkXNp8Ya3nd4KW5khvpG0ZgEOImutj/PgM8r9CdOU4HzdrZ ak2e0R448mwbdhNXqnwuTxBSSdL0jNCtujk6Vbs= X-Received: by 2002:a17:903:1247:b0:1ab:afd:903a with SMTP id u7-20020a170903124700b001ab0afd903amr44833633plh.24.1684161213068; Mon, 15 May 2023 07:33:33 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:32 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 21/54] tcg/arm: Use full load/store helpers in user-only mode Date: Mon, 15 May 2023 07:32:40 -0700 Message-Id: <20230515143313.734053-22-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62f; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Instead of using helper_unaligned_{ld,st}, use the full load/store helpers. This will allow the fast path to increase alignment to implement atomicity while not immediately raising an alignment exception. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/arm/tcg-target.c.inc | 45 ---------------------------------------- 1 file changed, 45 deletions(-) diff --git a/tcg/arm/tcg-target.c.inc b/tcg/arm/tcg-target.c.inc index a02804dd69..eb0542f32e 100644 --- a/tcg/arm/tcg-target.c.inc +++ b/tcg/arm/tcg-target.c.inc @@ -1325,7 +1325,6 @@ typedef struct { bool index_scratch; } HostAddress; -#ifdef CONFIG_SOFTMMU static TCGReg ldst_ra_gen(TCGContext *s, const TCGLabelQemuLdst *l, int arg) { /* We arrive at the slow path via "BLNE", so R14 contains l->raddr. */ @@ -1368,50 +1367,6 @@ static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *lb) tcg_out_goto(s, COND_AL, qemu_st_helpers[opc & MO_SIZE]); return true; } -#else -static bool tcg_out_fail_alignment(TCGContext *s, TCGLabelQemuLdst *l) -{ - if (!reloc_pc24(l->label_ptr[0], tcg_splitwx_to_rx(s->code_ptr))) { - return false; - } - - if (TARGET_LONG_BITS == 64) { - /* 64-bit target address is aligned into R2:R3. */ - TCGMovExtend ext[2] = { - { .dst = TCG_REG_R2, .dst_type = TCG_TYPE_I32, - .src = l->addrlo_reg, - .src_type = TCG_TYPE_I32, .src_ext = MO_UL }, - { .dst = TCG_REG_R3, .dst_type = TCG_TYPE_I32, - .src = l->addrhi_reg, - .src_type = TCG_TYPE_I32, .src_ext = MO_UL }, - }; - tcg_out_movext2(s, &ext[0], &ext[1], TCG_REG_TMP); - } else { - tcg_out_mov(s, TCG_TYPE_I32, TCG_REG_R1, l->addrlo_reg); - } - tcg_out_mov(s, TCG_TYPE_PTR, TCG_REG_R0, TCG_AREG0); - - /* - * Tail call to the helper, with the return address back inline, - * just for the clarity of the debugging traceback -- the helper - * cannot return. We have used BLNE to arrive here, so LR is - * already set. - */ - tcg_out_goto(s, COND_AL, (const void *) - (l->is_ld ? helper_unaligned_ld : helper_unaligned_st)); - return true; -} - -static bool tcg_out_qemu_ld_slow_path(TCGContext *s, TCGLabelQemuLdst *l) -{ - return tcg_out_fail_alignment(s, l); -} - -static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *l) -{ - return tcg_out_fail_alignment(s, l); -} -#endif /* SOFTMMU */ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, TCGReg addrlo, TCGReg addrhi, From patchwork Mon May 15 14:32:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681938 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1148836wrr; Mon, 15 May 2023 07:36:09 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6nXT1rvmvDopodXvjDd2HBIqnSCLUF1gQqlu61KGH7yd4McPfq0TEP7QZAxUJKoK1Ev+mp X-Received: by 2002:a05:622a:145:b0:3f5:15bf:94e5 with SMTP id v5-20020a05622a014500b003f515bf94e5mr12995470qtw.17.1684161369216; Mon, 15 May 2023 07:36:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161369; cv=none; d=google.com; s=arc-20160816; b=VmzqLGKmiVCCxgWTWIpMTRxuLY6wKLFjp3i/mksBWxJO1IO+ke706ghA8yZ7en+EXX QPHWZG1AJ0ZqjFS9WQCqJv7vneLTjHGt2JMrLvw7ojEL1i29Id4FM0d9p+I5QBVsMn+1 jJQNIAUVegB6OUKu+mcG3ZLpvC7PUooYwLU0UimAVxt3QFEuB+XLJvkwNhr1rQCpUMe2 Nc3XMsEVbpmfVrtQGsiNPkB4omw5tSohBly9uhBeOPJYQKXBtvXMtBZhJzBllr7LcphW OP5UdRblmJmX9moElCjiDO4o8rflHTo/PjMFcSVzHLy5DPfVL/23wJwf1BnB5qcsoWIj 4Gow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=K2D+W3Frhm1HnR66jCW7Uv+/eEEehjSzUkt7BPDzA4s=; b=gah7uQve9cdoPcMIFXW/FV+EtbhI6A4qRnQI4+DNwrFaqsAxGA4FSx3sLfmrgU5bg0 t9tMZdmFJu8BnmoGGyfdWU9g32bQyThZBQ96Inev9N0vMb1utrSZqOKMV3d0BMYTwoOl JaM7CuJvtneO4LyFPTV8lCVZOj5VjNO49ZOU3CYFjhgHavxbck6Zx6JIHgvX4T6KNa9M ztsk6xlHO+NDrOtOuFWGMhuCwLOynStjZKLtjAMFO9X4/XNj4ZzLNse16E5uQEq2nWqL vr02HD4Qajv050iRQTqqVGHhT+o8k8gOnKHyElZXraeuVKjFWjk+C8S1rOHMTiUx/B2f A/Kw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=J8qNx79y; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id p18-20020a05622a049200b003f0a4a28477si2968038qtx.79.2023.05.15.07.36.09 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:36:09 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=J8qNx79y; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIa-0000Al-SP; Mon, 15 May 2023 10:35:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHS-0006xv-P2 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:36 -0400 Received: from mail-pl1-x62b.google.com ([2607:f8b0:4864:20::62b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZGZ-0004z2-KK for qemu-devel@nongnu.org; Mon, 15 May 2023 10:33:47 -0400 Received: by mail-pl1-x62b.google.com with SMTP id d9443c01a7336-1aad6f2be8eso118634665ad.3 for ; Mon, 15 May 2023 07:33:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161214; x=1686753214; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K2D+W3Frhm1HnR66jCW7Uv+/eEEehjSzUkt7BPDzA4s=; b=J8qNx79yECQ4oVsJrZl9qNxLCQBJ6VMfU+liJyky9EZvbTY5GZVAq0zEKdCbSIvuWJ pGjpyuKUyrlh48BwFikkVR7+D9yQ7PZT44iplBI9nDzBcEom97HNRLipcwKtc1ytU3s/ b+zjhXKbGyjdkVJXCGPgnYE0rUkp4bLVuQHNbiM4s52BH2dbKvDPnnkiiMJ8JX1CUnlz E57vxqUIX3KsevCA8vjihY9iJT3DrKXyUefzlQiQEna5ikvtAqCY177RbYLL/D7UfQSY RnNw0UFo0NxGb4vU7OsPpgeo5sfBJU3P9pRFDedmcyeDpAT0SHrgzS/rcDtBzLRR5s4S Mfgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161214; x=1686753214; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K2D+W3Frhm1HnR66jCW7Uv+/eEEehjSzUkt7BPDzA4s=; b=DQ6Xano+MHJibZGIEXfEkNLt8l1Ad+HxM3SUuS78erw6R6VQqcLoTo9yDWRjnkPXrK RZG/fEzpyMg35wpXDA2+aFtBu61Ex6p21WmkAnQXMzs/Wo0qY47iyjZFhvn+/U5UUaID k18Fh0wol8c1Lx/WJUqw8EkJoOtZ4aZ/8Elc4jVknDIJ/1zyWPWKH/IVCsg8k4Mksjbg MLB72i75l3HR0xzwUFpcE1lyZECV5stJyOTrbKuAgCkioEYOW53XfBW+dbV/GlkYlABd 17RJN8HbN6+E2/hAubOSYsg/IwOyYWDQ1Z48bQD7n1oQXJCvuaRSIZauHOkhbH2DEpje GHdA== X-Gm-Message-State: AC+VfDxq90ZCqHRaUAtmFfZkqcmEvHkQlaMzFyE83iwx7sgYLCzH03lv L8RXtXeicMQldL38GrbdT/xw//s8L+0QW4SmTkI= X-Received: by 2002:a17:902:a402:b0:1a1:9020:f9c7 with SMTP id p2-20020a170902a40200b001a19020f9c7mr32295089plq.44.1684161213900; Mon, 15 May 2023 07:33:33 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:33 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 22/54] tcg/mips: Use full load/store helpers in user-only mode Date: Mon, 15 May 2023 07:32:41 -0700 Message-Id: <20230515143313.734053-23-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62b; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Instead of using helper_unaligned_{ld,st}, use the full load/store helpers. This will allow the fast path to increase alignment to implement atomicity while not immediately raising an alignment exception. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/mips/tcg-target.c.inc | 57 ++------------------------------------- 1 file changed, 2 insertions(+), 55 deletions(-) diff --git a/tcg/mips/tcg-target.c.inc b/tcg/mips/tcg-target.c.inc index 7770ef46bd..fa0f334e8d 100644 --- a/tcg/mips/tcg-target.c.inc +++ b/tcg/mips/tcg-target.c.inc @@ -1075,7 +1075,6 @@ static void tcg_out_call(TCGContext *s, const tcg_insn_unit *arg, tcg_out_nop(s); } -#if defined(CONFIG_SOFTMMU) /* We have four temps, we might as well expose three of them. */ static const TCGLdstHelperParam ldst_helper_param = { .ntmp = 3, .tmp = { TCG_TMP0, TCG_TMP1, TCG_TMP2 } @@ -1088,8 +1087,7 @@ static bool tcg_out_qemu_ld_slow_path(TCGContext *s, TCGLabelQemuLdst *l) /* resolve label address */ if (!reloc_pc16(l->label_ptr[0], tgt_rx) - || (TCG_TARGET_REG_BITS < TARGET_LONG_BITS - && !reloc_pc16(l->label_ptr[1], tgt_rx))) { + || (l->label_ptr[1] && !reloc_pc16(l->label_ptr[1], tgt_rx))) { return false; } @@ -1118,8 +1116,7 @@ static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *l) /* resolve label address */ if (!reloc_pc16(l->label_ptr[0], tgt_rx) - || (TCG_TARGET_REG_BITS < TARGET_LONG_BITS - && !reloc_pc16(l->label_ptr[1], tgt_rx))) { + || (l->label_ptr[1] && !reloc_pc16(l->label_ptr[1], tgt_rx))) { return false; } @@ -1139,56 +1136,6 @@ static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *l) return true; } -#else -static bool tcg_out_fail_alignment(TCGContext *s, TCGLabelQemuLdst *l) -{ - void *target; - - if (!reloc_pc16(l->label_ptr[0], tcg_splitwx_to_rx(s->code_ptr))) { - return false; - } - - if (TCG_TARGET_REG_BITS < TARGET_LONG_BITS) { - /* A0 is env, A1 is skipped, A2:A3 is the uint64_t address. */ - TCGReg a2 = MIPS_BE ? l->addrhi_reg : l->addrlo_reg; - TCGReg a3 = MIPS_BE ? l->addrlo_reg : l->addrhi_reg; - - if (a3 != TCG_REG_A2) { - tcg_out_mov(s, TCG_TYPE_I32, TCG_REG_A2, a2); - tcg_out_mov(s, TCG_TYPE_I32, TCG_REG_A3, a3); - } else if (a2 != TCG_REG_A3) { - tcg_out_mov(s, TCG_TYPE_I32, TCG_REG_A3, a3); - tcg_out_mov(s, TCG_TYPE_I32, TCG_REG_A2, a2); - } else { - tcg_out_mov(s, TCG_TYPE_I32, TCG_TMP0, TCG_REG_A2); - tcg_out_mov(s, TCG_TYPE_I32, TCG_REG_A2, TCG_REG_A3); - tcg_out_mov(s, TCG_TYPE_I32, TCG_REG_A3, TCG_TMP0); - } - } else { - tcg_out_mov(s, TCG_TYPE_TL, TCG_REG_A1, l->addrlo_reg); - } - tcg_out_mov(s, TCG_TYPE_PTR, TCG_REG_A0, TCG_AREG0); - - /* - * Tail call to the helper, with the return address back inline. - * We have arrived here via BNEL, so $31 is already set. - */ - target = (l->is_ld ? helper_unaligned_ld : helper_unaligned_st); - tcg_out_call_int(s, target, true); - return true; -} - -static bool tcg_out_qemu_ld_slow_path(TCGContext *s, TCGLabelQemuLdst *l) -{ - return tcg_out_fail_alignment(s, l); -} - -static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *l) -{ - return tcg_out_fail_alignment(s, l); -} -#endif /* SOFTMMU */ - typedef struct { TCGReg base; MemOp align; From patchwork Mon May 15 14:32:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681941 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1149550wrr; Mon, 15 May 2023 07:37:34 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7rvOxHkapbswLl7wHPCeFxHNP3Prz1/HacchCqZVoLjoozUxRS+7e2JtPuOjxccTPdqI/G X-Received: by 2002:a1c:f714:0:b0:3f5:e29:bcde with SMTP id v20-20020a1cf714000000b003f50e29bcdemr1552202wmh.17.1684161454179; Mon, 15 May 2023 07:37:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161454; cv=none; d=google.com; s=arc-20160816; b=1KGuJ8TS1zkRgq+kAn+s81SdGu7msUaKzyeTSDYc3/E8sGtb/Q6zOf6iyn+P6NZGVH 9a+0qiddiH5plHLcWcJpZ/PgOzC7NivG/2uQ8V+XMScr87ogK1gcHGdMpJPv1tsr89me n9r1Qpp2M49TvYVBbNgz1rHPZD69w0r6JnCrSvhQo1PEblrJUb3ecSW8OrZaaY1k7dxl yp0J3/797jacaudK8LwRrj1S/qNCNUb1We7tLyA30NcPdR9d+BLXL/3aau/nyz7TnGiK V6lvQpnRZWmgZdUhsGReX5UEuNfBdBff3YQuyr1r3HKI6box9CSdDunu6uRWu3jrAbcE EeZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=PEsWAFpnCMYNqvb5yhVEEwYFWaIpTnBv29asu3C8fXA=; b=h5kwH7Z7ugrA0W0IInFKkWLgwf1IQy+NBvuFn4434CSdDW9IykAOt8oS4kChdyVX5u IHgieA9C4h5hUAB4K+3aR3fYULiaQhSlp8JmJc4tKUOYiG7agOiNhO7faU/MOD9Ziip6 fYZKdbn+q1L177HF+HYPYmKuOZPdD8P/at2FQF16H5jh/Xfmv/iFD09E+i05Okpi38P3 soHcnn6vBzuNqt42cLjZzBvmMIg9Lo/SWnJXT1w3bvlfB9d1tk4+tHVQkUghdTIU3wQL QuVriFBDgOHkk0rDCcqvdaJFpPZBH6mO/lIyXHaDEiTT93SKqx3HDh+wCux0AJvyX+P5 +WAA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=viSPRN0b; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id j12-20020adff54c000000b003062639dc86si15577088wrp.650.2023.05.15.07.37.34 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:37:34 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=viSPRN0b; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIk-000132-Gu; Mon, 15 May 2023 10:35:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHc-00077s-RN for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:47 -0400 Received: from mail-pl1-x62b.google.com ([2607:f8b0:4864:20::62b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZHY-0004zZ-5Q for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:39 -0400 Received: by mail-pl1-x62b.google.com with SMTP id d9443c01a7336-1aad55244b7so97900595ad.2 for ; Mon, 15 May 2023 07:33:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161215; x=1686753215; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PEsWAFpnCMYNqvb5yhVEEwYFWaIpTnBv29asu3C8fXA=; b=viSPRN0bskYGKCe2vB2bTfhdcgLJEJpOZaEnEi5JqLgZuweOECFo2l5Dos3y86egFz PBDVbiFnFyNvJycmieq330p4un91Rmk52Zwl+zTTx3qASF0xwfy8shhNsDpaKFhCSeWm CIS0WiCEQQZuD8xwTFRWpsT569BTFPSgzYYRS0ENkDz7xmSbPewN+G32e5tvfdrADV6w hr5q3qQ/lLpM9IuRQ7GbsD4MOP5Wb1JpbxgBnd8zCCR3BuyFqd7+M0QhoMMAhm/n2psF edJeb/zUCFrW9G18TYBUEMkZFG0kNBDxUvkUVGQ4L0A0bV0F/ddjYA4N9dPiBzrfp29Y VWBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161215; x=1686753215; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PEsWAFpnCMYNqvb5yhVEEwYFWaIpTnBv29asu3C8fXA=; b=RqYWBtJvqq+JmfYgfeGWevg/EJESRqfBFKu56x6bvVI05BO/kWbU0XEzFaCnh/VKJO b6V/HJY0rFdi8uj4Arp0868PAEFyYPCvcyKRtwigGgq8HHGojgsTVYpXrEzOrromk9g5 CUA2P7CaRh5jqRZgB3vMt5nJ31xIOwLLxKOOuT+FeEorNZf0YPB0G7M3Wy2Cuhr6r5uL hvva1zDQuGCD8ObBOeUKP+I2KZg/vUs2kWatx2uvYWWwy21kyEvbujhLi8mpPcvIZ6rW l7wQYFp5VMQUhvWUKfIE0dzAMnRvEheOdAkM/quTrYxD+wBAfZ4RfZFg0UQC6QeaaXkC uAyA== X-Gm-Message-State: AC+VfDyIhJRlmXR5ejjGw3G6/X/oxRDJ39uQ5sLNDDof1T0hxbVwqbAU 4LIh4atf3Wx1u1gLV48e4r7g6wjpqP8eOLIybTc= X-Received: by 2002:a17:903:1205:b0:1ac:6d4c:c26a with SMTP id l5-20020a170903120500b001ac6d4cc26amr35717447plh.14.1684161214925; Mon, 15 May 2023 07:33:34 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:34 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 23/54] tcg/s390x: Use full load/store helpers in user-only mode Date: Mon, 15 May 2023 07:32:42 -0700 Message-Id: <20230515143313.734053-24-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62b; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Instead of using helper_unaligned_{ld,st}, use the full load/store helpers. This will allow the fast path to increase alignment to implement atomicity while not immediately raising an alignment exception. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/s390x/tcg-target.c.inc | 29 ----------------------------- 1 file changed, 29 deletions(-) diff --git a/tcg/s390x/tcg-target.c.inc b/tcg/s390x/tcg-target.c.inc index 968977be98..de8aed5f77 100644 --- a/tcg/s390x/tcg-target.c.inc +++ b/tcg/s390x/tcg-target.c.inc @@ -1679,7 +1679,6 @@ static void tcg_out_qemu_st_direct(TCGContext *s, MemOp opc, TCGReg data, } } -#if defined(CONFIG_SOFTMMU) static const TCGLdstHelperParam ldst_helper_param = { .ntmp = 1, .tmp = { TCG_TMP0 } }; @@ -1716,34 +1715,6 @@ static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *lb) tgen_gotoi(s, S390_CC_ALWAYS, lb->raddr); return true; } -#else -static bool tcg_out_fail_alignment(TCGContext *s, TCGLabelQemuLdst *l) -{ - if (!patch_reloc(l->label_ptr[0], R_390_PC16DBL, - (intptr_t)tcg_splitwx_to_rx(s->code_ptr), 2)) { - return false; - } - - tcg_out_mov(s, TCG_TYPE_TL, TCG_REG_R3, l->addrlo_reg); - tcg_out_mov(s, TCG_TYPE_PTR, TCG_REG_R2, TCG_AREG0); - - /* "Tail call" to the helper, with the return address back inline. */ - tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_R14, (uintptr_t)l->raddr); - tgen_gotoi(s, S390_CC_ALWAYS, (const void *)(l->is_ld ? helper_unaligned_ld - : helper_unaligned_st)); - return true; -} - -static bool tcg_out_qemu_ld_slow_path(TCGContext *s, TCGLabelQemuLdst *l) -{ - return tcg_out_fail_alignment(s, l); -} - -static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *l) -{ - return tcg_out_fail_alignment(s, l); -} -#endif /* CONFIG_SOFTMMU */ /* * For softmmu, perform the TLB load and compare. From patchwork Mon May 15 14:32:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681984 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1154797wrr; Mon, 15 May 2023 07:49:24 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5FIJp0naY2Zdgh624likq08KHyZqaxpF76W0Vr2tRSy6Z8WKG99jUTvLusHxF1Jm6K29Au X-Received: by 2002:a17:907:3ea8:b0:953:37d9:282f with SMTP id hs40-20020a1709073ea800b0095337d9282fmr28709186ejc.38.1684162164251; Mon, 15 May 2023 07:49:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684162164; cv=none; d=google.com; s=arc-20160816; b=jIAQqxYByrnXreUIfF9Mjz1JLhNHLjZE9VgtJd3ktRmw1QNf8iTOI6uN/jUI6YVogm HKCZeNCNjUWf6/+MHXFN4uTSTmqi0pbZvzOYL1hFuoRQVFpV7YZ/a/s/L12ZbxRUGD1N 2d65O1vyWm3bqYa9dSI8xO7MzakxwpVtx1ZlMFHPOpQcINJJxFz5BTKYuS0vNcbtcGTn BUxIpfJKzOVSbEgy/oIQQpLe9C4/f6tQfeb3TaAIQ3JFi9VuheRTBQuawjimV4wmPmTB qhSOe70jmQIr6R8oNdfESPCr8le1jJacykGBkyPF6u8FxDF2EgmV0WjklMn3r5DiXcke Vmjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=YyONh+q6+JPHpqSgz1eedz8qFY8evuyzGRq8DlknugQ=; b=dey/X3Hr32vvbaI0FpyMzQ8XIoaMWMTKKDQPJE/hocdvN3KLaxJpR/Ym5EESq7g2T6 KRyFPLOxHvg5L2FQzhjKMAccnIpxSRFWiK6cKxDuY5KIzJ58mDhW50L0cBCTVbm2ToYS AYrDHeY+Ylxow+JaOV2CPT1VX0E35+5X9Q+vONhvyKn4AljpU4G0NoG7LCVtwminwWLM 7vo32ekljRRLFX2R3/ijwzBFxDqZioEgRoXfKTAiiXI5eKEdPE08qqVsm4FmTC9HIsDA pw3ruu28ws0RFHvwatN/jkK10Re9VTUkfiwwqRmc28pcEAwT5fk3jay/H/C3x2KJcRJE DYkg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=O4YoGg8g; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id gv57-20020a1709072bf900b0096653f7a2f4si9341842ejc.543.2023.05.15.07.49.24 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:49:24 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=O4YoGg8g; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIv-0001z6-P4; Mon, 15 May 2023 10:36:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHc-00074p-Fd for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:47 -0400 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZHV-000503-6U for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:37 -0400 Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-1aaea43def7so88183165ad.2 for ; Mon, 15 May 2023 07:33:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161215; x=1686753215; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YyONh+q6+JPHpqSgz1eedz8qFY8evuyzGRq8DlknugQ=; b=O4YoGg8gOumIMlcnZX1ADy7TAwBiDuVYUnn1MrsPTWuNxoIL1rbCQN+RMBlgv57fZm kg0ALBvBxg4v6UQjbJ10IwRYzJPwQU6i40FuU+xs7sNiuvBhaNhWBUNUvNea0kM92zZt vgyrCji5j7oP/WleZFrNZNV8mWyqJ2VH3hRYWpn9qAO+abRwvPHPYqvi0kX5vSv3Fb/J LJ5MkIwVCfCSy5k2m1Hb+PVStblkBeLn5E3bPcp6NVoPH+8Pi5iCz49T2TDr4PihSBl6 pIQr07eC917PIYVraRuukBJoQHm5KibCkHBqTfK6w1nQUZepEvnZ+E+X3Uq607SeVgSz ou6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161215; x=1686753215; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YyONh+q6+JPHpqSgz1eedz8qFY8evuyzGRq8DlknugQ=; b=kUniqKEpo3qZv6t4q5k85KrzIPMeh08sSSFBKKW/CvgfMPoM5RyKxy3iQKNwDk018/ Kfmq+D5P0sGNGh3pbcbk3HE6HhbaeGRU0fH9p5mZjHqjikTY73gjqYvyeD0bLHoUU72D kYpDQQf6xXkfhpngSxslb8KmEYp7zgO9fkjZ+zyW2nJTtNDOAk5GBxkyzhsleueYYJ0X qwljCAelOJSWblOgNgm0K5pTen/CpCVPOvg7v1QJd9mwptGEl1StJgG8D/dne5mkFTwF VU5FoyinfBweC76efIPVDe5XeMhrTQ0yGrzTs/QX9Djz5UhXhkH2d/GfZTZgnzkVVywj vCEQ== X-Gm-Message-State: AC+VfDzGEwYYuo86klnTAUtMOvCdldLERWlmGHLfXX8iKOxg0UpuzkF4 ikIf00rcVBNeU85VJ4K9ExkowhS1t8JP6ytd84o= X-Received: by 2002:a17:903:552:b0:1ae:2013:4bc8 with SMTP id jo18-20020a170903055200b001ae20134bc8mr2546582plb.18.1684161215696; Mon, 15 May 2023 07:33:35 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:35 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 24/54] tcg/sparc64: Allocate %g2 as a third temporary Date: Mon, 15 May 2023 07:32:43 -0700 Message-Id: <20230515143313.734053-25-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::635; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x635.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- tcg/sparc64/tcg-target.c.inc | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc index e997db2645..64464ab363 100644 --- a/tcg/sparc64/tcg-target.c.inc +++ b/tcg/sparc64/tcg-target.c.inc @@ -83,9 +83,10 @@ static const char * const tcg_target_reg_names[TCG_TARGET_NB_REGS] = { #define ALL_GENERAL_REGS MAKE_64BIT_MASK(0, 32) #define ALL_QLDST_REGS (ALL_GENERAL_REGS & ~SOFTMMU_RESERVE_REGS) -/* Define some temporary registers. T2 is used for constant generation. */ +/* Define some temporary registers. T3 is used for constant generation. */ #define TCG_REG_T1 TCG_REG_G1 -#define TCG_REG_T2 TCG_REG_O7 +#define TCG_REG_T2 TCG_REG_G2 +#define TCG_REG_T3 TCG_REG_O7 #ifndef CONFIG_SOFTMMU # define TCG_GUEST_BASE_REG TCG_REG_I5 @@ -110,7 +111,6 @@ static const int tcg_target_reg_alloc_order[] = { TCG_REG_I4, TCG_REG_I5, - TCG_REG_G2, TCG_REG_G3, TCG_REG_G4, TCG_REG_G5, @@ -492,8 +492,8 @@ static void tcg_out_movi_int(TCGContext *s, TCGType type, TCGReg ret, static void tcg_out_movi(TCGContext *s, TCGType type, TCGReg ret, tcg_target_long arg) { - tcg_debug_assert(ret != TCG_REG_T2); - tcg_out_movi_int(s, type, ret, arg, false, TCG_REG_T2); + tcg_debug_assert(ret != TCG_REG_T3); + tcg_out_movi_int(s, type, ret, arg, false, TCG_REG_T3); } static void tcg_out_ext8s(TCGContext *s, TCGType type, TCGReg rd, TCGReg rs) @@ -885,10 +885,8 @@ static void tcg_out_jmpl_const(TCGContext *s, const tcg_insn_unit *dest, { uintptr_t desti = (uintptr_t)dest; - /* Be careful not to clobber %o7 for a tail call. */ tcg_out_movi_int(s, TCG_TYPE_PTR, TCG_REG_T1, - desti & ~0xfff, in_prologue, - tail_call ? TCG_REG_G2 : TCG_REG_O7); + desti & ~0xfff, in_prologue, TCG_REG_T2); tcg_out_arithi(s, tail_call ? TCG_REG_G0 : TCG_REG_O7, TCG_REG_T1, desti & 0xfff, JMPL); } @@ -1856,6 +1854,7 @@ static void tcg_target_init(TCGContext *s) tcg_regset_set_reg(s->reserved_regs, TCG_REG_O6); /* stack pointer */ tcg_regset_set_reg(s->reserved_regs, TCG_REG_T1); /* for internal use */ tcg_regset_set_reg(s->reserved_regs, TCG_REG_T2); /* for internal use */ + tcg_regset_set_reg(s->reserved_regs, TCG_REG_T3); /* for internal use */ } #define ELF_HOST_MACHINE EM_SPARCV9 From patchwork Mon May 15 14:32:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681969 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1152741wrr; Mon, 15 May 2023 07:44:31 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7UoLOP/DZXKd2w/e8eknCWB5dSHo3qkuyzGJJG3HXFYQ2+5s2jPXVm8jIYyA/lnekkfrwW X-Received: by 2002:a5d:4c8c:0:b0:2f5:3dfd:f4d2 with SMTP id z12-20020a5d4c8c000000b002f53dfdf4d2mr24833327wrs.64.1684161871143; Mon, 15 May 2023 07:44:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161871; cv=none; d=google.com; s=arc-20160816; b=hfpMgbp2IepTOfHLHS42qzPt4W42iXFeyGXlWbr9Ix0Qpd4AMxTOG9+WU4/kaxwTwy +rHvd92f8lGXzmAzkvECX0jbfZ1U8YwzE7kdu7wGPZ2uG01x6lQtktyXN5sYUpogu401 rqKdPHJ1PHi6FpQkCfEsXH2/SVfXy3nUhRqNnwzlfz9UQwsohyoTcCwHIfEmE2+TaVHe ktDKzyGNL4NgEjfs8VHus/K0EQ0Q/steO44bvnX9wMgn2QzTDp82RszF7EPLyUncIkd4 J694d35op1wzsP8htMn4qZ2fxulZdRgpRQeoBpYM+V3786MR1yTi+vXVOwUgvu1S9JzO 4dtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=8/oWD1U747odSVFFBC9M3Sdo3AJc8ITxvfJIeQOUz74=; b=awDYMj+dLjqWvykTqe0Y2W1lcugkjO/qXrf/8qWDOlC+jm9mEhAPMYlCJA0NvzBGyp hdonmgh8pGr/An/W7Pb2BbiG3jhe4UGmBxLwwCprf7vcx42OsUev12gIVx66CfWv1iOX 2XmTZ2A+2SLEuI1MSyn02lD9Q09s+HU+uc2Zm32heial3Ie7pckM9W+VP5hjkmxkUyeo AIWPpGAWib2exYorEPFFg8vZLEwIfiwDZFkxgzyuT8c315fhQilaOn99kqpOhBiiBefM 1DSajAse6b8L8UKkhxpZ6om+cPj7rq3a8DVBd3mYzULmECP9uwDIafQ3U3ZAI4hAwk99 jtlQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="Eln7J/z7"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id h14-20020a5d688e000000b002f9424dfacfsi13736661wru.803.2023.05.15.07.44.30 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:44:31 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="Eln7J/z7"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZJ4-0002OO-6S; Mon, 15 May 2023 10:36:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHh-00079j-1S for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:47 -0400 Received: from mail-pl1-x62a.google.com ([2607:f8b0:4864:20::62a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZHZ-00050I-Lj for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:42 -0400 Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-1aad5245571so88102875ad.1 for ; Mon, 15 May 2023 07:33:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161216; x=1686753216; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8/oWD1U747odSVFFBC9M3Sdo3AJc8ITxvfJIeQOUz74=; b=Eln7J/z7uhECyw02TtVZ7H6I39wO9KqkRVEtDU5gCVV/3jjVDt8rrjLSSBPRF/MH4k yDZvUvHVdrve854lhDDkvbgkwOQ/Nr5cc6K9dk+3rrmJbHsk41Eea00a2kUU3EAdgp2K BYXiOaecJOEyeaX6/qTPVl9MWzT6d7jFwBxpYGxnPMKoUPykASrr5Emqe0u0ySNXkog8 EJOLSVE7hT5/9UJt4IKn+W1yQas2XbSq8V2rJ5hsEVYSlfoJILKN3Lx7LLZJVTUR+ZOy dyu+soMKykN/vncGPGkVfI+T9uxvsAbOOVZ71BVznP7922gztE615Yf5VZOfYgkn//x8 O4BQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161216; x=1686753216; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8/oWD1U747odSVFFBC9M3Sdo3AJc8ITxvfJIeQOUz74=; b=PkSmTNnQ7T6jFwd7D9LYlck9djsYNws2sw/KNwPr/NdXtS6lV5GbRNJhzTZxTqccpw A6JpbMhP8ntD35QCfiLvnFDBkoo9YgNXQvZXBOfpBl/b1S3LL2TIWz2VhymB/uuBcwRU iIFbGQhbFgoEkZ6vbgmYuvxaFvs2s7k4xwbHLqjdv3ze9tYgPUtxHjdyP1EQysapKuTJ zM+0uUNZuicWbRvXbxQkDcXowZp0/FISHMCRZDQ4Hkv1A6J8Khb2qLnaI9GzKQAR1/ql 5xrZ+sbbtYOB98upij4K5DOLx2Do6sr46RU5emfsSW7/tkhOSr6xqZbz4vjY0NesDbWB rSRQ== X-Gm-Message-State: AC+VfDwSfDQyvH5pQu0iL9TcNGdQHt1CUPNp9attA8Pbh9+JKUW3HI/V /YdR8yWHGUOLEMOIfOX94AaxjvmEHf3DzSHINys= X-Received: by 2002:a17:903:22cd:b0:1aa:ee36:4095 with SMTP id y13-20020a17090322cd00b001aaee364095mr43063462plg.43.1684161216423; Mon, 15 May 2023 07:33:36 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:36 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 25/54] tcg/sparc64: Rename tcg_out_movi_imm13 to tcg_out_movi_s13 Date: Mon, 15 May 2023 07:32:44 -0700 Message-Id: <20230515143313.734053-26-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62a; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Emphasize that the constant is signed. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- tcg/sparc64/tcg-target.c.inc | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc index 64464ab363..15d6a9fd73 100644 --- a/tcg/sparc64/tcg-target.c.inc +++ b/tcg/sparc64/tcg-target.c.inc @@ -399,7 +399,8 @@ static void tcg_out_sethi(TCGContext *s, TCGReg ret, uint32_t arg) tcg_out32(s, SETHI | INSN_RD(ret) | ((arg & 0xfffffc00) >> 10)); } -static void tcg_out_movi_imm13(TCGContext *s, TCGReg ret, int32_t arg) +/* A 13-bit constant sign-extended to 64 bits. */ +static void tcg_out_movi_s13(TCGContext *s, TCGReg ret, int32_t arg) { tcg_out_arithi(s, ret, TCG_REG_G0, arg, ARITH_OR); } @@ -408,7 +409,7 @@ static void tcg_out_movi_imm32(TCGContext *s, TCGReg ret, int32_t arg) { if (check_fit_i32(arg, 13)) { /* A 13-bit constant sign-extended to 64-bits. */ - tcg_out_movi_imm13(s, ret, arg); + tcg_out_movi_s13(s, ret, arg); } else { /* A 32-bit constant zero-extended to 64 bits. */ tcg_out_sethi(s, ret, arg); @@ -433,7 +434,7 @@ static void tcg_out_movi_int(TCGContext *s, TCGType type, TCGReg ret, /* A 13-bit constant sign-extended to 64-bits. */ if (check_fit_tl(arg, 13)) { - tcg_out_movi_imm13(s, ret, arg); + tcg_out_movi_s13(s, ret, arg); return; } @@ -767,7 +768,7 @@ static void tcg_out_setcond_i32(TCGContext *s, TCGCond cond, TCGReg ret, default: tcg_out_cmp(s, c1, c2, c2const); - tcg_out_movi_imm13(s, ret, 0); + tcg_out_movi_s13(s, ret, 0); tcg_out_movcc(s, cond, MOVCC_ICC, ret, 1, 1); return; } @@ -803,11 +804,11 @@ static void tcg_out_setcond_i64(TCGContext *s, TCGCond cond, TCGReg ret, /* For 64-bit signed comparisons vs zero, we can avoid the compare if the input does not overlap the output. */ if (c2 == 0 && !is_unsigned_cond(cond) && c1 != ret) { - tcg_out_movi_imm13(s, ret, 0); + tcg_out_movi_s13(s, ret, 0); tcg_out_movr(s, cond, ret, c1, 1, 1); } else { tcg_out_cmp(s, c1, c2, c2const); - tcg_out_movi_imm13(s, ret, 0); + tcg_out_movi_s13(s, ret, 0); tcg_out_movcc(s, cond, MOVCC_XCC, ret, 1, 1); } } @@ -844,7 +845,7 @@ static void tcg_out_addsub2_i64(TCGContext *s, TCGReg rl, TCGReg rh, if (use_vis3_instructions && !is_sub) { /* Note that ADDXC doesn't accept immediates. */ if (bhconst && bh != 0) { - tcg_out_movi_imm13(s, TCG_REG_T2, bh); + tcg_out_movi_s13(s, TCG_REG_T2, bh); bh = TCG_REG_T2; } tcg_out_arith(s, rh, ah, bh, ARITH_ADDXC); @@ -866,7 +867,7 @@ static void tcg_out_addsub2_i64(TCGContext *s, TCGReg rl, TCGReg rh, * so the adjustment fits 12 bits. */ if (bhconst) { - tcg_out_movi_imm13(s, TCG_REG_T2, bh + (is_sub ? -1 : 1)); + tcg_out_movi_s13(s, TCG_REG_T2, bh + (is_sub ? -1 : 1)); } else { tcg_out_arithi(s, TCG_REG_T2, bh, 1, is_sub ? ARITH_SUB : ARITH_ADD); @@ -1036,7 +1037,7 @@ static void tcg_target_qemu_prologue(TCGContext *s) tcg_code_gen_epilogue = tcg_splitwx_to_rx(s->code_ptr); tcg_out_arithi(s, TCG_REG_G0, TCG_REG_I7, 8, RETURN); /* delay slot */ - tcg_out_movi_imm13(s, TCG_REG_O0, 0); + tcg_out_movi_s13(s, TCG_REG_O0, 0); build_trampolines(s); } @@ -1430,7 +1431,7 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0) { if (check_fit_ptr(a0, 13)) { tcg_out_arithi(s, TCG_REG_G0, TCG_REG_I7, 8, RETURN); - tcg_out_movi_imm13(s, TCG_REG_O0, a0); + tcg_out_movi_s13(s, TCG_REG_O0, a0); return; } else { intptr_t tb_diff = tcg_tbrel_diff(s, (void *)a0); From patchwork Mon May 15 14:32:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681986 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1154873wrr; Mon, 15 May 2023 07:49:32 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6riKVIXAEBXhs5YmWIzDh1V7aOmW3AE6K8l7QAyKlacwORd+meZqwsAWRRrRAB1mEKfZ0q X-Received: by 2002:a17:907:9602:b0:965:77d2:7802 with SMTP id gb2-20020a170907960200b0096577d27802mr33517661ejc.0.1684162172330; Mon, 15 May 2023 07:49:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684162172; cv=none; d=google.com; s=arc-20160816; b=QZVcCtHsDcN6t9CIZtAvbA1WLOrjnNVyVEXKLd4DaMNVitnustk6TbK0Nz9jhPowfH 52u7QMSjGrkGY48bIQ5TVMXxSO7/OwXK+BaXG+9eQ6Pe64Hi2xZgZlq9aPrPyJMhG9aD VkguoEhoXcYTgce5VXqFl3/ZnW27WmMad+qsk23suTcHgvviFNt2mBFuVXT7XsJSOR5v ZzUkIBq5toSyAwqwKquBVU9Bye1IweqUD9ki4sLgwicKNxwwqnxRFvTqsnWx7QoYqnyC uPRBIVmVdKwvmh2AwOaOn/AU+qayBLrFCpTOGtOZ5ufvamGrOtxaqapdWF0YEK7Xjnw+ vrbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=t1hIIKUo7PoLuxvqXryvPnYOSzjxwn2GYHszm/BGZZQ=; b=xsVVgKCybaNWMSoYMbkA9wcE3E9meyCQNgFZ4hQhL2qQ4of2KGg71fi/so8Hd49OqL EJga+/MVII3VPB5wAQmJsGV/rYc5jFL4ffCF2MCrW8A36z1nbYtLmlOQ8MT1vjMCWztv s+UMCafhUvcSBCkbumAuHvLxDQU1BssLQmFpWw5nQ2eimtzf79Xard0MznpdUN/3Pne2 LCdCJr2kZzZhyV01ljVb+5OiL9J+NwN7Zl6IEzB7Zz0U0E79wJQ15Whj+MEV16CQnGMP X4TKFUIqDfQoY8EVADi1Sk6KTPHjgITkCM8btNdGEMu+seOZgDs+qgCih5a7+dwYETDZ tjvw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=wx1aOCcM; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id fp1-20020a1709069e0100b0094f226d0073si12248549ejc.514.2023.05.15.07.49.32 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:49:32 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=wx1aOCcM; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZJP-000315-Ll; Mon, 15 May 2023 10:36:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHg-00079f-WA for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:47 -0400 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZHa-00050a-Fg for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:43 -0400 Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-1aaf21bb42bso90759715ad.2 for ; Mon, 15 May 2023 07:33:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161217; x=1686753217; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=t1hIIKUo7PoLuxvqXryvPnYOSzjxwn2GYHszm/BGZZQ=; b=wx1aOCcMxrdRBcClXzU/JVDXzLeYSFDD/d4zTHM1y1SWA25cLtkLgG7hS2iIvfR3tL TpxtXw4huShC+24M2GF0TS7KLUm4+7ZX4NeJEKhmGH7dypvjg6Gy+cep6BBIQqc+kTti swexNPwxc72S3x2oZiVEhwgtsRBdOAU8gTwh4wpv0MuO2BOqh9bGJAZylQBOi9gKtKf1 qzSs0LxI1+FsctKB3EFn23Ig0s59nk0eZFEQnr349kGKetPEAVIfxhicxL1/reVglkoa 1JOpZm45Gdc8ShRaT3CErkqgNxgs3Jv8Ex/y/9DTcKolvv/S2MONqAj/pA5IsTVK2gqk duzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161217; x=1686753217; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t1hIIKUo7PoLuxvqXryvPnYOSzjxwn2GYHszm/BGZZQ=; b=BcUHG0QvgwWDkafl4rYeFkmK6Q/ojuueucevdpOmbSyFbna6eElGM63Gmv9apUNSly 9TYNzUYfgd94AXTuUgvwI9+LsTu0bE+TvAttLbeKgHVfkn+zE7P9y55wi1pg0k3tjy5Q J/2KwgjuDLqbOf+fvywEpVxnEkNV9x0L2b0F367Gy5BdqBt6tooS3ObGBbFbhyYHqUE2 o6oJr5BMpZ1MWnq9aopaOQMRWkVWFWNYQkAYb6/0FirzXTFLoBolEZ9OqNiAWZgMYyZi +eJHfqd7N5fr5iUbQRGB/fEX3duK0KhlPAV33u0gWqmP+IzLXUhxPZAcVtVHMDS5f1AF CW2g== X-Gm-Message-State: AC+VfDx/VmIlpHrlYVM982JMHiBUovf9U+vkEASToZkKKhwerrZX+Q1s m8KIgh7znNmpHb3pwdI13fc+EYoBgSbj4FemRF4= X-Received: by 2002:a17:903:48e:b0:1a2:a904:c438 with SMTP id jj14-20020a170903048e00b001a2a904c438mr35834889plb.58.1684161217185; Mon, 15 May 2023 07:33:37 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:36 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 26/54] target/sparc64: Remove tcg_out_movi_s13 case from tcg_out_movi_imm32 Date: Mon, 15 May 2023 07:32:45 -0700 Message-Id: <20230515143313.734053-27-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::635; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x635.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Shuffle the order in tcg_out_movi_int to check s13 first, and drop this check from tcg_out_movi_imm32. This might make the sequence for in_prologue larger, but not with worrying about. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- tcg/sparc64/tcg-target.c.inc | 25 ++++++++++--------------- 1 file changed, 10 insertions(+), 15 deletions(-) diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc index 15d6a9fd73..2689599fd6 100644 --- a/tcg/sparc64/tcg-target.c.inc +++ b/tcg/sparc64/tcg-target.c.inc @@ -407,15 +407,10 @@ static void tcg_out_movi_s13(TCGContext *s, TCGReg ret, int32_t arg) static void tcg_out_movi_imm32(TCGContext *s, TCGReg ret, int32_t arg) { - if (check_fit_i32(arg, 13)) { - /* A 13-bit constant sign-extended to 64-bits. */ - tcg_out_movi_s13(s, ret, arg); - } else { - /* A 32-bit constant zero-extended to 64 bits. */ - tcg_out_sethi(s, ret, arg); - if (arg & 0x3ff) { - tcg_out_arithi(s, ret, ret, arg & 0x3ff, ARITH_OR); - } + /* A 32-bit constant zero-extended to 64 bits. */ + tcg_out_sethi(s, ret, arg); + if (arg & 0x3ff) { + tcg_out_arithi(s, ret, ret, arg & 0x3ff, ARITH_OR); } } @@ -426,18 +421,18 @@ static void tcg_out_movi_int(TCGContext *s, TCGType type, TCGReg ret, tcg_target_long hi, lo = (int32_t)arg; tcg_target_long test, lsb; - /* A 32-bit constant, or 32-bit zero-extended to 64-bits. */ - if (type == TCG_TYPE_I32 || arg == (uint32_t)arg) { - tcg_out_movi_imm32(s, ret, arg); - return; - } - /* A 13-bit constant sign-extended to 64-bits. */ if (check_fit_tl(arg, 13)) { tcg_out_movi_s13(s, ret, arg); return; } + /* A 32-bit constant, or 32-bit zero-extended to 64-bits. */ + if (type == TCG_TYPE_I32 || arg == (uint32_t)arg) { + tcg_out_movi_imm32(s, ret, arg); + return; + } + /* A 13-bit constant relative to the TB. */ if (!in_prologue) { test = tcg_tbrel_diff(s, (void *)arg); From patchwork Mon May 15 14:32:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681966 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1152444wrr; Mon, 15 May 2023 07:43:55 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ778uvk0x19v29y2B+w/w3wQipFVbBpsZ0vJLXfJWaCcc4sxQbChtNYrYt+yADoV1RlIaxj X-Received: by 2002:a05:6000:14a:b0:307:9592:846d with SMTP id r10-20020a056000014a00b003079592846dmr19334958wrx.14.1684161835525; Mon, 15 May 2023 07:43:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161835; cv=none; d=google.com; s=arc-20160816; b=jISofYA0eOQ4hHp9/Q1u0dEqIirM4Bsnx3gj9QNY5W1JKLqnMY8wNUyu8tUWRn0QUq TDOsR8a7+Z6DInM2zbZNH8jjvdGe91C0y11F3qM2wECgRA3I+1+c8JPeztForP5EW/X7 0e3ZFi0WhxJExYgbUpLdsOm0Oo7IYV2sfVpQeJg0dsEOBBSElg46W0UJYl4xcZ8MgkTQ EQww07f1NwulA+6SvCUc8P8VQfYtybQf3Rwf04+LtsW9+upzv6fDes+MV1FTI8MCcCk+ EwnKKvI5T6iHMXtkHyUK8qf0G41NHTajOxD+WuotD4wWurzdf3P1MfcnCJqEyMiBQDEK X/Aw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=QglhRsdw7AzGzQGFnKhJg0Itmtj7bWZIFp39BEmNv6E=; b=qb5t6PSXC+hl6Ui3lbQneWjUx/8GReZulrEX1f1c2FOI7ncav/u3n76TKDt9uf5O89 URIzdpMqyGXudM3f3IQTZhb5Ps5a4MZVk3NHD47mzWraTrEjt9MtK4PApNztcopChwtD Tvr8GqYVEvTmQTOGKJ4uuervpcStciUnx1KUmTxPVJluGRf+156P0D+8y6q0o3pLJJs9 Y377XTjv3ftfaUFgWHWhHww2QOEVmlOWh2rUuVDCA4u/iGFG1A554dz/He0xpTwHkmbe I1+Sj9tAtHKtU9Q8V+GL0e3R06LsSCMi+xqUMaZ/sI+zI7aCHVds8OOsyQjzr01nlRUh uPvQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="iCbi7E8/"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id p7-20020a5d4e07000000b0030640508244si14824948wrt.676.2023.05.15.07.43.55 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:43:55 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="iCbi7E8/"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIq-0001cH-JW; Mon, 15 May 2023 10:35:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHY-00070x-Ez for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:38 -0400 Received: from mail-pl1-x634.google.com ([2607:f8b0:4864:20::634]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZHS-0004rv-NE for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:33 -0400 Received: by mail-pl1-x634.google.com with SMTP id d9443c01a7336-1aaf91ae451so120245245ad.1 for ; Mon, 15 May 2023 07:33:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161218; x=1686753218; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QglhRsdw7AzGzQGFnKhJg0Itmtj7bWZIFp39BEmNv6E=; b=iCbi7E8/0QUKz7qYqiDmyVA7Td0HW5AX1NoUOvEbosqmjX4VmPwVloTvj+w6QudEiL LCA1Rugz502R6vcEUQ6qfwpzaqLLrXtCr9VqGaHvQGyX3BHoZEOqgj+CZm510NLOZ0+Q 79zIqzHMlB7NZ38c5luZ9eJiSK33mJB4HDgRH/t+/4VT9HxmNK1EgMFfcN4XDVXSCEui PnVcqxhEgvZ2RBqJINllHGxgerl2+nC4rVhNqYrGCTnrmtx3jEP/AcRbcWQPEe+7+Nv9 FwFAsEhl78fOYEq44Nh//X3T9ylnDXn3q/vow+ouqMJw/sgyp1veGhOw0YJ4I/Syua7c pfAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161218; x=1686753218; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QglhRsdw7AzGzQGFnKhJg0Itmtj7bWZIFp39BEmNv6E=; b=al7HomKWGoLfR3U9lpK3BNbzXJRHd5udpVw0J49HZArkMO3esEQkST1F26z3jOPlR2 PnbJwJ0nO4/hj69v7/KiIw3a4el2eHJvsmzPJST8vxR9/woLn5TnJa7HQnDD1f1RIKcy m0bc2gGiW0uupwItf4RQFvIM2XXu9JwVLCzQpTmDSBjKzVZv3tjprJV4fXvdOoVODhU/ I/nbb5YMnYlya3GwuszjCZvKQF8nVhBaAwDUwxEbRlYGc5uvy9HnwSpr/Nla5w3m1NIy kPUBA1KCBn73py1vVRO0NoDnqZCk+vxIJWrnVRr1zRNMVw1U0a2jI3cwpcPUc7vFGGkT ZyRg== X-Gm-Message-State: AC+VfDwe94bVDMSYhKjG3BvUntqp54i2/k+Tm3KXZ3awfRnohTDZMvfo MbcFIK6PYhPQs8VStQCmCY0wX3ClTkVxpgfss2E= X-Received: by 2002:a17:902:ce8c:b0:1ad:f138:b2f6 with SMTP id f12-20020a170902ce8c00b001adf138b2f6mr11645112plg.16.1684161218232; Mon, 15 May 2023 07:33:38 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:37 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 27/54] tcg/sparc64: Rename tcg_out_movi_imm32 to tcg_out_movi_u32 Date: Mon, 15 May 2023 07:32:46 -0700 Message-Id: <20230515143313.734053-28-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::634; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x634.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Emphasize that the constant is unsigned. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- tcg/sparc64/tcg-target.c.inc | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc index 2689599fd6..e244209890 100644 --- a/tcg/sparc64/tcg-target.c.inc +++ b/tcg/sparc64/tcg-target.c.inc @@ -405,9 +405,9 @@ static void tcg_out_movi_s13(TCGContext *s, TCGReg ret, int32_t arg) tcg_out_arithi(s, ret, TCG_REG_G0, arg, ARITH_OR); } -static void tcg_out_movi_imm32(TCGContext *s, TCGReg ret, int32_t arg) +/* A 32-bit constant zero-extended to 64 bits. */ +static void tcg_out_movi_u32(TCGContext *s, TCGReg ret, uint32_t arg) { - /* A 32-bit constant zero-extended to 64 bits. */ tcg_out_sethi(s, ret, arg); if (arg & 0x3ff) { tcg_out_arithi(s, ret, ret, arg & 0x3ff, ARITH_OR); @@ -429,7 +429,7 @@ static void tcg_out_movi_int(TCGContext *s, TCGType type, TCGReg ret, /* A 32-bit constant, or 32-bit zero-extended to 64-bits. */ if (type == TCG_TYPE_I32 || arg == (uint32_t)arg) { - tcg_out_movi_imm32(s, ret, arg); + tcg_out_movi_u32(s, ret, arg); return; } @@ -473,13 +473,13 @@ static void tcg_out_movi_int(TCGContext *s, TCGType type, TCGReg ret, /* A 64-bit constant decomposed into 2 32-bit pieces. */ if (check_fit_i32(lo, 13)) { hi = (arg - lo) >> 32; - tcg_out_movi_imm32(s, ret, hi); + tcg_out_movi_u32(s, ret, hi); tcg_out_arithi(s, ret, ret, 32, SHIFT_SLLX); tcg_out_arithi(s, ret, ret, lo, ARITH_ADD); } else { hi = arg >> 32; - tcg_out_movi_imm32(s, ret, hi); - tcg_out_movi_imm32(s, scratch, lo); + tcg_out_movi_u32(s, ret, hi); + tcg_out_movi_u32(s, scratch, lo); tcg_out_arithi(s, ret, ret, 32, SHIFT_SLLX); tcg_out_arith(s, ret, ret, scratch, ARITH_OR); } From patchwork Mon May 15 14:32:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681985 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1154831wrr; Mon, 15 May 2023 07:49:28 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6iqi/Qva2zFP65PbXXNv3ItxDObB/xEhhV2rKR8P0f31iWJB2AgPERh+ytnY0qb2hQ8Jjc X-Received: by 2002:aa7:c597:0:b0:508:3b23:d84c with SMTP id g23-20020aa7c597000000b005083b23d84cmr29089674edq.1.1684162168600; Mon, 15 May 2023 07:49:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684162168; cv=none; d=google.com; s=arc-20160816; b=EXcz8yxACi+QVBGYKaHVdUw8qqk4JF/x925A7rpz59sURvCp7wDYr20mf/TAcLT9gq y4sE+sOrQHfiqB1YnybN7roGfs6rOAjf5nipjKz/6A275FlXoTXlh0S5KTJNQ2Fy+ENM kFTBrCvdcyQ87R9j50SoVoQL+DIQpHgVH3RaUrnAZ6vbhqXPREvSVD9RdkCCeAY3TuDk vTVWppTW8rVI5lCNJSrqoJwwaHsQZ/bLGRbT29JVIbKFay7WGB/gKVa8j9ugt8VqUxAs 3Jkol16UHvgAbB6EaDarUPjAHHhintexHC5Sn5A3SOpWft5M/wnzn15yfj0p7DlxjkPV akUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=5+dVuSBLTF8ePfEeAWZlWoNLu6H+GdqE/W0YcabdOx4=; b=APe9kerwr9vimCnLYUnWB7WWVXH+H6zmPMM5Jxr2opYwzmkBVmhz3N9Ys2z7vLpG0B ZSiQ+hmlon3ZjH4f3w0xemQhumLhS3vayutaGx1khr7xGOpRkhDvykBdS42N5YTWrtAo 3tSKOY32fZLA//Qn/yp1UqIuIHRrc6CFbhNokzxqO5cKY7rpWdemvrq1KedxvgHCkseV CEwsz2c26XQLbm1hWuG+uwiHV53d8eISGcRg/c5CuNgnEi4pRxTh+5n2us2xTsjP9ppB +8kgZXXhkabrFAQHoXbr/Ecn3XVZqxK1cdiTaRSBYO2yRPZ2SVIrLQ3K0T9LPzjYjf0Q pGeA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=jfKYLz8A; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id w13-20020aa7da4d000000b0050be529bf70si10716402eds.332.2023.05.15.07.49.28 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:49:28 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=jfKYLz8A; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZJv-0005Di-9K; Mon, 15 May 2023 10:37:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHn-0007G4-8C for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:52 -0400 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZHc-000512-8r for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:50 -0400 Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-1ab032d9266so118753005ad.0 for ; Mon, 15 May 2023 07:33:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161219; x=1686753219; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5+dVuSBLTF8ePfEeAWZlWoNLu6H+GdqE/W0YcabdOx4=; b=jfKYLz8AoLK9EiTKyy4pi6rR+Di0inNpR7KTxQM/A6bObNx4+aO2L6A7W8HEBOG7Yl BFtaNOzEGdfRE3pbYE/nOEIUnxmYM4LnoxCd7poihr9gfPBBumTvGHVEuoINSvMAbtS1 ITJScN9xd/nfGh62eGb4YnjZaZZkJnMqlRPD+v9KLEnLU5GECnpYMNtLJSv+oLOTtm9O DKPXHwhOoIyWLv+WcoP+Q64o2fO7c/sDpxlus17T3Kk0ntom0Yqwr1j/5aQXY3oTwgul TBiG7pCDIwFdcpiECprM87rE2o/IEHmJaTE5I205BVuDz+w+7ipO29lgQwiLecBr3xki 5wfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161219; x=1686753219; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5+dVuSBLTF8ePfEeAWZlWoNLu6H+GdqE/W0YcabdOx4=; b=XfpCXp2+EapzZH9ZGxqWeXt3nlOiFMRpLKdn0pmr4fzs00M6ltyvZOl4e33xJrsmlM PkKDpG1CD74JBcSZsx3wvwIGeLVXqaWeLexccOugsoFCb16hqOJw4QO16e+xq8g/AUT2 5RXoAMyqw49XRofbJgBnvaT+q9/nXG969fSQNLwYo+mdVAo4gG2tQ8+5jLYxprHIhsYf mv0Vg7R+lxERTr8LOKi7u02IhpefneF9JS3fTvTDmawElezinnKK5hoJfdVe0I0EtJml TItfbLmpsr0FykNi1o8nBkoISda+OynA7FEqX1vLTDsH71sVrjwuBfGgcv2aqGajsx5q MsJA== X-Gm-Message-State: AC+VfDzdY082duilsrlm7tjI4+haIPxlBU8jeBHEmjOWrfGZKLmq/VBU rIttq3WWet46izgV5kNn9Typ1Az0Ob1NDfnZanA= X-Received: by 2002:a17:902:d4c9:b0:1ac:637d:5888 with SMTP id o9-20020a170902d4c900b001ac637d5888mr35448064plg.43.1684161219063; Mon, 15 May 2023 07:33:39 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:38 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 28/54] tcg/sparc64: Split out tcg_out_movi_s32 Date: Mon, 15 May 2023 07:32:47 -0700 Message-Id: <20230515143313.734053-29-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::635; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x635.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/sparc64/tcg-target.c.inc | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc index e244209890..4375a06377 100644 --- a/tcg/sparc64/tcg-target.c.inc +++ b/tcg/sparc64/tcg-target.c.inc @@ -405,6 +405,13 @@ static void tcg_out_movi_s13(TCGContext *s, TCGReg ret, int32_t arg) tcg_out_arithi(s, ret, TCG_REG_G0, arg, ARITH_OR); } +/* A 32-bit constant sign-extended to 64 bits. */ +static void tcg_out_movi_s32(TCGContext *s, TCGReg ret, int32_t arg) +{ + tcg_out_sethi(s, ret, ~arg); + tcg_out_arithi(s, ret, ret, (arg & 0x3ff) | -0x400, ARITH_XOR); +} + /* A 32-bit constant zero-extended to 64 bits. */ static void tcg_out_movi_u32(TCGContext *s, TCGReg ret, uint32_t arg) { @@ -444,8 +451,7 @@ static void tcg_out_movi_int(TCGContext *s, TCGType type, TCGReg ret, /* A 32-bit constant sign-extended to 64-bits. */ if (arg == lo) { - tcg_out_sethi(s, ret, ~arg); - tcg_out_arithi(s, ret, ret, (arg & 0x3ff) | -0x400, ARITH_XOR); + tcg_out_movi_s32(s, ret, arg); return; } From patchwork Mon May 15 14:32:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681949 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1150043wrr; Mon, 15 May 2023 07:38:35 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6M+/wyTT88j3Vc6u/VkDfXAJQNlHcigcEJkhoVkTefBO3UEdoS3ADBo0rtOZNeHhqw624p X-Received: by 2002:a5d:5043:0:b0:307:8e50:34c6 with SMTP id h3-20020a5d5043000000b003078e5034c6mr21854723wrt.52.1684161515198; Mon, 15 May 2023 07:38:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161515; cv=none; d=google.com; s=arc-20160816; b=knZVh8o7C0/QzhlmViHfiGyTlNXAil9+4pIawRRB/2+gwXYu70bn41gkAqLbxqo9oa KkQ//RrgBC81mlgfH4DXzlUuTYdHxeuOyckfsDwkQmpPTls8CPXDnNS18kreeffE12s/ X68+BCdSye4sa/MSPXUSI0kfeicBcuh5P9vF4nnE1t5ivvDYFDg0wfEu489UTtGP4uK2 HJQvD5GH82p+LYCaEbEduCrZxltcdWWaKzOvBFrdltSe6TcWlc15mD0W41nzrs+2FtCP zEMKZ1pxOmS9HbFNX7jXowOxM7dqtl3cVMvTfNPfudoQdsoDj+nY1HJ0p/MmHy6CCZ6+ AXiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=8KAB/H13Cm0D6bHkoWgDoGCikvqnWURt7Ewb7NyDsZI=; b=HBOvHZYt/IxTFm3aroOwuFleomPPSswJC+RmE4f4sgjY8a/A2lF4LasV92xTLTnsnG t1Tuzk99cJXfuUHbQ9rQXC5nMnPXBzc3wQp308yu322B2+LyWYLs3rT8e/u+SPG1yOo9 3DbFuI6kH5XjrkVjD0IN3uToScxZQB80n3s9Jk2Mn3gb9u49Qzj9pXS1/5fW0HCyCCi1 nM/I+2pa5mw+toKjnpG/AUOj3ZDJyfzgqQ3bSuFx3qpV9enVgSSOa69GkW+tkhEjuSRq UAtEV7LsrYog38qOf/cTD4S5nRfoxJG0MPmfnqoidIUe6WyNVhQ0kxliZO+NhSrOoHO7 j2tg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=rEZmQYoR; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id f1-20020adfe901000000b003062f8f0616si16249300wrm.909.2023.05.15.07.38.34 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:38:35 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=rEZmQYoR; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZJS-0003JJ-AD; Mon, 15 May 2023 10:36:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHo-0007GA-B1 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:52 -0400 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZHc-00051Q-Hk for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:52 -0400 Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-1ab01bf474aso97654285ad.1 for ; Mon, 15 May 2023 07:33:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161220; x=1686753220; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8KAB/H13Cm0D6bHkoWgDoGCikvqnWURt7Ewb7NyDsZI=; b=rEZmQYoRqR++BT2THJUCg7G6+c79v2D2hxoc43DMCfCaCrfYObp9Xp5VF1SXOfLklf um/c3vzcOYmg4tYcKmJp9hEM4k217xRdpFHfwfDMdxx5rZrkNnNKSly+eiV0M0kNlY2A b9n4OlomfHysCPQjd9u6P6eRYQFXc07JxE8Y604VmwWTABjlThio1Futg5XdM5/iKhii VXruehtDZV9yOR+z+I8VB3cghEaFtsMjexjhk0gKnogOIEinSw+q3yGAiLkRfsLCoCYa EEdl08mn0iowAOFoS3vcn1oTMJt92ptE9VOfefmPZF9IaKESaK04vHM837K8a4jkeDCV lvnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161220; x=1686753220; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8KAB/H13Cm0D6bHkoWgDoGCikvqnWURt7Ewb7NyDsZI=; b=CYpGVxvq3c/7dI5jfk+zVRs4HTQuOXyi1eDZa7gP6RCwExzZKP+wTAIuE3CAMQ5qKF 8N7tesOMFwJQ5w8K6rdbo5qQ6F5H+Fn29NBPfxYg3+ROlMt/81NXNXeZLd/a5qEsn50l oEC34CtlC/j/5I/PSzkQDgzkGCy8NX6I1xxx5IkFpyCMu3vzmm3KiN0xiuRITF7oMb0Y TBjlmEttdBJWWgHTJb75RmgaesnNrWXpdHY6b7b20DlOCUFARMCDvVf0kcXbtHOouJ9+ zwvegO1zRJbqMPciCMofeDzYfcb+h16H7RfplKwOuUz+D3SNwh7+lOhX2WcEmai9tQFR QMUQ== X-Gm-Message-State: AC+VfDztdIanR6yOy+rWBH6X5iWctXIGuj7vGkvWcYZXkPbW0Hz7xLeY PWMF7O6m+JOEPA7sgwGwAG+w6sOBMA5HdL401C8= X-Received: by 2002:a17:902:d2d2:b0:1ac:b4db:6a62 with SMTP id n18-20020a170902d2d200b001acb4db6a62mr23159208plc.65.1684161219819; Mon, 15 May 2023 07:33:39 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:39 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 29/54] tcg/sparc64: Use standard slow path for softmmu Date: Mon, 15 May 2023 07:32:48 -0700 Message-Id: <20230515143313.734053-30-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62e; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Drop the target-specific trampolines for the standard slow path. This lets us use tcg_out_helper_{ld,st}_args, and handles the new atomicity bits within MemOp. At the same time, use the full load/store helpers for user-only mode. Drop inline unaligned access support for user-only mode, as it does not handle atomicity. Use TCG_REG_T[1-3] in the tlb lookup, instead of TCG_REG_O[0-2]. This allows the constraints to be simplified. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- tcg/sparc64/tcg-target-con-set.h | 2 - tcg/sparc64/tcg-target-con-str.h | 1 - tcg/sparc64/tcg-target.h | 1 + tcg/sparc64/tcg-target.c.inc | 610 +++++++++---------------------- 4 files changed, 182 insertions(+), 432 deletions(-) diff --git a/tcg/sparc64/tcg-target-con-set.h b/tcg/sparc64/tcg-target-con-set.h index 31e6fea1fc..434bf25072 100644 --- a/tcg/sparc64/tcg-target-con-set.h +++ b/tcg/sparc64/tcg-target-con-set.h @@ -12,8 +12,6 @@ C_O0_I1(r) C_O0_I2(rZ, r) C_O0_I2(rZ, rJ) -C_O0_I2(sZ, s) -C_O1_I1(r, s) C_O1_I1(r, r) C_O1_I2(r, r, r) C_O1_I2(r, rZ, rJ) diff --git a/tcg/sparc64/tcg-target-con-str.h b/tcg/sparc64/tcg-target-con-str.h index 8f5c7aef97..0577ec4942 100644 --- a/tcg/sparc64/tcg-target-con-str.h +++ b/tcg/sparc64/tcg-target-con-str.h @@ -9,7 +9,6 @@ * REGS(letter, register_mask) */ REGS('r', ALL_GENERAL_REGS) -REGS('s', ALL_QLDST_REGS) /* * Define constraint letters for constants: diff --git a/tcg/sparc64/tcg-target.h b/tcg/sparc64/tcg-target.h index ffe22b1d21..7434cc99d4 100644 --- a/tcg/sparc64/tcg-target.h +++ b/tcg/sparc64/tcg-target.h @@ -155,6 +155,7 @@ extern bool use_vis3_instructions; #define TCG_TARGET_DEFAULT_MO (0) #define TCG_TARGET_HAS_MEMORY_BSWAP 1 +#define TCG_TARGET_NEED_LDST_LABELS #define TCG_TARGET_NEED_POOL_LABELS #endif diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc index 4375a06377..0237188d65 100644 --- a/tcg/sparc64/tcg-target.c.inc +++ b/tcg/sparc64/tcg-target.c.inc @@ -27,6 +27,7 @@ #error "unsupported code generation mode" #endif +#include "../tcg-ldst.c.inc" #include "../tcg-pool.c.inc" #ifdef CONFIG_DEBUG_TCG @@ -70,18 +71,7 @@ static const char * const tcg_target_reg_names[TCG_TARGET_NB_REGS] = { #define TCG_CT_CONST_S13 0x200 #define TCG_CT_CONST_ZERO 0x400 -/* - * For softmmu, we need to avoid conflicts with the first 3 - * argument registers to perform the tlb lookup, and to call - * the helper function. - */ -#ifdef CONFIG_SOFTMMU -#define SOFTMMU_RESERVE_REGS MAKE_64BIT_MASK(TCG_REG_O0, 3) -#else -#define SOFTMMU_RESERVE_REGS 0 -#endif -#define ALL_GENERAL_REGS MAKE_64BIT_MASK(0, 32) -#define ALL_QLDST_REGS (ALL_GENERAL_REGS & ~SOFTMMU_RESERVE_REGS) +#define ALL_GENERAL_REGS MAKE_64BIT_MASK(0, 32) /* Define some temporary registers. T3 is used for constant generation. */ #define TCG_REG_T1 TCG_REG_G1 @@ -918,82 +908,6 @@ static void tcg_out_mb(TCGContext *s, TCGArg a0) tcg_out32(s, MEMBAR | (a0 & TCG_MO_ALL)); } -#ifdef CONFIG_SOFTMMU -static const tcg_insn_unit *qemu_ld_trampoline[MO_SSIZE + 1]; -static const tcg_insn_unit *qemu_st_trampoline[MO_SIZE + 1]; - -static void build_trampolines(TCGContext *s) -{ - int i; - - for (i = 0; i < ARRAY_SIZE(qemu_ld_helpers); ++i) { - if (qemu_ld_helpers[i] == NULL) { - continue; - } - - /* May as well align the trampoline. */ - while ((uintptr_t)s->code_ptr & 15) { - tcg_out_nop(s); - } - qemu_ld_trampoline[i] = tcg_splitwx_to_rx(s->code_ptr); - - /* Set the retaddr operand. */ - tcg_out_mov(s, TCG_TYPE_PTR, TCG_REG_O3, TCG_REG_O7); - /* Tail call. */ - tcg_out_jmpl_const(s, qemu_ld_helpers[i], true, true); - /* delay slot -- set the env argument */ - tcg_out_mov_delay(s, TCG_REG_O0, TCG_AREG0); - } - - for (i = 0; i < ARRAY_SIZE(qemu_st_helpers); ++i) { - if (qemu_st_helpers[i] == NULL) { - continue; - } - - /* May as well align the trampoline. */ - while ((uintptr_t)s->code_ptr & 15) { - tcg_out_nop(s); - } - qemu_st_trampoline[i] = tcg_splitwx_to_rx(s->code_ptr); - - /* Set the retaddr operand. */ - tcg_out_mov(s, TCG_TYPE_PTR, TCG_REG_O4, TCG_REG_O7); - - /* Tail call. */ - tcg_out_jmpl_const(s, qemu_st_helpers[i], true, true); - /* delay slot -- set the env argument */ - tcg_out_mov_delay(s, TCG_REG_O0, TCG_AREG0); - } -} -#else -static const tcg_insn_unit *qemu_unalign_ld_trampoline; -static const tcg_insn_unit *qemu_unalign_st_trampoline; - -static void build_trampolines(TCGContext *s) -{ - for (int ld = 0; ld < 2; ++ld) { - void *helper; - - while ((uintptr_t)s->code_ptr & 15) { - tcg_out_nop(s); - } - - if (ld) { - helper = helper_unaligned_ld; - qemu_unalign_ld_trampoline = tcg_splitwx_to_rx(s->code_ptr); - } else { - helper = helper_unaligned_st; - qemu_unalign_st_trampoline = tcg_splitwx_to_rx(s->code_ptr); - } - - /* Tail call. */ - tcg_out_jmpl_const(s, helper, true, true); - /* delay slot -- set the env argument */ - tcg_out_mov_delay(s, TCG_REG_O0, TCG_AREG0); - } -} -#endif - /* Generate global QEMU prologue and epilogue code */ static void tcg_target_qemu_prologue(TCGContext *s) { @@ -1039,8 +953,6 @@ static void tcg_target_qemu_prologue(TCGContext *s) tcg_out_arithi(s, TCG_REG_G0, TCG_REG_I7, 8, RETURN); /* delay slot */ tcg_out_movi_s13(s, TCG_REG_O0, 0); - - build_trampolines(s); } static void tcg_out_nop_fill(tcg_insn_unit *p, int count) @@ -1051,381 +963,224 @@ static void tcg_out_nop_fill(tcg_insn_unit *p, int count) } } -#if defined(CONFIG_SOFTMMU) +static const TCGLdstHelperParam ldst_helper_param = { + .ntmp = 1, .tmp = { TCG_REG_T1 } +}; -/* We expect to use a 13-bit negative offset from ENV. */ -QEMU_BUILD_BUG_ON(TLB_MASK_TABLE_OFS(0) > 0); -QEMU_BUILD_BUG_ON(TLB_MASK_TABLE_OFS(0) < -(1 << 12)); - -/* Perform the TLB load and compare. - - Inputs: - ADDRLO and ADDRHI contain the possible two parts of the address. - - MEM_INDEX and S_BITS are the memory context and log2 size of the load. - - WHICH is the offset into the CPUTLBEntry structure of the slot to read. - This should be offsetof addr_read or addr_write. - - The result of the TLB comparison is in %[ix]cc. The sanitized address - is in the returned register, maybe %o0. The TLB addend is in %o1. */ - -static TCGReg tcg_out_tlb_load(TCGContext *s, TCGReg addr, int mem_index, - MemOp opc, int which) +static bool tcg_out_qemu_ld_slow_path(TCGContext *s, TCGLabelQemuLdst *lb) { + MemOp opc = get_memop(lb->oi); + MemOp sgn; + + if (!patch_reloc(lb->label_ptr[0], R_SPARC_WDISP19, + (intptr_t)tcg_splitwx_to_rx(s->code_ptr), 0)) { + return false; + } + + /* Use inline tcg_out_ext32s; otherwise let the helper sign-extend. */ + sgn = (opc & MO_SIZE) < MO_32 ? MO_SIGN : 0; + + tcg_out_ld_helper_args(s, lb, &ldst_helper_param); + tcg_out_call(s, qemu_ld_helpers[opc & (MO_SIZE | sgn)], NULL); + tcg_out_ld_helper_ret(s, lb, sgn, &ldst_helper_param); + + tcg_out_bpcc0(s, COND_A, BPCC_A | BPCC_PT, 0); + return patch_reloc(s->code_ptr - 1, R_SPARC_WDISP19, + (intptr_t)lb->raddr, 0); +} + +static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *lb) +{ + MemOp opc = get_memop(lb->oi); + + if (!patch_reloc(lb->label_ptr[0], R_SPARC_WDISP19, + (intptr_t)tcg_splitwx_to_rx(s->code_ptr), 0)) { + return false; + } + + tcg_out_st_helper_args(s, lb, &ldst_helper_param); + tcg_out_call(s, qemu_st_helpers[opc & MO_SIZE], NULL); + + tcg_out_bpcc0(s, COND_A, BPCC_A | BPCC_PT, 0); + return patch_reloc(s->code_ptr - 1, R_SPARC_WDISP19, + (intptr_t)lb->raddr, 0); +} + +typedef struct { + TCGReg base; + TCGReg index; +} HostAddress; + +/* + * For softmmu, perform the TLB load and compare. + * For useronly, perform any required alignment tests. + * In both cases, return a TCGLabelQemuLdst structure if the slow path + * is required and fill in @h with the host address for the fast path. + */ +static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, + TCGReg addr_reg, MemOpIdx oi, + bool is_ld) +{ + TCGLabelQemuLdst *ldst = NULL; + MemOp opc = get_memop(oi); + unsigned a_bits = get_alignment_bits(opc); + unsigned s_bits = opc & MO_SIZE; + unsigned a_mask; + + /* We don't support unaligned accesses. */ + a_bits = MAX(a_bits, s_bits); + a_mask = (1u << a_bits) - 1; + +#ifdef CONFIG_SOFTMMU + int mem_index = get_mmuidx(oi); int fast_off = TLB_MASK_TABLE_OFS(mem_index); int mask_off = fast_off + offsetof(CPUTLBDescFast, mask); int table_off = fast_off + offsetof(CPUTLBDescFast, table); - const TCGReg r0 = TCG_REG_O0; - const TCGReg r1 = TCG_REG_O1; - const TCGReg r2 = TCG_REG_O2; - unsigned s_bits = opc & MO_SIZE; - unsigned a_bits = get_alignment_bits(opc); - tcg_target_long compare_mask; + int cmp_off = is_ld ? offsetof(CPUTLBEntry, addr_read) + : offsetof(CPUTLBEntry, addr_write); + int add_off = offsetof(CPUTLBEntry, addend); + int compare_mask; + int cc; /* Load tlb_mask[mmu_idx] and tlb_table[mmu_idx]. */ - tcg_out_ld(s, TCG_TYPE_PTR, r0, TCG_AREG0, mask_off); - tcg_out_ld(s, TCG_TYPE_PTR, r1, TCG_AREG0, table_off); + QEMU_BUILD_BUG_ON(TLB_MASK_TABLE_OFS(0) > 0); + QEMU_BUILD_BUG_ON(TLB_MASK_TABLE_OFS(0) < -(1 << 12)); + tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_T2, TCG_AREG0, mask_off); + tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_T3, TCG_AREG0, table_off); /* Extract the page index, shifted into place for tlb index. */ - tcg_out_arithi(s, r2, addr, TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS, - SHIFT_SRL); - tcg_out_arith(s, r2, r2, r0, ARITH_AND); + tcg_out_arithi(s, TCG_REG_T1, addr_reg, + TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS, SHIFT_SRL); + tcg_out_arith(s, TCG_REG_T1, TCG_REG_T1, TCG_REG_T2, ARITH_AND); /* Add the tlb_table pointer, creating the CPUTLBEntry address into R2. */ - tcg_out_arith(s, r2, r2, r1, ARITH_ADD); + tcg_out_arith(s, TCG_REG_T1, TCG_REG_T1, TCG_REG_T3, ARITH_ADD); - /* Load the tlb comparator and the addend. */ - tcg_out_ld(s, TCG_TYPE_TL, r0, r2, which); - tcg_out_ld(s, TCG_TYPE_PTR, r1, r2, offsetof(CPUTLBEntry, addend)); + /* Load the tlb comparator and the addend. */ + tcg_out_ld(s, TCG_TYPE_TL, TCG_REG_T2, TCG_REG_T1, cmp_off); + tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_T1, TCG_REG_T1, add_off); + h->base = TCG_REG_T1; - /* Mask out the page offset, except for the required alignment. - We don't support unaligned accesses. */ - if (a_bits < s_bits) { - a_bits = s_bits; - } - compare_mask = (tcg_target_ulong)TARGET_PAGE_MASK | ((1 << a_bits) - 1); + /* Mask out the page offset, except for the required alignment. */ + compare_mask = TARGET_PAGE_MASK | a_mask; if (check_fit_tl(compare_mask, 13)) { - tcg_out_arithi(s, r2, addr, compare_mask, ARITH_AND); + tcg_out_arithi(s, TCG_REG_T3, addr_reg, compare_mask, ARITH_AND); } else { - tcg_out_movi(s, TCG_TYPE_TL, r2, compare_mask); - tcg_out_arith(s, r2, addr, r2, ARITH_AND); + tcg_out_movi_s32(s, TCG_REG_T3, compare_mask); + tcg_out_arith(s, TCG_REG_T3, addr_reg, TCG_REG_T3, ARITH_AND); } - tcg_out_cmp(s, r0, r2, 0); + tcg_out_cmp(s, TCG_REG_T2, TCG_REG_T3, 0); - /* If the guest address must be zero-extended, do so now. */ + ldst = new_ldst_label(s); + ldst->is_ld = is_ld; + ldst->oi = oi; + ldst->addrlo_reg = addr_reg; + ldst->label_ptr[0] = s->code_ptr; + + /* bne,pn %[xi]cc, label0 */ + cc = TARGET_LONG_BITS == 64 ? BPCC_XCC : BPCC_ICC; + tcg_out_bpcc0(s, COND_NE, BPCC_PN | cc, 0); +#else + if (a_bits != s_bits) { + /* + * Test for at least natural alignment, and defer + * everything else to the helper functions. + */ + tcg_debug_assert(check_fit_tl(a_mask, 13)); + tcg_out_arithi(s, TCG_REG_G0, addr_reg, a_mask, ARITH_ANDCC); + + ldst = new_ldst_label(s); + ldst->is_ld = is_ld; + ldst->oi = oi; + ldst->addrlo_reg = addr_reg; + ldst->label_ptr[0] = s->code_ptr; + + /* bne,pn %icc, label0 */ + tcg_out_bpcc0(s, COND_NE, BPCC_PN | BPCC_ICC, 0); + } + h->base = guest_base ? TCG_GUEST_BASE_REG : TCG_REG_G0; +#endif + + /* If the guest address must be zero-extended, do in the delay slot. */ if (TARGET_LONG_BITS == 32) { - tcg_out_ext32u(s, r0, addr); - return r0; + tcg_out_ext32u(s, TCG_REG_T2, addr_reg); + h->index = TCG_REG_T2; + } else { + if (ldst) { + tcg_out_nop(s); + } + h->index = addr_reg; } - return addr; + return ldst; } -#endif /* CONFIG_SOFTMMU */ - -static const int qemu_ld_opc[(MO_SSIZE | MO_BSWAP) + 1] = { - [MO_UB] = LDUB, - [MO_SB] = LDSB, - [MO_UB | MO_LE] = LDUB, - [MO_SB | MO_LE] = LDSB, - - [MO_BEUW] = LDUH, - [MO_BESW] = LDSH, - [MO_BEUL] = LDUW, - [MO_BESL] = LDSW, - [MO_BEUQ] = LDX, - [MO_BESQ] = LDX, - - [MO_LEUW] = LDUH_LE, - [MO_LESW] = LDSH_LE, - [MO_LEUL] = LDUW_LE, - [MO_LESL] = LDSW_LE, - [MO_LEUQ] = LDX_LE, - [MO_LESQ] = LDX_LE, -}; - -static const int qemu_st_opc[(MO_SIZE | MO_BSWAP) + 1] = { - [MO_UB] = STB, - - [MO_BEUW] = STH, - [MO_BEUL] = STW, - [MO_BEUQ] = STX, - - [MO_LEUW] = STH_LE, - [MO_LEUL] = STW_LE, - [MO_LEUQ] = STX_LE, -}; static void tcg_out_qemu_ld(TCGContext *s, TCGReg data, TCGReg addr, MemOpIdx oi, TCGType data_type) { - MemOp memop = get_memop(oi); - tcg_insn_unit *label_ptr; + static const int ld_opc[(MO_SSIZE | MO_BSWAP) + 1] = { + [MO_UB] = LDUB, + [MO_SB] = LDSB, + [MO_UB | MO_LE] = LDUB, + [MO_SB | MO_LE] = LDSB, -#ifdef CONFIG_SOFTMMU - unsigned memi = get_mmuidx(oi); - TCGReg addrz; - const tcg_insn_unit *func; + [MO_BEUW] = LDUH, + [MO_BESW] = LDSH, + [MO_BEUL] = LDUW, + [MO_BESL] = LDSW, + [MO_BEUQ] = LDX, + [MO_BESQ] = LDX, - addrz = tcg_out_tlb_load(s, addr, memi, memop, - offsetof(CPUTLBEntry, addr_read)); + [MO_LEUW] = LDUH_LE, + [MO_LESW] = LDSH_LE, + [MO_LEUL] = LDUW_LE, + [MO_LESL] = LDSW_LE, + [MO_LEUQ] = LDX_LE, + [MO_LESQ] = LDX_LE, + }; - /* The fast path is exactly one insn. Thus we can perform the - entire TLB Hit in the (annulled) delay slot of the branch - over the TLB Miss case. */ + TCGLabelQemuLdst *ldst; + HostAddress h; - /* beq,a,pt %[xi]cc, label0 */ - label_ptr = s->code_ptr; - tcg_out_bpcc0(s, COND_E, BPCC_A | BPCC_PT - | (TARGET_LONG_BITS == 64 ? BPCC_XCC : BPCC_ICC), 0); - /* delay slot */ - tcg_out_ldst_rr(s, data, addrz, TCG_REG_O1, - qemu_ld_opc[memop & (MO_BSWAP | MO_SSIZE)]); + ldst = prepare_host_addr(s, &h, addr, oi, true); - /* TLB Miss. */ + tcg_out_ldst_rr(s, data, h.base, h.index, + ld_opc[get_memop(oi) & (MO_BSWAP | MO_SSIZE)]); - tcg_out_mov(s, TCG_TYPE_REG, TCG_REG_O1, addrz); - - /* We use the helpers to extend SB and SW data, leaving the case - of SL needing explicit extending below. */ - if ((memop & MO_SSIZE) == MO_SL) { - func = qemu_ld_trampoline[MO_UL]; - } else { - func = qemu_ld_trampoline[memop & MO_SSIZE]; + if (ldst) { + ldst->type = data_type; + ldst->datalo_reg = data; + ldst->raddr = tcg_splitwx_to_rx(s->code_ptr); } - tcg_debug_assert(func != NULL); - tcg_out_call_nodelay(s, func, false); - /* delay slot */ - tcg_out_movi(s, TCG_TYPE_I32, TCG_REG_O2, oi); - - /* We let the helper sign-extend SB and SW, but leave SL for here. */ - if ((memop & MO_SSIZE) == MO_SL) { - tcg_out_ext32s(s, data, TCG_REG_O0); - } else { - tcg_out_mov(s, TCG_TYPE_REG, data, TCG_REG_O0); - } - - *label_ptr |= INSN_OFF19(tcg_ptr_byte_diff(s->code_ptr, label_ptr)); -#else - TCGReg index = (guest_base ? TCG_GUEST_BASE_REG : TCG_REG_G0); - unsigned a_bits = get_alignment_bits(memop); - unsigned s_bits = memop & MO_SIZE; - unsigned t_bits; - - if (TARGET_LONG_BITS == 32) { - tcg_out_ext32u(s, TCG_REG_T1, addr); - addr = TCG_REG_T1; - } - - /* - * Normal case: alignment equal to access size. - */ - if (a_bits == s_bits) { - tcg_out_ldst_rr(s, data, addr, index, - qemu_ld_opc[memop & (MO_BSWAP | MO_SSIZE)]); - return; - } - - /* - * Test for at least natural alignment, and assume most accesses - * will be aligned -- perform a straight load in the delay slot. - * This is required to preserve atomicity for aligned accesses. - */ - t_bits = MAX(a_bits, s_bits); - tcg_debug_assert(t_bits < 13); - tcg_out_arithi(s, TCG_REG_G0, addr, (1u << t_bits) - 1, ARITH_ANDCC); - - /* beq,a,pt %icc, label */ - label_ptr = s->code_ptr; - tcg_out_bpcc0(s, COND_E, BPCC_A | BPCC_PT | BPCC_ICC, 0); - /* delay slot */ - tcg_out_ldst_rr(s, data, addr, index, - qemu_ld_opc[memop & (MO_BSWAP | MO_SSIZE)]); - - if (a_bits >= s_bits) { - /* - * Overalignment: A successful alignment test will perform the memory - * operation in the delay slot, and failure need only invoke the - * handler for SIGBUS. - */ - tcg_out_call_nodelay(s, qemu_unalign_ld_trampoline, false); - /* delay slot -- move to low part of argument reg */ - tcg_out_mov_delay(s, TCG_REG_O1, addr); - } else { - /* Underalignment: load by pieces of minimum alignment. */ - int ld_opc, a_size, s_size, i; - - /* - * Force full address into T1 early; avoids problems with - * overlap between @addr and @data. - */ - tcg_out_arith(s, TCG_REG_T1, addr, index, ARITH_ADD); - - a_size = 1 << a_bits; - s_size = 1 << s_bits; - if ((memop & MO_BSWAP) == MO_BE) { - ld_opc = qemu_ld_opc[a_bits | MO_BE | (memop & MO_SIGN)]; - tcg_out_ldst(s, data, TCG_REG_T1, 0, ld_opc); - ld_opc = qemu_ld_opc[a_bits | MO_BE]; - for (i = a_size; i < s_size; i += a_size) { - tcg_out_ldst(s, TCG_REG_T2, TCG_REG_T1, i, ld_opc); - tcg_out_arithi(s, data, data, a_size, SHIFT_SLLX); - tcg_out_arith(s, data, data, TCG_REG_T2, ARITH_OR); - } - } else if (a_bits == 0) { - ld_opc = LDUB; - tcg_out_ldst(s, data, TCG_REG_T1, 0, ld_opc); - for (i = a_size; i < s_size; i += a_size) { - if ((memop & MO_SIGN) && i == s_size - a_size) { - ld_opc = LDSB; - } - tcg_out_ldst(s, TCG_REG_T2, TCG_REG_T1, i, ld_opc); - tcg_out_arithi(s, TCG_REG_T2, TCG_REG_T2, i * 8, SHIFT_SLLX); - tcg_out_arith(s, data, data, TCG_REG_T2, ARITH_OR); - } - } else { - ld_opc = qemu_ld_opc[a_bits | MO_LE]; - tcg_out_ldst_rr(s, data, TCG_REG_T1, TCG_REG_G0, ld_opc); - for (i = a_size; i < s_size; i += a_size) { - tcg_out_arithi(s, TCG_REG_T1, TCG_REG_T1, a_size, ARITH_ADD); - if ((memop & MO_SIGN) && i == s_size - a_size) { - ld_opc = qemu_ld_opc[a_bits | MO_LE | MO_SIGN]; - } - tcg_out_ldst_rr(s, TCG_REG_T2, TCG_REG_T1, TCG_REG_G0, ld_opc); - tcg_out_arithi(s, TCG_REG_T2, TCG_REG_T2, i * 8, SHIFT_SLLX); - tcg_out_arith(s, data, data, TCG_REG_T2, ARITH_OR); - } - } - } - - *label_ptr |= INSN_OFF19(tcg_ptr_byte_diff(s->code_ptr, label_ptr)); -#endif /* CONFIG_SOFTMMU */ } static void tcg_out_qemu_st(TCGContext *s, TCGReg data, TCGReg addr, MemOpIdx oi, TCGType data_type) { - MemOp memop = get_memop(oi); - tcg_insn_unit *label_ptr; + static const int st_opc[(MO_SIZE | MO_BSWAP) + 1] = { + [MO_UB] = STB, -#ifdef CONFIG_SOFTMMU - unsigned memi = get_mmuidx(oi); - TCGReg addrz; - const tcg_insn_unit *func; + [MO_BEUW] = STH, + [MO_BEUL] = STW, + [MO_BEUQ] = STX, - addrz = tcg_out_tlb_load(s, addr, memi, memop, - offsetof(CPUTLBEntry, addr_write)); + [MO_LEUW] = STH_LE, + [MO_LEUL] = STW_LE, + [MO_LEUQ] = STX_LE, + }; - /* The fast path is exactly one insn. Thus we can perform the entire - TLB Hit in the (annulled) delay slot of the branch over TLB Miss. */ - /* beq,a,pt %[xi]cc, label0 */ - label_ptr = s->code_ptr; - tcg_out_bpcc0(s, COND_E, BPCC_A | BPCC_PT - | (TARGET_LONG_BITS == 64 ? BPCC_XCC : BPCC_ICC), 0); - /* delay slot */ - tcg_out_ldst_rr(s, data, addrz, TCG_REG_O1, - qemu_st_opc[memop & (MO_BSWAP | MO_SIZE)]); + TCGLabelQemuLdst *ldst; + HostAddress h; - /* TLB Miss. */ + ldst = prepare_host_addr(s, &h, addr, oi, false); - tcg_out_mov(s, TCG_TYPE_REG, TCG_REG_O1, addrz); - tcg_out_movext(s, (memop & MO_SIZE) == MO_64 ? TCG_TYPE_I64 : TCG_TYPE_I32, - TCG_REG_O2, data_type, memop & MO_SIZE, data); + tcg_out_ldst_rr(s, data, h.base, h.index, + st_opc[get_memop(oi) & (MO_BSWAP | MO_SIZE)]); - func = qemu_st_trampoline[memop & MO_SIZE]; - tcg_debug_assert(func != NULL); - tcg_out_call_nodelay(s, func, false); - /* delay slot */ - tcg_out_movi(s, TCG_TYPE_I32, TCG_REG_O3, oi); - - *label_ptr |= INSN_OFF19(tcg_ptr_byte_diff(s->code_ptr, label_ptr)); -#else - TCGReg index = (guest_base ? TCG_GUEST_BASE_REG : TCG_REG_G0); - unsigned a_bits = get_alignment_bits(memop); - unsigned s_bits = memop & MO_SIZE; - unsigned t_bits; - - if (TARGET_LONG_BITS == 32) { - tcg_out_ext32u(s, TCG_REG_T1, addr); - addr = TCG_REG_T1; + if (ldst) { + ldst->type = data_type; + ldst->datalo_reg = data; + ldst->raddr = tcg_splitwx_to_rx(s->code_ptr); } - - /* - * Normal case: alignment equal to access size. - */ - if (a_bits == s_bits) { - tcg_out_ldst_rr(s, data, addr, index, - qemu_st_opc[memop & (MO_BSWAP | MO_SIZE)]); - return; - } - - /* - * Test for at least natural alignment, and assume most accesses - * will be aligned -- perform a straight store in the delay slot. - * This is required to preserve atomicity for aligned accesses. - */ - t_bits = MAX(a_bits, s_bits); - tcg_debug_assert(t_bits < 13); - tcg_out_arithi(s, TCG_REG_G0, addr, (1u << t_bits) - 1, ARITH_ANDCC); - - /* beq,a,pt %icc, label */ - label_ptr = s->code_ptr; - tcg_out_bpcc0(s, COND_E, BPCC_A | BPCC_PT | BPCC_ICC, 0); - /* delay slot */ - tcg_out_ldst_rr(s, data, addr, index, - qemu_st_opc[memop & (MO_BSWAP | MO_SIZE)]); - - if (a_bits >= s_bits) { - /* - * Overalignment: A successful alignment test will perform the memory - * operation in the delay slot, and failure need only invoke the - * handler for SIGBUS. - */ - tcg_out_call_nodelay(s, qemu_unalign_st_trampoline, false); - /* delay slot -- move to low part of argument reg */ - tcg_out_mov_delay(s, TCG_REG_O1, addr); - } else { - /* Underalignment: store by pieces of minimum alignment. */ - int st_opc, a_size, s_size, i; - - /* - * Force full address into T1 early; avoids problems with - * overlap between @addr and @data. - */ - tcg_out_arith(s, TCG_REG_T1, addr, index, ARITH_ADD); - - a_size = 1 << a_bits; - s_size = 1 << s_bits; - if ((memop & MO_BSWAP) == MO_BE) { - st_opc = qemu_st_opc[a_bits | MO_BE]; - for (i = 0; i < s_size; i += a_size) { - TCGReg d = data; - int shift = (s_size - a_size - i) * 8; - if (shift) { - d = TCG_REG_T2; - tcg_out_arithi(s, d, data, shift, SHIFT_SRLX); - } - tcg_out_ldst(s, d, TCG_REG_T1, i, st_opc); - } - } else if (a_bits == 0) { - tcg_out_ldst(s, data, TCG_REG_T1, 0, STB); - for (i = 1; i < s_size; i++) { - tcg_out_arithi(s, TCG_REG_T2, data, i * 8, SHIFT_SRLX); - tcg_out_ldst(s, TCG_REG_T2, TCG_REG_T1, i, STB); - } - } else { - /* Note that ST*A with immediate asi must use indexed address. */ - st_opc = qemu_st_opc[a_bits + MO_LE]; - tcg_out_ldst_rr(s, data, TCG_REG_T1, TCG_REG_G0, st_opc); - for (i = a_size; i < s_size; i += a_size) { - tcg_out_arithi(s, TCG_REG_T2, data, i * 8, SHIFT_SRLX); - tcg_out_arithi(s, TCG_REG_T1, TCG_REG_T1, a_size, ARITH_ADD); - tcg_out_ldst_rr(s, TCG_REG_T2, TCG_REG_T1, TCG_REG_G0, st_opc); - } - } - } - - *label_ptr |= INSN_OFF19(tcg_ptr_byte_diff(s->code_ptr, label_ptr)); -#endif /* CONFIG_SOFTMMU */ } static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0) @@ -1744,6 +1499,8 @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_extu_i32_i64: case INDEX_op_extrl_i64_i32: case INDEX_op_extrh_i64_i32: + case INDEX_op_qemu_ld_i32: + case INDEX_op_qemu_ld_i64: return C_O1_I1(r, r); case INDEX_op_st8_i32: @@ -1753,6 +1510,8 @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_st_i32: case INDEX_op_st32_i64: case INDEX_op_st_i64: + case INDEX_op_qemu_st_i32: + case INDEX_op_qemu_st_i64: return C_O0_I2(rZ, r); case INDEX_op_add_i32: @@ -1802,13 +1561,6 @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_muluh_i64: return C_O1_I2(r, r, r); - case INDEX_op_qemu_ld_i32: - case INDEX_op_qemu_ld_i64: - return C_O1_I1(r, s); - case INDEX_op_qemu_st_i32: - case INDEX_op_qemu_st_i64: - return C_O0_I2(sZ, s); - default: g_assert_not_reached(); } From patchwork Mon May 15 14:32:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681954 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1150410wrr; Mon, 15 May 2023 07:39:24 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ68sN2/It7Ttp+iRKUg4RNnPLfg4T/ug2RhnoNXyPv8GkhW3uc+HZWcKeLVJ9czNLy6GHip X-Received: by 2002:a05:6000:11c2:b0:307:9f8:4f30 with SMTP id i2-20020a05600011c200b0030709f84f30mr22640464wrx.70.1684161564727; Mon, 15 May 2023 07:39:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161564; cv=none; d=google.com; s=arc-20160816; b=pkthRP+RNDGRl9h/MYAPgCOxn4I7BL4DNofEVJ+2hJJPXht1MoEsWnhUppUbc1Ej5A BP/tGellxG/DBky6PYOt8VlFR0854f/dJ4mbVrA3vL9L5Nid2XaQIGMlUb0tqb7x0Twj KxPh0zb/wLdYsK/meG68oeWFUH+xHC3nxPA4htXuRVa+3EjWuewYOK0h6/dvh0UiqUN0 0dYb1iLstiQjasWaspEgVPptHRT7rsbeuNmES337bWNdnPkRsjvLk//LxsTLp/pJNYom VORzHFP/ao/KTOVZgDlMu7evbctruh7y18ynmXQypLveapzC3Zd6uvC/8I7q6jWICuWu fmVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=3QMQeFWwqcUFumXr1WKfxiG/Sk++Gp7SwpIzlewGlEA=; b=HHces7xSXdacyPDZxhueMp/eTyfLyZ+foVQs78l1wlPiIBtywzEea6euz4IcFLGaNj +AUWWPCdoY9ILg1zm865BJJPLuJPARB7GaC8FL8ZcRXliS04/XCjWLrXSxnbeoj2ZMDY 5C2NDKk6wVe2hzHD0Tk2QfgOwEOrnJ/VdbZt8P3QUjXkZp3vScDjKFE/dR39tL8fdvMn S6na35eyyDh4nrftAION7sr3iFBQmlcBVazDOT1+sr0pSdSjGg81rgXubYaRX35Ub2Km ey45dDxEm1025OZRt679dc1cFBkNhPFNRj/5h8thV3OhCY5DVyjBZHKjm07VZLg0NvZW 2UuQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=vY8W9qv6; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id s11-20020a5d6a8b000000b00307b2709799si9332131wru.494.2023.05.15.07.39.24 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:39:24 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=vY8W9qv6; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIm-0001FM-1X; Mon, 15 May 2023 10:35:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHo-0007G9-8o for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:52 -0400 Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZHc-00051Z-Jc for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:52 -0400 Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-1aaebed5bd6so90783635ad.1 for ; Mon, 15 May 2023 07:33:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161220; x=1686753220; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3QMQeFWwqcUFumXr1WKfxiG/Sk++Gp7SwpIzlewGlEA=; b=vY8W9qv6VVO3vZNzHxzp15ikXYAwSDUNLoUQKMp7UVlkIdNbXiPFWv9XW+nMGPnDYp n3QvtJ2ewT+qZzUEYdo2YXhxdOUHE34pgyqtXfHOCyrR0p7VCixtzy7/N5tl7BUlehA9 NXY9o7qNJPDD/3led1syGVQmromMrN9w5GafoWJocAquyw7fsQatd0+9Fq3TrhF2AUc7 9f9qHr0gDL8yqt6cjKoEJEWF76Cw//YKTf0YFxCwtupjvWwU5jDvsrAmcIHt2DuWbP3f Rl5iEPebCq6sKdTNd/GgfTBPoLczYsPHPW+wpM39pAxuH9yQn6u5us37TvUZHz+3AT9s ZLgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161220; x=1686753220; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3QMQeFWwqcUFumXr1WKfxiG/Sk++Gp7SwpIzlewGlEA=; b=BxCdvpJHwTFEN3UGrFQHSS84fflyuwbg25Fs4ARLrmt23EKp63IYb91YiUQgh3WpZX 7bKZBH7FoguUN26S+FyrAtvLc2CQop9mDBIKZr3+9y9OQV4JDY7xf8giQB5CpIZO38FY 2ZsSMLtQECEdxCnKGUde3fmdnENT4impoZRbq3tG3vYlTqh7hGcyCDFfRc0h6c0iV1Tn KKECgRl9Zz6aqCQSvN/N8+tpqE6xFkgXAgyFS3eSQvLCq3FRFSqKGTYffTbaU6d5wpRd auIzVi5DodNqLgXpydubCJ0MUUKrmdR2as9ZtFIP3Ds0mjdG93MHifsRKh/Pln4tWn7j Xqvg== X-Gm-Message-State: AC+VfDwQlzde86NsiYi18fsggEtU/E3UujuuDSaSWbecU3vAZxligMa1 ScSmZPz82aYA5IfrL73Qj5luzUdbckpATqKrRtQ= X-Received: by 2002:a17:903:1249:b0:1ab:220a:9068 with SMTP id u9-20020a170903124900b001ab220a9068mr45238720plh.42.1684161220627; Mon, 15 May 2023 07:33:40 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:40 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 30/54] accel/tcg: Remove helper_unaligned_{ld,st} Date: Mon, 15 May 2023 07:32:49 -0700 Message-Id: <20230515143313.734053-31-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62f; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org These functions are now unused. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- include/tcg/tcg-ldst.h | 6 ------ accel/tcg/user-exec.c | 10 ---------- 2 files changed, 16 deletions(-) diff --git a/include/tcg/tcg-ldst.h b/include/tcg/tcg-ldst.h index 64f48e6990..7dd57013e9 100644 --- a/include/tcg/tcg-ldst.h +++ b/include/tcg/tcg-ldst.h @@ -60,10 +60,4 @@ void helper_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, void helper_st16_mmu(CPUArchState *env, target_ulong addr, Int128 val, MemOpIdx oi, uintptr_t retaddr); -#ifdef CONFIG_USER_ONLY - -G_NORETURN void helper_unaligned_ld(CPUArchState *env, target_ulong addr); -G_NORETURN void helper_unaligned_st(CPUArchState *env, target_ulong addr); - -#endif /* CONFIG_USER_ONLY */ #endif /* TCG_LDST_H */ diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c index 8f86254eb4..7b824dcde8 100644 --- a/accel/tcg/user-exec.c +++ b/accel/tcg/user-exec.c @@ -889,16 +889,6 @@ void page_reset_target_data(target_ulong start, target_ulong last) { } /* The softmmu versions of these helpers are in cputlb.c. */ -void helper_unaligned_ld(CPUArchState *env, target_ulong addr) -{ - cpu_loop_exit_sigbus(env_cpu(env), addr, MMU_DATA_LOAD, GETPC()); -} - -void helper_unaligned_st(CPUArchState *env, target_ulong addr) -{ - cpu_loop_exit_sigbus(env_cpu(env), addr, MMU_DATA_STORE, GETPC()); -} - static void *cpu_mmu_lookup(CPUArchState *env, abi_ptr addr, MemOp mop, uintptr_t ra, MMUAccessType type) { From patchwork Mon May 15 14:32:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681979 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1154323wrr; Mon, 15 May 2023 07:48:19 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7M0VjURdYEi+cRNIkq6TxigwAN7GoDKwJfq56lcXCGKL3Vo+06iYo822HCOktoPxJd1TTG X-Received: by 2002:a5d:6885:0:b0:306:373e:7b14 with SMTP id h5-20020a5d6885000000b00306373e7b14mr26449657wru.63.1684162099679; Mon, 15 May 2023 07:48:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684162099; cv=none; d=google.com; s=arc-20160816; b=nXcY2nAzVnZ4bIHezlZlICMlK6QTwz2T5Ft+yRkY+HKOvTvXnWnLTp6vbYgMrAU1gD Tv5FZAcJnhtzJo88AQNTYI/Pq3lCglrWY0baXsNgE6LGQgn9OiQ6umqS+CNgu9WVzYq2 GwXwcFOwFWFVNELig9tkZ8GNPxI6UrbQLfmUV9QFClR+chh9r3j9DPgbg5yCL1Pd0Kdf 69ryuReby9P86xP9lS3mtIGegiInQzDFc/n12KWPt4x0IsUjn1TUOqfvl03oPGoT0Oxi 8irrM4KM2yBHbVyow9Ps0TKWiEk/LzRihWAL8aDvmUgim+c+4spzLIyodcfRr2ZXwKOp EZaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=C7A+mEX1ZrmjjLYqtzKZHi9DyEb4qEYbCIsoKmrRzio=; b=Ly+xfpVG193L0gDqwiRScf6NnLR1jFw9jz45jiD5vk9GhS8+KuV1yvW1U4jco+Cuxk EmnEzXqK+FJFKYBElnLoiLDko/GOY9epcbmwiycxVney9MmqJT8/TBPYlAz7XTYTWHT4 ZWAHZOhTM0PAk9leIBDIStDAE+2WKuRLJa7pNAsP1t67FCuMbfiyAtilnZA9kwws0TZ1 JjNw3HTLg5VZyOzcEbYJe4mVWH9pXPSSuC83aeR6pEMJ8ehWfIsB9P3Dh25Tc9nFcuZ/ FfU7/0VEWZx+vsyMMPZlD500ggFhVEO0YYMnm5nlJM/3B86t6Ijri43Ks/+WYLWn1z+l /dZA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Qx4vHVbi; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id n10-20020a5d484a000000b0030641171ee2si1225wrs.391.2023.05.15.07.48.19 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:48:19 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Qx4vHVbi; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIf-0000d0-4D; Mon, 15 May 2023 10:35:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHj-0007C6-Vy for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:50 -0400 Received: from mail-pj1-x102b.google.com ([2607:f8b0:4864:20::102b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZHc-00051t-D1 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:46 -0400 Received: by mail-pj1-x102b.google.com with SMTP id 98e67ed59e1d1-24e09b4153eso8789301a91.2 for ; Mon, 15 May 2023 07:33:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161221; x=1686753221; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C7A+mEX1ZrmjjLYqtzKZHi9DyEb4qEYbCIsoKmrRzio=; b=Qx4vHVbiUZILX0umSe3uJ//SoyiAOG2QS/k+csfwJlgO6E/jwDGcSvOOOYEhTcxepU NoIul6WxPoeKEK+8h+pRxVQtVC7EjN+hMMqqznWKTQtHzwsySZCcVmCmOyh9pGdtY0oi bztJ6KQJtKd6LAukLW9EB6D+4OsATa/GeLeP2Pf4tLzY8kdUIz5yBEr6T/FK/GBfTz3B yJpT8X5IhTYj3iNfeEpwpsO0fHr1S9fmP0A3osMPkiOqXWbivY2mjtdJVB0IyFK/OUJJ xUaQ7/JsYq6NMFd/buI6f/EtQgjKTqhsq9X8VCheSkj9WEqzlWl7zQXH6aE7YUODizzW fQAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161221; x=1686753221; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C7A+mEX1ZrmjjLYqtzKZHi9DyEb4qEYbCIsoKmrRzio=; b=Y1wvAdZ9MksxeG2gUt5G2SSMEJy3rHb3uIUbhxWyNl06H65Eyf8humxbfmmyRc+O9R Ufkgd8iwuNpHNsGWyfK+M3t/gyoszW/W4FelFTqBblE09HyeA7UMzRsCrpZIkYacyFD/ h+IcHhsG6HRCtJA+eLfwIjuCHPIUhkrVXnwQs7qJUA3nVDPq70kyFSNvU+Xyw815fZJn x95Lv0LesJItUnoktu3rh3HSFYePYixGYTu8DrtDZnmXtbsP1m5NGiEUWs93jeoyeuOc TVkW3tcuwqAn2sSas2rXrFG1D3IzmdB5NKhBwLzinlevCE39O3P1MXTSbcccpcrvwmjc F5aQ== X-Gm-Message-State: AC+VfDwIXusQ1VeQ+Va/T+YQ6mwCj9ldzsVNst5a6yfmLSPXTl1/4fbW jrmGnNZkCwxr0tWkG079K2h12nrC/GqfmVbvcgI= X-Received: by 2002:a17:90b:4c06:b0:246:5f9e:e4cf with SMTP id na6-20020a17090b4c0600b002465f9ee4cfmr33500057pjb.43.1684161221380; Mon, 15 May 2023 07:33:41 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:41 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 31/54] tcg/loongarch64: Check the host supports unaligned accesses Date: Mon, 15 May 2023 07:32:50 -0700 Message-Id: <20230515143313.734053-32-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::102b; envelope-from=richard.henderson@linaro.org; helo=mail-pj1-x102b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org This should be true of all loongarch64 running Linux. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- tcg/loongarch64/tcg-target.c.inc | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index e651ec5c71..33d8e67513 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -30,6 +30,7 @@ */ #include "../tcg-ldst.c.inc" +#include #ifdef CONFIG_DEBUG_TCG static const char * const tcg_target_reg_names[TCG_TARGET_NB_REGS] = { @@ -1674,6 +1675,14 @@ static void tcg_target_qemu_prologue(TCGContext *s) static void tcg_target_init(TCGContext *s) { + unsigned long hwcap = qemu_getauxval(AT_HWCAP); + + /* Server and desktop class cpus have UAL; embedded cpus do not. */ + if (!(hwcap & HWCAP_LOONGARCH_UAL)) { + error_report("TCG: unaligned access support required; exiting"); + exit(EXIT_FAILURE); + } + tcg_target_available_regs[TCG_TYPE_I32] = ALL_GENERAL_REGS; tcg_target_available_regs[TCG_TYPE_I64] = ALL_GENERAL_REGS; From patchwork Mon May 15 14:32:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681970 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1152763wrr; Mon, 15 May 2023 07:44:34 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5MnSiPO3w+50xGkX61V+mcAX2D3MN6sm1gIuZqyfZCR9xwc1f37Y9gXB0U/FeJEcz+eHG6 X-Received: by 2002:a05:600c:19c6:b0:3f5:b15:87ee with SMTP id u6-20020a05600c19c600b003f50b1587eemr2329128wmq.18.1684161873988; Mon, 15 May 2023 07:44:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161873; cv=none; d=google.com; s=arc-20160816; b=XyKdvjJLBVwI0SXr+MRsLiVEuqeMkAELzFD8iCV4pv82f9wTBH1DFGEVOd3Hv4HCuz r3Pfb32LhEk43TQsHfT6T7I51HkNNW4daXrza99kSi4aiqHWPiN0x0rSAJrETz08cf2b l7UIZIQaYCXJyKEeh9cQCayurGfliaR7JVBAahGusiXlzgAVH41dqSvv+45V8GrrQxtJ SgBTyRLqSyIHRU2xVvq1o4sv+BdbvBRHdAv/WdqDPQuQuVCgS4xMaBe++KefF5RssxhE f+s/fBT7vjlHy8BhUbL6e23qVPtp4iMtWG5A/NfCtSwk9NApYDtaZG6fPIN5pNNLN1TY MhmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=4qhfCxW8nZ+AIzOPAndxpM5sZuukyWLZqrgdNp4ChwI=; b=cbaxgLiJidhNc8xYvPxVYIFEx8e9GtYOVF5j6PiiWnluOCSn2632Pimr1pun+t996D UFchdpJoFLrg+JXr2GiPkvFV5FPstRvW8vthqAatuXSwWfF/ycAJY0TQ561fs7e75tTO /XC+UZmTMlVPDI1Hx5cXXCGLUz9McCb0fhF9eS+sWFOny2le/t/t06xmPiu5Z4xFq4OV zrI4uXhuy1vMauyIUv4D81/Rn+GE80shP7KfCz9PZBMaJmBkFQ6erfJoQzzXHIs5hh6h tmEboDygTfsVnI+biA8UxXBPM1/kp/mRjxQYmK+e3O2IpMhqRs1l2/+hv/jRFI+oDU99 nxkA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=JbczjSV8; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id fj3-20020a05600c0c8300b003f4b6c0e663si8900993wmb.175.2023.05.15.07.44.33 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:44:33 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=JbczjSV8; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZJc-0004Vh-V7; Mon, 15 May 2023 10:36:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHl-0007Ck-VO for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:50 -0400 Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZHc-00052A-Io for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:49 -0400 Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-1aae5c2423dso124343345ad.3 for ; Mon, 15 May 2023 07:33:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161222; x=1686753222; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4qhfCxW8nZ+AIzOPAndxpM5sZuukyWLZqrgdNp4ChwI=; b=JbczjSV8D2Dsp7LYY84GHTuQ3Oy/4SRTkgrexRE+8nhY9plFDJKp8YsoDG8+R6HbZA GmaUC7r0nllv3Hu8IesnALOV+QdmW12viR9uOsGgQizupqefhYuO+uTf2NW3o8Lce3QB 8Y0hTYFdVZm/nE3mVLFLxH/k9koK3kO9WkCvuBK/r9z77dbKK3J/EXLrl/MybjsP86o9 DWqCAnU+J5nvaDzlkqErNJoqqEgG3CYjajh65yK43nGN5kP3KR1XENxVJn4vg/d0kvsk TOndX5zt4/OdrHtCZAVJvfCVYm8Q/rbyPrWexs016beH5tgf2Q66HPRkt1CmNb/GEhc7 +iXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161222; x=1686753222; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4qhfCxW8nZ+AIzOPAndxpM5sZuukyWLZqrgdNp4ChwI=; b=IEuuEeA7s+J72oqqxJhAfsT7wF22Dz8nQ1Qtb2zhtW39RspYAUBUKnFeFnLr2u14aP q3MZ5bnFUS08Sz91Bt2go9CSnLsvCra+2UbTO9eYFjIsl2f6RV7B8EqdI0rJx9n+RXdO 5NblMcEKMLDP9hMy2PTduplEXQ4OqsR3BAwDUXciZWUoBPgHKp/7bULQ7gdEQauYEvyi rqt/WUddeGaBYsT0570YOOKv8h4vmlUJy6CNm+jRr72gpGR//HeTe1VTDcS0tKYNhB1u QbOLyBvA5mz2n0ekP0d//CyEYvSJd+lcPowTDrE9uWN/X2x0N2DiLtVW3adTJuPvcwxU COAQ== X-Gm-Message-State: AC+VfDzu7sejBLmDSopgBaDBAxXD5ZzmL8apneSClWLZxS+cvUWlkgwx 6aAGSjLyb8xazEvzH25DpzMbY90Ep7UbZwkZNBM= X-Received: by 2002:a17:902:b593:b0:1ad:ea13:1918 with SMTP id a19-20020a170902b59300b001adea131918mr10339312pls.69.1684161222326; Mon, 15 May 2023 07:33:42 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:41 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 32/54] tcg/loongarch64: Support softmmu unaligned accesses Date: Mon, 15 May 2023 07:32:51 -0700 Message-Id: <20230515143313.734053-33-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::630; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x630.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Test the final byte of an unaligned access. Use BSTRINS.D to clear the range of bits, rather than AND. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/loongarch64/tcg-target.c.inc | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index 33d8e67513..7d0165349d 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -848,7 +848,6 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, int fast_ofs = TLB_MASK_TABLE_OFS(mem_index); int mask_ofs = fast_ofs + offsetof(CPUTLBDescFast, mask); int table_ofs = fast_ofs + offsetof(CPUTLBDescFast, table); - tcg_target_long compare_mask; ldst = new_ldst_label(s); ldst->is_ld = is_ld; @@ -872,14 +871,20 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TMP2, TCG_REG_TMP2, offsetof(CPUTLBEntry, addend)); - /* We don't support unaligned accesses. */ + /* + * For aligned accesses, we check the first byte and include the alignment + * bits within the address. For unaligned access, we check that we don't + * cross pages using the address of the last byte of the access. + */ if (a_bits < s_bits) { - a_bits = s_bits; + unsigned a_mask = (1u << a_bits) - 1; + unsigned s_mask = (1u << s_bits) - 1; + tcg_out_addi(s, TCG_TYPE_TL, TCG_REG_TMP1, addr_reg, s_mask - a_mask); + } else { + tcg_out_mov(s, TCG_TYPE_TL, TCG_REG_TMP1, addr_reg); } - /* Clear the non-page, non-alignment bits from the address. */ - compare_mask = (tcg_target_long)TARGET_PAGE_MASK | ((1 << a_bits) - 1); - tcg_out_movi(s, TCG_TYPE_TL, TCG_REG_TMP1, compare_mask); - tcg_out_opc_and(s, TCG_REG_TMP1, TCG_REG_TMP1, addr_reg); + tcg_out_opc_bstrins_d(s, TCG_REG_TMP1, TCG_REG_ZERO, + a_bits, TARGET_PAGE_BITS - 1); /* Compare masked address with the TLB entry. */ ldst->label_ptr[0] = s->code_ptr; From patchwork Mon May 15 14:32:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681961 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1152162wrr; Mon, 15 May 2023 07:43:13 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5+GeoGgp8aPno7J0Y78eHJwRLyJRi/dhh6QMM9JY0rqIvpXaD7YDgMLZVxzBf28GQi7d45 X-Received: by 2002:adf:e909:0:b0:306:2db9:cc2c with SMTP id f9-20020adfe909000000b003062db9cc2cmr27032211wrm.32.1684161792902; Mon, 15 May 2023 07:43:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161792; cv=none; d=google.com; s=arc-20160816; b=BX8PajZRS6uNIWwMrMf2vJ8Kg/pNWHgBRXsJejrqNnVyF1xzNEQqfNKmKILWk8RvgA I9R8F91pAPlRp8pkoq6wd7/9N80ft11hsoJQdDCv8fu1RDZCIFMbFLB9xwayO91f5GQQ ZKCyXeIDGvAM1SKiIe1kFK4ydHqYbfRkd9PIP2P0gV+DlOOfl/swdo0RywWw7hjVABEp MBowgE1zbX0/0auATJnH84fPCAaVnHpZmiItKzvUqNse39FjQz+ywFr1zlQptNKklKGi SWmcbpCZnRcU/7qqXPeyvHMLOVQ63IqWZd+Tlh3E1RpPyxZIUduYGREvWqKpEHuUt+6R ldQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=RcueoQ7mogTzwxRR1biig20yNfysNgxqkYSawavzYAo=; b=lyCBhaspxp1L6j79z6PtNM9NIFE9Typ451X6vQTwT1u9k+qbl5yeiLxRC7tq/h2ZOf UegtG5H0pXMwKwy1exYFTeUYfbHD4Mq5IkzjRF2PgyEQD0G1KnKKW9Bhk+EoqDn0Arky Y1d0/h8UdE/o/X/cw/pAuV9NWyFD5jBxzaREIzXF+Yp58Mo9ce+LMHnyL072uEX6jqEp Bf81m4048i69QyIaRkNgxN3sK3NkcGZu/0XtqkxhFrEDQtBZK+C00f5m5yftwTRUs3fC OwkL+xBP3WvyPMTovSmasAZabCDkUbZ3lTsVSmoydj8I9eEJ5BaWr/wUpFejZa+a2RpR oOuA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ZZ9BTpD6; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id u11-20020adfdd4b000000b003062aa1a9ccsi11501wrm.41.2023.05.15.07.43.12 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:43:12 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ZZ9BTpD6; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZJS-0003KL-KS; Mon, 15 May 2023 10:36:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHh-00079p-54 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:47 -0400 Received: from mail-pl1-x62a.google.com ([2607:f8b0:4864:20::62a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZHa-0004sJ-3L for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:43 -0400 Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-1ab05018381so118665145ad.2 for ; Mon, 15 May 2023 07:33:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161223; x=1686753223; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RcueoQ7mogTzwxRR1biig20yNfysNgxqkYSawavzYAo=; b=ZZ9BTpD69lKmUJLOZGgT+fUwwlkRO7Q8KlzqvE7E66V5RcJsnMgpex4J0b+xF37yex N1OmRfIPGD5PxmtgNAX718e/cUG4a8DOm5jWetYRPuO6PutWEPLAgiGTitSAN6J5ICxE HwCU6kb3yJ3CuNWr9qc/RK/qebrPOZKzUjWl752dapPMHoiE3Bv5NbJN8HBNF6unepbZ 2kvOOg30yKhONpODMPKGapqe8dMJSY22UgsuVRMhGt/RVq6qwfIpS+fuQbm0YUMReQ5y o0OYJ04jD8M6tPK7r47iRhR7YHzI44C89Zw8APREQ9GP/gM97qdPGQOQv9bw/PEiWHwZ 5nJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161223; x=1686753223; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RcueoQ7mogTzwxRR1biig20yNfysNgxqkYSawavzYAo=; b=Hy7gGjPI+CFC4tXXI5bfI2WQPlYmvS5YLQXfu/giIkIq1xqOPVvi11Or2p2Iuf0YYE RNzxfCvpJkQsvQIu1kfu81svZeHzrt+vPidGxKM10RJanFtT/I1ysKDfK6HxQO3mTDUk Oel2w9KcV5A1SJkJAug3r/CynnCw10y2OC1n1Oy3MvhlaLO8hphTlnrjFF3XH13WZWE0 iHe9vVAQxnUYx208Ku52jRVTnSTAE/AnCpbannDTKiabJp0A/LOumxlycLntGg5hot6/ zWtkcSJmsGaiT8W5rathaaOjhH//uZGsbOtDujEK2wrmhzMosG3/hX1Do1sZcKeuHNx6 /xJw== X-Gm-Message-State: AC+VfDziBVrZpPSImEcPSYT0fMl2IWPraDLVUSHzQMW5HcjkuQK/j20K asUQdAwreXdg+iFmkCMg2Isq2N9YoP85SUco7sg= X-Received: by 2002:a17:903:244d:b0:1a5:2540:729 with SMTP id l13-20020a170903244d00b001a525400729mr44025420pls.56.1684161223150; Mon, 15 May 2023 07:33:43 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:42 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, LIU Zhiwei Subject: [PATCH v5 33/54] tcg/riscv: Support softmmu unaligned accesses Date: Mon, 15 May 2023 07:32:52 -0700 Message-Id: <20230515143313.734053-34-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62a; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org The system is required to emulate unaligned accesses, even if the hardware does not support it. The resulting trap may or may not be more efficient than the qemu slow path. There are linux kernel patches in flight to allow userspace to query hardware support; we can re-evaluate whether to enable this by default after that. In the meantime, softmmu now matches useronly, where we already assumed that unaligned accesses are supported. Reviewed-by: LIU Zhiwei Signed-off-by: Richard Henderson --- tcg/riscv/tcg-target.c.inc | 48 ++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 20 deletions(-) diff --git a/tcg/riscv/tcg-target.c.inc b/tcg/riscv/tcg-target.c.inc index 19cd4507fb..415e6c6e15 100644 --- a/tcg/riscv/tcg-target.c.inc +++ b/tcg/riscv/tcg-target.c.inc @@ -910,12 +910,13 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, TCGReg *pbase, #ifdef CONFIG_SOFTMMU unsigned s_bits = opc & MO_SIZE; + unsigned s_mask = (1u << s_bits) - 1; int mem_index = get_mmuidx(oi); int fast_ofs = TLB_MASK_TABLE_OFS(mem_index); int mask_ofs = fast_ofs + offsetof(CPUTLBDescFast, mask); int table_ofs = fast_ofs + offsetof(CPUTLBDescFast, table); - TCGReg mask_base = TCG_AREG0, table_base = TCG_AREG0; - tcg_target_long compare_mask; + int compare_mask; + TCGReg addr_adj; ldst = new_ldst_label(s); ldst->is_ld = is_ld; @@ -924,14 +925,33 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, TCGReg *pbase, QEMU_BUILD_BUG_ON(TLB_MASK_TABLE_OFS(0) > 0); QEMU_BUILD_BUG_ON(TLB_MASK_TABLE_OFS(0) < -(1 << 11)); - tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TMP0, mask_base, mask_ofs); - tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TMP1, table_base, table_ofs); + tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TMP0, TCG_AREG0, mask_ofs); + tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TMP1, TCG_AREG0, table_ofs); tcg_out_opc_imm(s, OPC_SRLI, TCG_REG_TMP2, addr_reg, TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS); tcg_out_opc_reg(s, OPC_AND, TCG_REG_TMP2, TCG_REG_TMP2, TCG_REG_TMP0); tcg_out_opc_reg(s, OPC_ADD, TCG_REG_TMP2, TCG_REG_TMP2, TCG_REG_TMP1); + /* + * For aligned accesses, we check the first byte and include the alignment + * bits within the address. For unaligned access, we check that we don't + * cross pages using the address of the last byte of the access. + */ + addr_adj = addr_reg; + if (a_bits < s_bits) { + addr_adj = TCG_REG_TMP0; + tcg_out_opc_imm(s, TARGET_LONG_BITS == 32 ? OPC_ADDIW : OPC_ADDI, + addr_adj, addr_reg, s_mask - a_mask); + } + compare_mask = TARGET_PAGE_MASK | a_mask; + if (compare_mask == sextreg(compare_mask, 0, 12)) { + tcg_out_opc_imm(s, OPC_ANDI, TCG_REG_TMP1, addr_adj, compare_mask); + } else { + tcg_out_movi(s, TCG_TYPE_TL, TCG_REG_TMP1, compare_mask); + tcg_out_opc_reg(s, OPC_AND, TCG_REG_TMP1, TCG_REG_TMP1, addr_adj); + } + /* Load the tlb comparator and the addend. */ tcg_out_ld(s, TCG_TYPE_TL, TCG_REG_TMP0, TCG_REG_TMP2, is_ld ? offsetof(CPUTLBEntry, addr_read) @@ -939,29 +959,17 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, TCGReg *pbase, tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TMP2, TCG_REG_TMP2, offsetof(CPUTLBEntry, addend)); - /* We don't support unaligned accesses. */ - if (a_bits < s_bits) { - a_bits = s_bits; - } - /* Clear the non-page, non-alignment bits from the address. */ - compare_mask = (tcg_target_long)TARGET_PAGE_MASK | a_mask; - if (compare_mask == sextreg(compare_mask, 0, 12)) { - tcg_out_opc_imm(s, OPC_ANDI, TCG_REG_TMP1, addr_reg, compare_mask); - } else { - tcg_out_movi(s, TCG_TYPE_TL, TCG_REG_TMP1, compare_mask); - tcg_out_opc_reg(s, OPC_AND, TCG_REG_TMP1, TCG_REG_TMP1, addr_reg); - } - /* Compare masked address with the TLB entry. */ ldst->label_ptr[0] = s->code_ptr; tcg_out_opc_branch(s, OPC_BNE, TCG_REG_TMP0, TCG_REG_TMP1, 0); /* TLB Hit - translate address using addend. */ + addr_adj = addr_reg; if (TARGET_LONG_BITS == 32) { - tcg_out_ext32u(s, TCG_REG_TMP0, addr_reg); - addr_reg = TCG_REG_TMP0; + addr_adj = TCG_REG_TMP0; + tcg_out_ext32u(s, addr_adj, addr_reg); } - tcg_out_opc_reg(s, OPC_ADD, TCG_REG_TMP0, TCG_REG_TMP2, addr_reg); + tcg_out_opc_reg(s, OPC_ADD, TCG_REG_TMP0, TCG_REG_TMP2, addr_adj); *pbase = TCG_REG_TMP0; #else if (a_mask) { From patchwork Mon May 15 14:32:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681972 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1152879wrr; Mon, 15 May 2023 07:44:54 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4+qizZOtoHf3I+zckRAwvmi4rYte64LOJTtxzTPV/sZXus9DktEKPZGyG48pOAx6vjuhjb X-Received: by 2002:a7b:cb92:0:b0:3f4:2148:e8d7 with SMTP id m18-20020a7bcb92000000b003f42148e8d7mr21911863wmi.35.1684161894737; Mon, 15 May 2023 07:44:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161894; cv=none; d=google.com; s=arc-20160816; b=Ltp2pYdewNxS2zS1NepGUOWrLkH0nR9VbkGCb4garyApysFeyDUnfMPb1U1BIrOHTj 0c3vuZJj8phAc8ZnGSEnpqLJHn43mARlSKmu3Jjoxe1BdDHo3+IE7zsqWPYielT6LFCI sfLcChNtdgEaIrExqmX7jlRrBKYRGbnvp9hz8Y2lxDUZtSeqc7QFEXh/uvoK9OboocSU znVbrEMP2nOmPuBuMEosRa2ufHwNJ3cKm3lLmIfYJrs276A+F2aa7fP9Z06Ug2cNdYBW VIQlCNHjlUiWqwobVnuOFWqUM3WqN3YXqmrHy+YApdXO3cSKYb9nk3mTIQV1Ts7xQXtC X+Eg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=HF3N9t+WR/vu9xazr4+Gr/alUcZraRNVjVLeq4yAhBM=; b=jhQO6tze/5QivbAXx+LtjkO4tfh7G92LshsnNuBWRZlCvSYzaj/2BtD4agzHrvQ8iw fDpGcOdXNppkUhdYgAm0kNyg1HVvhyRnOWp9mqUvk80YouVrXjbNEUBfpbwvpF2Bcj+/ v3T5cl2IbjN+f1UtE14T8W/OJXEqAQ1nyEQn4WApG9VCkGIX/Qj1hiwp1uMHSc/gGdn6 xmyvSfef9iTjyQO/9pZ81/nC9sM9REVJ+20lfNDbrnXIZcZVRzkXXjJK64FfplcYURmz L9H7PPv4orKcvOhWPSAke0BFrllWhgwCTp1SvaZ5tqjVAp84VYyHKLXNOjWg2atrlBOf UddA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=SQfmYQDU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id f15-20020a5d568f000000b002fb6c560f76si14544061wrv.466.2023.05.15.07.44.54 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:44:54 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=SQfmYQDU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZJr-0004iZ-4B; Mon, 15 May 2023 10:37:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHk-0007C7-0j for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:50 -0400 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZHc-00052l-Ar for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:47 -0400 Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-1ab13da70a3so124363575ad.1 for ; Mon, 15 May 2023 07:33:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161224; x=1686753224; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HF3N9t+WR/vu9xazr4+Gr/alUcZraRNVjVLeq4yAhBM=; b=SQfmYQDU4sFVunWxtWSFgMP/OTq02g5Ayq9ptppiATAgQum7ank2NvUdRxgPf8noh2 wZ33H0TqFCPEATuumvmhzDJ7TIF4B1YgXJLUDiJFt0hbOmtrdzBqyV1Y7UtFNvnNursN aAdgVGq7kXr2F39JAzbrkboqy5IGaAw7OREQWbXkGOOb3RV2KT1rZckPy5c9FUVVMQtR 0L03Ry5RzTlhd1n+uqWYzlfPvNRFDmFfasqYNOJL+d5qkmoWlK0hCyNuTEXC9JsisStX srfRHXliJflzQRFkzM2lyKY83BZALPR77lUiFr7zw0FXSEPSz8k1ftcF6/MGa9UReWu0 gucA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161224; x=1686753224; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HF3N9t+WR/vu9xazr4+Gr/alUcZraRNVjVLeq4yAhBM=; b=WgrrE58tg2bu8Ch5nchavz5rCnqrBZWcROjQvA4pSINRI0uzmttlceqMOX5sBBXYhF AnC2eLOD7NEuVQCVzHaNimT9Y7LHNj5xLMYEHE2J396PPJoP4GItvP39MeGf2VJqMmXH ak8FGQihhpIpa7BAwAO9GCyBVm0gk1ZHwY3mzt24s5GcVR8cNEaEJZGdZkG2UE+yAv6S 4sORh0pjB0YCmIFjuD+t5IRk/IlleYKnCR2cw/7aVHbpZTjgQa7idM2RGUp+fu7KjBQk yFUuSqNyWBug/9GwEh87ZDS6lBbTbMua5JhPDtah8DMZ0W1pnRAp/p3A4N8tcQo3bMzF mHog== X-Gm-Message-State: AC+VfDx6IfYgMGLXBZ9j8FTmkzjMbgwBdn+D+L2UrUnS3P0XxPw7/tQQ 5JXMpuu9PX53y9IlZwkf7WlQELm5Ezq6fXjO7oo= X-Received: by 2002:a17:902:b08d:b0:1ad:e099:fc04 with SMTP id p13-20020a170902b08d00b001ade099fc04mr11824726plr.38.1684161223907; Mon, 15 May 2023 07:33:43 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:43 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 34/54] tcg: Introduce tcg_target_has_memory_bswap Date: Mon, 15 May 2023 07:32:53 -0700 Message-Id: <20230515143313.734053-35-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::636; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x636.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Replace the unparameterized TCG_TARGET_HAS_MEMORY_BSWAP macro with a function with a memop argument. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/aarch64/tcg-target.h | 1 - tcg/arm/tcg-target.h | 1 - tcg/i386/tcg-target.h | 3 --- tcg/loongarch64/tcg-target.h | 2 -- tcg/mips/tcg-target.h | 2 -- tcg/ppc/tcg-target.h | 1 - tcg/riscv/tcg-target.h | 2 -- tcg/s390x/tcg-target.h | 2 -- tcg/sparc64/tcg-target.h | 1 - tcg/tcg-internal.h | 2 ++ tcg/tci/tcg-target.h | 2 -- tcg/tcg-op.c | 20 +++++++++++--------- tcg/aarch64/tcg-target.c.inc | 5 +++++ tcg/arm/tcg-target.c.inc | 5 +++++ tcg/i386/tcg-target.c.inc | 5 +++++ tcg/loongarch64/tcg-target.c.inc | 5 +++++ tcg/mips/tcg-target.c.inc | 5 +++++ tcg/ppc/tcg-target.c.inc | 5 +++++ tcg/riscv/tcg-target.c.inc | 5 +++++ tcg/s390x/tcg-target.c.inc | 5 +++++ tcg/sparc64/tcg-target.c.inc | 5 +++++ tcg/tci/tcg-target.c.inc | 5 +++++ 22 files changed, 63 insertions(+), 26 deletions(-) diff --git a/tcg/aarch64/tcg-target.h b/tcg/aarch64/tcg-target.h index 3c0b0d312d..378e01d9d8 100644 --- a/tcg/aarch64/tcg-target.h +++ b/tcg/aarch64/tcg-target.h @@ -154,7 +154,6 @@ extern bool have_lse2; #define TCG_TARGET_HAS_cmpsel_vec 0 #define TCG_TARGET_DEFAULT_MO (0) -#define TCG_TARGET_HAS_MEMORY_BSWAP 0 #define TCG_TARGET_NEED_LDST_LABELS #define TCG_TARGET_NEED_POOL_LABELS diff --git a/tcg/arm/tcg-target.h b/tcg/arm/tcg-target.h index def2a189e6..4c2d3332d5 100644 --- a/tcg/arm/tcg-target.h +++ b/tcg/arm/tcg-target.h @@ -150,7 +150,6 @@ extern bool use_neon_instructions; #define TCG_TARGET_HAS_cmpsel_vec 0 #define TCG_TARGET_DEFAULT_MO (0) -#define TCG_TARGET_HAS_MEMORY_BSWAP 0 #define TCG_TARGET_NEED_LDST_LABELS #define TCG_TARGET_NEED_POOL_LABELS diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h index 0421776cb8..8fe6958abd 100644 --- a/tcg/i386/tcg-target.h +++ b/tcg/i386/tcg-target.h @@ -240,9 +240,6 @@ extern bool have_atomic16; #include "tcg/tcg-mo.h" #define TCG_TARGET_DEFAULT_MO (TCG_MO_ALL & ~TCG_MO_ST_LD) - -#define TCG_TARGET_HAS_MEMORY_BSWAP have_movbe - #define TCG_TARGET_NEED_LDST_LABELS #define TCG_TARGET_NEED_POOL_LABELS diff --git a/tcg/loongarch64/tcg-target.h b/tcg/loongarch64/tcg-target.h index 17b8193aa5..75c3d80ed2 100644 --- a/tcg/loongarch64/tcg-target.h +++ b/tcg/loongarch64/tcg-target.h @@ -173,6 +173,4 @@ typedef enum { #define TCG_TARGET_NEED_LDST_LABELS -#define TCG_TARGET_HAS_MEMORY_BSWAP 0 - #endif /* LOONGARCH_TCG_TARGET_H */ diff --git a/tcg/mips/tcg-target.h b/tcg/mips/tcg-target.h index 42bd7fff01..47088af9cb 100644 --- a/tcg/mips/tcg-target.h +++ b/tcg/mips/tcg-target.h @@ -205,8 +205,6 @@ extern bool use_mips32r2_instructions; #endif #define TCG_TARGET_DEFAULT_MO 0 -#define TCG_TARGET_HAS_MEMORY_BSWAP 0 - #define TCG_TARGET_NEED_LDST_LABELS #endif diff --git a/tcg/ppc/tcg-target.h b/tcg/ppc/tcg-target.h index af81c5a57f..d55f0266bb 100644 --- a/tcg/ppc/tcg-target.h +++ b/tcg/ppc/tcg-target.h @@ -179,7 +179,6 @@ extern bool have_vsx; #define TCG_TARGET_HAS_cmpsel_vec 0 #define TCG_TARGET_DEFAULT_MO (0) -#define TCG_TARGET_HAS_MEMORY_BSWAP 1 #define TCG_TARGET_NEED_LDST_LABELS #define TCG_TARGET_NEED_POOL_LABELS diff --git a/tcg/riscv/tcg-target.h b/tcg/riscv/tcg-target.h index dddf2486c1..dece3b3c27 100644 --- a/tcg/riscv/tcg-target.h +++ b/tcg/riscv/tcg-target.h @@ -168,6 +168,4 @@ typedef enum { #define TCG_TARGET_NEED_LDST_LABELS #define TCG_TARGET_NEED_POOL_LABELS -#define TCG_TARGET_HAS_MEMORY_BSWAP 0 - #endif diff --git a/tcg/s390x/tcg-target.h b/tcg/s390x/tcg-target.h index a05b473117..fe05680124 100644 --- a/tcg/s390x/tcg-target.h +++ b/tcg/s390x/tcg-target.h @@ -172,8 +172,6 @@ extern uint64_t s390_facilities[3]; #define TCG_TARGET_CALL_ARG_I128 TCG_CALL_ARG_BY_REF #define TCG_TARGET_CALL_RET_I128 TCG_CALL_RET_BY_REF -#define TCG_TARGET_HAS_MEMORY_BSWAP 1 - #define TCG_TARGET_DEFAULT_MO (TCG_MO_ALL & ~TCG_MO_ST_LD) #define TCG_TARGET_NEED_LDST_LABELS #define TCG_TARGET_NEED_POOL_LABELS diff --git a/tcg/sparc64/tcg-target.h b/tcg/sparc64/tcg-target.h index 7434cc99d4..f6cd86975a 100644 --- a/tcg/sparc64/tcg-target.h +++ b/tcg/sparc64/tcg-target.h @@ -154,7 +154,6 @@ extern bool use_vis3_instructions; #define TCG_AREG0 TCG_REG_I0 #define TCG_TARGET_DEFAULT_MO (0) -#define TCG_TARGET_HAS_MEMORY_BSWAP 1 #define TCG_TARGET_NEED_LDST_LABELS #define TCG_TARGET_NEED_POOL_LABELS diff --git a/tcg/tcg-internal.h b/tcg/tcg-internal.h index 0f1ba01a9a..67b698bd5c 100644 --- a/tcg/tcg-internal.h +++ b/tcg/tcg-internal.h @@ -126,4 +126,6 @@ static inline TCGv_i64 TCGV128_HIGH(TCGv_i128 t) return temp_tcgv_i64(tcgv_i128_temp(t) + o); } +bool tcg_target_has_memory_bswap(MemOp memop); + #endif /* TCG_INTERNAL_H */ diff --git a/tcg/tci/tcg-target.h b/tcg/tci/tcg-target.h index 7140a76a73..364012e4d2 100644 --- a/tcg/tci/tcg-target.h +++ b/tcg/tci/tcg-target.h @@ -176,6 +176,4 @@ typedef enum { We prefer consistency across hosts on this. */ #define TCG_TARGET_DEFAULT_MO (0) -#define TCG_TARGET_HAS_MEMORY_BSWAP 1 - #endif /* TCG_TARGET_H */ diff --git a/tcg/tcg-op.c b/tcg/tcg-op.c index 22481a344c..b13ded10df 100644 --- a/tcg/tcg-op.c +++ b/tcg/tcg-op.c @@ -2959,7 +2959,7 @@ void tcg_gen_qemu_ld_i32(TCGv_i32 val, TCGv addr, TCGArg idx, MemOp memop) oi = make_memop_idx(memop, idx); orig_memop = memop; - if (!TCG_TARGET_HAS_MEMORY_BSWAP && (memop & MO_BSWAP)) { + if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) { memop &= ~MO_BSWAP; /* The bswap primitive benefits from zero-extended input. */ if ((memop & MO_SSIZE) == MO_SW) { @@ -2996,7 +2996,7 @@ void tcg_gen_qemu_st_i32(TCGv_i32 val, TCGv addr, TCGArg idx, MemOp memop) memop = tcg_canonicalize_memop(memop, 0, 1); oi = make_memop_idx(memop, idx); - if (!TCG_TARGET_HAS_MEMORY_BSWAP && (memop & MO_BSWAP)) { + if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) { swap = tcg_temp_ebb_new_i32(); switch (memop & MO_SIZE) { case MO_16: @@ -3045,7 +3045,7 @@ void tcg_gen_qemu_ld_i64(TCGv_i64 val, TCGv addr, TCGArg idx, MemOp memop) oi = make_memop_idx(memop, idx); orig_memop = memop; - if (!TCG_TARGET_HAS_MEMORY_BSWAP && (memop & MO_BSWAP)) { + if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) { memop &= ~MO_BSWAP; /* The bswap primitive benefits from zero-extended input. */ if ((memop & MO_SIGN) && (memop & MO_SIZE) < MO_64) { @@ -3091,7 +3091,7 @@ void tcg_gen_qemu_st_i64(TCGv_i64 val, TCGv addr, TCGArg idx, MemOp memop) memop = tcg_canonicalize_memop(memop, 1, 1); oi = make_memop_idx(memop, idx); - if (!TCG_TARGET_HAS_MEMORY_BSWAP && (memop & MO_BSWAP)) { + if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) { swap = tcg_temp_ebb_new_i64(); switch (memop & MO_SIZE) { case MO_16: @@ -3157,11 +3157,6 @@ static void canonicalize_memop_i128_as_i64(MemOp ret[2], MemOp orig) tcg_debug_assert((orig & MO_SIZE) == MO_128); tcg_debug_assert((orig & MO_SIGN) == 0); - /* Use a memory ordering implemented by the host. */ - if (!TCG_TARGET_HAS_MEMORY_BSWAP && (orig & MO_BSWAP)) { - mop_1 &= ~MO_BSWAP; - } - /* Reduce the size to 64-bit. */ mop_1 = (mop_1 & ~MO_SIZE) | MO_64; @@ -3191,6 +3186,13 @@ static void canonicalize_memop_i128_as_i64(MemOp ret[2], MemOp orig) default: g_assert_not_reached(); } + + /* Use a memory ordering implemented by the host. */ + if ((orig & MO_BSWAP) && !tcg_target_has_memory_bswap(mop_1)) { + mop_1 &= ~MO_BSWAP; + mop_2 &= ~MO_BSWAP; + } + ret[0] = mop_1; ret[1] = mop_2; } diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc index 36d8798bca..0cc719d799 100644 --- a/tcg/aarch64/tcg-target.c.inc +++ b/tcg/aarch64/tcg-target.c.inc @@ -1595,6 +1595,11 @@ typedef struct { TCGType index_ext; } HostAddress; +bool tcg_target_has_memory_bswap(MemOp memop) +{ + return false; +} + static const TCGLdstHelperParam ldst_helper_param = { .ntmp = 1, .tmp = { TCG_REG_TMP } }; diff --git a/tcg/arm/tcg-target.c.inc b/tcg/arm/tcg-target.c.inc index eb0542f32e..e5aed03247 100644 --- a/tcg/arm/tcg-target.c.inc +++ b/tcg/arm/tcg-target.c.inc @@ -1325,6 +1325,11 @@ typedef struct { bool index_scratch; } HostAddress; +bool tcg_target_has_memory_bswap(MemOp memop) +{ + return false; +} + static TCGReg ldst_ra_gen(TCGContext *s, const TCGLabelQemuLdst *l, int arg) { /* We arrive at the slow path via "BLNE", so R14 contains l->raddr. */ diff --git a/tcg/i386/tcg-target.c.inc b/tcg/i386/tcg-target.c.inc index 21553f3c39..6d55ba5a1c 100644 --- a/tcg/i386/tcg-target.c.inc +++ b/tcg/i386/tcg-target.c.inc @@ -1776,6 +1776,11 @@ typedef struct { int seg; } HostAddress; +bool tcg_target_has_memory_bswap(MemOp memop) +{ + return have_movbe; +} + /* * Because i686 has no register parameters and because x86_64 has xchg * to handle addr/data register overlap, we have placed all input arguments diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index 7d0165349d..d26174dde5 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -828,6 +828,11 @@ typedef struct { TCGReg index; } HostAddress; +bool tcg_target_has_memory_bswap(MemOp memop) +{ + return false; +} + /* * For softmmu, perform the TLB load and compare. * For useronly, perform any required alignment tests. diff --git a/tcg/mips/tcg-target.c.inc b/tcg/mips/tcg-target.c.inc index fa0f334e8d..cd0254a0d7 100644 --- a/tcg/mips/tcg-target.c.inc +++ b/tcg/mips/tcg-target.c.inc @@ -1141,6 +1141,11 @@ typedef struct { MemOp align; } HostAddress; +bool tcg_target_has_memory_bswap(MemOp memop) +{ + return false; +} + /* * For softmmu, perform the TLB load and compare. * For useronly, perform any required alignment tests. diff --git a/tcg/ppc/tcg-target.c.inc b/tcg/ppc/tcg-target.c.inc index 218602c10c..b62a163014 100644 --- a/tcg/ppc/tcg-target.c.inc +++ b/tcg/ppc/tcg-target.c.inc @@ -2017,6 +2017,11 @@ typedef struct { TCGReg index; } HostAddress; +bool tcg_target_has_memory_bswap(MemOp memop) +{ + return true; +} + /* * For softmmu, perform the TLB load and compare. * For useronly, perform any required alignment tests. diff --git a/tcg/riscv/tcg-target.c.inc b/tcg/riscv/tcg-target.c.inc index 415e6c6e15..37870c89fc 100644 --- a/tcg/riscv/tcg-target.c.inc +++ b/tcg/riscv/tcg-target.c.inc @@ -853,6 +853,11 @@ static void tcg_out_goto(TCGContext *s, const tcg_insn_unit *target) tcg_debug_assert(ok); } +bool tcg_target_has_memory_bswap(MemOp memop) +{ + return false; +} + /* We have three temps, we might as well expose them. */ static const TCGLdstHelperParam ldst_helper_param = { .ntmp = 3, .tmp = { TCG_REG_TMP0, TCG_REG_TMP1, TCG_REG_TMP2 } diff --git a/tcg/s390x/tcg-target.c.inc b/tcg/s390x/tcg-target.c.inc index de8aed5f77..22f0206b5a 100644 --- a/tcg/s390x/tcg-target.c.inc +++ b/tcg/s390x/tcg-target.c.inc @@ -1574,6 +1574,11 @@ typedef struct { int disp; } HostAddress; +bool tcg_target_has_memory_bswap(MemOp memop) +{ + return true; +} + static void tcg_out_qemu_ld_direct(TCGContext *s, MemOp opc, TCGReg data, HostAddress h) { diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc index 0237188d65..bb23038529 100644 --- a/tcg/sparc64/tcg-target.c.inc +++ b/tcg/sparc64/tcg-target.c.inc @@ -1011,6 +1011,11 @@ typedef struct { TCGReg index; } HostAddress; +bool tcg_target_has_memory_bswap(MemOp memop) +{ + return true; +} + /* * For softmmu, perform the TLB load and compare. * For useronly, perform any required alignment tests. diff --git a/tcg/tci/tcg-target.c.inc b/tcg/tci/tcg-target.c.inc index 4cf03a579c..41fbf042da 100644 --- a/tcg/tci/tcg-target.c.inc +++ b/tcg/tci/tcg-target.c.inc @@ -963,3 +963,8 @@ static void tcg_target_init(TCGContext *s) static inline void tcg_target_qemu_prologue(TCGContext *s) { } + +bool tcg_target_has_memory_bswap(MemOp memop) +{ + return true; +} From patchwork Mon May 15 14:32:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681943 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1149561wrr; Mon, 15 May 2023 07:37:35 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6AUKoUzYMQqQ12hs7QIvVIc9YOquRxGvsaFfCK9ITZq5kyY1fCmQ8b58z1TP/omzyHac+s X-Received: by 2002:adf:df0f:0:b0:306:4023:359a with SMTP id y15-20020adfdf0f000000b003064023359amr29306725wrl.9.1684161454724; Mon, 15 May 2023 07:37:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161454; cv=none; d=google.com; s=arc-20160816; b=vOfG0wfY44H8nIAhKY3s0PODIyX3xmJRRQMOpn9Xc0nOoi3goZUj5lU1q4WSJF2VhB A+8oDGS3lbbxKEpMO0hkTJiq3buDknIwlOiQwarD+iyFD5loxruJhI70HqUuCBIMEqwP XpxoHC6rmzKcQybSIKK65bz6EtYTkXdXNazUltZiahvVrIPMiW5NyLnZB81i1T99LtDv DthvybLuS0A33E8yjhsFVKlpt7iSDAblE8zkcJbUqFfS+cvAnj8z0yoJvGlMGSoWMZMF QouQu/IxdJffsOe9iqjAJtVfwiufEL9vtxU0wpNI8WkQoT6qLKz0hh7qixbM0gxDnmEE ny3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=4WK5/owdhFhhV3tBupHbnhKWf3MdxcfPHINI/58Czec=; b=b64E9/mVTsq3n5wH0o0DeRBxO4+mC9HAW8eWU8DfLNRPgr4bMZxw5yXoWsTuVFPNpI qjOMphaE1vuhLj1fsJRftbKRX5llUaTBEJfYLlzCEaVxhCGa88hSIhKbg7Lxdr3SWBWx /Scl7YpkMpAOcYkpSYO2LzvbZEtp8/tVrvZeJloqNJtIx+ZXDixTlqLj7k+bp2Lzubc3 grVLNY/Mqno9iUQ0dPUyJ4wZNRdDiHDmhxR6rLzzJBZssvSlAV0Ih4VtEg8iWKruxUnH gE1PJa4Fqt54YNXMNflfykAbDA8UL5VsZ5uP3k+i9uT+vfvemApuOLD6bXvhDOE31mQe iksA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=gOm+YL7n; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id y7-20020a5d6207000000b003062bdecebcsi14958593wru.245.2023.05.15.07.37.34 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:37:34 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=gOm+YL7n; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZJ7-0002bR-7C; Mon, 15 May 2023 10:36:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHY-000712-FK for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:38 -0400 Received: from mail-pl1-x62b.google.com ([2607:f8b0:4864:20::62b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZHS-0004rx-JB for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:33 -0400 Received: by mail-pl1-x62b.google.com with SMTP id d9443c01a7336-1aaea3909d1so120072335ad.2 for ; Mon, 15 May 2023 07:33:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161225; x=1686753225; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4WK5/owdhFhhV3tBupHbnhKWf3MdxcfPHINI/58Czec=; b=gOm+YL7nEWxfrAk7+EGGPth9b+zz7xZLT2wZJVi0KK7cjsIhrzJMvIU+OpKSbJfbpt acrMNEz42i3wi5OTNSwxsFgEb4CA100Yp4BNQxKvH/QXynxPndpthZZP+0kV7ISmz/C4 Cf5PUWlp4E+R9i5PTktuxz5WksrlU14ZPnLD4YHmqVpK0eIN25hiFe9STGWbgjhH1Ev6 cURAE5AykFyPkJ1kHYglK/LaeJj9VKqvoGfx+aIdcMDDedwEIBke7t85LNmul2sBt2op 4N4d07je1sbpyVZ9CAAcWXePOL8pz83LzSugOKA/6nB4SbpaWPLYSMrf1iAHPOKctYuH eyyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161225; x=1686753225; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4WK5/owdhFhhV3tBupHbnhKWf3MdxcfPHINI/58Czec=; b=ahed0vTaGRHvzlCZX19ng20C7EdMEJHquYLdsG4oTn2pILUJaPNiRUl/2K/4jb1PBL DLjfJ/NtFaQcM2dsMBYy1GL93YCeX59iTHp/mcKnFXceXIa6Hj3B2guq9g4sqMGBoGJI pYYVfMEuBnvAKkiHO/qmz7JjWldcZUScKBUgEnTPnEn8a6Jtz9a/YKCZLMuA239a37zo iRpDpDxkEPlanHZgIRwpzkz8JtetUzh8ZfErbBR5fyT/hqIiyNChLGY88oK1Jy7PBAm0 w07Z+B+rHaweEstfr7Pk0mILu+TygN6xBhmqMC/Ilv/Mgt8nXqze8uBFuaOOMnyYJ3Fc ARIQ== X-Gm-Message-State: AC+VfDzs4xamsBFPGIwPmt/fdMJbvVRIZNPZdLxse6fCyFfdujkVci0A gP+fb0xRYLWdnL5WIaBQ0mZKhtSUtHW1+1vkHLg= X-Received: by 2002:a17:902:e5c1:b0:1ab:1bb0:126d with SMTP id u1-20020a170902e5c100b001ab1bb0126dmr46244986plf.23.1684161224683; Mon, 15 May 2023 07:33:44 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:44 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell , =?utf-8?q?Philippe_Mathieu-Daud?= =?utf-8?q?=C3=A9?= Subject: [PATCH v5 35/54] tcg: Add INDEX_op_qemu_{ld,st}_i128 Date: Mon, 15 May 2023 07:32:54 -0700 Message-Id: <20230515143313.734053-36-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62b; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Add opcodes for backend support for 128-bit memory operations. Reviewed-by: Peter Maydell Reviewed-by: Philippe Mathieu-Daudé Signed-off-by: Richard Henderson --- docs/devel/tcg-ops.rst | 11 +++--- include/tcg/tcg-opc.h | 8 +++++ tcg/aarch64/tcg-target.h | 2 ++ tcg/arm/tcg-target.h | 2 ++ tcg/i386/tcg-target.h | 2 ++ tcg/loongarch64/tcg-target.h | 1 + tcg/mips/tcg-target.h | 2 ++ tcg/ppc/tcg-target.h | 2 ++ tcg/riscv/tcg-target.h | 2 ++ tcg/s390x/tcg-target.h | 2 ++ tcg/sparc64/tcg-target.h | 2 ++ tcg/tci/tcg-target.h | 2 ++ tcg/tcg-op.c | 69 ++++++++++++++++++++++++++++++++---- tcg/tcg.c | 4 +++ 14 files changed, 101 insertions(+), 10 deletions(-) diff --git a/docs/devel/tcg-ops.rst b/docs/devel/tcg-ops.rst index f3f451b77f..6a166c5665 100644 --- a/docs/devel/tcg-ops.rst +++ b/docs/devel/tcg-ops.rst @@ -672,19 +672,20 @@ QEMU specific operations | This operation is optional. If the TCG backend does not implement the goto_ptr opcode, emitting this op is equivalent to emitting exit_tb(0). - * - qemu_ld_i32/i64 *t0*, *t1*, *flags*, *memidx* + * - qemu_ld_i32/i64/i128 *t0*, *t1*, *flags*, *memidx* - qemu_st_i32/i64 *t0*, *t1*, *flags*, *memidx* + qemu_st_i32/i64/i128 *t0*, *t1*, *flags*, *memidx* qemu_st8_i32 *t0*, *t1*, *flags*, *memidx* - | Load data at the guest address *t1* into *t0*, or store data in *t0* at guest - address *t1*. The _i32/_i64 size applies to the size of the input/output + address *t1*. The _i32/_i64/_i128 size applies to the size of the input/output register *t0* only. The address *t1* is always sized according to the guest, and the width of the memory operation is controlled by *flags*. | | Both *t0* and *t1* may be split into little-endian ordered pairs of registers - if dealing with 64-bit quantities on a 32-bit host. + if dealing with 64-bit quantities on a 32-bit host, or 128-bit quantities on + a 64-bit host. | | The *memidx* selects the qemu tlb index to use (e.g. user or kernel access). The flags are the MemOp bits, selecting the sign, width, and endianness @@ -693,6 +694,8 @@ QEMU specific operations | For a 32-bit host, qemu_ld/st_i64 is guaranteed to only be used with a 64-bit memory access specified in *flags*. | + | For qemu_ld/st_i128, these are only supported for a 64-bit host. + | | For i386, qemu_st8_i32 is exactly like qemu_st_i32, except the size of the memory operation is known to be 8-bit. This allows the backend to provide a different set of register constraints. diff --git a/include/tcg/tcg-opc.h b/include/tcg/tcg-opc.h index dd444734d9..94cf7c5d6a 100644 --- a/include/tcg/tcg-opc.h +++ b/include/tcg/tcg-opc.h @@ -213,6 +213,14 @@ DEF(qemu_st8_i32, 0, TLADDR_ARGS + 1, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | IMPL(TCG_TARGET_HAS_qemu_st8_i32)) +/* Only for 64-bit hosts at the moment. */ +DEF(qemu_ld_i128, 2, 1, 1, + TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_64BIT | + IMPL(TCG_TARGET_HAS_qemu_ldst_i128)) +DEF(qemu_st_i128, 0, 3, 1, + TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_64BIT | + IMPL(TCG_TARGET_HAS_qemu_ldst_i128)) + /* Host vector support. */ #define IMPLVEC TCG_OPF_VECTOR | IMPL(TCG_TARGET_MAYBE_vec) diff --git a/tcg/aarch64/tcg-target.h b/tcg/aarch64/tcg-target.h index 378e01d9d8..74ee2ed255 100644 --- a/tcg/aarch64/tcg-target.h +++ b/tcg/aarch64/tcg-target.h @@ -129,6 +129,8 @@ extern bool have_lse2; #define TCG_TARGET_HAS_muluh_i64 1 #define TCG_TARGET_HAS_mulsh_i64 1 +#define TCG_TARGET_HAS_qemu_ldst_i128 0 + #define TCG_TARGET_HAS_v64 1 #define TCG_TARGET_HAS_v128 1 #define TCG_TARGET_HAS_v256 0 diff --git a/tcg/arm/tcg-target.h b/tcg/arm/tcg-target.h index 4c2d3332d5..65efc538f4 100644 --- a/tcg/arm/tcg-target.h +++ b/tcg/arm/tcg-target.h @@ -125,6 +125,8 @@ extern bool use_neon_instructions; #define TCG_TARGET_HAS_rem_i32 0 #define TCG_TARGET_HAS_qemu_st8_i32 0 +#define TCG_TARGET_HAS_qemu_ldst_i128 0 + #define TCG_TARGET_HAS_v64 use_neon_instructions #define TCG_TARGET_HAS_v128 use_neon_instructions #define TCG_TARGET_HAS_v256 0 diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h index 8fe6958abd..943af6775e 100644 --- a/tcg/i386/tcg-target.h +++ b/tcg/i386/tcg-target.h @@ -194,6 +194,8 @@ extern bool have_atomic16; #define TCG_TARGET_HAS_qemu_st8_i32 1 #endif +#define TCG_TARGET_HAS_qemu_ldst_i128 0 + /* We do not support older SSE systems, only beginning with AVX1. */ #define TCG_TARGET_HAS_v64 have_avx1 #define TCG_TARGET_HAS_v128 have_avx1 diff --git a/tcg/loongarch64/tcg-target.h b/tcg/loongarch64/tcg-target.h index 75c3d80ed2..482901ac15 100644 --- a/tcg/loongarch64/tcg-target.h +++ b/tcg/loongarch64/tcg-target.h @@ -168,6 +168,7 @@ typedef enum { #define TCG_TARGET_HAS_muls2_i64 0 #define TCG_TARGET_HAS_muluh_i64 1 #define TCG_TARGET_HAS_mulsh_i64 1 +#define TCG_TARGET_HAS_qemu_ldst_i128 0 #define TCG_TARGET_DEFAULT_MO (0) diff --git a/tcg/mips/tcg-target.h b/tcg/mips/tcg-target.h index 47088af9cb..7277a117ef 100644 --- a/tcg/mips/tcg-target.h +++ b/tcg/mips/tcg-target.h @@ -204,6 +204,8 @@ extern bool use_mips32r2_instructions; #define TCG_TARGET_HAS_ext16u_i64 0 /* andi rt, rs, 0xffff */ #endif +#define TCG_TARGET_HAS_qemu_ldst_i128 0 + #define TCG_TARGET_DEFAULT_MO 0 #define TCG_TARGET_NEED_LDST_LABELS diff --git a/tcg/ppc/tcg-target.h b/tcg/ppc/tcg-target.h index d55f0266bb..0914380bd7 100644 --- a/tcg/ppc/tcg-target.h +++ b/tcg/ppc/tcg-target.h @@ -149,6 +149,8 @@ extern bool have_vsx; #define TCG_TARGET_HAS_mulsh_i64 1 #endif +#define TCG_TARGET_HAS_qemu_ldst_i128 0 + /* * While technically Altivec could support V64, it has no 64-bit store * instruction and substituting two 32-bit stores makes the generated diff --git a/tcg/riscv/tcg-target.h b/tcg/riscv/tcg-target.h index dece3b3c27..494c986b49 100644 --- a/tcg/riscv/tcg-target.h +++ b/tcg/riscv/tcg-target.h @@ -163,6 +163,8 @@ typedef enum { #define TCG_TARGET_HAS_muluh_i64 1 #define TCG_TARGET_HAS_mulsh_i64 1 +#define TCG_TARGET_HAS_qemu_ldst_i128 0 + #define TCG_TARGET_DEFAULT_MO (0) #define TCG_TARGET_NEED_LDST_LABELS diff --git a/tcg/s390x/tcg-target.h b/tcg/s390x/tcg-target.h index fe05680124..170007bea5 100644 --- a/tcg/s390x/tcg-target.h +++ b/tcg/s390x/tcg-target.h @@ -140,6 +140,8 @@ extern uint64_t s390_facilities[3]; #define TCG_TARGET_HAS_muluh_i64 0 #define TCG_TARGET_HAS_mulsh_i64 0 +#define TCG_TARGET_HAS_qemu_ldst_i128 0 + #define TCG_TARGET_HAS_v64 HAVE_FACILITY(VECTOR) #define TCG_TARGET_HAS_v128 HAVE_FACILITY(VECTOR) #define TCG_TARGET_HAS_v256 0 diff --git a/tcg/sparc64/tcg-target.h b/tcg/sparc64/tcg-target.h index f6cd86975a..31c5537379 100644 --- a/tcg/sparc64/tcg-target.h +++ b/tcg/sparc64/tcg-target.h @@ -151,6 +151,8 @@ extern bool use_vis3_instructions; #define TCG_TARGET_HAS_muluh_i64 use_vis3_instructions #define TCG_TARGET_HAS_mulsh_i64 0 +#define TCG_TARGET_HAS_qemu_ldst_i128 0 + #define TCG_AREG0 TCG_REG_I0 #define TCG_TARGET_DEFAULT_MO (0) diff --git a/tcg/tci/tcg-target.h b/tcg/tci/tcg-target.h index 364012e4d2..28dc6d5cfc 100644 --- a/tcg/tci/tcg-target.h +++ b/tcg/tci/tcg-target.h @@ -127,6 +127,8 @@ #define TCG_TARGET_HAS_mulu2_i32 1 #endif /* TCG_TARGET_REG_BITS == 64 */ +#define TCG_TARGET_HAS_qemu_ldst_i128 0 + /* Number of registers available. */ #define TCG_TARGET_NB_REGS 16 diff --git a/tcg/tcg-op.c b/tcg/tcg-op.c index b13ded10df..c419228cc4 100644 --- a/tcg/tcg-op.c +++ b/tcg/tcg-op.c @@ -3205,7 +3205,7 @@ static void canonicalize_memop_i128_as_i64(MemOp ret[2], MemOp orig) void tcg_gen_qemu_ld_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop) { - MemOpIdx oi = make_memop_idx(memop, idx); + const MemOpIdx oi = make_memop_idx(memop, idx); tcg_debug_assert((memop & MO_SIZE) == MO_128); tcg_debug_assert((memop & MO_SIGN) == 0); @@ -3213,9 +3213,36 @@ void tcg_gen_qemu_ld_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop) tcg_gen_req_mo(TCG_MO_LD_LD | TCG_MO_ST_LD); addr = plugin_prep_mem_callbacks(addr); - /* TODO: allow the tcg backend to see the whole operation. */ + /* TODO: For now, force 32-bit hosts to use the helper. */ + if (TCG_TARGET_HAS_qemu_ldst_i128 && TCG_TARGET_REG_BITS == 64) { + TCGv_i64 lo, hi; + TCGArg addr_arg; + MemOpIdx adj_oi; + bool need_bswap = false; - if (use_two_i64_for_i128(memop)) { + if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) { + lo = TCGV128_HIGH(val); + hi = TCGV128_LOW(val); + adj_oi = make_memop_idx(memop & ~MO_BSWAP, idx); + need_bswap = true; + } else { + lo = TCGV128_LOW(val); + hi = TCGV128_HIGH(val); + adj_oi = oi; + } + +#if TARGET_LONG_BITS == 32 + addr_arg = tcgv_i32_arg(addr); +#else + addr_arg = tcgv_i64_arg(addr); +#endif + tcg_gen_op4ii_i64(INDEX_op_qemu_ld_i128, lo, hi, addr_arg, adj_oi); + + if (need_bswap) { + tcg_gen_bswap64_i64(lo, lo); + tcg_gen_bswap64_i64(hi, hi); + } + } else if (use_two_i64_for_i128(memop)) { MemOp mop[2]; TCGv addr_p8; TCGv_i64 x, y; @@ -3258,7 +3285,7 @@ void tcg_gen_qemu_ld_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop) void tcg_gen_qemu_st_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop) { - MemOpIdx oi = make_memop_idx(memop, idx); + const MemOpIdx oi = make_memop_idx(memop, idx); tcg_debug_assert((memop & MO_SIZE) == MO_128); tcg_debug_assert((memop & MO_SIGN) == 0); @@ -3266,9 +3293,39 @@ void tcg_gen_qemu_st_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop) tcg_gen_req_mo(TCG_MO_ST_LD | TCG_MO_ST_ST); addr = plugin_prep_mem_callbacks(addr); - /* TODO: allow the tcg backend to see the whole operation. */ + /* TODO: For now, force 32-bit hosts to use the helper. */ - if (use_two_i64_for_i128(memop)) { + if (TCG_TARGET_HAS_qemu_ldst_i128 && TCG_TARGET_REG_BITS == 64) { + TCGv_i64 lo, hi; + TCGArg addr_arg; + MemOpIdx adj_oi; + bool need_bswap = false; + + if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) { + lo = tcg_temp_new_i64(); + hi = tcg_temp_new_i64(); + tcg_gen_bswap64_i64(lo, TCGV128_HIGH(val)); + tcg_gen_bswap64_i64(hi, TCGV128_LOW(val)); + adj_oi = make_memop_idx(memop & ~MO_BSWAP, idx); + need_bswap = true; + } else { + lo = TCGV128_LOW(val); + hi = TCGV128_HIGH(val); + adj_oi = oi; + } + +#if TARGET_LONG_BITS == 32 + addr_arg = tcgv_i32_arg(addr); +#else + addr_arg = tcgv_i64_arg(addr); +#endif + tcg_gen_op4ii_i64(INDEX_op_qemu_st_i128, lo, hi, addr_arg, adj_oi); + + if (need_bswap) { + tcg_temp_free_i64(lo); + tcg_temp_free_i64(hi); + } + } else if (use_two_i64_for_i128(memop)) { MemOp mop[2]; TCGv addr_p8; TCGv_i64 x, y; diff --git a/tcg/tcg.c b/tcg/tcg.c index dc1c0fdf2b..aa0a6c3763 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -1735,6 +1735,10 @@ bool tcg_op_supported(TCGOpcode op) case INDEX_op_qemu_st8_i32: return TCG_TARGET_HAS_qemu_st8_i32; + case INDEX_op_qemu_ld_i128: + case INDEX_op_qemu_st_i128: + return TCG_TARGET_HAS_qemu_ldst_i128; + case INDEX_op_mov_i32: case INDEX_op_setcond_i32: case INDEX_op_brcond_i32: From patchwork Mon May 15 14:32:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681960 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1152151wrr; Mon, 15 May 2023 07:43:12 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ57KPLdy9mLYYtx2+fWMGlQNOHmRDW/2RRco8jl+pimXRSnqT+vaYug8dpfWZvaSSJHs0lp X-Received: by 2002:a05:6000:104b:b0:2e4:eebe:aee3 with SMTP id c11-20020a056000104b00b002e4eebeaee3mr21690823wrx.60.1684161792168; Mon, 15 May 2023 07:43:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161792; cv=none; d=google.com; s=arc-20160816; b=RyhLJqII4sesVmS7TVVXMbHaV80fZ+a3NGXE/TehrV6t8DtZ9Z3LLr0Tu83uj8sRSX ISIkCi9RebTQB43Eaqa5sXCyPb/dVoVV1Y/pFTidUI1Rp2Y7n99kwLoK+rhNvVXK+2dF kC/Qs1ifkXIqteQu2zFgqdt1zCWVgLqMu0DQInpmSe5y+l3V6XA5ifrvR857GC9Wy+tO sj672ixXjcHEkq+uWuBUrZg8T5l4lp9X5hIIu3IjgVoYgq/c5gjqqAMMqbRJF51ZDgg4 MCRyRDJv1OyoTD2CbXz60iEbssP7qzSAXqfkfp4gDdRND4kj98H+9tJQHOMrTzuVev6Z wzcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=OL2XP1Zb93dTXqFgRsxjJukwO3HhoSqGjshmJLLIfT8=; b=QAv64O/oo42jW3TsRqjFtWbb50UmJt2tPz5elCi9bJVGgFvvJhfEDPZmC0KwnqIx3i jybBkZSx0TKD33FbB7wIr3PLjdv6cyxuBUZX7uNrJ/eEqLU/Sig7hbJoFZUADdruJ3ha V8d+xicaP7luAn9ZULnuHiN/HRfdO9Qt0oOdTT/Jj7AHf6wa3qgfy57nRG5mlc+6PNAQ 8gaZndByuB4BvarJVcD/ltxVPI0VxMnn1TvLPrZUJxc348+ALzA6/puOw3nrEWe7QelG GWPpo3zY3gBJf8WySW962S1VGsvBkOVbzVI0G763cyB5KsKRuejtKOhDujBugpYn0rxz M9sg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="cW31/wa8"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id l3-20020adfe9c3000000b0030648993001si9439wrn.104.2023.05.15.07.43.12 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:43:12 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="cW31/wa8"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIw-00020e-EO; Mon, 15 May 2023 10:36:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHh-0007AI-U5 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:47 -0400 Received: from mail-pl1-x62c.google.com ([2607:f8b0:4864:20::62c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZHc-00053H-6P for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:45 -0400 Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-1aafa03f541so124390515ad.0 for ; Mon, 15 May 2023 07:33:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161225; x=1686753225; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OL2XP1Zb93dTXqFgRsxjJukwO3HhoSqGjshmJLLIfT8=; b=cW31/wa8O7gdur2bQrthPNfiw1vDI1ei6DdH3XU5CIRFHuNlMCoEt3lA6NLG5wgLXE PqGzocd3VhnlLedTsaFWmnLPt2a0Y6MuGzN7nk2meDGIvg3XHJyidACNmy/2/ilUHYCX l0F1+hs2Fc7GF3Mg6pDGx7jR5tSnWexI+FW6W8gtC2m9uBuWssp3RLBKSedPLc6l934a /3daWL4d1qciIV26Lf3yttNkT1gNV5y/iopBV4S5Ny7hldhJIUdBPG4v1E9BcsW2rV29 8sirB/vSanMSB18TMSSbj64RukzOzKQzTR0AwpRAA7F9caqu3OitgZ9TdWsw3OzE7/jl Q/Nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161225; x=1686753225; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OL2XP1Zb93dTXqFgRsxjJukwO3HhoSqGjshmJLLIfT8=; b=L+52U2DLSAAX2wNU4+i2wrt5MsFImHsJUDWsQ7RRdn/2CwAGS3Q36njcpYVLdvmCrz ciWcmb/bKt7HpFsFw/6vxLUeIktT5ofOViuVOzKhH1w1j/LeubsYtuJbjEEAxE67NAFI JhHNspLIbfmQm+MrrX3fW6/uAnxLdt183qGjhiL51NwOJN9xSlUTZh6Ywx1PfA1XrIRC M30dezutX3ykpx+nK5btDdHhcg5EhaNMGKT9owMNcRKGAnUXsGiUuL7nlXaSbMnLA0JA 4/hrTxRsEVP47BVaUF3Q3I9BVCt9IZNi66fGerksHoowCuSb66YD1GJtfbRcxy71booc BnXA== X-Gm-Message-State: AC+VfDwh3Acw6wc9rD1LtJdYe4xiJGOTmCXHCpNAcTEe19nNF6ZD0sLe xiniKnk62aDHn/RA11V44Lw4prKWj4TPOo+JuwM= X-Received: by 2002:a17:902:b08d:b0:1ad:e099:fc04 with SMTP id p13-20020a170902b08d00b001ade099fc04mr11824812plr.38.1684161225613; Mon, 15 May 2023 07:33:45 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:45 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 36/54] tcg: Introduce tcg_out_movext3 Date: Mon, 15 May 2023 07:32:55 -0700 Message-Id: <20230515143313.734053-37-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62c; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62c.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org With x86_64 as host, we do not have any temporaries with which to resolve cycles, but we do have xchg. As a side bonus, the set of graphs that can be made with 3 nodes and all nodes conflicting is small: two. We can solve the cycle with a single temp. This is required for x86_64 to handle stores of i128: 1 address register and 2 data registers. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- tcg/tcg.c | 138 ++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 107 insertions(+), 31 deletions(-) diff --git a/tcg/tcg.c b/tcg/tcg.c index aa0a6c3763..8688248284 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -532,6 +532,82 @@ static void tcg_out_movext2(TCGContext *s, const TCGMovExtend *i1, tcg_out_movext1_new_src(s, i1, src1); } +/** + * tcg_out_movext3 -- move and extend three pair + * @s: tcg context + * @i1: first move description + * @i2: second move description + * @i3: third move description + * @scratch: temporary register, or -1 for none + * + * As tcg_out_movext, for all of @i1, @i2 and @i3, caring for overlap + * between the sources and destinations. + */ + +static void tcg_out_movext3(TCGContext *s, const TCGMovExtend *i1, + const TCGMovExtend *i2, const TCGMovExtend *i3, + int scratch) +{ + TCGReg src1 = i1->src; + TCGReg src2 = i2->src; + TCGReg src3 = i3->src; + + if (i1->dst != src2 && i1->dst != src3) { + tcg_out_movext1(s, i1); + tcg_out_movext2(s, i2, i3, scratch); + return; + } + if (i2->dst != src1 && i2->dst != src3) { + tcg_out_movext1(s, i2); + tcg_out_movext2(s, i1, i3, scratch); + return; + } + if (i3->dst != src1 && i3->dst != src2) { + tcg_out_movext1(s, i3); + tcg_out_movext2(s, i1, i2, scratch); + return; + } + + /* + * There is a cycle. Since there are only 3 nodes, the cycle is + * either "clockwise" or "anti-clockwise", and can be solved with + * a single scratch or two xchg. + */ + if (i1->dst == src2 && i2->dst == src3 && i3->dst == src1) { + /* "Clockwise" */ + if (tcg_out_xchg(s, MAX(i1->src_type, i2->src_type), src1, src2)) { + tcg_out_xchg(s, MAX(i2->src_type, i3->src_type), src2, src3); + /* The data is now in the correct registers, now extend. */ + tcg_out_movext1_new_src(s, i1, i1->dst); + tcg_out_movext1_new_src(s, i2, i2->dst); + tcg_out_movext1_new_src(s, i3, i3->dst); + } else { + tcg_debug_assert(scratch >= 0); + tcg_out_mov(s, i1->src_type, scratch, src1); + tcg_out_movext1(s, i3); + tcg_out_movext1(s, i2); + tcg_out_movext1_new_src(s, i1, scratch); + } + } else if (i1->dst == src3 && i2->dst == src1 && i3->dst == src2) { + /* "Anti-clockwise" */ + if (tcg_out_xchg(s, MAX(i2->src_type, i3->src_type), src2, src3)) { + tcg_out_xchg(s, MAX(i1->src_type, i2->src_type), src1, src2); + /* The data is now in the correct registers, now extend. */ + tcg_out_movext1_new_src(s, i1, i1->dst); + tcg_out_movext1_new_src(s, i2, i2->dst); + tcg_out_movext1_new_src(s, i3, i3->dst); + } else { + tcg_debug_assert(scratch >= 0); + tcg_out_mov(s, i1->src_type, scratch, src1); + tcg_out_movext1(s, i2); + tcg_out_movext1(s, i3); + tcg_out_movext1_new_src(s, i1, scratch); + } + } else { + g_assert_not_reached(); + } +} + #define C_PFX1(P, A) P##A #define C_PFX2(P, A, B) P##A##_##B #define C_PFX3(P, A, B, C) P##A##_##B##_##C @@ -5149,46 +5225,46 @@ static int tcg_out_helper_stk_ofs(TCGType type, unsigned slot) static void tcg_out_helper_load_regs(TCGContext *s, unsigned nmov, TCGMovExtend *mov, - unsigned ntmp, const int *tmp) + const TCGLdstHelperParam *parm) { + TCGReg dst3; + switch (nmov) { - default: + case 4: /* The backend must have provided enough temps for the worst case. */ - tcg_debug_assert(ntmp + 1 >= nmov); + tcg_debug_assert(parm->ntmp >= 2); - for (unsigned i = nmov - 1; i >= 2; --i) { - TCGReg dst = mov[i].dst; - - for (unsigned j = 0; j < i; ++j) { - if (dst == mov[j].src) { - /* - * Conflict. - * Copy the source to a temporary, recurse for the - * remaining moves, perform the extension from our - * scratch on the way out. - */ - TCGReg scratch = tmp[--ntmp]; - tcg_out_mov(s, mov[i].src_type, scratch, mov[i].src); - mov[i].src = scratch; - - tcg_out_helper_load_regs(s, i, mov, ntmp, tmp); - tcg_out_movext1(s, &mov[i]); - return; - } + dst3 = mov[3].dst; + for (unsigned j = 0; j < 3; ++j) { + if (dst3 == mov[j].src) { + /* + * Conflict. Copy the source to a temporary, perform the + * remaining moves, then the extension from our scratch + * on the way out. + */ + TCGReg scratch = parm->tmp[1]; + tcg_out_movext3(s, mov, mov + 1, mov + 2, parm->tmp[0]); + tcg_out_movext1_new_src(s, &mov[3], scratch); + break; } - - /* No conflicts: perform this move and continue. */ - tcg_out_movext1(s, &mov[i]); } - /* fall through for the final two moves */ + /* No conflicts: perform this move and continue. */ + tcg_out_movext1(s, &mov[3]); + /* fall through */ + + case 3: + tcg_out_movext3(s, mov, mov + 1, mov + 2, + parm->ntmp ? parm->tmp[0] : -1); + break; case 2: - tcg_out_movext2(s, mov, mov + 1, ntmp ? tmp[0] : -1); - return; + tcg_out_movext2(s, mov, mov + 1, + parm->ntmp ? parm->tmp[0] : -1); + break; case 1: tcg_out_movext1(s, mov); - return; - case 0: + break; + default: g_assert_not_reached(); } } @@ -5235,7 +5311,7 @@ static void tcg_out_helper_load_slots(TCGContext *s, for (i = 0; i < nmov; ++i) { mov[i].dst = tcg_target_call_iarg_regs[mov[i].dst]; } - tcg_out_helper_load_regs(s, nmov, mov, parm->ntmp, parm->tmp); + tcg_out_helper_load_regs(s, nmov, mov, parm); } static void tcg_out_helper_load_imm(TCGContext *s, unsigned slot, From patchwork Mon May 15 14:32:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681950 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1150138wrr; Mon, 15 May 2023 07:38:48 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5TGGj7GqstUuvLgXOZQgdm/QTRE/cpsOlYetb/F2vST3K9HHKspdeVeYVITH+5CH7Y2LP6 X-Received: by 2002:a5d:408f:0:b0:306:3352:5ba3 with SMTP id o15-20020a5d408f000000b0030633525ba3mr21372508wrp.25.1684161528142; Mon, 15 May 2023 07:38:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161528; cv=none; d=google.com; s=arc-20160816; b=Vop0+s0vALD9nKI2JmFFtyVKghDeAq+OjA4+svbmsKpuXhEzQRgYRozs8TAyr7bBkZ kdkadFttyHl8AFsIVXszusnYJJe6zzR6xXC5QH8phGVGcM28DqDR1HqD9GQmZdSGJ106 SKrIlEHgUetIN4R3hKqe6i9OQ3zDvXhXXih7bBaO4K9NvWAlIzcWO7aY48BtYVRZWf+F 0G2s38ORQB/lDKUrA0nBMxuSscYngElr1X5MxX9xZdaJ/WgdtLxg+8SWNjW48wxkYXii WHMT/5esBgNAYDbXQ6yPduxXiyYE2CYDP8jDdsmVA4SXZWkeR+u4bTvAsXstuPj3Adsm Tlcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=kNGmF8W7v6Sxbzk1GnBwY+ErFtp6C+F1QfXFPbibcyI=; b=qv4iiqquZL23sMGs7rDI0Jd7d3QQ7dWTt4NUhv4E5kYDjvotqBGHCwWwJo6ZZ0A6l6 zHrkSNxUYIg3zoJD/3DqvbWa9rzd8v+2qK6Gbpzoen7zmD/HogQRe41ObBiVZd0BNbng /Klr4UfQpxVyjetmhuvNLmZqwc/BdEOuNoES2DK0r8OdMLeXeOlG8jkkJxK3yrdlvsUv PNCSL9Nu7UGpnvks0dxIZd3Zh+C32SQrrtl3Rhof/Aoa/css2zfV4SUwi/sli/1uVhqp xfET2OEYxNJFPmAt1Jq1u27vsMCsGunD0H50ekDkUZ0TCRbX8Z3JR5K2IEC5mwwj2WXA gKSQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=eQt0RcAW; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id v14-20020a5d610e000000b0030743a4776bsi4949wrt.16.2023.05.15.07.38.47 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:38:48 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=eQt0RcAW; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZJU-0003Vh-6b; Mon, 15 May 2023 10:36:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHr-0007Ic-Br for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:56 -0400 Received: from mail-pl1-x633.google.com ([2607:f8b0:4864:20::633]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZHg-00053W-MS for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:54 -0400 Received: by mail-pl1-x633.google.com with SMTP id d9443c01a7336-1aad6f2be8eso118636985ad.3 for ; Mon, 15 May 2023 07:33:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161226; x=1686753226; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kNGmF8W7v6Sxbzk1GnBwY+ErFtp6C+F1QfXFPbibcyI=; b=eQt0RcAWhzyW2Wopvg07MsRwJi1ZEuf6nPyHHDqUfCSAcNYMq2343bqW2PeRl1uKlz m8QY53+nzWBJCub/B6y2gc5xJfl773zXd1YE8mgSqhG+DmuhuMK8gSXnOjvJojvAevMp i6QUhrwpQRIFgVKtkdMpRRrMzG/ossiX/eist+TN1WV4Qn9Jl6I+FFRIvI5XFx3l5+0K q4DaEjHHok+EtMSr/WvlDbyY6JImSnh7II8Rx9/cOkdIhFTHgkA6YnNIHnRNLdhCKoWE bVuxfEb7TzQUPmbYLY+SaaZSltbXkSOQwFO+GnNAhIazJI8sXCK9EqF6rZTddth+SYXi JRyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161226; x=1686753226; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kNGmF8W7v6Sxbzk1GnBwY+ErFtp6C+F1QfXFPbibcyI=; b=aPES8ezTSHawpTKTWIKsnYPpxct/5M5dNOaASsBIbL6faFW90YT98LCODIGrxm8r4M l4+9/NqjZ38V1NRgtT5TUnDPGkj9BPn5SM3BcItZ2Hfp9lPTRYtWtyX8XoiReB4g83I4 tbCTPHRsa3CLo4i7gSJK/okA92DtsO8U3+6h6QQnPOPHzOarVAUcHidM9ZfcruBjV4Up APmNaUFXdTkIhaK+mFx+szRyb9sjM0NKp6dml2uLnpPXKUZkz4bDHBnvALRRnnuwvDNx fQXINaDP7WZyhC96hqE+eTzhtmyqSbUsyDFCGfWWa5n76JV0nVlQ4Yugc4SFfO+sgdOY IKjg== X-Gm-Message-State: AC+VfDyqMjEGcLbTe8dkwsmLnvXvYjs3pnprAd130WCtuzlYpBl7tDG1 B2whlBjdtRrJXPAQnCya1qXryoJYuXL3C8vI5Pc= X-Received: by 2002:a17:902:c94b:b0:1ab:14f2:e7e6 with SMTP id i11-20020a170902c94b00b001ab14f2e7e6mr42088614pla.65.1684161226395; Mon, 15 May 2023 07:33:46 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:46 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 37/54] tcg: Merge tcg_out_helper_load_regs into caller Date: Mon, 15 May 2023 07:32:56 -0700 Message-Id: <20230515143313.734053-38-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::633; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x633.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Now that tcg_out_helper_load_regs is not recursive, we can merge it into its only caller, tcg_out_helper_load_slots. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- tcg/tcg.c | 89 +++++++++++++++++++++++++------------------------------ 1 file changed, 41 insertions(+), 48 deletions(-) diff --git a/tcg/tcg.c b/tcg/tcg.c index 8688248284..ff7aec23e7 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -5223,12 +5223,50 @@ static int tcg_out_helper_stk_ofs(TCGType type, unsigned slot) return ofs; } -static void tcg_out_helper_load_regs(TCGContext *s, - unsigned nmov, TCGMovExtend *mov, - const TCGLdstHelperParam *parm) +static void tcg_out_helper_load_slots(TCGContext *s, + unsigned nmov, TCGMovExtend *mov, + const TCGLdstHelperParam *parm) { + unsigned i; TCGReg dst3; + /* + * Start from the end, storing to the stack first. + * This frees those registers, so we need not consider overlap. + */ + for (i = nmov; i-- > 0; ) { + unsigned slot = mov[i].dst; + + if (arg_slot_reg_p(slot)) { + goto found_reg; + } + + TCGReg src = mov[i].src; + TCGType dst_type = mov[i].dst_type; + MemOp dst_mo = dst_type == TCG_TYPE_I32 ? MO_32 : MO_64; + + /* The argument is going onto the stack; extend into scratch. */ + if ((mov[i].src_ext & MO_SIZE) != dst_mo) { + tcg_debug_assert(parm->ntmp != 0); + mov[i].dst = src = parm->tmp[0]; + tcg_out_movext1(s, &mov[i]); + } + + tcg_out_st(s, dst_type, src, TCG_REG_CALL_STACK, + tcg_out_helper_stk_ofs(dst_type, slot)); + } + return; + + found_reg: + /* + * The remaining arguments are in registers. + * Convert slot numbers to argument registers. + */ + nmov = i + 1; + for (i = 0; i < nmov; ++i) { + mov[i].dst = tcg_target_call_iarg_regs[mov[i].dst]; + } + switch (nmov) { case 4: /* The backend must have provided enough temps for the worst case. */ @@ -5269,51 +5307,6 @@ static void tcg_out_helper_load_regs(TCGContext *s, } } -static void tcg_out_helper_load_slots(TCGContext *s, - unsigned nmov, TCGMovExtend *mov, - const TCGLdstHelperParam *parm) -{ - unsigned i; - - /* - * Start from the end, storing to the stack first. - * This frees those registers, so we need not consider overlap. - */ - for (i = nmov; i-- > 0; ) { - unsigned slot = mov[i].dst; - - if (arg_slot_reg_p(slot)) { - goto found_reg; - } - - TCGReg src = mov[i].src; - TCGType dst_type = mov[i].dst_type; - MemOp dst_mo = dst_type == TCG_TYPE_I32 ? MO_32 : MO_64; - - /* The argument is going onto the stack; extend into scratch. */ - if ((mov[i].src_ext & MO_SIZE) != dst_mo) { - tcg_debug_assert(parm->ntmp != 0); - mov[i].dst = src = parm->tmp[0]; - tcg_out_movext1(s, &mov[i]); - } - - tcg_out_st(s, dst_type, src, TCG_REG_CALL_STACK, - tcg_out_helper_stk_ofs(dst_type, slot)); - } - return; - - found_reg: - /* - * The remaining arguments are in registers. - * Convert slot numbers to argument registers. - */ - nmov = i + 1; - for (i = 0; i < nmov; ++i) { - mov[i].dst = tcg_target_call_iarg_regs[mov[i].dst]; - } - tcg_out_helper_load_regs(s, nmov, mov, parm); -} - static void tcg_out_helper_load_imm(TCGContext *s, unsigned slot, TCGType type, tcg_target_long imm, const TCGLdstHelperParam *parm) From patchwork Mon May 15 14:32:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681944 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1149751wrr; Mon, 15 May 2023 07:37:59 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ46NRd8GO+b12Mx5UKY3w4sNoVMdu9Q9RiBR6WSAloL+GuY47JlOF40dPYO/iiMJdGAIqbc X-Received: by 2002:a05:600c:1149:b0:3f5:e7f:5328 with SMTP id z9-20020a05600c114900b003f50e7f5328mr1648418wmz.3.1684161478992; Mon, 15 May 2023 07:37:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161478; cv=none; d=google.com; s=arc-20160816; b=IdXOkNc1lkBGi15WuF2H+jjZNU6UZvmMeD+SQKnq8DmN841ag0WaYvrohdRMH4l29Y DG+JR2/eLFOuOCu79BFLe3gXczNgB5kD/BCTvuWYxFbRYy65A8nUt9IGZIVDYYT4yYW8 3lGumQDOeK99TtKrluLmLgskbJsnsuyvw2vXDeKZSEbkbcy5DDxw6+i2f4L7HrXOaJHV qqWTF9mOuklbJK+oeQQXG0wPlD7dseicHyNSw1Hd/uY2gi9fKhZUTFL8B4dnFxQArQCg mV4GUEav/NNb66T98oHzr6UbrvxN3A+1SZYjgot8TxYXMY7RFd+43iiRiyJ7GN0IaAWd GgiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=IAYOvh1bwAhumLvlgygLCHUMrrkGrbmEQT9VUhbIKvg=; b=mNvCkAMEdBGCw0l80OfGULoovhRRcM0FkGgyCq5k9uYYTADAxB3S70qioqrRmhWFKu uK0mglCBQUZH+3qbexzGSLf5uBiRG19fcocmvf0AHZ3gCdVVOgDWpkPTdVyx3oHV17s2 76LFOUkNvk9qCNbdbHAZB3WEJagRATXDf08kVnZvsxFuJ7ti1K/2fQwkFlaY8448bUCJ TQDZMdOwy+4cTyZwbqeEjCX+yBWT79EgjzAEC3xnrfaZzTMG+9qCh6s6hCWeGIFyHjUt hFf10DGYFHVN2sdT8/L0XriL95lcvbeB3/daqae2oNS6IWt1Rxd/8u7iwF26UUZJheXB XI3w== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=qmCiNUVp; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id a3-20020a05600c348300b003f4215d8632si14347077wmq.216.2023.05.15.07.37.58 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:37:58 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=qmCiNUVp; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIo-0001RD-JQ; Mon, 15 May 2023 10:35:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHY-00070w-Dy for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:38 -0400 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZHS-00053j-Ji for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:32 -0400 Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-1ab0c697c2bso118592985ad.1 for ; Mon, 15 May 2023 07:33:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161227; x=1686753227; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IAYOvh1bwAhumLvlgygLCHUMrrkGrbmEQT9VUhbIKvg=; b=qmCiNUVp0HA6wX2iE0+g8ygvkiJBd611/2xuWR2oPgtyU13sygWDDUWuVRsUe5af8V 63siCQJq+cROpGR9zIJHIJ0BRUoIcH5m49Ye/k0hdcMzmmi2gangqqA/9cYaH85Af8ah JZDTjyPwufJZv18Z/e67oFI1hvDmeW7fkks+3y5f7Jeg01yCXyzzq8anRyg+rvN5bJmi g1WwTMXr7SXqiJoGoaFxKnRvznPkTNVcBkpYqnGSqipzfJ5g36EGT49F95C2RmFgA+lX qDlSY033qV8yApbLoK3HetlCUKwCkc3sgO2+NBQx1dVMRs4G6yPnuk5AyYgfHfpDhevJ eLsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161227; x=1686753227; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IAYOvh1bwAhumLvlgygLCHUMrrkGrbmEQT9VUhbIKvg=; b=Cd/mELD9iteEl/4Ezd7DgCvdb7r/28tRGzSzg3ovtxmlADrwHKG7WuY/6Pw7jUWXBS /4DHNSU0X3uJN+VOXj2FpUps1LFVgC0LPgCKNYuQ5Q8LZ0j5qKdtPf/45euWI66FVcBT n9VAen9ga/+U9H0kzZ5lh+sRpUFRidJAKYAWWYFL07CFxCqbHLefJCPUcS7pBzvmbSzw YrcJJJ2RKcffzMfbUXcsQP2H3irM9A817eyQt49n892mDOB2nrYSMfg3HbE82k7pSM5i ZUi0z5tcd3syqDhym2KUKVnC+tPm8R9jmUVPZpxioQ1Pi0WxSJ6cT51BROKn1ujIuQy0 zpoQ== X-Gm-Message-State: AC+VfDwDK1DsxjlapN8mfm5nACuXIpbzNUTm/ZQnfpprqCcpKb8AdJ/F VWQmVEPsS/ZzPoraEOyT/A8IzqpFfIOPTrlCqhs= X-Received: by 2002:a17:902:8691:b0:19a:9890:eac6 with SMTP id g17-20020a170902869100b0019a9890eac6mr32126898plo.24.1684161227210; Mon, 15 May 2023 07:33:47 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:46 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 38/54] tcg: Support TCG_TYPE_I128 in tcg_out_{ld, st}_helper_{args, ret} Date: Mon, 15 May 2023 07:32:57 -0700 Message-Id: <20230515143313.734053-39-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62e; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/tcg.c | 196 +++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 163 insertions(+), 33 deletions(-) diff --git a/tcg/tcg.c b/tcg/tcg.c index ff7aec23e7..10429be039 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -206,6 +206,7 @@ static void * const qemu_ld_helpers[MO_SSIZE + 1] __attribute__((unused)) = { [MO_UQ] = helper_ldq_mmu, #if TCG_TARGET_REG_BITS == 64 [MO_SL] = helper_ldsl_mmu, + [MO_128] = helper_ld16_mmu, #endif }; @@ -214,6 +215,9 @@ static void * const qemu_st_helpers[MO_SIZE + 1] __attribute__((unused)) = { [MO_16] = helper_stw_mmu, [MO_32] = helper_stl_mmu, [MO_64] = helper_stq_mmu, +#if TCG_TARGET_REG_BITS == 64 + [MO_128] = helper_st16_mmu, +#endif }; TCGContext tcg_init_ctx; @@ -866,6 +870,15 @@ static TCGHelperInfo info_helper_ld64_mmu = { | dh_typemask(ptr, 4) /* uintptr_t ra */ }; +static TCGHelperInfo info_helper_ld128_mmu = { + .flags = TCG_CALL_NO_WG, + .typemask = dh_typemask(i128, 0) /* return Int128 */ + | dh_typemask(env, 1) + | dh_typemask(tl, 2) /* target_ulong addr */ + | dh_typemask(i32, 3) /* unsigned oi */ + | dh_typemask(ptr, 4) /* uintptr_t ra */ +}; + static TCGHelperInfo info_helper_st32_mmu = { .flags = TCG_CALL_NO_WG, .typemask = dh_typemask(void, 0) @@ -886,6 +899,16 @@ static TCGHelperInfo info_helper_st64_mmu = { | dh_typemask(ptr, 5) /* uintptr_t ra */ }; +static TCGHelperInfo info_helper_st128_mmu = { + .flags = TCG_CALL_NO_WG, + .typemask = dh_typemask(void, 0) + | dh_typemask(env, 1) + | dh_typemask(tl, 2) /* target_ulong addr */ + | dh_typemask(i128, 3) /* Int128 data */ + | dh_typemask(i32, 4) /* unsigned oi */ + | dh_typemask(ptr, 5) /* uintptr_t ra */ +}; + #ifdef CONFIG_TCG_INTERPRETER static ffi_type *typecode_to_ffi(int argmask) { @@ -1299,8 +1322,10 @@ static void tcg_context_init(unsigned max_cpus) init_call_layout(&info_helper_ld32_mmu); init_call_layout(&info_helper_ld64_mmu); + init_call_layout(&info_helper_ld128_mmu); init_call_layout(&info_helper_st32_mmu); init_call_layout(&info_helper_st64_mmu); + init_call_layout(&info_helper_st128_mmu); #ifdef CONFIG_TCG_INTERPRETER init_ffi_layouts(); @@ -5395,6 +5420,8 @@ static unsigned tcg_out_helper_add_mov(TCGMovExtend *mov, TCGType dst_type, TCGType src_type, TCGReg lo, TCGReg hi) { + MemOp reg_mo; + if (dst_type <= TCG_TYPE_REG) { MemOp src_ext; @@ -5422,19 +5449,25 @@ static unsigned tcg_out_helper_add_mov(TCGMovExtend *mov, return 1; } - assert(TCG_TARGET_REG_BITS == 32); + if (TCG_TARGET_REG_BITS == 32) { + assert(dst_type == TCG_TYPE_I64); + reg_mo = MO_32; + } else { + assert(dst_type == TCG_TYPE_I128); + reg_mo = MO_64; + } mov[0].dst = loc[HOST_BIG_ENDIAN].arg_slot; mov[0].src = lo; - mov[0].dst_type = TCG_TYPE_I32; - mov[0].src_type = TCG_TYPE_I32; - mov[0].src_ext = MO_32; + mov[0].dst_type = TCG_TYPE_REG; + mov[0].src_type = TCG_TYPE_REG; + mov[0].src_ext = reg_mo; mov[1].dst = loc[!HOST_BIG_ENDIAN].arg_slot; mov[1].src = hi; - mov[1].dst_type = TCG_TYPE_I32; - mov[1].src_type = TCG_TYPE_I32; - mov[1].src_ext = MO_32; + mov[1].dst_type = TCG_TYPE_REG; + mov[1].src_type = TCG_TYPE_REG; + mov[1].src_ext = reg_mo; return 2; } @@ -5457,6 +5490,9 @@ static void tcg_out_ld_helper_args(TCGContext *s, const TCGLabelQemuLdst *ldst, case MO_64: info = &info_helper_ld64_mmu; break; + case MO_128: + info = &info_helper_ld128_mmu; + break; default: g_assert_not_reached(); } @@ -5471,8 +5507,33 @@ static void tcg_out_ld_helper_args(TCGContext *s, const TCGLabelQemuLdst *ldst, tcg_out_helper_load_slots(s, nmov, mov, parm); - /* No special attention for 32 and 64-bit return values. */ - tcg_debug_assert(info->out_kind == TCG_CALL_RET_NORMAL); + switch (info->out_kind) { + case TCG_CALL_RET_NORMAL: + case TCG_CALL_RET_BY_VEC: + break; + case TCG_CALL_RET_BY_REF: + /* + * The return reference is in the first argument slot. + * We need memory in which to return: re-use the top of stack. + */ + { + int ofs_slot0 = TCG_TARGET_CALL_STACK_OFFSET; + + if (arg_slot_reg_p(0)) { + tcg_out_addi_ptr(s, tcg_target_call_iarg_regs[0], + TCG_REG_CALL_STACK, ofs_slot0); + } else { + tcg_debug_assert(parm->ntmp != 0); + tcg_out_addi_ptr(s, parm->tmp[0], + TCG_REG_CALL_STACK, ofs_slot0); + tcg_out_st(s, TCG_TYPE_PTR, parm->tmp[0], + TCG_REG_CALL_STACK, ofs_slot0); + } + } + break; + default: + g_assert_not_reached(); + } tcg_out_helper_load_common_args(s, ldst, parm, info, next_arg); } @@ -5481,11 +5542,18 @@ static void tcg_out_ld_helper_ret(TCGContext *s, const TCGLabelQemuLdst *ldst, bool load_sign, const TCGLdstHelperParam *parm) { + MemOp mop = get_memop(ldst->oi); TCGMovExtend mov[2]; + int ofs_slot0; - if (ldst->type <= TCG_TYPE_REG) { - MemOp mop = get_memop(ldst->oi); + switch (ldst->type) { + case TCG_TYPE_I64: + if (TCG_TARGET_REG_BITS == 32) { + break; + } + /* fall through */ + case TCG_TYPE_I32: mov[0].dst = ldst->datalo_reg; mov[0].src = tcg_target_call_oarg_reg(TCG_CALL_RET_NORMAL, 0); mov[0].dst_type = ldst->type; @@ -5511,25 +5579,49 @@ static void tcg_out_ld_helper_ret(TCGContext *s, const TCGLabelQemuLdst *ldst, mov[0].src_ext = mop & MO_SSIZE; } tcg_out_movext1(s, mov); - } else { - assert(TCG_TARGET_REG_BITS == 32); + return; - mov[0].dst = ldst->datalo_reg; - mov[0].src = - tcg_target_call_oarg_reg(TCG_CALL_RET_NORMAL, HOST_BIG_ENDIAN); - mov[0].dst_type = TCG_TYPE_I32; - mov[0].src_type = TCG_TYPE_I32; - mov[0].src_ext = MO_32; + case TCG_TYPE_I128: + tcg_debug_assert(TCG_TARGET_REG_BITS == 64); + ofs_slot0 = TCG_TARGET_CALL_STACK_OFFSET; + switch (TCG_TARGET_CALL_RET_I128) { + case TCG_CALL_RET_NORMAL: + break; + case TCG_CALL_RET_BY_VEC: + tcg_out_st(s, TCG_TYPE_V128, + tcg_target_call_oarg_reg(TCG_CALL_RET_BY_VEC, 0), + TCG_REG_CALL_STACK, ofs_slot0); + /* fall through */ + case TCG_CALL_RET_BY_REF: + tcg_out_ld(s, TCG_TYPE_I64, ldst->datalo_reg, + TCG_REG_CALL_STACK, ofs_slot0 + 8 * HOST_BIG_ENDIAN); + tcg_out_ld(s, TCG_TYPE_I64, ldst->datahi_reg, + TCG_REG_CALL_STACK, ofs_slot0 + 8 * !HOST_BIG_ENDIAN); + return; + default: + g_assert_not_reached(); + } + break; - mov[1].dst = ldst->datahi_reg; - mov[1].src = - tcg_target_call_oarg_reg(TCG_CALL_RET_NORMAL, !HOST_BIG_ENDIAN); - mov[1].dst_type = TCG_TYPE_REG; - mov[1].src_type = TCG_TYPE_REG; - mov[1].src_ext = MO_32; - - tcg_out_movext2(s, mov, mov + 1, parm->ntmp ? parm->tmp[0] : -1); + default: + g_assert_not_reached(); } + + mov[0].dst = ldst->datalo_reg; + mov[0].src = + tcg_target_call_oarg_reg(TCG_CALL_RET_NORMAL, HOST_BIG_ENDIAN); + mov[0].dst_type = TCG_TYPE_I32; + mov[0].src_type = TCG_TYPE_I32; + mov[0].src_ext = TCG_TARGET_REG_BITS == 32 ? MO_32 : MO_64; + + mov[1].dst = ldst->datahi_reg; + mov[1].src = + tcg_target_call_oarg_reg(TCG_CALL_RET_NORMAL, !HOST_BIG_ENDIAN); + mov[1].dst_type = TCG_TYPE_REG; + mov[1].src_type = TCG_TYPE_REG; + mov[1].src_ext = TCG_TARGET_REG_BITS == 32 ? MO_32 : MO_64; + + tcg_out_movext2(s, mov, mov + 1, parm->ntmp ? parm->tmp[0] : -1); } static void tcg_out_st_helper_args(TCGContext *s, const TCGLabelQemuLdst *ldst, @@ -5553,6 +5645,10 @@ static void tcg_out_st_helper_args(TCGContext *s, const TCGLabelQemuLdst *ldst, info = &info_helper_st64_mmu; data_type = TCG_TYPE_I64; break; + case MO_128: + info = &info_helper_st128_mmu; + data_type = TCG_TYPE_I128; + break; default: g_assert_not_reached(); } @@ -5570,13 +5666,47 @@ static void tcg_out_st_helper_args(TCGContext *s, const TCGLabelQemuLdst *ldst, /* Handle data argument. */ loc = &info->in[next_arg]; - n = tcg_out_helper_add_mov(mov + nmov, loc, data_type, ldst->type, - ldst->datalo_reg, ldst->datahi_reg); - next_arg += n; - nmov += n; - tcg_debug_assert(nmov <= ARRAY_SIZE(mov)); + switch (loc->kind) { + case TCG_CALL_ARG_NORMAL: + case TCG_CALL_ARG_EXTEND_U: + case TCG_CALL_ARG_EXTEND_S: + n = tcg_out_helper_add_mov(mov + nmov, loc, data_type, ldst->type, + ldst->datalo_reg, ldst->datahi_reg); + next_arg += n; + nmov += n; + tcg_out_helper_load_slots(s, nmov, mov, parm); + break; + + case TCG_CALL_ARG_BY_REF: + tcg_debug_assert(TCG_TARGET_REG_BITS == 64); + tcg_debug_assert(data_type == TCG_TYPE_I128); + tcg_out_st(s, TCG_TYPE_I64, + HOST_BIG_ENDIAN ? ldst->datahi_reg : ldst->datalo_reg, + TCG_REG_CALL_STACK, arg_slot_stk_ofs(loc[0].ref_slot)); + tcg_out_st(s, TCG_TYPE_I64, + HOST_BIG_ENDIAN ? ldst->datalo_reg : ldst->datahi_reg, + TCG_REG_CALL_STACK, arg_slot_stk_ofs(loc[1].ref_slot)); + + tcg_out_helper_load_slots(s, nmov, mov, parm); + + if (arg_slot_reg_p(loc->arg_slot)) { + tcg_out_addi_ptr(s, tcg_target_call_iarg_regs[loc->arg_slot], + TCG_REG_CALL_STACK, + arg_slot_stk_ofs(loc->ref_slot)); + } else { + tcg_debug_assert(parm->ntmp != 0); + tcg_out_addi_ptr(s, parm->tmp[0], TCG_REG_CALL_STACK, + arg_slot_stk_ofs(loc->ref_slot)); + tcg_out_st(s, TCG_TYPE_PTR, parm->tmp[0], + TCG_REG_CALL_STACK, arg_slot_stk_ofs(loc->arg_slot)); + } + next_arg += 2; + break; + + default: + g_assert_not_reached(); + } - tcg_out_helper_load_slots(s, nmov, mov, parm); tcg_out_helper_load_common_args(s, ldst, parm, info, next_arg); } From patchwork Mon May 15 14:32:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681963 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1152199wrr; Mon, 15 May 2023 07:43:18 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7tSqkwL4/S6WwhQMHFz0FUjG+3NNGYds2EK2u5me1SU9P0Eddx1MOLBLoO3teXjHbuuNgF X-Received: by 2002:adf:f04b:0:b0:2ee:da1c:381a with SMTP id t11-20020adff04b000000b002eeda1c381amr24526617wro.69.1684161797827; Mon, 15 May 2023 07:43:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161797; cv=none; d=google.com; s=arc-20160816; b=gBEwjA9noVR5xMJBbI/R42t4ef56WJcEcgdNC+vwAq9rmXO01u7p/yB7D8poKCXrvb 5ptOBH3PNpQ7n3s1Lv/RE/WMbZmL+aBzsMuwTo+e3AIu57n2Dii9a2ZvA05Z/Pgfvg4e s8a5Z4x/ntOgHumI2IXkwc8NMCoSyFTcupO0S/ME4/MSnwgx2j0DGcQppvfHskJWMKbS SQLykltMreysubP0LwX/qw4Yk9ZqWlCAzdzsTKo9iC9X8ASpnBouH1SuoqLvjGgDWABh gDSkSjFKoI9vUPBBH3fulXcR3ZzsFdnq3bddYkjCgpxTMPWuPq4tF6QGpwC3gQpvcrXh F7Gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=dp9FxODKTIpcK97erYATVmrCbIv6BXH7ZL9RlqvYq+w=; b=GqcEwOoRFd2dwwzvgPfI8zwOpJlHHz0wSw0Palo+5D3aeVPM/FYlOYffeXLdmkqjkg PHYbchHKhfTGAXLWfaCv9Nkxsuy5wv9+iGHljg7jBuwV6j5VK73A5IPpGHtCNPiNRGWT DwANL4mE/4WQFi11AGNiUcpVl1WQgL7DX5KD0Y+4qf44sV1GF/5Tt4Abh/vpmLOeY31b tOe5Lb0Ha0xhUJAmkFDcT5NRyL2lGUGOi1/nlOiDbV/oMpxjYQl8DCy4DKOPx9gUYi4p +HvYjj/Vg8UYfvq208fWb6ehaH+RVRnj8jm4gZWEXg+Lm07stuU1G94bo4kt6gwAIyDL erTQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=N1ObHclI; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id f8-20020a5d58e8000000b003078646c830si14505348wrd.991.2023.05.15.07.43.17 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:43:17 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=N1ObHclI; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZIz-0002Ad-BF; Mon, 15 May 2023 10:36:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZHY-00070s-E2 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:38 -0400 Received: from mail-pl1-x62b.google.com ([2607:f8b0:4864:20::62b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZHS-00053v-Kd for qemu-devel@nongnu.org; Mon, 15 May 2023 10:34:33 -0400 Received: by mail-pl1-x62b.google.com with SMTP id d9443c01a7336-1aaf706768cso98047235ad.0 for ; Mon, 15 May 2023 07:33:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161228; x=1686753228; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dp9FxODKTIpcK97erYATVmrCbIv6BXH7ZL9RlqvYq+w=; b=N1ObHclIYSmQj3x/Tqz0fa6BrVLfTfVc45/xtvmZ2EAv0lp5Pm5BQxEGeRSO7Y1+q+ ffWh438XbLCblxpar3Km17KOpTWVWqhGBMDZFboaPfTNWb3bUHLsqNjNIupaNLqtxwER j3i9OvT1MsL2axuQ9bqWUztSdwZfG2hEcrR8JFruNwf+83P1/LtvKfJ28xNxf/7VJVDn cqfbnsDpCkzzXyTRWFcLVtQFzz56o7Gas/xgTR+mkYEtaB0++cFhJkibs+CtYdH71d/N 60nbKvn1h7sHtJ/1qFvlvpyASBshC8Fc1qST9CfvpROtGsH2AbYuFEY16oNXscE5hs0K rRHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161228; x=1686753228; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dp9FxODKTIpcK97erYATVmrCbIv6BXH7ZL9RlqvYq+w=; b=fkfLFoU+4L/44TWoqbzbjG3dpCnioSkZ7Hf40XmU/1GTG8Z6LZiQlw7lleINIsOd+V TqysjKoIsDQMdUQ0xMmw1+6kGdIqtriETatSB30sYDrr7oGkGqKDGpivO/cMjgw00QEz iLux+P8wTd4/C4/DtCi8HmkDjiaox6Ne/3ZbR8y+hA8S9fWRLALZ2OPYPnj45+DGrspG n50D1cJJ5jcQBoPyGaDrZ7iaiXZMMc32dvKp7Mi6VnVsm0Tz9Ef6Ba1fPeFKOjArYrVh l+szgODr1I553OHkky281jUyIo2o4oLQKNb1RryTc+ZTpwrx2TKXV0s2e/Zyk6b+zMiL AYHg== X-Gm-Message-State: AC+VfDzadVRfWBFlxhclVhRgcFbj26RBVRIaXXd6wL/H/lmxaEmEcZkz 5c4ToHKGKRCAG86sl/Elucr9x5tZ2N3FZggKSAk= X-Received: by 2002:a17:902:ee89:b0:1a6:74f6:fa92 with SMTP id a9-20020a170902ee8900b001a674f6fa92mr36555098pld.19.1684161228124; Mon, 15 May 2023 07:33:48 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id ik9-20020a170902ab0900b001ab0aec388bsm13694335plb.135.2023.05.15.07.33.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:33:47 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 39/54] tcg: Introduce atom_and_align_for_opc Date: Mon, 15 May 2023 07:32:58 -0700 Message-Id: <20230515143313.734053-40-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62b; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Examine MemOp for atomicity and alignment, adjusting alignment as required to implement atomicity on the host. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- v5: Dropped r-b because of MO_ATOM_* reorg Return a struct with the result pair (phil). --- tcg/tcg.c | 95 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+) diff --git a/tcg/tcg.c b/tcg/tcg.c index 10429be039..279ba2ca6a 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -220,6 +220,15 @@ static void * const qemu_st_helpers[MO_SIZE + 1] __attribute__((unused)) = { #endif }; +typedef struct { + MemOp atom; /* lg2 bits of atomicity required */ + MemOp align; /* lg2 bits of alignment to use */ +} TCGAtomAlign; + +static TCGAtomAlign atom_and_align_for_opc(TCGContext *s, MemOp opc, + MemOp host_atom, bool allow_two_ops) + __attribute__((unused)); + TCGContext tcg_init_ctx; __thread TCGContext *tcg_ctx; @@ -5226,6 +5235,92 @@ static void tcg_reg_alloc_call(TCGContext *s, TCGOp *op) } } +/** + * atom_and_align_for_opc: + * @s: tcg context + * @opc: memory operation code + * @host_atom: MO_ATOM_{IFALIGN,WITHIN16,SUBALIGN} for host operations + * @allow_two_ops: true if we are prepared to issue two operations + * + * Return the alignment and atomicity to use for the inline fast path + * for the given memory operation. The alignment may be larger than + * that specified in @opc, and the correct alignment will be diagnosed + * by the slow path helper. + * + * If @allow_two_ops, the host is prepared to test for 2x alignment, + * and issue two loads or stores for subalignment. + */ +static TCGAtomAlign atom_and_align_for_opc(TCGContext *s, MemOp opc, + MemOp host_atom, bool allow_two_ops) +{ + MemOp align = get_alignment_bits(opc); + MemOp size = opc & MO_SIZE; + MemOp half = size ? size - 1 : 0; + MemOp atmax; + MemOp atom; + + /* When serialized, no further atomicity required. */ + if (s->gen_tb->cflags & CF_PARALLEL) { + atom = opc & MO_ATOM_MASK; + } else { + atom = MO_ATOM_NONE; + } + + switch (atom) { + case MO_ATOM_NONE: + /* The operation requires no specific atomicity. */ + atmax = MO_8; + break; + + case MO_ATOM_IFALIGN: + atmax = size; + break; + + case MO_ATOM_IFALIGN_PAIR: + atmax = half; + break; + + case MO_ATOM_WITHIN16: + atmax = size; + if (size == MO_128) { + /* Misalignment implies !within16, and therefore no atomicity. */ + } else if (host_atom != MO_ATOM_WITHIN16) { + /* The host does not implement within16, so require alignment. */ + align = MAX(align, size); + } + break; + + case MO_ATOM_WITHIN16_PAIR: + atmax = size; + /* + * Misalignment implies !within16, and therefore half atomicity. + * Any host prepared for two operations can implement this with + * half alignment. + */ + if (host_atom != MO_ATOM_WITHIN16 && allow_two_ops) { + align = MAX(align, half); + } + break; + + case MO_ATOM_SUBALIGN: + atmax = size; + if (host_atom != MO_ATOM_SUBALIGN) { + /* If unaligned but not odd, there are subobjects up to half. */ + if (allow_two_ops) { + align = MAX(align, half); + } else { + align = MAX(align, size); + } + } + break; + + default: + g_assert_not_reached(); + } + + return (TCGAtomAlign){ .atom = atmax, .align = align }; +} + /* * Similarly for qemu_ld/st slow path helpers. * We must re-implement tcg_gen_callN and tcg_reg_alloc_call simultaneously, From patchwork Mon May 15 14:32:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681982 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1154759wrr; Mon, 15 May 2023 07:49:18 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5y49sX0+nk0C+pvaOJln4Vf/2/nYCygr/dJ1OEQl3Amql5NlFYdKJpxAAXSHnmSU9Pdla+ X-Received: by 2002:a05:6402:515:b0:50b:f9d8:ff7f with SMTP id m21-20020a056402051500b0050bf9d8ff7fmr23988008edv.5.1684162158575; Mon, 15 May 2023 07:49:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684162158; cv=none; d=google.com; s=arc-20160816; b=Zlod46SufJj6L+auX+hF5s4tS1WViw+7EmTBgZ8xHAapIrluVf554GKB7f/AcRpov8 q7gIC0IPApJoF0Feo+g9xcGvgzaWg9HBkID/8LAQUNFZ7RN4eaTR1MA6YL4YuUxcz0Zi 6jMWFD6xvLYwlHyTrHl6H44g5gqpsf6PyWKYgkMzeRMKccaYpmbvb52iqie+We+Zzetc dV5TUpEAGk18Uee4/Hd3f6PFH4Ytie++PddkMSHvTFAA1M9Bq6mhYOSJ6bBNle3mrnVz m4mvfYNw8I0TG5MkUGNR1jh319yFuu5cQpKzgm/wpgNwoSlktbNEHOvyLAchp3Bf0dn8 rP3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=udci6EEhxJsFM3OJ4RCi5N8U0RcK0CcyHoB02ejCoyw=; b=HfOCQ4VQ4WL1+L2XBgdAWqoDSl4eg5Ex2ZjSK7Gb9fdA6+y/jofmBPcBoQeY0xgY05 Rq7RtUnP/rob2Uqs0cyYrVjjAi8g2JLOzNWKiqgKF7WY+lHHKvolnpt9ygqSqlf608Uu mFW/vZtK23IW7Yang2RF0/8KyQrbTARe/1wxDiIi09qpscIyiqnmwqkMxcTGzL9wDGl4 OsNv0febAqlgRXUQOPPQ53wUsZ4Ixq4TUJ2Huec+Pqs34Ec3k6JimkCA6AYpL5LWoz+a Flq1U9gtK/V/lAltLQ0O8fHlxVnHRaO1fflccVV32kDe6FYae3egbus2ELnPkzKvhOzZ TyPw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GUtI0Ick; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id q16-20020aa7da90000000b0050bca1bade9si10774586eds.583.2023.05.15.07.49.18 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:49:18 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GUtI0Ick; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZK5-0005n8-3j; Mon, 15 May 2023 10:37:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZJk-0004lt-4y for qemu-devel@nongnu.org; Mon, 15 May 2023 10:36:57 -0400 Received: from mail-pl1-x62b.google.com ([2607:f8b0:4864:20::62b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZJi-0005wq-FF for qemu-devel@nongnu.org; Mon, 15 May 2023 10:36:51 -0400 Received: by mail-pl1-x62b.google.com with SMTP id d9443c01a7336-1a516fb6523so120170055ad.3 for ; Mon, 15 May 2023 07:36:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161409; x=1686753409; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=udci6EEhxJsFM3OJ4RCi5N8U0RcK0CcyHoB02ejCoyw=; b=GUtI0IckYyMCaETSFMXz9jpRAioavRDtkAzg8O9UEOOfc9DCWCt38FQ1Wqhf+6uE/z 2NMv3/GtZxP24jmmmiScPagnNf9fReO7LLnhfUs7CaYvQC+XWZxiy7enU9TzM457gCxe 5zqN07howVaXg4nAwcvBDSTRWiw+ov5fqBcvIMWf3wM28HjOvKUYs81IndTHgCtbmRCH OvVJqQUOQmlDxZVnzlUahLohKJlZ6P0l33g8gbmDfhJprVSLTEHbziMSEBuNdYc6volQ W/UOcnukelK7NCU4QeYZiZaHxoAkHD/dzZuNu9Kmsh6bgYeokc7R6rNbuD4f3d/tz+j7 ga+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161409; x=1686753409; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=udci6EEhxJsFM3OJ4RCi5N8U0RcK0CcyHoB02ejCoyw=; b=Mo6dwGHAq0e10lgBlHlrHsBip/2U+6h2msgEJa/zb++na9vgCdw1GCAgdNOCDIx6GM lITn+zzRrqCfEtf/X6HN6IgGlrJbkbQ/b5VJ8wKfOcEil5Pw9UzpLsHJ5Yrc1Nxdqk26 iBy+eykhrz3pAGZSj8a7t0rj366E/jtr7IR/XTxN8GPTun/BXc70EfzHJ0Aj3ICCnB2Q maGbI3ZgQUiGuK1yVBGwE4XQIzjf4yOUz87O4s9neBoVw7mHKaY//38DAWpVzNyqt3tQ IVLGJh9ZijNk4uyZYFE08ew0LFrPlk5gByiOj5/XOtImzBQJUheo3vgpwsjpdD+IyAaj tEvA== X-Gm-Message-State: AC+VfDxpVOTHOMo6uhEbggJKZnK5pqN8uhgg8vNqPjoU527Z61WqDJ15 l7v98EgsrrhuhA59yXFhBTlvL91f4EvA4tvEfbk= X-Received: by 2002:a17:902:7b81:b0:1ae:89a:92 with SMTP id w1-20020a1709027b8100b001ae089a0092mr5437652pll.59.1684161409297; Mon, 15 May 2023 07:36:49 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id i6-20020a1709026ac600b001aadfdfcd06sm8942735plt.299.2023.05.15.07.36.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:36:48 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 40/54] tcg/i386: Use atom_and_align_for_opc Date: Mon, 15 May 2023 07:32:59 -0700 Message-Id: <20230515143313.734053-41-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62b; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org No change to the ultimate load/store routines yet, so some atomicity conditions not yet honored, but plumbs the change to alignment through the relevant functions. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- v5: Removed r-b for changes to HostAddress. --- tcg/i386/tcg-target.c.inc | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/tcg/i386/tcg-target.c.inc b/tcg/i386/tcg-target.c.inc index 6d55ba5a1c..3b8528e332 100644 --- a/tcg/i386/tcg-target.c.inc +++ b/tcg/i386/tcg-target.c.inc @@ -1774,6 +1774,7 @@ typedef struct { int index; int ofs; int seg; + TCGAtomAlign aa; } HostAddress; bool tcg_target_has_memory_bswap(MemOp memop) @@ -1895,8 +1896,18 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, { TCGLabelQemuLdst *ldst = NULL; MemOp opc = get_memop(oi); - unsigned a_bits = get_alignment_bits(opc); - unsigned a_mask = (1 << a_bits) - 1; + unsigned a_mask; + +#ifdef CONFIG_SOFTMMU + h->index = TCG_REG_L0; + h->ofs = 0; + h->seg = 0; +#else + *h = x86_guest_base; +#endif + h->base = addrlo; + h->aa = atom_and_align_for_opc(s, opc, MO_ATOM_IFALIGN, false); + a_mask = (1 << h->aa.align) - 1; #ifdef CONFIG_SOFTMMU int cmp_ofs = is_ld ? offsetof(CPUTLBEntry, addr_read) @@ -1946,7 +1957,7 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, * copy the address and mask. For lesser alignments, check that we don't * cross pages for the complete access. */ - if (a_bits >= s_bits) { + if (a_mask >= s_mask) { tcg_out_mov(s, ttype, TCG_REG_L1, addrlo); } else { tcg_out_modrm_offset(s, OPC_LEA + trexw, TCG_REG_L1, @@ -1977,13 +1988,8 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, /* TLB Hit. */ tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_L0, TCG_REG_L0, offsetof(CPUTLBEntry, addend)); - - *h = (HostAddress) { - .base = addrlo, - .index = TCG_REG_L0, - }; #else - if (a_bits) { + if (a_mask) { ldst = new_ldst_label(s); ldst->is_ld = is_ld; @@ -1997,9 +2003,6 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, ldst->label_ptr[0] = s->code_ptr; s->code_ptr += 4; } - - *h = x86_guest_base; - h->base = addrlo; #endif return ldst; From patchwork Mon May 15 14:33:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681988 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1155050wrr; Mon, 15 May 2023 07:49:57 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7We7kE70IZYjoDc9v8VJo4eTdyyMgmsK8/kRALRvwyslXOUgKuPWqcrzaUsO/LthWNmjjo X-Received: by 2002:aa7:c38e:0:b0:50d:ff73:64ef with SMTP id k14-20020aa7c38e000000b0050dff7364efmr9740856edq.20.1684162197736; Mon, 15 May 2023 07:49:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684162197; cv=none; d=google.com; s=arc-20160816; b=RJS1Ie3q0zeyvq6eUjoh3mvQG1izdVjUsrKfarke6v0Ja1FdCqd0b4S6574G06jxxT QmTtupfaOB/wakVmExvUpmzssp057djWy+dW0GIUP8dzVW+7M5UPwKc08+58fBUBW8G/ JaDYpjnnl+x1ZPSKH2n9qgxgKOIPelMlqJA5dxyFY4taq0HTXmaVn4jdHA/RbBW/LXpY sOOR92zKuHmPtWv/zo8OL2BnHs0Yig90c06K+ZD57uc8yCUSB8ZYi+VlC5BCQAAdoilX CvZR14fHOxwdgjGfXg7UTSyPxojzcQvpVskrJJY9TH4+8wedxsEIoVIwc9aBib9fEa1H QiAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=LPhN/AVmZfqDEN0zKKnHjQGokKYF75xLV5YHB0OEaoc=; b=xK3dWGnWVo/yF1PhVIIJBVSRDYbjZ/sJ1hNIMe3th0N7LN7T7GzTbBkBQ3P3omul4f Mmy4QMMtYA/SpPzxPxZuMqRhfYNdKqRIdLEEH+c7BDvnmVYbVBE2jNeqrgZFVW/0T2oe PFme+agxsQin1vSM76ofQcQCgpMY+w5YBS3YgX4QeIDQj4thwz3DdMdAZ8JBZ7CF56BZ dNMNRsKTF9i/EoDoPfximnVg3Nnj3BFgMUPWVJXp6c4m6q3gxMe3T8SBVqW6WKaRQcT7 zmWHxQUmvL0OZnWrSFdnIGwJBjmopKj1tbT1IudunvNEPKiqc3l0QwMgVfd34h5XO8k0 CcNA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=mvTdIKjv; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id l3-20020aa7c3c3000000b0050bc4de643fsi11121061edr.225.2023.05.15.07.49.57 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:49:57 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=mvTdIKjv; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZJv-0005Fw-If; Mon, 15 May 2023 10:37:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZJs-0004yO-CB for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:01 -0400 Received: from mail-pl1-x62b.google.com ([2607:f8b0:4864:20::62b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZJj-0005yV-F2 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:00 -0400 Received: by mail-pl1-x62b.google.com with SMTP id d9443c01a7336-1aaea3909d1so120110295ad.2 for ; Mon, 15 May 2023 07:36:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161410; x=1686753410; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LPhN/AVmZfqDEN0zKKnHjQGokKYF75xLV5YHB0OEaoc=; b=mvTdIKjvnkSuz2lCQmiB3DZC9JiVefPaeGnuFgM8g4fZDrKjEezR4UwAYTDhLytCU3 j8KMmFY4koHmQ86YyRTe/zNcE5ozl2fOJUE/4nbtFwblEIE360XBuJEWmdo73qb1bBGT dzOnXjRdfQ/xjPd2jXkfMCeWN6qMDC+f0q0+a3RkOdPpIou5XhL8e/ZB30sF3izZ58yP vmP+Tuu3G/hvcI3XRx9PNa0ngD3bmLLswf7rEzSEJL/gs2pbSwr3/epcrkN8dY9Ldogz mtUu1jCBfplAAv7X+pz4oPBtfvfsqQVJ5aPf6EJbKm9hCxCPayI6ZFIrbkdzUdeekGFz e+cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161410; x=1686753410; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LPhN/AVmZfqDEN0zKKnHjQGokKYF75xLV5YHB0OEaoc=; b=IXEqgz4fLI+FF9n4addPbJ+QmuNvmkWZQIyJPQSnG2NPvemgIAge457+JG/qZjGxCc KznK3p/8SE7OA3/ZTw4ZipfTZEgUvCo0hJjv/sUvROzsIohOl1GjkPmMCoBv06dhsZDn T41ABhrG0aZYMT+zYXtnn5Lv8vIjfncMzzvDkyAOwqfXfWm/gYhD7wmbkfI4G/1BfCYa dqrMKsAMRfiWmxm/0jsSZ2/lR5LqxGauO5JMgtcF8j+FxgiEZHU0FnnM5DDkzBYGdGNf EI9KkiStmhRyNCmKBxPCUNCnvbDBl1RVFLzHIHl2Lrgg8lg3Ab/A9i6KO8VC+lUl7ANi 04ig== X-Gm-Message-State: AC+VfDx8PJ6OjbyDfYavyWt+YkTJ1yACu9WE4cPkDZrNYmAjw4qhVvhA GZJuMNhw8fi3vgzzK6EM/7hkqsYrpJgWZy/fcS0= X-Received: by 2002:a17:902:efc5:b0:19c:f096:bbef with SMTP id ja5-20020a170902efc500b0019cf096bbefmr31630039plb.49.1684161410241; Mon, 15 May 2023 07:36:50 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id i6-20020a1709026ac600b001aadfdfcd06sm8942735plt.299.2023.05.15.07.36.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:36:49 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 41/54] tcg/aarch64: Use atom_and_align_for_opc Date: Mon, 15 May 2023 07:33:00 -0700 Message-Id: <20230515143313.734053-42-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62b; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/aarch64/tcg-target.c.inc | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc index 0cc719d799..ea4108d59c 100644 --- a/tcg/aarch64/tcg-target.c.inc +++ b/tcg/aarch64/tcg-target.c.inc @@ -1593,6 +1593,7 @@ typedef struct { TCGReg base; TCGReg index; TCGType index_ext; + TCGAtomAlign aa; } HostAddress; bool tcg_target_has_memory_bswap(MemOp memop) @@ -1646,8 +1647,13 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, TCGType addr_type = TARGET_LONG_BITS == 64 ? TCG_TYPE_I64 : TCG_TYPE_I32; TCGLabelQemuLdst *ldst = NULL; MemOp opc = get_memop(oi); - unsigned a_bits = get_alignment_bits(opc); - unsigned a_mask = (1u << a_bits) - 1; + unsigned a_mask; + + h->aa = atom_and_align_for_opc(s, opc, + have_lse2 ? MO_ATOM_WITHIN16 + : MO_ATOM_IFALIGN, + false); + a_mask = (1 << h->aa.align) - 1; #ifdef CONFIG_SOFTMMU unsigned s_bits = opc & MO_SIZE; @@ -1693,7 +1699,7 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, * bits within the address. For unaligned access, we check that we don't * cross pages using the address of the last byte of the access. */ - if (a_bits >= s_bits) { + if (a_mask >= s_mask) { x3 = addr_reg; } else { tcg_out_insn(s, 3401, ADDI, TARGET_LONG_BITS == 64, @@ -1713,11 +1719,9 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, ldst->label_ptr[0] = s->code_ptr; tcg_out_insn(s, 3202, B_C, TCG_COND_NE, 0); - *h = (HostAddress){ - .base = TCG_REG_X1, - .index = addr_reg, - .index_ext = addr_type - }; + h->base = TCG_REG_X1, + h->index = addr_reg; + h->index_ext = addr_type; #else if (a_mask) { ldst = new_ldst_label(s); @@ -1735,17 +1739,13 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, } if (USE_GUEST_BASE) { - *h = (HostAddress){ - .base = TCG_REG_GUEST_BASE, - .index = addr_reg, - .index_ext = addr_type - }; + h->base = TCG_REG_GUEST_BASE; + h->index = addr_reg; + h->index_ext = addr_type; } else { - *h = (HostAddress){ - .base = addr_reg, - .index = TCG_REG_XZR, - .index_ext = TCG_TYPE_I64 - }; + h->base = addr_reg; + h->index = TCG_REG_XZR; + h->index_ext = TCG_TYPE_I64; } #endif From patchwork Mon May 15 14:33:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681964 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1152226wrr; Mon, 15 May 2023 07:43:23 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4cnACO1UdnRfxvhMozl+TFXb1lzW2kkn53bo77BCYuMi0hYkw3R6oNCukdGi+3tOaWsG1s X-Received: by 2002:a7b:c357:0:b0:3ee:19b4:a2e6 with SMTP id l23-20020a7bc357000000b003ee19b4a2e6mr21520814wmj.19.1684161803256; Mon, 15 May 2023 07:43:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161803; cv=none; d=google.com; s=arc-20160816; b=nbiBsA9sD5qHm9PSrtOqdpdKBNEah4vIX7OUUnpHf8YYnDdQLn+w08WTwcadieI9I1 mK34sbm33vHXFpDUY+cDwArE46GvQC+NxIapQdC35oSlJCehM5dPNKg7rNPUS23BaFLz 2Y2ofDpvhqnFho153FrnK/KoIuF6RrUev2eGX9NDUyRtXOwZA/8E/1G8ILL5m3g+VirY Lf+wswSSo0Tg16qp+DcAlvFG14EPQv+Pi8/7XyZD4A+U+4YO6KvzLABSpU5XQF6q33j6 CmYlpz/Op+u/QVZClnG5YL9JEqhyOCjRmH6IMSYa0JE6woFM6I1TPqwvXTQpRR9DGKBX zB4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=1qqlVmOWLkkD4N0TUKCkau5U1AXPhMOrCn0VUAaOFSs=; b=AONp1ei+D+/MaG+a+6GLvWPQR50xbqSUbW+R29v1qXbY64sE0+9O7qskzNnAf5umAK M+4xcwEHZHc/WyZk8LQKbKCwRtLxgQKm5HzHh8+IQ7InCfGN9suLS1ZtVGZMNrvAt3kT vfpl4wgXOEKSXrvsFmkjKAsX5qIxov+BJ12J5VadG0s+6JNUztW1yxnzLMO22H5OduC6 rQKBrpQ2fcbWwFkUjp2Q6jEdUuWWEam40Wk5wkSuSSxz50eG/dz1R9WM9u90iy+vEM8D 6XS3XDIOO3P7iBnpu5FUQjZLdZY/li+oq76bwQB09gVouOwojAz/EoON8YDYWBhEH7e1 ekrQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=aQ3PGwvP; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id a3-20020a05600c348300b003f4215d8632si14353041wmq.216.2023.05.15.07.43.23 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:43:23 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=aQ3PGwvP; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZLJ-0007rC-86; Mon, 15 May 2023 10:38:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZJq-0004s1-F2 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:01 -0400 Received: from mail-pl1-x62d.google.com ([2607:f8b0:4864:20::62d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZJk-0005yz-Ar for qemu-devel@nongnu.org; Mon, 15 May 2023 10:36:58 -0400 Received: by mail-pl1-x62d.google.com with SMTP id d9443c01a7336-1ab032d9266so118797235ad.0 for ; Mon, 15 May 2023 07:36:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161411; x=1686753411; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1qqlVmOWLkkD4N0TUKCkau5U1AXPhMOrCn0VUAaOFSs=; b=aQ3PGwvPinfAAEnEb70YaxCNuLX3No6jB2q4yQYcsxAuXAJptV5lgMnabsGnL49rVh xXNov3k+t+pDSLSq9mFfmxCjOfHm2R09EfozwuysjcqblJZPB9v9Sx5D27l3XbltFnNs mjG8AGQ4qS+u0Yd2KK3xKIpCyWG2q8ntccfBFeTn3Q1uF8VtvkcKZRWugWLRMLRfMB/u HMKSJo4cdget0PImaa5PM6WVvMJPJX5IdpRKzP7pJRAMEm2XfizMjRmu7S76dmtdgxx/ tELbAupCH6+gzvPJgiYKa64Mtglf8ttg0mFSe3HFqQwdDXiafXBmARvo73D8bvHngvry oHUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161411; x=1686753411; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1qqlVmOWLkkD4N0TUKCkau5U1AXPhMOrCn0VUAaOFSs=; b=YgYLHESQb3bdv5kV/7gIMD0mtXLYfyT/dLzKfaLCUwu8Fe9hhk0xsK5jXYqEbJrHh7 y/zr+afIadKLf5PpIC7UmhjMl8l0UaGXtxKwNWf0YagyPn6OgYjl6UBcOyO0po0Rdim/ oOmDBAr6ANi/ZCebwHoB076tBeK428aWSiZhwOAgXc3/XckgI+CyOK04HZPtwepf8JOq FyPa8hsBApuaE4r7piqDjCmMbQjRx6/DBN+VdRG899DdNI4Cog9SPYozhnNzd0v9ewxU t7tZiguPMrFZwdxyhFS1lLqsZuL980rN428bPNXZsantrduYqsL9KeDQy5l1axUEb7Ll kN4w== X-Gm-Message-State: AC+VfDybP56BAitGHiNB8E8rTIeVdlQJ6bgRFkDOOMUSx9PUyPNpl54T iplq3qTGGrO89cR8AH/RIjU65Z9TPMPFxVt701w= X-Received: by 2002:a17:902:da84:b0:1ac:5b6b:df4c with SMTP id j4-20020a170902da8400b001ac5b6bdf4cmr39458614plx.69.1684161410916; Mon, 15 May 2023 07:36:50 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id i6-20020a1709026ac600b001aadfdfcd06sm8942735plt.299.2023.05.15.07.36.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:36:50 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 42/54] tcg/arm: Use atom_and_align_for_opc Date: Mon, 15 May 2023 07:33:01 -0700 Message-Id: <20230515143313.734053-43-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62d; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org No change to the ultimate load/store routines yet, so some atomicity conditions not yet honored, but plumbs the change to alignment through the relevant functions. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/arm/tcg-target.c.inc | 39 ++++++++++++++++++++++----------------- 1 file changed, 22 insertions(+), 17 deletions(-) diff --git a/tcg/arm/tcg-target.c.inc b/tcg/arm/tcg-target.c.inc index e5aed03247..add8cc1fd5 100644 --- a/tcg/arm/tcg-target.c.inc +++ b/tcg/arm/tcg-target.c.inc @@ -1323,6 +1323,7 @@ typedef struct { TCGReg base; int index; bool index_scratch; + TCGAtomAlign aa; } HostAddress; bool tcg_target_has_memory_bswap(MemOp memop) @@ -1379,8 +1380,26 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, { TCGLabelQemuLdst *ldst = NULL; MemOp opc = get_memop(oi); - MemOp a_bits = get_alignment_bits(opc); - unsigned a_mask = (1 << a_bits) - 1; + unsigned a_mask; + +#ifdef CONFIG_SOFTMMU + *h = (HostAddress){ + .cond = COND_AL, + .base = addrlo, + .index = TCG_REG_R1, + .index_scratch = true, + }; +#else + *h = (HostAddress){ + .cond = COND_AL, + .base = addrlo, + .index = guest_base ? TCG_REG_GUEST_BASE : -1, + .index_scratch = false, + }; +#endif + + h->aa = atom_and_align_for_opc(s, opc, MO_ATOM_IFALIGN, false); + a_mask = (1 << h->aa.align) - 1; #ifdef CONFIG_SOFTMMU int mem_index = get_mmuidx(oi); @@ -1469,13 +1488,6 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, if (TARGET_LONG_BITS == 64) { tcg_out_dat_reg(s, COND_EQ, ARITH_CMP, 0, TCG_REG_R3, addrhi, 0); } - - *h = (HostAddress){ - .cond = COND_AL, - .base = addrlo, - .index = TCG_REG_R1, - .index_scratch = true, - }; #else if (a_mask) { ldst = new_ldst_label(s); @@ -1484,18 +1496,11 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, ldst->addrlo_reg = addrlo; ldst->addrhi_reg = addrhi; - /* We are expecting a_bits to max out at 7 */ + /* We are expecting alignment to max out at 7 */ tcg_debug_assert(a_mask <= 0xff); /* tst addr, #mask */ tcg_out_dat_imm(s, COND_AL, ARITH_TST, 0, addrlo, a_mask); } - - *h = (HostAddress){ - .cond = COND_AL, - .base = addrlo, - .index = guest_base ? TCG_REG_GUEST_BASE : -1, - .index_scratch = false, - }; #endif return ldst; From patchwork Mon May 15 14:33:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681955 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1150416wrr; Mon, 15 May 2023 07:39:25 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7d3GJp9nLWKZPu55CcivjRliHWVY5DHyJqBYl3cHg69lhyp+C8G+biHb5Pw2lNwxm7sIoP X-Received: by 2002:a7b:c8c9:0:b0:3f4:f00f:5e09 with SMTP id f9-20020a7bc8c9000000b003f4f00f5e09mr9161127wml.11.1684161564917; Mon, 15 May 2023 07:39:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161564; cv=none; d=google.com; s=arc-20160816; b=G6ltXgB6u5iawYGWs6HqkmQuCVmk9iuuIDVrx/Nt1jYEQ6hcRefdtcqz/JEt+IxGhe Mlx8MFlCQvWnflvKCXvLecvk6z2R/U8fzbWh80Za90MnpCD4g0KsC2H7JtST6WJ9hTOU zlT33kU4mAG3+pOM50MATIlOty9qfupE0vgal6bKrApJYvWAyHH6ULhjqUV2DsTvTYq7 10u849buxhMjs4OQmS5Jv5JXgpwBKFMgwmRVjj7XsbB2e137UT+YG78BdrEvKk/yEgM4 q5IXZQTGj7ieVGdTx+CkUwxnK9/4G+bh7yaXbRBjBQD68m0HTf+OCN7u1KFgJRrnWmZx qOAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=ls4ciHximYxj1CuBwfXZE8IEsihukuIHW0VGJUoZUDc=; b=uS5WcbDOUa00ou7BKVWj0hA1XWafRhTtnxdn/cFwrsJj7AzngOOBJzhCesamZowCZP DDy9pQ53qX8C8GD2tmaJY/S/BverVgpsA8mb6JVjqbPUVuNPJlH5IR7KpgWzjy0HdpGL hQ3+U1TLMmf+h7c657ePYeXVFOi99waKc4M7yPGtamTktB/ThBUl24a/1EJ9wRdXbSG5 KaXtH7t6MJqwGEAeBnfKoeb2LwgfEISpbU6pKqND2WRq2Fq8NKxswteHLGxB408+A4Ib pLkNbcE2wNH7eeVA3SDg/TpJjg2imASko5JnuzA2Tyd4JFWOLNn5TDjvFsIFLDrzwVcH T48w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=s9b23tw5; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id z8-20020adfe548000000b0030631c8dd52si14507569wrm.565.2023.05.15.07.39.24 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:39:24 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=s9b23tw5; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZK1-0005W7-BD; Mon, 15 May 2023 10:37:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZJt-00050X-P4 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:01 -0400 Received: from mail-pf1-x434.google.com ([2607:f8b0:4864:20::434]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZJl-0005zV-46 for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:01 -0400 Received: by mail-pf1-x434.google.com with SMTP id d2e1a72fcca58-64ab2a37812so20442189b3a.1 for ; Mon, 15 May 2023 07:36:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161412; x=1686753412; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ls4ciHximYxj1CuBwfXZE8IEsihukuIHW0VGJUoZUDc=; b=s9b23tw5Hjb9D3k/SgxA9PWPSUn1Uq/Y+OPy5FITo4uY3j7eyLYna+N0GXKHXO5hsq SjuS9ASrqHbV/Da0a2GpmPQlmK8C7+UwkvMrU3OxhRlZUrMU6WBiFMrtKnWjodSUpx7Y ToKio7hJq3NWTD0tViL74D+evDEz3aAj1Z91xc7HVp2083RtcyXrg8MkGavOec1e/HMY XS2IarbNJCMUCaGJ4k32qnhVeT2OY7Lh1jCZDrwc06dC+ucaxoFa45bM339K+ouuAGWG eMh0IuaLOun+JPTZ3k8SIeX71rZ+a+g9wXZCaFoVYB6rGUVp4fWk/VaK4aiB4c6o438D sjOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161412; x=1686753412; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ls4ciHximYxj1CuBwfXZE8IEsihukuIHW0VGJUoZUDc=; b=b1LwHsgPJjYtnTPr+Yt+G4dFIwW4XMVcrwJPSLt8xUWeHKR6gUHipd3ML8XmUYU2O7 kzANDVFi12/P+BymtjCO12A3EltHH/2yuWHHJ2OWIOMMtMoVTEEkZCdSNAY0Yr/onBnp MB+lCepsQeO4ptzY148VX09WS+DMy88gFpbb3OX7kFPpCt0MR/Uk4w6OEOefx5KDPNZv gvDRwPm5hdEzpmS9rMoztDkdjXmw/6Bo1qrxd10ygfgSQLsB6RIE4vWjG6zn+uxxASMB 8uXHsVxj6YqPOitE3OwG5mWWYfv+H/0npOxEJtwE7YpAvdT/sTGJLzP4TT+sbFlHKe5k XF/g== X-Gm-Message-State: AC+VfDzx4CZmiACxpQhPbRjRtdDx5+uvsVRsp41ryTU6UV7p7bScNv3O AvYzR47W1Dx3gmioEu3FriQbPOLTU2XyPTfWPQw= X-Received: by 2002:a17:902:f54c:b0:1ac:3b69:bb9c with SMTP id h12-20020a170902f54c00b001ac3b69bb9cmr41573964plf.28.1684161411742; Mon, 15 May 2023 07:36:51 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id i6-20020a1709026ac600b001aadfdfcd06sm8942735plt.299.2023.05.15.07.36.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:36:51 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 43/54] tcg/loongarch64: Use atom_and_align_for_opc Date: Mon, 15 May 2023 07:33:02 -0700 Message-Id: <20230515143313.734053-44-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::434; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x434.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/loongarch64/tcg-target.c.inc | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index d26174dde5..07d35f92fa 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -826,6 +826,7 @@ static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *l) typedef struct { TCGReg base; TCGReg index; + TCGAtomAlign aa; } HostAddress; bool tcg_target_has_memory_bswap(MemOp memop) @@ -845,7 +846,10 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, { TCGLabelQemuLdst *ldst = NULL; MemOp opc = get_memop(oi); - unsigned a_bits = get_alignment_bits(opc); + MemOp a_bits; + + h->aa = atom_and_align_for_opc(s, opc, MO_ATOM_IFALIGN, false); + a_bits = h->aa.align; #ifdef CONFIG_SOFTMMU unsigned s_bits = opc & MO_SIZE; From patchwork Mon May 15 14:33:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681962 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1152179wrr; Mon, 15 May 2023 07:43:14 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6MX9g3DYH9BY54TN+UsxImLr68gkU3lm8My+q1DwxUZW0ngDZtY7eN49YI8kbzQkGtKa+I X-Received: by 2002:a5d:6d49:0:b0:2f7:8779:3bc3 with SMTP id k9-20020a5d6d49000000b002f787793bc3mr23814636wri.11.1684161794357; Mon, 15 May 2023 07:43:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161794; cv=none; d=google.com; s=arc-20160816; b=gxgJvnSH0O2BrmkfRp43GzF8USU5T/vw9jyB0Cmlkm1/f+HmvmGANrVyUAT2kL4Dl5 TCSC4aykSl98+t3v/aeN1QmnjLF1P927HyR4ayiNkIhN1yKA+mAky07h4DdVaw/4A49Q EJZm1nPz39EMVmI2pvBlB4Cc8TxdeFxKgsG7Yq5/Dt3KpiQE4LPG7GqNQ2vbnGhzwJ9w 5b18xHHeJ5oItj7dV9toiCC0q0Cmr4/YizEZ3Vex4zL4Ro4R75oSaFlJi1uwzjpaCFCv EOtvWKeAUwy/gQmy/Q+D1asBTArVHAqmchNSna6Zz4gD0hyTjphzJMrmJlJffo2aMCjB +HHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=COeCBcDhyHAzyrUtiEtch7tCJeKP75ntgsjZMHrLAcw=; b=bLHSk/Ubkq8/ITGvatibhd0pWZNg7z2NuUhUPKUm6H8y6ABYrU84D0d6VTLpL/JzF1 MfuGZ8iPTgIPtWu46lY8dDA5g7dmY4EPYgUuyCzKmYyupP7D0RHPvFj9ndmhNwEySZj9 uuobshdAqSzMvF/jSAnUNYZaEO4yzv3WebNyxVRHZvaNPCMTt1b+JxWhIfvcBK1UdZgj x2V0okG4oSRHmZyrVB/SAFHC5KT+woM58MSQtW+lJs/Zwp5Sc/m6GVILwEdCXQWgSrMb 2XPfXYbKyVCd+GXYbEhmEchzDNRsddyMFhtNQ6Hy8lZmmK/3JWItJmdD3eJg5/DSOI4l etSw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QYHhs7Gw; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id z3-20020a5d4c83000000b00307d158c08esi6164299wrs.463.2023.05.15.07.43.14 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:43:14 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QYHhs7Gw; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZLL-0008Cc-On; Mon, 15 May 2023 10:38:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZJu-00055q-7m for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:02 -0400 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZJl-0005zm-Nv for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:01 -0400 Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-1aaf706768cso98078675ad.0 for ; Mon, 15 May 2023 07:36:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161412; x=1686753412; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=COeCBcDhyHAzyrUtiEtch7tCJeKP75ntgsjZMHrLAcw=; b=QYHhs7GwDYTCfkOtLvKr7EQZw2y1cZHbUKG8jrwnxTJxjItI2fBxRK8zsJZYiwNiuY 5wtldvVk3P8dXOY/1vSIm05ts2Y2KQgInFfICycB1GgLQtRCiEMOBLJy/NUF8guCEQgj jsXX79l0T4JjOJ1Qpcgzbxd9087C0Goziqn7rNGN32EMYeQRIqBOW5g+7p30JCn1f/H6 WixxGmsvtTnQW/9osR8qc0IJ+c/GDqpwkxW98XZqwK0F4aYT4/O2opQPRXLNxRsfPVWc 5QDSoGlQKZQVQDSN1rS7ytrX1zMVC0+7uaIS4c+hs3iRLN3HaIesvTP80Jk3jMVZ2UGA YokA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161412; x=1686753412; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=COeCBcDhyHAzyrUtiEtch7tCJeKP75ntgsjZMHrLAcw=; b=YM7v3aPzCTtdXxQqrd+WoWXz7Qg2BYkiIJNcjuwXPWAc9bd7Han7pSTS94zG03MmxI Q8uTlkgQAiabfbCILFMgeI3iQrYAKPStzHq7ZXMLuqbjwDBbmoYTOdIUa31uNCQHx9Lj uzDRCnm8q/fj1KfdgjOOoKpMOn51n8cEvJgFxXEnVvR0HI4svA0r/4/Z+RfQ+sNh+Bs1 6Arnx7jGbm4yPNTspMLyyk3PiPTGXIFSjcnGdjKoIlaBlBJK/i1TifZ5vwXP+X9GfFh8 jU5sRepohiHobcsm28M64D4NkLWUyhtnqLSMdiHwJO0DIcq51e1Pv8H5nYnlUKtOf+Tg 0Tqw== X-Gm-Message-State: AC+VfDxr3mXmu1//vsLziQUIEAem3QP8Dtx1h88ur3MOnIzU9MS6/BnV 8CLpSgy4AVjy1rqVXd16mEQQtZ/fLVZKOXiRMDc= X-Received: by 2002:a17:902:ce91:b0:1ad:ddf0:131f with SMTP id f17-20020a170902ce9100b001adddf0131fmr16389454plg.51.1684161412527; Mon, 15 May 2023 07:36:52 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id i6-20020a1709026ac600b001aadfdfcd06sm8942735plt.299.2023.05.15.07.36.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:36:52 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 44/54] tcg/mips: Use atom_and_align_for_opc Date: Mon, 15 May 2023 07:33:03 -0700 Message-Id: <20230515143313.734053-45-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::636; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x636.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/mips/tcg-target.c.inc | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/tcg/mips/tcg-target.c.inc b/tcg/mips/tcg-target.c.inc index cd0254a0d7..3f3fe5b991 100644 --- a/tcg/mips/tcg-target.c.inc +++ b/tcg/mips/tcg-target.c.inc @@ -1138,7 +1138,7 @@ static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *l) typedef struct { TCGReg base; - MemOp align; + TCGAtomAlign aa; } HostAddress; bool tcg_target_has_memory_bswap(MemOp memop) @@ -1158,11 +1158,15 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, { TCGLabelQemuLdst *ldst = NULL; MemOp opc = get_memop(oi); - unsigned a_bits = get_alignment_bits(opc); + MemOp a_bits; unsigned s_bits = opc & MO_SIZE; - unsigned a_mask = (1 << a_bits) - 1; + unsigned a_mask; TCGReg base; + h->aa = atom_and_align_for_opc(s, opc, MO_ATOM_IFALIGN, false); + a_bits = h->aa.align; + a_mask = (1 << a_bits) - 1; + #ifdef CONFIG_SOFTMMU unsigned s_mask = (1 << s_bits) - 1; int mem_index = get_mmuidx(oi); @@ -1281,7 +1285,6 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, #endif h->base = base; - h->align = a_bits; return ldst; } @@ -1394,7 +1397,7 @@ static void tcg_out_qemu_ld(TCGContext *s, TCGReg datalo, TCGReg datahi, ldst = prepare_host_addr(s, &h, addrlo, addrhi, oi, true); - if (use_mips32r6_instructions || h.align >= (opc & MO_SIZE)) { + if (use_mips32r6_instructions || h.aa.align >= (opc & MO_SIZE)) { tcg_out_qemu_ld_direct(s, datalo, datahi, h.base, opc, data_type); } else { tcg_out_qemu_ld_unalign(s, datalo, datahi, h.base, opc, data_type); @@ -1481,7 +1484,7 @@ static void tcg_out_qemu_st(TCGContext *s, TCGReg datalo, TCGReg datahi, ldst = prepare_host_addr(s, &h, addrlo, addrhi, oi, false); - if (use_mips32r6_instructions || h.align >= (opc & MO_SIZE)) { + if (use_mips32r6_instructions || h.aa.align >= (opc & MO_SIZE)) { tcg_out_qemu_st_direct(s, datalo, datahi, h.base, opc); } else { tcg_out_qemu_st_unalign(s, datalo, datahi, h.base, opc); From patchwork Mon May 15 14:33:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681959 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1151652wrr; Mon, 15 May 2023 07:42:07 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6acihRca/8/gk3JPfLGpLan0wW4Ti1+snX+lvH9bf+VSI9wxnCnBaFgJ5nxl0zojjLOVGA X-Received: by 2002:adf:cf0b:0:b0:306:3c04:e4df with SMTP id o11-20020adfcf0b000000b003063c04e4dfmr21686580wrj.58.1684161727491; Mon, 15 May 2023 07:42:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161727; cv=none; d=google.com; s=arc-20160816; b=rJAmjB0STQ4GN+bAuBaI1ND9+bqWm6nBSnW3bmGffmNDLvA04W24TF+0X5Ik5LB8n4 ok3+oKEo0E0k2uMDKNM5zbxMiFQ31M0qZWc6xklRFi4nhj3LZxUILIs7eHtbl5+mu1v5 9Mlpvgmi3ndnKXRHuX/4NXhX1MyhgJw/8UxYcYhE+wxqUDgwlIz13bmkAkhslNIVRozp 8EVx9BQIyt8ujEOuxImk60k1Gw2R/ho5H8eORAK/5IwBNuvtz/AEts63yNXQIX496O7e 6d3p7g0k4FulCzCkpsHygVftFxyC9Qyhy09Z0nfG+QhS0dJCL6G2MEILsjB57MxgJW5W hXhA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=DCduZxszzPV9k+ZJiXPHrxUp5Yyx6mQ092je6pWUrHc=; b=iChF58+o4HYoXzPW/YS0023Xb1nkNFUk7o0xQJj/KO5Dgyuu4BTYF89aUs1f9YsAWB ciW8o0erlIGI60gk9al9IfxTgr3ryqbKFUJ0S2sqgcPBtZfHevg4/YC+FITFx4oc3U14 MFXc89hsIPJuoYc8VPUnBnC6Z6eMEDB2JDueODQ4+5qSNKDbbBmpR7XrhissTJ+gnOv9 F1MoLO+cG4bUFXfjnQWj9Ab1VFU8fy+8EY3LFtrG4ArUZMfjO0nqmpOQdpLHPr3UDqDf oeW4Ngvg0JXrkhxrzIMlC3EQkBg0Awo/tW6w8F1f7JD/jPahBnRyalkaap9tSMjg/GWU RURA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="WOBf4/Fu"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id k17-20020a5d66d1000000b00306319b56d1si15311418wrw.939.2023.05.15.07.42.07 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:42:07 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="WOBf4/Fu"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZLX-0000ff-K3; Mon, 15 May 2023 10:38:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZJw-0005P8-Tf for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:08 -0400 Received: from mail-pl1-x62a.google.com ([2607:f8b0:4864:20::62a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZJo-00060B-7H for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:04 -0400 Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-1aaff9c93a5so87248725ad.2 for ; Mon, 15 May 2023 07:36:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161413; x=1686753413; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DCduZxszzPV9k+ZJiXPHrxUp5Yyx6mQ092je6pWUrHc=; b=WOBf4/FuGoLoJGVXoHMkSiONeFatm8YlDzIbg4fdSjTlzHZY8ZMqr1B/hg87tz+tBV VF9alinKnIkbmYeiTI+2VM6V4N6s1dKMY9SqLTeatc3Yyku3FinHRDrX1J0EPOueiQUu gYfmiYIBCnC36KZ9CdA1uIT4fRgUObiv1NkDS55j5vOaz6UVxAh/NsfSbi+1VJ0L4/x6 0m2xzm6RwbroQRECR8OBJZaSzV0F6Nh+6nRecj/hGK/XzUiCi820oxDBPZAkZjozealh eaJ40weq7s/kTznVX6979afRVT7NIC4YWILVYNykswJ9ilKEtobNvIQnTHmx6kMBOWJp 7JNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161413; x=1686753413; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DCduZxszzPV9k+ZJiXPHrxUp5Yyx6mQ092je6pWUrHc=; b=IN8iByqWq9+lwr9Rm/J9UHs/Hp/ijk2Q4SbRZjeVniiCQhnrEGu4WaeVSOdZrHEE59 0GyCw5Jrlu5FnmThjuz8RL8yACv+cY5hI8gyjXpAVJsqoGJronWzG3GpSHSYbM7i9N/6 eeWgJheZwQVUQRj4pygIRSsPtrKeP8aRN09P/gmgO32u2E2zunBkPqk1VgYwh7brdPlE zDVDqL9PjUtLbbBLCxxx5cUifrlFq/PExjTl1UbHxgIgpxhelmC9ADCTSoT2GWu/SUP9 95GWJ7NegSpHNumkaE+QRbv3NIbsIIRunLfxvndS8C7LOqzRLcsBdk8XvjCkohgiL7UV 8nkg== X-Gm-Message-State: AC+VfDw2Bcmt+l7qJr6daqsYVmDvYfBNxEOIodyR7OR0hL5Yn0QDyqa/ 6PoMiYFNt56GDhzPOKWhd/uDlorSmQbJjlS0eBU= X-Received: by 2002:a17:903:120d:b0:1ae:b38:b6a with SMTP id l13-20020a170903120d00b001ae0b380b6amr7585127plh.58.1684161413302; Mon, 15 May 2023 07:36:53 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id i6-20020a1709026ac600b001aadfdfcd06sm8942735plt.299.2023.05.15.07.36.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:36:52 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 45/54] tcg/ppc: Use atom_and_align_for_opc Date: Mon, 15 May 2023 07:33:04 -0700 Message-Id: <20230515143313.734053-46-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62a; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- tcg/ppc/tcg-target.c.inc | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/tcg/ppc/tcg-target.c.inc b/tcg/ppc/tcg-target.c.inc index b62a163014..b5c49895f3 100644 --- a/tcg/ppc/tcg-target.c.inc +++ b/tcg/ppc/tcg-target.c.inc @@ -2015,6 +2015,7 @@ static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *lb) typedef struct { TCGReg base; TCGReg index; + TCGAtomAlign aa; } HostAddress; bool tcg_target_has_memory_bswap(MemOp memop) @@ -2034,7 +2035,23 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, { TCGLabelQemuLdst *ldst = NULL; MemOp opc = get_memop(oi); - unsigned a_bits = get_alignment_bits(opc); + MemOp a_bits; + + /* + * Book II, Section 1.4, Single-Copy Atomicity, specifies: + * + * Before 3.0, "An access that is not atomic is performed as a set of + * smaller disjoint atomic accesses. In general, the number and alignment + * of these accesses are implementation-dependent." Thus MO_ATOM_IFALIGN. + * + * As of 3.0, "the non-atomic access is performed as described in + * the corresponding list", which matches MO_ATOM_SUBALIGN. + */ + h->aa = atom_and_align_for_opc(s, opc, + have_isa_3_00 ? MO_ATOM_SUBALIGN + : MO_ATOM_IFALIGN, + false); + a_bits = h->aa.align; #ifdef CONFIG_SOFTMMU int mem_index = get_mmuidx(oi); From patchwork Mon May 15 14:33:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681956 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1150659wrr; Mon, 15 May 2023 07:39:52 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5cav/7asvjFjTyRVLTgu9G2nrhOtqs9OajFBfNp9ZaA0MGIdp/26IZxHrPUljzwII6lxrL X-Received: by 2002:adf:f24d:0:b0:2f8:3225:2bc2 with SMTP id b13-20020adff24d000000b002f832252bc2mr24074456wrp.41.1684161592309; Mon, 15 May 2023 07:39:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161592; cv=none; d=google.com; s=arc-20160816; b=0l8O4SYXKLnbQ8wu3uGg7ZnQgcz1Xlhe84pjSE4PYmpDS13aCCQJfHwQayfsiR2wA4 7POc35fF/p1pntCwQLDckT3TbdEWR36eCmYXo5BsIDx/rPAUUqMxsBDiXhu5HOVZOTry BhbPiQoJBSIZtaUkR8S3e4FgF0ozRLVeeQC0hk5KJBloRkQ5A2TeGDeB5yiV9B9F09ds iaccAy2kuCV9cT/teGNAlwLYq/uWlRzN2ZQOBKYmMbFARqUPfb4KotVeWbyQm+JKSKKH O7LZZTQ+h48LEFjPP23m6/2/GsnaGqMzEJLdQPiTaGFzxnm82mveQKwbjm4LaG4br2ic ohIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=StcC3iugyCuMy6suZA8eRiPDdLMl9BrRBxet+P7NYKE=; b=Cgl1hbD8WJbNmq+oAPA6tBvFR3WKgXXBwirpiUFfr3gntr8ngkzuhWAq0lVeMOfOnA /0oQ+7tfI9AmTr3VLzp47Eq7x7GtTquasykQ1x4HiaMyI/zOW4Jdvta3dWzhEb4RMeAl ppYhwhfJxqnJn+nyYy9Nbdl+cE2JVQqH6z5PZlR5x0xkImv/Bsldaqo8y+9nvTK5cRhv 8PGn2cOWtw+zsUbH61/owV1dOnGNm2dPYDuhl2x4xCDakz6u/MUphWgDE0Qa4703YYZM Oa0ZRDfynHPoZw41QG9+HR7PN4zj1vEcdUo68WwEpSLUn52UvK5JQ/CYlkEugeOyZ1r0 9Slw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=oTg1MZbU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id h8-20020adff188000000b0030780c0fa34si15141933wro.838.2023.05.15.07.39.52 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:39:52 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=oTg1MZbU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZKO-0006Fx-LZ; Mon, 15 May 2023 10:37:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZJz-0005Pz-AM for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:08 -0400 Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZJo-00060O-9A for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:07 -0400 Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-1aaea43def7so88216655ad.2 for ; Mon, 15 May 2023 07:36:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161414; x=1686753414; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=StcC3iugyCuMy6suZA8eRiPDdLMl9BrRBxet+P7NYKE=; b=oTg1MZbUdvG9e3Wg310VVv3I4mrqcVpYF5FpAURh3rnS2khQbFYNyl43X2I63fNafH 9bR6eTiK9O9GqR+mEjTWWzfROiUEZPmLK2YZqV13TJwOkGZR+hja6J01CoSECIVknj46 YVqlfWoWM0KduXkjZ3D/WXaqTyE6perwSOKxh6ebqX7AThIXnlE1FQlqIXr2qLDN+45g jTYysn65XeTrT4dG3PRvgrEeEDvLTs3c57oNPRrBd1l9mgo8dT4cW48o4gA8ZCCfe6+I m5efjds/qGFixo3ISI8J1SbgBxVnfloFN7HzI+i2ogS3kxsJgO2ZVDxYMytkxPoWI03K gj4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161414; x=1686753414; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=StcC3iugyCuMy6suZA8eRiPDdLMl9BrRBxet+P7NYKE=; b=i2UnAaCNxb5cTxRign9NL7owyKwq7repRGz+VMlXcsm3qj47CDhcuRfzh+QEaMohPA 49f1yffm9IIqSBfXd8DChUnHl8lqE8l+l/WVw3Lg9TP6c18ge1UnMilN6cjBmeappoJ4 am5nXDc/M7oiSCiSM2K2zuq9eYDb1HK+rhYII1+4UKGBFjkGxN/xVXItNm1P8U1l2+HC plA+YRm17W2hJeWQXZeyckrwFugp8Gl23zGnaYCzxylTVpDhDxTnxCsoZkFp9vu65mQA yyK2RfDQnU73aNt3URC26Kag8bF/HVEeQCOwbByMA5ge3CL4GR08m9uxJ48w1m5Vl5kX Orlg== X-Gm-Message-State: AC+VfDwPZexJ7mS4oIWfxQl82zYEkyoJ5nLgssFhn2/gnozcqqC91YGC FDHOWMA1KbhjtJvy09AIIleZn+Pr9IIP1YXCF4k= X-Received: by 2002:a17:902:a609:b0:1ac:946e:4690 with SMTP id u9-20020a170902a60900b001ac946e4690mr22778533plq.49.1684161413995; Mon, 15 May 2023 07:36:53 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id i6-20020a1709026ac600b001aadfdfcd06sm8942735plt.299.2023.05.15.07.36.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:36:53 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 46/54] tcg/riscv: Use atom_and_align_for_opc Date: Mon, 15 May 2023 07:33:05 -0700 Message-Id: <20230515143313.734053-47-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::630; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x630.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- tcg/riscv/tcg-target.c.inc | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/tcg/riscv/tcg-target.c.inc b/tcg/riscv/tcg-target.c.inc index 37870c89fc..1fc1a9199b 100644 --- a/tcg/riscv/tcg-target.c.inc +++ b/tcg/riscv/tcg-target.c.inc @@ -910,8 +910,11 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, TCGReg *pbase, { TCGLabelQemuLdst *ldst = NULL; MemOp opc = get_memop(oi); - unsigned a_bits = get_alignment_bits(opc); - unsigned a_mask = (1u << a_bits) - 1; + TCGAtomAlign aa; + unsigned a_mask; + + aa = atom_and_align_for_opc(s, opc, MO_ATOM_IFALIGN, false); + a_mask = (1u << aa.align) - 1; #ifdef CONFIG_SOFTMMU unsigned s_bits = opc & MO_SIZE; @@ -944,7 +947,7 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, TCGReg *pbase, * cross pages using the address of the last byte of the access. */ addr_adj = addr_reg; - if (a_bits < s_bits) { + if (a_mask < s_mask) { addr_adj = TCG_REG_TMP0; tcg_out_opc_imm(s, TARGET_LONG_BITS == 32 ? OPC_ADDIW : OPC_ADDI, addr_adj, addr_reg, s_mask - a_mask); @@ -983,8 +986,8 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, TCGReg *pbase, ldst->oi = oi; ldst->addrlo_reg = addr_reg; - /* We are expecting a_bits max 7, so we can always use andi. */ - tcg_debug_assert(a_bits < 12); + /* We are expecting alignment max 7, so we can always use andi. */ + tcg_debug_assert(a_mask == sextreg(a_mask, 0, 12)); tcg_out_opc_imm(s, OPC_ANDI, TCG_REG_TMP1, addr_reg, a_mask); ldst->label_ptr[0] = s->code_ptr; From patchwork Mon May 15 14:33:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681975 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1154245wrr; Mon, 15 May 2023 07:48:08 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7JvW9qoau/UARAm+B/bIxCLgffd7EGUdOeTB+Et1bkMWg0W053oDslCZjE9FLqKHTfVqIG X-Received: by 2002:a5d:5946:0:b0:2f5:d3d7:7af4 with SMTP id e6-20020a5d5946000000b002f5d3d77af4mr23068654wri.63.1684162088743; Mon, 15 May 2023 07:48:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684162088; cv=none; d=google.com; s=arc-20160816; b=D+/RTatu4GLV1auLwqNBipNM4onhV67NGwFfW3n+79wXqHF1kAhmt27N5GoxvC1CEl RFVdp/9SQUuBGnqF/h5QPWwS8LQg70xJ+0WX2sG25qkTTqquhefyKZkTHLEludKs8aIp zAvV6umLcgOuu7pM954taxwjwzhUHKapwUUDX+gukCOUcqjCZZ8ENluhEaslWheqtj8o fdMjA3h/WapYqhxRktE4HLXIQzZyboCiIMQ7DzqVniPqEDLZDz5JtFDh/uzgmlTzCsiD Tln5sZBpH9XvR8nx0jbhvYl9mcdwny7IQLnFSB0iEq0YDSIpB35yzxJFj8elbHCIQpi+ Ghbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=NDknFLWtWBKU8G+ToympZ5KLvdGdhGS895Fc1ZFjry8=; b=m+gv6El8zn4oOfErpK4szLT68WP8jqXzTMArTVJpKLjTFg6eVPc4i7LebgkIqxdhJU Ceatjp2z8J4Ucnv89vgkC30W9dlgsprx6v80vbh4txfPJXeveppVRULV2XK12SCUkqtK 6fnhq4W3bRUyRP6vwdwOzzthvvc7wql9v0IrG7xySYMw9DhXp9ZCf+H4bhNhrOxAK3uK NwTe4yD7Ty9298EojvLR9RcRjlpK/LtQdHLkXvTRWC7vH2JzygR/AlX9GnxZnlvtMcz4 2DcZ34HIVL+gwCtJNxdtRCstoSsPaXks1c7pyrdOZC2VZPdwu3zNrMEeZnf2KwHG6ikr 2vpQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=B5v3PTgu; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id l17-20020a5d6691000000b00307c7ef183esi6830594wru.1077.2023.05.15.07.48.08 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:48:08 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=B5v3PTgu; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZLH-0007cT-NZ; Mon, 15 May 2023 10:38:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZJx-0005P9-1i for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:08 -0400 Received: from mail-pl1-x633.google.com ([2607:f8b0:4864:20::633]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZJo-00060d-7d for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:04 -0400 Received: by mail-pl1-x633.google.com with SMTP id d9443c01a7336-1ab0c697c2bso118636415ad.1 for ; Mon, 15 May 2023 07:36:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161414; x=1686753414; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NDknFLWtWBKU8G+ToympZ5KLvdGdhGS895Fc1ZFjry8=; b=B5v3PTguSldXGeu30WDAIeCNkBRB28ND45sg9u5HXN09yfvTYoHqabv5Sq+pVW5Ox1 ApKE5w0rKfsEd3eHzWi0ju1hx7c8g0wLbl5G3ivYIa0ogpYPTF2DpLv0Ag8w8obCHAGc 7ahq/EaNKj/iKSKvflg50Baf2gBUrgABone1t96SAc4mTPinvO4IS1FRejf11KJHOg19 97uLpKu1g4DMhL8iwgqScMBxOAZC3O9UQI4KLLjmSCCJY0bBtq1havGkM4j38HUCD7nr A32cu5mbYjYQ9CdJA9L9SIZoK+BYG2e+RqaMscPRpdG987XCpvl+J5osaPzjPyF8Xi4M CmDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161414; x=1686753414; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NDknFLWtWBKU8G+ToympZ5KLvdGdhGS895Fc1ZFjry8=; b=GLmO6j5ce9Ui3HTXAJ1sbvDS2LpkLkxRgaViHZoQC5ysoRQjLI1EbWvZNkuwRFCpPa xfl6E66gI/uLTwzOnFfaWIBGvRFZDAXNGjEc3sxoFAq3R98/Oesl7InzEDIzWjQ6dY+o 1+tnHXbYREIfEzACKBLctDb7vDtszWR+vPhtf/8fKftE3pSCPINHfSA459vCdnjufTb/ KdzWDP65fQAt7IkHfPLfvnxmPSLPgMc/gpgvshQKIItcg6rRaihggxiPNlyDvkJ5Od5c XfQvcVqMuyTbrTlpB0n3MEbd7ycOeoY54thkI+ifStu66mEoJLdohoPnWeiRWZ1JJR4w +zbg== X-Gm-Message-State: AC+VfDwV3IDhxA8Hx0ym8Y2D797bQTCt12wsDAnmoD/ewT/W2EDZ6JOv hkEa7pYEryndulIZcQSWY/iadiwMxXBmPYEYQqM= X-Received: by 2002:a17:903:24e:b0:1ac:3b6f:c141 with SMTP id j14-20020a170903024e00b001ac3b6fc141mr41845261plh.11.1684161414745; Mon, 15 May 2023 07:36:54 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id i6-20020a1709026ac600b001aadfdfcd06sm8942735plt.299.2023.05.15.07.36.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:36:54 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 47/54] tcg/s390x: Use atom_and_align_for_opc Date: Mon, 15 May 2023 07:33:06 -0700 Message-Id: <20230515143313.734053-48-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::633; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x633.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/s390x/tcg-target.c.inc | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/tcg/s390x/tcg-target.c.inc b/tcg/s390x/tcg-target.c.inc index 22f0206b5a..8e34b214fc 100644 --- a/tcg/s390x/tcg-target.c.inc +++ b/tcg/s390x/tcg-target.c.inc @@ -1572,6 +1572,7 @@ typedef struct { TCGReg base; TCGReg index; int disp; + TCGAtomAlign aa; } HostAddress; bool tcg_target_has_memory_bswap(MemOp memop) @@ -1733,8 +1734,10 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, { TCGLabelQemuLdst *ldst = NULL; MemOp opc = get_memop(oi); - unsigned a_bits = get_alignment_bits(opc); - unsigned a_mask = (1u << a_bits) - 1; + unsigned a_mask; + + h->aa = atom_and_align_for_opc(s, opc, MO_ATOM_IFALIGN, false); + a_mask = (1 << h->aa.align) - 1; #ifdef CONFIG_SOFTMMU unsigned s_bits = opc & MO_SIZE; @@ -1764,7 +1767,7 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, * bits within the address. For unaligned access, we check that we don't * cross pages using the address of the last byte of the access. */ - a_off = (a_bits >= s_bits ? 0 : s_mask - a_mask); + a_off = (a_mask >= s_mask ? 0 : s_mask - a_mask); tlb_mask = (uint64_t)TARGET_PAGE_MASK | a_mask; if (a_off == 0) { tgen_andi_risbg(s, TCG_REG_R0, addr_reg, tlb_mask); @@ -1806,7 +1809,7 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, ldst->addrlo_reg = addr_reg; /* We are expecting a_bits to max out at 7, much lower than TMLL. */ - tcg_debug_assert(a_bits < 16); + tcg_debug_assert(a_mask <= 0xffff); tcg_out_insn(s, RI, TMLL, addr_reg, a_mask); tcg_out16(s, RI_BRC | (7 << 4)); /* CC in {1,2,3} */ From patchwork Mon May 15 14:33:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681983 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1154782wrr; Mon, 15 May 2023 07:49:21 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6L/WXGzCu95vBcQU58AVAAdRCSRS35e1ayIaw0SOZB6sMgdB54te5Tp9VaeMrLmhpj2Bwb X-Received: by 2002:a7b:cb90:0:b0:3f4:f8f9:bce7 with SMTP id m16-20020a7bcb90000000b003f4f8f9bce7mr6676823wmi.32.1684162161303; Mon, 15 May 2023 07:49:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684162161; cv=none; d=google.com; s=arc-20160816; b=Txiv9jpvQNK2CoAEXvb943uHSyiZmkekp5ddFFttBXmBpr1DAbHGzWIXdRaTMVV5Xq /4skHib6sN4PbbwA1dRGZhYx6J+Xs3GPhiYjo+Byw4OvFp+Q0iyBRalgkDrQWvdNgTK2 +4eaBUq3o1tn2QIaxotksFQTxG7taup+0Nd40wTJKwktcU4PqU3WvY6AJibBM2T2a4XN zK6aRvZEPeRSBPccNhmK7gM8cBDm+9WSZFPsmaVjRXh/1+1kNRYXM8/962Qs0+7HWVZZ wUsIgHJ/4BZiFt+nzMWMn269ZRBkcVIeHoaZ6wfcQpGRa3kEcpy26Kj3elf6Gm6t3u2b A+Pg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=DtY46dQ1btaM3hQ34MrgQ/0EC+dDpusLxNLWdyRhtMo=; b=VFBk1tQMySiBLq2Aw1SeUNQVO3EB4Z3Rito3VBOch6vHHmgd3wsa1MqkStJDz768BM WSWiXzN6TnQzSayQZqda57MAHwkpSQHSIED/UElkt3ongKeI1Hofu++dY4wqpbN2AUOv VzdKbGSE1JwarLoKj4Zbdd31Og18k4dfOnDYK6gJI90wCB0Q7R3ahswWLHlugKDXDAan kXqPOhp77Gut682OcAzi3gM7Ifz3/Xt3XSkzNgFuPXqG7v21+NriUpC+ePGomXC9M9L4 WQ0VblX2VkXTMvVrKYDTLyS0xW2YQ5CC0LPxlQe8Sjt7LABb1sgWnYYgzoHy4vQHlObD a9dw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=C1NxuCWA; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id s19-20020a05600c45d300b003f40cef02fcsi17122926wmo.25.2023.05.15.07.49.21 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:49:21 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=C1NxuCWA; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZLZ-0000rG-8b; Mon, 15 May 2023 10:38:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZJz-0005Q1-Gp for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:08 -0400 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZJo-00060s-TE for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:07 -0400 Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-1ab267e3528so91646755ad.0 for ; Mon, 15 May 2023 07:36:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161415; x=1686753415; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DtY46dQ1btaM3hQ34MrgQ/0EC+dDpusLxNLWdyRhtMo=; b=C1NxuCWASvRkqmkfxHc6ktErupL+w5No5cEXZ5xw6iGTz8gM7SpDRfVIoksPFtA8sj GdYFTHaJNMxcO4IfGkIt5xNmuqxeeuMGj4bqz3Jl36QlXTDRF9YJKdUC83yp1xR0+q9Q kGVU5NGx/axkBQ9U/Ifuu/+blj7Nv7GSPz78EFJBI5tQPiDUT8q3+qqZDHtiHyGBhgKn TI0fSX+d3wSruyd/VfmRiNeZtIXzwq1aSqizT8iGvoBkQarmAmSr6ZgnhMbqiwrrXXHs UWemPR0TImK4NkkdMWYhBGUWmj2KW230UVCJPgfqlMyaiOHP+joj3qs6UP6mIRDEGMcc sLcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161415; x=1686753415; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DtY46dQ1btaM3hQ34MrgQ/0EC+dDpusLxNLWdyRhtMo=; b=CGI0Wy/fkHslO/ekqwbS52VT8iQbNtAsHuADW8of/DJddwWL71FBU1i7zOJzDMrHRE ZaeHwH+gXDMX8fK/KN0BAg73xXU6TUUDI/+ZKkVsCDnm0CAhL1XJ0xzBJVING4ChtuIf KJcx8LBf+taUyK4Elu/jccc5iKo9Z78J/3fj746ioZwVtVoEmqW+Chl7Y0X6EM58XqT1 JNLSoDN2RkqiKRrhHkKftytQUFW05Z5gfcLKk5gaU5zwDS0gzgJEdO1ufuckQUL9LEKj SQKm5qr6wawdRgHxuQOThHer9MCfm0hV4UciTE6w0bJNvqYpE6UH2i0fouNBdoPv3yn/ tIVw== X-Gm-Message-State: AC+VfDyXYiqOj6AOxKP3kJt19SgRcGavuQidWAEi3hmhL+t8TvqrNqjr qYPKLNAbgnl5Ia7akmAW0jwS8fvFN60B0poxgoU= X-Received: by 2002:a17:902:dc81:b0:1ac:8cd6:9323 with SMTP id n1-20020a170902dc8100b001ac8cd69323mr21982705pld.22.1684161415640; Mon, 15 May 2023 07:36:55 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id i6-20020a1709026ac600b001aadfdfcd06sm8942735plt.299.2023.05.15.07.36.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:36:55 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 48/54] tcg/sparc64: Use atom_and_align_for_opc Date: Mon, 15 May 2023 07:33:07 -0700 Message-Id: <20230515143313.734053-49-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::635; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x635.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- tcg/sparc64/tcg-target.c.inc | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc index bb23038529..9676b745a2 100644 --- a/tcg/sparc64/tcg-target.c.inc +++ b/tcg/sparc64/tcg-target.c.inc @@ -1009,6 +1009,7 @@ static bool tcg_out_qemu_st_slow_path(TCGContext *s, TCGLabelQemuLdst *lb) typedef struct { TCGReg base; TCGReg index; + TCGAtomAlign aa; } HostAddress; bool tcg_target_has_memory_bswap(MemOp memop) @@ -1028,13 +1029,13 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, { TCGLabelQemuLdst *ldst = NULL; MemOp opc = get_memop(oi); - unsigned a_bits = get_alignment_bits(opc); - unsigned s_bits = opc & MO_SIZE; + MemOp s_bits = opc & MO_SIZE; unsigned a_mask; /* We don't support unaligned accesses. */ - a_bits = MAX(a_bits, s_bits); - a_mask = (1u << a_bits) - 1; + h->aa = atom_and_align_for_opc(s, opc, MO_ATOM_IFALIGN, false); + h->aa.align = MAX(h->aa.align, s_bits); + a_mask = (1u << h->aa.align) - 1; #ifdef CONFIG_SOFTMMU int mem_index = get_mmuidx(oi); @@ -1086,11 +1087,13 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, cc = TARGET_LONG_BITS == 64 ? BPCC_XCC : BPCC_ICC; tcg_out_bpcc0(s, COND_NE, BPCC_PN | cc, 0); #else - if (a_bits != s_bits) { - /* - * Test for at least natural alignment, and defer - * everything else to the helper functions. - */ + /* + * If the size equals the required alignment, we can skip the test + * and allow host SIGBUS to deliver SIGBUS to the guest. + * Otherwise, test for at least natural alignment and defer + * everything else to the helper functions. + */ + if (s_bits != get_alignment_bits(opc)) { tcg_debug_assert(check_fit_tl(a_mask, 13)); tcg_out_arithi(s, TCG_REG_G0, addr_reg, a_mask, ARITH_ANDCC); From patchwork Mon May 15 14:33:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681976 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1154272wrr; Mon, 15 May 2023 07:48:13 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7C9bnVPCJSyb+tTRgFNx3e31M+1qQXBfnF46/9UtyKK6yZMC9Yabu+RPvVZtOTHeFHIppS X-Received: by 2002:adf:f84b:0:b0:307:8b6f:dcff with SMTP id d11-20020adff84b000000b003078b6fdcffmr22035773wrq.37.1684162093020; Mon, 15 May 2023 07:48:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684162093; cv=none; d=google.com; s=arc-20160816; b=P1HxsD/C+bhV+dBv0l4HK3ZWv++cGYS+aF1HYCZD1FuPxijvf6kX4rEw24Gpu1NWnn eRf3aXAfA9q7TlzzOLVuzMZdFLMuSc8mq/RaFAV8CmhWL4M+5ISwSwigU6TffoK287tr RSAMnMcHD9vmyCDrguXvRDlNXmxXayi76mL296UhJZhmbk7i3HzXRamTnWeDXq5U9gHv TYotBYRL7sSNAWNBhNXFz9XgJ4PvxTpHpEmTeE11aQkqTYxbXY2+ktycjEmSabnPFLRK EDHgVWp+6E4f7Nrq5m/N3Gt1Ab1ARt6eRCQFPSmLbsGt4kx99L1SIvMJ/iVxO45zTCYA Wj1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=e3K+79nBiSc5crGCl8vbMQJ8GibyAa5XexENyD2+9r4=; b=flNYwDNCQ8y9h3Szz3IEsTaXBDCtmh+2BTKWC7zmRlRt0GFvo4l2upYt65lE7UmNjI PBv8o0djNlOcrF7s5ig0mslfTtxbw959LMJY6Q0wsSeOd2kgP3AsKYc0IQE+zrXPDuwY fwroNKG2XpY1DAKStuvm1s1KK/Bmm1RGMDCQSGwEQM2adpDCw76q/IdI2IGYfN33tMYC hkKJg4LXVahgAloNOgURjnHOqM624QUgfR82pQqUZ2cUuTobq5rI16r+POYEd4+adQhs Ql7LmjMa8Qcx+YyQkVZq42d4Epx4wQUSsV8ykovjEwduST5xLiYAUFthmZpqM4mgUiaG 9Frg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=rpmgzLPJ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id d11-20020adffd8b000000b0030639d02b12si14906660wrr.627.2023.05.15.07.48.12 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:48:12 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=rpmgzLPJ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZKl-0006vH-Lt; Mon, 15 May 2023 10:37:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZK1-0005W5-0O for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:09 -0400 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZJp-00061B-Kf for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:08 -0400 Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-1ab01bf474aso97689105ad.1 for ; Mon, 15 May 2023 07:36:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161416; x=1686753416; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e3K+79nBiSc5crGCl8vbMQJ8GibyAa5XexENyD2+9r4=; b=rpmgzLPJ+aNecBK2ne+VOKKD3v0+ZT+GFbRBOAFL9T8r02IirH6DCbLLgGAxPYoc28 UjocEzLn/ygh5Hdnn4beN4rvFd/hU/BcgFkzktsQFYkNsDSDTx0hl7XkWWBmMcMqMZXD jfz/zBCjyGNhVWFkG3u6tDwhzfbtGeJule+WySA6Zc0QB0JRddXjeMRAAUa673cbBl7e 8a+TxqRaPt8gbJTQ3baHa0Jllc1dKU6I5OPSHw/CUPIUI1kaiZoGsukWmorfeMOtxvgM O39tXRtRHx42OYJqQ+TytDWZGxS9MFkGDk/L9/+RKsDK75jJZfRoNCmjXQQ7dazKQ9/w iHMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161416; x=1686753416; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e3K+79nBiSc5crGCl8vbMQJ8GibyAa5XexENyD2+9r4=; b=AHbOw7lO+Ubic854uApqgiiEemYtprBKxcfTfQWI+P+l2RiK9qRdsjm8Cnkt4CSxe9 93PzI4ZO7hajOWwfgBG0L3IgUVTauPBlcvhPkpHgVQv5rMIwQq+6i4YZxwT+MkQoDPOk rSNWAGE6NvoX7+6K/lCDK1elt4hSyAhX7bv3hvVt0ZrTo899lu6FDwClxgl5QQwYRMRa xrE0si5PP4p3muoSlBmViCcOyJRspQvYSXshAjTsshuLOWlwN9hvLg4kAWD02YLXQlpu Ufe3I+xTPSQfU6SXFq/5fNB9/qLSRPuF33aJP90j7t88T/CSKJ64ymdXEZzI1vX+fJxA 11vw== X-Gm-Message-State: AC+VfDyH79z5/Vbeq8x2yuSKMs1YzUhJvyrKlN1Cl3HqL+HvngxPew2Y syonLVhKqRQVngmIzh6qGBcJ7DLGZJAON7x0Gtg= X-Received: by 2002:a17:902:ee89:b0:1a6:74f6:fa92 with SMTP id a9-20020a170902ee8900b001a674f6fa92mr36566171pld.19.1684161416415; Mon, 15 May 2023 07:36:56 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id i6-20020a1709026ac600b001aadfdfcd06sm8942735plt.299.2023.05.15.07.36.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:36:56 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 49/54] tcg/i386: Honor 64-bit atomicity in 32-bit mode Date: Mon, 15 May 2023 07:33:08 -0700 Message-Id: <20230515143313.734053-50-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::635; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x635.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Use the fpu to perform 64-bit loads and stores. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/i386/tcg-target.c.inc | 44 +++++++++++++++++++++++++++++++++------ 1 file changed, 38 insertions(+), 6 deletions(-) diff --git a/tcg/i386/tcg-target.c.inc b/tcg/i386/tcg-target.c.inc index 3b8528e332..0415ca2a4c 100644 --- a/tcg/i386/tcg-target.c.inc +++ b/tcg/i386/tcg-target.c.inc @@ -468,6 +468,10 @@ static bool tcg_target_const_match(int64_t val, TCGType type, int ct) #define OPC_GRP5 (0xff) #define OPC_GRP14 (0x73 | P_EXT | P_DATA16) +#define OPC_ESCDF (0xdf) +#define ESCDF_FILD_m64 5 +#define ESCDF_FISTP_m64 7 + /* Group 1 opcode extensions for 0x80-0x83. These are also used as modifiers for OPC_ARITH. */ #define ARITH_ADD 0 @@ -2086,7 +2090,20 @@ static void tcg_out_qemu_ld_direct(TCGContext *s, TCGReg datalo, TCGReg datahi, datalo = datahi; datahi = t; } - if (h.base == datalo || h.index == datalo) { + if (h.aa.atom == MO_64) { + /* + * Atomicity requires that we use use a single 8-byte load. + * For simplicity and code size, always use the FPU for this. + * Similar insns using SSE/AVX are merely larger. + * Load from memory in one go, then store back to the stack, + * from whence we can load into the correct integer regs. + */ + tcg_out_modrm_sib_offset(s, OPC_ESCDF + h.seg, ESCDF_FILD_m64, + h.base, h.index, 0, h.ofs); + tcg_out_modrm_offset(s, OPC_ESCDF, ESCDF_FISTP_m64, TCG_REG_ESP, 0); + tcg_out_modrm_offset(s, movop, datalo, TCG_REG_ESP, 0); + tcg_out_modrm_offset(s, movop, datahi, TCG_REG_ESP, 4); + } else if (h.base == datalo || h.index == datalo) { tcg_out_modrm_sib_offset(s, OPC_LEA, datahi, h.base, h.index, 0, h.ofs); tcg_out_modrm_offset(s, movop + h.seg, datalo, datahi, 0); @@ -2156,12 +2173,27 @@ static void tcg_out_qemu_st_direct(TCGContext *s, TCGReg datalo, TCGReg datahi, if (TCG_TARGET_REG_BITS == 64) { tcg_out_modrm_sib_offset(s, movop + P_REXW + h.seg, datalo, h.base, h.index, 0, h.ofs); + break; + } + if (use_movbe) { + TCGReg t = datalo; + datalo = datahi; + datahi = t; + } + if (h.aa.atom == MO_64) { + /* + * Atomicity requires that we use use one 8-byte store. + * For simplicity, and code size, always use the FPU for this. + * Similar insns using SSE/AVX are merely larger. + * Assemble the 8-byte quantity in required endianness + * on the stack, load to coproc unit, and store. + */ + tcg_out_modrm_offset(s, movop, datalo, TCG_REG_ESP, 0); + tcg_out_modrm_offset(s, movop, datahi, TCG_REG_ESP, 4); + tcg_out_modrm_offset(s, OPC_ESCDF, ESCDF_FILD_m64, TCG_REG_ESP, 0); + tcg_out_modrm_sib_offset(s, OPC_ESCDF + h.seg, ESCDF_FISTP_m64, + h.base, h.index, 0, h.ofs); } else { - if (use_movbe) { - TCGReg t = datalo; - datalo = datahi; - datahi = t; - } tcg_out_modrm_sib_offset(s, movop + h.seg, datalo, h.base, h.index, 0, h.ofs); tcg_out_modrm_sib_offset(s, movop + h.seg, datahi, From patchwork Mon May 15 14:33:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681967 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1152536wrr; Mon, 15 May 2023 07:44:05 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4xHtcP4zrNdcrO3x/k0degb71kPpeY9mQVzRWeNm+7P1DkPBs/SHXrNz3j9vM4gJlSaxfk X-Received: by 2002:a05:6000:11cd:b0:306:2638:6fea with SMTP id i13-20020a05600011cd00b0030626386feamr26558853wrx.54.1684161845495; Mon, 15 May 2023 07:44:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161845; cv=none; d=google.com; s=arc-20160816; b=HzzqvDqUyomEQzjtel2MVfUNy4rxNoPF+95+RFnu52jTNNtDgRMNEhxZtKBHXUVCGZ agqUFjV1O+nzwrZFOsk8A0rZ48/cy3q9vMX7M8QasWWJdjRzROl/39ms1Q+P0pcRCorY zXHZrv3CkVIMg8AamzNIUU1pryPKsVKbGH0ckp8AkzB0zPlZOFrxmvqnfTDoGhGNJbEl SYw3tl5of/lQK6/Z7zEZ7I+6q/Wd1+7HzGoIjfRebo4yXblEWjfLlZymOiT5TLKN3ELA O3ZKaFx4kvXcsvSG8EBNhS4zUb0DiPgtNmKCucvIMohdt7pOJYKoQ30oqUpSAnNXArCs PJ1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=t0lfTBvzVqVW4zHYDuaTXpybGovORRKOOKd48jLLK6c=; b=QuIyYobLmYw8ImFkquf6bBN4uE1s2L9efaR2jKF2wP9sziQHskHooPkzRBjsg5bBEr 3X5xSO/Ry4FzQHrThATLH5damO5GAY7p0lQjubJBALCp057jfhQ/q3LB0tUygMZ6C6S+ rvCYsEp+OYYcfgi9+I2Tz1kOsOMv26ggnjsjH9U7SM30ZZiFEPSwMinAprFAS78qyp1W Q0Axbpm7WCKlZbinICpBOR1KmAJoKZRzkm+3KQnwNhgDBozaBB5UYwvHytIwyea/LSJe 6psxto1vqarx+ijX9NMmsVSgBNz4rY8ueJlqQu0/lm9q/3cpuWaTy695rdOUDUeadKLX y/8Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=aQZUWQbv; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id h11-20020a5d548b000000b00307a9207a1dsi10155890wrv.746.2023.05.15.07.44.05 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:44:05 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=aQZUWQbv; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZL9-0007Zm-Nv; Mon, 15 May 2023 10:38:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZK2-0005g4-FE for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:10 -0400 Received: from mail-pl1-x62c.google.com ([2607:f8b0:4864:20::62c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZJq-00061T-Eb for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:10 -0400 Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-1aaf7067647so87347815ad.0 for ; Mon, 15 May 2023 07:36:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161417; x=1686753417; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=t0lfTBvzVqVW4zHYDuaTXpybGovORRKOOKd48jLLK6c=; b=aQZUWQbvFyuWihDuBSGU/9c+q/rSVEP49EE8f6t1AAyIHHCJM7TqE5M6g1Z/pWGOVt /NVVgl7ISyFBRDwfFo9gurTg3tuUGtBOj9PzQUWlqVxD/+WLlhccPGkFSyiM4/lAQrM0 IecF0p8a+DSk7Pk0X7N+ykW9Tl/sPwFAxeo1F3yUEmvwnOf/nhvHFBWldJ1wrl69Ug3C cGtkfwXM6VAG+LINiH/ahoEhytY/f9O0WFGhgWP6PiihlnJ+YQ9iCPpgplATxCBj979K Wg1SaVURAWV1061lA8fXbfVxWOPis35Wu62VowfyUg7yDDFHcp+sh37XNL1gqsFtTlgq kMfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161417; x=1686753417; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t0lfTBvzVqVW4zHYDuaTXpybGovORRKOOKd48jLLK6c=; b=ZpG7QD/icE1wTbbhl30/GcUVn4N50sAJGBaWkY8OTGW6curIGGFBxqn0nEEtLJW4Nu Yux5Rm0zYWj1mEDaqhuwI/PPIfDT8DzszW4n8eYk8Xp+mjHb4Q6w5JECO60hg2QWSwVk StuPhmOdLDb8clZ/XfKrdqp35FZKP4mmCm/8bNlgIi1DrIkxvQNBuFigcdc3GT9kra72 u9B7WjiMzcCKm+fpwI427po/+Sfod7/6wFOOXbitnbf4HcgR/TZ4D7gyTQ/DzHuKNn61 kpPz6Fc3uYulCWPrp3u2VVvOD3IKL8RETRKA8TECM6JWumPO8fgm+YL0ia89Xg3zmbKH w54w== X-Gm-Message-State: AC+VfDzjOOfHqCt9ltSMOefr0mATl8IMRc1iym01aANXSFxjuOmhyr3f P2NrSlVNqSR7T0qxfOX3vg9XCkDVLJzQIkhbHE0= X-Received: by 2002:a17:902:6ac3:b0:1ac:93ba:a5ec with SMTP id i3-20020a1709026ac300b001ac93baa5ecmr21115909plt.52.1684161417151; Mon, 15 May 2023 07:36:57 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id i6-20020a1709026ac600b001aadfdfcd06sm8942735plt.299.2023.05.15.07.36.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:36:56 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 50/54] tcg/i386: Support 128-bit load/store with have_atomic16 Date: Mon, 15 May 2023 07:33:09 -0700 Message-Id: <20230515143313.734053-51-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62c; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62c.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/i386/tcg-target.h | 3 +- tcg/i386/tcg-target.c.inc | 181 +++++++++++++++++++++++++++++++++++++- 2 files changed, 180 insertions(+), 4 deletions(-) diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h index 943af6775e..7f69997e30 100644 --- a/tcg/i386/tcg-target.h +++ b/tcg/i386/tcg-target.h @@ -194,7 +194,8 @@ extern bool have_atomic16; #define TCG_TARGET_HAS_qemu_st8_i32 1 #endif -#define TCG_TARGET_HAS_qemu_ldst_i128 0 +#define TCG_TARGET_HAS_qemu_ldst_i128 \ + (TCG_TARGET_REG_BITS == 64 && have_atomic16) /* We do not support older SSE systems, only beginning with AVX1. */ #define TCG_TARGET_HAS_v64 have_avx1 diff --git a/tcg/i386/tcg-target.c.inc b/tcg/i386/tcg-target.c.inc index 0415ca2a4c..390f126dc8 100644 --- a/tcg/i386/tcg-target.c.inc +++ b/tcg/i386/tcg-target.c.inc @@ -91,6 +91,8 @@ static const int tcg_target_reg_alloc_order[] = { #endif }; +#define TCG_TMP_VEC TCG_REG_XMM5 + static const int tcg_target_call_iarg_regs[] = { #if TCG_TARGET_REG_BITS == 64 #if defined(_WIN64) @@ -347,6 +349,8 @@ static bool tcg_target_const_match(int64_t val, TCGType type, int ct) #define OPC_PCMPGTW (0x65 | P_EXT | P_DATA16) #define OPC_PCMPGTD (0x66 | P_EXT | P_DATA16) #define OPC_PCMPGTQ (0x37 | P_EXT38 | P_DATA16) +#define OPC_PEXTRD (0x16 | P_EXT3A | P_DATA16) +#define OPC_PINSRD (0x22 | P_EXT3A | P_DATA16) #define OPC_PMAXSB (0x3c | P_EXT38 | P_DATA16) #define OPC_PMAXSW (0xee | P_EXT | P_DATA16) #define OPC_PMAXSD (0x3d | P_EXT38 | P_DATA16) @@ -1783,7 +1787,21 @@ typedef struct { bool tcg_target_has_memory_bswap(MemOp memop) { - return have_movbe; + TCGAtomAlign aa; + + if (!have_movbe) { + return false; + } + if ((memop & MO_SIZE) <= MO_64) { + return true; + } + + /* + * Reject 16-byte memop with 16-byte atomicity, i.e. VMOVDQA, + * but do allow a pair of 64-bit operations, i.e. MOVBEQ. + */ + aa = atom_and_align_for_opc(tcg_ctx, memop, MO_ATOM_IFALIGN, true); + return aa.atom <= MO_64; } /* @@ -1811,6 +1829,30 @@ static const TCGLdstHelperParam ldst_helper_param = { static const TCGLdstHelperParam ldst_helper_param = { }; #endif +static void tcg_out_vec_to_pair(TCGContext *s, TCGType type, + TCGReg l, TCGReg h, TCGReg v) +{ + int rexw = type == TCG_TYPE_I32 ? 0 : P_REXW; + + /* vpmov{d,q} %v, %l */ + tcg_out_vex_modrm(s, OPC_MOVD_EyVy + rexw, v, 0, l); + /* vpextr{d,q} $1, %v, %h */ + tcg_out_vex_modrm(s, OPC_PEXTRD + rexw, v, 0, h); + tcg_out8(s, 1); +} + +static void tcg_out_pair_to_vec(TCGContext *s, TCGType type, + TCGReg v, TCGReg l, TCGReg h) +{ + int rexw = type == TCG_TYPE_I32 ? 0 : P_REXW; + + /* vmov{d,q} %l, %v */ + tcg_out_vex_modrm(s, OPC_MOVD_VyEy + rexw, v, 0, l); + /* vpinsr{d,q} $1, %h, %v, %v */ + tcg_out_vex_modrm(s, OPC_PINSRD + rexw, v, v, h); + tcg_out8(s, 1); +} + /* * Generate code for the slow path for a load at the end of block */ @@ -1900,6 +1942,7 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, { TCGLabelQemuLdst *ldst = NULL; MemOp opc = get_memop(oi); + MemOp s_bits = opc & MO_SIZE; unsigned a_mask; #ifdef CONFIG_SOFTMMU @@ -1910,7 +1953,7 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, *h = x86_guest_base; #endif h->base = addrlo; - h->aa = atom_and_align_for_opc(s, opc, MO_ATOM_IFALIGN, false); + h->aa = atom_and_align_for_opc(s, opc, MO_ATOM_IFALIGN, s_bits == MO_128); a_mask = (1 << h->aa.align) - 1; #ifdef CONFIG_SOFTMMU @@ -1920,7 +1963,6 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, TCGType tlbtype = TCG_TYPE_I32; int trexw = 0, hrexw = 0, tlbrexw = 0; unsigned mem_index = get_mmuidx(oi); - unsigned s_bits = opc & MO_SIZE; unsigned s_mask = (1 << s_bits) - 1; target_ulong tlb_mask; @@ -2115,6 +2157,69 @@ static void tcg_out_qemu_ld_direct(TCGContext *s, TCGReg datalo, TCGReg datahi, h.base, h.index, 0, h.ofs + 4); } break; + + case MO_128: + { + TCGLabel *l1 = NULL, *l2 = NULL; + bool use_pair = h.aa.atom < MO_128; + + tcg_debug_assert(TCG_TARGET_REG_BITS == 64); + + if (!use_pair) { + tcg_debug_assert(!use_movbe); + /* + * Atomicity requires that we use use VMOVDQA. + * If we've already checked for 16-byte alignment, that's all + * we need. If we arrive here with lesser alignment, then we + * have determined that less than 16-byte alignment can be + * satisfied with two 8-byte loads. + */ + if (h.aa.align < MO_128) { + use_pair = true; + l1 = gen_new_label(); + l2 = gen_new_label(); + + tcg_out_testi(s, h.base, 15); + tcg_out_jxx(s, JCC_JNE, l2, true); + } + + tcg_out_vex_modrm_sib_offset(s, OPC_MOVDQA_VxWx + h.seg, + TCG_TMP_VEC, 0, + h.base, h.index, 0, h.ofs); + tcg_out_vec_to_pair(s, TCG_TYPE_I64, datalo, + datahi, TCG_TMP_VEC); + + if (use_pair) { + tcg_out_jxx(s, JCC_JMP, l1, true); + tcg_out_label(s, l2); + } + } + if (use_pair) { + if (use_movbe) { + TCGReg t = datalo; + datalo = datahi; + datahi = t; + } + if (h.base == datalo || h.index == datalo) { + tcg_out_modrm_sib_offset(s, OPC_LEA, datahi, + h.base, h.index, 0, h.ofs); + tcg_out_modrm_offset(s, movop + P_REXW + h.seg, + datalo, datahi, 0); + tcg_out_modrm_offset(s, movop + P_REXW + h.seg, + datahi, datahi, 8); + } else { + tcg_out_modrm_sib_offset(s, movop + P_REXW + h.seg, datalo, + h.base, h.index, 0, h.ofs); + tcg_out_modrm_sib_offset(s, movop + P_REXW + h.seg, datahi, + h.base, h.index, 0, h.ofs + 8); + } + } + if (l1) { + tcg_out_label(s, l1); + } + } + break; + default: g_assert_not_reached(); } @@ -2200,6 +2305,60 @@ static void tcg_out_qemu_st_direct(TCGContext *s, TCGReg datalo, TCGReg datahi, h.base, h.index, 0, h.ofs + 4); } break; + + case MO_128: + { + TCGLabel *l1 = NULL, *l2 = NULL; + bool use_pair = h.aa.atom < MO_128; + + tcg_debug_assert(TCG_TARGET_REG_BITS == 64); + + if (!use_pair) { + tcg_debug_assert(!use_movbe); + /* + * Atomicity requires that we use use VMOVDQA. + * If we've already checked for 16-byte alignment, that's all + * we need. If we arrive here with lesser alignment, then we + * have determined that less that 16-byte alignment can be + * satisfied with two 8-byte loads. + */ + if (h.aa.align < MO_128) { + use_pair = true; + l1 = gen_new_label(); + l2 = gen_new_label(); + + tcg_out_testi(s, h.base, 15); + tcg_out_jxx(s, JCC_JNE, l2, true); + } + + tcg_out_pair_to_vec(s, TCG_TYPE_I64, TCG_TMP_VEC, + datalo, datahi); + tcg_out_vex_modrm_sib_offset(s, OPC_MOVDQA_WxVx + h.seg, + TCG_TMP_VEC, 0, + h.base, h.index, 0, h.ofs); + + if (use_pair) { + tcg_out_jxx(s, JCC_JMP, l1, true); + tcg_out_label(s, l2); + } + } + if (use_pair) { + if (use_movbe) { + TCGReg t = datalo; + datalo = datahi; + datahi = t; + } + tcg_out_modrm_sib_offset(s, movop + P_REXW + h.seg, datalo, + h.base, h.index, 0, h.ofs); + tcg_out_modrm_sib_offset(s, movop + P_REXW + h.seg, datahi, + h.base, h.index, 0, h.ofs + 8); + } + if (l1) { + tcg_out_label(s, l1); + } + } + break; + default: g_assert_not_reached(); } @@ -2523,6 +2682,10 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc, tcg_out_qemu_ld(s, a0, a1, a2, args[3], args[4], TCG_TYPE_I64); } break; + case INDEX_op_qemu_ld_i128: + tcg_debug_assert(TCG_TARGET_REG_BITS == 64); + tcg_out_qemu_ld(s, a0, a1, a2, -1, args[3], TCG_TYPE_I128); + break; case INDEX_op_qemu_st_i32: case INDEX_op_qemu_st8_i32: if (TCG_TARGET_REG_BITS >= TARGET_LONG_BITS) { @@ -2540,6 +2703,10 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc, tcg_out_qemu_st(s, a0, a1, a2, args[3], args[4], TCG_TYPE_I64); } break; + case INDEX_op_qemu_st_i128: + tcg_debug_assert(TCG_TARGET_REG_BITS == 64); + tcg_out_qemu_st(s, a0, a1, a2, -1, args[3], TCG_TYPE_I128); + break; OP_32_64(mulu2): tcg_out_modrm(s, OPC_GRP3_Ev + rexw, EXT3_MUL, args[3]); @@ -3234,6 +3401,13 @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) : TARGET_LONG_BITS <= TCG_TARGET_REG_BITS ? C_O0_I3(L, L, L) : C_O0_I4(L, L, L, L)); + case INDEX_op_qemu_ld_i128: + tcg_debug_assert(TCG_TARGET_REG_BITS == 64); + return C_O2_I1(r, r, L); + case INDEX_op_qemu_st_i128: + tcg_debug_assert(TCG_TARGET_REG_BITS == 64); + return C_O0_I3(L, L, L); + case INDEX_op_brcond2_i32: return C_O0_I4(r, r, ri, ri); @@ -4090,6 +4264,7 @@ static void tcg_target_init(TCGContext *s) s->reserved_regs = 0; tcg_regset_set_reg(s->reserved_regs, TCG_REG_CALL_STACK); + tcg_regset_set_reg(s->reserved_regs, TCG_TMP_VEC); #ifdef _WIN64 /* These are call saved, and we don't save them, so don't use them. */ tcg_regset_set_reg(s->reserved_regs, TCG_REG_XMM6); From patchwork Mon May 15 14:33:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681952 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1150356wrr; Mon, 15 May 2023 07:39:17 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6+d2JtJ99h5rNquRKGUPCV4XDO5jRYkWQ8qloS0xqGi/Rv3ggrH8HBmae07TRxrZlvdm4Z X-Received: by 2002:a5d:4701:0:b0:306:31e0:964 with SMTP id y1-20020a5d4701000000b0030631e00964mr23012302wrq.55.1684161557127; Mon, 15 May 2023 07:39:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161557; cv=none; d=google.com; s=arc-20160816; b=WwnTVd9buqUjCR333n6Sbc+xHiNuDajFVMnnbKqH4a8eGawuRfpxwKeIUfQwNo0hak 4hHp9Nmlfw7MDccUYGhsUKJEHp8k714s++LRlEjH0qb40FPZTQoGOEy9zJEJERJhHPnG sB/mS98B7O6fvS+gO5zLbV0jBcy63zq3dS2pb1TY0wDqODTrLpEvsjKufV4aBVV/KhfB doEQv0hajm95i56BRlkr+YHdZKQ4GWpJxloxYxsf+/qwDiuYJoAfi2onCEhmUefLMu2A bSyeVRlOq95tiI1KnTfmuty8dODD2o+Rn4exIklTmXPIdaoVRUPdGiQhdNg+DKS4fGbZ nCFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=s0UJEF5x7D/tJuynxyknIi5yS26S+uTp0Nck/e1vPAw=; b=QFOKVc/mn9JK5enbF8g++chyUvPo62kSQQHDMsdLXoboLD3JNmPkTsMWZJP5ctwIDH LoMNQ7eXobhirqSFQOoCWQWk5LTBrX0gPRSpaGseKwPxgf2UavY8sUu43eN0jtRdQcCd I7bwKdUIgFaLLAAs5yiCG2eDrd5Ser4/S/s4tisGZZkRE/IlF/WCPyH0EA9ivcJlyxaY u1tPvbOGlIdM/M3Hk27/NVJv4mcxIvdCYp7deK1wZjFSg+5/Vzi3Bx5+/G7ifh+dvLlo TNEseh9xVQrt6onxOFy4BS8HYZirKL0J+guegsneZWzha1r68S/B/aByWjtsHB1+p472 c/GA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=OLl8LNwU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id w3-20020a5d6803000000b003079b67d145si11532765wru.398.2023.05.15.07.39.16 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:39:17 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=OLl8LNwU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZLK-00082D-NW; Mon, 15 May 2023 10:38:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZK2-0005gy-TI for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:11 -0400 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZJr-00061p-7X for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:10 -0400 Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-1aad5245571so88134455ad.1 for ; Mon, 15 May 2023 07:36:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161418; x=1686753418; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=s0UJEF5x7D/tJuynxyknIi5yS26S+uTp0Nck/e1vPAw=; b=OLl8LNwUg4e06GWQk1+Kq07UpHHMf+UlLRlAml3aUT3xOrnqVYVpkyt06G+hhSvzEH iEQ+twXDmUK6+mmlRgOuKdQEvDuZ3w6/o0rVPGc54k7uLYDeFCVgqF/2GxYwJT/oMLSC I6G6UEFMv1G/xs3rOZpLBM57Bvt/CnKpPDpoyIayRXPnkEXpWy8Sl8x/ZxjSaRpXHkcm JmN+JXfqhnxlz3Zo95/A4PWwvrbz2Qpl2/zfhOsoRtIc1QCgRaEJfuxLCl8vHfZcBLYj ug8b+VYmEgMyEstOEoBVygnzoEHaOyHG3QkOoKsndTZipxONsHwW5iK0KyQKh6zAjOw0 44iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161418; x=1686753418; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=s0UJEF5x7D/tJuynxyknIi5yS26S+uTp0Nck/e1vPAw=; b=ga16JdiUIN6YXAmpFE9GVlGiKbhqtZDqwjukbP7mU+oDhcl8TPUJfI5taoUAK+HOJ/ b0dPPCfMLrSTOFID4zoJ8/P7dr99LHMCslat8oQDMzyqY1WHvhEUJdVVIyxXGIw+DyNY t88TiDWdwyT/UP+d9VN5elHX717fJEQR8SocbpirRi3smuQT6S3f93RXFsEZK9VDHfBO ncIb37jIi520FEsJpsmz0izt7ffsdKcIUoQnAqHYGd/eWE/5MiGWY8kAUe1k8ERZiuuz xoIO9X9+MAZfQXSe0yZVbdFqUvvMUddobIQbz7yocmyTuh1R9mqbVvYrsONW0UB4jdUy 2AUw== X-Gm-Message-State: AC+VfDzgJFHIySgTaxP9gCXeV1a+u37T1aNNUwCidZOUKdVKnvyWwjd/ 6LyARm8TEjLBzxIzFMcrxVim2AelG/EhR+QS2Ok= X-Received: by 2002:a17:902:e888:b0:1ad:d500:19d3 with SMTP id w8-20020a170902e88800b001add50019d3mr17358339plg.23.1684161417918; Mon, 15 May 2023 07:36:57 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id i6-20020a1709026ac600b001aadfdfcd06sm8942735plt.299.2023.05.15.07.36.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:36:57 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 51/54] tcg/aarch64: Rename temporaries Date: Mon, 15 May 2023 07:33:10 -0700 Message-Id: <20230515143313.734053-52-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::636; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x636.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org We will need to allocate a second general-purpose temporary. Rename the existing temps to add a distinguishing number. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/aarch64/tcg-target.c.inc | 50 ++++++++++++++++++------------------ 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc index ea4108d59c..1ed5be2c00 100644 --- a/tcg/aarch64/tcg-target.c.inc +++ b/tcg/aarch64/tcg-target.c.inc @@ -80,8 +80,8 @@ static TCGReg tcg_target_call_oarg_reg(TCGCallReturnKind kind, int slot) bool have_lse; bool have_lse2; -#define TCG_REG_TMP TCG_REG_X30 -#define TCG_VEC_TMP TCG_REG_V31 +#define TCG_REG_TMP0 TCG_REG_X30 +#define TCG_VEC_TMP0 TCG_REG_V31 #ifndef CONFIG_SOFTMMU /* Note that XZR cannot be encoded in the address base register slot, @@ -998,7 +998,7 @@ static bool tcg_out_dup_vec(TCGContext *s, TCGType type, unsigned vece, static bool tcg_out_dupm_vec(TCGContext *s, TCGType type, unsigned vece, TCGReg r, TCGReg base, intptr_t offset) { - TCGReg temp = TCG_REG_TMP; + TCGReg temp = TCG_REG_TMP0; if (offset < -0xffffff || offset > 0xffffff) { tcg_out_movi(s, TCG_TYPE_PTR, temp, offset); @@ -1150,8 +1150,8 @@ static void tcg_out_ldst(TCGContext *s, AArch64Insn insn, TCGReg rd, } /* Worst-case scenario, move offset to temp register, use reg offset. */ - tcg_out_movi(s, TCG_TYPE_I64, TCG_REG_TMP, offset); - tcg_out_ldst_r(s, insn, rd, rn, TCG_TYPE_I64, TCG_REG_TMP); + tcg_out_movi(s, TCG_TYPE_I64, TCG_REG_TMP0, offset); + tcg_out_ldst_r(s, insn, rd, rn, TCG_TYPE_I64, TCG_REG_TMP0); } static bool tcg_out_mov(TCGContext *s, TCGType type, TCGReg ret, TCGReg arg) @@ -1367,8 +1367,8 @@ static void tcg_out_call_int(TCGContext *s, const tcg_insn_unit *target) if (offset == sextract64(offset, 0, 26)) { tcg_out_insn(s, 3206, BL, offset); } else { - tcg_out_movi(s, TCG_TYPE_I64, TCG_REG_TMP, (intptr_t)target); - tcg_out_insn(s, 3207, BLR, TCG_REG_TMP); + tcg_out_movi(s, TCG_TYPE_I64, TCG_REG_TMP0, (intptr_t)target); + tcg_out_insn(s, 3207, BLR, TCG_REG_TMP0); } } @@ -1505,7 +1505,7 @@ static void tcg_out_addsub2(TCGContext *s, TCGType ext, TCGReg rl, AArch64Insn insn; if (rl == ah || (!const_bh && rl == bh)) { - rl = TCG_REG_TMP; + rl = TCG_REG_TMP0; } if (const_bl) { @@ -1522,7 +1522,7 @@ static void tcg_out_addsub2(TCGContext *s, TCGType ext, TCGReg rl, possibility of adding 0+const in the low part, and the immediate add instructions encode XSP not XZR. Don't try anything more elaborate here than loading another zero. */ - al = TCG_REG_TMP; + al = TCG_REG_TMP0; tcg_out_movi(s, ext, al, 0); } tcg_out_insn_3401(s, insn, ext, rl, al, bl); @@ -1563,7 +1563,7 @@ static void tcg_out_cltz(TCGContext *s, TCGType ext, TCGReg d, { TCGReg a1 = a0; if (is_ctz) { - a1 = TCG_REG_TMP; + a1 = TCG_REG_TMP0; tcg_out_insn(s, 3507, RBIT, ext, a1, a0); } if (const_b && b == (ext ? 64 : 32)) { @@ -1572,7 +1572,7 @@ static void tcg_out_cltz(TCGContext *s, TCGType ext, TCGReg d, AArch64Insn sel = I3506_CSEL; tcg_out_cmp(s, ext, a0, 0, 1); - tcg_out_insn(s, 3507, CLZ, ext, TCG_REG_TMP, a1); + tcg_out_insn(s, 3507, CLZ, ext, TCG_REG_TMP0, a1); if (const_b) { if (b == -1) { @@ -1585,7 +1585,7 @@ static void tcg_out_cltz(TCGContext *s, TCGType ext, TCGReg d, b = d; } } - tcg_out_insn_3506(s, sel, ext, d, TCG_REG_TMP, b, TCG_COND_NE); + tcg_out_insn_3506(s, sel, ext, d, TCG_REG_TMP0, b, TCG_COND_NE); } } @@ -1602,7 +1602,7 @@ bool tcg_target_has_memory_bswap(MemOp memop) } static const TCGLdstHelperParam ldst_helper_param = { - .ntmp = 1, .tmp = { TCG_REG_TMP } + .ntmp = 1, .tmp = { TCG_REG_TMP0 } }; static bool tcg_out_qemu_ld_slow_path(TCGContext *s, TCGLabelQemuLdst *lb) @@ -1862,7 +1862,7 @@ static void tcg_out_goto_tb(TCGContext *s, int which) set_jmp_insn_offset(s, which); tcg_out32(s, I3206_B); - tcg_out_insn(s, 3207, BR, TCG_REG_TMP); + tcg_out_insn(s, 3207, BR, TCG_REG_TMP0); set_jmp_reset_offset(s, which); } @@ -1881,7 +1881,7 @@ void tb_target_set_jmp_target(const TranslationBlock *tb, int n, ptrdiff_t i_offset = i_addr - jmp_rx; /* Note that we asserted this in range in tcg_out_goto_tb. */ - insn = deposit32(I3305_LDR | TCG_REG_TMP, 5, 19, i_offset >> 2); + insn = deposit32(I3305_LDR | TCG_REG_TMP0, 5, 19, i_offset >> 2); } qatomic_set((uint32_t *)jmp_rw, insn); flush_idcache_range(jmp_rx, jmp_rw, 4); @@ -2075,13 +2075,13 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc, case INDEX_op_rem_i64: case INDEX_op_rem_i32: - tcg_out_insn(s, 3508, SDIV, ext, TCG_REG_TMP, a1, a2); - tcg_out_insn(s, 3509, MSUB, ext, a0, TCG_REG_TMP, a2, a1); + tcg_out_insn(s, 3508, SDIV, ext, TCG_REG_TMP0, a1, a2); + tcg_out_insn(s, 3509, MSUB, ext, a0, TCG_REG_TMP0, a2, a1); break; case INDEX_op_remu_i64: case INDEX_op_remu_i32: - tcg_out_insn(s, 3508, UDIV, ext, TCG_REG_TMP, a1, a2); - tcg_out_insn(s, 3509, MSUB, ext, a0, TCG_REG_TMP, a2, a1); + tcg_out_insn(s, 3508, UDIV, ext, TCG_REG_TMP0, a1, a2); + tcg_out_insn(s, 3509, MSUB, ext, a0, TCG_REG_TMP0, a2, a1); break; case INDEX_op_shl_i64: @@ -2125,8 +2125,8 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc, if (c2) { tcg_out_rotl(s, ext, a0, a1, a2); } else { - tcg_out_insn(s, 3502, SUB, 0, TCG_REG_TMP, TCG_REG_XZR, a2); - tcg_out_insn(s, 3508, RORV, ext, a0, a1, TCG_REG_TMP); + tcg_out_insn(s, 3502, SUB, 0, TCG_REG_TMP0, TCG_REG_XZR, a2); + tcg_out_insn(s, 3508, RORV, ext, a0, a1, TCG_REG_TMP0); } break; @@ -2528,8 +2528,8 @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, break; } } - tcg_out_dupi_vec(s, type, MO_8, TCG_VEC_TMP, 0); - a2 = TCG_VEC_TMP; + tcg_out_dupi_vec(s, type, MO_8, TCG_VEC_TMP0, 0); + a2 = TCG_VEC_TMP0; } if (is_scalar) { insn = cmp_scalar_insn[cond]; @@ -2938,9 +2938,9 @@ static void tcg_target_init(TCGContext *s) s->reserved_regs = 0; tcg_regset_set_reg(s->reserved_regs, TCG_REG_SP); tcg_regset_set_reg(s->reserved_regs, TCG_REG_FP); - tcg_regset_set_reg(s->reserved_regs, TCG_REG_TMP); tcg_regset_set_reg(s->reserved_regs, TCG_REG_X18); /* platform register */ - tcg_regset_set_reg(s->reserved_regs, TCG_VEC_TMP); + tcg_regset_set_reg(s->reserved_regs, TCG_REG_TMP0); + tcg_regset_set_reg(s->reserved_regs, TCG_VEC_TMP0); } /* Saving pairs: (X19, X20) .. (X27, X28), (X29(fp), X30(lr)). */ From patchwork Mon May 15 14:33:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681981 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1154631wrr; Mon, 15 May 2023 07:49:02 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ639qXluSAnQnb1/fvgbQRLpIwC9zMlYukfHIkarrfTgALmeZTAnASqkbpHHLr7HwsExfC3 X-Received: by 2002:aa7:d8cb:0:b0:50b:d76a:7904 with SMTP id k11-20020aa7d8cb000000b0050bd76a7904mr24391768eds.28.1684162142333; Mon, 15 May 2023 07:49:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684162142; cv=none; d=google.com; s=arc-20160816; b=uwNQLUjISg1JpJizOR5mPeZHCgTr408Y1dEC5VtPvPhm3nznDOg3/J7lk8bjzpcFyG 0wHeLKEEKfL78CFd9LE2vn/n48yHsBu5HIZZOAIxMVsDI9WE10QQc9fqKjIdAFKa1+c2 ZQefoFVv4t8cnKa9v8/iwJa7z3ncEqLoK2okJ7dfJTQJ1a2fXfsrljVUxvnEuSG36AFj 7TKV8km+HIzjxvuw/YtYc/VOo7DyJ94hssB+i6dxzn7FVWlD7gdePw9mm+ubT30minbI R6zEVGsx1JQAXy/E6fpveUceBAlMXK76AlKrtx4gXEdTdIn97RI6eLo8GejTMYoxD0Q6 LMJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=WhuyjHJ9pdLCLJ9H3nLGBV4xkPMQgMzQI/73pN8YJEU=; b=kJAOS/yzLZSxS5OJgEroL6fGElVjP3ilevVRdp7+jIRMfvIeMJHvf1a4JdfHkYaOwg qxJ1kP4gLyg0/ehwPqY5iyRBC0dteFrgvVQWfQiuEymX546XgHIvvLD+VfBXb9cU5Ac/ WrDm8ifY4tji39vcVQUMyXsciJpHRhwJBnTO8J1C2XImU2YtRIDhJ7SybYKlYJYzaddP sU2ynpUg84Ahdg08Hlo0nQ8EP62MoZAEBdih+YQW+xdZOxpp81bA11iwXEtg3ZsfALcj joChF3DkadJrqbphzPTVlYGNXfpRKUM2B09faOt4jWSJnq821nTjLLu971mRUQEkBQak 1Gmg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=w8AAGISi; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id a18-20020aa7d752000000b005067c8073casi12336617eds.135.2023.05.15.07.49.02 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:49:02 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=w8AAGISi; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZLv-0001sn-Pq; Mon, 15 May 2023 10:39:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZK4-0005oq-Mf for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:13 -0400 Received: from mail-pl1-x632.google.com ([2607:f8b0:4864:20::632]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZJs-00062K-1Z for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:12 -0400 Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-1aad5245632so91410275ad.3 for ; Mon, 15 May 2023 07:36:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161419; x=1686753419; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WhuyjHJ9pdLCLJ9H3nLGBV4xkPMQgMzQI/73pN8YJEU=; b=w8AAGISitY+Z+bdk6L50Mhj6G8RanqrYnXPlTh9DOASWJOHy48pWZp24JAUG3Gqvsa ud1Gsf343piARv1tS8qaYHmLyoBN+FHeJQSUjSpN9wXionvhrScYZKRGKbIJra91gHq8 3wBsTXeJViseNbqLWank5qv7b6JielTbFah6UleGGMjlH2j/Xnu8PDEFbgSW4JRDK02b 6WNc+l8Zq/mr8ttcrxEiKx11TLDyne7xZLtZbfMyld4cJgIy/4ubJKNn8HiOttfTRxVf iJ9GxN08nFTfnUrxuaKHzqE0U5L/jcKXiyi5b3uEgFxSMcoipbMIAp+S66PnjRAzDxoH KrPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161419; x=1686753419; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WhuyjHJ9pdLCLJ9H3nLGBV4xkPMQgMzQI/73pN8YJEU=; b=MDpVSFGLvbsTuJdbbk3YfhPU/8YlDP2xaMUzYe6wgH2B0LtkhH9MX929FpOZbhlBN5 3hPBHj6piArlTzQPDu0ojFz4XKuMGV4UVIaFjVsEgmomVED/ihCWNegDo26Gg2K5vegK SQSZyvbtLC0/Oyqt8Gzq/bB7LCbf0I/w6CyWmabrI35mRDkXvqe7CbCjJtYXH5S2uSgn HsgdAMkUzcYPWH7AP7Vo/a9+hL1wFWEg12wpphrsOR2D6PQ9TEIL+LIgWMyU2JJwGMq/ yfA4OlrjosO48O6ef8RRwXFUDLbQNBTri4SHwr4koYYW8FcUHn3SXBVJqTD0EGAPmCEc qk+Q== X-Gm-Message-State: AC+VfDwraNwpAOEnJ3fgHFnpWXxe2gS5ekGjbfT7zJXeOIZWtpwBbciR tYPsb8tbXIevz6sQ0Wg9MeGZ+8FlMcdyxwWPsPA= X-Received: by 2002:a17:902:e5c3:b0:1ad:fa2e:17fc with SMTP id u3-20020a170902e5c300b001adfa2e17fcmr11424689plf.2.1684161418662; Mon, 15 May 2023 07:36:58 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id i6-20020a1709026ac600b001aadfdfcd06sm8942735plt.299.2023.05.15.07.36.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:36:58 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Peter Maydell Subject: [PATCH v5 52/54] tcg/aarch64: Support 128-bit load/store Date: Mon, 15 May 2023 07:33:11 -0700 Message-Id: <20230515143313.734053-53-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::632; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x632.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Use LDXP+STXP when LSE2 is not present and 16-byte atomicity is required, and LDP/STP otherwise. This requires allocating a second general-purpose temporary, as Rs cannot overlap Rn in STXP. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- v5: Require lse2 for user-only. --- tcg/aarch64/tcg-target-con-set.h | 2 + tcg/aarch64/tcg-target.h | 11 +- tcg/aarch64/tcg-target.c.inc | 179 ++++++++++++++++++++++++++++++- 3 files changed, 189 insertions(+), 3 deletions(-) diff --git a/tcg/aarch64/tcg-target-con-set.h b/tcg/aarch64/tcg-target-con-set.h index d6c6866878..74065c7098 100644 --- a/tcg/aarch64/tcg-target-con-set.h +++ b/tcg/aarch64/tcg-target-con-set.h @@ -14,6 +14,7 @@ C_O0_I2(lZ, l) C_O0_I2(r, rA) C_O0_I2(rZ, r) C_O0_I2(w, r) +C_O0_I3(lZ, lZ, l) C_O1_I1(r, l) C_O1_I1(r, r) C_O1_I1(w, r) @@ -33,4 +34,5 @@ C_O1_I2(w, w, wO) C_O1_I2(w, w, wZ) C_O1_I3(w, w, w, w) C_O1_I4(r, r, rA, rZ, rZ) +C_O2_I1(r, r, l) C_O2_I4(r, r, rZ, rZ, rA, rMZ) diff --git a/tcg/aarch64/tcg-target.h b/tcg/aarch64/tcg-target.h index 74ee2ed255..2c079f21c2 100644 --- a/tcg/aarch64/tcg-target.h +++ b/tcg/aarch64/tcg-target.h @@ -129,7 +129,16 @@ extern bool have_lse2; #define TCG_TARGET_HAS_muluh_i64 1 #define TCG_TARGET_HAS_mulsh_i64 1 -#define TCG_TARGET_HAS_qemu_ldst_i128 0 +/* + * Without FEAT_LSE2, we must use LDXP+STXP to implement atomic 128-bit load, + * which requires writable pages. We must defer to the helper for user-only, + * but in system mode all ram is writable for the host. + */ +#ifdef CONFIG_USER_ONLY +#define TCG_TARGET_HAS_qemu_ldst_i128 have_lse2 +#else +#define TCG_TARGET_HAS_qemu_ldst_i128 1 +#endif #define TCG_TARGET_HAS_v64 1 #define TCG_TARGET_HAS_v128 1 diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc index 1ed5be2c00..893b3514bb 100644 --- a/tcg/aarch64/tcg-target.c.inc +++ b/tcg/aarch64/tcg-target.c.inc @@ -81,6 +81,7 @@ bool have_lse; bool have_lse2; #define TCG_REG_TMP0 TCG_REG_X30 +#define TCG_REG_TMP1 TCG_REG_X17 #define TCG_VEC_TMP0 TCG_REG_V31 #ifndef CONFIG_SOFTMMU @@ -404,6 +405,10 @@ typedef enum { I3305_LDR_v64 = 0x5c000000, I3305_LDR_v128 = 0x9c000000, + /* Load/store exclusive. */ + I3306_LDXP = 0xc8600000, + I3306_STXP = 0xc8200000, + /* Load/store register. Described here as 3.3.12, but the helper that emits them can transform to 3.3.10 or 3.3.13. */ I3312_STRB = 0x38000000 | LDST_ST << 22 | MO_8 << 30, @@ -468,6 +473,9 @@ typedef enum { I3406_ADR = 0x10000000, I3406_ADRP = 0x90000000, + /* Add/subtract extended register instructions. */ + I3501_ADD = 0x0b200000, + /* Add/subtract shifted register instructions (without a shift). */ I3502_ADD = 0x0b000000, I3502_ADDS = 0x2b000000, @@ -638,6 +646,12 @@ static void tcg_out_insn_3305(TCGContext *s, AArch64Insn insn, tcg_out32(s, insn | (imm19 & 0x7ffff) << 5 | rt); } +static void tcg_out_insn_3306(TCGContext *s, AArch64Insn insn, TCGReg rs, + TCGReg rt, TCGReg rt2, TCGReg rn) +{ + tcg_out32(s, insn | rs << 16 | rt2 << 10 | rn << 5 | rt); +} + static void tcg_out_insn_3201(TCGContext *s, AArch64Insn insn, TCGType ext, TCGReg rt, int imm19) { @@ -720,6 +734,14 @@ static void tcg_out_insn_3406(TCGContext *s, AArch64Insn insn, tcg_out32(s, insn | (disp & 3) << 29 | (disp & 0x1ffffc) << (5 - 2) | rd); } +static inline void tcg_out_insn_3501(TCGContext *s, AArch64Insn insn, + TCGType sf, TCGReg rd, TCGReg rn, + TCGReg rm, int opt, int imm3) +{ + tcg_out32(s, insn | sf << 31 | rm << 16 | opt << 13 | + imm3 << 10 | rn << 5 | rd); +} + /* This function is for both 3.5.2 (Add/Subtract shifted register), for the rare occasion when we actually want to supply a shift amount. */ static inline void tcg_out_insn_3502S(TCGContext *s, AArch64Insn insn, @@ -1647,16 +1669,16 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, TCGType addr_type = TARGET_LONG_BITS == 64 ? TCG_TYPE_I64 : TCG_TYPE_I32; TCGLabelQemuLdst *ldst = NULL; MemOp opc = get_memop(oi); + MemOp s_bits = opc & MO_SIZE; unsigned a_mask; h->aa = atom_and_align_for_opc(s, opc, have_lse2 ? MO_ATOM_WITHIN16 : MO_ATOM_IFALIGN, - false); + s_bits == MO_128); a_mask = (1 << h->aa.align) - 1; #ifdef CONFIG_SOFTMMU - unsigned s_bits = opc & MO_SIZE; unsigned s_mask = (1u << s_bits) - 1; unsigned mem_index = get_mmuidx(oi); TCGReg x3; @@ -1837,6 +1859,148 @@ static void tcg_out_qemu_st(TCGContext *s, TCGReg data_reg, TCGReg addr_reg, } } +static TCGLabelQemuLdst * +prepare_host_addr_base_only(TCGContext *s, HostAddress *h, TCGReg addr_reg, + MemOpIdx oi, bool is_ld) +{ + TCGLabelQemuLdst *ldst; + + ldst = prepare_host_addr(s, h, addr_reg, oi, true); + + /* Compose the final address, as LDP/STP have no indexing. */ + if (h->index != TCG_REG_XZR) { + tcg_out_insn(s, 3501, ADD, TCG_TYPE_I64, TCG_REG_TMP0, + h->base, h->index, + h->index_ext == TCG_TYPE_I32 ? MO_32 : MO_64, 0); + h->base = TCG_REG_TMP0; + h->index = TCG_REG_XZR; + h->index_ext = TCG_TYPE_I64; + } + + return ldst; +} + +static void tcg_out_qemu_ld128(TCGContext *s, TCGReg datalo, TCGReg datahi, + TCGReg addr_reg, MemOpIdx oi) +{ + TCGLabelQemuLdst *ldst; + HostAddress h; + + ldst = prepare_host_addr_base_only(s, &h, addr_reg, oi, true); + + if (h.aa.atom < MO_128 || have_lse2) { + tcg_out_insn(s, 3314, LDP, datalo, datahi, h.base, 0, 0, 0); + } else { + TCGLabel *l0, *l1 = NULL; + + /* + * 16-byte atomicity without LSE2 requires LDXP+STXP loop: + * 1: ldxp lo,hi,[addr] + * stxp tmp1,lo,hi,[addr] + * cbnz tmp1, 1b + * + * If we have already checked for 16-byte alignment, that's all + * we need. Otherwise we have determined that misaligned atomicity + * may be handled with two 8-byte loads. + */ + if (h.aa.align < MO_128) { + /* + * TODO: align should be MO_64, so we only need test bit 3, + * which means we could use TBNZ instead of AND+CBNE. + */ + l1 = gen_new_label(); + tcg_out_logicali(s, I3404_ANDI, 0, TCG_REG_TMP1, addr_reg, 15); + tcg_out_brcond(s, TCG_TYPE_I32, TCG_COND_NE, + TCG_REG_TMP1, 0, 1, l1); + } + + l0 = gen_new_label(); + tcg_out_label(s, l0); + + tcg_out_insn(s, 3306, LDXP, TCG_REG_XZR, datalo, datahi, h.base); + tcg_out_insn(s, 3306, STXP, TCG_REG_TMP1, datalo, datahi, h.base); + tcg_out_brcond(s, TCG_TYPE_I32, TCG_COND_NE, TCG_REG_TMP1, 0, 1, l0); + + if (l1) { + TCGLabel *l2 = gen_new_label(); + tcg_out_goto_label(s, l2); + + tcg_out_label(s, l1); + tcg_out_insn(s, 3314, LDP, datalo, datahi, h.base, 0, 0, 0); + + tcg_out_label(s, l2); + } + } + + if (ldst) { + ldst->type = TCG_TYPE_I128; + ldst->datalo_reg = datalo; + ldst->datahi_reg = datahi; + ldst->raddr = tcg_splitwx_to_rx(s->code_ptr); + } +} + +static void tcg_out_qemu_st128(TCGContext *s, TCGReg datalo, TCGReg datahi, + TCGReg addr_reg, MemOpIdx oi) +{ + TCGLabelQemuLdst *ldst; + HostAddress h; + + ldst = prepare_host_addr_base_only(s, &h, addr_reg, oi, false); + + if (h.aa.atom < MO_128 || have_lse2) { + tcg_out_insn(s, 3314, STP, datalo, datahi, h.base, 0, 0, 0); + } else { + TCGLabel *l0, *l1 = NULL; + + /* + * 16-byte atomicity without LSE2 requires LDXP+STXP loop: + * 1: ldxp xzr,tmp1,[addr] + * stxp tmp1,lo,hi,[addr] + * cbnz tmp1, 1b + * + * If we have already checked for 16-byte alignment, that's all + * we need. Otherwise we have determined that misaligned atomicity + * may be handled with two 8-byte stores. + */ + if (h.aa.align < MO_128) { + /* + * TODO: align should be MO_64, so we only need test bit 3, + * which means we could use TBNZ instead of AND+CBNE. + */ + l1 = gen_new_label(); + tcg_out_logicali(s, I3404_ANDI, 0, TCG_REG_TMP1, addr_reg, 15); + tcg_out_brcond(s, TCG_TYPE_I32, TCG_COND_NE, + TCG_REG_TMP1, 0, 1, l1); + } + + l0 = gen_new_label(); + tcg_out_label(s, l0); + + tcg_out_insn(s, 3306, LDXP, TCG_REG_XZR, + TCG_REG_XZR, TCG_REG_TMP1, h.base); + tcg_out_insn(s, 3306, STXP, TCG_REG_TMP1, datalo, datahi, h.base); + tcg_out_brcond(s, TCG_TYPE_I32, TCG_COND_NE, TCG_REG_TMP1, 0, 1, l0); + + if (l1) { + TCGLabel *l2 = gen_new_label(); + tcg_out_goto_label(s, l2); + + tcg_out_label(s, l1); + tcg_out_insn(s, 3314, STP, datalo, datahi, h.base, 0, 0, 0); + + tcg_out_label(s, l2); + } + } + + if (ldst) { + ldst->type = TCG_TYPE_I128; + ldst->datalo_reg = datalo; + ldst->datahi_reg = datahi; + ldst->raddr = tcg_splitwx_to_rx(s->code_ptr); + } +} + static const tcg_insn_unit *tb_ret_addr; static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0) @@ -2172,6 +2336,12 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc, case INDEX_op_qemu_st_i64: tcg_out_qemu_st(s, REG0(0), a1, a2, ext); break; + case INDEX_op_qemu_ld_i128: + tcg_out_qemu_ld128(s, a0, a1, a2, args[3]); + break; + case INDEX_op_qemu_st_i128: + tcg_out_qemu_st128(s, REG0(0), REG0(1), a2, args[3]); + break; case INDEX_op_bswap64_i64: tcg_out_rev(s, TCG_TYPE_I64, MO_64, a0, a1); @@ -2809,9 +2979,13 @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_qemu_ld_i32: case INDEX_op_qemu_ld_i64: return C_O1_I1(r, l); + case INDEX_op_qemu_ld_i128: + return C_O2_I1(r, r, l); case INDEX_op_qemu_st_i32: case INDEX_op_qemu_st_i64: return C_O0_I2(lZ, l); + case INDEX_op_qemu_st_i128: + return C_O0_I3(lZ, lZ, l); case INDEX_op_deposit_i32: case INDEX_op_deposit_i64: @@ -2940,6 +3114,7 @@ static void tcg_target_init(TCGContext *s) tcg_regset_set_reg(s->reserved_regs, TCG_REG_FP); tcg_regset_set_reg(s->reserved_regs, TCG_REG_X18); /* platform register */ tcg_regset_set_reg(s->reserved_regs, TCG_REG_TMP0); + tcg_regset_set_reg(s->reserved_regs, TCG_REG_TMP1); tcg_regset_set_reg(s->reserved_regs, TCG_VEC_TMP0); } From patchwork Mon May 15 14:33:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681968 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1152677wrr; Mon, 15 May 2023 07:44:23 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5oGBfPExF0nrCYWBbRR7bfzSAVODILWTdDe4xXfzjS+W1ty5Xk1Lpa1VCspHdWy+ilyDSq X-Received: by 2002:adf:fdd1:0:b0:306:2ff1:5227 with SMTP id i17-20020adffdd1000000b003062ff15227mr24111241wrs.23.1684161862792; Mon, 15 May 2023 07:44:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161862; cv=none; d=google.com; s=arc-20160816; b=nxHxTpKaqd4rFXJk3fDLj5FoUtcTdSKYeck7lSnwHPl6dusQ60jh3c/z1PEP/Cuw/p 5VjTLHB0L1yNphaK7MA8qXTKtxkjMPkwdiutRDYnULitMdM4NlaGqG4c/rZy7yzRmVfu RxlKn/XJsU1YvEG+BydOkce37a11lxwJlAoRvS2RqOmzTI/8TKy0yAAM6hDS0SqO20Av 7cdXDcqy/E/xxBulD0MlbSq/RDCEsfpQK4XjrAosrlX4pRiIzhEVv8N8GDWuiSgdEBe2 SYBK76bbRj0aBLIdG8AoqFLrwtTFinsTu6Co/w13KGxS/qJbBb3xY8JpwVEPYukTy5GZ 5LRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=a3C62E7I5KSH6hopm1N3+kJjnyiF8W6DIOatywqt9qE=; b=ILaMzlVLWPVtzG2IbbjJxsY0D1LmJyTzKBHRmZV0XpxBpQHuy+JXip+yOLK+hJ7Sbi hjDgn6mbbBfkR0ZwvpupKdvFIw+sU/x2bN41Bm4GUadMoc6OBpQNJg+2CjRo5N4eSZlG OKOpzg4jf9Dy8uJeS/fu/AYGpEABavWUqr9oftquYKzN1kyvupMsz2+nPhXzRcg5QkdX 7djGGmVkEC+nWaYAzTDk1Rr4We37voLRwi3GFXX1wO6lk8iLL7ydckVKbJ+G0Wy+Uuc+ KL77NInyFht3k1T+MJR8Zt9j+XqwKRttEV8VOWFPfFQXOSq4Xg4DaJGkt7BxsWJ/AtUh OegA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=kcvh4loy; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id z18-20020adff752000000b002ff15b4136fsi14652790wrp.418.2023.05.15.07.44.22 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:44:22 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=kcvh4loy; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZLN-0008Qr-Ph; Mon, 15 May 2023 10:38:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZK7-0005ti-0V for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:20 -0400 Received: from mail-pl1-x634.google.com ([2607:f8b0:4864:20::634]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZJs-00062d-Rj for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:14 -0400 Received: by mail-pl1-x634.google.com with SMTP id d9443c01a7336-1aaf2ede38fso124301315ad.2 for ; Mon, 15 May 2023 07:37:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161419; x=1686753419; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=a3C62E7I5KSH6hopm1N3+kJjnyiF8W6DIOatywqt9qE=; b=kcvh4loy5A+0KhiHPS28qq4x7z0Isll6eS2rML2ZU/gSpriYoEF9BahWdJ0w1nG7X/ imkfTW/oopVyrTDbxhDctOAs3HzXbJHeG1e8EBEDrbKsKlVNDix3dl1sVodXrTXlbyoO 3paDA2eqQL8ySpIWItxUhG/Phd8OXL/c+BDF2AjnJEmusPsLXqucO5gpbQHziLU/JODh 2KAY4cU73TGc0qS2jA75wOD/Gv14C9mUaILz8P489RftO6qWXAc1b0ToHSKKs0EupQHH J90xTO/3sI2aObEO60vryMXXB5n+9bT3CkXrSHPPfgj/HFL2oicl4HkdkrNaRY+SqRCx 2mfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161419; x=1686753419; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=a3C62E7I5KSH6hopm1N3+kJjnyiF8W6DIOatywqt9qE=; b=cLHN5a3bY2yLcbR4dYJZo36WStS8hQYHGyujLMoU30H+o8ifkb+E+OrjsRwON4nvYu EeB8Mvkh803ox5KcsaGLfn0MKo/0YQFNmXz2PkGUySBSqPZ2ilEMl1o5G45XSxmiqR7m 94hY180TC379f0/BLxbC2tnF93paDl2owDm0qZBgq1JHjw79LkEADVos7dYoz101xM93 jBxVkr9GaRvB1D1flHSR9J8MUTJpPDk++SrP1aqyZAsDWIhflQAyFL6F6plNUEgVeSrw 6iZsY9plSoiMUe0GhwtR/pj5vOcOvILm22GlGzismnLTAbGIqEzqFblca07eElbw1u1M TH3w== X-Gm-Message-State: AC+VfDyyd46CZN3vFMG0iEqSHfj1BjPHZrE8YwLE4wYnIErExKC5dBv5 mAVpb9yogCo/IWan5XyHj4xBQNe0VLHRVwrUIFc= X-Received: by 2002:a17:902:f547:b0:1ac:4a41:d38d with SMTP id h7-20020a170902f54700b001ac4a41d38dmr40019890plf.51.1684161419504; Mon, 15 May 2023 07:36:59 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id i6-20020a1709026ac600b001aadfdfcd06sm8942735plt.299.2023.05.15.07.36.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:36:59 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org, Daniel Henrique Barboza Subject: [PATCH v5 53/54] tcg/ppc: Support 128-bit load/store Date: Mon, 15 May 2023 07:33:12 -0700 Message-Id: <20230515143313.734053-54-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::634; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x634.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Use LQ/STQ with ISA v2.07, and 16-byte atomicity is required. Note that these instructions do not require 16-byte alignment. Reviewed-by: Daniel Henrique Barboza Signed-off-by: Richard Henderson --- tcg/ppc/tcg-target-con-set.h | 2 + tcg/ppc/tcg-target-con-str.h | 1 + tcg/ppc/tcg-target.h | 3 +- tcg/ppc/tcg-target.c.inc | 115 +++++++++++++++++++++++++++++++---- 4 files changed, 108 insertions(+), 13 deletions(-) diff --git a/tcg/ppc/tcg-target-con-set.h b/tcg/ppc/tcg-target-con-set.h index f206b29205..bbd7b21247 100644 --- a/tcg/ppc/tcg-target-con-set.h +++ b/tcg/ppc/tcg-target-con-set.h @@ -14,6 +14,7 @@ C_O0_I2(r, r) C_O0_I2(r, ri) C_O0_I2(v, r) C_O0_I3(r, r, r) +C_O0_I3(o, m, r) C_O0_I4(r, r, ri, ri) C_O0_I4(r, r, r, r) C_O1_I1(r, r) @@ -34,6 +35,7 @@ C_O1_I3(v, v, v, v) C_O1_I4(r, r, ri, rZ, rZ) C_O1_I4(r, r, r, ri, ri) C_O2_I1(r, r, r) +C_O2_I1(o, m, r) C_O2_I2(r, r, r, r) C_O2_I4(r, r, rI, rZM, r, r) C_O2_I4(r, r, r, r, rI, rZM) diff --git a/tcg/ppc/tcg-target-con-str.h b/tcg/ppc/tcg-target-con-str.h index 094613cbcb..20846901de 100644 --- a/tcg/ppc/tcg-target-con-str.h +++ b/tcg/ppc/tcg-target-con-str.h @@ -9,6 +9,7 @@ * REGS(letter, register_mask) */ REGS('r', ALL_GENERAL_REGS) +REGS('o', ALL_GENERAL_REGS & 0xAAAAAAAAu) /* odd registers */ REGS('v', ALL_VECTOR_REGS) /* diff --git a/tcg/ppc/tcg-target.h b/tcg/ppc/tcg-target.h index 0914380bd7..204b70f86a 100644 --- a/tcg/ppc/tcg-target.h +++ b/tcg/ppc/tcg-target.h @@ -149,7 +149,8 @@ extern bool have_vsx; #define TCG_TARGET_HAS_mulsh_i64 1 #endif -#define TCG_TARGET_HAS_qemu_ldst_i128 0 +#define TCG_TARGET_HAS_qemu_ldst_i128 \ + (TCG_TARGET_REG_BITS == 64 && have_isa_2_07) /* * While technically Altivec could support V64, it has no 64-bit store diff --git a/tcg/ppc/tcg-target.c.inc b/tcg/ppc/tcg-target.c.inc index b5c49895f3..c3a1527856 100644 --- a/tcg/ppc/tcg-target.c.inc +++ b/tcg/ppc/tcg-target.c.inc @@ -295,25 +295,27 @@ static bool tcg_target_const_match(int64_t val, TCGType type, int ct) #define B OPCD( 18) #define BC OPCD( 16) + #define LBZ OPCD( 34) #define LHZ OPCD( 40) #define LHA OPCD( 42) #define LWZ OPCD( 32) #define LWZUX XO31( 55) -#define STB OPCD( 38) -#define STH OPCD( 44) -#define STW OPCD( 36) - -#define STD XO62( 0) -#define STDU XO62( 1) -#define STDX XO31(149) - #define LD XO58( 0) #define LDX XO31( 21) #define LDU XO58( 1) #define LDUX XO31( 53) #define LWA XO58( 2) #define LWAX XO31(341) +#define LQ OPCD( 56) + +#define STB OPCD( 38) +#define STH OPCD( 44) +#define STW OPCD( 36) +#define STD XO62( 0) +#define STDU XO62( 1) +#define STDX XO31(149) +#define STQ XO62( 2) #define ADDIC OPCD( 12) #define ADDI OPCD( 14) @@ -2020,7 +2022,18 @@ typedef struct { bool tcg_target_has_memory_bswap(MemOp memop) { - return true; + TCGAtomAlign aa; + + if ((memop & MO_SIZE) <= MO_64) { + return true; + } + + /* + * Reject 16-byte memop with 16-byte atomicity, + * but do allow a pair of 64-bit operations. + */ + aa = atom_and_align_for_opc(tcg_ctx, memop, MO_ATOM_IFALIGN, true); + return aa.atom <= MO_64; } /* @@ -2035,7 +2048,7 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, { TCGLabelQemuLdst *ldst = NULL; MemOp opc = get_memop(oi); - MemOp a_bits; + MemOp a_bits, s_bits; /* * Book II, Section 1.4, Single-Copy Atomicity, specifies: @@ -2047,10 +2060,11 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, * As of 3.0, "the non-atomic access is performed as described in * the corresponding list", which matches MO_ATOM_SUBALIGN. */ + s_bits = opc & MO_SIZE; h->aa = atom_and_align_for_opc(s, opc, have_isa_3_00 ? MO_ATOM_SUBALIGN : MO_ATOM_IFALIGN, - false); + s_bits == MO_128); a_bits = h->aa.align; #ifdef CONFIG_SOFTMMU @@ -2060,7 +2074,6 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, int fast_off = TLB_MASK_TABLE_OFS(mem_index); int mask_off = fast_off + offsetof(CPUTLBDescFast, mask); int table_off = fast_off + offsetof(CPUTLBDescFast, table); - unsigned s_bits = opc & MO_SIZE; ldst = new_ldst_label(s); ldst->is_ld = is_ld; @@ -2303,6 +2316,70 @@ static void tcg_out_qemu_st(TCGContext *s, TCGReg datalo, TCGReg datahi, } } +static TCGLabelQemuLdst * +prepare_host_addr_index_only(TCGContext *s, HostAddress *h, TCGReg addr_reg, + MemOpIdx oi, bool is_ld) +{ + TCGLabelQemuLdst *ldst; + + ldst = prepare_host_addr(s, h, addr_reg, -1, oi, true); + + /* Compose the final address, as LQ/STQ have no indexing. */ + if (h->base != 0) { + tcg_out32(s, ADD | TAB(TCG_REG_TMP1, h->base, h->index)); + h->index = TCG_REG_TMP1; + h->base = 0; + } + + return ldst; +} + +static void tcg_out_qemu_ldst_i128(TCGContext *s, TCGReg datalo, TCGReg datahi, + TCGReg addr_reg, MemOpIdx oi, bool is_ld) +{ + TCGLabelQemuLdst *ldst; + HostAddress h; + bool need_bswap; + uint32_t insn; + + ldst = prepare_host_addr_index_only(s, &h, addr_reg, oi, is_ld); + need_bswap = get_memop(oi) & MO_BSWAP; + + if (h.aa.atom == MO_128) { + tcg_debug_assert(!need_bswap); + tcg_debug_assert(datalo & 1); + tcg_debug_assert(datahi == datalo - 1); + insn = is_ld ? LQ : STQ; + tcg_out32(s, insn | TAI(datahi, h.index, 0)); + } else { + TCGReg d1, d2; + + if (HOST_BIG_ENDIAN ^ need_bswap) { + d1 = datahi, d2 = datalo; + } else { + d1 = datalo, d2 = datahi; + } + + if (need_bswap) { + tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_R0, 8); + insn = is_ld ? LDBRX : STDBRX; + tcg_out32(s, insn | TAB(d1, 0, h.index)); + tcg_out32(s, insn | TAB(d2, h.index, TCG_REG_R0)); + } else { + insn = is_ld ? LD : STD; + tcg_out32(s, insn | TAI(d1, h.index, 0)); + tcg_out32(s, insn | TAI(d2, h.index, 8)); + } + } + + if (ldst) { + ldst->type = TCG_TYPE_I128; + ldst->datalo_reg = datalo; + ldst->datahi_reg = datahi; + ldst->raddr = tcg_splitwx_to_rx(s->code_ptr); + } +} + static void tcg_out_nop_fill(tcg_insn_unit *p, int count) { int i; @@ -2853,6 +2930,11 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc, args[4], TCG_TYPE_I64); } break; + case INDEX_op_qemu_ld_i128: + tcg_debug_assert(TCG_TARGET_REG_BITS == 64); + tcg_out_qemu_ldst_i128(s, args[0], args[1], args[2], args[3], true); + break; + case INDEX_op_qemu_st_i32: if (TCG_TARGET_REG_BITS >= TARGET_LONG_BITS) { tcg_out_qemu_st(s, args[0], -1, args[1], -1, @@ -2874,6 +2956,10 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc, args[4], TCG_TYPE_I64); } break; + case INDEX_op_qemu_st_i128: + tcg_debug_assert(TCG_TARGET_REG_BITS == 64); + tcg_out_qemu_ldst_i128(s, args[0], args[1], args[2], args[3], false); + break; case INDEX_op_setcond_i32: tcg_out_setcond(s, TCG_TYPE_I32, args[3], args[0], args[1], args[2], @@ -3709,6 +3795,11 @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) : TARGET_LONG_BITS == 32 ? C_O0_I3(r, r, r) : C_O0_I4(r, r, r, r)); + case INDEX_op_qemu_ld_i128: + return C_O2_I1(o, m, r); + case INDEX_op_qemu_st_i128: + return C_O0_I3(o, m, r); + case INDEX_op_add_vec: case INDEX_op_sub_vec: case INDEX_op_mul_vec: From patchwork Mon May 15 14:33:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 681973 Delivered-To: patch@linaro.org Received: by 2002:adf:fd8f:0:0:0:0:0 with SMTP id d15csp1152926wrr; Mon, 15 May 2023 07:45:01 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4h8rKEEwbwICmUxi5SRkXLVQOVuO/HYLB/UGx7l/31Wrg2T5dvJFaMpCJZ+a0umiRw9mad X-Received: by 2002:a05:6000:510:b0:306:2fab:1f81 with SMTP id a16-20020a056000051000b003062fab1f81mr26083586wrf.21.1684161901361; Mon, 15 May 2023 07:45:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161901; cv=none; d=google.com; s=arc-20160816; b=q1smsw2EXSNvYPPX14YtRuBeUg/lzsXoXlfpbJVGAqFbx1iadDRVfYaoL92jCfUy6t nfox8SiOgkz9wCJLzsrQ8+AssWV4DypJBZ07Ukmj0JMOyJDU+qx2LC76Pq/qVsIywdMY REeoDiqtpvwkAdA+4FXlH84dxpP9jy+nplCLhn9XpK6WUd5nhFrMJ+/rCJZLXGJ7KuEn kQXmI99qZhJk761tIi5j7N2Ahxwck254ZXFuhnsMyhpJP3nzf626WWNiKSlw9X+l7A7k /w+Vs0gv4KAl75TzW48TnZyf33F9US4vTwMq3NpB2G9TI6V+Sz30QuhKwUwjbdRJfByK sN+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=BwdovVLSIJ9sUIX28iM3Bk5pQGlPR2EuvtD4XWozcso=; b=zg7RhzkHDQGyPkWUG1gLTxzAjPZkXmhx5TDa8IlJAyBO6C97U+jD35qAQPXp5rk2Kn i+VvBwgh9zDLgP7LmVUzjalOzKiCKDgUJ0oOk0FzniM5XZZ/3iYUmjSz+jysnjm1GwTE 7XrT2tshgtCR4ICXUitlxjDxgkaQmJ8inb5aFxaacxKmfOaW/oDSUXMBGkNXo7gchOFu /CqkqK527o2nOB8gCrhybP1ZlHFiVQ47flhXGwwLCq8gqK52f6vKCHCcM1JRsIRKejJp VwCgVjAxYeJJaiui5LAHeKIpWR5h36y4J0ARcGr0sHVvNAmremUWWpqGukqKPI08i4YJ Tqpw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=toLjq8gX; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id t17-20020a5d6a51000000b00306281b8a84si15714243wrw.617.2023.05.15.07.45.01 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 May 2023 07:45:01 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=toLjq8gX; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pyZM3-0002LO-9D; Mon, 15 May 2023 10:39:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pyZK6-0005tg-GO for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:20 -0400 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pyZJs-0005yx-VG for qemu-devel@nongnu.org; Mon, 15 May 2023 10:37:13 -0400 Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-1ab05018381so118708775ad.2 for ; Mon, 15 May 2023 07:37:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684161420; x=1686753420; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BwdovVLSIJ9sUIX28iM3Bk5pQGlPR2EuvtD4XWozcso=; b=toLjq8gXa23rau//UMrLnw38baXSNt94IJiVDgLDLpMgMU7EUu1tTfH1/lz8y727TD G+3SnrFnl5w8FqKSP+qQbyAOSSpQVcY0tt+TRmNg4UYZ6VTfy2fT07VVETF6zy7Xn95/ 9O6BNYqJwzD0vbFUV6RCCRw0YYYdCS8W0XN8mSKO01xqmdqIccZMd2I1OXOKa2Kr7p64 Km53CUmAG+WzwG1NtOoMeX/S/Z7VNLfIm20YyYblN8E5Hll8WdSUH2CMJA0UMfePyIL8 E+TwNk5QYYZvIfX3ekGJCJc+K8tdF43tCan85HxczgPQCLkcZU8Si4UM4pMX9FkZLkwI ZJMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684161420; x=1686753420; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BwdovVLSIJ9sUIX28iM3Bk5pQGlPR2EuvtD4XWozcso=; b=X24tbu5tS28gmaslOu6bclipORtiZNagztoa5zW/kZblVMOu5tZe8jPE6pCtFhy/jM 8aqZqsWe6JNPlvYBuhWMlAvLdkyj65lG/QcO+ho4TipQWWb7eb9ySu3k1qctZ3anhaQz //cFRrKlq9fo6cndXzGxQYKqNV3ThdXVDHoRRTaycEbOiKThPDp2hi+pA1S4Bx3VHhR5 eRHqyHZgAt+8hdVdxC3DDJtSm64Oh64V134CcoMojQWSzgHNNIBip/sdoJedzHsDoUWj mm+Q58ZnSTcolNPfBJ/TPwDKELMjMmZ9V5AVhXWPNWKI/FXSjKjmL4y05qSPiiF/xDBA EWJA== X-Gm-Message-State: AC+VfDxzj8V2E83DJtmKqBOBUcu80l4PKtHOC5M/Jd3QVLIeis1IQ3Gz 3viCeaQedHoq9d4EbSRSjrclOWwzRQR9+vFzNAI= X-Received: by 2002:a17:902:e5ce:b0:1ae:e2b:3ad8 with SMTP id u14-20020a170902e5ce00b001ae0e2b3ad8mr5885732plf.66.1684161420258; Mon, 15 May 2023 07:37:00 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:9902:96ac:8d8c:4366]) by smtp.gmail.com with ESMTPSA id i6-20020a1709026ac600b001aadfdfcd06sm8942735plt.299.2023.05.15.07.36.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 07:36:59 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, qemu-s390x@nongnu.org Subject: [PATCH v5 54/54] tcg/s390x: Support 128-bit load/store Date: Mon, 15 May 2023 07:33:13 -0700 Message-Id: <20230515143313.734053-55-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230515143313.734053-1-richard.henderson@linaro.org> References: <20230515143313.734053-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62e; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Use LPQ/STPQ when 16-byte atomicity is required. Note that these instructions require 16-byte alignment. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- tcg/s390x/tcg-target-con-set.h | 2 + tcg/s390x/tcg-target.h | 2 +- tcg/s390x/tcg-target.c.inc | 103 ++++++++++++++++++++++++++++++++- 3 files changed, 103 insertions(+), 4 deletions(-) diff --git a/tcg/s390x/tcg-target-con-set.h b/tcg/s390x/tcg-target-con-set.h index ecc079bb6d..cbad91b2b5 100644 --- a/tcg/s390x/tcg-target-con-set.h +++ b/tcg/s390x/tcg-target-con-set.h @@ -14,6 +14,7 @@ C_O0_I2(r, r) C_O0_I2(r, ri) C_O0_I2(r, rA) C_O0_I2(v, r) +C_O0_I3(o, m, r) C_O1_I1(r, r) C_O1_I1(v, r) C_O1_I1(v, v) @@ -36,6 +37,7 @@ C_O1_I2(v, v, v) C_O1_I3(v, v, v, v) C_O1_I4(r, r, ri, rI, r) C_O1_I4(r, r, rA, rI, r) +C_O2_I1(o, m, r) C_O2_I2(o, m, 0, r) C_O2_I2(o, m, r, r) C_O2_I3(o, m, 0, 1, r) diff --git a/tcg/s390x/tcg-target.h b/tcg/s390x/tcg-target.h index 170007bea5..ec96952172 100644 --- a/tcg/s390x/tcg-target.h +++ b/tcg/s390x/tcg-target.h @@ -140,7 +140,7 @@ extern uint64_t s390_facilities[3]; #define TCG_TARGET_HAS_muluh_i64 0 #define TCG_TARGET_HAS_mulsh_i64 0 -#define TCG_TARGET_HAS_qemu_ldst_i128 0 +#define TCG_TARGET_HAS_qemu_ldst_i128 1 #define TCG_TARGET_HAS_v64 HAVE_FACILITY(VECTOR) #define TCG_TARGET_HAS_v128 HAVE_FACILITY(VECTOR) diff --git a/tcg/s390x/tcg-target.c.inc b/tcg/s390x/tcg-target.c.inc index 8e34b214fc..835daa51fa 100644 --- a/tcg/s390x/tcg-target.c.inc +++ b/tcg/s390x/tcg-target.c.inc @@ -243,6 +243,7 @@ typedef enum S390Opcode { RXY_LLGF = 0xe316, RXY_LLGH = 0xe391, RXY_LMG = 0xeb04, + RXY_LPQ = 0xe38f, RXY_LRV = 0xe31e, RXY_LRVG = 0xe30f, RXY_LRVH = 0xe31f, @@ -253,6 +254,7 @@ typedef enum S390Opcode { RXY_STG = 0xe324, RXY_STHY = 0xe370, RXY_STMG = 0xeb24, + RXY_STPQ = 0xe38e, RXY_STRV = 0xe33e, RXY_STRVG = 0xe32f, RXY_STRVH = 0xe33f, @@ -1577,7 +1579,18 @@ typedef struct { bool tcg_target_has_memory_bswap(MemOp memop) { - return true; + TCGAtomAlign aa; + + if ((memop & MO_SIZE) <= MO_64) { + return true; + } + + /* + * Reject 16-byte memop with 16-byte atomicity, + * but do allow a pair of 64-bit operations. + */ + aa = atom_and_align_for_opc(tcg_ctx, memop, MO_ATOM_IFALIGN, true); + return aa.atom <= MO_64; } static void tcg_out_qemu_ld_direct(TCGContext *s, MemOp opc, TCGReg data, @@ -1734,13 +1747,13 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h, { TCGLabelQemuLdst *ldst = NULL; MemOp opc = get_memop(oi); + MemOp s_bits = opc & MO_SIZE; unsigned a_mask; - h->aa = atom_and_align_for_opc(s, opc, MO_ATOM_IFALIGN, false); + h->aa = atom_and_align_for_opc(s, opc, MO_ATOM_IFALIGN, s_bits == MO_128); a_mask = (1 << h->aa.align) - 1; #ifdef CONFIG_SOFTMMU - unsigned s_bits = opc & MO_SIZE; unsigned s_mask = (1 << s_bits) - 1; int mem_index = get_mmuidx(oi); int fast_off = TLB_MASK_TABLE_OFS(mem_index); @@ -1865,6 +1878,80 @@ static void tcg_out_qemu_st(TCGContext* s, TCGReg data_reg, TCGReg addr_reg, } } +static void tcg_out_qemu_ldst_i128(TCGContext *s, TCGReg datalo, TCGReg datahi, + TCGReg addr_reg, MemOpIdx oi, bool is_ld) +{ + TCGLabel *l1 = NULL, *l2 = NULL; + TCGLabelQemuLdst *ldst; + HostAddress h; + bool need_bswap; + bool use_pair; + S390Opcode insn; + + ldst = prepare_host_addr(s, &h, addr_reg, oi, is_ld); + + use_pair = h.aa.atom < MO_128; + need_bswap = get_memop(oi) & MO_BSWAP; + + if (!use_pair) { + /* + * Atomicity requires we use LPQ. If we've already checked for + * 16-byte alignment, that's all we need. If we arrive with + * lesser alignment, we have determined that less than 16-byte + * alignment can be satisfied with two 8-byte loads. + */ + if (h.aa.align < MO_128) { + use_pair = true; + l1 = gen_new_label(); + l2 = gen_new_label(); + + tcg_out_insn(s, RI, TMLL, addr_reg, 15); + tgen_branch(s, 7, l1); /* CC in {1,2,3} */ + } + + tcg_debug_assert(!need_bswap); + tcg_debug_assert(datalo & 1); + tcg_debug_assert(datahi == datalo - 1); + insn = is_ld ? RXY_LPQ : RXY_STPQ; + tcg_out_insn_RXY(s, insn, datahi, h.base, h.index, h.disp); + + if (use_pair) { + tgen_branch(s, S390_CC_ALWAYS, l2); + tcg_out_label(s, l1); + } + } + if (use_pair) { + TCGReg d1, d2; + + if (need_bswap) { + d1 = datalo, d2 = datahi; + insn = is_ld ? RXY_LRVG : RXY_STRVG; + } else { + d1 = datahi, d2 = datalo; + insn = is_ld ? RXY_LG : RXY_STG; + } + + if (h.base == d1 || h.index == d1) { + tcg_out_insn(s, RXY, LAY, TCG_TMP0, h.base, h.index, h.disp); + h.base = TCG_TMP0; + h.index = TCG_REG_NONE; + h.disp = 0; + } + tcg_out_insn_RXY(s, insn, d1, h.base, h.index, h.disp); + tcg_out_insn_RXY(s, insn, d2, h.base, h.index, h.disp + 8); + } + if (l2) { + tcg_out_label(s, l2); + } + + if (ldst) { + ldst->type = TCG_TYPE_I128; + ldst->datalo_reg = datalo; + ldst->datahi_reg = datahi; + ldst->raddr = tcg_splitwx_to_rx(s->code_ptr); + } +} + static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0) { /* Reuse the zeroing that exists for goto_ptr. */ @@ -2222,6 +2309,12 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc, case INDEX_op_qemu_st_i64: tcg_out_qemu_st(s, args[0], args[1], args[2], TCG_TYPE_I64); break; + case INDEX_op_qemu_ld_i128: + tcg_out_qemu_ldst_i128(s, args[0], args[1], args[2], args[3], true); + break; + case INDEX_op_qemu_st_i128: + tcg_out_qemu_ldst_i128(s, args[0], args[1], args[2], args[3], false); + break; case INDEX_op_ld16s_i64: tcg_out_mem(s, 0, RXY_LGH, args[0], args[1], TCG_REG_NONE, args[2]); @@ -3099,6 +3192,10 @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_qemu_st_i64: case INDEX_op_qemu_st_i32: return C_O0_I2(r, r); + case INDEX_op_qemu_ld_i128: + return C_O2_I1(o, m, r); + case INDEX_op_qemu_st_i128: + return C_O0_I3(o, m, r); case INDEX_op_deposit_i32: case INDEX_op_deposit_i64: