From patchwork Wed Mar 11 06:44:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 184384 Delivered-To: patch@linaro.org Received: by 2002:a92:1f12:0:0:0:0:0 with SMTP id i18csp277263ile; Tue, 10 Mar 2020 23:44:53 -0700 (PDT) X-Google-Smtp-Source: ADFU+vu15NCU4ReopuxaVgzFwS3kCj+cV5zE1IbQEF6dld5enzmh77m7cMc8CGCI88lLL4U2Juy2 X-Received: by 2002:ac8:4cd1:: with SMTP id l17mr1357877qtv.165.1583909093479; Tue, 10 Mar 2020 23:44:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1583909093; cv=none; d=google.com; s=arc-20160816; b=PDaSACRDgx8ZpRzNa08IoNWwAxuWuOgQeRnkyn3w414vFKEHZEmU/jUR+7h7scrvF2 l6/63w5W/9b8Jf8hU229ekTZ2M43cJeQ9eDucZ8mbcs0kHOVL5GLITKm1DcLsEFGi4UM NsXEpEx/Of0T+1SJvgqr00CBQ1EEM16Xvbny4SRrXxXFqEDgT3twpXpvSLs3C+DB44wn Xq0+PzXqEDHUtGNf9ZVHXjG5nP/LdaypNK1R+9bvjQ9ZI6eEz0zAnAhgOjZZJ/cVl+m0 mekEG7sDa1TdUfhIkmIfvs2sPyO6v1XlR0Ga7N7TcoQ8Paj3oUUGEfhxreJttA9n/Tde oTqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=ME4wdyObWWzU36JOWOP068GfXM/kHUGKiy3+FE+Caek=; b=vxONop2FculRaC/mTgRVEcR/jdHGX4Vb19eOV5UBvcB80zxg/8ybAfarEc8419+Gpf 4Pz5ipKh0NrLSN5p5ZTPaJcr8LFVnfTS9QUlpGPB9co5X68ipl16AkE1XRvIXxW3PhOh 72/Fup2yqyXeFzW32Q5lKs2G4lpLomQQdZ+JOrBVlT8XQ/3eYVBK0NmpszTaK19n2S3N TUV//jIvSF7FrsuQwsmQsVL9+zvOlu0i8+XjM30XXpfJOYv3R99py70UpkzCQXyLCElL t0KwIOtUWqfo+z7z/XUoO5YYZ1v/6zpx+dzOpYCMc5mODvLhFGBLHxxrvSoAmAAzXwFq ZLWw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=nAqcHY80; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id w14si482183qki.211.2020.03.10.23.44.53 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Mar 2020 23:44:53 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=nAqcHY80; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:46170 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6q-00068B-UQ for patch@linaro.org; Wed, 11 Mar 2020 02:44:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59301) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6Q-000665-Es for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6O-0001f1-Hm for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:26 -0400 Received: from mail-pg1-x541.google.com ([2607:f8b0:4864:20::541]:38245) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6O-0001cs-Cw for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:24 -0400 Received: by mail-pg1-x541.google.com with SMTP id x7so639014pgh.5 for ; Tue, 10 Mar 2020 23:44:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ME4wdyObWWzU36JOWOP068GfXM/kHUGKiy3+FE+Caek=; b=nAqcHY80e4ejxIJMyR8NQ/oenp6q+JKoqoSjzsnDo3V0irsukHBJXuIFSZMeH2FA91 nqThih0v1cRFxjbr0+jn2U6EUvJt+351gAGkWdsQAusWjD+k7ZaTRrMfVcBzFUqudBDs 22boIFp7rItiG50EB5sHmTOm72nPPhIJS06g+UEHGhfUS73ZT/FP6OTlWitNLXQlBEzr 7TVgPHIA4hzZx87mHNTDbqn3DTtgSD9DTLzNLd578xOMPPNqmAivDyftR8KhKM84zPxh OVkWZsWp3ZHMK9Ce/V6QQEVWr58hC6Lnr3t+DW6KHbHRF0DdT54SXtBgkpMQSNF8FK7z LIfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ME4wdyObWWzU36JOWOP068GfXM/kHUGKiy3+FE+Caek=; b=nU5XCeWwXbqhto6JeU5oamSSXxTywmXgW6mm5lolLlmn26B1EiR7DOqggSL5AMxzVd VeFi/l2VU+9eCpMlD50VHsizCC/ZGbB2tIJ6kRp7lrWHJSDsnITp0m86q2W8CRiPiSdI /4ZOw7Z630bQ0sbqSj3+XmZW7RHwkpkqJ3UV0sHzCCDlzKGZiJ8fJwFdtbZ8JZBH8QzN jt5ucceGpM5Ngu8oOA221MKBTZlra12d2ivv/EfJ//t2yZMV6/GCTQQpH1kuBcnxj2RX Immfv2yJ6TyCD+nIqXX+f5WOIelmrwkYnL2okQWLnMBluTZpN2Vjc+Q30/BOgnOuK31E maJw== X-Gm-Message-State: ANhLgQ26GrGurwQBKKAYmJF1gs+xc7ZpaBFw65gdeS+BsAXTabOLp9cx 3tubebmVIU3TfBmQye0euEnr+4TlD7w= X-Received: by 2002:a62:7cc9:: with SMTP id x192mr767250pfc.176.1583909062988; Tue, 10 Mar 2020 23:44:22 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:22 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 01/16] accel/tcg: Add block comment for probe_access Date: Tue, 10 Mar 2020 23:44:05 -0700 Message-Id: <20200311064420.30606-2-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::541 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- include/exec/exec-all.h | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) -- 2.20.1 Reviewed-by: Peter Maydell diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index 350c4b451b..d656a1f05c 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -330,6 +330,23 @@ static inline void tlb_flush_by_mmuidx_all_cpus_synced(CPUState *cpu, { } #endif +/** + * probe_access: + * @env: CPUArchState + * @addr: guest virtual address to look up + * @size: size of the access + * @access_type: read, write or execute permission + * @mmu_idx: MMU index to use for lookup + * @retaddr: return address for unwinding + * + * Look up the guest virtual address @addr. Raise an exception if the + * page does not satisfy @access_type. Raise an exception if the + * access (@addr, @size) hits a watchpoint. For writes, mark a clean + * page as dirty. + * + * Finally, return the host address for a page that is backed by RAM, + * or NULL if the page requires I/O. + */ void *probe_access(CPUArchState *env, target_ulong addr, int size, MMUAccessType access_type, int mmu_idx, uintptr_t retaddr); From patchwork Wed Mar 11 06:44:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 184385 Delivered-To: patch@linaro.org Received: by 2002:a92:1f12:0:0:0:0:0 with SMTP id i18csp277296ile; Tue, 10 Mar 2020 23:44:56 -0700 (PDT) X-Google-Smtp-Source: ADFU+vtlAY7AMMoVIozWzpDsedr7UY8gJQHR+SUk7UQp2Nb/vxLtR3lZ/BuPNQ4s+9c1hDJuXR/R X-Received: by 2002:ac8:4e4d:: with SMTP id e13mr1364603qtw.272.1583909096291; Tue, 10 Mar 2020 23:44:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1583909096; cv=none; d=google.com; s=arc-20160816; b=kmSsvL6Ml+pjsh8PJWc58bNrXu4mg5tJQttRNeEPPIoum/OyiGs6Jv6BV01UXLS88U t5NguW2di30FHh2fUOMUfncVaaEY8ydWjquK0t55lRvQ+s+44iHgOIG1kogEPpEI0Hn1 Ny3oMwdVH8PlheAj9PQAcdDEhDWfAyKfqmIgvPTKtJGtvqpjdUTDf9g8bLc6wM4eTVJX h1w42XC2ZRd3wQwu/x6CzEIKHjM018NtGx2x9GydCpCxAwQQ1nmAgKKv8meMs2kDFLaw 0tHvXUCOalYCWsC1RRLvE5FZQhATHE8ULAUCw37e91UPOo4ITLz2BkxWlnBNO+syuzFt ZAww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=WFkeQZK5wV/2b9hgN0NoiBdZ/Pf2pozUeegzyTZXJXc=; b=CZmOfzgV+W4uYVs1mocnZULWWfejOqWojlJjJNDUBGGEi/4ilc0FUDH+Bs9qg/pszd XyRf19OTJeHp5eKklP/xkRQP+54rJMMURsXYCRrj7RGugUHt73BYVhbM8gZcIj5CIR9d fIVE7+yV95YXwRYXbj3D6Y9OgZJQZyNA8j/IM9aIeobe6WpWvyGDzLKBF+ocm4tagvOL 90zZyn1UV6iqW4rHXSOUjwWiUy7CJpFpLlSVZfkKagAH0s6+b7VSEynNCR9y+kCvF3Lu IKx+GdZhJ54Cy1Inn0Bt5Ksz9/2GkHWfkx+ON9HxA56Nx1yLPo84EljWaAPQr4/I2/We 7Vqw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="faxQT0/T"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id g23si494152qka.92.2020.03.10.23.44.56 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Mar 2020 23:44:56 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="faxQT0/T"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:46222 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6t-0006DK-LT for patch@linaro.org; Wed, 11 Mar 2020 02:44:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59385) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6V-00069K-US for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6S-0001mW-5h for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:30 -0400 Received: from mail-pg1-x544.google.com ([2607:f8b0:4864:20::544]:39444) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6Q-0001hB-D0 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:27 -0400 Received: by mail-pg1-x544.google.com with SMTP id s2so637509pgv.6 for ; Tue, 10 Mar 2020 23:44:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WFkeQZK5wV/2b9hgN0NoiBdZ/Pf2pozUeegzyTZXJXc=; b=faxQT0/TZ2lP8LdufRbHEAMgV5SqtHpRoMMBYOjXzi8/oIgSE3khGexXHAGUuNG0TG HBFx4yEsGDAuJfnSqyKCTqX3yy0UoII0WkRISbXRIvE1r5FuUy9PrM/cMnn7z5BD+ZoR Sm/7bl3KmX+FCEFDGgjYlJwxMRzggqNhG4pbEjEyH0aRzMhXhNnE75kJ3lv8fHbm7VNx gkOmIqWNgU1ALaPxbOXA/XouNF/Ej3x7j+H02mErdIAckpsh8fhVcNPDUlTBbQSyk+BU tqlm0WdPXZ0rXAXQsxHs4dVKNUX76eREDFyqqAzVgXlSMu0Rg7S4Xq2tI64LGIpxVXO5 9FXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WFkeQZK5wV/2b9hgN0NoiBdZ/Pf2pozUeegzyTZXJXc=; b=K31lWmGYEx0M7T/8X0rwbiiOfC/rQykIM7ng0gzcQyri5UwLgjV3pZpjxN6QpyDgs2 Cu9VX2kjqaS4++1ou4gX40nGfQnJqLcFV+Ypb+5/Ybmo6UMUNK2TeWrRo2r+Ufx9jTSP m+uLucfW6WQJ8L3yK/AK0Gr+uK3MzjJnCtodooZeUQr36WjMx5xJbWTqJjh7bkMwOhM1 m9dgTjK1K5Xe+fxQ0t0qNyLv0nYojG4/W+cmwE/pw7gSaq19Nin23N3pnUuOEAD8o6M9 NN9Oq84Jr+Eva3e5/r78iov6ixA1g3JG3lmF9bsPKaZif2tcC+6YRLywkCUpwyIBXwrR 2IJw== X-Gm-Message-State: ANhLgQ1w+RbMvjbhvItVqAJPmik5giBmKTfAgQBZEVhpS5f6yLnrS5JC hefIslMFGGDC1JEV7VJR7lesvzZlEEM= X-Received: by 2002:a63:a65:: with SMTP id z37mr1405166pgk.31.1583909064289; Tue, 10 Mar 2020 23:44:24 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:23 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 02/16] accel/tcg: Add probe_access_flags Date: Tue, 10 Mar 2020 23:44:06 -0700 Message-Id: <20200311064420.30606-3-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::544 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" This new interface will allow targets to probe for a page and then handle watchpoints themselves. This will be most useful for vector predicated memory operations, where one page lookup can be used for many operations, and one test can avoid many watchpoint checks. Signed-off-by: Richard Henderson --- include/exec/cpu-all.h | 13 ++- include/exec/exec-all.h | 22 +++++ accel/tcg/cputlb.c | 178 ++++++++++++++++++++-------------------- accel/tcg/user-exec.c | 36 +++++--- 4 files changed, 150 insertions(+), 99 deletions(-) -- 2.20.1 diff --git a/include/exec/cpu-all.h b/include/exec/cpu-all.h index 49384bb66a..43ddcf024c 100644 --- a/include/exec/cpu-all.h +++ b/include/exec/cpu-all.h @@ -328,7 +328,18 @@ CPUArchState *cpu_copy(CPUArchState *env); | CPU_INTERRUPT_TGT_EXT_3 \ | CPU_INTERRUPT_TGT_EXT_4) -#if !defined(CONFIG_USER_ONLY) +#ifdef CONFIG_USER_ONLY + +/* + * Allow some level of source compatibility with softmmu. We do not + * support any of the more exotic features, so only invalid pages may + * be signaled by probe_access_flags(). + */ +#define TLB_INVALID_MASK (1 << (TARGET_PAGE_BITS_MIN - 1)) +#define TLB_MMIO 0 +#define TLB_WATCHPOINT 0 + +#else /* * Flags stored in the low bits of the TLB virtual address. diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index d656a1f05c..8792bea07a 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -362,6 +362,28 @@ static inline void *probe_read(CPUArchState *env, target_ulong addr, int size, return probe_access(env, addr, size, MMU_DATA_LOAD, mmu_idx, retaddr); } +/** + * probe_access_flags: + * @env: CPUArchState + * @addr: guest virtual address to look up + * @access_type: read, write or execute permission + * @mmu_idx: MMU index to use for lookup + * @nonfault: suppress the fault + * @phost: return value for host address + * @retaddr: return address for unwinding + * + * Similar to probe_access, loosely returning the TLB_FLAGS_MASK for + * the page, and storing the host address for RAM in @phost. + * + * If @nonfault is set, do not raise an exception but return TLB_INVALID_MASK. + * Do not handle watchpoints, but include TLB_WATCHPOINT in the returned flags. + * Do handle clean pages, so exclude TLB_NOTDIRY from the returned flags. + * For simplicity, all "mmio-like" flags are folded to TLB_MMIO. + */ +int probe_access_flags(CPUArchState *env, target_ulong addr, + MMUAccessType access_type, int mmu_idx, + bool nonfault, void **phost, uintptr_t retaddr); + #define CODE_GEN_ALIGN 16 /* must be >= of the size of a icache line */ /* Estimated block size for TB allocation. */ diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index e3b5750c3b..687474ccea 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -1231,86 +1231,16 @@ static void notdirty_write(CPUState *cpu, vaddr mem_vaddr, unsigned size, } } -/* - * Probe for whether the specified guest access is permitted. If it is not - * permitted then an exception will be taken in the same way as if this - * were a real access (and we will not return). - * If the size is 0 or the page requires I/O access, returns NULL; otherwise, - * returns the address of the host page similar to tlb_vaddr_to_host(). - */ -void *probe_access(CPUArchState *env, target_ulong addr, int size, - MMUAccessType access_type, int mmu_idx, uintptr_t retaddr) +static int probe_access_internal(CPUArchState *env, target_ulong addr, + int fault_size, MMUAccessType access_type, + int mmu_idx, bool nonfault, + void **phost, uintptr_t retaddr) { uintptr_t index = tlb_index(env, mmu_idx, addr); CPUTLBEntry *entry = tlb_entry(env, mmu_idx, addr); - target_ulong tlb_addr; - size_t elt_ofs; - int wp_access; - - g_assert(-(addr | TARGET_PAGE_MASK) >= size); - - switch (access_type) { - case MMU_DATA_LOAD: - elt_ofs = offsetof(CPUTLBEntry, addr_read); - wp_access = BP_MEM_READ; - break; - case MMU_DATA_STORE: - elt_ofs = offsetof(CPUTLBEntry, addr_write); - wp_access = BP_MEM_WRITE; - break; - case MMU_INST_FETCH: - elt_ofs = offsetof(CPUTLBEntry, addr_code); - wp_access = BP_MEM_READ; - break; - default: - g_assert_not_reached(); - } - tlb_addr = tlb_read_ofs(entry, elt_ofs); - - if (unlikely(!tlb_hit(tlb_addr, addr))) { - if (!victim_tlb_hit(env, mmu_idx, index, elt_ofs, - addr & TARGET_PAGE_MASK)) { - tlb_fill(env_cpu(env), addr, size, access_type, mmu_idx, retaddr); - /* TLB resize via tlb_fill may have moved the entry. */ - index = tlb_index(env, mmu_idx, addr); - entry = tlb_entry(env, mmu_idx, addr); - } - tlb_addr = tlb_read_ofs(entry, elt_ofs); - } - - if (!size) { - return NULL; - } - - if (unlikely(tlb_addr & TLB_FLAGS_MASK)) { - CPUIOTLBEntry *iotlbentry = &env_tlb(env)->d[mmu_idx].iotlb[index]; - - /* Reject I/O access, or other required slow-path. */ - if (tlb_addr & (TLB_MMIO | TLB_BSWAP | TLB_DISCARD_WRITE)) { - return NULL; - } - - /* Handle watchpoints. */ - if (tlb_addr & TLB_WATCHPOINT) { - cpu_check_watchpoint(env_cpu(env), addr, size, - iotlbentry->attrs, wp_access, retaddr); - } - - /* Handle clean RAM pages. */ - if (tlb_addr & TLB_NOTDIRTY) { - notdirty_write(env_cpu(env), addr, size, iotlbentry, retaddr); - } - } - - return (void *)((uintptr_t)addr + entry->addend); -} - -void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr, - MMUAccessType access_type, int mmu_idx) -{ - CPUTLBEntry *entry = tlb_entry(env, mmu_idx, addr); - target_ulong tlb_addr, page; + target_ulong tlb_addr, page_addr; size_t elt_ofs; + int flags; switch (access_type) { case MMU_DATA_LOAD: @@ -1325,20 +1255,19 @@ void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr, default: g_assert_not_reached(); } - - page = addr & TARGET_PAGE_MASK; tlb_addr = tlb_read_ofs(entry, elt_ofs); - if (!tlb_hit_page(tlb_addr, page)) { - uintptr_t index = tlb_index(env, mmu_idx, addr); - - if (!victim_tlb_hit(env, mmu_idx, index, elt_ofs, page)) { + page_addr = addr & TARGET_PAGE_MASK; + if (!tlb_hit_page(tlb_addr, page_addr)) { + if (!victim_tlb_hit(env, mmu_idx, index, elt_ofs, page_addr)) { CPUState *cs = env_cpu(env); CPUClass *cc = CPU_GET_CLASS(cs); - if (!cc->tlb_fill(cs, addr, 0, access_type, mmu_idx, true, 0)) { + if (!cc->tlb_fill(cs, addr, fault_size, access_type, + mmu_idx, nonfault, retaddr)) { /* Non-faulting page table read failed. */ - return NULL; + *phost = NULL; + return TLB_INVALID_MASK; } /* TLB resize via tlb_fill may have moved the entry. */ @@ -1346,15 +1275,90 @@ void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr, } tlb_addr = tlb_read_ofs(entry, elt_ofs); } + flags = tlb_addr & TLB_FLAGS_MASK; - if (tlb_addr & ~TARGET_PAGE_MASK) { - /* IO access */ + /* Fold all "mmio-like" bits into TLB_MMIO. This is not RAM. */ + if (unlikely(flags & ~(TLB_WATCHPOINT | TLB_NOTDIRTY))) { + *phost = NULL; + return TLB_MMIO; + } + + /* Everything else is RAM. */ + *phost = (void *)((uintptr_t)addr + entry->addend); + return flags; +} + +int probe_access_flags(CPUArchState *env, target_ulong addr, + MMUAccessType access_type, int mmu_idx, + bool nonfault, void **phost, uintptr_t retaddr) +{ + void *host; + int flags; + + flags = probe_access_internal(env, addr, 0, access_type, mmu_idx, + nonfault, &host, retaddr); + + /* Handle clean RAM pages. */ + if (unlikely(flags & TLB_NOTDIRTY)) { + uintptr_t index = tlb_index(env, mmu_idx, addr); + CPUIOTLBEntry *iotlbentry = &env_tlb(env)->d[mmu_idx].iotlb[index]; + + notdirty_write(env_cpu(env), addr, 1, iotlbentry, retaddr); + flags &= ~TLB_NOTDIRTY; + } + + return flags; +} + +void *probe_access(CPUArchState *env, target_ulong addr, int size, + MMUAccessType access_type, int mmu_idx, uintptr_t retaddr) +{ + void *host; + int flags; + + g_assert(-(addr | TARGET_PAGE_MASK) >= size); + + flags = probe_access_internal(env, addr, size, access_type, mmu_idx, + false, &host, retaddr); + + /* Per the interface, size == 0 merely faults the access. */ + if (size == 0) { return NULL; } - return (void *)((uintptr_t)addr + entry->addend); + if (unlikely(flags & (TLB_NOTDIRTY | TLB_WATCHPOINT))) { + uintptr_t index = tlb_index(env, mmu_idx, addr); + CPUIOTLBEntry *iotlbentry = &env_tlb(env)->d[mmu_idx].iotlb[index]; + + /* Handle clean RAM pages. */ + if (flags & TLB_NOTDIRTY) { + notdirty_write(env_cpu(env), addr, 1, iotlbentry, retaddr); + } + + /* Handle watchpoints. */ + if (flags & TLB_WATCHPOINT) { + int wp_access = (access_type == MMU_DATA_STORE + ? BP_MEM_WRITE : BP_MEM_READ); + cpu_check_watchpoint(env_cpu(env), addr, size, + iotlbentry->attrs, wp_access, retaddr); + } + } + + return host; } +void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr, + MMUAccessType access_type, int mmu_idx) +{ + void *host; + int flags; + + flags = probe_access_internal(env, addr, 0, access_type, + mmu_idx, true, &host, 0); + + /* No combination of flags are expected by the caller. */ + return flags ? NULL : host; +} #ifdef CONFIG_PLUGIN /* diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c index 4be78eb9b3..c52dd8a95a 100644 --- a/accel/tcg/user-exec.c +++ b/accel/tcg/user-exec.c @@ -190,13 +190,12 @@ static inline int handle_cpu_signal(uintptr_t pc, siginfo_t *info, g_assert_not_reached(); } -void *probe_access(CPUArchState *env, target_ulong addr, int size, - MMUAccessType access_type, int mmu_idx, uintptr_t retaddr) +int probe_access_flags(CPUArchState *env, target_ulong addr, + MMUAccessType access_type, int mmu_idx, + bool nonfault, void **phost, uintptr_t retaddr) { int flags; - g_assert(-(addr | TARGET_PAGE_MASK) >= size); - switch (access_type) { case MMU_DATA_STORE: flags = PAGE_WRITE; @@ -211,15 +210,30 @@ void *probe_access(CPUArchState *env, target_ulong addr, int size, g_assert_not_reached(); } - if (!guest_addr_valid(addr) || page_check_range(addr, size, flags) < 0) { - CPUState *cpu = env_cpu(env); - CPUClass *cc = CPU_GET_CLASS(cpu); - cc->tlb_fill(cpu, addr, size, access_type, MMU_USER_IDX, false, - retaddr); - g_assert_not_reached(); + if (!guest_addr_valid(addr) || page_check_range(addr, 1, flags) < 0) { + if (nonfault) { + *phost = NULL; + return TLB_INVALID_MASK; + } else { + CPUState *cpu = env_cpu(env); + CPUClass *cc = CPU_GET_CLASS(cpu); + cc->tlb_fill(cpu, addr, 0, access_type, MMU_USER_IDX, false, retaddr); + g_assert_not_reached(); + } } - return size ? g2h(addr) : NULL; + *phost = g2h(addr); + return 0; +} + +void *probe_access(CPUArchState *env, target_ulong addr, int size, + MMUAccessType access_type, int mmu_idx, uintptr_t retaddr) +{ + void *host; + + g_assert(-(addr | TARGET_PAGE_MASK) >= size); + probe_access_flags(env, addr, access_type, mmu_idx, false, &host, retaddr); + return host; } #if defined(__i386__) From patchwork Wed Mar 11 06:44:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 184391 Delivered-To: patch@linaro.org Received: by 2002:a92:1f12:0:0:0:0:0 with SMTP id i18csp280952ile; Tue, 10 Mar 2020 23:49:49 -0700 (PDT) X-Google-Smtp-Source: ADFU+vvLTe19t/svn6sKetGKy4JXUWf8xSlwxbzdErhL7jPXXZWsAL2KEumQIRd+5VL6N41WdhyR X-Received: by 2002:ac8:6f70:: with SMTP id u16mr153833qtv.355.1583909389348; Tue, 10 Mar 2020 23:49:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1583909389; cv=none; d=google.com; s=arc-20160816; b=nPoYAS4KhSJO/qzwJD576SErrUvwQjidDui1QSTnPmEcApwySW0f4jd1KBYvtqW8KU l5Bdck1eDP2DEfab44a+CtvBRGoPR7Omq6fft3ko3FTqLFZMlomy6Xuug7ICxiyvq1oq VU7SQyeazudNChrVO5dezH5k1TaZsHm5amlccirVuND64ZiZC949rFYME4/YVghAN71G kR4bE625D1xGYGXc8gQ96LRJ0T2gfdtoMPaM9YQnapLkQ4z1Uw/grZHvP42wHMfKzcQ9 riPHaw7xrlAyJJ/ruHNk0b6KT+LA5/cmxj/006UKfuMBhcWbNid4HcDm9ek9LbAH8iL4 zMRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=B3XDGQrOIHo2Z/Fxm4hqpcuhDa9MTHMM7/GJHSmNf48=; b=xlsfb/LXEleM5XLqqANsIp0DT6M5vxX2aaVhWouDeAqwSH4jHNbGH7MTMyugMiHi01 JruNhzxSX0mE7A5oNLgxaN0iNCV1kMsgrHWz5qQD47dnpAov9mOSjA++Uz+ZQkiAE9pl /knqDAKKFAulN9ObJHb3Ac52rFN0Pe0ItmgtS6LvVqQHe/lv7IKfvrBVIGSgyAyEDo9U jaIg67YW3gJduZ11SCpSCKu2tVZ4P4SW6p8q2c6ecQn7euNF5oSHC/AFpwQstR8/hRcD 8H/Jz/s9AVJEDvT3psAxCYpa4nyd+dzNkLVG1QDWnLWlPYU11q6vDbSTAhyF4RDojfSa 7w4A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GshumYDG; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id z10si465295qkl.68.2020.03.10.23.49.49 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Mar 2020 23:49:49 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GshumYDG; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:46520 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvBc-0005q7-TU for patch@linaro.org; Wed, 11 Mar 2020 02:49:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59349) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6U-00067p-0u for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:31 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6S-0001nR-9i for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:29 -0400 Received: from mail-pg1-x542.google.com ([2607:f8b0:4864:20::542]:36474) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6S-0001hI-4P for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:28 -0400 Received: by mail-pg1-x542.google.com with SMTP id c7so645489pgw.3 for ; Tue, 10 Mar 2020 23:44:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=B3XDGQrOIHo2Z/Fxm4hqpcuhDa9MTHMM7/GJHSmNf48=; b=GshumYDGLFLuvesKBTQC9q0ilBA4N4T54w6gyQPr9gXCq4SD15BazgVA4Dze02YJEj RdGGW+qMy4eYhuAM0HTSXQYYOVfGekhn8MoWFP3NgI0+Zy0pi69L6o9X5xJJ+qruIMw4 dYH1Q8KSF1a6SwDOgg8y3js+Ib15hBL3oSa3V4M5tqp5C6pD5AIRJuxNY0GSDRpKkukt 4dDaPwC5j+or5ACIrzO0WHc0CQ6kE3Iojl84TVuDMy9u9ffvwHxcipzjndCplSySkJ6u dkGCMiaURON6IW0WxGhs1uTyMSL4kQhHmqUsoLARvngm3bTwVE4Cnn/CGh7RMRP1Zkmw DskA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=B3XDGQrOIHo2Z/Fxm4hqpcuhDa9MTHMM7/GJHSmNf48=; b=p/S70XvQ0F9fNM120i7gJOynfNH79XUsrFMZawi0+3M/htGclLPl6r80S33aR5vOMR W65/7pfSFsyUOrvTuwU3XHOfcm3+wn9/qojXsBUi9P6ZpFrzkslGrSmwJGZnv0Etrqkx HZAZZDYEiXSADCdCsBKxqB1kNGZvqUXAC3uBVEQO7uYIfdx6FNXcuMc+ZxgwFf5UGRww SUidxA4ica+ucIzL1bmoYNX1rLAZAuROcqU/WHTHTXjiy/hWTPCmO8lx8McAwq32pZ+j S9eMyoPDDpKCsdsthbdo8UocyCTe0OrTL5gJunwdDPTATnw1NBIWMhSYz75/CngZG8cg Q3Rg== X-Gm-Message-State: ANhLgQ0V8zPOtg0/cgOnwnRewbK2eOB5RNEFGM4ReSwwhNbjawbaats0 mBTcEUJ78dRqBuCeeJ/aQHcF6txKsXA= X-Received: by 2002:aa7:9732:: with SMTP id k18mr1368276pfg.231.1583909065441; Tue, 10 Mar 2020 23:44:25 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:24 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 03/16] exec: Add cpu_probe_watchpoint Date: Tue, 10 Mar 2020 23:44:07 -0700 Message-Id: <20200311064420.30606-4-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::542 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Allow probing of a watchpoint *without* raising an exception. This is of most use for no-fault loads, which should indicate via some architectural means that the load did not occur. Signed-off-by: Richard Henderson --- include/hw/core/cpu.h | 7 +++++++ exec.c | 19 +++++++++++++++++++ 2 files changed, 26 insertions(+) -- 2.20.1 diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index 73e9a869a4..8ec44267a4 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -1090,6 +1090,12 @@ static inline void cpu_check_watchpoint(CPUState *cpu, vaddr addr, vaddr len, { } +static inline bool cpu_probe_watchpoint(CPUState *cpu, vaddr addr, vaddr len, + int flags) +{ + return false; +} + static inline int cpu_watchpoint_address_matches(CPUState *cpu, vaddr addr, vaddr len) { @@ -1104,6 +1110,7 @@ void cpu_watchpoint_remove_by_ref(CPUState *cpu, CPUWatchpoint *watchpoint); void cpu_watchpoint_remove_all(CPUState *cpu, int mask); void cpu_check_watchpoint(CPUState *cpu, vaddr addr, vaddr len, MemTxAttrs attrs, int flags, uintptr_t ra); +bool cpu_probe_watchpoint(CPUState *cpu, vaddr addr, vaddr len, int flags); int cpu_watchpoint_address_matches(CPUState *cpu, vaddr addr, vaddr len); #endif diff --git a/exec.c b/exec.c index 0cc500d53a..2b8f601b9e 100644 --- a/exec.c +++ b/exec.c @@ -2735,6 +2735,25 @@ void cpu_check_watchpoint(CPUState *cpu, vaddr addr, vaddr len, } } +bool cpu_probe_watchpoint(CPUState *cpu, vaddr addr, vaddr len, int flags) +{ + CPUClass *cc = CPU_GET_CLASS(cpu); + CPUWatchpoint *wp; + + assert(tcg_enabled()); + + addr = cc->adjust_watchpoint_address(cpu, addr, len); + QTAILQ_FOREACH(wp, &cpu->watchpoints, entry) { + if (watchpoint_address_matches(wp, addr, len) && + (wp->flags & flags) && + (!(wp->flags & BP_CPU) || + !cc->debug_check_watchpoint(cpu, wp))) { + return true; + } + } + return false; +} + static MemTxResult flatview_read(FlatView *fv, hwaddr addr, MemTxAttrs attrs, void *buf, hwaddr len); static MemTxResult flatview_write(FlatView *fv, hwaddr addr, MemTxAttrs attrs, From patchwork Wed Mar 11 06:44:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 184389 Delivered-To: patch@linaro.org Received: by 2002:a92:1f12:0:0:0:0:0 with SMTP id i18csp280446ile; Tue, 10 Mar 2020 23:49:08 -0700 (PDT) X-Google-Smtp-Source: ADFU+vsIZ5u0P5PzbbQBdNHywaVdsz9CiqcNZugSFiW2K67lD/87xA9eQ1c6v77ucQps5td1gUp1 X-Received: by 2002:ac8:6d01:: with SMTP id o1mr1374133qtt.246.1583909348140; Tue, 10 Mar 2020 23:49:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1583909348; cv=none; d=google.com; s=arc-20160816; b=EyHbiYuMAMFQYDos5RqV50jahXxvkXEql5tC2bpjoQaStPkPL531g26jVop9MP5Qsr 3DsIuWD9taiSjXsY8DxPPHBOwNNrmqFF+WXdhqeAuDPeUSYc8dJARD3oXkdw5zW48ewd no5q+Zvjt5Gr/VdcDbFggvuK4x0ERYiS9Nbqi3gKYQ9o5U8kyAEfzazr4NeohbtkX1Tj YtC9sITgcdXv8lzh0tXI66pO3cNNrWb85d6NBiKn/J0HlngxqRrndhDiQ60x4xb1/VlM 87g5sxJUYpUdyUm8iSPzOps2t8dqrgDxxDx/uXjhZP0CA8eKL+HVXSudfVKJJnDyJ6Qt AE+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=GCUC3ck5APqYH1cRrCKzJ3VHOu3Vf4yYuFQi3uqihJ0=; b=hkw0FB58YaoVRtguUeh7rz1xk3RmHZdhoC0ncZodStVnGHb/dnpJQj+0deTLvtEb5K p3KN5DcWIDqyHjYh6W2MoAAc6PmMLRMfa6r/ER65d20ooYcLBmp/deaecHQ38g1VYiBP yfNSANH0kfhz5+nsbPn6V8HTgEJnm/Atyu2Yv1LUZbQTvVvaAnvkTbiP2zv25bCVk7gz /B75P9mcPdYMdogGhw/gsawN4UlDSB9nVdJpBBTbyvhUKroRBIJMkunJZUxljQLkpjpz CjYCbIaHA4HxhoMb5NYtDduqgXifDjuasHFqFF6jnGMumfYKJYbnZmdMkNcvMVFYaoc5 UrZw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="p/xn6S31"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id w13si415565qvp.1.2020.03.10.23.49.07 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Mar 2020 23:49:08 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="p/xn6S31"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:46486 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvAx-0004MD-EW for patch@linaro.org; Wed, 11 Mar 2020 02:49:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59390) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6W-00069U-0I for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6T-0001oQ-0j for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:31 -0400 Received: from mail-pf1-x442.google.com ([2607:f8b0:4864:20::442]:44859) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6S-0001n1-NS for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:28 -0400 Received: by mail-pf1-x442.google.com with SMTP id b72so732418pfb.11 for ; Tue, 10 Mar 2020 23:44:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GCUC3ck5APqYH1cRrCKzJ3VHOu3Vf4yYuFQi3uqihJ0=; b=p/xn6S31kRYlmK7AzUGp7ETctjPY7TgDPfSOK+iyfTxoaQBIUUimHloq9CJHNt7dWS ewo2luyE9ARy8RHIPLT1LUtaATfFnHpbvgwBC3fBpvzPAaOkFPIa90RPShzUPOyBwg0A 9VS9e9M1ImaR74B670ygBT8O8o2IjoX0jtRwvkPzyE+NVSYno4/UnUJpV7mrDAzQh/fa C4tGxtbRraf3BBZ4mSzN69ENcjFxALq3YbKwtY6lU8kysJdtL8he/+lMa6NZtt4E7lBT YHxZsikiiaiTP5fMbHRePhvKt1ThHc5rRkgmBLME0kLrXMozYGv78XD38jI1kfEbUxle yHCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GCUC3ck5APqYH1cRrCKzJ3VHOu3Vf4yYuFQi3uqihJ0=; b=PW7RdhGKg6y9AfIptFBZBfMdjotvDEUEQfuaOYHjwDnKeRXgvY9bE1Z2zPbPAQcvoK MO6HPiZEzM3L97Cs8dkPIFqTLV0jlxCdzDlIeshgwlSSuoZg596KfXrQ3MMEQx96YLhs HbPJSHRMVYurJIiMDaXqBwaRoIIFnP1pLG85g+NUKt1KO0KA1ZMf+uUZKGa6DeGH/WQc TSisAHCWdqQrLIHXXD8+bEV9nJF8Fb1FxTEZdx8oX9H34ypN9CHxJmjxNy/r84mtsT75 RC4TDLhU3twpOvMsD3kVxYZnGuHKqVbfgOR1JAx02UKKFR5hMwL3EpQw1hFsjck+SIu0 QWAQ== X-Gm-Message-State: ANhLgQ38CN8sKJ704Z4GnvI9cn7IXGkXKNCvDAECejzUZi77rW4lw/x8 G2hpgmS4vh9AsurSP6HorIIsoa6tykU= X-Received: by 2002:a62:2587:: with SMTP id l129mr474534pfl.180.1583909066713; Tue, 10 Mar 2020 23:44:26 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:26 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 04/16] target/arm: Use cpu_*_data_ra for sve_ldst_tlb_fn Date: Tue, 10 Mar 2020 23:44:08 -0700 Message-Id: <20200311064420.30606-5-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::442 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Use the "normal" memory access functions, rather than the softmmu internal helper functions directly. Since fb901c905dc3, cpu_mem_index is now a simple extract from env->hflags and not a large computation. Which means that it's now more work to pass around this value than it is to recompute it. This only adjusts the primitives, and does not clean up all of the uses within sve_helper.c. Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 241 ++++++++++++++++++---------------------- 1 file changed, 107 insertions(+), 134 deletions(-) -- 2.20.1 diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index fdfa652094..f8b310a16b 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -3991,12 +3991,15 @@ typedef intptr_t sve_ld1_host_fn(void *vd, void *vg, void *host, * Load one element into @vd + @reg_off from (@env, @vaddr, @ra). * The controlling predicate is known to be true. */ -typedef void sve_ld1_tlb_fn(CPUARMState *env, void *vd, intptr_t reg_off, - target_ulong vaddr, TCGMemOpIdx oi, uintptr_t ra); -typedef sve_ld1_tlb_fn sve_st1_tlb_fn; +typedef void sve_ldst1_tlb_fn(CPUARMState *env, void *vd, intptr_t reg_off, + target_ulong vaddr, uintptr_t retaddr); /* * Generate the above primitives. + * + * For *_tlb, this uses the cpu_*_data_ra helpers. There are not + * endian-specific versions of these, so we must handle endianness + * locally. */ #define DO_LD_HOST(NAME, H, TYPEE, TYPEM, HOST) \ @@ -4016,27 +4019,25 @@ static intptr_t sve_##NAME##_host(void *vd, void *vg, void *host, \ return mem_off; \ } -#ifdef CONFIG_SOFTMMU -#define DO_LD_TLB(NAME, H, TYPEE, TYPEM, HOST, MOEND, TLB) \ +#define DO_LD_TLB(NAME, H, TYPEE, TYPEM, BSWAP, TLB) \ static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ - target_ulong addr, TCGMemOpIdx oi, uintptr_t ra) \ + target_ulong addr, uintptr_t ra) \ { \ - TYPEM val = TLB(env, addr, oi, ra); \ + TYPEM val = BSWAP(TLB(env, addr, ra)); \ *(TYPEE *)(vd + H(reg_off)) = val; \ } -#else -#define DO_LD_TLB(NAME, H, TYPEE, TYPEM, HOST, MOEND, TLB) \ + +#define DO_ST_TLB(NAME, H, TYPEE, TYPEM, BSWAP, TLB) \ static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ - target_ulong addr, TCGMemOpIdx oi, uintptr_t ra) \ + target_ulong addr, uintptr_t ra) \ { \ - TYPEM val = HOST(g2h(addr)); \ - *(TYPEE *)(vd + H(reg_off)) = val; \ + TYPEM val = *(TYPEE *)(vd + H(reg_off)); \ + TLB(env, addr, BSWAP(val), ra); \ } -#endif #define DO_LD_PRIM_1(NAME, H, TE, TM) \ DO_LD_HOST(NAME, H, TE, TM, ldub_p) \ - DO_LD_TLB(NAME, H, TE, TM, ldub_p, 0, helper_ret_ldub_mmu) + DO_LD_TLB(NAME, H, TE, TM, , cpu_ldub_data_ra) DO_LD_PRIM_1(ld1bb, H1, uint8_t, uint8_t) DO_LD_PRIM_1(ld1bhu, H1_2, uint16_t, uint8_t) @@ -4046,39 +4047,67 @@ DO_LD_PRIM_1(ld1bss, H1_4, uint32_t, int8_t) DO_LD_PRIM_1(ld1bdu, , uint64_t, uint8_t) DO_LD_PRIM_1(ld1bds, , uint64_t, int8_t) -#define DO_LD_PRIM_2(NAME, end, MOEND, H, TE, TM, PH, PT) \ - DO_LD_HOST(NAME##_##end, H, TE, TM, PH##_##end##_p) \ - DO_LD_TLB(NAME##_##end, H, TE, TM, PH##_##end##_p, \ - MOEND, helper_##end##_##PT##_mmu) +#define DO_ST_PRIM_1(NAME, H, TE, TM) \ + DO_ST_TLB(st1##NAME, H, TE, TM, , cpu_stb_data_ra) -DO_LD_PRIM_2(ld1hh, le, MO_LE, H1_2, uint16_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hsu, le, MO_LE, H1_4, uint32_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hss, le, MO_LE, H1_4, uint32_t, int16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hdu, le, MO_LE, , uint64_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hds, le, MO_LE, , uint64_t, int16_t, lduw, lduw) +DO_ST_PRIM_1(bb, H1, uint8_t, uint8_t) +DO_ST_PRIM_1(bh, H1_2, uint16_t, uint8_t) +DO_ST_PRIM_1(bs, H1_4, uint32_t, uint8_t) +DO_ST_PRIM_1(bd, , uint64_t, uint8_t) -DO_LD_PRIM_2(ld1ss, le, MO_LE, H1_4, uint32_t, uint32_t, ldl, ldul) -DO_LD_PRIM_2(ld1sdu, le, MO_LE, , uint64_t, uint32_t, ldl, ldul) -DO_LD_PRIM_2(ld1sds, le, MO_LE, , uint64_t, int32_t, ldl, ldul) +#define DO_LD_PRIM_2(NAME, H, TE, TM, BSWAP, LD) \ + DO_LD_HOST(ld1##NAME##_be, H, TE, TM, LD##_be_p) \ + DO_LD_HOST(ld1##NAME##_le, H, TE, TM, LD##_le_p) \ + DO_LD_TLB(ld1##NAME##_be, H, TE, TM, be_##BSWAP, cpu_##LD##_data_ra) \ + DO_LD_TLB(ld1##NAME##_le, H, TE, TM, le_##BSWAP, cpu_##LD##_data_ra) -DO_LD_PRIM_2(ld1dd, le, MO_LE, , uint64_t, uint64_t, ldq, ldq) +#define DO_ST_PRIM_2(NAME, H, TE, TM, BSWAP, ST) \ + DO_ST_TLB(st1##NAME##_be, H, TE, TM, be_##BSWAP, cpu_##ST##_data_ra) \ + DO_ST_TLB(st1##NAME##_le, H, TE, TM, le_##BSWAP, cpu_##ST##_data_ra) -DO_LD_PRIM_2(ld1hh, be, MO_BE, H1_2, uint16_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hsu, be, MO_BE, H1_4, uint32_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hss, be, MO_BE, H1_4, uint32_t, int16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hdu, be, MO_BE, , uint64_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hds, be, MO_BE, , uint64_t, int16_t, lduw, lduw) +#ifdef TARGET_WORDS_BIGENDIAN +# define be_bswap16(x) ((uint16_t)(x)) +# define be_bswap32(x) ((uint32_t)(x)) +# define be_bswap64(x) ((uint64_t)(x)) +# define le_bswap16(x) bswap16(x) +# define le_bswap32(x) bswap32(x) +# define le_bswap64(x) bswap64(x) +#else +# define be_bswap16(x) bswap16(x) +# define be_bswap32(x) bswap32(x) +# define be_bswap64(x) bswap64(x) +# define le_bswap16(x) ((uint16_t)(x)) +# define le_bswap32(x) ((uint32_t)(x)) +# define le_bswap64(x) ((uint64_t)(x)) +#endif -DO_LD_PRIM_2(ld1ss, be, MO_BE, H1_4, uint32_t, uint32_t, ldl, ldul) -DO_LD_PRIM_2(ld1sdu, be, MO_BE, , uint64_t, uint32_t, ldl, ldul) -DO_LD_PRIM_2(ld1sds, be, MO_BE, , uint64_t, int32_t, ldl, ldul) +DO_LD_PRIM_2(hh, H1_2, uint16_t, uint16_t, bswap16, lduw) +DO_LD_PRIM_2(hsu, H1_4, uint32_t, uint16_t, bswap16, lduw) +DO_LD_PRIM_2(hss, H1_4, uint32_t, int16_t, bswap16, lduw) +DO_LD_PRIM_2(hdu, , uint64_t, uint16_t, bswap16, lduw) +DO_LD_PRIM_2(hds, , uint64_t, int16_t, bswap16, lduw) -DO_LD_PRIM_2(ld1dd, be, MO_BE, , uint64_t, uint64_t, ldq, ldq) +DO_ST_PRIM_2(hh, H1_2, uint16_t, uint16_t, bswap16, stw) +DO_ST_PRIM_2(hs, H1_4, uint32_t, uint16_t, bswap16, stw) +DO_ST_PRIM_2(hd, , uint64_t, uint16_t, bswap16, stw) + +DO_LD_PRIM_2(ss, H1_4, uint32_t, uint32_t, bswap32, ldl) +DO_LD_PRIM_2(sdu, , uint64_t, uint32_t, bswap32, ldl) +DO_LD_PRIM_2(sds, , uint64_t, int32_t, bswap32, ldl) + +DO_ST_PRIM_2(ss, H1_4, uint32_t, uint32_t, bswap32, stl) +DO_ST_PRIM_2(sd, , uint64_t, uint32_t, bswap32, stl) + +DO_LD_PRIM_2(dd, , uint64_t, uint64_t, bswap64, ldq) +DO_ST_PRIM_2(dd, , uint64_t, uint64_t, bswap64, stq) #undef DO_LD_TLB +#undef DO_ST_TLB #undef DO_LD_HOST #undef DO_LD_PRIM_1 +#undef DO_ST_PRIM_1 #undef DO_LD_PRIM_2 +#undef DO_ST_PRIM_2 /* * Skip through a sequence of inactive elements in the guarding predicate @vg, @@ -4152,7 +4181,7 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, uint32_t desc, const uintptr_t retaddr, const int esz, const int msz, sve_ld1_host_fn *host_fn, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const int mmu_idx = get_mmuidx(oi); @@ -4234,7 +4263,7 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, * on I/O memory, it may succeed but not bring in the TLB entry. * But even then we have still made forward progress. */ - tlb_fn(env, &scratch, reg_off, addr + mem_off, oi, retaddr); + tlb_fn(env, &scratch, reg_off, addr + mem_off, retaddr); reg_off += 1 << esz; } #endif @@ -4293,9 +4322,8 @@ DO_LD1_2(ld1dd, 3, 3) */ static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, int size, uintptr_t ra, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); ARMVectorReg scratch[2] = { }; @@ -4305,8 +4333,8 @@ static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, oi, ra); - tlb_fn(env, &scratch[1], i, addr + size, oi, ra); + tlb_fn(env, &scratch[0], i, addr, ra); + tlb_fn(env, &scratch[1], i, addr + size, ra); } i += size, pg >>= size; addr += 2 * size; @@ -4321,9 +4349,8 @@ static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, int size, uintptr_t ra, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); ARMVectorReg scratch[3] = { }; @@ -4333,9 +4360,9 @@ static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, oi, ra); - tlb_fn(env, &scratch[1], i, addr + size, oi, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, oi, ra); + tlb_fn(env, &scratch[0], i, addr, ra); + tlb_fn(env, &scratch[1], i, addr + size, ra); + tlb_fn(env, &scratch[2], i, addr + 2 * size, ra); } i += size, pg >>= size; addr += 3 * size; @@ -4351,9 +4378,8 @@ static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, int size, uintptr_t ra, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); ARMVectorReg scratch[4] = { }; @@ -4363,10 +4389,10 @@ static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, oi, ra); - tlb_fn(env, &scratch[1], i, addr + size, oi, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, oi, ra); - tlb_fn(env, &scratch[3], i, addr + 3 * size, oi, ra); + tlb_fn(env, &scratch[0], i, addr, ra); + tlb_fn(env, &scratch[1], i, addr + size, ra); + tlb_fn(env, &scratch[2], i, addr + 2 * size, ra); + tlb_fn(env, &scratch[3], i, addr + 3 * size, ra); } i += size, pg >>= size; addr += 4 * size; @@ -4459,7 +4485,7 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, uint32_t desc, const uintptr_t retaddr, const int esz, const int msz, sve_ld1_host_fn *host_fn, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const int mmu_idx = get_mmuidx(oi); @@ -4519,7 +4545,7 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, * Perform one normal read, which will fault or not. * But it is likely to bring the page into the tlb. */ - tlb_fn(env, vd, reg_off, addr + mem_off, oi, retaddr); + tlb_fn(env, vd, reg_off, addr + mem_off, retaddr); /* After any fault, zero any leading predicated false elts. */ swap_memzero(vd, reg_off); @@ -4671,60 +4697,14 @@ DO_LDFF1_LDNF1_2(dd, 3, 3) #undef DO_LDFF1_LDNF1_1 #undef DO_LDFF1_LDNF1_2 -/* - * Store contiguous data, protected by a governing predicate. - */ - -#ifdef CONFIG_SOFTMMU -#define DO_ST_TLB(NAME, H, TYPEM, HOST, MOEND, TLB) \ -static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ - target_ulong addr, TCGMemOpIdx oi, uintptr_t ra) \ -{ \ - TLB(env, addr, *(TYPEM *)(vd + H(reg_off)), oi, ra); \ -} -#else -#define DO_ST_TLB(NAME, H, TYPEM, HOST, MOEND, TLB) \ -static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ - target_ulong addr, TCGMemOpIdx oi, uintptr_t ra) \ -{ \ - HOST(g2h(addr), *(TYPEM *)(vd + H(reg_off))); \ -} -#endif - -DO_ST_TLB(st1bb, H1, uint8_t, stb_p, 0, helper_ret_stb_mmu) -DO_ST_TLB(st1bh, H1_2, uint16_t, stb_p, 0, helper_ret_stb_mmu) -DO_ST_TLB(st1bs, H1_4, uint32_t, stb_p, 0, helper_ret_stb_mmu) -DO_ST_TLB(st1bd, , uint64_t, stb_p, 0, helper_ret_stb_mmu) - -DO_ST_TLB(st1hh_le, H1_2, uint16_t, stw_le_p, MO_LE, helper_le_stw_mmu) -DO_ST_TLB(st1hs_le, H1_4, uint32_t, stw_le_p, MO_LE, helper_le_stw_mmu) -DO_ST_TLB(st1hd_le, , uint64_t, stw_le_p, MO_LE, helper_le_stw_mmu) - -DO_ST_TLB(st1ss_le, H1_4, uint32_t, stl_le_p, MO_LE, helper_le_stl_mmu) -DO_ST_TLB(st1sd_le, , uint64_t, stl_le_p, MO_LE, helper_le_stl_mmu) - -DO_ST_TLB(st1dd_le, , uint64_t, stq_le_p, MO_LE, helper_le_stq_mmu) - -DO_ST_TLB(st1hh_be, H1_2, uint16_t, stw_be_p, MO_BE, helper_be_stw_mmu) -DO_ST_TLB(st1hs_be, H1_4, uint32_t, stw_be_p, MO_BE, helper_be_stw_mmu) -DO_ST_TLB(st1hd_be, , uint64_t, stw_be_p, MO_BE, helper_be_stw_mmu) - -DO_ST_TLB(st1ss_be, H1_4, uint32_t, stl_be_p, MO_BE, helper_be_stl_mmu) -DO_ST_TLB(st1sd_be, , uint64_t, stl_be_p, MO_BE, helper_be_stl_mmu) - -DO_ST_TLB(st1dd_be, , uint64_t, stq_be_p, MO_BE, helper_be_stq_mmu) - -#undef DO_ST_TLB - /* * Common helpers for all contiguous 1,2,3,4-register predicated stores. */ static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, const uintptr_t ra, const int esize, const int msize, - sve_st1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); void *vd = &env->vfp.zregs[rd]; @@ -4734,7 +4714,7 @@ static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, vd, i, addr, oi, ra); + tlb_fn(env, vd, i, addr, ra); } i += esize, pg >>= esize; addr += msize; @@ -4746,9 +4726,8 @@ static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, const uintptr_t ra, const int esize, const int msize, - sve_st1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); void *d1 = &env->vfp.zregs[rd]; @@ -4759,8 +4738,8 @@ static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, d1, i, addr, oi, ra); - tlb_fn(env, d2, i, addr + msize, oi, ra); + tlb_fn(env, d1, i, addr, ra); + tlb_fn(env, d2, i, addr + msize, ra); } i += esize, pg >>= esize; addr += 2 * msize; @@ -4772,9 +4751,8 @@ static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, const uintptr_t ra, const int esize, const int msize, - sve_st1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); void *d1 = &env->vfp.zregs[rd]; @@ -4786,9 +4764,9 @@ static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, d1, i, addr, oi, ra); - tlb_fn(env, d2, i, addr + msize, oi, ra); - tlb_fn(env, d3, i, addr + 2 * msize, oi, ra); + tlb_fn(env, d1, i, addr, ra); + tlb_fn(env, d2, i, addr + msize, ra); + tlb_fn(env, d3, i, addr + 2 * msize, ra); } i += esize, pg >>= esize; addr += 3 * msize; @@ -4800,9 +4778,8 @@ static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, const uintptr_t ra, const int esize, const int msize, - sve_st1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); void *d1 = &env->vfp.zregs[rd]; @@ -4815,10 +4792,10 @@ static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, d1, i, addr, oi, ra); - tlb_fn(env, d2, i, addr + msize, oi, ra); - tlb_fn(env, d3, i, addr + 2 * msize, oi, ra); - tlb_fn(env, d4, i, addr + 3 * msize, oi, ra); + tlb_fn(env, d1, i, addr, ra); + tlb_fn(env, d2, i, addr + msize, ra); + tlb_fn(env, d3, i, addr + 2 * msize, ra); + tlb_fn(env, d4, i, addr + 3 * msize, ra); } i += esize, pg >>= esize; addr += 4 * msize; @@ -4914,9 +4891,8 @@ static target_ulong off_zd_d(void *reg, intptr_t reg_ofs) static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t i, oprsz = simd_oprsz(desc); ARMVectorReg scratch = { }; @@ -4927,7 +4903,7 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, do { if (likely(pg & 1)) { target_ulong off = off_fn(vm, i); - tlb_fn(env, &scratch, i, base + (off << scale), oi, ra); + tlb_fn(env, &scratch, i, base + (off << scale), ra); } i += 4, pg >>= 4; } while (i & 15); @@ -4940,9 +4916,8 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t i, oprsz = simd_oprsz(desc) / 8; ARMVectorReg scratch = { }; @@ -4952,7 +4927,7 @@ static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, uint8_t pg = *(uint8_t *)(vg + H1(i)); if (likely(pg & 1)) { target_ulong off = off_fn(vm, i * 8); - tlb_fn(env, &scratch, i * 8, base + (off << scale), oi, ra); + tlb_fn(env, &scratch, i * 8, base + (off << scale), ra); } } clear_helper_retaddr(); @@ -5114,7 +5089,7 @@ DO_LD_NF(dd_be, , uint64_t, uint64_t, ldq_be_p) */ static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn, + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn, sve_ld1_nf_fn *nonfault_fn) { const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); @@ -5130,7 +5105,7 @@ static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void *vm, set_helper_retaddr(ra); addr = off_fn(vm, reg_off); addr = base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, oi, ra); + tlb_fn(env, vd, reg_off, addr, ra); /* The rest of the reads will be non-faulting. */ clear_helper_retaddr(); @@ -5156,7 +5131,7 @@ static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void *vm, static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn, + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn, sve_ld1_nf_fn *nonfault_fn) { const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); @@ -5172,7 +5147,7 @@ static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void *vm, set_helper_retaddr(ra); addr = off_fn(vm, reg_off); addr = base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, oi, ra); + tlb_fn(env, vd, reg_off, addr, ra); /* The rest of the reads will be non-faulting. */ clear_helper_retaddr(); @@ -5282,9 +5257,8 @@ DO_LDFF1_ZPZ_D(dd_be, zd) static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t i, oprsz = simd_oprsz(desc); @@ -5294,7 +5268,7 @@ static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, do { if (likely(pg & 1)) { target_ulong off = off_fn(vm, i); - tlb_fn(env, vd, i, base + (off << scale), oi, ra); + tlb_fn(env, vd, i, base + (off << scale), ra); } i += 4, pg >>= 4; } while (i & 15); @@ -5304,9 +5278,8 @@ static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t i, oprsz = simd_oprsz(desc) / 8; @@ -5315,7 +5288,7 @@ static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, uint8_t pg = *(uint8_t *)(vg + H1(i)); if (likely(pg & 1)) { target_ulong off = off_fn(vm, i * 8); - tlb_fn(env, vd, i * 8, base + (off << scale), oi, ra); + tlb_fn(env, vd, i * 8, base + (off << scale), ra); } } clear_helper_retaddr(); From patchwork Wed Mar 11 06:44:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 184388 Delivered-To: patch@linaro.org Received: by 2002:a92:1f12:0:0:0:0:0 with SMTP id i18csp279039ile; Tue, 10 Mar 2020 23:47:21 -0700 (PDT) X-Google-Smtp-Source: ADFU+vuI0FUI/wjW9ez334moVwkXA04kNPswHfkunfsvGEUlU7PWpjDKqueJHc4fSjDgvbCePMvW X-Received: by 2002:ac8:735a:: with SMTP id q26mr1341653qtp.286.1583909241491; Tue, 10 Mar 2020 23:47:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1583909241; cv=none; d=google.com; s=arc-20160816; b=zqHhbnAXv3w4gGLz/kbgez54I15+Sq1ObQgDCoYtmm4dELU+w+3GFS6XCQT68zhAsB 5FrJTZR0Bs3lVcV80u7qZYMuzgIFvJtHXlW/YsQF94/eDiBWKPicCF6yQ2uqgFhbi6H/ Zv4kPX1c1t9sb3nRnVmaO0hZdVgNKnQaPjtEPU96CxmczrWZt0VHOEDesYStsyPN2NFs 0F8/5a4/IZW0+AaCyuQg5PbRh9shMK0UP1np+O4oZIS6q1L9R1ZyuouUpiUjV2EzMYUE eWZGccZ+znR+TspMPrf9F8Pufo1wkdDb7JZoGmsjTTr93f+dxSz3eVwT3EKlR5CGG+2W 9tLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=x2sQUQOTqOZ29kmZWF8+Fedva6/mV+pCTu/MNQTzpjU=; b=Ddtlm89OTzfYahKeGbKgG4Nt9FjwUhCqRnnrJ9TMgp6dFHZjp3ZP6Uzo7k/ZY1Bw2J r9cxWtNblrs8jmi0I0g5sJFenDqT+TpmrMLaFGyjN6OBkfMX7niIiDeGqMBVVxGNfqGU 3cwMFTpFpEiimg9TNFqftRug1qhe5o4Kh7JWKy6j/BLCelGtiDOQUSHnS+qenxQETSjz Nzp1i2JcfGQPOOn5W80aelgX58bjHMjl8zUURrDDH5yrDQ615lZpR3qqLrGUZnzhfhPb p3XSxnqKiif32SoMGRY9LltFEUTDDTL0MbW1a2XcndceiE6HRZUxJqhHPWbfyqwufFPb WwXw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=jUs5ATgi; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id j13si462845qkm.358.2020.03.10.23.47.21 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Mar 2020 23:47:21 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=jUs5ATgi; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:46430 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv9E-0001dK-U3 for patch@linaro.org; Wed, 11 Mar 2020 02:47:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59417) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6X-0006Az-S2 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6V-0001tI-UR for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:33 -0400 Received: from mail-pf1-x444.google.com ([2607:f8b0:4864:20::444]:34535) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6T-0001oH-UW for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:31 -0400 Received: by mail-pf1-x444.google.com with SMTP id 23so755088pfj.1 for ; Tue, 10 Mar 2020 23:44:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=x2sQUQOTqOZ29kmZWF8+Fedva6/mV+pCTu/MNQTzpjU=; b=jUs5ATgiOY2goWQDP8wHHTO4JnkBdWW94XVQdx35YRib2w1wDldTvX3S19jJvkQm8M Mp6lGiODavUH6VzuzQ8Ceme2SSze9je3T9syMM6QDcYpUMiYnCkI6Xw/xAg608PXky7N /6maAxk4PCIsgC/PfiqPCSai43ih5aqKJjJNI6fyZjfHlub0Y0XwXDVhEUoiRVhWs4Oe JmIaJggZlD4ceq8Un7bWOMrI1yJGTWuSBaOTcD5DSqhuyqWdfaeaxchUsDP4kG2xn55f TD65DC4mdajR/VmPc+s3sxUeVL+CMCwxX+lM+q0k5A0rOI8OKwNjsHRCWdNG9iHciogm 2beQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=x2sQUQOTqOZ29kmZWF8+Fedva6/mV+pCTu/MNQTzpjU=; b=UYqOFK41atvtWmrWHlSkVD0BpvrsPcp/DsGjeqEkoWufk/H1hOZY+r98d0danQKHCC eaQHX9b2ZnYJJ57r1B+FKpvnBTLqcuZmM/5fYSERp4W7a8C9t93I/fM7fmvqRjnSM8Ac eGj9KMMUK29j/RC29rRDlA023q3KW32kcCcfeaPCWIwa9WNVghaBCw3ECENrfdQVxy8x 1+l7VWxgBUexUxj1AbIFiBWKhU5Wn39BtM4moGnu6Nrj4p2Ez/npPeBp++ErTcEtUGjw or/3FESa1EbT+i8RoEigufZ+mw5H2Axf7Z04if3Pj0H4ujrJlJTL79OXqt4ghvW5dJek mUpA== X-Gm-Message-State: ANhLgQ3e2tWsGG4eFEkZCFzLJzePOuZTm3/7/e5VZWKZpO6ShsSEy1RQ jtMkJz8JfnHuDQvmklkhhRW+rLKzI/8= X-Received: by 2002:a63:257:: with SMTP id 84mr1436504pgc.304.1583909067912; Tue, 10 Mar 2020 23:44:27 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:27 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 05/16] target/arm: Drop manual handling of set/clear_helper_retaddr Date: Tue, 10 Mar 2020 23:44:09 -0700 Message-Id: <20200311064420.30606-6-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::444 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Since we converted back to cpu_*_data_ra, we do not need to do this ourselves. Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 38 -------------------------------------- 1 file changed, 38 deletions(-) -- 2.20.1 Reviewed-by: Peter Maydell diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index f8b310a16b..8b470991db 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4155,12 +4155,6 @@ static intptr_t max_for_page(target_ulong base, intptr_t mem_off, return MIN(split, mem_max - mem_off) + mem_off; } -#ifndef CONFIG_USER_ONLY -/* These are normally defined only for CONFIG_USER_ONLY in */ -static inline void set_helper_retaddr(uintptr_t ra) { } -static inline void clear_helper_retaddr(void) { } -#endif - /* * The result of tlb_vaddr_to_host for user-only is just g2h(x), * which is always non-null. Elide the useless test. @@ -4202,7 +4196,6 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, return; } mem_off = reg_off >> diffsz; - set_helper_retaddr(retaddr); /* * If the (remaining) load is entirely within a single page, then: @@ -4217,7 +4210,6 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, if (test_host_page(host)) { mem_off = host_fn(vd, vg, host - mem_off, mem_off, mem_max); tcg_debug_assert(mem_off == mem_max); - clear_helper_retaddr(); /* After having taken any fault, zero leading inactive elements. */ swap_memzero(vd, reg_off); return; @@ -4268,7 +4260,6 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, } #endif - clear_helper_retaddr(); memcpy(vd, &scratch, reg_max); } @@ -4328,7 +4319,6 @@ static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, intptr_t i, oprsz = simd_oprsz(desc); ARMVectorReg scratch[2] = { }; - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4340,7 +4330,6 @@ static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, addr += 2 * size; } while (i & 15); } - clear_helper_retaddr(); /* Wait until all exceptions have been raised to write back. */ memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); @@ -4355,7 +4344,6 @@ static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, intptr_t i, oprsz = simd_oprsz(desc); ARMVectorReg scratch[3] = { }; - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4368,7 +4356,6 @@ static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, addr += 3 * size; } while (i & 15); } - clear_helper_retaddr(); /* Wait until all exceptions have been raised to write back. */ memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); @@ -4384,7 +4371,6 @@ static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, intptr_t i, oprsz = simd_oprsz(desc); ARMVectorReg scratch[4] = { }; - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4398,7 +4384,6 @@ static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, addr += 4 * size; } while (i & 15); } - clear_helper_retaddr(); /* Wait until all exceptions have been raised to write back. */ memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); @@ -4505,7 +4490,6 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, return; } mem_off = reg_off >> diffsz; - set_helper_retaddr(retaddr); /* * If the (remaining) load is entirely within a single page, then: @@ -4520,7 +4504,6 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, if (test_host_page(host)) { mem_off = host_fn(vd, vg, host - mem_off, mem_off, mem_max); tcg_debug_assert(mem_off == mem_max); - clear_helper_retaddr(); /* After any fault, zero any leading inactive elements. */ swap_memzero(vd, reg_off); return; @@ -4563,7 +4546,6 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, } #endif - clear_helper_retaddr(); record_fault(env, reg_off, reg_max); } @@ -4709,7 +4691,6 @@ static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, intptr_t i, oprsz = simd_oprsz(desc); void *vd = &env->vfp.zregs[rd]; - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4720,7 +4701,6 @@ static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, addr += msize; } while (i & 15); } - clear_helper_retaddr(); } static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, @@ -4733,7 +4713,6 @@ static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, void *d1 = &env->vfp.zregs[rd]; void *d2 = &env->vfp.zregs[(rd + 1) & 31]; - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4745,7 +4724,6 @@ static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, addr += 2 * msize; } while (i & 15); } - clear_helper_retaddr(); } static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, @@ -4759,7 +4737,6 @@ static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, void *d2 = &env->vfp.zregs[(rd + 1) & 31]; void *d3 = &env->vfp.zregs[(rd + 2) & 31]; - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4772,7 +4749,6 @@ static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, addr += 3 * msize; } while (i & 15); } - clear_helper_retaddr(); } static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, @@ -4787,7 +4763,6 @@ static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, void *d3 = &env->vfp.zregs[(rd + 2) & 31]; void *d4 = &env->vfp.zregs[(rd + 3) & 31]; - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4801,7 +4776,6 @@ static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, addr += 4 * msize; } while (i & 15); } - clear_helper_retaddr(); } #define DO_STN_1(N, NAME, ESIZE) \ @@ -4897,7 +4871,6 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, intptr_t i, oprsz = simd_oprsz(desc); ARMVectorReg scratch = { }; - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4908,7 +4881,6 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, i += 4, pg >>= 4; } while (i & 15); } - clear_helper_retaddr(); /* Wait until all exceptions have been raised to write back. */ memcpy(vd, &scratch, oprsz); @@ -4922,7 +4894,6 @@ static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, intptr_t i, oprsz = simd_oprsz(desc) / 8; ARMVectorReg scratch = { }; - set_helper_retaddr(ra); for (i = 0; i < oprsz; i++) { uint8_t pg = *(uint8_t *)(vg + H1(i)); if (likely(pg & 1)) { @@ -4930,7 +4901,6 @@ static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, tlb_fn(env, &scratch, i * 8, base + (off << scale), ra); } } - clear_helper_retaddr(); /* Wait until all exceptions have been raised to write back. */ memcpy(vd, &scratch, oprsz * 8); @@ -5102,13 +5072,11 @@ static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void *vm, reg_off = find_next_active(vg, 0, reg_max, MO_32); if (likely(reg_off < reg_max)) { /* Perform one normal read, which will fault or not. */ - set_helper_retaddr(ra); addr = off_fn(vm, reg_off); addr = base + (addr << scale); tlb_fn(env, vd, reg_off, addr, ra); /* The rest of the reads will be non-faulting. */ - clear_helper_retaddr(); } /* After any fault, zero the leading predicated false elements. */ @@ -5144,13 +5112,11 @@ static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void *vm, reg_off = find_next_active(vg, 0, reg_max, MO_64); if (likely(reg_off < reg_max)) { /* Perform one normal read, which will fault or not. */ - set_helper_retaddr(ra); addr = off_fn(vm, reg_off); addr = base + (addr << scale); tlb_fn(env, vd, reg_off, addr, ra); /* The rest of the reads will be non-faulting. */ - clear_helper_retaddr(); } /* After any fault, zero the leading predicated false elements. */ @@ -5262,7 +5228,6 @@ static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t i, oprsz = simd_oprsz(desc); - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -5273,7 +5238,6 @@ static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, i += 4, pg >>= 4; } while (i & 15); } - clear_helper_retaddr(); } static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, @@ -5283,7 +5247,6 @@ static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t i, oprsz = simd_oprsz(desc) / 8; - set_helper_retaddr(ra); for (i = 0; i < oprsz; i++) { uint8_t pg = *(uint8_t *)(vg + H1(i)); if (likely(pg & 1)) { @@ -5291,7 +5254,6 @@ static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, tlb_fn(env, vd, i * 8, base + (off << scale), ra); } } - clear_helper_retaddr(); } #define DO_ST1_ZPZ_S(MEM, OFS) \ From patchwork Wed Mar 11 06:44:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 184393 Delivered-To: patch@linaro.org Received: by 2002:a92:1f12:0:0:0:0:0 with SMTP id i18csp281509ile; Tue, 10 Mar 2020 23:50:36 -0700 (PDT) X-Google-Smtp-Source: ADFU+vtLDXi9LQ89UWq8Tl+AOE076fDXITna6WMQdgKeM4HYrfRw/btSSXJUWlZ2AS2n03HI75Y/ X-Received: by 2002:a37:b886:: with SMTP id i128mr1435850qkf.410.1583909436772; Tue, 10 Mar 2020 23:50:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1583909436; cv=none; d=google.com; s=arc-20160816; b=rRjwIeTA/1GcN1DOB8kPJjGencLQLGLKqKMOE/M4h1hk+uvnZiUpqu9ItMQZf2tdnw yedJRQRiaF21B8Xuzf4jE6mzcHMPOKZISbkSGV8hL+wJ9zyF3W/j2Z8tziY6PsIkNLDc xjpIMGb11NzfczvruR7JT0ZmKyb5RD/loUOCfPED+7tHkg0x1jLxvhgiXrgP+xlEr6iA psyBZ+JsgRl6fdTgsOIRbeVit6ZmTBIWpO9geL5gs3V9wYbkWgs+BA1LgxHdLkrp3Ho2 aUWsDVazOVW2++W//kbEmGmatsVyjxIGmTOA0AxOIcalbYLL0kaCKM6azcFrxDA7gVAE 3FHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=WeaNzwzC03UA9+8arqHbMRnYm8SNdcLfajSQ8cx3zrw=; b=UBzk45oUvXvhr019jCWbgxMVfsnoOTedW1CjONRINgaVNMRyeHrEqaOpkwd1P2GH37 zkmos104G0hSrlAMqBvlbPl1Ss0kE7DbJd8xOm+h2bk/InB1fDrvvtRvmevDJC8DuQ1F 9ZI122pVHLhitH7YXFWr/ZHRSCjefi6ttt9+i8ZfcNmPVl5urmGV79E8GPbicwKF03dL JRuBcL3zYkxIX4U9fgXml/wpD0i6sj1XGEoAmeCYUEdJZfDmc8je+GvM+Bsxwc51ZZfm tozoFuyYeBC73DAg4mh2rSQmEyVdQi7pRlJ+tMURnPUbiNG5w/Z3ZiQIwlwOYPDy2o/B mf5w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=EL2RWt2D; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id t12si531998qtr.207.2020.03.10.23.50.36 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Mar 2020 23:50:36 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=EL2RWt2D; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:46554 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvCM-0007SU-VY for patch@linaro.org; Wed, 11 Mar 2020 02:50:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59454) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6Z-0006CR-Mi for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6X-0001vU-PY for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:35 -0400 Received: from mail-pl1-x642.google.com ([2607:f8b0:4864:20::642]:41528) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6V-0001rw-TO for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:33 -0400 Received: by mail-pl1-x642.google.com with SMTP id t14so614380plr.8 for ; Tue, 10 Mar 2020 23:44:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WeaNzwzC03UA9+8arqHbMRnYm8SNdcLfajSQ8cx3zrw=; b=EL2RWt2DGrX6kJg5C4+cQ18aibmZetpuMBLgbQxG11B+rOZfBhV54BmfC2uh+j8+Z6 SVWdFT4QIl7aQvjuGBj2T6f1Z3O8zNKAwSNEsYY8lY7nR+WnmC7yk1AWZhBOf4grB87Y 4q2WOi/C3HO4mG2QSTyVvDm/eaAG4eMJnZMwb7ZUiLuVlC3+tpFiBCFwEwpVlSwUcsXS TeBkoJsy6vpa84dCFdTK5TmBtCEA2d1Rqpvwo50X+HZXQZg/CBLHQ7GxlhoE07Z2ECnE nuSwpZ5gsSBImNhjcbEAakSOR28LDROD041GlDZiCJXCY7LeUPWi7efTC41AKXZKLVaz 21uA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WeaNzwzC03UA9+8arqHbMRnYm8SNdcLfajSQ8cx3zrw=; b=kEVQQuq8ye0JoE8LvcimlFow18shbpF9y8uyUatcXIwi0GDl7Q48c0lueFEfQLhH0h pImPzng/LwQdYUrIsZCtRfRxvJZfKKrOSDWFD2WB3esZKMhJbc4KrtsfKtJPVm7Qq4Nw 0z9Zm2GXemvjsTskmsKFJvVIkxeUpIUNwO7xBzQavBTbI/EXUhx9ni/ANNI1W0F78Wkb y2QMWd9G6SUHuw1OsX5Ugmlm7GAGhHF7GDSsPqiUvtCmniUJ8kIH9JYJGtl/APgpY0LY at3RE75s42RUJr7kgBC1kWGzLFdmx400TdGPK3lNcelGxjMhXpTxuRDBKl/BkzZsccSo 99jg== X-Gm-Message-State: ANhLgQ12Nw/0L3BrEsCnFvBeWbTQpINQi+NEm4J8GRmQb5kyT3Fdzg0W 7BzMQuSPyECQJ/6vifGc9SpCSn/2j0w= X-Received: by 2002:a17:902:a408:: with SMTP id p8mr1782284plq.132.1583909069346; Tue, 10 Mar 2020 23:44:29 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:28 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 06/16] target/arm: Add sve infrastructure for page lookup Date: Tue, 10 Mar 2020 23:44:10 -0700 Message-Id: <20200311064420.30606-7-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::642 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" For contiguous predicated memory operations, we want to minimize the number of tlb lookups performed. We have open-coded this for sve_ld1_r, but for correctness with MTE we will need this for all of the memory operations. Create a structure that holds the bounds of active elements, and metadata for two pages. Add routines to find those active elements, lookup the pages, and run watchpoints for those pages. Temporarily mark the functions unused to avoid Werror. Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 240 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 240 insertions(+) -- 2.20.1 diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 8b470991db..3f653e46a0 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4155,6 +4155,246 @@ static intptr_t max_for_page(target_ulong base, intptr_t mem_off, return MIN(split, mem_max - mem_off) + mem_off; } +/* + * Resolve the guest virtual address to info->host and info->flags. + * If @nofault, return false if the page is invalid, otherwise + * exit via page fault exception. + */ + +typedef struct { + void *host; + int flags; + MemTxAttrs attrs; +} SVEHostPage; + +static bool sve_probe_page(SVEHostPage *info, bool nofault, + CPUARMState *env, target_ulong addr, + int mem_off, MMUAccessType access_type, + int mmu_idx, uintptr_t retaddr) +{ + int flags; + + addr += mem_off; + flags = probe_access_flags(env, addr, access_type, mmu_idx, nofault, + &info->host, retaddr); + info->flags = flags; + + if (flags & TLB_INVALID_MASK) { + g_assert(nofault); + return false; + } + + /* Ensure that info->host[] is relative to addr, not addr + mem_off. */ + info->host -= mem_off; + +#ifdef CONFIG_USER_ONLY + memset(&info->attrs, 0, sizeof(info->attrs)); +#else + /* + * Find the iotlbentry for addr and return the transaction attributes. + * This *must* be present in the TLB because we just found the mapping. + */ + { + uintptr_t index = tlb_index(env, mmu_idx, addr); + +# ifdef CONFIG_DEBUG_TCG + CPUTLBEntry *entry = tlb_entry(env, mmu_idx, addr); + target_ulong comparator = (access_type == MMU_DATA_LOAD + ? entry->addr_read + : tlb_addr_write(entry)); + g_assert(tlb_hit(comparator, addr)); +# endif + + CPUIOTLBEntry *iotlbentry = &env_tlb(env)->d[mmu_idx].iotlb[index]; + info->attrs = iotlbentry->attrs; + } +#endif + + return true; +} + + +/* + * Analyse contiguous data, protected by a governing predicate. + */ + +typedef enum { + FAULT_NO, + FAULT_FIRST, + FAULT_ALL, +} SVEContFault; + +typedef struct { + /* First and last element wholy contained within the two pages. */ + int16_t mem_off_first[2]; + int16_t reg_off_first[2]; + int16_t reg_off_last[2]; + + /* One element that is misaligned and spans both pages. */ + int16_t mem_off_split; + int16_t reg_off_split; + int16_t page_split; + + /* TLB data for the two pages. */ + SVEHostPage page[2]; +} SVEContLdSt; + +/* + * Find first active element on each page, and a loose bound for the + * final element on each page. Identify any single element that spans + * the page boundary. Return true if there are any active elements. + */ +static bool __attribute__((unused)) +sve_cont_ldst_elements(SVEContLdSt *info, target_ulong addr, uint64_t *vg, + intptr_t reg_max, int esz, int msize) +{ + const int esize = 1 << esz; + const uint64_t pg_mask = pred_esz_masks[esz]; + intptr_t reg_off_first = -1, reg_off_last = -1, reg_off_split; + intptr_t mem_off_last, mem_off_split; + intptr_t page_split, elt_split; + intptr_t i; + + /* Set all of the element indicies to -1, and the TLB data to 0. */ + memset(info, -1, offsetof(SVEContLdSt, page)); + memset(info->page, 0, sizeof(info->page)); + + /* Gross scan over the entire predicate to find bounds. */ + i = 0; + do { + uint64_t pg = vg[i] & pg_mask; + if (pg) { + reg_off_last = i * 64 + 63 - clz64(pg); + if (reg_off_first < 0) { + reg_off_first = i * 64 + ctz64(pg); + } + } + } while (++i * 64 < reg_max); + + if (unlikely(reg_off_first < 0)) { + /* No active elements, no pages touched. */ + return false; + } + tcg_debug_assert(reg_off_last >= 0 && reg_off_last < reg_max); + + info->reg_off_first[0] = reg_off_first; + info->mem_off_first[0] = (reg_off_first >> esz) * msize; + mem_off_last = (reg_off_last >> esz) * msize; + + page_split = -(addr | TARGET_PAGE_MASK); + if (likely(mem_off_last + msize <= page_split)) { + /* The entire operation fits within a single page. */ + info->reg_off_last[0] = reg_off_last; + return true; + } + + info->page_split = page_split; + elt_split = page_split / msize; + reg_off_split = elt_split << esz; + mem_off_split = elt_split * msize; + + /* + * This is the last full element on the first page, but it is not + * necessarily active. If there is no full element, i.e. the first + * active element is the one that's split, this value remains -1. + * It is useful as iteration bounds. + */ + if (elt_split != 0) { + info->reg_off_last[0] = reg_off_split - esize; + } + + /* Determine if an unaligned element spans the pages. */ + if (page_split % msize != 0) { + /* It is helpful to know if the split element is active. */ + if ((vg[reg_off_split >> 6] >> (reg_off_split & 63)) & 1) { + info->reg_off_split = reg_off_split; + info->mem_off_split = mem_off_split; + + if (reg_off_split == reg_off_last) { + /* The page crossing element is last. */ + return true; + } + } + reg_off_split += esize; + mem_off_split += msize; + } + + /* + * We do want the first active element on the second page, because + * this may affect the address reported in an exception. + */ + reg_off_split = find_next_active(vg, reg_off_split, reg_max, esz); + tcg_debug_assert(reg_off_split <= reg_off_last); + info->reg_off_first[1] = reg_off_split; + info->mem_off_first[1] = (reg_off_split >> esz) * msize; + info->reg_off_last[1] = reg_off_last; + return true; +} + +/* + * Resolve the guest virtual addresses to info->page[]. + * Control the generation of page faults with @fault. Return false if + * there is no work to do, which can only happen with @fault == FAULT_NO. + */ +static bool __attribute__((unused)) +sve_cont_ldst_pages(SVEContLdSt *info, SVEContFault fault, CPUARMState *env, + target_ulong addr, MMUAccessType access_type, + uintptr_t retaddr) +{ + int mmu_idx = cpu_mmu_index(env, false); + int mem_off = info->mem_off_first[0]; + bool nofault = fault == FAULT_NO; + bool have_work = true; + + if (!sve_probe_page(&info->page[0], nofault, env, addr, mem_off, + access_type, mmu_idx, retaddr)) { + /* No work to be done. */ + return false; + } + + if (likely(info->page_split < 0)) { + /* The entire operation was on the one page. */ + return true; + } + + /* + * If the second page is invalid, then we want the fault address to be + * the first byte on that page which is accessed. + */ + if (info->mem_off_split >= 0) { + /* + * There is an element split across the pages. The fault address + * should be the first byte of the second page. + */ + mem_off = info->page_split; + /* + * If the split element is also the first active element + * of the vector, then: For first-fault we should continue + * to generate faults for the second page. For no-fault, + * we have work only if the second page is valid. + */ + if (info->mem_off_first[0] < info->mem_off_split) { + nofault = FAULT_FIRST; + have_work = false; + } + } else { + /* + * There is no element split across the pages. The fault address + * should be the first active element on the second page. + */ + mem_off = info->mem_off_first[1]; + /* + * There must have been one active element on the first page, + * so we're out of first-fault territory. + */ + nofault = fault != FAULT_ALL; + } + + have_work |= sve_probe_page(&info->page[1], nofault, env, addr, mem_off, + access_type, mmu_idx, retaddr); + return have_work; +} + /* * The result of tlb_vaddr_to_host for user-only is just g2h(x), * which is always non-null. Elide the useless test. From patchwork Wed Mar 11 06:44:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 184396 Delivered-To: patch@linaro.org Received: by 2002:a92:1f12:0:0:0:0:0 with SMTP id i18csp283677ile; Tue, 10 Mar 2020 23:53:40 -0700 (PDT) X-Google-Smtp-Source: ADFU+vvO2tAOx6ADvilv7m5ArXidM42zN+zY1auDSchr3kRigZ05Dvascj8uAzx6K0DfbinuyZi9 X-Received: by 2002:a0c:fdc5:: with SMTP id g5mr1624977qvs.194.1583909620730; Tue, 10 Mar 2020 23:53:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1583909620; cv=none; d=google.com; s=arc-20160816; b=tqSqur+uANmzaEmBsAr+R6Xl3F8x9NmjTKUnxEaVauD86/Hv2cSKWVZvFoIN9Glid8 UHPUUWL+JQQeQ1sSIK2I1oE9w9+y9lJix10js0iNb1FvzyRk67UwR+trDl/qzZryF0Lj 5HgnxqX+7JvMpbJ5SqbvF2rO5orBkqtMQ80vVGpjPncgPWfdYk9NIO7VL8uQx254YWCi JiW4jsZ7X0lJmBSatfvD2HuntaOxMMk36UYGILy1m4Ufm4MztLV1axfihbESfa2gDikk 3qnQGinPP6FYwIET4foOcxH6W6vGkn/X8NnF1smFRnAhp+etV4q44saRk7rsGy2gkJuS 43eA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=3qRBHeH49omwqgoxg8Z+ZrefAQ7kYCLg+akntsfuZdA=; b=mbiZ8sMnsH44jPBFkG1xuoqTB2sRLQmQPyQFZ6aVqHQfbQivV1za73wUHAOH1u23sq 8F01JsUYh+bIe9/5K0HGRFRdNIVbMMAtRpIHxLI/Z0iF+3iZV+jQByVkOkP+e/TW5kP4 eCw12dYVG0IbMblOYBDVHHxYEv0GCj/hhlmyeIKONly40EtBzQMD3OQF4vUrsAljA6p+ nlOTUgkZNaoihX5CtDjaHLqiqK58WcXdNzcYhRSJgo5AqmJJxnYx5fU+syemK8ejBZ1y HS4p6VJvg5PGPVcQvgWG9Yv15QhlW0yS5PJhQhmjWd1F4nAPRydVsC2DLIx/VKr1Xtye CZOA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QR0vyAfX; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id b25si551154qtq.275.2020.03.10.23.53.40 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Mar 2020 23:53:40 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QR0vyAfX; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:46668 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvFM-0003b3-6w for patch@linaro.org; Wed, 11 Mar 2020 02:53:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59515) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6b-0006FL-PO for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6Z-0001ws-Kd for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:37 -0400 Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f]:34341) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6Y-0001tB-7v for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:34 -0400 Received: by mail-pl1-x62f.google.com with SMTP id a23so630448plm.1 for ; Tue, 10 Mar 2020 23:44:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3qRBHeH49omwqgoxg8Z+ZrefAQ7kYCLg+akntsfuZdA=; b=QR0vyAfXuvl8umdBF3mzlxvOCfvxsRghnY2q01fR0IaVMAMSiFnoy9QsH7BNhA652F MFLrDpP6DPCiSXowIJfoAgQHSdz7uZeNQz2DeY50VZFkuJghMXu5FsU5zzvW+j4cVoaD bXO45GF7RwA5i+Dmsu5bPT8Nrs2XRII0sVS6dSeQhbXapVNWF/JIgV/bt7MjV7gu6BmX FyAIiUn88NWN47ByGImQFIVw2gZP/cWrTPUgX6L88NY3f+86Iy26fxNP135MFGqDTIKg /VqZ7aqTlJ3YFTPVc8xnS/qLAfcQx+0qffx71PjMhsrHEnJTyrUDAMBM+YL1v/XkU7FO kULg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3qRBHeH49omwqgoxg8Z+ZrefAQ7kYCLg+akntsfuZdA=; b=FfkvzGM8HwnfJEqJBrunbMTqi8nwct50dsjRhuL5MvkdlO50qFZfsNSnoOveocgz6m FCUN+CAwnDkfu8Scp1UUXLfvjn2a9aSYtj2IxI/7Mu053zSHbeZiY6rUY6947U94yVPh e/lAyl8DXjJgOQVcNWhyDO/dGyKvW5O3h0tdTsItjDIDlLsRC9MzYWCAzv4edcXKeOcQ dHNBADVhJTdwzQNGfwsxT+DT1LFxIymQ8AQ8irBZJ5jIh3m24rJbPikFdyhcKYiht2GI 8uuE73/LZRNb0RpDJ9q1pr2SP+Vya9M5oFfmY81W7PTkKz2X3aKt1s3V22O1GpYh5Jon 7cdQ== X-Gm-Message-State: ANhLgQ0rovBZPqlpPqEqApRjmAfZ41ZM0qExlr4Ck763POA0Hh6atZ6L itukmB0N3wsZSMpwDUW9UoWhRdMWo/w= X-Received: by 2002:a17:902:7b8d:: with SMTP id w13mr1795283pll.78.1583909070792; Tue, 10 Mar 2020 23:44:30 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:29 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 07/16] target/arm: Adjust interface of sve_ld1_host_fn Date: Tue, 10 Mar 2020 23:44:11 -0700 Message-Id: <20200311064420.30606-8-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::62f X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" The current interface includes a loop; change it to load a single element. We will then be able to use the function for ld{2,3,4} where individual vector elements are not adjacent. Replace each call with the simplest possible loop over active elements. Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 127 +++++++++++++++++++++------------------- 1 file changed, 66 insertions(+), 61 deletions(-) -- 2.20.1 Reviewed-by: Peter Maydell diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 3f653e46a0..1397c2b634 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -3972,20 +3972,10 @@ void HELPER(sve_fcmla_zpzzz_d)(CPUARMState *env, void *vg, uint32_t desc) */ /* - * Load elements into @vd, controlled by @vg, from @host + @mem_ofs. - * Memory is valid through @host + @mem_max. The register element - * indicies are inferred from @mem_ofs, as modified by the types for - * which the helper is built. Return the @mem_ofs of the first element - * not loaded (which is @mem_max if they are all loaded). - * - * For softmmu, we have fully validated the guest page. For user-only, - * we cannot fully validate without taking the mmap lock, but since we - * know the access is within one host page, if any access is valid they - * all must be valid. However, when @vg is all false, it may be that - * no access is valid. + * Load elements into @vd + @reg_off, from @host, + * or the reverse for stores. */ -typedef intptr_t sve_ld1_host_fn(void *vd, void *vg, void *host, - intptr_t mem_ofs, intptr_t mem_max); +typedef void sve_ldst1_host_fn(void *vd, intptr_t reg_off, void *host); /* * Load one element into @vd + @reg_off from (@env, @vaddr, @ra). @@ -4000,23 +3990,16 @@ typedef void sve_ldst1_tlb_fn(CPUARMState *env, void *vd, intptr_t reg_off, * For *_tlb, this uses the cpu_*_data_ra helpers. There are not * endian-specific versions of these, so we must handle endianness * locally. + * + * For *_host, this is a trivial application of the + * endian-specific access followed by a store into the vector register. */ #define DO_LD_HOST(NAME, H, TYPEE, TYPEM, HOST) \ -static intptr_t sve_##NAME##_host(void *vd, void *vg, void *host, \ - intptr_t mem_off, const intptr_t mem_max) \ -{ \ - intptr_t reg_off = mem_off * (sizeof(TYPEE) / sizeof(TYPEM)); \ - uint64_t *pg = vg; \ - while (mem_off + sizeof(TYPEM) <= mem_max) { \ - TYPEM val = 0; \ - if (likely((pg[reg_off >> 6] >> (reg_off & 63)) & 1)) { \ - val = HOST(host + mem_off); \ - } \ - *(TYPEE *)(vd + H(reg_off)) = val; \ - mem_off += sizeof(TYPEM), reg_off += sizeof(TYPEE); \ - } \ - return mem_off; \ +static void sve_##NAME##_host(void *vd, intptr_t reg_off, void *host) \ +{ \ + TYPEM val = HOST(host); \ + *(TYPEE *)(vd + H(reg_off)) = val; \ } #define DO_LD_TLB(NAME, H, TYPEE, TYPEM, BSWAP, TLB) \ @@ -4414,7 +4397,7 @@ static inline bool test_host_page(void *host) static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, uint32_t desc, const uintptr_t retaddr, const int esz, const int msz, - sve_ld1_host_fn *host_fn, + sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); @@ -4448,8 +4431,12 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, if (likely(split == mem_max)) { host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); if (test_host_page(host)) { - mem_off = host_fn(vd, vg, host - mem_off, mem_off, mem_max); - tcg_debug_assert(mem_off == mem_max); + intptr_t i = reg_off; + host -= mem_off; + do { + host_fn(vd, i, host + (i >> diffsz)); + i = find_next_active(vg, i + (1 << esz), reg_max, esz); + } while (i < reg_max); /* After having taken any fault, zero leading inactive elements. */ swap_memzero(vd, reg_off); return; @@ -4462,7 +4449,12 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, */ #ifdef CONFIG_USER_ONLY swap_memzero(&scratch, reg_off); - host_fn(&scratch, vg, g2h(addr), mem_off, mem_max); + host = g2h(addr); + do { + host_fn(&scratch, reg_off, host + (reg_off >> diffsz)); + reg_off += 1 << esz; + reg_off = find_next_active(vg, reg_off, reg_max, esz); + } while (reg_off < reg_max); #else memset(&scratch, 0, reg_max); goto start; @@ -4480,9 +4472,13 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); if (host) { - mem_off = host_fn(&scratch, vg, host - mem_off, - mem_off, split); - reg_off = mem_off << diffsz; + host -= mem_off; + do { + host_fn(&scratch, reg_off, host + mem_off); + reg_off += 1 << esz; + reg_off = find_next_active(vg, reg_off, reg_max, esz); + mem_off = reg_off >> diffsz; + } while (split - mem_off >= (1 << msz)); continue; } } @@ -4709,7 +4705,7 @@ static void record_fault(CPUARMState *env, uintptr_t i, uintptr_t oprsz) static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, uint32_t desc, const uintptr_t retaddr, const int esz, const int msz, - sve_ld1_host_fn *host_fn, + sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); @@ -4719,7 +4715,7 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, const int diffsz = esz - msz; const intptr_t reg_max = simd_oprsz(desc); const intptr_t mem_max = reg_max >> diffsz; - intptr_t split, reg_off, mem_off; + intptr_t split, reg_off, mem_off, i; void *host; /* Skip to the first active element. */ @@ -4742,28 +4738,18 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, if (likely(split == mem_max)) { host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); if (test_host_page(host)) { - mem_off = host_fn(vd, vg, host - mem_off, mem_off, mem_max); - tcg_debug_assert(mem_off == mem_max); + i = reg_off; + host -= mem_off; + do { + host_fn(vd, i, host + (i >> diffsz)); + i = find_next_active(vg, i + (1 << esz), reg_max, esz); + } while (i < reg_max); /* After any fault, zero any leading inactive elements. */ swap_memzero(vd, reg_off); return; } } -#ifdef CONFIG_USER_ONLY - /* - * The page(s) containing this first element at ADDR+MEM_OFF must - * be valid. Considering that this first element may be misaligned - * and cross a page boundary itself, take the rest of the page from - * the last byte of the element. - */ - split = max_for_page(addr, mem_off + (1 << msz) - 1, mem_max); - mem_off = host_fn(vd, vg, g2h(addr), mem_off, split); - - /* After any fault, zero any leading inactive elements. */ - swap_memzero(vd, reg_off); - reg_off = mem_off << diffsz; -#else /* * Perform one normal read, which will fault or not. * But it is likely to bring the page into the tlb. @@ -4780,11 +4766,15 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, if (split >= (1 << msz)) { host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); if (host) { - mem_off = host_fn(vd, vg, host - mem_off, mem_off, split); - reg_off = mem_off << diffsz; + host -= mem_off; + do { + host_fn(vd, reg_off, host + mem_off); + reg_off += 1 << esz; + reg_off = find_next_active(vg, reg_off, reg_max, esz); + mem_off = reg_off >> diffsz; + } while (split - mem_off >= (1 << msz)); } } -#endif record_fault(env, reg_off, reg_max); } @@ -4794,7 +4784,7 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, */ static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong addr, uint32_t desc, const int esz, const int msz, - sve_ld1_host_fn *host_fn) + sve_ldst1_host_fn *host_fn) { const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); void *vd = &env->vfp.zregs[rd]; @@ -4809,7 +4799,13 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong addr, host = tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx); if (likely(page_check_range(addr, mem_max, PAGE_READ) == 0)) { /* The entire operation is valid and will not fault. */ - host_fn(vd, vg, host, 0, mem_max); + reg_off = 0; + do { + mem_off = reg_off >> diffsz; + host_fn(vd, reg_off, host + mem_off); + reg_off += 1 << esz; + reg_off = find_next_active(vg, reg_off, reg_max, esz); + } while (reg_off < reg_max); return; } #endif @@ -4829,8 +4825,12 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong addr, if (page_check_range(addr + mem_off, 1 << msz, PAGE_READ) == 0) { /* At least one load is valid; take the rest of the page. */ split = max_for_page(addr, mem_off + (1 << msz) - 1, mem_max); - mem_off = host_fn(vd, vg, host, mem_off, split); - reg_off = mem_off << diffsz; + do { + host_fn(vd, reg_off, host + mem_off); + reg_off += 1 << esz; + reg_off = find_next_active(vg, reg_off, reg_max, esz); + mem_off = reg_off >> diffsz; + } while (split - mem_off >= (1 << msz)); } #else /* @@ -4851,8 +4851,13 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong addr, host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); split = max_for_page(addr, mem_off, mem_max); if (host && split >= (1 << msz)) { - mem_off = host_fn(vd, vg, host - mem_off, mem_off, split); - reg_off = mem_off << diffsz; + host -= mem_off; + do { + host_fn(vd, reg_off, host + mem_off); + reg_off += 1 << esz; + reg_off = find_next_active(vg, reg_off, reg_max, esz); + mem_off = reg_off >> diffsz; + } while (split - mem_off >= (1 << msz)); } #endif From patchwork Wed Mar 11 06:44:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 184392 Delivered-To: patch@linaro.org Received: by 2002:a92:1f12:0:0:0:0:0 with SMTP id i18csp281181ile; Tue, 10 Mar 2020 23:50:09 -0700 (PDT) X-Google-Smtp-Source: ADFU+vt3otHZFSwYhbMwhoJFmqvF4hQ4P3q5liUIlOnAy5c4CmECLqaRFY0V4JGpfrtSbTWDk0o6 X-Received: by 2002:ac8:b8d:: with SMTP id h13mr1349913qti.298.1583909409242; Tue, 10 Mar 2020 23:50:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1583909409; cv=none; d=google.com; s=arc-20160816; b=zGWc+5+ifVoIvXR65uBhnJwdplvsKsaqSiODO27hGUomxuoGC1DcMKBS+l6XAP/Qmx eQ/daCearHzMGzkqAviHvHwWddUf28S13D2YfPqRckgAksdk8Fz6M2JjxNzP0WHv9bah WWbPewejDnrTHNxKViItzdC+lS5uLPYNEwP2CaY0o06GuVHlgIILhAfaGs1Awml/4YfR Rtyl6AwoXwf61J4gdxZ9mecW1WMmaXVN+6DWUhX+M0PjI00vgqq920n4Snrz3QCKOHtg cIvVVORrn4JUegZciONQGXNgN0Ts3HxIgCfeuyG+B7fn88zhSGt1AI4k5jvPN86fUbHN 9+eA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=hwv0ZdGNzk1JPF3B9TKzm+UaZXEnJM80ZJXFaFLa00w=; b=vxtOdAUSB8Egthe0CD+vIlEJC1vki+hnwiSBFVlRX6IN55B7ioTWQl/cGBZjxRMlnB VxF24ajiiuDEtKP3xR9Og3ZS9e6jiJ4cvxvuY0Q9+lapsR0i3DOQil14RTwn95JpOJHJ EzQ4zh+8OYorTkCLW8Mxkf7rTORtN2TXpwsngMIx8kmlCCKl8HQQzVum6b87AQASQ8tE e50VutG7hk9Fk0wmMZqa0oP4/CR1g3E6JwsYdl+jT44i+DRCbSz9hvH9+h1QMHxPLeQo GH4m38FVn41CoRYB2vvzoxadPw4T5EI3r6mFt/2/Jf69YUe9UZMtF7QXdfZKKrajY30o C8Iw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="N2/L1msj"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id n186si507261qke.137.2020.03.10.23.50.09 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Mar 2020 23:50:09 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="N2/L1msj"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:46538 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvBw-0006Eu-OT for patch@linaro.org; Wed, 11 Mar 2020 02:50:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59513) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6b-0006FI-NW for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6Z-0001xD-MB for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:37 -0400 Received: from mail-pj1-x1041.google.com ([2607:f8b0:4864:20::1041]:34411) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6Z-0001v3-9A for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:35 -0400 Received: by mail-pj1-x1041.google.com with SMTP id 39so1174667pjo.1 for ; Tue, 10 Mar 2020 23:44:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hwv0ZdGNzk1JPF3B9TKzm+UaZXEnJM80ZJXFaFLa00w=; b=N2/L1msjt0mGGn+cirOIcg8NYShSTV/hYDQ59P8VIQpAf5S+ShR5sh2OuX2r3JvY8g UCDGflEv8oz8HRj0mlU47215JBpcOrDUjse83MJkXZaCosGYIGj1h/IeZMvuhr7ALRau CXJHgl+tJVFn20/93wkBMBRJPWOph/beNzPN82tDgaFzpfj5aCnFumB84CmnYp7XqVir 3GhWX5ZPFC0jDd6g24cjnS7/yoL7PMmIuw90Ba/P/Ifeb0Av1J2TwnI64Pf4guEJrsT5 7sq4MKLbk7KjQIHwAM9APieWLOMXVuJV9rjBltciNYy6ulCxq79tZQisXjOUNl/nQzen nOgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hwv0ZdGNzk1JPF3B9TKzm+UaZXEnJM80ZJXFaFLa00w=; b=Pi9UnlpLvwHWkurJlANl4yL4DeEW8beBH9mLkcIj9VVYF1vr6LVnNDIctZY9Rm78lQ 7mRMmrh+l9Hj6/K1WWdT+9+XMNO+T3uP+3GokbT4hJSBX1x1TVCJgbwQzVbCQZXz9OfJ hxmLRGSSPbl9O7GuGSyV+S69jCIv1s8Y8g0GhfEL6cAne2fHRi4ZuMqbKB7HeZf1PG9V 4R0cMBG0GrxhuYbhKo0ZQQA9ifJpYnRvYUnP8EbIXYH3TI5nxMhZvm3OKOgLjFJVMS9g DNMFvGskuKlfG/XBjhIqr2xNrk84VSJdbWMPb8IeK/PJin8/th8A2II+wmMAYBbJz39o p7fw== X-Gm-Message-State: ANhLgQ3BPBBh4F9e5PC0QXiG2lXcbCyPARRsHTpq4TfKSH+0smCd9y/T fT6G801uXTmLAxXUG8YWcHENv6Eh6hg= X-Received: by 2002:a17:90b:4c47:: with SMTP id np7mr1873292pjb.140.1583909072197; Tue, 10 Mar 2020 23:44:32 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:31 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 08/16] target/arm: Use SVEContLdSt in sve_ld1_r Date: Tue, 10 Mar 2020 23:44:12 -0700 Message-Id: <20200311064420.30606-9-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::1041 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" First use of the new helper functions, so we can remove the unused markup. No longer need a scratch for user-only, as we completely probe the page set before reading; system mode still requires a scratch for MMIO. Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 188 +++++++++++++++++++++------------------- 1 file changed, 97 insertions(+), 91 deletions(-) -- 2.20.1 Reviewed-by: Peter Maydell diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 1397c2b634..b827900a4e 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4227,9 +4227,9 @@ typedef struct { * final element on each page. Identify any single element that spans * the page boundary. Return true if there are any active elements. */ -static bool __attribute__((unused)) -sve_cont_ldst_elements(SVEContLdSt *info, target_ulong addr, uint64_t *vg, - intptr_t reg_max, int esz, int msize) +static bool sve_cont_ldst_elements(SVEContLdSt *info, target_ulong addr, + uint64_t *vg, intptr_t reg_max, + int esz, int msize) { const int esize = 1 << esz; const uint64_t pg_mask = pred_esz_masks[esz]; @@ -4319,10 +4319,9 @@ sve_cont_ldst_elements(SVEContLdSt *info, target_ulong addr, uint64_t *vg, * Control the generation of page faults with @fault. Return false if * there is no work to do, which can only happen with @fault == FAULT_NO. */ -static bool __attribute__((unused)) -sve_cont_ldst_pages(SVEContLdSt *info, SVEContFault fault, CPUARMState *env, - target_ulong addr, MMUAccessType access_type, - uintptr_t retaddr) +static bool sve_cont_ldst_pages(SVEContLdSt *info, SVEContFault fault, + CPUARMState *env, target_ulong addr, + MMUAccessType access_type, uintptr_t retaddr) { int mmu_idx = cpu_mmu_index(env, false); int mem_off = info->mem_off_first[0]; @@ -4394,109 +4393,116 @@ static inline bool test_host_page(void *host) /* * Common helper for all contiguous one-register predicated loads. */ -static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, - uint32_t desc, const uintptr_t retaddr, - const int esz, const int msz, - sve_ldst1_host_fn *host_fn, - sve_ldst1_tlb_fn *tlb_fn) +static inline QEMU_ALWAYS_INLINE +void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, + uint32_t desc, const uintptr_t retaddr, + const int esz, const int msz, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); - const int mmu_idx = get_mmuidx(oi); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); void *vd = &env->vfp.zregs[rd]; - const int diffsz = esz - msz; const intptr_t reg_max = simd_oprsz(desc); - const intptr_t mem_max = reg_max >> diffsz; - ARMVectorReg scratch; + intptr_t reg_off, reg_last, mem_off; + SVEContLdSt info; void *host; - intptr_t split, reg_off, mem_off; + int flags; - /* Find the first active element. */ - reg_off = find_next_active(vg, 0, reg_max, esz); - if (unlikely(reg_off == reg_max)) { + /* Find the active elements. */ + if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, 1 << msz)) { /* The entire predicate was false; no load occurs. */ memset(vd, 0, reg_max); return; } - mem_off = reg_off >> diffsz; - /* - * If the (remaining) load is entirely within a single page, then: - * For softmmu, and the tlb hits, then no faults will occur; - * For user-only, either the first load will fault or none will. - * We can thus perform the load directly to the destination and - * Vd will be unmodified on any exception path. - */ - split = max_for_page(addr, mem_off, mem_max); - if (likely(split == mem_max)) { - host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); - if (test_host_page(host)) { - intptr_t i = reg_off; - host -= mem_off; - do { - host_fn(vd, i, host + (i >> diffsz)); - i = find_next_active(vg, i + (1 << esz), reg_max, esz); - } while (i < reg_max); - /* After having taken any fault, zero leading inactive elements. */ - swap_memzero(vd, reg_off); - return; - } - } + /* Probe the page(s). Exit with exception for any invalid page. */ + sve_cont_ldst_pages(&info, FAULT_ALL, env, addr, MMU_DATA_LOAD, retaddr); - /* - * Perform the predicated read into a temporary, thus ensuring - * if the load of the last element faults, Vd is not modified. - */ + flags = info.page[0].flags | info.page[1].flags; + if (unlikely(flags != 0)) { #ifdef CONFIG_USER_ONLY - swap_memzero(&scratch, reg_off); - host = g2h(addr); - do { - host_fn(&scratch, reg_off, host + (reg_off >> diffsz)); - reg_off += 1 << esz; - reg_off = find_next_active(vg, reg_off, reg_max, esz); - } while (reg_off < reg_max); + g_assert_not_reached(); #else - memset(&scratch, 0, reg_max); - goto start; - while (1) { - reg_off = find_next_active(vg, reg_off, reg_max, esz); - if (reg_off >= reg_max) { - break; - } - mem_off = reg_off >> diffsz; - split = max_for_page(addr, mem_off, mem_max); + /* + * At least one page includes MMIO (or watchpoints). + * Any bus operation can fail with cpu_transaction_failed, + * which for ARM will raise SyncExternal. Perform the load + * into scratch memory to preserve register state until the end. + */ + ARMVectorReg scratch; - start: - if (split - mem_off >= (1 << msz)) { - /* At least one whole element on this page. */ - host = tlb_vaddr_to_host(env, addr + mem_off, - MMU_DATA_LOAD, mmu_idx); - if (host) { - host -= mem_off; - do { - host_fn(&scratch, reg_off, host + mem_off); - reg_off += 1 << esz; - reg_off = find_next_active(vg, reg_off, reg_max, esz); - mem_off = reg_off >> diffsz; - } while (split - mem_off >= (1 << msz)); - continue; + memset(&scratch, 0, reg_max); + mem_off = info.mem_off_first[0]; + reg_off = info.reg_off_first[0]; + reg_last = info.reg_off_last[1]; + if (reg_last < 0) { + reg_last = info.reg_off_split; + if (reg_last < 0) { + reg_last = info.reg_off_last[0]; } } - /* - * Perform one normal read. This may fault, longjmping out to the - * main loop in order to raise an exception. It may succeed, and - * as a side-effect load the TLB entry for the next round. Finally, - * in the extremely unlikely case we're performing this operation - * on I/O memory, it may succeed but not bring in the TLB entry. - * But even then we have still made forward progress. - */ - tlb_fn(env, &scratch, reg_off, addr + mem_off, retaddr); - reg_off += 1 << esz; - } -#endif + do { + uint64_t pg = vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + tlb_fn(env, &scratch, reg_off, addr + mem_off, retaddr); + } + reg_off += 1 << esz; + mem_off += 1 << msz; + } while (reg_off & 63); + } while (reg_off <= reg_last); - memcpy(vd, &scratch, reg_max); + memcpy(vd, &scratch, reg_max); + return; +#endif + } + + /* The entire operation is in RAM, on valid pages. */ + + memset(vd, 0, reg_max); + mem_off = info.mem_off_first[0]; + reg_off = info.reg_off_first[0]; + reg_last = info.reg_off_last[0]; + host = info.page[0].host; + + while (reg_off <= reg_last) { + uint64_t pg = vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + host_fn(vd, reg_off, host + mem_off); + } + reg_off += 1 << esz; + mem_off += 1 << msz; + } while (reg_off <= reg_last && (reg_off & 63)); + } + + /* + * Use the slow path to manage the cross-page misalignment. + * But we know this is RAM and cannot trap. + */ + mem_off = info.mem_off_split; + if (unlikely(mem_off >= 0)) { + tlb_fn(env, vd, info.reg_off_split, addr + mem_off, retaddr); + } + + mem_off = info.mem_off_first[1]; + if (unlikely(mem_off >= 0)) { + reg_off = info.reg_off_first[1]; + reg_last = info.reg_off_last[1]; + host = info.page[1].host; + + do { + uint64_t pg = vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + host_fn(vd, reg_off, host + mem_off); + } + reg_off += 1 << esz; + mem_off += 1 << msz; + } while (reg_off & 63); + } while (reg_off <= reg_last); + } } #define DO_LD1_1(NAME, ESZ) \ From patchwork Wed Mar 11 06:44:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 184390 Delivered-To: patch@linaro.org Received: by 2002:a92:1f12:0:0:0:0:0 with SMTP id i18csp280767ile; Tue, 10 Mar 2020 23:49:33 -0700 (PDT) X-Google-Smtp-Source: ADFU+vuKuG6Y16qcj2JJV+6xqKZwLJw4azKtakZUTU/sZUR2rHolmhBDc3SoSko7q0JPBEHawtV5 X-Received: by 2002:a37:688f:: with SMTP id d137mr1379870qkc.54.1583909373882; Tue, 10 Mar 2020 23:49:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1583909373; cv=none; d=google.com; s=arc-20160816; b=yLB5WXXDbedmPDw4qQwQrfozBxV89An3vvBqzhh6dMv0nfXfKv9zS0OwkldfH8e5a0 av+idrgq3s1wMAYcaIzp70LVCHnQtl5tzQCPmzUM3So34ETnMes1mHHuDYusSOeL03/9 TMRaBDFWOce/AuVLcLljbaJV05kb/stU9juXxcewPWiIvjHYWKneP++GDlywj8bBDtQJ V+Z0R+m/xtaSFAo6CXR24QlM662BjNyOaMbdWqKs8SvMrtiJ2FAjDbxXYKCmRRpTcWAY X4pTeBcryV81NdrjXyboF+QyRFk75tr+TPyplm6HkT7DsycRrIuJEhJ/+5nxgyy9dWG3 e9sw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=Tz6r3+bMnyO04HqqXq5aHLFFiU4Amy53tPfIpPhOcd0=; b=wHhXaawe0qruHMst5AnIwbTOEDEH58UsR5i0s0KiEyOqVqgIY9LvBJ5cgVF7nXfRMB ASaMXokeSZ9qOiqfuduQL6YZ0JmqLRhYljXtGzFPHr0XSlqoyCycr4/eUajBAU34D4nW bSec4ritcYdMtJoAEFzGHkrz9Fjg3qha9zAA/2VCgCcQbWS/5liBFvxnZErvBuTFBypy p0zuR68mRTcqXG1xf044ZmkwWaz5kuiMXIijlJQo78IPJ3WwKyExwiK90EeiEn+yZ0tB yV+cm+JmDk9SRUzYbhWozwKkxNim5wE0+yPROojPcTbInlXj3LbHJLXGanPOWN+fSmk3 Afxg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=iY6j3nvV; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id e11si516138qth.287.2020.03.10.23.49.33 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Mar 2020 23:49:33 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=iY6j3nvV; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:46516 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvBN-00050y-EO for patch@linaro.org; Wed, 11 Mar 2020 02:49:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59508) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6b-0006F9-MV for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6Z-0001xe-Q9 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:37 -0400 Received: from mail-pl1-x643.google.com ([2607:f8b0:4864:20::643]:42707) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6Z-0001vs-Jf for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:35 -0400 Received: by mail-pl1-x643.google.com with SMTP id t3so613231plz.9 for ; Tue, 10 Mar 2020 23:44:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Tz6r3+bMnyO04HqqXq5aHLFFiU4Amy53tPfIpPhOcd0=; b=iY6j3nvVY5+1mQagFTmecA511o9b0ILObTCimvRrrl3YbKGbCuEWO9y0TAksf9STzJ Jhc005Jx+vL3VyAm9GUq08M8WxBGQX/pMAqbJeCLUlLFY5KNytNBFNqfyRehExCwvYgC 4PTaZs7w8Rxs/yonODh+eY3QRjdVBmaXaoocQgugj5bpyRKiANwoG6paPfVmEiFJbKeb mLtYEg0SLg2Kl8EL+Gz19pF1lAmdFLA/6pGW0MouAdPS0yXuyYClmLQa2Mjyl0/cgEah IML880+wuN3TmdARqkKLmKC7gbbYJvOdXuNNnUN9mRfWxOpd6sYXDcW67uqkXonBAJWQ CFgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Tz6r3+bMnyO04HqqXq5aHLFFiU4Amy53tPfIpPhOcd0=; b=SvRecTS1Jm9+h/iSS2baDfsrki1W4DKIFMu+jDnXuP9aK+pNh9x9C1o2lgV7/Vr05c xotaku8K4rccIDgDJLWxVe+IFZq49Q2nqzsJGJRqv6j8iL+g7c+90Bm7sW4cgVjtuknv ehwD5oRZozGjG1MTPLBzCNM6oIyJ/GkBDHzwlY65PnnmhWD3KB3FsqCrPw7QG66jqIL6 iDXgkRuaSnZTwo6XtlLx5bqNgaGTlsIn/AFjnbq/ofwSINpE2p1RuxlXh4kCkVtl5vAO 8dfOE9OMcVr8GZjh1zabVx4uCy4qq/K7TZ+yI+dXgOJnPTaMwURBPkZSMF+480BmM3jf /OBw== X-Gm-Message-State: ANhLgQ2J/hf2cOnonC17a0Aw8YtGbHta0gv1rWSMdUynZKRxxFck4NI2 Hd852ffcYNYH2CQLb8ivbVau6f2+J/Q= X-Received: by 2002:a17:90b:3653:: with SMTP id nh19mr1896194pjb.41.1583909073613; Tue, 10 Mar 2020 23:44:33 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:32 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 09/16] target/arm: Handle watchpoints in sve_ld1_r Date: Tue, 10 Mar 2020 23:44:13 -0700 Message-Id: <20200311064420.30606-10-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::643 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Handle all of the watchpoints for active elements all at once, before we've modified the vector register. This removes the TLB_WATCHPOINT bit from page[].flags, which means that we can use the normal fast path via RAM. Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 70 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 69 insertions(+), 1 deletion(-) -- 2.20.1 Reviewed-by: Peter Maydell diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index b827900a4e..2c0e838ed3 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4377,6 +4377,68 @@ static bool sve_cont_ldst_pages(SVEContLdSt *info, SVEContFault fault, return have_work; } +static void sve_cont_ldst_watchpoints(SVEContLdSt *info, CPUARMState *env, + uint64_t *vg, target_ulong addr, + int esize, int msize, int wp_access, + uintptr_t retaddr) +{ +#ifndef CONFIG_USER_ONLY + intptr_t mem_off, reg_off, reg_last; + int flags0 = info->page[0].flags; + int flags1 = info->page[1].flags; + + if (likely(!((flags0 | flags1) & TLB_WATCHPOINT))) { + return; + } + + /* Indicate that watchpoints are handled. */ + info->page[0].flags = flags0 & ~TLB_WATCHPOINT; + info->page[1].flags = flags1 & ~TLB_WATCHPOINT; + + if (flags0 & TLB_WATCHPOINT) { + mem_off = info->mem_off_first[0]; + reg_off = info->reg_off_first[0]; + reg_last = info->reg_off_last[0]; + + while (reg_off <= reg_last) { + uint64_t pg = vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + cpu_check_watchpoint(env_cpu(env), addr + mem_off, msize, + info->page[0].attrs, wp_access, retaddr); + } + reg_off += esize; + mem_off += msize; + } while (reg_off <= reg_last && (reg_off & 63)); + } + } + + mem_off = info->mem_off_split; + if (mem_off >= 0) { + cpu_check_watchpoint(env_cpu(env), addr + mem_off, msize, + info->page[0].attrs, wp_access, retaddr); + } + + mem_off = info->mem_off_first[1]; + if ((flags1 & TLB_WATCHPOINT) && mem_off >= 0) { + reg_off = info->reg_off_first[1]; + reg_last = info->reg_off_last[1]; + + do { + uint64_t pg = vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + cpu_check_watchpoint(env_cpu(env), addr + mem_off, msize, + info->page[1].attrs, wp_access, retaddr); + } + reg_off += esize; + mem_off += msize; + } while (reg_off & 63); + } while (reg_off <= reg_last); + } +#endif +} + /* * The result of tlb_vaddr_to_host for user-only is just g2h(x), * which is always non-null. Elide the useless test. @@ -4418,13 +4480,19 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, /* Probe the page(s). Exit with exception for any invalid page. */ sve_cont_ldst_pages(&info, FAULT_ALL, env, addr, MMU_DATA_LOAD, retaddr); + /* Handle watchpoints for all active elements. */ + sve_cont_ldst_watchpoints(&info, env, vg, addr, 1 << esz, 1 << msz, + BP_MEM_READ, retaddr); + + /* TODO: MTE check. */ + flags = info.page[0].flags | info.page[1].flags; if (unlikely(flags != 0)) { #ifdef CONFIG_USER_ONLY g_assert_not_reached(); #else /* - * At least one page includes MMIO (or watchpoints). + * At least one page includes MMIO. * Any bus operation can fail with cpu_transaction_failed, * which for ARM will raise SyncExternal. Perform the load * into scratch memory to preserve register state until the end. From patchwork Wed Mar 11 06:44:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 184398 Delivered-To: patch@linaro.org Received: by 2002:a92:1f12:0:0:0:0:0 with SMTP id i18csp284736ile; Tue, 10 Mar 2020 23:55:13 -0700 (PDT) X-Google-Smtp-Source: ADFU+vuFTCTRHTcZ8wuX2Q3LAFOffSmP9JNISlJzNIH1faIelVAxtrHLpjSDcN9nBAbghoD63yZk X-Received: by 2002:a05:620a:1ea:: with SMTP id x10mr1392695qkn.412.1583909713300; Tue, 10 Mar 2020 23:55:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1583909713; cv=none; d=google.com; s=arc-20160816; b=hoe4kJTrGGL2Cf1EzJZnTIw6B4BanSQUdLgDPed9qgiuGYGqAvw2vubjHq0qxQvqVI 0UUtKPDr6vo1Acj9W6NxS52E1lASkaGGQEqxcn09uDqkBPLCGgnPmTL75U3lcjoGflxh rzgx78Pr7nBcX77bz/DXl+X48Ih/U5gMSycBi8vfULc7cTHCmOLSOdXVDYz4JHOtnQ6W qMB5wj3nj8le5o4FQhI4EQSHnkeqCudlpgRtrBZoSqCA2y9TGZ/+SPEZvYa8qS7VOLNR fCkJtUcOFztRoLvKkQ97cuaR27zJtj2MIQJSeknfevWjKcxFK7mNqDrD4zdiruAWO5km jl3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=2YeIG7aIPNgxF4xn6eg+uOsdMNeYSZujzfaE66vwnTQ=; b=tLZnbYaA93+Q5Yxmym77eqBy8ykD3SHKniyjpxQ2DZMa29EZo6Uudw1l8wMeBtkJWa li0EFw0qzYD333zGHfOgZqEVD9T4p6NMLl+1awbjS9Fir2JdxM73tU7qFs3RJ86otzLB i111/aS5cOcfJ7bXPIJkHGw74wXx+K71+HaAbGRAefuPa2G7jr03zeutatEiuQcYE+WG 2iJwtrBFmUarCaalNH9ad0eB+/mAJ3TEiWkWhhBsGdPAmtJYKPX50zplJDj2WYiHNygi EEskdDsaW8i3oGoAp0iMAZr/I3a4CsQJPqqv8FjX3xb2axfhcFN3KlQTVpWAvSchhlsk QqMQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=TwxJZbeP; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id j9si484425qtp.388.2020.03.10.23.55.13 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Mar 2020 23:55:13 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=TwxJZbeP; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:46744 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvGq-00066J-PX for patch@linaro.org; Wed, 11 Mar 2020 02:55:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59577) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6e-0006MW-78 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6b-0001zw-Th for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:40 -0400 Received: from mail-pl1-x643.google.com ([2607:f8b0:4864:20::643]:33829) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6b-0001yF-J7 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:37 -0400 Received: by mail-pl1-x643.google.com with SMTP id a23so630572plm.1 for ; Tue, 10 Mar 2020 23:44:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2YeIG7aIPNgxF4xn6eg+uOsdMNeYSZujzfaE66vwnTQ=; b=TwxJZbePUuZD0UKRO0B1c7EH1QHO/USGqld4kccmFkAim3AJz0p7HyYReHA7N4A2LA D05AVktBBaDVZ0u46Ka52UYZCOnLeB7uElinU1MoK26iIBcL9ilGQZ9yut4iy94rEy4j dct4zCcqTlAfVz9RTOe4tiZTBZnI1NSkmPawCohA4FONJHBTemY8/Ld0nSp5jlwoOsXj iy+JeXjzXFuqqSa9m0n1dLpzNEei1hBmrkHU8ExlzuZkX2venoAM5ZfJzY0cVpPfxtCq zWPUDyBH2gWg7fjye0cZMcJOLKapsuAChlwHl4wHI90zW5vrAO5LT2SImMWqPhHYTBlk ye4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2YeIG7aIPNgxF4xn6eg+uOsdMNeYSZujzfaE66vwnTQ=; b=odFj4sgvoDmIG++7YeoMZwnPd3tqy67roup23MTsz7QMwqc6Uv61Os2ol3L3VcGGPq Y/MmaVFvPBpdcIHtXL/QR43vFrsbuyXPzkbmXb6fKzXaO70fT+Pt8uJ21XW4x7CCF2jA zyB/DnipfnJLd1NXZV+tbP40AlItsfigDqKpfVmtzihHk5yA6LlXrBv0uGg/tmYHchnQ 1z0fsJXnMGM7qf1KFy6xJ1EywgdqgQ3LhcE0tbUVHhy5+QYMahw/8awn8FPXnSPvurJ7 jjHsqpAonokiGY5bxHz45dnRWzWcO6P1Vhmx47yWO4rNYMfvZ9fNu7p2kKHPKVtgDiee oaEw== X-Gm-Message-State: ANhLgQ1kvryJwrLbXnUmH3higW3+/MQw67JQIrp9HO9fGSpNbRJ455lt ZRdau8EzV9uboWR3SCNIypoXsFhkoOU= X-Received: by 2002:a17:90b:1983:: with SMTP id mv3mr1910165pjb.86.1583909075176; Tue, 10 Mar 2020 23:44:35 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:34 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 10/16] target/arm: Use SVEContLdSt for multi-register contiguous loads Date: Tue, 10 Mar 2020 23:44:14 -0700 Message-Id: <20200311064420.30606-11-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::643 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 223 ++++++++++++++-------------------------- 1 file changed, 79 insertions(+), 144 deletions(-) -- 2.20.1 Reviewed-by: Peter Maydell diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 2c0e838ed3..56407446eb 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4453,27 +4453,28 @@ static inline bool test_host_page(void *host) } /* - * Common helper for all contiguous one-register predicated loads. + * Common helper for all contiguous 1,2,3,4-register predicated stores. */ static inline QEMU_ALWAYS_INLINE -void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, +void sve_ldN_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, uint32_t desc, const uintptr_t retaddr, - const int esz, const int msz, + const int esz, const int msz, const int N, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - void *vd = &env->vfp.zregs[rd]; const intptr_t reg_max = simd_oprsz(desc); intptr_t reg_off, reg_last, mem_off; SVEContLdSt info; void *host; - int flags; + int flags, i; /* Find the active elements. */ - if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, 1 << msz)) { + if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, N << msz)) { /* The entire predicate was false; no load occurs. */ - memset(vd, 0, reg_max); + for (i = 0; i < N; ++i) { + memset(&env->vfp.zregs[(rd + i) & 31], 0, reg_max); + } return; } @@ -4481,7 +4482,7 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, sve_cont_ldst_pages(&info, FAULT_ALL, env, addr, MMU_DATA_LOAD, retaddr); /* Handle watchpoints for all active elements. */ - sve_cont_ldst_watchpoints(&info, env, vg, addr, 1 << esz, 1 << msz, + sve_cont_ldst_watchpoints(&info, env, vg, addr, 1 << esz, N << msz, BP_MEM_READ, retaddr); /* TODO: MTE check. */ @@ -4497,9 +4498,8 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, * which for ARM will raise SyncExternal. Perform the load * into scratch memory to preserve register state until the end. */ - ARMVectorReg scratch; + ARMVectorReg scratch[4] = { }; - memset(&scratch, 0, reg_max); mem_off = info.mem_off_first[0]; reg_off = info.reg_off_first[0]; reg_last = info.reg_off_last[1]; @@ -4514,21 +4514,29 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, uint64_t pg = vg[reg_off >> 6]; do { if ((pg >> (reg_off & 63)) & 1) { - tlb_fn(env, &scratch, reg_off, addr + mem_off, retaddr); + for (i = 0; i < N; ++i) { + tlb_fn(env, &scratch[i], reg_off, + addr + mem_off + (i << msz), retaddr); + } } reg_off += 1 << esz; - mem_off += 1 << msz; + mem_off += N << msz; } while (reg_off & 63); } while (reg_off <= reg_last); - memcpy(vd, &scratch, reg_max); + for (i = 0; i < N; ++i) { + memcpy(&env->vfp.zregs[(rd + i) & 31], &scratch[i], reg_max); + } return; #endif } /* The entire operation is in RAM, on valid pages. */ - memset(vd, 0, reg_max); + for (i = 0; i < N; ++i) { + memset(&env->vfp.zregs[(rd + i) & 31], 0, reg_max); + } + mem_off = info.mem_off_first[0]; reg_off = info.reg_off_first[0]; reg_last = info.reg_off_last[0]; @@ -4538,10 +4546,13 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, uint64_t pg = vg[reg_off >> 6]; do { if ((pg >> (reg_off & 63)) & 1) { - host_fn(vd, reg_off, host + mem_off); + for (i = 0; i < N; ++i) { + host_fn(&env->vfp.zregs[(rd + i) & 31], reg_off, + host + mem_off + (i << msz)); + } } reg_off += 1 << esz; - mem_off += 1 << msz; + mem_off += N << msz; } while (reg_off <= reg_last && (reg_off & 63)); } @@ -4551,7 +4562,11 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, */ mem_off = info.mem_off_split; if (unlikely(mem_off >= 0)) { - tlb_fn(env, vd, info.reg_off_split, addr + mem_off, retaddr); + reg_off = info.reg_off_split; + for (i = 0; i < N; ++i) { + tlb_fn(env, &env->vfp.zregs[(rd + i) & 31], reg_off, + addr + mem_off + (i << msz), retaddr); + } } mem_off = info.mem_off_first[1]; @@ -4564,10 +4579,13 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, uint64_t pg = vg[reg_off >> 6]; do { if ((pg >> (reg_off & 63)) & 1) { - host_fn(vd, reg_off, host + mem_off); + for (i = 0; i < N; ++i) { + host_fn(&env->vfp.zregs[(rd + i) & 31], reg_off, + host + mem_off + (i << msz)); + } } reg_off += 1 << esz; - mem_off += 1 << msz; + mem_off += N << msz; } while (reg_off & 63); } while (reg_off <= reg_last); } @@ -4577,7 +4595,7 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, void HELPER(sve_##NAME##_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, 0, \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, MO_8, 1, \ sve_##NAME##_host, sve_##NAME##_tlb); \ } @@ -4585,159 +4603,76 @@ void HELPER(sve_##NAME##_r)(CPUARMState *env, void *vg, \ void HELPER(sve_##NAME##_le_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, 1, \ sve_##NAME##_le_host, sve_##NAME##_le_tlb); \ } \ void HELPER(sve_##NAME##_be_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, 1, \ sve_##NAME##_be_host, sve_##NAME##_be_tlb); \ } -DO_LD1_1(ld1bb, 0) -DO_LD1_1(ld1bhu, 1) -DO_LD1_1(ld1bhs, 1) -DO_LD1_1(ld1bsu, 2) -DO_LD1_1(ld1bss, 2) -DO_LD1_1(ld1bdu, 3) -DO_LD1_1(ld1bds, 3) +DO_LD1_1(ld1bb, MO_8) +DO_LD1_1(ld1bhu, MO_16) +DO_LD1_1(ld1bhs, MO_16) +DO_LD1_1(ld1bsu, MO_32) +DO_LD1_1(ld1bss, MO_32) +DO_LD1_1(ld1bdu, MO_64) +DO_LD1_1(ld1bds, MO_64) -DO_LD1_2(ld1hh, 1, 1) -DO_LD1_2(ld1hsu, 2, 1) -DO_LD1_2(ld1hss, 2, 1) -DO_LD1_2(ld1hdu, 3, 1) -DO_LD1_2(ld1hds, 3, 1) +DO_LD1_2(ld1hh, MO_16, MO_16) +DO_LD1_2(ld1hsu, MO_32, MO_16) +DO_LD1_2(ld1hss, MO_32, MO_16) +DO_LD1_2(ld1hdu, MO_64, MO_16) +DO_LD1_2(ld1hds, MO_64, MO_16) -DO_LD1_2(ld1ss, 2, 2) -DO_LD1_2(ld1sdu, 3, 2) -DO_LD1_2(ld1sds, 3, 2) +DO_LD1_2(ld1ss, MO_32, MO_32) +DO_LD1_2(ld1sdu, MO_64, MO_32) +DO_LD1_2(ld1sds, MO_64, MO_32) -DO_LD1_2(ld1dd, 3, 3) +DO_LD1_2(ld1dd, MO_64, MO_64) #undef DO_LD1_1 #undef DO_LD1_2 -/* - * Common helpers for all contiguous 2,3,4-register predicated loads. - */ -static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, int size, uintptr_t ra, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - intptr_t i, oprsz = simd_oprsz(desc); - ARMVectorReg scratch[2] = { }; - - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, ra); - tlb_fn(env, &scratch[1], i, addr + size, ra); - } - i += size, pg >>= size; - addr += 2 * size; - } while (i & 15); - } - - /* Wait until all exceptions have been raised to write back. */ - memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); - memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); -} - -static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, int size, uintptr_t ra, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - intptr_t i, oprsz = simd_oprsz(desc); - ARMVectorReg scratch[3] = { }; - - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, ra); - tlb_fn(env, &scratch[1], i, addr + size, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, ra); - } - i += size, pg >>= size; - addr += 3 * size; - } while (i & 15); - } - - /* Wait until all exceptions have been raised to write back. */ - memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); - memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); - memcpy(&env->vfp.zregs[(rd + 2) & 31], &scratch[2], oprsz); -} - -static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, int size, uintptr_t ra, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - intptr_t i, oprsz = simd_oprsz(desc); - ARMVectorReg scratch[4] = { }; - - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, ra); - tlb_fn(env, &scratch[1], i, addr + size, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, ra); - tlb_fn(env, &scratch[3], i, addr + 3 * size, ra); - } - i += size, pg >>= size; - addr += 4 * size; - } while (i & 15); - } - - /* Wait until all exceptions have been raised to write back. */ - memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); - memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); - memcpy(&env->vfp.zregs[(rd + 2) & 31], &scratch[2], oprsz); - memcpy(&env->vfp.zregs[(rd + 3) & 31], &scratch[3], oprsz); -} - #define DO_LDN_1(N) \ -void QEMU_FLATTEN HELPER(sve_ld##N##bb_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ -{ \ - sve_ld##N##_r(env, vg, addr, desc, 1, GETPC(), sve_ld1bb_tlb); \ +void HELPER(sve_ld##N##bb_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + sve_ldN_r(env, vg, addr, desc, GETPC(), MO_8, MO_8, N, \ + sve_ld1bb_host, sve_ld1bb_tlb); \ } -#define DO_LDN_2(N, SUFF, SIZE) \ -void QEMU_FLATTEN HELPER(sve_ld##N##SUFF##_le_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +#define DO_LDN_2(N, SUFF, ESZ) \ +void HELPER(sve_ld##N##SUFF##_le_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_ld##N##_r(env, vg, addr, desc, SIZE, GETPC(), \ - sve_ld1##SUFF##_le_tlb); \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, ESZ, N, \ + sve_ld1##SUFF##_le_host, sve_ld1##SUFF##_le_tlb); \ } \ -void QEMU_FLATTEN HELPER(sve_ld##N##SUFF##_be_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +void HELPER(sve_ld##N##SUFF##_be_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_ld##N##_r(env, vg, addr, desc, SIZE, GETPC(), \ - sve_ld1##SUFF##_be_tlb); \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, ESZ, N, \ + sve_ld1##SUFF##_be_host, sve_ld1##SUFF##_be_tlb); \ } DO_LDN_1(2) DO_LDN_1(3) DO_LDN_1(4) -DO_LDN_2(2, hh, 2) -DO_LDN_2(3, hh, 2) -DO_LDN_2(4, hh, 2) +DO_LDN_2(2, hh, MO_16) +DO_LDN_2(3, hh, MO_16) +DO_LDN_2(4, hh, MO_16) -DO_LDN_2(2, ss, 4) -DO_LDN_2(3, ss, 4) -DO_LDN_2(4, ss, 4) +DO_LDN_2(2, ss, MO_32) +DO_LDN_2(3, ss, MO_32) +DO_LDN_2(4, ss, MO_32) -DO_LDN_2(2, dd, 8) -DO_LDN_2(3, dd, 8) -DO_LDN_2(4, dd, 8) +DO_LDN_2(2, dd, MO_64) +DO_LDN_2(3, dd, MO_64) +DO_LDN_2(4, dd, MO_64) #undef DO_LDN_1 #undef DO_LDN_2 From patchwork Wed Mar 11 06:44:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 184399 Delivered-To: patch@linaro.org Received: by 2002:a92:1f12:0:0:0:0:0 with SMTP id i18csp285114ile; Tue, 10 Mar 2020 23:55:47 -0700 (PDT) X-Google-Smtp-Source: ADFU+vsCN8ztDLbWEZdcFIS0tWHSUuJ4h9ur5OTHYkCRgaGYrMOApfWinnMzk9vEVk4ZAPSZA7v6 X-Received: by 2002:a37:b646:: with SMTP id g67mr1429180qkf.52.1583909747798; Tue, 10 Mar 2020 23:55:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1583909747; cv=none; d=google.com; s=arc-20160816; b=BOm/hplpeP8o2o6ikm11Pj4NUprMVzMd1NO9y+h0VJoUVyGgAoKEwaDZ+5bpEcqX5Z sscXY6BLcVPG9BfK52AeDzWGoV7m6PU/aahEJy8QE7H3vwS+yGXyofug8OdAQdtB7iGv 0vi1tWuyS8NRgpvKiZdj0ikoen0pp+XTjzMk6w0H8Cv81uJstJKeMOWian405VUpNeXl 0iPSCPbaSfBLOcAq+unHKpjSBdysmxrez01rY5o/nSlsOEi9WjN1SmePpc7Frtj9bJ6q ZKcU/94GS6S8lkBpnISJkor39m0ad4vo7v1uMcnKulVVNuc+Z8WR6TyXmpx8hJ9e3KYZ 64Iw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=K+5/whU+/G/MKfzhK6hriD90MgA0aJH5jQ1gy3fK8ao=; b=QrZ5HiQ0WvyDw2IptdDwa6PbkQC4TbPhdwxRbFSgodlMPSTvIza3LAbZMTf4w4HS3s Oh9WTG0Rzff59lMTwNU7LkuBJaZveXL9M2obZo/yjjh0IFpk873opG5Anm7QEZ/s5oYs TVETX1bWYpiuHWqeci4O2H+zRtpJm+Ls2XwIgiWWJiuGbqLqet2LgMbUUh+usORfAJhG y24h66WAJHdzOgbQfJzC/U08exMBDfzEbtk27HofWi9+TyQzHYmZbu0ncG1rFLBTXBs0 xR9Wn9RXtvKlwhF43ZR1Dluh4neb2TZmvS4uR4KCyFtm+wYcT+jESDXqFgO6En9rfGN/ nyMw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=o03UG8AU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id d13si525682qtr.356.2020.03.10.23.55.47 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Mar 2020 23:55:47 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=o03UG8AU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:46762 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvHP-0006lz-B2 for patch@linaro.org; Wed, 11 Mar 2020 02:55:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59606) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6f-0006PR-4w for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6c-00020b-H5 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:41 -0400 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]:42708) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6c-0001ze-8l for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:38 -0400 Received: by mail-pl1-x644.google.com with SMTP id t3so613331plz.9 for ; Tue, 10 Mar 2020 23:44:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=K+5/whU+/G/MKfzhK6hriD90MgA0aJH5jQ1gy3fK8ao=; b=o03UG8AUyo4rZxXQeYENCuwLQNV+8PdV6giiGk8VNWEQRkWsumInVv46Ovfqj68vp/ u8suePTXGneG2SPtuEKGeuDWZo6dizN+aKwLFDUw0Lx6OiuKCx4JbWNkFJFyteQLVPEm P+JxEzz+X7z9SRziSAxAZ5M8GSSyxmL7JylZVbNtJdNVMRJ8lPok77NhnD+qaGi4f1hJ 3xT2MH7wKuCu4J0zd5/wXCU07cEo6htpyLI1RTqINVgHLmWKelp9bd6c2eHTA15978b7 1nqSnPi71sAjbmt2JJRcjymFp04gOjtx45I22ciPbweYz+zVXhmyFD28EoAWW4dTfywG kqTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=K+5/whU+/G/MKfzhK6hriD90MgA0aJH5jQ1gy3fK8ao=; b=riCTDmU21138jWh17Qn7s9LcyZD7lp1Xnfev2W0m31KsKIMnqD8X0yuQmIav1XnWnG 1ZwXofFTHZNptTDqi3wQW3J5bjH4GejqIrRPD8fRNn4xvXZfx6V21zGeCaAv9VoFAPZJ MHIChX/9n0c/MxfewQLjO/GN56iChH0polPvGv6MOB9c8XPUfRVijTaMDSwS0HwJ9SyS 1VoPRD4IRXWdXrN80XX/x6TGZ9oul3YV8x0NkEXxZkVdKHYrRNMG4QfVczfF5WMzCPhv nsjFfH0avguf16yC9VeBNmMtHGFQSTcjrFPEFRCb7aSB7s49KWTo07PfrukB2jwfBtn0 zSew== X-Gm-Message-State: ANhLgQ0DrxtCrourco8s+4PTAklbLSXp8cA9gzycxFBrHyqLaxpVJDJ5 kVCVvHh/jpU6aQTK25hekINph4Gc+us= X-Received: by 2002:a17:90b:352:: with SMTP id fh18mr1896827pjb.168.1583909076323; Tue, 10 Mar 2020 23:44:36 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:35 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 11/16] target/arm: Update contiguous first-fault and no-fault loads Date: Tue, 10 Mar 2020 23:44:15 -0700 Message-Id: <20200311064420.30606-12-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::644 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" With sve_cont_ldst_pages, the differences between first-fault and no-fault are minimal, so unify the routines. With cpu_probe_watchpoint, we are able to make progress through pages with TLB_WATCHPOINT set when the watchpoint does not actually fire. Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 340 ++++++++++++++++++---------------------- 1 file changed, 156 insertions(+), 184 deletions(-) -- 2.20.1 diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 56407446eb..9198e1612a 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4126,18 +4126,6 @@ static intptr_t find_next_active(uint64_t *vg, intptr_t reg_off, return reg_off; } -/* - * Return the maximum offset <= @mem_max which is still within the page - * referenced by @base + @mem_off. - */ -static intptr_t max_for_page(target_ulong base, intptr_t mem_off, - intptr_t mem_max) -{ - target_ulong addr = base + mem_off; - intptr_t split = -(intptr_t)(addr | TARGET_PAGE_MASK); - return MIN(split, mem_max - mem_off) + mem_off; -} - /* * Resolve the guest virtual address to info->host and info->flags. * If @nofault, return false if the page is invalid, otherwise @@ -4439,19 +4427,6 @@ static void sve_cont_ldst_watchpoints(SVEContLdSt *info, CPUARMState *env, #endif } -/* - * The result of tlb_vaddr_to_host for user-only is just g2h(x), - * which is always non-null. Elide the useless test. - */ -static inline bool test_host_page(void *host) -{ -#ifdef CONFIG_USER_ONLY - return true; -#else - return likely(host != NULL); -#endif -} - /* * Common helper for all contiguous 1,2,3,4-register predicated stores. */ @@ -4709,167 +4684,161 @@ static void record_fault(CPUARMState *env, uintptr_t i, uintptr_t oprsz) } /* - * Common helper for all contiguous first-fault loads. + * Common helper for all contiguous no-fault and first-fault loads. */ -static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, - uint32_t desc, const uintptr_t retaddr, - const int esz, const int msz, - sve_ldst1_host_fn *host_fn, - sve_ldst1_tlb_fn *tlb_fn) +static inline QEMU_ALWAYS_INLINE +void sve_ldnfff1_r(CPUARMState *env, void *vg, const target_ulong addr, + uint32_t desc, const uintptr_t retaddr, + const int esz, const int msz, const SVEContFault fault, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); - const int mmu_idx = get_mmuidx(oi); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); void *vd = &env->vfp.zregs[rd]; - const int diffsz = esz - msz; const intptr_t reg_max = simd_oprsz(desc); - const intptr_t mem_max = reg_max >> diffsz; - intptr_t split, reg_off, mem_off, i; + intptr_t reg_off, mem_off, reg_last; + SVEContLdSt info; + int flags; void *host; - /* Skip to the first active element. */ - reg_off = find_next_active(vg, 0, reg_max, esz); - if (unlikely(reg_off == reg_max)) { + /* Find the active elements. */ + if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, 1 << msz)) { /* The entire predicate was false; no load occurs. */ memset(vd, 0, reg_max); return; } - mem_off = reg_off >> diffsz; + reg_off = info.reg_off_first[0]; - /* - * If the (remaining) load is entirely within a single page, then: - * For softmmu, and the tlb hits, then no faults will occur; - * For user-only, either the first load will fault or none will. - * We can thus perform the load directly to the destination and - * Vd will be unmodified on any exception path. - */ - split = max_for_page(addr, mem_off, mem_max); - if (likely(split == mem_max)) { - host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); - if (test_host_page(host)) { - i = reg_off; - host -= mem_off; - do { - host_fn(vd, i, host + (i >> diffsz)); - i = find_next_active(vg, i + (1 << esz), reg_max, esz); - } while (i < reg_max); - /* After any fault, zero any leading inactive elements. */ + /* Probe the page(s). */ + if (!sve_cont_ldst_pages(&info, fault, env, addr, MMU_DATA_LOAD, retaddr)) { + /* Fault on first element. */ + tcg_debug_assert(fault == FAULT_NO); + memset(vd, 0, reg_max); + goto do_fault; + } + + mem_off = info.mem_off_first[0]; + flags = info.page[0].flags; + + if (fault == FAULT_FIRST) { + /* + * Special handling of the first active element, + * if it crosses a page boundary or is MMIO. + */ + bool is_split = mem_off == info.mem_off_split; + /* TODO: MTE check. */ + if (unlikely(flags != 0) || unlikely(is_split)) { + /* + * Use the slow path for cross-page handling. + * Might trap for MMIO or watchpoints. + */ + tlb_fn(env, vd, reg_off, addr + mem_off, retaddr); + + /* After any fault, zero the other elements. */ swap_memzero(vd, reg_off); - return; + reg_off += 1 << esz; + mem_off += 1 << msz; + swap_memzero(vd + reg_off, reg_max - reg_off); + + if (is_split) { + goto second_page; + } + } else { + memset(vd, 0, reg_max); + } + } else { + memset(vd, 0, reg_max); + if (unlikely(mem_off == info.mem_off_split)) { + /* The first active element crosses a page boundary. */ + flags |= info.page[1].flags; + if (unlikely(flags & TLB_MMIO)) { + /* Some page is MMIO, see below. */ + goto do_fault; + } + if (unlikely(flags & TLB_WATCHPOINT) && + cpu_probe_watchpoint(env_cpu(env), addr + mem_off, + 1 << msz, BP_MEM_READ)) { + /* Watchpoint hit, see below. */ + goto do_fault; + } + /* TODO: MTE check. */ + /* + * Use the slow path for cross-page handling. + * This is RAM, without a watchpoint, and will not trap. + */ + tlb_fn(env, vd, reg_off, addr + mem_off, retaddr); + goto second_page; } } /* - * Perform one normal read, which will fault or not. - * But it is likely to bring the page into the tlb. + * From this point on, all memory operations are MemSingleNF. + * + * Per the MemSingleNF pseudocode, a no-fault load from Device memory + * must not actually hit the bus -- it returns (UNKNOWN, FAULT) instead. + * If you map non-RAM with Normal memory attributes and do a NF + * load then it should access the bus -- but doing so is illegal. + * + * While we do not have access to the memory attributes from the PTE + * to tell Device memory from Normal memory, we can validly assume that + * non-RAM has been mapped as Device memory. Thus we indicate fault + * on all MMIO. + * + * Similarly, CPU_BP breakpoints would raise exceptions, and so + * return (UNKNOWN, FAULT). For simplicity, we consider gdb and + * architectural breakpoints the same. */ - tlb_fn(env, vd, reg_off, addr + mem_off, retaddr); + if (unlikely(flags & TLB_MMIO)) { + goto do_fault; + } - /* After any fault, zero any leading predicated false elts. */ - swap_memzero(vd, reg_off); - mem_off += 1 << msz; - reg_off += 1 << esz; + reg_last = info.reg_off_last[0]; + host = info.page[0].host; - /* Try again to read the balance of the page. */ - split = max_for_page(addr, mem_off - 1, mem_max); - if (split >= (1 << msz)) { - host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); - if (host) { - host -= mem_off; - do { + do { + uint64_t pg = *(uint64_t *)(vg + (reg_off >> 3)); + do { + if ((pg >> (reg_off & 63)) & 1) { + if (unlikely(flags & TLB_WATCHPOINT) && + cpu_probe_watchpoint(env_cpu(env), addr + mem_off, + 1 << msz, BP_MEM_READ)) { + goto do_fault; + } + /* TODO: MTE check. */ host_fn(vd, reg_off, host + mem_off); - reg_off += 1 << esz; - reg_off = find_next_active(vg, reg_off, reg_max, esz); - mem_off = reg_off >> diffsz; - } while (split - mem_off >= (1 << msz)); - } - } - - record_fault(env, reg_off, reg_max); -} - -/* - * Common helper for all contiguous no-fault loads. - */ -static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong addr, - uint32_t desc, const int esz, const int msz, - sve_ldst1_host_fn *host_fn) -{ - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - void *vd = &env->vfp.zregs[rd]; - const int diffsz = esz - msz; - const intptr_t reg_max = simd_oprsz(desc); - const intptr_t mem_max = reg_max >> diffsz; - const int mmu_idx = cpu_mmu_index(env, false); - intptr_t split, reg_off, mem_off; - void *host; - -#ifdef CONFIG_USER_ONLY - host = tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx); - if (likely(page_check_range(addr, mem_max, PAGE_READ) == 0)) { - /* The entire operation is valid and will not fault. */ - reg_off = 0; - do { - mem_off = reg_off >> diffsz; - host_fn(vd, reg_off, host + mem_off); + } reg_off += 1 << esz; - reg_off = find_next_active(vg, reg_off, reg_max, esz); - } while (reg_off < reg_max); - return; - } -#endif + mem_off += 1 << msz; + } while (reg_off <= reg_last && (reg_off & 63)); + } while (reg_off <= reg_last); - /* There will be no fault, so we may modify in advance. */ - memset(vd, 0, reg_max); - - /* Skip to the first active element. */ - reg_off = find_next_active(vg, 0, reg_max, esz); - if (unlikely(reg_off == reg_max)) { - /* The entire predicate was false; no load occurs. */ - return; - } - mem_off = reg_off >> diffsz; - -#ifdef CONFIG_USER_ONLY - if (page_check_range(addr + mem_off, 1 << msz, PAGE_READ) == 0) { - /* At least one load is valid; take the rest of the page. */ - split = max_for_page(addr, mem_off + (1 << msz) - 1, mem_max); - do { - host_fn(vd, reg_off, host + mem_off); - reg_off += 1 << esz; - reg_off = find_next_active(vg, reg_off, reg_max, esz); - mem_off = reg_off >> diffsz; - } while (split - mem_off >= (1 << msz)); - } -#else /* - * If the address is not in the TLB, we have no way to bring the - * entry into the TLB without also risking a fault. Note that - * the corollary is that we never load from an address not in RAM. - * - * This last is out of spec, in a weird corner case. - * Per the MemNF/MemSingleNF pseudocode, a NF load from Device memory - * must not actually hit the bus -- it returns UNKNOWN data instead. - * But if you map non-RAM with Normal memory attributes and do a NF - * load then it should access the bus. (Nobody ought actually do this - * in the real world, obviously.) - * - * Then there are the annoying special cases with watchpoints... - * TODO: Add a form of non-faulting loads using cc->tlb_fill(probe=true). + * MemSingleNF is allowed to fail for any reason. We have special + * code above to handle the first element crossing a page boundary. + * As an implementation choice, decline to handle a cross-page element + * in any other position. */ - host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); - split = max_for_page(addr, mem_off, mem_max); - if (host && split >= (1 << msz)) { - host -= mem_off; - do { - host_fn(vd, reg_off, host + mem_off); - reg_off += 1 << esz; - reg_off = find_next_active(vg, reg_off, reg_max, esz); - mem_off = reg_off >> diffsz; - } while (split - mem_off >= (1 << msz)); + reg_off = info.reg_off_split; + if (reg_off >= 0) { + goto do_fault; } -#endif + second_page: + reg_off = info.reg_off_first[1]; + if (likely(reg_off < 0)) { + /* No active elements on the second page. All done. */ + return; + } + + /* + * MemSingleNF is allowed to fail for any reason. As an implementation + * choice, decline to handle elements on the second page. This should + * be low frequency as the guest walks through memory -- the next + * iteration of the guest's loop should be aligned on the page boundary, + * and then all following iterations will stay aligned. + */ + + do_fault: record_fault(env, reg_off, reg_max); } @@ -4877,58 +4846,61 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong addr, void HELPER(sve_ldff1##PART##_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, 0, \ - sve_ld1##PART##_host, sve_ld1##PART##_tlb); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MO_8, FAULT_FIRST, \ + sve_ld1##PART##_host, sve_ld1##PART##_tlb); \ } \ void HELPER(sve_ldnf1##PART##_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldnf1_r(env, vg, addr, desc, ESZ, 0, sve_ld1##PART##_host); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MO_8, FAULT_NO, \ + sve_ld1##PART##_host, sve_ld1##PART##_tlb); \ } #define DO_LDFF1_LDNF1_2(PART, ESZ, MSZ) \ void HELPER(sve_ldff1##PART##_le_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ - sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, FAULT_FIRST, \ + sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ } \ void HELPER(sve_ldnf1##PART##_le_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, sve_ld1##PART##_le_host); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, FAULT_NO, \ + sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ } \ void HELPER(sve_ldff1##PART##_be_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ - sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, FAULT_FIRST, \ + sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ } \ void HELPER(sve_ldnf1##PART##_be_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, sve_ld1##PART##_be_host); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, FAULT_NO, \ + sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ } -DO_LDFF1_LDNF1_1(bb, 0) -DO_LDFF1_LDNF1_1(bhu, 1) -DO_LDFF1_LDNF1_1(bhs, 1) -DO_LDFF1_LDNF1_1(bsu, 2) -DO_LDFF1_LDNF1_1(bss, 2) -DO_LDFF1_LDNF1_1(bdu, 3) -DO_LDFF1_LDNF1_1(bds, 3) +DO_LDFF1_LDNF1_1(bb, MO_8) +DO_LDFF1_LDNF1_1(bhu, MO_16) +DO_LDFF1_LDNF1_1(bhs, MO_16) +DO_LDFF1_LDNF1_1(bsu, MO_32) +DO_LDFF1_LDNF1_1(bss, MO_32) +DO_LDFF1_LDNF1_1(bdu, MO_64) +DO_LDFF1_LDNF1_1(bds, MO_64) -DO_LDFF1_LDNF1_2(hh, 1, 1) -DO_LDFF1_LDNF1_2(hsu, 2, 1) -DO_LDFF1_LDNF1_2(hss, 2, 1) -DO_LDFF1_LDNF1_2(hdu, 3, 1) -DO_LDFF1_LDNF1_2(hds, 3, 1) +DO_LDFF1_LDNF1_2(hh, MO_16, MO_16) +DO_LDFF1_LDNF1_2(hsu, MO_32, MO_16) +DO_LDFF1_LDNF1_2(hss, MO_32, MO_16) +DO_LDFF1_LDNF1_2(hdu, MO_64, MO_16) +DO_LDFF1_LDNF1_2(hds, MO_64, MO_16) -DO_LDFF1_LDNF1_2(ss, 2, 2) -DO_LDFF1_LDNF1_2(sdu, 3, 2) -DO_LDFF1_LDNF1_2(sds, 3, 2) +DO_LDFF1_LDNF1_2(ss, MO_32, MO_32) +DO_LDFF1_LDNF1_2(sdu, MO_64, MO_32) +DO_LDFF1_LDNF1_2(sds, MO_64, MO_32) -DO_LDFF1_LDNF1_2(dd, 3, 3) +DO_LDFF1_LDNF1_2(dd, MO_64, MO_64) #undef DO_LDFF1_LDNF1_1 #undef DO_LDFF1_LDNF1_2 From patchwork Wed Mar 11 06:44:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 184386 Delivered-To: patch@linaro.org Received: by 2002:a92:1f12:0:0:0:0:0 with SMTP id i18csp277588ile; Tue, 10 Mar 2020 23:45:19 -0700 (PDT) X-Google-Smtp-Source: ADFU+vvSBttnWgeNuGyV1vxNL5iGyqvI/K4Vsxak5O0+5VCzaxsaolcO4oO/cv/WNdL3591ZLLtn X-Received: by 2002:ac8:7755:: with SMTP id g21mr1350324qtu.372.1583909119463; Tue, 10 Mar 2020 23:45:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1583909119; cv=none; d=google.com; s=arc-20160816; b=z+r0SVABTyoTgY1KiChB/ywsR//QoE5hTdpr1EVFIR66B2rPg5GTtcRWro83PCyzD3 huy2Cw2CDeHoW68Glc0YteQeFyNhY81Wy3z4UPmLj3BmvY/29Ce9z0jQWsOnvKse8l1J xc3HH/yEFxUdSCBQ6iMCBH9IGqM57GyfhrG63TexRQSy78Uc5LYs+SA3KKKlqlU8rjGL 8HrBDOXqDv9UqJ2+7/sWIqjE8ZbLZNDuy6ygKajo3VChOmdl6obCZQAphmZEjIHYjHri 4u6UO/qgQo+vPUL9so+4i5KraXz0S23er9cXX+QeQoit2cnkbRx1rAheGKYem8OXgd60 8ntQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=KhxITN+NjmEXUXhdWp+bRLvbeT9VqflVpRPfCgP2sDA=; b=uEUDa2FWU2KYnTbPZTng9cwtAhweRlMLmac1LjFPUBqVdJ4DmVeUzG+ADojCzjgIGH AL9pyJDfIItr4ztx17om7A7vd6f2Dqe2vEEcR3iVF4xS9kHD801khjHeWyEf15n+kKnu oKvhu2WBTZ4E9KuqP2jWbwnJaOYcMSE6x9D4DYS1to9rpYRU9t2HaSVYjjkoE3dkgOHp Tdetdy9/JRl9AQGDdS39tKnzLgm89cTluryEeYUDNYonKCMUZ/gqaQyvYWP7lv8EZ1Yo vX3iEhnIya62qNHQgCh5Qi4BeprsMMcdCrK5hVnquOWKtRvSrV2pAZGyek6nXaq5XYm0 8Kqw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=K01SY383; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id r15si470125qtm.402.2020.03.10.23.45.19 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Mar 2020 23:45:19 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=K01SY383; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:46370 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv7G-0007Hi-R5 for patch@linaro.org; Wed, 11 Mar 2020 02:45:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59617) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6f-0006R3-P1 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6d-00021V-Fl for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:41 -0400 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]:33484) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6d-00020n-7q for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:39 -0400 Received: by mail-pl1-x644.google.com with SMTP id ay11so632423plb.0 for ; Tue, 10 Mar 2020 23:44:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KhxITN+NjmEXUXhdWp+bRLvbeT9VqflVpRPfCgP2sDA=; b=K01SY383f8aqADDrj0upIPIKGpQoU0zUlEzSfCwYctxykf5wgQFE9IQxNCfJvIUNHm OBByQVOUBuJtB+6VJ+HI14y2nNJPNWb6WNZVf9FFxwC7DRl8E3+3nNSi6t7kZlQ9zdcp PLGc1eEQVk6pT09At+jFxQbd3jBxo556CiGTzFKNiw2Zsm7n/uUf4gj7qfkNNfaTyLyC F4sT19Y0R7pevYh/JPYXQpJrn8NWN4ujHcBeqW/Lu83K7lIE/Lm25lUSLevv6IX+dMEq 7DIwcwj4DX0zeer+KgRew1kBfUu/HMFdZ/G0LqezpZAi6oywzgFLFCz6Ba+tZLnpTxIG MDFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KhxITN+NjmEXUXhdWp+bRLvbeT9VqflVpRPfCgP2sDA=; b=S//qHPpIUpZHU1mrljVodb+jGF6+NkauhGKf8Ag0JeFTPUleyFdylEr4dgrI0+EwmR YxwrhgDwSVDRvKLCGfu342NMH9TlEzLV1DtT8hwbbyl++eZ+agUTPjM1HI9M9Z6hR4Cu +6SM7Kp6z3m4Kp1pIRMl4D3O2WUY0spfKufmfoNRGXQVyqhe1JIkDtF9IoWXk9tw2hGw L7wNgkQcQvpGdZmbc4rze+dinQiZjoOuxjp5t7Sbbbcp+psYFLujfvbIWpcy6mRzKIrb Du2tjlzqGZkerVm9tDeR/wnKNEsSwsiV/e11mJb10CWAnn33OPv3rvVOvXSDjoJL1bX8 jCVw== X-Gm-Message-State: ANhLgQ2xHMaEaXX71GMfDi4vwMRkSuJN2DyvdO+PdoCjbBe3oAkM6Z0b x4y+ze9nTSxgUmCfbzmjn7Oz35hL+NU= X-Received: by 2002:a17:90a:7309:: with SMTP id m9mr1850242pjk.52.1583909077579; Tue, 10 Mar 2020 23:44:37 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:36 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 12/16] target/arm: Use SVEContLdSt for contiguous stores Date: Tue, 10 Mar 2020 23:44:16 -0700 Message-Id: <20200311064420.30606-13-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::644 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Follow the model set up for contiguous loads. This handles watchpoints correctly for contiguous stores, recognizing the exception before any changes to memory. Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 288 ++++++++++++++++++++++------------------ 1 file changed, 162 insertions(+), 126 deletions(-) -- 2.20.1 Reviewed-by: Peter Maydell diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 9198e1612a..c6f22928d5 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4002,6 +4002,13 @@ static void sve_##NAME##_host(void *vd, intptr_t reg_off, void *host) \ *(TYPEE *)(vd + H(reg_off)) = val; \ } +#define DO_ST_HOST(NAME, H, TYPEE, TYPEM, HOST) \ +static void sve_##NAME##_host(void *vd, intptr_t reg_off, void *host) \ +{ \ + TYPEM val = *(TYPEE *)(vd + H(reg_off)); \ + HOST(host, val); \ +} + #define DO_LD_TLB(NAME, H, TYPEE, TYPEM, BSWAP, TLB) \ static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ target_ulong addr, uintptr_t ra) \ @@ -4031,6 +4038,7 @@ DO_LD_PRIM_1(ld1bdu, , uint64_t, uint8_t) DO_LD_PRIM_1(ld1bds, , uint64_t, int8_t) #define DO_ST_PRIM_1(NAME, H, TE, TM) \ + DO_ST_HOST(st1##NAME, H, TE, TM, stb_p) \ DO_ST_TLB(st1##NAME, H, TE, TM, , cpu_stb_data_ra) DO_ST_PRIM_1(bb, H1, uint8_t, uint8_t) @@ -4045,6 +4053,8 @@ DO_ST_PRIM_1(bd, , uint64_t, uint8_t) DO_LD_TLB(ld1##NAME##_le, H, TE, TM, le_##BSWAP, cpu_##LD##_data_ra) #define DO_ST_PRIM_2(NAME, H, TE, TM, BSWAP, ST) \ + DO_ST_HOST(st1##NAME##_be, H, TE, TM, ST##_be_p) \ + DO_ST_HOST(st1##NAME##_le, H, TE, TM, ST##_le_p) \ DO_ST_TLB(st1##NAME##_be, H, TE, TM, be_##BSWAP, cpu_##ST##_data_ra) \ DO_ST_TLB(st1##NAME##_le, H, TE, TM, le_##BSWAP, cpu_##ST##_data_ra) @@ -4906,151 +4916,177 @@ DO_LDFF1_LDNF1_2(dd, MO_64, MO_64) #undef DO_LDFF1_LDNF1_2 /* - * Common helpers for all contiguous 1,2,3,4-register predicated stores. + * Common helper for all contiguous 1,2,3,4-register predicated stores. */ -static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, const uintptr_t ra, - const int esize, const int msize, - sve_ldst1_tlb_fn *tlb_fn) + +static inline QEMU_ALWAYS_INLINE +void sve_stN_r(CPUARMState *env, uint64_t *vg, target_ulong addr, uint32_t desc, + const uintptr_t retaddr, const int esz, + const int msz, const int N, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - intptr_t i, oprsz = simd_oprsz(desc); - void *vd = &env->vfp.zregs[rd]; + const intptr_t reg_max = simd_oprsz(desc); + intptr_t reg_off, reg_last, mem_off; + SVEContLdSt info; + void *host; + int i, flags; - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, vd, i, addr, ra); + /* Find the active elements. */ + if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, N << msz)) { + /* The entire predicate was false; no store occurs. */ + return; + } + + /* Probe the page(s). Exit with exception for any invalid page. */ + sve_cont_ldst_pages(&info, FAULT_ALL, env, addr, MMU_DATA_STORE, retaddr); + + /* Handle watchpoints for all active elements. */ + sve_cont_ldst_watchpoints(&info, env, vg, addr, 1 << esz, N << msz, + BP_MEM_WRITE, retaddr); + + /* TODO: MTE check. */ + + flags = info.page[0].flags | info.page[1].flags; + if (unlikely(flags != 0)) { +#ifdef CONFIG_USER_ONLY + g_assert_not_reached(); +#else + /* + * At least one page includes MMIO. + * Any bus operation can fail with cpu_transaction_failed, + * which for ARM will raise SyncExternal. We cannot avoid + * this fault and will leave with the store incomplete. + */ + mem_off = info.mem_off_first[0]; + reg_off = info.reg_off_first[0]; + reg_last = info.reg_off_last[1]; + if (reg_last < 0) { + reg_last = info.reg_off_split; + if (reg_last < 0) { + reg_last = info.reg_off_last[0]; } - i += esize, pg >>= esize; - addr += msize; - } while (i & 15); + } + + do { + uint64_t pg = vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + for (i = 0; i < N; ++i) { + tlb_fn(env, &env->vfp.zregs[(rd + i) & 31], reg_off, + addr + mem_off + (i << msz), retaddr); + } + } + reg_off += 1 << esz; + mem_off += N << msz; + } while (reg_off & 63); + } while (reg_off <= reg_last); + return; +#endif + } + + mem_off = info.mem_off_first[0]; + reg_off = info.reg_off_first[0]; + reg_last = info.reg_off_last[0]; + host = info.page[0].host; + + while (reg_off <= reg_last) { + uint64_t pg = vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + for (i = 0; i < N; ++i) { + host_fn(&env->vfp.zregs[(rd + i) & 31], reg_off, + host + mem_off + (i << msz)); + } + } + reg_off += 1 << esz; + mem_off += N << msz; + } while (reg_off <= reg_last && (reg_off & 63)); + } + + /* + * Use the slow path to manage the cross-page misalignment. + * But we know this is RAM and cannot trap. + */ + mem_off = info.mem_off_split; + if (unlikely(mem_off >= 0)) { + reg_off = info.reg_off_split; + for (i = 0; i < N; ++i) { + tlb_fn(env, &env->vfp.zregs[(rd + i) & 31], reg_off, + addr + mem_off + (i << msz), retaddr); + } + } + + mem_off = info.mem_off_first[1]; + if (unlikely(mem_off >= 0)) { + reg_off = info.reg_off_first[1]; + reg_last = info.reg_off_last[1]; + host = info.page[1].host; + + do { + uint64_t pg = vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + for (i = 0; i < N; ++i) { + host_fn(&env->vfp.zregs[(rd + i) & 31], reg_off, + host + mem_off + (i << msz)); + } + } + reg_off += 1 << esz; + mem_off += N << msz; + } while (reg_off & 63); + } while (reg_off <= reg_last); } } -static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, const uintptr_t ra, - const int esize, const int msize, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - intptr_t i, oprsz = simd_oprsz(desc); - void *d1 = &env->vfp.zregs[rd]; - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; - - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, d1, i, addr, ra); - tlb_fn(env, d2, i, addr + msize, ra); - } - i += esize, pg >>= esize; - addr += 2 * msize; - } while (i & 15); - } -} - -static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, const uintptr_t ra, - const int esize, const int msize, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - intptr_t i, oprsz = simd_oprsz(desc); - void *d1 = &env->vfp.zregs[rd]; - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; - void *d3 = &env->vfp.zregs[(rd + 2) & 31]; - - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, d1, i, addr, ra); - tlb_fn(env, d2, i, addr + msize, ra); - tlb_fn(env, d3, i, addr + 2 * msize, ra); - } - i += esize, pg >>= esize; - addr += 3 * msize; - } while (i & 15); - } -} - -static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, const uintptr_t ra, - const int esize, const int msize, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - intptr_t i, oprsz = simd_oprsz(desc); - void *d1 = &env->vfp.zregs[rd]; - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; - void *d3 = &env->vfp.zregs[(rd + 2) & 31]; - void *d4 = &env->vfp.zregs[(rd + 3) & 31]; - - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, d1, i, addr, ra); - tlb_fn(env, d2, i, addr + msize, ra); - tlb_fn(env, d3, i, addr + 2 * msize, ra); - tlb_fn(env, d4, i, addr + 3 * msize, ra); - } - i += esize, pg >>= esize; - addr += 4 * msize; - } while (i & 15); - } -} - -#define DO_STN_1(N, NAME, ESIZE) \ -void QEMU_FLATTEN HELPER(sve_st##N##NAME##_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +#define DO_STN_1(N, NAME, ESZ) \ +void HELPER(sve_st##N##NAME##_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, 1, \ - sve_st1##NAME##_tlb); \ + sve_stN_r(env, vg, addr, desc, GETPC(), ESZ, MO_8, N, \ + sve_st1##NAME##_host, sve_st1##NAME##_tlb); \ } -#define DO_STN_2(N, NAME, ESIZE, MSIZE) \ -void QEMU_FLATTEN HELPER(sve_st##N##NAME##_le_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +#define DO_STN_2(N, NAME, ESZ, MSZ) \ +void HELPER(sve_st##N##NAME##_le_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, MSIZE, \ - sve_st1##NAME##_le_tlb); \ + sve_stN_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, N, \ + sve_st1##NAME##_le_host, sve_st1##NAME##_le_tlb); \ } \ -void QEMU_FLATTEN HELPER(sve_st##N##NAME##_be_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +void HELPER(sve_st##N##NAME##_be_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, MSIZE, \ - sve_st1##NAME##_be_tlb); \ + sve_stN_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, N, \ + sve_st1##NAME##_be_host, sve_st1##NAME##_be_tlb); \ } -DO_STN_1(1, bb, 1) -DO_STN_1(1, bh, 2) -DO_STN_1(1, bs, 4) -DO_STN_1(1, bd, 8) -DO_STN_1(2, bb, 1) -DO_STN_1(3, bb, 1) -DO_STN_1(4, bb, 1) +DO_STN_1(1, bb, MO_8) +DO_STN_1(1, bh, MO_16) +DO_STN_1(1, bs, MO_32) +DO_STN_1(1, bd, MO_64) +DO_STN_1(2, bb, MO_8) +DO_STN_1(3, bb, MO_8) +DO_STN_1(4, bb, MO_8) -DO_STN_2(1, hh, 2, 2) -DO_STN_2(1, hs, 4, 2) -DO_STN_2(1, hd, 8, 2) -DO_STN_2(2, hh, 2, 2) -DO_STN_2(3, hh, 2, 2) -DO_STN_2(4, hh, 2, 2) +DO_STN_2(1, hh, MO_16, MO_16) +DO_STN_2(1, hs, MO_32, MO_16) +DO_STN_2(1, hd, MO_64, MO_16) +DO_STN_2(2, hh, MO_16, MO_16) +DO_STN_2(3, hh, MO_16, MO_16) +DO_STN_2(4, hh, MO_16, MO_16) -DO_STN_2(1, ss, 4, 4) -DO_STN_2(1, sd, 8, 4) -DO_STN_2(2, ss, 4, 4) -DO_STN_2(3, ss, 4, 4) -DO_STN_2(4, ss, 4, 4) +DO_STN_2(1, ss, MO_32, MO_32) +DO_STN_2(1, sd, MO_64, MO_32) +DO_STN_2(2, ss, MO_32, MO_32) +DO_STN_2(3, ss, MO_32, MO_32) +DO_STN_2(4, ss, MO_32, MO_32) -DO_STN_2(1, dd, 8, 8) -DO_STN_2(2, dd, 8, 8) -DO_STN_2(3, dd, 8, 8) -DO_STN_2(4, dd, 8, 8) +DO_STN_2(1, dd, MO_64, MO_64) +DO_STN_2(2, dd, MO_64, MO_64) +DO_STN_2(3, dd, MO_64, MO_64) +DO_STN_2(4, dd, MO_64, MO_64) #undef DO_STN_1 #undef DO_STN_2 From patchwork Wed Mar 11 06:44:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 184395 Delivered-To: patch@linaro.org Received: by 2002:a92:1f12:0:0:0:0:0 with SMTP id i18csp283621ile; Tue, 10 Mar 2020 23:53:36 -0700 (PDT) X-Google-Smtp-Source: ADFU+vvVfl4lnnrzfOoEKEIHO/32WmOWXmM0scpVuLog3hFuBl7C1s9VkSskWgrqV0AN87DR9XPt X-Received: by 2002:ac8:70c:: with SMTP id g12mr1330540qth.47.1583909616307; Tue, 10 Mar 2020 23:53:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1583909616; cv=none; d=google.com; s=arc-20160816; b=fDgC+NxCB7Yr+8z1NZ11oAuJIOAZgfsQjdSiA+8jLd1xc4fXCwg6DkQbUdUj8XrOJe 4SWT/bRD+depbFinD6tkqwtcLMPg12Tsz6duDGknUrJJrSJZf+x5ix921P3q6o9bQ8Jc HYAgoGakXEclPSo70CsbSm0KguR1TEgfotnvvPOPHKpMpd3mYcMFfaEAmMPYLXxmocaL SUMzn9LZYjtbyYFz1iNDZVLE2BhLDig4BYzocsy6chDmL3SfdSQx7jZEltMzV0jhqbug EcFJSu7znXhSTmY8rutR3iyQC5CbYk/o8LBJRMuiZF/yChq0YJFXXlQmV00SJ1t4EITw rtjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=x3t5PDoyjO7TC0DW2CC4BuCDJWVwgzq1cMlGkXCUhYU=; b=BVS6PfBNm9kIcCTIpTJCldA5EAObs+i6o0Zj5HpfhtOPvlAgnbQ8wZULj4t9VQfT9F hkfNHokc6YSsLBMAZu6J8qtfCHhGD7uN7owLagQOd3yg24+MG+h2PiIH8d+ew8yP/RqK wh09J4ksYoTKNx0v/4DX2cbIR7pQ1z10//Kr6WebXLkJuqevu4eMD7uyEFWpyNGnrwHl qSlrLELRiX5Imt3g4dy4fpBDCFExpHuR5YtCxEzg+meO6qUJ0LtGQdu17N1sOqvH72zU ORIHfPNEB/T4LqJ2sjuASxNySL2iy5qN9kovdXZzoQ5KkJ3VyvQ3VgbuYWJlome+Sucv ejPg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=KPCy48Bo; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id d204si52885qke.80.2020.03.10.23.53.36 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Mar 2020 23:53:36 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=KPCy48Bo; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:46656 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvFH-0003Qr-RK for patch@linaro.org; Wed, 11 Mar 2020 02:53:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59648) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6h-0006SS-47 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:45 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6e-00022a-VV for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:43 -0400 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]:46643) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6e-00022B-NE for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:40 -0400 Received: by mail-pl1-x644.google.com with SMTP id w12so604769pll.13 for ; Tue, 10 Mar 2020 23:44:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=x3t5PDoyjO7TC0DW2CC4BuCDJWVwgzq1cMlGkXCUhYU=; b=KPCy48BoRzZwRXMbdj1MeiGeFlCbpdtdfvqD8a6aYnpYkxsEFudnzhwI1zWvZqOGhl lG2fZwjmhLVCnB2tuuZoP1iFc8FHWmE5mnfpW2jL/pj3dwv4xoiAo1ZGbH7OTZCfpfbO LFS4r+MrmmSdOQUM6fDsWxhi7F3RR15G75YBsSA3wvbgnsMeVPpW2OcG4DyxhqZPPF4L iwglr61/imuf1IzXuAif8S2dbkKyeyh4LiLzJhm9GEzAc9yuaerZvAiBmCd1RmxxYyfU Doa9p59TmJZJR8pLW/QWDkcDeySVCD5lfJel8ehw/V+Bn/0x3pNT4YgY3TBUVyVZCnZm cNEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=x3t5PDoyjO7TC0DW2CC4BuCDJWVwgzq1cMlGkXCUhYU=; b=bw+zIG9tPJDNvddyr5zYMAFQoAcGvtf6YOfye5xrtH/6n7c8q8QSg2dakUkVIVxKT5 UYr79Rn5+uF4V3v1XBJ8g95N2Y5JfkWilWGWtINvk2Y8xROzRW8bJy/567HxrXLjnyQm kyad9vjTUaYln79oTM3zq94IZOzalDcloycWUDDQk4SG4H+Wsaa9X+N20zpDxfvSsSfz VDRQxSbKjUjEwxfPvY0+6xT59kE5Fit+gT0MgOmWZfhpEiquRgNFXczBCGaeHdvICAI7 Bp7mmOLsHYtW6BIEztMI506gVzV1K1K16OUSGdDAhpFe6RhLMq1lso+D3roPqJwDa9HH 6DOA== X-Gm-Message-State: ANhLgQ3Eeag16M2FVXI5pzTif0pZgXKp5fB16kbilR6QAUPvguzKWp9o o54VCzXKf3QIP0gEP8U2G9q3skFmcaU= X-Received: by 2002:a17:902:7c0a:: with SMTP id x10mr1641782pll.229.1583909079053; Tue, 10 Mar 2020 23:44:39 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:38 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 13/16] target/arm: Reuse sve_probe_page for gather first-fault loads Date: Tue, 10 Mar 2020 23:44:17 -0700 Message-Id: <20200311064420.30606-14-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::644 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" This avoids the need for a separate set of helpers to implement no-fault semantics, and will enable MTE in the future. Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 323 ++++++++++++++++------------------------ 1 file changed, 127 insertions(+), 196 deletions(-) -- 2.20.1 Reviewed-by: Peter Maydell diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index c6f22928d5..a33699adf8 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -5255,231 +5255,162 @@ DO_LD1_ZPZ_D(dd_be, zd) /* First fault loads with a vector index. */ -/* Load one element into VD+REG_OFF from (ENV,VADDR) without faulting. - * The controlling predicate is known to be true. Return true if the - * load was successful. - */ -typedef bool sve_ld1_nf_fn(CPUARMState *env, void *vd, intptr_t reg_off, - target_ulong vaddr, int mmu_idx); - -#ifdef CONFIG_SOFTMMU -#define DO_LD_NF(NAME, H, TYPEE, TYPEM, HOST) \ -static bool sve_ld##NAME##_nf(CPUARMState *env, void *vd, intptr_t reg_off, \ - target_ulong addr, int mmu_idx) \ -{ \ - target_ulong next_page = -(addr | TARGET_PAGE_MASK); \ - if (likely(next_page - addr >= sizeof(TYPEM))) { \ - void *host = tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx); \ - if (likely(host)) { \ - TYPEM val = HOST(host); \ - *(TYPEE *)(vd + H(reg_off)) = val; \ - return true; \ - } \ - } \ - return false; \ -} -#else -#define DO_LD_NF(NAME, H, TYPEE, TYPEM, HOST) \ -static bool sve_ld##NAME##_nf(CPUARMState *env, void *vd, intptr_t reg_off, \ - target_ulong addr, int mmu_idx) \ -{ \ - if (likely(page_check_range(addr, sizeof(TYPEM), PAGE_READ))) { \ - TYPEM val = HOST(g2h(addr)); \ - *(TYPEE *)(vd + H(reg_off)) = val; \ - return true; \ - } \ - return false; \ -} -#endif - -DO_LD_NF(bsu, H1_4, uint32_t, uint8_t, ldub_p) -DO_LD_NF(bss, H1_4, uint32_t, int8_t, ldsb_p) -DO_LD_NF(bdu, , uint64_t, uint8_t, ldub_p) -DO_LD_NF(bds, , uint64_t, int8_t, ldsb_p) - -DO_LD_NF(hsu_le, H1_4, uint32_t, uint16_t, lduw_le_p) -DO_LD_NF(hss_le, H1_4, uint32_t, int16_t, ldsw_le_p) -DO_LD_NF(hsu_be, H1_4, uint32_t, uint16_t, lduw_be_p) -DO_LD_NF(hss_be, H1_4, uint32_t, int16_t, ldsw_be_p) -DO_LD_NF(hdu_le, , uint64_t, uint16_t, lduw_le_p) -DO_LD_NF(hds_le, , uint64_t, int16_t, ldsw_le_p) -DO_LD_NF(hdu_be, , uint64_t, uint16_t, lduw_be_p) -DO_LD_NF(hds_be, , uint64_t, int16_t, ldsw_be_p) - -DO_LD_NF(ss_le, H1_4, uint32_t, uint32_t, ldl_le_p) -DO_LD_NF(ss_be, H1_4, uint32_t, uint32_t, ldl_be_p) -DO_LD_NF(sdu_le, , uint64_t, uint32_t, ldl_le_p) -DO_LD_NF(sds_le, , uint64_t, int32_t, ldl_le_p) -DO_LD_NF(sdu_be, , uint64_t, uint32_t, ldl_be_p) -DO_LD_NF(sds_be, , uint64_t, int32_t, ldl_be_p) - -DO_LD_NF(dd_le, , uint64_t, uint64_t, ldq_le_p) -DO_LD_NF(dd_be, , uint64_t, uint64_t, ldq_be_p) - /* - * Common helper for all gather first-faulting loads. + * Common helpers for all gather first-faulting loads. */ -static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn, - sve_ld1_nf_fn *nonfault_fn) + +static inline QEMU_ALWAYS_INLINE +void sve_ldff1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t retaddr, + const int esz, const int msz, zreg_off_fn *off_fn, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); - const int mmu_idx = get_mmuidx(oi); + const int mmu_idx = cpu_mmu_index(env, false); const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); - intptr_t reg_off, reg_max = simd_oprsz(desc); - target_ulong addr; + const int esize = 1 << esz; + const int msize = 1 << msz; + const intptr_t reg_max = simd_oprsz(desc); + intptr_t reg_off; + SVEHostPage info; + target_ulong addr, in_page; /* Skip to the first true predicate. */ - reg_off = find_next_active(vg, 0, reg_max, MO_32); - if (likely(reg_off < reg_max)) { - /* Perform one normal read, which will fault or not. */ - addr = off_fn(vm, reg_off); - addr = base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, ra); - - /* The rest of the reads will be non-faulting. */ + reg_off = find_next_active(vg, 0, reg_max, esz); + if (unlikely(reg_off >= reg_max)) { + /* The entire predicate was false; no load occurs. */ + memset(vd, 0, reg_max); + return; } - /* After any fault, zero the leading predicated false elements. */ + /* + * Probe the first element, allowing faults. + */ + addr = base + (off_fn(vm, reg_off) << scale); + tlb_fn(env, vd, reg_off, addr, retaddr); + + /* After any fault, zero the other elements. */ swap_memzero(vd, reg_off); + reg_off += esize; + swap_memzero(vd + reg_off, reg_max - reg_off); - while (likely((reg_off += 4) < reg_max)) { - uint64_t pg = *(uint64_t *)(vg + (reg_off >> 6) * 8); - if (likely((pg >> (reg_off & 63)) & 1)) { - addr = off_fn(vm, reg_off); - addr = base + (addr << scale); - if (!nonfault_fn(env, vd, reg_off, addr, mmu_idx)) { - record_fault(env, reg_off, reg_max); - break; + /* + * Probe the remaining elements, not allowing faults. + */ + while (reg_off < reg_max) { + uint64_t pg = vg[reg_off >> 6]; + do { + if (likely((pg >> (reg_off & 63)) & 1)) { + addr = base + (off_fn(vm, reg_off) << scale); + in_page = -(addr | TARGET_PAGE_MASK); + + if (unlikely(in_page < msize)) { + /* Stop if the element crosses a page boundary. */ + goto fault; + } + + sve_probe_page(&info, true, env, addr, 0, MMU_DATA_LOAD, + mmu_idx, retaddr); + if (unlikely(info.flags & (TLB_INVALID_MASK | TLB_MMIO))) { + goto fault; + } + if (unlikely(info.flags & TLB_WATCHPOINT) && + cpu_probe_watchpoint(env_cpu(env), addr, msize, + BP_MEM_READ)) { + goto fault; + } + /* TODO: MTE check. */ + + host_fn(vd, reg_off, info.host); } - } else { - *(uint32_t *)(vd + H1_4(reg_off)) = 0; - } + reg_off += esize; + } while (reg_off & 63); } + return; + + fault: + record_fault(env, reg_off, reg_max); } -static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn, - sve_ld1_nf_fn *nonfault_fn) -{ - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); - const int mmu_idx = get_mmuidx(oi); - const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); - intptr_t reg_off, reg_max = simd_oprsz(desc); - target_ulong addr; - - /* Skip to the first true predicate. */ - reg_off = find_next_active(vg, 0, reg_max, MO_64); - if (likely(reg_off < reg_max)) { - /* Perform one normal read, which will fault or not. */ - addr = off_fn(vm, reg_off); - addr = base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, ra); - - /* The rest of the reads will be non-faulting. */ - } - - /* After any fault, zero the leading predicated false elements. */ - swap_memzero(vd, reg_off); - - while (likely((reg_off += 8) < reg_max)) { - uint8_t pg = *(uint8_t *)(vg + H1(reg_off >> 3)); - if (likely(pg & 1)) { - addr = off_fn(vm, reg_off); - addr = base + (addr << scale); - if (!nonfault_fn(env, vd, reg_off, addr, mmu_idx)) { - record_fault(env, reg_off, reg_max); - break; - } - } else { - *(uint64_t *)(vd + reg_off) = 0; - } - } +#define DO_LDFF1_ZPZ_S(MEM, OFS, MSZ) \ +void HELPER(sve_ldff##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, \ + void *vm, target_ulong base, uint32_t desc) \ +{ \ + sve_ldff1_z(env, vd, vg, vm, base, desc, GETPC(), MO_32, MSZ, \ + off_##OFS##_s, sve_ld1##MEM##_host, sve_ld1##MEM##_tlb); \ } -#define DO_LDFF1_ZPZ_S(MEM, OFS) \ -void HELPER(sve_ldff##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_ldff1_zs(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_s, sve_ld1##MEM##_tlb, sve_ld##MEM##_nf); \ +#define DO_LDFF1_ZPZ_D(MEM, OFS, MSZ) \ +void HELPER(sve_ldff##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, \ + void *vm, target_ulong base, uint32_t desc) \ +{ \ + sve_ldff1_z(env, vd, vg, vm, base, desc, GETPC(), MO_64, MSZ, \ + off_##OFS##_d, sve_ld1##MEM##_host, sve_ld1##MEM##_tlb); \ } -#define DO_LDFF1_ZPZ_D(MEM, OFS) \ -void HELPER(sve_ldff##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_ldff1_zd(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_d, sve_ld1##MEM##_tlb, sve_ld##MEM##_nf); \ -} +DO_LDFF1_ZPZ_S(bsu, zsu, MO_8) +DO_LDFF1_ZPZ_S(bsu, zss, MO_8) +DO_LDFF1_ZPZ_D(bdu, zsu, MO_8) +DO_LDFF1_ZPZ_D(bdu, zss, MO_8) +DO_LDFF1_ZPZ_D(bdu, zd, MO_8) -DO_LDFF1_ZPZ_S(bsu, zsu) -DO_LDFF1_ZPZ_S(bsu, zss) -DO_LDFF1_ZPZ_D(bdu, zsu) -DO_LDFF1_ZPZ_D(bdu, zss) -DO_LDFF1_ZPZ_D(bdu, zd) +DO_LDFF1_ZPZ_S(bss, zsu, MO_8) +DO_LDFF1_ZPZ_S(bss, zss, MO_8) +DO_LDFF1_ZPZ_D(bds, zsu, MO_8) +DO_LDFF1_ZPZ_D(bds, zss, MO_8) +DO_LDFF1_ZPZ_D(bds, zd, MO_8) -DO_LDFF1_ZPZ_S(bss, zsu) -DO_LDFF1_ZPZ_S(bss, zss) -DO_LDFF1_ZPZ_D(bds, zsu) -DO_LDFF1_ZPZ_D(bds, zss) -DO_LDFF1_ZPZ_D(bds, zd) +DO_LDFF1_ZPZ_S(hsu_le, zsu, MO_16) +DO_LDFF1_ZPZ_S(hsu_le, zss, MO_16) +DO_LDFF1_ZPZ_D(hdu_le, zsu, MO_16) +DO_LDFF1_ZPZ_D(hdu_le, zss, MO_16) +DO_LDFF1_ZPZ_D(hdu_le, zd, MO_16) -DO_LDFF1_ZPZ_S(hsu_le, zsu) -DO_LDFF1_ZPZ_S(hsu_le, zss) -DO_LDFF1_ZPZ_D(hdu_le, zsu) -DO_LDFF1_ZPZ_D(hdu_le, zss) -DO_LDFF1_ZPZ_D(hdu_le, zd) +DO_LDFF1_ZPZ_S(hsu_be, zsu, MO_16) +DO_LDFF1_ZPZ_S(hsu_be, zss, MO_16) +DO_LDFF1_ZPZ_D(hdu_be, zsu, MO_16) +DO_LDFF1_ZPZ_D(hdu_be, zss, MO_16) +DO_LDFF1_ZPZ_D(hdu_be, zd, MO_16) -DO_LDFF1_ZPZ_S(hsu_be, zsu) -DO_LDFF1_ZPZ_S(hsu_be, zss) -DO_LDFF1_ZPZ_D(hdu_be, zsu) -DO_LDFF1_ZPZ_D(hdu_be, zss) -DO_LDFF1_ZPZ_D(hdu_be, zd) +DO_LDFF1_ZPZ_S(hss_le, zsu, MO_16) +DO_LDFF1_ZPZ_S(hss_le, zss, MO_16) +DO_LDFF1_ZPZ_D(hds_le, zsu, MO_16) +DO_LDFF1_ZPZ_D(hds_le, zss, MO_16) +DO_LDFF1_ZPZ_D(hds_le, zd, MO_16) -DO_LDFF1_ZPZ_S(hss_le, zsu) -DO_LDFF1_ZPZ_S(hss_le, zss) -DO_LDFF1_ZPZ_D(hds_le, zsu) -DO_LDFF1_ZPZ_D(hds_le, zss) -DO_LDFF1_ZPZ_D(hds_le, zd) +DO_LDFF1_ZPZ_S(hss_be, zsu, MO_16) +DO_LDFF1_ZPZ_S(hss_be, zss, MO_16) +DO_LDFF1_ZPZ_D(hds_be, zsu, MO_16) +DO_LDFF1_ZPZ_D(hds_be, zss, MO_16) +DO_LDFF1_ZPZ_D(hds_be, zd, MO_16) -DO_LDFF1_ZPZ_S(hss_be, zsu) -DO_LDFF1_ZPZ_S(hss_be, zss) -DO_LDFF1_ZPZ_D(hds_be, zsu) -DO_LDFF1_ZPZ_D(hds_be, zss) -DO_LDFF1_ZPZ_D(hds_be, zd) +DO_LDFF1_ZPZ_S(ss_le, zsu, MO_32) +DO_LDFF1_ZPZ_S(ss_le, zss, MO_32) +DO_LDFF1_ZPZ_D(sdu_le, zsu, MO_32) +DO_LDFF1_ZPZ_D(sdu_le, zss, MO_32) +DO_LDFF1_ZPZ_D(sdu_le, zd, MO_32) -DO_LDFF1_ZPZ_S(ss_le, zsu) -DO_LDFF1_ZPZ_S(ss_le, zss) -DO_LDFF1_ZPZ_D(sdu_le, zsu) -DO_LDFF1_ZPZ_D(sdu_le, zss) -DO_LDFF1_ZPZ_D(sdu_le, zd) +DO_LDFF1_ZPZ_S(ss_be, zsu, MO_32) +DO_LDFF1_ZPZ_S(ss_be, zss, MO_32) +DO_LDFF1_ZPZ_D(sdu_be, zsu, MO_32) +DO_LDFF1_ZPZ_D(sdu_be, zss, MO_32) +DO_LDFF1_ZPZ_D(sdu_be, zd, MO_32) -DO_LDFF1_ZPZ_S(ss_be, zsu) -DO_LDFF1_ZPZ_S(ss_be, zss) -DO_LDFF1_ZPZ_D(sdu_be, zsu) -DO_LDFF1_ZPZ_D(sdu_be, zss) -DO_LDFF1_ZPZ_D(sdu_be, zd) +DO_LDFF1_ZPZ_D(sds_le, zsu, MO_32) +DO_LDFF1_ZPZ_D(sds_le, zss, MO_32) +DO_LDFF1_ZPZ_D(sds_le, zd, MO_32) -DO_LDFF1_ZPZ_D(sds_le, zsu) -DO_LDFF1_ZPZ_D(sds_le, zss) -DO_LDFF1_ZPZ_D(sds_le, zd) +DO_LDFF1_ZPZ_D(sds_be, zsu, MO_32) +DO_LDFF1_ZPZ_D(sds_be, zss, MO_32) +DO_LDFF1_ZPZ_D(sds_be, zd, MO_32) -DO_LDFF1_ZPZ_D(sds_be, zsu) -DO_LDFF1_ZPZ_D(sds_be, zss) -DO_LDFF1_ZPZ_D(sds_be, zd) +DO_LDFF1_ZPZ_D(dd_le, zsu, MO_64) +DO_LDFF1_ZPZ_D(dd_le, zss, MO_64) +DO_LDFF1_ZPZ_D(dd_le, zd, MO_64) -DO_LDFF1_ZPZ_D(dd_le, zsu) -DO_LDFF1_ZPZ_D(dd_le, zss) -DO_LDFF1_ZPZ_D(dd_le, zd) - -DO_LDFF1_ZPZ_D(dd_be, zsu) -DO_LDFF1_ZPZ_D(dd_be, zss) -DO_LDFF1_ZPZ_D(dd_be, zd) +DO_LDFF1_ZPZ_D(dd_be, zsu, MO_64) +DO_LDFF1_ZPZ_D(dd_be, zss, MO_64) +DO_LDFF1_ZPZ_D(dd_be, zd, MO_64) /* Stores with a vector index. */ From patchwork Wed Mar 11 06:44:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 184400 Delivered-To: patch@linaro.org Received: by 2002:a92:1f12:0:0:0:0:0 with SMTP id i18csp287003ile; Tue, 10 Mar 2020 23:58:40 -0700 (PDT) X-Google-Smtp-Source: ADFU+vvwQ4TjnzKg1oTaz0/eParodpAFKLwZvVuBOX/rzrtcWrtvsVjFjpNWYCWlZLncI6NDm3B2 X-Received: by 2002:ac8:1248:: with SMTP id g8mr1375502qtj.390.1583909920653; Tue, 10 Mar 2020 23:58:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1583909920; cv=none; d=google.com; s=arc-20160816; b=MgnW5t9FXGQu2oWD3fbIkSZ/+5FHYQpI03aSBGUNeWD/N4PD+AH8dZCshAZd+VzEqv qRPuWKff1WmyrdVnyf1uD//9Y9m4ToPnVDwuI7SKspXE3mIl9QuILEgmYD2M3cZ2btN0 F1L402DMilQP+MxwXNaa3bJ8BY5TDB3qIhAp8sI4fE1wCnXXw3L/UzjqEQJ5zpbPYkHI 6naNUfsTHtedw1oItUeWmPuLRO4OeqwNPh7+rhA/7LPMvHr2aZUCiUlq4bMId2s+xS/p rvfo5YyIMu43fLXIusfBTaV0oD7uwUiPEukLhi3ae/AzT2MqAwQ0YwsbO9vWgJTWFhbN 5hiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=lMQ8Ls/5UNh2CeM9JrBjlxyyT05PO0oq+06rVR26ElQ=; b=cES646gRxS4kUWzlHvCkFSGXiQYUzCj95rvMMDqq51KbldGksybuo/ik3CYptO6GqN BInLrt7nHyyXeYDyzRUV99+pTFnAPQHM/6ZxfDleeGbRqBdLdKCXlqdME3yYIIGz+coc iu7/LiPoLNhCf3GmnJ1DyAd1467D/QE9z6eyrOp48GX+sWIVUuZgzPZ2Wjuea9C30G3M zoXvYqTjvRlA0DnRm0//Z6moQT4gvjPBQ7vsDBHXZCUfCJ62FyVhlkNCi4uO3hEQz59E GziaAj4MSg0MYgv9WcbwdCAvgSkk87GTXZQBM/v2p6xFXndOFI0ZgHSXJcA3AB0v2bts 7Wdg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=pnsSDz7u; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id x8si523589qkn.4.2020.03.10.23.58.40 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Mar 2020 23:58:40 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=pnsSDz7u; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47092 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvKC-0002j5-7U for patch@linaro.org; Wed, 11 Mar 2020 02:58:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59699) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6j-0006Uc-LB for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6h-00024i-Cs for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:45 -0400 Received: from mail-pj1-x1044.google.com ([2607:f8b0:4864:20::1044]:37668) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6h-00023q-1q for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:43 -0400 Received: by mail-pj1-x1044.google.com with SMTP id ca13so505618pjb.2 for ; Tue, 10 Mar 2020 23:44:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lMQ8Ls/5UNh2CeM9JrBjlxyyT05PO0oq+06rVR26ElQ=; b=pnsSDz7uX7GhYBZZYMXEfjVFEGPwaQGv7iR7MVDt6XU0LW1Tc/eITVr8UjAcfxvK4V 2B4Fu0DhX74i8csLOBBc+UDgPV85jn5k84zP86wz9JIuJ1V5T840XSH3obuDoTn/6Fbp tYzJOBB3QsLnT643Tp5+KdKKwjHOU+cNNHu2kh7Z37ZMteBQNAf8ah68jkLbsR9PGjK/ f+A6jOioYsKWmdEAHaoK1u2yLyT+ID8FJFKXIgPhvDeTc8HpoN7smPU3NCHOolFjgmW4 la7nrqjJmb6BQiW5r4AAPyoq602MxLUej4VkFpm2Lw410Poj3NiLhVHVxMEcjF9bl1ho 3x3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lMQ8Ls/5UNh2CeM9JrBjlxyyT05PO0oq+06rVR26ElQ=; b=uj4P/eco1zmtYZ9uFRY6BBQJOmaFe/QFB49STpAdWHyXIHDMeC7iRt+0uej2Vpd3O0 Z0G53vq0QIWqcRFfI5k7zA7SMz01apJX8i+5RosuPCiuuRjWnOtzKjQ/Ve2Qh6vE7gUR 9p4DAj081ahyg1fb5KEVuO3FWpPfwhFMi8Am+4HMBrCRd3qAfilyvCX7TaKWGMTl5NpD CbC918Zroxb938XOf6j5fZx9DnYaEC36mp407l9wfx/26Pajq4Uc7a/MM45GFGiSl03a TtbUBNYePxh09kgCdc2TqpNxOUu0RGNjuwhdIU57oX/mVGM++BzFChqPp3Hh6wtng2Z9 s/TA== X-Gm-Message-State: ANhLgQ1XKWO5j0Y6phn+gKMsuV6PFlgbWJtWfagt52GS+gMsCkFau716 1c+2CpSVF4Nrxh9XcImo1GlZt/RIuao= X-Received: by 2002:a17:90a:e7c8:: with SMTP id kb8mr1881923pjb.113.1583909080607; Tue, 10 Mar 2020 23:44:40 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:39 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 14/16] target/arm: Reuse sve_probe_page for scatter stores Date: Tue, 10 Mar 2020 23:44:18 -0700 Message-Id: <20200311064420.30606-15-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::1044 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 182 ++++++++++++++++++++++++---------------- 1 file changed, 111 insertions(+), 71 deletions(-) -- 2.20.1 Reviewed-by: Peter Maydell diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index a33699adf8..e8b8f2560d 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -5414,94 +5414,134 @@ DO_LDFF1_ZPZ_D(dd_be, zd, MO_64) /* Stores with a vector index. */ -static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) +static inline QEMU_ALWAYS_INLINE +void sve_st1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t retaddr, + int esize, int msize, zreg_off_fn *off_fn, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); - intptr_t i, oprsz = simd_oprsz(desc); + const int mmu_idx = cpu_mmu_index(env, false); + const intptr_t reg_max = simd_oprsz(desc); + void *host[ARM_MAX_VQ * 4]; + intptr_t reg_off, i; + SVEHostPage info, info2; - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + /* + * Probe all of the elements for host addresses and flags. + */ + i = reg_off = 0; + do { + uint64_t pg = vg[reg_off >> 6]; do { - if (likely(pg & 1)) { - target_ulong off = off_fn(vm, i); - tlb_fn(env, vd, i, base + (off << scale), ra); + target_ulong addr = base + (off_fn(vm, reg_off) << scale); + target_ulong in_page = -(addr | TARGET_PAGE_MASK); + + host[i] = NULL; + if (likely((pg >> (reg_off & 63)) & 1)) { + if (likely(in_page >= msize)) { + sve_probe_page(&info, false, env, addr, 0, MMU_DATA_STORE, + mmu_idx, retaddr); + host[i] = info.host; + } else { + /* + * Element crosses the page boundary. + * Probe both pages, but do not record the host address, + * so that we use the slow path. + */ + sve_probe_page(&info, false, env, addr, 0, + MMU_DATA_STORE, mmu_idx, retaddr); + sve_probe_page(&info2, false, env, addr + in_page, 0, + MMU_DATA_STORE, mmu_idx, retaddr); + info.flags |= info2.flags; + } + + if (unlikely(info.flags & TLB_WATCHPOINT)) { + cpu_check_watchpoint(env_cpu(env), addr, msize, + info.attrs, BP_MEM_WRITE, retaddr); + } + /* TODO: MTE check. */ } - i += 4, pg >>= 4; - } while (i & 15); - } -} + i += 1; + reg_off += esize; + } while (reg_off & 63); + } while (reg_off < reg_max); -static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) -{ - const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); - intptr_t i, oprsz = simd_oprsz(desc) / 8; - - for (i = 0; i < oprsz; i++) { - uint8_t pg = *(uint8_t *)(vg + H1(i)); - if (likely(pg & 1)) { - target_ulong off = off_fn(vm, i * 8); - tlb_fn(env, vd, i * 8, base + (off << scale), ra); + /* + * Now that we have recognized all exceptions except SyncExternal + * (from TLB_MMIO), which we cannot avoid, perform all of the stores. + * + * Note for the common case of an element in RAM, not crossing a page + * boundary, we have stored the host address in host[]. This doubles + * as a first-level check against the predicate, since only enabled + * elements have non-null host addresses. + */ + i = reg_off = 0; + do { + void *h = host[i]; + if (likely(h != NULL)) { + host_fn(vd, reg_off, h); + } else if ((vg[reg_off >> 6] >> (reg_off & 63)) & 1) { + target_ulong addr = base + (off_fn(vm, reg_off) << scale); + tlb_fn(env, vd, reg_off, addr, retaddr); } - } + i += 1; + reg_off += esize; + } while (reg_off < reg_max); } -#define DO_ST1_ZPZ_S(MEM, OFS) \ -void QEMU_FLATTEN HELPER(sve_st##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_st1_zs(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_s, sve_st1##MEM##_tlb); \ +#define DO_ST1_ZPZ_S(MEM, OFS, MSZ) \ +void HELPER(sve_st##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, \ + void *vm, target_ulong base, uint32_t desc) \ +{ \ + sve_st1_z(env, vd, vg, vm, base, desc, GETPC(), 4, 1 << MSZ, \ + off_##OFS##_s, sve_st1##MEM##_host, sve_st1##MEM##_tlb); \ } -#define DO_ST1_ZPZ_D(MEM, OFS) \ -void QEMU_FLATTEN HELPER(sve_st##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_st1_zd(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_d, sve_st1##MEM##_tlb); \ +#define DO_ST1_ZPZ_D(MEM, OFS, MSZ) \ +void HELPER(sve_st##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, \ + void *vm, target_ulong base, uint32_t desc) \ +{ \ + sve_st1_z(env, vd, vg, vm, base, desc, GETPC(), 8, 1 << MSZ, \ + off_##OFS##_d, sve_st1##MEM##_host, sve_st1##MEM##_tlb); \ } -DO_ST1_ZPZ_S(bs, zsu) -DO_ST1_ZPZ_S(hs_le, zsu) -DO_ST1_ZPZ_S(hs_be, zsu) -DO_ST1_ZPZ_S(ss_le, zsu) -DO_ST1_ZPZ_S(ss_be, zsu) +DO_ST1_ZPZ_S(bs, zsu, MO_8) +DO_ST1_ZPZ_S(hs_le, zsu, MO_16) +DO_ST1_ZPZ_S(hs_be, zsu, MO_16) +DO_ST1_ZPZ_S(ss_le, zsu, MO_32) +DO_ST1_ZPZ_S(ss_be, zsu, MO_32) -DO_ST1_ZPZ_S(bs, zss) -DO_ST1_ZPZ_S(hs_le, zss) -DO_ST1_ZPZ_S(hs_be, zss) -DO_ST1_ZPZ_S(ss_le, zss) -DO_ST1_ZPZ_S(ss_be, zss) +DO_ST1_ZPZ_S(bs, zss, MO_8) +DO_ST1_ZPZ_S(hs_le, zss, MO_16) +DO_ST1_ZPZ_S(hs_be, zss, MO_16) +DO_ST1_ZPZ_S(ss_le, zss, MO_32) +DO_ST1_ZPZ_S(ss_be, zss, MO_32) -DO_ST1_ZPZ_D(bd, zsu) -DO_ST1_ZPZ_D(hd_le, zsu) -DO_ST1_ZPZ_D(hd_be, zsu) -DO_ST1_ZPZ_D(sd_le, zsu) -DO_ST1_ZPZ_D(sd_be, zsu) -DO_ST1_ZPZ_D(dd_le, zsu) -DO_ST1_ZPZ_D(dd_be, zsu) +DO_ST1_ZPZ_D(bd, zsu, MO_8) +DO_ST1_ZPZ_D(hd_le, zsu, MO_16) +DO_ST1_ZPZ_D(hd_be, zsu, MO_16) +DO_ST1_ZPZ_D(sd_le, zsu, MO_32) +DO_ST1_ZPZ_D(sd_be, zsu, MO_32) +DO_ST1_ZPZ_D(dd_le, zsu, MO_64) +DO_ST1_ZPZ_D(dd_be, zsu, MO_64) -DO_ST1_ZPZ_D(bd, zss) -DO_ST1_ZPZ_D(hd_le, zss) -DO_ST1_ZPZ_D(hd_be, zss) -DO_ST1_ZPZ_D(sd_le, zss) -DO_ST1_ZPZ_D(sd_be, zss) -DO_ST1_ZPZ_D(dd_le, zss) -DO_ST1_ZPZ_D(dd_be, zss) +DO_ST1_ZPZ_D(bd, zss, MO_8) +DO_ST1_ZPZ_D(hd_le, zss, MO_16) +DO_ST1_ZPZ_D(hd_be, zss, MO_16) +DO_ST1_ZPZ_D(sd_le, zss, MO_32) +DO_ST1_ZPZ_D(sd_be, zss, MO_32) +DO_ST1_ZPZ_D(dd_le, zss, MO_64) +DO_ST1_ZPZ_D(dd_be, zss, MO_64) -DO_ST1_ZPZ_D(bd, zd) -DO_ST1_ZPZ_D(hd_le, zd) -DO_ST1_ZPZ_D(hd_be, zd) -DO_ST1_ZPZ_D(sd_le, zd) -DO_ST1_ZPZ_D(sd_be, zd) -DO_ST1_ZPZ_D(dd_le, zd) -DO_ST1_ZPZ_D(dd_be, zd) +DO_ST1_ZPZ_D(bd, zd, MO_8) +DO_ST1_ZPZ_D(hd_le, zd, MO_16) +DO_ST1_ZPZ_D(hd_be, zd, MO_16) +DO_ST1_ZPZ_D(sd_le, zd, MO_32) +DO_ST1_ZPZ_D(sd_be, zd, MO_32) +DO_ST1_ZPZ_D(dd_le, zd, MO_64) +DO_ST1_ZPZ_D(dd_be, zd, MO_64) #undef DO_ST1_ZPZ_S #undef DO_ST1_ZPZ_D From patchwork Wed Mar 11 06:44:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 184394 Delivered-To: patch@linaro.org Received: by 2002:a92:1f12:0:0:0:0:0 with SMTP id i18csp281986ile; Tue, 10 Mar 2020 23:51:19 -0700 (PDT) X-Google-Smtp-Source: ADFU+vv/M9LF6+BPg/IDwglFlumpmbZVIcc7sAIcheXxsZlfC4Mf+DK/0DtK0liCAxfv1mypnfLA X-Received: by 2002:a37:4a51:: with SMTP id x78mr1285052qka.445.1583909479663; Tue, 10 Mar 2020 23:51:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1583909474; cv=none; d=google.com; s=arc-20160816; b=s2qVVoZfKPSpIpe/+4ilUyh46EQoCxlvh+94hTEvyoCisdvKu/N5dGBn+2LG8LAZUH 8+uXlOhUoU5PEWO0rsRbkI6p5fWzd1sWUHp2J9+cN6p1Oz2WfHqdKO3S4OPsN5uthg1f emBv2UC4oTympbgih9c/8SRsCJgle8ZyKGfKcgclGz+IbTWedMWYBlyL9JIv/9sfvX3i jul5JCQB1n01TxnM6N6tHmRQ6Pz7u0eXKrznegk4t2TAoht1h0zMoP4UcxfK61C8JHzZ Zd2eoQlkbKUn33NaSpQmF42Uic3mb9DsCI3TYO63ZiSU5AzYhOMcopoMoZPDQ1NgyLaZ +cfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=JO9UDGNI/3FywIxMw2Y2YRQLM5xOitYFKQKOyJDpZrk=; b=O7nqL+e/nWKWwbR2u5qn2OoNFv3ngHJV3udRwKJFmeteHKBvDRtcATujCzxJ6f3eIr +t9yCIROTHMhhb6AdNyp4KB+bPGZd7YBY2h8cSL7P6uq4eaUlVR5HxRelOCRegBmKp3y YMht3Aftt1KpxjbTMqKR3VW2SwUr42W52uef3kC8M8/sp2ztkct9POWWvVZ0wlIdSMhC huZNAfCrZPnS4099NLkLkruDyXdUNSvlZvPETMmaR2SPXjEY2C2BpON2U6/Emz8ao8Qe P/NTkRZXNzWnnSo4oTiiydaOTkWfw6/j8QiQ9s90vqWTB53hzUgFA+VTdXzPrCl3Mwyv 0UEw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=M+Sfwx6b; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id s37si558637qth.84.2020.03.10.23.51.14 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Mar 2020 23:51:14 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=M+Sfwx6b; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:46578 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvCz-0007yg-Jc for patch@linaro.org; Wed, 11 Mar 2020 02:51:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59700) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6j-0006Ud-Kv for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6h-00025F-Ro for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:45 -0400 Received: from mail-pf1-x441.google.com ([2607:f8b0:4864:20::441]:36836) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6h-00024P-KX for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:43 -0400 Received: by mail-pf1-x441.google.com with SMTP id i13so751166pfe.3 for ; Tue, 10 Mar 2020 23:44:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JO9UDGNI/3FywIxMw2Y2YRQLM5xOitYFKQKOyJDpZrk=; b=M+Sfwx6b6MM1KXMzEOpPGdsCRxA/iZj8DUjKXgTHEyzUTW3VwXzdgwViEcdW+c6zyP zn0xpW2rrCXV5gpIgQKUgWqybgDX3mbePqLm8GRCXbodPhO/U5A36zi0/QUcuubZQUXG c9FC/7vf1XdSoc4FN2H48kQ2EUJFPgCp7rd8gxBfrBAU+vRvQf3HBbJgEjSaJ/tej7JI acnQh5NNkh+cgiISJJssHOuypnUD6LdEgcQuG77TdILlFQIngnQBprYjkmGzZIcibTim q+vN04OuozaQLky0O3veMW2r4nY3cF75y66qgqS7wo6WiDX8Y4VA5N6GG9cdkXWJIFLx JCUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JO9UDGNI/3FywIxMw2Y2YRQLM5xOitYFKQKOyJDpZrk=; b=O6uyd2YdwONtorl8F8TWf9aHWT9dqH5ox1xTevtWgVk2R1gWcEFjBgqG2+3wAUnikq KQHstb27UOtdjpuwDnmxVgrphTZbJG3JBlSeI8+EbRxhVOfwSoFikPFRgVB8AQG/u63/ MDP6FHbwk3lD/995ZRSkHa5lprHq2+jL5ngvtl5icaDnCl/ZxRMyo8SaITJzUbutHRfG ECG7r+9dy+PzwZdrsEW2lHKfK3QzXvvyhgWcfd1QUPWcF0kvGDBjQ9Ihl7WzBYNLAIOe M8dG8PlAYXfjNYxatH/wF7qvER/UTSYu/NzZdxNevSEJFIditEwWWZVdA4a3iZxlEXVb Tn3Q== X-Gm-Message-State: ANhLgQ3paJZVsFxZeOtN5OOwn0bjoUty0qJJa8i00OxztbbNxnPLEt8D aukkVkQ+7NZOgZzok8YXvID3ryWyv+o= X-Received: by 2002:a62:be04:: with SMTP id l4mr1473040pff.234.1583909082120; Tue, 10 Mar 2020 23:44:42 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:41 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 15/16] target/arm: Reuse sve_probe_page for gather loads Date: Tue, 10 Mar 2020 23:44:19 -0700 Message-Id: <20200311064420.30606-16-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::441 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 208 +++++++++++++++++++++------------------- 1 file changed, 109 insertions(+), 99 deletions(-) -- 2.20.1 Reviewed-by: Peter Maydell diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index e8b8f2560d..4ad870f1ee 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -5125,130 +5125,140 @@ static target_ulong off_zd_d(void *reg, intptr_t reg_ofs) return *(uint64_t *)(reg + reg_ofs); } -static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) +static inline QEMU_ALWAYS_INLINE +void sve_ld1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t retaddr, + int esize, int msize, zreg_off_fn *off_fn, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); - intptr_t i, oprsz = simd_oprsz(desc); - ARMVectorReg scratch = { }; + const int mmu_idx = cpu_mmu_index(env, false); + const intptr_t reg_max = simd_oprsz(desc); + ARMVectorReg scratch; + intptr_t reg_off; + SVEHostPage info, info2; - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + memset(&scratch, 0, reg_max); + reg_off = 0; + do { + uint64_t pg = vg[reg_off >> 6]; do { if (likely(pg & 1)) { - target_ulong off = off_fn(vm, i); - tlb_fn(env, &scratch, i, base + (off << scale), ra); + target_ulong addr = base + (off_fn(vm, reg_off) << scale); + target_ulong in_page = -(addr | TARGET_PAGE_MASK); + + sve_probe_page(&info, false, env, addr, 0, MMU_DATA_LOAD, + mmu_idx, retaddr); + + if (likely(in_page >= msize)) { + if (unlikely(info.flags & TLB_WATCHPOINT)) { + cpu_check_watchpoint(env_cpu(env), addr, msize, + info.attrs, BP_MEM_READ, retaddr); + } + /* TODO: MTE check */ + host_fn(&scratch, reg_off, info.host); + } else { + /* Element crosses the page boundary. */ + sve_probe_page(&info2, false, env, addr + in_page, 0, + MMU_DATA_LOAD, mmu_idx, retaddr); + if (unlikely((info.flags | info2.flags) & TLB_WATCHPOINT)) { + cpu_check_watchpoint(env_cpu(env), addr, + msize, info.attrs, + BP_MEM_READ, retaddr); + } + /* TODO: MTE check */ + tlb_fn(env, &scratch, reg_off, addr, retaddr); + } } - i += 4, pg >>= 4; - } while (i & 15); - } + reg_off += esize; + pg >>= esize; + } while (reg_off & 63); + } while (reg_off < reg_max); /* Wait until all exceptions have been raised to write back. */ - memcpy(vd, &scratch, oprsz); + memcpy(vd, &scratch, reg_max); } -static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) -{ - const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); - intptr_t i, oprsz = simd_oprsz(desc) / 8; - ARMVectorReg scratch = { }; - - for (i = 0; i < oprsz; i++) { - uint8_t pg = *(uint8_t *)(vg + H1(i)); - if (likely(pg & 1)) { - target_ulong off = off_fn(vm, i * 8); - tlb_fn(env, &scratch, i * 8, base + (off << scale), ra); - } - } - - /* Wait until all exceptions have been raised to write back. */ - memcpy(vd, &scratch, oprsz * 8); +#define DO_LD1_ZPZ_S(MEM, OFS, MSZ) \ +void HELPER(sve_ld##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, \ + void *vm, target_ulong base, uint32_t desc) \ +{ \ + sve_ld1_z(env, vd, vg, vm, base, desc, GETPC(), 4, 1 << MSZ, \ + off_##OFS##_s, sve_ld1##MEM##_host, sve_ld1##MEM##_tlb); \ } -#define DO_LD1_ZPZ_S(MEM, OFS) \ -void QEMU_FLATTEN HELPER(sve_ld##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_ld1_zs(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_s, sve_ld1##MEM##_tlb); \ +#define DO_LD1_ZPZ_D(MEM, OFS, MSZ) \ +void HELPER(sve_ld##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, \ + void *vm, target_ulong base, uint32_t desc) \ +{ \ + sve_ld1_z(env, vd, vg, vm, base, desc, GETPC(), 8, 1 << MSZ, \ + off_##OFS##_d, sve_ld1##MEM##_host, sve_ld1##MEM##_tlb); \ } -#define DO_LD1_ZPZ_D(MEM, OFS) \ -void QEMU_FLATTEN HELPER(sve_ld##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_ld1_zd(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_d, sve_ld1##MEM##_tlb); \ -} +DO_LD1_ZPZ_S(bsu, zsu, MO_8) +DO_LD1_ZPZ_S(bsu, zss, MO_8) +DO_LD1_ZPZ_D(bdu, zsu, MO_8) +DO_LD1_ZPZ_D(bdu, zss, MO_8) +DO_LD1_ZPZ_D(bdu, zd, MO_8) -DO_LD1_ZPZ_S(bsu, zsu) -DO_LD1_ZPZ_S(bsu, zss) -DO_LD1_ZPZ_D(bdu, zsu) -DO_LD1_ZPZ_D(bdu, zss) -DO_LD1_ZPZ_D(bdu, zd) +DO_LD1_ZPZ_S(bss, zsu, MO_8) +DO_LD1_ZPZ_S(bss, zss, MO_8) +DO_LD1_ZPZ_D(bds, zsu, MO_8) +DO_LD1_ZPZ_D(bds, zss, MO_8) +DO_LD1_ZPZ_D(bds, zd, MO_8) -DO_LD1_ZPZ_S(bss, zsu) -DO_LD1_ZPZ_S(bss, zss) -DO_LD1_ZPZ_D(bds, zsu) -DO_LD1_ZPZ_D(bds, zss) -DO_LD1_ZPZ_D(bds, zd) +DO_LD1_ZPZ_S(hsu_le, zsu, MO_16) +DO_LD1_ZPZ_S(hsu_le, zss, MO_16) +DO_LD1_ZPZ_D(hdu_le, zsu, MO_16) +DO_LD1_ZPZ_D(hdu_le, zss, MO_16) +DO_LD1_ZPZ_D(hdu_le, zd, MO_16) -DO_LD1_ZPZ_S(hsu_le, zsu) -DO_LD1_ZPZ_S(hsu_le, zss) -DO_LD1_ZPZ_D(hdu_le, zsu) -DO_LD1_ZPZ_D(hdu_le, zss) -DO_LD1_ZPZ_D(hdu_le, zd) +DO_LD1_ZPZ_S(hsu_be, zsu, MO_16) +DO_LD1_ZPZ_S(hsu_be, zss, MO_16) +DO_LD1_ZPZ_D(hdu_be, zsu, MO_16) +DO_LD1_ZPZ_D(hdu_be, zss, MO_16) +DO_LD1_ZPZ_D(hdu_be, zd, MO_16) -DO_LD1_ZPZ_S(hsu_be, zsu) -DO_LD1_ZPZ_S(hsu_be, zss) -DO_LD1_ZPZ_D(hdu_be, zsu) -DO_LD1_ZPZ_D(hdu_be, zss) -DO_LD1_ZPZ_D(hdu_be, zd) +DO_LD1_ZPZ_S(hss_le, zsu, MO_16) +DO_LD1_ZPZ_S(hss_le, zss, MO_16) +DO_LD1_ZPZ_D(hds_le, zsu, MO_16) +DO_LD1_ZPZ_D(hds_le, zss, MO_16) +DO_LD1_ZPZ_D(hds_le, zd, MO_16) -DO_LD1_ZPZ_S(hss_le, zsu) -DO_LD1_ZPZ_S(hss_le, zss) -DO_LD1_ZPZ_D(hds_le, zsu) -DO_LD1_ZPZ_D(hds_le, zss) -DO_LD1_ZPZ_D(hds_le, zd) +DO_LD1_ZPZ_S(hss_be, zsu, MO_16) +DO_LD1_ZPZ_S(hss_be, zss, MO_16) +DO_LD1_ZPZ_D(hds_be, zsu, MO_16) +DO_LD1_ZPZ_D(hds_be, zss, MO_16) +DO_LD1_ZPZ_D(hds_be, zd, MO_16) -DO_LD1_ZPZ_S(hss_be, zsu) -DO_LD1_ZPZ_S(hss_be, zss) -DO_LD1_ZPZ_D(hds_be, zsu) -DO_LD1_ZPZ_D(hds_be, zss) -DO_LD1_ZPZ_D(hds_be, zd) +DO_LD1_ZPZ_S(ss_le, zsu, MO_32) +DO_LD1_ZPZ_S(ss_le, zss, MO_32) +DO_LD1_ZPZ_D(sdu_le, zsu, MO_32) +DO_LD1_ZPZ_D(sdu_le, zss, MO_32) +DO_LD1_ZPZ_D(sdu_le, zd, MO_32) -DO_LD1_ZPZ_S(ss_le, zsu) -DO_LD1_ZPZ_S(ss_le, zss) -DO_LD1_ZPZ_D(sdu_le, zsu) -DO_LD1_ZPZ_D(sdu_le, zss) -DO_LD1_ZPZ_D(sdu_le, zd) +DO_LD1_ZPZ_S(ss_be, zsu, MO_32) +DO_LD1_ZPZ_S(ss_be, zss, MO_32) +DO_LD1_ZPZ_D(sdu_be, zsu, MO_32) +DO_LD1_ZPZ_D(sdu_be, zss, MO_32) +DO_LD1_ZPZ_D(sdu_be, zd, MO_32) -DO_LD1_ZPZ_S(ss_be, zsu) -DO_LD1_ZPZ_S(ss_be, zss) -DO_LD1_ZPZ_D(sdu_be, zsu) -DO_LD1_ZPZ_D(sdu_be, zss) -DO_LD1_ZPZ_D(sdu_be, zd) +DO_LD1_ZPZ_D(sds_le, zsu, MO_32) +DO_LD1_ZPZ_D(sds_le, zss, MO_32) +DO_LD1_ZPZ_D(sds_le, zd, MO_32) -DO_LD1_ZPZ_D(sds_le, zsu) -DO_LD1_ZPZ_D(sds_le, zss) -DO_LD1_ZPZ_D(sds_le, zd) +DO_LD1_ZPZ_D(sds_be, zsu, MO_32) +DO_LD1_ZPZ_D(sds_be, zss, MO_32) +DO_LD1_ZPZ_D(sds_be, zd, MO_32) -DO_LD1_ZPZ_D(sds_be, zsu) -DO_LD1_ZPZ_D(sds_be, zss) -DO_LD1_ZPZ_D(sds_be, zd) +DO_LD1_ZPZ_D(dd_le, zsu, MO_64) +DO_LD1_ZPZ_D(dd_le, zss, MO_64) +DO_LD1_ZPZ_D(dd_le, zd, MO_64) -DO_LD1_ZPZ_D(dd_le, zsu) -DO_LD1_ZPZ_D(dd_le, zss) -DO_LD1_ZPZ_D(dd_le, zd) - -DO_LD1_ZPZ_D(dd_be, zsu) -DO_LD1_ZPZ_D(dd_be, zss) -DO_LD1_ZPZ_D(dd_be, zd) +DO_LD1_ZPZ_D(dd_be, zsu, MO_64) +DO_LD1_ZPZ_D(dd_be, zss, MO_64) +DO_LD1_ZPZ_D(dd_be, zd, MO_64) #undef DO_LD1_ZPZ_S #undef DO_LD1_ZPZ_D From patchwork Wed Mar 11 06:44:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 184397 Delivered-To: patch@linaro.org Received: by 2002:a92:1f12:0:0:0:0:0 with SMTP id i18csp284201ile; Tue, 10 Mar 2020 23:54:27 -0700 (PDT) X-Google-Smtp-Source: ADFU+vuG5nXutHPe9Do/UN+l/QzVaecnv1qYizdMh1PcDntps2wtHYU9VOeLqOSqsAUAVK2t9cvI X-Received: by 2002:ac8:4084:: with SMTP id p4mr1362495qtl.30.1583909667805; Tue, 10 Mar 2020 23:54:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1583909667; cv=none; d=google.com; s=arc-20160816; b=OSfubHiKdl1AJY7yFIFDWLzkOGYNElbduCoihPT9r4mccCdxZXzz3U/azkvalYN10X a+94Kkd6d70VZK1PiC76Fh1xNmZVlGooRcaZyjyNUj+tflzTNeqa3PesZyAI7GubFx8Q Ytf21OcEDwg2v284dhqYf5ZCwd6XullGHSxICTnq53OBKWLsM1I/pC6xVXyojY9MqERq NQqn2K8ZaQh8Atrxir+MlhXpy51CYZ1NrILyA0XbVymn1TRg681pCsjFx1GoSbQ2q4tc eFJJm6JHjzvWsLR133R5zu4CDkl54+fEzWJCufqROhLCHbQj4zd5eG03HqyPez0UkboF hZAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=j19UswlgyGkpUPAbfqRH89tG+gAVqbIVGxthtVV4itA=; b=b3UMsHW8seXmnMUkxxs8UYxLAfR2O/XafzWA8zJsdD7oSfjXLRjvTQHyu82DQiFmMM AHbXMP6ZM8bY8yJuuXt4W3YY9+ZEZK0ABXyhSchHvu2q/JLFEdyR6iPxb8/5upr9bBrS x5QrzlKPYw+ipwxsUpLHCsBz/jZQfPbenMhgOQsLOqIMuokqpVaCToxw2GOebEUWHEeQ fGPI0a1r2wpnuzZc3Z/0pafI9MoaL5Vci+JM1o3+OTqs/u4UxEXlaEypzmw24poLsEGm 9hKKY21M2u+5Ck0Wghx1s+iKxXRnnI7yXwGqf6vONDEFiusxWK7VKibHWmUvozfegK0U iLnA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=CrtmWX0N; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id z1si470491qkd.37.2020.03.10.23.54.27 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Mar 2020 23:54:27 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=CrtmWX0N; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:46712 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvG7-0004mf-BJ for patch@linaro.org; Wed, 11 Mar 2020 02:54:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59745) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6l-0006Xj-TB for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6j-00026Y-Kh for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:46 -0400 Received: from mail-pg1-x541.google.com ([2607:f8b0:4864:20::541]:40532) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6j-00025r-F3 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:45 -0400 Received: by mail-pg1-x541.google.com with SMTP id t24so635026pgj.7 for ; Tue, 10 Mar 2020 23:44:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=j19UswlgyGkpUPAbfqRH89tG+gAVqbIVGxthtVV4itA=; b=CrtmWX0NIBqwTNOCpdyYTnzEErpf+EQg+JsW/Sv2a7jjjAG5F1LzZJLIq8zeni96rz z4fCuaKRU9ANW99qtYHBq5vK3OeZYUy4/crU7iXPTPFuIC1PqGKEVVKXuw1aDFQ0RGZ0 ldl6/jqFKrH4g3p8AQHi8E3fd66wA78XLLuMZkHukNdWL4EeJFD7IlbmZHSihj4DcR7X QqdQI/mwtZwBscZo68mxG0yGy1MyUIo3K4hDxjJujyPtJgAIICvgqRyLP9Q+1dyZo1dt xIgEce01eEgsqsOylmCdyivRQlJYCBxx2zTabKZzh2mB74p0zWvw3YHRZqKwjXHlnAKg vhGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=j19UswlgyGkpUPAbfqRH89tG+gAVqbIVGxthtVV4itA=; b=j1PLcEgOu3TDjr9dhZZymrMAPEXg7736pXUO5ccpqHT+/4qFk5BZRIJ3HLisIfT3Tv Lt81BV4etKAdJZduCA+pB4moKhx+HDr5ghmvAKLo6gC7/jNOcqUnw38vqWaR4sKiChR5 BeZwcH87fGioFRiVy9TK3Zs2rZz7mG0AK6di7PCqh7GeimqxnTeicHppBelda9pOYa6U 9Tgxn3EgrpzGcGyHW5puEncfv7tkWRd+ZHHC4T7wa+9AAcHwjy+PB9tdyRWdITt6sZce BFDMq1IX6ZUqp2r59j+w8hk13CTMa0KeQdji9jMuc6jqoNvksfRTHmpFaIaGOhrarr+z b7dw== X-Gm-Message-State: ANhLgQ37KnOfmeRGt5orNST7HmyHpuqoZ8eaVNcbalrzhMGNGK6Vwoh1 2j8KuEPBaB1KOHnzvlw2Ovcq+TEMcyA= X-Received: by 2002:a63:82c2:: with SMTP id w185mr1505089pgd.382.1583909083428; Tue, 10 Mar 2020 23:44:43 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:42 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 16/16] target/arm: Remove sve_memopidx Date: Tue, 10 Mar 2020 23:44:20 -0700 Message-Id: <20200311064420.30606-17-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::541 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" None of the sve helpers use TCGMemOpIdx any longer, so we can stop passing it. Signed-off-by: Richard Henderson --- target/arm/internals.h | 5 ----- target/arm/sve_helper.c | 14 +++++++------- target/arm/translate-sve.c | 17 +++-------------- 3 files changed, 10 insertions(+), 26 deletions(-) -- 2.20.1 Reviewed-by: Peter Maydell diff --git a/target/arm/internals.h b/target/arm/internals.h index e633aff36e..a833e3941d 100644 --- a/target/arm/internals.h +++ b/target/arm/internals.h @@ -979,11 +979,6 @@ static inline int arm_num_ctx_cmps(ARMCPU *cpu) } } -/* Note make_memop_idx reserves 4 bits for mmu_idx, and MO_BSWAP is bit 3. - * Thus a TCGMemOpIdx, without any MO_ALIGN bits, fits in 8 bits. - */ -#define MEMOPIDX_SHIFT 8 - /** * v7m_using_psp: Return true if using process stack pointer * Return true if the CPU is currently using the process stack diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 4ad870f1ee..80453953ad 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4447,7 +4447,7 @@ void sve_ldN_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); + const unsigned rd = simd_data(desc); const intptr_t reg_max = simd_oprsz(desc); intptr_t reg_off, reg_last, mem_off; SVEContLdSt info; @@ -4703,7 +4703,7 @@ void sve_ldnfff1_r(CPUARMState *env, void *vg, const target_ulong addr, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); + const unsigned rd = simd_data(desc); void *vd = &env->vfp.zregs[rd]; const intptr_t reg_max = simd_oprsz(desc); intptr_t reg_off, mem_off, reg_last; @@ -4926,7 +4926,7 @@ void sve_stN_r(CPUARMState *env, uint64_t *vg, target_ulong addr, uint32_t desc, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); + const unsigned rd = simd_data(desc); const intptr_t reg_max = simd_oprsz(desc); intptr_t reg_off, reg_last, mem_off; SVEContLdSt info; @@ -5132,9 +5132,9 @@ void sve_ld1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); const int mmu_idx = cpu_mmu_index(env, false); const intptr_t reg_max = simd_oprsz(desc); + const int scale = simd_data(desc); ARMVectorReg scratch; intptr_t reg_off; SVEHostPage info, info2; @@ -5277,10 +5277,10 @@ void sve_ldff1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, sve_ldst1_tlb_fn *tlb_fn) { const int mmu_idx = cpu_mmu_index(env, false); - const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); + const intptr_t reg_max = simd_oprsz(desc); + const int scale = simd_data(desc); const int esize = 1 << esz; const int msize = 1 << msz; - const intptr_t reg_max = simd_oprsz(desc); intptr_t reg_off; SVEHostPage info; target_ulong addr, in_page; @@ -5431,9 +5431,9 @@ void sve_st1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); const int mmu_idx = cpu_mmu_index(env, false); const intptr_t reg_max = simd_oprsz(desc); + const int scale = simd_data(desc); void *host[ARM_MAX_VQ * 4]; intptr_t reg_off, i; SVEHostPage info, info2; diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index b35bad245e..7bd7de80e6 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -4584,11 +4584,6 @@ static const uint8_t dtype_esz[16] = { 3, 2, 1, 3 }; -static TCGMemOpIdx sve_memopidx(DisasContext *s, int dtype) -{ - return make_memop_idx(s->be_data | dtype_mop[dtype], get_mem_index(s)); -} - static void do_mem_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, int dtype, gen_helper_gvec_mem *fn) { @@ -4601,9 +4596,7 @@ static void do_mem_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, * registers as pointers, so encode the regno into the data field. * For consistency, do this even for LD1. */ - desc = sve_memopidx(s, dtype); - desc |= zt << MEMOPIDX_SHIFT; - desc = simd_desc(vsz, vsz, desc); + desc = simd_desc(vsz, vsz, zt); t_desc = tcg_const_i32(desc); t_pg = tcg_temp_new_ptr(); @@ -4835,9 +4828,7 @@ static void do_ldrq(DisasContext *s, int zt, int pg, TCGv_i64 addr, int msz) int desc, poff; /* Load the first quadword using the normal predicated load helpers. */ - desc = sve_memopidx(s, msz_dtype(s, msz)); - desc |= zt << MEMOPIDX_SHIFT; - desc = simd_desc(16, 16, desc); + desc = simd_desc(16, 16, zt); t_desc = tcg_const_i32(desc); poff = pred_full_reg_offset(s, pg); @@ -5066,9 +5057,7 @@ static void do_mem_zpz(DisasContext *s, int zt, int pg, int zm, TCGv_i32 t_desc; int desc; - desc = sve_memopidx(s, msz_dtype(s, msz)); - desc |= scale << MEMOPIDX_SHIFT; - desc = simd_desc(vsz, vsz, desc); + desc = simd_desc(vsz, vsz, scale); t_desc = tcg_const_i32(desc); tcg_gen_addi_ptr(t_pg, cpu_env, pred_full_reg_offset(s, pg));