From patchwork Fri Oct 5 17:53:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 148283 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp756809lji; Fri, 5 Oct 2018 11:05:54 -0700 (PDT) X-Google-Smtp-Source: ACcGV61HYaJnXvLBdIqMBC8xBpwV06Euk6TjvTRofo1bNayPbO41c7l1NjQVpluLtoFdJMw5rcCj X-Received: by 2002:a37:650b:: with SMTP id z11-v6mr9497992qkb.221.1538762754024; Fri, 05 Oct 2018 11:05:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1538762754; cv=none; d=google.com; s=arc-20160816; b=huvPC8dL/yLwls/1MiqaB+kYhPE4njIFjUGJ+uL5hmpi8Fmov8lWEzUwYFZdLSBN/U l0AOB7ac3yLtR7W4VPrCtkJED3IlzYqAxbe1a2blKfadwAecuWuS4bIwM4qFwNgmemna pJDySk9RFZQDUeIMr8Ool7X3IDQ8dRtFVw90SF80YSvn+nqZa/oDMMyca1r/rU39XObe xO0zPieq9uXVJQ/nFqXNo3h0vdXCHMZp8fCcWhd/4YmfaXBsDc8aV5Mfw9AJ3GWtdrWZ bQgCMNVHPymY5FmEek5E2kjfEbEEdPk1yGtR7MUiMlZRZduMYjkuphJhjL3e2XVvh4/w OOiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature; bh=Ux1rR2xkBloFKcutvqBFIY+BHfGYUefIOvfsD6OV/kE=; b=e+VwLX5fWnsIBY8gM455CbwAD5QfyI0SEHPGDnV2rF0D4K0IsEBAK/lfEMeeFB11+H YsgX73StT79Rlve+8AytsjZWocoLXfB0GneScvvIZq+6BqTZo01sRr7N5Sb2bpJcxECO DxG1UaXladEaURvfFF+eT6u0je9w9K8ZqvEUzm6oLIJFQeFNFILZENIbreorlx/ti/dP MDFKXi1haxTPCVIxqDFncBiVZdxG29TUfcJ6phadrOlrnyJ+v7yJf6PH2BfH59XpFR2C L68uqqGIqdRPc6rstS/J64Cr1RQFjVaAQJ9G3/hWDVDD3eyRmybbhhiuWFvLC9DyFNIN xZ6A== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b="DPTH/FW3"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id m19-v6si1478139qkg.357.2018.10.05.11.05.53 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 05 Oct 2018 11:05:54 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b="DPTH/FW3"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36402 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g8UU5-00043p-BX for patch@linaro.org; Fri, 05 Oct 2018 14:05:53 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:35528) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g8UIl-00017J-Hx for qemu-devel@nongnu.org; Fri, 05 Oct 2018 13:54:12 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g8UIk-0003Kg-77 for qemu-devel@nongnu.org; Fri, 05 Oct 2018 13:54:11 -0400 Received: from mail-oi1-x244.google.com ([2607:f8b0:4864:20::244]:37995) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1g8UIj-0003KQ-Us for qemu-devel@nongnu.org; Fri, 05 Oct 2018 13:54:10 -0400 Received: by mail-oi1-x244.google.com with SMTP id u197-v6so11093596oif.5 for ; Fri, 05 Oct 2018 10:54:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Ux1rR2xkBloFKcutvqBFIY+BHfGYUefIOvfsD6OV/kE=; b=DPTH/FW3tQPiWgtrjQ3maGkyuyHKWOlEWVeHYd/xkBWwC1eH6BP1fPd8R6MymttGnC hMfhWGsMxG3b5AKwyPCH/8jpD1KtpTMiJ5t6tId9TCGaAwY1efjYToy406cStRKmnU1M 1eikETiTrAuB38M1HRE1LaRqGtGZaJPcH/kro= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Ux1rR2xkBloFKcutvqBFIY+BHfGYUefIOvfsD6OV/kE=; b=D5d53c0cUHv+47JyQiqtm1O05tGRnT6RrdSKi9EjNhkujyBUbfpPMym+xdbzC5IgR1 CnsRAewIQ+LomEvG+qe7wlIkAJsJvgXwffrGj1iicLI/jcymRDnUfT+nAwij60rVGRgs tn00OIMmr0RevgLMVgRfJosFEf211RJMBrS5Lc2kLRBWUxEA5L1mR15U4fXzAvwEJxsM ojhT3OwgcEohkHkx70fRIZCPjn4FPBovhPCofUn8yol69CqdrOus/ZxPohDi/I2mxf92 8FZHcN1iF6FJljn8w00D8FvaJz649QPltESogbLTmRwrG7M3+uE3KQZrrArJI+nsj8ZJ YxRQ== X-Gm-Message-State: ABuFfojH32/xUKQUuB6Xw7t/gQUc4XESOIA7OClCPginm6Gbynov1c5i WdY4OxpspEoIiSx8NDOyy8/TiEyjQCC7PbwGpTU= X-Received: by 2002:aca:d841:: with SMTP id p62-v6mr26067oig.118.1538762048781; Fri, 05 Oct 2018 10:54:08 -0700 (PDT) Received: from cloudburst.twiddle.net ([187.217.230.84]) by smtp.gmail.com with ESMTPSA id q10-v6sm2672278otg.31.2018.10.05.10.54.07 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 05 Oct 2018 10:54:07 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 5 Oct 2018 12:53:43 -0500 Message-Id: <20181005175350.30752-9-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181005175350.30752-1-richard.henderson@linaro.org> References: <20181005175350.30752-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::244 Subject: [Qemu-devel] [PATCH v3 08/15] target/arm: Rewrite helper_sve_ld[234]*_r X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Use the same *_tlb primitives as we use for ld1. For linux-user, this hoists the set of helper_retaddr. For softmmu, hoists the computation of the current mmu_idx outside the loop, fixes the endianness problem, and moves the main loop out of a macro and into an inlined function. Reviewed-by: Peter Maydell Tested-by: Laurent Desnogues Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 210 ++++++++++++++++++++++------------------ 1 file changed, 117 insertions(+), 93 deletions(-) -- 2.17.1 diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index d628978431..f712b382f8 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4285,109 +4285,133 @@ DO_LD1_2(ld1dd, 3, 3) #undef DO_LD1_1 #undef DO_LD1_2 -#define DO_LD2(NAME, FN, TYPEE, TYPEM, H) \ -void HELPER(NAME)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ -{ \ - intptr_t i, oprsz = simd_oprsz(desc); \ - intptr_t ra = GETPC(); \ - unsigned rd = simd_data(desc); \ - void *d1 = &env->vfp.zregs[rd]; \ - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ - for (i = 0; i < oprsz; ) { \ - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ - do { \ - TYPEM m1 = 0, m2 = 0; \ - if (pg & 1) { \ - m1 = FN(env, addr, ra); \ - m2 = FN(env, addr + sizeof(TYPEM), ra); \ - } \ - *(TYPEE *)(d1 + H(i)) = m1; \ - *(TYPEE *)(d2 + H(i)) = m2; \ - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ - addr += 2 * sizeof(TYPEM); \ - } while (i & 15); \ - } \ +/* + * Common helpers for all contiguous 2,3,4-register predicated loads. + */ +static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, + uint32_t desc, int size, uintptr_t ra, + sve_ld1_tlb_fn *tlb_fn) +{ + const int mmu_idx = cpu_mmu_index(env, false); + intptr_t i, oprsz = simd_oprsz(desc); + unsigned rd = simd_data(desc); + ARMVectorReg scratch[2] = { }; + + set_helper_retaddr(ra); + for (i = 0; i < oprsz; ) { + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + do { + if (pg & 1) { + tlb_fn(env, &scratch[0], i, addr, mmu_idx, ra); + tlb_fn(env, &scratch[1], i, addr + size, mmu_idx, ra); + } + i += size, pg >>= size; + addr += 2 * size; + } while (i & 15); + } + set_helper_retaddr(0); + + /* Wait until all exceptions have been raised to write back. */ + memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); + memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); } -#define DO_LD3(NAME, FN, TYPEE, TYPEM, H) \ -void HELPER(NAME)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ -{ \ - intptr_t i, oprsz = simd_oprsz(desc); \ - intptr_t ra = GETPC(); \ - unsigned rd = simd_data(desc); \ - void *d1 = &env->vfp.zregs[rd]; \ - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ - void *d3 = &env->vfp.zregs[(rd + 2) & 31]; \ - for (i = 0; i < oprsz; ) { \ - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ - do { \ - TYPEM m1 = 0, m2 = 0, m3 = 0; \ - if (pg & 1) { \ - m1 = FN(env, addr, ra); \ - m2 = FN(env, addr + sizeof(TYPEM), ra); \ - m3 = FN(env, addr + 2 * sizeof(TYPEM), ra); \ - } \ - *(TYPEE *)(d1 + H(i)) = m1; \ - *(TYPEE *)(d2 + H(i)) = m2; \ - *(TYPEE *)(d3 + H(i)) = m3; \ - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ - addr += 3 * sizeof(TYPEM); \ - } while (i & 15); \ - } \ +static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, + uint32_t desc, int size, uintptr_t ra, + sve_ld1_tlb_fn *tlb_fn) +{ + const int mmu_idx = cpu_mmu_index(env, false); + intptr_t i, oprsz = simd_oprsz(desc); + unsigned rd = simd_data(desc); + ARMVectorReg scratch[3] = { }; + + set_helper_retaddr(ra); + for (i = 0; i < oprsz; ) { + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + do { + if (pg & 1) { + tlb_fn(env, &scratch[0], i, addr, mmu_idx, ra); + tlb_fn(env, &scratch[1], i, addr + size, mmu_idx, ra); + tlb_fn(env, &scratch[2], i, addr + 2 * size, mmu_idx, ra); + } + i += size, pg >>= size; + addr += 3 * size; + } while (i & 15); + } + set_helper_retaddr(0); + + /* Wait until all exceptions have been raised to write back. */ + memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); + memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); + memcpy(&env->vfp.zregs[(rd + 2) & 31], &scratch[2], oprsz); } -#define DO_LD4(NAME, FN, TYPEE, TYPEM, H) \ -void HELPER(NAME)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ -{ \ - intptr_t i, oprsz = simd_oprsz(desc); \ - intptr_t ra = GETPC(); \ - unsigned rd = simd_data(desc); \ - void *d1 = &env->vfp.zregs[rd]; \ - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ - void *d3 = &env->vfp.zregs[(rd + 2) & 31]; \ - void *d4 = &env->vfp.zregs[(rd + 3) & 31]; \ - for (i = 0; i < oprsz; ) { \ - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ - do { \ - TYPEM m1 = 0, m2 = 0, m3 = 0, m4 = 0; \ - if (pg & 1) { \ - m1 = FN(env, addr, ra); \ - m2 = FN(env, addr + sizeof(TYPEM), ra); \ - m3 = FN(env, addr + 2 * sizeof(TYPEM), ra); \ - m4 = FN(env, addr + 3 * sizeof(TYPEM), ra); \ - } \ - *(TYPEE *)(d1 + H(i)) = m1; \ - *(TYPEE *)(d2 + H(i)) = m2; \ - *(TYPEE *)(d3 + H(i)) = m3; \ - *(TYPEE *)(d4 + H(i)) = m4; \ - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ - addr += 4 * sizeof(TYPEM); \ - } while (i & 15); \ - } \ +static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, + uint32_t desc, int size, uintptr_t ra, + sve_ld1_tlb_fn *tlb_fn) +{ + const int mmu_idx = cpu_mmu_index(env, false); + intptr_t i, oprsz = simd_oprsz(desc); + unsigned rd = simd_data(desc); + ARMVectorReg scratch[4] = { }; + + set_helper_retaddr(ra); + for (i = 0; i < oprsz; ) { + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + do { + if (pg & 1) { + tlb_fn(env, &scratch[0], i, addr, mmu_idx, ra); + tlb_fn(env, &scratch[1], i, addr + size, mmu_idx, ra); + tlb_fn(env, &scratch[2], i, addr + 2 * size, mmu_idx, ra); + tlb_fn(env, &scratch[3], i, addr + 3 * size, mmu_idx, ra); + } + i += size, pg >>= size; + addr += 4 * size; + } while (i & 15); + } + set_helper_retaddr(0); + + /* Wait until all exceptions have been raised to write back. */ + memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); + memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); + memcpy(&env->vfp.zregs[(rd + 2) & 31], &scratch[2], oprsz); + memcpy(&env->vfp.zregs[(rd + 3) & 31], &scratch[3], oprsz); } -DO_LD2(sve_ld2bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) -DO_LD3(sve_ld3bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) -DO_LD4(sve_ld4bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) +#define DO_LDN_1(N) \ +void __attribute__((flatten)) HELPER(sve_ld##N##bb_r) \ + (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +{ \ + sve_ld##N##_r(env, vg, addr, desc, 1, GETPC(), sve_ld1bb_tlb); \ +} -DO_LD2(sve_ld2hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) -DO_LD3(sve_ld3hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) -DO_LD4(sve_ld4hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) +#define DO_LDN_2(N, SUFF, SIZE) \ +void __attribute__((flatten)) HELPER(sve_ld##N##SUFF##_r) \ + (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +{ \ + sve_ld##N##_r(env, vg, addr, desc, SIZE, GETPC(), \ + arm_cpu_data_is_big_endian(env) \ + ? sve_ld1##SUFF##_be_tlb : sve_ld1##SUFF##_le_tlb); \ +} -DO_LD2(sve_ld2ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) -DO_LD3(sve_ld3ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) -DO_LD4(sve_ld4ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) +DO_LDN_1(2) +DO_LDN_1(3) +DO_LDN_1(4) -DO_LD2(sve_ld2dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) -DO_LD3(sve_ld3dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) -DO_LD4(sve_ld4dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) +DO_LDN_2(2, hh, 2) +DO_LDN_2(3, hh, 2) +DO_LDN_2(4, hh, 2) -#undef DO_LD2 -#undef DO_LD3 -#undef DO_LD4 +DO_LDN_2(2, ss, 4) +DO_LDN_2(3, ss, 4) +DO_LDN_2(4, ss, 4) + +DO_LDN_2(2, dd, 8) +DO_LDN_2(3, dd, 8) +DO_LDN_2(4, dd, 8) + +#undef DO_LDN_1 +#undef DO_LDN_2 /* * Load contiguous data, first-fault and no-fault.