From patchwork Thu Jul 13 21:14:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702240 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp742190wrj; Thu, 13 Jul 2023 14:18:23 -0700 (PDT) X-Google-Smtp-Source: APBJJlG2o+2zn5iA+7gxOEitgMqtsgDWTdB5KLNWT0i5yKHmS+jyVSRd88aEhtiM9HFWSIAuL3M/ X-Received: by 2002:a05:620a:4555:b0:762:4e2:847b with SMTP id u21-20020a05620a455500b0076204e2847bmr3668705qkp.54.1689283103200; Thu, 13 Jul 2023 14:18:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689283103; cv=none; d=google.com; s=arc-20160816; b=kq8Vagr7BKCfCwJ8rBnKLbrpC1PyvD/4LHUTxnfye2l0etpD7SVQV+tikvJSEDQxH0 XH8JmN0PfE6iF9xNi4UdHMg804pC6AfTzUU6HsCZt15OXVVn4LWaCg7KTQe4zWxmJv32 z80gf6LwqVssQDVlgmsCkO9yfZwF41aEHeWOfffsUQavDsxlBJBpeG7+ZrH7Apq4VNkX AIaIzArSGgx5fyvm2udyE68FzrTZ8mgueDZEHvpy7TkvYhgWjlALgeozwHMl2Y4gv92y lSjC8C3uMpcy14stHdhjH3bY5y7LBdXKrRGVbRbslA9inpTZrUJ1paFAzv1u21dT0Dlp 2+eA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=l0ncNfWBAGzw2Z/9Y0q3EwyOwTxxOZI+c9r6426RCJU=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=nLUOa3/QzkR6aMsR8H8GAOzx3tZxgOUipeMiCLIk6stT1UBIqfF/f/uCGKJHc22MNa 2/a98DJcvfIg6jEZa+eK5LhvtV4bC8IMgTIAj3gFbzf7Vu/pbKUI5V9zAgLfrH8MphQy gJKdKrnxtjD5StlgNdBP4b865DTLAxgRui4BSl2uhdujnjbsZ5qSfY6sKGDFDLTsX/mG EX/W+xmiHU2VFnbtM+amBnCzF1y+lFBLVpWN5LiAqnaeYMSQeef6GlZ+KmbXYWHUV/Av 9TUQBJ+miMl8jdSgiIMZ9bL87+wGPBKGCMooGdhwkVMSmDbLpAdxXmdx8xN5WtJuMuQX Wfsw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=BsC33UBa; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id e29-20020a05620a209d00b00763ad5556eesi3394633qka.780.2023.07.13.14.18.23 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:18:23 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=BsC33UBa; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3eZ-0000PW-71; Thu, 13 Jul 2023 17:15:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eK-0000JL-Q7 for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:56 -0400 Received: from mail-wr1-x429.google.com ([2a00:1450:4864:20::429]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3eA-00015n-Ee for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:56 -0400 Received: by mail-wr1-x429.google.com with SMTP id ffacd0b85a97d-314172bac25so1314823f8f.3 for ; Thu, 13 Jul 2023 14:14:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282883; x=1691874883; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=l0ncNfWBAGzw2Z/9Y0q3EwyOwTxxOZI+c9r6426RCJU=; b=BsC33UBaPd62v8wYZAzibIOrEqPjHaemGFzIHaluNLbbFA7PoVDhbQLxiIoP4ZKeGN a5d+shms0krmastJj2MsGiqJccH64HeMgkmg/r/L4o/cysFOuDS3DiSXy/l5IAIbKCEc c7elm1Z0+GXa43d6h+j9DTHCTX0Hp8NGgXAS1G91HaCVs6jpBE6n6j1LNRnPs0DNFxV2 eDgobUC7ePWtXLhgyPwcimFTBPSgs56NOQ7WcClekv+EybQsRZ8AJ9gdkDRJevUqgMqZ qFzvMQDPIESuLqs12gjxMRCe/os6+Zf6TvfftQwB0kGNwFxN8pVNuPPfZgZP3MLdKXkn TRmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282883; x=1691874883; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l0ncNfWBAGzw2Z/9Y0q3EwyOwTxxOZI+c9r6426RCJU=; b=ZJSbULchtjy0ow+2aqVNRlaWAXlJ7NUyUSyyNmPxTHfRfdgBrmy0JEVMs+0vl69uBo quAI5grjm+4uPM4h+0aH1yEJleiIelQb2Jl9aeqOcrR6zaNA1/cEmbu740g35DxC1lx9 GMxmW2W8Bo7kaEwRnTzR5xD5zjRl30LAYpKqgqLd03rbpcSx/jEjyKm8oYnX0TeXb6zg dphxSxhL9tvZCjWVvcJiY0G9tRw9ANXeLLdFry1zcLYhKnVE/7W9bbXD4ZqlIZNGkqeX TsoMevdwHNjsztIrs6hdDIbVCnwtBbA59zghXOWMFdmGRM5rYvXrLy+p1q605qrNi6/P 9EZg== X-Gm-Message-State: ABy/qLYNRryCFC9prB87Hf8vsl0FOJhqpNb0/8ZBwOmZKwEMfUjWJIk5 yP49UvXN2XxOIqHTlg6DQkjcy4UuJhmsdfjOQPWGFXJb X-Received: by 2002:adf:ecc9:0:b0:314:182a:3d96 with SMTP id s9-20020adfecc9000000b00314182a3d96mr2662392wro.28.1689282883414; Thu, 13 Jul 2023 14:14:43 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:43 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 01/18] crypto: Add generic 8-bit carry-less multiply routines Date: Thu, 13 Jul 2023 22:14:18 +0100 Message-Id: <20230713211435.13505-2-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::429; envelope-from=richard.henderson@linaro.org; helo=mail-wr1-x429.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Signed-off-by: Richard Henderson --- host/include/generic/host/crypto/clmul.h | 17 ++++++ include/crypto/clmul.h | 61 +++++++++++++++++++ crypto/clmul.c | 76 ++++++++++++++++++++++++ crypto/meson.build | 9 ++- 4 files changed, 160 insertions(+), 3 deletions(-) create mode 100644 host/include/generic/host/crypto/clmul.h create mode 100644 include/crypto/clmul.h create mode 100644 crypto/clmul.c diff --git a/host/include/generic/host/crypto/clmul.h b/host/include/generic/host/crypto/clmul.h new file mode 100644 index 0000000000..694705f703 --- /dev/null +++ b/host/include/generic/host/crypto/clmul.h @@ -0,0 +1,17 @@ +/* + * No host specific carry-less multiply acceleration. + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#ifndef GENERIC_HOST_CRYPTO_CLMUL_H +#define GENERIC_HOST_CRYPTO_CLMUL_H + +/* Defer everything to the generic routines. */ +#define clmul_8x8_low clmul_8x8_low_gen +#define clmul_8x4_even clmul_8x4_even_gen +#define clmul_8x4_odd clmul_8x4_odd_gen +#define clmul_8x8_even clmul_8x8_even_gen +#define clmul_8x8_odd clmul_8x8_odd_gen +#define clmul_8x8_packed clmul_8x8_packed_gen + +#endif /* GENERIC_HOST_CRYPTO_CLMUL_H */ diff --git a/include/crypto/clmul.h b/include/crypto/clmul.h new file mode 100644 index 0000000000..7f19205d6f --- /dev/null +++ b/include/crypto/clmul.h @@ -0,0 +1,61 @@ +/* + * Carry-less multiply + * SPDX-License-Identifier: GPL-2.0-or-later + * + * Copyright (C) 2023 Linaro, Ltd. + */ + +#ifndef CRYPTO_CLMUL_H +#define CRYPTO_CLMUL_H + +#include "qemu/int128.h" + +/** + * clmul_8x8_low: + * + * Perform eight 8x8->8 carry-less multiplies. + */ +uint64_t clmul_8x8_low_gen(uint64_t, uint64_t); + +/** + * clmul_8x4_even: + * + * Perform four 8x8->16 carry-less multiplies. + * The odd bytes of the inputs are ignored. + */ +uint64_t clmul_8x4_even_gen(uint64_t, uint64_t); + +/** + * clmul_8x4_odd: + * + * Perform four 8x8->16 carry-less multiplies. + * The even bytes of the inputs are ignored. + */ +uint64_t clmul_8x4_odd_gen(uint64_t, uint64_t); + +/** + * clmul_8x8_even: + * + * Perform eight 8x8->16 carry-less multiplies. + * The odd bytes of the inputs are ignored. + */ +Int128 clmul_8x8_even_gen(Int128, Int128); + +/** + * clmul_8x8_odd: + * + * Perform eight 8x8->16 carry-less multiplies. + * The even bytes of the inputs are ignored. + */ +Int128 clmul_8x8_odd_gen(Int128, Int128); + +/** + * clmul_8x8_packed: + * + * Perform eight 8x8->16 carry-less multiplies. + */ +Int128 clmul_8x8_packed_gen(uint64_t, uint64_t); + +#include "host/crypto/clmul.h" + +#endif /* CRYPTO_CLMUL_H */ diff --git a/crypto/clmul.c b/crypto/clmul.c new file mode 100644 index 0000000000..866704e751 --- /dev/null +++ b/crypto/clmul.c @@ -0,0 +1,76 @@ +/* + * No host specific carry-less multiply acceleration. + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include "qemu/osdep.h" +#include "crypto/clmul.h" + +uint64_t clmul_8x8_low_gen(uint64_t n, uint64_t m) +{ + uint64_t r = 0; + + for (int i = 0; i < 8; ++i) { + uint64_t mask = (n & 0x0101010101010101ull) * 0xff; + r ^= m & mask; + m = (m << 1) & 0xfefefefefefefefeull; + n >>= 1; + } + return r; +} + +uint64_t clmul_8x4_even_gen(uint64_t n, uint64_t m) +{ + uint64_t r = 0; + + n &= 0x00ff00ff00ff00ffull; + m &= 0x00ff00ff00ff00ffull; + + for (int i = 0; i < 8; ++i) { + uint64_t mask = (n & 0x0001000100010001ull) * 0xffff; + r ^= m & mask; + n >>= 1; + m <<= 1; + } + return r; +} + +uint64_t clmul_8x4_odd_gen(uint64_t n, uint64_t m) +{ + return clmul_8x4_even_gen(n >> 8, m >> 8); +} + +Int128 clmul_8x8_even_gen(Int128 n, Int128 m) +{ + uint64_t rl, rh; + + rl = clmul_8x4_even_gen(int128_getlo(n), int128_getlo(m)); + rh = clmul_8x4_even_gen(int128_gethi(n), int128_gethi(m)); + return int128_make128(rl, rh); +} + +Int128 clmul_8x8_odd_gen(Int128 n, Int128 m) +{ + uint64_t rl, rh; + + rl = clmul_8x4_odd_gen(int128_getlo(n), int128_getlo(m)); + rh = clmul_8x4_odd_gen(int128_gethi(n), int128_gethi(m)); + return int128_make128(rl, rh); +} + +static uint64_t unpack_8_to_16(uint64_t x) +{ + return (x & 0x000000ff) + | ((x & 0x0000ff00) << 8) + | ((x & 0x00ff0000) << 16) + | ((x & 0xff000000) << 24); +} + +Int128 clmul_8x8_packed_gen(uint64_t n, uint64_t m) +{ + uint64_t rl, rh; + + rl = clmul_8x4_even_gen(unpack_8_to_16(n), unpack_8_to_16(m)); + rh = clmul_8x4_even_gen(unpack_8_to_16(n >> 32), unpack_8_to_16(m >> 32)); + return int128_make128(rl, rh); +} diff --git a/crypto/meson.build b/crypto/meson.build index 5f03a30d34..9ac1a89802 100644 --- a/crypto/meson.build +++ b/crypto/meson.build @@ -48,9 +48,12 @@ if have_afalg endif crypto_ss.add(when: gnutls, if_true: files('tls-cipher-suites.c')) -util_ss.add(files('sm4.c')) -util_ss.add(files('aes.c')) -util_ss.add(files('init.c')) +util_ss.add(files( + 'aes.c', + 'clmul.c', + 'init.c', + 'sm4.c', +)) if gnutls.found() util_ss.add(gnutls) endif From patchwork Thu Jul 13 21:14:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702232 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp740953wrj; Thu, 13 Jul 2023 14:16:11 -0700 (PDT) X-Google-Smtp-Source: APBJJlEIrqkWPH9mMbE7ZaUlPvCn3syhy60CgJldS5gPZEETjUs39S+9KLZIIcoDlhHCPY4F+IEM X-Received: by 2002:a37:f708:0:b0:766:f9b8:5602 with SMTP id q8-20020a37f708000000b00766f9b85602mr2398824qkj.33.1689282971421; Thu, 13 Jul 2023 14:16:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689282971; cv=none; d=google.com; s=arc-20160816; b=uEoUDcp/cvs2g09e1YbV4176iluGDpS4bVRAh5D1FpFNROAvkZzGz5RQgZk948K1dl wEns0XLCxc/bykrOva1W3h+dsD9Iir9RGbUeWBSGww/zHWIZ5mCxXcpR3KhYwic0KHgl HIoaqbMZrXA9SHExUinr49NEjna50rLqRLOI3bNaiNxSZQTL5BerSJKglwgKmQGfuVq4 Sp32ONJoagzzQyKPuec0/LFqBSLwV2SBhbt47vB4gubaD9rHRpFwce5MSXA1kx2GTUXy DMx8hdniPkmhsD+H6R6j4XpP75OF71JuCmM4ZhLq5yTVNTk1SoR3SFcO3Hy7wBsnPEDN zDxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=ECUmLrNCqZ555vv9lbCs7tRpJNSGqq32nj0uXtVER3w=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=OIxyL7iyaz+Bz8ndIAUjp+gBd4ysGN9E+/sexDDiUqONkur/8jMwhEqYY9iuglk6Jh DjO3slohbN8a5XvdqZC41/Jy2X+kno8Kyyt4ZhEq4n61Gj4+46s6mQd7KwE8ww8ywIGn avbdCe0FkVTpg2hNbilx8ZFJtPqWQf4Hu35I3td4YdvIaaaKJ9iwUbESMDPwGKSoZXFd trwfxmvm3dEgzxKPb9YVNfVcHR+9gdN8O5DEonDI07gCEp8y/AC2+fP+IOHp0SGaKigT WtXb8/RM1mwVZ27/mTLTMVslLc9KwShvJe982pxzPRWyuJZCIY/NoQDQNQadLuHCnj97 nlzg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Hv8oVkwO; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id 16-20020a05620a049000b00767ee53f0absi2252487qkr.340.2023.07.13.14.16.11 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:16:11 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Hv8oVkwO; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3eJ-0000Ik-3V; Thu, 13 Jul 2023 17:14:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eF-0000HU-8U for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:51 -0400 Received: from mail-wr1-x434.google.com ([2a00:1450:4864:20::434]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3e9-00015r-Od for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:50 -0400 Received: by mail-wr1-x434.google.com with SMTP id ffacd0b85a97d-3143b72c5ffso1381132f8f.3 for ; Thu, 13 Jul 2023 14:14:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282884; x=1691874884; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ECUmLrNCqZ555vv9lbCs7tRpJNSGqq32nj0uXtVER3w=; b=Hv8oVkwOKaCPvjob4KH39+wO4CMkpG4G7b2cLB/nEgUopWSn1CxrY5NVHuvn88iI3e hLuILbBpANTIw/Ro7RXzM0hXQI3MKOF3b8O6QWSN5BrkLgc4VM48uTQEFWmi01iodu0L NSiXETdZ4CgE8TTDO2+/gjGD7XvYvltedD0BTYKZpRz+HaMt7AGeKHbIO7kkGwymm2tR ClCLyNutfy9yZ4g50p74OfnvFmMGEnqDrxWMCY7LZ6PUze1yYrWZNRVNpPtLtqAz/Et/ byqST9UbISZ7yzBnPRK1FZHveop3nLFoK3mfMuOC37i0LdDYSJ/Htk0FpCv0jVWsMMaQ w5kQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282884; x=1691874884; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ECUmLrNCqZ555vv9lbCs7tRpJNSGqq32nj0uXtVER3w=; b=Ezp5axgzFdVy6+qlAIITqfnpb8Fjw26a87bq4w73sZC+yWTvfOUdxrd2JMyWrhVQPt fr1Hv2mIk88/4xSc4qVXU8T+O56TPKdYee36WUHD43JWHJC2TEt7wcOe9Jqbf/FiCHZ+ 5tyGam0xfF5O+P7Tvj+QuAQaSX26b3whXeA9GerRss30HTuZfWNdf1d7Z534DggUdVa7 6eoPutoBeAZxVciQMNod6aDNloe/Xce2MdedjNG+aT8ZHtUpGkisfEescbYgmvxYpkZr ok1jUmbKCktmRYxVdvsgQqxGQUBeN+FuYo4GtFms9lImtNPq7CdC0z80fCaW3W9YXIfW ohfg== X-Gm-Message-State: ABy/qLaCxRcEvSQU1IvwCAWehyFGW8nMynWUxRM0IXFtfGX82zfAo5yR t3Qro2zJoyZeKa1dTRiirOq5XXu6JMq37ouQDU9oBuZu X-Received: by 2002:adf:cf11:0:b0:313:e520:936f with SMTP id o17-20020adfcf11000000b00313e520936fmr2864031wrj.17.1689282884324; Thu, 13 Jul 2023 14:14:44 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:43 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 02/18] target/arm: Use clmul_8* routines Date: Thu, 13 Jul 2023 22:14:19 +0100 Message-Id: <20230713211435.13505-3-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::434; envelope-from=richard.henderson@linaro.org; helo=mail-wr1-x434.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, T_SPF_HELO_TEMPERROR=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Use generic routines for 8-bit carry-less multiply. Remove our local version of pmull_h. Signed-off-by: Richard Henderson Reviewed-by: Philippe Mathieu-Daudé --- target/arm/tcg/vec_internal.h | 5 --- target/arm/tcg/mve_helper.c | 8 ++--- target/arm/tcg/vec_helper.c | 63 +++++++---------------------------- 3 files changed, 15 insertions(+), 61 deletions(-) diff --git a/target/arm/tcg/vec_internal.h b/target/arm/tcg/vec_internal.h index 1f4ed80ff7..c4afba6d9f 100644 --- a/target/arm/tcg/vec_internal.h +++ b/target/arm/tcg/vec_internal.h @@ -219,11 +219,6 @@ int16_t do_sqrdmlah_h(int16_t, int16_t, int16_t, bool, bool, uint32_t *); int32_t do_sqrdmlah_s(int32_t, int32_t, int32_t, bool, bool, uint32_t *); int64_t do_sqrdmlah_d(int64_t, int64_t, int64_t, bool, bool); -/* - * 8 x 8 -> 16 vector polynomial multiply where the inputs are - * in the low 8 bits of each 16-bit element -*/ -uint64_t pmull_h(uint64_t op1, uint64_t op2); /* * 16 x 16 -> 32 vector polynomial multiply where the inputs are * in the low 16 bits of each 32-bit element diff --git a/target/arm/tcg/mve_helper.c b/target/arm/tcg/mve_helper.c index 403b345ea3..96ddfb4b3a 100644 --- a/target/arm/tcg/mve_helper.c +++ b/target/arm/tcg/mve_helper.c @@ -26,6 +26,7 @@ #include "exec/exec-all.h" #include "tcg/tcg.h" #include "fpu/softfloat.h" +#include "crypto/clmul.h" static uint16_t mve_eci_mask(CPUARMState *env) { @@ -984,15 +985,12 @@ DO_2OP_L(vmulltuw, 1, 4, uint32_t, 8, uint64_t, DO_MUL) * Polynomial multiply. We can always do this generating 64 bits * of the result at a time, so we don't need to use DO_2OP_L. */ -#define VMULLPH_MASK 0x00ff00ff00ff00ffULL #define VMULLPW_MASK 0x0000ffff0000ffffULL -#define DO_VMULLPBH(N, M) pmull_h((N) & VMULLPH_MASK, (M) & VMULLPH_MASK) -#define DO_VMULLPTH(N, M) DO_VMULLPBH((N) >> 8, (M) >> 8) #define DO_VMULLPBW(N, M) pmull_w((N) & VMULLPW_MASK, (M) & VMULLPW_MASK) #define DO_VMULLPTW(N, M) DO_VMULLPBW((N) >> 16, (M) >> 16) -DO_2OP(vmullpbh, 8, uint64_t, DO_VMULLPBH) -DO_2OP(vmullpth, 8, uint64_t, DO_VMULLPTH) +DO_2OP(vmullpbh, 8, uint64_t, clmul_8x4_even) +DO_2OP(vmullpth, 8, uint64_t, clmul_8x4_odd) DO_2OP(vmullpbw, 8, uint64_t, DO_VMULLPBW) DO_2OP(vmullptw, 8, uint64_t, DO_VMULLPTW) diff --git a/target/arm/tcg/vec_helper.c b/target/arm/tcg/vec_helper.c index f59d3b26ea..4384b6c188 100644 --- a/target/arm/tcg/vec_helper.c +++ b/target/arm/tcg/vec_helper.c @@ -23,6 +23,7 @@ #include "tcg/tcg-gvec-desc.h" #include "fpu/softfloat.h" #include "qemu/int128.h" +#include "crypto/clmul.h" #include "vec_internal.h" /* @@ -1986,21 +1987,11 @@ void HELPER(gvec_ushl_h)(void *vd, void *vn, void *vm, uint32_t desc) */ void HELPER(gvec_pmul_b)(void *vd, void *vn, void *vm, uint32_t desc) { - intptr_t i, j, opr_sz = simd_oprsz(desc); + intptr_t i, opr_sz = simd_oprsz(desc); uint64_t *d = vd, *n = vn, *m = vm; for (i = 0; i < opr_sz / 8; ++i) { - uint64_t nn = n[i]; - uint64_t mm = m[i]; - uint64_t rr = 0; - - for (j = 0; j < 8; ++j) { - uint64_t mask = (nn & 0x0101010101010101ull) * 0xff; - rr ^= mm & mask; - mm = (mm << 1) & 0xfefefefefefefefeull; - nn >>= 1; - } - d[i] = rr; + d[i] = clmul_8x8_low(n[i], m[i]); } clear_tail(d, opr_sz, simd_maxsz(desc)); } @@ -2038,22 +2029,6 @@ void HELPER(gvec_pmull_q)(void *vd, void *vn, void *vm, uint32_t desc) clear_tail(d, opr_sz, simd_maxsz(desc)); } -/* - * 8x8->16 polynomial multiply. - * - * The byte inputs are expanded to (or extracted from) half-words. - * Note that neon and sve2 get the inputs from different positions. - * This allows 4 bytes to be processed in parallel with uint64_t. - */ - -static uint64_t expand_byte_to_half(uint64_t x) -{ - return (x & 0x000000ff) - | ((x & 0x0000ff00) << 8) - | ((x & 0x00ff0000) << 16) - | ((x & 0xff000000) << 24); -} - uint64_t pmull_w(uint64_t op1, uint64_t op2) { uint64_t result = 0; @@ -2067,30 +2042,14 @@ uint64_t pmull_w(uint64_t op1, uint64_t op2) return result; } -uint64_t pmull_h(uint64_t op1, uint64_t op2) -{ - uint64_t result = 0; - int i; - for (i = 0; i < 8; ++i) { - uint64_t mask = (op1 & 0x0001000100010001ull) * 0xffff; - result ^= op2 & mask; - op1 >>= 1; - op2 <<= 1; - } - return result; -} - void HELPER(neon_pmull_h)(void *vd, void *vn, void *vm, uint32_t desc) { int hi = simd_data(desc); uint64_t *d = vd, *n = vn, *m = vm; - uint64_t nn = n[hi], mm = m[hi]; - - d[0] = pmull_h(expand_byte_to_half(nn), expand_byte_to_half(mm)); - nn >>= 32; - mm >>= 32; - d[1] = pmull_h(expand_byte_to_half(nn), expand_byte_to_half(mm)); + Int128 r = clmul_8x8_packed(n[hi], m[hi]); + d[0] = int128_getlo(r); + d[1] = int128_gethi(r); clear_tail(d, 16, simd_maxsz(desc)); } @@ -2101,11 +2060,13 @@ void HELPER(sve2_pmull_h)(void *vd, void *vn, void *vm, uint32_t desc) intptr_t i, opr_sz = simd_oprsz(desc); uint64_t *d = vd, *n = vn, *m = vm; - for (i = 0; i < opr_sz / 8; ++i) { - uint64_t nn = (n[i] >> shift) & 0x00ff00ff00ff00ffull; - uint64_t mm = (m[i] >> shift) & 0x00ff00ff00ff00ffull; + for (i = 0; i < opr_sz / 8; i += 2) { + Int128 nn = int128_make128(n[i] >> shift, n[i + 1] >> shift); + Int128 mm = int128_make128(m[i] >> shift, m[i + 1] >> shift); + Int128 r = clmul_8x8_even(nn, mm); - d[i] = pmull_h(nn, mm); + d[0] = int128_getlo(r); + d[1] = int128_gethi(r); } } From patchwork Thu Jul 13 21:14:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702235 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp741494wrj; Thu, 13 Jul 2023 14:16:54 -0700 (PDT) X-Google-Smtp-Source: APBJJlHIJL8VA8+ou7VSx3Hs3Godxo1NzeuLP3CAmws8+hRfmuSH+xy4UgIVmk4Uv2QydtSVsZRg X-Received: by 2002:a05:620a:4728:b0:767:dc46:eb6c with SMTP id bs40-20020a05620a472800b00767dc46eb6cmr1369346qkb.0.1689283014689; Thu, 13 Jul 2023 14:16:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689283014; cv=none; d=google.com; s=arc-20160816; b=gVcJgrhFmplOpAYR+KoIGny9MjetIgddR5cewA4Jv8EDA+IyQjFIc8YNjwc5RUTVeh RJnPZcgDsRxuBMH3yAv4XM72ep1/pvbLzP2260oI+EO/aQQGEu9BiC+Jafv6s2aYPEZ+ mWid/0V6SwUTsPTL0Nu7RrnbtdBCdiXiudChwzoBKMF6eLtd8pCDC4bLeSe5a5LawVgZ KBVBAx/UqG2Ziu3aiedfZgjcwoHtlTd+rGDTb3HNw0S9F0ie6+ENpwTxDi/SbSq4CXLs Zd563iHIgQywpQIQqlYMZXOvu5cWCXvUt8MR9J8LRhc+X8paTFHws+uPYy4Hme9oOPdL 48RQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=H93abcQMfpcVmKf+SmjckREj83SkP9I5kccuQ+f4FCc=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=JkQxr87kHpTFogySkNGtK5dlqqdDvpLtSvvvwF+2qez+/24/yGV4KLFT/Dk5e4d222 RVrgfWxLHw33Arq5R18Fx+AX/0WgCDcPEW11BNvirjXq/nULgSCK4+CdNRyEIdi1thv4 zBh/hR5/vYKmuKGJgyLDXTsKQaDAsXMy4tFkvuHppeMpJ4bWv8nvrTsB+uae9cbszK1P 0zpZc2adEr9oLviENIKuCduS79M6+hodNfXbPgmJjI+o277VLvQ59xMe3+jMdU8PjT3p jfEIPJEAZhfdLLkZ/XaybpmSYTetLjCaCzZXAeU5u22OwKlInJn3qkEFcPo6AmJO+7LP E3Lg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=vsG2NBtk; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id v22-20020a05620a441600b00767dc9dddeesi3696256qkp.619.2023.07.13.14.16.54 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:16:54 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=vsG2NBtk; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3eG-0000Hz-WB; Thu, 13 Jul 2023 17:14:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eE-0000HN-Dk for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:50 -0400 Received: from mail-wm1-x331.google.com ([2a00:1450:4864:20::331]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3eB-000164-OC for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:49 -0400 Received: by mail-wm1-x331.google.com with SMTP id 5b1f17b1804b1-3fc04692e20so11243125e9.0 for ; Thu, 13 Jul 2023 14:14:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282885; x=1691874885; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H93abcQMfpcVmKf+SmjckREj83SkP9I5kccuQ+f4FCc=; b=vsG2NBtkBSfDV6qbPry0b/iRTj8SvYNpBk5CQwGPL5EFCbtfu3JWS11JZb5/u0kgTW Xdws6eXl+CV+IIoHhLiD6QPdGvppTYCOIw6zZvPqKtuJok0Ee1Y1LfhtY5YOSFD6OCOy YNqOadonC7Fg0Wp52Ti1F/4q7wmzAdyYAW5s30v2h7eG/Uy9GgF+tUHg6IO7ZA6Cj1vG KRT6btxSJ4nTiAJ/1yPGUoIr0Vyiyu3yO4I2D2k9/SpJWFVI1SPy+WPI9fKTEckOXC+o ewz8V/kCRKsHciyFfI8dARxo+K0ktc0TL0CyuGXT/iQvGJPMcQXhN0eIuMQLxNXyOXQ7 GlAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282885; x=1691874885; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H93abcQMfpcVmKf+SmjckREj83SkP9I5kccuQ+f4FCc=; b=IaWOGogMPftK8MH70dTd65mFSH01vHvUVl/u8Y9W6wVvfQpHwa+jF+iSHZ1zwpMAje y37U9P+YbNbz+uyyQUDguJgn6oAMGbNYf5TezhYAYtH4LLDU3aVzv5a1cnwk6LhTs6MA 2rGcCQkkR1KmZd6d9nN2kDHD/kUa0tYiN79zBBz0zKCgt9FMykTmRCoc+5B+vIbf1r+N 9uZMtfobzz6/+/H39mubm0ezX7kzzMyzk7CtYCfRSDEPogh2w2kM/n4+JxQI/k/ZgrSb q/c0TPFtN8GVmALEXsxvcfOwLp0/+xrkJwQVx1bWyLLKgTVBbCyLFARAIpwyWodgmo2m rSkg== X-Gm-Message-State: ABy/qLZRMpJsVEq7TKQDWAAjqUYAGNqGYF0xtTl62x7+Wk1KZVPLrKpb YEvp9ug31nOAtU5ZbDsRu9bhVFd4Y+euUnM7P9pcKhhe X-Received: by 2002:a5d:5650:0:b0:313:e88d:e6cf with SMTP id j16-20020a5d5650000000b00313e88de6cfmr2256359wrw.69.1689282885052; Thu, 13 Jul 2023 14:14:45 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:44 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 03/18] target/s390x: Use clmul_8* routines Date: Thu, 13 Jul 2023 22:14:20 +0100 Message-Id: <20230713211435.13505-4-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::331; envelope-from=richard.henderson@linaro.org; helo=mail-wm1-x331.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Use generic routines for 8-bit carry-less multiply. Remove our local version of galois_multiply8. Signed-off-by: Richard Henderson --- target/s390x/tcg/vec_int_helper.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/target/s390x/tcg/vec_int_helper.c b/target/s390x/tcg/vec_int_helper.c index 53ab5c5eb3..e110a7581a 100644 --- a/target/s390x/tcg/vec_int_helper.c +++ b/target/s390x/tcg/vec_int_helper.c @@ -14,6 +14,7 @@ #include "vec.h" #include "exec/helper-proto.h" #include "tcg/tcg-gvec-desc.h" +#include "crypto/clmul.h" static bool s390_vec_is_zero(const S390Vector *v) { @@ -179,7 +180,6 @@ static uint##TBITS##_t galois_multiply##BITS(uint##TBITS##_t a, \ } \ return res; \ } -DEF_GALOIS_MULTIPLY(8, 16) DEF_GALOIS_MULTIPLY(16, 32) DEF_GALOIS_MULTIPLY(32, 64) @@ -203,6 +203,29 @@ static S390Vector galois_multiply64(uint64_t a, uint64_t b) return res; } +static Int128 do_gfm8(Int128 n, Int128 m) +{ + Int128 e = clmul_8x8_even(n, m); + Int128 o = clmul_8x8_odd(n, m); + return int128_xor(e, o); +} + +void HELPER(gvec_vgfm8)(void *v1, const void *v2, const void *v3, uint32_t d) +{ + /* + * There is no carry across the two doublewords, so their order + * does not matter, so we need not care for host endianness. + */ + *(Int128 *)v1 = do_gfm8(*(const Int128 *)v2, *(const Int128 *)v3); +} + +void HELPER(gvec_vgfma8)(void *v1, const void *v2, const void *v3, + const void *v4, uint32_t d) +{ + Int128 r = do_gfm8(*(const Int128 *)v2, *(const Int128 *)v3); + *(Int128 *)v1 = int128_xor(r, *(Int128 *)v4); +} + #define DEF_VGFM(BITS, TBITS) \ void HELPER(gvec_vgfm##BITS)(void *v1, const void *v2, const void *v3, \ uint32_t desc) \ @@ -220,7 +243,6 @@ void HELPER(gvec_vgfm##BITS)(void *v1, const void *v2, const void *v3, \ s390_vec_write_element##TBITS(v1, i, d); \ } \ } -DEF_VGFM(8, 16) DEF_VGFM(16, 32) DEF_VGFM(32, 64) @@ -257,7 +279,6 @@ void HELPER(gvec_vgfma##BITS)(void *v1, const void *v2, const void *v3, \ s390_vec_write_element##TBITS(v1, i, d); \ } \ } -DEF_VGFMA(8, 16) DEF_VGFMA(16, 32) DEF_VGFMA(32, 64) From patchwork Thu Jul 13 21:14:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702241 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp742241wrj; Thu, 13 Jul 2023 14:18:30 -0700 (PDT) X-Google-Smtp-Source: APBJJlGFY9Rj/FN9SWjsYoCqxBBxo9ZO8j/PF0HkAdB/681NfcLNyxQj+4Dsk1ILcXXEZOlE9lsh X-Received: by 2002:ac8:7f91:0:b0:403:a6f7:a839 with SMTP id z17-20020ac87f91000000b00403a6f7a839mr3558227qtj.49.1689283110290; Thu, 13 Jul 2023 14:18:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689283110; cv=none; d=google.com; s=arc-20160816; b=KzKndv2EXLsJNhxme29tyU0FCEKXCWCHIyrgkdyMhewuO+k1L+ihta5zqM1z+nAVdo SJLs3HhMT/LraWED20BHiG1mjDP+LYwc+8xk9cDD3n0YM5shPIUJBR4c920ATcwTIDX5 3BquRaXQR8QqJr12p7qlCQW8LwpYAllEAadeYTaaVHH4jN96yNZLVbXEPFuKh4qtGad0 XSH60+DwMPN8rg6OGl779D2MDN+ekKWBGCkjMi4kGkWq8CjapdiuCqn27+Q2olgShuLL eMOaY5DMeMoQmfYtYwirTjeQPGhHoqEddFyNwqvGqlS7zwtn/3zjvDWTdOLVH+PHuzoc fSLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=uiRxUWQ/HM/fmJF0BXMvciSYviZrpuPjzgLczxVKcDA=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=EyZpJAmL934SYut++Vfs4qFjJ1kv8oBbf2yAQQqpH8XUx2bhm1RTrL2WF9rIJEX4Mh SBvJIN6mTlztFBHeVQjpvGPp67b1dXbbmhzc2oSD/sY7efzZN3aEPhMsYwBHnQTvWW2m 0VyVa4JxAQAiQ75EHODIWQNfTlEy49awZ3Oj+u516/6zij+03M0FDlT9MkDS5XOVU8jc mwEuZvQ4NPeAy/JSgFtEVcvtxFD5PO8hGN23nFjC9yRZhmKwQgLEgNVz31b4G9trbOGJ Kz/Ytd37gSAJrZalHwLOv9eZXaw2A6kAoTAGdH9m7RZ/LnQcBwY4YjnKmdicM2iEKDBV awtw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="VlDqWrH/"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id v23-20020a05622a189700b00400a725621esi3677302qtc.508.2023.07.13.14.18.30 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:18:30 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="VlDqWrH/"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3ea-0000QP-Pb; Thu, 13 Jul 2023 17:15:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eQ-0000Mr-7T for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:15:04 -0400 Received: from mail-wr1-x430.google.com ([2a00:1450:4864:20::430]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3eC-00016F-Cu for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:15:01 -0400 Received: by mail-wr1-x430.google.com with SMTP id ffacd0b85a97d-3141c3a7547so1322193f8f.2 for ; Thu, 13 Jul 2023 14:14:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282885; x=1691874885; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uiRxUWQ/HM/fmJF0BXMvciSYviZrpuPjzgLczxVKcDA=; b=VlDqWrH/ORjSn+/ojaVyuYiDoF8hi91RgmPXuZw1PpXzH3X8nfDlYA6T5NEvvDWpiy Jr8EGE9k1e6CNXDLoBsHF91xGt4kvKLwRKfDHHqQsK9bwFFIbySRS+gBzYnsmrlHJRy9 ZYU08Z/9kwbDPOeGGDe3ECNruoQAeACnXGC48DN3+osnWYBR3gm+BOce/xpQ/VqU0Jyj a5j8CYoHiVm4us5EMlcADlZ5KgN4QBwU28RPVd//Xmh7qgDSaqYfewfFLHOJOrNLgT/G 1wVEhU2Is3LKIDmZg7cH11ZqMZdRl7Ursrx1vdSQ+Wk4lcr6bvjS4A9sJ0Mv4mQMboxj ITmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282885; x=1691874885; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uiRxUWQ/HM/fmJF0BXMvciSYviZrpuPjzgLczxVKcDA=; b=K5MhPdGfWqhKcj9X9T/mhUr3OmPQ1dAD/M0TSVXfdnEVCYFuEVwP6nvPBFofFYwo3j ISnUkSNYirQ7iol2HjCP34CGJeb8oyIREePr9kmsUZmP+I8xSLn3JHfMHo/cTcnA73ZJ HfJ+efUQQpJByeVzBMtwWyMhcjQzCAF0KNOESdjqsOD0BNTkm5GfZk7W5Q98jNyW3m3r vUpbuKlh8wt5mt0VIUKWZkboMGmxwPL20aGb3RWgbIr4injCK34QC+h/Uwt7vFzJpJCf faxY7dEeRCRedBuiI0f3MQnvul9px1JElBcFqjXuP4XfYwOAaYq3kcSnw7tP86KCT+sN 8W8g== X-Gm-Message-State: ABy/qLbuJANSXpCTKNwZ4Sd3ziY3ktd5mPovm9YUN0SJfreZd4kr4QqF eZoLPjuUvXbDajiVTOPM5ADkaf1mBQpF9GGe2jN37ZOx X-Received: by 2002:adf:fa52:0:b0:313:f07b:8025 with SMTP id y18-20020adffa52000000b00313f07b8025mr2404490wrr.47.1689282885800; Thu, 13 Jul 2023 14:14:45 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:45 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 04/18] target/ppc: Use clmul_8* routines Date: Thu, 13 Jul 2023 22:14:21 +0100 Message-Id: <20230713211435.13505-5-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::430; envelope-from=richard.henderson@linaro.org; helo=mail-wr1-x430.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Use generic routines for 8-bit carry-less multiply. Signed-off-by: Richard Henderson --- target/ppc/int_helper.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/target/ppc/int_helper.c b/target/ppc/int_helper.c index 834da80fe3..3bf0f5dbe5 100644 --- a/target/ppc/int_helper.c +++ b/target/ppc/int_helper.c @@ -26,6 +26,7 @@ #include "exec/helper-proto.h" #include "crypto/aes.h" #include "crypto/aes-round.h" +#include "crypto/clmul.h" #include "fpu/softfloat.h" #include "qapi/error.h" #include "qemu/guest-random.h" @@ -1425,6 +1426,15 @@ void helper_vbpermq(ppc_avr_t *r, ppc_avr_t *a, ppc_avr_t *b) #undef VBPERMQ_INDEX #undef VBPERMQ_DW +void helper_vpmsumb(ppc_avr_t *r, ppc_avr_t *a, ppc_avr_t *b) +{ + Int128 ia = a->s128; + Int128 ib = b->s128; + Int128 e = clmul_8x8_even(ia, ib); + Int128 o = clmul_8x8_odd(ia, ib); + r->s128 = int128_xor(e, o); +} + #define PMSUM(name, srcfld, trgfld, trgtyp) \ void helper_##name(ppc_avr_t *r, ppc_avr_t *a, ppc_avr_t *b) \ { \ @@ -1445,7 +1455,6 @@ void helper_##name(ppc_avr_t *r, ppc_avr_t *a, ppc_avr_t *b) \ } \ } -PMSUM(vpmsumb, u8, u16, uint16_t) PMSUM(vpmsumh, u16, u32, uint32_t) PMSUM(vpmsumw, u32, u64, uint64_t) From patchwork Thu Jul 13 21:14:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702245 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp742412wrj; Thu, 13 Jul 2023 14:18:55 -0700 (PDT) X-Google-Smtp-Source: APBJJlFwURlvazk+nE+5tonEID7Jkbe/Rc/TbB5j6sy6iPnJgS4FF4MnYZUv6UEJPHinur52m6J5 X-Received: by 2002:a0c:aa42:0:b0:635:f412:6a75 with SMTP id e2-20020a0caa42000000b00635f4126a75mr2404959qvb.48.1689283135438; Thu, 13 Jul 2023 14:18:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689283135; cv=none; d=google.com; s=arc-20160816; b=Qvrl3qy88szE9MWXe4vXadMeWpNoJUNWJT4PbzwQtGcD/XnfOM/whMQjc+0drIH/Xo HGbRoo1nv6xr69m9BQogSbOcDZv/6T93PWCsLnVerUfmzQ98FWsIpEBfex8KD0CQd04P xJ1cPQAgpyR0ckf78h4fYSRwcFlexTAnmcnb2Uz2zLXmDT8ZUo7cOvBY15s6RT2EGiv9 Q/xpmnTLzVAEUFn5pUGeXcePlAe5CeuHLGc+bBUdFh5s2f/ZRoc7NL78fxOlVkcaEGT0 l4oLjK1ymRI0bHv+8epBOeEcAghyVgZN/gf9LimWls+GDiAoZsmA8RT7U9tVRipze+xO Bi2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=hpm+Tse7Y103FOMmNwDNKcmWp0woIdWlhDNb7L+eKdk=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=cS8qXe7H0LH8sC+Q31kahaiqlFE9XBn3OgTYgazncfhyo154ZS0zL5s2zOHJE1XnnC l2qsMPFX1NK2pCeMm4zl8j+l8fn4xnWrCJqoB8Ip8uT96qA9+UlhBZ4SarPyfIzp1Mmu 0RQEvEUoB5nQyxONMI+lZ29hU9Hxz0hKcqrwM93t3nBWyt/5xLg6L5EY7NTm9/zadRPe 5DONcDoWSW/yQQE9hr0DeetpDOEaHiaV9Kvs1BbHIiB/oj/6b5X999OY6hp+LnY5LOLb DQlHyZw8PEIqs4MsKf5w6UHF6j+zB3POQ4nuhLQKEfhUYCuTj62kGXGoQFHTOm++KutA axhg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Ymkx1ZkB; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id q15-20020a0ce20f000000b0062388040b4csi3612456qvl.203.2023.07.13.14.18.55 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:18:55 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Ymkx1ZkB; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3ea-0000QL-5T; Thu, 13 Jul 2023 17:15:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eH-0000I5-Ng for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:54 -0400 Received: from mail-wm1-x331.google.com ([2a00:1450:4864:20::331]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3eC-00016U-C5 for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:53 -0400 Received: by mail-wm1-x331.google.com with SMTP id 5b1f17b1804b1-3fbd33a57dcso11896265e9.0 for ; Thu, 13 Jul 2023 14:14:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282887; x=1691874887; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hpm+Tse7Y103FOMmNwDNKcmWp0woIdWlhDNb7L+eKdk=; b=Ymkx1ZkBA98FvqUJN4LQVtifHOY4M47/ksi6Gj3oSgYXTnHDop4dcUOnWIZm78I70/ Llpsa1zsKVQCd2DY7+3nRoBPflcXWxyocFxsjXjlVUIshM+aTXHz96OwEKjD74lBaHO7 k3OfrG5xqqhAy4zzU+iBRfZSaImbJSzes7917nLMa4yb3A9CdMyjy3aVBzqEVVYhXyq/ zQFJ7Mj+A31+GrjseWvg9NwByLBnfplOo2HthjEd+vKovQpWE97+MVKfw19n603p+EXX pg79dPJz/E6tHS95Z6OpIwx8DGlUhwwrutd0PXNNjOGUJZT5F0ESkYIWx2UPxF1so2Ct kA1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282887; x=1691874887; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hpm+Tse7Y103FOMmNwDNKcmWp0woIdWlhDNb7L+eKdk=; b=IvQwCPsIgcdZAzXel2f7/m2l5ZiE6sqEBMrzSpusKBV3MD2poiVd2DJGDspkNK/Way f64GwEzt9qKKdmVCmdC+rWUICZ9FwCCfDAL9gdVPJ3ZPqNLXYKZq5UN+zcMVb9J11Xu0 kBCZlZ4ETpoPji7rzDZ3WSMvkpqpHWLDsHM0mJsFLCBu6OOAmlkwjO2t8co2rHdoI05f UiFmg4MH+54mag+WsvhKUrlxrX5iUFAqFSsaAs2a4V+h1GZt9IPnfa3DypYfcnZfoLWc 89M9QYJNBvZ7P1nw+ZM0TrA7Tog2N8C+Q1Zt6tqpL9vdLnmfpNBYpryaSUrxvOWrY3Sa +jkw== X-Gm-Message-State: ABy/qLZ5qGPnTQBQfDPS+Lkpg2seMz3HaGHI9Diujn+YJre/a3Yh/hmD p/ijLsphacw5LXuhXTEhthHhUaNZ8vwq7SZmePyOE9SK X-Received: by 2002:a05:6000:86:b0:311:1a45:3606 with SMTP id m6-20020a056000008600b003111a453606mr2378255wrx.30.1689282887123; Thu, 13 Jul 2023 14:14:47 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:46 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 05/18] crypto: Add generic 16-bit carry-less multiply routines Date: Thu, 13 Jul 2023 22:14:22 +0100 Message-Id: <20230713211435.13505-6-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::331; envelope-from=richard.henderson@linaro.org; helo=mail-wm1-x331.google.com X-Spam_score_int: -16 X-Spam_score: -1.7 X-Spam_bar: - X-Spam_report: (-1.7 / 5.0 requ) BAYES_00=-1.9, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Signed-off-by: Richard Henderson --- host/include/generic/host/crypto/clmul.h | 5 +++ include/crypto/clmul.h | 32 +++++++++++++++++++ crypto/clmul.c | 39 ++++++++++++++++++++++++ 3 files changed, 76 insertions(+) diff --git a/host/include/generic/host/crypto/clmul.h b/host/include/generic/host/crypto/clmul.h index 694705f703..cba8bbf3e4 100644 --- a/host/include/generic/host/crypto/clmul.h +++ b/host/include/generic/host/crypto/clmul.h @@ -14,4 +14,9 @@ #define clmul_8x8_odd clmul_8x8_odd_gen #define clmul_8x8_packed clmul_8x8_packed_gen +#define clmul_16x2_even clmul_16x2_even_gen +#define clmul_16x2_odd clmul_16x2_odd_gen +#define clmul_16x4_even clmul_16x4_even_gen +#define clmul_16x4_odd clmul_16x4_odd_gen + #endif /* GENERIC_HOST_CRYPTO_CLMUL_H */ diff --git a/include/crypto/clmul.h b/include/crypto/clmul.h index 7f19205d6f..b701bac9d6 100644 --- a/include/crypto/clmul.h +++ b/include/crypto/clmul.h @@ -56,6 +56,38 @@ Int128 clmul_8x8_odd_gen(Int128, Int128); */ Int128 clmul_8x8_packed_gen(uint64_t, uint64_t); +/** + * clmul_16x2_even: + * + * Perform two 16x16->32 carry-less multiplies. + * The odd words of the inputs are ignored. + */ +uint64_t clmul_16x2_even_gen(uint64_t, uint64_t); + +/** + * clmul_16x2_odd: + * + * Perform two 16x16->32 carry-less multiplies. + * The even bytes of the inputs are ignored. + */ +uint64_t clmul_16x2_odd_gen(uint64_t, uint64_t); + +/** + * clmul_16x4_even: + * + * Perform four 16x16->32 carry-less multiplies. + * The odd bytes of the inputs are ignored. + */ +Int128 clmul_16x4_even_gen(Int128, Int128); + +/** + * clmul_16x4_odd: + * + * Perform eight 16x16->32 carry-less multiplies. + * The even bytes of the inputs are ignored. + */ +Int128 clmul_16x4_odd_gen(Int128, Int128); + #include "host/crypto/clmul.h" #endif /* CRYPTO_CLMUL_H */ diff --git a/crypto/clmul.c b/crypto/clmul.c index 866704e751..69a3b6f7ff 100644 --- a/crypto/clmul.c +++ b/crypto/clmul.c @@ -74,3 +74,42 @@ Int128 clmul_8x8_packed_gen(uint64_t n, uint64_t m) rh = clmul_8x4_even_gen(unpack_8_to_16(n >> 32), unpack_8_to_16(m >> 32)); return int128_make128(rl, rh); } + +uint64_t clmul_16x2_even_gen(uint64_t n, uint64_t m) +{ + uint64_t r = 0; + + n &= 0x0000ffff0000ffffull; + m &= 0x0000ffff0000ffffull; + + for (int i = 0; i < 16; ++i) { + uint64_t mask = (n & 0x0000000100000001ull) * 0xffffffffull; + r ^= m & mask; + n >>= 1; + m <<= 1; + } + return r; +} + +uint64_t clmul_16x2_odd_gen(uint64_t n, uint64_t m) +{ + return clmul_16x2_even_gen(n >> 16, m >> 16); +} + +Int128 clmul_16x4_even_gen(Int128 n, Int128 m) +{ + uint64_t rl, rh; + + rl = clmul_16x2_even_gen(int128_getlo(n), int128_getlo(m)); + rh = clmul_16x2_even_gen(int128_gethi(n), int128_gethi(m)); + return int128_make128(rl, rh); +} + +Int128 clmul_16x4_odd_gen(Int128 n, Int128 m) +{ + uint64_t rl, rh; + + rl = clmul_16x2_odd_gen(int128_getlo(n), int128_getlo(m)); + rh = clmul_16x2_odd_gen(int128_gethi(n), int128_gethi(m)); + return int128_make128(rl, rh); +} From patchwork Thu Jul 13 21:14:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702227 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp740702wrj; Thu, 13 Jul 2023 14:15:40 -0700 (PDT) X-Google-Smtp-Source: APBJJlGZYXuSE0UDgx+DxhGuYWngGMG7FS1/IWWAmQZ4l5drYDuWyfjMrmXsURedyyKupTc887aO X-Received: by 2002:ac8:7f92:0:b0:403:b2d9:fbbb with SMTP id z18-20020ac87f92000000b00403b2d9fbbbmr3846200qtj.7.1689282940179; Thu, 13 Jul 2023 14:15:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689282940; cv=none; d=google.com; s=arc-20160816; b=gE7s/qGBftNctm/2vNnSRYERCKmDVfUNiexCT9KWr6nQhgfAkT4rzh5ibXAPMyp0w9 gK6YeX/bhXIs61wSgUOJO+fAxJ82lmzGU6t6yJMSSf2tkwDKpdQi9p2k0xkFCUSpp9nJ LVd+ETzqEAO/IcVRQ665u3IktQROqg6NZmFWs6avBccnWy53CL/kZLvAmhlpteQoPvTs C9/pijQyvWRN1PN3FoYfseGS4Bb3EyuVfDiY1J/0QTVRH4ySDIzSC4t/V3R9O7dGUQJ6 aykk9bYrgHRVxQSuVNRsNSoUwvzY594aZShdkS2MiZ6Lill6EpSw9lG239vL9+iLOEte zm8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=joE7iM6kvkwj0QEtXA447MLGCwlorW8Lqpd1o16zxAo=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=UIB1ThanF4GGBdcEbhF+g+CXb8B5O1CGuZXQUvk1yDCWPI5poarAl/5/4brSIYZkbo X2rcG+LFD6KIps3++qn0v22BHiA/TDsvrhqCwIf2/UflCEGnD6pghLEGHsOLMIztMVnB j1XXMFgFH58RSl1bZFr3zOoGIcx5n+f9vpbc3+xu07bhB3weBNShN30C7mOg7K+/4kQU X9uoUIlEBm7TDycsAW35Ed6uE+El9CZ2p/AH1vPLppdzdh3hczUTFJR8/C5d4B781vf3 B7s19FUWqwCWQm4CUr+NnNV/V1tHX0KKeIwQELxcP3hc2NgdynSjkhd8vaHzSUc7ta+n ctnA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=FrdDIhdq; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id a21-20020ac85b95000000b00401e0febe01si3643596qta.215.2023.07.13.14.15.40 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:15:40 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=FrdDIhdq; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3eI-0000I4-Ec; Thu, 13 Jul 2023 17:14:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eG-0000He-M9 for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:52 -0400 Received: from mail-wr1-x42b.google.com ([2a00:1450:4864:20::42b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3eE-00016g-9g for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:52 -0400 Received: by mail-wr1-x42b.google.com with SMTP id ffacd0b85a97d-3094910b150so1342729f8f.0 for ; Thu, 13 Jul 2023 14:14:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282888; x=1691874888; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=joE7iM6kvkwj0QEtXA447MLGCwlorW8Lqpd1o16zxAo=; b=FrdDIhdqXHSAtYxZK82P+sbDZXBm9z4pOPiExBL1CvvkBu15dy7uqGfyMAHvznqnSW WPUXf+HZq9P0iOiiVcIjmv2Ew13PyyU4NCTyvEyjiwy1b29hFoYJcuq47/zJiscyVbgW HmpIve4kpL28wHY0qQHnj9OXpTA0irmSK/4UHIiFR90tY2EWUFohrEy+lbbRwAjObLOR ZufIYxp2D4RWINF3X4/PMjkp0XBwgmCKY7CDyd/P2dZWkBTqEy1noSYl7gF+HgFvptGg etEBoIsAI2QUU/c3+16oXFTHMG7zgXNd5wGYhOJ+6sE8Iz7ICSSRRvP/kJQzLEwE6901 CBBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282888; x=1691874888; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=joE7iM6kvkwj0QEtXA447MLGCwlorW8Lqpd1o16zxAo=; b=YLCWC+LMbo9p2Dv2BnUzJbOihczUz+Y93npa6CUWt+L60IRRt7tK6RUcjRxJCEAkru bNUIMFA6g/wl7xsSAbysuACSTof+PvCOiiM629byRCn0KQ8xWy+e4JHSUyRgh/QY7COd LLpd3KdHphD4neRVlUMWUSY5eL6rUMLSFUMOUCNYKpx3os8r2u4qFqx95/GJat8qa/6B J0/yeQl/smul5OcXWyEC7mteXtXTQWF9WGYlZoq+OIFWPn3TaPszEYsFKiKDTRcfO5IW cVy5yc7JVmSvMNINVYl+lEUcBSYEputpG3/uu6WxYoZSYpGXxF7JWrtcki999JM7a8kw dtKA== X-Gm-Message-State: ABy/qLbZer8cOz1QW9JAQL2EnKx4BXwxf9UEyVpLP2YSxcHpTjEt/qYK YRy6t8q/jCZ+U+daO0G9JrDEMmfxnlb5qQdJ5VtmiK3M X-Received: by 2002:adf:d4c8:0:b0:313:f07b:8027 with SMTP id w8-20020adfd4c8000000b00313f07b8027mr2606481wrk.36.1689282887984; Thu, 13 Jul 2023 14:14:47 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:47 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 06/18] target/arm: Use clmul_16* routines Date: Thu, 13 Jul 2023 22:14:23 +0100 Message-Id: <20230713211435.13505-7-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::42b; envelope-from=richard.henderson@linaro.org; helo=mail-wr1-x42b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Use generic routines for 16-bit carry-less multiply. Remove our local version of pmull_w. Signed-off-by: Richard Henderson --- target/arm/tcg/vec_internal.h | 6 ------ target/arm/tcg/mve_helper.c | 8 ++------ target/arm/tcg/vec_helper.c | 13 ------------- 3 files changed, 2 insertions(+), 25 deletions(-) diff --git a/target/arm/tcg/vec_internal.h b/target/arm/tcg/vec_internal.h index c4afba6d9f..3ca1b94ccf 100644 --- a/target/arm/tcg/vec_internal.h +++ b/target/arm/tcg/vec_internal.h @@ -219,12 +219,6 @@ int16_t do_sqrdmlah_h(int16_t, int16_t, int16_t, bool, bool, uint32_t *); int32_t do_sqrdmlah_s(int32_t, int32_t, int32_t, bool, bool, uint32_t *); int64_t do_sqrdmlah_d(int64_t, int64_t, int64_t, bool, bool); -/* - * 16 x 16 -> 32 vector polynomial multiply where the inputs are - * in the low 16 bits of each 32-bit element - */ -uint64_t pmull_w(uint64_t op1, uint64_t op2); - /** * bfdotadd: * @sum: addend diff --git a/target/arm/tcg/mve_helper.c b/target/arm/tcg/mve_helper.c index 96ddfb4b3a..c666a96ba1 100644 --- a/target/arm/tcg/mve_helper.c +++ b/target/arm/tcg/mve_helper.c @@ -985,14 +985,10 @@ DO_2OP_L(vmulltuw, 1, 4, uint32_t, 8, uint64_t, DO_MUL) * Polynomial multiply. We can always do this generating 64 bits * of the result at a time, so we don't need to use DO_2OP_L. */ -#define VMULLPW_MASK 0x0000ffff0000ffffULL -#define DO_VMULLPBW(N, M) pmull_w((N) & VMULLPW_MASK, (M) & VMULLPW_MASK) -#define DO_VMULLPTW(N, M) DO_VMULLPBW((N) >> 16, (M) >> 16) - DO_2OP(vmullpbh, 8, uint64_t, clmul_8x4_even) DO_2OP(vmullpth, 8, uint64_t, clmul_8x4_odd) -DO_2OP(vmullpbw, 8, uint64_t, DO_VMULLPBW) -DO_2OP(vmullptw, 8, uint64_t, DO_VMULLPTW) +DO_2OP(vmullpbw, 8, uint64_t, clmul_16x2_even) +DO_2OP(vmullptw, 8, uint64_t, clmul_16x2_odd) /* * Because the computation type is at least twice as large as required, diff --git a/target/arm/tcg/vec_helper.c b/target/arm/tcg/vec_helper.c index 4384b6c188..1b1d5fccbc 100644 --- a/target/arm/tcg/vec_helper.c +++ b/target/arm/tcg/vec_helper.c @@ -2029,19 +2029,6 @@ void HELPER(gvec_pmull_q)(void *vd, void *vn, void *vm, uint32_t desc) clear_tail(d, opr_sz, simd_maxsz(desc)); } -uint64_t pmull_w(uint64_t op1, uint64_t op2) -{ - uint64_t result = 0; - int i; - for (i = 0; i < 16; ++i) { - uint64_t mask = (op1 & 0x0000000100000001ull) * 0xffffffff; - result ^= op2 & mask; - op1 >>= 1; - op2 <<= 1; - } - return result; -} - void HELPER(neon_pmull_h)(void *vd, void *vn, void *vm, uint32_t desc) { int hi = simd_data(desc); From patchwork Thu Jul 13 21:14:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702243 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp742405wrj; Thu, 13 Jul 2023 14:18:54 -0700 (PDT) X-Google-Smtp-Source: APBJJlFIXuBwEdStSk8+Z+Ngv0fHy5HAR+hwu2GmdsHrR/yac3Q5hgkFgRyt4XYoQge5iM1u+Hg9 X-Received: by 2002:a05:622a:551:b0:3ff:42df:7bfd with SMTP id m17-20020a05622a055100b003ff42df7bfdmr2964289qtx.63.1689283134304; Thu, 13 Jul 2023 14:18:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689283134; cv=none; d=google.com; s=arc-20160816; b=HjGqAUz4HNpvECooflS1xCTfFhUtb3KPfSsRiDUT8ONr2JviyriH/4xoDLk2lp0wcN SSlJPuxHKY19J1Ckl4K17Bx4fp9PQAYNkDVmW8n4chfGP6tFBzDQprVh7DBoTjmSHhu9 t/44huSLmj80B3wCoxM4+elORTS8RffdU/4WEB1ukvle4lnGE9O5I6ycyCjkGd7EKPFk t+CfbogdBqSR+IbnwZfbQHg4jnomjNLX6MDMY5DvIYADA7nGtiknlIzW2lmIfvNgJ5l1 PLuxOugeXclo4c2LSpRsltkK0ZtnE30gkbgVxQrggILGS9sm+3CEIrnHFtvInHj0z17d iApA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=rL0bmNffPB7WGyLli9yh/J2zSVJX7fi9BvkXQFIA5Kc=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=sk6vjTZeBIoo+02DDqupBVPs5OjONIe7tHZUJgFT7zlKAK4cjGHAuW/KYxxivxXMMl EmtwsCqOEiCmhMw8ez6FhCBrABYhqcSzZa3Xv6nrWxFqvMbx7akoURTcJjFgCJpHThxq sDFCf1FKumQXzWDDc4llBkqIQ+SadPcXzHFxpplRSYXAs6UfyZIdf0kbcPd/8h/IBzUO 8NkxVFcLEqn+vVJPKvfLyCnEyIc43YLTuN/xTF28XVSCIrj3hEkkfNFHG/xL6JdRyOK+ /2yStpyMj3VuvT9m5ueLw20f5NDtLYLkbnLgxLIJlp61UEGYuYuFQWY1p1Sf4nxsE6rY J7GQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=eHSZynIw; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id u16-20020a05622a14d000b0040383b3aea2si3597939qtx.778.2023.07.13.14.18.54 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:18:54 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=eHSZynIw; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3eK-0000J8-K2; Thu, 13 Jul 2023 17:14:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eH-0000I3-CK for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:54 -0400 Received: from mail-wr1-x433.google.com ([2a00:1450:4864:20::433]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3eE-00016t-JZ for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:53 -0400 Received: by mail-wr1-x433.google.com with SMTP id ffacd0b85a97d-3163eb69487so1259511f8f.1 for ; Thu, 13 Jul 2023 14:14:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282889; x=1691874889; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rL0bmNffPB7WGyLli9yh/J2zSVJX7fi9BvkXQFIA5Kc=; b=eHSZynIwUPhXF+gDw9Jsj3l0JfmRCWB0/NyfL9zLrpISrxs7GKTMGNIARIUQNKbEd/ JGMvtmKhsIfpsVPQG8lQwjqPG/SNdNgBZ/TSCTnM0t/WmQMt4yZha3G2N5ddnMPNrJ1O n7LnjZXqkEFJNBXQIwUbAO3kK/bf7R6aW1XMl+Mx4knnag1KdSWiA8Zs/Vsje7w7kNpn gQEYY5WOq67SmzsX8GqLfHRII+j9g8EVUT/pnclOsGfUH+rHnUyH0EWsohznkVqjpe5S eBfDpaB/PSTZE099/A7rrkynQkWg2NsvhyOqPu0cca5Gc9MEkDv+38bq7qILJ73CCNb4 1r8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282889; x=1691874889; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rL0bmNffPB7WGyLli9yh/J2zSVJX7fi9BvkXQFIA5Kc=; b=gIgylqWXJ8n8ObBz/iYOvsZFDZzKad4VCZzPgmzngBjnlB9xKXDCZ5xxrKsFwUr287 0l1h/yrNc9SPYBAiPmXyq4q0/EjcW7aXfhAUelIwnbtSjkwsQIag92l5a9hy74mOfMx0 rmiqx27QSOypAwiTM/CE0UZ/jGI1J75gJCKZeBLTuP1FBqnWPI14qe2icHqx82dLH1We oGAuE8VVCYfuaMiwOs2ZUKA7amQnfU6ZNaoqRqabK3yrH0s5YMEB6JQKiewuBNfNoLXy q2vjsesmkM0NUPpIyBWQnABxlIDnqdPCUtHo5m36dkNbKGwFivEUejQDY+R0rttXPRuw nE5A== X-Gm-Message-State: ABy/qLYy31bcJpsqCMHPdkk22pN7dlU+moeX6gieYoK2t6KwV6W3naAB yyNXWzGTzn8jML+k214BpELR5bn0vPPEFEolpkFvrxVr X-Received: by 2002:adf:f44d:0:b0:314:1d6:8aa7 with SMTP id f13-20020adff44d000000b0031401d68aa7mr595907wrp.29.1689282889209; Thu, 13 Jul 2023 14:14:49 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:48 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 07/18] target/s390x: Use clmul_16* routines Date: Thu, 13 Jul 2023 22:14:24 +0100 Message-Id: <20230713211435.13505-8-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::433; envelope-from=richard.henderson@linaro.org; helo=mail-wr1-x433.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Use generic routines for 16-bit carry-less multiply. Remove our local version of galois_multiply16. Signed-off-by: Richard Henderson --- target/s390x/tcg/vec_int_helper.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/target/s390x/tcg/vec_int_helper.c b/target/s390x/tcg/vec_int_helper.c index e110a7581a..523d6375bb 100644 --- a/target/s390x/tcg/vec_int_helper.c +++ b/target/s390x/tcg/vec_int_helper.c @@ -180,7 +180,6 @@ static uint##TBITS##_t galois_multiply##BITS(uint##TBITS##_t a, \ } \ return res; \ } -DEF_GALOIS_MULTIPLY(16, 32) DEF_GALOIS_MULTIPLY(32, 64) static S390Vector galois_multiply64(uint64_t a, uint64_t b) @@ -226,6 +225,25 @@ void HELPER(gvec_vgfma8)(void *v1, const void *v2, const void *v3, *(Int128 *)v1 = int128_xor(r, *(Int128 *)v4); } +static Int128 do_gfm16(Int128 n, Int128 m) +{ + Int128 e = clmul_16x4_even(n, m); + Int128 o = clmul_16x4_odd(n, m); + return int128_xor(e, o); +} + +void HELPER(gvec_vgfm16)(void *v1, const void *v2, const void *v3, uint32_t d) +{ + *(Int128 *)v1 = do_gfm16(*(const Int128 *)v2, *(const Int128 *)v3); +} + +void HELPER(gvec_vgfma16)(void *v1, const void *v2, const void *v3, + const void *v4, uint32_t d) +{ + Int128 r = do_gfm16(*(const Int128 *)v2, *(const Int128 *)v3); + *(Int128 *)v1 = int128_xor(r, *(Int128 *)v4); +} + #define DEF_VGFM(BITS, TBITS) \ void HELPER(gvec_vgfm##BITS)(void *v1, const void *v2, const void *v3, \ uint32_t desc) \ @@ -243,7 +261,6 @@ void HELPER(gvec_vgfm##BITS)(void *v1, const void *v2, const void *v3, \ s390_vec_write_element##TBITS(v1, i, d); \ } \ } -DEF_VGFM(16, 32) DEF_VGFM(32, 64) void HELPER(gvec_vgfm64)(void *v1, const void *v2, const void *v3, @@ -279,7 +296,6 @@ void HELPER(gvec_vgfma##BITS)(void *v1, const void *v2, const void *v3, \ s390_vec_write_element##TBITS(v1, i, d); \ } \ } -DEF_VGFMA(16, 32) DEF_VGFMA(32, 64) void HELPER(gvec_vgfma64)(void *v1, const void *v2, const void *v3, From patchwork Thu Jul 13 21:14:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702242 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp742382wrj; Thu, 13 Jul 2023 14:18:52 -0700 (PDT) X-Google-Smtp-Source: APBJJlG3k24QACfb4/pRXEzhUvA31jw+ColLpMARVeA9wGWQ8xuiq42F4nWo8nm4ToT3MyHUDgUp X-Received: by 2002:a05:620a:bc1:b0:75b:23a0:e7de with SMTP id s1-20020a05620a0bc100b0075b23a0e7demr3475667qki.63.1689283131886; Thu, 13 Jul 2023 14:18:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689283131; cv=none; d=google.com; s=arc-20160816; b=BOyRqHsg0Q9uEsZy4mSBBbGXHExlEKlg8xxDr40vKGsNqFhQdVOuOxf5VUOYltKuFe EdTY6KJk47NYgkqoBDYZiq1eVlmn43OLxUOC2ARy70fym8qLstj8bXc/UGIQToA+a6qt vWWPOJUWLhMNo3biMGOMDotbOsfE09k+MJtJuDNHnCparAr+95cTGQSklJstsG3JFKrS 6Y1G6cfHJ/LkV/b4jbJzRAgUCi45DHGe/JeyhUreRhmnBQzEgMiy62KgJoeOlI8xYvLy F1XVyek7KqSBk9T5XCBaZvlYRe1XTXuL12cOMBBCWGacMc6OUtDBBVz6wDLWdG/eoEjS Wa/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=mkMN+Ym75aKqSSlyQfMGwZmblrlERSKZPoX/RaUMIgw=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=OAdqIRLhajnvWt+m4IR4rxzLC2DUtNAOA62k3ogXzH6o1R/hN5pk4TWZ5mVR09GLsA DiZ3Z3DZmpnfhbOywyJbpiVbWzHPbLMmu4jtpzrho2L8JvcVUhtlwHvyBo2p3PkqwCX/ zANUTftXjrGtE29hwtqYnMMJIGs5VBU+YQ6S6HKpD44mB69SAFwj+TCTZmDzgdLSuQBu da1w2if9ZrKpGsrJUPRS+e8IKFpfT426oGzTz82OIg6mENzc6O7m8oFcrRTk7Jyu2Iwm gKWnY8TBAOWC9VZyWVZ1P44TbuHpqIPoEtv5Gjspy+EBjAkrd7ZgQz9h9DH6ytDn4KRl gvZQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=s87Fh+Wv; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id q22-20020a05620a0d9600b0076729aa3775si3648268qkl.609.2023.07.13.14.18.51 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:18:51 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=s87Fh+Wv; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3eK-0000J9-Ke; Thu, 13 Jul 2023 17:14:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eI-0000IL-Kg for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:54 -0400 Received: from mail-wm1-x330.google.com ([2a00:1450:4864:20::330]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3eG-00017G-QU for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:54 -0400 Received: by mail-wm1-x330.google.com with SMTP id 5b1f17b1804b1-3fbc0609cd6so11285645e9.1 for ; Thu, 13 Jul 2023 14:14:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282890; x=1691874890; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mkMN+Ym75aKqSSlyQfMGwZmblrlERSKZPoX/RaUMIgw=; b=s87Fh+WvZVFOS0o7bh6N3wcnQi+96xnWjFhEsza75Sc8obbGlJDhVMDMaaGjIR4d0d 1bPtdF8+N4ZcNoYYKQOS+Moff3/Zb4kkr9hfh/gF2nogqeDjlB/qe//bwz/yUfn2Fyd+ bpYORJDXpHknejCOOYRbijtsV5F6LBoGwANrRi/2v6wMx3+NDxUt4h1D8AmO9yv0CDSM Z45ORZA99nFhO+Q67TFVPXwpvKktGtol1sa/q08yjJcyutRqdKQXEKkb2eAfRZ/sLfAd Yq24eEgrV8tqWPnFHx7QUxIocpbTWappxdVC7DhVCSue7wMKkKUR7wEKKpaQUxhQRTP0 TPEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282890; x=1691874890; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mkMN+Ym75aKqSSlyQfMGwZmblrlERSKZPoX/RaUMIgw=; b=Rk7sY4uzW3uclpTXNdF0TB3lynJG50+TUhn0extOJTgFCNQf9Tzyfh7Pz1M7TUjy/V 3L4QmzcBHUDgvKkXE2/98tVLcnQ2yZQWroZbrK9TkmX3X6a3wdGLm1qgcL3K37riuNsl SJ3jutvqw0IyM9kMUGyzIfHr0oGMejXbBEdoM75cxnYWnbTr+sZptrQt86dwH86Pe8vw h5lO22i7SsfdiAr50EbhlW5bEB5RoGnV+V57t4Srpe2iKxdVsZ0Lm3Lm4GrqrjOymFID R9auSLWhIy4lPw3CO8VkwX9ZxaITRS+A4+04l61ZGg6ACOkB/k+l2gse9kmmATq4pYii zQtA== X-Gm-Message-State: ABy/qLYEosOF8FbgOWvzPUDeDrhmUjWTAnagbIqPjn5BpjSCtuVpkHNe 1AxWZBElyX1+pRIxyqvuAGMqzl5T8SHiwId0h2rhYKlo X-Received: by 2002:a5d:4d02:0:b0:314:17cc:31d0 with SMTP id z2-20020a5d4d02000000b0031417cc31d0mr2185561wrt.34.1689282890235; Thu, 13 Jul 2023 14:14:50 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:49 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 08/18] target/ppc: Use clmul_16* routines Date: Thu, 13 Jul 2023 22:14:25 +0100 Message-Id: <20230713211435.13505-9-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::330; envelope-from=richard.henderson@linaro.org; helo=mail-wm1-x330.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Use generic routines for 16-bit carry-less multiply. Signed-off-by: Richard Henderson --- target/ppc/int_helper.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/target/ppc/int_helper.c b/target/ppc/int_helper.c index 3bf0f5dbe5..98d6310f59 100644 --- a/target/ppc/int_helper.c +++ b/target/ppc/int_helper.c @@ -1435,6 +1435,15 @@ void helper_vpmsumb(ppc_avr_t *r, ppc_avr_t *a, ppc_avr_t *b) r->s128 = int128_xor(e, o); } +void helper_vpmsumh(ppc_avr_t *r, ppc_avr_t *a, ppc_avr_t *b) +{ + Int128 ia = a->s128; + Int128 ib = b->s128; + Int128 e = clmul_16x4_even(ia, ib); + Int128 o = clmul_16x4_odd(ia, ib); + r->s128 = int128_xor(e, o); +} + #define PMSUM(name, srcfld, trgfld, trgtyp) \ void helper_##name(ppc_avr_t *r, ppc_avr_t *a, ppc_avr_t *b) \ { \ @@ -1455,7 +1464,6 @@ void helper_##name(ppc_avr_t *r, ppc_avr_t *a, ppc_avr_t *b) \ } \ } -PMSUM(vpmsumh, u16, u32, uint32_t) PMSUM(vpmsumw, u32, u64, uint64_t) void helper_VPMSUMD(ppc_avr_t *r, ppc_avr_t *a, ppc_avr_t *b) From patchwork Thu Jul 13 21:14:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702239 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp742173wrj; Thu, 13 Jul 2023 14:18:22 -0700 (PDT) X-Google-Smtp-Source: APBJJlGxivmqYMWvw0off4Bp4xgit2f+T1JdE0NWOmjj1zQfu3CSW+vdBwevt+EsG6N1nXI5i4re X-Received: by 2002:a05:622a:16:b0:403:b112:1435 with SMTP id x22-20020a05622a001600b00403b1121435mr3654562qtw.28.1689283102187; Thu, 13 Jul 2023 14:18:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689283102; cv=none; d=google.com; s=arc-20160816; b=XEUgjBAVmFash04zTnXHubvHzCFCEPZ6Rj+6pgyQNd/boVPvJErVhM7eaUQVN7QwpG lTJuF9lzuG3LLS2GLyOjZS69+5Hv4W/W4Rf1HJacaLp6btFfXh51dQm/w7/5mbtqRIrp yA0KXAr7ydUfyN+cF292yi8EB77G9dBcwGJt0JgDyubWE+/sTPTLstpSoFcHaPE++hPT jbI4OmpVfGZr6JXieksnB9izTxyd/z68Z1naa3+kbUuNlEnqq6a9p9I2GOMF6K34h63e O+SyCJnRUNRt545TARF6rwCyY1mqhfz7ZWWWIyOoKZ/VFox6mobo0wBUBRJrGuIL81Qb aYmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=/VchOJmp/PdkjGrWpO7FtoJjXcBbpTQPyllwuZ+9STU=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=i1pdw6UfzxACR0Oqp9Riq6nRjD4ovQa0bBQ0PnFiz7k2kZnur+D9wpEdE7ACpciQaw Iq5yoEyF/p6bOe9HzhEt4IFf5fkvcvd4qWImJmuvlwgrcaDBekpikL/95MsdBohFTBSw /UwSlTPIc+bL4gNBsUV4XpMiO1QUs3xH5+pG9f1/0riGZLpeRtQsxYZewj8t985mBCED 6H/LlTURHWLFaBbGe6l9kL8k73JjAf8//X/nTHH52PqGfaO82+HZZ6RglMC/ZKxS9QtM ng2TesPV7/YVsJbsuC5qgSB9ODp5CNfodUwX1Br5zJBiL5YZe0l5IDVaNjgdMCVveqnW zXzg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=kQIxRxw+; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id z15-20020a05622a028f00b004039f269d11si3641983qtw.100.2023.07.13.14.18.22 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:18:22 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=kQIxRxw+; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3eM-0000Jw-4S; Thu, 13 Jul 2023 17:14:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eI-0000I9-FO for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:54 -0400 Received: from mail-wm1-x32f.google.com ([2a00:1450:4864:20::32f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3eG-00017U-Pq for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:54 -0400 Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-3fbca8935bfso11255705e9.3 for ; Thu, 13 Jul 2023 14:14:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282891; x=1691874891; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/VchOJmp/PdkjGrWpO7FtoJjXcBbpTQPyllwuZ+9STU=; b=kQIxRxw+RpNmTfB3c1AJOyZkE7mchlHaPAT2bg6WzywfUo8ez/Yd7xbPK/RVIX57sH NJIme9uP6JJ7Crj2jt9tWtynUR0XvMF9szTlWYQ1HNLvadNUBlmf9aheVhCJwWpK0IMa JZ8ka0+8liKeZYqjQOYO0HRRPSFxosYlqD34hx4LpXsMKpF0AqwILFzVWeZgaKFJEYiT CTyrxJB66E49ypDteL2SDkN7vv5J/zVB7rM/lXH0C1IeKoemmJx+sY8y8E/j3JMCSyTv pO7sli3LScNCQ77N6oJd0zC69LRIyAOgJrXqCOfa9IBpLhMczH6ozD4+Lz2wVYiXrA2F Uozg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282891; x=1691874891; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/VchOJmp/PdkjGrWpO7FtoJjXcBbpTQPyllwuZ+9STU=; b=eN2seIlvjbkXJhLlKbQnzTEsG3S4DmSedpzt5aEKqombqvDPtqcuLETK6HTMkIwV2i X1YHONrlZjNsa0NxHxj69BHSrWmuPEABd1CDI72ZUmiUtuyVOUbJuLZ5Jhw3Jyp6mdiu x2FiEosbt1Cb6QD76H8aP5GB0pWNY4EO1AdcD6J04L2U8Fe/HADTIjhSBL7ONaTX0GDF 1TYQTXlP0g0yrV4bkrw5Q9MrC+YBWDVgGRUsqLS5QJIwMjItaWRCLBKo8EuWJ4f5v3Nv nYSW33dH64ccQTND4q7LinORj8q+QnXBX5+ROg1/DhwGvQ4Aq7vEkei1Ai5Yv/X2V3XE S/OQ== X-Gm-Message-State: ABy/qLZ1PZzP5w4hq01twc9e4a5MkaLHod+AOBuUXUEIxcXNGX127k03 kGEVXVEglDmuFce3rDP0Usq/D3dEXSPLRlYBxNW1vvBw X-Received: by 2002:adf:ed4c:0:b0:313:f347:eea0 with SMTP id u12-20020adfed4c000000b00313f347eea0mr2351551wro.60.1689282891100; Thu, 13 Jul 2023 14:14:51 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:50 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 09/18] crypto: Add generic 32-bit carry-less multiply routines Date: Thu, 13 Jul 2023 22:14:26 +0100 Message-Id: <20230713211435.13505-10-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::32f; envelope-from=richard.henderson@linaro.org; helo=mail-wm1-x32f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Signed-off-by: Richard Henderson --- host/include/generic/host/crypto/clmul.h | 4 +++ include/crypto/clmul.h | 23 ++++++++++++++++++ crypto/clmul.c | 31 ++++++++++++++++++++++++ 3 files changed, 58 insertions(+) diff --git a/host/include/generic/host/crypto/clmul.h b/host/include/generic/host/crypto/clmul.h index cba8bbf3e4..3fbb1576cf 100644 --- a/host/include/generic/host/crypto/clmul.h +++ b/host/include/generic/host/crypto/clmul.h @@ -19,4 +19,8 @@ #define clmul_16x4_even clmul_16x4_even_gen #define clmul_16x4_odd clmul_16x4_odd_gen +#define clmul_32 clmul_32_gen +#define clmul_32x2_even clmul_32x2_even_gen +#define clmul_32x2_odd clmul_32x2_odd_gen + #endif /* GENERIC_HOST_CRYPTO_CLMUL_H */ diff --git a/include/crypto/clmul.h b/include/crypto/clmul.h index b701bac9d6..ce43c9aeb1 100644 --- a/include/crypto/clmul.h +++ b/include/crypto/clmul.h @@ -88,6 +88,29 @@ Int128 clmul_16x4_even_gen(Int128, Int128); */ Int128 clmul_16x4_odd_gen(Int128, Int128); +/** + * clmul_32: + * + * Perform a 32x32->64 carry-less multiply. + */ +uint64_t clmul_32_gen(uint32_t, uint32_t); + +/** + * clmul_32x2_even: + * + * Perform two 32x32->64 carry-less multiplies. + * The odd words of the inputs are ignored. + */ +Int128 clmul_32x2_even_gen(Int128, Int128); + +/** + * clmul_32x2_odd: + * + * Perform two 32x32->64 carry-less multiplies. + * The even words of the inputs are ignored. + */ +Int128 clmul_32x2_odd_gen(Int128, Int128); + #include "host/crypto/clmul.h" #endif /* CRYPTO_CLMUL_H */ diff --git a/crypto/clmul.c b/crypto/clmul.c index 69a3b6f7ff..c197cd5f21 100644 --- a/crypto/clmul.c +++ b/crypto/clmul.c @@ -113,3 +113,34 @@ Int128 clmul_16x4_odd_gen(Int128 n, Int128 m) rh = clmul_16x2_odd_gen(int128_gethi(n), int128_gethi(m)); return int128_make128(rl, rh); } + +uint64_t clmul_32_gen(uint32_t n, uint32_t m32) +{ + uint64_t r = 0; + uint64_t m = m32; + + for (int i = 0; i < 32; ++i) { + r ^= n & 1 ? m : 0; + n >>= 1; + m <<= 1; + } + return r; +} + +Int128 clmul_32x2_even_gen(Int128 n, Int128 m) +{ + uint64_t rl, rh; + + rl = clmul_32_gen(int128_getlo(n), int128_getlo(m)); + rh = clmul_32_gen(int128_gethi(n), int128_gethi(m)); + return int128_make128(rl, rh); +} + +Int128 clmul_32x2_odd_gen(Int128 n, Int128 m) +{ + uint64_t rl, rh; + + rl = clmul_32_gen(int128_getlo(n) >> 32, int128_getlo(m) >> 32); + rh = clmul_32_gen(int128_gethi(n) >> 32, int128_gethi(m) >> 32); + return int128_make128(rl, rh); +} From patchwork Thu Jul 13 21:14:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702244 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp742409wrj; Thu, 13 Jul 2023 14:18:55 -0700 (PDT) X-Google-Smtp-Source: APBJJlEP3WBVTirDhQsNSn4nOOsWvS6FCZae/PESLbNoPTYxE8fAD/8lqLbcLFPuKNXf+5H1/uXQ X-Received: by 2002:a05:620a:f0b:b0:75f:5df:1652 with SMTP id v11-20020a05620a0f0b00b0075f05df1652mr2805572qkl.68.1689283134905; Thu, 13 Jul 2023 14:18:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689283134; cv=none; d=google.com; s=arc-20160816; b=oPmiDS7XzIr2ZCk8DOsETcCCuP8kOH1R149PDmveZzRCOUumkQRE4sg8ChHC1MbRX4 0yrVeGYFl6h7D4Q9jptt2UvcEAQJnpzrq2MmfC/w4RsP8cnZWX80WKwKjn43l3nRIbvE r4GUxrBSaUw8z45pjOPwPkW9mRZXjjJ2ZRCoVrISFLzMJUkVANMeW7z+7Qg3DBobjQHf iNa3HHYrZmpzo+d7zm5yVbPFrn2J6+lqPbrjjzxura6s+f4GrL9EM3Wwfjh34ZqtGlap 0PVSOatmnIuQx4CTRZM/K44cQGKsuzNtGv2T2zF2qtf9NFhZ+fWzFrwo2u94fqIrc7Eq v4qw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=Fnmyc9DudD/G12QVt/hdI4bqnT7KT3GDxNaZP+FHbB4=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=ISX5mwEe/m/jqjgD4DezDAyNZZSHGdTQtoFfR/WsRxyLgssc6qTaDj9HUVApkCFB/G rX/l/PkUDD15ywTdNC8CQmNytGm20qR+4ld3GcR04kO1Mrd7K+xPRZ6ijx5w5Chfu3t6 qhnv3k7+IEdqUsxxON3SkvwRS69U+dw9SsDtrdTZ3qvclBnAVX7fDK6kNLHtqpBJKb1u nupizuH+RO8Sf6rLw5tj44MDCGKQc1IdSTgZLCePVRH1Fyya0Eeq8DPNHLHBoHHwXQx+ 6122IL0AFuLOv8Ciq7qOzikCbkbcutAVduNJAfF5JpAYUGku1ZDSuI4UB3zdsYCQPXlr BwzQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ROlyrFZu; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id g9-20020a05620a40c900b00765368d3156si3690404qko.624.2023.07.13.14.18.54 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:18:54 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ROlyrFZu; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3eM-0000Ju-4o; Thu, 13 Jul 2023 17:14:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eI-0000Ib-U9 for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:54 -0400 Received: from mail-wm1-x32c.google.com ([2a00:1450:4864:20::32c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3eH-00017Y-9c for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:54 -0400 Received: by mail-wm1-x32c.google.com with SMTP id 5b1f17b1804b1-3fbc63c2e84so9863475e9.3 for ; Thu, 13 Jul 2023 14:14:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282892; x=1691874892; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Fnmyc9DudD/G12QVt/hdI4bqnT7KT3GDxNaZP+FHbB4=; b=ROlyrFZuJgP+QGcoz0Wqn0TvnKcKqg4P2HwOCGvUzbpwlwS+VAIhC0sH8CJyeSw2MH EW4W7P6iFwgVadlbtoL+o2GSrLs7XSep3PQPRwAxtyS7f4Xyv/+w3BrNpbB29DTWqLMJ 2sCzWtAyVcNkgHZTcs8uTCPUDvc9taur65X/o/Xz3sSte5R3f8qXTjbGFga9251n/y6l hgOyUKE7NuPJeaZpNRw7m/RhUhS6R/P5JbrywP97QOgpZCJPWh4yPHNbE/zniuN8ss/i Ro5zjycV6Vudwld7BT/7JA9lPylDxkqvaxEYSpX/kwE4KhEVWcS/iRKnQ+CDSD4jxZw+ UGqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282892; x=1691874892; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Fnmyc9DudD/G12QVt/hdI4bqnT7KT3GDxNaZP+FHbB4=; b=MhrXTeJoje/HjmVA3kSR2Z4TXF9T5Ywc7G1zKfT1bCRZDnMHogJ2MWXYnxLkwrpbXM zlIzKU+4X7PheuSztEkSzxyqcRwCioBODbC02/TTPO4tyb/6GZfLn6/ETK6f4fBo2wni Tv2g3Ietr9pHhU7Cs8T4bEklBjG7CWQhjVWzp/YRPAN+SVqq/B6mFkw1EPSQ9nu3kGRx pIjxcw+JdSWdRy3EDWfYTEbcYCdQcC0+4M/1bP2EXgnlnzv6iGH24pza9J/mWvnJ5rWM hEzy+84wWXcIX2fpoBJMmqQY/Bil16eMpcJd8VHxvDidY6F6mXRMVisBIt/G9kOFqytZ zSZA== X-Gm-Message-State: ABy/qLYNYdrNsQU2oG79A7gQrLnZEejtbEVjdR4CxITAh7MTPyN2YFQL IRkuIjbRJNtdUnOmJ5K3k0p4E80AYnnXj+QtKBPqxmtu X-Received: by 2002:a7b:ce0f:0:b0:3fa:964e:e85 with SMTP id m15-20020a7bce0f000000b003fa964e0e85mr2810554wmc.5.1689282891832; Thu, 13 Jul 2023 14:14:51 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:51 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 10/18] target/arm: Use clmul_32* routines Date: Thu, 13 Jul 2023 22:14:27 +0100 Message-Id: <20230713211435.13505-11-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::32c; envelope-from=richard.henderson@linaro.org; helo=mail-wm1-x32c.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Use generic routines for 32-bit carry-less multiply. Remove our local version of pmull_d. Signed-off-by: Richard Henderson --- target/arm/tcg/vec_helper.c | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/target/arm/tcg/vec_helper.c b/target/arm/tcg/vec_helper.c index 1b1d5fccbc..c81447e674 100644 --- a/target/arm/tcg/vec_helper.c +++ b/target/arm/tcg/vec_helper.c @@ -2057,18 +2057,6 @@ void HELPER(sve2_pmull_h)(void *vd, void *vn, void *vm, uint32_t desc) } } -static uint64_t pmull_d(uint64_t op1, uint64_t op2) -{ - uint64_t result = 0; - int i; - - for (i = 0; i < 32; ++i) { - uint64_t mask = -((op1 >> i) & 1); - result ^= (op2 << i) & mask; - } - return result; -} - void HELPER(sve2_pmull_d)(void *vd, void *vn, void *vm, uint32_t desc) { intptr_t sel = H4(simd_data(desc)); @@ -2077,7 +2065,7 @@ void HELPER(sve2_pmull_d)(void *vd, void *vn, void *vm, uint32_t desc) uint64_t *d = vd; for (i = 0; i < opr_sz / 8; ++i) { - d[i] = pmull_d(n[2 * i + sel], m[2 * i + sel]); + d[i] = clmul_32(n[2 * i + sel], m[2 * i + sel]); } } #endif From patchwork Thu Jul 13 21:14:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702231 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp740807wrj; Thu, 13 Jul 2023 14:15:55 -0700 (PDT) X-Google-Smtp-Source: APBJJlF1X1lwDYxZnbYnu98Kl3daxxKbhvyaO+z+CLAPhCkFhJuR3n1GPVghfEbAnCUgc9CJosgi X-Received: by 2002:a05:622a:148:b0:400:7de9:635a with SMTP id v8-20020a05622a014800b004007de9635amr3423710qtw.57.1689282955684; Thu, 13 Jul 2023 14:15:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689282955; cv=none; d=google.com; s=arc-20160816; b=eqevxVbj0XzeNFcA8+9wb6bvtmlZ9m8f3NaCYvgCJ6Hys02dsumLsdKzE/tKtAm601 qsEggTNoqK9jjZmSnAtqv4xFAjVBjJMehdleQ5CTLNPWyzCUbymplqutLoaS4kSs5QjX Qqn/Ap4F6bXXz504aZFjCubDj/jlYxBixpm062xk0F+YjO8rpKBDmYrk8XA4koxonVMS YI/jvwgVXCIzwwuaLxT7cEE6eubfqbjTGhJD9f08tJJFr7xRkUlP3vgIvRDJ8MG2uoll PBD6LunJ4MJ1X8f38a0hhWQg6LQhQ+dPwVoCh+FMALzv7QESMjhpB1jEcQOeDpHS7rnO rccw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=Af+EKJ8ZN4mASGCOe+AcfD/joyYUaDCEWZVqXWvkjKU=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=hxQ3Wp0CTD2NRsRT/tQEjsdKodenHWFpyuLIm8AiRsnR4bWRUNYJH/KgbSXYQIfCFM Kqw83vLiqWTlA66isRsR1qecYuCRtA3E0EVxMbKj4rtwTE16OqwjtCgQ6crcK+BEAy8w p+9zZP0nI5lVpgJlbidkXA78AN6EFs0ABJSaWw2XhGEmHERKJffWJVROFcBB3bJtci2L rFEnZeftwprOM2Qi573SP/CBv47dHwqmt815MjEfCz+WP1BpcfoyHkMzoq8adn1FkLvE 4QyjktqD4Qy0oQ1+LESCTJgrI4DtQNsboCpmaYAJVa9C1Houvue+JhaZvdhXiS3ZSrzw Nbsg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Kgk868F9; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id h18-20020ac87d52000000b00403b84be504si3408358qtb.580.2023.07.13.14.15.55 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:15:55 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Kgk868F9; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3eV-0000Ly-9f; Thu, 13 Jul 2023 17:15:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eJ-0000J7-Vk for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:56 -0400 Received: from mail-wr1-x430.google.com ([2a00:1450:4864:20::430]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3eI-00017x-CO for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:55 -0400 Received: by mail-wr1-x430.google.com with SMTP id ffacd0b85a97d-316eabffaa6so795136f8f.2 for ; Thu, 13 Jul 2023 14:14:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282893; x=1691874893; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Af+EKJ8ZN4mASGCOe+AcfD/joyYUaDCEWZVqXWvkjKU=; b=Kgk868F9D3WQqXnTIH4bYiv9vobnCbu/5H+mHKO6g72pnhiIfww7SV590eCybpaKZu rcttQ34S1VHEwAtj5lHIU1i9oru3StKvmYFgTTgPqNFTMo9OC4NmOSRWfdYnEkP/MPfy hDBNkoyfLygNG3vXBZtSN2TNV1POJSYQykOjkFRDiDxnGtjQHyAv1jTK/BsHSxyiLSYK U/1Iv8Qs8j6VrLY3MjLpX+80VasyOFWC61hCWAxn6G1pVBvoUedK0E6LW1XDNegoNaJG ZaFTIrLnRYbO6ADkhn0MzKx+zoyvhgHwkVA8RnsJhOOvMmbPYHWszNxuxosnV5Ur6NKB 8OkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282893; x=1691874893; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Af+EKJ8ZN4mASGCOe+AcfD/joyYUaDCEWZVqXWvkjKU=; b=I9HRFNKNTdCx31Deh/1zyQFu6IY01gUm3o7ThInBfkax2kVHUqWcneUSoXOdQSiqbF oEz27ezbbsO61xsvV8RX0giLVZUce3KIDDKiJAetKTuCREY3XUK5DK9JfjD4w/jZnaTL mc+Um+AbufeGFolnegRtGkuI0XbGopo+E0yRWwVAk5UDTSxZEthfHZwXT4QIseb+gmbv W2p6n+7/uEYUvYzwx2ctVNdKRlikKGD/2xdH+5c5rdDgUCw6TRkRgV0BOaJUr+E5A9tN ikK2IW4MYSAtYREsE6i36LdoP9j+iGBoVg65uGG8p7CjLrFxCahGipByyPEX4Z8zNSU+ JNrw== X-Gm-Message-State: ABy/qLa+7/eXKSf0nu6cZ8Ue/NUrFMdorTD83kMDAgNu1I5RrBp1H0oU TlmmmPuRUt1Jc53JAI0W1QPcCEXigUAunKk/ZP9SKgop X-Received: by 2002:a5d:4482:0:b0:314:2b0a:dabe with SMTP id j2-20020a5d4482000000b003142b0adabemr2268057wrq.30.1689282892950; Thu, 13 Jul 2023 14:14:52 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:52 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 11/18] target/s390x: Use clmul_32* routines Date: Thu, 13 Jul 2023 22:14:28 +0100 Message-Id: <20230713211435.13505-12-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::430; envelope-from=richard.henderson@linaro.org; helo=mail-wr1-x430.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Use generic routines for 32-bit carry-less multiply. Remove our local version of galois_multiply32. Signed-off-by: Richard Henderson --- target/s390x/tcg/vec_int_helper.c | 70 ++++++++----------------------- 1 file changed, 17 insertions(+), 53 deletions(-) diff --git a/target/s390x/tcg/vec_int_helper.c b/target/s390x/tcg/vec_int_helper.c index 523d6375bb..f5eea2330a 100644 --- a/target/s390x/tcg/vec_int_helper.c +++ b/target/s390x/tcg/vec_int_helper.c @@ -165,22 +165,6 @@ DEF_VCTZ(8) DEF_VCTZ(16) /* like binary multiplication, but XOR instead of addition */ -#define DEF_GALOIS_MULTIPLY(BITS, TBITS) \ -static uint##TBITS##_t galois_multiply##BITS(uint##TBITS##_t a, \ - uint##TBITS##_t b) \ -{ \ - uint##TBITS##_t res = 0; \ - \ - while (b) { \ - if (b & 0x1) { \ - res = res ^ a; \ - } \ - a = a << 1; \ - b = b >> 1; \ - } \ - return res; \ -} -DEF_GALOIS_MULTIPLY(32, 64) static S390Vector galois_multiply64(uint64_t a, uint64_t b) { @@ -244,24 +228,24 @@ void HELPER(gvec_vgfma16)(void *v1, const void *v2, const void *v3, *(Int128 *)v1 = int128_xor(r, *(Int128 *)v4); } -#define DEF_VGFM(BITS, TBITS) \ -void HELPER(gvec_vgfm##BITS)(void *v1, const void *v2, const void *v3, \ - uint32_t desc) \ -{ \ - int i; \ - \ - for (i = 0; i < (128 / TBITS); i++) { \ - uint##BITS##_t a = s390_vec_read_element##BITS(v2, i * 2); \ - uint##BITS##_t b = s390_vec_read_element##BITS(v3, i * 2); \ - uint##TBITS##_t d = galois_multiply##BITS(a, b); \ - \ - a = s390_vec_read_element##BITS(v2, i * 2 + 1); \ - b = s390_vec_read_element##BITS(v3, i * 2 + 1); \ - d = d ^ galois_multiply32(a, b); \ - s390_vec_write_element##TBITS(v1, i, d); \ - } \ +static Int128 do_gfm32(Int128 n, Int128 m) +{ + Int128 e = clmul_32x2_even(n, m); + Int128 o = clmul_32x2_odd(n, m); + return int128_xor(e, o); +} + +void HELPER(gvec_vgfm32)(void *v1, const void *v2, const void *v3, uint32_t d) +{ + *(Int128 *)v1 = do_gfm32(*(const Int128 *)v2, *(const Int128 *)v3); +} + +void HELPER(gvec_vgfma32)(void *v1, const void *v2, const void *v3, + const void *v4, uint32_t d) +{ + Int128 r = do_gfm32(*(const Int128 *)v2, *(const Int128 *)v3); + *(Int128 *)v1 = int128_xor(r, *(Int128 *)v4); } -DEF_VGFM(32, 64) void HELPER(gvec_vgfm64)(void *v1, const void *v2, const void *v3, uint32_t desc) @@ -278,26 +262,6 @@ void HELPER(gvec_vgfm64)(void *v1, const void *v2, const void *v3, s390_vec_xor(v1, &tmp1, &tmp2); } -#define DEF_VGFMA(BITS, TBITS) \ -void HELPER(gvec_vgfma##BITS)(void *v1, const void *v2, const void *v3, \ - const void *v4, uint32_t desc) \ -{ \ - int i; \ - \ - for (i = 0; i < (128 / TBITS); i++) { \ - uint##BITS##_t a = s390_vec_read_element##BITS(v2, i * 2); \ - uint##BITS##_t b = s390_vec_read_element##BITS(v3, i * 2); \ - uint##TBITS##_t d = galois_multiply##BITS(a, b); \ - \ - a = s390_vec_read_element##BITS(v2, i * 2 + 1); \ - b = s390_vec_read_element##BITS(v3, i * 2 + 1); \ - d = d ^ galois_multiply32(a, b); \ - d = d ^ s390_vec_read_element##TBITS(v4, i); \ - s390_vec_write_element##TBITS(v1, i, d); \ - } \ -} -DEF_VGFMA(32, 64) - void HELPER(gvec_vgfma64)(void *v1, const void *v2, const void *v3, const void *v4, uint32_t desc) { From patchwork Thu Jul 13 21:14:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702236 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp741674wrj; Thu, 13 Jul 2023 14:17:18 -0700 (PDT) X-Google-Smtp-Source: APBJJlHFaT+TsYACeZfuIuTwA0Kd8T0JYSObmrACKqFZu/9npbNSqAVssgrqrBG8BR2+ta/y0Ncw X-Received: by 2002:a05:6102:3026:b0:443:8eab:c664 with SMTP id v6-20020a056102302600b004438eabc664mr2116745vsa.13.1689283037948; Thu, 13 Jul 2023 14:17:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689283037; cv=none; d=google.com; s=arc-20160816; b=hfCZ+cNehkie3v8hZZzORye6IXWIOJGkH9DfrQWyXHhDMmD1jIgWllWrXggQsDWmos MFrZj3EJRXBszvFcR2QNN8uammuAI/wPfzc4XU7tHI572FMFsTD/QXUujOMw3wSMtQeP x14UgDV0gGL++oBGRT15Z5g9Smg91vMrYDnwennULWbQPmmyVNWHgcxYlb04MKXz0efI un9OZVULGJ4DTOsYzJrTLZfsZhZzCyvEHfLH/i6rgyqFG6XOxl3RD9wIwiPNLSQwwoGJ OEZ0ff9QGO0nNmyCTVAl7qdkLTyAi2yN1iWSIRdQAHLklDg2ho5O8ZY1yRwVhTbE8cSu IUIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=nIVnJ7xXPoFHeKhRtV8u1nP3hp+hYXvngLOg/MOoyhs=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=apIGSV0Sj9nkjhYfNlCmJk8aWJ8teniIRu8m67Ou9iv9kP904GclDpyTuxrUTj1JqR 1Dj4xxs/yJWEc7jxSWcKOsHrXA567gHmJZeQ812FS0gH1N3nDMUNP1jvOTZywyFQT+a+ okoRWmJGGm1UxTIb5Fw+JqI87Kx5Bv4PGf9adwlDi6+9zyQeTXSs9Zd/0UP9YnwMJfON vGSenN+lKUCIaK2qNZL0ucQNjMUYy57TUHx43woy9E/G8M7KovCmNZNWj1U6T77k+7Me rwgjnA/FFUHsGkNsTDqttgOV+0ahnHkhrjvsl+SYyaraIQw/DiaOCNyRnPb0TR8S82ca IckQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=fLtEgnnc; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id bn39-20020a05620a2ae700b00767ba3c23ecsi3622152qkb.284.2023.07.13.14.17.17 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:17:17 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=fLtEgnnc; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3eU-0000Lv-HR; Thu, 13 Jul 2023 17:15:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eK-0000JI-NA for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:56 -0400 Received: from mail-wm1-x330.google.com ([2a00:1450:4864:20::330]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3eI-000186-RR for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:56 -0400 Received: by mail-wm1-x330.google.com with SMTP id 5b1f17b1804b1-3fbf1b82de7so7720355e9.1 for ; Thu, 13 Jul 2023 14:14:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282893; x=1691874893; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nIVnJ7xXPoFHeKhRtV8u1nP3hp+hYXvngLOg/MOoyhs=; b=fLtEgnncCjkpJKMR5EUcqTThok6V4283ZQNWuxOUcnZGvSQ7KnVx6WbaRA7Rz5w5c9 MHv8ot8qQm0NE/bsjxse0JJq/hO/36lOwEnnU+SEJu0tzWiAccmrl4Fm525s5f1o7CtD k3+dvDaJMkbVUvxtC3tk0cRoyLQ+QE8rQzLAFjDPumdBmwfWg3N2jSMjG4ZZwZRP9xGY /xaSvSY0yNORREHO7R/ARuZTe5XXwjK2S4h6ClOws1A8KUGYI1EcF0TcPYC7rH9DA4fe sEfSFCZAp0ksx43P7ybIlTIC9ah6brtPvPO0/MgBG7um3w+HJ/XCJdiKfP4u00pIe/8w 251Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282893; x=1691874893; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nIVnJ7xXPoFHeKhRtV8u1nP3hp+hYXvngLOg/MOoyhs=; b=NA5VY3OG1PkBz3z0BJj+CNbLMyiR1NlgPyZzyhph3KmQL0SK+HubqaEEbIgFyjZQIp /qETjg93mObT10X+FX+8vsEO9Jppqu+A4xgfNDWMPee+U5FwajPEyCTInwWI18olUq0W FmWQrgkff1uASlcoWI0dJn/fOV1iHtMtKpWlyi4HMsh/5Vr97MfkexhWpVhdWUACaAH6 CWUSQLEd7Lt9kRKlIOCft4iO1wTz2jmcXplo3gu6jpoPtz4oFn8oNXjFHIFkHuSUGr+H CDeh6fvd2yqRxlaNY+PktpBHaHByg9Xc99sA88sX1M7ntpi9pd/i/HaMR9zlRqCHNzXQ R79Q== X-Gm-Message-State: ABy/qLbUgP2f6QuxSG2Z0KCyCNzoNvWkDoYy0b7bVAMouJ/61SUEAfkS l7BgoLvCmKzPs1ybuVcuPBMadUY2Lv1zDdMumRQs+Fet X-Received: by 2002:a7b:ce87:0:b0:3fa:8422:158d with SMTP id q7-20020a7bce87000000b003fa8422158dmr560883wmj.18.1689282893616; Thu, 13 Jul 2023 14:14:53 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:53 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 12/18] target/ppc: Use clmul_32* routines Date: Thu, 13 Jul 2023 22:14:29 +0100 Message-Id: <20230713211435.13505-13-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::330; envelope-from=richard.henderson@linaro.org; helo=mail-wm1-x330.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Use generic routines for 32-bit carry-less multiply. Signed-off-by: Richard Henderson --- target/ppc/int_helper.c | 27 +++++++-------------------- 1 file changed, 7 insertions(+), 20 deletions(-) diff --git a/target/ppc/int_helper.c b/target/ppc/int_helper.c index 98d6310f59..828f04bce7 100644 --- a/target/ppc/int_helper.c +++ b/target/ppc/int_helper.c @@ -1444,28 +1444,15 @@ void helper_vpmsumh(ppc_avr_t *r, ppc_avr_t *a, ppc_avr_t *b) r->s128 = int128_xor(e, o); } -#define PMSUM(name, srcfld, trgfld, trgtyp) \ -void helper_##name(ppc_avr_t *r, ppc_avr_t *a, ppc_avr_t *b) \ -{ \ - int i, j; \ - trgtyp prod[sizeof(ppc_avr_t) / sizeof(a->srcfld[0])]; \ - \ - VECTOR_FOR_INORDER_I(i, srcfld) { \ - prod[i] = 0; \ - for (j = 0; j < sizeof(a->srcfld[0]) * 8; j++) { \ - if (a->srcfld[i] & (1ull << j)) { \ - prod[i] ^= ((trgtyp)b->srcfld[i] << j); \ - } \ - } \ - } \ - \ - VECTOR_FOR_INORDER_I(i, trgfld) { \ - r->trgfld[i] = prod[2 * i] ^ prod[2 * i + 1]; \ - } \ +void helper_vpmsumw(ppc_avr_t *r, ppc_avr_t *a, ppc_avr_t *b) +{ + Int128 ia = a->s128; + Int128 ib = b->s128; + Int128 e = clmul_32x2_even(ia, ib); + Int128 o = clmul_32x2_odd(ia, ib); + r->s128 = int128_xor(e, o); } -PMSUM(vpmsumw, u32, u64, uint64_t) - void helper_VPMSUMD(ppc_avr_t *r, ppc_avr_t *a, ppc_avr_t *b) { int i, j; From patchwork Thu Jul 13 21:14:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702230 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp740709wrj; Thu, 13 Jul 2023 14:15:40 -0700 (PDT) X-Google-Smtp-Source: APBJJlGBcjLCGGSpWb6chklwpmLXOZGVCRLQM72W3R8pdGQ6I/tWV9Aaex7s0521eRsoqqpdoqUJ X-Received: by 2002:a05:622a:2d6:b0:403:9abf:805c with SMTP id a22-20020a05622a02d600b004039abf805cmr3351984qtx.40.1689282940736; Thu, 13 Jul 2023 14:15:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689282940; cv=none; d=google.com; s=arc-20160816; b=zmbcXbpxq6cxkQ/To7t4bvbZBK4tBolSJgETnFbNuDZ58my0DWlLf4DXHZOzp09cLp 2QJdW8PJyZi+BKr88bq45DQO/xcnWegQVejR4wAN+ag/ORKzARQIdmjpDmckRxaeXlUM EN9sqnC4XZ+LYbxtnph/6TLiAx+pwiEVwkOx7hqZrVVXKFov5tExiybFuAilEF9uhqsw y9BKaTBb6Qquf+zaWnYd5y4N9FB2dIo7etP8nZ7Fxoixpwu6myaU9g+J3WEloEP4AVXi cxHY4oidcroMO5QDpT1Ypk6W9fwE/gC1Y7k5eXeQxKqGXFFbEG8aaIk1g1jFDWiARC4d iF4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=xK0qTtWWEN2PFwA7pUogqEsxDTZZLq0zFIIqlW+dA/A=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=TrTN5Gw46zuGCWOAE6qWMKHYneGTKflOdu2mMbVz7wK6/Y12WS8JEei3y5yu4A//F4 vI09HEv3uV5opau+y68WkStQt9A7I+QFES43vzWSbBsh3/YxkvDrRIeA6kbQnPl15cRO 8BBB6BwGzwwUKBDF4IlJYBbdLazs5TD3LCoM3KAjIpyyPAV1ycUKb2w01G3iu2s0hee1 +bPN7m/b2Bl7pxROFCbHo762YOn7ilVrJ6VnARGJhOwaLywbvRa8BP2sIjdwkwv3uqAx 0ktAcNl2fChVpuvK36ms3ijPXfoAdmLTbqN+SGtdCiHROLisnNkQrWvciHrIc/V2+/39 EekA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=OGUieX20; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id g19-20020ac85813000000b00400ed9e712fsi3636975qtg.113.2023.07.13.14.15.40 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:15:40 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=OGUieX20; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3ea-0000Q8-0W; Thu, 13 Jul 2023 17:15:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eL-0000Je-VQ for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:57 -0400 Received: from mail-wm1-x333.google.com ([2a00:1450:4864:20::333]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3eK-00018V-4k for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:57 -0400 Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-3fbef8ad9bbso10026315e9.0 for ; Thu, 13 Jul 2023 14:14:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282895; x=1691874895; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xK0qTtWWEN2PFwA7pUogqEsxDTZZLq0zFIIqlW+dA/A=; b=OGUieX20zuHlPj8X1Z1sK2vXL0FyybB9otlgkVcNVaxMWdnatdBb0FvZMBIzSywf8w VzjG/NAieG3LtYYfzuCY98LoWyHEjVvvDXTHBk5ZB9xoqCKLHsj3MXVA0ZSFg0x3ZxFv lTznQ+m1jgfjfAG7L3gUxzuPOzObIO3XsL5s64VSV3j/Slf+sfaBwD4d9fp6FlGWFVvT yGeOyn4x5oyyN7MnQzD5yaedzqCKAl5CtxUKtdbpMMQPF3GG55WjhOeRPqe9mcRq5peq kYb/zpRh/wV69MRP6I4lkVkfBUO90xfbLQFBYVMdM7Ko04e0CzFl+CAHWWafvTu7wD4+ 8FuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282895; x=1691874895; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xK0qTtWWEN2PFwA7pUogqEsxDTZZLq0zFIIqlW+dA/A=; b=QVogE1p+faw5Bs/LwmA3iThVHb7o2lZINgBiJYGZYtg5WAAG6O/rcq+D+rCSQLTxgF V1ZoVfi3ETAY4/TfgPi6Jo5/w19Gy0+IXOOhzIc6d0xuxGhmCmfO/uZLqQugi4lv2LNG jnzIa/5qIvbC1KdSsSpfHNSYN7MCEpB3352fwPCy2jm+WWE0XISoW+ufzAc690hcijbl cDYu8Ym4EBPY/4wjup1aIKyaPQcR5PAka+88lgu4TCcZM0R5T06LITFGuXGKYo9XBVlU WlaqWCs1540DueELz9rIz5GTVtSNN3YOhmNVX80fDNUCIJ8BvpEYPU97F5K+Dqjxy3PY Vedg== X-Gm-Message-State: ABy/qLa0d/tMo2xng/TZcPliGXZoH6HWyJYBkSSSPax0huyu2UXp+4D+ j6R7zb9PiPVkjZtT3yxjkMp8QmruHUJ4e3hbVik/hvws X-Received: by 2002:adf:e483:0:b0:314:db7:d132 with SMTP id i3-20020adfe483000000b003140db7d132mr2544451wrm.61.1689282894830; Thu, 13 Jul 2023 14:14:54 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:54 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 13/18] crypto: Add generic 64-bit carry-less multiply routine Date: Thu, 13 Jul 2023 22:14:30 +0100 Message-Id: <20230713211435.13505-14-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::333; envelope-from=richard.henderson@linaro.org; helo=mail-wm1-x333.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Signed-off-by: Richard Henderson --- host/include/generic/host/crypto/clmul.h | 2 ++ include/crypto/clmul.h | 7 +++++++ crypto/clmul.c | 17 +++++++++++++++++ 3 files changed, 26 insertions(+) diff --git a/host/include/generic/host/crypto/clmul.h b/host/include/generic/host/crypto/clmul.h index 3fbb1576cf..7f70afeb57 100644 --- a/host/include/generic/host/crypto/clmul.h +++ b/host/include/generic/host/crypto/clmul.h @@ -23,4 +23,6 @@ #define clmul_32x2_even clmul_32x2_even_gen #define clmul_32x2_odd clmul_32x2_odd_gen +#define clmul_64 clmul_64_gen + #endif /* GENERIC_HOST_CRYPTO_CLMUL_H */ diff --git a/include/crypto/clmul.h b/include/crypto/clmul.h index ce43c9aeb1..8b4c263459 100644 --- a/include/crypto/clmul.h +++ b/include/crypto/clmul.h @@ -111,6 +111,13 @@ Int128 clmul_32x2_even_gen(Int128, Int128); */ Int128 clmul_32x2_odd_gen(Int128, Int128); +/** + * clmul_64: + * + * Perform a 64x64->128 carry-less multiply. + */ +Int128 clmul_64_gen(uint64_t, uint64_t); + #include "host/crypto/clmul.h" #endif /* CRYPTO_CLMUL_H */ diff --git a/crypto/clmul.c b/crypto/clmul.c index c197cd5f21..0be06073f0 100644 --- a/crypto/clmul.c +++ b/crypto/clmul.c @@ -144,3 +144,20 @@ Int128 clmul_32x2_odd_gen(Int128 n, Int128 m) rh = clmul_32_gen(int128_gethi(n) >> 32, int128_gethi(m) >> 32); return int128_make128(rl, rh); } + +Int128 clmul_64_gen(uint64_t n, uint64_t m) +{ + uint64_t rl = 0, rh = 0; + + /* Bit 0 can only influence the low 64-bit result. */ + if (n & 1) { + rl = m; + } + + for (int i = 1; i < 64; ++i) { + uint64_t mask = -((n >> i) & 1); + rl ^= (m << i) & mask; + rh ^= (m >> (64 - i)) & mask; + } + return int128_make128(rl, rh); +} From patchwork Thu Jul 13 21:14:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702233 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp741485wrj; Thu, 13 Jul 2023 14:16:54 -0700 (PDT) X-Google-Smtp-Source: APBJJlHPBvzlnPg+k2UA00fs7oEgsdzXHmMej+Dy3T8EUenu36L6AAZi92U21ktdhMoT+keVSnnz X-Received: by 2002:a05:622a:412:b0:3ff:2a6b:5a83 with SMTP id n18-20020a05622a041200b003ff2a6b5a83mr4356807qtx.10.1689283013941; Thu, 13 Jul 2023 14:16:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689283013; cv=none; d=google.com; s=arc-20160816; b=y7NWNTqdCnL3CHw0OvDhakmaNf46SlqsPa2H+9b/biJKCdENrdoSk0jJhQ8d2aBQI1 8V6h6E5TuQD2h38+t8UHRywsuERu/XHGySZsc85k3EWdWdf/kEpwu1ipJa9dhRvwfcNL Id7598icQZ8wQZCRhdUGpopAiivFXnST37FL1otHwykElmZBNKzxvAgYE8VSlk0bvz+w LEist0zboFJZcNlrQGwlbx2sw8GStf9nbcSn/F4d936fWkAr3liJYRfFw9TjN9/Mx6C4 /QVwLwsVVNgCRn37GivzNiAddJvKuVP9TvAgpvyD2+AVKqzmZtT/wvq7U6STGIz5i1zi 46Ug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=O1koV3EZIffRqpF6kXzdOYGupm/qybouRR/ROCDFTB8=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=Lz/pZ+6eSpXxyiL4qmiP1b0Cs1wOa8f06YAUZbIBIky9MjlaKcDzaXR/Ksjse67VCh VgvSk6D+GCYF9QCdr/tOeIWNGNBTfSmh6mO+9NHR8otnuFxqmeRDOoQOMqbxEqZ944h5 AiTF0heNK2uxzoNeuETPwdNWeKwkD5ozvSBZRmIfEYhsraHGzx2yXnMufWOsad8F6aOM e/SeDiZGAIKx9a8/Di0PFf3a6hI3/qYj7jHeWEiA6+3XLUSlKqW4P2R6WIQM3Tl6y+oy 3a7VeR6U4TLvxxb8wa0R89EsrzTHD2HjiD11j1fI84aFG/Q0E0tLxh84d4uYQM+oefaG ZURA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=dF0eVDXN; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id d16-20020a05622a15d000b0040091263f0esi3566445qty.708.2023.07.13.14.16.53 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:16:53 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=dF0eVDXN; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3ej-0000S0-Mz; Thu, 13 Jul 2023 17:15:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eM-0000K3-FU for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:58 -0400 Received: from mail-wm1-x32f.google.com ([2a00:1450:4864:20::32f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3eK-00018i-U3 for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:58 -0400 Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-3fbea14700bso11059565e9.3 for ; Thu, 13 Jul 2023 14:14:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282895; x=1691874895; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O1koV3EZIffRqpF6kXzdOYGupm/qybouRR/ROCDFTB8=; b=dF0eVDXN1HFujTd60wEqCwxFXFel0kx9yNUNo5g23hWIK75gy7VusAx46n+tMJ38pH /N/+VOFcq71qiBjsdCdYYVixf0RLdn82vZxtnrV3FaHyXdewMhlsby80wBKurIo1XBYW CXcl3I02pdMHP6x9z4pAWen7MzRSNJxaTlo+qq3FvMWksq9prPAStLwgq2FjtD0oErNj 50GB4W/bYTpsYHG/rBezlIw1QIsAiPnAvLfyEckgbXk0bibCEezR/IfGRaPWLoE5qIjz zSJOenUFM9zavspuhtcPlxRhZXw6Fcn26xfybYYU487ckl8ZDk2jiYa5sxKTGOFMlN4q E8rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282895; x=1691874895; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O1koV3EZIffRqpF6kXzdOYGupm/qybouRR/ROCDFTB8=; b=L0ZooBqZBQ5n64MkOaf/MD+WZBDmbKFSFhTsxkHGHGINaIcNxbHwlWHMeJmF8u8HKx af+zAfdZNyyfpc+Sgrt3t398FzVsrXj6hWonGNs8UdiIZn17WsNdh67zxcPIo4etdJ7d oDK8o93gXV4drEbGsoT4G7pRZrfEEK3IOY2FwgGzIQ8LNpiujf/BZ+uwEM7q+uaIaAl3 P1osChBjgDY2YPVT/iZ1eI3Ms6P5vbXXIGTW1lC31A6D/8AIAH683VPdmQ5ktzTkkDxp iY/5VNv1DUp3WzPJUTppkXK2cjLrAWh0VVeJr/npVKDq5xqn8aH1h4A2qEmFe24AU9uQ PMjw== X-Gm-Message-State: ABy/qLah9RUaQTG7Y9HzguLirW9F+iwMORY9TYs4m3p/5Hj7HHNqOONW V2nlqoWsblLPzy+XhMha4eAViG0OypkWDY21E1G76Y0B X-Received: by 2002:a7b:cb10:0:b0:3fc:21:2c43 with SMTP id u16-20020a7bcb10000000b003fc00212c43mr2547618wmj.13.1689282895738; Thu, 13 Jul 2023 14:14:55 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:55 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 14/18] target/arm: Use clmul_64 Date: Thu, 13 Jul 2023 22:14:31 +0100 Message-Id: <20230713211435.13505-15-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::32f; envelope-from=richard.henderson@linaro.org; helo=mail-wm1-x32f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Use generic routine for 64-bit carry-less multiply. Signed-off-by: Richard Henderson --- target/arm/tcg/vec_helper.c | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/target/arm/tcg/vec_helper.c b/target/arm/tcg/vec_helper.c index c81447e674..1a21aff4d9 100644 --- a/target/arm/tcg/vec_helper.c +++ b/target/arm/tcg/vec_helper.c @@ -2003,28 +2003,14 @@ void HELPER(gvec_pmul_b)(void *vd, void *vn, void *vm, uint32_t desc) */ void HELPER(gvec_pmull_q)(void *vd, void *vn, void *vm, uint32_t desc) { - intptr_t i, j, opr_sz = simd_oprsz(desc); + intptr_t i, opr_sz = simd_oprsz(desc); intptr_t hi = simd_data(desc); uint64_t *d = vd, *n = vn, *m = vm; for (i = 0; i < opr_sz / 8; i += 2) { - uint64_t nn = n[i + hi]; - uint64_t mm = m[i + hi]; - uint64_t rhi = 0; - uint64_t rlo = 0; - - /* Bit 0 can only influence the low 64-bit result. */ - if (nn & 1) { - rlo = mm; - } - - for (j = 1; j < 64; ++j) { - uint64_t mask = -((nn >> j) & 1); - rlo ^= (mm << j) & mask; - rhi ^= (mm >> (64 - j)) & mask; - } - d[i] = rlo; - d[i + 1] = rhi; + Int128 r = clmul_64(n[i + hi], m[i + hi]); + d[i] = int128_getlo(r); + d[i + 1] = int128_gethi(r); } clear_tail(d, opr_sz, simd_maxsz(desc)); } From patchwork Thu Jul 13 21:14:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702234 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp741492wrj; Thu, 13 Jul 2023 14:16:54 -0700 (PDT) X-Google-Smtp-Source: APBJJlFmVILPOKX79kjuPL2ltsNw8ds7/1vZGUf6JKLzMApd+aa2S7vLgUxx4FsscCiTdVCCGqTO X-Received: by 2002:ac8:7d94:0:b0:403:afe6:c173 with SMTP id c20-20020ac87d94000000b00403afe6c173mr3975370qtd.8.1689283014573; Thu, 13 Jul 2023 14:16:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689283014; cv=none; d=google.com; s=arc-20160816; b=KK2nKBjVoDA/bdQWfBt14IO3kMn3A/Mil7POslatsdtnl/6e4GbV+C0l66oqoS6URT W2TxLxgRYc+yNA3WJidHP9/4mY8GsiHoFrcHAuEnTtVNP/K24H2lvGMCrbgBUYwwOyxC /L728g+8Rl0OJs+CdcsJAfdTDTzJVRD2amq6yuwJjxwd/iUjh/w5NiuG+mscbRWR/XzO B+VOQKHil9tGoarA+SZIDLSEE7PWeXElzgMF/mstAFwqDWCS7ITBws2fc+lQ1aR8uDSz c1sP207qelGYAEF5xm0S0ODhm5aLDltmwT5DHGsz2Q+MCN8JgjrsGp3A+Upf3sIWAuPq fVgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=gzKHXQvYOd2P2OgIBYbUsN5miYHmmP4uVxotaWuoE7o=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=B5+A8KBogQMRPecSpTrFi9+2kvIkdac0F8PAbEHaQbBLWcByTtDFUXFH7obAxdPCKB 6Ps/glmnxIzGQO34VeU4kS7Vm6emKTE6Lxwdyv9srczRROJfulBaGtaxHIc+kTVvF1R9 0AkM3m8hfGyfLYSqFgJYJN0iDMsOwC2vIfCeEKaVZuo3a+VXKQwqjWvA9u+BIKUt/tGT yKfK9YxAqyJQYPDPIzOYFoU8bL/CZdOGNpoNvqGg954wvtn5z1GAPVKIIYP3FcxC3Imx qewkloBe6NYY7fMz2GxUTBgIjq4ucpRlbpryVvGVX0osUw5F3u6mIdjLM5SoiGqYQetn Df1g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=s8OMZUUa; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id u15-20020a05622a010f00b00403620cac93si3761894qtw.200.2023.07.13.14.16.54 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:16:54 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=s8OMZUUa; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3ec-0000RD-8T; Thu, 13 Jul 2023 17:15:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eN-0000KF-C5 for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:15:00 -0400 Received: from mail-wr1-x42d.google.com ([2a00:1450:4864:20::42d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3eL-00018v-LW for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:59 -0400 Received: by mail-wr1-x42d.google.com with SMTP id ffacd0b85a97d-314319c0d3eso1210573f8f.0 for ; Thu, 13 Jul 2023 14:14:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282896; x=1691874896; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gzKHXQvYOd2P2OgIBYbUsN5miYHmmP4uVxotaWuoE7o=; b=s8OMZUUakfW5G7zjXN29swx1Ay3uIIK8BcxyI9RnuOcLEqxHy8qsTek1EhEdHK50nr o0YxuuyR90jLnooRV+O1AyyDQwSj6Jh48crKu+BJRaN9Pq1n4P8NGkmgN2apUQEvY9wS V3xQ/j97QShn7mc5gHMiYUmTFUpuAoeXxF27562cZDja33mAjBBS9TGpKv3GPWK0SstD Jc6WxayijSlOezswgU6gzzmseLE/gXo6tZsoKyPaiUR/ga0ah+/rUbBDP5WDhgSwFeEA r4dUHY087GNfIIsbpdm3KR2G3xexVQ3M5FGlh18245nA8JD9EYQxGfQUZvX8g/9+P7XX 2GQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282896; x=1691874896; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gzKHXQvYOd2P2OgIBYbUsN5miYHmmP4uVxotaWuoE7o=; b=eSREaMz2GDW9LrINoIwZyrug4jK7roVgoGAK5COOGPfhUbMyeWdeHStVOEHOtY1zVA SNDXVc/rtdWrXfq0cZgT8TXIoU1CvxR6ZVPCnjemaqOY6m/ZfXL2spUVjK3TKOgvN+ny ALgur1DNZz3rrM9AqQo3kh2BGry7+NCBBU5IfQRWc1vld11Hbo/7zA+JI/L2G68W8qFc F9TaVoqMxg6/IU/lt2zTpiAh8zO13AYi9X+J6cyKhvxrtsUgbI9ydptoRh9mCrSCZ9KL +24PpepuDWFhX/Cn1+4kVH2VvwosW56ZtVWBW70WyvZYE4km+xhORE1JMvkW1cbbB3Cr QE/g== X-Gm-Message-State: ABy/qLaKKKsjI/ekP6iOJ4a0470RWpd7UMfzN7w5x4lecFnROZeXnLKs tjwaJ+qlwHo47sDXRuhXKwEO0QLiAS3NabPzVl2KDSK3 X-Received: by 2002:adf:fe4c:0:b0:313:e8bf:a6e with SMTP id m12-20020adffe4c000000b00313e8bf0a6emr2761804wrs.21.1689282896404; Thu, 13 Jul 2023 14:14:56 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:56 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 15/18] target/s390x: Use clmul_64 Date: Thu, 13 Jul 2023 22:14:32 +0100 Message-Id: <20230713211435.13505-16-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::42d; envelope-from=richard.henderson@linaro.org; helo=mail-wr1-x42d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Use the generic routine for 64-bit carry-less multiply. Remove our local version of galois_multiply64. Signed-off-by: Richard Henderson --- target/s390x/tcg/vec_int_helper.c | 62 +++++++------------------------ 1 file changed, 14 insertions(+), 48 deletions(-) diff --git a/target/s390x/tcg/vec_int_helper.c b/target/s390x/tcg/vec_int_helper.c index f5eea2330a..002ba67b11 100644 --- a/target/s390x/tcg/vec_int_helper.c +++ b/target/s390x/tcg/vec_int_helper.c @@ -21,13 +21,6 @@ static bool s390_vec_is_zero(const S390Vector *v) return !v->doubleword[0] && !v->doubleword[1]; } -static void s390_vec_xor(S390Vector *res, const S390Vector *a, - const S390Vector *b) -{ - res->doubleword[0] = a->doubleword[0] ^ b->doubleword[0]; - res->doubleword[1] = a->doubleword[1] ^ b->doubleword[1]; -} - static void s390_vec_and(S390Vector *res, const S390Vector *a, const S390Vector *b) { @@ -166,26 +159,6 @@ DEF_VCTZ(16) /* like binary multiplication, but XOR instead of addition */ -static S390Vector galois_multiply64(uint64_t a, uint64_t b) -{ - S390Vector res = {}; - S390Vector va = { - .doubleword[1] = a, - }; - S390Vector vb = { - .doubleword[1] = b, - }; - - while (!s390_vec_is_zero(&vb)) { - if (vb.doubleword[1] & 0x1) { - s390_vec_xor(&res, &res, &va); - } - s390_vec_shl(&va, &va, 1); - s390_vec_shr(&vb, &vb, 1); - } - return res; -} - static Int128 do_gfm8(Int128 n, Int128 m) { Int128 e = clmul_8x8_even(n, m); @@ -247,35 +220,28 @@ void HELPER(gvec_vgfma32)(void *v1, const void *v2, const void *v3, *(Int128 *)v1 = int128_xor(r, *(Int128 *)v4); } +static Int128 do_gfm64(Int128 n, Int128 m) +{ + /* + * The two 64-bit halves are treated identically, + * therefore host ordering does not matter. + */ + Int128 e = clmul_64(int128_getlo(n), int128_getlo(m)); + Int128 o = clmul_64(int128_gethi(n), int128_gethi(m)); + return int128_xor(e, o); +} + void HELPER(gvec_vgfm64)(void *v1, const void *v2, const void *v3, uint32_t desc) { - S390Vector tmp1, tmp2; - uint64_t a, b; - - a = s390_vec_read_element64(v2, 0); - b = s390_vec_read_element64(v3, 0); - tmp1 = galois_multiply64(a, b); - a = s390_vec_read_element64(v2, 1); - b = s390_vec_read_element64(v3, 1); - tmp2 = galois_multiply64(a, b); - s390_vec_xor(v1, &tmp1, &tmp2); + *(Int128 *)v1 = do_gfm64(*(const Int128 *)v2, *(const Int128 *)v3); } void HELPER(gvec_vgfma64)(void *v1, const void *v2, const void *v3, const void *v4, uint32_t desc) { - S390Vector tmp1, tmp2; - uint64_t a, b; - - a = s390_vec_read_element64(v2, 0); - b = s390_vec_read_element64(v3, 0); - tmp1 = galois_multiply64(a, b); - a = s390_vec_read_element64(v2, 1); - b = s390_vec_read_element64(v3, 1); - tmp2 = galois_multiply64(a, b); - s390_vec_xor(&tmp1, &tmp1, &tmp2); - s390_vec_xor(v1, &tmp1, v4); + Int128 r = do_gfm64(*(const Int128 *)v2, *(const Int128 *)v3); + *(Int128 *)v1 = int128_xor(r, *(Int128 *)v4); } #define DEF_VMAL(BITS) \ From patchwork Thu Jul 13 21:14:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702237 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp741731wrj; Thu, 13 Jul 2023 14:17:26 -0700 (PDT) X-Google-Smtp-Source: APBJJlHxbsXLfDicNqUGkPsn2KUmh4BkfyV0BzYsfEBIJTMdDs4F8LkOx5TTVmK9+QvcztWVFAvA X-Received: by 2002:a05:622a:1750:b0:402:d96c:c840 with SMTP id l16-20020a05622a175000b00402d96cc840mr3258618qtk.18.1689283046345; Thu, 13 Jul 2023 14:17:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689283046; cv=none; d=google.com; s=arc-20160816; b=qro7wSSPsX2AngZgU9uYqnwikI4SVzirNdCQkDUDAv6PDMZ5EsooTVMIJsgGU7byVw 4jaeRfuTVdEQL5+8uLzeL70JRLWvjunMnCf0+KJg7spe7DSlxEZ+rJHIP9NH3SXtHtHK Fcl5OCRkNFQIcEyj23cA6RCCxbRSSpmplynGug2IrHxx1yPnONcftAVUHkx8iXULIt+p 7r1AZjiHA/76QU1z78Mwqnpt4vq14F65LUNTn9F5DZoEkxxyjQUbfTLRkjgkR4LL+jMv r3+iARMX/zFmbKzkvOYa2+vutrDXl/OyyrUsFaQBBBOzmn/bmNggeRvUtu9DTsIYmiCx wHKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=66pSwadG/9AX/ZsrllJ7+T0lOcbfRbkA+50ErEBSa5M=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=zbfdSBj+H+kq4x6DK0aypEjJT4w/2JcEwxWbRo4K/0EpxWHr22NV8GKiz6peTa/imu ISYoJKj8pQ5vrSfzQ9feieQdb5pN/bvuof7vnuQuoD1fWp+1jcEYLrWomgqeuv65xN3K aCcI3zJkZYnK44EEtkdkPz87P5NsmQtQuOCP16ID9fWaBAHNeGAfjvllJebuVbKIq96h l3os6Ks9ZbjP9RNB235WdtT01sUdeiLwE11+ExrwJLxdIVZ8bAfkF2T9XY8gxy4OuM/x rwGs68Tje3FffasKOiX6BFqvEnsH+NCGY6/U5DBnmMHi9D7TUCQGe5Ci91Rv7qIVxMTy NjKg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ydtfPoH9; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id l6-20020a05622a174600b00403a4d391e2si3602190qtk.65.2023.07.13.14.17.26 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:17:26 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ydtfPoH9; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3eb-0000Qv-Mi; Thu, 13 Jul 2023 17:15:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eO-0000KJ-4K for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:15:00 -0400 Received: from mail-wm1-x334.google.com ([2a00:1450:4864:20::334]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3eM-000192-Hv for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:14:59 -0400 Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-3fbd33a57dcso11897605e9.0 for ; Thu, 13 Jul 2023 14:14:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282897; x=1691874897; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=66pSwadG/9AX/ZsrllJ7+T0lOcbfRbkA+50ErEBSa5M=; b=ydtfPoH97BucvQ0xFcFr5i9651XLHRcp0kx6dcqjIYeMAm6HE5S/souUKGmadIkajq wK6JerpNQQffnUCq3bfyQBmpzp7HEMKv1S6V1a2fZf2hsoQeSj51eNXsAoyETAffsIlE rBEgYvMlJSYJX3SxIk+VE+VUZEw86MUTEretoHC39NwpKiJc5Q+Xis8fvpKhnKNeQOtA vtc511LfSgcouccMDLr6gKmBhoha26WfsChsMPi1smu+h6gUXc/5GiP63G2tk9MFXhfu vjjuFDvblwRmKVFAP0Mga7fTpx0EDnZsXXGrpXpcNts2C64InzefeyBLaznFVvUDZDfU RwsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282897; x=1691874897; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=66pSwadG/9AX/ZsrllJ7+T0lOcbfRbkA+50ErEBSa5M=; b=btLt8QZ5qfHz/hYVALc/AMN/iT3GJusgtV7DhN64nnXKeh4UQphctkMMeW1L6ryRqK uKhZJ/ws2DuVX+HagXPd/sfQHir+7SmvDMPWYx+jddeefvBvuk7gLo/YoQpOCZBzxODJ iies68MMruZI9vQBsjOnNwYzNNzoyN5vENOsT/jBqVbrbXdsLQ3eYBmaelmU7I+j47k9 3z+DwU1eYjz+iFf4vJsWFm6nP6UbK/y+qASfDLHSbRtPv/hUaK/viJpis18Ia5T2cuYn A24LbG4vQNC65yRtLwKVN1wT0jOD9dfXYp+exYw9g94MgI76mxTtBtKtakHL4wLzRPQ0 s8/Q== X-Gm-Message-State: ABy/qLbid/pSMhb9a3zKAPKYMcCekU+yC//HMGOmNwWeudFmkc369saH iFOKit4aoaZdY337jajpaycEF7BLyrn7IwAesj/Zojog X-Received: by 2002:adf:f30a:0:b0:313:dfb8:b4d0 with SMTP id i10-20020adff30a000000b00313dfb8b4d0mr2490155wro.66.1689282897123; Thu, 13 Jul 2023 14:14:57 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:56 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 16/18] target/ppc: Use clmul_64 Date: Thu, 13 Jul 2023 22:14:33 +0100 Message-Id: <20230713211435.13505-17-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::334; envelope-from=richard.henderson@linaro.org; helo=mail-wm1-x334.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Use generic routine for 64-bit carry-less multiply. Signed-off-by: Richard Henderson --- target/ppc/int_helper.c | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/target/ppc/int_helper.c b/target/ppc/int_helper.c index 828f04bce7..4e1fa2fd68 100644 --- a/target/ppc/int_helper.c +++ b/target/ppc/int_helper.c @@ -1455,20 +1455,9 @@ void helper_vpmsumw(ppc_avr_t *r, ppc_avr_t *a, ppc_avr_t *b) void helper_VPMSUMD(ppc_avr_t *r, ppc_avr_t *a, ppc_avr_t *b) { - int i, j; - Int128 tmp, prod[2] = {int128_zero(), int128_zero()}; - - for (j = 0; j < 64; j++) { - for (i = 0; i < ARRAY_SIZE(r->u64); i++) { - if (a->VsrD(i) & (1ull << j)) { - tmp = int128_make64(b->VsrD(i)); - tmp = int128_lshift(tmp, j); - prod[i] = int128_xor(prod[i], tmp); - } - } - } - - r->s128 = int128_xor(prod[0], prod[1]); + Int128 e = clmul_64(a->u64[0], b->u64[0]); + Int128 o = clmul_64(a->u64[1], b->u64[1]); + r->s128 = int128_xor(e, o); } #if HOST_BIG_ENDIAN From patchwork Thu Jul 13 21:14:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702229 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp740705wrj; Thu, 13 Jul 2023 14:15:40 -0700 (PDT) X-Google-Smtp-Source: APBJJlHf0Q9RnKfbM0e1m6sHz53Q84RWhyDBv4aiJaqV/IUI6U7lQ9gZxAh9XTAwoFZl3jcjk0ow X-Received: by 2002:a05:620a:4053:b0:767:81fc:c6d5 with SMTP id i19-20020a05620a405300b0076781fcc6d5mr3460955qko.39.1689282940343; Thu, 13 Jul 2023 14:15:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689282940; cv=none; d=google.com; s=arc-20160816; b=Zv3H6PeHgQ7a8F+JvUlC6JTdUjoCRnOoMCU4rufmozOwr1QbSj80Hn1w1j/HONczig xbyLKbDEpHxVz5ZN0SRzehnyS64RTLuHKEgq4c/GJpWS6VqKgnBgbKKeJnHTmrReqqWJ 4cagfeph3qljUoN0Gv7tL1p9oKyYUNoYD0PVZYQriT9J2gbEY9VqE9MMu7zR0N6pmmEb G4qGYpka3wNsXQ/iWOaZkI3wuObz7Do1SRWJ7X2JWnDMimCOW2NvdnOWjDGAYK2Wihzi PDDFNlOSnMdcIx1O4iVTQ55lYp9nYCsscxditTKGr+DVZxfaum88UymfrimnRGfXsOzX Ps4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=FzGMoLNjkDe1yX7tuu779EsO1joTMqPUtVQXclt+8ac=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=ihebqJN0/4mvWQ2o9N2nDWLIF7DyuvAB506cpjd67TiGnRHEM0j3gaAfsxgMUuqEoz EiGD/ZgG3GUFv3EYvBnENqCreSFZxDIjTIrhr9YstPSJGZGRHHGhv86NJSCNb2YFOsAP uaxXdjEe46V/8KFwCAwEE1nYwLuYGwbzgYhajaxRZ5HCkjFAfs0D3HoZDYfdZ0mYUWZG VtMmzJRRWBSSm3o5qTlFT4a7pAZi9FETJmIPETM5GNW3qwPCeeI2vFke0gKLTERdqpzI rKuUHOBS/jUkk5vhd4tHgnxB2Bsgos3W3sFvr/WzNKnsVGCU+dJq6BQbIGYxILnx2lBJ dxoQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=LqnqEjsR; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id az18-20020a05620a171200b00767f1e50572si1981138qkb.39.2023.07.13.14.15.40 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:15:40 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=LqnqEjsR; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3ec-0000Ri-TX; Thu, 13 Jul 2023 17:15:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eQ-0000Mq-6g for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:15:04 -0400 Received: from mail-wr1-x434.google.com ([2a00:1450:4864:20::434]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3eN-00019J-KF for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:15:01 -0400 Received: by mail-wr1-x434.google.com with SMTP id ffacd0b85a97d-314417861b9so1313929f8f.0 for ; Thu, 13 Jul 2023 14:14:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282898; x=1691874898; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FzGMoLNjkDe1yX7tuu779EsO1joTMqPUtVQXclt+8ac=; b=LqnqEjsRBK+opcPjBUcVbLDs0MsRdBOa+vWNDBuLiRd2zpYFGNEAJDY1WLcw7FO3KA wbeWXcW0X4bjpo4l+1rH1bicn27nH2HdsRjA9s/8mPwkiNZivilBHw01z8GkaMcKbVvU bk02Y5NEp2m7opNdxZmhN22D6rW0tHamYXF+Yx8BtrlYmM1axdtkvvgBEI1x7iDyNoV/ uyHxz9MA+8mdg6cHoe4HC1JpuZeyHZg7tx3KenzhCObzymvvhdTQX3S7MmBwYaobS/E9 U28mBwB5b7wjwQFzSni2wRyUAuQxmX/UJ/PHU+2y9y30JNW50159U6lIkrBB0k+TfwRW GgAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282898; x=1691874898; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FzGMoLNjkDe1yX7tuu779EsO1joTMqPUtVQXclt+8ac=; b=VV0NOir2DXSIgLkfsJ18b1Bg0MhdB1sVJSDkGnI0EO7ZO2nSUQ2TSchsCG+bLMOpaT ZkOKhUFjctJxVKTZG1uyu+rLI8s8BZFU2tBrA0F15Q8gdx1R47XquFuEfZ8/Ec2J8xKa Ro7El9smB0e0in0464c2WvfZkLYcfgwKERAuKmuu9+D0VeQl7xcsZkguxnLT6WFtAKFj c2r29W86Sw1BaDlMDmlxc3t0tyN2jVaD7FLtO+CvZDIcvu/ijzren8BY49YshUxgivV+ iFdqIqesNtZYSIxJh9Zz9hPlpHjm3eRYlossoHsuPu0f/bONv50IbVDa9mf75PidKWJ9 7vUA== X-Gm-Message-State: ABy/qLYiaMkVfE018/a1flcf+Gw31T1g5r8BosGlBo9XlbpTJsIIWnFc fr++BUSJTyleseVU0FjyJWECRpPaOrXAZW9yBW+kyDZI X-Received: by 2002:a5d:6b8f:0:b0:316:ee41:f1bf with SMTP id n15-20020a5d6b8f000000b00316ee41f1bfmr213435wrx.12.1689282898299; Thu, 13 Jul 2023 14:14:58 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:58 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 17/18] host/include/i386: Implement clmul.h Date: Thu, 13 Jul 2023 22:14:34 +0100 Message-Id: <20230713211435.13505-18-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::434; envelope-from=richard.henderson@linaro.org; helo=mail-wr1-x434.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Detect PCLMUL in cpuinfo; implement the accel hooks. Signed-off-by: Richard Henderson --- host/include/i386/host/cpuinfo.h | 1 + host/include/i386/host/crypto/clmul.h | 187 ++++++++++++++++++++++++ host/include/x86_64/host/crypto/clmul.h | 1 + util/cpuinfo-i386.c | 1 + 4 files changed, 190 insertions(+) create mode 100644 host/include/i386/host/crypto/clmul.h create mode 100644 host/include/x86_64/host/crypto/clmul.h diff --git a/host/include/i386/host/cpuinfo.h b/host/include/i386/host/cpuinfo.h index 073d0a426f..7ae21568f7 100644 --- a/host/include/i386/host/cpuinfo.h +++ b/host/include/i386/host/cpuinfo.h @@ -27,6 +27,7 @@ #define CPUINFO_ATOMIC_VMOVDQA (1u << 16) #define CPUINFO_ATOMIC_VMOVDQU (1u << 17) #define CPUINFO_AES (1u << 18) +#define CPUINFO_PCLMUL (1u << 19) /* Initialized with a constructor. */ extern unsigned cpuinfo; diff --git a/host/include/i386/host/crypto/clmul.h b/host/include/i386/host/crypto/clmul.h new file mode 100644 index 0000000000..0877d65ab6 --- /dev/null +++ b/host/include/i386/host/crypto/clmul.h @@ -0,0 +1,187 @@ +/* + * x86 specific clmul acceleration. + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#ifndef X86_HOST_CRYPTO_CLMUL_H +#define X86_HOST_CRYPTO_CLMUL_H + +#include "host/cpuinfo.h" +#include + +#if defined(__PCLMUL__) +# define HAVE_CLMUL_ACCEL true +# define ATTR_CLMUL_ACCEL +#else +# define HAVE_CLMUL_ACCEL likely(cpuinfo & CPUINFO_PCLMUL) +# define ATTR_CLMUL_ACCEL __attribute__((target("pclmul"))) +#endif + +static inline Int128 ATTR_CLMUL_ACCEL +clmul_64(uint64_t n, uint64_t m) +{ + union { __m128i v; Int128 s; } u; + + if (!HAVE_CLMUL_ACCEL) { + return clmul_64_gen(n, m); + } + + u.v = _mm_clmulepi64_si128(_mm_set_epi64x(0, n), _mm_set_epi64x(0, m), 0); + return u.s; +} + +static inline uint64_t ATTR_CLMUL_ACCEL +clmul_32(uint32_t n, uint32_t m) +{ + __m128i r; + + if (!HAVE_CLMUL_ACCEL) { + return clmul_32_gen(n, m); + } + + r = _mm_clmulepi64_si128(_mm_cvtsi32_si128(n), _mm_cvtsi32_si128(m), 0); + return ((__v2di)r)[0]; +} + +static inline Int128 ATTR_CLMUL_ACCEL +clmul_32x2_even(Int128 n, Int128 m) +{ + union { __m128i v; Int128 s; } ur, un, um; + __m128i n02, m02, r0, r2; + + if (!HAVE_CLMUL_ACCEL) { + return clmul_32x2_even_gen(n, m); + } + + un.s = n; + um.s = m; + n02 = _mm_slli_epi64(un.v, 32); + m02 = _mm_slli_epi64(um.v, 32); + r0 = _mm_clmulepi64_si128(n02, m02, 0x00); + r2 = _mm_clmulepi64_si128(n02, m02, 0x11); + ur.v = _mm_unpackhi_epi64(r0, r2); + return ur.s; +} + +static inline Int128 ATTR_CLMUL_ACCEL +clmul_32x2_odd(Int128 n, Int128 m) +{ + union { __m128i v; Int128 s; } ur, un, um; + __m128i n13, m13, r1, r3; + + if (!HAVE_CLMUL_ACCEL) { + return clmul_32x2_odd_gen(n, m); + } + + un.s = n; + um.s = m; + n13 = _mm_srli_epi64(un.v, 32); + m13 = _mm_srli_epi64(um.v, 32); + r1 = _mm_clmulepi64_si128(n13, m13, 0x00); + r3 = _mm_clmulepi64_si128(n13, m13, 0x11); + ur.v = _mm_unpacklo_epi64(r1, r3); + return ur.s; +} + +static inline uint64_t ATTR_CLMUL_ACCEL +clmul_16x2_even(uint64_t n, uint64_t m) +{ + __m128i r0, r2; + + if (!HAVE_CLMUL_ACCEL) { + return clmul_16x2_even_gen(n, m); + } + + r0 = _mm_clmulepi64_si128(_mm_cvtsi32_si128(n & 0xffff), + _mm_cvtsi32_si128(m & 0xffff), 0); + r2 = _mm_clmulepi64_si128(_mm_cvtsi32_si128((n >> 32) & 0xffff), + _mm_cvtsi32_si128((m >> 32) & 0xffff), 0); + r0 = _mm_unpacklo_epi32(r0, r2); + return ((__v2di)r0)[0]; +} + +static inline uint64_t ATTR_CLMUL_ACCEL +clmul_16x2_odd(uint64_t n, uint64_t m) +{ + __m128i r1, r3; + + if (!HAVE_CLMUL_ACCEL) { + return clmul_16x2_even_gen(n, m); + } + + r1 = _mm_clmulepi64_si128(_mm_cvtsi32_si128((n >> 16) & 0xffff), + _mm_cvtsi32_si128((m >> 16) & 0xffff), 0); + r3 = _mm_clmulepi64_si128(_mm_cvtsi32_si128((n >> 48) & 0xffff), + _mm_cvtsi32_si128((m >> 48) & 0xffff), 0); + r1 = _mm_unpacklo_epi32(r1, r3); + return ((__v2di)r1)[0]; +} + +static inline Int128 ATTR_CLMUL_ACCEL +clmul_16x4_even(Int128 n, Int128 m) +{ + union { __m128i v; Int128 s; } ur, un, um; + __m128i mask = _mm_set_epi16(0, 0, 0, -1, 0, 0, 0, -1); + __m128i n04, m04, n26, m26, r0, r2, r4, r6; + + if (!HAVE_CLMUL_ACCEL) { + return clmul_16x4_even_gen(n, m); + } + + un.s = n; + um.s = m; + n04 = _mm_and_si128(un.v, mask); + m04 = _mm_and_si128(um.v, mask); + r0 = _mm_clmulepi64_si128(n04, m04, 0x00); + r4 = _mm_clmulepi64_si128(n04, m04, 0x11); + n26 = _mm_and_si128(_mm_srli_epi64(un.v, 32), mask); + m26 = _mm_and_si128(_mm_srli_epi64(um.v, 32), mask); + r2 = _mm_clmulepi64_si128(n26, m26, 0x00); + r6 = _mm_clmulepi64_si128(n26, m26, 0x11); + + r0 = _mm_unpacklo_epi32(r0, r2); + r4 = _mm_unpacklo_epi32(r4, r6); + ur.v = _mm_unpacklo_epi64(r0, r4); + return ur.s; +} + +static inline Int128 ATTR_CLMUL_ACCEL +clmul_16x4_odd(Int128 n, Int128 m) +{ + union { __m128i v; Int128 s; } ur, un, um; + __m128i mask = _mm_set_epi16(0, 0, 0, -1, 0, 0, 0, -1); + __m128i n15, m15, n37, m37, r1, r3, r5, r7; + + if (!HAVE_CLMUL_ACCEL) { + return clmul_16x4_odd_gen(n, m); + } + + un.s = n; + um.s = m; + n15 = _mm_and_si128(_mm_srli_epi64(un.v, 16), mask); + m15 = _mm_and_si128(_mm_srli_epi64(um.v, 16), mask); + r1 = _mm_clmulepi64_si128(n15, m15, 0x00); + r5 = _mm_clmulepi64_si128(n15, m15, 0x11); + n37 = _mm_srli_epi64(un.v, 48); + m37 = _mm_srli_epi64(um.v, 48); + r3 = _mm_clmulepi64_si128(n37, m37, 0x00); + r7 = _mm_clmulepi64_si128(n37, m37, 0x11); + + r1 = _mm_unpacklo_epi32(r1, r3); + r5 = _mm_unpacklo_epi32(r5, r7); + ur.v = _mm_unpacklo_epi64(r1, r5); + return ur.s; +} + +/* + * Defer everything else to the generic routines. + * We could implement them with even more element manipulation. + */ +#define clmul_8x8_low clmul_8x8_low_gen +#define clmul_8x4_even clmul_8x4_even_gen +#define clmul_8x4_odd clmul_8x4_odd_gen +#define clmul_8x8_even clmul_8x8_even_gen +#define clmul_8x8_odd clmul_8x8_odd_gen +#define clmul_8x8_packed clmul_8x8_packed_gen + +#endif /* X86_HOST_CRYPTO_CLMUL_H */ diff --git a/host/include/x86_64/host/crypto/clmul.h b/host/include/x86_64/host/crypto/clmul.h new file mode 100644 index 0000000000..f25eced416 --- /dev/null +++ b/host/include/x86_64/host/crypto/clmul.h @@ -0,0 +1 @@ +#include "host/include/i386/host/crypto/clmul.h" diff --git a/util/cpuinfo-i386.c b/util/cpuinfo-i386.c index 3a7b7e0ad1..c6f6364826 100644 --- a/util/cpuinfo-i386.c +++ b/util/cpuinfo-i386.c @@ -39,6 +39,7 @@ unsigned __attribute__((constructor)) cpuinfo_init(void) info |= (c & bit_SSE4_1 ? CPUINFO_SSE4 : 0); info |= (c & bit_MOVBE ? CPUINFO_MOVBE : 0); info |= (c & bit_POPCNT ? CPUINFO_POPCNT : 0); + info |= (c & bit_PCLMULQDQ ? CPUINFO_PCLMUL : 0); /* Our AES support requires PSHUFB as well. */ info |= ((c & bit_AES) && (c & bit_SSSE3) ? CPUINFO_AES : 0); From patchwork Thu Jul 13 21:14:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 702238 Delivered-To: patch@linaro.org Received: by 2002:adf:db4e:0:b0:314:25da:8dc4 with SMTP id f14csp741780wrj; Thu, 13 Jul 2023 14:17:34 -0700 (PDT) X-Google-Smtp-Source: APBJJlFM/2o3OatB7xW5fStMZY4q4ZLkvHc3rdDSrptD1Etc8rCJncUo/qHBgeAQE9nEOGMFJQov X-Received: by 2002:a0c:f194:0:b0:635:5a95:bb48 with SMTP id m20-20020a0cf194000000b006355a95bb48mr2789522qvl.24.1689283053954; Thu, 13 Jul 2023 14:17:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689283053; cv=none; d=google.com; s=arc-20160816; b=dC4978bCsXeqmPcZwrG5fjPk2ZaUowCEYuQbqwJhvmOJloWvZdxl0TZo7w9WAuelU+ 1rQ245Z+zS8y61QYgn8sUTKP17lcj74pCh/YfsXClPOOXzMYCCOXsoNpiu5Yt+UEZajD JzPPzYpeBUqM2QzNpgxsV5c+SpWJxZ40HDsc+2rguYtTLfYZNKoKiPYXAkmWqJw3H35U I3Dj9C4a/rD8byJHIGpI42RCep6QM6ndWIyTFPKUXv8qzTyRGvM7ln+q42Jvd0VIGieZ ERwquOpz2vwD4Ejn7FFbr/g4WN6b2Jlz7JN5qj2wLy5v9dENCfqLtprDIpHljECTwIm3 dPQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=epV34O9DTR5ZBAxnORvbfiEcBH2IbFQIc3x2ZASYHbI=; fh=4xNVIrgzKzsUJFZ/LK5iWzslbipp0obh3zmyxXTMM5Q=; b=ovTW5zVssKMv70mK6/iMS57ZW4JY/G+K+g9i3qiMkyAwt+MjDVQvZbTuVVvkptO+QR QOYRr5Hj3rxpqcZ/DghyXCIfJcwEmitFy8P7wKwZLathNtcS870R4APqRRBEOzidUdcE OpVkrwaSyDF9Hv9bDb+VR9SdiXWQJerjfC1F+QIAbFlbX91n5lmRcXKK9SWUw74j3AwK KJ2FANW3GrltbyX3TgbvJhLJfMEed/D+ikahKNoU9gczHShv8mfLRZbkuGVn4mTnbCMV HPvvmPePIHxFnzIhLEO3WjLsLBaz5AZ1FePCyPvfkTHOxhQiDWwWtBtbILz3QCrVa3DS etjQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=LW96DSsn; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id q15-20020a0ce20f000000b00635dfafb75csi3592964qvl.71.2023.07.13.14.17.33 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2023 14:17:33 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=LW96DSsn; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qK3ej-0000S1-Mj; Thu, 13 Jul 2023 17:15:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qK3eR-0000NA-2g for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:15:04 -0400 Received: from mail-wr1-x435.google.com ([2a00:1450:4864:20::435]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qK3eO-00019U-PW for qemu-devel@nongnu.org; Thu, 13 Jul 2023 17:15:02 -0400 Received: by mail-wr1-x435.google.com with SMTP id ffacd0b85a97d-3141c3a7547so1322340f8f.2 for ; Thu, 13 Jul 2023 14:15:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1689282899; x=1691874899; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=epV34O9DTR5ZBAxnORvbfiEcBH2IbFQIc3x2ZASYHbI=; b=LW96DSsnnHGNraV3OFF6ViK62/0drwKKRdgpzRMf2YZdyoXcrNke52sWyhtlzxcVBd 5W+O8XrO2LyPSr3fKecKWD0e0VLhMNggjUr3bWcRrMPDU4e7J+hyM2jGaReftGTfNfgC EYCTaG6qQJ7DXAYK71SYj4jPq4nYyJNwWN7P2fQ+4NLmiwxJdMortOaHpm+T4jckG9gY dRbqPu7E8hium8P6Z9d5MbdLqHWaoyXnmzX08Bp5xAQVp+gmfeAisK8TB+eCHx3WONh8 e0uu6tCUSSPBj8yFQ1axLRvw8K6qCdeFvguhm+7gxkAgp12ilLDydUPj8bmsya7TW4Ak J5AQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689282899; x=1691874899; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=epV34O9DTR5ZBAxnORvbfiEcBH2IbFQIc3x2ZASYHbI=; b=FgJlkQZMbzqMgqOnZtXy3ubXoAedPVos3o+jIByGDmETXhwVskNmp6oV7+b4oO61Vk Lnem5RBCXTBM7YhvcmeKFvRsaW5zixn502+dQvp5DrD9VaD4s2afCZwePvdTDG2YzjzZ SZQA5Ksuo1uh9ZD7utBQqU/KQd90IYhzoVIOXVQq6ww+sHr93KmNQVQR3+eKxwF5XLfQ /aZjOU2oIOipeH8JsZQDFHiAF9RbiO4NIlTpABPj5wbKvvUZqxFU3mzoAPpeMPmr8SWX DsvOn736aw4eMz96ngFKrmBbLwqd5/sOLwUHnaaWIy/HtaazFDO2LjrPiaMAITKUa5B8 29mw== X-Gm-Message-State: ABy/qLYMzZjUAmYzJ1cawAEugECTRcyl+jfbx0BglVtyWt5ou/8AurT2 lhuPPVBeUmjZtN+e9IITLwOYN4gPKwgwIIzAnEcWCkdm X-Received: by 2002:a5d:4e4d:0:b0:313:eee0:89a4 with SMTP id r13-20020a5d4e4d000000b00313eee089a4mr2051943wrt.12.1689282899379; Thu, 13 Jul 2023 14:14:59 -0700 (PDT) Received: from stoup.. ([85.193.156.66]) by smtp.gmail.com with ESMTPSA id n12-20020adff08c000000b0030ada01ca78sm8973747wro.10.2023.07.13.14.14.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:14:59 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: berrange@redhat.com, ardb@kernel.org Subject: [PATCH 18/18] host/include/aarch64: Implement clmul.h Date: Thu, 13 Jul 2023 22:14:35 +0100 Message-Id: <20230713211435.13505-19-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230713211435.13505-1-richard.henderson@linaro.org> References: <20230713211435.13505-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::435; envelope-from=richard.henderson@linaro.org; helo=mail-wr1-x435.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Detect PMULL in cpuinfo; implement the accel hooks. Signed-off-by: Richard Henderson --- host/include/aarch64/host/cpuinfo.h | 1 + host/include/aarch64/host/crypto/clmul.h | 230 +++++++++++++++++++++++ util/cpuinfo-aarch64.c | 4 +- 3 files changed, 234 insertions(+), 1 deletion(-) create mode 100644 host/include/aarch64/host/crypto/clmul.h diff --git a/host/include/aarch64/host/cpuinfo.h b/host/include/aarch64/host/cpuinfo.h index 05feeb4f43..da268dce13 100644 --- a/host/include/aarch64/host/cpuinfo.h +++ b/host/include/aarch64/host/cpuinfo.h @@ -10,6 +10,7 @@ #define CPUINFO_LSE (1u << 1) #define CPUINFO_LSE2 (1u << 2) #define CPUINFO_AES (1u << 3) +#define CPUINFO_PMULL (1u << 4) /* Initialized with a constructor. */ extern unsigned cpuinfo; diff --git a/host/include/aarch64/host/crypto/clmul.h b/host/include/aarch64/host/crypto/clmul.h new file mode 100644 index 0000000000..7fd827898b --- /dev/null +++ b/host/include/aarch64/host/crypto/clmul.h @@ -0,0 +1,230 @@ +/* + * AArch64 specific clmul acceleration. + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#ifndef AARCH64_HOST_CRYPTO_CLMUL_H +#define AARCH64_HOST_CRYPTO_CLMUL_H + +#include "host/cpuinfo.h" +#include + +/* Both FEAT_AES and FEAT_PMULL are covered under the same macro. */ +#ifdef __ARM_FEATURE_AES +# define HAVE_CLMUL_ACCEL true +#else +# define HAVE_CLMUL_ACCEL likely(cpuinfo & CPUINFO_PMULL) +#endif +#if !defined(__ARM_FEATURE_AES) && defined(CONFIG_ARM_AES_BUILTIN) +# define ATTR_CLMUL_ACCEL __attribute__((target("+crypto"))) +#else +# define ATTR_CLMUL_ACCEL +#endif + +/* + * The 8x8->8 pmul and 8x8->16 pmull are available unconditionally. + */ + +static inline uint64_t clmul_8x8_low(uint64_t n, uint64_t m) +{ + return (uint64_t)vmul_p8((poly8x8_t)n, (poly8x8_t)m); +} + +static inline Int128 clmul_8x8_packed(uint64_t n, uint64_t m) +{ + union { poly16x8_t v; Int128 s; } u; + u.v = vmull_p8((poly8x8_t)n, (poly8x8_t)m); + return u.s; +} + +static inline Int128 clmul_8x8_even(Int128 n, Int128 m) +{ + union { uint16x8_t v; Int128 s; } un, um; + uint8x8_t pn, pm; + + un.s = n; + um.s = m; + pn = vmovn_u16(un.v); + pm = vmovn_u16(um.v); + return clmul_8x8_packed((uint64_t)pn, (uint64_t)pm); +} + +static inline Int128 clmul_8x8_odd(Int128 n, Int128 m) +{ + union { uint8x16_t v; Int128 s; } un, um; + uint8x8_t pn, pm; + + un.s = n; + um.s = m; + pn = vqtbl1_u8(un.v, (uint8x8_t){ 1, 3, 5, 7, 9, 11, 13, 15 }); + pm = vqtbl1_u8(um.v, (uint8x8_t){ 1, 3, 5, 7, 9, 11, 13, 15 }); + return clmul_8x8_packed((uint64_t)pn, (uint64_t)pm); +} + +static inline uint64_t clmul_8x4_even(uint64_t n, uint64_t m) +{ + return int128_getlo(clmul_8x8_even(int128_make64(n), int128_make64(m))); +} + +static inline uint64_t clmul_8x4_odd(uint64_t n, uint64_t m) +{ + return int128_getlo(clmul_8x8_odd(int128_make64(n), int128_make64(m))); +} + +static inline Int128 clmul_16x4_packed_accel(uint16x4_t n, uint16x4_t m) +{ + union { uint32x4_t v; Int128 s; } u; + uint32x4_t r0, r1, r2; + + /* + * Considering the per-byte multiplication: + * ab + * cd + * ----- + * bd << 0 + * bc << 8 + * ad << 8 + * ac << 16 + * + * We get the ac and bd rows of the result for free from the expanding + * packed multiply. Reverse the two bytes in M, repeat, and we get the + * ad and bc results, but in the wrong column; shift to fix and sum all. + */ + r0 = (uint32x4_t)vmull_p8((poly8x8_t)n, (poly8x8_t)m); + r1 = (uint32x4_t)vmull_p8((poly8x8_t)n, vrev16_p8((poly8x8_t)m)); + r2 = r1 << 8; /* bc */ + r1 = r1 >> 8; /* ad */ + r1 &= (uint32x4_t){ 0x00ffff00, 0x00ffff00, 0x00ffff00, 0x00ffff00 }; + r2 &= (uint32x4_t){ 0x00ffff00, 0x00ffff00, 0x00ffff00, 0x00ffff00 }; + r0 = r0 ^ r1 ^ r2; + + u.v = r0; + return u.s; +} + +static inline Int128 clmul_16x4_even(Int128 n, Int128 m) +{ + union { uint32x4_t v; Int128 s; } um, un; + uint16x4_t pn, pm; + + /* Extract even uint16_t. */ + un.s = n; + um.s = m; + pn = vmovn_u32(un.v); + pm = vmovn_u32(um.v); + return clmul_16x4_packed_accel(pn, pm); +} + +static inline Int128 clmul_16x4_odd(Int128 n, Int128 m) +{ + union { uint8x16_t v; Int128 s; } um, un; + uint16x4_t pn, pm; + + /* Extract odd uint16_t. */ + un.s = n; + um.s = m; + pn = (uint16x4_t)vqtbl1_u8(un.v, (uint8x8_t){ 2, 3, 6, 7, 10, 11, 14, 15 }); + pm = (uint16x4_t)vqtbl1_u8(um.v, (uint8x8_t){ 2, 3, 6, 7, 10, 11, 14, 15 }); + return clmul_16x4_packed_accel(pn, pm); +} + +static inline uint64_t clmul_16x2_even(uint64_t n, uint64_t m) +{ + return int128_getlo(clmul_16x4_even(int128_make64(n), int128_make64(m))); +} + +static inline uint64_t clmul_16x2_odd(uint64_t n, uint64_t m) +{ + return int128_getlo(clmul_16x4_odd(int128_make64(n), int128_make64(m))); +} + +/* + * The 64x64->128 pmull is available with FEAT_PMULL. + */ + +static inline Int128 ATTR_CLMUL_ACCEL +clmul_64(uint64_t n, uint64_t m) +{ + union { poly128_t v; Int128 s; } u; + + if (!HAVE_CLMUL_ACCEL) { + return clmul_64_gen(n, m); + } + +#ifdef CONFIG_ARM_AES_BUILTIN + u.v = vmull_p64((poly64_t)n, (poly64_t)m); +#else + asm(".arch_extension aes\n\t" + "pmull %0.1q, %1.1d, %2.1d" : "=w"(u.v) : "w"(n), "w"(m)); +#endif + return u.s; +} + +static inline uint64_t ATTR_CLMUL_ACCEL +clmul_32(uint32_t n, uint32_t m) +{ + if (!HAVE_CLMUL_ACCEL) { + return clmul_32_gen(n, m); + } + return int128_getlo(clmul_64(n, m)); +} + +static inline Int128 ATTR_CLMUL_ACCEL +clmul_32x2_even(Int128 n, Int128 m) +{ + union { uint64x2_t v; poly64_t h; Int128 s; } um, un, ur; + uint64x2_t r0, r2; + + if (!HAVE_CLMUL_ACCEL) { + return clmul_32x2_even_gen(n, m); + } + + un.s = n; + um.s = m; + un.v &= (uint64x2_t){ 0xffffffffu, 0xffffffffu }; + um.v &= (uint64x2_t){ 0xffffffffu, 0xffffffffu }; + +#ifdef CONFIG_ARM_AES_BUILTIN + r0 = (uint64x2_t)vmull_p64(un.h, um.h); + r2 = (uint64x2_t)vmull_high_p64((poly64x2_t)un.v, (poly64x2_t)um.v); +#else + asm(".arch_extension aes\n\t" + "pmull %0.1q, %2.1d, %3.1d\n\t" + "pmull2 %1.1q, %2.2d, %3.2d" + : "=&w"(r0), "=w"(r2) : "w"(un.v), "w"(um.v)); +#endif + + ur.v = vzip1q_u64(r0, r2); + return ur.s; +} + +static inline Int128 ATTR_CLMUL_ACCEL +clmul_32x2_odd(Int128 n, Int128 m) +{ + union { uint64x2_t v; poly64_t h; Int128 s; } um, un, ur; + uint64x2_t r0, r2; + + if (!HAVE_CLMUL_ACCEL) { + return clmul_32x2_odd_gen(n, m); + } + + un.s = n; + um.s = m; + un.v &= (uint64x2_t){ 0xffffffff00000000ull, 0xffffffff00000000ull }; + um.v &= (uint64x2_t){ 0xffffffff00000000ull, 0xffffffff00000000ull }; + +#ifdef CONFIG_ARM_AES_BUILTIN + r0 = (uint64x2_t)vmull_p64(un.h, um.h); + r2 = (uint64x2_t)vmull_high_p64((poly64x2_t)un.v, (poly64x2_t)um.v); +#else + asm(".arch_extension aes\n\t" + "pmull %0.1q, %2.1d, %3.1d\n\t" + "pmull2 %1.1q, %2.2d, %3.2d" + : "=&w"(r0), "=w"(r2) : "w"(un.v), "w"(um.v)); +#endif + + ur.v = vzip2q_u64(r0, r2); + return ur.s; +} + +#endif /* AARCH64_HOST_CRYPTO_CLMUL_H */ diff --git a/util/cpuinfo-aarch64.c b/util/cpuinfo-aarch64.c index ababc39550..1d565b8420 100644 --- a/util/cpuinfo-aarch64.c +++ b/util/cpuinfo-aarch64.c @@ -56,12 +56,14 @@ unsigned __attribute__((constructor)) cpuinfo_init(void) unsigned long hwcap = qemu_getauxval(AT_HWCAP); info |= (hwcap & HWCAP_ATOMICS ? CPUINFO_LSE : 0); info |= (hwcap & HWCAP_USCAT ? CPUINFO_LSE2 : 0); - info |= (hwcap & HWCAP_AES ? CPUINFO_AES: 0); + info |= (hwcap & HWCAP_AES ? CPUINFO_AES : 0); + info |= (hwcap & HWCAP_PMULL ? CPUINFO_PMULL : 0); #endif #ifdef CONFIG_DARWIN info |= sysctl_for_bool("hw.optional.arm.FEAT_LSE") * CPUINFO_LSE; info |= sysctl_for_bool("hw.optional.arm.FEAT_LSE2") * CPUINFO_LSE2; info |= sysctl_for_bool("hw.optional.arm.FEAT_AES") * CPUINFO_AES; + info |= sysctl_for_bool("hw.optional.arm.FEAT_PMULL") * CPUINFO_PMULL; #endif cpuinfo = info;