From patchwork Fri Jul 22 12:21:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 592462 Delivered-To: patch@linaro.org Received: by 2002:a05:7000:b811:0:0:0:0 with SMTP id fc17csp451011mab; Fri, 22 Jul 2022 05:22:45 -0700 (PDT) X-Google-Smtp-Source: AGRyM1udesYx2l45PhWdgz0LLDCtgnvAbazs3Dj12hX4V4hA9NCQ1Ssv17hUu/+dVOaXDS/+3Lbk X-Received: by 2002:a05:6402:d76:b0:43b:c772:58b5 with SMTP id ec54-20020a0564020d7600b0043bc77258b5mr374896edb.47.1658492565666; Fri, 22 Jul 2022 05:22:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1658492565; cv=none; d=google.com; s=arc-20160816; b=vnQPwM3QTZsXIzhe5Pse9qDITPGhwDqP48pARRRQmzGBf76KwjKS8+J5ze0TQ8Gn/w UM72ufV08WVn8QXhawxnBfeB7bIeQrGOLpxKw4Y/8wkVjRYMaOOJ9maPNT74AvXkWUsX 3t8FLZWvWWJWPB6hMhyztxJNThlEvKxwIFwdkBpARPaSBk+ix0VjegT8Kg21KDX2rUv3 H7sSXOF+LYBKjpGtpfME5a5LXgKHP79kKpuojeE1Y/1xalnubNg63p2n/0utcIg8omYx Iivgxwh9h8xk1YE51Paa41sK3JrVPZ61sCLWHqBH0oHj4lSHv/U6xLKmoRl6wmiz17bq dzKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:reply-to:from:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:dmarc-filter:delivered-to:dkim-signature :dkim-filter; bh=/nenQLnK07XiYYjT11evXg/vylH0zq4YojiBl4sdplY=; b=l1Bz/ft82Q7lY/J81JOZmjDCt9Dyn7x5m1Zd8lf94A2oIQnhm2qqxYurom4ayg08Tz 9dhc2DFSSiXeOm0NiD9AQEyhCeZAIZWRTHdLBkpUQ4FrvXdE0bxzpjgFmtv4W+uD52ZJ YiEu7yprV82WZEM2XKGaAmTNAyjNgATxP40LnrqaCstaPY480YwOOf1cruOjPUiH19rK 2gcB6Eo8PNF2FmcQZyq0lfFmUfikMUgzi/cb0NPl8zWgdpkK+oro8E4hhkr1AutSqiIc Cd/0SEEwBolfNgtpgdrMxV3CEMK8G0drA0QAO9mjBrLduPHEpV/HEbbe4tm8i/efFd+h KECA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sourceware.org header.s=default header.b=c5obRsDp; spf=pass (google.com: domain of libc-alpha-bounces+patch=linaro.org@sourceware.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="libc-alpha-bounces+patch=linaro.org@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sourceware.org Return-Path: Received: from sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id i10-20020a1709064fca00b0072afc15af40si6225668ejw.39.2022.07.22.05.22.45 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Jul 2022 05:22:45 -0700 (PDT) Received-SPF: pass (google.com: domain of libc-alpha-bounces+patch=linaro.org@sourceware.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; dkim=pass header.i=@sourceware.org header.s=default header.b=c5obRsDp; spf=pass (google.com: domain of libc-alpha-bounces+patch=linaro.org@sourceware.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="libc-alpha-bounces+patch=linaro.org@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 3ECED382C153 for ; Fri, 22 Jul 2022 12:22:44 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 3ECED382C153 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1658492564; bh=/nenQLnK07XiYYjT11evXg/vylH0zq4YojiBl4sdplY=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=c5obRsDpf4i60PWkyTsEuh55qiIdNTUVk7Fy4jy/qejFbSwpoMMhz+HYqMJVSODqu yXRjnBvElGUUtocq0g0ei3DPa/zVX2XJsTJpnot4aMNuQh3Pxp/kI1CR769MaK9PZ6 L5WHOKsLgkAsUEHLDa1bsI0jMSr/azd0FNrRKjKI= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-oi1-x236.google.com (mail-oi1-x236.google.com [IPv6:2607:f8b0:4864:20::236]) by sourceware.org (Postfix) with ESMTPS id C6D5F3835438 for ; Fri, 22 Jul 2022 12:21:59 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org C6D5F3835438 Received: by mail-oi1-x236.google.com with SMTP id v130so185364oie.13 for ; Fri, 22 Jul 2022 05:21:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/nenQLnK07XiYYjT11evXg/vylH0zq4YojiBl4sdplY=; b=xweYNB/snto77zBJAbTY9KaCGS27dXgmSO8LxTNTYJp0EhGiu6A5B6nKH3rhC7pRzo skq/oYgrOIoko++QNWZiWA0egeOnGTOueJG1T8bNb4DR/l1zG46g21ykfyJe5xNO0dE6 HMIVRsO28BJtqW9ZyDXaOPqITdAxN1k5mmu0EXlbqqykpuSnUJUU+jbRrkRvq/+tV7LA f/Mpa2GfTeWoN3ObvDAI49VKVuivXO20ixUk12jQnz2FO7LwFOqNwahuACb5BUzs6DIe yUALyI7Pa+njwVsHuiIA5lbuDMml9znb0DmGS54BV5UgYKc0IMUB+/vQCU312mvgTSRx tMlw== X-Gm-Message-State: AJIora9AByYCBN5SCEB8S0Y20PMkcMfMYQ3I9Oq/B1K1YYofbRkcFdqg cOTB+baG9QaYanoeZq3xJDzt5JeC6v35rw== X-Received: by 2002:a54:4e8d:0:b0:325:988d:afaf with SMTP id c13-20020a544e8d000000b00325988dafafmr6559612oiy.135.1658492518715; Fri, 22 Jul 2022 05:21:58 -0700 (PDT) Received: from mandiga.. ([2804:431:c7cb:8ded:a055:4527:6234:319b]) by smtp.gmail.com with ESMTPSA id 188-20020a4a1bc5000000b0035eb4e5a6cbsm1831953oop.33.2022.07.22.05.21.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Jul 2022 05:21:58 -0700 (PDT) To: libc-alpha@sourceware.org, Florian Weimer , Carlos O'Donell , Szabolcs Nagy Subject: [PATCH v12 7/9] powerpc64: Add optimized chacha20 Date: Fri, 22 Jul 2022 09:21:35 -0300 Message-Id: <20220722122137.3270666-8-adhemerval.zanella@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220722122137.3270666-1-adhemerval.zanella@linaro.org> References: <20220722122137.3270666-1-adhemerval.zanella@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-11.4 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_ASCII_DIVIDERS, KAM_NUMSUBJECT, KAM_SHORT, RCVD_IN_DNSWL_NONE, SCC_5_SHORT_WORD_LINES, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Adhemerval Zanella via Libc-alpha From: Adhemerval Zanella Netto Reply-To: Adhemerval Zanella Cc: "Paul E . Murphy" Errors-To: libc-alpha-bounces+patch=linaro.org@sourceware.org Sender: "Libc-alpha" From: Adhemerval Zanella Netto It adds vectorized ChaCha20 implementation based on libgcrypt cipher/chacha20-ppc.c. It targets POWER8 and it is used on default for LE. On a POWER8 it shows the following improvements (using formatted bench-arc4random data): POWER8 GENERIC MB/s ----------------------------------------------- arc4random [single-thread] 138.77 arc4random_buf(16) [single-thread] 174.36 arc4random_buf(32) [single-thread] 228.11 arc4random_buf(48) [single-thread] 252.31 arc4random_buf(64) [single-thread] 270.11 arc4random_buf(80) [single-thread] 278.97 arc4random_buf(96) [single-thread] 287.78 arc4random_buf(112) [single-thread] 291.92 arc4random_buf(128) [single-thread] 295.25 POWER8 MB/s ----------------------------------------------- arc4random [single-thread] 198.06 arc4random_buf(16) [single-thread] 278.79 arc4random_buf(32) [single-thread] 448.89 arc4random_buf(48) [single-thread] 551.09 arc4random_buf(64) [single-thread] 646.12 arc4random_buf(80) [single-thread] 698.04 arc4random_buf(96) [single-thread] 756.06 arc4random_buf(112) [single-thread] 784.12 arc4random_buf(128) [single-thread] 808.04 ----------------------------------------------- Checked on powerpc64-linux-gnu and powerpc64le-linux-gnu. Reviewed-by: Paul E. Murphy --- LICENSES | 3 +- .../powerpc/powerpc64/be/multiarch/Makefile | 4 + .../powerpc64/be/multiarch/chacha20-ppc.c | 1 + .../powerpc64/be/multiarch/chacha20_arch.h | 41 +++ sysdeps/powerpc/powerpc64/power8/Makefile | 5 + .../powerpc/powerpc64/power8/chacha20-ppc.c | 256 ++++++++++++++++++ .../powerpc/powerpc64/power8/chacha20_arch.h | 36 +++ 7 files changed, 345 insertions(+), 1 deletion(-) create mode 100644 sysdeps/powerpc/powerpc64/be/multiarch/Makefile create mode 100644 sysdeps/powerpc/powerpc64/be/multiarch/chacha20-ppc.c create mode 100644 sysdeps/powerpc/powerpc64/be/multiarch/chacha20_arch.h create mode 100644 sysdeps/powerpc/powerpc64/power8/chacha20-ppc.c create mode 100644 sysdeps/powerpc/powerpc64/power8/chacha20_arch.h diff --git a/LICENSES b/LICENSES index 80168d0b1b..e177af6035 100644 --- a/LICENSES +++ b/LICENSES @@ -391,7 +391,8 @@ Copyright 2001 by Stephen L. Moshier . */ sysdeps/aarch64/chacha20-aarch64.S, sysdeps/x86_64/chacha20-amd64-sse2.S, -and sysdeps/x86_64/chacha20-amd64-avx2.S imports code from libgcrypt, +sysdeps/x86_64/chacha20-amd64-avx2.S, and +sysdeps/powerpc/powerpc64/power8/chacha20-ppc.c imports code from libgcrypt, with the following notices: Copyright (C) 2017-2019 Jussi Kivilinna diff --git a/sysdeps/powerpc/powerpc64/be/multiarch/Makefile b/sysdeps/powerpc/powerpc64/be/multiarch/Makefile new file mode 100644 index 0000000000..8c75165f7f --- /dev/null +++ b/sysdeps/powerpc/powerpc64/be/multiarch/Makefile @@ -0,0 +1,4 @@ +ifeq ($(subdir),stdlib) +sysdep_routines += chacha20-ppc +CFLAGS-chacha20-ppc.c += -mcpu=power8 +endif diff --git a/sysdeps/powerpc/powerpc64/be/multiarch/chacha20-ppc.c b/sysdeps/powerpc/powerpc64/be/multiarch/chacha20-ppc.c new file mode 100644 index 0000000000..cf9e735326 --- /dev/null +++ b/sysdeps/powerpc/powerpc64/be/multiarch/chacha20-ppc.c @@ -0,0 +1 @@ +#include diff --git a/sysdeps/powerpc/powerpc64/be/multiarch/chacha20_arch.h b/sysdeps/powerpc/powerpc64/be/multiarch/chacha20_arch.h new file mode 100644 index 0000000000..c9ff8b0d27 --- /dev/null +++ b/sysdeps/powerpc/powerpc64/be/multiarch/chacha20_arch.h @@ -0,0 +1,41 @@ +/* PowerPC optimization for ChaCha20. + Copyright (C) 2022 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include + +unsigned int __chacha20_power8_blocks4 (uint32_t *state, uint8_t *dst, + const uint8_t *src, size_t nblks) + attribute_hidden; + +static void +chacha20_crypt (uint32_t *state, uint8_t *dst, const uint8_t *src) +{ + _Static_assert (CHACHA20_BUFSIZE % 4 == 0, + "CHACHA20_BUFSIZE not multiple of 4"); + _Static_assert (CHACHA20_BUFSIZE >= CHACHA20_BLOCK_SIZE * 4, + "CHACHA20_BUFSIZE < CHACHA20_BLOCK_SIZE * 4"); + + unsigned long int hwcap = GLRO(dl_hwcap); + unsigned long int hwcap2 = GLRO(dl_hwcap2); + if (hwcap2 & PPC_FEATURE2_ARCH_2_07 && hwcap & PPC_FEATURE_HAS_ALTIVEC) + __chacha20_power8_blocks4 (state, dst, src, + CHACHA20_BUFSIZE / CHACHA20_BLOCK_SIZE); + else + chacha20_crypt_generic (state, dst, src); +} diff --git a/sysdeps/powerpc/powerpc64/power8/Makefile b/sysdeps/powerpc/powerpc64/power8/Makefile index 71a59529f3..abb0aa3f11 100644 --- a/sysdeps/powerpc/powerpc64/power8/Makefile +++ b/sysdeps/powerpc/powerpc64/power8/Makefile @@ -1,3 +1,8 @@ ifeq ($(subdir),string) sysdep_routines += strcasestr-ppc64 endif + +ifeq ($(subdir),stdlib) +sysdep_routines += chacha20-ppc +CFLAGS-chacha20-ppc.c += -mcpu=power8 +endif diff --git a/sysdeps/powerpc/powerpc64/power8/chacha20-ppc.c b/sysdeps/powerpc/powerpc64/power8/chacha20-ppc.c new file mode 100644 index 0000000000..0bbdcb9363 --- /dev/null +++ b/sysdeps/powerpc/powerpc64/power8/chacha20-ppc.c @@ -0,0 +1,256 @@ +/* Optimized PowerPC implementation of ChaCha20 cipher. + Copyright (C) 2022 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +/* chacha20-ppc.c - PowerPC vector implementation of ChaCha20 + Copyright (C) 2019 Jussi Kivilinna + + This file is part of Libgcrypt. + + Libgcrypt is free software; you can redistribute it and/or modify + it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of + the License, or (at your option) any later version. + + Libgcrypt is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this program; if not, see . + */ + +#include +#include +#include +#include +#include + +typedef vector unsigned char vector16x_u8; +typedef vector unsigned int vector4x_u32; +typedef vector unsigned long long vector2x_u64; + +#if __BYTE_ORDER == __BIG_ENDIAN +static const vector16x_u8 le_bswap_const = + { 3, 2, 1, 0, 7, 6, 5, 4, 11, 10, 9, 8, 15, 14, 13, 12 }; +#endif + +static inline vector4x_u32 +vec_rol_elems (vector4x_u32 v, unsigned int idx) +{ +#if __BYTE_ORDER != __BIG_ENDIAN + return vec_sld (v, v, (16 - (4 * idx)) & 15); +#else + return vec_sld (v, v, (4 * idx) & 15); +#endif +} + +static inline vector4x_u32 +vec_load_le (unsigned long offset, const unsigned char *ptr) +{ + vector4x_u32 vec; + vec = vec_vsx_ld (offset, (const uint32_t *)ptr); +#if __BYTE_ORDER == __BIG_ENDIAN + vec = (vector4x_u32) vec_perm ((vector16x_u8)vec, (vector16x_u8)vec, + le_bswap_const); +#endif + return vec; +} + +static inline void +vec_store_le (vector4x_u32 vec, unsigned long offset, unsigned char *ptr) +{ +#if __BYTE_ORDER == __BIG_ENDIAN + vec = (vector4x_u32)vec_perm((vector16x_u8)vec, (vector16x_u8)vec, + le_bswap_const); +#endif + vec_vsx_st (vec, offset, (uint32_t *)ptr); +} + + +static inline vector4x_u32 +vec_add_ctr_u64 (vector4x_u32 v, vector4x_u32 a) +{ +#if __BYTE_ORDER == __BIG_ENDIAN + static const vector16x_u8 swap32 = + { 4, 5, 6, 7, 0, 1, 2, 3, 12, 13, 14, 15, 8, 9, 10, 11 }; + vector2x_u64 vec, add, sum; + + vec = (vector2x_u64)vec_perm ((vector16x_u8)v, (vector16x_u8)v, swap32); + add = (vector2x_u64)vec_perm ((vector16x_u8)a, (vector16x_u8)a, swap32); + sum = vec + add; + return (vector4x_u32)vec_perm ((vector16x_u8)sum, (vector16x_u8)sum, swap32); +#else + return (vector4x_u32)((vector2x_u64)(v) + (vector2x_u64)(a)); +#endif +} + +/********************************************************************** + 4-way chacha20 + **********************************************************************/ + +#define ROTATE(v1,rolv) \ + __asm__ ("vrlw %0,%1,%2\n\t" : "=v" (v1) : "v" (v1), "v" (rolv)) + +#define PLUS(ds,s) \ + ((ds) += (s)) + +#define XOR(ds,s) \ + ((ds) ^= (s)) + +#define ADD_U64(v,a) \ + (v = vec_add_ctr_u64(v, a)) + +/* 4x4 32-bit integer matrix transpose */ +#define transpose_4x4(x0, x1, x2, x3) ({ \ + vector4x_u32 t1 = vec_mergeh(x0, x2); \ + vector4x_u32 t2 = vec_mergel(x0, x2); \ + vector4x_u32 t3 = vec_mergeh(x1, x3); \ + x3 = vec_mergel(x1, x3); \ + x0 = vec_mergeh(t1, t3); \ + x1 = vec_mergel(t1, t3); \ + x2 = vec_mergeh(t2, x3); \ + x3 = vec_mergel(t2, x3); \ + }) + +#define QUARTERROUND2(a1,b1,c1,d1,a2,b2,c2,d2) \ + PLUS(a1,b1); PLUS(a2,b2); XOR(d1,a1); XOR(d2,a2); \ + ROTATE(d1, rotate_16); ROTATE(d2, rotate_16); \ + PLUS(c1,d1); PLUS(c2,d2); XOR(b1,c1); XOR(b2,c2); \ + ROTATE(b1, rotate_12); ROTATE(b2, rotate_12); \ + PLUS(a1,b1); PLUS(a2,b2); XOR(d1,a1); XOR(d2,a2); \ + ROTATE(d1, rotate_8); ROTATE(d2, rotate_8); \ + PLUS(c1,d1); PLUS(c2,d2); XOR(b1,c1); XOR(b2,c2); \ + ROTATE(b1, rotate_7); ROTATE(b2, rotate_7); + +unsigned int attribute_hidden +__chacha20_power8_blocks4 (uint32_t *state, uint8_t *dst, const uint8_t *src, + size_t nblks) +{ + vector4x_u32 counters_0123 = { 0, 1, 2, 3 }; + vector4x_u32 counter_4 = { 4, 0, 0, 0 }; + vector4x_u32 rotate_16 = { 16, 16, 16, 16 }; + vector4x_u32 rotate_12 = { 12, 12, 12, 12 }; + vector4x_u32 rotate_8 = { 8, 8, 8, 8 }; + vector4x_u32 rotate_7 = { 7, 7, 7, 7 }; + vector4x_u32 state0, state1, state2, state3; + vector4x_u32 v0, v1, v2, v3, v4, v5, v6, v7; + vector4x_u32 v8, v9, v10, v11, v12, v13, v14, v15; + vector4x_u32 tmp; + int i; + + /* Force preload of constants to vector registers. */ + __asm__ ("": "+v" (counters_0123) :: "memory"); + __asm__ ("": "+v" (counter_4) :: "memory"); + __asm__ ("": "+v" (rotate_16) :: "memory"); + __asm__ ("": "+v" (rotate_12) :: "memory"); + __asm__ ("": "+v" (rotate_8) :: "memory"); + __asm__ ("": "+v" (rotate_7) :: "memory"); + + state0 = vec_vsx_ld (0 * 16, state); + state1 = vec_vsx_ld (1 * 16, state); + state2 = vec_vsx_ld (2 * 16, state); + state3 = vec_vsx_ld (3 * 16, state); + + do + { + v0 = vec_splat (state0, 0); + v1 = vec_splat (state0, 1); + v2 = vec_splat (state0, 2); + v3 = vec_splat (state0, 3); + v4 = vec_splat (state1, 0); + v5 = vec_splat (state1, 1); + v6 = vec_splat (state1, 2); + v7 = vec_splat (state1, 3); + v8 = vec_splat (state2, 0); + v9 = vec_splat (state2, 1); + v10 = vec_splat (state2, 2); + v11 = vec_splat (state2, 3); + v12 = vec_splat (state3, 0); + v13 = vec_splat (state3, 1); + v14 = vec_splat (state3, 2); + v15 = vec_splat (state3, 3); + + v12 += counters_0123; + v13 -= vec_cmplt (v12, counters_0123); + + for (i = 20; i > 0; i -= 2) + { + QUARTERROUND2 (v0, v4, v8, v12, v1, v5, v9, v13) + QUARTERROUND2 (v2, v6, v10, v14, v3, v7, v11, v15) + QUARTERROUND2 (v0, v5, v10, v15, v1, v6, v11, v12) + QUARTERROUND2 (v2, v7, v8, v13, v3, v4, v9, v14) + } + + v0 += vec_splat (state0, 0); + v1 += vec_splat (state0, 1); + v2 += vec_splat (state0, 2); + v3 += vec_splat (state0, 3); + v4 += vec_splat (state1, 0); + v5 += vec_splat (state1, 1); + v6 += vec_splat (state1, 2); + v7 += vec_splat (state1, 3); + v8 += vec_splat (state2, 0); + v9 += vec_splat (state2, 1); + v10 += vec_splat (state2, 2); + v11 += vec_splat (state2, 3); + tmp = vec_splat( state3, 0); + tmp += counters_0123; + v12 += tmp; + v13 += vec_splat (state3, 1) - vec_cmplt (tmp, counters_0123); + v14 += vec_splat (state3, 2); + v15 += vec_splat (state3, 3); + ADD_U64 (state3, counter_4); + + transpose_4x4 (v0, v1, v2, v3); + transpose_4x4 (v4, v5, v6, v7); + transpose_4x4 (v8, v9, v10, v11); + transpose_4x4 (v12, v13, v14, v15); + + vec_store_le (v0, (64 * 0 + 16 * 0), dst); + vec_store_le (v1, (64 * 1 + 16 * 0), dst); + vec_store_le (v2, (64 * 2 + 16 * 0), dst); + vec_store_le (v3, (64 * 3 + 16 * 0), dst); + + vec_store_le (v4, (64 * 0 + 16 * 1), dst); + vec_store_le (v5, (64 * 1 + 16 * 1), dst); + vec_store_le (v6, (64 * 2 + 16 * 1), dst); + vec_store_le (v7, (64 * 3 + 16 * 1), dst); + + vec_store_le (v8, (64 * 0 + 16 * 2), dst); + vec_store_le (v9, (64 * 1 + 16 * 2), dst); + vec_store_le (v10, (64 * 2 + 16 * 2), dst); + vec_store_le (v11, (64 * 3 + 16 * 2), dst); + + vec_store_le (v12, (64 * 0 + 16 * 3), dst); + vec_store_le (v13, (64 * 1 + 16 * 3), dst); + vec_store_le (v14, (64 * 2 + 16 * 3), dst); + vec_store_le (v15, (64 * 3 + 16 * 3), dst); + + src += 4*64; + dst += 4*64; + + nblks -= 4; + } + while (nblks); + + vec_vsx_st (state3, 3 * 16, state); + + return 0; +} diff --git a/sysdeps/powerpc/powerpc64/power8/chacha20_arch.h b/sysdeps/powerpc/powerpc64/power8/chacha20_arch.h new file mode 100644 index 0000000000..8aa0bb026c --- /dev/null +++ b/sysdeps/powerpc/powerpc64/power8/chacha20_arch.h @@ -0,0 +1,36 @@ +/* PowerPC optimization for ChaCha20. + Copyright (C) 2022 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include + +unsigned int __chacha20_power8_blocks4 (uint32_t *state, uint8_t *dst, + const uint8_t *src, size_t nblks) + attribute_hidden; + +static void +chacha20_crypt (uint32_t *state, uint8_t *dst, const uint8_t *src) +{ + _Static_assert (CHACHA20_BUFSIZE % 4 == 0, + "CHACHA20_BUFSIZE not multiple of 4"); + _Static_assert (CHACHA20_BUFSIZE >= CHACHA20_BLOCK_SIZE * 4, + "CHACHA20_BUFSIZE < CHACHA20_BLOCK_SIZE * 4"); + + __chacha20_power8_blocks4 (state, dst, src, + CHACHA20_BUFSIZE / CHACHA20_BLOCK_SIZE); +}