From patchwork Thu Dec 15 06:38:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Elliott, Robert \(Servers\)" X-Patchwork-Id: 634637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE13BC4167B for ; Thu, 15 Dec 2022 06:40:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229815AbiLOGk2 (ORCPT ); Thu, 15 Dec 2022 01:40:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42310 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229766AbiLOGjy (ORCPT ); Thu, 15 Dec 2022 01:39:54 -0500 Received: from mx0b-002e3701.pphosted.com (mx0b-002e3701.pphosted.com [148.163.143.35]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4BAB35C741; Wed, 14 Dec 2022 22:39:44 -0800 (PST) Received: from pps.filterd (m0134425.ppops.net [127.0.0.1]) by mx0b-002e3701.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2BF52DDF014421; Thu, 15 Dec 2022 06:39:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hpe.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pps0720; bh=IJh7WEYxyt2LrgouyYo+ovdw+Y2aJpHTxe4DsSblkuA=; b=T8/+Ydys0fxnqXiC4Z08IzI/kpgdRhaF5kmJfCiYP2XpuSF85n3728YNEdleXdl5HCwb 9VQ40Uxri5KFuV/QphZISDm8D0ieF0rA3wDh5/oua6FwoniGiOZT72xUhp+AtCw8RoAi RgNv0MTxfT3bHqzpv0OWQCKhSeqxcaFnDl/BtSFVzcydJmc9HLm8HeXte0I51oEWEVwS mn/HVIzrHZBx0l2AeK3JOnWs4Yz4yi5loZyCB/2FPde/7GUiMEScgnnQyrYf63sTzaXp WPA4EgpzR35qyZbS1nfSnTCr82iA8EJPcDoaEHCK2duB5xGHTHkPpjazw0iVZkh8Kw0E zg== Received: from p1lg14880.it.hpe.com (p1lg14880.it.hpe.com [16.230.97.201]) by mx0b-002e3701.pphosted.com (PPS) with ESMTPS id 3mfvv2gn2h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 15 Dec 2022 06:39:39 +0000 Received: from p1lg14886.dc01.its.hpecorp.net (unknown [10.119.18.237]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by p1lg14880.it.hpe.com (Postfix) with ESMTPS id EB35580711A; Thu, 15 Dec 2022 06:39:38 +0000 (UTC) Received: from adevxp033-sys.us.rdlabs.hpecorp.net (unknown [16.231.227.36]) by p1lg14886.dc01.its.hpecorp.net (Postfix) with ESMTP id 99FAC80BE0E; Thu, 15 Dec 2022 06:39:38 +0000 (UTC) From: Robert Elliott To: herbert@gondor.apana.org.au, davem@davemloft.net, corbet@lwn.net, linux-crypto@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Robert Elliott Subject: [PATCH 8/8] crypto: x86/chacha - add kernel-doc comments to assembly Date: Thu, 15 Dec 2022 00:38:57 -0600 Message-Id: <20221215063857.161665-9-elliott@hpe.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221215063857.161665-1-elliott@hpe.com> References: <20221215063857.161665-1-elliott@hpe.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: Rseu9TnYuIn6EfjrU9krUBw4arC8KQAn X-Proofpoint-GUID: Rseu9TnYuIn6EfjrU9krUBw4arC8KQAn X-HPE-SCL: -1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-15_02,2022-12-14_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 phishscore=0 mlxlogscore=999 suspectscore=0 adultscore=0 mlxscore=0 bulkscore=0 impostorscore=0 malwarescore=0 spamscore=0 priorityscore=1501 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2212150050 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Add kernel-doc comments for assembly language functions exported to C glue code. Signed-off-by: Robert Elliott --- arch/x86/crypto/chacha-avx2-x86_64.S | 90 +++++++++++++++-------- arch/x86/crypto/chacha-avx512vl-x86_64.S | 94 +++++++++++++++--------- arch/x86/crypto/chacha-ssse3-x86_64.S | 75 ++++++++++++------- 3 files changed, 170 insertions(+), 89 deletions(-) diff --git a/arch/x86/crypto/chacha-avx2-x86_64.S b/arch/x86/crypto/chacha-avx2-x86_64.S index f3d8fc018249..5ebced6f32c3 100644 --- a/arch/x86/crypto/chacha-avx2-x86_64.S +++ b/arch/x86/crypto/chacha-avx2-x86_64.S @@ -34,18 +34,26 @@ CTR4BL: .octa 0x00000000000000000000000000000002 .text +/** + * chacha_2block_xor_avx2 - Encrypt 2 blocks using the x86 AVX2 feature set + * @state: address of input state matrix, s (%rdi) + * @dst: address of up to 2 data blocks output, o (%rsi) + * @src: address of up to 2 data blocks input, i (%rdx) + * @len: input/output length in bytes (%rcx) + * @nrounds: number of rounds (%r8d) + * + * This function supports 64-bit CPUs. + * + * This function encrypts two ChaCha blocks by loading the state + * matrix twice across four AVX registers. It performs matrix operations + * on four words in each matrix in parallel, but requires shuffling to + * rearrange the words after each round. + * + * Return: none + * Prototype: asmlinkage void chacha_2block_xor_avx2(u32 *state, u8 *dst, const u8 *src, + * unsigned int len, int nrounds); + */ SYM_FUNC_START(chacha_2block_xor_avx2) - # %rdi: Input state matrix, s - # %rsi: up to 2 data blocks output, o - # %rdx: up to 2 data blocks input, i - # %rcx: input/output length in bytes - # %r8d: nrounds - - # This function encrypts two ChaCha blocks by loading the state - # matrix twice across four AVX registers. It performs matrix operations - # on four words in each matrix in parallel, but requires shuffling to - # rearrange the words after each round. - vzeroupper # x0..3[0-2] = s0..3 @@ -226,20 +234,28 @@ SYM_FUNC_START(chacha_2block_xor_avx2) SYM_FUNC_END(chacha_2block_xor_avx2) +/** + * chacha_4block_xor_avx2 - Encrypt 4 blocks using the x86 AVX2 feature set + * @state: address of input state matrix, s (%rdi) + * @dst: address of up to 4 data blocks output, o (%rsi) + * @src: address of up to 4 data blocks input, i (%rdx) + * @len: input/output length in bytes (%rcx) + * @nrounds: number of rounds (%r8d) + * + * This function supports 64-bit CPUs. + * + * This function encrypts four ChaCha blocks by loading the state + * matrix four times across eight AVX registers. It performs matrix + * operations on four words in two matrices in parallel, sequentially + * to the operations on the four words of the other two matrices. The + * required word shuffling has a rather high latency, we can do the + * arithmetic on two matrix-pairs without much slowdown. + * + * Return: none + * Prototype: asmlinkage void chacha_4block_xor_avx2(u32 *state, u8 *dst, const u8 *src, + * unsigned int len, int nrounds); + */ SYM_FUNC_START(chacha_4block_xor_avx2) - # %rdi: Input state matrix, s - # %rsi: up to 4 data blocks output, o - # %rdx: up to 4 data blocks input, i - # %rcx: input/output length in bytes - # %r8d: nrounds - - # This function encrypts four ChaCha blocks by loading the state - # matrix four times across eight AVX registers. It performs matrix - # operations on four words in two matrices in parallel, sequentially - # to the operations on the four words of the other two matrices. The - # required word shuffling has a rather high latency, we can do the - # arithmetic on two matrix-pairs without much slowdown. - vzeroupper # x0..3[0-4] = s0..3 @@ -531,12 +547,28 @@ SYM_FUNC_START(chacha_4block_xor_avx2) SYM_FUNC_END(chacha_4block_xor_avx2) +/** + * chacha_8block_xor_avx2 - Encrypt 8 blocks using the x86 AVX2 feature set + * @state: address of input state matrix, s (%rdi) + * @dst: address of up to 8 data blocks output, o (%rsi) + * @src: address of up to 8 data blocks input, i (%rdx) + * @len: input/output length in bytes (%rcx) + * @nrounds: number of rounds (%r8d) + * + * This function supports 64-bit CPUs. + * + * This function encrypts four ChaCha blocks by loading the state + * matrix four times across eight AVX registers. It performs matrix + * operations on four words in two matrices in parallel, sequentially + * to the operations on the four words of the other two matrices. The + * required word shuffling has a rather high latency, we can do the + * arithmetic on two matrix-pairs without much slowdown. + * + * Return: none + * Prototype: asmlinkage void chacha_8block_xor_avx2(u32 *state, u8 *dst, const u8 *src, + * unsigned int len, int nrounds); + */ SYM_FUNC_START(chacha_8block_xor_avx2) - # %rdi: Input state matrix, s - # %rsi: up to 8 data blocks output, o - # %rdx: up to 8 data blocks input, i - # %rcx: input/output length in bytes - # %r8d: nrounds # This function encrypts eight consecutive ChaCha blocks by loading # the state matrix in AVX registers eight times. As we need some diff --git a/arch/x86/crypto/chacha-avx512vl-x86_64.S b/arch/x86/crypto/chacha-avx512vl-x86_64.S index 259383e1ad44..b4a85365e164 100644 --- a/arch/x86/crypto/chacha-avx512vl-x86_64.S +++ b/arch/x86/crypto/chacha-avx512vl-x86_64.S @@ -24,18 +24,26 @@ CTR8BL: .octa 0x00000003000000020000000100000000 .text +/** + * chacha_2block_xor_avx512vl - Encrypt 2 blocks using the x86 AVX512VL feature set + * @state: address of input state matrix, s (%rdi) + * @dst: address of up to 2 data blocks output, o (%rsi) + * @src: address of up to 2 data blocks input, i (%rdx) + * @len: input/output length in bytes (%rcx) + * @nrounds: number of rounds (%r8d) + * + * This function supports 64-bit CPUs. + * + * This function encrypts two ChaCha blocks by loading the state + * matrix twice across four AVX registers. It performs matrix operations + * on four words in each matrix in parallel, but requires shuffling to + * rearrange the words after each round. + * + * Return: none + * Prototype: asmlinkage void chacha_2block_xor_avx512vl(u32 *state, u8 *dst, const u8 *src, + * unsigned int len, int nrounds); + */ SYM_FUNC_START(chacha_2block_xor_avx512vl) - # %rdi: Input state matrix, s - # %rsi: up to 2 data blocks output, o - # %rdx: up to 2 data blocks input, i - # %rcx: input/output length in bytes - # %r8d: nrounds - - # This function encrypts two ChaCha blocks by loading the state - # matrix twice across four AVX registers. It performs matrix operations - # on four words in each matrix in parallel, but requires shuffling to - # rearrange the words after each round. - vzeroupper # x0..3[0-2] = s0..3 @@ -189,20 +197,28 @@ SYM_FUNC_START(chacha_2block_xor_avx512vl) SYM_FUNC_END(chacha_2block_xor_avx512vl) +/** + * chacha_4block_xor_avx512vl - Encrypt 4 blocks using the x86 AVX512VL feature set + * @state: address of input state matrix, s (%rdi) + * @dst: address of up to 4 data blocks output, o (%rsi) + * @src: address of up to 4 data blocks input, i (%rdx) + * @len: input/output length in bytes (%rcx) + * @nrounds: number of rounds (%r8d) + * + * This function supports 64-bit CPUs. + * + * This function encrypts four ChaCha blocks by loading the state + * matrix four times across eight AVX registers. It performs matrix + * operations on four words in two matrices in parallel, sequentially + * to the operations on the four words of the other two matrices. The + * required word shuffling has a rather high latency, we can do the + * arithmetic on two matrix-pairs without much slowdown. + * + * Return: none + * Prototype: asmlinkage void chacha_4block_xor_avx512vl(u32 *state, u8 *dst, const u8 *src, + * unsigned int len, int nrounds); + */ SYM_FUNC_START(chacha_4block_xor_avx512vl) - # %rdi: Input state matrix, s - # %rsi: up to 4 data blocks output, o - # %rdx: up to 4 data blocks input, i - # %rcx: input/output length in bytes - # %r8d: nrounds - - # This function encrypts four ChaCha blocks by loading the state - # matrix four times across eight AVX registers. It performs matrix - # operations on four words in two matrices in parallel, sequentially - # to the operations on the four words of the other two matrices. The - # required word shuffling has a rather high latency, we can do the - # arithmetic on two matrix-pairs without much slowdown. - vzeroupper # x0..3[0-4] = s0..3 @@ -455,18 +471,26 @@ SYM_FUNC_START(chacha_4block_xor_avx512vl) SYM_FUNC_END(chacha_4block_xor_avx512vl) +/** + * chacha_8block_xor_avx512vl - Encrypt 8 blocks using the x86 AVX512VL feature set + * @state: address of input state matrix, s (%rdi) + * @dst: address of up to 8 data blocks output, o (%rsi) + * @src: address of up to 8 data blocks input, i (%rdx) + * @len: input/output length in bytes (%rcx) + * @nrounds: number of rounds (%r8d) + * + * This function supports 64-bit CPUs. + * + * This function encrypts eight consecutive ChaCha blocks by loading + * the state matrix in AVX registers eight times. Compared to AVX2, this + * mostly benefits from the new rotate instructions in VL and the + * additional registers. + * + * Return: none + * Prototype: asmlinkage void chacha_8block_xor_avx512vl(u32 *state, u8 *dst, const u8 *src, + * unsigned int len, int nrounds); + */ SYM_FUNC_START(chacha_8block_xor_avx512vl) - # %rdi: Input state matrix, s - # %rsi: up to 8 data blocks output, o - # %rdx: up to 8 data blocks input, i - # %rcx: input/output length in bytes - # %r8d: nrounds - - # This function encrypts eight consecutive ChaCha blocks by loading - # the state matrix in AVX registers eight times. Compared to AVX2, this - # mostly benefits from the new rotate instructions in VL and the - # additional registers. - vzeroupper # x0..15[0-7] = s[0..15] diff --git a/arch/x86/crypto/chacha-ssse3-x86_64.S b/arch/x86/crypto/chacha-ssse3-x86_64.S index 7111949cd5b9..6f5395ba54ab 100644 --- a/arch/x86/crypto/chacha-ssse3-x86_64.S +++ b/arch/x86/crypto/chacha-ssse3-x86_64.S @@ -34,7 +34,6 @@ CTRINC: .octa 0x00000003000000020000000100000000 * Clobbers: %r8d, %xmm4-%xmm7 */ SYM_FUNC_START_LOCAL(chacha_permute) - movdqa ROT8(%rip),%xmm4 movdqa ROT16(%rip),%xmm5 @@ -111,12 +110,21 @@ SYM_FUNC_START_LOCAL(chacha_permute) RET SYM_FUNC_END(chacha_permute) +/** + * chacha_block_xor_ssse3 - Encrypt 1 block using the x86 SSSE3 feature set + * @state: address of input state matrix, s (%rdi) + * @dst: address of up to 1 data block output, o (%rsi) + * @src: address of up to 1 data block input, i (%rdx) + * @len: input/output length in bytes (%rcx) + * @nrounds: number of rounds (%r8d) + * + * This function supports 64-bit CPUs. + * + * Return: none + * Prototype: asmlinkage void chacha_block_xor_ssse3(u32 *state, u8 *dst, const u8 *src, + * unsigned int len, int nrounds); + */ SYM_FUNC_START(chacha_block_xor_ssse3) - # %rdi: Input state matrix, s - # %rsi: up to 1 data block output, o - # %rdx: up to 1 data block input, i - # %rcx: input/output length in bytes - # %r8d: nrounds FRAME_BEGIN # x0..3 = s0..3 @@ -199,10 +207,19 @@ SYM_FUNC_START(chacha_block_xor_ssse3) SYM_FUNC_END(chacha_block_xor_ssse3) +/** + * hchacha_block_ssse3 - Encrypt 1 block using the x86 SSSE3 feature set + * @state: address of input state matrix, s (%rdu) + * @out: address of output (8 32-bit words)(%rsi) + * @nrounds: number of rounds (%edx); + * only uses lower 32 bits + * + * This function supports 64-bit CPUs. + * + * Return: none + * Prototype: asmlinkage void hchacha_block_ssse3(const u32 *state, u32 *out, int nrounds); + */ SYM_FUNC_START(hchacha_block_ssse3) - # %rdi: Input state matrix, s - # %rsi: output (8 32-bit words) - # %edx: nrounds FRAME_BEGIN movdqu 0x00(%rdi),%xmm0 @@ -220,23 +237,31 @@ SYM_FUNC_START(hchacha_block_ssse3) RET SYM_FUNC_END(hchacha_block_ssse3) +/** + * chacha_4block_xor_ssse3 - Encrypt 4 blocks using the x86 SSSE3 feature set + * @state: address of input state matrix, s (%rdi) + * @dst: address of up to 4 data blocks output, o (%rsi) + * @src: address of up to 4 data blocks input, i (%rdx) + * @len: input/output length in bytes (%rcx) + * @nrounds: number of rounds (%r8d) + * + * This function supports 64-bit CPUs. + * + * This function encrypts four consecutive ChaCha blocks by loading the + * state matrix in SSE registers four times. As we need some scratch + * registers, we save the first four registers on the stack. The + * algorithm performs each operation on the corresponding word of each + * state matrix, hence requires no word shuffling. For final XORing step + * we transpose the matrix by interleaving 32- and then 64-bit words, + * which allows us to do XOR in SSE registers. 8/16-bit word rotation is + * done with the slightly better performing SSSE3 byte shuffling, + * 7/12-bit word rotation uses traditional shift+OR. + * + * Return: none + * Prototype: asmlinkage void chacha_4block_xor_ssse3(u32 *state, u8 *dst, const u8 *src, + * unsigned int len, int nrounds); + */ SYM_FUNC_START(chacha_4block_xor_ssse3) - # %rdi: Input state matrix, s - # %rsi: up to 4 data blocks output, o - # %rdx: up to 4 data blocks input, i - # %rcx: input/output length in bytes - # %r8d: nrounds - - # This function encrypts four consecutive ChaCha blocks by loading the - # the state matrix in SSE registers four times. As we need some scratch - # registers, we save the first four registers on the stack. The - # algorithm performs each operation on the corresponding word of each - # state matrix, hence requires no word shuffling. For final XORing step - # we transpose the matrix by interleaving 32- and then 64-bit words, - # which allows us to do XOR in SSE registers. 8/16-bit word rotation is - # done with the slightly better performing SSSE3 byte shuffling, - # 7/12-bit word rotation uses traditional shift+OR. - lea 8(%rsp),%r10 sub $0x80,%rsp and $~63,%rsp