From patchwork Sun Nov 17 00:22:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 843993 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EF736B67A; Sun, 17 Nov 2024 00:23:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731803035; cv=none; b=jKvacCQ/lETFgtOiwh2vJvn2uGyt5ydkLXHqItA9G4HThGKD8M7saxJfKwARg2//zr5zdFbze05F7grp8Vinixh0kQcJmOjrdnd7fNLRoVMAMffA6bKu/FGfP7ChbZa63XQKDfdC4WOOH5WpRFtbv80EiJoydK/KkqaC15u1is8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731803035; c=relaxed/simple; bh=SOmKtc1U+Cth3ARmtiU8OIUv0APQu6lfcmWi1oOE0T4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PsmrXphSxK4JxIwweVrNDTwQ5YsyopEjUPcz2/Se92O5E9Z7Z1HoCVOu9ktep85lZIypC/z5iSfg9/RLq6ifpaw9LC9tp2G6HoK5Zx1uhr7NyajqFw3dOWms/g2jBsQAyeVf/QFJfmywg2PILPsXrKAdeBNjGnWnC2Z2oFrmgHM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Ae3xjGCt; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Ae3xjGCt" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2EC2CC4CEDB; Sun, 17 Nov 2024 00:23:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731803034; bh=SOmKtc1U+Cth3ARmtiU8OIUv0APQu6lfcmWi1oOE0T4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ae3xjGCtOs/y/2IA9+TlSWBoIoshQQNPXgr0SgZ3f/r3v0tK6ppp+TNKKkk5FhMaW fzhPJxnAGNYoXtldO4TD34DmtPZGZJVe6afCKZa77lMI67XEvqDcdJsy/eOf4UQzu+ UWMfcIvzW7ZZUSkslybJsRHzXpnJ9NwfZeU9Hl8H24dX7fea6Wx8fkBvKzt0IJevAp +Oa1FxY9qXH1f+iivqzfVTgaHpriJPKUjS8BHISYZq/3g2CZr3mrS7oCMW9kzxxkPJ jiuDIbfYJj5EPXDpkPR8PqnoUcYxPOZMjuZXUqo2eVxc9uVn3W3IGA73Qbdf+fCtM+ kIcqazZYknmnw== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-crypto@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, Zhihang Shao , Ard Biesheuvel Subject: [PATCH 02/11] lib/crc-t10dif: add support for arch overrides Date: Sat, 16 Nov 2024 16:22:35 -0800 Message-ID: <20241117002244.105200-3-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241117002244.105200-1-ebiggers@kernel.org> References: <20241117002244.105200-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Following what was done for CRC32, add support for architecture-specific override of the CRC-T10DIF library. This will allow the CRC-T10DIF library functions to access architecture-optimized code directly. Signed-off-by: Eric Biggers --- include/linux/crc-t10dif.h | 12 ++++++++++++ lib/Kconfig | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+) diff --git a/include/linux/crc-t10dif.h b/include/linux/crc-t10dif.h index 206ba2305483..16787c1cee21 100644 --- a/include/linux/crc-t10dif.h +++ b/include/linux/crc-t10dif.h @@ -5,18 +5,30 @@ #include #define CRC_T10DIF_DIGEST_SIZE 2 #define CRC_T10DIF_BLOCK_SIZE 1 +u16 crc_t10dif_arch(u16 crc, const u8 *p, size_t len); u16 crc_t10dif_generic(u16 crc, const u8 *p, size_t len); static inline u16 crc_t10dif_update(u16 crc, const u8 *p, size_t len) { + if (IS_ENABLED(CONFIG_CRC_T10DIF_ARCH)) + return crc_t10dif_arch(crc, p, len); return crc_t10dif_generic(crc, p, len); } static inline u16 crc_t10dif(const u8 *p, size_t len) { return crc_t10dif_update(0, p, len); } +#if IS_ENABLED(CONFIG_CRC_T10DIF_ARCH) +bool crc_t10dif_is_optimized(void); +#else +static inline bool crc_t10dif_is_optimized(void) +{ + return false; +} +#endif + #endif diff --git a/lib/Kconfig b/lib/Kconfig index eb6c7a023be9..b84aa06ade30 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -147,10 +147,42 @@ config CRC_T10DIF help This option is only needed if a module that's not in the kernel tree needs to calculate CRC checks for use with the SCSI data integrity subsystem. +config ARCH_HAS_CRC_T10DIF + bool + +choice + prompt "CRC-T10DIF implementation" + depends on CRC_T10DIF + default CRC_T10DIF_IMPL_ARCH if ARCH_HAS_CRC_T10DIF + default CRC_T10DIF_IMPL_GENERIC if !ARCH_HAS_CRC_T10DIF + help + This option allows you to override the default choice of CRC-T10DIF + implementation. + +config CRC_T10DIF_IMPL_ARCH + bool "Architecture-optimized" if ARCH_HAS_CRC_T10DIF + help + Use the optimized implementation of CRC-T10DIF for the selected + architecture. It is recommended to keep this enabled, as it can + greatly improve CRC-T10DIF performance. + +config CRC_T10DIF_IMPL_GENERIC + bool "Generic implementation" + help + Use the generic table-based implementation of CRC-T10DIF. Selecting + this will reduce code size slightly but can greatly reduce CRC-T10DIF + performance. + +endchoice + +config CRC_T10DIF_ARCH + tristate + default CRC_T10DIF if CRC_T10DIF_IMPL_ARCH + config CRC64_ROCKSOFT tristate "CRC calculation for the Rocksoft model CRC64" select CRC64 select CRYPTO select CRYPTO_CRC64_ROCKSOFT From patchwork Sun Nov 17 00:22:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 843992 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 66AD0D529; Sun, 17 Nov 2024 00:23:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731803035; cv=none; b=hN68wb2Z5AV73wHph4CGeXE02xIjwK6gVZWETZ48N7tXoNp5yBE2Rk67NEyOSfQQ1a9Z8hujgHXzuf0aYD+85G058AD+KQPctyD1Omb0VCKJQ5znT0hUwtmXheSZZWAYGKnDb72wL0S9stDWXlZMjdoNzM1d87Fttd3sqSnt2Wg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731803035; c=relaxed/simple; bh=gzdPLhhkLERJCguDKATRZ050D7HdtyySUPbi/TiqkkA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FfEaVNmvENP7WTY1avgQ36hACIl4PeD2pUn5B4X0xC55Kh/B8dLqgEuLwMSfATAnyZtDj8qeN2GJ2DxKTy0oU9JIKRmoiJpRlbpwA4ae0A8ZlXElA2+oatgAqmie5tdhgJ+708YWsCcOvjwRZz13UgewMm3Xofd+XycWV478Pj8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ZhVnyRJc; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ZhVnyRJc" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 05267C4CED2; Sun, 17 Nov 2024 00:23:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731803035; bh=gzdPLhhkLERJCguDKATRZ050D7HdtyySUPbi/TiqkkA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZhVnyRJcHee6aSGYwXhwyotedN0bibRkeEgtSzh1Rf/aHpK4SyDEc2MgRnxGIMXQt VlouS8AgLVpiBu/BOyEvXD61Oca4Rz3eQtQIlgqfzUW5/02gZ1zpUnX56WKGWdZGIK EN8TUTZ2N7bc0F1Cp0kq0KM+QiPlM7CxVeCDURJzod0IaeFF1M1UyGFVPRZlfGklJZ DVIHEhE86CzX3aSKfvRTSFu060/yNxH6pV1ceFmcP2trn+if4ruwqFVBCj20Y+WyzU +wSvGHjaxO8dHANwyKTwGTR1H8zPM72q4o0WLFzvnlHS20MPs5XRMye4YBUT8BlDAi 8oEoLy7ogzILg== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-crypto@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, Zhihang Shao , Ard Biesheuvel Subject: [PATCH 04/11] x86/crc-t10dif: expose CRC-T10DIF function through lib Date: Sat, 16 Nov 2024 16:22:37 -0800 Message-ID: <20241117002244.105200-5-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241117002244.105200-1-ebiggers@kernel.org> References: <20241117002244.105200-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Move the x86 CRC-T10DIF assembly code into the lib directory and wire it up to the library interface. This allows it to be used without going through the crypto API. It remains usable via the crypto API too via the shash algorithms that use the library interface. Thus all the arch-specific "shash" code becomes unnecessary and is removed. Signed-off-by: Eric Biggers --- arch/x86/Kconfig | 1 + arch/x86/crypto/Kconfig | 10 -- arch/x86/crypto/Makefile | 3 - arch/x86/crypto/crct10dif-pclmul_glue.c | 143 ------------------ arch/x86/lib/Makefile | 3 + arch/x86/lib/crc-t10dif-glue.c | 51 +++++++ .../{crypto => lib}/crct10dif-pcl-asm_64.S | 0 7 files changed, 55 insertions(+), 156 deletions(-) delete mode 100644 arch/x86/crypto/crct10dif-pclmul_glue.c create mode 100644 arch/x86/lib/crc-t10dif-glue.c rename arch/x86/{crypto => lib}/crct10dif-pcl-asm_64.S (100%) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index e7470de11cec..fd3f9be3b932 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -75,10 +75,11 @@ config X86 select ARCH_HAS_CACHE_LINE_SIZE select ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION select ARCH_HAS_CPU_FINALIZE_INIT select ARCH_HAS_CPU_PASID if IOMMU_SVA select ARCH_HAS_CRC32 + select ARCH_HAS_CRC_T10DIF if X86_64 select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEBUG_VM_PGTABLE if !X86_PAE select ARCH_HAS_DEVMEM_IS_ALLOWED select ARCH_HAS_DMA_OPS if GART_IOMMU || XEN diff --git a/arch/x86/crypto/Kconfig b/arch/x86/crypto/Kconfig index 1ca53e847966..2ab31fda4d88 100644 --- a/arch/x86/crypto/Kconfig +++ b/arch/x86/crypto/Kconfig @@ -490,16 +490,6 @@ config CRYPTO_GHASH_CLMUL_NI_INTEL GCM GHASH hash function (NIST SP800-38D) Architecture: x86_64 using: - CLMUL-NI (carry-less multiplication new instructions) -config CRYPTO_CRCT10DIF_PCLMUL - tristate "CRCT10DIF (PCLMULQDQ)" - depends on X86 && 64BIT && CRC_T10DIF - select CRYPTO_HASH - help - CRC16 CRC algorithm used for the T10 (SCSI) Data Integrity Field (DIF) - - Architecture: x86_64 using: - - PCLMULQDQ (carry-less multiplication) - endmenu diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile index 030b925ca4e2..07b00bfca64b 100644 --- a/arch/x86/crypto/Makefile +++ b/arch/x86/crypto/Makefile @@ -73,13 +73,10 @@ obj-$(CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL) += ghash-clmulni-intel.o ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o obj-$(CONFIG_CRYPTO_POLYVAL_CLMUL_NI) += polyval-clmulni.o polyval-clmulni-y := polyval-clmulni_asm.o polyval-clmulni_glue.o -obj-$(CONFIG_CRYPTO_CRCT10DIF_PCLMUL) += crct10dif-pclmul.o -crct10dif-pclmul-y := crct10dif-pcl-asm_64.o crct10dif-pclmul_glue.o - obj-$(CONFIG_CRYPTO_POLY1305_X86_64) += poly1305-x86_64.o poly1305-x86_64-y := poly1305-x86_64-cryptogams.o poly1305_glue.o targets += poly1305-x86_64-cryptogams.S obj-$(CONFIG_CRYPTO_NHPOLY1305_SSE2) += nhpoly1305-sse2.o diff --git a/arch/x86/crypto/crct10dif-pclmul_glue.c b/arch/x86/crypto/crct10dif-pclmul_glue.c deleted file mode 100644 index 71291d5af9f4..000000000000 --- a/arch/x86/crypto/crct10dif-pclmul_glue.c +++ /dev/null @@ -1,143 +0,0 @@ -/* - * Cryptographic API. - * - * T10 Data Integrity Field CRC16 Crypto Transform using PCLMULQDQ Instructions - * - * Copyright (C) 2013 Intel Corporation - * Author: Tim Chen - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the Free - * Software Foundation; either version 2 of the License, or (at your option) - * any later version. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF - * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS - * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN - * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN - * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE - * SOFTWARE. - * - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -asmlinkage u16 crc_t10dif_pcl(u16 init_crc, const u8 *buf, size_t len); - -struct chksum_desc_ctx { - __u16 crc; -}; - -static int chksum_init(struct shash_desc *desc) -{ - struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); - - ctx->crc = 0; - - return 0; -} - -static int chksum_update(struct shash_desc *desc, const u8 *data, - unsigned int length) -{ - struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); - - if (length >= 16 && crypto_simd_usable()) { - kernel_fpu_begin(); - ctx->crc = crc_t10dif_pcl(ctx->crc, data, length); - kernel_fpu_end(); - } else - ctx->crc = crc_t10dif_generic(ctx->crc, data, length); - return 0; -} - -static int chksum_final(struct shash_desc *desc, u8 *out) -{ - struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); - - *(__u16 *)out = ctx->crc; - return 0; -} - -static int __chksum_finup(__u16 crc, const u8 *data, unsigned int len, u8 *out) -{ - if (len >= 16 && crypto_simd_usable()) { - kernel_fpu_begin(); - *(__u16 *)out = crc_t10dif_pcl(crc, data, len); - kernel_fpu_end(); - } else - *(__u16 *)out = crc_t10dif_generic(crc, data, len); - return 0; -} - -static int chksum_finup(struct shash_desc *desc, const u8 *data, - unsigned int len, u8 *out) -{ - struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); - - return __chksum_finup(ctx->crc, data, len, out); -} - -static int chksum_digest(struct shash_desc *desc, const u8 *data, - unsigned int length, u8 *out) -{ - return __chksum_finup(0, data, length, out); -} - -static struct shash_alg alg = { - .digestsize = CRC_T10DIF_DIGEST_SIZE, - .init = chksum_init, - .update = chksum_update, - .final = chksum_final, - .finup = chksum_finup, - .digest = chksum_digest, - .descsize = sizeof(struct chksum_desc_ctx), - .base = { - .cra_name = "crct10dif", - .cra_driver_name = "crct10dif-pclmul", - .cra_priority = 200, - .cra_blocksize = CRC_T10DIF_BLOCK_SIZE, - .cra_module = THIS_MODULE, - } -}; - -static const struct x86_cpu_id crct10dif_cpu_id[] = { - X86_MATCH_FEATURE(X86_FEATURE_PCLMULQDQ, NULL), - {} -}; -MODULE_DEVICE_TABLE(x86cpu, crct10dif_cpu_id); - -static int __init crct10dif_intel_mod_init(void) -{ - if (!x86_match_cpu(crct10dif_cpu_id)) - return -ENODEV; - - return crypto_register_shash(&alg); -} - -static void __exit crct10dif_intel_mod_fini(void) -{ - crypto_unregister_shash(&alg); -} - -module_init(crct10dif_intel_mod_init); -module_exit(crct10dif_intel_mod_fini); - -MODULE_AUTHOR("Tim Chen "); -MODULE_DESCRIPTION("T10 DIF CRC calculation accelerated with PCLMULQDQ."); -MODULE_LICENSE("GPL"); - -MODULE_ALIAS_CRYPTO("crct10dif"); -MODULE_ALIAS_CRYPTO("crct10dif-pclmul"); diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile index 17510da06c9f..8a59c61624c2 100644 --- a/arch/x86/lib/Makefile +++ b/arch/x86/lib/Makefile @@ -40,10 +40,13 @@ lib-$(CONFIG_MITIGATION_RETPOLINE) += retpoline.o obj-$(CONFIG_CRC32_ARCH) += crc32-x86.o crc32-x86-y := crc32-glue.o crc32-pclmul.o crc32-x86-$(CONFIG_64BIT) += crc32c-3way.o +obj-$(CONFIG_CRC_T10DIF_ARCH) += crc-t10dif-x86.o +crc-t10dif-x86-y := crc-t10dif-glue.o crct10dif-pcl-asm_64.o + obj-y += msr.o msr-reg.o msr-reg-export.o hweight.o obj-y += iomem.o ifeq ($(CONFIG_X86_32),y) obj-y += atomic64_32.o diff --git a/arch/x86/lib/crc-t10dif-glue.c b/arch/x86/lib/crc-t10dif-glue.c new file mode 100644 index 000000000000..13f07ddc9122 --- /dev/null +++ b/arch/x86/lib/crc-t10dif-glue.c @@ -0,0 +1,51 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * CRC-T10DIF using PCLMULQDQ instructions + * + * Copyright 2024 Google LLC + */ + +#include +#include +#include +#include +#include + +static DEFINE_STATIC_KEY_FALSE(have_pclmulqdq); + +asmlinkage u16 crc_t10dif_pcl(u16 init_crc, const u8 *buf, size_t len); + +u16 crc_t10dif_arch(u16 crc, const u8 *p, size_t len) +{ + if (len >= 16 && + static_key_enabled(&have_pclmulqdq) && crypto_simd_usable()) { + kernel_fpu_begin(); + crc = crc_t10dif_pcl(crc, p, len); + kernel_fpu_end(); + return crc; + } + return crc_t10dif_generic(crc, p, len); +} +EXPORT_SYMBOL(crc_t10dif_arch); + +static int __init crc_t10dif_x86_init(void) +{ + if (boot_cpu_has(X86_FEATURE_PCLMULQDQ)) + static_branch_enable(&have_pclmulqdq); + return 0; +} +arch_initcall(crc_t10dif_x86_init); + +static void __exit crc_t10dif_x86_exit(void) +{ +} +module_exit(crc_t10dif_x86_exit); + +bool crc_t10dif_is_optimized(void) +{ + return static_key_enabled(&have_pclmulqdq); +} +EXPORT_SYMBOL(crc_t10dif_is_optimized); + +MODULE_DESCRIPTION("CRC-T10DIF using PCLMULQDQ instructions"); +MODULE_LICENSE("GPL"); diff --git a/arch/x86/crypto/crct10dif-pcl-asm_64.S b/arch/x86/lib/crct10dif-pcl-asm_64.S similarity index 100% rename from arch/x86/crypto/crct10dif-pcl-asm_64.S rename to arch/x86/lib/crct10dif-pcl-asm_64.S From patchwork Sun Nov 17 00:22:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 843991 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8E0291DFDE; Sun, 17 Nov 2024 00:23:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731803036; cv=none; b=eZISOOPt/4s9GCjo9YYtIhyoYb2JubRdTDv+xOspAZq06bBEA0xNovCIpeajoYD2qm9JNl5T6arkiPkX2SprXfKG0+pFeInwm0iW2YGHOEK8LjpJ9+UAcFX5pi32RRjNEcy7+15zeyPSa0Krug08Oe1mTg9XG+ThIfXa7peeTCU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731803036; c=relaxed/simple; bh=WxUhYnP1fx0cGeoy+OGHDs01GlaiqwZIcvRWEbAyWeg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bZzDzIvOT+/u6ns3UWYqReK1y+jf3i2zllwPDgVzYoJY5ZXhs510nH1w5BcR+kyJizyQ4BtdxIaMZxq5dnISLwQeWpQJWrVI8JXksyQ83mLhoK9yq0qmpyKT/GC8yC+xwPtfvGO6jisS9uQBxrZ+PkRwKdwLCfJlh6gAwi/P3ys= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=B9o5GmY+; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="B9o5GmY+" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DAE23C4CED7; Sun, 17 Nov 2024 00:23:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731803036; bh=WxUhYnP1fx0cGeoy+OGHDs01GlaiqwZIcvRWEbAyWeg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=B9o5GmY+34Ohxwq3v/3m8iIIBd0VOiuDairRoTYO8XVbdHG/H4vucvO/JboNkFZwZ Z4HgmywYB3qspA6mvIsnPKbZugBj9qczi1Gzf6hYCbn1sGcAnuSgvh266J0BYcXx/O P47n9PP/l4Q+AXh2O7o+LPlBOIqEPG9wB1zeELKFMHwomj+MJmEIlVO2mmUA84fTpK FOWn3BTEJjm2esi7YM35N4UuM1NmzG4cuEE35ZoMkmfqW2Eu1qoRcp4trKwr4pb/2a mYfx5AXpkwt5aABotk+f/65AoIICu2N5seykhLl/g4AXvq7yhwHXPuZbbJSLpp8v1r OVKC0YrhyjCMQ== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-crypto@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, Zhihang Shao , Ard Biesheuvel Subject: [PATCH 06/11] arm64/crc-t10dif: expose CRC-T10DIF function through lib Date: Sat, 16 Nov 2024 16:22:39 -0800 Message-ID: <20241117002244.105200-7-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241117002244.105200-1-ebiggers@kernel.org> References: <20241117002244.105200-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Move the arm64 CRC-T10DIF assembly code into the lib directory and wire it up to the library interface. This allows it to be used without going through the crypto API. It remains usable via the crypto API too via the shash algorithms that use the library interface. Thus all the arch-specific "shash" code becomes unnecessary and is removed. Note: to see the diff from arch/arm64/crypto/crct10dif-ce-glue.c to arch/arm64/lib/crc-t10dif-glue.c, view this commit with 'git show -M10'. Signed-off-by: Eric Biggers --- arch/arm64/Kconfig | 1 + arch/arm64/configs/defconfig | 1 - arch/arm64/crypto/Kconfig | 10 -- arch/arm64/crypto/Makefile | 3 - arch/arm64/crypto/crct10dif-ce-glue.c | 132 ------------------ arch/arm64/lib/Makefile | 3 + .../crc-t10dif-core.S} | 0 arch/arm64/lib/crc-t10dif-glue.c | 78 +++++++++++ .../testing/selftests/arm64/fp/kernel-test.c | 3 +- 9 files changed, 83 insertions(+), 148 deletions(-) delete mode 100644 arch/arm64/crypto/crct10dif-ce-glue.c rename arch/arm64/{crypto/crct10dif-ce-core.S => lib/crc-t10dif-core.S} (100%) create mode 100644 arch/arm64/lib/crc-t10dif-glue.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 1e48f40f654e..44cdc7ccbc67 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -20,10 +20,11 @@ config ARM64 select ARCH_ENABLE_MEMORY_HOTREMOVE select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE select ARCH_HAS_CACHE_LINE_SIZE select ARCH_HAS_CRC32 + select ARCH_HAS_CRC_T10DIF if KERNEL_MODE_NEON select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEBUG_VM_PGTABLE select ARCH_HAS_DMA_OPS if XEN select ARCH_HAS_DMA_PREP_COHERENT diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig index 5fdbfea7a5b2..5688fc4ec6e9 100644 --- a/arch/arm64/configs/defconfig +++ b/arch/arm64/configs/defconfig @@ -1689,11 +1689,10 @@ CONFIG_CRYPTO_SHA512_ARM64_CE=m CONFIG_CRYPTO_SHA3_ARM64=m CONFIG_CRYPTO_SM3_ARM64_CE=m CONFIG_CRYPTO_AES_ARM64_CE_BLK=y CONFIG_CRYPTO_AES_ARM64_BS=m CONFIG_CRYPTO_AES_ARM64_CE_CCM=y -CONFIG_CRYPTO_CRCT10DIF_ARM64_CE=m CONFIG_CRYPTO_DEV_SUN8I_CE=m CONFIG_CRYPTO_DEV_FSL_CAAM=m CONFIG_CRYPTO_DEV_FSL_DPAA2_CAAM=m CONFIG_CRYPTO_DEV_QCE=m CONFIG_CRYPTO_DEV_QCOM_RNG=m diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index e7d9bd8e4709..5636ab83f22a 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -310,17 +310,7 @@ config CRYPTO_SM4_ARM64_CE_GCM Architecture: arm64 using: - ARMv8 Crypto Extensions - PMULL (Polynomial Multiply Long) instructions - NEON (Advanced SIMD) extensions -config CRYPTO_CRCT10DIF_ARM64_CE - tristate "CRCT10DIF (PMULL)" - depends on KERNEL_MODE_NEON && CRC_T10DIF - select CRYPTO_HASH - help - CRC16 CRC algorithm used for the T10 (SCSI) Data Integrity Field (DIF) - - Architecture: arm64 using - - PMULL (Polynomial Multiply Long) instructions - endmenu diff --git a/arch/arm64/crypto/Makefile b/arch/arm64/crypto/Makefile index fbe64dce66e0..e7139c4768ce 100644 --- a/arch/arm64/crypto/Makefile +++ b/arch/arm64/crypto/Makefile @@ -42,13 +42,10 @@ obj-$(CONFIG_CRYPTO_GHASH_ARM64_CE) += ghash-ce.o ghash-ce-y := ghash-ce-glue.o ghash-ce-core.o obj-$(CONFIG_CRYPTO_POLYVAL_ARM64_CE) += polyval-ce.o polyval-ce-y := polyval-ce-glue.o polyval-ce-core.o -obj-$(CONFIG_CRYPTO_CRCT10DIF_ARM64_CE) += crct10dif-ce.o -crct10dif-ce-y := crct10dif-ce-core.o crct10dif-ce-glue.o - obj-$(CONFIG_CRYPTO_AES_ARM64_CE) += aes-ce-cipher.o aes-ce-cipher-y := aes-ce-core.o aes-ce-glue.o obj-$(CONFIG_CRYPTO_AES_ARM64_CE_CCM) += aes-ce-ccm.o aes-ce-ccm-y := aes-ce-ccm-glue.o aes-ce-ccm-core.o diff --git a/arch/arm64/crypto/crct10dif-ce-glue.c b/arch/arm64/crypto/crct10dif-ce-glue.c deleted file mode 100644 index 08bcbd884395..000000000000 --- a/arch/arm64/crypto/crct10dif-ce-glue.c +++ /dev/null @@ -1,132 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* - * Accelerated CRC-T10DIF using arm64 NEON and Crypto Extensions instructions - * - * Copyright (C) 2016 - 2017 Linaro Ltd - */ - -#include -#include -#include -#include -#include -#include - -#include -#include - -#include -#include - -#define CRC_T10DIF_PMULL_CHUNK_SIZE 16U - -asmlinkage void crc_t10dif_pmull_p8(u16 init_crc, const u8 *buf, size_t len, - u8 out[16]); -asmlinkage u16 crc_t10dif_pmull_p64(u16 init_crc, const u8 *buf, size_t len); - -static int crct10dif_init(struct shash_desc *desc) -{ - u16 *crc = shash_desc_ctx(desc); - - *crc = 0; - return 0; -} - -static int crct10dif_update_pmull_p8(struct shash_desc *desc, const u8 *data, - unsigned int length) -{ - u16 *crcp = shash_desc_ctx(desc); - u16 crc = *crcp; - u8 buf[16]; - - if (length > CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) { - kernel_neon_begin(); - crc_t10dif_pmull_p8(crc, data, length, buf); - kernel_neon_end(); - - crc = 0; - data = buf; - length = sizeof(buf); - } - - *crcp = crc_t10dif_generic(crc, data, length); - return 0; -} - -static int crct10dif_update_pmull_p64(struct shash_desc *desc, const u8 *data, - unsigned int length) -{ - u16 *crc = shash_desc_ctx(desc); - - if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) { - kernel_neon_begin(); - *crc = crc_t10dif_pmull_p64(*crc, data, length); - kernel_neon_end(); - } else { - *crc = crc_t10dif_generic(*crc, data, length); - } - - return 0; -} - -static int crct10dif_final(struct shash_desc *desc, u8 *out) -{ - u16 *crc = shash_desc_ctx(desc); - - *(u16 *)out = *crc; - return 0; -} - -static struct shash_alg crc_t10dif_alg[] = {{ - .digestsize = CRC_T10DIF_DIGEST_SIZE, - .init = crct10dif_init, - .update = crct10dif_update_pmull_p8, - .final = crct10dif_final, - .descsize = CRC_T10DIF_DIGEST_SIZE, - - .base.cra_name = "crct10dif", - .base.cra_driver_name = "crct10dif-arm64-neon", - .base.cra_priority = 150, - .base.cra_blocksize = CRC_T10DIF_BLOCK_SIZE, - .base.cra_module = THIS_MODULE, -}, { - .digestsize = CRC_T10DIF_DIGEST_SIZE, - .init = crct10dif_init, - .update = crct10dif_update_pmull_p64, - .final = crct10dif_final, - .descsize = CRC_T10DIF_DIGEST_SIZE, - - .base.cra_name = "crct10dif", - .base.cra_driver_name = "crct10dif-arm64-ce", - .base.cra_priority = 200, - .base.cra_blocksize = CRC_T10DIF_BLOCK_SIZE, - .base.cra_module = THIS_MODULE, -}}; - -static int __init crc_t10dif_mod_init(void) -{ - if (cpu_have_named_feature(PMULL)) - return crypto_register_shashes(crc_t10dif_alg, - ARRAY_SIZE(crc_t10dif_alg)); - else - /* only register the first array element */ - return crypto_register_shash(crc_t10dif_alg); -} - -static void __exit crc_t10dif_mod_exit(void) -{ - if (cpu_have_named_feature(PMULL)) - crypto_unregister_shashes(crc_t10dif_alg, - ARRAY_SIZE(crc_t10dif_alg)); - else - crypto_unregister_shash(crc_t10dif_alg); -} - -module_cpu_feature_match(ASIMD, crc_t10dif_mod_init); -module_exit(crc_t10dif_mod_exit); - -MODULE_AUTHOR("Ard Biesheuvel "); -MODULE_DESCRIPTION("CRC-T10DIF using arm64 NEON and Crypto Extensions"); -MODULE_LICENSE("GPL v2"); -MODULE_ALIAS_CRYPTO("crct10dif"); -MODULE_ALIAS_CRYPTO("crct10dif-arm64-ce"); diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile index 5fbcf0d56665..4d49dff721a8 100644 --- a/arch/arm64/lib/Makefile +++ b/arch/arm64/lib/Makefile @@ -14,10 +14,13 @@ endif lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o obj-$(CONFIG_CRC32_ARCH) += crc32-arm64.o crc32-arm64-y := crc32.o crc32-glue.o +obj-$(CONFIG_CRC_T10DIF_ARCH) += crc-t10dif-arm64.o +crc-t10dif-arm64-y := crc-t10dif-glue.o crc-t10dif-core.o + obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-$(CONFIG_KASAN_SW_TAGS) += kasan_sw_tags.o diff --git a/arch/arm64/crypto/crct10dif-ce-core.S b/arch/arm64/lib/crc-t10dif-core.S similarity index 100% rename from arch/arm64/crypto/crct10dif-ce-core.S rename to arch/arm64/lib/crc-t10dif-core.S diff --git a/arch/arm64/lib/crc-t10dif-glue.c b/arch/arm64/lib/crc-t10dif-glue.c new file mode 100644 index 000000000000..99058749db67 --- /dev/null +++ b/arch/arm64/lib/crc-t10dif-glue.c @@ -0,0 +1,78 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Accelerated CRC-T10DIF using arm64 NEON and Crypto Extensions instructions + * + * Copyright (C) 2016 - 2017 Linaro Ltd + */ + +#include +#include +#include +#include +#include +#include + +#include + +#include +#include + +static DEFINE_STATIC_KEY_FALSE(have_asimd); +static DEFINE_STATIC_KEY_FALSE(have_pmull); + +#define CRC_T10DIF_PMULL_CHUNK_SIZE 16U + +asmlinkage void crc_t10dif_pmull_p8(u16 init_crc, const u8 *buf, size_t len, + u8 out[16]); +asmlinkage u16 crc_t10dif_pmull_p64(u16 init_crc, const u8 *buf, size_t len); + +u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) +{ + if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && + static_branch_likely(&have_pmull) && crypto_simd_usable()) { + kernel_neon_begin(); + crc = crc_t10dif_pmull_p64(crc, data, length); + kernel_neon_end(); + return crc; + } + if (length > CRC_T10DIF_PMULL_CHUNK_SIZE && + static_branch_likely(&have_asimd) && crypto_simd_usable()) { + u8 buf[16]; + + kernel_neon_begin(); + crc_t10dif_pmull_p8(crc, data, length, buf); + kernel_neon_end(); + + crc = 0; + data = buf; + length = sizeof(buf); + } + return crc_t10dif_generic(crc, data, length); +} +EXPORT_SYMBOL(crc_t10dif_arch); + +static int __init crc_t10dif_arm64_init(void) +{ + if (cpu_have_named_feature(ASIMD)) { + static_branch_enable(&have_asimd); + if (cpu_have_named_feature(PMULL)) + static_branch_enable(&have_pmull); + } + return 0; +} +arch_initcall(crc_t10dif_arm64_init); + +static void __exit crc_t10dif_arm64_exit(void) +{ +} +module_exit(crc_t10dif_arm64_exit); + +bool crc_t10dif_is_optimized(void) +{ + return static_key_enabled(&have_asimd); +} +EXPORT_SYMBOL(crc_t10dif_is_optimized); + +MODULE_AUTHOR("Ard Biesheuvel "); +MODULE_DESCRIPTION("CRC-T10DIF using arm64 NEON and Crypto Extensions"); +MODULE_LICENSE("GPL v2"); diff --git a/tools/testing/selftests/arm64/fp/kernel-test.c b/tools/testing/selftests/arm64/fp/kernel-test.c index e8da3b4cbd23..9d4c5cc5ae7e 100644 --- a/tools/testing/selftests/arm64/fp/kernel-test.c +++ b/tools/testing/selftests/arm64/fp/kernel-test.c @@ -44,12 +44,11 @@ static void handle_kick_signal(int sig, siginfo_t *info, void *context) { sigs++; } static char *drivers[] = { - "crct10dif-arm64-ce", - /* "crct10dif-arm64-neon", - Same priority as generic */ + "crct10dif-arm64", "sha1-ce", "sha224-arm64", "sha224-arm64-neon", "sha224-ce", "sha256-arm64", From patchwork Sun Nov 17 00:22:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 843990 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F8D93AC2B; Sun, 17 Nov 2024 00:23:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731803037; cv=none; b=su6EjzeqcJfo9AB1GB1GPqyCDsATu+HFtr2k133jya9me87VOHEa7RYZzXn4axkx0tMKKK7LbnM0PisNj8m91s5jTUEW0gwPtx8yqyYYpruFqpUiSamzAZM5wUvomA4it2XTjNsQUiUMKcJw/WX6EM3JkfIc0rBEg6eHWwrOuPM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731803037; c=relaxed/simple; bh=1JUku0y+4FR2wwQl/nc1RGyM0jOhmjqHDta/W8dcw9c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=b0YodlfcoIMo2jukaP6Noihldv+PSf7vnkEZiEGkM6fc6+fA5s0KPwBMuDINSvcfr4V1I0GErqlxVHN/D+MBmGRZe/8bVIxedz9QTqjLkYbKdM+RqteytoIPvxjNc5qZm/cwO3rnOQ/7wSQRR7sHDo+FrhV/a6GrGazwbLPdZsg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=f6ReEFx4; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="f6ReEFx4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CF58EC4CEDC; Sun, 17 Nov 2024 00:23:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731803037; bh=1JUku0y+4FR2wwQl/nc1RGyM0jOhmjqHDta/W8dcw9c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=f6ReEFx4o4Q8D77Llzf7CuHJqbEPHYTJH4FTrVHpn8aXIiGiAwgiJBN/RWaeMVbLg 2qn/OH0zrC+5+Mow0uSzPlYjDpjeMSRJcODlPCQCMAKD7WzFo82e9R+nkB59MjQSwQ 6N+R2nYpcPXgpMTpkbWEAo6FxxW2LAelErPcwYO6oJ1IRFdf62j0s62BR1yKK9iSko hfHTHqFBVThZkTR5YT750/Z76v55ZSJe5UZSOoBQ11zTMur5dmS1XGG24n2MfTTpIM +KzUYeMjUe42rx9UrzF6oXVUpQqHHAm/lJT+JF6VPhpCK/+k4f3Uatz8rSNYSrkPM8 OjSI1Cr1ixhoA== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-crypto@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, Zhihang Shao , Ard Biesheuvel Subject: [PATCH 08/11] lib/crc_kunit.c: add KUnit test suite for CRC library functions Date: Sat, 16 Nov 2024 16:22:41 -0800 Message-ID: <20241117002244.105200-9-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241117002244.105200-1-ebiggers@kernel.org> References: <20241117002244.105200-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Add a KUnit test suite for the crc16, crc_t10dif, crc32_le, crc32_be, crc32c, and crc64_be library functions. It avoids code duplication by sharing most logic among all CRC variants. The test suite includes: - Differential fuzz test of each CRC function against a simple bit-at-a-time reference implementation. - Test for CRC combination, when implemented by a CRC variant. - Optional benchmark of each CRC function with various data lengths. This is intended as a replacement for crc32test as well as a new test for CRC variants which didn't previously have a test. Signed-off-by: Eric Biggers --- lib/Kconfig.debug | 20 +++ lib/Makefile | 1 + lib/crc_kunit.c | 428 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 449 insertions(+) create mode 100644 lib/crc_kunit.c diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 7312ae7c3cc5..c5f4b5827665 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2828,10 +2828,30 @@ config HW_BREAKPOINT_KUNIT_TEST help Tests for hw_breakpoint constraints accounting. If unsure, say N. +config CRC_KUNIT_TEST + tristate "KUnit tests for CRC functions" + depends on KUNIT + default KUNIT_ALL_TESTS + select CRC16 + select CRC_T10DIF + select CRC32 + select CRC64 + help + Unit tests for the CRC library functions. + + This is intended to help people writing architecture-specific + optimized versions. If unsure, say N. + +config CRC_BENCHMARK + bool "Benchmark for the CRC functions" + depends on CRC_KUNIT_TEST + help + Include benchmarks in the KUnit test suite for the CRC functions. + config SIPHASH_KUNIT_TEST tristate "Perform selftest on siphash functions" if !KUNIT_ALL_TESTS depends on KUNIT default KUNIT_ALL_TESTS help diff --git a/lib/Makefile b/lib/Makefile index 15646679aee2..75b0d4f583ed 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -384,10 +384,11 @@ obj-$(CONFIG_STACKINIT_KUNIT_TEST) += stackinit_kunit.o CFLAGS_fortify_kunit.o += $(call cc-disable-warning, unsequenced) CFLAGS_fortify_kunit.o += $(call cc-disable-warning, stringop-overread) CFLAGS_fortify_kunit.o += $(call cc-disable-warning, stringop-truncation) CFLAGS_fortify_kunit.o += $(DISABLE_STRUCTLEAK_PLUGIN) obj-$(CONFIG_FORTIFY_KUNIT_TEST) += fortify_kunit.o +obj-$(CONFIG_CRC_KUNIT_TEST) += crc_kunit.o obj-$(CONFIG_SIPHASH_KUNIT_TEST) += siphash_kunit.o obj-$(CONFIG_USERCOPY_KUNIT_TEST) += usercopy_kunit.o obj-$(CONFIG_GENERIC_LIB_DEVMEM_IS_ALLOWED) += devmem_is_allowed.o diff --git a/lib/crc_kunit.c b/lib/crc_kunit.c new file mode 100644 index 000000000000..e0536a3f544d --- /dev/null +++ b/lib/crc_kunit.c @@ -0,0 +1,428 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Unit tests and benchmarks for the CRC library functions + * + * Copyright 2024 Google LLC + * + * Author: Eric Biggers + */ +#include +#include +#include +#include +#include +#include +#include +#include + +#define CRC_KUNIT_SEED 42 +#define CRC_KUNIT_MAX_LEN 16384 +#define CRC_KUNIT_NUM_TEST_ITERS 1000 + +static struct rnd_state rng; +static u8 *test_buffer; +static int test_buflen; + +/** + * struct crc_variant - describes a CRC variant + * @bits: Number of bits in the CRC, 1 <= @bits <= 64. + * @le: true if it's a "little endian" CRC (reversed mapping between bits and + * polynomial coefficients in each byte), false if it's a "big endian" CRC + * (natural mapping between bits and polynomial coefficients in each byte) + * @poly: The generator polynomial with the highest-order term omitted. + * Bit-reversed if @le is true. + * @func: The function to compute a CRC. The type signature uses u64 so that it + * can fit any CRC up to CRC-64. + * @combine_func: Optional function to combine two CRCs. + */ +struct crc_variant { + u64 poly; + int bits; + bool le; + u64 (*func)(u64 crc, const void *p, size_t len); + u64 (*combine_func)(u64 crc1, u64 crc2, size_t len2); +}; + +static u32 rand32(void) +{ + return prandom_u32_state(&rng); +} + +static u64 rand64(void) +{ + u32 n = rand32(); + + return ((u64)n << 32) | rand32(); +} + +static u64 crc_mask(const struct crc_variant *v) +{ + return (u64)-1 >> (64 - v->bits); +} + +/* Reference implementation of any CRC variant */ +static u64 crc_ref(const struct crc_variant *v, + u64 crc, const u8 *p, size_t len) +{ + size_t i, j; + + for (i = 0; i < len; i++) { + for (j = 0; j < 8; j++) { + if (v->le) { + crc ^= (p[i] >> j) & 1; + crc = (crc >> 1) ^ ((crc & 1) ? v->poly : 0); + } else { + crc ^= (u64)((p[i] >> (7 - j)) & 1) << + (v->bits - 1); + if (crc & (1ULL << (v->bits - 1))) + crc = ((crc << 1) ^ v->poly) & + crc_mask(v); + else + crc <<= 1; + } + } + } + return crc; +} + +static int crc_suite_init(struct kunit_suite *suite) +{ + /* + * Allocate the test buffer using vmalloc() with a page-aligned length + * so that it is immediately followed by a guard page. This allows + * buffer overreads to be detected, even in assembly code. + */ + test_buflen = round_up(CRC_KUNIT_MAX_LEN, PAGE_SIZE); + test_buffer = vmalloc(test_buflen); + if (!test_buffer) + return -ENOMEM; + + prandom_seed_state(&rng, CRC_KUNIT_SEED); + prandom_bytes_state(&rng, test_buffer, test_buflen); + return 0; +} + +/* Generate a random initial CRC. */ +static u64 generate_random_initial_crc(const struct crc_variant *v) +{ + switch (rand32() % 4) { + case 0: + return 0; + case 1: + return crc_mask(v); /* All 1 bits */ + default: + return rand64() & crc_mask(v); + } +} + +/* Generate a random length, preferring small lengths. */ +static size_t generate_random_length(size_t max_length) +{ + size_t len; + + switch (rand32() % 3) { + case 0: + len = rand32() % 128; + break; + case 1: + len = rand32() % 3072; + break; + default: + len = rand32(); + break; + } + return len % (max_length + 1); +} + +static void crc_main_test(struct kunit *test, const struct crc_variant *v) +{ + size_t i; + + for (i = 0; i < CRC_KUNIT_NUM_TEST_ITERS; i++) { + u64 init_crc, expected_crc, actual_crc; + size_t len, offset; + bool nosimd; + + init_crc = generate_random_initial_crc(v); + len = generate_random_length(CRC_KUNIT_MAX_LEN); + + /* Generate a random offset. */ + if (rand32() & 1) { + /* Use a random alignment mod 64 */ + offset = rand32() % 64; + offset = min(offset, CRC_KUNIT_MAX_LEN - len); + } else { + /* Go up to the guard page, to catch buffer overreads */ + offset = test_buflen - len; + } + + if (rand32() % 8 == 0) + /* Refresh the data occasionally. */ + prandom_bytes_state(&rng, &test_buffer[offset], len); + + nosimd = rand32() % 8 == 0; + + /* + * Compute the CRC, and verify that it equals the CRC computed + * by a simple bit-at-a-time reference implementation. + */ + expected_crc = crc_ref(v, init_crc, &test_buffer[offset], len); + if (nosimd) + local_irq_disable(); + actual_crc = v->func(init_crc, &test_buffer[offset], len); + if (nosimd) + local_irq_enable(); + KUNIT_EXPECT_EQ_MSG(test, expected_crc, actual_crc, + "Wrong result with len=%zu offset=%zu nosimd=%d", + len, offset, nosimd); + } +} + +static void crc_combine_test(struct kunit *test, const struct crc_variant *v) +{ + int i; + + for (i = 0; i < 100; i++) { + u64 init_crc = generate_random_initial_crc(v); + size_t len1 = generate_random_length(CRC_KUNIT_MAX_LEN); + size_t len2 = generate_random_length(CRC_KUNIT_MAX_LEN - len1); + u64 crc1, crc2, expected_crc, actual_crc; + + prandom_bytes_state(&rng, test_buffer, len1 + len2); + crc1 = v->func(init_crc, test_buffer, len1); + crc2 = v->func(0, &test_buffer[len1], len2); + expected_crc = v->func(init_crc, test_buffer, len1 + len2); + actual_crc = v->combine_func(crc1, crc2, len2); + KUNIT_EXPECT_EQ_MSG(test, expected_crc, actual_crc, + "CRC combination gave wrong result with len1=%zu len2=%zu\n", + len1, len2); + } +} + +static void crc_test(struct kunit *test, const struct crc_variant *v) +{ + crc_main_test(test, v); + if (v->combine_func) + crc_combine_test(test, v); +} + +static __always_inline void +crc_benchmark(struct kunit *test, + u64 (*crc_func)(u64 crc, const void *p, size_t len)) +{ + static const size_t lens_to_test[] = { + 1, 16, 64, 127, 128, 200, 256, 511, 512, 1024, 3173, 4096, 16384, + }; + size_t len, i, j, num_iters; + /* + * Some of the CRC library functions are marked as __pure, so use + * volatile to ensure that all calls are really made as intended. + */ + volatile u64 crc = 0; + u64 t; + + if (!IS_ENABLED(CONFIG_CRC_BENCHMARK)) + kunit_skip(test, "not enabled"); + + preempt_disable(); + + /* warm-up */ + for (i = 0; i < 10000000; i += CRC_KUNIT_MAX_LEN) + crc = crc_func(crc, test_buffer, CRC_KUNIT_MAX_LEN); + + for (i = 0; i < ARRAY_SIZE(lens_to_test); i++) { + len = lens_to_test[i]; + KUNIT_ASSERT_LE(test, len, CRC_KUNIT_MAX_LEN); + num_iters = 10000000 / (len + 128); + t = ktime_get_ns(); + for (j = 0; j < num_iters; j++) + crc = crc_func(crc, test_buffer, len); + t = ktime_get_ns() - t; + kunit_info(test, "len=%zu: %llu MB/s\n", + len, div64_u64((u64)len * num_iters * 1000, t)); + } + + preempt_enable(); +} + +/* crc16 */ + +static u64 crc16_wrapper(u64 crc, const void *p, size_t len) +{ + return crc16(crc, p, len); +} + +static const struct crc_variant crc_variant_crc16 = { + .bits = 16, + .le = true, + .poly = 0xa001, + .func = crc16_wrapper, +}; + +static void crc16_test(struct kunit *test) +{ + crc_test(test, &crc_variant_crc16); +} + +static void crc16_benchmark(struct kunit *test) +{ + crc_benchmark(test, crc16_wrapper); +} + +/* crc_t10dif */ + +static u64 crc_t10dif_wrapper(u64 crc, const void *p, size_t len) +{ + return crc_t10dif_update(crc, p, len); +} + +static const struct crc_variant crc_variant_crc_t10dif = { + .bits = 16, + .le = false, + .poly = 0x8bb7, + .func = crc_t10dif_wrapper, +}; + +static void crc_t10dif_test(struct kunit *test) +{ + crc_test(test, &crc_variant_crc_t10dif); +} + +static void crc_t10dif_benchmark(struct kunit *test) +{ + crc_benchmark(test, crc_t10dif_wrapper); +} + +/* crc32_le */ + +static u64 crc32_le_wrapper(u64 crc, const void *p, size_t len) +{ + return crc32_le(crc, p, len); +} + +static u64 crc32_le_combine_wrapper(u64 crc1, u64 crc2, size_t len2) +{ + return crc32_le_combine(crc1, crc2, len2); +} + +static const struct crc_variant crc_variant_crc32_le = { + .bits = 32, + .le = true, + .poly = 0xedb88320, + .func = crc32_le_wrapper, + .combine_func = crc32_le_combine_wrapper, +}; + +static void crc32_le_test(struct kunit *test) +{ + crc_test(test, &crc_variant_crc32_le); +} + +static void crc32_le_benchmark(struct kunit *test) +{ + crc_benchmark(test, crc32_le_wrapper); +} + +/* crc32_be */ + +static u64 crc32_be_wrapper(u64 crc, const void *p, size_t len) +{ + return crc32_be(crc, p, len); +} + +static const struct crc_variant crc_variant_crc32_be = { + .bits = 32, + .le = false, + .poly = 0x04c11db7, + .func = crc32_be_wrapper, +}; + +static void crc32_be_test(struct kunit *test) +{ + crc_test(test, &crc_variant_crc32_be); +} + +static void crc32_be_benchmark(struct kunit *test) +{ + crc_benchmark(test, crc32_be_wrapper); +} + +/* crc32c */ + +static u64 crc32c_wrapper(u64 crc, const void *p, size_t len) +{ + return crc32c(crc, p, len); +} + +static u64 crc32c_combine_wrapper(u64 crc1, u64 crc2, size_t len2) +{ + return __crc32c_le_combine(crc1, crc2, len2); +} + +static const struct crc_variant crc_variant_crc32c = { + .bits = 32, + .le = true, + .poly = 0x82f63b78, + .func = crc32c_wrapper, + .combine_func = crc32c_combine_wrapper, +}; + +static void crc32c_test(struct kunit *test) +{ + crc_test(test, &crc_variant_crc32c); +} + +static void crc32c_benchmark(struct kunit *test) +{ + crc_benchmark(test, crc32c_wrapper); +} + +/* crc64_be */ + +static u64 crc64_be_wrapper(u64 crc, const void *p, size_t len) +{ + return crc64_be(crc, p, len); +} + +static const struct crc_variant crc_variant_crc64_be = { + .bits = 64, + .le = false, + .poly = 0x42f0e1eba9ea3693, + .func = crc64_be_wrapper, +}; + +static void crc64_be_test(struct kunit *test) +{ + crc_test(test, &crc_variant_crc64_be); +} + +static void crc64_be_benchmark(struct kunit *test) +{ + crc_benchmark(test, crc64_be_wrapper); +} + +static struct kunit_case crc_test_cases[] = { + KUNIT_CASE(crc16_test), + KUNIT_CASE(crc16_benchmark), + KUNIT_CASE(crc_t10dif_test), + KUNIT_CASE(crc_t10dif_benchmark), + KUNIT_CASE(crc32_le_test), + KUNIT_CASE(crc32_le_benchmark), + KUNIT_CASE(crc32_be_test), + KUNIT_CASE(crc32_be_benchmark), + KUNIT_CASE(crc32c_test), + KUNIT_CASE(crc32c_benchmark), + KUNIT_CASE(crc64_be_test), + KUNIT_CASE(crc64_be_benchmark), + {}, +}; + +static struct kunit_suite crc_test_suite = { + .name = "crc", + .test_cases = crc_test_cases, + .suite_init = crc_suite_init, +}; +kunit_test_suite(crc_test_suite); + +MODULE_DESCRIPTION("Unit tests and benchmarks for the CRC library functions"); +MODULE_LICENSE("GPL"); From patchwork Sun Nov 17 00:22:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 843989 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 39DCB83A17; Sun, 17 Nov 2024 00:23:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731803038; cv=none; b=ifoonNMVlGvfj4aTPN4bDQ2ULyuTC59iOKzn5cNwi/ISnBdo8MDOBPVtu/bZM3Lw/Oe8HX9QwJXx6j72FJkuT4+WVOASL1BGQGtL5lmGx1LwU0aLufKWHtIcm3d4pbSaGVsysmiDnTq8XMeNdwDprQM2w91UffKWI9EHwEYMFdU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731803038; c=relaxed/simple; bh=L8iJ9JStDTqpmtdnZloUKPKtnPySgcuceskFsXeoeLQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QP2+WHLLjL3BRP69MVrLXWwWdklwj9q2cn5ehLme6YD9G14Bnlt/5AjbBYuWGdUndLVtFd6aUUuYebp9dPhFObxE9lGcUabbE5C2zlmVeW0bJDWEBenVWGwgl0mUStsNBHsx6qSREzOWuZW8KvxwUi2mZ0qy1OJX10UKC+un7f4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KEihOEeZ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KEihOEeZ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D37CCC4CEC3; Sun, 17 Nov 2024 00:23:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731803038; bh=L8iJ9JStDTqpmtdnZloUKPKtnPySgcuceskFsXeoeLQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KEihOEeZ9pps1Gmhd03DWXm2C9+PD1QlRllHUMri7/q2NQvuv0/RFaVQWNRbEW8oW BtUyHKZRQZi1Tz2zk1m+P1LsHGo9Hv1HtqBFLY974f/LLQt7LPGuUwlFeQrWL3UBed KxIf0qSTOsaup8AqhiXce7laYg2kjw4/wYEDCbfAv9UfzR3lx8slmqYw/THIALTrjO akw/eb3eXF18JfGYvONTeKAQJfP7V50dgIo1sa0UQ294tqk3vH/DRDN+I8oDNku1Hx FTkU7VkZkR6TdhFCuDMLz7ZDDT3BatSTPmOZWO/yq4jRu0FYj9IUG7TH7YJwlj31Uz XzKKBvHnTL8bw== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-crypto@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, Zhihang Shao , Ard Biesheuvel Subject: [PATCH 10/11] powerpc/crc: delete obsolete crc-vpmsum_test.c Date: Sat, 16 Nov 2024 16:22:43 -0800 Message-ID: <20241117002244.105200-11-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241117002244.105200-1-ebiggers@kernel.org> References: <20241117002244.105200-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Delete crc-vpmsum_test.c, since its functionality is now covered by the new crc_kunit.c as well as the crypto subsystem's fuzz tests. Signed-off-by: Eric Biggers --- arch/powerpc/configs/ppc64_defconfig | 1 - arch/powerpc/crypto/Kconfig | 9 -- arch/powerpc/crypto/Makefile | 1 - arch/powerpc/crypto/crc-vpmsum_test.c | 133 -------------------------- 4 files changed, 144 deletions(-) delete mode 100644 arch/powerpc/crypto/crc-vpmsum_test.c diff --git a/arch/powerpc/configs/ppc64_defconfig b/arch/powerpc/configs/ppc64_defconfig index ec91e0104713..c6445c2b3131 100644 --- a/arch/powerpc/configs/ppc64_defconfig +++ b/arch/powerpc/configs/ppc64_defconfig @@ -388,11 +388,10 @@ CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_PCBC=m CONFIG_CRYPTO_MICHAEL_MIC=m CONFIG_CRYPTO_SHA256=y CONFIG_CRYPTO_WP512=m CONFIG_CRYPTO_LZO=m -CONFIG_CRYPTO_VPMSUM_TESTER=m CONFIG_CRYPTO_MD5_PPC=m CONFIG_CRYPTO_SHA1_PPC=m CONFIG_CRYPTO_AES_GCM_P10=m CONFIG_CRYPTO_DEV_NX=y CONFIG_CRYPTO_DEV_NX_ENCRYPT=m diff --git a/arch/powerpc/crypto/Kconfig b/arch/powerpc/crypto/Kconfig index 90ac79513923..e48d23fb7ea5 100644 --- a/arch/powerpc/crypto/Kconfig +++ b/arch/powerpc/crypto/Kconfig @@ -11,19 +11,10 @@ config CRYPTO_CURVE25519_PPC64 Curve25519 algorithm Architecture: PowerPC64 - Little-endian -config CRYPTO_VPMSUM_TESTER - tristate "CRC32c and CRC32T10DIF hardware acceleration tester" - depends on CRYPTO_CRC32C && CRC32_ARCH - depends on CRYPTO_CRCT10DIF && CRC_T10DIF_ARCH - help - Stress test for CRC32c and CRCT10DIF algorithms implemented with - powerpc64 AltiVec extensions (POWER8 vpmsum instructions). - Unless you are testing these algorithms, you don't need this. - config CRYPTO_MD5_PPC tristate "Digests: MD5" depends on PPC select CRYPTO_HASH help diff --git a/arch/powerpc/crypto/Makefile b/arch/powerpc/crypto/Makefile index d2238ac7e52c..9b38f4a7bc15 100644 --- a/arch/powerpc/crypto/Makefile +++ b/arch/powerpc/crypto/Makefile @@ -8,11 +8,10 @@ obj-$(CONFIG_CRYPTO_AES_PPC_SPE) += aes-ppc-spe.o obj-$(CONFIG_CRYPTO_MD5_PPC) += md5-ppc.o obj-$(CONFIG_CRYPTO_SHA1_PPC) += sha1-powerpc.o obj-$(CONFIG_CRYPTO_SHA1_PPC_SPE) += sha1-ppc-spe.o obj-$(CONFIG_CRYPTO_SHA256_PPC_SPE) += sha256-ppc-spe.o -obj-$(CONFIG_CRYPTO_VPMSUM_TESTER) += crc-vpmsum_test.o obj-$(CONFIG_CRYPTO_AES_GCM_P10) += aes-gcm-p10-crypto.o obj-$(CONFIG_CRYPTO_CHACHA20_P10) += chacha-p10-crypto.o obj-$(CONFIG_CRYPTO_POLY1305_P10) += poly1305-p10-crypto.o obj-$(CONFIG_CRYPTO_DEV_VMX_ENCRYPT) += vmx-crypto.o obj-$(CONFIG_CRYPTO_CURVE25519_PPC64) += curve25519-ppc64le.o diff --git a/arch/powerpc/crypto/crc-vpmsum_test.c b/arch/powerpc/crypto/crc-vpmsum_test.c deleted file mode 100644 index c61a874a3a5c..000000000000 --- a/arch/powerpc/crypto/crc-vpmsum_test.c +++ /dev/null @@ -1,133 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* - * CRC vpmsum tester - * Copyright 2017 Daniel Axtens, IBM Corporation. - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -static unsigned long iterations = 10000; - -#define MAX_CRC_LENGTH 65535 - - -static int __init crc_test_init(void) -{ - u16 crc16 = 0, verify16 = 0; - __le32 verify32le = 0; - unsigned char *data; - u32 verify32 = 0; - unsigned long i; - __le32 crc32; - int ret; - - struct crypto_shash *crct10dif_tfm; - struct crypto_shash *crc32c_tfm; - - if (!cpu_has_feature(CPU_FTR_ARCH_207S)) - return -ENODEV; - - data = kmalloc(MAX_CRC_LENGTH, GFP_KERNEL); - if (!data) - return -ENOMEM; - - crct10dif_tfm = crypto_alloc_shash("crct10dif", 0, 0); - - if (IS_ERR(crct10dif_tfm)) { - pr_err("Error allocating crc-t10dif\n"); - goto free_buf; - } - - crc32c_tfm = crypto_alloc_shash("crc32c", 0, 0); - - if (IS_ERR(crc32c_tfm)) { - pr_err("Error allocating crc32c\n"); - goto free_16; - } - - do { - SHASH_DESC_ON_STACK(crct10dif_shash, crct10dif_tfm); - SHASH_DESC_ON_STACK(crc32c_shash, crc32c_tfm); - - crct10dif_shash->tfm = crct10dif_tfm; - ret = crypto_shash_init(crct10dif_shash); - - if (ret) { - pr_err("Error initing crc-t10dif\n"); - goto free_32; - } - - - crc32c_shash->tfm = crc32c_tfm; - ret = crypto_shash_init(crc32c_shash); - - if (ret) { - pr_err("Error initing crc32c\n"); - goto free_32; - } - - pr_info("crc-vpmsum_test begins, %lu iterations\n", iterations); - for (i=0; i"); -MODULE_DESCRIPTION("Vector polynomial multiply-sum CRC tester"); -MODULE_LICENSE("GPL");