From patchwork Mon Jan 11 16:52:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 360625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49C12C433E6 for ; Mon, 11 Jan 2021 16:53:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F1CAD2250F for ; Mon, 11 Jan 2021 16:53:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389041AbhAKQxf (ORCPT ); Mon, 11 Jan 2021 11:53:35 -0500 Received: from mail.kernel.org ([198.145.29.99]:46592 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389086AbhAKQxe (ORCPT ); Mon, 11 Jan 2021 11:53:34 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id EC2D8229CA; Mon, 11 Jan 2021 16:52:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1610383973; bh=dbwT85pGXyl/xe9gDylWrwUYY28BJ2J4ql/h9Mis97Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YbRR07m9AY8S8X8N3DC1VKJU5UuEm+kkeJdU5hAWXxyUyvKlamsAbwWex2gzdjgFX nkHBHjIvr6nbW0BRm+TQ0qNFmVSU7rgAex52KrxR3+CSTdh3OtR9qI3PaJIQSiSjom AcXh5VqmoVVNfci4fxrFlP+AKfbgmBmim3Un4pcWO3CpDWT+tn58UKFnzoi3jkSrvY caOMHsMwmVeco4Z0eslLKisHfV/BXsBZo3cO54xwovZ4/frQQKdk4r5cHFgGqnLQBr JnlO+AdfHn3/1mcpMD8LMCXaTyUHUqWBKk6fy5L9B+QerpuLRRbp09iK/ip2c6gnOK r43n5PCBHqqMA== From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, Ard Biesheuvel , "Martin K. Petersen" , Eric Biggers , Peter Zijlstra Subject: [PATCH 1/7] crypto: crc-t10dif - turn library wrapper for shash into generic library Date: Mon, 11 Jan 2021 17:52:31 +0100 Message-Id: <20210111165237.18178-2-ardb@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210111165237.18178-1-ardb@kernel.org> References: <20210111165237.18178-1-ardb@kernel.org> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org As a first step towards moving CRC-T10DIF out of the crypto API, drop the shash wrapping code from the library implementation, and move the generic core CRC-T10DIF into it from the generic crypto SHASH driver. In a future patch, the library interface will be augmented with the facilities to register and unregister optimized implementations. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/Kconfig | 2 +- arch/arm64/crypto/Kconfig | 3 +- crypto/Kconfig | 7 +- crypto/Makefile | 2 +- crypto/crct10dif_common.c | 82 ------------ lib/Kconfig | 2 - lib/crc-t10dif.c | 136 +++++++------------- 7 files changed, 57 insertions(+), 177 deletions(-) diff --git a/arch/arm/crypto/Kconfig b/arch/arm/crypto/Kconfig index 2b575792363e..939de9ceed0f 100644 --- a/arch/arm/crypto/Kconfig +++ b/arch/arm/crypto/Kconfig @@ -138,8 +138,8 @@ config CRYPTO_GHASH_ARM_CE config CRYPTO_CRCT10DIF_ARM_CE tristate "CRCT10DIF digest algorithm using PMULL instructions" depends on KERNEL_MODE_NEON - depends on CRC_T10DIF select CRYPTO_HASH + select CRC_T10DIF config CRYPTO_CRC32_ARM_CE tristate "CRC32(C) digest algorithm using CRC and/or PMULL instructions" diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index b8eb0453123d..e5d2f989521f 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -62,8 +62,9 @@ config CRYPTO_GHASH_ARM64_CE config CRYPTO_CRCT10DIF_ARM64_CE tristate "CRCT10DIF digest algorithm using PMULL instructions" - depends on KERNEL_MODE_NEON && CRC_T10DIF + depends on KERNEL_MODE_NEON select CRYPTO_HASH + select CRC_T10DIF config CRYPTO_AES_ARM64 tristate "AES core cipher using scalar instructions" diff --git a/crypto/Kconfig b/crypto/Kconfig index 94f0fde06b94..3e8cf6c2215a 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -700,6 +700,7 @@ config CRYPTO_BLAKE2S_X86 config CRYPTO_CRCT10DIF tristate "CRCT10DIF algorithm" select CRYPTO_HASH + select CRC_T10DIF help CRC T10 Data Integrity Field computation is being cast as a crypto transform. This allows for faster crc t10 diff @@ -707,8 +708,9 @@ config CRYPTO_CRCT10DIF config CRYPTO_CRCT10DIF_PCLMUL tristate "CRCT10DIF PCLMULQDQ hardware acceleration" - depends on X86 && 64BIT && CRC_T10DIF + depends on X86 && 64BIT select CRYPTO_HASH + select CRC_T10DIF help For x86_64 processors with SSE4.2 and PCLMULQDQ supported, CRC T10 DIF PCLMULQDQ computation can be hardware @@ -718,8 +720,9 @@ config CRYPTO_CRCT10DIF_PCLMUL config CRYPTO_CRCT10DIF_VPMSUM tristate "CRC32T10DIF powerpc64 hardware acceleration" - depends on PPC64 && ALTIVEC && CRC_T10DIF + depends on PPC64 && ALTIVEC select CRYPTO_HASH + select CRC_T10DIF help CRC10T10DIF algorithm implemented using vector polynomial multiply-sum (vpmsum) instructions, introduced in POWER8. Enable on diff --git a/crypto/Makefile b/crypto/Makefile index b279483fba50..69d06a28c8e6 100644 --- a/crypto/Makefile +++ b/crypto/Makefile @@ -148,7 +148,7 @@ obj-$(CONFIG_CRYPTO_DEFLATE) += deflate.o obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += michael_mic.o obj-$(CONFIG_CRYPTO_CRC32C) += crc32c_generic.o obj-$(CONFIG_CRYPTO_CRC32) += crc32_generic.o -obj-$(CONFIG_CRYPTO_CRCT10DIF) += crct10dif_common.o crct10dif_generic.o +obj-$(CONFIG_CRYPTO_CRCT10DIF) += crct10dif_generic.o obj-$(CONFIG_CRYPTO_AUTHENC) += authenc.o authencesn.o obj-$(CONFIG_CRYPTO_LZO) += lzo.o lzo-rle.o obj-$(CONFIG_CRYPTO_LZ4) += lz4.o diff --git a/crypto/crct10dif_common.c b/crypto/crct10dif_common.c deleted file mode 100644 index b2fab366f518..000000000000 --- a/crypto/crct10dif_common.c +++ /dev/null @@ -1,82 +0,0 @@ -/* - * Cryptographic API. - * - * T10 Data Integrity Field CRC16 Crypto Transform - * - * Copyright (c) 2007 Oracle Corporation. All rights reserved. - * Written by Martin K. Petersen - * Copyright (C) 2013 Intel Corporation - * Author: Tim Chen - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the Free - * Software Foundation; either version 2 of the License, or (at your option) - * any later version. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF - * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS - * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN - * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN - * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE - * SOFTWARE. - * - */ - -#include -#include -#include - -/* Table generated using the following polynomium: - * x^16 + x^15 + x^11 + x^9 + x^8 + x^7 + x^5 + x^4 + x^2 + x + 1 - * gt: 0x8bb7 - */ -static const __u16 t10_dif_crc_table[256] = { - 0x0000, 0x8BB7, 0x9CD9, 0x176E, 0xB205, 0x39B2, 0x2EDC, 0xA56B, - 0xEFBD, 0x640A, 0x7364, 0xF8D3, 0x5DB8, 0xD60F, 0xC161, 0x4AD6, - 0x54CD, 0xDF7A, 0xC814, 0x43A3, 0xE6C8, 0x6D7F, 0x7A11, 0xF1A6, - 0xBB70, 0x30C7, 0x27A9, 0xAC1E, 0x0975, 0x82C2, 0x95AC, 0x1E1B, - 0xA99A, 0x222D, 0x3543, 0xBEF4, 0x1B9F, 0x9028, 0x8746, 0x0CF1, - 0x4627, 0xCD90, 0xDAFE, 0x5149, 0xF422, 0x7F95, 0x68FB, 0xE34C, - 0xFD57, 0x76E0, 0x618E, 0xEA39, 0x4F52, 0xC4E5, 0xD38B, 0x583C, - 0x12EA, 0x995D, 0x8E33, 0x0584, 0xA0EF, 0x2B58, 0x3C36, 0xB781, - 0xD883, 0x5334, 0x445A, 0xCFED, 0x6A86, 0xE131, 0xF65F, 0x7DE8, - 0x373E, 0xBC89, 0xABE7, 0x2050, 0x853B, 0x0E8C, 0x19E2, 0x9255, - 0x8C4E, 0x07F9, 0x1097, 0x9B20, 0x3E4B, 0xB5FC, 0xA292, 0x2925, - 0x63F3, 0xE844, 0xFF2A, 0x749D, 0xD1F6, 0x5A41, 0x4D2F, 0xC698, - 0x7119, 0xFAAE, 0xEDC0, 0x6677, 0xC31C, 0x48AB, 0x5FC5, 0xD472, - 0x9EA4, 0x1513, 0x027D, 0x89CA, 0x2CA1, 0xA716, 0xB078, 0x3BCF, - 0x25D4, 0xAE63, 0xB90D, 0x32BA, 0x97D1, 0x1C66, 0x0B08, 0x80BF, - 0xCA69, 0x41DE, 0x56B0, 0xDD07, 0x786C, 0xF3DB, 0xE4B5, 0x6F02, - 0x3AB1, 0xB106, 0xA668, 0x2DDF, 0x88B4, 0x0303, 0x146D, 0x9FDA, - 0xD50C, 0x5EBB, 0x49D5, 0xC262, 0x6709, 0xECBE, 0xFBD0, 0x7067, - 0x6E7C, 0xE5CB, 0xF2A5, 0x7912, 0xDC79, 0x57CE, 0x40A0, 0xCB17, - 0x81C1, 0x0A76, 0x1D18, 0x96AF, 0x33C4, 0xB873, 0xAF1D, 0x24AA, - 0x932B, 0x189C, 0x0FF2, 0x8445, 0x212E, 0xAA99, 0xBDF7, 0x3640, - 0x7C96, 0xF721, 0xE04F, 0x6BF8, 0xCE93, 0x4524, 0x524A, 0xD9FD, - 0xC7E6, 0x4C51, 0x5B3F, 0xD088, 0x75E3, 0xFE54, 0xE93A, 0x628D, - 0x285B, 0xA3EC, 0xB482, 0x3F35, 0x9A5E, 0x11E9, 0x0687, 0x8D30, - 0xE232, 0x6985, 0x7EEB, 0xF55C, 0x5037, 0xDB80, 0xCCEE, 0x4759, - 0x0D8F, 0x8638, 0x9156, 0x1AE1, 0xBF8A, 0x343D, 0x2353, 0xA8E4, - 0xB6FF, 0x3D48, 0x2A26, 0xA191, 0x04FA, 0x8F4D, 0x9823, 0x1394, - 0x5942, 0xD2F5, 0xC59B, 0x4E2C, 0xEB47, 0x60F0, 0x779E, 0xFC29, - 0x4BA8, 0xC01F, 0xD771, 0x5CC6, 0xF9AD, 0x721A, 0x6574, 0xEEC3, - 0xA415, 0x2FA2, 0x38CC, 0xB37B, 0x1610, 0x9DA7, 0x8AC9, 0x017E, - 0x1F65, 0x94D2, 0x83BC, 0x080B, 0xAD60, 0x26D7, 0x31B9, 0xBA0E, - 0xF0D8, 0x7B6F, 0x6C01, 0xE7B6, 0x42DD, 0xC96A, 0xDE04, 0x55B3 -}; - -__u16 crc_t10dif_generic(__u16 crc, const unsigned char *buffer, size_t len) -{ - unsigned int i; - - for (i = 0 ; i < len ; i++) - crc = (crc << 8) ^ t10_dif_crc_table[((crc >> 8) ^ buffer[i]) & 0xff]; - - return crc; -} -EXPORT_SYMBOL(crc_t10dif_generic); - -MODULE_DESCRIPTION("T10 DIF CRC calculation common code"); -MODULE_LICENSE("GPL"); diff --git a/lib/Kconfig b/lib/Kconfig index 46806332a8cc..203a50674602 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -120,8 +120,6 @@ config CRC16 config CRC_T10DIF tristate "CRC calculation for the T10 Data Integrity Field" - select CRYPTO - select CRYPTO_CRCT10DIF help This option is only needed if a module that's not in the kernel tree needs to calculate CRC checks for use with the diff --git a/lib/crc-t10dif.c b/lib/crc-t10dif.c index 1ed2ed487097..cbb739f0a0d7 100644 --- a/lib/crc-t10dif.c +++ b/lib/crc-t10dif.c @@ -4,6 +4,8 @@ * * Copyright (c) 2007 Oracle Corporation. All rights reserved. * Written by Martin K. Petersen + * Copyright (C) 2013 Intel Corporation + * Author: Tim Chen */ #include @@ -16,70 +18,59 @@ #include #include -static struct crypto_shash __rcu *crct10dif_tfm; -static DEFINE_STATIC_KEY_TRUE(crct10dif_fallback); -static DEFINE_MUTEX(crc_t10dif_mutex); -static struct work_struct crct10dif_rehash_work; - -static int crc_t10dif_notify(struct notifier_block *self, unsigned long val, void *data) -{ - struct crypto_alg *alg = data; - - if (val != CRYPTO_MSG_ALG_LOADED || - strcmp(alg->cra_name, CRC_T10DIF_STRING)) - return NOTIFY_DONE; - - schedule_work(&crct10dif_rehash_work); - return NOTIFY_OK; -} +/* Table generated using the following polynomium: + * x^16 + x^15 + x^11 + x^9 + x^8 + x^7 + x^5 + x^4 + x^2 + x + 1 + * gt: 0x8bb7 + */ +static const __u16 __cacheline_aligned t10_dif_crc_table[256] = { + 0x0000, 0x8BB7, 0x9CD9, 0x176E, 0xB205, 0x39B2, 0x2EDC, 0xA56B, + 0xEFBD, 0x640A, 0x7364, 0xF8D3, 0x5DB8, 0xD60F, 0xC161, 0x4AD6, + 0x54CD, 0xDF7A, 0xC814, 0x43A3, 0xE6C8, 0x6D7F, 0x7A11, 0xF1A6, + 0xBB70, 0x30C7, 0x27A9, 0xAC1E, 0x0975, 0x82C2, 0x95AC, 0x1E1B, + 0xA99A, 0x222D, 0x3543, 0xBEF4, 0x1B9F, 0x9028, 0x8746, 0x0CF1, + 0x4627, 0xCD90, 0xDAFE, 0x5149, 0xF422, 0x7F95, 0x68FB, 0xE34C, + 0xFD57, 0x76E0, 0x618E, 0xEA39, 0x4F52, 0xC4E5, 0xD38B, 0x583C, + 0x12EA, 0x995D, 0x8E33, 0x0584, 0xA0EF, 0x2B58, 0x3C36, 0xB781, + 0xD883, 0x5334, 0x445A, 0xCFED, 0x6A86, 0xE131, 0xF65F, 0x7DE8, + 0x373E, 0xBC89, 0xABE7, 0x2050, 0x853B, 0x0E8C, 0x19E2, 0x9255, + 0x8C4E, 0x07F9, 0x1097, 0x9B20, 0x3E4B, 0xB5FC, 0xA292, 0x2925, + 0x63F3, 0xE844, 0xFF2A, 0x749D, 0xD1F6, 0x5A41, 0x4D2F, 0xC698, + 0x7119, 0xFAAE, 0xEDC0, 0x6677, 0xC31C, 0x48AB, 0x5FC5, 0xD472, + 0x9EA4, 0x1513, 0x027D, 0x89CA, 0x2CA1, 0xA716, 0xB078, 0x3BCF, + 0x25D4, 0xAE63, 0xB90D, 0x32BA, 0x97D1, 0x1C66, 0x0B08, 0x80BF, + 0xCA69, 0x41DE, 0x56B0, 0xDD07, 0x786C, 0xF3DB, 0xE4B5, 0x6F02, + 0x3AB1, 0xB106, 0xA668, 0x2DDF, 0x88B4, 0x0303, 0x146D, 0x9FDA, + 0xD50C, 0x5EBB, 0x49D5, 0xC262, 0x6709, 0xECBE, 0xFBD0, 0x7067, + 0x6E7C, 0xE5CB, 0xF2A5, 0x7912, 0xDC79, 0x57CE, 0x40A0, 0xCB17, + 0x81C1, 0x0A76, 0x1D18, 0x96AF, 0x33C4, 0xB873, 0xAF1D, 0x24AA, + 0x932B, 0x189C, 0x0FF2, 0x8445, 0x212E, 0xAA99, 0xBDF7, 0x3640, + 0x7C96, 0xF721, 0xE04F, 0x6BF8, 0xCE93, 0x4524, 0x524A, 0xD9FD, + 0xC7E6, 0x4C51, 0x5B3F, 0xD088, 0x75E3, 0xFE54, 0xE93A, 0x628D, + 0x285B, 0xA3EC, 0xB482, 0x3F35, 0x9A5E, 0x11E9, 0x0687, 0x8D30, + 0xE232, 0x6985, 0x7EEB, 0xF55C, 0x5037, 0xDB80, 0xCCEE, 0x4759, + 0x0D8F, 0x8638, 0x9156, 0x1AE1, 0xBF8A, 0x343D, 0x2353, 0xA8E4, + 0xB6FF, 0x3D48, 0x2A26, 0xA191, 0x04FA, 0x8F4D, 0x9823, 0x1394, + 0x5942, 0xD2F5, 0xC59B, 0x4E2C, 0xEB47, 0x60F0, 0x779E, 0xFC29, + 0x4BA8, 0xC01F, 0xD771, 0x5CC6, 0xF9AD, 0x721A, 0x6574, 0xEEC3, + 0xA415, 0x2FA2, 0x38CC, 0xB37B, 0x1610, 0x9DA7, 0x8AC9, 0x017E, + 0x1F65, 0x94D2, 0x83BC, 0x080B, 0xAD60, 0x26D7, 0x31B9, 0xBA0E, + 0xF0D8, 0x7B6F, 0x6C01, 0xE7B6, 0x42DD, 0xC96A, 0xDE04, 0x55B3 +}; -static void crc_t10dif_rehash(struct work_struct *work) +__u16 crc_t10dif_generic(__u16 crc, const unsigned char *buffer, size_t len) { - struct crypto_shash *new, *old; + unsigned int i; - mutex_lock(&crc_t10dif_mutex); - old = rcu_dereference_protected(crct10dif_tfm, - lockdep_is_held(&crc_t10dif_mutex)); - new = crypto_alloc_shash(CRC_T10DIF_STRING, 0, 0); - if (IS_ERR(new)) { - mutex_unlock(&crc_t10dif_mutex); - return; - } - rcu_assign_pointer(crct10dif_tfm, new); - mutex_unlock(&crc_t10dif_mutex); + for (i = 0 ; i < len ; i++) + crc = (crc << 8) ^ t10_dif_crc_table[((crc >> 8) ^ buffer[i]) & 0xff]; - if (old) { - synchronize_rcu(); - crypto_free_shash(old); - } else { - static_branch_disable(&crct10dif_fallback); - } + return crc; } - -static struct notifier_block crc_t10dif_nb = { - .notifier_call = crc_t10dif_notify, -}; +EXPORT_SYMBOL(crc_t10dif_generic); __u16 crc_t10dif_update(__u16 crc, const unsigned char *buffer, size_t len) { - struct { - struct shash_desc shash; - __u16 crc; - } desc; - int err; - - if (static_branch_unlikely(&crct10dif_fallback)) - return crc_t10dif_generic(crc, buffer, len); - - rcu_read_lock(); - desc.shash.tfm = rcu_dereference(crct10dif_tfm); - desc.crc = crc; - err = crypto_shash_update(&desc.shash, buffer, len); - rcu_read_unlock(); - - BUG_ON(err); - - return desc.crc; + return crc_t10dif_generic(crc, buffer, len); } EXPORT_SYMBOL(crc_t10dif_update); @@ -89,43 +80,12 @@ __u16 crc_t10dif(const unsigned char *buffer, size_t len) } EXPORT_SYMBOL(crc_t10dif); -static int __init crc_t10dif_mod_init(void) -{ - INIT_WORK(&crct10dif_rehash_work, crc_t10dif_rehash); - crypto_register_notifier(&crc_t10dif_nb); - crc_t10dif_rehash(&crct10dif_rehash_work); - return 0; -} - -static void __exit crc_t10dif_mod_fini(void) -{ - crypto_unregister_notifier(&crc_t10dif_nb); - cancel_work_sync(&crct10dif_rehash_work); - crypto_free_shash(rcu_dereference_protected(crct10dif_tfm, 1)); -} - -module_init(crc_t10dif_mod_init); -module_exit(crc_t10dif_mod_fini); - static int crc_t10dif_transform_show(char *buffer, const struct kernel_param *kp) { - struct crypto_shash *tfm; - int len; - - if (static_branch_unlikely(&crct10dif_fallback)) - return sprintf(buffer, "fallback\n"); - - rcu_read_lock(); - tfm = rcu_dereference(crct10dif_tfm); - len = snprintf(buffer, PAGE_SIZE, "%s\n", - crypto_shash_driver_name(tfm)); - rcu_read_unlock(); - - return len; + return sprintf(buffer, "fallback\n"); } module_param_call(transform, NULL, crc_t10dif_transform_show, NULL, 0444); MODULE_DESCRIPTION("T10 DIF CRC calculation (library API)"); MODULE_LICENSE("GPL"); -MODULE_SOFTDEP("pre: crct10dif"); From patchwork Mon Jan 11 16:52:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 361336 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20D5AC433DB for ; Mon, 11 Jan 2021 16:53:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C92AB2250F for ; Mon, 11 Jan 2021 16:53:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389094AbhAKQxh (ORCPT ); Mon, 11 Jan 2021 11:53:37 -0500 Received: from mail.kernel.org ([198.145.29.99]:46606 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728911AbhAKQxf (ORCPT ); Mon, 11 Jan 2021 11:53:35 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id AF74122AAD; Mon, 11 Jan 2021 16:52:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1610383975; bh=FJmpS5/x409gUNG+5LuompKBu98uDrt0lWpADeOtLqQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VBs4Sot7qg5OJcFDWlt21fNAQn/usBhOuLsUj0f7sCBmDpqBIUO+BH7GbPgHcgDzl k5VFchotKiRyp4o1hq1cGBYz/gOSOpOdkoTkUT3Ku5LxxyPqB0ix3blmnua3UplDyE Wky88jCZho4R7ku6hyjiJuAAUtuKw3XSyA3FDwzEOa3se1D9tUkI2J6SH1F/pFA03t 9HaDmboHjNicPqw07uWIzWNcq+tNlXWHQ7l01QSysfMrUAcmUCLH45qDjn7v8veztP qC7xRC4bqL8svQLRYdn1MxyP4Enxvt/kjHRmH+kCLqbXTro84B0SWPIs99JcxDok+R CjXTS+ZEYCARw== From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, Ard Biesheuvel , "Martin K. Petersen" , Eric Biggers , Peter Zijlstra Subject: [PATCH 2/7] crypto: lib/crc-t10dif - add static call support for optimized versions Date: Mon, 11 Jan 2021 17:52:32 +0100 Message-Id: <20210111165237.18178-3-ardb@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210111165237.18178-1-ardb@kernel.org> References: <20210111165237.18178-1-ardb@kernel.org> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Wire up the new static call facility to the CRC-T10DIF library code, so that optimized implementations can be swapped in easily, without having to rely on the complexity of the crypto API shash infrastructure. Signed-off-by: Ard Biesheuvel --- include/linux/crc-t10dif.h | 21 ++++++++++++-- lib/crc-t10dif.c | 30 +++++++++++++++----- 2 files changed, 42 insertions(+), 9 deletions(-) diff --git a/include/linux/crc-t10dif.h b/include/linux/crc-t10dif.h index 6bb0c0bf357b..ad8b0a358680 100644 --- a/include/linux/crc-t10dif.h +++ b/include/linux/crc-t10dif.h @@ -3,6 +3,7 @@ #define _LINUX_CRC_T10DIF_H #include +#include #define CRC_T10DIF_DIGEST_SIZE 2 #define CRC_T10DIF_BLOCK_SIZE 1 @@ -10,7 +11,23 @@ extern __u16 crc_t10dif_generic(__u16 crc, const unsigned char *buffer, size_t len); -extern __u16 crc_t10dif(unsigned char const *, size_t); -extern __u16 crc_t10dif_update(__u16 crc, unsigned char const *, size_t); + +DECLARE_STATIC_CALL(crc_t10dif_arch, crc_t10dif_generic); + +static inline __u16 crc_t10dif_update(__u16 crc, unsigned char const *buffer, + size_t len) +{ + return static_call(crc_t10dif_arch)(crc, buffer, len); +} + +static inline __u16 crc_t10dif(unsigned char const *buffer, size_t len) +{ + return crc_t10dif_update(0, buffer, len); +} + +int crc_t10dif_register(__u16 (*)(__u16, const unsigned char *, size_t), + const char *name); + +int crc_t10dif_unregister(void); #endif diff --git a/lib/crc-t10dif.c b/lib/crc-t10dif.c index cbb739f0a0d7..b657cdfb118a 100644 --- a/lib/crc-t10dif.c +++ b/lib/crc-t10dif.c @@ -18,6 +18,11 @@ #include #include +DEFINE_STATIC_CALL(crc_t10dif_arch, crc_t10dif_generic); +EXPORT_STATIC_CALL(crc_t10dif_arch); + +static char const *crc_t10dif_arch_name; + /* Table generated using the following polynomium: * x^16 + x^15 + x^11 + x^9 + x^8 + x^7 + x^5 + x^4 + x^2 + x + 1 * gt: 0x8bb7 @@ -68,21 +73,32 @@ __u16 crc_t10dif_generic(__u16 crc, const unsigned char *buffer, size_t len) } EXPORT_SYMBOL(crc_t10dif_generic); -__u16 crc_t10dif_update(__u16 crc, const unsigned char *buffer, size_t len) +int crc_t10dif_register(__u16 (*func)(__u16, const unsigned char *, size_t), + const char *name) { - return crc_t10dif_generic(crc, buffer, len); + if (!name || cmpxchg(&crc_t10dif_arch_name, NULL, name) != NULL) + return -EBUSY; + + static_call_update(crc_t10dif_arch, func); + return 0; } -EXPORT_SYMBOL(crc_t10dif_update); +EXPORT_SYMBOL_NS(crc_t10dif_register, CRYPTO_INTERNAL); -__u16 crc_t10dif(const unsigned char *buffer, size_t len) +int crc_t10dif_unregister(void) { - return crc_t10dif_update(0, buffer, len); + // revert to generic implementation + static_call_update(crc_t10dif_arch, crc_t10dif_generic); + crc_t10dif_arch_name = NULL; + + // wait for all potential callers of the unregistered routine to finish + synchronize_rcu_tasks(); + return 0; } -EXPORT_SYMBOL(crc_t10dif); +EXPORT_SYMBOL_NS(crc_t10dif_unregister, CRYPTO_INTERNAL); static int crc_t10dif_transform_show(char *buffer, const struct kernel_param *kp) { - return sprintf(buffer, "fallback\n"); + return sprintf(buffer, "%s\n", crc_t10dif_arch_name ?: "fallback"); } module_param_call(transform, NULL, crc_t10dif_transform_show, NULL, 0444); From patchwork Mon Jan 11 16:52:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 360624 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63315C433E0 for ; Mon, 11 Jan 2021 16:53:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 38E5E2242A for ; Mon, 11 Jan 2021 16:53:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728911AbhAKQxh (ORCPT ); Mon, 11 Jan 2021 11:53:37 -0500 Received: from mail.kernel.org ([198.145.29.99]:46622 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389086AbhAKQxh (ORCPT ); Mon, 11 Jan 2021 11:53:37 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id 6CB0A22ADF; Mon, 11 Jan 2021 16:52:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1610383976; bh=1sXKue0EfaLTfGbGolSyeQaFsOf9FB5CSBB20r3snh0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sTIqcUHR/Xe1KQN+Cjvq/6quoa+HxCnDNH8wxbMS0t3hEukQTaw2/dcMc6C3Qf0xd e1PDAaKHqObTbnuozgyCo0LVRzoeL8Gp/vqCjojq4Y/deRZh322ceqg9Upicsn/qAW nvPP0TfnVgeaFKbgOPM2PWjfgyRdUnBfg28lgbpgXAqU83yJ/1b28SBEA+ONGG8x+i HX5FVQb+1GKeZtn1+TRAMxl3YbEkvJBq3d09vnk81t+F+fVImyQMDjm/g6OZZS52/I H1V90AnEN1mhI6GA8vHW+/kIrnDorL+K99R6B61cpJVvrWYfSiYu3ppHWrC3johyg5 MskgvZuunBduQ== From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, Ard Biesheuvel , "Martin K. Petersen" , Eric Biggers , Peter Zijlstra Subject: [PATCH 3/7] crypto: generic/crc-t10dif - expose both arch and generic shashes Date: Mon, 11 Jan 2021 17:52:33 +0100 Message-Id: <20210111165237.18178-4-ardb@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210111165237.18178-1-ardb@kernel.org> References: <20210111165237.18178-1-ardb@kernel.org> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org In order to retain the ability to check accelerated implementations of CRC-T10DIF against the generic C implementation, expose both via the generic crypto API shash driver. This relies on the arch specific version to be registered beforehand - this will generally be the case, given that the modules in question are autoloaded via CPU feature matching, but let's use a module soft-dependency as well to trigger a load of such modules if they exist on the system. Signed-off-by: Ard Biesheuvel --- crypto/crct10dif_generic.c | 100 ++++++++++++++------ 1 file changed, 72 insertions(+), 28 deletions(-) diff --git a/crypto/crct10dif_generic.c b/crypto/crct10dif_generic.c index e843982073bb..7878eda7513e 100644 --- a/crypto/crct10dif_generic.c +++ b/crypto/crct10dif_generic.c @@ -48,8 +48,8 @@ static int chksum_init(struct shash_desc *desc) return 0; } -static int chksum_update(struct shash_desc *desc, const u8 *data, - unsigned int length) +static int chksum_generic_update(struct shash_desc *desc, const u8 *data, + unsigned int length) { struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); @@ -65,54 +65,97 @@ static int chksum_final(struct shash_desc *desc, u8 *out) return 0; } -static int __chksum_finup(__u16 crc, const u8 *data, unsigned int len, u8 *out) +static int __chksum_generic_finup(__u16 crc, const u8 *data, unsigned int len, + u8 *out) { *(__u16 *)out = crc_t10dif_generic(crc, data, len); return 0; } -static int chksum_finup(struct shash_desc *desc, const u8 *data, - unsigned int len, u8 *out) +static int chksum_generic_finup(struct shash_desc *desc, const u8 *data, + unsigned int len, u8 *out) { struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); - return __chksum_finup(ctx->crc, data, len, out); + return __chksum_generic_finup(ctx->crc, data, len, out); } -static int chksum_digest(struct shash_desc *desc, const u8 *data, - unsigned int length, u8 *out) +static int chksum_generic_digest(struct shash_desc *desc, const u8 *data, + unsigned int length, u8 *out) { - return __chksum_finup(0, data, length, out); + return __chksum_generic_finup(0, data, length, out); } -static struct shash_alg alg = { - .digestsize = CRC_T10DIF_DIGEST_SIZE, - .init = chksum_init, - .update = chksum_update, - .final = chksum_final, - .finup = chksum_finup, - .digest = chksum_digest, - .descsize = sizeof(struct chksum_desc_ctx), - .base = { - .cra_name = "crct10dif", - .cra_driver_name = "crct10dif-generic", - .cra_priority = 100, - .cra_blocksize = CRC_T10DIF_BLOCK_SIZE, - .cra_module = THIS_MODULE, - } -}; +static int chksum_arch_update(struct shash_desc *desc, const u8 *data, + unsigned int length) +{ + struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); + + ctx->crc = crc_t10dif_update(ctx->crc, data, length); + return 0; +} + +static int __chksum_arch_finup(__u16 crc, const u8 *data, unsigned int len, + u8 *out) +{ + *(__u16 *)out = crc_t10dif_update(crc, data, len); + return 0; +} + +static int chksum_arch_finup(struct shash_desc *desc, const u8 *data, + unsigned int len, u8 *out) +{ + struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); + + return __chksum_arch_finup(ctx->crc, data, len, out); +} + +static int chksum_arch_digest(struct shash_desc *desc, const u8 *data, + unsigned int length, u8 *out) +{ + return __chksum_arch_finup(0, data, length, out); +} + +static struct shash_alg algs[] = {{ + .digestsize = CRC_T10DIF_DIGEST_SIZE, + .init = chksum_init, + .update = chksum_generic_update, + .final = chksum_final, + .finup = chksum_generic_finup, + .digest = chksum_generic_digest, + .descsize = sizeof(struct chksum_desc_ctx), + + .base.cra_name = "crct10dif", + .base.cra_driver_name = "crct10dif-generic", + .base.cra_blocksize = CRC_T10DIF_BLOCK_SIZE, + .base.cra_module = THIS_MODULE, +}, { + .digestsize = CRC_T10DIF_DIGEST_SIZE, + .init = chksum_init, + .update = chksum_arch_update, + .final = chksum_final, + .finup = chksum_arch_finup, + .digest = chksum_arch_digest, + .descsize = sizeof(struct chksum_desc_ctx), + + .base.cra_name = "crct10dif", + .base.cra_driver_name = "crct10dif-arch", + .base.cra_priority = 100, + .base.cra_blocksize = CRC_T10DIF_BLOCK_SIZE, + .base.cra_module = THIS_MODULE, +}}; static int __init crct10dif_mod_init(void) { - return crypto_register_shash(&alg); + return crypto_register_shashes(algs, ARRAY_SIZE(algs)); } static void __exit crct10dif_mod_fini(void) { - crypto_unregister_shash(&alg); + crypto_unregister_shashes(algs, ARRAY_SIZE(algs)); } -subsys_initcall(crct10dif_mod_init); +module_init(crct10dif_mod_init); module_exit(crct10dif_mod_fini); MODULE_AUTHOR("Tim Chen "); @@ -120,3 +163,4 @@ MODULE_DESCRIPTION("T10 DIF CRC calculation."); MODULE_LICENSE("GPL"); MODULE_ALIAS_CRYPTO("crct10dif"); MODULE_ALIAS_CRYPTO("crct10dif-generic"); +MODULE_SOFTDEP("pre: crct10dif-arch"); From patchwork Mon Jan 11 16:52:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 361335 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72686C433DB for ; Mon, 11 Jan 2021 16:53:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2A7032242A for ; Mon, 11 Jan 2021 16:53:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389098AbhAKQxj (ORCPT ); Mon, 11 Jan 2021 11:53:39 -0500 Received: from mail.kernel.org ([198.145.29.99]:46634 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389086AbhAKQxj (ORCPT ); Mon, 11 Jan 2021 11:53:39 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id 257EB22B30; Mon, 11 Jan 2021 16:52:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1610383978; bh=bDMPvU8UNU86OkjcEmn2+senxHqtQ5Y473/EwGLnoI0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=c7M53jAAcVKo1ZOMeLfLeZPJwEvn/bAtaWxYSu1gLnIMfhX1TmqVIVu3jNCn1StD0 iSvdT9n8Iq6MzxoNLdkAmIIHsAW3B9kJNQ+ghEopNXA26Fn9NuEYep3aORIs6K82CQ 0H61NDs5EW3JcGRUKcVagBBTy9AsrmWtiW6XHvNDwOF2XhSkb0FwPyBUttz+EbEUpB r2DENp7N5gYwhykjqSY0rr1hiUcrSpPXHbVYDzRh3jZFVmuQM9NbM1yF2Q90pjOwOh pUor2XFcvL5ZdVGRer9RoIn8FLhfbJp7gPowmZw8hG6EWelE+fwHvSr+/HlgP6eoYV jCqMTXLjfdUZw== From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, Ard Biesheuvel , "Martin K. Petersen" , Eric Biggers , Peter Zijlstra Subject: [PATCH 4/7] crypto: x86/crc-t10dif - convert to static call library API Date: Mon, 11 Jan 2021 17:52:34 +0100 Message-Id: <20210111165237.18178-5-ardb@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210111165237.18178-1-ardb@kernel.org> References: <20210111165237.18178-1-ardb@kernel.org> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Get rid of the shash boilerplate, and register the accelerated x86 version of the CRC-T10DIF algorithm with the library interface instead. Signed-off-by: Ard Biesheuvel --- arch/x86/crypto/crct10dif-pclmul_glue.c | 90 +++----------------- 1 file changed, 10 insertions(+), 80 deletions(-) diff --git a/arch/x86/crypto/crct10dif-pclmul_glue.c b/arch/x86/crypto/crct10dif-pclmul_glue.c index 71291d5af9f4..6f06499d96a3 100644 --- a/arch/x86/crypto/crct10dif-pclmul_glue.c +++ b/arch/x86/crypto/crct10dif-pclmul_glue.c @@ -25,7 +25,6 @@ #include #include #include -#include #include #include #include @@ -36,82 +35,17 @@ asmlinkage u16 crc_t10dif_pcl(u16 init_crc, const u8 *buf, size_t len); -struct chksum_desc_ctx { - __u16 crc; -}; - -static int chksum_init(struct shash_desc *desc) -{ - struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); - - ctx->crc = 0; - - return 0; -} - -static int chksum_update(struct shash_desc *desc, const u8 *data, - unsigned int length) -{ - struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); - - if (length >= 16 && crypto_simd_usable()) { - kernel_fpu_begin(); - ctx->crc = crc_t10dif_pcl(ctx->crc, data, length); - kernel_fpu_end(); - } else - ctx->crc = crc_t10dif_generic(ctx->crc, data, length); - return 0; -} - -static int chksum_final(struct shash_desc *desc, u8 *out) -{ - struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); - - *(__u16 *)out = ctx->crc; - return 0; -} - -static int __chksum_finup(__u16 crc, const u8 *data, unsigned int len, u8 *out) +static u16 crc_t10dif_x86(u16 crc, const u8 *data, size_t len) { if (len >= 16 && crypto_simd_usable()) { kernel_fpu_begin(); - *(__u16 *)out = crc_t10dif_pcl(crc, data, len); + crc = crc_t10dif_pcl(crc, data, len); kernel_fpu_end(); - } else - *(__u16 *)out = crc_t10dif_generic(crc, data, len); - return 0; -} - -static int chksum_finup(struct shash_desc *desc, const u8 *data, - unsigned int len, u8 *out) -{ - struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); - - return __chksum_finup(ctx->crc, data, len, out); -} - -static int chksum_digest(struct shash_desc *desc, const u8 *data, - unsigned int length, u8 *out) -{ - return __chksum_finup(0, data, length, out); -} - -static struct shash_alg alg = { - .digestsize = CRC_T10DIF_DIGEST_SIZE, - .init = chksum_init, - .update = chksum_update, - .final = chksum_final, - .finup = chksum_finup, - .digest = chksum_digest, - .descsize = sizeof(struct chksum_desc_ctx), - .base = { - .cra_name = "crct10dif", - .cra_driver_name = "crct10dif-pclmul", - .cra_priority = 200, - .cra_blocksize = CRC_T10DIF_BLOCK_SIZE, - .cra_module = THIS_MODULE, + } else { + crc = crc_t10dif_generic(crc, data, len); } -}; + return crc; +} static const struct x86_cpu_id crct10dif_cpu_id[] = { X86_MATCH_FEATURE(X86_FEATURE_PCLMULQDQ, NULL), @@ -121,15 +55,12 @@ MODULE_DEVICE_TABLE(x86cpu, crct10dif_cpu_id); static int __init crct10dif_intel_mod_init(void) { - if (!x86_match_cpu(crct10dif_cpu_id)) - return -ENODEV; - - return crypto_register_shash(&alg); + return crc_t10dif_register(crc_t10dif_x86, "crct10dif-pclmul"); } static void __exit crct10dif_intel_mod_fini(void) { - crypto_unregister_shash(&alg); + crc_t10dif_unregister(); } module_init(crct10dif_intel_mod_init); @@ -138,6 +69,5 @@ module_exit(crct10dif_intel_mod_fini); MODULE_AUTHOR("Tim Chen "); MODULE_DESCRIPTION("T10 DIF CRC calculation accelerated with PCLMULQDQ."); MODULE_LICENSE("GPL"); - -MODULE_ALIAS_CRYPTO("crct10dif"); -MODULE_ALIAS_CRYPTO("crct10dif-pclmul"); +MODULE_ALIAS("crct10dif-arch"); +MODULE_IMPORT_NS(CRYPTO_INTERNAL); From patchwork Mon Jan 11 16:52:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 361334 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30BCAC433DB for ; Mon, 11 Jan 2021 16:54:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E816F2250F for ; Mon, 11 Jan 2021 16:54:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389053AbhAKQyN (ORCPT ); Mon, 11 Jan 2021 11:54:13 -0500 Received: from mail.kernel.org ([198.145.29.99]:46722 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387574AbhAKQyM (ORCPT ); Mon, 11 Jan 2021 11:54:12 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id D421122B51; Mon, 11 Jan 2021 16:52:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1610383980; bh=0knrbaQvNbxq4+wTxIRfAEHDDOjfn9lPOK3J0RS2wsc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gNP1z2yEtNYn2IluLxqp/o0+rEF26hdwmv3ZaPxixw1ec2wO+cQfAynV7qXpmFNSh 8QlcMjnhtiZakLf47Kv9JcS3hcSpFgi5+LWYKqNJHAQbXVzv3UYlsC0w8gtg0jVozZ TEZ936rmwxxf22Lb8doQjal5l5azFKRJCO1lr0corPbOWC5RbTfYIr3zmSbpGFOyQL eSBzZR7Jo2NdjdAKvMgRbxwmX3slB0ThWGpfAriRERKSD9fQWSsATey1bvEVM4ixTP JBRjgEm08cdqcs0+uuDErb7vBU4s7d0XOXBr335G6NBX0H4WwJJNtAIAnr+32ZMDFK 3n+g3QUhxEMuQ== From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, Ard Biesheuvel , "Martin K. Petersen" , Eric Biggers , Peter Zijlstra Subject: [PATCH 5/7] crypto: arm/crc-t10dif - convert to static call library API Date: Mon, 11 Jan 2021 17:52:35 +0100 Message-Id: <20210111165237.18178-6-ardb@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210111165237.18178-1-ardb@kernel.org> References: <20210111165237.18178-1-ardb@kernel.org> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Get rid of the shash boilerplate, and register the accelerated ARM version of the CRC-T10DIF algorithm with the library interface instead. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/crct10dif-ce-glue.c | 58 ++++---------------- 1 file changed, 11 insertions(+), 47 deletions(-) diff --git a/arch/arm/crypto/crct10dif-ce-glue.c b/arch/arm/crypto/crct10dif-ce-glue.c index e9191a8c87b9..ce21f958fd49 100644 --- a/arch/arm/crypto/crct10dif-ce-glue.c +++ b/arch/arm/crypto/crct10dif-ce-glue.c @@ -5,13 +5,13 @@ * Copyright (C) 2016 Linaro Ltd */ +#include #include #include #include #include #include -#include #include #include @@ -21,68 +21,32 @@ asmlinkage u16 crc_t10dif_pmull(u16 init_crc, const u8 *buf, size_t len); -static int crct10dif_init(struct shash_desc *desc) +static u16 crc_t10dif_arm(u16 crc, const u8 *data, size_t len) { - u16 *crc = shash_desc_ctx(desc); - - *crc = 0; - return 0; -} - -static int crct10dif_update(struct shash_desc *desc, const u8 *data, - unsigned int length) -{ - u16 *crc = shash_desc_ctx(desc); - - if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) { + if (len >= CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) { kernel_neon_begin(); - *crc = crc_t10dif_pmull(*crc, data, length); + crc = crc_t10dif_pmull(crc, data, len); kernel_neon_end(); } else { - *crc = crc_t10dif_generic(*crc, data, length); + crc = crc_t10dif_generic(crc, data, len); } - - return 0; + return crc; } -static int crct10dif_final(struct shash_desc *desc, u8 *out) -{ - u16 *crc = shash_desc_ctx(desc); - - *(u16 *)out = *crc; - return 0; -} - -static struct shash_alg crc_t10dif_alg = { - .digestsize = CRC_T10DIF_DIGEST_SIZE, - .init = crct10dif_init, - .update = crct10dif_update, - .final = crct10dif_final, - .descsize = CRC_T10DIF_DIGEST_SIZE, - - .base.cra_name = "crct10dif", - .base.cra_driver_name = "crct10dif-arm-ce", - .base.cra_priority = 200, - .base.cra_blocksize = CRC_T10DIF_BLOCK_SIZE, - .base.cra_module = THIS_MODULE, -}; - static int __init crc_t10dif_mod_init(void) { - if (!(elf_hwcap2 & HWCAP2_PMULL)) - return -ENODEV; - - return crypto_register_shash(&crc_t10dif_alg); + return crc_t10dif_register(crc_t10dif_arm, "crct10dif-arm-ce"); } static void __exit crc_t10dif_mod_exit(void) { - crypto_unregister_shash(&crc_t10dif_alg); + crc_t10dif_unregister(); } -module_init(crc_t10dif_mod_init); +module_cpu_feature_match(PMULL, crc_t10dif_mod_init); module_exit(crc_t10dif_mod_exit); MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); -MODULE_ALIAS_CRYPTO("crct10dif"); +MODULE_ALIAS("crct10dif-arch"); +MODULE_IMPORT_NS(CRYPTO_INTERNAL); From patchwork Mon Jan 11 16:52:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 360623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1443AC433E0 for ; Mon, 11 Jan 2021 16:54:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C39BB22288 for ; Mon, 11 Jan 2021 16:54:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389154AbhAKQyN (ORCPT ); Mon, 11 Jan 2021 11:54:13 -0500 Received: from mail.kernel.org ([198.145.29.99]:46724 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389053AbhAKQyM (ORCPT ); Mon, 11 Jan 2021 11:54:12 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id 927D722BE9; Mon, 11 Jan 2021 16:53:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1610383981; bh=O6/RYp8CVubXs/Bqm7W26Mnby0JlQwuVMOuvid3rSkg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Hoki5xpiTdOzKYr9+8cag6XptcMh+EYuDSaouNdNXNRsZdgiD8dl4Or6xD6ABg4FZ TKa5vmJvf9qCowHM5lTQsiKsv9i0aZ8ZG6MAerMG2M/x1E/AvYAg1ZhBEJuu8Ogcde +XYjn366ZFEMgEVHczsrFr+N+FiWot6nKvFJxeq+6oadHitN7SoRxuoiquKL8U3iA4 7tObAkNy3/HfPsoUNV3P2MTRoimZiTxO/epEMhbc+oUTIHWFcG+rO7J9FPFR2Vv70N lu9IzMse/BZNmxO0OBaQJgWuUS/HdKeRyQVzvUhQIA6AgHhWdDV2uRUBfKT4lVjwJ0 +mWDTqZ97kN2w== From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, Ard Biesheuvel , "Martin K. Petersen" , Eric Biggers , Peter Zijlstra Subject: [PATCH 6/7] crypto: arm64/crc-t10dif - convert to static call API Date: Mon, 11 Jan 2021 17:52:36 +0100 Message-Id: <20210111165237.18178-7-ardb@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210111165237.18178-1-ardb@kernel.org> References: <20210111165237.18178-1-ardb@kernel.org> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Get rid of the shash boilerplate, and register the accelerated arm64 version of the CRC-T10DIF algorithm with the library interface instead. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/crct10dif-ce-glue.c | 85 ++++---------------- 1 file changed, 15 insertions(+), 70 deletions(-) diff --git a/arch/arm64/crypto/crct10dif-ce-glue.c b/arch/arm64/crypto/crct10dif-ce-glue.c index ccc3f6067742..70c730866d4a 100644 --- a/arch/arm64/crypto/crct10dif-ce-glue.c +++ b/arch/arm64/crypto/crct10dif-ce-glue.c @@ -12,7 +12,6 @@ #include #include -#include #include #include @@ -23,97 +22,43 @@ asmlinkage u16 crc_t10dif_pmull_p8(u16 init_crc, const u8 *buf, size_t len); asmlinkage u16 crc_t10dif_pmull_p64(u16 init_crc, const u8 *buf, size_t len); -static int crct10dif_init(struct shash_desc *desc) +static u16 crct10dif_update_pmull_p8(u16 crc, const u8 *data, size_t length) { - u16 *crc = shash_desc_ctx(desc); - - *crc = 0; - return 0; -} - -static int crct10dif_update_pmull_p8(struct shash_desc *desc, const u8 *data, - unsigned int length) -{ - u16 *crc = shash_desc_ctx(desc); - if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) { kernel_neon_begin(); - *crc = crc_t10dif_pmull_p8(*crc, data, length); + crc = crc_t10dif_pmull_p8(crc, data, length); kernel_neon_end(); } else { - *crc = crc_t10dif_generic(*crc, data, length); + crc = crc_t10dif_generic(crc, data, length); } - - return 0; + return crc; } -static int crct10dif_update_pmull_p64(struct shash_desc *desc, const u8 *data, - unsigned int length) +static u16 crct10dif_update_pmull_p64(u16 crc, const u8 *data, size_t length) { - u16 *crc = shash_desc_ctx(desc); - if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) { kernel_neon_begin(); - *crc = crc_t10dif_pmull_p64(*crc, data, length); + crc = crc_t10dif_pmull_p64(crc, data, length); kernel_neon_end(); } else { - *crc = crc_t10dif_generic(*crc, data, length); + crc = crc_t10dif_generic(crc, data, length); } - - return 0; + return crc; } -static int crct10dif_final(struct shash_desc *desc, u8 *out) -{ - u16 *crc = shash_desc_ctx(desc); - - *(u16 *)out = *crc; - return 0; -} - -static struct shash_alg crc_t10dif_alg[] = {{ - .digestsize = CRC_T10DIF_DIGEST_SIZE, - .init = crct10dif_init, - .update = crct10dif_update_pmull_p8, - .final = crct10dif_final, - .descsize = CRC_T10DIF_DIGEST_SIZE, - - .base.cra_name = "crct10dif", - .base.cra_driver_name = "crct10dif-arm64-neon", - .base.cra_priority = 100, - .base.cra_blocksize = CRC_T10DIF_BLOCK_SIZE, - .base.cra_module = THIS_MODULE, -}, { - .digestsize = CRC_T10DIF_DIGEST_SIZE, - .init = crct10dif_init, - .update = crct10dif_update_pmull_p64, - .final = crct10dif_final, - .descsize = CRC_T10DIF_DIGEST_SIZE, - - .base.cra_name = "crct10dif", - .base.cra_driver_name = "crct10dif-arm64-ce", - .base.cra_priority = 200, - .base.cra_blocksize = CRC_T10DIF_BLOCK_SIZE, - .base.cra_module = THIS_MODULE, -}}; - static int __init crc_t10dif_mod_init(void) { if (cpu_have_named_feature(PMULL)) - return crypto_register_shashes(crc_t10dif_alg, - ARRAY_SIZE(crc_t10dif_alg)); + return crc_t10dif_register(crct10dif_update_pmull_p64, + "crct10dif-arm64-ce"); else - /* only register the first array element */ - return crypto_register_shash(crc_t10dif_alg); + return crc_t10dif_register(crct10dif_update_pmull_p8, + "crct10dif-arm64-neon"); } static void __exit crc_t10dif_mod_exit(void) { - if (cpu_have_named_feature(PMULL)) - crypto_unregister_shashes(crc_t10dif_alg, - ARRAY_SIZE(crc_t10dif_alg)); - else - crypto_unregister_shash(crc_t10dif_alg); + crc_t10dif_unregister(); } module_cpu_feature_match(ASIMD, crc_t10dif_mod_init); @@ -121,5 +66,5 @@ module_exit(crc_t10dif_mod_exit); MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); -MODULE_ALIAS_CRYPTO("crct10dif"); -MODULE_ALIAS_CRYPTO("crct10dif-arm64-ce"); +MODULE_ALIAS("crct10dif-arch"); +MODULE_IMPORT_NS(CRYPTO_INTERNAL); From patchwork Mon Jan 11 16:52:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 360622 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5392BC433E6 for ; Mon, 11 Jan 2021 16:54:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 181BB22AAD for ; Mon, 11 Jan 2021 16:54:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389155AbhAKQyQ (ORCPT ); Mon, 11 Jan 2021 11:54:16 -0500 Received: from mail.kernel.org ([198.145.29.99]:46732 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387574AbhAKQyP (ORCPT ); Mon, 11 Jan 2021 11:54:15 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id 5AC8E22BED; Mon, 11 Jan 2021 16:53:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1610383983; bh=M7+Zu321ZKbQuqu/bQyhrejFbYMgPSTQhX6ej4RnGKw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IbTPZD+cOFCZQ1KZizzjgCbiKxWG/RTuXvLdzlTApBKLgKDKCjw4m6tfmZ758Qx0w PBTkxKq7r1043sP2G+dxEVdH0QULIjKEr38e/EjLSZt5V/A05HwyKhQpr36Z7bomlv P8PRucXXtaolRxlxRS13MpZQd3RV1wzMKisk+26JR+RxiXQNXz/tp4/u7+t+y2EtLZ 10CR0YCRQX6JSNMhxDN/ra7qCL7ZOHGzfV7dxC0/PLcwuDR34wkJKEg5CHqz7XliXT rRnkK93vxHbZZcO4aLSIhhKQsV5H97UNVvRg3ib9eU5Oyf6lukSjwWbnHPl74Omgax S+K71x6t2jC8w== From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, Ard Biesheuvel , "Martin K. Petersen" , Eric Biggers , Peter Zijlstra Subject: [PATCH 7/7] crypto: powerpc/crc-t10dif - convert to static call API Date: Mon, 11 Jan 2021 17:52:37 +0100 Message-Id: <20210111165237.18178-8-ardb@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210111165237.18178-1-ardb@kernel.org> References: <20210111165237.18178-1-ardb@kernel.org> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Get rid of the shash boilerplate, and register the accelerated PowerPC version of the CRC-T10DIF algorithm with the library interface instead. Signed-off-by: Ard Biesheuvel --- arch/powerpc/crypto/crct10dif-vpmsum_glue.c | 51 ++------------------ 1 file changed, 4 insertions(+), 47 deletions(-) diff --git a/arch/powerpc/crypto/crct10dif-vpmsum_glue.c b/arch/powerpc/crypto/crct10dif-vpmsum_glue.c index 1dc8b6915178..26c7da181018 100644 --- a/arch/powerpc/crypto/crct10dif-vpmsum_glue.c +++ b/arch/powerpc/crypto/crct10dif-vpmsum_glue.c @@ -7,7 +7,6 @@ */ #include -#include #include #include #include @@ -61,59 +60,17 @@ static u16 crct10dif_vpmsum(u16 crci, unsigned char const *p, size_t len) return crc & 0xffff; } -static int crct10dif_vpmsum_init(struct shash_desc *desc) -{ - u16 *crc = shash_desc_ctx(desc); - - *crc = 0; - return 0; -} - -static int crct10dif_vpmsum_update(struct shash_desc *desc, const u8 *data, - unsigned int length) -{ - u16 *crc = shash_desc_ctx(desc); - - *crc = crct10dif_vpmsum(*crc, data, length); - - return 0; -} - - -static int crct10dif_vpmsum_final(struct shash_desc *desc, u8 *out) -{ - u16 *crcp = shash_desc_ctx(desc); - - *(u16 *)out = *crcp; - return 0; -} - -static struct shash_alg alg = { - .init = crct10dif_vpmsum_init, - .update = crct10dif_vpmsum_update, - .final = crct10dif_vpmsum_final, - .descsize = CRC_T10DIF_DIGEST_SIZE, - .digestsize = CRC_T10DIF_DIGEST_SIZE, - .base = { - .cra_name = "crct10dif", - .cra_driver_name = "crct10dif-vpmsum", - .cra_priority = 200, - .cra_blocksize = CRC_T10DIF_BLOCK_SIZE, - .cra_module = THIS_MODULE, - } -}; - static int __init crct10dif_vpmsum_mod_init(void) { if (!cpu_has_feature(CPU_FTR_ARCH_207S)) return -ENODEV; - return crypto_register_shash(&alg); + return crc_t10dif_register(crct10dif_vpmsum, "crct10dif-vpmsum"); } static void __exit crct10dif_vpmsum_mod_fini(void) { - crypto_unregister_shash(&alg); + crc_t10dif_unregister(); } module_cpu_feature_match(PPC_MODULE_FEATURE_VEC_CRYPTO, crct10dif_vpmsum_mod_init); @@ -122,5 +79,5 @@ module_exit(crct10dif_vpmsum_mod_fini); MODULE_AUTHOR("Daniel Axtens "); MODULE_DESCRIPTION("CRCT10DIF using vector polynomial multiply-sum instructions"); MODULE_LICENSE("GPL"); -MODULE_ALIAS_CRYPTO("crct10dif"); -MODULE_ALIAS_CRYPTO("crct10dif-vpmsum"); +MODULE_ALIAS("crct10dif-arch"); +MODULE_IMPORT_NS(CRYPTO_INTERNAL);