From patchwork Wed Oct 2 14:16:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 174999 Delivered-To: patch@linaro.org Received: by 2002:a92:7e96:0:0:0:0:0 with SMTP id q22csp818025ill; Wed, 2 Oct 2019 07:17:44 -0700 (PDT) X-Google-Smtp-Source: APXvYqwSdny+yEyByt4BvZjrKta9t/crEDqc63+KBb+TrJc8BGMRp6vjd2fWoKqppYimbgAyJcE+ X-Received: by 2002:a50:9f42:: with SMTP id b60mr4150675edf.192.1570025864658; Wed, 02 Oct 2019 07:17:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1570025864; cv=none; d=google.com; s=arc-20160816; b=oyO2YBBrcTXZUyiHrSgDivI9zJs4AR3bGEe70ugFbWG495htFxuueJ6DfQkDw90eQ+ 7YpLcNVq4I6U6myVE53fqTnvwtom4m/ITnwNPU75tagmr1XLvq8dgZz6pzmQoHOcdLYx t2zPiXMJfvGEL6jRjGlfdQvf8kbAqABoNMwSeJWY+t5tKa7LCBCudLTWP5phQ3ZgH6hL 8pVCRqF26XezLIIoxLken+8BMhliCwxAdTIw0ycm95XkA9NmY0ACMvRnkAkUwpwuNQ4S YRn9jTpijMpr0LRm5SNJy2LEQwRxW9AhDwoVwogOf9lsySeeRz3BoK1wIpQpCYBLecu9 JFaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9eoG0Ja/MkKdV4fIyrYwW1AyCVa60yb4N0sHgj42YTs=; b=l5/q+Se2GKlac10WQzTaltOgYLesMiYyuhzC7REyhGEcbGR6LqhP2ExzC5AdjpFXAw NOzmw5Kt02PlyNiS5idGH+vi1/r0qhlAgpaluIv4+gwA7tb4UNszaj8gedGzV9wR+Rep u1+66rbkHvJg30jp4uaunarAqVeYFL8C+ejGKh2OfPw4BBF4n8QSpw6Fx3VVhzMSfSuH 3OMX4xxDFXylyW8Nm0+Zn6/P54EWeuXQu/z/YoOtkJrSlAnVRZ2Ip5UjF8SyVhX1Xr/w ZTWyIHSmrmzmN+/DSvzRLLo6raem1zUIX3PNXGVN2JQGn5OeVC+mZwstpMEY4fAMB1zv e3yA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="B+lm8R/n"; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id va7si10939370ejb.4.2019.10.02.07.17.44; Wed, 02 Oct 2019 07:17:44 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="B+lm8R/n"; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727656AbfJBORo (ORCPT + 3 others); Wed, 2 Oct 2019 10:17:44 -0400 Received: from mail-wm1-f68.google.com ([209.85.128.68]:52801 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727210AbfJBORn (ORCPT ); Wed, 2 Oct 2019 10:17:43 -0400 Received: by mail-wm1-f68.google.com with SMTP id r19so7410765wmh.2 for ; Wed, 02 Oct 2019 07:17:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9eoG0Ja/MkKdV4fIyrYwW1AyCVa60yb4N0sHgj42YTs=; b=B+lm8R/nN/ZWjtHJoIeJNWQxYFloKyi1DRQo8Ky/B8LNhnhVxligrSXzQdCWBJvZHZ 8gRjC3oe8686Iha7gOdKkCj3/QtNBsIQw632s7nA2hxw5awYOp/UVJXyUhHCBDZvthfy sS51VSaokdKNQJRAhc1jBg+JPWlV/K0VpaXQ0KGtRdA0Ost8du9jjqYJa/wqgImMARog QDdEmNhwMtoMOdRV+1yh02s2LwT6EzmLAMZr4sCo7feIsgUl6KApHFsW1wC4Es8hOEoh hkihIRQRp8qiGqdO8YYMcbhWoxTRN7IooneWfm8Rx1l8STr+56YmzHRfYqv8ozVRPJVn WbcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9eoG0Ja/MkKdV4fIyrYwW1AyCVa60yb4N0sHgj42YTs=; b=H5YSnrQzlEN3Fv7yATUEfoyXIEjJyd/KOb8iuEdEZhTglm1SZ+tPIYdCSiNwgSXuZL EGy92h+BkPuZqzkyk+x9gLxa3BrlKT5vkwnya4t1AAlUMtHFI2GrvTUF4nDFFZ+PNe64 WSgO95Y1HwOnQ3HNynUnCnoMdytjZ1z38Ct6V/NkDlnY9Na/IluH3RQ9lAA3S+U0SwyS vZUsP+4kp1EM+8w+Bjn9uWsT7xosD+YnlIWj0RnjCt/yNQrZyoEiKlJc7jI6e3Ymx/zV aytPdg7NrJ+jXlOKEZjqbf3MN20PyMnnSM3Div3345xuJ4ZNx8Hr9NZRtHfYACmLo3Pn 5P0g== X-Gm-Message-State: APjAAAX1ls3GQr4XGdk3hJhEsHj+uyMtGQSRgLB80rXlGGb53pJEZb0j JxwvAXcUTsi0O7zZVm2yce58hfPcPL0KSMoU X-Received: by 2002:a05:600c:351:: with SMTP id u17mr3033643wmd.1.1570025861369; Wed, 02 Oct 2019 07:17:41 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:f145:3252:fc29:76c9]) by smtp.gmail.com with ESMTPSA id t13sm41078149wra.70.2019.10.02.07.17.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 02 Oct 2019 07:17:40 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: Ard Biesheuvel , Herbert Xu , David Miller , Greg KH , Linus Torvalds , "Jason A . Donenfeld" , Samuel Neves , Dan Carpenter , Arnd Bergmann , Eric Biggers , Andy Lutomirski , Will Deacon , Marc Zyngier , Catalin Marinas , Martin Willi , Peter Zijlstra , Josh Poimboeuf Subject: [PATCH v2 02/20] crypto: x86/chacha - expose SIMD ChaCha routine as library function Date: Wed, 2 Oct 2019 16:16:55 +0200 Message-Id: <20191002141713.31189-3-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191002141713.31189-1-ard.biesheuvel@linaro.org> References: <20191002141713.31189-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Wire the existing x86 SIMD ChaCha code into the new ChaCha library interface. Signed-off-by: Ard Biesheuvel --- arch/x86/crypto/chacha_glue.c | 36 ++++++++++++++++++++ crypto/Kconfig | 1 + include/crypto/chacha.h | 6 ++++ 3 files changed, 43 insertions(+) -- 2.20.1 diff --git a/arch/x86/crypto/chacha_glue.c b/arch/x86/crypto/chacha_glue.c index bc62daa8dafd..fd9ef42842cf 100644 --- a/arch/x86/crypto/chacha_glue.c +++ b/arch/x86/crypto/chacha_glue.c @@ -123,6 +123,42 @@ static void chacha_dosimd(u32 *state, u8 *dst, const u8 *src, } } +void hchacha_block(const u32 *state, u32 *stream, int nrounds) +{ + state = PTR_ALIGN(state, CHACHA_STATE_ALIGN); + + if (!crypto_simd_usable()) { + hchacha_block_generic(state, stream, nrounds); + } else { + kernel_fpu_begin(); + hchacha_block_ssse3(state, stream, nrounds); + kernel_fpu_end(); + } +} +EXPORT_SYMBOL(hchacha_block); + +void chacha_init(u32 *state, const u32 *key, const u8 *iv) +{ + state = PTR_ALIGN(state, CHACHA_STATE_ALIGN); + + chacha_init_generic(state, key, iv); +} +EXPORT_SYMBOL(chacha_init); + +void chacha_crypt(u32 *state, u8 *dst, const u8 *src, unsigned int bytes, + int nrounds) +{ + state = PTR_ALIGN(state, CHACHA_STATE_ALIGN); + + if (bytes <= CHACHA_BLOCK_SIZE || !crypto_simd_usable()) + return chacha_crypt_generic(state, dst, src, bytes, nrounds); + + kernel_fpu_begin(); + chacha_dosimd(state, dst, src, bytes, nrounds); + kernel_fpu_end(); +} +EXPORT_SYMBOL(chacha_crypt); + static int chacha_simd_stream_xor(struct skcipher_walk *walk, const struct chacha_ctx *ctx, const u8 *iv) { diff --git a/crypto/Kconfig b/crypto/Kconfig index 074b125819b0..f90b53a526ba 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -1436,6 +1436,7 @@ config CRYPTO_CHACHA20_X86_64 depends on X86 && 64BIT select CRYPTO_BLKCIPHER select CRYPTO_CHACHA20 + select CRYPTO_ARCH_HAVE_LIB_CHACHA help SSSE3, AVX2, and AVX-512VL optimized implementations of the ChaCha20, XChaCha20, and XChaCha12 stream ciphers. diff --git a/include/crypto/chacha.h b/include/crypto/chacha.h index 95a4a0ff4f7d..58192096679d 100644 --- a/include/crypto/chacha.h +++ b/include/crypto/chacha.h @@ -25,6 +25,12 @@ #define CHACHA_BLOCK_SIZE 64 #define CHACHAPOLY_IV_SIZE 12 +#ifdef CONFIG_X86_64 +#define CHACHA_STATE_WORDS ((CHACHA_BLOCK_SIZE + 12) / sizeof(u32)) +#else +#define CHACHA_STATE_WORDS (CHACHA_BLOCK_SIZE / sizeof(u32)) +#endif + /* 192-bit nonce, then 64-bit stream position */ #define XCHACHA_IV_SIZE 32