From patchwork Thu Oct 17 19:09:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 176706 Delivered-To: patch@linaro.org Received: by 2002:a92:7e96:0:0:0:0:0 with SMTP id q22csp1359490ill; Thu, 17 Oct 2019 12:10:07 -0700 (PDT) X-Google-Smtp-Source: APXvYqx1dxoD3f0x7Jd95Em972qUWk0Ve3Vp6rGaMvpMqrCOiZhSMw8yggy7LTPrbGfYp0UW/K9a X-Received: by 2002:aa7:cd43:: with SMTP id v3mr5413489edw.235.1571339407626; Thu, 17 Oct 2019 12:10:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1571339407; cv=none; d=google.com; s=arc-20160816; b=pDBju6Dap2eGjJ8+B2on991b9NwTMdKglPYfQMj3xW2rH7Y+XX+cnlPGfYpt+5eGIK lzywWguOBT0napSgBXSg85OamDujskOsP9KCqy0wmMj8iBIvMRIO5Flj5pOdV8W7RzhR S65XbMRj2YVIVZ/VzvirAIrjMrTs9jCsIDqBhiZ3LIWs1F+rlHUNuftXQbEh5HPy30W3 l+m3FGhDTi+zgULNGA4jY0NhlGq8hoeP9/LTUapxqN35BKDkflHcjKc2AqENmzIAGlg4 0Zu0gxv0/iJhO73j7PXJdudgAxJlDVohqPxkWtJXWPqJ+dpwnptNJk6Ty78FRc+I51jP m0SQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1BZFKQh4dFsI0CY4Tdkg4GCkEZnZ05oiXvCaqESz0Zg=; b=rI67xvKnZ7/e8Six7WJzbfCDBA/NSVbROguOfNzuLr9B58vfhMQ7zDeOw9TFxPXmaK YoS0ueVljIso/qcZ0ET5Ho8bvz0mt0fNNCU1KXgq7l+kteAMazZYkIXysuA3OL9uxmUp SBcCS1oGL22SCAB/Tp/13UIHCnSUR358SVUG/tnKcLY+IAl85DwrfdB2kqi3PKuyP/4G iRgjKrh35j/xBVBIPwaEbUTsfBelKjhyV3RhCcUAhx5pqXDOlH3Sq4BZaoll97EzRkr+ CKCCNhWQKYdpI38AQSsiDog7J6rk6V714qIgrF9ZQU0oEhz2V4+99UNd5znzQwPCE+NZ vUUA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=euKOtiAb; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e31si2292605ede.199.2019.10.17.12.10.07; Thu, 17 Oct 2019 12:10:07 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=euKOtiAb; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2503384AbfJQTKG (ORCPT + 3 others); Thu, 17 Oct 2019 15:10:06 -0400 Received: from mail-wm1-f68.google.com ([209.85.128.68]:37728 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2503387AbfJQTKG (ORCPT ); Thu, 17 Oct 2019 15:10:06 -0400 Received: by mail-wm1-f68.google.com with SMTP id f22so3682721wmc.2 for ; Thu, 17 Oct 2019 12:10:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1BZFKQh4dFsI0CY4Tdkg4GCkEZnZ05oiXvCaqESz0Zg=; b=euKOtiAb8QNlVFl1X8LHJcEXBTt2rh2H+/nPek23rT4MPsktA+/uYWD13kXR9p8ZBV j1Yu87tUaqLpCI6cE1NYcCJQ9i8rXkiG0GCyizIvtjRbURvFw6oqWxQ0pcKeRq81Uy9l Suksb5j/41PCc8lp8EpfF7eOjLXgSHP50aYj0/ktaOQSRxQTtL0RRDSHyO9MocEacb9s D5iOB/tzn9kuGu6doDCEW7CxclB0n0blSNuTEtV0mDK9rGNWtRk1OhuLZvQaLSyWWIJb xaNtyzvugFx4LIKv0X6l0oMco1pGZAwxQ1SCkEw3Gyu/PoAkwAeX7Vc9H3LjED8N6SR7 izWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1BZFKQh4dFsI0CY4Tdkg4GCkEZnZ05oiXvCaqESz0Zg=; b=sc1PxBkv1BoTD3+MEtAG2uApv/chxwtbSp/ERyMK/J9PSbLqUdb0tVFr0636kkCPL0 MNGZ3f8tMJehY7+mnFgT1ViJabrhqNjUiVd1ksr/mZtdeGg27ZhiqHfiGTcT0lYqRAyr y+0mZEItU4IZly6KcMcjnFmH9CIuZLi+E9J/6NdcU5HkQB3kcSLzvJG97SCEWw+RNhjl 2w9Kyz3or9jKY7NbzxJhp0wS7TyjGyVlM/WPvAYjLH853Tb69Eq1Nw5RUEpwhZL/8KZa rRSAPeqoXGbWlPf7VRvVR0sNQ1s4rPjxWWvgXLcju4LcPwHzgN5nUJ5OWynvX0kLA8zp RnfQ== X-Gm-Message-State: APjAAAXH+RxZVg0gbKmma/Pm+FTSWAWz0+vvLx/MQ/u92Kww1RTriYQt 2H18yBKPH+2B9Nq08etqVpnkXHMieX3zOz9Y X-Received: by 2002:a1c:6709:: with SMTP id b9mr4395274wmc.14.1571339404242; Thu, 17 Oct 2019 12:10:04 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:ccb6:e9d4:c1bc:d107]) by smtp.gmail.com with ESMTPSA id y3sm5124528wro.36.2019.10.17.12.10.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 17 Oct 2019 12:10:03 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: Ard Biesheuvel , Herbert Xu , David Miller , "Jason A . Donenfeld" , Samuel Neves , Arnd Bergmann , Eric Biggers , Andy Lutomirski , Martin Willi , Rene van Dorst , David Sterba Subject: [PATCH v4 09/35] crypto: arm/chacha - expose ARM ChaCha routine as library function Date: Thu, 17 Oct 2019 21:09:06 +0200 Message-Id: <20191017190932.1947-10-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191017190932.1947-1-ard.biesheuvel@linaro.org> References: <20191017190932.1947-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Expose the accelerated NEON ChaCha routine directly as a symbol export so that users of the ChaCha library API can use it directly. Given that calls into the library API will always go through the routines in this module if it is enabled, switch to static keys to select the optimal implementation available (which may be none at all, in which case we defer to the generic implementation for all invocations). Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/Kconfig | 1 + arch/arm/crypto/chacha-glue.c | 41 +++++++++++++++++++- 2 files changed, 41 insertions(+), 1 deletion(-) -- 2.20.1 diff --git a/arch/arm/crypto/Kconfig b/arch/arm/crypto/Kconfig index a31b0b95548d..265da3801e4f 100644 --- a/arch/arm/crypto/Kconfig +++ b/arch/arm/crypto/Kconfig @@ -128,6 +128,7 @@ config CRYPTO_CRC32_ARM_CE config CRYPTO_CHACHA20_NEON tristate "NEON and scalar accelerated ChaCha stream cipher algorithms" select CRYPTO_BLKCIPHER + select CRYPTO_ARCH_HAVE_LIB_CHACHA config CRYPTO_NHPOLY1305_NEON tristate "NEON accelerated NHPoly1305 hash function (for Adiantum)" diff --git a/arch/arm/crypto/chacha-glue.c b/arch/arm/crypto/chacha-glue.c index 2c0f76bc08b2..fc6e69184d59 100644 --- a/arch/arm/crypto/chacha-glue.c +++ b/arch/arm/crypto/chacha-glue.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include @@ -29,9 +30,11 @@ asmlinkage void hchacha_block_neon(const u32 *state, u32 *out, int nrounds); asmlinkage void chacha_doarm(u8 *dst, const u8 *src, unsigned int bytes, const u32 *state, int nrounds); +static __ro_after_init DEFINE_STATIC_KEY_FALSE(use_neon); + static inline bool neon_usable(void) { - return crypto_simd_usable(); + return static_branch_likely(&use_neon) && crypto_simd_usable(); } static void chacha_doneon(u32 *state, u8 *dst, const u8 *src, @@ -60,6 +63,40 @@ static void chacha_doneon(u32 *state, u8 *dst, const u8 *src, } } +void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds) +{ + if (!IS_ENABLED(CONFIG_KERNEL_MODE_NEON) || !neon_usable()) { + hchacha_block_arm(state, stream, nrounds); + } else { + kernel_neon_begin(); + hchacha_block_neon(state, stream, nrounds); + kernel_neon_end(); + } +} +EXPORT_SYMBOL(hchacha_block_arch); + +void chacha_init_arch(u32 *state, const u32 *key, const u8 *iv) +{ + chacha_init_generic(state, key, iv); +} +EXPORT_SYMBOL(chacha_init_arch); + +void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src, unsigned int bytes, + int nrounds) +{ + if (!IS_ENABLED(CONFIG_KERNEL_MODE_NEON) || !neon_usable() || + bytes <= CHACHA_BLOCK_SIZE) { + chacha_doarm(dst, src, bytes, state, nrounds); + state[12] += DIV_ROUND_UP(bytes, CHACHA_BLOCK_SIZE); + return; + } + + kernel_neon_begin(); + chacha_doneon(state, dst, src, bytes, nrounds); + kernel_neon_end(); +} +EXPORT_SYMBOL(chacha_crypt_arch); + static int chacha_stream_xor(struct skcipher_request *req, const struct chacha_ctx *ctx, const u8 *iv, bool neon) @@ -281,6 +318,8 @@ static int __init chacha_simd_mod_init(void) for (i = 0; i < ARRAY_SIZE(neon_algs); i++) neon_algs[i].base.cra_priority = 0; break; + default: + static_branch_enable(&use_neon); } err = crypto_register_skciphers(neon_algs, ARRAY_SIZE(neon_algs));