From patchwork Mon Oct 7 16:45:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 175369 Delivered-To: patch@linaro.org Received: by 2002:a92:7e96:0:0:0:0:0 with SMTP id q22csp4529108ill; Mon, 7 Oct 2019 09:46:29 -0700 (PDT) X-Google-Smtp-Source: APXvYqxZ2AaKsH/5C8hOngQ49QX3neBRLSGSrsbAlJdQ977EV6Gxx59An7TcwOS45qXGEdJXlrGB X-Received: by 2002:a05:6402:281:: with SMTP id l1mr30005781edv.40.1570466789184; Mon, 07 Oct 2019 09:46:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1570466789; cv=none; d=google.com; s=arc-20160816; b=VkvdCxcY4W21vvGWcBd5JgCVKUYkRmiKfsPY8P7soFB92L+mxLLx9NGTaj13rG2Gta TYn5Uv/EtUBSonDu51wQxV6g9UNVVZfkTIfvvWMWFJ/SHY9qNv0VbBoYC/4qczrqzVfy kyDX8SG92R0YqNTcowgswpV6OmX/0NZaPmvHYdiW/z5VP7mEhV7BLACNb69yf9xm1Wyz 2PKDL6Dd5MJPcX6gFu8clVEdP+J2NKR13WzfsynFhNQuN6zfRFgJFTk1AyF+Q63Ri3AR Xu801o/8nFcEhR0VUbgDE+Sx/R3aR2EPL+mqmsNyBvncLP9QOjwgnVkGdqrngZmqX+2N +h4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JUtSu2dFwCY1DgItWOyayZqegd+iOxLLlyUn/RxwQH4=; b=B94zpWqd9JDgqHPqs2U2N4MZF5VVjpIipMNBQ8lrsMUgtrSmtAns24BVj4ELZ7id1b tXg7UCtk6Otz0S+a0sHlpSeuFveXNZsz+a8kbofCImUe8jX3wT1eZzaJERvi3zj5sGxR HdQKCpu3rw7zPApBQ/DEfAa0ldIWxsAAXoAwuGaat0yCL7EyJpNRWGYsKKZwVg5Z7egC Hg2BvuJ7jrcCRRzHIZk5yWhJduc5dVSJLtvTulV0V6ViPGo50AWht0GNVsSaZpS5qdGG W543P+3cHeqmXZQA6EFN2qIYayrJq6WUdSDLktneRUBsoEbkagBlp1SfNtP60sDUOqec oVsg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ee7QoeFu; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f17si11357081edf.328.2019.10.07.09.46.29; Mon, 07 Oct 2019 09:46:29 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ee7QoeFu; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728939AbfJGQq2 (ORCPT + 3 others); Mon, 7 Oct 2019 12:46:28 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:36234 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729003AbfJGQq2 (ORCPT ); Mon, 7 Oct 2019 12:46:28 -0400 Received: by mail-wm1-f65.google.com with SMTP id m18so211915wmc.1 for ; Mon, 07 Oct 2019 09:46:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JUtSu2dFwCY1DgItWOyayZqegd+iOxLLlyUn/RxwQH4=; b=ee7QoeFuMaJ/4z/z0PSsyJNlt6IzvHrCKHoj213MKF7s83aFtpEe8Mm7N79VI+ZLiC UKQkKv0T8rUUwtrVJbl7rKR1v+qzYl+EWqbUYtzA60Z5O/mfLO3Zv7GKpdbWYyZObnyP F2Plzoo5dpF2cDD686uZBk5j83cnFRxXzuKlOiJp0kw56Jlp5SqU9nNoTGenBl/X6C1j xgpus41rLKea/XhYl4YzYqpO+4rArAhEY2GcVH2pAicHWkEsveWS3pN2buAQJ/ACw9+N 0So+uv3Kc26xVfAnxtT8+jQuwHi93rjAsYrudsa9X3/hQyIkaSTXCW6ZBTH8bFglhS6f NGnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JUtSu2dFwCY1DgItWOyayZqegd+iOxLLlyUn/RxwQH4=; b=DGzDylDpkhzQPx6HA+e46TxOlFZZ8catuCoL8mnB34AG/226yjJ8ZgJ/X8ALra5n+y NJMu4uYxhXEwiWZEwg/7iQv4xeQLKI2Wj1hYSpNnwlgBtByvTQiE7ECX6xl9Z8W6415b 6NghS3ztXpJk5pg2/BNwZut/pgZGf717DbHw/BqMLsAD0mZQb+X1dt+4Gf9YLqFdlJj7 IkEXXzFTyg9ybaUwbp/HEhXi/xzrPfhHeSisW93xeGbrgIwdnfLcMNdW49AMIHD5pHgz HNg9AvJ5upUVLVLsS9JdIHltmrWvO1amVj4PLrcfSnPatKCq4xlUHvO7sj9GgOE2a9z8 tmsw== X-Gm-Message-State: APjAAAXfK19Xoe7I0FQoMDUE73Nqhp+AfaqPlMD+r3edvZoki2KU1lVl 798CoAS5oFr646l0k84WC6WJv555N2Fk5A== X-Received: by 2002:a1c:6709:: with SMTP id b9mr202093wmc.14.1570466785874; Mon, 07 Oct 2019 09:46:25 -0700 (PDT) Received: from localhost.localdomain (laubervilliers-657-1-83-120.w92-154.abo.wanadoo.fr. [92.154.90.120]) by smtp.gmail.com with ESMTPSA id b22sm68507wmj.36.2019.10.07.09.46.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Oct 2019 09:46:25 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: Ard Biesheuvel , Herbert Xu , David Miller , "Jason A . Donenfeld" , Samuel Neves , Arnd Bergmann , Eric Biggers , Andy Lutomirski , Martin Willi , Rene van Dorst Subject: [PATCH v3 05/29] crypto: arm64/chacha - expose arm64 ChaCha routine as library function Date: Mon, 7 Oct 2019 18:45:46 +0200 Message-Id: <20191007164610.6881-6-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191007164610.6881-1-ard.biesheuvel@linaro.org> References: <20191007164610.6881-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Expose the accelerated NEON ChaCha routine directly as a symbol export so that users of the ChaCha library API can use it directly. Given that calls into the library API will always go through the routines in this module if it is enabled, switch to static keys to select the optimal implementation available (which may be none at all, in which case we defer to the generic implementation for all invocations). Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/Kconfig | 1 + arch/arm64/crypto/chacha-neon-glue.c | 47 ++++++++++++++++++-- 2 files changed, 44 insertions(+), 4 deletions(-) -- 2.20.1 diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index fdf52d5f18f9..17bada4b9dd2 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -104,6 +104,7 @@ config CRYPTO_CHACHA20_NEON depends on KERNEL_MODE_NEON select CRYPTO_BLKCIPHER select CRYPTO_LIB_CHACHA_GENERIC + select CRYPTO_ARCH_HAVE_LIB_CHACHA config CRYPTO_NHPOLY1305_NEON tristate "NHPoly1305 hash function using NEON instructions (for Adiantum)" diff --git a/arch/arm64/crypto/chacha-neon-glue.c b/arch/arm64/crypto/chacha-neon-glue.c index 6450bb9f55f4..57e50acf35f8 100644 --- a/arch/arm64/crypto/chacha-neon-glue.c +++ b/arch/arm64/crypto/chacha-neon-glue.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include @@ -36,6 +37,8 @@ asmlinkage void chacha_4block_xor_neon(u32 *state, u8 *dst, const u8 *src, int nrounds, int bytes); asmlinkage void hchacha_block_neon(const u32 *state, u32 *out, int nrounds); +static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon); + static void chacha_doneon(u32 *state, u8 *dst, const u8 *src, int bytes, int nrounds) { @@ -59,6 +62,37 @@ static void chacha_doneon(u32 *state, u8 *dst, const u8 *src, } } +void hchacha_block(const u32 *state, u32 *stream, int nrounds) +{ + if (!static_branch_likely(&have_neon) || !crypto_simd_usable()) { + hchacha_block_generic(state, stream, nrounds); + } else { + kernel_neon_begin(); + hchacha_block_neon(state, stream, nrounds); + kernel_neon_end(); + } +} +EXPORT_SYMBOL(hchacha_block); + +void chacha_init(u32 *state, const u32 *key, const u8 *iv) +{ + chacha_init_generic(state, key, iv); +} +EXPORT_SYMBOL(chacha_init); + +void chacha_crypt(u32 *state, u8 *dst, const u8 *src, unsigned int bytes, + int nrounds) +{ + if (!static_branch_likely(&have_neon) || bytes <= CHACHA_BLOCK_SIZE || + !crypto_simd_usable()) + return chacha_crypt_generic(state, dst, src, bytes, nrounds); + + kernel_neon_begin(); + chacha_doneon(state, dst, src, bytes, nrounds); + kernel_neon_end(); +} +EXPORT_SYMBOL(chacha_crypt); + static int chacha_neon_stream_xor(struct skcipher_request *req, const struct chacha_ctx *ctx, const u8 *iv) { @@ -78,7 +112,7 @@ static int chacha_neon_stream_xor(struct skcipher_request *req, if (nbytes < walk.total) nbytes = rounddown(nbytes, walk.stride); - if (!do_neon) { + if (!static_branch_likely(&have_neon) || !do_neon) { chacha_crypt_generic(state, walk.dst.virt.addr, walk.src.virt.addr, nbytes, ctx->nrounds); @@ -112,7 +146,9 @@ static int xchacha_neon(struct skcipher_request *req) chacha_init_generic(state, ctx->key, req->iv); - if (req->cryptlen > CHACHA_BLOCK_SIZE && crypto_simd_usable()) { + if (static_branch_likely(&have_neon) && + req->cryptlen > CHACHA_BLOCK_SIZE && + crypto_simd_usable()) { kernel_neon_begin(); hchacha_block_neon(state, subctx.key, ctx->nrounds); kernel_neon_end(); @@ -193,14 +229,17 @@ static struct skcipher_alg algs[] = { static int __init chacha_simd_mod_init(void) { if (!cpu_have_named_feature(ASIMD)) - return -ENODEV; + return 0; + + static_branch_enable(&have_neon); return crypto_register_skciphers(algs, ARRAY_SIZE(algs)); } static void __exit chacha_simd_mod_fini(void) { - crypto_unregister_skciphers(algs, ARRAY_SIZE(algs)); + if (cpu_have_named_feature(ASIMD)) + crypto_unregister_skciphers(algs, ARRAY_SIZE(algs)); } module_init(chacha_simd_mod_init);