From patchwork Mon Oct 7 16:45:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 175365 Delivered-To: patch@linaro.org Received: by 2002:a92:7e96:0:0:0:0:0 with SMTP id q22csp4529030ill; Mon, 7 Oct 2019 09:46:25 -0700 (PDT) X-Google-Smtp-Source: APXvYqxPboJE9r/iid4BsyLM2ZTdQamIvJHWRtix2CLKDupmhgUlA7cffclcdPKW7iTvLSkn1cVU X-Received: by 2002:a17:906:c822:: with SMTP id dd2mr24265952ejb.165.1570466785758; Mon, 07 Oct 2019 09:46:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1570466785; cv=none; d=google.com; s=arc-20160816; b=cz4gJVgi7r3Le29lExk0SewIuDRbm1BuBVkPdg2e99NTQ/+GEdW63uYhj8XEj9reaR yjOSWbxWyQpL6o1pV0Ss8x8Ihzm6ara9aPnjCpyxXNnUdgKCO88ea9TEcC8KtQigYSul SuGD2I3QZ6M2GOOTPPdbRAqs86+W/vzyrQ0mR1fahGZYnq2pqndxPuZONWNkt9U0SKf9 6oX+T5MyqahJ807FXvwB9Guzl3KK6zBBgC/pyDIaeCY3hcw22Hqa2stDewFyWbt2AmT8 s3klR4iXqYDL8AXYHJEM05T1FWn05Gjpb4r13r1IwkBYuwuIdtsuYIc9FJ2IwbdTWAv7 h9rg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Wacjnu5s+UlDpKfDkG2erDF4AHgi0dg4fYncQyjWISM=; b=sKVjHU0OVl3Cj31RDPjpjbBfjzMI5orx2onEG2l2SJT45uuXsnrmFtwE5IzxV2WlQc J1DcF7RzfxpJ8DQvM+ksxEFUAIemPYh17583bdL85vbrhPSrnrn2wRzerA8HCvFuBBjg yUBdwRlEJzM+pyAKHRq69ytawPgOavwQYvjZgygKcn4S8DmRJyvPQLxDylYuOl8Uc5O5 7L3gPGOkG2ojD74JgF7VCo8dQ14ISlYsLPmqfHfnLb8beCc5mjNlHV9IiqqF+VXXfzgq 3lPuiOiu1uldAt4+BAVBjZrmOfIWBrlm+m/t7qZJX4TAHXp8JXxFuOck2OGnQYXwvGY6 EfbA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IEODKYqO; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f17si11357081edf.328.2019.10.07.09.46.25; Mon, 07 Oct 2019 09:46:25 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IEODKYqO; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729000AbfJGQqY (ORCPT + 3 others); Mon, 7 Oct 2019 12:46:24 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:43372 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728996AbfJGQqX (ORCPT ); Mon, 7 Oct 2019 12:46:23 -0400 Received: by mail-wr1-f65.google.com with SMTP id j18so15344283wrq.10 for ; Mon, 07 Oct 2019 09:46:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Wacjnu5s+UlDpKfDkG2erDF4AHgi0dg4fYncQyjWISM=; b=IEODKYqO1OXyor34Q+y4F1Rab1ZJ3bHIW9f/2T/VAOycnVkCsTtJo2xVEOl3w1M0NG yBfa6DO1sV4LZ6A4WBGwibsHBJDCHjrnmjTJbepB//17HViMrT082zaKkcxbmUn1/Qew f1zcEaJPj9FeZWt0wfvVoRyFjhur1zWLqzN8Bbgc4PmUz2hE/B/2fqmSjWueLfV089Jv 9SxAua90+YJ3IPC79bPGJse/qpn9AtaejjHhDhxyYeHHRNpZ6QP7ha8RSHGAu5fo3MuE +WxxJGVPxycH7ez4vbNmpq0XB7G+ssun0odIeQ4/WBEBpuSAo5KTZppN5mkvV7zA/whK qiaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Wacjnu5s+UlDpKfDkG2erDF4AHgi0dg4fYncQyjWISM=; b=pmqW1X1xozuP6j8TxC6/0DiAfYoJZ9pDXF0MG/RYxjHKSVVA+Wzb3nvEXZm2JV8Ye8 0O0hHrTWs6Wa2MLdrcf/x4XOKAexLF3Kt//p3Qple2AJqNbfjseh0G4BBpauacUkOCk7 hLEW4cEKR02gr0EP8jYkyM2edEXgAPLMxxWEXvN5CWKsjDzeQyJ84YtMT4AYgck7/D19 llM5BhFI1uRa5aikK53IFd3gU8P96wP7AfJ3mgwx3DjN9BZ000luorQFBqaI97d0p9xz wyI5Wz2pH3lmqS4oWhThhiht83sIMamLc4AAYsZ3kl8Pfyb6Qkinh/ZULtdkEal0xrsO gacA== X-Gm-Message-State: APjAAAXNhGJVUuoSMrVo3QemU+L3AOxkBAn2ULld/tDzqsKmTNwRBri2 GJCFTd72nzFXMLmBtci3zd8lqfYac/6vbQ== X-Received: by 2002:adf:8163:: with SMTP id 90mr19685075wrm.129.1570466781821; Mon, 07 Oct 2019 09:46:21 -0700 (PDT) Received: from localhost.localdomain (laubervilliers-657-1-83-120.w92-154.abo.wanadoo.fr. [92.154.90.120]) by smtp.gmail.com with ESMTPSA id b22sm68507wmj.36.2019.10.07.09.46.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Oct 2019 09:46:21 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: Ard Biesheuvel , Herbert Xu , David Miller , "Jason A . Donenfeld" , Samuel Neves , Arnd Bergmann , Eric Biggers , Andy Lutomirski , Martin Willi , Rene van Dorst Subject: [PATCH v3 02/29] crypto: x86/chacha - depend on generic chacha library instead of crypto driver Date: Mon, 7 Oct 2019 18:45:43 +0200 Message-Id: <20191007164610.6881-3-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191007164610.6881-1-ard.biesheuvel@linaro.org> References: <20191007164610.6881-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org In preparation of extending the x86 ChaCha driver to also expose the ChaCha library interface, drop the dependency on the chacha_generic crypto driver as a non-SIMD fallback, and depend on the generic ChaCha library directly. This way, we only pull in the code we actually need, without registering a set of ChaCha skciphers that we will never use. Since turning the FPU on and off is cheap these days, simplify the SIMD routine by dropping the per-page yield, which makes for a cleaner switch to the library API as well. Signed-off-by: Ard Biesheuvel --- arch/x86/crypto/chacha_glue.c | 77 ++++++++++---------- crypto/Kconfig | 2 +- 2 files changed, 40 insertions(+), 39 deletions(-) -- 2.20.1 diff --git a/arch/x86/crypto/chacha_glue.c b/arch/x86/crypto/chacha_glue.c index bc62daa8dafd..3a1a11a4326d 100644 --- a/arch/x86/crypto/chacha_glue.c +++ b/arch/x86/crypto/chacha_glue.c @@ -127,32 +127,32 @@ static int chacha_simd_stream_xor(struct skcipher_walk *walk, const struct chacha_ctx *ctx, const u8 *iv) { u32 *state, state_buf[16 + 2] __aligned(8); - int next_yield = 4096; /* bytes until next FPU yield */ + bool do_simd; int err = 0; BUILD_BUG_ON(CHACHA_STATE_ALIGN != 16); state = PTR_ALIGN(state_buf + 0, CHACHA_STATE_ALIGN); - crypto_chacha_init(state, ctx, iv); + chacha_init_generic(state, ctx->key, iv); + do_simd = (walk->total > CHACHA_BLOCK_SIZE) && crypto_simd_usable(); while (walk->nbytes > 0) { unsigned int nbytes = walk->nbytes; - if (nbytes < walk->total) { + if (nbytes < walk->total) nbytes = round_down(nbytes, walk->stride); - next_yield -= nbytes; - } - - chacha_dosimd(state, walk->dst.virt.addr, walk->src.virt.addr, - nbytes, ctx->nrounds); - if (next_yield <= 0) { - /* temporarily allow preemption */ - kernel_fpu_end(); + if (!do_simd) { + chacha_crypt_generic(state, walk->dst.virt.addr, + walk->src.virt.addr, nbytes, + ctx->nrounds); + } else { kernel_fpu_begin(); - next_yield = 4096; + chacha_dosimd(state, walk->dst.virt.addr, + walk->src.virt.addr, nbytes, + ctx->nrounds); + kernel_fpu_end(); } - err = skcipher_walk_done(walk, walk->nbytes - nbytes); } @@ -164,19 +164,9 @@ static int chacha_simd(struct skcipher_request *req) struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct chacha_ctx *ctx = crypto_skcipher_ctx(tfm); struct skcipher_walk walk; - int err; - if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable()) - return crypto_chacha_crypt(req); - - err = skcipher_walk_virt(&walk, req, true); - if (err) - return err; - - kernel_fpu_begin(); - err = chacha_simd_stream_xor(&walk, ctx, req->iv); - kernel_fpu_end(); - return err; + return skcipher_walk_virt(&walk, req, true) ?: + chacha_simd_stream_xor(&walk, ctx, req->iv); } static int xchacha_simd(struct skcipher_request *req) @@ -189,31 +179,42 @@ static int xchacha_simd(struct skcipher_request *req) u8 real_iv[16]; int err; - if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable()) - return crypto_xchacha_crypt(req); - err = skcipher_walk_virt(&walk, req, true); if (err) return err; BUILD_BUG_ON(CHACHA_STATE_ALIGN != 16); state = PTR_ALIGN(state_buf + 0, CHACHA_STATE_ALIGN); - crypto_chacha_init(state, ctx, req->iv); - - kernel_fpu_begin(); - - hchacha_block_ssse3(state, subctx.key, ctx->nrounds); + chacha_init_generic(state, ctx->key, req->iv); + + if (req->cryptlen > CHACHA_BLOCK_SIZE && crypto_simd_usable()) { + kernel_fpu_begin(); + hchacha_block_ssse3(state, subctx.key, ctx->nrounds); + kernel_fpu_end(); + } else { + hchacha_block_generic(state, subctx.key, ctx->nrounds); + } subctx.nrounds = ctx->nrounds; memcpy(&real_iv[0], req->iv + 24, 8); memcpy(&real_iv[8], req->iv + 16, 8); err = chacha_simd_stream_xor(&walk, &subctx, real_iv); - kernel_fpu_end(); - return err; } +static int chacha20_setkey(struct crypto_skcipher *tfm, const u8 *key, + unsigned int keysize) +{ + return chacha_setkey(tfm, key, keysize, 20); +} + +static int chacha12_setkey(struct crypto_skcipher *tfm, const u8 *key, + unsigned int keysize) +{ + return chacha_setkey(tfm, key, keysize, 12); +} + static struct skcipher_alg algs[] = { { .base.cra_name = "chacha20", @@ -227,7 +228,7 @@ static struct skcipher_alg algs[] = { .max_keysize = CHACHA_KEY_SIZE, .ivsize = CHACHA_IV_SIZE, .chunksize = CHACHA_BLOCK_SIZE, - .setkey = crypto_chacha20_setkey, + .setkey = chacha20_setkey, .encrypt = chacha_simd, .decrypt = chacha_simd, }, { @@ -242,7 +243,7 @@ static struct skcipher_alg algs[] = { .max_keysize = CHACHA_KEY_SIZE, .ivsize = XCHACHA_IV_SIZE, .chunksize = CHACHA_BLOCK_SIZE, - .setkey = crypto_chacha20_setkey, + .setkey = chacha20_setkey, .encrypt = xchacha_simd, .decrypt = xchacha_simd, }, { @@ -257,7 +258,7 @@ static struct skcipher_alg algs[] = { .max_keysize = CHACHA_KEY_SIZE, .ivsize = XCHACHA_IV_SIZE, .chunksize = CHACHA_BLOCK_SIZE, - .setkey = crypto_chacha12_setkey, + .setkey = chacha12_setkey, .encrypt = xchacha_simd, .decrypt = xchacha_simd, }, diff --git a/crypto/Kconfig b/crypto/Kconfig index b39ca79ef65f..86732709b171 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -1439,7 +1439,7 @@ config CRYPTO_CHACHA20_X86_64 tristate "ChaCha stream cipher algorithms (x86_64/SSSE3/AVX2/AVX-512VL)" depends on X86 && 64BIT select CRYPTO_BLKCIPHER - select CRYPTO_CHACHA20 + select CRYPTO_LIB_CHACHA_GENERIC help SSSE3, AVX2, and AVX-512VL optimized implementations of the ChaCha20, XChaCha20, and XChaCha12 stream ciphers.