From patchwork Sat Jun 10 16:22:48 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 103557 Delivered-To: patch@linaro.org Received: by 10.140.91.77 with SMTP id y71csp270805qgd; Sat, 10 Jun 2017 09:23:13 -0700 (PDT) X-Received: by 10.84.132.42 with SMTP id 39mr36686502ple.226.1497111793538; Sat, 10 Jun 2017 09:23:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1497111793; cv=none; d=google.com; s=arc-20160816; b=VzgnMs8yxOxura8RZbcWHBJ5aOfJwtLZKGDKvA0uUkSaE0nxvEAN6uE8ehzM/5oL4K vq+rtDncklFJ5EsuKVOxAvcwyowVYlCRqDrgBL3hmrohTEAPknb+dsNSM7xYmjIZjKhf i0yMeVuji4lc/8sVdru4eqBF+uCAg1slBysB+4DgFm1ta5GREmwSLKvYKsP4af5iW/0L KFeluXyK5ii+bW2aSi9Fr5QASN3RWZiukTLjOh62bal2pIIxp/OIVON5u24iZ9ySvHsf 2ktstp/J25OPommysTp4eKE74gmqYMmvHqx4SzGJY5NjdGjEwp8exa5zw9++C/8WUr78 bTUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=Y0U+ubMgLhP7VrOh5gl5x3F1FjQb0idG1cJqm/DPRl0=; b=l/w0jgUP7ieAMr0H0P+AVK3Vkc5Wg9pF8EqRvQ8iElgv9Hf83FWbX76xs033GpDUxt R0LDjBJniqYFvJBG/ZWjDyZbxXKy3AaE/HQofQxv5KBE0zEMd68d2nyux/uhxBLFAxj2 d6mS1hO2keSg1SQVR5ImVK1qywcxZWws+wJ9F5x68lLq7u2Ih1fbTqUgziK3b4QcEcrl hg7DVw5roNycY5OruF6O+OnPRalfKC+lsLsdN7mQm8INQC3x7KEv0iiKlDKx1lCPpkd4 gwO7x9vtwsbXGkqtuPlHKaGxDXf36o0RtGL2IXChglA8kPOh4g88exyluD3hyCJgNcxG G3eQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m11si9340618pgc.389.2017.06.10.09.23.13; Sat, 10 Jun 2017 09:23:13 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752119AbdFJQXM (ORCPT + 1 other); Sat, 10 Jun 2017 12:23:12 -0400 Received: from mail-wr0-f170.google.com ([209.85.128.170]:34061 "EHLO mail-wr0-f170.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752070AbdFJQXL (ORCPT ); Sat, 10 Jun 2017 12:23:11 -0400 Received: by mail-wr0-f170.google.com with SMTP id g76so58539401wrd.1 for ; Sat, 10 Jun 2017 09:23:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Y0U+ubMgLhP7VrOh5gl5x3F1FjQb0idG1cJqm/DPRl0=; b=I26IOelNNZD+D9+lNhMxCOEWemKZ/akXRIsnx3xSfc+LgxHRaTo4GLwxISBa0Epf4k 6EkXuBFwqFQXuog/nf2So4I/IqcdUUBu22nayWQ/Zugdd/JxOS9Nqcdcp/sR+/9Q1P/t D60kcttA8DMVM5m9fz6X7IsrN66uJfJUOLn3A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Y0U+ubMgLhP7VrOh5gl5x3F1FjQb0idG1cJqm/DPRl0=; b=RrRZ67t5fBgnAvDrgodmJB5kiLNj1ZlnIVUSxrwK349YPSblOV4q50UCnSeaDVVQZS 9ZaFoOB9IS59RMwUhM3NayOjJ6zw5FeXCIPW5K20/Oi+ZGJGyXxgl7avGlxZHe6P2xLP pczaliXewCuVeTan0/J674WHoUMWzuaE1RVLQj8QMKiG0zSHVNJLHxNtHf2UPwtt1BUn S0V5U/ioM2KqaRI0avqSDP7uZ8d8YfcHeFC4MAPnld+mn+tWzryAvRGBcDuqgv1w3lwZ oQocCy1MZLUAJFq8CBXK9qTEyMVJKV+xwvkdubE9Jfhed/e7cEAYmBMDBOPbiJzaPbkq jxjA== X-Gm-Message-State: AKS2vOzbMS247bTQ+Sh/4g+PL2yQw0vawthtt8L+aDUmcOSd1DKfOypP 2aMH9zVfSyhuWmI4ecSc2w== X-Received: by 10.28.172.69 with SMTP id v66mr3212818wme.64.1497111789334; Sat, 10 Jun 2017 09:23:09 -0700 (PDT) Received: from localhost.localdomain ([160.165.120.116]) by smtp.gmail.com with ESMTPSA id k35sm4440181wre.9.2017.06.10.09.23.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Jun 2017 09:23:08 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will.deacon@arm.com, dave.martin@arm.com Cc: Ard Biesheuvel Subject: [PATCH 02/12] crypto: arm64/ghash-ce - add non-SIMD scalar fallback Date: Sat, 10 Jun 2017 16:22:48 +0000 Message-Id: <1497111778-4210-3-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1497111778-4210-1-git-send-email-ard.biesheuvel@linaro.org> References: <1497111778-4210-1-git-send-email-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The arm64 kernel will shortly disallow nested kernel mode NEON, so add a fallback to scalar C code that can be invoked in that case. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/Kconfig | 3 +- arch/arm64/crypto/ghash-ce-glue.c | 49 ++++++++++++++++---- 2 files changed, 43 insertions(+), 9 deletions(-) -- 2.7.4 diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index d92293747d63..7d75a363e317 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -28,8 +28,9 @@ config CRYPTO_SHA2_ARM64_CE config CRYPTO_GHASH_ARM64_CE tristate "GHASH (for GCM chaining mode) using ARMv8 Crypto Extensions" - depends on ARM64 && KERNEL_MODE_NEON + depends on KERNEL_MODE_NEON select CRYPTO_HASH + select CRYPTO_GF128MUL config CRYPTO_CRCT10DIF_ARM64_CE tristate "CRCT10DIF digest algorithm using PMULL instructions" diff --git a/arch/arm64/crypto/ghash-ce-glue.c b/arch/arm64/crypto/ghash-ce-glue.c index 833ec1e3f3e9..3e1a778b181a 100644 --- a/arch/arm64/crypto/ghash-ce-glue.c +++ b/arch/arm64/crypto/ghash-ce-glue.c @@ -1,7 +1,7 @@ /* * Accelerated GHASH implementation with ARMv8 PMULL instructions. * - * Copyright (C) 2014 Linaro Ltd. + * Copyright (C) 2014 - 2017 Linaro Ltd. * * This program is free software; you can redistribute it and/or modify it * under the terms of the GNU General Public License version 2 as published @@ -9,7 +9,9 @@ */ #include +#include #include +#include #include #include #include @@ -25,6 +27,7 @@ MODULE_LICENSE("GPL v2"); struct ghash_key { u64 a; u64 b; + be128 k; }; struct ghash_desc_ctx { @@ -44,6 +47,36 @@ static int ghash_init(struct shash_desc *desc) return 0; } +static void ghash_do_update(int blocks, u64 dg[], const char *src, + struct ghash_key *key, const char *head) +{ + if (may_use_simd()) { + kernel_neon_begin(); + pmull_ghash_update(blocks, dg, src, key, head); + kernel_neon_end(); + } else { + be128 dst = { cpu_to_be64(dg[1]), cpu_to_be64(dg[0]) }; + + do { + const u8 *in = src; + + if (head) { + in = head; + blocks++; + head = NULL; + } else { + src += GHASH_BLOCK_SIZE; + } + + crypto_xor((u8 *)&dst, in, GHASH_BLOCK_SIZE); + gf128mul_lle(&dst, &key->k); + } while (--blocks); + + dg[0] = be64_to_cpu(dst.b); + dg[1] = be64_to_cpu(dst.a); + } +} + static int ghash_update(struct shash_desc *desc, const u8 *src, unsigned int len) { @@ -67,10 +100,9 @@ static int ghash_update(struct shash_desc *desc, const u8 *src, blocks = len / GHASH_BLOCK_SIZE; len %= GHASH_BLOCK_SIZE; - kernel_neon_begin_partial(8); - pmull_ghash_update(blocks, ctx->digest, src, key, - partial ? ctx->buf : NULL); - kernel_neon_end(); + ghash_do_update(blocks, ctx->digest, src, key, + partial ? ctx->buf : NULL); + src += blocks * GHASH_BLOCK_SIZE; partial = 0; } @@ -89,9 +121,7 @@ static int ghash_final(struct shash_desc *desc, u8 *dst) memset(ctx->buf + partial, 0, GHASH_BLOCK_SIZE - partial); - kernel_neon_begin_partial(8); - pmull_ghash_update(1, ctx->digest, ctx->buf, key, NULL); - kernel_neon_end(); + ghash_do_update(1, ctx->digest, ctx->buf, key, NULL); } put_unaligned_be64(ctx->digest[1], dst); put_unaligned_be64(ctx->digest[0], dst + 8); @@ -111,6 +141,9 @@ static int ghash_setkey(struct crypto_shash *tfm, return -EINVAL; } + /* needed for the fallback */ + memcpy(&key->k, inkey, GHASH_BLOCK_SIZE); + /* perform multiplication by 'x' in GF(2^128) */ b = get_unaligned_be64(inkey); a = get_unaligned_be64(inkey + 8);