From patchwork Tue Jul 2 19:41:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 168368 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp4653666ilk; Tue, 2 Jul 2019 12:42:43 -0700 (PDT) X-Google-Smtp-Source: APXvYqxc69pbpLeQZxHjS2Oou85uLddMlChouRyThO0pCG5F2mvr365fmd4qJ1QminyI3JW8IcmS X-Received: by 2002:a65:41c6:: with SMTP id b6mr2244964pgq.269.1562096563060; Tue, 02 Jul 2019 12:42:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562096563; cv=none; d=google.com; s=arc-20160816; b=G7JzxEdI99cYZH5Ycm9OJhpc0LE9GN/kRrsuJLuMG+KGUhhW5CHd/cqcoK27Y9m42o wEBuHRlgyKvrJ6E28n/53GgJPwSN3OgAaXV21spbS8okLt0GI/yItsZAe1uw5ydhh3Fb 194jlGROgaxlVKDC4M6DZuZef4bpbVWOfp0LOTze4bFAu4lmw/q4GTtjbns+1aAm/MMZ JTUOoYWyHwlrTVW9dgkBGjGYRXUTtgeXp/O1YesMm8lYb18HzUIBMpa1xGXdnM/H2+z3 EZJ6Lb/TI/4TQ8PNxsgCTixOAGRkED/lPSg1U+LbIYg1S1ooCs0+zxdcjqh5Uxhc+ppX vlOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=VA1NX4OTJyuJ+sHjTv3EgQQCR56i+Qi8C4n3+5icP3I=; b=MjrBoVe/EqlpMtn9VNXnRqeFMc5qp7ryf63XOrWn9UPmXzTT5bURIcNQkxVHwuxHIm 9G4Ycu0fkwIRwc/0BjL3MYiCb/bRviUqG/u6DJXmr4vF9yiCXI6WSx/Dpa0K8O2rElfT o3Fak8hgPsvc+CzSHJxEIXXs4SGuHSn1F6MGXANWyStErIJd0+K2ZmNOTA5ZKEfX+gt9 3/Jq9z/iPy/J6V/JtX9791H3ix/dPygS8jNUcB2y/ubVh4HzwC4PdEB3ARWTTE15sYqL 0UnbRmsrjOjcRQsMvahVLVS2rkBYJTQTaE1bdqxd6OR5YQvWd0Z743uB9WH17LnDgrlv hQcA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=gubS8BLH; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x13si13373092pgh.116.2019.07.02.12.42.42; Tue, 02 Jul 2019 12:42:43 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=gubS8BLH; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727092AbfGBTmm (ORCPT + 3 others); Tue, 2 Jul 2019 15:42:42 -0400 Received: from mail-lj1-f196.google.com ([209.85.208.196]:40428 "EHLO mail-lj1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727069AbfGBTmm (ORCPT ); Tue, 2 Jul 2019 15:42:42 -0400 Received: by mail-lj1-f196.google.com with SMTP id a21so18169330ljh.7 for ; Tue, 02 Jul 2019 12:42:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=VA1NX4OTJyuJ+sHjTv3EgQQCR56i+Qi8C4n3+5icP3I=; b=gubS8BLHsB5NbC39fRYRNOlmUqVxLMuCXolIY+O+z7WZMFn4CibcpVlUlw0snTj7hC S1MHCspjJWaQXoACWaw1T4lBLRAJyvLX55lbQcIwcU0Ji1DMdw32MZ5u3L4APxZICP5g bWQ2CklluUVYKdYfJTOWou+0m7mp/aM38Fs29RdRaooiO2geHajp2L7lIVdKXlIbnAg5 sc9XlEZtoLMcsSi3QfBSYo7cUNvGn5PFmB6Tu1eb+TLp1Fl5Y2T/kyIcFpkQYC6Pxkjv MMBA6H8Ebzxxj1uqw7SBZnDjXmYmsYF6eSSnJPMzA/UOl65abwHf8fjQ+Gu2Xjbx/Ac9 X5UA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=VA1NX4OTJyuJ+sHjTv3EgQQCR56i+Qi8C4n3+5icP3I=; b=YjCHgmK2C1ePflwAxnfBw8aD0wDrUIpvT/wNbb/Kgbsr+SXC77n7yfIs6uYxjie//F rlcux5rzH8JM1ZDo/RB7cKutbiDpOQW/EIwtkWjYIKuvnn3Q6ayHbsk54032smkk7LWr 5Ph+aNzVWalYZ8UGIunWFkO+CirjnO4Y5iuwaL2pjO24lYC06mM8aOxVMxHLAbUk+wcN bLF/1eUo6KbHPVkfLhz+s+jvBBRiYS1TjtUm3W5GL0O0TXU+LGiFXXt6DBmlrL5iZevm MEuWfwT6O6N7vsop7J9rFtH8sgmgUwOkaGrb7d4MWas5EM82bk1aLtbNqFTm1hMexhTW otRA== X-Gm-Message-State: APjAAAXpZZcoKrm6X8sGCObcemXVpuGfLck8MpR/lpRU/0Bsjx+RUjZU 4mROwLKTK16vhan8LCdiLUgWsXYusvA++b1D X-Received: by 2002:a2e:5d46:: with SMTP id r67mr17553560ljb.187.1562096559561; Tue, 02 Jul 2019 12:42:39 -0700 (PDT) Received: from e111045-lin.arm.com (89-212-78-239.static.t-2.net. [89.212.78.239]) by smtp.gmail.com with ESMTPSA id 24sm4475163ljs.63.2019.07.02.12.42.38 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Tue, 02 Jul 2019 12:42:38 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v4 22/32] crypto: arm/ghash - provide a synchronous version Date: Tue, 2 Jul 2019 21:41:40 +0200 Message-Id: <20190702194150.10405-23-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190702194150.10405-1-ard.biesheuvel@linaro.org> References: <20190702194150.10405-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org GHASH is used by the GCM mode, which is often used in contexts where only synchronous ciphers are permitted. So provide a synchronous version of GHASH based on the existing code. This requires a non-SIMD fallback to deal with invocations occurring from a context where SIMD instructions may not be used. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/ghash-ce-glue.c | 78 +++++++++++++------- 1 file changed, 52 insertions(+), 26 deletions(-) -- 2.17.1 diff --git a/arch/arm/crypto/ghash-ce-glue.c b/arch/arm/crypto/ghash-ce-glue.c index 39d1ccec1aab..ebb237ca874b 100644 --- a/arch/arm/crypto/ghash-ce-glue.c +++ b/arch/arm/crypto/ghash-ce-glue.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -33,6 +34,8 @@ struct ghash_key { u64 h2[2]; u64 h3[2]; u64 h4[2]; + + be128 k; }; struct ghash_desc_ctx { @@ -65,6 +68,36 @@ static int ghash_init(struct shash_desc *desc) return 0; } +static void ghash_do_update(int blocks, u64 dg[], const char *src, + struct ghash_key *key, const char *head) +{ + if (likely(crypto_simd_usable())) { + kernel_neon_begin(); + pmull_ghash_update(blocks, dg, src, key, head); + kernel_neon_end(); + } else { + be128 dst = { cpu_to_be64(dg[1]), cpu_to_be64(dg[0]) }; + + do { + const u8 *in = src; + + if (head) { + in = head; + blocks++; + head = NULL; + } else { + src += GHASH_BLOCK_SIZE; + } + + crypto_xor((u8 *)&dst, in, GHASH_BLOCK_SIZE); + gf128mul_lle(&dst, &key->k); + } while (--blocks); + + dg[0] = be64_to_cpu(dst.b); + dg[1] = be64_to_cpu(dst.a); + } +} + static int ghash_update(struct shash_desc *desc, const u8 *src, unsigned int len) { @@ -88,10 +121,8 @@ static int ghash_update(struct shash_desc *desc, const u8 *src, blocks = len / GHASH_BLOCK_SIZE; len %= GHASH_BLOCK_SIZE; - kernel_neon_begin(); - pmull_ghash_update(blocks, ctx->digest, src, key, - partial ? ctx->buf : NULL); - kernel_neon_end(); + ghash_do_update(blocks, ctx->digest, src, key, + partial ? ctx->buf : NULL); src += blocks * GHASH_BLOCK_SIZE; partial = 0; } @@ -109,9 +140,7 @@ static int ghash_final(struct shash_desc *desc, u8 *dst) struct ghash_key *key = crypto_shash_ctx(desc->tfm); memset(ctx->buf + partial, 0, GHASH_BLOCK_SIZE - partial); - kernel_neon_begin(); - pmull_ghash_update(1, ctx->digest, ctx->buf, key, NULL); - kernel_neon_end(); + ghash_do_update(1, ctx->digest, ctx->buf, key, NULL); } put_unaligned_be64(ctx->digest[1], dst); put_unaligned_be64(ctx->digest[0], dst + 8); @@ -135,24 +164,25 @@ static int ghash_setkey(struct crypto_shash *tfm, const u8 *inkey, unsigned int keylen) { struct ghash_key *key = crypto_shash_ctx(tfm); - be128 h, k; + be128 h; if (keylen != GHASH_BLOCK_SIZE) { crypto_shash_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN); return -EINVAL; } - memcpy(&k, inkey, GHASH_BLOCK_SIZE); - ghash_reflect(key->h, &k); + /* needed for the fallback */ + memcpy(&key->k, inkey, GHASH_BLOCK_SIZE); + ghash_reflect(key->h, &key->k); - h = k; - gf128mul_lle(&h, &k); + h = key->k; + gf128mul_lle(&h, &key->k); ghash_reflect(key->h2, &h); - gf128mul_lle(&h, &k); + gf128mul_lle(&h, &key->k); ghash_reflect(key->h3, &h); - gf128mul_lle(&h, &k); + gf128mul_lle(&h, &key->k); ghash_reflect(key->h4, &h); return 0; @@ -165,15 +195,13 @@ static struct shash_alg ghash_alg = { .final = ghash_final, .setkey = ghash_setkey, .descsize = sizeof(struct ghash_desc_ctx), - .base = { - .cra_name = "__ghash", - .cra_driver_name = "__driver-ghash-ce", - .cra_priority = 0, - .cra_flags = CRYPTO_ALG_INTERNAL, - .cra_blocksize = GHASH_BLOCK_SIZE, - .cra_ctxsize = sizeof(struct ghash_key), - .cra_module = THIS_MODULE, - }, + + .base.cra_name = "ghash", + .base.cra_driver_name = "ghash-ce-sync", + .base.cra_priority = 300 - 1, + .base.cra_blocksize = GHASH_BLOCK_SIZE, + .base.cra_ctxsize = sizeof(struct ghash_key), + .base.cra_module = THIS_MODULE, }; static int ghash_async_init(struct ahash_request *req) @@ -288,9 +316,7 @@ static int ghash_async_init_tfm(struct crypto_tfm *tfm) struct cryptd_ahash *cryptd_tfm; struct ghash_async_ctx *ctx = crypto_tfm_ctx(tfm); - cryptd_tfm = cryptd_alloc_ahash("__driver-ghash-ce", - CRYPTO_ALG_INTERNAL, - CRYPTO_ALG_INTERNAL); + cryptd_tfm = cryptd_alloc_ahash("ghash-ce-sync", 0, 0); if (IS_ERR(cryptd_tfm)) return PTR_ERR(cryptd_tfm); ctx->cryptd_tfm = cryptd_tfm;