From patchwork Sat Jan 28 23:33:32 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 92779 Delivered-To: patch@linaro.org Received: by 10.140.20.99 with SMTP id 90csp849301qgi; Sat, 28 Jan 2017 15:36:42 -0800 (PST) X-Received: by 10.98.16.201 with SMTP id 70mr16103463pfq.25.1485646602746; Sat, 28 Jan 2017 15:36:42 -0800 (PST) Return-Path: Received: from bombadil.infradead.org (bombadil.infradead.org. [65.50.211.133]) by mx.google.com with ESMTPS id c2si4291084pgf.100.2017.01.28.15.36.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 Jan 2017 15:36:42 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org designates 65.50.211.133 as permitted sender) client-ip=65.50.211.133; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org designates 65.50.211.133 as permitted sender) smtp.mailfrom=linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1cXcXw-0000JQ-ST; Sat, 28 Jan 2017 23:36:40 +0000 Received: from mail-wm0-f42.google.com ([74.125.82.42]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1cXcWg-0007uA-Sx for linux-arm-kernel@lists.infradead.org; Sat, 28 Jan 2017 23:35:25 +0000 Received: by mail-wm0-f42.google.com with SMTP id r126so35530630wmr.1 for ; Sat, 28 Jan 2017 15:35:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Z8w+gjCOcNV2Gny4oVYVwnaR+rzw8N9Kamh/2dkIwTg=; b=RJ/VchGmuBjPrUsFHQc5JQfgJW5Fda3beVnt5iVe8yC4OuH46Fa8CYxGro6OHTNc7w FOIvFMCv3ZdMufeMFUw1NQkAfHuFYqouuhbAekeeo2FrnrznSjOY6YhPjMZBXVzlnWP1 VGc9gpPmuRtWA7vda7xoaC7ppZZaVAkLdEZZk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Z8w+gjCOcNV2Gny4oVYVwnaR+rzw8N9Kamh/2dkIwTg=; b=av6LTStyhHHzSvsT5ZI0hYxbsqEXmOeJkhf68+iaUc3brh2FdyB+vp75mFeAB/NiZj 2F9zYTljevoT5M6juHkyRGFwJBA6Vy/sHtVgWG3RQGauUtJNRmeaClN8stIM29h/m6rf a9JzN+qiudKyNlZf4SYckiO30BZgL5OXDQ/0pyQlahqNC1kFHu3VMoN8jH/dimczkEXu KtNW6MnG+3LsOL4A1HL1WvBXsluIqFfKKuFnRAHF4USayp5f28nzGUwxMS0Z61CnjdIh sKio8cNEm+hXfUL+ub2ZPe4xYsM6CfllY65okqUM4xv4j0Lb5GBlbny7SJBK5FbRhaOW Yayw== X-Gm-Message-State: AIkVDXJamkGmU6sFkoCzKKjLaF/KCoN99heu4F/nJ8fhn3aPIanwwQkyxUdmCwXIn3y/XDBa X-Received: by 10.223.153.135 with SMTP id y7mr12998585wrb.55.1485646445300; Sat, 28 Jan 2017 15:34:05 -0800 (PST) Received: from localhost.localdomain ([160.163.215.165]) by smtp.gmail.com with ESMTPSA id 40sm15141221wry.22.2017.01.28.15.34.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 28 Jan 2017 15:34:04 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Subject: [PATCH v2 3/4] crypto: arm64/aes - add NEON and Crypto Extension CBC-MAC driver Date: Sat, 28 Jan 2017 23:33:32 +0000 Message-Id: <1485646413-17491-4-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1485646413-17491-1-git-send-email-ard.biesheuvel@linaro.org> References: <1485646413-17491-1-git-send-email-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170128_153523_096817_4275DA64 X-CRM114-Status: GOOD ( 16.34 ) X-Spam-Score: -1.5 (-) X-Spam-Report: SpamAssassin version 3.4.1 on bombadil.infradead.org summary: Content analysis details: (-1.5 points) pts rule name description ---- ---------------------- -------------------------------------------------- 0.5 RCVD_IN_SORBS_SPAM RBL: SORBS: sender is a spam source [74.125.82.42 listed in dnsbl.sorbs.net] -0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3) [74.125.82.42 listed in wl.mailspike.net] -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [74.125.82.42 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org On ARMv8 implementations that do not support the Crypto Extensions, such as the Raspberry Pi 3, the CCM driver falls back to the generic table based AES implementation to perform the MAC part of the algorithm, which is slow and not time invariant. So add a CBCMAC implementation to the shared glue code between NEON AES and Crypto Extensions AES, so that it can be used instead now that the CCM driver has been updated to look for CBCMAC implementations other than the one it supplies itself. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-glue.c | 107 ++++++++++++++++++++ arch/arm64/crypto/aes-modes.S | 22 ++++ 2 files changed, 129 insertions(+) -- 2.7.4 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index 055bc3f61138..3265a6e79a4e 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -31,6 +32,7 @@ #define aes_ctr_encrypt ce_aes_ctr_encrypt #define aes_xts_encrypt ce_aes_xts_encrypt #define aes_xts_decrypt ce_aes_xts_decrypt +#define aes_cbcmac_update ce_aes_cbcmac_update MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 Crypto Extensions"); #else #define MODE "neon" @@ -44,11 +46,13 @@ MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 Crypto Extensions"); #define aes_ctr_encrypt neon_aes_ctr_encrypt #define aes_xts_encrypt neon_aes_xts_encrypt #define aes_xts_decrypt neon_aes_xts_decrypt +#define aes_cbcmac_update neon_aes_cbcmac_update MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 NEON"); MODULE_ALIAS_CRYPTO("ecb(aes)"); MODULE_ALIAS_CRYPTO("cbc(aes)"); MODULE_ALIAS_CRYPTO("ctr(aes)"); MODULE_ALIAS_CRYPTO("xts(aes)"); +MODULE_ALIAS_CRYPTO("cbcmac(aes)"); #endif MODULE_AUTHOR("Ard Biesheuvel "); @@ -75,11 +79,19 @@ asmlinkage void aes_xts_decrypt(u8 out[], u8 const in[], u8 const rk1[], int rounds, int blocks, u8 const rk2[], u8 iv[], int first); +asmlinkage void aes_cbcmac_update(u8 const in[], u32 const rk[], int rounds, + int blocks, u8 dg[], int skip_first_load); + struct crypto_aes_xts_ctx { struct crypto_aes_ctx key1; struct crypto_aes_ctx __aligned(8) key2; }; +struct cbcmac_desc_ctx { + unsigned int len; + u8 dg[AES_BLOCK_SIZE]; +}; + static int skcipher_aes_setkey(struct crypto_skcipher *tfm, const u8 *in_key, unsigned int key_len) { @@ -357,6 +369,94 @@ static struct skcipher_alg aes_algs[] = { { .decrypt = xts_decrypt, } }; +static int cbcmac_setkey(struct crypto_shash *tfm, + const u8 *in_key, unsigned int key_len) +{ + struct crypto_aes_ctx *ctx = crypto_shash_ctx(tfm); + int err; + + err = aes_expandkey(ctx, in_key, key_len); + if (err) + crypto_shash_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN); + + return err; +} + +static int cbcmac_init(struct shash_desc *desc) +{ + struct cbcmac_desc_ctx *ctx = shash_desc_ctx(desc); + + memset(ctx->dg, 0, AES_BLOCK_SIZE); + ctx->len = 0; + + return 0; +} + +static int cbcmac_update(struct shash_desc *desc, const u8 *p, + unsigned int len) +{ + struct crypto_aes_ctx *tctx = crypto_shash_ctx(desc->tfm); + struct cbcmac_desc_ctx *ctx = shash_desc_ctx(desc); + int rounds = 6 + tctx->key_length / 4; + int queued = (ctx->len != 0); + + while (ctx->len != 0 && len > 0) { + ctx->dg[ctx->len++] ^= *p++; + ctx->len %= AES_BLOCK_SIZE; + len--; + } + + if (len >= AES_BLOCK_SIZE || (!ctx->len && queued)) { + int blocks = len / AES_BLOCK_SIZE; + + kernel_neon_begin(); + aes_cbcmac_update(p, tctx->key_enc, rounds, blocks, ctx->dg, + queued); + kernel_neon_end(); + + p += blocks * AES_BLOCK_SIZE; + len %= AES_BLOCK_SIZE; + } + + while (len--) + ctx->dg[ctx->len++] ^= *p++; + + return 0; +} + +static int cbcmac_final(struct shash_desc *desc, u8 *out) +{ + struct crypto_aes_ctx *tctx = crypto_shash_ctx(desc->tfm); + struct cbcmac_desc_ctx *ctx = shash_desc_ctx(desc); + int rounds = 6 + tctx->key_length / 4; + + if (ctx->len) { + kernel_neon_begin(); + aes_cbcmac_update(NULL, tctx->key_enc, rounds, 0, ctx->dg, 1); + kernel_neon_end(); + } + memcpy(out, ctx->dg, AES_BLOCK_SIZE); + + return 0; +} + +static struct shash_alg cbcmac_alg = { + .base.cra_name = "cbcmac(aes)", + .base.cra_driver_name = "cbcmac-aes-" MODE, + .base.cra_priority = PRIO, + .base.cra_flags = CRYPTO_ALG_TYPE_SHASH, + .base.cra_blocksize = 1, + .base.cra_ctxsize = sizeof(struct crypto_aes_ctx), + .base.cra_module = THIS_MODULE, + + .digestsize = AES_BLOCK_SIZE, + .init = cbcmac_init, + .update = cbcmac_update, + .final = cbcmac_final, + .setkey = cbcmac_setkey, + .descsize = sizeof(struct cbcmac_desc_ctx), +}; + static struct simd_skcipher_alg *aes_simd_algs[ARRAY_SIZE(aes_algs)]; static void aes_exit(void) @@ -367,6 +467,7 @@ static void aes_exit(void) if (aes_simd_algs[i]) simd_skcipher_free(aes_simd_algs[i]); + crypto_unregister_shash(&cbcmac_alg); crypto_unregister_skciphers(aes_algs, ARRAY_SIZE(aes_algs)); } @@ -383,6 +484,10 @@ static int __init aes_init(void) if (err) return err; + err = crypto_register_shash(&cbcmac_alg); + if (err) + goto unregister_ciphers; + for (i = 0; i < ARRAY_SIZE(aes_algs); i++) { if (!(aes_algs[i].base.cra_flags & CRYPTO_ALG_INTERNAL)) continue; @@ -402,6 +507,8 @@ static int __init aes_init(void) unregister_simds: aes_exit(); +unregister_ciphers: + crypto_unregister_skciphers(aes_algs, ARRAY_SIZE(aes_algs)); return err; } diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index 92b982a8b112..f2bd43451885 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -525,3 +525,25 @@ AES_ENTRY(aes_xts_decrypt) FRAME_POP ret AES_ENDPROC(aes_xts_decrypt) + + /* + * aes_cbcmac_update(u8 const in[], u32 const rk[], int rounds, + * int blocks, u8 dg[], int skip_first_load) + */ +AES_ENTRY(aes_cbcmac_update) + ld1 {v0.16b}, [x4] /* get dg */ + enc_prepare w2, x1, x6 + cbnz w5, .Lcbcmacenc + +.Lcbcmacloop: + ld1 {v1.16b}, [x0], #16 /* get next pt block */ + eor v0.16b, v0.16b, v1.16b /* ..and xor with dg */ + sub w3, w3, #1 + +.Lcbcmacenc: + encrypt_block v0, w2, x1, x5, w6 + cbnz w3, .Lcbcmacloop + + st1 {v0.16b}, [x4] /* return dg */ + ret +AES_ENDPROC(aes_cbcmac_update)