From patchwork Thu Feb 2 15:56:28 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 93113 Delivered-To: patch@linaro.org Received: by 10.140.20.99 with SMTP id 90csp163603qgi; Thu, 2 Feb 2017 07:56:37 -0800 (PST) X-Received: by 10.98.113.9 with SMTP id m9mr11246077pfc.121.1486050997675; Thu, 02 Feb 2017 07:56:37 -0800 (PST) Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id u22si22661217plk.137.2017.02.02.07.56.37; Thu, 02 Feb 2017 07:56:37 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751201AbdBBP4g (ORCPT + 1 other); Thu, 2 Feb 2017 10:56:36 -0500 Received: from mail-wm0-f50.google.com ([74.125.82.50]:36584 "EHLO mail-wm0-f50.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751618AbdBBP4f (ORCPT ); Thu, 2 Feb 2017 10:56:35 -0500 Received: by mail-wm0-f50.google.com with SMTP id c85so96611472wmi.1 for ; Thu, 02 Feb 2017 07:56:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id; bh=nL+FNlR3NK87KhMtUXjpHhynjvRGn/dWdH5EQeW1U+8=; b=aCWdybDFV+AFmU7iCeA0VZgHJcstagPBwZ6EChe373voDJWBnX5Ku2sjD4gMfYBMnc SkHmhxvG8SyZ6ZthL+3mHEwSHmSS3vNNnxp9nTEbcIua7AYV6et/OK5ybCyyDtjALVq2 4Ya22nnFaWZtfgu164pi6rzVnjz7hpW6oZN7E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=nL+FNlR3NK87KhMtUXjpHhynjvRGn/dWdH5EQeW1U+8=; b=uoaXWT7qsVpYn/Bx789QvYKa+ni1e96Nhcxktg8huU/ApmTF8togmvNwayvKoKwqba dm19ioUfhbZOE/TznT+LgS92iOqPsetwa7NbqfvPgv/M//VZqJuLEcGkX6RBfnuWfkJQ mn7mxjEXab2fAlU3bGsw+x9f8NBH6AppOhe1kyl6GjStedI2+ts4GBqXHTa8fBcjdGic lIQsa4D98dTmzktHSd7wtOWm/pdzfQJKMsUwrNVsaY5uSN+dc/Muz/wMaIOYUIjogIu0 k/dyGhttyGGkMCIP9Cx5RTWy6qlM4HC7Q/mglx8lRPTDQ3CxDL/eW5YU/6rQcmSGwEjc nDoQ== X-Gm-Message-State: AIkVDXJo60NAI9zUM1iG+tvegyPnCoKJBKJYRO8xqua8cN7AOtdB52pTg/nt2OS6VWA9geiT X-Received: by 10.28.216.14 with SMTP id p14mr26945760wmg.56.1486050994006; Thu, 02 Feb 2017 07:56:34 -0800 (PST) Received: from localhost.localdomain ([105.130.17.13]) by smtp.gmail.com with ESMTPSA id w70sm40401961wrc.47.2017.02.02.07.56.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 02 Feb 2017 07:56:32 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers3@gmail.com Cc: Ard Biesheuvel Subject: [PATCH v2] crypto: algapi - make crypto_xor() and crypto_inc() alignment agnostic Date: Thu, 2 Feb 2017 15:56:28 +0000 Message-Id: <1486050988-20407-1-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.7.4 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Instead of unconditionally forcing 4 byte alignment for all generic chaining modes that rely on crypto_xor() or crypto_inc() (which may result in unnecessary copying of data when the underlying hardware can perform unaligned accesses efficiently), make those functions deal with unaligned input explicitly, but only if the Kconfig symbol HAVE_EFFICIENT_UNALIGNED_ACCESS is set. This will allow us to drop the alignmasks from the CBC, CMAC, CTR, CTS, PCBC and SEQIV drivers. For crypto_inc(), this simply involves making the 4-byte stride conditional on HAVE_EFFICIENT_UNALIGNED_ACCESS being set, given that it typically operates on 16 byte buffers. For crypto_xor(), an algorithm is implemented that simply runs through the input using the largest strides possible if unaligned accesses are allowed. If they are not, an optimal sequence of memory accesses is emitted that takes the relative alignment of the input buffers into account, e.g., if the relative misalignment of dst and src is 4 bytes, the entire xor operation will be completed using 4 byte loads and stores (modulo unaligned bits at the start and end). Note that all expressions involving misalign are simply eliminated by the compiler when HAVE_EFFICIENT_UNALIGNED_ACCESS is defined. Signed-off-by: Ard Biesheuvel --- I have greatly simplified the code, but it should still emit an optimal sequence of loads and stores depending on the misalignment. crypto/algapi.c | 65 +++++++++++++++----- crypto/cbc.c | 3 - crypto/cmac.c | 3 +- crypto/ctr.c | 2 +- crypto/cts.c | 3 - crypto/pcbc.c | 3 - crypto/seqiv.c | 2 - 7 files changed, 50 insertions(+), 31 deletions(-) -- 2.7.4 diff --git a/crypto/algapi.c b/crypto/algapi.c index df939b54b09f..e05ed1c5f5d1 100644 --- a/crypto/algapi.c +++ b/crypto/algapi.c @@ -961,32 +961,63 @@ void crypto_inc(u8 *a, unsigned int size) __be32 *b = (__be32 *)(a + size); u32 c; - for (; size >= 4; size -= 4) { - c = be32_to_cpu(*--b) + 1; - *b = cpu_to_be32(c); - if (c) - return; - } + if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) || + !((unsigned long)b & (__alignof__(*b) - 1))) + for (; size >= 4; size -= 4) { + c = be32_to_cpu(*--b) + 1; + *b = cpu_to_be32(c); + if (c) + return; + } crypto_inc_byte(a, size); } EXPORT_SYMBOL_GPL(crypto_inc); -static inline void crypto_xor_byte(u8 *a, const u8 *b, unsigned int size) +void crypto_xor(u8 *dst, const u8 *src, unsigned int len) { - for (; size; size--) - *a++ ^= *b++; -} + const int size = sizeof(unsigned long); + int delta = ((unsigned long)dst ^ (unsigned long)src) & (size - 1); + int misalign = 0; + + if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) && delta) { + misalign = 1 << __ffs(delta); + + /* + * If we care about alignment, process as many bytes as + * needed to advance dst and src to values whose alignments + * equal their relative misalignment. This will allow us to + * process the remainder of the input using optimal strides. + */ + while (((unsigned long)dst & (misalign - 1)) && len > 0) { + *dst++ ^= *src++; + len--; + } + } -void crypto_xor(u8 *dst, const u8 *src, unsigned int size) -{ - u32 *a = (u32 *)dst; - u32 *b = (u32 *)src; + while (len >= size && !misalign) { + *(unsigned long *)dst ^= *(unsigned long *)src; + dst += size; + src += size; + len -= size; + } - for (; size >= 4; size -= 4) - *a++ ^= *b++; + while (IS_ENABLED(CONFIG_64BIT) && len >= 4 && !(misalign & 3)) { + *(u32 *)dst ^= *(u32 *)src; + dst += 4; + src += 4; + len -= 4; + } + + while (len >= 2 && !(misalign & 1)) { + *(u16 *)dst ^= *(u16 *)src; + dst += 2; + src += 2; + len -= 2; + } - crypto_xor_byte((u8 *)a, (u8 *)b, size); + while (len--) + *dst++ ^= *src++; } EXPORT_SYMBOL_GPL(crypto_xor); diff --git a/crypto/cbc.c b/crypto/cbc.c index 68f751a41a84..bc160a3186dc 100644 --- a/crypto/cbc.c +++ b/crypto/cbc.c @@ -145,9 +145,6 @@ static int crypto_cbc_create(struct crypto_template *tmpl, struct rtattr **tb) inst->alg.base.cra_blocksize = alg->cra_blocksize; inst->alg.base.cra_alignmask = alg->cra_alignmask; - /* We access the data as u32s when xoring. */ - inst->alg.base.cra_alignmask |= __alignof__(u32) - 1; - inst->alg.ivsize = alg->cra_blocksize; inst->alg.min_keysize = alg->cra_cipher.cia_min_keysize; inst->alg.max_keysize = alg->cra_cipher.cia_max_keysize; diff --git a/crypto/cmac.c b/crypto/cmac.c index 04080dca8f0c..16301f52858c 100644 --- a/crypto/cmac.c +++ b/crypto/cmac.c @@ -260,8 +260,7 @@ static int cmac_create(struct crypto_template *tmpl, struct rtattr **tb) if (err) goto out_free_inst; - /* We access the data as u32s when xoring. */ - alignmask = alg->cra_alignmask | (__alignof__(u32) - 1); + alignmask = alg->cra_alignmask; inst->alg.base.cra_alignmask = alignmask; inst->alg.base.cra_priority = alg->cra_priority; inst->alg.base.cra_blocksize = alg->cra_blocksize; diff --git a/crypto/ctr.c b/crypto/ctr.c index a9a7a44f2783..a4f4a8983169 100644 --- a/crypto/ctr.c +++ b/crypto/ctr.c @@ -209,7 +209,7 @@ static struct crypto_instance *crypto_ctr_alloc(struct rtattr **tb) inst->alg.cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER; inst->alg.cra_priority = alg->cra_priority; inst->alg.cra_blocksize = 1; - inst->alg.cra_alignmask = alg->cra_alignmask | (__alignof__(u32) - 1); + inst->alg.cra_alignmask = alg->cra_alignmask; inst->alg.cra_type = &crypto_blkcipher_type; inst->alg.cra_blkcipher.ivsize = alg->cra_blocksize; diff --git a/crypto/cts.c b/crypto/cts.c index a1335d6c35fb..243f591dc409 100644 --- a/crypto/cts.c +++ b/crypto/cts.c @@ -374,9 +374,6 @@ static int crypto_cts_create(struct crypto_template *tmpl, struct rtattr **tb) inst->alg.base.cra_blocksize = alg->base.cra_blocksize; inst->alg.base.cra_alignmask = alg->base.cra_alignmask; - /* We access the data as u32s when xoring. */ - inst->alg.base.cra_alignmask |= __alignof__(u32) - 1; - inst->alg.ivsize = alg->base.cra_blocksize; inst->alg.chunksize = crypto_skcipher_alg_chunksize(alg); inst->alg.min_keysize = crypto_skcipher_alg_min_keysize(alg); diff --git a/crypto/pcbc.c b/crypto/pcbc.c index 11d248673ad4..29dd2b4a3b85 100644 --- a/crypto/pcbc.c +++ b/crypto/pcbc.c @@ -260,9 +260,6 @@ static int crypto_pcbc_create(struct crypto_template *tmpl, struct rtattr **tb) inst->alg.base.cra_blocksize = alg->cra_blocksize; inst->alg.base.cra_alignmask = alg->cra_alignmask; - /* We access the data as u32s when xoring. */ - inst->alg.base.cra_alignmask |= __alignof__(u32) - 1; - inst->alg.ivsize = alg->cra_blocksize; inst->alg.min_keysize = alg->cra_cipher.cia_min_keysize; inst->alg.max_keysize = alg->cra_cipher.cia_max_keysize; diff --git a/crypto/seqiv.c b/crypto/seqiv.c index c7049231861f..570b7d1aa0ca 100644 --- a/crypto/seqiv.c +++ b/crypto/seqiv.c @@ -153,8 +153,6 @@ static int seqiv_aead_create(struct crypto_template *tmpl, struct rtattr **tb) if (IS_ERR(inst)) return PTR_ERR(inst); - inst->alg.base.cra_alignmask |= __alignof__(u32) - 1; - spawn = aead_instance_ctx(inst); alg = crypto_spawn_aead_alg(spawn);