From patchwork Mon Jul 24 10:28:11 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 108556 Delivered-To: patch@linaro.org Received: by 10.140.101.44 with SMTP id t41csp3887557qge; Mon, 24 Jul 2017 03:28:50 -0700 (PDT) X-Received: by 10.99.168.5 with SMTP id o5mr15422375pgf.207.1500892129973; Mon, 24 Jul 2017 03:28:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1500892129; cv=none; d=google.com; s=arc-20160816; b=OpFPYPeFm5kBQa4UvNZPthpIPVFcJWuZTdqCFlsslSN6Tu3e6KrG5lEUvsLaTmrJoV 0MNBy3lpvRvH0XvLRUItu8SkxyZqWtkA8iPNkYnCsdXAI/4XTw63lJ0R64docbbd/tp3 1EDX+oU5KxtsKfuHOZkA2TDhUewmkNl5OsdgvAnP2u0AgJlg9bMzRBS170aaZU9ps0AQ wegoNxNGGXdcPpsyOxbJ6DrDy2iLRv3enX+zKdg16dh3mP1Tj9ZrtsoSRA8qffd7yF07 VozsKqzyCo1/yxywpsH5+YLpDIUEUbIdKyFc7UXdNDB6IdWg24lfoKgVMxRncExWdJKb AeeA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=Q8qNtVVM7A1mNOWXz5wWpoWXHkKfLWT1lj0d0U7Z318=; b=paDAOAZrHmfql/mjQgoCbQdI+MUmp5f3NzraYrLfaSo99igjW9LcPM1MBu77wRq+UZ +OKfn9PxBHdYwLzgaKlnrEAdV40B5nUu4YcHSE0uKFiI+qkeih9RnmC1xhENp8PIWa57 Wm0Ffap/7p1Afw6to+ulXV2O67vDePoDT18G9N/6VRK5x4mOz8SIYiZdciG/Lgs4RlX1 4tH5CHwzOWfOcWZ6iOn/06E2JnMvK+m/2sWKC0KHlwL2p9gi6csbdvxfu0kyae++eK+Y u0o+QCsw+oKKpCaJJ5BNe74kZ+b6jZZ1j/OFLdNylY2uO8IWnj4IRKYNeFoEtPT7+Ihu 325A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.b=FEot+7Al; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 38si3348411pld.76.2017.07.24.03.28.49; Mon, 24 Jul 2017 03:28:49 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.b=FEot+7Al; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752441AbdGXK2t (ORCPT + 1 other); Mon, 24 Jul 2017 06:28:49 -0400 Received: from mail-wr0-f177.google.com ([209.85.128.177]:36470 "EHLO mail-wr0-f177.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751766AbdGXK2s (ORCPT ); Mon, 24 Jul 2017 06:28:48 -0400 Received: by mail-wr0-f177.google.com with SMTP id y43so104225090wrd.3 for ; Mon, 24 Jul 2017 03:28:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Q8qNtVVM7A1mNOWXz5wWpoWXHkKfLWT1lj0d0U7Z318=; b=FEot+7AlgtJrDLKye8MZpeMRppmRChRX7Az5sMyqWCLPf9bsEdAY4K10EkrNZHv194 yUQ6ls6NrmX2fMB1b6MjIEsOhPUxzN1khafxikXLdjb+0pBGJz/i5F6mRzjHiuavdSXr I5B6BiY4odggSLLiXwBsKNYgB1A2z97fU4OIM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Q8qNtVVM7A1mNOWXz5wWpoWXHkKfLWT1lj0d0U7Z318=; b=GCvhg+/HggKDfSWZkVjYUfVrOWufT6MGvGOj/mwDaMdfwlku9ZYOph87ajhKsyqDal TOdWhvlDWplxOn7gI2/i2+pxYF9Q8JqtZZt8zmVqaWrHTiMGCF3CwDpjuQOBJfbC26B8 NvEnv+cTmItEKDa1M8jVRRlvYPQ8xmKdmUfqLyQgTQ5i+isfqMa+RvS4xwZMiWc+CJVA Q4pXiLwRuJyWjux26twGITQHjTFhUEjBZxjgF1QKs0/bbHh0wnagOL61HfWHxMWFO/vj /OzZ7JGQAwIk5+j5k5Tya5Op3NNgkEhK/zUpBE3ZiEBet3gB1q4pBn3GX/iEu+EEqeNj fc4w== X-Gm-Message-State: AIVw1103jkywfVVzM+pIaHXHkjZF5BcPSOuGjhvbxUuIwVuAg9GCMq0u /Zb7U8LCn1zmA/VrH7ji+g== X-Received: by 10.223.166.173 with SMTP id t42mr12044646wrc.272.1500892126980; Mon, 24 Jul 2017 03:28:46 -0700 (PDT) Received: from localhost.localdomain ([105.148.195.69]) by smtp.gmail.com with ESMTPSA id v44sm13205400wrb.53.2017.07.24.03.28.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 24 Jul 2017 03:28:46 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: herbert@gondor.apana.org.au, dave.martin@arm.com, Ard Biesheuvel Subject: [PATCH resend 09/18] crypto: arm64/aes-ce-cipher: add non-SIMD generic fallback Date: Mon, 24 Jul 2017 11:28:11 +0100 Message-Id: <20170724102820.16534-10-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.9.3 In-Reply-To: <20170724102820.16534-1-ard.biesheuvel@linaro.org> References: <20170724102820.16534-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The arm64 kernel will shortly disallow nested kernel mode NEON, so add a fallback to scalar code that can be invoked in that case. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/Kconfig | 1 + arch/arm64/crypto/aes-ce-cipher.c | 20 +++++++++++++++++--- 2 files changed, 18 insertions(+), 3 deletions(-) -- 2.9.3 diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index 8cd145f9c1ff..2fd4bb6d0b5a 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -52,6 +52,7 @@ config CRYPTO_AES_ARM64_CE tristate "AES core cipher using ARMv8 Crypto Extensions" depends on ARM64 && KERNEL_MODE_NEON select CRYPTO_ALGAPI + select CRYPTO_AES_ARM64 config CRYPTO_AES_ARM64_CE_CCM tristate "AES in CCM mode using ARMv8 Crypto Extensions" diff --git a/arch/arm64/crypto/aes-ce-cipher.c b/arch/arm64/crypto/aes-ce-cipher.c index a0a0e5e3a8b5..6a75cd75ed11 100644 --- a/arch/arm64/crypto/aes-ce-cipher.c +++ b/arch/arm64/crypto/aes-ce-cipher.c @@ -9,6 +9,7 @@ */ #include +#include #include #include #include @@ -21,6 +22,9 @@ MODULE_DESCRIPTION("Synchronous AES cipher using ARMv8 Crypto Extensions"); MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); +asmlinkage void __aes_arm64_encrypt(u32 *rk, u8 *out, const u8 *in, int rounds); +asmlinkage void __aes_arm64_decrypt(u32 *rk, u8 *out, const u8 *in, int rounds); + struct aes_block { u8 b[AES_BLOCK_SIZE]; }; @@ -45,7 +49,12 @@ static void aes_cipher_encrypt(struct crypto_tfm *tfm, u8 dst[], u8 const src[]) void *dummy0; int dummy1; - kernel_neon_begin_partial(4); + if (!may_use_simd()) { + __aes_arm64_encrypt(ctx->key_enc, dst, src, num_rounds(ctx)); + return; + } + + kernel_neon_begin(); __asm__(" ld1 {v0.16b}, %[in] ;" " ld1 {v1.4s}, [%[key]], #16 ;" @@ -90,7 +99,12 @@ static void aes_cipher_decrypt(struct crypto_tfm *tfm, u8 dst[], u8 const src[]) void *dummy0; int dummy1; - kernel_neon_begin_partial(4); + if (!may_use_simd()) { + __aes_arm64_decrypt(ctx->key_dec, dst, src, num_rounds(ctx)); + return; + } + + kernel_neon_begin(); __asm__(" ld1 {v0.16b}, %[in] ;" " ld1 {v1.4s}, [%[key]], #16 ;" @@ -170,7 +184,7 @@ int ce_aes_expandkey(struct crypto_aes_ctx *ctx, const u8 *in_key, for (i = 0; i < kwords; i++) ctx->key_enc[i] = get_unaligned_le32(in_key + i * sizeof(u32)); - kernel_neon_begin_partial(2); + kernel_neon_begin(); for (i = 0; i < sizeof(rcon); i++) { u32 *rki = ctx->key_enc + (i * kwords); u32 *rko = rki + kwords;