From patchwork Sat Jun 10 16:22:54 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 103563 Delivered-To: patch@linaro.org Received: by 10.140.91.77 with SMTP id y71csp270871qgd; Sat, 10 Jun 2017 09:23:26 -0700 (PDT) X-Received: by 10.84.238.139 with SMTP id v11mr47732906plk.182.1497111806335; Sat, 10 Jun 2017 09:23:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1497111806; cv=none; d=google.com; s=arc-20160816; b=z4Fl6Z8Rn2iYOflKEXmMTySJnUEKXFQ4lx1vgtk9bbHcXeQeCRRyLOUQ8b3+6qEYkt eSB0Tv5cYMUTXnz6KTh2b5/p8dBc7OsHIsn0Xxe3Vobu+CIOndlKdgpFpmcbEPEeIJbT Onc3TQA+yNq81U83/UwuFaTOJnbPpgMXucacPdM0utN7d+OVXhLoYskgTKSAwfbWfSCb A+y9UUIL7ACofLzpp5hYw99ZeoKA4I7CSbpCGR+ddUPV77qrOdVfoDF2wctiDeZD45cG J9LedYWZTkOUB2WxSv26x/D2t1K+DeoQV2YJGusiwuFdEbAVMVFMvp9H2akl2YRbZZVR Zu2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=Hp2doILegMbs6y4QwSEc7j8vbpgicAqKMc+INz1yHGs=; b=oU7X7RvszWg0m9yglJzehY7Pq5OrH2aYlSNTin7dT7nAZjK5E5eOPC34FR0wlFW3Nb oP7SGKjt/WrH7eXsDExzkeW4fYb2PGKxEIiJdgln1Knen4KnEa5tQPhFdxwx/XJexznd yxqt8F+tXiTqJtBO8lXamBxzY0m82iLnp2OaeKKMlv1aqzGAhejcpYX+i2Sy+NnfXi8s AbsKBKMYIfYq+f3+6FrqTYTbgStuIzqrqfBQpaSPj70OizXeUx+3vg2H+CKt9jzJLvn+ wz5sIzKjKK5dDkFihS3/B4kT821RX9/LFaoOi7FaDvfXgZTkiLnAlP78mRQJvRMyvNYR gF/Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m11si9340618pgc.389.2017.06.10.09.23.26; Sat, 10 Jun 2017 09:23:26 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752202AbdFJQXZ (ORCPT + 1 other); Sat, 10 Jun 2017 12:23:25 -0400 Received: from mail-wr0-f173.google.com ([209.85.128.173]:36064 "EHLO mail-wr0-f173.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752199AbdFJQXY (ORCPT ); Sat, 10 Jun 2017 12:23:24 -0400 Received: by mail-wr0-f173.google.com with SMTP id v111so58817631wrc.3 for ; Sat, 10 Jun 2017 09:23:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Hp2doILegMbs6y4QwSEc7j8vbpgicAqKMc+INz1yHGs=; b=VRSHiuaVDg4UINFM877ewuXI1TgsX7WBnis4EFq+sCJ87BBsk1WfeaIJ23ubgQHabM kD1EFv0PCflud2luGMqzkhRBHzCbBKsrkfPCYDcqtUXB77rkALRSwOUTVYku2aEGxKAg l+3psq7VL6ZMHxb+gcqdiMFGbf1xtstWPv/5E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Hp2doILegMbs6y4QwSEc7j8vbpgicAqKMc+INz1yHGs=; b=YRUyb+TPDvz0LgTvcn6VJNxrToIvzyQaXjxlkGC/HezXF5He2fTEAzB3FU+sSpxzag QXf6drkUA4SkRaK9ujbgJuT33cWLe0hYqt92z5/hUhdLSKVq3pc0uzs+QvzdnH3v1dcV M/X+eXfzV3vAMQD7JN0d4BEncR3tH2gDiOx8QLyek8e6EPlyzOeTOfLZWKKE+oe2hF5g Ao1B+C9xszPHKzzPIckF5EfKp8bGXtQUqG8psSHkhQF88nA9cOtA11E2Jz0klG7Flwx/ qsoPvEaC+O777iqrWx8T9xG0C7Iv/P1cqQ44kbli5tXo8tBYZMQJz2v0Maf5QPz/ti9H WdtQ== X-Gm-Message-State: AKS2vOzF+BB1UsuH4xHTE1icWO9anXu0sF9C5t3Tf+CZraDv09yZmi1r C1wS2kvvqmM8KliuHn26Pg== X-Received: by 10.28.47.138 with SMTP id v132mr3109018wmv.77.1497111803193; Sat, 10 Jun 2017 09:23:23 -0700 (PDT) Received: from localhost.localdomain ([160.165.120.116]) by smtp.gmail.com with ESMTPSA id k35sm4440181wre.9.2017.06.10.09.23.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Jun 2017 09:23:22 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will.deacon@arm.com, dave.martin@arm.com Cc: Ard Biesheuvel Subject: [PATCH 08/12] crypto: arm64/aes-ce-cipher: add non-SIMD generic fallback Date: Sat, 10 Jun 2017 16:22:54 +0000 Message-Id: <1497111778-4210-9-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1497111778-4210-1-git-send-email-ard.biesheuvel@linaro.org> References: <1497111778-4210-1-git-send-email-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The arm64 kernel will shortly disallow nested kernel mode NEON, so add a fallback to scalar code that can be invoked in that case. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/Kconfig | 3 ++- arch/arm64/crypto/aes-ce-cipher.c | 20 +++++++++++++++++--- 2 files changed, 19 insertions(+), 4 deletions(-) -- 2.7.4 diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index 8cd145f9c1ff..772801f263d9 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -50,8 +50,9 @@ config CRYPTO_AES_ARM64 config CRYPTO_AES_ARM64_CE tristate "AES core cipher using ARMv8 Crypto Extensions" - depends on ARM64 && KERNEL_MODE_NEON + depends on KERNEL_MODE_NEON select CRYPTO_ALGAPI + select CRYPTO_AES_ARM64 config CRYPTO_AES_ARM64_CE_CCM tristate "AES in CCM mode using ARMv8 Crypto Extensions" diff --git a/arch/arm64/crypto/aes-ce-cipher.c b/arch/arm64/crypto/aes-ce-cipher.c index a0a0e5e3a8b5..6a75cd75ed11 100644 --- a/arch/arm64/crypto/aes-ce-cipher.c +++ b/arch/arm64/crypto/aes-ce-cipher.c @@ -9,6 +9,7 @@ */ #include +#include #include #include #include @@ -21,6 +22,9 @@ MODULE_DESCRIPTION("Synchronous AES cipher using ARMv8 Crypto Extensions"); MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); +asmlinkage void __aes_arm64_encrypt(u32 *rk, u8 *out, const u8 *in, int rounds); +asmlinkage void __aes_arm64_decrypt(u32 *rk, u8 *out, const u8 *in, int rounds); + struct aes_block { u8 b[AES_BLOCK_SIZE]; }; @@ -45,7 +49,12 @@ static void aes_cipher_encrypt(struct crypto_tfm *tfm, u8 dst[], u8 const src[]) void *dummy0; int dummy1; - kernel_neon_begin_partial(4); + if (!may_use_simd()) { + __aes_arm64_encrypt(ctx->key_enc, dst, src, num_rounds(ctx)); + return; + } + + kernel_neon_begin(); __asm__(" ld1 {v0.16b}, %[in] ;" " ld1 {v1.4s}, [%[key]], #16 ;" @@ -90,7 +99,12 @@ static void aes_cipher_decrypt(struct crypto_tfm *tfm, u8 dst[], u8 const src[]) void *dummy0; int dummy1; - kernel_neon_begin_partial(4); + if (!may_use_simd()) { + __aes_arm64_decrypt(ctx->key_dec, dst, src, num_rounds(ctx)); + return; + } + + kernel_neon_begin(); __asm__(" ld1 {v0.16b}, %[in] ;" " ld1 {v1.4s}, [%[key]], #16 ;" @@ -170,7 +184,7 @@ int ce_aes_expandkey(struct crypto_aes_ctx *ctx, const u8 *in_key, for (i = 0; i < kwords; i++) ctx->key_enc[i] = get_unaligned_le32(in_key + i * sizeof(u32)); - kernel_neon_begin_partial(2); + kernel_neon_begin(); for (i = 0; i < sizeof(rcon); i++) { u32 *rki = ctx->key_enc + (i * kwords); u32 *rko = rki + kwords;