From patchwork Tue Jan 17 15:22:28 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 91673 Delivered-To: patch@linaro.org Received: by 10.140.20.99 with SMTP id 90csp542975qgi; Tue, 17 Jan 2017 07:23:21 -0800 (PST) X-Received: by 10.99.56.94 with SMTP id h30mr25061756pgn.23.1484666601336; Tue, 17 Jan 2017 07:23:21 -0800 (PST) Return-Path: Received: from bombadil.infradead.org (bombadil.infradead.org. [2001:1868:205::9]) by mx.google.com with ESMTPS id c10si25245507pfh.201.2017.01.17.07.23.21 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 17 Jan 2017 07:23:21 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org designates 2001:1868:205::9 as permitted sender) client-ip=2001:1868:205::9; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org designates 2001:1868:205::9 as permitted sender) smtp.mailfrom=linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1cTVbT-0002QZ-5g; Tue, 17 Jan 2017 15:23:19 +0000 Received: from mail-wm0-x22a.google.com ([2a00:1450:400c:c09::22a]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1cTVbG-00029q-P5 for linux-arm-kernel@lists.infradead.org; Tue, 17 Jan 2017 15:23:08 +0000 Received: by mail-wm0-x22a.google.com with SMTP id c85so204591844wmi.1 for ; Tue, 17 Jan 2017 07:22:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=CzjTHOxkTE4Qgd4iwVqP3xmJj/nHZVRS08KnoIo68+M=; b=XNolBKw5bnwS3b6IGC3vGoJYH4bIdg4SE62PMn3WIm+iBsbk7G8cwcQHy6QuHVkOO1 6O52cRxrRwZM7fsRY7B2RGQ+5VMMUtyuGng5aTCLY/RDHNLafJFnOJofV1PpFxBUGrnG If2KDAGaDLuc3bYS2mlR10012S69GrxjGgXVU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=CzjTHOxkTE4Qgd4iwVqP3xmJj/nHZVRS08KnoIo68+M=; b=aqeOBAiWk4e2HFHBQ7DZJQxFLUGPmrNNno+GhiLaGr60PxCMV7Jd0y1qLd85Ak61Ma OKrXqVW/FsGuJBPEkTflglZvNFiFDbCirw2HBPKoAx2YHLyP0AETMD2rKplZyhJepFHC e/31oCulIc7ejDcF7EanAzKrwplaKlVgxep/xQFOZCkwgSInmfQdUrbdLBCKMpLTtZvM cZebkujWbKJnf5FmCTxXcpf3jx1LWKhlWYDQzxkse/haKu5e4ilWLHlrF8NMXySMO45Z S1T1FStVReM1gwpaDzjpb3adFSaTL8H6I1AkDALylBtC+AcyBiR0Vr+1zHv7wVe4wwJA yVzA== X-Gm-Message-State: AIkVDXKuuRObfLaRJusaLwK2kUQ4RCZlhjbfTT9CGRYzcysUYnHjVi+hm+qId6K0tQFJWeu3 X-Received: by 10.28.100.70 with SMTP id y67mr15894255wmb.45.1484666567074; Tue, 17 Jan 2017 07:22:47 -0800 (PST) Received: from localhost.localdomain ([160.167.203.25]) by smtp.gmail.com with ESMTPSA id y127sm37590359wmg.12.2017.01.17.07.22.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 17 Jan 2017 07:22:46 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au Subject: [PATCH 01/10] crypto: arm64/aes-neon-bs - honour iv_out requirement in CTR mode Date: Tue, 17 Jan 2017 15:22:28 +0000 Message-Id: <1484666557-31458-2-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> References: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170117_072306_983657_2A36C8F5 X-CRM114-Status: GOOD ( 10.45 ) X-Spam-Score: -2.7 (--) X-Spam-Report: SpamAssassin version 3.4.1 on bombadil.infradead.org summary: Content analysis details: (-2.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low trust [2a00:1450:400c:c09:0:0:0:22a listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arm-kernel@lists.infradead.org, Ard Biesheuvel MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org Update the new bitsliced NEON AES implementation in CTR mode to return the next IV back to the skcipher API client. This is necessary for chaining to work correctly. Note that this is only done if the request is a round multiple of the block size, since otherwise, chaining is impossible anyway. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-neonbs-core.S | 25 +++++++++++++------- 1 file changed, 16 insertions(+), 9 deletions(-) -- 2.7.4 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel diff --git a/arch/arm64/crypto/aes-neonbs-core.S b/arch/arm64/crypto/aes-neonbs-core.S index 8d0cdaa2768d..2ada12dd768e 100644 --- a/arch/arm64/crypto/aes-neonbs-core.S +++ b/arch/arm64/crypto/aes-neonbs-core.S @@ -874,12 +874,19 @@ CPU_LE( rev x8, x8 ) csel x4, x4, xzr, pl csel x9, x9, xzr, le + tbnz x9, #1, 0f next_ctr v1 + tbnz x9, #2, 0f next_ctr v2 + tbnz x9, #3, 0f next_ctr v3 + tbnz x9, #4, 0f next_ctr v4 + tbnz x9, #5, 0f next_ctr v5 + tbnz x9, #6, 0f next_ctr v6 + tbnz x9, #7, 0f next_ctr v7 0: mov bskey, x2 @@ -928,11 +935,11 @@ CPU_LE( rev x8, x8 ) eor v5.16b, v5.16b, v15.16b st1 {v5.16b}, [x0], #16 - next_ctr v0 +8: next_ctr v0 cbnz x4, 99b 0: st1 {v0.16b}, [x5] -8: ldp x29, x30, [sp], #16 +9: ldp x29, x30, [sp], #16 ret /* @@ -941,23 +948,23 @@ CPU_LE( rev x8, x8 ) */ 1: cbz x6, 8b st1 {v1.16b}, [x5] - b 8b + b 9b 2: cbz x6, 8b st1 {v4.16b}, [x5] - b 8b + b 9b 3: cbz x6, 8b st1 {v6.16b}, [x5] - b 8b + b 9b 4: cbz x6, 8b st1 {v3.16b}, [x5] - b 8b + b 9b 5: cbz x6, 8b st1 {v7.16b}, [x5] - b 8b + b 9b 6: cbz x6, 8b st1 {v2.16b}, [x5] - b 8b + b 9b 7: cbz x6, 8b st1 {v5.16b}, [x5] - b 8b + b 9b ENDPROC(aesbs_ctr_encrypt) From patchwork Tue Jan 17 15:22:29 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 91682 Delivered-To: patch@linaro.org Received: by 10.182.3.34 with SMTP id 2csp499364obz; Tue, 17 Jan 2017 07:31:22 -0800 (PST) X-Received: by 10.84.209.203 with SMTP id y69mr59471586plh.115.1484667082823; Tue, 17 Jan 2017 07:31:22 -0800 (PST) Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 6si11639381pfr.161.2017.01.17.07.31.22; Tue, 17 Jan 2017 07:31:22 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751023AbdAQPbV (ORCPT + 1 other); Tue, 17 Jan 2017 10:31:21 -0500 Received: from mail-wm0-f47.google.com ([74.125.82.47]:37008 "EHLO mail-wm0-f47.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750932AbdAQPbU (ORCPT ); Tue, 17 Jan 2017 10:31:20 -0500 Received: by mail-wm0-f47.google.com with SMTP id c206so228314972wme.0 for ; Tue, 17 Jan 2017 07:31:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uMxznzyzjyL6KPmIcSlEFgXHzKWi8SJ3MJxOaj5XmxA=; b=AXKezKPGD4yX8FySuY0AMRPvZuaIJnDUkwddO6UUpsgiXUoPw9s24epEsjlqOtTAWf aAiHaqszaBrt0NbZZM2lNQAFEanWOtzlc+7Dl8EYaWdw+d5Vo40rtXmvkEpddsbpEXZy V2SsoCbfYLyBdue9ya5JveK7Ng5MylYI56QzM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=uMxznzyzjyL6KPmIcSlEFgXHzKWi8SJ3MJxOaj5XmxA=; b=GAfgV0n6wGUP1Nq1mXqzKe+NuYFQcQxuTrIiOCJtN4xCFo5OWJwoRNWxAEnYiCyITY 4XL53FEuY9ABlrmOGErWtkj4Br8K7gUsAYR4ezU2309Sbo5gsrZ1nH03/4FF15rcmt57 GOIOoPe5Ax50VepCdkNeuTflftOcTL46ghrkv3nYrsyl+atM70HZJ256MZErbxX2ZEM+ yDC+2nAi6sK6sbcqC+iH5v8c0rtt/9eMC+hhEaaz4VbogDh4NlV2Xwgjyh8zzpAzROL+ uXGDYw5ttIru+OjwwykgJD8kbz1POriO9C+2U9CDL+m1eidhIdENH/+Vt97JezMjliRF KQyA== X-Gm-Message-State: AIkVDXJp5G2bw3J2eJ+4sCA1tGek2cqqA6ZP9E733ZVhSM6RbzFKC0hRQt2n3tqImrxgIQbr X-Received: by 10.223.160.114 with SMTP id l47mr7596191wrl.73.1484666570094; Tue, 17 Jan 2017 07:22:50 -0800 (PST) Received: from localhost.localdomain ([160.167.203.25]) by smtp.gmail.com with ESMTPSA id y127sm37590359wmg.12.2017.01.17.07.22.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 17 Jan 2017 07:22:49 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au Cc: linux-arm-kernel@lists.infradead.org, Ard Biesheuvel Subject: [PATCH 02/10] crypto: arm/aes-ce - remove cra_alignmask Date: Tue, 17 Jan 2017 15:22:29 +0000 Message-Id: <1484666557-31458-3-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> References: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Remove the unnecessary alignmask: it is much more efficient to deal with the misalignment in the core algorithm than relying on the crypto API to copy the data to a suitably aligned buffer. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/aes-ce-core.S | 84 ++++++++++---------- arch/arm/crypto/aes-ce-glue.c | 15 ++-- 2 files changed, 47 insertions(+), 52 deletions(-) -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-crypto" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm/crypto/aes-ce-core.S b/arch/arm/crypto/aes-ce-core.S index 987aa632c9f0..ba8e6a32fdc9 100644 --- a/arch/arm/crypto/aes-ce-core.S +++ b/arch/arm/crypto/aes-ce-core.S @@ -169,19 +169,19 @@ ENTRY(ce_aes_ecb_encrypt) .Lecbencloop3x: subs r4, r4, #3 bmi .Lecbenc1x - vld1.8 {q0-q1}, [r1, :64]! - vld1.8 {q2}, [r1, :64]! + vld1.8 {q0-q1}, [r1]! + vld1.8 {q2}, [r1]! bl aes_encrypt_3x - vst1.8 {q0-q1}, [r0, :64]! - vst1.8 {q2}, [r0, :64]! + vst1.8 {q0-q1}, [r0]! + vst1.8 {q2}, [r0]! b .Lecbencloop3x .Lecbenc1x: adds r4, r4, #3 beq .Lecbencout .Lecbencloop: - vld1.8 {q0}, [r1, :64]! + vld1.8 {q0}, [r1]! bl aes_encrypt - vst1.8 {q0}, [r0, :64]! + vst1.8 {q0}, [r0]! subs r4, r4, #1 bne .Lecbencloop .Lecbencout: @@ -195,19 +195,19 @@ ENTRY(ce_aes_ecb_decrypt) .Lecbdecloop3x: subs r4, r4, #3 bmi .Lecbdec1x - vld1.8 {q0-q1}, [r1, :64]! - vld1.8 {q2}, [r1, :64]! + vld1.8 {q0-q1}, [r1]! + vld1.8 {q2}, [r1]! bl aes_decrypt_3x - vst1.8 {q0-q1}, [r0, :64]! - vst1.8 {q2}, [r0, :64]! + vst1.8 {q0-q1}, [r0]! + vst1.8 {q2}, [r0]! b .Lecbdecloop3x .Lecbdec1x: adds r4, r4, #3 beq .Lecbdecout .Lecbdecloop: - vld1.8 {q0}, [r1, :64]! + vld1.8 {q0}, [r1]! bl aes_decrypt - vst1.8 {q0}, [r0, :64]! + vst1.8 {q0}, [r0]! subs r4, r4, #1 bne .Lecbdecloop .Lecbdecout: @@ -226,10 +226,10 @@ ENTRY(ce_aes_cbc_encrypt) vld1.8 {q0}, [r5] prepare_key r2, r3 .Lcbcencloop: - vld1.8 {q1}, [r1, :64]! @ get next pt block + vld1.8 {q1}, [r1]! @ get next pt block veor q0, q0, q1 @ ..and xor with iv bl aes_encrypt - vst1.8 {q0}, [r0, :64]! + vst1.8 {q0}, [r0]! subs r4, r4, #1 bne .Lcbcencloop vst1.8 {q0}, [r5] @@ -244,8 +244,8 @@ ENTRY(ce_aes_cbc_decrypt) .Lcbcdecloop3x: subs r4, r4, #3 bmi .Lcbcdec1x - vld1.8 {q0-q1}, [r1, :64]! - vld1.8 {q2}, [r1, :64]! + vld1.8 {q0-q1}, [r1]! + vld1.8 {q2}, [r1]! vmov q3, q0 vmov q4, q1 vmov q5, q2 @@ -254,19 +254,19 @@ ENTRY(ce_aes_cbc_decrypt) veor q1, q1, q3 veor q2, q2, q4 vmov q6, q5 - vst1.8 {q0-q1}, [r0, :64]! - vst1.8 {q2}, [r0, :64]! + vst1.8 {q0-q1}, [r0]! + vst1.8 {q2}, [r0]! b .Lcbcdecloop3x .Lcbcdec1x: adds r4, r4, #3 beq .Lcbcdecout vmov q15, q14 @ preserve last round key .Lcbcdecloop: - vld1.8 {q0}, [r1, :64]! @ get next ct block + vld1.8 {q0}, [r1]! @ get next ct block veor q14, q15, q6 @ combine prev ct with last key vmov q6, q0 bl aes_decrypt - vst1.8 {q0}, [r0, :64]! + vst1.8 {q0}, [r0]! subs r4, r4, #1 bne .Lcbcdecloop .Lcbcdecout: @@ -300,15 +300,15 @@ ENTRY(ce_aes_ctr_encrypt) rev ip, r6 add r6, r6, #1 vmov s11, ip - vld1.8 {q3-q4}, [r1, :64]! - vld1.8 {q5}, [r1, :64]! + vld1.8 {q3-q4}, [r1]! + vld1.8 {q5}, [r1]! bl aes_encrypt_3x veor q0, q0, q3 veor q1, q1, q4 veor q2, q2, q5 rev ip, r6 - vst1.8 {q0-q1}, [r0, :64]! - vst1.8 {q2}, [r0, :64]! + vst1.8 {q0-q1}, [r0]! + vst1.8 {q2}, [r0]! vmov s27, ip b .Lctrloop3x .Lctr1x: @@ -318,10 +318,10 @@ ENTRY(ce_aes_ctr_encrypt) vmov q0, q6 bl aes_encrypt subs r4, r4, #1 - bmi .Lctrhalfblock @ blocks < 0 means 1/2 block - vld1.8 {q3}, [r1, :64]! + bmi .Lctrtailblock @ blocks < 0 means tail block + vld1.8 {q3}, [r1]! veor q3, q0, q3 - vst1.8 {q3}, [r0, :64]! + vst1.8 {q3}, [r0]! adds r6, r6, #1 @ increment BE ctr rev ip, r6 @@ -333,10 +333,8 @@ ENTRY(ce_aes_ctr_encrypt) vst1.8 {q6}, [r5] pop {r4-r6, pc} -.Lctrhalfblock: - vld1.8 {d1}, [r1, :64] - veor d0, d0, d1 - vst1.8 {d0}, [r0, :64] +.Lctrtailblock: + vst1.8 {q0}, [r0, :64] @ return just the key stream pop {r4-r6, pc} .Lctrcarry: @@ -405,8 +403,8 @@ ENTRY(ce_aes_xts_encrypt) .Lxtsenc3x: subs r4, r4, #3 bmi .Lxtsenc1x - vld1.8 {q0-q1}, [r1, :64]! @ get 3 pt blocks - vld1.8 {q2}, [r1, :64]! + vld1.8 {q0-q1}, [r1]! @ get 3 pt blocks + vld1.8 {q2}, [r1]! next_tweak q4, q3, q7, q6 veor q0, q0, q3 next_tweak q5, q4, q7, q6 @@ -416,8 +414,8 @@ ENTRY(ce_aes_xts_encrypt) veor q0, q0, q3 veor q1, q1, q4 veor q2, q2, q5 - vst1.8 {q0-q1}, [r0, :64]! @ write 3 ct blocks - vst1.8 {q2}, [r0, :64]! + vst1.8 {q0-q1}, [r0]! @ write 3 ct blocks + vst1.8 {q2}, [r0]! vmov q3, q5 teq r4, #0 beq .Lxtsencout @@ -426,11 +424,11 @@ ENTRY(ce_aes_xts_encrypt) adds r4, r4, #3 beq .Lxtsencout .Lxtsencloop: - vld1.8 {q0}, [r1, :64]! + vld1.8 {q0}, [r1]! veor q0, q0, q3 bl aes_encrypt veor q0, q0, q3 - vst1.8 {q0}, [r0, :64]! + vst1.8 {q0}, [r0]! subs r4, r4, #1 beq .Lxtsencout next_tweak q3, q3, q7, q6 @@ -456,8 +454,8 @@ ENTRY(ce_aes_xts_decrypt) .Lxtsdec3x: subs r4, r4, #3 bmi .Lxtsdec1x - vld1.8 {q0-q1}, [r1, :64]! @ get 3 ct blocks - vld1.8 {q2}, [r1, :64]! + vld1.8 {q0-q1}, [r1]! @ get 3 ct blocks + vld1.8 {q2}, [r1]! next_tweak q4, q3, q7, q6 veor q0, q0, q3 next_tweak q5, q4, q7, q6 @@ -467,8 +465,8 @@ ENTRY(ce_aes_xts_decrypt) veor q0, q0, q3 veor q1, q1, q4 veor q2, q2, q5 - vst1.8 {q0-q1}, [r0, :64]! @ write 3 pt blocks - vst1.8 {q2}, [r0, :64]! + vst1.8 {q0-q1}, [r0]! @ write 3 pt blocks + vst1.8 {q2}, [r0]! vmov q3, q5 teq r4, #0 beq .Lxtsdecout @@ -477,12 +475,12 @@ ENTRY(ce_aes_xts_decrypt) adds r4, r4, #3 beq .Lxtsdecout .Lxtsdecloop: - vld1.8 {q0}, [r1, :64]! + vld1.8 {q0}, [r1]! veor q0, q0, q3 add ip, r2, #32 @ 3rd round key bl aes_decrypt veor q0, q0, q3 - vst1.8 {q0}, [r0, :64]! + vst1.8 {q0}, [r0]! subs r4, r4, #1 beq .Lxtsdecout next_tweak q3, q3, q7, q6 diff --git a/arch/arm/crypto/aes-ce-glue.c b/arch/arm/crypto/aes-ce-glue.c index 8857531915bf..883b84d828c5 100644 --- a/arch/arm/crypto/aes-ce-glue.c +++ b/arch/arm/crypto/aes-ce-glue.c @@ -278,14 +278,15 @@ static int ctr_encrypt(struct skcipher_request *req) u8 *tsrc = walk.src.virt.addr; /* - * Minimum alignment is 8 bytes, so if nbytes is <= 8, we need - * to tell aes_ctr_encrypt() to only read half a block. + * Tell aes_ctr_encrypt() to process a tail block. */ - blocks = (nbytes <= 8) ? -1 : 1; + blocks = -1; - ce_aes_ctr_encrypt(tail, tsrc, (u8 *)ctx->key_enc, + ce_aes_ctr_encrypt(tail, NULL, (u8 *)ctx->key_enc, num_rounds(ctx), blocks, walk.iv); - memcpy(tdst, tail, nbytes); + if (tdst != tsrc) + memcpy(tdst, tsrc, nbytes); + crypto_xor(tdst, tail, nbytes); err = skcipher_walk_done(&walk, 0); } kernel_neon_end(); @@ -345,7 +346,6 @@ static struct skcipher_alg aes_algs[] = { { .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = AES_BLOCK_SIZE, .cra_ctxsize = sizeof(struct crypto_aes_ctx), - .cra_alignmask = 7, .cra_module = THIS_MODULE, }, .min_keysize = AES_MIN_KEY_SIZE, @@ -361,7 +361,6 @@ static struct skcipher_alg aes_algs[] = { { .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = AES_BLOCK_SIZE, .cra_ctxsize = sizeof(struct crypto_aes_ctx), - .cra_alignmask = 7, .cra_module = THIS_MODULE, }, .min_keysize = AES_MIN_KEY_SIZE, @@ -378,7 +377,6 @@ static struct skcipher_alg aes_algs[] = { { .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = 1, .cra_ctxsize = sizeof(struct crypto_aes_ctx), - .cra_alignmask = 7, .cra_module = THIS_MODULE, }, .min_keysize = AES_MIN_KEY_SIZE, @@ -396,7 +394,6 @@ static struct skcipher_alg aes_algs[] = { { .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = AES_BLOCK_SIZE, .cra_ctxsize = sizeof(struct crypto_aes_xts_ctx), - .cra_alignmask = 7, .cra_module = THIS_MODULE, }, .min_keysize = 2 * AES_MIN_KEY_SIZE, From patchwork Tue Jan 17 15:22:30 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 91674 Delivered-To: patch@linaro.org Received: by 10.140.20.99 with SMTP id 90csp543162qgi; Tue, 17 Jan 2017 07:23:45 -0800 (PST) X-Received: by 10.98.63.210 with SMTP id z79mr44490269pfj.134.1484666625399; Tue, 17 Jan 2017 07:23:45 -0800 (PST) Return-Path: Received: from bombadil.infradead.org (bombadil.infradead.org. [2001:1868:205::9]) by mx.google.com with ESMTPS id 145si25230864pfa.222.2017.01.17.07.23.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 17 Jan 2017 07:23:45 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org designates 2001:1868:205::9 as permitted sender) client-ip=2001:1868:205::9; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org designates 2001:1868:205::9 as permitted sender) smtp.mailfrom=linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1cTVbr-00030s-JX; Tue, 17 Jan 2017 15:23:43 +0000 Received: from mail-wm0-x234.google.com ([2a00:1450:400c:c09::234]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1cTVbP-0002A8-78 for linux-arm-kernel@lists.infradead.org; Tue, 17 Jan 2017 15:23:17 +0000 Received: by mail-wm0-x234.google.com with SMTP id r144so227513178wme.1 for ; Tue, 17 Jan 2017 07:22:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Twhl3TaN4Yleujf68Qk0fFCweF7qBJDAw0TmYcZHr54=; b=LaEelGgUYQ8wmmkdPw193rHSBrKGMSTXMpXjBD6ShClf3MC3TZDKbC9CzDdAMUINho +zweqZxTE6WScQvRYIsGnz0TcePXGR5IJTzzxvWlroukP4Jeqw593jq15D0f6K9/GGeX Va/OWP0puIdCi9xt3R3pZErtiLsBt/mog46so= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Twhl3TaN4Yleujf68Qk0fFCweF7qBJDAw0TmYcZHr54=; b=MJ4PovakO9MuEOVWHxT5BINtsXMAPs0OfnQYEF+JQLA0oRbqTkPbUfq2YZPvfykHmO pKAf2NonScVPZ/KBLXBht2zLrqHh4UEHRcnS571vjqtIgr0q2S9tv7+JeFaoc3iQMsV2 UIl6SfgXoRNJLW5Acn3hfzJ3zwRdACcGWm8bDFKXsGRm0jv6jFeI5VoF/2fNc7P3Yydy Oackdf1ABp9eGNazKV9FZbp1s9awYSg0+Rjr0O+5/XT/N9smODmoGILhWZA8Bx5JUj9q UAhMxGM3MLHg+Ab6zSf5PhXXjVD3DeQEonwzVXv3sVajSPDrSEmTiTWLEhAz65R4uBMi Iufw== X-Gm-Message-State: AIkVDXISYwR5f220hkJthkTzbWHUaOjm+baprjOBzQtYvs46TKD/b4T4BCucGhlmTpdoPlg8 X-Received: by 10.28.100.70 with SMTP id y67mr15894596wmb.45.1484666573346; Tue, 17 Jan 2017 07:22:53 -0800 (PST) Received: from localhost.localdomain ([160.167.203.25]) by smtp.gmail.com with ESMTPSA id y127sm37590359wmg.12.2017.01.17.07.22.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 17 Jan 2017 07:22:52 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au Subject: [PATCH 03/10] crypto: arm/chacha20 - remove cra_alignmask Date: Tue, 17 Jan 2017 15:22:30 +0000 Message-Id: <1484666557-31458-4-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> References: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170117_072315_538616_8D76D591 X-CRM114-Status: GOOD ( 10.54 ) X-Spam-Score: -2.7 (--) X-Spam-Report: SpamAssassin version 3.4.1 on bombadil.infradead.org summary: Content analysis details: (-2.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low trust [2a00:1450:400c:c09:0:0:0:234 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arm-kernel@lists.infradead.org, Ard Biesheuvel MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org Remove the unnecessary alignmask: it is much more efficient to deal with the misalignment in the core algorithm than relying on the crypto API to copy the data to a suitably aligned buffer. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/chacha20-neon-glue.c | 1 - 1 file changed, 1 deletion(-) -- 2.7.4 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel diff --git a/arch/arm/crypto/chacha20-neon-glue.c b/arch/arm/crypto/chacha20-neon-glue.c index 592f75ae4fa1..59a7be08e80c 100644 --- a/arch/arm/crypto/chacha20-neon-glue.c +++ b/arch/arm/crypto/chacha20-neon-glue.c @@ -94,7 +94,6 @@ static struct skcipher_alg alg = { .base.cra_priority = 300, .base.cra_blocksize = 1, .base.cra_ctxsize = sizeof(struct chacha20_ctx), - .base.cra_alignmask = 1, .base.cra_module = THIS_MODULE, .min_keysize = CHACHA20_KEY_SIZE, From patchwork Tue Jan 17 15:22:31 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 91676 Delivered-To: patch@linaro.org Received: by 10.140.20.99 with SMTP id 90csp543269qgi; Tue, 17 Jan 2017 07:24:01 -0800 (PST) X-Received: by 10.84.231.9 with SMTP id f9mr30897958plk.28.1484666641082; Tue, 17 Jan 2017 07:24:01 -0800 (PST) Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e13si25201758pgf.220.2017.01.17.07.24.00; Tue, 17 Jan 2017 07:24:01 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1750803AbdAQPX5 (ORCPT + 1 other); Tue, 17 Jan 2017 10:23:57 -0500 Received: from mail-wm0-f54.google.com ([74.125.82.54]:35194 "EHLO mail-wm0-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750770AbdAQPX4 (ORCPT ); Tue, 17 Jan 2017 10:23:56 -0500 Received: by mail-wm0-f54.google.com with SMTP id r126so205365367wmr.0 for ; Tue, 17 Jan 2017 07:22:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uEavdoflWkEgxBCfHtZRMITUTt3rHQF+vKAmVwYFXzM=; b=VcPULplGtNBNwCEWZ7LE54azPdlBMhpy+Nh+dT15QB+gt/I9/UI+xfXZ7j2ezsflZw QhZRuW0ZR8HTSPOh+Dbqr0aLOkatFQHeSCTGCkurLEFw7V1TMHTDFnRlkyYMQCexSQnR h5gFNFEO+6UAqlhBOdvf/nwra3cPeK+k87Es4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=uEavdoflWkEgxBCfHtZRMITUTt3rHQF+vKAmVwYFXzM=; b=Sw3lvqfGzSqrq2m1snVcV8+rG/8haWVSoLPaTCx7kvRDtER9chPk0iHXlXu3IDNigL JkCa3msHQ6tAL1+q90yTkZWZJDgbOlhogN88QW2W67btA8Mv2o9lAIEzajfiITJzzjq/ nMZsvrmeTVorL+oICIfU3hSF13efpk1IztxWgJUF1qDuj1SFd5XrOnlcYTYRN/iKU5Kc TUR2saHGgSQ7xmAqScl+19ikiubn+pB7YqJewaDeL01rMYE/AX+8G6ZxbkaWinT2XfZc 2E6aaQOb511P04vcM6ol3eMOLMAJzdyJd/rgrRadLgBbR2gobR1ik3Zk1ati2YuIVjHS BwcA== X-Gm-Message-State: AIkVDXJGoocx9xXsvxkqbGJ/LLV9ZYhgGHFG+1Sv0Km5S6go/tI/8NbQUwqi/VCJ0SKLx2lx X-Received: by 10.28.107.77 with SMTP id g74mr15812205wmc.109.1484666576089; Tue, 17 Jan 2017 07:22:56 -0800 (PST) Received: from localhost.localdomain ([160.167.203.25]) by smtp.gmail.com with ESMTPSA id y127sm37590359wmg.12.2017.01.17.07.22.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 17 Jan 2017 07:22:55 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au Cc: linux-arm-kernel@lists.infradead.org, Ard Biesheuvel Subject: [PATCH 04/10] crypto: arm64/aes-ce-ccm - remove cra_alignmask Date: Tue, 17 Jan 2017 15:22:31 +0000 Message-Id: <1484666557-31458-5-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> References: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Remove the unnecessary alignmask: it is much more efficient to deal with the misalignment in the core algorithm than relying on the crypto API to copy the data to a suitably aligned buffer. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-ce-ccm-glue.c | 1 - 1 file changed, 1 deletion(-) -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-crypto" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce-ccm-glue.c index cc5515dac74a..6a7dbc7c83a6 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -258,7 +258,6 @@ static struct aead_alg ccm_aes_alg = { .cra_priority = 300, .cra_blocksize = 1, .cra_ctxsize = sizeof(struct crypto_aes_ctx), - .cra_alignmask = 7, .cra_module = THIS_MODULE, }, .ivsize = AES_BLOCK_SIZE, From patchwork Tue Jan 17 15:22:32 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 91678 Delivered-To: patch@linaro.org Received: by 10.140.20.99 with SMTP id 90csp543283qgi; Tue, 17 Jan 2017 07:24:03 -0800 (PST) X-Received: by 10.84.236.4 with SMTP id q4mr53833584plk.110.1484666643154; Tue, 17 Jan 2017 07:24:03 -0800 (PST) Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e13si25201758pgf.220.2017.01.17.07.24.03; Tue, 17 Jan 2017 07:24:03 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751007AbdAQPYA (ORCPT + 1 other); Tue, 17 Jan 2017 10:24:00 -0500 Received: from mail-wm0-f44.google.com ([74.125.82.44]:36319 "EHLO mail-wm0-f44.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750817AbdAQPX4 (ORCPT ); Tue, 17 Jan 2017 10:23:56 -0500 Received: by mail-wm0-f44.google.com with SMTP id c85so204600595wmi.1 for ; Tue, 17 Jan 2017 07:23:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=DteLWZVBH8uLjZJOK0l1y6OIZZLs4TaicSNQ92TWNoI=; b=JKyMa9d46h924doBRbzdnJ8PgXn+LDZFlae7Fjpc00WG1dl14F6joXUmTz3nMykUce bgL1jnHWcKCAlOTjg6J39Q6D6+N+n4N+F6EcGMazGCWPyCskL55U3mPzkcpjukvGwqGM PQlh9/0yknosnTPPqvb4NVAX4F8/OcUi4RXd8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=DteLWZVBH8uLjZJOK0l1y6OIZZLs4TaicSNQ92TWNoI=; b=HzPr3If5WGrSBNBoRohjYmmjEHkhOhseDioygnxbMB7A1s65WXGU+oq0bik2zuqXct 6SvBp/yfFCAgEYwu0pv6X6yPfLT7a98Fp/5xQzHNSFzR6l5iECXUrOhAZ60/rnPEUpHp T2M8NYVgmjWzlAQBD2ujFtZK1at3zcU7fofMpwh7OaAzI4qdqHvv3aADs8EJPy7aIjsI 2hFYlIcqdq7bz5FYpLOz34hNlezl7XDl/wci4VHsaXXj07HliuFH0N1uybg7w5KVPGNf Ykd0ZP+viAaDnNkCx/amKQLH2OMamArhChsryhfVV7kUicc+hixTMR2/IA2bD7uC35Kv 21Iw== X-Gm-Message-State: AIkVDXKaNDdtSPzYhWIrdtgrABKJzDC0eNUADVkG9O8NdvXB6gSK2e7kWMA9jd4XX76j8p0a X-Received: by 10.28.191.79 with SMTP id p76mr16193400wmf.21.1484666579087; Tue, 17 Jan 2017 07:22:59 -0800 (PST) Received: from localhost.localdomain ([160.167.203.25]) by smtp.gmail.com with ESMTPSA id y127sm37590359wmg.12.2017.01.17.07.22.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 17 Jan 2017 07:22:58 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au Cc: linux-arm-kernel@lists.infradead.org, Ard Biesheuvel Subject: [PATCH 05/10] crypto: arm64/aes-blk - remove cra_alignmask Date: Tue, 17 Jan 2017 15:22:32 +0000 Message-Id: <1484666557-31458-6-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> References: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Remove the unnecessary alignmask: it is much more efficient to deal with the misalignment in the core algorithm than relying on the crypto API to copy the data to a suitably aligned buffer. Signed-off-by: Ard Biesheuvel --- NOTE: this won't apply unless 'crypto: arm64/aes-blk - honour iv_out requirement in CBC and CTR modes' is applied first, which was sent out separately as a bugfix for v3.16 - v4.9 arch/arm64/crypto/aes-glue.c | 16 ++++++---------- arch/arm64/crypto/aes-modes.S | 8 +++----- 2 files changed, 9 insertions(+), 15 deletions(-) -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-crypto" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index 5164aaf82c6a..8ee1fb7aaa4f 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -215,14 +215,15 @@ static int ctr_encrypt(struct skcipher_request *req) u8 *tsrc = walk.src.virt.addr; /* - * Minimum alignment is 8 bytes, so if nbytes is <= 8, we need - * to tell aes_ctr_encrypt() to only read half a block. + * Tell aes_ctr_encrypt() to process a tail block. */ - blocks = (nbytes <= 8) ? -1 : 1; + blocks = -1; - aes_ctr_encrypt(tail, tsrc, (u8 *)ctx->key_enc, rounds, + aes_ctr_encrypt(tail, NULL, (u8 *)ctx->key_enc, rounds, blocks, walk.iv, first); - memcpy(tdst, tail, nbytes); + if (tdst != tsrc) + memcpy(tdst, tsrc, nbytes); + crypto_xor(tdst, tail, nbytes); err = skcipher_walk_done(&walk, 0); } kernel_neon_end(); @@ -282,7 +283,6 @@ static struct skcipher_alg aes_algs[] = { { .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = AES_BLOCK_SIZE, .cra_ctxsize = sizeof(struct crypto_aes_ctx), - .cra_alignmask = 7, .cra_module = THIS_MODULE, }, .min_keysize = AES_MIN_KEY_SIZE, @@ -298,7 +298,6 @@ static struct skcipher_alg aes_algs[] = { { .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = AES_BLOCK_SIZE, .cra_ctxsize = sizeof(struct crypto_aes_ctx), - .cra_alignmask = 7, .cra_module = THIS_MODULE, }, .min_keysize = AES_MIN_KEY_SIZE, @@ -315,7 +314,6 @@ static struct skcipher_alg aes_algs[] = { { .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = 1, .cra_ctxsize = sizeof(struct crypto_aes_ctx), - .cra_alignmask = 7, .cra_module = THIS_MODULE, }, .min_keysize = AES_MIN_KEY_SIZE, @@ -332,7 +330,6 @@ static struct skcipher_alg aes_algs[] = { { .cra_priority = PRIO - 1, .cra_blocksize = 1, .cra_ctxsize = sizeof(struct crypto_aes_ctx), - .cra_alignmask = 7, .cra_module = THIS_MODULE, }, .min_keysize = AES_MIN_KEY_SIZE, @@ -350,7 +347,6 @@ static struct skcipher_alg aes_algs[] = { { .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = AES_BLOCK_SIZE, .cra_ctxsize = sizeof(struct crypto_aes_xts_ctx), - .cra_alignmask = 7, .cra_module = THIS_MODULE, }, .min_keysize = 2 * AES_MIN_KEY_SIZE, diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index 838dad5c209f..92b982a8b112 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -337,7 +337,7 @@ AES_ENTRY(aes_ctr_encrypt) .Lctrcarrydone: subs w4, w4, #1 - bmi .Lctrhalfblock /* blocks < 0 means 1/2 block */ + bmi .Lctrtailblock /* blocks <0 means tail block */ ld1 {v3.16b}, [x1], #16 eor v3.16b, v0.16b, v3.16b st1 {v3.16b}, [x0], #16 @@ -348,10 +348,8 @@ AES_ENTRY(aes_ctr_encrypt) FRAME_POP ret -.Lctrhalfblock: - ld1 {v3.8b}, [x1] - eor v3.8b, v0.8b, v3.8b - st1 {v3.8b}, [x0] +.Lctrtailblock: + st1 {v0.16b}, [x0] FRAME_POP ret From patchwork Tue Jan 17 15:22:33 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 91675 Delivered-To: patch@linaro.org Received: by 10.140.20.99 with SMTP id 90csp543249qgi; Tue, 17 Jan 2017 07:23:59 -0800 (PST) X-Received: by 10.98.131.136 with SMTP id h130mr40968921pfe.4.1484666639026; Tue, 17 Jan 2017 07:23:59 -0800 (PST) Return-Path: Received: from bombadil.infradead.org (bombadil.infradead.org. [2001:1868:205::9]) by mx.google.com with ESMTPS id 23si24098837pfy.74.2017.01.17.07.23.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 17 Jan 2017 07:23:59 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org designates 2001:1868:205::9 as permitted sender) client-ip=2001:1868:205::9; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org designates 2001:1868:205::9 as permitted sender) smtp.mailfrom=linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1cTVc5-0003LB-Bh; Tue, 17 Jan 2017 15:23:57 +0000 Received: from mail-wm0-x22a.google.com ([2a00:1450:400c:c09::22a]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1cTVbQ-0002AN-By for linux-arm-kernel@lists.infradead.org; Tue, 17 Jan 2017 15:23:18 +0000 Received: by mail-wm0-x22a.google.com with SMTP id c206so227924739wme.0 for ; Tue, 17 Jan 2017 07:23:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=zfpd1eO7xxGzNokheRELTOmW0w9HWUScNlNa89gvn3Y=; b=IyL2SJ7/2+LvlZU8MSF6NlzHN0e3zkljoLKPGwpxEd+QIOUwzTQVmyPxZ8agelEkU1 BBlu2AxD8OqoGo90/wOWlqg2iY6fcAaX/0AjRlMEKLpTUy2O9jvkYz5kkCQALBu47HIx HMLbIU4CiccwTs42P92Mi3RZ9gcKWf/YdUT9k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=zfpd1eO7xxGzNokheRELTOmW0w9HWUScNlNa89gvn3Y=; b=Xqx1vrkBTSCc7LdcggFdTRHBM+kAOe3rnFjl+ROWuCxJJvyny/ydYrM65EYCuniH0L SNhSuLMdeWPbZG9/KaFdUyvWeY3MQuUPFg7fcp/Hwll3EpNAyk/X14VcwZjEPsM3FMdz sBZhMEtYRYRBfun5MDHoToKBiu+0j5lQrZQRmn1eVsKBKNlx/QFP0mrlArXUU1Pu/A/y bEqqXBuDjzAuxkGSqOAq0ppLq1FmDEqOuEUH4uansKRUXXnF6Bq3aU7oZi6KbTqXMpTe mOtjnB7kx0lyV5cK2g3u7NswdoWVSp/zHp5AjsJjkgIQHw+itoJuu/0IuPhg99P2fMAP jyHQ== X-Gm-Message-State: AIkVDXINCpEb3/00hKMuhsI//ZijBuA4pbp7B3Hh43Fh5ohY9jDP2Knd0RuNgUvWvedEqULF X-Received: by 10.223.154.132 with SMTP id a4mr32128225wrc.188.1484666582841; Tue, 17 Jan 2017 07:23:02 -0800 (PST) Received: from localhost.localdomain ([160.167.203.25]) by smtp.gmail.com with ESMTPSA id y127sm37590359wmg.12.2017.01.17.07.22.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 17 Jan 2017 07:23:01 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au Subject: [PATCH 06/10] crypto: arm64/chacha20 - remove cra_alignmask Date: Tue, 17 Jan 2017 15:22:33 +0000 Message-Id: <1484666557-31458-7-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> References: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170117_072316_676653_0A43567E X-CRM114-Status: GOOD ( 10.54 ) X-Spam-Score: -2.7 (--) X-Spam-Report: SpamAssassin version 3.4.1 on bombadil.infradead.org summary: Content analysis details: (-2.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low trust [2a00:1450:400c:c09:0:0:0:22a listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arm-kernel@lists.infradead.org, Ard Biesheuvel MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org Remove the unnecessary alignmask: it is much more efficient to deal with the misalignment in the core algorithm than relying on the crypto API to copy the data to a suitably aligned buffer. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/chacha20-neon-glue.c | 1 - 1 file changed, 1 deletion(-) -- 2.7.4 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel diff --git a/arch/arm64/crypto/chacha20-neon-glue.c b/arch/arm64/crypto/chacha20-neon-glue.c index a7f2337d46cf..a7cd575ea223 100644 --- a/arch/arm64/crypto/chacha20-neon-glue.c +++ b/arch/arm64/crypto/chacha20-neon-glue.c @@ -93,7 +93,6 @@ static struct skcipher_alg alg = { .base.cra_priority = 300, .base.cra_blocksize = 1, .base.cra_ctxsize = sizeof(struct chacha20_ctx), - .base.cra_alignmask = 1, .base.cra_module = THIS_MODULE, .min_keysize = CHACHA20_KEY_SIZE, From patchwork Tue Jan 17 15:22:34 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 91681 Delivered-To: patch@linaro.org Received: by 10.140.20.99 with SMTP id 90csp543302qgi; Tue, 17 Jan 2017 07:24:05 -0800 (PST) X-Received: by 10.84.217.216 with SMTP id d24mr59356895plj.101.1484666645555; Tue, 17 Jan 2017 07:24:05 -0800 (PST) Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e13si25201758pgf.220.2017.01.17.07.24.05; Tue, 17 Jan 2017 07:24:05 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1750920AbdAQPYD (ORCPT + 1 other); Tue, 17 Jan 2017 10:24:03 -0500 Received: from mail-wm0-f41.google.com ([74.125.82.41]:36937 "EHLO mail-wm0-f41.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750892AbdAQPX4 (ORCPT ); Tue, 17 Jan 2017 10:23:56 -0500 Received: by mail-wm0-f41.google.com with SMTP id c206so227927090wme.0 for ; Tue, 17 Jan 2017 07:23:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=i8PTGdBLEln38Rqn6mFXAiQkiYiMzfbJpCdlXeMJ1E4=; b=HjZ79z2W6jeuwLydyhGdg4E+nVzqWr3k5v1es0wm3vwlkfxLwQsZwaICVUDsUvheXp 5byI7oJ4lghae6+RZQqZssl3K1y7wycaGFPpEcyLEAaST0lJ0zVLeVyNCZJSu6VDEPSB 8cF7T3Cb4icnFBFuKMpuu5gKoenycD2v8Cknw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=i8PTGdBLEln38Rqn6mFXAiQkiYiMzfbJpCdlXeMJ1E4=; b=dwyAmwY9jWVE85SjNNAQ6Zzzmk1G1oEsFY90JGLpB0xFTG5UN5d0z753sALbhI/txE P3lk0ZfcNfLb3Y1EjqWMA1Hbzlnc2/gm9Jvw2qHbVL9HgZe0Y/CiLUp2gsbHaAlGSfx2 1VizgxlQ28QLNniffem2wdtTipnEE7MwQUzKCNlehI2Qf5unkkMJcwoiyr2x2fqz8N1l EbmPbhoPsMJEXNkX+M8hBpUytSlQRflRuoFlB/miKTZwc+ikiBXigHIM9c3vOPD0rHo5 axrFRzn22nH1vJN5k/ofD1VVB6Q/wQPuCZKhCk3n0kxjtGjSX7gjGAuaYizD7JDnMMPt nr0Q== X-Gm-Message-State: AIkVDXLDR9p3RUURldh9IYCDGOJEpF9gbql5XyTKJ+A9mZskoXlqckzjwWhGRLK8T+L3HwB1 X-Received: by 10.223.150.183 with SMTP id u52mr27157274wrb.180.1484666585839; Tue, 17 Jan 2017 07:23:05 -0800 (PST) Received: from localhost.localdomain ([160.167.203.25]) by smtp.gmail.com with ESMTPSA id y127sm37590359wmg.12.2017.01.17.07.23.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 17 Jan 2017 07:23:04 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au Cc: linux-arm-kernel@lists.infradead.org, Ard Biesheuvel Subject: [PATCH 07/10] crypto: arm64/aes - avoid literals for cross-module symbol references Date: Tue, 17 Jan 2017 15:22:34 +0000 Message-Id: <1484666557-31458-8-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> References: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Using simple adrp/add pairs to refer to the AES lookup tables exposed by the generic AES driver (which could be loaded far away from this driver when KASLR is in effect) was unreliable at module load time before commit 41c066f2c4d4 ("arm64: assembler: make adr_l work in modules under KASLR"), which is why the AES code used literals instead. So now we can get rid of the literals, and switch to the adr_l macro. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-cipher-core.S | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-crypto" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/aes-cipher-core.S b/arch/arm64/crypto/aes-cipher-core.S index 37590ab8121a..cd58c61e6677 100644 --- a/arch/arm64/crypto/aes-cipher-core.S +++ b/arch/arm64/crypto/aes-cipher-core.S @@ -89,8 +89,8 @@ CPU_BE( rev w8, w8 ) eor w7, w7, w11 eor w8, w8, w12 - ldr tt, =\ttab - ldr lt, =\ltab + adr_l tt, \ttab + adr_l lt, \ltab tbnz rounds, #1, 1f @@ -111,9 +111,6 @@ CPU_BE( rev w8, w8 ) stp w5, w6, [out] stp w7, w8, [out, #8] ret - - .align 4 - .ltorg .endm .align 5 From patchwork Tue Jan 17 15:22:35 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 91677 Delivered-To: patch@linaro.org Received: by 10.140.20.99 with SMTP id 90csp543276qgi; Tue, 17 Jan 2017 07:24:02 -0800 (PST) X-Received: by 10.99.204.81 with SMTP id q17mr46847363pgi.168.1484666641979; Tue, 17 Jan 2017 07:24:01 -0800 (PST) Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e13si25201758pgf.220.2017.01.17.07.24.01; Tue, 17 Jan 2017 07:24:01 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1750947AbdAQPX7 (ORCPT + 1 other); Tue, 17 Jan 2017 10:23:59 -0500 Received: from mail-wm0-f54.google.com ([74.125.82.54]:35379 "EHLO mail-wm0-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750918AbdAQPX4 (ORCPT ); Tue, 17 Jan 2017 10:23:56 -0500 Received: by mail-wm0-f54.google.com with SMTP id r126so205373708wmr.0 for ; Tue, 17 Jan 2017 07:23:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Cy9yiOmXCCJTU3CWvHqx5pREGmQmzaCRklJ9F/Bb1ZQ=; b=f3o5yeDD2oOal5vVG+0NVGTJbOfmfe1kO3rbDt5ggkiXIzrQnYIXEakvkWxXiU54mi bwYKGSzfuVLggSKLWaFyuZcspQE575HkqrlGj+EyygChomctnAhzqvNkHyttaRwIIPCl qB/baS0nfKAccWRMV0802JcAVVkkuHoFj8imM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Cy9yiOmXCCJTU3CWvHqx5pREGmQmzaCRklJ9F/Bb1ZQ=; b=i3/3iXMgtLsl4TCUaBAtdM4iThHyrtOzeG5zKX7JKqEB3qX6ik74h4dBfxPDP5c2NG IKiRlxGkVclGG34/LYtHLjPDbBYq3X8wP6foKVuKtARIGfpoke2QgReVYxhP9Z8HqdKU 6QPlwYZGrTABIUJPrCY+zefkpcJG3ZV8xuY97j/l88sznjOQFa0pw3ibfd312OHkPQ1m wnCmGqlyA7G1xglN9iC4kKnsWyKizYiYvZg4a34YfAAnVLVd+BGjLkcYON3P2SpA29Yc C8s1iLe/E3MsrCPq0r0YY4b+RkkbM7u911ckEEk29I8aiE7InO1Ez+g9vPm5QbY9T9OB 9FUw== X-Gm-Message-State: AIkVDXKmKGFc751/GBvIxjt3yh2l7kX4q9PWs5+gexk7+VMFRUE/MolW70MFrfxMvv3rdu0l X-Received: by 10.223.145.161 with SMTP id 30mr20215400wri.199.1484666588028; Tue, 17 Jan 2017 07:23:08 -0800 (PST) Received: from localhost.localdomain ([160.167.203.25]) by smtp.gmail.com with ESMTPSA id y127sm37590359wmg.12.2017.01.17.07.23.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 17 Jan 2017 07:23:07 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au Cc: linux-arm-kernel@lists.infradead.org, Ard Biesheuvel Subject: [PATCH 08/10] crypto: arm64/aes - performance tweak Date: Tue, 17 Jan 2017 15:22:35 +0000 Message-Id: <1484666557-31458-9-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> References: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Shuffle some instructions around in the __hround macro to shave off 0.1 cycles per byte on Cortex-A57. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-cipher-core.S | 52 +++++++------------- 1 file changed, 19 insertions(+), 33 deletions(-) -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-crypto" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/aes-cipher-core.S b/arch/arm64/crypto/aes-cipher-core.S index cd58c61e6677..f2f9cc519309 100644 --- a/arch/arm64/crypto/aes-cipher-core.S +++ b/arch/arm64/crypto/aes-cipher-core.S @@ -20,46 +20,32 @@ tt .req x4 lt .req x2 - .macro __hround, out0, out1, in0, in1, in2, in3, t0, t1, enc - ldp \out0, \out1, [rk], #8 - - ubfx w13, \in0, #0, #8 - ubfx w14, \in1, #8, #8 - ldr w13, [tt, w13, uxtw #2] - ldr w14, [tt, w14, uxtw #2] - + .macro __pair, enc, reg0, reg1, in0, in1e, in1d, shift + ubfx \reg0, \in0, #\shift, #8 .if \enc - ubfx w17, \in1, #0, #8 - ubfx w18, \in2, #8, #8 + ubfx \reg1, \in1e, #\shift, #8 .else - ubfx w17, \in3, #0, #8 - ubfx w18, \in0, #8, #8 + ubfx \reg1, \in1d, #\shift, #8 .endif - ldr w17, [tt, w17, uxtw #2] - ldr w18, [tt, w18, uxtw #2] + ldr \reg0, [tt, \reg0, uxtw #2] + ldr \reg1, [tt, \reg1, uxtw #2] + .endm - ubfx w15, \in2, #16, #8 - ubfx w16, \in3, #24, #8 - ldr w15, [tt, w15, uxtw #2] - ldr w16, [tt, w16, uxtw #2] + .macro __hround, out0, out1, in0, in1, in2, in3, t0, t1, enc + ldp \out0, \out1, [rk], #8 - .if \enc - ubfx \t0, \in3, #16, #8 - ubfx \t1, \in0, #24, #8 - .else - ubfx \t0, \in1, #16, #8 - ubfx \t1, \in2, #24, #8 - .endif - ldr \t0, [tt, \t0, uxtw #2] - ldr \t1, [tt, \t1, uxtw #2] + __pair \enc, w13, w14, \in0, \in1, \in3, 0 + __pair \enc, w15, w16, \in1, \in2, \in0, 8 + __pair \enc, w17, w18, \in2, \in3, \in1, 16 + __pair \enc, \t0, \t1, \in3, \in0, \in2, 24 eor \out0, \out0, w13 - eor \out1, \out1, w17 - eor \out0, \out0, w14, ror #24 - eor \out1, \out1, w18, ror #24 - eor \out0, \out0, w15, ror #16 - eor \out1, \out1, \t0, ror #16 - eor \out0, \out0, w16, ror #8 + eor \out1, \out1, w14 + eor \out0, \out0, w15, ror #24 + eor \out1, \out1, w16, ror #24 + eor \out0, \out0, w17, ror #16 + eor \out1, \out1, w18, ror #16 + eor \out0, \out0, \t0, ror #8 eor \out1, \out1, \t1, ror #8 .endm From patchwork Tue Jan 17 15:22:36 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 91679 Delivered-To: patch@linaro.org Received: by 10.140.20.99 with SMTP id 90csp543289qgi; Tue, 17 Jan 2017 07:24:03 -0800 (PST) X-Received: by 10.98.108.196 with SMTP id h187mr5580104pfc.176.1484666643815; Tue, 17 Jan 2017 07:24:03 -0800 (PST) Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e13si25201758pgf.220.2017.01.17.07.24.03; Tue, 17 Jan 2017 07:24:03 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1750817AbdAQPYB (ORCPT + 1 other); Tue, 17 Jan 2017 10:24:01 -0500 Received: from mail-wm0-f50.google.com ([74.125.82.50]:38586 "EHLO mail-wm0-f50.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750920AbdAQPX4 (ORCPT ); Tue, 17 Jan 2017 10:23:56 -0500 Received: by mail-wm0-f50.google.com with SMTP id r144so227526415wme.1 for ; Tue, 17 Jan 2017 07:23:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=S20ee06oNQRset7AtETN4R+rVsA2p9fyD6Toi2f3NmI=; b=ElWRLuhkscu6D8u9E8YaI5o3KE0ydwQGcRYLbIeddgGpnyZ1GUi/wGHM4HkIFoL+wS jvMEPVuQ7/AaGD3f3BFe3G1xppPP2pijn1FVM18Wp3wcN+LYvntSzLlNa53vQxwm8ZZM Syk0E+OB+nQOJzIJjqDkjKzwq9fwGE64MUAlA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=S20ee06oNQRset7AtETN4R+rVsA2p9fyD6Toi2f3NmI=; b=o/7OYu45/+PsJ2Hy/xmqaYNXq6WAUSxkSVRw2F87uNS5dYu4O7y+OJaRkrsRRHWkUR MZVbl0xBk1hrRrWVAsZzZMjGAjgEo3ej+LPwuue7Lx5mLKozGz+z1XTfZKgX425L44wb ueMtV0jujoWLaePlpt9UX6bA/VEm0yaHb4VYnH6daFZZ7+T6D/3p93/JSJTV/8YxjOD6 NX1OhgJbZKc20zgu48KcphB311zv4TM/OMsBacOWT65Clqe10/JIZX+rK30Zv3GtyYHt C8olpMG0ezWDK+I9WoLuNwewhoKmq16hMHzDpmu78IHURJvC11tdJ6/54vhdnkRMY95p Dfug== X-Gm-Message-State: AIkVDXJp34vWgm3b4fNDFP5+XwNrhqyqez90BY8V4bwX4uEsms7c4bzLwI5VUdfLjJL1o0gr X-Received: by 10.223.128.202 with SMTP id 68mr28547384wrl.148.1484666590339; Tue, 17 Jan 2017 07:23:10 -0800 (PST) Received: from localhost.localdomain ([160.167.203.25]) by smtp.gmail.com with ESMTPSA id y127sm37590359wmg.12.2017.01.17.07.23.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 17 Jan 2017 07:23:09 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au Cc: linux-arm-kernel@lists.infradead.org, Ard Biesheuvel Subject: [PATCH 09/10] crypto: arm64/aes-neon-blk - tweak performance for low end cores Date: Tue, 17 Jan 2017 15:22:36 +0000 Message-Id: <1484666557-31458-10-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> References: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The non-bitsliced AES implementation using the NEON is highly sensitive to micro-architectural details, and, as it turns out, the Cortex-A53 on the Raspberry Pi 3 is a core that can benefit from this code, given that its scalar AES performance is abysmal (32.9 cycles per byte). The new bitsliced AES code manages 19.8 cycles per byte on this core, but can only operate on 8 blocks at a time, which is not supported by all chaining modes. With a bit of tweaking, we can get the plain NEON code to run at 24.0 cycles per byte, making it useful for sequential modes like CBC encryption. (Like bitsliced NEON, the plain NEON implementation does not use any lookup tables, which makes it easy on the D-cache, and invulnerable to cache timing attacks) So tweak the plain NEON AES code to use tbl instructions rather than shl/sri pairs, and to avoid the need to reload permutation vectors or other constants from memory in every round. To allow the ECB and CBC encrypt routines to be reused by the bitsliced NEON code in a subsequent patch, export them from the module. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-glue.c | 2 + arch/arm64/crypto/aes-neon.S | 199 ++++++++------------ 2 files changed, 77 insertions(+), 124 deletions(-) -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-crypto" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index 8ee1fb7aaa4f..055bc3f61138 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -409,5 +409,7 @@ static int __init aes_init(void) module_cpu_feature_match(AES, aes_init); #else module_init(aes_init); +EXPORT_SYMBOL(neon_aes_ecb_encrypt); +EXPORT_SYMBOL(neon_aes_cbc_encrypt); #endif module_exit(aes_exit); diff --git a/arch/arm64/crypto/aes-neon.S b/arch/arm64/crypto/aes-neon.S index 85f07ead7c5c..67c68462bc20 100644 --- a/arch/arm64/crypto/aes-neon.S +++ b/arch/arm64/crypto/aes-neon.S @@ -1,7 +1,7 @@ /* * linux/arch/arm64/crypto/aes-neon.S - AES cipher for ARMv8 NEON * - * Copyright (C) 2013 Linaro Ltd + * Copyright (C) 2013 - 2017 Linaro Ltd. * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as @@ -25,9 +25,9 @@ /* preload the entire Sbox */ .macro prepare, sbox, shiftrows, temp adr \temp, \sbox - movi v12.16b, #0x40 + movi v12.16b, #0x1b ldr q13, \shiftrows - movi v14.16b, #0x1b + ldr q14, .Lror32by8 ld1 {v16.16b-v19.16b}, [\temp], #64 ld1 {v20.16b-v23.16b}, [\temp], #64 ld1 {v24.16b-v27.16b}, [\temp], #64 @@ -50,37 +50,33 @@ /* apply SubBytes transformation using the the preloaded Sbox */ .macro sub_bytes, in - sub v9.16b, \in\().16b, v12.16b + sub v9.16b, \in\().16b, v15.16b tbl \in\().16b, {v16.16b-v19.16b}, \in\().16b - sub v10.16b, v9.16b, v12.16b + sub v10.16b, v9.16b, v15.16b tbx \in\().16b, {v20.16b-v23.16b}, v9.16b - sub v11.16b, v10.16b, v12.16b + sub v11.16b, v10.16b, v15.16b tbx \in\().16b, {v24.16b-v27.16b}, v10.16b tbx \in\().16b, {v28.16b-v31.16b}, v11.16b .endm /* apply MixColumns transformation */ - .macro mix_columns, in - mul_by_x v10.16b, \in\().16b, v9.16b, v14.16b - rev32 v8.8h, \in\().8h - eor \in\().16b, v10.16b, \in\().16b - shl v9.4s, v8.4s, #24 - shl v11.4s, \in\().4s, #24 - sri v9.4s, v8.4s, #8 - sri v11.4s, \in\().4s, #8 - eor v9.16b, v9.16b, v8.16b - eor v10.16b, v10.16b, v9.16b - eor \in\().16b, v10.16b, v11.16b - .endm - + .macro mix_columns, in, enc + .if \enc == 0 /* Inverse MixColumns: pre-multiply by { 5, 0, 4, 0 } */ - .macro inv_mix_columns, in - mul_by_x v11.16b, \in\().16b, v10.16b, v14.16b - mul_by_x v11.16b, v11.16b, v10.16b, v14.16b + mul_by_x v11.16b, \in\().16b, v10.16b, v12.16b + mul_by_x v11.16b, v11.16b, v10.16b, v12.16b eor \in\().16b, \in\().16b, v11.16b rev32 v11.8h, v11.8h eor \in\().16b, \in\().16b, v11.16b - mix_columns \in + .endif + + mul_by_x v10.16b, \in\().16b, v9.16b, v12.16b + rev32 v8.8h, \in\().8h + eor \in\().16b, \in\().16b, v10.16b + eor v10.16b, v10.16b, v8.16b + eor v11.16b, \in\().16b, v8.16b + tbl v11.16b, {v11.16b}, v14.16b + eor \in\().16b, v10.16b, v11.16b .endm .macro do_block, enc, in, rounds, rk, rkp, i @@ -88,16 +84,13 @@ add \rkp, \rk, #16 mov \i, \rounds 1111: eor \in\().16b, \in\().16b, v15.16b /* ^round key */ + movi v15.16b, #0x40 tbl \in\().16b, {\in\().16b}, v13.16b /* ShiftRows */ sub_bytes \in - ld1 {v15.4s}, [\rkp], #16 subs \i, \i, #1 + ld1 {v15.4s}, [\rkp], #16 beq 2222f - .if \enc == 1 - mix_columns \in - .else - inv_mix_columns \in - .endif + mix_columns \in, \enc b 1111b 2222: eor \in\().16b, \in\().16b, v15.16b /* ^round key */ .endm @@ -116,48 +109,48 @@ */ .macro sub_bytes_2x, in0, in1 - sub v8.16b, \in0\().16b, v12.16b - sub v9.16b, \in1\().16b, v12.16b + sub v8.16b, \in0\().16b, v15.16b tbl \in0\().16b, {v16.16b-v19.16b}, \in0\().16b + sub v9.16b, \in1\().16b, v15.16b tbl \in1\().16b, {v16.16b-v19.16b}, \in1\().16b - sub v10.16b, v8.16b, v12.16b - sub v11.16b, v9.16b, v12.16b + sub v10.16b, v8.16b, v15.16b tbx \in0\().16b, {v20.16b-v23.16b}, v8.16b + sub v11.16b, v9.16b, v15.16b tbx \in1\().16b, {v20.16b-v23.16b}, v9.16b - sub v8.16b, v10.16b, v12.16b - sub v9.16b, v11.16b, v12.16b + sub v8.16b, v10.16b, v15.16b tbx \in0\().16b, {v24.16b-v27.16b}, v10.16b + sub v9.16b, v11.16b, v15.16b tbx \in1\().16b, {v24.16b-v27.16b}, v11.16b tbx \in0\().16b, {v28.16b-v31.16b}, v8.16b tbx \in1\().16b, {v28.16b-v31.16b}, v9.16b .endm .macro sub_bytes_4x, in0, in1, in2, in3 - sub v8.16b, \in0\().16b, v12.16b + sub v8.16b, \in0\().16b, v15.16b tbl \in0\().16b, {v16.16b-v19.16b}, \in0\().16b - sub v9.16b, \in1\().16b, v12.16b + sub v9.16b, \in1\().16b, v15.16b tbl \in1\().16b, {v16.16b-v19.16b}, \in1\().16b - sub v10.16b, \in2\().16b, v12.16b + sub v10.16b, \in2\().16b, v15.16b tbl \in2\().16b, {v16.16b-v19.16b}, \in2\().16b - sub v11.16b, \in3\().16b, v12.16b + sub v11.16b, \in3\().16b, v15.16b tbl \in3\().16b, {v16.16b-v19.16b}, \in3\().16b tbx \in0\().16b, {v20.16b-v23.16b}, v8.16b tbx \in1\().16b, {v20.16b-v23.16b}, v9.16b - sub v8.16b, v8.16b, v12.16b + sub v8.16b, v8.16b, v15.16b tbx \in2\().16b, {v20.16b-v23.16b}, v10.16b - sub v9.16b, v9.16b, v12.16b + sub v9.16b, v9.16b, v15.16b tbx \in3\().16b, {v20.16b-v23.16b}, v11.16b - sub v10.16b, v10.16b, v12.16b + sub v10.16b, v10.16b, v15.16b tbx \in0\().16b, {v24.16b-v27.16b}, v8.16b - sub v11.16b, v11.16b, v12.16b + sub v11.16b, v11.16b, v15.16b tbx \in1\().16b, {v24.16b-v27.16b}, v9.16b - sub v8.16b, v8.16b, v12.16b + sub v8.16b, v8.16b, v15.16b tbx \in2\().16b, {v24.16b-v27.16b}, v10.16b - sub v9.16b, v9.16b, v12.16b + sub v9.16b, v9.16b, v15.16b tbx \in3\().16b, {v24.16b-v27.16b}, v11.16b - sub v10.16b, v10.16b, v12.16b + sub v10.16b, v10.16b, v15.16b tbx \in0\().16b, {v28.16b-v31.16b}, v8.16b - sub v11.16b, v11.16b, v12.16b + sub v11.16b, v11.16b, v15.16b tbx \in1\().16b, {v28.16b-v31.16b}, v9.16b tbx \in2\().16b, {v28.16b-v31.16b}, v10.16b tbx \in3\().16b, {v28.16b-v31.16b}, v11.16b @@ -174,59 +167,32 @@ eor \out1\().16b, \out1\().16b, \tmp1\().16b .endm - .macro mix_columns_2x, in0, in1 - mul_by_x_2x v8, v9, \in0, \in1, v10, v11, v14 - rev32 v10.8h, \in0\().8h - rev32 v11.8h, \in1\().8h - eor \in0\().16b, v8.16b, \in0\().16b - eor \in1\().16b, v9.16b, \in1\().16b - shl v12.4s, v10.4s, #24 - shl v13.4s, v11.4s, #24 - eor v8.16b, v8.16b, v10.16b - sri v12.4s, v10.4s, #8 - shl v10.4s, \in0\().4s, #24 - eor v9.16b, v9.16b, v11.16b - sri v13.4s, v11.4s, #8 - shl v11.4s, \in1\().4s, #24 - sri v10.4s, \in0\().4s, #8 - eor \in0\().16b, v8.16b, v12.16b - sri v11.4s, \in1\().4s, #8 - eor \in1\().16b, v9.16b, v13.16b - eor \in0\().16b, v10.16b, \in0\().16b - eor \in1\().16b, v11.16b, \in1\().16b - .endm - - .macro inv_mix_cols_2x, in0, in1 - mul_by_x_2x v8, v9, \in0, \in1, v10, v11, v14 - mul_by_x_2x v8, v9, v8, v9, v10, v11, v14 + .macro mix_columns_2x, in0, in1, enc + .if \enc == 0 + /* Inverse MixColumns: pre-multiply by { 5, 0, 4, 0 } */ + mul_by_x_2x v8, v9, \in0, \in1, v10, v11, v12 + mul_by_x_2x v8, v9, v8, v9, v10, v11, v12 eor \in0\().16b, \in0\().16b, v8.16b - eor \in1\().16b, \in1\().16b, v9.16b rev32 v8.8h, v8.8h + eor \in1\().16b, \in1\().16b, v9.16b rev32 v9.8h, v9.8h eor \in0\().16b, \in0\().16b, v8.16b eor \in1\().16b, \in1\().16b, v9.16b - mix_columns_2x \in0, \in1 - .endm + .endif - .macro inv_mix_cols_4x, in0, in1, in2, in3 - mul_by_x_2x v8, v9, \in0, \in1, v10, v11, v14 - mul_by_x_2x v10, v11, \in2, \in3, v12, v13, v14 - mul_by_x_2x v8, v9, v8, v9, v12, v13, v14 - mul_by_x_2x v10, v11, v10, v11, v12, v13, v14 + mul_by_x_2x v8, v9, \in0, \in1, v10, v11, v12 + rev32 v10.8h, \in0\().8h eor \in0\().16b, \in0\().16b, v8.16b + rev32 v11.8h, \in1\().8h eor \in1\().16b, \in1\().16b, v9.16b - eor \in2\().16b, \in2\().16b, v10.16b - eor \in3\().16b, \in3\().16b, v11.16b - rev32 v8.8h, v8.8h - rev32 v9.8h, v9.8h - rev32 v10.8h, v10.8h - rev32 v11.8h, v11.8h + eor v8.16b, v8.16b, v10.16b + eor \in0\().16b, \in0\().16b, v10.16b + eor v9.16b, v9.16b, v11.16b + eor \in1\().16b, \in1\().16b, v11.16b + tbl \in0\().16b, {\in0\().16b}, v14.16b + tbl \in1\().16b, {\in1\().16b}, v14.16b eor \in0\().16b, \in0\().16b, v8.16b eor \in1\().16b, \in1\().16b, v9.16b - eor \in2\().16b, \in2\().16b, v10.16b - eor \in3\().16b, \in3\().16b, v11.16b - mix_columns_2x \in0, \in1 - mix_columns_2x \in2, \in3 .endm .macro do_block_2x, enc, in0, in1 rounds, rk, rkp, i @@ -235,20 +201,14 @@ mov \i, \rounds 1111: eor \in0\().16b, \in0\().16b, v15.16b /* ^round key */ eor \in1\().16b, \in1\().16b, v15.16b /* ^round key */ - sub_bytes_2x \in0, \in1 + movi v15.16b, #0x40 tbl \in0\().16b, {\in0\().16b}, v13.16b /* ShiftRows */ tbl \in1\().16b, {\in1\().16b}, v13.16b /* ShiftRows */ - ld1 {v15.4s}, [\rkp], #16 + sub_bytes_2x \in0, \in1 subs \i, \i, #1 + ld1 {v15.4s}, [\rkp], #16 beq 2222f - .if \enc == 1 - mix_columns_2x \in0, \in1 - ldr q13, .LForward_ShiftRows - .else - inv_mix_cols_2x \in0, \in1 - ldr q13, .LReverse_ShiftRows - .endif - movi v12.16b, #0x40 + mix_columns_2x \in0, \in1, \enc b 1111b 2222: eor \in0\().16b, \in0\().16b, v15.16b /* ^round key */ eor \in1\().16b, \in1\().16b, v15.16b /* ^round key */ @@ -262,23 +222,17 @@ eor \in1\().16b, \in1\().16b, v15.16b /* ^round key */ eor \in2\().16b, \in2\().16b, v15.16b /* ^round key */ eor \in3\().16b, \in3\().16b, v15.16b /* ^round key */ - sub_bytes_4x \in0, \in1, \in2, \in3 + movi v15.16b, #0x40 tbl \in0\().16b, {\in0\().16b}, v13.16b /* ShiftRows */ tbl \in1\().16b, {\in1\().16b}, v13.16b /* ShiftRows */ tbl \in2\().16b, {\in2\().16b}, v13.16b /* ShiftRows */ tbl \in3\().16b, {\in3\().16b}, v13.16b /* ShiftRows */ - ld1 {v15.4s}, [\rkp], #16 + sub_bytes_4x \in0, \in1, \in2, \in3 subs \i, \i, #1 + ld1 {v15.4s}, [\rkp], #16 beq 2222f - .if \enc == 1 - mix_columns_2x \in0, \in1 - mix_columns_2x \in2, \in3 - ldr q13, .LForward_ShiftRows - .else - inv_mix_cols_4x \in0, \in1, \in2, \in3 - ldr q13, .LReverse_ShiftRows - .endif - movi v12.16b, #0x40 + mix_columns_2x \in0, \in1, \enc + mix_columns_2x \in2, \in3, \enc b 1111b 2222: eor \in0\().16b, \in0\().16b, v15.16b /* ^round key */ eor \in1\().16b, \in1\().16b, v15.16b /* ^round key */ @@ -305,19 +259,7 @@ #include "aes-modes.S" .text - .align 4 -.LForward_ShiftRows: -CPU_LE( .byte 0x0, 0x5, 0xa, 0xf, 0x4, 0x9, 0xe, 0x3 ) -CPU_LE( .byte 0x8, 0xd, 0x2, 0x7, 0xc, 0x1, 0x6, 0xb ) -CPU_BE( .byte 0xb, 0x6, 0x1, 0xc, 0x7, 0x2, 0xd, 0x8 ) -CPU_BE( .byte 0x3, 0xe, 0x9, 0x4, 0xf, 0xa, 0x5, 0x0 ) - -.LReverse_ShiftRows: -CPU_LE( .byte 0x0, 0xd, 0xa, 0x7, 0x4, 0x1, 0xe, 0xb ) -CPU_LE( .byte 0x8, 0x5, 0x2, 0xf, 0xc, 0x9, 0x6, 0x3 ) -CPU_BE( .byte 0x3, 0x6, 0x9, 0xc, 0xf, 0x2, 0x5, 0x8 ) -CPU_BE( .byte 0xb, 0xe, 0x1, 0x4, 0x7, 0xa, 0xd, 0x0 ) - + .align 6 .LForward_Sbox: .byte 0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5 .byte 0x30, 0x01, 0x67, 0x2b, 0xfe, 0xd7, 0xab, 0x76 @@ -385,3 +327,12 @@ CPU_BE( .byte 0xb, 0xe, 0x1, 0x4, 0x7, 0xa, 0xd, 0x0 ) .byte 0xc8, 0xeb, 0xbb, 0x3c, 0x83, 0x53, 0x99, 0x61 .byte 0x17, 0x2b, 0x04, 0x7e, 0xba, 0x77, 0xd6, 0x26 .byte 0xe1, 0x69, 0x14, 0x63, 0x55, 0x21, 0x0c, 0x7d + +.LForward_ShiftRows: + .octa 0x0b06010c07020d08030e09040f0a0500 + +.LReverse_ShiftRows: + .octa 0x0306090c0f0205080b0e0104070a0d00 + +.Lror32by8: + .octa 0x0c0f0e0d080b0a090407060500030201 From patchwork Tue Jan 17 15:22:37 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 91680 Delivered-To: patch@linaro.org Received: by 10.140.20.99 with SMTP id 90csp543301qgi; Tue, 17 Jan 2017 07:24:05 -0800 (PST) X-Received: by 10.84.130.5 with SMTP id 5mr58509213plc.69.1484666645388; Tue, 17 Jan 2017 07:24:05 -0800 (PST) Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e13si25201758pgf.220.2017.01.17.07.24.05; Tue, 17 Jan 2017 07:24:05 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751008AbdAQPYC (ORCPT + 1 other); Tue, 17 Jan 2017 10:24:02 -0500 Received: from mail-wm0-f48.google.com ([74.125.82.48]:35459 "EHLO mail-wm0-f48.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750924AbdAQPX4 (ORCPT ); Tue, 17 Jan 2017 10:23:56 -0500 Received: by mail-wm0-f48.google.com with SMTP id r126so205377522wmr.0 for ; Tue, 17 Jan 2017 07:23:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ElVhTH7rbhdNxoCETCPbiBrPki0mmyu75g9pZfBtnwA=; b=dRxJZQrQq8u+0FYeKPrjpyIv/qHN/wc89LeDhWCn+YqFZlmLNnFPi0k3SijCIJISLJ eXZSnjhcjQhZzbMY+2a6l4QPObUBRaw0KeGhejBlDHSYGR+2zbNTsaqmZ/IOdb6zWjZy 6enFeGbbktKaINr63OWl2VelrV2ug3putZwrk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ElVhTH7rbhdNxoCETCPbiBrPki0mmyu75g9pZfBtnwA=; b=WOIfl5lrWh72D9durTCsTTggycILHpc8qRaqjcOijHeZTPrbeW8S6rO1FuSMEKQ/VW 4+ZYWr/ZsddBr4QoB2Fkf4FM2IT4QVOgiKRCTe4399Um72lEVd6nu1HLeX0Vv5MVkyYF a6eT8xsDG/p1ZXN5HmdxfQIgS83P0+dgGf23UsrFBjzah95Vhi3/f08Vb9w34Rtz9oEz 7otZ5QF4b177LfeE1ibg5QB7xWXTQQdMMHrkcoukPXtLkmTL+IyUe8ZdYFMWgR2371FA I6JN/5jQhs6YCKZxBV8GlQrWzHNy0LPTMrijOo54GFUGDtHibzZ613wXsyGkVccGemVQ whXA== X-Gm-Message-State: AIkVDXLUO9FisqUO21d4YE4XfaeIxT9nzIUNuf1CmAhQNl0MXYglttD4MkFa8u+FZr2Wlhgf X-Received: by 10.28.172.7 with SMTP id v7mr15722294wme.37.1484666592597; Tue, 17 Jan 2017 07:23:12 -0800 (PST) Received: from localhost.localdomain ([160.167.203.25]) by smtp.gmail.com with ESMTPSA id y127sm37590359wmg.12.2017.01.17.07.23.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 17 Jan 2017 07:23:11 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au Cc: linux-arm-kernel@lists.infradead.org, Ard Biesheuvel Subject: [PATCH 10/10] crypto: arm64/aes - replace scalar fallback with plain NEON fallback Date: Tue, 17 Jan 2017 15:22:37 +0000 Message-Id: <1484666557-31458-11-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> References: <1484666557-31458-1-git-send-email-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The new bitsliced NEON implementation of AES uses a fallback in two places: CBC encryption (which is strictly sequential, whereas this driver can only operate efficiently on 8 blocks at a time), and the XTS tweak generation, which involves encrypting a single AES block with a different key schedule. The plain (i.e., non-bitsliced) NEON code is more suitable as a fallback, given that it is faster than scalar on low end cores (which is what the NEON implementations target, since high end cores have dedicated instructions for AES), and shows similar behavior in terms of D-cache footprint and sensitivity to cache timing attacks. So switch the fallback handling to the plain NEON driver. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/Kconfig | 2 +- arch/arm64/crypto/aes-neonbs-glue.c | 38 ++++++++++++++------ 2 files changed, 29 insertions(+), 11 deletions(-) -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-crypto" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index 5de75c3dcbd4..bed7feddfeed 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -86,7 +86,7 @@ config CRYPTO_AES_ARM64_BS tristate "AES in ECB/CBC/CTR/XTS modes using bit-sliced NEON algorithm" depends on KERNEL_MODE_NEON select CRYPTO_BLKCIPHER - select CRYPTO_AES_ARM64 + select CRYPTO_AES_ARM64_NEON_BLK select CRYPTO_SIMD endif diff --git a/arch/arm64/crypto/aes-neonbs-glue.c b/arch/arm64/crypto/aes-neonbs-glue.c index 323dd76ae5f0..863e436ecf89 100644 --- a/arch/arm64/crypto/aes-neonbs-glue.c +++ b/arch/arm64/crypto/aes-neonbs-glue.c @@ -10,7 +10,6 @@ #include #include -#include #include #include #include @@ -42,7 +41,12 @@ asmlinkage void aesbs_xts_encrypt(u8 out[], u8 const in[], u8 const rk[], asmlinkage void aesbs_xts_decrypt(u8 out[], u8 const in[], u8 const rk[], int rounds, int blocks, u8 iv[]); -asmlinkage void __aes_arm64_encrypt(u32 *rk, u8 *out, const u8 *in, int rounds); +/* borrowed from aes-neon-blk.ko */ +asmlinkage void neon_aes_ecb_encrypt(u8 out[], u8 const in[], u32 const rk[], + int rounds, int blocks, int first); +asmlinkage void neon_aes_cbc_encrypt(u8 out[], u8 const in[], u32 const rk[], + int rounds, int blocks, u8 iv[], + int first); struct aesbs_ctx { u8 rk[13 * (8 * AES_BLOCK_SIZE) + 32]; @@ -140,16 +144,28 @@ static int aesbs_cbc_setkey(struct crypto_skcipher *tfm, const u8 *in_key, return 0; } -static void cbc_encrypt_one(struct crypto_skcipher *tfm, const u8 *src, u8 *dst) +static int cbc_encrypt(struct skcipher_request *req) { + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct aesbs_cbc_ctx *ctx = crypto_skcipher_ctx(tfm); + struct skcipher_walk walk; + int err, first = 1; - __aes_arm64_encrypt(ctx->enc, dst, src, ctx->key.rounds); -} + err = skcipher_walk_virt(&walk, req, true); -static int cbc_encrypt(struct skcipher_request *req) -{ - return crypto_cbc_encrypt_walk(req, cbc_encrypt_one); + kernel_neon_begin(); + while (walk.nbytes >= AES_BLOCK_SIZE) { + unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; + + /* fall back to the non-bitsliced NEON implementation */ + neon_aes_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->enc, ctx->key.rounds, blocks, walk.iv, + first); + err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); + first = 0; + } + kernel_neon_end(); + return err; } static int cbc_decrypt(struct skcipher_request *req) @@ -254,9 +270,11 @@ static int __xts_crypt(struct skcipher_request *req, err = skcipher_walk_virt(&walk, req, true); - __aes_arm64_encrypt(ctx->twkey, walk.iv, walk.iv, ctx->key.rounds); - kernel_neon_begin(); + + neon_aes_ecb_encrypt(walk.iv, walk.iv, ctx->twkey, + ctx->key.rounds, 1, 1); + while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE;