From patchwork Wed Dec 6 19:43:34 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 120891 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7468832qgn; Wed, 6 Dec 2017 11:44:31 -0800 (PST) X-Google-Smtp-Source: AGs4zMZJhY1miJe/m/rJART59ELOUC3eedlcP9BW7afoumF96m2I/VQ3/P+lFNGPaBkKAzkgirrM X-Received: by 10.98.21.17 with SMTP id 17mr4028097pfv.120.1512589471589; Wed, 06 Dec 2017 11:44:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512589471; cv=none; d=google.com; s=arc-20160816; b=bF/j8VY8hVUMOXSUnsy8fKqX8j3cYnm1MnmZy93zQpAt4tiiLkE/xkd/i5yvUkFtGy ICJVgYH3FswYuePyfRZ7+TwRJFeYDZBkx+2rdzModNuNk8p7yZQm2e1vbnAzF81WK/Y3 eRaM9StyGFAGu6fgTIHOvfRhpI+y8CpS4H8pze72k8x3Pe5Dct0RiR+afYFph83CnHkN kMhCf0B+uO5LAf4ArIuVbqZRoKEDcsWl8jMO5we5OwKYMx/RyMhMfWn6KWPSQd3GkAbg 0GP40tOLhJVs0uG7OWNZyjgKkGYY2Beg8/SZkeuCfs4n2rWMyRBdZOov7ncTuvCajPYG pRrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=4L8VBFJ8/vUZayezPEZFRvTau06CygKiThORT60NjPo=; b=VF3xQ7vtXM2EJ9Hhi+9yf9sLjbfuaWNRBr6NeD4GElabvbwdzSUvcvcnRRztGqSA7h 4FRJ4LMKRV/AgPsxPTf6YZ9U+tdlYGNklE3C95l41Of/U4SziBk3Y2/xeyAa1Ce82bz8 23IbNh5sASV0nVKMMKOskJZhkW86qdxNb6/7+mkpUlljTxHDx0YO4hFM8nwBZydgPN+7 7MbIkuc5t1gBnv0vcwqYBz+tilGnd+ZVShRWoC6czauQIkK+YHqKxkeN/nSuUzzYlZft gbEHTNsAdkwEZBf+ORW2+t31HFhYjRhfK9xaw7WONsKkULv7VQF1XhQ6NRuBZ9rSQ8bW j0kQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=BfeaCIe3; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i16si2382524pgv.496.2017.12.06.11.44.31; Wed, 06 Dec 2017 11:44:31 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=BfeaCIe3; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752344AbdLFTo3 (ORCPT + 4 others); Wed, 6 Dec 2017 14:44:29 -0500 Received: from mail-wr0-f196.google.com ([209.85.128.196]:35633 "EHLO mail-wr0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752132AbdLFToT (ORCPT ); Wed, 6 Dec 2017 14:44:19 -0500 Received: by mail-wr0-f196.google.com with SMTP id g53so5128980wra.2 for ; Wed, 06 Dec 2017 11:44:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=AVGr1XiAAq1ruDzOgmfaJd5iqwO5GoptNYnkdzAZJA8=; b=BfeaCIe3bP4WigQt68E0qH/qrUXmozDFVhJ9mVfSpVq8sWTT3H1/xKTP0S5ULBRNeE i+UVvst52Nr+FIHkQbwrd1cASaskFmYHscpH0XkAXRAKmXEWcmpRgCZY8D9nFRxbQ+JJ KYB59CdIaJwWuiJVhT3r6HljuCYh8nsmJ7Maw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=AVGr1XiAAq1ruDzOgmfaJd5iqwO5GoptNYnkdzAZJA8=; b=KeffHrOJLfaTOLFIgpqxBfqy3Er3RKEgzyox8GccWy2m5BwQKJMX188QXKws6+9AqY hcd+vs2xHnkiUkiuUsLE5LyFgOS0pUP8HqjIUZLKcA/CGWR6K0STbt6RkSYStjzi5iCE Rpqkg25f3KxVzNYphG1XSfsdbRK/VJQOY9kH2cYeeeXtohFPw4spTaxoyPGbtb+fVYCx PFh5Z5hBUD50uGoOsmoezpe5AA1qAjY+SDcqQfMCAxcvIhRGvioavfinB3gTO1O8pPk/ 5twPub4UK2fu8igicJIUjGcJXpvXFuW/Qd5jbh//1TWBmiHgjWA+qrgQ8Lz2ZzcoTvM4 IeGA== X-Gm-Message-State: AKGB3mJYS5DLJjRGqScdZLOhb7R8SBYNynnfDIaUQyUOhbE09qTu9YK6 WJx6osX9YYSESSW5c8jFMssIRg== X-Received: by 10.223.134.184 with SMTP id 53mr509613wrx.17.1512589458265; Wed, 06 Dec 2017 11:44:18 -0800 (PST) Received: from localhost.localdomain ([105.150.171.234]) by smtp.gmail.com with ESMTPSA id b66sm3596594wmh.32.2017.12.06.11.44.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Dec 2017 11:44:17 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v3 08/20] crypto: arm64/aes-blk - add 4 way interleave to CBC-MAC encrypt path Date: Wed, 6 Dec 2017 19:43:34 +0000 Message-Id: <20171206194346.24393-9-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171206194346.24393-1-ard.biesheuvel@linaro.org> References: <20171206194346.24393-1-ard.biesheuvel@linaro.org> Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org CBC MAC is strictly sequential, and so the current AES code simply processes the input one block at a time. However, we are about to add yield support, which adds a bit of overhead, and which we prefer to align with other modes in terms of granularity (i.e., it is better to have all routines yield every 64 bytes and not have an exception for CBC MAC which yields every 16 bytes) So unroll the loop by 4. We still cannot perform the AES algorithm in parallel, but we can at least merge the loads and stores. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-modes.S | 23 ++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) -- 2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index e86535a1329d..a68412e1e3a4 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -395,8 +395,28 @@ AES_ENDPROC(aes_xts_decrypt) AES_ENTRY(aes_mac_update) ld1 {v0.16b}, [x4] /* get dg */ enc_prepare w2, x1, x7 - cbnz w5, .Lmacenc + cbz w5, .Lmacloop4x + encrypt_block v0, w2, x1, x7, w8 + +.Lmacloop4x: + subs w3, w3, #4 + bmi .Lmac1x + ld1 {v1.16b-v4.16b}, [x0], #64 /* get next pt block */ + eor v0.16b, v0.16b, v1.16b /* ..and xor with dg */ + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v2.16b + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v3.16b + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v4.16b + cmp w3, wzr + csinv x5, x6, xzr, eq + cbz w5, .Lmacout + encrypt_block v0, w2, x1, x7, w8 + b .Lmacloop4x +.Lmac1x: + add w3, w3, #4 .Lmacloop: cbz w3, .Lmacout ld1 {v1.16b}, [x0], #16 /* get next pt block */ @@ -406,7 +426,6 @@ AES_ENTRY(aes_mac_update) csinv x5, x6, xzr, eq cbz w5, .Lmacout -.Lmacenc: encrypt_block v0, w2, x1, x7, w8 b .Lmacloop