From patchwork Mon Jul 24 10:28:02 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 108547 Delivered-To: patch@linaro.org Received: by 10.140.101.44 with SMTP id t41csp3887308qge; Mon, 24 Jul 2017 03:28:31 -0700 (PDT) X-Received: by 10.99.126.86 with SMTP id o22mr15606156pgn.367.1500892111201; Mon, 24 Jul 2017 03:28:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1500892111; cv=none; d=google.com; s=arc-20160816; b=B1WgvFeu4exNChE6jzGy26sPPs4jtdHWceuafd9k3jQSlRD6OOB0pMejJmO8tmKrt+ 2WwHKS//eNsaszrNi6lxoTR6P1hnvkCuDl70jPkYwoB5arA4078I/fwxK19wFff544fr Cn+XPQRiFEmo5VzK2dzDdbXu5ytKintEx0GaSh39wqlVHhgDuWIkqe8ggQsApozesn5J e5Ls/5s4fvpG1I5hpbrH39dZIHmO5joiQzkxlgJyBFTOz6Is2DsXjebr0SEJ8ekO9nbV 2M6GswkjiONFzxVLJtuc4sjCc7iFsl4+5ZhPOE81qPg5z3JZFXxJNXdybjKOcBxybEwQ s0iA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=bMTua6OX/RKu4qY5yuBIl/nVUGA/yTqyb4ZCaruJ8jI=; b=B5gUwXtvJBtGoEG/GPvIX+AVPE8Dth1wT5jgNVYwQO/iDf1dZA89snMsAEkWO2CJfY zv4fT4L4a7s2Ci/8hx8tF5MreaAoJQxA4ZN7oaXy1GYZ9o5bex63N+BHsLn6wH5UIHce Amhz4rAmQznF4fSyRbCya/U7hslInE6czIsBub+zYDd0vVmZ77reCOfpdChLMoIywx7Z kvuvk6TN9VH2L5jWnII2ZBZUe273dpJebde+e61BgrIxtxn+YYE66ULMmaMU6zU96aUJ kKlXRpih/LPnfFTJnwXTd2qkXqPDpcJxUr3/bB/Cf6Yuupcxp1CCmHJi0B09XhDIfnAU oCbQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.b=chPYlENz; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 38si3348411pld.76.2017.07.24.03.28.30; Mon, 24 Jul 2017 03:28:31 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.b=chPYlENz; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751520AbdGXK2a (ORCPT + 1 other); Mon, 24 Jul 2017 06:28:30 -0400 Received: from mail-wm0-f52.google.com ([74.125.82.52]:38854 "EHLO mail-wm0-f52.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751413AbdGXK23 (ORCPT ); Mon, 24 Jul 2017 06:28:29 -0400 Received: by mail-wm0-f52.google.com with SMTP id m85so13405575wma.1 for ; Mon, 24 Jul 2017 03:28:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id; bh=bMTua6OX/RKu4qY5yuBIl/nVUGA/yTqyb4ZCaruJ8jI=; b=chPYlENz6Zb2gsbsulSN7fV3A0fqeOqh4rggw+nv1vAWzFIGJCll8Sn2ztkVZtH6QV fzWVxhkRzYlBDObia1/y8KrCdYgZknchpPR8Ou693AlEEoiNWbNeF9WfdiPEQqt2cyS3 yW9R//GCR0aUB/UMztk4BM72Cofs7+lJN8bvQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=bMTua6OX/RKu4qY5yuBIl/nVUGA/yTqyb4ZCaruJ8jI=; b=RGPaP0bUfHB3okOe2VwvYQD1igsY1rc4tM0UuBiadF0acWGHKSEXduHsmkjAqtOWls FEAQqj4lFwDyJPVSh7nl/xs3THGZrTZe3mah2/TJTn5nujWs6K+YHgcE8XjKPBIz0QjT W39Ap2CgkGHhytYBZ9P+9QWHK/hEd1UxJ8qSzk6UjjfMcMnrM8g2l0bTpWufC0cAuMMb lwmHmlFtpMZufen37/0YS+cD7ZqQ4IB4dxB4QhiGeMU3oQR/b406XCI6oa9gc3SrKMut UEnRNCinFeeoZHX+t6Kywf/Q9s6yZaeFrDiEI7dYEoVwOiD8jkMzQL+Ip07CUfsTtDPx 7S8w== X-Gm-Message-State: AIVw110c/yvremRS6n4Wk8yUaZnOf8giCPQ0CO0lHMe5sExqeKdnCJI9 JcJQxH71yooGsWVWddnDJA== X-Received: by 10.28.155.84 with SMTP id d81mr4719269wme.163.1500892107935; Mon, 24 Jul 2017 03:28:27 -0700 (PDT) Received: from localhost.localdomain ([105.148.195.69]) by smtp.gmail.com with ESMTPSA id v44sm13205400wrb.53.2017.07.24.03.28.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 24 Jul 2017 03:28:26 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: herbert@gondor.apana.org.au, dave.martin@arm.com, Ard Biesheuvel Subject: [PATCH resend 00/18] crypto: ARM/arm64 roundup for v4.14 Date: Mon, 24 Jul 2017 11:28:02 +0100 Message-Id: <20170724102820.16534-1-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.9.3 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org This is a resend of all the patches I sent out recently that I would like to be considered for v4.14. Their main purpose is to prepare the arm64 crypto code to deal with situations where the SIMD register file is unavailable, which never occurs at present, but this will change in the future when support for SVE is added. Patches #1 and #2 have been sent out last week as 'crypto/algapi - refactor crypto_xor() to avoid memcpy()s' (v2). This version of #2 fixes an error caught by kbuild. The non-SIMD fallback code added in the remaining patches relies on crypto_xor() extensively, which is why these patches have been included here. Patches #3 - #13 implement the non-SIMD fallbacks for the various NEON based drivers. Patch #14 implements AES-GCM natively instead of relying on the generic GCM module to wire accelerated AES-CTR and GHASH together, resulting in a ~37% speedup. Patches #15 and #16 implement an accelerated GHASH algorithm for ARM cores that lack the 64x64 PMULL instruction. Patches #17 and #18 update the scalar AES implementations to stop using the expanded lookup tables for the final round. This reduces the Dcache footprint, and thus the key correlated jitter. This supersedes all other crypto patches I have outstanding, including the AES refactor ones which I will rework later. Ard Biesheuvel (18): crypto/algapi - use separate dst and src operands for __crypto_xor() crypto/algapi - make crypto_xor() take separate dst and src arguments crypto: arm64/ghash-ce - add non-SIMD scalar fallback crypto: arm64/crct10dif - add non-SIMD generic fallback crypto: arm64/crc32 - add non-SIMD scalar fallback crypto: arm64/sha1-ce - add non-SIMD generic fallback crypto: arm64/sha2-ce - add non-SIMD scalar fallback crypto: arm64/aes-ce-cipher - match round key endianness with generic code crypto: arm64/aes-ce-cipher: add non-SIMD generic fallback crypto: arm64/aes-ce-ccm: add non-SIMD generic fallback crypto: arm64/aes-blk - add a non-SIMD fallback for synchronous CTR crypto: arm64/chacha20 - take may_use_simd() into account crypto: arm64/aes-bs - implement non-SIMD fallback for AES-CTR crypto: arm64/gcm - implement native driver using v8 Crypto Extensions crypto: arm/ghash - add NEON accelerated fallback for vmull.p64 crypto: arm64/ghash - add NEON accelerated fallback for 64-bit PMULL crypto: arm/aes - avoid expanded lookup tables in the final round crypto: arm64/aes - avoid expanded lookup tables in the final round arch/arm/crypto/Kconfig | 5 +- arch/arm/crypto/aes-ce-glue.c | 4 +- arch/arm/crypto/aes-cipher-core.S | 88 +++- arch/arm/crypto/aes-neonbs-glue.c | 5 +- arch/arm/crypto/ghash-ce-core.S | 234 +++++++-- arch/arm/crypto/ghash-ce-glue.c | 24 +- arch/arm64/crypto/Kconfig | 22 +- arch/arm64/crypto/aes-ce-ccm-core.S | 30 +- arch/arm64/crypto/aes-ce-ccm-glue.c | 174 +++++-- arch/arm64/crypto/aes-ce-cipher.c | 55 ++- arch/arm64/crypto/aes-ce.S | 12 +- arch/arm64/crypto/aes-cipher-core.S | 152 ++++-- arch/arm64/crypto/aes-ctr-fallback.h | 53 ++ arch/arm64/crypto/aes-glue.c | 63 ++- arch/arm64/crypto/aes-neonbs-glue.c | 53 +- arch/arm64/crypto/chacha20-neon-glue.c | 5 +- arch/arm64/crypto/crc32-ce-glue.c | 11 +- arch/arm64/crypto/crct10dif-ce-glue.c | 13 +- arch/arm64/crypto/ghash-ce-core.S | 401 ++++++++++++++- arch/arm64/crypto/ghash-ce-glue.c | 517 ++++++++++++++++++-- arch/arm64/crypto/sha1-ce-glue.c | 18 +- arch/arm64/crypto/sha2-ce-glue.c | 30 +- arch/arm64/crypto/sha256-glue.c | 1 + arch/sparc/crypto/aes_glue.c | 3 +- arch/x86/crypto/aesni-intel_glue.c | 4 +- arch/x86/crypto/blowfish_glue.c | 3 +- arch/x86/crypto/cast5_avx_glue.c | 3 +- arch/x86/crypto/des3_ede_glue.c | 3 +- crypto/algapi.c | 25 +- crypto/ctr.c | 3 +- crypto/pcbc.c | 12 +- drivers/crypto/vmx/aes_ctr.c | 3 +- drivers/md/dm-crypt.c | 11 +- include/crypto/algapi.h | 23 +- 34 files changed, 1719 insertions(+), 344 deletions(-) create mode 100644 arch/arm64/crypto/aes-ctr-fallback.h -- 2.9.3