From patchwork Tue Jan 16 08:29:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 124615 Delivered-To: patch@linaro.org Received: by 10.46.64.148 with SMTP id r20csp936526lje; Tue, 16 Jan 2018 00:29:37 -0800 (PST) X-Google-Smtp-Source: ACJfBovnsNpt1ozFEvkf1f8la7vX/OTIJyhDnv/AK/v/622SYrnX7hLdM46PloTEW2PLd1PIOP+F X-Received: by 10.98.59.80 with SMTP id i77mr26693091pfa.146.1516091377626; Tue, 16 Jan 2018 00:29:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1516091377; cv=none; d=google.com; s=arc-20160816; b=LyDH0FlFTDxcgI7nlDyt9GzxU+roiU6aLGB40CVR/UZGl5Q6+bCDq/GoG5K6QDs6Ac h5ias/lYs2rjJyxrmHzv+eo28XUGnofVVyLn9ls2E744Dt6ofzQI37BUsPmqe33mxxP6 A+uXtRYQ7+i4n+kIPkmd+V0OMCdOojms0qK+iAx0gePBlZ9HN0gC9WkQbAb0Zm8SC/it hwpVLc2g1t19vgeHLvKhUhQI48Rf/MifNSzBIg4YN/JMHrmzl43Tsg/I0JpOcnMB+Qhp BopWIgh0ongN2kJJvVVC6p4t7mqpJGhLZ3hxvLxRUgRfjjFEMY64XaaGgVIPxDwhdlqZ 2+WA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=wWDxzV5EuNnKoNFgBvuu+oc53I5seOdEw6wT1j/tJUo=; b=Kn2DClnzb/v8c1J/yiTCuVYbitURpb2ZsALjEmOGcRDO1icWGYab7DVjiNaGLbt9QU 9i91RENfx3Vlnrl2OwoSta+2d1HTEhjVAluQ4D5zsIjv9jJaNL0eXGopWGpKGBKDiqmz 7/12U6BCxaFkLjg7qtjgqRuWJKR5F6kl3Ns2H7C8/95OBkcV59gTt03XfzD37su36nmd xK5gvE/GkpdL+ZmfvBt9fVmVOA+/M4Q7DjXrpDjuEEfAW4puATyYPfOT9mztYS3grl7D W12VCWiq4QGLbnp/z4SHi+Xhgy6hv2SaEN9nidY6cpiHq//D61nNYeDIPfiL0LLN7cm1 HLhA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=FfiIL8MG; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f6si1405936pfg.125.2018.01.16.00.29.37; Tue, 16 Jan 2018 00:29:37 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=FfiIL8MG; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751009AbeAPI3g (ORCPT + 1 other); Tue, 16 Jan 2018 03:29:36 -0500 Received: from mail-wr0-f196.google.com ([209.85.128.196]:37931 "EHLO mail-wr0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750798AbeAPI3f (ORCPT ); Tue, 16 Jan 2018 03:29:35 -0500 Received: by mail-wr0-f196.google.com with SMTP id x1so10125457wrb.5 for ; Tue, 16 Jan 2018 00:29:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id; bh=wWDxzV5EuNnKoNFgBvuu+oc53I5seOdEw6wT1j/tJUo=; b=FfiIL8MGJxs/Q0z9BAVaZ62qkM3+HnD7xCXCBFu+3ZKG4RWjmVeh+tf8Rt5CvnGHag LtuF1rZOzGkD4wYHBtsDj6UUI25lQhGWD13ajILR9VJW0bcqIPAOB9MPw3ad1FMPxg5s 1dCAuF9uSN3HXVA8ZeA69xxaCBGRoaJQVHeyk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=wWDxzV5EuNnKoNFgBvuu+oc53I5seOdEw6wT1j/tJUo=; b=phWkAhIQgUQVPQ4dSjL08HqVOqjhOtNlAoCbphTvHAgGYr4a2Q/YgCacNRZLv8q5bj MKBUK4y3kBT/8SsvO9URHBWLGfBKLGgskSvyyFjQSi+9+MeCWdJMzSc2vLO13o6DFBtv d8V3m5LxU4Ji14DxGz67DvbtY+ZlKVA0Akf8VgGkgHDWSJrgzfh/GRQOt/apFeg+ouFy w3uoGO2HRHw7gPaGaoaX2GejVsjTUVOo5f3M/QoqauY3NqR9GsD2sfcGJBkPD+01NHmx VIH/3RUZtT32IgoTPm41CQZnHrvNSizlJCZpwdi6HQcG2Uw7OZko5iRlf1YTywC/cNb1 I+Ug== X-Gm-Message-State: AKwxyteZG+5EosaEHXLFbsZMc6RloN8X+dHebshc1CUDz9QlY5M15aCW yaByETL2ZIbzTq3G426dEoZX05p8C3Q= X-Received: by 10.223.157.146 with SMTP id p18mr15981063wre.71.1516091373712; Tue, 16 Jan 2018 00:29:33 -0800 (PST) Received: from localhost.localdomain ([160.90.52.105]) by smtp.gmail.com with ESMTPSA id 34sm946440wro.7.2018.01.16.00.29.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Jan 2018 00:29:32 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, steve.capper@arm.com, catalin.marinas@arm.com, will.deacon@arm.com, gilad@benyossef.com, Ard Biesheuvel Subject: [RFT PATCH] crypto: arm64 - implement SM3 secure hash using special instructions Date: Tue, 16 Jan 2018 08:29:22 +0000 Message-Id: <20180116082922.19465-1-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Implement the Chinese SM3 secure hash algorithm using the new special instructions that have been introduced as an optional extension in ARMv8.2. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/Kconfig | 5 ++ arch/arm64/crypto/Makefile | 3 + arch/arm64/crypto/sm3-ce-core.S | 142 ++++++++++++++++++++++++++++++++++++++++ arch/arm64/crypto/sm3-ce-glue.c | 92 ++++++++++++++++++++++++++ 4 files changed, 242 insertions(+) create mode 100644 arch/arm64/crypto/sm3-ce-core.S create mode 100644 arch/arm64/crypto/sm3-ce-glue.c -- 2.11.0 Tested-by: Steve Capper Reviewed-by: Gilad Ben-Yossef diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index 71293e049a5d..225c3842644c 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -105,4 +105,9 @@ config CRYPTO_AES_ARM64_BS select CRYPTO_AES_ARM64 select CRYPTO_SIMD +config CRYPTO_SM3_ARM64_CE + tristate "SM3 digest algorithm (ARMv8.2 Crypto Extensions)" + depends on KERNEL_MODE_NEON + select CRYPTO_HASH + endif diff --git a/arch/arm64/crypto/Makefile b/arch/arm64/crypto/Makefile index 267764473ef6..989d6e51f6c9 100644 --- a/arch/arm64/crypto/Makefile +++ b/arch/arm64/crypto/Makefile @@ -56,6 +56,9 @@ aes-arm64-y := aes-cipher-core.o aes-cipher-glue.o obj-$(CONFIG_CRYPTO_AES_ARM64_BS) += aes-neon-bs.o aes-neon-bs-y := aes-neonbs-core.o aes-neonbs-glue.o +obj-$(CONFIG_CRYPTO_SM3_ARM64_CE) += sm3-ce.o +sm3-ce-y := sm3-ce-glue.o sm3-ce-core.o + AFLAGS_aes-ce.o := -DINTERLEAVE=4 AFLAGS_aes-neon.o := -DINTERLEAVE=4 diff --git a/arch/arm64/crypto/sm3-ce-core.S b/arch/arm64/crypto/sm3-ce-core.S new file mode 100644 index 000000000000..961d01764886 --- /dev/null +++ b/arch/arm64/crypto/sm3-ce-core.S @@ -0,0 +1,142 @@ +/* + * sm3-ce-core.S - SM3 secure hash using ARMv8.2 Crypto Extensions + * + * Copyright (C) 2018 Linaro Ltd + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include + + .irp b,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 + .set .Lv\b\().4s, \b + .endr + + .macro sm3partw1, rd, rn, rm + .inst 0xce60c000 | .L\rd | (.L\rn << 5) | (.L\rm << 16) + .endm + + .macro sm3partw2, rd, rn, rm + .inst 0xce60c400 | .L\rd | (.L\rn << 5) | (.L\rm << 16) + .endm + + .macro sm3ss1, rd, rn, rm, ra + .inst 0xce400000 | .L\rd | (.L\rn << 5) | (.L\ra << 10) | (.L\rm << 16) + .endm + + .macro sm3tt1a, rd, rn, rm, imm2 + .inst 0xce408000 | .L\rd | (.L\rn << 5) | ((\imm2) << 12) | (.L\rm << 16) + .endm + + .macro sm3tt1b, rd, rn, rm, imm2 + .inst 0xce408400 | .L\rd | (.L\rn << 5) | ((\imm2) << 12) | (.L\rm << 16) + .endm + + .macro sm3tt2a, rd, rn, rm, imm2 + .inst 0xce408800 | .L\rd | (.L\rn << 5) | ((\imm2) << 12) | (.L\rm << 16) + .endm + + .macro sm3tt2b, rd, rn, rm, imm2 + .inst 0xce408c00 | .L\rd | (.L\rn << 5) | ((\imm2) << 12) | (.L\rm << 16) + .endm + + .macro round, ab, s0, t0, t1, i + sm3ss1 v10.4s, v8.4s, \t0\().4s, v9.4s + shl \t1\().4s, \t0\().4s, #1 + sri \t1\().4s, \t0\().4s, #31 + sm3tt1\ab v8.4s, v10.4s, v15.4s, \i + sm3tt2\ab v9.4s, v10.4s, \s0\().4s, \i + .endm + + .macro qround, ab, s0, s1, s2, s3, s4 + .ifnb \s4 + ext \s4\().16b, \s1\().16b, \s2\().16b, #12 + ext v13.16b, \s0\().16b, \s1\().16b, #12 + ext v14.16b, \s2\().16b, \s3\().16b, #8 + sm3partw1 \s4\().4s, \s0\().4s, \s3\().4s + .endif + + eor v15.16b, \s0\().16b, \s1\().16b + + round \ab, \s0, v11, v12, 0 + round \ab, \s0, v12, v11, 1 + round \ab, \s0, v11, v12, 2 + round \ab, \s0, v12, v11, 3 + + .ifnb \s4 + sm3partw2 \s4\().4s, v14.4s, v13.4s + .endif + .endm + + /* + * void sm3_ce_transform(struct sm3_state *sst, u8 const *src, + * int blocks) + */ + .text +ENTRY(sm3_ce_transform) + /* load state */ + ld1 {v8.4s-v9.4s}, [x0] + rev64 v8.4s, v8.4s + rev64 v9.4s, v9.4s + ext v8.16b, v8.16b, v8.16b, #8 + ext v9.16b, v9.16b, v9.16b, #8 + + adrp x8, .Lt + ldr s16, [x8, :lo12:.Lt] + ldr s17, [x8, :lo12:.Lt + 4] + + /* load input */ +0: ld1 {v0.16b-v3.16b}, [x1], #64 + sub w2, w2, #1 + + mov v18.16b, v8.16b + mov v19.16b, v9.16b + +CPU_LE( rev32 v0.16b, v0.16b ) +CPU_LE( rev32 v1.16b, v1.16b ) +CPU_LE( rev32 v2.16b, v2.16b ) +CPU_LE( rev32 v3.16b, v3.16b ) + + ext v11.16b, v16.16b, v16.16b, #4 + + qround a, v0, v1, v2, v3, v4 + qround a, v1, v2, v3, v4, v5 + qround a, v2, v3, v4, v5, v6 + qround a, v3, v4, v5, v6, v7 + + ext v11.16b, v17.16b, v17.16b, #4 + + qround b, v4, v5, v6, v7, v0 + qround b, v5, v6, v7, v0, v1 + qround b, v6, v7, v0, v1, v2 + qround b, v7, v0, v1, v2, v3 + qround b, v0, v1, v2, v3, v4 + qround b, v1, v2, v3, v4, v5 + qround b, v2, v3, v4, v5, v6 + qround b, v3, v4, v5, v6, v7 + qround b, v4, v5, v6, v7, v0 + qround b, v5, v6 + qround b, v6, v7 + qround b, v7, v0 + + eor v8.16b, v8.16b, v18.16b + eor v9.16b, v9.16b, v19.16b + + /* handled all input blocks? */ + cbnz w2, 0b + + /* save state */ + rev64 v8.4s, v8.4s + rev64 v9.4s, v9.4s + ext v8.16b, v8.16b, v8.16b, #8 + ext v9.16b, v9.16b, v9.16b, #8 + st1 {v8.4s-v9.4s}, [x0] + ret +ENDPROC(sm3_ce_transform) + + .section ".rodata", "a" + .align 3 +.Lt: .word 0x79cc4519, 0x9d8a7a87 diff --git a/arch/arm64/crypto/sm3-ce-glue.c b/arch/arm64/crypto/sm3-ce-glue.c new file mode 100644 index 000000000000..3b4948f7e26f --- /dev/null +++ b/arch/arm64/crypto/sm3-ce-glue.c @@ -0,0 +1,92 @@ +/* + * sm3-ce-glue.c - SM3 secure hash using ARMv8.2 Crypto Extensions + * + * Copyright (C) 2018 Linaro Ltd + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +MODULE_DESCRIPTION("SM3 secure hash using ARMv8 Crypto Extensions"); +MODULE_AUTHOR("Ard Biesheuvel "); +MODULE_LICENSE("GPL v2"); + +asmlinkage void sm3_ce_transform(struct sm3_state *sst, u8 const *src, + int blocks); + +static int sm3_ce_update(struct shash_desc *desc, const u8 *data, + unsigned int len) +{ + if (!may_use_simd()) + return crypto_sm3_update(desc, data, len); + + kernel_neon_begin(); + sm3_base_do_update(desc, data, len, sm3_ce_transform); + kernel_neon_end(); + + return 0; +} + +static int sm3_ce_final(struct shash_desc *desc, u8 *out) +{ + if (!may_use_simd()) + return crypto_sm3_finup(desc, NULL, 0, out); + + kernel_neon_begin(); + sm3_base_do_finalize(desc, sm3_ce_transform); + kernel_neon_end(); + + return sm3_base_finish(desc, out); +} + +static int sm3_ce_finup(struct shash_desc *desc, const u8 *data, + unsigned int len, u8 *out) +{ + if (!may_use_simd()) + return crypto_sm3_finup(desc, data, len, out); + + kernel_neon_begin(); + sm3_base_do_update(desc, data, len, sm3_ce_transform); + kernel_neon_end(); + + return sm3_ce_final(desc, out); +} + +static struct shash_alg sm3_alg = { + .digestsize = SM3_DIGEST_SIZE, + .init = sm3_base_init, + .update = sm3_ce_update, + .final = sm3_ce_final, + .finup = sm3_ce_finup, + .descsize = sizeof(struct sm3_state), + .base.cra_name = "sm3", + .base.cra_driver_name = "sm3-ce", + .base.cra_flags = CRYPTO_ALG_TYPE_SHASH, + .base.cra_blocksize = SM3_BLOCK_SIZE, + .base.cra_module = THIS_MODULE, + .base.cra_priority = 200, +}; + +static int __init sm3_ce_mod_init(void) +{ + return crypto_register_shash(&sm3_alg); +} + +static void __exit sm3_ce_mod_fini(void) +{ + crypto_unregister_shash(&sm3_alg); +} + +module_cpu_feature_match(SM3, sm3_ce_mod_init); +module_exit(sm3_ce_mod_fini);