From patchwork Sat Jun 22 19:34:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167485 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2255819ilk; Sat, 22 Jun 2019 12:34:45 -0700 (PDT) X-Google-Smtp-Source: APXvYqynd3VGV5VbH1CeLYu91uW1U6robvbZGk4bGgV1Ho+WWC0Ea62VTug58vDt4+IMCq9Y8sVA X-Received: by 2002:a17:90a:d983:: with SMTP id d3mr14415767pjv.88.1561232085073; Sat, 22 Jun 2019 12:34:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232085; cv=none; d=google.com; s=arc-20160816; b=SNK+vOMI/pnQkoTewYYde7Y5gqTYn5+0Uq1AdfstYpEGyW1HYxsACPI2Vg5+N38oNg we/CE/SHZuHjUpLJnoCjGiyrea2TrsYO1jt4U+TeRUqxwa0xmeNNeGddwVx0dGNO6LLq MLgNOJhreO0PFw9QQHFzp2n/ZrqBEffI3o8ToYbSPXog9obH39KED4iQvv+zCFWxfPZI BM2jQc9Ns1eNJnm1Lf7JZbE+zbqz3bhepUuFSDLRgOkAD/SkFru5+Rc/Nfmz0IQblmPW kMoxiqdE+nptQykNeprQyqyRkaibuuaNdMvxQek4t2iBI+mlM2+bfrcWpe9Wc/2KehsZ ZwLg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=BVjKB83CGDhuLXJgz703gQuRW2ORbNsUnSLW3NNqkVc=; b=viN/+PzQ3BaZCSD96DB3NrO93/7xxQoAgFd02VysClbzM0l2y2NEUbhWGtae6s7MWr zW+Un+E7ghHqH8CetMZLY6aAI4CVAqXQn97jactsVybTkOd7lrndHS6DZP5oSAxJ+Rvm Zha0iD2YQpExBUR8r/SlWWTYNg07qLrVnAtycJv35f3HEOlIEWZ4Xzc1MgcR+7XNB/Sl bJNX6MmhcD9mNGKFG+xbbZWNyt7ud5xGxSki74b93a6BRQWIp58GWAHlv95YzcKiRb7H IN2ZeBme9fJga5D6SyjqiQ9Uhn1b0AkyinmZ239sxAS+jSl+CZ8bXQHydo2Q0kJHBoEo LTpA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=cHSgDKcu; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.34.44; Sat, 22 Jun 2019 12:34:45 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=cHSgDKcu; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726276AbfFVTen (ORCPT + 3 others); Sat, 22 Jun 2019 15:34:43 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:35941 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726286AbfFVTen (ORCPT ); Sat, 22 Jun 2019 15:34:43 -0400 Received: by mail-wr1-f68.google.com with SMTP id n4so8516961wrs.3 for ; Sat, 22 Jun 2019 12:34:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BVjKB83CGDhuLXJgz703gQuRW2ORbNsUnSLW3NNqkVc=; b=cHSgDKcuFMWYNb1pWS6aRdjp/rWrXTERCfIoFtcTtDIpaPxn7MWw9CCY4djWKBKq6n dl03jsNcdHz7noxOcNJZfh7LbIcbKyLj6wDzOD5YUAKRiRsvaAHmoM6R3xR5Gg+TNEO6 aFVZTwQ6fkWYTbevLZXf8UaU481qRjwMm58zyitx4vvz3a+hlBo6l4MoxlWxhRf7abj3 Yo5gQoP7l2ZEYv/i/2sP8re8hN+0sxJzI4FrhLEvm+1G+OJM05eI5rbIoS0qd81ynJxv sTAb+Dy5DDQebUXJoMzft2go4ItS7+ARQFL+HNM37FyUQjoJgzhN4/SxRgY1z9YM3yFE W6DQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BVjKB83CGDhuLXJgz703gQuRW2ORbNsUnSLW3NNqkVc=; b=F9WMeDBcyVKSR5xTzhCiqFKEzNTm/LZV9KTLf8qrfrTmv/KZORhnGsjX4XEcG6cuw4 LCRmI8PV6T3n1UBeeEk/7patY0D0LQ2o5CAza8tSUFZtQUuAIe991dgOit3oVRmyE2ZA g+uYQG9VQUDkf9PpUyJJCY/98wF37NPDPP9awcWwgv9Rqj04id12hxpPZaMfV3Moqt5K zerZOKEVz46csL3mXXhGNmcGbXMMuNLKPzDwMph6Ij/a4+UVvUEDcyWem3F3fWfOpOdb MyN0d3cFpC0+WFA2fGSyDww0mUM+VyxxgqD9WKV1jdnYYNUdcpeTCxmkYREdVN/uqUpR e1CQ== X-Gm-Message-State: APjAAAWEnVAf9sZAGndpCmM0LeVrkR9VE4PU289Et3GM/x8S8kgu/e6L s+kix6A6EotO5+zudq4RxccJD0MOeY/RgHVr X-Received: by 2002:a5d:5607:: with SMTP id l7mr15715149wrv.228.1561232080494; Sat, 22 Jun 2019 12:34:40 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:39 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 01/26] crypto: arm/aes-ce - cosmetic/whitespace cleanup Date: Sat, 22 Jun 2019 21:34:02 +0200 Message-Id: <20190622193427.20336-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Rearrange the aes_algs[] array for legibility. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/aes-ce-glue.c | 116 ++++++++++---------- 1 file changed, 56 insertions(+), 60 deletions(-) -- 2.20.1 diff --git a/arch/arm/crypto/aes-ce-glue.c b/arch/arm/crypto/aes-ce-glue.c index 5affb8482379..04ba66903674 100644 --- a/arch/arm/crypto/aes-ce-glue.c +++ b/arch/arm/crypto/aes-ce-glue.c @@ -337,69 +337,65 @@ static int xts_decrypt(struct skcipher_request *req) } static struct skcipher_alg aes_algs[] = { { - .base = { - .cra_name = "__ecb(aes)", - .cra_driver_name = "__ecb-aes-ce", - .cra_priority = 300, - .cra_flags = CRYPTO_ALG_INTERNAL, - .cra_blocksize = AES_BLOCK_SIZE, - .cra_ctxsize = sizeof(struct crypto_aes_ctx), - .cra_module = THIS_MODULE, - }, - .min_keysize = AES_MIN_KEY_SIZE, - .max_keysize = AES_MAX_KEY_SIZE, - .setkey = ce_aes_setkey, - .encrypt = ecb_encrypt, - .decrypt = ecb_decrypt, + .base.cra_name = "__ecb(aes)", + .base.cra_driver_name = "__ecb-aes-ce", + .base.cra_priority = 300, + .base.cra_flags = CRYPTO_ALG_INTERNAL, + .base.cra_blocksize = AES_BLOCK_SIZE, + .base.cra_ctxsize = sizeof(struct crypto_aes_ctx), + .base.cra_module = THIS_MODULE, + + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .setkey = ce_aes_setkey, + .encrypt = ecb_encrypt, + .decrypt = ecb_decrypt, }, { - .base = { - .cra_name = "__cbc(aes)", - .cra_driver_name = "__cbc-aes-ce", - .cra_priority = 300, - .cra_flags = CRYPTO_ALG_INTERNAL, - .cra_blocksize = AES_BLOCK_SIZE, - .cra_ctxsize = sizeof(struct crypto_aes_ctx), - .cra_module = THIS_MODULE, - }, - .min_keysize = AES_MIN_KEY_SIZE, - .max_keysize = AES_MAX_KEY_SIZE, - .ivsize = AES_BLOCK_SIZE, - .setkey = ce_aes_setkey, - .encrypt = cbc_encrypt, - .decrypt = cbc_decrypt, + .base.cra_name = "__cbc(aes)", + .base.cra_driver_name = "__cbc-aes-ce", + .base.cra_priority = 300, + .base.cra_flags = CRYPTO_ALG_INTERNAL, + .base.cra_blocksize = AES_BLOCK_SIZE, + .base.cra_ctxsize = sizeof(struct crypto_aes_ctx), + .base.cra_module = THIS_MODULE, + + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + .setkey = ce_aes_setkey, + .encrypt = cbc_encrypt, + .decrypt = cbc_decrypt, }, { - .base = { - .cra_name = "__ctr(aes)", - .cra_driver_name = "__ctr-aes-ce", - .cra_priority = 300, - .cra_flags = CRYPTO_ALG_INTERNAL, - .cra_blocksize = 1, - .cra_ctxsize = sizeof(struct crypto_aes_ctx), - .cra_module = THIS_MODULE, - }, - .min_keysize = AES_MIN_KEY_SIZE, - .max_keysize = AES_MAX_KEY_SIZE, - .ivsize = AES_BLOCK_SIZE, - .chunksize = AES_BLOCK_SIZE, - .setkey = ce_aes_setkey, - .encrypt = ctr_encrypt, - .decrypt = ctr_encrypt, + .base.cra_name = "__ctr(aes)", + .base.cra_driver_name = "__ctr-aes-ce", + .base.cra_priority = 300, + .base.cra_flags = CRYPTO_ALG_INTERNAL, + .base.cra_blocksize = 1, + .base.cra_ctxsize = sizeof(struct crypto_aes_ctx), + .base.cra_module = THIS_MODULE, + + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + .chunksize = AES_BLOCK_SIZE, + .setkey = ce_aes_setkey, + .encrypt = ctr_encrypt, + .decrypt = ctr_encrypt, }, { - .base = { - .cra_name = "__xts(aes)", - .cra_driver_name = "__xts-aes-ce", - .cra_priority = 300, - .cra_flags = CRYPTO_ALG_INTERNAL, - .cra_blocksize = AES_BLOCK_SIZE, - .cra_ctxsize = sizeof(struct crypto_aes_xts_ctx), - .cra_module = THIS_MODULE, - }, - .min_keysize = 2 * AES_MIN_KEY_SIZE, - .max_keysize = 2 * AES_MAX_KEY_SIZE, - .ivsize = AES_BLOCK_SIZE, - .setkey = xts_set_key, - .encrypt = xts_encrypt, - .decrypt = xts_decrypt, + .base.cra_name = "__xts(aes)", + .base.cra_driver_name = "__xts-aes-ce", + .base.cra_priority = 300, + .base.cra_flags = CRYPTO_ALG_INTERNAL, + .base.cra_blocksize = AES_BLOCK_SIZE, + .base.cra_ctxsize = sizeof(struct crypto_aes_xts_ctx), + .base.cra_module = THIS_MODULE, + + .min_keysize = 2 * AES_MIN_KEY_SIZE, + .max_keysize = 2 * AES_MAX_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + .setkey = xts_set_key, + .encrypt = xts_encrypt, + .decrypt = xts_decrypt, } }; static struct simd_skcipher_alg *aes_simd_algs[ARRAY_SIZE(aes_algs)]; From patchwork Sat Jun 22 19:34:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167486 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2255839ilk; Sat, 22 Jun 2019 12:34:45 -0700 (PDT) X-Google-Smtp-Source: APXvYqyUer9BID4ONJbPNznSkIzq11XOXIftco9GCNUlliIewvrUaHQQNi2Jnb0+6ACewgfX3Vga X-Received: by 2002:a17:90a:37c8:: with SMTP id v66mr14953671pjb.33.1561232085863; Sat, 22 Jun 2019 12:34:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232085; cv=none; d=google.com; s=arc-20160816; b=jm3ixJZK2X2wys7SywPOiyqvgGanuThgKvCZcnpXj5xiyMjYYofidDUfCffFt4VM1Z CwUFOZh7q/mr10/QiSp9aIVJtfE683cK9SnmxQprh15NZ9BopOaRWPLUFXjYG7mOJhRo nAsQe3F1aUBim4JthdNV2txVSLNxelb00SMqEHlASM3mcQBBdlWDu1nWWsEA3cm8jmE3 w4GVKIDpl2LajeyNbMPqD0J0J4DCw/Dg4DqZZqFdmwmpKj4FkIAchW+skJK3wSnIX9l4 0jibtpcwNwAzg2qRBHFowi34WfljbbRAAm5B5UNYX+HXqfo+M3rH2TEZKFYyjGOhP5K5 hrGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9jtH53Jk7KStQcdKvZdfqjCIjywTrKC4erKe/oL3K6E=; b=FIqN6OiZMSXeahXnIhKMT9Pb0bSy93TR82b3je4Jk2I8X61Iv3RXU9a1kNttkgf3qX M6kkk5S93vraUw9WiV2m/VVbVIYsqAlyGxeyS/D/Xk4e+3/DU4jfOIO9c+wIcuQy1tVH 7z75ILvG5sBF40cEsJmRgBKImQVDgRFNjl0WjKAbS7nmgP2RiXgYJW0ldj2sMiibDKva MkQCMfEcHADpxoul+bpbtncf2lImk8RUXDEaL30etzVdIQ4XLVwLOzOKyrUlrHPf7jy1 5A23R4KZ1W54EvlvANpKhdkgF+Jkkh3IonHqkV3H9MhkNHDLcKxnNJl4IQd57mIArAUT EXeA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=rjelpwRl; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.34.45; Sat, 22 Jun 2019 12:34:45 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=rjelpwRl; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726328AbfFVTep (ORCPT + 3 others); Sat, 22 Jun 2019 15:34:45 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:35006 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726290AbfFVTeo (ORCPT ); Sat, 22 Jun 2019 15:34:44 -0400 Received: by mail-wr1-f66.google.com with SMTP id m3so9743937wrv.2 for ; Sat, 22 Jun 2019 12:34:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9jtH53Jk7KStQcdKvZdfqjCIjywTrKC4erKe/oL3K6E=; b=rjelpwRlEGl8nDu1hlHJKl5gKKNiR5f23R8ycwG41M44RVvWF/etZuWgvSZGBO3dSt kXk51LyjJBDRBTFf5iMk2zBtfldLHm+TKlsjuZgo5mwuKHtPbRHdjZ9uSAq0cRRDCG0/ 1mYrI2Gasxagzbxr4vyW5w9Ix2RkIGntxbZWCuOd3F87Vwq98s5RtlPayQXBfBwQmLx4 9OnBaEOZIhHr177ZShVOZBa4+EYWAeeutgwZDCUTDULlaCbCl1Pw2mVaHNRn0yID+AM3 u4IsyDvhXpa/ZeAa3iT6flMwslQW46b4tsG/0qzox3GS2ZmWf8ldjU9qAjQGKsasPEvm a0tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9jtH53Jk7KStQcdKvZdfqjCIjywTrKC4erKe/oL3K6E=; b=WE/A/f1t/F5TsVaIHsUM6naEfijqBtz2G9Mks33yx2hRrFASWYHNScAlaxsVB/JoOC Iw0ugbb1smsvi7E9xD206/oR08HW7BMMgo64Qjp65E9SYtb9xohYwpG43PK0XYEPminT OpLsvZ/V/G1JmBoZaNwnopnbACyx+34y0mxJXw7/MUNjWSVwyMj1MS4T9Rlw3a9mJZJ4 QFGGQSXRtOHLSUukZEv0vRoARisM5I1Xw5ULHYwBEOi6DBqbRCr2Q8x7GExvYx5PFAhJ rrNjP+HEKUQKYScg1DSQIqAHAJQJRbvg+SwpDRydo/WmqGQrrXw8k/eSEHYd7xeN2sE5 8tRQ== X-Gm-Message-State: APjAAAXSPm9IEM85ITB68twEh+dAzVKicXIpse75w3aMLHRUYCL2E2kq +Z2yKOeYalijU4EWNYhIDS7zuN8cqu1d+T6S X-Received: by 2002:a5d:4d84:: with SMTP id b4mr21162089wru.242.1561232081740; Sat, 22 Jun 2019 12:34:41 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:41 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 02/26] crypto: aes - rename local routines to prevent future clashes Date: Sat, 22 Jun 2019 21:34:03 +0200 Message-Id: <20190622193427.20336-3-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Rename some local AES encrypt/decrypt routines so they don't clash with the names we are about to introduce for the routines exposed by the generic AES library. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/aes-cipher-glue.c | 8 ++++---- arch/arm64/crypto/aes-cipher-glue.c | 8 ++++---- arch/x86/crypto/aesni-intel_glue.c | 8 ++++---- crypto/aes_generic.c | 8 ++++---- drivers/crypto/padlock-aes.c | 8 ++++---- 5 files changed, 20 insertions(+), 20 deletions(-) -- 2.20.1 diff --git a/arch/arm/crypto/aes-cipher-glue.c b/arch/arm/crypto/aes-cipher-glue.c index c222f6e072ad..f6c07867b8ff 100644 --- a/arch/arm/crypto/aes-cipher-glue.c +++ b/arch/arm/crypto/aes-cipher-glue.c @@ -19,7 +19,7 @@ EXPORT_SYMBOL(__aes_arm_encrypt); asmlinkage void __aes_arm_decrypt(u32 *rk, int rounds, const u8 *in, u8 *out); EXPORT_SYMBOL(__aes_arm_decrypt); -static void aes_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) +static void aes_arm_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) { struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm); int rounds = 6 + ctx->key_length / 4; @@ -27,7 +27,7 @@ static void aes_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) __aes_arm_encrypt(ctx->key_enc, rounds, in, out); } -static void aes_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) +static void aes_arm_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) { struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm); int rounds = 6 + ctx->key_length / 4; @@ -47,8 +47,8 @@ static struct crypto_alg aes_alg = { .cra_cipher.cia_min_keysize = AES_MIN_KEY_SIZE, .cra_cipher.cia_max_keysize = AES_MAX_KEY_SIZE, .cra_cipher.cia_setkey = crypto_aes_set_key, - .cra_cipher.cia_encrypt = aes_encrypt, - .cra_cipher.cia_decrypt = aes_decrypt, + .cra_cipher.cia_encrypt = aes_arm_encrypt, + .cra_cipher.cia_decrypt = aes_arm_decrypt, #ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS .cra_alignmask = 3, diff --git a/arch/arm64/crypto/aes-cipher-glue.c b/arch/arm64/crypto/aes-cipher-glue.c index 7288e7cbebff..0e90b06ebcec 100644 --- a/arch/arm64/crypto/aes-cipher-glue.c +++ b/arch/arm64/crypto/aes-cipher-glue.c @@ -18,7 +18,7 @@ EXPORT_SYMBOL(__aes_arm64_encrypt); asmlinkage void __aes_arm64_decrypt(u32 *rk, u8 *out, const u8 *in, int rounds); EXPORT_SYMBOL(__aes_arm64_decrypt); -static void aes_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) +static void aes_arm64_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) { struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm); int rounds = 6 + ctx->key_length / 4; @@ -26,7 +26,7 @@ static void aes_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) __aes_arm64_encrypt(ctx->key_enc, out, in, rounds); } -static void aes_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) +static void aes_arm64_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) { struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm); int rounds = 6 + ctx->key_length / 4; @@ -46,8 +46,8 @@ static struct crypto_alg aes_alg = { .cra_cipher.cia_min_keysize = AES_MIN_KEY_SIZE, .cra_cipher.cia_max_keysize = AES_MAX_KEY_SIZE, .cra_cipher.cia_setkey = crypto_aes_set_key, - .cra_cipher.cia_encrypt = aes_encrypt, - .cra_cipher.cia_decrypt = aes_decrypt + .cra_cipher.cia_encrypt = aes_arm64_encrypt, + .cra_cipher.cia_decrypt = aes_arm64_decrypt }; static int __init aes_init(void) diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c index c95bd397dc07..836d50bd096f 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -349,7 +349,7 @@ static int aes_set_key(struct crypto_tfm *tfm, const u8 *in_key, return aes_set_key_common(tfm, crypto_tfm_ctx(tfm), in_key, key_len); } -static void aes_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) +static void aesni_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) { struct crypto_aes_ctx *ctx = aes_ctx(crypto_tfm_ctx(tfm)); @@ -362,7 +362,7 @@ static void aes_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) } } -static void aes_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) +static void aesni_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) { struct crypto_aes_ctx *ctx = aes_ctx(crypto_tfm_ctx(tfm)); @@ -923,8 +923,8 @@ static struct crypto_alg aesni_cipher_alg = { .cia_min_keysize = AES_MIN_KEY_SIZE, .cia_max_keysize = AES_MAX_KEY_SIZE, .cia_setkey = aes_set_key, - .cia_encrypt = aes_encrypt, - .cia_decrypt = aes_decrypt + .cia_encrypt = aesni_encrypt, + .cia_decrypt = aesni_decrypt } } }; diff --git a/crypto/aes_generic.c b/crypto/aes_generic.c index f217568917e4..3aa4a715c216 100644 --- a/crypto/aes_generic.c +++ b/crypto/aes_generic.c @@ -1332,7 +1332,7 @@ EXPORT_SYMBOL_GPL(crypto_aes_set_key); f_rl(bo, bi, 3, k); \ } while (0) -static void aes_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) +static void crypto_aes_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) { const struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm); u32 b0[4], b1[4]; @@ -1402,7 +1402,7 @@ static void aes_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) i_rl(bo, bi, 3, k); \ } while (0) -static void aes_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) +static void crypto_aes_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) { const struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm); u32 b0[4], b1[4]; @@ -1454,8 +1454,8 @@ static struct crypto_alg aes_alg = { .cia_min_keysize = AES_MIN_KEY_SIZE, .cia_max_keysize = AES_MAX_KEY_SIZE, .cia_setkey = crypto_aes_set_key, - .cia_encrypt = aes_encrypt, - .cia_decrypt = aes_decrypt + .cia_encrypt = crypto_aes_encrypt, + .cia_decrypt = crypto_aes_decrypt } } }; diff --git a/drivers/crypto/padlock-aes.c b/drivers/crypto/padlock-aes.c index 09d823d36d3a..854539512c35 100644 --- a/drivers/crypto/padlock-aes.c +++ b/drivers/crypto/padlock-aes.c @@ -299,7 +299,7 @@ static inline u8 *padlock_xcrypt_cbc(const u8 *input, u8 *output, void *key, return iv; } -static void aes_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) +static void padlock_aes_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) { struct aes_ctx *ctx = aes_ctx(tfm); @@ -308,7 +308,7 @@ static void aes_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) padlock_store_cword(&ctx->cword.encrypt); } -static void aes_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) +static void padlock_aes_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) { struct aes_ctx *ctx = aes_ctx(tfm); @@ -331,8 +331,8 @@ static struct crypto_alg aes_alg = { .cia_min_keysize = AES_MIN_KEY_SIZE, .cia_max_keysize = AES_MAX_KEY_SIZE, .cia_setkey = aes_set_key, - .cia_encrypt = aes_encrypt, - .cia_decrypt = aes_decrypt, + .cia_encrypt = padlock_aes_encrypt, + .cia_decrypt = padlock_aes_decrypt, } } }; From patchwork Sat Jun 22 19:34:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167487 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2255854ilk; Sat, 22 Jun 2019 12:34:47 -0700 (PDT) X-Google-Smtp-Source: APXvYqxTpvPnt4Aiuzl3O8b+/d1JLLBqNqd8i+zwqP3wDtN8x3yc2hgHnUzN7KRzqfkZ4AcYEzzj X-Received: by 2002:a17:902:b093:: with SMTP id p19mr344077plr.141.1561232087391; Sat, 22 Jun 2019 12:34:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232087; cv=none; d=google.com; s=arc-20160816; b=QpWnLWyvQGXoNNeJrag2jUL4T+YqK/rBguZ2eQ9JMBS/NDMdfNY3sIjc1n8gCzojvF BeJ86mLdAqDykc9CJ0pOYpJ//B7cymkl/nCYBNP2NERmsDclVNr0J03maC8UVDaABg0j JTH85nJ09fjpvktKfH1PzsGv/VclIUHZcfgtIYRdkez/zBp8qI4uCDkNAD9cl1aZ+HvF BVRQXWGS8ogTRcop9vQo/uzAvZjogndmVtIgfyaV3h78nVaJV3+HEQUSGrNzz4nKnRq5 cnkuNRo744hhmOIUJbSX1JIEdVUeU08NEH5D2sUlAfHNxstlx7Is36oMEjyqnmP51/LI a19g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=OVlu3R/XDhYHCSEro6NMIXQLAFRHmhGDxGC17BFWD8A=; b=AQ0yQ/QqFOXvkcaTeAUvdm1wbe2gSdyJXGfUIX1TsyUrJ7BuhCzlS/H4bXVI8oSHGS hhON6FZ+WhaL4VqJopXumKR1JoX3M1vhI9NWm5q1fdWh6ueZ/VIlB9NA4wxew5omByIP 4FvtLn2jl1qVbDXv73MSrHNNTOmA9uVyXq3p6W1JrmQL17XsuFnsYm6wBKE1wcTatNMm Ku6BjYNXRLdQcnmCm6uu160+6LXIpXCj60gvEh5qc7C97LgKbl9CTErphudtRLWNBnjg 1hr5WH9QeV3+WC6tmEUHwNWIKzZ6rOP1rz6N8uNaTQs2PaNTbGFYzDMRgMn3jCBiOTs2 ffRA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=tJXTXngG; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.34.47; Sat, 22 Jun 2019 12:34:47 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=tJXTXngG; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726339AbfFVTeq (ORCPT + 3 others); Sat, 22 Jun 2019 15:34:46 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:53464 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726286AbfFVTeq (ORCPT ); Sat, 22 Jun 2019 15:34:46 -0400 Received: by mail-wm1-f66.google.com with SMTP id x15so9147901wmj.3 for ; Sat, 22 Jun 2019 12:34:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OVlu3R/XDhYHCSEro6NMIXQLAFRHmhGDxGC17BFWD8A=; b=tJXTXngGIkeIZQnGKLux/rSvP5U7mbJawhh3jmft7Y/pQGJb+TpzXp780X+OOM65qF b6pkJwz8McIRpkJKNLg8KGTlFY7iBxHW9217gC455xLoOqgvsay3Nxhc+pCnM88frjlD YmA1ZH8x9jfKzjaXxYMngDCZAnCKSzjoJGeHObQb5UKgDfgtJlC8HPKPAvd9w4rCmvhZ JeA+M+3+JsEagV2/ST3lkUV6ZkNHelsfycvBBA0Ar/C/0ak+rHImTzwlLMcOkwda5vFb 70ny267/gmzXz1Qt4TKuzre6cPikhx9WChKNyVy/rQVMUE5Mfavbu5JzTHERS8yxxKfv D+JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OVlu3R/XDhYHCSEro6NMIXQLAFRHmhGDxGC17BFWD8A=; b=NCcFOR2HKdv1yVbsVKKcbQwlxkGfyHS3DtKa2gQA/pbp7a9PE4W8INQpDDnS4hAb1O hY8fMwux4gaYKWJjHZcZkNqp6guk7hhgt+1gcY4zkCxnCz6GRe4NX9/Dk+18omMFamCv XyQBrgcrArwV2RT0Nkfxg0YhZ4/bRYVRV449nAUc2LrD0OUrBaY2JUACz1ro1dI0PYli 8VU8S3NqEl9bYKdHdqYmOdIdPQWiaCBpwnyPCLWZvTJXe+yKWcmj8+60Khyxx+IUheao cGHQONPQ7B5t471ZkwlybGRNkE2ar3Cfi1anDWi9y5y7KmqN+osotyNwpc7PolZ4LZbn i6dA== X-Gm-Message-State: APjAAAW2ns1Hz6vTV8yPuGHY7tLzbxnTsesuXsDJHA8opY2fpaX59TVX /htAw7Pco4Qentg2VO9nJ1O9meAmP0q7n5Wl X-Received: by 2002:a7b:c450:: with SMTP id l16mr9294082wmi.0.1561232083236; Sat, 22 Jun 2019 12:34:43 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:42 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 03/26] crypto: aes/fixed-time - align key schedule with other implementations Date: Sat, 22 Jun 2019 21:34:04 +0200 Message-Id: <20190622193427.20336-4-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The fixed time AES code mangles the key schedule so that xoring the first round key with values at fixed offsets across the Sbox produces the correct value. This primes the D-cache with the entire Sbox before any data dependent lookups are done, making it more difficult to infer key bits from timing variances when the plaintext is known. The downside of this approach is that it renders the key schedule incompatible with other implementations of AES in the kernel, which makes it cumbersome to use this implementation as a fallback for SIMD based AES in contexts where this is not allowed. So let's tweak the fixed Sbox indexes so that they add up to zero under the xor operation. While at it, increase the granularity to 16 bytes so we cover the entire Sbox even on systems with 16 byte cachelines. Signed-off-by: Ard Biesheuvel --- crypto/aes_ti.c | 52 ++++++++------------ 1 file changed, 21 insertions(+), 31 deletions(-) -- 2.20.1 diff --git a/crypto/aes_ti.c b/crypto/aes_ti.c index 1ff9785b30f5..fd70dc322634 100644 --- a/crypto/aes_ti.c +++ b/crypto/aes_ti.c @@ -237,30 +237,8 @@ static int aesti_set_key(struct crypto_tfm *tfm, const u8 *in_key, unsigned int key_len) { struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm); - int err; - err = aesti_expand_key(ctx, in_key, key_len); - if (err) - return err; - - /* - * In order to force the compiler to emit data independent Sbox lookups - * at the start of each block, xor the first round key with values at - * fixed indexes in the Sbox. This will need to be repeated each time - * the key is used, which will pull the entire Sbox into the D-cache - * before any data dependent Sbox lookups are performed. - */ - ctx->key_enc[0] ^= __aesti_sbox[ 0] ^ __aesti_sbox[128]; - ctx->key_enc[1] ^= __aesti_sbox[32] ^ __aesti_sbox[160]; - ctx->key_enc[2] ^= __aesti_sbox[64] ^ __aesti_sbox[192]; - ctx->key_enc[3] ^= __aesti_sbox[96] ^ __aesti_sbox[224]; - - ctx->key_dec[0] ^= __aesti_inv_sbox[ 0] ^ __aesti_inv_sbox[128]; - ctx->key_dec[1] ^= __aesti_inv_sbox[32] ^ __aesti_inv_sbox[160]; - ctx->key_dec[2] ^= __aesti_inv_sbox[64] ^ __aesti_inv_sbox[192]; - ctx->key_dec[3] ^= __aesti_inv_sbox[96] ^ __aesti_inv_sbox[224]; - - return 0; + return aesti_expand_key(ctx, in_key, key_len); } static void aesti_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) @@ -283,10 +261,16 @@ static void aesti_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) */ local_irq_save(flags); - st0[0] ^= __aesti_sbox[ 0] ^ __aesti_sbox[128]; - st0[1] ^= __aesti_sbox[32] ^ __aesti_sbox[160]; - st0[2] ^= __aesti_sbox[64] ^ __aesti_sbox[192]; - st0[3] ^= __aesti_sbox[96] ^ __aesti_sbox[224]; + /* + * Force the compiler to emit data independent Sbox references, + * by xoring the input with Sbox values that are known to add up + * to zero. This pulls the entire Sbox into the D-cache before any + * data dependent lookups are done. + */ + st0[0] ^= __aesti_sbox[ 0] ^ __aesti_sbox[ 64] ^ __aesti_sbox[134] ^ __aesti_sbox[195]; + st0[1] ^= __aesti_sbox[16] ^ __aesti_sbox[ 82] ^ __aesti_sbox[158] ^ __aesti_sbox[221]; + st0[2] ^= __aesti_sbox[32] ^ __aesti_sbox[ 96] ^ __aesti_sbox[160] ^ __aesti_sbox[234]; + st0[3] ^= __aesti_sbox[48] ^ __aesti_sbox[112] ^ __aesti_sbox[186] ^ __aesti_sbox[241]; for (round = 0;; round += 2, rkp += 8) { st1[0] = mix_columns(subshift(st0, 0)) ^ rkp[0]; @@ -331,10 +315,16 @@ static void aesti_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) */ local_irq_save(flags); - st0[0] ^= __aesti_inv_sbox[ 0] ^ __aesti_inv_sbox[128]; - st0[1] ^= __aesti_inv_sbox[32] ^ __aesti_inv_sbox[160]; - st0[2] ^= __aesti_inv_sbox[64] ^ __aesti_inv_sbox[192]; - st0[3] ^= __aesti_inv_sbox[96] ^ __aesti_inv_sbox[224]; + /* + * Force the compiler to emit data independent Sbox references, + * by xoring the input with Sbox values that are known to add up + * to zero. This pulls the entire Sbox into the D-cache before any + * data dependent lookups are done. + */ + st0[0] ^= __aesti_inv_sbox[ 0] ^ __aesti_inv_sbox[ 64] ^ __aesti_inv_sbox[129] ^ __aesti_inv_sbox[200]; + st0[1] ^= __aesti_inv_sbox[16] ^ __aesti_inv_sbox[ 83] ^ __aesti_inv_sbox[150] ^ __aesti_inv_sbox[212]; + st0[2] ^= __aesti_inv_sbox[32] ^ __aesti_inv_sbox[ 96] ^ __aesti_inv_sbox[160] ^ __aesti_inv_sbox[236]; + st0[3] ^= __aesti_inv_sbox[48] ^ __aesti_inv_sbox[112] ^ __aesti_inv_sbox[187] ^ __aesti_inv_sbox[247]; for (round = 0;; round += 2, rkp += 8) { st1[0] = inv_mix_columns(inv_subshift(st0, 0)) ^ rkp[0]; From patchwork Sat Jun 22 19:34:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167491 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2255883ilk; Sat, 22 Jun 2019 12:34:51 -0700 (PDT) X-Google-Smtp-Source: APXvYqw2sJQ3b92s+EKlST7YnvAr3psq1jpyMlf4VbT+wksT1IuxjzBNt7wapCIn8wjeFEtpHaea X-Received: by 2002:a63:6149:: with SMTP id v70mr16427510pgb.191.1561232091078; Sat, 22 Jun 2019 12:34:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232091; cv=none; d=google.com; s=arc-20160816; b=dfnAb1QMoquYRfLnqZwfqU2kV6XyOeI5LtQR8srWNaqqQwDpG9E7/XcXqgR8x8+0Vq pGCWGbXyEpgsshFxKOsnxxorS9YqG0qD8iJ11dCXPB5P+Rm9tIwh2bpv1vkMZH5bDfMn HrHzNX4a9/q2Iz8yevtUR64AUiDstWTOWcP0hE85rKeHuZum+U7Kzv1VvsxdudObOBih RSdY7eWQcx/whNp2eZG4VgrDBSeCJcezBqLPAAVhU6WffS6GWA9eZQiJe2vfJP0qxX1T n69psxTMb4PYHqHlCaRhO7yVCI5anuey/uedjlqHPZPSfUgDfnE4+S4AJ9oJwnpLND3q staQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JYfyuXtkWyqjgUR6IexmK+iDcDL5/HP/psv9afBMZ2s=; b=V9uaSs3q4xT//YSiXzsoT9rgEwp2ChvBAqZoPSUSWKl6ZSTpp5VSFlfwoefjxVg0BP lHPdm7IKxMemHMIZsXQP1neg09OZSMwYR8a2Q72o5rr/PugOt8UisHSXD3Hj0b54u9pv lDBOpD+FzWsZcISeG+oYf2qyP6rG6p6USDimb9/1G/vdsFDaeUJWKnkBduUzhDT2CqxS q9O2EDHLwFDhkaildLHWumrly37HWoBU4FSVlR4xotH57xzd+RpdqbZ/WHgAjuUAlARC Z6NB2hlUjADEptcLxE02hZrvHd3DzYeUYk/UG+hIS+Z7bNb56VB0Fx1h752mveNSP1Iv BcYw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=sLauLZDN; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.34.50; Sat, 22 Jun 2019 12:34:51 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=sLauLZDN; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726350AbfFVTeu (ORCPT + 3 others); Sat, 22 Jun 2019 15:34:50 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:34851 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726290AbfFVTeu (ORCPT ); Sat, 22 Jun 2019 15:34:50 -0400 Received: by mail-wm1-f65.google.com with SMTP id c6so9631252wml.0 for ; Sat, 22 Jun 2019 12:34:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JYfyuXtkWyqjgUR6IexmK+iDcDL5/HP/psv9afBMZ2s=; b=sLauLZDNvRWMaZX1QtEPVft9AGIVxdv/7w6ZnU/2d/m11nB1zFBwC+pBJhKeExTHn2 Oui337C2f/udRwYtQ0PPQAbdz6VjjGYHFGzf0X9uKB5PlVAF4v/b7zKGajtMnqz4ijQQ Qa/98NR6lMbSfmhNIXzm/z8Uj3N4fM8/DyShv+sLiqBb7AdY+BT2h8C/oXLCp/w4vgCN BtyMuiz2DiRLnltrlL/vTbbnchTDBDl/Y+I0Rwm16VyWRgsXa5jdllumuzV+4bP4Hzry wX+IVYAj3UkHbag9I3ahGEzSpgndeMF8LOhy9++D8PWYeIqWDDsspN1dPC5tceyXOS+I COzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JYfyuXtkWyqjgUR6IexmK+iDcDL5/HP/psv9afBMZ2s=; b=PES3YL/Uzupy96Kh3qTUdlZ6x0N7oFLn0mXvVEGrpbQVVkVZ5h39BRyWS5dcbc6xtX SmNSC7lLjREI4bqYac+E5X+pI4+/fz8Zwkg9DBHtQJumbxh6HvAfAMaft7aJ93vps3aQ Np3wr0yHpqBM9v2NM+Ewi4CkPlS9UuWHawWtCssnSw84adyE70YEf2QoR4XZcnnZ9ngR 0hCNDOLD1IBzG8vH1vaosPO1dxlJFdq1umCjaVdS8c8ldktdM6GaqRYaGUBip16tTpBH olXBQSKD0+PLHzbpfKAhXkyxM0TIdCs+e825aDzi9u+RM4PMl+LGLG5lIMt5m2HW4oYH lZOA== X-Gm-Message-State: APjAAAVS3cDPbjcI4vjrkrGsZg3OxGLoq86kXxYfKJ+tSsL7zfUX6y3t Njj5I5mvIZt+Xi0fUaXZtu/NEA9og+/mBCh9 X-Received: by 2002:a05:600c:20ca:: with SMTP id y10mr8474098wmm.72.1561232084461; Sat, 22 Jun 2019 12:34:44 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:43 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 04/26] crypto: aes - create AES library based on the fixed time AES code Date: Sat, 22 Jun 2019 21:34:05 +0200 Message-Id: <20190622193427.20336-5-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Take the existing small footprint and mostly time invariant C code and turn it into a AES library that can be used for non-performance critical, casual use of AES, and as a fallback for, e.g., SIMD code that needs a secondary path that can be taken in contexts where the SIMD unit is off limits (e.g., in hard interrupts taken from kernel context) Signed-off-by: Ard Biesheuvel --- crypto/Kconfig | 4 + crypto/aes_ti.c | 307 +---------------- include/crypto/aes.h | 34 ++ lib/crypto/Makefile | 3 + lib/crypto/aes.c | 350 ++++++++++++++++++++ 5 files changed, 395 insertions(+), 303 deletions(-) -- 2.20.1 diff --git a/crypto/Kconfig b/crypto/Kconfig index e801450bcb1c..091ebbbc9655 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -1066,6 +1066,9 @@ config CRYPTO_GHASH_CLMUL_NI_INTEL comment "Ciphers" +config CRYPTO_LIB_AES + tristate + config CRYPTO_AES tristate "AES cipher algorithms" select CRYPTO_ALGAPI @@ -1089,6 +1092,7 @@ config CRYPTO_AES config CRYPTO_AES_TI tristate "Fixed time AES cipher" select CRYPTO_ALGAPI + select CRYPTO_LIB_AES help This is a generic implementation of AES that attempts to eliminate data dependent latencies as much as possible without affecting diff --git a/crypto/aes_ti.c b/crypto/aes_ti.c index fd70dc322634..339915db9aeb 100644 --- a/crypto/aes_ti.c +++ b/crypto/aes_ti.c @@ -1,259 +1,27 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Scalar fixed time AES core transform * * Copyright (C) 2017 Linaro Ltd - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. */ #include #include #include -#include - -/* - * Emit the sbox as volatile const to prevent the compiler from doing - * constant folding on sbox references involving fixed indexes. - */ -static volatile const u8 __cacheline_aligned __aesti_sbox[] = { - 0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5, - 0x30, 0x01, 0x67, 0x2b, 0xfe, 0xd7, 0xab, 0x76, - 0xca, 0x82, 0xc9, 0x7d, 0xfa, 0x59, 0x47, 0xf0, - 0xad, 0xd4, 0xa2, 0xaf, 0x9c, 0xa4, 0x72, 0xc0, - 0xb7, 0xfd, 0x93, 0x26, 0x36, 0x3f, 0xf7, 0xcc, - 0x34, 0xa5, 0xe5, 0xf1, 0x71, 0xd8, 0x31, 0x15, - 0x04, 0xc7, 0x23, 0xc3, 0x18, 0x96, 0x05, 0x9a, - 0x07, 0x12, 0x80, 0xe2, 0xeb, 0x27, 0xb2, 0x75, - 0x09, 0x83, 0x2c, 0x1a, 0x1b, 0x6e, 0x5a, 0xa0, - 0x52, 0x3b, 0xd6, 0xb3, 0x29, 0xe3, 0x2f, 0x84, - 0x53, 0xd1, 0x00, 0xed, 0x20, 0xfc, 0xb1, 0x5b, - 0x6a, 0xcb, 0xbe, 0x39, 0x4a, 0x4c, 0x58, 0xcf, - 0xd0, 0xef, 0xaa, 0xfb, 0x43, 0x4d, 0x33, 0x85, - 0x45, 0xf9, 0x02, 0x7f, 0x50, 0x3c, 0x9f, 0xa8, - 0x51, 0xa3, 0x40, 0x8f, 0x92, 0x9d, 0x38, 0xf5, - 0xbc, 0xb6, 0xda, 0x21, 0x10, 0xff, 0xf3, 0xd2, - 0xcd, 0x0c, 0x13, 0xec, 0x5f, 0x97, 0x44, 0x17, - 0xc4, 0xa7, 0x7e, 0x3d, 0x64, 0x5d, 0x19, 0x73, - 0x60, 0x81, 0x4f, 0xdc, 0x22, 0x2a, 0x90, 0x88, - 0x46, 0xee, 0xb8, 0x14, 0xde, 0x5e, 0x0b, 0xdb, - 0xe0, 0x32, 0x3a, 0x0a, 0x49, 0x06, 0x24, 0x5c, - 0xc2, 0xd3, 0xac, 0x62, 0x91, 0x95, 0xe4, 0x79, - 0xe7, 0xc8, 0x37, 0x6d, 0x8d, 0xd5, 0x4e, 0xa9, - 0x6c, 0x56, 0xf4, 0xea, 0x65, 0x7a, 0xae, 0x08, - 0xba, 0x78, 0x25, 0x2e, 0x1c, 0xa6, 0xb4, 0xc6, - 0xe8, 0xdd, 0x74, 0x1f, 0x4b, 0xbd, 0x8b, 0x8a, - 0x70, 0x3e, 0xb5, 0x66, 0x48, 0x03, 0xf6, 0x0e, - 0x61, 0x35, 0x57, 0xb9, 0x86, 0xc1, 0x1d, 0x9e, - 0xe1, 0xf8, 0x98, 0x11, 0x69, 0xd9, 0x8e, 0x94, - 0x9b, 0x1e, 0x87, 0xe9, 0xce, 0x55, 0x28, 0xdf, - 0x8c, 0xa1, 0x89, 0x0d, 0xbf, 0xe6, 0x42, 0x68, - 0x41, 0x99, 0x2d, 0x0f, 0xb0, 0x54, 0xbb, 0x16, -}; - -static volatile const u8 __cacheline_aligned __aesti_inv_sbox[] = { - 0x52, 0x09, 0x6a, 0xd5, 0x30, 0x36, 0xa5, 0x38, - 0xbf, 0x40, 0xa3, 0x9e, 0x81, 0xf3, 0xd7, 0xfb, - 0x7c, 0xe3, 0x39, 0x82, 0x9b, 0x2f, 0xff, 0x87, - 0x34, 0x8e, 0x43, 0x44, 0xc4, 0xde, 0xe9, 0xcb, - 0x54, 0x7b, 0x94, 0x32, 0xa6, 0xc2, 0x23, 0x3d, - 0xee, 0x4c, 0x95, 0x0b, 0x42, 0xfa, 0xc3, 0x4e, - 0x08, 0x2e, 0xa1, 0x66, 0x28, 0xd9, 0x24, 0xb2, - 0x76, 0x5b, 0xa2, 0x49, 0x6d, 0x8b, 0xd1, 0x25, - 0x72, 0xf8, 0xf6, 0x64, 0x86, 0x68, 0x98, 0x16, - 0xd4, 0xa4, 0x5c, 0xcc, 0x5d, 0x65, 0xb6, 0x92, - 0x6c, 0x70, 0x48, 0x50, 0xfd, 0xed, 0xb9, 0xda, - 0x5e, 0x15, 0x46, 0x57, 0xa7, 0x8d, 0x9d, 0x84, - 0x90, 0xd8, 0xab, 0x00, 0x8c, 0xbc, 0xd3, 0x0a, - 0xf7, 0xe4, 0x58, 0x05, 0xb8, 0xb3, 0x45, 0x06, - 0xd0, 0x2c, 0x1e, 0x8f, 0xca, 0x3f, 0x0f, 0x02, - 0xc1, 0xaf, 0xbd, 0x03, 0x01, 0x13, 0x8a, 0x6b, - 0x3a, 0x91, 0x11, 0x41, 0x4f, 0x67, 0xdc, 0xea, - 0x97, 0xf2, 0xcf, 0xce, 0xf0, 0xb4, 0xe6, 0x73, - 0x96, 0xac, 0x74, 0x22, 0xe7, 0xad, 0x35, 0x85, - 0xe2, 0xf9, 0x37, 0xe8, 0x1c, 0x75, 0xdf, 0x6e, - 0x47, 0xf1, 0x1a, 0x71, 0x1d, 0x29, 0xc5, 0x89, - 0x6f, 0xb7, 0x62, 0x0e, 0xaa, 0x18, 0xbe, 0x1b, - 0xfc, 0x56, 0x3e, 0x4b, 0xc6, 0xd2, 0x79, 0x20, - 0x9a, 0xdb, 0xc0, 0xfe, 0x78, 0xcd, 0x5a, 0xf4, - 0x1f, 0xdd, 0xa8, 0x33, 0x88, 0x07, 0xc7, 0x31, - 0xb1, 0x12, 0x10, 0x59, 0x27, 0x80, 0xec, 0x5f, - 0x60, 0x51, 0x7f, 0xa9, 0x19, 0xb5, 0x4a, 0x0d, - 0x2d, 0xe5, 0x7a, 0x9f, 0x93, 0xc9, 0x9c, 0xef, - 0xa0, 0xe0, 0x3b, 0x4d, 0xae, 0x2a, 0xf5, 0xb0, - 0xc8, 0xeb, 0xbb, 0x3c, 0x83, 0x53, 0x99, 0x61, - 0x17, 0x2b, 0x04, 0x7e, 0xba, 0x77, 0xd6, 0x26, - 0xe1, 0x69, 0x14, 0x63, 0x55, 0x21, 0x0c, 0x7d, -}; - -static u32 mul_by_x(u32 w) -{ - u32 x = w & 0x7f7f7f7f; - u32 y = w & 0x80808080; - - /* multiply by polynomial 'x' (0b10) in GF(2^8) */ - return (x << 1) ^ (y >> 7) * 0x1b; -} - -static u32 mul_by_x2(u32 w) -{ - u32 x = w & 0x3f3f3f3f; - u32 y = w & 0x80808080; - u32 z = w & 0x40404040; - - /* multiply by polynomial 'x^2' (0b100) in GF(2^8) */ - return (x << 2) ^ (y >> 7) * 0x36 ^ (z >> 6) * 0x1b; -} - -static u32 mix_columns(u32 x) -{ - /* - * Perform the following matrix multiplication in GF(2^8) - * - * | 0x2 0x3 0x1 0x1 | | x[0] | - * | 0x1 0x2 0x3 0x1 | | x[1] | - * | 0x1 0x1 0x2 0x3 | x | x[2] | - * | 0x3 0x1 0x1 0x2 | | x[3] | - */ - u32 y = mul_by_x(x) ^ ror32(x, 16); - - return y ^ ror32(x ^ y, 8); -} - -static u32 inv_mix_columns(u32 x) -{ - /* - * Perform the following matrix multiplication in GF(2^8) - * - * | 0xe 0xb 0xd 0x9 | | x[0] | - * | 0x9 0xe 0xb 0xd | | x[1] | - * | 0xd 0x9 0xe 0xb | x | x[2] | - * | 0xb 0xd 0x9 0xe | | x[3] | - * - * which can conveniently be reduced to - * - * | 0x2 0x3 0x1 0x1 | | 0x5 0x0 0x4 0x0 | | x[0] | - * | 0x1 0x2 0x3 0x1 | | 0x0 0x5 0x0 0x4 | | x[1] | - * | 0x1 0x1 0x2 0x3 | x | 0x4 0x0 0x5 0x0 | x | x[2] | - * | 0x3 0x1 0x1 0x2 | | 0x0 0x4 0x0 0x5 | | x[3] | - */ - u32 y = mul_by_x2(x); - - return mix_columns(x ^ y ^ ror32(y, 16)); -} - -static __always_inline u32 subshift(u32 in[], int pos) -{ - return (__aesti_sbox[in[pos] & 0xff]) ^ - (__aesti_sbox[(in[(pos + 1) % 4] >> 8) & 0xff] << 8) ^ - (__aesti_sbox[(in[(pos + 2) % 4] >> 16) & 0xff] << 16) ^ - (__aesti_sbox[(in[(pos + 3) % 4] >> 24) & 0xff] << 24); -} - -static __always_inline u32 inv_subshift(u32 in[], int pos) -{ - return (__aesti_inv_sbox[in[pos] & 0xff]) ^ - (__aesti_inv_sbox[(in[(pos + 3) % 4] >> 8) & 0xff] << 8) ^ - (__aesti_inv_sbox[(in[(pos + 2) % 4] >> 16) & 0xff] << 16) ^ - (__aesti_inv_sbox[(in[(pos + 1) % 4] >> 24) & 0xff] << 24); -} - -static u32 subw(u32 in) -{ - return (__aesti_sbox[in & 0xff]) ^ - (__aesti_sbox[(in >> 8) & 0xff] << 8) ^ - (__aesti_sbox[(in >> 16) & 0xff] << 16) ^ - (__aesti_sbox[(in >> 24) & 0xff] << 24); -} - -static int aesti_expand_key(struct crypto_aes_ctx *ctx, const u8 *in_key, - unsigned int key_len) -{ - u32 kwords = key_len / sizeof(u32); - u32 rc, i, j; - - if (key_len != AES_KEYSIZE_128 && - key_len != AES_KEYSIZE_192 && - key_len != AES_KEYSIZE_256) - return -EINVAL; - - ctx->key_length = key_len; - - for (i = 0; i < kwords; i++) - ctx->key_enc[i] = get_unaligned_le32(in_key + i * sizeof(u32)); - - for (i = 0, rc = 1; i < 10; i++, rc = mul_by_x(rc)) { - u32 *rki = ctx->key_enc + (i * kwords); - u32 *rko = rki + kwords; - - rko[0] = ror32(subw(rki[kwords - 1]), 8) ^ rc ^ rki[0]; - rko[1] = rko[0] ^ rki[1]; - rko[2] = rko[1] ^ rki[2]; - rko[3] = rko[2] ^ rki[3]; - - if (key_len == 24) { - if (i >= 7) - break; - rko[4] = rko[3] ^ rki[4]; - rko[5] = rko[4] ^ rki[5]; - } else if (key_len == 32) { - if (i >= 6) - break; - rko[4] = subw(rko[3]) ^ rki[4]; - rko[5] = rko[4] ^ rki[5]; - rko[6] = rko[5] ^ rki[6]; - rko[7] = rko[6] ^ rki[7]; - } - } - - /* - * Generate the decryption keys for the Equivalent Inverse Cipher. - * This involves reversing the order of the round keys, and applying - * the Inverse Mix Columns transformation to all but the first and - * the last one. - */ - ctx->key_dec[0] = ctx->key_enc[key_len + 24]; - ctx->key_dec[1] = ctx->key_enc[key_len + 25]; - ctx->key_dec[2] = ctx->key_enc[key_len + 26]; - ctx->key_dec[3] = ctx->key_enc[key_len + 27]; - - for (i = 4, j = key_len + 20; j > 0; i += 4, j -= 4) { - ctx->key_dec[i] = inv_mix_columns(ctx->key_enc[j]); - ctx->key_dec[i + 1] = inv_mix_columns(ctx->key_enc[j + 1]); - ctx->key_dec[i + 2] = inv_mix_columns(ctx->key_enc[j + 2]); - ctx->key_dec[i + 3] = inv_mix_columns(ctx->key_enc[j + 3]); - } - ctx->key_dec[i] = ctx->key_enc[0]; - ctx->key_dec[i + 1] = ctx->key_enc[1]; - ctx->key_dec[i + 2] = ctx->key_enc[2]; - ctx->key_dec[i + 3] = ctx->key_enc[3]; - - return 0; -} static int aesti_set_key(struct crypto_tfm *tfm, const u8 *in_key, unsigned int key_len) { struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm); - return aesti_expand_key(ctx, in_key, key_len); + return aes_expandkey(ctx, in_key, key_len); } static void aesti_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) { const struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm); - const u32 *rkp = ctx->key_enc + 4; - int rounds = 6 + ctx->key_length / 4; - u32 st0[4], st1[4]; unsigned long flags; - int round; - - st0[0] = ctx->key_enc[0] ^ get_unaligned_le32(in); - st0[1] = ctx->key_enc[1] ^ get_unaligned_le32(in + 4); - st0[2] = ctx->key_enc[2] ^ get_unaligned_le32(in + 8); - st0[3] = ctx->key_enc[3] ^ get_unaligned_le32(in + 12); /* * Temporarily disable interrupts to avoid races where cachelines are @@ -261,36 +29,7 @@ static void aesti_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) */ local_irq_save(flags); - /* - * Force the compiler to emit data independent Sbox references, - * by xoring the input with Sbox values that are known to add up - * to zero. This pulls the entire Sbox into the D-cache before any - * data dependent lookups are done. - */ - st0[0] ^= __aesti_sbox[ 0] ^ __aesti_sbox[ 64] ^ __aesti_sbox[134] ^ __aesti_sbox[195]; - st0[1] ^= __aesti_sbox[16] ^ __aesti_sbox[ 82] ^ __aesti_sbox[158] ^ __aesti_sbox[221]; - st0[2] ^= __aesti_sbox[32] ^ __aesti_sbox[ 96] ^ __aesti_sbox[160] ^ __aesti_sbox[234]; - st0[3] ^= __aesti_sbox[48] ^ __aesti_sbox[112] ^ __aesti_sbox[186] ^ __aesti_sbox[241]; - - for (round = 0;; round += 2, rkp += 8) { - st1[0] = mix_columns(subshift(st0, 0)) ^ rkp[0]; - st1[1] = mix_columns(subshift(st0, 1)) ^ rkp[1]; - st1[2] = mix_columns(subshift(st0, 2)) ^ rkp[2]; - st1[3] = mix_columns(subshift(st0, 3)) ^ rkp[3]; - - if (round == rounds - 2) - break; - - st0[0] = mix_columns(subshift(st1, 0)) ^ rkp[4]; - st0[1] = mix_columns(subshift(st1, 1)) ^ rkp[5]; - st0[2] = mix_columns(subshift(st1, 2)) ^ rkp[6]; - st0[3] = mix_columns(subshift(st1, 3)) ^ rkp[7]; - } - - put_unaligned_le32(subshift(st1, 0) ^ rkp[4], out); - put_unaligned_le32(subshift(st1, 1) ^ rkp[5], out + 4); - put_unaligned_le32(subshift(st1, 2) ^ rkp[6], out + 8); - put_unaligned_le32(subshift(st1, 3) ^ rkp[7], out + 12); + aes_encrypt(ctx, out, in); local_irq_restore(flags); } @@ -298,16 +37,7 @@ static void aesti_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) static void aesti_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) { const struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm); - const u32 *rkp = ctx->key_dec + 4; - int rounds = 6 + ctx->key_length / 4; - u32 st0[4], st1[4]; unsigned long flags; - int round; - - st0[0] = ctx->key_dec[0] ^ get_unaligned_le32(in); - st0[1] = ctx->key_dec[1] ^ get_unaligned_le32(in + 4); - st0[2] = ctx->key_dec[2] ^ get_unaligned_le32(in + 8); - st0[3] = ctx->key_dec[3] ^ get_unaligned_le32(in + 12); /* * Temporarily disable interrupts to avoid races where cachelines are @@ -315,36 +45,7 @@ static void aesti_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) */ local_irq_save(flags); - /* - * Force the compiler to emit data independent Sbox references, - * by xoring the input with Sbox values that are known to add up - * to zero. This pulls the entire Sbox into the D-cache before any - * data dependent lookups are done. - */ - st0[0] ^= __aesti_inv_sbox[ 0] ^ __aesti_inv_sbox[ 64] ^ __aesti_inv_sbox[129] ^ __aesti_inv_sbox[200]; - st0[1] ^= __aesti_inv_sbox[16] ^ __aesti_inv_sbox[ 83] ^ __aesti_inv_sbox[150] ^ __aesti_inv_sbox[212]; - st0[2] ^= __aesti_inv_sbox[32] ^ __aesti_inv_sbox[ 96] ^ __aesti_inv_sbox[160] ^ __aesti_inv_sbox[236]; - st0[3] ^= __aesti_inv_sbox[48] ^ __aesti_inv_sbox[112] ^ __aesti_inv_sbox[187] ^ __aesti_inv_sbox[247]; - - for (round = 0;; round += 2, rkp += 8) { - st1[0] = inv_mix_columns(inv_subshift(st0, 0)) ^ rkp[0]; - st1[1] = inv_mix_columns(inv_subshift(st0, 1)) ^ rkp[1]; - st1[2] = inv_mix_columns(inv_subshift(st0, 2)) ^ rkp[2]; - st1[3] = inv_mix_columns(inv_subshift(st0, 3)) ^ rkp[3]; - - if (round == rounds - 2) - break; - - st0[0] = inv_mix_columns(inv_subshift(st1, 0)) ^ rkp[4]; - st0[1] = inv_mix_columns(inv_subshift(st1, 1)) ^ rkp[5]; - st0[2] = inv_mix_columns(inv_subshift(st1, 2)) ^ rkp[6]; - st0[3] = inv_mix_columns(inv_subshift(st1, 3)) ^ rkp[7]; - } - - put_unaligned_le32(inv_subshift(st1, 0) ^ rkp[4], out); - put_unaligned_le32(inv_subshift(st1, 1) ^ rkp[5], out + 4); - put_unaligned_le32(inv_subshift(st1, 2) ^ rkp[6], out + 8); - put_unaligned_le32(inv_subshift(st1, 3) ^ rkp[7], out + 12); + aes_decrypt(ctx, out, in); local_irq_restore(flags); } diff --git a/include/crypto/aes.h b/include/crypto/aes.h index 0fdb542c70cd..d0067fca0cd0 100644 --- a/include/crypto/aes.h +++ b/include/crypto/aes.h @@ -37,4 +37,38 @@ int crypto_aes_set_key(struct crypto_tfm *tfm, const u8 *in_key, unsigned int key_len); int crypto_aes_expand_key(struct crypto_aes_ctx *ctx, const u8 *in_key, unsigned int key_len); + +/** + * aes_expandkey - Expands the AES key as described in FIPS-197 + * @ctx: The location where the computed key will be stored. + * @in_key: The supplied key. + * @key_len: The length of the supplied key. + * + * Returns 0 on success. The function fails only if an invalid key size (or + * pointer) is supplied. + * The expanded key size is 240 bytes (max of 14 rounds with a unique 16 bytes + * key schedule plus a 16 bytes key which is used before the first round). + * The decryption key is prepared for the "Equivalent Inverse Cipher" as + * described in FIPS-197. The first slot (16 bytes) of each key (enc or dec) is + * for the initial combination, the second slot for the first round and so on. + */ +int aes_expandkey(struct crypto_aes_ctx *ctx, const u8 *in_key, + unsigned int key_len); + +/** + * aes_encrypt - Encrypt a single AES block + * @ctx: Context struct containing the key schedule + * @out: Buffer to store the ciphertext + * @in: Buffer containing the plaintext + */ +void aes_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in); + +/** + * aes_decrypt - Decrypt a single AES block + * @ctx: Context struct containing the key schedule + * @out: Buffer to store the plaintext + * @in: Buffer containing the ciphertext + */ +void aes_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in); + #endif diff --git a/lib/crypto/Makefile b/lib/crypto/Makefile index 88195c34932d..42a91c62d96d 100644 --- a/lib/crypto/Makefile +++ b/lib/crypto/Makefile @@ -1,4 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 +obj-$(CONFIG_CRYPTO_LIB_AES) += libaes.o +libaes-y := aes.o + obj-$(CONFIG_CRYPTO_LIB_ARC4) += libarc4.o libarc4-y := arc4.o diff --git a/lib/crypto/aes.c b/lib/crypto/aes.c new file mode 100644 index 000000000000..9928b23e0a8a --- /dev/null +++ b/lib/crypto/aes.c @@ -0,0 +1,350 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2017-2019 Linaro Ltd + */ + +#include +#include +#include +#include + +/* + * Emit the sbox as volatile const to prevent the compiler from doing + * constant folding on sbox references involving fixed indexes. + */ +static volatile const u8 __cacheline_aligned aes_sbox[] = { + 0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5, + 0x30, 0x01, 0x67, 0x2b, 0xfe, 0xd7, 0xab, 0x76, + 0xca, 0x82, 0xc9, 0x7d, 0xfa, 0x59, 0x47, 0xf0, + 0xad, 0xd4, 0xa2, 0xaf, 0x9c, 0xa4, 0x72, 0xc0, + 0xb7, 0xfd, 0x93, 0x26, 0x36, 0x3f, 0xf7, 0xcc, + 0x34, 0xa5, 0xe5, 0xf1, 0x71, 0xd8, 0x31, 0x15, + 0x04, 0xc7, 0x23, 0xc3, 0x18, 0x96, 0x05, 0x9a, + 0x07, 0x12, 0x80, 0xe2, 0xeb, 0x27, 0xb2, 0x75, + 0x09, 0x83, 0x2c, 0x1a, 0x1b, 0x6e, 0x5a, 0xa0, + 0x52, 0x3b, 0xd6, 0xb3, 0x29, 0xe3, 0x2f, 0x84, + 0x53, 0xd1, 0x00, 0xed, 0x20, 0xfc, 0xb1, 0x5b, + 0x6a, 0xcb, 0xbe, 0x39, 0x4a, 0x4c, 0x58, 0xcf, + 0xd0, 0xef, 0xaa, 0xfb, 0x43, 0x4d, 0x33, 0x85, + 0x45, 0xf9, 0x02, 0x7f, 0x50, 0x3c, 0x9f, 0xa8, + 0x51, 0xa3, 0x40, 0x8f, 0x92, 0x9d, 0x38, 0xf5, + 0xbc, 0xb6, 0xda, 0x21, 0x10, 0xff, 0xf3, 0xd2, + 0xcd, 0x0c, 0x13, 0xec, 0x5f, 0x97, 0x44, 0x17, + 0xc4, 0xa7, 0x7e, 0x3d, 0x64, 0x5d, 0x19, 0x73, + 0x60, 0x81, 0x4f, 0xdc, 0x22, 0x2a, 0x90, 0x88, + 0x46, 0xee, 0xb8, 0x14, 0xde, 0x5e, 0x0b, 0xdb, + 0xe0, 0x32, 0x3a, 0x0a, 0x49, 0x06, 0x24, 0x5c, + 0xc2, 0xd3, 0xac, 0x62, 0x91, 0x95, 0xe4, 0x79, + 0xe7, 0xc8, 0x37, 0x6d, 0x8d, 0xd5, 0x4e, 0xa9, + 0x6c, 0x56, 0xf4, 0xea, 0x65, 0x7a, 0xae, 0x08, + 0xba, 0x78, 0x25, 0x2e, 0x1c, 0xa6, 0xb4, 0xc6, + 0xe8, 0xdd, 0x74, 0x1f, 0x4b, 0xbd, 0x8b, 0x8a, + 0x70, 0x3e, 0xb5, 0x66, 0x48, 0x03, 0xf6, 0x0e, + 0x61, 0x35, 0x57, 0xb9, 0x86, 0xc1, 0x1d, 0x9e, + 0xe1, 0xf8, 0x98, 0x11, 0x69, 0xd9, 0x8e, 0x94, + 0x9b, 0x1e, 0x87, 0xe9, 0xce, 0x55, 0x28, 0xdf, + 0x8c, 0xa1, 0x89, 0x0d, 0xbf, 0xe6, 0x42, 0x68, + 0x41, 0x99, 0x2d, 0x0f, 0xb0, 0x54, 0xbb, 0x16, +}; + +static volatile const u8 __cacheline_aligned aes_inv_sbox[] = { + 0x52, 0x09, 0x6a, 0xd5, 0x30, 0x36, 0xa5, 0x38, + 0xbf, 0x40, 0xa3, 0x9e, 0x81, 0xf3, 0xd7, 0xfb, + 0x7c, 0xe3, 0x39, 0x82, 0x9b, 0x2f, 0xff, 0x87, + 0x34, 0x8e, 0x43, 0x44, 0xc4, 0xde, 0xe9, 0xcb, + 0x54, 0x7b, 0x94, 0x32, 0xa6, 0xc2, 0x23, 0x3d, + 0xee, 0x4c, 0x95, 0x0b, 0x42, 0xfa, 0xc3, 0x4e, + 0x08, 0x2e, 0xa1, 0x66, 0x28, 0xd9, 0x24, 0xb2, + 0x76, 0x5b, 0xa2, 0x49, 0x6d, 0x8b, 0xd1, 0x25, + 0x72, 0xf8, 0xf6, 0x64, 0x86, 0x68, 0x98, 0x16, + 0xd4, 0xa4, 0x5c, 0xcc, 0x5d, 0x65, 0xb6, 0x92, + 0x6c, 0x70, 0x48, 0x50, 0xfd, 0xed, 0xb9, 0xda, + 0x5e, 0x15, 0x46, 0x57, 0xa7, 0x8d, 0x9d, 0x84, + 0x90, 0xd8, 0xab, 0x00, 0x8c, 0xbc, 0xd3, 0x0a, + 0xf7, 0xe4, 0x58, 0x05, 0xb8, 0xb3, 0x45, 0x06, + 0xd0, 0x2c, 0x1e, 0x8f, 0xca, 0x3f, 0x0f, 0x02, + 0xc1, 0xaf, 0xbd, 0x03, 0x01, 0x13, 0x8a, 0x6b, + 0x3a, 0x91, 0x11, 0x41, 0x4f, 0x67, 0xdc, 0xea, + 0x97, 0xf2, 0xcf, 0xce, 0xf0, 0xb4, 0xe6, 0x73, + 0x96, 0xac, 0x74, 0x22, 0xe7, 0xad, 0x35, 0x85, + 0xe2, 0xf9, 0x37, 0xe8, 0x1c, 0x75, 0xdf, 0x6e, + 0x47, 0xf1, 0x1a, 0x71, 0x1d, 0x29, 0xc5, 0x89, + 0x6f, 0xb7, 0x62, 0x0e, 0xaa, 0x18, 0xbe, 0x1b, + 0xfc, 0x56, 0x3e, 0x4b, 0xc6, 0xd2, 0x79, 0x20, + 0x9a, 0xdb, 0xc0, 0xfe, 0x78, 0xcd, 0x5a, 0xf4, + 0x1f, 0xdd, 0xa8, 0x33, 0x88, 0x07, 0xc7, 0x31, + 0xb1, 0x12, 0x10, 0x59, 0x27, 0x80, 0xec, 0x5f, + 0x60, 0x51, 0x7f, 0xa9, 0x19, 0xb5, 0x4a, 0x0d, + 0x2d, 0xe5, 0x7a, 0x9f, 0x93, 0xc9, 0x9c, 0xef, + 0xa0, 0xe0, 0x3b, 0x4d, 0xae, 0x2a, 0xf5, 0xb0, + 0xc8, 0xeb, 0xbb, 0x3c, 0x83, 0x53, 0x99, 0x61, + 0x17, 0x2b, 0x04, 0x7e, 0xba, 0x77, 0xd6, 0x26, + 0xe1, 0x69, 0x14, 0x63, 0x55, 0x21, 0x0c, 0x7d, +}; + +static u32 mul_by_x(u32 w) +{ + u32 x = w & 0x7f7f7f7f; + u32 y = w & 0x80808080; + + /* multiply by polynomial 'x' (0b10) in GF(2^8) */ + return (x << 1) ^ (y >> 7) * 0x1b; +} + +static u32 mul_by_x2(u32 w) +{ + u32 x = w & 0x3f3f3f3f; + u32 y = w & 0x80808080; + u32 z = w & 0x40404040; + + /* multiply by polynomial 'x^2' (0b100) in GF(2^8) */ + return (x << 2) ^ (y >> 7) * 0x36 ^ (z >> 6) * 0x1b; +} + +static u32 mix_columns(u32 x) +{ + /* + * Perform the following matrix multiplication in GF(2^8) + * + * | 0x2 0x3 0x1 0x1 | | x[0] | + * | 0x1 0x2 0x3 0x1 | | x[1] | + * | 0x1 0x1 0x2 0x3 | x | x[2] | + * | 0x3 0x1 0x1 0x2 | | x[3] | + */ + u32 y = mul_by_x(x) ^ ror32(x, 16); + + return y ^ ror32(x ^ y, 8); +} + +static u32 inv_mix_columns(u32 x) +{ + /* + * Perform the following matrix multiplication in GF(2^8) + * + * | 0xe 0xb 0xd 0x9 | | x[0] | + * | 0x9 0xe 0xb 0xd | | x[1] | + * | 0xd 0x9 0xe 0xb | x | x[2] | + * | 0xb 0xd 0x9 0xe | | x[3] | + * + * which can conveniently be reduced to + * + * | 0x2 0x3 0x1 0x1 | | 0x5 0x0 0x4 0x0 | | x[0] | + * | 0x1 0x2 0x3 0x1 | | 0x0 0x5 0x0 0x4 | | x[1] | + * | 0x1 0x1 0x2 0x3 | x | 0x4 0x0 0x5 0x0 | x | x[2] | + * | 0x3 0x1 0x1 0x2 | | 0x0 0x4 0x0 0x5 | | x[3] | + */ + u32 y = mul_by_x2(x); + + return mix_columns(x ^ y ^ ror32(y, 16)); +} + +static __always_inline u32 subshift(u32 in[], int pos) +{ + return (aes_sbox[in[pos] & 0xff]) ^ + (aes_sbox[(in[(pos + 1) % 4] >> 8) & 0xff] << 8) ^ + (aes_sbox[(in[(pos + 2) % 4] >> 16) & 0xff] << 16) ^ + (aes_sbox[(in[(pos + 3) % 4] >> 24) & 0xff] << 24); +} + +static __always_inline u32 inv_subshift(u32 in[], int pos) +{ + return (aes_inv_sbox[in[pos] & 0xff]) ^ + (aes_inv_sbox[(in[(pos + 3) % 4] >> 8) & 0xff] << 8) ^ + (aes_inv_sbox[(in[(pos + 2) % 4] >> 16) & 0xff] << 16) ^ + (aes_inv_sbox[(in[(pos + 1) % 4] >> 24) & 0xff] << 24); +} + +static u32 subw(u32 in) +{ + return (aes_sbox[in & 0xff]) ^ + (aes_sbox[(in >> 8) & 0xff] << 8) ^ + (aes_sbox[(in >> 16) & 0xff] << 16) ^ + (aes_sbox[(in >> 24) & 0xff] << 24); +} + +/** + * aes_expandkey - Expands the AES key as described in FIPS-197 + * @ctx: The location where the computed key will be stored. + * @in_key: The supplied key. + * @key_len: The length of the supplied key. + * + * Returns 0 on success. The function fails only if an invalid key size (or + * pointer) is supplied. + * The expanded key size is 240 bytes (max of 14 rounds with a unique 16 bytes + * key schedule plus a 16 bytes key which is used before the first round). + * The decryption key is prepared for the "Equivalent Inverse Cipher" as + * described in FIPS-197. The first slot (16 bytes) of each key (enc or dec) is + * for the initial combination, the second slot for the first round and so on. + */ +int aes_expandkey(struct crypto_aes_ctx *ctx, const u8 *in_key, + unsigned int key_len) +{ + u32 kwords = key_len / sizeof(u32); + u32 rc, i, j; + + if (key_len != AES_KEYSIZE_128 && + key_len != AES_KEYSIZE_192 && + key_len != AES_KEYSIZE_256) + return -EINVAL; + + ctx->key_length = key_len; + + for (i = 0; i < kwords; i++) + ctx->key_enc[i] = get_unaligned_le32(in_key + i * sizeof(u32)); + + for (i = 0, rc = 1; i < 10; i++, rc = mul_by_x(rc)) { + u32 *rki = ctx->key_enc + (i * kwords); + u32 *rko = rki + kwords; + + rko[0] = ror32(subw(rki[kwords - 1]), 8) ^ rc ^ rki[0]; + rko[1] = rko[0] ^ rki[1]; + rko[2] = rko[1] ^ rki[2]; + rko[3] = rko[2] ^ rki[3]; + + if (key_len == AES_KEYSIZE_192) { + if (i >= 7) + break; + rko[4] = rko[3] ^ rki[4]; + rko[5] = rko[4] ^ rki[5]; + } else if (key_len == AES_KEYSIZE_256) { + if (i >= 6) + break; + rko[4] = subw(rko[3]) ^ rki[4]; + rko[5] = rko[4] ^ rki[5]; + rko[6] = rko[5] ^ rki[6]; + rko[7] = rko[6] ^ rki[7]; + } + } + + /* + * Generate the decryption keys for the Equivalent Inverse Cipher. + * This involves reversing the order of the round keys, and applying + * the Inverse Mix Columns transformation to all but the first and + * the last one. + */ + ctx->key_dec[0] = ctx->key_enc[key_len + 24]; + ctx->key_dec[1] = ctx->key_enc[key_len + 25]; + ctx->key_dec[2] = ctx->key_enc[key_len + 26]; + ctx->key_dec[3] = ctx->key_enc[key_len + 27]; + + for (i = 4, j = key_len + 20; j > 0; i += 4, j -= 4) { + ctx->key_dec[i] = inv_mix_columns(ctx->key_enc[j]); + ctx->key_dec[i + 1] = inv_mix_columns(ctx->key_enc[j + 1]); + ctx->key_dec[i + 2] = inv_mix_columns(ctx->key_enc[j + 2]); + ctx->key_dec[i + 3] = inv_mix_columns(ctx->key_enc[j + 3]); + } + + ctx->key_dec[i] = ctx->key_enc[0]; + ctx->key_dec[i + 1] = ctx->key_enc[1]; + ctx->key_dec[i + 2] = ctx->key_enc[2]; + ctx->key_dec[i + 3] = ctx->key_enc[3]; + + return 0; +} +EXPORT_SYMBOL(aes_expandkey); + +/** + * aes_encrypt - Encrypt a single AES block + * @ctx: Context struct containing the key schedule + * @out: Buffer to store the ciphertext + * @in: Buffer containing the plaintext + */ +void aes_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in) +{ + const u32 *rkp = ctx->key_enc + 4; + int rounds = 6 + ctx->key_length / 4; + u32 st0[4], st1[4]; + int round; + + st0[0] = ctx->key_enc[0] ^ get_unaligned_le32(in); + st0[1] = ctx->key_enc[1] ^ get_unaligned_le32(in + 4); + st0[2] = ctx->key_enc[2] ^ get_unaligned_le32(in + 8); + st0[3] = ctx->key_enc[3] ^ get_unaligned_le32(in + 12); + + /* + * Force the compiler to emit data independent Sbox references, + * by xoring the input with Sbox values that are known to add up + * to zero. This pulls the entire Sbox into the D-cache before any + * data dependent lookups are done. + */ + st0[0] ^= aes_sbox[ 0] ^ aes_sbox[ 64] ^ aes_sbox[134] ^ aes_sbox[195]; + st0[1] ^= aes_sbox[16] ^ aes_sbox[ 82] ^ aes_sbox[158] ^ aes_sbox[221]; + st0[2] ^= aes_sbox[32] ^ aes_sbox[ 96] ^ aes_sbox[160] ^ aes_sbox[234]; + st0[3] ^= aes_sbox[48] ^ aes_sbox[112] ^ aes_sbox[186] ^ aes_sbox[241]; + + for (round = 0;; round += 2, rkp += 8) { + st1[0] = mix_columns(subshift(st0, 0)) ^ rkp[0]; + st1[1] = mix_columns(subshift(st0, 1)) ^ rkp[1]; + st1[2] = mix_columns(subshift(st0, 2)) ^ rkp[2]; + st1[3] = mix_columns(subshift(st0, 3)) ^ rkp[3]; + + if (round == rounds - 2) + break; + + st0[0] = mix_columns(subshift(st1, 0)) ^ rkp[4]; + st0[1] = mix_columns(subshift(st1, 1)) ^ rkp[5]; + st0[2] = mix_columns(subshift(st1, 2)) ^ rkp[6]; + st0[3] = mix_columns(subshift(st1, 3)) ^ rkp[7]; + } + + put_unaligned_le32(subshift(st1, 0) ^ rkp[4], out); + put_unaligned_le32(subshift(st1, 1) ^ rkp[5], out + 4); + put_unaligned_le32(subshift(st1, 2) ^ rkp[6], out + 8); + put_unaligned_le32(subshift(st1, 3) ^ rkp[7], out + 12); +} +EXPORT_SYMBOL(aes_encrypt); + +/** + * aes_decrypt - Decrypt a single AES block + * @ctx: Context struct containing the key schedule + * @out: Buffer to store the plaintext + * @in: Buffer containing the ciphertext + */ +void aes_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in) +{ + const u32 *rkp = ctx->key_dec + 4; + int rounds = 6 + ctx->key_length / 4; + u32 st0[4], st1[4]; + int round; + + st0[0] = ctx->key_dec[0] ^ get_unaligned_le32(in); + st0[1] = ctx->key_dec[1] ^ get_unaligned_le32(in + 4); + st0[2] = ctx->key_dec[2] ^ get_unaligned_le32(in + 8); + st0[3] = ctx->key_dec[3] ^ get_unaligned_le32(in + 12); + + /* + * Force the compiler to emit data independent Sbox references, + * by xoring the input with Sbox values that are known to add up + * to zero. This pulls the entire Sbox into the D-cache before any + * data dependent lookups are done. + */ + st0[0] ^= aes_inv_sbox[ 0] ^ aes_inv_sbox[ 64] ^ aes_inv_sbox[129] ^ aes_inv_sbox[200]; + st0[1] ^= aes_inv_sbox[16] ^ aes_inv_sbox[ 83] ^ aes_inv_sbox[150] ^ aes_inv_sbox[212]; + st0[2] ^= aes_inv_sbox[32] ^ aes_inv_sbox[ 96] ^ aes_inv_sbox[160] ^ aes_inv_sbox[236]; + st0[3] ^= aes_inv_sbox[48] ^ aes_inv_sbox[112] ^ aes_inv_sbox[187] ^ aes_inv_sbox[247]; + + for (round = 0;; round += 2, rkp += 8) { + st1[0] = inv_mix_columns(inv_subshift(st0, 0)) ^ rkp[0]; + st1[1] = inv_mix_columns(inv_subshift(st0, 1)) ^ rkp[1]; + st1[2] = inv_mix_columns(inv_subshift(st0, 2)) ^ rkp[2]; + st1[3] = inv_mix_columns(inv_subshift(st0, 3)) ^ rkp[3]; + + if (round == rounds - 2) + break; + + st0[0] = inv_mix_columns(inv_subshift(st1, 0)) ^ rkp[4]; + st0[1] = inv_mix_columns(inv_subshift(st1, 1)) ^ rkp[5]; + st0[2] = inv_mix_columns(inv_subshift(st1, 2)) ^ rkp[6]; + st0[3] = inv_mix_columns(inv_subshift(st1, 3)) ^ rkp[7]; + } + + put_unaligned_le32(inv_subshift(st1, 0) ^ rkp[4], out); + put_unaligned_le32(inv_subshift(st1, 1) ^ rkp[5], out + 4); + put_unaligned_le32(inv_subshift(st1, 2) ^ rkp[6], out + 8); + put_unaligned_le32(inv_subshift(st1, 3) ^ rkp[7], out + 12); +} +EXPORT_SYMBOL(aes_decrypt); + +MODULE_DESCRIPTION("Generic AES library"); +MODULE_AUTHOR("Ard Biesheuvel "); +MODULE_LICENSE("GPL v2"); From patchwork Sat Jun 22 19:34:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167488 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2255866ilk; Sat, 22 Jun 2019 12:34:49 -0700 (PDT) X-Google-Smtp-Source: APXvYqxNESAsMVORXmcMwXIXSdxJYV6Ku4mH++fQWzn+yKi7QoiZvBN2yDf/UTpqos0qsQDsXRwC X-Received: by 2002:a17:90a:710c:: with SMTP id h12mr14381220pjk.36.1561232089029; Sat, 22 Jun 2019 12:34:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232089; cv=none; d=google.com; s=arc-20160816; b=Ul4jS4/k2YjvqUCPQIb3O3dOqKxfz0tI+XCUa3r12r8ASgdnjrhAoLZ57HmplzzJfX 8UVs3iDY2GT6CmqYqNnmkKemjkX2OzhaRRnMDNBDGjtuauPH0z/EeeIXCJNyZgg/ox02 JqJpvkA9U0TTcUWBwKFabYqepgo5TL8BeNQR08qOQJhanR9zFLE87Jg7EF47ZzJcvGhO gTP/nBfmtRLPEZghBuPjgGdOKNNhdcNibi8AQLBFysn3VnMqa5HfKkZQ1yxpnl/U+SMc zTf1AsTK/5vU7JKi7W1EModJJ2pTe06h2gk35897M1yuGd0Oc/Jsa4BMTU2t47wonwbv KJoQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5Oqvcot1yZKpSWtboJ7OjflLCxxMos1RB3yPJW9XMLo=; b=XdLltKD9leyXwdIG9aa4iRjReDw4IykJkP7Cl6EdIv7m33IU8UB2TcQJC+61C15Oz4 neT76xtSctqbKMsWu71QZd7nVTl25XGEV6koMQghaLXZw7LJYZs96sej4XRO4HlnXkmH prRPGrip0HNQiuU1Hp87DnUFj06wmBV6XBS/tGj9CWpv7uQp7/oWDgfNqgBU8J12eFrR qcAF4RTZd07noCIRGiwk0o84I7zjOGNvmIsHxdYyNQc0S4WoqnwuaOTuanpISHoHBXw8 ny3wA0Z7kCRjT4HqnHNHhq/AW9IMoV6L2ncGVhOwswUPfYOABE8g+zi4SgMFrsPSTxJP OnIA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=pTL9q9uQ; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.34.48; Sat, 22 Jun 2019 12:34:49 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=pTL9q9uQ; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726342AbfFVTes (ORCPT + 3 others); Sat, 22 Jun 2019 15:34:48 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:33739 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726338AbfFVTes (ORCPT ); Sat, 22 Jun 2019 15:34:48 -0400 Received: by mail-wm1-f65.google.com with SMTP id h19so11351617wme.0 for ; Sat, 22 Jun 2019 12:34:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5Oqvcot1yZKpSWtboJ7OjflLCxxMos1RB3yPJW9XMLo=; b=pTL9q9uQWGDoEDBmSirmxQxs1RKz+z9+olTB1MNYmmYhAR764ygt7Hfbtnioif+CTO wbpdJch9/BLv6CLdXAg7flE9434mH9Si1UpokHFmutWE7sZ2xRyR15dfKjAEBBG5IRWR ub+6GhcU0dvQO2QvblFqove/vzIFBt823QaWQ9Hk/XJx6f1YcX2p3KRZzCOMtCBcKgM6 f8raGMHokHvX67dn6ftXUPtZ8DBSUr4+nzMIoM/yOttOn3qOgCi2MGHVQWNoMl0Dn4f8 5UXcrpfNwCOrQrOkJEbcMlb86292qtKz1/sgxoaOyA+RNJh2FmkP8bBu0EzonvtQWu24 NtcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5Oqvcot1yZKpSWtboJ7OjflLCxxMos1RB3yPJW9XMLo=; b=TWTd6FtugXmuBlaqC+3i559EELuPu/App1Ip0h2hcIqfuOY/ZVSGlQv1UA9+aXbznB 2XK2WZcr3tb4Xps5vS06DXxJrY6rduUlpzoct97rn12H5MRuagwvYPdlvGTVb2sgCnHv h2MfPS4ql2xvCKukhnwVQgo5ay6D1KoxNNwzJnrbQGrhf2q3j3Uv8m3JyubGEAEtTFks ZGFDjDxO4vquuAGZzhiCuS9CsSQIpma8qCLBG6UhFlbl4VMcTTXnhGxeCuEDDT5yhEH0 p5czKK/LSA4fJnaReGTY+j4Db1qgIrde8ouuUUemrlaAD57xv8ghkG7+Yf0u96Jw3cQe mJFw== X-Gm-Message-State: APjAAAX5g1qeCH4NVclN2yYlDHA3OFNVyuSoA7U09Sudbjow5edQ9gAD Aa85eABA0HU6Bh3+ImDicHpO11zGxj+4Hzpy X-Received: by 2002:a05:600c:20d:: with SMTP id 13mr8680233wmi.141.1561232085469; Sat, 22 Jun 2019 12:34:45 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:44 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 05/26] crypto: x86/aes-ni - switch to generic for fallback and key routines Date: Sat, 22 Jun 2019 21:34:06 +0200 Message-Id: <20190622193427.20336-6-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The AES-NI code contains fallbacks for invocations that occur from a context where the SIMD unit is unavailable, which really only occurs when running in softirq context that was entered from a hard IRQ that was taken while running kernel code that was already using the FPU. That means performance is not really a consideration, and we can just use the new library code for this use case, which has a smaller footprint and is believed to be time invariant. This will allow us to drop the non-SIMD asm routines in a subsequent patch. Signed-off-by: Ard Biesheuvel --- arch/x86/crypto/aesni-intel_glue.c | 15 +++++++-------- arch/x86/include/asm/crypto/aes.h | 12 ------------ crypto/Kconfig | 3 +-- 3 files changed, 8 insertions(+), 22 deletions(-) -- 2.20.1 diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c index 836d50bd096f..42873c1f6bb4 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -30,7 +30,6 @@ #include #include #include -#include #include #include #include @@ -333,7 +332,7 @@ static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx, } if (!crypto_simd_usable()) - err = crypto_aes_expand_key(ctx, in_key, key_len); + err = aes_expandkey(ctx, in_key, key_len); else { kernel_fpu_begin(); err = aesni_set_key(ctx, in_key, key_len); @@ -353,9 +352,9 @@ static void aesni_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) { struct crypto_aes_ctx *ctx = aes_ctx(crypto_tfm_ctx(tfm)); - if (!crypto_simd_usable()) - crypto_aes_encrypt_x86(ctx, dst, src); - else { + if (!crypto_simd_usable()) { + aes_encrypt(ctx, dst, src); + } else { kernel_fpu_begin(); aesni_enc(ctx, dst, src); kernel_fpu_end(); @@ -366,9 +365,9 @@ static void aesni_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) { struct crypto_aes_ctx *ctx = aes_ctx(crypto_tfm_ctx(tfm)); - if (!crypto_simd_usable()) - crypto_aes_decrypt_x86(ctx, dst, src); - else { + if (!crypto_simd_usable()) { + aes_decrypt(ctx, dst, src); + } else { kernel_fpu_begin(); aesni_dec(ctx, dst, src); kernel_fpu_end(); diff --git a/arch/x86/include/asm/crypto/aes.h b/arch/x86/include/asm/crypto/aes.h deleted file mode 100644 index c508521dd190..000000000000 --- a/arch/x86/include/asm/crypto/aes.h +++ /dev/null @@ -1,12 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef ASM_X86_AES_H -#define ASM_X86_AES_H - -#include -#include - -void crypto_aes_encrypt_x86(struct crypto_aes_ctx *ctx, u8 *dst, - const u8 *src); -void crypto_aes_decrypt_x86(struct crypto_aes_ctx *ctx, u8 *dst, - const u8 *src); -#endif diff --git a/crypto/Kconfig b/crypto/Kconfig index 091ebbbc9655..20af58068e6b 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -1156,8 +1156,7 @@ config CRYPTO_AES_NI_INTEL tristate "AES cipher algorithms (AES-NI)" depends on X86 select CRYPTO_AEAD - select CRYPTO_AES_X86_64 if 64BIT - select CRYPTO_AES_586 if !64BIT + select CRYPTO_LIB_AES select CRYPTO_ALGAPI select CRYPTO_BLKCIPHER select CRYPTO_GLUE_HELPER_X86 if 64BIT From patchwork Sat Jun 22 19:34:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167492 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2255900ilk; Sat, 22 Jun 2019 12:34:52 -0700 (PDT) X-Google-Smtp-Source: APXvYqyT/Vtkim76dTerCSr2V9QQCYATQQUUk7e9zizF8YYr1p1dIeViKUk00SwCzmkvvghLHg5P X-Received: by 2002:a63:f510:: with SMTP id w16mr25205370pgh.0.1561232092394; Sat, 22 Jun 2019 12:34:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232092; cv=none; d=google.com; s=arc-20160816; b=wtS+VZVow+8n3r996+IySCF5PHXnYbHtQNfH+YQn90bkQ+qGoiwn+NxKsidMwkmt+m 4hKrQ8k4X1ZKuIHI/Oy7gHZhPdAgFRzOS6RTQqz9N3ixPjF7pj9jqhgGrxPRjnCbCt61 WwXiBiCAw6q3YBVn+2e9e3VdrfMAhtKRdchCxuuAC/terLNwXN9sIb12tP6ayghTy9O5 OwdrXmw6IPedo6utaJiKHulq9Cw1PlpS3LwyDt0585p/y+h4F6kHidTVZNlDY83feZRa 6/lbC1L0XEQUTno2HGfEKOV0pSnwxAbRHw6R1ESeyezT14vQ1P4dUc0m3/auOg30oPiN AzUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Rk3H4Hh084k052p7bt4Y0A038t9wYhRs8MD3q3JWnkU=; b=P3dZLiBU8B5vA72Si9roIyQwSR+1uTo5FzkXqDiDp7lwViZvlp+v5JoJHug0xMrJ40 Dj5Hliyv3E4ciMzeukmMJ5VbMlhBhnMzN3sC/J3hSfdPm0TcrKntkFKFlnijDT/mUdK2 +llyNndeAXznCJJA/WbHkqe1MGxrx1SyT4W3KA5yytpgsyBPLJD1WoQkCzsSSks4y1C4 +WJr8VCv+ZxBFAUYqJ4ZBjLvOLQpmJLppp/8OrHeQAO2nfEvMguvNAaPSkSNFy8JzH3m 5aY235D4/5zOMN/MS4QzCxecMlEyQSnHRkQnRmj/ur5bw7ikUPdVgpNrRpdXjN/o5Vhq p1zA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=krbrbRi6; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.34.52; Sat, 22 Jun 2019 12:34:52 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=krbrbRi6; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726343AbfFVTev (ORCPT + 3 others); Sat, 22 Jun 2019 15:34:51 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:35945 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726286AbfFVTev (ORCPT ); Sat, 22 Jun 2019 15:34:51 -0400 Received: by mail-wr1-f65.google.com with SMTP id n4so8517081wrs.3 for ; Sat, 22 Jun 2019 12:34:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Rk3H4Hh084k052p7bt4Y0A038t9wYhRs8MD3q3JWnkU=; b=krbrbRi6R0EkYvFDlLyTdN8hbrygpu8HU5Q5phY7Oym5Xtks9p1Xt7vV8Z8DLP/aBF 9dDUHE+K3/MJjgEk2nP5SEeNkS/qfXr1h/2kwiMF8VrQV8xe6LPQVxRsll0figaYIdBb JQ7Uh5bdKc6DnUkxCDgNek8gnSmdpBwvIbqi0S37c3NKizURo2vl/yT1LGmjunwhlzPu 5dj+R7JyJ77tWyWviHW9mln31DxWe6x7SII/GDjfPk8W3NxbAEWW2FUCvAmeL8upxpba 8y9NQxmMxkZ54QZPaW1k9zbqHJoQ1DsIpYivV6gGeerIDDXRvoGpeWy+djyPvJBwzOsQ 539g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Rk3H4Hh084k052p7bt4Y0A038t9wYhRs8MD3q3JWnkU=; b=MmRo9JulPO8QRrpK/2iYSr12vrOT+R+UrEb656yB1riuDkyCBYMDmiB3pVWKh1hG8z RgUIB/RX5bHLOScuL2g5tx6egmy7ROxtCJCuPaTr2V6HpjAXtxsyyYF4opZu+YHUTBKn aY3dY/JrMNikaBnU2GExtT96Gf5UgV5gU6BOt6yyZgxAyybOus1pmSN3YVdezZBCz5lQ YLx/r0DyxoKkOmygw02xTLSOWfOAObGEN75GWa3hfBzy1+KNwX559B0iJ+/Inn2Khd1n nPipB+t1ZUZeTDX5PDNmPD7UiE0fgz3SiBuPoU8Ex7PpzWnpmro/inNg0/t3saDtG1U9 vmeg== X-Gm-Message-State: APjAAAXolz+ITzpNCyrBRB04CYV3QEseFHwWo2cIl5CHtuXsdOUIKly6 HTuR8/ZdIABkHDuvOkTJCx98AooSM8SDwr3E X-Received: by 2002:adf:fa4c:: with SMTP id y12mr87193688wrr.282.1561232086735; Sat, 22 Jun 2019 12:34:46 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:46 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 06/26] crypto: x86/aes - drop scalar assembler implementations Date: Sat, 22 Jun 2019 21:34:07 +0200 Message-Id: <20190622193427.20336-7-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The AES assembler code for x86 isn't actually faster than code generated by the compiler from aes_generic.c, and considering the disproportionate maintenance burden of assembler code on x86, it is better just to drop it entirely. Modern x86 systems will use AES-NI anyway, and given that the modules being removed have a dependency on aes_generic already, we can remove them without running the risk of regressions. Signed-off-by: Ard Biesheuvel --- arch/x86/crypto/Makefile | 4 - arch/x86/crypto/aes-i586-asm_32.S | 362 -------------------- arch/x86/crypto/aes-x86_64-asm_64.S | 185 ---------- arch/x86/crypto/aes_glue.c | 70 ---- crypto/Kconfig | 44 --- 5 files changed, 665 deletions(-) -- 2.20.1 diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile index 45734e1cf967..b96a14e67ab0 100644 --- a/arch/x86/crypto/Makefile +++ b/arch/x86/crypto/Makefile @@ -14,11 +14,9 @@ sha256_ni_supported :=$(call as-instr,sha256msg1 %xmm0$(comma)%xmm1,yes,no) obj-$(CONFIG_CRYPTO_GLUE_HELPER_X86) += glue_helper.o -obj-$(CONFIG_CRYPTO_AES_586) += aes-i586.o obj-$(CONFIG_CRYPTO_TWOFISH_586) += twofish-i586.o obj-$(CONFIG_CRYPTO_SERPENT_SSE2_586) += serpent-sse2-i586.o -obj-$(CONFIG_CRYPTO_AES_X86_64) += aes-x86_64.o obj-$(CONFIG_CRYPTO_DES3_EDE_X86_64) += des3_ede-x86_64.o obj-$(CONFIG_CRYPTO_CAMELLIA_X86_64) += camellia-x86_64.o obj-$(CONFIG_CRYPTO_BLOWFISH_X86_64) += blowfish-x86_64.o @@ -68,11 +66,9 @@ ifeq ($(avx2_supported),yes) obj-$(CONFIG_CRYPTO_MORUS1280_AVX2) += morus1280-avx2.o endif -aes-i586-y := aes-i586-asm_32.o aes_glue.o twofish-i586-y := twofish-i586-asm_32.o twofish_glue.o serpent-sse2-i586-y := serpent-sse2-i586-asm_32.o serpent_sse2_glue.o -aes-x86_64-y := aes-x86_64-asm_64.o aes_glue.o des3_ede-x86_64-y := des3_ede-asm_64.o des3_ede_glue.o camellia-x86_64-y := camellia-x86_64-asm_64.o camellia_glue.o blowfish-x86_64-y := blowfish-x86_64-asm_64.o blowfish_glue.o diff --git a/arch/x86/crypto/aes-i586-asm_32.S b/arch/x86/crypto/aes-i586-asm_32.S deleted file mode 100644 index 2849dbc59e11..000000000000 --- a/arch/x86/crypto/aes-i586-asm_32.S +++ /dev/null @@ -1,362 +0,0 @@ -// ------------------------------------------------------------------------- -// Copyright (c) 2001, Dr Brian Gladman < >, Worcester, UK. -// All rights reserved. -// -// LICENSE TERMS -// -// The free distribution and use of this software in both source and binary -// form is allowed (with or without changes) provided that: -// -// 1. distributions of this source code include the above copyright -// notice, this list of conditions and the following disclaimer// -// -// 2. distributions in binary form include the above copyright -// notice, this list of conditions and the following disclaimer -// in the documentation and/or other associated materials// -// -// 3. the copyright holder's name is not used to endorse products -// built using this software without specific written permission. -// -// -// ALTERNATIVELY, provided that this notice is retained in full, this product -// may be distributed under the terms of the GNU General Public License (GPL), -// in which case the provisions of the GPL apply INSTEAD OF those given above. -// -// Copyright (c) 2004 Linus Torvalds -// Copyright (c) 2004 Red Hat, Inc., James Morris - -// DISCLAIMER -// -// This software is provided 'as is' with no explicit or implied warranties -// in respect of its properties including, but not limited to, correctness -// and fitness for purpose. -// ------------------------------------------------------------------------- -// Issue Date: 29/07/2002 - -.file "aes-i586-asm.S" -.text - -#include -#include - -#define tlen 1024 // length of each of 4 'xor' arrays (256 32-bit words) - -/* offsets to parameters with one register pushed onto stack */ -#define ctx 8 -#define out_blk 12 -#define in_blk 16 - -/* offsets in crypto_aes_ctx structure */ -#define klen (480) -#define ekey (0) -#define dkey (240) - -// register mapping for encrypt and decrypt subroutines - -#define r0 eax -#define r1 ebx -#define r2 ecx -#define r3 edx -#define r4 esi -#define r5 edi - -#define eaxl al -#define eaxh ah -#define ebxl bl -#define ebxh bh -#define ecxl cl -#define ecxh ch -#define edxl dl -#define edxh dh - -#define _h(reg) reg##h -#define h(reg) _h(reg) - -#define _l(reg) reg##l -#define l(reg) _l(reg) - -// This macro takes a 32-bit word representing a column and uses -// each of its four bytes to index into four tables of 256 32-bit -// words to obtain values that are then xored into the appropriate -// output registers r0, r1, r4 or r5. - -// Parameters: -// table table base address -// %1 out_state[0] -// %2 out_state[1] -// %3 out_state[2] -// %4 out_state[3] -// idx input register for the round (destroyed) -// tmp scratch register for the round -// sched key schedule - -#define do_col(table, a1,a2,a3,a4, idx, tmp) \ - movzx %l(idx),%tmp; \ - xor table(,%tmp,4),%a1; \ - movzx %h(idx),%tmp; \ - shr $16,%idx; \ - xor table+tlen(,%tmp,4),%a2; \ - movzx %l(idx),%tmp; \ - movzx %h(idx),%idx; \ - xor table+2*tlen(,%tmp,4),%a3; \ - xor table+3*tlen(,%idx,4),%a4; - -// initialise output registers from the key schedule -// NB1: original value of a3 is in idx on exit -// NB2: original values of a1,a2,a4 aren't used -#define do_fcol(table, a1,a2,a3,a4, idx, tmp, sched) \ - mov 0 sched,%a1; \ - movzx %l(idx),%tmp; \ - mov 12 sched,%a2; \ - xor table(,%tmp,4),%a1; \ - mov 4 sched,%a4; \ - movzx %h(idx),%tmp; \ - shr $16,%idx; \ - xor table+tlen(,%tmp,4),%a2; \ - movzx %l(idx),%tmp; \ - movzx %h(idx),%idx; \ - xor table+3*tlen(,%idx,4),%a4; \ - mov %a3,%idx; \ - mov 8 sched,%a3; \ - xor table+2*tlen(,%tmp,4),%a3; - -// initialise output registers from the key schedule -// NB1: original value of a3 is in idx on exit -// NB2: original values of a1,a2,a4 aren't used -#define do_icol(table, a1,a2,a3,a4, idx, tmp, sched) \ - mov 0 sched,%a1; \ - movzx %l(idx),%tmp; \ - mov 4 sched,%a2; \ - xor table(,%tmp,4),%a1; \ - mov 12 sched,%a4; \ - movzx %h(idx),%tmp; \ - shr $16,%idx; \ - xor table+tlen(,%tmp,4),%a2; \ - movzx %l(idx),%tmp; \ - movzx %h(idx),%idx; \ - xor table+3*tlen(,%idx,4),%a4; \ - mov %a3,%idx; \ - mov 8 sched,%a3; \ - xor table+2*tlen(,%tmp,4),%a3; - - -// original Gladman had conditional saves to MMX regs. -#define save(a1, a2) \ - mov %a2,4*a1(%esp) - -#define restore(a1, a2) \ - mov 4*a2(%esp),%a1 - -// These macros perform a forward encryption cycle. They are entered with -// the first previous round column values in r0,r1,r4,r5 and -// exit with the final values in the same registers, using stack -// for temporary storage. - -// round column values -// on entry: r0,r1,r4,r5 -// on exit: r2,r1,r4,r5 -#define fwd_rnd1(arg, table) \ - save (0,r1); \ - save (1,r5); \ - \ - /* compute new column values */ \ - do_fcol(table, r2,r5,r4,r1, r0,r3, arg); /* idx=r0 */ \ - do_col (table, r4,r1,r2,r5, r0,r3); /* idx=r4 */ \ - restore(r0,0); \ - do_col (table, r1,r2,r5,r4, r0,r3); /* idx=r1 */ \ - restore(r0,1); \ - do_col (table, r5,r4,r1,r2, r0,r3); /* idx=r5 */ - -// round column values -// on entry: r2,r1,r4,r5 -// on exit: r0,r1,r4,r5 -#define fwd_rnd2(arg, table) \ - save (0,r1); \ - save (1,r5); \ - \ - /* compute new column values */ \ - do_fcol(table, r0,r5,r4,r1, r2,r3, arg); /* idx=r2 */ \ - do_col (table, r4,r1,r0,r5, r2,r3); /* idx=r4 */ \ - restore(r2,0); \ - do_col (table, r1,r0,r5,r4, r2,r3); /* idx=r1 */ \ - restore(r2,1); \ - do_col (table, r5,r4,r1,r0, r2,r3); /* idx=r5 */ - -// These macros performs an inverse encryption cycle. They are entered with -// the first previous round column values in r0,r1,r4,r5 and -// exit with the final values in the same registers, using stack -// for temporary storage - -// round column values -// on entry: r0,r1,r4,r5 -// on exit: r2,r1,r4,r5 -#define inv_rnd1(arg, table) \ - save (0,r1); \ - save (1,r5); \ - \ - /* compute new column values */ \ - do_icol(table, r2,r1,r4,r5, r0,r3, arg); /* idx=r0 */ \ - do_col (table, r4,r5,r2,r1, r0,r3); /* idx=r4 */ \ - restore(r0,0); \ - do_col (table, r1,r4,r5,r2, r0,r3); /* idx=r1 */ \ - restore(r0,1); \ - do_col (table, r5,r2,r1,r4, r0,r3); /* idx=r5 */ - -// round column values -// on entry: r2,r1,r4,r5 -// on exit: r0,r1,r4,r5 -#define inv_rnd2(arg, table) \ - save (0,r1); \ - save (1,r5); \ - \ - /* compute new column values */ \ - do_icol(table, r0,r1,r4,r5, r2,r3, arg); /* idx=r2 */ \ - do_col (table, r4,r5,r0,r1, r2,r3); /* idx=r4 */ \ - restore(r2,0); \ - do_col (table, r1,r4,r5,r0, r2,r3); /* idx=r1 */ \ - restore(r2,1); \ - do_col (table, r5,r0,r1,r4, r2,r3); /* idx=r5 */ - -// AES (Rijndael) Encryption Subroutine -/* void aes_enc_blk(struct crypto_aes_ctx *ctx, u8 *out_blk, const u8 *in_blk) */ - -.extern crypto_ft_tab -.extern crypto_fl_tab - -ENTRY(aes_enc_blk) - push %ebp - mov ctx(%esp),%ebp - -// CAUTION: the order and the values used in these assigns -// rely on the register mappings - -1: push %ebx - mov in_blk+4(%esp),%r2 - push %esi - mov klen(%ebp),%r3 // key size - push %edi -#if ekey != 0 - lea ekey(%ebp),%ebp // key pointer -#endif - -// input four columns and xor in first round key - - mov (%r2),%r0 - mov 4(%r2),%r1 - mov 8(%r2),%r4 - mov 12(%r2),%r5 - xor (%ebp),%r0 - xor 4(%ebp),%r1 - xor 8(%ebp),%r4 - xor 12(%ebp),%r5 - - sub $8,%esp // space for register saves on stack - add $16,%ebp // increment to next round key - cmp $24,%r3 - jb 4f // 10 rounds for 128-bit key - lea 32(%ebp),%ebp - je 3f // 12 rounds for 192-bit key - lea 32(%ebp),%ebp - -2: fwd_rnd1( -64(%ebp), crypto_ft_tab) // 14 rounds for 256-bit key - fwd_rnd2( -48(%ebp), crypto_ft_tab) -3: fwd_rnd1( -32(%ebp), crypto_ft_tab) // 12 rounds for 192-bit key - fwd_rnd2( -16(%ebp), crypto_ft_tab) -4: fwd_rnd1( (%ebp), crypto_ft_tab) // 10 rounds for 128-bit key - fwd_rnd2( +16(%ebp), crypto_ft_tab) - fwd_rnd1( +32(%ebp), crypto_ft_tab) - fwd_rnd2( +48(%ebp), crypto_ft_tab) - fwd_rnd1( +64(%ebp), crypto_ft_tab) - fwd_rnd2( +80(%ebp), crypto_ft_tab) - fwd_rnd1( +96(%ebp), crypto_ft_tab) - fwd_rnd2(+112(%ebp), crypto_ft_tab) - fwd_rnd1(+128(%ebp), crypto_ft_tab) - fwd_rnd2(+144(%ebp), crypto_fl_tab) // last round uses a different table - -// move final values to the output array. CAUTION: the -// order of these assigns rely on the register mappings - - add $8,%esp - mov out_blk+12(%esp),%ebp - mov %r5,12(%ebp) - pop %edi - mov %r4,8(%ebp) - pop %esi - mov %r1,4(%ebp) - pop %ebx - mov %r0,(%ebp) - pop %ebp - ret -ENDPROC(aes_enc_blk) - -// AES (Rijndael) Decryption Subroutine -/* void aes_dec_blk(struct crypto_aes_ctx *ctx, u8 *out_blk, const u8 *in_blk) */ - -.extern crypto_it_tab -.extern crypto_il_tab - -ENTRY(aes_dec_blk) - push %ebp - mov ctx(%esp),%ebp - -// CAUTION: the order and the values used in these assigns -// rely on the register mappings - -1: push %ebx - mov in_blk+4(%esp),%r2 - push %esi - mov klen(%ebp),%r3 // key size - push %edi -#if dkey != 0 - lea dkey(%ebp),%ebp // key pointer -#endif - -// input four columns and xor in first round key - - mov (%r2),%r0 - mov 4(%r2),%r1 - mov 8(%r2),%r4 - mov 12(%r2),%r5 - xor (%ebp),%r0 - xor 4(%ebp),%r1 - xor 8(%ebp),%r4 - xor 12(%ebp),%r5 - - sub $8,%esp // space for register saves on stack - add $16,%ebp // increment to next round key - cmp $24,%r3 - jb 4f // 10 rounds for 128-bit key - lea 32(%ebp),%ebp - je 3f // 12 rounds for 192-bit key - lea 32(%ebp),%ebp - -2: inv_rnd1( -64(%ebp), crypto_it_tab) // 14 rounds for 256-bit key - inv_rnd2( -48(%ebp), crypto_it_tab) -3: inv_rnd1( -32(%ebp), crypto_it_tab) // 12 rounds for 192-bit key - inv_rnd2( -16(%ebp), crypto_it_tab) -4: inv_rnd1( (%ebp), crypto_it_tab) // 10 rounds for 128-bit key - inv_rnd2( +16(%ebp), crypto_it_tab) - inv_rnd1( +32(%ebp), crypto_it_tab) - inv_rnd2( +48(%ebp), crypto_it_tab) - inv_rnd1( +64(%ebp), crypto_it_tab) - inv_rnd2( +80(%ebp), crypto_it_tab) - inv_rnd1( +96(%ebp), crypto_it_tab) - inv_rnd2(+112(%ebp), crypto_it_tab) - inv_rnd1(+128(%ebp), crypto_it_tab) - inv_rnd2(+144(%ebp), crypto_il_tab) // last round uses a different table - -// move final values to the output array. CAUTION: the -// order of these assigns rely on the register mappings - - add $8,%esp - mov out_blk+12(%esp),%ebp - mov %r5,12(%ebp) - pop %edi - mov %r4,8(%ebp) - pop %esi - mov %r1,4(%ebp) - pop %ebx - mov %r0,(%ebp) - pop %ebp - ret -ENDPROC(aes_dec_blk) diff --git a/arch/x86/crypto/aes-x86_64-asm_64.S b/arch/x86/crypto/aes-x86_64-asm_64.S deleted file mode 100644 index 8739cf7795de..000000000000 --- a/arch/x86/crypto/aes-x86_64-asm_64.S +++ /dev/null @@ -1,185 +0,0 @@ -/* AES (Rijndael) implementation (FIPS PUB 197) for x86_64 - * - * Copyright (C) 2005 Andreas Steinmetz, - * - * License: - * This code can be distributed under the terms of the GNU General Public - * License (GPL) Version 2 provided that the above header down to and - * including this sentence is retained in full. - */ - -.extern crypto_ft_tab -.extern crypto_it_tab -.extern crypto_fl_tab -.extern crypto_il_tab - -.text - -#include -#include - -#define R1 %rax -#define R1E %eax -#define R1X %ax -#define R1H %ah -#define R1L %al -#define R2 %rbx -#define R2E %ebx -#define R2X %bx -#define R2H %bh -#define R2L %bl -#define R3 %rcx -#define R3E %ecx -#define R3X %cx -#define R3H %ch -#define R3L %cl -#define R4 %rdx -#define R4E %edx -#define R4X %dx -#define R4H %dh -#define R4L %dl -#define R5 %rsi -#define R5E %esi -#define R6 %rdi -#define R6E %edi -#define R7 %r9 /* don't use %rbp; it breaks stack traces */ -#define R7E %r9d -#define R8 %r8 -#define R10 %r10 -#define R11 %r11 - -#define prologue(FUNC,KEY,B128,B192,r1,r2,r5,r6,r7,r8,r9,r10,r11) \ - ENTRY(FUNC); \ - movq r1,r2; \ - leaq KEY+48(r8),r9; \ - movq r10,r11; \ - movl (r7),r5 ## E; \ - movl 4(r7),r1 ## E; \ - movl 8(r7),r6 ## E; \ - movl 12(r7),r7 ## E; \ - movl 480(r8),r10 ## E; \ - xorl -48(r9),r5 ## E; \ - xorl -44(r9),r1 ## E; \ - xorl -40(r9),r6 ## E; \ - xorl -36(r9),r7 ## E; \ - cmpl $24,r10 ## E; \ - jb B128; \ - leaq 32(r9),r9; \ - je B192; \ - leaq 32(r9),r9; - -#define epilogue(FUNC,r1,r2,r5,r6,r7,r8,r9) \ - movq r1,r2; \ - movl r5 ## E,(r9); \ - movl r6 ## E,4(r9); \ - movl r7 ## E,8(r9); \ - movl r8 ## E,12(r9); \ - ret; \ - ENDPROC(FUNC); - -#define round(TAB,OFFSET,r1,r2,r3,r4,r5,r6,r7,r8,ra,rb,rc,rd) \ - movzbl r2 ## H,r5 ## E; \ - movzbl r2 ## L,r6 ## E; \ - movl TAB+1024(,r5,4),r5 ## E;\ - movw r4 ## X,r2 ## X; \ - movl TAB(,r6,4),r6 ## E; \ - roll $16,r2 ## E; \ - shrl $16,r4 ## E; \ - movzbl r4 ## L,r7 ## E; \ - movzbl r4 ## H,r4 ## E; \ - xorl OFFSET(r8),ra ## E; \ - xorl OFFSET+4(r8),rb ## E; \ - xorl TAB+3072(,r4,4),r5 ## E;\ - xorl TAB+2048(,r7,4),r6 ## E;\ - movzbl r1 ## L,r7 ## E; \ - movzbl r1 ## H,r4 ## E; \ - movl TAB+1024(,r4,4),r4 ## E;\ - movw r3 ## X,r1 ## X; \ - roll $16,r1 ## E; \ - shrl $16,r3 ## E; \ - xorl TAB(,r7,4),r5 ## E; \ - movzbl r3 ## L,r7 ## E; \ - movzbl r3 ## H,r3 ## E; \ - xorl TAB+3072(,r3,4),r4 ## E;\ - xorl TAB+2048(,r7,4),r5 ## E;\ - movzbl r1 ## L,r7 ## E; \ - movzbl r1 ## H,r3 ## E; \ - shrl $16,r1 ## E; \ - xorl TAB+3072(,r3,4),r6 ## E;\ - movl TAB+2048(,r7,4),r3 ## E;\ - movzbl r1 ## L,r7 ## E; \ - movzbl r1 ## H,r1 ## E; \ - xorl TAB+1024(,r1,4),r6 ## E;\ - xorl TAB(,r7,4),r3 ## E; \ - movzbl r2 ## H,r1 ## E; \ - movzbl r2 ## L,r7 ## E; \ - shrl $16,r2 ## E; \ - xorl TAB+3072(,r1,4),r3 ## E;\ - xorl TAB+2048(,r7,4),r4 ## E;\ - movzbl r2 ## H,r1 ## E; \ - movzbl r2 ## L,r2 ## E; \ - xorl OFFSET+8(r8),rc ## E; \ - xorl OFFSET+12(r8),rd ## E; \ - xorl TAB+1024(,r1,4),r3 ## E;\ - xorl TAB(,r2,4),r4 ## E; - -#define move_regs(r1,r2,r3,r4) \ - movl r3 ## E,r1 ## E; \ - movl r4 ## E,r2 ## E; - -#define entry(FUNC,KEY,B128,B192) \ - prologue(FUNC,KEY,B128,B192,R2,R8,R1,R3,R4,R6,R10,R5,R11) - -#define return(FUNC) epilogue(FUNC,R8,R2,R5,R6,R3,R4,R11) - -#define encrypt_round(TAB,OFFSET) \ - round(TAB,OFFSET,R1,R2,R3,R4,R5,R6,R7,R10,R5,R6,R3,R4) \ - move_regs(R1,R2,R5,R6) - -#define encrypt_final(TAB,OFFSET) \ - round(TAB,OFFSET,R1,R2,R3,R4,R5,R6,R7,R10,R5,R6,R3,R4) - -#define decrypt_round(TAB,OFFSET) \ - round(TAB,OFFSET,R2,R1,R4,R3,R6,R5,R7,R10,R5,R6,R3,R4) \ - move_regs(R1,R2,R5,R6) - -#define decrypt_final(TAB,OFFSET) \ - round(TAB,OFFSET,R2,R1,R4,R3,R6,R5,R7,R10,R5,R6,R3,R4) - -/* void aes_enc_blk(stuct crypto_tfm *tfm, u8 *out, const u8 *in) */ - - entry(aes_enc_blk,0,.Le128,.Le192) - encrypt_round(crypto_ft_tab,-96) - encrypt_round(crypto_ft_tab,-80) -.Le192: encrypt_round(crypto_ft_tab,-64) - encrypt_round(crypto_ft_tab,-48) -.Le128: encrypt_round(crypto_ft_tab,-32) - encrypt_round(crypto_ft_tab,-16) - encrypt_round(crypto_ft_tab, 0) - encrypt_round(crypto_ft_tab, 16) - encrypt_round(crypto_ft_tab, 32) - encrypt_round(crypto_ft_tab, 48) - encrypt_round(crypto_ft_tab, 64) - encrypt_round(crypto_ft_tab, 80) - encrypt_round(crypto_ft_tab, 96) - encrypt_final(crypto_fl_tab,112) - return(aes_enc_blk) - -/* void aes_dec_blk(struct crypto_tfm *tfm, u8 *out, const u8 *in) */ - - entry(aes_dec_blk,240,.Ld128,.Ld192) - decrypt_round(crypto_it_tab,-96) - decrypt_round(crypto_it_tab,-80) -.Ld192: decrypt_round(crypto_it_tab,-64) - decrypt_round(crypto_it_tab,-48) -.Ld128: decrypt_round(crypto_it_tab,-32) - decrypt_round(crypto_it_tab,-16) - decrypt_round(crypto_it_tab, 0) - decrypt_round(crypto_it_tab, 16) - decrypt_round(crypto_it_tab, 32) - decrypt_round(crypto_it_tab, 48) - decrypt_round(crypto_it_tab, 64) - decrypt_round(crypto_it_tab, 80) - decrypt_round(crypto_it_tab, 96) - decrypt_final(crypto_il_tab,112) - return(aes_dec_blk) diff --git a/arch/x86/crypto/aes_glue.c b/arch/x86/crypto/aes_glue.c deleted file mode 100644 index e26984f7ab8d..000000000000 --- a/arch/x86/crypto/aes_glue.c +++ /dev/null @@ -1,70 +0,0 @@ -/* - * Glue Code for the asm optimized version of the AES Cipher Algorithm - * - */ - -#include -#include -#include - -asmlinkage void aes_enc_blk(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in); -asmlinkage void aes_dec_blk(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in); - -void crypto_aes_encrypt_x86(struct crypto_aes_ctx *ctx, u8 *dst, const u8 *src) -{ - aes_enc_blk(ctx, dst, src); -} -EXPORT_SYMBOL_GPL(crypto_aes_encrypt_x86); - -void crypto_aes_decrypt_x86(struct crypto_aes_ctx *ctx, u8 *dst, const u8 *src) -{ - aes_dec_blk(ctx, dst, src); -} -EXPORT_SYMBOL_GPL(crypto_aes_decrypt_x86); - -static void aes_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) -{ - aes_enc_blk(crypto_tfm_ctx(tfm), dst, src); -} - -static void aes_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) -{ - aes_dec_blk(crypto_tfm_ctx(tfm), dst, src); -} - -static struct crypto_alg aes_alg = { - .cra_name = "aes", - .cra_driver_name = "aes-asm", - .cra_priority = 200, - .cra_flags = CRYPTO_ALG_TYPE_CIPHER, - .cra_blocksize = AES_BLOCK_SIZE, - .cra_ctxsize = sizeof(struct crypto_aes_ctx), - .cra_module = THIS_MODULE, - .cra_u = { - .cipher = { - .cia_min_keysize = AES_MIN_KEY_SIZE, - .cia_max_keysize = AES_MAX_KEY_SIZE, - .cia_setkey = crypto_aes_set_key, - .cia_encrypt = aes_encrypt, - .cia_decrypt = aes_decrypt - } - } -}; - -static int __init aes_init(void) -{ - return crypto_register_alg(&aes_alg); -} - -static void __exit aes_fini(void) -{ - crypto_unregister_alg(&aes_alg); -} - -module_init(aes_init); -module_exit(aes_fini); - -MODULE_DESCRIPTION("Rijndael (AES) Cipher Algorithm, asm optimized"); -MODULE_LICENSE("GPL"); -MODULE_ALIAS_CRYPTO("aes"); -MODULE_ALIAS_CRYPTO("aes-asm"); diff --git a/crypto/Kconfig b/crypto/Kconfig index 20af58068e6b..df6f0be66574 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -1108,50 +1108,6 @@ config CRYPTO_AES_TI block. Interrupts are also disabled to avoid races where cachelines are evicted when the CPU is interrupted to do something else. -config CRYPTO_AES_586 - tristate "AES cipher algorithms (i586)" - depends on (X86 || UML_X86) && !64BIT - select CRYPTO_ALGAPI - select CRYPTO_AES - help - AES cipher algorithms (FIPS-197). AES uses the Rijndael - algorithm. - - Rijndael appears to be consistently a very good performer in - both hardware and software across a wide range of computing - environments regardless of its use in feedback or non-feedback - modes. Its key setup time is excellent, and its key agility is - good. Rijndael's very low memory requirements make it very well - suited for restricted-space environments, in which it also - demonstrates excellent performance. Rijndael's operations are - among the easiest to defend against power and timing attacks. - - The AES specifies three key sizes: 128, 192 and 256 bits - - See for more information. - -config CRYPTO_AES_X86_64 - tristate "AES cipher algorithms (x86_64)" - depends on (X86 || UML_X86) && 64BIT - select CRYPTO_ALGAPI - select CRYPTO_AES - help - AES cipher algorithms (FIPS-197). AES uses the Rijndael - algorithm. - - Rijndael appears to be consistently a very good performer in - both hardware and software across a wide range of computing - environments regardless of its use in feedback or non-feedback - modes. Its key setup time is excellent, and its key agility is - good. Rijndael's very low memory requirements make it very well - suited for restricted-space environments, in which it also - demonstrates excellent performance. Rijndael's operations are - among the easiest to defend against power and timing attacks. - - The AES specifies three key sizes: 128, 192 and 256 bits - - See for more information. - config CRYPTO_AES_NI_INTEL tristate "AES cipher algorithms (AES-NI)" depends on X86 From patchwork Sat Jun 22 19:34:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167489 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2255890ilk; Sat, 22 Jun 2019 12:34:51 -0700 (PDT) X-Google-Smtp-Source: APXvYqzXESPo01xV5PIL203RSkAJFMtkC8DWI7b5I1/YXWxefuo8VJICagb2OaB5AGtkufV9R/hn X-Received: by 2002:a63:e151:: with SMTP id h17mr2085743pgk.307.1561232091505; Sat, 22 Jun 2019 12:34:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232091; cv=none; d=google.com; s=arc-20160816; b=Zx5yTTIA82ANfb066lPncvTAtzlzywq9B2j+D5apgDez5iMjP/Gps9O1OX7NOiU8c/ PTHQJKmr9PJPTNapsYnvcvZ44O4WmtqenO9D/k489sXIAhynJbaUx/4E1rEodpNjve00 /ZmQBR6kbdZ4OK+yRpza3K3sWKv+tp/TXra2owKEPQCP0xhNuPJo1xHaMYOwon6EneXT qkhrN+QyU5XZIpB4Ogjw/Z4P8hM5hIre8TrpJ0OGMr2rV0XYaw3Soc0IwkG1oeNNBD/D LvSQLGOSmxywZBIHkcFxlmotXFZSsOBAed/YjXeyVAHjJ+0NcfcHzKlLWBO4rL+uGHO/ 7Hvg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QMunKwfP+ecJeJsjLgotGX9cYxNfDTWVUzGiozUxkpA=; b=q5dZ0NdaUBfODnEOTLzeeOq0xvA81HXAd4ZcAR44HF0RKDiyZ5g7stz5b7bKR3z2O3 u7FGbd4AOxSWUjZPp0t48Ey6U+RxwiFUZc46wAVhT/5MTsQTInXDH6KehCeotU4i69vG mdtlr1Ot1gt3XpWTFmpv0Upy0K5NbdNLRtSy5eenqAvu4IHpMevKx+hnMUwf8/M0vo7j 7rd/kxll3O2Yb0kduFOZbN9muEeDB5DSW++0hzt8PQV1NqNv0GK/hQmbSBIKbmmS19kG 0xuzx+Ep0+CKbVzOt66sS/H7cfL88zKCuWYwaetkwL2wJKC1DAeTvEQ2tkpuL/xYGbre m7Ng== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=HRL4wYu2; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.34.51; Sat, 22 Jun 2019 12:34:51 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=HRL4wYu2; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726290AbfFVTeu (ORCPT + 3 others); Sat, 22 Jun 2019 15:34:50 -0400 Received: from mail-wm1-f67.google.com ([209.85.128.67]:53471 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726338AbfFVTeu (ORCPT ); Sat, 22 Jun 2019 15:34:50 -0400 Received: by mail-wm1-f67.google.com with SMTP id x15so9147968wmj.3 for ; Sat, 22 Jun 2019 12:34:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QMunKwfP+ecJeJsjLgotGX9cYxNfDTWVUzGiozUxkpA=; b=HRL4wYu2j3ZC8ztFcw5jusvcO70KmvcLgz/loyCoCvT4/QqUZ2KGzp4oEzGr6zSvQP z9uBHfwDsrks9peolBIaZ3LI7xvxLkZW8NK+vEhm5HXBiLD05vCph9smlQ1zLtrSmFLi GZNqkbrPsxSj1r1XFjIZe+0OyMC/eiPbT0DzXSwYLMFi+NcoB+arPCwqUhgaIIl3j/IY TY2Q17mbD/wxjM7wJfxfOx2+fC8gahXqf6TMdvlDjXmih9XU/jI427uc6o0TQjwuRZch wgy0iieM62PaOoZX++/M92IbZ53JtpBrwUBMeqB+bMygBWuRip3zZdJH/ZbQDgsnMt9H anYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QMunKwfP+ecJeJsjLgotGX9cYxNfDTWVUzGiozUxkpA=; b=lxXPpdAxpSPGjm8Il1VUqUTD9sHEBDmRgsPan42HnEeIwytr98TpQ907nP0TPuLlvW Ra46grIWj733jPv5HkBLwW/vUR+29Sbp7Mkmd+Y3wW/zrI/6hrmf6bKG25OY4zrcMLOA 1EZFmwY7H2MM0+LmnaoZjf59J1bEDtLiuS1QseAPNrEIkaTVzVBPxE6XpIgFdoW9pWXR 0X0THwk2yrqjFusjP86RkPygyzji9zpWFdSlh22w46I5J1UGX1kxGndsq9RC1MvY6Jxj 0iDp+APt0NVqePR7Y/x+9sfNPxyx4+oIBpfNPnIBNiptHiYrwnZdut6Fu6gPXT6tKN1S L3TA== X-Gm-Message-State: APjAAAUa1ZEP6dqHSqKQtuV0pme7AefJJ+WhWALZDR6XAkjWogOnGtqG ge0DKQM24p3A+wwXqgfOuBHOTCBzPxz4yMei X-Received: by 2002:a05:600c:28d:: with SMTP id 13mr8899720wmk.5.1561232087743; Sat, 22 Jun 2019 12:34:47 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:47 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 07/26] crypto: padlock/aes - switch to library version of key expansion routine Date: Sat, 22 Jun 2019 21:34:08 +0200 Message-Id: <20190622193427.20336-8-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Switch to the new AES library that also provides an implementation of the AES key expansion routine. This removes the dependency on the generic AES cipher, allowing it to be omitted entirely in the future. Signed-off-by: Ard Biesheuvel --- drivers/crypto/Kconfig | 2 +- drivers/crypto/padlock-aes.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- 2.20.1 diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig index 67af688d7d84..3fca5f7e38f0 100644 --- a/drivers/crypto/Kconfig +++ b/drivers/crypto/Kconfig @@ -26,7 +26,7 @@ config CRYPTO_DEV_PADLOCK_AES tristate "PadLock driver for AES algorithm" depends on CRYPTO_DEV_PADLOCK select CRYPTO_BLKCIPHER - select CRYPTO_AES + select CRYPTO_LIB_AES help Use VIA PadLock for AES algorithm. diff --git a/drivers/crypto/padlock-aes.c b/drivers/crypto/padlock-aes.c index 854539512c35..af90138eddb7 100644 --- a/drivers/crypto/padlock-aes.c +++ b/drivers/crypto/padlock-aes.c @@ -144,7 +144,7 @@ static int aes_set_key(struct crypto_tfm *tfm, const u8 *in_key, ctx->cword.encrypt.keygen = 1; ctx->cword.decrypt.keygen = 1; - if (crypto_aes_expand_key(&gen_aes, in_key, key_len)) { + if (aes_expandkey(&gen_aes, in_key, key_len)) { *flags |= CRYPTO_TFM_RES_BAD_KEY_LEN; return -EINVAL; } From patchwork Sat Jun 22 19:34:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167490 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2255896ilk; Sat, 22 Jun 2019 12:34:52 -0700 (PDT) X-Google-Smtp-Source: APXvYqxLRVh3ogsS82kZcYViu9Emsi5/FfCKzXFdKWN57DlyvXdO2zqsxw3P1mR9ud0OngsqEr+j X-Received: by 2002:a17:90a:8a8e:: with SMTP id x14mr14225411pjn.103.1561232092081; Sat, 22 Jun 2019 12:34:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232092; cv=none; d=google.com; s=arc-20160816; b=cngyaBkr8t1EveXbN7ryWJs0NoC9j6frYt/ErH9JlaKdxbYaL/vmauBkqaJEi9xAE2 zX7YLfATpUqTx+Hf7m8by2dnaPPUkBHI/v5gHR/sVxC5Adyq4UQi4svMMIyHwqbyk/e4 gJT0ykpbX/7S78j6q6Hvl9pp4AY3PWxkrn6pEj9h6xPVu00BY7JbawJC1oNZ9UwxMwY8 WvMymYO2bJFdy74o/cWwYs91bjWtDSfjIjzY0+pwjcwRmA97rzDaqMHyn0yJ7o3LkcMD XEzFFrAtnJ67TvelT6i4WOAnn1bt7KUN+fTEfWigx/p/nBdtSLA9pUS5s5M3qZdjxuwh c4mg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=lQLA1Z6u9tD3If/MHOVikbtefg1FLPAfk8jJsqm/d/c=; b=nl3DxnEbqPBMiXc6v+JOJLEsM6G24rN+9CQlayPQoeNpr6JXULHxV/rugFrw0xwhFV 57HOH9TYuZBQiijAOwiAt+RCIQEktwXdEedFh3Z2pd9TTwG0zSys8lNW5lB5PoQrs69z 6lP3gLJ0sfVR0os8RPMbFVoZYO5Uw9SO9LsI6OYbRCfnt3IkbRSUSOcfxIkFZugm/35o JEbKAH9XwoUNEjcEwN+n2RP27r6O/26Cu+79zYsiEY756hp8tHYJAzoi25GyFrdltBo8 NuaaJtWWktiNhJFfVsVIZEMCtYgqAFEjzlpVlOVdeZjkkVi8k3rTcTan3m8ikJuQ0jyl QTrg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=MYRAFNet; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.34.51; Sat, 22 Jun 2019 12:34:52 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=MYRAFNet; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726351AbfFVTev (ORCPT + 3 others); Sat, 22 Jun 2019 15:34:51 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:37423 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726343AbfFVTev (ORCPT ); Sat, 22 Jun 2019 15:34:51 -0400 Received: by mail-wm1-f66.google.com with SMTP id f17so9609732wme.2 for ; Sat, 22 Jun 2019 12:34:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lQLA1Z6u9tD3If/MHOVikbtefg1FLPAfk8jJsqm/d/c=; b=MYRAFNetilQRuZslBml8aKhDISEudeoPejyATZfvssgTxaDE/R1CKYQt6EVyk0b4jg Rh6Xxk8espv5gC83N6yobsZ2u3+e/bPXfbY4vgjDK6pbyepwv3NufLqIqo1DpVtGjxG5 LuZVwxF3ZFCsWDOBrAVGL9pKcj+k2Dit3xkITDL9AeB+qiv8h4pS+X9eQNtOpTs5EI9p PuB5hVD755KqNcIbZfbHTAr4G78BS/qFDiQrPDx9xZ6mTRJ3paRktbKbAk8DLDCdexzk 02XLXUhiUXeu/M51zGgMPXL1ofFPl2LQqy7JrkCSDoO3I9fldGnGLo+jgeOjn4qLPMEa jt3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lQLA1Z6u9tD3If/MHOVikbtefg1FLPAfk8jJsqm/d/c=; b=pbZlTIcqXQF4OEDdZtJ7whRWfzA06YxQEXEvbeqszo6sfFBVHOupAoo8iLPaMRHZu/ LcoSoO6bJMDG0A99mSVNQklktkFk87T9Xswux1T4AQWQYq2kE/4y1a3DfiFrJYZjGCaj U8VRRcZp1gEkU/aWVCHWxFXqMFTWzZdexL3PyV2DfI9jeAJ5CwIJ5r/SLeanQ9DvSigy tSou3CpdOrry+u0Ifsu4wUqpJtRK2MsZa7ky2+Mr2OgbtL5hAV87vksP9kInvQok5tCV 498s3SdN+U1sX9YNU/sUYrvXnRf5BOqwPYYVcYaZmyPVCKWw8Ycma4O4/HtL1YmYfY1T pqzA== X-Gm-Message-State: APjAAAV2r0v2GZU4wVyz4IxcdSJS8Nwjq/jlSI9FBLjBYLuqb//0GIz6 C8xYszOh7K9Lgdg7ji5+GaMcdWeoHFq/3KBB X-Received: by 2002:a1c:80c1:: with SMTP id b184mr8037126wmd.24.1561232088802; Sat, 22 Jun 2019 12:34:48 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:48 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 08/26] crypto: cesa/aes - switch to library version of key expansion routine Date: Sat, 22 Jun 2019 21:34:09 +0200 Message-Id: <20190622193427.20336-9-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Switch to the new AES library that also provides an implementation of the AES key expansion routine. This removes the dependency on the generic AES cipher, allowing it to be omitted entirely in the future. Signed-off-by: Ard Biesheuvel --- drivers/crypto/Kconfig | 2 +- drivers/crypto/marvell/cipher.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- 2.20.1 diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig index 3fca5f7e38f0..fdccadc94819 100644 --- a/drivers/crypto/Kconfig +++ b/drivers/crypto/Kconfig @@ -213,7 +213,7 @@ config CRYPTO_CRC32_S390 config CRYPTO_DEV_MARVELL_CESA tristate "Marvell's Cryptographic Engine driver" depends on PLAT_ORION || ARCH_MVEBU - select CRYPTO_AES + select CRYPTO_LIB_AES select CRYPTO_DES select CRYPTO_BLKCIPHER select CRYPTO_HASH diff --git a/drivers/crypto/marvell/cipher.c b/drivers/crypto/marvell/cipher.c index 2fd936b19c6d..debe7d9f00ae 100644 --- a/drivers/crypto/marvell/cipher.c +++ b/drivers/crypto/marvell/cipher.c @@ -257,7 +257,7 @@ static int mv_cesa_aes_setkey(struct crypto_skcipher *cipher, const u8 *key, int ret; int i; - ret = crypto_aes_expand_key(&ctx->aes, key, len); + ret = aes_expandkey(&ctx->aes, key, len); if (ret) { crypto_skcipher_set_flags(cipher, CRYPTO_TFM_RES_BAD_KEY_LEN); return ret; From patchwork Sat Jun 22 19:34:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167493 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2255908ilk; Sat, 22 Jun 2019 12:34:53 -0700 (PDT) X-Google-Smtp-Source: APXvYqwmuCQ6uBU85fPH7qaL7U/bXtU8YF8J4yvXwIDyBMWPcol/W/317apz6iS94LBftBNUbquR X-Received: by 2002:a63:730c:: with SMTP id o12mr999219pgc.18.1561232093659; Sat, 22 Jun 2019 12:34:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232093; cv=none; d=google.com; s=arc-20160816; b=V4nz1Ty1/oWF79p4zA5hJxMhKH1ZhgwRS2b+GZpjDgZk6CymT8XSfXXsHeZ6vg2Njk LSGH2Tbm5sViN6Gsup+9wYTIUmbxxdMzvb+5VOGj7ts99ENg9Tz9KdhWWKbfEp3QFWnX 6FODa5fBPpG9FFzKlb/JisluI5Lqga/xqK+lBFNgIYbnO/1mkzG23MGhMBYTLYthxB8J tBiBpp/h1sHrHPoy1O4ww5CI2Qrwd9M95R/dzYlZS4e5sJ0ELSFemR960K8lAHEstS51 ErI/wTg6LTIWf8NGzSlStl0wniB32+xiuCrLEmdZJs6F+gAPdkQ9t7POETgCRwFxnq6E /iaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=TEkyEzYA9z24dLcVZzAphhOs6uMPB3TRScCj5mulxhM=; b=kls4cF02ISCNh9JRBLboPlb7blLpm5rvJZRDehuf9HJvXgmWrrCSRA5B/HOuNgM/VT 2N1TRwGTQG225/4oTw4gHEaqnzYgNa6RCck5EWn2er6QMyIoodkMYxW7fqUF5vl+c85a omI4ASl6H9ELDRwE7ktgI5uq//q1wEDSiSjjM5MsB9Ou1h13Ckgm9999z234E7XeGcDT mlcmlimToJ+K4OpNTivBUTT1Y2mC0Zoxw2KiQzuYNZ3Nd4PL8oDTpFHroG/LfsqhAfwQ dFzIX4dxeMVbjfu6LJnHZXME62QVTwdY/1CgRVDUAbBnXHDUUwVzeHi08xuBe5+8V0RW oWhw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=MGwPA4QG; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.34.53; Sat, 22 Jun 2019 12:34:53 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=MGwPA4QG; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726353AbfFVTew (ORCPT + 3 others); Sat, 22 Jun 2019 15:34:52 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:33742 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726338AbfFVTew (ORCPT ); Sat, 22 Jun 2019 15:34:52 -0400 Received: by mail-wm1-f65.google.com with SMTP id h19so11351678wme.0 for ; Sat, 22 Jun 2019 12:34:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TEkyEzYA9z24dLcVZzAphhOs6uMPB3TRScCj5mulxhM=; b=MGwPA4QG2l1a7CsZmkd46vcd1qlG3ozWuiFkR6iVpRp2qCFSNs2q8yctk/eCr3booL baHALEcUZPE4O6OG5Xh8T0qsI2UWIx1UmBnUK//JNF273U1Aeml5h1OYZOSWkV4+2tT+ qCNzT9fG3QRVHaRmLoSJeezaM70E5+l4TH9pVU2op8s3fYmrq+jpEa+sxhZnE6fdUDeM Nk8+sc/2gZ+tr1Sev0vKS3aQ+WcnoJpP64hzODQldUi46qAybqDoc12KDeSx+hzkjT/n O+R//nIitNcWHldcKhVdP8mPwpPjA1zpkK4JKPBg2aJrFuRkYzJNaT6vxOdLHcP+aF7U u9eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TEkyEzYA9z24dLcVZzAphhOs6uMPB3TRScCj5mulxhM=; b=eFoUEni2Grbk4tPSjFBXKHCPYQ6Ftevz/7kBuOYC1Z1zB1L3qFRqp08N/6BYdXMzpn iALtG37KEW7dnLKkZslDAjTndfePOyMonpHE+H0d5mkHik+fiZigRMx7soIqZJdCa+bU P5czwFLk5xABOf+d8h30DNHrii1CyN4SJ+v7ZfLuwz9PJAJgIoV3QmjAqTd+eGOgb0ZH obJClDIU08+WswbviZur4v0P3q9yGuaNDYWWJgNcmLy0gRcq0wWyA5hIuLORMwJyQZxk Ek8IcUYbVZQBiBm67f1DSFtxOzSQIQa+42i09t+1+kcHK/5iw6xIjmUtA6H1wncevnc1 g8cA== X-Gm-Message-State: APjAAAXRvks+1hUrR24lDXpDB5sNKiMXpPlRUrTrqJ6OaA6hsHEY1fTF 11LekU7If7pQJAbDW7bSFlwNwQc134xMEMCy X-Received: by 2002:a1c:7e90:: with SMTP id z138mr8384504wmc.128.1561232089800; Sat, 22 Jun 2019 12:34:49 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:49 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 09/26] crypto: safexcel/aes - switch to library version of key expansion routine Date: Sat, 22 Jun 2019 21:34:10 +0200 Message-Id: <20190622193427.20336-10-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Switch to the new AES library that also provides an implementation of the AES key expansion routine. This removes the dependency on the generic AES cipher, allowing it to be omitted entirely in the future. Signed-off-by: Ard Biesheuvel --- drivers/crypto/Kconfig | 2 +- drivers/crypto/inside-secure/safexcel_cipher.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- 2.20.1 diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig index fdccadc94819..b30b84089d11 100644 --- a/drivers/crypto/Kconfig +++ b/drivers/crypto/Kconfig @@ -718,7 +718,7 @@ config CRYPTO_DEV_SAFEXCEL tristate "Inside Secure's SafeXcel cryptographic engine driver" depends on OF depends on (ARM64 && ARCH_MVEBU) || (COMPILE_TEST && 64BIT) - select CRYPTO_AES + select CRYPTO_LIB_AES select CRYPTO_AUTHENC select CRYPTO_BLKCIPHER select CRYPTO_DES diff --git a/drivers/crypto/inside-secure/safexcel_cipher.c b/drivers/crypto/inside-secure/safexcel_cipher.c index 8cdbdbe35681..19ec086dce4f 100644 --- a/drivers/crypto/inside-secure/safexcel_cipher.c +++ b/drivers/crypto/inside-secure/safexcel_cipher.c @@ -178,7 +178,7 @@ static int safexcel_skcipher_aes_setkey(struct crypto_skcipher *ctfm, struct crypto_aes_ctx aes; int ret, i; - ret = crypto_aes_expand_key(&aes, key, len); + ret = aes_expandkey(&aes, key, len); if (ret) { crypto_skcipher_set_flags(ctfm, CRYPTO_TFM_RES_BAD_KEY_LEN); return ret; From patchwork Sat Jun 22 19:34:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167495 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2255929ilk; Sat, 22 Jun 2019 12:34:55 -0700 (PDT) X-Google-Smtp-Source: APXvYqx0lEyd6NQnMiCTjgZXxfdMNku9BTIhaQpAJKFouxmhaKqNz9zwzsKjk9CD90XBssXkUThA X-Received: by 2002:a63:5b58:: with SMTP id l24mr24478377pgm.303.1561232095570; Sat, 22 Jun 2019 12:34:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232095; cv=none; d=google.com; s=arc-20160816; b=mf+JmobDMoZ9Gh4xw3/eHH8h70sU7BicxH4X2X71GWJFL05UeGRPrW34l68hg3fCT6 XBhULABuF1dIpJN5bhifFrfn+q7cRL+t08Kv7YQ1tiQ06WUt8eN59fqdPIe9SUpIxUR1 a6mxymzRYWvQ+kl8iNC8HJyXpyzBEVzXUv97cLXL7y3m1Tn0wVfu4u5y6luNNFEfY06u INPA0uZDDnl365k9Y+BDM+Yg6M2ncuf4oXwdkKI+Y/zrcodkFeOMQMuR4B6/ej1JQHr6 7E5sgne2rCJYS5Enlg+Ys0ddbNtC/C3w5ge5+2XbwfBheh05vTSTWgGBD7n7lEIXnYCd i0IQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=8rzDm2pxsxDof2+7Zzrqa2tkF2eXq+Yv+cQAP5RFetY=; b=pbIaldns+QZu6LlTXuFGBOd/j97LedR1Of7YfuZH6wPFgJ5zHC2oL8uVS1AGRCbV7Y Zia4Q8fXGRKzv79AOp5xO5abhqhIv2BGcGW2CvmOYZ9hJKKJw3/QROlNuA89Q8qtrtQz jtZ0WjIYSKMSvBJSZJ0/ww3ufhAfkm/7KOy3GzEv4TnWQxe+bfWuqYPmEKzxC/B7HYA9 ZcgmWLeojEDq6QNW+OznxcT1vN8KjVGH7irCiTzbEcSIRWb4RmnpatXy5E7LrfSNSdSr 4AYHh81wt0Xx5Lnmp8Kn91tK9/wdqTi6UBW3kxwjWYpG/kjsuQW/YKWhMwE7WFhYpJsJ c4MQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=kBHPINNO; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.34.55; Sat, 22 Jun 2019 12:34:55 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=kBHPINNO; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726338AbfFVTey (ORCPT + 3 others); Sat, 22 Jun 2019 15:34:54 -0400 Received: from mail-wr1-f67.google.com ([209.85.221.67]:40961 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726286AbfFVTey (ORCPT ); Sat, 22 Jun 2019 15:34:54 -0400 Received: by mail-wr1-f67.google.com with SMTP id c2so9704617wrm.8 for ; Sat, 22 Jun 2019 12:34:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8rzDm2pxsxDof2+7Zzrqa2tkF2eXq+Yv+cQAP5RFetY=; b=kBHPINNOkXhgNeZrO083oUFhDSW9AT0ZnejWBybqSGkBaGJ+gfR2PxCRH3oOidz5Dl 4UUnIK5Udm5KrB/Bueydeice4U/6dBOsBMvNR/XllDrWId3+CFccntD1Coj4OQyt7EEg 2qN5FGMIyIaqUI4OuwMbOE2fRpraXZTrl2kWHemj5WHH9gletc4JTrJyPcHXh5mNje9P edGZF9YeJjeLotEiJEvy2/x0yUTKIxWODBMa5G0rwTilrlro3zo3lKWv0p0XqIko4oyn dkGAGyYWVzHKCCEsk48D5UuVKMAG0nwqwTb7+sPzydZdKTdxmitCHybhgHplIMAjh98o 65fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8rzDm2pxsxDof2+7Zzrqa2tkF2eXq+Yv+cQAP5RFetY=; b=MG6+VRdAIyJCSAcS8mPbOlE3mKic4DTq1cVASAayeVHNv+5z+XRJCiA7308adVU2Ke Tfs1qeoHBCjbTHEPEntItqG+TfOOvB6N5rfPSVNCpw3yrRknEQg+DzZVH+26aW76jo4y MXA22hooq2ejJymFIeea05yEZBBoPBYu5oLIyEUpG1D2lWzotmOnF+mPcdr8muurG/nQ P0LSsZPyO+aGR+BDDPDoTmq4A69sbfeXt3N+ZOnk763d1tYKlp+xelJOjbxl7tXKcjd9 zjzhpayGptJrWNfDQ/RPzOI/c+cyF1/W1ZP2WA9fMMwQ8eeO45F26tjOCBBrb8ULJCNM WoAA== X-Gm-Message-State: APjAAAVUNgAlzJMuFuvAfWqdGljUdCT2xcRcoXlmz+ch5hsCKcXEHzEx JomADrTEMXha2g0UfpE6fBpNk/4x6c8wMNbs X-Received: by 2002:a5d:4949:: with SMTP id r9mr41692492wrs.289.1561232090966; Sat, 22 Jun 2019 12:34:50 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:50 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 10/26] crypto: arm64/ghash - switch to AES library Date: Sat, 22 Jun 2019 21:34:11 +0200 Message-Id: <20190622193427.20336-11-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The GHASH code uses the generic AES key expansion routines, and calls directly into the scalar table based AES cipher for arm64 from the fallback path, and since this implementation is known to be non-time invariant, doing so from a time invariant SIMD cipher is a bit nasty. So let's switch to the AES library - this makes the code more robust, and drops the dependency on the generic AES cipher, allowing us to omit it entirely in the future. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/Kconfig | 3 +- arch/arm64/crypto/ghash-ce-glue.c | 30 +++++++------------- 2 files changed, 11 insertions(+), 22 deletions(-) -- 2.20.1 diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index d9a523ecdd83..1762055e7093 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -58,8 +58,7 @@ config CRYPTO_GHASH_ARM64_CE depends on KERNEL_MODE_NEON select CRYPTO_HASH select CRYPTO_GF128MUL - select CRYPTO_AES - select CRYPTO_AES_ARM64 + select CRYPTO_LIB_AES config CRYPTO_CRCT10DIF_ARM64_CE tristate "CRCT10DIF digest algorithm using PMULL instructions" diff --git a/arch/arm64/crypto/ghash-ce-glue.c b/arch/arm64/crypto/ghash-ce-glue.c index b39ed99b06fb..90496765d22f 100644 --- a/arch/arm64/crypto/ghash-ce-glue.c +++ b/arch/arm64/crypto/ghash-ce-glue.c @@ -73,8 +73,6 @@ asmlinkage void pmull_gcm_decrypt(int blocks, u64 dg[], u8 dst[], asmlinkage void pmull_gcm_encrypt_block(u8 dst[], u8 const src[], u32 const rk[], int rounds); -asmlinkage void __aes_arm64_encrypt(u32 *rk, u8 *out, const u8 *in, int rounds); - static int ghash_init(struct shash_desc *desc) { struct ghash_desc_ctx *ctx = shash_desc_ctx(desc); @@ -312,14 +310,13 @@ static int gcm_setkey(struct crypto_aead *tfm, const u8 *inkey, u8 key[GHASH_BLOCK_SIZE]; int ret; - ret = crypto_aes_expand_key(&ctx->aes_key, inkey, keylen); + ret = aes_expandkey(&ctx->aes_key, inkey, keylen); if (ret) { tfm->base.crt_flags |= CRYPTO_TFM_RES_BAD_KEY_LEN; return -EINVAL; } - __aes_arm64_encrypt(ctx->aes_key.key_enc, key, (u8[AES_BLOCK_SIZE]){}, - num_rounds(&ctx->aes_key)); + aes_encrypt(&ctx->aes_key, key, (u8[AES_BLOCK_SIZE]){}); return __ghash_setkey(&ctx->ghash_key, key, sizeof(be128)); } @@ -470,7 +467,7 @@ static int gcm_encrypt(struct aead_request *req) rk = ctx->aes_key.key_enc; } while (walk.nbytes >= 2 * AES_BLOCK_SIZE); } else { - __aes_arm64_encrypt(ctx->aes_key.key_enc, tag, iv, nrounds); + aes_encrypt(&ctx->aes_key, tag, iv); put_unaligned_be32(2, iv + GCM_IV_SIZE); while (walk.nbytes >= (2 * AES_BLOCK_SIZE)) { @@ -481,8 +478,7 @@ static int gcm_encrypt(struct aead_request *req) int remaining = blocks; do { - __aes_arm64_encrypt(ctx->aes_key.key_enc, - ks, iv, nrounds); + aes_encrypt(&ctx->aes_key, ks, iv); crypto_xor_cpy(dst, src, ks, AES_BLOCK_SIZE); crypto_inc(iv, AES_BLOCK_SIZE); @@ -498,13 +494,10 @@ static int gcm_encrypt(struct aead_request *req) walk.nbytes % (2 * AES_BLOCK_SIZE)); } if (walk.nbytes) { - __aes_arm64_encrypt(ctx->aes_key.key_enc, ks, iv, - nrounds); + aes_encrypt(&ctx->aes_key, ks, iv); if (walk.nbytes > AES_BLOCK_SIZE) { crypto_inc(iv, AES_BLOCK_SIZE); - __aes_arm64_encrypt(ctx->aes_key.key_enc, - ks + AES_BLOCK_SIZE, iv, - nrounds); + aes_encrypt(&ctx->aes_key, ks + AES_BLOCK_SIZE, iv); } } } @@ -608,7 +601,7 @@ static int gcm_decrypt(struct aead_request *req) rk = ctx->aes_key.key_enc; } while (walk.nbytes >= 2 * AES_BLOCK_SIZE); } else { - __aes_arm64_encrypt(ctx->aes_key.key_enc, tag, iv, nrounds); + aes_encrypt(&ctx->aes_key, tag, iv); put_unaligned_be32(2, iv + GCM_IV_SIZE); while (walk.nbytes >= (2 * AES_BLOCK_SIZE)) { @@ -621,8 +614,7 @@ static int gcm_decrypt(struct aead_request *req) pmull_ghash_update_p64); do { - __aes_arm64_encrypt(ctx->aes_key.key_enc, - buf, iv, nrounds); + aes_encrypt(&ctx->aes_key, buf, iv); crypto_xor_cpy(dst, src, buf, AES_BLOCK_SIZE); crypto_inc(iv, AES_BLOCK_SIZE); @@ -640,11 +632,9 @@ static int gcm_decrypt(struct aead_request *req) memcpy(iv2, iv, AES_BLOCK_SIZE); crypto_inc(iv2, AES_BLOCK_SIZE); - __aes_arm64_encrypt(ctx->aes_key.key_enc, iv2, - iv2, nrounds); + aes_encrypt(&ctx->aes_key, iv2, iv2); } - __aes_arm64_encrypt(ctx->aes_key.key_enc, iv, iv, - nrounds); + aes_encrypt(&ctx->aes_key, iv, iv); } } From patchwork Sat Jun 22 19:34:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167494 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2255921ilk; Sat, 22 Jun 2019 12:34:55 -0700 (PDT) X-Google-Smtp-Source: APXvYqwQzR3hu7tNCrMWavNFg8X1cDcXIm56B68syKliN99TvjdwQHIYROU5tEQveZmgrs5gf/a5 X-Received: by 2002:a63:9e53:: with SMTP id r19mr15900255pgo.442.1561232095217; Sat, 22 Jun 2019 12:34:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232095; cv=none; d=google.com; s=arc-20160816; b=oPIHN9UZmZAI2DHKNZpOcG8fRgsIDIGeN3v5deqdb0kZvqmM4De9hJyMcrzLa14vi8 PStt6I5ThOzjxFpyBn4GsD//A6vCySMyXlNyBVU2Jo4f2xwDsv1cJxO4UMIWaYt+oC6w iQ++rRihFwDRnL1NTSKiK7QC3MCmeHjA01uLapTgpRdTtZcc5/oZvAzEdiXJ5fDLMfo5 12FiuOwV3+4b0FpPZ3Xi9zbmElL2qnF045uPNc6jz5aQ0u7RTMbBYyxRYwMbX8Jh2MwQ KEb9/CFoKgh44KYZptRxAIqr++polrWoEZC6HptERj3pChcmZ8SAGqK+6Tvg5QDedvJg YShA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QXrlKaueW49CmOlW9q/RYTZvkffZ+6o0SuGgZq9akzA=; b=tV2C6yOmT2zslmf8cUEzM/Bsng0A6Ez8yDRn5Cuv75sD53duh8NjIrsNxkR+e5gt9R rIG30v+LDx1PhIlrOdnbChkjvsvHBpnGXu8lb4daljhGMtBaPi9i7/JawYbGTHDkW5yo /NtQbZm9otA5XYakGUVfE+RXG09K14sAj/U3IQzc2N4XMD7TnKuCyp/63XBMPoYF1GuM 730XIXlFXWMPVDqMVNBxgSAlSS4Z9HQxFXWT327XCwBUdzdAHxuvllsk9haqOE49ifgS RerrPGjZ3z78xCmiYbxxdIyVHPdE4KuHmc8KTK1ZLGcYt5t3Wp6AGXhw7QQj+3ndW6Ds WXWA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=se4WvrAk; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.34.55; Sat, 22 Jun 2019 12:34:55 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=se4WvrAk; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726362AbfFVTey (ORCPT + 3 others); Sat, 22 Jun 2019 15:34:54 -0400 Received: from mail-wm1-f68.google.com ([209.85.128.68]:34664 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726338AbfFVTex (ORCPT ); Sat, 22 Jun 2019 15:34:53 -0400 Received: by mail-wm1-f68.google.com with SMTP id w9so11348958wmd.1 for ; Sat, 22 Jun 2019 12:34:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QXrlKaueW49CmOlW9q/RYTZvkffZ+6o0SuGgZq9akzA=; b=se4WvrAkQddJuEKOWJWEobWEFH4GFHDCCINWdq/2PGUoMXh7mI+J7ZhN6Z1xR/MSC6 lIzGId01l5ijUmJkVoaymPFWv/NkcA2M8GuDraBWeZP3Laz0m9VHtrhLSwklTQ9whIve 71KGUDOgD41lfRpd/0LrfPgZgvBSrr3TLJmdg3NpJs6vKdhjBU/6JVSvId7ofZnIzzOQ he7Jas4GTIrKORP5DBcfd4oW2LO788MKI2UkHkavRJMiFCsuSFLlXSqnAiICfkKweOci LrVeefvVndHELfMu4XnzPfcMPp135V20aTAWwjjGm5crcl4MVme6kH6eR2OzLytGD+w7 STSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QXrlKaueW49CmOlW9q/RYTZvkffZ+6o0SuGgZq9akzA=; b=mXwkWJ3hXQ5GbsXjHL9JdJ0K0LIK4XvjZYxVTk5DpjiFbT3sHpwshfIqk/3+UGYShg zdxbpfNFT25a1wsYC8AN8tNRjz0STSf89lYBLi6tiPChIOw1wXjvtGAYaufXDxIZW4hE zxHmnig8msmbFfvT2jKKKxBlxQVY3zapiFaAYKYpW0Me24SEZlK64u2+oUJ3qi3mPJff BxZB6qfrGq489UlteyvZx6SLa5KgVub0f+6nrKLsTbyU8xA3flvUHWBp8THwbK+ys09+ V1BaEbyXC07QX4NpGl83bxQpNCSg8am200VZ487R48m3Zk45as+n7G/PmwHy44nJ/PL6 YrcA== X-Gm-Message-State: APjAAAUzxCTHIC39W/npTfZeWblUzfklPhs9Kjx3dlpjKJAbOVrR1FTu 6bCj8w92mfCtvNFrfJxfi3UOkwfm/xfJU/oL X-Received: by 2002:a7b:c766:: with SMTP id x6mr9077532wmk.40.1561232091975; Sat, 22 Jun 2019 12:34:51 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:51 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 11/26] crypto: arm/aes-neonbs - switch to library version of key expansion routine Date: Sat, 22 Jun 2019 21:34:12 +0200 Message-Id: <20190622193427.20336-12-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Switch to the new AES library that also provides an implementation of the AES key expansion routine. This removes the dependency on the generic AES cipher, allowing it to be omitted entirely in the future. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/Kconfig | 2 +- arch/arm/crypto/aes-neonbs-glue.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) -- 2.20.1 diff --git a/arch/arm/crypto/Kconfig b/arch/arm/crypto/Kconfig index a95322b59799..b24df84a1d7a 100644 --- a/arch/arm/crypto/Kconfig +++ b/arch/arm/crypto/Kconfig @@ -82,8 +82,8 @@ config CRYPTO_AES_ARM_BS tristate "Bit sliced AES using NEON instructions" depends on KERNEL_MODE_NEON select CRYPTO_BLKCIPHER + select CRYPTO_LIB_AES select CRYPTO_SIMD - select CRYPTO_AES help Use a faster and more secure NEON based implementation of AES in CBC, CTR and XTS modes diff --git a/arch/arm/crypto/aes-neonbs-glue.c b/arch/arm/crypto/aes-neonbs-glue.c index 617c2c99ebfb..f43c9365b6a9 100644 --- a/arch/arm/crypto/aes-neonbs-glue.c +++ b/arch/arm/crypto/aes-neonbs-glue.c @@ -64,7 +64,7 @@ static int aesbs_setkey(struct crypto_skcipher *tfm, const u8 *in_key, struct crypto_aes_ctx rk; int err; - err = crypto_aes_expand_key(&rk, in_key, key_len); + err = aes_expandkey(&rk, in_key, key_len); if (err) return err; @@ -123,7 +123,7 @@ static int aesbs_cbc_setkey(struct crypto_skcipher *tfm, const u8 *in_key, struct crypto_aes_ctx rk; int err; - err = crypto_aes_expand_key(&rk, in_key, key_len); + err = aes_expandkey(&rk, in_key, key_len); if (err) return err; From patchwork Sat Jun 22 19:34:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167496 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2255944ilk; Sat, 22 Jun 2019 12:34:56 -0700 (PDT) X-Google-Smtp-Source: APXvYqxukQtyCMag14RP8BPIxqRWMM+ijApp5JK6xvaMDjHz11ldpKuk695ZYAzrWgEYbn/yOHC9 X-Received: by 2002:a17:902:28c9:: with SMTP id f67mr140095512plb.19.1561232096824; Sat, 22 Jun 2019 12:34:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232096; cv=none; d=google.com; s=arc-20160816; b=dAUG2R9aEqz7o/ymenajmSWb6LWlLZg98gEX+kjeHqW/pYoIbomPKTWcdUgMDJVA2M iacX/9EBXu1lhZWqqOOT33H957pJiEb6Uzwp/r3+t9gKrxjG2uhFJwN7Felz84sbeMlG giVQRwoKdkvRqgmSm68qsMiUIGDUNvv+YMmAWctRWkled5acsfr06UxaIWZ8+hBNQ1sl uPl6Fcbk9WeoESxccPesIMr0ozm1pnUgHHc2GdogKMjemfjZLSB0m3VfwOLqr4IfvIhS Gz2SFbYbhrK/42p3CtwThDoEfT8UasZ4swGRJ0crTcuhkjFf7KoIJAS+Wn3D8tdNIv23 QPzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Pgzq94PpKRgLR2IqqS+QLXIksHPk3/jPtTuBsqGb9fM=; b=CV0w9FWksu1PFrDC6vA/TCkCAs9DsGmj+6Daogd+q/euhO6EAC8/rP7jS2CPMAw0n/ sfyAiXQkIT2A0xMZUswy1cTIPpO/2Xa003IzUUJtRt7O4f52IjZBwKkzlaYas8WyT0W7 h6ifTLybfKUb0vETtRD7TqyAwJIGKIfiz92nYfmJ1xxzPvvGRSXSrGUPb8ZysvBDHg7e gc0g+bN3kJYFQ+Zu17DrO/imtSujS6b6lDlFAKqEtY2n4GudJy6eTIVh1BRNAS7mC5VX 7wKIlwX231yoM+4x1nstsOusWZAXu5aQUH12YgM7gunb1w8herDPp3YcYsMCu1Y1hvIu niKQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=jJ3qN3P7; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.34.56; Sat, 22 Jun 2019 12:34:56 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=jJ3qN3P7; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726370AbfFVTe4 (ORCPT + 3 others); Sat, 22 Jun 2019 15:34:56 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:40963 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726359AbfFVTez (ORCPT ); Sat, 22 Jun 2019 15:34:55 -0400 Received: by mail-wr1-f66.google.com with SMTP id c2so9704653wrm.8 for ; Sat, 22 Jun 2019 12:34:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Pgzq94PpKRgLR2IqqS+QLXIksHPk3/jPtTuBsqGb9fM=; b=jJ3qN3P7GB+LlOmKMZdJl6R+axoPADCBKhPa5hcIDq7yq3IJ3g06DH8JkGaZ5ArLd3 DQVRyVzYlmDx9Msk1nlD/j3pe2RFyrapO00lU5nYQoAiFH+q0jnOkb32uHjD92gMoIu+ jP6EH1wzF+cN93pZEYoQThimIrdpgiKtxagV4DunVLv8T0TOO2yt4b/xHS+YHWFcetwd DBnUu09qQ3D4IqoKEIPeWajOvYk4dbcOJ6rL+eNasQ1tgLIZ2sKYmI0tAlBlvAkmzogJ 9q96mI/HO4Mx+MQyghDeubACjAlCAYJWSjvfW+FGG4YHcqfU5CImVncNhCTgPTvVWtdi gsyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Pgzq94PpKRgLR2IqqS+QLXIksHPk3/jPtTuBsqGb9fM=; b=V2TSxoU1U423MQni2VelkntoOB845YW3tD9PVjA3EG+z9BgloZus1d0Qk4Nxhdo2i4 VB22I33TNOGsAna6NHWALBzbwQ1pB33e9DK1Bs8ZB9uBUwBMIdyTEy5mv9lBrHAghneK wLxSC+k0xE6BL8RlHnHPPb4L9x1fgVpG0omTdW17mJ49TkmAjyfisyHfwaLKc1gkLJUb HyNe3yPUdSOr6RZSrpPGukH9wLEz3c3026r7rbDqvypCC2Y4J9LH3tHlrT54YnFXFzq1 hd37Xl6/DoKcZMsumIu8oVPjmrO6MD/OthG53fpUwqfDTtkBJZqwLSXqxFTZ9EUzif3E Jtyw== X-Gm-Message-State: APjAAAUb8SFKOCiA78wchj8wb+z2f8IOdIuK8R/HtNU6kbS7qPDMp38N U1UnoFr8mHhKniyTpE5qHOOB4nln5y1555Ex X-Received: by 2002:adf:e8cb:: with SMTP id k11mr4318379wrn.244.1561232092962; Sat, 22 Jun 2019 12:34:52 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:52 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 12/26] crypto: arm64/aes-ccm - switch to AES library Date: Sat, 22 Jun 2019 21:34:13 +0200 Message-Id: <20190622193427.20336-13-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The CCM code calls directly into the scalar table based AES cipher for arm64 from the fallback path, and since this implementation is known to be non-time invariant, doing so from a time invariant SIMD cipher is a bit nasty. So let's switch to the AES library - this makes the code more robust, and drops the dependency on the generic AES cipher, allowing us to omit it entirely in the future. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/Kconfig | 2 +- arch/arm64/crypto/aes-ce-ccm-glue.c | 18 ++++++------------ 2 files changed, 7 insertions(+), 13 deletions(-) -- 2.20.1 diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index 1762055e7093..c6032bfb44fb 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -80,8 +80,8 @@ config CRYPTO_AES_ARM64_CE_CCM depends on ARM64 && KERNEL_MODE_NEON select CRYPTO_ALGAPI select CRYPTO_AES_ARM64_CE - select CRYPTO_AES_ARM64 select CRYPTO_AEAD + select CRYPTO_LIB_AES config CRYPTO_AES_ARM64_CE_BLK tristate "AES in ECB/CBC/CTR/XTS modes using ARMv8 Crypto Extensions" diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce-ccm-glue.c index cb89c80800b5..b9b7cf4b5a8f 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -46,8 +46,6 @@ asmlinkage void ce_aes_ccm_decrypt(u8 out[], u8 const in[], u32 cbytes, asmlinkage void ce_aes_ccm_final(u8 mac[], u8 const ctr[], u32 const rk[], u32 rounds); -asmlinkage void __aes_arm64_encrypt(u32 *rk, u8 *out, const u8 *in, int rounds); - static int ccm_setkey(struct crypto_aead *tfm, const u8 *in_key, unsigned int key_len) { @@ -127,8 +125,7 @@ static void ccm_update_mac(struct crypto_aes_ctx *key, u8 mac[], u8 const in[], } while (abytes >= AES_BLOCK_SIZE) { - __aes_arm64_encrypt(key->key_enc, mac, mac, - num_rounds(key)); + aes_encrypt(key, mac, mac); crypto_xor(mac, in, AES_BLOCK_SIZE); in += AES_BLOCK_SIZE; @@ -136,8 +133,7 @@ static void ccm_update_mac(struct crypto_aes_ctx *key, u8 mac[], u8 const in[], } if (abytes > 0) { - __aes_arm64_encrypt(key->key_enc, mac, mac, - num_rounds(key)); + aes_encrypt(key, mac, mac); crypto_xor(mac, in, abytes); *macp = abytes; } @@ -209,10 +205,8 @@ static int ccm_crypt_fallback(struct skcipher_walk *walk, u8 mac[], u8 iv0[], bsize = nbytes; crypto_inc(walk->iv, AES_BLOCK_SIZE); - __aes_arm64_encrypt(ctx->key_enc, buf, walk->iv, - num_rounds(ctx)); - __aes_arm64_encrypt(ctx->key_enc, mac, mac, - num_rounds(ctx)); + aes_encrypt(ctx, buf, walk->iv); + aes_encrypt(ctx, mac, mac); if (enc) crypto_xor(mac, src, bsize); crypto_xor_cpy(dst, src, buf, bsize); @@ -227,8 +221,8 @@ static int ccm_crypt_fallback(struct skcipher_walk *walk, u8 mac[], u8 iv0[], } if (!err) { - __aes_arm64_encrypt(ctx->key_enc, buf, iv0, num_rounds(ctx)); - __aes_arm64_encrypt(ctx->key_enc, mac, mac, num_rounds(ctx)); + aes_encrypt(ctx, buf, iv0); + aes_encrypt(ctx, mac, mac); crypto_xor(mac, buf, AES_BLOCK_SIZE); } return err; From patchwork Sat Jun 22 19:34:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167497 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2255954ilk; Sat, 22 Jun 2019 12:34:57 -0700 (PDT) X-Google-Smtp-Source: APXvYqzzqwqhg97K2DpH0JD6CsZZPKIfdEffkch0lVYNnawmBQfSl8iscxrkfyQzeIYL4MesVlDe X-Received: by 2002:a17:90a:ac0e:: with SMTP id o14mr14795054pjq.142.1561232097495; Sat, 22 Jun 2019 12:34:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232097; cv=none; d=google.com; s=arc-20160816; b=F6Zea9I974L2mwzUqSTFiEx17wEK2mIJbUaKf7O8c/dNJx8lw2oLVl+LG3d+VTtsaw Qy/68aY/Z0+Tm2BodxzNhmYGQDZH1eBrR8yW49lzW6YSk0yeIkWK9slbeN7mfB4hzbDt TJrZCF+7T3OxJ0S6jETDDSVe3xHUoz20SzCYGJp4cD2H7690TUK88t9I4VgBCEqKVVpU V3BP5qjzpvTxr5fhEDD4qGXYROJveOgNKjIrpDQeUfb5hPzLdtmea9JskZCKouTwPYlw eL/DrgZsYhDx4cAxzzK727pN/p/LGU9hFkyyBvUzrfaGXyWEAoa326ft9gdd0/uc9jb6 6bQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0+abpCskFH64uUUzWz6MrJWQeWRPxd08F5mJmqOJSaY=; b=TQkmfsJAKncXa89RFtqbJUxMfy9+ewURBYAUtWajHB1OFYgkwGbkf2LB16LzRohB9A HBkbEC/3cGvRcUvMlH3FV5WYsVchU4PaYPoYeC0F8RNQDAPvpYT9zV7Cgi/QPlEPJmYE vbb20b395CP8ne0/ym5bZMY7iaEK89izzLyXTbRr3GR1aqEozrFNuJOpI0C4n6XPhR6d iVlzKaK0ROgpj9mh+jXKBX55dv2vA7yL0s7aqyukGVLBFl8qo03VcfDv5AnbKe733dTx zHBwr3pPBHpGf1cW0/Itt/huV3TSfsZ0gO/c7VQJHiIPG3DJRPYqkd9cEwGAoICE/F7d szVQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=tG+zChu1; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.34.57; Sat, 22 Jun 2019 12:34:57 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=tG+zChu1; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726359AbfFVTe4 (ORCPT + 3 others); Sat, 22 Jun 2019 15:34:56 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:34664 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726286AbfFVTe4 (ORCPT ); Sat, 22 Jun 2019 15:34:56 -0400 Received: by mail-wm1-f66.google.com with SMTP id w9so11348986wmd.1 for ; Sat, 22 Jun 2019 12:34:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0+abpCskFH64uUUzWz6MrJWQeWRPxd08F5mJmqOJSaY=; b=tG+zChu1TY57HmPBHNOBr8ZTS5Jg27t/A8d8SsgNlYBqbOpiXe7548t9OubC0JH1Xu vrd0A/PhM6ADLoMzc+n3JJFJrOhQg6hB9pRiyT/2zq/LTPGTNfFfYF5rDXCyrx1UWM3W rBoShNAyWgIJxndSg4OARGF1XAUzH8uCn6tXROm+c64j8MlRChJLLa9OEgXcjetsWB1W v1nZHq/9+JCZ2cSIj0o/tgr2z6icIsEQR94XEqa5nfOAuPrTv9TfLFupxvuzUGojLddG I+H2VR+nmGifnSIWqm2zKr5rT9tHNvGXyD4LBeCMiX+9dY6wQBpT4Ez2G1XP29aevM7O ZQkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0+abpCskFH64uUUzWz6MrJWQeWRPxd08F5mJmqOJSaY=; b=GV9pu5jDEers1HOozJ+Ck+jF3ZZAgBuT7wLg4uJebVv7VX0MyH2rG5WFBIO6QzqVJF rZtGY2Rd7qg4S/VBSvd0VngyBZGCkLmIyQjYqZa7XMs8Ej9bzxLsxyPj4azao0eiQAjg tRC43ioVmRGHaHW+zotkHe4WAVf8ci6Vkyv0FF4zmOgIFrw379MjGycmyIIbN+eB6pe4 90rXbvNfKcK/po2tmKfTt41DMpBId15XZicMKad/A0hn+9odBGWUN5uNO65fXzrnxgNW vtloH9PHXAk34WnnlvBOE2JeNGsGuQutw8a8mW7TYbzpnn8GVJnTTZpNcNLH5HCPsro1 Fcqw== X-Gm-Message-State: APjAAAV1qVYRecuzDs64wABAOdihssSuJYH8y1otOEnKFcBrg2qheNqS 9+BJspt+WAOsLEDhaxXoTSbbYr7ULPtvOMCC X-Received: by 2002:a7b:cbc6:: with SMTP id n6mr6114671wmi.14.1561232093962; Sat, 22 Jun 2019 12:34:53 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:53 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 13/26] crypto: arm64/aes-neonbs - switch to library version of key expansion routine Date: Sat, 22 Jun 2019 21:34:14 +0200 Message-Id: <20190622193427.20336-14-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Switch to the new AES library that also provides an implementation of the AES key expansion routine. This removes the dependency on the generic AES cipher, allowing it to be omitted entirely in the future. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/Kconfig | 1 + arch/arm64/crypto/aes-neonbs-glue.c | 8 ++++---- 2 files changed, 5 insertions(+), 4 deletions(-) -- 2.20.1 diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index c6032bfb44fb..17bf5dc10aad 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -116,6 +116,7 @@ config CRYPTO_AES_ARM64_BS select CRYPTO_BLKCIPHER select CRYPTO_AES_ARM64_NEON_BLK select CRYPTO_AES_ARM64 + select CRYPTO_LIB_AES select CRYPTO_SIMD endif diff --git a/arch/arm64/crypto/aes-neonbs-glue.c b/arch/arm64/crypto/aes-neonbs-glue.c index 02b65d9eb947..cb8d90f795a0 100644 --- a/arch/arm64/crypto/aes-neonbs-glue.c +++ b/arch/arm64/crypto/aes-neonbs-glue.c @@ -77,7 +77,7 @@ static int aesbs_setkey(struct crypto_skcipher *tfm, const u8 *in_key, struct crypto_aes_ctx rk; int err; - err = crypto_aes_expand_key(&rk, in_key, key_len); + err = aes_expandkey(&rk, in_key, key_len); if (err) return err; @@ -136,7 +136,7 @@ static int aesbs_cbc_setkey(struct crypto_skcipher *tfm, const u8 *in_key, struct crypto_aes_ctx rk; int err; - err = crypto_aes_expand_key(&rk, in_key, key_len); + err = aes_expandkey(&rk, in_key, key_len); if (err) return err; @@ -208,7 +208,7 @@ static int aesbs_ctr_setkey_sync(struct crypto_skcipher *tfm, const u8 *in_key, struct aesbs_ctr_ctx *ctx = crypto_skcipher_ctx(tfm); int err; - err = crypto_aes_expand_key(&ctx->fallback, in_key, key_len); + err = aes_expandkey(&ctx->fallback, in_key, key_len); if (err) return err; @@ -274,7 +274,7 @@ static int aesbs_xts_setkey(struct crypto_skcipher *tfm, const u8 *in_key, return err; key_len /= 2; - err = crypto_aes_expand_key(&rk, in_key + key_len, key_len); + err = aes_expandkey(&rk, in_key + key_len, key_len); if (err) return err; From patchwork Sat Jun 22 19:34:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167498 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2255966ilk; Sat, 22 Jun 2019 12:34:58 -0700 (PDT) X-Google-Smtp-Source: APXvYqzs+UrFL+m45HBrqHh143Wc/6D9UogRJ1cza/64imLMdyvbGe730f6oY40IZX48Es5pmiHK X-Received: by 2002:a17:90a:d983:: with SMTP id d3mr14416516pjv.88.1561232098172; Sat, 22 Jun 2019 12:34:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232098; cv=none; d=google.com; s=arc-20160816; b=dQTWVTguOXP2ovdT5psBxeVPcSvFZYpw93FPopdYzIMCVBppCwkRTJfKVxucgrmYrj +MBBUkT0aveM4snKsq41U0ad0mEpwRBXzPzVu5mx8DYIy0yXMPULPM5rEK1if5D4raOA R5QtSK5/FsZijAtl1wIsGiHkmVVGeN0ZMxN4ppZPd+wOaJxMQtVsyJ0O43klrfAJYcaR 4xz79L8y49GNcRFliCAfJ0G5HVhW1iaohDzN05aJSBazaGV3jF1zFamOSpAm2Edc/HPC Q/yKnpa2BTnujcwHx3/W2rpXTfXLuDS7h7Kgu63TJHh6Cam6Qf3LVGudokFcMSLi6Gdh Xldg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=MzqBn3zXqdqT5rykfpJWuWfbyyxJXTIyDmCZ4k6FneI=; b=kLdFeHPrgwOCktCWPsE+hTmLjy4KLH/AZpnZVlSdF86S7WEdgWVnHpUl397VDjn7qs IR/lSQCM9u9Tr9bUh7xluW4NBzJX6KPjnhivp5TUxlRQa0n9jj42zCf+fM3MOMdQK1eg IkqLjN3SPEZxd5qg8nySHezVBjPJ1jG+3vtuGB/5vOGc4fZCrKSYqhs3pZn9eCOZxahD lLnPqABQUq4NZb2yjI+/8qENJztWQI61hEZAc+S5Usr4NI0dHE80G7140YoYFy6s8i+7 oGTFJpMP7stENDvzx6mhUmCyG3pDPWBldsz+8FK6oQO8eQtyVQ235lWhJbIvsG3UjfcW fPjQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=E9syHCVT; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.34.57; Sat, 22 Jun 2019 12:34:58 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=E9syHCVT; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726372AbfFVTe5 (ORCPT + 3 others); Sat, 22 Jun 2019 15:34:57 -0400 Received: from mail-wm1-f68.google.com ([209.85.128.68]:39674 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726368AbfFVTe5 (ORCPT ); Sat, 22 Jun 2019 15:34:57 -0400 Received: by mail-wm1-f68.google.com with SMTP id z23so9601914wma.4 for ; Sat, 22 Jun 2019 12:34:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MzqBn3zXqdqT5rykfpJWuWfbyyxJXTIyDmCZ4k6FneI=; b=E9syHCVTDPMLucp3RnHYQY+qNyIRICSrZTsJqEdZrKZuMfKt3+rLxR8wHsUFgFIZWf pO6ESMNs4KklPu6xy8TI0JkA/eaG9YXQH9RGWVPtfNDP0TAarTi+NO2LNrQG/V/DDwHL YiScaHnLm0FE3ujt6oUYT4vsSWF0OK90rYRAgVHfx2uXxol+cm0MnR4bpY70TgQqOboY +Spm9I5Gb96zoOvkVBnAJIG4WVi72ZNpGfxOSKyrILQJkFtnqo0B/dU4DTRWIFzR6wtP LyeTY+AqYgDNzyqEtkj2yIEX4x8zf34dwPEFeUIqfZe2G3LSFAoU32h02rt+3i0Rv3IR B4DQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MzqBn3zXqdqT5rykfpJWuWfbyyxJXTIyDmCZ4k6FneI=; b=PDZ6jtx0QTQ1ycK5vqhIwlZ/1BKr5iKnfIxVwDEPrDjAQe7e+IR+6Dn/tFJPGiRY/l ffPbCvR1wwos6NnkOzGEU9oeUeOsCOKHZVuwdE+cbHo7kdgnZuGxMKLn42mtu6dVRnls x77c/h26rkbV/Egx2K+SxlDCsvGPEmmLSeFPCKK++C1wfm+au2fiQJdTI8cz2xAK2TyY iV8VZ/UD23nvvpNrPnwVyNwALv7bq8w0dFW7ITEfXj9N+sIU3iLk+J1kFSgG/65jljz1 f2kwjhtozu0T8GkbLC2/IbdCIe8222FTfVyUNbNUs4SsVzSEb8oZEz+Bw8UOveV1Mrj+ J4ow== X-Gm-Message-State: APjAAAW4P46PgLeadGQQqG+OtT17WiuioFn2qY24wg03qrJKjWOL5N7K 8nmHLXzQgRBAqaoeqtcxYHA9k9fCmZicuzNh X-Received: by 2002:a7b:cf27:: with SMTP id m7mr8911315wmg.7.1561232095024; Sat, 22 Jun 2019 12:34:55 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:54 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 14/26] crypto: arm64/aes-ce - switch to library version of key expansion routine Date: Sat, 22 Jun 2019 21:34:15 +0200 Message-Id: <20190622193427.20336-15-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Switch to the new AES library that also provides an implementation of the AES key expansion routine. This removes the dependency on the generic AES cipher, allowing it to be omitted entirely in the future. While at it, remove some references to the table based arm64 version of AES and replace them with AES library calls as well. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/Kconfig | 2 +- arch/arm64/crypto/aes-glue.c | 17 ++++++++++------- 2 files changed, 11 insertions(+), 8 deletions(-) -- 2.20.1 diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index 17bf5dc10aad..66dea518221c 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -96,7 +96,7 @@ config CRYPTO_AES_ARM64_NEON_BLK depends on KERNEL_MODE_NEON select CRYPTO_BLKCIPHER select CRYPTO_AES_ARM64 - select CRYPTO_AES + select CRYPTO_LIB_AES select CRYPTO_SIMD config CRYPTO_CHACHA20_NEON diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index f0ceb545bd1e..3c80345d914f 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -26,7 +26,6 @@ #ifdef USE_V8_CRYPTO_EXTENSIONS #define MODE "ce" #define PRIO 300 -#define aes_setkey ce_aes_setkey #define aes_expandkey ce_aes_expandkey #define aes_ecb_encrypt ce_aes_ecb_encrypt #define aes_ecb_decrypt ce_aes_ecb_decrypt @@ -42,8 +41,6 @@ MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 Crypto Extensions"); #else #define MODE "neon" #define PRIO 200 -#define aes_setkey crypto_aes_set_key -#define aes_expandkey crypto_aes_expand_key #define aes_ecb_encrypt neon_aes_ecb_encrypt #define aes_ecb_decrypt neon_aes_ecb_decrypt #define aes_cbc_encrypt neon_aes_cbc_encrypt @@ -121,7 +118,14 @@ struct mac_desc_ctx { static int skcipher_aes_setkey(struct crypto_skcipher *tfm, const u8 *in_key, unsigned int key_len) { - return aes_setkey(crypto_skcipher_tfm(tfm), in_key, key_len); + struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); + int ret; + + ret = aes_expandkey(ctx, in_key, key_len); + if (ret) + crypto_skcipher_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN); + + return ret; } static int xts_set_key(struct crypto_skcipher *tfm, const u8 *in_key, @@ -649,15 +653,14 @@ static void mac_do_update(struct crypto_aes_ctx *ctx, u8 const in[], int blocks, kernel_neon_end(); } else { if (enc_before) - __aes_arm64_encrypt(ctx->key_enc, dg, dg, rounds); + aes_encrypt(ctx, dg, dg); while (blocks--) { crypto_xor(dg, in, AES_BLOCK_SIZE); in += AES_BLOCK_SIZE; if (blocks || enc_after) - __aes_arm64_encrypt(ctx->key_enc, dg, dg, - rounds); + aes_encrypt(ctx, dg, dg); } } } From patchwork Sat Jun 22 19:34:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167499 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2255991ilk; Sat, 22 Jun 2019 12:34:59 -0700 (PDT) X-Google-Smtp-Source: APXvYqzLrMbiJxiFQB2qHrj3B89eymZSzq8Wn+jvbrVnTtR60zO/vmMadsaYH5qmTM4n2vFu/XoG X-Received: by 2002:a17:90a:3225:: with SMTP id k34mr14462747pjb.31.1561232099699; Sat, 22 Jun 2019 12:34:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232099; cv=none; d=google.com; s=arc-20160816; b=W6MZZWyKVXtQqt3J+o3Om9t8HiJkPVADVd+uwpQNUdBHbpRhKCtmX5SUCdEnnzpjp2 7DA5Vve/+nW+VbgUZNP6nm75Dsao6ZiBvLdoOFRA0bLzLslLooW7kLTlIdWUqFkGAFPs P7TlJPr3whc5P29BGdsnwmEUSg+6SGOF85lOm02b+25pStCMK9AbS2gwFZr/3ZJxarrX n0a4MRKBjl0TIIsWQzl5E2gwrh4WCdyn939TDzkFOqYRdzp9pWzGII5B90WjoObZnMF5 KHfvHWa7DggCIBBBYmcX7HbkL7nQjjl5HqNCB7asm9QLm3qky7m0tEdgDlF6WIU97Ihm YNKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=m8xE/gQHhxeiNIomdxtjEbSnrgED29gmEg1crTL/DPU=; b=fYymZIV/XieHEP91Sx7V7cvmeTHyZlC+PxuNk5IU9BMgjvM4zw9txJHnUJXl4p271S sbfdQozZznlpoAkqtb3HuXePMRumBXpkLPIWGpPg1PNkkm4SZRWyvIcy8DesE/wIgBQ1 Bu6/IOZ09YdElvZIsMjNQds2ScOqf6f4L8oWHfFuQB0pM/kgeyEblqCxh+3xqaXF54tg PYAjKszxjood0eyFijnIKdMBp3M6siNdfIf5fGsAJ6wIm896FCh0x/vXIVKK+yo6pV3s mPwZM+ym5qYtzOSiaj2IN/Z+ut+KVhEIvKbxKcn22qrpsRfZDY8Ls2oiDdcLaCOKL2Km 8d0w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QQmeNaik; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.34.59; Sat, 22 Jun 2019 12:34:59 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QQmeNaik; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726374AbfFVTe7 (ORCPT + 3 others); Sat, 22 Jun 2019 15:34:59 -0400 Received: from mail-wm1-f68.google.com ([209.85.128.68]:33749 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726286AbfFVTe6 (ORCPT ); Sat, 22 Jun 2019 15:34:58 -0400 Received: by mail-wm1-f68.google.com with SMTP id h19so11351776wme.0 for ; Sat, 22 Jun 2019 12:34:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=m8xE/gQHhxeiNIomdxtjEbSnrgED29gmEg1crTL/DPU=; b=QQmeNaikMHZjrX1wwWJjAdu0i78Io2j+/Ojq9GGLuAMBAkQk2YGQZykbZVXTMBDvHh NQRkdtDU84L4ERYF7N79airj52nGVgSBcfESH3z+bGpvv/BBaoix/aLG/M2cwNsRZx9b 8WuBDedsjNTAt4p08an9+uj/ZygCzmpQTqxgYLoSzzTGrUyTXX7+/g24IU0TCLaYewUk qvBr+gtID1juvAp6Fx0a/icU2W1jWzzbB5tEFOAyZOatWpP51TwGNYmuNTSHR2uNrCfB I3gRajgXDLv5kkj9GNA3WfDvtSQ7FREp7bDAkiI3Ix3NeZFIkVtu6hSK0jsIFAPvoNDp bBww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=m8xE/gQHhxeiNIomdxtjEbSnrgED29gmEg1crTL/DPU=; b=F2aGfHFc6P2qYXjjeq7q5Agc1IfBXvELuAwl1UgI4NE6g1rRb3JSmQ1k/60XcwA0O/ v8mpxm5yCUEQsWXwc/eSLShRTOEzCK01SbTwMlpnHuWTzvmbjFsbFdnLXRLNwdOiDjTG OhODHNNjgr2AFaNQJLi347dj5zEshdB3Yl83J/OKz5ZcqQGfapYYp3krtNZhY3tiq3h+ W7LMLcdkqGOE5dyvgj/IFM98vR3VHpRp5XA6D1jcBG2UhAfaeIzvnfgDWLKqQfgnCV5Z XHeB53tS4YwLme3MqykmZk21vw0AACjMWRuQa0ZWZdpJLkhQoMZX+RBF6KPhlgeceMm+ 9ZwA== X-Gm-Message-State: APjAAAUohRMq+pN1ev4AS241HQStOQsx6Qy20CKzoPLnecyNWFOfUtwH BDjOJ3SF76gh0xC7tqDE/7ggpDpk41rBzBUs X-Received: by 2002:a7b:cb08:: with SMTP id u8mr8573772wmj.167.1561232096039; Sat, 22 Jun 2019 12:34:56 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:55 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 15/26] crypto: generic/aes - drop key expansion routine in favor of library version Date: Sat, 22 Jun 2019 21:34:16 +0200 Message-Id: <20190622193427.20336-16-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Drop aes-generic's version of crypto_aes_expand_key(), and switch to the key expansion routine provided by the AES library. AES key expansion is not performance critical, and it is better to have a single version shared by all AES implementations. Signed-off-by: Ard Biesheuvel --- crypto/Kconfig | 1 + crypto/aes_generic.c | 153 +------------------- include/crypto/aes.h | 2 - 3 files changed, 3 insertions(+), 153 deletions(-) -- 2.20.1 diff --git a/crypto/Kconfig b/crypto/Kconfig index df6f0be66574..80ea118600ab 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -1072,6 +1072,7 @@ config CRYPTO_LIB_AES config CRYPTO_AES tristate "AES cipher algorithms" select CRYPTO_ALGAPI + select CRYPTO_LIB_AES help AES cipher algorithms (FIPS-197). AES uses the Rijndael algorithm. diff --git a/crypto/aes_generic.c b/crypto/aes_generic.c index 3aa4a715c216..426deb437f19 100644 --- a/crypto/aes_generic.c +++ b/crypto/aes_generic.c @@ -1125,155 +1125,6 @@ EXPORT_SYMBOL_GPL(crypto_fl_tab); EXPORT_SYMBOL_GPL(crypto_it_tab); EXPORT_SYMBOL_GPL(crypto_il_tab); -/* initialise the key schedule from the user supplied key */ - -#define star_x(x) (((x) & 0x7f7f7f7f) << 1) ^ ((((x) & 0x80808080) >> 7) * 0x1b) - -#define imix_col(y, x) do { \ - u = star_x(x); \ - v = star_x(u); \ - w = star_x(v); \ - t = w ^ (x); \ - (y) = u ^ v ^ w; \ - (y) ^= ror32(u ^ t, 8) ^ \ - ror32(v ^ t, 16) ^ \ - ror32(t, 24); \ -} while (0) - -#define ls_box(x) \ - crypto_fl_tab[0][byte(x, 0)] ^ \ - crypto_fl_tab[1][byte(x, 1)] ^ \ - crypto_fl_tab[2][byte(x, 2)] ^ \ - crypto_fl_tab[3][byte(x, 3)] - -#define loop4(i) do { \ - t = ror32(t, 8); \ - t = ls_box(t) ^ rco_tab[i]; \ - t ^= ctx->key_enc[4 * i]; \ - ctx->key_enc[4 * i + 4] = t; \ - t ^= ctx->key_enc[4 * i + 1]; \ - ctx->key_enc[4 * i + 5] = t; \ - t ^= ctx->key_enc[4 * i + 2]; \ - ctx->key_enc[4 * i + 6] = t; \ - t ^= ctx->key_enc[4 * i + 3]; \ - ctx->key_enc[4 * i + 7] = t; \ -} while (0) - -#define loop6(i) do { \ - t = ror32(t, 8); \ - t = ls_box(t) ^ rco_tab[i]; \ - t ^= ctx->key_enc[6 * i]; \ - ctx->key_enc[6 * i + 6] = t; \ - t ^= ctx->key_enc[6 * i + 1]; \ - ctx->key_enc[6 * i + 7] = t; \ - t ^= ctx->key_enc[6 * i + 2]; \ - ctx->key_enc[6 * i + 8] = t; \ - t ^= ctx->key_enc[6 * i + 3]; \ - ctx->key_enc[6 * i + 9] = t; \ - t ^= ctx->key_enc[6 * i + 4]; \ - ctx->key_enc[6 * i + 10] = t; \ - t ^= ctx->key_enc[6 * i + 5]; \ - ctx->key_enc[6 * i + 11] = t; \ -} while (0) - -#define loop8tophalf(i) do { \ - t = ror32(t, 8); \ - t = ls_box(t) ^ rco_tab[i]; \ - t ^= ctx->key_enc[8 * i]; \ - ctx->key_enc[8 * i + 8] = t; \ - t ^= ctx->key_enc[8 * i + 1]; \ - ctx->key_enc[8 * i + 9] = t; \ - t ^= ctx->key_enc[8 * i + 2]; \ - ctx->key_enc[8 * i + 10] = t; \ - t ^= ctx->key_enc[8 * i + 3]; \ - ctx->key_enc[8 * i + 11] = t; \ -} while (0) - -#define loop8(i) do { \ - loop8tophalf(i); \ - t = ctx->key_enc[8 * i + 4] ^ ls_box(t); \ - ctx->key_enc[8 * i + 12] = t; \ - t ^= ctx->key_enc[8 * i + 5]; \ - ctx->key_enc[8 * i + 13] = t; \ - t ^= ctx->key_enc[8 * i + 6]; \ - ctx->key_enc[8 * i + 14] = t; \ - t ^= ctx->key_enc[8 * i + 7]; \ - ctx->key_enc[8 * i + 15] = t; \ -} while (0) - -/** - * crypto_aes_expand_key - Expands the AES key as described in FIPS-197 - * @ctx: The location where the computed key will be stored. - * @in_key: The supplied key. - * @key_len: The length of the supplied key. - * - * Returns 0 on success. The function fails only if an invalid key size (or - * pointer) is supplied. - * The expanded key size is 240 bytes (max of 14 rounds with a unique 16 bytes - * key schedule plus a 16 bytes key which is used before the first round). - * The decryption key is prepared for the "Equivalent Inverse Cipher" as - * described in FIPS-197. The first slot (16 bytes) of each key (enc or dec) is - * for the initial combination, the second slot for the first round and so on. - */ -int crypto_aes_expand_key(struct crypto_aes_ctx *ctx, const u8 *in_key, - unsigned int key_len) -{ - u32 i, t, u, v, w, j; - - if (key_len != AES_KEYSIZE_128 && key_len != AES_KEYSIZE_192 && - key_len != AES_KEYSIZE_256) - return -EINVAL; - - ctx->key_length = key_len; - - ctx->key_enc[0] = get_unaligned_le32(in_key); - ctx->key_enc[1] = get_unaligned_le32(in_key + 4); - ctx->key_enc[2] = get_unaligned_le32(in_key + 8); - ctx->key_enc[3] = get_unaligned_le32(in_key + 12); - - ctx->key_dec[key_len + 24] = ctx->key_enc[0]; - ctx->key_dec[key_len + 25] = ctx->key_enc[1]; - ctx->key_dec[key_len + 26] = ctx->key_enc[2]; - ctx->key_dec[key_len + 27] = ctx->key_enc[3]; - - switch (key_len) { - case AES_KEYSIZE_128: - t = ctx->key_enc[3]; - for (i = 0; i < 10; ++i) - loop4(i); - break; - - case AES_KEYSIZE_192: - ctx->key_enc[4] = get_unaligned_le32(in_key + 16); - t = ctx->key_enc[5] = get_unaligned_le32(in_key + 20); - for (i = 0; i < 8; ++i) - loop6(i); - break; - - case AES_KEYSIZE_256: - ctx->key_enc[4] = get_unaligned_le32(in_key + 16); - ctx->key_enc[5] = get_unaligned_le32(in_key + 20); - ctx->key_enc[6] = get_unaligned_le32(in_key + 24); - t = ctx->key_enc[7] = get_unaligned_le32(in_key + 28); - for (i = 0; i < 6; ++i) - loop8(i); - loop8tophalf(i); - break; - } - - ctx->key_dec[0] = ctx->key_enc[key_len + 24]; - ctx->key_dec[1] = ctx->key_enc[key_len + 25]; - ctx->key_dec[2] = ctx->key_enc[key_len + 26]; - ctx->key_dec[3] = ctx->key_enc[key_len + 27]; - - for (i = 4; i < key_len + 24; ++i) { - j = key_len + 24 - (i & ~3) + (i & 3); - imix_col(ctx->key_dec[j], ctx->key_enc[i]); - } - return 0; -} -EXPORT_SYMBOL_GPL(crypto_aes_expand_key); - /** * crypto_aes_set_key - Set the AES key. * @tfm: The %crypto_tfm that is used in the context. @@ -1281,7 +1132,7 @@ EXPORT_SYMBOL_GPL(crypto_aes_expand_key); * @key_len: The size of the key. * * Returns 0 on success, on failure the %CRYPTO_TFM_RES_BAD_KEY_LEN flag in tfm - * is set. The function uses crypto_aes_expand_key() to expand the key. + * is set. The function uses aes_expand_key() to expand the key. * &crypto_aes_ctx _must_ be the private data embedded in @tfm which is * retrieved with crypto_tfm_ctx(). */ @@ -1292,7 +1143,7 @@ int crypto_aes_set_key(struct crypto_tfm *tfm, const u8 *in_key, u32 *flags = &tfm->crt_flags; int ret; - ret = crypto_aes_expand_key(ctx, in_key, key_len); + ret = aes_expandkey(ctx, in_key, key_len); if (!ret) return 0; diff --git a/include/crypto/aes.h b/include/crypto/aes.h index d0067fca0cd0..0a64a977f9b3 100644 --- a/include/crypto/aes.h +++ b/include/crypto/aes.h @@ -35,8 +35,6 @@ extern const u32 crypto_il_tab[4][256] ____cacheline_aligned; int crypto_aes_set_key(struct crypto_tfm *tfm, const u8 *in_key, unsigned int key_len); -int crypto_aes_expand_key(struct crypto_aes_ctx *ctx, const u8 *in_key, - unsigned int key_len); /** * aes_expandkey - Expands the AES key as described in FIPS-197 From patchwork Sat Jun 22 19:34:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167500 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2255999ilk; Sat, 22 Jun 2019 12:35:00 -0700 (PDT) X-Google-Smtp-Source: APXvYqxQO3uqqL4V2X548oxz6x1jGz+YWmKXAnwu8Dcl3jx6JchluPtWCxJAn1U6WXGSPnXsBRAe X-Received: by 2002:a17:90a:24ac:: with SMTP id i41mr14504377pje.124.1561232100698; Sat, 22 Jun 2019 12:35:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232100; cv=none; d=google.com; s=arc-20160816; b=mYDYNAD4pgD/6wpuS90FMEjPeaRzz/391J25I/LY1ujb/Si/xLaamEkaHa/ZGiX+iX teuPJqRiPQ405meLhrYRUNjR8KR+R+PEWyxwPAdvVwuKY+rhjzfen+5K/cjOC2OTVt6s t54yG4tM0BPY8Na7/WlZhHvsOqHnRQca4hQTD6Q1mipyYmWYFyNsVndDYeVzdqA/DjPC Hjog+3XI0QE53bypwtvP15gaYSJDoW/OgszwjrwsxeGxVv2vrzp52bKe+dAcur3COb4U 09HrJkXeL1MXJ7lY84a4TsET5ckpN0ne0wZz5SEYJ+TJRVg6dHhoQp0xEJB4NaMFiUTp mvVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=wZx82U9voWItQ7XjqUeJC5jib2swTQ8RphpE15DMS1M=; b=KeYnPNMYu6jstX5OyhEyqaqK2JnxzdpIW+Bqv73NfbWM3fy21LaxSGRacEH+ofsoik TaxAQletQSYel2IWpQVrIkYJ0u9cvQqjM6k36EoEXBofA6jW/ZlBiJJeM27LWfpjWJ6D 2XAoKdf0q9oApuNgj0f2qpYvJo9mOLGUqIRcnQ0Tyj6tTr3VcSeIBUetxOq/NiJV2KhW z0EBuOsI6vhgFY2ACM08d7CAEBTg2KuE01h/VDVwdUGfUfzf6ELCBp0L/iUnf3u2hk/r o/pG+V2Uhrkz6TiLrZ52is/ME3INNlMglyLRKeQKwk6ZaPwgy45qii1uuqJZIava5SNH MKNg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ZhEjEvQI; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.35.00; Sat, 22 Jun 2019 12:35:00 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ZhEjEvQI; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726376AbfFVTe7 (ORCPT + 3 others); Sat, 22 Jun 2019 15:34:59 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:43957 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726368AbfFVTe7 (ORCPT ); Sat, 22 Jun 2019 15:34:59 -0400 Received: by mail-wr1-f68.google.com with SMTP id p13so9694043wru.10 for ; Sat, 22 Jun 2019 12:34:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wZx82U9voWItQ7XjqUeJC5jib2swTQ8RphpE15DMS1M=; b=ZhEjEvQIqhA5GX3sES588EqzaZ0VJ2PTS8uVFfg7XONw+MNc+ACw15XwxZsHu0RC/Q Ht/GVajnrlhCzulhZhDNLW4sDkmggVVmC1qBPzTkX/32sLMFLHf8FRxevQgeBnvv4nyc G+rmAo+m6NIaiIdf1gox9fRBqVscbmRo3AGjjT76BzXXwXNdrk4w3nmrqPfWiDetBnUz bG5mQei/re7ml0qWsDrYinGkQmyLT47zeQc278WBrBCjnfWVnrAbEdJerPxDMLz0YtUD FBhpkfBECKsGQlkNz4tP3I5MME0dEpQK/JaQulCMzac7Ony+m6CwI3PEwHJG7psPQ7KP /b+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wZx82U9voWItQ7XjqUeJC5jib2swTQ8RphpE15DMS1M=; b=lUynpF3AuNckWnkqrFCmMT7BJ7JXFsMx47wQnnQHPcTfzk/Gi0OBlOEobY3Pp0TrFF Wcr8vgqiSXv7YpgoPJFD8dwysLLg8wpbtjPjJiOyzEMKxXXaHkdRBI+djHnemLRHvLyF qFkMnx/6iS9NEdLHYsQQCntO3KNIgR9dXzbO6PHUCw+KZ2RkD1lGGcqz2rIAFSVRrXE4 wzn6SGEWzN3u8Cmep75iRb3RB3Xc+joxKeCdeg6D2N217Y3KByt85/9OM0DJo3mczVRo zLzxCD6IvxBnG8uMGDYP+6qBfPOOkf4FtNNzKAF6368jbaBNwYmBR0kVPhJ9yzJIMFVc y+ZQ== X-Gm-Message-State: APjAAAXdbAZuo+BU2cM63AiSFJjWdFz8UkAx/IFm6o9X+1urNiCdttxv NPrNrfXdS3qwvIhkwSLmqjeX5qSOFP7WhTlZ X-Received: by 2002:adf:e8c8:: with SMTP id k8mr73848941wrn.285.1561232097032; Sat, 22 Jun 2019 12:34:57 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:56 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 16/26] crypto: ctr - add helper for performing a CTR encryption walk Date: Sat, 22 Jun 2019 21:34:17 +0200 Message-Id: <20190622193427.20336-17-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Add a static inline helper modeled after crypto_cbc_encrypt_walk() that can be reused for SIMD algorithms that need to implement a non-SIMD fallback for performing CTR encryption. Signed-off-by: Ard Biesheuvel --- include/crypto/ctr.h | 53 ++++++++++++++++++++ 1 file changed, 53 insertions(+) -- 2.20.1 diff --git a/include/crypto/ctr.h b/include/crypto/ctr.h index 4180fc080e3b..b441274e9b27 100644 --- a/include/crypto/ctr.h +++ b/include/crypto/ctr.h @@ -13,8 +13,61 @@ #ifndef _CRYPTO_CTR_H #define _CRYPTO_CTR_H +#include +#include +#include +#include + #define CTR_RFC3686_NONCE_SIZE 4 #define CTR_RFC3686_IV_SIZE 8 #define CTR_RFC3686_BLOCK_SIZE 16 +#define CTR_HELPER_MAX_BLOCK_SIZE 16 + +static inline int crypto_ctr_encrypt_walk(struct skcipher_request *req, + void (*fn)(struct crypto_skcipher *, + const u8 *, u8 *)) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + int blocksize = crypto_skcipher_blocksize(tfm); + u8 buf[CTR_HELPER_MAX_BLOCK_SIZE]; + struct skcipher_walk walk; + int err; + + /* verify some assumptions that help us keep the code simple */ + if (WARN_ON_ONCE(!is_power_of_2(blocksize) || + blocksize > CTR_HELPER_MAX_BLOCK_SIZE)) + return -EINVAL; + + err = skcipher_walk_virt(&walk, req, false); + + while (walk.nbytes > 0) { + u8 *dst = walk.dst.virt.addr; + u8 *src = walk.src.virt.addr; + int nbytes = walk.nbytes; + int tail = 0; + + if (nbytes < walk.total) { + nbytes = round_down(nbytes, blocksize); + tail = walk.nbytes & (blocksize - 1); + } + + do { + int bsize = min(nbytes, blocksize); + + fn(tfm, walk.iv, buf); + + crypto_xor_cpy(dst, src, buf, bsize); + crypto_inc(walk.iv, blocksize); + + dst += blocksize; + src += blocksize; + nbytes -= blocksize; + } while (nbytes > 0); + + err = skcipher_walk_done(&walk, tail); + } + return err; +} + #endif /* _CRYPTO_CTR_H */ From patchwork Sat Jun 22 19:34:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167502 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2256037ilk; Sat, 22 Jun 2019 12:35:02 -0700 (PDT) X-Google-Smtp-Source: APXvYqxYO4Fnp5XLyaU0VWRvUW2tJRZ1QTwJUkLuvwLNYjQRGax4BZWq5pPY0NrcGxxv61WL3Cst X-Received: by 2002:a63:f4e:: with SMTP id 14mr24787115pgp.58.1561232102753; Sat, 22 Jun 2019 12:35:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232102; cv=none; d=google.com; s=arc-20160816; b=vIOaeBD9wTAcpzc1xCg0lr0kLbXzFJdSzF2UEepCX7+QF/U9Xp5ZMT++WQhtajEXuL m8BlYcMURmUMbfB4VmNT16EO0n/wX/Pe7nOuAgoC7czW4tQ0EGxxMd7CmlJgo0wN7M6T Az29oRoPsGh7iwhhXF2drbhG4hM531C7B9CMbBXXcYaNTiCwvwVPsWgCf/GjmgPV/j9A HFXYQ3u8SvliA/oEHAgBledKfIhWC9ndFQX6Oi+oUH1MyY/h5n9Ui0H4RpA18oPgbRlN rCloyytIi/so+2TFwIUHHp2Dlo3euoUXj4irxQT8sfOgEnvKxvZV48+rabYampPAEhwm oMgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=oSCqTZSbnLGprwAW+e+bdrFNNEkfE8SXrdCsMDCAbr0=; b=uX8o97+LchwVKAtUKn69q7tW61FO1uP+7lrjvwpV5O5x0Bj+liyuLoGaWrplqLjRWu R0pAOseZXorhy8/+0bl4HxrdOQhapZoEIi390/n0rJB5UUDf8E87bcMdyag+4gwMdxlb l3R5CqUlG3p9DjeM8tn9UEC9QYSWQUIV/Mif44dNXrI4Y2mmgONCJUndFj5RV4f/kFf/ n4apwNxk6Sj6+Ap1FJ8syVXlemXAuQuiW4jaMx1Ao3XrOWuTU+6NDQxC5403p27qcpAq pLtfh/e7062z8pvnGD3ZnWcEpJCV/ZgpyUkkd7DDUHtj52GNiEFr2sBfyb2Ji0PEy8XY BC8w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=XLvYdmzx; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.35.02; Sat, 22 Jun 2019 12:35:02 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=XLvYdmzx; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726368AbfFVTfC (ORCPT + 3 others); Sat, 22 Jun 2019 15:35:02 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:42724 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726286AbfFVTfB (ORCPT ); Sat, 22 Jun 2019 15:35:01 -0400 Received: by mail-wr1-f68.google.com with SMTP id x17so9693741wrl.9 for ; Sat, 22 Jun 2019 12:34:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oSCqTZSbnLGprwAW+e+bdrFNNEkfE8SXrdCsMDCAbr0=; b=XLvYdmzxeQ2M1tGuF+68xj4dQmIlodjFDHEdTN8LhTKOP5H7ZcCelcfoTo1bLG2HXg VGmzxKuinpIWzKFX0MBT/RZgkFdwvbXOgS/GMqGtpSos/+P7M9XFahSXvnyJ4nUqR014 E3jSxIxcAet0H9D9UcYPBsnZ31gxgh5ZfF5n8RSgkKL0M0yvAvjePHhc2YHqM/ygqCGb NLYMjJGwKL5lxv6sFK8k09CSpeOhEOo1NtZW31ajLjQDqhXFocepnOM1UnIIPd70LI4y UyV1V4bHSz54M4D0anatWFm5BuRG++FhrCb7LZShlMYV983FhFzug5RqR9lhC4lzEjvM OBUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oSCqTZSbnLGprwAW+e+bdrFNNEkfE8SXrdCsMDCAbr0=; b=RchRO7Gj766YjnLrDABNs4meOTlf9s2QpMx4EMVC2W+KzXLb9bS5/a98vzmViIkb09 VL/G28f4mwjSm4aeVaGUuOb1hpOc/h1JVpl8QQ+X/MbQEEBdG+ym8dNLS3uiKeh2tPoh iBG/hSq+1+I1WbpGPapuUAe8v2sCf3HzYC4ZuinXLVIabU7FxzEMEadQoAJtQ4TuZezD 1MTNxKUDOYVeBEiYQka5WNZqca7xOLJ0l8wTljzUtcWi6agDdbrCrtBTDsPrqs6R08ZY nb4m+OZ9oY6lvHluYk3b820VenP+lceWSqYJA2oNjRLRA00finHsZ9EoBqbInGNHWgrE AAIQ== X-Gm-Message-State: APjAAAUiH7sHc1YCvPjgSBn2RJpAigGS6kmZL+SzOXITkwoYvVmlYNfx DPGShFzfGhCyRXa3l3YwgISshNdHC65jmvhy X-Received: by 2002:a5d:4642:: with SMTP id j2mr3319100wrs.211.1561232098125; Sat, 22 Jun 2019 12:34:58 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:57 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 17/26] crypto: aes - move sync ctr(aes) to AES library and generic helper Date: Sat, 22 Jun 2019 21:34:18 +0200 Message-Id: <20190622193427.20336-18-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org In preparation of duplicating the sync ctr(aes) functionality to modules under arch/arm, move the helper function from a inline .h file to the AES library, which is already depended upon by the drivers that use this fallback. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-ctr-fallback.h | 53 -------------------- arch/arm64/crypto/aes-glue.c | 22 ++++++-- arch/arm64/crypto/aes-neonbs-glue.c | 21 ++++++-- 3 files changed, 33 insertions(+), 63 deletions(-) -- 2.20.1 diff --git a/arch/arm64/crypto/aes-ctr-fallback.h b/arch/arm64/crypto/aes-ctr-fallback.h deleted file mode 100644 index c9285717b6b5..000000000000 --- a/arch/arm64/crypto/aes-ctr-fallback.h +++ /dev/null @@ -1,53 +0,0 @@ -/* - * Fallback for sync aes(ctr) in contexts where kernel mode NEON - * is not allowed - * - * Copyright (C) 2017 Linaro Ltd - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - */ - -#include -#include - -asmlinkage void __aes_arm64_encrypt(u32 *rk, u8 *out, const u8 *in, int rounds); - -static inline int aes_ctr_encrypt_fallback(struct crypto_aes_ctx *ctx, - struct skcipher_request *req) -{ - struct skcipher_walk walk; - u8 buf[AES_BLOCK_SIZE]; - int err; - - err = skcipher_walk_virt(&walk, req, true); - - while (walk.nbytes > 0) { - u8 *dst = walk.dst.virt.addr; - u8 *src = walk.src.virt.addr; - int nbytes = walk.nbytes; - int tail = 0; - - if (nbytes < walk.total) { - nbytes = round_down(nbytes, AES_BLOCK_SIZE); - tail = walk.nbytes % AES_BLOCK_SIZE; - } - - do { - int bsize = min(nbytes, AES_BLOCK_SIZE); - - __aes_arm64_encrypt(ctx->key_enc, buf, walk.iv, - 6 + ctx->key_length / 4); - crypto_xor_cpy(dst, src, buf, bsize); - crypto_inc(walk.iv, AES_BLOCK_SIZE); - - dst += AES_BLOCK_SIZE; - src += AES_BLOCK_SIZE; - nbytes -= AES_BLOCK_SIZE; - } while (nbytes > 0); - - err = skcipher_walk_done(&walk, tail); - } - return err; -} diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index 3c80345d914f..6dc90557282d 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -21,7 +22,6 @@ #include #include "aes-ce-setkey.h" -#include "aes-ctr-fallback.h" #ifdef USE_V8_CRYPTO_EXTENSIONS #define MODE "ce" @@ -404,13 +404,25 @@ static int ctr_encrypt(struct skcipher_request *req) return err; } -static int ctr_encrypt_sync(struct skcipher_request *req) +static void ctr_encrypt_one(struct crypto_skcipher *tfm, const u8 *src, u8 *dst) { - struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); - struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); + const struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); + unsigned long flags; + + /* + * Temporarily disable interrupts to avoid races where + * evicted when the CPU is interrupted to do something + * else. + */ + local_irq_save(flags); + aes_encrypt(ctx, dst, src); + local_irq_restore(flags); +} +static int ctr_encrypt_sync(struct skcipher_request *req) +{ if (!crypto_simd_usable()) - return aes_ctr_encrypt_fallback(ctx, req); + return crypto_ctr_encrypt_walk(req, ctr_encrypt_one); return ctr_encrypt(req); } diff --git a/arch/arm64/crypto/aes-neonbs-glue.c b/arch/arm64/crypto/aes-neonbs-glue.c index cb8d90f795a0..933ce70a2504 100644 --- a/arch/arm64/crypto/aes-neonbs-glue.c +++ b/arch/arm64/crypto/aes-neonbs-glue.c @@ -11,13 +11,12 @@ #include #include #include +#include #include #include #include #include -#include "aes-ctr-fallback.h" - MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); @@ -283,13 +282,25 @@ static int aesbs_xts_setkey(struct crypto_skcipher *tfm, const u8 *in_key, return aesbs_setkey(tfm, in_key, key_len); } -static int ctr_encrypt_sync(struct skcipher_request *req) +static void ctr_encrypt_one(struct crypto_skcipher *tfm, const u8 *src, u8 *dst) { - struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct aesbs_ctr_ctx *ctx = crypto_skcipher_ctx(tfm); + unsigned long flags; + + /* + * Temporarily disable interrupts to avoid races where + * evicted when the CPU is interrupted to do something + * else. + */ + local_irq_save(flags); + aes_encrypt(&ctx->fallback, dst, src); + local_irq_restore(flags); +} +static int ctr_encrypt_sync(struct skcipher_request *req) +{ if (!crypto_simd_usable()) - return aes_ctr_encrypt_fallback(&ctx->fallback, req); + return crypto_ctr_encrypt_walk(req, ctr_encrypt_one); return ctr_encrypt(req); } From patchwork Sat Jun 22 19:34:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167501 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2256029ilk; Sat, 22 Jun 2019 12:35:02 -0700 (PDT) X-Google-Smtp-Source: APXvYqyMhF9srwOy9JpJPde2r2pXhrkekcRFyjvNAfkNh0o8SfePynkY2NC2LRx6TmdPelyJtbq9 X-Received: by 2002:a17:902:7883:: with SMTP id q3mr136918627pll.89.1561232102448; Sat, 22 Jun 2019 12:35:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232102; cv=none; d=google.com; s=arc-20160816; b=hMqkLuM8Lb8oPZERUiRGsrmONS0sOjxXmC4/EtLdcPTRdCo5vjCKti7huElNao6lNK kIVWG+L55BnJwuPlu/csfUjlPNxeX9KmBPNrNFSU93M9sBkvv2Gz7Wr6Wu1X8TBgQ7Ep kus1AyJ5IcrS0Bn1CzEQkWMCv/poBbhactBoLvd6IbnZWOT9RjWa2nRQdmghcNNh3qLL D9HQ9rMIjlEk2g2fmAtp9KAXt4Utinzka2snuHHBPNnYhPYhApLyPcc7w5dH0MJiOwk1 Lt9ycXADyqjwAethMNqI05UajK8jRZt/VA0ENaR3p4/7jNw8TvoB8oZREHbbG7/vLE9U L6xw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Lm7oGps4teOEwcLPMuq0r39VG4DYBydnuDjdvE8qRsQ=; b=ZbaEgN9Ng/BrMcsKP5fXYU8Sk+STsqE1WXr4L3+e85J+sZeoWrwjPRlMHb4N+n46UP 5Q1COghHQiL1iSFB1DIeSy4JFRcw87x/o0yyKQOO5AhlI0SmW1RDOIvLS4axYqmNpD0R 0LgK5eGU+/dLQKdMsSA1lI4iXNR2qpST97cVZVloGyIkzaEZ+Qp6e9lS7H8KeAVVGtZP Yg+E99ASDNbuKPKQa8oRUIdRRS3Wx/StQEIrdCMONqvdcQS6j/MCV6ezfIMop/bMlBM7 TuycptoLaQLeLyxc3z2jwzVG+efNfEMx2SI47FTgkewARqVmqQw0Qu8MIgpNcgigGCUs ajZw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=OEI3R8Zd; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.35.02; Sat, 22 Jun 2019 12:35:02 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=OEI3R8Zd; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726382AbfFVTfB (ORCPT + 3 others); Sat, 22 Jun 2019 15:35:01 -0400 Received: from mail-wm1-f68.google.com ([209.85.128.68]:37437 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726368AbfFVTfB (ORCPT ); Sat, 22 Jun 2019 15:35:01 -0400 Received: by mail-wm1-f68.google.com with SMTP id f17so9609959wme.2 for ; Sat, 22 Jun 2019 12:35:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Lm7oGps4teOEwcLPMuq0r39VG4DYBydnuDjdvE8qRsQ=; b=OEI3R8ZdEFOf2IK+/cF7HPZfZj4AkMIKfVOxytWBcOxnlrXK9fTyom7onykz9NbC5C euufNlTWgEZEqS6bgABNwEWRaxXV51aaoGTDogrwcaO4sVP+o8PBLQtc+GBmgRar1KWA HG3AbG1TViD3gypmMAuNDKBuyOC8wmVhw6jgdNzKnM5sVK+LmdqQYrQz3ZePevNJQu1P wZKer/5Ws7/2h92qHpw81y1uwXyrzmvcLyYssd159glKG6uq+21yIDyq2JZiwRZyj4hF EF6DaElXL1TiVDTsZdGHrpvwr1UcITG2hwNWItjHFkBhMaYoYLN8lvPRucTJXV5COkE2 69Ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Lm7oGps4teOEwcLPMuq0r39VG4DYBydnuDjdvE8qRsQ=; b=j1+mg8BGrd1kCk24g6wOnJS6bhW/NvfDLa7KkF1QnU0elkU0UwMQEvDyC99+8RV6j4 ZQpn8DVpbBO75siF/uYXOim2mysL7FlSJlv3isTCkZWPjFoE5VQzyk3QgpC2eUZHPbZI e0dgz0erJ/ESshq6oak6bw/F949kVF+t3a+ID8QIEOyHvwIaEj9ZLu6KYpfwOe4oUw1b Z4CHb4t4hWXo0Ov2at0UhOWM76jysatsqYtSl1UHSK4pymBed3maZUthXMf5nx1yXMsp WXtQtqkh3jCT2vaGoR7awNJXCwyItb7pXSAXGgfgnPx+oznqCdoJxrbvtLiaLuWvmZJy xuwA== X-Gm-Message-State: APjAAAXlO0jo9eaKsbeAvRMma3xuE3N7AN7g95bSBK84OiXJ5fvVtk2c KqAe90CxVMcJi2mOKQuEMe8spe3EaqvZ4kY7 X-Received: by 2002:a1c:452:: with SMTP id 79mr9014805wme.149.1561232099327; Sat, 22 Jun 2019 12:34:59 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:58 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 18/26] crypto: arm64/aes-ce-cipher - use AES library as fallback Date: Sat, 22 Jun 2019 21:34:19 +0200 Message-Id: <20190622193427.20336-19-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Instead of calling into the table based scalar AES code in situations where the SIMD unit may not be used, use the generic AES code, which is more appropriate since it is less likely to be susceptible to timing attacks. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/Kconfig | 2 +- arch/arm64/crypto/aes-ce-glue.c | 7 ++----- arch/arm64/crypto/aes-cipher-glue.c | 3 --- 3 files changed, 3 insertions(+), 9 deletions(-) -- 2.20.1 diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index 66dea518221c..4922c4451e7c 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -73,7 +73,7 @@ config CRYPTO_AES_ARM64_CE tristate "AES core cipher using ARMv8 Crypto Extensions" depends on ARM64 && KERNEL_MODE_NEON select CRYPTO_ALGAPI - select CRYPTO_AES_ARM64 + select CRYPTO_LIB_AES config CRYPTO_AES_ARM64_CE_CCM tristate "AES in CCM mode using ARMv8 Crypto Extensions" diff --git a/arch/arm64/crypto/aes-ce-glue.c b/arch/arm64/crypto/aes-ce-glue.c index 3213843fcb46..6890e003b8f1 100644 --- a/arch/arm64/crypto/aes-ce-glue.c +++ b/arch/arm64/crypto/aes-ce-glue.c @@ -23,9 +23,6 @@ MODULE_DESCRIPTION("Synchronous AES cipher using ARMv8 Crypto Extensions"); MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); -asmlinkage void __aes_arm64_encrypt(u32 *rk, u8 *out, const u8 *in, int rounds); -asmlinkage void __aes_arm64_decrypt(u32 *rk, u8 *out, const u8 *in, int rounds); - struct aes_block { u8 b[AES_BLOCK_SIZE]; }; @@ -54,7 +51,7 @@ static void aes_cipher_encrypt(struct crypto_tfm *tfm, u8 dst[], u8 const src[]) struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm); if (!crypto_simd_usable()) { - __aes_arm64_encrypt(ctx->key_enc, dst, src, num_rounds(ctx)); + aes_encrypt(ctx, dst, src); return; } @@ -68,7 +65,7 @@ static void aes_cipher_decrypt(struct crypto_tfm *tfm, u8 dst[], u8 const src[]) struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm); if (!crypto_simd_usable()) { - __aes_arm64_decrypt(ctx->key_dec, dst, src, num_rounds(ctx)); + aes_decrypt(ctx, dst, src); return; } diff --git a/arch/arm64/crypto/aes-cipher-glue.c b/arch/arm64/crypto/aes-cipher-glue.c index 0e90b06ebcec..bf32cc6489e1 100644 --- a/arch/arm64/crypto/aes-cipher-glue.c +++ b/arch/arm64/crypto/aes-cipher-glue.c @@ -13,10 +13,7 @@ #include asmlinkage void __aes_arm64_encrypt(u32 *rk, u8 *out, const u8 *in, int rounds); -EXPORT_SYMBOL(__aes_arm64_encrypt); - asmlinkage void __aes_arm64_decrypt(u32 *rk, u8 *out, const u8 *in, int rounds); -EXPORT_SYMBOL(__aes_arm64_decrypt); static void aes_arm64_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) { From patchwork Sat Jun 22 19:34:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167503 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2256046ilk; Sat, 22 Jun 2019 12:35:03 -0700 (PDT) X-Google-Smtp-Source: APXvYqwj88tkpqz18uXekWmrCNGDc/psioo18onak+5YQETBggP7Su1LPg3Ux7Mm1s/c7f2XbJvm X-Received: by 2002:a63:d1d:: with SMTP id c29mr14824068pgl.251.1561232103494; Sat, 22 Jun 2019 12:35:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232103; cv=none; d=google.com; s=arc-20160816; b=LKRH7nhlQZ44XRtYWbnvigv4wPFpZiiQVL33IN0fzCXqAjT6JJgXYFL0sae1NmoUJQ KvCW4JHF7nGgm12kOHpp1DWR/ASvOdLQ8QOKBz9a3yM1+9R8RguV8WIRTD64ysvEqHoP /Uf1Y0LBq5j73TcQtiGlU0/+IcxSKlQj+Yf1zAbx+lpZhaFEDjV0GyIQFfMDxV0kPjpj ln2175PD9T672aAtrerLFcGP3UpcCYOChe64VfdgtRkyzBQcX0eT1AnwVWwsPtHTpdwk 97Dxb9RlW99nsxkPOzFt8MKitzImBo04UqPm2+M5FZz+cBF9rfTEc1Nr9NHSecw5fuJ7 UZQw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JgRwk7rV4YDW0Cr7MyntD6TN2vlr6LPUNeCX8j451cs=; b=MWRF9AzfbWD4PA4JIKM1Go71jInnYFemtsBpvI0gTBWJ5hzg9XGwVEp4imYCzmwvKr lhhAZDa9jvWtbCLzsBmdkZ8yDRgznX6CupdHYnrWPrG8vzBDNLMFN1B0bAyXaUR99Lj3 9DSMsox1gG48eu8Dv3VkF1ovsn9ZrvautRaFWlwaIhfAOa110VR4wlWkT8K5enodFuZV 6c5AV4FxRPCS/OmkuaRUBmMdDk16Hu8nfUzrm+AbH/XP5D8a/I8fZ+VXIl45x+nLIWv2 uzewUpIn985WDN1C28waOvkiHKNxhe+/DOV0T4rShcxl6d1C5yp1cb1F+jv2fpHIGG/F jHJA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ge1zLkXR; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.35.03; Sat, 22 Jun 2019 12:35:03 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ge1zLkXR; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726386AbfFVTfC (ORCPT + 3 others); Sat, 22 Jun 2019 15:35:02 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:40983 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726378AbfFVTfC (ORCPT ); Sat, 22 Jun 2019 15:35:02 -0400 Received: by mail-wr1-f68.google.com with SMTP id c2so9704832wrm.8 for ; Sat, 22 Jun 2019 12:35:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JgRwk7rV4YDW0Cr7MyntD6TN2vlr6LPUNeCX8j451cs=; b=ge1zLkXRgySqIvDpzQa7s1BcIusdV6uU80ACw2ZbtUmazBYKJVd93dvlO8TfcUXyaj wCOOI1uyObgEWUIGoAfEfPS11gVaK+M+Uc/7Qcg2mROc8IXhYW/krvQz+CJ2IVlnqjRr AaLb0E1Pk9GtbkUYccRy/Ijt+Dn/RCU1yGtbAYhGWCQBlSfblpUnpR4RuVKPIg1ChnJJ PCtUiKaIZ3zzNZ5Vic/p5cAxEJF3VUc8klrX69eibxSys8S2m1rtJvHsPKejdOaRJADp zZpetAlUlNByQ388Tb9Q15/+MYgqf3bBFMkYBKDSB2jCENZZSVEDJ6d3Xy6npJ03c9A7 +v1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JgRwk7rV4YDW0Cr7MyntD6TN2vlr6LPUNeCX8j451cs=; b=KQkzzlOpEMiaXAhWZDrQcx6WmppPeu63PAe6/i2IQZAGNA0tKqaxlYjp6gwi84p1zI OwAiR1fqEveIw3Tgpcrl3MoEg85SZbiCGulGJrJ2YLz6YN7fD7mN9vuMDS89NYKGRcmI NGy52gJmZHY1ZdRe4VVVNlfpLIfdecFLWQTh7T6ujW2CuJTtZCJ7urY1ERZ9Y6jWw03v w+n69DbruHsa3LeIEa5hkxr1SZoI9wU/3byBByvq4toAACvllSDM0fHqvdw61LjN7heM GGeSBRkcIB7O5IVyMjtbh25+JuLI4bg3h/gOnYVNH7hP1+4YLiTzaeXddbEo19WDRBWK ynyw== X-Gm-Message-State: APjAAAUeu2Nyag6aQb92oebFw2nyWDh83zEDQDl6XkPjzQ7zCM4oWBkh 9WtwnP0h8jxOnIrUdcMiMLTIWkuIF4+qNhx5 X-Received: by 2002:adf:ff90:: with SMTP id j16mr23616872wrr.135.1561232100290; Sat, 22 Jun 2019 12:35:00 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.34.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:34:59 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 19/26] crypto: aes/arm - use native endiannes for key schedule Date: Sat, 22 Jun 2019 21:34:20 +0200 Message-Id: <20190622193427.20336-20-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Align ARM's hw instruction based AES implementation with other versions that keep the key schedule in native endianness. This will allow us to merge the various implementations going forward. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/aes-ce-core.S | 20 ++++++++++---------- arch/arm/crypto/aes-ce-glue.c | 9 +++------ 2 files changed, 13 insertions(+), 16 deletions(-) -- 2.20.1 diff --git a/arch/arm/crypto/aes-ce-core.S b/arch/arm/crypto/aes-ce-core.S index bc53bcaa772e..3692b8735ef7 100644 --- a/arch/arm/crypto/aes-ce-core.S +++ b/arch/arm/crypto/aes-ce-core.S @@ -91,19 +91,19 @@ .macro do_block, dround, fround cmp r3, #12 @ which key size? - vld1.8 {q10-q11}, [ip]! + vld1.32 {q10-q11}, [ip]! \dround q8, q9 - vld1.8 {q12-q13}, [ip]! + vld1.32 {q12-q13}, [ip]! \dround q10, q11 - vld1.8 {q10-q11}, [ip]! + vld1.32 {q10-q11}, [ip]! \dround q12, q13 - vld1.8 {q12-q13}, [ip]! + vld1.32 {q12-q13}, [ip]! \dround q10, q11 blo 0f @ AES-128: 10 rounds - vld1.8 {q10-q11}, [ip]! + vld1.32 {q10-q11}, [ip]! \dround q12, q13 beq 1f @ AES-192: 12 rounds - vld1.8 {q12-q13}, [ip] + vld1.32 {q12-q13}, [ip] \dround q10, q11 0: \fround q12, q13, q14 bx lr @@ -152,8 +152,8 @@ ENDPROC(aes_decrypt_3x) .macro prepare_key, rk, rounds add ip, \rk, \rounds, lsl #4 - vld1.8 {q8-q9}, [\rk] @ load first 2 round keys - vld1.8 {q14}, [ip] @ load last round key + vld1.32 {q8-q9}, [\rk] @ load first 2 round keys + vld1.32 {q14}, [ip] @ load last round key .endm /* @@ -508,8 +508,8 @@ ENDPROC(ce_aes_sub) * operation on round key *src */ ENTRY(ce_aes_invert) - vld1.8 {q0}, [r1] + vld1.32 {q0}, [r1] aesimc.8 q0, q0 - vst1.8 {q0}, [r0] + vst1.32 {q0}, [r0] bx lr ENDPROC(ce_aes_invert) diff --git a/arch/arm/crypto/aes-ce-glue.c b/arch/arm/crypto/aes-ce-glue.c index 04ba66903674..e6da3e30018b 100644 --- a/arch/arm/crypto/aes-ce-glue.c +++ b/arch/arm/crypto/aes-ce-glue.c @@ -10,6 +10,7 @@ #include #include +#include #include #include #include @@ -80,21 +81,17 @@ static int ce_aes_expandkey(struct crypto_aes_ctx *ctx, const u8 *in_key, key_len != AES_KEYSIZE_256) return -EINVAL; - memcpy(ctx->key_enc, in_key, key_len); ctx->key_length = key_len; + for (i = 0; i < kwords; i++) + ctx->key_enc[i] = get_unaligned_le32(in_key + i * sizeof(u32)); kernel_neon_begin(); for (i = 0; i < sizeof(rcon); i++) { u32 *rki = ctx->key_enc + (i * kwords); u32 *rko = rki + kwords; -#ifndef CONFIG_CPU_BIG_ENDIAN rko[0] = ror32(ce_aes_sub(rki[kwords - 1]), 8); rko[0] = rko[0] ^ rki[0] ^ rcon[i]; -#else - rko[0] = rol32(ce_aes_sub(rki[kwords - 1]), 8); - rko[0] = rko[0] ^ rki[0] ^ (rcon[i] << 24); -#endif rko[1] = rko[0] ^ rki[1]; rko[2] = rko[1] ^ rki[2]; rko[3] = rko[2] ^ rki[3]; From patchwork Sat Jun 22 19:34:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167504 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2256070ilk; Sat, 22 Jun 2019 12:35:05 -0700 (PDT) X-Google-Smtp-Source: APXvYqy6RIZY4VjOyUZWCpYi8gDu/8Xw8eYhpmQFIaXwB+Yd7qSqu0kz95j3OLF4deCE73z1x7PV X-Received: by 2002:a17:902:2a27:: with SMTP id i36mr21271940plb.161.1561232105655; Sat, 22 Jun 2019 12:35:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232105; cv=none; d=google.com; s=arc-20160816; b=fxdWsqBhiY+rEN3xcs2UL6BASIkcXaTdnTtgQ8GF0VFhtqIcjcxs1yF9vrecvFRl6f IkGsBQVJ3r/gyKTi+5KGOtfq7r+NBSVZatBiaEN9semNhYB/KH6fgumGtdA0wRXAU8eg r4J00634l0m6jhhPoY+qqK0IhWDGBkqQmRF4rY0hysU52XGmv1V1wbys+ZMy7b2u0gBB X9MYQikGGsZL2Ie4QKko20mc6pZVL/31FrXuBnjDQSNjcgUr60YjrXx48qKzNqwKlEGJ FZkc0DNhh+6qERF7sC6omMsNVYyNMD8ZH2hOW8FVyoMR1ITRaC0jLSFyPG8/aK4vgBzj 3p3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=i5YAml7KVGLr/1+CAScarRaj3NSVjbEGTREd9Bd1Ltw=; b=CWGSadrhBl6t4jeVR66S81M9gLc09YIpmXA/zek13lR+FRvTXTDsbnOFWzHYRxLUoz gtmLCb707xMPXp1nVI6nKlAO/fwj3YEjpuzeu7dCzs0UjjXb8uxLxEHPwycLqd1xok/x 9n95XUAJzVdt8jK9DP6tjSJCk5k1RxYmuI9Rq8uT9lOz9jObj48grPG/xcl0f4d4E2Ce cZVxSph2FOMu+Z/2CCI4hx8SkW8cWZlGRGg7nfB6/7oJbYivvtQtAMeuMwdgV4tSqz7w WIQqwxt6QdsXMYLNibMizcH8vJ7bZcnV68wPiOSgOcEhQP1bucBjtD4hPeCn7w3OBjke rKAQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=MbMFFh0L; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.35.05; Sat, 22 Jun 2019 12:35:05 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=MbMFFh0L; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726388AbfFVTfF (ORCPT + 3 others); Sat, 22 Jun 2019 15:35:05 -0400 Received: from mail-wm1-f67.google.com ([209.85.128.67]:34690 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726286AbfFVTfD (ORCPT ); Sat, 22 Jun 2019 15:35:03 -0400 Received: by mail-wm1-f67.google.com with SMTP id w9so11349209wmd.1 for ; Sat, 22 Jun 2019 12:35:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=i5YAml7KVGLr/1+CAScarRaj3NSVjbEGTREd9Bd1Ltw=; b=MbMFFh0LVBuA2Ka3k502n9AtagHKkTgUoB+sYJXNU6ryLYM9VrunnEJz2HHY+Sng7D 1hWgtcFtTSyeYji16HNVHN9hKNBLRs/iGdQdiTjh0xg1ycfiD3ndE5/Q0ko7EhBSoO5f efkGR/9VeB9g1RYM93GUR8DVnAjXR2DbZw06JxXj2epN7C6xlZHhXFa/lo1+1Q5pnvNT oIJDgMqZPBNsAxGGAGghocZRdGi376hUOy9ZrQFJxfXoPB0NZ8bvluP8Z6+fxgXtcBAt uidDQhxIPb7FyidBTE0hS8wrGWAl8BPOadwJFXsg7Kb9eYbbA30o8THue20ceXTSCQ/u wQyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=i5YAml7KVGLr/1+CAScarRaj3NSVjbEGTREd9Bd1Ltw=; b=o4kgYbz43WaqSdtawcOvf6jv2NsRY/2t6AA71cyZRLTdDP+c6n31vrbZ+quIA1/OWz bVv5rji9hlnaX6hDQiUQO6WaVslglcGXmeVVmTLvjRuJCkwP+/J4Ji2KOE6HREBswVQ7 KpW2hL57lG38Eor6U9Zh853taTwpeHBGL4A1T3IUZMTTlFdcpEfxBU5cC4h/5CZXTl/v tntRWVYHwD0vkdljoJ0vWMtcd/eUh/U3zJD2uKk2d7dE/9t1LQRUHaDGF67MU1Hw+Wce 5C7JayxL02LSTD7RKQ4ttczsoHEutXKSvwFlKOfwCZ1R71mAbyGpnHYCj44y3WD+RgUN klOg== X-Gm-Message-State: APjAAAUH3FxRzWxdvw2mA0O5KrN4Je9Qks4pI/n2KPfdy5tMUrMUMNgd mFQtFiOWYLcjtyOhdDINT2KRzMCx9NuEfPM+ X-Received: by 2002:a7b:cd9a:: with SMTP id y26mr9220661wmj.44.1561232101491; Sat, 22 Jun 2019 12:35:01 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.35.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:35:00 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 20/26] crypto: arm/aes-ce - provide a synchronous version of ctr(aes) Date: Sat, 22 Jun 2019 21:34:21 +0200 Message-Id: <20190622193427.20336-21-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org AES in CTR mode is used by modes such as GCM and CCM, which are often used in contexts where only synchronous ciphers are permitted. So provide a synchronous version of ctr(aes) based on the existing code. This requires a non-SIMD fallback to deal with invocations occurring from a context where SIMD instructions may not be used. We have a helper for this now in the AES library, so wire that up. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/aes-ce-glue.c | 43 ++++++++++++++++++++ 1 file changed, 43 insertions(+) -- 2.20.1 diff --git a/arch/arm/crypto/aes-ce-glue.c b/arch/arm/crypto/aes-ce-glue.c index e6da3e30018b..c3a78c5a5c35 100644 --- a/arch/arm/crypto/aes-ce-glue.c +++ b/arch/arm/crypto/aes-ce-glue.c @@ -10,8 +10,10 @@ #include #include +#include #include #include +#include #include #include #include @@ -289,6 +291,29 @@ static int ctr_encrypt(struct skcipher_request *req) return err; } +static void ctr_encrypt_one(struct crypto_skcipher *tfm, const u8 *src, u8 *dst) +{ + struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); + unsigned long flags; + + /* + * Temporarily disable interrupts to avoid races where + * evicted when the CPU is interrupted to do something + * else. + */ + local_irq_save(flags); + aes_encrypt(ctx, dst, src); + local_irq_restore(flags); +} + +static int ctr_encrypt_sync(struct skcipher_request *req) +{ + if (!crypto_simd_usable()) + return crypto_ctr_encrypt_walk(req, ctr_encrypt_one); + + return ctr_encrypt(req); +} + static int xts_encrypt(struct skcipher_request *req) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); @@ -378,6 +403,21 @@ static struct skcipher_alg aes_algs[] = { { .setkey = ce_aes_setkey, .encrypt = ctr_encrypt, .decrypt = ctr_encrypt, +}, { + .base.cra_name = "ctr(aes)", + .base.cra_driver_name = "ctr-aes-ce-sync", + .base.cra_priority = 300 - 1, + .base.cra_blocksize = 1, + .base.cra_ctxsize = sizeof(struct crypto_aes_ctx), + .base.cra_module = THIS_MODULE, + + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + .chunksize = AES_BLOCK_SIZE, + .setkey = ce_aes_setkey, + .encrypt = ctr_encrypt_sync, + .decrypt = ctr_encrypt_sync, }, { .base.cra_name = "__xts(aes)", .base.cra_driver_name = "__xts-aes-ce", @@ -421,6 +461,9 @@ static int __init aes_init(void) return err; for (i = 0; i < ARRAY_SIZE(aes_algs); i++) { + if (!(aes_algs[i].base.cra_flags & CRYPTO_ALG_INTERNAL)) + continue; + algname = aes_algs[i].base.cra_name + 2; drvname = aes_algs[i].base.cra_driver_name + 2; basename = aes_algs[i].base.cra_driver_name; From patchwork Sat Jun 22 19:34:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167505 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2256078ilk; Sat, 22 Jun 2019 12:35:06 -0700 (PDT) X-Google-Smtp-Source: APXvYqyEk9zFN8meqiRq0cP7C85hK1iijMjgH13z+EnEp2vvRWzyGqQAKcrAAraSh51rr39iQ5Ez X-Received: by 2002:a63:db05:: with SMTP id e5mr24724965pgg.121.1561232105958; Sat, 22 Jun 2019 12:35:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232105; cv=none; d=google.com; s=arc-20160816; b=cX1jZQo07LVDG7SnnT3KvuLBMwd8PZ+8PMCFW253zb+QZpODVgNo3pCeNsFja30LhD 6oj+HwiLBgfZ797S7OCQd7P4ZSGvUWhKcHzKo3MtnmKmAydWVRv7fZ46vDdrudpc+/Z7 aQbkfpOrhNk/nbaXQUJkrS8oqz1qmDIgq+7bXHN7PnplokeACs1tCW17rik91zBKCASL DUsDoOcd8nDyzqVuk9JZH5rrFtdbhTuscWg7my3TK1CHLdER7MdAviXb9knZQ+lZaz/6 H18lVDvqzLvQiTcei9CsvLUlg+HhwSLPKSsAmJDha8dDYwgmyyYWgm/9OLALogGHRYKx WgPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vL6L5hDUn9mhzlJsgchovHgAG3ROBqwVQK1Apo8D4Bw=; b=fWxFkzWEI07n3elpjSD/AI4Dw/UVnfX4tMU+HsZ0ZZcspDrUXz307WCrGwgyEwZQ9K oity7HK0axo+KaEgXm5PmbLF7hWe0/TPkCWcVfBT/FD1sqzYFOoc3Ny4SZDWRVJEwZ7P Ks40B1qMHh38SrZMuej1vwV8CWpWe7UhILry0XhOi7FY672821DDOwKPTwo7ILxHH4OU Ge6oQknume1+Rc5Fx9Wjvy79KrYt26KpfUZZWUa+hL4Qep79pnlQOjPB7FFYCBuNAM7V FW7qRjth1KSVIT1iRhW12K+NzEtheTpamBmR4ZxONCGumHEQomAL1hR6qhuXXxEeZ+Dt fIVw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=WhzvsXaM; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.35.05; Sat, 22 Jun 2019 12:35:05 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=WhzvsXaM; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726286AbfFVTfF (ORCPT + 3 others); Sat, 22 Jun 2019 15:35:05 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:46345 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726378AbfFVTfE (ORCPT ); Sat, 22 Jun 2019 15:35:04 -0400 Received: by mail-wr1-f66.google.com with SMTP id n4so9666050wrw.13 for ; Sat, 22 Jun 2019 12:35:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vL6L5hDUn9mhzlJsgchovHgAG3ROBqwVQK1Apo8D4Bw=; b=WhzvsXaMq4jr9ZNdR2e5Z3+WuuVRSfdwgoJWFzepBHWVIFtw4SJFdFc9Zf+Aj8mfOn Pyu7trs2uhvrJ5c52amgY4SuY/vAvLezoluLtCgbL9M9Cfxugqey5xvibRK1mL+briPO l3BVAOhcl+uSETi8LgvHrawTG5ZOuTAKC7aiMN3OBdpVuQwVgUtbctDkEC4g8SBL5+aH XEIpXFC4Mp5xAdftufoqYvrNqSe+F/WMfIYjD2wyp43lQYJ6VQoIzl9NBLsyR8DGUgSY Dj4FmqNbysPpblbedgJuwhbMgw89ZZDZSX8KdvfbcwE8o1AOLvV3vfj28tF8/e7//kMd Pwew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vL6L5hDUn9mhzlJsgchovHgAG3ROBqwVQK1Apo8D4Bw=; b=quz+NGwl57wjLmSjE3ibq2YT1TJLYWuvAPm87f+KWsVMVk6rzFxQUiWpFxjdcOiGPq OI+Lk1EUJTddmef6MKGofQI6hBbMrXKuc84KpfjNRaL7fGn/t42JRnd+0CuAWibvrKZJ x1NBnoxyYsPQEf5HK+soQfM6EbHYzEV+8CzdYsofxZvSbi7trv8DzaE04zERmLBkSv83 CBs3F3ZoJR9gt9VSkjXOdIiF7JtnB7Pn6cdWYmUZijoNj7lWIT3iaL2ImhGir3spJlxJ xB4O4xWwgZawS7ymPTxTTi96Gn2MpMyqnPXnMSgSZ3eQ8wkTp9Ih4PhI8UKNNlju4g3W 6mww== X-Gm-Message-State: APjAAAVjtMTg3JhDp2WEudxjTX7YQ/Meh5lL72FSqsrbCJfgi9Dgr9Kn Msu7Xhn4i2os4lvFA9e+a00vekR6/bDqRoQ0 X-Received: by 2002:adf:fb81:: with SMTP id a1mr16653991wrr.329.1561232102531; Sat, 22 Jun 2019 12:35:02 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.35.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:35:01 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 21/26] crypto: arm/aes-neonbs - provide a synchronous version of ctr(aes) Date: Sat, 22 Jun 2019 21:34:22 +0200 Message-Id: <20190622193427.20336-22-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org AES in CTR mode is used by modes such as GCM and CCM, which are often used in contexts where only synchronous ciphers are permitted. So provide a synchronous version of ctr(aes) based on the existing code. This requires a non-SIMD fallback to deal with invocations occurring from a context where SIMD instructions may not be used. We have a helper for this now in the AES library, so wire that up. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/aes-neonbs-glue.c | 65 ++++++++++++++++++++ 1 file changed, 65 insertions(+) -- 2.20.1 diff --git a/arch/arm/crypto/aes-neonbs-glue.c b/arch/arm/crypto/aes-neonbs-glue.c index f43c9365b6a9..2f1aa199926c 100644 --- a/arch/arm/crypto/aes-neonbs-glue.c +++ b/arch/arm/crypto/aes-neonbs-glue.c @@ -9,8 +9,10 @@ */ #include +#include #include #include +#include #include #include #include @@ -57,6 +59,11 @@ struct aesbs_xts_ctx { struct crypto_cipher *tweak_tfm; }; +struct aesbs_ctr_ctx { + struct aesbs_ctx key; /* must be first member */ + struct crypto_aes_ctx fallback; +}; + static int aesbs_setkey(struct crypto_skcipher *tfm, const u8 *in_key, unsigned int key_len) { @@ -192,6 +199,25 @@ static void cbc_exit(struct crypto_tfm *tfm) crypto_free_cipher(ctx->enc_tfm); } +static int aesbs_ctr_setkey_sync(struct crypto_skcipher *tfm, const u8 *in_key, + unsigned int key_len) +{ + struct aesbs_ctr_ctx *ctx = crypto_skcipher_ctx(tfm); + int err; + + err = aes_expandkey(&ctx->fallback, in_key, key_len); + if (err) + return err; + + ctx->key.rounds = 6 + key_len / 4; + + kernel_neon_begin(); + aesbs_convert_key(ctx->key.rk, ctx->fallback.key_enc, ctx->key.rounds); + kernel_neon_end(); + + return 0; +} + static int ctr_encrypt(struct skcipher_request *req) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); @@ -234,6 +260,29 @@ static int ctr_encrypt(struct skcipher_request *req) return err; } +static void ctr_encrypt_one(struct crypto_skcipher *tfm, const u8 *src, u8 *dst) +{ + struct aesbs_ctr_ctx *ctx = crypto_skcipher_ctx(tfm); + unsigned long flags; + + /* + * Temporarily disable interrupts to avoid races where + * evicted when the CPU is interrupted to do something + * else. + */ + local_irq_save(flags); + aes_encrypt(&ctx->fallback, dst, src); + local_irq_restore(flags); +} + +static int ctr_encrypt_sync(struct skcipher_request *req) +{ + if (!crypto_simd_usable()) + return crypto_ctr_encrypt_walk(req, ctr_encrypt_one); + + return ctr_encrypt(req); +} + static int aesbs_xts_setkey(struct crypto_skcipher *tfm, const u8 *in_key, unsigned int key_len) { @@ -361,6 +410,22 @@ static struct skcipher_alg aes_algs[] = { { .setkey = aesbs_setkey, .encrypt = ctr_encrypt, .decrypt = ctr_encrypt, +}, { + .base.cra_name = "ctr(aes)", + .base.cra_driver_name = "ctr-aes-neonbs-sync", + .base.cra_priority = 250 - 1, + .base.cra_blocksize = 1, + .base.cra_ctxsize = sizeof(struct aesbs_ctr_ctx), + .base.cra_module = THIS_MODULE, + + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .chunksize = AES_BLOCK_SIZE, + .walksize = 8 * AES_BLOCK_SIZE, + .ivsize = AES_BLOCK_SIZE, + .setkey = aesbs_ctr_setkey_sync, + .encrypt = ctr_encrypt_sync, + .decrypt = ctr_encrypt_sync, }, { .base.cra_name = "__xts(aes)", .base.cra_driver_name = "__xts-aes-neonbs", From patchwork Sat Jun 22 19:34:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167506 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2256090ilk; Sat, 22 Jun 2019 12:35:07 -0700 (PDT) X-Google-Smtp-Source: APXvYqyAVMSTTv7mDg9FjwC8b1jD+vC1hRhs7IPyANplH/vehEYvoFpFHjLMfgfIEya+pvIFVBsk X-Received: by 2002:a17:90a:3688:: with SMTP id t8mr14224803pjb.35.1561232106946; Sat, 22 Jun 2019 12:35:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232106; cv=none; d=google.com; s=arc-20160816; b=WoH2LtAz4jgWAcJ97bF8Y/IfF7GkRWWC7KH8kMNviFIAhKLbXouEBoo5GGulTwuFF8 +4v4U5/9RZqcxF5ICX3xFKRDGmHdfRFscBTha+C8SRxD1wkVewiCIG7cDcJVlwZfFI1g jgP+uIWiA83B8mFdjTporj/Fl9QzrWM7Zdt45XoJ41k6qH2YUWNp5RycGK8k1+9I4vo/ sPl+sKxkYBt3rOZynXh//AkoOaKbtAfwc3KvPSlgq9BbkE7XI11WA5A/hHiEJVBx3Ikq dnG2sD4N2wnDL+1N4LEATPxEosYiTJJjbdxQlM5jv+l5cOAi4MO0WA0+EZKtAQKHXiP+ R/4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0vLqup8kXDSW4TBicl2CoUwy3k2oxTX/PfRrsOfcNio=; b=GCGFd8eVFw3jY5Dwbflh2zSslo9H+i6wxdRt/aRdhTKDbE/nJCiwXg0fUp7/Q9uU7T MwwdoZgoNilvHOJHgio6x302kEuezSp8+laQ6xm+67vHI7uSwS+Xk/p8BIS7cc4WvVGS MNeXJMEMgVUrj37k6NLKNnmhJHJgf8CEvALnP8e4UAWW+ypqiJdNZsNpnzZ93dc06MBV 3ikJw6bl9PFi0xVIWZ6uJSwxT9Wv37q8in5sNotln9yYvxibErvPhuOUEn96ty6cUP/1 vmIeQOiEImmy7jREPgfmc9Bckp00Y6rdv0XFwVb9h9UuNsvdXx3vQ6j1OGceFFz7+QxZ IgPg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=pwWgggii; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.35.06; Sat, 22 Jun 2019 12:35:06 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=pwWgggii; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726393AbfFVTfG (ORCPT + 3 others); Sat, 22 Jun 2019 15:35:06 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:43964 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726379AbfFVTfF (ORCPT ); Sat, 22 Jun 2019 15:35:05 -0400 Received: by mail-wr1-f66.google.com with SMTP id p13so9694169wru.10 for ; Sat, 22 Jun 2019 12:35:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0vLqup8kXDSW4TBicl2CoUwy3k2oxTX/PfRrsOfcNio=; b=pwWgggiiOEiZv9fan738/o6lf5f+wRxptxIdjGbdWs2OXka6l1N360LFxunvx5riBc upxwz35BO5ywJ9gZ1W2ZgcisKy/CgOvILPLYWwX/eiV1ePBTpK+05YLs/KO7Iyq1Sf/E Qb5SFdkXQHegXp8L3OUNFgPJgcZb0Qh1IDy3p2/Wt4l5GAu5IoQPWn7/s0bQh7PJu+Uh 4L7UsycxCjaD8Eiyf5zcG5enYYAyzkNy8er178QW32256s4bFgE41HgmCXKHksoCo5cz SLum4nL/s+KNLPqgMF6wfCDKXM8zN0Hg3twrU1bBom2BU+ooL5B73TA0Cwp2L6p5hNfd vzHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0vLqup8kXDSW4TBicl2CoUwy3k2oxTX/PfRrsOfcNio=; b=uKvOLs28OKNrhfPJqvhm5ItlzrqUq3Ehl678DJY8qwy9+nhTK8QdQOMtjYthiYTYTR vDqLqUQy47KRlf9zinl79XQDj2FgV44nCASUJe4c+vsT91jRke4nrn9yfJIbKSII8vOr zHewz3O5CQyhwLdoOrC9kH9cR9zyFjm0A0Vif/Z4vJhf4PorC046q4sytnpc5ctTN/c2 0rv7H3U5kIQAGRzktz32s6EpfFgc5hxs0SiyY3mYBccQur17INybMtt14/aO4pv7I7kI UdYZW1L5XDZRjXV7NbPKM4B98bVgNx9tIrkwWnBgFwg6X9aTeYVJ4dDzwwSIjm9CCz1o vteg== X-Gm-Message-State: APjAAAUXl17m2Byai7AXVrreBV9Xwdyz0D6k4sb9VbFUwFsG93Vmozum 9CLZeZKsr1LZFmsoQZ1C91gRhr6lBnqUL8yO X-Received: by 2002:adf:dcc2:: with SMTP id x2mr61601195wrm.55.1561232103495; Sat, 22 Jun 2019 12:35:03 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.35.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:35:02 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 22/26] crypto: arm/ghash - provide a synchronous version Date: Sat, 22 Jun 2019 21:34:23 +0200 Message-Id: <20190622193427.20336-23-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org GHASH is used by the GCM mode, which is often used in contexts where only synchronous ciphers are permitted. So provide a synchronous version of GHASH based on the existing code. This requires a non-SIMD fallback to deal with invocations occurring from a context where SIMD instructions may not be used. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/ghash-ce-glue.c | 78 +++++++++++++------- 1 file changed, 52 insertions(+), 26 deletions(-) -- 2.20.1 diff --git a/arch/arm/crypto/ghash-ce-glue.c b/arch/arm/crypto/ghash-ce-glue.c index 39d1ccec1aab..ebb237ca874b 100644 --- a/arch/arm/crypto/ghash-ce-glue.c +++ b/arch/arm/crypto/ghash-ce-glue.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -33,6 +34,8 @@ struct ghash_key { u64 h2[2]; u64 h3[2]; u64 h4[2]; + + be128 k; }; struct ghash_desc_ctx { @@ -65,6 +68,36 @@ static int ghash_init(struct shash_desc *desc) return 0; } +static void ghash_do_update(int blocks, u64 dg[], const char *src, + struct ghash_key *key, const char *head) +{ + if (likely(crypto_simd_usable())) { + kernel_neon_begin(); + pmull_ghash_update(blocks, dg, src, key, head); + kernel_neon_end(); + } else { + be128 dst = { cpu_to_be64(dg[1]), cpu_to_be64(dg[0]) }; + + do { + const u8 *in = src; + + if (head) { + in = head; + blocks++; + head = NULL; + } else { + src += GHASH_BLOCK_SIZE; + } + + crypto_xor((u8 *)&dst, in, GHASH_BLOCK_SIZE); + gf128mul_lle(&dst, &key->k); + } while (--blocks); + + dg[0] = be64_to_cpu(dst.b); + dg[1] = be64_to_cpu(dst.a); + } +} + static int ghash_update(struct shash_desc *desc, const u8 *src, unsigned int len) { @@ -88,10 +121,8 @@ static int ghash_update(struct shash_desc *desc, const u8 *src, blocks = len / GHASH_BLOCK_SIZE; len %= GHASH_BLOCK_SIZE; - kernel_neon_begin(); - pmull_ghash_update(blocks, ctx->digest, src, key, - partial ? ctx->buf : NULL); - kernel_neon_end(); + ghash_do_update(blocks, ctx->digest, src, key, + partial ? ctx->buf : NULL); src += blocks * GHASH_BLOCK_SIZE; partial = 0; } @@ -109,9 +140,7 @@ static int ghash_final(struct shash_desc *desc, u8 *dst) struct ghash_key *key = crypto_shash_ctx(desc->tfm); memset(ctx->buf + partial, 0, GHASH_BLOCK_SIZE - partial); - kernel_neon_begin(); - pmull_ghash_update(1, ctx->digest, ctx->buf, key, NULL); - kernel_neon_end(); + ghash_do_update(1, ctx->digest, ctx->buf, key, NULL); } put_unaligned_be64(ctx->digest[1], dst); put_unaligned_be64(ctx->digest[0], dst + 8); @@ -135,24 +164,25 @@ static int ghash_setkey(struct crypto_shash *tfm, const u8 *inkey, unsigned int keylen) { struct ghash_key *key = crypto_shash_ctx(tfm); - be128 h, k; + be128 h; if (keylen != GHASH_BLOCK_SIZE) { crypto_shash_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN); return -EINVAL; } - memcpy(&k, inkey, GHASH_BLOCK_SIZE); - ghash_reflect(key->h, &k); + /* needed for the fallback */ + memcpy(&key->k, inkey, GHASH_BLOCK_SIZE); + ghash_reflect(key->h, &key->k); - h = k; - gf128mul_lle(&h, &k); + h = key->k; + gf128mul_lle(&h, &key->k); ghash_reflect(key->h2, &h); - gf128mul_lle(&h, &k); + gf128mul_lle(&h, &key->k); ghash_reflect(key->h3, &h); - gf128mul_lle(&h, &k); + gf128mul_lle(&h, &key->k); ghash_reflect(key->h4, &h); return 0; @@ -165,15 +195,13 @@ static struct shash_alg ghash_alg = { .final = ghash_final, .setkey = ghash_setkey, .descsize = sizeof(struct ghash_desc_ctx), - .base = { - .cra_name = "__ghash", - .cra_driver_name = "__driver-ghash-ce", - .cra_priority = 0, - .cra_flags = CRYPTO_ALG_INTERNAL, - .cra_blocksize = GHASH_BLOCK_SIZE, - .cra_ctxsize = sizeof(struct ghash_key), - .cra_module = THIS_MODULE, - }, + + .base.cra_name = "ghash", + .base.cra_driver_name = "ghash-ce-sync", + .base.cra_priority = 300 - 1, + .base.cra_blocksize = GHASH_BLOCK_SIZE, + .base.cra_ctxsize = sizeof(struct ghash_key), + .base.cra_module = THIS_MODULE, }; static int ghash_async_init(struct ahash_request *req) @@ -288,9 +316,7 @@ static int ghash_async_init_tfm(struct crypto_tfm *tfm) struct cryptd_ahash *cryptd_tfm; struct ghash_async_ctx *ctx = crypto_tfm_ctx(tfm); - cryptd_tfm = cryptd_alloc_ahash("__driver-ghash-ce", - CRYPTO_ALG_INTERNAL, - CRYPTO_ALG_INTERNAL); + cryptd_tfm = cryptd_alloc_ahash("ghash-ce-sync", 0, 0); if (IS_ERR(cryptd_tfm)) return PTR_ERR(cryptd_tfm); ctx->cryptd_tfm = cryptd_tfm; From patchwork Sat Jun 22 19:34:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167508 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2256117ilk; Sat, 22 Jun 2019 12:35:09 -0700 (PDT) X-Google-Smtp-Source: APXvYqzC3P+BavQIYGgKERvw+6Uqa6FmsYdcB6jlI6EuUDV9cMqndOSYgu2ve10nw4b2WO3ZO/aA X-Received: by 2002:a65:518d:: with SMTP id h13mr24428183pgq.22.1561232109615; Sat, 22 Jun 2019 12:35:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232109; cv=none; d=google.com; s=arc-20160816; b=t8Gy0c9AmerxsNS0hJyjoFmHEKlQvAR0HZBTZv+OC4KNl2JcS6oZA87xfrWVMuEjvy jkz1tV7jk1spJGis8KKcX4+QwXAIXvCcQFMajuGPDQ7Px0b3+bBNveV9k814XCor6LUU lMlEQ0VI/Pp9cl1FsT9IueUrQCVwqfb5nWNuyzzWy4Jt+p8LMRKVJTITysKCOJJSiyK2 oN092XmXVR7UnpCXnI20aOceg7iHeFy0riiOgNze+oUPlcnYjfGj5Cc1HIP5WgUxkR63 Zud+EWXe7BrfVqqNGyEbKyd3FHzCL1m0CxPrwUNjQYuGGFTvEmC08TLtLW6OvfNrH1oe Coyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Q96/29TdV0TBH+QbXhY2WMZ256VAP+kFBl0pIhaOEg0=; b=M5SDvqMf4ciZ0JKSsmUXheAG7ju8+NAkKdFIIyRAkw+cCHdWcEfy3FpYMfyedQt5f7 GZnFsiwfFio4FrGp0atoQq2OFKTn3fO2k3bjhV7wQ+zO4Vjhh9i2xHxJf910DCKiLIrD GSndZWcddWc3WEukEHb70FHYUtLqN6X3/8bNFLe3+Y37dTWwBktxNFEWrXjPPcgPox05 zlNRXILBnXdWpIJLE9MFI36L7b1wU5q8/fvm9i8XPqr7p1hr7chPeFnesMgtBLKtjLXT doLcLoOXAcfrt6h6BJuhdVdli2gUyfBpH0em1M+i4npZ5X7162cJOAsOPHC0DR79WODn qWFg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ZHd1HwMs; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.35.09; Sat, 22 Jun 2019 12:35:09 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ZHd1HwMs; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726401AbfFVTfI (ORCPT + 3 others); Sat, 22 Jun 2019 15:35:08 -0400 Received: from mail-wm1-f68.google.com ([209.85.128.68]:37443 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726378AbfFVTfI (ORCPT ); Sat, 22 Jun 2019 15:35:08 -0400 Received: by mail-wm1-f68.google.com with SMTP id f17so9610064wme.2 for ; Sat, 22 Jun 2019 12:35:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Q96/29TdV0TBH+QbXhY2WMZ256VAP+kFBl0pIhaOEg0=; b=ZHd1HwMsJylHn/YpXFYKNv//iCkHvOZ/ZkkjnhVQ46djDb5pGASPbvibq/jrM4B5ta ropwSSc9L0y/Q+2Fo4M4ZRIztdNGqYzyk1BWJCWr7z2KExgCcPPJUQuvAEqlc9MSOWWK fAexdJcGfrKuS7lZwqUxlgFPohLs1C4nvXmNkT3uRY3GRQCaS+4O5kLhq6G9mjX85m81 rh+jkHr6Sw9wTNpLYT93PAy3KjGbQMRmgSk759pKwhcs4WlaHIk11bXeKnnXfFZsw2gj TKerCHwcid/sSfNj2r/KZ6WEAnm7MpqkQOByV7m7KfM7yJ/JWq9ht4DOw+fd4oM0b796 koGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Q96/29TdV0TBH+QbXhY2WMZ256VAP+kFBl0pIhaOEg0=; b=ld1K9y8BzPEBkuT7Nqn5wwmmuswF4oN3eyI4EGzCb3ALDMeY+9ybXrm5E4NeikU4TH P1GFXeSZQ+EWC3b7DIK5YQgsJsYKXyIlYzAqQamKwBOjpa9SYfy72KCAyHJpSBsrmQ7e aLOdGHTzkB4YAA2WC/40rDMAfYWJFBMnnggdxZ1yMiswcEpiNoZ2NlKWldjO28G8ohMw Qcj28fPYg7oZo3RNEzdQgrrvj2IRgqWW8Yr5jItw6i5RZP6jI+hm25rg2wCDaxJmzwA3 bjPvMRVrLFN87O3JAlUuztNSR7ccXAk1ur3GtiZGVaiNCVXeFSff4LFWxTDNdiWvZ2BJ qhBQ== X-Gm-Message-State: APjAAAWXehByHZV0tuzW+hwZcxt3x2fB6UHhoBM8sw+ckqJof8MOcjfN jYJfxW6ogxgF5nm47HHTJKa01s3tEM2L6FS8 X-Received: by 2002:a7b:cb84:: with SMTP id m4mr9340198wmi.50.1561232104489; Sat, 22 Jun 2019 12:35:04 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.35.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:35:03 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 23/26] bluetooth: switch to AES library Date: Sat, 22 Jun 2019 21:34:24 +0200 Message-Id: <20190622193427.20336-24-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The bluetooth code uses a bare AES cipher for the encryption operations. Given that it carries out a set_key() operation right before every encryption operation, this is clearly not a hot path, and so the use of the cipher interface (which provides the best implementation available on the system) is not really required. In fact, when using a cipher like AES-NI or AES-CE, both the set_key() and the encrypt() operations involve en/disabling preemption as well as stacking and unstacking the SIMD context, and this is most certainly not worth it for encrypting 16 bytes of data. So let's switch to the new lightweight library interface instead. Signed-off-by: Ard Biesheuvel --- net/bluetooth/Kconfig | 3 +- net/bluetooth/smp.c | 103 ++++++-------------- 2 files changed, 33 insertions(+), 73 deletions(-) -- 2.20.1 diff --git a/net/bluetooth/Kconfig b/net/bluetooth/Kconfig index db82a40875e8..a9d83ec4ee33 100644 --- a/net/bluetooth/Kconfig +++ b/net/bluetooth/Kconfig @@ -9,7 +9,8 @@ menuconfig BT select CRC16 select CRYPTO select CRYPTO_BLKCIPHER - select CRYPTO_AES + select CRYPTO_LIB_AES + imply CRYPTO_AES select CRYPTO_CMAC select CRYPTO_ECB select CRYPTO_SHA256 diff --git a/net/bluetooth/smp.c b/net/bluetooth/smp.c index e68c715f8d37..b5045b57ead3 100644 --- a/net/bluetooth/smp.c +++ b/net/bluetooth/smp.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -88,7 +89,6 @@ struct smp_dev { u8 local_rand[16]; bool debug_key; - struct crypto_cipher *tfm_aes; struct crypto_shash *tfm_cmac; struct crypto_kpp *tfm_ecdh; }; @@ -127,7 +127,6 @@ struct smp_chan { u8 dhkey[32]; u8 mackey[16]; - struct crypto_cipher *tfm_aes; struct crypto_shash *tfm_cmac; struct crypto_kpp *tfm_ecdh; }; @@ -377,22 +376,18 @@ static int smp_h7(struct crypto_shash *tfm_cmac, const u8 w[16], * s1 and ah. */ -static int smp_e(struct crypto_cipher *tfm, const u8 *k, u8 *r) +static int smp_e(const u8 *k, u8 *r) { + struct crypto_aes_ctx ctx; uint8_t tmp[16], data[16]; int err; SMP_DBG("k %16phN r %16phN", k, r); - if (!tfm) { - BT_ERR("tfm %p", tfm); - return -EINVAL; - } - /* The most significant octet of key corresponds to k[0] */ swap_buf(k, tmp, 16); - err = crypto_cipher_setkey(tfm, tmp, 16); + err = aes_expandkey(&ctx, tmp, 16); if (err) { BT_ERR("cipher setkey failed: %d", err); return err; @@ -401,17 +396,18 @@ static int smp_e(struct crypto_cipher *tfm, const u8 *k, u8 *r) /* Most significant octet of plaintextData corresponds to data[0] */ swap_buf(r, data, 16); - crypto_cipher_encrypt_one(tfm, data, data); + aes_encrypt(&ctx, data, data); /* Most significant octet of encryptedData corresponds to data[0] */ swap_buf(data, r, 16); SMP_DBG("r %16phN", r); + memzero_explicit(&ctx, sizeof (ctx)); return err; } -static int smp_c1(struct crypto_cipher *tfm_aes, const u8 k[16], +static int smp_c1(const u8 k[16], const u8 r[16], const u8 preq[7], const u8 pres[7], u8 _iat, const bdaddr_t *ia, u8 _rat, const bdaddr_t *ra, u8 res[16]) { @@ -436,7 +432,7 @@ static int smp_c1(struct crypto_cipher *tfm_aes, const u8 k[16], u128_xor((u128 *) res, (u128 *) r, (u128 *) p1); /* res = e(k, res) */ - err = smp_e(tfm_aes, k, res); + err = smp_e(k, res); if (err) { BT_ERR("Encrypt data error"); return err; @@ -453,14 +449,14 @@ static int smp_c1(struct crypto_cipher *tfm_aes, const u8 k[16], u128_xor((u128 *) res, (u128 *) res, (u128 *) p2); /* res = e(k, res) */ - err = smp_e(tfm_aes, k, res); + err = smp_e(k, res); if (err) BT_ERR("Encrypt data error"); return err; } -static int smp_s1(struct crypto_cipher *tfm_aes, const u8 k[16], +static int smp_s1(const u8 k[16], const u8 r1[16], const u8 r2[16], u8 _r[16]) { int err; @@ -469,15 +465,14 @@ static int smp_s1(struct crypto_cipher *tfm_aes, const u8 k[16], memcpy(_r, r2, 8); memcpy(_r + 8, r1, 8); - err = smp_e(tfm_aes, k, _r); + err = smp_e(k, _r); if (err) BT_ERR("Encrypt data error"); return err; } -static int smp_ah(struct crypto_cipher *tfm, const u8 irk[16], - const u8 r[3], u8 res[3]) +static int smp_ah(const u8 irk[16], const u8 r[3], u8 res[3]) { u8 _res[16]; int err; @@ -486,7 +481,7 @@ static int smp_ah(struct crypto_cipher *tfm, const u8 irk[16], memcpy(_res, r, 3); memset(_res + 3, 0, 13); - err = smp_e(tfm, irk, _res); + err = smp_e(irk, _res); if (err) { BT_ERR("Encrypt error"); return err; @@ -518,7 +513,7 @@ bool smp_irk_matches(struct hci_dev *hdev, const u8 irk[16], BT_DBG("RPA %pMR IRK %*phN", bdaddr, 16, irk); - err = smp_ah(smp->tfm_aes, irk, &bdaddr->b[3], hash); + err = smp_ah(irk, &bdaddr->b[3], hash); if (err) return false; @@ -541,7 +536,7 @@ int smp_generate_rpa(struct hci_dev *hdev, const u8 irk[16], bdaddr_t *rpa) rpa->b[5] &= 0x3f; /* Clear two most significant bits */ rpa->b[5] |= 0x40; /* Set second most significant bit */ - err = smp_ah(smp->tfm_aes, irk, &rpa->b[3], rpa->b); + err = smp_ah(irk, &rpa->b[3], rpa->b); if (err < 0) return err; @@ -768,7 +763,6 @@ static void smp_chan_destroy(struct l2cap_conn *conn) kzfree(smp->slave_csrk); kzfree(smp->link_key); - crypto_free_cipher(smp->tfm_aes); crypto_free_shash(smp->tfm_cmac); crypto_free_kpp(smp->tfm_ecdh); @@ -957,7 +951,7 @@ static u8 smp_confirm(struct smp_chan *smp) BT_DBG("conn %p", conn); - ret = smp_c1(smp->tfm_aes, smp->tk, smp->prnd, smp->preq, smp->prsp, + ret = smp_c1(smp->tk, smp->prnd, smp->preq, smp->prsp, conn->hcon->init_addr_type, &conn->hcon->init_addr, conn->hcon->resp_addr_type, &conn->hcon->resp_addr, cp.confirm_val); @@ -983,12 +977,9 @@ static u8 smp_random(struct smp_chan *smp) u8 confirm[16]; int ret; - if (IS_ERR_OR_NULL(smp->tfm_aes)) - return SMP_UNSPECIFIED; - BT_DBG("conn %p %s", conn, conn->hcon->out ? "master" : "slave"); - ret = smp_c1(smp->tfm_aes, smp->tk, smp->rrnd, smp->preq, smp->prsp, + ret = smp_c1(smp->tk, smp->rrnd, smp->preq, smp->prsp, hcon->init_addr_type, &hcon->init_addr, hcon->resp_addr_type, &hcon->resp_addr, confirm); if (ret) @@ -1005,7 +996,7 @@ static u8 smp_random(struct smp_chan *smp) __le64 rand = 0; __le16 ediv = 0; - smp_s1(smp->tfm_aes, smp->tk, smp->rrnd, smp->prnd, stk); + smp_s1(smp->tk, smp->rrnd, smp->prnd, stk); if (test_and_set_bit(HCI_CONN_ENCRYPT_PEND, &hcon->flags)) return SMP_UNSPECIFIED; @@ -1021,7 +1012,7 @@ static u8 smp_random(struct smp_chan *smp) smp_send_cmd(conn, SMP_CMD_PAIRING_RANDOM, sizeof(smp->prnd), smp->prnd); - smp_s1(smp->tfm_aes, smp->tk, smp->prnd, smp->rrnd, stk); + smp_s1(smp->tk, smp->prnd, smp->rrnd, stk); if (hcon->pending_sec_level == BT_SECURITY_HIGH) auth = 1; @@ -1389,16 +1380,10 @@ static struct smp_chan *smp_chan_create(struct l2cap_conn *conn) if (!smp) return NULL; - smp->tfm_aes = crypto_alloc_cipher("aes", 0, 0); - if (IS_ERR(smp->tfm_aes)) { - BT_ERR("Unable to create AES crypto context"); - goto zfree_smp; - } - smp->tfm_cmac = crypto_alloc_shash("cmac(aes)", 0, 0); if (IS_ERR(smp->tfm_cmac)) { BT_ERR("Unable to create CMAC crypto context"); - goto free_cipher; + goto zfree_smp; } smp->tfm_ecdh = crypto_alloc_kpp("ecdh", CRYPTO_ALG_INTERNAL, 0); @@ -1420,8 +1405,6 @@ static struct smp_chan *smp_chan_create(struct l2cap_conn *conn) free_shash: crypto_free_shash(smp->tfm_cmac); -free_cipher: - crypto_free_cipher(smp->tfm_aes); zfree_smp: kzfree(smp); return NULL; @@ -3219,7 +3202,6 @@ static struct l2cap_chan *smp_add_cid(struct hci_dev *hdev, u16 cid) { struct l2cap_chan *chan; struct smp_dev *smp; - struct crypto_cipher *tfm_aes; struct crypto_shash *tfm_cmac; struct crypto_kpp *tfm_ecdh; @@ -3232,17 +3214,9 @@ static struct l2cap_chan *smp_add_cid(struct hci_dev *hdev, u16 cid) if (!smp) return ERR_PTR(-ENOMEM); - tfm_aes = crypto_alloc_cipher("aes", 0, 0); - if (IS_ERR(tfm_aes)) { - BT_ERR("Unable to create AES crypto context"); - kzfree(smp); - return ERR_CAST(tfm_aes); - } - tfm_cmac = crypto_alloc_shash("cmac(aes)", 0, 0); if (IS_ERR(tfm_cmac)) { BT_ERR("Unable to create CMAC crypto context"); - crypto_free_cipher(tfm_aes); kzfree(smp); return ERR_CAST(tfm_cmac); } @@ -3251,13 +3225,11 @@ static struct l2cap_chan *smp_add_cid(struct hci_dev *hdev, u16 cid) if (IS_ERR(tfm_ecdh)) { BT_ERR("Unable to create ECDH crypto context"); crypto_free_shash(tfm_cmac); - crypto_free_cipher(tfm_aes); kzfree(smp); return ERR_CAST(tfm_ecdh); } smp->local_oob = false; - smp->tfm_aes = tfm_aes; smp->tfm_cmac = tfm_cmac; smp->tfm_ecdh = tfm_ecdh; @@ -3265,7 +3237,6 @@ static struct l2cap_chan *smp_add_cid(struct hci_dev *hdev, u16 cid) chan = l2cap_chan_create(); if (!chan) { if (smp) { - crypto_free_cipher(smp->tfm_aes); crypto_free_shash(smp->tfm_cmac); crypto_free_kpp(smp->tfm_ecdh); kzfree(smp); @@ -3313,7 +3284,6 @@ static void smp_del_chan(struct l2cap_chan *chan) smp = chan->data; if (smp) { chan->data = NULL; - crypto_free_cipher(smp->tfm_aes); crypto_free_shash(smp->tfm_cmac); crypto_free_kpp(smp->tfm_ecdh); kzfree(smp); @@ -3569,7 +3539,7 @@ static int __init test_debug_key(struct crypto_kpp *tfm_ecdh) return 0; } -static int __init test_ah(struct crypto_cipher *tfm_aes) +static int __init test_ah(void) { const u8 irk[16] = { 0x9b, 0x7d, 0x39, 0x0a, 0xa6, 0x10, 0x10, 0x34, @@ -3579,7 +3549,7 @@ static int __init test_ah(struct crypto_cipher *tfm_aes) u8 res[3]; int err; - err = smp_ah(tfm_aes, irk, r, res); + err = smp_ah(irk, r, res); if (err) return err; @@ -3589,7 +3559,7 @@ static int __init test_ah(struct crypto_cipher *tfm_aes) return 0; } -static int __init test_c1(struct crypto_cipher *tfm_aes) +static int __init test_c1(void) { const u8 k[16] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, @@ -3609,7 +3579,7 @@ static int __init test_c1(struct crypto_cipher *tfm_aes) u8 res[16]; int err; - err = smp_c1(tfm_aes, k, r, preq, pres, _iat, &ia, _rat, &ra, res); + err = smp_c1(k, r, preq, pres, _iat, &ia, _rat, &ra, res); if (err) return err; @@ -3619,7 +3589,7 @@ static int __init test_c1(struct crypto_cipher *tfm_aes) return 0; } -static int __init test_s1(struct crypto_cipher *tfm_aes) +static int __init test_s1(void) { const u8 k[16] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, @@ -3634,7 +3604,7 @@ static int __init test_s1(struct crypto_cipher *tfm_aes) u8 res[16]; int err; - err = smp_s1(tfm_aes, k, r1, r2, res); + err = smp_s1(k, r1, r2, res); if (err) return err; @@ -3815,8 +3785,7 @@ static const struct file_operations test_smp_fops = { .llseek = default_llseek, }; -static int __init run_selftests(struct crypto_cipher *tfm_aes, - struct crypto_shash *tfm_cmac, +static int __init run_selftests(struct crypto_shash *tfm_cmac, struct crypto_kpp *tfm_ecdh) { ktime_t calltime, delta, rettime; @@ -3831,19 +3800,19 @@ static int __init run_selftests(struct crypto_cipher *tfm_aes, goto done; } - err = test_ah(tfm_aes); + err = test_ah(); if (err) { BT_ERR("smp_ah test failed"); goto done; } - err = test_c1(tfm_aes); + err = test_c1(); if (err) { BT_ERR("smp_c1 test failed"); goto done; } - err = test_s1(tfm_aes); + err = test_s1(); if (err) { BT_ERR("smp_s1 test failed"); goto done; @@ -3900,21 +3869,13 @@ static int __init run_selftests(struct crypto_cipher *tfm_aes, int __init bt_selftest_smp(void) { - struct crypto_cipher *tfm_aes; struct crypto_shash *tfm_cmac; struct crypto_kpp *tfm_ecdh; int err; - tfm_aes = crypto_alloc_cipher("aes", 0, 0); - if (IS_ERR(tfm_aes)) { - BT_ERR("Unable to create AES crypto context"); - return PTR_ERR(tfm_aes); - } - tfm_cmac = crypto_alloc_shash("cmac(aes)", 0, 0); if (IS_ERR(tfm_cmac)) { BT_ERR("Unable to create CMAC crypto context"); - crypto_free_cipher(tfm_aes); return PTR_ERR(tfm_cmac); } @@ -3922,14 +3883,12 @@ int __init bt_selftest_smp(void) if (IS_ERR(tfm_ecdh)) { BT_ERR("Unable to create ECDH crypto context"); crypto_free_shash(tfm_cmac); - crypto_free_cipher(tfm_aes); return PTR_ERR(tfm_ecdh); } - err = run_selftests(tfm_aes, tfm_cmac, tfm_ecdh); + err = run_selftests(tfm_cmac, tfm_ecdh); crypto_free_shash(tfm_cmac); - crypto_free_cipher(tfm_aes); crypto_free_kpp(tfm_ecdh); return err; From patchwork Sat Jun 22 19:34:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167507 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2256106ilk; Sat, 22 Jun 2019 12:35:08 -0700 (PDT) X-Google-Smtp-Source: APXvYqwOgZj2Oi7tLLlmyY+4nK28dn+Oc7QftRscvEmdZv8A3Igor/q4fv3fJ5D26o1tlYHC4H0w X-Received: by 2002:a17:90a:37c8:: with SMTP id v66mr14955054pjb.33.1561232108517; Sat, 22 Jun 2019 12:35:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232108; cv=none; d=google.com; s=arc-20160816; b=JQkSWW+muV9YeYgT4a50YRnrauKgaDxtFOzDXXrvNnm081EWCUb8aZDsv4iuccxw6i WBfsM73MD7JmMlZO1Ur4yEudmyQtfZvx6jfRPaqs7t9ELQzX+5coGACh/5Om/OASNhJA n12Iso//3hRxr8ttlfw8YRMsgmMyvY5Sap3ThhFd6wjOmaE3UhR+OYIOjnDQ8dn5XlWM jgPIwTSlWVEzVQ67kcO6jnpyopjKR8AhUSCJkpgE3CD8s/HrTF3YYa4Nke6pkyIS/rf2 cvZZaH/FfYzL3yKatsDfGUXQwBxVGYb+5Cd3SIUW5z6vIR7F5i4oucWngN2jMKMKIxlL JHmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=h9xdExBrqYFWsuLQCJksPQkQTr4z/marqcQjFd4StkE=; b=vPi168HSws+raaMAl/zLKem1Uk3p0ui3QYfWPjkKehNRwTCcYq9D7Z1hwemgV/Wl8s Emz3ea85ZDOHEaw7TQplg5MEqAWkJxQdrBKC2kajLpoFTVyIvJDAfs85QGQkG+sa1dnZ IRCFTgwus5QoPySOXcbpWGjbhgn5BYqGuDSkQl2wykAMzLWxZURkiONIRW9Bn0B9zqs4 UWwEw23TnAnzkot7wL9DF2VZ8YBkQg1MMTlw4Q2y0Nodrr6wlYT1QJL81R6sD6DNiCdN 9XWdkbtOSAarAgwIZznMNtXR2uY2ZdCxAN2B6ZH89hXN9nMz64/zcjhHsZBEFlPgQR67 OsQw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=PsBUE0nV; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.35.08; Sat, 22 Jun 2019 12:35:08 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=PsBUE0nV; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726397AbfFVTfH (ORCPT + 3 others); Sat, 22 Jun 2019 15:35:07 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:39432 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726379AbfFVTfH (ORCPT ); Sat, 22 Jun 2019 15:35:07 -0400 Received: by mail-wr1-f66.google.com with SMTP id x4so9707938wrt.6 for ; Sat, 22 Jun 2019 12:35:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=h9xdExBrqYFWsuLQCJksPQkQTr4z/marqcQjFd4StkE=; b=PsBUE0nVB+7NLTRtR6PyHt8kVIKwtKhXM6MbVB3kmBDZ2Q5zJhkIli8SQnxDQIVcO+ LTzL/3qbyvZUvuaC7BsoW+212GWkRVUA1/3CUSu5l9zMD92q5WiRmf17khCwR2dB6/gD x4YvZZ+MmpRkcf0iELtnKBcr0QUlDve0+gI1V7+WD77ba4mgSNpXA8dyxW5VbuXJKGIS nWCAWMn42S/scchluEKUaN+rWc0fuKBFufPf2+pk+L1EmlAsRtFoiAXSviPVjlgIMOyX 8V1jXySbsLywmRGOlOpYXIhykxYKCa//Ss3BK3jsLW42RqNBLDQrkkYz2PJH1rh3Gjib WRcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=h9xdExBrqYFWsuLQCJksPQkQTr4z/marqcQjFd4StkE=; b=FxLjAkDkpFgxchSQdLgk7qx+j7cqUi6PYRt1MGmvHucYxBoN9dnT8MPGC9QVOVvbad sPYpmBSNPAnalTFLNdJWJYRETHtRe09kRw9l5dRga9wtBcGzsuO9mAP8zCsTtNKHCHXj SfUP8KxVdl9XUGl3ywSyMndjKF9IHHArwFc2z+de0AdPsfWv2v5cKD59fU5qXZ3ejwPM tAfPmG5vpkzedDWjpsSYFczLAfaW4CQRPzZNq6sAEbnUHFLAizLUW+zaB4+teZjAdJU7 a7adfpjPExfHgxoOet50IYKTOGQQDwDynsc3V3HpeL1OOTbgBvyb+NYFw7tvbIwvLij2 Xnzw== X-Gm-Message-State: APjAAAWYL0X/bmhHV7IT4fIFRq7R+dNrOAQ9D0ZK8ASp/T/IAKhcUVWt 2w9+e5e8Wg5+1jEVmGavDXSeFTZ6EVZEQTFN X-Received: by 2002:a5d:5607:: with SMTP id l7mr15715931wrv.228.1561232105612; Sat, 22 Jun 2019 12:35:05 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.35.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:35:04 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 24/26] crypto: amcc/aes - switch to AES library for GCM key derivation Date: Sat, 22 Jun 2019 21:34:25 +0200 Message-Id: <20190622193427.20336-25-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The AMCC code for GCM key derivation allocates a AES cipher to perform a single block encryption. So let's switch to the new and more lightweight AES library instead. Signed-off-by: Ard Biesheuvel --- drivers/crypto/Kconfig | 2 +- drivers/crypto/amcc/crypto4xx_alg.c | 24 +++++++------------- 2 files changed, 9 insertions(+), 17 deletions(-) -- 2.20.1 diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig index b30b84089d11..c7ac1e6d23d4 100644 --- a/drivers/crypto/Kconfig +++ b/drivers/crypto/Kconfig @@ -311,7 +311,7 @@ config CRYPTO_DEV_PPC4XX depends on PPC && 4xx select CRYPTO_HASH select CRYPTO_AEAD - select CRYPTO_AES + select CRYPTO_LIB_AES select CRYPTO_CCM select CRYPTO_CTR select CRYPTO_GCM diff --git a/drivers/crypto/amcc/crypto4xx_alg.c b/drivers/crypto/amcc/crypto4xx_alg.c index 26f86fd7532b..d3660703a36c 100644 --- a/drivers/crypto/amcc/crypto4xx_alg.c +++ b/drivers/crypto/amcc/crypto4xx_alg.c @@ -536,28 +536,20 @@ static int crypto4xx_aes_gcm_validate_keylen(unsigned int keylen) static int crypto4xx_compute_gcm_hash_key_sw(__le32 *hash_start, const u8 *key, unsigned int keylen) { - struct crypto_cipher *aes_tfm = NULL; + struct crypto_aes_ctx ctx; uint8_t src[16] = { 0 }; - int rc = 0; - - aes_tfm = crypto_alloc_cipher("aes", 0, CRYPTO_ALG_NEED_FALLBACK); - if (IS_ERR(aes_tfm)) { - rc = PTR_ERR(aes_tfm); - pr_warn("could not load aes cipher driver: %d\n", rc); - return rc; - } + int rc; - rc = crypto_cipher_setkey(aes_tfm, key, keylen); + rc = aes_expandkey(&ctx, key, keylen); if (rc) { - pr_err("setkey() failed: %d\n", rc); - goto out; + pr_err("aes_expandkey() failed: %d\n", rc); + return rc; } - crypto_cipher_encrypt_one(aes_tfm, src, src); + aes_encrypt(&ctx, src, src); crypto4xx_memcpy_to_le32(hash_start, src, 16); -out: - crypto_free_cipher(aes_tfm); - return rc; + memzero_explicit(&ctx, sizeof(ctx)); + return 0; } int crypto4xx_setkey_aes_gcm(struct crypto_aead *cipher, From patchwork Sat Jun 22 19:34:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167509 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2256125ilk; Sat, 22 Jun 2019 12:35:11 -0700 (PDT) X-Google-Smtp-Source: APXvYqxtyf2CEPaNEASt181jOcze+8TVQ+seQVQ4MmjS9yu/g0nMDvZYta3HPLbo1JlH3Tt4DHay X-Received: by 2002:a63:9e53:: with SMTP id r19mr15901054pgo.442.1561232110901; Sat, 22 Jun 2019 12:35:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232110; cv=none; d=google.com; s=arc-20160816; b=zKrktgdl4jGOUOEFSjZZPwE8nNJbqQNscN3xdPot1L1TsYzDxEv0utkUgw98hnrpiV pVcd2NNerPpwyARVJnwkstZM7CqWfVDz76m/BJfyGubikoPCCa/jhrCMc5j46et8lvvl 1+BO4gGgmO9nJj/9YcfYoy09VwCmPEEzgtSPw35p/AM7Gkf9gQVtArK5ezSrSqyF57kE s5OjGRX7T38YZilAxLOIKl1Hg9Kf9RNWu2gHv6wUtZT6DclGiok8DCM+M8MbJWechYJF dKj6KY/n70/PEJ42Jyc29N+3MEJ5Tqsuvk6Qm79YvY92meY8mJt3PH5jsDDlEYA2egSH HL1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Fm4ramXN1vd7JQGliUbSFSJzkgKkgIE9Zzs9yUh/C1k=; b=xaySqaVupmvkcIGUFbk1/Oxyb3Yz9vFUWkquSQ8FusbZV7j7+6tHWlxznioavMl4p5 ZXpKtIl9thPxiCy+396IF5y4dmKBTQxmaufNn91EEBt1bMQxIWpdks4hq25dPtmziMZB x/XtBjEAXZTwIOmn/LAumoEDW/p6TvasGbHwLaV9r1OkaYixLuMHexvaJvswEoWj66n3 B6rwLqqThFnv3kpcR/OFfuqLPJOpFqG+DZuUqzKJG1uTVx5VG4VlHDFkQNjQFd+yIoyI mW7C2QKnv8IbBstF4CpOdBQY8kXTogMS56i7A1JoeAGXm5OfAZCGqQsV+UBJTo4V91Zf jFBg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=cQa5bBNH; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.35.10; Sat, 22 Jun 2019 12:35:10 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=cQa5bBNH; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726373AbfFVTfK (ORCPT + 3 others); Sat, 22 Jun 2019 15:35:10 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:35981 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726396AbfFVTfJ (ORCPT ); Sat, 22 Jun 2019 15:35:09 -0400 Received: by mail-wr1-f65.google.com with SMTP id n4so8517508wrs.3 for ; Sat, 22 Jun 2019 12:35:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Fm4ramXN1vd7JQGliUbSFSJzkgKkgIE9Zzs9yUh/C1k=; b=cQa5bBNHUb9giayw2fJSuMPHtlyBny32MYddKn3/NiDaWpOUGAYFyv5J1BcRXqCvW8 2477L3mzeFfASZPC3G6dOWtEp35PFuN9GF4Cn6uIoNX+4QamMsPb5OSuhC1EG221IHlA NBAiP0UbPpmxoYN40gRdGM34rCEbdiXwa5kGkF72nNYEEHgFiiSN352P8PvO3WZmBLgF tZOMuMDBmTI+AAIkZWq1QRdUFQb+/SeCYK8/o9lmlaUGM+E+caSV3n46ULkIFbpVi347 y16YTPI1kczNKwlp8q1wtSSd4xt0racCk//mY8YJNJMe9FM5rpGKlQ6fIr4EiOY0tYxf o+Wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Fm4ramXN1vd7JQGliUbSFSJzkgKkgIE9Zzs9yUh/C1k=; b=CoWz1WsLx1uVFoJOwrvy3Q1PsX5a9J8Z4+msdvysPuuxmAvP325McOZJDVtCjVwFs8 Aqa3u8szxBVY6D9CThmfGCoZIH4/33j/oUvSWN2QdcINkc74v8a8SuuKOhCtSgMwmd+B cz/PRI2uGe586+gxaBhd97/cs23dqWCcD/K5+YJm3Yw4rAfKdWu/+W3FMm6wZDuGVQs/ 5sl3GTJQqCoETGFNmrq8dV1tv/8dkFOol2W11UbIvcWzZilYlIqRLC2dzoV5YiezNEnY e7aDk5o+XfF8s+1lcGtvLMgl/n8u4xeV/GmEyyExoIyXLG5944tmdy7IpcQYTG7OBeEW DFXQ== X-Gm-Message-State: APjAAAVHjLTBBXAsmmc6kegfo9RCpQBLLoUGpJRqHu7VGgtTYFJPmTd8 B2wxw5FP3jx/YWeUSkHWsExLgTGe67T0qjzA X-Received: by 2002:adf:ff84:: with SMTP id j4mr23369809wrr.71.1561232106710; Sat, 22 Jun 2019 12:35:06 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.35.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:35:06 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 25/26] crypto: ccp - move to AES library for CMAC key derivation Date: Sat, 22 Jun 2019 21:34:26 +0200 Message-Id: <20190622193427.20336-26-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Use the AES library instead of the cipher interface to perform the single block of AES processing involved in updating the key of the cmac(aes) hash. Signed-off-by: Ard Biesheuvel --- drivers/crypto/ccp/Kconfig | 1 + drivers/crypto/ccp/ccp-crypto-aes-cmac.c | 25 ++++---------------- drivers/crypto/ccp/ccp-crypto.h | 3 --- 3 files changed, 5 insertions(+), 24 deletions(-) -- 2.20.1 diff --git a/drivers/crypto/ccp/Kconfig b/drivers/crypto/ccp/Kconfig index b9dfae47aefd..ee06d0fccdb5 100644 --- a/drivers/crypto/ccp/Kconfig +++ b/drivers/crypto/ccp/Kconfig @@ -29,6 +29,7 @@ config CRYPTO_DEV_CCP_CRYPTO select CRYPTO_BLKCIPHER select CRYPTO_AUTHENC select CRYPTO_RSA + select CRYPTO_LIB_AES help Support for using the cryptographic API with the AMD Cryptographic Coprocessor. This module supports offload of SHA and AES algorithms. diff --git a/drivers/crypto/ccp/ccp-crypto-aes-cmac.c b/drivers/crypto/ccp/ccp-crypto-aes-cmac.c index f6e252c1d6fb..c8f4b29bf044 100644 --- a/drivers/crypto/ccp/ccp-crypto-aes-cmac.c +++ b/drivers/crypto/ccp/ccp-crypto-aes-cmac.c @@ -264,6 +264,7 @@ static int ccp_aes_cmac_setkey(struct crypto_ahash *tfm, const u8 *key, ccp_crypto_ahash_alg(crypto_ahash_tfm(tfm)); u64 k0_hi, k0_lo, k1_hi, k1_lo, k2_hi, k2_lo; u64 rb_hi = 0x00, rb_lo = 0x87; + struct crypto_aes_ctx aes; __be64 *gk; int ret; @@ -287,14 +288,14 @@ static int ccp_aes_cmac_setkey(struct crypto_ahash *tfm, const u8 *key, ctx->u.aes.key_len = 0; /* Set the key for the AES cipher used to generate the keys */ - ret = crypto_cipher_setkey(ctx->u.aes.tfm_cipher, key, key_len); + ret = aes_expandkey(&aes, key, key_len); if (ret) return ret; /* Encrypt a block of zeroes - use key area in context */ memset(ctx->u.aes.key, 0, sizeof(ctx->u.aes.key)); - crypto_cipher_encrypt_one(ctx->u.aes.tfm_cipher, ctx->u.aes.key, - ctx->u.aes.key); + aes_encrypt(&aes, ctx->u.aes.key, ctx->u.aes.key); + memzero_explicit(&aes, sizeof(aes)); /* Generate K1 and K2 */ k0_hi = be64_to_cpu(*((__be64 *)ctx->u.aes.key)); @@ -339,32 +340,15 @@ static int ccp_aes_cmac_cra_init(struct crypto_tfm *tfm) { struct ccp_ctx *ctx = crypto_tfm_ctx(tfm); struct crypto_ahash *ahash = __crypto_ahash_cast(tfm); - struct crypto_cipher *cipher_tfm; ctx->complete = ccp_aes_cmac_complete; ctx->u.aes.key_len = 0; crypto_ahash_set_reqsize(ahash, sizeof(struct ccp_aes_cmac_req_ctx)); - cipher_tfm = crypto_alloc_cipher("aes", 0, CRYPTO_ALG_NEED_FALLBACK); - if (IS_ERR(cipher_tfm)) { - pr_warn("could not load aes cipher driver\n"); - return PTR_ERR(cipher_tfm); - } - ctx->u.aes.tfm_cipher = cipher_tfm; - return 0; } -static void ccp_aes_cmac_cra_exit(struct crypto_tfm *tfm) -{ - struct ccp_ctx *ctx = crypto_tfm_ctx(tfm); - - if (ctx->u.aes.tfm_cipher) - crypto_free_cipher(ctx->u.aes.tfm_cipher); - ctx->u.aes.tfm_cipher = NULL; -} - int ccp_register_aes_cmac_algs(struct list_head *head) { struct ccp_crypto_ahash_alg *ccp_alg; @@ -404,7 +388,6 @@ int ccp_register_aes_cmac_algs(struct list_head *head) base->cra_ctxsize = sizeof(struct ccp_ctx); base->cra_priority = CCP_CRA_PRIORITY; base->cra_init = ccp_aes_cmac_cra_init; - base->cra_exit = ccp_aes_cmac_cra_exit; base->cra_module = THIS_MODULE; ret = crypto_register_ahash(alg); diff --git a/drivers/crypto/ccp/ccp-crypto.h b/drivers/crypto/ccp/ccp-crypto.h index 28819e11db96..9100df77a7b3 100644 --- a/drivers/crypto/ccp/ccp-crypto.h +++ b/drivers/crypto/ccp/ccp-crypto.h @@ -90,9 +90,6 @@ struct ccp_aes_ctx { /* Fallback cipher for XTS with unsupported unit sizes */ struct crypto_sync_skcipher *tfm_skcipher; - /* Cipher used to generate CMAC K1/K2 keys */ - struct crypto_cipher *tfm_cipher; - enum ccp_engine engine; enum ccp_aes_type type; enum ccp_aes_mode mode; From patchwork Sat Jun 22 19:34:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 167510 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp2256138ilk; Sat, 22 Jun 2019 12:35:11 -0700 (PDT) X-Google-Smtp-Source: APXvYqwZ51eBheVijx//EZzqw2wUsv9GqovD8cuL11BrrqRrZMIE33hV7T+4mpZQOu8kz8PWEDJX X-Received: by 2002:a65:518d:: with SMTP id h13mr24428270pgq.22.1561232111459; Sat, 22 Jun 2019 12:35:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561232111; cv=none; d=google.com; s=arc-20160816; b=YDFqdOffnw/uSRdMIrk78RMzFMGUsgt42zoOUsgn8xBad483NvX0xDFkj0YsJZLXvE X2/wRef3Klb+0k6g4vVSSo1qG0h7wz6ihPHnxundpVfP8FPJPvugnOAii72atycW2rDv 7OfZMhQRulrWm3wzi8jFEabAZNQjSZcs4eizPIZDxTn0kB5XVZYEVXWJvYe7NdvqKIQe daXvBU9kS85j7Divc9G8/j6RcB7G3O9wywYYjur3QKN6yLirTkFdEuWprBIkx4aUkvCk in9wmQ0gQtpufGL5Nob4RJbxMHPVR+fSBeGzWWdogYqGGwgnczuLgrKAeAQ40ITfhlPM VHJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=SI+i3SEIksVLr/oeokNkhVP46cu/BheSlfNWTBwHojU=; b=ZmCE/rsTYv4NAoHXHqEBi9joE9BL/KiTNYZ93AMqN+DE8LeApsWBKL3HisFu1royyn Y5gTUAX1AFOu+8+1Z+gNMtsoKliREMiMgMZGaYAZLKfLqvdJULDR9My2C2263jFy5oXc H81ZEOySkuq3xC0AeIr3K+axRVZAY52SEmEUTsxgBb1+2IwLlFF3i2lQA/TxBLMQN4pl bGA4uDhn9IZ9hfvNRVhEnpHqOI3zMRWz0Zafc3xS6d4kX+xzuYpSz4JzYHq/iU3mjdGr g84nZanUfvHXaVXN/xMZdsMFVUbeN5E2sJ3MzGh5rJjKPRDpY0+1QHQypepeYbrNydIe TSpw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=RAPTAHXa; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l21si5511699pgb.409.2019.06.22.12.35.11; Sat, 22 Jun 2019 12:35:11 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=RAPTAHXa; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726378AbfFVTfK (ORCPT + 3 others); Sat, 22 Jun 2019 15:35:10 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:44233 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726379AbfFVTfK (ORCPT ); Sat, 22 Jun 2019 15:35:10 -0400 Received: by mail-wr1-f66.google.com with SMTP id r16so9681178wrl.11 for ; Sat, 22 Jun 2019 12:35:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SI+i3SEIksVLr/oeokNkhVP46cu/BheSlfNWTBwHojU=; b=RAPTAHXaoDVe/Z2mTTJE4TiShHfJsA6E3wSmcOxYCfehlwCeEMUaXxnfpwCej+QIrd UXKg/LnVi97BHsAEhOlsMZnLWIGU+gEYnldHDRShax93fNEtSZxmNtlpQcnHkB/0GCIt Z/nlIpApoHjmKkO2lvLKzWlQmU2QkT20xmjo8MfdN+mCybFE0Eh/tPAZcdbJG6HI6cg3 +ya2iecnsC9Y6AJXk3EtQB6VxFpiNTX1DwhAxe13LWzoV3kUwUjhN3pbu24lyrS16nkj pNq4xwp1c5io677a3xVl2/sjge8CM7E7cp1S3LujPJHmletbiImsdMea6Y2YotozOqvT +8fA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SI+i3SEIksVLr/oeokNkhVP46cu/BheSlfNWTBwHojU=; b=dCKBndgsuS2BbXiWTMAkP630KE5mRZCAzjmkGJFC05tsOG+j/dm9KLiJ4bnpWE/dYk w1KVH4BPYcLgVh76ReaeYAJ3kBX7umRdxFYbgGFFKP8kNG9JpJT85tRBu2rYpJjpIoDy SrSC2aAFcSXCYkYFtsO1y8N7+f0CiqIXSzdcx0ih2POCUDWU342pmhaytd/8mfShyGf5 1VOHXlkVY1zhnP1/QXh7LFm95Q96TGNn2V4A1KNTgYos3QjscuHN1HSzJR4V7Xb6ThJN EwPcpxrWNPRikX+yLxdFrLndldWzLFxY+uC17748W8u6vpO1s1NZvv37SYqpwSWAltBp 7t4Q== X-Gm-Message-State: APjAAAU4I0eijzABV6b4cy75E8dsId3hzdBZ/4h4nfiOC+W6OrQSwQ3T gfNkmi5ojfn8uG65Fris85bMUvL/cFu+L8W5 X-Received: by 2002:adf:fa4c:: with SMTP id y12mr87194296wrr.282.1561232107762; Sat, 22 Jun 2019 12:35:07 -0700 (PDT) Received: from sudo.home ([2a01:cb1d:112:6f00:4bd:3f91:4ef8:ae7e]) by smtp.gmail.com with ESMTPSA id h8sm4814494wmf.12.2019.06.22.12.35.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 12:35:07 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@google.com, Ard Biesheuvel Subject: [PATCH v2 26/26] crypto: chelsio/aes - replace AES cipher calls with library calls Date: Sat, 22 Jun 2019 21:34:27 +0200 Message-Id: <20190622193427.20336-27-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190622193427.20336-1-ard.biesheuvel@linaro.org> References: <20190622193427.20336-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Replace a couple of occurrences where the "aes-generic" cipher is instantiated explicitly and only used for encryption of a single block. Use AES library calls instead. Signed-off-by: Ard Biesheuvel --- drivers/crypto/chelsio/Kconfig | 1 + drivers/crypto/chelsio/chcr_algo.c | 46 ++++++-------------- drivers/crypto/chelsio/chcr_crypto.h | 1 - drivers/crypto/chelsio/chcr_ipsec.c | 19 +++----- drivers/crypto/chelsio/chtls/chtls_hw.c | 20 +++------ 5 files changed, 26 insertions(+), 61 deletions(-) -- 2.20.1 diff --git a/drivers/crypto/chelsio/Kconfig b/drivers/crypto/chelsio/Kconfig index 930d82d991f2..36402ba63b50 100644 --- a/drivers/crypto/chelsio/Kconfig +++ b/drivers/crypto/chelsio/Kconfig @@ -1,6 +1,7 @@ config CRYPTO_DEV_CHELSIO tristate "Chelsio Crypto Co-processor Driver" depends on CHELSIO_T4 + select CRYPTO_LIB_AES select CRYPTO_SHA1 select CRYPTO_SHA256 select CRYPTO_SHA512 diff --git a/drivers/crypto/chelsio/chcr_algo.c b/drivers/crypto/chelsio/chcr_algo.c index 177f572b9589..38ee38b37ae6 100644 --- a/drivers/crypto/chelsio/chcr_algo.c +++ b/drivers/crypto/chelsio/chcr_algo.c @@ -1023,22 +1023,21 @@ static int chcr_update_tweak(struct ablkcipher_request *req, u8 *iv, struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req); struct ablk_ctx *ablkctx = ABLK_CTX(c_ctx(tfm)); struct chcr_blkcipher_req_ctx *reqctx = ablkcipher_request_ctx(req); - struct crypto_cipher *cipher; + struct crypto_aes_ctx aes; int ret, i; u8 *key; unsigned int keylen; int round = reqctx->last_req_len / AES_BLOCK_SIZE; int round8 = round / 8; - cipher = ablkctx->aes_generic; memcpy(iv, reqctx->iv, AES_BLOCK_SIZE); keylen = ablkctx->enckey_len / 2; key = ablkctx->key + keylen; - ret = crypto_cipher_setkey(cipher, key, keylen); + ret = aes_expandkey(&aes, key, keylen); if (ret) - goto out; - crypto_cipher_encrypt_one(cipher, iv, iv); + return ret; + aes_encrypt(&aes, iv, iv); for (i = 0; i < round8; i++) gf128mul_x8_ble((le128 *)iv, (le128 *)iv); @@ -1046,9 +1045,10 @@ static int chcr_update_tweak(struct ablkcipher_request *req, u8 *iv, gf128mul_x_ble((le128 *)iv, (le128 *)iv); if (!isfinal) - crypto_cipher_decrypt_one(cipher, iv, iv); -out: - return ret; + aes_decrypt(&aes, iv, iv); + + memzero_explicit(&aes, sizeof(aes)); + return 0; } static int chcr_update_cipher_iv(struct ablkcipher_request *req, @@ -1411,16 +1411,6 @@ static int chcr_cra_init(struct crypto_tfm *tfm) return PTR_ERR(ablkctx->sw_cipher); } - if (get_cryptoalg_subtype(tfm) == CRYPTO_ALG_SUB_TYPE_XTS) { - /* To update tweak*/ - ablkctx->aes_generic = crypto_alloc_cipher("aes-generic", 0, 0); - if (IS_ERR(ablkctx->aes_generic)) { - pr_err("failed to allocate aes cipher for tweak\n"); - return PTR_ERR(ablkctx->aes_generic); - } - } else - ablkctx->aes_generic = NULL; - tfm->crt_ablkcipher.reqsize = sizeof(struct chcr_blkcipher_req_ctx); return chcr_device_init(crypto_tfm_ctx(tfm)); } @@ -1451,8 +1441,6 @@ static void chcr_cra_exit(struct crypto_tfm *tfm) struct ablk_ctx *ablkctx = ABLK_CTX(ctx); crypto_free_sync_skcipher(ablkctx->sw_cipher); - if (ablkctx->aes_generic) - crypto_free_cipher(ablkctx->aes_generic); } static int get_alg_config(struct algo_param *params, @@ -3364,9 +3352,9 @@ static int chcr_gcm_setkey(struct crypto_aead *aead, const u8 *key, { struct chcr_aead_ctx *aeadctx = AEAD_CTX(a_ctx(aead)); struct chcr_gcm_ctx *gctx = GCM_CTX(aeadctx); - struct crypto_cipher *cipher; unsigned int ck_size; int ret = 0, key_ctx_size = 0; + struct crypto_aes_ctx aes; aeadctx->enckey_len = 0; crypto_aead_clear_flags(aeadctx->sw_cipher, CRYPTO_TFM_REQ_MASK); @@ -3409,23 +3397,15 @@ static int chcr_gcm_setkey(struct crypto_aead *aead, const u8 *key, /* Calculate the H = CIPH(K, 0 repeated 16 times). * It will go in key context */ - cipher = crypto_alloc_cipher("aes-generic", 0, 0); - if (IS_ERR(cipher)) { - aeadctx->enckey_len = 0; - ret = -ENOMEM; - goto out; - } - - ret = crypto_cipher_setkey(cipher, key, keylen); + ret = aes_expandkey(&aes, key, keylen); if (ret) { aeadctx->enckey_len = 0; - goto out1; + goto out; } memset(gctx->ghash_h, 0, AEAD_H_SIZE); - crypto_cipher_encrypt_one(cipher, gctx->ghash_h, gctx->ghash_h); + aes_encrypt(&aes, gctx->ghash_h, gctx->ghash_h); + memzero_explicit(&aes, sizeof(aes)); -out1: - crypto_free_cipher(cipher); out: return ret; } diff --git a/drivers/crypto/chelsio/chcr_crypto.h b/drivers/crypto/chelsio/chcr_crypto.h index 655606f2e4d0..993c97e70565 100644 --- a/drivers/crypto/chelsio/chcr_crypto.h +++ b/drivers/crypto/chelsio/chcr_crypto.h @@ -172,7 +172,6 @@ static inline struct chcr_context *h_ctx(struct crypto_ahash *tfm) struct ablk_ctx { struct crypto_sync_skcipher *sw_cipher; - struct crypto_cipher *aes_generic; __be32 key_ctx_hdr; unsigned int enckey_len; unsigned char ciph_mode; diff --git a/drivers/crypto/chelsio/chcr_ipsec.c b/drivers/crypto/chelsio/chcr_ipsec.c index f429aae72542..24355680f30a 100644 --- a/drivers/crypto/chelsio/chcr_ipsec.c +++ b/drivers/crypto/chelsio/chcr_ipsec.c @@ -132,11 +132,11 @@ static inline int chcr_ipsec_setauthsize(struct xfrm_state *x, static inline int chcr_ipsec_setkey(struct xfrm_state *x, struct ipsec_sa_entry *sa_entry) { - struct crypto_cipher *cipher; int keylen = (x->aead->alg_key_len + 7) / 8; unsigned char *key = x->aead->alg_key; int ck_size, key_ctx_size = 0; unsigned char ghash_h[AEAD_H_SIZE]; + struct crypto_aes_ctx aes; int ret = 0; if (keylen > 3) { @@ -170,26 +170,19 @@ static inline int chcr_ipsec_setkey(struct xfrm_state *x, /* Calculate the H = CIPH(K, 0 repeated 16 times). * It will go in key context */ - cipher = crypto_alloc_cipher("aes-generic", 0, 0); - if (IS_ERR(cipher)) { - sa_entry->enckey_len = 0; - ret = -ENOMEM; - goto out; - } - - ret = crypto_cipher_setkey(cipher, key, keylen); + ret = aes_expandkey(&aes, key, keylen); if (ret) { sa_entry->enckey_len = 0; - goto out1; + goto out; } memset(ghash_h, 0, AEAD_H_SIZE); - crypto_cipher_encrypt_one(cipher, ghash_h, ghash_h); + aes_encrypt(&aes, ghash_h, ghash_h); + memzero_explicit(&aes, sizeof(aes)); + memcpy(sa_entry->key + (DIV_ROUND_UP(sa_entry->enckey_len, 16) * 16), ghash_h, AEAD_H_SIZE); sa_entry->kctx_len = ((DIV_ROUND_UP(sa_entry->enckey_len, 16)) << 4) + AEAD_H_SIZE; -out1: - crypto_free_cipher(cipher); out: return ret; } diff --git a/drivers/crypto/chelsio/chtls/chtls_hw.c b/drivers/crypto/chelsio/chtls/chtls_hw.c index 490960755864..a6f0278f3597 100644 --- a/drivers/crypto/chelsio/chtls/chtls_hw.c +++ b/drivers/crypto/chelsio/chtls/chtls_hw.c @@ -216,8 +216,8 @@ static int chtls_key_info(struct chtls_sock *csk, unsigned char key[AES_KEYSIZE_128]; struct tls12_crypto_info_aes_gcm_128 *gcm_ctx; unsigned char ghash_h[AEAD_H_SIZE]; - struct crypto_cipher *cipher; int ck_size, key_ctx_size; + struct crypto_aes_ctx aes; int ret; gcm_ctx = (struct tls12_crypto_info_aes_gcm_128 *) @@ -237,18 +237,13 @@ static int chtls_key_info(struct chtls_sock *csk, /* Calculate the H = CIPH(K, 0 repeated 16 times). * It will go in key context */ - cipher = crypto_alloc_cipher("aes", 0, 0); - if (IS_ERR(cipher)) { - ret = -ENOMEM; - goto out; - } - - ret = crypto_cipher_setkey(cipher, key, keylen); + ret = aes_expandkey(&aes, key, keylen); if (ret) - goto out1; + return ret; memset(ghash_h, 0, AEAD_H_SIZE); - crypto_cipher_encrypt_one(cipher, ghash_h, ghash_h); + aes_encrypt(&aes, ghash_h, ghash_h); + memzero_explicit(&aes, sizeof(aes)); csk->tlshws.keylen = key_ctx_size; /* Copy the Key context */ @@ -272,10 +267,7 @@ static int chtls_key_info(struct chtls_sock *csk, /* erase key info from driver */ memset(gcm_ctx->key, 0, keylen); -out1: - crypto_free_cipher(cipher); -out: - return ret; + return 0; } static void chtls_set_scmd(struct chtls_sock *csk)