From patchwork Sat Mar 10 15:21:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 131294 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp2244488lja; Sat, 10 Mar 2018 07:22:46 -0800 (PST) X-Google-Smtp-Source: AG47ELtgx93VfEzBSij5USKGtZ2mk+ylkvZcLY7sVcShOrW6Je6pS0Fou9pxOj2+svoQNNGHznf0 X-Received: by 10.99.120.13 with SMTP id t13mr1932082pgc.35.1520695365902; Sat, 10 Mar 2018 07:22:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520695365; cv=none; d=google.com; s=arc-20160816; b=QWRDTmxQAUnbZYa2tvX//AmxNJaK6bAxMSz95yFS7AN8pt6yQAWJCF2OrV57mjXfMA +94T0hPHJOpHywdbM290+KUawNH+YiTHOjlt6lzMZBtSY8vlcqPgLjwj9SI8WOOmJPig ebA7pWIORn7N/gADnTALrJTruvTEl9JTHWEIz5BMn8KFehTvAKKr/hx1bGZr1tnSy3vg 3/2biljvYRk2JlL2O9Z+7ZliR6DH+y0WmZzVHtk6h7SEx+wsYmFrhEuV/7PzIIjS1zXk tD42GB+xDdnlnLErgUnhqEwFCPtaWUyLO/eT7RuwACfRZtVndb9soYrHoTNHIiLf6b5p dICw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=aVJMpK+ItX9Ku1Ngl1sTNCOZ4+v3QvBf89t8LQLjCgc=; b=I4Xx4X+xh4XeAlby1AEEaHp+S4+ZvSGA0+xMnObe2eivg8jGCD3Yzg3nKUxlsfFi0Z dBbtmbQnHPiSTReFbN1dWh6rPyjlhYqMkRD45eicNM7gB57bWr4AVqvgrq8BkDOFNVlp +qWnYowFG4bqHgVW1N3b0c+Z63BPKeC7GUBrUMLFCR4qVj/TJF6+Vnkos97hOmKbjO4i 42J4kitIRYQSZGay+I1y5QwvUwTVcOUDcKhwStLEo1bBfX+oGFHubSIf1lOvpRVWxOej UC5bBDSAn8FDc0BhvyLcdT4L83EjYzfa3GeIFqtRPqz7ynNW3g5BfVcYsk/PkgPyC7GS EVaw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=TiQWRhe7; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q16si2824191pfg.221.2018.03.10.07.22.45; Sat, 10 Mar 2018 07:22:45 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=TiQWRhe7; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932130AbeCJPWn (ORCPT + 1 other); Sat, 10 Mar 2018 10:22:43 -0500 Received: from mail-wr0-f195.google.com ([209.85.128.195]:45839 "EHLO mail-wr0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932271AbeCJPWl (ORCPT ); Sat, 10 Mar 2018 10:22:41 -0500 Received: by mail-wr0-f195.google.com with SMTP id h2so4348392wre.12 for ; Sat, 10 Mar 2018 07:22:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=aVJMpK+ItX9Ku1Ngl1sTNCOZ4+v3QvBf89t8LQLjCgc=; b=TiQWRhe7AyjhEUN2gPsRFJOR+jmW3N7aICigRu2p08JBVf2ouzFuBJhNoUI/7P/nHt lAP4yoy+vF0F0G9ve/9duqLja12a+Z/+Z5bCSNQYTWhJ8AnS+KoMJSlohWZfjL25/z5S VAcOKKv1tQYjEdt5A4TcS1qkEB/eNkrEHsnrI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=aVJMpK+ItX9Ku1Ngl1sTNCOZ4+v3QvBf89t8LQLjCgc=; b=Uw1vSGHxH2wU/535h032vrQ4clGVB4CfRhuiyz2foUk0fisvGGjlhwORjxJhPtsQaJ q8CYS3h8gyBJXEsoWNzX8aZ1Fy0oqp4cbhOHI9bVk53VP07qwZv5ExLzoRsmjX1jNdh7 sTVk5ik1ddC8G+7+HVKeWAiGrhAzGE1eb543OfEIGHD4eBAjQ5oEOjGFE7JTxkLD7kBS qd76KNLel+dLicJgAgKdZh3tLPs5n/NhWtMPwAAXMXLVDO1uaf6e9TRwAJBueVLxDrQ6 86INWdHFihuhOyU1uNu5RSvcc4TzbUHIy7IxxHtPrAhZEdtVoxgp4ij8rivnWTcgpTWT 87XA== X-Gm-Message-State: AElRT7HCiC8s0l1ayIVP5kCnDq0xvjxIFJjtlOAcJLC0ZOx8k3fPDrHZ LPzkM/H8YiJomlKhbP+kVRNyk2aPk74= X-Received: by 10.223.176.86 with SMTP id g22mr1718146wra.11.1520695360376; Sat, 10 Mar 2018 07:22:40 -0800 (PST) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id m9sm7027531wrf.13.2018.03.10.07.22.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:22:39 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v5 04/23] crypto: arm64/aes-bs - move kernel mode neon en/disable into loop Date: Sat, 10 Mar 2018 15:21:49 +0000 Message-Id: <20180310152208.10369-5-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180310152208.10369-1-ard.biesheuvel@linaro.org> References: <20180310152208.10369-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org When kernel mode NEON was first introduced on arm64, the preserve and restore of the userland NEON state was completely unoptimized, and involved saving all registers on each call to kernel_neon_begin(), and restoring them on each call to kernel_neon_end(). For this reason, the NEON crypto code that was introduced at the time keeps the NEON enabled throughout the execution of the crypto API methods, which may include calls back into the crypto API that could result in memory allocation or other actions that we should avoid when running with preemption disabled. Since then, we have optimized the kernel mode NEON handling, which now restores lazily (upon return to userland), and so the preserve action is only costly the first time it is called after entering the kernel. So let's put the kernel_neon_begin() and kernel_neon_end() calls around the actual invocations of the NEON crypto code, and run the remainder of the code with kernel mode NEON disabled (and preemption enabled) Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-neonbs-glue.c | 36 +++++++++----------- 1 file changed, 17 insertions(+), 19 deletions(-) -- 2.15.1 diff --git a/arch/arm64/crypto/aes-neonbs-glue.c b/arch/arm64/crypto/aes-neonbs-glue.c index 9d823c77ec84..e7a95a566462 100644 --- a/arch/arm64/crypto/aes-neonbs-glue.c +++ b/arch/arm64/crypto/aes-neonbs-glue.c @@ -99,9 +99,8 @@ static int __ecb_crypt(struct skcipher_request *req, struct skcipher_walk walk; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; @@ -109,12 +108,13 @@ static int __ecb_crypt(struct skcipher_request *req, blocks = round_down(blocks, walk.stride / AES_BLOCK_SIZE); + kernel_neon_begin(); fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->rk, ctx->rounds, blocks); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } - kernel_neon_end(); return err; } @@ -158,19 +158,19 @@ static int cbc_encrypt(struct skcipher_request *req) struct skcipher_walk walk; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; /* fall back to the non-bitsliced NEON implementation */ + kernel_neon_begin(); neon_aes_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->enc, ctx->key.rounds, blocks, walk.iv); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } - kernel_neon_end(); return err; } @@ -181,9 +181,8 @@ static int cbc_decrypt(struct skcipher_request *req) struct skcipher_walk walk; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; @@ -191,13 +190,14 @@ static int cbc_decrypt(struct skcipher_request *req) blocks = round_down(blocks, walk.stride / AES_BLOCK_SIZE); + kernel_neon_begin(); aesbs_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->key.rk, ctx->key.rounds, blocks, walk.iv); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } - kernel_neon_end(); return err; } @@ -229,9 +229,8 @@ static int ctr_encrypt(struct skcipher_request *req) u8 buf[AES_BLOCK_SIZE]; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while (walk.nbytes > 0) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; u8 *final = (walk.total % AES_BLOCK_SIZE) ? buf : NULL; @@ -242,8 +241,10 @@ static int ctr_encrypt(struct skcipher_request *req) final = NULL; } + kernel_neon_begin(); aesbs_ctr_encrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->rk, ctx->rounds, blocks, walk.iv, final); + kernel_neon_end(); if (final) { u8 *dst = walk.dst.virt.addr + blocks * AES_BLOCK_SIZE; @@ -258,8 +259,6 @@ static int ctr_encrypt(struct skcipher_request *req) err = skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } - kernel_neon_end(); - return err; } @@ -304,12 +303,11 @@ static int __xts_crypt(struct skcipher_request *req, struct skcipher_walk walk; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); kernel_neon_begin(); - - neon_aes_ecb_encrypt(walk.iv, walk.iv, ctx->twkey, - ctx->key.rounds, 1); + neon_aes_ecb_encrypt(walk.iv, walk.iv, ctx->twkey, ctx->key.rounds, 1); + kernel_neon_end(); while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; @@ -318,13 +316,13 @@ static int __xts_crypt(struct skcipher_request *req, blocks = round_down(blocks, walk.stride / AES_BLOCK_SIZE); + kernel_neon_begin(); fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->key.rk, ctx->key.rounds, blocks, walk.iv); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } - kernel_neon_end(); - return err; }