From patchwork Sat Mar 10 15:21:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 131295 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp2244525lja; Sat, 10 Mar 2018 07:22:49 -0800 (PST) X-Google-Smtp-Source: AG47ELv5qLUtyd360NtDroeENcNs7On1waRyINWFj5cNxjqIspMbzzaBrgBYRR8WRPsRvHOWi+5N X-Received: by 10.99.66.65 with SMTP id p62mr1878572pga.378.1520695369194; Sat, 10 Mar 2018 07:22:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520695369; cv=none; d=google.com; s=arc-20160816; b=l8z8A65FR3ua1m2L9ykNE53wDuGS5XkGy0ZHny4QuVo4zp6NBJ/07AqPelLaNOlASL u3OPosikevV/Zu+6lD3ks/5q8dX7e2paDrgFWRvGk3k4FmsLhgpoF+Gl+wIeqEuiND2F NH9nXnL8LozBAwHmcBhMT/wW3g303IB9ckXfSdg/VfUfFkJBhZtVTxOlTEQjXsolBjWJ 5muZRqIt2J7ZA5qrpght0vEc98RqJ/OLq4iBJ/zEPb9NuSP5GssOdCuVziKq0LH+VxR+ tEcyO3kw/4qITbg3IQWtX7PQzSvyqqxIpmOfdUlxrCeoS9FIH2n2zIbfSX0WLsBEaamk IchA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=o9DpJHdXibth1JpdGJrmSP+z+3/O3SvHO7sSQoHB+GE=; b=s+Gqe86O97fyFWwSjCFiHlJjNlekzonHYjafYUJDl+KM6uiMUzfymKdP4vZ83zAabM 4hEv6z20g/s5iTpWGH5YMsYpLnAiItj8Z0INup7p9HFwVe4MrKzwj8ZPEVMcNR4bIxrZ cLiaNaWgytwD20lJ+xAH7mYR3yfDZ4+ldQjDiN3fZ5c3STbB9k5s5gS14/bB41TQ1f6l 8wtyAl0uYhHN1x5MeJS4Ja2ciFntht6oILySrUO12Lw7q3bXgNxUFtIilYKx4nUXwN9j eiS2SBGW70EhRwSkbVPqH7yWDwIvNwuOzLr4/r2BGle7DrmdrP8NICEPDEMjiIWV2zkm 780w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Nc80v7ii; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q16si2824191pfg.221.2018.03.10.07.22.48; Sat, 10 Mar 2018 07:22:49 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Nc80v7ii; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932292AbeCJPWq (ORCPT + 1 other); Sat, 10 Mar 2018 10:22:46 -0500 Received: from mail-wm0-f68.google.com ([74.125.82.68]:33967 "EHLO mail-wm0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932277AbeCJPWp (ORCPT ); Sat, 10 Mar 2018 10:22:45 -0500 Received: by mail-wm0-f68.google.com with SMTP id a20so7233502wmd.1 for ; Sat, 10 Mar 2018 07:22:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=o9DpJHdXibth1JpdGJrmSP+z+3/O3SvHO7sSQoHB+GE=; b=Nc80v7ii0S8D+KJt8wa4vOl+GDzwV1Kka+jcAUJiwWvaMRm3Zb6wyFwN8DcPWLTr87 NaGF1vcrk2pbf9JekkSemNPsthQVkJj87zu5bG3pmpLWJK+bOe5wuboIrXxNJq2lQYcm HK0ereFQisULL6CyFhM4qkZLEkGPHZSoqKvQM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=o9DpJHdXibth1JpdGJrmSP+z+3/O3SvHO7sSQoHB+GE=; b=Py+aUURlVBqIRo4ifUFaEeYwnA8dG6qRxrjUoMucGNgSHIQyLTUPcr0QaLJwcqemII 8R/2GXKWFzBVb9y90QCvUM1sK0NJO0pXpu4BOT5aWiWD3Qv6JQwlJzefgBDZIbKObUv6 md8oUYoz6WHdc3BfM10W9i3bklPBCE2vxVhJ2Htm3GaMBT+DYx+YcYtx8iEMEsSEAmbx LmX+04WaJBEcgH4v1lYyotCn0vvTcMlM3KIcd29K/AbFdvGorQVepGmR6UOV1VNUWebe AwhvWOnTWDYhB9HhSRgW84XVXky62U0DHE5rzo2fwI+N/lPPzqECCjlwNjpTLkk17L4u Wp5w== X-Gm-Message-State: AElRT7ESnBNEvE1t8H7LCTJ7gzjFuHnSB2M5OQxmZc5gPQT0BYUKqHNy T8LvEDLyhfALyDepFpyj3ZD2QEc0D8k= X-Received: by 10.28.26.202 with SMTP id a193mr1436610wma.138.1520695363656; Sat, 10 Mar 2018 07:22:43 -0800 (PST) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id m9sm7027531wrf.13.2018.03.10.07.22.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:22:43 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v5 05/23] crypto: arm64/chacha20 - move kernel mode neon en/disable into loop Date: Sat, 10 Mar 2018 15:21:50 +0000 Message-Id: <20180310152208.10369-6-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180310152208.10369-1-ard.biesheuvel@linaro.org> References: <20180310152208.10369-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org When kernel mode NEON was first introduced on arm64, the preserve and restore of the userland NEON state was completely unoptimized, and involved saving all registers on each call to kernel_neon_begin(), and restoring them on each call to kernel_neon_end(). For this reason, the NEON crypto code that was introduced at the time keeps the NEON enabled throughout the execution of the crypto API methods, which may include calls back into the crypto API that could result in memory allocation or other actions that we should avoid when running with preemption disabled. Since then, we have optimized the kernel mode NEON handling, which now restores lazily (upon return to userland), and so the preserve action is only costly the first time it is called after entering the kernel. So let's put the kernel_neon_begin() and kernel_neon_end() calls around the actual invocations of the NEON crypto code, and run the remainder of the code with kernel mode NEON disabled (and preemption enabled) Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/chacha20-neon-glue.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) -- 2.15.1 diff --git a/arch/arm64/crypto/chacha20-neon-glue.c b/arch/arm64/crypto/chacha20-neon-glue.c index cbdb75d15cd0..727579c93ded 100644 --- a/arch/arm64/crypto/chacha20-neon-glue.c +++ b/arch/arm64/crypto/chacha20-neon-glue.c @@ -37,12 +37,19 @@ static void chacha20_doneon(u32 *state, u8 *dst, const u8 *src, u8 buf[CHACHA20_BLOCK_SIZE]; while (bytes >= CHACHA20_BLOCK_SIZE * 4) { + kernel_neon_begin(); chacha20_4block_xor_neon(state, dst, src); + kernel_neon_end(); bytes -= CHACHA20_BLOCK_SIZE * 4; src += CHACHA20_BLOCK_SIZE * 4; dst += CHACHA20_BLOCK_SIZE * 4; state[12] += 4; } + + if (!bytes) + return; + + kernel_neon_begin(); while (bytes >= CHACHA20_BLOCK_SIZE) { chacha20_block_xor_neon(state, dst, src); bytes -= CHACHA20_BLOCK_SIZE; @@ -55,6 +62,7 @@ static void chacha20_doneon(u32 *state, u8 *dst, const u8 *src, chacha20_block_xor_neon(state, buf, buf); memcpy(dst, buf, bytes); } + kernel_neon_end(); } static int chacha20_neon(struct skcipher_request *req) @@ -68,11 +76,10 @@ static int chacha20_neon(struct skcipher_request *req) if (!may_use_simd() || req->cryptlen <= CHACHA20_BLOCK_SIZE) return crypto_chacha20_crypt(req); - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); crypto_chacha20_init(state, ctx, walk.iv); - kernel_neon_begin(); while (walk.nbytes > 0) { unsigned int nbytes = walk.nbytes; @@ -83,7 +90,6 @@ static int chacha20_neon(struct skcipher_request *req) nbytes); err = skcipher_walk_done(&walk, walk.nbytes - nbytes); } - kernel_neon_end(); return err; }