From patchwork Fri Dec 1 21:19:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 120388 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp1655577qgn; Fri, 1 Dec 2017 13:26:16 -0800 (PST) X-Google-Smtp-Source: AGs4zMb1VUBF6UYzTZA4p30rfqAzG3FuGCmkP89nAjnR5F+xdBVlaFuQi/lIXxLOP2qh7loMqz0c X-Received: by 10.98.11.218 with SMTP id 87mr11602917pfl.228.1512163576843; Fri, 01 Dec 2017 13:26:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512163576; cv=none; d=google.com; s=arc-20160816; b=S7gbo8oxnOWEC6xtf47qnLHRbpshnd9JyX6mQuu3beq4i2lLtr1GVBgoYi7Ayv7oje KTTeiw4DgCmuV870xcw4XDSn+mSjWiXzLhUzSV6Il5HzuaYnJfM3u0W3QPZQVctLRjsv 03XU1+UKye9RXkAxCpbYWsCwTYode01XFFGLiAqvV9JlUlM3LFeGzVPDb4Rsa+obKvpw M/T889QoO4iIT6UZTGbDYhy5Gnl0dVU78bPapjw+vTO/bmcfS9TKHZIRFCHSoz1hmln3 ZyQyWmY2eQ5REP9nfEiPPkGeZBi+C8bbxwOlLjdnZIClhejPQaE8vZQxpQZrVhWPvt74 ObMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=SgKsLu2PdMAkeEjJB77DO8HL4XwrPtctezT2ceVL2+w=; b=GF0Hn7WsdAvAOCQVTRkAESTGvZrEnVnl0tZS3E50vyN78o1e2S+8Pv1iYQle1SCp8I qkDzc/t99JHWVZ4DOTd1tjLSz3eB+yvTimArrWbsMB0zMYNK1g09Y8YjPtqeIDssqRKx PKAlqU1YgQqDbcOLK1Wyy9M3F7mOhnzEs0aHSBRgNS5SpVoz4JQ+FT/ngiRCf5RcHxrS nB4/GZFkCYrrrl/PO4sH0vgTySGInp7735okC2v9jcvBU/YPukClycz/o56BRdTCg9Iz S2qzYNf4hiT3wgkyFUQqKl4RqzsfTYYStnjnJ5flrFAgFw7GSfhKm+DgqWjCco/EFyb/ R7OQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=TomDMick; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id u66si5760540pfa.237.2017.12.01.13.26.16; Fri, 01 Dec 2017 13:26:16 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=TomDMick; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751039AbdLAV0P (ORCPT + 1 other); Fri, 1 Dec 2017 16:26:15 -0500 Received: from mail-wr0-f194.google.com ([209.85.128.194]:42460 "EHLO mail-wr0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751188AbdLAV0O (ORCPT ); Fri, 1 Dec 2017 16:26:14 -0500 Received: by mail-wr0-f194.google.com with SMTP id s66so11440075wrc.9 for ; Fri, 01 Dec 2017 13:26:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=SgKsLu2PdMAkeEjJB77DO8HL4XwrPtctezT2ceVL2+w=; b=TomDMickzVqZNh64eSjZtwrzYo30etBaLKXiOAWPaaU7JdIAw+FpiedspC/C+MJbY3 skW85cd2T4R3j5zrtPbPDL4Ch1jJZ6lrUy5gGzi2kmkr1z8ADvmdO7RFcdjdo+ZZY598 gW6/OzozKtUHWkoJGWORQ2j/fDuvt8e9MC1bs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=SgKsLu2PdMAkeEjJB77DO8HL4XwrPtctezT2ceVL2+w=; b=oHb9X3M/NY0gIJEdwUVj6HuER2BSF3SM7eYxEHlkg/iekMUnf9XGnsxeY+K/lLFPHf lX8hYPTH8JNKqt5Ae3KOQCIH1uk5tcyjophQQJRE/sfc3NUvJFYMY5PeGBNdIuNSOHnO WUH17eCDuP+dqLhEnqJ9ikTnDzTh3ePNH9V8/T6U02DWfIIeNwLZP0dGaKKChdkaNd9c V7EZncx+hjOM1jAotrq0FVo9XY9e6nMt2mEzAYS2aeT5T+vmuuHBVvmpXo7atjE1lGkl R5Prg7UpDZnA2JnpDBftnUfhXgruKS/z0AqsGzoOq5EZqUOUBJYPjDeYBaJ4gMq+0HeM kpzg== X-Gm-Message-State: AJaThX7TUSRXgLLoUBQjkHX4DQIZP+apkLwK8QjfObOwwr9iUw1dD0kN AY8m+KcGRuDltNbM/W+hXBn9+lswzVk= X-Received: by 10.223.162.137 with SMTP id s9mr6921536wra.272.1512163572481; Fri, 01 Dec 2017 13:26:12 -0800 (PST) Received: from localhost.localdomain ([105.150.171.234]) by smtp.gmail.com with ESMTPSA id p17sm2161682wma.23.2017.12.01.13.26.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 01 Dec 2017 13:26:11 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH 5/5] crypto: arm64/ghash - move kernel mode neon en/disable into loop Date: Fri, 1 Dec 2017 21:19:27 +0000 Message-Id: <20171201211927.24653-6-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171201211927.24653-1-ard.biesheuvel@linaro.org> References: <20171201211927.24653-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org When kernel mode NEON was first introduced on arm64, the preserve and restore of the userland NEON state was completely unoptimized, and involved saving all registers on each call to kernel_neon_begin(), and restoring them on each call to kernel_neon_end(). For this reason, the NEON crypto code that was introduced at the time keeps the NEON enabled throughout the execution of the crypto API methods, which may include calls back into the crypto API that could result in memory allocation or other actions that we should avoid when running with preemption disabled. Since then, we have optimized the kernel mode NEON handling, which now restores lazily (upon return to userland), and so the preserve action is only costly the first time it is called after entering the kernel. So let's put the kernel_neon_begin() and kernel_neon_end() calls around the actual invocations of the NEON crypto code, and run the remainder of the code with kernel mode NEON disabled (and preemption enabled) Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/ghash-ce-glue.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) -- 2.11.0 diff --git a/arch/arm64/crypto/ghash-ce-glue.c b/arch/arm64/crypto/ghash-ce-glue.c index cfc9c92814fd..ddd9b565d775 100644 --- a/arch/arm64/crypto/ghash-ce-glue.c +++ b/arch/arm64/crypto/ghash-ce-glue.c @@ -368,20 +368,22 @@ static int gcm_encrypt(struct aead_request *req) pmull_gcm_encrypt_block(ks, iv, NULL, num_rounds(&ctx->aes_key)); put_unaligned_be32(3, iv + GCM_IV_SIZE); + kernel_neon_end(); err = skcipher_walk_aead_encrypt(&walk, req, true); while (walk.nbytes >= AES_BLOCK_SIZE) { int blocks = walk.nbytes / AES_BLOCK_SIZE; + kernel_neon_begin(); pmull_gcm_encrypt(blocks, dg, walk.dst.virt.addr, walk.src.virt.addr, &ctx->ghash_key, iv, num_rounds(&ctx->aes_key), ks); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } - kernel_neon_end(); } else { __aes_arm64_encrypt(ctx->aes_key.key_enc, tag, iv, num_rounds(&ctx->aes_key)); @@ -467,15 +469,18 @@ static int gcm_decrypt(struct aead_request *req) pmull_gcm_encrypt_block(tag, iv, ctx->aes_key.key_enc, num_rounds(&ctx->aes_key)); put_unaligned_be32(2, iv + GCM_IV_SIZE); + kernel_neon_end(); err = skcipher_walk_aead_decrypt(&walk, req, true); while (walk.nbytes >= AES_BLOCK_SIZE) { int blocks = walk.nbytes / AES_BLOCK_SIZE; + kernel_neon_begin(); pmull_gcm_decrypt(blocks, dg, walk.dst.virt.addr, walk.src.virt.addr, &ctx->ghash_key, iv, num_rounds(&ctx->aes_key)); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); @@ -483,8 +488,6 @@ static int gcm_decrypt(struct aead_request *req) if (walk.nbytes) pmull_gcm_encrypt_block(iv, iv, NULL, num_rounds(&ctx->aes_key)); - - kernel_neon_end(); } else { __aes_arm64_encrypt(ctx->aes_key.key_enc, tag, iv, num_rounds(&ctx->aes_key));