From patchwork Wed May 14 18:17:29 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 30187 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-pa0-f70.google.com (mail-pa0-f70.google.com [209.85.220.70]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id F013620446 for ; Wed, 14 May 2014 18:18:07 +0000 (UTC) Received: by mail-pa0-f70.google.com with SMTP id lj1sf10691888pab.5 for ; Wed, 14 May 2014 11:18:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:delivered-to:from:to:cc:subject :date:message-id:in-reply-to:references:sender:precedence:list-id :x-original-sender:x-original-authentication-results:mailing-list :list-post:list-help:list-archive:list-unsubscribe; bh=ew660hm3lLzpW1DSMje7teTYVrB6MSDo3eszbz/36x0=; b=E9G2nAKAInoAG+oNxRuQFzOXdL0cmSkM9T4HkJtWXVC5D4ZP6U/aDRPtY28QavOmCl K2xBNyS6zGo45NWeQsdali2SCCbXkZuxejUQQgaQY3ViPBEkALWVXLlMpyTiKyRJ68jR 5x0AeK0rP83TnIRIvvPcwU1fCdNmsVEdLtV3hu6EEPBnLJIZn2unW3VzobbHNlQzbEzO D3HhnwtV9JfzIhrwb4ZHL77ubh+ADiTzBD0pJGbJls9nIBDk/Yhot7l4txoW1/Y6VLk0 OW6AVLtUzvSEzkR2/kB792rP3zBTBC074c3fyhWlm65cXBKqC5aIsy0GSbyVosUpck86 LlSQ== X-Gm-Message-State: ALoCoQlXK4lR38ARQTKkTrkQUDVBGexOeuiPd1s3N4BXs32duWQ7qxF5G5fs47lG5tCZXQybN4Yg X-Received: by 10.66.240.4 with SMTP id vw4mr2388311pac.10.1400091487308; Wed, 14 May 2014 11:18:07 -0700 (PDT) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.23.147 with SMTP id 19ls2550987qgp.68.gmail; Wed, 14 May 2014 11:18:07 -0700 (PDT) X-Received: by 10.52.93.132 with SMTP id cu4mr27695vdb.76.1400091487108; Wed, 14 May 2014 11:18:07 -0700 (PDT) Received: from mail-ve0-f180.google.com (mail-ve0-f180.google.com [209.85.128.180]) by mx.google.com with ESMTPS id ui2si455950vdc.154.2014.05.14.11.18.07 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 14 May 2014 11:18:07 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.128.180 as permitted sender) client-ip=209.85.128.180; Received: by mail-ve0-f180.google.com with SMTP id db12so2874679veb.39 for ; Wed, 14 May 2014 11:18:07 -0700 (PDT) X-Received: by 10.58.38.40 with SMTP id d8mr40803vek.61.1400091487019; Wed, 14 May 2014 11:18:07 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.221.72 with SMTP id ib8csp261280vcb; Wed, 14 May 2014 11:18:06 -0700 (PDT) X-Received: by 10.66.122.36 with SMTP id lp4mr6505301pab.82.1400091486215; Wed, 14 May 2014 11:18:06 -0700 (PDT) Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id qe5si1341172pbc.453.2014.05.14.11.18.05; Wed, 14 May 2014 11:18:05 -0700 (PDT) Received-SPF: none (google.com: linux-crypto-owner@vger.kernel.org does not designate permitted sender hosts) client-ip=209.132.180.67; Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751361AbaENSSF (ORCPT ); Wed, 14 May 2014 14:18:05 -0400 Received: from mail-yh0-f53.google.com ([209.85.213.53]:52499 "EHLO mail-yh0-f53.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751143AbaENSSC (ORCPT ); Wed, 14 May 2014 14:18:02 -0400 Received: by mail-yh0-f53.google.com with SMTP id i57so2027753yha.26 for ; Wed, 14 May 2014 11:18:01 -0700 (PDT) X-Received: by 10.236.167.105 with SMTP id h69mr7713456yhl.30.1400091481524; Wed, 14 May 2014 11:18:01 -0700 (PDT) Received: from ards-macbook-pro.swisscom.com ([12.153.182.133]) by mx.google.com with ESMTPSA id c66sm3641080yhk.23.2014.05.14.11.17.59 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 14 May 2014 11:18:00 -0700 (PDT) From: Ard Biesheuvel To: catalin.marinas@arm.com, jussi.kivilinna@iki.fi, herbert@gondor.apana.org.au Cc: linux-arm-kernel@lists.infradead.org, linux-crypto@vger.kernel.org, Ard Biesheuvel Subject: [PATCH v2 09/11] arm64/crypto: add voluntary preemption to Crypto Extensions SHA1 Date: Wed, 14 May 2014 11:17:29 -0700 Message-Id: <1400091451-9117-10-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 1.8.3.2 In-Reply-To: <1400091451-9117-1-git-send-email-ard.biesheuvel@linaro.org> References: <1400091451-9117-1-git-send-email-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: list List-ID: X-Mailing-List: linux-crypto@vger.kernel.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: ard.biesheuvel@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.128.180 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , The Crypto Extensions based SHA1 implementation uses the NEON register file, and hence runs with preemption disabled. This patch adds a TIF_NEED_RESCHED check to its inner loop so we at least give up the CPU voluntarily when we are running in process context and have been tagged for preemption by the scheduler. Signed-off-by: Ard Biesheuvel Acked-by: Herbert Xu --- arch/arm64/crypto/sha1-ce-core.S | 19 ++++++++------- arch/arm64/crypto/sha1-ce-glue.c | 52 ++++++++++++++++++++++++++-------------- 2 files changed, 44 insertions(+), 27 deletions(-) diff --git a/arch/arm64/crypto/sha1-ce-core.S b/arch/arm64/crypto/sha1-ce-core.S index 09d57d98609c..0cb9b8f4906b 100644 --- a/arch/arm64/crypto/sha1-ce-core.S +++ b/arch/arm64/crypto/sha1-ce-core.S @@ -66,8 +66,8 @@ .word 0x5a827999, 0x6ed9eba1, 0x8f1bbcdc, 0xca62c1d6 /* - * void sha1_ce_transform(int blocks, u8 const *src, u32 *state, - * u8 *head, long bytes) + * int sha1_ce_transform(int blocks, u8 const *src, u32 *state, + * u8 *head, long bytes, struct thread_info *ti) */ ENTRY(sha1_ce_transform) /* load round constants */ @@ -127,7 +127,13 @@ CPU_LE( rev32 v11.16b, v11.16b ) add dgbv.2s, dgbv.2s, dg1v.2s add dgav.4s, dgav.4s, dg0v.4s - cbnz w0, 0b + cbz w0, 4f + b_if_no_resched x5, x8, 0b + + /* store new state */ +3: str dga, [x2] + str dgb, [x2, #16] + ret /* * Final block: add padding and total bit count. @@ -135,7 +141,7 @@ CPU_LE( rev32 v11.16b, v11.16b ) * size was not a round multiple of the block size, and the padding is * handled by the C code. */ - cbz x4, 3f +4: cbz x4, 3b movi v9.2d, #0 mov x8, #0x80000000 movi v10.2d, #0 @@ -145,9 +151,4 @@ CPU_LE( rev32 v11.16b, v11.16b ) mov v11.d[0], xzr mov v11.d[1], x7 b 2b - - /* store new state */ -3: str dga, [x2] - str dgb, [x2, #16] - ret ENDPROC(sha1_ce_transform) diff --git a/arch/arm64/crypto/sha1-ce-glue.c b/arch/arm64/crypto/sha1-ce-glue.c index 6fe83f37a750..b195f7104706 100644 --- a/arch/arm64/crypto/sha1-ce-glue.c +++ b/arch/arm64/crypto/sha1-ce-glue.c @@ -20,8 +20,8 @@ MODULE_DESCRIPTION("SHA1 secure hash using ARMv8 Crypto Extensions"); MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); -asmlinkage void sha1_ce_transform(int blocks, u8 const *src, u32 *state, - u8 *head, long bytes); +asmlinkage int sha1_ce_transform(int blocks, u8 const *src, u32 *state, + u8 *head, long bytes, struct thread_info *ti); static int sha1_init(struct shash_desc *desc) { @@ -33,6 +33,34 @@ static int sha1_init(struct shash_desc *desc) return 0; } +static u8 const *sha1_do_update(struct shash_desc *desc, const u8 *data, + int blocks, u8 *head, unsigned int len) +{ + struct sha1_state *sctx = shash_desc_ctx(desc); + struct thread_info *ti = NULL; + + /* + * Pass current's thread info pointer to sha1_ce_transform() + * below if we want it to play nice under preemption. + */ + if ((IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY) || IS_ENABLED(CONFIG_PREEMPT)) + && (desc->flags & CRYPTO_TFM_REQ_MAY_SLEEP)) + ti = current_thread_info(); + + do { + int rem; + + kernel_neon_begin_partial(16); + rem = sha1_ce_transform(blocks, data, sctx->state, head, 0, ti); + kernel_neon_end(); + + data += (blocks - rem) * SHA1_BLOCK_SIZE; + blocks = rem; + head = NULL; + } while (unlikely(ti && blocks > 0)); + return data; +} + static int sha1_update(struct shash_desc *desc, const u8 *data, unsigned int len) { @@ -42,8 +70,6 @@ static int sha1_update(struct shash_desc *desc, const u8 *data, sctx->count += len; if ((partial + len) >= SHA1_BLOCK_SIZE) { - int blocks; - if (partial) { int p = SHA1_BLOCK_SIZE - partial; @@ -52,15 +78,10 @@ static int sha1_update(struct shash_desc *desc, const u8 *data, len -= p; } - blocks = len / SHA1_BLOCK_SIZE; - len %= SHA1_BLOCK_SIZE; + data = sha1_do_update(desc, data, len / SHA1_BLOCK_SIZE, + partial ? sctx->buffer : NULL, 0); - kernel_neon_begin_partial(16); - sha1_ce_transform(blocks, data, sctx->state, - partial ? sctx->buffer : NULL, 0); - kernel_neon_end(); - - data += blocks * SHA1_BLOCK_SIZE; + len %= SHA1_BLOCK_SIZE; partial = 0; } if (len) @@ -95,7 +116,6 @@ static int sha1_finup(struct shash_desc *desc, const u8 *data, { struct sha1_state *sctx = shash_desc_ctx(desc); __be32 *dst = (__be32 *)out; - int blocks; int i; if (sctx->count || !len || (len % SHA1_BLOCK_SIZE)) { @@ -109,11 +129,7 @@ static int sha1_finup(struct shash_desc *desc, const u8 *data, * perform the entire digest calculation in a single invocation * of sha1_ce_transform() */ - blocks = len / SHA1_BLOCK_SIZE; - - kernel_neon_begin_partial(16); - sha1_ce_transform(blocks, data, sctx->state, NULL, len); - kernel_neon_end(); + sha1_do_update(desc, data, len / SHA1_BLOCK_SIZE, NULL, len); for (i = 0; i < SHA1_DIGEST_SIZE / sizeof(__be32); i++) put_unaligned_be32(sctx->state[i], dst++);