From patchwork Tue Apr 1 13:47:38 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 27543 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-qg0-f71.google.com (mail-qg0-f71.google.com [209.85.192.71]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id D0FF420341 for ; Tue, 1 Apr 2014 13:50:03 +0000 (UTC) Received: by mail-qg0-f71.google.com with SMTP id j5sf14321034qga.6 for ; Tue, 01 Apr 2014 06:50:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:subject:date:message-id :in-reply-to:references:cc:precedence:list-id:list-unsubscribe :list-archive:list-post:list-help:list-subscribe:mime-version:sender :errors-to:x-original-sender:x-original-authentication-results :mailing-list:content-type:content-transfer-encoding; bh=DkO3U6URkudkHHTg/zstBx9PoKEbZGSYRqqgcHpS4Xk=; b=gmE5LG289rdVta4faOech+LBt4MMgDty1BlpVwIJ6rAkLK1dL5C2dogGLX2aD4yCnD ddGlFZ+MPoLTyYnMMiMI4Ya/aR6hALOrAinqaN9toJssb2UIN3Kb1mfvIgBLW4APRdXP k2zNUARd3Stb6zJ/F86ka6z9/1ruprtnMkGHBNO5kYxyDjpVP5D8T7RbWgi6tX/iA445 g/4udRXZdPGM8T0QUPEAreqJZbYveSjLs/VwVlAKJse6Mlv0SSLxRytGeABGQE5L2s35 45SI9NKpdz9G/isUYZMvTtdzXZ53fJRE1PQUogr08XnVExDTfUVmV3UlrVmOr1AsdCKO NS9g== X-Gm-Message-State: ALoCoQmsJaMZf02fAODQpwx7gbJbAxz6fEoOer4E/48DJN+sN8Kx137k0sLXPGE37o7/3OGcabU1 X-Received: by 10.236.216.39 with SMTP id f37mr4884766yhp.57.1396360203222; Tue, 01 Apr 2014 06:50:03 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.42.20 with SMTP id b20ls11464qga.5.gmail; Tue, 01 Apr 2014 06:50:03 -0700 (PDT) X-Received: by 10.52.243.167 with SMTP id wz7mr382172vdc.47.1396360203133; Tue, 01 Apr 2014 06:50:03 -0700 (PDT) Received: from mail-vc0-f174.google.com (mail-vc0-f174.google.com [209.85.220.174]) by mx.google.com with ESMTPS id rx10si3613917vdc.186.2014.04.01.06.50.03 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 01 Apr 2014 06:50:03 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.174 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.220.174; Received: by mail-vc0-f174.google.com with SMTP id ld13so9527213vcb.19 for ; Tue, 01 Apr 2014 06:50:03 -0700 (PDT) X-Received: by 10.52.171.68 with SMTP id as4mr24132980vdc.0.1396360203052; Tue, 01 Apr 2014 06:50:03 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.12.8 with SMTP id v8csp239011vcv; Tue, 1 Apr 2014 06:50:02 -0700 (PDT) X-Received: by 10.180.89.211 with SMTP id bq19mr20029972wib.58.1396360201950; Tue, 01 Apr 2014 06:50:01 -0700 (PDT) Received: from casper.infradead.org (casper.infradead.org. [2001:770:15f::2]) by mx.google.com with ESMTPS id ck5si5017290wjc.84.2014.04.01.06.50.01 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 01 Apr 2014 06:50:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org designates 2001:770:15f::2 as permitted sender) client-ip=2001:770:15f::2; Received: from merlin.infradead.org ([2001:4978:20e::2]) by casper.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1WUz3G-0007FV-2S; Tue, 01 Apr 2014 13:48:30 +0000 Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1WUz3B-0001sd-E6; Tue, 01 Apr 2014 13:48:25 +0000 Received: from mail-we0-f178.google.com ([74.125.82.178]) by merlin.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1WUz2x-0001oY-Sx for linux-arm-kernel@lists.infradead.org; Tue, 01 Apr 2014 13:48:14 +0000 Received: by mail-we0-f178.google.com with SMTP id u56so6152786wes.23 for ; Tue, 01 Apr 2014 06:47:53 -0700 (PDT) X-Received: by 10.180.89.102 with SMTP id bn6mr20160844wib.28.1396360073410; Tue, 01 Apr 2014 06:47:53 -0700 (PDT) Received: from ards-macbook-pro.local ([95.129.121.210]) by mx.google.com with ESMTPSA id 48sm40727180eee.2.2014.04.01.06.47.52 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 01 Apr 2014 06:47:52 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, nico@linaro.org Subject: [PATCH v3 6/7] arm64/crypto: add voluntary preemption to Crypto Extensions SHA2 Date: Tue, 1 Apr 2014 15:47:38 +0200 Message-Id: <1396360059-31949-7-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 1.8.3.2 In-Reply-To: <1396360059-31949-1-git-send-email-ard.biesheuvel@linaro.org> References: <1396360059-31949-1-git-send-email-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20140401_094812_085217_17A37F9F X-CRM114-Status: GOOD ( 14.60 ) X-Spam-Score: -2.6 (--) X-Spam-Report: SpamAssassin version 3.3.2 on merlin.infradead.org summary: Content analysis details: (-2.6 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low trust [74.125.82.178 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] Cc: Ard Biesheuvel X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: ard.biesheuvel@linaro.org X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.220.174 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 The Crypto Extensions based SHA2 implementation uses the NEON register file, and hence runs with preemption disabled. This patch adds a TIF_NEED_RESCHED check to its inner loop so we at least give up the CPU voluntarily when if we are running in process context and have been tagged for preemption by the scheduler. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sha2-ce-core.S | 19 +++++++++------- arch/arm64/crypto/sha2-ce-glue.c | 49 +++++++++++++++++++++++++++++++--------- 2 files changed, 49 insertions(+), 19 deletions(-) diff --git a/arch/arm64/crypto/sha2-ce-core.S b/arch/arm64/crypto/sha2-ce-core.S index 53e750614169..46b669d91c29 100644 --- a/arch/arm64/crypto/sha2-ce-core.S +++ b/arch/arm64/crypto/sha2-ce-core.S @@ -73,8 +73,8 @@ .word 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2 /* - * void sha2_ce_transform(int blocks, u8 const *src, u32 *state, - * u8 *head, long bytes) + * int sha2_ce_transform(int blocks, u8 const *src, u32 *state, + * u8 *head, long bytes, struct thread_info *ti) */ ENTRY(sha2_ce_transform) /* load round constants */ @@ -131,7 +131,14 @@ CPU_LE( rev32 v19.16b, v19.16b ) add dgbv.4s, dgbv.4s, dg1v.4s /* handled all input blocks? */ - cbnz w0, 0b + cbz w0, 4f + + /* should we exit early? */ + b_if_no_resched x5, x8, 0b + + /* store new state */ +3: stp dga, dgb, [x2] + ret /* * Final block: add padding and total bit count. @@ -139,7 +146,7 @@ CPU_LE( rev32 v19.16b, v19.16b ) * size was not a round multiple of the block size, and the padding is * handled by the C code. */ - cbz x4, 3f +4: cbz x4, 3b movi v17.2d, #0 mov x8, #0x80000000 movi v18.2d, #0 @@ -149,8 +156,4 @@ CPU_LE( rev32 v19.16b, v19.16b ) mov v19.d[0], xzr mov v19.d[1], x7 b 2b - - /* store new state */ -3: stp dga, dgb, [x2] - ret ENDPROC(sha2_ce_transform) diff --git a/arch/arm64/crypto/sha2-ce-glue.c b/arch/arm64/crypto/sha2-ce-glue.c index 81617262b3df..333d6427f7b6 100644 --- a/arch/arm64/crypto/sha2-ce-glue.c +++ b/arch/arm64/crypto/sha2-ce-glue.c @@ -20,8 +20,8 @@ MODULE_DESCRIPTION("SHA-224/SHA-256 secure hash using ARMv8 Crypto Extensions"); MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); -asmlinkage void sha2_ce_transform(int blocks, u8 const *src, u32 *state, - u8 *head, long bytes); +asmlinkage int sha2_ce_transform(int blocks, u8 const *src, u32 *state, + u8 *head, long bytes, struct thread_info *ti); static int sha224_init(struct shash_desc *desc) { @@ -58,6 +58,7 @@ static int sha2_update(struct shash_desc *desc, const u8 *data, sctx->count += len; if ((partial + len) >= SHA256_BLOCK_SIZE) { + struct thread_info *ti = NULL; int blocks; if (partial) { @@ -68,16 +69,30 @@ static int sha2_update(struct shash_desc *desc, const u8 *data, len -= p; } + /* + * Pass current's thread info pointer to sha2_ce_transform() + * below if we want it to play nice under preemption. + */ + if ((IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY) || + IS_ENABLED(CONFIG_PREEMPT)) && !in_interrupt()) + ti = current_thread_info(); + blocks = len / SHA256_BLOCK_SIZE; len %= SHA256_BLOCK_SIZE; - kernel_neon_begin_partial(28); - sha2_ce_transform(blocks, data, sctx->state, - partial ? sctx->buf : NULL, 0); - kernel_neon_end(); + do { + int rem; + + kernel_neon_begin_partial(28); + rem = sha2_ce_transform(blocks, data, sctx->state, + partial ? sctx->buf : NULL, + 0, ti); + kernel_neon_end(); - data += blocks * SHA256_BLOCK_SIZE; - partial = 0; + data += (blocks - rem) * SHA256_BLOCK_SIZE; + blocks = rem; + partial = 0; + } while (unlikely(ti && blocks > 0)); } if (len) memcpy(sctx->buf + partial, data, len); @@ -131,6 +146,7 @@ static void sha2_finup(struct shash_desc *desc, const u8 *data, unsigned int len) { struct sha256_state *sctx = shash_desc_ctx(desc); + struct thread_info *ti = NULL; int blocks; if (sctx->count || !len || (len % SHA256_BLOCK_SIZE)) { @@ -147,9 +163,20 @@ static void sha2_finup(struct shash_desc *desc, const u8 *data, */ blocks = len / SHA256_BLOCK_SIZE; - kernel_neon_begin_partial(28); - sha2_ce_transform(blocks, data, sctx->state, NULL, len); - kernel_neon_end(); + if ((IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY) || + IS_ENABLED(CONFIG_PREEMPT)) && !in_interrupt()) + ti = current_thread_info(); + + do { + int rem; + + kernel_neon_begin_partial(28); + rem = sha2_ce_transform(blocks, data, sctx->state, + NULL, len, ti); + kernel_neon_end(); + data += (blocks - rem) * SHA256_BLOCK_SIZE; + blocks = rem; + } while (unlikely(ti && blocks > 0)); } static int sha224_finup(struct shash_desc *desc, const u8 *data,