From patchwork Tue Apr 1 13:47:39 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 27547 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-ob0-f200.google.com (mail-ob0-f200.google.com [209.85.214.200]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 35BFA20341 for ; Tue, 1 Apr 2014 13:52:53 +0000 (UTC) Received: by mail-ob0-f200.google.com with SMTP id gq1sf34215140obb.7 for ; Tue, 01 Apr 2014 06:52:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:subject:date:message-id :in-reply-to:references:cc:precedence:list-id:list-unsubscribe :list-archive:list-post:list-help:list-subscribe:mime-version:sender :errors-to:x-original-sender:x-original-authentication-results :mailing-list:content-type:content-transfer-encoding; bh=96YMF6G52fcTasD7iyrR/DMCIMxoD2G6XShh3MoI7KE=; b=JF2JHjZfRWGYMaceCM4uXgPZTUUGX2sV03m1Dp09kHw/1RhC7q7ZBe7+H7hDmYshhI NX7qkchetne8dGZGstdYI+NhyEGxYPgP/p4BAg0II44Ub6IKEnHVjMlBe1MvgeY+qCdg 8CeFPFhALZYSixkrRxpP2Wt7Tnc/AvesfBrOh4U841Rf8j0GeQb5sGjIeN6htq4vhdbC vlnoE/u4cEKIMpUkBPfdrnoTRRr4WWcWtDGRoLPHbQgl7VjSvH5V8Dt1Wdr41xqUlVAv FT1V5z9u0iRqtjcB6zYzD7dH8JmXcg8L8HhKpvkwMD9qUTpIHwE5W/eA8zH4yneNT5qC GYFw== X-Gm-Message-State: ALoCoQlCbjapH2EJxYPach06yEEdpMBSu8aBliTeexDk/GJa/J4BweMdjOIeX/55xZTDKvc2CRgD X-Received: by 10.182.60.37 with SMTP id e5mr13264682obr.30.1396360372740; Tue, 01 Apr 2014 06:52:52 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.33.197 with SMTP id j63ls11580qgj.4.gmail; Tue, 01 Apr 2014 06:52:52 -0700 (PDT) X-Received: by 10.52.125.170 with SMTP id mr10mr90547vdb.71.1396360372641; Tue, 01 Apr 2014 06:52:52 -0700 (PDT) Received: from mail-vc0-f169.google.com (mail-vc0-f169.google.com [209.85.220.169]) by mx.google.com with ESMTPS id tz5si3619866vdc.187.2014.04.01.06.52.52 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 01 Apr 2014 06:52:52 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.169 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.220.169; Received: by mail-vc0-f169.google.com with SMTP id ik5so10070701vcb.0 for ; Tue, 01 Apr 2014 06:52:52 -0700 (PDT) X-Received: by 10.220.237.146 with SMTP id ko18mr171759vcb.49.1396360371972; Tue, 01 Apr 2014 06:52:51 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.12.8 with SMTP id v8csp239229vcv; Tue, 1 Apr 2014 06:52:51 -0700 (PDT) X-Received: by 10.194.60.114 with SMTP id g18mr2006247wjr.61.1396360371073; Tue, 01 Apr 2014 06:52:51 -0700 (PDT) Received: from casper.infradead.org (casper.infradead.org. [2001:770:15f::2]) by mx.google.com with ESMTPS id wx5si5020638wjb.53.2014.04.01.06.52.50 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 01 Apr 2014 06:52:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org designates 2001:770:15f::2 as permitted sender) client-ip=2001:770:15f::2; Received: from merlin.infradead.org ([2001:4978:20e::2]) by casper.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1WUz47-0007b4-Sw; Tue, 01 Apr 2014 13:49:24 +0000 Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1WUz3d-0001v9-9d; Tue, 01 Apr 2014 13:48:53 +0000 Received: from mail-we0-f180.google.com ([74.125.82.180]) by merlin.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1WUz32-0001oZ-P4 for linux-arm-kernel@lists.infradead.org; Tue, 01 Apr 2014 13:48:21 +0000 Received: by mail-we0-f180.google.com with SMTP id p61so6366163wes.11 for ; Tue, 01 Apr 2014 06:47:55 -0700 (PDT) X-Received: by 10.194.174.100 with SMTP id br4mr4078293wjc.83.1396360074596; Tue, 01 Apr 2014 06:47:54 -0700 (PDT) Received: from ards-macbook-pro.local ([95.129.121.210]) by mx.google.com with ESMTPSA id 48sm40727180eee.2.2014.04.01.06.47.53 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 01 Apr 2014 06:47:53 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, nico@linaro.org Subject: [PATCH v3 7/7] arm64/crypto: add voluntary preemption to Crypto Extensions GHASH Date: Tue, 1 Apr 2014 15:47:39 +0200 Message-Id: <1396360059-31949-8-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 1.8.3.2 In-Reply-To: <1396360059-31949-1-git-send-email-ard.biesheuvel@linaro.org> References: <1396360059-31949-1-git-send-email-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20140401_094816_976949_1284E611 X-CRM114-Status: GOOD ( 13.10 ) X-Spam-Score: -2.6 (--) X-Spam-Report: SpamAssassin version 3.3.2 on merlin.infradead.org summary: Content analysis details: (-2.6 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low trust [74.125.82.180 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] Cc: Ard Biesheuvel X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: ard.biesheuvel@linaro.org X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.220.169 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 The Crypto Extensions based GHASH implementation uses the NEON register file, and hence runs with preemption disabled. This patch adds a TIF_NEED_RESCHED check to its inner loop so we at least give up the CPU voluntarily when if we are running in process context and have been tagged for preemption by the scheduler. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/ghash-ce-core.S | 10 ++++++---- arch/arm64/crypto/ghash-ce-glue.c | 33 +++++++++++++++++++++++++-------- 2 files changed, 31 insertions(+), 12 deletions(-) diff --git a/arch/arm64/crypto/ghash-ce-core.S b/arch/arm64/crypto/ghash-ce-core.S index 1ca719ce9323..14240c6dc343 100644 --- a/arch/arm64/crypto/ghash-ce-core.S +++ b/arch/arm64/crypto/ghash-ce-core.S @@ -32,8 +32,9 @@ .align 3 /* - * void pmull_ghash_update(int blocks, u64 dg[], const char *src, - * struct ghash_key const *k, const char *head) + * int pmull_ghash_update(int blocks, u64 dg[], const char *src, + * struct ghash_key const *k, const char *head, + * struct thread_info *ti) */ ENTRY(pmull_ghash_update) ld1 {DATA.16b}, [x1] @@ -89,8 +90,9 @@ CPU_LE( rev64 IN1.16b, IN1.16b ) eor T1.16b, T1.16b, T2.16b eor DATA.16b, DATA.16b, T1.16b - cbnz w0, 0b + cbz w0, 2f + b_if_no_resched x5, x7, 0b - st1 {DATA.16b}, [x1] +2: st1 {DATA.16b}, [x1] ret ENDPROC(pmull_ghash_update) diff --git a/arch/arm64/crypto/ghash-ce-glue.c b/arch/arm64/crypto/ghash-ce-glue.c index b92baf3f68c7..4df64832617d 100644 --- a/arch/arm64/crypto/ghash-ce-glue.c +++ b/arch/arm64/crypto/ghash-ce-glue.c @@ -33,8 +33,9 @@ struct ghash_desc_ctx { u32 count; }; -asmlinkage void pmull_ghash_update(int blocks, u64 dg[], const char *src, - struct ghash_key const *k, const char *head); +asmlinkage int pmull_ghash_update(int blocks, u64 dg[], const char *src, + struct ghash_key const *k, const char *head, + struct thread_info *ti); static int ghash_init(struct shash_desc *desc) { @@ -54,6 +55,7 @@ static int ghash_update(struct shash_desc *desc, const u8 *src, if ((partial + len) >= GHASH_BLOCK_SIZE) { struct ghash_key *key = crypto_shash_ctx(desc->tfm); + struct thread_info *ti = NULL; int blocks; if (partial) { @@ -64,14 +66,29 @@ static int ghash_update(struct shash_desc *desc, const u8 *src, len -= p; } + /* + * Pass current's thread info pointer to pmull_ghash_update() + * below if we want it to play nice under preemption. + */ + if ((IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY) || + IS_ENABLED(CONFIG_PREEMPT)) && !in_interrupt()) + ti = current_thread_info(); + blocks = len / GHASH_BLOCK_SIZE; len %= GHASH_BLOCK_SIZE; - kernel_neon_begin_partial(6); - pmull_ghash_update(blocks, ctx->digest, src, key, - partial ? ctx->buf : NULL); - kernel_neon_end(); - src += blocks * GHASH_BLOCK_SIZE; + do { + int rem; + + kernel_neon_begin_partial(6); + rem = pmull_ghash_update(blocks, ctx->digest, src, key, + partial ? ctx->buf : NULL, ti); + kernel_neon_end(); + + src += (blocks - rem) * GHASH_BLOCK_SIZE; + blocks = rem; + partial = 0; + } while (unlikely(ti && blocks > 0)); } if (len) memcpy(ctx->buf + partial, src, len); @@ -89,7 +106,7 @@ static int ghash_final(struct shash_desc *desc, u8 *dst) memset(ctx->buf + partial, 0, GHASH_BLOCK_SIZE - partial); kernel_neon_begin_partial(6); - pmull_ghash_update(1, ctx->digest, ctx->buf, key, NULL); + pmull_ghash_update(1, ctx->digest, ctx->buf, key, NULL, NULL); kernel_neon_end(); } put_unaligned_be64(ctx->digest[1], dst);