From patchwork Thu Mar 27 17:14:40 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 27233 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-ig0-f199.google.com (mail-ig0-f199.google.com [209.85.213.199]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id C65D0202FA for ; Thu, 27 Mar 2014 17:14:34 +0000 (UTC) Received: by mail-ig0-f199.google.com with SMTP id t19sf4894911igi.2 for ; Thu, 27 Mar 2014 10:14:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:delivered-to:from:to:cc:subject :date:message-id:sender:precedence:list-id:x-original-sender :x-original-authentication-results:mailing-list:list-post:list-help :list-archive:list-unsubscribe; bh=VmKymFA50Y74uVy4VlbHAqaAHc+kwhoUYJeV1ad9zE0=; b=ODsEZKTKAUI5Z74X7OJn9/fbVY1i5kOIT2qWacky/At/TUQz9w1gqGAsyoKwZw2pRF jDou9uIVmGKK9FOtqREV2LWeTWiilnsaZfVkq6wTr+FqGbR+6KFSxKs3p+a33QCW97uL rSTDv/ntysQ3i6EJisLm69b4PJfxZ+Nppdi1KPFJwz76CtscmHxmCTMDuLK4o3dzuvlJ FTcMn0uDmzyvPyAxCDxwDKoqmUDp76z8UekdpGUFUkAaccJ7VokHBsnc+rIvSycoPN90 NsuVgCzjCaBGI6xxL1gYk5AV1Df9FDHajJUvO1c3um1egA0kLcS7Rvyp4MBn01eXEwS2 fxug== X-Gm-Message-State: ALoCoQlJB9bSibX4LQ5jLaBOkTZEVh1aghjNq/Wjnjw+12ltBUUR9GIXTdRVKdfJoPFKydkc3N0J X-Received: by 10.43.88.73 with SMTP id az9mr1008484icc.5.1395940474284; Thu, 27 Mar 2014 10:14:34 -0700 (PDT) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.25.196 with SMTP id 62ls1088904qgt.76.gmail; Thu, 27 Mar 2014 10:14:34 -0700 (PDT) X-Received: by 10.221.55.133 with SMTP id vy5mr2387649vcb.17.1395940474071; Thu, 27 Mar 2014 10:14:34 -0700 (PDT) Received: from mail-vc0-f181.google.com (mail-vc0-f181.google.com [209.85.220.181]) by mx.google.com with ESMTPS id bn6si593001vcb.186.2014.03.27.10.14.34 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 27 Mar 2014 10:14:34 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.181 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.220.181; Received: by mail-vc0-f181.google.com with SMTP id id10so4534019vcb.26 for ; Thu, 27 Mar 2014 10:14:34 -0700 (PDT) X-Received: by 10.221.22.71 with SMTP id qv7mr1290592vcb.34.1395940473968; Thu, 27 Mar 2014 10:14:33 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.78.9 with SMTP id i9csp45172vck; Thu, 27 Mar 2014 10:14:33 -0700 (PDT) X-Received: by 10.66.148.134 with SMTP id ts6mr3005830pab.113.1395940472511; Thu, 27 Mar 2014 10:14:32 -0700 (PDT) Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m9si1961764pab.167.2014.03.27.10.14.32; Thu, 27 Mar 2014 10:14:32 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756172AbaC0RO1 (ORCPT + 1 other); Thu, 27 Mar 2014 13:14:27 -0400 Received: from mail-we0-f176.google.com ([74.125.82.176]:39095 "EHLO mail-we0-f176.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756045AbaC0ROV (ORCPT ); Thu, 27 Mar 2014 13:14:21 -0400 Received: by mail-we0-f176.google.com with SMTP id x48so2019005wes.35 for ; Thu, 27 Mar 2014 10:14:20 -0700 (PDT) X-Received: by 10.180.101.166 with SMTP id fh6mr6616147wib.2.1395940460391; Thu, 27 Mar 2014 10:14:20 -0700 (PDT) Received: from ards-macbook-pro.local (cag06-7-83-153-85-71.fbx.proxad.net. [83.153.85.71]) by mx.google.com with ESMTPSA id d6sm9009037wiz.4.2014.03.27.10.14.18 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 27 Mar 2014 10:14:19 -0700 (PDT) From: Ard Biesheuvel To: herbert@gondor.apana.org.au, linux-crypto@vger.kernel.org, x86@kernel.org, hpa@zytor.com Cc: Ard Biesheuvel Subject: [PATCH v2] x86/crypto: ghash: use C implementation for setkey() Date: Thu, 27 Mar 2014 18:14:40 +0100 Message-Id: <1395940480-3827-1-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 1.8.3.2 Sender: linux-crypto-owner@vger.kernel.org Precedence: list List-ID: X-Mailing-List: linux-crypto@vger.kernel.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: ard.biesheuvel@linaro.org X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.220.181 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , The GHASH setkey() function uses SSE registers but fails to call kernel_fpu_begin()/kernel_fpu_end(). Instead of adding these calls, and then having to deal with the restriction that they cannot be called from interrupt context, move the setkey() implementation to the C domain. Note that setkey() does not use any particular SSE features and is not expected to become a performance bottleneck. Signed-off-by: Ard Biesheuvel Acked-by: H. Peter Anvin Fixes: 0e1227d356e9b (crypto: ghash - Add PCLMULQDQ accelerated implementation) --- Changes since v1: - added hpa's ack and 'fixes' line for cc stable annotation - removed clmul_ghash_setkey() forward declaration arch/x86/crypto/ghash-clmulni-intel_asm.S | 29 ----------------------------- arch/x86/crypto/ghash-clmulni-intel_glue.c | 14 +++++++++++--- 2 files changed, 11 insertions(+), 32 deletions(-) diff --git a/arch/x86/crypto/ghash-clmulni-intel_asm.S b/arch/x86/crypto/ghash-clmulni-intel_asm.S index 586f41aac361..185fad49d86f 100644 --- a/arch/x86/crypto/ghash-clmulni-intel_asm.S +++ b/arch/x86/crypto/ghash-clmulni-intel_asm.S @@ -24,10 +24,6 @@ .align 16 .Lbswap_mask: .octa 0x000102030405060708090a0b0c0d0e0f -.Lpoly: - .octa 0xc2000000000000000000000000000001 -.Ltwo_one: - .octa 0x00000001000000000000000000000001 #define DATA %xmm0 #define SHASH %xmm1 @@ -134,28 +130,3 @@ ENTRY(clmul_ghash_update) .Lupdate_just_ret: ret ENDPROC(clmul_ghash_update) - -/* - * void clmul_ghash_setkey(be128 *shash, const u8 *key); - * - * Calculate hash_key << 1 mod poly - */ -ENTRY(clmul_ghash_setkey) - movaps .Lbswap_mask, BSWAP - movups (%rsi), %xmm0 - PSHUFB_XMM BSWAP %xmm0 - movaps %xmm0, %xmm1 - psllq $1, %xmm0 - psrlq $63, %xmm1 - movaps %xmm1, %xmm2 - pslldq $8, %xmm1 - psrldq $8, %xmm2 - por %xmm1, %xmm0 - # reduction - pshufd $0b00100100, %xmm2, %xmm1 - pcmpeqd .Ltwo_one, %xmm1 - pand .Lpoly, %xmm1 - pxor %xmm1, %xmm0 - movups %xmm0, (%rdi) - ret -ENDPROC(clmul_ghash_setkey) diff --git a/arch/x86/crypto/ghash-clmulni-intel_glue.c b/arch/x86/crypto/ghash-clmulni-intel_glue.c index 6759dd1135be..d785cf2c529c 100644 --- a/arch/x86/crypto/ghash-clmulni-intel_glue.c +++ b/arch/x86/crypto/ghash-clmulni-intel_glue.c @@ -30,8 +30,6 @@ void clmul_ghash_mul(char *dst, const be128 *shash); void clmul_ghash_update(char *dst, const char *src, unsigned int srclen, const be128 *shash); -void clmul_ghash_setkey(be128 *shash, const u8 *key); - struct ghash_async_ctx { struct cryptd_ahash *cryptd_tfm; }; @@ -58,13 +56,23 @@ static int ghash_setkey(struct crypto_shash *tfm, const u8 *key, unsigned int keylen) { struct ghash_ctx *ctx = crypto_shash_ctx(tfm); + be128 *x = (be128 *)key; + u64 a, b; if (keylen != GHASH_BLOCK_SIZE) { crypto_shash_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN); return -EINVAL; } - clmul_ghash_setkey(&ctx->shash, key); + /* perform multiplication by 'x' in GF(2^128) */ + a = be64_to_cpu(x->a); + b = be64_to_cpu(x->b); + + ctx->shash.a = (__be64)((b << 1) | (a >> 63)); + ctx->shash.b = (__be64)((a << 1) | (b >> 63)); + + if (a >> 63) + ctx->shash.b ^= cpu_to_be64(0xc2); return 0; }