From patchwork Mon Apr 30 16:18:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 134719 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp3948345lji; Mon, 30 Apr 2018 09:19:04 -0700 (PDT) X-Google-Smtp-Source: AB8JxZqESADwYTW1Xygmr/i91aJ75rD7+ZVBYZJpzvFVvwcDfQ8mzXrg+4Ohh9PZTaxqYTiMyRaH X-Received: by 2002:a65:5244:: with SMTP id q4-v6mr10405454pgp.201.1525105144529; Mon, 30 Apr 2018 09:19:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525105144; cv=none; d=google.com; s=arc-20160816; b=kuonh+WNk1rHqDbJA9SfA3NoXfSOOTXIBF3v+GWMgYTLRJLXblBcV9OlRpPtn0PMV/ RHAgMB3MUmJyC/jryHwaxC1ZxmzRFzVDZmG1kYtv/6Hp5DIxhW9ii0VdXxnfIdvqt2CV 4B9k4pqkPHKcJCTXmIflO1kvFFvW7vaEbZJxs7dShSAvAOWv52vgbER/zSjo57N8cNBc dC3fCRF1Hc/W+DQrsMdmYpPlmt3YXCrZD2JdPB//gwT5KehhAWYAbfKHR9Lqgm9/KyjP VycLOs2IyahhVHbxgQqC4MJLnLykFzHKwqhEN0DbG6BQKYW7h7hAxn7sCceLFvf7Gdyr xBXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=bOSfwQobMJzi1OJdCB+etKpYG/fpkT32nfTf6cOmpZQ=; b=yemYxIx/Man9yabIn2tqFuv5zPEWt1y0Yp5+zVuAT9IHzX9XDS/j8qe8KmprBaQUHz pHZsuOO9vGCszEo9UrHIebW00xOlOGekfcfxaTxmWNATfriIJDCxQE6ksc0SHAXprMnh XcPIfbStYKxEOnWGrncMFicI1D6fpAmtCUnyUBdQZEQYZ644ScQ0C7lrJ3JlQoCwE55/ wYiV+APnGysE75RrE1IF4b2d4sKM06ijNUnvQhbifdMknp35Z6eO6qFJk1movAVyrs+E t1NhYs7iDT3L5H4d2mkfkscQgWHOQPew7dO93lkvOsR7Twqq79BZo2H3AmZAZ9LXDzYh 4Qjg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Qbd0+15Y; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f66-v6si5264259pgc.391.2018.04.30.09.19.04; Mon, 30 Apr 2018 09:19:04 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Qbd0+15Y; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754651AbeD3QTD (ORCPT + 1 other); Mon, 30 Apr 2018 12:19:03 -0400 Received: from mail-wr0-f193.google.com ([209.85.128.193]:38353 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753334AbeD3QTD (ORCPT ); Mon, 30 Apr 2018 12:19:03 -0400 Received: by mail-wr0-f193.google.com with SMTP id 94-v6so7263645wrf.5 for ; Mon, 30 Apr 2018 09:19:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=bOSfwQobMJzi1OJdCB+etKpYG/fpkT32nfTf6cOmpZQ=; b=Qbd0+15Y2pCybt6hKDfX+JqXZky05wobZH+2WaManVKCkyJSKsUj/wLDp9o0QnS55l 3l5CfqUo6w0Hd94kMfn49+pIflEC1FLZJa5PvWM+fAYTlpPtx5yE8xhMmtD6MACVCnsA 15NXn6EvcSnaYCV0/Fd0k0I4GSl6hrhSCycX8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=bOSfwQobMJzi1OJdCB+etKpYG/fpkT32nfTf6cOmpZQ=; b=Xo3yF2JVGooFEKcvabT8WO7f5CYPkChPC5Abc+Xed9fZSDexbDy9czGFk4z62dON3C rbTzbJ6rjQYL9qc2OMWkRCHiAD0TiU/vo63ZNWp8eoHYqxWgUQbvqoW3QDNAE/eY7jjK AUp28GqEnYBTa85a1n951qq+ZxK7t2dAljxCOzjAOqEMsznpHfuS36yH1EYcF0U/Ys/r aDznGQBTWNormBfNzYcAksS1p7DvsgTjAzmDSjyHaYv2vgfS5WJ54BpCS8nrN05IJ2mX NwSP12u374yCrL+eMWsZU8mtvGjDkbfsocWLLwuwg4Mf0SC0O7G4V4d/fcmjfpuGDcSA HyGg== X-Gm-Message-State: ALQs6tAfCxmCBgu01mkdekg6dKtsbdFxYdhO39c1Wid0KnxNB/+yZQKG BkuWzm60/VGc+QUYSTYl/qrdD503Lqk= X-Received: by 2002:adf:c104:: with SMTP id r4-v6mr9881361wre.84.1525105141405; Mon, 30 Apr 2018 09:19:01 -0700 (PDT) Received: from localhost.localdomain ([2a01:e35:3995:5470:200:1aff:fe1b:b328]) by smtp.gmail.com with ESMTPSA id l1-v6sm5753845wre.54.2018.04.30.09.18.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 30 Apr 2018 09:19:00 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au Cc: linux-arm-kernel@lists.infradead.org, dave.martin@arm.com, will.deacon@arm.com, Ard Biesheuvel Subject: [PATCH resend 09/10] crypto: arm64/sha3-ce - yield NEON after every block of input Date: Mon, 30 Apr 2018 18:18:29 +0200 Message-Id: <20180430161830.14892-10-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180430161830.14892-1-ard.biesheuvel@linaro.org> References: <20180430161830.14892-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Avoid excessive scheduling delays under a preemptible kernel by conditionally yielding the NEON after every block of input. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sha3-ce-core.S | 77 +++++++++++++------- 1 file changed, 50 insertions(+), 27 deletions(-) -- 2.17.0 diff --git a/arch/arm64/crypto/sha3-ce-core.S b/arch/arm64/crypto/sha3-ce-core.S index 332ad7530690..a7d587fa54f6 100644 --- a/arch/arm64/crypto/sha3-ce-core.S +++ b/arch/arm64/crypto/sha3-ce-core.S @@ -41,9 +41,16 @@ */ .text ENTRY(sha3_ce_transform) - /* load state */ - add x8, x0, #32 - ld1 { v0.1d- v3.1d}, [x0] + frame_push 4 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + +0: /* load state */ + add x8, x19, #32 + ld1 { v0.1d- v3.1d}, [x19] ld1 { v4.1d- v7.1d}, [x8], #32 ld1 { v8.1d-v11.1d}, [x8], #32 ld1 {v12.1d-v15.1d}, [x8], #32 @@ -51,13 +58,13 @@ ENTRY(sha3_ce_transform) ld1 {v20.1d-v23.1d}, [x8], #32 ld1 {v24.1d}, [x8] -0: sub w2, w2, #1 +1: sub w21, w21, #1 mov w8, #24 adr_l x9, .Lsha3_rcon /* load input */ - ld1 {v25.8b-v28.8b}, [x1], #32 - ld1 {v29.8b-v31.8b}, [x1], #24 + ld1 {v25.8b-v28.8b}, [x20], #32 + ld1 {v29.8b-v31.8b}, [x20], #24 eor v0.8b, v0.8b, v25.8b eor v1.8b, v1.8b, v26.8b eor v2.8b, v2.8b, v27.8b @@ -66,10 +73,10 @@ ENTRY(sha3_ce_transform) eor v5.8b, v5.8b, v30.8b eor v6.8b, v6.8b, v31.8b - tbnz x3, #6, 2f // SHA3-512 + tbnz x22, #6, 3f // SHA3-512 - ld1 {v25.8b-v28.8b}, [x1], #32 - ld1 {v29.8b-v30.8b}, [x1], #16 + ld1 {v25.8b-v28.8b}, [x20], #32 + ld1 {v29.8b-v30.8b}, [x20], #16 eor v7.8b, v7.8b, v25.8b eor v8.8b, v8.8b, v26.8b eor v9.8b, v9.8b, v27.8b @@ -77,34 +84,34 @@ ENTRY(sha3_ce_transform) eor v11.8b, v11.8b, v29.8b eor v12.8b, v12.8b, v30.8b - tbnz x3, #4, 1f // SHA3-384 or SHA3-224 + tbnz x22, #4, 2f // SHA3-384 or SHA3-224 // SHA3-256 - ld1 {v25.8b-v28.8b}, [x1], #32 + ld1 {v25.8b-v28.8b}, [x20], #32 eor v13.8b, v13.8b, v25.8b eor v14.8b, v14.8b, v26.8b eor v15.8b, v15.8b, v27.8b eor v16.8b, v16.8b, v28.8b - b 3f + b 4f -1: tbz x3, #2, 3f // bit 2 cleared? SHA-384 +2: tbz x22, #2, 4f // bit 2 cleared? SHA-384 // SHA3-224 - ld1 {v25.8b-v28.8b}, [x1], #32 - ld1 {v29.8b}, [x1], #8 + ld1 {v25.8b-v28.8b}, [x20], #32 + ld1 {v29.8b}, [x20], #8 eor v13.8b, v13.8b, v25.8b eor v14.8b, v14.8b, v26.8b eor v15.8b, v15.8b, v27.8b eor v16.8b, v16.8b, v28.8b eor v17.8b, v17.8b, v29.8b - b 3f + b 4f // SHA3-512 -2: ld1 {v25.8b-v26.8b}, [x1], #16 +3: ld1 {v25.8b-v26.8b}, [x20], #16 eor v7.8b, v7.8b, v25.8b eor v8.8b, v8.8b, v26.8b -3: sub w8, w8, #1 +4: sub w8, w8, #1 eor3 v29.16b, v4.16b, v9.16b, v14.16b eor3 v26.16b, v1.16b, v6.16b, v11.16b @@ -183,17 +190,33 @@ ENTRY(sha3_ce_transform) eor v0.16b, v0.16b, v31.16b - cbnz w8, 3b - cbnz w2, 0b + cbnz w8, 4b + cbz w21, 5f + + if_will_cond_yield_neon + add x8, x19, #32 + st1 { v0.1d- v3.1d}, [x19] + st1 { v4.1d- v7.1d}, [x8], #32 + st1 { v8.1d-v11.1d}, [x8], #32 + st1 {v12.1d-v15.1d}, [x8], #32 + st1 {v16.1d-v19.1d}, [x8], #32 + st1 {v20.1d-v23.1d}, [x8], #32 + st1 {v24.1d}, [x8] + do_cond_yield_neon + b 0b + endif_yield_neon + + b 1b /* save state */ - st1 { v0.1d- v3.1d}, [x0], #32 - st1 { v4.1d- v7.1d}, [x0], #32 - st1 { v8.1d-v11.1d}, [x0], #32 - st1 {v12.1d-v15.1d}, [x0], #32 - st1 {v16.1d-v19.1d}, [x0], #32 - st1 {v20.1d-v23.1d}, [x0], #32 - st1 {v24.1d}, [x0] +5: st1 { v0.1d- v3.1d}, [x19], #32 + st1 { v4.1d- v7.1d}, [x19], #32 + st1 { v8.1d-v11.1d}, [x19], #32 + st1 {v12.1d-v15.1d}, [x19], #32 + st1 {v16.1d-v19.1d}, [x19], #32 + st1 {v20.1d-v23.1d}, [x19], #32 + st1 {v24.1d}, [x19] + frame_pop ret ENDPROC(sha3_ce_transform)