From patchwork Sat Mar 10 15:21:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 131291 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp2244332lja; Sat, 10 Mar 2018 07:22:34 -0800 (PST) X-Google-Smtp-Source: AG47ELsGsVBlBhElEYGgK8/D53RExmPRqrrvIsiV9jqYGQqAEYE49qwqFiB0DNH6S8UkhTZ0IzTA X-Received: by 2002:a17:902:8492:: with SMTP id c18-v6mr2342771plo.40.1520695354595; Sat, 10 Mar 2018 07:22:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520695354; cv=none; d=google.com; s=arc-20160816; b=LGpDB6g0FxK2sIosZa5xV+tKMtvyXiwijibdrPNnZEiecFy5/aHsk+6bPmPCAkGaiR mLGJgtnfprVgbI+1wHaPBmxEcYAqgm5OJ+KOagcJ2b9EqC1Nj/x73CW839yz6n7CPhup ZjiuAnoVbyDtrsQgUkDjXCzCkGHSJz4JIP0qbMIp44LT4KN3RVaNRmii5ElIe8lEAZZO Y9v2Y6sRGnWloHuiSopgqzntfX9j+WDNpq0mHnp44OWM5Yrbw07BKGzypK90wNpYHCk+ IVlPgvdeClnyv6nO1UFWwH/7r8ZhalRnaG3Vsm9TqL4358ELyQgtf8a6Rzzi5S7Yu9/r ZzIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=EUqJGq7kexprt6eJ1AfDCt3BJ0o5psDpuDk6fuvRfFc=; b=FlHgVsIxdy8tAVeBT9JEXUzEk92r5wuTOhYuxy+sLm73LC800DjbsM5DIVHssDF624 aEyMYCCaIryAAdjW8c+QMDRjC5J0GCyGrguE6G275UPt9i2fL/Cb3J8ZSnJBuQ7zVHg7 7HZuXVzUSua4k0D2S742Cw4cXNa3UisS+0ahITn0IVWbDcDNKfMCdUHyGpCxOomYRUAD of8G4PMKguUH4KO7b6R+zpM9Y+vApsLRGOiPDhkfN9KhANnFUW9MqW3bR3yzW2V0tmIk Y9Elx027+4A/31fkY0vOsKtVHp+c+Rid16mAHCAFm4WQHN4svodyH79uVfJB1lJ6bjoD ln+Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Ass9ti7N; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q16si2824191pfg.221.2018.03.10.07.22.34; Sat, 10 Mar 2018 07:22:34 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Ass9ti7N; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932209AbeCJPWc (ORCPT + 1 other); Sat, 10 Mar 2018 10:22:32 -0500 Received: from mail-wm0-f68.google.com ([74.125.82.68]:37564 "EHLO mail-wm0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932130AbeCJPWa (ORCPT ); Sat, 10 Mar 2018 10:22:30 -0500 Received: by mail-wm0-f68.google.com with SMTP id 139so8784766wmn.2 for ; Sat, 10 Mar 2018 07:22:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=EUqJGq7kexprt6eJ1AfDCt3BJ0o5psDpuDk6fuvRfFc=; b=Ass9ti7Nta8JgbO68bgbhq2aF7TzYBHpXQqai3Jc4lPoMsdTUHhjAW3+OCs+knlXKu 8xa1vG1dfX/kVTQ3x7HxOFjDqxxHr0wJdvxYV1sBIkkqU7DyiDEdkhGrAd3YmRDlqLnA K2TO3Zrt2WIYXIv3C/dMfjDv6Eqf+o2ci9k4k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=EUqJGq7kexprt6eJ1AfDCt3BJ0o5psDpuDk6fuvRfFc=; b=KmXsH9WwimOBgUW0xkFHc47ub1YPsIOKPtNlmuxR7m24pG5owlEwnNGZGwYskLroh2 oBgN4yVyRXg5AMyqckTjoGTlIHrLUhoLeBv1B6e+ExQebJjNBajYqebfKCbLSmqR46ad xfgoU2E8E2uF9a5F0s78ZG9ZMk5oC0Fy6KY5fcpsFNE7DQO5k2yjJT+gZWgOmZZVFgzM Lg++HX5jI1eXigCtKUBmXNa0WvmbLoJplTV/Plgzf2+4K79MYSU89mqbgkOdBG45HaFI 5ECZbfP7vJC9eEt6XuMhv80ZFsW7dFtpmW/oRE9a+X/bGCV+2KSrzT517aqbxz+lMfGv GPCg== X-Gm-Message-State: AElRT7Edcd7v3ZVdezNk50o9o6adciJ6RQfDyKOpVScFsZN1QvJx/lBM pGxXMAdATzUx2sbf/7z8si3iTDuumYU= X-Received: by 10.28.156.13 with SMTP id f13mr1449873wme.11.1520695348738; Sat, 10 Mar 2018 07:22:28 -0800 (PST) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id m9sm7027531wrf.13.2018.03.10.07.22.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:22:27 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v5 01/23] crypto: testmgr - add a new test case for CRC-T10DIF Date: Sat, 10 Mar 2018 15:21:46 +0000 Message-Id: <20180310152208.10369-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180310152208.10369-1-ard.biesheuvel@linaro.org> References: <20180310152208.10369-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org In order to be able to test yield support under preempt, add a test vector for CRC-T10DIF that is long enough to take multiple iterations (and thus possible preemption between them) of the primary loop of the accelerated x86 and arm64 implementations. Signed-off-by: Ard Biesheuvel --- crypto/testmgr.h | 259 ++++++++++++++++++++ 1 file changed, 259 insertions(+) -- 2.15.1 diff --git a/crypto/testmgr.h b/crypto/testmgr.h index 6044f6906bd6..52d9ff93beac 100644 --- a/crypto/testmgr.h +++ b/crypto/testmgr.h @@ -2044,6 +2044,265 @@ static const struct hash_testvec crct10dif_tv_template[] = { .digest = (u8 *)(u16 []){ 0x44c6 }, .np = 4, .tap = { 1, 255, 57, 6 }, + }, { + .plaintext = "\x6e\x05\x79\x10\xa7\x1b\xb2\x49" + "\xe0\x54\xeb\x82\x19\x8d\x24\xbb" + "\x2f\xc6\x5d\xf4\x68\xff\x96\x0a" + "\xa1\x38\xcf\x43\xda\x71\x08\x7c" + "\x13\xaa\x1e\xb5\x4c\xe3\x57\xee" + "\x85\x1c\x90\x27\xbe\x32\xc9\x60" + "\xf7\x6b\x02\x99\x0d\xa4\x3b\xd2" + "\x46\xdd\x74\x0b\x7f\x16\xad\x21" + "\xb8\x4f\xe6\x5a\xf1\x88\x1f\x93" + "\x2a\xc1\x35\xcc\x63\xfa\x6e\x05" + "\x9c\x10\xa7\x3e\xd5\x49\xe0\x77" + "\x0e\x82\x19\xb0\x24\xbb\x52\xe9" + "\x5d\xf4\x8b\x22\x96\x2d\xc4\x38" + "\xcf\x66\xfd\x71\x08\x9f\x13\xaa" + "\x41\xd8\x4c\xe3\x7a\x11\x85\x1c" + "\xb3\x27\xbe\x55\xec\x60\xf7\x8e" + "\x02\x99\x30\xc7\x3b\xd2\x69\x00" + "\x74\x0b\xa2\x16\xad\x44\xdb\x4f" + "\xe6\x7d\x14\x88\x1f\xb6\x2a\xc1" + "\x58\xef\x63\xfa\x91\x05\x9c\x33" + "\xca\x3e\xd5\x6c\x03\x77\x0e\xa5" + "\x19\xb0\x47\xde\x52\xe9\x80\x17" + "\x8b\x22\xb9\x2d\xc4\x5b\xf2\x66" + "\xfd\x94\x08\x9f\x36\xcd\x41\xd8" + "\x6f\x06\x7a\x11\xa8\x1c\xb3\x4a" + "\xe1\x55\xec\x83\x1a\x8e\x25\xbc" + "\x30\xc7\x5e\xf5\x69\x00\x97\x0b" + "\xa2\x39\xd0\x44\xdb\x72\x09\x7d" + "\x14\xab\x1f\xb6\x4d\xe4\x58\xef" + "\x86\x1d\x91\x28\xbf\x33\xca\x61" + "\xf8\x6c\x03\x9a\x0e\xa5\x3c\xd3" + "\x47\xde\x75\x0c\x80\x17\xae\x22" + "\xb9\x50\xe7\x5b\xf2\x89\x20\x94" + "\x2b\xc2\x36\xcd\x64\xfb\x6f\x06" + "\x9d\x11\xa8\x3f\xd6\x4a\xe1\x78" + "\x0f\x83\x1a\xb1\x25\xbc\x53\xea" + "\x5e\xf5\x8c\x00\x97\x2e\xc5\x39" + "\xd0\x67\xfe\x72\x09\xa0\x14\xab" + "\x42\xd9\x4d\xe4\x7b\x12\x86\x1d" + "\xb4\x28\xbf\x56\xed\x61\xf8\x8f" + "\x03\x9a\x31\xc8\x3c\xd3\x6a\x01" + "\x75\x0c\xa3\x17\xae\x45\xdc\x50" + "\xe7\x7e\x15\x89\x20\xb7\x2b\xc2" + "\x59\xf0\x64\xfb\x92\x06\x9d\x34" + "\xcb\x3f\xd6\x6d\x04\x78\x0f\xa6" + "\x1a\xb1\x48\xdf\x53\xea\x81\x18" + "\x8c\x23\xba\x2e\xc5\x5c\xf3\x67" + "\xfe\x95\x09\xa0\x37\xce\x42\xd9" + "\x70\x07\x7b\x12\xa9\x1d\xb4\x4b" + "\xe2\x56\xed\x84\x1b\x8f\x26\xbd" + "\x31\xc8\x5f\xf6\x6a\x01\x98\x0c" + "\xa3\x3a\xd1\x45\xdc\x73\x0a\x7e" + "\x15\xac\x20\xb7\x4e\xe5\x59\xf0" + "\x87\x1e\x92\x29\xc0\x34\xcb\x62" + "\xf9\x6d\x04\x9b\x0f\xa6\x3d\xd4" + "\x48\xdf\x76\x0d\x81\x18\xaf\x23" + "\xba\x51\xe8\x5c\xf3\x8a\x21\x95" + "\x2c\xc3\x37\xce\x65\xfc\x70\x07" + "\x9e\x12\xa9\x40\xd7\x4b\xe2\x79" + "\x10\x84\x1b\xb2\x26\xbd\x54\xeb" + "\x5f\xf6\x8d\x01\x98\x2f\xc6\x3a" + "\xd1\x68\xff\x73\x0a\xa1\x15\xac" + "\x43\xda\x4e\xe5\x7c\x13\x87\x1e" + "\xb5\x29\xc0\x57\xee\x62\xf9\x90" + "\x04\x9b\x32\xc9\x3d\xd4\x6b\x02" + "\x76\x0d\xa4\x18\xaf\x46\xdd\x51" + "\xe8\x7f\x16\x8a\x21\xb8\x2c\xc3" + "\x5a\xf1\x65\xfc\x93\x07\x9e\x35" + "\xcc\x40\xd7\x6e\x05\x79\x10\xa7" + "\x1b\xb2\x49\xe0\x54\xeb\x82\x19" + "\x8d\x24\xbb\x2f\xc6\x5d\xf4\x68" + "\xff\x96\x0a\xa1\x38\xcf\x43\xda" + "\x71\x08\x7c\x13\xaa\x1e\xb5\x4c" + "\xe3\x57\xee\x85\x1c\x90\x27\xbe" + "\x32\xc9\x60\xf7\x6b\x02\x99\x0d" + "\xa4\x3b\xd2\x46\xdd\x74\x0b\x7f" + "\x16\xad\x21\xb8\x4f\xe6\x5a\xf1" + "\x88\x1f\x93\x2a\xc1\x35\xcc\x63" + "\xfa\x6e\x05\x9c\x10\xa7\x3e\xd5" + "\x49\xe0\x77\x0e\x82\x19\xb0\x24" + "\xbb\x52\xe9\x5d\xf4\x8b\x22\x96" + "\x2d\xc4\x38\xcf\x66\xfd\x71\x08" + "\x9f\x13\xaa\x41\xd8\x4c\xe3\x7a" + "\x11\x85\x1c\xb3\x27\xbe\x55\xec" + "\x60\xf7\x8e\x02\x99\x30\xc7\x3b" + "\xd2\x69\x00\x74\x0b\xa2\x16\xad" + "\x44\xdb\x4f\xe6\x7d\x14\x88\x1f" + "\xb6\x2a\xc1\x58\xef\x63\xfa\x91" + "\x05\x9c\x33\xca\x3e\xd5\x6c\x03" + "\x77\x0e\xa5\x19\xb0\x47\xde\x52" + "\xe9\x80\x17\x8b\x22\xb9\x2d\xc4" + "\x5b\xf2\x66\xfd\x94\x08\x9f\x36" + "\xcd\x41\xd8\x6f\x06\x7a\x11\xa8" + "\x1c\xb3\x4a\xe1\x55\xec\x83\x1a" + "\x8e\x25\xbc\x30\xc7\x5e\xf5\x69" + "\x00\x97\x0b\xa2\x39\xd0\x44\xdb" + "\x72\x09\x7d\x14\xab\x1f\xb6\x4d" + "\xe4\x58\xef\x86\x1d\x91\x28\xbf" + "\x33\xca\x61\xf8\x6c\x03\x9a\x0e" + "\xa5\x3c\xd3\x47\xde\x75\x0c\x80" + "\x17\xae\x22\xb9\x50\xe7\x5b\xf2" + "\x89\x20\x94\x2b\xc2\x36\xcd\x64" + "\xfb\x6f\x06\x9d\x11\xa8\x3f\xd6" + "\x4a\xe1\x78\x0f\x83\x1a\xb1\x25" + "\xbc\x53\xea\x5e\xf5\x8c\x00\x97" + "\x2e\xc5\x39\xd0\x67\xfe\x72\x09" + "\xa0\x14\xab\x42\xd9\x4d\xe4\x7b" + "\x12\x86\x1d\xb4\x28\xbf\x56\xed" + "\x61\xf8\x8f\x03\x9a\x31\xc8\x3c" + "\xd3\x6a\x01\x75\x0c\xa3\x17\xae" + "\x45\xdc\x50\xe7\x7e\x15\x89\x20" + "\xb7\x2b\xc2\x59\xf0\x64\xfb\x92" + "\x06\x9d\x34\xcb\x3f\xd6\x6d\x04" + "\x78\x0f\xa6\x1a\xb1\x48\xdf\x53" + "\xea\x81\x18\x8c\x23\xba\x2e\xc5" + "\x5c\xf3\x67\xfe\x95\x09\xa0\x37" + "\xce\x42\xd9\x70\x07\x7b\x12\xa9" + "\x1d\xb4\x4b\xe2\x56\xed\x84\x1b" + "\x8f\x26\xbd\x31\xc8\x5f\xf6\x6a" + "\x01\x98\x0c\xa3\x3a\xd1\x45\xdc" + "\x73\x0a\x7e\x15\xac\x20\xb7\x4e" + "\xe5\x59\xf0\x87\x1e\x92\x29\xc0" + "\x34\xcb\x62\xf9\x6d\x04\x9b\x0f" + "\xa6\x3d\xd4\x48\xdf\x76\x0d\x81" + "\x18\xaf\x23\xba\x51\xe8\x5c\xf3" + "\x8a\x21\x95\x2c\xc3\x37\xce\x65" + "\xfc\x70\x07\x9e\x12\xa9\x40\xd7" + "\x4b\xe2\x79\x10\x84\x1b\xb2\x26" + "\xbd\x54\xeb\x5f\xf6\x8d\x01\x98" + "\x2f\xc6\x3a\xd1\x68\xff\x73\x0a" + "\xa1\x15\xac\x43\xda\x4e\xe5\x7c" + "\x13\x87\x1e\xb5\x29\xc0\x57\xee" + "\x62\xf9\x90\x04\x9b\x32\xc9\x3d" + "\xd4\x6b\x02\x76\x0d\xa4\x18\xaf" + "\x46\xdd\x51\xe8\x7f\x16\x8a\x21" + "\xb8\x2c\xc3\x5a\xf1\x65\xfc\x93" + "\x07\x9e\x35\xcc\x40\xd7\x6e\x05" + "\x79\x10\xa7\x1b\xb2\x49\xe0\x54" + "\xeb\x82\x19\x8d\x24\xbb\x2f\xc6" + "\x5d\xf4\x68\xff\x96\x0a\xa1\x38" + "\xcf\x43\xda\x71\x08\x7c\x13\xaa" + "\x1e\xb5\x4c\xe3\x57\xee\x85\x1c" + "\x90\x27\xbe\x32\xc9\x60\xf7\x6b" + "\x02\x99\x0d\xa4\x3b\xd2\x46\xdd" + "\x74\x0b\x7f\x16\xad\x21\xb8\x4f" + "\xe6\x5a\xf1\x88\x1f\x93\x2a\xc1" + "\x35\xcc\x63\xfa\x6e\x05\x9c\x10" + "\xa7\x3e\xd5\x49\xe0\x77\x0e\x82" + "\x19\xb0\x24\xbb\x52\xe9\x5d\xf4" + "\x8b\x22\x96\x2d\xc4\x38\xcf\x66" + "\xfd\x71\x08\x9f\x13\xaa\x41\xd8" + "\x4c\xe3\x7a\x11\x85\x1c\xb3\x27" + "\xbe\x55\xec\x60\xf7\x8e\x02\x99" + "\x30\xc7\x3b\xd2\x69\x00\x74\x0b" + "\xa2\x16\xad\x44\xdb\x4f\xe6\x7d" + "\x14\x88\x1f\xb6\x2a\xc1\x58\xef" + "\x63\xfa\x91\x05\x9c\x33\xca\x3e" + "\xd5\x6c\x03\x77\x0e\xa5\x19\xb0" + "\x47\xde\x52\xe9\x80\x17\x8b\x22" + "\xb9\x2d\xc4\x5b\xf2\x66\xfd\x94" + "\x08\x9f\x36\xcd\x41\xd8\x6f\x06" + "\x7a\x11\xa8\x1c\xb3\x4a\xe1\x55" + "\xec\x83\x1a\x8e\x25\xbc\x30\xc7" + "\x5e\xf5\x69\x00\x97\x0b\xa2\x39" + "\xd0\x44\xdb\x72\x09\x7d\x14\xab" + "\x1f\xb6\x4d\xe4\x58\xef\x86\x1d" + "\x91\x28\xbf\x33\xca\x61\xf8\x6c" + "\x03\x9a\x0e\xa5\x3c\xd3\x47\xde" + "\x75\x0c\x80\x17\xae\x22\xb9\x50" + "\xe7\x5b\xf2\x89\x20\x94\x2b\xc2" + "\x36\xcd\x64\xfb\x6f\x06\x9d\x11" + "\xa8\x3f\xd6\x4a\xe1\x78\x0f\x83" + "\x1a\xb1\x25\xbc\x53\xea\x5e\xf5" + "\x8c\x00\x97\x2e\xc5\x39\xd0\x67" + "\xfe\x72\x09\xa0\x14\xab\x42\xd9" + "\x4d\xe4\x7b\x12\x86\x1d\xb4\x28" + "\xbf\x56\xed\x61\xf8\x8f\x03\x9a" + "\x31\xc8\x3c\xd3\x6a\x01\x75\x0c" + "\xa3\x17\xae\x45\xdc\x50\xe7\x7e" + "\x15\x89\x20\xb7\x2b\xc2\x59\xf0" + "\x64\xfb\x92\x06\x9d\x34\xcb\x3f" + "\xd6\x6d\x04\x78\x0f\xa6\x1a\xb1" + "\x48\xdf\x53\xea\x81\x18\x8c\x23" + "\xba\x2e\xc5\x5c\xf3\x67\xfe\x95" + "\x09\xa0\x37\xce\x42\xd9\x70\x07" + "\x7b\x12\xa9\x1d\xb4\x4b\xe2\x56" + "\xed\x84\x1b\x8f\x26\xbd\x31\xc8" + "\x5f\xf6\x6a\x01\x98\x0c\xa3\x3a" + "\xd1\x45\xdc\x73\x0a\x7e\x15\xac" + "\x20\xb7\x4e\xe5\x59\xf0\x87\x1e" + "\x92\x29\xc0\x34\xcb\x62\xf9\x6d" + "\x04\x9b\x0f\xa6\x3d\xd4\x48\xdf" + "\x76\x0d\x81\x18\xaf\x23\xba\x51" + "\xe8\x5c\xf3\x8a\x21\x95\x2c\xc3" + "\x37\xce\x65\xfc\x70\x07\x9e\x12" + "\xa9\x40\xd7\x4b\xe2\x79\x10\x84" + "\x1b\xb2\x26\xbd\x54\xeb\x5f\xf6" + "\x8d\x01\x98\x2f\xc6\x3a\xd1\x68" + "\xff\x73\x0a\xa1\x15\xac\x43\xda" + "\x4e\xe5\x7c\x13\x87\x1e\xb5\x29" + "\xc0\x57\xee\x62\xf9\x90\x04\x9b" + "\x32\xc9\x3d\xd4\x6b\x02\x76\x0d" + "\xa4\x18\xaf\x46\xdd\x51\xe8\x7f" + "\x16\x8a\x21\xb8\x2c\xc3\x5a\xf1" + "\x65\xfc\x93\x07\x9e\x35\xcc\x40" + "\xd7\x6e\x05\x79\x10\xa7\x1b\xb2" + "\x49\xe0\x54\xeb\x82\x19\x8d\x24" + "\xbb\x2f\xc6\x5d\xf4\x68\xff\x96" + "\x0a\xa1\x38\xcf\x43\xda\x71\x08" + "\x7c\x13\xaa\x1e\xb5\x4c\xe3\x57" + "\xee\x85\x1c\x90\x27\xbe\x32\xc9" + "\x60\xf7\x6b\x02\x99\x0d\xa4\x3b" + "\xd2\x46\xdd\x74\x0b\x7f\x16\xad" + "\x21\xb8\x4f\xe6\x5a\xf1\x88\x1f" + "\x93\x2a\xc1\x35\xcc\x63\xfa\x6e" + "\x05\x9c\x10\xa7\x3e\xd5\x49\xe0" + "\x77\x0e\x82\x19\xb0\x24\xbb\x52" + "\xe9\x5d\xf4\x8b\x22\x96\x2d\xc4" + "\x38\xcf\x66\xfd\x71\x08\x9f\x13" + "\xaa\x41\xd8\x4c\xe3\x7a\x11\x85" + "\x1c\xb3\x27\xbe\x55\xec\x60\xf7" + "\x8e\x02\x99\x30\xc7\x3b\xd2\x69" + "\x00\x74\x0b\xa2\x16\xad\x44\xdb" + "\x4f\xe6\x7d\x14\x88\x1f\xb6\x2a" + "\xc1\x58\xef\x63\xfa\x91\x05\x9c" + "\x33\xca\x3e\xd5\x6c\x03\x77\x0e" + "\xa5\x19\xb0\x47\xde\x52\xe9\x80" + "\x17\x8b\x22\xb9\x2d\xc4\x5b\xf2" + "\x66\xfd\x94\x08\x9f\x36\xcd\x41" + "\xd8\x6f\x06\x7a\x11\xa8\x1c\xb3" + "\x4a\xe1\x55\xec\x83\x1a\x8e\x25" + "\xbc\x30\xc7\x5e\xf5\x69\x00\x97" + "\x0b\xa2\x39\xd0\x44\xdb\x72\x09" + "\x7d\x14\xab\x1f\xb6\x4d\xe4\x58" + "\xef\x86\x1d\x91\x28\xbf\x33\xca" + "\x61\xf8\x6c\x03\x9a\x0e\xa5\x3c" + "\xd3\x47\xde\x75\x0c\x80\x17\xae" + "\x22\xb9\x50\xe7\x5b\xf2\x89\x20" + "\x94\x2b\xc2\x36\xcd\x64\xfb\x6f" + "\x06\x9d\x11\xa8\x3f\xd6\x4a\xe1" + "\x78\x0f\x83\x1a\xb1\x25\xbc\x53" + "\xea\x5e\xf5\x8c\x00\x97\x2e\xc5" + "\x39\xd0\x67\xfe\x72\x09\xa0\x14" + "\xab\x42\xd9\x4d\xe4\x7b\x12\x86" + "\x1d\xb4\x28\xbf\x56\xed\x61\xf8" + "\x8f\x03\x9a\x31\xc8\x3c\xd3\x6a" + "\x01\x75\x0c\xa3\x17\xae\x45\xdc" + "\x50\xe7\x7e\x15\x89\x20\xb7\x2b" + "\xc2\x59\xf0\x64\xfb\x92\x06\x9d" + "\x34\xcb\x3f\xd6\x6d\x04\x78\x0f" + "\xa6\x1a\xb1\x48\xdf\x53\xea\x81" + "\x18\x8c\x23\xba\x2e\xc5\x5c\xf3" + "\x67\xfe\x95\x09\xa0\x37\xce\x42" + "\xd9\x70\x07\x7b\x12\xa9\x1d\xb4" + "\x4b\xe2\x56\xed\x84\x1b\x8f\x26" + "\xbd\x31\xc8\x5f\xf6\x6a\x01\x98", + .psize = 2048, + .digest = (u8 *)(u16 []){ 0x23ca }, } }; From patchwork Sat Mar 10 15:21:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 131292 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp2244379lja; Sat, 10 Mar 2018 07:22:38 -0800 (PST) X-Google-Smtp-Source: AG47ELs+Iv58ziBjgFPoZaJx+Lds2WPf/XN+oz8nRhO0oKGLL0j/9xNpmt3+8ZP2eVpFGb4qc17N X-Received: by 10.99.66.135 with SMTP id p129mr1941541pga.220.1520695358155; Sat, 10 Mar 2018 07:22:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520695358; cv=none; d=google.com; s=arc-20160816; b=SyVke4wsb01wM9NVwe2OVhw1dt7arn9WQNsUtznFPaIWUZDnqRKU4pYFJe+eArj076 Cyc5bjjvim1gyGeRXPK9iUQ/8WiwhafAlqZOC61XyFosSM5wzkPXU24Ipa59XVhOD8iP hKwMj6D8FwfXbJrrbfWK3Wm9kpvAclAXstHcXLPd+xtthcbCk50XQ+5Bf84ZdEVJnBuH Wlh6x22GsZNDkUQU/cK9fMlVnCACzDZIfWcdN9JVi/rANcP9EayoXBdVylJQ9eiqBwy8 OgmZnyDNPVTiQecFGYilPDBxBPTn9JdRDUwIBmL3/0I3OVjYI4F7IwpFqueWKLA7SOh8 hxow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=JdJLNL1b2SAPFDYe5cR+GVEkPFZb9F6GBJ002Vtvdw4=; b=SbOY5Xd8JIBPwTvSdRJUr+ASOr3pihEW3qpaHWNtQ4rY91eQuQSL8ARUP9KR5Pr7/w TljL/mxCtzIFmTuLjtBiTGdtBFGFtY0OyLWz2n3Ee2OCsGcE+EMbDK+2NQ4XMQxzGxJf ZmaJlM+XhArPL/fZI3Tzd3Y+kZKIH80tCzhj8jy/FjKzqihnftU/yjaLLSwMHIvuMLIg Bl2cCGpeC6sW1Dt83onz+R+cL8pf/1JsPD3dKDvyQG/ci+NZHexvKp2uDHOs4CAjCOfu qMsxFwPJ6rPKF5Zs1vpilUWFZW2570Lo6wXBcgjcaqvCcq4mVkDvU1ZusFuMn0RzHVIW EuXA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=AblMJqQ+; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q16si2824191pfg.221.2018.03.10.07.22.37; Sat, 10 Mar 2018 07:22:38 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=AblMJqQ+; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932227AbeCJPWf (ORCPT + 1 other); Sat, 10 Mar 2018 10:22:35 -0500 Received: from mail-wr0-f196.google.com ([209.85.128.196]:45833 "EHLO mail-wr0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932130AbeCJPWe (ORCPT ); Sat, 10 Mar 2018 10:22:34 -0500 Received: by mail-wr0-f196.google.com with SMTP id h2so4348236wre.12 for ; Sat, 10 Mar 2018 07:22:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=JdJLNL1b2SAPFDYe5cR+GVEkPFZb9F6GBJ002Vtvdw4=; b=AblMJqQ+0tQhrO7zvn+8UzEVPiExppjRyJfqIw/WOr6XAlbIvl0bJdgmS5QvbsdkBg pSCipKxlyJrsTx7671u8Y19ky0bKsnbWacgsWTPy6Re7Aq4rFyANXYLPMVfhMXvikt23 hJAe9XGqzBaIx6V3tEoaHPdQUcYOkFAR7O+a4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=JdJLNL1b2SAPFDYe5cR+GVEkPFZb9F6GBJ002Vtvdw4=; b=DtNIANqnNN5cz7YXL3c3YIY8Bp7iJLqYXTmOU8TXOm6kveGRzS2FhLThmDO/EaKSC/ l+s4VorkUu+GNKlljjRdi/uRCJPlqSlMM7nS7J+E7fAmNS6CX7/Rvt6v+onLz/4nEjJu dFg8gIsKW0LbkpilIHjGzET8WVUH34JFuJQsVS/egUzdW/+A8onV3kGhyFvh6meJsRXa Gh1BggTbDCooIBUlTOv38cZUmNHOxSLkdUfQ3zfrTEId+ja7nMZ9O25X4mIPjhxTYgdh byJ32wfcJdgP/H0P0E4HQLSzvM/IGZ8Y1B+o4wDHzv0oGN9MwgQukoc6sLh02KIQne1M wbbw== X-Gm-Message-State: AElRT7HC4+FKgVQOTxdFleD4AVAfusze2QJ5GOHBj9hM1j0kUnbCctJR X11Qf2T4GxqUR43t5sKdl6p8r7Mtczk= X-Received: by 10.223.134.136 with SMTP id 8mr1814997wrx.86.1520695352440; Sat, 10 Mar 2018 07:22:32 -0800 (PST) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id m9sm7027531wrf.13.2018.03.10.07.22.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:22:30 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v5 02/23] crypto: arm64/aes-ce-ccm - move kernel mode neon en/disable into loop Date: Sat, 10 Mar 2018 15:21:47 +0000 Message-Id: <20180310152208.10369-3-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180310152208.10369-1-ard.biesheuvel@linaro.org> References: <20180310152208.10369-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org When kernel mode NEON was first introduced on arm64, the preserve and restore of the userland NEON state was completely unoptimized, and involved saving all registers on each call to kernel_neon_begin(), and restoring them on each call to kernel_neon_end(). For this reason, the NEON crypto code that was introduced at the time keeps the NEON enabled throughout the execution of the crypto API methods, which may include calls back into the crypto API that could result in memory allocation or other actions that we should avoid when running with preemption disabled. Since then, we have optimized the kernel mode NEON handling, which now restores lazily (upon return to userland), and so the preserve action is only costly the first time it is called after entering the kernel. So let's put the kernel_neon_begin() and kernel_neon_end() calls around the actual invocations of the NEON crypto code, and run the remainder of the code with kernel mode NEON disabled (and preemption enabled) Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-ce-ccm-glue.c | 47 ++++++++++---------- 1 file changed, 23 insertions(+), 24 deletions(-) -- 2.15.1 diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce-ccm-glue.c index a1254036f2b1..68b11aa690e4 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -107,11 +107,13 @@ static int ccm_init_mac(struct aead_request *req, u8 maciv[], u32 msglen) } static void ccm_update_mac(struct crypto_aes_ctx *key, u8 mac[], u8 const in[], - u32 abytes, u32 *macp, bool use_neon) + u32 abytes, u32 *macp) { - if (likely(use_neon)) { + if (may_use_simd()) { + kernel_neon_begin(); ce_aes_ccm_auth_data(mac, in, abytes, macp, key->key_enc, num_rounds(key)); + kernel_neon_end(); } else { if (*macp > 0 && *macp < AES_BLOCK_SIZE) { int added = min(abytes, AES_BLOCK_SIZE - *macp); @@ -143,8 +145,7 @@ static void ccm_update_mac(struct crypto_aes_ctx *key, u8 mac[], u8 const in[], } } -static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[], - bool use_neon) +static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[]) { struct crypto_aead *aead = crypto_aead_reqtfm(req); struct crypto_aes_ctx *ctx = crypto_aead_ctx(aead); @@ -163,7 +164,7 @@ static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[], ltag.len = 6; } - ccm_update_mac(ctx, mac, (u8 *)<ag, ltag.len, &macp, use_neon); + ccm_update_mac(ctx, mac, (u8 *)<ag, ltag.len, &macp); scatterwalk_start(&walk, req->src); do { @@ -175,7 +176,7 @@ static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[], n = scatterwalk_clamp(&walk, len); } p = scatterwalk_map(&walk); - ccm_update_mac(ctx, mac, p, n, &macp, use_neon); + ccm_update_mac(ctx, mac, p, n, &macp); len -= n; scatterwalk_unmap(p); @@ -242,43 +243,42 @@ static int ccm_encrypt(struct aead_request *req) u8 __aligned(8) mac[AES_BLOCK_SIZE]; u8 buf[AES_BLOCK_SIZE]; u32 len = req->cryptlen; - bool use_neon = may_use_simd(); int err; err = ccm_init_mac(req, mac, len); if (err) return err; - if (likely(use_neon)) - kernel_neon_begin(); - if (req->assoclen) - ccm_calculate_auth_mac(req, mac, use_neon); + ccm_calculate_auth_mac(req, mac); /* preserve the original iv for the final round */ memcpy(buf, req->iv, AES_BLOCK_SIZE); err = skcipher_walk_aead_encrypt(&walk, req, true); - if (likely(use_neon)) { + if (may_use_simd()) { while (walk.nbytes) { u32 tail = walk.nbytes % AES_BLOCK_SIZE; if (walk.nbytes == walk.total) tail = 0; + kernel_neon_begin(); ce_aes_ccm_encrypt(walk.dst.virt.addr, walk.src.virt.addr, walk.nbytes - tail, ctx->key_enc, num_rounds(ctx), mac, walk.iv); + kernel_neon_end(); err = skcipher_walk_done(&walk, tail); } - if (!err) + if (!err) { + kernel_neon_begin(); ce_aes_ccm_final(mac, buf, ctx->key_enc, num_rounds(ctx)); - - kernel_neon_end(); + kernel_neon_end(); + } } else { err = ccm_crypt_fallback(&walk, mac, buf, ctx, true); } @@ -301,43 +301,42 @@ static int ccm_decrypt(struct aead_request *req) u8 __aligned(8) mac[AES_BLOCK_SIZE]; u8 buf[AES_BLOCK_SIZE]; u32 len = req->cryptlen - authsize; - bool use_neon = may_use_simd(); int err; err = ccm_init_mac(req, mac, len); if (err) return err; - if (likely(use_neon)) - kernel_neon_begin(); - if (req->assoclen) - ccm_calculate_auth_mac(req, mac, use_neon); + ccm_calculate_auth_mac(req, mac); /* preserve the original iv for the final round */ memcpy(buf, req->iv, AES_BLOCK_SIZE); err = skcipher_walk_aead_decrypt(&walk, req, true); - if (likely(use_neon)) { + if (may_use_simd()) { while (walk.nbytes) { u32 tail = walk.nbytes % AES_BLOCK_SIZE; if (walk.nbytes == walk.total) tail = 0; + kernel_neon_begin(); ce_aes_ccm_decrypt(walk.dst.virt.addr, walk.src.virt.addr, walk.nbytes - tail, ctx->key_enc, num_rounds(ctx), mac, walk.iv); + kernel_neon_end(); err = skcipher_walk_done(&walk, tail); } - if (!err) + if (!err) { + kernel_neon_begin(); ce_aes_ccm_final(mac, buf, ctx->key_enc, num_rounds(ctx)); - - kernel_neon_end(); + kernel_neon_end(); + } } else { err = ccm_crypt_fallback(&walk, mac, buf, ctx, false); } From patchwork Sat Mar 10 15:21:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 131294 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp2244488lja; Sat, 10 Mar 2018 07:22:46 -0800 (PST) X-Google-Smtp-Source: AG47ELtgx93VfEzBSij5USKGtZ2mk+ylkvZcLY7sVcShOrW6Je6pS0Fou9pxOj2+svoQNNGHznf0 X-Received: by 10.99.120.13 with SMTP id t13mr1932082pgc.35.1520695365902; Sat, 10 Mar 2018 07:22:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520695365; cv=none; d=google.com; s=arc-20160816; b=QWRDTmxQAUnbZYa2tvX//AmxNJaK6bAxMSz95yFS7AN8pt6yQAWJCF2OrV57mjXfMA +94T0hPHJOpHywdbM290+KUawNH+YiTHOjlt6lzMZBtSY8vlcqPgLjwj9SI8WOOmJPig ebA7pWIORn7N/gADnTALrJTruvTEl9JTHWEIz5BMn8KFehTvAKKr/hx1bGZr1tnSy3vg 3/2biljvYRk2JlL2O9Z+7ZliR6DH+y0WmZzVHtk6h7SEx+wsYmFrhEuV/7PzIIjS1zXk tD42GB+xDdnlnLErgUnhqEwFCPtaWUyLO/eT7RuwACfRZtVndb9soYrHoTNHIiLf6b5p dICw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=aVJMpK+ItX9Ku1Ngl1sTNCOZ4+v3QvBf89t8LQLjCgc=; b=I4Xx4X+xh4XeAlby1AEEaHp+S4+ZvSGA0+xMnObe2eivg8jGCD3Yzg3nKUxlsfFi0Z dBbtmbQnHPiSTReFbN1dWh6rPyjlhYqMkRD45eicNM7gB57bWr4AVqvgrq8BkDOFNVlp +qWnYowFG4bqHgVW1N3b0c+Z63BPKeC7GUBrUMLFCR4qVj/TJF6+Vnkos97hOmKbjO4i 42J4kitIRYQSZGay+I1y5QwvUwTVcOUDcKhwStLEo1bBfX+oGFHubSIf1lOvpRVWxOej UC5bBDSAn8FDc0BhvyLcdT4L83EjYzfa3GeIFqtRPqz7ynNW3g5BfVcYsk/PkgPyC7GS EVaw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=TiQWRhe7; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q16si2824191pfg.221.2018.03.10.07.22.45; Sat, 10 Mar 2018 07:22:45 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=TiQWRhe7; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932130AbeCJPWn (ORCPT + 1 other); Sat, 10 Mar 2018 10:22:43 -0500 Received: from mail-wr0-f195.google.com ([209.85.128.195]:45839 "EHLO mail-wr0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932271AbeCJPWl (ORCPT ); Sat, 10 Mar 2018 10:22:41 -0500 Received: by mail-wr0-f195.google.com with SMTP id h2so4348392wre.12 for ; Sat, 10 Mar 2018 07:22:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=aVJMpK+ItX9Ku1Ngl1sTNCOZ4+v3QvBf89t8LQLjCgc=; b=TiQWRhe7AyjhEUN2gPsRFJOR+jmW3N7aICigRu2p08JBVf2ouzFuBJhNoUI/7P/nHt lAP4yoy+vF0F0G9ve/9duqLja12a+Z/+Z5bCSNQYTWhJ8AnS+KoMJSlohWZfjL25/z5S VAcOKKv1tQYjEdt5A4TcS1qkEB/eNkrEHsnrI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=aVJMpK+ItX9Ku1Ngl1sTNCOZ4+v3QvBf89t8LQLjCgc=; b=Uw1vSGHxH2wU/535h032vrQ4clGVB4CfRhuiyz2foUk0fisvGGjlhwORjxJhPtsQaJ q8CYS3h8gyBJXEsoWNzX8aZ1Fy0oqp4cbhOHI9bVk53VP07qwZv5ExLzoRsmjX1jNdh7 sTVk5ik1ddC8G+7+HVKeWAiGrhAzGE1eb543OfEIGHD4eBAjQ5oEOjGFE7JTxkLD7kBS qd76KNLel+dLicJgAgKdZh3tLPs5n/NhWtMPwAAXMXLVDO1uaf6e9TRwAJBueVLxDrQ6 86INWdHFihuhOyU1uNu5RSvcc4TzbUHIy7IxxHtPrAhZEdtVoxgp4ij8rivnWTcgpTWT 87XA== X-Gm-Message-State: AElRT7HCiC8s0l1ayIVP5kCnDq0xvjxIFJjtlOAcJLC0ZOx8k3fPDrHZ LPzkM/H8YiJomlKhbP+kVRNyk2aPk74= X-Received: by 10.223.176.86 with SMTP id g22mr1718146wra.11.1520695360376; Sat, 10 Mar 2018 07:22:40 -0800 (PST) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id m9sm7027531wrf.13.2018.03.10.07.22.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:22:39 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v5 04/23] crypto: arm64/aes-bs - move kernel mode neon en/disable into loop Date: Sat, 10 Mar 2018 15:21:49 +0000 Message-Id: <20180310152208.10369-5-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180310152208.10369-1-ard.biesheuvel@linaro.org> References: <20180310152208.10369-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org When kernel mode NEON was first introduced on arm64, the preserve and restore of the userland NEON state was completely unoptimized, and involved saving all registers on each call to kernel_neon_begin(), and restoring them on each call to kernel_neon_end(). For this reason, the NEON crypto code that was introduced at the time keeps the NEON enabled throughout the execution of the crypto API methods, which may include calls back into the crypto API that could result in memory allocation or other actions that we should avoid when running with preemption disabled. Since then, we have optimized the kernel mode NEON handling, which now restores lazily (upon return to userland), and so the preserve action is only costly the first time it is called after entering the kernel. So let's put the kernel_neon_begin() and kernel_neon_end() calls around the actual invocations of the NEON crypto code, and run the remainder of the code with kernel mode NEON disabled (and preemption enabled) Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-neonbs-glue.c | 36 +++++++++----------- 1 file changed, 17 insertions(+), 19 deletions(-) -- 2.15.1 diff --git a/arch/arm64/crypto/aes-neonbs-glue.c b/arch/arm64/crypto/aes-neonbs-glue.c index 9d823c77ec84..e7a95a566462 100644 --- a/arch/arm64/crypto/aes-neonbs-glue.c +++ b/arch/arm64/crypto/aes-neonbs-glue.c @@ -99,9 +99,8 @@ static int __ecb_crypt(struct skcipher_request *req, struct skcipher_walk walk; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; @@ -109,12 +108,13 @@ static int __ecb_crypt(struct skcipher_request *req, blocks = round_down(blocks, walk.stride / AES_BLOCK_SIZE); + kernel_neon_begin(); fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->rk, ctx->rounds, blocks); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } - kernel_neon_end(); return err; } @@ -158,19 +158,19 @@ static int cbc_encrypt(struct skcipher_request *req) struct skcipher_walk walk; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; /* fall back to the non-bitsliced NEON implementation */ + kernel_neon_begin(); neon_aes_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->enc, ctx->key.rounds, blocks, walk.iv); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } - kernel_neon_end(); return err; } @@ -181,9 +181,8 @@ static int cbc_decrypt(struct skcipher_request *req) struct skcipher_walk walk; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; @@ -191,13 +190,14 @@ static int cbc_decrypt(struct skcipher_request *req) blocks = round_down(blocks, walk.stride / AES_BLOCK_SIZE); + kernel_neon_begin(); aesbs_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->key.rk, ctx->key.rounds, blocks, walk.iv); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } - kernel_neon_end(); return err; } @@ -229,9 +229,8 @@ static int ctr_encrypt(struct skcipher_request *req) u8 buf[AES_BLOCK_SIZE]; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while (walk.nbytes > 0) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; u8 *final = (walk.total % AES_BLOCK_SIZE) ? buf : NULL; @@ -242,8 +241,10 @@ static int ctr_encrypt(struct skcipher_request *req) final = NULL; } + kernel_neon_begin(); aesbs_ctr_encrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->rk, ctx->rounds, blocks, walk.iv, final); + kernel_neon_end(); if (final) { u8 *dst = walk.dst.virt.addr + blocks * AES_BLOCK_SIZE; @@ -258,8 +259,6 @@ static int ctr_encrypt(struct skcipher_request *req) err = skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } - kernel_neon_end(); - return err; } @@ -304,12 +303,11 @@ static int __xts_crypt(struct skcipher_request *req, struct skcipher_walk walk; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); kernel_neon_begin(); - - neon_aes_ecb_encrypt(walk.iv, walk.iv, ctx->twkey, - ctx->key.rounds, 1); + neon_aes_ecb_encrypt(walk.iv, walk.iv, ctx->twkey, ctx->key.rounds, 1); + kernel_neon_end(); while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; @@ -318,13 +316,13 @@ static int __xts_crypt(struct skcipher_request *req, blocks = round_down(blocks, walk.stride / AES_BLOCK_SIZE); + kernel_neon_begin(); fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->key.rk, ctx->key.rounds, blocks, walk.iv); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } - kernel_neon_end(); - return err; } From patchwork Sat Mar 10 15:21:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 131295 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp2244525lja; Sat, 10 Mar 2018 07:22:49 -0800 (PST) X-Google-Smtp-Source: AG47ELv5qLUtyd360NtDroeENcNs7On1waRyINWFj5cNxjqIspMbzzaBrgBYRR8WRPsRvHOWi+5N X-Received: by 10.99.66.65 with SMTP id p62mr1878572pga.378.1520695369194; Sat, 10 Mar 2018 07:22:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520695369; cv=none; d=google.com; s=arc-20160816; b=l8z8A65FR3ua1m2L9ykNE53wDuGS5XkGy0ZHny4QuVo4zp6NBJ/07AqPelLaNOlASL u3OPosikevV/Zu+6lD3ks/5q8dX7e2paDrgFWRvGk3k4FmsLhgpoF+Gl+wIeqEuiND2F NH9nXnL8LozBAwHmcBhMT/wW3g303IB9ckXfSdg/VfUfFkJBhZtVTxOlTEQjXsolBjWJ 5muZRqIt2J7ZA5qrpght0vEc98RqJ/OLq4iBJ/zEPb9NuSP5GssOdCuVziKq0LH+VxR+ tEcyO3kw/4qITbg3IQWtX7PQzSvyqqxIpmOfdUlxrCeoS9FIH2n2zIbfSX0WLsBEaamk IchA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=o9DpJHdXibth1JpdGJrmSP+z+3/O3SvHO7sSQoHB+GE=; b=s+Gqe86O97fyFWwSjCFiHlJjNlekzonHYjafYUJDl+KM6uiMUzfymKdP4vZ83zAabM 4hEv6z20g/s5iTpWGH5YMsYpLnAiItj8Z0INup7p9HFwVe4MrKzwj8ZPEVMcNR4bIxrZ cLiaNaWgytwD20lJ+xAH7mYR3yfDZ4+ldQjDiN3fZ5c3STbB9k5s5gS14/bB41TQ1f6l 8wtyAl0uYhHN1x5MeJS4Ja2ciFntht6oILySrUO12Lw7q3bXgNxUFtIilYKx4nUXwN9j eiS2SBGW70EhRwSkbVPqH7yWDwIvNwuOzLr4/r2BGle7DrmdrP8NICEPDEMjiIWV2zkm 780w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Nc80v7ii; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q16si2824191pfg.221.2018.03.10.07.22.48; Sat, 10 Mar 2018 07:22:49 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Nc80v7ii; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932292AbeCJPWq (ORCPT + 1 other); Sat, 10 Mar 2018 10:22:46 -0500 Received: from mail-wm0-f68.google.com ([74.125.82.68]:33967 "EHLO mail-wm0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932277AbeCJPWp (ORCPT ); Sat, 10 Mar 2018 10:22:45 -0500 Received: by mail-wm0-f68.google.com with SMTP id a20so7233502wmd.1 for ; Sat, 10 Mar 2018 07:22:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=o9DpJHdXibth1JpdGJrmSP+z+3/O3SvHO7sSQoHB+GE=; b=Nc80v7ii0S8D+KJt8wa4vOl+GDzwV1Kka+jcAUJiwWvaMRm3Zb6wyFwN8DcPWLTr87 NaGF1vcrk2pbf9JekkSemNPsthQVkJj87zu5bG3pmpLWJK+bOe5wuboIrXxNJq2lQYcm HK0ereFQisULL6CyFhM4qkZLEkGPHZSoqKvQM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=o9DpJHdXibth1JpdGJrmSP+z+3/O3SvHO7sSQoHB+GE=; b=Py+aUURlVBqIRo4ifUFaEeYwnA8dG6qRxrjUoMucGNgSHIQyLTUPcr0QaLJwcqemII 8R/2GXKWFzBVb9y90QCvUM1sK0NJO0pXpu4BOT5aWiWD3Qv6JQwlJzefgBDZIbKObUv6 md8oUYoz6WHdc3BfM10W9i3bklPBCE2vxVhJ2Htm3GaMBT+DYx+YcYtx8iEMEsSEAmbx LmX+04WaJBEcgH4v1lYyotCn0vvTcMlM3KIcd29K/AbFdvGorQVepGmR6UOV1VNUWebe AwhvWOnTWDYhB9HhSRgW84XVXky62U0DHE5rzo2fwI+N/lPPzqECCjlwNjpTLkk17L4u Wp5w== X-Gm-Message-State: AElRT7ESnBNEvE1t8H7LCTJ7gzjFuHnSB2M5OQxmZc5gPQT0BYUKqHNy T8LvEDLyhfALyDepFpyj3ZD2QEc0D8k= X-Received: by 10.28.26.202 with SMTP id a193mr1436610wma.138.1520695363656; Sat, 10 Mar 2018 07:22:43 -0800 (PST) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id m9sm7027531wrf.13.2018.03.10.07.22.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:22:43 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v5 05/23] crypto: arm64/chacha20 - move kernel mode neon en/disable into loop Date: Sat, 10 Mar 2018 15:21:50 +0000 Message-Id: <20180310152208.10369-6-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180310152208.10369-1-ard.biesheuvel@linaro.org> References: <20180310152208.10369-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org When kernel mode NEON was first introduced on arm64, the preserve and restore of the userland NEON state was completely unoptimized, and involved saving all registers on each call to kernel_neon_begin(), and restoring them on each call to kernel_neon_end(). For this reason, the NEON crypto code that was introduced at the time keeps the NEON enabled throughout the execution of the crypto API methods, which may include calls back into the crypto API that could result in memory allocation or other actions that we should avoid when running with preemption disabled. Since then, we have optimized the kernel mode NEON handling, which now restores lazily (upon return to userland), and so the preserve action is only costly the first time it is called after entering the kernel. So let's put the kernel_neon_begin() and kernel_neon_end() calls around the actual invocations of the NEON crypto code, and run the remainder of the code with kernel mode NEON disabled (and preemption enabled) Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/chacha20-neon-glue.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) -- 2.15.1 diff --git a/arch/arm64/crypto/chacha20-neon-glue.c b/arch/arm64/crypto/chacha20-neon-glue.c index cbdb75d15cd0..727579c93ded 100644 --- a/arch/arm64/crypto/chacha20-neon-glue.c +++ b/arch/arm64/crypto/chacha20-neon-glue.c @@ -37,12 +37,19 @@ static void chacha20_doneon(u32 *state, u8 *dst, const u8 *src, u8 buf[CHACHA20_BLOCK_SIZE]; while (bytes >= CHACHA20_BLOCK_SIZE * 4) { + kernel_neon_begin(); chacha20_4block_xor_neon(state, dst, src); + kernel_neon_end(); bytes -= CHACHA20_BLOCK_SIZE * 4; src += CHACHA20_BLOCK_SIZE * 4; dst += CHACHA20_BLOCK_SIZE * 4; state[12] += 4; } + + if (!bytes) + return; + + kernel_neon_begin(); while (bytes >= CHACHA20_BLOCK_SIZE) { chacha20_block_xor_neon(state, dst, src); bytes -= CHACHA20_BLOCK_SIZE; @@ -55,6 +62,7 @@ static void chacha20_doneon(u32 *state, u8 *dst, const u8 *src, chacha20_block_xor_neon(state, buf, buf); memcpy(dst, buf, bytes); } + kernel_neon_end(); } static int chacha20_neon(struct skcipher_request *req) @@ -68,11 +76,10 @@ static int chacha20_neon(struct skcipher_request *req) if (!may_use_simd() || req->cryptlen <= CHACHA20_BLOCK_SIZE) return crypto_chacha20_crypt(req); - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); crypto_chacha20_init(state, ctx, walk.iv); - kernel_neon_begin(); while (walk.nbytes > 0) { unsigned int nbytes = walk.nbytes; @@ -83,7 +90,6 @@ static int chacha20_neon(struct skcipher_request *req) nbytes); err = skcipher_walk_done(&walk, walk.nbytes - nbytes); } - kernel_neon_end(); return err; } From patchwork Sat Mar 10 15:21:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 131296 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp2244569lja; Sat, 10 Mar 2018 07:22:52 -0800 (PST) X-Google-Smtp-Source: AG47ELvtMWlWKk2979jGFNc+CSZPYnz0dBeC/Vne5lwg0sgkg+xhdwXmZrC76IiwcJKMfASsvtSF X-Received: by 2002:a17:902:2cc1:: with SMTP id n59-v6mr2299951plb.215.1520695372432; Sat, 10 Mar 2018 07:22:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520695372; cv=none; d=google.com; s=arc-20160816; b=Il5OAc9eOZAxnQwT2h5BAhCKKvOqarcoH51/CrOgIK0Y7BPMWjE7NBP8FjpruWDEWD f8d9pPMDuluAwmK6cCvspNVC29E8r9l+R7yuB9Rv/OX1VU7tfmlMuuM/RTCw77TgMMNI oEYOUwGMVR3Oh1HAlGS4aFt70G+DSlEBXZHSRXXoFbrPp//8lJIBxyJqdxexq5I/YxYM taje12h2AR2djbnBSDwMm4nq9dPnLht12BdN9a7u5rXqALcJJUyHEQtNE69aP53/caC4 9YC8QT0Kumt7gDZshbhGzmYAg5U6FsweJ0DZebmLuxjXhZMCuLPJtX6IY2V51WQHO2a7 OfmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=s8TtMAo57z1jG6FAAZUog2c3ePpi5jVMwADCnedkZIs=; b=G5dTZlQcceixiqm5q4FT7RvcPch8ZJadgW+FQZb4Id6trfHTYRP+tkKVMVVmhi/utj lmUJvxDWxD2CimKq7GpeLdtVFrxcAVdgLlDTqWEaE777mge2pEiCcXBTZhQ7wLM7kiQM Cz45Tyx7p2GQQaCZ1dccxbhBNH8dbrPun2w4ri4wCyrE2ZZTwzz0Vm6L4HRlVijLv3na DufYOy/YlJ4RKuIFIH8NKJA8sAEYXedmGgLZdTGOFDsrNrk5lT6tiJ0Kcc4DlmMkMyau 1rd1bg4PXV4tH72GHvEYjrxOllcdTTfC/gh6cYWwGmuPIl3KJ6hZkPhUJEsdUggWEth5 kMQQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=BgjTYABx; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q16si2824191pfg.221.2018.03.10.07.22.52; Sat, 10 Mar 2018 07:22:52 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=BgjTYABx; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932313AbeCJPWu (ORCPT + 1 other); Sat, 10 Mar 2018 10:22:50 -0500 Received: from mail-wr0-f193.google.com ([209.85.128.193]:44344 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932277AbeCJPWs (ORCPT ); Sat, 10 Mar 2018 10:22:48 -0500 Received: by mail-wr0-f193.google.com with SMTP id v65so11629310wrc.11 for ; Sat, 10 Mar 2018 07:22:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=s8TtMAo57z1jG6FAAZUog2c3ePpi5jVMwADCnedkZIs=; b=BgjTYABxh7eBB5GIupxqtlh5e1/N6CM9mD4gsMdmaHMtM4F1iNjet1VE99atxhkrxy uuLB632SrueNtd88RYWCMKDfe4IR2pDDn4VHXOtts1Fmmp9Ri3BIX0aLVlj9xvKcbpsx h2Mpy85XTriAYPvQytrA/lP4atvfyLE4BvjEs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=s8TtMAo57z1jG6FAAZUog2c3ePpi5jVMwADCnedkZIs=; b=QjlHzpF6J+CSYncQFWsJ+hsB+rkZEuQ6fcC8G67KVa6tcIeKG0DWcwaCTU4ayHgfEN n/FoZnnfJEhSuksAvHzS+glzRO6pGGgxq4AqBHR9+zpMd/uOMCupa4bchRLSYvgD3MNS rPJGTUqKhVfXbf4UwsvliSRqGfDVsDB5QWNncc/In/TYdX8vMVhny8EvxzYHSk/vc6as tzbhXF91jCv/sSdSWrmL5odHio1NOUMDVlBeHcqLQb58VimRwGNeB6fStM1X2D6UV8BN +r5psQYZroOWw4I9BPTjOVF9H2dgXbQ9BTux8NLN16DDZKavzkXXQQrV9w5hIlpnWBXp VAXQ== X-Gm-Message-State: AElRT7GORWco/h/47FQ0+NDDrMTg1zU8v5xUjOD2I8rj+mTjMopAapKO 5MQDcdbI36AU0jFs3I/MbAbna0VeO/k= X-Received: by 10.223.169.248 with SMTP id b111mr1666453wrd.237.1520695366777; Sat, 10 Mar 2018 07:22:46 -0800 (PST) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id m9sm7027531wrf.13.2018.03.10.07.22.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:22:45 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v5 06/23] crypto: arm64/aes-blk - remove configurable interleave Date: Sat, 10 Mar 2018 15:21:51 +0000 Message-Id: <20180310152208.10369-7-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180310152208.10369-1-ard.biesheuvel@linaro.org> References: <20180310152208.10369-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The AES block mode implementation using Crypto Extensions or plain NEON was written before real hardware existed, and so its interleave factor was made build time configurable (as well as an option to instantiate all interleaved sequences inline rather than as subroutines) We ended up using INTERLEAVE=4 with inlining disabled for both flavors of the core AES routines, so let's stick with that, and remove the option to configure this at build time. This makes the code easier to modify, which is nice now that we're adding yield support. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/Makefile | 3 - arch/arm64/crypto/aes-modes.S | 237 ++++---------------- 2 files changed, 40 insertions(+), 200 deletions(-) -- 2.15.1 diff --git a/arch/arm64/crypto/Makefile b/arch/arm64/crypto/Makefile index cee9b8d9830b..b6b624602582 100644 --- a/arch/arm64/crypto/Makefile +++ b/arch/arm64/crypto/Makefile @@ -59,9 +59,6 @@ aes-arm64-y := aes-cipher-core.o aes-cipher-glue.o obj-$(CONFIG_CRYPTO_AES_ARM64_BS) += aes-neon-bs.o aes-neon-bs-y := aes-neonbs-core.o aes-neonbs-glue.o -AFLAGS_aes-ce.o := -DINTERLEAVE=4 -AFLAGS_aes-neon.o := -DINTERLEAVE=4 - CFLAGS_aes-glue-ce.o := -DUSE_V8_CRYPTO_EXTENSIONS $(obj)/aes-glue-%.o: $(src)/aes-glue.c FORCE diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index 65b273667b34..27a235b2ddee 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -13,44 +13,6 @@ .text .align 4 -/* - * There are several ways to instantiate this code: - * - no interleave, all inline - * - 2-way interleave, 2x calls out of line (-DINTERLEAVE=2) - * - 2-way interleave, all inline (-DINTERLEAVE=2 -DINTERLEAVE_INLINE) - * - 4-way interleave, 4x calls out of line (-DINTERLEAVE=4) - * - 4-way interleave, all inline (-DINTERLEAVE=4 -DINTERLEAVE_INLINE) - * - * Macros imported by this code: - * - enc_prepare - setup NEON registers for encryption - * - dec_prepare - setup NEON registers for decryption - * - enc_switch_key - change to new key after having prepared for encryption - * - encrypt_block - encrypt a single block - * - decrypt block - decrypt a single block - * - encrypt_block2x - encrypt 2 blocks in parallel (if INTERLEAVE == 2) - * - decrypt_block2x - decrypt 2 blocks in parallel (if INTERLEAVE == 2) - * - encrypt_block4x - encrypt 4 blocks in parallel (if INTERLEAVE == 4) - * - decrypt_block4x - decrypt 4 blocks in parallel (if INTERLEAVE == 4) - */ - -#if defined(INTERLEAVE) && !defined(INTERLEAVE_INLINE) -#define FRAME_PUSH stp x29, x30, [sp,#-16]! ; mov x29, sp -#define FRAME_POP ldp x29, x30, [sp],#16 - -#if INTERLEAVE == 2 - -aes_encrypt_block2x: - encrypt_block2x v0, v1, w3, x2, x8, w7 - ret -ENDPROC(aes_encrypt_block2x) - -aes_decrypt_block2x: - decrypt_block2x v0, v1, w3, x2, x8, w7 - ret -ENDPROC(aes_decrypt_block2x) - -#elif INTERLEAVE == 4 - aes_encrypt_block4x: encrypt_block4x v0, v1, v2, v3, w3, x2, x8, w7 ret @@ -61,48 +23,6 @@ aes_decrypt_block4x: ret ENDPROC(aes_decrypt_block4x) -#else -#error INTERLEAVE should equal 2 or 4 -#endif - - .macro do_encrypt_block2x - bl aes_encrypt_block2x - .endm - - .macro do_decrypt_block2x - bl aes_decrypt_block2x - .endm - - .macro do_encrypt_block4x - bl aes_encrypt_block4x - .endm - - .macro do_decrypt_block4x - bl aes_decrypt_block4x - .endm - -#else -#define FRAME_PUSH -#define FRAME_POP - - .macro do_encrypt_block2x - encrypt_block2x v0, v1, w3, x2, x8, w7 - .endm - - .macro do_decrypt_block2x - decrypt_block2x v0, v1, w3, x2, x8, w7 - .endm - - .macro do_encrypt_block4x - encrypt_block4x v0, v1, v2, v3, w3, x2, x8, w7 - .endm - - .macro do_decrypt_block4x - decrypt_block4x v0, v1, v2, v3, w3, x2, x8, w7 - .endm - -#endif - /* * aes_ecb_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds, * int blocks) @@ -111,28 +31,21 @@ ENDPROC(aes_decrypt_block4x) */ AES_ENTRY(aes_ecb_encrypt) - FRAME_PUSH + stp x29, x30, [sp, #-16]! + mov x29, sp enc_prepare w3, x2, x5 .LecbencloopNx: -#if INTERLEAVE >= 2 - subs w4, w4, #INTERLEAVE + subs w4, w4, #4 bmi .Lecbenc1x -#if INTERLEAVE == 2 - ld1 {v0.16b-v1.16b}, [x1], #32 /* get 2 pt blocks */ - do_encrypt_block2x - st1 {v0.16b-v1.16b}, [x0], #32 -#else ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 pt blocks */ - do_encrypt_block4x + bl aes_encrypt_block4x st1 {v0.16b-v3.16b}, [x0], #64 -#endif b .LecbencloopNx .Lecbenc1x: - adds w4, w4, #INTERLEAVE + adds w4, w4, #4 beq .Lecbencout -#endif .Lecbencloop: ld1 {v0.16b}, [x1], #16 /* get next pt block */ encrypt_block v0, w3, x2, x5, w6 @@ -140,34 +53,27 @@ AES_ENTRY(aes_ecb_encrypt) subs w4, w4, #1 bne .Lecbencloop .Lecbencout: - FRAME_POP + ldp x29, x30, [sp], #16 ret AES_ENDPROC(aes_ecb_encrypt) AES_ENTRY(aes_ecb_decrypt) - FRAME_PUSH + stp x29, x30, [sp, #-16]! + mov x29, sp dec_prepare w3, x2, x5 .LecbdecloopNx: -#if INTERLEAVE >= 2 - subs w4, w4, #INTERLEAVE + subs w4, w4, #4 bmi .Lecbdec1x -#if INTERLEAVE == 2 - ld1 {v0.16b-v1.16b}, [x1], #32 /* get 2 ct blocks */ - do_decrypt_block2x - st1 {v0.16b-v1.16b}, [x0], #32 -#else ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 ct blocks */ - do_decrypt_block4x + bl aes_decrypt_block4x st1 {v0.16b-v3.16b}, [x0], #64 -#endif b .LecbdecloopNx .Lecbdec1x: - adds w4, w4, #INTERLEAVE + adds w4, w4, #4 beq .Lecbdecout -#endif .Lecbdecloop: ld1 {v0.16b}, [x1], #16 /* get next ct block */ decrypt_block v0, w3, x2, x5, w6 @@ -175,7 +81,7 @@ AES_ENTRY(aes_ecb_decrypt) subs w4, w4, #1 bne .Lecbdecloop .Lecbdecout: - FRAME_POP + ldp x29, x30, [sp], #16 ret AES_ENDPROC(aes_ecb_decrypt) @@ -204,30 +110,20 @@ AES_ENDPROC(aes_cbc_encrypt) AES_ENTRY(aes_cbc_decrypt) - FRAME_PUSH + stp x29, x30, [sp, #-16]! + mov x29, sp ld1 {v7.16b}, [x5] /* get iv */ dec_prepare w3, x2, x6 .LcbcdecloopNx: -#if INTERLEAVE >= 2 - subs w4, w4, #INTERLEAVE + subs w4, w4, #4 bmi .Lcbcdec1x -#if INTERLEAVE == 2 - ld1 {v0.16b-v1.16b}, [x1], #32 /* get 2 ct blocks */ - mov v2.16b, v0.16b - mov v3.16b, v1.16b - do_decrypt_block2x - eor v0.16b, v0.16b, v7.16b - eor v1.16b, v1.16b, v2.16b - mov v7.16b, v3.16b - st1 {v0.16b-v1.16b}, [x0], #32 -#else ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 ct blocks */ mov v4.16b, v0.16b mov v5.16b, v1.16b mov v6.16b, v2.16b - do_decrypt_block4x + bl aes_decrypt_block4x sub x1, x1, #16 eor v0.16b, v0.16b, v7.16b eor v1.16b, v1.16b, v4.16b @@ -235,12 +131,10 @@ AES_ENTRY(aes_cbc_decrypt) eor v2.16b, v2.16b, v5.16b eor v3.16b, v3.16b, v6.16b st1 {v0.16b-v3.16b}, [x0], #64 -#endif b .LcbcdecloopNx .Lcbcdec1x: - adds w4, w4, #INTERLEAVE + adds w4, w4, #4 beq .Lcbcdecout -#endif .Lcbcdecloop: ld1 {v1.16b}, [x1], #16 /* get next ct block */ mov v0.16b, v1.16b /* ...and copy to v0 */ @@ -251,8 +145,8 @@ AES_ENTRY(aes_cbc_decrypt) subs w4, w4, #1 bne .Lcbcdecloop .Lcbcdecout: - FRAME_POP st1 {v7.16b}, [x5] /* return iv */ + ldp x29, x30, [sp], #16 ret AES_ENDPROC(aes_cbc_decrypt) @@ -263,34 +157,19 @@ AES_ENDPROC(aes_cbc_decrypt) */ AES_ENTRY(aes_ctr_encrypt) - FRAME_PUSH + stp x29, x30, [sp, #-16]! + mov x29, sp enc_prepare w3, x2, x6 ld1 {v4.16b}, [x5] umov x6, v4.d[1] /* keep swabbed ctr in reg */ rev x6, x6 -#if INTERLEAVE >= 2 cmn w6, w4 /* 32 bit overflow? */ bcs .Lctrloop .LctrloopNx: - subs w4, w4, #INTERLEAVE + subs w4, w4, #4 bmi .Lctr1x -#if INTERLEAVE == 2 - mov v0.8b, v4.8b - mov v1.8b, v4.8b - rev x7, x6 - add x6, x6, #1 - ins v0.d[1], x7 - rev x7, x6 - add x6, x6, #1 - ins v1.d[1], x7 - ld1 {v2.16b-v3.16b}, [x1], #32 /* get 2 input blocks */ - do_encrypt_block2x - eor v0.16b, v0.16b, v2.16b - eor v1.16b, v1.16b, v3.16b - st1 {v0.16b-v1.16b}, [x0], #32 -#else ldr q8, =0x30000000200000001 /* addends 1,2,3[,0] */ dup v7.4s, w6 mov v0.16b, v4.16b @@ -303,23 +182,21 @@ AES_ENTRY(aes_ctr_encrypt) mov v2.s[3], v8.s[1] mov v3.s[3], v8.s[2] ld1 {v5.16b-v7.16b}, [x1], #48 /* get 3 input blocks */ - do_encrypt_block4x + bl aes_encrypt_block4x eor v0.16b, v5.16b, v0.16b ld1 {v5.16b}, [x1], #16 /* get 1 input block */ eor v1.16b, v6.16b, v1.16b eor v2.16b, v7.16b, v2.16b eor v3.16b, v5.16b, v3.16b st1 {v0.16b-v3.16b}, [x0], #64 - add x6, x6, #INTERLEAVE -#endif + add x6, x6, #4 rev x7, x6 ins v4.d[1], x7 cbz w4, .Lctrout b .LctrloopNx .Lctr1x: - adds w4, w4, #INTERLEAVE + adds w4, w4, #4 beq .Lctrout -#endif .Lctrloop: mov v0.16b, v4.16b encrypt_block v0, w3, x2, x8, w7 @@ -339,12 +216,12 @@ AES_ENTRY(aes_ctr_encrypt) .Lctrout: st1 {v4.16b}, [x5] /* return next CTR value */ - FRAME_POP + ldp x29, x30, [sp], #16 ret .Lctrtailblock: st1 {v0.16b}, [x0] - FRAME_POP + ldp x29, x30, [sp], #16 ret .Lctrcarry: @@ -378,7 +255,9 @@ CPU_LE( .quad 1, 0x87 ) CPU_BE( .quad 0x87, 1 ) AES_ENTRY(aes_xts_encrypt) - FRAME_PUSH + stp x29, x30, [sp, #-16]! + mov x29, sp + ld1 {v4.16b}, [x6] cbz w7, .Lxtsencnotfirst @@ -394,25 +273,8 @@ AES_ENTRY(aes_xts_encrypt) ldr q7, .Lxts_mul_x next_tweak v4, v4, v7, v8 .LxtsencNx: -#if INTERLEAVE >= 2 - subs w4, w4, #INTERLEAVE + subs w4, w4, #4 bmi .Lxtsenc1x -#if INTERLEAVE == 2 - ld1 {v0.16b-v1.16b}, [x1], #32 /* get 2 pt blocks */ - next_tweak v5, v4, v7, v8 - eor v0.16b, v0.16b, v4.16b - eor v1.16b, v1.16b, v5.16b - do_encrypt_block2x - eor v0.16b, v0.16b, v4.16b - eor v1.16b, v1.16b, v5.16b - st1 {v0.16b-v1.16b}, [x0], #32 - cbz w4, .LxtsencoutNx - next_tweak v4, v5, v7, v8 - b .LxtsencNx -.LxtsencoutNx: - mov v4.16b, v5.16b - b .Lxtsencout -#else ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 pt blocks */ next_tweak v5, v4, v7, v8 eor v0.16b, v0.16b, v4.16b @@ -421,7 +283,7 @@ AES_ENTRY(aes_xts_encrypt) eor v2.16b, v2.16b, v6.16b next_tweak v7, v6, v7, v8 eor v3.16b, v3.16b, v7.16b - do_encrypt_block4x + bl aes_encrypt_block4x eor v3.16b, v3.16b, v7.16b eor v0.16b, v0.16b, v4.16b eor v1.16b, v1.16b, v5.16b @@ -430,11 +292,9 @@ AES_ENTRY(aes_xts_encrypt) mov v4.16b, v7.16b cbz w4, .Lxtsencout b .LxtsencloopNx -#endif .Lxtsenc1x: - adds w4, w4, #INTERLEAVE + adds w4, w4, #4 beq .Lxtsencout -#endif .Lxtsencloop: ld1 {v1.16b}, [x1], #16 eor v0.16b, v1.16b, v4.16b @@ -447,13 +307,15 @@ AES_ENTRY(aes_xts_encrypt) b .Lxtsencloop .Lxtsencout: st1 {v4.16b}, [x6] - FRAME_POP + ldp x29, x30, [sp], #16 ret AES_ENDPROC(aes_xts_encrypt) AES_ENTRY(aes_xts_decrypt) - FRAME_PUSH + stp x29, x30, [sp, #-16]! + mov x29, sp + ld1 {v4.16b}, [x6] cbz w7, .Lxtsdecnotfirst @@ -469,25 +331,8 @@ AES_ENTRY(aes_xts_decrypt) ldr q7, .Lxts_mul_x next_tweak v4, v4, v7, v8 .LxtsdecNx: -#if INTERLEAVE >= 2 - subs w4, w4, #INTERLEAVE + subs w4, w4, #4 bmi .Lxtsdec1x -#if INTERLEAVE == 2 - ld1 {v0.16b-v1.16b}, [x1], #32 /* get 2 ct blocks */ - next_tweak v5, v4, v7, v8 - eor v0.16b, v0.16b, v4.16b - eor v1.16b, v1.16b, v5.16b - do_decrypt_block2x - eor v0.16b, v0.16b, v4.16b - eor v1.16b, v1.16b, v5.16b - st1 {v0.16b-v1.16b}, [x0], #32 - cbz w4, .LxtsdecoutNx - next_tweak v4, v5, v7, v8 - b .LxtsdecNx -.LxtsdecoutNx: - mov v4.16b, v5.16b - b .Lxtsdecout -#else ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 ct blocks */ next_tweak v5, v4, v7, v8 eor v0.16b, v0.16b, v4.16b @@ -496,7 +341,7 @@ AES_ENTRY(aes_xts_decrypt) eor v2.16b, v2.16b, v6.16b next_tweak v7, v6, v7, v8 eor v3.16b, v3.16b, v7.16b - do_decrypt_block4x + bl aes_decrypt_block4x eor v3.16b, v3.16b, v7.16b eor v0.16b, v0.16b, v4.16b eor v1.16b, v1.16b, v5.16b @@ -505,11 +350,9 @@ AES_ENTRY(aes_xts_decrypt) mov v4.16b, v7.16b cbz w4, .Lxtsdecout b .LxtsdecloopNx -#endif .Lxtsdec1x: - adds w4, w4, #INTERLEAVE + adds w4, w4, #4 beq .Lxtsdecout -#endif .Lxtsdecloop: ld1 {v1.16b}, [x1], #16 eor v0.16b, v1.16b, v4.16b @@ -522,7 +365,7 @@ AES_ENTRY(aes_xts_decrypt) b .Lxtsdecloop .Lxtsdecout: st1 {v4.16b}, [x6] - FRAME_POP + ldp x29, x30, [sp], #16 ret AES_ENDPROC(aes_xts_decrypt) From patchwork Sat Mar 10 15:21:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 131297 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp2244604lja; Sat, 10 Mar 2018 07:22:55 -0800 (PST) X-Google-Smtp-Source: AG47ELunon6CPCJtb+wJCKBVm+8Q+2jaThPrNaMOdfKX9uQzqX+lVE6esZNX+3zA0K14YoXMmAIB X-Received: by 10.98.75.129 with SMTP id d1mr2276651pfj.19.1520695375347; Sat, 10 Mar 2018 07:22:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520695375; cv=none; d=google.com; s=arc-20160816; b=mPpwpRRK00tBR9dIN4/2o/Wy/6uZztWNGqvcl7jvtYAj/1R7jyj2DCMnsiv/uF+uRS J/UWQX6fBYGRuzOUh79DyrhNi2YPMwxuUiWNqGotwvbE6gecBlqCOyzckwxbrApTnv2U fpOuXWKdejpxCpVzS0w+Jof1sAwjzE0WhxA9aGFPQbMvRCcY8/VIeiT/WMKOWReKqfs/ lOzP7ptU+WwuA+lJBG1EoDG9LPwwO814o2OU9tyMGDf6z0Fbq4KyPlP/1YYorXL8/F9e G424bmVoJyxUG1uYTHlMZ54WTwP1cZi3huQoSLkQfzk4fqHWlX3r4i3B5qTACSlT839i U7qg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=045pc8ZcXNQ6enBteVqbqVjC9IyYTZHYnUU9BOMiWIg=; b=AeWcco3jKOpQSrkQyBH+UijVbvqVZG+zAauLlSBBbSlhaqFp5xs0Y2q8ZGulVBS6JD uCpR1mX5GpoL9hNstUNXeVQ1AXp/FvOmwFGu8gI93IAKbwSalHYHk/j3Y2xLWR7s6lEs k9xknxxlha4umjh8QS+Vprhv9ILfpct2EqcN1Ri0JH06tcEBXGN10dybnh5/p3eenkYD chRwH3kXKZ1FbY5z/ocdkCNOC29OhkJHAPnNl8h6unlUktzv8WtBbp8wvP124WdiHqnh 0EFhUog0IeMFmUa2+i1Nv9Ecg+WUk58RN7gW2lzQufRibGw+JLvTV7TLZzluw87RCUx4 cedg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=gymxINhg; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q16si2824191pfg.221.2018.03.10.07.22.55; Sat, 10 Mar 2018 07:22:55 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=gymxINhg; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932271AbeCJPWx (ORCPT + 1 other); Sat, 10 Mar 2018 10:22:53 -0500 Received: from mail-wr0-f193.google.com ([209.85.128.193]:42067 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932308AbeCJPWu (ORCPT ); Sat, 10 Mar 2018 10:22:50 -0500 Received: by mail-wr0-f193.google.com with SMTP id k9so11632260wre.9 for ; Sat, 10 Mar 2018 07:22:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=045pc8ZcXNQ6enBteVqbqVjC9IyYTZHYnUU9BOMiWIg=; b=gymxINhgItz+nuDPpQXrUwzDGnQLyQFYiBcxPpGA+owu+IcV0NuO7N1is247R5MHKQ jOAvbsyvv+NQ7mRXwUS7PhgQRtSCx2d3cNfdzmm4yvFDFHpnolBxukky0JMhhEID/Yz3 624aIfxCturwrbvsh45JVKmSjEbH8s8yCHIB8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=045pc8ZcXNQ6enBteVqbqVjC9IyYTZHYnUU9BOMiWIg=; b=mMQUtxj8/DXqYDTutj5rCu+AkZE6X4be4uMtEujkZ41kpe40yrXdwSeOv7rLSkQUTb fte3sUaflmrgEuy6CcUSiN9osklWLjuzxiFuQMt2SLSxCsO4PLaiaR3JI00pv6c4oo1A ktdXK1J+Fu1klY1QwkImWrdPYqf8G74HkzA6Hdrqsyd4C6tEmrAv21aMeo6cs5HgDy1o 2JXOjYujqy0CaK5rixR28dGVFRoEHpREUbKNlu/5HckicNpxdqx1W3V7QRRP3cmgRe5W t+DnqWP0R2EmEvS/HHyKZbjP38Oe1rl3N6RrV5bn/fsuCOVxuoYkr4GhIUXaZ8uS0O5o 91Jw== X-Gm-Message-State: AElRT7ElAHlFYEwM1NZXCadthAfVx7teBl90LmFeBaqhW/EbKTfGJ7Ak 4Mo0UTfLCBS9bUPovoo2KA968YNRtJ8= X-Received: by 10.223.201.142 with SMTP id f14mr1899909wrh.40.1520695369357; Sat, 10 Mar 2018 07:22:49 -0800 (PST) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id m9sm7027531wrf.13.2018.03.10.07.22.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:22:48 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v5 07/23] crypto: arm64/aes-blk - add 4 way interleave to CBC encrypt path Date: Sat, 10 Mar 2018 15:21:52 +0000 Message-Id: <20180310152208.10369-8-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180310152208.10369-1-ard.biesheuvel@linaro.org> References: <20180310152208.10369-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org CBC encryption is strictly sequential, and so the current AES code simply processes the input one block at a time. However, we are about to add yield support, which adds a bit of overhead, and which we prefer to align with other modes in terms of granularity (i.e., it is better to have all routines yield every 64 bytes and not have an exception for CBC encrypt which yields every 16 bytes) So unroll the loop by 4. We still cannot perform the AES algorithm in parallel, but we can at least merge the loads and stores. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-modes.S | 31 ++++++++++++++++---- 1 file changed, 25 insertions(+), 6 deletions(-) -- 2.15.1 diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index 27a235b2ddee..e86535a1329d 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -94,17 +94,36 @@ AES_ENDPROC(aes_ecb_decrypt) */ AES_ENTRY(aes_cbc_encrypt) - ld1 {v0.16b}, [x5] /* get iv */ + ld1 {v4.16b}, [x5] /* get iv */ enc_prepare w3, x2, x6 -.Lcbcencloop: - ld1 {v1.16b}, [x1], #16 /* get next pt block */ - eor v0.16b, v0.16b, v1.16b /* ..and xor with iv */ +.Lcbcencloop4x: + subs w4, w4, #4 + bmi .Lcbcenc1x + ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 pt blocks */ + eor v0.16b, v0.16b, v4.16b /* ..and xor with iv */ encrypt_block v0, w3, x2, x6, w7 - st1 {v0.16b}, [x0], #16 + eor v1.16b, v1.16b, v0.16b + encrypt_block v1, w3, x2, x6, w7 + eor v2.16b, v2.16b, v1.16b + encrypt_block v2, w3, x2, x6, w7 + eor v3.16b, v3.16b, v2.16b + encrypt_block v3, w3, x2, x6, w7 + st1 {v0.16b-v3.16b}, [x0], #64 + mov v4.16b, v3.16b + b .Lcbcencloop4x +.Lcbcenc1x: + adds w4, w4, #4 + beq .Lcbcencout +.Lcbcencloop: + ld1 {v0.16b}, [x1], #16 /* get next pt block */ + eor v4.16b, v4.16b, v0.16b /* ..and xor with iv */ + encrypt_block v4, w3, x2, x6, w7 + st1 {v4.16b}, [x0], #16 subs w4, w4, #1 bne .Lcbcencloop - st1 {v0.16b}, [x5] /* return iv */ +.Lcbcencout: + st1 {v4.16b}, [x5] /* return iv */ ret AES_ENDPROC(aes_cbc_encrypt) From patchwork Sat Mar 10 15:21:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 131298 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp2244651lja; Sat, 10 Mar 2018 07:22:58 -0800 (PST) X-Google-Smtp-Source: AG47ELvw4JpeQwZ0/LHjWWakByFV2dy6EjAueweyrvEJipb5cxfC4hzm5awGyubt6IQB3/rOl7yr X-Received: by 2002:a17:902:b785:: with SMTP id e5-v6mr1703604pls.354.1520695378084; Sat, 10 Mar 2018 07:22:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520695378; cv=none; d=google.com; s=arc-20160816; b=e91LDzX5JwQI+NzX3cBHbc3ghP8Gng8E3WXljkb/Wa/8HdnO4mUEYsbFQj9cC1D4fv IA+d4ZVdc+CPDyE9EsMVpab6jpi8szklWjod9OYv2GdH8R7tOh5BLNJxquqHKnTultys rIl+OhWCahkgae3jwbfTaosyAfRoYZPYHzK6g8fA0FEcMRGIxAQvHK6oCMpe2Mt3KoJZ 0GL3LdWglc1sDwQEmivpT5uS4q/erUPW5OVSiAq3ApdWNzkeqr10btK+Q+vsy6SRgBYM GVa8O7xgl6TWeducPN9t5OOhpG8BdchrOCbq7vNiTrfcn57arck204GMA0xiG/PZX8t/ dhfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=SWaTAqImlUy+j9GIerQtkvTvfHo8IM7KHMAA2NnX1Ag=; b=Ud+IszENUiVt1mjRoG4IyJ2ihAGo57KzQjRIr3h+OL8t4sK9QzA5C9Jd+oLnBecLIS 7o9EwXzLUYuVy+B6zFug3aVLLv456AH1ny9FaKU5XeIt59NnysSJoKcRbBMPtkoJ1q/P BmcUxx352vZ1aVyreYL8NtyKD1TBI3iiw9LlAPVgW6VunrwQKAAE8MnP0RSPN4lA2sA8 IhkPcFuMPNpHhEriDjZn+F1HZ2ORX4JUS+VSJqggdGw2/nA6ex+q5hQuir90kWkKKNwk UWB38kLeMuWLK8cXh4qMxk+VZxQ4MLWMoqh2QblK0VEX3pr+HFY0PprPai7mKTCzZlbl hohA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GPqtIFW2; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q16si2824191pfg.221.2018.03.10.07.22.57; Sat, 10 Mar 2018 07:22:58 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GPqtIFW2; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932321AbeCJPW4 (ORCPT + 1 other); Sat, 10 Mar 2018 10:22:56 -0500 Received: from mail-wm0-f67.google.com ([74.125.82.67]:40210 "EHLO mail-wm0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932308AbeCJPWy (ORCPT ); Sat, 10 Mar 2018 10:22:54 -0500 Received: by mail-wm0-f67.google.com with SMTP id t6so8750195wmt.5 for ; Sat, 10 Mar 2018 07:22:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=SWaTAqImlUy+j9GIerQtkvTvfHo8IM7KHMAA2NnX1Ag=; b=GPqtIFW2E2k9LNStc3Z9Oro8B0dz4DGxsgTQLwqMAi4I34CzMjbs2U9ZToAW2lXiaP an07aC0SUfD7RTP5RmhTkjNLlX2MISFzXHf/pWtM6J3X3hQQh4qhKbLKnSd+pq7RJw8X JkhVAziEZUiliixX6YJKMmxe8o6z1u/hWQJ2Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=SWaTAqImlUy+j9GIerQtkvTvfHo8IM7KHMAA2NnX1Ag=; b=EtbX9+G2LCEJMVN4kKWH0kwRLQuT0YbjPnRWMWMDjLx0s+KBVXNOme+ryDj2Rtd7N3 gqEKoPi/adrsE3eWBJ9VTFGywNKmc0i4oD+r6+FZe00ViG/xT0edhex5gdSlgwC0pv/O sKnDqtLW2JbZgt8kLSTBqmwbc75rFNEsT/BruOg+pFpIAWRYtquqTufqQagurIGMHKho RBy08mXKd3yMgYdnhVSZeJ1Pe6KOD3A2W/bS3W+HX2vvcT95HY1ic6gCegseOVNWYV0Q GIP914MOZAPWGguj4rburhg0loNi5UOXluHyHpXlI/4CBpxvbupB3rpl2HWzDRJBwL9M gnVQ== X-Gm-Message-State: AElRT7G1Bsf97AwwzIO0bvb5KCDkTMMo2kXvSt+Q5zs6LinNxWS7/8Ch Bb42aD5pse2pyTMUWydfH/03brCufZA= X-Received: by 10.28.69.197 with SMTP id l66mr1359264wmi.34.1520695373023; Sat, 10 Mar 2018 07:22:53 -0800 (PST) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id m9sm7027531wrf.13.2018.03.10.07.22.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:22:51 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v5 08/23] crypto: arm64/aes-blk - add 4 way interleave to CBC-MAC encrypt path Date: Sat, 10 Mar 2018 15:21:53 +0000 Message-Id: <20180310152208.10369-9-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180310152208.10369-1-ard.biesheuvel@linaro.org> References: <20180310152208.10369-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org CBC MAC is strictly sequential, and so the current AES code simply processes the input one block at a time. However, we are about to add yield support, which adds a bit of overhead, and which we prefer to align with other modes in terms of granularity (i.e., it is better to have all routines yield every 64 bytes and not have an exception for CBC MAC which yields every 16 bytes) So unroll the loop by 4. We still cannot perform the AES algorithm in parallel, but we can at least merge the loads and stores. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-modes.S | 23 ++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) -- 2.15.1 diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index e86535a1329d..a68412e1e3a4 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -395,8 +395,28 @@ AES_ENDPROC(aes_xts_decrypt) AES_ENTRY(aes_mac_update) ld1 {v0.16b}, [x4] /* get dg */ enc_prepare w2, x1, x7 - cbnz w5, .Lmacenc + cbz w5, .Lmacloop4x + encrypt_block v0, w2, x1, x7, w8 + +.Lmacloop4x: + subs w3, w3, #4 + bmi .Lmac1x + ld1 {v1.16b-v4.16b}, [x0], #64 /* get next pt block */ + eor v0.16b, v0.16b, v1.16b /* ..and xor with dg */ + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v2.16b + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v3.16b + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v4.16b + cmp w3, wzr + csinv x5, x6, xzr, eq + cbz w5, .Lmacout + encrypt_block v0, w2, x1, x7, w8 + b .Lmacloop4x +.Lmac1x: + add w3, w3, #4 .Lmacloop: cbz w3, .Lmacout ld1 {v1.16b}, [x0], #16 /* get next pt block */ @@ -406,7 +426,6 @@ AES_ENTRY(aes_mac_update) csinv x5, x6, xzr, eq cbz w5, .Lmacout -.Lmacenc: encrypt_block v0, w2, x1, x7, w8 b .Lmacloop From patchwork Sat Mar 10 15:21:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 131300 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp2244727lja; Sat, 10 Mar 2018 07:23:04 -0800 (PST) X-Google-Smtp-Source: AG47ELuzq3i8luHrTHEuT1cBWyTUl3epxeLCdmZI3g+hJ26qAVB/+qE91FxRWxMjU5xLHkSobl9y X-Received: by 10.101.77.195 with SMTP id q3mr1841354pgt.283.1520695383954; Sat, 10 Mar 2018 07:23:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520695383; cv=none; d=google.com; s=arc-20160816; b=VgWe2p2TC+aoh7iSm21tgzr8cEgOMa5oEAIY5mRXhlKiyuuo/WjP04AhEipZsxCzVV 6qR6DFzr8akNO94D5c+fLNWrFKC2+GJV5aLltvqJHTBpmeyklauArrve5LJ5+7FbwHAP HG2Nb6/Kq4wsCcb8nA41vHRKq3BUY/vavR5DcTc78QonV1+4KvFX7jsSYsXUK6rgPFuQ 6y//nq2kxzWaCrsKu3ti4BT1XHcYRZrSizBSdkidn3rB1DIMS5aaOvq+TvgCxOR+Ctyf nnb33rugi1qBTF1sx8fG+HMUvrfiFZKP3jxrCNjOlsZRI9+FpPNIWRl3TuQU/QLXCKmF Z/3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=ghtE/+HMhJ7+BEn8JVj07Uf/eaORWShrkPULyd1WCu8=; b=xJ4qGBwVo6ySMiD8Ppsus3yJlHGxd7eS6pa1gi30+vbs3BkvbMgtlvx/1LkQI9/ee5 r8Y0QtTm2Oa4NGOuXRuMomB1sNwSq6aIcHxjRxD1KP13tAm9Aj38eVQL4gS/diBULdA+ pN9kqpmYYWKADd544xqWN5t+TGYcKAjic9gaNlZXxO9LuZZjbPR+Gb+FdM7V+GGNdaaf XfgTlf2v7goIsd94m0u6rdfQCNRuqbLsGiw05kU+POVlK1aAKWUS3fYNlg+MKBY6FWpr uFsMNi3AUVmzVu+XG7tJYWogubC6FjmHf19gq3Lo3lRnaeeeaeix2CloKGMpthzFBSWn Z22g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=VMf6V7QQ; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q16si2824191pfg.221.2018.03.10.07.23.03; Sat, 10 Mar 2018 07:23:03 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=VMf6V7QQ; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932308AbeCJPXC (ORCPT + 1 other); Sat, 10 Mar 2018 10:23:02 -0500 Received: from mail-wm0-f66.google.com ([74.125.82.66]:50589 "EHLO mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932325AbeCJPXA (ORCPT ); Sat, 10 Mar 2018 10:23:00 -0500 Received: by mail-wm0-f66.google.com with SMTP id w128so8941889wmw.0 for ; Sat, 10 Mar 2018 07:22:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ghtE/+HMhJ7+BEn8JVj07Uf/eaORWShrkPULyd1WCu8=; b=VMf6V7QQDK1FdmxfpKtNnLbFsdW5ywaElZ+MUDBevO5/6hGLvC2p+mpzayQBOluVa4 DVZ4pLMe3/1JAHYXiLF829Or0FglWvOUFrOEQH3+2oY54mis9Np5hyaUs/shwLn6yEpz s7rQGiyK9CbdbJoA/C5VfHJwEycb5VH3TTEbM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ghtE/+HMhJ7+BEn8JVj07Uf/eaORWShrkPULyd1WCu8=; b=stRVMQpjd/2iB8b2mtQzx5wEtO+4oXS7zk+UHRtoN5voyUF1pa3cP0lqRpJNKwikVn TFio6z2GbM8Dinavd8Jld3Z5HRBvVAsUJ1Vnbdndkc6eaW2mvtS+J77RiHufmQBW444J v1pVeD1OdPg1OqaLPnHuFYUJhtVKrgIOWTHX+Czh0uEVA+9mjbnRFWrw4a4jkj0fr9l9 OH+BECLiZaDCl95iJQeF9XXgMv0a6iL0b1V6rlD4Yjyw/XCwjYSJjapyECys3ZMrnsug Iy/uLIA95hu5UuynjDHdV9M0X0how6uipGkzE0fMCWbmFIXVONCQXbUFfFpkJ9jb215J fU+A== X-Gm-Message-State: AElRT7Gb1XULrk8eb0TI6xRQypY85zjBYhXMfStbMwijB47Ho5rLHaAq 00lfoqoFkMFM6b93Wm77k3dFoSfMxOA= X-Received: by 10.28.1.14 with SMTP id 14mr1318939wmb.40.1520695378429; Sat, 10 Mar 2018 07:22:58 -0800 (PST) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id m9sm7027531wrf.13.2018.03.10.07.22.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:22:57 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v5 10/23] arm64: assembler: add utility macros to push/pop stack frames Date: Sat, 10 Mar 2018 15:21:55 +0000 Message-Id: <20180310152208.10369-11-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180310152208.10369-1-ard.biesheuvel@linaro.org> References: <20180310152208.10369-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org We are going to add code to all the NEON crypto routines that will turn them into non-leaf functions, so we need to manage the stack frames. To make this less tedious and error prone, add some macros that take the number of callee saved registers to preserve and the extra size to allocate in the stack frame (for locals) and emit the ldp/stp sequences. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 70 ++++++++++++++++++++ 1 file changed, 70 insertions(+) -- 2.15.1 diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 053d83e8db6f..eef1fd2c1c0b 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -555,6 +555,19 @@ USER(\label, ic ivau, \tmp2) // invalidate I line PoU #endif .endm +/* + * Errata workaround post TTBR0_EL1 update. + */ + .macro post_ttbr0_update_workaround +#ifdef CONFIG_CAVIUM_ERRATUM_27456 +alternative_if ARM64_WORKAROUND_CAVIUM_27456 + ic iallu + dsb nsh + isb +alternative_else_nop_endif +#endif + .endm + /** * Errata workaround prior to disable MMU. Insert an ISB immediately prior * to executing the MSR that will change SCTLR_ELn[M] from a value of 1 to 0. @@ -565,4 +578,61 @@ USER(\label, ic ivau, \tmp2) // invalidate I line PoU #endif .endm + /* + * frame_push - Push @regcount callee saved registers to the stack, + * starting at x19, as well as x29/x30, and set x29 to + * the new value of sp. Add @extra bytes of stack space + * for locals. + */ + .macro frame_push, regcount:req, extra + __frame st, \regcount, \extra + .endm + + /* + * frame_pop - Pop the callee saved registers from the stack that were + * pushed in the most recent call to frame_push, as well + * as x29/x30 and any extra stack space that may have been + * allocated. + */ + .macro frame_pop + __frame ld + .endm + + .macro __frame_regs, reg1, reg2, op, num + .if .Lframe_regcount == \num + \op\()r \reg1, [sp, #(\num + 1) * 8] + .elseif .Lframe_regcount > \num + \op\()p \reg1, \reg2, [sp, #(\num + 1) * 8] + .endif + .endm + + .macro __frame, op, regcount, extra=0 + .ifc \op, st + .if (\regcount) < 0 || (\regcount) > 10 + .error "regcount should be in the range [0 ... 10]" + .endif + .if ((\extra) % 16) != 0 + .error "extra should be a multiple of 16 bytes" + .endif + .set .Lframe_regcount, \regcount + .set .Lframe_extra, \extra + .set .Lframe_local_offset, ((\regcount + 3) / 2) * 16 + stp x29, x30, [sp, #-.Lframe_local_offset - .Lframe_extra]! + mov x29, sp + .elseif .Lframe_regcount == -1 // && op == 'ld' + .error "frame_push/frame_pop may not be nested" + .endif + + __frame_regs x19, x20, \op, 1 + __frame_regs x21, x22, \op, 3 + __frame_regs x23, x24, \op, 5 + __frame_regs x25, x26, \op, 7 + __frame_regs x27, x28, \op, 9 + + .ifc \op, ld + ldp x29, x30, [sp], #.Lframe_local_offset + .Lframe_extra + .set .Lframe_regcount, -1 + .endif + .endm + #endif /* __ASM_ASSEMBLER_H */ From patchwork Sat Mar 10 15:21:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 131301 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp2244829lja; Sat, 10 Mar 2018 07:23:10 -0800 (PST) X-Google-Smtp-Source: AG47ELvRPR+KWYkWbGL5GFR4TSeM1RDtmSjiEMqa+vqiKZgseNwuCkKymf0uhi4+ELK4m8uUskvs X-Received: by 10.99.96.84 with SMTP id u81mr1372017pgb.231.1520695390223; Sat, 10 Mar 2018 07:23:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520695390; cv=none; d=google.com; s=arc-20160816; b=QMG2dXIIVRd5r8ICFtsfQpOaP+Zk5TRRsPB8PbGwxPY9wdqvLjRIRBqfZWgjxLoFgJ pk4s+8SJ8TlVAd3c8u+J3B8QSdDxE1DrqsTEa4QOxh7YDYuvl0QJ+WBqhgYTJnRVpdLU 3JSlSqGmBL26hqeJEB6xAq5wmtvrUaYbxBySh+ekwhO5hkqVov8AfsH6YZ/pCctS66Rl 8o6AduXJjN++4h/oDGnjx+XxpQTsyM7FXeQrDijHefZEWLrt/w0lPWIKRXxFYw2Xlb22 9D+6VcPkRN6kMbWMkrCjLDD4HLXLDszVPPPmY8byfPgddpeNMXUHbfDmc3oduRuVEGX0 YX3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=igH9KImZTLC7b3FEy2vhduXKBfEy6hhH3sJVJ6VEP8Q=; b=sP1ISq2h/HbZqN/gXewN7sGVtyChO1rlFzECNknCzUesKUkXcnOsPNLDvV78CibZNB B6bydKwSAwgmOx18IbkA8WcbwNgm9QiJC5Jeg+yqPGRS74EHWonc3xa3anOfhQ/pl20R moVF8kwnK4gwXQJ7iLC4c9dP5gWaCnA762BzIq+r8+uBnqrGXaFLrv2OaLyF+oVmDOIv cpxMTubpBdNQ+7M5qatOOMIKHouUz8gaEVtBb2ihDXGfTxPEiW9nhAfluzsvgu5CXDl6 j/nouIhmCmRJAatg06WWZ71lhfr1oz1h6mOUhWOaSJNqspMjIpdH+eUu/g/nSKkPAJvn CSDA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=FVsvPWHi; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q16si2824191pfg.221.2018.03.10.07.23.10; Sat, 10 Mar 2018 07:23:10 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=FVsvPWHi; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932338AbeCJPXH (ORCPT + 1 other); Sat, 10 Mar 2018 10:23:07 -0500 Received: from mail-wm0-f65.google.com ([74.125.82.65]:33338 "EHLO mail-wm0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932324AbeCJPXD (ORCPT ); Sat, 10 Mar 2018 10:23:03 -0500 Received: by mail-wm0-f65.google.com with SMTP id s206so7245065wme.0 for ; Sat, 10 Mar 2018 07:23:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=igH9KImZTLC7b3FEy2vhduXKBfEy6hhH3sJVJ6VEP8Q=; b=FVsvPWHitHJSBOohv0LlBjibo2VrwyfVHxbYh4IhXPj6AMPo41FiXMR2a8MlyBIYyU xKJgA5UQzEAHOAeVWumMh/oPI3S8vy+QeSuxNYdzc7rgXz9AXQfR5a/W9ZUNYTwnD/Qa NBTMgEPCUbHkANZHWDzeqJql6L01j2f+pOWPk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=igH9KImZTLC7b3FEy2vhduXKBfEy6hhH3sJVJ6VEP8Q=; b=Ai7ZHH/A4kwSjBZGbAp6CeXgea3sIQUj4gGvLMiGcNOE3/9ynfX3auDnk90zbCjpk1 xvUeRwauwFSEQTkd9G6n5lYvjbQhlOloGNmna3M/ZKM3Gu/DDElOPZIli2MRvzfMdbVA LN3lJkSglSUXAlKx0IktNKLxCsfK7S9erGEsTE4Mh5yavv/Zpc7GBqZqmwRchvl6m7kQ QrN0bW7nRkz2OtCH2Apsm1GNaOjLEVdRiNMMzSDSq3rTt2yf8ChQQMWFaacZX8md3bKY uJZZuyGRO7VD538oHZp6ReJ97V8DhebcegWeAxnn41q9nEoxQa8m/X2iwz/BNq5gOo+w /j1g== X-Gm-Message-State: AElRT7EniOszoLXva/KesPRymVh4ETMuZe068DfbDcwFGtum1nVnEcHr kyJ3phHIj9QVJjsJNX0Qd+46bOVQAx8= X-Received: by 10.28.191.138 with SMTP id o10mr1328691wmi.26.1520695381839; Sat, 10 Mar 2018 07:23:01 -0800 (PST) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id m9sm7027531wrf.13.2018.03.10.07.22.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:23:01 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v5 11/23] arm64: assembler: add macros to conditionally yield the NEON under PREEMPT Date: Sat, 10 Mar 2018 15:21:56 +0000 Message-Id: <20180310152208.10369-12-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180310152208.10369-1-ard.biesheuvel@linaro.org> References: <20180310152208.10369-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Add support macros to conditionally yield the NEON (and thus the CPU) that may be called from the assembler code. In some cases, yielding the NEON involves saving and restoring a non trivial amount of context (especially in the CRC folding algorithms), and so the macro is split into three, and the code in between is only executed when the yield path is taken, allowing the context to be preserved. The third macro takes an optional label argument that marks the resume path after a yield has been performed. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 64 ++++++++++++++++++++ arch/arm64/kernel/asm-offsets.c | 2 + 2 files changed, 66 insertions(+) -- 2.15.1 diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index eef1fd2c1c0b..61168cbe9781 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -635,4 +635,68 @@ alternative_else_nop_endif .endif .endm +/* + * Check whether to yield to another runnable task from kernel mode NEON code + * (which runs with preemption disabled). + * + * if_will_cond_yield_neon + * // pre-yield patchup code + * do_cond_yield_neon + * // post-yield patchup code + * endif_yield_neon