From patchwork Thu Jan 24 18:27:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 156505 Delivered-To: patch@linaro.org Received: by 2002:a02:48:0:0:0:0:0 with SMTP id 69csp2284421jaa; Thu, 24 Jan 2019 10:27:23 -0800 (PST) X-Google-Smtp-Source: ALg8bN5gp1gWy1pgeYrKIrl87FB7/0Tbt7KOvxsjepAlLA+hh3f8heQIXg4jcuf0Jj1NLrXCIkp/ X-Received: by 2002:a63:8c0d:: with SMTP id m13mr6987188pgd.422.1548354443028; Thu, 24 Jan 2019 10:27:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1548354443; cv=none; d=google.com; s=arc-20160816; b=zNopygkoseHODWGBTGDjzNNf7x0QOjU34BT/nKaF/R6mtcxlSYXXyKc0ZDUB/xxrEL cANW6Z7cEt51Qe08H2MnByg8Mdop3ePa0OLm1fpEb+Hl2qJfUhqO3qBZoBIikUIspJG3 c5M7nSzHRokIvbn7Xx84s9j1MVK9CNwcTBgxj44XZZdUCYaykkZ8qm7oGmZjaW5qKIoG g1fmP1ORBklTGbZXgZmIWx55EdVYxAp9NMb/L7l99dQDaFA/e757Rva+mCOUm6B+VQLi Jy/7LB0Eifb/SJ8z/JbVhQK+xDfeEE7BNQr5/3qBqZKcfvdWEs/0y8gPB47y83TKDRx+ SyCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=usyANgO+md6MAxAYC+xPQ71SM8cvmj1kbE42YVBL7tM=; b=0/ZVq1ZIhz9NPoAywImAOrzJLxRy1UT//MieOX1gOn6ZMUoM0IjvZ6CVxXwcKnZD8N +5sqtIiiv3qvoDlZv22qEXOt57/aSzrRfVyRdZgVZa1OnMqvaE01LzIYjQcGx0WQ54ag vBFONR/R6gX1mDWQPrQsJxXg5Qx644pLGsE6RyBMbyNJ8RmjFHOveRnfgIaI25TP+VdP T6TWwQrANp3k20YQH62j0eSvA3kXqNY4Qm8btPqYj9ECwQjWuZ3mXtQxP0FPkDjsolGc jdV69ZvY7DCJ6+QIUU4wZC2duwjf/lSQuiwGdT7HdtdWuoVY8FwDvBil/SZfe+z55jvf /Qxw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=hl6687nM; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id n4si22707825pgm.487.2019.01.24.10.27.22; Thu, 24 Jan 2019 10:27:23 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=hl6687nM; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726026AbfAXS1W (ORCPT + 3 others); Thu, 24 Jan 2019 13:27:22 -0500 Received: from mail-wm1-f65.google.com ([209.85.128.65]:39538 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725909AbfAXS1W (ORCPT ); Thu, 24 Jan 2019 13:27:22 -0500 Received: by mail-wm1-f65.google.com with SMTP id y8so4106117wmi.4 for ; Thu, 24 Jan 2019 10:27:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=usyANgO+md6MAxAYC+xPQ71SM8cvmj1kbE42YVBL7tM=; b=hl6687nMxC/Lvjr/tM2DjrjI/c8ecYNEFmGYcpWIz77p8QVJs+H+jeO7eVzqdxvccp mTMjUSf5HKVBt5Zic8+nAiGOYWobjoBpdi5d5nsFkkGsb42uTDGMuzhsyH06T4Yk6AE3 jnTYDN0edqWfUgdVxi/XmW+dAlLOU3jED3uRQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=usyANgO+md6MAxAYC+xPQ71SM8cvmj1kbE42YVBL7tM=; b=cn8va3o4vYbEUpyUCxWLzPXWRhgZrTx7NXyvlVNj7+5RDGRfUkAth7kaK6frJGRe7s r9Q2cOE+PFJEQUGZ3ciVxtuNGrPRHesrX/uO/h0kpYTEGVSjHehvTp6y9+zJmiVGQaxT USisBf494T3jn46tCGCJguRkNbBihsNkaHxmEQz/YY/fx/e9e4Dkup4y/OQedSzeNg3l Anvb8KVlEO2NHb8GvrQGwB3JGsIyChvaZcGiXomxXUDliAqFMwmd8m6SlQdHgJOruyiL 4Zj37+ZSQaqB4StJNrq4C2jMaaIR18bqKIrv4OKbOkCkNJPtW9vEz4tZlb7AUNj/kG8V x/OQ== X-Gm-Message-State: AJcUukfz+Wgz3Wtnv6RqlSGJIdRoOyAdtv3Ar99copnNqaLX5vBq6wDy sHZVYW47xMXUubE9sFYRtmGqQwN1n5rpag== X-Received: by 2002:a1c:e044:: with SMTP id x65mr3685792wmg.86.1548354439745; Thu, 24 Jan 2019 10:27:19 -0800 (PST) Received: from dogfood.home ([2a01:cb1d:112:6f00:bcd4:806e:230a:673e]) by smtp.gmail.com with ESMTPSA id 202sm73247138wmt.8.2019.01.24.10.27.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 Jan 2019 10:27:18 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 1/2] crypto: arm/crct10dif - revert to C code for short inputs Date: Thu, 24 Jan 2019 19:27:11 +0100 Message-Id: <20190124182712.7142-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190124182712.7142-1-ard.biesheuvel@linaro.org> References: <20190124182712.7142-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The SIMD routine ported from x86 used to have a special code path for inputs < 16 bytes, which got lost somewhere along the way. Instead, the current glue code aligns the input pointer to permit the NEON routine to use special versions of the vld1 instructions that assume 16 byte alignment, but this could result in inputs of less than 16 bytes to be passed in. This not only fails the new extended tests that Eric has implemented, it also results in the code reading before the input pointer, which could potentially result in crashes when dealing with less than 16 bytes of input at the start of a page which is preceded by an unmapped page. So update the glue code to only invoke the NEON routine if the input is more than 16 bytes. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/crct10dif-ce-core.S | 20 ++++++++--------- arch/arm/crypto/crct10dif-ce-glue.c | 23 +++++--------------- 2 files changed, 16 insertions(+), 27 deletions(-) -- 2.17.1 diff --git a/arch/arm/crypto/crct10dif-ce-core.S b/arch/arm/crypto/crct10dif-ce-core.S index ce45ba0c0687..3fd13d7c842c 100644 --- a/arch/arm/crypto/crct10dif-ce-core.S +++ b/arch/arm/crypto/crct10dif-ce-core.S @@ -124,10 +124,10 @@ ENTRY(crc_t10dif_pmull) vext.8 q10, qzr, q0, #4 // receive the initial 64B data, xor the initial crc value - vld1.64 {q0-q1}, [arg2, :128]! - vld1.64 {q2-q3}, [arg2, :128]! - vld1.64 {q4-q5}, [arg2, :128]! - vld1.64 {q6-q7}, [arg2, :128]! + vld1.64 {q0-q1}, [arg2]! + vld1.64 {q2-q3}, [arg2]! + vld1.64 {q4-q5}, [arg2]! + vld1.64 {q6-q7}, [arg2]! CPU_LE( vrev64.8 q0, q0 ) CPU_LE( vrev64.8 q1, q1 ) CPU_LE( vrev64.8 q2, q2 ) @@ -150,7 +150,7 @@ CPU_LE( vrev64.8 q7, q7 ) veor.8 q0, q0, q10 adr ip, rk3 - vld1.64 {q10}, [ip, :128] // xmm10 has rk3 and rk4 + vld1.64 {q10}, [ip] // xmm10 has rk3 and rk4 // // we subtract 256 instead of 128 to save one instruction from the loop @@ -167,7 +167,7 @@ CPU_LE( vrev64.8 q7, q7 ) _fold_64_B_loop: .macro fold64, reg1, reg2 - vld1.64 {q11-q12}, [arg2, :128]! + vld1.64 {q11-q12}, [arg2]! vmull.p64 q8, \reg1\()h, d21 vmull.p64 \reg1, \reg1\()l, d20 @@ -203,13 +203,13 @@ CPU_LE( vrev64.8 q12, q12 ) // constants adr ip, rk9 - vld1.64 {q10}, [ip, :128]! + vld1.64 {q10}, [ip]! .macro fold16, reg, rk vmull.p64 q8, \reg\()l, d20 vmull.p64 \reg, \reg\()h, d21 .ifnb \rk - vld1.64 {q10}, [ip, :128]! + vld1.64 {q10}, [ip]! .endif veor.8 q7, q7, q8 veor.8 q7, q7, \reg @@ -238,7 +238,7 @@ _16B_reduction_loop: vmull.p64 q7, d15, d21 veor.8 q7, q7, q8 - vld1.64 {q0}, [arg2, :128]! + vld1.64 {q0}, [arg2]! CPU_LE( vrev64.8 q0, q0 ) vswp d0, d1 veor.8 q7, q7, q0 @@ -335,7 +335,7 @@ _less_than_128: vmov.i8 q0, #0 vmov s3, arg1_low32 // get the initial crc value - vld1.64 {q7}, [arg2, :128]! + vld1.64 {q7}, [arg2]! CPU_LE( vrev64.8 q7, q7 ) vswp d14, d15 veor.8 q7, q7, q0 diff --git a/arch/arm/crypto/crct10dif-ce-glue.c b/arch/arm/crypto/crct10dif-ce-glue.c index d428355cf38d..14c19c70a841 100644 --- a/arch/arm/crypto/crct10dif-ce-glue.c +++ b/arch/arm/crypto/crct10dif-ce-glue.c @@ -35,26 +35,15 @@ static int crct10dif_update(struct shash_desc *desc, const u8 *data, unsigned int length) { u16 *crc = shash_desc_ctx(desc); - unsigned int l; - if (!may_use_simd()) { - *crc = crc_t10dif_generic(*crc, data, length); + if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) { + kernel_neon_begin(); + *crc = crc_t10dif_pmull(*crc, data, length); + kernel_neon_end(); } else { - if (unlikely((u32)data % CRC_T10DIF_PMULL_CHUNK_SIZE)) { - l = min_t(u32, length, CRC_T10DIF_PMULL_CHUNK_SIZE - - ((u32)data % CRC_T10DIF_PMULL_CHUNK_SIZE)); - - *crc = crc_t10dif_generic(*crc, data, l); - - length -= l; - data += l; - } - if (length > 0) { - kernel_neon_begin(); - *crc = crc_t10dif_pmull(*crc, data, length); - kernel_neon_end(); - } + *crc = crc_t10dif_generic(*crc, data, length); } + return 0; } From patchwork Thu Jan 24 18:27:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 156506 Delivered-To: patch@linaro.org Received: by 2002:a02:48:0:0:0:0:0 with SMTP id 69csp2284441jaa; Thu, 24 Jan 2019 10:27:24 -0800 (PST) X-Google-Smtp-Source: ALg8bN5TJC87XXFvm0Se/ELHNxcRRF7UqA8gei9vcHblX7wdFSxGMYL4S8ne6RJP3bwIkrpkQJzb X-Received: by 2002:a17:902:9687:: with SMTP id n7mr7384399plp.94.1548354444116; Thu, 24 Jan 2019 10:27:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1548354444; cv=none; d=google.com; s=arc-20160816; b=iFiwqKa8AXDkU6NKW63XnSP8WM2eNHkhinPpKBfTawNR9PVL3NM6NMfH7mhKUOyyij ky5rktVUnkq/YP77h8OYLZifrfuDTJJcdmQmmjNtc4ktK80VgIyKWysr7hLQjtyL3+hB o4reHOfa8ckL252dAqa2Bbiv3UvVb57/9RXQnMCx9L6uF+cccMiF0pwGpMr0NDKAv2rJ euncy2nSSE0j9RKYPaBBF47z9WwM/oPjFAQky2n7WdqQ6ZqS2GgsiIPux3b1fNXzbk07 JEcQJlyIUqO4Lx0dGiMCn8LTxxuOxJNd0wN2T0vOzqsnGdf+WUp2J3glDhRHqy1G0YHa bOzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=3/1JsJbOO9AapMWTRThG5LlPeoGHhP1SPxErwHi+5nY=; b=X4W69KW0jtzJaiPqlGhXCUeA8tLAYzl3JX1572aHL/aVgMIs30LgI3vi9uDr5IUum5 H2Vdc46dK1KWo2fZ83STLmuf2YvSbUCrACwUI3Wecaupn/j/+pTKJEfb7vy8dPwsnECR y5Q8GJ8bSYOinFwVSo/5oXS5J7ARavTV+o1bpHmTUqUGyPDXN7wubfY02XhFiighsFKU lHyZ/KQkEkyK0bDxl4lOh12OYL/gFk+6Ei31wOY935CaBeKJKwQna1zxCkJa9T2xboWk 2jXdnvwkFZjnYXCjxX9Ilr1XRIW5UqUz48B4vLrScKbUiyt5+TepMlPxEjAGne3Bcmz1 lO1w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=aIpSZUY3; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id n4si22707825pgm.487.2019.01.24.10.27.23; Thu, 24 Jan 2019 10:27:24 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=aIpSZUY3; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725909AbfAXS1X (ORCPT + 3 others); Thu, 24 Jan 2019 13:27:23 -0500 Received: from mail-wr1-f66.google.com ([209.85.221.66]:34529 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726040AbfAXS1X (ORCPT ); Thu, 24 Jan 2019 13:27:23 -0500 Received: by mail-wr1-f66.google.com with SMTP id f7so7621514wrp.1 for ; Thu, 24 Jan 2019 10:27:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3/1JsJbOO9AapMWTRThG5LlPeoGHhP1SPxErwHi+5nY=; b=aIpSZUY3HDECVs6ewDdoEN9Hdyis2ferKU36Vhx9iyGGId0acU/LfAtbxIhx6gcIuE 7XMpikyGfot/vqiMvqbqgqh7pC9Qnp7EPME7IXlJRVzkUbJFdwNoHJfXc82NiVyZWluV 7PGeqVprHgmqPoVodW4vgj54HJ0cZh7Y33O5A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3/1JsJbOO9AapMWTRThG5LlPeoGHhP1SPxErwHi+5nY=; b=GdXNK/WowIHtb6udMnXriiQx5PW8FacQ0lQ2i7Gee5iirQOzcq7XiJ8o7MBh0kWZsC TfBudkpSAt94eXTq8iJmlwqvieWNLy63wVvOyfSHxNjiqFkjAGGjNlaiJ1GZJZm+PUyl 483gWTq2br5mSUMB+UuCrZD9u04KdBdoQmgHIDIwi8Ns5fDbXUIfEMao22ee+Z5C02wr EynjfgWMAIu1U5W6iRdWIzwFkbe/edsYDFa2UWljZeUm38wZjE+AYq9jn2PGxJ+McuxK jXiquPPw0mWlDc1xeFTC6SE6oHHyuKbLozQ0C0hLoX7ayNvlXeOBvmN80xwt7u0AhqiJ mprA== X-Gm-Message-State: AJcUukc9nGMY1f/m7NigUZrwFJPU2RYIFOg9qtDXsQNzLcJCCDsT2tPB WoDd0y1NBVSL7AYBHEGqDBogsS4v/WBveA== X-Received: by 2002:adf:f550:: with SMTP id j16mr8145818wrp.258.1548354441226; Thu, 24 Jan 2019 10:27:21 -0800 (PST) Received: from dogfood.home ([2a01:cb1d:112:6f00:bcd4:806e:230a:673e]) by smtp.gmail.com with ESMTPSA id 202sm73247138wmt.8.2019.01.24.10.27.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 Jan 2019 10:27:20 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 2/2] crypto: arm64/crct10dif - revert to C code for short inputs Date: Thu, 24 Jan 2019 19:27:12 +0100 Message-Id: <20190124182712.7142-3-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190124182712.7142-1-ard.biesheuvel@linaro.org> References: <20190124182712.7142-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The SIMD routine ported from x86 used to have a special code path for inputs < 16 bytes, which got lost somewhere along the way. Instead, the current glue code aligns the input pointer to permit the NEON routine to use special versions of the vld1 instructions that assume 16 byte alignment, but this could result in inputs of less than 16 bytes to be passed in. This not only fails the new extended tests that Eric has implemented, it also results in the code reading before the input pointer, which could potentially result in crashes when dealing with less than 16 bytes of input at the start of a page which is preceded by an unmapped page. So update the glue code to only invoke the NEON routine if the input is more than 16 bytes. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/crct10dif-ce-glue.c | 25 +++++--------------- 1 file changed, 6 insertions(+), 19 deletions(-) -- 2.17.1 diff --git a/arch/arm64/crypto/crct10dif-ce-glue.c b/arch/arm64/crypto/crct10dif-ce-glue.c index b461d62023f2..567c24f3d224 100644 --- a/arch/arm64/crypto/crct10dif-ce-glue.c +++ b/arch/arm64/crypto/crct10dif-ce-glue.c @@ -39,26 +39,13 @@ static int crct10dif_update(struct shash_desc *desc, const u8 *data, unsigned int length) { u16 *crc = shash_desc_ctx(desc); - unsigned int l; - if (unlikely((u64)data % CRC_T10DIF_PMULL_CHUNK_SIZE)) { - l = min_t(u32, length, CRC_T10DIF_PMULL_CHUNK_SIZE - - ((u64)data % CRC_T10DIF_PMULL_CHUNK_SIZE)); - - *crc = crc_t10dif_generic(*crc, data, l); - - length -= l; - data += l; - } - - if (length > 0) { - if (may_use_simd()) { - kernel_neon_begin(); - *crc = crc_t10dif_pmull(*crc, data, length); - kernel_neon_end(); - } else { - *crc = crc_t10dif_generic(*crc, data, length); - } + if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) { + kernel_neon_begin(); + *crc = crc_t10dif_pmull(*crc, data, length); + kernel_neon_end(); + } else { + *crc = crc_t10dif_generic(*crc, data, length); } return 0;