From patchwork Mon Oct 8 21:15:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 148445 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp4125256lji; Mon, 8 Oct 2018 14:16:09 -0700 (PDT) X-Google-Smtp-Source: ACcGV61iqVvZj3TnGK3n2wr/ugFJ4OZnmMO5CNtoOwmOKrWrY0VWKoKkHUPls4BTmsz0KdU34ME1 X-Received: by 2002:a62:70c7:: with SMTP id l190-v6mr26639166pfc.186.1539033369543; Mon, 08 Oct 2018 14:16:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539033369; cv=none; d=google.com; s=arc-20160816; b=gdevJ7+I39AuxKtzp9TuMUP8rYPOuTZWCtIo4mZdw+moQ+4DqZtZHOKrX0hkMo7rcc leHADcrLcuiyRIVwkvrXNDU3clG5pTc8xMWOUnufZ/nlCSSJ1qDhdqummJoxk+QTSAVp rrCoQAAdyhnfge7I9lbpjyPJt4RVSlPYuRPHYim3J4GBQZwTimfc8sDIw+eldIjfqPhS cV8oO7+zoWHAp/P+F1bwLwiPnWDIptnsGWBw06HqhFxAYAcemDQdyRHS5+Is+pRuv3Qd eqCAFbSWfrrBY/NKipLflCTpIzqs+DCtz+M6K4CbudvayJNaLNCYzHefMfhQd0d7XfvT JeYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=yVyxxH1ii59qYZF1tTV85yOwfr9+lABlveFtUiVcJ1o=; b=TeZjE9KrGoU5h/FYJ2MPtlUB37srAZqqF0Mi25xFtc5PxxpVIPjWhHwSRRosJ8jAys DPNdGvOFLWtQT1SM9PypQbePWl0Kd591jBlzvTdWmty2ROFg0XsZUL4zinnJtGtS4Vk0 0NNnkULO7YAgQWgeSpxZsbVitjKqxbKM9oHQSPK/7DCB0Ggqi4CDpyvEgYdqf/iOJ7r6 rEiV1vTkBaSCz0tGpo1Cpvl4BGEy9gUSc811J9pcA6EeBuwB3OMDdCYA5PvEfOOtPVxo rgqWA/qNBaKPdxY3Z6hHMGalNfBQ3DStx3vhuf48wf8+DethNBd37lUmo4dJ7MTCQwcJ oPFA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=PBqbW6uT; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w4-v6si18552811pll.214.2018.10.08.14.16.09; Mon, 08 Oct 2018 14:16:09 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=PBqbW6uT; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726830AbeJIE3t (ORCPT + 2 others); Tue, 9 Oct 2018 00:29:49 -0400 Received: from mail-wm1-f67.google.com ([209.85.128.67]:40723 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725828AbeJIE3t (ORCPT ); Tue, 9 Oct 2018 00:29:49 -0400 Received: by mail-wm1-f67.google.com with SMTP id z204-v6so9612029wmc.5 for ; Mon, 08 Oct 2018 14:16:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=yVyxxH1ii59qYZF1tTV85yOwfr9+lABlveFtUiVcJ1o=; b=PBqbW6uTfN5iffbPRE1SIC5r3SoXgOP4NpaaUNKHUlamM5mbc7Y7g5XrgIAlvLYQrN lL+nGpFu9zOhdoBJsSE4TIyUcq5aX+0vRITgY4CXvXiWROA5n6RutE5HQ7av3uDRu4PJ CrcyaB/LWlQR5HTRec71Ei5nAmBeBTtuqjJEA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=yVyxxH1ii59qYZF1tTV85yOwfr9+lABlveFtUiVcJ1o=; b=LYntiDLxevbeAYBv+ujuIsjN25arHhBUISpb5xfqOZ1/EA3aPn8YOFlVJjbUHpj+G1 RDDJttY+NJZ0neTxXOc3b7jouoHxlB+6WMX8r0XFxGosPgUGfTK4yIwvdCTiZho6drpS bFTS/SA9ct5HoIJrT6GIUi36iEsNrEdidh8QxtEvSEmGSP4BiIEbEUVpAls3j0db2kbH 2CqOx0iqbvW8XYqvFmQNr65NpEXAIAF6euT3mVPwIc8MdOEwiZZc97Fk/ZwhNscJJPI/ DveAvZMbgWnHIvaon8PNfaI0gBljhMESx51CR17st1/n3kiUY+DnihUc35lX8Qujoe2F trmg== X-Gm-Message-State: ABuFfohjob3VVirn4AbHlY3H6HHemWeIgnfp8RPWhT/Fwcf/+/FK/1do vqcSwneQrZ2HElmV3PepsoU2zHiKIgw= X-Received: by 2002:a1c:b58e:: with SMTP id e136-v6mr15619838wmf.114.1539033366609; Mon, 08 Oct 2018 14:16:06 -0700 (PDT) Received: from localhost.localdomain ([2a01:cb1d:112:6f00:8084:9715:d038:c67d]) by smtp.gmail.com with ESMTPSA id s24-v6sm7563308wmc.7.2018.10.08.14.16.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 08 Oct 2018 14:16:05 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, arnd@arndb.de, jason@zx2c4.com, ebiggers@google.com, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel Subject: [PATCH 1/3] crypto: memneq - use unaligned accessors for aligned fast path Date: Mon, 8 Oct 2018 23:15:52 +0200 Message-Id: <20181008211554.5355-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20181008211554.5355-1-ard.biesheuvel@linaro.org> References: <20181008211554.5355-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org On ARM v6 and later, we define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS because the ordinary load/store instructions (ldr, ldrh, ldrb) can tolerate any misalignment of the memory address. However, load/store double and load/store multiple instructions (ldrd, ldm) may still only be used on memory addresses that are 32-bit aligned, and so we have to use the CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS macro with care, or we may end up with a severe performance hit due to alignment traps that require fixups by the kernel. Fortunately, the get_unaligned() accessors do the right thing: when building for ARMv6 or later, the compiler will emit unaligned accesses using the ordinary load/store instructions (but avoid the ones that require 32-bit alignment). When building for older ARM, those accessors will emit the appropriate sequence of ldrb/mov/orr instructions. And on architectures that can truly tolerate any kind of misalignment, the get_unaligned() accessors resolve to the leXX_to_cpup accessors that operate on aligned addresses. So switch to the unaligned accessors for the aligned fast path. This will create the exact same code on architectures that can really tolerate any kind of misalignment, and generate code for ARMv6+ that avoids load/store instructions that trigger alignment faults. Signed-off-by: Ard Biesheuvel --- crypto/memneq.c | 24 ++++++++++++++------ 1 file changed, 17 insertions(+), 7 deletions(-) -- 2.11.0 diff --git a/crypto/memneq.c b/crypto/memneq.c index afed1bd16aee..0f46a6150f22 100644 --- a/crypto/memneq.c +++ b/crypto/memneq.c @@ -60,6 +60,7 @@ */ #include +#include #ifndef __HAVE_ARCH_CRYPTO_MEMNEQ @@ -71,7 +72,10 @@ __crypto_memneq_generic(const void *a, const void *b, size_t size) #if defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) while (size >= sizeof(unsigned long)) { - neq |= *(unsigned long *)a ^ *(unsigned long *)b; + unsigned long const *p = a; + unsigned long const *q = b; + + neq |= get_unaligned(p) ^ get_unaligned(q); OPTIMIZER_HIDE_VAR(neq); a += sizeof(unsigned long); b += sizeof(unsigned long); @@ -95,18 +99,24 @@ static inline unsigned long __crypto_memneq_16(const void *a, const void *b) #ifdef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS if (sizeof(unsigned long) == 8) { - neq |= *(unsigned long *)(a) ^ *(unsigned long *)(b); + unsigned long const *p = a; + unsigned long const *q = b; + + neq |= get_unaligned(p++) ^ get_unaligned(q++); OPTIMIZER_HIDE_VAR(neq); - neq |= *(unsigned long *)(a+8) ^ *(unsigned long *)(b+8); + neq |= get_unaligned(p) ^ get_unaligned(q); OPTIMIZER_HIDE_VAR(neq); } else if (sizeof(unsigned int) == 4) { - neq |= *(unsigned int *)(a) ^ *(unsigned int *)(b); + unsigned int const *p = a; + unsigned int const *q = b; + + neq |= get_unaligned(p++) ^ get_unaligned(q++); OPTIMIZER_HIDE_VAR(neq); - neq |= *(unsigned int *)(a+4) ^ *(unsigned int *)(b+4); + neq |= get_unaligned(p++) ^ get_unaligned(q++); OPTIMIZER_HIDE_VAR(neq); - neq |= *(unsigned int *)(a+8) ^ *(unsigned int *)(b+8); + neq |= get_unaligned(p++) ^ get_unaligned(q++); OPTIMIZER_HIDE_VAR(neq); - neq |= *(unsigned int *)(a+12) ^ *(unsigned int *)(b+12); + neq |= get_unaligned(p) ^ get_unaligned(q); OPTIMIZER_HIDE_VAR(neq); } else #endif /* CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS */ From patchwork Mon Oct 8 21:15:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 148446 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp4125303lji; Mon, 8 Oct 2018 14:16:11 -0700 (PDT) X-Google-Smtp-Source: ACcGV63AJv+puRyWTGd4d3u47OhrXz9WNjB/V/9tfvfOvqkrAREFtO1p+vwGxAfu2xqLnvF22jlr X-Received: by 2002:a17:902:167:: with SMTP id 94-v6mr25073209plb.142.1539033371748; Mon, 08 Oct 2018 14:16:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539033371; cv=none; d=google.com; s=arc-20160816; b=XKi9EnffdirYUIghRJ/1q4jPka0DXmsllgGofnC/voXXUgN13wud/kGtLXDo1ioAIZ +83JQ9lTP4IQ70KqfeZ+MottPu+XnenD5As8KESynrzeYPvIYf3i1m4WI8oPzgjWkfMv 1wzlVnep6bEcHygI2Afeo8AjLufz3XJrrxkQb9QipyiqUQ/8gv9mH97VU7USOkYhXkXV k/cmdQXLqk9EpOTa+d67akZbOFb86I/RDELZ5Xr752N7ulvgUyGymqs+IdzPN04SdkWx x0abpwVvYkZcaziLjP2X60lz4gSb7FqFUiXDAiRLoL8D2CDU0DHPK0Cg/zNGbTpKu4qZ EVYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=J0aEQXXrkbEOrgPdvQLuNJVqzwZkLxj5YarnobqFpmY=; b=JzYWUsiafTHDDqbsrJ1jUUEz+ebdGMHk/6d5tVpnMMSlvnL4Y6+IPWScZjXuhyb9X+ ebakAjIzmgpm2tRWOp59iHKjX387X3eITKUGQrnTlv9aKSwzTE/W4sOFy6zbd0/cN2gg fnF9AeuLkTKbLHMpgnjm8xPYXvuaOSHG9vhsDCepfivXsUQtHCK/whgv0bpn9BF33RIj HfHpDRlDX2oSDdo6E8RqcE/n0JMmlkogYl8JIVXn9fmseGKjCU4htm/SvW5r8zjmSoQO iqcq2n4m39AWJZ7REs0pgxJGu/JIPHvuzk0miDHtv7HQ0LzapM5lrdCH2gMS7vzlMmrl OnCg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QICMMtwn; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w4-v6si18552811pll.214.2018.10.08.14.16.11; Mon, 08 Oct 2018 14:16:11 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QICMMtwn; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726907AbeJIE3v (ORCPT + 2 others); Tue, 9 Oct 2018 00:29:51 -0400 Received: from mail-wm1-f68.google.com ([209.85.128.68]:37859 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725828AbeJIE3v (ORCPT ); Tue, 9 Oct 2018 00:29:51 -0400 Received: by mail-wm1-f68.google.com with SMTP id 185-v6so9631087wmt.2 for ; Mon, 08 Oct 2018 14:16:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=J0aEQXXrkbEOrgPdvQLuNJVqzwZkLxj5YarnobqFpmY=; b=QICMMtwn3ZdSKPwUMQdTRIHUAtlJNyjNIadf/MSj58IyxcxqKwNtGER7cJEyvzXeXa V37zUin8m75tjh26a9BCszmlTBQYd1uXX/aRuTK0ZiMiQNBQCWZeIFEODgXxmwtabViZ ZRFxj+5OCxE215VNU4ZKErCM+4Qlujm6sNcqE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=J0aEQXXrkbEOrgPdvQLuNJVqzwZkLxj5YarnobqFpmY=; b=ci+NWhIfmKCkp4scx8AIpnolMSNB4OjAFvmHHRezqxIm9p68QS86a6+WcjK+kz3/h7 a/7JmID9KP34mc2WnRlsXPNcH6uz66VyXVmmQx3RTJJhHIbCU0bfTYK3sxdeaX4QNwpY BhlltYL418iabS2odLbubejlxOG59fWQZ6HbWDQEeVWOgFddLyfc+U65mj+aewDG0R1X Hkh4jW8C1ttbeDPS4ocJD4GxqfpmahKHn+BhdCFtoDraIhZHfxUT1SlhJone3xDXYqkv LYv5iOxn57/Jd7E/1ILeh8OjAWygBbzOPutyb8JCGOAjYrU3pIx/z9qJy5zElBXfFsKL dcfw== X-Gm-Message-State: ABuFfogw8Eer6tCDCstm1EOvJl8tuxFEe94NXNk8Biaj8C5qFq+CeNwI zwcw/Rhjnzlm+XpgVzP7PY2w6G4M8Gc= X-Received: by 2002:a1c:bce:: with SMTP id 197-v6mr15370424wml.15.1539033367816; Mon, 08 Oct 2018 14:16:07 -0700 (PDT) Received: from localhost.localdomain ([2a01:cb1d:112:6f00:8084:9715:d038:c67d]) by smtp.gmail.com with ESMTPSA id s24-v6sm7563308wmc.7.2018.10.08.14.16.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 08 Oct 2018 14:16:07 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, arnd@arndb.de, jason@zx2c4.com, ebiggers@google.com, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel Subject: [PATCH 2/3] crypto: crypto_xor - use unaligned accessors for aligned fast path Date: Mon, 8 Oct 2018 23:15:53 +0200 Message-Id: <20181008211554.5355-3-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20181008211554.5355-1-ard.biesheuvel@linaro.org> References: <20181008211554.5355-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org On ARM v6 and later, we define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS because the ordinary load/store instructions (ldr, ldrh, ldrb) can tolerate any misalignment of the memory address. However, load/store double and load/store multiple instructions (ldrd, ldm) may still only be used on memory addresses that are 32-bit aligned, and so we have to use the CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS macro with care, or we may end up with a severe performance hit due to alignment traps that require fixups by the kernel. Fortunately, the get_unaligned() accessors do the right thing: when building for ARMv6 or later, the compiler will emit unaligned accesses using the ordinary load/store instructions (but avoid the ones that require 32-bit alignment). When building for older ARM, those accessors will emit the appropriate sequence of ldrb/mov/orr instructions. And on architectures that can truly tolerate any kind of misalignment, the get_unaligned() accessors resolve to the leXX_to_cpup accessors that operate on aligned addresses. So switch to the unaligned accessors for the aligned fast path. This will create the exact same code on architectures that can really tolerate any kind of misalignment, and generate code for ARMv6+ that avoids load/store instructions that trigger alignment faults. Signed-off-by: Ard Biesheuvel --- crypto/algapi.c | 7 +++---- include/crypto/algapi.h | 11 +++++++++-- 2 files changed, 12 insertions(+), 6 deletions(-) -- 2.11.0 diff --git a/crypto/algapi.c b/crypto/algapi.c index 2545c5f89c4c..52ce3c5a0499 100644 --- a/crypto/algapi.c +++ b/crypto/algapi.c @@ -988,11 +988,10 @@ void crypto_inc(u8 *a, unsigned int size) __be32 *b = (__be32 *)(a + size); u32 c; - if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) || - IS_ALIGNED((unsigned long)b, __alignof__(*b))) + if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)) for (; size >= 4; size -= 4) { - c = be32_to_cpu(*--b) + 1; - *b = cpu_to_be32(c); + c = get_unaligned_be32(--b) + 1; + put_unaligned_be32(c, b); if (likely(c)) return; } diff --git a/include/crypto/algapi.h b/include/crypto/algapi.h index 4a5ad10e75f0..86267c232f34 100644 --- a/include/crypto/algapi.h +++ b/include/crypto/algapi.h @@ -17,6 +17,8 @@ #include #include +#include + /* * Maximum values for blocksize and alignmask, used to allocate * static buffers that are big enough for any combination of @@ -212,7 +214,9 @@ static inline void crypto_xor(u8 *dst, const u8 *src, unsigned int size) unsigned long *s = (unsigned long *)src; while (size > 0) { - *d++ ^= *s++; + put_unaligned(get_unaligned(d) ^ get_unaligned(s), d); + d++; + s++; size -= sizeof(unsigned long); } } else { @@ -231,7 +235,10 @@ static inline void crypto_xor_cpy(u8 *dst, const u8 *src1, const u8 *src2, unsigned long *s2 = (unsigned long *)src2; while (size > 0) { - *d++ = *s1++ ^ *s2++; + put_unaligned(get_unaligned(s1) ^ get_unaligned(s2), d); + d++; + s1++; + s2++; size -= sizeof(unsigned long); } } else { From patchwork Mon Oct 8 21:15:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 148447 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp4125337lji; Mon, 8 Oct 2018 14:16:14 -0700 (PDT) X-Google-Smtp-Source: ACcGV6340LpgB+0hL/eHeJa9cXGHR4RXI9w90Vba1wVQBSMXJFZhBN0ov4+zOfdG+sPLHUt+ant5 X-Received: by 2002:a17:902:bc8c:: with SMTP id bb12-v6mr25033947plb.275.1539033374712; Mon, 08 Oct 2018 14:16:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539033374; cv=none; d=google.com; s=arc-20160816; b=MHrplzdzSF1W+glgbl4NonNgepoTSMs6TOqukmpRpCpEbl5Z30KDOPUDc24vl1YZM5 N1bamMxOxeJd5vxy2yxQHy9mbRKNYDQCJ2fr6lV2D3Fd7UfNqQXj9yiU5fyqa3TwLRaa pRHFs/gXU2CGfRNUSsOOnLFjfoRe/z0nB6cJjjRu6TbH3vii2c1wNT4G+9KwP/UxnEbm ko+DqqpPbouEUV91uAR30tvWwYlkWD++5Uvdirj0imZhlRm+4aTT0pKuAi/DWvC4kWuU nAKJcodsrWVuhdkWgkEL6wIKQrwSTLlspKwUbn2vCGdxHtZkCWpbY3WI0bX1g9uOC9Lo r/qw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=gav2KcErAu5oEWztCs050y15OhvmwcN8SK23NHcd/0Q=; b=DxtoaNqXoGwQNLE62OQXb64iyObmQRCWiVhwzRerJkEj6G6aaWtjA4/zgD2vONT1l3 5fqza/C+mA66R4KT2lJiqNszN7/xCwVsbYvb3k9rdyTiTzeVfHFQSzrLyNbbQEs4Mo1r MWAFbY1YYM/GcWchS/RGDwFxuJov60HIFDhSyq8lzLthlh/BnguDfq1U+YFzpnXL7Q6l bLM6e55kfeKK6y2H4ukWP0fZ89h0/bhmexx8W0xmkI5fJDrh97SucEbFBgvXrUvqIf3h JVY00aotwVGQc8FmTCGXPArNlsOmzwDGYJfzx4a66meO1Mu4S3Q7ataKNjNJ22d0yUDH Bciw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IklIDQ37; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 68-v6si18572575pld.314.2018.10.08.14.16.14; Mon, 08 Oct 2018 14:16:14 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IklIDQ37; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726523AbeJIE3x (ORCPT + 2 others); Tue, 9 Oct 2018 00:29:53 -0400 Received: from mail-wm1-f67.google.com ([209.85.128.67]:56098 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726529AbeJIE3w (ORCPT ); Tue, 9 Oct 2018 00:29:52 -0400 Received: by mail-wm1-f67.google.com with SMTP id 206-v6so9413133wmb.5 for ; Mon, 08 Oct 2018 14:16:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=gav2KcErAu5oEWztCs050y15OhvmwcN8SK23NHcd/0Q=; b=IklIDQ37fH9qB/852NY84rDYYN6XPnb3cTOS8BbxQpapijNSMREM3JGnd7xCAl9PcM KDwmXpYeKnR6kQ8ZN6GcHD3uCNU65Mqb6yHf4/M1COcfjFLloBBd1WI/9z/RDav8pamO CzjSg2jvzOBRSH0qDqh49OsAmnZEguKWzRlHQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=gav2KcErAu5oEWztCs050y15OhvmwcN8SK23NHcd/0Q=; b=kg8PuNy9JDjeabGQUkgUuAC3g1KUlpMh7w/KgiuQVp39ncaYCdgZk7SEVrlu2ujXZf ja6H/8Eyomvi/yj+TFdp+ZGmlHjuEbxbbFx2lL9IN0zu+465eIgs0BS4/euLFOtGMkP/ XIN/za74g3OvNwOHFGYIkQ4y5my2+BrhAXh9wXqZur9LO2ohtNrt3SS92GlY0AEPD+9G 8/qcyEgHqOoYp5dJQPKnuA8lDW3aqFCeBEk2tWtZd/ik/Zf9G5w6Rznzxb0ayC9+cMev 6X1GoEL5HQ7mX0urQe/hpjJ252D9E0YzWTxqtGheX57y4siFPBQKb4YUA/vIBidOeAMX 9T4A== X-Gm-Message-State: ABuFfogVcsLdijs3umktKgxDYfNvXZVRg/UTwLgpfQ5GA07YgbrhsInm 38NRlZCfspKudtpmsMJxdwGDVRTuABo= X-Received: by 2002:a1c:4887:: with SMTP id v129-v6mr15717607wma.139.1539033369088; Mon, 08 Oct 2018 14:16:09 -0700 (PDT) Received: from localhost.localdomain ([2a01:cb1d:112:6f00:8084:9715:d038:c67d]) by smtp.gmail.com with ESMTPSA id s24-v6sm7563308wmc.7.2018.10.08.14.16.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 08 Oct 2018 14:16:08 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, arnd@arndb.de, jason@zx2c4.com, ebiggers@google.com, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel Subject: [PATCH 3/3] crypto: siphash - drop _aligned variants Date: Mon, 8 Oct 2018 23:15:54 +0200 Message-Id: <20181008211554.5355-4-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20181008211554.5355-1-ard.biesheuvel@linaro.org> References: <20181008211554.5355-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org On ARM v6 and later, we define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS because the ordinary load/store instructions (ldr, ldrh, ldrb) can tolerate any misalignment of the memory address. However, load/store double and load/store multiple instructions (ldrd, ldm) may still only be used on memory addresses that are 32-bit aligned, and so we have to use the CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS macro with care, or we may end up with a severe performance hit due to alignment traps that require fixups by the kernel. Fortunately, the get_unaligned() accessors do the right thing: when building for ARMv6 or later, the compiler will emit unaligned accesses using the ordinary load/store instructions (but avoid the ones that require 32-bit alignment). When building for older ARM, those accessors will emit the appropriate sequence of ldrb/mov/orr instructions. And on architectures that can truly tolerate any kind of misalignment, the get_unaligned() accessors resolve to the leXX_to_cpup accessors that operate on aligned addresses. Since the compiler will in fact emit ldrd or ldm instructions when building this code for ARM v6 or later, the solution is to use the unaligned accessors on the aligned code paths. Given the above, this either produces the same code, or better in the ARMv6+ case. However, since that removes the only difference between the aligned and unaligned variants, we can drop the aligned variant entirely. Signed-off-by: Ard Biesheuvel --- include/linux/siphash.h | 106 +++++++++----------- lib/siphash.c | 103 ++----------------- 2 files changed, 54 insertions(+), 155 deletions(-) -- 2.11.0 diff --git a/include/linux/siphash.h b/include/linux/siphash.h index fa7a6b9cedbf..ef3c36b0ae0f 100644 --- a/include/linux/siphash.h +++ b/include/linux/siphash.h @@ -15,16 +15,14 @@ #include #include +#include #define SIPHASH_ALIGNMENT __alignof__(u64) typedef struct { u64 key[2]; } siphash_key_t; -u64 __siphash_aligned(const void *data, size_t len, const siphash_key_t *key); -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS -u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key); -#endif +u64 __siphash(const void *data, size_t len, const siphash_key_t *key); u64 siphash_1u64(const u64 a, const siphash_key_t *key); u64 siphash_2u64(const u64 a, const u64 b, const siphash_key_t *key); @@ -48,26 +46,6 @@ static inline u64 siphash_4u32(const u32 a, const u32 b, const u32 c, } -static inline u64 ___siphash_aligned(const __le64 *data, size_t len, - const siphash_key_t *key) -{ - if (__builtin_constant_p(len) && len == 4) - return siphash_1u32(le32_to_cpup((const __le32 *)data), key); - if (__builtin_constant_p(len) && len == 8) - return siphash_1u64(le64_to_cpu(data[0]), key); - if (__builtin_constant_p(len) && len == 16) - return siphash_2u64(le64_to_cpu(data[0]), le64_to_cpu(data[1]), - key); - if (__builtin_constant_p(len) && len == 24) - return siphash_3u64(le64_to_cpu(data[0]), le64_to_cpu(data[1]), - le64_to_cpu(data[2]), key); - if (__builtin_constant_p(len) && len == 32) - return siphash_4u64(le64_to_cpu(data[0]), le64_to_cpu(data[1]), - le64_to_cpu(data[2]), le64_to_cpu(data[3]), - key); - return __siphash_aligned(data, len, key); -} - /** * siphash - compute 64-bit siphash PRF value * @data: buffer to hash @@ -77,11 +55,30 @@ static inline u64 ___siphash_aligned(const __le64 *data, size_t len, static inline u64 siphash(const void *data, size_t len, const siphash_key_t *key) { -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS - if (!IS_ALIGNED((unsigned long)data, SIPHASH_ALIGNMENT)) - return __siphash_unaligned(data, len, key); -#endif - return ___siphash_aligned(data, len, key); + if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)) { + if (__builtin_constant_p(len) && len == 4) + return siphash_1u32(get_unaligned_le32(data), + key); + if (__builtin_constant_p(len) && len == 8) + return siphash_1u64(get_unaligned_le64(data), + key); + if (__builtin_constant_p(len) && len == 16) + return siphash_2u64(get_unaligned_le64(data), + get_unaligned_le64(data + 8), + key); + if (__builtin_constant_p(len) && len == 24) + return siphash_3u64(get_unaligned_le64(data), + get_unaligned_le64(data + 8), + get_unaligned_le64(data + 16), + key); + if (__builtin_constant_p(len) && len == 32) + return siphash_4u64(get_unaligned_le64(data), + get_unaligned_le64(data + 8), + get_unaligned_le64(data + 16), + get_unaligned_le64(data + 24), + key); + } + return __siphash(data, len, key); } #define HSIPHASH_ALIGNMENT __alignof__(unsigned long) @@ -89,12 +86,7 @@ typedef struct { unsigned long key[2]; } hsiphash_key_t; -u32 __hsiphash_aligned(const void *data, size_t len, - const hsiphash_key_t *key); -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS -u32 __hsiphash_unaligned(const void *data, size_t len, - const hsiphash_key_t *key); -#endif +u32 __hsiphash(const void *data, size_t len, const hsiphash_key_t *key); u32 hsiphash_1u32(const u32 a, const hsiphash_key_t *key); u32 hsiphash_2u32(const u32 a, const u32 b, const hsiphash_key_t *key); @@ -103,24 +95,6 @@ u32 hsiphash_3u32(const u32 a, const u32 b, const u32 c, u32 hsiphash_4u32(const u32 a, const u32 b, const u32 c, const u32 d, const hsiphash_key_t *key); -static inline u32 ___hsiphash_aligned(const __le32 *data, size_t len, - const hsiphash_key_t *key) -{ - if (__builtin_constant_p(len) && len == 4) - return hsiphash_1u32(le32_to_cpu(data[0]), key); - if (__builtin_constant_p(len) && len == 8) - return hsiphash_2u32(le32_to_cpu(data[0]), le32_to_cpu(data[1]), - key); - if (__builtin_constant_p(len) && len == 12) - return hsiphash_3u32(le32_to_cpu(data[0]), le32_to_cpu(data[1]), - le32_to_cpu(data[2]), key); - if (__builtin_constant_p(len) && len == 16) - return hsiphash_4u32(le32_to_cpu(data[0]), le32_to_cpu(data[1]), - le32_to_cpu(data[2]), le32_to_cpu(data[3]), - key); - return __hsiphash_aligned(data, len, key); -} - /** * hsiphash - compute 32-bit hsiphash PRF value * @data: buffer to hash @@ -130,11 +104,27 @@ static inline u32 ___hsiphash_aligned(const __le32 *data, size_t len, static inline u32 hsiphash(const void *data, size_t len, const hsiphash_key_t *key) { -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS - if (!IS_ALIGNED((unsigned long)data, HSIPHASH_ALIGNMENT)) - return __hsiphash_unaligned(data, len, key); -#endif - return ___hsiphash_aligned(data, len, key); + if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)) { + if (__builtin_constant_p(len) && len == 4) + return hsiphash_1u32(get_unaligned_le32(data), + key); + if (__builtin_constant_p(len) && len == 8) + return hsiphash_2u32(get_unaligned_le32(data), + get_unaligned_le32(data + 4), + key); + if (__builtin_constant_p(len) && len == 12) + return hsiphash_3u32(get_unaligned_le32(data), + get_unaligned_le32(data + 4), + get_unaligned_le32(data + 8), + key); + if (__builtin_constant_p(len) && len == 16) + return hsiphash_4u32(get_unaligned_le32(data), + get_unaligned_le32(data + 4), + get_unaligned_le32(data + 8), + get_unaligned_le32(data + 12), + key); + } + return __hsiphash(data, len, key); } #endif /* _LINUX_SIPHASH_H */ diff --git a/lib/siphash.c b/lib/siphash.c index 3ae58b4edad6..3b2ba1a10ad9 100644 --- a/lib/siphash.c +++ b/lib/siphash.c @@ -49,40 +49,7 @@ SIPROUND; \ return (v0 ^ v1) ^ (v2 ^ v3); -u64 __siphash_aligned(const void *data, size_t len, const siphash_key_t *key) -{ - const u8 *end = data + len - (len % sizeof(u64)); - const u8 left = len & (sizeof(u64) - 1); - u64 m; - PREAMBLE(len) - for (; data != end; data += sizeof(u64)) { - m = le64_to_cpup(data); - v3 ^= m; - SIPROUND; - SIPROUND; - v0 ^= m; - } -#if defined(CONFIG_DCACHE_WORD_ACCESS) && BITS_PER_LONG == 64 - if (left) - b |= le64_to_cpu((__force __le64)(load_unaligned_zeropad(data) & - bytemask_from_count(left))); -#else - switch (left) { - case 7: b |= ((u64)end[6]) << 48; - case 6: b |= ((u64)end[5]) << 40; - case 5: b |= ((u64)end[4]) << 32; - case 4: b |= le32_to_cpup(data); break; - case 3: b |= ((u64)end[2]) << 16; - case 2: b |= le16_to_cpup(data); break; - case 1: b |= end[0]; - } -#endif - POSTAMBLE -} -EXPORT_SYMBOL(__siphash_aligned); - -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS -u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key) +u64 __siphash(const void *data, size_t len, const siphash_key_t *key) { const u8 *end = data + len - (len % sizeof(u64)); const u8 left = len & (sizeof(u64) - 1); @@ -112,8 +79,7 @@ u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key) #endif POSTAMBLE } -EXPORT_SYMBOL(__siphash_unaligned); -#endif +EXPORT_SYMBOL(__siphash); /** * siphash_1u64 - compute 64-bit siphash PRF value of a u64 @@ -250,39 +216,7 @@ EXPORT_SYMBOL(siphash_3u32); HSIPROUND; \ return (v0 ^ v1) ^ (v2 ^ v3); -u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key) -{ - const u8 *end = data + len - (len % sizeof(u64)); - const u8 left = len & (sizeof(u64) - 1); - u64 m; - HPREAMBLE(len) - for (; data != end; data += sizeof(u64)) { - m = le64_to_cpup(data); - v3 ^= m; - HSIPROUND; - v0 ^= m; - } -#if defined(CONFIG_DCACHE_WORD_ACCESS) && BITS_PER_LONG == 64 - if (left) - b |= le64_to_cpu((__force __le64)(load_unaligned_zeropad(data) & - bytemask_from_count(left))); -#else - switch (left) { - case 7: b |= ((u64)end[6]) << 48; - case 6: b |= ((u64)end[5]) << 40; - case 5: b |= ((u64)end[4]) << 32; - case 4: b |= le32_to_cpup(data); break; - case 3: b |= ((u64)end[2]) << 16; - case 2: b |= le16_to_cpup(data); break; - case 1: b |= end[0]; - } -#endif - HPOSTAMBLE -} -EXPORT_SYMBOL(__hsiphash_aligned); - -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS -u32 __hsiphash_unaligned(const void *data, size_t len, +u32 __hsiphash(const void *data, size_t len, const hsiphash_key_t *key) { const u8 *end = data + len - (len % sizeof(u64)); @@ -312,8 +246,7 @@ u32 __hsiphash_unaligned(const void *data, size_t len, #endif HPOSTAMBLE } -EXPORT_SYMBOL(__hsiphash_unaligned); -#endif +EXPORT_SYMBOL(__hsiphash); /** * hsiphash_1u32 - compute 64-bit hsiphash PRF value of a u32 @@ -418,30 +351,7 @@ EXPORT_SYMBOL(hsiphash_4u32); HSIPROUND; \ return v1 ^ v3; -u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key) -{ - const u8 *end = data + len - (len % sizeof(u32)); - const u8 left = len & (sizeof(u32) - 1); - u32 m; - HPREAMBLE(len) - for (; data != end; data += sizeof(u32)) { - m = le32_to_cpup(data); - v3 ^= m; - HSIPROUND; - v0 ^= m; - } - switch (left) { - case 3: b |= ((u32)end[2]) << 16; - case 2: b |= le16_to_cpup(data); break; - case 1: b |= end[0]; - } - HPOSTAMBLE -} -EXPORT_SYMBOL(__hsiphash_aligned); - -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS -u32 __hsiphash_unaligned(const void *data, size_t len, - const hsiphash_key_t *key) +u32 __hsiphash(const void *data, size_t len, const hsiphash_key_t *key) { const u8 *end = data + len - (len % sizeof(u32)); const u8 left = len & (sizeof(u32) - 1); @@ -460,8 +370,7 @@ u32 __hsiphash_unaligned(const void *data, size_t len, } HPOSTAMBLE } -EXPORT_SYMBOL(__hsiphash_unaligned); -#endif +EXPORT_SYMBOL(__hsiphash); /** * hsiphash_1u32 - compute 32-bit hsiphash PRF value of a u32