From patchwork Mon Oct 8 21:15:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 148447 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp4125337lji; Mon, 8 Oct 2018 14:16:14 -0700 (PDT) X-Google-Smtp-Source: ACcGV6340LpgB+0hL/eHeJa9cXGHR4RXI9w90Vba1wVQBSMXJFZhBN0ov4+zOfdG+sPLHUt+ant5 X-Received: by 2002:a17:902:bc8c:: with SMTP id bb12-v6mr25033947plb.275.1539033374712; Mon, 08 Oct 2018 14:16:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539033374; cv=none; d=google.com; s=arc-20160816; b=MHrplzdzSF1W+glgbl4NonNgepoTSMs6TOqukmpRpCpEbl5Z30KDOPUDc24vl1YZM5 N1bamMxOxeJd5vxy2yxQHy9mbRKNYDQCJ2fr6lV2D3Fd7UfNqQXj9yiU5fyqa3TwLRaa pRHFs/gXU2CGfRNUSsOOnLFjfoRe/z0nB6cJjjRu6TbH3vii2c1wNT4G+9KwP/UxnEbm ko+DqqpPbouEUV91uAR30tvWwYlkWD++5Uvdirj0imZhlRm+4aTT0pKuAi/DWvC4kWuU nAKJcodsrWVuhdkWgkEL6wIKQrwSTLlspKwUbn2vCGdxHtZkCWpbY3WI0bX1g9uOC9Lo r/qw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=gav2KcErAu5oEWztCs050y15OhvmwcN8SK23NHcd/0Q=; b=DxtoaNqXoGwQNLE62OQXb64iyObmQRCWiVhwzRerJkEj6G6aaWtjA4/zgD2vONT1l3 5fqza/C+mA66R4KT2lJiqNszN7/xCwVsbYvb3k9rdyTiTzeVfHFQSzrLyNbbQEs4Mo1r MWAFbY1YYM/GcWchS/RGDwFxuJov60HIFDhSyq8lzLthlh/BnguDfq1U+YFzpnXL7Q6l bLM6e55kfeKK6y2H4ukWP0fZ89h0/bhmexx8W0xmkI5fJDrh97SucEbFBgvXrUvqIf3h JVY00aotwVGQc8FmTCGXPArNlsOmzwDGYJfzx4a66meO1Mu4S3Q7ataKNjNJ22d0yUDH Bciw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IklIDQ37; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 68-v6si18572575pld.314.2018.10.08.14.16.14; Mon, 08 Oct 2018 14:16:14 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IklIDQ37; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726523AbeJIE3x (ORCPT + 2 others); Tue, 9 Oct 2018 00:29:53 -0400 Received: from mail-wm1-f67.google.com ([209.85.128.67]:56098 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726529AbeJIE3w (ORCPT ); Tue, 9 Oct 2018 00:29:52 -0400 Received: by mail-wm1-f67.google.com with SMTP id 206-v6so9413133wmb.5 for ; Mon, 08 Oct 2018 14:16:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=gav2KcErAu5oEWztCs050y15OhvmwcN8SK23NHcd/0Q=; b=IklIDQ37fH9qB/852NY84rDYYN6XPnb3cTOS8BbxQpapijNSMREM3JGnd7xCAl9PcM KDwmXpYeKnR6kQ8ZN6GcHD3uCNU65Mqb6yHf4/M1COcfjFLloBBd1WI/9z/RDav8pamO CzjSg2jvzOBRSH0qDqh49OsAmnZEguKWzRlHQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=gav2KcErAu5oEWztCs050y15OhvmwcN8SK23NHcd/0Q=; b=kg8PuNy9JDjeabGQUkgUuAC3g1KUlpMh7w/KgiuQVp39ncaYCdgZk7SEVrlu2ujXZf ja6H/8Eyomvi/yj+TFdp+ZGmlHjuEbxbbFx2lL9IN0zu+465eIgs0BS4/euLFOtGMkP/ XIN/za74g3OvNwOHFGYIkQ4y5my2+BrhAXh9wXqZur9LO2ohtNrt3SS92GlY0AEPD+9G 8/qcyEgHqOoYp5dJQPKnuA8lDW3aqFCeBEk2tWtZd/ik/Zf9G5w6Rznzxb0ayC9+cMev 6X1GoEL5HQ7mX0urQe/hpjJ252D9E0YzWTxqtGheX57y4siFPBQKb4YUA/vIBidOeAMX 9T4A== X-Gm-Message-State: ABuFfogVcsLdijs3umktKgxDYfNvXZVRg/UTwLgpfQ5GA07YgbrhsInm 38NRlZCfspKudtpmsMJxdwGDVRTuABo= X-Received: by 2002:a1c:4887:: with SMTP id v129-v6mr15717607wma.139.1539033369088; Mon, 08 Oct 2018 14:16:09 -0700 (PDT) Received: from localhost.localdomain ([2a01:cb1d:112:6f00:8084:9715:d038:c67d]) by smtp.gmail.com with ESMTPSA id s24-v6sm7563308wmc.7.2018.10.08.14.16.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 08 Oct 2018 14:16:08 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, arnd@arndb.de, jason@zx2c4.com, ebiggers@google.com, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel Subject: [PATCH 3/3] crypto: siphash - drop _aligned variants Date: Mon, 8 Oct 2018 23:15:54 +0200 Message-Id: <20181008211554.5355-4-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20181008211554.5355-1-ard.biesheuvel@linaro.org> References: <20181008211554.5355-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org On ARM v6 and later, we define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS because the ordinary load/store instructions (ldr, ldrh, ldrb) can tolerate any misalignment of the memory address. However, load/store double and load/store multiple instructions (ldrd, ldm) may still only be used on memory addresses that are 32-bit aligned, and so we have to use the CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS macro with care, or we may end up with a severe performance hit due to alignment traps that require fixups by the kernel. Fortunately, the get_unaligned() accessors do the right thing: when building for ARMv6 or later, the compiler will emit unaligned accesses using the ordinary load/store instructions (but avoid the ones that require 32-bit alignment). When building for older ARM, those accessors will emit the appropriate sequence of ldrb/mov/orr instructions. And on architectures that can truly tolerate any kind of misalignment, the get_unaligned() accessors resolve to the leXX_to_cpup accessors that operate on aligned addresses. Since the compiler will in fact emit ldrd or ldm instructions when building this code for ARM v6 or later, the solution is to use the unaligned accessors on the aligned code paths. Given the above, this either produces the same code, or better in the ARMv6+ case. However, since that removes the only difference between the aligned and unaligned variants, we can drop the aligned variant entirely. Signed-off-by: Ard Biesheuvel --- include/linux/siphash.h | 106 +++++++++----------- lib/siphash.c | 103 ++----------------- 2 files changed, 54 insertions(+), 155 deletions(-) -- 2.11.0 diff --git a/include/linux/siphash.h b/include/linux/siphash.h index fa7a6b9cedbf..ef3c36b0ae0f 100644 --- a/include/linux/siphash.h +++ b/include/linux/siphash.h @@ -15,16 +15,14 @@ #include #include +#include #define SIPHASH_ALIGNMENT __alignof__(u64) typedef struct { u64 key[2]; } siphash_key_t; -u64 __siphash_aligned(const void *data, size_t len, const siphash_key_t *key); -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS -u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key); -#endif +u64 __siphash(const void *data, size_t len, const siphash_key_t *key); u64 siphash_1u64(const u64 a, const siphash_key_t *key); u64 siphash_2u64(const u64 a, const u64 b, const siphash_key_t *key); @@ -48,26 +46,6 @@ static inline u64 siphash_4u32(const u32 a, const u32 b, const u32 c, } -static inline u64 ___siphash_aligned(const __le64 *data, size_t len, - const siphash_key_t *key) -{ - if (__builtin_constant_p(len) && len == 4) - return siphash_1u32(le32_to_cpup((const __le32 *)data), key); - if (__builtin_constant_p(len) && len == 8) - return siphash_1u64(le64_to_cpu(data[0]), key); - if (__builtin_constant_p(len) && len == 16) - return siphash_2u64(le64_to_cpu(data[0]), le64_to_cpu(data[1]), - key); - if (__builtin_constant_p(len) && len == 24) - return siphash_3u64(le64_to_cpu(data[0]), le64_to_cpu(data[1]), - le64_to_cpu(data[2]), key); - if (__builtin_constant_p(len) && len == 32) - return siphash_4u64(le64_to_cpu(data[0]), le64_to_cpu(data[1]), - le64_to_cpu(data[2]), le64_to_cpu(data[3]), - key); - return __siphash_aligned(data, len, key); -} - /** * siphash - compute 64-bit siphash PRF value * @data: buffer to hash @@ -77,11 +55,30 @@ static inline u64 ___siphash_aligned(const __le64 *data, size_t len, static inline u64 siphash(const void *data, size_t len, const siphash_key_t *key) { -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS - if (!IS_ALIGNED((unsigned long)data, SIPHASH_ALIGNMENT)) - return __siphash_unaligned(data, len, key); -#endif - return ___siphash_aligned(data, len, key); + if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)) { + if (__builtin_constant_p(len) && len == 4) + return siphash_1u32(get_unaligned_le32(data), + key); + if (__builtin_constant_p(len) && len == 8) + return siphash_1u64(get_unaligned_le64(data), + key); + if (__builtin_constant_p(len) && len == 16) + return siphash_2u64(get_unaligned_le64(data), + get_unaligned_le64(data + 8), + key); + if (__builtin_constant_p(len) && len == 24) + return siphash_3u64(get_unaligned_le64(data), + get_unaligned_le64(data + 8), + get_unaligned_le64(data + 16), + key); + if (__builtin_constant_p(len) && len == 32) + return siphash_4u64(get_unaligned_le64(data), + get_unaligned_le64(data + 8), + get_unaligned_le64(data + 16), + get_unaligned_le64(data + 24), + key); + } + return __siphash(data, len, key); } #define HSIPHASH_ALIGNMENT __alignof__(unsigned long) @@ -89,12 +86,7 @@ typedef struct { unsigned long key[2]; } hsiphash_key_t; -u32 __hsiphash_aligned(const void *data, size_t len, - const hsiphash_key_t *key); -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS -u32 __hsiphash_unaligned(const void *data, size_t len, - const hsiphash_key_t *key); -#endif +u32 __hsiphash(const void *data, size_t len, const hsiphash_key_t *key); u32 hsiphash_1u32(const u32 a, const hsiphash_key_t *key); u32 hsiphash_2u32(const u32 a, const u32 b, const hsiphash_key_t *key); @@ -103,24 +95,6 @@ u32 hsiphash_3u32(const u32 a, const u32 b, const u32 c, u32 hsiphash_4u32(const u32 a, const u32 b, const u32 c, const u32 d, const hsiphash_key_t *key); -static inline u32 ___hsiphash_aligned(const __le32 *data, size_t len, - const hsiphash_key_t *key) -{ - if (__builtin_constant_p(len) && len == 4) - return hsiphash_1u32(le32_to_cpu(data[0]), key); - if (__builtin_constant_p(len) && len == 8) - return hsiphash_2u32(le32_to_cpu(data[0]), le32_to_cpu(data[1]), - key); - if (__builtin_constant_p(len) && len == 12) - return hsiphash_3u32(le32_to_cpu(data[0]), le32_to_cpu(data[1]), - le32_to_cpu(data[2]), key); - if (__builtin_constant_p(len) && len == 16) - return hsiphash_4u32(le32_to_cpu(data[0]), le32_to_cpu(data[1]), - le32_to_cpu(data[2]), le32_to_cpu(data[3]), - key); - return __hsiphash_aligned(data, len, key); -} - /** * hsiphash - compute 32-bit hsiphash PRF value * @data: buffer to hash @@ -130,11 +104,27 @@ static inline u32 ___hsiphash_aligned(const __le32 *data, size_t len, static inline u32 hsiphash(const void *data, size_t len, const hsiphash_key_t *key) { -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS - if (!IS_ALIGNED((unsigned long)data, HSIPHASH_ALIGNMENT)) - return __hsiphash_unaligned(data, len, key); -#endif - return ___hsiphash_aligned(data, len, key); + if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)) { + if (__builtin_constant_p(len) && len == 4) + return hsiphash_1u32(get_unaligned_le32(data), + key); + if (__builtin_constant_p(len) && len == 8) + return hsiphash_2u32(get_unaligned_le32(data), + get_unaligned_le32(data + 4), + key); + if (__builtin_constant_p(len) && len == 12) + return hsiphash_3u32(get_unaligned_le32(data), + get_unaligned_le32(data + 4), + get_unaligned_le32(data + 8), + key); + if (__builtin_constant_p(len) && len == 16) + return hsiphash_4u32(get_unaligned_le32(data), + get_unaligned_le32(data + 4), + get_unaligned_le32(data + 8), + get_unaligned_le32(data + 12), + key); + } + return __hsiphash(data, len, key); } #endif /* _LINUX_SIPHASH_H */ diff --git a/lib/siphash.c b/lib/siphash.c index 3ae58b4edad6..3b2ba1a10ad9 100644 --- a/lib/siphash.c +++ b/lib/siphash.c @@ -49,40 +49,7 @@ SIPROUND; \ return (v0 ^ v1) ^ (v2 ^ v3); -u64 __siphash_aligned(const void *data, size_t len, const siphash_key_t *key) -{ - const u8 *end = data + len - (len % sizeof(u64)); - const u8 left = len & (sizeof(u64) - 1); - u64 m; - PREAMBLE(len) - for (; data != end; data += sizeof(u64)) { - m = le64_to_cpup(data); - v3 ^= m; - SIPROUND; - SIPROUND; - v0 ^= m; - } -#if defined(CONFIG_DCACHE_WORD_ACCESS) && BITS_PER_LONG == 64 - if (left) - b |= le64_to_cpu((__force __le64)(load_unaligned_zeropad(data) & - bytemask_from_count(left))); -#else - switch (left) { - case 7: b |= ((u64)end[6]) << 48; - case 6: b |= ((u64)end[5]) << 40; - case 5: b |= ((u64)end[4]) << 32; - case 4: b |= le32_to_cpup(data); break; - case 3: b |= ((u64)end[2]) << 16; - case 2: b |= le16_to_cpup(data); break; - case 1: b |= end[0]; - } -#endif - POSTAMBLE -} -EXPORT_SYMBOL(__siphash_aligned); - -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS -u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key) +u64 __siphash(const void *data, size_t len, const siphash_key_t *key) { const u8 *end = data + len - (len % sizeof(u64)); const u8 left = len & (sizeof(u64) - 1); @@ -112,8 +79,7 @@ u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key) #endif POSTAMBLE } -EXPORT_SYMBOL(__siphash_unaligned); -#endif +EXPORT_SYMBOL(__siphash); /** * siphash_1u64 - compute 64-bit siphash PRF value of a u64 @@ -250,39 +216,7 @@ EXPORT_SYMBOL(siphash_3u32); HSIPROUND; \ return (v0 ^ v1) ^ (v2 ^ v3); -u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key) -{ - const u8 *end = data + len - (len % sizeof(u64)); - const u8 left = len & (sizeof(u64) - 1); - u64 m; - HPREAMBLE(len) - for (; data != end; data += sizeof(u64)) { - m = le64_to_cpup(data); - v3 ^= m; - HSIPROUND; - v0 ^= m; - } -#if defined(CONFIG_DCACHE_WORD_ACCESS) && BITS_PER_LONG == 64 - if (left) - b |= le64_to_cpu((__force __le64)(load_unaligned_zeropad(data) & - bytemask_from_count(left))); -#else - switch (left) { - case 7: b |= ((u64)end[6]) << 48; - case 6: b |= ((u64)end[5]) << 40; - case 5: b |= ((u64)end[4]) << 32; - case 4: b |= le32_to_cpup(data); break; - case 3: b |= ((u64)end[2]) << 16; - case 2: b |= le16_to_cpup(data); break; - case 1: b |= end[0]; - } -#endif - HPOSTAMBLE -} -EXPORT_SYMBOL(__hsiphash_aligned); - -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS -u32 __hsiphash_unaligned(const void *data, size_t len, +u32 __hsiphash(const void *data, size_t len, const hsiphash_key_t *key) { const u8 *end = data + len - (len % sizeof(u64)); @@ -312,8 +246,7 @@ u32 __hsiphash_unaligned(const void *data, size_t len, #endif HPOSTAMBLE } -EXPORT_SYMBOL(__hsiphash_unaligned); -#endif +EXPORT_SYMBOL(__hsiphash); /** * hsiphash_1u32 - compute 64-bit hsiphash PRF value of a u32 @@ -418,30 +351,7 @@ EXPORT_SYMBOL(hsiphash_4u32); HSIPROUND; \ return v1 ^ v3; -u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key) -{ - const u8 *end = data + len - (len % sizeof(u32)); - const u8 left = len & (sizeof(u32) - 1); - u32 m; - HPREAMBLE(len) - for (; data != end; data += sizeof(u32)) { - m = le32_to_cpup(data); - v3 ^= m; - HSIPROUND; - v0 ^= m; - } - switch (left) { - case 3: b |= ((u32)end[2]) << 16; - case 2: b |= le16_to_cpup(data); break; - case 1: b |= end[0]; - } - HPOSTAMBLE -} -EXPORT_SYMBOL(__hsiphash_aligned); - -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS -u32 __hsiphash_unaligned(const void *data, size_t len, - const hsiphash_key_t *key) +u32 __hsiphash(const void *data, size_t len, const hsiphash_key_t *key) { const u8 *end = data + len - (len % sizeof(u32)); const u8 left = len & (sizeof(u32) - 1); @@ -460,8 +370,7 @@ u32 __hsiphash_unaligned(const void *data, size_t len, } HPOSTAMBLE } -EXPORT_SYMBOL(__hsiphash_unaligned); -#endif +EXPORT_SYMBOL(__hsiphash); /** * hsiphash_1u32 - compute 32-bit hsiphash PRF value of a u32