From patchwork Mon Aug 27 11:02:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 145171 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp4048176ljw; Mon, 27 Aug 2018 04:03:09 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZZOB2bImX0s3H0/Hk3V2epZkqsJQFMsezmQvbifS1kOFsaYlDzv8LFzcmSfgKboiPOhK/K X-Received: by 2002:a63:de10:: with SMTP id f16-v6mr11758006pgg.97.1535367789677; Mon, 27 Aug 2018 04:03:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535367789; cv=none; d=google.com; s=arc-20160816; b=pW7tE54n3Q6E0KfC4RCuO6YHaCpoNu2g1qq9SiODnv274xd+XjYbPAfICZHkLjbONn QS8GwOcB4NJJ7GopisA9a6h9HCL9+pPms1vhdAVXdGNnYy1obvJ2whQlnpqncbgLk/7o /FkMw3l6116jlagcLhRhmDYpzqOf+dmxFOWAezjUBQ2kzSb/z+FcWqSFl4MsDSRVle8s g1QP3DkVca5NTMBZk1T5C+S1J2jnB4/jpIiSAOT7xWmeB/zW2ai67as78AZHwx4oUeSZ hfQBhh5c3dciYlG0P9cRpEee3CXhfaCqWPr4Fz2/mj3PnwUAfIpEOWymyTBt0Zv/Pj19 4+0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=RBWagJrS8izqpWfSj/uAIGFEJFmec204/M/NCMHphvQ=; b=jNrSqafCTHj93NHiSbmy5fVuRE3FMwuwaTOiSBboSeWXbN8GZarbvb/LxjFBKIQCad WM2wzR0Ef4P60sFTE8rZSvi8UVUbgj92IOAa0ZLDbEKzGHG1GzcEj1KWvyQYXauDxU8T +zfswPWAC2YHYrL/OLYz8KXS4BApU8+fDHyyNaMpp8+cD859+u8vYwNsmX+LbByOP+cQ /YAtACDsY74dXWsaA553vgi+KMuXtuIc6lReQXI4zQMMr7Qp8GCNHry4YpEjcnrdN1uR ImKVCUmquVMTF83O7iLJg4jt4N9CfoYxjqcXmFpn2gKm/JcVU70W1XykeGmMO4fK35Cv YcCw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=JoYpMQeD; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id c41-v6si13901454plj.509.2018.08.27.04.03.09; Mon, 27 Aug 2018 04:03:09 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=JoYpMQeD; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727306AbeH0OtR (ORCPT + 32 others); Mon, 27 Aug 2018 10:49:17 -0400 Received: from mail-ed1-f67.google.com ([209.85.208.67]:46508 "EHLO mail-ed1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726949AbeH0OtQ (ORCPT ); Mon, 27 Aug 2018 10:49:16 -0400 Received: by mail-ed1-f67.google.com with SMTP id k14-v6so3775493edr.13 for ; Mon, 27 Aug 2018 04:03:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RBWagJrS8izqpWfSj/uAIGFEJFmec204/M/NCMHphvQ=; b=JoYpMQeDGKZ1zPIb0BMTotGaKg0SeZ72J+k3jcjwarBgImQcQVqOO6lafaAInE26Jb K3luZE7h/FLn/GR8AopVaDIhhpcGxZUvQRhxg6UR/95nwjim2/Ll8c9PGqeQz54smymW 2+v2iqer31zdShwwuAKCTtkmD7B45t1uZowzg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RBWagJrS8izqpWfSj/uAIGFEJFmec204/M/NCMHphvQ=; b=Rj/U9jjWxqLJpHUOqH5ifLClFU/5srORysz4dphlwnfM5/XHowPl4/oyC8BMde0rW6 6ETQY/yD/C2LcGC345jH1G4glqeashjjdFLYneRpbenpGtQzs5dI6U/rcYuA2bDOb4ly VmBn31bFS1yATtX2lfKFsWSQ4OYHFEp+bb4jpdQEgbAvTPt5+Kilvrg2540LHSVawk26 UghQ+7hlAbd4L6zONgkRPKrhlmSWPR/mKObmg+9LokiO5kMKcwNURQW98QoLZ8DGhriK MouXuIY83Dtprq1kBMgHWLkkDGe/7DrQXjHMtB7e79VBqTjkxphNOTZfMs+nekuu01P3 19Bw== X-Gm-Message-State: APzg51AC177Uqldw0Jj09Z7sQQSL6yXo0PxLsmoIyaJs68w59wPp3hU4 Z8jSnA+r4hftVTAghNuwxYmoWQ== X-Received: by 2002:a50:8386:: with SMTP id 6-v6mr16195527edi.170.1535367783703; Mon, 27 Aug 2018 04:03:03 -0700 (PDT) Received: from rev02.home ([2a02:a212:9283:9800:24b9:e2d6:9acc:50dd]) by smtp.gmail.com with ESMTPSA id r2-v6sm3114344eda.89.2018.08.27.04.03.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 27 Aug 2018 04:03:02 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux-crypto@vger.kernel.org Cc: will.deacon@arm.com, catalin.marinas@arm.com, herbert@gondor.apana.org.au, ebiggers@google.com, suzuki.poulose@arm.com, linux-kernel@vger.kernel.org, Ard Biesheuvel Subject: [PATCH 3/4] arm64/lib: add accelerated crc32 routines Date: Mon, 27 Aug 2018 13:02:44 +0200 Message-Id: <20180827110245.14812-4-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180827110245.14812-1-ard.biesheuvel@linaro.org> References: <20180827110245.14812-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Unlike crc32c(), which is wired up to the crypto API internally so the optimal driver is selected based on the platform's capabilities, crc32_le() is implemented as a library function using a slice-by-8 table based C implementation. Even though few of the call sites may be bottlenecks, calling a time variant implementation with a non-negligible D-cache footprint is a bit of a waste, given that ARMv8.1 and up mandates support for the CRC32 instructions that were optional in ARMv8.0, but are already widely available, even on the Cortex-A53 based Raspberry Pi. So implement routines that use these instructions if available, and fall back to the existing generic routines otherwise. The selection is based on alternatives patching. Note that this unconditionally selects CONFIG_CRC32 as a builtin. Since CRC32 is relied upon by core functionality such as CONFIG_OF_FLATTREE, this just codifies the status quo. Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 1 + arch/arm64/lib/Makefile | 2 + arch/arm64/lib/crc32.S | 60 ++++++++++++++++++++ 3 files changed, 63 insertions(+) -- 2.18.0 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 29e75b47becd..0625355f12fa 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -75,6 +75,7 @@ config ARM64 select CLONE_BACKWARDS select COMMON_CLK select CPU_PM if (SUSPEND || CPU_IDLE) + select CRC32 select DCACHE_WORD_ACCESS select DMA_DIRECT_OPS select EDAC_SUPPORT diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile index 68755fd70dcf..f28f91fd96a2 100644 --- a/arch/arm64/lib/Makefile +++ b/arch/arm64/lib/Makefile @@ -25,3 +25,5 @@ KCOV_INSTRUMENT_atomic_ll_sc.o := n UBSAN_SANITIZE_atomic_ll_sc.o := n lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o + +obj-$(CONFIG_CRC32) += crc32.o diff --git a/arch/arm64/lib/crc32.S b/arch/arm64/lib/crc32.S new file mode 100644 index 000000000000..5bc1e85b4e1c --- /dev/null +++ b/arch/arm64/lib/crc32.S @@ -0,0 +1,60 @@ +/* + * Accelerated CRC32(C) using AArch64 CRC instructions + * + * Copyright (C) 2016 - 2018 Linaro Ltd + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include +#include + + .cpu generic+crc + + .macro __crc32, c +0: subs x2, x2, #16 + b.mi 8f + ldp x3, x4, [x1], #16 +CPU_BE( rev x3, x3 ) +CPU_BE( rev x4, x4 ) + crc32\c\()x w0, w0, x3 + crc32\c\()x w0, w0, x4 + b.ne 0b + ret + +8: tbz x2, #3, 4f + ldr x3, [x1], #8 +CPU_BE( rev x3, x3 ) + crc32\c\()x w0, w0, x3 +4: tbz x2, #2, 2f + ldr w3, [x1], #4 +CPU_BE( rev w3, w3 ) + crc32\c\()w w0, w0, w3 +2: tbz x2, #1, 1f + ldrh w3, [x1], #2 +CPU_BE( rev16 w3, w3 ) + crc32\c\()h w0, w0, w3 +1: tbz x2, #0, 0f + ldrb w3, [x1] + crc32\c\()b w0, w0, w3 +0: ret + .endm + + .align 5 +ENTRY(crc32_le) +alternative_if_not ARM64_HAS_CRC32 + b crc32_le_base +alternative_else_nop_endif + __crc32 +ENDPROC(crc32_le) + + .align 5 +ENTRY(__crc32c_le) +alternative_if_not ARM64_HAS_CRC32 + b __crc32c_le_base +alternative_else_nop_endif + __crc32 c +ENDPROC(__crc32c_le)