| Message ID | 20241018075347.2821102-5-ardb+git@google.com |
|---|---|
| Headers | show |
| Series | arm64: Speed up CRC-32 using PMULL instructions | expand |
On Fri, 18 Oct 2024 09:53:48 +0200, Ard Biesheuvel wrote: > The CRC-32 code is library code, and is not part of the crypto > subsystem. This means that callers may not generally be aware of the > kind of implementation that backs it, and so we've refrained from using > FP/SIMD code in the past, as it disables preemption, and this may incur > scheduling latencies that the caller did not anticipate. > > This was solved a while ago, and on arm64, kernel mode FP/SIMD no longer > disables preemption. > > [...] Applied to arm64 (for-next/crc32), thanks! [1/3] arm64/lib: Handle CRC-32 alternative in C code https://git.kernel.org/arm64/c/fc7454107d1b [2/3] arm64/crc32: Reorganize bit/byte ordering macros https://git.kernel.org/arm64/c/b98b23e19492 [3/3] arm64/crc32: Implement 4-way interleave using PMULL https://git.kernel.org/arm64/c/a6478d69cf56
From: Ard Biesheuvel <ardb@kernel.org> The CRC-32 code is library code, and is not part of the crypto subsystem. This means that callers may not generally be aware of the kind of implementation that backs it, and so we've refrained from using FP/SIMD code in the past, as it disables preemption, and this may incur scheduling latencies that the caller did not anticipate. This was solved a while ago, and on arm64, kernel mode FP/SIMD no longer disables preemption. This means we can happily use PMULL instructions in the CRC-32 library code, which permits an optimization to be implemented that results in a speedup of 2 - 2.8x for inputs >1k in size (on Apple M2) Patch #1 implements some prepwork to handle the scalar CRC-32 alternatives patching in C code. Changes since v3: - fix broken crc32be version - add patch to tidy up existing code for reuse - add 4-way code to existing .S file Changes since v2: - drop alternatives.h #include (#1) - drop unneeded branch (#2) - fix comment max -> min (#2) - add Eric's Rb Changes since v1: - rename crc32-pmull.S to crc32-4way.S and avoid pmull in the function names to avoid confusion about the nature of the implementation; - polish the asm a bit, and add some comments - don't return via the scalar code if len dropped to 0 after calling the 4-way code. Cc: Eric Biggers <ebiggers@kernel.org> Cc: Kees Cook <kees@kernel.org> Ard Biesheuvel (3): arm64/lib: Handle CRC-32 alternative in C code arm64/crc32: Reorganize bit/byte ordering macros arm64/crc32: Implement 4-way interleave using PMULL arch/arm64/lib/Makefile | 2 +- arch/arm64/lib/crc32-glue.c | 82 +++++ arch/arm64/lib/crc32.S | 344 ++++++++++++++++---- 3 files changed, 356 insertions(+), 72 deletions(-) create mode 100644 arch/arm64/lib/crc32-glue.c