From patchwork Thu Jun 6 15:03:04 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 17652 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-fa0-f71.google.com (mail-fa0-f71.google.com [209.85.161.71]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 2638620F47 for ; Thu, 6 Jun 2013 15:03:39 +0000 (UTC) Received: by mail-fa0-f71.google.com with SMTP id s1sf3129063fas.6 for ; Thu, 06 Jun 2013 08:03:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-beenthere:x-forwarded-to:x-forwarded-for :delivered-to:from:to:cc:subject:date:message-id:x-mailer :in-reply-to:references:x-gm-message-state:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-google-group-id:list-post:list-help:list-archive:list-unsubscribe; bh=iBFhMNMPZf/89612P89c2+AK1kTlqA8EsVuLgcxGk2I=; b=lPGag/9BOTktS+UQo5Brp2WAf1basqjwy5pYQFDHKlUIogPakBB70BMpF/hUSWbm49 nk1vvfHFjBxNzpnGdIrGZTCaKAcvoyVg1J5sQW8UxP3dyunTXViQn51jtF0aPngUMo6O WNczOnWMo0cP38ag2hkNP/xZSxb0ORytpxAWGQ17zItYMpeoInI/MnrRVGz0jITJADkB 5CtU5nVCB7TYQeudspG2yXGJWG4GPGGc6vk6OKYkaCuqe+pgPT/AUTOBVYmziJsP0ibD 2xUm4QNEBP5BVcOdp8MctachDWVk1JLYDG2PRw/qCW0CQITwDftobcm8qvhKk8DIPWDl Qo7w== X-Received: by 10.180.206.107 with SMTP id ln11mr5078493wic.7.1370531017793; Thu, 06 Jun 2013 08:03:37 -0700 (PDT) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.180.108.103 with SMTP id hj7ls2014600wib.22.canary; Thu, 06 Jun 2013 08:03:37 -0700 (PDT) X-Received: by 10.194.119.195 with SMTP id kw3mr1269219wjb.64.1370531017470; Thu, 06 Jun 2013 08:03:37 -0700 (PDT) Received: from mail-ve0-x22a.google.com (mail-ve0-x22a.google.com [2607:f8b0:400c:c01::22a]) by mx.google.com with ESMTPS id yw5si28502577wjc.109.2013.06.06.08.03.37 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 06 Jun 2013 08:03:37 -0700 (PDT) Received-SPF: neutral (google.com: 2607:f8b0:400c:c01::22a is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=2607:f8b0:400c:c01::22a; Received: by mail-ve0-f170.google.com with SMTP id 14so2288763vea.1 for ; Thu, 06 Jun 2013 08:03:36 -0700 (PDT) X-Received: by 10.52.36.115 with SMTP id p19mr18843872vdj.8.1370531016244; Thu, 06 Jun 2013 08:03:36 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patches@linaro.org Received: by 10.221.10.206 with SMTP id pb14csp70247vcb; Thu, 6 Jun 2013 08:03:35 -0700 (PDT) X-Received: by 10.180.107.104 with SMTP id hb8mr10359934wib.26.1370531015221; Thu, 06 Jun 2013 08:03:35 -0700 (PDT) Received: from mail-wi0-x232.google.com (mail-wi0-x232.google.com [2a00:1450:400c:c05::232]) by mx.google.com with ESMTPS id bp1si23018770wjc.55.2013.06.06.08.03.34 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 06 Jun 2013 08:03:35 -0700 (PDT) Received-SPF: neutral (google.com: 2a00:1450:400c:c05::232 is neither permitted nor denied by best guess record for domain of ard.biesheuvel@linaro.org) client-ip=2a00:1450:400c:c05::232; Received: by mail-wi0-f178.google.com with SMTP id k10so450957wiv.17 for ; Thu, 06 Jun 2013 08:03:34 -0700 (PDT) X-Received: by 10.180.188.141 with SMTP id ga13mr10379710wic.9.1370531014736; Thu, 06 Jun 2013 08:03:34 -0700 (PDT) Received: from ards-mac-mini.local (cag06-7-83-153-85-71.fbx.proxad.net. [83.153.85.71]) by mx.google.com with ESMTPSA id cw8sm16357697wib.7.2013.06.06.08.03.33 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 06 Jun 2013 08:03:34 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: nico@linaro.org, rob.herring@calxeda.com, linux@arm.linux.org.uk, patches@linaro.org, Ard Biesheuvel Subject: [PATCH 4/5] ARM: crypto: add NEON accelerated XOR implementation Date: Thu, 6 Jun 2013 17:03:04 +0200 Message-Id: <1370530985-20619-5-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 1.8.1.2 In-Reply-To: <1370530985-20619-1-git-send-email-ard.biesheuvel@linaro.org> References: <1370530985-20619-1-git-send-email-ard.biesheuvel@linaro.org> X-Gm-Message-State: ALoCoQmqoHBHpQiB3YX1dDyu2IbZ50ONoJaagSVu7RlB7iWzoM51e+ymFyURkL2EEPuq7qyoXrZv X-Original-Sender: ard.biesheuvel@linaro.org X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 2607:f8b0:400c:c01::22a is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Precedence: list Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org List-ID: X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , Add a source file xor-neon.c (which is really just the reference C implementation passed through the GCC vectorizer) and hook it up to the XOR framework. Output captured from a Cortex-A15 @ 1.7 GHz: xor: measuring software checksum speed arm4regs : 2261.600 MB/sec 8regs : 1771.600 MB/sec 32regs : 1441.600 MB/sec neon : 3619.600 MB/sec xor: using function: neon (3619.600 MB/sec) Signed-off-by: Ard Biesheuvel Acked-by: Nicolas Pitre --- arch/arm/include/asm/xor.h | 73 ++++++++++++++++++++++++++++++++++++++++++++++ arch/arm/lib/Makefile | 6 ++++ arch/arm/lib/xor-neon.c | 42 ++++++++++++++++++++++++++ 3 files changed, 121 insertions(+) create mode 100644 arch/arm/lib/xor-neon.c diff --git a/arch/arm/include/asm/xor.h b/arch/arm/include/asm/xor.h index 7604673..4ffb26d 100644 --- a/arch/arm/include/asm/xor.h +++ b/arch/arm/include/asm/xor.h @@ -7,7 +7,10 @@ * it under the terms of the GNU General Public License version 2 as * published by the Free Software Foundation. */ +#include #include +#include +#include #define __XOR(a1, a2) a1 ^= a2 @@ -138,4 +141,74 @@ static struct xor_block_template xor_block_arm4regs = { xor_speed(&xor_block_arm4regs); \ xor_speed(&xor_block_8regs); \ xor_speed(&xor_block_32regs); \ + NEON_TEMPLATES; \ } while (0) + +#ifdef CONFIG_KERNEL_MODE_NEON + +extern struct xor_block_template const xor_block_neon_inner; + +static void +xor_neon_2(unsigned long bytes, unsigned long *p1, unsigned long *p2) +{ + if (in_interrupt()) { + xor_arm4regs_2(bytes, p1, p2); + } else { + kernel_neon_begin(); + xor_block_neon_inner.do_2(bytes, p1, p2); + kernel_neon_end(); + } +} + +static void +xor_neon_3(unsigned long bytes, unsigned long *p1, unsigned long *p2, + unsigned long *p3) +{ + if (in_interrupt()) { + xor_arm4regs_3(bytes, p1, p2, p3); + } else { + kernel_neon_begin(); + xor_block_neon_inner.do_3(bytes, p1, p2, p3); + kernel_neon_end(); + } +} + +static void +xor_neon_4(unsigned long bytes, unsigned long *p1, unsigned long *p2, + unsigned long *p3, unsigned long *p4) +{ + if (in_interrupt()) { + xor_arm4regs_4(bytes, p1, p2, p3, p4); + } else { + kernel_neon_begin(); + xor_block_neon_inner.do_4(bytes, p1, p2, p3, p4); + kernel_neon_end(); + } +} + +static void +xor_neon_5(unsigned long bytes, unsigned long *p1, unsigned long *p2, + unsigned long *p3, unsigned long *p4, unsigned long *p5) +{ + if (in_interrupt()) { + xor_arm4regs_5(bytes, p1, p2, p3, p4, p5); + } else { + kernel_neon_begin(); + xor_block_neon_inner.do_5(bytes, p1, p2, p3, p4, p5); + kernel_neon_end(); + } +} + +static struct xor_block_template xor_block_neon = { + .name = "neon", + .do_2 = xor_neon_2, + .do_3 = xor_neon_3, + .do_4 = xor_neon_4, + .do_5 = xor_neon_5 +}; + +#define NEON_TEMPLATES \ + do { if (cpu_has_neon()) xor_speed(&xor_block_neon); } while (0) +#else +#define NEON_TEMPLATES +#endif diff --git a/arch/arm/lib/Makefile b/arch/arm/lib/Makefile index af72969..aaf3a87 100644 --- a/arch/arm/lib/Makefile +++ b/arch/arm/lib/Makefile @@ -45,3 +45,9 @@ lib-$(CONFIG_ARCH_SHARK) += io-shark.o $(obj)/csumpartialcopy.o: $(obj)/csumpartialcopygeneric.S $(obj)/csumpartialcopyuser.o: $(obj)/csumpartialcopygeneric.S + +ifeq ($(CONFIG_KERNEL_MODE_NEON),y) + NEON_FLAGS := -mfloat-abi=softfp -mfpu=neon + CFLAGS_xor-neon.o += $(NEON_FLAGS) + lib-$(CONFIG_XOR_BLOCKS) += xor-neon.o +endif diff --git a/arch/arm/lib/xor-neon.c b/arch/arm/lib/xor-neon.c new file mode 100644 index 0000000..f485e5a --- /dev/null +++ b/arch/arm/lib/xor-neon.c @@ -0,0 +1,42 @@ +/* + * linux/arch/arm/lib/xor-neon.c + * + * Copyright (C) 2013 Linaro Ltd + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include + +#ifndef __ARM_NEON__ +#error You should compile this file with '-mfloat-abi=softfp -mfpu=neon' +#endif + +/* + * Pull in the reference implementations while instructing GCC (through + * -ftree-vectorize) to attempt to exploit implicit parallelism and emit + * NEON instructions. + */ +#if __GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 6) +#pragma GCC optimize "tree-vectorize" +#else +/* + * While older versions of GCC do not generate incorrect code, they fail to + * recognize the parallel nature of these functions, and emit plain ARM code, + * which is known to be slower than the optimized ARM code in asm-arm/xor.h. + */ +#warning This code requires at least version 4.6 of GCC +#endif + +#pragma GCC diagnostic ignored "-Wunused-variable" +#include + +struct xor_block_template const xor_block_neon_inner = { + .name = "__inner_neon__", + .do_2 = xor_8regs_2, + .do_3 = xor_8regs_3, + .do_4 = xor_8regs_4, + .do_5 = xor_8regs_5, +};