From patchwork Thu Mar 6 04:12:47 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 25773 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-oa0-f69.google.com (mail-oa0-f69.google.com [209.85.219.69]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id DD553203C3 for ; Thu, 6 Mar 2014 04:14:17 +0000 (UTC) Received: by mail-oa0-f69.google.com with SMTP id i7sf7750838oag.0 for ; Wed, 05 Mar 2014 20:14:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:subject:date:message-id :in-reply-to:references:cc:precedence:list-id:list-unsubscribe :list-archive:list-post:list-help:list-subscribe:mime-version:sender :errors-to:x-original-sender:x-original-authentication-results :mailing-list:content-type:content-transfer-encoding; bh=LNUZfe0DbRNTn3qSqrXK0jEcqfl5eq/HCY2s8A3IOZ8=; b=KrhKb/Da0536vbnEA9W5hGOYLXGFhUPd3mngcO3FCpjq69O8lO4pRiqPXin82/aZHB YCbWKv8qVriCFtGWDbJ5vnHHc9BD2WbnXWo0BfG+5qJvc1rP5Bynf4cszHqk6U3rFxgz Ai/6WPavffpPCe1RfhRIdcbC247DOy8JtGF/ckFwq9yNDjlkUagKNsq05difySe4CHr2 BupY/0cnSen9sxMolfC4cv2elwPqb+3gjywJZ8M+WIyzYaOgSwJyHB8b4TJhW3B1iVsN fZ7xDjeOlP16kl/0bAXs3O3zhjhg1sLwiGqBXvdRdHG4HiXRJ9lX4PcFG0I+o+i+Y3ry vLhQ== X-Gm-Message-State: ALoCoQm0Zh7H9smODa7IMPTEumsXfjys74SI63VJu1g09sX9A0kAsoBj4QbY5X+G/JwRYwfRfris X-Received: by 10.43.161.202 with SMTP id mh10mr3646990icc.23.1394079257356; Wed, 05 Mar 2014 20:14:17 -0800 (PST) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.25.212 with SMTP id 78ls622036qgt.30.gmail; Wed, 05 Mar 2014 20:14:17 -0800 (PST) X-Received: by 10.221.66.73 with SMTP id xp9mr59935vcb.27.1394079257141; Wed, 05 Mar 2014 20:14:17 -0800 (PST) Received: from mail-vc0-f176.google.com (mail-vc0-f176.google.com [209.85.220.176]) by mx.google.com with ESMTPS id eb8si1462005vdb.151.2014.03.05.20.14.17 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 05 Mar 2014 20:14:17 -0800 (PST) Received-SPF: neutral (google.com: 209.85.220.176 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.220.176; Received: by mail-vc0-f176.google.com with SMTP id lc6so1533454vcb.7 for ; Wed, 05 Mar 2014 20:14:17 -0800 (PST) X-Received: by 10.58.37.67 with SMTP id w3mr3378945vej.22.1394079257007; Wed, 05 Mar 2014 20:14:17 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.78.9 with SMTP id i9csp9545vck; Wed, 5 Mar 2014 20:14:16 -0800 (PST) X-Received: by 10.194.190.10 with SMTP id gm10mr6284158wjc.55.1394079255929; Wed, 05 Mar 2014 20:14:15 -0800 (PST) Received: from casper.infradead.org (casper.infradead.org. [2001:770:15f::2]) by mx.google.com with ESMTPS id h7si3178593wja.53.2014.03.05.20.14.15 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Mar 2014 20:14:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org designates 2001:770:15f::2 as permitted sender) client-ip=2001:770:15f::2; Received: from merlin.infradead.org ([2001:4978:20e::2]) by casper.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1WLPgb-0002KT-Bc; Thu, 06 Mar 2014 04:13:33 +0000 Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1WLPgW-0001xf-8U; Thu, 06 Mar 2014 04:13:28 +0000 Received: from mail-pb0-f47.google.com ([209.85.160.47]) by merlin.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1WLPgS-0001wO-8m for linux-arm-kernel@lists.infradead.org; Thu, 06 Mar 2014 04:13:26 +0000 Received: by mail-pb0-f47.google.com with SMTP id up15so2038202pbc.20 for ; Wed, 05 Mar 2014 20:13:01 -0800 (PST) X-Received: by 10.66.141.165 with SMTP id rp5mr11792588pab.90.1394079181049; Wed, 05 Mar 2014 20:13:01 -0800 (PST) Received: from ards-macbook-pro.local (z88l218.static.ctm.net. [202.175.88.218]) by mx.google.com with ESMTPSA id fk4sm27773287pab.23.2014.03.05.20.12.58 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 05 Mar 2014 20:13:00 -0800 (PST) From: Ard Biesheuvel To: catalin.marinas@arm.com, will.deacon@arm.com, linux-arm-kernel@lists.infradead.org Subject: [PATCH 1/2] arm64: add kernel emulation for AES and SHA1 instructions Date: Thu, 6 Mar 2014 12:12:47 +0800 Message-Id: <1394079168-27666-2-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 1.8.3.2 In-Reply-To: <1394079168-27666-1-git-send-email-ard.biesheuvel@linaro.org> References: <1394079168-27666-1-git-send-email-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20140305_231324_455436_876B2883 X-CRM114-Status: GOOD ( 20.83 ) X-Spam-Score: -2.6 (--) X-Spam-Report: SpamAssassin version 3.3.2 on merlin.infradead.org summary: Content analysis details: (-2.6 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low trust [209.85.160.47 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] Cc: Ard Biesheuvel , nico@linaro.org, steve.capper@arm.com X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: ard.biesheuvel@linaro.org X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.220.176 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 This patch adds emulation of the AES and SHA1 instructions in Aarch64 mode. The mechanism it uses to hook into the undefined instruction exception handler was borrowed from the ARM tree. Signed-off-by: Ard Biesheuvel --- arch/arm64/Makefile | 1 + arch/arm64/emu/Makefile | 11 ++ arch/arm64/emu/ce-emu.c | 430 +++++++++++++++++++++++++++++++++++++++++ arch/arm64/include/asm/traps.h | 10 + arch/arm64/kernel/entry.S | 4 +- arch/arm64/kernel/traps.c | 49 +++++ 6 files changed, 504 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/emu/Makefile create mode 100644 arch/arm64/emu/ce-emu.c diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 8185a913c5ed..a4b3e253557d 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -42,6 +42,7 @@ TEXT_OFFSET := 0x00080000 export TEXT_OFFSET GZFLAGS +core-y += arch/arm64/emu/ core-y += arch/arm64/kernel/ arch/arm64/mm/ core-$(CONFIG_KVM) += arch/arm64/kvm/ core-$(CONFIG_XEN) += arch/arm64/xen/ diff --git a/arch/arm64/emu/Makefile b/arch/arm64/emu/Makefile new file mode 100644 index 000000000000..7df0397041ef --- /dev/null +++ b/arch/arm64/emu/Makefile @@ -0,0 +1,11 @@ +# +# linux/arch/arm64/emu/Makefile +# +# Copyright (C) 2013 Linaro Ltd +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# + +obj-y += ce-emu.o diff --git a/arch/arm64/emu/ce-emu.c b/arch/arm64/emu/ce-emu.c new file mode 100644 index 000000000000..daead19f323d --- /dev/null +++ b/arch/arm64/emu/ce-emu.c @@ -0,0 +1,430 @@ +/* + * ce-emu.c - emulate ARMv8 Crypto Instructions + * + * Copyright (C) 2013 Linaro Ltd + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include +#include +#include +#include +#include +#include + +union neon_reg { + u8 bytes[16]; + u32 words[4]; + u64 l[2]; +} __aligned(8); + +static void add_sub_shift(union neon_reg *st, union neon_reg *rk, int inv); +static void mix_columns(union neon_reg *out, union neon_reg *in, int inv); + +#define REG_ACCESS(op, r, mem) \ + do { case r: asm(#op " {v" #r ".16b}, [%0]" : : "r"(mem)); goto out; \ + } while (0) + +#define REG_SWITCH(reg, op, m) do { switch (reg) { \ + REG_ACCESS(op, 0, m); REG_ACCESS(op, 1, m); REG_ACCESS(op, 2, m); \ + REG_ACCESS(op, 3, m); REG_ACCESS(op, 4, m); REG_ACCESS(op, 5, m); \ + REG_ACCESS(op, 6, m); REG_ACCESS(op, 7, m); REG_ACCESS(op, 8, m); \ + REG_ACCESS(op, 9, m); REG_ACCESS(op, 10, m); REG_ACCESS(op, 11, m); \ + REG_ACCESS(op, 12, m); REG_ACCESS(op, 13, m); REG_ACCESS(op, 14, m); \ + REG_ACCESS(op, 15, m); REG_ACCESS(op, 16, m); REG_ACCESS(op, 17, m); \ + REG_ACCESS(op, 18, m); REG_ACCESS(op, 19, m); REG_ACCESS(op, 20, m); \ + REG_ACCESS(op, 21, m); REG_ACCESS(op, 22, m); REG_ACCESS(op, 23, m); \ + REG_ACCESS(op, 24, m); REG_ACCESS(op, 25, m); REG_ACCESS(op, 26, m); \ + REG_ACCESS(op, 27, m); REG_ACCESS(op, 28, m); REG_ACCESS(op, 29, m); \ + REG_ACCESS(op, 30, m); REG_ACCESS(op, 31, m); \ + } out:; } while (0) + +static void load_neon_reg(void *mem, int reg) +{ + REG_SWITCH(reg, st1, mem); +} + +static void save_neon_reg(void *mem, int reg) +{ + REG_SWITCH(reg, ld1, mem); +} + +static void aesce_do_emulate(unsigned int instr) +{ + enum { AESE, AESD, AESMC, AESIMC } kind = (instr >> 12) & 3; + int rn = (instr >> 5) & 0x1f; + int rd = instr & 0x1f; + union neon_reg in, out; + + load_neon_reg(&in, rn); + + switch (kind) { + case AESE: + case AESD: + load_neon_reg(&out, rd); + add_sub_shift(&out, &in, kind & 1); + break; + case AESMC: + case AESIMC: + mix_columns(&out, &in, kind & 1); + break; + } + save_neon_reg(&out, rd); +} + +static int aesce_emu_instr(struct pt_regs *regs, unsigned int instr); + +static struct undef_hook aesce_emu_uh = { + .instr_val = 0x4e284800, + .instr_mask = 0xffffcc00, + .fn = aesce_emu_instr, +}; + +static int aesce_emu_instr(struct pt_regs *regs, unsigned int instr) +{ + do { + aesce_do_emulate(instr); + regs->pc += 4; + get_user(instr, (u32 __user *)regs->pc); + } while ((instr & aesce_emu_uh.instr_mask) == aesce_emu_uh.instr_val); + + return 0; +} + +static void sha1ce_do_emulate(unsigned int instr) +{ + enum { + SHA1C, SHA1H, SHA1P, SHA1SU1, SHA1M, SHA1SU0 = 6 + } kind = (instr >> 11) & 7; + + int rn = (instr >> 5) & 0x1f; + int rm = (instr >> 16) & 0x1f; + int rd = instr & 0x1f; + + union neon_reg op_n, op_d; + + load_neon_reg(&op_n, rn); + load_neon_reg(&op_d, rd); + + if (kind == SHA1H) { + op_d.words[0] = (op_n.words[0] << 30) | (op_n.words[0] >> 2); + op_d.words[1] = op_d.words[2] = op_d.words[3] = 0; + } else if (kind == SHA1SU1) { + op_d.words[3] = rol32(op_d.words[3], 1) ^ + rol32(op_d.words[0] ^ op_n.words[1], 2); + op_d.words[2] = rol32(op_d.words[2] ^ op_n.words[3], 1); + op_d.words[1] = rol32(op_d.words[1] ^ op_n.words[2], 1); + op_d.words[0] = rol32(op_d.words[0] ^ op_n.words[1], 1); + } else { + union neon_reg op_m; + + load_neon_reg(&op_m, rm); + + if (kind == SHA1SU0) { + op_d.l[0] = op_d.l[1] ^ op_d.l[0] ^ op_m.l[0]; + op_d.l[1] = op_n.l[0] ^ op_d.l[1] ^ op_m.l[1]; + } else { + u32 t; + int i; + + for (i = 0; i < 4; i++) { + switch (kind) { + case SHA1C: + /* choose */ + t = ((op_d.words[2] ^ op_d.words[3]) + & op_d.words[1]) ^ op_d.words[3]; + break; + case SHA1M: + /* majority */ + t = (op_d.words[1] & op_d.words[2]) + + ((op_d.words[1] ^ op_d.words[2]) + & op_d.words[3]); + break; + default: /* == case SHA1P: */ + /* parity */ + t = op_d.words[1] ^ op_d.words[2] + ^ op_d.words[3]; + break; + } + t += op_n.words[0] + rol32(op_d.words[0], 5) + + op_m.words[i]; + + op_n.words[0] = op_d.words[3]; + op_d.words[3] = op_d.words[2]; + op_d.words[2] = rol32(op_d.words[1], 30); + op_d.words[1] = op_d.words[0]; + op_d.words[0] = t; + } + } + } + save_neon_reg(&op_d, rd); +} + +static int sha1ce_emu_instr(struct pt_regs *regs, unsigned int instr); + +static struct undef_hook sha1ce_emu_uh[] = { { + /* 3 reg variant (sha1c, sha1m, sha1p, sha1su0) */ + .instr_val = 0x5e000000, + .instr_mask = 0xffe0cc00, + .fn = sha1ce_emu_instr, +}, { + /* 2 reg variant (sha1h, sha1su1) */ + .instr_val = 0x5e280800, + .instr_mask = 0xffffec00, + .fn = sha1ce_emu_instr, +} }; + +static int sha1ce_emu_instr(struct pt_regs *regs, unsigned int instr) +{ + do { + sha1ce_do_emulate(instr); + regs->pc += 4; + get_user(instr, (u32 __user *)regs->pc); + } while ((instr & sha1ce_emu_uh[0].instr_mask) == + sha1ce_emu_uh[0].instr_val + || (instr & sha1ce_emu_uh[1].instr_mask) == + sha1ce_emu_uh[1].instr_val); + + return 0; +} + +static int ce_emu_init(void) +{ + register_undef_hook(&aesce_emu_uh); + register_undef_hook(sha1ce_emu_uh); + register_undef_hook(sha1ce_emu_uh + 1); + elf_hwcap |= HWCAP_AES | HWCAP_SHA1; + return 0; +} + +arch_initcall(ce_emu_init); + +static void add_sub_shift(union neon_reg *st, union neon_reg *rk, int inv) +{ + static u8 const sbox[][256] = { { + 0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5, + 0x30, 0x01, 0x67, 0x2b, 0xfe, 0xd7, 0xab, 0x76, + 0xca, 0x82, 0xc9, 0x7d, 0xfa, 0x59, 0x47, 0xf0, + 0xad, 0xd4, 0xa2, 0xaf, 0x9c, 0xa4, 0x72, 0xc0, + 0xb7, 0xfd, 0x93, 0x26, 0x36, 0x3f, 0xf7, 0xcc, + 0x34, 0xa5, 0xe5, 0xf1, 0x71, 0xd8, 0x31, 0x15, + 0x04, 0xc7, 0x23, 0xc3, 0x18, 0x96, 0x05, 0x9a, + 0x07, 0x12, 0x80, 0xe2, 0xeb, 0x27, 0xb2, 0x75, + 0x09, 0x83, 0x2c, 0x1a, 0x1b, 0x6e, 0x5a, 0xa0, + 0x52, 0x3b, 0xd6, 0xb3, 0x29, 0xe3, 0x2f, 0x84, + 0x53, 0xd1, 0x00, 0xed, 0x20, 0xfc, 0xb1, 0x5b, + 0x6a, 0xcb, 0xbe, 0x39, 0x4a, 0x4c, 0x58, 0xcf, + 0xd0, 0xef, 0xaa, 0xfb, 0x43, 0x4d, 0x33, 0x85, + 0x45, 0xf9, 0x02, 0x7f, 0x50, 0x3c, 0x9f, 0xa8, + 0x51, 0xa3, 0x40, 0x8f, 0x92, 0x9d, 0x38, 0xf5, + 0xbc, 0xb6, 0xda, 0x21, 0x10, 0xff, 0xf3, 0xd2, + 0xcd, 0x0c, 0x13, 0xec, 0x5f, 0x97, 0x44, 0x17, + 0xc4, 0xa7, 0x7e, 0x3d, 0x64, 0x5d, 0x19, 0x73, + 0x60, 0x81, 0x4f, 0xdc, 0x22, 0x2a, 0x90, 0x88, + 0x46, 0xee, 0xb8, 0x14, 0xde, 0x5e, 0x0b, 0xdb, + 0xe0, 0x32, 0x3a, 0x0a, 0x49, 0x06, 0x24, 0x5c, + 0xc2, 0xd3, 0xac, 0x62, 0x91, 0x95, 0xe4, 0x79, + 0xe7, 0xc8, 0x37, 0x6d, 0x8d, 0xd5, 0x4e, 0xa9, + 0x6c, 0x56, 0xf4, 0xea, 0x65, 0x7a, 0xae, 0x08, + 0xba, 0x78, 0x25, 0x2e, 0x1c, 0xa6, 0xb4, 0xc6, + 0xe8, 0xdd, 0x74, 0x1f, 0x4b, 0xbd, 0x8b, 0x8a, + 0x70, 0x3e, 0xb5, 0x66, 0x48, 0x03, 0xf6, 0x0e, + 0x61, 0x35, 0x57, 0xb9, 0x86, 0xc1, 0x1d, 0x9e, + 0xe1, 0xf8, 0x98, 0x11, 0x69, 0xd9, 0x8e, 0x94, + 0x9b, 0x1e, 0x87, 0xe9, 0xce, 0x55, 0x28, 0xdf, + 0x8c, 0xa1, 0x89, 0x0d, 0xbf, 0xe6, 0x42, 0x68, + 0x41, 0x99, 0x2d, 0x0f, 0xb0, 0x54, 0xbb, 0x16 + }, { + 0x52, 0x09, 0x6a, 0xd5, 0x30, 0x36, 0xa5, 0x38, + 0xbf, 0x40, 0xa3, 0x9e, 0x81, 0xf3, 0xd7, 0xfb, + 0x7c, 0xe3, 0x39, 0x82, 0x9b, 0x2f, 0xff, 0x87, + 0x34, 0x8e, 0x43, 0x44, 0xc4, 0xde, 0xe9, 0xcb, + 0x54, 0x7b, 0x94, 0x32, 0xa6, 0xc2, 0x23, 0x3d, + 0xee, 0x4c, 0x95, 0x0b, 0x42, 0xfa, 0xc3, 0x4e, + 0x08, 0x2e, 0xa1, 0x66, 0x28, 0xd9, 0x24, 0xb2, + 0x76, 0x5b, 0xa2, 0x49, 0x6d, 0x8b, 0xd1, 0x25, + 0x72, 0xf8, 0xf6, 0x64, 0x86, 0x68, 0x98, 0x16, + 0xd4, 0xa4, 0x5c, 0xcc, 0x5d, 0x65, 0xb6, 0x92, + 0x6c, 0x70, 0x48, 0x50, 0xfd, 0xed, 0xb9, 0xda, + 0x5e, 0x15, 0x46, 0x57, 0xa7, 0x8d, 0x9d, 0x84, + 0x90, 0xd8, 0xab, 0x00, 0x8c, 0xbc, 0xd3, 0x0a, + 0xf7, 0xe4, 0x58, 0x05, 0xb8, 0xb3, 0x45, 0x06, + 0xd0, 0x2c, 0x1e, 0x8f, 0xca, 0x3f, 0x0f, 0x02, + 0xc1, 0xaf, 0xbd, 0x03, 0x01, 0x13, 0x8a, 0x6b, + 0x3a, 0x91, 0x11, 0x41, 0x4f, 0x67, 0xdc, 0xea, + 0x97, 0xf2, 0xcf, 0xce, 0xf0, 0xb4, 0xe6, 0x73, + 0x96, 0xac, 0x74, 0x22, 0xe7, 0xad, 0x35, 0x85, + 0xe2, 0xf9, 0x37, 0xe8, 0x1c, 0x75, 0xdf, 0x6e, + 0x47, 0xf1, 0x1a, 0x71, 0x1d, 0x29, 0xc5, 0x89, + 0x6f, 0xb7, 0x62, 0x0e, 0xaa, 0x18, 0xbe, 0x1b, + 0xfc, 0x56, 0x3e, 0x4b, 0xc6, 0xd2, 0x79, 0x20, + 0x9a, 0xdb, 0xc0, 0xfe, 0x78, 0xcd, 0x5a, 0xf4, + 0x1f, 0xdd, 0xa8, 0x33, 0x88, 0x07, 0xc7, 0x31, + 0xb1, 0x12, 0x10, 0x59, 0x27, 0x80, 0xec, 0x5f, + 0x60, 0x51, 0x7f, 0xa9, 0x19, 0xb5, 0x4a, 0x0d, + 0x2d, 0xe5, 0x7a, 0x9f, 0x93, 0xc9, 0x9c, 0xef, + 0xa0, 0xe0, 0x3b, 0x4d, 0xae, 0x2a, 0xf5, 0xb0, + 0xc8, 0xeb, 0xbb, 0x3c, 0x83, 0x53, 0x99, 0x61, + 0x17, 0x2b, 0x04, 0x7e, 0xba, 0x77, 0xd6, 0x26, + 0xe1, 0x69, 0x14, 0x63, 0x55, 0x21, 0x0c, 0x7d + } }; + static u8 const permute[][16] = { + { 0, 5, 10, 15, 4, 9, 14, 3, 8, 13, 2, 7, 12, 1, 6, 11 }, + { 0, 13, 10, 7, 4, 1, 14, 11, 8, 5, 2, 15, 12, 9, 6, 3 }, + }; + int i; + + rk->l[0] ^= st->l[0]; + rk->l[1] ^= st->l[1]; + + for (i = 0; i < 16; i++) + st->bytes[i] = sbox[inv][rk->bytes[permute[inv][i]]]; +} + +static void mix_columns(union neon_reg *out, union neon_reg *in, int inv) +{ + static u32 const mc[][256] = { { + 0x00000000, 0x03010102, 0x06020204, 0x05030306, + 0x0c040408, 0x0f05050a, 0x0a06060c, 0x0907070e, + 0x18080810, 0x1b090912, 0x1e0a0a14, 0x1d0b0b16, + 0x140c0c18, 0x170d0d1a, 0x120e0e1c, 0x110f0f1e, + 0x30101020, 0x33111122, 0x36121224, 0x35131326, + 0x3c141428, 0x3f15152a, 0x3a16162c, 0x3917172e, + 0x28181830, 0x2b191932, 0x2e1a1a34, 0x2d1b1b36, + 0x241c1c38, 0x271d1d3a, 0x221e1e3c, 0x211f1f3e, + 0x60202040, 0x63212142, 0x66222244, 0x65232346, + 0x6c242448, 0x6f25254a, 0x6a26264c, 0x6927274e, + 0x78282850, 0x7b292952, 0x7e2a2a54, 0x7d2b2b56, + 0x742c2c58, 0x772d2d5a, 0x722e2e5c, 0x712f2f5e, + 0x50303060, 0x53313162, 0x56323264, 0x55333366, + 0x5c343468, 0x5f35356a, 0x5a36366c, 0x5937376e, + 0x48383870, 0x4b393972, 0x4e3a3a74, 0x4d3b3b76, + 0x443c3c78, 0x473d3d7a, 0x423e3e7c, 0x413f3f7e, + 0xc0404080, 0xc3414182, 0xc6424284, 0xc5434386, + 0xcc444488, 0xcf45458a, 0xca46468c, 0xc947478e, + 0xd8484890, 0xdb494992, 0xde4a4a94, 0xdd4b4b96, + 0xd44c4c98, 0xd74d4d9a, 0xd24e4e9c, 0xd14f4f9e, + 0xf05050a0, 0xf35151a2, 0xf65252a4, 0xf55353a6, + 0xfc5454a8, 0xff5555aa, 0xfa5656ac, 0xf95757ae, + 0xe85858b0, 0xeb5959b2, 0xee5a5ab4, 0xed5b5bb6, + 0xe45c5cb8, 0xe75d5dba, 0xe25e5ebc, 0xe15f5fbe, + 0xa06060c0, 0xa36161c2, 0xa66262c4, 0xa56363c6, + 0xac6464c8, 0xaf6565ca, 0xaa6666cc, 0xa96767ce, + 0xb86868d0, 0xbb6969d2, 0xbe6a6ad4, 0xbd6b6bd6, + 0xb46c6cd8, 0xb76d6dda, 0xb26e6edc, 0xb16f6fde, + 0x907070e0, 0x937171e2, 0x967272e4, 0x957373e6, + 0x9c7474e8, 0x9f7575ea, 0x9a7676ec, 0x997777ee, + 0x887878f0, 0x8b7979f2, 0x8e7a7af4, 0x8d7b7bf6, + 0x847c7cf8, 0x877d7dfa, 0x827e7efc, 0x817f7ffe, + 0x9b80801b, 0x98818119, 0x9d82821f, 0x9e83831d, + 0x97848413, 0x94858511, 0x91868617, 0x92878715, + 0x8388880b, 0x80898909, 0x858a8a0f, 0x868b8b0d, + 0x8f8c8c03, 0x8c8d8d01, 0x898e8e07, 0x8a8f8f05, + 0xab90903b, 0xa8919139, 0xad92923f, 0xae93933d, + 0xa7949433, 0xa4959531, 0xa1969637, 0xa2979735, + 0xb398982b, 0xb0999929, 0xb59a9a2f, 0xb69b9b2d, + 0xbf9c9c23, 0xbc9d9d21, 0xb99e9e27, 0xba9f9f25, + 0xfba0a05b, 0xf8a1a159, 0xfda2a25f, 0xfea3a35d, + 0xf7a4a453, 0xf4a5a551, 0xf1a6a657, 0xf2a7a755, + 0xe3a8a84b, 0xe0a9a949, 0xe5aaaa4f, 0xe6abab4d, + 0xefacac43, 0xecadad41, 0xe9aeae47, 0xeaafaf45, + 0xcbb0b07b, 0xc8b1b179, 0xcdb2b27f, 0xceb3b37d, + 0xc7b4b473, 0xc4b5b571, 0xc1b6b677, 0xc2b7b775, + 0xd3b8b86b, 0xd0b9b969, 0xd5baba6f, 0xd6bbbb6d, + 0xdfbcbc63, 0xdcbdbd61, 0xd9bebe67, 0xdabfbf65, + 0x5bc0c09b, 0x58c1c199, 0x5dc2c29f, 0x5ec3c39d, + 0x57c4c493, 0x54c5c591, 0x51c6c697, 0x52c7c795, + 0x43c8c88b, 0x40c9c989, 0x45caca8f, 0x46cbcb8d, + 0x4fcccc83, 0x4ccdcd81, 0x49cece87, 0x4acfcf85, + 0x6bd0d0bb, 0x68d1d1b9, 0x6dd2d2bf, 0x6ed3d3bd, + 0x67d4d4b3, 0x64d5d5b1, 0x61d6d6b7, 0x62d7d7b5, + 0x73d8d8ab, 0x70d9d9a9, 0x75dadaaf, 0x76dbdbad, + 0x7fdcdca3, 0x7cdddda1, 0x79dedea7, 0x7adfdfa5, + 0x3be0e0db, 0x38e1e1d9, 0x3de2e2df, 0x3ee3e3dd, + 0x37e4e4d3, 0x34e5e5d1, 0x31e6e6d7, 0x32e7e7d5, + 0x23e8e8cb, 0x20e9e9c9, 0x25eaeacf, 0x26ebebcd, + 0x2fececc3, 0x2cededc1, 0x29eeeec7, 0x2aefefc5, + 0x0bf0f0fb, 0x08f1f1f9, 0x0df2f2ff, 0x0ef3f3fd, + 0x07f4f4f3, 0x04f5f5f1, 0x01f6f6f7, 0x02f7f7f5, + 0x13f8f8eb, 0x10f9f9e9, 0x15fafaef, 0x16fbfbed, + 0x1ffcfce3, 0x1cfdfde1, 0x19fefee7, 0x1affffe5, + }, { + 0x00000000, 0x0b0d090e, 0x161a121c, 0x1d171b12, + 0x2c342438, 0x27392d36, 0x3a2e3624, 0x31233f2a, + 0x58684870, 0x5365417e, 0x4e725a6c, 0x457f5362, + 0x745c6c48, 0x7f516546, 0x62467e54, 0x694b775a, + 0xb0d090e0, 0xbbdd99ee, 0xa6ca82fc, 0xadc78bf2, + 0x9ce4b4d8, 0x97e9bdd6, 0x8afea6c4, 0x81f3afca, + 0xe8b8d890, 0xe3b5d19e, 0xfea2ca8c, 0xf5afc382, + 0xc48cfca8, 0xcf81f5a6, 0xd296eeb4, 0xd99be7ba, + 0x7bbb3bdb, 0x70b632d5, 0x6da129c7, 0x66ac20c9, + 0x578f1fe3, 0x5c8216ed, 0x41950dff, 0x4a9804f1, + 0x23d373ab, 0x28de7aa5, 0x35c961b7, 0x3ec468b9, + 0x0fe75793, 0x04ea5e9d, 0x19fd458f, 0x12f04c81, + 0xcb6bab3b, 0xc066a235, 0xdd71b927, 0xd67cb029, + 0xe75f8f03, 0xec52860d, 0xf1459d1f, 0xfa489411, + 0x9303e34b, 0x980eea45, 0x8519f157, 0x8e14f859, + 0xbf37c773, 0xb43ace7d, 0xa92dd56f, 0xa220dc61, + 0xf66d76ad, 0xfd607fa3, 0xe07764b1, 0xeb7a6dbf, + 0xda595295, 0xd1545b9b, 0xcc434089, 0xc74e4987, + 0xae053edd, 0xa50837d3, 0xb81f2cc1, 0xb31225cf, + 0x82311ae5, 0x893c13eb, 0x942b08f9, 0x9f2601f7, + 0x46bde64d, 0x4db0ef43, 0x50a7f451, 0x5baafd5f, + 0x6a89c275, 0x6184cb7b, 0x7c93d069, 0x779ed967, + 0x1ed5ae3d, 0x15d8a733, 0x08cfbc21, 0x03c2b52f, + 0x32e18a05, 0x39ec830b, 0x24fb9819, 0x2ff69117, + 0x8dd64d76, 0x86db4478, 0x9bcc5f6a, 0x90c15664, + 0xa1e2694e, 0xaaef6040, 0xb7f87b52, 0xbcf5725c, + 0xd5be0506, 0xdeb30c08, 0xc3a4171a, 0xc8a91e14, + 0xf98a213e, 0xf2872830, 0xef903322, 0xe49d3a2c, + 0x3d06dd96, 0x360bd498, 0x2b1ccf8a, 0x2011c684, + 0x1132f9ae, 0x1a3ff0a0, 0x0728ebb2, 0x0c25e2bc, + 0x656e95e6, 0x6e639ce8, 0x737487fa, 0x78798ef4, + 0x495ab1de, 0x4257b8d0, 0x5f40a3c2, 0x544daacc, + 0xf7daec41, 0xfcd7e54f, 0xe1c0fe5d, 0xeacdf753, + 0xdbeec879, 0xd0e3c177, 0xcdf4da65, 0xc6f9d36b, + 0xafb2a431, 0xa4bfad3f, 0xb9a8b62d, 0xb2a5bf23, + 0x83868009, 0x888b8907, 0x959c9215, 0x9e919b1b, + 0x470a7ca1, 0x4c0775af, 0x51106ebd, 0x5a1d67b3, + 0x6b3e5899, 0x60335197, 0x7d244a85, 0x7629438b, + 0x1f6234d1, 0x146f3ddf, 0x097826cd, 0x02752fc3, + 0x335610e9, 0x385b19e7, 0x254c02f5, 0x2e410bfb, + 0x8c61d79a, 0x876cde94, 0x9a7bc586, 0x9176cc88, + 0xa055f3a2, 0xab58faac, 0xb64fe1be, 0xbd42e8b0, + 0xd4099fea, 0xdf0496e4, 0xc2138df6, 0xc91e84f8, + 0xf83dbbd2, 0xf330b2dc, 0xee27a9ce, 0xe52aa0c0, + 0x3cb1477a, 0x37bc4e74, 0x2aab5566, 0x21a65c68, + 0x10856342, 0x1b886a4c, 0x069f715e, 0x0d927850, + 0x64d90f0a, 0x6fd40604, 0x72c31d16, 0x79ce1418, + 0x48ed2b32, 0x43e0223c, 0x5ef7392e, 0x55fa3020, + 0x01b79aec, 0x0aba93e2, 0x17ad88f0, 0x1ca081fe, + 0x2d83bed4, 0x268eb7da, 0x3b99acc8, 0x3094a5c6, + 0x59dfd29c, 0x52d2db92, 0x4fc5c080, 0x44c8c98e, + 0x75ebf6a4, 0x7ee6ffaa, 0x63f1e4b8, 0x68fcedb6, + 0xb1670a0c, 0xba6a0302, 0xa77d1810, 0xac70111e, + 0x9d532e34, 0x965e273a, 0x8b493c28, 0x80443526, + 0xe90f427c, 0xe2024b72, 0xff155060, 0xf418596e, + 0xc53b6644, 0xce366f4a, 0xd3217458, 0xd82c7d56, + 0x7a0ca137, 0x7101a839, 0x6c16b32b, 0x671bba25, + 0x5638850f, 0x5d358c01, 0x40229713, 0x4b2f9e1d, + 0x2264e947, 0x2969e049, 0x347efb5b, 0x3f73f255, + 0x0e50cd7f, 0x055dc471, 0x184adf63, 0x1347d66d, + 0xcadc31d7, 0xc1d138d9, 0xdcc623cb, 0xd7cb2ac5, + 0xe6e815ef, 0xede51ce1, 0xf0f207f3, 0xfbff0efd, + 0x92b479a7, 0x99b970a9, 0x84ae6bbb, 0x8fa362b5, + 0xbe805d9f, 0xb58d5491, 0xa89a4f83, 0xa397468d, + } }; + + int i; + + for (i = 0; i < 16; i += 4) + out->words[i >> 2] = cpu_to_le32( + mc[inv][in->bytes[i]] ^ + rol32(mc[inv][in->bytes[i + 1]], 8) ^ + rol32(mc[inv][in->bytes[i + 2]], 16) ^ + rol32(mc[inv][in->bytes[i + 3]], 24)); +} diff --git a/arch/arm64/include/asm/traps.h b/arch/arm64/include/asm/traps.h index 10ca8ff93cc2..781e50cb2f03 100644 --- a/arch/arm64/include/asm/traps.h +++ b/arch/arm64/include/asm/traps.h @@ -27,4 +27,14 @@ static inline int in_exception_text(unsigned long ptr) ptr < (unsigned long)&__exception_text_end; } +struct undef_hook { + struct list_head node; + u32 instr_mask; + u32 instr_val; + int (*fn)(struct pt_regs *regs, unsigned int instr); +}; + +void register_undef_hook(struct undef_hook *hook); +void unregister_undef_hook(struct undef_hook *hook); + #endif diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 80464e2fb1a5..a1af17360339 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -283,7 +283,9 @@ el1_undef: * Undefined instruction */ mov x0, sp - b do_undefinstr + bl do_undefinstr + + kernel_exit 1 el1_dbg: /* * Debug exception handling diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index 7ffadddb645d..3cc4c915b73f 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -257,11 +257,60 @@ void arm64_notify_die(const char *str, struct pt_regs *regs, die(str, regs, err); } +static LIST_HEAD(undef_hook); +static DEFINE_RAW_SPINLOCK(undef_lock); + +void register_undef_hook(struct undef_hook *hook) +{ + unsigned long flags; + + raw_spin_lock_irqsave(&undef_lock, flags); + list_add(&hook->node, &undef_hook); + raw_spin_unlock_irqrestore(&undef_lock, flags); +} + +void unregister_undef_hook(struct undef_hook *hook) +{ + unsigned long flags; + + raw_spin_lock_irqsave(&undef_lock, flags); + list_del(&hook->node); + raw_spin_unlock_irqrestore(&undef_lock, flags); +} + +static int call_undef_hook(struct pt_regs *regs, void __user *pc) +{ + struct undef_hook *hook; + unsigned long flags; + int (*fn)(struct pt_regs *regs, unsigned int instr) = NULL; + unsigned int instr; + mm_segment_t fs; + int ret; + + fs = get_fs(); + set_fs(KERNEL_DS); + + get_user(instr, (u32 __user *)pc); + + raw_spin_lock_irqsave(&undef_lock, flags); + list_for_each_entry(hook, &undef_hook, node) + if ((instr & hook->instr_mask) == hook->instr_val) + fn = hook->fn; + raw_spin_unlock_irqrestore(&undef_lock, flags); + + ret = fn ? fn(regs, instr) : 1; + set_fs(fs); + return ret; +} + asmlinkage void __exception do_undefinstr(struct pt_regs *regs) { siginfo_t info; void __user *pc = (void __user *)instruction_pointer(regs); + if (call_undef_hook(regs, pc) == 0) + return; + /* check for AArch32 breakpoint instructions */ if (!aarch32_break_handler(regs)) return;