From patchwork Fri Aug 8 03:37:36 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Nan X-Patchwork-Id: 35071 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-pa0-f69.google.com (mail-pa0-f69.google.com [209.85.220.69]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 5AE9321137 for ; Fri, 8 Aug 2014 03:52:04 +0000 (UTC) Received: by mail-pa0-f69.google.com with SMTP id kx10sf32349159pab.4 for ; Thu, 07 Aug 2014 20:52:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:subject:date:message-id :mime-version:cc:precedence:list-id:list-unsubscribe:list-archive :list-post:list-help:list-subscribe:sender:errors-to :x-original-sender:x-original-authentication-results:mailing-list :content-type:content-transfer-encoding; bh=A9yHlpjoTG9sU5d18g37nMpNNd0XSiczAvQzy9HFsNQ=; b=ZQTi7+o/EIei1EdmdYN4DnJtudG08c7v7TdpRJif42dB9MBnJFxw9ugCaugft9j4L7 iDFMXrUK+tLarD8Yc5Y8ChxEwf0kdGidOrAOu4mRRSeHXhP7L/WNEosZ1heLbD8aGpSM YMm8rFJqCRixt8L+N296bGYsS6SvC5KKJC4i+BPmluzbdvGjzVqfLg8fBo/gLvdaL2FE emC43KNlUpHvP89T0oqgl+/9e5KHcR9FS6sQkG7+tfzD5BKp8lOmgInTrJSNzQO6V7KO +Y5D5a8UZKJB68BJdqpOlo6A6VoeOtGFhLpJQ4d3ChbEQIKY288VT4MJEC2KAX1iurJf ucow== X-Gm-Message-State: ALoCoQm7oJlcRwfm4qVu9XF2rjh8rcnqrpfjEA0uP7yOYVewTVgWONmj3s4Hy64xVZIOVV+jaDoG X-Received: by 10.68.69.98 with SMTP id d2mr11332032pbu.0.1407469923681; Thu, 07 Aug 2014 20:52:03 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.51.170 with SMTP id u39ls386717qga.21.gmail; Thu, 07 Aug 2014 20:52:03 -0700 (PDT) X-Received: by 10.221.44.69 with SMTP id uf5mr19898995vcb.4.1407469923555; Thu, 07 Aug 2014 20:52:03 -0700 (PDT) Received: from mail-vc0-f173.google.com (mail-vc0-f173.google.com [209.85.220.173]) by mx.google.com with ESMTPS id jw6si1339368veb.67.2014.08.07.20.52.03 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 07 Aug 2014 20:52:03 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.173 as permitted sender) client-ip=209.85.220.173; Received: by mail-vc0-f173.google.com with SMTP id hy10so7571073vcb.32 for ; Thu, 07 Aug 2014 20:52:03 -0700 (PDT) X-Received: by 10.221.64.80 with SMTP id xh16mr19719699vcb.35.1407469923390; Thu, 07 Aug 2014 20:52:03 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.221.37.5 with SMTP id tc5csp86046vcb; Thu, 7 Aug 2014 20:52:02 -0700 (PDT) X-Received: by 10.68.209.225 with SMTP id mp1mr11212219pbc.80.1407469922436; Thu, 07 Aug 2014 20:52:02 -0700 (PDT) Received: from bombadil.infradead.org (bombadil.infradead.org. [2001:1868:205::9]) by mx.google.com with ESMTPS id oj5si1435656pdb.250.2014.08.07.20.52.02 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Aug 2014 20:52:02 -0700 (PDT) Received-SPF: none (google.com: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org does not designate permitted sender hosts) client-ip=2001:1868:205::9; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1XFbBJ-0006gT-Jq; Fri, 08 Aug 2014 03:49:29 +0000 Received: from szxga01-in.huawei.com ([119.145.14.64]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1XFbBE-0006fn-9T for linux-arm-kernel@lists.infradead.org; Fri, 08 Aug 2014 03:49:26 +0000 Received: from 172.24.2.119 (EHLO lggeml420-hub.china.huawei.com) ([172.24.2.119]) by szxrg01-dlp.huawei.com (MOS 4.3.7-GA FastPath queued) with ESMTP id BZW04981; Fri, 08 Aug 2014 11:48:25 +0800 (CST) Received: from kernel-host.huawei (10.107.197.247) by lggeml420-hub.china.huawei.com (10.72.61.115) with Microsoft SMTP Server id 14.3.158.1; Fri, 8 Aug 2014 11:48:15 +0800 From: Wang Nan To: Russell King - ARM Linux , Masami Hiramatsu , "Jon Medhurst (Tixy)" , , , , Will Deacon , , Subject: [RFC PATCH v2] kprobes: arm: enable OPTPROBES for ARM 32 Date: Fri, 8 Aug 2014 11:37:36 +0800 Message-ID: <1407469056-53478-1-git-send-email-wangnan0@huawei.com> X-Mailer: git-send-email 1.8.4 MIME-Version: 1.0 X-Originating-IP: [10.107.197.247] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20140807_204925_473802_CEF523B4 X-CRM114-Status: GOOD ( 24.96 ) X-Spam-Score: -1.4 (-) X-Spam-Report: SpamAssassin version 3.4.0 on bombadil.infradead.org summary: Content analysis details: (-1.4 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low trust [119.145.14.64 listed in list.dnswl.org] -0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3) [119.145.14.64 listed in wl.mailspike.net] -0.7 RP_MATCHES_RCVD Envelope sender domain matches handover relay domain -0.0 SPF_PASS SPF: sender matches SPF record -0.0 RCVD_IN_MSPIKE_WL Mailspike good senders Cc: wangnan0@huawei.com, lizefan@huawei.com, peifeiyue@huawei.com X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: wangnan0@huawei.com X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.173 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 This patch introduce kprobeopt for ARM 32. Limitations: - Currently only kernel compiled with ARM ISA is supported. - Offset between probe point and optinsn slot must not larger than 32MiB. Masami Hiramatsu suggests replacing 2 words, it will make things complex. Futher patch can make such optimization. Kprobe opt on ARM is relatively simpler than kprobe opt on x86 because ARM instruction is always 4 bytes aligned and 4 bytes long. This patch replace probed instruction by a 'b', branch to trampoline code and then calls optimized_callback(). optimized_callback() calls opt_pre_handler() to execute kprobe handler. It also emulate/simulate replaced instruction. When unregistering kprobe, the deferred manner of unoptimizer may leave branch instruction before optimizer is called. Different from x86_64, which only copy the probed insn after optprobe_template_end and reexecute them, this patch call singlestep to emulate/simulate the insn directly. Futher patch can optimize this behavior. v1 -> v2: - Improvement: if replaced instruction is conditional, generate a conditional branch instruction for it; - Introduces RELATIVEJUMP_OPCODES due to ARM kprobe_opcode_t is 4 bytes; - Removes size field in struct arch_optimized_insn; - Use arm_gen_branch() to generate branch instruction; - Remove all recover logic: ARM doesn't use tail buffer, no need recover replaced instructions like x86; - Remove incorrect CONFIG_THUMB checking; - can_optimize() always returns true if address is well aligned; - Improve optimized_callback: using opt_pre_handler(); - Bugfix: correct range checking code and improve comments; - Fix commit message. Signed-off-by: Wang Nan Cc: Masami Hiramatsu Cc: Jon Medhurst (Tixy) Cc: Russell King - ARM Linux --- arch/arm/Kconfig | 1 + arch/arm/include/asm/kprobes.h | 26 +++++ arch/arm/kernel/Makefile | 3 +- arch/arm/kernel/kprobes-opt.c | 257 +++++++++++++++++++++++++++++++++++++++++ 4 files changed, 286 insertions(+), 1 deletion(-) create mode 100644 arch/arm/kernel/kprobes-opt.c diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 290f02ee..2106918 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -57,6 +57,7 @@ config ARM select HAVE_MEMBLOCK select HAVE_MOD_ARCH_SPECIFIC if ARM_UNWIND select HAVE_OPROFILE if (HAVE_PERF_EVENTS) + select HAVE_OPTPROBES if (!THUMB2_KERNEL) select HAVE_PERF_EVENTS select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP diff --git a/arch/arm/include/asm/kprobes.h b/arch/arm/include/asm/kprobes.h index 49fa0df..caacc7c 100644 --- a/arch/arm/include/asm/kprobes.h +++ b/arch/arm/include/asm/kprobes.h @@ -51,5 +51,31 @@ int kprobe_fault_handler(struct pt_regs *regs, unsigned int fsr); int kprobe_exceptions_notify(struct notifier_block *self, unsigned long val, void *data); +/* optinsn template addresses */ +extern __visible kprobe_opcode_t optprobe_template_entry; +extern __visible kprobe_opcode_t optprobe_template_val; +extern __visible kprobe_opcode_t optprobe_template_call; +extern __visible kprobe_opcode_t optprobe_template_end; + +#define MAX_OPTIMIZED_LENGTH (4) +#define MAX_OPTINSN_SIZE \ + (((unsigned long)&optprobe_template_end - \ + (unsigned long)&optprobe_template_entry)) +#define RELATIVEJUMP_SIZE (4) +#define RELATIVEJUMP_OPCODES ((RELATIVEJUMP_SIZE) / sizeof(kprobe_opcode_t)) + +struct arch_optimized_insn { + /* + * copy of the original instructions. + * Different from x86, ARM kprobe_opcode_t is u32. + */ + kprobe_opcode_t copied_insn[RELATIVEJUMP_OPCODES]; + /* detour code buffer */ + kprobe_opcode_t *insn; + /* + * we always copies one instruction on arm32, + * size always be 4, so no size field. + */ +}; #endif /* _ARM_KPROBES_H */ diff --git a/arch/arm/kernel/Makefile b/arch/arm/kernel/Makefile index 38ddd9f..6a38ec1 100644 --- a/arch/arm/kernel/Makefile +++ b/arch/arm/kernel/Makefile @@ -52,11 +52,12 @@ obj-$(CONFIG_FUNCTION_GRAPH_TRACER) += ftrace.o insn.o obj-$(CONFIG_JUMP_LABEL) += jump_label.o insn.o patch.o obj-$(CONFIG_KEXEC) += machine_kexec.o relocate_kernel.o obj-$(CONFIG_UPROBES) += probes.o probes-arm.o uprobes.o uprobes-arm.o -obj-$(CONFIG_KPROBES) += probes.o kprobes.o kprobes-common.o patch.o +obj-$(CONFIG_KPROBES) += probes.o kprobes.o kprobes-common.o patch.o insn.o ifdef CONFIG_THUMB2_KERNEL obj-$(CONFIG_KPROBES) += kprobes-thumb.o probes-thumb.o else obj-$(CONFIG_KPROBES) += kprobes-arm.o probes-arm.o +obj-$(CONFIG_OPTPROBES) += kprobes-opt.o endif obj-$(CONFIG_ARM_KPROBES_TEST) += test-kprobes.o test-kprobes-objs := kprobes-test.o diff --git a/arch/arm/kernel/kprobes-opt.c b/arch/arm/kernel/kprobes-opt.c new file mode 100644 index 0000000..3b721e7 --- /dev/null +++ b/arch/arm/kernel/kprobes-opt.c @@ -0,0 +1,257 @@ +/* + * Kernel Probes Jump Optimization (Optprobes) + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * Copyright (C) IBM Corporation, 2002, 2004 + * Copyright (C) Hitachi Ltd., 2012 + * Copyright (C) Huawei Inc., 2014 + */ + +#include +#include +#include +#include +/* for arm_gen_branch */ +#include "insn.h" +/* for patch_text */ +#include "patch.h" + +asm ( + ".global optprobe_template_entry\n" + "optprobe_template_entry:\n" + " sub sp, sp, #80\n" + " stmia sp, {r0 - r14} \n" + " add r3, sp, #80\n" + " str r3, [sp, #52]\n" + " mrs r4, cpsr\n" + " str r4, [sp, #64]\n" + " mov r1, sp\n" + " ldr r0, 1f\n" + " ldr r2, 2f\n" + " blx r2\n" + " ldr r1, [sp, #64]\n" + " msr cpsr_fs, r1\n" + " ldmia sp, {r0 - r15}\n" + ".global optprobe_template_val\n" + "optprobe_template_val:\n" + "1: nop\n" + ".global optprobe_template_call\n" + "optprobe_template_call:\n" + "2: nop\n" + ".global optprobe_template_end\n" + "optprobe_template_end:\n"); + +#define TMPL_VAL_IDX \ + ((long)&optprobe_template_val - (long)&optprobe_template_entry) +#define TMPL_CALL_IDX \ + ((long)&optprobe_template_call - (long)&optprobe_template_entry) +#define TMPL_END_IDX \ + ((long)&optprobe_template_end - (long)&optprobe_template_entry) + +/* + * ARM can always optimize an instruction when using ARM ISA. + */ +int arch_prepared_optinsn(struct arch_optimized_insn *optinsn) +{ + return 1; +} + +int arch_check_optimized_kprobe(struct optimized_kprobe *op) +{ + int i; + struct kprobe *p; + + for (i = 1; i < RELATIVEJUMP_SIZE; i++) { + p = get_kprobe(op->kp.addr + i); + if (p && !kprobe_disabled(p)) + return -EEXIST; + } + + return 0; +} + +static int can_optimize(unsigned long paddr) +{ + /* we can always optimize a arm instruction */ + return ((paddr & 3) == 0); +} + +/* Free optimized instruction slot */ +static +void __arch_remove_optimized_kprobe(struct optimized_kprobe *op, int dirty) +{ + if (op->optinsn.insn) { + free_optinsn_slot(op->optinsn.insn, dirty); + op->optinsn.insn = NULL; + } +} + +extern void kprobe_handler(struct pt_regs *regs); + +static void +optimized_callback(struct optimized_kprobe *op, struct pt_regs *regs) +{ + unsigned long flags; + struct kprobe *p = &op->kp; + struct kprobe_ctlblk *kcb = get_kprobe_ctlblk(); + + /* Save skipped registers */ + regs->ARM_pc = (unsigned long)op->kp.addr; + regs->ARM_ORIG_r0 = ~0UL; + + local_irq_save(flags); + + if (kprobe_running()) { + kprobes_inc_nmissed_count(&op->kp); + } else { + __this_cpu_write(current_kprobe, &op->kp); + kcb->kprobe_status = KPROBE_HIT_ACTIVE; + opt_pre_handler(&op->kp, regs); + __this_cpu_write(current_kprobe, NULL); + } + + /* In each case, we must singlestep the replaced instruction. */ + op->kp.ainsn.insn_singlestep(p->opcode, &p->ainsn, regs); + + local_irq_restore(flags); +} + +int arch_prepare_optimized_kprobe(struct optimized_kprobe *op) +{ + u8 *buf; + unsigned long rel_chk; + unsigned long val; + + if (!can_optimize((unsigned long)op->kp.addr)) + return -EILSEQ; + + op->optinsn.insn = get_optinsn_slot(); + if (!op->optinsn.insn) + return -ENOMEM; + + /* + * Verify if the address gap is in 32MiB range, because this uses + * a relative jump. + * + * kprobe opt use a 'b' instruction to branch to optinsn.insn. + * According to ARM manual, branch instruction is: + * + * 31 28 27 24 23 0 + * +------+---+---+---+---+----------------+ + * | cond | 1 | 0 | 1 | 0 | imm24 | + * +------+---+---+---+---+----------------+ + * + * imm24 is a signed 24 bits integer. The real branch offset is computed + * by: imm32 = SignExtend(imm24:'00', 32); + * + * So the maximum forward branch should be: + * (0x007fffff << 2) = 0x01fffffc = 0x1fffffc + * The maximum backword branch should be: + * (0xff800000 << 2) = 0xfe000000 = -0x2000000 + * + * We can simply check (rel & 0xfe000003): + * if rel is positive, (rel & 0xfe000000) shoule be 0 + * if rel is negitive, (rel & 0xfe000000) should be 0xfe000000 + * the last '3' is used for alignment checking. + */ + rel_chk = (unsigned long)((long)op->optinsn.insn - + (long)op->kp.addr + 8) & 0xfe000003; + + if ((rel_chk != 0) && (rel_chk != 0xfe000000)) { + __arch_remove_optimized_kprobe(op, 0); + return -ERANGE; + } + + buf = (u8 *)op->optinsn.insn; + + /* Copy arch-dep-instance from template */ + memcpy(buf, &optprobe_template_entry, TMPL_END_IDX); + + /* Set probe information */ + val = (unsigned long)op; + memcpy(buf + TMPL_VAL_IDX, &val, sizeof(val)); + + /* Set probe function call */ + val = (unsigned long)optimized_callback; + memcpy(buf + TMPL_CALL_IDX, &val, sizeof(val)); + return 0; +} + +void arch_optimize_kprobes(struct list_head *oplist) +{ + struct optimized_kprobe *op, *tmp; + + list_for_each_entry_safe(op, tmp, oplist, list) { + unsigned long insn; + WARN_ON(kprobe_disabled(&op->kp)); + + /* + * Backup instructions which will be replaced + * by jump address + */ + memcpy(op->optinsn.copied_insn, op->kp.addr, + RELATIVEJUMP_SIZE); + + insn = arm_gen_branch((unsigned long)op->kp.addr, + (unsigned long)op->optinsn.insn); + BUG_ON(insn == 0); + + /* + * Make it a conditional branch if replaced insn + * is consitional + */ + insn = (op->optinsn.copied_insn[0] & 0xf0000000) | + (insn & 0x0fffffff); + + patch_text(op->kp.addr, insn); + + list_del_init(&op->list); + } + return; +} + +void arch_unoptimize_kprobe(struct optimized_kprobe *op) +{ + arch_arm_kprobe(&op->kp); + return; +} + +/* + * Recover original instructions and breakpoints from relative jumps. + * Caller must call with locking kprobe_mutex. + */ +void arch_unoptimize_kprobes(struct list_head *oplist, + struct list_head *done_list) +{ + struct optimized_kprobe *op, *tmp; + + list_for_each_entry_safe(op, tmp, oplist, list) { + arch_unoptimize_kprobe(op); + list_move(&op->list, done_list); + } +} + +int arch_within_optimized_kprobe(struct optimized_kprobe *op, + unsigned long addr) +{ + return ((unsigned long)op->kp.addr <= addr && + (unsigned long)op->kp.addr + RELATIVEJUMP_SIZE > addr); +} + +void arch_remove_optimized_kprobe(struct optimized_kprobe *op) +{ + __arch_remove_optimized_kprobe(op, 1); +}