From patchwork Mon Apr 20 20:19:46 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Long X-Patchwork-Id: 47373 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-lb0-f200.google.com (mail-lb0-f200.google.com [209.85.217.200]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 2673E2121F for ; Mon, 20 Apr 2015 20:20:12 +0000 (UTC) Received: by lbdc7 with SMTP id c7sf40062755lbd.2 for ; Mon, 20 Apr 2015 13:20:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:delivered-to:from:to:cc:subject :date:message-id:in-reply-to:references:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :list-post:list-help:list-archive:list-unsubscribe; bh=ZfkbW50pPImOCqFQOn2Q7D5xECvw7RJAY6wLDLZ0dfc=; b=FBsabJLI2w9mr+HkQtwhs9p8KEfe569a2fTF/rgADaj3hgV/01al/AmpfS/Myvs4Ov 4HzfbuA8Yu1/6OX/zayF2Y2eZBj/SBncv8M0hpPSRvjWr9mUjr4NOe92Kyvwd5oSlM/S /TRICiD9GqA4ov2ah/Ckf7LU3KRvagMiwixCwYD+aSg+rUa3pbig9wQbI4xkUqaszpcE y1729foH4cWdSAWr9bWzjpoJCnZgT9qFu2vnB8wZbDxYwcSs9H0FCR2zob3bXXyDquMd kPr2ZCJYjdKqGdCx2k4lxx/Mjtv0oW9tGQQX6itiOew8s1d1vbgEOZmlHzVAE/n7KWEl 60oA== X-Gm-Message-State: ALoCoQmcvdN/kNoGVJxnbN3i+7oKGuUl24epKcktMmEULzjhxMaErrYDuYtJa0PLUu4eyTaiTJHa X-Received: by 10.112.166.137 with SMTP id zg9mr7896407lbb.11.1429561211123; Mon, 20 Apr 2015 13:20:11 -0700 (PDT) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.205.37 with SMTP id ld5ls882402lac.52.gmail; Mon, 20 Apr 2015 13:20:10 -0700 (PDT) X-Received: by 10.112.180.201 with SMTP id dq9mr17489644lbc.78.1429561210912; Mon, 20 Apr 2015 13:20:10 -0700 (PDT) Received: from mail-la0-f50.google.com (mail-la0-f50.google.com. [209.85.215.50]) by mx.google.com with ESMTPS id rl7si15744455lac.80.2015.04.20.13.20.10 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 20 Apr 2015 13:20:10 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.50 as permitted sender) client-ip=209.85.215.50; Received: by lagv1 with SMTP id v1so135851352lag.3 for ; Mon, 20 Apr 2015 13:20:10 -0700 (PDT) X-Received: by 10.152.8.78 with SMTP id p14mr17229140laa.19.1429561210816; Mon, 20 Apr 2015 13:20:10 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patches@linaro.org Received: by 10.112.67.65 with SMTP id l1csp232782lbt; Mon, 20 Apr 2015 13:20:10 -0700 (PDT) X-Received: by 10.55.21.31 with SMTP id f31mr31839008qkh.95.1429561198209; Mon, 20 Apr 2015 13:19:58 -0700 (PDT) Received: from mail-qg0-f53.google.com (mail-qg0-f53.google.com. [209.85.192.53]) by mx.google.com with ESMTPS id 17si20635332qhh.38.2015.04.20.13.19.57 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 20 Apr 2015 13:19:58 -0700 (PDT) Received-SPF: pass (google.com: domain of dave.long@linaro.org designates 209.85.192.53 as permitted sender) client-ip=209.85.192.53; Received: by qgeb100 with SMTP id b100so58527551qge.3 for ; Mon, 20 Apr 2015 13:19:57 -0700 (PDT) X-Received: by 10.140.80.170 with SMTP id c39mr19190976qgd.55.1429561197608; Mon, 20 Apr 2015 13:19:57 -0700 (PDT) Received: from localhost.localdomain (pool-72-71-243-249.cncdnh.fast00.myfairpoint.net. [72.71.243.249]) by mx.google.com with ESMTPSA id 77sm15240386qhf.44.2015.04.20.13.19.55 (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 20 Apr 2015 13:19:56 -0700 (PDT) From: David Long To: linux-arm-kernel@lists.infradead.org, Russell King Cc: sandeepa.s.prabhu@gmail.com, William Cohen , Steve Capper , Catalin Marinas , Will Deacon , "Jon Medhurst (Tixy)" , Masami Hiramatsu , Ananth N Mavinakayanahalli , Anil S Keshavamurthy , , linux-kernel@vger.kernel.org Subject: [PATCH v6 5/6] arm64: Add kernel return probes support (kretprobes) Date: Mon, 20 Apr 2015 16:19:46 -0400 Message-Id: <1429561187-3661-6-git-send-email-dave.long@linaro.org> X-Mailer: git-send-email 1.8.1.2 In-Reply-To: <1429561187-3661-1-git-send-email-dave.long@linaro.org> References: <1429561187-3661-1-git-send-email-dave.long@linaro.org> X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: dave.long@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.50 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Precedence: list Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org List-ID: X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , From: Sandeepa Prabhu AArch64 ISA does not have instructions to pop the PC register value from the stack(like ARM v7 has ldmia {...,pc}) without using one of the general purpose registers. This means return probes cannot return to the actual return address directly without modifying register context, and without trapping into debug exception. So, like many other architectures, we prepare a global routine with NOPs which serve as a trampoline to hack away the function return address by placing an extra kprobe on the trampoline entry. The pre-handler of this special 'trampoline' kprobe executes the return probe handler functions and restores original return address in ELR_EL1. This way the saved pt_regs still hold the original register context to be carried back to the probed kernel function. Signed-off-by: Sandeepa Prabhu Signed-off-by: David A. Long --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/kprobes.h | 1 + arch/arm64/kernel/kprobes.c | 112 ++++++++++++++++++++++++++++++++++++++- 3 files changed, 113 insertions(+), 1 deletion(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 8e06a03..84f3c9e 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -69,6 +69,7 @@ config ARM64 select HAVE_RCU_TABLE_FREE select HAVE_SYSCALL_TRACEPOINTS select HAVE_KPROBES + select HAVE_KRETPROBES if HAVE_KPROBES select IRQ_DOMAIN select MODULES_USE_ELF_RELA select NO_BOOTMEM diff --git a/arch/arm64/include/asm/kprobes.h b/arch/arm64/include/asm/kprobes.h index af31c4d..d081f49 100644 --- a/arch/arm64/include/asm/kprobes.h +++ b/arch/arm64/include/asm/kprobes.h @@ -58,5 +58,6 @@ int kprobe_exceptions_notify(struct notifier_block *self, unsigned long val, void *data); int kprobe_breakpoint_handler(struct pt_regs *regs, unsigned int esr); int kprobe_single_step_handler(struct pt_regs *regs, unsigned int esr); +void kretprobe_trampoline(void); #endif /* _ARM_KPROBES_H */ diff --git a/arch/arm64/kernel/kprobes.c b/arch/arm64/kernel/kprobes.c index 6255814..2b3ef17 100644 --- a/arch/arm64/kernel/kprobes.c +++ b/arch/arm64/kernel/kprobes.c @@ -560,7 +560,117 @@ int __kprobes longjmp_break_handler(struct kprobe *p, struct pt_regs *regs) return 0; } +/* + * Kretprobes: kernel return probes handling + * + * AArch64 mode does not support popping the PC value from the + * stack like on ARM 32-bit (ldmia {..,pc}), so atleast one + * register need to be used to achieve branching/return. + * It means return probes cannot return back to the original + * return address directly without modifying the register context. + * + * So like other architectures, we prepare a global routine + * with NOPs, which serve as trampoline address that hack away the + * function return, with the exact register context. + * Placing a kprobe on trampoline routine entry will trap again to + * execute return probe handlers and restore original return address + * in ELR_EL1, this way saved pt_regs still hold the original + * register values to be carried back to the caller. + */ +static void __used kretprobe_trampoline_holder(void) +{ + asm volatile (".global kretprobe_trampoline\n" + "kretprobe_trampoline:\n" + "NOP\n\t" + "NOP\n\t"); +} + +static int __kprobes +trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) +{ + struct kretprobe_instance *ri = NULL; + struct hlist_head *head, empty_rp; + struct hlist_node *tmp; + unsigned long flags, orig_ret_addr = 0; + unsigned long trampoline_address = + (unsigned long)&kretprobe_trampoline; + + INIT_HLIST_HEAD(&empty_rp); + kretprobe_hash_lock(current, &head, &flags); + + /* + * It is possible to have multiple instances associated with a given + * task either because multiple functions in the call path have + * a return probe installed on them, and/or more than one return + * probe was registered for a target function. + * + * We can handle this because: + * - instances are always inserted at the head of the list + * - when multiple return probes are registered for the same + * function, the first instance's ret_addr will point to the + * real return address, and all the rest will point to + * kretprobe_trampoline + */ + hlist_for_each_entry_safe(ri, tmp, head, hlist) { + if (ri->task != current) + /* another task is sharing our hash bucket */ + continue; + + if (ri->rp && ri->rp->handler) { + __this_cpu_write(current_kprobe, &ri->rp->kp); + get_kprobe_ctlblk()->kprobe_status = KPROBE_HIT_ACTIVE; + ri->rp->handler(ri, regs); + __this_cpu_write(current_kprobe, NULL); + } + + orig_ret_addr = (unsigned long)ri->ret_addr; + recycle_rp_inst(ri, &empty_rp); + + if (orig_ret_addr != trampoline_address) + /* + * This is the real return address. Any other + * instances associated with this task are for + * other calls deeper on the call stack + */ + break; + } + + kretprobe_assert(ri, orig_ret_addr, trampoline_address); + /* restore the original return address */ + instruction_pointer(regs) = orig_ret_addr; + reset_current_kprobe(); + kretprobe_hash_unlock(current, &flags); + + hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) { + hlist_del(&ri->hlist); + kfree(ri); + } + + /* return 1 so that post handlers not called */ + return 1; +} + +void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, + struct pt_regs *regs) +{ + ri->ret_addr = (kprobe_opcode_t *)regs->regs[30]; + + /* replace return addr (x30) with trampoline */ + regs->regs[30] = (long)&kretprobe_trampoline; +} + +static struct kprobe trampoline = { + .addr = (kprobe_opcode_t *) &kretprobe_trampoline, + .pre_handler = trampoline_probe_handler +}; + +int __kprobes arch_trampoline_kprobe(struct kprobe *p) +{ + return p->addr == (kprobe_opcode_t *) &kretprobe_trampoline; +} + int __init arch_init_kprobes(void) { - return 0; + /* register trampoline for kret probe */ + return register_kprobe(&trampoline); }