From patchwork Wed Dec 27 08:50:33 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 122765 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp1920285qgn; Wed, 27 Dec 2017 00:51:44 -0800 (PST) X-Google-Smtp-Source: ACJfBouKUjseEOBxotv55QHFQp/+1fHZBF/Hf4ljUNIBfiZB7Fgq5YLJyvrKIH4nBxDBTJVE5suM X-Received: by 10.159.207.145 with SMTP id z17mr27598023plo.269.1514364704097; Wed, 27 Dec 2017 00:51:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514364704; cv=none; d=google.com; s=arc-20160816; b=E2g4PsVsqGC8ia9/s2kJ7PtY+wip6WYqH1q8HYE1mcYxRv/g/O8iVWSUq5FoC27Wf/ Evd+0lzzVLZZjZ4NqnxbQbwfrXqripHxH6bPdSwPI8QvXWHeW7hzDU+gVyjGrJwc5KHb U+VakR28UOVVwkqUo8/9oqaqiRWpaigUN6aLXvtm5qgeYFoc1GVReBtPKOwxh1o/LZzt joohd4hj7FHuyn4KPxfl0RuHbokGW5ZXta8V7JRlqo8K4bMGFliJwVVkdjc8Qz+o+K8p oX/NdxlwGTPVfcRJsfbK+6uBsjKonUaxqc9LHQLsUSLTRbmq5q6tnI6RhlgqGgvF4Oy1 hVsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=zBo/6EncMhQE+u7uWHtusHrsjwb/Xhu+WleYCGKp7Zw=; b=zElWAjGbKJvErNldfH350LpsmzZavWYaZeW5i2aQ41tRUI2comiihLs1m/Q6v+FgJZ MPiUTJKF6Qyi0RnV3xwkpDjBEBu2Hd/C45KMtuKeDYqQqJnaF9OHo4AIS+qSDjgvPy5J JmdX2xwTmf0ixSMNpDmKmfNTn7e3YnGp23iYTxRs+MUd3EzAUw5rPZMxwI0gcIlGdzYj GtSgVwEvMltVdgrhARjU5ObrsCkvgf9bhhSDKDAU+kHT+p74bdAS8+g3wQNenYF8n9Pe NUvRk0qKPNVfG+irBqWxOgApr4Tlt09YMkxqNeIDLO+jzwkL3litIbE8gunOTJCAtRYu BYCg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=YDuC1/ee; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j4si24477080pfc.22.2017.12.27.00.51.43; Wed, 27 Dec 2017 00:51:44 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=YDuC1/ee; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751983AbdL0Ivj (ORCPT + 28 others); Wed, 27 Dec 2017 03:51:39 -0500 Received: from mail-wm0-f65.google.com ([74.125.82.65]:46979 "EHLO mail-wm0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751934AbdL0Ive (ORCPT ); Wed, 27 Dec 2017 03:51:34 -0500 Received: by mail-wm0-f65.google.com with SMTP id r78so38448234wme.5 for ; Wed, 27 Dec 2017 00:51:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=zBo/6EncMhQE+u7uWHtusHrsjwb/Xhu+WleYCGKp7Zw=; b=YDuC1/eeZUOl9wiL7YxX5dzXK0PSaQTy3t4NObHTF4xOFkUGsRmXQq8KAG7NRYYRGH yvFc+Kxhj7WKoU4/JjOhY3SOKcgu16yIUO32QBwK2XFsI/VBDg+H0zfroSAdow5nrDgc Pp3jK9/mizaywl6ACAMCQbDUwGaVDmnL3yJEw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=zBo/6EncMhQE+u7uWHtusHrsjwb/Xhu+WleYCGKp7Zw=; b=JQMkRB6wsyljEACsWNqbL3htQZmQxCcJoTBoy6gzj5Sbs5iN1qljJOV4wbMQxiZZOR yRCzAjRPPMSfLrPgizfwBiUZ8f/ftj5d7N3xjb8tRNAfmRPM0aRgH4NG+ImZM5BTsGpg k0Sq+024+BCEujY1SGx4TRAeSR8DdagUVTqNfrBMo0GWB7KNQnzZEFm1/kHlTUs8tYqw PWWl2+4OtwcIhyuSXE0Y0CG1PUMvBCmozo1/NsAyRWN9oeHp7swsEcOcoFICjip0oC8u 499fbCZpXVdgrP/AjOrAjpDEmGBaL+XMJ7RZz/OJacaiEdIwBmxxVOL2sKc64X47RvYR wYzA== X-Gm-Message-State: AKGB3mKRIcU/+7+4TJ2Y3IDQaEuTCtHhdlDg5kZXNAtMvRtggzAMWyze r+SgH/3IOeMvh8tyAbwts815pWsDMyk= X-Received: by 10.28.4.146 with SMTP id 140mr21020116wme.38.1514364692449; Wed, 27 Dec 2017 00:51:32 -0800 (PST) Received: from localhost.localdomain ([105.137.110.132]) by smtp.gmail.com with ESMTPSA id q74sm32677226wmg.22.2017.12.27.00.51.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Dec 2017 00:51:31 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v6 8/8] x86/kernel: jump_table: use relative references Date: Wed, 27 Dec 2017 08:50:33 +0000 Message-Id: <20171227085033.22389-9-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171227085033.22389-1-ard.biesheuvel@linaro.org> References: <20171227085033.22389-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Similar to the arm64 case, 64-bit x86 can benefit from using 32-bit relative references rather than 64-bit absolute ones when emitting struct jump_entry instances. Not only does this reduce the memory footprint of the entries themselves by 50%, it also removes the need for carrying relocation metadata on relocatable builds (i.e., for KASLR) which saves a fair chunk of .init space as well (although the savings are not as dramatic as on arm64) Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/jump_label.h | 35 +++++++----- arch/x86/kernel/jump_label.c | 59 ++++++++++++++------ tools/objtool/special.c | 4 +- 3 files changed, 65 insertions(+), 33 deletions(-) -- 2.11.0 diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h index 009ff2699d07..91c01af96907 100644 --- a/arch/x86/include/asm/jump_label.h +++ b/arch/x86/include/asm/jump_label.h @@ -36,8 +36,8 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran asm_volatile_goto("1:" ".byte " __stringify(STATIC_KEY_INIT_NOP) "\n\t" ".pushsection __jump_table, \"aw\" \n\t" - _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".balign 4\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -52,8 +52,8 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool ".byte 0xe9\n\t .long %l[l_yes] - 2f\n\t" "2:\n\t" ".pushsection __jump_table, \"aw\" \n\t" - _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".balign 4\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -69,19 +69,26 @@ typedef u32 jump_label_t; #endif struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; + s32 code; + s32 target; + s32 key; }; static inline jump_label_t jump_entry_code(const struct jump_entry *entry) { - return entry->code; + return (jump_label_t)&entry->code + entry->code; +} + +static inline jump_label_t jump_entry_target(const struct jump_entry *entry) +{ + return (jump_label_t)&entry->target + entry->target; } static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - return (struct static_key *)((unsigned long)entry->key & ~1UL); + unsigned long key = (unsigned long)&entry->key + entry->key; + + return (struct static_key *)(key & ~1UL); } static inline bool jump_entry_is_branch(const struct jump_entry *entry) @@ -99,7 +106,7 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) entry->code = 0; } -#define jump_label_swap NULL +void jump_label_swap(void *a, void *b, int size); #else /* __ASSEMBLY__ */ @@ -114,8 +121,8 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) .byte STATIC_KEY_INIT_NOP .endif .pushsection __jump_table, "aw" - _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + .balign 4 + .long .Lstatic_jump_\@ - ., \target - ., \key - . .popsection .endm @@ -130,8 +137,8 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) .Lstatic_jump_after_\@: .endif .pushsection __jump_table, "aw" - _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + 1 + .balign 4 + .long .Lstatic_jump_\@ - ., \target - ., \key - . + 1 .popsection .endm diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index e56c95be2808..cc5034b42335 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -52,22 +52,24 @@ static void __jump_label_transform(struct jump_entry *entry, * Jump label is enabled for the first time. * So we expect a default_nop... */ - if (unlikely(memcmp((void *)entry->code, default_nop, 5) - != 0)) - bug_at((void *)entry->code, __LINE__); + if (unlikely(memcmp((void *)jump_entry_code(entry), + default_nop, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } else { /* * ...otherwise expect an ideal_nop. Otherwise * something went horribly wrong. */ - if (unlikely(memcmp((void *)entry->code, ideal_nop, 5) - != 0)) - bug_at((void *)entry->code, __LINE__); + if (unlikely(memcmp((void *)jump_entry_code(entry), + ideal_nop, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } code.jump = 0xe9; - code.offset = entry->target - - (entry->code + JUMP_LABEL_NOP_SIZE); + code.offset = jump_entry_target(entry) - + (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); } else { /* * We are disabling this jump label. If it is not what @@ -76,14 +78,18 @@ static void __jump_label_transform(struct jump_entry *entry, * are converting the default nop to the ideal nop. */ if (init) { - if (unlikely(memcmp((void *)entry->code, default_nop, 5) != 0)) - bug_at((void *)entry->code, __LINE__); + if (unlikely(memcmp((void *)jump_entry_code(entry), + default_nop, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } else { code.jump = 0xe9; - code.offset = entry->target - - (entry->code + JUMP_LABEL_NOP_SIZE); - if (unlikely(memcmp((void *)entry->code, &code, 5) != 0)) - bug_at((void *)entry->code, __LINE__); + code.offset = jump_entry_target(entry) - + (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); + if (unlikely(memcmp((void *)jump_entry_code(entry), + &code, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } memcpy(&code, ideal_nops[NOP_ATOMIC5], JUMP_LABEL_NOP_SIZE); } @@ -97,10 +103,13 @@ static void __jump_label_transform(struct jump_entry *entry, * */ if (poker) - (*poker)((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE); + (*poker)((void *)jump_entry_code(entry), &code, + JUMP_LABEL_NOP_SIZE); else - text_poke_bp((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE, - (void *)entry->code + JUMP_LABEL_NOP_SIZE); + text_poke_bp((void *)jump_entry_code(entry), &code, + JUMP_LABEL_NOP_SIZE, + (void *)jump_entry_code(entry) + + JUMP_LABEL_NOP_SIZE); } void arch_jump_label_transform(struct jump_entry *entry, @@ -140,4 +149,20 @@ __init_or_module void arch_jump_label_transform_static(struct jump_entry *entry, __jump_label_transform(entry, type, text_poke_early, 1); } +void jump_label_swap(void *a, void *b, int size) +{ + long delta = (unsigned long)a - (unsigned long)b; + struct jump_entry *jea = a; + struct jump_entry *jeb = b; + struct jump_entry tmp = *jea; + + jea->code = jeb->code - delta; + jea->target = jeb->target - delta; + jea->key = jeb->key - delta; + + jeb->code = tmp.code + delta; + jeb->target = tmp.target + delta; + jeb->key = tmp.key + delta; +} + #endif diff --git a/tools/objtool/special.c b/tools/objtool/special.c index 84f001d52322..98ae55b39037 100644 --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -30,9 +30,9 @@ #define EX_ORIG_OFFSET 0 #define EX_NEW_OFFSET 4 -#define JUMP_ENTRY_SIZE 24 +#define JUMP_ENTRY_SIZE 12 #define JUMP_ORIG_OFFSET 0 -#define JUMP_NEW_OFFSET 8 +#define JUMP_NEW_OFFSET 4 #define ALT_ENTRY_SIZE 13 #define ALT_ORIG_OFFSET 0