From patchwork Sun Mar 11 12:38:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 131344 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp3055595lja; Sun, 11 Mar 2018 05:39:20 -0700 (PDT) X-Google-Smtp-Source: AG47ELvVAF+8hCJ62m9R3Os8kzRBQoExDkPynlzdZQ2t1FLLaKLPaLTGL8g0Dh10Y0qhawMoD9L6 X-Received: by 2002:a17:902:40a:: with SMTP id 10-v6mr4853026ple.245.1520771959837; Sun, 11 Mar 2018 05:39:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1520771959; cv=none; d=google.com; s=arc-20160816; b=jsHnpI3bqC4AGQgWmfZIpaFoCAF6UOcugpB9IuWh5iVrHO/AauIJ0RnN3XOgmCY+TO l7Zd90WFptsGW/FnMMi0c0z7zIti78kHnhWDzYymU5Hwm9m3uw/G6gwxeUR2WLVH2T+Q xEDWLE+ScYPGZ/NUKMuNRubtHBn1RTXVdaUdC6xLK8eKxOIpiBA2Y2cIXCYFEArm8UuX VCLHqCFW43mURC8mhih97PdMjKQC+B2nojy/BmiSq4EK/+Ns8+ob2bQC1tL6KQp9DAs7 67ofo6kt3D9b6R0f85f0InrxeyllyP5RcyIGeEmXpQ7fDrtEes2gdDdM2X5wN3VRL+40 tgOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=cIBQXEJp/WJqYgkyL2azlB0OGK67eNhC55dbvil4JQQ=; b=NN2t6lE7Smt4Lj3uX+un9iTnFbw86REWAC3lco/nU5EKZYO8R1wDgY5m0TD9fLtl6o l8GLl9/9A5nsZc/YYH0UaN0L+mP7LI3ODpDBJqFfxI6KWieSIOhrbwv3ehy1qpToZRTV ePfgtSs8PtKuvKlUSAg8Cl3xAZDMwgUaS933qIG8vqvoeMLwwwjs8KVY1OrzBDJex3WK sh9U8HdTeKcSc9H96sxak+3ABsvPz0DGilMPf1zdT895pPyRJF/ib9pDLzaaEV0iaMOx yXdvBHgNCoCJd/BpMSBpPz/E9Cdn/VjJ1F+8J1RUeZS64ay277xkK6kCG4Pf7QLsjBBS ULJg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=EuJsZEJ0; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id d5si4238997pfl.224.2018.03.11.05.39.19; Sun, 11 Mar 2018 05:39:19 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=EuJsZEJ0; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932266AbeCKMjN (ORCPT + 28 others); Sun, 11 Mar 2018 08:39:13 -0400 Received: from mail-wm0-f66.google.com ([74.125.82.66]:34290 "EHLO mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932126AbeCKMjH (ORCPT ); Sun, 11 Mar 2018 08:39:07 -0400 Received: by mail-wm0-f66.google.com with SMTP id a20so9510889wmd.1 for ; Sun, 11 Mar 2018 05:39:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=cIBQXEJp/WJqYgkyL2azlB0OGK67eNhC55dbvil4JQQ=; b=EuJsZEJ0OWBxCIzipBL3jwFr+sdpqkmXwyJU9csqFhrK/CKa4Sqmy0lB0LJVS5dmQX Usl8lfNNjbt4u9g6z0bHa1NlnPIO5cHF0Cy9DQDvXPyRKdvt+XQsXiD/kb4O7BDtykEf aaQTsALS19hjJdFXT1K//lbiHCDaNNXfv1vQ4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=cIBQXEJp/WJqYgkyL2azlB0OGK67eNhC55dbvil4JQQ=; b=Dg7J21uQJK4dyi/0ABNOcU0ILDR8e/5XHTyQimkO+P6Ixn7JBDqlJxRUB7or7hWsKg Pwrc/gilEBJjOJD28gEzEADYZFC3aqDbB84xNsdXgWDd3k3xHrt4YzvsNho8QncC7++l r5buMfJzflIiXHSBcj3tmprgE2sjKg4aWgFT4hzgjDvYZRzgnBhSmRHyS+BsXYVkjOR0 /3WlMrYEmnIpfZFtm3ikqPreGvm9UVapdUm576oB+3IqMwWoD2Mp1e+1MzA164IQaYGA iXkxcGNiah6Fc5pB3bxLwEEPK14EzTeH65sasSzU208ciOR0+W1ZE8LTO7kz9NY3NJeu 0WyQ== X-Gm-Message-State: AElRT7Gmseh+ryBwalwd16W9KHIeFTDx+TN60qt6i80K6L1AANgCEVz8 wrKS0GcVvvwKPz3OBkVrgUT5ZA== X-Received: by 10.28.108.6 with SMTP id h6mr3057680wmc.6.1520771946296; Sun, 11 Mar 2018 05:39:06 -0700 (PDT) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id h188sm2569103wmf.37.2018.03.11.05.39.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 11 Mar 2018 05:39:05 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Arnd Bergmann , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, Ingo Molnar Subject: [PATCH v8 3/6] module: use relative references for __ksymtab entries Date: Sun, 11 Mar 2018 12:38:12 +0000 Message-Id: <20180311123815.17916-4-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180311123815.17916-1-ard.biesheuvel@linaro.org> References: <20180311123815.17916-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org An ordinary arm64 defconfig build has ~64 KB worth of __ksymtab entries, each consisting of two 64-bit fields containing absolute references, to the symbol itself and to a char array containing its name, respectively. When we build the same configuration with KASLR enabled, we end up with an additional ~192 KB of relocations in the .init section, i.e., one 24 byte entry for each absolute reference, which all need to be processed at boot time. Given how the struct kernel_symbol that describes each entry is completely local to module.c (except for the references emitted by EXPORT_SYMBOL() itself), we can easily modify it to contain two 32-bit relative references instead. This reduces the size of the __ksymtab section by 50% for all 64-bit architectures, and gets rid of the runtime relocations entirely for architectures implementing KASLR, either via standard PIE linking (arm64) or using custom host tools (x86). Note that the binary search involving __ksymtab contents relies on each section being sorted by symbol name. This is implemented based on the input section names, not the names in the ksymtab entries, so this patch does not interfere with that. Given that the use of place-relative relocations requires support both in the toolchain and in the module loader, we cannot enable this feature for all architectures. So make it dependent on whether CONFIG_HAVE_ARCH_PREL32_RELOCATIONS is defined. Cc: Arnd Bergmann Cc: Andrew Morton Cc: Ingo Molnar Cc: Kees Cook Cc: Thomas Garnier Cc: Nicolas Pitre Acked-by: Jessica Yu Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/Kbuild | 1 + arch/x86/include/asm/export.h | 5 --- include/asm-generic/export.h | 12 ++++- include/linux/compiler.h | 19 ++++++++ include/linux/export.h | 46 +++++++++++++++----- kernel/module.c | 32 +++++++++++--- 6 files changed, 91 insertions(+), 24 deletions(-) -- 2.15.1 diff --git a/arch/x86/include/asm/Kbuild b/arch/x86/include/asm/Kbuild index de690c2d2e33..a0ab9ab61c75 100644 --- a/arch/x86/include/asm/Kbuild +++ b/arch/x86/include/asm/Kbuild @@ -8,5 +8,6 @@ generated-y += xen-hypercalls.h generic-y += dma-contiguous.h generic-y += early_ioremap.h +generic-y += export.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h diff --git a/arch/x86/include/asm/export.h b/arch/x86/include/asm/export.h deleted file mode 100644 index 2a51d66689c5..000000000000 --- a/arch/x86/include/asm/export.h +++ /dev/null @@ -1,5 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifdef CONFIG_64BIT -#define KSYM_ALIGN 16 -#endif -#include diff --git a/include/asm-generic/export.h b/include/asm-generic/export.h index 719db1968d81..97ce606459ae 100644 --- a/include/asm-generic/export.h +++ b/include/asm-generic/export.h @@ -5,12 +5,10 @@ #define KSYM_FUNC(x) x #endif #ifdef CONFIG_64BIT -#define __put .quad #ifndef KSYM_ALIGN #define KSYM_ALIGN 8 #endif #else -#define __put .long #ifndef KSYM_ALIGN #define KSYM_ALIGN 4 #endif @@ -25,6 +23,16 @@ #define KSYM(name) name #endif +.macro __put, val, name +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + .long \val - ., \name - . +#elif defined(CONFIG_64BIT) + .quad \val, \name +#else + .long \val, \name +#endif +.endm + /* * note on .section use: @progbits vs %progbits nastiness doesn't matter, * since we immediately emit into those sections anyway. diff --git a/include/linux/compiler.h b/include/linux/compiler.h index ab4711c63601..0a9328ea9dbd 100644 --- a/include/linux/compiler.h +++ b/include/linux/compiler.h @@ -280,6 +280,25 @@ unsigned long read_word_at_a_time(const void *addr) #endif /* __KERNEL__ */ +/* + * Force the compiler to emit 'sym' as a symbol, so that we can reference + * it from inline assembler. Necessary in case 'sym' could be inlined + * otherwise, or eliminated entirely due to lack of references that are + * visible to the compiler. + */ +#define __ADDRESSABLE(sym) \ + static void * const __attribute__((section(".discard"), used)) \ + __PASTE(__addressable_##sym, __LINE__) = (void *)&sym; + +/** + * offset_to_ptr - convert a relative memory offset to an absolute pointer + * @off: the address of the 32-bit offset value + */ +static inline void *offset_to_ptr(const int *off) +{ + return (void *)((unsigned long)off + *off); +} + #endif /* __ASSEMBLY__ */ #ifndef __optimize diff --git a/include/linux/export.h b/include/linux/export.h index 25005b55b079..04c78e6bfec9 100644 --- a/include/linux/export.h +++ b/include/linux/export.h @@ -24,12 +24,6 @@ #define VMLINUX_SYMBOL_STR(x) __VMLINUX_SYMBOL_STR(x) #ifndef __ASSEMBLY__ -struct kernel_symbol -{ - unsigned long value; - const char *name; -}; - #ifdef MODULE extern struct module __this_module; #define THIS_MODULE (&__this_module) @@ -60,17 +54,47 @@ extern struct module __this_module; #define __CRC_SYMBOL(sym, sec) #endif +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS +#include +/* + * Emit the ksymtab entry as a pair of relative references: this reduces + * the size by half on 64-bit architectures, and eliminates the need for + * absolute relocations that require runtime processing on relocatable + * kernels. + */ +#define __KSYMTAB_ENTRY(sym, sec) \ + __ADDRESSABLE(sym) \ + asm(" .section \"___ksymtab" sec "+" #sym "\", \"a\" \n" \ + " .balign 8 \n" \ + VMLINUX_SYMBOL_STR(__ksymtab_##sym) ": \n" \ + " .long " VMLINUX_SYMBOL_STR(sym) "- . \n" \ + " .long " VMLINUX_SYMBOL_STR(__kstrtab_##sym) "- .\n" \ + " .previous \n") + +struct kernel_symbol { + int value_offset; + int name_offset; +}; +#else +#define __KSYMTAB_ENTRY(sym, sec) \ + static const struct kernel_symbol __ksymtab_##sym \ + __attribute__((section("___ksymtab" sec "+" #sym), used)) \ + = { (unsigned long)&sym, __kstrtab_##sym } + +struct kernel_symbol { + unsigned long value; + const char *name; +}; +#endif + /* For every exported symbol, place a struct in the __ksymtab section */ #define ___EXPORT_SYMBOL(sym, sec) \ extern typeof(sym) sym; \ __CRC_SYMBOL(sym, sec) \ static const char __kstrtab_##sym[] \ - __attribute__((section("__ksymtab_strings"), aligned(1))) \ + __attribute__((section("__ksymtab_strings"), used, aligned(1))) \ = VMLINUX_SYMBOL_STR(sym); \ - static const struct kernel_symbol __ksymtab_##sym \ - __used \ - __attribute__((section("___ksymtab" sec "+" #sym), used)) \ - = { (unsigned long)&sym, __kstrtab_##sym } + __KSYMTAB_ENTRY(sym, sec) #if defined(__DISABLE_EXPORTS) diff --git a/kernel/module.c b/kernel/module.c index ad2d420024f6..b4782cfbb79b 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -549,12 +549,30 @@ static bool check_symbol(const struct symsearch *syms, return true; } +static unsigned long kernel_symbol_value(const struct kernel_symbol *sym) +{ +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + return (unsigned long)offset_to_ptr(&sym->value_offset); +#else + return sym->value; +#endif +} + +static const char *kernel_symbol_name(const struct kernel_symbol *sym) +{ +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + return offset_to_ptr(&sym->name_offset); +#else + return sym->name; +#endif +} + static int cmp_name(const void *va, const void *vb) { const char *a; const struct kernel_symbol *b; a = va; b = vb; - return strcmp(a, b->name); + return strcmp(a, kernel_symbol_name(b)); } static bool find_symbol_in_section(const struct symsearch *syms, @@ -2198,7 +2216,7 @@ void *__symbol_get(const char *symbol) sym = NULL; preempt_enable(); - return sym ? (void *)sym->value : NULL; + return sym ? (void *)kernel_symbol_value(sym) : NULL; } EXPORT_SYMBOL_GPL(__symbol_get); @@ -2228,10 +2246,12 @@ static int verify_export_symbols(struct module *mod) for (i = 0; i < ARRAY_SIZE(arr); i++) { for (s = arr[i].sym; s < arr[i].sym + arr[i].num; s++) { - if (find_symbol(s->name, &owner, NULL, true, false)) { + if (find_symbol(kernel_symbol_name(s), &owner, NULL, + true, false)) { pr_err("%s: exports duplicate symbol %s" " (owned by %s)\n", - mod->name, s->name, module_name(owner)); + mod->name, kernel_symbol_name(s), + module_name(owner)); return -ENOEXEC; } } @@ -2280,7 +2300,7 @@ static int simplify_symbols(struct module *mod, const struct load_info *info) ksym = resolve_symbol_wait(mod, info, name); /* Ok if resolved. */ if (ksym && !IS_ERR(ksym)) { - sym[i].st_value = ksym->value; + sym[i].st_value = kernel_symbol_value(ksym); break; } @@ -2540,7 +2560,7 @@ static int is_exported(const char *name, unsigned long value, ks = lookup_symbol(name, __start___ksymtab, __stop___ksymtab); else ks = lookup_symbol(name, mod->syms, mod->syms + mod->num_syms); - return ks != NULL && ks->value == value; + return ks != NULL && kernel_symbol_value(ks) == value; } /* As per nm */