From patchwork Sun Jul 28 20:29:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rong Xu X-Patchwork-Id: 814903 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B1427BAF4 for ; Sun, 28 Jul 2024 20:31:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722198662; cv=none; b=gVsFlEs61ijbHFxNWT3Odn76hw/SVRokVWcq6sX3EVH5kPKdooX4UX9TamSwKoPYnIqIJGUDip8N9W1QJdgdzYWOa7xgLtaHYlCUQkcFTgawL5uE7Z7HZwsPAHkjhvr4LFJhaqtTkn4IiIx4EOesfRCXAF+iOI9tPdl9CH0kjgs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722198662; c=relaxed/simple; bh=beGZIYM3TPtatxrKmLYPAD0/CJjAgMIMbsJ074Ik9tg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=RKGmifxzgOcjGM4Y14LuYUdEC6kxH6V00zfTGD2PdhgmWvAzfbpekeD4Jr9Kn2NjPUdWDX4wsy2KWBU9W+5uBunzyMK/p1wQnZ+neNLmFBgoYKEUV8B/SXmOYc5F+FTEfONdzjr3TlgGmMSTyt/LWak9APxwr0/GtIWZVnMa6Bw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--xur.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Cbw9W6ak; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--xur.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Cbw9W6ak" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e035f7b5976so4592991276.0 for ; Sun, 28 Jul 2024 13:31:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722198659; x=1722803459; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Nypg1QHLbTqa/1p2hSGPQd9/Gf8EHrTdP6hUDec9EdE=; b=Cbw9W6akg0cNIlC5zMjJAlxlLpwYzGB4rcfnqydC9Jze3jMb2XKqKzGywKdmsn/ijN c6MhSTyYHyEAmibBRh9dGqbOGFNum3nck1HFGOVA7duZoxDFns+AxrCKucm4tkyTgn6m K5oHXQhhQKUpCdP93JbAk3kYLeo5iMoJBXN9Y3pakzBjoBT63FrRkzA2GaanOCaXXENX ykkQGAP9vfGLVmDJaQgxg7lJ/Z1gxWfVIwXRrq8pzDMJggfmixLuHZZNFaZLeASdGd9I oiWiiteyOxAjMHTIhX6t1izT/TlPofqhWVMWpGZyvre1tO/+TyFNyfOSed1VXklQ6SqX nsFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722198659; x=1722803459; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Nypg1QHLbTqa/1p2hSGPQd9/Gf8EHrTdP6hUDec9EdE=; b=gOOzOtFzmjqKS/FTkkDMXmD78i+vXUJqSi9sG4vfM/B5FOg2y/8iXdtp1KtZhwA0LP SogH/3BGty45yAUQOt66/YYFHuFqelrVrwDQC1o1rwwej8uEJVOm9EPE7yTNFRDIePZy pIgtE3/8b2IhB/kWzDbF03hHfvvYIQvgi6dzOtJvoYJBKDNqy++n+ou0D1d8dv1K5Nnl UcncN1Y0zN0WIErGCrTkoA8UfqTGWVwg+QZd35Bp96ECHu5inYwYXWoX1OL6HAoNn5D8 poYk28fMahGBUZ8dKqtJGn+S9iz/XMMosoS4xWbMLsVaCLB8/MRVh6MWz+W9D1pCGHWT /XiA== X-Forwarded-Encrypted: i=1; AJvYcCXGNGxd7DIkPLiq35A7w+ppui4kkK6xBLASSB1T7jsIR3VaoTUZ5VWtqsLq9n4WuDCv2sCfvjiq889AR3YFxvtkCy1gc5E1X5+e X-Gm-Message-State: AOJu0YxIYW9lZuDU0YTcM4aXZ7EGdmVGIMk5AQxWZ+nHuJiPVBTEuizp BbDRbk5++SRKPMghETP4s5ebAbsXpjj08F6fhvVqGRgdmXadXeDW13hc6qIbEet/+Q== X-Google-Smtp-Source: AGHT+IGhcfBcAn7qASZpEvCq18PhN0EBXHuUDxWLgxYpaHcb/kQ/zFsB5MAYY4oJLzn0xh4JiWgv4U8= X-Received: from xur.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:2330]) (user=xur job=sendgmr) by 2002:a25:6844:0:b0:e0b:3432:73d4 with SMTP id 3f1490d57ef6-e0b5560e6b3mr58714276.1.1722198659209; Sun, 28 Jul 2024 13:30:59 -0700 (PDT) Date: Sun, 28 Jul 2024 13:29:57 -0700 In-Reply-To: <20240728203001.2551083-1-xur@google.com> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240728203001.2551083-1-xur@google.com> X-Mailer: git-send-email 2.46.0.rc1.232.g9752f9e123-goog Message-ID: <20240728203001.2551083-5-xur@google.com> Subject: [PATCH 4/6] AutoFDO: Enable -ffunction-sections for the AutoFDO build From: Rong Xu To: Rong Xu , Han Shen , Sriraman Tallam , David Li , Jonathan Corbet , Masahiro Yamada , Nathan Chancellor , Nicolas Schier , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Ard Biesheuvel , Arnd Bergmann , Josh Poimboeuf , Peter Zijlstra , Nick Desaulniers , Bill Wendling , Justin Stitt , Vegard Nossum , John Moon , Andrew Morton , Heiko Carstens , Luis Chamberlain , Samuel Holland , Mike Rapoport , "Paul E . McKenney" , Rafael Aquini , Petr Pavlu , Eric DeVolder , Bjorn Helgaas , Randy Dunlap , Benjamin Segall , Breno Leitao , Wei Yang , Brian Gerst , Juergen Gross , Palmer Dabbelt , Alexandre Ghiti , Kees Cook , Sami Tolvanen , Xiao Wang , Jan Kiszka Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kbuild@vger.kernel.org, linux-efi@vger.kernel.org, linux-arch@vger.kernel.org, llvm@lists.linux.dev Enable -ffunction-sections by default for the AutoFDO build. With -ffunction-sections, the compiler places each function in its own section named .text.function_name instead of placing all functions in the .text section. In the AutoFDO build, this allows the linker to utilize profile information to reorganize functions for improved utilization of iCache and iTLB. Co-developed-by: Han Shen Signed-off-by: Han Shen Signed-off-by: Rong Xu Suggested-by: Sriraman Tallam --- include/asm-generic/vmlinux.lds.h | 37 ++++++++++++++++++++++++------- scripts/Makefile.autofdo | 2 +- 2 files changed, 30 insertions(+), 9 deletions(-) diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index f3de66bda293..97c8399e5532 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -95,18 +95,25 @@ * With LTO_CLANG, the linker also splits sections by default, so we need * these macros to combine the sections during the final link. * + * With LTO_CLANG, the linker also splits sections by default, so we need + * these macros to combine the sections during the final link. + * * RODATA_MAIN is not used because existing code already defines .rodata.x * sections to be brought in with rodata. */ -#if defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || defined(CONFIG_LTO_CLANG) +#if defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || defined(CONFIG_LTO_CLANG) || \ +defined(CONFIG_AUTOFDO_CLANG) #define TEXT_MAIN .text .text.[0-9a-zA-Z_]* +#else +#define TEXT_MAIN .text +#endif +#if defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || defined(CONFIG_LTO_CLANG) #define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..L* .data..compoundliteral* .data.$__unnamed_* .data.$L* #define SDATA_MAIN .sdata .sdata.[0-9a-zA-Z_]* #define RODATA_MAIN .rodata .rodata.[0-9a-zA-Z_]* .rodata..L* #define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* .bss..compoundliteral* #define SBSS_MAIN .sbss .sbss.[0-9a-zA-Z_]* #else -#define TEXT_MAIN .text #define DATA_MAIN .data #define SDATA_MAIN .sdata #define RODATA_MAIN .rodata @@ -577,6 +584,20 @@ __cpuidle_text_end = .; \ __noinstr_text_end = .; +#ifdef CONFIG_AUTOFDO_CLANG +#define TEXT_HOT \ + __hot_text_start = .; \ + *(.text.hot .text.hot.*) \ + __hot_text_end = .; +#define TEXT_UNLIKELY \ + __unlikely_text_start = .; \ + *(.text.unlikely .text.unlikely.*) \ + __unlikely_text_end = .; +#else +#define TEXT_HOT *(.text.hot .text.hot.*) +#define TEXT_UNLIKELY *(.text.unlikely .text.unlikely.*) +#endif + /* * .text section. Map to function alignment to avoid address changes * during second ld run in second ld pass when generating System.map @@ -585,13 +606,14 @@ * code elimination or function-section is enabled. Match these symbols * first when in these builds. */ -#if defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || defined(CONFIG_LTO_CLANG) +#if defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || defined(CONFIG_LTO_CLANG) || \ +defined(CONFIG_AUTOFDO_CLANG) #define TEXT_TEXT \ *(.text.asan.* .text.tsan.*) \ *(.text.unknown .text.unknown.*) \ - *(.text.unlikely .text.unlikely.*) \ + TEXT_UNLIKELY \ ALIGN_FUNCTION(); \ - *(.text.hot .text.hot.*) \ + TEXT_HOT \ *(TEXT_MAIN .text.fixup) \ NOINSTR_TEXT \ *(.ref.text) \ @@ -599,9 +621,9 @@ #else #define TEXT_TEXT \ ALIGN_FUNCTION(); \ - *(.text.hot .text.hot.*) \ + TEXT_HOT \ *(TEXT_MAIN .text.fixup) \ - *(.text.unlikely .text.unlikely.*) \ + TEXT_UNLIKELY \ *(.text.unknown .text.unknown.*) \ NOINSTR_TEXT \ *(.ref.text) \ @@ -609,7 +631,6 @@ MEM_KEEP(init.text*) #endif - /* sched.text is aling to function alignment to secure we have same * address even at second ld pass when generating System.map */ #define SCHED_TEXT \ diff --git a/scripts/Makefile.autofdo b/scripts/Makefile.autofdo index 882fa7e801ba..f765bd9e81d7 100644 --- a/scripts/Makefile.autofdo +++ b/scripts/Makefile.autofdo @@ -5,7 +5,7 @@ CFLAGS_AUTOFDO_CLANG := -fdebug-info-for-profiling -mllvm -enable-fs-discriminator=true -mllvm -improved-fs-discriminator=true ifdef CLANG_AUTOFDO_PROFILE -CFLAGS_AUTOFDO_CLANG += -fprofile-sample-use=$(CLANG_AUTOFDO_PROFILE) +CFLAGS_AUTOFDO_CLANG += -fprofile-sample-use=$(CLANG_AUTOFDO_PROFILE) -ffunction-sections endif ifdef CONFIG_LTO_CLANG