From patchwork Mon Nov 6 20:25:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella X-Patchwork-Id: 741462 Delivered-To: patch@linaro.org Received: by 2002:adf:fd90:0:b0:32d:baff:b0ca with SMTP id d16csp1294733wrr; Mon, 6 Nov 2023 12:27:29 -0800 (PST) X-Google-Smtp-Source: AGHT+IHeJmTaEXxd8Tayyrn83EZncXyqzdfIWToxaV3kinobzQBl6VFfh08s6iDLrVgZ92jf92Os X-Received: by 2002:a54:4799:0:b0:3a7:82e8:8fd1 with SMTP id o25-20020a544799000000b003a782e88fd1mr30068333oic.20.1699302449568; Mon, 06 Nov 2023 12:27:29 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1699302449; cv=pass; d=google.com; s=arc-20160816; b=on8jdqNxoAhLWvzvVehwdYWfIxbXJqfLNZKc1ZKYexsdHKa20i0YTN4nGVWTm+A6PN 6ygyVEyrkngO1Gf+wG//6uuHY2kaQovDVTJMJHwSSiVnnDxGE7DlHnlY2kVQWY0lUxgw k8dOB9ZpNJ4nefIwIIV6C017juv2qxCR6wNQdeg8HBzckTGajN4PARRoAEaooL2mFtSp xaltC0CWIs7MewKDi30zskGRtKWCz0woA8TYNwuWUdFE867Mx19/JAfRwW/LfF5EqHls 4rNWF64IZ25MIokNi1WE7eCyT3OK8Pi9A9W8wOu/TvrjX8YCcGDHUigjF4GY0YbpZOP2 BJzw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature:arc-filter:dmarc-filter:delivered-to; bh=mwKBe68nGz40YXHE5GIXD0G78lQPmNvQ11oO/WZQghs=; fh=sF2t3pyYcPOK+OvSTe5TZdX8fUNw2wsg3A//CY2PtUw=; b=OzXQcvUMizBXOGPtKZI9EtwhhlqhIGhNHPdQumP6MPPvKOz5IeVhDoATfvJJ43J9Hc fHlBVptVQUnGIT4nbVHu9phoQn5iUzafv7fq54uQgwkrO4FT8nWBrlG6zyaM3+77Z4b5 oBZsZf6Ja7wvO0GLYCgMgOKxZYzuCK101Ndr+aYXGtwX6c2pimkjBt3nIAr0bH9B5mTM MGEiKHCmuQYqYJYZIwFcK3nokvAEvDhKB/1M07g1E+JHUk5Nol6RE7FPDxNJTRaBE7d1 NAG5CGmxyEdPz76E2JLcixP1/VeWgQT0btJ3zOchiW9WQWNHr8nGnkByR/cxzp9yeUAO lkwQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Wk9eoYnq; arc=pass (i=1); spf=pass (google.com: domain of libc-alpha-bounces+patch=linaro.org@sourceware.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="libc-alpha-bounces+patch=linaro.org@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from server2.sourceware.org (server2.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id d3-20020a05620a204300b0077767d37ffbsi6048988qka.782.2023.11.06.12.27.29 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Nov 2023 12:27:29 -0800 (PST) Received-SPF: pass (google.com: domain of libc-alpha-bounces+patch=linaro.org@sourceware.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Wk9eoYnq; arc=pass (i=1); spf=pass (google.com: domain of libc-alpha-bounces+patch=linaro.org@sourceware.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="libc-alpha-bounces+patch=linaro.org@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 3BE3E38555A0 for ; Mon, 6 Nov 2023 20:27:29 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yw1-x112f.google.com (mail-yw1-x112f.google.com [IPv6:2607:f8b0:4864:20::112f]) by sourceware.org (Postfix) with ESMTPS id C6150385783F for ; Mon, 6 Nov 2023 20:26:16 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org C6150385783F Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=linaro.org ARC-Filter: OpenARC Filter v1.0.0 sourceware.org C6150385783F Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::112f ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1699302379; cv=none; b=il8DQK9m5kk3DVQQ93LU8ZuLJpNRx2fKYSLietQiMk/ye7eEL7drvI+ibzAUG1iXt5Os+ykoGYI/ZLRURgCd5sr3ZMP1/ReiZk6zSXQjeJArO9O6qifLHGhSAbvQ7oxg1/QKGOgC+DOLVgRknY395RYp9C+p5xXTW+ahpJkUeFk= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1699302379; c=relaxed/simple; bh=sqh0rRH1d8vEuSpJWMlgry+ZPLL6nhduPLTCPhiW/EQ=; h=DKIM-Signature:From:To:Subject:Date:Message-Id:MIME-Version; b=fy5Q4/F2lxXFlZvbZTCXel2OzBS6MUWpnh5H0V4gBbbfRFG6nKBAXqP7tdp/msu3ZtVlfqVhbfPqWj9u4lrchlpMNayrYcKVEE+coU2g8meJ2OEU5T8bEo5JI1Kowv5xbS6wRcfrgwHFaPtUZJzTY8BT4Gtn23tvahta6WW2QqI= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-yw1-x112f.google.com with SMTP id 00721157ae682-5b499b18b28so59299067b3.0 for ; Mon, 06 Nov 2023 12:26:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1699302375; x=1699907175; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mwKBe68nGz40YXHE5GIXD0G78lQPmNvQ11oO/WZQghs=; b=Wk9eoYnqPvKcTBDW5lLtTdMo047hk4fenUd1w1qyBEaxWooiUpfsvfXbbmx6DGfWNW mvLrMUrMw/Q7YojQ/pnNUrmmkYzNJdrMpbOoqzLdB9KZaXssuvWCP9qbvfRmUxCXnGQZ 2KK2vYfS1u1ljsui42lk57Uoc8zn1FP8BEJp/fQ4NKd/S/UKoK5KSDXxEtqz/M4uTr76 kfzNcHga8gY1KcY+ecUAm6rMEH0rjufu1qvRGZLHtAMfpW9VAA0M/ADjsXt0ASL2w27E C+DmY/CgpRzN+cPOoqmzylGDVBHCMEDZAMKY/mVgtlTUT+9kXTHGa47iGEcFtskSY3gP SmoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699302375; x=1699907175; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mwKBe68nGz40YXHE5GIXD0G78lQPmNvQ11oO/WZQghs=; b=VZO9eUjN+Oamh3kqtwPBnZONH061QJb2Gif8Xs2xl//zgfXzyTXnMkuGxLFZmS6wLj cx04vBlp/opW2HzCezanF0LyRPV7KAa2FnsUpTs6etxvwFHjjdu0xRYMyBVuGuzcIH62 bBXvywfCEwORoUSoipZ7qPdi76fP0M4e+I4Lp7KMFxnIhcf1NzboWO134BkMP4Maeovp cGVvipBS2hAMMZaKMJbfvlX7hgPZJfl3wQNC0vVMVRck+TexQXGa53bca8zbknxdUiMv +gp5yjC2ELuHIsx623UasNT9FJZCgzy0YP6K+T9tiXs7bj5gCBm5us24ZKVyWUADCAD2 eYyg== X-Gm-Message-State: AOJu0Yzlj7tWx1GHzTvl0wxA6rSfhpQ6pIwp1+M+fohwU3W/GJ5v/dne opXzulgjwVtrgb5s9+xhop9dfia6XN0g06SZH7el5A== X-Received: by 2002:a0d:e681:0:b0:589:8b55:fe09 with SMTP id p123-20020a0de681000000b005898b55fe09mr10839511ywe.50.1699302375524; Mon, 06 Nov 2023 12:26:15 -0800 (PST) Received: from mandiga.. ([2804:1b3:a7c0:a715:c1a0:7281:6384:2ee9]) by smtp.gmail.com with ESMTPSA id ci7-20020a05690c0a8700b005a7b8fddfedsm4707154ywb.41.2023.11.06.12.26.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Nov 2023 12:26:14 -0800 (PST) From: Adhemerval Zanella To: libc-alpha@sourceware.org, Siddhesh Poyarekar Cc: Noah Goldstein Subject: [PATCH v3 09/19] x86: Use dl-symbol-redir-ifunc.h on cpu-tunables Date: Mon, 6 Nov 2023 17:25:42 -0300 Message-Id: <20231106202552.3404059-10-adhemerval.zanella@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231106202552.3404059-1-adhemerval.zanella@linaro.org> References: <20231106202552.3404059-1-adhemerval.zanella@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-13.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+patch=linaro.org@sourceware.org The dl-symbol-redir-ifunc.h redirects compiler-generated libcalls to arch-specific memory implementations to avoid ifunc calls where it is not yet possible. The memcmp-isa-default-impl.h aims to fix the same issue by calling the specific memset implementation directly. Using the memcmp symbol directly allows the compiler to inline the memset calls (especially because _dl_tunable_set_hwcaps uses constants values), generating better code. Checked on x86_64-linux-gnu. Reviewed-by: Noah Goldstein Reviewed-by: Siddhesh Poyarekar --- .../i686/multiarch/dl-symbol-redir-ifunc.h | 5 +++ sysdeps/x86/cpu-tunables.c | 39 ++++++------------- sysdeps/x86_64/memcmp-isa-default-impl.h | 28 ------------- .../x86_64/multiarch/dl-symbol-redir-ifunc.h | 15 +++++++ 4 files changed, 32 insertions(+), 55 deletions(-) delete mode 100644 sysdeps/x86_64/memcmp-isa-default-impl.h diff --git a/sysdeps/i386/i686/multiarch/dl-symbol-redir-ifunc.h b/sysdeps/i386/i686/multiarch/dl-symbol-redir-ifunc.h index dee69d19db..220c586bd2 100644 --- a/sysdeps/i386/i686/multiarch/dl-symbol-redir-ifunc.h +++ b/sysdeps/i386/i686/multiarch/dl-symbol-redir-ifunc.h @@ -19,6 +19,11 @@ #ifndef _DL_IFUNC_GENERIC_H #define _DL_IFUNC_GENERIC_H +#ifndef SHARED + asm ("memset = __memset_ia32"); +asm ("memcmp = __memcmp_ia32"); + +#endif /* SHARED */ #endif diff --git a/sysdeps/x86/cpu-tunables.c b/sysdeps/x86/cpu-tunables.c index 0d4f328585..5697885226 100644 --- a/sysdeps/x86/cpu-tunables.c +++ b/sysdeps/x86/cpu-tunables.c @@ -24,24 +24,11 @@ #include #include #include - -/* We can't use IFUNC memcmp nor strlen in init_cpu_features from libc.a - since IFUNC must be set up by init_cpu_features. */ -#if defined USE_MULTIARCH && !defined SHARED -# ifdef __x86_64__ -/* DEFAULT_MEMCMP by sysdeps/x86_64/memcmp-isa-default-impl.h. */ -# include -# else -# define DEFAULT_MEMCMP __memcmp_ia32 -# endif -extern __typeof (memcmp) DEFAULT_MEMCMP; -#else -# define DEFAULT_MEMCMP memcmp -#endif +#include #define CHECK_GLIBC_IFUNC_CPU_OFF(f, cpu_features, name, len) \ _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ - if (!DEFAULT_MEMCMP (f, #name, len)) \ + if (memcmp (f, #name, len) == 0) \ { \ CPU_FEATURE_UNSET (cpu_features, name) \ break; \ @@ -51,7 +38,7 @@ extern __typeof (memcmp) DEFAULT_MEMCMP; which isn't available. */ #define CHECK_GLIBC_IFUNC_PREFERRED_OFF(f, cpu_features, name, len) \ _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ - if (!DEFAULT_MEMCMP (f, #name, len)) \ + if (memcmp (f, #name, len) == 0) \ { \ cpu_features->preferred[index_arch_##name] \ &= ~bit_arch_##name; \ @@ -62,7 +49,7 @@ extern __typeof (memcmp) DEFAULT_MEMCMP; #define CHECK_GLIBC_IFUNC_PREFERRED_BOTH(f, cpu_features, name, \ disable, len) \ _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ - if (!DEFAULT_MEMCMP (f, #name, len)) \ + if (memcmp (f, #name, len) == 0) \ { \ if (disable) \ cpu_features->preferred[index_arch_##name] &= ~bit_arch_##name; \ @@ -76,7 +63,7 @@ extern __typeof (memcmp) DEFAULT_MEMCMP; #define CHECK_GLIBC_IFUNC_PREFERRED_NEED_BOTH(f, cpu_features, name, \ need, disable, len) \ _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ - if (!DEFAULT_MEMCMP (f, #name, len)) \ + if (memcmp (f, #name, len) == 0) \ { \ if (disable) \ cpu_features->preferred[index_arch_##name] &= ~bit_arch_##name; \ @@ -177,7 +164,7 @@ TUNABLE_CALLBACK (set_hwcaps) (tunable_val_t *valp) CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, POPCNT, 6); CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSE4_1, 6); CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSE4_2, 6); - if (!DEFAULT_MEMCMP (n, "XSAVEC", 6)) + if (memcmp (n, "XSAVEC", 6) == 0) { /* Update xsave_state_size to XSAVE state size. */ cpu_features->xsave_state_size @@ -290,12 +277,11 @@ attribute_hidden void TUNABLE_CALLBACK (set_x86_ibt) (tunable_val_t *valp) { - if (DEFAULT_MEMCMP (valp->strval, "on", sizeof ("on")) == 0) + if (memcmp (valp->strval, "on", sizeof ("on")) == 0) GL(dl_x86_feature_control).ibt = cet_always_on; - else if (DEFAULT_MEMCMP (valp->strval, "off", sizeof ("off")) == 0) + else if (memcmp (valp->strval, "off", sizeof ("off")) == 0) GL(dl_x86_feature_control).ibt = cet_always_off; - else if (DEFAULT_MEMCMP (valp->strval, "permissive", - sizeof ("permissive")) == 0) + else if (memcmp (valp->strval, "permissive", sizeof ("permissive")) == 0) GL(dl_x86_feature_control).ibt = cet_permissive; } @@ -303,12 +289,11 @@ attribute_hidden void TUNABLE_CALLBACK (set_x86_shstk) (tunable_val_t *valp) { - if (DEFAULT_MEMCMP (valp->strval, "on", sizeof ("on")) == 0) + if (memcmp (valp->strval, "on", sizeof ("on")) == 0) GL(dl_x86_feature_control).shstk = cet_always_on; - else if (DEFAULT_MEMCMP (valp->strval, "off", sizeof ("off")) == 0) + else if (memcmp (valp->strval, "off", sizeof ("off")) == 0) GL(dl_x86_feature_control).shstk = cet_always_off; - else if (DEFAULT_MEMCMP (valp->strval, "permissive", - sizeof ("permissive")) == 0) + else if (memcmp (valp->strval, "permissive", sizeof ("permissive")) == 0) GL(dl_x86_feature_control).shstk = cet_permissive; } #endif diff --git a/sysdeps/x86_64/memcmp-isa-default-impl.h b/sysdeps/x86_64/memcmp-isa-default-impl.h deleted file mode 100644 index 0962e83c3d..0000000000 --- a/sysdeps/x86_64/memcmp-isa-default-impl.h +++ /dev/null @@ -1,28 +0,0 @@ -/* Set default memcmp impl based on ISA level. - Copyright (C) 2022-2023 Free Software Foundation, Inc. - This file is part of the GNU C Library. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -#include -#if MINIMUM_X86_ISA_LEVEL == 1 || MINIMUM_X86_ISA_LEVEL == 2 -# define DEFAULT_MEMCMP __memcmp_sse2 -#elif MINIMUM_X86_ISA_LEVEL == 3 -# define DEFAULT_MEMCMP __memcmp_avx2_movbe -#elif MINIMUM_X86_ISA_LEVEL == 4 -# define DEFAULT_MEMCMP __memcmp_evex_movbe -#else -# error "Unknown default memcmp implementation" -#endif diff --git a/sysdeps/x86_64/multiarch/dl-symbol-redir-ifunc.h b/sysdeps/x86_64/multiarch/dl-symbol-redir-ifunc.h index 3fe73ca1c3..c7d8961bb6 100644 --- a/sysdeps/x86_64/multiarch/dl-symbol-redir-ifunc.h +++ b/sysdeps/x86_64/multiarch/dl-symbol-redir-ifunc.h @@ -19,6 +19,8 @@ #ifndef _DL_IFUNC_GENERIC_H #define _DL_IFUNC_GENERIC_H +#ifndef SHARED + #include #if MINIMUM_X86_ISA_LEVEL >= 4 @@ -31,4 +33,17 @@ asm ("memset = " HAVE_MEMSET_IFUNC_GENERIC); + +#if MINIMUM_X86_ISA_LEVEL >= 4 +# define HAVE_MEMCMP_IFUNC_GENERIC "__memcmp_evex_movbe" +#elif MINIMUM_X86_ISA_LEVEL == 3 +# define HAVE_MEMCMP_IFUNC_GENERIC "__memcmp_avx2_movbe" +#else +# define HAVE_MEMCMP_IFUNC_GENERIC "__memcmp_sse2" +#endif + +asm ("memcmp = " HAVE_MEMCMP_IFUNC_GENERIC); + +#endif /* SHARED */ + #endif