From patchwork Wed Sep 26 05:03:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 147537 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp263006lji; Tue, 25 Sep 2018 22:05:59 -0700 (PDT) X-Google-Smtp-Source: ACcGV63P3gUb0EiycbmPpD0U8jfKnIaeEZCTjAU3cSmuNnsvRe5DcbNFZ8pfW3qKt+9bfJBK6iV1 X-Received: by 2002:a63:d154:: with SMTP id c20-v6mr3948704pgj.188.1537938359385; Tue, 25 Sep 2018 22:05:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537938359; cv=none; d=google.com; s=arc-20160816; b=rRBw2EML+UjPH4u5t3HXouwnrBlgpNmkZaJ+QU5zXwxbIinIqzWidwlq+rjbA71cPK k24PGf73JaOFwgppnsh4bz69a8ltjVVC7t0M8XWAiaqnJOBJSPgT7RFBuLNg2pz+PXR5 FtSNGlMh5gyeahzqZkza1/O4BfXg0Nz26qLa/4Z5mzPyvd+LqwBjExP3rDl1UEo1Mgbt /qNQuFaCB9roO7QHBkXpBqym8XYRTqU0mSvsNatZYVTCdv8eKJp7vGu1jawc7F1AJS1y qDuMC+97G1v3ssyGEIGARHKNUCTm/bgzM8p+4hLwOrz6xknvTxb52zF8SsScQRB//LnR CN8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature; bh=5g5c4EL3+eSicB2HKkt8RUEf5G9bIKncW/GTtxREBSI=; b=dwL8T/n4KytceEmwfvFn/dpsQs4cDSxbInsHmWY/xO/joRexhXeEmSt4ozhFnzquLg ZAeGXQKpXs2jZ6zfc6KgEeuy7QV1nK3vi8GZmUBeWQyMtl4a1Hq3Rl27gcFrjsp/dfdX xpRtxUNutksTreoHXBXJdLmYIPypEllqbZknvbSosednL5oy8EGJFQQZpSzQKcM/MWAa NSq/OsndHgE7m5JgzsQrBDMFT1frXTTKPoGDSpZCU9sLqocKx6sPUQ7J1YVRrAUJfIn/ kTMPoArAvemY3CqXpuNYI05dqI+0DCV/vRSJm+2lcKUkY38X//cLVmyHdshw8o0zk874 hZHw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=c0htpFfK; dkim=pass header.i=@gmail.com header.s=20161025 header.b=pXB+KoKR; spf=pass (google.com: domain of gcc-patches-return-486390-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486390-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id d1-v6si4515782pld.322.2018.09.25.22.05.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Sep 2018 22:05:59 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-486390-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=c0htpFfK; dkim=pass header.i=@gmail.com header.s=20161025 header.b=pXB+KoKR; spf=pass (google.com: domain of gcc-patches-return-486390-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486390-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; q=dns; s= default; b=UuKKmJj6IHKAoHgVM9pmEuvU61rfTfgCn88CvB7D+JrAgQb7ZefNY riyUrR/ApmNWdvxL+ZstufmFkaZzSXtTV2j4UOzSMRwZanSJco73TvCnXSnFeNqV 2saP1waDZ/eHf7tlM6SogMSoRwxtKuPanE0klt2jyX/RX038//fGUY= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; s= default; bh=MV6tsDYtghrI/0Fc6RfUtB4rp2s=; b=c0htpFfKRM0TxBFN+zWu UnJ8MPSvZHTEnsrvxSl26757yOuGUN94uyee0agqL2U+CMUYUgGqptN9iuP1FA5X coiaxYbZ/RRdVPGuHGOG0oOaTqOIp+ny4zQPSfHJ/KGUoScuRUgKHsRl0sM6pZB+ FA9gI4LkgC89OcwdMfeE57Y= Received: (qmail 74612 invoked by alias); 26 Sep 2018 05:04:12 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 74493 invoked by uid 89); 26 Sep 2018 05:04:11 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.5 required=5.0 tests=AWL, BAYES_00, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=_, H*r:sk:c10-v6s, 3.1, HX-Received:sk:72-v6mr X-HELO: mail-pg1-f169.google.com Received: from mail-pg1-f169.google.com (HELO mail-pg1-f169.google.com) (209.85.215.169) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Wed, 26 Sep 2018 05:04:07 +0000 Received: by mail-pg1-f169.google.com with SMTP id c10-v6so6228223pgq.4 for ; Tue, 25 Sep 2018 22:04:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=5g5c4EL3+eSicB2HKkt8RUEf5G9bIKncW/GTtxREBSI=; b=pXB+KoKR2ZB2JZ/mYfOjt7zabf1aqtQE8D2MxjSce35Y0HccgNRProhzfcyWP8rWwh +6ySJ0lZNN0w1Sc1z1sB113twsI27SM8xv3QKaMYZzMkXHp32kB+kaYmMPT0i2mwdUPk +XKkvvHPqfCxFkEjIzo+YtHk7Ej33r4uRIxoW0s8TuDC4XMjH96WUCRTrnKWUNHOmL9E tLiKlG5rRhWZ9aOGRxkdiVChdJCB1IPJE4VM+UtKtNHwe6NQtyXzCcj/OkJYko57Ixb3 8C1JZ3liBZB9jsi3/4h2GLGKY/veV7mz68zQysSmgDBFty0d/Vn2izwLCnSXR00FWxVb ULUA== Return-Path: Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id j22-v6sm4954650pfh.45.2018.09.25.22.04.04 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 25 Sep 2018 22:04:04 -0700 (PDT) From: rth7680@gmail.com To: gcc-patches@gcc.gnu.org Cc: ramana.radhakrishnan@arm.com, agraf@suse.de, matz@suse.de, Richard Henderson Subject: [PATCH, AArch64 08/11] aarch64: Add out-of-line functions for LSE atomics Date: Tue, 25 Sep 2018 22:03:52 -0700 Message-Id: <20180926050355.32746-9-richard.henderson@linaro.org> In-Reply-To: <20180926050355.32746-1-richard.henderson@linaro.org> References: <20180926050355.32746-1-richard.henderson@linaro.org> From: Richard Henderson This is the libgcc part of the interface -- providing the functions. Rationale is provided at the top of libgcc/config/aarch64/lse.c. * config/aarch64/lse.c: New file. * config/aarch64/t-lse: New file. * config.host: Add t-lse to all aarch64 tuples. --- libgcc/config/aarch64/lse.c | 258 ++++++++++++++++++++++++++++++++++++ libgcc/config.host | 4 + libgcc/config/aarch64/t-lse | 44 ++++++ 3 files changed, 306 insertions(+) create mode 100644 libgcc/config/aarch64/lse.c create mode 100644 libgcc/config/aarch64/t-lse -- 2.17.1 diff --git a/libgcc/config/aarch64/lse.c b/libgcc/config/aarch64/lse.c new file mode 100644 index 00000000000..20f4bde741f --- /dev/null +++ b/libgcc/config/aarch64/lse.c @@ -0,0 +1,258 @@ +/* Out-of-line LSE atomics for AArch64 architecture. + Copyright (C) 2018 Free Software Foundation, Inc. + Contributed by Linaro Ltd. + +This file is part of GCC. + +GCC is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation; either version 3, or (at your option) any later +version. + +GCC is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +for more details. + +Under Section 7 of GPL version 3, you are granted additional +permissions described in the GCC Runtime Library Exception, version +3.1, as published by the Free Software Foundation. + +You should have received a copy of the GNU General Public License and +a copy of the GCC Runtime Library Exception along with this program; +see the files COPYING3 and COPYING.RUNTIME respectively. If not, see +. */ + +/* + * The problem that we are trying to solve is operating system deployment + * of ARMv8.1-Atomics, also known as Large System Exensions (LSE). + * + * There are a number of potential solutions for this problem which have + * been proposed and rejected for various reasons. To recap: + * + * (1) Multiple builds. The dynamic linker will examine /lib64/atomics/ + * if HWCAP_ATOMICS is set, allowing entire libraries to be overwritten. + * However, not all Linux distributions are happy with multiple builds, + * and anyway it has no effect on main applications. + * + * (2) IFUNC. We could put these functions into libgcc_s.so, and have + * a single copy of each function for all DSOs. However, ARM is concerned + * that the branch-to-indirect-branch that is implied by using a PLT, + * as required by IFUNC, is too much overhead for smaller cpus. + * + * (3) Statically predicted direct branches. This is the approach that + * is taken here. These functions are linked into every DSO that uses them. + * All of the symbols are hidden, so that the functions are called via a + * direct branch. The choice of LSE vs non-LSE is done via one byte load + * followed by a well-predicted direct branch. The functions are compiled + * separately to minimize code size. + */ + +/* Define or declare the symbol gating the LSE implementations. */ +#ifndef L_have_atomics +extern +#endif +_Bool __aa64_have_atomics __attribute__((visibility("hidden"), nocommon)); + +/* The branch controlled by this test should be easily predicted, in that + it will, after constructors, always branch the same way. The expectation + is that systems that implement ARMv8.1-Atomics are "beefier" than those + that omit the extension. By arranging for the fall-through path to use + load-store-exclusive insns, we aid the branch predictor of the + smallest cpus. */ +#define have_atomics __builtin_expect(__aa64_have_atomics, 0) + +#ifdef L_have_atomics +/* Disable initialization of __aa64_have_atomics during bootstrap. */ +# ifndef inhibit_libc +# include + +static void __attribute__((constructor)) +init_have_atomics(void) +{ + unsigned long hwcap = getauxval(AT_HWCAP); + __aa64_have_atomics = (hwcap & HWCAP_ATOMICS) != 0; +} +# endif /* inhibit_libc */ +#else + +/* Tell the assembler to accept LSE instructions. */ +asm(".arch armv8-a+lse"); + +/* Turn size and memory model defines into mnemonic fragments. */ +#if SIZE == 1 +# define S "b" +# define MASK ", uxtb" +#elif SIZE == 2 +# define S "h" +# define MASK ", uxth" +#elif SIZE == 4 || SIZE == 8 +# define S "" +# define MASK "" +#else +# error +#endif + +#if SIZE < 8 +# define T unsigned int +# define W "w" +#else +# define T unsigned long long +# define W "" +#endif + +#if MODEL == 1 +# define SUFF _relax +# define A "" +# define L "" +#elif MODEL == 2 +# define SUFF _acq +# define A "a" +# define L "" +#elif MODEL == 3 +# define SUFF _rel +# define A "" +# define L "l" +#elif MODEL == 4 +# define SUFF _acq_rel +# define A "a" +# define L "l" +#else +# error +#endif + +#define NAME2(B, S, X) __aa64_ ## B ## S ## X +#define NAME1(B, S, X) NAME2(B, S, X) +#define NAME(BASE) NAME1(BASE, SIZE, SUFF) + +#define str1(S) #S +#define str(S) str1(S) + +#ifdef L_cas +T NAME(cas)(T cmp, T new, T *ptr) __attribute__((visibility("hidden"))); +T NAME(cas)(T cmp, T new, T *ptr) +{ + T old; + unsigned tmp; + + if (have_atomics) + __asm__("cas" A L S " %"W"0, %"W"2, %1" + : "=r"(old), "+m"(*ptr) : "r"(new), "0"(cmp)); + else + __asm__( + "0: " + "ld" A "xr"S" %"W"0, %1\n\t" + "cmp %"W"0, %"W"4" MASK "\n\t" + "bne 1f\n\t" + "st" L "xr"S" %w2, %"W"3, %1\n\t" + "cbnz %w2, 0b\n" + "1:" + : "=&r"(old), "+m"(*ptr), "=&r"(tmp) : "r"(new), "r"(cmp)); + + return old; +} +#endif + +#ifdef L_swp +T NAME(swp)(T new, T *ptr) __attribute__((visibility("hidden"))); +T NAME(swp)(T new, T *ptr) +{ + T old; + unsigned tmp; + + if (have_atomics) + __asm__("swp" A L S " %"W"2, %"W"0, %1" + : "=r"(old), "+m"(*ptr) : "r"(new)); + else + __asm__( + "0: " + "ld" A "xr"S" %"W"0, %1\n\t" + "st" L "xr"S" %w2, %"W"3, %1\n\t" + "cbnz %w2, 0b\n" + "1:" + : "=&r"(old), "+m"(*ptr), "=&r"(tmp) : "r"(new)); + + return old; +} +#endif + +#if defined(L_ldadd) || defined(L_ldclr) \ + || defined(L_ldeor) || defined(L_ldset) + +#ifdef L_ldadd +#define LDOP ldadd +#define OP add +#elif defined(L_ldclr) +#define LDOP ldclr +#define OP bic +#elif defined(L_ldeor) +#define LDOP ldeor +#define OP eor +#elif defined(L_ldset) +#define LDOP ldset +#define OP orr +#else +#error +#endif + +T NAME(LDOP)(T val, T *ptr) __attribute__((visibility("hidden"))); +T NAME(LDOP)(T val, T *ptr) +{ + T old; + unsigned tmp; + + if (have_atomics) + __asm__(str(LDOP) A L S " %"W"2, %"W"0, %1" + : "=r"(old), "+m"(*ptr) : "r"(val)); + else + __asm__( + "0: " + "ld" A "xr"S" %"W"0, %1\n\t" + str(OP) " %"W"2, %"W"0, %"W"3\n\t" + "st" L "xr"S" %w2, %"W"2, %1\n\t" + "cbnz %w2, 0b\n" + "1:" + : "=&r"(old), "+m"(*ptr), "=&r"(tmp) : "r"(val)); + + return old; +} +#endif + +#if defined(L_stadd) || defined(L_stclr) \ + || defined(L_steor) || defined(L_stset) + +#ifdef L_stadd +#define STOP stadd +#define OP add +#elif defined(L_stclr) +#define STOP stclr +#define OP bic +#elif defined(L_steor) +#define STOP steor +#define OP eor +#elif defined(L_stset) +#define STOP stset +#define OP orr +#else +#error +#endif + +void NAME(STOP)(T val, T *ptr) __attribute__((visibility("hidden"))); +void NAME(STOP)(T val, T *ptr) +{ + unsigned tmp; + + if (have_atomics) + __asm__(str(STOP) L S " %"W"1, %0" : "+m"(*ptr) : "r"(val)); + else + __asm__( + "0: " + "ldxr"S" %"W"1, %0\n\t" + str(OP) " %"W"1, %"W"1, %"W"2\n\t" + "st" L "xr"S" %w1, %"W"1, %0\n\t" + "cbnz %w1, 0b\n" + "1:" + : "+m"(*ptr), "=&r"(tmp) : "r"(val)); +} +#endif +#endif /* L_have_atomics */ diff --git a/libgcc/config.host b/libgcc/config.host index 029f6569caf..2c4a05d69c5 100644 --- a/libgcc/config.host +++ b/libgcc/config.host @@ -340,23 +340,27 @@ aarch64*-*-elf | aarch64*-*-rtems*) extra_parts="$extra_parts crtbegin.o crtend.o crti.o crtn.o" extra_parts="$extra_parts crtfastmath.o" tmake_file="${tmake_file} ${cpu_type}/t-aarch64" + tmake_file="${tmake_file} ${cpu_type}/t-lse" tmake_file="${tmake_file} ${cpu_type}/t-softfp t-softfp t-crtfm" md_unwind_header=aarch64/aarch64-unwind.h ;; aarch64*-*-freebsd*) extra_parts="$extra_parts crtfastmath.o" tmake_file="${tmake_file} ${cpu_type}/t-aarch64" + tmake_file="${tmake_file} ${cpu_type}/t-lse" tmake_file="${tmake_file} ${cpu_type}/t-softfp t-softfp t-crtfm" md_unwind_header=aarch64/freebsd-unwind.h ;; aarch64*-*-fuchsia*) tmake_file="${tmake_file} ${cpu_type}/t-aarch64" + tmake_file="${tmake_file} ${cpu_type}/t-lse" tmake_file="${tmake_file} ${cpu_type}/t-softfp t-softfp" ;; aarch64*-*-linux*) extra_parts="$extra_parts crtfastmath.o" md_unwind_header=aarch64/linux-unwind.h tmake_file="${tmake_file} ${cpu_type}/t-aarch64" + tmake_file="${tmake_file} ${cpu_type}/t-lse" tmake_file="${tmake_file} ${cpu_type}/t-softfp t-softfp t-crtfm" ;; alpha*-*-linux*) diff --git a/libgcc/config/aarch64/t-lse b/libgcc/config/aarch64/t-lse new file mode 100644 index 00000000000..e862b0c2448 --- /dev/null +++ b/libgcc/config/aarch64/t-lse @@ -0,0 +1,44 @@ +# Out-of-line LSE atomics for AArch64 architecture. +# Copyright (C) 2018 Free Software Foundation, Inc. +# Contributed by Linaro Ltd. +# +# This file is part of GCC. +# +# GCC is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3, or (at your option) +# any later version. +# +# GCC is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GCC; see the file COPYING3. If not see +# . + +# CAS, Swap, Load-and-operate have 4 sizes and 4 memory models +S1 := $(foreach s, 1 2 4 8, $(addsuffix _$(s), cas swp ldadd ldclr ldeor ldset)) +O1 := $(foreach m, 1 2 3 4, $(addsuffix _$(m)$(objext), $(S1))) + +# Store-and-operate has 4 sizes but only 2 memory models (relaxed, release). +S2 := $(foreach s, 1 2 4 8, $(addsuffix _$(s), stadd stclr steor stset)) +O2 := $(foreach m, 1 3, $(addsuffix _$(m)$(objext), $(S2))) + +LSE_OBJS := $(O1) $(O2) + +libgcc-objects += $(LSE_OBJS) have_atomic$(objext) + +empty = +space = $(empty) $(empty) +PAT_SPLIT = $(subst _,$(space),$(*F)) +PAT_BASE = $(word 1,$(PAT_SPLIT)) +PAT_N = $(word 2,$(PAT_SPLIT)) +PAT_M = $(word 3,$(PAT_SPLIT)) + +have_atomic$(objext): $(srcdir)/config/aarch64/lse.c + $(gcc_compile) -DL_have_atomics -c $< + +$(LSE_OBJS): $(srcdir)/config/aarch64/lse.c + $(gcc_compile) -DL_$(PAT_BASE) -DSIZE=$(PAT_N) -DMODEL=$(PAT_M) -c $<