From patchwork Wed Sep 26 05:03:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 147540 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp263578lji; Tue, 25 Sep 2018 22:06:38 -0700 (PDT) X-Google-Smtp-Source: ACcGV61MycIGQkGfi52WLgikCmGI0oqBzSp5nM1SqySHjJ8PawkAeRomYIwgDOX7JfUFzho8dGfu X-Received: by 2002:a63:2587:: with SMTP id l129-v6mr3913604pgl.29.1537938398767; Tue, 25 Sep 2018 22:06:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537938398; cv=none; d=google.com; s=arc-20160816; b=x0ylstXhFOrvlIwDKy5lpytP3dz3wKtGtWlMPoQwKhyia3MdbZgBAXCne7clIAnu4T HXut9tNWPMk5lc44of5Fss7IuFs/3Yg90Tea1fVifsCImrwZgHTpJhjiykIuxZ9wYtAf G4SXCVj81aOaJANY6C2GsU+51C5Bt014/yrF6rowAL5a3VAUO0Bys+CZyjMnN6LpPoGf /1fw7kOj1w1kEf3GLpBsrd6QaEkEZ5ceOdgPLdrfOoxvJ+qVIgONaYvDPIdvjIdZKOD6 LVq9VGpGIpyZ719Aun/dkiv/PgNJU18AtH13tVRnAEpdnpX1yJ/yn4CaI+c4O/U349k+ aJuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature; bh=IA7OzxRjbSsPhl+kGerDzGhtm/i66GrI+X18o+vVf8g=; b=1FNqL2gLkcgn1UJ/9AZXuXhfQq0B+714KshbOVnsN0Y1dNbEG6pgmysQo2UPr3FCCM KyNmDp22VVc90y6ygVu4DrL0WGLO9e0VTjrEvIcennwsmmnspoT8fo8rk//Wr1Ti2Mwn qgPgftxbdu+KP0R+88OC4TbiE/8M28kN+cR4XpTL7ohGBYbQ2GSufToYjRbWD/ZRVwm+ ZFuxEQSr/miB0vbxZWB/gKx6x54PdJjhFPyq8yzyWmsNipHInbn+QldFe0LYszwG/TPD C7TDyIV5HzzfIruFWX5wXZyX6sPPjoGVsTM8StU+SPTQwm5rGvHO83ePJWDDefkwA301 oV9w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=tSqoVfhH; dkim=pass header.i=@gmail.com header.s=20161025 header.b=fBLJCca3; spf=pass (google.com: domain of gcc-patches-return-486393-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486393-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id c37-v6si4238237pgl.689.2018.09.25.22.06.38 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Sep 2018 22:06:38 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-486393-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=tSqoVfhH; dkim=pass header.i=@gmail.com header.s=20161025 header.b=fBLJCca3; spf=pass (google.com: domain of gcc-patches-return-486393-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486393-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; q=dns; s= default; b=n34ocKdo666RIZ9rLfyjWVXWwp8Tuap+n0ERUs+yw3NZ/HbZAR90k x2KBdjUaJZNXUG/aby7tvFNBnLXXvegZVhRz1cDPLvsYkOb8CJOrYHSacPT2Z8zp GdZiafInv0XgyoQQleIO7LtBKCuiD+qKyJTvw2iuhNyeTN83bJbMxU= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; s= default; bh=M6wvqfkifQq9ec6nvBBXOVyNqD0=; b=tSqoVfhHHIbJROS8tVp3 OIHUxWERbobeJDaim5p0slNnu2oISj1ydR9m0YQ87RA2NxjCsLzildbPGW6m1ROr bBvORAF6HSgAtXDYGRLDzi28R4GyDN4XC4JFUtbL1o9XuP9MnhhBp0tahTc9pdyC 7U6IigxK4iHZ017wVGppskc= Received: (qmail 75452 invoked by alias); 26 Sep 2018 05:04:18 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 74838 invoked by uid 89); 26 Sep 2018 05:04:13 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.5 required=5.0 tests=AWL, BAYES_00, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=NAMES X-HELO: mail-pg1-f172.google.com Received: from mail-pg1-f172.google.com (HELO mail-pg1-f172.google.com) (209.85.215.172) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Wed, 26 Sep 2018 05:04:09 +0000 Received: by mail-pg1-f172.google.com with SMTP id t70-v6so8414258pgd.12 for ; Tue, 25 Sep 2018 22:04:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=IA7OzxRjbSsPhl+kGerDzGhtm/i66GrI+X18o+vVf8g=; b=fBLJCca3GN8ch5Ze+FVwlWe6mAWgPFUOSx8xYz02p4r6nUYPjfgT9Tl8RXPOkCqXPd izc+pWY5D0QX+ndOB5YERTYsx+SO5VMnJw7srf9Q64to8BTy42Q2XcpAt9Ugy/Lgrvpq g3AptD7BGAHvPTGTVWmp9xd1oozPiF5zE7y9skURL4WDTkqRwxYEBpBrt6byW+bFG87n DLvKZwvSD7ZdQvxCuOethBPn3JdiBObmmGpqX7cVmrSaYSm0gmpMQ965uF4X24dreZCW +bIlXQl+hwKn3V4iuWvvhwMqkZGmMOA73tPlXjmdxeCydJ3l/JiUf1xNpNN3D889vvBP kKxg== Return-Path: Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id j22-v6sm4954650pfh.45.2018.09.25.22.04.06 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 25 Sep 2018 22:04:06 -0700 (PDT) From: rth7680@gmail.com To: gcc-patches@gcc.gnu.org Cc: ramana.radhakrishnan@arm.com, agraf@suse.de, matz@suse.de, Richard Henderson Subject: [PATCH, AArch64 10/11] aarch64: Implement TImode compare-and-swap Date: Tue, 25 Sep 2018 22:03:54 -0700 Message-Id: <20180926050355.32746-11-richard.henderson@linaro.org> In-Reply-To: <20180926050355.32746-1-richard.henderson@linaro.org> References: <20180926050355.32746-1-richard.henderson@linaro.org> From: Richard Henderson This pattern will only be used with the __sync functions, because we do not yet have a bare TImode atomic load. * config/aarch64/aarch64.c (aarch64_gen_compare_reg): Add support for NE comparison of TImode values. (aarch64_print_operand): Extend %R to handle general registers. (aarch64_emit_load_exclusive): Add support for TImode. (aarch64_emit_store_exclusive): Likewise. (aarch64_atomic_ool_func): Likewise. (aarch64_ool_cas_names): Likewise. * config/aarch64/atomics.md (@atomic_compare_and_swap): Change iterator from ALLI to ALLI_TI. (@atomic_compare_and_swap): New. (@atomic_compare_and_swap_lse): New. (aarch64_load_exclusive_pair): New. (aarch64_store_exclusive_pair): New. * config/aarch64/iterators.md (JUST_TI): New. * config/aarch64/lse.c (cas): Add support for SIZE == 16. * config/aarch64/t-lse (S0, O0): Split out cas. (LSE_OBJS): Include $(O0). --- gcc/config/aarch64/aarch64-protos.h | 2 +- gcc/config/aarch64/aarch64.c | 72 ++++++++++++++++++----- libgcc/config/aarch64/lse.c | 48 ++++++++++----- gcc/config/aarch64/atomics.md | 91 +++++++++++++++++++++++++++-- gcc/config/aarch64/iterators.md | 3 + libgcc/config/aarch64/t-lse | 10 +++- 6 files changed, 189 insertions(+), 37 deletions(-) -- 2.17.1 diff --git a/gcc/config/aarch64/aarch64-protos.h b/gcc/config/aarch64/aarch64-protos.h index c7b96b12bbe..f735c4e5ad8 100644 --- a/gcc/config/aarch64/aarch64-protos.h +++ b/gcc/config/aarch64/aarch64-protos.h @@ -626,7 +626,7 @@ bool aarch64_high_bits_all_ones_p (HOST_WIDE_INT); struct atomic_ool_names { - const char *str[4][4]; + const char *str[5][4]; }; rtx aarch64_atomic_ool_func(machine_mode mode, rtx model_rtx, diff --git a/gcc/config/aarch64/aarch64.c b/gcc/config/aarch64/aarch64.c index 78b30d68884..eca47784730 100644 --- a/gcc/config/aarch64/aarch64.c +++ b/gcc/config/aarch64/aarch64.c @@ -1606,10 +1606,33 @@ emit_set_insn (rtx x, rtx y) rtx aarch64_gen_compare_reg (RTX_CODE code, rtx x, rtx y) { - machine_mode mode = SELECT_CC_MODE (code, x, y); - rtx cc_reg = gen_rtx_REG (mode, CC_REGNUM); + machine_mode cmp_mode = GET_MODE (x); + machine_mode cc_mode; + rtx cc_reg; - emit_set_insn (cc_reg, gen_rtx_COMPARE (mode, x, y)); + if (cmp_mode == E_TImode) + { + gcc_assert (code == NE); + + cc_mode = E_CCmode; + cc_reg = gen_rtx_REG (cc_mode, CC_REGNUM); + + rtx x_lo = operand_subword (x, 0, 0, TImode); + rtx y_lo = operand_subword (y, 0, 0, TImode); + emit_set_insn (cc_reg, gen_rtx_COMPARE (cc_mode, x_lo, y_lo)); + + rtx x_hi = operand_subword (x, 1, 0, TImode); + rtx y_hi = operand_subword (y, 1, 0, TImode); + emit_insn (gen_ccmpdi (cc_reg, cc_reg, x_hi, y_hi, + gen_rtx_EQ (cc_mode, cc_reg, const0_rtx), + GEN_INT (AARCH64_EQ))); + } + else + { + cc_mode = SELECT_CC_MODE (code, x, y); + cc_reg = gen_rtx_REG (cc_mode, CC_REGNUM); + emit_set_insn (cc_reg, gen_rtx_COMPARE (cc_mode, x, y)); + } return cc_reg; } @@ -6689,7 +6712,7 @@ sizetochar (int size) 'S/T/U/V': Print a FP/SIMD register name for a register list. The register printed is the FP/SIMD register name of X + 0/1/2/3 for S/T/U/V. - 'R': Print a scalar FP/SIMD register name + 1. + 'R': Print a scalar Integer/FP/SIMD register name + 1. 'X': Print bottom 16 bits of integer constant in hex. 'w/x': Print a general register name or the zero register (32-bit or 64-bit). @@ -6881,12 +6904,13 @@ aarch64_print_operand (FILE *f, rtx x, int code) break; case 'R': - if (!REG_P (x) || !FP_REGNUM_P (REGNO (x))) - { - output_operand_lossage ("incompatible floating point / vector register operand for '%%%c'", code); - return; - } - asm_fprintf (f, "q%d", REGNO (x) - V0_REGNUM + 1); + if (REG_P (x) && FP_REGNUM_P (REGNO (x))) + asm_fprintf (f, "q%d", REGNO (x) - V0_REGNUM + 1); + else if (REG_P (x) && GP_REGNUM_P (REGNO (x))) + asm_fprintf (f, "x%d", REGNO (x) - R0_REGNUM + 1); + else + output_operand_lossage ("incompatible register operand for '%%%c'", + code); break; case 'X': @@ -14139,16 +14163,26 @@ static void aarch64_emit_load_exclusive (machine_mode mode, rtx rval, rtx mem, rtx model_rtx) { - emit_insn (gen_aarch64_load_exclusive (mode, rval, mem, model_rtx)); + if (mode == E_TImode) + emit_insn (gen_aarch64_load_exclusive_pair (gen_lowpart (DImode, rval), + gen_highpart (DImode, rval), + mem, model_rtx)); + else + emit_insn (gen_aarch64_load_exclusive (mode, rval, mem, model_rtx)); } /* Emit store exclusive. */ static void aarch64_emit_store_exclusive (machine_mode mode, rtx bval, - rtx rval, rtx mem, rtx model_rtx) + rtx mem, rtx val, rtx model_rtx) { - emit_insn (gen_aarch64_store_exclusive (mode, bval, rval, mem, model_rtx)); + if (mode == E_TImode) + emit_insn (gen_aarch64_store_exclusive_pair + (bval, mem, operand_subword (val, 0, 0, TImode), + operand_subword (val, 1, 0, TImode), model_rtx)); + else + emit_insn (gen_aarch64_store_exclusive (mode, bval, mem, val, model_rtx)); } /* Mark the previous jump instruction as unlikely. */ @@ -14160,7 +14194,7 @@ aarch64_emit_unlikely_jump (rtx insn) add_reg_br_prob_note (jump, profile_probability::very_unlikely ()); } -/* We store the names of the various atomic helpers in a 4x4 array. +/* We store the names of the various atomic helpers in a 5x4 array. Return the libcall function given MODE, MODEL and NAMES. */ rtx @@ -14184,6 +14218,9 @@ aarch64_atomic_ool_func(machine_mode mode, rtx model_rtx, case E_DImode: mode_idx = 3; break; + case E_TImode: + mode_idx = 4; + break; default: gcc_unreachable (); } @@ -14218,9 +14255,11 @@ aarch64_atomic_ool_func(machine_mode mode, rtx model_rtx, "__aa64_" #B #N "_rel", \ "__aa64_" #B #N "_acq_rel" } -#define DEF4(B) DEF0(B, 1), DEF0(B, 2), DEF0(B, 4), DEF0(B, 8) +#define DEF4(B) DEF0(B, 1), DEF0(B, 2), DEF0(B, 4), DEF0(B, 8), \ + { NULL, NULL, NULL, NULL } +#define DEF5(B) DEF0(B, 1), DEF0(B, 2), DEF0(B, 4), DEF0(B, 8), DEF0(B, 16) -static const atomic_ool_names aarch64_ool_cas_names = { { DEF4(cas) } }; +static const atomic_ool_names aarch64_ool_cas_names = { { DEF5(cas) } }; const atomic_ool_names aarch64_ool_swp_names = { { DEF4(swp) } }; const atomic_ool_names aarch64_ool_ldadd_names = { { DEF4(ldadd) } }; const atomic_ool_names aarch64_ool_ldset_names = { { DEF4(ldset) } }; @@ -14243,6 +14282,7 @@ const atomic_ool_names aarch64_ool_steor_names = { { DEF4(eor) } }; #undef DEF0 #undef DEF4 +#undef DEF5 /* Expand a compare and swap pattern. */ diff --git a/libgcc/config/aarch64/lse.c b/libgcc/config/aarch64/lse.c index 20f4bde741f..8f84820fa78 100644 --- a/libgcc/config/aarch64/lse.c +++ b/libgcc/config/aarch64/lse.c @@ -89,6 +89,7 @@ asm(".arch armv8-a+lse"); #elif SIZE == 4 || SIZE == 8 # define S "" # define MASK "" +#elif SIZE == 16 #else # error #endif @@ -96,9 +97,11 @@ asm(".arch armv8-a+lse"); #if SIZE < 8 # define T unsigned int # define W "w" -#else +#elif SIZE == 8 # define T unsigned long long # define W "" +#else +# define T unsigned __int128 #endif #if MODEL == 1 @@ -136,19 +139,38 @@ T NAME(cas)(T cmp, T new, T *ptr) unsigned tmp; if (have_atomics) - __asm__("cas" A L S " %"W"0, %"W"2, %1" - : "=r"(old), "+m"(*ptr) : "r"(new), "0"(cmp)); + { +#if SIZE == 16 + __asm__("casp" A L " %0, %R0, %2, %R2, %1" + : "=r"(old), "+m"(*ptr) : "r"(new), "0"(cmp)); +#else + __asm__("cas" A L S " %"W"0, %"W"2, %1" + : "=r"(old), "+m"(*ptr) : "r"(new), "0"(cmp)); +#endif + } else - __asm__( - "0: " - "ld" A "xr"S" %"W"0, %1\n\t" - "cmp %"W"0, %"W"4" MASK "\n\t" - "bne 1f\n\t" - "st" L "xr"S" %w2, %"W"3, %1\n\t" - "cbnz %w2, 0b\n" - "1:" - : "=&r"(old), "+m"(*ptr), "=&r"(tmp) : "r"(new), "r"(cmp)); - + { +#if SIZE == 16 + __asm__("0: " + "ld" A "xp %0, %R0, %1\n\t" + "cmp %0, %4\n\t" + "ccmp %R0, %R4, #0, eq\n\t" + "bne 1f\n\t" + "st" L "xp %w2, %3, %R3, %1\n\t" + "cbnz %w2, 0b\n" + "1:" + : "=&r"(old), "+m"(*ptr), "=&r"(tmp) : "r"(new), "r"(cmp)); +#else + __asm__("0: " + "ld" A "xr"S" %"W"0, %1\n\t" + "cmp %"W"0, %"W"4" MASK "\n\t" + "bne 1f\n\t" + "st" L "xr"S" %w2, %"W"3, %1\n\t" + "cbnz %w2, 0b\n" + "1:" + : "=&r"(old), "+m"(*ptr), "=&r"(tmp) : "r"(new), "r"(cmp)); +#endif + } return old; } #endif diff --git a/gcc/config/aarch64/atomics.md b/gcc/config/aarch64/atomics.md index 470c85d6ac3..568e4c831d1 100644 --- a/gcc/config/aarch64/atomics.md +++ b/gcc/config/aarch64/atomics.md @@ -22,10 +22,10 @@ (define_expand "@atomic_compare_and_swap" [(match_operand:SI 0 "register_operand" "") ;; bool out - (match_operand:ALLI 1 "register_operand" "") ;; val out - (match_operand:ALLI 2 "aarch64_sync_memory_operand" "") ;; memory - (match_operand:ALLI 3 "nonmemory_operand" "") ;; expected - (match_operand:ALLI 4 "aarch64_reg_or_zero" "") ;; desired + (match_operand:ALLI_TI 1 "register_operand" "") ;; val out + (match_operand:ALLI_TI 2 "aarch64_sync_memory_operand" "") ;; memory + (match_operand:ALLI_TI 3 "nonmemory_operand" "") ;; expected + (match_operand:ALLI_TI 4 "aarch64_reg_or_zero" "") ;; desired (match_operand:SI 5 "const_int_operand") ;; is_weak (match_operand:SI 6 "const_int_operand") ;; mod_s (match_operand:SI 7 "const_int_operand")] ;; mod_f @@ -88,6 +88,30 @@ } ) +(define_insn_and_split "@aarch64_compare_and_swap" + [(set (reg:CC CC_REGNUM) ;; bool out + (unspec_volatile:CC [(const_int 0)] UNSPECV_ATOMIC_CMPSW)) + (set (match_operand:JUST_TI 0 "register_operand" "=&r") ;; val out + (match_operand:JUST_TI 1 "aarch64_sync_memory_operand" "+Q")) ;; memory + (set (match_dup 1) + (unspec_volatile:JUST_TI + [(match_operand:JUST_TI 2 "register_operand" "r") ;; expect + (match_operand:JUST_TI 3 "aarch64_reg_or_zero" "rZ") ;; desired + (match_operand:SI 4 "const_int_operand") ;; is_weak + (match_operand:SI 5 "const_int_operand") ;; mod_s + (match_operand:SI 6 "const_int_operand")] ;; mod_f + UNSPECV_ATOMIC_CMPSW)) + (clobber (match_scratch:SI 7 "=&r"))] + "" + "#" + "&& reload_completed" + [(const_int 0)] + { + aarch64_split_compare_and_swap (operands); + DONE; + } +) + (define_insn "@aarch64_compare_and_swap_lse" [(set (match_operand:SI 0 "register_operand" "=r") ;; val out (zero_extend:SI @@ -133,6 +157,28 @@ return "casal\t%0, %3, %1"; }) +(define_insn "@aarch64_compare_and_swap_lse" + [(set (match_operand:JUST_TI 0 "register_operand" "=r") ;; val out + (match_operand:JUST_TI 1 "aarch64_sync_memory_operand" "+Q")) ;; memory + (set (match_dup 1) + (unspec_volatile:JUST_TI + [(match_operand:JUST_TI 2 "register_operand" "0") ;; expect + (match_operand:JUST_TI 3 "register_operand" "r") ;; desired + (match_operand:SI 4 "const_int_operand")] ;; mod_s + UNSPECV_ATOMIC_CMPSW))] + "TARGET_LSE" +{ + enum memmodel model = memmodel_from_int (INTVAL (operands[4])); + if (is_mm_relaxed (model)) + return "casp\t%0, %R0, %3, %R3, %1"; + else if (is_mm_acquire (model) || is_mm_consume (model)) + return "caspa\t%0, %R0, %3, %R3, %1"; + else if (is_mm_release (model)) + return "caspl\t%0, %R0, %3, %R3, %1"; + else + return "caspal\t%0, %R0, %3, %R3, %1"; +}) + (define_expand "atomic_exchange" [(match_operand:ALLI 0 "register_operand" "") (match_operand:ALLI 1 "aarch64_sync_memory_operand" "") @@ -650,6 +696,24 @@ } ) +(define_insn "aarch64_load_exclusive_pair" + [(set (match_operand:DI 0 "register_operand" "=r") + (unspec_volatile:DI + [(match_operand:TI 2 "aarch64_sync_memory_operand" "Q") + (match_operand:SI 3 "const_int_operand")] + UNSPECV_LX)) + (set (match_operand:DI 1 "register_operand" "=r") + (unspec_volatile:DI [(match_dup 2) (match_dup 3)] UNSPECV_LX))] + "" + { + enum memmodel model = memmodel_from_int (INTVAL (operands[3])); + if (is_mm_relaxed (model) || is_mm_consume (model) || is_mm_release (model)) + return "ldxp\t%0, %1, %2"; + else + return "ldaxp\t%0, %1, %2"; + } +) + (define_insn "@aarch64_store_exclusive" [(set (match_operand:SI 0 "register_operand" "=&r") (unspec_volatile:SI [(const_int 0)] UNSPECV_SX)) @@ -668,6 +732,25 @@ } ) +(define_insn "aarch64_store_exclusive_pair" + [(set (match_operand:SI 0 "register_operand" "=&r") + (unspec_volatile:SI [(const_int 0)] UNSPECV_SX)) + (set (match_operand:TI 1 "aarch64_sync_memory_operand" "=Q") + (unspec_volatile:TI + [(match_operand:DI 2 "aarch64_reg_or_zero" "rZ") + (match_operand:DI 3 "aarch64_reg_or_zero" "rZ") + (match_operand:SI 4 "const_int_operand")] + UNSPECV_SX))] + "" + { + enum memmodel model = memmodel_from_int (INTVAL (operands[3])); + if (is_mm_relaxed (model) || is_mm_consume (model) || is_mm_acquire (model)) + return "stxp\t%w0, %x2, %x3, %1"; + else + return "stlxp\t%w0, %x2, %x3, %1"; + } +) + (define_expand "mem_thread_fence" [(match_operand:SI 0 "const_int_operand" "")] "" diff --git a/gcc/config/aarch64/iterators.md b/gcc/config/aarch64/iterators.md index 524e4e6929b..dd26bdbbc6b 100644 --- a/gcc/config/aarch64/iterators.md +++ b/gcc/config/aarch64/iterators.md @@ -29,6 +29,9 @@ ;; Iterator for HI, SI, DI, some instructions can only work on these modes. (define_mode_iterator GPI_I16 [(HI "AARCH64_ISA_F16") SI DI]) +;; "Iterator" for just TI -- features like @pattern only work with iterators. +(define_mode_iterator JUST_TI [TI]) + ;; Iterator for QI and HI modes (define_mode_iterator SHORT [QI HI]) diff --git a/libgcc/config/aarch64/t-lse b/libgcc/config/aarch64/t-lse index e862b0c2448..534ff6efea8 100644 --- a/libgcc/config/aarch64/t-lse +++ b/libgcc/config/aarch64/t-lse @@ -18,15 +18,19 @@ # along with GCC; see the file COPYING3. If not see # . -# CAS, Swap, Load-and-operate have 4 sizes and 4 memory models -S1 := $(foreach s, 1 2 4 8, $(addsuffix _$(s), cas swp ldadd ldclr ldeor ldset)) +# Compare-and-swap has 5 sizes and 4 memory models. +S0 := $(foreach s, 1 2 4 8 16, $(addsuffix _$(s), cas)) +O0 := $(foreach m, 1 2 3 4, $(addsuffix _$(m)$(objext), $(S0))) + +# Swap, Load-and-operate have 4 sizes and 4 memory models +S1 := $(foreach s, 1 2 4 8, $(addsuffix _$(s), swp ldadd ldclr ldeor ldset)) O1 := $(foreach m, 1 2 3 4, $(addsuffix _$(m)$(objext), $(S1))) # Store-and-operate has 4 sizes but only 2 memory models (relaxed, release). S2 := $(foreach s, 1 2 4 8, $(addsuffix _$(s), stadd stclr steor stset)) O2 := $(foreach m, 1 3, $(addsuffix _$(m)$(objext), $(S2))) -LSE_OBJS := $(O1) $(O2) +LSE_OBJS := $(O0) $(O1) $(O2) libgcc-objects += $(LSE_OBJS) have_atomic$(objext)