From patchwork Wed Sep 26 05:03:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 147539 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp263396lji; Tue, 25 Sep 2018 22:06:25 -0700 (PDT) X-Google-Smtp-Source: ACcGV601JFf5uB+M3mQOUzBa1tGloQovL2nQs1jYVhRPIqS0k0G7pYX2Eiu/oTyThQyHzYKBF2MR X-Received: by 2002:a17:902:b092:: with SMTP id p18-v6mr4262724plr.90.1537938385674; Tue, 25 Sep 2018 22:06:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537938385; cv=none; d=google.com; s=arc-20160816; b=QQKUohNbyzV4/hBXkBz9rzdtP3u61DrDcUyGOMJXH22GquluxO3OKT554wW7Mv/fKV VBPtuP8pU53pgr5ZhSA439Yq5X8v57XtSs6mza9kkcs72QSgNtnj2aFY7H4BnxevuK37 UlU2FDQxqOcVTh546qHzmzxuk/ZSVX17FU0QD3/Lpnd4ZtZoTShHjMW04xid9THDJdeF xywpYmRtZbQBn+5FaTDWytz4Qfg0/AANWixuJ1GCOWALpQmNSHfjVoMBVwLep4wpedUS xM0gYMREfaTavplJYA/jXraZkc6gncncKs6h0idPhsgkXkdmoD0jG/qPjDVqLuDwtgLR 5iEA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature; bh=Wv48GDrpSDVfkxPW52L9GfsTCrD6HbZWoijmVAGKRSs=; b=B/zB3jUQ5FvDHKhTrtUWueQHLfYRHsGGGMqAqhyyKMMOCKq+z1vCB1hOqxZzeb+Nym UT7F8t3GKNnSdotEcueCRXbF2GjYpIiuWayS+puzerH+tzB39c2/lT1I5LG5VHHSUsdl TQpVIT5soabrfvsmgHmmVmadz6yRQel2BQdec49/7JlOevfrnssb/ECqaFNHJPMMYeyy yAahfNWvtcPyd+ipB4QpLW9KXHhbvJggyBl6iJGyB7ED3TQe8yI6oPhDZP9xQ4tDIi0M 6bNQ1akOzfNDOsaBSJ3BSCaRVPOI6k8ODxdGmSFR/QicIOqv//HO4Q83zhxINkwy95KW ZWyw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=hwMir1bb; dkim=pass header.i=@gmail.com header.s=20161025 header.b=InFuDcE7; spf=pass (google.com: domain of gcc-patches-return-486392-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486392-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id 59-v6si4290510plp.87.2018.09.25.22.06.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Sep 2018 22:06:25 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-486392-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=hwMir1bb; dkim=pass header.i=@gmail.com header.s=20161025 header.b=InFuDcE7; spf=pass (google.com: domain of gcc-patches-return-486392-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486392-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; q=dns; s= default; b=dE/BuqT0QOAmRKCs44DY8aWPnlmMFMDJihq68dV9qFD1hgeucYOPX c1hFzbGIz7E0udmjuh62infyquW3XqOHEJts+fvWa3/FT85ju3TEIBRwflL+w2nQ tZ2qjNWtMuupM1HaswsNPM7OcPpfwHm1Eqchh13NQ4+3NQnsSlALuc= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; s= default; bh=ibQkAGI69mVI7CvNlxiH2ZHnfI4=; b=hwMir1bbzgzfHVx0jFBb b1xX/EG+GRmpp/Q+r5TroH7P3au2jdObexbZh+uEuKPTXMITSchXBJXLqpKOvlHW UynnMIFkwVL8Rf+yhZnKLNJ7YyocPoUGlI8RX/K+gbkTK3pWzYv7JnqaC9wR3aiB yEG0OMdHaw6uPPMEX+ddYEI= Received: (qmail 75303 invoked by alias); 26 Sep 2018 05:04:17 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 74780 invoked by uid 89); 26 Sep 2018 05:04:13 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.5 required=5.0 tests=AWL, BAYES_00, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=H*r:sk:85-v6so, NAMES X-HELO: mail-pg1-f175.google.com Received: from mail-pg1-f175.google.com (HELO mail-pg1-f175.google.com) (209.85.215.175) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Wed, 26 Sep 2018 05:04:08 +0000 Received: by mail-pg1-f175.google.com with SMTP id 85-v6so10147335pge.6 for ; Tue, 25 Sep 2018 22:04:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Wv48GDrpSDVfkxPW52L9GfsTCrD6HbZWoijmVAGKRSs=; b=InFuDcE70OWUujvyzun7jYmsiQgerlRg3DtrHhdGSAXMmTVsJNdfV7pAtLhp31f1tf 5twSqvPcA7ZeLXK4VkuZCWuZV2p4Lu5ABMiUemL46Yb+XrsXkzr60GvOGW2/4/dItSLp biugmlv/FX9fP2rHPxdgOb2+weJP4UpJoxO3EgMx6R58MhQixI1jHrzM9DrsXdRgsV/0 Tffims6iTJ9M/D3/kEto6D060OWJSZEcDEadMtYxwT48tVmC3tgth0vvlskvNBwgfL4G npi7VFDCfEsnYVx1ykab7p88cUav5Sz1R5rACWvq/cp9zbZTp0zheYEnDPWHGFz3BeAc DOoA== Return-Path: Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id j22-v6sm4954650pfh.45.2018.09.25.22.04.05 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 25 Sep 2018 22:04:05 -0700 (PDT) From: rth7680@gmail.com To: gcc-patches@gcc.gnu.org Cc: ramana.radhakrishnan@arm.com, agraf@suse.de, matz@suse.de, Richard Henderson Subject: [PATCH, AArch64 09/11] aarch64: Implement -matomic-ool Date: Tue, 25 Sep 2018 22:03:53 -0700 Message-Id: <20180926050355.32746-10-richard.henderson@linaro.org> In-Reply-To: <20180926050355.32746-1-richard.henderson@linaro.org> References: <20180926050355.32746-1-richard.henderson@linaro.org> From: Richard Henderson * config/aarch64/aarch64.opt (-matomic-ool): New. * config/aarch64/aarch64.c (aarch64_atomic_ool_func): New. (aarch64_ool_cas_names, aarch64_ool_swp_names): New. (aarch64_ool_ldadd_names, aarch64_ool_ldset_names): New. (aarch64_ool_ldclr_names, aarch64_ool_ldeor_names): New. (aarch64_ool_stadd_names, aarch64_ool_stset_names): New. (aarch64_ool_stclr_names, aarch64_ool_steor_names): New. (aarch64_expand_compare_and_swap): Honor TARGET_ATOMIC_OOL. * config/aarch64/atomics.md (atomic_exchange): Likewise. (atomic_): Likewise. (atomic_fetch_): Likewise. (atomic__fetch): Likewise. --- gcc/config/aarch64/aarch64-protos.h | 17 ++++ gcc/config/aarch64/aarch64.c | 95 +++++++++++++++++++ .../atomic-comp-swap-release-acquire.c | 2 +- .../gcc.target/aarch64/atomic-op-acq_rel.c | 2 +- .../gcc.target/aarch64/atomic-op-acquire.c | 2 +- .../gcc.target/aarch64/atomic-op-char.c | 2 +- .../gcc.target/aarch64/atomic-op-consume.c | 2 +- .../gcc.target/aarch64/atomic-op-imm.c | 2 +- .../gcc.target/aarch64/atomic-op-int.c | 2 +- .../gcc.target/aarch64/atomic-op-long.c | 2 +- .../gcc.target/aarch64/atomic-op-relaxed.c | 2 +- .../gcc.target/aarch64/atomic-op-release.c | 2 +- .../gcc.target/aarch64/atomic-op-seq_cst.c | 2 +- .../gcc.target/aarch64/atomic-op-short.c | 2 +- .../aarch64/atomic_cmp_exchange_zero_reg_1.c | 2 +- .../atomic_cmp_exchange_zero_strong_1.c | 2 +- .../gcc.target/aarch64/sync-comp-swap.c | 2 +- .../gcc.target/aarch64/sync-op-acquire.c | 2 +- .../gcc.target/aarch64/sync-op-full.c | 2 +- gcc/config/aarch64/aarch64.opt | 4 + gcc/config/aarch64/atomics.md | 94 ++++++++++++++++-- gcc/doc/invoke.texi | 14 ++- 22 files changed, 232 insertions(+), 26 deletions(-) -- 2.17.1 diff --git a/gcc/config/aarch64/aarch64-protos.h b/gcc/config/aarch64/aarch64-protos.h index 1d2f8487d1a..c7b96b12bbe 100644 --- a/gcc/config/aarch64/aarch64-protos.h +++ b/gcc/config/aarch64/aarch64-protos.h @@ -624,4 +624,21 @@ poly_uint64 aarch64_regmode_natural_size (machine_mode); bool aarch64_high_bits_all_ones_p (HOST_WIDE_INT); +struct atomic_ool_names +{ + const char *str[4][4]; +}; + +rtx aarch64_atomic_ool_func(machine_mode mode, rtx model_rtx, + const atomic_ool_names *names); +extern const atomic_ool_names aarch64_ool_swp_names; +extern const atomic_ool_names aarch64_ool_stadd_names; +extern const atomic_ool_names aarch64_ool_stset_names; +extern const atomic_ool_names aarch64_ool_stclr_names; +extern const atomic_ool_names aarch64_ool_steor_names; +extern const atomic_ool_names aarch64_ool_ldadd_names; +extern const atomic_ool_names aarch64_ool_ldset_names; +extern const atomic_ool_names aarch64_ool_ldclr_names; +extern const atomic_ool_names aarch64_ool_ldeor_names; + #endif /* GCC_AARCH64_PROTOS_H */ diff --git a/gcc/config/aarch64/aarch64.c b/gcc/config/aarch64/aarch64.c index 1e00fdc801c..78b30d68884 100644 --- a/gcc/config/aarch64/aarch64.c +++ b/gcc/config/aarch64/aarch64.c @@ -14160,6 +14160,90 @@ aarch64_emit_unlikely_jump (rtx insn) add_reg_br_prob_note (jump, profile_probability::very_unlikely ()); } +/* We store the names of the various atomic helpers in a 4x4 array. + Return the libcall function given MODE, MODEL and NAMES. */ + +rtx +aarch64_atomic_ool_func(machine_mode mode, rtx model_rtx, + const atomic_ool_names *names) +{ + memmodel model = memmodel_base (INTVAL (model_rtx)); + int mode_idx, model_idx; + + switch (mode) + { + case E_QImode: + mode_idx = 0; + break; + case E_HImode: + mode_idx = 1; + break; + case E_SImode: + mode_idx = 2; + break; + case E_DImode: + mode_idx = 3; + break; + default: + gcc_unreachable (); + } + + switch (model) + { + case MEMMODEL_RELAXED: + model_idx = 0; + break; + case MEMMODEL_CONSUME: + case MEMMODEL_ACQUIRE: + model_idx = 1; + break; + case MEMMODEL_RELEASE: + model_idx = 2; + break; + case MEMMODEL_ACQ_REL: + case MEMMODEL_SEQ_CST: + model_idx = 3; + break; + default: + gcc_unreachable (); + } + + return init_one_libfunc_visibility (names->str[mode_idx][model_idx], + VISIBILITY_HIDDEN); +} + +#define DEF0(B, N) \ + { "__aa64_" #B #N "_relax", \ + "__aa64_" #B #N "_acq", \ + "__aa64_" #B #N "_rel", \ + "__aa64_" #B #N "_acq_rel" } + +#define DEF4(B) DEF0(B, 1), DEF0(B, 2), DEF0(B, 4), DEF0(B, 8) + +static const atomic_ool_names aarch64_ool_cas_names = { { DEF4(cas) } }; +const atomic_ool_names aarch64_ool_swp_names = { { DEF4(swp) } }; +const atomic_ool_names aarch64_ool_ldadd_names = { { DEF4(ldadd) } }; +const atomic_ool_names aarch64_ool_ldset_names = { { DEF4(ldset) } }; +const atomic_ool_names aarch64_ool_ldclr_names = { { DEF4(ldclr) } }; +const atomic_ool_names aarch64_ool_ldeor_names = { { DEF4(ldeor) } }; + +/* Note that the store forms are only available for relax and release + memory models. For the other models, re-use the load forms. */ +#undef DEF0 +#define DEF0(B, N) \ + { "__aa64_st" #B #N "_relax", \ + "__aa64_ld" #B #N "_acq", \ + "__aa64_st" #B #N "_rel", \ + "__aa64_ld" #B #N "_acq_rel" } + +const atomic_ool_names aarch64_ool_stadd_names = { { DEF4(add) } }; +const atomic_ool_names aarch64_ool_stset_names = { { DEF4(set) } }; +const atomic_ool_names aarch64_ool_stclr_names = { { DEF4(clr) } }; +const atomic_ool_names aarch64_ool_steor_names = { { DEF4(eor) } }; + +#undef DEF0 +#undef DEF4 + /* Expand a compare and swap pattern. */ void @@ -14204,6 +14288,17 @@ aarch64_expand_compare_and_swap (rtx operands[]) newval, mod_s)); cc_reg = aarch64_gen_compare_reg_maybe_ze (NE, rval, oldval, mode); } + else if (TARGET_ATOMIC_OOL) + { + /* Oldval must satisfy compare afterward. */ + if (!aarch64_plus_operand (oldval, mode)) + oldval = force_reg (mode, oldval); + rtx func = aarch64_atomic_ool_func (mode, mod_s, &aarch64_ool_cas_names); + rval = emit_library_call_value (func, NULL_RTX, LCT_NORMAL, r_mode, + oldval, mode, newval, mode, + XEXP (mem, 0), ptr_mode); + cc_reg = aarch64_gen_compare_reg_maybe_ze (NE, rval, oldval, mode); + } else { /* The oldval predicate varies by mode. Test it and force to reg. */ diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-comp-swap-release-acquire.c b/gcc/testsuite/gcc.target/aarch64/atomic-comp-swap-release-acquire.c index 49ca5d0d09c..e92f205c3a8 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-comp-swap-release-acquire.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-comp-swap-release-acquire.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2 -fno-ipa-icf" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -fno-ipa-icf -mno-atomic-ool" } */ #include "atomic-comp-swap-release-acquire.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-acq_rel.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-acq_rel.c index 74f26348e42..6965431f7d9 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-acq_rel.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-acq_rel.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-acq_rel.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-acquire.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-acquire.c index 66c1b1efe20..07dbca49d56 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-acquire.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-acquire.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-acquire.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-char.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-char.c index c09d0434ecf..73bfbb7afc9 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-char.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-char.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-char.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-consume.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-consume.c index 5783ab84f5c..c7945b3a22d 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-consume.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-consume.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-consume.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-imm.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-imm.c index 18b8f0b04e9..e46bb3de7c1 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-imm.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-imm.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ int v = 0; diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-int.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-int.c index 8520f0839ba..9b55deb5225 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-int.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-int.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-int.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-long.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-long.c index d011f8c5ce2..2622f75331f 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-long.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-long.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ long v = 0; diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-relaxed.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-relaxed.c index ed96bfdb978..f118a37a352 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-relaxed.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-relaxed.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-relaxed.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-release.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-release.c index fc4be17de89..579634b08e8 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-release.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-release.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-release.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-seq_cst.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-seq_cst.c index 613000fe490..016b0d6619f 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-seq_cst.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-seq_cst.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-seq_cst.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-short.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-short.c index e82c8118ece..978bd1d8377 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-short.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-short.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-short.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_reg_1.c b/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_reg_1.c index f2a21ddf2e1..77430ecdbce 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_reg_1.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_reg_1.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-O2 -march=armv8-a+nolse" } */ +/* { dg-options "-O2 -march=armv8-a+nolse -mno-atomic-ool" } */ /* { dg-skip-if "" { *-*-* } { "-mcpu=*" } { "" } } */ int diff --git a/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_strong_1.c b/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_strong_1.c index 8d2ae67dfbe..7d58b2f6bd0 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_strong_1.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_strong_1.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-O2 -march=armv8-a+nolse" } */ +/* { dg-options "-O2 -march=armv8-a+nolse -mno-atomic-ool" } */ /* { dg-skip-if "" { *-*-* } { "-mcpu=*" } { "" } } */ int diff --git a/gcc/testsuite/gcc.target/aarch64/sync-comp-swap.c b/gcc/testsuite/gcc.target/aarch64/sync-comp-swap.c index e571b2f13b3..7fc5885d0fd 100644 --- a/gcc/testsuite/gcc.target/aarch64/sync-comp-swap.c +++ b/gcc/testsuite/gcc.target/aarch64/sync-comp-swap.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2 -fno-ipa-icf" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -fno-ipa-icf -mno-atomic-ool" } */ #include "sync-comp-swap.x" diff --git a/gcc/testsuite/gcc.target/aarch64/sync-op-acquire.c b/gcc/testsuite/gcc.target/aarch64/sync-op-acquire.c index 357bf1be3b2..6ad0daa8998 100644 --- a/gcc/testsuite/gcc.target/aarch64/sync-op-acquire.c +++ b/gcc/testsuite/gcc.target/aarch64/sync-op-acquire.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "sync-op-acquire.x" diff --git a/gcc/testsuite/gcc.target/aarch64/sync-op-full.c b/gcc/testsuite/gcc.target/aarch64/sync-op-full.c index c6ba1629965..9a7afeb70d3 100644 --- a/gcc/testsuite/gcc.target/aarch64/sync-op-full.c +++ b/gcc/testsuite/gcc.target/aarch64/sync-op-full.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "sync-op-full.x" diff --git a/gcc/config/aarch64/aarch64.opt b/gcc/config/aarch64/aarch64.opt index b2e80cbf6f1..83166834165 100644 --- a/gcc/config/aarch64/aarch64.opt +++ b/gcc/config/aarch64/aarch64.opt @@ -218,3 +218,7 @@ Enables verbose cost model dumping in the debug dump files. mtrack-speculation Target Var(aarch64_track_speculation) Generate code to track when the CPU might be speculating incorrectly. + +matomic-ool +Target Report Mask(ATOMIC_OOL) Save +Generate local calls to out-of-line atomic operations. diff --git a/gcc/config/aarch64/atomics.md b/gcc/config/aarch64/atomics.md index f74521f885d..470c85d6ac3 100644 --- a/gcc/config/aarch64/atomics.md +++ b/gcc/config/aarch64/atomics.md @@ -140,16 +140,27 @@ (match_operand:SI 3 "const_int_operand" "")] "" { - rtx (*gen) (rtx, rtx, rtx, rtx); - /* Use an atomic SWP when available. */ if (TARGET_LSE) - gen = gen_aarch64_atomic_exchange_lse; + { + emit_insn (gen_aarch64_atomic_exchange_lse + (operands[0], operands[1], operands[2], operands[3])); + } + else if (TARGET_ATOMIC_OOL) + { + machine_mode mode = mode; + rtx func = aarch64_atomic_ool_func (mode, operands[3], + &aarch64_ool_swp_names); + rtx rval = emit_library_call_value (func, operands[0], LCT_NORMAL, + mode, operands[2], mode, + XEXP (operands[1], 0), ptr_mode); + emit_move_insn (operands[0], rval); + } else - gen = gen_aarch64_atomic_exchange; - - emit_insn (gen (operands[0], operands[1], operands[2], operands[3])); - + { + emit_insn (gen_aarch64_atomic_exchange + (operands[0], operands[1], operands[2], operands[3])); + } DONE; } ) @@ -234,6 +245,39 @@ } operands[1] = force_reg (mode, operands[1]); } + else if (TARGET_ATOMIC_OOL) + { + const atomic_ool_names *names; + switch () + { + case MINUS: + operands[1] = expand_simple_unop (mode, NEG, operands[1], + NULL, 1); + /* fallthru */ + case PLUS: + names = &aarch64_ool_stadd_names; + break; + case IOR: + names = &aarch64_ool_stset_names; + break; + case XOR: + names = &aarch64_ool_steor_names; + break; + case AND: + operands[1] = expand_simple_unop (mode, NOT, operands[1], + NULL, 1); + names = &aarch64_ool_stclr_names; + break; + default: + gcc_unreachable (); + } + machine_mode mode = mode; + rtx func = aarch64_atomic_ool_func (mode, operands[2], names); + emit_library_call_value (func, NULL_RTX, LCT_NORMAL, mode, + operands[1], mode, + XEXP (operands[0], 0), ptr_mode); + DONE; + } else gen = gen_aarch64_atomic_; @@ -350,6 +394,40 @@ } operands[2] = force_reg (mode, operands[2]); } + else if (TARGET_ATOMIC_OOL) + { + const atomic_ool_names *names; + switch () + { + case MINUS: + operands[2] = expand_simple_unop (mode, NEG, operands[2], + NULL, 1); + /* fallthru */ + case PLUS: + names = &aarch64_ool_ldadd_names; + break; + case IOR: + names = &aarch64_ool_ldset_names; + break; + case XOR: + names = &aarch64_ool_ldeor_names; + break; + case AND: + operands[2] = expand_simple_unop (mode, NOT, operands[2], + NULL, 1); + names = &aarch64_ool_ldclr_names; + break; + default: + gcc_unreachable (); + } + machine_mode mode = mode; + rtx func = aarch64_atomic_ool_func (mode, operands[3], names); + rtx rval = emit_library_call_value (func, operands[0], LCT_NORMAL, mode, + operands[2], mode, + XEXP (operands[1], 0), ptr_mode); + emit_move_insn (operands[0], rval); + DONE; + } else gen = gen_aarch64_atomic_fetch_; @@ -439,7 +517,7 @@ { /* Use an atomic load-operate instruction when possible. In this case we will re-compute the result from the original mem value. */ - if (TARGET_LSE) + if (TARGET_LSE || TARGET_ATOMIC_OOL) { rtx tmp = gen_reg_rtx (mode); operands[2] = force_reg (mode, operands[2]); diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi index 7ef4e7a449b..229a6f06aae 100644 --- a/gcc/doc/invoke.texi +++ b/gcc/doc/invoke.texi @@ -621,7 +621,7 @@ Objective-C and Objective-C++ Dialects}. -mpc-relative-literal-loads @gol -msign-return-address=@var{scope} @gol -march=@var{name} -mcpu=@var{name} -mtune=@var{name} @gol --moverride=@var{string} -mverbose-cost-dump -mtrack-speculation} +-moverride=@var{string} -mverbose-cost-dump -mtrack-speculation -matomic-ool} @emph{Adapteva Epiphany Options} @gccoptlist{-mhalf-reg-file -mprefer-short-insn-regs @gol @@ -15039,6 +15039,18 @@ be used by the compiler when expanding calls to @code{__builtin_speculation_safe_copy} to permit a more efficient code sequence to be generated. +@item -matomic-ool +@itemx -mno-atomic-ool +Enable or disable calls to out-of-line helpers to implement atomic operations. +These helpers will, at runtime, determine if ARMv8.1-Atomics instructions +should be used; if not, they will use the load/store-exclusive instructions +that are present in the base ARMv8.0 ISA. + +This option is only applicable when compiling for the base ARMv8.0 +instruction set. If using a later revision, e.g. @option{-march=armv8.1-a} +or @option{-march=armv8-a+lse}, the ARMv8.1-Atomics instructions will be +used directly. + @item -march=@var{name} @opindex march Specify the name of the target architecture and, optionally, one or