From patchwork Mon Jul 9 16:38:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Richard Earnshaw \(lists\)" X-Patchwork-Id: 141470 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp2872238ljj; Mon, 9 Jul 2018 09:40:20 -0700 (PDT) X-Google-Smtp-Source: AAOMgpeWlsL3+N4nSdswbvovi/xlArvRgJgjrRoPHycGuTcItIgFTfiUGKFABpyvXQ/vDuG75PN3 X-Received: by 2002:a17:902:7e43:: with SMTP id a3-v6mr21088433pln.151.1531154420522; Mon, 09 Jul 2018 09:40:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1531154420; cv=none; d=google.com; s=arc-20160816; b=cV1XAxv0clUkB08FnJJeeyFoaX+/wAjOvCEsdg1mrbBO8Y7Fx2UURqyKWIBYjzd4W3 v+Abgk0CzUOrFahq3/oixcMnasKScK8A16IT7iv33agdlAwLlIhewOqBtGyGJJA6Vi/F mp60lkvRacrrajGpi4zfU2wlIvLxCSRs3zlPblXC43918yJl8z6tTArVoiQouIiA4xnd jVGNZJV4KhpJSW4/nvQbU2tnrg0BrzbQzxqvRiIjpQBXJYlz2lak/B1++lqgMc+5uoQa V4aXEvfaEhNBCbg3Qa1XYvm9afAaVlSdN4Yd4kyQW9bKRQmIpJZfJ1FlMVIvIjYhMwP1 vO1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature:arc-authentication-results; bh=vfBsfP0gzmAM5CjhnMavb2fqb6d62Jk1xKzL0lq6fqw=; b=RzB4sdVZx3z5lBxAGDQGvc0qfsYbk1MfAoCJ0lc9KmB2HBQqMcLXrJtPaNQxtIl1E+ y34h0Bg8AOQNo48Z6miXezSxSy6/DOJRc3+9sebiYya9wdQZzBBZ57iKGL5WAIm065IS v6t1j2q/veWH76dAR9cmXYWqfhDPmxg42QvzwhvgxIDL5ALU8vQCao5oRvBRJRm3+tSx yPkfEZhYpyzrdoCwK9My3ww1OGrJ8eXgtBfVhl0RY9kt75R3gQzA9la09mylTYXyXvRH zIeFTImg8gWeKgovyvX29zaS0GZQJE9ZaosM/wNoDxKUbtQdRKGDSt8kiP/8erO0MLmw BZKA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=iGal6fGG; spf=pass (google.com: domain of gcc-patches-return-481230-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-481230-patch=linaro.org@gcc.gnu.org" Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id o7-v6si14284734pga.393.2018.07.09.09.40.20 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 Jul 2018 09:40:20 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-481230-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=iGal6fGG; spf=pass (google.com: domain of gcc-patches-return-481230-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-481230-patch=linaro.org@gcc.gnu.org" DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type; q=dns; s=default; b=t45xcNtTEIxgaWuj 5g9OP9qMmiEpsnCocbkdw/gnuemfDgyiRrOYidVsbOzp+Z0mxJKe9R9ETq+9qbOf A2vk1eytETc4/1tAQimyyuLpgch0z2/KWH4nrhc1wDVxS7JA6IghdwtSXZENaK/S O5F2/8OF9DfbrmXkiukQopSerXs= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type; s=default; bh=VEHK9p18Q75bJ+EcXvB+8M QeASM=; b=iGal6fGGLDSsHmzHk69McmxBYLgUZ0pHP3SwttRcC2x9QMcNzUlE0j fjfezL/a9Eig3An2AlhIWJS2dgk8hN9vdrhLpknJAuQdTvZ4Who+iVav+Bbt1qkr PKT5htLQkwuQZJt+1wzCrlB6qpM7FULooy1HHQUDDpbeKzq34scn4= Received: (qmail 90403 invoked by alias); 9 Jul 2018 16:39:20 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 90186 invoked by uid 89); 9 Jul 2018 16:39:18 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, SPF_PASS autolearn=ham version=3.3.2 spammy=pause, UD:md.texi, md.texi, findex X-HELO: foss.arm.com Received: from usa-sjc-mx-foss1.foss.arm.com (HELO foss.arm.com) (217.140.101.70) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Mon, 09 Jul 2018 16:39:12 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 89C3A1682; Mon, 9 Jul 2018 09:39:11 -0700 (PDT) Received: from e120077-lin.cambridge.arm.com (e120077-lin.cambridge.arm.com [10.2.206.23]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 7B6CE3F589; Mon, 9 Jul 2018 09:39:10 -0700 (PDT) From: Richard Earnshaw To: gcc-patches@gcc.gnu.org Cc: Richard Earnshaw , richard.earnshaw@arm.com Subject: [PATCH 1/7] Add __builtin_speculation_safe_value Date: Mon, 9 Jul 2018 17:38:13 +0100 Message-Id: <1531154299-28349-2-git-send-email-Richard.Earnshaw@arm.com> In-Reply-To: <1531154299-28349-1-git-send-email-Richard.Earnshaw@arm.com> References: <1531154299-28349-1-git-send-email-Richard.Earnshaw@arm.com> MIME-Version: 1.0 This patch defines a new intrinsic function __builtin_speculation_safe_value. A generic default implementation is defined which will attempt to use the backend pattern "speculation_safe_barrier". If this pattern is not defined, or if it is not available, then the compiler will emit a warning, but compilation will continue. Note that the test spec-barrier-1.c will currently fail on all targets. This is deliberate, the failure will go away when appropriate action is taken for each target backend. gcc: * builtin-types.def (BT_FN_PTR_PTR_VAR): New function type. (BT_FN_I1_I1_VAR, BT_FN_I2_I2_VAR, BT_FN_I4_I4_VAR): Likewise. (BT_FN_I8_I8_VAR, BT_FN_I16_I16_VAR): Likewise. * builtins.def (BUILT_IN_SPECULATION_SAFE_VALUE_N): New builtin. (BUILT_IN_SPECULATION_SAFE_VALUE_PTR): New internal builtin. (BUILT_IN_SPECULATION_SAFE_VALUE_1): Likewise. (BUILT_IN_SPECULATION_SAFE_VALUE_2): Likewise. (BUILT_IN_SPECULATION_SAFE_VALUE_4): Likewise. (BUILT_IN_SPECULATION_SAFE_VALUE_8): Likewise. (BUILT_IN_SPECULATION_SAFE_VALUE_16): Likewise. * builtins.c (expand_speculation_safe_value): New function. (expand_builtin): Call it. * doc/cpp.texi: Document predefine __HAVE_SPECULATION_SAFE_VALUE. * doc/extend.texi: Document __builtin_speculation_safe_value. * doc/md.texi: Document "speculation_barrier" pattern. * doc/tm.texi.in: Pull in TARGET_SPECULATION_SAFE_VALUE. * doc/tm.texi: Regenerated. * target.def (speculation_safe_value): New hook. * targhooks.c (default_speculation_safe_value): New function. * targhooks.h (default_speculation_safe_value): Add prototype. c-family: * c-common.c (speculation_safe_resolve_size): New function. (speculation_safe_resolve_params): New function. (speculation_safe_resolve_return): New function. (resolve_overloaded_builtin): Handle __builtin_speculation_safe_value. * c-cppbuiltin.c (c_cpp_builtins): Add pre-define for __HAVE_SPECULATION_SAFE_VALUE. testsuite: * gcc.dg/spec-barrier-1.c: New test. * gcc.dg/spec-barrier-2.c: New test. * gcc.dg/spec-barrier-3.c: New test. --- gcc/builtin-types.def | 6 ++ gcc/builtins.c | 57 ++++++++++++++ gcc/builtins.def | 20 +++++ gcc/c-family/c-common.c | 143 ++++++++++++++++++++++++++++++++++ gcc/c-family/c-cppbuiltin.c | 5 +- gcc/doc/cpp.texi | 4 + gcc/doc/extend.texi | 29 +++++++ gcc/doc/md.texi | 15 ++++ gcc/doc/tm.texi | 20 +++++ gcc/doc/tm.texi.in | 2 + gcc/target.def | 23 ++++++ gcc/targhooks.c | 27 +++++++ gcc/targhooks.h | 2 + gcc/testsuite/gcc.dg/spec-barrier-1.c | 40 ++++++++++ gcc/testsuite/gcc.dg/spec-barrier-2.c | 19 +++++ gcc/testsuite/gcc.dg/spec-barrier-3.c | 13 ++++ 16 files changed, 424 insertions(+), 1 deletion(-) create mode 100644 gcc/testsuite/gcc.dg/spec-barrier-1.c create mode 100644 gcc/testsuite/gcc.dg/spec-barrier-2.c create mode 100644 gcc/testsuite/gcc.dg/spec-barrier-3.c diff --git a/gcc/builtin-types.def b/gcc/builtin-types.def index b01095c..70fae35 100644 --- a/gcc/builtin-types.def +++ b/gcc/builtin-types.def @@ -763,6 +763,12 @@ DEF_FUNCTION_TYPE_VAR_1 (BT_FN_VOID_LONG_VAR, BT_VOID, BT_LONG) DEF_FUNCTION_TYPE_VAR_1 (BT_FN_VOID_ULL_VAR, BT_VOID, BT_ULONGLONG) +DEF_FUNCTION_TYPE_VAR_1 (BT_FN_PTR_PTR_VAR, BT_PTR, BT_PTR) +DEF_FUNCTION_TYPE_VAR_1 (BT_FN_I1_I1_VAR, BT_I1, BT_I1) +DEF_FUNCTION_TYPE_VAR_1 (BT_FN_I2_I2_VAR, BT_I2, BT_I2) +DEF_FUNCTION_TYPE_VAR_1 (BT_FN_I4_I4_VAR, BT_I4, BT_I4) +DEF_FUNCTION_TYPE_VAR_1 (BT_FN_I8_I8_VAR, BT_I8, BT_I8) +DEF_FUNCTION_TYPE_VAR_1 (BT_FN_I16_I16_VAR, BT_I16, BT_I16) DEF_FUNCTION_TYPE_VAR_2 (BT_FN_INT_FILEPTR_CONST_STRING_VAR, BT_INT, BT_FILEPTR, BT_CONST_STRING) diff --git a/gcc/builtins.c b/gcc/builtins.c index 91658e8..9f97ecf 100644 --- a/gcc/builtins.c +++ b/gcc/builtins.c @@ -6716,6 +6716,52 @@ expand_builtin_goacc_parlevel_id_size (tree exp, rtx target, int ignore) return target; } +/* Expand a call to __builtin_speculation_safe_value_. MODE + represents the size of the first argument to that call, or VOIDmode + if the argument is a pointer. IGNORE will be true if the result + isn't used. */ +static rtx +expand_speculation_safe_value (machine_mode mode, tree exp, rtx target, + bool ignore) +{ + rtx val, failsafe; + unsigned nargs = call_expr_nargs (exp); + + tree arg0 = CALL_EXPR_ARG (exp, 0); + + if (mode == VOIDmode) + { + mode = TYPE_MODE (TREE_TYPE (arg0)); + gcc_assert (GET_MODE_CLASS (mode) == MODE_INT); + } + + val = expand_expr (arg0, NULL_RTX, mode, EXPAND_NORMAL); + + /* An optional second argument can be used as a failsafe value on + some machines. If it isn't present, then the failsafe value is + assumed to be 0. */ + if (nargs > 1) + { + tree arg1 = CALL_EXPR_ARG (exp, 1); + failsafe = expand_expr (arg1, NULL_RTX, mode, EXPAND_NORMAL); + } + else + failsafe = const0_rtx; + + /* If the result isn't used, the behavior is undefined. It would be + nice to emit a warning here, but path splitting means this might + happen with legitimate code. So simply drop the builtin + expansion in that case; we've handled any side-effects above. */ + if (ignore) + return const0_rtx; + + /* If we don't have a suitable target, create one to hold the result. */ + if (target == NULL) + target = gen_reg_rtx (mode); + + return targetm.speculation_safe_value (mode, target, val, failsafe); +} + /* Expand an expression EXP that calls a built-in function, with result going to TARGET if that's convenient (and in mode MODE if that's convenient). @@ -7827,6 +7873,17 @@ expand_builtin (tree exp, rtx target, rtx subtarget, machine_mode mode, case BUILT_IN_GOACC_PARLEVEL_SIZE: return expand_builtin_goacc_parlevel_id_size (exp, target, ignore); + case BUILT_IN_SPECULATION_SAFE_VALUE_PTR: + return expand_speculation_safe_value (VOIDmode, exp, target, ignore); + + case BUILT_IN_SPECULATION_SAFE_VALUE_1: + case BUILT_IN_SPECULATION_SAFE_VALUE_2: + case BUILT_IN_SPECULATION_SAFE_VALUE_4: + case BUILT_IN_SPECULATION_SAFE_VALUE_8: + case BUILT_IN_SPECULATION_SAFE_VALUE_16: + mode = get_builtin_sync_mode (fcode - BUILT_IN_SPECULATION_SAFE_VALUE_1); + return expand_speculation_safe_value (mode, exp, target, ignore); + default: /* just do library call, if unknown builtin */ break; } diff --git a/gcc/builtins.def b/gcc/builtins.def index aacbd51..b71d89c 100644 --- a/gcc/builtins.def +++ b/gcc/builtins.def @@ -1003,6 +1003,26 @@ DEF_BUILTIN (BUILT_IN_EMUTLS_REGISTER_COMMON, true, true, true, ATTR_NOTHROW_LEAF_LIST, false, !targetm.have_tls) +/* Suppressing speculation. Users are expected to use the first (N) + variant, which will be translated internally into one of the other + types. */ + +DEF_GCC_BUILTIN (BUILT_IN_SPECULATION_SAFE_VALUE_N, "speculation_safe_value", + BT_FN_VOID_VAR, ATTR_NULL) + +DEF_GCC_BUILTIN (BUILT_IN_SPECULATION_SAFE_VALUE_PTR, + "speculation_safe_value_ptr", BT_FN_PTR_PTR_VAR, ATTR_NULL) +DEF_GCC_BUILTIN (BUILT_IN_SPECULATION_SAFE_VALUE_1, "speculation_safe_value_1", + BT_FN_I1_I1_VAR, ATTR_NULL) +DEF_GCC_BUILTIN (BUILT_IN_SPECULATION_SAFE_VALUE_2, "speculation_safe_value_2", + BT_FN_I2_I2_VAR, ATTR_NULL) +DEF_GCC_BUILTIN (BUILT_IN_SPECULATION_SAFE_VALUE_4, "speculation_safe_value_4", + BT_FN_I4_I4_VAR, ATTR_NULL) +DEF_GCC_BUILTIN (BUILT_IN_SPECULATION_SAFE_VALUE_8, "speculation_safe_value_8", + BT_FN_I8_I8_VAR, ATTR_NULL) +DEF_GCC_BUILTIN (BUILT_IN_SPECULATION_SAFE_VALUE_16, + "speculation_safe_value_16", BT_FN_I16_I16_VAR, ATTR_NULL) + /* Exception support. */ DEF_BUILTIN_STUB (BUILT_IN_UNWIND_RESUME, "__builtin_unwind_resume") DEF_BUILTIN_STUB (BUILT_IN_CXA_END_CLEANUP, "__builtin_cxa_end_cleanup") diff --git a/gcc/c-family/c-common.c b/gcc/c-family/c-common.c index f5e1111..32a2de2 100644 --- a/gcc/c-family/c-common.c +++ b/gcc/c-family/c-common.c @@ -6457,6 +6457,121 @@ builtin_type_for_size (int size, bool unsignedp) return type ? type : error_mark_node; } +/* Work out the size of the first argument of a call to + __builtin_speculation_safe_value. Only pointers and integral types + are permitted. Return -1 if the argument type is not supported or + the size is too large; 0 if the argument type is a pointer or the + size if it is integral. */ +static int +speculation_safe_value_resolve_size (tree function, vec *params) +{ + /* Type of the argument. */ + tree type; + int size; + + if (vec_safe_is_empty (params)) + { + error ("too few arguments to function %qE", function); + return -1; + } + + type = TREE_TYPE ((*params)[0]); + if (TREE_CODE (type) == ARRAY_TYPE && c_dialect_cxx ()) + { + /* Force array-to-pointer decay for C++. */ + (*params)[0] = default_conversion ((*params)[0]); + type = TREE_TYPE ((*params)[0]); + } + + if (POINTER_TYPE_P (type)) + return 0; + + if (!INTEGRAL_TYPE_P (type)) + goto incompatible; + + if (!COMPLETE_TYPE_P (type)) + goto incompatible; + + size = tree_to_uhwi (TYPE_SIZE_UNIT (type)); + if (size == 1 || size == 2 || size == 4 || size == 8 || size == 16) + return size; + + incompatible: + /* Issue the diagnostic only if the argument is valid, otherwise + it would be redundant at best and could be misleading. */ + if (type != error_mark_node) + error ("operand type %qT is incompatible with argument %d of %qE", + type, 1, function); + + return -1; +} + +/* Validate and coerce PARAMS, the arguments to ORIG_FUNCTION to fit + the prototype for FUNCTION. The first argument is mandatory, a second + argument, if present, must be type compatible with the first. */ +static bool +speculation_safe_value_resolve_params (location_t loc, tree orig_function, + vec *params) +{ + tree val; + + if (params->length () == 0) + { + error_at (loc, "too few arguments to function %qE", orig_function); + return false; + } + + else if (params->length () > 2) + { + error_at (loc, "too many arguments to function %qE", orig_function); + return false; + } + + val = (*params)[0]; + if (TREE_CODE (TREE_TYPE (val)) == ARRAY_TYPE) + val = default_conversion (val); + if (!(TREE_CODE (TREE_TYPE (val)) == POINTER_TYPE + || TREE_CODE (TREE_TYPE (val)) == INTEGER_TYPE)) + { + error_at (loc, + "expecting argument of type pointer or of type integer " + "for argument 1"); + return false; + } + (*params)[0] = val; + + if (params->length () == 2) + { + tree val2 = (*params)[1]; + if (TREE_CODE (TREE_TYPE (val2)) == ARRAY_TYPE) + val2 = default_conversion (val2); + if (!(TREE_TYPE (val) == TREE_TYPE (val2) + || useless_type_conversion_p (TREE_TYPE (val), TREE_TYPE (val2)))) + { + error_at (loc, "both arguments must be compatible"); + return false; + } + (*params)[1] = val2; + } + + return true; +} + +/* Cast the result of the builtin back to the type of the first argument, + preserving any qualifiers that it might have. */ +static tree +speculation_safe_value_resolve_return (tree first_param, tree result) +{ + tree ptype = TREE_TYPE (first_param); + tree rtype = TREE_TYPE (result); + ptype = TYPE_MAIN_VARIANT (ptype); + + if (tree_int_cst_equal (TYPE_SIZE (ptype), TYPE_SIZE (rtype))) + return convert (ptype, result); + + return result; +} + /* A helper function for resolve_overloaded_builtin in resolving the overloaded __sync_ builtins. Returns a positive power of 2 if the first operand of PARAMS is a pointer to a supported data type. @@ -7111,6 +7226,34 @@ resolve_overloaded_builtin (location_t loc, tree function, /* Handle BUILT_IN_NORMAL here. */ switch (orig_code) { + case BUILT_IN_SPECULATION_SAFE_VALUE_N: + { + int n = speculation_safe_value_resolve_size (function, params); + tree new_function, first_param, result; + enum built_in_function fncode; + + if (n == -1) + return error_mark_node; + else if (n == 0) + fncode = (enum built_in_function)((int)orig_code + 1); + else + fncode + = (enum built_in_function)((int)orig_code + exact_log2 (n) + 2); + + new_function = builtin_decl_explicit (fncode); + first_param = (*params)[0]; + if (!speculation_safe_value_resolve_params (loc, function, params)) + return error_mark_node; + + result = build_function_call_vec (loc, vNULL, new_function, params, + NULL); + + if (result == error_mark_node) + return result; + + return speculation_safe_value_resolve_return (first_param, result); + } + case BUILT_IN_ATOMIC_EXCHANGE: case BUILT_IN_ATOMIC_COMPARE_EXCHANGE: case BUILT_IN_ATOMIC_LOAD: diff --git a/gcc/c-family/c-cppbuiltin.c b/gcc/c-family/c-cppbuiltin.c index bdb5691..0b10e65 100644 --- a/gcc/c-family/c-cppbuiltin.c +++ b/gcc/c-family/c-cppbuiltin.c @@ -1361,7 +1361,10 @@ c_cpp_builtins (cpp_reader *pfile) cpp_define (pfile, "__WCHAR_UNSIGNED__"); cpp_atomic_builtins (pfile); - + + /* Show support for __builtin_speculation_safe_value (). */ + cpp_define (pfile, "__HAVE_SPECULATION_SAFE_VALUE"); + #ifdef DWARF2_UNWIND_INFO if (dwarf2out_do_cfi_asm ()) cpp_define (pfile, "__GCC_HAVE_DWARF2_CFI_ASM"); diff --git a/gcc/doc/cpp.texi b/gcc/doc/cpp.texi index 3f7a8fc..efad2c8 100644 --- a/gcc/doc/cpp.texi +++ b/gcc/doc/cpp.texi @@ -2381,6 +2381,10 @@ If GCC cannot determine the current date, it will emit a warning message These macros are defined when the target processor supports atomic compare and swap operations on operands 1, 2, 4, 8 or 16 bytes in length, respectively. +@item __HAVE_SPECULATION_SAFE_VALUE +This macro is defined with the value 1 to show that this version of GCC +supports @code{__builtin_speculation_safe_value}. + @item __GCC_HAVE_DWARF2_CFI_ASM This macro is defined when the compiler is emitting DWARF CFI directives to the assembler. When this is defined, it is possible to emit those same diff --git a/gcc/doc/extend.texi b/gcc/doc/extend.texi index c7745c4..6eb0c6b 100644 --- a/gcc/doc/extend.texi +++ b/gcc/doc/extend.texi @@ -10935,6 +10935,7 @@ is called and the @var{flag} argument passed to it. @findex __builtin_powi @findex __builtin_powif @findex __builtin_powil +@findex __builtin_speculation_safe_value @findex _Exit @findex _exit @findex abort @@ -11579,6 +11580,34 @@ check its compatibility with @var{size}. @end deftypefn +@deftypefn {Built-in Function} @var{type} __builtin_speculation_safe_value (@var{type} val, @var{type} failval) + +This builtin can be used to help mitigate against unsafe speculative +execution. @var{type} may be any integral type or any pointer type. + +@enumerate +@item +If the CPU is not speculatively executing the code, then @var{val} +is returned. +@item +If the CPU is executing speculatively then either: +@itemize +@item +The function may cause execution to pause until it is known that the +code is no-longer being executed speculatively (in which case +@var{val} can be returned, as above); or +@item +The function may use target-dependent speculation tracking state to cause +@var{failval} to be returned when it is known that speculative +execution has incorrectly predicted a conditional branch operation. +@end itemize +@end enumerate + +The second argument, @var{failval}, is optional and defaults to zero +if omitted. + +@end deftypefn + @deftypefn {Built-in Function} int __builtin_types_compatible_p (@var{type1}, @var{type2}) You can use the built-in function @code{__builtin_types_compatible_p} to diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi index 6d15d99..5de27f6 100644 --- a/gcc/doc/md.texi +++ b/gcc/doc/md.texi @@ -7026,6 +7026,21 @@ should be defined to an instruction that orders both loads and stores before the instruction with respect to loads and stores after the instruction. This pattern has no operands. +@cindex @code{speculation_barrier} instruction pattern +@item @samp{speculation_barrier} +If the target can support speculative execution, then this pattern should +be defined to an instruction that will block subsequent execution until +any prior speculation conditions has been resolved. The pattern must also +ensure that the compiler cannot move memory operations past the barrier, +so it needs to be an UNSPEC_VOLATILE pattern. The pattern has no +operands. + +If this pattern is not defined then the default expansion of +@code{__builtin_speculation_safe_value} will emit a warning. You can +suppress this warning by defining this pattern with a final condition +of @code{0} (zero), which tells the compiler that a speculation +barrier is not needed for this target. + @cindex @code{sync_compare_and_swap@var{mode}} instruction pattern @item @samp{sync_compare_and_swap@var{mode}} This pattern, if defined, emits code for an atomic compare-and-swap diff --git a/gcc/doc/tm.texi b/gcc/doc/tm.texi index 7e2cdc2..681e53b 100644 --- a/gcc/doc/tm.texi +++ b/gcc/doc/tm.texi @@ -11932,6 +11932,26 @@ maintainer is familiar with. @end defmac +@deftypefn {Target Hook} rtx TARGET_SPECULATION_SAFE_VALUE (machine_mode @var{mode}, rtx @var{result}, rtx @var{val}, rtx @var{failval}) +This target hook can be used to generate a target-specific code + sequence that implements the @code{__builtin_speculation_safe_value} + built-in function. The function must always return @var{val} in + @var{result} in mode @var{mode} when the cpu is not executing + speculatively, but must never return that when speculating until it + is known that the speculation will not be unwound. The hook supports + two primary mechanisms for implementing the requirements. The first + is to emit a speculation barrier which forces the processor to wait + until all prior speculative operations have been resolved; the second + is to use a target-specific mechanism that can track the speculation + state and to return @var{failval} if it can determine that + speculation must be unwound at a later time. + + The default implementation simply copies @var{val} to @var{result} and + emits a @code{speculation_barrier} instruction if that is defined. If + @code{speculation_barrier} is not defined for the target a warning will + be generated. +@end deftypefn + @deftypefn {Target Hook} void TARGET_RUN_TARGET_SELFTESTS (void) If selftests are enabled, run any selftests for this target. @end deftypefn diff --git a/gcc/doc/tm.texi.in b/gcc/doc/tm.texi.in index b7b0e8a..6e20afb 100644 --- a/gcc/doc/tm.texi.in +++ b/gcc/doc/tm.texi.in @@ -8107,4 +8107,6 @@ maintainer is familiar with. @end defmac +@hook TARGET_SPECULATION_SAFE_VALUE + @hook TARGET_RUN_TARGET_SELFTESTS diff --git a/gcc/target.def b/gcc/target.def index 112c772..c8bd7f8 100644 --- a/gcc/target.def +++ b/gcc/target.def @@ -4177,6 +4177,29 @@ DEFHOOK hook_bool_void_true) DEFHOOK +(speculation_safe_value, +"This target hook can be used to generate a target-specific code\n\ + sequence that implements the @code{__builtin_speculation_safe_value}\n\ + built-in function. The function must always return @var{val} in\n\ + @var{result} in mode @var{mode} when the cpu is not executing\n\ + speculatively, but must never return that when speculating until it\n\ + is known that the speculation will not be unwound. The hook supports\n\ + two primary mechanisms for implementing the requirements. The first\n\ + is to emit a speculation barrier which forces the processor to wait\n\ + until all prior speculative operations have been resolved; the second\n\ + is to use a target-specific mechanism that can track the speculation\n\ + state and to return @var{failval} if it can determine that\n\ + speculation must be unwound at a later time.\n\ + \n\ + The default implementation simply copies @var{val} to @var{result} and\n\ + emits a @code{speculation_barrier} instruction if that is defined. If\n\ + @code{speculation_barrier} is not defined for the target a warning will\n\ + be generated.", +rtx, (machine_mode mode, rtx result, rtx val, rtx failval), + default_speculation_safe_value) + + +DEFHOOK (can_use_doloop_p, "Return true if it is possible to use low-overhead loops (@code{doloop_end}\n\ and @code{doloop_begin}) for a particular loop. @var{iterations} gives the\n\ diff --git a/gcc/targhooks.c b/gcc/targhooks.c index 7315f1a..2061f07 100644 --- a/gcc/targhooks.c +++ b/gcc/targhooks.c @@ -2306,4 +2306,31 @@ default_select_early_remat_modes (sbitmap) { } +/* Default implementation of the speculation-safe-load builtin. This + implementation simply copies val to result and generates a + speculation_barrier insn, if such a pattern is defined. If + speculation_barrier is not defined at all, a warning is generated. */ + +rtx +default_speculation_safe_value (machine_mode mode ATTRIBUTE_UNUSED, + rtx result, rtx val, + rtx failval ATTRIBUTE_UNUSED) +{ + emit_move_insn (result, val); +#ifdef HAVE_speculation_barrier + /* Assume the target knows what it is doing: if it defines a + speculation barrier, but it is not enabled, then assume that one + isn't needed. */ + if (HAVE_speculation_barrier) + emit_insn (gen_speculation_barrier ()); + +#else + warning_at (input_location, 0, + "this target does not define a speculation barrier; " + "your program will still execute correctly, but speculation " + "will not be inhibited"); +#endif + return result; +} + #include "gt-targhooks.h" diff --git a/gcc/targhooks.h b/gcc/targhooks.h index 4107e22..80ac283 100644 --- a/gcc/targhooks.h +++ b/gcc/targhooks.h @@ -284,4 +284,6 @@ default_excess_precision (enum excess_precision_type ATTRIBUTE_UNUSED); extern bool default_stack_clash_protection_final_dynamic_probe (rtx); extern void default_select_early_remat_modes (sbitmap); +extern rtx default_speculation_safe_value (machine_mode, rtx, rtx, rtx); + #endif /* GCC_TARGHOOKS_H */ diff --git a/gcc/testsuite/gcc.dg/spec-barrier-1.c b/gcc/testsuite/gcc.dg/spec-barrier-1.c new file mode 100644 index 0000000..106f89a --- /dev/null +++ b/gcc/testsuite/gcc.dg/spec-barrier-1.c @@ -0,0 +1,40 @@ +/* { dg-do run } */ +/* { dg-options "-O" } */ + +/* Test that __builtin_speculation_safe_value returns the correct value. */ +/* This test will cause an unfiltered warning to be emitted on targets + that have not implemented support for speculative execution + barriers. They should fix that rather than disabling this + test. */ +char a = 1; +short b = 2; +int c = 3; +long d = 4; +long long e = 5; +int *f = (int*) &c; +#ifdef __SIZEOF_INT128__ +__int128 g = 9; +#endif + +extern void abort (void); + +int main () +{ + if (__builtin_speculation_safe_value (a) != 1) + abort (); + if (__builtin_speculation_safe_value (b) != 2) + abort (); + if (__builtin_speculation_safe_value (c) != 3) + abort (); + if (__builtin_speculation_safe_value (d) != 4) + abort (); + if (__builtin_speculation_safe_value (e) != 5) + abort (); + if (__builtin_speculation_safe_value (f) != &c) + abort (); +#ifdef __SIZEOF_INT128__ + if (__builtin_speculation_safe_value (g) != 9) + abort (); +#endif + return 0; +} diff --git a/gcc/testsuite/gcc.dg/spec-barrier-2.c b/gcc/testsuite/gcc.dg/spec-barrier-2.c new file mode 100644 index 0000000..7e9c497 --- /dev/null +++ b/gcc/testsuite/gcc.dg/spec-barrier-2.c @@ -0,0 +1,19 @@ +/* { dg-do run } */ + +/* Even on targets that don't need the optional failval parameter, + side-effects on the operand should still be calculated. */ + +int x = 3; +volatile int y = 9; + +extern void abort (void); + +int main () +{ + int z = __builtin_speculation_safe_value (x, y++); + if (z != 3 || y != 10) + abort (); + return 0; +} + +/* { dg-prune-output "this target does not define a speculation barrier;" } */ diff --git a/gcc/testsuite/gcc.dg/spec-barrier-3.c b/gcc/testsuite/gcc.dg/spec-barrier-3.c new file mode 100644 index 0000000..3ed4d39 --- /dev/null +++ b/gcc/testsuite/gcc.dg/spec-barrier-3.c @@ -0,0 +1,13 @@ +/* { dg-do compile } */ +/* { dg-options "-Wpedantic" } */ + +/* __builtin_speculation_safe_value returns a value with the same type + as its first argument. There should be a warning if that isn't + type-compatible with the use. */ +int * +f (int x) +{ + return __builtin_speculation_safe_value (x); /* { dg-warning "returning 'int' from a function with return type 'int \\*' makes pointer from integer without a cast" } */ +} + +/* { dg-prune-output "this target does not define a speculation barrier;" } */ From patchwork Mon Jul 9 16:38:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Richard Earnshaw \(lists\)" X-Patchwork-Id: 141467 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp2871622ljj; Mon, 9 Jul 2018 09:39:40 -0700 (PDT) X-Google-Smtp-Source: AAOMgpfg9+J2P86TDkfYvOL4sjTl1Eq6mCbDF5VeoNT1JFBO9X7IskWCmKlkUZ0k8VnA5YycywIo X-Received: by 2002:a62:8917:: with SMTP id v23-v6mr22241421pfd.127.1531154380877; Mon, 09 Jul 2018 09:39:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1531154380; cv=none; d=google.com; s=arc-20160816; b=FKFHu63xiN89a7pBvDK2dfTS1WPtYzCGQwtSsXx34Bg2cm5fsi5+l11ip3z5fSEvBy xjL7u9QRdr+EbYP4yWFkUkppLBUBGlk5aQmxoEyaQGaR9EmEbjTktpvNO22jMQGyFJQg 1uaES14c0Fvi0gNRScqqFMCtYq+04xXhbUzWFyDJoX2jiCT0svJBBhayVJZqHeeoDl0H ASenYcRCTAnFvPAFEs8Qp10APeXU3ztKwvC+uw8LazlMC9G0YzPIjFfnGJAaRaWluUzY xr2bBfdwx+LyLctGswHVizuwfA40MHY6/OduxPCZjF/awo3B1k8m6v85elPykxm3WEB8 ilgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature:arc-authentication-results; bh=m8RvBy1C1FfkZciGh9kBFMLkklyo+EfjWH9/QV02BGM=; b=vt1+xKdLLx65fdMJmRgoLskiXEsV3ZLufO7l8Xav+gJDr57/9PD+RUGRRAB3QPJ37T 2kTx8/mUcl+6ZfE1rnlAjDyEqEalxfRq75nGpOUlCT/XQNICdrOzP36NSvFZ2eUE/b+B EeHWb8CD2lavN4lD2Owl1akS3qAjQAH4/ksa367HGehZUyIe4IyO73NlkH3SOXSeAOqk c+AfEJdytAM8cnP3xoIVgfu+LBFDLhO/a4blB4yYs4O3KwzbP4c2vWanr5EyqytCeXFR /Lgck7fP2nxdg8nxB6qwFvmoE/t2ExEdnooavEZmDlPXpGWky56T8KXqhI2kVmOYzM9H W08g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=ZjQlS9Sl; spf=pass (google.com: domain of gcc-patches-return-481227-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-481227-patch=linaro.org@gcc.gnu.org" Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id a5-v6si14104047pgd.400.2018.07.09.09.39.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 Jul 2018 09:39:40 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-481227-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=ZjQlS9Sl; spf=pass (google.com: domain of gcc-patches-return-481227-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-481227-patch=linaro.org@gcc.gnu.org" DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type; q=dns; s=default; b=xc2ttkBofRi3XrhG HSrrDg5N/yTmkfc3dk0hZTgq1EgiZ4y6cBTUeIrX1Jha02SoYnAeWrUY9f3rw24r ElB5NTKE8/EqsUJ6o+IM+NHQv1AJG5VJyMrINRrHcKzbOgMoTa5uINUWUJlxK275 slkysgKz2SoD7C/58paiYhEFIQQ= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type; s=default; bh=AdI5PGpkQBGUs5pQsM0Q7+ rCcK8=; b=ZjQlS9SlVEySC4ASM/dgQSqOxrYa8ot+FBx1HqnHvrl8+0zOzOaSo3 aLp0wXCDe50KzjXSfkf4ZBPIyGxWdpEQ38QSvpQs75ejjsaRNnKGJlTBT3WUOG2W fDMMtNwGC4gmGUp8asSzgx6VX+Wo8AmyaCTaZWkz/CqcNnyqq66QY= Received: (qmail 89966 invoked by alias); 9 Jul 2018 16:39:16 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 89860 invoked by uid 89); 9 Jul 2018 16:39:15 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, SPF_PASS autolearn=ham version=3.3.2 spammy=multimedia, wireless, Wireless, Multimedia X-HELO: foss.arm.com Received: from usa-sjc-mx-foss1.foss.arm.com (HELO foss.arm.com) (217.140.101.70) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Mon, 09 Jul 2018 16:39:13 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8E8591684; Mon, 9 Jul 2018 09:39:12 -0700 (PDT) Received: from e120077-lin.cambridge.arm.com (e120077-lin.cambridge.arm.com [10.2.206.23]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C7B463F589; Mon, 9 Jul 2018 09:39:11 -0700 (PDT) From: Richard Earnshaw To: gcc-patches@gcc.gnu.org Cc: Richard Earnshaw , richard.earnshaw@arm.com Subject: [PATCH 2/7] Arm - add speculation_barrier pattern Date: Mon, 9 Jul 2018 17:38:14 +0100 Message-Id: <1531154299-28349-3-git-send-email-Richard.Earnshaw@arm.com> In-Reply-To: <1531154299-28349-1-git-send-email-Richard.Earnshaw@arm.com> References: <1531154299-28349-1-git-send-email-Richard.Earnshaw@arm.com> MIME-Version: 1.0 This patch defines a speculation barrier for AArch32. * config/arm/unspecs.md (unspecv): Add VUNSPEC_SPECULATION_BARRIER. * config/arm/arm.md (speculation_barrier): New expand. (speculation_barrier_insn): New pattern. --- gcc/config/arm/arm.md | 21 +++++++++++++++++++++ gcc/config/arm/unspecs.md | 1 + 2 files changed, 22 insertions(+) diff --git a/gcc/config/arm/arm.md b/gcc/config/arm/arm.md index 361a026..ca2a2f5 100644 --- a/gcc/config/arm/arm.md +++ b/gcc/config/arm/arm.md @@ -12012,6 +12012,27 @@ (define_insn "" [(set_attr "length" "4") (set_attr "type" "coproc")]) +(define_expand "speculation_barrier" + [(unspec_volatile [(const_int 0)] VUNSPEC_SPECULATION_BARRIER)] + "TARGET_EITHER" + " + /* Don't emit anything for Thumb1 and suppress the warning from the + generic expansion. */ + if (!TARGET_32BIT) + DONE; + " +) + +;; Generate a hard speculation barrier when we have not enabled speculation +;; tracking. +(define_insn "*speculation_barrier_insn" + [(unspec_volatile [(const_int 0)] VUNSPEC_SPECULATION_BARRIER)] + "TARGET_32BIT" + "isb\;dsb\\tsy" + [(set_attr "type" "block") + (set_attr "length" "8")] +) + ;; Vector bits common to IWMMXT and Neon (include "vec-common.md") ;; Load the Intel Wireless Multimedia Extension patterns diff --git a/gcc/config/arm/unspecs.md b/gcc/config/arm/unspecs.md index b05f85e..1941673 100644 --- a/gcc/config/arm/unspecs.md +++ b/gcc/config/arm/unspecs.md @@ -168,6 +168,7 @@ (define_c_enum "unspecv" [ VUNSPEC_MCRR2 ; Represent the coprocessor mcrr2 instruction. VUNSPEC_MRRC ; Represent the coprocessor mrrc instruction. VUNSPEC_MRRC2 ; Represent the coprocessor mrrc2 instruction. + VUNSPEC_SPECULATION_BARRIER ; Represents an unconditional speculation barrier. ]) ;; Enumerators for NEON unspecs. From patchwork Mon Jul 9 16:38:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Richard Earnshaw \(lists\)" X-Patchwork-Id: 141468 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp2871799ljj; Mon, 9 Jul 2018 09:39:53 -0700 (PDT) X-Google-Smtp-Source: AAOMgpetg6ucKJKp6EhIYzrFWhsgAu4+fR3f2AJ9RHnD4hVH/C5NtQxOXbHJiG1B69wvhZ9/9asp X-Received: by 2002:a17:902:d88d:: with SMTP id b13-v6mr4256901plz.314.1531154393179; Mon, 09 Jul 2018 09:39:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1531154393; cv=none; d=google.com; s=arc-20160816; b=vX9yXmawK1jX/8QOZjNZH6z0HZ/gBWVu+Mbp9DrZ9JHjj6p6UIRZm7ow5KhVFaxguZ SmYzF1DHKnj5frM0DtLCiRcZ3nu+KAQsMzElhMnLLN+5VaKPi/bcbA0TQt65P61EzmVE cKQPbw78Ws3xhMGx4StXPpg96uk8805eQzBTFhFlATdwFbbAka+dTFOUTo/u9Di/MLeD xEKwyE1LTASVxv2MGuxfsKrEoYMzBA2ORsdXpno+pa7L7mJOTH8IuzE8yz4psqznhyvV 2TJB2sQwPJk5tobhNCld48ZAvWxDZzNnaG3NIqsepeYEt5BalTNCeO3TOIWInVLTcv09 1f/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature:arc-authentication-results; bh=WW/GzLumh7o2IWzFZ6L0Y8ZXueWnV6LFQILu2i02ucM=; b=V7Qf0LOFO1ruSUKb9VwuNbMikLnzOMDMd6xnbwUetLY8Vf3NWgol46YykXj/CP5VP2 5R6Uz3Pr1ak0/7QBDn3r9VKSrO10FzWIT1VoWHMF2rmp2Lwy/6p5MW+2FFrgQmZ6e8R4 PPyNe9MkQxxiCDZ0JcRRXxWPyJ6RS+4NDRKFLAEqYpBQaUXbrutBS03gvkxPstLTtA7C ALzE4/6G3/cepLL0AJ085Z4TaENsVYJg9UQyIARt7zPjVb9lhvqE8dh4JW0uGuOjow3u dpEvgRwNHcRAEXSV2b3X/wf43pqA40+Nbr58ES8VCvLHbdq9PV4I+xIyplD/IN+AJ1Ak zFhA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=nCFbfj1Y; spf=pass (google.com: domain of gcc-patches-return-481228-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-481228-patch=linaro.org@gcc.gnu.org" Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id g33-v6si15171850plb.297.2018.07.09.09.39.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 Jul 2018 09:39:53 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-481228-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=nCFbfj1Y; spf=pass (google.com: domain of gcc-patches-return-481228-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-481228-patch=linaro.org@gcc.gnu.org" DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type; q=dns; s=default; b=bgAodLPRjewBUQ8C tx8LkSav/4XsRhVBmDz3FMtc2+FAsB9uZW6eO7FnhxUTl1B4r+EfqNEqrL2dur69 dYAE7L8/0Rki6lv3DvQC2hu0YlZMFh7FDWXGlIvaK4yTHih4YePB7JovwHuYS4I6 4EwNpD+IdrCZ22gr5Oa0I7qru+o= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type; s=default; bh=LXdV+RzKviWvtG9UDSjIeu iVwlk=; b=nCFbfj1YZsSMp+Tuj9bWPRSCuBJZlF4hVBTPY0Mz19fisdwuK3iJJz cJq9SXqnAnDL4T4U+60XLRb34RWdlsHsIhIoK7AATfLHvnQemjM777Mz1gFrtieR z4TREjPl1SW+5Sduw+7LFrzx5XuG8HXXYgkiztVHCeDHi27uE0Vk8= Received: (qmail 90224 invoked by alias); 9 Jul 2018 16:39:18 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 89899 invoked by uid 89); 9 Jul 2018 16:39:16 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, SPF_PASS autolearn=ham version=3.3.2 spammy=Hard, Stuff X-HELO: foss.arm.com Received: from usa-sjc-mx-foss1.foss.arm.com (HELO foss.arm.com) (217.140.101.70) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Mon, 09 Jul 2018 16:39:14 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 934AF16A3; Mon, 9 Jul 2018 09:39:13 -0700 (PDT) Received: from e120077-lin.cambridge.arm.com (e120077-lin.cambridge.arm.com [10.2.206.23]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id CC7BB3F589; Mon, 9 Jul 2018 09:39:12 -0700 (PDT) From: Richard Earnshaw To: gcc-patches@gcc.gnu.org Cc: Richard Earnshaw , richard.earnshaw@arm.com Subject: [PATCH 3/7] AArch64 - add speculation barrier Date: Mon, 9 Jul 2018 17:38:15 +0100 Message-Id: <1531154299-28349-4-git-send-email-Richard.Earnshaw@arm.com> In-Reply-To: <1531154299-28349-1-git-send-email-Richard.Earnshaw@arm.com> References: <1531154299-28349-1-git-send-email-Richard.Earnshaw@arm.com> MIME-Version: 1.0 Similar to Arm, this adds an unconditional speculation barrier for AArch64. * config/aarch64.md (unspecv): Add UNSPECV_SPECULAION_BARRIER. (speculation_barrier): New insn. --- gcc/config/aarch64/aarch64.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/gcc/config/aarch64/aarch64.md b/gcc/config/aarch64/aarch64.md index a014a01..c135ada 100644 --- a/gcc/config/aarch64/aarch64.md +++ b/gcc/config/aarch64/aarch64.md @@ -205,6 +205,7 @@ (define_c_enum "unspecv" [ UNSPECV_SET_FPSR ; Represent assign of FPSR content. UNSPECV_BLOCKAGE ; Represent a blockage UNSPECV_PROBE_STACK_RANGE ; Represent stack range probing. + UNSPECV_SPECULATION_BARRIER ; Represent speculation barrier. ] ) @@ -6093,6 +6094,15 @@ (define_expand "set_clobber_cc" (match_operand 1)) (clobber (reg:CC CC_REGNUM))])]) +;; Hard speculation barrier. +(define_insn "speculation_barrier" + [(unspec_volatile [(const_int 0)] UNSPECV_SPECULATION_BARRIER)] + "" + "isb\;dsb\\tsy" + [(set_attr "length" "8") + (set_attr "type" "block")] +) + ;; AdvSIMD Stuff (include "aarch64-simd.md") From patchwork Mon Jul 9 16:38:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Richard Earnshaw \(lists\)" X-Patchwork-Id: 141469 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp2871967ljj; Mon, 9 Jul 2018 09:40:05 -0700 (PDT) X-Google-Smtp-Source: AAOMgpdUd3rw2irUk9NvhMffZl8weAqr3JNPEJV/h2L9WXslzfOvcS3oby5SiQSlCzh48TtLiRCG X-Received: by 2002:a17:902:b596:: with SMTP id a22-v6mr20902879pls.154.1531154405824; Mon, 09 Jul 2018 09:40:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1531154405; cv=none; d=google.com; s=arc-20160816; b=u1e7T4/8jgLQ0rfTKxJt9LopcGiuIaxlLWMpHl+wXBsQnavsQea5KGl7ZIZJ2mXKYz PGAojZuwOU2rfbpxsNIe7VeqC1GnLEvLZDHYoXTVfnO5JnaK/OEP7i60Il7krFMj+oBN LkAZSpe2or+1pjfId7bu6xdMsf1kmxZHrFBQ4YRfNXYwL8OBE8TrwvCyuxfD57uFkkw1 0RDDKDyP+/2F3h+6w9QbC12PbWb6z6tZZ93fpuNa0RZn5HbXQSMHOXL6U2iWFZozFaoH AKHC3A4XpbATsyug8daWStioM/e+2hO3Q3VH1rqAN1A8tewFSfZUq+qbKewoMoQIr73A 5Nig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature:arc-authentication-results; bh=+/1U6HWux2mB5WLXVKXFRQlld9T5+xhbmFfc3shCFtY=; b=rvP11LCQCncE7oRfE8ZWEpRJULDKUFz5cr/UGgZRfmmUAm5VXhu4Qp6GtY/hm9tPhK bgVknBIXE4aIlMKq52v2JRvkWXHwYqp2OMVquncGVccPuw/PMjzqtKhqe3dW2MQD975J 2ZDngPhc4sggRK+yocDPNx9ldeTyKyZiFFf7BU7p9pZDVK50hmZsU+HrPnDiZ9Vye6UH JLhg/I++lEhXtoo1z1ZYbA0ZIgainZk8an+I73Ndds5TXTffse09L0xfyBAAbPY3xKXx JPX2uvR/cE0uAJbzJSk6tDXPoZPv4wPCbK1c1QUW+ywcMk025EfanbWcXFOceenoZZH8 4n+Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=dTxS6I9o; spf=pass (google.com: domain of gcc-patches-return-481229-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-481229-patch=linaro.org@gcc.gnu.org" Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id n63-v6si14426437pgn.639.2018.07.09.09.40.05 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 Jul 2018 09:40:05 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-481229-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=dTxS6I9o; spf=pass (google.com: domain of gcc-patches-return-481229-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-481229-patch=linaro.org@gcc.gnu.org" DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type; q=dns; s=default; b=AAr6RAVpdy8VckvR fFL/kBfjV4WjJaHIE7uSRppxkaLvx2NNDzrT0g/SanonoITVZUFMXBJk2O9ZqzEI 1OqNErEnQZxPykV/Q4ZKKPY07OjXYOEwL+Y7mN4DH07mhioVbj1Vhb24mBqxWfHD WPRzsPI6SKWkEl9Zc4zcMwtxA5Y= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type; s=default; bh=+YEQ2tHPw+ZsahyKU1Z2uN w2zvc=; b=dTxS6I9o7M2K3KFyokEp4gwF8DE9e5icAbZ5ShwD398wNAdafE2IuE UjM73nYbbE3hgaAk+apDOV6EeyHV3Am1rsPTgOQvje2f6ergE4aa19BO0CDjHZkA VkR5Wb//3f4xukw9nFZwcxB621nTnwdP3HLbQOXbj/qi1U/Ia4sHI= Received: (qmail 90343 invoked by alias); 9 Jul 2018 16:39:19 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 90181 invoked by uid 89); 9 Jul 2018 16:39:18 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, SPF_PASS autolearn=ham version=3.3.2 spammy= X-HELO: foss.arm.com Received: from foss.arm.com (HELO foss.arm.com) (217.140.101.70) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Mon, 09 Jul 2018 16:39:15 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9814B7A9; Mon, 9 Jul 2018 09:39:14 -0700 (PDT) Received: from e120077-lin.cambridge.arm.com (e120077-lin.cambridge.arm.com [10.2.206.23]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D162D3F589; Mon, 9 Jul 2018 09:39:13 -0700 (PDT) From: Richard Earnshaw To: gcc-patches@gcc.gnu.org Cc: Richard Earnshaw , richard.earnshaw@arm.com Subject: [PATCH 4/7] AArch64 - Add new option -mtrack-speculation Date: Mon, 9 Jul 2018 17:38:16 +0100 Message-Id: <1531154299-28349-5-git-send-email-Richard.Earnshaw@arm.com> In-Reply-To: <1531154299-28349-1-git-send-email-Richard.Earnshaw@arm.com> References: <1531154299-28349-1-git-send-email-Richard.Earnshaw@arm.com> MIME-Version: 1.0 This patch doesn't do anything useful, it simply adds a new command-line option -mtrack-speculation to AArch64. Subsequent patches build on this. * config/aarch64/aarch64.opt (mtrack-speculation): New target option. --- gcc/config/aarch64/aarch64.opt | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/gcc/config/aarch64/aarch64.opt b/gcc/config/aarch64/aarch64.opt index 1426b45..bc9b22a 100644 --- a/gcc/config/aarch64/aarch64.opt +++ b/gcc/config/aarch64/aarch64.opt @@ -214,3 +214,7 @@ Target RejectNegative Joined Enum(sve_vector_bits) Var(aarch64_sve_vector_bits) mverbose-cost-dump Common Undocumented Var(flag_aarch64_verbose_cost) Enables verbose cost model dumping in the debug dump files. + +mtrack-speculation +Target Var(aarch64_track_speculation) +Generate code to track when the CPU might be speculating incorrectly. From patchwork Mon Jul 9 16:38:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Richard Earnshaw \(lists\)" X-Patchwork-Id: 141473 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp2872922ljj; Mon, 9 Jul 2018 09:41:04 -0700 (PDT) X-Google-Smtp-Source: AAOMgpebsr2XTiLQMJtBtsYQDeBOSkHbEYil+8CNeCLXMU1BKHAvEXAePiZZcotnIvEL8EKyS5nC X-Received: by 2002:a63:7847:: with SMTP id t68-v6mr13348531pgc.329.1531154464837; Mon, 09 Jul 2018 09:41:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1531154464; cv=none; d=google.com; s=arc-20160816; b=nFDsORguBGHEVxrf9z8T9f9Z4rjNYMRfYptW/AE4ReKtgNCwEc3m2uKIG27b/UWyyt NgPyZQnISZvEXrDQd5fIVhYAVz9v9mRPo44qDoJ44vbyC5uN19zUI4qFWE89R36ezfNn pvuegbmDvrNco67arSuL03pNC4XmzNCOx7PrMaI0Dp95bf8NbjOmBwoJBXkGvn0J+Bv/ FfQ79TqL1XuRUdkqBoxx7m+6ER/gD8Z+dbPyTL2AD/ehYkz9jPvdgQtOwmSmWQp6fd47 WeGL6JDR9mi6MQjURGFGfxHKQjw1V8TBns2ukC7JSK5+Sh3HRgFM71b6lyS0rfxJm96L 24NA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature:arc-authentication-results; bh=zjCMsnokAB+WBEEv+Y/3DhR/3pGtxGLCkJa2ub0nkbk=; b=ndMNuwo4E7PAXB2d16jGiWbJ5qQE+LKxT348r8gPD4gmIG8gwG3VQ1HbdmAJmHEiso IZZJsI1xc1lzWWip8FwVRUk7kXMUB7YVMGxwzuAG8HQZph0cRJhqwafmQB6nXn15ierZ vB/U3x9frmYACrF7gPGm1g2SBntYxM7wFH7NCV2Xnwmn0GT1yTiDLOjL0nUamwPALc7b 53NBCuJjnyMvZpdLqPn5KebBKcgZWhs3+Mm2gkSjAFH12dFPrcZhZPffkjlGMLVkPCtp vBC99f5U5znfoh07PKsAgqsaNWltG0oz7Z6g4XgnrD75fHuq+lGrwdiKf558S3BMKVP/ o07A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=aRSgzuE9; spf=pass (google.com: domain of gcc-patches-return-481233-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-481233-patch=linaro.org@gcc.gnu.org" Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id h17-v6si14119791pgg.218.2018.07.09.09.41.04 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 Jul 2018 09:41:04 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-481233-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=aRSgzuE9; spf=pass (google.com: domain of gcc-patches-return-481233-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-481233-patch=linaro.org@gcc.gnu.org" DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type; q=dns; s=default; b=Gil/DC3b++v2lBuj aI1yVz9bBgD0eN0djgFpbOVdP+l168EXgPz218WDq3kBy++68S9LgQEG7oPohCSZ GzXcsij3DMjVGvD5TlABrYvQ9rNBYMe1gBB+lZnvNO/ROZCWQ+TyEvu5WyE1Cfq2 WIwGmwrrUPdunuUa5XwMGzblr3Q= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type; s=default; bh=2bp0hhV537A7n4HMgoN/gL 7U2nI=; b=aRSgzuE96MlrJ/xZxpCwUxFwgvsBj8ZR4799mVSx1DgYwnaJfAjYXD Pta2llXdq3617FLQBe5s77qIq7IWgwQhBXIdi9sC2Xkuha3ozqwy+9fYvT8M/8O7 5qOuwQBsajTGp6INY8aVvvxvtNUVren87H9Fc8LVrI26PUJjP+1D0= Received: (qmail 91560 invoked by alias); 9 Jul 2018 16:39:28 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 91332 invoked by uid 89); 9 Jul 2018 16:39:27 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, SPF_FAIL autolearn=ham version=3.3.2 spammy= X-HELO: eggs.gnu.org Received: from eggs.gnu.org (HELO eggs.gnu.org) (208.118.235.92) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Mon, 09 Jul 2018 16:39:24 +0000 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fcZC2-0003FS-3c for gcc-patches@gcc.gnu.org; Mon, 09 Jul 2018 12:39:23 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:42130 helo=foss.arm.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fcZC1-0003Ed-PX for gcc-patches@gcc.gnu.org; Mon, 09 Jul 2018 12:39:18 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9C7E51713; Mon, 9 Jul 2018 09:39:15 -0700 (PDT) Received: from e120077-lin.cambridge.arm.com (e120077-lin.cambridge.arm.com [10.2.206.23]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D5F883F589; Mon, 9 Jul 2018 09:39:14 -0700 (PDT) From: Richard Earnshaw To: gcc-patches@gcc.gnu.org Cc: Richard Earnshaw , richard.earnshaw@arm.com Subject: [PATCH 5/7] AArch64 - disable CB[N]Z TB[N]Z when tracking speculation Date: Mon, 9 Jul 2018 17:38:17 +0100 Message-Id: <1531154299-28349-6-git-send-email-Richard.Earnshaw@arm.com> In-Reply-To: <1531154299-28349-1-git-send-email-Richard.Earnshaw@arm.com> References: <1531154299-28349-1-git-send-email-Richard.Earnshaw@arm.com> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 217.140.101.70 The CB[N]Z and TB[N]Z instructions do not expose the comparison through the condition code flags. This makes it impossible to track speculative execution through such a branch. We can handle this relatively easily by simply disabling the patterns in this case. A side effect of this is that the split patterns for the atomic operations need to also avoid generating these instructions. They mostly have simple fall-backs for this already. * config/aarch64/aarch64.md (cb1): Disable when aarch64_track_speculation is true. (tb1): Likewise. * config/aarch64/aarch64.c (aarch64_split_compare_regs): Do not generate CB[N]Z when tracking speculation. (aarch64_split_compare_and_swap): Likewise. (aarch64_split_atomic_op): Likewise. --- gcc/config/aarch64/aarch64.c | 33 ++++++++++++++++++++++++++++++--- gcc/config/aarch64/aarch64.md | 6 +++--- 2 files changed, 33 insertions(+), 6 deletions(-) diff --git a/gcc/config/aarch64/aarch64.c b/gcc/config/aarch64/aarch64.c index 01f35f8..da96afd 100644 --- a/gcc/config/aarch64/aarch64.c +++ b/gcc/config/aarch64/aarch64.c @@ -14465,7 +14465,16 @@ aarch64_split_compare_and_swap (rtx operands[]) if (strong_zero_p) { - x = gen_rtx_NE (VOIDmode, rval, const0_rtx); + if (aarch64_track_speculation) + { + /* Emit an explicit compare instruction, so that we can correctly + track the condition codes. */ + rtx cc_reg = aarch64_gen_compare_reg (NE, rval, const0_rtx); + x = gen_rtx_NE (GET_MODE (cc_reg), cc_reg, const0_rtx); + } + else + x = gen_rtx_NE (VOIDmode, rval, const0_rtx); + x = gen_rtx_IF_THEN_ELSE (VOIDmode, x, gen_rtx_LABEL_REF (Pmode, label2), pc_rtx); aarch64_emit_unlikely_jump (gen_rtx_SET (pc_rtx, x)); @@ -14483,7 +14492,16 @@ aarch64_split_compare_and_swap (rtx operands[]) if (!is_weak) { - x = gen_rtx_NE (VOIDmode, scratch, const0_rtx); + if (aarch64_track_speculation) + { + /* Emit an explicit compare instruction, so that we can correctly + track the condition codes. */ + rtx cc_reg = aarch64_gen_compare_reg (NE, scratch, const0_rtx); + x = gen_rtx_NE (GET_MODE (cc_reg), cc_reg, const0_rtx); + } + else + x = gen_rtx_NE (VOIDmode, scratch, const0_rtx); + x = gen_rtx_IF_THEN_ELSE (VOIDmode, x, gen_rtx_LABEL_REF (Pmode, label1), pc_rtx); aarch64_emit_unlikely_jump (gen_rtx_SET (pc_rtx, x)); @@ -14819,7 +14837,16 @@ aarch64_split_atomic_op (enum rtx_code code, rtx old_out, rtx new_out, rtx mem, aarch64_emit_store_exclusive (mode, cond, mem, gen_lowpart (mode, new_out), model_rtx); - x = gen_rtx_NE (VOIDmode, cond, const0_rtx); + if (aarch64_track_speculation) + { + /* Emit an explicit compare instruction, so that we can correctly + track the condition codes. */ + rtx cc_reg = aarch64_gen_compare_reg (NE, cond, const0_rtx); + x = gen_rtx_NE (GET_MODE (cc_reg), cc_reg, const0_rtx); + } + else + x = gen_rtx_NE (VOIDmode, cond, const0_rtx); + x = gen_rtx_IF_THEN_ELSE (VOIDmode, x, gen_rtx_LABEL_REF (Pmode, label), pc_rtx); aarch64_emit_unlikely_jump (gen_rtx_SET (pc_rtx, x)); diff --git a/gcc/config/aarch64/aarch64.md b/gcc/config/aarch64/aarch64.md index c135ada..259a07d 100644 --- a/gcc/config/aarch64/aarch64.md +++ b/gcc/config/aarch64/aarch64.md @@ -690,7 +690,7 @@ (define_insn "*cb1" (const_int 0)) (label_ref (match_operand 1 "" "")) (pc)))] - "" + "!aarch64_track_speculation" { if (get_attr_length (insn) == 8) return aarch64_gen_far_branch (operands, 1, "Lcb", "\\t%0, "); @@ -720,7 +720,7 @@ (define_insn "*tb1" (label_ref (match_operand 2 "" "")) (pc))) (clobber (reg:CC CC_REGNUM))] - "" + "!aarch64_track_speculation" { if (get_attr_length (insn) == 8) { @@ -756,7 +756,7 @@ (define_insn "*cb1" (label_ref (match_operand 1 "" "")) (pc))) (clobber (reg:CC CC_REGNUM))] - "" + "!aarch64_track_speculation" { if (get_attr_length (insn) == 8) { From patchwork Mon Jul 9 16:38:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Richard Earnshaw \(lists\)" X-Patchwork-Id: 141472 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp2872678ljj; Mon, 9 Jul 2018 09:40:48 -0700 (PDT) X-Google-Smtp-Source: AAOMgpc+Y3jqO3Tj6DhUnJg+75kp1Mie5q83qZo1KCq0TV0XKG8Ql6FKAtvT1lwxuNa0LlOg+DCy X-Received: by 2002:a63:1015:: with SMTP id f21-v6mr18670058pgl.354.1531154448422; Mon, 09 Jul 2018 09:40:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1531154448; cv=none; d=google.com; s=arc-20160816; b=mo8AwEb93o8YueAFX+lF922dAvyZbac5HbHWIgJ3MPn8LdJc0Msm9yQCQiJLvdyqBF C9HMTMCTFLYf9dMfkFxLTNjgR4zIJnan70WBMBgDxgHtEwt38oee9udOL4Igwe6AMPP9 7PfHzk7iwfnrkV8wjhCnnEWbZbwt/qfSXiXKkZthwEPiSGUUmdXIZs3GBjUCYuKPm/uK WFybA1Oq5huxnJrJxOPTV+6awMJy3EetKW9PnCr9EQ5CQuQnDJ66JJ5Pz7+d0QRi31hr QqfQbhb9i3UvLruECL1Nn7vmabWNS2VNacgAFgyppMld/wrlPSysKUk24cPUSWEOYt6Q VdlQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature:arc-authentication-results; bh=oisib0+9ljJUSXGQJGPB8SB1IMlNltc0ibWW/smZnJ0=; b=gdghrVhEzrEMl3JYc5Ve/YgxEjfSHU6EN+yo+oEUkM/ynZ3x2gpcTP5qIjC9yIJcOI 7PKr8LZxKnQbdpdWbDZCRhJDuCeRnGbV47ashQQyfitetOZ/jr69hIbVwAZak5md5osz kbQ8wvH3oQruAkKE7K28qaGjSEw2jc6OsGlgQCqadHsRnZuaJbwsn91kTMCLFDDAWlNv ie9B8CWGYfAE2+WQ3EkXnoJAHps6B+OFJ10ncb1yh6kLRzZuZCaz9dVzNg8IZ6uWmhA+ aPxwkdjQf0VS/1oyvDvJORiREXgS+OB9FTJ2VOUJ1cpCOoO8cQ3AsNneMVn/Ds8sWApl thrg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=XfEwItwd; spf=pass (google.com: domain of gcc-patches-return-481232-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-481232-patch=linaro.org@gcc.gnu.org" Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id s184-v6si13942374pgs.492.2018.07.09.09.40.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 Jul 2018 09:40:48 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-481232-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=XfEwItwd; spf=pass (google.com: domain of gcc-patches-return-481232-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-481232-patch=linaro.org@gcc.gnu.org" DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type; q=dns; s=default; b=iNufzmtFeBOZq6j7 +zb9XwPuDYyURNFoUZOeSfYxGKWLMkA5+OSbSp7VN7S94FGLVUhz0VVYOPBgMCLP 7woI6bEYVE3APRQGjpabT1AWtlUBpQvTKitbyBeXtmIShPcV6cVot0mQcVUX3Edv vqPe0t2MHkBY5GecHS0XAZd/O6k= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type; s=default; bh=MZ2477eFWhcvME02ONfCez 8B5j8=; b=XfEwItwda131zl6/RFB9QdqfsWY9QPABWr2Hkw1+dbX3d9tR6ltI3x Hh/fSqtXTFCWVELSftSGOT8VeeIeefgsoLHrWZftiEiGG6I4ubd0uj0f7q+h+oI1 v4cBTqdwqCsKFcDe6FqZNSxr7Ph6G3sMFlXRoRQtw3Tx2BjQ4Wydk= Received: (qmail 91092 invoked by alias); 9 Jul 2018 16:39:25 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 90970 invoked by uid 89); 9 Jul 2018 16:39:24 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.1 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_ASCII_DIVIDERS, KAM_SHORT, SPF_PASS autolearn=ham version=3.3.2 spammy=1997, tricks, numerous, seq X-HELO: foss.arm.com Received: from usa-sjc-mx-foss1.foss.arm.com (HELO foss.arm.com) (217.140.101.70) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Mon, 09 Jul 2018 16:39:18 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C4AE31684; Mon, 9 Jul 2018 09:39:16 -0700 (PDT) Received: from e120077-lin.cambridge.arm.com (e120077-lin.cambridge.arm.com [10.2.206.23]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DAAAB3F589; Mon, 9 Jul 2018 09:39:15 -0700 (PDT) From: Richard Earnshaw To: gcc-patches@gcc.gnu.org Cc: Richard Earnshaw , richard.earnshaw@arm.com Subject: [PATCH 6/7] AArch64 - new pass to add conditional-branch speculation tracking Date: Mon, 9 Jul 2018 17:38:18 +0100 Message-Id: <1531154299-28349-7-git-send-email-Richard.Earnshaw@arm.com> In-Reply-To: <1531154299-28349-1-git-send-email-Richard.Earnshaw@arm.com> References: <1531154299-28349-1-git-send-email-Richard.Earnshaw@arm.com> MIME-Version: 1.0 This patch is the main part of the speculation tracking code. It adds a new target-specific pass that is run just before the final branch reorg pass (so that it can clean up any new edge insertions we make). The pass is only run with -mtrack-speculation is passed on the command line. One thing that did come to light as part of this was that the stack pointer register was not being permitted in comparision instructions. We rely on that for moving the tracking state between SP and the scratch register at function call boundaries. * config/aarch64/aarch64-speculation.cc: New file. * config/aarch64/aarch64-passes.def (pass_track_speculation): Add before pass_reorder_blocks. * config/aarch64/aarch64-protos.h (make_pass_track_speculation): Add prototype. * config/aarch64/aarch64.c (aarch64_conditional_register_usage): Fix X14 and X15 when tracking speculation. * config/aarch64/aarch64.md (register name constants): Add SPECULATION_TRACKER_REGNUM and SPECULATION_SCRATCH_REGNUM. (unspec): Add UNSPEC_SPECULATION_TRACKER. (speculation_barrier): New insn attribute. (cmp): Allow SP in comparisons. (speculation_tracker): New insn. (speculation_barrier): Add speculation_barrier attribute. * config/aarch64/t-aarch64: Add make rule for aarch64-speculation.o. * config.gcc (aarch64*-*-*): Add aarch64-speculation.o to extra_objs. * doc/invoke.texi (AArch64 Options): Document -mtrack-speculation. --- gcc/config.gcc | 2 +- gcc/config/aarch64/aarch64-passes.def | 1 + gcc/config/aarch64/aarch64-protos.h | 3 +- gcc/config/aarch64/aarch64-speculation.cc | 494 ++++++++++++++++++++++++++++++ gcc/config/aarch64/aarch64.c | 13 + gcc/config/aarch64/aarch64.md | 30 +- gcc/config/aarch64/t-aarch64 | 10 + gcc/doc/invoke.texi | 10 +- 8 files changed, 558 insertions(+), 5 deletions(-) create mode 100644 gcc/config/aarch64/aarch64-speculation.cc diff --git a/gcc/config.gcc b/gcc/config.gcc index 78e84c2..b17fdba 100644 --- a/gcc/config.gcc +++ b/gcc/config.gcc @@ -304,7 +304,7 @@ aarch64*-*-*) extra_headers="arm_fp16.h arm_neon.h arm_acle.h" c_target_objs="aarch64-c.o" cxx_target_objs="aarch64-c.o" - extra_objs="aarch64-builtins.o aarch-common.o cortex-a57-fma-steering.o" + extra_objs="aarch64-builtins.o aarch-common.o cortex-a57-fma-steering.o aarch64-speculation.o" target_gtfiles="\$(srcdir)/config/aarch64/aarch64-builtins.c" target_has_targetm_common=yes ;; diff --git a/gcc/config/aarch64/aarch64-passes.def b/gcc/config/aarch64/aarch64-passes.def index 87747b4..3d6a254 100644 --- a/gcc/config/aarch64/aarch64-passes.def +++ b/gcc/config/aarch64/aarch64-passes.def @@ -19,3 +19,4 @@ . */ INSERT_PASS_AFTER (pass_regrename, 1, pass_fma_steering); +INSERT_PASS_BEFORE (pass_reorder_blocks, 1, pass_track_speculation); diff --git a/gcc/config/aarch64/aarch64-protos.h b/gcc/config/aarch64/aarch64-protos.h index bc11a78..e80ffcf 100644 --- a/gcc/config/aarch64/aarch64-protos.h +++ b/gcc/config/aarch64/aarch64-protos.h @@ -554,7 +554,8 @@ enum aarch64_parse_opt_result aarch64_parse_extension (const char *, std::string aarch64_get_extension_string_for_isa_flags (unsigned long, unsigned long); -rtl_opt_pass *make_pass_fma_steering (gcc::context *ctxt); +rtl_opt_pass *make_pass_fma_steering (gcc::context *); +rtl_opt_pass *make_pass_track_speculation (gcc::context *); poly_uint64 aarch64_regmode_natural_size (machine_mode); diff --git a/gcc/config/aarch64/aarch64-speculation.cc b/gcc/config/aarch64/aarch64-speculation.cc new file mode 100644 index 0000000..2dd06ae --- /dev/null +++ b/gcc/config/aarch64/aarch64-speculation.cc @@ -0,0 +1,494 @@ +/* Speculation tracking and mitigation (e.g. CVE 2017-5753) for AArch64. + Copyright (C) 2018 Free Software Foundation, Inc. + Contributed by ARM Ltd. + + This file is part of GCC. + + GCC is free software; you can redistribute it and/or modify it + under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 3, or (at your option) + any later version. + + GCC is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received a copy of the GNU General Public License + along with GCC; see the file COPYING3. If not see + . */ + +#include "config.h" +#include "system.h" +#include "coretypes.h" +#include "target.h" +#include "rtl.h" +#include "tree-pass.h" +#include "profile-count.h" +#include "cfg.h" +#include "cfgbuild.h" +#include "print-rtl.h" +#include "cfgrtl.h" +#include "function.h" +#include "basic-block.h" +#include "memmodel.h" +#include "emit-rtl.h" +#include "insn-attr.h" +#include "df.h" +#include "tm_p.h" +#include "insn-config.h" +#include "recog.h" + +/* This pass scans the RTL just before the final branch + re-organisation pass. The aim is to identify all places where + there is conditional control flow and to insert code that tracks + any speculative execution of a conditional branch. + + To do this we reserve a call-clobbered register (so that it can be + initialized very early in the function prologue) that can then be + updated each time there is a conditional branch. At each such + branch we then generate a code sequence that uses conditional + select operations that are not subject to speculation themselves + (we ignore for the moment situations where that might not always be + strictly true). For example, a branch sequence such as: + + B.EQ + ... + : + + is transformed to: + + B.EQ + CSEL tracker, tracker, XZr, ne + ... + : + CSEL tracker, tracker, XZr, eq + + Since we start with the tracker initialized to all bits one, if at any + time the predicted control flow diverges from the architectural program + behavior, then the tracker will become zero (but not otherwise). + + The tracker value can be used at any time at which a value needs + guarding against incorrect speculation. This can be done in + several ways, but they all amount to the same thing. For an + untrusted address, or an untrusted offset to a trusted address, we + can simply mask the address with the tracker with the untrusted + value. If the CPU is not speculating, or speculating correctly, + then the value will remain unchanged, otherwise it will be clamped + to zero. For more complex scenarios we can compare the tracker + against zero and use the flags to form a new selection with an + alternate safe value. + + On implementations where the data processing instructions may + themselves produce speculative values, the architecture requires + that a CSDB instruction will resolve such data speculation, so each + time we use the tracker for protecting a vulnerable value we also + emit a CSDB: we do not need to do that each time the tracker itself + is updated. + + At function boundaries, we need to communicate the speculation + tracking state with the caller or the callee. This is tricky + because there is no register available for such a purpose without + creating a new ABI. We deal with this by relying on the principle + that in all real programs the stack pointer, SP will never be NULL + at a function boundary; we can thus encode the speculation state in + SP by clearing SP if the speculation tracker itself is NULL. After + the call we recover the tracking state back from SP into the + tracker register. The results is that a function call sequence is + transformed to + + MOV tmp, SP + AND tmp, tmp, tracker + MOV SP, tmp + BL + CMP SP, #0 + CSETM tracker, ne + + The additional MOV instructions in the pre-call sequence are needed + because SP cannot be used directly with the AND instruction. + + The code inside a function body uses the post-call sequence in the + prologue to establish the tracker and the pre-call sequence in the + epilogue to re-encode the state for the return. + + The code sequences have the nice property that if called from, or + calling a function that does not track speculation then the stack pointer + will always be non-NULL and hence the tracker will be initialized to all + bits one as we need: we lose the ability to fully track speculation in that + case, but we are still architecturally safe. + + Tracking speculation in this way is quite expensive, both in code + size and execution time. We employ a number of tricks to try to + limit this: + + 1) Simple leaf functions with no conditional branches (or use of + the tracker) do not need to establish a new tracker: they simply + carry the tracking state through SP for the duration of the call. + The same is also true for leaf functions that end in a tail-call. + + 2) Back-to-back function calls in a single basic block also do not + need to re-establish the tracker between the calls. Again, we can + carry the tracking state in SP for this period of time unless the + tracker value is needed at that point in time. + + We run the pass just before the final branch reorganization pass so + that we can handle most of the conditional branch cases using the + standard edge insertion code. The reorg pass will hopefully clean + things up for afterwards so that the results aren't too + horrible. */ + +/* Generate a code sequence to clobber SP if speculating incorreclty. */ +static rtx_insn * +aarch64_speculation_clobber_sp () +{ + rtx sp = gen_rtx_REG (DImode, SP_REGNUM); + rtx tracker = gen_rtx_REG (DImode, SPECULATION_TRACKER_REGNUM); + rtx scratch = gen_rtx_REG (DImode, SPECULATION_SCRATCH_REGNUM); + + start_sequence (); + emit_insn (gen_rtx_SET (scratch, sp)); + emit_insn (gen_anddi3 (scratch, scratch, tracker)); + emit_insn (gen_rtx_SET (sp, scratch)); + rtx_insn *seq = get_insns (); + end_sequence (); + return seq; +} + +/* Generate a code sequence to establish the tracker variable from the + contents of SP. */ +static rtx_insn * +aarch64_speculation_establish_tracker () +{ + rtx sp = gen_rtx_REG (DImode, SP_REGNUM); + rtx tracker = gen_rtx_REG (DImode, SPECULATION_TRACKER_REGNUM); + start_sequence (); + rtx cc = aarch64_gen_compare_reg (EQ, sp, const0_rtx); + emit_insn (gen_cstoredi_neg (tracker, + gen_rtx_NE (CCmode, cc, const0_rtx), cc)); + rtx_insn *seq = get_insns (); + end_sequence (); + return seq; +} + +/* Main speculation tracking pass. */ +unsigned int +aarch64_do_track_speculation () +{ + basic_block bb; + bool needs_tracking = false; + bool need_second_pass = false; + rtx_insn *insn; + int fixups_pending = 0; + + FOR_EACH_BB_FN (bb, cfun) + { + insn = BB_END (bb); + + if (dump_file) + fprintf (dump_file, "Basic block %d:\n", bb->index); + + while (insn != BB_HEAD (bb) + && NOTE_P (insn)) + insn = PREV_INSN (insn); + + if (control_flow_insn_p (insn)) + { + if (any_condjump_p (insn)) + { + if (dump_file) + { + fprintf (dump_file, " condjump\n"); + dump_insn_slim (dump_file, insn); + } + + rtx src = SET_SRC (pc_set (insn)); + + /* Check for an inverted jump, where the fall-through edge + appears first. */ + bool inverted = GET_CODE (XEXP (src, 2)) != PC; + /* The other edge must be the PC (we assume that we don't + have conditional return instructions). */ + gcc_assert (GET_CODE (XEXP (src, 1 + !inverted)) == PC); + + rtx cond = copy_rtx (XEXP (src, 0)); + gcc_assert (COMPARISON_P (cond) + && REG_P (XEXP (cond, 0)) + && REGNO (XEXP (cond, 0)) == CC_REGNUM + && XEXP (cond, 1) == const0_rtx); + enum rtx_code inv_cond_code + = reversed_comparison_code (cond, insn); + /* We should be able to reverse all conditions. */ + gcc_assert (inv_cond_code != UNKNOWN); + rtx inv_cond = gen_rtx_fmt_ee (inv_cond_code, GET_MODE (cond), + copy_rtx (XEXP (cond, 0)), + copy_rtx (XEXP (cond, 1))); + if (inverted) + std::swap (cond, inv_cond); + + insert_insn_on_edge (gen_speculation_tracker (cond), + BRANCH_EDGE (bb)); + insert_insn_on_edge (gen_speculation_tracker (inv_cond), + FALLTHRU_EDGE (bb)); + needs_tracking = true; + } + else if (GET_CODE (PATTERN (insn)) == RETURN) + { + /* If we already know we'll need a second pass, don't put + out the return sequence now, or we might end up with + two copies. Instead, we'll do all return statements + during the second pass. However, if this is the + first return insn we've found and we already + know that we'll need to emit the code, we can save a + second pass by emitting the code now. */ + if (needs_tracking && ! need_second_pass) + { + rtx_insn *seq = aarch64_speculation_clobber_sp (); + emit_insn_before (seq, insn); + } + else + { + fixups_pending++; + need_second_pass = true; + } + } + else if (find_reg_note (insn, REG_NON_LOCAL_GOTO, NULL_RTX)) + { + rtx_insn *seq = aarch64_speculation_clobber_sp (); + emit_insn_before (seq, insn); + needs_tracking = true; + } + } + else + { + if (dump_file) + { + fprintf (dump_file, " other\n"); + dump_insn_slim (dump_file, insn); + } + } + } + + FOR_EACH_BB_FN (bb, cfun) + { + rtx_insn *end = BB_END (bb); + rtx_insn *call_insn = NULL; + + if (bb->flags & BB_NON_LOCAL_GOTO_TARGET) + { + rtx_insn *label = NULL; + /* For non-local goto targets we have to recover the + speculation state from SP. Find the last code label at + the head of the block and place the fixup sequence after + that. */ + for (insn = BB_HEAD (bb); insn != end; insn = NEXT_INSN (insn)) + { + if (LABEL_P (insn)) + label = insn; + /* Never put anything before the basic block note. */ + if (NOTE_INSN_BASIC_BLOCK_P (insn)) + label = insn; + if (INSN_P (insn)) + break; + } + + gcc_assert (label); + emit_insn_after (aarch64_speculation_establish_tracker (), label); + } + + /* Scan the insns looking for calls. We need to pass the + speculation tracking state encoded in to SP. After a call we + restore the speculation tracking into the tracker register. + To avoid unnecessary transfers we look for two or more calls + within a single basic block and eliminate, where possible, + any redundant operations. */ + for (insn = BB_HEAD (bb); ; insn = NEXT_INSN (insn)) + { + if (NONDEBUG_INSN_P (insn) + && recog_memoized (insn) >= 0 + && (get_attr_speculation_barrier (insn) + == SPECULATION_BARRIER_TRUE)) + { + if (call_insn) + { + /* This instruction requires the speculation + tracking to be in the tracker register. If there + was an earlier call in this block, we need to + copy the speculation tracking back there. */ + emit_insn_after (aarch64_speculation_establish_tracker (), + call_insn); + call_insn = NULL; + } + + needs_tracking = true; + } + + if (CALL_P (insn)) + { + bool tailcall + = (SIBLING_CALL_P (insn) + || find_reg_note (insn, REG_NORETURN, NULL_RTX)); + + /* Tailcalls are like returns, we can eliminate the + transfer between the tracker register and SP if we + know that this function does not itself need + tracking. */ + if (tailcall && (need_second_pass || !needs_tracking)) + { + /* Don't clear call_insn if it is set - needs_tracking + will be true in that case and so we will end + up putting out mitigation sequences. */ + fixups_pending++; + need_second_pass = true; + break; + } + + needs_tracking = true; + + /* We always need a transfer before the first call in a BB. */ + if (!call_insn) + emit_insn_before (aarch64_speculation_clobber_sp (), insn); + + /* Tail-calls and no-return calls don't need any post-call + reestablishment of the tracker. */ + if (! tailcall) + call_insn = insn; + else + call_insn = NULL; + } + + if (insn == end) + break; + } + + if (call_insn) + { + rtx_insn *seq = aarch64_speculation_establish_tracker (); + + /* Handle debug insns at the end of the BB. Put the extra + insns after them. This ensures that we have consistent + behaviour for the placement of the extra insns between + debug and non-debug builds. */ + for (insn = call_insn; + insn != end && DEBUG_INSN_P (NEXT_INSN (insn)); + insn = NEXT_INSN (insn)) + ; + + if (insn == end) + { + edge e = find_fallthru_edge (bb->succs); + /* We need to be very careful about some calls that + appear at the end of a basic block. If the call + involves exceptions, then the compiler may depend on + this being the last instruction in the block. The + easiest way to handle this is to commit the new + instructions on the fall-through edge and to let + commit_edge_insertions clean things up for us. + + Sometimes, eg with OMP, there may not even be an + outgoing edge after the call. In that case, there's + not much we can do, presumably the compiler has + decided that the call can never return in this + context. */ + if (e) + { + /* We need to set the location lists explicitly in + this case. */ + if (! INSN_P (seq)) + { + start_sequence (); + emit_insn (seq); + seq = get_insns (); + end_sequence (); + } + + for (rtx_insn *list = seq; list; list = NEXT_INSN (list)) + INSN_LOCATION (list) = INSN_LOCATION (call_insn); + + insert_insn_on_edge (seq, e); + } + } + else + emit_insn_after (seq, call_insn); + } + } + + if (needs_tracking) + { + if (need_second_pass) + { + /* We found a return instruction before we found out whether + or not we need to emit the tracking code, but we now + know we do. Run quickly over the basic blocks and + fix up the return insns. */ + FOR_EACH_BB_FN (bb, cfun) + { + insn = BB_END (bb); + + while (insn != BB_HEAD (bb) + && NOTE_P (insn)) + insn = PREV_INSN (insn); + + if ((control_flow_insn_p (insn) + && GET_CODE (PATTERN (insn)) == RETURN) + || (CALL_P (insn) + && (SIBLING_CALL_P (insn) + || find_reg_note (insn, REG_NORETURN, NULL_RTX)))) + { + rtx_insn *seq = aarch64_speculation_clobber_sp (); + emit_insn_before (seq, insn); + fixups_pending--; + } + } + gcc_assert (fixups_pending == 0); + } + + /* Set up the initial value of the tracker, using the incoming SP. */ + insert_insn_on_edge (aarch64_speculation_establish_tracker (), + single_succ_edge (ENTRY_BLOCK_PTR_FOR_FN (cfun))); + commit_edge_insertions (); + } + + return 0; +} + +namespace { + +const pass_data pass_data_aarch64_track_speculation = +{ + RTL_PASS, /* type. */ + "speculation", /* name. */ + OPTGROUP_NONE, /* optinfo_flags. */ + TV_MACH_DEP, /* tv_id. */ + 0, /* properties_required. */ + 0, /* properties_provided. */ + 0, /* properties_destroyed. */ + 0, /* todo_flags_start. */ + 0 /* todo_flags_finish. */ +}; + +class pass_track_speculation : public rtl_opt_pass +{ + public: + pass_track_speculation(gcc::context *ctxt) + : rtl_opt_pass(pass_data_aarch64_track_speculation, ctxt) + {} + + /* opt_pass methods: */ + virtual bool gate (function *) + { + return aarch64_track_speculation; + } + + virtual unsigned int execute (function *) + { + return aarch64_do_track_speculation (); + } +}; // class pass_track_speculation. +} // anon namespace. + +/* Create a new pass instance. */ +rtl_opt_pass * +make_pass_track_speculation (gcc::context *ctxt) +{ + return new pass_track_speculation (ctxt); +} diff --git a/gcc/config/aarch64/aarch64.c b/gcc/config/aarch64/aarch64.c index da96afd..b11d768 100644 --- a/gcc/config/aarch64/aarch64.c +++ b/gcc/config/aarch64/aarch64.c @@ -12573,6 +12573,19 @@ aarch64_conditional_register_usage (void) fixed_regs[i] = 1; call_used_regs[i] = 1; } + + /* When tracking speculation, we need a couple of call-clobbered registers + to track the speculation state. It would be nice to just use + IP0 and IP1, but currently there are numerous places that just + assume these registers are free for other uses (eg pointer + authentication). */ + if (aarch64_track_speculation) + { + fixed_regs[SPECULATION_TRACKER_REGNUM] = 1; + call_used_regs[SPECULATION_TRACKER_REGNUM] = 1; + fixed_regs[SPECULATION_SCRATCH_REGNUM] = 1; + call_used_regs[SPECULATION_SCRATCH_REGNUM] = 1; + } } /* Walk down the type tree of TYPE counting consecutive base elements. diff --git a/gcc/config/aarch64/aarch64.md b/gcc/config/aarch64/aarch64.md index 259a07d..528d03d 100644 --- a/gcc/config/aarch64/aarch64.md +++ b/gcc/config/aarch64/aarch64.md @@ -88,6 +88,10 @@ (define_constants (P13_REGNUM 81) (P14_REGNUM 82) (P15_REGNUM 83) + ;; A couple of call-clobbered registers that we need to reserve when + ;; tracking speculation this is not ABI, so is subject to change. + (SPECULATION_TRACKER_REGNUM 15) + (SPECULATION_SCRATCH_REGNUM 14) ] ) @@ -195,6 +199,7 @@ (define_c_enum "unspec" [ UNSPEC_CLASTB UNSPEC_FADDA UNSPEC_REV_SUBREG + UNSPEC_SPECULATION_TRACKER ]) (define_c_enum "unspecv" [ @@ -287,6 +292,11 @@ (define_attr "length" "" ;; no predicated insns. (define_attr "predicated" "yes,no" (const_string "no")) +;; Set to true on an insn that requires the speculation tracking state to be +;; in the tracking register before the insn issues. Otherwise the compiler +;; may chose to hold the tracking state encoded in SP. +(define_attr "speculation_barrier" "true,false" (const_string "false")) + ;; ------------------------------------------------------------------- ;; Pipeline descriptions and scheduling ;; ------------------------------------------------------------------- @@ -3079,7 +3089,7 @@ (define_insn "*divsi3_uxtw" (define_insn "cmp" [(set (reg:CC CC_REGNUM) - (compare:CC (match_operand:GPI 0 "register_operand" "r,r,r") + (compare:CC (match_operand:GPI 0 "register_operand" "rk,rk,rk") (match_operand:GPI 1 "aarch64_plus_operand" "r,I,J")))] "" "@ @@ -6088,6 +6098,21 @@ (define_expand "doloop_end" DONE; }) +;; Track speculation through conditional branches. We assume that +;; SPECULATION_TRACKER_REGNUM is reserved for this purpose when necessary. +(define_insn "speculation_tracker" + [(set (reg:DI SPECULATION_TRACKER_REGNUM) + (unspec [(reg:DI SPECULATION_TRACKER_REGNUM) (match_operand 0)] + UNSPEC_SPECULATION_TRACKER))] + "" + { + operands[1] = gen_rtx_REG (DImode, SPECULATION_TRACKER_REGNUM); + output_asm_insn ("csel\\t%1, %1, xzr, %m0", operands); + return ""; + } + [(set_attr "type" "csel")] +) + ;; Helper for aarch64.c code. (define_expand "set_clobber_cc" [(parallel [(set (match_operand 0) @@ -6100,7 +6125,8 @@ (define_insn "speculation_barrier" "" "isb\;dsb\\tsy" [(set_attr "length" "8") - (set_attr "type" "block")] + (set_attr "type" "block") + (set_attr "speculation_barrier" "true")] ) ;; AdvSIMD Stuff diff --git a/gcc/config/aarch64/t-aarch64 b/gcc/config/aarch64/t-aarch64 index 0be1f0d..5d54853 100644 --- a/gcc/config/aarch64/t-aarch64 +++ b/gcc/config/aarch64/t-aarch64 @@ -67,6 +67,16 @@ cortex-a57-fma-steering.o: $(srcdir)/config/aarch64/cortex-a57-fma-steering.c \ $(COMPILER) -c $(ALL_COMPILERFLAGS) $(ALL_CPPFLAGS) $(INCLUDES) \ $(srcdir)/config/aarch64/cortex-a57-fma-steering.c +aarch64-speculation.o: $(srcdir)/config/aarch64/aarch64-speculation.cc \ + $(CONFIG_H) \ + $(SYSTEM_H) \ + $(TM_H) \ + $(TARGET_H) \ + $(RTL_BASE_H) \ + $(TREE_PASS_H) + $(COMPILER) -c $(ALL_COMPILERFLAGS) $(ALL_SPPFLAGS) $(INCLUDES) \ + $(srcdir)/config/aarch64/aarch64-speculation.cc + comma=, MULTILIB_OPTIONS = $(subst $(comma),/, $(patsubst %, mabi=%, $(subst $(comma),$(comma)mabi=,$(TM_MULTILIB_CONFIG)))) MULTILIB_DIRNAMES = $(subst $(comma), ,$(TM_MULTILIB_CONFIG)) diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi index e0e59f6..db39304 100644 --- a/gcc/doc/invoke.texi +++ b/gcc/doc/invoke.texi @@ -598,7 +598,7 @@ Objective-C and Objective-C++ Dialects}. -mpc-relative-literal-loads @gol -msign-return-address=@var{scope} @gol -march=@var{name} -mcpu=@var{name} -mtune=@var{name} @gol --moverride=@var{string} -mverbose-cost-dump} +-moverride=@var{string} -mverbose-cost-dump -mtrack-speculation} @emph{Adapteva Epiphany Options} @gccoptlist{-mhalf-reg-file -mprefer-short-insn-regs @gol @@ -14649,6 +14649,14 @@ This option only has an effect if @option{-ffast-math} or precision of division results to about 16 bits for single precision and to 32 bits for double precision. +@item -mtrack-speculation +@itemx -mno-track-speculation +Enable or disable generation of additional code to track speculative +execution through conditional branches. The tracking state can then +be used by the compiler when expanding calls to +@code{__builtin_speculation_safe_copy} to permit a more efficient code +sequence to be generated. + @item -march=@var{name} @opindex march Specify the name of the target architecture and, optionally, one or From patchwork Mon Jul 9 16:38:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Richard Earnshaw \(lists\)" X-Patchwork-Id: 141471 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp2872430ljj; Mon, 9 Jul 2018 09:40:33 -0700 (PDT) X-Google-Smtp-Source: AAOMgpeTWx6spqCIkDA2N+Kf/4lrWIrfh4kJP20G5c5E/fMYocagA1aOhcIrFRf9zetYX9RZxOkd X-Received: by 2002:a62:da07:: with SMTP id c7-v6mr22025388pfh.106.1531154433257; Mon, 09 Jul 2018 09:40:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1531154433; cv=none; d=google.com; s=arc-20160816; b=Bk20DfKT8+EbB8UNY8Ov9RSDZBp5h3UVSIXIBalBlko3gi+Fbt6RcHbzSCMloOT57w gybBt0LJAe3HERBL737PlBlSSgVExQULaqs0PdmMFsAqVksvzOYMj5FhFHEF9bPJ6J+Z DuThw96Lo0yt89hQKXyq8obORT/N24VnV5bzZK2/h4V+ZDuCkfY+3Fkh1dWS+u6/5DoG igk6ri44ObX326U/AgpZ6DSfG2+AngD47KJWH0XBuWL6BK7hDPxvb8K3vrBRhwm2mIXq ieKgxBelco0C78C/fhMIOn5DTYZmWkslwrpAzZc+ucC3rfi0efspetqDuN2zPvgtJ0EO qtog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature:arc-authentication-results; bh=FpJM0xqBkPDWHF9HHuR6UArIFGiauFAe09HCYtzIAVc=; b=SOTbji2VXZ2/NXWZucsk/MN2hs465WrFgwR5VzxdQnQUbJ9w0TQCVbH7SjYYGs8Bhn XMW05UcfcUL6hN/ZDtQxZLykplMPsnXDmlcZTEJluYOpyW52FiZdjL9KBpS24lfhfJGD nc42I6amc/J+iKHEshm2MDwhJxaG5fup5HH2Hw09eEldcYPee+tOqcRkD+iTmPeERdY4 ghP+JmK3gIddzm8RTGuhtzERIrZTmyYyizc/CXLnVwp9Aund/4nsf6vjhXizn/KXzDuj LAeEBUjgvZzlgu0EMpEtdGNwvrjTqLevF7bEczGFXYQgC/a0VR6xd4PndQD2aUK+TAVc QJXA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=BlWPstOT; spf=pass (google.com: domain of gcc-patches-return-481231-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-481231-patch=linaro.org@gcc.gnu.org" Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id t16-v6si13934613pga.442.2018.07.09.09.40.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 Jul 2018 09:40:33 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-481231-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=BlWPstOT; spf=pass (google.com: domain of gcc-patches-return-481231-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-481231-patch=linaro.org@gcc.gnu.org" DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type; q=dns; s=default; b=rIeWJ9iugSgBAUn1 X73xbkR9P56nFeku8VSmUq/h3+Nv/X7fhcQsjiDKl8Ej5IL4xQJVKwkJKN5EB2Km LuRl36oHIYQKtyHEEU9xFSG329CRjiPMqR1aR2qW3OZeHOIebsxvzLEDrihgnX5S TkCB0bG0/5jV9hzsHAE4UYBjC1I= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type; s=default; bh=VY3LNvFwtzFqdVoD1pwIR8 ol+pg=; b=BlWPstOTbgxIyOvRQkAj4F7zu2zixMjSxBsIkmvc1InFSqTrCrIYaa mEX1HME8pQ2R1tfH7TO6yllw3VLZgZgKGgQjbIJZ8Bl5dconvirenJzNL9WBlEvq TON5h4IDvA8qVzWrYYlhM2bFGq1Zzs38zzm7YugqT14+qKTA7PUOc= Received: (qmail 90769 invoked by alias); 9 Jul 2018 16:39:22 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 90641 invoked by uid 89); 9 Jul 2018 16:39:21 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, SPF_PASS autolearn=ham version=3.3.2 spammy=falling, tracker X-HELO: foss.arm.com Received: from usa-sjc-mx-foss1.foss.arm.com (HELO foss.arm.com) (217.140.101.70) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Mon, 09 Jul 2018 16:39:19 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C98CA16A3; Mon, 9 Jul 2018 09:39:17 -0700 (PDT) Received: from e120077-lin.cambridge.arm.com (e120077-lin.cambridge.arm.com [10.2.206.23]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0EBFF3F589; Mon, 9 Jul 2018 09:39:16 -0700 (PDT) From: Richard Earnshaw To: gcc-patches@gcc.gnu.org Cc: Richard Earnshaw , richard.earnshaw@arm.com Subject: [PATCH 7/7] AArch64 - use CSDB based sequences if speculation tracking is enabled Date: Mon, 9 Jul 2018 17:38:19 +0100 Message-Id: <1531154299-28349-8-git-send-email-Richard.Earnshaw@arm.com> In-Reply-To: <1531154299-28349-1-git-send-email-Richard.Earnshaw@arm.com> References: <1531154299-28349-1-git-send-email-Richard.Earnshaw@arm.com> MIME-Version: 1.0 In this final patch, now that we can track speculation through conditional branches, we can use this information to use a less expensive CSDB based speculation barrier. * config/aarch64/iterators.md (ALLI_TI): New iterator. * config/aarch64/aarch64.md (despeculate_copy): New expand. (despeculate_copy_insn): New insn. (despeculate_copyti_insn): New insn. (despeculate_simple): New insn (despeculate_simpleti): New insn. * config/aarch64/aarch64.c (aarch64_speculation_safe_value): New function. (TARGET_SPECULATION_SAFE_VALUE): Redefine to aarch64_speculation_safe_value. --- gcc/config/aarch64/aarch64.c | 42 ++++++++++++++++++ gcc/config/aarch64/aarch64.md | 96 +++++++++++++++++++++++++++++++++++++++++ gcc/config/aarch64/iterators.md | 3 ++ 3 files changed, 141 insertions(+) diff --git a/gcc/config/aarch64/aarch64.c b/gcc/config/aarch64/aarch64.c index b11d768..b30b857 100644 --- a/gcc/config/aarch64/aarch64.c +++ b/gcc/config/aarch64/aarch64.c @@ -17648,6 +17648,45 @@ aarch64_select_early_remat_modes (sbitmap modes) } } +/* Override the default target speculation_safe_value. */ +static rtx +aarch64_speculation_safe_value (machine_mode mode, + rtx result, rtx val, rtx failval) +{ + /* Maybe we should warn if falling back to hard barriers. They are + likely to be noticably more expensive than the alternative below. */ + if (!aarch64_track_speculation) + return default_speculation_safe_value (mode, result, val, failval); + + if (!REG_P (val)) + val = copy_to_mode_reg (mode, val); + + if (!aarch64_reg_or_zero (failval, mode)) + failval = copy_to_mode_reg (mode, failval); + + switch (mode) + { + case E_QImode: + emit_insn (gen_despeculate_copyqi (result, val, failval)); + break; + case E_HImode: + emit_insn (gen_despeculate_copyhi (result, val, failval)); + break; + case E_SImode: + emit_insn (gen_despeculate_copysi (result, val, failval)); + break; + case E_DImode: + emit_insn (gen_despeculate_copydi (result, val, failval)); + break; + case E_TImode: + emit_insn (gen_despeculate_copyti (result, val, failval)); + break; + default: + gcc_unreachable (); + } + return result; +} + /* Target-specific selftests. */ #if CHECKING_P @@ -18117,6 +18156,9 @@ aarch64_libgcc_floating_mode_supported_p #undef TARGET_SELECT_EARLY_REMAT_MODES #define TARGET_SELECT_EARLY_REMAT_MODES aarch64_select_early_remat_modes +#undef TARGET_SPECULATION_SAFE_VALUE +#define TARGET_SPECULATION_SAFE_VALUE aarch64_speculation_safe_value + #if CHECKING_P #undef TARGET_RUN_TARGET_SELFTESTS #define TARGET_RUN_TARGET_SELFTESTS selftest::aarch64_run_selftests diff --git a/gcc/config/aarch64/aarch64.md b/gcc/config/aarch64/aarch64.md index 528d03d..cbcada2 100644 --- a/gcc/config/aarch64/aarch64.md +++ b/gcc/config/aarch64/aarch64.md @@ -6129,6 +6129,102 @@ (define_insn "speculation_barrier" (set_attr "speculation_barrier" "true")] ) +;; Support for __builtin_speculation_safe_value when we have speculation +;; tracking enabled. Use the speculation tracker to decide whether to +;; copy operand 1 to the target, or to copy the fail value (operand 2). +(define_expand "despeculate_copy" + [(set (match_operand:ALLI_TI 0 "register_operand" "=r") + (unspec_volatile:ALLI_TI + [(match_operand:ALLI_TI 1 "register_operand" "r") + (match_operand:ALLI_TI 2 "aarch64_reg_or_zero" "rZ") + (use (reg:DI SPECULATION_TRACKER_REGNUM)) + (clobber (reg:CC CC_REGNUM))] UNSPECV_SPECULATION_BARRIER))] + "" + " + { + if (operands[2] == const0_rtx) + { + rtx tracker; + if (mode == TImode) + tracker = gen_rtx_REG (DImode, SPECULATION_TRACKER_REGNUM); + else + tracker = gen_rtx_REG (mode, SPECULATION_TRACKER_REGNUM); + + emit_insn (gen_despeculate_simple (operands[0], operands[1], + tracker)); + DONE; + } + } + " +) + +;; Pattern to match despeculate_copy +(define_insn "*despeculate_copy_insn" + [(set (match_operand:ALLI 0 "register_operand" "=r") + (unspec_volatile:ALLI + [(match_operand:ALLI 1 "register_operand" "r") + (match_operand:ALLI 2 "aarch64_reg_or_zero" "rZ") + (use (reg:DI SPECULATION_TRACKER_REGNUM)) + (clobber (reg:CC CC_REGNUM))] UNSPECV_SPECULATION_BARRIER))] + "" + { + operands[3] = gen_rtx_REG (DImode, SPECULATION_TRACKER_REGNUM); + output_asm_insn ("cmp\\t%3, #0\;csel\\t%0, %1, %2, ne\;csdb", + operands); + return ""; + } + [(set_attr "length" "12") + (set_attr "type" "block") + (set_attr "speculation_barrier" "true")] +) + +;; Pattern to match despeculate_copyti +(define_insn "*despeculate_copyti_insn" + [(set (match_operand:TI 0 "register_operand" "=r") + (unspec_volatile:TI + [(match_operand:TI 1 "register_operand" "r") + (match_operand:TI 2 "aarch64_reg_or_zero" "rZ") + (use (reg:DI SPECULATION_TRACKER_REGNUM)) + (clobber (reg:CC CC_REGNUM))] UNSPECV_SPECULATION_BARRIER))] + "" + { + operands[3] = gen_rtx_REG (DImode, SPECULATION_TRACKER_REGNUM); + output_asm_insn + ("cmp\\t%3, #0\;csel\\t%0, %1, %2, ne\;csel\\t%R0, %R1, %R2, ne\;csdb", + operands); + return ""; + } + [(set_attr "length" "16") + (set_attr "type" "block") + (set_attr "speculation_barrier" "true")] +) + +(define_insn "despeculate_simple" + [(set (match_operand:ALLI 0 "register_operand" "=r") + (unspec_volatile:ALLI + [(match_operand:ALLI 1 "register_operand" "r") + (use (match_operand:ALLI 2 "register_operand" ""))] + UNSPECV_SPECULATION_BARRIER))] + "" + "and\\t%0, %1, %2\;csdb" + [(set_attr "type" "block") + (set_attr "length" "8") + (set_attr "speculation_barrier" "true")] +) + +(define_insn "despeculate_simpleti" + [(set (match_operand:TI 0 "register_operand" "=r") + (unspec_volatile:TI + [(match_operand:TI 1 "register_operand" "r") + (use (match_operand:DI 2 "register_operand" ""))] + UNSPECV_SPECULATION_BARRIER))] + "" + "and\\t%0, %1, %2\;and\\t%R0, %R1, %2\;csdb" + [(set_attr "type" "block") + (set_attr "length" "12") + (set_attr "speculation_barrier" "true")] +) + ;; AdvSIMD Stuff (include "aarch64-simd.md") diff --git a/gcc/config/aarch64/iterators.md b/gcc/config/aarch64/iterators.md index 949fd2a..d2b7fff 100644 --- a/gcc/config/aarch64/iterators.md +++ b/gcc/config/aarch64/iterators.md @@ -35,6 +35,9 @@ (define_mode_iterator SHORT [QI HI]) ;; Iterator for all integer modes (up to 64-bit) (define_mode_iterator ALLI [QI HI SI DI]) +;; Iterator for all integer modes (up to 128-bit) +(define_mode_iterator ALLI_TI [QI HI SI DI TI]) + ;; Iterator for all integer modes that can be extended (up to 64-bit) (define_mode_iterator ALLX [QI HI SI])