From patchwork Fri Nov 17 09:20:15 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Sandiford X-Patchwork-Id: 119107 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp282842qgn; Fri, 17 Nov 2017 01:21:36 -0800 (PST) X-Google-Smtp-Source: AGs4zMZQXITwLrmcUMmqrMbvyi+lb5XvvZ/Gtk5qczhPAa70XiNqIwEPING3xKXrV0n9hA/I6HBm X-Received: by 10.159.216.141 with SMTP id s13mr4622771plp.377.1510910496597; Fri, 17 Nov 2017 01:21:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510910496; cv=none; d=google.com; s=arc-20160816; b=FaBj41N67rMXek1zmAsuTMICT7xDcspO349ypblBi5ZrgTuxqsrxWj2Cmpzl4ELzPa 4KmYRoRnY27skU8fFY/2h8D9jCOdqNqPFpzFrRU06lfmaJYKGZuVio6/EjsqA7uEt25Z 8C2k9600QDdhZjsoSQ0/tT+vXa0OcNLtt4Vqg4GJGxwN3g9bGARoBpDr86fQ1HR+TfTk DkKsNvhaVu9U7S52cd+3Su7vZr1UExRcLNzMcNau9K5P+iaRWmhIVJZPtBxXoRDlUskG KHUEj+eHpSFSr078xG+PCcNYY3hzw48PNwTa0bebcfdUph3br232oOjrCySxf9uU42OH JRYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:mail-followup-to:to:from:delivered-to:sender:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence :mailing-list:dkim-signature:domainkey-signature :arc-authentication-results; bh=zm6j6rgksZOrgkaLnP7anExteg+1rW0C5P0fr5AfvvU=; b=d8Z5fV6QA98xPZ1rmp+TmYmVJg+sPl6gwPyrUXpXMINHUU2vD+bNjg8fk6hKVYCDgv 5zWIQy4vPk71MywjK92MxqywS1Q7yJbiAnDV7wmr6iAbCVosrZ+Zt2kj8GQJ/n0OfimT GTvDBdrpX9gscBnR1/W4y0C0UEEt99t5k9hsPeqasyg0W/C1eyMK/M+HyK75rUL33yRG SR3P3BkeF/TMqaxCrdaqbMuBfpBP4RZzLsndBe4nE0SYQKkDP1OEZ/KVwilHirYpPpgq FfjiCXEZRyc+WFdGdWWNRDJ1zsog8mrmwnZ/7ucG4961/DagNtWQEv3Rt7rdPcEQT7jI 249Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=rmP6UuYN; spf=pass (google.com: domain of gcc-patches-return-467100-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-467100-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id i8si2384738pgr.690.2017.11.17.01.21.36 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 17 Nov 2017 01:21:36 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-return-467100-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=rmP6UuYN; spf=pass (google.com: domain of gcc-patches-return-467100-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-467100-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:references:date:in-reply-to:message-id:mime-version :content-type; q=dns; s=default; b=L7FmDUNqQm34EOvd83yLyhyimNok2 OXKtGwPpKlLawlftKRQDzRH6rVNLo00gy+wz9tQww/8/+c9kHIRl0BpgmpIuXST2 d0AITftQF2mCPTDpUJHmKtUY+rstJlTTTgvqSiYEIhLYkJrnOq/eqfZiqh3LflcG 11Hi+ZT2HYsffA= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:references:date:in-reply-to:message-id:mime-version :content-type; s=default; bh=eOYHOVSyG4AvYyTxy/NWNA0TpUc=; b=rmP 6UuYNvezetBZy+85uRdPNl8UPsTHm6UiHXQ78fbvjDFgPN3eLXDLW509BCT1iVW9 7xL/TQA1EldbAig3Wxts+yHUi1CKRe8dcoLFY5Hl7GEnmE1O5IGwGuwgtGnqeJNU E8MZfRV0Om71NPvhpGWBSharrs6l67MdLYHq7S88= Received: (qmail 123432 invoked by alias); 17 Nov 2017 09:20:24 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 122640 invoked by uid 89); 17 Nov 2017 09:20:21 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-11.5 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_2, GIT_PATCH_3, KB_WAM_FROM_NAME_SINGLEWORD, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy= X-HELO: mail-wr0-f177.google.com Received: from mail-wr0-f177.google.com (HELO mail-wr0-f177.google.com) (209.85.128.177) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Fri, 17 Nov 2017 09:20:17 +0000 Received: by mail-wr0-f177.google.com with SMTP id y42so1525101wrd.3 for ; Fri, 17 Nov 2017 01:20:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:mail-followup-to:subject:references:date :in-reply-to:message-id:user-agent:mime-version; bh=zm6j6rgksZOrgkaLnP7anExteg+1rW0C5P0fr5AfvvU=; b=jEpM7Yp5SkTIik5ENmZ6au46pfMIGhFXEAvFktW8kaKiLJnv/ZBKt+UOUr9E18qdVc Q7m8WrgIVXZkOfklcj4vnaumb4bZ2p6Aa7MWCh8LbWlHpLIPyRoXuhrmCnCcQEdqCqIa AsMP00zSc7wokxC9/OkJXrnmK6zdQcCUt8skIwfFkCVRcLUbTt6Qvr6SjcIkyP1+xGOW 5AiLe6fZZWmzY0M5kPV95w1WMXAkmCtMBC69fS3lBjKvJmiUklSRiyDcPE5+XWsqh+kK EBnuSphyxiLErnoOAhZH7rFgMBlKF3zsQ8qxw9o/qbFPBWhXn4A/Jzd1BrxnEhGV+ck3 huVQ== X-Gm-Message-State: AJaThX7IDKexE+ZDZgu+/ngADCqTE1v2S46Y8Arv3TfiCBAiDWpGLMOj 1jU8TxlpayCFJtWdOofL5eWId+NzWXw= X-Received: by 10.223.186.71 with SMTP id t7mr3569330wrg.75.1510910414887; Fri, 17 Nov 2017 01:20:14 -0800 (PST) Received: from localhost ([2.25.234.120]) by smtp.gmail.com with ESMTPSA id m187sm2396373wmg.18.2017.11.17.01.20.13 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 17 Nov 2017 01:20:14 -0800 (PST) From: Richard Sandiford To: gcc-patches@gcc.gnu.org Mail-Followup-To: gcc-patches@gcc.gnu.org, richard.sandiford@linaro.org Subject: [6/7] Split gather load handling out of vectorizable_{mask_load_store, load} References: <87375d2rkr.fsf@linaro.org> Date: Fri, 17 Nov 2017 09:20:15 +0000 In-Reply-To: <87375d2rkr.fsf@linaro.org> (Richard Sandiford's message of "Fri, 17 Nov 2017 09:16:20 +0000") Message-ID: <87d14h1cts.fsf@linaro.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) MIME-Version: 1.0 vectorizable_mask_load_store and vectorizable_load used the same code to build a gather load call, except that the former also vectorised a mask argument and used it for both the src and mask inputs. The latter instead used a src input of zero and a mask input of all-ones. This patch splits the code out into a subroutine. Richard 2017-11-17 Richard Sandiford gcc/ * tree-vect-stmts.c (vect_build_gather_load_calls): New function, split out from... (vectorizable_mask_load_store): ...here. (vectorizable_load): ...and here. Index: gcc/tree-vect-stmts.c =================================================================== --- gcc/tree-vect-stmts.c 2017-11-17 09:06:56.444527585 +0000 +++ gcc/tree-vect-stmts.c 2017-11-17 09:07:00.001060463 +0000 @@ -2193,6 +2193,206 @@ vect_build_zero_merge_argument (gimple * return vect_init_vector (stmt, merge, vectype, NULL); } +/* Build a gather load call while vectorizing STMT. Insert new instructions + before GSI and add them to VEC_STMT. GS_INFO describes the gather load + operation. If the load is conditional, MASK is the unvectorized + condition, otherwise MASK is null. */ + +static void +vect_build_gather_load_calls (gimple *stmt, gimple_stmt_iterator *gsi, + gimple **vec_stmt, gather_scatter_info *gs_info, + tree mask) +{ + stmt_vec_info stmt_info = vinfo_for_stmt (stmt); + loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_info); + struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo); + tree vectype = STMT_VINFO_VECTYPE (stmt_info); + poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype); + int ncopies = vect_get_num_copies (loop_vinfo, vectype); + edge pe = loop_preheader_edge (loop); + enum { NARROW, NONE, WIDEN } modifier; + poly_uint64 gather_off_nunits + = TYPE_VECTOR_SUBPARTS (gs_info->offset_vectype); + + tree arglist = TYPE_ARG_TYPES (TREE_TYPE (gs_info->decl)); + tree rettype = TREE_TYPE (TREE_TYPE (gs_info->decl)); + tree srctype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); + tree ptrtype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); + tree idxtype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); + tree masktype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); + tree scaletype = TREE_VALUE (arglist); + gcc_checking_assert (types_compatible_p (srctype, rettype) + && (!mask || types_compatible_p (srctype, masktype))); + + tree perm_mask = NULL_TREE; + tree mask_perm_mask = NULL_TREE; + if (must_eq (nunits, gather_off_nunits)) + modifier = NONE; + else if (must_eq (nunits * 2, gather_off_nunits)) + { + modifier = WIDEN; + + /* Currently widening gathers and scatters are only supported for + fixed-length vectors. */ + int count = gather_off_nunits.to_constant (); + auto_vec_perm_indices sel (count); + for (int i = 0; i < count; ++i) + sel.quick_push (i | (count / 2)); + + perm_mask = vect_gen_perm_mask_checked (gs_info->offset_vectype, sel); + } + else if (must_eq (nunits, gather_off_nunits * 2)) + { + modifier = NARROW; + + /* Currently narrowing gathers and scatters are only supported for + fixed-length vectors. */ + int count = nunits.to_constant (); + auto_vec_perm_indices sel (count); + sel.quick_grow (count); + for (int i = 0; i < count; ++i) + sel[i] = i < count / 2 ? i : i + count / 2; + perm_mask = vect_gen_perm_mask_checked (vectype, sel); + + ncopies *= 2; + + if (mask) + { + for (int i = 0; i < count; ++i) + sel[i] = i | (count / 2); + mask_perm_mask = vect_gen_perm_mask_checked (masktype, sel); + } + } + else + gcc_unreachable (); + + tree vec_dest = vect_create_destination_var (gimple_get_lhs (stmt), + vectype); + + tree ptr = fold_convert (ptrtype, gs_info->base); + if (!is_gimple_min_invariant (ptr)) + { + gimple_seq seq; + ptr = force_gimple_operand (ptr, &seq, true, NULL_TREE); + basic_block new_bb = gsi_insert_seq_on_edge_immediate (pe, seq); + gcc_assert (!new_bb); + } + + tree scale = build_int_cst (scaletype, gs_info->scale); + + tree vec_oprnd0 = NULL_TREE; + tree vec_mask = NULL_TREE; + tree src_op = NULL_TREE; + tree mask_op = NULL_TREE; + tree prev_res = NULL_TREE; + stmt_vec_info prev_stmt_info = NULL; + + if (!mask) + { + src_op = vect_build_zero_merge_argument (stmt, rettype); + mask_op = vect_build_all_ones_mask (stmt, masktype); + } + + for (int j = 0; j < ncopies; ++j) + { + tree op, var; + gimple *new_stmt; + if (modifier == WIDEN && (j & 1)) + op = permute_vec_elements (vec_oprnd0, vec_oprnd0, + perm_mask, stmt, gsi); + else if (j == 0) + op = vec_oprnd0 + = vect_get_vec_def_for_operand (gs_info->offset, stmt); + else + op = vec_oprnd0 + = vect_get_vec_def_for_stmt_copy (gs_info->offset_dt, vec_oprnd0); + + if (!useless_type_conversion_p (idxtype, TREE_TYPE (op))) + { + gcc_assert (must_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (op)), + TYPE_VECTOR_SUBPARTS (idxtype))); + var = vect_get_new_ssa_name (idxtype, vect_simple_var); + op = build1 (VIEW_CONVERT_EXPR, idxtype, op); + new_stmt = gimple_build_assign (var, VIEW_CONVERT_EXPR, op); + vect_finish_stmt_generation (stmt, new_stmt, gsi); + op = var; + } + + if (mask) + { + if (mask_perm_mask && (j & 1)) + mask_op = permute_vec_elements (mask_op, mask_op, + mask_perm_mask, stmt, gsi); + else + { + if (j == 0) + vec_mask = vect_get_vec_def_for_operand (mask, stmt); + else + { + gimple *def_stmt; + enum vect_def_type dt; + vect_is_simple_use (vec_mask, loop_vinfo, &def_stmt, &dt); + vec_mask = vect_get_vec_def_for_stmt_copy (dt, vec_mask); + } + + mask_op = vec_mask; + if (!useless_type_conversion_p (masktype, TREE_TYPE (vec_mask))) + { + gcc_assert + (must_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (mask_op)), + TYPE_VECTOR_SUBPARTS (masktype))); + var = vect_get_new_ssa_name (masktype, vect_simple_var); + mask_op = build1 (VIEW_CONVERT_EXPR, masktype, mask_op); + new_stmt = gimple_build_assign (var, VIEW_CONVERT_EXPR, + mask_op); + vect_finish_stmt_generation (stmt, new_stmt, gsi); + mask_op = var; + } + } + src_op = mask_op; + } + + new_stmt = gimple_build_call (gs_info->decl, 5, src_op, ptr, op, + mask_op, scale); + + if (!useless_type_conversion_p (vectype, rettype)) + { + gcc_assert (must_eq (TYPE_VECTOR_SUBPARTS (vectype), + TYPE_VECTOR_SUBPARTS (rettype))); + op = vect_get_new_ssa_name (rettype, vect_simple_var); + gimple_call_set_lhs (new_stmt, op); + vect_finish_stmt_generation (stmt, new_stmt, gsi); + var = make_ssa_name (vec_dest); + op = build1 (VIEW_CONVERT_EXPR, vectype, op); + new_stmt = gimple_build_assign (var, VIEW_CONVERT_EXPR, op); + } + else + { + var = make_ssa_name (vec_dest, new_stmt); + gimple_call_set_lhs (new_stmt, var); + } + + vect_finish_stmt_generation (stmt, new_stmt, gsi); + + if (modifier == NARROW) + { + if ((j & 1) == 0) + { + prev_res = var; + continue; + } + var = permute_vec_elements (prev_res, var, perm_mask, stmt, gsi); + new_stmt = SSA_NAME_DEF_STMT (var); + } + + if (prev_stmt_info == NULL) + STMT_VINFO_VEC_STMT (stmt_info) = *vec_stmt = new_stmt; + else + STMT_VINFO_RELATED_STMT (prev_stmt_info) = new_stmt; + prev_stmt_info = vinfo_for_stmt (new_stmt); + } +} + /* Function vectorizable_mask_load_store. Check if STMT performs a conditional load or store that can be vectorized. @@ -2219,9 +2419,8 @@ vectorizable_mask_load_store (gimple *st tree dummy; tree dataref_ptr = NULL_TREE; gimple *ptr_incr; - poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype); int ncopies; - int i, j; + int i; bool inv_p; gather_scatter_info gs_info; vec_load_store_type vls_type; @@ -2319,170 +2518,7 @@ vectorizable_mask_load_store (gimple *st if (memory_access_type == VMAT_GATHER_SCATTER) { - tree vec_oprnd0 = NULL_TREE, op; - tree arglist = TYPE_ARG_TYPES (TREE_TYPE (gs_info.decl)); - tree rettype, srctype, ptrtype, idxtype, masktype, scaletype; - tree ptr, vec_mask = NULL_TREE, mask_op = NULL_TREE, var, scale; - tree perm_mask = NULL_TREE, prev_res = NULL_TREE; - tree mask_perm_mask = NULL_TREE; - edge pe = loop_preheader_edge (loop); - gimple_seq seq; - basic_block new_bb; - enum { NARROW, NONE, WIDEN } modifier; - poly_uint64 gather_off_nunits - = TYPE_VECTOR_SUBPARTS (gs_info.offset_vectype); - - rettype = TREE_TYPE (TREE_TYPE (gs_info.decl)); - srctype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - ptrtype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - idxtype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - masktype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - scaletype = TREE_VALUE (arglist); - gcc_checking_assert (types_compatible_p (srctype, rettype) - && types_compatible_p (srctype, masktype)); - - if (must_eq (nunits, gather_off_nunits)) - modifier = NONE; - else if (must_eq (nunits * 2, gather_off_nunits)) - { - modifier = WIDEN; - - /* Currently widening gathers and scatters are only supported for - fixed-length vectors. */ - int count = gather_off_nunits.to_constant (); - auto_vec_perm_indices sel (count); - for (i = 0; i < count; ++i) - sel.quick_push (i | (count / 2)); - - perm_mask = vect_gen_perm_mask_checked (gs_info.offset_vectype, sel); - } - else if (must_eq (nunits, gather_off_nunits * 2)) - { - modifier = NARROW; - - /* Currently narrowing gathers and scatters are only supported for - fixed-length vectors. */ - int count = nunits.to_constant (); - auto_vec_perm_indices sel (count); - sel.quick_grow (count); - for (i = 0; i < count; ++i) - sel[i] = i < count / 2 ? i : i + count / 2; - - perm_mask = vect_gen_perm_mask_checked (vectype, sel); - ncopies *= 2; - for (i = 0; i < count; ++i) - sel[i] = i | (count / 2); - mask_perm_mask = vect_gen_perm_mask_checked (masktype, sel); - } - else - gcc_unreachable (); - - vec_dest = vect_create_destination_var (gimple_call_lhs (stmt), vectype); - - ptr = fold_convert (ptrtype, gs_info.base); - if (!is_gimple_min_invariant (ptr)) - { - ptr = force_gimple_operand (ptr, &seq, true, NULL_TREE); - new_bb = gsi_insert_seq_on_edge_immediate (pe, seq); - gcc_assert (!new_bb); - } - - scale = build_int_cst (scaletype, gs_info.scale); - - prev_stmt_info = NULL; - for (j = 0; j < ncopies; ++j) - { - if (modifier == WIDEN && (j & 1)) - op = permute_vec_elements (vec_oprnd0, vec_oprnd0, - perm_mask, stmt, gsi); - else if (j == 0) - op = vec_oprnd0 - = vect_get_vec_def_for_operand (gs_info.offset, stmt); - else - op = vec_oprnd0 - = vect_get_vec_def_for_stmt_copy (gs_info.offset_dt, vec_oprnd0); - - if (!useless_type_conversion_p (idxtype, TREE_TYPE (op))) - { - gcc_assert (must_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (op)), - TYPE_VECTOR_SUBPARTS (idxtype))); - var = vect_get_new_ssa_name (idxtype, vect_simple_var); - op = build1 (VIEW_CONVERT_EXPR, idxtype, op); - new_stmt - = gimple_build_assign (var, VIEW_CONVERT_EXPR, op); - vect_finish_stmt_generation (stmt, new_stmt, gsi); - op = var; - } - - if (mask_perm_mask && (j & 1)) - mask_op = permute_vec_elements (mask_op, mask_op, - mask_perm_mask, stmt, gsi); - else - { - if (j == 0) - vec_mask = vect_get_vec_def_for_operand (mask, stmt); - else - { - vect_is_simple_use (vec_mask, loop_vinfo, &def_stmt, &dt); - vec_mask = vect_get_vec_def_for_stmt_copy (dt, vec_mask); - } - - mask_op = vec_mask; - if (!useless_type_conversion_p (masktype, TREE_TYPE (vec_mask))) - { - gcc_assert - (must_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (mask_op)), - TYPE_VECTOR_SUBPARTS (masktype))); - var = vect_get_new_ssa_name (masktype, vect_simple_var); - mask_op = build1 (VIEW_CONVERT_EXPR, masktype, mask_op); - new_stmt - = gimple_build_assign (var, VIEW_CONVERT_EXPR, mask_op); - vect_finish_stmt_generation (stmt, new_stmt, gsi); - mask_op = var; - } - } - - new_stmt - = gimple_build_call (gs_info.decl, 5, mask_op, ptr, op, mask_op, - scale); - - if (!useless_type_conversion_p (vectype, rettype)) - { - gcc_assert (must_eq (TYPE_VECTOR_SUBPARTS (vectype), - TYPE_VECTOR_SUBPARTS (rettype))); - op = vect_get_new_ssa_name (rettype, vect_simple_var); - gimple_call_set_lhs (new_stmt, op); - vect_finish_stmt_generation (stmt, new_stmt, gsi); - var = make_ssa_name (vec_dest); - op = build1 (VIEW_CONVERT_EXPR, vectype, op); - new_stmt = gimple_build_assign (var, VIEW_CONVERT_EXPR, op); - } - else - { - var = make_ssa_name (vec_dest, new_stmt); - gimple_call_set_lhs (new_stmt, var); - } - - vect_finish_stmt_generation (stmt, new_stmt, gsi); - - if (modifier == NARROW) - { - if ((j & 1) == 0) - { - prev_res = var; - continue; - } - var = permute_vec_elements (prev_res, var, - perm_mask, stmt, gsi); - new_stmt = SSA_NAME_DEF_STMT (var); - } - - if (prev_stmt_info == NULL) - STMT_VINFO_VEC_STMT (stmt_info) = *vec_stmt = new_stmt; - else - STMT_VINFO_RELATED_STMT (prev_stmt_info) = new_stmt; - prev_stmt_info = vinfo_for_stmt (new_stmt); - } + vect_build_gather_load_calls (stmt, gsi, vec_stmt, &gs_info, mask); return true; } else if (vls_type != VLS_LOAD) @@ -6980,139 +7016,7 @@ vectorizable_load (gimple *stmt, gimple_ if (memory_access_type == VMAT_GATHER_SCATTER) { - tree vec_oprnd0 = NULL_TREE, op; - tree arglist = TYPE_ARG_TYPES (TREE_TYPE (gs_info.decl)); - tree rettype, srctype, ptrtype, idxtype, masktype, scaletype; - tree ptr, mask, var, scale, merge, perm_mask = NULL_TREE, prev_res = NULL_TREE; - edge pe = loop_preheader_edge (loop); - gimple_seq seq; - basic_block new_bb; - enum { NARROW, NONE, WIDEN } modifier; - poly_uint64 gather_off_nunits - = TYPE_VECTOR_SUBPARTS (gs_info.offset_vectype); - - if (must_eq (nunits, gather_off_nunits)) - modifier = NONE; - else if (must_eq (nunits * 2, gather_off_nunits)) - { - modifier = WIDEN; - - /* Currently widening gathers are only supported for - fixed-length vectors. */ - int count = gather_off_nunits.to_constant (); - auto_vec_perm_indices sel (count); - for (i = 0; i < count; ++i) - sel.quick_push (i | (count / 2)); - - perm_mask = vect_gen_perm_mask_checked (gs_info.offset_vectype, sel); - } - else if (must_eq (nunits, gather_off_nunits * 2)) - { - modifier = NARROW; - - /* Currently narrowing gathers are only supported for - fixed-length vectors. */ - int count = nunits.to_constant (); - auto_vec_perm_indices sel (count); - for (i = 0; i < count; ++i) - sel.quick_push (i < count / 2 ? i : i + count / 2); - - perm_mask = vect_gen_perm_mask_checked (vectype, sel); - ncopies *= 2; - } - else - gcc_unreachable (); - - rettype = TREE_TYPE (TREE_TYPE (gs_info.decl)); - srctype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - ptrtype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - idxtype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - masktype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - scaletype = TREE_VALUE (arglist); - gcc_checking_assert (types_compatible_p (srctype, rettype)); - - vec_dest = vect_create_destination_var (scalar_dest, vectype); - - ptr = fold_convert (ptrtype, gs_info.base); - if (!is_gimple_min_invariant (ptr)) - { - ptr = force_gimple_operand (ptr, &seq, true, NULL_TREE); - new_bb = gsi_insert_seq_on_edge_immediate (pe, seq); - gcc_assert (!new_bb); - } - - /* Currently we support only unconditional gather loads, - so mask should be all ones. */ - mask = vect_build_all_ones_mask (stmt, masktype); - scale = build_int_cst (scaletype, gs_info.scale); - merge = vect_build_zero_merge_argument (stmt, rettype); - - prev_stmt_info = NULL; - for (j = 0; j < ncopies; ++j) - { - if (modifier == WIDEN && (j & 1)) - op = permute_vec_elements (vec_oprnd0, vec_oprnd0, - perm_mask, stmt, gsi); - else if (j == 0) - op = vec_oprnd0 - = vect_get_vec_def_for_operand (gs_info.offset, stmt); - else - op = vec_oprnd0 - = vect_get_vec_def_for_stmt_copy (gs_info.offset_dt, vec_oprnd0); - - if (!useless_type_conversion_p (idxtype, TREE_TYPE (op))) - { - gcc_assert (must_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (op)), - TYPE_VECTOR_SUBPARTS (idxtype))); - var = vect_get_new_ssa_name (idxtype, vect_simple_var); - op = build1 (VIEW_CONVERT_EXPR, idxtype, op); - new_stmt - = gimple_build_assign (var, VIEW_CONVERT_EXPR, op); - vect_finish_stmt_generation (stmt, new_stmt, gsi); - op = var; - } - - new_stmt - = gimple_build_call (gs_info.decl, 5, merge, ptr, op, mask, scale); - - if (!useless_type_conversion_p (vectype, rettype)) - { - gcc_assert (must_eq (TYPE_VECTOR_SUBPARTS (vectype), - TYPE_VECTOR_SUBPARTS (rettype))); - op = vect_get_new_ssa_name (rettype, vect_simple_var); - gimple_call_set_lhs (new_stmt, op); - vect_finish_stmt_generation (stmt, new_stmt, gsi); - var = make_ssa_name (vec_dest); - op = build1 (VIEW_CONVERT_EXPR, vectype, op); - new_stmt - = gimple_build_assign (var, VIEW_CONVERT_EXPR, op); - } - else - { - var = make_ssa_name (vec_dest, new_stmt); - gimple_call_set_lhs (new_stmt, var); - } - - vect_finish_stmt_generation (stmt, new_stmt, gsi); - - if (modifier == NARROW) - { - if ((j & 1) == 0) - { - prev_res = var; - continue; - } - var = permute_vec_elements (prev_res, var, - perm_mask, stmt, gsi); - new_stmt = SSA_NAME_DEF_STMT (var); - } - - if (prev_stmt_info == NULL) - STMT_VINFO_VEC_STMT (stmt_info) = *vec_stmt = new_stmt; - else - STMT_VINFO_RELATED_STMT (prev_stmt_info) = new_stmt; - prev_stmt_info = vinfo_for_stmt (new_stmt); - } + vect_build_gather_load_calls (stmt, gsi, vec_stmt, &gs_info, NULL_TREE); return true; }