From patchwork Mon Nov 12 21:44:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 150888 Delivered-To: patch@linaro.org Received: by 2002:a2e:299d:0:0:0:0:0 with SMTP id p29-v6csp3654420ljp; Mon, 12 Nov 2018 14:01:14 -0800 (PST) X-Google-Smtp-Source: AJdET5erfQ1k8lF0RwOaFVB5YLwWn54ldn08pmH3NiEiLoNl7Y4HB6K/Js3M1LilZllMToZqqI3X X-Received: by 2002:ac8:6606:: with SMTP id c6mr2576674qtp.376.1542060074242; Mon, 12 Nov 2018 14:01:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542060074; cv=none; d=google.com; s=arc-20160816; b=yUm+J3jjl83XX2HlAJeeY0UGn1kdDWAh6CPK8euSZegVFvYyz4cNd5916hdE53F+FY dFXj0QPxemn9A+zIi4JH2WAQ/RBsfOVnT+N80mDJUSG/AfFKxBznKWcqoPBJqQWh2P+w A5A7Cd6tuhZ0xjODimZqaMyQcd6f99z77dnWsJJ1+24i5uxDXDkhtZV6l3TuQJiDDO4F +xM+JfVpGOXa6I5pRKohQN3K6W5xRbmI5NuJTbSTFMKwVigvhmfvq1Y7SPglohuYuJPr nqdQz/oftn3nYvVKd0l2JmR3LutFPG7iL3gcjJGHl/w0ya6pu4I5s4bbAq3Eqh4sy/wm cHVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature; bh=8y1UmhfH9rH2EQVi6l23g/hflBkPX47jvRyrizN6T+o=; b=f8yFx6ddfvKiPyNx9U+vkLr0OyH+VIEDDJzFTEBun5P8Ep0p5kwo0YJc6Mhtda1BI3 8IE2LmTMvQzyxWExb5cFNNcA8PoOVXZFd5amrRMRYfRfsL2agQzQ6NGy2JLVfbCvIFhQ rjVED4/2eat4HrW1hxMP1PFA6Q4XwFTVZEdad1/29d9bgqHBqTWwTuhR9u5CUusz7Trg JkW04JG39LkEVWp+m6vSeMaWqPoWuqVTFEtOJxGOWxaa6K+idlchNNft7YOtJoutbGPH hydhUcaFiUzjPRU176wXXdS/iTO7XbaGgcTzz/iFTvanuqJJclv7cHtEGXqE5hERqkAn LtZA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=O5UFjoZy; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id t82si14189782qkl.141.2018.11.12.14.01.13 for (version=TLS1 cipher=AES128-SHA bits=128/128); Mon, 12 Nov 2018 14:01:14 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=O5UFjoZy; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:50881 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMKGf-00084u-6j for patch@linaro.org; Mon, 12 Nov 2018 17:01:13 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54110) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMK39-0003X2-I9 for qemu-devel@nongnu.org; Mon, 12 Nov 2018 16:47:16 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMK38-0004xl-G5 for qemu-devel@nongnu.org; Mon, 12 Nov 2018 16:47:15 -0500 Received: from mail-wm1-x341.google.com ([2a00:1450:4864:20::341]:51118) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gMK38-0004xQ-9h for qemu-devel@nongnu.org; Mon, 12 Nov 2018 16:47:14 -0500 Received: by mail-wm1-x341.google.com with SMTP id 124-v6so10004274wmw.0 for ; Mon, 12 Nov 2018 13:47:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=8y1UmhfH9rH2EQVi6l23g/hflBkPX47jvRyrizN6T+o=; b=O5UFjoZyoUOIAjKQXiijFiqY67oN8Tp260SE4x3cqORaoxFTBFhN/1T7WMGI3Ljr+t lBalSG5tu9uvlzEIX61i1yDVHbBhIarIB9R9ZRYLBtONX2bsPMX9mrL8/xf1CgOfnWDQ 5JxTnzQObmk6phmk2rGiErZ9ReBr0gTv0BikI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=8y1UmhfH9rH2EQVi6l23g/hflBkPX47jvRyrizN6T+o=; b=lvZDau8kzGKaPOBcRrWxaCDkHMSWVSZE71b7Jm/zH8cIMsvDlDaRtYOsNU+Z+Pgnhu oefcHe70EVmfuEag+1BPanrcDKh84D2BcjbHRieG8TZv6wCSPIuIt4Qgobl3+rCrmfoe 5TG8Iz6CZ07frP1fL8VY6rl8mQnoZe/d7XwkcMDeU1O4YSa0PDU6Y6wdHiZb0v5JdHYn G1PHFujXCQS9CV556110l5CS3ydl2bcYEW8k+vAXnZDKNdmWgID4r5rdxlIYMrp8xzbt VU0Nmfd5rR07upBbVwlpAxdR5V14DOQ7/NjUu3HNCocqjyWsdYtzrW2XuXkm+fNtHLFi fBIQ== X-Gm-Message-State: AGRZ1gJkWl9aWKu0KjzGZcG6HPGVOHhDHtML9axIYf/cq/4rKurQGQ/8 TOuGpiGNtfdAH3useqRi6Uddzastr1BxxA== X-Received: by 2002:a1c:154a:: with SMTP id 71-v6mr1027322wmv.83.1542059232984; Mon, 12 Nov 2018 13:47:12 -0800 (PST) Received: from cloudburst.twiddle.net (178.red-213-99-154.dynamicip.rima-tde.net. [213.99.154.178]) by smtp.gmail.com with ESMTPSA id y123-v6sm3946907wme.38.2018.11.12.13.47.11 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 12 Nov 2018 13:47:12 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Mon, 12 Nov 2018 22:44:52 +0100 Message-Id: <20181112214503.22941-7-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181112214503.22941-1-richard.henderson@linaro.org> References: <20181112214503.22941-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::341 Subject: [Qemu-devel] [PATCH for-4.0 06/17] tcg: Add TCG_TARGET_NEED_LDST_OOL_LABELS X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cota@braap.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" This variant of tcg-ldst.inc.c allows the entire thunk to be moved out-of-line, with caching across TBs within a region. Signed-off-by: Richard Henderson --- tcg/tcg.h | 4 ++ tcg/tcg-ldst-ool.inc.c | 94 ++++++++++++++++++++++++++++++++++++++++++ tcg/tcg.c | 20 +++++++++ 3 files changed, 118 insertions(+) create mode 100644 tcg/tcg-ldst-ool.inc.c -- 2.17.2 diff --git a/tcg/tcg.h b/tcg/tcg.h index f4efbaa680..1255d2a2c6 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -706,6 +706,10 @@ struct TCGContext { #ifdef TCG_TARGET_NEED_LDST_LABELS QSIMPLEQ_HEAD(ldst_labels, TCGLabelQemuLdst) ldst_labels; #endif +#ifdef TCG_TARGET_NEED_LDST_OOL_LABELS + QSIMPLEQ_HEAD(ldst_labels, TCGLabelQemuLdstOol) ldst_ool_labels; + GHashTable *ldst_ool_thunks; +#endif #ifdef TCG_TARGET_NEED_POOL_LABELS struct TCGLabelPoolData *pool_labels; #endif diff --git a/tcg/tcg-ldst-ool.inc.c b/tcg/tcg-ldst-ool.inc.c new file mode 100644 index 0000000000..8fb6550a8d --- /dev/null +++ b/tcg/tcg-ldst-ool.inc.c @@ -0,0 +1,94 @@ +/* + * TCG Backend Data: load-store optimization only. + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + * THE SOFTWARE. + */ + +typedef struct TCGLabelQemuLdstOol { + QSIMPLEQ_ENTRY(TCGLabelQemuLdstOol) next; + tcg_insn_unit *label; /* label pointer to be updated */ + int reloc; /* relocation type from label_ptr */ + intptr_t addend; /* relocation addend from label_ptr */ + uint32_t key; /* oi : is_64 : is_ld */ +} TCGLabelQemuLdstOol; + + +/* + * Generate TB finalization at the end of block + */ + +static tcg_insn_unit *tcg_out_qemu_ldst_ool(TCGContext *s, bool is_ld, + bool is64, TCGMemOpIdx oi); + +static bool tcg_out_ldst_ool_finalize(TCGContext *s) +{ + TCGLabelQemuLdstOol *lb; + + /* qemu_ld/st slow paths */ + QSIMPLEQ_FOREACH(lb, &s->ldst_ool_labels, next) { + gpointer dest, key = (gpointer)(uintptr_t)lb->key; + TCGMemOpIdx oi; + bool is_ld, is_64, ok; + + /* If we have generated the thunk, and it's still in range, all ok. */ + dest = g_hash_table_lookup(s->ldst_ool_thunks, key); + if (dest && + patch_reloc(lb->label, lb->reloc, (intptr_t)dest, lb->addend)) { + continue; + } + + /* Generate a new thunk. */ + is_ld = extract32(lb->key, 0, 1); + is_64 = extract32(lb->key, 1, 1); + oi = extract32(lb->key, 2, 30); + dest = tcg_out_qemu_ldst_ool(s, is_ld, is_64, oi); + + /* Test for (pending) buffer overflow. The assumption is that any + one thunk beginning below the high water mark cannot overrun + the buffer completely. Thus we can test for overflow after + generating code without having to check during generation. */ + if (unlikely((void *)s->code_ptr > s->code_gen_highwater)) { + return false; + } + + /* Remember the thunk for next time. */ + g_hash_table_replace(s->ldst_ool_thunks, key, dest); + + /* The new thunk must be in range. */ + ok = patch_reloc(lb->label, lb->reloc, (intptr_t)dest, lb->addend); + tcg_debug_assert(ok); + } + return true; +} + +/* + * Allocate a new TCGLabelQemuLdstOol entry. + */ + +static void add_ldst_ool_label(TCGContext *s, bool is_ld, bool is_64, + TCGMemOpIdx oi, int reloc, intptr_t addend) +{ + TCGLabelQemuLdstOol *lb = tcg_malloc(sizeof(*lb)); + + QSIMPLEQ_INSERT_TAIL(&s->ldst_ool_labels, lb, next); + lb->label = s->code_ptr; + lb->reloc = reloc; + lb->addend = addend; + lb->key = is_ld | (is_64 << 1) | (oi << 2); +} diff --git a/tcg/tcg.c b/tcg/tcg.c index 54f1272187..885d842a12 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -521,6 +521,13 @@ static void tcg_region_assign(TCGContext *s, size_t curr_region) s->code_gen_ptr = start; s->code_gen_buffer_size = end - start; s->code_gen_highwater = end - TCG_HIGHWATER; + +#ifdef TCG_TARGET_NEED_LDST_OOL_LABELS + /* No thunks yet generated this region. Even if they were in range, + this is also the most convenient place to clear the table after a + full tb_flush. */ + g_hash_table_remove_all(s->ldst_ool_thunks); +#endif } static bool tcg_region_alloc__locked(TCGContext *s) @@ -964,6 +971,11 @@ void tcg_context_init(TCGContext *s) tcg_debug_assert(!tcg_regset_test_reg(s->reserved_regs, TCG_AREG0)); ts = tcg_global_reg_new_internal(s, TCG_TYPE_PTR, TCG_AREG0, "env"); cpu_env = temp_tcgv_ptr(ts); + +#ifdef TCG_TARGET_NEED_LDST_OOL_LABELS + /* Both key and value are raw pointers. */ + s->ldst_ool_thunks = g_hash_table_new(NULL, NULL); +#endif } /* @@ -3540,6 +3552,9 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb) #ifdef TCG_TARGET_NEED_LDST_LABELS QSIMPLEQ_INIT(&s->ldst_labels); #endif +#ifdef TCG_TARGET_NEED_LDST_OOL_LABELS + QSIMPLEQ_INIT(&s->ldst_ool_labels); +#endif #ifdef TCG_TARGET_NEED_POOL_LABELS s->pool_labels = NULL; #endif @@ -3620,6 +3635,11 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb) return -1; } #endif +#ifdef TCG_TARGET_NEED_LDST_OOL_LABELS + if (!tcg_out_ldst_ool_finalize(s)) { + return -1; + } +#endif #ifdef TCG_TARGET_NEED_POOL_LABELS if (!tcg_out_pool_finalize(s)) { return -1;