From patchwork Tue Dec 3 22:53:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 180772 Delivered-To: patch@linaro.org Received: by 2002:a92:3001:0:0:0:0:0 with SMTP id x1csp2184102ile; Tue, 3 Dec 2019 15:16:41 -0800 (PST) X-Google-Smtp-Source: APXvYqwPeX/LdNekM5SAy/7RPzA3SmlSjOzhgmyGEgu57gCG3Mza1VXXSRpshHtkJa1MP/ivL2dY X-Received: by 2002:a05:6214:1711:: with SMTP id db17mr117549qvb.199.1575415001095; Tue, 03 Dec 2019 15:16:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1575415001; cv=none; d=google.com; s=arc-20160816; b=OpQorCdaosUIV788w6/iBCkjgl5ABuP9t6FxE805T7n2YB7jg4DP4KdPD5LMZeJaYy WbYtfWCN5EPs86sXOqFvX6Q31he4cCl7qbWYK0SMIA+MuCWn2GlZXtzH/w8SkYzRZkVk nlHtlnNyoGdpsOEszdp5HZP91FbtDXnw4t474vFkXrfqGxfWPVtCj03T8isC4jnu0S5c YqVlUDm5IEpQnZkfLyAqjKqIK+pjcemWX4uTJ4UqfAIyRghppf3dGJjLm6xpSTueSN+0 DADTFFjTHtTstsV8dD0yUTNl5kqf0EGP7dRQJKx5NBUwqWEen5OdENy54qja+YlZFVg+ fUgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:references:in-reply-to :message-id:date:subject:to:from:dkim-signature; bh=GpVUc819EtKJmXmcFCtrJ6j6UAZ0zHMd3RprAICGD1Q=; b=Er6BCI/vdyFYRPVTzANjdz16HQLzmFrkiYeI4+UzeD0Gwg8Xx2NG6lkeCH0uut1rG1 fO6bJVyUNhwJ9LuJf6rB65T3fOAB5ME7+ygdS17I9CpbgOAbnysng4H6l18MxaEDkjmt I0o7rmBzyg19YBVs/UhlFZl0LHBaRZYf+QY7bpU1rd5q+yZ4v78PLzZGEDNjnyKma0p9 efGosHvWDYga81klqE8hJKZOSQ9RIumayvoCUUMSejMUxwYwSYs0gpCMui9xZbNsXD1u NLWeWtiF6xQSsrbPoca9oWoZKEVcYSEY8gic5qXLNQ3MI2OX0rcjdrAWDtqzhfhLezOL msdw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=BXOWQtpM; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id u2si390287qku.279.2019.12.03.15.16.41 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 03 Dec 2019 15:16:41 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=BXOWQtpM; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:60280 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1icHPM-0007Uy-Di for patch@linaro.org; Tue, 03 Dec 2019 18:16:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:58045) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1icH37-0002XK-1E for qemu-devel@nongnu.org; Tue, 03 Dec 2019 17:53:42 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1icH35-0006Fg-5P for qemu-devel@nongnu.org; Tue, 03 Dec 2019 17:53:40 -0500 Received: from mail-pf1-x441.google.com ([2607:f8b0:4864:20::441]:41813) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1icH34-0006Cg-Qg for qemu-devel@nongnu.org; Tue, 03 Dec 2019 17:53:39 -0500 Received: by mail-pf1-x441.google.com with SMTP id s18so2566108pfd.8 for ; Tue, 03 Dec 2019 14:53:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=GpVUc819EtKJmXmcFCtrJ6j6UAZ0zHMd3RprAICGD1Q=; b=BXOWQtpMs2xHzrkqMezlvx/Rx3H/5spS7nryedTFylz+s3IAZXQZrzLW5SPwAyoGbr rCmrLe8aCGEw8YbxfhE245cEuuqL8VG0oSUXW8ZJ10ntF6I4VaBOzWxi51q4nRpzGJa2 HIO4xew7OX+onjwGnQkhlD9N8ExoaLR/bJ6O6z0pixDwzLMP3i79vH+DyrlhiOfT1cIn MvBMP0GZkiem2Dsg1GiGRHb/8yWsFxtl2Ey2Ldl3pETgfZuvBBGYxmIBt+e717Akz680 F6OhH2+T2KNSvS6kpL5YXieVggjlKNutqvxZcqgO1woTZWAQgAFdF3z0N4Ly24dvRiWe cE8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=GpVUc819EtKJmXmcFCtrJ6j6UAZ0zHMd3RprAICGD1Q=; b=eKbqVsPnkdukHhkJRSanY9mCpqJf80dZJtniZQLEHrqUo/sOGcp+VsqCsiK6KRle9I eJsbHtQOaTRKZr+Xr4O0RKvgfjvHbZyu17NYG+8Bo42ddz2dFlsHZcuppka/cRgYTsJt NZvoFgEQ7+NApsgylkGohDl3ufk4UqfhxWwafZj99+KERmH+WNg5Sc7tOnhSNM4a9nCn Oe6yXo24Ald3YwTGn9XJ4cTaBooFZV1xtq/5+mKy/pGgHiHn0pWmALvi/V57mxGxkM/T wwXObk302rPkeX+O81wqS9VQ2F04cYK38KvceN2haLXysYh4dL3FpPwswLBmoo64KO8F V4Mw== X-Gm-Message-State: APjAAAWwwEIVKtdSde/bDcdeUxf1MU0NyBi3ZOzpIhgIG9M8/MXJgJ8U 9X8gNWDfPBf9VJQrP1ltAF03ubvWXQI= X-Received: by 2002:a62:e914:: with SMTP id j20mr266652pfh.245.1575413616536; Tue, 03 Dec 2019 14:53:36 -0800 (PST) Received: from localhost.localdomain (97-113-7-119.tukw.qwest.net. [97.113.7.119]) by smtp.gmail.com with ESMTPSA id d22sm3789713pjd.2.2019.12.03.14.53.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2019 14:53:35 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 01/11] cputlb: Handle NB_MMU_MODES > TARGET_PAGE_BITS_MIN Date: Tue, 3 Dec 2019 14:53:23 -0800 Message-Id: <20191203225333.17055-2-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20191203225333.17055-1-richard.henderson@linaro.org> References: <20191203225333.17055-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::441 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" In target/arm we will shortly have "too many" mmu_idx. The current minimum barrier is caused by the way in which tlb_flush_page_by_mmuidx is coded. We can remove this limitation by allocating memory for consumption by the worker. Let us assume that this is the unlikely case, as will be the case for the majority of targets which have so far satisfied the BUILD_BUG_ON, and only allocate memory when necessary. Signed-off-by: Richard Henderson --- accel/tcg/cputlb.c | 167 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 132 insertions(+), 35 deletions(-) -- 2.17.1 Reviewed-by: Peter Maydell diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index 98221948d6..0c2adb93ea 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -447,28 +447,29 @@ static void tlb_flush_page_locked(CPUArchState *env, int midx, } } -/* As we are going to hijack the bottom bits of the page address for a - * mmuidx bit mask we need to fail to build if we can't do that +/** + * tlb_flush_page_by_mmuidx_async_0: + * @cpu: cpu on which to flush + * @addr: page of virtual address to flush + * @idxmap: set of mmu_idx to flush + * + * Helper for tlb_flush_page_by_mmuidx and friends, flush one page + * at @addr from the tlbs indicated by @idxmap from @cpu. */ -QEMU_BUILD_BUG_ON(NB_MMU_MODES > TARGET_PAGE_BITS_MIN); - -static void tlb_flush_page_by_mmuidx_async_work(CPUState *cpu, - run_on_cpu_data data) +static void tlb_flush_page_by_mmuidx_async_0(CPUState *cpu, + target_ulong addr, + uint16_t idxmap) { CPUArchState *env = cpu->env_ptr; - target_ulong addr_and_mmuidx = (target_ulong) data.target_ptr; - target_ulong addr = addr_and_mmuidx & TARGET_PAGE_MASK; - unsigned long mmu_idx_bitmap = addr_and_mmuidx & ALL_MMUIDX_BITS; int mmu_idx; assert_cpu_is_self(cpu); - tlb_debug("page addr:" TARGET_FMT_lx " mmu_map:0x%lx\n", - addr, mmu_idx_bitmap); + tlb_debug("page addr:" TARGET_FMT_lx " mmu_map:0x%x\n", addr, idxmap); qemu_spin_lock(&env_tlb(env)->c.lock); for (mmu_idx = 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { - if (test_bit(mmu_idx, &mmu_idx_bitmap)) { + if ((idxmap >> mmu_idx) & 1) { tlb_flush_page_locked(env, mmu_idx, addr); } } @@ -477,22 +478,75 @@ static void tlb_flush_page_by_mmuidx_async_work(CPUState *cpu, tb_flush_jmp_cache(cpu, addr); } +/** + * tlb_flush_page_by_mmuidx_async_1: + * @cpu: cpu on which to flush + * @data: encoded addr + idxmap + * + * Helper for tlb_flush_page_by_mmuidx and friends, called through + * async_run_on_cpu. The idxmap parameter is encoded in the page + * offset of the target_ptr field. This limits the set of mmu_idx + * that can be passed via this method. + */ +static void tlb_flush_page_by_mmuidx_async_1(CPUState *cpu, + run_on_cpu_data data) +{ + target_ulong addr_and_idxmap = (target_ulong) data.target_ptr; + target_ulong addr = addr_and_idxmap & TARGET_PAGE_MASK; + uint16_t idxmap = addr_and_idxmap & ~TARGET_PAGE_MASK; + + tlb_flush_page_by_mmuidx_async_0(cpu, addr, idxmap); +} + +typedef struct { + target_ulong addr; + uint16_t idxmap; +} TLBFlushPageByMMUIdxData; + +/** + * tlb_flush_page_by_mmuidx_async_2: + * @cpu: cpu on which to flush + * @data: allocated addr + idxmap + * + * Helper for tlb_flush_page_by_mmuidx and friends, called through + * async_run_on_cpu. The addr+idxmap parameters are stored in a + * TLBFlushPageByMMUIdxData structure that has been allocated + * specifically for this helper. Free the structure when done. + */ +static void tlb_flush_page_by_mmuidx_async_2(CPUState *cpu, + run_on_cpu_data data) +{ + TLBFlushPageByMMUIdxData *d = data.host_ptr; + + tlb_flush_page_by_mmuidx_async_0(cpu, d->addr, d->idxmap); + g_free(d); +} + void tlb_flush_page_by_mmuidx(CPUState *cpu, target_ulong addr, uint16_t idxmap) { - target_ulong addr_and_mmu_idx; - tlb_debug("addr: "TARGET_FMT_lx" mmu_idx:%" PRIx16 "\n", addr, idxmap); /* This should already be page aligned */ - addr_and_mmu_idx = addr & TARGET_PAGE_MASK; - addr_and_mmu_idx |= idxmap; + addr &= TARGET_PAGE_MASK; - if (!qemu_cpu_is_self(cpu)) { - async_run_on_cpu(cpu, tlb_flush_page_by_mmuidx_async_work, - RUN_ON_CPU_TARGET_PTR(addr_and_mmu_idx)); + if (qemu_cpu_is_self(cpu)) { + tlb_flush_page_by_mmuidx_async_0(cpu, addr, idxmap); + } else if (idxmap < TARGET_PAGE_SIZE) { + /* + * Most targets have only a few mmu_idx. In the case where + * we can stuff idxmap into the low TARGET_PAGE_BITS, avoid + * allocating memory for this operation. + */ + async_run_on_cpu(cpu, tlb_flush_page_by_mmuidx_async_1, + RUN_ON_CPU_TARGET_PTR(addr | idxmap)); } else { - tlb_flush_page_by_mmuidx_async_work( - cpu, RUN_ON_CPU_TARGET_PTR(addr_and_mmu_idx)); + TLBFlushPageByMMUIdxData *d = g_new(TLBFlushPageByMMUIdxData, 1); + + /* Otherwise allocate a structure, freed by the worker. */ + d->addr = addr; + d->idxmap = idxmap; + async_run_on_cpu(cpu, tlb_flush_page_by_mmuidx_async_2, + RUN_ON_CPU_HOST_PTR(d)); } } @@ -504,17 +558,36 @@ void tlb_flush_page(CPUState *cpu, target_ulong addr) void tlb_flush_page_by_mmuidx_all_cpus(CPUState *src_cpu, target_ulong addr, uint16_t idxmap) { - const run_on_cpu_func fn = tlb_flush_page_by_mmuidx_async_work; - target_ulong addr_and_mmu_idx; - tlb_debug("addr: "TARGET_FMT_lx" mmu_idx:%"PRIx16"\n", addr, idxmap); /* This should already be page aligned */ - addr_and_mmu_idx = addr & TARGET_PAGE_MASK; - addr_and_mmu_idx |= idxmap; + addr &= TARGET_PAGE_MASK; - flush_all_helper(src_cpu, fn, RUN_ON_CPU_TARGET_PTR(addr_and_mmu_idx)); - fn(src_cpu, RUN_ON_CPU_TARGET_PTR(addr_and_mmu_idx)); + /* + * Allocate memory to hold addr+idxmap only when needed. + * See tlb_flush_page_by_mmuidx for details. + */ + if (idxmap < TARGET_PAGE_SIZE) { + flush_all_helper(src_cpu, tlb_flush_page_by_mmuidx_async_1, + RUN_ON_CPU_TARGET_PTR(addr | idxmap)); + } else { + CPUState *dst_cpu; + + /* Allocate a separate data block for each destination cpu. */ + CPU_FOREACH(dst_cpu) { + if (dst_cpu != src_cpu) { + TLBFlushPageByMMUIdxData *d + = g_new(TLBFlushPageByMMUIdxData, 1); + + d->addr = addr; + d->idxmap = idxmap; + async_run_on_cpu(dst_cpu, tlb_flush_page_by_mmuidx_async_2, + RUN_ON_CPU_HOST_PTR(d)); + } + } + } + + tlb_flush_page_by_mmuidx_async_0(src_cpu, addr, idxmap); } void tlb_flush_page_all_cpus(CPUState *src, target_ulong addr) @@ -526,17 +599,41 @@ void tlb_flush_page_by_mmuidx_all_cpus_synced(CPUState *src_cpu, target_ulong addr, uint16_t idxmap) { - const run_on_cpu_func fn = tlb_flush_page_by_mmuidx_async_work; - target_ulong addr_and_mmu_idx; - tlb_debug("addr: "TARGET_FMT_lx" mmu_idx:%"PRIx16"\n", addr, idxmap); /* This should already be page aligned */ - addr_and_mmu_idx = addr & TARGET_PAGE_MASK; - addr_and_mmu_idx |= idxmap; + addr &= TARGET_PAGE_MASK; - flush_all_helper(src_cpu, fn, RUN_ON_CPU_TARGET_PTR(addr_and_mmu_idx)); - async_safe_run_on_cpu(src_cpu, fn, RUN_ON_CPU_TARGET_PTR(addr_and_mmu_idx)); + /* + * Allocate memory to hold addr+idxmap only when needed. + * See tlb_flush_page_by_mmuidx for details. + */ + if (idxmap < TARGET_PAGE_SIZE) { + flush_all_helper(src_cpu, tlb_flush_page_by_mmuidx_async_1, + RUN_ON_CPU_TARGET_PTR(addr | idxmap)); + async_safe_run_on_cpu(src_cpu, tlb_flush_page_by_mmuidx_async_1, + RUN_ON_CPU_TARGET_PTR(addr | idxmap)); + } else { + CPUState *dst_cpu; + TLBFlushPageByMMUIdxData *d; + + /* Allocate a separate data block for each destination cpu. */ + CPU_FOREACH(dst_cpu) { + if (dst_cpu != src_cpu) { + d = g_new(TLBFlushPageByMMUIdxData, 1); + d->addr = addr; + d->idxmap = idxmap; + async_run_on_cpu(dst_cpu, tlb_flush_page_by_mmuidx_async_2, + RUN_ON_CPU_HOST_PTR(d)); + } + } + + d = g_new(TLBFlushPageByMMUIdxData, 1); + d->addr = addr; + d->idxmap = idxmap; + async_safe_run_on_cpu(src_cpu, tlb_flush_page_by_mmuidx_async_2, + RUN_ON_CPU_HOST_PTR(d)); + } } void tlb_flush_page_all_cpus_synced(CPUState *src, target_ulong addr)