From patchwork Mon Oct 24 05:18:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 617843 Delivered-To: patch@linaro.org Received: by 2002:a17:522:c983:b0:460:3032:e3c4 with SMTP id kr3csp2511073pvb; Sun, 23 Oct 2022 23:04:49 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6cufhUXu/wnUHwD1aPJnUbp0OWhKA48aysQ4LbOgVdmTozbcOFGor+z2h6993830m8qlHi X-Received: by 2002:a05:6214:250c:b0:4b1:a1e7:1d89 with SMTP id gf12-20020a056214250c00b004b1a1e71d89mr26419477qvb.1.1666591489576; Sun, 23 Oct 2022 23:04:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666591489; cv=none; d=google.com; s=arc-20160816; b=eJ/+7I9cfVEjEYdhxGAYcbgf0cu1nqXXDKv2Djg5Ye7yZNsM0x+Z2y9jAcaKclAYK5 SB9C1Of56QSO0fzfDlKJjwUsX/nsEDKKpNSP6dSVPC4yRTofmJWIrQrZYJz0paRKqsVc 1IJD6OxQ1miVrnZW9UyWAJW/dz3cHWaN5ff1pdCLBWol1iiWZNPC8wVZxLiHEw9GPUbT pfHSDIH3iDQWI/Q4bh1Eqo0rvgiLoieC1GDvCPrbrTIM0mhhbWoyFGJymh72WxD6IABg tVs/fcJc3Ylb6I64OK6ccL+SIOSHMNvM1XJvWViBlt8MRYSr5XlpM2+TayjyVxs2X7dN DM7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=vqqhe7ok+952FMeoYC9dG+eusN1LKoZKt0UmNCjl0DQ=; b=JY4M3tzloNVDgLXPxexY9gkUzRxIvy6Tdxn+/zCh5jo8jt3OsYqMV7fsmUAJL5lHLF HjL/eNPvUCMz2kmo7tSUB0zX5Ew2iiIGwYR6Qzr0DBVNIA9TaVAQzS8I2Xu4k5Tg6gNn WsF75cd7gdS71CSB5urkKyH7IF8v8qh7Bz9ayfqB4ediJU6yAMMxJz6i2YFccyjNi4Zu pcahIRmfA4tKxVTLcr6Hu0pS/gvN6ROIoN2SHboX1uDOQ2LPmoVwSbYbhE2rlwSf3HgB tqgNWMfiIn8HAsYZKE4nDkuHO6ZMGY4E9H6XZpRola6X+Yixe+hxxIK+vUe5U6mhKDC3 dC7w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=lA3VuMzH; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id s20-20020ac85ed4000000b0035bb6064313si15445862qtx.170.2022.10.23.23.04.49 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Sun, 23 Oct 2022 23:04:49 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=lA3VuMzH; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ompt5-0007lq-FN; Mon, 24 Oct 2022 01:20:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ompsC-0006sk-Lp for qemu-devel@nongnu.org; Mon, 24 Oct 2022 01:19:43 -0400 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1omps8-0002RD-1y for qemu-devel@nongnu.org; Mon, 24 Oct 2022 01:19:39 -0400 Received: by mail-pl1-x62e.google.com with SMTP id g24so2582660plq.3 for ; Sun, 23 Oct 2022 22:19:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vqqhe7ok+952FMeoYC9dG+eusN1LKoZKt0UmNCjl0DQ=; b=lA3VuMzHfeZVjcDGVcj/tcuhSkjE/pMPlcJX0gPXzzGjiePf8V76ex2LDe+dQxo7Z4 RvTH6GgDWweeW0z7rBBIGbY0GjSLuQYvVn69/QlQW04udvF+GFjUWbmB8En2zuwMiKLO teOoELMUkTnRRr0dOsC3HhYMjTsvcwJ3/yTDd9lI7gjKIJ9OW9wobXWPET9oX+2mmf2g RXpEKNYh4jNIIWRrkp8bIaWm+pXdJzpLgpCQx0zkQc1acA2Vh74j/pbMT9xkEJq3hIPc YPM4+m1b7YQ3aI/ednmmkl1h1t53pKW6yxd+JydBUXIKsTggZeDeMrCLq/6o4OoxYPcs y49w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vqqhe7ok+952FMeoYC9dG+eusN1LKoZKt0UmNCjl0DQ=; b=LDGqji9yW00xBrTgpaXCpki7SdSXStHzxItcyZah6h67VwhWGuUK09WRqQFTiZVNXE oTYdhQWH5CfejW1khNbV7osNhvBbEfiqgWj0iX3NTPzySqLu4a/HWonS5xWAMNQQBDKK PC75CnUBtK8JxwHYW8SwwmoZKAUeAtIFxg6dX2wPGJ8gObrYatgeCFuYxbiR11KTfrsj rM75ZU5GwmQ8oUEAMf1XAyVo4yRUWl5xHMSkQ7G2GpaL0OTJfrWPZTjJjVqM95+ackE4 U4PfQGp4+SJXsNWOQjFjwZ8wFIm0uX/NNq9fCLY2CC3Dlj+5kN5HIp1NyILMmEo5l32+ zxdg== X-Gm-Message-State: ACrzQf0ZB5nWRqJkTFVerXCMaRfx07PHzQQ8QIMnN2xBiB5WgHqnUonY Ka+i2X+FK1dLeMKQXmMSpHZjR+Q07z3ee9Zu X-Received: by 2002:a17:90a:fa42:b0:212:f0d3:2ff4 with SMTP id dt2-20020a17090afa4200b00212f0d32ff4mr9406255pjb.104.1666588771768; Sun, 23 Oct 2022 22:19:31 -0700 (PDT) Received: from stoup.. ([103.100.225.182]) by smtp.gmail.com with ESMTPSA id q12-20020a170902f34c00b00172951ddb12sm1895231ple.42.2022.10.23.22.19.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Oct 2022 22:19:31 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, Peter Maydell Subject: [PATCH v6 12/14] target/arm: Implement FEAT_HAFDBS, access flag portion Date: Mon, 24 Oct 2022 15:18:49 +1000 Message-Id: <20221024051851.3074715-13-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221024051851.3074715-1-richard.henderson@linaro.org> References: <20221024051851.3074715-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62e; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x62e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Qemu-devel" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Perform the atomic update for hardware management of the access flag. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- v4: Raise permission fault if pte read-only and atomic update reqd. Split out dirty bit portion. Prepare for a single update for AF + DB. v5: Fix s1ns typo; incorporate i_yzsvv comment. Move the AF update before attributes are extracted and merged. Do permission check before pte writeback, per AArch64.S1Translate. --- docs/system/arm/emulation.rst | 1 + target/arm/cpu64.c | 1 + target/arm/ptw.c | 176 +++++++++++++++++++++++++++++----- 3 files changed, 156 insertions(+), 22 deletions(-) diff --git a/docs/system/arm/emulation.rst b/docs/system/arm/emulation.rst index cfb4b0768b..580e67b190 100644 --- a/docs/system/arm/emulation.rst +++ b/docs/system/arm/emulation.rst @@ -32,6 +32,7 @@ the following architecture extensions: - FEAT_FlagM (Flag manipulation instructions v2) - FEAT_FlagM2 (Enhancements to flag manipulation instructions) - FEAT_GTG (Guest translation granule size) +- FEAT_HAFDBS (Hardware management of the access flag and dirty bit state) - FEAT_HCX (Support for the HCRX_EL2 register) - FEAT_HPDS (Hierarchical permission disables) - FEAT_I8MM (AArch64 Int8 matrix multiplication instructions) diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index 85e0d1daf1..fe1369fe96 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -1165,6 +1165,7 @@ static void aarch64_max_initfn(Object *obj) cpu->isar.id_aa64mmfr0 = t; t = cpu->isar.id_aa64mmfr1; + t = FIELD_DP64(t, ID_AA64MMFR1, HAFDBS, 1); /* FEAT_HAFDBS, AF only */ t = FIELD_DP64(t, ID_AA64MMFR1, VMIDBITS, 2); /* FEAT_VMID16 */ t = FIELD_DP64(t, ID_AA64MMFR1, VH, 1); /* FEAT_VHE */ t = FIELD_DP64(t, ID_AA64MMFR1, HPDS, 1); /* FEAT_HPDS */ diff --git a/target/arm/ptw.c b/target/arm/ptw.c index 282828992e..d9f89b6105 100644 --- a/target/arm/ptw.c +++ b/target/arm/ptw.c @@ -21,7 +21,9 @@ typedef struct S1Translate { bool in_secure; bool in_debug; bool out_secure; + bool out_rw; bool out_be; + hwaddr out_virt; hwaddr out_phys; void *out_host; } S1Translate; @@ -238,6 +240,8 @@ static bool S1_ptw_translate(CPUARMState *env, S1Translate *ptw, uint8_t pte_attrs; bool pte_secure; + ptw->out_virt = addr; + if (unlikely(ptw->in_debug)) { /* * From gdbstub, do not use softmmu so that we don't modify the @@ -266,6 +270,7 @@ static bool S1_ptw_translate(CPUARMState *env, S1Translate *ptw, pte_secure = is_secure; } ptw->out_host = NULL; + ptw->out_rw = false; } else { CPUTLBEntryFull *full; int flags; @@ -280,6 +285,7 @@ static bool S1_ptw_translate(CPUARMState *env, S1Translate *ptw, goto fail; } ptw->out_phys = full->phys_addr; + ptw->out_rw = full->prot & PROT_WRITE; pte_attrs = full->pte_attrs; pte_secure = full->attrs.secure; } @@ -323,14 +329,16 @@ static uint32_t arm_ldl_ptw(CPUARMState *env, S1Translate *ptw, ARMMMUFaultInfo *fi) { CPUState *cs = env_cpu(env); + void *host = ptw->out_host; uint32_t data; - if (likely(ptw->out_host)) { + if (likely(host)) { /* Page tables are in RAM, and we have the host address. */ + data = qatomic_read((uint32_t *)host); if (ptw->out_be) { - data = ldl_be_p(ptw->out_host); + data = be32_to_cpu(data); } else { - data = ldl_le_p(ptw->out_host); + data = le32_to_cpu(data); } } else { /* Page tables are in MMIO. */ @@ -356,15 +364,25 @@ static uint64_t arm_ldq_ptw(CPUARMState *env, S1Translate *ptw, ARMMMUFaultInfo *fi) { CPUState *cs = env_cpu(env); + void *host = ptw->out_host; uint64_t data; - if (likely(ptw->out_host)) { + if (likely(host)) { /* Page tables are in RAM, and we have the host address. */ +#ifdef CONFIG_ATOMIC64 + data = qatomic_read__nocheck((uint64_t *)host); if (ptw->out_be) { - data = ldq_be_p(ptw->out_host); + data = be64_to_cpu(data); } else { - data = ldq_le_p(ptw->out_host); + data = le64_to_cpu(data); } +#else + if (ptw->out_be) { + data = ldq_be_p(host); + } else { + data = ldq_le_p(host); + } +#endif } else { /* Page tables are in MMIO. */ MemTxAttrs attrs = { .secure = ptw->out_secure }; @@ -385,6 +403,91 @@ static uint64_t arm_ldq_ptw(CPUARMState *env, S1Translate *ptw, return data; } +static uint64_t arm_casq_ptw(CPUARMState *env, uint64_t old_val, + uint64_t new_val, S1Translate *ptw, + ARMMMUFaultInfo *fi) +{ + uint64_t cur_val; + void *host = ptw->out_host; + + if (unlikely(!host)) { + fi->type = ARMFault_UnsuppAtomicUpdate; + fi->s1ptw = true; + return 0; + } + + /* + * Raising a stage2 Protection fault for an atomic update to a read-only + * page is delayed until it is certain that there is a change to make. + */ + if (unlikely(!ptw->out_rw)) { + int flags; + void *discard; + + env->tlb_fi = fi; + flags = probe_access_flags(env, ptw->out_virt, MMU_DATA_STORE, + arm_to_core_mmu_idx(ptw->in_ptw_idx), + true, &discard, 0); + env->tlb_fi = NULL; + + if (unlikely(flags & TLB_INVALID_MASK)) { + assert(fi->type != ARMFault_None); + fi->s2addr = ptw->out_virt; + fi->stage2 = true; + fi->s1ptw = true; + fi->s1ns = !ptw->in_secure; + return 0; + } + + /* In case CAS mismatches and we loop, remember writability. */ + ptw->out_rw = true; + } + +#ifdef CONFIG_ATOMIC64 + if (ptw->out_be) { + old_val = cpu_to_be64(old_val); + new_val = cpu_to_be64(new_val); + cur_val = qatomic_cmpxchg__nocheck((uint64_t *)host, old_val, new_val); + cur_val = be64_to_cpu(cur_val); + } else { + old_val = cpu_to_le64(old_val); + new_val = cpu_to_le64(new_val); + cur_val = qatomic_cmpxchg__nocheck((uint64_t *)host, old_val, new_val); + cur_val = le64_to_cpu(cur_val); + } +#else + /* + * We can't support the full 64-bit atomic cmpxchg on the host. + * Because this is only used for FEAT_HAFDBS, which is only for AA64, + * we know that TCG_OVERSIZED_GUEST is set, which means that we are + * running in round-robin mode and could only race with dma i/o. + */ +#ifndef TCG_OVERSIZED_GUEST +# error "Unexpected configuration" +#endif + bool locked = qemu_mutex_iothread_locked(); + if (!locked) { + qemu_mutex_lock_iothread(); + } + if (ptw->out_be) { + cur_val = ldq_be_p(host); + if (cur_val == old_val) { + stq_be_p(host, new_val); + } + } else { + cur_val = ldq_le_p(host); + if (cur_val == old_val) { + stq_le_p(host, new_val); + } + } + if (!locked) { + qemu_mutex_unlock_iothread(); + } +#endif + + return cur_val; +} + static bool get_level1_table_address(CPUARMState *env, ARMMMUIdx mmu_idx, uint32_t *table, uint32_t address) { @@ -1077,7 +1180,7 @@ static bool get_phys_addr_lpae(CPUARMState *env, S1Translate *ptw, uint32_t el = regime_el(env, mmu_idx); uint64_t descaddrmask; bool aarch64 = arm_el_is_aa64(env, el); - uint64_t descriptor; + uint64_t descriptor, new_descriptor; bool nstable; /* TODO: This code does not support shareability levels. */ @@ -1291,7 +1394,9 @@ static bool get_phys_addr_lpae(CPUARMState *env, S1Translate *ptw, if (fi->type != ARMFault_None) { goto do_fault; } + new_descriptor = descriptor; + restart_atomic_update: if (!(descriptor & 1) || (!(descriptor & 2) && (level == 3))) { /* Invalid, or the Reserved level 3 encoding */ goto do_translation_fault; @@ -1337,17 +1442,36 @@ static bool get_phys_addr_lpae(CPUARMState *env, S1Translate *ptw, * to give a correct page or table address, the address field * in a block descriptor is smaller; so we need to explicitly * clear the lower bits here before ORing in the low vaddr bits. + * + * Afterward, descaddr is the final physical address. */ page_size = (1ULL << ((stride * (4 - level)) + 3)); descaddr &= ~(hwaddr)(page_size - 1); descaddr |= (address & (page_size - 1)); + if (likely(!ptw->in_debug)) { + /* + * Access flag. + * If HA is enabled, prepare to update the descriptor below. + * Otherwise, pass the access fault on to software. + */ + if (!(descriptor & (1 << 10))) { + if (param.ha) { + new_descriptor |= 1 << 10; /* AF */ + } else { + fi->type = ARMFault_AccessFlag; + goto do_fault; + } + } + } + /* - * Extract attributes from the descriptor, and apply table descriptors. - * Stage 2 table descriptors do not include any attribute fields. - * HPD disables all the table attributes except NSTable. + * Extract attributes from the (modified) descriptor, and apply + * table descriptors. Stage 2 table descriptors do not include + * any attribute fields. HPD disables all the table attributes + * except NSTable. */ - attrs = descriptor & (MAKE_64BIT_MASK(2, 10) | MAKE_64BIT_MASK(50, 14)); + attrs = new_descriptor & (MAKE_64BIT_MASK(2, 10) | MAKE_64BIT_MASK(50, 14)); if (!regime_is_stage2(mmu_idx)) { attrs |= nstable << 5; /* NS */ if (!param.hpd) { @@ -1361,18 +1485,7 @@ static bool get_phys_addr_lpae(CPUARMState *env, S1Translate *ptw, } } - /* - * Here descaddr is the final physical address, and attributes - * are all in attrs. - */ - if ((attrs & (1 << 10)) == 0) { - /* Access flag */ - fi->type = ARMFault_AccessFlag; - goto do_fault; - } - ap = extract32(attrs, 6, 2); - if (regime_is_stage2(mmu_idx)) { ns = mmu_idx == ARMMMUIdx_Stage2; xn = extract64(attrs, 53, 2); @@ -1389,6 +1502,25 @@ static bool get_phys_addr_lpae(CPUARMState *env, S1Translate *ptw, goto do_fault; } + /* If FEAT_HAFDBS has made changes, update the PTE. */ + if (new_descriptor != descriptor) { + new_descriptor = arm_casq_ptw(env, descriptor, new_descriptor, ptw, fi); + if (fi->type != ARMFault_None) { + goto do_fault; + } + /* + * I_YZSVV says that if the in-memory descriptor has changed, + * then we must use the information in that new value + * (which might include a different output address, different + * attributes, or generate a fault). + * Restart the handling of the descriptor value from scratch. + */ + if (new_descriptor != descriptor) { + descriptor = new_descriptor; + goto restart_atomic_update; + } + } + if (ns) { /* * The NS bit will (as required by the architecture) have no effect if