From patchwork Wed Jan 22 16:35:31 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Campbell X-Patchwork-Id: 23554 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-oa0-f71.google.com (mail-oa0-f71.google.com [209.85.219.71]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 3BA01218CB for ; Wed, 22 Jan 2014 18:39:38 +0000 (UTC) Received: by mail-oa0-f71.google.com with SMTP id g12sf2782958oah.2 for ; Wed, 22 Jan 2014 10:39:37 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:message-id:from:to:date :organization:mime-version:cc:subject:precedence:list-id :list-unsubscribe:list-post:list-help:list-subscribe:sender :errors-to:x-original-sender:x-original-authentication-results :mailing-list:list-archive:content-type:content-transfer-encoding; bh=sNDy09tkwtw713VOdrmeX0UElaNjb/WPsVi+KtuywXA=; b=Zkx29v1jYVSo5Z429vULxNnJtdSTxp8cfEbxfqmaidtqFsEYEBce44xHQmRaVSeDl/ Di+EXzXCjSoEOYxyy4eabvfVzuYnZLUvy6NvT43Q7H5nVHxj5GJ4QZc68wldqf7f7I5t CdVJQ/pFemBAriFQnreNr5tEJP8RNXRnPJQL1wM9C2+0NSXutwkjKPOSwMPEUoeSUgbW fNAvCExjFLcqA02eWX9LWq5x5YaUDaGxUjJuKHTWj8oTMyzx1Jur1EX2EFzs5uQLHfO7 yBthe9O5/LuvzqiPtmyn9AyLB3woqVmr3/ZG4l5qJteFDn5i9k2/Zo2o5ORgvJLy3YX3 3GJA== X-Gm-Message-State: ALoCoQl5RfKoRoHtJoGqKbgj1vu+pAtr1bUvgcfvCN7uKniIB4CEnU0xiPG26ABduBwvz8oT7fxp X-Received: by 10.182.33.6 with SMTP id n6mr1170472obi.6.1390415977324; Wed, 22 Jan 2014 10:39:37 -0800 (PST) X-BeenThere: patchwork-forward@linaro.org Received: by 10.49.108.65 with SMTP id hi1ls138959qeb.24.gmail; Wed, 22 Jan 2014 10:39:37 -0800 (PST) X-Received: by 10.58.211.130 with SMTP id nc2mr1861235vec.7.1390415977209; Wed, 22 Jan 2014 10:39:37 -0800 (PST) Received: from mail-ve0-f180.google.com (mail-ve0-f180.google.com [209.85.128.180]) by mx.google.com with ESMTPS id xg12si5040223vcb.102.2014.01.22.10.39.37 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 22 Jan 2014 10:39:37 -0800 (PST) Received-SPF: neutral (google.com: 209.85.128.180 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.128.180; Received: by mail-ve0-f180.google.com with SMTP id db12so469419veb.25 for ; Wed, 22 Jan 2014 10:39:37 -0800 (PST) X-Received: by 10.221.37.1 with SMTP id tc1mr1928548vcb.32.1390415977100; Wed, 22 Jan 2014 10:39:37 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.174.196 with SMTP id u4csp204118vcz; Wed, 22 Jan 2014 10:39:36 -0800 (PST) X-Received: by 10.42.226.66 with SMTP id iv2mr2500399icb.11.1390415976258; Wed, 22 Jan 2014 10:39:36 -0800 (PST) Received: from lists.xen.org (lists.xen.org. [50.57.142.19]) by mx.google.com with ESMTPS id ha7si15276838icc.38.2014.01.22.10.39.35 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Wed, 22 Jan 2014 10:39:36 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of xen-devel-bounces@lists.xen.org designates 50.57.142.19 as permitted sender) client-ip=50.57.142.19; Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1W60mA-0001G6-Ey; Wed, 22 Jan 2014 16:35:39 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1W60m8-0001G1-VR for xen-devel@lists.xen.org; Wed, 22 Jan 2014 16:35:37 +0000 Received: from [85.158.137.68:18669] by server-4.bemta-3.messagelabs.com id 21/96-10414-853FFD25; Wed, 22 Jan 2014 16:35:36 +0000 X-Env-Sender: Ian.Campbell@citrix.com X-Msg-Ref: server-12.tower-31.messagelabs.com!1390408533!7061452!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n X-StarScan-Received: X-StarScan-Version: 6.9.16; banners=-,-,- X-VirusChecked: Checked Received: (qmail 1268 invoked from network); 22 Jan 2014 16:35:34 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-12.tower-31.messagelabs.com with RC4-SHA encrypted SMTP; 22 Jan 2014 16:35:34 -0000 X-IronPort-AV: E=Sophos;i="4.95,700,1384300800"; d="scan'208";a="93338131" Received: from accessns.citrite.net (HELO FTLPEX01CL02.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 22 Jan 2014 16:35:32 +0000 Received: from [10.80.2.80] (10.80.2.80) by FTLPEX01CL02.citrite.net (10.13.107.79) with Microsoft SMTP Server id 14.2.342.4; Wed, 22 Jan 2014 11:35:32 -0500 Message-ID: <1390408531.32519.78.camel@kazak.uk.xensource.com> From: Ian Campbell To: Julien Grall , Stefano Stabellini Date: Wed, 22 Jan 2014 16:35:31 +0000 Organization: Citrix Systems, Inc. X-Mailer: Evolution 3.4.4-3 MIME-Version: 1.0 X-Originating-IP: [10.80.2.80] X-DLP: MIA2 Cc: xen-devel Subject: [Xen-devel] xen/arm: Alternative start of day cache coherency X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Post: , List-Help: , List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: ian.campbell@citrix.com X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.128.180 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Archive: Julien, I wonder if the following is any better than the current stuff in staging for the issue you are seeing with BSD at start of day? Can you try it please. It has survived >1000 bootloops on Midway and >50 on Mustang, both are still going. It basically does a cache clean on all RAM mapped in the p2m. Anything in the cache is either the result of an earlier scrub of the page or something toolstack just wrote, so there is no need to be concerned about clean vs. invalidate -- clean is always correct. This should ensure that the guest has no dirty pages when it starts. This nobbles the HCR_DC based stuff, too since it is no longer necessary. This avoids concerns about guests which enable MMU before caches. It contains debug BUG()s in various trap locations to trap if the guest experiences any incoherence and has lots of other debugging left in etc. (and hacks wrt prototypes not in headers) Ian. diff --git a/tools/libxc/xc_domain.c b/tools/libxc/xc_domain.c index e1d1bec..60c3091 100644 --- a/tools/libxc/xc_domain.c +++ b/tools/libxc/xc_domain.c @@ -48,6 +48,14 @@ int xc_domain_create(xc_interface *xch, return 0; } +int xc_domain_cacheflush(xc_interface *xch, uint32_t domid) +{ + DECLARE_DOMCTL; + domctl.cmd = XEN_DOMCTL_cacheflush; + domctl.domain = (domid_t)domid; + domctl.u.cacheflush.start_mfn = 0; + return do_domctl(xch, &domctl); +} int xc_domain_pause(xc_interface *xch, uint32_t domid) diff --git a/tools/libxc/xenctrl.h b/tools/libxc/xenctrl.h index 13f816b..43dae5c 100644 --- a/tools/libxc/xenctrl.h +++ b/tools/libxc/xenctrl.h @@ -453,6 +453,7 @@ int xc_domain_create(xc_interface *xch, xen_domain_handle_t handle, uint32_t flags, uint32_t *pdomid); +int xc_domain_cacheflush(xc_interface *xch, uint32_t domid); /* Functions to produce a dump of a given domain diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c index a604cd8..55c86f0 100644 --- a/tools/libxl/libxl_create.c +++ b/tools/libxl/libxl_create.c @@ -1364,7 +1364,10 @@ static void domain_create_cb(libxl__egc *egc, STATE_AO_GC(cdcs->dcs.ao); if (!rc) + { *cdcs->domid_out = domid; + xc_domain_cacheflush(CTX->xch, domid); + } libxl__ao_complete(egc, ao, rc); } diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c index 635a9a4..2edd09d 100644 --- a/xen/arch/arm/domain.c +++ b/xen/arch/arm/domain.c @@ -475,7 +475,8 @@ int vcpu_initialise(struct vcpu *v) return rc; v->arch.sctlr = SCTLR_GUEST_INIT; - v->arch.default_cache = true; + //v->arch.default_cache = true; + v->arch.default_cache = false; /* * By default exposes an SMP system with AFF0 set to the VCPU ID diff --git a/xen/arch/arm/domctl.c b/xen/arch/arm/domctl.c index 546e86b..9e3b37d 100644 --- a/xen/arch/arm/domctl.c +++ b/xen/arch/arm/domctl.c @@ -11,12 +11,24 @@ #include #include #include +#include + +extern long p2m_cache_flush(struct domain *d, xen_pfn_t *start_mfn); long arch_do_domctl(struct xen_domctl *domctl, struct domain *d, XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl) { switch ( domctl->cmd ) { + case XEN_DOMCTL_cacheflush: + { + long rc = p2m_cache_flush(d, &domctl->u.cacheflush.start_mfn); + if ( __copy_to_guest(u_domctl, domctl, 1) ) + rc = -EFAULT; + + return rc; + } + default: return subarch_do_domctl(domctl, d, u_domctl); } diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c index 85ca330..f35ed57 100644 --- a/xen/arch/arm/p2m.c +++ b/xen/arch/arm/p2m.c @@ -228,15 +228,26 @@ enum p2m_operation { ALLOCATE, REMOVE, RELINQUISH, + CACHEFLUSH, }; +static void do_one_cacheflush(paddr_t mfn) +{ + void *v = map_domain_page(mfn); + + flush_xen_dcache_va_range(v, PAGE_SIZE); + + unmap_domain_page(v); +} + static int create_p2m_entries(struct domain *d, enum p2m_operation op, paddr_t start_gpaddr, paddr_t end_gpaddr, paddr_t maddr, int mattr, - p2m_type_t t) + p2m_type_t t, + xen_pfn_t *last_mfn) { int rc; struct p2m_domain *p2m = &d->arch.p2m; @@ -381,18 +392,42 @@ static int create_p2m_entries(struct domain *d, count++; } break; + case CACHEFLUSH: + { + if ( !pte.p2m.valid || !p2m_is_ram(pte.p2m.type) ) + { + count++; + break; + } + + count += 0x10; + + do_one_cacheflush(pte.p2m.base); + } + break; } + if ( last_mfn ) + *last_mfn = addr >> PAGE_SHIFT; + /* Preempt every 2MiB (mapped) or 32 MiB (unmapped) - arbitrary */ - if ( op == RELINQUISH && count >= 0x2000 ) + switch ( op ) { - if ( hypercall_preempt_check() ) + case RELINQUISH: + case CACHEFLUSH: + if (count >= 0x2000 && hypercall_preempt_check() ) { p2m->next_gfn_to_relinquish = addr >> PAGE_SHIFT; rc = -EAGAIN; goto out; } count = 0; + break; + case INSERT: + case ALLOCATE: + case REMOVE: + /* No preemption */ + break; } /* Got the next page */ @@ -439,7 +474,7 @@ int p2m_populate_ram(struct domain *d, paddr_t end) { return create_p2m_entries(d, ALLOCATE, start, end, - 0, MATTR_MEM, p2m_ram_rw); + 0, MATTR_MEM, p2m_ram_rw, NULL); } int map_mmio_regions(struct domain *d, @@ -448,7 +483,7 @@ int map_mmio_regions(struct domain *d, paddr_t maddr) { return create_p2m_entries(d, INSERT, start_gaddr, end_gaddr, - maddr, MATTR_DEV, p2m_mmio_direct); + maddr, MATTR_DEV, p2m_mmio_direct, NULL); } int guest_physmap_add_entry(struct domain *d, @@ -460,7 +495,7 @@ int guest_physmap_add_entry(struct domain *d, return create_p2m_entries(d, INSERT, pfn_to_paddr(gpfn), pfn_to_paddr(gpfn + (1 << page_order)), - pfn_to_paddr(mfn), MATTR_MEM, t); + pfn_to_paddr(mfn), MATTR_MEM, t, NULL); } void guest_physmap_remove_page(struct domain *d, @@ -470,7 +505,7 @@ void guest_physmap_remove_page(struct domain *d, create_p2m_entries(d, REMOVE, pfn_to_paddr(gpfn), pfn_to_paddr(gpfn + (1<next_gfn_to_relinquish), pfn_to_paddr(p2m->max_mapped_gfn), pfn_to_paddr(INVALID_MFN), - MATTR_MEM, p2m_invalid); + MATTR_MEM, p2m_invalid, NULL); +} + +long p2m_cache_flush(struct domain *d, xen_pfn_t *start_mfn) +{ + struct p2m_domain *p2m = &d->arch.p2m; + + printk("dom%d p2m cache flush from mfn %"PRI_xen_pfn" RELIN %lx\n", + d->domain_id, *start_mfn, p2m->next_gfn_to_relinquish); + + *start_mfn = MAX(*start_mfn, p2m->next_gfn_to_relinquish); + + printk("dom%d p2m cache flush: %"PRIpaddr"-%"PRIpaddr"\n", + d->domain_id, + pfn_to_paddr(*start_mfn), + pfn_to_paddr(p2m->max_mapped_gfn)); + + return create_p2m_entries(d, CACHEFLUSH, + pfn_to_paddr(*start_mfn), + pfn_to_paddr(p2m->max_mapped_gfn), + pfn_to_paddr(INVALID_MFN), + MATTR_MEM, p2m_invalid, start_mfn); } unsigned long gmfn_to_mfn(struct domain *d, unsigned long gpfn) diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c index 48a6fcc..546f7ce 100644 --- a/xen/arch/arm/traps.c +++ b/xen/arch/arm/traps.c @@ -1283,6 +1283,7 @@ static void advance_pc(struct cpu_user_regs *regs, union hsr hsr) static void update_sctlr(struct vcpu *v, uint32_t val) { + BUG(); /* * If MMU (SCTLR_M) is now enabled then we must disable HCR.DC * because they are incompatible. @@ -1628,6 +1629,7 @@ static void do_trap_instr_abort_guest(struct cpu_user_regs *regs, union hsr hsr) { register_t addr = READ_SYSREG(FAR_EL2); + BUG(); inject_iabt_exception(regs, addr, hsr.len); } @@ -1683,6 +1685,8 @@ static void do_trap_data_abort_guest(struct cpu_user_regs *regs, } bad_data_abort: + show_execution_state(regs); + panic("DABT"); inject_dabt_exception(regs, info.gva, hsr.len); } diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h index 91f01fa..d7b22c3 100644 --- a/xen/include/public/domctl.h +++ b/xen/include/public/domctl.h @@ -885,6 +885,13 @@ struct xen_domctl_set_max_evtchn { typedef struct xen_domctl_set_max_evtchn xen_domctl_set_max_evtchn_t; DEFINE_XEN_GUEST_HANDLE(xen_domctl_set_max_evtchn_t); +struct xen_domctl_cacheflush { + /* Updated for progress */ + xen_pfn_t start_mfn; +}; +typedef struct xen_domctl_cacheflush xen_domctl_cacheflush_t; +DEFINE_XEN_GUEST_HANDLE(xen_domctl_cacheflush_t); + struct xen_domctl { uint32_t cmd; #define XEN_DOMCTL_createdomain 1 @@ -954,6 +961,7 @@ struct xen_domctl { #define XEN_DOMCTL_setnodeaffinity 68 #define XEN_DOMCTL_getnodeaffinity 69 #define XEN_DOMCTL_set_max_evtchn 70 +#define XEN_DOMCTL_cacheflush 71 #define XEN_DOMCTL_gdbsx_guestmemio 1000 #define XEN_DOMCTL_gdbsx_pausevcpu 1001 #define XEN_DOMCTL_gdbsx_unpausevcpu 1002 @@ -1012,6 +1020,7 @@ struct xen_domctl { struct xen_domctl_set_max_evtchn set_max_evtchn; struct xen_domctl_gdbsx_memio gdbsx_guest_memio; struct xen_domctl_set_broken_page_p2m set_broken_page_p2m; + struct xen_domctl_cacheflush cacheflush; struct xen_domctl_gdbsx_pauseunp_vcpu gdbsx_pauseunp_vcpu; struct xen_domctl_gdbsx_domstatus gdbsx_domstatus; uint8_t pad[128];