From patchwork Wed Jun 11 16:40:04 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Campbell X-Patchwork-Id: 31791 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-pb0-f69.google.com (mail-pb0-f69.google.com [209.85.160.69]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id EAB7B2054B for ; Wed, 11 Jun 2014 16:41:21 +0000 (UTC) Received: by mail-pb0-f69.google.com with SMTP id up15sf3775597pbc.8 for ; Wed, 11 Jun 2014 09:41:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:date:message-id:in-reply-to :references:mime-version:cc:subject:precedence:list-id :list-unsubscribe:list-post:list-help:list-subscribe:sender :errors-to:x-original-sender:x-original-authentication-results :mailing-list:list-archive:content-type:content-transfer-encoding; bh=4sh140oGfeTAE+S/Hw/B79MXbphIkhFqF53Zn3ha/Ic=; b=i6lNpia2T0ogd1VpkzD7wACWpvuR8qP8ys/PrfRICBUtj8LKafR+r5TnkGPN5sfsoC GeyKaR/7o9Vvz1bH4zQzt/10s+Ibh2E0TYscf1mStUXa3hlmc5rX/xxrSZs5Ga605aWI jG8cfj8jso2/x9RRKrRuyVyhDzMY0kqNydGWLblCS3oKvWrTvZug0rvwkJraR3kOZaie ALEo3oCpgPPgrAedJSUt1jHdxIJWhRf8Ek/0lJBCeDIbC0pn81Zbqz8JZFPFrzCsr96Z cgUVBQwbZZpB3l5Vlm4dy0UvgNctRGRD4VDmGIXSF6eJpplxSluuI4R5n9ql45FvRrZS NSZA== X-Gm-Message-State: ALoCoQnDUJBZo2JXg1JVspBueavPJQNjD0XX8L8ei1rRQQApLtPEUHwUgRvUyF/8h+ykuzbmfFgc X-Received: by 10.68.253.66 with SMTP id zy2mr991712pbc.1.1402504881285; Wed, 11 Jun 2014 09:41:21 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.22.39 with SMTP id 36ls2583363qgm.75.gmail; Wed, 11 Jun 2014 09:41:21 -0700 (PDT) X-Received: by 10.52.130.18 with SMTP id oa18mr965948vdb.78.1402504881009; Wed, 11 Jun 2014 09:41:21 -0700 (PDT) Received: from mail-ve0-f172.google.com (mail-ve0-f172.google.com [209.85.128.172]) by mx.google.com with ESMTPS id vs2si14410791veb.10.2014.06.11.09.41.20 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 11 Jun 2014 09:41:20 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.128.172 as permitted sender) client-ip=209.85.128.172; Received: by mail-ve0-f172.google.com with SMTP id jz11so7449358veb.17 for ; Wed, 11 Jun 2014 09:41:20 -0700 (PDT) X-Received: by 10.220.68.83 with SMTP id u19mr1500700vci.52.1402504880910; Wed, 11 Jun 2014 09:41:20 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.221.54.6 with SMTP id vs6csp317883vcb; Wed, 11 Jun 2014 09:41:20 -0700 (PDT) X-Received: by 10.52.68.72 with SMTP id u8mr3973338vdt.29.1402504880439; Wed, 11 Jun 2014 09:41:20 -0700 (PDT) Received: from lists.xen.org (lists.xen.org. [50.57.142.19]) by mx.google.com with ESMTPS id tv5si13134855vcb.38.2014.06.11.09.41.19 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Wed, 11 Jun 2014 09:41:20 -0700 (PDT) Received-SPF: none (google.com: xen-devel-bounces@lists.xen.org does not designate permitted sender hosts) client-ip=50.57.142.19; Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1WulZd-0001DL-RY; Wed, 11 Jun 2014 16:40:29 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1WulZW-00015x-MO for xen-devel@lists.xen.org; Wed, 11 Jun 2014 16:40:23 +0000 Received: from [85.158.137.68:14880] by server-1.bemta-3.messagelabs.com id 67/26-11134-57688935; Wed, 11 Jun 2014 16:40:21 +0000 X-Env-Sender: Ian.Campbell@citrix.com X-Msg-Ref: server-7.tower-31.messagelabs.com!1402504818!9471676!2 X-Originating-IP: [66.165.176.89] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni44OSA9PiAyMDMwMDc=\n X-StarScan-Received: X-StarScan-Version: 6.11.3; banners=-,-,- X-VirusChecked: Checked Received: (qmail 15192 invoked from network); 11 Jun 2014 16:40:20 -0000 Received: from smtp.citrix.com (HELO SMTP.CITRIX.COM) (66.165.176.89) by server-7.tower-31.messagelabs.com with RC4-SHA encrypted SMTP; 11 Jun 2014 16:40:20 -0000 X-IronPort-AV: E=Sophos;i="5.01,459,1400025600"; d="scan'208";a="142556080" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 11 Jun 2014 16:40:06 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.3.181.6; Wed, 11 Jun 2014 12:40:05 -0400 Received: from marilith-n13-p0.uk.xensource.com ([10.80.229.115] helo=marilith-n13.uk.xensource.com.) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1WulZF-0004h3-4T; Wed, 11 Jun 2014 17:40:05 +0100 From: Ian Campbell To: Date: Wed, 11 Jun 2014 17:40:04 +0100 Message-ID: <1402504804-29173-8-git-send-email-ian.campbell@citrix.com> X-Mailer: git-send-email 1.7.10.4 In-Reply-To: <1402504640.16332.50.camel@kazak.uk.xensource.com> References: <1402504640.16332.50.camel@kazak.uk.xensource.com> MIME-Version: 1.0 X-DLP: MIA2 Cc: julien.grall@linaro.org, tim@xen.org, Ian Campbell , stefano.stabellini@eu.citrix.com Subject: [Xen-devel] [PATCH v2 8/8] xen: arm: allocate more than one bank for 1:1 domain 0 if needed X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Post: , List-Help: , List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: ian.campbell@citrix.com X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.128.172 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Archive: Depending on where Xen and the initial modules were loaded into RAM we may not be able to allocate a suitably contiguous block of memory for dom0. Especially since the allocations made by alloc_domheap_pages are naturally aligned (e.g. a 1GB allocation must be at a 1GB boundary). The alignment requirement in particular is a huge limitation, meaning that even dom0_mem0=1G on a 2GB system is pretty likely to fail unless you are very careful with your load addresses. People were also having trouble with dom0 > 128MB on the 1GB cubieboard2 when using fairly standard load addresses for things. This patch tries to allocate multiple banks of memory in order to try and satisfy the entire requested domain 0 allocation. Sadly this turns out to be pretty tricky to arrange (see the large comment in the code). Signed-off-by: Ian Campbell Cc: Karim Raslan --- v2: New patch --- xen/arch/arm/domain_build.c | 278 +++++++++++++++++++++++++++++++++++++++---- 1 file changed, 256 insertions(+), 22 deletions(-) diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c index 46a3619..f37127f 100644 --- a/xen/arch/arm/domain_build.c +++ b/xen/arch/arm/domain_build.c @@ -44,6 +44,13 @@ custom_param("dom0_mem", parse_dom0_mem); # define DPRINT(fmt, args...) do {} while ( 0 ) #endif +//#define DEBUG_11_ALLOCATION +#ifdef DEBUG_11_ALLOCATION +# define D11PRINT(fmt, args...) printk(XENLOG_DEBUG fmt, ##args) +#else +# define D11PRINT(fmt, args...) do {} while ( 0 ) +#endif + /* * Amount of extra space required to dom0's device tree. No new nodes * are added (yet) but one terminating reserve map entry (16 bytes) is @@ -66,39 +73,266 @@ struct vcpu *__init alloc_dom0_vcpu0(struct domain *dom0) return alloc_vcpu(dom0, 0, 0); } -static void allocate_memory_11(struct domain *d, struct kernel_info *kinfo) +static unsigned int get_11_allocation_size(paddr_t size) { - paddr_t start; - paddr_t size; - struct page_info *pg; - unsigned int order = get_order_from_bytes(dom0_mem); - int res; - paddr_t spfn; + /* + * get_order_from_bytes returns the order greater than or equal to + * the given size, but we need less than or equal. Adding one to + * the size pushes an evenly aligned size into the next order, so + * we can then unconditionally subtract 1 from the order which is + * returned. + */ + return get_order_from_bytes(size + 1) - 1; +} - if ( is_32bit_domain(d) ) - pg = alloc_domheap_pages(d, order, MEMF_bits(32)); - else - pg = alloc_domheap_pages(d, order, 0); - if ( !pg ) - panic("Failed to allocate contiguous memory for dom0"); +/* + * Insert the given pages into a memory bank, banks are ordered by address. + * + * Returns false if the memory would be below bank 0. + */ +static bool_t insert_11_bank(struct domain *d, + struct kernel_info *kinfo, + struct page_info *pg, + unsigned int order) +{ + int res, i; + paddr_t spfn; + paddr_t start, size; spfn = page_to_mfn(pg); start = pfn_to_paddr(spfn); size = pfn_to_paddr((1 << order)); - // 1:1 mapping - printk("Populate P2M %#"PRIx64"->%#"PRIx64" (1:1 mapping for dom0)\n", - start, start + size); - res = guest_physmap_add_page(d, spfn, spfn, order); + D11PRINT("Allocated %#"PRIpaddr"-%#"PRIpaddr" (%ldMB/%ldMB, order %d)\n", + start, start + size, + 1UL << (order+PAGE_SHIFT-20), + /* Don't want format this as PRIpaddr (16 digit hex) */ + (unsigned long)(kinfo->unassigned_mem >> 20), + order); - if ( res ) - panic("Unable to add pages in DOM0: %d", res); + if ( kinfo->mem.nr_banks > 0 && start + size < kinfo->mem.bank[0].start ) + { + D11PRINT("Allocation is below bank 0, not using\n"); + free_domheap_pages(pg, order); + return false; + } - kinfo->mem.bank[0].start = start; - kinfo->mem.bank[0].size = size; - kinfo->mem.nr_banks = 1; + res = guest_physmap_add_page(d, spfn, spfn, order); + if ( res ) + panic("Failed map pages to DOM0: %d", res); kinfo->unassigned_mem -= size; + + if ( kinfo->mem.nr_banks == 0 ) + { + kinfo->mem.bank[0].start = start; + kinfo->mem.bank[0].size = size; + kinfo->mem.nr_banks = 1; + return true; + } + + for( i = 0; i < kinfo->mem.nr_banks; i++ ) + { + struct membank *bank = &kinfo->mem.bank[i]; + + /* If possible merge new memory into the start of the bank */ + if ( bank->start == start+size ) + { + bank->start = start; + bank->size += size; + return true; + } + + /* If possible merge new memory onto the end of the bank */ + if ( start == bank->start + bank->size ) + { + bank->size += size; + return true; + } + + /* + * Otherwise if it is below this bank insert new memory in a + * new bank before this one. If there was a lower bank we + * could have inserted the memory into/before we would already + * have done so, so this must be the right place. + */ + if ( start + size < bank->start && kinfo->mem.nr_banks < NR_MEM_BANKS ) + { + memmove(bank + 1, bank, sizeof(*bank)*(kinfo->mem.nr_banks - i)); + kinfo->mem.nr_banks++; + bank->start = start; + bank->size = size; + return true; + } + } + + if ( i == kinfo->mem.nr_banks && kinfo->mem.nr_banks < NR_MEM_BANKS ) + { + struct membank *bank = &kinfo->mem.bank[kinfo->mem.nr_banks]; + + bank->start = start; + bank->size = size; + kinfo->mem.nr_banks++; + return true; + } + + panic("Failed add pages to DOM0 memory bank"); +} + +/* + * This is all pretty horrible. + * + * Requirements: + * + * 1. The dom0 kernel should be loaded within the first 128MB of RAM. This + * is necessary at least for Linux zImage kernels, which are all we + * support today. + * 2. We want to put the dom0 kernel, ramdisk and DTB in the same + * bank. Partly this is just easier for us to deal with, but also + * the ramdisk and DTB must be placed within a certain proximity of + * the kernel within RAM. + * 3. For 32-bit dom0 we want to place as much of the RAM as we + * reasonably can below 4GB, so that it can be used by non-LPAE + * enabled kernels. + * 4. For 32-bit dom0 the kernel must be located below 4GB. + * 5. We want to have a few largers banks rather than many smaller ones. + * + * For the first two requirements we need to make sure that the lowest + * bank is sufficiently large. + * + * For convenience we also sort the banks by physical address. + * + * The memory allocator does not really give us the flexibility to + * meet these requirements directly. So instead of proceed as follows: + * + * We first allocate the largest allocation we can as low as we + * can. This then becomes the first bank. This bank must be at least + * 128MB (or dom0_mem if that is smaller). + * + * Then we start allocating more memory, trying to allocate the + * largest possible size and trying smaller sizes until we + * successfully allocate something. + * + * We then try and insert this memory in to the list of banks. If it + * can be merged into an existing bank then this is trivial. + * + * If the new memory is before the first bank (and cannot be merged + * into it) then we give up. Since the allocator prefers to allocate + * high addresses first and the first bank has already been allocated + * to be as low as possible this likely means we wouldn't have been + * able to allocate much more memory anyway. + * + * Otherwise we insert a new bank. If we've reached MAX_NR_BANKS then + * we give up. + * + * For 32-bit domain we require that the initial allocation for the + * first bank is under 4G. Then for the subsequent allocations we + * initially allocate memory only from below 4GB. Once that runs out + * (as described above) we allow higher allocations and continue until + * that runs out (or we have allocated sufficient dom0 memory). + */ +static void allocate_memory_11(struct domain *d, struct kernel_info *kinfo) +{ + const unsigned int min_low_order = + get_order_from_bytes(min_t(paddr_t, dom0_mem, MB(128))); + const unsigned int min_order = get_order_from_bytes(MB(4)); + struct page_info *pg; + unsigned int order = get_11_allocation_size(kinfo->unassigned_mem); + int i; + + bool_t lowmem = is_32bit_domain(d); + unsigned int bits; + + printk("Allocating 1:1 mappings totalling %ldMB for dom0:\n", + /* Don't want format this as PRIpaddr (16 digit hex) */ + (unsigned long)(kinfo->unassigned_mem >> 20)); + + kinfo->mem.nr_banks = 0; + + /* + * First try and allocate the largest thing we can as low as + * possible to be bank 0. + */ + while ( order > min_low_order ) + { + for ( bits = order ; bits < (lowmem ? 32 : PADDR_BITS); bits++ ) + { + pg = alloc_domheap_pages(d, order, MEMF_bits(bits)); + if ( pg != NULL ) + goto got_bank0; + } + order--; + } + + panic("Unable to allocate first memory bank"); + + got_bank0: + + if ( !insert_11_bank(d, kinfo, pg, order) ) + BUG(); /* Cannot fail for first bank */ + + /* Now allocate more memory and fill in additional banks */ + + order = get_11_allocation_size(kinfo->unassigned_mem); + while ( kinfo->unassigned_mem && kinfo->mem.nr_banks < NR_MEM_BANKS ) + { + pg = alloc_domheap_pages(d, order, lowmem ? MEMF_bits(32) : 0); + if ( !pg ) + { + order --; + + if ( lowmem && order < min_low_order) + { + D11PRINT("Failed at min_low_order, allow high allocations\n"); + order = get_11_allocation_size(kinfo->unassigned_mem); + lowmem = false; + continue; + } + if ( order >= min_order ) + continue; + + /* No more we can do */ + break; + } + + if ( !insert_11_bank(d, kinfo, pg, order) ) + { + if ( lowmem ) + { + D11PRINT("Allocation below bank 0, allow high allocations\n"); + order = get_11_allocation_size(kinfo->unassigned_mem); + lowmem = false; + continue; + } + else + { + D11PRINT("Allocation below bank 0\n"); + break; + } + } + + /* + * Success, next time around try again to get the largest order + * allocation possible. + */ + order = get_11_allocation_size(kinfo->unassigned_mem); + } + + if ( kinfo->unassigned_mem ) + printk("WARNING: Failed to allocate requested dom0 memory." + /* Don't want format this as PRIpaddr (16 digit hex) */ + " %ldMB unallocated\n", + (unsigned long)kinfo->unassigned_mem >> 20); + + for( i = 0; i < kinfo->mem.nr_banks; i++ ) + { + printk("BANK[%d] %#"PRIpaddr"-%#"PRIpaddr" (%ldMB)\n", + i, + kinfo->mem.bank[i].start, + kinfo->mem.bank[i].start + kinfo->mem.bank[i].size, + /* Don't want format this as PRIpaddr (16 digit hex) */ + (unsigned long)(kinfo->mem.bank[i].size >> 20)); + } } static void allocate_memory(struct domain *d, struct kernel_info *kinfo)