From patchwork Mon May 11 07:13:04 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 48213 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-la0-f69.google.com (mail-la0-f69.google.com [209.85.215.69]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id F1B8D2121F for ; Mon, 11 May 2015 07:18:08 +0000 (UTC) Received: by layy10 with SMTP id y10sf39174039lay.0 for ; Mon, 11 May 2015 00:18:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:subject:date:message-id :in-reply-to:references:cc:precedence:list-id:list-unsubscribe :list-archive:list-post:list-help:list-subscribe:mime-version :content-type:content-transfer-encoding:sender:errors-to :x-original-sender:x-original-authentication-results:mailing-list; bh=Vunl7kInwaJfjDjsx1A1FndRMlol499ZJExgJYBP0I8=; b=EKZi2R1vCd2Ju/8T4PKfcPRcbpjLfC/fG7Sbfh8x7N5/RvFlQ3LBVYOJa33AbdKVlJ pxKitgmAwfiOg9Ky9ZS+snvTCZhzd5v9dOWo5PFCjsm5xX0E+lsLKrw6pazn+23idQOC gBgejNztGg0SqZ/VC/aJif/I+m1x16Fg1HoG5jqUUHT8toy8SB4l06+nsra5GcW9SF9/ 7V+gWN3qyjn/EHMT4fGN8wFK9avu7BjjshwsZzzWnW71ChYwvtgPoW34MQqaUa9aTSBD CFzNXntyYg/cP1UptGkx3glbra+GZ0IzCmemo4JlOOFQXhPGgWgbvZdSvDXEPRyxwDC3 0ugw== X-Gm-Message-State: ALoCoQnz6co0gdqbXnhsniSh4pyirwEOkMV4ZYPX39UIthohJB7lDNBRoNRT7k1w70njEVe9xTUz X-Received: by 10.180.211.168 with SMTP id nd8mr6779366wic.4.1431328687956; Mon, 11 May 2015 00:18:07 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.170.163 with SMTP id an3ls616765lac.90.gmail; Mon, 11 May 2015 00:18:07 -0700 (PDT) X-Received: by 10.112.202.103 with SMTP id kh7mr6786380lbc.118.1431328687802; Mon, 11 May 2015 00:18:07 -0700 (PDT) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com. [209.85.217.182]) by mx.google.com with ESMTPS id qa2si7851398lbb.153.2015.05.11.00.18.07 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 11 May 2015 00:18:07 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.182 as permitted sender) client-ip=209.85.217.182; Received: by lbbqq2 with SMTP id qq2so87553616lbb.3 for ; Mon, 11 May 2015 00:18:07 -0700 (PDT) X-Received: by 10.112.140.231 with SMTP id rj7mr6879289lbb.76.1431328687658; Mon, 11 May 2015 00:18:07 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.108.230 with SMTP id hn6csp1346415lbb; Mon, 11 May 2015 00:18:06 -0700 (PDT) X-Received: by 10.70.90.162 with SMTP id bx2mr16755211pdb.60.1431328685707; Mon, 11 May 2015 00:18:05 -0700 (PDT) Received: from bombadil.infradead.org (bombadil.infradead.org. [2001:1868:205::9]) by mx.google.com with ESMTPS id z14si16872883pas.56.2015.05.11.00.18.04 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 11 May 2015 00:18:05 -0700 (PDT) Received-SPF: none (google.com: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org does not designate permitted sender hosts) client-ip=2001:1868:205::9; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1Yrhwp-0003w7-8Z; Mon, 11 May 2015 07:16:19 +0000 Received: from mail-wi0-f179.google.com ([209.85.212.179]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1YrhuN-0000ji-Bz for linux-arm-kernel@lists.infradead.org; Mon, 11 May 2015 07:13:48 +0000 Received: by wizk4 with SMTP id k4so93527308wiz.1 for ; Mon, 11 May 2015 00:13:25 -0700 (PDT) X-Received: by 10.194.61.133 with SMTP id p5mr17928424wjr.132.1431328405214; Mon, 11 May 2015 00:13:25 -0700 (PDT) Received: from localhost.localdomain (cag06-7-83-153-85-71.fbx.proxad.net. [83.153.85.71]) by mx.google.com with ESMTPSA id o5sm10728933wia.0.2015.05.11.00.13.23 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 11 May 2015 00:13:24 -0700 (PDT) From: Ard Biesheuvel To: catalin.marinas@arm.com, mark.rutland@arm.com, linux-arm-kernel@lists.infradead.org Subject: [PATCH 06/10] arm64: mm: explicitly bootstrap the linear mapping Date: Mon, 11 May 2015 09:13:04 +0200 Message-Id: <1431328388-3051-7-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1431328388-3051-1-git-send-email-ard.biesheuvel@linaro.org> References: <1431328388-3051-1-git-send-email-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20150511_001347_750034_2A35A3D3 X-CRM114-Status: GOOD ( 21.30 ) X-Spam-Score: -0.7 (/) X-Spam-Report: SpamAssassin version 3.4.0 on bombadil.infradead.org summary: Content analysis details: (-0.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low trust [209.85.212.179 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record -0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3) [209.85.212.179 listed in wl.mailspike.net] -0.0 RCVD_IN_MSPIKE_WL Mailspike good senders Cc: Ard Biesheuvel X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: ard.biesheuvel@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.182 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 In preparation of moving the kernel text out of the linear mapping, ensure that the part of the kernel Image that contains the statically allocated page tables is made accessible via the linear mapping before performing the actual mapping of all of memory. This is needed by the normal mapping routines, that rely on the linear mapping to walk the page tables while manipulating them. In addition, explicitly map the start of DRAM and set the memblock limit so that all early memblock allocations are done from a region that is guaranteed to be mapped. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/vmlinux.lds.S | 19 ++++++- arch/arm64/mm/mmu.c | 109 +++++++++++++++++++++++++++++----------- 2 files changed, 98 insertions(+), 30 deletions(-) diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index ceec4def354b..0b82c4c203fb 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -68,6 +68,18 @@ PECOFF_FILE_ALIGNMENT = 0x200; #define ALIGN_DEBUG_RO_MIN(min) . = ALIGN(min); #endif +/* + * The pgdir region needs to be mappable using a single PMD or PUD sized region, + * so it should not cross a 512 MB or 1 GB alignment boundary, respectively + * (depending on page size). So align to a power-of-2 upper bound of the size + * of the entire __pgdir section. + */ +#if CONFIG_ARM64_PGTABLE_LEVELS == 2 +#define PGDIR_ALIGN (8 * PAGE_SIZE) +#else +#define PGDIR_ALIGN (16 * PAGE_SIZE) +#endif + SECTIONS { /* @@ -160,7 +172,7 @@ SECTIONS BSS_SECTION(0, 0, 0) - .pgdir (NOLOAD) : ALIGN(PAGE_SIZE) { + .pgdir (NOLOAD) : ALIGN(PGDIR_ALIGN) { idmap_pg_dir = .; . += IDMAP_DIR_SIZE; swapper_pg_dir = .; @@ -185,6 +197,11 @@ ASSERT(__idmap_text_end - (__idmap_text_start & ~(SZ_4K - 1)) <= SZ_4K, "ID map text too big or misaligned") /* + * Check that the chosen PGDIR_ALIGN value is sufficient. + */ +ASSERT(SIZEOF(.pgdir) <= ALIGNOF(.pgdir), ".pgdir size exceeds its alignment") + +/* * If padding is applied before .head.text, virt<->phys conversions will fail. */ ASSERT(_text == (PAGE_OFFSET + TEXT_OFFSET), "HEAD is misaligned") diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index f64a817af469..32ce481d90bf 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -380,26 +380,92 @@ static void __init bootstrap_early_mapping(unsigned long addr, } } -static void __init map_mem(void) +/* + * Bootstrap a memory mapping in such a way that it does not require allocation + * of page tables beyond the ones that were allocated statically by + * bootstrap_early_mapping(). + * This is done by finding the memblock that covers pa_base, and intersecting + * it with the naturally aligned 512 MB of 1 GB region (depending on page size) + * that covers pa_base as well and (on 4k pages) round it to section size. + */ +static unsigned long __init bootstrap_region(struct bootstrap_pgtables *reg, + phys_addr_t pa_base, + unsigned long va_offset) { - struct memblock_region *reg; - phys_addr_t limit; + unsigned long va_base = __phys_to_virt(pa_base) + va_offset; + struct memblock_region *mr; + + bootstrap_early_mapping(va_base, reg, + IS_ENABLED(CONFIG_ARM64_64K_PAGES)); + + for_each_memblock(memory, mr) { + phys_addr_t start = mr->base; + phys_addr_t end = start + mr->size; + unsigned long vstart, vend; + + if (start > pa_base || end <= pa_base) + continue; + +#ifdef CONFIG_ARM64_64K_PAGES + /* clip the region to PMD size */ + vstart = max(va_base & PMD_MASK, + round_up(__phys_to_virt(start) + va_offset, + PAGE_SIZE)); + vend = min(round_up(va_base + 1, PMD_SIZE), + round_down(__phys_to_virt(end) + va_offset, + PAGE_SIZE)); +#else + /* clip the region to PUD size */ + vstart = max(va_base & PUD_MASK, + round_up(__phys_to_virt(start) + va_offset, + PMD_SIZE)); + vend = min(round_up(va_base + 1, PUD_SIZE), + round_down(__phys_to_virt(end) + va_offset, + PMD_SIZE)); +#endif + + create_mapping(__pa(vstart - va_offset), vstart, vend - vstart, + PAGE_KERNEL_EXEC); + + return vend; + } + return 0; +} + +/* + * Bootstrap the linear ranges that cover the start of DRAM and swapper_pg_dir + * so that the statically allocated page tables as well as newly allocated ones + * are accessible via the linear mapping. + */ +static void __init bootstrap_linear_mapping(unsigned long va_offset) +{ + static struct bootstrap_pgtables __pgdir bs_pgdir_low, bs_pgdir_high; + unsigned long vend; + + /* Bootstrap the mapping for the beginning of RAM */ + vend = bootstrap_region(&bs_pgdir_low, memblock_start_of_DRAM(), + va_offset); + BUG_ON(vend == 0); /* * Temporarily limit the memblock range. We need to do this as * create_mapping requires puds, pmds and ptes to be allocated from - * memory addressable from the initial direct kernel mapping. - * - * The initial direct kernel mapping, located at swapper_pg_dir, gives - * us PUD_SIZE (4K pages) or PMD_SIZE (64K pages) memory starting from - * PHYS_OFFSET (which must be aligned to 2MB as per - * Documentation/arm64/booting.txt). + * memory addressable from the early linear mapping. */ - if (IS_ENABLED(CONFIG_ARM64_64K_PAGES)) - limit = PHYS_OFFSET + PMD_SIZE; - else - limit = PHYS_OFFSET + PUD_SIZE; - memblock_set_current_limit(limit); + memblock_set_current_limit(__pa(vend - va_offset)); + + /* Bootstrap the linear mapping of the kernel image */ + vend = bootstrap_region(&bs_pgdir_high, __pa(swapper_pg_dir), + va_offset); + if (vend == 0) + panic("Kernel image not covered by memblock"); +} + +static void __init map_mem(void) +{ + struct memblock_region *reg; + + bootstrap_linear_mapping(0); /* map all the memory banks */ for_each_memblock(memory, reg) { @@ -409,21 +475,6 @@ static void __init map_mem(void) if (start >= end) break; -#ifndef CONFIG_ARM64_64K_PAGES - /* - * For the first memory bank align the start address and - * current memblock limit to prevent create_mapping() from - * allocating pte page tables from unmapped memory. - * When 64K pages are enabled, the pte page table for the - * first PGDIR_SIZE is already present in swapper_pg_dir. - */ - if (start < limit) - start = ALIGN(start, PMD_SIZE); - if (end < limit) { - limit = end & PMD_MASK; - memblock_set_current_limit(limit); - } -#endif __map_memblock(start, end); }