From patchwork Wed Aug 19 12:13:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 247957 Delivered-To: patch@linaro.org Received: by 2002:a05:6e02:522:0:0:0:0 with SMTP id h2csp404488ils; Wed, 19 Aug 2020 05:14:03 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzB3+xZlq0GlfZ2JujYZ9bcMz+AlD9tVrZYiE4pw6GQiwx5CVvXYcqqFkToIxdD8YlGGC8R X-Received: by 2002:a50:a165:: with SMTP id 92mr21071356edj.320.1597839242998; Wed, 19 Aug 2020 05:14:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1597839242; cv=none; d=google.com; s=arc-20160816; b=rSQ8ovcnB9SFYxn0s8czzyGLNuRduOQYIjZwoippkQJCdCtMmZ4YB0pTRduvtfzJBT rSLt3RwR0lNME3/kmRbR6vuQN54fvY4yfyu5YBniIml3zYmHuCCRLtYWrpf4ocZXvzLq wbbcNpqEo4ZMGNe5X2CvoGBZdqCQEM4CpD0Uzz1RfU5ipKNQcTaTcFvHaxZwQe1sGJ55 5Qiod9yHHzetZqaGnZ1O1gd3JaVp6GkcmKftUnbNC5qmFJvyDdd1pj1wPyGvGG0ZcI4G jUtnmKrXWGNHtGJd5MPcwoEibPeESvVS7uLwkDTL3TE9BNDFw8FwEkSWPXDYp/pTGAfN VMyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=dWRN3UitV84HGoOlJlYMpiRZtz1R8um5tM2e0ssLEoM=; b=tux6eodHGLAOMS/YDk21wiXtiLzCLU9HufuEjSnbVVaKwdO3vBSv7ClFko64kC9onr lPzQ5KjFfZeESfKP8zWzwF6HoT+Y25qXxeFNCukuKhbqJRNYa/7QWnaewPZ2Uyed3Vow PCTjVOYKqTdesrd1ypOMKcXzFuxdO123C/JxHoS9ai7ivQ6+TRs2FYoyVjTLvHldi+xg XcPociFbAiOG18PXEmofL1pdfWfSwSq5LLT7vHTDDxQmaI56D3vbPAzmqkOGBc5DLOgi Gc1Ff8zWpCG6AicjbCybndW68cM7ER9LM5lX2cNE7udGxvZUUmucHY64kE2a7lk8SOh5 KzUQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=zFciX799; spf=pass (google.com: domain of linux-kselftest-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kselftest-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id b6si16772273eja.559.2020.08.19.05.14.02; Wed, 19 Aug 2020 05:14:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kselftest-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=zFciX799; spf=pass (google.com: domain of linux-kselftest-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kselftest-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726961AbgHSMOC (ORCPT + 6 others); Wed, 19 Aug 2020 08:14:02 -0400 Received: from mail.kernel.org ([198.145.29.99]:51308 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726798AbgHSMOB (ORCPT ); Wed, 19 Aug 2020 08:14:01 -0400 Received: from localhost (fw-tnat.cambridge.arm.com [217.140.96.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5D0C720738; Wed, 19 Aug 2020 12:14:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1597839240; bh=cU6dUF2JesHsQbTewHHBbh8QA6GTXyIQPYFFzE/8QWk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=zFciX799O1aIbdBKyUraqxG+H21PFFm1inO08gpDk6VLww0Uaisiyic/PK7vHxChH gjH23BMddwKk9LCZ/K5onXUmlUiS1s4lXQF0vGTPPLwEu7Q+fEJpwGS/9SV9XW+Fk/ 09mO1Drjcl2rZMV3njg2HkvKZ3ZQVdTb3ggP14x0= From: Mark Brown To: Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, Vincenzo Frascino , linux-kselftest@vger.kernel.org, Shuah Khan , Mark Brown Subject: [PATCH v3 1/5] arm64: vdso: Provide a define when building the vDSO Date: Wed, 19 Aug 2020 13:13:14 +0100 Message-Id: <20200819121318.52158-2-broonie@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200819121318.52158-1-broonie@kernel.org> References: <20200819121318.52158-1-broonie@kernel.org> MIME-Version: 1.0 Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Provide a define identifying if code is being built for the vDSO to help with writing headers that are shared between the kernel and the vDSO. Signed-off-by: Mark Brown --- arch/arm64/kernel/vdso/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- 2.20.1 diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile index 45d5cfe46429..88cf0f0b91ed 100644 --- a/arch/arm64/kernel/vdso/Makefile +++ b/arch/arm64/kernel/vdso/Makefile @@ -28,7 +28,7 @@ ldflags-y := -shared -nostdlib -soname=linux-vdso.so.1 --hash-style=sysv \ $(btildflags-y) -T ccflags-y := -fno-common -fno-builtin -fno-stack-protector -ffixed-x18 -ccflags-y += -DDISABLE_BRANCH_PROFILING +ccflags-y += -DDISABLE_BRANCH_PROFILING -D__VDSO__ CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) $(GCC_PLUGINS_CFLAGS) KBUILD_CFLAGS += $(DISABLE_LTO) From patchwork Wed Aug 19 12:13:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 247956 Delivered-To: patch@linaro.org Received: by 2002:a05:6e02:522:0:0:0:0 with SMTP id h2csp404547ils; Wed, 19 Aug 2020 05:14:08 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwFFgbVuCAPeJ63gzjLf6/OLdlcSJfwavuf5X7yCSyaEMx+go31/D91QWmj1v3WAnkM7j0K X-Received: by 2002:aa7:ca0c:: with SMTP id y12mr23691852eds.251.1597839248012; Wed, 19 Aug 2020 05:14:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1597839248; cv=none; d=google.com; s=arc-20160816; b=hE1Sg+zeQzvFRuzfgur199jK/aYsLDVUXyk8ydTzcZH9LRzlIXPY8rMvDNOhwV4Q+c ZYV7P+odX4PKANhS0WMmIcprq/3ocIDIXu1UZXmNSwd1pSMWA+Xi9ulJjjt123jzVhZJ eULMZvvePYKB8CwnW9yzla0+y5Hq5VZRb87/L8xNC5unUW6sRJUK5CEENIOav0KFlkQz 6V5JX3QQDx5sHC7o/OOelsKskQKpPd+fFy+botwLrt8Pql8jyGMKSbRmyoA7D92HQngE qibrUbdtWjfP8sfHHvghHPv/6Kd4cS+61F8HPLvcJcy/PlFmYVv8ctaQlho6x/3oL3RN 35ZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ocaKXXfLPkrUzWULqpYZ2ZtAKtlNacg1/KdGynFCJgc=; b=yiU7oKpumsvdI43Bbgj9gPmJ4n7iyJr3bb/YaCHenTxqjtqmEBaxxDGi3W4bQ1ieN+ ye4bJvCCBvVtO8/eQU8U5hJhEdYIJq0K51nypf+KSMDgBaS33BFyzfi7lZE5983CXVIs AtsmnG+C2eVkrg9zzddokkY4rJp0YF4dHN9lugruz2VDBod9HEgmSU3QkoeHyqOhixIC /2fRsXnj+7TgOtqVb1FRsb4plqzYqNQmGhmisA4TqhM7U/4W2GQWzHRLiAStRbaZ9Qok aM+6kkeuPX3kxn0qxizmTpgT/TKnaFL8E9MPudZdxkPCZrJEqY5YZQ/ZMgAgjxAJyo5a AjjQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=uIiwEtfV; spf=pass (google.com: domain of linux-kselftest-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kselftest-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id b6si16772273eja.559.2020.08.19.05.14.07; Wed, 19 Aug 2020 05:14:08 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kselftest-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=uIiwEtfV; spf=pass (google.com: domain of linux-kselftest-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kselftest-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727020AbgHSMOH (ORCPT + 6 others); Wed, 19 Aug 2020 08:14:07 -0400 Received: from mail.kernel.org ([198.145.29.99]:51344 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726752AbgHSMOE (ORCPT ); Wed, 19 Aug 2020 08:14:04 -0400 Received: from localhost (fw-tnat.cambridge.arm.com [217.140.96.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E2BF0207BB; Wed, 19 Aug 2020 12:14:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1597839243; bh=HR11PdYBvh2kcllXkA7X72cUQrugXv/fyUKLaDrD1NM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uIiwEtfVwOI1891Y3mg15Z62mKkn2wU8U4oDMyYAsM+LxmedSKVGWbsnbN6t7FWi1 TyXxeu0E7sJrl9kF6zegPd7t5mQ3vVwjHNihK7hvU2yIVNu+uJa1EXVFioU+T2FLdX 6Ak2O8wTP4fcOkBKhu2Sd6vBL1YsRWHtqUYRMwEY= From: Mark Brown To: Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, Vincenzo Frascino , linux-kselftest@vger.kernel.org, Shuah Khan , Mark Brown Subject: [PATCH v3 2/5] arm64: vdso: Add per-CPU data Date: Wed, 19 Aug 2020 13:13:15 +0100 Message-Id: <20200819121318.52158-3-broonie@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200819121318.52158-1-broonie@kernel.org> References: <20200819121318.52158-1-broonie@kernel.org> MIME-Version: 1.0 Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org In order to support a vDSO getcpu() implementation add per-CPU data to the vDSO data page. Do this by wrapping the generic vdso_data struct in an arm64 specific one with an array of per-CPU data. The offset of the per-CPU data applying to a CPU will be stored in TPIDRRO_EL0, this allows us to get to the per-CPU data without doing any multiplications. Since we currently only map a single data page for the vDSO but support very large numbers of CPUs TPIDRRO may be set to zero for CPUs which don't fit in the data page. This will also happen when KPTI is active since kernel_ventry uses TPIDRRO_EL0 as a scratch register in that case, add a comment to the code explaining this. Acessors for the data are provided in the header since they will be needed in multiple files and it seems neater to keep things together. Signed-off-by: Mark Brown --- arch/arm64/include/asm/processor.h | 12 +----- arch/arm64/include/asm/vdso/datapage.h | 54 ++++++++++++++++++++++++++ arch/arm64/kernel/process.c | 26 ++++++++++++- arch/arm64/kernel/vdso.c | 5 ++- 4 files changed, 83 insertions(+), 14 deletions(-) create mode 100644 arch/arm64/include/asm/vdso/datapage.h -- 2.20.1 diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 240fe5e5b720..db7a804030b3 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -207,17 +207,7 @@ static inline void set_compat_ssbs_bit(struct pt_regs *regs) regs->pstate |= PSR_AA32_SSBS_BIT; } -static inline void start_thread(struct pt_regs *regs, unsigned long pc, - unsigned long sp) -{ - start_thread_common(regs, pc); - regs->pstate = PSR_MODE_EL0t; - - if (arm64_get_ssbd_state() != ARM64_SSBD_FORCE_ENABLE) - set_ssbs_bit(regs); - - regs->sp = sp; -} +void start_thread(struct pt_regs *regs, unsigned long pc, unsigned long sp); static inline bool is_ttbr0_addr(unsigned long addr) { diff --git a/arch/arm64/include/asm/vdso/datapage.h b/arch/arm64/include/asm/vdso/datapage.h new file mode 100644 index 000000000000..e88d97238c52 --- /dev/null +++ b/arch/arm64/include/asm/vdso/datapage.h @@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2020 ARM Limited + */ +#ifndef __ASM_VDSO_DATAPAGE_H +#define __ASM_VDSO_DATAPAGE_H + +#include + +struct vdso_cpu_data { + unsigned int cpu; + unsigned int node; +}; + +struct arm64_vdso_data { + /* Must be first in struct, we cast to vdso_data */ + struct vdso_data data[CS_BASES]; + struct vdso_cpu_data cpu_data[]; +}; + +#ifdef __VDSO__ +static inline struct vdso_cpu_data *__vdso_cpu_data(void) +{ + unsigned long offset; + + asm volatile( + "mrs %0, tpidrro_el0\n" + : "=r" (offset) + : + : "cc"); + + if (offset) + return (void *)(_vdso_data) + offset; + + return NULL; +} +#else +static inline size_t vdso_cpu_offset(void) +{ + size_t offset, data_end; + + offset = offsetof(struct arm64_vdso_data, cpu_data) + + smp_processor_id() * sizeof(struct vdso_cpu_data); + data_end = offset + sizeof(struct vdso_cpu_data) + 1; + + /* We only map a single page for vDSO data currently */ + if (data_end > PAGE_SIZE) + return 0; + + return offset; +} +#endif + +#endif diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 84ec630b8ab5..89b400f9397d 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -55,6 +55,7 @@ #include #include #include +#include #if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_STACKPROTECTOR_PER_TASK) #include @@ -309,6 +310,28 @@ void show_regs(struct pt_regs * regs) dump_backtrace(regs, NULL, KERN_DEFAULT); } +void start_thread(struct pt_regs *regs, unsigned long pc, unsigned long sp) +{ + start_thread_common(regs, pc); + regs->pstate = PSR_MODE_EL0t; + + if (arm64_get_ssbd_state() != ARM64_SSBD_FORCE_ENABLE) + set_ssbs_bit(regs); + + regs->sp = sp; + + /* + * Store the vDSO per-CPU offset if supported. Disable + * preemption to make sure we read the CPU offset on the CPU + * we write it on. + */ + if (!arm64_kernel_unmapped_at_el0()) { + preempt_disable(); + write_sysreg(vdso_cpu_offset(), tpidrro_el0); + preempt_enable(); + } +} + static void tls_thread_flush(void) { write_sysreg(0, tpidr_el0); @@ -452,7 +475,8 @@ static void tls_thread_switch(struct task_struct *next) if (is_compat_thread(task_thread_info(next))) write_sysreg(next->thread.uw.tp_value, tpidrro_el0); else if (!arm64_kernel_unmapped_at_el0()) - write_sysreg(0, tpidrro_el0); + /* Used as scratch in KPTI trampoline so don't set here. */ + write_sysreg(vdso_cpu_offset(), tpidrro_el0); write_sysreg(*task_user_tls(next), tpidr_el0); } diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c index d4202a32abc9..2a8d7ab76bee 100644 --- a/arch/arm64/kernel/vdso.c +++ b/arch/arm64/kernel/vdso.c @@ -28,6 +28,7 @@ #include #include #include +#include extern char vdso_start[], vdso_end[]; #ifdef CONFIG_COMPAT_VDSO @@ -77,10 +78,10 @@ static struct vdso_abi_info vdso_info[] __ro_after_init = { * The vDSO data page. */ static union { - struct vdso_data data[CS_BASES]; + struct arm64_vdso_data data; u8 page[PAGE_SIZE]; } vdso_data_store __page_aligned_data; -struct vdso_data *vdso_data = vdso_data_store.data; +struct vdso_data *vdso_data = vdso_data_store.data.data; static int __vdso_remap(enum vdso_abi abi, const struct vm_special_mapping *sm, From patchwork Wed Aug 19 12:13:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 247958 Delivered-To: patch@linaro.org Received: by 2002:a05:6e02:522:0:0:0:0 with SMTP id h2csp404576ils; Wed, 19 Aug 2020 05:14:10 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxs0dt5lOpFJe8n+xOZfzBOFlA51BdLs3y8TAVerJPch44/kYcg2YmN1//3140cF8q3aD90 X-Received: by 2002:a17:907:100f:: with SMTP id ox15mr4159322ejb.125.1597839250629; Wed, 19 Aug 2020 05:14:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1597839250; cv=none; d=google.com; s=arc-20160816; b=Zxhtt57uttzqv6glb7UumbjGGT06kYiKqXY6Sh4htTI9UeK9ryiw3udIcXa6hwpCpA Nq1XZFeAQQBKIWN9BO/TqGSgrVwnFVY+HrGmLLevhka88CCNXaFChtutyU0KgIhHqBnI wjHz5EVDJOYd8btdnrdatXgABGtb9+GEUfGWUyG9LCMrEuajw4uvGXDin6+mmHqtJy8L EkJuWHVpUv4DL9h8DdC188YzxVzZQ2B9c2K9SBFq84+BQcf4WS+lyKXfEEr7/icKJlH0 d1g8eu6MdkwFdQebHl8QVhQAP8u9ObLTg+fvrTSxomIOnikhMsgyzceYYB8+oh8m87+4 lq/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=X8lkUvX5IkIr5hd2D2n/jqShANSuHR5K3yAONjn5VZk=; b=mkWxHTpxGGkMqc7sWzDlkd4fCQBGX7ysVwp9QULgKsFyJE0NOz2icBAme0xO+NlAL/ 3ZHdjjTit4OpIM63Y6s9wEYiTpspT6qgvQbwDUCN02PBPYEQuszjgLf9YOgCkQXyoinM pSOD5wL6lXauK6/J8VXsf0Mc4rwLHzJzCDIPPFcptt88zQHqp0Nc7pCdomX2wt52RHG7 QzEgey3VQ8ChzGW7YyxMD7rCrLLEE2CgrRtgjRDj6+6T7gPkCngpwe7mQ0cuSbTtZKqM dF4pNH0AsSzDWjYINEEe32b5n7XnWV3XPRhddIYvgGgKmy8eIc3M7vApWOBtpWzh5sVs XZfw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=hGdWwUzz; spf=pass (google.com: domain of linux-kselftest-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kselftest-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id b6si16772273eja.559.2020.08.19.05.14.10; Wed, 19 Aug 2020 05:14:10 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kselftest-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=hGdWwUzz; spf=pass (google.com: domain of linux-kselftest-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kselftest-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726798AbgHSMOJ (ORCPT + 6 others); Wed, 19 Aug 2020 08:14:09 -0400 Received: from mail.kernel.org ([198.145.29.99]:51382 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727018AbgHSMOH (ORCPT ); Wed, 19 Aug 2020 08:14:07 -0400 Received: from localhost (fw-tnat.cambridge.arm.com [217.140.96.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 874C6206DA; Wed, 19 Aug 2020 12:14:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1597839246; bh=Ftm+BgFLcJzi0bjTiz8IjEJjLINWXFqX3Rrl97pTpb8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hGdWwUzzLPnSon0iOKaq9SG/crBOPwsDLkl71Tmfh0jMLTfAf5jQ4WHrDimzHvqzI OLN3vtTxqe9E7ih91kchWM5gj1ra76v0Nyi+KCdk8cnu3GXosZ+xi8pK9yc/Yw0CjH GdTJGU0vhRSRF45VGT10z4h5NCYSpUPUYb7Hs2sA= From: Mark Brown To: Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, Vincenzo Frascino , linux-kselftest@vger.kernel.org, Shuah Khan , Mark Brown Subject: [PATCH v3 3/5] arm64: vdso: Initialise the per-CPU vDSO data Date: Wed, 19 Aug 2020 13:13:16 +0100 Message-Id: <20200819121318.52158-4-broonie@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200819121318.52158-1-broonie@kernel.org> References: <20200819121318.52158-1-broonie@kernel.org> MIME-Version: 1.0 Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Register with the CPU hotplug system to initialise the per-CPU data for getcpu(). Signed-off-by: Mark Brown --- arch/arm64/kernel/vdso.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) -- 2.20.1 diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c index 2a8d7ab76bee..d9743c659341 100644 --- a/arch/arm64/kernel/vdso.c +++ b/arch/arm64/kernel/vdso.c @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -18,6 +19,7 @@ #include #include #include +#include #include #include #include @@ -466,6 +468,26 @@ int aarch32_setup_additional_pages(struct linux_binprm *bprm, int uses_interp) } #endif /* CONFIG_COMPAT */ +static void vdso_cpu_init(void *p) +{ + struct arm64_vdso_data *data = (struct arm64_vdso_data *)vdso_data; + unsigned int cpu; + + if (vdso_cpu_offset()) { + cpu = smp_processor_id(); + + data->cpu_data[cpu].cpu = cpu; + data->cpu_data[cpu].node = cpu_to_node(cpu); + } +} + +static int vdso_cpu_online(unsigned int cpu) +{ + smp_call_function_single(cpu, vdso_cpu_init, NULL, 1); + + return 0; +} + static int vdso_mremap(const struct vm_special_mapping *sm, struct vm_area_struct *new_vma) { @@ -494,6 +516,12 @@ static int __init vdso_init(void) vdso_info[VDSO_ABI_AA64].dm = &aarch64_vdso_maps[AA64_MAP_VVAR]; vdso_info[VDSO_ABI_AA64].cm = &aarch64_vdso_maps[AA64_MAP_VDSO]; + /* + * Initialize per-CPU data, callback runs for all current and + * future CPUs. + */ + cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "vdso", vdso_cpu_online, NULL); + return __vdso_init(VDSO_ABI_AA64); } arch_initcall(vdso_init); From patchwork Wed Aug 19 12:13:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 247959 Delivered-To: patch@linaro.org Received: by 2002:a05:6e02:522:0:0:0:0 with SMTP id h2csp404601ils; Wed, 19 Aug 2020 05:14:12 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwe07xzttOPnxS177CvBEwCxmzWT+KARIjTPd0OcjGtkiAElT7j7qbNGoV2+TX3pHqcpGP6 X-Received: by 2002:a50:c38b:: with SMTP id h11mr24547657edf.308.1597839252793; Wed, 19 Aug 2020 05:14:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1597839252; cv=none; d=google.com; s=arc-20160816; b=QZYiNH2zmWuo5Eti/RpmvAPnu81kuGvzEYQ+6fI2A1Pd6N2Fogzb2jy3Z2QL62Tn3o +TxAcMwBK+zzI2D9bpiQGqzVO5fuHMvBRWGxXIyGPKF+sFYhNrBV0y1SfpppjhJg31YU e8DKMyZJUdpXIM0I+LnfFxtcXh2V197LysR2L/lwmyL84UMVHPSZTmIK+kZ1q9rTOTNW HlM82H8X7AYX4V+mae8QCZpQg/TD0D9pkLUwVCbynb3uiYtpKLG4l83xq0kH8QYAT3yO 0SxMj8RD6IgQ0JQMi2/hH457iHCeCZog+02OvGd7P9Lt5LuBK/rddcl5KJ/LSB7ScJKl KNrQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JRTD0E1USqv27tKpmxT7DLVTgpP9KsLIGpqLmzJk6dc=; b=Bl7K35nZYuN0IiOz3kcDmDpADG/4G1fghoffAaUbQqHW1DygCu8vPjAhhOC1/CAl+B gkoERk0erHF4i5ktloBERl5DGzZusz992XLcf563D6/RB6EWwAgV+xjNj6kYQ3R1mAai 51Wz0AFMerlPztRsRzjF/oS0BVEQoAIuNqawoGOqgjz0Rx/fW8mqLxRcPe36ayBjEt0o mpbXDmGSstmXXfjy+/XnFZVvUXaM5JMdvi7Nlsp1AnTafxUa8n3XsUdgFQSNapz6ZSYQ Haqu0fjmf5mJ42Ds8n9fC5sHtPAdhZhJbi7L42L7/ARNUbbXWC2m8Wm4XMFdy1UPJsDN mQog== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=eTeToc67; spf=pass (google.com: domain of linux-kselftest-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kselftest-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id b6si16772273eja.559.2020.08.19.05.14.12; Wed, 19 Aug 2020 05:14:12 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kselftest-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=eTeToc67; spf=pass (google.com: domain of linux-kselftest-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kselftest-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726952AbgHSMOK (ORCPT + 6 others); Wed, 19 Aug 2020 08:14:10 -0400 Received: from mail.kernel.org ([198.145.29.99]:51404 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726752AbgHSMOJ (ORCPT ); Wed, 19 Aug 2020 08:14:09 -0400 Received: from localhost (fw-tnat.cambridge.arm.com [217.140.96.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 1B60720738; Wed, 19 Aug 2020 12:14:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1597839248; bh=KV4mnGXGCrFRun+EDN9OxgHUnFqbU7ZkJMYyr+CEL94=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eTeToc67nVsh1gRKfuZxYg5Scyvc3Cr2zYdt+z2Q+WGLVMhzFeHmNU1s/qZSPTazs z0aAmBdhWsef0uqrpoauI/XkedNz8KOetq9wjnE7nR9P3ywxWUXJpp6t3Q8rtQ4RDW K/nrRyxQcW6z/jwcN8oXmds0QwkErAfmeo9c4gTI= From: Mark Brown To: Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, Vincenzo Frascino , linux-kselftest@vger.kernel.org, Shuah Khan , Mark Brown Subject: [PATCH v3 4/5] arm64: vdso: Add getcpu() implementation Date: Wed, 19 Aug 2020 13:13:17 +0100 Message-Id: <20200819121318.52158-5-broonie@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200819121318.52158-1-broonie@kernel.org> References: <20200819121318.52158-1-broonie@kernel.org> MIME-Version: 1.0 Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Some applications, especially trace ones, benefit from avoiding the syscall overhead on getcpu() calls so provide a vDSO implementation of it. Signed-off-by: Mark Brown --- arch/arm64/kernel/vdso/Makefile | 2 +- arch/arm64/kernel/vdso/vdso.lds.S | 1 + arch/arm64/kernel/vdso/vgetcpu.c | 48 +++++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kernel/vdso/vgetcpu.c -- 2.20.1 diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile index 88cf0f0b91ed..ff350e69b8b6 100644 --- a/arch/arm64/kernel/vdso/Makefile +++ b/arch/arm64/kernel/vdso/Makefile @@ -11,7 +11,7 @@ ARCH_REL_TYPE_ABS := R_AARCH64_JUMP_SLOT|R_AARCH64_GLOB_DAT|R_AARCH64_ABS64 include $(srctree)/lib/vdso/Makefile -obj-vdso := vgettimeofday.o note.o sigreturn.o +obj-vdso := vgettimeofday.o note.o sigreturn.o vgetcpu.o # Build rules targets := $(obj-vdso) vdso.so vdso.so.dbg diff --git a/arch/arm64/kernel/vdso/vdso.lds.S b/arch/arm64/kernel/vdso/vdso.lds.S index d808ad31e01f..ef3fb80e0349 100644 --- a/arch/arm64/kernel/vdso/vdso.lds.S +++ b/arch/arm64/kernel/vdso/vdso.lds.S @@ -80,6 +80,7 @@ VERSION __kernel_gettimeofday; __kernel_clock_gettime; __kernel_clock_getres; + __kernel_getcpu; local: *; }; } diff --git a/arch/arm64/kernel/vdso/vgetcpu.c b/arch/arm64/kernel/vdso/vgetcpu.c new file mode 100644 index 000000000000..e8972e561e08 --- /dev/null +++ b/arch/arm64/kernel/vdso/vgetcpu.c @@ -0,0 +1,48 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * ARM64 userspace implementations of getcpu() + * + * Copyright (C) 2020 ARM Limited + * + */ + +#include +#include + +struct getcpucache; + +static __always_inline +int getcpu_fallback(unsigned int *_cpu, unsigned int *_node, + struct getcpucache *_c) +{ + register unsigned int *cpu asm("x0") = _cpu; + register unsigned int *node asm("x1") = _node; + register struct getcpucache *c asm("x2") = _c; + register long ret asm ("x0"); + register long nr asm("x8") = __NR_getcpu; + + asm volatile( + " svc #0\n" + : "=r" (ret) + : "r" (cpu), "r" (node), "r" (c), "r" (nr) + : "memory"); + + return ret; +} + +int __kernel_getcpu(unsigned int *cpu, unsigned int *node, + struct getcpucache *c) +{ + struct vdso_cpu_data *cpu_data = __vdso_cpu_data(); + + if (cpu_data) { + if (cpu) + *cpu = cpu_data->cpu; + if (node) + *node = cpu_data->node; + + return 0; + } + + return getcpu_fallback(cpu, node, c); +} From patchwork Wed Aug 19 12:13:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 247960 Delivered-To: patch@linaro.org Received: by 2002:a05:6e02:522:0:0:0:0 with SMTP id h2csp404609ils; Wed, 19 Aug 2020 05:14:13 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwuUAvBmxA0Ezxr+1rUVIBvZqG3Z+YFRuorEvFMfRINFkC8NcFQUbVtukQ4R/IKcupDqvjP X-Received: by 2002:a17:906:78e:: with SMTP id l14mr26443798ejc.67.1597839253566; Wed, 19 Aug 2020 05:14:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1597839253; cv=none; d=google.com; s=arc-20160816; b=az4LyeYUJR3lJnb12FhKfwitCt27DDK31p5x/7pY0Udzvinl7zSZ3VWyYX+vhYOMDW dpuZ17sr7KpyONkVZlS/YHStCNVvPvES5HCInS3ikBkVPptWVNme1db40etI/8TOht+b OAiUJ6HGbuNtMLyP6yaNGIwjLf9Y3+3wX8Vz4lDyImESYFwoKcTM2XcFhSXuGoS7AJSB PY1jWsXDOnDWkzhC3iB4H5cI1iIqxrjLj482ESIL+PZuwMOVbO36BvSX6EK826HT4Rwv 4xZs6pqg7toyYDAIkBbhhtl6djaODzqUzL6VeBU82WveYyN2JjYyKvspFc6SkcIzP335 gcXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=rzsL2X8+dIxcbWdscjZhV1ge9hK/Hg8ooAlJsiD1yD0=; b=c7Bd7d53SClep8W0JbGJEKzDAJCPTSEKv+kxbuUe1fUOZvQKVceKy1V5gG7OHiv7+Z f86JTjBNLxQHvUzEE/82Gh/NPNS2XIqwUM/W8bVmUXUjlW+JM2KS2t6caZX2t7Tv6KKg KQjaFC16zDh5n4Qkr7Fz58zAxiF1iBp5Wel3Fwj05siyNRmnAIDsiA1CEmlaYxLDvvLA oFPY2u5ikYXl9nB5edi5kDNbiZzfqQHjJYeJMymx9Vynd5k/mFZh242KqHKOuCMRXvyW /Ogl6o2OTrT37BgT4OghW5DAdYUfNGQmSHIjt7euIKyGQD0jhgGDR0opJ56iwGkZcMZu EJOQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=tTpFukSQ; spf=pass (google.com: domain of linux-kselftest-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kselftest-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id b6si16772273eja.559.2020.08.19.05.14.13; Wed, 19 Aug 2020 05:14:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kselftest-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=tTpFukSQ; spf=pass (google.com: domain of linux-kselftest-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kselftest-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726752AbgHSMON (ORCPT + 6 others); Wed, 19 Aug 2020 08:14:13 -0400 Received: from mail.kernel.org ([198.145.29.99]:51448 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727018AbgHSMOL (ORCPT ); Wed, 19 Aug 2020 08:14:11 -0400 Received: from localhost (fw-tnat.cambridge.arm.com [217.140.96.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AF5AE206DA; Wed, 19 Aug 2020 12:14:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1597839251; bh=Qm1D4s+M/mO7mwIKm4XuWbPwM6fNbujUhQSw8qCmPVo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tTpFukSQmJyyrAXX0yhMBRTggEOGVM7aeRXckhVLbgCQi3LgF58ukbdELkTN/v+/s UVIuUi9W3MAQZNMAuX/8iUqNsw1gyXwY0oqY3sCBfFFU5axWh8MshuvTA5yPuckMPp n3yXaa3UY9WcZFA8Szu0ifRmeBHC8Fmv0cRQU/Mg= From: Mark Brown To: Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, Vincenzo Frascino , linux-kselftest@vger.kernel.org, Shuah Khan , Mark Brown Subject: [PATCH v3 5/5] selftests: vdso: Support arm64 in getcpu() test Date: Wed, 19 Aug 2020 13:13:18 +0100 Message-Id: <20200819121318.52158-6-broonie@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200819121318.52158-1-broonie@kernel.org> References: <20200819121318.52158-1-broonie@kernel.org> MIME-Version: 1.0 Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org arm64 exports the vDSO ABI with a version of LINUX_2.6.39 and symbols prefixed with __kernel rather than __vdso. Update the getcpu() test to handle this. Signed-off-by: Mark Brown --- tools/testing/selftests/vDSO/vdso_test_getcpu.c | 10 ++++++++++ 1 file changed, 10 insertions(+) -- 2.20.1 diff --git a/tools/testing/selftests/vDSO/vdso_test_getcpu.c b/tools/testing/selftests/vDSO/vdso_test_getcpu.c index fc25ede131b8..4aeb65012b81 100644 --- a/tools/testing/selftests/vDSO/vdso_test_getcpu.c +++ b/tools/testing/selftests/vDSO/vdso_test_getcpu.c @@ -14,8 +14,18 @@ #include "../kselftest.h" #include "parse_vdso.h" +/* + * ARM64's vDSO exports its getcpu() implementation with a different + * name and version from other architectures, so we need to handle it + * as a special case. + */ +#if defined(__aarch64__) +const char *version = "LINUX_2.6.39"; +const char *name = "__kernel_getcpu"; +#else const char *version = "LINUX_2.6"; const char *name = "__vdso_getcpu"; +#endif struct getcpu_cache; typedef long (*getcpu_t)(unsigned int *, unsigned int *,