From patchwork Tue Apr 23 13:57:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Grall X-Patchwork-Id: 162701 Delivered-To: patch@linaro.org Received: by 2002:a02:c6d8:0:0:0:0:0 with SMTP id r24csp3819061jan; Tue, 23 Apr 2019 06:57:32 -0700 (PDT) X-Google-Smtp-Source: APXvYqwaeK4w6CLmrh2pb8Z6MMuUu+XtdMLRJSJJ+pz2RhnZIYkroAyZQB9PftfRMXBDa95RCOkM X-Received: by 2002:a63:5041:: with SMTP id q1mr1684569pgl.386.1556027852875; Tue, 23 Apr 2019 06:57:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556027852; cv=none; d=google.com; s=arc-20160816; b=vUiLnhH41zTJYmtJtolZ/MHyRZeQ9ZZoYMny0iB6bglXfLenfoUUK6JjAuAgUWsBBk oMOOCXN9V9fF7JsIijB/LBdvWK6kJAFjsQi7adI7ywghhpajOXLx4I9Bc0yIGGvj1NVa bxz/H1ENGL/FT6m5y8iwAPFYFCgY7PbO4xWEtmTsLSJ30QA2C+p4hYtsB00Pakdhbpij 6mh8qDrTrQNs2uYkj98DpkcRbx+M19gMTrx47zgfn6BRb4KE0kLfHBSS3ks3f/VSdDRT 3yWGHpsjkZPJTjIZShOHmPHvuak2hWpdwRLNkyrYe47e7PG4yAfM5/bQWsgm9i5BZo8a 4/JQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=r0VU6D15gzv6yB+V7Wxfb7jOx8N7gll1toqHo/B76/I=; b=F/fBPdbDI+6HXluETdSwcRjEv87XXk3CYFOBBqcQT1hENOofjQV3FKeqyRn3cZDoY/ FbtKFhN6TKXVW1A5eYa0xsJtAwSPSNkaPFSrgQ6qs51fEbyFxf++TYkDoUlPmRYkAkPY qkERYVX6idGvsHxWGXfg45O2Q6CslCnoroMqDFQlZwpc47/Sje+8Dys2BkO6sG0yhNPX 6Tx7K5i4zl/X4gnT8BDloB86PotcN0sX+0nrVBoaukjXcpgQDCZmOW26S7EetZaS2rn+ re3t4zLJ9pYRDnTofSNNZ6uog7FnluUe8QGIPoV1iX+gAhg1JUVJs2Mx6pLl0mqQRGYo Q/lA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e24si14459509pgv.555.2019.04.23.06.57.32; Tue, 23 Apr 2019 06:57:32 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728066AbfDWN5b (ORCPT + 30 others); Tue, 23 Apr 2019 09:57:31 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:56924 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726659AbfDWN53 (ORCPT ); Tue, 23 Apr 2019 09:57:29 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 172F7EBD; Tue, 23 Apr 2019 06:57:29 -0700 (PDT) Received: from e108454-lin.cambridge.arm.com (e108454-lin.cambridge.arm.com [10.1.196.50]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E248E3F238; Tue, 23 Apr 2019 06:57:26 -0700 (PDT) From: Julien Grall To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, will.deacon@arm.com, christoffer.dall@arm.com, marc.zyngier@arm.com, james.morse@arm.com, julien.thierry@arm.com, suzuki.poulose@arm.com, Dave.Martin@arm.com, ard.biesheuvel@linaro.org, Julien Grall Subject: [PATCH v3 1/3] arm64/fpsimd: Remove the prototype for sve_flush_cpu_state() Date: Tue, 23 Apr 2019 14:57:17 +0100 Message-Id: <20190423135719.11306-2-julien.grall@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20190423135719.11306-1-julien.grall@arm.com> References: <20190423135719.11306-1-julien.grall@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The function sve_flush_cpu_state() has been removed in commit 21cdd7fd76e3 ("KVM: arm64: Remove eager host SVE state saving"). So remove the associated prototype in asm/fpsimd.h. Signed-off-by: Julien Grall Reviewed-by: Dave Martin --- Changes in v3: - Add Dave's reviewed-by - Fix checkpatch style error when mentioning a commit Changes in v2: - Patch added --- arch/arm64/include/asm/fpsimd.h | 1 - 1 file changed, 1 deletion(-) -- 2.11.0 diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index df62bbd33a9a..b73d12fcc7f9 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -64,7 +64,6 @@ extern void fpsimd_bind_state_to_cpu(struct user_fpsimd_state *state, extern void fpsimd_flush_task_state(struct task_struct *target); extern void fpsimd_flush_cpu_state(void); -extern void sve_flush_cpu_state(void); /* Maximum VL that SVE VL-agnostic software can transparently support */ #define SVE_VL_ARCH_MAX 0x100 From patchwork Tue Apr 23 13:57:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Grall X-Patchwork-Id: 162702 Delivered-To: patch@linaro.org Received: by 2002:a02:c6d8:0:0:0:0:0 with SMTP id r24csp3819114jan; Tue, 23 Apr 2019 06:57:36 -0700 (PDT) X-Google-Smtp-Source: APXvYqyzAVbVSTBJ7iVuu9VXs5fkvbDvI5DEoXnklIbvOwWRzwdsMjlsOQq0qYYiiNbsreCVv14b X-Received: by 2002:a62:8381:: with SMTP id h123mr26222423pfe.226.1556027856285; Tue, 23 Apr 2019 06:57:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556027856; cv=none; d=google.com; s=arc-20160816; b=aNL+Wy5LLnYa08F4WC85r9GC4kuWZ8wPMklvinETo92l4tOS2bYh86SU0BslhDS/WO y02kEo9A8jFIAGs6SYYFM8h7QCXgNC85yJfwjYFhWVNwNF4aAZn6a2kyY7/tP5ZNAkxX ALn4lJhWnYHiSpAUFzXp+EWa+FxmuujpFoHlRhJ34RwXIXmDMX2d2KLt3jBKL74WKe4l OQ+HX70nZKwIdRJNWCy9kqwPEMllHwIjtwk/Sf2Yes24EElDJaDfzwFxVDnPYxI7kZ2G gYUfdk6/sWgl9aB9D0HfEhLX1jnqx6P/XmWeCDPbiPOZQjUyCBM5wvO2V8n603IzBiag LQ9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=bay+0SHz9NdqjWCILEn37rfLuSdBe2p/IZJdFdjGC7k=; b=V1zy/fK11PXMH+NdsZT9vFbW/q9aXLMPGjT4JDyMTBrO/LXUTLe1JfQdr7zhSRsaZ7 bj5+7FofLv0A7kfrWfVyk7kHfEjc687OfFgfZmcvQy202+07V4D/ocBet1AoAQYvUcfp eeR81s1PU+Paq4HFXbvD/V0Xnm57UFhm6j+OnqFMuzrPmbnxpvqhSdPC4L+7cHJynMYc IWIzlQXrfyhJzQxJsaG5SDVD721zWhhRho7wWKW9B+ixYFUEovI51BWXlvv92bqsnQKJ pJW43slRXbu+Ax/De1/CBUxC3xEF9mLP18TYPFY8iZs2V8O7gPG6O6U1bpzf+sjEmHPf pNlA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e24si14459509pgv.555.2019.04.23.06.57.36; Tue, 23 Apr 2019 06:57:36 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728098AbfDWN5f (ORCPT + 30 others); Tue, 23 Apr 2019 09:57:35 -0400 Received: from foss.arm.com ([217.140.101.70]:56932 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726659AbfDWN5c (ORCPT ); Tue, 23 Apr 2019 09:57:32 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 81654A78; Tue, 23 Apr 2019 06:57:31 -0700 (PDT) Received: from e108454-lin.cambridge.arm.com (e108454-lin.cambridge.arm.com [10.1.196.50]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 584043F238; Tue, 23 Apr 2019 06:57:29 -0700 (PDT) From: Julien Grall To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, will.deacon@arm.com, christoffer.dall@arm.com, marc.zyngier@arm.com, james.morse@arm.com, julien.thierry@arm.com, suzuki.poulose@arm.com, Dave.Martin@arm.com, ard.biesheuvel@linaro.org, Julien Grall Subject: [PATCH v3 2/3] arch/arm64: fpsimd: Introduce fpsimd_save_and_flush_cpu_state() and use it Date: Tue, 23 Apr 2019 14:57:18 +0100 Message-Id: <20190423135719.11306-3-julien.grall@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20190423135719.11306-1-julien.grall@arm.com> References: <20190423135719.11306-1-julien.grall@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The only external user of fpsimd_save() and fpsimd_flush_cpu_state() is the KVM FPSIMD code. A following patch will introduce a mechanism to acquire owernship of the FPSIMD/SVE context for performing context management operations. Rather than having to export the new helpers to get/put the context, we can just introduce a new function to combine fpsimd_save() and fpsimd_flush_cpu_state(). This has also the advantage to remove any external call of fpsimd_save() and fpsimd_flush_cpu_state(), so they can be turned static. Lastly, the new function can also be used in the PM notifier. Signed-off-by: Julien Grall --- kernel_neon_begin() does not use fpsimd_save_and_flush_cpu_state() because the next patch will modify the function to also grab the FPSIMD/SVE context. Changes in v3: - Rework the commit message - Move the prototype of fpsimd_save_and_flush_cpu_state() further down in the header - Remove comment in kvm_arch_vcpu_put_fp() Changes in v2: - Patch added --- arch/arm64/include/asm/fpsimd.h | 5 ++--- arch/arm64/kernel/fpsimd.c | 17 +++++++++++++---- arch/arm64/kvm/fpsimd.c | 4 +--- 3 files changed, 16 insertions(+), 10 deletions(-) -- 2.11.0 Reviewed-by: Dave Martin diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index b73d12fcc7f9..c311d36ecffe 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -48,8 +48,6 @@ struct task_struct; extern void fpsimd_save_state(struct user_fpsimd_state *state); extern void fpsimd_load_state(struct user_fpsimd_state *state); -extern void fpsimd_save(void); - extern void fpsimd_thread_switch(struct task_struct *next); extern void fpsimd_flush_thread(void); @@ -63,7 +61,8 @@ extern void fpsimd_bind_state_to_cpu(struct user_fpsimd_state *state, void *sve_state, unsigned int sve_vl); extern void fpsimd_flush_task_state(struct task_struct *target); -extern void fpsimd_flush_cpu_state(void); + +extern void fpsimd_save_and_flush_cpu_state(void); /* Maximum VL that SVE VL-agnostic software can transparently support */ #define SVE_VL_ARCH_MAX 0x100 diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 56afa40263d9..5313aa257be6 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -246,7 +246,7 @@ static void task_fpsimd_load(void) * * Softirqs (and preemption) must be disabled. */ -void fpsimd_save(void) +static void fpsimd_save(void) { struct fpsimd_last_state_struct const *last = this_cpu_ptr(&fpsimd_last_state); @@ -1122,12 +1122,22 @@ void fpsimd_flush_task_state(struct task_struct *t) * Invalidate any task's FPSIMD state that is present on this cpu. * This function must be called with softirqs disabled. */ -void fpsimd_flush_cpu_state(void) +static void fpsimd_flush_cpu_state(void) { __this_cpu_write(fpsimd_last_state.st, NULL); set_thread_flag(TIF_FOREIGN_FPSTATE); } +/* + * Save the FPSIMD state to memory and invalidate cpu view. + * This function must be called with softirqs (and preemption) disabled. + */ +void fpsimd_save_and_flush_cpu_state(void) +{ + fpsimd_save(); + fpsimd_flush_cpu_state(); +} + #ifdef CONFIG_KERNEL_MODE_NEON DEFINE_PER_CPU(bool, kernel_neon_busy); @@ -1284,8 +1294,7 @@ static int fpsimd_cpu_pm_notifier(struct notifier_block *self, { switch (cmd) { case CPU_PM_ENTER: - fpsimd_save(); - fpsimd_flush_cpu_state(); + fpsimd_save_and_flush_cpu_state(); break; case CPU_PM_EXIT: break; diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 6e3c9c8b2df9..525010504f9d 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -112,9 +112,7 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) if (vcpu->arch.flags & KVM_ARM64_FP_ENABLED) { u64 *guest_zcr = &vcpu->arch.ctxt.sys_regs[ZCR_EL1]; - /* Clean guest FP state to memory and invalidate cpu view */ - fpsimd_save(); - fpsimd_flush_cpu_state(); + fpsimd_save_and_flush_cpu_state(); if (guest_has_sve) *guest_zcr = read_sysreg_s(SYS_ZCR_EL12); From patchwork Tue Apr 23 13:57:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Grall X-Patchwork-Id: 162703 Delivered-To: patch@linaro.org Received: by 2002:a02:c6d8:0:0:0:0:0 with SMTP id r24csp3819152jan; Tue, 23 Apr 2019 06:57:39 -0700 (PDT) X-Google-Smtp-Source: APXvYqxGiTlRQ/AtKgNF0gpEEnhTwWu8OT1j6CgaJ4uCQkzMaLA2+M3oAlgXHIgDDtA/O8xMS1mr X-Received: by 2002:a17:902:b589:: with SMTP id a9mr1570145pls.66.1556027859196; Tue, 23 Apr 2019 06:57:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556027859; cv=none; d=google.com; s=arc-20160816; b=F0y3PyjDFQoQEiYn9Fnd7sRz/6TKFVAt6s94YcBJWxL3W/kjubRVZgbauA4qk93C7X 7x+Fnbtx7RjqKGYlEjZLGDuWxViAxQsLgF0+Wb7QodXMv1SlXGfKi+TnioHCIeuF0Vnm L5XqC02VQ9rRxd5oetQpD+C99nZsUiWtXvOan1fJMnAjqeAaGaaiygHeIFB+TzSg3ZYO b8ZGBa/7Gdx6mpKdrqe2J0u97F3wlwyL73xXvxRGn6EeQoNP/3cljGLg0Cc7uXzeoh/X /2Bzxv8sIDFNJ6zU/TKQERHtBdYm5bs/G4RUwdgfCUUkbUFl57m78jqsMT+Sr3eHE/w5 5eQw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=+EnshdQvAd68XZO26MgZ6gD+BVF5wWiYXo3jhMdJLUM=; b=BptqQR0Gm9rN8LDCxBcdCuSrpW0eh2KTh0YwdmkWiJvEWzJ0FzfGqVF4htX0s1nmti yqpyuvQe38QI4i1BzZlgJdLM+0FgX0u1PjlfIJZwD+0lBB7IfbZ3ZSicJxrE1ZO5bwAy 7hT7u6DOwuhaRJtLxuOkc7hz13LW9mYcW+qDDacNUbdHfQ2FzViY8shK8l3McEQicLwX CuGd9w5nBIm069pRxa7b4uohLClZo+ovPE0z1XPnWtc8uNhfCH9+Rq5K958rDGubsXyf SsfySiX9lVvwgDLxol54SFJa3mngt0wwFYHw8kB2zIxFvFRk6EGipEMotBDdlSriAhxw CFWQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f34si15223460pgb.49.2019.04.23.06.57.38; Tue, 23 Apr 2019 06:57:39 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728115AbfDWN5i (ORCPT + 30 others); Tue, 23 Apr 2019 09:57:38 -0400 Received: from foss.arm.com ([217.140.101.70]:56946 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728075AbfDWN5f (ORCPT ); Tue, 23 Apr 2019 09:57:35 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1CECAEBD; Tue, 23 Apr 2019 06:57:34 -0700 (PDT) Received: from e108454-lin.cambridge.arm.com (e108454-lin.cambridge.arm.com [10.1.196.50]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C2B1F3F238; Tue, 23 Apr 2019 06:57:31 -0700 (PDT) From: Julien Grall To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, will.deacon@arm.com, christoffer.dall@arm.com, marc.zyngier@arm.com, james.morse@arm.com, julien.thierry@arm.com, suzuki.poulose@arm.com, Dave.Martin@arm.com, ard.biesheuvel@linaro.org, Julien Grall Subject: [PATCH v3 3/3] arm64/fpsimd: Don't disable softirq when touching FPSIMD/SVE state Date: Tue, 23 Apr 2019 14:57:19 +0100 Message-Id: <20190423135719.11306-4-julien.grall@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20190423135719.11306-1-julien.grall@arm.com> References: <20190423135719.11306-1-julien.grall@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When the kernel is compiled with CONFIG_KERNEL_MODE_NEON, some part of the kernel may be able to use FPSIMD/SVE. This is for instance the case for crypto code. Any use of FPSIMD/SVE in the kernel are clearly marked by using the function kernel_neon_{begin, end}. Furthermore, this can only be used when may_use_simd() returns true. The current implementation of may_use_simd() allows softirq to use FPSIMD/SVE unless it is currently in use (i.e kernel_neon_busy is true). When in use, softirqs usually fall back to a software method. At the moment, as a softirq may use FPSIMD/SVE, softirqs are disabled when touching the FPSIMD/SVE context. This has the drawback to disable all softirqs even if they are not using FPSIMD/SVE. Since a softirq is supposed to check may_use_simd() anyway before attempting to use FPSIMD/SVE, there is limited reason to keep softirq disabled when touching the FPSIMD/SVE context. Instead, we can simply disable preemption and mark the FPSIMD/SVE context as in use by setting CPU's kernel_neon_busy flag. Two new helpers {get, put}_cpu_fpsimd_context is introduced to mark the area using FPSIMD/SVE context and uses them in replacement of local_bh_{disable, enable}. The functions kernel_neon_{begin, end} are also re-implemented to use the new helpers. For use in fpsimd_thread_switch(), which is a critical path where preemption is already disabled, doule-underscored versions of the helpers are provided to avoid disabling preemption again. The change has been benchmarked on Linux 5.1-rc4 with defconfig. On Juno2: * hackbench 100 process 1000 (10 times) * .7% quicker On ThunderX 2: * hackbench 1000 process 1000 (20 times) * 3.4% quicker Signed-off-by: Julien Grall --- Changes in v3: - Fix typoes in the commit message - Rework a bit the commit message - Use imperative mood - Rename kernel_neon_busy to fpsimd_context_busy - Remove debug code - Update comments - Don't require preemption when calling fpsimd_save_and_flush_cpu_state() Changes in v2: - Remove spurious call to kernel_neon_enable in kernel_neon_begin. - Rename kernel_neon_{enable, disable} to {get, put}_cpu_fpsimd_context - Introduce a double-underscore version of the helpers for case where preemption is already disabled - Introduce have_cpu_fpsimd_context() and use it in WARN_ON(...) - Surround more places in the code with the new helpers - Rework the comments - Update the commit message with the benchmark result --- arch/arm64/include/asm/simd.h | 10 ++-- arch/arm64/kernel/fpsimd.c | 126 ++++++++++++++++++++++++++++-------------- 2 files changed, 88 insertions(+), 48 deletions(-) -- 2.11.0 diff --git a/arch/arm64/include/asm/simd.h b/arch/arm64/include/asm/simd.h index 6495cc51246f..a6307e43b8c2 100644 --- a/arch/arm64/include/asm/simd.h +++ b/arch/arm64/include/asm/simd.h @@ -15,9 +15,9 @@ #include #include -#ifdef CONFIG_KERNEL_MODE_NEON +DECLARE_PER_CPU(bool, fpsimd_context_busy); -DECLARE_PER_CPU(bool, kernel_neon_busy); +#ifdef CONFIG_KERNEL_MODE_NEON /* * may_use_simd - whether it is allowable at this time to issue SIMD @@ -29,15 +29,15 @@ DECLARE_PER_CPU(bool, kernel_neon_busy); static __must_check inline bool may_use_simd(void) { /* - * kernel_neon_busy is only set while preemption is disabled, + * fpsimd_context_busy is only set while preemption is disabled, * and is clear whenever preemption is enabled. Since - * this_cpu_read() is atomic w.r.t. preemption, kernel_neon_busy + * this_cpu_read() is atomic w.r.t. preemption, fpsimd_context_busy * cannot change under our feet -- if it's set we cannot be * migrated, and if it's clear we cannot be migrated to a CPU * where it is set. */ return !in_irq() && !irqs_disabled() && !in_nmi() && - !this_cpu_read(kernel_neon_busy); + !this_cpu_read(fpsimd_context_busy); } #else /* ! CONFIG_KERNEL_MODE_NEON */ diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 5313aa257be6..6168d06bbd20 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -92,7 +92,8 @@ * To prevent this from racing with the manipulation of the task's FPSIMD state * from task context and thereby corrupting the state, it is necessary to * protect any manipulation of a task's fpsimd_state or TIF_FOREIGN_FPSTATE - * flag with local_bh_disable() unless softirqs are already masked. + * flag with {, __}get_cpu_fpsimd_context(). This will still allow softirqs to + * run but prevent them to use FPSIMD. * * For a certain task, the sequence may look something like this: * - the task gets scheduled in; if both the task's fpsimd_cpu field @@ -155,6 +156,56 @@ extern void __percpu *efi_sve_state; #endif /* ! CONFIG_ARM64_SVE */ +DEFINE_PER_CPU(bool, fpsimd_context_busy); +EXPORT_PER_CPU_SYMBOL(fpsimd_context_busy); + +static void __get_cpu_fpsimd_context(void) +{ + bool busy = __this_cpu_xchg(fpsimd_context_busy, true); + + WARN_ON(busy); +} + +/* + * Claim ownership of the CPU FPSIMD context for use by the calling context. + * + * The caller may freely modify FPSIMD context until *put_cpu_fpsimd_context() + * is called. + * + * The double-underscore version must only be called if you know the task + * can't be preempted. + */ +static void get_cpu_fpsimd_context(void) +{ + preempt_disable(); + __get_cpu_fpsimd_context(); +} + +static void __put_cpu_fpsimd_context(void) +{ + bool busy = __this_cpu_xchg(fpsimd_context_busy, false); + + WARN_ON(!busy); /* No matching get_cpu_fpsimd_context()? */ +} + +/* + * Release the CPU FPSIMD context. + * + * Must be called from a context in which get_cpu_fpsimd_context() was + * previously called, with no call to put_cpu_fpsimd_context() in the + * meantime. + */ +static void put_cpu_fpsimd_context(void) +{ + __put_cpu_fpsimd_context(); + preempt_enable(); +} + +static bool have_cpu_fpsimd_context(void) +{ + return !preemptible() && __this_cpu_read(fpsimd_context_busy); +} + /* * Call __sve_free() directly only if you know task can't be scheduled * or preempted. @@ -225,12 +276,10 @@ static void sve_free(struct task_struct *task) * This function should be called only when the FPSIMD/SVE state in * thread_struct is known to be up to date, when preparing to enter * userspace. - * - * Softirqs (and preemption) must be disabled. */ static void task_fpsimd_load(void) { - WARN_ON(!in_softirq() && !irqs_disabled()); + WARN_ON(!have_cpu_fpsimd_context()); if (system_supports_sve() && test_thread_flag(TIF_SVE)) sve_load_state(sve_pffr(¤t->thread), @@ -243,8 +292,6 @@ static void task_fpsimd_load(void) /* * Ensure FPSIMD/SVE storage in memory for the loaded context is up to * date with respect to the CPU registers. - * - * Softirqs (and preemption) must be disabled. */ static void fpsimd_save(void) { @@ -252,7 +299,7 @@ static void fpsimd_save(void) this_cpu_ptr(&fpsimd_last_state); /* set by fpsimd_bind_task_to_cpu() or fpsimd_bind_state_to_cpu() */ - WARN_ON(!in_softirq() && !irqs_disabled()); + WARN_ON(!have_cpu_fpsimd_context()); if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) { if (system_supports_sve() && test_thread_flag(TIF_SVE)) { @@ -357,7 +404,8 @@ static int __init sve_sysctl_init(void) { return 0; } * task->thread.sve_state. * * Task can be a non-runnable task, or current. In the latter case, - * softirqs (and preemption) must be disabled. + * the caller must have ownership of the cpu FPSIMD context before calling + * this function. * task->thread.sve_state must point to at least sve_state_size(task) * bytes of allocated kernel memory. * task->thread.uw.fpsimd_state must be up to date before calling this @@ -384,7 +432,8 @@ static void fpsimd_to_sve(struct task_struct *task) * task->thread.uw.fpsimd_state. * * Task can be a non-runnable task, or current. In the latter case, - * softirqs (and preemption) must be disabled. + * the caller must have ownership of the cpu FPSIMD context before calling + * this function. * task->thread.sve_state must point to at least sve_state_size(task) * bytes of allocated kernel memory. * task->thread.sve_state must be up to date before calling this function. @@ -544,7 +593,7 @@ int sve_set_vector_length(struct task_struct *task, * non-SVE thread. */ if (task == current) { - local_bh_disable(); + get_cpu_fpsimd_context(); fpsimd_save(); } @@ -554,7 +603,7 @@ int sve_set_vector_length(struct task_struct *task, sve_to_fpsimd(task); if (task == current) - local_bh_enable(); + put_cpu_fpsimd_context(); /* * Force reallocation of task SVE state to the correct size @@ -867,7 +916,7 @@ asmlinkage void do_sve_acc(unsigned int esr, struct pt_regs *regs) sve_alloc(current); - local_bh_disable(); + get_cpu_fpsimd_context(); fpsimd_save(); @@ -878,7 +927,7 @@ asmlinkage void do_sve_acc(unsigned int esr, struct pt_regs *regs) if (test_and_set_thread_flag(TIF_SVE)) WARN_ON(1); /* SVE access shouldn't have trapped */ - local_bh_enable(); + put_cpu_fpsimd_context(); } /* @@ -922,6 +971,8 @@ void fpsimd_thread_switch(struct task_struct *next) if (!system_supports_fpsimd()) return; + __get_cpu_fpsimd_context(); + /* Save unsaved fpsimd state, if any: */ fpsimd_save(); @@ -936,6 +987,8 @@ void fpsimd_thread_switch(struct task_struct *next) update_tsk_thread_flag(next, TIF_FOREIGN_FPSTATE, wrong_task || wrong_cpu); + + __put_cpu_fpsimd_context(); } void fpsimd_flush_thread(void) @@ -945,7 +998,7 @@ void fpsimd_flush_thread(void) if (!system_supports_fpsimd()) return; - local_bh_disable(); + get_cpu_fpsimd_context(); fpsimd_flush_task_state(current); memset(¤t->thread.uw.fpsimd_state, 0, @@ -986,7 +1039,7 @@ void fpsimd_flush_thread(void) current->thread.sve_vl_onexec = 0; } - local_bh_enable(); + put_cpu_fpsimd_context(); } /* @@ -998,9 +1051,9 @@ void fpsimd_preserve_current_state(void) if (!system_supports_fpsimd()) return; - local_bh_disable(); + get_cpu_fpsimd_context(); fpsimd_save(); - local_bh_enable(); + put_cpu_fpsimd_context(); } /* @@ -1017,7 +1070,8 @@ void fpsimd_signal_preserve_current_state(void) /* * Associate current's FPSIMD context with this cpu - * Preemption must be disabled when calling this function. + * The caller must have ownership of the cpu FPSIMD context before calling + * this function. */ void fpsimd_bind_task_to_cpu(void) { @@ -1063,14 +1117,14 @@ void fpsimd_restore_current_state(void) if (!system_supports_fpsimd()) return; - local_bh_disable(); + get_cpu_fpsimd_context(); if (test_and_clear_thread_flag(TIF_FOREIGN_FPSTATE)) { task_fpsimd_load(); fpsimd_bind_task_to_cpu(); } - local_bh_enable(); + put_cpu_fpsimd_context(); } /* @@ -1083,7 +1137,7 @@ void fpsimd_update_current_state(struct user_fpsimd_state const *state) if (!system_supports_fpsimd()) return; - local_bh_disable(); + get_cpu_fpsimd_context(); current->thread.uw.fpsimd_state = *state; if (system_supports_sve() && test_thread_flag(TIF_SVE)) @@ -1094,7 +1148,7 @@ void fpsimd_update_current_state(struct user_fpsimd_state const *state) clear_thread_flag(TIF_FOREIGN_FPSTATE); - local_bh_enable(); + put_cpu_fpsimd_context(); } /* @@ -1120,7 +1174,8 @@ void fpsimd_flush_task_state(struct task_struct *t) /* * Invalidate any task's FPSIMD state that is present on this cpu. - * This function must be called with softirqs disabled. + * The FPSIMD context should be acquired with get_cpu_fpsimd_context() + * before calling this function. */ static void fpsimd_flush_cpu_state(void) { @@ -1128,21 +1183,17 @@ static void fpsimd_flush_cpu_state(void) set_thread_flag(TIF_FOREIGN_FPSTATE); } -/* - * Save the FPSIMD state to memory and invalidate cpu view. - * This function must be called with softirqs (and preemption) disabled. - */ +/* Save the FPSIMD state to memory and invalidate cpu view. */ void fpsimd_save_and_flush_cpu_state(void) { + get_cpu_fpsimd_context(); fpsimd_save(); fpsimd_flush_cpu_state(); + put_cpu_fpsimd_context(); } #ifdef CONFIG_KERNEL_MODE_NEON -DEFINE_PER_CPU(bool, kernel_neon_busy); -EXPORT_PER_CPU_SYMBOL(kernel_neon_busy); - /* * Kernel-side NEON support functions */ @@ -1167,19 +1218,13 @@ void kernel_neon_begin(void) BUG_ON(!may_use_simd()); - local_bh_disable(); - - __this_cpu_write(kernel_neon_busy, true); + get_cpu_fpsimd_context(); /* Save unsaved fpsimd state, if any: */ fpsimd_save(); /* Invalidate any task state remaining in the fpsimd regs: */ fpsimd_flush_cpu_state(); - - preempt_disable(); - - local_bh_enable(); } EXPORT_SYMBOL(kernel_neon_begin); @@ -1194,15 +1239,10 @@ EXPORT_SYMBOL(kernel_neon_begin); */ void kernel_neon_end(void) { - bool busy; - if (!system_supports_fpsimd()) return; - busy = __this_cpu_xchg(kernel_neon_busy, false); - WARN_ON(!busy); /* No matching kernel_neon_begin()? */ - - preempt_enable(); + put_cpu_fpsimd_context(); } EXPORT_SYMBOL(kernel_neon_end);