From patchwork Fri Nov 30 17:34:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 152576 Delivered-To: patch@linaro.org Received: by 2002:a2e:299d:0:0:0:0:0 with SMTP id p29-v6csp3923639ljp; Fri, 30 Nov 2018 09:34:17 -0800 (PST) X-Google-Smtp-Source: AFSGD/UOGArIhX1qJM1pX1xq7hxvRKSl3+XUaECY63j8Ww8SokiumsrTs00lP6uJrPeuItq57OuY X-Received: by 2002:a63:eb0e:: with SMTP id t14mr5619407pgh.445.1543599257249; Fri, 30 Nov 2018 09:34:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1543599257; cv=none; d=google.com; s=arc-20160816; b=VnmMEoQqnMszpzKOLCSC3Xm9AZuQQiIMWfw+rAgCTVFBuzykF28au+p0QgEB0CG+yk Zv2xZirvw8EkOEa4Lwzdw4A9E3iLsDDxQoFXwhTvyRzBVa362ZQ6Uo8v94CM1AhmUVG5 e4OmMKCDP2SGXsvpTdnsDoK3atUUEbQQz+vmeVNKaTBnkrng0OD2HZWa48YWYh2NnP/E mz0iY/PrvVydDbC27+ACfewB+D9ZwLSki8nbwgUHaRDvKGnMYFqX2rSUm2p7TzOzLKxt 1mHpqlbZ7lS6AH0h0djVGpinT0CdUsGZgg9fwZC4AYU5jVJUNoZKQQaz5Xj3NuAR2Hvc LFlg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=mYHNG8lfEzIyMHUhDDLvx87Kw5p0THgL4LdZGWas/wk=; b=vikUjEmtYA0I4e3JoqulmPrRCllh5ZHTjCdWtCso/wA6QSfYhZfvTfQwluOZWjjHaP Ski0mD2S8FHgN84sICK9PTwHSa+DwsLcq6s7wf/F2zmNIqqrkFXx2vReiLh0zK/wWNww XyD19yXIH8uSDI0/gpToxywqJ7YBYp0sLj5QSo9l656WBMu27eBoZaRhboFkbOkyT0Gb VG+PN83uqppvNgKrptGZuiuFku7DjMkU9wO73I3X6aHeUu3TAxZ+yxLGAz/B6v1HCbF5 UU3LNr5acmrGcuWwzEJYzv2613fyCn0B37f0xkHr1xx9Gs30Z4FSPFP0FpYV0hjW755W oCrA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id y8si5927551pfn.26.2018.11.30.09.34.16; Fri, 30 Nov 2018 09:34:17 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727189AbeLAEoP (ORCPT + 32 others); Fri, 30 Nov 2018 23:44:15 -0500 Received: from foss.arm.com ([217.140.101.70]:33166 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726650AbeLAEoO (ORCPT ); Fri, 30 Nov 2018 23:44:14 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E4C8B1596; Fri, 30 Nov 2018 09:34:13 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B50883F73C; Fri, 30 Nov 2018 09:34:13 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 846C21AE0808; Fri, 30 Nov 2018 17:34:32 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, ard.biesheuvel@linaro.org, catalin.marinas@arm.com, rml@tech9.net, tglx@linutronix.de, peterz@infradead.org, schwidefsky@de.ibm.com, Will Deacon Subject: [PATCH v2 1/2] preempt: Move PREEMPT_NEED_RESCHED definition into arch code Date: Fri, 30 Nov 2018 17:34:30 +0000 Message-Id: <1543599271-14339-2-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1543599271-14339-1-git-send-email-will.deacon@arm.com> References: <1543599271-14339-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org PREEMPT_NEED_RESCHED is never used directly, so move it into the arch code where it can potentially be implemented using either a different bit in the preempt count or as an entirely separate entity. Cc: Robert Love Cc: Thomas Gleixner Cc: Peter Zijlstra Cc: Martin Schwidefsky Signed-off-by: Will Deacon --- arch/s390/include/asm/preempt.h | 2 ++ arch/x86/include/asm/preempt.h | 3 +++ include/linux/preempt.h | 3 --- 3 files changed, 5 insertions(+), 3 deletions(-) -- 2.1.4 diff --git a/arch/s390/include/asm/preempt.h b/arch/s390/include/asm/preempt.h index 23a14d187fb1..b5ea9e14c017 100644 --- a/arch/s390/include/asm/preempt.h +++ b/arch/s390/include/asm/preempt.h @@ -8,6 +8,8 @@ #ifdef CONFIG_HAVE_MARCH_Z196_FEATURES +/* We use the MSB mostly because its available */ +#define PREEMPT_NEED_RESCHED 0x80000000 #define PREEMPT_ENABLED (0 + PREEMPT_NEED_RESCHED) static inline int preempt_count(void) diff --git a/arch/x86/include/asm/preempt.h b/arch/x86/include/asm/preempt.h index 90cb2f36c042..99a7fa9ab0a3 100644 --- a/arch/x86/include/asm/preempt.h +++ b/arch/x86/include/asm/preempt.h @@ -8,6 +8,9 @@ DECLARE_PER_CPU(int, __preempt_count); +/* We use the MSB mostly because its available */ +#define PREEMPT_NEED_RESCHED 0x80000000 + /* * We use the PREEMPT_NEED_RESCHED bit as an inverted NEED_RESCHED such * that a decrement hitting 0 means we can and should reschedule. diff --git a/include/linux/preempt.h b/include/linux/preempt.h index c01813c3fbe9..dd92b1a93919 100644 --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -53,9 +53,6 @@ #define SOFTIRQ_DISABLE_OFFSET (2 * SOFTIRQ_OFFSET) -/* We use the MSB mostly because its available */ -#define PREEMPT_NEED_RESCHED 0x80000000 - #define PREEMPT_DISABLED (PREEMPT_DISABLE_OFFSET + PREEMPT_ENABLED) /* From patchwork Fri Nov 30 17:34:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 152577 Delivered-To: patch@linaro.org Received: by 2002:a2e:299d:0:0:0:0:0 with SMTP id p29-v6csp3923761ljp; Fri, 30 Nov 2018 09:34:24 -0800 (PST) X-Google-Smtp-Source: AFSGD/V71C0z9VGeOyzPvUi3rlWBqRpYVpRW4lnEMKB6L2w68zJsYi/gx/Rmv2hLjrfP/VWqn2+9 X-Received: by 2002:a63:eb0e:: with SMTP id t14mr5619697pgh.445.1543599264833; Fri, 30 Nov 2018 09:34:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1543599264; cv=none; d=google.com; s=arc-20160816; b=QFoMHzfF0ZeqrKkr5XsF8krtymBIbbr0UZ9witxw1haAeOHft/wicEnhE7Eew17ZSO W5+Uw23LEW6PWUaVX8ZxnbPmUqvl9Fugkyds4Ub4PgWlMyXzBRwoH+2zrdvEI1WXzx4f P1MElHGDUyPjYzFEXTlH/skjziu5hSgdw+7W++CvaS/NlIDibCldsfdEAqUM7JwKQZZI s6wDxASjHXiAPUheyqXHWurYTmmhRBaidqTay3gMTN+CM0UDSVASOH2rlAUJcCeaDao/ fHPTCHHs5xo3EezSLIZCzaUtN12puSVXu1P9PKTytlgf7MPUHb/0GJP75aFSNrx38OiF KGHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=no8khM+H21+oEcW+H1IIWVZYQHHCZssWHaWmYWbGnRk=; b=lRLhfKxYQpfUWrA8QxcTZ8pA/FFqkKVSiENlNQv+7iLNw2UAW5X70UaT/PdmcWF4/u uLLBdxZhSUrTzf2X9vSonP9zybt0BPj18wtiAKzDIR+e5cIFzX+O6S0JfLVHo78UX7Ew M2JWgQ4CQG/1UwOlP7C6GG32w5C5VNtZ7370WgfnnxAFZdRXq/AxG+J3zWocYPp1u5DW iwTCjYtlsG0+8CKDJCnN8PZJLKg8Xwv40QueV5DpBX6jHszMEfD1VCX1eNNqBHH9P8Pm 0CQpVFcSWCmyWJHiO3OycE9kykuQNuUq2g4J9qUmwlPma+vLKFpx2Dq2z9NAB7oWSyJo XChQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id t10si5262509pgn.551.2018.11.30.09.34.24; Fri, 30 Nov 2018 09:34:24 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727296AbeLAEoX (ORCPT + 32 others); Fri, 30 Nov 2018 23:44:23 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:33172 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726712AbeLAEoO (ORCPT ); Fri, 30 Nov 2018 23:44:14 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id F390615AB; Fri, 30 Nov 2018 09:34:13 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C40923F763; Fri, 30 Nov 2018 09:34:13 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 98CC91AE1007; Fri, 30 Nov 2018 17:34:32 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, ard.biesheuvel@linaro.org, catalin.marinas@arm.com, rml@tech9.net, tglx@linutronix.de, peterz@infradead.org, schwidefsky@de.ibm.com, Will Deacon Subject: [PATCH v2 2/2] arm64: preempt: Provide our own implementation of asm/preempt.h Date: Fri, 30 Nov 2018 17:34:31 +0000 Message-Id: <1543599271-14339-3-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1543599271-14339-1-git-send-email-will.deacon@arm.com> References: <1543599271-14339-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The asm-generic/preempt.h implementation doesn't make use of the PREEMPT_NEED_RESCHED flag, since this can interact badly with load/store architectures which rely on the preempt_count word being unchanged across an interrupt. However, since we're a 64-bit architecture and the preempt count is only 32 bits wide, we can simply pack it next to the resched flag and load the whole thing in one go, so that a dec-and-test operation doesn't need to load twice. Signed-off-by: Will Deacon --- arch/arm64/include/asm/Kbuild | 1 - arch/arm64/include/asm/preempt.h | 88 ++++++++++++++++++++++++++++++++++++ arch/arm64/include/asm/thread_info.h | 13 +++++- 3 files changed, 100 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/include/asm/preempt.h -- 2.1.4 Reviewed-by: Ard Biesheuvel diff --git a/arch/arm64/include/asm/Kbuild b/arch/arm64/include/asm/Kbuild index 6cd5d77b6b44..33498f900390 100644 --- a/arch/arm64/include/asm/Kbuild +++ b/arch/arm64/include/asm/Kbuild @@ -14,7 +14,6 @@ generic-y += local64.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += msi.h -generic-y += preempt.h generic-y += qrwlock.h generic-y += qspinlock.h generic-y += rwsem.h diff --git a/arch/arm64/include/asm/preempt.h b/arch/arm64/include/asm/preempt.h new file mode 100644 index 000000000000..f1c1398cf065 --- /dev/null +++ b/arch/arm64/include/asm/preempt.h @@ -0,0 +1,88 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_PREEMPT_H +#define __ASM_PREEMPT_H + +#include + +#define PREEMPT_NEED_RESCHED BIT(32) +#define PREEMPT_ENABLED (PREEMPT_NEED_RESCHED) + +static inline int preempt_count(void) +{ + return READ_ONCE(current_thread_info()->preempt.count); +} + +static inline void preempt_count_set(u64 pc) +{ + /* Preserve existing value of PREEMPT_NEED_RESCHED */ + WRITE_ONCE(current_thread_info()->preempt.count, pc); +} + +#define init_task_preempt_count(p) do { \ + task_thread_info(p)->preempt_count = FORK_PREEMPT_COUNT; \ +} while (0) + +#define init_idle_preempt_count(p, cpu) do { \ + task_thread_info(p)->preempt_count = PREEMPT_ENABLED; \ +} while (0) + +static inline void set_preempt_need_resched(void) +{ + current_thread_info()->preempt.need_resched = 0; +} + +static inline void clear_preempt_need_resched(void) +{ + current_thread_info()->preempt.need_resched = 1; +} + +static inline bool test_preempt_need_resched(void) +{ + return !current_thread_info()->preempt.need_resched; +} + +static inline void __preempt_count_add(int val) +{ + u32 pc = READ_ONCE(current_thread_info()->preempt.count); + pc += val; + WRITE_ONCE(current_thread_info()->preempt.count, pc); +} + +static inline void __preempt_count_sub(int val) +{ + u32 pc = READ_ONCE(current_thread_info()->preempt.count); + pc -= val; + WRITE_ONCE(current_thread_info()->preempt.count, pc); +} + +static inline bool __preempt_count_dec_and_test(void) +{ + struct thread_info *ti = current_thread_info(); + u64 pc = READ_ONCE(ti->preempt_count); + + WRITE_ONCE(ti->preempt.count, --pc); + + /* + * If we wrote back all zeroes, then we're preemptible and in + * need of a reschedule. Otherwise, we need to reload the + * preempt_count in case the need_resched flag was cleared by an + * interrupt occurring between the non-atomic READ_ONCE/WRITE_ONCE + * pair. + */ + return !pc || !READ_ONCE(ti->preempt_count); +} + +static inline bool should_resched(int preempt_offset) +{ + u64 pc = READ_ONCE(current_thread_info()->preempt_count); + return pc == preempt_offset; +} + +#ifdef CONFIG_PREEMPT +void preempt_schedule(void); +#define __preempt_schedule() preempt_schedule() +void preempt_schedule_notrace(void); +#define __preempt_schedule_notrace() preempt_schedule_notrace() +#endif /* CONFIG_PREEMPT */ + +#endif /* __ASM_PREEMPT_H */ diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index cb2c10a8f0a8..bbca68b54732 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -42,7 +42,18 @@ struct thread_info { #ifdef CONFIG_ARM64_SW_TTBR0_PAN u64 ttbr0; /* saved TTBR0_EL1 */ #endif - int preempt_count; /* 0 => preemptable, <0 => bug */ + union { + u64 preempt_count; /* 0 => preemptible, <0 => bug */ + struct { +#ifdef CONFIG_CPU_BIG_ENDIAN + u32 need_resched; + u32 count; +#else + u32 count; + u32 need_resched; +#endif + } preempt; + }; }; #define thread_saved_pc(tsk) \