From patchwork Tue May 19 23:25:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anchal Agarwal X-Patchwork-Id: 211999 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.4 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E83ACC433E2 for ; Tue, 19 May 2020 23:25:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AE9F320823 for ; Tue, 19 May 2020 23:25:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="add5YAzh" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728316AbgESXZx (ORCPT ); Tue, 19 May 2020 19:25:53 -0400 Received: from smtp-fw-4101.amazon.com ([72.21.198.25]:50535 "EHLO smtp-fw-4101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725998AbgESXZx (ORCPT ); Tue, 19 May 2020 19:25:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1589930752; x=1621466752; h=date:from:to:subject:message-id:references:mime-version: in-reply-to; bh=u2I4SVbLijVxq/T2OZK7zwcQCdIZfr/59rBd0BiigZo=; b=add5YAzhSeZGNYThHYUwV2AuZnIsvo7hN5N8bwZDQa5sTmZh+/kmic0V pQoYTLE3bAGprOhIAd4vYnnkHi0gdYN4/i5Wbcr3tZXb5NbIECp8reQJw yyajBQsL8pu0UQNVYRHnD/myL0fgJ4qzVJm8WkopGuOFjfsYFW5MRy/S4 Q=; IronPort-SDR: PUzafnrD9npruL/kMbXLm5hpJ1popdhgSPJqodhqecm7/PsBTkLK6Um1IrM4U5sGzedbKOXRIc CO06+Sbl0hSw== X-IronPort-AV: E=Sophos;i="5.73,411,1583193600"; d="scan'208";a="31182118" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2b-4e24fd92.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-4101.iad4.amazon.com with ESMTP; 19 May 2020 23:25:37 +0000 Received: from EX13MTAUEE002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2b-4e24fd92.us-west-2.amazon.com (Postfix) with ESMTPS id 76EF6A06D8; Tue, 19 May 2020 23:25:35 +0000 (UTC) Received: from EX13D08UEE002.ant.amazon.com (10.43.62.92) by EX13MTAUEE002.ant.amazon.com (10.43.62.24) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 19 May 2020 23:25:09 +0000 Received: from EX13MTAUEA002.ant.amazon.com (10.43.61.77) by EX13D08UEE002.ant.amazon.com (10.43.62.92) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 19 May 2020 23:25:09 +0000 Received: from dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (172.22.96.68) by mail-relay.amazon.com (10.43.61.169) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Tue, 19 May 2020 23:25:09 +0000 Received: by dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (Postfix, from userid 4335130) id CE7BE40712; Tue, 19 May 2020 23:25:08 +0000 (UTC) Date: Tue, 19 May 2020 23:25:08 +0000 From: Anchal Agarwal To: , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH 02/12] xenbus: add freeze/thaw/restore callbacks support Message-ID: <7fd12227f923eacc5841b47bd69f72b4105843a7.1589926004.git.anchalag@amazon.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-pm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org From: Munehisa Kamata Since commit b3e96c0c7562 ("xen: use freeze/restore/thaw PM events for suspend/resume/chkpt"), xenbus uses PMSG_FREEZE, PMSG_THAW and PMSG_RESTORE events for Xen suspend. However, they're actually assigned to xenbus_dev_suspend(), xenbus_dev_cancel() and xenbus_dev_resume() respectively, and only suspend and resume callbacks are supported at driver level. To support PM suspend and PM hibernation, modify the bus level PM callbacks to invoke not only device driver's suspend/resume but also freeze/thaw/restore. Note that we'll use freeze/restore callbacks even for PM suspend whereas suspend/resume callbacks are normally used in the case, becausae the existing xenbus device drivers already have suspend/resume callbacks specifically designed for Xen suspend. So we can allow the device drivers to keep the existing callbacks wihtout modification. [Anchal Changelog: Refactored the callbacks code] Signed-off-by: Agarwal Anchal Signed-off-by: Munehisa Kamata --- drivers/xen/xenbus/xenbus_probe.c | 99 +++++++++++++++++++++++++------ include/xen/xenbus.h | 3 + 2 files changed, 84 insertions(+), 18 deletions(-) diff --git a/drivers/xen/xenbus/xenbus_probe.c b/drivers/xen/xenbus/xenbus_probe.c index 8c4d05b687b7..1589b9b2cb56 100644 --- a/drivers/xen/xenbus/xenbus_probe.c +++ b/drivers/xen/xenbus/xenbus_probe.c @@ -49,6 +49,7 @@ #include #include #include +#include #include #include @@ -599,27 +600,44 @@ int xenbus_dev_suspend(struct device *dev) struct xenbus_driver *drv; struct xenbus_device *xdev = container_of(dev, struct xenbus_device, dev); - + bool xen_suspend = xen_suspend_mode_is_xen_suspend(); DPRINTK("%s", xdev->nodename); if (dev->driver == NULL) return 0; drv = to_xenbus_driver(dev->driver); - if (drv->suspend) - err = drv->suspend(xdev); - if (err) - pr_warn("suspend %s failed: %i\n", dev_name(dev), err); + + if (xen_suspend) { + if (drv->suspend) + err = drv->suspend(xdev); + } else { + if (drv->freeze) { + err = drv->freeze(xdev); + if (!err) { + free_otherend_watch(xdev); + free_otherend_details(xdev); + return 0; + } + } + } + + if (err) { + pr_warn("%s %s failed: %i\n", xen_suspend ? + "suspend" : "freeze", dev_name(dev), err); + return err; + } + return 0; } EXPORT_SYMBOL_GPL(xenbus_dev_suspend); int xenbus_dev_resume(struct device *dev) { - int err; + int err = 0; struct xenbus_driver *drv; struct xenbus_device *xdev = container_of(dev, struct xenbus_device, dev); - + bool xen_suspend = xen_suspend_mode_is_xen_suspend(); DPRINTK("%s", xdev->nodename); if (dev->driver == NULL) @@ -627,24 +645,32 @@ int xenbus_dev_resume(struct device *dev) drv = to_xenbus_driver(dev->driver); err = talk_to_otherend(xdev); if (err) { - pr_warn("resume (talk_to_otherend) %s failed: %i\n", + pr_warn("%s (talk_to_otherend) %s failed: %i\n", + xen_suspend ? "resume" : "restore", dev_name(dev), err); return err; } - xdev->state = XenbusStateInitialising; + if (xen_suspend) { + xdev->state = XenbusStateInitialising; + if (drv->resume) + err = drv->resume(xdev); + } else { + if (drv->restore) + err = drv->restore(xdev); + } - if (drv->resume) { - err = drv->resume(xdev); - if (err) { - pr_warn("resume %s failed: %i\n", dev_name(dev), err); - return err; - } + if (err) { + pr_warn("%s %s failed: %i\n", + xen_suspend ? "resume" : "restore", + dev_name(dev), err); + return err; } err = watch_otherend(xdev); if (err) { - pr_warn("resume (watch_otherend) %s failed: %d.\n", + pr_warn("%s (watch_otherend) %s failed: %d.\n", + xen_suspend ? "resume" : "restore", dev_name(dev), err); return err; } @@ -655,8 +681,45 @@ EXPORT_SYMBOL_GPL(xenbus_dev_resume); int xenbus_dev_cancel(struct device *dev) { - /* Do nothing */ - DPRINTK("cancel"); + int err = 0; + struct xenbus_driver *drv; + struct xenbus_device *xdev + = container_of(dev, struct xenbus_device, dev); + bool xen_suspend = xen_suspend_mode_is_xen_suspend(); + + if (xen_suspend) { + /* Do nothing */ + DPRINTK("cancel"); + return 0; + } + + DPRINTK("%s", xdev->nodename); + + if (dev->driver == NULL) + return 0; + drv = to_xenbus_driver(dev->driver); + err = talk_to_otherend(xdev); + if (err) { + pr_warn("thaw (talk_to_otherend) %s failed: %d.\n", + dev_name(dev), err); + return err; + } + + if (drv->thaw) { + err = drv->thaw(xdev); + if (err) { + pr_warn("thaw %s failed: %i\n", dev_name(dev), err); + return err; + } + } + + err = watch_otherend(xdev); + if (err) { + pr_warn("thaw (watch_otherend) %s failed: %d.\n", + dev_name(dev), err); + return err; + } + return 0; } EXPORT_SYMBOL_GPL(xenbus_dev_cancel); diff --git a/include/xen/xenbus.h b/include/xen/xenbus.h index 5a8315e6d8a6..8da964763255 100644 --- a/include/xen/xenbus.h +++ b/include/xen/xenbus.h @@ -104,6 +104,9 @@ struct xenbus_driver { int (*remove)(struct xenbus_device *dev); int (*suspend)(struct xenbus_device *dev); int (*resume)(struct xenbus_device *dev); + int (*freeze)(struct xenbus_device *dev); + int (*thaw)(struct xenbus_device *dev); + int (*restore)(struct xenbus_device *dev); int (*uevent)(struct xenbus_device *, struct kobj_uevent_env *); struct device_driver driver; int (*read_otherend_details)(struct xenbus_device *dev); From patchwork Tue May 19 23:26:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anchal Agarwal X-Patchwork-Id: 211998 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.4 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4EB5C433DF for ; Tue, 19 May 2020 23:26:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7ED5220823 for ; Tue, 19 May 2020 23:26:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="K46XGdq+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726344AbgESX0j (ORCPT ); Tue, 19 May 2020 19:26:39 -0400 Received: from smtp-fw-2101.amazon.com ([72.21.196.25]:64802 "EHLO smtp-fw-2101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726178AbgESX0i (ORCPT ); Tue, 19 May 2020 19:26:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1589930798; x=1621466798; h=date:from:to:subject:message-id:references:mime-version: in-reply-to; bh=+5sqW/9tRYGzDifQrxQJ7j+i6WgVO1jvWkTIJZXYHhc=; b=K46XGdq+bQjjsBSF58nOLshjyQJUeGKZesZ9jXdJfsN/covNZ3BRqVU/ tGtr/k9ujNTkUHmXPTzVco+WvZo0/HKbfy54uVrjU49o5CA54bJ8MVNWP rwysXkR544eRybIl/BESTHj2ng4qsUgqFYGKKqACW/qPKEdfAMH2a6b71 Q=; IronPort-SDR: mWw8AR8ej5ULAmO2NWPA9U3fjvDcfrEySiPbnKH0hbJ0nS6cPVOD41yjXFXqxCxYYtwfKRalTW k2MS/P3b7VvQ== X-IronPort-AV: E=Sophos;i="5.73,411,1583193600"; d="scan'208";a="31243916" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-2a-538b0bfb.us-west-2.amazon.com) ([10.43.8.2]) by smtp-border-fw-out-2101.iad2.amazon.com with ESMTP; 19 May 2020 23:26:23 +0000 Received: from EX13MTAUWB001.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2a-538b0bfb.us-west-2.amazon.com (Postfix) with ESMTPS id CA0F8A1EEC; Tue, 19 May 2020 23:26:20 +0000 (UTC) Received: from EX13D01UWB002.ant.amazon.com (10.43.161.136) by EX13MTAUWB001.ant.amazon.com (10.43.161.249) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 19 May 2020 23:26:15 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13d01UWB002.ant.amazon.com (10.43.161.136) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 19 May 2020 23:26:15 +0000 Received: from dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (172.22.96.68) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Tue, 19 May 2020 23:26:15 +0000 Received: by dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (Postfix, from userid 4335130) id 5012E40712; Tue, 19 May 2020 23:26:15 +0000 (UTC) Date: Tue, 19 May 2020 23:26:15 +0000 From: Anchal Agarwal To: , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH 04/12] x86/xen: add system core suspend and resume callbacks Message-ID: <79cf02631dc00e62ebf90410bfbbdb52fe7024cb.1589926004.git.anchalag@amazon.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-pm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org From: Munehisa Kamata Add Xen PVHVM specific system core callbacks for PM suspend and hibernation support. The callbacks suspend and resume Xen primitives,like shared_info, pvclock and grant table. Note that Xen suspend can handle them in a different manner, but system core callbacks are called from the context. So if the callbacks are called from Xen suspend context, return immediately. Signed-off-by: Agarwal Anchal Signed-off-by: Munehisa Kamata --- arch/x86/xen/enlighten_hvm.c | 1 + arch/x86/xen/suspend.c | 53 ++++++++++++++++++++++++++++++++++++ include/xen/xen-ops.h | 3 ++ 3 files changed, 57 insertions(+) diff --git a/arch/x86/xen/enlighten_hvm.c b/arch/x86/xen/enlighten_hvm.c index 75b1ec7a0fcd..138e71786e03 100644 --- a/arch/x86/xen/enlighten_hvm.c +++ b/arch/x86/xen/enlighten_hvm.c @@ -204,6 +204,7 @@ static void __init xen_hvm_guest_init(void) if (xen_feature(XENFEAT_hvm_callback_vector)) xen_have_vector_callback = 1; + xen_setup_syscore_ops(); xen_hvm_smp_init(); WARN_ON(xen_cpuhp_setup(xen_cpu_up_prepare_hvm, xen_cpu_dead_hvm)); xen_unplug_emulated_devices(); diff --git a/arch/x86/xen/suspend.c b/arch/x86/xen/suspend.c index 1d83152c761b..784c4484100b 100644 --- a/arch/x86/xen/suspend.c +++ b/arch/x86/xen/suspend.c @@ -2,17 +2,22 @@ #include #include #include +#include +#include #include #include +#include #include #include +#include #include #include #include #include #include +#include #include "xen-ops.h" #include "mmu.h" @@ -82,3 +87,51 @@ void xen_arch_suspend(void) on_each_cpu(xen_vcpu_notify_suspend, NULL, 1); } + +static int xen_syscore_suspend(void) +{ + struct xen_remove_from_physmap xrfp; + int ret; + + /* Xen suspend does similar stuffs in its own logic */ + if (xen_suspend_mode_is_xen_suspend()) + return 0; + + xrfp.domid = DOMID_SELF; + xrfp.gpfn = __pa(HYPERVISOR_shared_info) >> PAGE_SHIFT; + + ret = HYPERVISOR_memory_op(XENMEM_remove_from_physmap, &xrfp); + if (!ret) + HYPERVISOR_shared_info = &xen_dummy_shared_info; + + return ret; +} + +static void xen_syscore_resume(void) +{ + /* Xen suspend does similar stuffs in its own logic */ + if (xen_suspend_mode_is_xen_suspend()) + return; + + /* No need to setup vcpu_info as it's already moved off */ + xen_hvm_map_shared_info(); + + pvclock_resume(); + + gnttab_resume(); +} + +/* + * These callbacks will be called with interrupts disabled and when having only + * one CPU online. + */ +static struct syscore_ops xen_hvm_syscore_ops = { + .suspend = xen_syscore_suspend, + .resume = xen_syscore_resume +}; + +void __init xen_setup_syscore_ops(void) +{ + if (xen_hvm_domain()) + register_syscore_ops(&xen_hvm_syscore_ops); +} diff --git a/include/xen/xen-ops.h b/include/xen/xen-ops.h index 4ffe031adfc7..89b1e88712d6 100644 --- a/include/xen/xen-ops.h +++ b/include/xen/xen-ops.h @@ -43,6 +43,9 @@ int xen_setup_shutdown_event(void); bool xen_suspend_mode_is_xen_suspend(void); bool xen_suspend_mode_is_pm_suspend(void); bool xen_suspend_mode_is_pm_hibernation(void); + +void xen_setup_syscore_ops(void); + extern unsigned long *xen_contiguous_bitmap; #if defined(CONFIG_XEN_PV) || defined(CONFIG_ARM) || defined(CONFIG_ARM64) From patchwork Tue May 19 23:34:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anchal Agarwal X-Patchwork-Id: 211994 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.4 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61F4CC433E1 for ; Tue, 19 May 2020 23:35:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 35746207C4 for ; Tue, 19 May 2020 23:35:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="E/2LWijB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726344AbgESXfC (ORCPT ); Tue, 19 May 2020 19:35:02 -0400 Received: from smtp-fw-2101.amazon.com ([72.21.196.25]:1590 "EHLO smtp-fw-2101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726178AbgESXfC (ORCPT ); Tue, 19 May 2020 19:35:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1589931301; x=1621467301; h=date:from:to:subject:message-id:references:mime-version: in-reply-to; bh=8UFUtTeRpjYPvMFrw037W1OadSP0YXrEVVUVgG1wRzY=; b=E/2LWijB4b2A0aKW1XVM79IKadw9EpRav99LgKTtTDYhAvtZ1poUw2yK j7fN1jgwrfbd+TNz/WZ0U4Zl/0h1Rk0iyYNtUaM14kqSnqRMwdKX5t6+S UqwL1xnYMb5tsFh2L97x5BpuyrcABEOkfnzQxsEpkB+Oz9EAJL50vBtzc U=; IronPort-SDR: PstKWIwiKXwlx7AwfI2MpbFwnnvbl4PRGOucOXTFL8WICxveEsjrTgaNNQtydlcjq5vYiBt8hp NCHu/6FrICEg== X-IronPort-AV: E=Sophos;i="5.73,411,1583193600"; d="scan'208";a="31244955" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-2a-d0be17ee.us-west-2.amazon.com) ([10.43.8.2]) by smtp-border-fw-out-2101.iad2.amazon.com with ESMTP; 19 May 2020 23:34:58 +0000 Received: from EX13MTAUEE002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2a-d0be17ee.us-west-2.amazon.com (Postfix) with ESMTPS id E4F73A2003; Tue, 19 May 2020 23:34:55 +0000 (UTC) Received: from EX13D08UEE002.ant.amazon.com (10.43.62.92) by EX13MTAUEE002.ant.amazon.com (10.43.62.24) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 19 May 2020 23:34:46 +0000 Received: from EX13MTAUEA002.ant.amazon.com (10.43.61.77) by EX13D08UEE002.ant.amazon.com (10.43.62.92) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 19 May 2020 23:34:45 +0000 Received: from dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (172.22.96.68) by mail-relay.amazon.com (10.43.61.169) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Tue, 19 May 2020 23:34:45 +0000 Received: by dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (Postfix, from userid 4335130) id 12AFA40712; Tue, 19 May 2020 23:34:45 +0000 (UTC) Date: Tue, 19 May 2020 23:34:45 +0000 From: Anchal Agarwal To: , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH 05/12] genirq: Shutdown irq chips in suspend/resume during hibernation Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-pm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Message-ID: <20200519233445._ffERAkOIKFWOq6NJvUQ3vPsno7S2-rEcYH8aUSWn_Q@z> Resending with fixed Signed-off-by. Many legacy device drivers do not implement power management (PM) functions which means that interrupts requested by these drivers stay in active state when the kernel is hibernated. This does not matter on bare metal and on most hypervisors because the interrupt is restored on resume without any noticable side effects as it stays connected to the same physical or virtual interrupt line. The XEN interrupt mechanism is different as it maintains a mapping between the Linux interrupt number and a XEN event channel. If the interrupt stays active on hibernation this mapping is preserved but there is unfortunately no guarantee that on resume the same event channels are reassigned to these devices. This can result in event channel conflicts which prevent the affected devices from being restored correctly. One way to solve this would be to add the necessary power management functions to all affected legacy device drivers, but that's a questionable effort which does not provide any benefits on non-XEN environments. The least intrusive and most efficient solution is to provide a mechanism which allows the core interrupt code to tear down these interrupts on hibernation and bring them back up again on resume. This allows the XEN event channel mechanism to assign an arbitrary event channel on resume without affecting the functionality of these devices. Fortunately all these device interrupts are handled by a dedicated XEN interrupt chip so the chip can be marked that all interrupts connected to it are handled this way. This is pretty much in line with the other interrupt chip specific quirks, e.g. IRQCHIP_MASK_ON_SUSPEND. Add a new quirk flag IRQCHIP_SHUTDOWN_ON_SUSPEND and add support for it the core interrupt suspend/resume paths. Signed-off-by: Anchal Agarwal Signed-off-by: Thomas Gleixner --- drivers/xen/events/events_base.c | 1 + include/linux/irq.h | 2 ++ kernel/irq/chip.c | 2 +- kernel/irq/internals.h | 1 + kernel/irq/pm.c | 31 ++++++++++++++++++++++--------- 5 files changed, 27 insertions(+), 10 deletions(-) diff --git a/drivers/xen/events/events_base.c b/drivers/xen/events/events_base.c index 3a791c8485d0..decf65bd3451 100644 --- a/drivers/xen/events/events_base.c +++ b/drivers/xen/events/events_base.c @@ -1613,6 +1613,7 @@ static struct irq_chip xen_pirq_chip __read_mostly = { .irq_set_affinity = set_affinity_irq, .irq_retrigger = retrigger_dynirq, + .flags = IRQCHIP_SHUTDOWN_ON_SUSPEND, }; static struct irq_chip xen_percpu_chip __read_mostly = { diff --git a/include/linux/irq.h b/include/linux/irq.h index 8d5bc2c237d7..94cb8c994d06 100644 --- a/include/linux/irq.h +++ b/include/linux/irq.h @@ -542,6 +542,7 @@ struct irq_chip { * IRQCHIP_EOI_THREADED: Chip requires eoi() on unmask in threaded mode * IRQCHIP_SUPPORTS_LEVEL_MSI Chip can provide two doorbells for Level MSIs * IRQCHIP_SUPPORTS_NMI: Chip can deliver NMIs, only for root irqchips + * IRQCHIP_SHUTDOWN_ON_SUSPEND: Shutdown non wake irqs in the suspend path */ enum { IRQCHIP_SET_TYPE_MASKED = (1 << 0), @@ -553,6 +554,7 @@ enum { IRQCHIP_EOI_THREADED = (1 << 6), IRQCHIP_SUPPORTS_LEVEL_MSI = (1 << 7), IRQCHIP_SUPPORTS_NMI = (1 << 8), + IRQCHIP_SHUTDOWN_ON_SUSPEND = (1 << 9), }; #include diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c index 41e7e37a0928..fd59489ff14b 100644 --- a/kernel/irq/chip.c +++ b/kernel/irq/chip.c @@ -233,7 +233,7 @@ __irq_startup_managed(struct irq_desc *desc, struct cpumask *aff, bool force) } #endif -static int __irq_startup(struct irq_desc *desc) +int __irq_startup(struct irq_desc *desc) { struct irq_data *d = irq_desc_get_irq_data(desc); int ret = 0; diff --git a/kernel/irq/internals.h b/kernel/irq/internals.h index 7db284b10ac9..b6fca5eacff7 100644 --- a/kernel/irq/internals.h +++ b/kernel/irq/internals.h @@ -80,6 +80,7 @@ extern void __enable_irq(struct irq_desc *desc); extern int irq_activate(struct irq_desc *desc); extern int irq_activate_and_startup(struct irq_desc *desc, bool resend); extern int irq_startup(struct irq_desc *desc, bool resend, bool force); +extern int __irq_startup(struct irq_desc *desc); extern void irq_shutdown(struct irq_desc *desc); extern void irq_shutdown_and_deactivate(struct irq_desc *desc); diff --git a/kernel/irq/pm.c b/kernel/irq/pm.c index 8f557fa1f4fe..dc48a25f1756 100644 --- a/kernel/irq/pm.c +++ b/kernel/irq/pm.c @@ -85,16 +85,25 @@ static bool suspend_device_irq(struct irq_desc *desc) } desc->istate |= IRQS_SUSPENDED; - __disable_irq(desc); - /* - * Hardware which has no wakeup source configuration facility - * requires that the non wakeup interrupts are masked at the - * chip level. The chip implementation indicates that with - * IRQCHIP_MASK_ON_SUSPEND. + * Some irq chips (e.g. XEN PIRQ) require a full shutdown on suspend + * as some of the legacy drivers(e.g. floppy) do nothing during the + * suspend path */ - if (irq_desc_get_chip(desc)->flags & IRQCHIP_MASK_ON_SUSPEND) - mask_irq(desc); + if (irq_desc_get_chip(desc)->flags & IRQCHIP_SHUTDOWN_ON_SUSPEND) { + irq_shutdown(desc); + } else { + __disable_irq(desc); + + /* + * Hardware which has no wakeup source configuration facility + * requires that the non wakeup interrupts are masked at the + * chip level. The chip implementation indicates that with + * IRQCHIP_MASK_ON_SUSPEND. + */ + if (irq_desc_get_chip(desc)->flags & IRQCHIP_MASK_ON_SUSPEND) + mask_irq(desc); + } return true; } @@ -152,7 +161,11 @@ static void resume_irq(struct irq_desc *desc) irq_state_set_masked(desc); resume: desc->istate &= ~IRQS_SUSPENDED; - __enable_irq(desc); + + if (irq_desc_get_chip(desc)->flags & IRQCHIP_SHUTDOWN_ON_SUSPEND) + __irq_startup(desc); + else + __enable_irq(desc); } static void resume_irqs(bool want_early) From patchwork Tue May 19 23:27:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anchal Agarwal X-Patchwork-Id: 211997 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.4 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE563C433E1 for ; Tue, 19 May 2020 23:28:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8EE8920823 for ; Tue, 19 May 2020 23:28:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="VVjaHDi1" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726898AbgESX2G (ORCPT ); Tue, 19 May 2020 19:28:06 -0400 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:58793 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726178AbgESX2F (ORCPT ); Tue, 19 May 2020 19:28:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1589930883; x=1621466883; h=date:from:to:subject:message-id:references:mime-version: in-reply-to; bh=uqwrgatq91TkK7sGNplkebnhm9CTfApD3Bn8LMVFgwk=; b=VVjaHDi112E57VG/tFGXhmR1d0KHAVV/30ZtVkL6P4vNPZ8/T6Lb5Ag9 Qi7m5gXROCOCQ7XfvNudulZPowHtV54I5/vkMTzMGglO6GK3KB25eOkBU JKo5LTo7WO2hiNAry4feYi6KSCn7TUlNZNhHYMCwvKErszTGBMoawChol I=; IronPort-SDR: 3iRxtVvg/NRGfue+knyqXXV5q/cvhhWNlq9Qv/b65nkzfZBywqzjcqXj7j7ZSwS3nKzzI0ldwW 0MeVqrDokKEQ== X-IronPort-AV: E=Sophos;i="5.73,411,1583193600"; d="scan'208";a="45949651" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2a-e7be2041.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 19 May 2020 23:28:03 +0000 Received: from EX13MTAUEB002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2a-e7be2041.us-west-2.amazon.com (Postfix) with ESMTPS id 7F40BA04A4; Tue, 19 May 2020 23:28:01 +0000 (UTC) Received: from EX13D08UEB001.ant.amazon.com (10.43.60.245) by EX13MTAUEB002.ant.amazon.com (10.43.60.12) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 19 May 2020 23:27:51 +0000 Received: from EX13MTAUEA001.ant.amazon.com (10.43.61.82) by EX13D08UEB001.ant.amazon.com (10.43.60.245) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 19 May 2020 23:27:51 +0000 Received: from dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (172.22.96.68) by mail-relay.amazon.com (10.43.61.243) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Tue, 19 May 2020 23:27:51 +0000 Received: by dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (Postfix, from userid 4335130) id ED41340712; Tue, 19 May 2020 23:27:50 +0000 (UTC) Date: Tue, 19 May 2020 23:27:50 +0000 From: Anchal Agarwal To: , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH 06/12] xen-blkfront: add callbacks for PM suspend and hibernation Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-pm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org From: Munehisa Kamata S4 power transition states are much different than xen suspend/resume. Former is visible to the guest and frontend drivers should be aware of the state transitions and should be able to take appropriate actions when needed. In transition to S4 we need to make sure that at least all the in-flight blkif requests get completed, since they probably contain bits of the guest's memory image and that's not going to get saved any other way. Hence, re-issuing of in-flight requests as in case of xen resume will not work here. This is in contrast to xen-suspend where we need to freeze with as little processing as possible to avoid dirtying RAM late in the migration cycle and we know that in-flight data can wait. Add freeze, thaw and restore callbacks for PM suspend and hibernation support. All frontend drivers that needs to use PM_HIBERNATION/PM_SUSPEND events, need to implement these xenbus_driver callbacks. The freeze handler stops block-layer queue and disconnect the frontend from the backend while freeing ring_info and associated resources. Before disconnecting from the backend, we need to prevent any new IO from being queued and wait for existing IO to complete. Freeze/unfreeze of the queues will guarantee that there are no requests in use on the shared ring. However, for sanity we should check state of the ring before disconnecting to make sure that there are no outstanding requests to be processed on the ring. The restore handler re-allocates ring_info, unquiesces and unfreezes the queue and re-connect to the backend, so that rest of the kernel can continue to use the block device transparently. Note:For older backends,if a backend doesn't have commit'12ea729645ace' xen/blkback: unmap all persistent grants when frontend gets disconnected, the frontend may see massive amount of grant table warning when freeing resources. [ 36.852659] deferring g.e. 0xf9 (pfn 0xffffffffffffffff) [ 36.855089] xen:grant_table: WARNING:e.g. 0x112 still in use! In this case, persistent grants would need to be disabled. [Anchal Changelog: Removed timeout/request during blkfront freeze. Reworked the whole patch to work with blk-mq and incorporate upstream's comments] Signed-off-by: Anchal Agarwal Signed-off-by: Munehisa Kamata --- drivers/block/xen-blkfront.c | 122 +++++++++++++++++++++++++++++++++-- 1 file changed, 115 insertions(+), 7 deletions(-) diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c index 3b889ea950c2..464863ed7093 100644 --- a/drivers/block/xen-blkfront.c +++ b/drivers/block/xen-blkfront.c @@ -48,6 +48,8 @@ #include #include #include +#include +#include #include #include @@ -80,6 +82,8 @@ enum blkif_state { BLKIF_STATE_DISCONNECTED, BLKIF_STATE_CONNECTED, BLKIF_STATE_SUSPENDED, + BLKIF_STATE_FREEZING, + BLKIF_STATE_FROZEN }; struct grant { @@ -219,6 +223,7 @@ struct blkfront_info struct list_head requests; struct bio_list bio_list; struct list_head info_list; + struct completion wait_backend_disconnected; }; static unsigned int nr_minors; @@ -1005,6 +1010,7 @@ static int xlvbd_init_blk_queue(struct gendisk *gd, u16 sector_size, info->sector_size = sector_size; info->physical_sector_size = physical_sector_size; blkif_set_queue_limits(info); + init_completion(&info->wait_backend_disconnected); return 0; } @@ -1057,7 +1063,7 @@ static int xen_translate_vdev(int vdevice, int *minor, unsigned int *offset) case XEN_SCSI_DISK5_MAJOR: case XEN_SCSI_DISK6_MAJOR: case XEN_SCSI_DISK7_MAJOR: - *offset = (*minor / PARTS_PER_DISK) + + *offset = (*minor / PARTS_PER_DISK) + ((major - XEN_SCSI_DISK1_MAJOR + 1) * 16) + EMULATED_SD_DISK_NAME_OFFSET; *minor = *minor + @@ -1072,7 +1078,7 @@ static int xen_translate_vdev(int vdevice, int *minor, unsigned int *offset) case XEN_SCSI_DISK13_MAJOR: case XEN_SCSI_DISK14_MAJOR: case XEN_SCSI_DISK15_MAJOR: - *offset = (*minor / PARTS_PER_DISK) + + *offset = (*minor / PARTS_PER_DISK) + ((major - XEN_SCSI_DISK8_MAJOR + 8) * 16) + EMULATED_SD_DISK_NAME_OFFSET; *minor = *minor + @@ -1353,6 +1359,8 @@ static void blkif_free(struct blkfront_info *info, int suspend) unsigned int i; struct blkfront_ring_info *rinfo; + if (info->connected == BLKIF_STATE_FREEZING) + goto free_rings; /* Prevent new requests being issued until we fix things up. */ info->connected = suspend ? BLKIF_STATE_SUSPENDED : BLKIF_STATE_DISCONNECTED; @@ -1360,6 +1368,7 @@ static void blkif_free(struct blkfront_info *info, int suspend) if (info->rq) blk_mq_stop_hw_queues(info->rq); +free_rings: for_each_rinfo(info, rinfo, i) blkif_free_ring(rinfo); @@ -1563,8 +1572,10 @@ static irqreturn_t blkif_interrupt(int irq, void *dev_id) struct blkfront_ring_info *rinfo = (struct blkfront_ring_info *)dev_id; struct blkfront_info *info = rinfo->dev_info; - if (unlikely(info->connected != BLKIF_STATE_CONNECTED)) - return IRQ_HANDLED; + if (unlikely(info->connected != BLKIF_STATE_CONNECTED + && info->connected != BLKIF_STATE_FREEZING)){ + return IRQ_HANDLED; + } spin_lock_irqsave(&rinfo->ring_lock, flags); again: @@ -2027,6 +2038,7 @@ static int blkif_recover(struct blkfront_info *info) unsigned int segs; struct blkfront_ring_info *rinfo; + bool frozen = info->connected == BLKIF_STATE_FROZEN; blkfront_gather_backend_features(info); /* Reset limits changed by blk_mq_update_nr_hw_queues(). */ blkif_set_queue_limits(info); @@ -2048,6 +2060,9 @@ static int blkif_recover(struct blkfront_info *info) kick_pending_request_queues(rinfo); } + if (frozen) + return 0; + list_for_each_entry_safe(req, n, &info->requests, queuelist) { /* Requeue pending requests (flush or discard) */ list_del_init(&req->queuelist); @@ -2364,6 +2379,7 @@ static void blkfront_connect(struct blkfront_info *info) return; case BLKIF_STATE_SUSPENDED: + case BLKIF_STATE_FROZEN: /* * If we are recovering from suspension, we need to wait * for the backend to announce it's features before @@ -2481,12 +2497,36 @@ static void blkback_changed(struct xenbus_device *dev, break; case XenbusStateClosed: - if (dev->state == XenbusStateClosed) + if (dev->state == XenbusStateClosed) { + if (info->connected == BLKIF_STATE_FREEZING) { + blkif_free(info, 0); + info->connected = BLKIF_STATE_FROZEN; + complete(&info->wait_backend_disconnected); + break; + } + break; + } + + /* + * We may somehow receive backend's Closed again while thawing + * or restoring and it causes thawing or restoring to fail. + * Ignore such unexpected state regardless of the backend state. + */ + if (info->connected == BLKIF_STATE_FROZEN) { + dev_dbg(&dev->dev, + "ignore the backend's Closed state: %s", + dev->nodename); + break; + } /* fall through */ case XenbusStateClosing: - if (info) - blkfront_closing(info); + if (info) { + if (info->connected == BLKIF_STATE_FREEZING) + xenbus_frontend_closed(dev); + else + blkfront_closing(info); + } break; } } @@ -2630,6 +2670,71 @@ static void blkif_release(struct gendisk *disk, fmode_t mode) mutex_unlock(&blkfront_mutex); } +static int blkfront_freeze(struct xenbus_device *dev) +{ + unsigned int i; + struct blkfront_info *info = dev_get_drvdata(&dev->dev); + struct blkfront_ring_info *rinfo; + /* This would be reasonable timeout as used in xenbus_dev_shutdown() */ + unsigned int timeout = 5 * HZ; + unsigned long flags; + int err = 0; + + info->connected = BLKIF_STATE_FREEZING; + + blk_mq_freeze_queue(info->rq); + blk_mq_quiesce_queue(info->rq); + + for_each_rinfo(info, rinfo, i) { + /* No more gnttab callback work. */ + gnttab_cancel_free_callback(&rinfo->callback); + /* Flush gnttab callback work. Must be done with no locks held. */ + flush_work(&rinfo->work); + } + + for_each_rinfo(info, rinfo, i) { + spin_lock_irqsave(&rinfo->ring_lock, flags); + if (RING_FULL(&rinfo->ring) + || RING_HAS_UNCONSUMED_RESPONSES(&rinfo->ring)) { + xenbus_dev_error(dev, err, "Hibernation Failed. + The ring is still busy"); + info->connected = BLKIF_STATE_CONNECTED; + spin_unlock_irqrestore(&rinfo->ring_lock, flags); + return -EBUSY; + } + spin_unlock_irqrestore(&rinfo->ring_lock, flags); + } + /* Kick the backend to disconnect */ + xenbus_switch_state(dev, XenbusStateClosing); + + /* + * We don't want to move forward before the frontend is diconnected + * from the backend cleanly. + */ + timeout = wait_for_completion_timeout(&info->wait_backend_disconnected, + timeout); + if (!timeout) { + err = -EBUSY; + xenbus_dev_error(dev, err, "Freezing timed out;" + "the device may become inconsistent state"); + } + + return err; +} + +static int blkfront_restore(struct xenbus_device *dev) +{ + struct blkfront_info *info = dev_get_drvdata(&dev->dev); + int err = 0; + + err = talk_to_blkback(dev, info); + blk_mq_unquiesce_queue(info->rq); + blk_mq_unfreeze_queue(info->rq); + if (!err) + blk_mq_update_nr_hw_queues(&info->tag_set, info->nr_rings); + return err; +} + static const struct block_device_operations xlvbd_block_fops = { .owner = THIS_MODULE, @@ -2653,6 +2758,9 @@ static struct xenbus_driver blkfront_driver = { .resume = blkfront_resume, .otherend_changed = blkback_changed, .is_ready = blkfront_is_ready, + .freeze = blkfront_freeze, + .thaw = blkfront_restore, + .restore = blkfront_restore }; static void purge_persistent_grants(struct blkfront_info *info) From patchwork Tue May 19 23:28:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anchal Agarwal X-Patchwork-Id: 211996 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.4 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C31F4C433E1 for ; Tue, 19 May 2020 23:28:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8B54320826 for ; Tue, 19 May 2020 23:28:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="rfzpMep9" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726178AbgESX2l (ORCPT ); Tue, 19 May 2020 19:28:41 -0400 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:29867 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726536AbgESX2l (ORCPT ); Tue, 19 May 2020 19:28:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1589930920; x=1621466920; h=date:from:to:subject:message-id:references:mime-version: in-reply-to; bh=8oFwSiGiXazezHFt6ilNhdZTejmHfEW4a5eyQv6e/gQ=; b=rfzpMep9LFWBd/N2D7eR5ym3SsoZl+bxPrEntwVEPAChnja6g+2geCSk fdp3E1eu6JLnV5RzP9iFK8bRlWiP0XuLfIwrr1n5anQ0NBan+zceib0bS qmogj+cpla7fCMQFwOuUoYSZSyb099YFk6LOByF1mw4TFbCbxSm8NW8+r o=; IronPort-SDR: ypi20ESB/vUTFj2KU/00Es79bg7nB7GrK8X0Zx694qkzKCpMWUQK1Y3nZ8y0w9sxgpu4l6EFGx eREArqyAyGtA== X-IronPort-AV: E=Sophos;i="5.73,411,1583193600"; d="scan'208";a="31067058" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-c5104f52.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 19 May 2020 23:28:37 +0000 Received: from EX13MTAUWA001.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2a-c5104f52.us-west-2.amazon.com (Postfix) with ESMTPS id 03FA7A1E06; Tue, 19 May 2020 23:28:35 +0000 (UTC) Received: from EX13D07UWA001.ant.amazon.com (10.43.160.145) by EX13MTAUWA001.ant.amazon.com (10.43.160.58) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 19 May 2020 23:28:30 +0000 Received: from EX13MTAUWA001.ant.amazon.com (10.43.160.58) by EX13D07UWA001.ant.amazon.com (10.43.160.145) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 19 May 2020 23:28:30 +0000 Received: from dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (172.22.96.68) by mail-relay.amazon.com (10.43.160.118) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Tue, 19 May 2020 23:28:30 +0000 Received: by dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (Postfix, from userid 4335130) id 8B45240712; Tue, 19 May 2020 23:28:30 +0000 (UTC) Date: Tue, 19 May 2020 23:28:30 +0000 From: Anchal Agarwal To: , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH 08/12] xen/time: introduce xen_{save,restore}_steal_clock Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-pm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org From: Munehisa Kamata Currently, steal time accounting code in scheduler expects steal clock callback to provide monotonically increasing value. If the accounting code receives a smaller value than previous one, it uses a negative value to calculate steal time and results in incorrectly updated idle and steal time accounting. This breaks userspace tools which read /proc/stat. top - 08:05:35 up 2:12, 3 users, load average: 0.00, 0.07, 0.23 Tasks: 80 total, 1 running, 79 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,30100.0%id, 0.0%wa, 0.0%hi, 0.0%si,-1253874204672.0%st This can actually happen when a Xen PVHVM guest gets restored from hibernation, because such a restored guest is just a fresh domain from Xen perspective and the time information in runstate info starts over from scratch. This patch introduces xen_save_steal_clock() which saves current values in runstate info into per-cpu variables. Its couterpart, xen_restore_steal_clock(), sets offset if it found the current values in runstate info are smaller than previous ones. xen_steal_clock() is also modified to use the offset to ensure that scheduler only sees monotonically increasing number. Signed-off-by: Munehisa Kamata Signed-off-by: Anchal Agarwal --- drivers/xen/time.c | 29 ++++++++++++++++++++++++++++- include/xen/xen-ops.h | 2 ++ 2 files changed, 30 insertions(+), 1 deletion(-) diff --git a/drivers/xen/time.c b/drivers/xen/time.c index 0968859c29d0..3560222cc0dd 100644 --- a/drivers/xen/time.c +++ b/drivers/xen/time.c @@ -23,6 +23,9 @@ static DEFINE_PER_CPU(struct vcpu_runstate_info, xen_runstate); static DEFINE_PER_CPU(u64[4], old_runstate_time); +static DEFINE_PER_CPU(u64, xen_prev_steal_clock); +static DEFINE_PER_CPU(u64, xen_steal_clock_offset); + /* return an consistent snapshot of 64-bit time/counter value */ static u64 get64(const u64 *p) { @@ -149,7 +152,7 @@ bool xen_vcpu_stolen(int vcpu) return per_cpu(xen_runstate, vcpu).state == RUNSTATE_runnable; } -u64 xen_steal_clock(int cpu) +static u64 __xen_steal_clock(int cpu) { struct vcpu_runstate_info state; @@ -157,6 +160,30 @@ u64 xen_steal_clock(int cpu) return state.time[RUNSTATE_runnable] + state.time[RUNSTATE_offline]; } +u64 xen_steal_clock(int cpu) +{ + return __xen_steal_clock(cpu) + per_cpu(xen_steal_clock_offset, cpu); +} + +void xen_save_steal_clock(int cpu) +{ + per_cpu(xen_prev_steal_clock, cpu) = xen_steal_clock(cpu); +} + +void xen_restore_steal_clock(int cpu) +{ + u64 steal_clock = __xen_steal_clock(cpu); + + if (per_cpu(xen_prev_steal_clock, cpu) > steal_clock) { + /* Need to update the offset */ + per_cpu(xen_steal_clock_offset, cpu) = + per_cpu(xen_prev_steal_clock, cpu) - steal_clock; + } else { + /* Avoid unnecessary steal clock warp */ + per_cpu(xen_steal_clock_offset, cpu) = 0; + } +} + void xen_setup_runstate_info(int cpu) { struct vcpu_register_runstate_memory_area area; diff --git a/include/xen/xen-ops.h b/include/xen/xen-ops.h index 89b1e88712d6..74fb5eb3aad8 100644 --- a/include/xen/xen-ops.h +++ b/include/xen/xen-ops.h @@ -37,6 +37,8 @@ void xen_time_setup_guest(void); void xen_manage_runstate_time(int action); void xen_get_runstate_snapshot(struct vcpu_runstate_info *res); u64 xen_steal_clock(int cpu); +void xen_save_steal_clock(int cpu); +void xen_restore_steal_clock(int cpu); int xen_setup_shutdown_event(void); From patchwork Tue May 19 23:29:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anchal Agarwal X-Patchwork-Id: 211995 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.4 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E0BCC433DF for ; Tue, 19 May 2020 23:30:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EAA5D20823 for ; Tue, 19 May 2020 23:30:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="YygAM4rx" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728508AbgESXaD (ORCPT ); Tue, 19 May 2020 19:30:03 -0400 Received: from smtp-fw-9101.amazon.com ([207.171.184.25]:40836 "EHLO smtp-fw-9101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728432AbgESXaC (ORCPT ); Tue, 19 May 2020 19:30:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1589931002; x=1621467002; h=date:from:to:subject:message-id:references:mime-version: in-reply-to; bh=fitAmmYR+siRHznU7MJYKk9c4Tw82/jy6e5chPwlJLw=; b=YygAM4rxhevUTIpwXDea+e1znhZ7YOkK+Xbwgvyk2RpqEX1HneGo34qH FUR9cIh77T27+TwpzuT/FfNlYH1JbfxCVlwArylM/KP1tL7hgMuWok+zE H0QLewXYVd0zlqr1NelNitMrBpv7JsNH4728d9RmELnUU2LHe0srjoA2i M=; IronPort-SDR: Wl3dSMwhpOL5k2tH39DJuBMefCB10ew0UQM8WlE40AAwlaQq7gFVjSYFqv7rMMiJuNURig22t8 62l/Qk1vmD/Q== X-IronPort-AV: E=Sophos;i="5.73,411,1583193600"; d="scan'208";a="36216330" Received: from sea32-co-svc-lb4-vlan2.sea.corp.amazon.com (HELO email-inbound-relay-2b-859fe132.us-west-2.amazon.com) ([10.47.23.34]) by smtp-border-fw-out-9101.sea19.amazon.com with ESMTP; 19 May 2020 23:30:01 +0000 Received: from EX13MTAUWB001.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2b-859fe132.us-west-2.amazon.com (Postfix) with ESMTPS id C56E42225D4; Tue, 19 May 2020 23:29:59 +0000 (UTC) Received: from EX13D07UWB004.ant.amazon.com (10.43.161.196) by EX13MTAUWB001.ant.amazon.com (10.43.161.207) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 19 May 2020 23:29:52 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D07UWB004.ant.amazon.com (10.43.161.196) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 19 May 2020 23:29:52 +0000 Received: from dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (172.22.96.68) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Tue, 19 May 2020 23:29:52 +0000 Received: by dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com (Postfix, from userid 4335130) id 847C740712; Tue, 19 May 2020 23:29:52 +0000 (UTC) Date: Tue, 19 May 2020 23:29:52 +0000 From: Anchal Agarwal To: , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH 12/12] PM / hibernate: update the resume offset on SNAPSHOT_SET_SWAP_AREA Message-ID: <40de33ca69c0d3bcf8c827862768ae5d399698d6.1589926004.git.anchalag@amazon.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-pm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org From: Aleksei Besogonov The SNAPSHOT_SET_SWAP_AREA is supposed to be used to set the hibernation offset on a running kernel to enable hibernating to a swap file. However, it doesn't actually update the swsusp_resume_block variable. As a result, the hibernation fails at the last step (after all the data is written out) in the validation of the swap signature in mark_swapfiles(). Before this patch, the command line processing was the only place where swsusp_resume_block was set. [Changelog: Resolved patch conflict as code fragmented to snapshot_set_swap_area] Signed-off-by: Aleksei Besogonov Signed-off-by: Munehisa Kamata Signed-off-by: Anchal Agarwal --- kernel/power/user.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/kernel/power/user.c b/kernel/power/user.c index 7959449765d9..1afa1f0a223e 100644 --- a/kernel/power/user.c +++ b/kernel/power/user.c @@ -235,8 +235,12 @@ static int snapshot_set_swap_area(struct snapshot_data *data, return -EINVAL; } data->swap = swap_type_of(swdev, offset, NULL); - if (data->swap < 0) + if (data->swap < 0) { return -ENODEV; + } else { + swsusp_resume_device = swdev; + swsusp_resume_block = offset; + } return 0; }