From patchwork Tue Sep 6 22:23:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bjorn Helgaas X-Patchwork-Id: 603252 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F037EC6FA86 for ; Tue, 6 Sep 2022 22:24:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229842AbiIFWYA (ORCPT ); Tue, 6 Sep 2022 18:24:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48130 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229810AbiIFWX7 (ORCPT ); Tue, 6 Sep 2022 18:23:59 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 05851A9C27; Tue, 6 Sep 2022 15:23:59 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A2C7461595; Tue, 6 Sep 2022 22:23:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C7980C433C1; Tue, 6 Sep 2022 22:23:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1662503038; bh=RT3RQzwcZmsdKMDfRVB08SNXYiMqaAszRBdVwIagBPc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KoWQN+DtFhGgaXGAEecu3YKNsWQGG8N0v6iE7sATpBchhXm81rsAlv98qJNIh9Wpd cmA+qKlBeGudRRi5UO7LtHjB7er9Ovu6uwvHXrMRCLp6SYZlUcyRoGjQ8skCEsjALw tQbrfA/dG5mD9GFkmLvzmats17r8MvOPMTIcrKAVUCHNoOTRGBkbnfHqwx2e4rlPVI caS99Ak3oYEFWQbHW1M7RUOaoi+zUItiul2iXoMdgpAOyUqIY2qF3IWBBzWbTbB32P sZZeciu0P1DXzhqK9+4CPNqX/ByvEtKyqCEAE2Fe7EKYaZ7S9hn/5IqreTJ/IPyJB9 10Uu0OAAIPx0g== From: Bjorn Helgaas To: Kai-Heng Feng , Rajvi Jingar , "Rafael J . Wysocki" Cc: Koba Ko , Mika Westerberg , "David E . Box" , Sathyanarayanan Kuppuswamy , linux-pci@vger.kernel.org, linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org, Bjorn Helgaas Subject: [PATCH v3 01/10] PCI/PTM: Preserve PTM Root Select Date: Tue, 6 Sep 2022 17:23:42 -0500 Message-Id: <20220906222351.64760-2-helgaas@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220906222351.64760-1-helgaas@kernel.org> References: <20220906222351.64760-1-helgaas@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org From: Bjorn Helgaas When disabling PTM, there's no need to clear the Root Select bit. We disable PTM during suspend, and we want to re-enable it during resume. Clearing Root Select here makes re-enabling more complicated. Per PCIe r6.0, sec 7.9.15.3, "When set, if the PTM Enable bit is also Set, this Time Source is the PTM Root," so if PTM Enable is cleared, the value of Root Select should be irrelevant. Preserve Root Select to simplify re-enabling PTM. Signed-off-by: Bjorn Helgaas Cc: David E. Box --- drivers/pci/pcie/ptm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/pci/pcie/ptm.c b/drivers/pci/pcie/ptm.c index 368a254e3124..b6a417247ce3 100644 --- a/drivers/pci/pcie/ptm.c +++ b/drivers/pci/pcie/ptm.c @@ -42,7 +42,7 @@ void pci_disable_ptm(struct pci_dev *dev) return; pci_read_config_word(dev, ptm + PCI_PTM_CTRL, &ctrl); - ctrl &= ~(PCI_PTM_CTRL_ENABLE | PCI_PTM_CTRL_ROOT); + ctrl &= ~PCI_PTM_CTRL_ENABLE; pci_write_config_word(dev, ptm + PCI_PTM_CTRL, ctrl); } From patchwork Tue Sep 6 22:23:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bjorn Helgaas X-Patchwork-Id: 603251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D91EFC38145 for ; Tue, 6 Sep 2022 22:24:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230002AbiIFWYK (ORCPT ); Tue, 6 Sep 2022 18:24:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48452 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229886AbiIFWYI (ORCPT ); Tue, 6 Sep 2022 18:24:08 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6320CAA37A; Tue, 6 Sep 2022 15:24:04 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id DA27DB81AAC; Tue, 6 Sep 2022 22:24:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5E235C43470; Tue, 6 Sep 2022 22:24:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1662503041; bh=9D8mgVY5gHcE6/P2q6z5Fluh/zLBUAmVl7NPUlk2Lew=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UJNy3ohIHRZmJ7ExidmVPQszAoQxTHEG0ha6tBNlcF6B61WQ1ytjO30zoGmWeQg8v sliC9dTUVQCvcMju7iHIapWq3TD0CunwOeCCZtCdwmfI5SWKcblCxwl0iKGCRtRA2o rtqLmbxjE4V4z2XmDfBw2kYTAUiaQX7JtOSRkR1zpEVDRo+rz90N+e97fgJBCymMs2 /m7Df03YMEV7HSqkoE8Ilgwq+6v163+Rmkfws3Q54X81M6JBzdgQb0aISzkof5/xRC 0yYya7s7J5uOwtpDiFlAqRJZA+Mmhi1z0nTXUuIFIm44jlIM3HumNDB8LVDWNxRqRw zSiDW4+4ytdwA== From: Bjorn Helgaas To: Kai-Heng Feng , Rajvi Jingar , "Rafael J . Wysocki" Cc: Koba Ko , Mika Westerberg , "David E . Box" , Sathyanarayanan Kuppuswamy , linux-pci@vger.kernel.org, linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org, Bjorn Helgaas Subject: [PATCH v3 03/10] PCI/PTM: Add pci_upstream_ptm() helper Date: Tue, 6 Sep 2022 17:23:44 -0500 Message-Id: <20220906222351.64760-4-helgaas@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220906222351.64760-1-helgaas@kernel.org> References: <20220906222351.64760-1-helgaas@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org From: Bjorn Helgaas PTM requires an unbroken path of PTM-supporting devices between the PTM Root and the ultimate PTM Requester, but if a Switch supports PTM, only the Upstream Port can have a PTM Capability; the Downstream Ports do not. Previously we copied the PTM configuration from the Switch Upstream Port to the Downstream Ports so dev->ptm_enabled for any device implied that all the upstream devices support PTM. Instead of making it look like Downstream Ports have their own PTM config, add pci_upstream_ptm(), which returns the upstream device that has a PTM Capability (either a Root Port or a Switch Upstream Port). Signed-off-by: Bjorn Helgaas --- drivers/pci/pcie/ptm.c | 39 +++++++++++++++++++++++++-------------- 1 file changed, 25 insertions(+), 14 deletions(-) diff --git a/drivers/pci/pcie/ptm.c b/drivers/pci/pcie/ptm.c index 6ac7ff48be57..8729c3e452ee 100644 --- a/drivers/pci/pcie/ptm.c +++ b/drivers/pci/pcie/ptm.c @@ -76,6 +76,29 @@ void pci_restore_ptm_state(struct pci_dev *dev) pci_write_config_word(dev, ptm + PCI_PTM_CTRL, *cap); } +/* + * If the next upstream device supports PTM, return it; otherwise return + * NULL. PTM Messages are local, so both link partners must support it. + */ +static struct pci_dev *pci_upstream_ptm(struct pci_dev *dev) +{ + struct pci_dev *ups = pci_upstream_bridge(dev); + + /* + * Switch Downstream Ports are not permitted to have a PTM + * capability; their PTM behavior is controlled by the Upstream + * Port (PCIe r5.0, sec 7.9.16), so if the upstream bridge is a + * Switch Downstream Port, look up one more level. + */ + if (ups && pci_pcie_type(ups) == PCI_EXP_TYPE_DOWNSTREAM) + ups = pci_upstream_bridge(ups); + + if (ups && ups->ptm_cap) + return ups; + + return NULL; +} + void pci_ptm_init(struct pci_dev *dev) { int ptm; @@ -95,19 +118,6 @@ void pci_ptm_init(struct pci_dev *dev) pci_pcie_type(dev) == PCI_EXP_TYPE_RC_END)) return; - /* - * Switch Downstream Ports are not permitted to have a PTM - * capability; their PTM behavior is controlled by the Upstream - * Port (PCIe r5.0, sec 7.9.16). - */ - ups = pci_upstream_bridge(dev); - if (pci_pcie_type(dev) == PCI_EXP_TYPE_DOWNSTREAM && - ups && ups->ptm_enabled) { - dev->ptm_granularity = ups->ptm_granularity; - dev->ptm_enabled = 1; - return; - } - ptm = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_PTM); if (!ptm) return; @@ -124,6 +134,7 @@ void pci_ptm_init(struct pci_dev *dev) * the spec recommendation (PCIe r3.1, sec 7.32.3), select the * furthest upstream Time Source as the PTM Root. */ + ups = pci_upstream_ptm(dev); if (ups && ups->ptm_enabled) { ctrl = PCI_PTM_CTRL_ENABLE; if (ups->ptm_granularity == 0) @@ -173,7 +184,7 @@ int pci_enable_ptm(struct pci_dev *dev, u8 *granularity) * associate the endpoint with a time source. */ if (pci_pcie_type(dev) == PCI_EXP_TYPE_ENDPOINT) { - ups = pci_upstream_bridge(dev); + ups = pci_upstream_ptm(dev); if (!ups || !ups->ptm_enabled) return -EINVAL; From patchwork Tue Sep 6 22:23:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bjorn Helgaas X-Patchwork-Id: 603250 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D080BC38145 for ; Tue, 6 Sep 2022 22:24:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230072AbiIFWYS (ORCPT ); Tue, 6 Sep 2022 18:24:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48478 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229916AbiIFWYL (ORCPT ); Tue, 6 Sep 2022 18:24:11 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6BB9BAA3E4; Tue, 6 Sep 2022 15:24:08 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 63BF2B81AAA; Tue, 6 Sep 2022 22:24:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EB9A0C433D6; Tue, 6 Sep 2022 22:24:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1662503045; bh=cD6LaKtHHzCoaSefmit3nfjumtZ0PHa2t3oTDXVD7L0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vPfusRYKGLk3TY22O1v2vmVECesOBY1pyCymDRrMmhZTxwwMfxH0QL9+8jceQ4FsD 3lVV+/x4o0Awsf4PzqYz3/b7D2BUIOYFhLpK6R1lc0QS3i03lTgPPz4+OMTwVsCHqU CFsj66y4Oevjj+CKseAunSovND2QHNJJxr4izQZbVX6AeIN7/+D6xaJt80fLmmcR1f 3QD4Gflny2ftp4Rh3v1BTfXurPs1yUA5DNO5+5NpQxwg9n1rAZbKw54nhk2sx9WJBz tmmpzwLx6snG5lkhUwWTzSArO9FqPB691kJxHlsv5jHrVRYxjmlZzhExNuJ2h7zuc4 4SAE+HtCYM/sw== From: Bjorn Helgaas To: Kai-Heng Feng , Rajvi Jingar , "Rafael J . Wysocki" Cc: Koba Ko , Mika Westerberg , "David E . Box" , Sathyanarayanan Kuppuswamy , linux-pci@vger.kernel.org, linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org, Bjorn Helgaas Subject: [PATCH v3 05/10] PCI/PTM: Add pci_disable_ptm() wrapper Date: Tue, 6 Sep 2022 17:23:46 -0500 Message-Id: <20220906222351.64760-6-helgaas@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220906222351.64760-1-helgaas@kernel.org> References: <20220906222351.64760-1-helgaas@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org From: Bjorn Helgaas Implement pci_disable_ptm() as a wrapper around an internal __pci_disable_ptm() that we can use during suspend to disable PTM without clearing dev->ptm_enabled. A future commit will re-enable PTM during resume when dev->ptm_enabled is set. Export pci_disable_ptm(), which *does* clear dev->ptm_enabled, for use by drivers that want to disable PTM. Signed-off-by: Bjorn Helgaas --- drivers/pci/pci.h | 2 -- drivers/pci/pcie/ptm.c | 9 ++++++++- include/linux/pci.h | 2 ++ 3 files changed, 10 insertions(+), 3 deletions(-) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 785f31086313..91a465460d0f 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -507,11 +507,9 @@ static inline int pci_iov_bus_range(struct pci_bus *bus) #ifdef CONFIG_PCIE_PTM void pci_save_ptm_state(struct pci_dev *dev); void pci_restore_ptm_state(struct pci_dev *dev); -void pci_disable_ptm(struct pci_dev *dev); #else static inline void pci_save_ptm_state(struct pci_dev *dev) { } static inline void pci_restore_ptm_state(struct pci_dev *dev) { } -static inline void pci_disable_ptm(struct pci_dev *dev) { } #endif unsigned long pci_cardbus_resource_alignment(struct resource *); diff --git a/drivers/pci/pcie/ptm.c b/drivers/pci/pcie/ptm.c index ac51cd84793f..762299984469 100644 --- a/drivers/pci/pcie/ptm.c +++ b/drivers/pci/pcie/ptm.c @@ -29,7 +29,7 @@ static void pci_ptm_info(struct pci_dev *dev) dev->ptm_root ? " (root)" : "", clock_desc); } -void pci_disable_ptm(struct pci_dev *dev) +static void __pci_disable_ptm(struct pci_dev *dev) { int ptm = dev->ptm_cap; u16 ctrl; @@ -42,6 +42,13 @@ void pci_disable_ptm(struct pci_dev *dev) pci_write_config_word(dev, ptm + PCI_PTM_CTRL, ctrl); } +void pci_disable_ptm(struct pci_dev *dev) +{ + __pci_disable_ptm(dev); + dev->ptm_enabled = 0; +} +EXPORT_SYMBOL(pci_disable_ptm); + void pci_save_ptm_state(struct pci_dev *dev) { int ptm = dev->ptm_cap; diff --git a/include/linux/pci.h b/include/linux/pci.h index 54be939023a3..cb5f796e3319 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -1678,10 +1678,12 @@ bool pci_ats_disabled(void); #ifdef CONFIG_PCIE_PTM int pci_enable_ptm(struct pci_dev *dev, u8 *granularity); +void pci_disable_ptm(struct pci_dev *dev); bool pcie_ptm_enabled(struct pci_dev *dev); #else static inline int pci_enable_ptm(struct pci_dev *dev, u8 *granularity) { return -EINVAL; } +static inline void pci_disable_ptm(struct pci_dev *dev) { } static inline bool pcie_ptm_enabled(struct pci_dev *dev) { return false; } #endif From patchwork Tue Sep 6 22:23:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bjorn Helgaas X-Patchwork-Id: 603249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 956EDC6FA89 for ; Tue, 6 Sep 2022 22:24:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230104AbiIFWYd (ORCPT ); Tue, 6 Sep 2022 18:24:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48466 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229995AbiIFWYN (ORCPT ); Tue, 6 Sep 2022 18:24:13 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4EB26ABF0E; Tue, 6 Sep 2022 15:24:10 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 30606616F5; Tue, 6 Sep 2022 22:24:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6CA26C433C1; Tue, 6 Sep 2022 22:24:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1662503048; bh=GZ1t3eEPe9WAKIeqnRdMruDBNlk/eilfZ8AziF6/6PI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DeD/nKSAZsxT6ztAhrLodFx5OfRGCevqJTbqV9M0/SXRzVpdVIJkCYT9MOMs6v5rQ 9fgsbljrwM7837zZsdZsuYe0kfiNoZDsq1QyXojGodg+FmI7V7w1YEY5ntG4XF2HQz yikQXO7zGnnqozfDYCVe0YZm5Dd0pThK1+kPO40FztEjlesNUPvM+W8wD5+zabncp7 DJuKd9X8qxaDiiemhtP6P4mdxplnjkZhjjcz0M+MlMt5SLMxw0+HuGY9go68I61/02 kx6z22rlodkosTtmoV/p0pAvpmyfhURIgG4ZYfHbOympDprTJJ6lV4B/2Lps/aMcIG lHyeBSD3YLeVQ== From: Bjorn Helgaas To: Kai-Heng Feng , Rajvi Jingar , "Rafael J . Wysocki" Cc: Koba Ko , Mika Westerberg , "David E . Box" , Sathyanarayanan Kuppuswamy , linux-pci@vger.kernel.org, linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org, Bjorn Helgaas Subject: [PATCH v3 07/10] PCI/PTM: Add suspend/resume Date: Tue, 6 Sep 2022 17:23:48 -0500 Message-Id: <20220906222351.64760-8-helgaas@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220906222351.64760-1-helgaas@kernel.org> References: <20220906222351.64760-1-helgaas@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org From: Bjorn Helgaas --- drivers/pci/pci.c | 4 ++-- drivers/pci/pci.h | 4 ++++ drivers/pci/pcie/ptm.c | 15 +++++++++++++++ 3 files changed, 21 insertions(+), 2 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 95bc329e74c0..83818f81577d 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -2714,7 +2714,7 @@ int pci_prepare_to_sleep(struct pci_dev *dev) * lower-power idle state as a whole. */ if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT) - pci_disable_ptm(dev); + pci_suspend_ptm(dev); pci_enable_wake(dev, target_state, wakeup); @@ -2772,7 +2772,7 @@ int pci_finish_runtime_suspend(struct pci_dev *dev) * lower-power idle state as a whole. */ if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT) - pci_disable_ptm(dev); + pci_suspend_ptm(dev); __pci_enable_wake(dev, target_state, pci_dev_run_wake(dev)); diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 91a465460d0f..ce4a277e3f41 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -507,9 +507,13 @@ static inline int pci_iov_bus_range(struct pci_bus *bus) #ifdef CONFIG_PCIE_PTM void pci_save_ptm_state(struct pci_dev *dev); void pci_restore_ptm_state(struct pci_dev *dev); +void pci_suspend_ptm(struct pci_dev *dev); +void pci_resume_ptm(struct pci_dev *dev); #else static inline void pci_save_ptm_state(struct pci_dev *dev) { } static inline void pci_restore_ptm_state(struct pci_dev *dev) { } +static inline void pci_suspend_ptm(struct pci_dev *dev) { } +static inline void pci_resume_ptm(struct pci_dev *dev) { } #endif unsigned long pci_cardbus_resource_alignment(struct resource *); diff --git a/drivers/pci/pcie/ptm.c b/drivers/pci/pcie/ptm.c index 4a9f045126ca..8ac844212436 100644 --- a/drivers/pci/pcie/ptm.c +++ b/drivers/pci/pcie/ptm.c @@ -198,6 +198,21 @@ int pci_enable_ptm(struct pci_dev *dev, u8 *granularity) } EXPORT_SYMBOL(pci_enable_ptm); +/* + * Disable PTM, but leave dev->ptm_enabled so we silently re-enable it on + * resume. + */ +void pci_suspend_ptm(struct pci_dev *dev) +{ + __pci_disable_ptm(dev); +} + +void pci_resume_ptm(struct pci_dev *dev) +{ + if (dev->ptm_enabled) + __pci_enable_ptm(dev); +} + bool pcie_ptm_enabled(struct pci_dev *dev) { if (!dev) From patchwork Tue Sep 6 22:23:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bjorn Helgaas X-Patchwork-Id: 603248 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28D13ECAAA1 for ; Tue, 6 Sep 2022 22:24:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230168AbiIFWYr (ORCPT ); Tue, 6 Sep 2022 18:24:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49482 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230080AbiIFWY0 (ORCPT ); Tue, 6 Sep 2022 18:24:26 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9EE80AE86F; Tue, 6 Sep 2022 15:24:15 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A378861713; Tue, 6 Sep 2022 22:24:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DAE96C433C1; Tue, 6 Sep 2022 22:24:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1662503054; bh=8zBtTybYX8wofqg/4M5b5Eesn1tmGcj3alXYa5B6SS8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jTF2mls+dk/XL02WdDwayJCVNRSAdDhBURu3wckxOru9a5/wQKNLb4P6TPbmkSIKh bzQb8xJ/qYcu56kRUVpLaWHZLUwzo8iFfG6qZ6etUMrP0R7idmfxLO/yxpNEAlPebj XgEpTnQgLZDegIE+t+qwm5vozzOLCSlS7KORLibCFL8GZ+0czg7iQEr+fqKN5GtM7N wgCtCDsDJE5DdbD6/kaEQGCUhhXaJLnXULnB2hZ8xqYMHZ/OD9rZagdEY7ZfdeqPJ0 INRsniFzvKiQE7cXiouUc4v7XEW7341ma0ApVVjqFnw0ZAt5kDLcTajzAXGTxd7GMA YRVqMpdEIuKyw== From: Bjorn Helgaas To: Kai-Heng Feng , Rajvi Jingar , "Rafael J . Wysocki" Cc: Koba Ko , Mika Westerberg , "David E . Box" , Sathyanarayanan Kuppuswamy , linux-pci@vger.kernel.org, linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org, Bjorn Helgaas Subject: [PATCH v3 10/10] PCI/PM: Always disable PTM for all devices during suspend Date: Tue, 6 Sep 2022 17:23:51 -0500 Message-Id: <20220906222351.64760-11-helgaas@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220906222351.64760-1-helgaas@kernel.org> References: <20220906222351.64760-1-helgaas@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org From: Bjorn Helgaas We want to disable PTM on Root Ports because that allows some chips, e.g., Intel mobile chips since Coffee Lake, to enter a lower-power PM state. That means we also have to disable PTM on downstream devices. PCIe r6.0, sec 2.2.8, recommends that functions support generation of messages in non-D0 states, so we have to assume Switch Upstream Ports or Endpoints may send PTM Requests while in D1, D2, and D3hot. A PTM message received by a Downstream Port (including a Root Port) with PTM disabled must be treated as an Unsupported Request (sec 6.21.3). PTM was previously disabled only for Root Ports, and it was disabled in pci_prepare_to_sleep(), which is not called at all if a driver supports legacy PM or does its own state saving. Instead, disable PTM early in pci_pm_suspend() and pci_pm_runtime_suspend() so we do it in all cases. Previously PTM was disabled *after* saving device state, so the state restore on resume automatically re-enabled it. Since we now disable PTM *before* saving state, we must explicitly re-enable it in pci_pm_resume() and pci_pm_runtime_resume(). Here's a sample of errors that occur when PTM is disabled only on the Root Port. With this topology: 0000:00:1d.0 Root Port to [bus 08-71] 0000:08:00.0 Switch Upstream Port to [bus 09-71] Kai-Heng reported errors like this: pcieport 0000:00:1d.0: [20] UnsupReq (First) pcieport 0000:00:1d.0: AER: TLP Header: 34000000 08000052 00000000 00000000 Decoding TLP header 0x34...... (0011 0100b) and 0x08000052: Fmt 001b 4 DW header, no data Type 1 0100b Msg (Local - Terminate at Receiver) Requester ID 0x0800 Bus 08 Devfn 00.0 Message Code 0x52 0101 0010b PTM Request The 00:1d.0 Root Port logged an Unsupported Request error when it received a PTM Request with Requester ID 08:00.0. Fixes: a697f072f5da ("PCI: Disable PTM during suspend to save power") Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=215453 Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=216210 Reported-by: Kai-Heng Feng Signed-off-by: Bjorn Helgaas --- drivers/pci/pci-driver.c | 11 +++++++++++ drivers/pci/pci.c | 28 ++-------------------------- 2 files changed, 13 insertions(+), 26 deletions(-) diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c index 2815922ac525..107d77f3c846 100644 --- a/drivers/pci/pci-driver.c +++ b/drivers/pci/pci-driver.c @@ -774,6 +774,12 @@ static int pci_pm_suspend(struct device *dev) pci_dev->skip_bus_pm = false; + /* + * Disabling PTM allows some systems, e.g., Intel mobile chips + * since Coffee Lake, to enter a lower-power PM state. + */ + pci_suspend_ptm(pci_dev); + if (pci_has_legacy_pm_support(pci_dev)) return pci_legacy_suspend(dev, PMSG_SUSPEND); @@ -982,6 +988,8 @@ static int pci_pm_resume(struct device *dev) if (pci_dev->state_saved) pci_restore_standard_config(pci_dev); + pci_resume_ptm(pci_dev); + if (pci_has_legacy_pm_support(pci_dev)) return pci_legacy_resume(dev); @@ -1269,6 +1277,8 @@ static int pci_pm_runtime_suspend(struct device *dev) pci_power_t prev = pci_dev->current_state; int error; + pci_suspend_ptm(pci_dev); + /* * If pci_dev->driver is not set (unbound), we leave the device in D0, * but it may go to D3cold when the bridge above it runtime suspends. @@ -1330,6 +1340,7 @@ static int pci_pm_runtime_resume(struct device *dev) * D3cold when the bridge above it runtime suspended. */ pci_pm_default_resume_early(pci_dev); + pci_resume_ptm(pci_dev); if (!pci_dev->driver) return 0; diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 83818f81577d..107afa0a5b03 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -2706,24 +2706,12 @@ int pci_prepare_to_sleep(struct pci_dev *dev) if (target_state == PCI_POWER_ERROR) return -EIO; - /* - * There are systems (for example, Intel mobile chips since Coffee - * Lake) where the power drawn while suspended can be significantly - * reduced by disabling PTM on PCIe root ports as this allows the - * port to enter a lower-power PM state and the SoC to reach a - * lower-power idle state as a whole. - */ - if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT) - pci_suspend_ptm(dev); - pci_enable_wake(dev, target_state, wakeup); error = pci_set_power_state(dev, target_state); - if (error) { + if (error) pci_enable_wake(dev, target_state, false); - pci_restore_ptm_state(dev); - } return error; } @@ -2764,24 +2752,12 @@ int pci_finish_runtime_suspend(struct pci_dev *dev) if (target_state == PCI_POWER_ERROR) return -EIO; - /* - * There are systems (for example, Intel mobile chips since Coffee - * Lake) where the power drawn while suspended can be significantly - * reduced by disabling PTM on PCIe root ports as this allows the - * port to enter a lower-power PM state and the SoC to reach a - * lower-power idle state as a whole. - */ - if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT) - pci_suspend_ptm(dev); - __pci_enable_wake(dev, target_state, pci_dev_run_wake(dev)); error = pci_set_power_state(dev, target_state); - if (error) { + if (error) pci_enable_wake(dev, target_state, false); - pci_restore_ptm_state(dev); - } return error; }