From patchwork Tue Jun 3 22:19:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Smita Koralahalli X-Patchwork-Id: 893970 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2076.outbound.protection.outlook.com [40.107.93.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2CBFD2550D3; Tue, 3 Jun 2025 22:20:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.76 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748989214; cv=fail; b=ChJzVpDZVnFprjmiQdHZJ7uiwMVnC20itPZu6whJbtggLL6Wn8HzYFrlmtnlEdbXeRrKQL3PNbnemJ3xIwAwUcnp+VNRAygCc6n9Id8lJ13yRJIPPtinIpMINoDJllfAchZgeFZ3G1UfsW53WP7w0bBx4ygHe8JP9hoNsQo9eMA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748989214; c=relaxed/simple; bh=1osBlMvFUlp2kGJU/zmJFdZEI8qYHmB0DKaac0xYUU8=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=oPO9QMqzsnhusmOWDpnPXTMjqmzwwN+oxDCrdIiARwa1GqV6C72lRzXP2E6yHW5cBHIMzIZN5tfssZiWrPaF5uVPel1q8S/enEPH1ArAw2GrYx86cOINPHpaHaHjMjmB0vDwZI2jp9ipGgUXF2b/L2glRHVvbdRnlMgtKYYs7S0= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=DcuzcDVf; arc=fail smtp.client-ip=40.107.93.76 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="DcuzcDVf" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=uGyQtz3fQ93gFFw74s6V7UPZF6yFGjecd4euR9aJMbRXPsvdVn94XDbjw0dAaFVE0zDzN4ApRsq5cNAzElsEpttXPCCpxlafg8VIIixZJXQC8NIxyeFGS5BrqkAyJbzhhNqxzjd9atjQCpAL6oa0XmLFnzGEHlpVznWeDC3SeURy9DzLcODpA1nPzsYAP7qWgrxD2OCvG58gwbSHcfSYAmnCc4Gn27aoGw97/DP9zci22YP8fm7/IKmYbeibhsh0wbKbUcnR8pBR6hKfFLQ8nTbwQ1ZmedRToXG6efXoYJRpgXg3NWuVPiqW3gR8/21Lh2noaopMfivOP5E1gTHLWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=K9fmUy2piLdGGKbNxeBJ3C6cYSaNbZu7eNQuByQYbSo=; b=YTBJ3dJu98boKtzuACPy8LjdRijDDCsAzY5kFz2LLFIDy9F9EbHa4CM67EGcSXWeRcV8yxa5tyuU9EI4eVj85uRn9FgyO0wE3xpciRU4GOXJ4GjNjdrPScgFIr281IwQS+jSX4/kymANOH+aVcj6opqja+EUuKaGCJCxjdiVcjPJA+I985HFPy45UGrSDS8CYGJrHTYfTr37ORKtobyDqtOJWA9bpwAlVzHtElo1HosZh5oGm0zE7YvKL0zhtnW3Z8pu1lSMEO7Lxy2QUSvwkA+VVNIKlbGPmxgTt65HVsg75CozMSHOcrsBg0bOeYyUzBdRUcpQRPH/T2T+7SD1/Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=K9fmUy2piLdGGKbNxeBJ3C6cYSaNbZu7eNQuByQYbSo=; b=DcuzcDVfwlU0pWUObrdCff0cA4ZZQX+l2cbFaPrwagRXL11d9DXEAu7W4rmLc7AaGoaDX3r+Hv8g+FlwnIHM9Ik3YLDkt85Qfsgq9JMjEMuvs19moc/XSQhYpSxBbHawb9cRZXYHe/KukWO48xcEVvBE9BQ9qHTY+OIiSwYbN1g= Received: from SJ0PR13CA0199.namprd13.prod.outlook.com (2603:10b6:a03:2c3::24) by DM4PR12MB6039.namprd12.prod.outlook.com (2603:10b6:8:aa::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8792.34; Tue, 3 Jun 2025 22:20:09 +0000 Received: from CY4PEPF0000FCBF.namprd03.prod.outlook.com (2603:10b6:a03:2c3:cafe::dd) by SJ0PR13CA0199.outlook.office365.com (2603:10b6:a03:2c3::24) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8792.26 via Frontend Transport; Tue, 3 Jun 2025 22:20:08 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000FCBF.mail.protection.outlook.com (10.167.242.101) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8792.29 via Frontend Transport; Tue, 3 Jun 2025 22:20:08 +0000 Received: from ethanolx50f7host.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 3 Jun 2025 17:20:06 -0500 From: Smita Koralahalli To: , , , , CC: Davidlohr Bueso , Jonathan Cameron , Dave Jiang , "Alison Schofield" , Vishal Verma , Ira Weiny , Dan Williams , Matthew Wilcox , Jan Kara , "Rafael J . Wysocki" , Len Brown , Pavel Machek , Li Ming , Jeff Johnson , "Ying Huang" , Yao Xingtao , Peter Zijlstra , Greg KH , Nathan Fontenot , Smita Koralahalli , Terry Bowman , Robert Richter , Benjamin Cheatham , PradeepVineshReddy Kodamati , Zhijian Li Subject: [PATCH v4 7/7] cxl/dax: Defer DAX consumption of SOFT RESERVED resources until after CXL region creation Date: Tue, 3 Jun 2025 22:19:49 +0000 Message-ID: <20250603221949.53272-8-Smita.KoralahalliChannabasappa@amd.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20250603221949.53272-1-Smita.KoralahalliChannabasappa@amd.com> References: <20250603221949.53272-1-Smita.KoralahalliChannabasappa@amd.com> Precedence: bulk X-Mailing-List: linux-pm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000FCBF:EE_|DM4PR12MB6039:EE_ X-MS-Office365-Filtering-Correlation-Id: cac8e9a6-6d1b-427c-a759-08dda2eccca4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|1800799024|376014|7416014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?utf-8?q?gxHnGmwn5G7HhCMMBAx7Xm7Jm7R9ooJ?= =?utf-8?q?GfaTHy5Y7IhPgyaFEwTEBpSl6KOGkhEHvYaWEr7Cpy84ka0scujd9oZSal3auSN1X?= =?utf-8?q?x2izwIc8rRHF0bOiyjDU9l4bqW0f0Va7lXUP2QyI62fiK4V+CfmZDbtpLucgRuEA8?= =?utf-8?q?nvVTlCvDfbnkNePsN8rGBzU3nTBjLPOffnATdY3VXYorbsT7D9x9NViJmUGM1RrDV?= =?utf-8?q?W7lYPlG3rG4/dOQxB5V16FrS4zKuBdmfMaCaizW8WRapg5hceizYJr2VXnjiXuh8x?= =?utf-8?q?EM1rx3hB9M9LYrGMDwWkGtWddegNvrDR0HgAMTCEgWelsNTfCEs1XVC9nC/lEaBJv?= =?utf-8?q?G6QJPvNy7neE7IjESN0mKlTKTu2CExgREbBnFmGJIQm7RvAs58Br902zRFEigFgIH?= =?utf-8?q?/numSMJzo0DZHiz58L6PBxKGoVBDRboO12w1pOZR0FCNu/EquuHQRoG7OStCFj3Ht?= =?utf-8?q?4PeEXHmjNaJqkrUwFFJWM1AHA7aLDfwlLf8ZW/B8MNzvyDEg3n+0pqzWog2koDkCJ?= =?utf-8?q?AI4pdRXDQ4KJD2vxIhbLmOh/Vk2Q9NaJj3bYUtG2Oc6p/PYys+bw4i8mrQdM6QzOv?= =?utf-8?q?0/rfTuWFbW/Pp9ETRsGFn3U91kDNPg5mMfEmIYnnZBbjlrZOPKtmM0pr0I6SMOWmQ?= =?utf-8?q?nHQXIjZuMg34s8EO1MGZIo40j+qoFTsc0J6RrytM/2KNct8wHAIX2F6DY4sWZqWZi?= =?utf-8?q?uB0/KDqPFSI2yMiUxv9ycvX7917ddokaJXZ+4BdiuOwsE8Ir+qdwFfn91tVICd5cs?= =?utf-8?q?j1P5L0u0T7QSVEpNoluWg44QXL/V3EHc+T/oegBjefkantrftcH3Iuez3vcjEO8Bk?= =?utf-8?q?SdWcKvSxRh5QgxyETuV6St7UeR5+kTpWWxwtJfsJv5iPxAqLGMFQSwQyifimBKkB0?= =?utf-8?q?Rwm86bvIkjQtvc5e42RxJqWrKNFhsSQg202KWrRCuLPYkBc75w1MHgpRTiBKXwUL9?= =?utf-8?q?CNzjCasjFPcxQf/47ptn7FFvHp4cK/SDYtWnbtXInnpOs1YVwy39M2tyXGWJOjH5L?= =?utf-8?q?RoaXvFjHXRuSuBdz8pmX2oXHpBXO8dWdaONIP0dm3Av4mSY3ut80AP4JbLRJAXLM+?= =?utf-8?q?G+ZsjHL5+UuNsxctD5knxG7D8gb0ZVmZPMbHAePJcioxSAwO8BZkZvsNmriugCpVu?= =?utf-8?q?AeAw/YhJ8HYLh5a+zkyYjXOEVu1oaMW6f5OrqE74yvTID8LHRAvEdkCeSoii8lgq3?= =?utf-8?q?xUYseFCpM/GAo/8N5oSDzmuxGJklrpBZf5UCi1LGbeJLoJK2vQbFbninKNcyLtZb2?= =?utf-8?q?5yagkW1UcWLkZ5cc84a1O6GIL36TMq7RAZgBXEYCH/2rKQHDSdTHrQD+lTtWRb6NI?= =?utf-8?q?3ggfRO+Dn3bTy7BxXT0Rxpd/TuiOT8DMbJ/sdZIuxJG10E/8Rm7W/Q6sRNl/BuDzD?= =?utf-8?q?TPs6sbMHxayQ2x+EL3C9lM2WPtEo23fJt2+gFeoJakyIA/WFFqjF5DAWe0dMhk2Gr?= =?utf-8?q?RDscyv9g+y?= X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:SATLEXMB04.amd.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(1800799024)(376014)(7416014)(82310400026)(36860700013); DIR:OUT; SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2025 22:20:08.5214 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cac8e9a6-6d1b-427c-a759-08dda2eccca4 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000FCBF.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6039 From: Nathan Fontenot The DAX HMEM driver currently consumes all SOFT RESERVED iomem resources during initialization. This interferes with the CXL driver’s ability to create regions and trim overlapping SOFT RESERVED ranges before DAX uses them. To resolve this, defer the DAX driver's resource consumption if the cxl_acpi driver is enabled. The DAX HMEM initialization skips walking the iomem resource tree in this case. After CXL region creation completes, any remaining SOFT RESERVED resources are explicitly registered with the DAX driver by the CXL driver. This sequencing ensures proper handling of overlaps and fixes hotplug failures. Co-developed-by: Nathan Fontenot Signed-off-by: Nathan Fontenot Co-developed-by: Terry Bowman Signed-off-by: Terry Bowman Signed-off-by: Smita Koralahalli --- drivers/cxl/core/region.c | 10 +++++++++ drivers/dax/hmem/device.c | 43 ++++++++++++++++++++------------------- drivers/dax/hmem/hmem.c | 3 ++- include/linux/dax.h | 6 ++++++ 4 files changed, 40 insertions(+), 22 deletions(-) diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index 3a5ca44d65f3..c6c0c7ba3b20 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include "core.h" @@ -3553,6 +3554,11 @@ static struct resource *normalize_resource(struct resource *res) return NULL; } +static int cxl_softreserv_mem_register(struct resource *res, void *unused) +{ + return hmem_register_device(phys_to_target_node(res->start), res); +} + static int __cxl_region_softreserv_update(struct resource *soft, void *_cxlr) { @@ -3590,6 +3596,10 @@ int cxl_region_softreserv_update(void) __cxl_region_softreserv_update); } + /* Now register any remaining SOFT RESERVES with DAX */ + walk_iomem_res_desc(IORES_DESC_SOFT_RESERVED, IORESOURCE_MEM, + 0, -1, NULL, cxl_softreserv_mem_register); + return 0; } EXPORT_SYMBOL_NS_GPL(cxl_region_softreserv_update, "CXL"); diff --git a/drivers/dax/hmem/device.c b/drivers/dax/hmem/device.c index 59ad44761191..cc1ed7bbdb1a 100644 --- a/drivers/dax/hmem/device.c +++ b/drivers/dax/hmem/device.c @@ -8,7 +8,6 @@ static bool nohmem; module_param_named(disable, nohmem, bool, 0444); -static bool platform_initialized; static DEFINE_MUTEX(hmem_resource_lock); static struct resource hmem_active = { .name = "HMEM devices", @@ -35,9 +34,7 @@ EXPORT_SYMBOL_GPL(walk_hmem_resources); static void __hmem_register_resource(int target_nid, struct resource *res) { - struct platform_device *pdev; struct resource *new; - int rc; new = __request_region(&hmem_active, res->start, resource_size(res), "", 0); @@ -47,21 +44,6 @@ static void __hmem_register_resource(int target_nid, struct resource *res) } new->desc = target_nid; - - if (platform_initialized) - return; - - pdev = platform_device_alloc("hmem_platform", 0); - if (!pdev) { - pr_err_once("failed to register device-dax hmem_platform device\n"); - return; - } - - rc = platform_device_add(pdev); - if (rc) - platform_device_put(pdev); - else - platform_initialized = true; } void hmem_register_resource(int target_nid, struct resource *res) @@ -83,9 +65,28 @@ static __init int hmem_register_one(struct resource *res, void *data) static __init int hmem_init(void) { - walk_iomem_res_desc(IORES_DESC_SOFT_RESERVED, - IORESOURCE_MEM, 0, -1, NULL, hmem_register_one); - return 0; + struct platform_device *pdev; + int rc; + + if (!IS_ENABLED(CONFIG_CXL_ACPI)) { + walk_iomem_res_desc(IORES_DESC_SOFT_RESERVED, + IORESOURCE_MEM, 0, -1, NULL, + hmem_register_one); + } + + pdev = platform_device_alloc("hmem_platform", 0); + if (!pdev) { + pr_err("failed to register device-dax hmem_platform device\n"); + return -1; + } + + rc = platform_device_add(pdev); + if (rc) { + pr_err("failed to add device-dax hmem_platform device\n"); + platform_device_put(pdev); + } + + return rc; } /* diff --git a/drivers/dax/hmem/hmem.c b/drivers/dax/hmem/hmem.c index 3aedef5f1be1..a206b9b383e4 100644 --- a/drivers/dax/hmem/hmem.c +++ b/drivers/dax/hmem/hmem.c @@ -61,7 +61,7 @@ static void release_hmem(void *pdev) platform_device_unregister(pdev); } -static int hmem_register_device(int target_nid, const struct resource *res) +int hmem_register_device(int target_nid, const struct resource *res) { struct device *host = &dax_hmem_pdev->dev; struct platform_device *pdev; @@ -124,6 +124,7 @@ static int hmem_register_device(int target_nid, const struct resource *res) platform_device_put(pdev); return rc; } +EXPORT_SYMBOL_GPL(hmem_register_device); static int dax_hmem_platform_probe(struct platform_device *pdev) { diff --git a/include/linux/dax.h b/include/linux/dax.h index a4ad3708ea35..5052dca8b3bc 100644 --- a/include/linux/dax.h +++ b/include/linux/dax.h @@ -299,10 +299,16 @@ static inline int dax_mem2blk_err(int err) #ifdef CONFIG_DEV_DAX_HMEM_DEVICES void hmem_register_resource(int target_nid, struct resource *r); +int hmem_register_device(int target_nid, const struct resource *res); #else static inline void hmem_register_resource(int target_nid, struct resource *r) { } + +static inline int hmem_register_device(int target_nid, const struct resource *res) +{ + return 0; +} #endif typedef int (*walk_hmem_fn)(int target_nid, const struct resource *res);