From patchwork Wed Nov 11 05:43:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Widawsky X-Patchwork-Id: 322980 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFEACC55ABD for ; Wed, 11 Nov 2020 05:45:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 698A022227 for ; Wed, 11 Nov 2020 05:45:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725947AbgKKFoH (ORCPT ); Wed, 11 Nov 2020 00:44:07 -0500 Received: from mga12.intel.com ([192.55.52.136]:54155 "EHLO mga12.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725900AbgKKFoH (ORCPT ); Wed, 11 Nov 2020 00:44:07 -0500 IronPort-SDR: EHQ43h7e8DrqVsxDSzi6/8EFCnvv/nt/i5uyHSJcmMF5Em1HbhWv7xSBNn87vp35XNa28IYicE yYClnbfqMvRQ== X-IronPort-AV: E=McAfee;i="6000,8403,9801"; a="149372956" X-IronPort-AV: E=Sophos;i="5.77,468,1596524400"; d="scan'208";a="149372956" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Nov 2020 21:44:04 -0800 IronPort-SDR: Jm3eKzSLS+VC8sYNrsKiLonFsR01pgEsrjtH2W6euLvi1AulpKq4hcIEk30rI0T7gqTeL3an50 Or5YaNtR3Y1g== X-IronPort-AV: E=Sophos;i="5.77,468,1596524400"; d="scan'208";a="360414741" Received: from hccoutan-mobl1.amr.corp.intel.com (HELO bwidawsk-mobl5.local) ([10.252.131.159]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Nov 2020 21:44:03 -0800 From: Ben Widawsky To: linux-cxl@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-acpi@vger.kernel.org, Dan Williams , Ira Weiny , Vishal Verma , "Kelley, Sean V" , Bjorn Helgaas , "Rafael J . Wysocki" , Ben Widawsky Subject: [RFC PATCH 2/9] cxl/acpi: add OSC support Date: Tue, 10 Nov 2020 21:43:49 -0800 Message-Id: <20201111054356.793390-3-ben.widawsky@intel.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201111054356.793390-1-ben.widawsky@intel.com> References: <20201111054356.793390-1-ben.widawsky@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org From: Vishal Verma Add support to advertise OS capabilities, and request OS control for CXL features using the ACPI _OSC mechanism. Advertise support for all possible CXL features, and attempt to request control too for all possible features. Based on a patch by Sean Kelley. Signed-off-by: Vishal Verma Signed-off-by: Ben Widawsky --- This uses a non-standard UUID for CXL and is meant as a change proposal to the CXL specification. The definition for _OSC intermixes PCIe and CXL Dwords, which makes a clean separation of CXL capabilities difficult, if not impossible. This is therefore subject to change. --- drivers/cxl/acpi.c | 210 ++++++++++++++++++++++++++++++++++++++++++++- drivers/cxl/acpi.h | 18 ++++ 2 files changed, 226 insertions(+), 2 deletions(-) diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c index 26e4f73838a7..c3e2f7f6ea02 100644 --- a/drivers/cxl/acpi.c +++ b/drivers/cxl/acpi.c @@ -12,6 +12,16 @@ #include #include "acpi.h" +/* + * TODO: These are global for now to save the OSC state. + * This works if OSC can be expected to be uniform for every ACPI0016 object. + * If that is not the case, revisit this. These need to be saved because + * relinquishing control of a previously granted capability is disallowed. + */ +static u32 cxl_osc_support_set; +static u32 cxl_osc_control_set; +static DEFINE_MUTEX(acpi_desc_lock); + static void acpi_cxl_desc_init(struct acpi_cxl_desc *acpi_desc, struct device *dev) { dev_set_drvdata(dev, acpi_desc); @@ -74,6 +84,199 @@ static struct acpi_driver acpi_cxl_driver = { }, }; +struct pci_osc_bit_struct { + u32 bit; + char *desc; +}; + +static struct pci_osc_bit_struct cxl_osc_support_bit[] = { + { CXL_OSC_PORT_REG_ACCESS_SUPPORT, "CXLPortRegAccess" }, + { CXL_OSC_PORT_DEV_REG_ACCESS_SUPPORT, "CXLPortDevRegAccess" }, + { CXL_OSC_PER_SUPPORT, "CXLProtocolErrorReporting" }, + { CXL_OSC_NATIVE_HP_SUPPORT, "CXLNativeHotPlug" }, +}; + +static struct pci_osc_bit_struct cxl_osc_control_bit[] = { + { CXL_OSC_MEM_ERROR_CONTROL, "CXLMemErrorReporting" }, +}; + +/* + * CXL 2.0 spec UUID - unusable as it PCI and CXL OSC are mixed + * static u8 cxl_osc_uuid_str[] = "68F2D50B-C469-4d8A-BD3D-941A103FD3FC"; + */ +/* New, proposed UUID for CXL-only OSC */ +static u8 cxl_osc_uuid_str[] = "A4D1629D-FF52-4888-BE96-E5CADE548DB1"; + +static void decode_osc_bits(struct device *dev, char *msg, u32 word, + struct pci_osc_bit_struct *table, int size) +{ + char buf[80]; + int i, len = 0; + struct pci_osc_bit_struct *entry; + + buf[0] = '\0'; + for (i = 0, entry = table; i < size; i++, entry++) + if (word & entry->bit) + len += scnprintf(buf + len, sizeof(buf) - len, "%s%s", + len ? " " : "", entry->desc); + + dev_info(dev, "_OSC: %s [%s]\n", msg, buf); +} + +static void decode_cxl_osc_support(struct device *dev, char *msg, u32 word) +{ + decode_osc_bits(dev, msg, word, cxl_osc_support_bit, + ARRAY_SIZE(cxl_osc_support_bit)); +} + +static void decode_cxl_osc_control(struct device *dev, char *msg, u32 word) +{ + decode_osc_bits(dev, msg, word, cxl_osc_control_bit, + ARRAY_SIZE(cxl_osc_control_bit)); +} + +static acpi_status acpi_cap_run_osc(acpi_handle handle, + const u32 *capbuf, u8 *uuid_str, u32 *retval) +{ + struct acpi_osc_context context = { + .uuid_str = uuid_str, + .rev = 1, + .cap.length = 20, + .cap.pointer = (void *)capbuf, + }; + acpi_status status; + + status = acpi_run_osc(handle, &context); + if (ACPI_SUCCESS(status)) { + *retval = *((u32 *)(context.ret.pointer + 8)); + kfree(context.ret.pointer); + } + return status; +} + +static acpi_status cxl_query_osc(acpi_handle handle, u32 support, u32 *control) +{ + acpi_status status; + u32 result, capbuf[5]; + + support &= CXL_OSC_SUPPORT_VALID_MASK; + support |= cxl_osc_support_set; + + capbuf[OSC_QUERY_DWORD] = OSC_QUERY_ENABLE; + capbuf[CXL_OSC_SUPPORT_DWORD] = support; + if (control) { + *control &= CXL_OSC_CONTROL_VALID_MASK; + capbuf[CXL_OSC_CONTROL_DWORD] = *control | cxl_osc_control_set; + } else { + /* Run _OSC query only with existing controls. */ + capbuf[CXL_OSC_CONTROL_DWORD] = cxl_osc_control_set; + } + + status = acpi_cap_run_osc(handle, capbuf, cxl_osc_uuid_str, &result); + if (ACPI_SUCCESS(status)) { + cxl_osc_support_set = support; + if (control) + *control = result; + } + return status; +} + +static acpi_status cxl_osc_advertise_support(acpi_handle handle, u32 flags) +{ + acpi_status status; + + mutex_lock(&acpi_desc_lock); + status = cxl_query_osc(handle, flags, NULL); + mutex_unlock(&acpi_desc_lock); + return status; +} + +/** + * cxl_osc_request_control - Request control of CXL root _OSC features. + * @adev: ACPI device for the PCI root bridge (or PCIe Root Complex). + * @mask: Mask of _OSC bits to request control of, place to store control mask. + * @req: Mask of _OSC bits the control of is essential to the caller. + **/ +static acpi_status cxl_osc_request_control(struct acpi_device *adev, u32 *mask, u32 req) +{ + acpi_handle handle = adev->handle; + struct device *dev = &adev->dev; + acpi_status status = AE_OK; + u32 ctrl, capbuf[5]; + + if (!mask) + return AE_BAD_PARAMETER; + + ctrl = *mask & CXL_OSC_MEM_ERROR_CONTROL; + if ((ctrl & req) != req) + return AE_TYPE; + + mutex_lock(&acpi_desc_lock); + + *mask = ctrl | cxl_osc_control_set; + /* No need to evaluate _OSC if the control was already granted. */ + if ((cxl_osc_control_set & ctrl) == ctrl) + goto out; + + /* Need to check the available controls bits before requesting them. */ + while (*mask) { + status = cxl_query_osc(handle, cxl_osc_support_set, mask); + if (ACPI_FAILURE(status)) + goto out; + if (ctrl == *mask) + break; + decode_cxl_osc_control(dev, "platform does not support", ctrl & ~(*mask)); + ctrl = *mask; + } + + if ((ctrl & req) != req) { + decode_cxl_osc_control(dev, "not requesting control; platform does not support", + req & ~(ctrl)); + status = AE_SUPPORT; + goto out; + } + + capbuf[OSC_QUERY_DWORD] = 0; + capbuf[CXL_OSC_SUPPORT_DWORD] = cxl_osc_support_set; + capbuf[CXL_OSC_CONTROL_DWORD] = ctrl; + status = acpi_cap_run_osc(handle, capbuf, cxl_osc_uuid_str, mask); + if (ACPI_SUCCESS(status)) + cxl_osc_control_set = *mask; +out: + mutex_unlock(&acpi_desc_lock); + return status; +} + +static int cxl_negotiate_osc(struct acpi_device *adev) +{ + u32 cxl_support, cxl_control, requested; + acpi_handle handle = adev->handle; + struct device *dev = &adev->dev; + acpi_status status; + + /* Declare support for everything */ + cxl_support = CXL_OSC_SUPPORT_VALID_MASK; + decode_cxl_osc_support(dev, "OS supports", cxl_support); + status = cxl_osc_advertise_support(handle, cxl_support); + if (ACPI_FAILURE(status)) { + dev_info(dev, "CXL_OSC failed (%s)\n", acpi_format_exception(status)); + return -ENXIO; + } + + /* Request control for everything */ + cxl_control = CXL_OSC_CONTROL_VALID_MASK; + requested = cxl_control; + status = cxl_osc_request_control(adev, &cxl_control, CXL_OSC_MEM_ERROR_CONTROL); + if (ACPI_SUCCESS(status)) { + decode_cxl_osc_control(dev, "OS now controls", cxl_control); + } else { + decode_cxl_osc_control(dev, "OS requested", requested); + decode_cxl_osc_control(dev, "platform willing to grant", cxl_control); + dev_info(dev, "_OSC failed (%s)\n", acpi_format_exception(status)); + } + return 0; +} + /* * If/when CXL support is defined by other platform firmware the kernel * will need a mechanism to select between the platform specific version @@ -84,6 +287,7 @@ int cxl_bus_prepared(struct pci_dev *pdev) struct acpi_device *adev; struct pci_dev *root_port; struct device *root; + int rc; root_port = pcie_find_root_port(pdev); if (!root_port) @@ -97,9 +301,11 @@ int cxl_bus_prepared(struct pci_dev *pdev) if (!adev) return -ENXIO; - /* TODO: OSC enabling */ + rc = cxl_negotiate_osc(adev); + if (rc) + dev_err(&pdev->dev, "Failed to negotiate OSC\n"); - return 0; + return rc; } EXPORT_SYMBOL_GPL(cxl_bus_prepared); diff --git a/drivers/cxl/acpi.h b/drivers/cxl/acpi.h index 011505475cc6..bf8d078e1b2a 100644 --- a/drivers/cxl/acpi.h +++ b/drivers/cxl/acpi.h @@ -10,6 +10,24 @@ struct acpi_cxl_desc { struct device *dev; }; +/* Indexes into _OSC Capabilities Buffer */ +#define CXL_OSC_SUPPORT_DWORD 1 /* DWORD 2 */ +#define CXL_OSC_CONTROL_DWORD 2 /* DWORD 3 */ + +/* CXL Host Bridge _OSC: Capabilities DWORD 2: Support Field */ +#define CXL_OSC_PORT_REG_ACCESS_SUPPORT 0x00000001 +#define CXL_OSC_PORT_DEV_REG_ACCESS_SUPPORT 0x00000002 +#define CXL_OSC_PER_SUPPORT 0x00000004 +#define CXL_OSC_NATIVE_HP_SUPPORT 0x00000008 +#define CXL_OSC_SUPPORT_VALID_MASK (CXL_OSC_PORT_REG_ACCESS_SUPPORT | \ + CXL_OSC_PORT_DEV_REG_ACCESS_SUPPORT | \ + CXL_OSC_PER_SUPPORT | \ + CXL_OSC_NATIVE_HP_SUPPORT) + +/* CXL Host Bridge _OSC: Capabilities DWORD 3: Control Field */ +#define CXL_OSC_MEM_ERROR_CONTROL 0x00000001 +#define CXL_OSC_CONTROL_VALID_MASK (CXL_OSC_MEM_ERROR_CONTROL) + int cxl_bus_prepared(struct pci_dev *pci_dev); #endif /* __CXL_ACPI_H__ */ From patchwork Wed Nov 11 05:43:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Widawsky X-Patchwork-Id: 322981 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 405C5C63699 for ; Wed, 11 Nov 2020 05:44:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E7059216C4 for ; Wed, 11 Nov 2020 05:44:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725996AbgKKFoo (ORCPT ); Wed, 11 Nov 2020 00:44:44 -0500 Received: from mga12.intel.com ([192.55.52.136]:54150 "EHLO mga12.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725904AbgKKFoH (ORCPT ); Wed, 11 Nov 2020 00:44:07 -0500 IronPort-SDR: Gn52N7/v4HVmeOcNx/VTWozOXjE+nFcEBV3BMOiGgUQnu8+fYMwh9/tZBqE0Fu8xXubLS8rlAA XC6mOlIGjAiw== X-IronPort-AV: E=McAfee;i="6000,8403,9801"; a="149372962" X-IronPort-AV: E=Sophos;i="5.77,468,1596524400"; d="scan'208";a="149372962" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Nov 2020 21:44:05 -0800 IronPort-SDR: y+L7DoqIphxCfgbmHMu89ucgdp18Wkpq5lLB1IOm/8A8IMmV0QO3615+uwLRN+SgFXL5NsWGbF 5JoKjMFRp7Og== X-IronPort-AV: E=Sophos;i="5.77,468,1596524400"; d="scan'208";a="360414751" Received: from hccoutan-mobl1.amr.corp.intel.com (HELO bwidawsk-mobl5.local) ([10.252.131.159]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Nov 2020 21:44:04 -0800 From: Ben Widawsky To: linux-cxl@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-acpi@vger.kernel.org, Dan Williams , Ira Weiny , Vishal Verma , "Kelley, Sean V" , Bjorn Helgaas , "Rafael J . Wysocki" , Ben Widawsky Subject: [RFC PATCH 4/9] cxl/mem: Map memory device registers Date: Tue, 10 Nov 2020 21:43:51 -0800 Message-Id: <20201111054356.793390-5-ben.widawsky@intel.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201111054356.793390-1-ben.widawsky@intel.com> References: <20201111054356.793390-1-ben.widawsky@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org All the necessary bits are initialized in order to find and map the register space for CXL Memory Devices. This is accomplished by using the Register Locator DVSEC (CXL 2.0 - 8.1.9.1) to determine which PCI BAR to use, and how much of an offset from that BAR should be added. If the memory device registers are found and mapped a new internal data structure tracking device state is allocated. Signed-off-by: Ben Widawsky --- drivers/cxl/mem.c | 68 +++++++++++++++++++++++++++++++++++++++++++---- drivers/cxl/pci.h | 6 +++++ 2 files changed, 69 insertions(+), 5 deletions(-) diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c index aa7d881fa47b..8d9b9ab6c5ea 100644 --- a/drivers/cxl/mem.c +++ b/drivers/cxl/mem.c @@ -7,9 +7,49 @@ #include "pci.h" struct cxl_mem { + struct pci_dev *pdev; void __iomem *regs; }; +static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi) +{ + struct device *dev = &pdev->dev; + struct cxl_mem *cxlm; + void __iomem *regs; + u64 offset; + u8 bar; + int rc; + + offset = ((u64)reg_hi << 32) | (reg_lo & 0xffff0000); + bar = reg_lo & 0x7; + + /* Basic sanity check that BAR is big enough */ + if (pci_resource_len(pdev, bar) < offset) { + dev_err(dev, "bar%d: %pr: too small (offset: %#llx)\n", + bar, &pdev->resource[bar], (unsigned long long) offset); + return ERR_PTR(-ENXIO); + } + + rc = pcim_iomap_regions(pdev, 1 << bar, pci_name(pdev)); + if (rc != 0) { + dev_err(dev, "failed to map registers\n"); + return ERR_PTR(-ENXIO); + } + + cxlm = devm_kzalloc(&pdev->dev, sizeof(*cxlm), GFP_KERNEL); + if (!cxlm) { + dev_err(dev, "No memory available\n"); + return ERR_PTR(-ENOMEM); + } + + regs = pcim_iomap_table(pdev)[bar]; + cxlm->pdev = pdev; + cxlm->regs = regs + offset; + + dev_dbg(dev, "Mapped CXL Memory Device resource\n"); + return cxlm; +} + static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec) { int pos; @@ -34,9 +74,9 @@ static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec) static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) { + struct cxl_mem *cxlm = ERR_PTR(-ENXIO); struct device *dev = &pdev->dev; - struct cxl_mem *cxlm; - int rc, regloc; + int rc, regloc, i; rc = cxl_bus_prepared(pdev); if (rc != 0) { @@ -44,15 +84,33 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) return rc; } + rc = pcim_enable_device(pdev); + if (rc) + return rc; + regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC); if (!regloc) { dev_err(dev, "register location dvsec not found\n"); return -ENXIO; } + regloc += 0xc; /* Skip DVSEC + reserved fields */ + + for (i = regloc; i < regloc + 0x24; i += 8) { + u32 reg_lo, reg_hi; + + pci_read_config_dword(pdev, i, ®_lo); + pci_read_config_dword(pdev, i + 4, ®_hi); + + if (CXL_REGLOG_IS_MEMDEV(reg_lo)) { + cxlm = cxl_mem_create(pdev, reg_lo, reg_hi); + break; + } + } + + if (IS_ERR(cxlm)) + return -ENXIO; - cxlm = devm_kzalloc(dev, sizeof(*cxlm), GFP_KERNEL); - if (!cxlm) - return -ENOMEM; + pci_set_drvdata(pdev, cxlm); return 0; } diff --git a/drivers/cxl/pci.h b/drivers/cxl/pci.h index beb03921e6da..be87f62e9132 100644 --- a/drivers/cxl/pci.h +++ b/drivers/cxl/pci.h @@ -12,4 +12,10 @@ #define PCI_DVSEC_ID_CXL 0x0 #define PCI_DVSEC_ID_CXL_REGLOC 0x8 +#define CXL_REGLOG_RBI_EMPTY 0 +#define CXL_REGLOG_RBI_COMPONENT 1 +#define CXL_REGLOG_RBI_VIRT 2 +#define CXL_REGLOG_RBI_MEMDEV 3 +#define CXL_REGLOG_IS_MEMDEV(x) ((((x) >> 8) & 0xff) == CXL_REGLOG_RBI_MEMDEV) + #endif /* __CXL_PCI_H__ */ From patchwork Wed Nov 11 05:43:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Widawsky X-Patchwork-Id: 322983 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52852C388F9 for ; Wed, 11 Nov 2020 05:44:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0C50A221E2 for ; Wed, 11 Nov 2020 05:44:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726063AbgKKFoZ (ORCPT ); Wed, 11 Nov 2020 00:44:25 -0500 Received: from mga12.intel.com ([192.55.52.136]:54147 "EHLO mga12.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725965AbgKKFoI (ORCPT ); Wed, 11 Nov 2020 00:44:08 -0500 IronPort-SDR: Xib6Unz4nvgjBZTQXI9MC9fGyBlfhq/0pkbCt6yMEadn7hemiUtVsOokAp7vW1GH501oV+T4WC osAY/XH/szTg== X-IronPort-AV: E=McAfee;i="6000,8403,9801"; a="149372969" X-IronPort-AV: E=Sophos;i="5.77,468,1596524400"; d="scan'208";a="149372969" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Nov 2020 21:44:07 -0800 IronPort-SDR: 3HSWyoqmBuMGPPs8yl+Dj+egPIPs7BqYQ6BXx49QZFpg3xFCZgeIyF0zpbrasoVJksghbJiA09 h02KmG0Zw2dA== X-IronPort-AV: E=Sophos;i="5.77,468,1596524400"; d="scan'208";a="360414807" Received: from hccoutan-mobl1.amr.corp.intel.com (HELO bwidawsk-mobl5.local) ([10.252.131.159]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Nov 2020 21:44:06 -0800 From: Ben Widawsky To: linux-cxl@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-acpi@vger.kernel.org, Dan Williams , Ira Weiny , Vishal Verma , "Kelley, Sean V" , Bjorn Helgaas , "Rafael J . Wysocki" , Ben Widawsky Subject: [RFC PATCH 6/9] cxl/mem: Initialize the mailbox interface Date: Tue, 10 Nov 2020 21:43:53 -0800 Message-Id: <20201111054356.793390-7-ben.widawsky@intel.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201111054356.793390-1-ben.widawsky@intel.com> References: <20201111054356.793390-1-ben.widawsky@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Provide enough functionality to utilize the mailbox of a memory device. The mailbox is used to interact with the firmware running on the memory device. The CXL specification defines separate capabilities for the mailbox and the memory device. While we can confirm the mailbox is ready, in order to actually interact with the memory device, the driver must also confirm the device's firmware is ready. Signed-off-by: Ben Widawsky --- drivers/cxl/cxl.h | 28 ++++++++++++++++++++++ drivers/cxl/mem.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 87 insertions(+) diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 02858ae63d6d..482fc9cdc890 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -19,14 +19,41 @@ #define CXLDEV_MB_CAPS 0x00 #define CXLDEV_MB_CAP_PAYLOAD_SIZE(cap) ((cap) & 0x1F) #define CXLDEV_MB_CTRL 0x04 +#define CXLDEV_MB_CTRL_DOORBELL BIT(0) #define CXLDEV_MB_CMD 0x08 #define CXLDEV_MB_STATUS 0x10 #define CXLDEV_MB_BG_CMD_STATUS 0x18 +/* Memory Device */ +#define CXLMDEV_STATUS 0 +#define CXLMDEV_DEV_FATAL BIT(0) +#define CXLMDEV_FW_HALT BIT(1) +#define CXLMDEV_MEDIA_STATUS_SHIFT 2 +#define CXLMDEV_MEDIA_STATUS_MASK 0x3 +#define CXLMDEV_READY(status) \ + ((((status) >> CXLMDEV_MEDIA_STATUS_SHIFT) & \ + CXLMDEV_MEDIA_STATUS_MASK) == CXLMDEV_MS_READY) +#define CXLMDEV_MS_NOT_READY 0 +#define CXLMDEV_MS_READY 1 +#define CXLMDEV_MS_ERROR 2 +#define CXLMDEV_MS_DISABLED 3 +#define CXLMDEV_MBOX_IF_READY BIT(4) +#define CXLMDEV_RESET_NEEDED_SHIFT 5 +#define CXLMDEV_RESET_NEEDED_MASK 0x7 +#define CXLMDEV_RESET_NEEDED(status) \ + (((status) >> CXLMDEV_RESET_NEEDED_SHIFT) & CXLMDEV_RESET_NEEDED_MASK) +#define CXLMDEV_RESET_NEEDED_NOT 0 +#define CXLMDEV_RESET_NEEDED_COLD 1 +#define CXLMDEV_RESET_NEEDED_WARM 2 +#define CXLMDEV_RESET_NEEDED_HOT 3 +#define CXLMDEV_RESET_NEEDED_CXL 4 + struct cxl_mem { struct pci_dev *pdev; void __iomem *regs; + spinlock_t mbox_lock; /* Protects device mailbox and firmware */ + /* Cap 0000h */ struct { void __iomem *regs; @@ -72,6 +99,7 @@ struct cxl_mem { cxl_reg(status) cxl_reg(mbox) +cxl_reg(mem) static inline u32 __cxl_raw_read_reg32(struct cxl_mem *cxlm, u32 reg) { diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c index 4109ef7c3ecb..9fd2d1daa534 100644 --- a/drivers/cxl/mem.c +++ b/drivers/cxl/mem.c @@ -7,6 +7,56 @@ #include "pci.h" #include "cxl.h" +static int cxl_mem_mbox_get(struct cxl_mem *cxlm) +{ + u64 md_status; + u32 ctrl; + int rc = -EBUSY; + + spin_lock(&cxlm->mbox_lock); + + ctrl = cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL); + if (ctrl & CXLDEV_MB_CTRL_DOORBELL) + goto out; + + md_status = cxl_read_mem_reg64(cxlm, CXLMDEV_STATUS); + if (md_status & CXLMDEV_MBOX_IF_READY && CXLMDEV_READY(md_status)) { + /* + * Hardware shouldn't allow a ready status but also have failure + * bits set. Spit out an error, this should be a bug report + */ + if (md_status & CXLMDEV_DEV_FATAL) { + dev_err(&cxlm->pdev->dev, + "CXL device reporting ready and fatal\n"); + rc = -EFAULT; + goto out; + } + if (md_status & CXLMDEV_FW_HALT) { + dev_err(&cxlm->pdev->dev, + "CXL device reporting ready and halted\n"); + rc = -EFAULT; + goto out; + } + if (CXLMDEV_RESET_NEEDED(md_status)) { + dev_err(&cxlm->pdev->dev, + "CXL device reporting ready and reset needed\n"); + rc = -EFAULT; + goto out; + } + + return 0; + } + +out: + spin_unlock(&cxlm->mbox_lock); + return rc; +} + +static void cxl_mem_mbox_put(struct cxl_mem *cxlm) +{ + spin_unlock(&cxlm->mbox_lock); +} + static int cxl_mem_setup_regs(struct cxl_mem *cxlm) { u64 cap_array; @@ -88,6 +138,8 @@ static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev, u32 reg_lo, u32 reg_ return ERR_PTR(-ENOMEM); } + spin_lock_init(&cxlm->mbox_lock); + regs = pcim_iomap_table(pdev)[bar]; cxlm->pdev = pdev; cxlm->regs = regs + offset; @@ -160,6 +212,13 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (rc) return rc; + /* Check that hardware "looks" okay. */ + rc = cxl_mem_mbox_get(cxlm); + if (rc) + return rc; + + cxl_mem_mbox_put(cxlm); + dev_dbg(&pdev->dev, "CXL Memory Device Interface Up\n"); pci_set_drvdata(pdev, cxlm); return 0; From patchwork Wed Nov 11 05:43:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Widawsky X-Patchwork-Id: 322982 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47C0FC55ABD for ; Wed, 11 Nov 2020 05:44:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E7B6C20825 for ; Wed, 11 Nov 2020 05:44:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726028AbgKKFoY (ORCPT ); Wed, 11 Nov 2020 00:44:24 -0500 Received: from mga12.intel.com ([192.55.52.136]:54147 "EHLO mga12.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725977AbgKKFoJ (ORCPT ); Wed, 11 Nov 2020 00:44:09 -0500 IronPort-SDR: Z09yrwLwcb1UcrEgoy+xxXQp1QhDhvRWkKHsygWmsIxePfiPaQOizJbcB1KTuqM1hnzXrcIc6C eqAaExmPEf4w== X-IronPort-AV: E=McAfee;i="6000,8403,9801"; a="149372975" X-IronPort-AV: E=Sophos;i="5.77,468,1596524400"; d="scan'208";a="149372975" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Nov 2020 21:44:09 -0800 IronPort-SDR: UmJc0Z0R+ywJjvrS9sQhlLGXxukrPhbcAcbaMsq235opUrOzPvAPKfPJ47aXv4U7Fjrh/iVMZ7 Avl9TnV+szrw== X-IronPort-AV: E=Sophos;i="5.77,468,1596524400"; d="scan'208";a="360414859" Received: from hccoutan-mobl1.amr.corp.intel.com (HELO bwidawsk-mobl5.local) ([10.252.131.159]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Nov 2020 21:44:08 -0800 From: Ben Widawsky To: linux-cxl@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-acpi@vger.kernel.org, Dan Williams , Ira Weiny , Vishal Verma , "Kelley, Sean V" , Bjorn Helgaas , "Rafael J . Wysocki" , Ben Widawsky Subject: [RFC PATCH 9/9] MAINTAINERS: Add maintainers of the CXL driver Date: Tue, 10 Nov 2020 21:43:56 -0800 Message-Id: <20201111054356.793390-10-ben.widawsky@intel.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201111054356.793390-1-ben.widawsky@intel.com> References: <20201111054356.793390-1-ben.widawsky@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Cc: Dan Williams Cc: Vishal Verma Cc: Ira Weiny Signed-off-by: Ben Widawsky --- MAINTAINERS | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index b516bb34a8d5..25e294031376 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -4428,6 +4428,15 @@ F: fs/configfs/ F: include/linux/configfs.h F: samples/configfs/ +COMPUTE EXPRESS LINK (CXL) +M: Vishal Verma +M: Ira Weiny +M: Ben Widawsky +M: Dan Williams +L: linux-cxl@vger.kernel.org +S: Maintained +F: drivers/cxl/ + CONSOLE SUBSYSTEM M: Greg Kroah-Hartman S: Supported