| Message ID | 20201111054356.793390-1-ben.widawsky@intel.com |
|---|---|
| Headers | show |
| Series | CXL 2.0 Support | expand |
On Tue, Nov 10, 2020 at 11:12 PM Christoph Hellwig <hch@infradead.org> wrote: > > On Tue, Nov 10, 2020 at 09:43:50PM -0800, Ben Widawsky wrote: > > +config CXL_MEM > > + tristate "CXL.mem Device Support" > > + depends on PCI && CXL_BUS_PROVIDER != n > > depend on PCI && CXL_BUS_PROVIDER > > > + default m if CXL_BUS_PROVIDER > > Please don't set weird defaults for new code. Especially not default > to module crap like this. This goes back to what people like Dave C. asked for LIBNVDIMM / DAX, a way to blanket turn on a subsystem without needing to go hunt down individual configs. All of CXL is "default n", but if someone turns on a piece of it they get all of it by default. The user can then opt-out on pieces after that first opt-in. If there's a better way to turn on suggested configs I'm open to switch to that style. As for the "default m" I was worried that it would be "default y" without the specificity, but I did not test that... will check. There have been times when I wished that distros defaulted bleeding edge new enabling to 'm' and putting that default in the Kconfig maybe saves me from needing to file individual config changes to distros after the fact. > > > +// Copyright(c) 2020 Intel Corporation. All rights reserved. > > Please don't use '//' for anything but the SPDX header. Ok, I find // following by /* */ a bit ugly, but I don't care enough to fight. > > > + > > + pci_read_config_word(pdev, pos + PCI_DVSEC_VENDOR_OFFSET, &vendor); > > + pci_read_config_word(pdev, pos + PCI_DVSEC_ID_OFFSET, &id); > > + if (vendor == PCI_DVSEC_VENDOR_CXL && dvsec == id) > > + return pos; > > + > > + pos = pci_find_next_ext_capability(pdev, pos, PCI_EXT_CAP_ID_DVSEC); > > Overly long lines again. I thought 100 is the new 80 these days? > > +static void cxl_mem_remove(struct pci_dev *pdev) > > +{ > > +} > > No need for the empty remove callback. True, will fix. > > > +MODULE_AUTHOR("Intel Corporation"); > > A module author is not a company. At least I don't have a copyright assignment clause, I don't agree with the vanity of listing multiple people here especially when MAINTAINERS has the contact info, and I don't want to maintain a list as people do drive-by contributions and we need to figure out at what level of contribution mandates a new MODULE_AUTHOR line. Now, that said I would be ok to duplicate the MAINTAINERS as MODULE_AUTHOR lines, but I otherwise expect MAINTAINERS is the central source for module contact info.
On Wed, Nov 11, 2020 at 9:17 AM Dan Williams <dan.j.williams@intel.com> wrote: [..] > > > + > > > + pci_read_config_word(pdev, pos + PCI_DVSEC_VENDOR_OFFSET, &vendor); > > > + pci_read_config_word(pdev, pos + PCI_DVSEC_ID_OFFSET, &id); > > > + if (vendor == PCI_DVSEC_VENDOR_CXL && dvsec == id) > > > + return pos; > > > + > > > + pos = pci_find_next_ext_capability(pdev, pos, PCI_EXT_CAP_ID_DVSEC); > > > > Overly long lines again. > > I thought 100 is the new 80 these days? Saw your clarification to Vishal, I had missed that. Will trim.
On 11/11/20 9:17 AM, Dan Williams wrote: > On Tue, Nov 10, 2020 at 11:12 PM Christoph Hellwig <hch@infradead.org> wrote: >> >> On Tue, Nov 10, 2020 at 09:43:50PM -0800, Ben Widawsky wrote: >>> +config CXL_MEM >>> + tristate "CXL.mem Device Support" >>> + depends on PCI && CXL_BUS_PROVIDER != n >> >> depend on PCI && CXL_BUS_PROVIDER >> >>> + default m if CXL_BUS_PROVIDER >> >> Please don't set weird defaults for new code. Especially not default >> to module crap like this. > > This goes back to what people like Dave C. asked for LIBNVDIMM / DAX, > a way to blanket turn on a subsystem without needing to go hunt down > individual configs. All of CXL is "default n", but if someone turns on > a piece of it they get all of it by default. The user can then opt-out > on pieces after that first opt-in. If there's a better way to turn on > suggested configs I'm open to switch to that style. As for the > "default m" I was worried that it would be "default y" without the > specificity, but I did not test that... will check. There have been > times when I wished that distros defaulted bleeding edge new enabling > to 'm' and putting that default in the Kconfig maybe saves me from > needing to file individual config changes to distros after the fact. What we as developers put into mainline kernel Kconfig files has nothing to do with what distros use in their distro config files. Or at least it shouldn't. Maybe your experience has been different. >> >>> +// Copyright(c) 2020 Intel Corporation. All rights reserved. >> >> Please don't use '//' for anything but the SPDX header. > > Ok, I find // following by /* */ a bit ugly, but I don't care enough to fight. > Hm, it's not in coding-style AFAICT but Linus has OK-ed C99 style comments: http://lkml.iu.edu/hypermail/linux/kernel/1607.1/00627.html >>> +MODULE_AUTHOR("Intel Corporation"); >> >> A module author is not a company. > > At least I don't have a copyright assignment clause, I don't agree > with the vanity of listing multiple people here especially when > MAINTAINERS has the contact info, and I don't want to maintain a list > as people do drive-by contributions and we need to figure out at what > level of contribution mandates a new MODULE_AUTHOR line. Now, that > said I would be ok to duplicate the MAINTAINERS as MODULE_AUTHOR > lines, but I otherwise expect MAINTAINERS is the central source for > module contact info. Sure, MAINTAINERS is fine, but the MODULE_AUTHOR() above provides no useful information. Even saying (made up) linux-devel@linux.intel.com would be slightly better, but some kind of contact info would be great. Otherwise just delete that line. -- ~Randy
Adding a cross reference to the QEMU work since I sent those patches after this: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0 https://lists.nongnu.org/archive/html/qemu-devel/2020-11/msg02886.html [snip]
On Wed, Nov 11, 2020 at 1:42 PM Randy Dunlap <rdunlap@infradead.org> wrote: > > On 11/11/20 9:17 AM, Dan Williams wrote: > > On Tue, Nov 10, 2020 at 11:12 PM Christoph Hellwig <hch@infradead.org> wrote: > >> > >> On Tue, Nov 10, 2020 at 09:43:50PM -0800, Ben Widawsky wrote: > >>> +config CXL_MEM > >>> + tristate "CXL.mem Device Support" > >>> + depends on PCI && CXL_BUS_PROVIDER != n > >> > >> depend on PCI && CXL_BUS_PROVIDER > >> > >>> + default m if CXL_BUS_PROVIDER > >> > >> Please don't set weird defaults for new code. Especially not default > >> to module crap like this. > > > > This goes back to what people like Dave C. asked for LIBNVDIMM / DAX, > > a way to blanket turn on a subsystem without needing to go hunt down > > individual configs. All of CXL is "default n", but if someone turns on > > a piece of it they get all of it by default. The user can then opt-out > > on pieces after that first opt-in. If there's a better way to turn on > > suggested configs I'm open to switch to that style. As for the > > "default m" I was worried that it would be "default y" without the > > specificity, but I did not test that... will check. There have been > > times when I wished that distros defaulted bleeding edge new enabling > > to 'm' and putting that default in the Kconfig maybe saves me from > > needing to file individual config changes to distros after the fact. > > What we as developers put into mainline kernel Kconfig files has nothing > to do with what distros use in their distro config files. > Or at least it shouldn't. Maybe your experience has been different. I agree with that sentiment, but walk it back through the requirement I mentioned above... *if* we want a top-level CXL option (default n) that goes and enables many CXL sub-options the default for those sub-options is something that needs to be listed in the Kconfig. 'm' is more flexible than 'y', so if a user wants CXL at all, and doesn't care about how, I'd prefer it's 'm' rather than 'y'. I have had to go submit distro config fixes when Kconfig defaulted to 'y' when 'm' was available, and the reasoning for why it was 'y' was "oh, that was the Kconfig default when I flipped this other option". > >>> +// Copyright(c) 2020 Intel Corporation. All rights reserved. > >> > >> Please don't use '//' for anything but the SPDX header. > > > > Ok, I find // following by /* */ a bit ugly, but I don't care enough to fight. > > > > Hm, it's not in coding-style AFAICT but Linus has OK-ed C99 style comments: > http://lkml.iu.edu/hypermail/linux/kernel/1607.1/00627.html > > > >>> +MODULE_AUTHOR("Intel Corporation"); > >> > >> A module author is not a company. > > > > At least I don't have a copyright assignment clause, I don't agree > > with the vanity of listing multiple people here especially when > > MAINTAINERS has the contact info, and I don't want to maintain a list > > as people do drive-by contributions and we need to figure out at what > > level of contribution mandates a new MODULE_AUTHOR line. Now, that > > said I would be ok to duplicate the MAINTAINERS as MODULE_AUTHOR > > lines, but I otherwise expect MAINTAINERS is the central source for > > module contact info. > > Sure, MAINTAINERS is fine, but the MODULE_AUTHOR() above provides > no useful information. > Even saying (made up) linux-devel@linux.intel.com would be slightly better, > but some kind of contact info would be great. Otherwise just delete that line. True, if the goal is to allow random end users to email support questions about this module I'd rather not put my email there. Instead, if it's someone that has kernel development questions then they should be able to use MAINTAINERS for that contact.
On Tue, Nov 10, 2020 at 09:43:47PM -0800, Ben Widawsky wrote: > ... > Ben Widawsky (5): > cxl/mem: Map memory device registers > cxl/mem: Find device capabilities > cxl/mem: Initialize the mailbox interface > cxl/mem: Implement polled mode mailbox > MAINTAINERS: Add maintainers of the CXL driver > > Dan Williams (2): > cxl/mem: Add a driver for the type-3 mailbox To include important words first and use "Type 3" as in spec: cxl/mem: Add Type 3 mailbox driver > cxl/mem: Register CXL memX devices > > Vishal Verma (2): > cxl/acpi: Add an acpi_cxl module for the CXL interconnect > cxl/acpi: add OSC support For consistency: cxl/acpi: Add _OSC support It's conventional in drivers/acpi and drivers/pci to capitalize the "ACPI" and "PCI" initialisms except in actual C code. Seems like you're mostly doing the same with "CXL", except in the subject lines above. Since you're making a new directory, I guess you get to choose. I use "PCIe" (not "PCIE" or "PCI-E"; you have a mix) because that seems to be the official abbreviation.
On Wed, Nov 11, 2020 at 09:17:37AM -0800, Dan Williams wrote: > > > +config CXL_MEM > > > + tristate "CXL.mem Device Support" > > > + depends on PCI && CXL_BUS_PROVIDER != n > > > > depend on PCI && CXL_BUS_PROVIDER > > > > > + default m if CXL_BUS_PROVIDER > > > > Please don't set weird defaults for new code. Especially not default > > to module crap like this. > > This goes back to what people like Dave C. asked for LIBNVDIMM / DAX, > a way to blanket turn on a subsystem without needing to go hunt down > individual configs. Then at least do a default CXL_BUS_PROVIDER but we really don't do this elsewhere. E.g. we don't default the scsi disk driver on if there is some host adapter selected. > > > +MODULE_AUTHOR("Intel Corporation"); > > > > A module author is not a company. > > At least I don't have a copyright assignment clause, I don't agree > with the vanity of listing multiple people here especially when > MAINTAINERS has the contact info, and I don't want to maintain a list > as people do drive-by contributions and we need to figure out at what > level of contribution mandates a new MODULE_AUTHOR line. Now, that > said I would be ok to duplicate the MAINTAINERS as MODULE_AUTHOR > lines, but I otherwise expect MAINTAINERS is the central source for > module contact info. IMHO MODULE_AUTHOR is completely pointless. I haven't used for ~15 years. Especially as the concept that a module has a single author is a rather strange one.
On Tue, 10 Nov 2020 21:43:48 -0800 Ben Widawsky <ben.widawsky@intel.com> wrote: > From: Vishal Verma <vishal.l.verma@intel.com> > > Add an acpi_cxl module to coordinate the ACPI portions of the CXL > (Compute eXpress Link) interconnect. This driver binds to ACPI0017 > objects in the ACPI tree, and coordinates access to the resources > provided by the ACPI CEDT (CXL Early Discovery Table). I think the qemu series notes that this ACPI0017 is just a proposal at this stage. Please make sure that's highlighted here as well unless that status is out of date. > > It also coordinates operations of the root port _OSC object to notify > platform firmware that the OS has native support for the CXL > capabilities of endpoints. > > Note: the actbl1.h changes are speculative. The expectation is that they > will arrive through the ACPICA tree in due time. > > Cc: Ben Widawsky <ben.widawsky@intel.com> > Cc: Dan Williams <dan.j.williams@intel.com> > Signed-off-by: Vishal Verma <vishal.l.verma@intel.com> > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com> > --- > drivers/Kconfig | 1 + > drivers/Makefile | 1 + > drivers/cxl/Kconfig | 30 +++++++++++ > drivers/cxl/Makefile | 5 ++ > drivers/cxl/acpi.c | 119 ++++++++++++++++++++++++++++++++++++++++++ > drivers/cxl/acpi.h | 15 ++++++ > include/acpi/actbl1.h | 52 ++++++++++++++++++ > 7 files changed, 223 insertions(+) > create mode 100644 drivers/cxl/Kconfig > create mode 100644 drivers/cxl/Makefile > create mode 100644 drivers/cxl/acpi.c > create mode 100644 drivers/cxl/acpi.h > > diff --git a/drivers/Kconfig b/drivers/Kconfig > index dcecc9f6e33f..62c753a73651 100644 > --- a/drivers/Kconfig > +++ b/drivers/Kconfig > @@ -6,6 +6,7 @@ menu "Device Drivers" > source "drivers/amba/Kconfig" > source "drivers/eisa/Kconfig" > source "drivers/pci/Kconfig" > +source "drivers/cxl/Kconfig" > source "drivers/pcmcia/Kconfig" > source "drivers/rapidio/Kconfig" > > diff --git a/drivers/Makefile b/drivers/Makefile > index c0cd1b9075e3..5dad349de73b 100644 > --- a/drivers/Makefile > +++ b/drivers/Makefile > @@ -73,6 +73,7 @@ obj-$(CONFIG_NVM) += lightnvm/ > obj-y += base/ block/ misc/ mfd/ nfc/ > obj-$(CONFIG_LIBNVDIMM) += nvdimm/ > obj-$(CONFIG_DAX) += dax/ > +obj-$(CONFIG_CXL_BUS) += cxl/ > obj-$(CONFIG_DMA_SHARED_BUFFER) += dma-buf/ > obj-$(CONFIG_NUBUS) += nubus/ > obj-y += macintosh/ > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig > new file mode 100644 > index 000000000000..dd724bd364df > --- /dev/null > +++ b/drivers/cxl/Kconfig > @@ -0,0 +1,30 @@ > +# SPDX-License-Identifier: GPL-2.0-only > +menuconfig CXL_BUS > + tristate "CXL (Compute Express Link) Devices Support" > + help > + CXL is a bus that is electrically compatible with PCI-E, but layers > + three protocols on that signalling (CXL.io, CXL.cache, and CXL.mem). The > + CXL.cache protocol allows devices to hold cachelines locally, the > + CXL.mem protocol allows devices to be fully coherent memory targets, the > + CXL.io protocol is equivalent to PCI-E. Say 'y' to enable support for > + the configuration and management of devices supporting these protocols. > + > +if CXL_BUS > + > +config CXL_BUS_PROVIDER > + tristate > + > +config CXL_ACPI > + tristate "CXL Platform Support" > + depends on ACPI > + default CXL_BUS > + select CXL_BUS_PROVIDER > + help > + CXL Platform Support is a prerequisite for any CXL device driver that > + wants to claim ownership of the component register space. By default > + platform firmware assumes Linux is unaware of CXL capabilities and > + requires explicit opt-in. This platform component also mediates > + resources described by the CEDT (CXL Early Discovery Table) > + > + Say 'y' to enable CXL (Compute Express Link) drivers. > +endif > diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile > new file mode 100644 > index 000000000000..d38cd34a2582 > --- /dev/null > +++ b/drivers/cxl/Makefile > @@ -0,0 +1,5 @@ > +# SPDX-License-Identifier: GPL-2.0 > +obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o > + > +ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL > +cxl_acpi-y := acpi.o > diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c > new file mode 100644 > index 000000000000..26e4f73838a7 > --- /dev/null > +++ b/drivers/cxl/acpi.c > @@ -0,0 +1,119 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +/* > + * Copyright(c) 2020 Intel Corporation. All rights reserved. > + */ > +#include <linux/list_sort.h> > +#include <linux/module.h> > +#include <linux/mutex.h> > +#include <linux/sysfs.h> > +#include <linux/list.h> > +#include <linux/acpi.h> > +#include <linux/sort.h> > +#include <linux/pci.h> > +#include "acpi.h" > + > +static void acpi_cxl_desc_init(struct acpi_cxl_desc *acpi_desc, struct device *dev) > +{ > + dev_set_drvdata(dev, acpi_desc); No need to have this wrapper + it hides the fact you are not just initialsing the acpi_desc structure. > + acpi_desc->dev = dev; > +} > + > +static void acpi_cedt_put_table(void *table) > +{ > + acpi_put_table(table); > +} > + > +static int acpi_cxl_add(struct acpi_device *adev) > +{ > + struct acpi_cxl_desc *acpi_desc; > + struct device *dev = &adev->dev; > + struct acpi_table_header *tbl; > + acpi_status status = AE_OK; Set below, so don't do it here. > + acpi_size sz; > + int rc = 0; Set in paths in which it's used so don't do it here. > + > + status = acpi_get_table(ACPI_SIG_CEDT, 0, &tbl); > + if (ACPI_FAILURE(status)) { > + dev_err(dev, "failed to find CEDT at startup\n"); > + return 0; > + } > + > + rc = devm_add_action_or_reset(dev, acpi_cedt_put_table, tbl); > + if (rc) > + return rc; blank line here preferred for readability (do something, then check errors as block) > + sz = tbl->length; > + dev_info(dev, "found CEDT at startup: %lld bytes\n", sz); > + > + acpi_desc = devm_kzalloc(dev, sizeof(*acpi_desc), GFP_KERNEL); > + if (!acpi_desc) > + return -ENOMEM; blank line here slightly helps readability. > + acpi_cxl_desc_init(acpi_desc, &adev->dev); > + > + acpi_desc->acpi_header = *tbl; > + > + return 0; > +} > + > +static int acpi_cxl_remove(struct acpi_device *adev) > +{ > + return 0; Don't think empty remove is needed. > +} > + > +static const struct acpi_device_id acpi_cxl_ids[] = { > + { "ACPI0017", 0 }, > + { "", 0 }, > +}; > +MODULE_DEVICE_TABLE(acpi, acpi_cxl_ids); > + > +static struct acpi_driver acpi_cxl_driver = { > + .name = KBUILD_MODNAME, > + .ids = acpi_cxl_ids, > + .ops = { > + .add = acpi_cxl_add, > + .remove = acpi_cxl_remove, > + }, > +}; > + > +/* > + * If/when CXL support is defined by other platform firmware the kernel > + * will need a mechanism to select between the platform specific version > + * of this routine, until then, hard-code ACPI assumptions > + */ > +int cxl_bus_prepared(struct pci_dev *pdev) > +{ > + struct acpi_device *adev; > + struct pci_dev *root_port; > + struct device *root; > + > + root_port = pcie_find_root_port(pdev); > + if (!root_port) > + return -ENXIO; > + > + root = root_port->dev.parent; > + if (!root) > + return -ENXIO; > + > + adev = ACPI_COMPANION(root); > + if (!adev) > + return -ENXIO; > + > + /* TODO: OSC enabling */ > + > + return 0; > +} > +EXPORT_SYMBOL_GPL(cxl_bus_prepared); > + > +static __init int acpi_cxl_init(void) > +{ > + return acpi_bus_register_driver(&acpi_cxl_driver); > +} > + > +static __exit void acpi_cxl_exit(void) > +{ > + acpi_bus_unregister_driver(&acpi_cxl_driver); > +} > + > +module_init(acpi_cxl_init); > +module_exit(acpi_cxl_exit); > +MODULE_LICENSE("GPL v2"); > +MODULE_AUTHOR("Intel Corporation"); > diff --git a/drivers/cxl/acpi.h b/drivers/cxl/acpi.h > new file mode 100644 > index 000000000000..011505475cc6 > --- /dev/null > +++ b/drivers/cxl/acpi.h > @@ -0,0 +1,15 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +// Copyright(c) 2020 Intel Corporation. All rights reserved. > + > +#ifndef __CXL_ACPI_H__ > +#define __CXL_ACPI_H__ > +#include <linux/acpi.h> > + > +struct acpi_cxl_desc { > + struct acpi_table_header acpi_header; > + struct device *dev; > +}; > + > +int cxl_bus_prepared(struct pci_dev *pci_dev); > + > +#endif /* __CXL_ACPI_H__ */ > diff --git a/include/acpi/actbl1.h b/include/acpi/actbl1.h > index 43549547ed3e..70f745f526e3 100644 > --- a/include/acpi/actbl1.h > +++ b/include/acpi/actbl1.h > @@ -28,6 +28,7 @@ > #define ACPI_SIG_BERT "BERT" /* Boot Error Record Table */ > #define ACPI_SIG_BGRT "BGRT" /* Boot Graphics Resource Table */ > #define ACPI_SIG_BOOT "BOOT" /* Simple Boot Flag Table */ > +#define ACPI_SIG_CEDT "CEDT" /* CXL Early Discovery Table */ > #define ACPI_SIG_CPEP "CPEP" /* Corrected Platform Error Polling table */ > #define ACPI_SIG_CSRT "CSRT" /* Core System Resource Table */ > #define ACPI_SIG_DBG2 "DBG2" /* Debug Port table type 2 */ > @@ -1624,6 +1625,57 @@ struct acpi_ibft_target { > u16 reverse_chap_secret_offset; > }; > > +/******************************************************************************* > + * > + * CEDT - CXL Early Discovery Table (ACPI 6.4) > + * Version 1 > + * > + ******************************************************************************/ > + > +struct acpi_table_cedt { > + struct acpi_table_header header; /* Common ACPI table header */ > + u32 reserved; > +}; > + > +/* Values for CEDT structure types */ > + > +enum acpi_cedt_type { > + ACPI_CEDT_TYPE_HOST_BRIDGE = 0, /* CHBS - CXL Host Bridge Structure */ > + ACPI_CEDT_TYPE_CFMWS = 1, /* CFMWS - CXL Fixed Memory Window Structure */ This isn't in the 2.0 spec, so I guess also part of some proposed changes. > +}; > + > +struct acpi_cedt_structure { > + u8 type; > + u8 reserved; > + u16 length; > +}; > + > +/* > + * CEDT Structures, correspond to Type in struct acpi_cedt_structure > + */ > + > +/* 0: CXL Host Bridge Structure */ > + > +struct acpi_cedt_chbs { > + struct acpi_cedt_structure header; > + u32 uid; > + u32 version; > + u32 reserved1; > + u64 base; > + u32 length; > + u32 reserved2; > +}; > + > +/* Values for version field above */ > + > +#define ACPI_CEDT_CHBS_VERSION_CXL11 (0) > +#define ACPI_CEDT_CHBS_VERSION_CXL20 (1) > + > +/* Values for length field above */ > + > +#define ACPI_CEDT_CHBS_LENGTH_CXL11 (0x2000) > +#define ACPI_CEDT_CHBS_LENGTH_CXL20 (0x10000) > + > /* Reset to default packing */ > > #pragma pack()
On Mon, 2020-11-16 at 17:59 +0000, Jonathan Cameron wrote: > On Tue, 10 Nov 2020 21:43:48 -0800 > Ben Widawsky <ben.widawsky@intel.com> wrote: > > > From: Vishal Verma <vishal.l.verma@intel.com> > > > > Add an acpi_cxl module to coordinate the ACPI portions of the CXL > > (Compute eXpress Link) interconnect. This driver binds to ACPI0017 > > objects in the ACPI tree, and coordinates access to the resources > > provided by the ACPI CEDT (CXL Early Discovery Table). > > I think the qemu series notes that this ACPI0017 is just a proposal at > this stage. Please make sure that's highlighted here as well unless > that status is out of date. Hi Jonathan, Thank you for the review. The cover letter talks about this, but I agree it would be worth repeating in this patch briefly as I did with the OSC patch. If it is still in a proposal state by the next posting, I'll make sure to add a note here too. > > > + > > +static void acpi_cxl_desc_init(struct acpi_cxl_desc *acpi_desc, struct device *dev) > > +{ > > + dev_set_drvdata(dev, acpi_desc); > > No need to have this wrapper + it hides the fact you are not just initialsing > the acpi_desc structure. > > > + acpi_desc->dev = dev; > > +} > > + > > +static void acpi_cedt_put_table(void *table) > > +{ > > + acpi_put_table(table); > > +} > > + > > +static int acpi_cxl_add(struct acpi_device *adev) > > +{ > > + struct acpi_cxl_desc *acpi_desc; > > + struct device *dev = &adev->dev; > > + struct acpi_table_header *tbl; > > + acpi_status status = AE_OK; > > Set below, so don't do it here. > > > + acpi_size sz; > > + int rc = 0; > > Set in paths in which it's used so don't do it here. > > > + > > + status = acpi_get_table(ACPI_SIG_CEDT, 0, &tbl); > > + if (ACPI_FAILURE(status)) { > > + dev_err(dev, "failed to find CEDT at startup\n"); > > + return 0; > > + } > > + > > + rc = devm_add_action_or_reset(dev, acpi_cedt_put_table, tbl); > > + if (rc) > > + return rc; > > blank line here preferred for readability (do something, then check errors as > block) > > > + sz = tbl->length; > > + dev_info(dev, "found CEDT at startup: %lld bytes\n", sz); > > + > > + acpi_desc = devm_kzalloc(dev, sizeof(*acpi_desc), GFP_KERNEL); > > + if (!acpi_desc) > > + return -ENOMEM; > > blank line here slightly helps readability. > > > + acpi_cxl_desc_init(acpi_desc, &adev->dev); > > + > > + acpi_desc->acpi_header = *tbl; > > + > > + return 0; > > +} > > + > > +static int acpi_cxl_remove(struct acpi_device *adev) > > +{ > > + return 0; > > Don't think empty remove is needed. > Agreed with all of the above, I'll fix for the next posting. > > + > > +/* Values for CEDT structure types */ > > + > > +enum acpi_cedt_type { > > + ACPI_CEDT_TYPE_HOST_BRIDGE = 0, /* CHBS - CXL Host Bridge Structure */ > > + ACPI_CEDT_TYPE_CFMWS = 1, /* CFMWS - CXL Fixed Memory Window Structure */ > > This isn't in the 2.0 spec, so I guess also part of some proposed changes. Yes this got in accidentally, will remove.
On Wed, Nov 11, 2020 at 6:45 AM Ben Widawsky <ben.widawsky@intel.com> wrote: > > From: Vishal Verma <vishal.l.verma@intel.com> > > Add an acpi_cxl module to coordinate the ACPI portions of the CXL > (Compute eXpress Link) interconnect. This driver binds to ACPI0017 > objects in the ACPI tree, and coordinates access to the resources > provided by the ACPI CEDT (CXL Early Discovery Table). > > It also coordinates operations of the root port _OSC object to notify > platform firmware that the OS has native support for the CXL > capabilities of endpoints. > > Note: the actbl1.h changes are speculative. The expectation is that they > will arrive through the ACPICA tree in due time. > > Cc: Ben Widawsky <ben.widawsky@intel.com> > Cc: Dan Williams <dan.j.williams@intel.com> > Signed-off-by: Vishal Verma <vishal.l.verma@intel.com> > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com> > --- > drivers/Kconfig | 1 + > drivers/Makefile | 1 + > drivers/cxl/Kconfig | 30 +++++++++++ > drivers/cxl/Makefile | 5 ++ > drivers/cxl/acpi.c | 119 ++++++++++++++++++++++++++++++++++++++++++ > drivers/cxl/acpi.h | 15 ++++++ > include/acpi/actbl1.h | 52 ++++++++++++++++++ > 7 files changed, 223 insertions(+) > create mode 100644 drivers/cxl/Kconfig > create mode 100644 drivers/cxl/Makefile > create mode 100644 drivers/cxl/acpi.c > create mode 100644 drivers/cxl/acpi.h > > diff --git a/drivers/Kconfig b/drivers/Kconfig > index dcecc9f6e33f..62c753a73651 100644 > --- a/drivers/Kconfig > +++ b/drivers/Kconfig > @@ -6,6 +6,7 @@ menu "Device Drivers" > source "drivers/amba/Kconfig" > source "drivers/eisa/Kconfig" > source "drivers/pci/Kconfig" > +source "drivers/cxl/Kconfig" > source "drivers/pcmcia/Kconfig" > source "drivers/rapidio/Kconfig" > > diff --git a/drivers/Makefile b/drivers/Makefile > index c0cd1b9075e3..5dad349de73b 100644 > --- a/drivers/Makefile > +++ b/drivers/Makefile > @@ -73,6 +73,7 @@ obj-$(CONFIG_NVM) += lightnvm/ > obj-y += base/ block/ misc/ mfd/ nfc/ > obj-$(CONFIG_LIBNVDIMM) += nvdimm/ > obj-$(CONFIG_DAX) += dax/ > +obj-$(CONFIG_CXL_BUS) += cxl/ > obj-$(CONFIG_DMA_SHARED_BUFFER) += dma-buf/ > obj-$(CONFIG_NUBUS) += nubus/ > obj-y += macintosh/ > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig > new file mode 100644 > index 000000000000..dd724bd364df > --- /dev/null > +++ b/drivers/cxl/Kconfig > @@ -0,0 +1,30 @@ > +# SPDX-License-Identifier: GPL-2.0-only > +menuconfig CXL_BUS > + tristate "CXL (Compute Express Link) Devices Support" > + help > + CXL is a bus that is electrically compatible with PCI-E, but layers > + three protocols on that signalling (CXL.io, CXL.cache, and CXL.mem). The > + CXL.cache protocol allows devices to hold cachelines locally, the > + CXL.mem protocol allows devices to be fully coherent memory targets, the > + CXL.io protocol is equivalent to PCI-E. Say 'y' to enable support for > + the configuration and management of devices supporting these protocols. > + > +if CXL_BUS > + > +config CXL_BUS_PROVIDER > + tristate > + > +config CXL_ACPI > + tristate "CXL Platform Support" > + depends on ACPI > + default CXL_BUS > + select CXL_BUS_PROVIDER > + help > + CXL Platform Support is a prerequisite for any CXL device driver that > + wants to claim ownership of the component register space. By default > + platform firmware assumes Linux is unaware of CXL capabilities and > + requires explicit opt-in. This platform component also mediates > + resources described by the CEDT (CXL Early Discovery Table) > + > + Say 'y' to enable CXL (Compute Express Link) drivers. > +endif > diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile > new file mode 100644 > index 000000000000..d38cd34a2582 > --- /dev/null > +++ b/drivers/cxl/Makefile > @@ -0,0 +1,5 @@ > +# SPDX-License-Identifier: GPL-2.0 > +obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o > + > +ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL > +cxl_acpi-y := acpi.o > diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c > new file mode 100644 > index 000000000000..26e4f73838a7 > --- /dev/null > +++ b/drivers/cxl/acpi.c > @@ -0,0 +1,119 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +/* > + * Copyright(c) 2020 Intel Corporation. All rights reserved. > + */ > +#include <linux/list_sort.h> > +#include <linux/module.h> > +#include <linux/mutex.h> > +#include <linux/sysfs.h> > +#include <linux/list.h> > +#include <linux/acpi.h> > +#include <linux/sort.h> > +#include <linux/pci.h> > +#include "acpi.h" > + > +static void acpi_cxl_desc_init(struct acpi_cxl_desc *acpi_desc, struct device *dev) > +{ > + dev_set_drvdata(dev, acpi_desc); > + acpi_desc->dev = dev; > +} > + > +static void acpi_cedt_put_table(void *table) > +{ > + acpi_put_table(table); > +} > + > +static int acpi_cxl_add(struct acpi_device *adev) > +{ > + struct acpi_cxl_desc *acpi_desc; > + struct device *dev = &adev->dev; > + struct acpi_table_header *tbl; > + acpi_status status = AE_OK; > + acpi_size sz; > + int rc = 0; > + > + status = acpi_get_table(ACPI_SIG_CEDT, 0, &tbl); > + if (ACPI_FAILURE(status)) { > + dev_err(dev, "failed to find CEDT at startup\n"); > + return 0; > + } > + > + rc = devm_add_action_or_reset(dev, acpi_cedt_put_table, tbl); > + if (rc) > + return rc; > + sz = tbl->length; > + dev_info(dev, "found CEDT at startup: %lld bytes\n", sz); > + > + acpi_desc = devm_kzalloc(dev, sizeof(*acpi_desc), GFP_KERNEL); > + if (!acpi_desc) > + return -ENOMEM; > + acpi_cxl_desc_init(acpi_desc, &adev->dev); > + > + acpi_desc->acpi_header = *tbl; > + > + return 0; > +} > + > +static int acpi_cxl_remove(struct acpi_device *adev) > +{ > + return 0; > +} > + > +static const struct acpi_device_id acpi_cxl_ids[] = { > + { "ACPI0017", 0 }, > + { "", 0 }, > +}; > +MODULE_DEVICE_TABLE(acpi, acpi_cxl_ids); > + > +static struct acpi_driver acpi_cxl_driver = { First of all, no new acpi_driver instances, pretty please! acpi_default_enumeration() should create a platform device with the ACPI0017 ID for you. Can't you provide a driver for this one? > + .name = KBUILD_MODNAME, > + .ids = acpi_cxl_ids, > + .ops = { > + .add = acpi_cxl_add, > + .remove = acpi_cxl_remove, > + }, > +}; > + > +/* > + * If/when CXL support is defined by other platform firmware the kernel > + * will need a mechanism to select between the platform specific version > + * of this routine, until then, hard-code ACPI assumptions > + */ > +int cxl_bus_prepared(struct pci_dev *pdev) > +{ > + struct acpi_device *adev; > + struct pci_dev *root_port; > + struct device *root; > + > + root_port = pcie_find_root_port(pdev); > + if (!root_port) > + return -ENXIO; > + > + root = root_port->dev.parent; > + if (!root) > + return -ENXIO; > + > + adev = ACPI_COMPANION(root); > + if (!adev) > + return -ENXIO; > + > + /* TODO: OSC enabling */ > + > + return 0; > +} > +EXPORT_SYMBOL_GPL(cxl_bus_prepared); > + > +static __init int acpi_cxl_init(void) > +{ > + return acpi_bus_register_driver(&acpi_cxl_driver); > +} > + > +static __exit void acpi_cxl_exit(void) > +{ > + acpi_bus_unregister_driver(&acpi_cxl_driver); > +} > + > +module_init(acpi_cxl_init); > +module_exit(acpi_cxl_exit); > +MODULE_LICENSE("GPL v2"); > +MODULE_AUTHOR("Intel Corporation"); > diff --git a/drivers/cxl/acpi.h b/drivers/cxl/acpi.h > new file mode 100644 > index 000000000000..011505475cc6 > --- /dev/null > +++ b/drivers/cxl/acpi.h > @@ -0,0 +1,15 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +// Copyright(c) 2020 Intel Corporation. All rights reserved. > + > +#ifndef __CXL_ACPI_H__ > +#define __CXL_ACPI_H__ > +#include <linux/acpi.h> > + > +struct acpi_cxl_desc { > + struct acpi_table_header acpi_header; > + struct device *dev; > +}; > + > +int cxl_bus_prepared(struct pci_dev *pci_dev); > + > +#endif /* __CXL_ACPI_H__ */ > diff --git a/include/acpi/actbl1.h b/include/acpi/actbl1.h > index 43549547ed3e..70f745f526e3 100644 > --- a/include/acpi/actbl1.h > +++ b/include/acpi/actbl1.h > @@ -28,6 +28,7 @@ > #define ACPI_SIG_BERT "BERT" /* Boot Error Record Table */ > #define ACPI_SIG_BGRT "BGRT" /* Boot Graphics Resource Table */ > #define ACPI_SIG_BOOT "BOOT" /* Simple Boot Flag Table */ > +#define ACPI_SIG_CEDT "CEDT" /* CXL Early Discovery Table */ > #define ACPI_SIG_CPEP "CPEP" /* Corrected Platform Error Polling table */ > #define ACPI_SIG_CSRT "CSRT" /* Core System Resource Table */ > #define ACPI_SIG_DBG2 "DBG2" /* Debug Port table type 2 */ > @@ -1624,6 +1625,57 @@ struct acpi_ibft_target { > u16 reverse_chap_secret_offset; > }; > > +/******************************************************************************* > + * > + * CEDT - CXL Early Discovery Table (ACPI 6.4) > + * Version 1 > + * > + ******************************************************************************/ > + > +struct acpi_table_cedt { > + struct acpi_table_header header; /* Common ACPI table header */ > + u32 reserved; > +}; > + > +/* Values for CEDT structure types */ > + > +enum acpi_cedt_type { > + ACPI_CEDT_TYPE_HOST_BRIDGE = 0, /* CHBS - CXL Host Bridge Structure */ > + ACPI_CEDT_TYPE_CFMWS = 1, /* CFMWS - CXL Fixed Memory Window Structure */ > +}; > + > +struct acpi_cedt_structure { > + u8 type; > + u8 reserved; > + u16 length; > +}; > + > +/* > + * CEDT Structures, correspond to Type in struct acpi_cedt_structure > + */ > + > +/* 0: CXL Host Bridge Structure */ > + > +struct acpi_cedt_chbs { > + struct acpi_cedt_structure header; > + u32 uid; > + u32 version; > + u32 reserved1; > + u64 base; > + u32 length; > + u32 reserved2; > +}; > + > +/* Values for version field above */ > + > +#define ACPI_CEDT_CHBS_VERSION_CXL11 (0) > +#define ACPI_CEDT_CHBS_VERSION_CXL20 (1) > + > +/* Values for length field above */ > + > +#define ACPI_CEDT_CHBS_LENGTH_CXL11 (0x2000) > +#define ACPI_CEDT_CHBS_LENGTH_CXL20 (0x10000) > + > /* Reset to default packing */ > > #pragma pack() > -- > 2.29.2 >
On Tue, 10 Nov 2020 21:43:50 -0800 Ben Widawsky <ben.widawsky@intel.com> wrote: > From: Dan Williams <dan.j.williams@intel.com> > > The CXL.mem protocol allows a device to act as a provider of "System > RAM" and/or "Persistent Memory" that is fully coherent as if the memory > was attached to the typical CPU memory controller. > > The memory range exported by the device may optionally be described by > the platform firmware memory map, or by infrastructure like LIBNVDIMM to > provision persistent memory capacity from one, or more, CXL.mem devices. > > A pre-requisite for Linux-managed memory-capacity provisioning is this > cxl_mem driver that can speak the "type-3 mailbox" protocol. > > For now just land the driver boiler-plate and fill it in with > functionality in subsequent commits. > > Signed-off-by: Dan Williams <dan.j.williams@intel.com> > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com> I've tried to avoid repeats, so mostly this is me moaning about naming! Jonathan > --- > drivers/cxl/Kconfig | 20 +++++++++++ > drivers/cxl/Makefile | 2 ++ > drivers/cxl/mem.c | 82 ++++++++++++++++++++++++++++++++++++++++++++ > drivers/cxl/pci.h | 15 ++++++++ > 4 files changed, 119 insertions(+) > create mode 100644 drivers/cxl/mem.c > create mode 100644 drivers/cxl/pci.h > > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig > index dd724bd364df..15548f5c77ff 100644 > --- a/drivers/cxl/Kconfig > +++ b/drivers/cxl/Kconfig > @@ -27,4 +27,24 @@ config CXL_ACPI > resources described by the CEDT (CXL Early Discovery Table) > > Say 'y' to enable CXL (Compute Express Link) drivers. > + > +config CXL_MEM > + tristate "CXL.mem Device Support" > + depends on PCI && CXL_BUS_PROVIDER != n > + default m if CXL_BUS_PROVIDER > + help > + The CXL.mem protocol allows a device to act as a provider of > + "System RAM" and/or "Persistent Memory" that is fully coherent > + as if the memory was attached to the typical CPU memory > + controller. > + > + Say 'y/m' to enable a driver named "cxl_mem.ko" that will attach > + to CXL.mem devices for configuration, provisioning, and health > + monitoring, the so called "type-3 mailbox". Note, this driver > + is required for dynamic provisioning of CXL.mem attached > + memory, a pre-requisite for persistent memory support, but > + devices that provide volatile memory may be fully described by > + existing platform firmware memory enumeration. > + > + If unsure say 'n'. > endif > diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile > index d38cd34a2582..97fdffb00f2d 100644 > --- a/drivers/cxl/Makefile > +++ b/drivers/cxl/Makefile > @@ -1,5 +1,7 @@ > # SPDX-License-Identifier: GPL-2.0 > obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o > +obj-$(CONFIG_CXL_MEM) += cxl_mem.o > > ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL > cxl_acpi-y := acpi.o > +cxl_mem-y := mem.o > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c > new file mode 100644 > index 000000000000..aa7d881fa47b > --- /dev/null > +++ b/drivers/cxl/mem.c > @@ -0,0 +1,82 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +// Copyright(c) 2020 Intel Corporation. All rights reserved. > +#include <linux/module.h> > +#include <linux/pci.h> > +#include <linux/io.h> > +#include "acpi.h" > +#include "pci.h" > + > +struct cxl_mem { > + void __iomem *regs; > +}; > + > +static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec) > +{ > + int pos; > + > + pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DVSEC); > + if (!pos) > + return 0; > + > + while (pos) { > + u16 vendor, id; > + > + pci_read_config_word(pdev, pos + PCI_DVSEC_VENDOR_OFFSET, &vendor); > + pci_read_config_word(pdev, pos + PCI_DVSEC_ID_OFFSET, &id); > + if (vendor == PCI_DVSEC_VENDOR_CXL && dvsec == id) > + return pos; > + > + pos = pci_find_next_ext_capability(pdev, pos, PCI_EXT_CAP_ID_DVSEC); This is good generic code and wouldn't cause much backport effort (even if needed to bring in a local copy), so perhaps make it a generic function and move to core PCI code? Mind you I guess that can happen the 'second' time someone wants to find a DVSEC. > + } > + > + return 0; > +} > + > +static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) > +{ > + struct device *dev = &pdev->dev; > + struct cxl_mem *cxlm; > + int rc, regloc; > + > + rc = cxl_bus_prepared(pdev); > + if (rc != 0) { > + dev_err(dev, "failed to acquire interface\n"); > + return rc; > + } > + > + regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC); > + if (!regloc) { > + dev_err(dev, "register location dvsec not found\n"); > + return -ENXIO; > + } > + > + cxlm = devm_kzalloc(dev, sizeof(*cxlm), GFP_KERNEL); > + if (!cxlm) > + return -ENOMEM; > + > + return 0; > +} > + > +static void cxl_mem_remove(struct pci_dev *pdev) > +{ > +} I'd bring this in only when needed in later patch. > + > +static const struct pci_device_id cxl_mem_pci_tbl[] = { > + /* PCI class code for CXL.mem Type-3 Devices */ > + { PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID, > + PCI_CLASS_MEMORY_CXL, 0xffffff, 0 }, > + { /* terminate list */ }, > +}; > +MODULE_DEVICE_TABLE(pci, cxl_mem_pci_tbl); > + > +static struct pci_driver cxl_mem_driver = { > + .name = KBUILD_MODNAME, > + .id_table = cxl_mem_pci_tbl, > + .probe = cxl_mem_probe, > + .remove = cxl_mem_remove, > +}; > + > +MODULE_LICENSE("GPL v2"); > +MODULE_AUTHOR("Intel Corporation"); > +module_pci_driver(cxl_mem_driver); > +MODULE_IMPORT_NS(CXL); > diff --git a/drivers/cxl/pci.h b/drivers/cxl/pci.h > new file mode 100644 > index 000000000000..beb03921e6da > --- /dev/null > +++ b/drivers/cxl/pci.h > @@ -0,0 +1,15 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +// Copyright(c) 2020 Intel Corporation. All rights reserved. > +#ifndef __CXL_PCI_H__ > +#define __CXL_PCI_H__ > + > +#define PCI_CLASS_MEMORY_CXL 0x050210 > + > +#define PCI_EXT_CAP_ID_DVSEC 0x23 > +#define PCI_DVSEC_VENDOR_CXL 0x1E98 Hmm. The magic question of what to call a vendor ID that isn't a vendor ID but just a magic number that talks like a duck and quacks like a duck (for anyone wondering what I'm talking about, there is a nice bit of legal boilerplate on this in the CXL spec) This name is definitely not accurate however. PCI_UNIQUE_VALUE_CXL maybe? It is used for other things than DVSEC (VDMs etc), though possibly this is the only software visible use. > +#define PCI_DVSEC_VENDOR_OFFSET 0x4 > +#define PCI_DVSEC_ID_OFFSET 0x8 Put a line break here perhaps and maybe a spec reference to where to find the various DVSEC IDs. > +#define PCI_DVSEC_ID_CXL 0x0 That's definitely a confusing name as well. PCI_DEVSEC_ID_CXL_DEVICE maybe? > +#define PCI_DVSEC_ID_CXL_REGLOC 0x8 > + > +#endif /* __CXL_PCI_H__ */
On Tue, 10 Nov 2020 21:43:52 -0800 Ben Widawsky <ben.widawsky@intel.com> wrote: > CXL devices contain an array of capabilities that describe the > interactions software can interact with the device, or firmware running > on the device. A CXL compliant device must implement the device status > and the mailbox capability. A CXL compliant memory device must implement > the memory device capability. > > Each of the capabilities can [will] provide an offset within the MMIO > region for interacting with the CXL device. > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com> A few really minor things in this one. Jonathan > --- > drivers/cxl/cxl.h | 89 +++++++++++++++++++++++++++++++++++++++++++++++ > drivers/cxl/mem.c | 58 +++++++++++++++++++++++++++--- > 2 files changed, 143 insertions(+), 4 deletions(-) > create mode 100644 drivers/cxl/cxl.h > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h > new file mode 100644 > index 000000000000..02858ae63d6d > --- /dev/null > +++ b/drivers/cxl/cxl.h > @@ -0,0 +1,89 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +// Copyright(c) 2020 Intel Corporation. All rights reserved. > + > +#ifndef __CXL_H__ > +#define __CXL_H__ > + > +/* Device */ > +#define CXLDEV_CAP_ARRAY_REG 0x0 > +#define CXLDEV_CAP_ARRAY_CAP_ID 0 > +#define CXLDEV_CAP_ARRAY_ID(x) ((x) & 0xffff) > +#define CXLDEV_CAP_ARRAY_COUNT(x) (((x) >> 32) & 0xffff) > + > +#define CXL_CAPABILITIES_CAP_ID_DEVICE_STATUS 1 I'm not sure what you can do about consistent naming, but perhaps this really does need to be CXLDEV_CAP_CAP_ID_x however silly that looks. It's a funny exercise I've only seen done once in a spec, but I wish everyone would try working out what their fully expanded field names will end up as and make sure the long form naming shortens to something sensible. It definitely helps with clarity! > +#define CXL_CAPABILITIES_CAP_ID_PRIMARY_MAILBOX 2 > +#define CXL_CAPABILITIES_CAP_ID_SECONDARY_MAILBOX 3 > +#define CXL_CAPABILITIES_CAP_ID_MEMDEV 0x4000 > + > +/* Mailbox */ > +#define CXLDEV_MB_CAPS 0x00 Elsewhere you've used _OFFSET postfix. That's useful so I'd do it here as well. Cross references to the spec section always appreciated as well! > +#define CXLDEV_MB_CAP_PAYLOAD_SIZE(cap) ((cap) & 0x1F) > +#define CXLDEV_MB_CTRL 0x04 > +#define CXLDEV_MB_CMD 0x08 > +#define CXLDEV_MB_STATUS 0x10 > +#define CXLDEV_MB_BG_CMD_STATUS 0x18 > + > +struct cxl_mem { > + struct pci_dev *pdev; > + void __iomem *regs; > + > + /* Cap 0000h */ What are the numbers here? These capabilities have too many indexes associated with them in different ways for it to be obvious, so perhaps more detail in the comments? > + struct { > + void __iomem *regs; > + } status; > + > + /* Cap 0002h */ > + struct { > + void __iomem *regs; > + size_t payload_size; > + } mbox; > + > + /* Cap 0040h */ > + struct { > + void __iomem *regs; > + } mem; > +};
On Tue, 10 Nov 2020 21:43:54 -0800 Ben Widawsky <ben.widawsky@intel.com> wrote: > Create a function to handle sending a command, optionally with a > payload, to the memory device, polling on a result, and then optionally > copying out the payload. The algorithm for doing this come straight out > of the CXL 2.0 specification. > > Primary mailboxes are capable of generating an interrupt when submitting > a command in the background. That implementation is saved for a later > time. > > Secondary mailboxes aren't implemented at this time. > > WARNING: This is untested with actual timeouts occurring. > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com> Question inline for why the preempt / local timer dance is worth bothering with. What am I missing? Thanks, Jonathan > --- > drivers/cxl/cxl.h | 16 +++++++ > drivers/cxl/mem.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 123 insertions(+) > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h > index 482fc9cdc890..f49ab80f68bd 100644 > --- a/drivers/cxl/cxl.h > +++ b/drivers/cxl/cxl.h > @@ -21,8 +21,12 @@ > #define CXLDEV_MB_CTRL 0x04 > #define CXLDEV_MB_CTRL_DOORBELL BIT(0) > #define CXLDEV_MB_CMD 0x08 > +#define CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT 16 > #define CXLDEV_MB_STATUS 0x10 > +#define CXLDEV_MB_STATUS_RET_CODE_SHIFT 32 > +#define CXLDEV_MB_STATUS_RET_CODE_MASK 0xffff > #define CXLDEV_MB_BG_CMD_STATUS 0x18 > +#define CXLDEV_MB_PAYLOAD 0x20 > > /* Memory Device */ > #define CXLMDEV_STATUS 0 > @@ -114,4 +118,16 @@ static inline u64 __cxl_raw_read_reg64(struct cxl_mem *cxlm, u32 reg) > > return readq(reg_addr + reg); > } > + > +static inline void cxl_mbox_payload_fill(struct cxl_mem *cxlm, u8 *input, > + unsigned int length) > +{ > + memcpy_toio(cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, input, length); > +} > + > +static inline void cxl_mbox_payload_drain(struct cxl_mem *cxlm, > + u8 *output, unsigned int length) > +{ > + memcpy_fromio(output, cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, length); > +} > #endif /* __CXL_H__ */ > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c > index 9fd2d1daa534..08913360d500 100644 > --- a/drivers/cxl/mem.c > +++ b/drivers/cxl/mem.c > @@ -1,5 +1,6 @@ > // SPDX-License-Identifier: GPL-2.0-only > // Copyright(c) 2020 Intel Corporation. All rights reserved. > +#include <linux/sched/clock.h> > #include <linux/module.h> > #include <linux/pci.h> > #include <linux/io.h> > @@ -7,6 +8,112 @@ > #include "pci.h" > #include "cxl.h" > > +struct mbox_cmd { > + u16 cmd; > + u8 *payload; > + size_t payload_size; > + u16 return_code; > +}; > + > +static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm) > +{ > + u64 start, now; > + int cpu, ret, timeout = 2000000000; > + > + start = local_clock(); > + preempt_disable(); > + cpu = smp_processor_id(); > + for (;;) { > + now = local_clock(); > + preempt_enable(); What do we ever do with this mailbox that is particularly performance critical? I'd like to understand why we care enough to mess around with the preemption changes and local clock etc. > + if ((cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL) & > + CXLDEV_MB_CTRL_DOORBELL) == 0) { > + ret = 0; > + break; > + } > + > + if (now - start >= timeout) { > + ret = -ETIMEDOUT; > + break; > + } > + > + cpu_relax(); > + preempt_disable(); > + if (unlikely(cpu != smp_processor_id())) { > + timeout -= (now - start); > + cpu = smp_processor_id(); > + start = local_clock(); > + } > + } > + > + return ret; > +} > + > +/* > + * Returns 0 if the doorbell transaction was successful from a protocol level. > + * Caller should check the return code in @mbox_cmd to make sure it succeeded. > + */ > +static int __maybe_unused cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm, struct mbox_cmd *mbox_cmd) > +{ > + u64 cmd, status; > + int rc; > + > + lockdep_assert_held(&cxlm->mbox_lock); > + > + /* > + * Here are the steps from 8.2.8.4 of the CXL 2.0 spec. > + * 1. Caller reads MB Control Register to verify doorbell is clear > + * 2. Caller writes Command Register > + * 3. Caller writes Command Payload Registers if input payload is non-empty > + * 4. Caller writes MB Control Register to set doorbell > + * 5. Caller either polls for doorbell to be clear or waits for interrupt if configured > + * 6. Caller reads MB Status Register to fetch Return code > + * 7. If command successful, Caller reads Command Register to get Payload Length > + * 8. If output payload is non-empty, host reads Command Payload Registers > + */ > + > + cmd = mbox_cmd->cmd; > + if (mbox_cmd->payload_size) { > + /* #3 */ Having just given the steps above, having them out of order feels like it needs a comment to state why. > + cmd |= mbox_cmd->payload_size > + << CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT; > + cxl_mbox_payload_fill(cxlm, mbox_cmd->payload, mbox_cmd->payload_size); > + } > + > + /* #2 */ > + cxl_write_mbox_reg64(cxlm, CXLDEV_MB_CMD, cmd); > + > + /* #4 */ > + cxl_write_mbox_reg32(cxlm, CXLDEV_MB_CTRL, CXLDEV_MB_CTRL_DOORBELL); > + > + /* #5 */ > + rc = cxldev_wait_for_doorbell(cxlm); > + if (rc == -ETIMEDOUT) { > + dev_warn(&cxlm->pdev->dev, "Mailbox command timed out\n"); > + return rc; > + } > + > + /* #6 */ > + status = cxl_read_mbox_reg64(cxlm, CXLDEV_MB_STATUS); > + cmd = cxl_read_mbox_reg64(cxlm, CXLDEV_MB_CMD); > + > + mbox_cmd->return_code = (status >> CXLDEV_MB_STATUS_RET_CODE_SHIFT) & > + CXLDEV_MB_STATUS_RET_CODE_MASK; > + > + /* There was a problem, let the caller deal with it */ > + if (mbox_cmd->return_code != 0) > + return 0; > + > + /* #7 */ > + mbox_cmd->payload_size = cmd >> CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT; > + > + /* #8 */ > + if (mbox_cmd->payload_size) > + cxl_mbox_payload_drain(cxlm, mbox_cmd->payload, mbox_cmd->payload_size); > + > + return 0; > +} > + > static int cxl_mem_mbox_get(struct cxl_mem *cxlm) > { > u64 md_status;
On Tue, 10 Nov 2020 21:43:55 -0800 Ben Widawsky <ben.widawsky@intel.com> wrote: > From: Dan Williams <dan.j.williams@intel.com> > > Create the /sys/bus/cxl hierarchy to enumerate memory devices > (per-endpoint control devices), memory address space devices (platform > address ranges with interleaving, performance, and persistence > attributes), and memory regions (active provisioned memory from an > address space device that is in use as System RAM or delegated to > libnvdimm as Persistent Memory regions). > > For now, only the per-endpoint control devices are registered on the > 'cxl' bus. Reviewing ABI without documentation is challenging even when it's simple so please add that for v2. This patch feels somewhat unpolished, but I guess it is mainly here to give an illustration of how stuff might fit together rather than any expectation of detailed review. So in that spirit I've just pointed out stuff that jumped out at me during a quick read through. Thanks, Jonathan > > Signed-off-by: Dan Williams <dan.j.williams@intel.com> > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com> > --- > drivers/cxl/Makefile | 2 + > drivers/cxl/bus.c | 35 ++++++ > drivers/cxl/bus.h | 8 ++ > drivers/cxl/cxl.h | 33 +++++ > drivers/cxl/mem.c | 287 ++++++++++++++++++++++++++++++++++++++++++- > 5 files changed, 359 insertions(+), 6 deletions(-) > create mode 100644 drivers/cxl/bus.c > create mode 100644 drivers/cxl/bus.h > > diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile > index 97fdffb00f2d..1cc032092852 100644 > --- a/drivers/cxl/Makefile > +++ b/drivers/cxl/Makefile > @@ -1,7 +1,9 @@ > # SPDX-License-Identifier: GPL-2.0 > obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o > obj-$(CONFIG_CXL_MEM) += cxl_mem.o > +obj-$(CONFIG_CXL_BUS_PROVIDER) += cxl_bus.o > > ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL > cxl_acpi-y := acpi.o > cxl_mem-y := mem.o > +cxl_bus-y := bus.o > diff --git a/drivers/cxl/bus.c b/drivers/cxl/bus.c > new file mode 100644 > index 000000000000..8594366955f7 > --- /dev/null > +++ b/drivers/cxl/bus.c > @@ -0,0 +1,35 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +// Copyright(c) 2020 Intel Corporation. All rights reserved. > +#include <linux/device.h> > +#include <linux/module.h> > + > +static struct bus_type cxl_bus_type = { > + .name = "cxl", > +}; > + > +int cxl_register(struct device *dev) > +{ > + int rc; > + > + dev->bus = &cxl_bus_type; > + rc = device_add(dev); > + if (rc) > + put_device(dev); > + return rc; > +} > +EXPORT_SYMBOL_GPL(cxl_register); > + > +static __init int cxl_bus_init(void) > +{ > + return bus_register(&cxl_bus_type); > +} > + > +static void cxl_bus_exit(void) > +{ > + bus_unregister(&cxl_bus_type); > +} > + > +module_init(cxl_bus_init); > +module_exit(cxl_bus_exit); > +MODULE_LICENSE("GPL v2"); > +MODULE_AUTHOR("Intel Corporation"); > diff --git a/drivers/cxl/bus.h b/drivers/cxl/bus.h > new file mode 100644 > index 000000000000..fe2bea2bbc3c > --- /dev/null > +++ b/drivers/cxl/bus.h > @@ -0,0 +1,8 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +// Copyright(c) 2020 Intel Corporation. All rights reserved. > +#ifndef __CXL_BUS_H__ > +#define __CXL_BUS_H__ > + > +int cxl_register(struct device *dev); > + > +#endif /* __CXL_BUS_H__ */ > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h > index f49ab80f68bd..cef5fd9ea68b 100644 > --- a/drivers/cxl/cxl.h > +++ b/drivers/cxl/cxl.h > @@ -3,6 +3,7 @@ > > #ifndef __CXL_H__ > #define __CXL_H__ > +#include <linux/range.h> > > /* Device */ > #define CXLDEV_CAP_ARRAY_REG 0x0 > @@ -52,12 +53,24 @@ > #define CXLMDEV_RESET_NEEDED_HOT 3 > #define CXLMDEV_RESET_NEEDED_CXL 4 > > +struct cxl_memdev; > struct cxl_mem { > struct pci_dev *pdev; > void __iomem *regs; > + struct cxl_memdev *cxlmd; > > spinlock_t mbox_lock; /* Protects device mailbox and firmware */ > > + struct { > + struct range range; > + } pmem; > + > + struct { > + struct range range; > + } ram; > + > + char firmware_version[0x10]; > + > /* Cap 0000h */ > struct { > void __iomem *regs; > @@ -130,4 +143,24 @@ static inline void cxl_mbox_payload_drain(struct cxl_mem *cxlm, > { > memcpy_fromio(output, cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, length); > } > + > +#define CXL_MBOX_IDENTIFY 0x4000 > + > +struct cxl_mbox_identify { > + char fw_revision[0x10]; > + __le64 total_capacity; > + __le64 volatile_capacity; > + __le64 persistent_capacity; > + __le64 partition_align; > + __le16 info_event_log_size; > + __le16 warning_event_log_size; > + __le16 failure_event_log_size; > + __le16 fatal_event_log_size; > + __le32 lsa_size; > + u8 poison_list_max_mer[3]; > + __le16 inject_poison_limit; > + u8 poison_caps; > + u8 qos_telemetry_caps; > +} __packed; > + > #endif /* __CXL_H__ */ > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c > index 08913360d500..54743d196feb 100644 > --- a/drivers/cxl/mem.c > +++ b/drivers/cxl/mem.c > @@ -2,11 +2,15 @@ > // Copyright(c) 2020 Intel Corporation. All rights reserved. > #include <linux/sched/clock.h> > #include <linux/module.h> > +#include <linux/mutex.h> > +#include <linux/cdev.h> > +#include <linux/idr.h> > #include <linux/pci.h> > #include <linux/io.h> > #include "acpi.h" > #include "pci.h" > #include "cxl.h" > +#include "bus.h" > > struct mbox_cmd { > u16 cmd; > @@ -15,6 +19,53 @@ struct mbox_cmd { > u16 return_code; > }; > > +/* > + * An entire PCI topology full of devices should be enough for any > + * config > + */ > +#define CXL_MEM_MAX_DEVS 65536 > + > +struct cxl_memdev { > + struct device dev; > + struct cxl_mem *cxlm; > + int id; > +}; > + > +static int cxl_mem_major; > +static struct cdev cxl_mem_cdev; > +static DEFINE_IDR(cxl_mem_idr); > +static DEFINE_MUTEX(cxl_memdev_lock); Define scope of this lock with a comment. > + > +static int cxl_mem_open(struct inode *inode, struct file *file) > +{ > + long minor = iminor(inode); > + struct cxl_memdev *cxlmd; > + > + rcu_read_lock(); > + cxlmd = idr_find(&cxl_mem_idr, minor); > + rcu_read_unlock(); > + > + if (!cxlmd) > + return -ENXIO; > + > + file->private_data = cxlmd; > + > + return 0; > +} > + > +static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg) > +{ > + return -ENOTTY; > +} > + > +static const struct file_operations cxl_mem_fops = { > + .owner = THIS_MODULE, > + .open = cxl_mem_open, > + .unlocked_ioctl = cxl_mem_ioctl, > + .compat_ioctl = compat_ptr_ioctl, > + .llseek = noop_llseek, > +}; > + > static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm) > { > u64 start, now; > @@ -53,7 +104,7 @@ static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm) > * Returns 0 if the doorbell transaction was successful from a protocol level. > * Caller should check the return code in @mbox_cmd to make sure it succeeded. > */ > -static int __maybe_unused cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm, struct mbox_cmd *mbox_cmd) > +static int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm, struct mbox_cmd *mbox_cmd) > { > u64 cmd, status; > int rc; > @@ -277,10 +328,185 @@ static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec) > return 0; > } > > +static struct cxl_memdev *to_cxl_memdev(struct device *dev) > +{ > + return container_of(dev, struct cxl_memdev, dev); > +} > + > +static void cxl_memdev_release(struct device *dev) > +{ > + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); > + > + mutex_lock(&cxl_memdev_lock); > + idr_remove(&cxl_mem_idr, cxlmd->id); > + mutex_unlock(&cxl_memdev_lock); > + > + kfree(cxlmd); > +} > + > +static char *cxl_memdev_devnode(struct device *dev, umode_t *mode, kuid_t *uid, kgid_t *gid) > +{ > + return kasprintf(GFP_KERNEL, "cxl/%s", dev_name(dev)); > +} > + > +static ssize_t firmware_version_show(struct device *dev, struct device_attribute *attr, char *buf) > +{ > + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); > + struct cxl_mem *cxlm = cxlmd->cxlm; > + > + return sprintf(buf, "%.16s\n", cxlm->firmware_version); > +} > +static DEVICE_ATTR_RO(firmware_version); > + > +static ssize_t ram_size_show(struct device *dev, struct device_attribute *attr, char *buf) > +{ > + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); > + struct cxl_mem *cxlm = cxlmd->cxlm; > + > + return sprintf(buf, "%#llx\n", (unsigned long long) range_len(&cxlm->ram.range)); > +} > +static struct device_attribute dev_attr_ram_size = __ATTR(size, 0444, ram_size_show, NULL); > + > +static ssize_t pmem_size_show(struct device *dev, struct device_attribute *attr, char *buf) > +{ > + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); > + struct cxl_mem *cxlm = cxlmd->cxlm; > + > + return sprintf(buf, "%#llx\n", (unsigned long long) range_len(&cxlm->pmem.range)); > +} > +static struct device_attribute dev_attr_pmem_size = __ATTR(size, 0444, pmem_size_show, NULL); > + > +static struct attribute *cxl_memdev_attributes[] = { > + &dev_attr_firmware_version.attr, > + NULL, > +}; > + > +static struct attribute *cxl_memdev_pmem_attributes[] = { > + &dev_attr_pmem_size.attr, It's simple, but should still have docs in Documentation/ABI/testing/sysfs... > + NULL, > +}; > + > +static struct attribute *cxl_memdev_ram_attributes[] = { > + &dev_attr_ram_size.attr, > + NULL, > +}; > + > +static struct attribute_group cxl_memdev_attribute_group = { > + .attrs = cxl_memdev_attributes, > +}; > + > +static struct attribute_group cxl_memdev_ram_attribute_group = { > + .name = "ram", > + .attrs = cxl_memdev_ram_attributes, > +}; > + > +static struct attribute_group cxl_memdev_pmem_attribute_group = { > + .name = "pmem", > + .attrs = cxl_memdev_pmem_attributes, > +}; > + > +static const struct attribute_group *cxl_memdev_attribute_groups[] = { > + &cxl_memdev_attribute_group, > + &cxl_memdev_ram_attribute_group, > + &cxl_memdev_pmem_attribute_group, > + NULL, > +}; > + > +static const struct device_type cxl_memdev_type = { > + .name = "cxl_memdev", > + .release = cxl_memdev_release, > + .devnode = cxl_memdev_devnode, > + .groups = cxl_memdev_attribute_groups, > +}; > + > +static struct cxl_memdev *cxl_mem_add_memdev(struct cxl_mem *cxlm) > +{ > + struct pci_dev *pdev = cxlm->pdev; > + struct cxl_memdev *cxlmd; > + struct device *dev; > + int id, rc; > + > + cxlmd = kzalloc(sizeof(*cxlmd), GFP_KERNEL); Maybe I missed it, but I'm not seeing this freed anywhere. > + if (!cxlmd) > + return ERR_PTR(-ENOMEM); > + > + cxlmd->cxlm = cxlm; > + cxlm->cxlmd = cxlmd; > + > + mutex_lock(&cxl_memdev_lock); > + id = idr_alloc(&cxl_mem_idr, cxlmd, 0, CXL_MEM_MAX_DEVS, GFP_KERNEL); > + mutex_unlock(&cxl_memdev_lock); > + if (id < 0) { > + rc = id; > + goto err_idr; > + } > + > + cxlmd->id = id; > + > + dev = &cxlmd->dev; > + > + device_initialize(dev); > + dev->parent = &pdev->dev; > + dev->devt = MKDEV(cxl_mem_major, id); > + dev->type = &cxl_memdev_type; > + dev_set_name(dev, "mem%d", id); blank line here > + rc = cxl_register(dev); > + if (rc) > + return ERR_PTR(rc); > + > + return cxlmd; > + > +err_idr: > + kfree(cxlmd); > + > + return ERR_PTR(rc); > +} > + ... > static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) > { > struct cxl_mem *cxlm = ERR_PTR(-ENXIO); > struct device *dev = &pdev->dev; > + struct cxl_memdev *cxlmd; > int rc, regloc, i; > > rc = cxl_bus_prepared(pdev); > @@ -319,20 +545,31 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) > if (rc) > return rc; > > - /* Check that hardware "looks" okay. */ > - rc = cxl_mem_mbox_get(cxlm); > + rc = cxl_mem_identify(cxlm); > if (rc) > return rc; > - > - cxl_mem_mbox_put(cxlm); It was kind of nice to see the flow earlier, but I'm also thinking it made a slightly harder to read patch. Hmm. Maybe just drop the version earlier in favour of a todo comment that you then do here? > dev_dbg(&pdev->dev, "CXL Memory Device Interface Up\n"); > + Nice to tidy that up by moving to earlier patch. > pci_set_drvdata(pdev, cxlm); > > + cxlmd = cxl_mem_add_memdev(cxlm); > + if (IS_ERR(cxlmd)) > + return PTR_ERR(cxlmd); Given we don't actually use cxlmd perhaps a simple return value of 0 or error would be better from cxl_mem_add_memdev() (I guess you may have follow up patches that do something with it here, though it feels wrong to ever do so given it is now registered and hence exposed to the system). > + > return 0; > } > > static void cxl_mem_remove(struct pci_dev *pdev) > { > + struct cxl_mem *cxlm = pci_get_drvdata(pdev); > + struct cxl_memdev *cxlmd = cxlm->cxlmd; > + > + device_lock(&cxlmd->dev); > + cxlm->cxlmd = NULL; > + cxlmd->cxlm = NULL; > + device_unlock(&cxlmd->dev); > + > + device_unregister(&cxlmd->dev); Why device_unregister last? Normally removing exposure to the system is the first thing you do in a remove() call. Particularly as you'll get NULL ptr dereferences if anyone manages a sysfs read between the pointers being set to NULL above and the device_unregister() taking away the sysfs files. > } > > static const struct pci_device_id cxl_mem_pci_tbl[] = { > @@ -350,7 +587,45 @@ static struct pci_driver cxl_mem_driver = { > .remove = cxl_mem_remove, > }; > > +static __init int cxl_mem_init(void) > +{ > + int rc; > + dev_t devt; > + > + rc = alloc_chrdev_region(&devt, 0, CXL_MEM_MAX_DEVS, "cxl"); > + if (rc) > + return rc; > + > + cxl_mem_major = MAJOR(devt); > + > + cdev_init(&cxl_mem_cdev, &cxl_mem_fops); > + rc = cdev_add(&cxl_mem_cdev, MKDEV(cxl_mem_major, 0), CXL_MEM_MAX_DEVS); > + if (rc) > + goto err_cdev; > + > + rc = pci_register_driver(&cxl_mem_driver); > + if (rc) > + goto err_driver; > + > + return 0; > + > +err_driver: > + cdev_del(&cxl_mem_cdev); > +err_cdev: > + unregister_chrdev_region(MKDEV(cxl_mem_major, 0), CXL_MEM_MAX_DEVS); > + > + return rc; > +} > + > +static __exit void cxl_mem_exit(void) > +{ > + pci_unregister_driver(&cxl_mem_driver); > + unregister_chrdev_region(MKDEV(cxl_mem_major, 0), CXL_MEM_MAX_DEVS); > + cdev_del(&cxl_mem_cdev); Ordering? cdev_dev should be before unregister_chrdev_region to match error handling in init() > +} > + > MODULE_LICENSE("GPL v2"); > MODULE_AUTHOR("Intel Corporation"); > -module_pci_driver(cxl_mem_driver); > +module_init(cxl_mem_init); > +module_exit(cxl_mem_exit); > MODULE_IMPORT_NS(CXL);
On 20-11-17 15:31:22, Jonathan Cameron wrote: > On Tue, 10 Nov 2020 21:43:54 -0800 > Ben Widawsky <ben.widawsky@intel.com> wrote: > > > Create a function to handle sending a command, optionally with a > > payload, to the memory device, polling on a result, and then optionally > > copying out the payload. The algorithm for doing this come straight out > > of the CXL 2.0 specification. > > > > Primary mailboxes are capable of generating an interrupt when submitting > > a command in the background. That implementation is saved for a later > > time. > > > > Secondary mailboxes aren't implemented at this time. > > > > WARNING: This is untested with actual timeouts occurring. > > > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com> > > Question inline for why the preempt / local timer dance is worth bothering with. > What am I missing? > > Thanks, > > Jonathan > > > --- > > drivers/cxl/cxl.h | 16 +++++++ > > drivers/cxl/mem.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++ > > 2 files changed, 123 insertions(+) > > > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h > > index 482fc9cdc890..f49ab80f68bd 100644 > > --- a/drivers/cxl/cxl.h > > +++ b/drivers/cxl/cxl.h > > @@ -21,8 +21,12 @@ > > #define CXLDEV_MB_CTRL 0x04 > > #define CXLDEV_MB_CTRL_DOORBELL BIT(0) > > #define CXLDEV_MB_CMD 0x08 > > +#define CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT 16 > > #define CXLDEV_MB_STATUS 0x10 > > +#define CXLDEV_MB_STATUS_RET_CODE_SHIFT 32 > > +#define CXLDEV_MB_STATUS_RET_CODE_MASK 0xffff > > #define CXLDEV_MB_BG_CMD_STATUS 0x18 > > +#define CXLDEV_MB_PAYLOAD 0x20 > > > > /* Memory Device */ > > #define CXLMDEV_STATUS 0 > > @@ -114,4 +118,16 @@ static inline u64 __cxl_raw_read_reg64(struct cxl_mem *cxlm, u32 reg) > > > > return readq(reg_addr + reg); > > } > > + > > +static inline void cxl_mbox_payload_fill(struct cxl_mem *cxlm, u8 *input, > > + unsigned int length) > > +{ > > + memcpy_toio(cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, input, length); > > +} > > + > > +static inline void cxl_mbox_payload_drain(struct cxl_mem *cxlm, > > + u8 *output, unsigned int length) > > +{ > > + memcpy_fromio(output, cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, length); > > +} > > #endif /* __CXL_H__ */ > > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c > > index 9fd2d1daa534..08913360d500 100644 > > --- a/drivers/cxl/mem.c > > +++ b/drivers/cxl/mem.c > > @@ -1,5 +1,6 @@ > > // SPDX-License-Identifier: GPL-2.0-only > > // Copyright(c) 2020 Intel Corporation. All rights reserved. > > +#include <linux/sched/clock.h> > > #include <linux/module.h> > > #include <linux/pci.h> > > #include <linux/io.h> > > @@ -7,6 +8,112 @@ > > #include "pci.h" > > #include "cxl.h" > > > > +struct mbox_cmd { > > + u16 cmd; > > + u8 *payload; > > + size_t payload_size; > > + u16 return_code; > > +}; > > + > > +static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm) > > +{ > > + u64 start, now; > > + int cpu, ret, timeout = 2000000000; > > + > > + start = local_clock(); > > + preempt_disable(); > > + cpu = smp_processor_id(); > > + for (;;) { > > + now = local_clock(); > > + preempt_enable(); > > What do we ever do with this mailbox that is particularly > performance critical? I'd like to understand why we care enough > to mess around with the preemption changes and local clock etc. > It is quite obviously a premature optimization at this point (since we only support a single command in QEMU). However, the polling can be anywhere from instant to 2 seconds. QEMU implementation aside again, some devices may never support interrupts on completion, and so I thought providing a poll function now that is capable of working for most [all?] cases was wise. > > + if ((cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL) & > > + CXLDEV_MB_CTRL_DOORBELL) == 0) { > > + ret = 0; > > + break; > > + } > > + > > + if (now - start >= timeout) { > > + ret = -ETIMEDOUT; > > + break; > > + } > > + > > + cpu_relax(); > > + preempt_disable(); > > + if (unlikely(cpu != smp_processor_id())) { > > + timeout -= (now - start); > > + cpu = smp_processor_id(); > > + start = local_clock(); > > + } > > + } > > + > > + return ret; > > +} > > + > > +/* > > + * Returns 0 if the doorbell transaction was successful from a protocol level. > > + * Caller should check the return code in @mbox_cmd to make sure it succeeded. > > + */ > > +static int __maybe_unused cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm, struct mbox_cmd *mbox_cmd) > > +{ > > + u64 cmd, status; > > + int rc; > > + > > + lockdep_assert_held(&cxlm->mbox_lock); > > + > > + /* > > + * Here are the steps from 8.2.8.4 of the CXL 2.0 spec. > > + * 1. Caller reads MB Control Register to verify doorbell is clear > > + * 2. Caller writes Command Register > > + * 3. Caller writes Command Payload Registers if input payload is non-empty > > + * 4. Caller writes MB Control Register to set doorbell > > + * 5. Caller either polls for doorbell to be clear or waits for interrupt if configured > > + * 6. Caller reads MB Status Register to fetch Return code > > + * 7. If command successful, Caller reads Command Register to get Payload Length > > + * 8. If output payload is non-empty, host reads Command Payload Registers > > + */ > > + > > + cmd = mbox_cmd->cmd; > > + if (mbox_cmd->payload_size) { > > + /* #3 */ > > Having just given the steps above, having them out of order feels like it needs > a comment to state why. > > > + cmd |= mbox_cmd->payload_size > > + << CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT; > > + cxl_mbox_payload_fill(cxlm, mbox_cmd->payload, mbox_cmd->payload_size); > > + } > > + > > + /* #2 */ > > + cxl_write_mbox_reg64(cxlm, CXLDEV_MB_CMD, cmd); > > + > > + /* #4 */ > > + cxl_write_mbox_reg32(cxlm, CXLDEV_MB_CTRL, CXLDEV_MB_CTRL_DOORBELL); > > + > > + /* #5 */ > > + rc = cxldev_wait_for_doorbell(cxlm); > > + if (rc == -ETIMEDOUT) { > > + dev_warn(&cxlm->pdev->dev, "Mailbox command timed out\n"); > > + return rc; > > + } > > + > > + /* #6 */ > > + status = cxl_read_mbox_reg64(cxlm, CXLDEV_MB_STATUS); > > + cmd = cxl_read_mbox_reg64(cxlm, CXLDEV_MB_CMD); > > + > > + mbox_cmd->return_code = (status >> CXLDEV_MB_STATUS_RET_CODE_SHIFT) & > > + CXLDEV_MB_STATUS_RET_CODE_MASK; > > + > > + /* There was a problem, let the caller deal with it */ > > + if (mbox_cmd->return_code != 0) > > + return 0; > > + > > + /* #7 */ > > + mbox_cmd->payload_size = cmd >> CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT; > > + > > + /* #8 */ > > + if (mbox_cmd->payload_size) > > + cxl_mbox_payload_drain(cxlm, mbox_cmd->payload, mbox_cmd->payload_size); > > + > > + return 0; > > +} > > + > > static int cxl_mem_mbox_get(struct cxl_mem *cxlm) > > { > > u64 md_status; >
On Tue, 17 Nov 2020 08:34:38 -0800 Ben Widawsky <ben.widawsky@intel.com> wrote: > On 20-11-17 15:31:22, Jonathan Cameron wrote: > > On Tue, 10 Nov 2020 21:43:54 -0800 > > Ben Widawsky <ben.widawsky@intel.com> wrote: > > > > > Create a function to handle sending a command, optionally with a > > > payload, to the memory device, polling on a result, and then optionally > > > copying out the payload. The algorithm for doing this come straight out > > > of the CXL 2.0 specification. > > > > > > Primary mailboxes are capable of generating an interrupt when submitting > > > a command in the background. That implementation is saved for a later > > > time. > > > > > > Secondary mailboxes aren't implemented at this time. > > > > > > WARNING: This is untested with actual timeouts occurring. > > > > > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com> > > > > Question inline for why the preempt / local timer dance is worth bothering with. > > What am I missing? > > > > Thanks, > > > > Jonathan > > > > > --- > > > drivers/cxl/cxl.h | 16 +++++++ > > > drivers/cxl/mem.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++ > > > 2 files changed, 123 insertions(+) > > > > > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h > > > index 482fc9cdc890..f49ab80f68bd 100644 > > > --- a/drivers/cxl/cxl.h > > > +++ b/drivers/cxl/cxl.h > > > @@ -21,8 +21,12 @@ > > > #define CXLDEV_MB_CTRL 0x04 > > > #define CXLDEV_MB_CTRL_DOORBELL BIT(0) > > > #define CXLDEV_MB_CMD 0x08 > > > +#define CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT 16 > > > #define CXLDEV_MB_STATUS 0x10 > > > +#define CXLDEV_MB_STATUS_RET_CODE_SHIFT 32 > > > +#define CXLDEV_MB_STATUS_RET_CODE_MASK 0xffff > > > #define CXLDEV_MB_BG_CMD_STATUS 0x18 > > > +#define CXLDEV_MB_PAYLOAD 0x20 > > > > > > /* Memory Device */ > > > #define CXLMDEV_STATUS 0 > > > @@ -114,4 +118,16 @@ static inline u64 __cxl_raw_read_reg64(struct cxl_mem *cxlm, u32 reg) > > > > > > return readq(reg_addr + reg); > > > } > > > + > > > +static inline void cxl_mbox_payload_fill(struct cxl_mem *cxlm, u8 *input, > > > + unsigned int length) > > > +{ > > > + memcpy_toio(cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, input, length); > > > +} > > > + > > > +static inline void cxl_mbox_payload_drain(struct cxl_mem *cxlm, > > > + u8 *output, unsigned int length) > > > +{ > > > + memcpy_fromio(output, cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, length); > > > +} > > > #endif /* __CXL_H__ */ > > > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c > > > index 9fd2d1daa534..08913360d500 100644 > > > --- a/drivers/cxl/mem.c > > > +++ b/drivers/cxl/mem.c > > > @@ -1,5 +1,6 @@ > > > // SPDX-License-Identifier: GPL-2.0-only > > > // Copyright(c) 2020 Intel Corporation. All rights reserved. > > > +#include <linux/sched/clock.h> > > > #include <linux/module.h> > > > #include <linux/pci.h> > > > #include <linux/io.h> > > > @@ -7,6 +8,112 @@ > > > #include "pci.h" > > > #include "cxl.h" > > > > > > +struct mbox_cmd { > > > + u16 cmd; > > > + u8 *payload; > > > + size_t payload_size; > > > + u16 return_code; > > > +}; > > > + > > > +static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm) > > > +{ > > > + u64 start, now; > > > + int cpu, ret, timeout = 2000000000; > > > + > > > + start = local_clock(); > > > + preempt_disable(); > > > + cpu = smp_processor_id(); > > > + for (;;) { > > > + now = local_clock(); > > > + preempt_enable(); > > > > What do we ever do with this mailbox that is particularly > > performance critical? I'd like to understand why we care enough > > to mess around with the preemption changes and local clock etc. > > > > It is quite obviously a premature optimization at this point (since we only > support a single command in QEMU). However, the polling can be anywhere from > instant to 2 seconds. QEMU implementation aside again, some devices may never > support interrupts on completion, and so I thought providing a poll function now > that is capable of working for most [all?] cases was wise. Definitely seems premature. I'd want to see real numbers on hardware to justify this sort of complexity. Maybe others disagree though. Jonathan > > > > + if ((cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL) & > > > + CXLDEV_MB_CTRL_DOORBELL) == 0) { > > > + ret = 0; > > > + break; > > > + } > > > + > > > + if (now - start >= timeout) { > > > + ret = -ETIMEDOUT; > > > + break; > > > + } > > > + > > > + cpu_relax(); > > > + preempt_disable(); > > > + if (unlikely(cpu != smp_processor_id())) { > > > + timeout -= (now - start); > > > + cpu = smp_processor_id(); > > > + start = local_clock(); > > > + } > > > + } > > > + > > > + return ret; > > > +} > > > + > > > +/* > > > + * Returns 0 if the doorbell transaction was successful from a protocol level. > > > + * Caller should check the return code in @mbox_cmd to make sure it succeeded. > > > + */ > > > +static int __maybe_unused cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm, struct mbox_cmd *mbox_cmd) > > > +{ > > > + u64 cmd, status; > > > + int rc; > > > + > > > + lockdep_assert_held(&cxlm->mbox_lock); > > > + > > > + /* > > > + * Here are the steps from 8.2.8.4 of the CXL 2.0 spec. > > > + * 1. Caller reads MB Control Register to verify doorbell is clear > > > + * 2. Caller writes Command Register > > > + * 3. Caller writes Command Payload Registers if input payload is non-empty > > > + * 4. Caller writes MB Control Register to set doorbell > > > + * 5. Caller either polls for doorbell to be clear or waits for interrupt if configured > > > + * 6. Caller reads MB Status Register to fetch Return code > > > + * 7. If command successful, Caller reads Command Register to get Payload Length > > > + * 8. If output payload is non-empty, host reads Command Payload Registers > > > + */ > > > + > > > + cmd = mbox_cmd->cmd; > > > + if (mbox_cmd->payload_size) { > > > + /* #3 */ > > > > Having just given the steps above, having them out of order feels like it needs > > a comment to state why. > > > > > + cmd |= mbox_cmd->payload_size > > > + << CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT; > > > + cxl_mbox_payload_fill(cxlm, mbox_cmd->payload, mbox_cmd->payload_size); > > > + } > > > + > > > + /* #2 */ > > > + cxl_write_mbox_reg64(cxlm, CXLDEV_MB_CMD, cmd); > > > + > > > + /* #4 */ > > > + cxl_write_mbox_reg32(cxlm, CXLDEV_MB_CTRL, CXLDEV_MB_CTRL_DOORBELL); > > > + > > > + /* #5 */ > > > + rc = cxldev_wait_for_doorbell(cxlm); > > > + if (rc == -ETIMEDOUT) { > > > + dev_warn(&cxlm->pdev->dev, "Mailbox command timed out\n"); > > > + return rc; > > > + } > > > + > > > + /* #6 */ > > > + status = cxl_read_mbox_reg64(cxlm, CXLDEV_MB_STATUS); > > > + cmd = cxl_read_mbox_reg64(cxlm, CXLDEV_MB_CMD); > > > + > > > + mbox_cmd->return_code = (status >> CXLDEV_MB_STATUS_RET_CODE_SHIFT) & > > > + CXLDEV_MB_STATUS_RET_CODE_MASK; > > > + > > > + /* There was a problem, let the caller deal with it */ > > > + if (mbox_cmd->return_code != 0) > > > + return 0; > > > + > > > + /* #7 */ > > > + mbox_cmd->payload_size = cmd >> CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT; > > > + > > > + /* #8 */ > > > + if (mbox_cmd->payload_size) > > > + cxl_mbox_payload_drain(cxlm, mbox_cmd->payload, mbox_cmd->payload_size); > > > + > > > + return 0; > > > +} > > > + > > > static int cxl_mem_mbox_get(struct cxl_mem *cxlm) > > > { > > > u64 md_status; > >
On Tue, Nov 17, 2020 at 10:07 AM Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote: > > On Tue, 17 Nov 2020 08:34:38 -0800 > Ben Widawsky <ben.widawsky@intel.com> wrote: > > > On 20-11-17 15:31:22, Jonathan Cameron wrote: > > > On Tue, 10 Nov 2020 21:43:54 -0800 > > > Ben Widawsky <ben.widawsky@intel.com> wrote: > > > > > > > Create a function to handle sending a command, optionally with a > > > > payload, to the memory device, polling on a result, and then optionally > > > > copying out the payload. The algorithm for doing this come straight out > > > > of the CXL 2.0 specification. > > > > > > > > Primary mailboxes are capable of generating an interrupt when submitting > > > > a command in the background. That implementation is saved for a later > > > > time. > > > > > > > > Secondary mailboxes aren't implemented at this time. > > > > > > > > WARNING: This is untested with actual timeouts occurring. > > > > > > > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com> > > > > > > Question inline for why the preempt / local timer dance is worth bothering with. > > > What am I missing? > > > > > > Thanks, > > > > > > Jonathan > > > > > > > --- > > > > drivers/cxl/cxl.h | 16 +++++++ > > > > drivers/cxl/mem.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++ > > > > 2 files changed, 123 insertions(+) > > > > > > > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h > > > > index 482fc9cdc890..f49ab80f68bd 100644 > > > > --- a/drivers/cxl/cxl.h > > > > +++ b/drivers/cxl/cxl.h > > > > @@ -21,8 +21,12 @@ > > > > #define CXLDEV_MB_CTRL 0x04 > > > > #define CXLDEV_MB_CTRL_DOORBELL BIT(0) > > > > #define CXLDEV_MB_CMD 0x08 > > > > +#define CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT 16 > > > > #define CXLDEV_MB_STATUS 0x10 > > > > +#define CXLDEV_MB_STATUS_RET_CODE_SHIFT 32 > > > > +#define CXLDEV_MB_STATUS_RET_CODE_MASK 0xffff > > > > #define CXLDEV_MB_BG_CMD_STATUS 0x18 > > > > +#define CXLDEV_MB_PAYLOAD 0x20 > > > > > > > > /* Memory Device */ > > > > #define CXLMDEV_STATUS 0 > > > > @@ -114,4 +118,16 @@ static inline u64 __cxl_raw_read_reg64(struct cxl_mem *cxlm, u32 reg) > > > > > > > > return readq(reg_addr + reg); > > > > } > > > > + > > > > +static inline void cxl_mbox_payload_fill(struct cxl_mem *cxlm, u8 *input, > > > > + unsigned int length) > > > > +{ > > > > + memcpy_toio(cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, input, length); > > > > +} > > > > + > > > > +static inline void cxl_mbox_payload_drain(struct cxl_mem *cxlm, > > > > + u8 *output, unsigned int length) > > > > +{ > > > > + memcpy_fromio(output, cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, length); > > > > +} > > > > #endif /* __CXL_H__ */ > > > > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c > > > > index 9fd2d1daa534..08913360d500 100644 > > > > --- a/drivers/cxl/mem.c > > > > +++ b/drivers/cxl/mem.c > > > > @@ -1,5 +1,6 @@ > > > > // SPDX-License-Identifier: GPL-2.0-only > > > > // Copyright(c) 2020 Intel Corporation. All rights reserved. > > > > +#include <linux/sched/clock.h> > > > > #include <linux/module.h> > > > > #include <linux/pci.h> > > > > #include <linux/io.h> > > > > @@ -7,6 +8,112 @@ > > > > #include "pci.h" > > > > #include "cxl.h" > > > > > > > > +struct mbox_cmd { > > > > + u16 cmd; > > > > + u8 *payload; > > > > + size_t payload_size; > > > > + u16 return_code; > > > > +}; > > > > + > > > > +static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm) > > > > +{ > > > > + u64 start, now; > > > > + int cpu, ret, timeout = 2000000000; > > > > + > > > > + start = local_clock(); > > > > + preempt_disable(); > > > > + cpu = smp_processor_id(); > > > > + for (;;) { > > > > + now = local_clock(); > > > > + preempt_enable(); > > > > > > What do we ever do with this mailbox that is particularly > > > performance critical? I'd like to understand why we care enough > > > to mess around with the preemption changes and local clock etc. > > > > > > > It is quite obviously a premature optimization at this point (since we only > > support a single command in QEMU). However, the polling can be anywhere from > > instant to 2 seconds. QEMU implementation aside again, some devices may never > > support interrupts on completion, and so I thought providing a poll function now > > that is capable of working for most [all?] cases was wise. > > Definitely seems premature. I'd want to see real numbers on hardware > to justify this sort of complexity. Maybe others disagree though. The polling is definitely needed, but I think it can be a simple jiffies based loop and avoid this sched_clock() complexity.
On Tue, Nov 17, 2020 at 6:33 AM Rafael J. Wysocki <rafael@kernel.org> wrote: [..] > > +static struct acpi_driver acpi_cxl_driver = { > > First of all, no new acpi_driver instances, pretty please! > > acpi_default_enumeration() should create a platform device with the > ACPI0017 ID for you. Can't you provide a driver for this one? > Ah, yes, I recall we had this discussion around the time the ACPI0012 NFIT driver was developed. IIRC the reason ACPI0012 remaining an acpi_driver was due to a need to handle ACPI notifications, is that the deciding factor? ACPI0017 does not have any notifications so it seems like platform driver is the way to go.
On Tue, Nov 17, 2020 at 10:45 PM Dan Williams <dan.j.williams@intel.com> wrote: > > On Tue, Nov 17, 2020 at 6:33 AM Rafael J. Wysocki <rafael@kernel.org> wrote: > [..] > > > +static struct acpi_driver acpi_cxl_driver = { > > > > First of all, no new acpi_driver instances, pretty please! > > > > acpi_default_enumeration() should create a platform device with the > > ACPI0017 ID for you. Can't you provide a driver for this one? > > > > Ah, yes, I recall we had this discussion around the time the ACPI0012 > NFIT driver was developed. IIRC the reason ACPI0012 remaining an > acpi_driver was due to a need to handle ACPI notifications, is that > the deciding factor? Sort of. In fact, a platform device driver can still handle ACPI notifications just fine, it just needs to install a notify handler for that. The cases when an acpi_driver is really needed are basically when creating the platform device during the enumeration is not desirable, like in the PCI or PNP cases (because they both create device objects of a different type to represent the "physical" device). It doesn't look like it is really needed for ACPI0012, but since it is there already, well ... > ACPI0017 does not have any notifications so it seems like platform driver is the way to go. Indeed.
On Tue, Nov 17, 2020 at 7:57 AM Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote: > > On Tue, 10 Nov 2020 21:43:55 -0800 > Ben Widawsky <ben.widawsky@intel.com> wrote: > > > From: Dan Williams <dan.j.williams@intel.com> > > > > Create the /sys/bus/cxl hierarchy to enumerate memory devices > > (per-endpoint control devices), memory address space devices (platform > > address ranges with interleaving, performance, and persistence > > attributes), and memory regions (active provisioned memory from an > > address space device that is in use as System RAM or delegated to > > libnvdimm as Persistent Memory regions). > > > > For now, only the per-endpoint control devices are registered on the > > 'cxl' bus. > > Reviewing ABI without documentation is challenging even when it's simple > so please add that for v2. > > This patch feels somewhat unpolished, but I guess it is mainly here to > give an illustration of how stuff might fit together rather than > any expectation of detailed review. Yeah, this is definitely an early look in the spirit of "Release early / release often". > > So in that spirit I've just pointed out stuff that jumped out at me > during a quick read through. > > Thanks, > > Jonathan > > > > > > Signed-off-by: Dan Williams <dan.j.williams@intel.com> > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com> > > --- > > drivers/cxl/Makefile | 2 + > > drivers/cxl/bus.c | 35 ++++++ > > drivers/cxl/bus.h | 8 ++ > > drivers/cxl/cxl.h | 33 +++++ > > drivers/cxl/mem.c | 287 ++++++++++++++++++++++++++++++++++++++++++- > > 5 files changed, 359 insertions(+), 6 deletions(-) > > create mode 100644 drivers/cxl/bus.c > > create mode 100644 drivers/cxl/bus.h > > > > diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile > > index 97fdffb00f2d..1cc032092852 100644 > > --- a/drivers/cxl/Makefile > > +++ b/drivers/cxl/Makefile > > @@ -1,7 +1,9 @@ > > # SPDX-License-Identifier: GPL-2.0 > > obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o > > obj-$(CONFIG_CXL_MEM) += cxl_mem.o > > +obj-$(CONFIG_CXL_BUS_PROVIDER) += cxl_bus.o > > > > ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL > > cxl_acpi-y := acpi.o > > cxl_mem-y := mem.o > > +cxl_bus-y := bus.o > > diff --git a/drivers/cxl/bus.c b/drivers/cxl/bus.c > > new file mode 100644 > > index 000000000000..8594366955f7 > > --- /dev/null > > +++ b/drivers/cxl/bus.c > > @@ -0,0 +1,35 @@ > > +// SPDX-License-Identifier: GPL-2.0-only > > +// Copyright(c) 2020 Intel Corporation. All rights reserved. > > +#include <linux/device.h> > > +#include <linux/module.h> > > + > > +static struct bus_type cxl_bus_type = { > > + .name = "cxl", > > +}; > > + > > +int cxl_register(struct device *dev) > > +{ > > + int rc; > > + > > + dev->bus = &cxl_bus_type; > > + rc = device_add(dev); > > + if (rc) > > + put_device(dev); > > + return rc; > > +} > > +EXPORT_SYMBOL_GPL(cxl_register); > > + > > +static __init int cxl_bus_init(void) > > +{ > > + return bus_register(&cxl_bus_type); > > +} > > + > > +static void cxl_bus_exit(void) > > +{ > > + bus_unregister(&cxl_bus_type); > > +} > > + > > +module_init(cxl_bus_init); > > +module_exit(cxl_bus_exit); > > +MODULE_LICENSE("GPL v2"); > > +MODULE_AUTHOR("Intel Corporation"); > > diff --git a/drivers/cxl/bus.h b/drivers/cxl/bus.h > > new file mode 100644 > > index 000000000000..fe2bea2bbc3c > > --- /dev/null > > +++ b/drivers/cxl/bus.h > > @@ -0,0 +1,8 @@ > > +// SPDX-License-Identifier: GPL-2.0-only > > +// Copyright(c) 2020 Intel Corporation. All rights reserved. > > +#ifndef __CXL_BUS_H__ > > +#define __CXL_BUS_H__ > > + > > +int cxl_register(struct device *dev); > > + > > +#endif /* __CXL_BUS_H__ */ > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h > > index f49ab80f68bd..cef5fd9ea68b 100644 > > --- a/drivers/cxl/cxl.h > > +++ b/drivers/cxl/cxl.h > > @@ -3,6 +3,7 @@ > > > > #ifndef __CXL_H__ > > #define __CXL_H__ > > +#include <linux/range.h> > > > > /* Device */ > > #define CXLDEV_CAP_ARRAY_REG 0x0 > > @@ -52,12 +53,24 @@ > > #define CXLMDEV_RESET_NEEDED_HOT 3 > > #define CXLMDEV_RESET_NEEDED_CXL 4 > > > > +struct cxl_memdev; > > struct cxl_mem { > > struct pci_dev *pdev; > > void __iomem *regs; > > + struct cxl_memdev *cxlmd; > > > > spinlock_t mbox_lock; /* Protects device mailbox and firmware */ > > > > + struct { > > + struct range range; > > + } pmem; > > + > > + struct { > > + struct range range; > > + } ram; > > + > > + char firmware_version[0x10]; > > + > > /* Cap 0000h */ > > struct { > > void __iomem *regs; > > @@ -130,4 +143,24 @@ static inline void cxl_mbox_payload_drain(struct cxl_mem *cxlm, > > { > > memcpy_fromio(output, cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, length); > > } > > + > > +#define CXL_MBOX_IDENTIFY 0x4000 > > + > > +struct cxl_mbox_identify { > > + char fw_revision[0x10]; > > + __le64 total_capacity; > > + __le64 volatile_capacity; > > + __le64 persistent_capacity; > > + __le64 partition_align; > > + __le16 info_event_log_size; > > + __le16 warning_event_log_size; > > + __le16 failure_event_log_size; > > + __le16 fatal_event_log_size; > > + __le32 lsa_size; > > + u8 poison_list_max_mer[3]; > > + __le16 inject_poison_limit; > > + u8 poison_caps; > > + u8 qos_telemetry_caps; > > +} __packed; > > + > > #endif /* __CXL_H__ */ > > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c > > index 08913360d500..54743d196feb 100644 > > --- a/drivers/cxl/mem.c > > +++ b/drivers/cxl/mem.c > > @@ -2,11 +2,15 @@ > > // Copyright(c) 2020 Intel Corporation. All rights reserved. > > #include <linux/sched/clock.h> > > #include <linux/module.h> > > +#include <linux/mutex.h> > > +#include <linux/cdev.h> > > +#include <linux/idr.h> > > #include <linux/pci.h> > > #include <linux/io.h> > > #include "acpi.h" > > #include "pci.h" > > #include "cxl.h" > > +#include "bus.h" > > > > struct mbox_cmd { > > u16 cmd; > > @@ -15,6 +19,53 @@ struct mbox_cmd { > > u16 return_code; > > }; > > > > +/* > > + * An entire PCI topology full of devices should be enough for any > > + * config > > + */ > > +#define CXL_MEM_MAX_DEVS 65536 > > + > > +struct cxl_memdev { > > + struct device dev; > > + struct cxl_mem *cxlm; > > + int id; > > +}; > > + > > +static int cxl_mem_major; > > +static struct cdev cxl_mem_cdev; > > +static DEFINE_IDR(cxl_mem_idr); > > +static DEFINE_MUTEX(cxl_memdev_lock); > > Define scope of this lock with a comment. Will do. > > > + > > +static int cxl_mem_open(struct inode *inode, struct file *file) > > +{ > > + long minor = iminor(inode); > > + struct cxl_memdev *cxlmd; > > + > > + rcu_read_lock(); > > + cxlmd = idr_find(&cxl_mem_idr, minor); > > + rcu_read_unlock(); > > + > > + if (!cxlmd) > > + return -ENXIO; > > + > > + file->private_data = cxlmd; > > + > > + return 0; > > +} > > + > > +static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg) > > +{ > > + return -ENOTTY; > > +} > > + > > +static const struct file_operations cxl_mem_fops = { > > + .owner = THIS_MODULE, > > + .open = cxl_mem_open, > > + .unlocked_ioctl = cxl_mem_ioctl, > > + .compat_ioctl = compat_ptr_ioctl, > > + .llseek = noop_llseek, > > +}; > > + > > static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm) > > { > > u64 start, now; > > @@ -53,7 +104,7 @@ static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm) > > * Returns 0 if the doorbell transaction was successful from a protocol level. > > * Caller should check the return code in @mbox_cmd to make sure it succeeded. > > */ > > -static int __maybe_unused cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm, struct mbox_cmd *mbox_cmd) > > +static int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm, struct mbox_cmd *mbox_cmd) > > { > > u64 cmd, status; > > int rc; > > @@ -277,10 +328,185 @@ static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec) > > return 0; > > } > > > > +static struct cxl_memdev *to_cxl_memdev(struct device *dev) > > +{ > > + return container_of(dev, struct cxl_memdev, dev); > > +} > > + > > +static void cxl_memdev_release(struct device *dev) > > +{ > > + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); > > + > > + mutex_lock(&cxl_memdev_lock); > > + idr_remove(&cxl_mem_idr, cxlmd->id); > > + mutex_unlock(&cxl_memdev_lock); > > + > > + kfree(cxlmd); > > +} > > + > > +static char *cxl_memdev_devnode(struct device *dev, umode_t *mode, kuid_t *uid, kgid_t *gid) > > +{ > > + return kasprintf(GFP_KERNEL, "cxl/%s", dev_name(dev)); > > +} > > + > > +static ssize_t firmware_version_show(struct device *dev, struct device_attribute *attr, char *buf) > > +{ > > + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); > > + struct cxl_mem *cxlm = cxlmd->cxlm; > > + > > + return sprintf(buf, "%.16s\n", cxlm->firmware_version); > > +} > > +static DEVICE_ATTR_RO(firmware_version); > > + > > +static ssize_t ram_size_show(struct device *dev, struct device_attribute *attr, char *buf) > > +{ > > + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); > > + struct cxl_mem *cxlm = cxlmd->cxlm; > > + > > + return sprintf(buf, "%#llx\n", (unsigned long long) range_len(&cxlm->ram.range)); > > +} > > +static struct device_attribute dev_attr_ram_size = __ATTR(size, 0444, ram_size_show, NULL); > > + > > +static ssize_t pmem_size_show(struct device *dev, struct device_attribute *attr, char *buf) > > +{ > > + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); > > + struct cxl_mem *cxlm = cxlmd->cxlm; > > + > > + return sprintf(buf, "%#llx\n", (unsigned long long) range_len(&cxlm->pmem.range)); > > +} > > +static struct device_attribute dev_attr_pmem_size = __ATTR(size, 0444, pmem_size_show, NULL); > > + > > +static struct attribute *cxl_memdev_attributes[] = { > > + &dev_attr_firmware_version.attr, > > + NULL, > > +}; > > + > > +static struct attribute *cxl_memdev_pmem_attributes[] = { > > + &dev_attr_pmem_size.attr, > > It's simple, but should still have docs in Documentation/ABI/testing/sysfs... Agree. > > > + NULL, > > +}; > > + > > +static struct attribute *cxl_memdev_ram_attributes[] = { > > + &dev_attr_ram_size.attr, > > + NULL, > > +}; > > + > > +static struct attribute_group cxl_memdev_attribute_group = { > > + .attrs = cxl_memdev_attributes, > > +}; > > + > > +static struct attribute_group cxl_memdev_ram_attribute_group = { > > + .name = "ram", > > + .attrs = cxl_memdev_ram_attributes, > > +}; > > + > > +static struct attribute_group cxl_memdev_pmem_attribute_group = { > > + .name = "pmem", > > + .attrs = cxl_memdev_pmem_attributes, > > +}; > > + > > +static const struct attribute_group *cxl_memdev_attribute_groups[] = { > > + &cxl_memdev_attribute_group, > > + &cxl_memdev_ram_attribute_group, > > + &cxl_memdev_pmem_attribute_group, > > + NULL, > > +}; > > + > > +static const struct device_type cxl_memdev_type = { > > + .name = "cxl_memdev", > > + .release = cxl_memdev_release, > > + .devnode = cxl_memdev_devnode, > > + .groups = cxl_memdev_attribute_groups, > > +}; > > + > > +static struct cxl_memdev *cxl_mem_add_memdev(struct cxl_mem *cxlm) > > +{ > > + struct pci_dev *pdev = cxlm->pdev; > > + struct cxl_memdev *cxlmd; > > + struct device *dev; > > + int id, rc; > > + > > + cxlmd = kzalloc(sizeof(*cxlmd), GFP_KERNEL); > > Maybe I missed it, but I'm not seeing this freed anywhere. See: cxl_memdev_release() > > > + if (!cxlmd) > > + return ERR_PTR(-ENOMEM); > > + > > + cxlmd->cxlm = cxlm; > > + cxlm->cxlmd = cxlmd; > > + > > + mutex_lock(&cxl_memdev_lock); > > + id = idr_alloc(&cxl_mem_idr, cxlmd, 0, CXL_MEM_MAX_DEVS, GFP_KERNEL); > > + mutex_unlock(&cxl_memdev_lock); > > + if (id < 0) { > > + rc = id; > > + goto err_idr; > > + } > > + > > + cxlmd->id = id; > > + > > + dev = &cxlmd->dev; > > + > > + device_initialize(dev); > > + dev->parent = &pdev->dev; > > + dev->devt = MKDEV(cxl_mem_major, id); > > + dev->type = &cxl_memdev_type; > > + dev_set_name(dev, "mem%d", id); > > blank line here ok > > > + rc = cxl_register(dev); > > + if (rc) > > + return ERR_PTR(rc); > > + > > + return cxlmd; > > + > > +err_idr: > > + kfree(cxlmd); > > + > > + return ERR_PTR(rc); > > +} > > + > > ... > > > static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) > > { > > struct cxl_mem *cxlm = ERR_PTR(-ENXIO); > > struct device *dev = &pdev->dev; > > + struct cxl_memdev *cxlmd; > > int rc, regloc, i; > > > > rc = cxl_bus_prepared(pdev); > > @@ -319,20 +545,31 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) > > if (rc) > > return rc; > > > > - /* Check that hardware "looks" okay. */ > > - rc = cxl_mem_mbox_get(cxlm); > > + rc = cxl_mem_identify(cxlm); > > if (rc) > > return rc; > > - > > - cxl_mem_mbox_put(cxlm); > > It was kind of nice to see the flow earlier, but I'm also thinking it made > a slightly harder to read patch. Hmm. Maybe just drop the version earlier > in favour of a todo comment that you then do here? Not sure I follow, but I think you're saying don't bother with an initial patch introducing just doing the raw cxl_mem_mbox_get() in this path, jump straight to cxl_mem_identify()? > > > dev_dbg(&pdev->dev, "CXL Memory Device Interface Up\n"); > > + > > Nice to tidy that up by moving to earlier patch. Sure. > > > pci_set_drvdata(pdev, cxlm); > > > > + cxlmd = cxl_mem_add_memdev(cxlm); > > + if (IS_ERR(cxlmd)) > > + return PTR_ERR(cxlmd); > > Given we don't actually use cxlmd perhaps a simple return value > of 0 or error would be better from cxl_mem_add_memdev() > > (I guess you may have follow up patches that do something with it > here, though it feels wrong to ever do so given it is now registered > and hence exposed to the system). It's not added if IS_ERR() is true, but it would be simpler to just have cxl_mem_add_memdev() return an int since ->probe() doesn't use it. > > > + > > return 0; > > } > > > > static void cxl_mem_remove(struct pci_dev *pdev) > > { > > + struct cxl_mem *cxlm = pci_get_drvdata(pdev); > > + struct cxl_memdev *cxlmd = cxlm->cxlmd; > > + > > + device_lock(&cxlmd->dev); > > + cxlm->cxlmd = NULL; > > + cxlmd->cxlm = NULL; > > + device_unlock(&cxlmd->dev); > > + > > + device_unregister(&cxlmd->dev); > > Why device_unregister last? Normally removing exposure to the > system is the first thing you do in a remove() call. > Particularly as you'll get NULL ptr dereferences if anyone > manages a sysfs read between the pointers being set to NULL above > and the device_unregister() taking away the sysfs files. Yes, the unregister should be before the invalidation in this case. In fact I'll likely drop the invalidation or do some other synchronization requirement for any races between the ioctl path and the remove path. > > > > > } > > > > static const struct pci_device_id cxl_mem_pci_tbl[] = { > > @@ -350,7 +587,45 @@ static struct pci_driver cxl_mem_driver = { > > .remove = cxl_mem_remove, > > }; > > > > +static __init int cxl_mem_init(void) > > +{ > > + int rc; > > + dev_t devt; > > + > > + rc = alloc_chrdev_region(&devt, 0, CXL_MEM_MAX_DEVS, "cxl"); > > + if (rc) > > + return rc; > > + > > + cxl_mem_major = MAJOR(devt); > > + > > + cdev_init(&cxl_mem_cdev, &cxl_mem_fops); > > + rc = cdev_add(&cxl_mem_cdev, MKDEV(cxl_mem_major, 0), CXL_MEM_MAX_DEVS); > > + if (rc) > > + goto err_cdev; > > + > > + rc = pci_register_driver(&cxl_mem_driver); > > + if (rc) > > + goto err_driver; > > + > > + return 0; > > + > > +err_driver: > > + cdev_del(&cxl_mem_cdev); > > +err_cdev: > > + unregister_chrdev_region(MKDEV(cxl_mem_major, 0), CXL_MEM_MAX_DEVS); > > + > > + return rc; > > +} > > + > > +static __exit void cxl_mem_exit(void) > > +{ > > + pci_unregister_driver(&cxl_mem_driver); > > + unregister_chrdev_region(MKDEV(cxl_mem_major, 0), CXL_MEM_MAX_DEVS); > > + cdev_del(&cxl_mem_cdev); > > Ordering? cdev_dev should be before unregister_chrdev_region to match > error handling in init() Yes. Thanks for taking a look!
On Thu, 19 Nov 2020 18:16:19 -0800 Dan Williams <dan.j.williams@intel.com> wrote: > On Tue, Nov 17, 2020 at 7:57 AM Jonathan Cameron > <Jonathan.Cameron@huawei.com> wrote: > > > > On Tue, 10 Nov 2020 21:43:55 -0800 > > Ben Widawsky <ben.widawsky@intel.com> wrote: > > > > > From: Dan Williams <dan.j.williams@intel.com> > > > > > > Create the /sys/bus/cxl hierarchy to enumerate memory devices > > > (per-endpoint control devices), memory address space devices (platform > > > address ranges with interleaving, performance, and persistence > > > attributes), and memory regions (active provisioned memory from an > > > address space device that is in use as System RAM or delegated to > > > libnvdimm as Persistent Memory regions). > > > > > > For now, only the per-endpoint control devices are registered on the > > > 'cxl' bus. > > > > Reviewing ABI without documentation is challenging even when it's simple > > so please add that for v2. > > > > This patch feels somewhat unpolished, but I guess it is mainly here to > > give an illustration of how stuff might fit together rather than > > any expectation of detailed review. > > Yeah, this is definitely an early look in the spirit of "Release early > / release often". > Definitely a good idea. ... > > > > > static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) > > > { > > > struct cxl_mem *cxlm = ERR_PTR(-ENXIO); > > > struct device *dev = &pdev->dev; > > > + struct cxl_memdev *cxlmd; > > > int rc, regloc, i; > > > > > > rc = cxl_bus_prepared(pdev); > > > @@ -319,20 +545,31 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) > > > if (rc) > > > return rc; > > > > > > - /* Check that hardware "looks" okay. */ > > > - rc = cxl_mem_mbox_get(cxlm); > > > + rc = cxl_mem_identify(cxlm); > > > if (rc) > > > return rc; > > > - > > > - cxl_mem_mbox_put(cxlm); > > > > It was kind of nice to see the flow earlier, but I'm also thinking it made > > a slightly harder to read patch. Hmm. Maybe just drop the version earlier > > in favour of a todo comment that you then do here? > > Not sure I follow, but I think you're saying don't bother with an > initial patch introducing just doing the raw cxl_mem_mbox_get() in > this path, jump straight to cxl_mem_identify()? Exactly. > > > > > > dev_dbg(&pdev->dev, "CXL Memory Device Interface Up\n"); > > > + > > > > Nice to tidy that up by moving to earlier patch. > > Sure. > > > > > > pci_set_drvdata(pdev, cxlm); > > > > > > + cxlmd = cxl_mem_add_memdev(cxlm); > > > + if (IS_ERR(cxlmd)) > > > + return PTR_ERR(cxlmd); > > > > Given we don't actually use cxlmd perhaps a simple return value > > of 0 or error would be better from cxl_mem_add_memdev() > > > > (I guess you may have follow up patches that do something with it > > here, though it feels wrong to ever do so given it is now registered > > and hence exposed to the system). > > It's not added if IS_ERR() is true, but it would be simpler to just > have cxl_mem_add_memdev() return an int since ->probe() doesn't use > it. Agreed. > > > > > > + > > > return 0; > > > } > > > ...
On 20-11-17 15:15:19, Jonathan Cameron wrote: > On Tue, 10 Nov 2020 21:43:52 -0800 > Ben Widawsky <ben.widawsky@intel.com> wrote: > > > CXL devices contain an array of capabilities that describe the > > interactions software can interact with the device, or firmware running > > on the device. A CXL compliant device must implement the device status > > and the mailbox capability. A CXL compliant memory device must implement > > the memory device capability. > > > > Each of the capabilities can [will] provide an offset within the MMIO > > region for interacting with the CXL device. > > > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com> > > A few really minor things in this one. > > Jonathan > > > --- > > drivers/cxl/cxl.h | 89 +++++++++++++++++++++++++++++++++++++++++++++++ > > drivers/cxl/mem.c | 58 +++++++++++++++++++++++++++--- > > 2 files changed, 143 insertions(+), 4 deletions(-) > > create mode 100644 drivers/cxl/cxl.h > > > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h > > new file mode 100644 > > index 000000000000..02858ae63d6d > > --- /dev/null > > +++ b/drivers/cxl/cxl.h > > @@ -0,0 +1,89 @@ > > +// SPDX-License-Identifier: GPL-2.0-only > > +// Copyright(c) 2020 Intel Corporation. All rights reserved. > > + > > +#ifndef __CXL_H__ > > +#define __CXL_H__ > > + > > +/* Device */ > > +#define CXLDEV_CAP_ARRAY_REG 0x0 > > +#define CXLDEV_CAP_ARRAY_CAP_ID 0 > > +#define CXLDEV_CAP_ARRAY_ID(x) ((x) & 0xffff) > > +#define CXLDEV_CAP_ARRAY_COUNT(x) (((x) >> 32) & 0xffff) > > + > > +#define CXL_CAPABILITIES_CAP_ID_DEVICE_STATUS 1 > > I'm not sure what you can do about consistent naming, but > perhaps this really does need to be > CXLDEV_CAP_CAP_ID_x however silly that looks. > > It's a funny exercise I've only seen done once in a spec, but > I wish everyone would try working out what their fully expanded > field names will end up as and make sure the long form naming shortens > to something sensible. It definitely helps with clarity! > > > +#define CXL_CAPABILITIES_CAP_ID_PRIMARY_MAILBOX 2 > > +#define CXL_CAPABILITIES_CAP_ID_SECONDARY_MAILBOX 3 > > +#define CXL_CAPABILITIES_CAP_ID_MEMDEV 0x4000 > > + > > +/* Mailbox */ > > +#define CXLDEV_MB_CAPS 0x00 > > Elsewhere you've used _OFFSET postfix. That's useful so I'd do it here > as well. Cross references to the spec section always appreciated as well! > > > +#define CXLDEV_MB_CAP_PAYLOAD_SIZE(cap) ((cap) & 0x1F) > > +#define CXLDEV_MB_CTRL 0x04 > > +#define CXLDEV_MB_CMD 0x08 > > +#define CXLDEV_MB_STATUS 0x10 > > +#define CXLDEV_MB_BG_CMD_STATUS 0x18 > > + > > +struct cxl_mem { > > + struct pci_dev *pdev; > > + void __iomem *regs; > > + > > + /* Cap 0000h */ > > What are the numbers here? These capabilities have too > many indexes associated with them in different ways for it > to be obvious, so perhaps more detail in the comments? This comment was a bug. The cap is 0001h actually. I've added the hash define for its cap id as part of the comment. Everything else is accepted. > > > + struct { > > + void __iomem *regs; > > + } status; > > + > > + /* Cap 0002h */ > > + struct { > > + void __iomem *regs; > > + size_t payload_size; > > + } mbox; > > + > > + /* Cap 0040h */ > > + struct { > > + void __iomem *regs; > > + } mem; > > +};
On 11/11/20 12:43 AM, Ben Widawsky wrote: > + case CXL_CAPABILITIES_CAP_ID_SECONDARY_MAILBOX: > + dev_dbg(&cxlm->pdev->dev, > + "found UNSUPPORTED Secondary Mailbox capability\n"); Per spec, the secondary mailbox is intended for use by platform firmware, so Linux should never be using it anyway. Maybe that message is slightly misleading? Jon. P.S. Related - I've severe doubts about the mailbox approach being proposed by CXL and have begun to push back through the spec org. -- Computer Architect
On 20-11-26 01:05:56, Jon Masters wrote: > On 11/11/20 12:43 AM, Ben Widawsky wrote: > > > + case CXL_CAPABILITIES_CAP_ID_SECONDARY_MAILBOX: > > + dev_dbg(&cxlm->pdev->dev, > > + "found UNSUPPORTED Secondary Mailbox capability\n"); > > Per spec, the secondary mailbox is intended for use by platform firmware, so > Linux should never be using it anyway. Maybe that message is slightly > misleading? Yeah, I think the message could be reworded, but it is dev_dbg, so I wasn't too worried about the wording in the first place. I think it is a mistake in this case for the spec to describe the intended purpose. If the expectation is for platform firmware to use it, but there is no negotiation mechanism in place, it's essentially useless. > > Jon. > > P.S. Related - I've severe doubts about the mailbox approach being proposed > by CXL and have begun to push back through the spec org. Any reason not to articulate that here? Now that the spec is public, I don't see any reason not to disclose that publicly.
On Tue, Nov 17, 2020 at 6:50 AM Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote: > > On Tue, 10 Nov 2020 21:43:50 -0800 > Ben Widawsky <ben.widawsky@intel.com> wrote: > > > From: Dan Williams <dan.j.williams@intel.com> > > > > The CXL.mem protocol allows a device to act as a provider of "System > > RAM" and/or "Persistent Memory" that is fully coherent as if the memory > > was attached to the typical CPU memory controller. > > > > The memory range exported by the device may optionally be described by > > the platform firmware memory map, or by infrastructure like LIBNVDIMM to > > provision persistent memory capacity from one, or more, CXL.mem devices. > > > > A pre-requisite for Linux-managed memory-capacity provisioning is this > > cxl_mem driver that can speak the "type-3 mailbox" protocol. > > > > For now just land the driver boiler-plate and fill it in with > > functionality in subsequent commits. > > > > Signed-off-by: Dan Williams <dan.j.williams@intel.com> > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com> > > I've tried to avoid repeats, so mostly this is me moaning about naming! > > Jonathan > > > --- > > drivers/cxl/Kconfig | 20 +++++++++++ > > drivers/cxl/Makefile | 2 ++ > > drivers/cxl/mem.c | 82 ++++++++++++++++++++++++++++++++++++++++++++ > > drivers/cxl/pci.h | 15 ++++++++ > > 4 files changed, 119 insertions(+) > > create mode 100644 drivers/cxl/mem.c > > create mode 100644 drivers/cxl/pci.h > > > > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig > > index dd724bd364df..15548f5c77ff 100644 > > --- a/drivers/cxl/Kconfig > > +++ b/drivers/cxl/Kconfig > > @@ -27,4 +27,24 @@ config CXL_ACPI > > resources described by the CEDT (CXL Early Discovery Table) > > > > Say 'y' to enable CXL (Compute Express Link) drivers. > > + > > +config CXL_MEM > > + tristate "CXL.mem Device Support" > > + depends on PCI && CXL_BUS_PROVIDER != n > > + default m if CXL_BUS_PROVIDER > > + help > > + The CXL.mem protocol allows a device to act as a provider of > > + "System RAM" and/or "Persistent Memory" that is fully coherent > > + as if the memory was attached to the typical CPU memory > > + controller. > > + > > + Say 'y/m' to enable a driver named "cxl_mem.ko" that will attach > > + to CXL.mem devices for configuration, provisioning, and health > > + monitoring, the so called "type-3 mailbox". Note, this driver > > + is required for dynamic provisioning of CXL.mem attached > > + memory, a pre-requisite for persistent memory support, but > > + devices that provide volatile memory may be fully described by > > + existing platform firmware memory enumeration. > > + > > + If unsure say 'n'. > > endif > > diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile > > index d38cd34a2582..97fdffb00f2d 100644 > > --- a/drivers/cxl/Makefile > > +++ b/drivers/cxl/Makefile > > @@ -1,5 +1,7 @@ > > # SPDX-License-Identifier: GPL-2.0 > > obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o > > +obj-$(CONFIG_CXL_MEM) += cxl_mem.o > > > > ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL > > cxl_acpi-y := acpi.o > > +cxl_mem-y := mem.o > > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c > > new file mode 100644 > > index 000000000000..aa7d881fa47b > > --- /dev/null > > +++ b/drivers/cxl/mem.c > > @@ -0,0 +1,82 @@ > > +// SPDX-License-Identifier: GPL-2.0-only > > +// Copyright(c) 2020 Intel Corporation. All rights reserved. > > +#include <linux/module.h> > > +#include <linux/pci.h> > > +#include <linux/io.h> > > +#include "acpi.h" > > +#include "pci.h" > > + > > +struct cxl_mem { > > + void __iomem *regs; > > +}; > > + > > +static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec) > > +{ > > + int pos; > > + > > + pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DVSEC); > > + if (!pos) > > + return 0; > > + > > + while (pos) { > > + u16 vendor, id; > > + > > + pci_read_config_word(pdev, pos + PCI_DVSEC_VENDOR_OFFSET, &vendor); > > + pci_read_config_word(pdev, pos + PCI_DVSEC_ID_OFFSET, &id); > > + if (vendor == PCI_DVSEC_VENDOR_CXL && dvsec == id) > > + return pos; > > + > > + pos = pci_find_next_ext_capability(pdev, pos, PCI_EXT_CAP_ID_DVSEC); > > This is good generic code and wouldn't cause much backport effort (even if needed > to bring in a local copy), so perhaps make it a generic function and move to > core PCI code? > > Mind you I guess that can happen the 'second' time someone wants to find a DVSEC. > > > + } > > + > > + return 0; > > +} > > + > > +static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) > > +{ > > + struct device *dev = &pdev->dev; > > + struct cxl_mem *cxlm; > > + int rc, regloc; > > + > > + rc = cxl_bus_prepared(pdev); > > + if (rc != 0) { > > + dev_err(dev, "failed to acquire interface\n"); > > + return rc; > > + } > > + > > + regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC); > > + if (!regloc) { > > + dev_err(dev, "register location dvsec not found\n"); > > + return -ENXIO; > > + } > > + > > + cxlm = devm_kzalloc(dev, sizeof(*cxlm), GFP_KERNEL); > > + if (!cxlm) > > + return -ENOMEM; > > + > > + return 0; > > +} > > + > > +static void cxl_mem_remove(struct pci_dev *pdev) > > +{ > > +} > > I'd bring this in only when needed in later patch. > > > + > > +static const struct pci_device_id cxl_mem_pci_tbl[] = { > > + /* PCI class code for CXL.mem Type-3 Devices */ > > + { PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID, > > + PCI_CLASS_MEMORY_CXL, 0xffffff, 0 }, > > + { /* terminate list */ }, > > +}; > > +MODULE_DEVICE_TABLE(pci, cxl_mem_pci_tbl); > > + > > +static struct pci_driver cxl_mem_driver = { > > + .name = KBUILD_MODNAME, > > + .id_table = cxl_mem_pci_tbl, > > + .probe = cxl_mem_probe, > > + .remove = cxl_mem_remove, > > +}; > > + > > +MODULE_LICENSE("GPL v2"); > > +MODULE_AUTHOR("Intel Corporation"); > > +module_pci_driver(cxl_mem_driver); > > +MODULE_IMPORT_NS(CXL); > > diff --git a/drivers/cxl/pci.h b/drivers/cxl/pci.h > > new file mode 100644 > > index 000000000000..beb03921e6da > > --- /dev/null > > +++ b/drivers/cxl/pci.h > > @@ -0,0 +1,15 @@ > > +// SPDX-License-Identifier: GPL-2.0-only > > +// Copyright(c) 2020 Intel Corporation. All rights reserved. > > +#ifndef __CXL_PCI_H__ > > +#define __CXL_PCI_H__ > > + > > +#define PCI_CLASS_MEMORY_CXL 0x050210 > > + > > +#define PCI_EXT_CAP_ID_DVSEC 0x23 > > +#define PCI_DVSEC_VENDOR_CXL 0x1E98 > > Hmm. The magic question of what to call a vendor ID that isn't a vendor > ID but just a magic number that talks like a duck and quacks like a duck > (for anyone wondering what I'm talking about, there is a nice bit of legal > boilerplate on this in the CXL spec) > > This name is definitely not accurate however. > > PCI_UNIQUE_VALUE_CXL maybe? It is used for other things than DVSEC (VDMs etc), > though possibly this is the only software visible use. Finally working my way back through this review to make the changes. If 0x1E98 becomes visible to software somewhere else then this can become something like the following: #define PCI_DVSEC_VENDOR_CXL PCI_UNIQUE_VALUE_CXL ...or whatever the generic name is, but this field per the specification is the DVSEC-vendor-id and calling it PCI_UNIQUE_VALUE_CXL does not have any basis in the spec. I will rename it though to: PCI_DVSEC_VENDOR_ID_CXL ...since include/linux/pci_ids.h includes the _ID_ part. > > > > +#define PCI_DVSEC_VENDOR_OFFSET 0x4 > > +#define PCI_DVSEC_ID_OFFSET 0x8 > > Put a line break here perhaps and maybe a spec reference to where to find > the various DVSEC IDs. Ok. > > > +#define PCI_DVSEC_ID_CXL 0x0 > > That's definitely a confusing name as well. Yeah, should be PCI_DVSEC_DEVICE_ID_CXL > > PCI_DEVSEC_ID_CXL_DEVICE maybe? > > > > +#define PCI_DVSEC_ID_CXL_REGLOC 0x8 > > + > > +#endif /* __CXL_PCI_H__ */ >
On Thu, Dec 3, 2020 at 11:22 PM Dan Williams <dan.j.williams@intel.com> wrote: > > On Tue, Nov 17, 2020 at 6:50 AM Jonathan Cameron > <Jonathan.Cameron@huawei.com> wrote: > > > > On Tue, 10 Nov 2020 21:43:50 -0800 > > Ben Widawsky <ben.widawsky@intel.com> wrote: > > > > > From: Dan Williams <dan.j.williams@intel.com> > > > > > > The CXL.mem protocol allows a device to act as a provider of "System > > > RAM" and/or "Persistent Memory" that is fully coherent as if the memory > > > was attached to the typical CPU memory controller. > > > > > > The memory range exported by the device may optionally be described by > > > the platform firmware memory map, or by infrastructure like LIBNVDIMM to > > > provision persistent memory capacity from one, or more, CXL.mem devices. > > > > > > A pre-requisite for Linux-managed memory-capacity provisioning is this > > > cxl_mem driver that can speak the "type-3 mailbox" protocol. > > > > > > For now just land the driver boiler-plate and fill it in with > > > functionality in subsequent commits. > > > > > > Signed-off-by: Dan Williams <dan.j.williams@intel.com> > > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com> > > > > I've tried to avoid repeats, so mostly this is me moaning about naming! > > > > Jonathan > > > > > --- > > > drivers/cxl/Kconfig | 20 +++++++++++ > > > drivers/cxl/Makefile | 2 ++ > > > drivers/cxl/mem.c | 82 ++++++++++++++++++++++++++++++++++++++++++++ > > > drivers/cxl/pci.h | 15 ++++++++ > > > 4 files changed, 119 insertions(+) > > > create mode 100644 drivers/cxl/mem.c > > > create mode 100644 drivers/cxl/pci.h > > > > > > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig > > > index dd724bd364df..15548f5c77ff 100644 > > > --- a/drivers/cxl/Kconfig > > > +++ b/drivers/cxl/Kconfig > > > @@ -27,4 +27,24 @@ config CXL_ACPI > > > resources described by the CEDT (CXL Early Discovery Table) > > > > > > Say 'y' to enable CXL (Compute Express Link) drivers. > > > + > > > +config CXL_MEM > > > + tristate "CXL.mem Device Support" > > > + depends on PCI && CXL_BUS_PROVIDER != n > > > + default m if CXL_BUS_PROVIDER > > > + help > > > + The CXL.mem protocol allows a device to act as a provider of > > > + "System RAM" and/or "Persistent Memory" that is fully coherent > > > + as if the memory was attached to the typical CPU memory > > > + controller. > > > + > > > + Say 'y/m' to enable a driver named "cxl_mem.ko" that will attach > > > + to CXL.mem devices for configuration, provisioning, and health > > > + monitoring, the so called "type-3 mailbox". Note, this driver > > > + is required for dynamic provisioning of CXL.mem attached > > > + memory, a pre-requisite for persistent memory support, but > > > + devices that provide volatile memory may be fully described by > > > + existing platform firmware memory enumeration. > > > + > > > + If unsure say 'n'. > > > endif > > > diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile > > > index d38cd34a2582..97fdffb00f2d 100644 > > > --- a/drivers/cxl/Makefile > > > +++ b/drivers/cxl/Makefile > > > @@ -1,5 +1,7 @@ > > > # SPDX-License-Identifier: GPL-2.0 > > > obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o > > > +obj-$(CONFIG_CXL_MEM) += cxl_mem.o > > > > > > ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL > > > cxl_acpi-y := acpi.o > > > +cxl_mem-y := mem.o > > > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c > > > new file mode 100644 > > > index 000000000000..aa7d881fa47b > > > --- /dev/null > > > +++ b/drivers/cxl/mem.c > > > @@ -0,0 +1,82 @@ > > > +// SPDX-License-Identifier: GPL-2.0-only > > > +// Copyright(c) 2020 Intel Corporation. All rights reserved. > > > +#include <linux/module.h> > > > +#include <linux/pci.h> > > > +#include <linux/io.h> > > > +#include "acpi.h" > > > +#include "pci.h" > > > + > > > +struct cxl_mem { > > > + void __iomem *regs; > > > +}; > > > + > > > +static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec) > > > +{ > > > + int pos; > > > + > > > + pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DVSEC); > > > + if (!pos) > > > + return 0; > > > + > > > + while (pos) { > > > + u16 vendor, id; > > > + > > > + pci_read_config_word(pdev, pos + PCI_DVSEC_VENDOR_OFFSET, &vendor); > > > + pci_read_config_word(pdev, pos + PCI_DVSEC_ID_OFFSET, &id); > > > + if (vendor == PCI_DVSEC_VENDOR_CXL && dvsec == id) > > > + return pos; > > > + > > > + pos = pci_find_next_ext_capability(pdev, pos, PCI_EXT_CAP_ID_DVSEC); > > > > This is good generic code and wouldn't cause much backport effort (even if needed > > to bring in a local copy), so perhaps make it a generic function and move to > > core PCI code? > > > > Mind you I guess that can happen the 'second' time someone wants to find a DVSEC. > > > > > + } > > > + > > > + return 0; > > > +} > > > + > > > +static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) > > > +{ > > > + struct device *dev = &pdev->dev; > > > + struct cxl_mem *cxlm; > > > + int rc, regloc; > > > + > > > + rc = cxl_bus_prepared(pdev); > > > + if (rc != 0) { > > > + dev_err(dev, "failed to acquire interface\n"); > > > + return rc; > > > + } > > > + > > > + regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC); > > > + if (!regloc) { > > > + dev_err(dev, "register location dvsec not found\n"); > > > + return -ENXIO; > > > + } > > > + > > > + cxlm = devm_kzalloc(dev, sizeof(*cxlm), GFP_KERNEL); > > > + if (!cxlm) > > > + return -ENOMEM; > > > + > > > + return 0; > > > +} > > > + > > > +static void cxl_mem_remove(struct pci_dev *pdev) > > > +{ > > > +} > > > > I'd bring this in only when needed in later patch. > > > > > + > > > +static const struct pci_device_id cxl_mem_pci_tbl[] = { > > > + /* PCI class code for CXL.mem Type-3 Devices */ > > > + { PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID, > > > + PCI_CLASS_MEMORY_CXL, 0xffffff, 0 }, > > > + { /* terminate list */ }, > > > +}; > > > +MODULE_DEVICE_TABLE(pci, cxl_mem_pci_tbl); > > > + > > > +static struct pci_driver cxl_mem_driver = { > > > + .name = KBUILD_MODNAME, > > > + .id_table = cxl_mem_pci_tbl, > > > + .probe = cxl_mem_probe, > > > + .remove = cxl_mem_remove, > > > +}; > > > + > > > +MODULE_LICENSE("GPL v2"); > > > +MODULE_AUTHOR("Intel Corporation"); > > > +module_pci_driver(cxl_mem_driver); > > > +MODULE_IMPORT_NS(CXL); > > > diff --git a/drivers/cxl/pci.h b/drivers/cxl/pci.h > > > new file mode 100644 > > > index 000000000000..beb03921e6da > > > --- /dev/null > > > +++ b/drivers/cxl/pci.h > > > @@ -0,0 +1,15 @@ > > > +// SPDX-License-Identifier: GPL-2.0-only > > > +// Copyright(c) 2020 Intel Corporation. All rights reserved. > > > +#ifndef __CXL_PCI_H__ > > > +#define __CXL_PCI_H__ > > > + > > > +#define PCI_CLASS_MEMORY_CXL 0x050210 > > > + > > > +#define PCI_EXT_CAP_ID_DVSEC 0x23 > > > +#define PCI_DVSEC_VENDOR_CXL 0x1E98 > > > > Hmm. The magic question of what to call a vendor ID that isn't a vendor > > ID but just a magic number that talks like a duck and quacks like a duck > > (for anyone wondering what I'm talking about, there is a nice bit of legal > > boilerplate on this in the CXL spec) > > > > This name is definitely not accurate however. > > > > PCI_UNIQUE_VALUE_CXL maybe? It is used for other things than DVSEC (VDMs etc), > > though possibly this is the only software visible use. > > Finally working my way back through this review to make the changes. > If 0x1E98 becomes visible to software somewhere else then this can > become something like the following: > > #define PCI_DVSEC_VENDOR_CXL PCI_UNIQUE_VALUE_CXL > > ...or whatever the generic name is, but this field per the > specification is the DVSEC-vendor-id and calling it > PCI_UNIQUE_VALUE_CXL does not have any basis in the spec. > > I will rename it though to: > > PCI_DVSEC_VENDOR_ID_CXL > > ...since include/linux/pci_ids.h includes the _ID_ part. > > > > > > > > +#define PCI_DVSEC_VENDOR_OFFSET 0x4 > > > +#define PCI_DVSEC_ID_OFFSET 0x8 > > > > Put a line break here perhaps and maybe a spec reference to where to find > > the various DVSEC IDs. > > Ok. > > > > > > +#define PCI_DVSEC_ID_CXL 0x0 > > > > That's definitely a confusing name as well. > > Yeah, should be PCI_DVSEC_DEVICE_ID_CXL Actually, no, the spec calls this the "DVSEC id" so PCI_DVSEC_ID_CXL seems suitable to me. This is from: Table 126. PCI Express DVSEC Register Settings for CXL Device In the CXL 2.0 Specification.
On Wed, Nov 25, 2020 at 10:06 PM Jon Masters <jcm@jonmasters.org> wrote: > > On 11/11/20 12:43 AM, Ben Widawsky wrote: > > > + case CXL_CAPABILITIES_CAP_ID_SECONDARY_MAILBOX: > > + dev_dbg(&cxlm->pdev->dev, > > + "found UNSUPPORTED Secondary Mailbox capability\n"); > > Per spec, the secondary mailbox is intended for use by platform > firmware, so Linux should never be using it anyway. Maybe that message > is slightly misleading? > > Jon. > > P.S. Related - I've severe doubts about the mailbox approach being > proposed by CXL and have begun to push back through the spec org. The more Linux software voices the better. At the same time the spec is released so we're into xkcd territory [1] of what the driver will be expected to support for any future improvements. [1]: https://xkcd.com/927/
On Tue, Nov 10, 2020 at 9:44 PM Ben Widawsky <ben.widawsky@intel.com> wrote: > > CXL devices contain an array of capabilities that describe the > interactions software can interact with the device, or firmware running > on the device. A CXL compliant device must implement the device status > and the mailbox capability. A CXL compliant memory device must implement > the memory device capability. > > Each of the capabilities can [will] provide an offset within the MMIO > region for interacting with the CXL device. > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com> > --- > drivers/cxl/cxl.h | 89 +++++++++++++++++++++++++++++++++++++++++++++++ > drivers/cxl/mem.c | 58 +++++++++++++++++++++++++++--- > 2 files changed, 143 insertions(+), 4 deletions(-) > create mode 100644 drivers/cxl/cxl.h > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h > new file mode 100644 > index 000000000000..02858ae63d6d > --- /dev/null > +++ b/drivers/cxl/cxl.h [..] > +static inline u32 __cxl_raw_read_reg32(struct cxl_mem *cxlm, u32 reg) Going through my reworks and the "raw" jumped out at me. My typical interpretation of "raw" in respect to register access macros is the difference between readl() and __raw_readl() which means "don't do bus endian swizzling, and don't do a memory clobber barrier". Any heartburn to drop the "raw"? ...is it only me that reacts that way?
... > > > > +MODULE_IMPORT_NS(CXL); > > > > diff --git a/drivers/cxl/pci.h b/drivers/cxl/pci.h > > > > new file mode 100644 > > > > index 000000000000..beb03921e6da > > > > --- /dev/null > > > > +++ b/drivers/cxl/pci.h > > > > @@ -0,0 +1,15 @@ > > > > +// SPDX-License-Identifier: GPL-2.0-only > > > > +// Copyright(c) 2020 Intel Corporation. All rights reserved. > > > > +#ifndef __CXL_PCI_H__ > > > > +#define __CXL_PCI_H__ > > > > + > > > > +#define PCI_CLASS_MEMORY_CXL 0x050210 > > > > + > > > > +#define PCI_EXT_CAP_ID_DVSEC 0x23 > > > > +#define PCI_DVSEC_VENDOR_CXL 0x1E98 > > > > > > Hmm. The magic question of what to call a vendor ID that isn't a vendor > > > ID but just a magic number that talks like a duck and quacks like a duck > > > (for anyone wondering what I'm talking about, there is a nice bit of legal > > > boilerplate on this in the CXL spec) > > > > > > This name is definitely not accurate however. > > > > > > PCI_UNIQUE_VALUE_CXL maybe? It is used for other things than DVSEC (VDMs etc), > > > though possibly this is the only software visible use. > > > > Finally working my way back through this review to make the changes. > > If 0x1E98 becomes visible to software somewhere else then this can > > become something like the following: > > > > #define PCI_DVSEC_VENDOR_CXL PCI_UNIQUE_VALUE_CXL > > > > ...or whatever the generic name is, but this field per the > > specification is the DVSEC-vendor-id and calling it > > PCI_UNIQUE_VALUE_CXL does not have any basis in the spec. There is a big statement about it as a footnote to 3.1.2 in CXL 2.0 "The Unique Value that is provided in this specification for use in ... Designated Vendor Specific Extended Capabilities.." And for extra amusement in the "Notice Regarding PCI-SIG Unique Value" that forms part of the click through https://www.computeexpresslink.org/download-the-specification (that's the only use of "PCI-SIG Unique Value" that Google finds but I know of one other similar statement) However, I agree it's being used in DVSEC field only (from software point of view) so fair enough to name it after where it is used rather than what it is. > > > > I will rename it though to: > > > > PCI_DVSEC_VENDOR_ID_CXL > > > > ...since include/linux/pci_ids.h includes the _ID_ part. > > > > > > > > > > > > +#define PCI_DVSEC_VENDOR_OFFSET 0x4 > > > > +#define PCI_DVSEC_ID_OFFSET 0x8 > > > > > > Put a line break here perhaps and maybe a spec reference to where to find > > > the various DVSEC IDs. > > > > Ok. > > > > > > > > > +#define PCI_DVSEC_ID_CXL 0x0 > > > > > > That's definitely a confusing name as well. > > > > Yeah, should be PCI_DVSEC_DEVICE_ID_CXL > > Actually, no, the spec calls this the "DVSEC id" so PCI_DVSEC_ID_CXL > seems suitable to me. This is from: > > Table 126. PCI Express DVSEC Register Settings for CXL Device > > In the CXL 2.0 Specification. The DVSEC ID naming is straight from the PCI spec so that part is fine, my issue is this is one of a whole bunch of CXL related DVSEC ID so it needs a more specific name. PCI_DVSEC_ID_CXL_DEVICE would work in line with table 124. I'm not that bothered though. Jonathan
On 20-12-03 23:41:16, Dan Williams wrote: > On Tue, Nov 10, 2020 at 9:44 PM Ben Widawsky <ben.widawsky@intel.com> wrote: > > > > CXL devices contain an array of capabilities that describe the > > interactions software can interact with the device, or firmware running > > on the device. A CXL compliant device must implement the device status > > and the mailbox capability. A CXL compliant memory device must implement > > the memory device capability. > > > > Each of the capabilities can [will] provide an offset within the MMIO > > region for interacting with the CXL device. > > > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com> > > --- > > drivers/cxl/cxl.h | 89 +++++++++++++++++++++++++++++++++++++++++++++++ > > drivers/cxl/mem.c | 58 +++++++++++++++++++++++++++--- > > 2 files changed, 143 insertions(+), 4 deletions(-) > > create mode 100644 drivers/cxl/cxl.h > > > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h > > new file mode 100644 > > index 000000000000..02858ae63d6d > > --- /dev/null > > +++ b/drivers/cxl/cxl.h > [..] > > +static inline u32 __cxl_raw_read_reg32(struct cxl_mem *cxlm, u32 reg) > > Going through my reworks and the "raw" jumped out at me. My typical > interpretation of "raw" in respect to register access macros is the > difference between readl() and __raw_readl() which means "don't do > bus endian swizzling, and don't do a memory clobber barrier". Any > heartburn to drop the "raw"? > > ...is it only me that reacts that way? I will drop "raw". Especially given that I intend to reuse the word in v2 for something entirely different, it makes sense. My idea of "raw" was that it's just unfettered access to the device's MMIO space. No offsets, no checks. I'm not sure of a better adjective to describe that, but if you have any in mind, I'd like to add it.