From patchwork Thu May 14 14:07:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeffrey Hugo X-Patchwork-Id: 189148 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 400A4C433E2 for ; Thu, 14 May 2020 14:08:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1AEAB2070A for ; Thu, 14 May 2020 14:08:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=mg.codeaurora.org header.i=@mg.codeaurora.org header.b="hFBKs93H" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727939AbgENOIc (ORCPT ); Thu, 14 May 2020 10:08:32 -0400 Received: from mail27.static.mailgun.info ([104.130.122.27]:19130 "EHLO mail27.static.mailgun.info" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727903AbgENOIb (ORCPT ); Thu, 14 May 2020 10:08:31 -0400 DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg.codeaurora.org; q=dns/txt; s=smtp; t=1589465310; h=References: In-Reply-To: Message-Id: Date: Subject: Cc: To: From: Sender; bh=44LNTWvl0HXLuuSgfcp8QfvHF49lLFxWFT5IEPtC01E=; b=hFBKs93Hmumyl/K9qwcAJEOwfbHDZvxdfeZuIFWZp60LlFX6o8jSYyNwlv+g2KIRktxP4gFX mXWL9AvYghoVEl/n1Sl1TWvEOSKq2hTdnADeNpHsoDVNs4hk8++PVniCHLeHFFYrzDFIcG4b 3CI3MQ8RMI1IyGzwODHeC4dYbrU= X-Mailgun-Sending-Ip: 104.130.122.27 X-Mailgun-Sid: WyI1MzIzYiIsICJsaW51eC1hcm0tbXNtQHZnZXIua2VybmVsLm9yZyIsICJiZTllNGEiXQ== Received: from smtp.codeaurora.org (ec2-35-166-182-171.us-west-2.compute.amazonaws.com [35.166.182.171]) by mxa.mailgun.org with ESMTP id 5ebd50cc.7f337c0c1928-smtp-out-n02; Thu, 14 May 2020 14:08:12 -0000 (UTC) Received: by smtp.codeaurora.org (Postfix, from userid 1001) id D2C41C43637; Thu, 14 May 2020 14:08:11 +0000 (UTC) Received: from jhugo-perf-lnx.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: jhugo) by smtp.codeaurora.org (Postfix) with ESMTPSA id 537C3C433F2; Thu, 14 May 2020 14:08:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org 537C3C433F2 Authentication-Results: aws-us-west-2-caf-mail-1.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: aws-us-west-2-caf-mail-1.web.codeaurora.org; spf=none smtp.mailfrom=jhugo@codeaurora.org From: Jeffrey Hugo To: arnd@arndb.de, gregkh@linuxfoundation.org Cc: manivannan.sadhasivam@linaro.org, bjorn.andersson@linaro.org, wufan@codeaurora.org, pratanan@codeaurora.org, linux-arm-msm@vger.kernel.org, linux-kernel@vger.kernel.org, Jeffrey Hugo Subject: [RFC PATCH 1/8] qaic: Add skeleton driver Date: Thu, 14 May 2020 08:07:39 -0600 Message-Id: <1589465266-20056-2-git-send-email-jhugo@codeaurora.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1589465266-20056-1-git-send-email-jhugo@codeaurora.org> References: <1589465266-20056-1-git-send-email-jhugo@codeaurora.org> Sender: linux-arm-msm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Add the Qualcomm Cloud AI 100 skeleton driver that does nothing but registers a pci driver, probes on the proper device, and supports removal of the module. Signed-off-by: Jeffrey Hugo --- drivers/misc/Kconfig | 1 + drivers/misc/Makefile | 1 + drivers/misc/qaic/Kconfig | 20 ++++++++++++++ drivers/misc/qaic/Makefile | 10 +++++++ drivers/misc/qaic/qaic_drv.c | 62 ++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 94 insertions(+) create mode 100644 drivers/misc/qaic/Kconfig create mode 100644 drivers/misc/qaic/Makefile create mode 100644 drivers/misc/qaic/qaic_drv.c diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig index 99e1514..ef91d062 100644 --- a/drivers/misc/Kconfig +++ b/drivers/misc/Kconfig @@ -481,4 +481,5 @@ source "drivers/misc/ocxl/Kconfig" source "drivers/misc/cardreader/Kconfig" source "drivers/misc/habanalabs/Kconfig" source "drivers/misc/uacce/Kconfig" +source "drivers/misc/qaic/Kconfig" endmenu diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile index 9abf292..64f0770 100644 --- a/drivers/misc/Makefile +++ b/drivers/misc/Makefile @@ -58,3 +58,4 @@ obj-$(CONFIG_PVPANIC) += pvpanic.o obj-$(CONFIG_HABANA_AI) += habanalabs/ obj-$(CONFIG_UACCE) += uacce/ obj-$(CONFIG_XILINX_SDFEC) += xilinx_sdfec.o +obj-$(CONFIG_QAIC) += qaic/ diff --git a/drivers/misc/qaic/Kconfig b/drivers/misc/qaic/Kconfig new file mode 100644 index 0000000..502d176 --- /dev/null +++ b/drivers/misc/qaic/Kconfig @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# Qualcomm Cloud AI 100 accelerators driver +# + +config QAIC + tristate "Qualcomm Cloud AI 100 accelerators" + depends on PCI && HAS_IOMEM + depends on MHI_BUS + help + Enables driver for Qualcomm's Cloud AI 100 accelerator PCIe cards + that are designed to accelerate Deep Learning inference workloads. + + The driver manages the PCIe devices and provides an IOCTL interface + for users to submit workloads to the devices. + + If unsure, say N. + + To compile this driver as a module, choose M here: the + module will be called qaic. diff --git a/drivers/misc/qaic/Makefile b/drivers/misc/qaic/Makefile new file mode 100644 index 0000000..42149ac --- /dev/null +++ b/drivers/misc/qaic/Makefile @@ -0,0 +1,10 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# Makefile for Qualcomm Cloud AI 100 accelerators driver +# + + +obj-$(CONFIG_QAIC) := qaic.o + +qaic-y := \ + qaic_drv.o diff --git a/drivers/misc/qaic/qaic_drv.c b/drivers/misc/qaic/qaic_drv.c new file mode 100644 index 0000000..addd9ea --- /dev/null +++ b/drivers/misc/qaic/qaic_drv.c @@ -0,0 +1,62 @@ +// SPDX-License-Identifier: GPL-2.0-only + +/* Copyright (c) 2019-2020, The Linux Foundation. All rights reserved. */ + +#include +#include +#include + +#define PCI_DEV_AIC100 0xa100 + +#define QAIC_NAME "Qualcomm Cloud AI 100" + +static int qaic_pci_probe(struct pci_dev *pdev, + const struct pci_device_id *id) +{ + pci_dbg(pdev, "%s\n", __func__); + pci_dbg(pdev, "%s: successful init\n", __func__); + return 0; +} + +static void qaic_pci_remove(struct pci_dev *pdev) +{ + pci_dbg(pdev, "%s\n", __func__); +} + +static const struct pci_device_id ids[] = { + { PCI_DEVICE(PCI_VENDOR_ID_QCOM, PCI_DEV_AIC100), }, + { 0, } +}; +MODULE_DEVICE_TABLE(pci, ids); + +static struct pci_driver qaic_pci_driver = { + .name = QAIC_NAME, + .id_table = ids, + .probe = qaic_pci_probe, + .remove = qaic_pci_remove, +}; + +static int __init qaic_init(void) +{ + int ret; + + pr_debug("qaic: init\n"); + + ret = pci_register_driver(&qaic_pci_driver); + + return ret; +} + +static void __exit qaic_exit(void) +{ + pr_debug("qaic: exit\n"); + pci_unregister_driver(&qaic_pci_driver); +} + +module_init(qaic_init); +module_exit(qaic_exit); + +MODULE_AUTHOR("Qualcomm Cloud AI 100 Accelerator Kernel Driver Team"); +MODULE_DESCRIPTION("Qualcomm Cloud 100 AI Accelerators Driver"); +MODULE_LICENSE("GPL v2"); +MODULE_VERSION("0.0.0"); /* MAJOR.MINOR.PATCH */ From patchwork Thu May 14 14:07:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeffrey Hugo X-Patchwork-Id: 189147 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F3A2C433E1 for ; Thu, 14 May 2020 14:08:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 48AFE2074A for ; Thu, 14 May 2020 14:08:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=mg.codeaurora.org header.i=@mg.codeaurora.org header.b="GcT05nCz" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727958AbgENOIi (ORCPT ); Thu, 14 May 2020 10:08:38 -0400 Received: from mail26.static.mailgun.info ([104.130.122.26]:37514 "EHLO mail26.static.mailgun.info" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727903AbgENOIh (ORCPT ); Thu, 14 May 2020 10:08:37 -0400 DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg.codeaurora.org; q=dns/txt; s=smtp; t=1589465316; h=References: In-Reply-To: Message-Id: Date: Subject: Cc: To: From: Sender; bh=/r1bgVMeJkBXiz1H+2qM7BXH6fCeQ1Tn4MROHmsuqVs=; b=GcT05nCzsl+vV6nCF9abLKIao2znkULwzbVouAA3t6tuvNKYAisXX0Ygo7nAIvDp9+Bvgc84 Y+PO3WObJkUH6tKcdHCw2NIDRix9+bLKTODHCmdP0Bz96k57dEW/hekErUTXwu3GqZ/mMSVM lD93520O7jCnK1ke/9Wik12d72E= X-Mailgun-Sending-Ip: 104.130.122.26 X-Mailgun-Sid: WyI1MzIzYiIsICJsaW51eC1hcm0tbXNtQHZnZXIua2VybmVsLm9yZyIsICJiZTllNGEiXQ== Received: from smtp.codeaurora.org (ec2-35-166-182-171.us-west-2.compute.amazonaws.com [35.166.182.171]) by mxa.mailgun.org with ESMTP id 5ebd50d2.7fa03765f0a0-smtp-out-n02; Thu, 14 May 2020 14:08:18 -0000 (UTC) Received: by smtp.codeaurora.org (Postfix, from userid 1001) id A14DDC433D2; Thu, 14 May 2020 14:08:17 +0000 (UTC) Received: from jhugo-perf-lnx.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: jhugo) by smtp.codeaurora.org (Postfix) with ESMTPSA id ADA0DC433D2; Thu, 14 May 2020 14:08:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org ADA0DC433D2 Authentication-Results: aws-us-west-2-caf-mail-1.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: aws-us-west-2-caf-mail-1.web.codeaurora.org; spf=none smtp.mailfrom=jhugo@codeaurora.org From: Jeffrey Hugo To: arnd@arndb.de, gregkh@linuxfoundation.org Cc: manivannan.sadhasivam@linaro.org, bjorn.andersson@linaro.org, wufan@codeaurora.org, pratanan@codeaurora.org, linux-arm-msm@vger.kernel.org, linux-kernel@vger.kernel.org, Jeffrey Hugo Subject: [RFC PATCH 3/8] qaic: Create char dev Date: Thu, 14 May 2020 08:07:41 -0600 Message-Id: <1589465266-20056-4-git-send-email-jhugo@codeaurora.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1589465266-20056-1-git-send-email-jhugo@codeaurora.org> References: <1589465266-20056-1-git-send-email-jhugo@codeaurora.org> Sender: linux-arm-msm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Now that we can fully boot the device, we should start making it usable. The primary interface to the device will be via a char dev. Add the necessary framework to detect when the device is fully booted, and create the char dev at that point. The device is only usable when it is fully booted. The char dev does nothing useful yet, but we can easily build on this to provide functionality. Signed-off-by: Jeffrey Hugo --- drivers/misc/qaic/qaic.h | 19 +++ drivers/misc/qaic/qaic_drv.c | 303 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 321 insertions(+), 1 deletion(-) diff --git a/drivers/misc/qaic/qaic.h b/drivers/misc/qaic/qaic.h index 379aa82..58ca167 100644 --- a/drivers/misc/qaic/qaic.h +++ b/drivers/misc/qaic/qaic.h @@ -6,13 +6,32 @@ #ifndef QAICINTERNAL_H_ #define QAICINTERNAL_H_ +#include +#include #include +#include #include +#include + +struct qaic_user { + pid_t handle; + struct qaic_device *qdev; + struct list_head node; + struct srcu_struct qdev_lock; + struct kref ref_count; +}; struct qaic_device { struct pci_dev *pdev; int bars; void __iomem *bar_0; struct mhi_controller *mhi_cntl; + struct mhi_device *cntl_ch; + struct cdev *cdev; + struct device *dev; + bool in_reset; + struct srcu_struct dev_lock; + struct list_head users; + struct mutex users_mutex; }; #endif /* QAICINTERNAL_H_ */ diff --git a/drivers/misc/qaic/qaic_drv.c b/drivers/misc/qaic/qaic_drv.c index b624daa..6e4b936 100644 --- a/drivers/misc/qaic/qaic_drv.c +++ b/drivers/misc/qaic/qaic_drv.c @@ -2,8 +2,14 @@ /* Copyright (c) 2019-2020, The Linux Foundation. All rights reserved. */ +#include +#include +#include +#include +#include #include #include +#include #include #include @@ -13,9 +19,242 @@ #define PCI_DEV_AIC100 0xa100 #define QAIC_NAME "Qualcomm Cloud AI 100" +#define QAIC_MAX_MINORS 256 +static int qaic_major; +static struct class *qaic_class; +static DEFINE_IDR(qaic_devs); +static DEFINE_MUTEX(qaic_devs_lock); static bool link_up; +static int qaic_device_open(struct inode *inode, struct file *filp); +static int qaic_device_release(struct inode *inode, struct file *filp); + +static const struct file_operations qaic_ops = { + .owner = THIS_MODULE, + .open = qaic_device_open, + .release = qaic_device_release, +}; + +static void free_usr(struct kref *kref) +{ + struct qaic_user *usr = container_of(kref, struct qaic_user, ref_count); + + list_del(&usr->node); + cleanup_srcu_struct(&usr->qdev_lock); + kfree(usr); +} + +static int qaic_device_open(struct inode *inode, struct file *filp) +{ + struct qaic_device *qdev; + struct qaic_user *usr; + int rcu_id; + int ret; + + ret = mutex_lock_interruptible(&qaic_devs_lock); + if (ret) + return ret; + qdev = idr_find(&qaic_devs, iminor(inode)); + mutex_unlock(&qaic_devs_lock); + + pci_dbg(qdev->pdev, "%s pid:%d\n", __func__, current->pid); + + rcu_id = srcu_read_lock(&qdev->dev_lock); + if (qdev->in_reset) { + srcu_read_unlock(&qdev->dev_lock, rcu_id); + return -ENODEV; + } + + usr = kmalloc(sizeof(*usr), GFP_KERNEL); + if (!usr) + return -ENOMEM; + + usr->handle = current->pid; + usr->qdev = qdev; + init_srcu_struct(&usr->qdev_lock); + kref_init(&usr->ref_count); + + ret = mutex_lock_interruptible(&qdev->users_mutex); + if (ret) { + cleanup_srcu_struct(&usr->qdev_lock); + kfree(usr); + srcu_read_unlock(&qdev->dev_lock, rcu_id); + return ret; + } + + list_add(&usr->node, &qdev->users); + mutex_unlock(&qdev->users_mutex); + + filp->private_data = usr; + nonseekable_open(inode, filp); + + srcu_read_unlock(&qdev->dev_lock, rcu_id); + return 0; +} + +static int qaic_device_release(struct inode *inode, struct file *filp) +{ + struct qaic_user *usr = filp->private_data; + struct qaic_device *qdev = usr->qdev; + int qdev_rcu_id; + int usr_rcu_id; + + usr_rcu_id = srcu_read_lock(&usr->qdev_lock); + if (qdev) { + qdev_rcu_id = srcu_read_lock(&qdev->dev_lock); + if (!qdev->in_reset) { + pci_dbg(qdev->pdev, "%s pid:%d\n", __func__, + current->pid); + } + srcu_read_unlock(&qdev->dev_lock, qdev_rcu_id); + + srcu_read_unlock(&usr->qdev_lock, usr_rcu_id); + mutex_lock(&qdev->users_mutex); + kref_put(&usr->ref_count, free_usr); + mutex_unlock(&qdev->users_mutex); + } else { + srcu_read_unlock(&usr->qdev_lock, usr_rcu_id); + /* safe to do without the mutex because reset already has ref */ + kref_put(&usr->ref_count, free_usr); + } + + filp->private_data = NULL; + return 0; +} + +static int qaic_mhi_probe(struct mhi_device *mhi_dev, + const struct mhi_device_id *id) +{ + struct qaic_device *qdev; + dev_t devno; + int ret; + + /* + * Invoking this function indicates that the control channel to the + * device is available. We use that as a signal to indicate that + * the device side firmware has booted. The device side firmware + * manages the device resources, so we need to communicate with it + * via the control channel in order to utilize the device. Therefore + * we wait until this signal to create the char dev that userspace will + * use to control the device, because without the device side firmware, + * userspace can't do anything useful. + */ + + qdev = (struct qaic_device *)pci_get_drvdata( + to_pci_dev(mhi_dev->mhi_cntrl->cntrl_dev)); + + pci_dbg(qdev->pdev, "%s\n", __func__); + qdev->in_reset = false; + + dev_set_drvdata(&mhi_dev->dev, qdev); + qdev->cntl_ch = mhi_dev; + + mutex_lock(&qaic_devs_lock); + ret = idr_alloc(&qaic_devs, qdev, 0, QAIC_MAX_MINORS, GFP_KERNEL); + mutex_unlock(&qaic_devs_lock); + + if (ret < 0) { + pci_dbg(qdev->pdev, "%s: idr_alloc failed %d\n", __func__, ret); + goto err; + } + + devno = MKDEV(qaic_major, ret); + + qdev->cdev = cdev_alloc(); + if (!qdev->cdev) { + pci_dbg(qdev->pdev, "%s: cdev_alloc failed\n", __func__); + ret = -ENOMEM; + goto free_idr; + } + + qdev->cdev->owner = THIS_MODULE; + qdev->cdev->ops = &qaic_ops; + ret = cdev_add(qdev->cdev, devno, 1); + if (ret) { + pci_dbg(qdev->pdev, "%s: cdev_add failed %d\n", __func__, ret); + goto free_cdev; + } + + qdev->dev = device_create(qaic_class, NULL, devno, NULL, + "qaic_aic100_%04x:%02x:%02x.%d", + pci_domain_nr(qdev->pdev->bus), + qdev->pdev->bus->number, + PCI_SLOT(qdev->pdev->devfn), + PCI_FUNC(qdev->pdev->devfn)); + if (IS_ERR(qdev->dev)) { + ret = PTR_ERR(qdev->dev); + pci_dbg(qdev->pdev, "%s: device_create failed %d\n", __func__, ret); + goto free_cdev; + } + + dev_set_drvdata(qdev->dev, qdev); + + return 0; + +free_cdev: + cdev_del(qdev->cdev); +free_idr: + mutex_lock(&qaic_devs_lock); + idr_remove(&qaic_devs, MINOR(devno)); + mutex_unlock(&qaic_devs_lock); +err: + return ret; +} + +static void qaic_mhi_remove(struct mhi_device *mhi_dev) +{ +} + +static void qaic_mhi_ul_xfer_cb(struct mhi_device *mhi_dev, + struct mhi_result *mhi_result) +{ +} + +static void qaic_mhi_dl_xfer_cb(struct mhi_device *mhi_dev, + struct mhi_result *mhi_result) +{ +} + +void qaic_dev_reset_clean_local_state(struct qaic_device *qdev) +{ + struct qaic_user *usr; + struct qaic_user *u; + dev_t devno; + + qdev->in_reset = true; + synchronize_srcu(&qdev->dev_lock); + + /* + * while the usr still has access to the qdev, use the mutex to add + * a reference for later. This makes sure the usr can't disappear on + * us at the wrong time. The mutex use in close() system call handling + * makes sure the usr will be valid or complete not exist here. + */ + mutex_lock(&qdev->users_mutex); + list_for_each_entry_safe(usr, u, &qdev->users, node) + kref_get(&usr->ref_count); + mutex_unlock(&qdev->users_mutex); + + /* remove chardev to prevent new users from coming in */ + if (qdev->dev) { + devno = qdev->dev->devt; + qdev->dev = NULL; + device_destroy(qaic_class, devno); + cdev_del(qdev->cdev); + mutex_lock(&qaic_devs_lock); + idr_remove(&qaic_devs, MINOR(devno)); + mutex_unlock(&qaic_devs_lock); + } + + /* make existing users get unresolvable errors until they close FDs */ + list_for_each_entry_safe(usr, u, &qdev->users, node) { + usr->qdev = NULL; + synchronize_srcu(&usr->qdev_lock); + kref_put(&usr->ref_count, free_usr); + } +} + static int qaic_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) { @@ -33,6 +272,9 @@ static int qaic_pci_probe(struct pci_dev *pdev, pci_set_drvdata(pdev, qdev); qdev->pdev = pdev; + init_srcu_struct(&qdev->dev_lock); + INIT_LIST_HEAD(&qdev->users); + mutex_init(&qdev->users_mutex); qdev->bars = pci_select_bars(pdev, IORESOURCE_MEM); @@ -107,6 +349,7 @@ static int qaic_pci_probe(struct pci_dev *pdev, enable_fail: pci_set_drvdata(pdev, NULL); bar_fail: + cleanup_srcu_struct(&qdev->dev_lock); kfree(qdev); qdev_fail: return ret; @@ -120,6 +363,7 @@ static void qaic_pci_remove(struct pci_dev *pdev) if (!qdev) return; + qaic_dev_reset_clean_local_state(qdev); qaic_mhi_free_controller(qdev->mhi_cntl, link_up); pci_free_irq_vectors(pdev); iounmap(qdev->bar_0); @@ -127,9 +371,27 @@ static void qaic_pci_remove(struct pci_dev *pdev) pci_release_selected_regions(pdev, qdev->bars); pci_disable_device(pdev); pci_set_drvdata(pdev, NULL); + cleanup_srcu_struct(&qdev->dev_lock); kfree(qdev); } +static const struct mhi_device_id qaic_mhi_match_table[] = { + { .chan = "QAIC_CONTROL", }, + {}, +}; + +static struct mhi_driver qaic_mhi_driver = { + .id_table = qaic_mhi_match_table, + .remove = qaic_mhi_remove, + .probe = qaic_mhi_probe, + .ul_xfer_cb = qaic_mhi_ul_xfer_cb, + .dl_xfer_cb = qaic_mhi_dl_xfer_cb, + .driver = { + .name = "qaic_mhi", + .owner = THIS_MODULE, + }, +}; + static const struct pci_device_id ids[] = { { PCI_DEVICE(PCI_VENDOR_ID_QCOM, PCI_DEV_AIC100), }, { 0, } @@ -146,13 +408,48 @@ static struct pci_driver qaic_pci_driver = { static int __init qaic_init(void) { int ret; + dev_t dev; pr_debug("qaic: init\n"); + ret = alloc_chrdev_region(&dev, 0, QAIC_MAX_MINORS, QAIC_NAME); + if (ret < 0) { + pr_debug("qaic: alloc_chrdev_region failed %d\n", ret); + goto out; + } + + qaic_major = MAJOR(dev); + + qaic_class = class_create(THIS_MODULE, QAIC_NAME); + if (IS_ERR(qaic_class)) { + ret = PTR_ERR(qaic_class); + pr_debug("qaic: class_create failed %d\n", ret); + goto free_major; + } + + ret = mhi_driver_register(&qaic_mhi_driver); + if (ret) { + pr_debug("qaic: mhi_driver_register failed %d\n", ret); + goto free_class; + } + ret = pci_register_driver(&qaic_pci_driver); - pr_debug("qaic: init success\n"); + if (ret) { + pr_debug("qaic: pci_register_driver failed %d\n", ret); + goto free_mhi; + } + pr_debug("qaic: init success\n"); + goto out; + +free_mhi: + mhi_driver_unregister(&qaic_mhi_driver); +free_class: + class_destroy(qaic_class); +free_major: + unregister_chrdev_region(MKDEV(qaic_major, 0), QAIC_MAX_MINORS); +out: return ret; } @@ -161,6 +458,10 @@ static void __exit qaic_exit(void) pr_debug("qaic: exit\n"); link_up = true; pci_unregister_driver(&qaic_pci_driver); + mhi_driver_unregister(&qaic_mhi_driver); + class_destroy(qaic_class); + unregister_chrdev_region(MKDEV(qaic_major, 0), QAIC_MAX_MINORS); + idr_destroy(&qaic_devs); } module_init(qaic_init); From patchwork Thu May 14 14:07:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeffrey Hugo X-Patchwork-Id: 189146 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71A0EC433E0 for ; Thu, 14 May 2020 14:08:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4E4E820767 for ; Thu, 14 May 2020 14:08:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=mg.codeaurora.org header.i=@mg.codeaurora.org header.b="lBw6Rrqe" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727900AbgENOIo (ORCPT ); Thu, 14 May 2020 10:08:44 -0400 Received: from mail26.static.mailgun.info ([104.130.122.26]:37514 "EHLO mail26.static.mailgun.info" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727956AbgENOIo (ORCPT ); Thu, 14 May 2020 10:08:44 -0400 DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg.codeaurora.org; q=dns/txt; s=smtp; t=1589465323; h=References: In-Reply-To: Message-Id: Date: Subject: Cc: To: From: Sender; bh=4N3Qgu7jkYHdJTEPxc6flfM5C9+ZWgywokKlAJopoq0=; b=lBw6RrqeGltw1Ch8d6Ih+IvwamIw11wfZK09ocxJ9YIlVs39W3uk+uMuNjza7zMTyV11nduo FUe2hIPYd5QmrcMaU7njcsZOH/+RdOUl41MEuwUFi+rtv+OCiJJzTZUntIwnVeNttckFv/FX mxro1MFERrdYOItlHv8P++bUUag= X-Mailgun-Sending-Ip: 104.130.122.26 X-Mailgun-Sid: WyI1MzIzYiIsICJsaW51eC1hcm0tbXNtQHZnZXIua2VybmVsLm9yZyIsICJiZTllNGEiXQ== Received: from smtp.codeaurora.org (ec2-35-166-182-171.us-west-2.compute.amazonaws.com [35.166.182.171]) by mxa.mailgun.org with ESMTP id 5ebd50d9.7f1279b30308-smtp-out-n05; Thu, 14 May 2020 14:08:25 -0000 (UTC) Received: by smtp.codeaurora.org (Postfix, from userid 1001) id 1DDE5C44793; Thu, 14 May 2020 14:08:23 +0000 (UTC) Received: from jhugo-perf-lnx.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: jhugo) by smtp.codeaurora.org (Postfix) with ESMTPSA id F01AAC072B5; Thu, 14 May 2020 14:08:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org F01AAC072B5 Authentication-Results: aws-us-west-2-caf-mail-1.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: aws-us-west-2-caf-mail-1.web.codeaurora.org; spf=none smtp.mailfrom=jhugo@codeaurora.org From: Jeffrey Hugo To: arnd@arndb.de, gregkh@linuxfoundation.org Cc: manivannan.sadhasivam@linaro.org, bjorn.andersson@linaro.org, wufan@codeaurora.org, pratanan@codeaurora.org, linux-arm-msm@vger.kernel.org, linux-kernel@vger.kernel.org, Jeffrey Hugo Subject: [RFC PATCH 6/8] qaic: Implement PCI link status error handlers Date: Thu, 14 May 2020 08:07:44 -0600 Message-Id: <1589465266-20056-7-git-send-email-jhugo@codeaurora.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1589465266-20056-1-git-send-email-jhugo@codeaurora.org> References: <1589465266-20056-1-git-send-email-jhugo@codeaurora.org> Sender: linux-arm-msm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Whenever the link goes down, the device loses all state. Therefore, we need to be aware of when the link goes down and comes back up so that our state is kept in sync with the device. Implement the PCI error handlers to be notified of links tate changes. Signed-off-by: Jeffrey Hugo --- drivers/misc/qaic/mhi_controller.c | 16 ++++++++++++++++ drivers/misc/qaic/mhi_controller.h | 4 ++++ drivers/misc/qaic/qaic_drv.c | 29 +++++++++++++++++++++++++++++ 3 files changed, 49 insertions(+) diff --git a/drivers/misc/qaic/mhi_controller.c b/drivers/misc/qaic/mhi_controller.c index ba4808c..18fc830 100644 --- a/drivers/misc/qaic/mhi_controller.c +++ b/drivers/misc/qaic/mhi_controller.c @@ -493,3 +493,19 @@ void qaic_mhi_free_controller(struct mhi_controller *mhi_cntl, bool link_up) kfree(mhi_cntl->irq); kfree(mhi_cntl); } + +void qaic_mhi_link_down(struct mhi_controller *mhi_cntl) +{ + mhi_power_down(mhi_cntl, false); +} + +void qaic_mhi_link_up(struct mhi_controller *mhi_cntl) +{ + struct pci_dev *pci_dev = container_of(mhi_cntl->cntrl_dev, + struct pci_dev, dev); + int ret; + + ret = mhi_async_power_up(mhi_cntl); + if (ret) + pci_err(pci_dev, "mhi_async_power_up failed when link came back %d\n", ret); +} diff --git a/drivers/misc/qaic/mhi_controller.h b/drivers/misc/qaic/mhi_controller.h index c81725e..5eca586 100644 --- a/drivers/misc/qaic/mhi_controller.h +++ b/drivers/misc/qaic/mhi_controller.h @@ -11,4 +11,8 @@ struct mhi_controller *qaic_mhi_register_controller(struct pci_dev *pci_dev, int mhi_irq); void qaic_mhi_free_controller(struct mhi_controller *mhi_cntl, bool link_up); + +void qaic_mhi_link_down(struct mhi_controller *mhi_cntl); +void qaic_mhi_link_up(struct mhi_controller *mhi_cntl); + #endif /* MHICONTROLLERQAIC_H_ */ diff --git a/drivers/misc/qaic/qaic_drv.c b/drivers/misc/qaic/qaic_drv.c index 800769d..1c8eefd 100644 --- a/drivers/misc/qaic/qaic_drv.c +++ b/drivers/misc/qaic/qaic_drv.c @@ -558,6 +558,28 @@ static void qaic_pci_remove(struct pci_dev *pdev) kfree(qdev); } +static pci_ers_result_t qaic_pci_error_detected(struct pci_dev *pdev, + enum pci_channel_state error) +{ + return PCI_ERS_RESULT_NEED_RESET; +} + +static void qaic_pci_reset_prepare(struct pci_dev *pdev) +{ + struct qaic_device *qdev = pci_get_drvdata(pdev); + + qaic_dev_reset_clean_local_state(qdev); + qaic_mhi_link_down(qdev->mhi_cntl); +} + +static void qaic_pci_reset_done(struct pci_dev *pdev) +{ + struct qaic_device *qdev = pci_get_drvdata(pdev); + + qdev->in_reset = false; + qaic_mhi_link_up(qdev->mhi_cntl); +} + static const struct mhi_device_id qaic_mhi_match_table[] = { { .chan = "QAIC_CONTROL", }, {}, @@ -581,11 +603,18 @@ static const struct pci_device_id ids[] = { }; MODULE_DEVICE_TABLE(pci, ids); +static const struct pci_error_handlers qaic_pci_err_handler = { + .error_detected = qaic_pci_error_detected, + .reset_prepare = qaic_pci_reset_prepare, + .reset_done = qaic_pci_reset_done, +}; + static struct pci_driver qaic_pci_driver = { .name = QAIC_NAME, .id_table = ids, .probe = qaic_pci_probe, .remove = qaic_pci_remove, + .err_handler = &qaic_pci_err_handler, }; static int __init qaic_init(void)