From patchwork Mon Dec 11 17:36:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Karan Tilak Kumar \(kartilak\)" X-Patchwork-Id: 752953 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="h+UuXWUc" Received: from alln-iport-6.cisco.com (alln-iport-6.cisco.com [173.37.142.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80A1EF5; Mon, 11 Dec 2023 09:37:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=3417; q=dns/txt; s=iport; t=1702316230; x=1703525830; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6SK31m/xgzin/ld6/lGXf2IyXG9SZiePNHpiDRsZ/fI=; b=h+UuXWUcuBY8BLrQjBGfQga3EqFA3iPYUJP+58XBHx8qAr2lERQSkQFc HQ+L125V6pg4vB8K+ffY9O60QgCB6ksyBoMKgjQkK458a0odvfJHMyP8q ylELZyZAq8KCK8YMAh385YCW6kTR2lv+srr3M9s8qskfWFBvm+nl4OCHi U=; X-CSE-ConnectionGUID: frN6eqE3Q9aXZYLtXLYzLw== X-CSE-MsgGUID: wuAWw6OpQsaT3JlWtZytGQ== X-IronPort-AV: E=Sophos;i="6.04,268,1695686400"; d="scan'208";a="193289207" Received: from rcdn-core-1.cisco.com ([173.37.93.152]) by alln-iport-6.cisco.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Dec 2023 17:37:07 +0000 Received: from localhost.cisco.com ([10.193.101.253]) (authenticated bits=0) by rcdn-core-1.cisco.com (8.15.2/8.15.2) with ESMTPSA id 3BBHaKqu009547 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 11 Dec 2023 17:37:07 GMT From: Karan Tilak Kumar To: sebaddel@cisco.com Cc: arulponn@cisco.com, djhawar@cisco.com, gcboffa@cisco.com, mkai2@cisco.com, satishkh@cisco.com, jejb@linux.ibm.com, martin.petersen@oracle.com, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org, Karan Tilak Kumar Subject: [PATCH v6 02/13] scsi: fnic: Add and use fnic number Date: Mon, 11 Dec 2023 09:36:06 -0800 Message-Id: <20231211173617.932990-3-kartilak@cisco.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231211173617.932990-1-kartilak@cisco.com> References: <20231211173617.932990-1-kartilak@cisco.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Authenticated-User: kartilak@cisco.com X-Outbound-SMTP-Client: 10.193.101.253, [10.193.101.253] X-Outbound-Node: rcdn-core-1.cisco.com Add fnic_num in fnic.h to identify fnic in a multi-fnic environment. Increment and set the fnic number during driver load in fnic_probe. Replace the host number with fnic number in debugfs. Reviewed-by: Sesidhar Baddela Reviewed-by: Arulprabhu Ponnusamy Signed-off-by: Karan Tilak Kumar --- Changes between v4 and v5: Incorporate review comments from Martin: Modify patch commits to include a "---" separator. Changes between v3 and v4: Incorporate review comments from Martin and Hannes: Undo the change to replace host number with fnic number in debugfs since kernel stack uses host number. Use ida_alloc to allocate ID for fnic number. --- drivers/scsi/fnic/fnic.h | 1 + drivers/scsi/fnic/fnic_main.c | 18 ++++++++++++++++-- 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/drivers/scsi/fnic/fnic.h b/drivers/scsi/fnic/fnic.h index 22cef283b2b9..721a31a03628 100644 --- a/drivers/scsi/fnic/fnic.h +++ b/drivers/scsi/fnic/fnic.h @@ -216,6 +216,7 @@ struct fnic_event { /* Per-instance private data structure */ struct fnic { + int fnic_num; struct fc_lport *lport; struct fcoe_ctlr ctlr; /* FIP FCoE controller structure */ struct vnic_dev_bar bar0; diff --git a/drivers/scsi/fnic/fnic_main.c b/drivers/scsi/fnic/fnic_main.c index f27f9319e0b2..3044723e125e 100644 --- a/drivers/scsi/fnic/fnic_main.c +++ b/drivers/scsi/fnic/fnic_main.c @@ -39,6 +39,7 @@ static struct kmem_cache *fnic_sgl_cache[FNIC_SGL_NUM_CACHES]; static struct kmem_cache *fnic_io_req_cache; static LIST_HEAD(fnic_list); static DEFINE_SPINLOCK(fnic_list_lock); +static DEFINE_IDA(fnic_ida); /* Supported devices by fnic module */ static struct pci_device_id fnic_id_table[] = { @@ -583,7 +584,8 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) struct fc_lport *lp; struct fnic *fnic; mempool_t *pool; - int err; + int err = 0; + int fnic_id = 0; int i; unsigned long flags; @@ -597,8 +599,16 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) err = -ENOMEM; goto err_out; } + host = lp->host; fnic = lport_priv(lp); + + fnic_id = ida_alloc(&fnic_ida, GFP_KERNEL); + if (fnic_id < 0) { + pr_err("Unable to alloc fnic ID\n"); + err = fnic_id; + goto err_out_ida_alloc; + } fnic->lport = lp; fnic->ctlr.lp = lp; @@ -608,7 +618,7 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) host->host_no); host->transportt = fnic_fc_transport; - + fnic->fnic_num = fnic_id; fnic_stats_debugfs_init(fnic); /* Setup PCI resources */ @@ -951,6 +961,8 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) pci_disable_device(pdev); err_out_free_hba: fnic_stats_debugfs_remove(fnic); + ida_free(&fnic_ida, fnic->fnic_num); +err_out_ida_alloc: scsi_host_put(lp->host); err_out: return err; @@ -1031,6 +1043,7 @@ static void fnic_remove(struct pci_dev *pdev) fnic_iounmap(fnic); pci_release_regions(pdev); pci_disable_device(pdev); + ida_free(&fnic_ida, fnic->fnic_num); scsi_host_put(lp->host); } @@ -1168,6 +1181,7 @@ static void __exit fnic_cleanup_module(void) fnic_trace_free(); fnic_fc_trace_free(); fnic_debugfs_terminate(); + ida_destroy(&fnic_ida); } module_init(fnic_init_module); From patchwork Mon Dec 11 17:36:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Karan Tilak Kumar \(kartilak\)" X-Patchwork-Id: 752952 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="PD4uwNv5" Received: from rcdn-iport-7.cisco.com (rcdn-iport-7.cisco.com [173.37.86.78]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F3D0119; Mon, 11 Dec 2023 09:37:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=6629; q=dns/txt; s=iport; t=1702316255; x=1703525855; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=WYa63LUfBeKl9vRgk3uqf+cKBjEsHT/9sm32R3sDq9g=; b=PD4uwNv5ZD1+MiLwU3+tAw6AElUe+9DBIbTRFgexJq5SgSpcbOFrVZTo wL7229D3TF6KbhNdcVVJOzxtzvmu6gNRuOVQ0L5pg04G9Y0aKKGpKF2D0 ukig5gUYq5cjvHdEb91WssAmI4gJ8omEtxkyxISpVqIU/NlExmuaqcV2f c=; X-CSE-ConnectionGUID: zUJ+R8lTTkG9XYJW+9EfvA== X-CSE-MsgGUID: OyfFuzmOTAyY24yOkLcP5w== X-IronPort-AV: E=Sophos;i="6.04,268,1695686400"; d="scan'208";a="158062904" Received: from rcdn-core-1.cisco.com ([173.37.93.152]) by rcdn-iport-7.cisco.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Dec 2023 17:37:34 +0000 Received: from localhost.cisco.com ([10.193.101.253]) (authenticated bits=0) by rcdn-core-1.cisco.com (8.15.2/8.15.2) with ESMTPSA id 3BBHaKqw009547 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 11 Dec 2023 17:37:33 GMT From: Karan Tilak Kumar To: sebaddel@cisco.com Cc: arulponn@cisco.com, djhawar@cisco.com, gcboffa@cisco.com, mkai2@cisco.com, satishkh@cisco.com, jejb@linux.ibm.com, martin.petersen@oracle.com, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org, Karan Tilak Kumar , Hannes Reinecke Subject: [PATCH v6 04/13] scsi: fnic: Rename wq_copy to hw_copy_wq Date: Mon, 11 Dec 2023 09:36:08 -0800 Message-Id: <20231211173617.932990-5-kartilak@cisco.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231211173617.932990-1-kartilak@cisco.com> References: <20231211173617.932990-1-kartilak@cisco.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Authenticated-User: kartilak@cisco.com X-Outbound-SMTP-Client: 10.193.101.253, [10.193.101.253] X-Outbound-Node: rcdn-core-1.cisco.com Rename wq_copy to hw_copy_wq to accurately describe the copy workqueue. This will also help distinguish this data structure from software data structures that can be introduced. Reviewed-by: Sesidhar Baddela Reviewed-by: Arulprabhu Ponnusamy Reviewed-by: Hannes Reinecke Signed-off-by: Karan Tilak Kumar --- drivers/scsi/fnic/fnic.h | 2 +- drivers/scsi/fnic/fnic_isr.c | 2 +- drivers/scsi/fnic/fnic_main.c | 8 ++++---- drivers/scsi/fnic/fnic_res.c | 6 +++--- drivers/scsi/fnic/fnic_scsi.c | 12 ++++++------ 5 files changed, 15 insertions(+), 15 deletions(-) diff --git a/drivers/scsi/fnic/fnic.h b/drivers/scsi/fnic/fnic.h index 4dde8afbb69b..2e68209181a8 100644 --- a/drivers/scsi/fnic/fnic.h +++ b/drivers/scsi/fnic/fnic.h @@ -307,7 +307,7 @@ struct fnic { /*** FIP related data members -- end ***/ /* copy work queue cache line section */ - ____cacheline_aligned struct vnic_wq_copy wq_copy[FNIC_WQ_COPY_MAX]; + ____cacheline_aligned struct vnic_wq_copy hw_copy_wq[FNIC_WQ_COPY_MAX]; /* completion queue cache line section */ ____cacheline_aligned struct vnic_cq cq[FNIC_CQ_MAX]; diff --git a/drivers/scsi/fnic/fnic_isr.c b/drivers/scsi/fnic/fnic_isr.c index 8896758fed8c..dff9689023e4 100644 --- a/drivers/scsi/fnic/fnic_isr.c +++ b/drivers/scsi/fnic/fnic_isr.c @@ -224,7 +224,7 @@ int fnic_set_intr_mode(struct fnic *fnic) { unsigned int n = ARRAY_SIZE(fnic->rq); unsigned int m = ARRAY_SIZE(fnic->wq); - unsigned int o = ARRAY_SIZE(fnic->wq_copy); + unsigned int o = ARRAY_SIZE(fnic->hw_copy_wq); /* * Set interrupt mode (INTx, MSI, MSI-X) depending diff --git a/drivers/scsi/fnic/fnic_main.c b/drivers/scsi/fnic/fnic_main.c index cc20c12012e3..dacefbeb44f0 100644 --- a/drivers/scsi/fnic/fnic_main.c +++ b/drivers/scsi/fnic/fnic_main.c @@ -355,7 +355,7 @@ void fnic_log_q_error(struct fnic *fnic) } for (i = 0; i < fnic->wq_copy_count; i++) { - error_status = ioread32(&fnic->wq_copy[i].ctrl->error_status); + error_status = ioread32(&fnic->hw_copy_wq[i].ctrl->error_status); if (error_status) shost_printk(KERN_ERR, fnic->lport->host, "CWQ[%d] error_status" @@ -492,7 +492,7 @@ static int fnic_cleanup(struct fnic *fnic) return err; } for (i = 0; i < fnic->wq_copy_count; i++) { - err = vnic_wq_copy_disable(&fnic->wq_copy[i]); + err = vnic_wq_copy_disable(&fnic->hw_copy_wq[i]); if (err) return err; } @@ -508,7 +508,7 @@ static int fnic_cleanup(struct fnic *fnic) for (i = 0; i < fnic->rq_count; i++) vnic_rq_clean(&fnic->rq[i], fnic_free_rq_buf); for (i = 0; i < fnic->wq_copy_count; i++) - vnic_wq_copy_clean(&fnic->wq_copy[i], + vnic_wq_copy_clean(&fnic->hw_copy_wq[i], fnic_wq_copy_cleanup_handler); for (i = 0; i < fnic->cq_count; i++) @@ -909,7 +909,7 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) for (i = 0; i < fnic->raw_wq_count; i++) vnic_wq_enable(&fnic->wq[i]); for (i = 0; i < fnic->wq_copy_count; i++) - vnic_wq_copy_enable(&fnic->wq_copy[i]); + vnic_wq_copy_enable(&fnic->hw_copy_wq[i]); fc_fabric_login(lp); diff --git a/drivers/scsi/fnic/fnic_res.c b/drivers/scsi/fnic/fnic_res.c index a1c9cfcace7f..109316cc4ad9 100644 --- a/drivers/scsi/fnic/fnic_res.c +++ b/drivers/scsi/fnic/fnic_res.c @@ -203,7 +203,7 @@ void fnic_free_vnic_resources(struct fnic *fnic) vnic_wq_free(&fnic->wq[i]); for (i = 0; i < fnic->wq_copy_count; i++) - vnic_wq_copy_free(&fnic->wq_copy[i]); + vnic_wq_copy_free(&fnic->hw_copy_wq[i]); for (i = 0; i < fnic->rq_count; i++) vnic_rq_free(&fnic->rq[i]); @@ -250,7 +250,7 @@ int fnic_alloc_vnic_resources(struct fnic *fnic) /* Allocate Copy WQs used for SCSI IOs */ for (i = 0; i < fnic->wq_copy_count; i++) { - err = vnic_wq_copy_alloc(fnic->vdev, &fnic->wq_copy[i], + err = vnic_wq_copy_alloc(fnic->vdev, &fnic->hw_copy_wq[i], (fnic->raw_wq_count + i), fnic->config.wq_copy_desc_count, sizeof(struct fcpio_host_req)); @@ -357,7 +357,7 @@ int fnic_alloc_vnic_resources(struct fnic *fnic) } for (i = 0; i < fnic->wq_copy_count; i++) { - vnic_wq_copy_init(&fnic->wq_copy[i], + vnic_wq_copy_init(&fnic->hw_copy_wq[i], 0 /* cq_index 0 - always */, error_interrupt_enable, error_interrupt_offset); diff --git a/drivers/scsi/fnic/fnic_scsi.c b/drivers/scsi/fnic/fnic_scsi.c index 416d81954819..b7dc304446f8 100644 --- a/drivers/scsi/fnic/fnic_scsi.c +++ b/drivers/scsi/fnic/fnic_scsi.c @@ -193,7 +193,7 @@ __fnic_set_state_flags(struct fnic *fnic, unsigned long st_flags, */ int fnic_fw_reset_handler(struct fnic *fnic) { - struct vnic_wq_copy *wq = &fnic->wq_copy[0]; + struct vnic_wq_copy *wq = &fnic->hw_copy_wq[0]; int ret = 0; unsigned long flags; @@ -246,7 +246,7 @@ int fnic_fw_reset_handler(struct fnic *fnic) */ int fnic_flogi_reg_handler(struct fnic *fnic, u32 fc_id) { - struct vnic_wq_copy *wq = &fnic->wq_copy[0]; + struct vnic_wq_copy *wq = &fnic->hw_copy_wq[0]; enum fcpio_flogi_reg_format_type format; struct fc_lport *lp = fnic->lport; u8 gw_mac[ETH_ALEN]; @@ -551,7 +551,7 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc) fnic_priv(sc)->flags |= FNIC_IO_INITIALIZED; /* create copy wq desc and enqueue it */ - wq = &fnic->wq_copy[0]; + wq = &fnic->hw_copy_wq[0]; ret = fnic_queue_wq_copy_desc(fnic, wq, io_req, sc, sg_count); if (ret) { /* @@ -782,7 +782,7 @@ static inline void fnic_fcpio_ack_handler(struct fnic *fnic, u64 *ox_id_tag = (u64 *)(void *)desc; /* mark the ack state */ - wq = &fnic->wq_copy[cq_index - fnic->raw_wq_count - fnic->rq_count]; + wq = &fnic->hw_copy_wq[cq_index - fnic->raw_wq_count - fnic->rq_count]; spin_lock_irqsave(&fnic->wq_copy_lock[0], flags); fnic->fnic_stats.misc_stats.last_ack_time = jiffies; @@ -1502,7 +1502,7 @@ static inline int fnic_queue_abort_io_req(struct fnic *fnic, int tag, u32 task_req, u8 *fc_lun, struct fnic_io_req *io_req) { - struct vnic_wq_copy *wq = &fnic->wq_copy[0]; + struct vnic_wq_copy *wq = &fnic->hw_copy_wq[0]; struct Scsi_Host *host = fnic->lport->host; struct misc_stats *misc_stats = &fnic->fnic_stats.misc_stats; unsigned long flags; @@ -1948,7 +1948,7 @@ static inline int fnic_queue_dr_io_req(struct fnic *fnic, struct scsi_cmnd *sc, struct fnic_io_req *io_req) { - struct vnic_wq_copy *wq = &fnic->wq_copy[0]; + struct vnic_wq_copy *wq = &fnic->hw_copy_wq[0]; struct Scsi_Host *host = fnic->lport->host; struct misc_stats *misc_stats = &fnic->fnic_stats.misc_stats; struct scsi_lun fc_lun; From patchwork Mon Dec 11 17:36:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Karan Tilak Kumar \(kartilak\)" X-Patchwork-Id: 752951 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="b7rkc9iV" Received: from alln-iport-8.cisco.com (alln-iport-8.cisco.com [173.37.142.95]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 401B29B; Mon, 11 Dec 2023 09:38:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=3263; q=dns/txt; s=iport; t=1702316334; x=1703525934; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Q0xWd2ThYF+V4l8oL+rdzstG0URCVrdCbQqXz/xjPCs=; b=b7rkc9iVzhnrYw0Zp3EHf79x8GjmI5cUKYSlviNTUXsLrtmNGhxua/5q fhVExWTzJDOePvOg7utIxxZzsWRl5m1A40VWBPxdrB8FIoPsjzdgwBGP3 ZKV0wdIkJwiLEgC3vYtSoznE/BOWI+s4vt6Y2IzmQdo/MFiwbhm2HQI3b k=; X-CSE-ConnectionGUID: meCwM58qSiuwxEoMoyUcOg== X-CSE-MsgGUID: RDp+kURuSIOMJ2ZSI8CWFQ== X-IronPort-AV: E=Sophos;i="6.04,268,1695686400"; d="scan'208";a="193670948" Received: from rcdn-core-1.cisco.com ([173.37.93.152]) by alln-iport-8.cisco.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Dec 2023 17:38:53 +0000 Received: from localhost.cisco.com ([10.193.101.253]) (authenticated bits=0) by rcdn-core-1.cisco.com (8.15.2/8.15.2) with ESMTPSA id 3BBHaKr0009547 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 11 Dec 2023 17:38:52 GMT From: Karan Tilak Kumar To: sebaddel@cisco.com Cc: arulponn@cisco.com, djhawar@cisco.com, gcboffa@cisco.com, mkai2@cisco.com, satishkh@cisco.com, jejb@linux.ibm.com, martin.petersen@oracle.com, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org, Karan Tilak Kumar Subject: [PATCH v6 06/13] scsi: fnic: Refactor and redefine fnic.h for multiqueue Date: Mon, 11 Dec 2023 09:36:10 -0800 Message-Id: <20231211173617.932990-7-kartilak@cisco.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231211173617.932990-1-kartilak@cisco.com> References: <20231211173617.932990-1-kartilak@cisco.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Authenticated-User: kartilak@cisco.com X-Outbound-SMTP-Client: 10.193.101.253, [10.193.101.253] X-Outbound-Node: rcdn-core-1.cisco.com Refactor and re-define values in fnic.h to implement multiqueue(MQ) functionality. VIC firmware allows fnic to create up to 64 copy workqueues. Update the copy workqueue max to 64. Modify the interrupt index to be in sync with the firmware to support MQ. Add irq number to the MSIX entry. Define a software workqueue table to track the status of io_reqs. Define a base for the copy workqueue. Reviewed-by: Sesidhar Baddela Reviewed-by: Arulprabhu Ponnusamy Signed-off-by: Karan Tilak Kumar --- Changes between v4 and v5: Incorporate review comments from Martin: Modify patch commits to include a "---" separator. Changes between v2 and v3: Replace cpy_wq_base with copy_wq_base. --- drivers/scsi/fnic/fnic.h | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/drivers/scsi/fnic/fnic.h b/drivers/scsi/fnic/fnic.h index 2e68209181a8..47173f1eec1e 100644 --- a/drivers/scsi/fnic/fnic.h +++ b/drivers/scsi/fnic/fnic.h @@ -167,12 +167,21 @@ do { \ #define FNIC_MAIN_NOTE(kern_level, host, fmt, args...) \ shost_printk(kern_level, host, fmt, ##args) +#define FNIC_WQ_COPY_MAX 64 +#define FNIC_WQ_MAX 1 +#define FNIC_RQ_MAX 1 +#define FNIC_CQ_MAX (FNIC_WQ_COPY_MAX + FNIC_WQ_MAX + FNIC_RQ_MAX) +#define FNIC_DFLT_IO_COMPLETIONS 256 + +#define FNIC_MQ_CQ_INDEX 2 + extern const char *fnic_state_str[]; enum fnic_intx_intr_index { FNIC_INTX_WQ_RQ_COPYWQ, - FNIC_INTX_ERR, + FNIC_INTX_DUMMY, FNIC_INTX_NOTIFY, + FNIC_INTX_ERR, FNIC_INTX_INTR_MAX, }; @@ -180,7 +189,7 @@ enum fnic_msix_intr_index { FNIC_MSIX_RQ, FNIC_MSIX_WQ, FNIC_MSIX_WQ_COPY, - FNIC_MSIX_ERR_NOTIFY, + FNIC_MSIX_ERR_NOTIFY = FNIC_MSIX_WQ_COPY + FNIC_WQ_COPY_MAX, FNIC_MSIX_INTR_MAX, }; @@ -189,6 +198,7 @@ struct fnic_msix_entry { char devname[IFNAMSIZ + 11]; irqreturn_t (*isr)(int, void *); void *devid; + int irq_num; }; enum fnic_state { @@ -198,12 +208,6 @@ enum fnic_state { FNIC_IN_ETH_TRANS_FC_MODE, }; -#define FNIC_WQ_COPY_MAX 1 -#define FNIC_WQ_MAX 1 -#define FNIC_RQ_MAX 1 -#define FNIC_CQ_MAX (FNIC_WQ_COPY_MAX + FNIC_WQ_MAX + FNIC_RQ_MAX) -#define FNIC_DFLT_IO_COMPLETIONS 256 - struct mempool; enum fnic_evt { @@ -218,6 +222,13 @@ struct fnic_event { enum fnic_evt event; }; +struct fnic_cpy_wq { + unsigned long hw_lock_flags; + u16 active_ioreq_count; + u16 ioreq_table_size; + ____cacheline_aligned struct fnic_io_req **io_req_table; +}; + /* Per-instance private data structure */ struct fnic { int fnic_num; @@ -289,6 +300,7 @@ struct fnic { mempool_t *io_sgl_pool[FNIC_SGL_NUM_CACHES]; spinlock_t io_req_lock[FNIC_IO_LOCKS]; /* locks for scsi cmnds */ + unsigned int copy_wq_base; struct work_struct link_work; struct work_struct frame_work; struct sk_buff_head frame_queue; @@ -308,6 +320,8 @@ struct fnic { /* copy work queue cache line section */ ____cacheline_aligned struct vnic_wq_copy hw_copy_wq[FNIC_WQ_COPY_MAX]; + ____cacheline_aligned struct fnic_cpy_wq sw_copy_wq[FNIC_WQ_COPY_MAX]; + /* completion queue cache line section */ ____cacheline_aligned struct vnic_cq cq[FNIC_CQ_MAX]; From patchwork Mon Dec 11 17:36:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Karan Tilak Kumar \(kartilak\)" X-Patchwork-Id: 752950 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="ixppTE1r" Received: from rcdn-iport-1.cisco.com (rcdn-iport-1.cisco.com [173.37.86.72]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4096C7; Mon, 11 Dec 2023 09:40:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=2336; q=dns/txt; s=iport; t=1702316416; x=1703526016; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ov+22JfMWX71JUrjb9pbSfvsIvqm6DfNrkyUATv3g3I=; b=ixppTE1r9T4+kzEHUPBMiRnirVJ0vsIBMQLxzr3wVuS0NRUX61kz7n33 8bgSoZD+FLs8t0ksBVwhRZ5n2HNWN/d2PloY85iEDoDWbekO9eNr6H8xW kGOGBz3owqrY4Un9rPZlhi03G3I3grfEVR5C9H8kVarrCRME9YMtxgNb/ I=; X-CSE-ConnectionGUID: KdjS/1psTCaTV/hgYiUwog== X-CSE-MsgGUID: j6MxhxOHQCOm8xUeSkm/vw== X-IronPort-AV: E=Sophos;i="6.04,268,1695686400"; d="scan'208";a="158328742" Received: from rcdn-core-1.cisco.com ([173.37.93.152]) by rcdn-iport-1.cisco.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Dec 2023 17:40:14 +0000 Received: from localhost.cisco.com ([10.193.101.253]) (authenticated bits=0) by rcdn-core-1.cisco.com (8.15.2/8.15.2) with ESMTPSA id 3BBHaKr2009547 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 11 Dec 2023 17:40:14 GMT From: Karan Tilak Kumar To: sebaddel@cisco.com Cc: arulponn@cisco.com, djhawar@cisco.com, gcboffa@cisco.com, mkai2@cisco.com, satishkh@cisco.com, jejb@linux.ibm.com, martin.petersen@oracle.com, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org, Karan Tilak Kumar , Hannes Reinecke Subject: [PATCH v6 08/13] scsi: fnic: Define stats to track multiqueue (MQ) IOs Date: Mon, 11 Dec 2023 09:36:12 -0800 Message-Id: <20231211173617.932990-9-kartilak@cisco.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231211173617.932990-1-kartilak@cisco.com> References: <20231211173617.932990-1-kartilak@cisco.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Authenticated-User: kartilak@cisco.com X-Outbound-SMTP-Client: 10.193.101.253, [10.193.101.253] X-Outbound-Node: rcdn-core-1.cisco.com Define an array to track IOs for the different queues, print the IO stats in fnic get stats data. Reviewed-by: Sesidhar Baddela Reviewed-by: Arulprabhu Ponnusamy Reviewed-by: Hannes Reinecke Signed-off-by: Karan Tilak Kumar --- drivers/scsi/fnic/fnic_stats.h | 2 ++ drivers/scsi/fnic/fnic_trace.c | 11 +++++++++++ 2 files changed, 13 insertions(+) diff --git a/drivers/scsi/fnic/fnic_stats.h b/drivers/scsi/fnic/fnic_stats.h index 07d1556e3c32..9d7f98c452dd 100644 --- a/drivers/scsi/fnic/fnic_stats.h +++ b/drivers/scsi/fnic/fnic_stats.h @@ -2,6 +2,7 @@ /* Copyright 2013 Cisco Systems, Inc. All rights reserved. */ #ifndef _FNIC_STATS_H_ #define _FNIC_STATS_H_ +#define FNIC_MQ_MAX_QUEUES 64 struct stats_timestamps { struct timespec64 last_reset_time; @@ -26,6 +27,7 @@ struct io_path_stats { atomic64_t io_btw_10000_to_30000_msec; atomic64_t io_greater_than_30000_msec; atomic64_t current_max_io_time; + atomic64_t ios[FNIC_MQ_MAX_QUEUES]; }; struct abort_stats { diff --git a/drivers/scsi/fnic/fnic_trace.c b/drivers/scsi/fnic/fnic_trace.c index be0d7c57b242..aaa4ea02fb7c 100644 --- a/drivers/scsi/fnic/fnic_trace.c +++ b/drivers/scsi/fnic/fnic_trace.c @@ -204,6 +204,7 @@ int fnic_get_stats_data(struct stats_debug_info *debug, int len = 0; int buf_size = debug->buf_size; struct timespec64 val1, val2; + int i = 0; ktime_get_real_ts64(&val1); len = scnprintf(debug->debug_buffer + len, buf_size - len, @@ -266,6 +267,16 @@ int fnic_get_stats_data(struct stats_debug_info *debug, (u64)atomic64_read(&stats->io_stats.io_btw_10000_to_30000_msec), (u64)atomic64_read(&stats->io_stats.io_greater_than_30000_msec)); + len += scnprintf(debug->debug_buffer + len, buf_size - len, + "------------------------------------------\n" + "\t\tIO Queues and cumulative IOs\n" + "------------------------------------------\n"); + + for (i = 0; i < FNIC_MQ_MAX_QUEUES; i++) { + len += scnprintf(debug->debug_buffer + len, buf_size - len, + "Q:%d -> %lld\n", i, (u64)atomic64_read(&stats->io_stats.ios[i])); + } + len += scnprintf(debug->debug_buffer + len, buf_size - len, "\nCurrent Max IO time : %lld\n", (u64)atomic64_read(&stats->io_stats.current_max_io_time)); From patchwork Mon Dec 11 17:36:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Karan Tilak Kumar \(kartilak\)" X-Patchwork-Id: 752949 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="MB01OxqD" Received: from alln-iport-6.cisco.com (alln-iport-6.cisco.com [173.37.142.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91BC3D0; Mon, 11 Dec 2023 09:41:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=10128; q=dns/txt; s=iport; t=1702316495; x=1703526095; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=C1Op3uVfWwQ0AGy+vXJw9fLd3TzW6SmqLiQ2mrLmgSg=; b=MB01OxqDoaFPGpsXxpG5PQ8FcZ4XmOQ7DE+DA+bFxBVfH7VPA+UGJ/f9 1PlIe/FtmtakVvqUNh4HBbn0PWn4FtN0QP5bVlN7Hp1667wU5Qzg8786j zWegG5KdKXlzhKi2OO1wyFJS2LrEDjmUM2TjolE1ACG+NF+HQ5oXL+MAv Y=; X-CSE-ConnectionGUID: NpBgdNW7TWe7KLfy3lVQcg== X-CSE-MsgGUID: 7Hl5l11jR/Wf8tnQaeMaCw== X-IronPort-AV: E=Sophos;i="6.04,268,1695686400"; d="scan'208";a="193291613" Received: from rcdn-core-1.cisco.com ([173.37.93.152]) by alln-iport-6.cisco.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Dec 2023 17:41:35 +0000 Received: from localhost.cisco.com ([10.193.101.253]) (authenticated bits=0) by rcdn-core-1.cisco.com (8.15.2/8.15.2) with ESMTPSA id 3BBHaKr4009547 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 11 Dec 2023 17:41:34 GMT From: Karan Tilak Kumar To: sebaddel@cisco.com Cc: arulponn@cisco.com, djhawar@cisco.com, gcboffa@cisco.com, mkai2@cisco.com, satishkh@cisco.com, jejb@linux.ibm.com, martin.petersen@oracle.com, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org, Karan Tilak Kumar , Hannes Reinecke , John Garry Subject: [PATCH v6 10/13] scsi: fnic: Add support for multiqueue (MQ) in fnic_main.c Date: Mon, 11 Dec 2023 09:36:14 -0800 Message-Id: <20231211173617.932990-11-kartilak@cisco.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231211173617.932990-1-kartilak@cisco.com> References: <20231211173617.932990-1-kartilak@cisco.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Authenticated-User: kartilak@cisco.com X-Outbound-SMTP-Client: 10.193.101.253, [10.193.101.253] X-Outbound-Node: rcdn-core-1.cisco.com Set map_queues in the fnic_host_template to fnic_mq_map_queues_cpus. Define fnic_mq_map_queues_cpus to set cpu assignment to fnic queues. Refactor code in fnic_probe to enable vnic queues before scsi_add_host. Modify notify set to the correct index. Reviewed-by: Sesidhar Baddela Reviewed-by: Arulprabhu Ponnusamy Reviewed-by: Hannes Reinecke Reviewed-by: John Garry Signed-off-by: Karan Tilak Kumar --- Changes between v4 and v5: Incorporate review comments from Martin: Modify patch commits to include a "---" separator. Changes between v2 and v3: Incorporate review comment from Hannes: Replace cpy_wq_base with copy_wq_base. Incorporate review comment from John Garry: Replace code in fnic_mq_map_queues_cpus with blk_mq_pci_map_queues. Replace shost_printk logs with FNIC_MAIN_DBG. --- drivers/scsi/fnic/fnic.h | 4 +- drivers/scsi/fnic/fnic_main.c | 107 ++++++++++++++++++++++++---------- 2 files changed, 77 insertions(+), 34 deletions(-) diff --git a/drivers/scsi/fnic/fnic.h b/drivers/scsi/fnic/fnic.h index 8b3e4fe1ce92..879e0f5b9b0a 100644 --- a/drivers/scsi/fnic/fnic.h +++ b/drivers/scsi/fnic/fnic.h @@ -109,7 +109,7 @@ static inline u64 fnic_flags_and_state(struct scsi_cmnd *cmd) #define FNIC_ABT_TERM_DELAY_TIMEOUT 500 /* mSec */ #define FNIC_MAX_FCP_TARGET 256 - +#define FNIC_PCI_OFFSET 2 /** * state_flags to identify host state along along with fnic's state **/ @@ -386,7 +386,7 @@ void fnic_wq_copy_cleanup_handler(struct vnic_wq_copy *wq, int fnic_fw_reset_handler(struct fnic *fnic); void fnic_terminate_rport_io(struct fc_rport *); const char *fnic_state_to_str(unsigned int state); - +void fnic_mq_map_queues_cpus(struct Scsi_Host *host); void fnic_log_q_error(struct fnic *fnic); void fnic_handle_link_event(struct fnic *fnic); diff --git a/drivers/scsi/fnic/fnic_main.c b/drivers/scsi/fnic/fnic_main.c index 5726c0fb6f0f..7fee8a024edb 100644 --- a/drivers/scsi/fnic/fnic_main.c +++ b/drivers/scsi/fnic/fnic_main.c @@ -12,9 +12,11 @@ #include #include #include +#include #include #include #include +#include #include #include #include @@ -114,6 +116,7 @@ static const struct scsi_host_template fnic_host_template = { .shost_groups = fnic_host_groups, .track_queue_depth = 1, .cmd_size = sizeof(struct fnic_cmd_priv), + .map_queues = fnic_mq_map_queues_cpus, }; static void @@ -390,7 +393,7 @@ static int fnic_notify_set(struct fnic *fnic) err = vnic_dev_notify_set(fnic->vdev, -1); break; case VNIC_DEV_INTR_MODE_MSIX: - err = vnic_dev_notify_set(fnic->vdev, FNIC_MSIX_ERR_NOTIFY); + err = vnic_dev_notify_set(fnic->vdev, fnic->wq_copy_count + fnic->copy_wq_base); break; default: shost_printk(KERN_ERR, fnic->lport->host, @@ -563,11 +566,6 @@ static int fnic_scsi_drv_init(struct fnic *fnic) host->max_cmd_len = FCOE_MAX_CMD_LEN; host->nr_hw_queues = fnic->wq_copy_count; - if (host->nr_hw_queues > 1) - shost_printk(KERN_ERR, host, - "fnic: blk-mq is not supported"); - - host->nr_hw_queues = fnic->wq_copy_count = 1; shost_printk(KERN_INFO, host, "fnic: can_queue: %d max_lun: %llu", @@ -580,6 +578,32 @@ static int fnic_scsi_drv_init(struct fnic *fnic) return 0; } +void fnic_mq_map_queues_cpus(struct Scsi_Host *host) +{ + struct fc_lport *lp = shost_priv(host); + struct fnic *fnic = lport_priv(lp); + struct pci_dev *l_pdev = fnic->pdev; + int intr_mode = fnic->config.intr_mode; + struct blk_mq_queue_map *qmap = &host->tag_set.map[HCTX_TYPE_DEFAULT]; + + if (intr_mode == VNIC_DEV_INTR_MODE_MSI || intr_mode == VNIC_DEV_INTR_MODE_INTX) { + FNIC_MAIN_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "intr_mode is not msix\n"); + return; + } + + FNIC_MAIN_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "qmap->nr_queues: %d\n", qmap->nr_queues); + + if (l_pdev == NULL) { + FNIC_MAIN_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "l_pdev is null\n"); + return; + } + + blk_mq_pci_map_queues(qmap, l_pdev, FNIC_PCI_OFFSET); +} + static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) { struct Scsi_Host *host; @@ -590,6 +614,7 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) int fnic_id = 0; int i; unsigned long flags; + int hwq; /* * Allocate SCSI Host and set up association between host, @@ -613,8 +638,8 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) } fnic->lport = lp; fnic->ctlr.lp = lp; - fnic->link_events = 0; + fnic->pdev = pdev; snprintf(fnic->name, sizeof(fnic->name) - 1, "%s%d", DRV_NAME, host->host_no); @@ -623,11 +648,6 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) fnic->fnic_num = fnic_id; fnic_stats_debugfs_init(fnic); - /* Setup PCI resources */ - pci_set_drvdata(pdev, fnic); - - fnic->pdev = pdev; - err = pci_enable_device(pdev); if (err) { shost_printk(KERN_ERR, fnic->lport->host, @@ -729,7 +749,8 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_out_dev_close; } - fnic_scsi_drv_init(fnic); + /* Setup PCI resources */ + pci_set_drvdata(pdev, fnic); fnic_get_res_counts(fnic); @@ -749,6 +770,16 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_out_clear_intr; } + fnic_scsi_drv_init(fnic); + + for (hwq = 0; hwq < fnic->wq_copy_count; hwq++) { + fnic->sw_copy_wq[hwq].ioreq_table_size = fnic->fnic_max_tag_id; + fnic->sw_copy_wq[hwq].io_req_table = + kzalloc((fnic->sw_copy_wq[hwq].ioreq_table_size + 1) * + sizeof(struct fnic_io_req *), GFP_KERNEL); + } + shost_printk(KERN_INFO, fnic->lport->host, "fnic copy wqs: %d, Q0 ioreq table size: %d\n", + fnic->wq_copy_count, fnic->sw_copy_wq[0].ioreq_table_size); /* initialize all fnic locks */ spin_lock_init(&fnic->fnic_lock); @@ -835,16 +866,32 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) /* allocate RQ buffers and post them to RQ*/ for (i = 0; i < fnic->rq_count; i++) { - vnic_rq_enable(&fnic->rq[i]); err = vnic_rq_fill(&fnic->rq[i], fnic_alloc_rq_frame); if (err) { shost_printk(KERN_ERR, fnic->lport->host, "fnic_alloc_rq_frame can't alloc " "frame\n"); - goto err_out_free_rq_buf; + goto err_out_rq_buf; } } + /* Enable all queues */ + for (i = 0; i < fnic->raw_wq_count; i++) + vnic_wq_enable(&fnic->wq[i]); + for (i = 0; i < fnic->rq_count; i++) { + if (!ioread32(&fnic->rq[i].ctrl->enable)) + vnic_rq_enable(&fnic->rq[i]); + } + for (i = 0; i < fnic->wq_copy_count; i++) + vnic_wq_copy_enable(&fnic->hw_copy_wq[i]); + + err = fnic_request_intr(fnic); + if (err) { + shost_printk(KERN_ERR, fnic->lport->host, + "Unable to request irq.\n"); + goto err_out_request_intr; + } + /* * Initialization done with PCI system, hardware, firmware. * Add host to SCSI @@ -853,9 +900,10 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) if (err) { shost_printk(KERN_ERR, fnic->lport->host, "fnic: scsi_add_host failed...exiting\n"); - goto err_out_free_rq_buf; + goto err_out_scsi_add_host; } + /* Start local port initiatialization */ lp->link_up = 0; @@ -879,7 +927,7 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) if (!fc_exch_mgr_alloc(lp, FC_CLASS_3, FCPIO_HOST_EXCH_RANGE_START, FCPIO_HOST_EXCH_RANGE_END, NULL)) { err = -ENOMEM; - goto err_out_remove_scsi_host; + goto err_out_fc_exch_mgr_alloc; } fc_lport_init_stats(lp); @@ -907,21 +955,8 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) skb_queue_head_init(&fnic->frame_queue); skb_queue_head_init(&fnic->tx_queue); - /* Enable all queues */ - for (i = 0; i < fnic->raw_wq_count; i++) - vnic_wq_enable(&fnic->wq[i]); - for (i = 0; i < fnic->wq_copy_count; i++) - vnic_wq_copy_enable(&fnic->hw_copy_wq[i]); - fc_fabric_login(lp); - err = fnic_request_intr(fnic); - if (err) { - shost_printk(KERN_ERR, fnic->lport->host, - "Unable to request irq.\n"); - goto err_out_free_exch_mgr; - } - vnic_dev_enable(fnic->vdev); for (i = 0; i < fnic->intr_count; i++) @@ -933,12 +968,15 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) err_out_free_exch_mgr: fc_exch_mgr_free(lp); -err_out_remove_scsi_host: +err_out_fc_exch_mgr_alloc: fc_remove_host(lp->host); scsi_remove_host(lp->host); -err_out_free_rq_buf: +err_out_scsi_add_host: + fnic_free_intr(fnic); +err_out_request_intr: for (i = 0; i < fnic->rq_count; i++) vnic_rq_clean(&fnic->rq[i], fnic_free_rq_buf); +err_out_rq_buf: vnic_dev_notify_unset(fnic->vdev); err_out_free_max_pool: mempool_destroy(fnic->io_sgl_pool[FNIC_SGL_CACHE_MAX]); @@ -947,6 +985,8 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) err_out_free_ioreq_pool: mempool_destroy(fnic->io_req_pool); err_out_free_resources: + for (hwq = 0; hwq < fnic->wq_copy_count; hwq++) + kfree(fnic->sw_copy_wq[hwq].io_req_table); fnic_free_vnic_resources(fnic); err_out_clear_intr: fnic_clear_intr_mode(fnic); @@ -975,6 +1015,7 @@ static void fnic_remove(struct pci_dev *pdev) struct fnic *fnic = pci_get_drvdata(pdev); struct fc_lport *lp = fnic->lport; unsigned long flags; + int hwq; /* * Mark state so that the workqueue thread stops forwarding @@ -1035,6 +1076,8 @@ static void fnic_remove(struct pci_dev *pdev) fc_remove_host(fnic->lport->host); scsi_remove_host(fnic->lport->host); + for (hwq = 0; hwq < fnic->wq_copy_count; hwq++) + kfree(fnic->sw_copy_wq[hwq].io_req_table); fc_exch_mgr_free(fnic->lport); vnic_dev_notify_unset(fnic->vdev); fnic_free_intr(fnic); From patchwork Mon Dec 11 17:36:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Karan Tilak Kumar \(kartilak\)" X-Patchwork-Id: 752948 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="iBuZAndH" Received: from alln-iport-3.cisco.com (alln-iport-3.cisco.com [173.37.142.90]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EDBB3D5; Mon, 11 Dec 2023 09:42:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=32138; q=dns/txt; s=iport; t=1702316564; x=1703526164; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qDw2r2rDc3t3J5rbWI5NFcp74D3afYODZh06sjgot80=; b=iBuZAndH0oc6QcHixEAsNGu8CEb0uw7oHvCmAQu+Wk2f5jO5d1l/miHJ eB59tSXZA2aMouj5u2mAHDwsQQKzpPKk1hRosK7hugTJGOHDzJptgxpWS iPNEaYQZEqsORt+PirMWOiGO9YQtnqCuseq7x4lzmSC7hFJuXDwPGms/S Y=; X-CSE-ConnectionGUID: kUi08uTmSse0cYTjJ+13dw== X-CSE-MsgGUID: kslSweEARbuSthoZ2Kvpkw== X-IronPort-AV: E=Sophos;i="6.04,268,1695686400"; d="scan'208";a="186635536" Received: from rcdn-core-1.cisco.com ([173.37.93.152]) by alln-iport-3.cisco.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Dec 2023 17:42:43 +0000 Received: from localhost.cisco.com ([10.193.101.253]) (authenticated bits=0) by rcdn-core-1.cisco.com (8.15.2/8.15.2) with ESMTPSA id 3BBHaKr6009547 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 11 Dec 2023 17:42:42 GMT From: Karan Tilak Kumar To: sebaddel@cisco.com Cc: arulponn@cisco.com, djhawar@cisco.com, gcboffa@cisco.com, mkai2@cisco.com, satishkh@cisco.com, jejb@linux.ibm.com, martin.petersen@oracle.com, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org, Karan Tilak Kumar , Hannes Reinecke Subject: [PATCH v6 12/13] scsi: fnic: Improve logs and add support for multiqueue (MQ) Date: Mon, 11 Dec 2023 09:36:16 -0800 Message-Id: <20231211173617.932990-13-kartilak@cisco.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231211173617.932990-1-kartilak@cisco.com> References: <20231211173617.932990-1-kartilak@cisco.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Authenticated-User: kartilak@cisco.com X-Outbound-SMTP-Client: 10.193.101.253, [10.193.101.253] X-Outbound-Node: rcdn-core-1.cisco.com Improve existing logs by adding fnic number, hardware queue, tag, and mqtag in the prints. Add logs with the above elements for effective debugging. Reviewed-by: Sesidhar Baddela Reviewed-by: Arulprabhu Ponnusamy Tested-by: Karan Tilak Kumar Reviewed-by: Hannes Reinecke Signed-off-by: Karan Tilak Kumar --- Changes between v4 and v5: Incorporate review comments from Martin: Modify patch commits to include a "---" separator. Changes between v2 and v3: Incorporate the following review comment from Hannes: Use fnic_num as an argument to FNIC_SCSI_DBG. Modify definition of FNIC_SCSI_DBG. Host number is still an argument since FNIC_SCSI_DBG in turn uses shost_printk. --- drivers/scsi/fnic/fnic.h | 6 +- drivers/scsi/fnic/fnic_scsi.c | 288 ++++++++++++++++++---------------- 2 files changed, 154 insertions(+), 140 deletions(-) diff --git a/drivers/scsi/fnic/fnic.h b/drivers/scsi/fnic/fnic.h index 34de8f0aacb2..28f402932b3c 100644 --- a/drivers/scsi/fnic/fnic.h +++ b/drivers/scsi/fnic/fnic.h @@ -155,9 +155,11 @@ do { \ "fnic<%d>: %s: %d: " fmt, fnic_num,\ __func__, __LINE__, ##args);) -#define FNIC_SCSI_DBG(kern_level, host, fmt, args...) \ +#define FNIC_SCSI_DBG(kern_level, host, fnic_num, fmt, args...) \ FNIC_CHECK_LOGGING(FNIC_SCSI_LOGGING, \ - shost_printk(kern_level, host, fmt, ##args);) + shost_printk(kern_level, host, \ + "fnic<%d>: %s: %d: " fmt, fnic_num,\ + __func__, __LINE__, ##args);) #define FNIC_ISR_DBG(kern_level, host, fnic_num, fmt, args...) \ FNIC_CHECK_LOGGING(FNIC_ISR_LOGGING, \ diff --git a/drivers/scsi/fnic/fnic_scsi.c b/drivers/scsi/fnic/fnic_scsi.c index 42807e89859c..4d6db4509e75 100644 --- a/drivers/scsi/fnic/fnic_scsi.c +++ b/drivers/scsi/fnic/fnic_scsi.c @@ -211,12 +211,12 @@ int fnic_fw_reset_handler(struct fnic *fnic) if (!ret) { atomic64_inc(&fnic->fnic_stats.reset_stats.fw_resets); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, - "Issued fw reset\n"); + FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "Issued fw reset\n"); } else { fnic_clear_state_flags(fnic, FNIC_FLAGS_FWRESET); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, - "Failed to issue fw reset\n"); + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "Failed to issue fw reset\n"); } return ret; @@ -259,15 +259,15 @@ int fnic_flogi_reg_handler(struct fnic *fnic, u32 fc_id) fc_id, gw_mac, fnic->data_src_addr, lp->r_a_tov, lp->e_d_tov); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "FLOGI FIP reg issued fcid %x src %pM dest %pM\n", fc_id, fnic->data_src_addr, gw_mac); } else { fnic_queue_wq_copy_desc_flogi_reg(wq, SCSI_NO_TAG, format, fc_id, gw_mac); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, - "FLOGI reg issued fcid %x map %d dest %pM\n", - fc_id, fnic->ctlr.map_dest, gw_mac); + FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "FLOGI reg issued fcid 0x%x map %d dest 0x%p\n", + fc_id, fnic->ctlr.map_dest, gw_mac); } atomic64_inc(&fnic->fnic_stats.fw_stats.active_fw_reqs); @@ -342,7 +342,7 @@ static inline int fnic_queue_wq_copy_desc(struct fnic *fnic, free_wq_copy_descs(fnic, wq, hwq); if (unlikely(!vnic_wq_copy_desc_avail(wq))) { - FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, + FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, "fnic_queue_wq_copy_desc failure - no descriptors\n"); atomic64_inc(&misc_stats->io_cpwq_alloc_failures); return SCSI_MLQUEUE_HOST_BUSY; @@ -408,24 +408,24 @@ int fnic_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *sc) if (unlikely(fnic_chk_state_flags_locked(fnic, FNIC_FLAGS_IO_BLOCKED))) { spin_unlock_irqrestore(&fnic->fnic_lock, flags); - FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, - "fnic<%d>: %s: %d: fnic IO blocked flags: 0x%lx. Returning SCSI_MLQUEUE_HOST_BUSY\n", - fnic->fnic_num, __func__, __LINE__, fnic->state_flags); + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "fnic IO blocked flags: 0x%lx. Returning SCSI_MLQUEUE_HOST_BUSY\n", + fnic->state_flags); return SCSI_MLQUEUE_HOST_BUSY; } if (unlikely(fnic_chk_state_flags_locked(fnic, FNIC_FLAGS_FWRESET))) { spin_unlock_irqrestore(&fnic->fnic_lock, flags); - FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, - "fnic<%d>: %s: %d: fnic flags: 0x%lx. Returning SCSI_MLQUEUE_HOST_BUSY\n", - fnic->fnic_num, __func__, __LINE__, fnic->state_flags); + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "fnic flags: 0x%lx. Returning SCSI_MLQUEUE_HOST_BUSY\n", + fnic->state_flags); return SCSI_MLQUEUE_HOST_BUSY; } rport = starget_to_rport(scsi_target(sc->device)); if (!rport) { spin_unlock_irqrestore(&fnic->fnic_lock, flags); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "returning DID_NO_CONNECT for IO as rport is NULL\n"); sc->result = DID_NO_CONNECT << 16; done(sc); @@ -435,7 +435,7 @@ int fnic_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *sc) ret = fc_remote_port_chkready(rport); if (ret) { spin_unlock_irqrestore(&fnic->fnic_lock, flags); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "rport is not ready\n"); atomic64_inc(&fnic_stats->misc_stats.rport_not_ready); sc->result = ret; @@ -446,7 +446,7 @@ int fnic_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *sc) rp = rport->dd_data; if (!rp || rp->rp_state == RPORT_ST_DELETE) { spin_unlock_irqrestore(&fnic->fnic_lock, flags); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "rport 0x%x removed, returning DID_NO_CONNECT\n", rport->port_id); @@ -458,7 +458,7 @@ int fnic_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *sc) if (rp->rp_state != RPORT_ST_READY) { spin_unlock_irqrestore(&fnic->fnic_lock, flags); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "rport 0x%x in state 0x%x, returning DID_IMM_RETRY\n", rport->port_id, rp->rp_state); @@ -469,9 +469,9 @@ int fnic_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *sc) if (lp->state != LPORT_ST_READY || !(lp->link_up)) { spin_unlock_irqrestore(&fnic->fnic_lock, flags); - FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, - "fnic<%d>: %s: %d: state not ready: %d/link not up: %d Returning HOST_BUSY\n", - fnic->fnic_num, __func__, __LINE__, lp->state, lp->link_up); + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "state not ready: %d/link not up: %d Returning HOST_BUSY\n", + lp->state, lp->link_up); return SCSI_MLQUEUE_HOST_BUSY; } @@ -636,15 +636,14 @@ static int fnic_fcpio_fw_reset_cmpl_handler(struct fnic *fnic, if (fnic->state == FNIC_IN_FC_TRANS_ETH_MODE) { /* Check status of reset completion */ if (!hdr_status) { - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, - "reset cmpl success\n"); + FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "reset cmpl success\n"); /* Ready to send flogi out */ fnic->state = FNIC_IN_ETH_MODE; } else { - FNIC_SCSI_DBG(KERN_DEBUG, - fnic->lport->host, - "fnic fw_reset : failed %s\n", - fnic_fcpio_status_to_str(hdr_status)); + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "reset failed with header status: %s\n", + fnic_fcpio_status_to_str(hdr_status)); /* * Unable to change to eth mode, cannot send out flogi @@ -657,10 +656,9 @@ static int fnic_fcpio_fw_reset_cmpl_handler(struct fnic *fnic, ret = -1; } } else { - FNIC_SCSI_DBG(KERN_DEBUG, - fnic->lport->host, - "Unexpected state %s while processing" - " reset cmpl\n", fnic_state_to_str(fnic->state)); + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "Unexpected state while processing reset completion: %s\n", + fnic_state_to_str(fnic->state)); atomic64_inc(&reset_stats->fw_reset_failures); ret = -1; } @@ -711,19 +709,19 @@ static int fnic_fcpio_flogi_reg_cmpl_handler(struct fnic *fnic, /* Check flogi registration completion status */ if (!hdr_status) { - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "flog reg succeeded\n"); fnic->state = FNIC_IN_FC_MODE; } else { FNIC_SCSI_DBG(KERN_DEBUG, - fnic->lport->host, + fnic->lport->host, fnic->fnic_num, "fnic flogi reg :failed %s\n", fnic_fcpio_status_to_str(hdr_status)); fnic->state = FNIC_IN_ETH_MODE; ret = -1; } } else { - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "Unexpected fnic state %s while" " processing flogi reg completion\n", fnic_state_to_str(fnic->state)); @@ -834,22 +832,20 @@ static void fnic_fcpio_icmnd_cmpl_handler(struct fnic *fnic, unsigned int cq_ind hwq = blk_mq_unique_tag_to_hwq(mqtag); if (hwq != cq_index) { - FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, - "fnic<%d>: %s: %d: hwq: %d mqtag: 0x%x tag: 0x%x cq index: %d ", - fnic->fnic_num, __func__, __LINE__, hwq, mqtag, tag, cq_index); - FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, - "fnic<%d>: %s: %d: hdr status: %s icmnd completion on the wrong queue\n", - fnic->fnic_num, __func__, __LINE__, + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "hwq: %d mqtag: 0x%x tag: 0x%x cq index: %d ", + hwq, mqtag, tag, cq_index); + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "hdr status: %s icmnd completion on the wrong queue\n", fnic_fcpio_status_to_str(hdr_status)); } if (tag >= fnic->fnic_max_tag_id) { - FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, - "fnic<%d>: %s: %d: hwq: %d mqtag: 0x%x tag: 0x%x cq index: %d ", - fnic->fnic_num, __func__, __LINE__, hwq, mqtag, tag, cq_index); - FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, - "fnic<%d>: %s: %d: hdr status: %s Out of range tag\n", - fnic->fnic_num, __func__, __LINE__, + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "hwq: %d mqtag: 0x%x tag: 0x%x cq index: %d ", + hwq, mqtag, tag, cq_index); + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "hdr status: %s Out of range tag\n", fnic_fcpio_status_to_str(hdr_status)); return; } @@ -915,7 +911,7 @@ static void fnic_fcpio_icmnd_cmpl_handler(struct fnic *fnic, unsigned int cq_ind if(FCPIO_ABORTED == hdr_status) fnic_priv(sc)->flags |= FNIC_IO_ABORTED; - FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, + FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, "icmnd_cmpl abts pending " "hdr status = %s tag = 0x%x sc = 0x%p " "scsi_status = %x residual = %d\n", @@ -1100,31 +1096,28 @@ static void fnic_fcpio_itmf_cmpl_handler(struct fnic *fnic, unsigned int cq_inde hwq = blk_mq_unique_tag_to_hwq(id & FNIC_TAG_MASK); if (hwq != cq_index) { - FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, - "%s: %d: hwq: %d mqtag: 0x%x tag: 0x%x cq index: %d ", - __func__, __LINE__, hwq, mqtag, tag, cq_index); - FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, - "%s: %d: hdr status: %s ITMF completion on the wrong queue\n", - __func__, __LINE__, + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "hwq: %d mqtag: 0x%x tag: 0x%x cq index: %d ", + hwq, mqtag, tag, cq_index); + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "hdr status: %s ITMF completion on the wrong queue\n", fnic_fcpio_status_to_str(hdr_status)); } if (tag > fnic->fnic_max_tag_id) { - FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, - "%s: %d: hwq: %d mqtag: 0x%x tag: 0x%x cq index: %d ", - __func__, __LINE__, hwq, mqtag, tag, cq_index); - FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, - "%s: %d: hdr status: %s Tag out of range\n", - __func__, __LINE__, + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "hwq: %d mqtag: 0x%x tag: 0x%x cq index: %d ", + hwq, mqtag, tag, cq_index); + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "hdr status: %s Tag out of range\n", fnic_fcpio_status_to_str(hdr_status)); return; } else if ((tag == fnic->fnic_max_tag_id) && !(id & FNIC_TAG_DEV_RST)) { - FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, - "%s: %d: hwq: %d mqtag: 0x%x tag: 0x%x cq index: %d ", - __func__, __LINE__, hwq, mqtag, tag, cq_index); - FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, - "%s: %d: hdr status: %s Tag out of range\n", - __func__, __LINE__, + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "hwq: %d mqtag: 0x%x tag: 0x%x cq index: %d ", + hwq, mqtag, tag, cq_index); + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "hdr status: %s Tag out of range\n", fnic_fcpio_status_to_str(hdr_status)); return; } @@ -1169,9 +1162,10 @@ static void fnic_fcpio_itmf_cmpl_handler(struct fnic *fnic, unsigned int cq_inde if ((id & FNIC_TAG_ABORT) && (id & FNIC_TAG_DEV_RST)) { /* Abort and terminate completion of device reset req */ /* REVISIT : Add asserts about various flags */ - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, - "dev reset abts cmpl recd. id %x status %s\n", - id, fnic_fcpio_status_to_str(hdr_status)); + FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "hwq: %d mqtag: 0x%x tag: 0x%x hst: %s Abt/term completion received\n", + hwq, mqtag, tag, + fnic_fcpio_status_to_str(hdr_status)); fnic_priv(sc)->state = FNIC_IOREQ_ABTS_COMPLETE; fnic_priv(sc)->abts_status = hdr_status; fnic_priv(sc)->flags |= FNIC_DEV_RST_DONE; @@ -1180,6 +1174,10 @@ static void fnic_fcpio_itmf_cmpl_handler(struct fnic *fnic, unsigned int cq_inde spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); } else if (id & FNIC_TAG_ABORT) { /* Completion of abort cmd */ + shost_printk(KERN_DEBUG, fnic->lport->host, + "hwq: %d mqtag: 0x%x tag: 0x%x Abort header status: %s\n", + hwq, mqtag, tag, + fnic_fcpio_status_to_str(hdr_status)); switch (hdr_status) { case FCPIO_SUCCESS: break; @@ -1191,7 +1189,7 @@ static void fnic_fcpio_itmf_cmpl_handler(struct fnic *fnic, unsigned int cq_inde &term_stats->terminate_fw_timeouts); break; case FCPIO_ITMF_REJECTED: - FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, + FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, "abort reject recd. id %d\n", (int)(id & FNIC_TAG_MASK)); break; @@ -1226,7 +1224,7 @@ static void fnic_fcpio_itmf_cmpl_handler(struct fnic *fnic, unsigned int cq_inde if (!(fnic_priv(sc)->flags & (FNIC_IO_ABORTED | FNIC_IO_DONE))) atomic64_inc(&misc_stats->no_icmnd_itmf_cmpls); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "abts cmpl recd. id %d status %s\n", (int)(id & FNIC_TAG_MASK), fnic_fcpio_status_to_str(hdr_status)); @@ -1239,9 +1237,14 @@ static void fnic_fcpio_itmf_cmpl_handler(struct fnic *fnic, unsigned int cq_inde if (io_req->abts_done) { complete(io_req->abts_done); spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); + shost_printk(KERN_INFO, fnic->lport->host, + "hwq: %d mqtag: 0x%x tag: 0x%x Waking up abort thread\n", + hwq, mqtag, tag); } else { - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, - "abts cmpl, completing IO\n"); + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, + "hwq: %d mqtag: 0x%x tag: 0x%x hst: %s Completing IO\n", + hwq, mqtag, + tag, fnic_fcpio_status_to_str(hdr_status)); fnic_priv(sc)->io_req = NULL; sc->result = (DID_ERROR << 16); fnic->sw_copy_wq[hwq].io_req_table[tag] = NULL; @@ -1269,6 +1272,10 @@ static void fnic_fcpio_itmf_cmpl_handler(struct fnic *fnic, unsigned int cq_inde } } else if (id & FNIC_TAG_DEV_RST) { /* Completion of device reset */ + shost_printk(KERN_INFO, fnic->lport->host, + "hwq: %d mqtag: 0x%x tag: 0x%x DR hst: %s\n", + hwq, mqtag, + tag, fnic_fcpio_status_to_str(hdr_status)); fnic_priv(sc)->lr_status = hdr_status; if (fnic_priv(sc)->state == FNIC_IOREQ_ABTS_PENDING) { spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); @@ -1277,11 +1284,10 @@ static void fnic_fcpio_itmf_cmpl_handler(struct fnic *fnic, unsigned int cq_inde sc->device->host->host_no, id, sc, jiffies_to_msecs(jiffies - start_time), desc, 0, fnic_flags_and_state(sc)); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, - "Terminate pending " - "dev reset cmpl recd. id %d status %s\n", - (int)(id & FNIC_TAG_MASK), - fnic_fcpio_status_to_str(hdr_status)); + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, + "hwq: %d mqtag: 0x%x tag: 0x%x hst: %s Terminate pending\n", + hwq, mqtag, + tag, fnic_fcpio_status_to_str(hdr_status)); return; } if (fnic_priv(sc)->flags & FNIC_DEV_RST_TIMED_OUT) { @@ -1291,7 +1297,7 @@ static void fnic_fcpio_itmf_cmpl_handler(struct fnic *fnic, unsigned int cq_inde sc->device->host->host_no, id, sc, jiffies_to_msecs(jiffies - start_time), desc, 0, fnic_flags_and_state(sc)); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "dev reset cmpl recd after time out. " "id %d status %s\n", (int)(id & FNIC_TAG_MASK), @@ -1300,18 +1306,18 @@ static void fnic_fcpio_itmf_cmpl_handler(struct fnic *fnic, unsigned int cq_inde } fnic_priv(sc)->state = FNIC_IOREQ_CMD_COMPLETE; fnic_priv(sc)->flags |= FNIC_DEV_RST_DONE; - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, - "dev reset cmpl recd. id %d status %s\n", - (int)(id & FNIC_TAG_MASK), - fnic_fcpio_status_to_str(hdr_status)); + FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "hwq: %d mqtag: 0x%x tag: 0x%x hst: %s DR completion received\n", + hwq, mqtag, + tag, fnic_fcpio_status_to_str(hdr_status)); if (io_req->dr_done) complete(io_req->dr_done); spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); } else { shost_printk(KERN_ERR, fnic->lport->host, - "Unexpected itmf io state %s tag %x\n", - fnic_ioreq_state_to_str(fnic_priv(sc)->state), id); + "%s: Unexpected itmf io state: hwq: %d tag 0x%x %s\n", + __func__, hwq, id, fnic_ioreq_state_to_str(fnic_priv(sc)->state)); spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); } @@ -1364,7 +1370,7 @@ static int fnic_fcpio_cmpl_handler(struct vnic_dev *vdev, break; default: - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "firmware completion type %d\n", desc->hdr.type); break; @@ -1425,9 +1431,9 @@ static bool fnic_cleanup_io_iter(struct scsi_cmnd *sc, void *data) io_req = fnic_priv(sc)->io_req; if (!io_req) { spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); - FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, - "fnic<%d>: %s: %d: hwq: %d mqtag: 0x%x tag: 0x%x flags: 0x%x No ioreq. Returning\n", - fnic->fnic_num, __func__, __LINE__, hwq, mqtag, tag, fnic_priv(sc)->flags); + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "hwq: %d mqtag: 0x%x tag: 0x%x flags: 0x%x No ioreq. Returning\n", + hwq, mqtag, tag, fnic_priv(sc)->flags); return true; } @@ -1462,9 +1468,9 @@ static bool fnic_cleanup_io_iter(struct scsi_cmnd *sc, void *data) mempool_free(io_req, fnic->io_req_pool); sc->result = DID_TRANSPORT_DISRUPTED << 16; - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, - "fnic_cleanup_io: tag:0x%x : sc:0x%p duration = %lu DID_TRANSPORT_DISRUPTED\n", - tag, sc, jiffies - start_time); + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "mqtag:0x%x tag: 0x%x sc:0x%p duration = %lu DID_TRANSPORT_DISRUPTED\n", + mqtag, tag, sc, (jiffies - start_time)); if (atomic64_read(&fnic->io_cmpl_skip)) atomic64_dec(&fnic->io_cmpl_skip); @@ -1538,7 +1544,7 @@ void fnic_wq_copy_cleanup_handler(struct vnic_wq_copy *wq, wq_copy_cleanup_scsi_cmd: sc->result = DID_NO_CONNECT << 16; - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, "wq_copy_cleanup_handler:" + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "wq_copy_cleanup_handler:" " DID_NO_CONNECT\n"); FNIC_TRACE(fnic_wq_copy_cleanup_handler, @@ -1578,7 +1584,7 @@ static inline int fnic_queue_abort_io_req(struct fnic *fnic, int tag, if (!vnic_wq_copy_desc_avail(wq)) { spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); atomic_dec(&fnic->in_flight); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "fnic_queue_abort_io_req: failure: no descriptors\n"); atomic64_inc(&misc_stats->abts_cpwq_alloc_failures); return 1; @@ -1633,9 +1639,9 @@ static bool fnic_rport_abort_io_iter(struct scsi_cmnd *sc, void *data) if ((fnic_priv(sc)->flags & FNIC_DEVICE_RESET) && !(fnic_priv(sc)->flags & FNIC_DEV_RST_ISSUED)) { - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, - "fnic_rport_exch_reset dev rst not pending sc 0x%p\n", - sc); + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "hwq: %d abt_tag: 0x%x flags: 0x%x Device reset is not pending\n", + hwq, abt_tag, fnic_priv(sc)->flags); spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); return true; } @@ -1669,11 +1675,11 @@ static bool fnic_rport_abort_io_iter(struct scsi_cmnd *sc, void *data) atomic64_inc(&reset_stats->device_reset_terminates); abt_tag |= FNIC_TAG_DEV_RST; } - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "fnic_rport_exch_reset dev rst sc 0x%p\n", sc); BUG_ON(io_req->abts_done); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "fnic_rport_reset_exch: Issuing abts\n"); spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); @@ -1691,6 +1697,9 @@ static bool fnic_rport_abort_io_iter(struct scsi_cmnd *sc, void *data) * lun reset */ spin_lock_irqsave(&fnic->wq_copy_lock[hwq], flags); + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "hwq: %d abt_tag: 0x%x flags: 0x%x Queuing abort failed\n", + hwq, abt_tag, fnic_priv(sc)->flags); if (fnic_priv(sc)->state == FNIC_IOREQ_ABTS_PENDING) fnic_priv(sc)->state = old_ioreq_state; spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); @@ -1717,7 +1726,7 @@ static void fnic_rport_exch_reset(struct fnic *fnic, u32 port_id) }; FNIC_SCSI_DBG(KERN_DEBUG, - fnic->lport->host, + fnic->lport->host, fnic->fnic_num, "fnic_rport_exch_reset called portid 0x%06x\n", port_id); @@ -1754,9 +1763,8 @@ void fnic_terminate_rport_io(struct fc_rport *rport) return; } fnic = lport_priv(lport); - FNIC_SCSI_DBG(KERN_DEBUG, - fnic->lport->host, "fnic_terminate_rport_io called" - " wwpn 0x%llx, wwnn0x%llx, rport 0x%p, portid 0x%06x\n", + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, + "wwpn 0x%llx, wwnn0x%llx, rport 0x%p, portid 0x%06x\n", rport->port_name, rport->node_name, rport, rport->port_id); @@ -1861,8 +1869,9 @@ int fnic_abort_cmd(struct scsi_cmnd *sc) else atomic64_inc(&abts_stats->abort_issued_greater_than_60_sec); - FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, - "CBD Opcode: %02x Abort issued time: %lu msec\n", sc->cmnd[0], abt_issued_time); + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, + "CDB Opcode: 0x%02x Abort issued time: %lu msec\n", + sc->cmnd[0], abt_issued_time); /* * Command is still pending, need to abort it * If the firmware completes the command after this point, @@ -1951,8 +1960,8 @@ int fnic_abort_cmd(struct scsi_cmnd *sc) if (!(fnic_priv(sc)->flags & (FNIC_IO_ABORTED | FNIC_IO_DONE))) { spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, - "Issuing Host reset due to out of order IO\n"); + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "Issuing host reset due to out of order IO\n"); ret = FAILED; goto fnic_abort_cmd_end; @@ -1999,7 +2008,7 @@ int fnic_abort_cmd(struct scsi_cmnd *sc) (u64)sc->cmnd[4] << 8 | sc->cmnd[5]), fnic_flags_and_state(sc)); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "Returning from abort cmd type %x %s\n", task_req, (ret == SUCCESS) ? "SUCCESS" : "FAILED"); @@ -2037,7 +2046,7 @@ static inline int fnic_queue_dr_io_req(struct fnic *fnic, free_wq_copy_descs(fnic, wq, hwq); if (!vnic_wq_copy_desc_avail(wq)) { - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "queue_dr_io_req failure - no descriptors\n"); atomic64_inc(&misc_stats->devrst_cpwq_alloc_failures); ret = -EAGAIN; @@ -2104,7 +2113,7 @@ static bool fnic_pending_aborts_iter(struct scsi_cmnd *sc, void *data) * Found IO that is still pending with firmware and * belongs to the LUN that we are resetting */ - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "Found IO in %s on lun\n", fnic_ioreq_state_to_str(fnic_priv(sc)->state)); @@ -2114,9 +2123,8 @@ static bool fnic_pending_aborts_iter(struct scsi_cmnd *sc, void *data) } if ((fnic_priv(sc)->flags & FNIC_DEVICE_RESET) && (!(fnic_priv(sc)->flags & FNIC_DEV_RST_ISSUED))) { - FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, - "%s dev rst not pending sc 0x%p\n", __func__, - sc); + FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "dev rst not pending sc 0x%p\n", sc); spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); return true; } @@ -2138,8 +2146,8 @@ static bool fnic_pending_aborts_iter(struct scsi_cmnd *sc, void *data) BUG_ON(io_req->abts_done); if (fnic_priv(sc)->flags & FNIC_DEVICE_RESET) { - FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, - "%s: dev rst sc 0x%p\n", __func__, sc); + FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "dev rst sc 0x%p\n", sc); } fnic_priv(sc)->abts_status = FCPIO_INVALID_CODE; @@ -2160,6 +2168,9 @@ static bool fnic_pending_aborts_iter(struct scsi_cmnd *sc, void *data) fnic_priv(sc)->state = old_ioreq_state; spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); iter_data->ret = FAILED; + FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "hwq: %d abt_tag: 0x%lx Abort could not be queued\n", + hwq, abt_tag); return false; } else { spin_lock_irqsave(&fnic->wq_copy_lock[hwq], flags); @@ -2249,8 +2260,8 @@ static int fnic_clean_pending_aborts(struct fnic *fnic, ret = 1; clean_pending_aborts_end: - FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, - "%s: exit status: %d\n", __func__, ret); + FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "exit status: %d\n", ret); return ret; } @@ -2291,9 +2302,10 @@ int fnic_device_reset(struct scsi_cmnd *sc) atomic64_inc(&reset_stats->device_resets); rport = starget_to_rport(scsi_target(sc->device)); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, - "Device reset called FCID 0x%x, LUN 0x%llx sc 0x%p\n", - rport->port_id, sc->device->lun, sc); + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, + "fcid: 0x%x lun: 0x%llx hwq: %d mqtag: 0x%x flags: 0x%x Device reset\n", + rport->port_id, sc->device->lun, hwq, mqtag, + fnic_priv(sc)->flags); if (lp->state != LPORT_ST_READY || !(lp->link_up)) goto fnic_device_reset_end; @@ -2353,7 +2365,7 @@ int fnic_device_reset(struct scsi_cmnd *sc) fnic_priv(sc)->lr_status = FCPIO_INVALID_CODE; spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, "TAG %x\n", mqtag); + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "TAG %x\n", mqtag); /* * issue the device reset, if enqueue failed, clean up the ioreq @@ -2381,7 +2393,7 @@ int fnic_device_reset(struct scsi_cmnd *sc) io_req = fnic_priv(sc)->io_req; if (!io_req) { spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "io_req is null mqtag 0x%x sc 0x%p\n", mqtag, sc); goto fnic_device_reset_end; } @@ -2395,7 +2407,7 @@ int fnic_device_reset(struct scsi_cmnd *sc) */ if (status == FCPIO_INVALID_CODE) { atomic64_inc(&reset_stats->device_reset_timeouts); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "Device reset timed out\n"); fnic_priv(sc)->flags |= FNIC_DEV_RST_TIMED_OUT; spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); @@ -2423,7 +2435,7 @@ int fnic_device_reset(struct scsi_cmnd *sc) fnic_priv(sc)->state = FNIC_IOREQ_ABTS_PENDING; io_req->abts_done = &tm_done; spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "Abort and terminate issued on Device reset mqtag 0x%x sc 0x%p\n", mqtag, sc); break; @@ -2450,7 +2462,7 @@ int fnic_device_reset(struct scsi_cmnd *sc) if (status != FCPIO_SUCCESS) { spin_lock_irqsave(&fnic->wq_copy_lock[hwq], flags); FNIC_SCSI_DBG(KERN_DEBUG, - fnic->lport->host, + fnic->lport->host, fnic->fnic_num, "Device reset completed - failed\n"); io_req = fnic_priv(sc)->io_req; goto fnic_device_reset_clean; @@ -2466,7 +2478,7 @@ int fnic_device_reset(struct scsi_cmnd *sc) if (fnic_clean_pending_aborts(fnic, sc, new_sc)) { spin_lock_irqsave(&fnic->wq_copy_lock[hwq], flags); io_req = fnic_priv(sc)->io_req; - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "Device reset failed" " since could not abort all IOs\n"); goto fnic_device_reset_clean; @@ -2507,7 +2519,7 @@ int fnic_device_reset(struct scsi_cmnd *sc) mutex_unlock(&fnic->sgreset_mutex); } - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "Returning from device reset %s\n", (ret == SUCCESS) ? "SUCCESS" : "FAILED"); @@ -2530,8 +2542,8 @@ int fnic_reset(struct Scsi_Host *shost) fnic = lport_priv(lp); reset_stats = &fnic->fnic_stats.reset_stats; - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, - "fnic_reset called\n"); + FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "Issuing fnic reset\n"); atomic64_inc(&reset_stats->fnic_resets); @@ -2541,10 +2553,9 @@ int fnic_reset(struct Scsi_Host *shost) */ ret = fc_lport_reset(lp); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, - "Returning from fnic reset %s\n", - (ret == 0) ? - "SUCCESS" : "FAILED"); + FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "Returning from fnic reset with: %s\n", + (ret == 0) ? "SUCCESS" : "FAILED"); if (ret == 0) atomic64_inc(&reset_stats->fnic_reset_completions); @@ -2577,7 +2588,7 @@ int fnic_host_reset(struct scsi_cmnd *sc) fnic->internal_reset_inprogress = true; } else { spin_unlock_irqrestore(&fnic->fnic_lock, flags); - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "host reset in progress skipping another host reset\n"); return SUCCESS; } @@ -2652,7 +2663,7 @@ void fnic_scsi_abort_io(struct fc_lport *lp) spin_lock_irqsave(&fnic->fnic_lock, flags); fnic->remove_wait = NULL; - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "fnic_scsi_abort_io %s\n", (fnic->state == FNIC_IN_ETH_MODE) ? "SUCCESS" : "FAILED"); @@ -2759,9 +2770,10 @@ static bool fnic_abts_pending_iter(struct scsi_cmnd *sc, void *data) * Found IO that is still pending with firmware and * belongs to the LUN that we are resetting */ - FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, - "Found IO in %s on lun\n", - fnic_ioreq_state_to_str(fnic_priv(sc)->state)); + FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "hwq: %d tag: 0x%x Found IO in state: %s on lun\n", + hwq, tag, + fnic_ioreq_state_to_str(fnic_priv(sc)->state)); cmd_state = fnic_priv(sc)->state; spin_unlock_irqrestore(&fnic->wq_copy_lock[hwq], flags); if (cmd_state == FNIC_IOREQ_ABTS_PENDING)