From patchwork Thu Apr 29 19:09:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shai Malin X-Patchwork-Id: 429662 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18C8BC433B4 for ; Thu, 29 Apr 2021 19:17:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D3EDC61077 for ; Thu, 29 Apr 2021 19:16:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230316AbhD2TRj (ORCPT ); Thu, 29 Apr 2021 15:17:39 -0400 Received: from mx0a-0016f401.pphosted.com ([67.231.148.174]:23496 "EHLO mx0b-0016f401.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S239474AbhD2TKt (ORCPT ); Thu, 29 Apr 2021 15:10:49 -0400 Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13TJ6WFD019663; Thu, 29 Apr 2021 12:09:54 -0700 Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com with ESMTP id 387erumvxq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 29 Apr 2021 12:09:54 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 29 Apr 2021 12:09:53 -0700 Received: from lbtlvb-pcie154.il.qlogic.org (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 29 Apr 2021 12:09:50 -0700 From: Shai Malin To: , , , , , CC: =?utf-8?q?David_S_=2E_Miller_davem_=40_davemloft_=2E_net_=C2=A0--cc?= =?utf-8?q?=3DJakub_Kicinski?= , , , , , , Subject: [RFC PATCH v4 04/27] qed: Add support of HW filter block Date: Thu, 29 Apr 2021 22:09:03 +0300 Message-ID: <20210429190926.5086-5-smalin@marvell.com> X-Mailer: git-send-email 2.16.6 In-Reply-To: <20210429190926.5086-1-smalin@marvell.com> References: <20210429190926.5086-1-smalin@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: 9xuTMRwhvtQhf6pizexO5SkHBRDh1_se X-Proofpoint-ORIG-GUID: 9xuTMRwhvtQhf6pizexO5SkHBRDh1_se X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-04-29_10:2021-04-28,2021-04-29 signatures=0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Prabhakar Kushwaha This patch introduces the functionality of HW filter block. It adds and removes filters based on source and target TCP port. It also add functionality to clear all filters at once. Acked-by: Igor Russkikh Signed-off-by: Prabhakar Kushwaha Signed-off-by: Omkar Kulkarni Signed-off-by: Shai Malin Signed-off-by: Michal Kalderon Signed-off-by: Ariel Elior Reviewed-by: Hannes Reinecke --- drivers/net/ethernet/qlogic/qed/qed.h | 10 ++ drivers/net/ethernet/qlogic/qed/qed_dev.c | 107 ++++++++++++++++++ drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c | 5 + include/linux/qed/qed_nvmetcp_if.h | 24 ++++ 4 files changed, 146 insertions(+) diff --git a/drivers/net/ethernet/qlogic/qed/qed.h b/drivers/net/ethernet/qlogic/qed/qed.h index 7ae648c4edba..c2305ff5bdc6 100644 --- a/drivers/net/ethernet/qlogic/qed/qed.h +++ b/drivers/net/ethernet/qlogic/qed/qed.h @@ -49,6 +49,8 @@ extern const struct qed_common_ops qed_common_ops_pass; #define QED_MIN_WIDS (4) #define QED_PF_DEMS_SIZE (4) +#define QED_LLH_DONT_CARE 0 + /* cau states */ enum qed_coalescing_mode { QED_COAL_MODE_DISABLE, @@ -1005,4 +1007,12 @@ int qed_mfw_fill_tlv_data(struct qed_hwfn *hwfn, void qed_hw_info_set_offload_tc(struct qed_hw_info *p_info, u8 tc); void qed_periodic_db_rec_start(struct qed_hwfn *p_hwfn); + +int qed_llh_add_src_tcp_port_filter(struct qed_dev *cdev, u16 src_port); +int qed_llh_add_dst_tcp_port_filter(struct qed_dev *cdev, u16 dest_port); + +void qed_llh_remove_src_tcp_port_filter(struct qed_dev *cdev, u16 src_port); +void qed_llh_remove_dst_tcp_port_filter(struct qed_dev *cdev, u16 src_port); + +void qed_llh_clear_all_filters(struct qed_dev *cdev); #endif /* _QED_H */ diff --git a/drivers/net/ethernet/qlogic/qed/qed_dev.c b/drivers/net/ethernet/qlogic/qed/qed_dev.c index d3f8cc42d07e..12382e0c0419 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_dev.c +++ b/drivers/net/ethernet/qlogic/qed/qed_dev.c @@ -5360,3 +5360,110 @@ void qed_set_fw_mac_addr(__le16 *fw_msb, ((u8 *)fw_lsb)[0] = mac[5]; ((u8 *)fw_lsb)[1] = mac[4]; } + +static int qed_llh_shadow_remove_all_filters(struct qed_dev *cdev, u8 ppfid) +{ + struct qed_llh_info *p_llh_info = cdev->p_llh_info; + struct qed_llh_filter_info *p_filters; + int rc; + + rc = qed_llh_shadow_sanity(cdev, ppfid, 0, "remove_all"); + if (rc) + return rc; + + p_filters = p_llh_info->pp_filters[ppfid]; + memset(p_filters, 0, NIG_REG_LLH_FUNC_FILTER_EN_SIZE * + sizeof(*p_filters)); + + return 0; +} + +int qed_abs_ppfid(struct qed_dev *cdev, u8 rel_ppfid, u8 *p_abs_ppfid) +{ + struct qed_llh_info *p_llh_info = cdev->p_llh_info; + + if (rel_ppfid >= p_llh_info->num_ppfid) { + DP_NOTICE(cdev, + "rel_ppfid %d is not valid, available indices are 0..%hhu\n", + rel_ppfid, p_llh_info->num_ppfid - 1); + + return -EINVAL; + } + + *p_abs_ppfid = p_llh_info->ppfid_array[rel_ppfid]; + + return 0; +} + +void qed_llh_clear_ppfid_filters(struct qed_dev *cdev, u8 ppfid) +{ + struct qed_hwfn *p_hwfn = QED_LEADING_HWFN(cdev); + struct qed_ptt *p_ptt = qed_ptt_acquire(p_hwfn); + u8 filter_idx, abs_ppfid; + int rc = 0; + + if (!p_ptt) + return; + + if (!test_bit(QED_MF_LLH_PROTO_CLSS, &cdev->mf_bits) && + !test_bit(QED_MF_LLH_MAC_CLSS, &cdev->mf_bits)) + goto out; + + rc = qed_abs_ppfid(cdev, ppfid, &abs_ppfid); + if (rc) + goto out; + + rc = qed_llh_shadow_remove_all_filters(cdev, ppfid); + if (rc) + goto out; + + for (filter_idx = 0; filter_idx < NIG_REG_LLH_FUNC_FILTER_EN_SIZE; + filter_idx++) { + rc = qed_llh_remove_filter(p_hwfn, p_ptt, + abs_ppfid, filter_idx); + if (rc) + goto out; + } +out: + qed_ptt_release(p_hwfn, p_ptt); +} + +int qed_llh_add_src_tcp_port_filter(struct qed_dev *cdev, u16 src_port) +{ + return qed_llh_add_protocol_filter(cdev, 0, + QED_LLH_FILTER_TCP_SRC_PORT, + src_port, QED_LLH_DONT_CARE); +} + +void qed_llh_remove_src_tcp_port_filter(struct qed_dev *cdev, u16 src_port) +{ + qed_llh_remove_protocol_filter(cdev, 0, + QED_LLH_FILTER_TCP_SRC_PORT, + src_port, QED_LLH_DONT_CARE); +} + +int qed_llh_add_dst_tcp_port_filter(struct qed_dev *cdev, u16 dest_port) +{ + return qed_llh_add_protocol_filter(cdev, 0, + QED_LLH_FILTER_TCP_DEST_PORT, + QED_LLH_DONT_CARE, dest_port); +} + +void qed_llh_remove_dst_tcp_port_filter(struct qed_dev *cdev, u16 dest_port) +{ + qed_llh_remove_protocol_filter(cdev, 0, + QED_LLH_FILTER_TCP_DEST_PORT, + QED_LLH_DONT_CARE, dest_port); +} + +void qed_llh_clear_all_filters(struct qed_dev *cdev) +{ + u8 ppfid; + + if (!test_bit(QED_MF_LLH_PROTO_CLSS, &cdev->mf_bits) && + !test_bit(QED_MF_LLH_MAC_CLSS, &cdev->mf_bits)) + return; + + for (ppfid = 0; ppfid < cdev->p_llh_info->num_ppfid; ppfid++) + qed_llh_clear_ppfid_filters(cdev, ppfid); +} diff --git a/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c index 79bd1cc6677f..1e2eb6dcbd6e 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c +++ b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c @@ -844,6 +844,11 @@ static const struct qed_nvmetcp_ops qed_nvmetcp_ops_pass = { .update_conn = &qed_nvmetcp_update_conn, .destroy_conn = &qed_nvmetcp_destroy_conn, .clear_sq = &qed_nvmetcp_clear_conn_sq, + .add_src_tcp_port_filter = &qed_llh_add_src_tcp_port_filter, + .remove_src_tcp_port_filter = &qed_llh_remove_src_tcp_port_filter, + .add_dst_tcp_port_filter = &qed_llh_add_dst_tcp_port_filter, + .remove_dst_tcp_port_filter = &qed_llh_remove_dst_tcp_port_filter, + .clear_all_filters = &qed_llh_clear_all_filters }; const struct qed_nvmetcp_ops *qed_get_nvmetcp_ops(void) diff --git a/include/linux/qed/qed_nvmetcp_if.h b/include/linux/qed/qed_nvmetcp_if.h index 96263e3cfa1e..686f924238e3 100644 --- a/include/linux/qed/qed_nvmetcp_if.h +++ b/include/linux/qed/qed_nvmetcp_if.h @@ -124,6 +124,20 @@ struct qed_nvmetcp_cb_ops { * @param cdev * @param handle - the connection handle. * @return 0 on success, otherwise error value. + * @add_src_tcp_port_filter: Add source tcp port filter + * @param cdev + * @param src_port + * @remove_src_tcp_port_filter: Remove source tcp port filter + * @param cdev + * @param src_port + * @add_dst_tcp_port_filter: Add destination tcp port filter + * @param cdev + * @param dest_port + * @remove_dst_tcp_port_filter: Remove destination tcp port filter + * @param cdev + * @param dest_port + * @clear_all_filters: Clear all filters. + * @param cdev */ struct qed_nvmetcp_ops { const struct qed_common_ops *common; @@ -159,6 +173,16 @@ struct qed_nvmetcp_ops { int (*destroy_conn)(struct qed_dev *cdev, u32 handle, u8 abrt_conn); int (*clear_sq)(struct qed_dev *cdev, u32 handle); + + int (*add_src_tcp_port_filter)(struct qed_dev *cdev, u16 src_port); + + void (*remove_src_tcp_port_filter)(struct qed_dev *cdev, u16 src_port); + + int (*add_dst_tcp_port_filter)(struct qed_dev *cdev, u16 dest_port); + + void (*remove_dst_tcp_port_filter)(struct qed_dev *cdev, u16 dest_port); + + void (*clear_all_filters)(struct qed_dev *cdev); }; const struct qed_nvmetcp_ops *qed_get_nvmetcp_ops(void); From patchwork Thu Apr 29 19:09:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shai Malin X-Patchwork-Id: 429661 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21038C43470 for ; Thu, 29 Apr 2021 19:18:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F1A726144E for ; Thu, 29 Apr 2021 19:18:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233073AbhD2TRo (ORCPT ); Thu, 29 Apr 2021 15:17:44 -0400 Received: from mx0b-0016f401.pphosted.com ([67.231.156.173]:65080 "EHLO mx0b-0016f401.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239527AbhD2TK4 (ORCPT ); Thu, 29 Apr 2021 15:10:56 -0400 Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13TJ5Ara027348; Thu, 29 Apr 2021 12:09:59 -0700 Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com with ESMTP id 387rpnammr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 29 Apr 2021 12:09:59 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 29 Apr 2021 12:09:57 -0700 Received: from lbtlvb-pcie154.il.qlogic.org (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 29 Apr 2021 12:09:54 -0700 From: Shai Malin To: , , , , , CC: =?utf-8?q?David_S_=2E_Miller_davem_=40_davemloft_=2E_net_=C2=A0--cc?= =?utf-8?q?=3DJakub_Kicinski?= , , , , , , Subject: [RFC PATCH v4 05/27] qed: Add NVMeTCP Offload IO Level FW and HW HSI Date: Thu, 29 Apr 2021 22:09:04 +0300 Message-ID: <20210429190926.5086-6-smalin@marvell.com> X-Mailer: git-send-email 2.16.6 In-Reply-To: <20210429190926.5086-1-smalin@marvell.com> References: <20210429190926.5086-1-smalin@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: 39ddqi2-31yBsprMvz6R838HFKJmS4xL X-Proofpoint-ORIG-GUID: 39ddqi2-31yBsprMvz6R838HFKJmS4xL X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-04-29_10:2021-04-28,2021-04-29 signatures=0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This patch introduces the NVMeTCP Offload FW and HW HSI in order to initialize the IO level configuration into a per IO HW resource ("task") as part of the IO path flow. Acked-by: Igor Russkikh Signed-off-by: Prabhakar Kushwaha Signed-off-by: Omkar Kulkarni Signed-off-by: Shai Malin Signed-off-by: Michal Kalderon Signed-off-by: Ariel Elior --- include/linux/qed/nvmetcp_common.h | 418 ++++++++++++++++++++++++++++- include/linux/qed/qed_nvmetcp_if.h | 37 +++ 2 files changed, 454 insertions(+), 1 deletion(-) diff --git a/include/linux/qed/nvmetcp_common.h b/include/linux/qed/nvmetcp_common.h index c8836b71b866..dda7a785c321 100644 --- a/include/linux/qed/nvmetcp_common.h +++ b/include/linux/qed/nvmetcp_common.h @@ -7,6 +7,7 @@ #include "tcp_common.h" #define NVMETCP_SLOW_PATH_LAYER_CODE (6) +#define NVMETCP_WQE_NUM_SGES_SLOWIO (0xf) /* NVMeTCP firmware function init parameters */ struct nvmetcp_spe_func_init { @@ -194,4 +195,419 @@ struct nvmetcp_wqe { #define NVMETCP_WQE_CDB_SIZE_OR_NVMETCP_CMD_SHIFT 24 }; -#endif /* __NVMETCP_COMMON__ */ +struct nvmetcp_host_cccid_itid_entry { + __le16 itid; +}; + +struct nvmetcp_connect_done_results { + __le16 icid; + __le16 conn_id; + struct tcp_ulp_connect_done_params params; +}; + +struct nvmetcp_eqe_data { + __le16 icid; + __le16 conn_id; + __le16 reserved; + u8 error_code; + u8 error_pdu_opcode_reserved; +#define NVMETCP_EQE_DATA_ERROR_PDU_OPCODE_MASK 0x3F +#define NVMETCP_EQE_DATA_ERROR_PDU_OPCODE_SHIFT 0 +#define NVMETCP_EQE_DATA_ERROR_PDU_OPCODE_VALID_MASK 0x1 +#define NVMETCP_EQE_DATA_ERROR_PDU_OPCODE_VALID_SHIFT 6 +#define NVMETCP_EQE_DATA_RESERVED0_MASK 0x1 +#define NVMETCP_EQE_DATA_RESERVED0_SHIFT 7 +}; + +enum nvmetcp_task_type { + NVMETCP_TASK_TYPE_HOST_WRITE, + NVMETCP_TASK_TYPE_HOST_READ, + NVMETCP_TASK_TYPE_INIT_CONN_REQUEST, + NVMETCP_TASK_TYPE_RESERVED0, + NVMETCP_TASK_TYPE_CLEANUP, + NVMETCP_TASK_TYPE_HOST_READ_NO_CQE, + MAX_NVMETCP_TASK_TYPE +}; + +struct nvmetcp_db_data { + u8 params; +#define NVMETCP_DB_DATA_DEST_MASK 0x3 /* destination of doorbell (use enum db_dest) */ +#define NVMETCP_DB_DATA_DEST_SHIFT 0 +#define NVMETCP_DB_DATA_AGG_CMD_MASK 0x3 /* aggregative command to CM (use enum db_agg_cmd_sel) */ +#define NVMETCP_DB_DATA_AGG_CMD_SHIFT 2 +#define NVMETCP_DB_DATA_BYPASS_EN_MASK 0x1 /* enable QM bypass */ +#define NVMETCP_DB_DATA_BYPASS_EN_SHIFT 4 +#define NVMETCP_DB_DATA_RESERVED_MASK 0x1 +#define NVMETCP_DB_DATA_RESERVED_SHIFT 5 +#define NVMETCP_DB_DATA_AGG_VAL_SEL_MASK 0x3 /* aggregative value selection */ +#define NVMETCP_DB_DATA_AGG_VAL_SEL_SHIFT 6 + u8 agg_flags; /* bit for every DQ counter flags in CM context that DQ can increment */ + __le16 sq_prod; +}; + +struct nvmetcp_fw_cqe_error_bitmap { + u8 cqe_error_status_bits; +#define CQE_ERROR_BITMAP_DIF_ERR_BITS_MASK 0x7 +#define CQE_ERROR_BITMAP_DIF_ERR_BITS_SHIFT 0 +#define CQE_ERROR_BITMAP_DATA_DIGEST_ERR_MASK 0x1 +#define CQE_ERROR_BITMAP_DATA_DIGEST_ERR_SHIFT 3 +#define CQE_ERROR_BITMAP_RCV_ON_INVALID_CONN_MASK 0x1 +#define CQE_ERROR_BITMAP_RCV_ON_INVALID_CONN_SHIFT 4 +}; + +struct nvmetcp_nvmf_cqe { + __le32 reserved[4]; +}; + +struct nvmetcp_fw_cqe_data { + struct nvmetcp_nvmf_cqe nvme_cqe; + struct regpair task_opaque; + __le32 reserved[6]; +}; + +struct nvmetcp_fw_cqe { + __le16 conn_id; + u8 cqe_type; + struct nvmetcp_fw_cqe_error_bitmap error_bitmap; + __le16 itid; + u8 task_type; + u8 fw_dbg_field; + u8 caused_conn_err; + u8 reserved0[3]; + __le32 reserved1; + struct nvmetcp_nvmf_cqe nvme_cqe; + struct regpair task_opaque; + __le32 reserved[6]; +}; + +enum nvmetcp_fw_cqes_type { + NVMETCP_FW_CQE_TYPE_NORMAL = 1, + NVMETCP_FW_CQE_TYPE_RESERVED0, + NVMETCP_FW_CQE_TYPE_RESERVED1, + NVMETCP_FW_CQE_TYPE_CLEANUP, + NVMETCP_FW_CQE_TYPE_DUMMY, + MAX_NVMETCP_FW_CQES_TYPE +}; + +struct ystorm_nvmetcp_task_state { + struct scsi_cached_sges data_desc; + struct scsi_sgl_params sgl_params; + __le32 resrved0; + __le32 buffer_offset; + __le16 cccid; + struct nvmetcp_dif_flags dif_flags; + u8 flags; +#define YSTORM_NVMETCP_TASK_STATE_LOCAL_COMP_MASK 0x1 +#define YSTORM_NVMETCP_TASK_STATE_LOCAL_COMP_SHIFT 0 +#define YSTORM_NVMETCP_TASK_STATE_SLOW_IO_MASK 0x1 +#define YSTORM_NVMETCP_TASK_STATE_SLOW_IO_SHIFT 1 +#define YSTORM_NVMETCP_TASK_STATE_SET_DIF_OFFSET_MASK 0x1 +#define YSTORM_NVMETCP_TASK_STATE_SET_DIF_OFFSET_SHIFT 2 +#define YSTORM_NVMETCP_TASK_STATE_SEND_W_RSP_MASK 0x1 +#define YSTORM_NVMETCP_TASK_STATE_SEND_W_RSP_SHIFT 3 +}; + +struct ystorm_nvmetcp_task_rxmit_opt { + __le32 reserved[4]; +}; + +struct nvmetcp_task_hdr { + __le32 reg[18]; +}; + +struct nvmetcp_task_hdr_aligned { + struct nvmetcp_task_hdr task_hdr; + __le32 reserved[2]; /* HSI_COMMENT: Align to QREG */ +}; + +struct e5_tdif_task_context { + __le32 reserved[16]; +}; + +struct e5_rdif_task_context { + __le32 reserved[12]; +}; + +struct ystorm_nvmetcp_task_st_ctx { + struct ystorm_nvmetcp_task_state state; + struct ystorm_nvmetcp_task_rxmit_opt rxmit_opt; + struct nvmetcp_task_hdr_aligned pdu_hdr; +}; + +struct mstorm_nvmetcp_task_st_ctx { + struct scsi_cached_sges data_desc; + struct scsi_sgl_params sgl_params; + __le32 rem_task_size; + __le32 data_buffer_offset; + u8 task_type; + struct nvmetcp_dif_flags dif_flags; + __le16 dif_task_icid; + struct regpair reserved0; + __le32 expected_itt; + __le32 reserved1; +}; + +struct nvmetcp_reg1 { + __le32 reg1_map; +#define NVMETCP_REG1_NUM_SGES_MASK 0xF +#define NVMETCP_REG1_NUM_SGES_SHIFT 0 +#define NVMETCP_REG1_RESERVED1_MASK 0xFFFFFFF +#define NVMETCP_REG1_RESERVED1_SHIFT 4 +}; + +struct ustorm_nvmetcp_task_st_ctx { + __le32 rem_rcv_len; + __le32 exp_data_transfer_len; + __le32 exp_data_sn; + struct regpair reserved0; + struct nvmetcp_reg1 reg1; + u8 flags2; +#define USTORM_NVMETCP_TASK_ST_CTX_AHS_EXIST_MASK 0x1 +#define USTORM_NVMETCP_TASK_ST_CTX_AHS_EXIST_SHIFT 0 +#define USTORM_NVMETCP_TASK_ST_CTX_RESERVED1_MASK 0x7F +#define USTORM_NVMETCP_TASK_ST_CTX_RESERVED1_SHIFT 1 + struct nvmetcp_dif_flags dif_flags; + __le16 reserved3; + __le16 tqe_opaque[2]; + __le32 reserved5; + __le32 nvme_tcp_opaque_lo; + __le32 nvme_tcp_opaque_hi; + u8 task_type; + u8 error_flags; +#define USTORM_NVMETCP_TASK_ST_CTX_DATA_DIGEST_ERROR_MASK 0x1 +#define USTORM_NVMETCP_TASK_ST_CTX_DATA_DIGEST_ERROR_SHIFT 0 +#define USTORM_NVMETCP_TASK_ST_CTX_DATA_TRUNCATED_ERROR_MASK 0x1 +#define USTORM_NVMETCP_TASK_ST_CTX_DATA_TRUNCATED_ERROR_SHIFT 1 +#define USTORM_NVMETCP_TASK_ST_CTX_UNDER_RUN_ERROR_MASK 0x1 +#define USTORM_NVMETCP_TASK_ST_CTX_UNDER_RUN_ERROR_SHIFT 2 +#define USTORM_NVMETCP_TASK_ST_CTX_NVME_TCP_MASK 0x1 +#define USTORM_NVMETCP_TASK_ST_CTX_NVME_TCP_SHIFT 3 + u8 flags; +#define USTORM_NVMETCP_TASK_ST_CTX_CQE_WRITE_MASK 0x3 +#define USTORM_NVMETCP_TASK_ST_CTX_CQE_WRITE_SHIFT 0 +#define USTORM_NVMETCP_TASK_ST_CTX_LOCAL_COMP_MASK 0x1 +#define USTORM_NVMETCP_TASK_ST_CTX_LOCAL_COMP_SHIFT 2 +#define USTORM_NVMETCP_TASK_ST_CTX_Q0_R2TQE_WRITE_MASK 0x1 +#define USTORM_NVMETCP_TASK_ST_CTX_Q0_R2TQE_WRITE_SHIFT 3 +#define USTORM_NVMETCP_TASK_ST_CTX_TOTAL_DATA_ACKED_DONE_MASK 0x1 +#define USTORM_NVMETCP_TASK_ST_CTX_TOTAL_DATA_ACKED_DONE_SHIFT 4 +#define USTORM_NVMETCP_TASK_ST_CTX_HQ_SCANNED_DONE_MASK 0x1 +#define USTORM_NVMETCP_TASK_ST_CTX_HQ_SCANNED_DONE_SHIFT 5 +#define USTORM_NVMETCP_TASK_ST_CTX_R2T2RECV_DONE_MASK 0x1 +#define USTORM_NVMETCP_TASK_ST_CTX_R2T2RECV_DONE_SHIFT 6 + u8 cq_rss_number; +}; + +struct e5_ystorm_nvmetcp_task_ag_ctx { + u8 reserved /* cdu_validation */; + u8 byte1 /* state_and_core_id */; + __le16 word0 /* icid */; + u8 flags0; + u8 flags1; + u8 flags2; + u8 flags3; + __le32 TTT; + u8 byte2; + u8 byte3; + u8 byte4; + u8 e4_reserved7; +}; + +struct e5_mstorm_nvmetcp_task_ag_ctx { + u8 cdu_validation; + u8 byte1; + __le16 task_cid; + u8 flags0; +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CONNECTION_TYPE_MASK 0xF +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CONNECTION_TYPE_SHIFT 0 +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_EXIST_IN_QM0_MASK 0x1 +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_EXIST_IN_QM0_SHIFT 4 +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CONN_CLEAR_SQ_FLAG_MASK 0x1 +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CONN_CLEAR_SQ_FLAG_SHIFT 5 +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_VALID_MASK 0x1 +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_VALID_SHIFT 6 +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_FLAG_MASK 0x1 +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_FLAG_SHIFT 7 + u8 flags1; +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_CF_MASK 0x3 +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_CF_SHIFT 0 +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF1_MASK 0x3 +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF1_SHIFT 2 +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF2_MASK 0x3 +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF2_SHIFT 4 +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_CF_EN_MASK 0x1 +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_CF_EN_SHIFT 6 +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF1EN_MASK 0x1 +#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF1EN_SHIFT 7 + u8 flags2; + u8 flags3; + __le32 reg0; + u8 byte2; + u8 byte3; + u8 byte4; + u8 e4_reserved7; +}; + +struct e5_ustorm_nvmetcp_task_ag_ctx { + u8 reserved; + u8 state_and_core_id; + __le16 icid; + u8 flags0; +#define E5_USTORM_NVMETCP_TASK_AG_CTX_CONNECTION_TYPE_MASK 0xF +#define E5_USTORM_NVMETCP_TASK_AG_CTX_CONNECTION_TYPE_SHIFT 0 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_EXIST_IN_QM0_MASK 0x1 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_EXIST_IN_QM0_SHIFT 4 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_CONN_CLEAR_SQ_FLAG_MASK 0x1 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_CONN_CLEAR_SQ_FLAG_SHIFT 5 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_HQ_SCANNED_CF_MASK 0x3 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_HQ_SCANNED_CF_SHIFT 6 + u8 flags1; +#define E5_USTORM_NVMETCP_TASK_AG_CTX_RESERVED1_MASK 0x3 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_RESERVED1_SHIFT 0 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_R2T2RECV_MASK 0x3 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_R2T2RECV_SHIFT 2 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_CF3_MASK 0x3 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_CF3_SHIFT 4 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_CF_MASK 0x3 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_CF_SHIFT 6 + u8 flags2; +#define E5_USTORM_NVMETCP_TASK_AG_CTX_HQ_SCANNED_CF_EN_MASK 0x1 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_HQ_SCANNED_CF_EN_SHIFT 0 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_DISABLE_DATA_ACKED_MASK 0x1 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_DISABLE_DATA_ACKED_SHIFT 1 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_R2T2RECV_EN_MASK 0x1 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_R2T2RECV_EN_SHIFT 2 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_CF3EN_MASK 0x1 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_CF3EN_SHIFT 3 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_CF_EN_MASK 0x1 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_CF_EN_SHIFT 4 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_CMP_DATA_TOTAL_EXP_EN_MASK 0x1 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_CMP_DATA_TOTAL_EXP_EN_SHIFT 5 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_RULE1EN_MASK 0x1 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_RULE1EN_SHIFT 6 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_CMP_CONT_RCV_EXP_EN_MASK 0x1 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_CMP_CONT_RCV_EXP_EN_SHIFT 7 + u8 flags3; + u8 flags4; +#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED5_MASK 0x3 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED5_SHIFT 0 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED6_MASK 0x1 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED6_SHIFT 2 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED7_MASK 0x1 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED7_SHIFT 3 +#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_TYPE_MASK 0xF +#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_TYPE_SHIFT 4 + u8 byte2; + u8 byte3; + u8 e4_reserved8; + __le32 dif_err_intervals; + __le32 dif_error_1st_interval; + __le32 rcv_cont_len; + __le32 exp_cont_len; + __le32 total_data_acked; + __le32 exp_data_acked; + __le16 word1; + __le16 next_tid; + __le32 hdr_residual_count; + __le32 exp_r2t_sn; +}; + +struct e5_nvmetcp_task_context { + struct ystorm_nvmetcp_task_st_ctx ystorm_st_context; + struct e5_ystorm_nvmetcp_task_ag_ctx ystorm_ag_context; + struct regpair ystorm_ag_padding[2]; + struct e5_tdif_task_context tdif_context; + struct e5_mstorm_nvmetcp_task_ag_ctx mstorm_ag_context; + struct regpair mstorm_ag_padding[2]; + struct e5_ustorm_nvmetcp_task_ag_ctx ustorm_ag_context; + struct regpair ustorm_ag_padding[2]; + struct mstorm_nvmetcp_task_st_ctx mstorm_st_context; + struct regpair mstorm_st_padding[2]; + struct ustorm_nvmetcp_task_st_ctx ustorm_st_context; + struct regpair ustorm_st_padding[2]; + struct e5_rdif_task_context rdif_context; +}; + +/* NVMe TCP common header in network order */ +struct nvmetcp_common_hdr { + u8 pdo; + u8 hlen; + u8 flags; +#define NVMETCP_COMMON_HDR_HDGSTF_MASK 0x1 +#define NVMETCP_COMMON_HDR_HDGSTF_SHIFT 0 +#define NVMETCP_COMMON_HDR_DDGSTF_MASK 0x1 +#define NVMETCP_COMMON_HDR_DDGSTF_SHIFT 1 +#define NVMETCP_COMMON_HDR_LAST_PDU_MASK 0x1 +#define NVMETCP_COMMON_HDR_LAST_PDU_SHIFT 2 +#define NVMETCP_COMMON_HDR_SUCCESS_MASK 0x1 +#define NVMETCP_COMMON_HDR_SUCCESS_SHIFT 3 +#define NVMETCP_COMMON_HDR_RESERVED_MASK 0xF +#define NVMETCP_COMMON_HDR_RESERVED_SHIFT 4 + u8 pdu_type; + __le32 plen_swapped; +}; + +/* We don't need the entire 128 Bytes of the ICReq, hence passing only 16 + * Bytes to the FW in network order. + */ +struct nvmetcp_icreq_hdr_psh { + __le16 pfv; + u8 hpda; + u8 digest; +#define NVMETCP_ICREQ_HDR_PSH_16B_HDGST_EN_MASK 0x1 +#define NVMETCP_ICREQ_HDR_PSH_16B_HDGST_EN_SHIFT 0 +#define NVMETCP_ICREQ_HDR_PSH_16B_DDGST_EN_MASK 0x1 +#define NVMETCP_ICREQ_HDR_PSH_16B_DDGST_EN_SHIFT 1 +#define NVMETCP_ICREQ_HDR_PSH_16B_RESERVED1_MASK 0x3F +#define NVMETCP_ICREQ_HDR_PSH_16B_RESERVED1_SHIFT 2 + __le32 maxr2t; + u8 reserved[8]; +}; + +struct nvmetcp_cmd_capsule_hdr_psh { + __le32 raw_swapped[16]; +}; + +struct nvmetcp_cmd_capsule_hdr { + struct nvmetcp_common_hdr chdr; + struct nvmetcp_cmd_capsule_hdr_psh pshdr; +}; + +struct nvmetcp_data_hdr { + __le32 data[6]; +}; + +struct nvmetcp_h2c_hdr_psh { + __le16 ttag_swapped; + __le16 command_id_swapped; + __le32 data_offset_swapped; + __le32 data_length_swapped; + __le32 reserved1; +}; + +struct nvmetcp_h2c_hdr { + struct nvmetcp_common_hdr chdr; + struct nvmetcp_h2c_hdr_psh pshdr; +}; + +/* We don't need the entire 128 Bytes of the ICResp, hence passing only 16 + * Bytes to the FW in network order. + */ +struct nvmetcp_icresp_hdr_psh { + u8 digest; +#define NVMETCP_ICRESP_HDR_PSH_16B_HDGST_EN_MASK 0x1 +#define NVMETCP_ICRESP_HDR_PSH_16B_HDGST_EN_SHIFT 0 +#define NVMETCP_ICRESP_HDR_PSH_16B_DDGST_EN_MASK 0x1 +#define NVMETCP_ICRESP_HDR_PSH_16B_DDGST_EN_SHIFT 1 + u8 cpda; + __le16 pfv_swapped; + __le32 maxdata_swapped; + __le16 reserved2[4]; +}; + +struct nvmetcp_init_conn_req_hdr { + struct nvmetcp_common_hdr chdr; + struct nvmetcp_icreq_hdr_psh pshdr; +}; + +#endif /* __NVMETCP_COMMON__*/ diff --git a/include/linux/qed/qed_nvmetcp_if.h b/include/linux/qed/qed_nvmetcp_if.h index 686f924238e3..04e90dc42c12 100644 --- a/include/linux/qed/qed_nvmetcp_if.h +++ b/include/linux/qed/qed_nvmetcp_if.h @@ -5,6 +5,8 @@ #define _QED_NVMETCP_IF_H #include #include +#include +#include #define QED_NVMETCP_MAX_IO_SIZE 0x800000 @@ -73,6 +75,41 @@ struct qed_nvmetcp_cb_ops { struct qed_common_cb_ops common; }; +struct nvmetcp_sge { + struct regpair sge_addr; /* SGE address */ + __le32 sge_len; /* SGE length */ + __le32 reserved; +}; + +/* IO path HSI function SGL params */ +struct storage_sgl_task_params { + struct nvmetcp_sge *sgl; + struct regpair sgl_phys_addr; + u32 total_buffer_size; + u16 num_sges; + bool small_mid_sge; +}; + +/* IO path HSI function FW task context params */ +struct nvmetcp_task_params { + void *context; /* Output parameter - set/filled by the HSI function */ + struct nvmetcp_wqe *sqe; + u32 tx_io_size; /* in bytes (Without DIF, if exists) */ + u32 rx_io_size; /* in bytes (Without DIF, if exists) */ + u16 conn_icid; + u16 itid; + struct regpair opq; /* qedn_task_ctx address */ + u16 host_cccid; + u8 cq_rss_number; + bool send_write_incapsule; +}; + +/* IO path HSI function FW conn level input params */ + +struct nvmetcp_conn_params { + u32 max_burst_length; +}; + /** * struct qed_nvmetcp_ops - qed NVMeTCP operations. * @common: common operations pointer From patchwork Thu Apr 29 19:09:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shai Malin X-Patchwork-Id: 429658 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FCDFC4363C for ; Thu, 29 Apr 2021 19:18:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 378B76142A for ; Thu, 29 Apr 2021 19:18:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233922AbhD2TSP (ORCPT ); Thu, 29 Apr 2021 15:18:15 -0400 Received: from mx0b-0016f401.pphosted.com ([67.231.156.173]:23672 "EHLO mx0b-0016f401.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241759AbhD2TNM (ORCPT ); Thu, 29 Apr 2021 15:13:12 -0400 Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13TJ55Ne027326; Thu, 29 Apr 2021 12:10:16 -0700 Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com with ESMTP id 387rpnamph-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 29 Apr 2021 12:10:16 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 29 Apr 2021 12:10:14 -0700 Received: from lbtlvb-pcie154.il.qlogic.org (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 29 Apr 2021 12:10:11 -0700 From: Shai Malin To: , , , , , CC: =?utf-8?q?David_S_=2E_Miller_davem_=40_davemloft_=2E_net_=C2=A0--cc?= =?utf-8?q?=3DJakub_Kicinski?= , , , , , , , Arie Gershberg Subject: [RFC PATCH v4 09/27] nvme-fabrics: Move NVMF_ALLOWED_OPTS and NVMF_REQUIRED_OPTS definitions Date: Thu, 29 Apr 2021 22:09:08 +0300 Message-ID: <20210429190926.5086-10-smalin@marvell.com> X-Mailer: git-send-email 2.16.6 In-Reply-To: <20210429190926.5086-1-smalin@marvell.com> References: <20210429190926.5086-1-smalin@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: RuB4JunV1JJ_uzOF0w6mzHM2oCsuXurX X-Proofpoint-ORIG-GUID: RuB4JunV1JJ_uzOF0w6mzHM2oCsuXurX X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-04-29_10:2021-04-28,2021-04-29 signatures=0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Arie Gershberg Move NVMF_ALLOWED_OPTS and NVMF_REQUIRED_OPTS definitions to header file, so it can be used by transport modules. Acked-by: Igor Russkikh Signed-off-by: Arie Gershberg Signed-off-by: Prabhakar Kushwaha Signed-off-by: Omkar Kulkarni Signed-off-by: Michal Kalderon Signed-off-by: Ariel Elior Signed-off-by: Shai Malin --- drivers/nvme/host/fabrics.c | 7 ------- drivers/nvme/host/fabrics.h | 7 +++++++ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c index 604ab0e5a2ad..55d7125c8483 100644 --- a/drivers/nvme/host/fabrics.c +++ b/drivers/nvme/host/fabrics.c @@ -1001,13 +1001,6 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts) } EXPORT_SYMBOL_GPL(nvmf_free_options); -#define NVMF_REQUIRED_OPTS (NVMF_OPT_TRANSPORT | NVMF_OPT_NQN) -#define NVMF_ALLOWED_OPTS (NVMF_OPT_QUEUE_SIZE | NVMF_OPT_NR_IO_QUEUES | \ - NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \ - NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\ - NVMF_OPT_DISABLE_SQFLOW |\ - NVMF_OPT_FAIL_FAST_TMO) - static struct nvme_ctrl * nvmf_create_ctrl(struct device *dev, const char *buf) { diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h index 888b108d87a4..b7627e8dcaaf 100644 --- a/drivers/nvme/host/fabrics.h +++ b/drivers/nvme/host/fabrics.h @@ -68,6 +68,13 @@ enum { NVMF_OPT_FAIL_FAST_TMO = 1 << 20, }; +#define NVMF_REQUIRED_OPTS (NVMF_OPT_TRANSPORT | NVMF_OPT_NQN) +#define NVMF_ALLOWED_OPTS (NVMF_OPT_QUEUE_SIZE | NVMF_OPT_NR_IO_QUEUES | \ + NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \ + NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\ + NVMF_OPT_DISABLE_SQFLOW |\ + NVMF_OPT_FAIL_FAST_TMO) + /** * struct nvmf_ctrl_options - Used to hold the options specified * with the parsing opts enum. From patchwork Thu Apr 29 19:09:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shai Malin X-Patchwork-Id: 429656 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8915DC41602 for ; Thu, 29 Apr 2021 19:18:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 71A536144E for ; Thu, 29 Apr 2021 19:18:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233964AbhD2TS0 (ORCPT ); Thu, 29 Apr 2021 15:18:26 -0400 Received: from mx0b-0016f401.pphosted.com ([67.231.156.173]:18286 "EHLO mx0b-0016f401.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241783AbhD2TNR (ORCPT ); Thu, 29 Apr 2021 15:13:17 -0400 Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13TJ4xn2027101; Thu, 29 Apr 2021 12:10:21 -0700 Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com with ESMTP id 387rpnampx-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 29 Apr 2021 12:10:20 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 29 Apr 2021 12:10:18 -0700 Received: from lbtlvb-pcie154.il.qlogic.org (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 29 Apr 2021 12:10:15 -0700 From: Shai Malin To: , , , , , CC: =?utf-8?q?David_S_=2E_Miller_davem_=40_davemloft_=2E_net_=C2=A0--cc?= =?utf-8?q?=3DJakub_Kicinski?= , , , , , , , Dean Balandin Subject: [RFC PATCH v4 10/27] nvme-tcp-offload: Add device scan implementation Date: Thu, 29 Apr 2021 22:09:09 +0300 Message-ID: <20210429190926.5086-11-smalin@marvell.com> X-Mailer: git-send-email 2.16.6 In-Reply-To: <20210429190926.5086-1-smalin@marvell.com> References: <20210429190926.5086-1-smalin@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: Mfd2Uo2uZPkqm8aE005jduqU9ZnsvE47 X-Proofpoint-ORIG-GUID: Mfd2Uo2uZPkqm8aE005jduqU9ZnsvE47 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-04-29_10:2021-04-28,2021-04-29 signatures=0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Dean Balandin As part of create_ctrl(), it scans the registered devices and calls the claim_dev op on each of them, to find the first devices that matches the connection params. Once the correct devices is found (claim_dev returns true), we raise the refcnt of that device and return that device as the device to be used for ctrl currently being created. Acked-by: Igor Russkikh Signed-off-by: Dean Balandin Signed-off-by: Prabhakar Kushwaha Signed-off-by: Omkar Kulkarni Signed-off-by: Michal Kalderon Signed-off-by: Ariel Elior Signed-off-by: Shai Malin --- drivers/nvme/host/tcp-offload.c | 94 +++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c index 711232eba339..aa7cc239abf2 100644 --- a/drivers/nvme/host/tcp-offload.c +++ b/drivers/nvme/host/tcp-offload.c @@ -13,6 +13,11 @@ static LIST_HEAD(nvme_tcp_ofld_devices); static DECLARE_RWSEM(nvme_tcp_ofld_devices_rwsem); +static inline struct nvme_tcp_ofld_ctrl *to_tcp_ofld_ctrl(struct nvme_ctrl *nctrl) +{ + return container_of(nctrl, struct nvme_tcp_ofld_ctrl, nctrl); +} + /** * nvme_tcp_ofld_register_dev() - NVMeTCP Offload Library registration * function. @@ -98,6 +103,94 @@ void nvme_tcp_ofld_req_done(struct nvme_tcp_ofld_req *req, /* Placeholder - complete request with/without error */ } +struct nvme_tcp_ofld_dev * +nvme_tcp_ofld_lookup_dev(struct nvme_tcp_ofld_ctrl *ctrl) +{ + struct nvme_tcp_ofld_dev *dev; + + down_read(&nvme_tcp_ofld_devices_rwsem); + list_for_each_entry(dev, &nvme_tcp_ofld_devices, entry) { + if (dev->ops->claim_dev(dev, &ctrl->conn_params)) { + /* Increase driver refcnt */ + if (!try_module_get(dev->ops->module)) { + pr_err("try_module_get failed\n"); + dev = NULL; + } + + goto out; + } + } + + dev = NULL; +out: + up_read(&nvme_tcp_ofld_devices_rwsem); + + return dev; +} + +static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new) +{ + /* Placeholder - validates inputs and creates admin and IO queues */ + + return 0; +} + +static struct nvme_ctrl * +nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts) +{ + struct nvme_tcp_ofld_ctrl *ctrl; + struct nvme_tcp_ofld_dev *dev; + struct nvme_ctrl *nctrl; + int rc = 0; + + ctrl = kzalloc(sizeof(*ctrl), GFP_KERNEL); + if (!ctrl) + return ERR_PTR(-ENOMEM); + + nctrl = &ctrl->nctrl; + + /* Init nvme_tcp_ofld_ctrl and nvme_ctrl params based on received opts */ + + /* Find device that can reach the dest addr */ + dev = nvme_tcp_ofld_lookup_dev(ctrl); + if (!dev) { + pr_info("no device found for addr %s:%s.\n", + opts->traddr, opts->trsvcid); + rc = -EINVAL; + goto out_free_ctrl; + } + + ctrl->dev = dev; + + if (ctrl->dev->ops->max_hw_sectors) + nctrl->max_hw_sectors = ctrl->dev->ops->max_hw_sectors; + if (ctrl->dev->ops->max_segments) + nctrl->max_segments = ctrl->dev->ops->max_segments; + + /* Init queues */ + + /* Call nvme_init_ctrl */ + + rc = ctrl->dev->ops->setup_ctrl(ctrl, true); + if (rc) + goto out_module_put; + + rc = nvme_tcp_ofld_setup_ctrl(nctrl, true); + if (rc) + goto out_uninit_ctrl; + + return nctrl; + +out_uninit_ctrl: + ctrl->dev->ops->release_ctrl(ctrl); +out_module_put: + module_put(dev->ops->module); +out_free_ctrl: + kfree(ctrl); + + return ERR_PTR(rc); +} + static struct nvmf_transport_ops nvme_tcp_ofld_transport = { .name = "tcp_offload", .module = THIS_MODULE, @@ -107,6 +200,7 @@ static struct nvmf_transport_ops nvme_tcp_ofld_transport = { NVMF_OPT_RECONNECT_DELAY | NVMF_OPT_HDR_DIGEST | NVMF_OPT_DATA_DIGEST | NVMF_OPT_NR_POLL_QUEUES | NVMF_OPT_TOS, + .create_ctrl = nvme_tcp_ofld_create_ctrl, }; static int __init nvme_tcp_ofld_init_module(void) From patchwork Thu Apr 29 19:09:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shai Malin X-Patchwork-Id: 429652 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB05EC433ED for ; Thu, 29 Apr 2021 19:18:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8E80161450 for ; Thu, 29 Apr 2021 19:18:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233984AbhD2TS2 (ORCPT ); Thu, 29 Apr 2021 15:18:28 -0400 Received: from mx0b-0016f401.pphosted.com ([67.231.156.173]:1468 "EHLO mx0b-0016f401.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241855AbhD2TNc (ORCPT ); Thu, 29 Apr 2021 15:13:32 -0400 Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13TJ4xdd027122; Thu, 29 Apr 2021 12:10:34 -0700 Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com with ESMTP id 387rpnamqp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 29 Apr 2021 12:10:34 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 29 Apr 2021 12:10:32 -0700 Received: from lbtlvb-pcie154.il.qlogic.org (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 29 Apr 2021 12:10:28 -0700 From: Shai Malin To: , , , , , CC: =?utf-8?q?David_S_=2E_Miller_davem_=40_davemloft_=2E_net_=C2=A0--cc?= =?utf-8?q?=3DJakub_Kicinski?= , , , , , , , Dean Balandin Subject: [RFC PATCH v4 13/27] nvme-tcp-offload: Add queue level implementation Date: Thu, 29 Apr 2021 22:09:12 +0300 Message-ID: <20210429190926.5086-14-smalin@marvell.com> X-Mailer: git-send-email 2.16.6 In-Reply-To: <20210429190926.5086-1-smalin@marvell.com> References: <20210429190926.5086-1-smalin@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: yMoC7QJaQiHXCEKqquxRBvRT1SZp2LyG X-Proofpoint-ORIG-GUID: yMoC7QJaQiHXCEKqquxRBvRT1SZp2LyG X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-04-29_10:2021-04-28,2021-04-29 signatures=0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Dean Balandin In this patch we implement queue level functionality. The implementation is similar to the nvme-tcp module, the main difference being that we call the vendor specific create_queue op which creates the TCP connection, and NVMeTPC connection including icreq+icresp negotiation. Once create_queue returns successfully, we can move on to the fabrics connect. Acked-by: Igor Russkikh Signed-off-by: Dean Balandin Signed-off-by: Prabhakar Kushwaha Signed-off-by: Omkar Kulkarni Signed-off-by: Michal Kalderon Signed-off-by: Ariel Elior Signed-off-by: Shai Malin --- drivers/nvme/host/tcp-offload.c | 415 ++++++++++++++++++++++++++++++-- drivers/nvme/host/tcp-offload.h | 2 +- 2 files changed, 390 insertions(+), 27 deletions(-) diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c index 9082b11c133f..8ddce2257100 100644 --- a/drivers/nvme/host/tcp-offload.c +++ b/drivers/nvme/host/tcp-offload.c @@ -22,6 +22,11 @@ static inline struct nvme_tcp_ofld_ctrl *to_tcp_ofld_ctrl(struct nvme_ctrl *nctr return container_of(nctrl, struct nvme_tcp_ofld_ctrl, nctrl); } +static inline int nvme_tcp_ofld_qid(struct nvme_tcp_ofld_queue *queue) +{ + return queue - queue->ctrl->queues; +} + /** * nvme_tcp_ofld_register_dev() - NVMeTCP Offload Library registration * function. @@ -191,12 +196,94 @@ nvme_tcp_ofld_alloc_tagset(struct nvme_ctrl *nctrl, bool admin) return set; } +static void __nvme_tcp_ofld_stop_queue(struct nvme_tcp_ofld_queue *queue) +{ + queue->dev->ops->drain_queue(queue); + queue->dev->ops->destroy_queue(queue); +} + +static void nvme_tcp_ofld_stop_queue(struct nvme_ctrl *nctrl, int qid) +{ + struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl); + struct nvme_tcp_ofld_queue *queue = &ctrl->queues[qid]; + + if (!test_and_clear_bit(NVME_TCP_OFLD_Q_LIVE, &queue->flags)) + return; + + __nvme_tcp_ofld_stop_queue(queue); +} + +static void nvme_tcp_ofld_stop_io_queues(struct nvme_ctrl *ctrl) +{ + int i; + + for (i = 1; i < ctrl->queue_count; i++) + nvme_tcp_ofld_stop_queue(ctrl, i); +} + +static void nvme_tcp_ofld_free_queue(struct nvme_ctrl *nctrl, int qid) +{ + struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl); + struct nvme_tcp_ofld_queue *queue = &ctrl->queues[qid]; + + if (!test_and_clear_bit(NVME_TCP_OFLD_Q_ALLOCATED, &queue->flags)) + return; + + queue = &ctrl->queues[qid]; + queue->ctrl = NULL; + queue->dev = NULL; + queue->report_err = NULL; +} + +static void nvme_tcp_ofld_destroy_admin_queue(struct nvme_ctrl *nctrl, bool remove) +{ + nvme_tcp_ofld_stop_queue(nctrl, 0); + if (remove) { + blk_cleanup_queue(nctrl->admin_q); + blk_cleanup_queue(nctrl->fabrics_q); + blk_mq_free_tag_set(nctrl->admin_tagset); + } +} + +static int nvme_tcp_ofld_start_queue(struct nvme_ctrl *nctrl, int qid) +{ + struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl); + struct nvme_tcp_ofld_queue *queue = &ctrl->queues[qid]; + int rc; + + queue = &ctrl->queues[qid]; + if (qid) { + queue->cmnd_capsule_len = nctrl->ioccsz * 16; + rc = nvmf_connect_io_queue(nctrl, qid, false); + } else { + queue->cmnd_capsule_len = sizeof(struct nvme_command) + NVME_TCP_ADMIN_CCSZ; + rc = nvmf_connect_admin_queue(nctrl); + } + + if (!rc) { + set_bit(NVME_TCP_OFLD_Q_LIVE, &queue->flags); + } else { + if (test_bit(NVME_TCP_OFLD_Q_ALLOCATED, &queue->flags)) + __nvme_tcp_ofld_stop_queue(queue); + dev_err(nctrl->device, + "failed to connect queue: %d ret=%d\n", qid, rc); + } + + return rc; +} + static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl, bool new) { + struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl); + struct nvme_tcp_ofld_queue *queue = &ctrl->queues[0]; int rc; - /* Placeholder - alloc_admin_queue */ + rc = ctrl->dev->ops->create_queue(queue, 0, NVME_AQ_DEPTH); + if (rc) + return rc; + + set_bit(NVME_TCP_OFLD_Q_ALLOCATED, &queue->flags); if (new) { nctrl->admin_tagset = nvme_tcp_ofld_alloc_tagset(nctrl, true); @@ -221,7 +308,9 @@ static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl, } } - /* Placeholder - nvme_tcp_ofld_start_queue */ + rc = nvme_tcp_ofld_start_queue(nctrl, 0); + if (rc) + goto out_cleanup_queue; rc = nvme_enable_ctrl(nctrl); if (rc) @@ -238,11 +327,12 @@ static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl, out_quiesce_queue: blk_mq_quiesce_queue(nctrl->admin_q); blk_sync_queue(nctrl->admin_q); - out_stop_queue: - /* Placeholder - stop offload queue */ + nvme_tcp_ofld_stop_queue(nctrl, 0); nvme_cancel_admin_tagset(nctrl); - +out_cleanup_queue: + if (new) + blk_cleanup_queue(nctrl->admin_q); out_cleanup_fabrics_q: if (new) blk_cleanup_queue(nctrl->fabrics_q); @@ -250,7 +340,127 @@ static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl, if (new) blk_mq_free_tag_set(nctrl->admin_tagset); out_free_queue: - /* Placeholder - free admin queue */ + nvme_tcp_ofld_free_queue(nctrl, 0); + + return rc; +} + +static unsigned int nvme_tcp_ofld_nr_io_queues(struct nvme_ctrl *nctrl) +{ + unsigned int nr_io_queues; + + nr_io_queues = min(nctrl->opts->nr_io_queues, num_online_cpus()); + nr_io_queues += min(nctrl->opts->nr_write_queues, num_online_cpus()); + nr_io_queues += min(nctrl->opts->nr_poll_queues, num_online_cpus()); + + return nr_io_queues; +} + +static void +nvme_tcp_ofld_set_io_queues(struct nvme_ctrl *nctrl, unsigned int nr_io_queues) +{ + struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl); + struct nvmf_ctrl_options *opts = nctrl->opts; + + if (opts->nr_write_queues && opts->nr_io_queues < nr_io_queues) { + /* + * separate read/write queues + * hand out dedicated default queues only after we have + * sufficient read queues. + */ + ctrl->io_queues[HCTX_TYPE_READ] = opts->nr_io_queues; + nr_io_queues -= ctrl->io_queues[HCTX_TYPE_READ]; + ctrl->io_queues[HCTX_TYPE_DEFAULT] = + min(opts->nr_write_queues, nr_io_queues); + nr_io_queues -= ctrl->io_queues[HCTX_TYPE_DEFAULT]; + } else { + /* + * shared read/write queues + * either no write queues were requested, or we don't have + * sufficient queue count to have dedicated default queues. + */ + ctrl->io_queues[HCTX_TYPE_DEFAULT] = + min(opts->nr_io_queues, nr_io_queues); + nr_io_queues -= ctrl->io_queues[HCTX_TYPE_DEFAULT]; + } + + if (opts->nr_poll_queues && nr_io_queues) { + /* map dedicated poll queues only if we have queues left */ + ctrl->io_queues[HCTX_TYPE_POLL] = + min(opts->nr_poll_queues, nr_io_queues); + } +} + +static void +nvme_tcp_ofld_terminate_io_queues(struct nvme_ctrl *nctrl, int start_from) +{ + int i; + + /* admin-q will be ignored because of the loop condition */ + for (i = start_from; i >= 1; i--) + nvme_tcp_ofld_stop_queue(nctrl, i); +} + +static int nvme_tcp_ofld_create_io_queues(struct nvme_ctrl *nctrl) +{ + struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl); + int i, rc; + + for (i = 1; i < nctrl->queue_count; i++) { + rc = ctrl->dev->ops->create_queue(&ctrl->queues[i], + i, nctrl->sqsize + 1); + if (rc) + goto out_free_queues; + + set_bit(NVME_TCP_OFLD_Q_ALLOCATED, &ctrl->queues[i].flags); + } + + return 0; + +out_free_queues: + nvme_tcp_ofld_terminate_io_queues(nctrl, --i); + + return rc; +} + +static int nvme_tcp_ofld_alloc_io_queues(struct nvme_ctrl *nctrl) +{ + unsigned int nr_io_queues; + int rc; + + nr_io_queues = nvme_tcp_ofld_nr_io_queues(nctrl); + rc = nvme_set_queue_count(nctrl, &nr_io_queues); + if (rc) + return rc; + + nctrl->queue_count = nr_io_queues + 1; + if (nctrl->queue_count < 2) { + dev_err(nctrl->device, + "unable to set any I/O queues\n"); + + return -ENOMEM; + } + + dev_info(nctrl->device, "creating %d I/O queues.\n", nr_io_queues); + nvme_tcp_ofld_set_io_queues(nctrl, nr_io_queues); + + return nvme_tcp_ofld_create_io_queues(nctrl); +} + +static int nvme_tcp_ofld_start_io_queues(struct nvme_ctrl *nctrl) +{ + int i, rc = 0; + + for (i = 1; i < nctrl->queue_count; i++) { + rc = nvme_tcp_ofld_start_queue(nctrl, i); + if (rc) + goto terminate_queues; + } + + return 0; + +terminate_queues: + nvme_tcp_ofld_terminate_io_queues(nctrl, --i); return rc; } @@ -258,9 +468,10 @@ static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl, static int nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new) { - int rc; + int rc = nvme_tcp_ofld_alloc_io_queues(nctrl); - /* Placeholder - alloc_io_queues */ + if (rc) + return rc; if (new) { nctrl->tagset = nvme_tcp_ofld_alloc_tagset(nctrl, false); @@ -278,7 +489,9 @@ nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new) } } - /* Placeholder - start_io_queues */ + rc = nvme_tcp_ofld_start_io_queues(nctrl); + if (rc) + goto out_cleanup_connect_q; if (!new) { nvme_start_queues(nctrl); @@ -300,16 +513,16 @@ nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new) out_wait_freeze_timed_out: nvme_stop_queues(nctrl); nvme_sync_io_queues(nctrl); - - /* Placeholder - Stop IO queues */ - + nvme_tcp_ofld_stop_io_queues(nctrl); +out_cleanup_connect_q: + nvme_cancel_tagset(nctrl); if (new) blk_cleanup_queue(nctrl->connect_q); out_free_tag_set: if (new) blk_mq_free_tag_set(nctrl->tagset); out_free_io_queues: - /* Placeholder - free_io_queues */ + nvme_tcp_ofld_terminate_io_queues(nctrl, nctrl->queue_count); return rc; } @@ -336,6 +549,26 @@ static void nvme_tcp_ofld_reconnect_or_remove(struct nvme_ctrl *nctrl) } } +static int +nvme_tcp_ofld_init_admin_hctx(struct blk_mq_hw_ctx *hctx, void *data, + unsigned int hctx_idx) +{ + struct nvme_tcp_ofld_ctrl *ctrl = data; + + hctx->driver_data = &ctrl->queues[0]; + + return 0; +} + +static void nvme_tcp_ofld_destroy_io_queues(struct nvme_ctrl *nctrl, bool remove) +{ + nvme_tcp_ofld_stop_io_queues(nctrl); + if (remove) { + blk_cleanup_queue(nctrl->connect_q); + blk_mq_free_tag_set(nctrl->tagset); + } +} + static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new) { struct nvmf_ctrl_options *opts = nctrl->opts; @@ -387,9 +620,19 @@ static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new) return 0; destroy_io: - /* Placeholder - stop and destroy io queues*/ + if (nctrl->queue_count > 1) { + nvme_stop_queues(nctrl); + nvme_sync_io_queues(nctrl); + nvme_tcp_ofld_stop_io_queues(nctrl); + nvme_cancel_tagset(nctrl); + nvme_tcp_ofld_destroy_io_queues(nctrl, new); + } destroy_admin: - /* Placeholder - stop and destroy admin queue*/ + blk_mq_quiesce_queue(nctrl->admin_q); + blk_sync_queue(nctrl->admin_q); + nvme_tcp_ofld_stop_queue(nctrl, 0); + nvme_cancel_admin_tagset(nctrl); + nvme_tcp_ofld_destroy_admin_queue(nctrl, new); return rc; } @@ -410,6 +653,18 @@ nvme_tcp_ofld_check_dev_opts(struct nvmf_ctrl_options *opts, return 0; } +static void nvme_tcp_ofld_free_ctrl_queues(struct nvme_ctrl *nctrl) +{ + struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl); + int i; + + for (i = 0; i < nctrl->queue_count; ++i) + nvme_tcp_ofld_free_queue(nctrl, i); + + kfree(ctrl->queues); + ctrl->queues = NULL; +} + static void nvme_tcp_ofld_free_ctrl(struct nvme_ctrl *nctrl) { struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl); @@ -419,6 +674,7 @@ static void nvme_tcp_ofld_free_ctrl(struct nvme_ctrl *nctrl) goto free_ctrl; down_write(&nvme_tcp_ofld_ctrl_rwsem); + nvme_tcp_ofld_free_ctrl_queues(nctrl); ctrl->dev->ops->release_ctrl(ctrl); list_del(&ctrl->list); up_write(&nvme_tcp_ofld_ctrl_rwsem); @@ -436,15 +692,37 @@ static void nvme_tcp_ofld_submit_async_event(struct nvme_ctrl *arg) } static void -nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *ctrl, bool remove) +nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *nctrl, bool remove) { - /* Placeholder - teardown_admin_queue */ + blk_mq_quiesce_queue(nctrl->admin_q); + blk_sync_queue(nctrl->admin_q); + + nvme_tcp_ofld_stop_queue(nctrl, 0); + nvme_cancel_admin_tagset(nctrl); + + if (remove) + blk_mq_unquiesce_queue(nctrl->admin_q); + + nvme_tcp_ofld_destroy_admin_queue(nctrl, remove); } static void nvme_tcp_ofld_teardown_io_queues(struct nvme_ctrl *nctrl, bool remove) { - /* Placeholder - teardown_io_queues */ + if (nctrl->queue_count <= 1) + return; + + blk_mq_quiesce_queue(nctrl->admin_q); + nvme_start_freeze(nctrl); + nvme_stop_queues(nctrl); + nvme_sync_io_queues(nctrl); + nvme_tcp_ofld_stop_io_queues(nctrl); + nvme_cancel_tagset(nctrl); + + if (remove) + nvme_start_queues(nctrl); + + nvme_tcp_ofld_destroy_io_queues(nctrl, remove); } static void nvme_tcp_ofld_reconnect_ctrl_work(struct work_struct *work) @@ -572,6 +850,17 @@ nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set, return 0; } +inline size_t nvme_tcp_ofld_inline_data_size(struct nvme_tcp_ofld_queue *queue) +{ + return queue->cmnd_capsule_len - sizeof(struct nvme_command); +} +EXPORT_SYMBOL_GPL(nvme_tcp_ofld_inline_data_size); + +static void nvme_tcp_ofld_commit_rqs(struct blk_mq_hw_ctx *hctx) +{ + /* Call ops->commit_rqs */ +} + static blk_status_t nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx, const struct blk_mq_queue_data *bd) @@ -583,22 +872,96 @@ nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx, return BLK_STS_OK; } +static void +nvme_tcp_ofld_exit_request(struct blk_mq_tag_set *set, + struct request *rq, unsigned int hctx_idx) +{ + /* + * Nothing is allocated in nvme_tcp_ofld_init_request, + * hence empty. + */ +} + +static int +nvme_tcp_ofld_init_hctx(struct blk_mq_hw_ctx *hctx, void *data, + unsigned int hctx_idx) +{ + struct nvme_tcp_ofld_ctrl *ctrl = data; + + hctx->driver_data = &ctrl->queues[hctx_idx + 1]; + + return 0; +} + +static int nvme_tcp_ofld_map_queues(struct blk_mq_tag_set *set) +{ + struct nvme_tcp_ofld_ctrl *ctrl = set->driver_data; + struct nvmf_ctrl_options *opts = ctrl->nctrl.opts; + + if (opts->nr_write_queues && ctrl->io_queues[HCTX_TYPE_READ]) { + /* separate read/write queues */ + set->map[HCTX_TYPE_DEFAULT].nr_queues = + ctrl->io_queues[HCTX_TYPE_DEFAULT]; + set->map[HCTX_TYPE_DEFAULT].queue_offset = 0; + set->map[HCTX_TYPE_READ].nr_queues = + ctrl->io_queues[HCTX_TYPE_READ]; + set->map[HCTX_TYPE_READ].queue_offset = + ctrl->io_queues[HCTX_TYPE_DEFAULT]; + } else { + /* shared read/write queues */ + set->map[HCTX_TYPE_DEFAULT].nr_queues = + ctrl->io_queues[HCTX_TYPE_DEFAULT]; + set->map[HCTX_TYPE_DEFAULT].queue_offset = 0; + set->map[HCTX_TYPE_READ].nr_queues = + ctrl->io_queues[HCTX_TYPE_DEFAULT]; + set->map[HCTX_TYPE_READ].queue_offset = 0; + } + blk_mq_map_queues(&set->map[HCTX_TYPE_DEFAULT]); + blk_mq_map_queues(&set->map[HCTX_TYPE_READ]); + + if (opts->nr_poll_queues && ctrl->io_queues[HCTX_TYPE_POLL]) { + /* map dedicated poll queues only if we have queues left */ + set->map[HCTX_TYPE_POLL].nr_queues = + ctrl->io_queues[HCTX_TYPE_POLL]; + set->map[HCTX_TYPE_POLL].queue_offset = + ctrl->io_queues[HCTX_TYPE_DEFAULT] + + ctrl->io_queues[HCTX_TYPE_READ]; + blk_mq_map_queues(&set->map[HCTX_TYPE_POLL]); + } + + dev_info(ctrl->nctrl.device, + "mapped %d/%d/%d default/read/poll queues.\n", + ctrl->io_queues[HCTX_TYPE_DEFAULT], + ctrl->io_queues[HCTX_TYPE_READ], + ctrl->io_queues[HCTX_TYPE_POLL]); + + return 0; +} + +static int nvme_tcp_ofld_poll(struct blk_mq_hw_ctx *hctx) +{ + /* Placeholder - Implement polling mechanism */ + + return 0; +} + static struct blk_mq_ops nvme_tcp_ofld_mq_ops = { .queue_rq = nvme_tcp_ofld_queue_rq, + .commit_rqs = nvme_tcp_ofld_commit_rqs, + .complete = nvme_complete_rq, .init_request = nvme_tcp_ofld_init_request, - /* - * All additional ops will be also implemented and registered similar to - * tcp.c - */ + .exit_request = nvme_tcp_ofld_exit_request, + .init_hctx = nvme_tcp_ofld_init_hctx, + .map_queues = nvme_tcp_ofld_map_queues, + .poll = nvme_tcp_ofld_poll, }; static struct blk_mq_ops nvme_tcp_ofld_admin_mq_ops = { .queue_rq = nvme_tcp_ofld_queue_rq, + .complete = nvme_complete_rq, .init_request = nvme_tcp_ofld_init_request, - /* - * All additional ops will be also implemented and registered similar to - * tcp.c - */ + .exit_request = nvme_tcp_ofld_exit_request, + .init_hctx = nvme_tcp_ofld_init_admin_hctx, }; static const struct nvme_ctrl_ops nvme_tcp_ofld_ctrl_ops = { diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h index b23b1d7ea6fa..d82645fcf9da 100644 --- a/drivers/nvme/host/tcp-offload.h +++ b/drivers/nvme/host/tcp-offload.h @@ -105,7 +105,6 @@ struct nvme_tcp_ofld_ctrl { * Each entry in the array indicates the number of queues of * corresponding type. */ - u32 queue_type_mapping[HCTX_MAX_TYPES]; u32 io_queues[HCTX_MAX_TYPES]; /* Connectivity params */ @@ -205,3 +204,4 @@ struct nvme_tcp_ofld_ops { int nvme_tcp_ofld_register_dev(struct nvme_tcp_ofld_dev *dev); void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev); void nvme_tcp_ofld_error_recovery(struct nvme_ctrl *nctrl); +inline size_t nvme_tcp_ofld_inline_data_size(struct nvme_tcp_ofld_queue *queue); From patchwork Thu Apr 29 19:09:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shai Malin X-Patchwork-Id: 429654 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC777C05031 for ; Thu, 29 Apr 2021 19:18:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ACA5E6143E for ; Thu, 29 Apr 2021 19:18:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233997AbhD2TSa (ORCPT ); Thu, 29 Apr 2021 15:18:30 -0400 Received: from mx0a-0016f401.pphosted.com ([67.231.148.174]:39082 "EHLO mx0b-0016f401.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S241861AbhD2TNd (ORCPT ); Thu, 29 Apr 2021 15:13:33 -0400 Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13TJ6QM7019652; Thu, 29 Apr 2021 12:10:38 -0700 Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com with ESMTP id 387erumw2y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 29 Apr 2021 12:10:37 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 29 Apr 2021 12:10:36 -0700 Received: from lbtlvb-pcie154.il.qlogic.org (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 29 Apr 2021 12:10:32 -0700 From: Shai Malin To: , , , , , CC: =?utf-8?q?David_S_=2E_Miller_davem_=40_davemloft_=2E_net_=C2=A0--cc?= =?utf-8?q?=3DJakub_Kicinski?= , , , , , , , Dean Balandin Subject: [RFC PATCH v4 14/27] nvme-tcp-offload: Add IO level implementation Date: Thu, 29 Apr 2021 22:09:13 +0300 Message-ID: <20210429190926.5086-15-smalin@marvell.com> X-Mailer: git-send-email 2.16.6 In-Reply-To: <20210429190926.5086-1-smalin@marvell.com> References: <20210429190926.5086-1-smalin@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: vnu6k2WQV2-qQen6zzGq1CzDuMBJWUca X-Proofpoint-ORIG-GUID: vnu6k2WQV2-qQen6zzGq1CzDuMBJWUca X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-04-29_10:2021-04-28,2021-04-29 signatures=0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Dean Balandin In this patch, we present the IO level functionality. The nvme-tcp-offload shall work on the IO-level, meaning the nvme-tcp-offload ULP module shall pass the request to the nvme-tcp-offload vendor driver and shall expect for the request compilation. No additional handling is needed in between, this design will reduce the CPU utilization as we will describe below. The nvme-tcp-offload vendor driver shall register to nvme-tcp-offload ULP with the following IO-path ops: - init_req - send_req - in order to pass the request to the handling of the offload driver that shall pass it to the vendor specific device - poll_queue The vendor driver will manage the context from which the request will be executed and the request aggregations. Once the IO completed, the nvme-tcp-offload vendor driver shall call command.done() that shall invoke the nvme-tcp-offload ULP layer for completing the request. This patch also contains initial definition of nvme_tcp_ofld_queue_rq(). Acked-by: Igor Russkikh Signed-off-by: Dean Balandin Signed-off-by: Prabhakar Kushwaha Signed-off-by: Omkar Kulkarni Signed-off-by: Michal Kalderon Signed-off-by: Ariel Elior Signed-off-by: Shai Malin Reviewed-by: Hannes Reinecke --- drivers/nvme/host/tcp-offload.c | 95 ++++++++++++++++++++++++++++++--- 1 file changed, 87 insertions(+), 8 deletions(-) diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c index 8ddce2257100..0cdf5a432208 100644 --- a/drivers/nvme/host/tcp-offload.c +++ b/drivers/nvme/host/tcp-offload.c @@ -127,7 +127,10 @@ void nvme_tcp_ofld_req_done(struct nvme_tcp_ofld_req *req, union nvme_result *result, __le16 status) { - /* Placeholder - complete request with/without error */ + struct request *rq = blk_mq_rq_from_pdu(req); + + if (!nvme_try_complete_req(rq, cpu_to_le16(status << 1), *result)) + nvme_complete_rq(rq); } struct nvme_tcp_ofld_dev * @@ -686,6 +689,34 @@ static void nvme_tcp_ofld_free_ctrl(struct nvme_ctrl *nctrl) kfree(ctrl); } +static void nvme_tcp_ofld_set_sg_null(struct nvme_command *c) +{ + struct nvme_sgl_desc *sg = &c->common.dptr.sgl; + + sg->addr = 0; + sg->length = 0; + sg->type = (NVME_TRANSPORT_SGL_DATA_DESC << 4) | NVME_SGL_FMT_TRANSPORT_A; +} + +inline void nvme_tcp_ofld_set_sg_inline(struct nvme_tcp_ofld_queue *queue, + struct nvme_command *c, u32 data_len) +{ + struct nvme_sgl_desc *sg = &c->common.dptr.sgl; + + sg->addr = cpu_to_le64(queue->ctrl->nctrl.icdoff); + sg->length = cpu_to_le32(data_len); + sg->type = (NVME_SGL_FMT_DATA_DESC << 4) | NVME_SGL_FMT_OFFSET; +} + +void nvme_tcp_ofld_map_data(struct nvme_command *c, u32 data_len) +{ + struct nvme_sgl_desc *sg = &c->common.dptr.sgl; + + sg->addr = 0; + sg->length = cpu_to_le32(data_len); + sg->type = (NVME_TRANSPORT_SGL_DATA_DESC << 4) | NVME_SGL_FMT_TRANSPORT_A; +} + static void nvme_tcp_ofld_submit_async_event(struct nvme_ctrl *arg) { /* Placeholder - submit_async_event */ @@ -841,9 +872,11 @@ nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set, { struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(rq); struct nvme_tcp_ofld_ctrl *ctrl = set->driver_data; + int qid; - /* Placeholder - init request */ - + qid = (set == &ctrl->tag_set) ? hctx_idx + 1 : 0; + req->queue = &ctrl->queues[qid]; + nvme_req(rq)->ctrl = &ctrl->nctrl; req->done = nvme_tcp_ofld_req_done; ctrl->dev->ops->init_req(req); @@ -858,16 +891,60 @@ EXPORT_SYMBOL_GPL(nvme_tcp_ofld_inline_data_size); static void nvme_tcp_ofld_commit_rqs(struct blk_mq_hw_ctx *hctx) { - /* Call ops->commit_rqs */ + struct nvme_tcp_ofld_queue *queue = hctx->driver_data; + struct nvme_tcp_ofld_dev *dev = queue->dev; + struct nvme_tcp_ofld_ops *ops = dev->ops; + + ops->commit_rqs(queue); } static blk_status_t nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx, const struct blk_mq_queue_data *bd) { - /* Call nvme_setup_cmd(...) */ + struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(bd->rq); + struct nvme_tcp_ofld_queue *queue = hctx->driver_data; + struct nvme_tcp_ofld_ctrl *ctrl = queue->ctrl; + struct nvme_ns *ns = hctx->queue->queuedata; + struct nvme_tcp_ofld_dev *dev = queue->dev; + struct nvme_tcp_ofld_ops *ops = dev->ops; + struct nvme_command *nvme_cmd; + struct request *rq; + bool queue_ready; + u32 data_len; + int rc; + + queue_ready = test_bit(NVME_TCP_OFLD_Q_LIVE, &queue->flags); + + req->rq = bd->rq; + req->async = false; + rq = req->rq; + + if (!nvmf_check_ready(&ctrl->nctrl, req->rq, queue_ready)) + return nvmf_fail_nonready_command(&ctrl->nctrl, req->rq); + + rc = nvme_setup_cmd(ns, req->rq, &req->nvme_cmd); + if (unlikely(rc)) + return rc; - /* Call ops->send_req(...) */ + blk_mq_start_request(req->rq); + req->last = bd->last; + + nvme_cmd = &req->nvme_cmd; + nvme_cmd->common.flags |= NVME_CMD_SGL_METABUF; + + data_len = blk_rq_nr_phys_segments(rq) ? blk_rq_payload_bytes(rq) : 0; + if (!data_len) + nvme_tcp_ofld_set_sg_null(&req->nvme_cmd); + else if ((rq_data_dir(rq) == WRITE) && + data_len <= nvme_tcp_ofld_inline_data_size(queue)) + nvme_tcp_ofld_set_sg_inline(queue, nvme_cmd, data_len); + else + nvme_tcp_ofld_map_data(nvme_cmd, data_len); + + rc = ops->send_req(req); + if (unlikely(rc)) + return rc; return BLK_STS_OK; } @@ -940,9 +1017,11 @@ static int nvme_tcp_ofld_map_queues(struct blk_mq_tag_set *set) static int nvme_tcp_ofld_poll(struct blk_mq_hw_ctx *hctx) { - /* Placeholder - Implement polling mechanism */ + struct nvme_tcp_ofld_queue *queue = hctx->driver_data; + struct nvme_tcp_ofld_dev *dev = queue->dev; + struct nvme_tcp_ofld_ops *ops = dev->ops; - return 0; + return ops->poll_queue(queue); } static struct blk_mq_ops nvme_tcp_ofld_mq_ops = { From patchwork Thu Apr 29 19:09:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shai Malin X-Patchwork-Id: 429660 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F3D1C43603 for ; Thu, 29 Apr 2021 19:18:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 805DF6146D for ; Thu, 29 Apr 2021 19:18:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233639AbhD2TRz (ORCPT ); Thu, 29 Apr 2021 15:17:55 -0400 Received: from mx0a-0016f401.pphosted.com ([67.231.148.174]:34482 "EHLO mx0b-0016f401.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S239562AbhD2TLo (ORCPT ); Thu, 29 Apr 2021 15:11:44 -0400 Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13TJ6QMA019652; Thu, 29 Apr 2021 12:10:50 -0700 Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0a-0016f401.pphosted.com with ESMTP id 387erumw45-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 29 Apr 2021 12:10:50 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 29 Apr 2021 12:10:49 -0700 Received: from lbtlvb-pcie154.il.qlogic.org (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 29 Apr 2021 12:10:45 -0700 From: Shai Malin To: , , , , , CC: =?utf-8?q?David_S_=2E_Miller_davem_=40_davemloft_=2E_net_=C2=A0--cc?= =?utf-8?q?=3DJakub_Kicinski?= , , , , , , , Dean Balandin Subject: [RFC PATCH v4 17/27] qedn: Add qedn probe Date: Thu, 29 Apr 2021 22:09:16 +0300 Message-ID: <20210429190926.5086-18-smalin@marvell.com> X-Mailer: git-send-email 2.16.6 In-Reply-To: <20210429190926.5086-1-smalin@marvell.com> References: <20210429190926.5086-1-smalin@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: g1PNubqQ7GSJ8LU2bhX4l93vhmq6_n98 X-Proofpoint-ORIG-GUID: g1PNubqQ7GSJ8LU2bhX4l93vhmq6_n98 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-04-29_10:2021-04-28,2021-04-29 signatures=0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This patch introduces the functionality of loading and unloading physical function. qedn_probe() loads the offload device PF(physical function), and initialize the HW and the FW with the PF parameters using the HW ops->qed_nvmetcp_ops, which are similar to other "qed_*_ops" which are used by the qede, qedr, qedf and qedi device drivers. qedn_remove() unloads the offload device PF, re-initialize the HW and the FW with the PF parameters. The struct qedn_ctx is per PF container for PF-specific attributes and resources. Acked-by: Igor Russkikh Signed-off-by: Dean Balandin Signed-off-by: Prabhakar Kushwaha Signed-off-by: Omkar Kulkarni Signed-off-by: Michal Kalderon Signed-off-by: Ariel Elior Signed-off-by: Shai Malin Reviewed-by: Hannes Reinecke --- drivers/nvme/hw/Kconfig | 1 + drivers/nvme/hw/qedn/qedn.h | 49 ++++++++ drivers/nvme/hw/qedn/qedn_main.c | 191 ++++++++++++++++++++++++++++++- 3 files changed, 236 insertions(+), 5 deletions(-) diff --git a/drivers/nvme/hw/Kconfig b/drivers/nvme/hw/Kconfig index 374f1f9dbd3d..91b1bd6f07d8 100644 --- a/drivers/nvme/hw/Kconfig +++ b/drivers/nvme/hw/Kconfig @@ -2,6 +2,7 @@ config NVME_QEDN tristate "Marvell NVM Express over Fabrics TCP offload" depends on NVME_TCP_OFFLOAD + select QED_NVMETCP help This enables the Marvell NVMe TCP offload support (qedn). diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h index bcd0748a10fd..c1ac17eabcb7 100644 --- a/drivers/nvme/hw/qedn/qedn.h +++ b/drivers/nvme/hw/qedn/qedn.h @@ -6,14 +6,63 @@ #ifndef _QEDN_H_ #define _QEDN_H_ +#include +#include + /* Driver includes */ #include "../../host/tcp-offload.h" +#define QEDN_MAJOR_VERSION 8 +#define QEDN_MINOR_VERSION 62 +#define QEDN_REVISION_VERSION 10 +#define QEDN_ENGINEERING_VERSION 0 +#define DRV_MODULE_VERSION __stringify(QEDE_MAJOR_VERSION) "." \ + __stringify(QEDE_MINOR_VERSION) "." \ + __stringify(QEDE_REVISION_VERSION) "." \ + __stringify(QEDE_ENGINEERING_VERSION) + #define QEDN_MODULE_NAME "qedn" +#define QEDN_MAX_TASKS_PER_PF (16 * 1024) +#define QEDN_MAX_CONNS_PER_PF (4 * 1024) +#define QEDN_FW_CQ_SIZE (4 * 1024) +#define QEDN_PROTO_CQ_PROD_IDX 0 +#define QEDN_NVMETCP_NUM_FW_CONN_QUEUE_PAGES 2 + +enum qedn_state { + QEDN_STATE_CORE_PROBED = 0, + QEDN_STATE_CORE_OPEN, + QEDN_STATE_GL_PF_LIST_ADDED, + QEDN_STATE_MFW_STATE, + QEDN_STATE_REGISTERED_OFFLOAD_DEV, + QEDN_STATE_MODULE_REMOVE_ONGOING, +}; + struct qedn_ctx { struct pci_dev *pdev; + struct qed_dev *cdev; + struct qed_dev_nvmetcp_info dev_info; struct nvme_tcp_ofld_dev qedn_ofld_dev; + struct qed_pf_params pf_params; + + /* Global PF list entry */ + struct list_head gl_pf_entry; + + /* Accessed with atomic bit ops, used with enum qedn_state */ + unsigned long state; + + /* Fast path queues */ + u8 num_fw_cqs; +}; + +struct qedn_global { + struct list_head qedn_pf_list; + + /* Host mode */ + struct list_head ctrl_list; + + /* Mutex for accessing the global struct */ + struct mutex glb_mutex; }; #endif /* _QEDN_H_ */ diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c index 31d6d86d6eb7..e3e8e3676b79 100644 --- a/drivers/nvme/hw/qedn/qedn_main.c +++ b/drivers/nvme/hw/qedn/qedn_main.c @@ -14,6 +14,10 @@ #define CHIP_NUM_AHP_NVMETCP 0x8194 +const struct qed_nvmetcp_ops *qed_ops; + +/* Global context instance */ +struct qedn_global qedn_glb; static struct pci_device_id qedn_pci_tbl[] = { { PCI_VDEVICE(QLOGIC, CHIP_NUM_AHP_NVMETCP), 0 }, {0, 0}, @@ -99,12 +103,132 @@ static struct nvme_tcp_ofld_ops qedn_ofld_ops = { .commit_rqs = qedn_commit_rqs, }; +static inline void qedn_init_pf_struct(struct qedn_ctx *qedn) +{ + /* Placeholder - Initialize qedn fields */ +} + +static inline void +qedn_init_core_probe_params(struct qed_probe_params *probe_params) +{ + memset(probe_params, 0, sizeof(*probe_params)); + probe_params->protocol = QED_PROTOCOL_NVMETCP; + probe_params->is_vf = false; + probe_params->recov_in_prog = 0; +} + +static inline int qedn_core_probe(struct qedn_ctx *qedn) +{ + struct qed_probe_params probe_params; + int rc = 0; + + qedn_init_core_probe_params(&probe_params); + pr_info("Starting QED probe\n"); + qedn->cdev = qed_ops->common->probe(qedn->pdev, &probe_params); + if (!qedn->cdev) { + rc = -ENODEV; + pr_err("QED probe failed\n"); + } + + return rc; +} + +static void qedn_add_pf_to_gl_list(struct qedn_ctx *qedn) +{ + mutex_lock(&qedn_glb.glb_mutex); + list_add_tail(&qedn->gl_pf_entry, &qedn_glb.qedn_pf_list); + mutex_unlock(&qedn_glb.glb_mutex); +} + +static void qedn_remove_pf_from_gl_list(struct qedn_ctx *qedn) +{ + mutex_lock(&qedn_glb.glb_mutex); + list_del_init(&qedn->gl_pf_entry); + mutex_unlock(&qedn_glb.glb_mutex); +} + +static int qedn_set_nvmetcp_pf_param(struct qedn_ctx *qedn) +{ + u32 fw_conn_queue_pages = QEDN_NVMETCP_NUM_FW_CONN_QUEUE_PAGES; + struct qed_nvmetcp_pf_params *pf_params; + + pf_params = &qedn->pf_params.nvmetcp_pf_params; + memset(pf_params, 0, sizeof(*pf_params)); + qedn->num_fw_cqs = min_t(u8, qedn->dev_info.num_cqs, num_online_cpus()); + + pf_params->num_cons = QEDN_MAX_CONNS_PER_PF; + pf_params->num_tasks = QEDN_MAX_TASKS_PER_PF; + + /* Placeholder - Initialize function level queues */ + + /* Placeholder - Initialize TCP params */ + + /* Queues */ + pf_params->num_sq_pages_in_ring = fw_conn_queue_pages; + pf_params->num_r2tq_pages_in_ring = fw_conn_queue_pages; + pf_params->num_uhq_pages_in_ring = fw_conn_queue_pages; + pf_params->num_queues = qedn->num_fw_cqs; + pf_params->cq_num_entries = QEDN_FW_CQ_SIZE; + + /* the CQ SB pi */ + pf_params->gl_rq_pi = QEDN_PROTO_CQ_PROD_IDX; + + return 0; +} + +static inline int qedn_slowpath_start(struct qedn_ctx *qedn) +{ + struct qed_slowpath_params sp_params = {}; + int rc = 0; + + /* Start the Slowpath-process */ + sp_params.int_mode = QED_INT_MODE_MSIX; + sp_params.drv_major = QEDN_MAJOR_VERSION; + sp_params.drv_minor = QEDN_MINOR_VERSION; + sp_params.drv_rev = QEDN_REVISION_VERSION; + sp_params.drv_eng = QEDN_ENGINEERING_VERSION; + strscpy(sp_params.name, "qedn NVMeTCP", QED_DRV_VER_STR_SIZE); + rc = qed_ops->common->slowpath_start(qedn->cdev, &sp_params); + if (rc) + pr_err("Cannot start slowpath\n"); + + return rc; +} + static void __qedn_remove(struct pci_dev *pdev) { struct qedn_ctx *qedn = pci_get_drvdata(pdev); + int rc; + + pr_notice("qedn remove started: abs PF id=%u\n", + qedn->dev_info.common.abs_pf_id); + + if (test_and_set_bit(QEDN_STATE_MODULE_REMOVE_ONGOING, &qedn->state)) { + pr_err("Remove already ongoing\n"); + + return; + } + + if (test_and_clear_bit(QEDN_STATE_REGISTERED_OFFLOAD_DEV, &qedn->state)) + nvme_tcp_ofld_unregister_dev(&qedn->qedn_ofld_dev); + + if (test_and_clear_bit(QEDN_STATE_GL_PF_LIST_ADDED, &qedn->state)) + qedn_remove_pf_from_gl_list(qedn); + else + pr_err("Failed to remove from global PF list\n"); + + if (test_and_clear_bit(QEDN_STATE_MFW_STATE, &qedn->state)) { + rc = qed_ops->common->update_drv_state(qedn->cdev, false); + if (rc) + pr_err("Failed to send drv state to MFW\n"); + } + + if (test_and_clear_bit(QEDN_STATE_CORE_OPEN, &qedn->state)) + qed_ops->common->slowpath_stop(qedn->cdev); + + if (test_and_clear_bit(QEDN_STATE_CORE_PROBED, &qedn->state)) + qed_ops->common->remove(qedn->cdev); - pr_notice("Starting qedn_remove\n"); - nvme_tcp_ofld_unregister_dev(&qedn->qedn_ofld_dev); kfree(qedn); pr_notice("Ending qedn_remove successfully\n"); } @@ -144,15 +268,55 @@ static int __qedn_probe(struct pci_dev *pdev) if (!qedn) return -ENODEV; + qedn_init_pf_struct(qedn); + + /* QED probe */ + rc = qedn_core_probe(qedn); + if (rc) + goto exit_probe_and_release_mem; + + set_bit(QEDN_STATE_CORE_PROBED, &qedn->state); + + rc = qed_ops->fill_dev_info(qedn->cdev, &qedn->dev_info); + if (rc) { + pr_err("fill_dev_info failed\n"); + goto exit_probe_and_release_mem; + } + + qedn_add_pf_to_gl_list(qedn); + set_bit(QEDN_STATE_GL_PF_LIST_ADDED, &qedn->state); + + rc = qedn_set_nvmetcp_pf_param(qedn); + if (rc) + goto exit_probe_and_release_mem; + + qed_ops->common->update_pf_params(qedn->cdev, &qedn->pf_params); + rc = qedn_slowpath_start(qedn); + if (rc) + goto exit_probe_and_release_mem; + + set_bit(QEDN_STATE_CORE_OPEN, &qedn->state); + + rc = qed_ops->common->update_drv_state(qedn->cdev, true); + if (rc) { + pr_err("Failed to send drv state to MFW\n"); + goto exit_probe_and_release_mem; + } + + set_bit(QEDN_STATE_MFW_STATE, &qedn->state); + qedn->qedn_ofld_dev.ops = &qedn_ofld_ops; INIT_LIST_HEAD(&qedn->qedn_ofld_dev.entry); rc = nvme_tcp_ofld_register_dev(&qedn->qedn_ofld_dev); if (rc) - goto release_qedn; + goto exit_probe_and_release_mem; + + set_bit(QEDN_STATE_REGISTERED_OFFLOAD_DEV, &qedn->state); return 0; -release_qedn: - kfree(qedn); +exit_probe_and_release_mem: + __qedn_remove(pdev); + pr_err("probe ended with error\n"); return rc; } @@ -170,10 +334,26 @@ static struct pci_driver qedn_pci_driver = { .shutdown = qedn_shutdown, }; +static inline void qedn_init_global_contxt(void) +{ + INIT_LIST_HEAD(&qedn_glb.qedn_pf_list); + INIT_LIST_HEAD(&qedn_glb.ctrl_list); + mutex_init(&qedn_glb.glb_mutex); +} + static int __init qedn_init(void) { int rc; + qedn_init_global_contxt(); + + qed_ops = qed_get_nvmetcp_ops(); + if (!qed_ops) { + pr_err("Failed to get QED NVMeTCP ops\n"); + + return -EINVAL; + } + rc = pci_register_driver(&qedn_pci_driver); if (rc) { pr_err("Failed to register pci driver\n"); @@ -189,6 +369,7 @@ static int __init qedn_init(void) static void __exit qedn_cleanup(void) { pci_unregister_driver(&qedn_pci_driver); + qed_put_nvmetcp_ops(); pr_notice("Unloading qedn ended\n"); } From patchwork Thu Apr 29 19:09:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shai Malin X-Patchwork-Id: 429653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26405C41515 for ; Thu, 29 Apr 2021 19:19:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 146756143A for ; Thu, 29 Apr 2021 19:19:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234054AbhD2TSf (ORCPT ); Thu, 29 Apr 2021 15:18:35 -0400 Received: from mx0b-0016f401.pphosted.com ([67.231.156.173]:11568 "EHLO mx0b-0016f401.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241984AbhD2TNu (ORCPT ); Thu, 29 Apr 2021 15:13:50 -0400 Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13TJ5GSr027367; Thu, 29 Apr 2021 12:10:55 -0700 Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com with ESMTP id 387rpnamsv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 29 Apr 2021 12:10:55 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 29 Apr 2021 12:10:53 -0700 Received: from lbtlvb-pcie154.il.qlogic.org (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 29 Apr 2021 12:10:50 -0700 From: Shai Malin To: , , , , , CC: =?utf-8?q?David_S_=2E_Miller_davem_=40_davemloft_=2E_net_=C2=A0--cc?= =?utf-8?q?=3DJakub_Kicinski?= , , , , , , , Nikolay Assa Subject: [RFC PATCH v4 18/27] qedn: Add qedn_claim_dev API support Date: Thu, 29 Apr 2021 22:09:17 +0300 Message-ID: <20210429190926.5086-19-smalin@marvell.com> X-Mailer: git-send-email 2.16.6 In-Reply-To: <20210429190926.5086-1-smalin@marvell.com> References: <20210429190926.5086-1-smalin@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: JNay_2PNs0-8k6GGjFuyCvYOPGSTGelf X-Proofpoint-ORIG-GUID: JNay_2PNs0-8k6GGjFuyCvYOPGSTGelf X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-04-29_10:2021-04-28,2021-04-29 signatures=0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Nikolay Assa This patch introduces the qedn_claim_dev() network service which the offload device (qedn) is using through the paired net-device (qede). qedn_claim_dev() returns true if the IP addr(IPv4 or IPv6) of the target server is reachable via the net-device which is paired with the offloaded device. Acked-by: Igor Russkikh Signed-off-by: Nikolay Assa Signed-off-by: Prabhakar Kushwaha Signed-off-by: Omkar Kulkarni Signed-off-by: Michal Kalderon Signed-off-by: Ariel Elior Signed-off-by: Shai Malin Reviewed-by: Hannes Reinecke --- drivers/nvme/hw/qedn/qedn.h | 4 +++ drivers/nvme/hw/qedn/qedn_main.c | 42 ++++++++++++++++++++++++++++++-- 2 files changed, 44 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h index c1ac17eabcb7..7efe2366eb7c 100644 --- a/drivers/nvme/hw/qedn/qedn.h +++ b/drivers/nvme/hw/qedn/qedn.h @@ -8,6 +8,10 @@ #include #include +#include +#include +#include +#include /* Driver includes */ #include "../../host/tcp-offload.h" diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c index e3e8e3676b79..52007d35622d 100644 --- a/drivers/nvme/hw/qedn/qedn_main.c +++ b/drivers/nvme/hw/qedn/qedn_main.c @@ -27,9 +27,47 @@ static int qedn_claim_dev(struct nvme_tcp_ofld_dev *dev, struct nvme_tcp_ofld_ctrl_con_params *conn_params) { - /* Placeholder - qedn_claim_dev */ + struct pci_dev *qede_pdev = NULL; + struct net_device *ndev = NULL; + u16 vlan_id = 0; + int rc = 0; - return 0; + /* qedn utilizes host network stack through paired qede device for + * non-offload traffic. First we verify there is valid route to remote + * peer. + */ + if (conn_params->remote_ip_addr.ss_family == AF_INET) { + rc = qed_route_ipv4(&conn_params->local_ip_addr, + &conn_params->remote_ip_addr, + &conn_params->remote_mac_addr, + &ndev); + } else if (conn_params->remote_ip_addr.ss_family == AF_INET6) { + rc = qed_route_ipv6(&conn_params->local_ip_addr, + &conn_params->remote_ip_addr, + &conn_params->remote_mac_addr, + &ndev); + } else { + pr_err("address family %d not supported\n", + conn_params->remote_ip_addr.ss_family); + + return false; + } + + if (rc) + return false; + + qed_vlan_get_ndev(&ndev, &vlan_id); + conn_params->vlan_id = vlan_id; + + /* route found through ndev - validate this is qede*/ + qede_pdev = qed_validate_ndev(ndev); + if (!qede_pdev) + return false; + + dev->qede_pdev = qede_pdev; + dev->ndev = ndev; + + return true; } static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid, From patchwork Thu Apr 29 19:09:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shai Malin X-Patchwork-Id: 429657 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5419C4363E for ; Thu, 29 Apr 2021 19:18:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7EDF36145E for ; Thu, 29 Apr 2021 19:18:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233851AbhD2TSC (ORCPT ); Thu, 29 Apr 2021 15:18:02 -0400 Received: from mx0a-0016f401.pphosted.com ([67.231.148.174]:9234 "EHLO mx0b-0016f401.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S241662AbhD2TM7 (ORCPT ); Thu, 29 Apr 2021 15:12:59 -0400 Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13TJ6FOY019575; Thu, 29 Apr 2021 12:11:03 -0700 Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0a-0016f401.pphosted.com with ESMTP id 387erumw58-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 29 Apr 2021 12:11:03 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 29 Apr 2021 12:11:01 -0700 Received: from lbtlvb-pcie154.il.qlogic.org (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 29 Apr 2021 12:10:58 -0700 From: Shai Malin To: , , , , , CC: =?utf-8?q?David_S_=2E_Miller_davem_=40_davemloft_=2E_net_=C2=A0--cc?= =?utf-8?q?=3DJakub_Kicinski?= , , , , , , Subject: [RFC PATCH v4 20/27] qedn: Add connection-level slowpath functionality Date: Thu, 29 Apr 2021 22:09:19 +0300 Message-ID: <20210429190926.5086-21-smalin@marvell.com> X-Mailer: git-send-email 2.16.6 In-Reply-To: <20210429190926.5086-1-smalin@marvell.com> References: <20210429190926.5086-1-smalin@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: rblbXDVKIaInsOo1k3shlaYQ3oaVrYZS X-Proofpoint-ORIG-GUID: rblbXDVKIaInsOo1k3shlaYQ3oaVrYZS X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-04-29_10:2021-04-28,2021-04-29 signatures=0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Prabhakar Kushwaha This patch will present the connection (queue) level slowpath implementation relevant for create_queue flow. The internal implementation: - Add per controller slowpath workqeueue via pre_setup_ctrl - qedn_main.c: Includes qedn's implementation of the create_queue op. - qedn_conn.c will include main slowpath connection level functions, including: 1. Per-queue resources allocation. 2. Creating a new connection. 3. Offloading the connection to the FW for TCP handshake. 4. Destroy of a connection. 5. Support of delete and free controller. 6. TCP port management via qed_fetch_tcp_port, qed_return_tcp_port Acked-by: Igor Russkikh Signed-off-by: Prabhakar Kushwaha Signed-off-by: Omkar Kulkarni Signed-off-by: Michal Kalderon Signed-off-by: Ariel Elior Signed-off-by: Shai Malin Reviewed-by: Hannes Reinecke --- drivers/nvme/hw/qedn/Makefile | 5 +- drivers/nvme/hw/qedn/qedn.h | 173 ++++++++++- drivers/nvme/hw/qedn/qedn_conn.c | 508 +++++++++++++++++++++++++++++++ drivers/nvme/hw/qedn/qedn_main.c | 208 ++++++++++++- 4 files changed, 883 insertions(+), 11 deletions(-) create mode 100644 drivers/nvme/hw/qedn/qedn_conn.c diff --git a/drivers/nvme/hw/qedn/Makefile b/drivers/nvme/hw/qedn/Makefile index cb169bbaae18..d8b343afcd16 100644 --- a/drivers/nvme/hw/qedn/Makefile +++ b/drivers/nvme/hw/qedn/Makefile @@ -1,5 +1,4 @@ # SPDX-License-Identifier: GPL-2.0-only -obj-$(CONFIG_NVME_QEDN) := qedn.o - -qedn-y := qedn_main.o \ No newline at end of file +obj-$(CONFIG_NVME_QEDN) += qedn.o +qedn-y := qedn_main.o qedn_conn.o diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h index 5d4d04d144e4..ed0d43163da2 100644 --- a/drivers/nvme/hw/qedn/qedn.h +++ b/drivers/nvme/hw/qedn/qedn.h @@ -6,6 +6,7 @@ #ifndef _QEDN_H_ #define _QEDN_H_ +#include #include #include #include @@ -37,8 +38,41 @@ #define QEDN_IRQ_NAME_LEN 24 #define QEDN_IRQ_NO_FLAGS 0 -/* TCP defines */ +/* Destroy connection defines */ +#define QEDN_NON_ABORTIVE_TERMINATION 0 +#define QEDN_ABORTIVE_TERMINATION 1 + +/* + * TCP offload stack default configurations and defines. + * Future enhancements will allow controlling the configurable + * parameters via devlink. + */ #define QEDN_TCP_RTO_DEFAULT 280 +#define QEDN_TCP_ECN_EN 0 +#define QEDN_TCP_TS_EN 0 +#define QEDN_TCP_DA_EN 0 +#define QEDN_TCP_KA_EN 0 +#define QEDN_TCP_TOS 0 +#define QEDN_TCP_TTL 0xfe +#define QEDN_TCP_FLOW_LABEL 0 +#define QEDN_TCP_KA_TIMEOUT 7200000 +#define QEDN_TCP_KA_INTERVAL 10000 +#define QEDN_TCP_KA_MAX_PROBE_COUNT 10 +#define QEDN_TCP_MAX_RT_TIME 1200 +#define QEDN_TCP_MAX_CWND 4 +#define QEDN_TCP_RCV_WND_SCALE 2 +#define QEDN_TCP_TS_OPTION_LEN 12 + +/* SP Work queue defines */ +#define QEDN_SP_WORKQUEUE "qedn_sp_wq" +#define QEDN_SP_WORKQUEUE_MAX_ACTIVE 1 + +#define QEDN_HOST_MAX_SQ_SIZE (512) +#define QEDN_SQ_SIZE (2 * QEDN_HOST_MAX_SQ_SIZE) + +/* Timeouts and delay constants */ +#define QEDN_WAIT_CON_ESTABLSH_TMO 10000 /* 10 seconds */ +#define QEDN_RLS_CONS_TMO 5000 /* 5 sec */ enum qedn_state { QEDN_STATE_CORE_PROBED = 0, @@ -78,6 +112,12 @@ struct qedn_ctx { /* Accessed with atomic bit ops, used with enum qedn_state */ unsigned long state; + u8 local_mac_addr[ETH_ALEN]; + u16 mtu; + + /* Connections */ + DECLARE_HASHTABLE(conn_ctx_hash, 16); + /* Fast path queues */ u8 num_fw_cqs; struct qedn_fp_queue *fp_q_arr; @@ -85,6 +125,126 @@ struct qedn_ctx { dma_addr_t fw_cq_array_phy; /* Physical address of fw_cq_array_virt */ }; +struct qedn_endpoint { + /* FW Params */ + struct qed_chain fw_sq_chain; + void __iomem *p_doorbell; + + /* TCP Params */ + __be32 dst_addr[4]; /* In network order */ + __be32 src_addr[4]; /* In network order */ + u16 src_port; + u16 dst_port; + u16 vlan_id; + u8 src_mac[ETH_ALEN]; + u8 dst_mac[ETH_ALEN]; + u8 ip_type; +}; + +enum sp_work_agg_action { + CREATE_CONNECTION = 0, + SEND_ICREQ, + HANDLE_ICRESP, + DESTROY_CONNECTION, +}; + +enum qedn_ctrl_agg_state { + QEDN_CTRL_SET_TO_OFLD_CTRL = 0, /* CTRL set to OFLD_CTRL */ + QEDN_STATE_SP_WORK_THREAD_SET, /* slow patch WQ was created*/ + LLH_FILTER, /* LLH filter added */ + QEDN_RECOVERY, + ADMINQ_CONNECTED, /* At least one connection has attempted offload */ + ERR_FLOW, +}; + +enum qedn_ctrl_sp_wq_state { + QEDN_CTRL_STATE_UNINITIALIZED = 0, + QEDN_CTRL_STATE_FREE_CTRL, + QEDN_CTRL_STATE_CTRL_ERR, +}; + +/* Any change to this enum requires an update of qedn_conn_state_str */ +enum qedn_conn_state { + CONN_STATE_CONN_IDLE = 0, + CONN_STATE_CREATE_CONNECTION, + CONN_STATE_WAIT_FOR_CONNECT_DONE, + CONN_STATE_OFFLOAD_COMPLETE, + CONN_STATE_WAIT_FOR_UPDATE_EQE, + CONN_STATE_WAIT_FOR_IC_COMP, + CONN_STATE_NVMETCP_CONN_ESTABLISHED, + CONN_STATE_DESTROY_CONNECTION, + CONN_STATE_WAIT_FOR_DESTROY_DONE, + CONN_STATE_DESTROY_COMPLETE +}; + +struct qedn_ctrl { + struct list_head glb_entry; + struct list_head pf_entry; + + struct qedn_ctx *qedn; + struct nvme_tcp_ofld_queue *queue; + struct nvme_tcp_ofld_ctrl *ctrl; + + struct workqueue_struct *sp_wq; + enum qedn_ctrl_sp_wq_state sp_wq_state; + + struct work_struct sp_wq_entry; + + struct qedn_llh_filter *llh_filter; + + unsigned long agg_state; + + atomic_t host_num_active_conns; +}; + +/* Connection level struct */ +struct qedn_conn_ctx { + struct qedn_ctx *qedn; + struct nvme_tcp_ofld_queue *queue; + struct nvme_tcp_ofld_ctrl *ctrl; + u32 conn_handle; + u32 fw_cid; + + atomic_t est_conn_indicator; + atomic_t destroy_conn_indicator; + wait_queue_head_t conn_waitq; + + struct work_struct sp_wq_entry; + + /* Connection aggregative state. + * Can have different states independently. + */ + unsigned long agg_work_action; + + struct hlist_node hash_node; + struct nvmetcp_host_cccid_itid_entry *host_cccid_itid; + dma_addr_t host_cccid_itid_phy_addr; + struct qedn_endpoint ep; + int abrt_flag; + + /* Connection resources - turned on to indicate what resource was + * allocated, to that it can later be released. + */ + unsigned long resrc_state; + + /* Connection state */ + spinlock_t conn_state_lock; + enum qedn_conn_state state; + + size_t sq_depth; + + /* "dummy" socket */ + struct socket *sock; +}; + +enum qedn_conn_resources_state { + QEDN_CONN_RESRC_FW_SQ, + QEDN_CONN_RESRC_ACQUIRE_CONN, + QEDN_CONN_RESRC_CCCID_ITID_MAP, + QEDN_CONN_RESRC_TCP_PORT, + QEDN_CONN_RESRC_MAX = 64 +}; + struct qedn_global { struct list_head qedn_pf_list; @@ -95,4 +255,15 @@ struct qedn_global { struct mutex glb_mutex; }; +struct qedn_conn_ctx *qedn_get_conn_hash(struct qedn_ctx *qedn, u16 icid); +int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data); +void qedn_sp_wq_handler(struct work_struct *work); +void qedn_set_sp_wa(struct qedn_conn_ctx *conn_ctx, u32 bit); +void qedn_clr_sp_wa(struct qedn_conn_ctx *conn_ctx, u32 bit); +int qedn_initialize_endpoint(struct qedn_endpoint *ep, u8 *local_mac_addr, + struct nvme_tcp_ofld_ctrl_con_params *conn_params); +int qedn_wait_for_conn_est(struct qedn_conn_ctx *conn_ctx); +int qedn_set_con_state(struct qedn_conn_ctx *conn_ctx, enum qedn_conn_state new_state); +void qedn_terminate_connection(struct qedn_conn_ctx *conn_ctx, int abrt_flag); + #endif /* _QEDN_H_ */ diff --git a/drivers/nvme/hw/qedn/qedn_conn.c b/drivers/nvme/hw/qedn/qedn_conn.c new file mode 100644 index 000000000000..9bfc0a5f0cdb --- /dev/null +++ b/drivers/nvme/hw/qedn/qedn_conn.c @@ -0,0 +1,508 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright 2021 Marvell. All rights reserved. + */ + +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + + /* Kernel includes */ +#include +#include +#include + +/* Driver includes */ +#include "qedn.h" + +extern const struct qed_nvmetcp_ops *qed_ops; + +static const char * const qedn_conn_state_str[] = { + "CONN_IDLE", + "CREATE_CONNECTION", + "WAIT_FOR_CONNECT_DONE", + "OFFLOAD_COMPLETE", + "WAIT_FOR_UPDATE_EQE", + "WAIT_FOR_IC_COMP", + "NVMETCP_CONN_ESTABLISHED", + "DESTROY_CONNECTION", + "WAIT_FOR_DESTROY_DONE", + "DESTROY_COMPLETE", + NULL +}; + +int qedn_set_con_state(struct qedn_conn_ctx *conn_ctx, enum qedn_conn_state new_state) +{ + spin_lock_bh(&conn_ctx->conn_state_lock); + conn_ctx->state = new_state; + spin_unlock_bh(&conn_ctx->conn_state_lock); + + return 0; +} + +static void qedn_return_tcp_port(struct qedn_conn_ctx *conn_ctx) +{ + if (conn_ctx->sock && conn_ctx->sock->sk) { + qed_return_tcp_port(conn_ctx->sock); + conn_ctx->sock = NULL; + } + + conn_ctx->ep.src_port = 0; +} + +int qedn_wait_for_conn_est(struct qedn_conn_ctx *conn_ctx) +{ + int wrc, rc; + + wrc = wait_event_interruptible_timeout(conn_ctx->conn_waitq, + atomic_read(&conn_ctx->est_conn_indicator) > 0, + msecs_to_jiffies(QEDN_WAIT_CON_ESTABLSH_TMO)); + atomic_set(&conn_ctx->est_conn_indicator, 0); + if (!wrc || + conn_ctx->state != CONN_STATE_NVMETCP_CONN_ESTABLISHED) { + rc = -ETIMEDOUT; + + /* If error was prior or during offload, conn_ctx was released. + * If the error was after offload sync has completed, we need to + * terminate the connection ourselves. + */ + if (conn_ctx && + conn_ctx->state >= CONN_STATE_WAIT_FOR_CONNECT_DONE && + conn_ctx->state <= CONN_STATE_NVMETCP_CONN_ESTABLISHED) + qedn_terminate_connection(conn_ctx, + QEDN_ABORTIVE_TERMINATION); + } else { + rc = 0; + } + + return rc; +} + +int qedn_fill_ep_addr4(struct qedn_endpoint *ep, + struct nvme_tcp_ofld_ctrl_con_params *conn_params) +{ + struct sockaddr_in *raddr = (struct sockaddr_in *)&conn_params->remote_ip_addr; + struct sockaddr_in *laddr = (struct sockaddr_in *)&conn_params->local_ip_addr; + + ep->ip_type = TCP_IPV4; + ep->src_port = laddr->sin_port; + ep->dst_port = ntohs(raddr->sin_port); + + ep->src_addr[0] = laddr->sin_addr.s_addr; + ep->dst_addr[0] = raddr->sin_addr.s_addr; + + return 0; +} + +int qedn_fill_ep_addr6(struct qedn_endpoint *ep, + struct nvme_tcp_ofld_ctrl_con_params *conn_params) +{ + struct sockaddr_in6 *raddr6 = (struct sockaddr_in6 *)&conn_params->remote_ip_addr; + struct sockaddr_in6 *laddr6 = (struct sockaddr_in6 *)&conn_params->local_ip_addr; + int i; + + ep->ip_type = TCP_IPV6; + ep->src_port = laddr6->sin6_port; + ep->dst_port = ntohs(raddr6->sin6_port); + + for (i = 0; i < 4; i++) { + ep->src_addr[i] = laddr6->sin6_addr.in6_u.u6_addr32[i]; + ep->dst_addr[i] = raddr6->sin6_addr.in6_u.u6_addr32[i]; + } + + return 0; +} + +int qedn_initialize_endpoint(struct qedn_endpoint *ep, u8 *local_mac_addr, + struct nvme_tcp_ofld_ctrl_con_params *conn_params) +{ + ether_addr_copy(ep->dst_mac, conn_params->remote_mac_addr.sa_data); + ether_addr_copy(ep->src_mac, local_mac_addr); + ep->vlan_id = conn_params->vlan_id; + if (conn_params->remote_ip_addr.ss_family == AF_INET) + qedn_fill_ep_addr4(ep, conn_params); + else + qedn_fill_ep_addr6(ep, conn_params); + + return -1; +} + +static void qedn_release_conn_ctx(struct qedn_conn_ctx *conn_ctx) +{ + struct qedn_ctx *qedn = conn_ctx->qedn; + int rc = 0; + + if (test_bit(QEDN_CONN_RESRC_FW_SQ, &conn_ctx->resrc_state)) { + qed_ops->common->chain_free(qedn->cdev, + &conn_ctx->ep.fw_sq_chain); + clear_bit(QEDN_CONN_RESRC_FW_SQ, &conn_ctx->resrc_state); + } + + if (test_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state)) { + hash_del(&conn_ctx->hash_node); + rc = qed_ops->release_conn(qedn->cdev, conn_ctx->conn_handle); + if (rc) + pr_warn("Release_conn returned with an error %u\n", + rc); + + clear_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state); + } + + if (test_bit(QEDN_CONN_RESRC_CCCID_ITID_MAP, &conn_ctx->resrc_state)) { + dma_free_coherent(&qedn->pdev->dev, + conn_ctx->sq_depth * + sizeof(struct nvmetcp_host_cccid_itid_entry), + conn_ctx->host_cccid_itid, + conn_ctx->host_cccid_itid_phy_addr); + clear_bit(QEDN_CONN_RESRC_CCCID_ITID_MAP, + &conn_ctx->resrc_state); + } + + if (test_bit(QEDN_CONN_RESRC_TCP_PORT, &conn_ctx->resrc_state)) { + qedn_return_tcp_port(conn_ctx); + clear_bit(QEDN_CONN_RESRC_TCP_PORT, + &conn_ctx->resrc_state); + } + + if (conn_ctx->resrc_state) + pr_err("Conn resources state isn't 0 as expected 0x%lx\n", + conn_ctx->resrc_state); + + atomic_inc(&conn_ctx->destroy_conn_indicator); + qedn_set_con_state(conn_ctx, CONN_STATE_DESTROY_COMPLETE); + wake_up_interruptible(&conn_ctx->conn_waitq); +} + +static int qedn_alloc_fw_sq(struct qedn_ctx *qedn, + struct qedn_endpoint *ep) +{ + struct qed_chain_init_params params = { + .mode = QED_CHAIN_MODE_PBL, + .intended_use = QED_CHAIN_USE_TO_PRODUCE, + .cnt_type = QED_CHAIN_CNT_TYPE_U16, + .num_elems = QEDN_SQ_SIZE, + .elem_size = sizeof(struct nvmetcp_wqe), + }; + int rc; + + rc = qed_ops->common->chain_alloc(qedn->cdev, + &ep->fw_sq_chain, + ¶ms); + if (rc) { + pr_err("Failed to allocate SQ chain\n"); + + return -ENOMEM; + } + + return 0; +} + +static int qedn_nvmetcp_offload_conn(struct qedn_conn_ctx *conn_ctx) +{ + struct qed_nvmetcp_params_offload offld_prms = { 0 }; + struct qedn_endpoint *qedn_ep = &conn_ctx->ep; + struct qedn_ctx *qedn = conn_ctx->qedn; + u8 ts_hdr_size = 0; + u32 hdr_size; + int rc, i; + + ether_addr_copy(offld_prms.src.mac, qedn_ep->src_mac); + ether_addr_copy(offld_prms.dst.mac, qedn_ep->dst_mac); + offld_prms.vlan_id = qedn_ep->vlan_id; + offld_prms.ecn_en = QEDN_TCP_ECN_EN; + offld_prms.timestamp_en = QEDN_TCP_TS_EN; + offld_prms.delayed_ack_en = QEDN_TCP_DA_EN; + offld_prms.tcp_keep_alive_en = QEDN_TCP_KA_EN; + offld_prms.ip_version = qedn_ep->ip_type; + + offld_prms.src.ip[0] = ntohl(qedn_ep->src_addr[0]); + offld_prms.dst.ip[0] = ntohl(qedn_ep->dst_addr[0]); + if (qedn_ep->ip_type == TCP_IPV6) { + for (i = 1; i < 4; i++) { + offld_prms.src.ip[i] = ntohl(qedn_ep->src_addr[i]); + offld_prms.dst.ip[i] = ntohl(qedn_ep->dst_addr[i]); + } + } + + offld_prms.ttl = QEDN_TCP_TTL; + offld_prms.tos_or_tc = QEDN_TCP_TOS; + offld_prms.dst.port = qedn_ep->dst_port; + offld_prms.src.port = qedn_ep->src_port; + offld_prms.nvmetcp_cccid_itid_table_addr = + conn_ctx->host_cccid_itid_phy_addr; + offld_prms.nvmetcp_cccid_max_range = conn_ctx->sq_depth; + + /* Calculate MSS */ + if (offld_prms.timestamp_en) + ts_hdr_size = QEDN_TCP_TS_OPTION_LEN; + + hdr_size = qedn_ep->ip_type == TCP_IPV4 ? + sizeof(struct iphdr) : sizeof(struct ipv6hdr); + hdr_size += sizeof(struct tcphdr) + ts_hdr_size; + + offld_prms.mss = qedn->mtu - hdr_size; + offld_prms.rcv_wnd_scale = QEDN_TCP_RCV_WND_SCALE; + offld_prms.cwnd = QEDN_TCP_MAX_CWND * offld_prms.mss; + offld_prms.ka_max_probe_cnt = QEDN_TCP_KA_MAX_PROBE_COUNT; + offld_prms.ka_timeout = QEDN_TCP_KA_TIMEOUT; + offld_prms.ka_interval = QEDN_TCP_KA_INTERVAL; + offld_prms.max_rt_time = QEDN_TCP_MAX_RT_TIME; + offld_prms.sq_pbl_addr = + (u64)qed_chain_get_pbl_phys(&qedn_ep->fw_sq_chain); + + rc = qed_ops->offload_conn(qedn->cdev, + conn_ctx->conn_handle, + &offld_prms); + if (rc) + pr_err("offload_conn returned with an error\n"); + + return rc; +} + +static int qedn_fetch_tcp_port(struct qedn_conn_ctx *conn_ctx) +{ + struct nvme_tcp_ofld_ctrl *ctrl; + struct qedn_ctrl *qctrl; + int rc = 0; + + ctrl = conn_ctx->ctrl; + qctrl = (struct qedn_ctrl *)ctrl->private_data; + + rc = qed_fetch_tcp_port(ctrl->conn_params.local_ip_addr, + &conn_ctx->sock, &conn_ctx->ep.src_port); + + return rc; +} + +static void qedn_decouple_conn(struct qedn_conn_ctx *conn_ctx) +{ + struct nvme_tcp_ofld_queue *queue; + + queue = conn_ctx->queue; + queue->private_data = NULL; +} + +void qedn_terminate_connection(struct qedn_conn_ctx *conn_ctx, int abrt_flag) +{ + struct qedn_ctrl *qctrl; + + if (!conn_ctx) + return; + + qctrl = (struct qedn_ctrl *)conn_ctx->ctrl->private_data; + + if (test_and_set_bit(DESTROY_CONNECTION, &conn_ctx->agg_work_action)) + return; + + qedn_set_con_state(conn_ctx, CONN_STATE_DESTROY_CONNECTION); + conn_ctx->abrt_flag = abrt_flag; + + queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry); +} + +/* Slowpath EQ Callback */ +int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data) +{ + struct nvmetcp_connect_done_results *eqe_connect_done; + struct nvmetcp_eqe_data *eqe_data; + struct nvme_tcp_ofld_ctrl *ctrl; + struct qedn_conn_ctx *conn_ctx; + struct qedn_ctrl *qctrl; + struct qedn_ctx *qedn; + u16 icid; + int rc; + + if (!context || !event_ring_data) { + pr_err("Recv event with ctx NULL\n"); + + return -EINVAL; + } + + qedn = (struct qedn_ctx *)context; + + if (fw_event_code != NVMETCP_EVENT_TYPE_ASYN_CONNECT_COMPLETE) { + eqe_data = (struct nvmetcp_eqe_data *)event_ring_data; + icid = le16_to_cpu(eqe_data->icid); + pr_err("EQE Info: icid=0x%x, conn_id=0x%x, err-code=0x%x, err-pdu-opcode-reserved=0x%x\n", + eqe_data->icid, eqe_data->conn_id, + eqe_data->error_code, + eqe_data->error_pdu_opcode_reserved); + } else { + eqe_connect_done = + (struct nvmetcp_connect_done_results *)event_ring_data; + icid = le16_to_cpu(eqe_connect_done->icid); + } + + conn_ctx = qedn_get_conn_hash(qedn, icid); + if (!conn_ctx) { + pr_err("Connection with icid=0x%x doesn't exist in conn list\n", icid); + + return -EINVAL; + } + + ctrl = conn_ctx->ctrl; + qctrl = (struct qedn_ctrl *)ctrl->private_data; + + switch (fw_event_code) { + case NVMETCP_EVENT_TYPE_ASYN_CONNECT_COMPLETE: + if (conn_ctx->state != CONN_STATE_WAIT_FOR_CONNECT_DONE) { + pr_err("CID=0x%x - ASYN_CONNECT_COMPLETE: Unexpected connection state %u\n", + conn_ctx->fw_cid, conn_ctx->state); + } else { + rc = qedn_set_con_state(conn_ctx, CONN_STATE_OFFLOAD_COMPLETE); + + if (rc) + return rc; + + /* Placeholder - for ICReq flow */ + } + + break; + case NVMETCP_EVENT_TYPE_ASYN_TERMINATE_DONE: + if (conn_ctx->state != CONN_STATE_WAIT_FOR_DESTROY_DONE) + pr_err("CID=0x%x - ASYN_TERMINATE_DONE: Unexpected connection state %u\n", + conn_ctx->fw_cid, conn_ctx->state); + else + queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry); + + break; + default: + pr_err("CID=0x%x - Recv Unknown Event %u\n", conn_ctx->fw_cid, fw_event_code); + break; + } + + return 0; +} + +static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx) +{ + struct qedn_ctx *qedn = conn_ctx->qedn; + size_t dma_size; + int rc; + + rc = qedn_alloc_fw_sq(qedn, &conn_ctx->ep); + if (rc) { + pr_err("Failed to allocate FW SQ\n"); + goto rel_conn; + } + + set_bit(QEDN_CONN_RESRC_FW_SQ, &conn_ctx->resrc_state); + rc = qed_ops->acquire_conn(qedn->cdev, + &conn_ctx->conn_handle, + &conn_ctx->fw_cid, + &conn_ctx->ep.p_doorbell); + if (rc) { + pr_err("Couldn't acquire connection\n"); + goto rel_conn; + } + + hash_add(qedn->conn_ctx_hash, &conn_ctx->hash_node, + conn_ctx->conn_handle); + set_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state); + + /* Placeholder - Allocate task resources and initialize fields */ + + rc = qedn_fetch_tcp_port(conn_ctx); + if (rc) + goto rel_conn; + + set_bit(QEDN_CONN_RESRC_TCP_PORT, &conn_ctx->resrc_state); + dma_size = conn_ctx->sq_depth * + sizeof(struct nvmetcp_host_cccid_itid_entry); + conn_ctx->host_cccid_itid = + dma_alloc_coherent(&qedn->pdev->dev, + dma_size, + &conn_ctx->host_cccid_itid_phy_addr, + GFP_ATOMIC); + if (!conn_ctx->host_cccid_itid) { + pr_err("CCCID-iTID Map allocation failed\n"); + goto rel_conn; + } + + memset(conn_ctx->host_cccid_itid, 0xFF, dma_size); + set_bit(QEDN_CONN_RESRC_CCCID_ITID_MAP, &conn_ctx->resrc_state); + rc = qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_CONNECT_DONE); + if (rc) + goto rel_conn; + + rc = qedn_nvmetcp_offload_conn(conn_ctx); + if (rc) { + pr_err("Offload error: CID=0x%x\n", conn_ctx->fw_cid); + goto rel_conn; + } + + return 0; + +rel_conn: + pr_err("qedn create queue ended with ERROR\n"); + qedn_release_conn_ctx(conn_ctx); + + return -EINVAL; +} + +void qedn_destroy_connection(struct qedn_conn_ctx *conn_ctx) +{ + struct qedn_ctx *qedn = conn_ctx->qedn; + int rc; + + qedn_decouple_conn(conn_ctx); + + if (qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_DESTROY_DONE)) + return; + + /* Placeholder - task cleanup */ + + rc = qed_ops->destroy_conn(qedn->cdev, conn_ctx->conn_handle, + conn_ctx->abrt_flag); + if (rc) + pr_warn("destroy_conn failed - rc %u\n", rc); +} + +void qedn_sp_wq_handler(struct work_struct *work) +{ + struct qedn_conn_ctx *conn_ctx; + struct qedn_ctx *qedn; + int rc; + + conn_ctx = container_of(work, struct qedn_conn_ctx, sp_wq_entry); + qedn = conn_ctx->qedn; + + if (conn_ctx->state == CONN_STATE_DESTROY_COMPLETE) { + pr_err("Connection already released!\n"); + + return; + } + + if (conn_ctx->state == CONN_STATE_WAIT_FOR_DESTROY_DONE) { + qedn_release_conn_ctx(conn_ctx); + + return; + } + + qedn = conn_ctx->qedn; + if (test_bit(DESTROY_CONNECTION, &conn_ctx->agg_work_action)) { + qedn_destroy_connection(conn_ctx); + + return; + } + + if (test_bit(CREATE_CONNECTION, &conn_ctx->agg_work_action)) { + qedn_clr_sp_wa(conn_ctx, CREATE_CONNECTION); + rc = qedn_prep_and_offload_queue(conn_ctx); + if (rc) { + pr_err("Error in queue prepare & firmware offload\n"); + + return; + } + } +} + +/* Clear connection aggregative slowpath work action */ +void qedn_clr_sp_wa(struct qedn_conn_ctx *conn_ctx, u32 bit) +{ + clear_bit(bit, &conn_ctx->agg_work_action); +} + +/* Set connection aggregative slowpath work action */ +void qedn_set_sp_wa(struct qedn_conn_ctx *conn_ctx, u32 bit) +{ + set_bit(bit, &conn_ctx->agg_work_action); +} diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c index 0135a1f490da..70beb84b9793 100644 --- a/drivers/nvme/hw/qedn/qedn_main.c +++ b/drivers/nvme/hw/qedn/qedn_main.c @@ -23,6 +23,38 @@ static struct pci_device_id qedn_pci_tbl[] = { {0, 0}, }; +static bool qedn_matches_qede(struct qedn_ctx *qedn, struct pci_dev *qede_pdev) +{ + struct pci_dev *qedn_pdev = qedn->pdev; + + return (qede_pdev->bus->number == qedn_pdev->bus->number && + PCI_SLOT(qede_pdev->devfn) == PCI_SLOT(qedn_pdev->devfn) && + PCI_FUNC(qede_pdev->devfn) == qedn->dev_info.port_id); +} + +static struct qedn_ctx *qedn_get_pf_from_pdev(struct pci_dev *qede_pdev) +{ + struct list_head *pf_list = NULL; + struct qedn_ctx *qedn = NULL; + int rc; + + pf_list = &qedn_glb.qedn_pf_list; + if (!pf_list) { + pr_err("Failed fetching pf list for nvmet_add_port\n"); + rc = -EFAULT; + goto pf_list_err; + } + + list_for_each_entry(qedn, pf_list, gl_pf_entry) { + if (qedn_matches_qede(qedn, qede_pdev)) + return qedn; + } + +pf_list_err: + + return NULL; +} + static int qedn_claim_dev(struct nvme_tcp_ofld_dev *dev, struct nvme_tcp_ofld_ctrl_con_params *conn_params) @@ -70,22 +102,167 @@ qedn_claim_dev(struct nvme_tcp_ofld_dev *dev, return true; } -static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid, - size_t q_size) +static int qedn_setup_ctrl(struct nvme_tcp_ofld_ctrl *ctrl, bool new) { - /* Placeholder - qedn_create_queue */ + struct nvme_tcp_ofld_dev *dev = ctrl->dev; + struct qedn_ctrl *qctrl = NULL; + struct qedn_ctx *qedn = NULL; + int rc = 0; + + if (new) { + qctrl = kzalloc(sizeof(*qctrl), GFP_KERNEL); + if (!qctrl) + return -ENOMEM; + + ctrl->private_data = (void *)qctrl; + set_bit(QEDN_CTRL_SET_TO_OFLD_CTRL, &qctrl->agg_state); + + qctrl->sp_wq = alloc_workqueue(QEDN_SP_WORKQUEUE, WQ_MEM_RECLAIM, + QEDN_SP_WORKQUEUE_MAX_ACTIVE); + if (!qctrl->sp_wq) { + rc = -ENODEV; + pr_err("Unable to create slowpath work queue!\n"); + kfree(qctrl); + + return rc; + } + + set_bit(QEDN_STATE_SP_WORK_THREAD_SET, &qctrl->agg_state); + } + + qedn = qedn_get_pf_from_pdev(dev->qede_pdev); + if (!qedn) { + pr_err("Failed locating QEDN for ip=%pIS\n", + &ctrl->conn_params.local_ip_addr); + rc = -EFAULT; + goto err_out; + } + + qctrl->qedn = qedn; + + /* Placeholder - setup LLH filter */ + + return 0; + +err_out: + flush_workqueue(qctrl->sp_wq); + kfree(qctrl); + + return rc; +} + +static int qedn_release_ctrl(struct nvme_tcp_ofld_ctrl *ctrl) +{ + struct qedn_ctrl *qctrl = (struct qedn_ctrl *)ctrl->private_data; + + if (test_and_clear_bit(QEDN_STATE_SP_WORK_THREAD_SET, &qctrl->agg_state)) + flush_workqueue(qctrl->sp_wq); + + if (test_and_clear_bit(QEDN_CTRL_SET_TO_OFLD_CTRL, &qctrl->agg_state)) { + kfree(qctrl); + ctrl->private_data = NULL; + } + + qctrl->agg_state = 0; + kfree(ctrl); + + return 0; +} + +static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid, size_t q_size) +{ + struct nvme_tcp_ofld_ctrl *ctrl = queue->ctrl; + struct qedn_conn_ctx *conn_ctx; + struct qedn_ctrl *qctrl; + struct qedn_ctx *qedn; + int rc; + + qctrl = (struct qedn_ctrl *)ctrl->private_data; + qedn = qctrl->qedn; + + /* Allocate qedn connection context */ + conn_ctx = kzalloc(sizeof(*conn_ctx), GFP_KERNEL); + if (!conn_ctx) + return -ENOMEM; + + queue->private_data = conn_ctx; + conn_ctx->qedn = qedn; + conn_ctx->queue = queue; + conn_ctx->ctrl = ctrl; + conn_ctx->sq_depth = q_size; + + init_waitqueue_head(&conn_ctx->conn_waitq); + atomic_set(&conn_ctx->est_conn_indicator, 0); + atomic_set(&conn_ctx->destroy_conn_indicator, 0); + + spin_lock_init(&conn_ctx->conn_state_lock); + + qedn_initialize_endpoint(&conn_ctx->ep, qedn->local_mac_addr, + &ctrl->conn_params); + + atomic_inc(&qctrl->host_num_active_conns); + + qedn_set_sp_wa(conn_ctx, CREATE_CONNECTION); + qedn_set_con_state(conn_ctx, CONN_STATE_CREATE_CONNECTION); + INIT_WORK(&conn_ctx->sp_wq_entry, qedn_sp_wq_handler); + queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry); + + /* Wait for the connection establishment to complete - this includes the + * FW TCP connection establishment and the NVMeTCP ICReq & ICResp + */ + rc = qedn_wait_for_conn_est(conn_ctx); + if (rc) + return -ENXIO; return 0; } static void qedn_drain_queue(struct nvme_tcp_ofld_queue *queue) { - /* Placeholder - qedn_drain_queue */ + /* No queue drain is required */ +} + +#define ATOMIC_READ_DESTROY_IND atomic_read(&conn_ctx->destroy_conn_indicator) +#define TERMINATE_TIMEOUT msecs_to_jiffies(QEDN_RLS_CONS_TMO) +static inline void +qedn_queue_wait_for_terminate_complete(struct qedn_conn_ctx *conn_ctx) +{ + /* Returns valid non-0 */ + int wrc, state; + + wrc = wait_event_interruptible_timeout(conn_ctx->conn_waitq, + ATOMIC_READ_DESTROY_IND > 0, + TERMINATE_TIMEOUT); + + atomic_set(&conn_ctx->destroy_conn_indicator, 0); + + spin_lock_bh(&conn_ctx->conn_state_lock); + state = conn_ctx->state; + spin_unlock_bh(&conn_ctx->conn_state_lock); + + if (!wrc || state != CONN_STATE_DESTROY_COMPLETE) + pr_warn("Timed out waiting for clear-SQ on FW conns"); } static void qedn_destroy_queue(struct nvme_tcp_ofld_queue *queue) { - /* Placeholder - qedn_destroy_queue */ + struct qedn_conn_ctx *conn_ctx; + + if (!queue) { + pr_err("ctrl has no queues\n"); + + return; + } + + conn_ctx = (struct qedn_conn_ctx *)queue->private_data; + if (!conn_ctx) + return; + + qedn_terminate_connection(conn_ctx, QEDN_ABORTIVE_TERMINATION); + + qedn_queue_wait_for_terminate_complete(conn_ctx); + + kfree(conn_ctx); } static int qedn_poll_queue(struct nvme_tcp_ofld_queue *queue) @@ -132,6 +309,8 @@ static struct nvme_tcp_ofld_ops qedn_ofld_ops = { * NVMF_OPT_NR_POLL_QUEUES | NVMF_OPT_TOS */ .claim_dev = qedn_claim_dev, + .setup_ctrl = qedn_setup_ctrl, + .release_ctrl = qedn_release_ctrl, .create_queue = qedn_create_queue, .drain_queue = qedn_drain_queue, .destroy_queue = qedn_destroy_queue, @@ -141,6 +320,21 @@ static struct nvme_tcp_ofld_ops qedn_ofld_ops = { .commit_rqs = qedn_commit_rqs, }; +struct qedn_conn_ctx *qedn_get_conn_hash(struct qedn_ctx *qedn, u16 icid) +{ + struct qedn_conn_ctx *conn = NULL; + + hash_for_each_possible(qedn->conn_ctx_hash, conn, hash_node, icid) { + if (conn->conn_handle == icid) + break; + } + + if (!conn || conn->conn_handle != icid) + return NULL; + + return conn; +} + /* Fastpath IRQ handler */ static irqreturn_t qedn_irq_handler(int irq, void *dev_id) { @@ -241,7 +435,7 @@ static int qedn_setup_irq(struct qedn_ctx *qedn) static inline void qedn_init_pf_struct(struct qedn_ctx *qedn) { - /* Placeholder - Initialize qedn fields */ + hash_init(qedn->conn_ctx_hash); } static inline void @@ -607,7 +801,7 @@ static int __qedn_probe(struct pci_dev *pdev) rc = qed_ops->start(qedn->cdev, NULL /* Placeholder for FW IO-path resources */, qedn, - NULL /* Placeholder for FW Event callback */); + qedn_event_cb); if (rc) { rc = -ENODEV; pr_err("Cannot start NVMeTCP Function\n"); From patchwork Thu Apr 29 19:09:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shai Malin X-Patchwork-Id: 429650 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C7F3C4161D for ; Thu, 29 Apr 2021 19:18:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5521B6143E for ; Thu, 29 Apr 2021 19:18:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233942AbhD2TSX (ORCPT ); Thu, 29 Apr 2021 15:18:23 -0400 Received: from mx0a-0016f401.pphosted.com ([67.231.148.174]:2708 "EHLO mx0b-0016f401.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S241773AbhD2TNP (ORCPT ); Thu, 29 Apr 2021 15:13:15 -0400 Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13TJ6GPf019581; Thu, 29 Apr 2021 12:11:15 -0700 Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com with ESMTP id 387erumw6d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 29 Apr 2021 12:11:15 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 29 Apr 2021 12:11:14 -0700 Received: from lbtlvb-pcie154.il.qlogic.org (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 29 Apr 2021 12:11:11 -0700 From: Shai Malin To: , , , , , CC: =?utf-8?q?David_S_=2E_Miller_davem_=40_davemloft_=2E_net_=C2=A0--cc?= =?utf-8?q?=3DJakub_Kicinski?= , , , , , , Subject: [RFC PATCH v4 23/27] qedn: Add support of Task and SGL Date: Thu, 29 Apr 2021 22:09:22 +0300 Message-ID: <20210429190926.5086-24-smalin@marvell.com> X-Mailer: git-send-email 2.16.6 In-Reply-To: <20210429190926.5086-1-smalin@marvell.com> References: <20210429190926.5086-1-smalin@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: ikDK5O3WNNBuHS-KZLyXBEqpXSiZnK_m X-Proofpoint-ORIG-GUID: ikDK5O3WNNBuHS-KZLyXBEqpXSiZnK_m X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-04-29_10:2021-04-28,2021-04-29 signatures=0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Prabhakar Kushwaha This patch will add support of Task and SGL which is used for slowpath and fast path IO. here Task is IO granule used by firmware to perform tasks The internal implementation: - Create task/sgl resources used by all connection - Provide APIs to allocate and free task. - Add task support during connection establishment i.e. slowpath Acked-by: Igor Russkikh Signed-off-by: Prabhakar Kushwaha Signed-off-by: Omkar Kulkarni Signed-off-by: Michal Kalderon Signed-off-by: Ariel Elior Signed-off-by: Shai Malin --- drivers/nvme/hw/qedn/qedn.h | 66 +++++ drivers/nvme/hw/qedn/qedn_conn.c | 43 +++- drivers/nvme/hw/qedn/qedn_main.c | 34 ++- drivers/nvme/hw/qedn/qedn_task.c | 411 +++++++++++++++++++++++++++++++ 4 files changed, 550 insertions(+), 4 deletions(-) diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h index bd9a250cb2f5..880ca245b02c 100644 --- a/drivers/nvme/hw/qedn/qedn.h +++ b/drivers/nvme/hw/qedn/qedn.h @@ -50,6 +50,21 @@ #define QEDN_FW_CQ_FP_WQ_WORKQUEUE "qedn_fw_cq_fp_wq" #define QEDN_NVME_REQ_FP_WQ_WORKQUEUE "qedn_nvme_req_fp_wq" +/* Protocol defines */ +#define QEDN_MAX_IO_SIZE QED_NVMETCP_MAX_IO_SIZE + +#define QEDN_SGE_BUFF_SIZE 4096 +#define QEDN_MAX_SGES_PER_TASK DIV_ROUND_UP(QEDN_MAX_IO_SIZE, QEDN_SGE_BUFF_SIZE) +#define QEDN_FW_SGE_SIZE sizeof(struct nvmetcp_sge) +#define QEDN_MAX_FW_SGL_SIZE ((QEDN_MAX_SGES_PER_TASK) * QEDN_FW_SGE_SIZE) +#define QEDN_FW_SLOW_IO_MIN_SGE_LIMIT (9700 / 6) + +#define QEDN_MAX_HW_SECTORS (QEDN_MAX_IO_SIZE / 512) +#define QEDN_MAX_SEGMENTS QEDN_MAX_SGES_PER_TASK + +#define QEDN_TASK_INSIST_TMO 1000 /* 1 sec */ +#define QEDN_INVALID_ITID 0xFFFF + /* * TCP offload stack default configurations and defines. * Future enhancements will allow controlling the configurable @@ -95,6 +110,15 @@ enum qedn_state { QEDN_STATE_MODULE_REMOVE_ONGOING, }; +struct qedn_io_resources { + /* Lock for IO resources */ + spinlock_t resources_lock; + struct list_head task_free_list; + u32 num_alloc_tasks; + u32 num_free_tasks; + u32 no_avail_resrc_cnt; +}; + /* Per CPU core params */ struct qedn_fp_queue { struct qed_chain cq_chain; @@ -104,6 +128,10 @@ struct qedn_fp_queue { struct qed_sb_info *sb_info; unsigned int cpu; struct work_struct fw_cq_fp_wq_entry; + + /* IO related resources for host */ + struct qedn_io_resources host_resrc; + u16 sb_id; char irqname[QEDN_IRQ_NAME_LEN]; }; @@ -130,6 +158,8 @@ struct qedn_ctx { /* Connections */ DECLARE_HASHTABLE(conn_ctx_hash, 16); + u32 num_tasks_per_pool; + /* Fast path queues */ u8 num_fw_cqs; struct qedn_fp_queue *fp_q_arr; @@ -137,6 +167,27 @@ struct qedn_ctx { dma_addr_t fw_cq_array_phy; /* Physical address of fw_cq_array_virt */ struct workqueue_struct *nvme_req_fp_wq; struct workqueue_struct *fw_cq_fp_wq; + + /* Fast Path Tasks */ + struct qed_nvmetcp_tid tasks; +}; + +struct qedn_task_ctx { + struct qedn_conn_ctx *qedn_conn; + struct qedn_ctx *qedn; + void *fw_task_ctx; + struct qedn_fp_queue *fp_q; + struct scatterlist *nvme_sg; + struct nvme_tcp_ofld_req *req; /* currently proccessed request */ + struct list_head entry; + spinlock_t lock; /* To protect task resources */ + bool valid; + unsigned long flags; /* Used by qedn_task_flags */ + u32 task_size; + u16 itid; + u16 cccid; + int req_direction; + struct storage_sgl_task_params sgl_task_params; }; struct qedn_endpoint { @@ -243,6 +294,7 @@ struct qedn_conn_ctx { struct nvme_tcp_ofld_ctrl *ctrl; u32 conn_handle; u32 fw_cid; + u8 default_cq; atomic_t est_conn_indicator; atomic_t destroy_conn_indicator; @@ -260,6 +312,11 @@ struct qedn_conn_ctx { dma_addr_t host_cccid_itid_phy_addr; struct qedn_endpoint ep; int abrt_flag; + /* Spinlock for accessing active_task_list */ + spinlock_t task_list_lock; + struct list_head active_task_list; + atomic_t num_active_tasks; + atomic_t num_active_fw_tasks; /* Connection resources - turned on to indicate what resource was * allocated, to that it can later be released. @@ -279,6 +336,7 @@ struct qedn_conn_ctx { enum qedn_conn_resources_state { QEDN_CONN_RESRC_FW_SQ, QEDN_CONN_RESRC_ACQUIRE_CONN, + QEDN_CONN_RESRC_TASKS, QEDN_CONN_RESRC_CCCID_ITID_MAP, QEDN_CONN_RESRC_TCP_PORT, QEDN_CONN_RESRC_MAX = 64 @@ -309,5 +367,13 @@ inline int qedn_validate_cccid_in_range(struct qedn_conn_ctx *conn_ctx, u16 ccci void qedn_queue_request(struct qedn_conn_ctx *qedn_conn, struct nvme_tcp_ofld_req *req); void qedn_nvme_req_fp_wq_handler(struct work_struct *work); void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe); +int qedn_alloc_tasks(struct qedn_conn_ctx *conn_ctx); +inline int qedn_qid(struct nvme_tcp_ofld_queue *queue); +struct qedn_task_ctx * + qedn_get_task_from_pool_insist(struct qedn_conn_ctx *conn_ctx, u16 cccid); +void qedn_common_clear_fw_sgl(struct storage_sgl_task_params *sgl_task_params); +void qedn_return_active_tasks(struct qedn_conn_ctx *conn_ctx); +void qedn_destroy_free_tasks(struct qedn_fp_queue *fp_q, + struct qedn_io_resources *io_resrc); #endif /* _QEDN_H_ */ diff --git a/drivers/nvme/hw/qedn/qedn_conn.c b/drivers/nvme/hw/qedn/qedn_conn.c index 90d8aa36d219..10a80fbeac43 100644 --- a/drivers/nvme/hw/qedn/qedn_conn.c +++ b/drivers/nvme/hw/qedn/qedn_conn.c @@ -29,6 +29,11 @@ static const char * const qedn_conn_state_str[] = { NULL }; +inline int qedn_qid(struct nvme_tcp_ofld_queue *queue) +{ + return queue - queue->ctrl->queues; +} + int qedn_set_con_state(struct qedn_conn_ctx *conn_ctx, enum qedn_conn_state new_state) { spin_lock_bh(&conn_ctx->conn_state_lock); @@ -146,6 +151,11 @@ static void qedn_release_conn_ctx(struct qedn_conn_ctx *conn_ctx) clear_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state); } + if (test_bit(QEDN_CONN_RESRC_TASKS, &conn_ctx->resrc_state)) { + clear_bit(QEDN_CONN_RESRC_TASKS, &conn_ctx->resrc_state); + qedn_return_active_tasks(conn_ctx); + } + if (test_bit(QEDN_CONN_RESRC_CCCID_ITID_MAP, &conn_ctx->resrc_state)) { dma_free_coherent(&qedn->pdev->dev, conn_ctx->sq_depth * @@ -247,6 +257,7 @@ static int qedn_nvmetcp_offload_conn(struct qedn_conn_ctx *conn_ctx) offld_prms.max_rt_time = QEDN_TCP_MAX_RT_TIME; offld_prms.sq_pbl_addr = (u64)qed_chain_get_pbl_phys(&qedn_ep->fw_sq_chain); + offld_prms.default_cq = conn_ctx->default_cq; rc = qed_ops->offload_conn(qedn->cdev, conn_ctx->conn_handle, @@ -375,6 +386,9 @@ int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data) static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx) { struct qedn_ctx *qedn = conn_ctx->qedn; + struct qedn_io_resources *io_resrc; + struct qedn_fp_queue *fp_q; + u8 default_cq_idx, qid; size_t dma_size; int rc; @@ -387,6 +401,8 @@ static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx) set_bit(QEDN_CONN_RESRC_FW_SQ, &conn_ctx->resrc_state); INIT_LIST_HEAD(&conn_ctx->host_pend_req_list); spin_lock_init(&conn_ctx->nvme_req_lock); + atomic_set(&conn_ctx->num_active_tasks, 0); + atomic_set(&conn_ctx->num_active_fw_tasks, 0); rc = qed_ops->acquire_conn(qedn->cdev, &conn_ctx->conn_handle, @@ -401,7 +417,32 @@ static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx) conn_ctx->conn_handle); set_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state); - /* Placeholder - Allocate task resources and initialize fields */ + qid = qedn_qid(conn_ctx->queue); + default_cq_idx = qid ? qid - 1 : 0; /* Offset adminq */ + + conn_ctx->default_cq = (default_cq_idx % qedn->num_fw_cqs); + fp_q = &qedn->fp_q_arr[conn_ctx->default_cq]; + conn_ctx->fp_q = fp_q; + io_resrc = &fp_q->host_resrc; + + /* The first connection on each fp_q will fill task + * resources + */ + spin_lock(&io_resrc->resources_lock); + if (io_resrc->num_alloc_tasks == 0) { + rc = qedn_alloc_tasks(conn_ctx); + if (rc) { + pr_err("Failed allocating tasks: CID=0x%x\n", + conn_ctx->fw_cid); + spin_unlock(&io_resrc->resources_lock); + goto rel_conn; + } + } + spin_unlock(&io_resrc->resources_lock); + + spin_lock_init(&conn_ctx->task_list_lock); + INIT_LIST_HEAD(&conn_ctx->active_task_list); + set_bit(QEDN_CONN_RESRC_TASKS, &conn_ctx->resrc_state); rc = qedn_fetch_tcp_port(conn_ctx); if (rc) diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c index 38f23dbb03a5..8d9c19d63480 100644 --- a/drivers/nvme/hw/qedn/qedn_main.c +++ b/drivers/nvme/hw/qedn/qedn_main.c @@ -30,6 +30,12 @@ __be16 qedn_get_in_port(struct sockaddr_storage *sa) : ((struct sockaddr_in6 *)sa)->sin6_port; } +static void qedn_init_io_resc(struct qedn_io_resources *io_resrc) +{ + spin_lock_init(&io_resrc->resources_lock); + INIT_LIST_HEAD(&io_resrc->task_free_list); +} + struct qedn_llh_filter *qedn_add_llh_filter(struct qedn_ctx *qedn, u16 tcp_port) { struct qedn_llh_filter *llh_filter = NULL; @@ -436,6 +442,8 @@ static struct nvme_tcp_ofld_ops qedn_ofld_ops = { * NVMF_OPT_HDR_DIGEST | NVMF_OPT_DATA_DIGEST | * NVMF_OPT_NR_POLL_QUEUES | NVMF_OPT_TOS */ + .max_hw_sectors = QEDN_MAX_HW_SECTORS, + .max_segments = QEDN_MAX_SEGMENTS, .claim_dev = qedn_claim_dev, .setup_ctrl = qedn_setup_ctrl, .release_ctrl = qedn_release_ctrl, @@ -657,8 +665,24 @@ static void qedn_remove_pf_from_gl_list(struct qedn_ctx *qedn) mutex_unlock(&qedn_glb.glb_mutex); } +static void qedn_call_destroy_free_tasks(struct qedn_fp_queue *fp_q, + struct qedn_io_resources *io_resrc) +{ + if (list_empty(&io_resrc->task_free_list)) + return; + + if (io_resrc->num_alloc_tasks != io_resrc->num_free_tasks) + pr_err("Task Pool:Not all returned allocated=0x%x, free=0x%x\n", + io_resrc->num_alloc_tasks, io_resrc->num_free_tasks); + + qedn_destroy_free_tasks(fp_q, io_resrc); + if (io_resrc->num_free_tasks) + pr_err("Expected num_free_tasks to be 0\n"); +} + static void qedn_free_function_queues(struct qedn_ctx *qedn) { + struct qedn_io_resources *host_resrc; struct qed_sb_info *sb_info = NULL; struct qedn_fp_queue *fp_q; int i; @@ -673,6 +697,9 @@ static void qedn_free_function_queues(struct qedn_ctx *qedn) /* Free the fast path queues*/ for (i = 0; i < qedn->num_fw_cqs; i++) { fp_q = &qedn->fp_q_arr[i]; + host_resrc = &fp_q->host_resrc; + + qedn_call_destroy_free_tasks(fp_q, host_resrc); /* Free SB */ sb_info = fp_q->sb_info; @@ -769,7 +796,8 @@ static int qedn_alloc_function_queues(struct qedn_ctx *qedn) goto mem_alloc_failure; } - /* placeholder - create task pools */ + qedn->num_tasks_per_pool = + qedn->pf_params.nvmetcp_pf_params.num_tasks / qedn->num_fw_cqs; for (i = 0; i < qedn->num_fw_cqs; i++) { fp_q = &qedn->fp_q_arr[i]; @@ -811,7 +839,7 @@ static int qedn_alloc_function_queues(struct qedn_ctx *qedn) fp_q->qedn = qedn; INIT_WORK(&fp_q->fw_cq_fp_wq_entry, qedn_fw_cq_fq_wq_handler); - /* Placeholder - Init IO-path resources */ + qedn_init_io_resc(&fp_q->host_resrc); } return 0; @@ -1005,7 +1033,7 @@ static int __qedn_probe(struct pci_dev *pdev) /* NVMeTCP start HW PF */ rc = qed_ops->start(qedn->cdev, - NULL /* Placeholder for FW IO-path resources */, + &qedn->tasks, qedn, qedn_event_cb); if (rc) { diff --git a/drivers/nvme/hw/qedn/qedn_task.c b/drivers/nvme/hw/qedn/qedn_task.c index d3474188efdc..54f2f4cba6ea 100644 --- a/drivers/nvme/hw/qedn/qedn_task.c +++ b/drivers/nvme/hw/qedn/qedn_task.c @@ -11,6 +11,263 @@ /* Driver includes */ #include "qedn.h" +static bool qedn_sgl_has_small_mid_sge(struct nvmetcp_sge *sgl, u16 sge_count) +{ + u16 sge_num; + + if (sge_count > 8) { + for (sge_num = 0; sge_num < sge_count; sge_num++) { + if (le32_to_cpu(sgl[sge_num].sge_len) < + QEDN_FW_SLOW_IO_MIN_SGE_LIMIT) + return true; /* small middle SGE found */ + } + } + + return false; /* no small middle SGEs */ +} + +static int qedn_init_sgl(struct qedn_ctx *qedn, struct qedn_task_ctx *qedn_task) +{ + struct storage_sgl_task_params *sgl_task_params; + enum dma_data_direction dma_dir; + struct scatterlist *sg; + struct request *rq; + u16 num_sges; + int index; + int rc; + + sgl_task_params = &qedn_task->sgl_task_params; + rq = blk_mq_rq_from_pdu(qedn_task->req); + if (qedn_task->task_size == 0) { + sgl_task_params->num_sges = 0; + + return 0; + } + + /* Convert BIO to scatterlist */ + num_sges = blk_rq_map_sg(rq->q, rq, qedn_task->nvme_sg); + if (qedn_task->req_direction == WRITE) + dma_dir = DMA_TO_DEVICE; + else + dma_dir = DMA_FROM_DEVICE; + + /* DMA map the scatterlist */ + if (dma_map_sg(&qedn->pdev->dev, qedn_task->nvme_sg, num_sges, dma_dir) != num_sges) { + pr_err("Couldn't map sgl\n"); + rc = -EPERM; + + return rc; + } + + sgl_task_params->total_buffer_size = qedn_task->task_size; + sgl_task_params->num_sges = num_sges; + + for_each_sg(qedn_task->nvme_sg, sg, num_sges, index) { + DMA_REGPAIR_LE(sgl_task_params->sgl[index].sge_addr, sg_dma_address(sg)); + sgl_task_params->sgl[index].sge_len = cpu_to_le32(sg_dma_len(sg)); + } + + /* Relevant for Host Write Only */ + sgl_task_params->small_mid_sge = (qedn_task->req_direction == READ) ? + false : + qedn_sgl_has_small_mid_sge(sgl_task_params->sgl, + sgl_task_params->num_sges); + + return 0; +} + +static void qedn_free_nvme_sg(struct qedn_task_ctx *qedn_task) +{ + kfree(qedn_task->nvme_sg); + qedn_task->nvme_sg = NULL; +} + +static void qedn_free_fw_sgl(struct qedn_task_ctx *qedn_task) +{ + struct qedn_ctx *qedn = qedn_task->qedn; + dma_addr_t sgl_pa; + + sgl_pa = HILO_DMA_REGPAIR(qedn_task->sgl_task_params.sgl_phys_addr); + dma_free_coherent(&qedn->pdev->dev, + QEDN_MAX_FW_SGL_SIZE, + qedn_task->sgl_task_params.sgl, + sgl_pa); + qedn_task->sgl_task_params.sgl = NULL; +} + +static void qedn_destroy_single_task(struct qedn_task_ctx *qedn_task) +{ + u16 itid; + + itid = qedn_task->itid; + list_del(&qedn_task->entry); + qedn_free_nvme_sg(qedn_task); + qedn_free_fw_sgl(qedn_task); + kfree(qedn_task); + qedn_task = NULL; +} + +void qedn_destroy_free_tasks(struct qedn_fp_queue *fp_q, + struct qedn_io_resources *io_resrc) +{ + struct qedn_task_ctx *qedn_task, *task_tmp; + + /* Destroy tasks from the free task list */ + list_for_each_entry_safe(qedn_task, task_tmp, + &io_resrc->task_free_list, entry) { + qedn_destroy_single_task(qedn_task); + io_resrc->num_free_tasks -= 1; + } +} + +static int qedn_alloc_nvme_sg(struct qedn_task_ctx *qedn_task) +{ + int rc; + + qedn_task->nvme_sg = kcalloc(QEDN_MAX_SGES_PER_TASK, + sizeof(*qedn_task->nvme_sg), GFP_KERNEL); + if (!qedn_task->nvme_sg) { + rc = -ENOMEM; + + return rc; + } + + return 0; +} + +static int qedn_alloc_fw_sgl(struct qedn_task_ctx *qedn_task) +{ + struct qedn_ctx *qedn = qedn_task->qedn_conn->qedn; + dma_addr_t fw_sgl_phys; + + qedn_task->sgl_task_params.sgl = + dma_alloc_coherent(&qedn->pdev->dev, QEDN_MAX_FW_SGL_SIZE, + &fw_sgl_phys, GFP_KERNEL); + if (!qedn_task->sgl_task_params.sgl) { + pr_err("Couldn't allocate FW sgl\n"); + + return -ENOMEM; + } + + DMA_REGPAIR_LE(qedn_task->sgl_task_params.sgl_phys_addr, fw_sgl_phys); + + return 0; +} + +static inline void *qedn_get_fw_task(struct qed_nvmetcp_tid *info, u16 itid) +{ + return (void *)(info->blocks[itid / info->num_tids_per_block] + + (itid % info->num_tids_per_block) * info->size); +} + +static struct qedn_task_ctx *qedn_alloc_task(struct qedn_conn_ctx *conn_ctx, u16 itid) +{ + struct qedn_ctx *qedn = conn_ctx->qedn; + struct qedn_task_ctx *qedn_task; + void *fw_task_ctx; + int rc = 0; + + qedn_task = kzalloc(sizeof(*qedn_task), GFP_KERNEL); + if (!qedn_task) + return NULL; + + spin_lock_init(&qedn_task->lock); + fw_task_ctx = qedn_get_fw_task(&qedn->tasks, itid); + if (!fw_task_ctx) { + pr_err("iTID: 0x%x; Failed getting fw_task_ctx memory\n", itid); + goto release_task; + } + + /* No need to memset fw_task_ctx - its done in the HSI func */ + qedn_task->qedn_conn = conn_ctx; + qedn_task->qedn = qedn; + qedn_task->fw_task_ctx = fw_task_ctx; + qedn_task->valid = 0; + qedn_task->flags = 0; + qedn_task->itid = itid; + rc = qedn_alloc_fw_sgl(qedn_task); + if (rc) { + pr_err("iTID: 0x%x; Failed allocating FW sgl\n", itid); + goto release_task; + } + + rc = qedn_alloc_nvme_sg(qedn_task); + if (rc) { + pr_err("iTID: 0x%x; Failed allocating FW sgl\n", itid); + goto release_fw_sgl; + } + + return qedn_task; + +release_fw_sgl: + qedn_free_fw_sgl(qedn_task); +release_task: + kfree(qedn_task); + + return NULL; +} + +int qedn_alloc_tasks(struct qedn_conn_ctx *conn_ctx) +{ + struct qedn_ctx *qedn = conn_ctx->qedn; + struct qedn_task_ctx *qedn_task = NULL; + struct qedn_io_resources *io_resrc; + u16 itid, start_itid, offset; + struct qedn_fp_queue *fp_q; + int i, rc; + + fp_q = conn_ctx->fp_q; + + offset = fp_q->sb_id; + io_resrc = &fp_q->host_resrc; + + start_itid = qedn->num_tasks_per_pool * offset; + for (i = 0; i < qedn->num_tasks_per_pool; ++i) { + itid = start_itid + i; + qedn_task = qedn_alloc_task(conn_ctx, itid); + if (!qedn_task) { + pr_err("Failed allocating task\n"); + rc = -ENOMEM; + goto release_tasks; + } + + qedn_task->fp_q = fp_q; + io_resrc->num_free_tasks += 1; + list_add_tail(&qedn_task->entry, &io_resrc->task_free_list); + } + + io_resrc->num_alloc_tasks = io_resrc->num_free_tasks; + + return 0; + +release_tasks: + qedn_destroy_free_tasks(fp_q, io_resrc); + + return rc; +} + +void qedn_common_clear_fw_sgl(struct storage_sgl_task_params *sgl_task_params) +{ + u16 sge_cnt = sgl_task_params->num_sges; + + memset(&sgl_task_params->sgl[(sge_cnt - 1)], 0, + sizeof(struct nvmetcp_sge)); + sgl_task_params->total_buffer_size = 0; + sgl_task_params->small_mid_sge = false; + sgl_task_params->num_sges = 0; +} + +inline void qedn_host_reset_cccid_itid_entry(struct qedn_conn_ctx *conn_ctx, + u16 cccid) +{ + conn_ctx->host_cccid_itid[cccid].itid = cpu_to_le16(QEDN_INVALID_ITID); +} + +inline void qedn_host_set_cccid_itid_entry(struct qedn_conn_ctx *conn_ctx, u16 cccid, u16 itid) +{ + conn_ctx->host_cccid_itid[cccid].itid = cpu_to_le16(itid); +} + inline int qedn_validate_cccid_in_range(struct qedn_conn_ctx *conn_ctx, u16 cccid) { int rc = 0; @@ -23,6 +280,160 @@ inline int qedn_validate_cccid_in_range(struct qedn_conn_ctx *conn_ctx, u16 ccci return rc; } +static void qedn_clear_sgl(struct qedn_ctx *qedn, + struct qedn_task_ctx *qedn_task) +{ + struct storage_sgl_task_params *sgl_task_params; + enum dma_data_direction dma_dir; + u32 sge_cnt; + + sgl_task_params = &qedn_task->sgl_task_params; + sge_cnt = sgl_task_params->num_sges; + + /* Nothing to do if no SGEs were used */ + if (!qedn_task->task_size || !sge_cnt) + return; + + dma_dir = (qedn_task->req_direction == WRITE ? DMA_TO_DEVICE : DMA_FROM_DEVICE); + dma_unmap_sg(&qedn->pdev->dev, qedn_task->nvme_sg, sge_cnt, dma_dir); + memset(&qedn_task->nvme_sg[(sge_cnt - 1)], 0, sizeof(struct scatterlist)); + qedn_common_clear_fw_sgl(sgl_task_params); + qedn_task->task_size = 0; +} + +static void qedn_clear_task(struct qedn_conn_ctx *conn_ctx, + struct qedn_task_ctx *qedn_task) +{ + /* Task lock isn't needed since it is no longer in use */ + qedn_clear_sgl(conn_ctx->qedn, qedn_task); + qedn_task->valid = 0; + qedn_task->flags = 0; + + atomic_dec(&conn_ctx->num_active_tasks); +} + +void qedn_return_active_tasks(struct qedn_conn_ctx *conn_ctx) +{ + struct qedn_fp_queue *fp_q = conn_ctx->fp_q; + struct qedn_task_ctx *qedn_task, *task_tmp; + struct qedn_io_resources *io_resrc; + int num_returned_tasks = 0; + int num_active_tasks; + + io_resrc = &fp_q->host_resrc; + + /* Return tasks that aren't "Used by FW" to the pool */ + list_for_each_entry_safe(qedn_task, task_tmp, + &conn_ctx->active_task_list, entry) { + qedn_clear_task(conn_ctx, qedn_task); + num_returned_tasks++; + } + + if (num_returned_tasks) { + spin_lock(&io_resrc->resources_lock); + /* Return tasks to FP_Q pool in one shot */ + + list_splice_tail_init(&conn_ctx->active_task_list, + &io_resrc->task_free_list); + io_resrc->num_free_tasks += num_returned_tasks; + spin_unlock(&io_resrc->resources_lock); + } + + num_active_tasks = atomic_read(&conn_ctx->num_active_tasks); + if (num_active_tasks) + pr_err("num_active_tasks is %u after cleanup.\n", num_active_tasks); +} + +void qedn_return_task_to_pool(struct qedn_conn_ctx *conn_ctx, + struct qedn_task_ctx *qedn_task) +{ + struct qedn_fp_queue *fp_q = conn_ctx->fp_q; + struct qedn_io_resources *io_resrc; + unsigned long lock_flags; + + io_resrc = &fp_q->host_resrc; + + spin_lock_irqsave(&qedn_task->lock, lock_flags); + qedn_task->valid = 0; + qedn_task->flags = 0; + qedn_clear_sgl(conn_ctx->qedn, qedn_task); + spin_unlock_irqrestore(&qedn_task->lock, lock_flags); + + spin_lock(&conn_ctx->task_list_lock); + list_del(&qedn_task->entry); + qedn_host_reset_cccid_itid_entry(conn_ctx, qedn_task->cccid); + spin_unlock(&conn_ctx->task_list_lock); + + atomic_dec(&conn_ctx->num_active_tasks); + atomic_dec(&conn_ctx->num_active_fw_tasks); + + spin_lock(&io_resrc->resources_lock); + list_add_tail(&qedn_task->entry, &io_resrc->task_free_list); + io_resrc->num_free_tasks += 1; + spin_unlock(&io_resrc->resources_lock); +} + +struct qedn_task_ctx * +qedn_get_free_task_from_pool(struct qedn_conn_ctx *conn_ctx, u16 cccid) +{ + struct qedn_task_ctx *qedn_task = NULL; + struct qedn_io_resources *io_resrc; + struct qedn_fp_queue *fp_q; + + fp_q = conn_ctx->fp_q; + io_resrc = &fp_q->host_resrc; + + spin_lock(&io_resrc->resources_lock); + qedn_task = list_first_entry_or_null(&io_resrc->task_free_list, + struct qedn_task_ctx, entry); + if (unlikely(!qedn_task)) { + spin_unlock(&io_resrc->resources_lock); + + return NULL; + } + list_del(&qedn_task->entry); + io_resrc->num_free_tasks -= 1; + spin_unlock(&io_resrc->resources_lock); + + spin_lock(&conn_ctx->task_list_lock); + list_add_tail(&qedn_task->entry, &conn_ctx->active_task_list); + qedn_host_set_cccid_itid_entry(conn_ctx, cccid, qedn_task->itid); + spin_unlock(&conn_ctx->task_list_lock); + + atomic_inc(&conn_ctx->num_active_tasks); + qedn_task->cccid = cccid; + qedn_task->qedn_conn = conn_ctx; + qedn_task->valid = 1; + + return qedn_task; +} + +struct qedn_task_ctx * +qedn_get_task_from_pool_insist(struct qedn_conn_ctx *conn_ctx, u16 cccid) +{ + struct qedn_task_ctx *qedn_task = NULL; + unsigned long timeout; + + qedn_task = qedn_get_free_task_from_pool(conn_ctx, cccid); + if (unlikely(!qedn_task)) { + timeout = msecs_to_jiffies(QEDN_TASK_INSIST_TMO) + jiffies; + while (1) { + qedn_task = qedn_get_free_task_from_pool(conn_ctx, cccid); + if (likely(qedn_task)) + break; + + msleep(100); + if (time_after(jiffies, timeout)) { + pr_err("Failed on timeout of fetching task\n"); + + return NULL; + } + } + } + + return qedn_task; +} + static bool qedn_process_req(struct qedn_conn_ctx *qedn_conn) { return true; From patchwork Thu Apr 29 19:09:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shai Malin X-Patchwork-Id: 429655 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFFE1C43616 for ; Thu, 29 Apr 2021 19:18:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B4EDF6145D for ; Thu, 29 Apr 2021 19:18:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233733AbhD2TR6 (ORCPT ); Thu, 29 Apr 2021 15:17:58 -0400 Received: from mx0a-0016f401.pphosted.com ([67.231.148.174]:43708 "EHLO mx0b-0016f401.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S234754AbhD2TMW (ORCPT ); Thu, 29 Apr 2021 15:12:22 -0400 Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13TJ6FJA019574; Thu, 29 Apr 2021 12:11:20 -0700 Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0a-0016f401.pphosted.com with ESMTP id 387erumw6h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 29 Apr 2021 12:11:19 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 29 Apr 2021 12:11:18 -0700 Received: from lbtlvb-pcie154.il.qlogic.org (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 29 Apr 2021 12:11:15 -0700 From: Shai Malin To: , , , , , CC: =?utf-8?q?David_S_=2E_Miller_davem_=40_davemloft_=2E_net_=C2=A0--cc?= =?utf-8?q?=3DJakub_Kicinski?= , , , , , , Subject: [RFC PATCH v4 24/27] qedn: Add support of NVME ICReq & ICResp Date: Thu, 29 Apr 2021 22:09:23 +0300 Message-ID: <20210429190926.5086-25-smalin@marvell.com> X-Mailer: git-send-email 2.16.6 In-Reply-To: <20210429190926.5086-1-smalin@marvell.com> References: <20210429190926.5086-1-smalin@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: wnovlya5CaaZBftyeStPXW-71TDgas-Z X-Proofpoint-ORIG-GUID: wnovlya5CaaZBftyeStPXW-71TDgas-Z X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-04-29_10:2021-04-28,2021-04-29 signatures=0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Prabhakar Kushwaha Once a TCP connection established, the host sends an Initialize Connection Request (ICReq) PDU to the controller. Further Initialize Connection Response (ICResp) PDU received from controller is processed by host to establish a connection and exchange connection configuration parameters. This patch present support of generation of ICReq and processing of ICResp. It also update host configuration based on exchanged parameters. Acked-by: Igor Russkikh Signed-off-by: Prabhakar Kushwaha Signed-off-by: Omkar Kulkarni Signed-off-by: Michal Kalderon Signed-off-by: Ariel Elior Signed-off-by: Shai Malin --- drivers/nvme/hw/qedn/qedn.h | 36 ++++ drivers/nvme/hw/qedn/qedn_conn.c | 317 ++++++++++++++++++++++++++++++- drivers/nvme/hw/qedn/qedn_main.c | 22 +++ drivers/nvme/hw/qedn/qedn_task.c | 8 +- 4 files changed, 379 insertions(+), 4 deletions(-) diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h index 880ca245b02c..773a57994148 100644 --- a/drivers/nvme/hw/qedn/qedn.h +++ b/drivers/nvme/hw/qedn/qedn.h @@ -16,6 +16,7 @@ /* Driver includes */ #include "../../host/tcp-offload.h" +#include #define QEDN_MAJOR_VERSION 8 #define QEDN_MINOR_VERSION 62 @@ -52,6 +53,8 @@ /* Protocol defines */ #define QEDN_MAX_IO_SIZE QED_NVMETCP_MAX_IO_SIZE +#define QEDN_MAX_PDU_SIZE 0x80000 /* 512KB */ +#define QEDN_MAX_OUTSTANDING_R2T_PDUS 0 /* 0 Based == 1 max R2T */ #define QEDN_SGE_BUFF_SIZE 4096 #define QEDN_MAX_SGES_PER_TASK DIV_ROUND_UP(QEDN_MAX_IO_SIZE, QEDN_SGE_BUFF_SIZE) @@ -65,6 +68,11 @@ #define QEDN_TASK_INSIST_TMO 1000 /* 1 sec */ #define QEDN_INVALID_ITID 0xFFFF +#define QEDN_ICREQ_FW_PAYLOAD (sizeof(struct nvme_tcp_icreq_pdu) - \ + sizeof(struct nvmetcp_init_conn_req_hdr)) +/* The FW will handle the ICReq as CCCID 0 (FW internal design) */ +#define QEDN_ICREQ_CCCID 0 + /* * TCP offload stack default configurations and defines. * Future enhancements will allow controlling the configurable @@ -136,6 +144,16 @@ struct qedn_fp_queue { char irqname[QEDN_IRQ_NAME_LEN]; }; +struct qedn_negotiation_params { + u32 maxh2cdata; /* Negotiation */ + u32 maxr2t; /* Validation */ + u16 pfv; /* Validation */ + bool hdr_digest; /* Negotiation */ + bool data_digest; /* Negotiation */ + u8 cpda; /* Negotiation */ + u8 hpda; /* Validation */ +}; + struct qedn_ctx { struct pci_dev *pdev; struct qed_dev *cdev; @@ -195,6 +213,9 @@ struct qedn_endpoint { struct qed_chain fw_sq_chain; void __iomem *p_doorbell; + /* Spinlock for accessing FW queue */ + spinlock_t doorbell_lock; + /* TCP Params */ __be32 dst_addr[4]; /* In network order */ __be32 src_addr[4]; /* In network order */ @@ -268,6 +289,12 @@ struct qedn_ctrl { atomic_t host_num_active_conns; }; +struct qedn_icreq_padding { + u32 *buffer; + dma_addr_t pa; + struct nvmetcp_sge sge; +}; + /* Connection level struct */ struct qedn_conn_ctx { /* IO path */ @@ -329,6 +356,11 @@ struct qedn_conn_ctx { size_t sq_depth; + struct qedn_negotiation_params required_params; + struct qedn_negotiation_params pdu_params; + struct nvmetcp_icresp_hdr_psh icresp; + struct qedn_icreq_padding *icreq_pad; + /* "dummy" socket */ struct socket *sock; }; @@ -337,6 +369,7 @@ enum qedn_conn_resources_state { QEDN_CONN_RESRC_FW_SQ, QEDN_CONN_RESRC_ACQUIRE_CONN, QEDN_CONN_RESRC_TASKS, + QEDN_CONN_RESRC_ICREQ_PAD, QEDN_CONN_RESRC_CCCID_ITID_MAP, QEDN_CONN_RESRC_TCP_PORT, QEDN_CONN_RESRC_MAX = 64 @@ -375,5 +408,8 @@ void qedn_common_clear_fw_sgl(struct storage_sgl_task_params *sgl_task_params); void qedn_return_active_tasks(struct qedn_conn_ctx *conn_ctx); void qedn_destroy_free_tasks(struct qedn_fp_queue *fp_q, struct qedn_io_resources *io_resrc); +void qedn_swap_bytes(u32 *p, int size); +void qedn_prep_icresp(struct qedn_conn_ctx *conn_ctx, struct nvmetcp_fw_cqe *cqe); +void qedn_ring_doorbell(struct qedn_conn_ctx *conn_ctx); #endif /* _QEDN_H_ */ diff --git a/drivers/nvme/hw/qedn/qedn_conn.c b/drivers/nvme/hw/qedn/qedn_conn.c index 10a80fbeac43..5679354aa0e0 100644 --- a/drivers/nvme/hw/qedn/qedn_conn.c +++ b/drivers/nvme/hw/qedn/qedn_conn.c @@ -34,6 +34,25 @@ inline int qedn_qid(struct nvme_tcp_ofld_queue *queue) return queue - queue->ctrl->queues; } +void qedn_ring_doorbell(struct qedn_conn_ctx *conn_ctx) +{ + struct nvmetcp_db_data dbell = { 0 }; + u16 prod_idx; + + dbell.agg_flags = 0; + dbell.params |= DB_DEST_XCM << NVMETCP_DB_DATA_DEST_SHIFT; + dbell.params |= DB_AGG_CMD_SET << NVMETCP_DB_DATA_AGG_CMD_SHIFT; + dbell.params |= + DQ_XCM_ISCSI_SQ_PROD_CMD << NVMETCP_DB_DATA_AGG_VAL_SEL_SHIFT; + dbell.params |= 1 << NVMETCP_DB_DATA_BYPASS_EN_SHIFT; + prod_idx = qed_chain_get_prod_idx(&conn_ctx->ep.fw_sq_chain); + dbell.sq_prod = cpu_to_le16(prod_idx); + + /* wmb - Make sure fw idx is coherent */ + wmb(); + writel(*(u32 *)&dbell, conn_ctx->ep.p_doorbell); +} + int qedn_set_con_state(struct qedn_conn_ctx *conn_ctx, enum qedn_conn_state new_state) { spin_lock_bh(&conn_ctx->conn_state_lock); @@ -130,6 +149,71 @@ int qedn_initialize_endpoint(struct qedn_endpoint *ep, u8 *local_mac_addr, return -1; } +static int qedn_alloc_icreq_pad(struct qedn_conn_ctx *conn_ctx) +{ + struct qedn_ctx *qedn = conn_ctx->qedn; + struct qedn_icreq_padding *icreq_pad; + u32 *buffer; + int rc = 0; + + icreq_pad = kzalloc(sizeof(*icreq_pad), GFP_KERNEL); + if (!icreq_pad) + return -ENOMEM; + + conn_ctx->icreq_pad = icreq_pad; + memset(&icreq_pad->sge, 0, sizeof(icreq_pad->sge)); + buffer = dma_alloc_coherent(&qedn->pdev->dev, + QEDN_ICREQ_FW_PAYLOAD, + &icreq_pad->pa, + GFP_KERNEL); + if (!buffer) { + pr_err("Could not allocate icreq_padding SGE buffer.\n"); + rc = -ENOMEM; + goto release_icreq_pad; + } + + DMA_REGPAIR_LE(icreq_pad->sge.sge_addr, icreq_pad->pa); + icreq_pad->sge.sge_len = cpu_to_le32(QEDN_ICREQ_FW_PAYLOAD); + icreq_pad->buffer = buffer; + set_bit(QEDN_CONN_RESRC_ICREQ_PAD, &conn_ctx->resrc_state); + + return 0; + +release_icreq_pad: + kfree(icreq_pad); + conn_ctx->icreq_pad = NULL; + + return rc; +} + +static void qedn_free_icreq_pad(struct qedn_conn_ctx *conn_ctx) +{ + struct qedn_ctx *qedn = conn_ctx->qedn; + struct qedn_icreq_padding *icreq_pad; + u32 *buffer; + + icreq_pad = conn_ctx->icreq_pad; + if (unlikely(!icreq_pad)) { + pr_err("null ptr in icreq_pad in conn_ctx\n"); + goto finally; + } + + buffer = icreq_pad->buffer; + if (buffer) { + dma_free_coherent(&qedn->pdev->dev, + QEDN_ICREQ_FW_PAYLOAD, + (void *)buffer, + icreq_pad->pa); + icreq_pad->buffer = NULL; + } + + kfree(icreq_pad); + conn_ctx->icreq_pad = NULL; + +finally: + clear_bit(QEDN_CONN_RESRC_ICREQ_PAD, &conn_ctx->resrc_state); +} + static void qedn_release_conn_ctx(struct qedn_conn_ctx *conn_ctx) { struct qedn_ctx *qedn = conn_ctx->qedn; @@ -151,6 +235,9 @@ static void qedn_release_conn_ctx(struct qedn_conn_ctx *conn_ctx) clear_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state); } + if (test_bit(QEDN_CONN_RESRC_ICREQ_PAD, &conn_ctx->resrc_state)) + qedn_free_icreq_pad(conn_ctx); + if (test_bit(QEDN_CONN_RESRC_TASKS, &conn_ctx->resrc_state)) { clear_bit(QEDN_CONN_RESRC_TASKS, &conn_ctx->resrc_state); qedn_return_active_tasks(conn_ctx); @@ -309,6 +396,194 @@ void qedn_terminate_connection(struct qedn_conn_ctx *conn_ctx, int abrt_flag) queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry); } +static int qedn_nvmetcp_update_conn(struct qedn_ctx *qedn, struct qedn_conn_ctx *conn_ctx) +{ + struct qedn_negotiation_params *pdu_params = &conn_ctx->pdu_params; + struct qed_nvmetcp_params_update *conn_info; + int rc; + + conn_info = kzalloc(sizeof(*conn_info), GFP_KERNEL); + if (!conn_info) + return -ENOMEM; + + conn_info->hdr_digest_en = pdu_params->hdr_digest; + conn_info->data_digest_en = pdu_params->data_digest; + conn_info->max_recv_pdu_length = QEDN_MAX_PDU_SIZE; + conn_info->max_io_size = QEDN_MAX_IO_SIZE; + conn_info->max_send_pdu_length = pdu_params->maxh2cdata; + + rc = qed_ops->update_conn(qedn->cdev, conn_ctx->conn_handle, conn_info); + if (rc) { + pr_err("Could not update connection\n"); + rc = -ENXIO; + } + + kfree(conn_info); + + return rc; +} + +static int qedn_update_ramrod(struct qedn_conn_ctx *conn_ctx) +{ + struct qedn_ctx *qedn = conn_ctx->qedn; + int rc = 0; + + rc = qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_UPDATE_EQE); + if (rc) + return rc; + + rc = qedn_nvmetcp_update_conn(qedn, conn_ctx); + if (rc) + return rc; + + if (conn_ctx->state != CONN_STATE_WAIT_FOR_UPDATE_EQE) { + pr_err("cid 0x%x: Unexpected state 0x%x after update ramrod\n", + conn_ctx->fw_cid, conn_ctx->state); + + return -EINVAL; + } + + return rc; +} + +static int qedn_send_icreq(struct qedn_conn_ctx *conn_ctx) +{ + struct nvmetcp_init_conn_req_hdr *icreq_ptr = NULL; + struct storage_sgl_task_params *sgl_task_params; + struct nvmetcp_task_params task_params; + struct qedn_task_ctx *qedn_task = NULL; + struct nvme_tcp_icreq_pdu icreq; + struct nvmetcp_wqe *chain_sqe; + struct nvmetcp_wqe local_sqe; + + qedn_task = qedn_get_task_from_pool_insist(conn_ctx, QEDN_ICREQ_CCCID); + if (!qedn_task) + return -EINVAL; + + memset(&icreq, 0, sizeof(icreq)); + memset(&local_sqe, 0, sizeof(local_sqe)); + + /* Initialize ICReq */ + icreq.hdr.type = nvme_tcp_icreq; + icreq.hdr.hlen = sizeof(icreq); + icreq.hdr.pdo = 0; + icreq.hdr.plen = cpu_to_le32(icreq.hdr.hlen); + icreq.pfv = cpu_to_le16(conn_ctx->required_params.pfv); + icreq.maxr2t = cpu_to_le32(conn_ctx->required_params.maxr2t); + icreq.hpda = conn_ctx->required_params.hpda; + if (conn_ctx->required_params.hdr_digest) + icreq.digest |= NVME_TCP_HDR_DIGEST_ENABLE; + if (conn_ctx->required_params.data_digest) + icreq.digest |= NVME_TCP_DATA_DIGEST_ENABLE; + + qedn_swap_bytes((u32 *)&icreq, + (sizeof(icreq) - QEDN_ICREQ_FW_PAYLOAD) / + sizeof(u32)); + + /* Initialize task params */ + task_params.opq.lo = cpu_to_le32(((u64)(qedn_task)) & 0xffffffff); + task_params.opq.hi = cpu_to_le32(((u64)(qedn_task)) >> 32); + task_params.context = qedn_task->fw_task_ctx; + task_params.sqe = &local_sqe; + task_params.conn_icid = (u16)conn_ctx->conn_handle; + task_params.itid = qedn_task->itid; + task_params.cq_rss_number = conn_ctx->default_cq; + task_params.tx_io_size = QEDN_ICREQ_FW_PAYLOAD; + task_params.rx_io_size = 0; /* Rx doesn't use SGL for icresp */ + + /* Init SGE for ICReq padding */ + sgl_task_params = &qedn_task->sgl_task_params; + sgl_task_params->total_buffer_size = task_params.tx_io_size; + sgl_task_params->small_mid_sge = false; + sgl_task_params->num_sges = 1; + memcpy(sgl_task_params->sgl, &conn_ctx->icreq_pad->sge, + sizeof(conn_ctx->icreq_pad->sge)); + icreq_ptr = (struct nvmetcp_init_conn_req_hdr *)&icreq; + + qed_ops->init_icreq_exchange(&task_params, icreq_ptr, sgl_task_params, NULL); + + qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_IC_COMP); + atomic_inc(&conn_ctx->num_active_fw_tasks); + + /* spin_lock - doorbell is accessed both Rx flow and response flow */ + spin_lock(&conn_ctx->ep.doorbell_lock); + chain_sqe = qed_chain_produce(&conn_ctx->ep.fw_sq_chain); + memcpy(chain_sqe, &local_sqe, sizeof(local_sqe)); + qedn_ring_doorbell(conn_ctx); + spin_unlock(&conn_ctx->ep.doorbell_lock); + + return 0; +} + +void qedn_prep_icresp(struct qedn_conn_ctx *conn_ctx, struct nvmetcp_fw_cqe *cqe) +{ + struct nvmetcp_icresp_hdr_psh *icresp_from_cqe = + (struct nvmetcp_icresp_hdr_psh *)&cqe->nvme_cqe; + struct nvme_tcp_ofld_ctrl *ctrl = conn_ctx->ctrl; + struct qedn_ctrl *qctrl = NULL; + + qctrl = (struct qedn_ctrl *)ctrl->private_data; + + memcpy(&conn_ctx->icresp, icresp_from_cqe, sizeof(conn_ctx->icresp)); + qedn_set_sp_wa(conn_ctx, HANDLE_ICRESP); + queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry); +} + +static int qedn_handle_icresp(struct qedn_conn_ctx *conn_ctx) +{ + struct nvmetcp_icresp_hdr_psh *icresp = &conn_ctx->icresp; + u16 pfv = __swab16(le16_to_cpu(icresp->pfv_swapped)); + int rc = 0; + + qedn_free_icreq_pad(conn_ctx); + + /* Validate ICResp */ + if (pfv != conn_ctx->required_params.pfv) { + pr_err("cid %u: unsupported pfv %u\n", conn_ctx->fw_cid, pfv); + + return -EINVAL; + } + + if (icresp->cpda > conn_ctx->required_params.cpda) { + pr_err("cid %u: unsupported cpda %u\n", conn_ctx->fw_cid, icresp->cpda); + + return -EINVAL; + } + + if ((NVME_TCP_HDR_DIGEST_ENABLE & icresp->digest) != + conn_ctx->required_params.hdr_digest) { + if ((NVME_TCP_HDR_DIGEST_ENABLE & icresp->digest) > + conn_ctx->required_params.hdr_digest) { + pr_err("cid 0x%x: invalid header digest bit\n", conn_ctx->fw_cid); + } + } + + if ((NVME_TCP_DATA_DIGEST_ENABLE & icresp->digest) != + conn_ctx->required_params.data_digest) { + if ((NVME_TCP_DATA_DIGEST_ENABLE & icresp->digest) > + conn_ctx->required_params.data_digest) { + pr_err("cid 0x%x: invalid data digest bit\n", conn_ctx->fw_cid); + } + } + + memset(&conn_ctx->pdu_params, 0, sizeof(conn_ctx->pdu_params)); + conn_ctx->pdu_params.maxh2cdata = + __swab32(le32_to_cpu(icresp->maxdata_swapped)); + conn_ctx->pdu_params.maxh2cdata = QEDN_MAX_PDU_SIZE; + if (conn_ctx->pdu_params.maxh2cdata > QEDN_MAX_PDU_SIZE) + conn_ctx->pdu_params.maxh2cdata = QEDN_MAX_PDU_SIZE; + + conn_ctx->pdu_params.pfv = pfv; + conn_ctx->pdu_params.cpda = icresp->cpda; + conn_ctx->pdu_params.hpda = conn_ctx->required_params.hpda; + conn_ctx->pdu_params.hdr_digest = NVME_TCP_HDR_DIGEST_ENABLE & icresp->digest; + conn_ctx->pdu_params.data_digest = NVME_TCP_DATA_DIGEST_ENABLE & icresp->digest; + conn_ctx->pdu_params.maxr2t = conn_ctx->required_params.maxr2t; + rc = qedn_update_ramrod(conn_ctx); + + return rc; +} + /* Slowpath EQ Callback */ int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data) { @@ -363,7 +638,8 @@ int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data) if (rc) return rc; - /* Placeholder - for ICReq flow */ + qedn_set_sp_wa(conn_ctx, SEND_ICREQ); + queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry); } break; @@ -399,6 +675,7 @@ static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx) } set_bit(QEDN_CONN_RESRC_FW_SQ, &conn_ctx->resrc_state); + spin_lock_init(&conn_ctx->ep.doorbell_lock); INIT_LIST_HEAD(&conn_ctx->host_pend_req_list); spin_lock_init(&conn_ctx->nvme_req_lock); atomic_set(&conn_ctx->num_active_tasks, 0); @@ -463,6 +740,11 @@ static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx) memset(conn_ctx->host_cccid_itid, 0xFF, dma_size); set_bit(QEDN_CONN_RESRC_CCCID_ITID_MAP, &conn_ctx->resrc_state); + + rc = qedn_alloc_icreq_pad(conn_ctx); + if (rc) + goto rel_conn; + rc = qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_CONNECT_DONE); if (rc) goto rel_conn; @@ -523,6 +805,9 @@ void qedn_sp_wq_handler(struct work_struct *work) qedn = conn_ctx->qedn; if (test_bit(DESTROY_CONNECTION, &conn_ctx->agg_work_action)) { + if (test_bit(HANDLE_ICRESP, &conn_ctx->agg_work_action)) + qedn_clr_sp_wa(conn_ctx, HANDLE_ICRESP); + qedn_destroy_connection(conn_ctx); return; @@ -537,6 +822,36 @@ void qedn_sp_wq_handler(struct work_struct *work) return; } } + + if (test_bit(SEND_ICREQ, &conn_ctx->agg_work_action)) { + qedn_clr_sp_wa(conn_ctx, SEND_ICREQ); + rc = qedn_send_icreq(conn_ctx); + if (rc) + return; + + return; + } + + if (test_bit(HANDLE_ICRESP, &conn_ctx->agg_work_action)) { + rc = qedn_handle_icresp(conn_ctx); + + qedn_clr_sp_wa(conn_ctx, HANDLE_ICRESP); + if (rc) { + pr_err("IC handling returned with 0x%x\n", rc); + if (test_and_set_bit(DESTROY_CONNECTION, &conn_ctx->agg_work_action)) + return; + + qedn_destroy_connection(conn_ctx); + + return; + } + + atomic_inc(&conn_ctx->est_conn_indicator); + qedn_set_con_state(conn_ctx, CONN_STATE_NVMETCP_CONN_ESTABLISHED); + wake_up_interruptible(&conn_ctx->conn_waitq); + + return; + } } /* Clear connection aggregative slowpath work action */ diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c index 8d9c19d63480..a6756d7250b7 100644 --- a/drivers/nvme/hw/qedn/qedn_main.c +++ b/drivers/nvme/hw/qedn/qedn_main.c @@ -285,6 +285,19 @@ static void qedn_set_ctrl_io_cpus(struct qedn_conn_ctx *conn_ctx, int qid) conn_ctx->cpu = fp_q->cpu; } +static void qedn_set_pdu_params(struct qedn_conn_ctx *conn_ctx) +{ + /* Enable digest once supported */ + conn_ctx->required_params.hdr_digest = 0; + conn_ctx->required_params.data_digest = 0; + + conn_ctx->required_params.maxr2t = QEDN_MAX_OUTSTANDING_R2T_PDUS; + conn_ctx->required_params.pfv = NVME_TCP_PFV_1_0; + conn_ctx->required_params.cpda = 0; + conn_ctx->required_params.hpda = 0; + conn_ctx->required_params.maxh2cdata = QEDN_MAX_PDU_SIZE; +} + static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid, size_t q_size) { struct nvme_tcp_ofld_ctrl *ctrl = queue->ctrl; @@ -307,6 +320,7 @@ static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid, size_t conn_ctx->ctrl = ctrl; conn_ctx->sq_depth = q_size; qedn_set_ctrl_io_cpus(conn_ctx, qid); + qedn_set_pdu_params(conn_ctx); init_waitqueue_head(&conn_ctx->conn_waitq); atomic_set(&conn_ctx->est_conn_indicator, 0); @@ -1073,6 +1087,14 @@ static int qedn_probe(struct pci_dev *pdev, const struct pci_device_id *id) return __qedn_probe(pdev); } +void qedn_swap_bytes(u32 *p, int size) +{ + int i; + + for (i = 0; i < size; ++i, ++p) + *p = __swab32(*p); +} + static struct pci_driver qedn_pci_driver = { .name = QEDN_MODULE_NAME, .id_table = qedn_pci_tbl, diff --git a/drivers/nvme/hw/qedn/qedn_task.c b/drivers/nvme/hw/qedn/qedn_task.c index 54f2f4cba6ea..9cb84883e95e 100644 --- a/drivers/nvme/hw/qedn/qedn_task.c +++ b/drivers/nvme/hw/qedn/qedn_task.c @@ -536,9 +536,11 @@ void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe) break; case NVMETCP_TASK_TYPE_INIT_CONN_REQUEST: - - /* Placeholder - ICReq flow */ - + /* Clear ICReq-padding SGE from SGL */ + qedn_common_clear_fw_sgl(&qedn_task->sgl_task_params); + /* Task is not required for icresp processing */ + qedn_return_task_to_pool(conn_ctx, qedn_task); + qedn_prep_icresp(conn_ctx, cqe); break; default: pr_info("Could not identify task type\n"); From patchwork Thu Apr 29 19:09:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shai Malin X-Patchwork-Id: 429651 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B2AAC070DE for ; Thu, 29 Apr 2021 19:19:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 672E961434 for ; Thu, 29 Apr 2021 19:19:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234097AbhD2TSj (ORCPT ); Thu, 29 Apr 2021 15:18:39 -0400 Received: from mx0b-0016f401.pphosted.com ([67.231.156.173]:44960 "EHLO mx0b-0016f401.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242214AbhD2TOY (ORCPT ); Thu, 29 Apr 2021 15:14:24 -0400 Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13TJ5Arl027348; Thu, 29 Apr 2021 12:11:28 -0700 Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com with ESMTP id 387rpnamw5-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 29 Apr 2021 12:11:28 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 29 Apr 2021 12:11:26 -0700 Received: from lbtlvb-pcie154.il.qlogic.org (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 29 Apr 2021 12:11:23 -0700 From: Shai Malin To: , , , , , CC: =?utf-8?q?David_S_=2E_Miller_davem_=40_davemloft_=2E_net_=C2=A0--cc?= =?utf-8?q?=3DJakub_Kicinski?= , , , , , , Subject: [RFC PATCH v4 26/27] qedn: Add Connection and IO level recovery flows Date: Thu, 29 Apr 2021 22:09:25 +0300 Message-ID: <20210429190926.5086-27-smalin@marvell.com> X-Mailer: git-send-email 2.16.6 In-Reply-To: <20210429190926.5086-1-smalin@marvell.com> References: <20210429190926.5086-1-smalin@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: I36LMHgOdciCgAWS0lHUGmCccygAGpb7 X-Proofpoint-ORIG-GUID: I36LMHgOdciCgAWS0lHUGmCccygAGpb7 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-04-29_10:2021-04-28,2021-04-29 signatures=0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This patch will present the connection level functionalities: - conn clear-sq: will release the FW restrictions in order to flush all the pending IOs. - drain: in case clear-sq is stuck, will release all the device FW restrictions in order to flush all the pending IOs. - task cleanup - will flush the IO level resources. Acked-by: Igor Russkikh Signed-off-by: Prabhakar Kushwaha Signed-off-by: Omkar Kulkarni Signed-off-by: Michal Kalderon Signed-off-by: Ariel Elior Signed-off-by: Shai Malin Reviewed-by: Hannes Reinecke --- drivers/nvme/hw/qedn/qedn.h | 8 ++ drivers/nvme/hw/qedn/qedn_conn.c | 133 ++++++++++++++++++++++++++++++- drivers/nvme/hw/qedn/qedn_main.c | 1 + drivers/nvme/hw/qedn/qedn_task.c | 27 ++++++- 4 files changed, 166 insertions(+), 3 deletions(-) diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h index 065e4324e30c..fed4252392e0 100644 --- a/drivers/nvme/hw/qedn/qedn.h +++ b/drivers/nvme/hw/qedn/qedn.h @@ -51,6 +51,8 @@ #define QEDN_FW_CQ_FP_WQ_WORKQUEUE "qedn_fw_cq_fp_wq" #define QEDN_NVME_REQ_FP_WQ_WORKQUEUE "qedn_nvme_req_fp_wq" +#define QEDN_DRAIN_MAX_ATTEMPTS 3 + /* Protocol defines */ #define QEDN_MAX_IO_SIZE QED_NVMETCP_MAX_IO_SIZE #define QEDN_MAX_PDU_SIZE 0x80000 /* 512KB */ @@ -104,6 +106,8 @@ /* Timeouts and delay constants */ #define QEDN_WAIT_CON_ESTABLSH_TMO 10000 /* 10 seconds */ #define QEDN_RLS_CONS_TMO 5000 /* 5 sec */ +#define QEDN_TASK_CLEANUP_TMO 3000 /* 3 sec */ +#define QEDN_DRAIN_TMO 1000 /* 1 sec */ enum qedn_state { QEDN_STATE_CORE_PROBED = 0, @@ -191,7 +195,9 @@ struct qedn_ctx { }; enum qedn_task_flags { + QEDN_TASK_IS_ICREQ, QEDN_TASK_USED_BY_FW, + QEDN_TASK_WAIT_FOR_CLEANUP, }; struct qedn_task_ctx { @@ -348,6 +354,8 @@ struct qedn_conn_ctx { struct list_head active_task_list; atomic_t num_active_tasks; atomic_t num_active_fw_tasks; + atomic_t task_cleanups_cnt; + wait_queue_head_t cleanup_waitq; /* Connection resources - turned on to indicate what resource was * allocated, to that it can later be released. diff --git a/drivers/nvme/hw/qedn/qedn_conn.c b/drivers/nvme/hw/qedn/qedn_conn.c index fa8d414eb888..8af119202b91 100644 --- a/drivers/nvme/hw/qedn/qedn_conn.c +++ b/drivers/nvme/hw/qedn/qedn_conn.c @@ -585,6 +585,11 @@ static int qedn_handle_icresp(struct qedn_conn_ctx *conn_ctx) return rc; } +void qedn_error_recovery(struct nvme_ctrl *nctrl) +{ + nvme_tcp_ofld_error_recovery(nctrl); +} + /* Slowpath EQ Callback */ int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data) { @@ -644,6 +649,7 @@ int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data) } break; + case NVMETCP_EVENT_TYPE_ASYN_TERMINATE_DONE: if (conn_ctx->state != CONN_STATE_WAIT_FOR_DESTROY_DONE) pr_err("CID=0x%x - ASYN_TERMINATE_DONE: Unexpected connection state %u\n", @@ -652,6 +658,19 @@ int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data) queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry); break; + + case NVMETCP_EVENT_TYPE_ASYN_CLOSE_RCVD: + case NVMETCP_EVENT_TYPE_ASYN_ABORT_RCVD: + case NVMETCP_EVENT_TYPE_ASYN_MAX_RT_TIME: + case NVMETCP_EVENT_TYPE_ASYN_MAX_RT_CNT: + case NVMETCP_EVENT_TYPE_ASYN_SYN_RCVD: + case NVMETCP_EVENT_TYPE_ASYN_MAX_KA_PROBES_CNT: + case NVMETCP_EVENT_TYPE_NVMETCP_CONN_ERROR: + case NVMETCP_EVENT_TYPE_TCP_CONN_ERROR: + qedn_error_recovery(&conn_ctx->ctrl->nctrl); + + break; + default: pr_err("CID=0x%x - Recv Unknown Event %u\n", conn_ctx->fw_cid, fw_event_code); break; @@ -765,8 +784,110 @@ static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx) return -EINVAL; } +static void qedn_cleanup_fw_task(struct qedn_ctx *qedn, struct qedn_task_ctx *qedn_task) +{ + struct qedn_conn_ctx *conn_ctx = qedn_task->qedn_conn; + struct nvmetcp_task_params task_params; + struct nvmetcp_wqe *chain_sqe; + struct nvmetcp_wqe local_sqe; + unsigned long lock_flags; + + /* Take lock to prevent race with fastpath, we don't want to + * invoke cleanup flows on tasks that already returned. + */ + spin_lock_irqsave(&qedn_task->lock, lock_flags); + if (!qedn_task->valid) { + spin_unlock_irqrestore(&qedn_task->lock, lock_flags); + + return; + } + /* Skip tasks not used by FW */ + if (!test_bit(QEDN_TASK_USED_BY_FW, &qedn_task->flags)) { + spin_unlock_irqrestore(&qedn_task->lock, lock_flags); + + return; + } + /* Skip tasks that were already invoked for cleanup */ + if (unlikely(test_bit(QEDN_TASK_WAIT_FOR_CLEANUP, &qedn_task->flags))) { + spin_unlock_irqrestore(&qedn_task->lock, lock_flags); + + return; + } + set_bit(QEDN_TASK_WAIT_FOR_CLEANUP, &qedn_task->flags); + spin_unlock_irqrestore(&qedn_task->lock, lock_flags); + + atomic_inc(&conn_ctx->task_cleanups_cnt); + + task_params.sqe = &local_sqe; + task_params.itid = qedn_task->itid; + qed_ops->init_task_cleanup(&task_params); + + /* spin_lock - doorbell is accessed both Rx flow and response flow */ + spin_lock(&conn_ctx->ep.doorbell_lock); + chain_sqe = qed_chain_produce(&conn_ctx->ep.fw_sq_chain); + memcpy(chain_sqe, &local_sqe, sizeof(local_sqe)); + qedn_ring_doorbell(conn_ctx); + spin_unlock(&conn_ctx->ep.doorbell_lock); +} + +inline int qedn_drain(struct qedn_conn_ctx *conn_ctx) +{ + int drain_iter = QEDN_DRAIN_MAX_ATTEMPTS; + struct qedn_ctx *qedn = conn_ctx->qedn; + int wrc; + + while (drain_iter) { + qed_ops->common->drain(qedn->cdev); + msleep(100); + + wrc = wait_event_interruptible_timeout(conn_ctx->cleanup_waitq, + !atomic_read(&conn_ctx->task_cleanups_cnt), + msecs_to_jiffies(QEDN_DRAIN_TMO)); + if (!wrc) { + drain_iter--; + continue; + } + + return 0; + } + + pr_err("CID 0x%x: cleanup after drain failed - need hard reset.\n", conn_ctx->fw_cid); + + return -EINVAL; +} + +void qedn_cleanup_all_fw_tasks(struct qedn_conn_ctx *conn_ctx) +{ + struct qedn_task_ctx *qedn_task, *task_tmp; + struct qedn_ctx *qedn = conn_ctx->qedn; + int wrc; + + list_for_each_entry_safe_reverse(qedn_task, task_tmp, &conn_ctx->active_task_list, entry) { + qedn_cleanup_fw_task(qedn, qedn_task); + } + + wrc = wait_event_interruptible_timeout(conn_ctx->cleanup_waitq, + atomic_read(&conn_ctx->task_cleanups_cnt) == 0, + msecs_to_jiffies(QEDN_TASK_CLEANUP_TMO)); + if (!wrc) { + if (qedn_drain(conn_ctx)) + return; + } +} + +static void qedn_clear_fw_sq(struct qedn_conn_ctx *conn_ctx) +{ + struct qedn_ctx *qedn = conn_ctx->qedn; + int rc; + + rc = qed_ops->clear_sq(qedn->cdev, conn_ctx->conn_handle); + if (rc) + pr_warn("clear_sq failed - rc %u\n", rc); +} + void qedn_destroy_connection(struct qedn_conn_ctx *conn_ctx) { + struct nvme_tcp_ofld_req *req, *req_tmp; struct qedn_ctx *qedn = conn_ctx->qedn; int rc; @@ -775,7 +896,17 @@ void qedn_destroy_connection(struct qedn_conn_ctx *conn_ctx) if (qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_DESTROY_DONE)) return; - /* Placeholder - task cleanup */ + spin_lock(&conn_ctx->nvme_req_lock); + list_for_each_entry_safe(req, req_tmp, &conn_ctx->host_pend_req_list, queue_entry) { + list_del(&req->queue_entry); + } + spin_unlock(&conn_ctx->nvme_req_lock); + + if (atomic_read(&conn_ctx->num_active_fw_tasks)) { + conn_ctx->abrt_flag = QEDN_ABORTIVE_TERMINATION; + qedn_clear_fw_sq(conn_ctx); + qedn_cleanup_all_fw_tasks(conn_ctx); + } rc = qed_ops->destroy_conn(qedn->cdev, conn_ctx->conn_handle, conn_ctx->abrt_flag); diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c index a6756d7250b7..63a4e88d826d 100644 --- a/drivers/nvme/hw/qedn/qedn_main.c +++ b/drivers/nvme/hw/qedn/qedn_main.c @@ -323,6 +323,7 @@ static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid, size_t qedn_set_pdu_params(conn_ctx); init_waitqueue_head(&conn_ctx->conn_waitq); + init_waitqueue_head(&conn_ctx->cleanup_waitq); atomic_set(&conn_ctx->est_conn_indicator, 0); atomic_set(&conn_ctx->destroy_conn_indicator, 0); diff --git a/drivers/nvme/hw/qedn/qedn_task.c b/drivers/nvme/hw/qedn/qedn_task.c index 13d9fb6ed5b6..4ae6c0f66258 100644 --- a/drivers/nvme/hw/qedn/qedn_task.c +++ b/drivers/nvme/hw/qedn/qedn_task.c @@ -327,6 +327,17 @@ void qedn_return_active_tasks(struct qedn_conn_ctx *conn_ctx) /* Return tasks that aren't "Used by FW" to the pool */ list_for_each_entry_safe(qedn_task, task_tmp, &conn_ctx->active_task_list, entry) { + /* If we got this far, cleanup was already done + * in which case we want to return the task to the pool and + * release it. So we make sure the cleanup indication is down + */ + clear_bit(QEDN_TASK_WAIT_FOR_CLEANUP, &qedn_task->flags); + + /* Special handling in case of ICREQ task */ + if (unlikely(conn_ctx->state == CONN_STATE_WAIT_FOR_IC_COMP && + test_bit(QEDN_TASK_IS_ICREQ, &(qedn_task)->flags))) + qedn_common_clear_fw_sgl(&qedn_task->sgl_task_params); + qedn_clear_task(conn_ctx, qedn_task); num_returned_tasks++; } @@ -770,7 +781,8 @@ void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe) return; if (likely(cqe->cqe_type == NVMETCP_FW_CQE_TYPE_NORMAL)) { - /* Placeholder - verify the connection was established */ + if (unlikely(test_bit(QEDN_TASK_WAIT_FOR_CLEANUP, &qedn_task->flags))) + return; switch (cqe->task_type) { case NVMETCP_TASK_TYPE_HOST_WRITE: @@ -811,6 +823,17 @@ void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe) pr_info("Could not identify task type\n"); } } else { - /* Placeholder - Recovery flows */ + if (cqe->cqe_type == NVMETCP_FW_CQE_TYPE_CLEANUP) { + clear_bit(QEDN_TASK_WAIT_FOR_CLEANUP, &qedn_task->flags); + qedn_return_task_to_pool(conn_ctx, qedn_task); + atomic_dec(&conn_ctx->task_cleanups_cnt); + wake_up_interruptible(&conn_ctx->cleanup_waitq); + + return; + } + + /* The else is NVMETCP_FW_CQE_TYPE_DUMMY - in which don't return the task. + * The task will return during NVMETCP_FW_CQE_TYPE_CLEANUP. + */ } } From patchwork Thu Apr 29 19:09:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shai Malin X-Patchwork-Id: 429659 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DD18C43617 for ; Thu, 29 Apr 2021 19:18:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EE8F26144E for ; Thu, 29 Apr 2021 19:18:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233758AbhD2TR7 (ORCPT ); Thu, 29 Apr 2021 15:17:59 -0400 Received: from mx0a-0016f401.pphosted.com ([67.231.148.174]:24468 "EHLO mx0b-0016f401.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S239594AbhD2TM2 (ORCPT ); Thu, 29 Apr 2021 15:12:28 -0400 Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13TJ6QMN019652; Thu, 29 Apr 2021 12:11:32 -0700 Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com with ESMTP id 387erumw77-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 29 Apr 2021 12:11:32 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 29 Apr 2021 12:11:30 -0700 Received: from lbtlvb-pcie154.il.qlogic.org (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 29 Apr 2021 12:11:27 -0700 From: Shai Malin To: , , , , , CC: =?utf-8?q?David_S_=2E_Miller_davem_=40_davemloft_=2E_net_=C2=A0--cc?= =?utf-8?q?=3DJakub_Kicinski?= , , , , , , Subject: [RFC PATCH v4 27/27] qedn: Add support of ASYNC Date: Thu, 29 Apr 2021 22:09:26 +0300 Message-ID: <20210429190926.5086-28-smalin@marvell.com> X-Mailer: git-send-email 2.16.6 In-Reply-To: <20210429190926.5086-1-smalin@marvell.com> References: <20210429190926.5086-1-smalin@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: jZJA06uKBK6Om4KgjLtrVG3eQRAdBkRl X-Proofpoint-ORIG-GUID: jZJA06uKBK6Om4KgjLtrVG3eQRAdBkRl X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-04-29_10:2021-04-28,2021-04-29 signatures=0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Prabhakar Kushwaha This patch implement ASYNC request and response event notification handling at qedn driver level. NVME Ofld layer's ASYNC request is treated similar to read with fake CCCID. This CCCID used to route ASYNC notification back to the NVME ofld layer. Acked-by: Igor Russkikh Signed-off-by: Prabhakar Kushwaha Signed-off-by: Omkar Kulkarni Signed-off-by: Michal Kalderon Signed-off-by: Ariel Elior Signed-off-by: Shai Malin Reviewed-by: Hannes Reinecke --- drivers/nvme/hw/qedn/qedn.h | 8 ++ drivers/nvme/hw/qedn/qedn_main.c | 1 + drivers/nvme/hw/qedn/qedn_task.c | 156 +++++++++++++++++++++++++++++-- 3 files changed, 156 insertions(+), 9 deletions(-) diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h index fed4252392e0..067dc45027d4 100644 --- a/drivers/nvme/hw/qedn/qedn.h +++ b/drivers/nvme/hw/qedn/qedn.h @@ -109,6 +109,9 @@ #define QEDN_TASK_CLEANUP_TMO 3000 /* 3 sec */ #define QEDN_DRAIN_TMO 1000 /* 1 sec */ +#define QEDN_MAX_OUTSTAND_ASYNC 32 +#define QEDN_INVALID_CCCID (-1) + enum qedn_state { QEDN_STATE_CORE_PROBED = 0, QEDN_STATE_CORE_OPEN, @@ -196,6 +199,7 @@ struct qedn_ctx { enum qedn_task_flags { QEDN_TASK_IS_ICREQ, + QEDN_TASK_ASYNC, QEDN_TASK_USED_BY_FW, QEDN_TASK_WAIT_FOR_CLEANUP, }; @@ -373,6 +377,10 @@ struct qedn_conn_ctx { struct nvmetcp_icresp_hdr_psh icresp; struct qedn_icreq_padding *icreq_pad; + DECLARE_BITMAP(async_cccid_idx_map, QEDN_MAX_OUTSTAND_ASYNC); + /* Spinlock for fetching pseudo CCCID for async request */ + spinlock_t async_cccid_bitmap_lock; + /* "dummy" socket */ struct socket *sock; }; diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c index 63a4e88d826d..d2575767be47 100644 --- a/drivers/nvme/hw/qedn/qedn_main.c +++ b/drivers/nvme/hw/qedn/qedn_main.c @@ -328,6 +328,7 @@ static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid, size_t atomic_set(&conn_ctx->destroy_conn_indicator, 0); spin_lock_init(&conn_ctx->conn_state_lock); + spin_lock_init(&conn_ctx->async_cccid_bitmap_lock); INIT_WORK(&conn_ctx->nvme_req_fp_wq_entry, qedn_nvme_req_fp_wq_handler); conn_ctx->nvme_req_fp_wq = qedn->nvme_req_fp_wq; diff --git a/drivers/nvme/hw/qedn/qedn_task.c b/drivers/nvme/hw/qedn/qedn_task.c index 4ae6c0f66258..9ed6bc6c5c8a 100644 --- a/drivers/nvme/hw/qedn/qedn_task.c +++ b/drivers/nvme/hw/qedn/qedn_task.c @@ -259,10 +259,45 @@ void qedn_common_clear_fw_sgl(struct storage_sgl_task_params *sgl_task_params) sgl_task_params->num_sges = 0; } -inline void qedn_host_reset_cccid_itid_entry(struct qedn_conn_ctx *conn_ctx, - u16 cccid) +inline void qedn_host_reset_cccid_itid_entry(struct qedn_conn_ctx *conn_ctx, u16 cccid, bool async) { conn_ctx->host_cccid_itid[cccid].itid = cpu_to_le16(QEDN_INVALID_ITID); + if (unlikely(async)) + clear_bit(cccid - NVME_AQ_DEPTH, + conn_ctx->async_cccid_idx_map); +} + +static int qedn_get_free_idx(struct qedn_conn_ctx *conn_ctx, unsigned int size) +{ + int idx; + + spin_lock(&conn_ctx->async_cccid_bitmap_lock); + idx = find_first_zero_bit(conn_ctx->async_cccid_idx_map, size); + if (unlikely(idx >= size)) { + idx = -1; + spin_unlock(&conn_ctx->async_cccid_bitmap_lock); + goto err_idx; + } + set_bit(idx, conn_ctx->async_cccid_idx_map); + spin_unlock(&conn_ctx->async_cccid_bitmap_lock); + +err_idx: + + return idx; +} + +int qedn_get_free_async_cccid(struct qedn_conn_ctx *conn_ctx) +{ + int async_cccid; + + async_cccid = + qedn_get_free_idx(conn_ctx, QEDN_MAX_OUTSTAND_ASYNC); + if (unlikely(async_cccid == QEDN_INVALID_CCCID)) + pr_err("No available CCCID for Async.\n"); + else + async_cccid += NVME_AQ_DEPTH; + + return async_cccid; } inline void qedn_host_set_cccid_itid_entry(struct qedn_conn_ctx *conn_ctx, u16 cccid, u16 itid) @@ -363,10 +398,12 @@ void qedn_return_task_to_pool(struct qedn_conn_ctx *conn_ctx, struct qedn_fp_queue *fp_q = conn_ctx->fp_q; struct qedn_io_resources *io_resrc; unsigned long lock_flags; + bool async; io_resrc = &fp_q->host_resrc; spin_lock_irqsave(&qedn_task->lock, lock_flags); + async = test_bit(QEDN_TASK_ASYNC, &(qedn_task)->flags); qedn_task->valid = 0; qedn_task->flags = 0; qedn_clear_sgl(conn_ctx->qedn, qedn_task); @@ -374,7 +411,7 @@ void qedn_return_task_to_pool(struct qedn_conn_ctx *conn_ctx, spin_lock(&conn_ctx->task_list_lock); list_del(&qedn_task->entry); - qedn_host_reset_cccid_itid_entry(conn_ctx, qedn_task->cccid); + qedn_host_reset_cccid_itid_entry(conn_ctx, qedn_task->cccid, async); spin_unlock(&conn_ctx->task_list_lock); atomic_dec(&conn_ctx->num_active_tasks); @@ -447,6 +484,67 @@ qedn_get_task_from_pool_insist(struct qedn_conn_ctx *conn_ctx, u16 cccid) return qedn_task; } +void qedn_send_async_event_cmd(struct qedn_task_ctx *qedn_task, + struct qedn_conn_ctx *conn_ctx) +{ + struct nvme_tcp_ofld_req *async_req = qedn_task->req; + struct nvme_command *nvme_cmd = &async_req->nvme_cmd; + struct storage_sgl_task_params *sgl_task_params; + struct nvmetcp_task_params task_params; + struct nvmetcp_cmd_capsule_hdr cmd_hdr; + struct nvmetcp_conn_params conn_params; + struct nvmetcp_wqe *chain_sqe; + struct nvmetcp_wqe local_sqe; + int i; + + set_bit(QEDN_TASK_ASYNC, &qedn_task->flags); + nvme_cmd->common.command_id = qedn_task->cccid; + qedn_task->task_size = 0; + + /* Initialize sgl params */ + sgl_task_params = &qedn_task->sgl_task_params; + sgl_task_params->total_buffer_size = 0; + sgl_task_params->num_sges = 0; + sgl_task_params->small_mid_sge = false; + + task_params.opq.lo = cpu_to_le32(((u64)(qedn_task)) & 0xffffffff); + task_params.opq.hi = cpu_to_le32(((u64)(qedn_task)) >> 32); + + /* Initialize task params */ + task_params.context = qedn_task->fw_task_ctx; + task_params.sqe = &local_sqe; + task_params.tx_io_size = 0; + task_params.rx_io_size = 0; + task_params.conn_icid = (u16)conn_ctx->conn_handle; + task_params.itid = qedn_task->itid; + task_params.cq_rss_number = conn_ctx->default_cq; + task_params.send_write_incapsule = 0; + + /* Initialize conn params */ + conn_params.max_burst_length = QEDN_MAX_IO_SIZE; + + /* Internal impl. - async is treated like zero len read */ + cmd_hdr.chdr.pdu_type = nvme_tcp_cmd; + cmd_hdr.chdr.flags = 0; + cmd_hdr.chdr.hlen = sizeof(cmd_hdr); + cmd_hdr.chdr.pdo = 0x0; + cmd_hdr.chdr.plen_swapped = cpu_to_le32(__swab32(cmd_hdr.chdr.hlen)); + + for (i = 0; i < 16; i++) + cmd_hdr.pshdr.raw_swapped[i] = cpu_to_le32(__swab32(((u32 *)nvme_cmd)[i])); + + qed_ops->init_read_io(&task_params, &conn_params, &cmd_hdr, &qedn_task->sgl_task_params); + + set_bit(QEDN_TASK_USED_BY_FW, &qedn_task->flags); + atomic_inc(&conn_ctx->num_active_fw_tasks); + + spin_lock(&conn_ctx->ep.doorbell_lock); + chain_sqe = qed_chain_produce(&conn_ctx->ep.fw_sq_chain); + memcpy(chain_sqe, &local_sqe, sizeof(local_sqe)); + qedn_ring_doorbell(conn_ctx); + spin_unlock(&conn_ctx->ep.doorbell_lock); +} + int qedn_send_read_cmd(struct qedn_task_ctx *qedn_task, struct qedn_conn_ctx *conn_ctx) { struct nvme_command *nvme_cmd = &qedn_task->req->nvme_cmd; @@ -580,6 +678,24 @@ static void qedn_fetch_request(struct qedn_conn_ctx *qedn_conn) spin_unlock(&qedn_conn->nvme_req_lock); } +static void qedn_return_error_req(struct nvme_tcp_ofld_req *req) +{ + __le16 status = cpu_to_le16(NVME_SC_HOST_PATH_ERROR << 1); + union nvme_result res = {}; + struct request *rq; + + if (!req) + return; + + rq = blk_mq_rq_from_pdu(req); + + /* Call request done to compelete the request */ + if (req->done) + req->done(req, &res, status); + else + pr_err("request done not set !!!\n"); +} + static bool qedn_process_req(struct qedn_conn_ctx *qedn_conn) { struct qedn_task_ctx *qedn_task; @@ -595,9 +711,16 @@ static bool qedn_process_req(struct qedn_conn_ctx *qedn_conn) req = qedn_conn->req; rq = blk_mq_rq_from_pdu(req); - /* Placeholder - async */ + if (unlikely(req->async)) { + cccid = qedn_get_free_async_cccid(qedn_conn); + if (cccid == QEDN_INVALID_CCCID) { + qedn_return_error_req(req); + goto doorbell; + } + } else { + cccid = rq->tag; + } - cccid = rq->tag; qedn_task = qedn_get_task_from_pool_insist(qedn_conn, cccid); if (unlikely(!qedn_task)) { pr_err("Not able to allocate task context\n"); @@ -607,7 +730,10 @@ static bool qedn_process_req(struct qedn_conn_ctx *qedn_conn) req->private_data = qedn_task; qedn_task->req = req; - /* Placeholder - handle (req->async) */ + if (unlikely(req->async)) { + qedn_send_async_event_cmd(qedn_task, qedn_conn); + goto doorbell; + } /* Check if there are physical segments in request to determine the task size. * The logic of nvme_tcp_set_sg_null() will be implemented as part of @@ -732,14 +858,26 @@ static inline int qedn_comp_valid_task(struct qedn_task_ctx *qedn_task, int qedn_process_nvme_cqe(struct qedn_task_ctx *qedn_task, struct nvme_completion *cqe) { + struct qedn_conn_ctx *conn_ctx = qedn_task->qedn_conn; + struct nvme_tcp_ofld_req *req; int rc = 0; + bool async; + + async = test_bit(QEDN_TASK_ASYNC, &(qedn_task)->flags); /* cqe arrives swapped */ qedn_swap_bytes((u32 *)cqe, (sizeof(*cqe) / sizeof(u32))); - /* Placeholder - async */ - - rc = qedn_comp_valid_task(qedn_task, &cqe->result, cqe->status); + if (unlikely(async)) { + qedn_return_task_to_pool(conn_ctx, qedn_task); + req = qedn_task->req; + if (req->done) + req->done(req, &cqe->result, cqe->status); + else + pr_err("request done not set for async request !!!\n"); + } else { + rc = qedn_comp_valid_task(qedn_task, &cqe->result, cqe->status); + } return rc; }