From patchwork Tue Feb 16 16:20:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eli Cohen X-Patchwork-Id: 383647 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51C4EC433DB for ; Tue, 16 Feb 2021 16:21:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1997164DE0 for ; Tue, 16 Feb 2021 16:21:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230303AbhBPQUs (ORCPT ); Tue, 16 Feb 2021 11:20:48 -0500 Received: from hqnvemgate25.nvidia.com ([216.228.121.64]:17089 "EHLO hqnvemgate25.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230204AbhBPQUr (ORCPT ); Tue, 16 Feb 2021 11:20:47 -0500 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate25.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 16 Feb 2021 08:20:06 -0800 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 16 Feb 2021 16:20:05 +0000 Received: from vdi.nvidia.com (172.20.145.6) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Tue, 16 Feb 2021 16:20:04 +0000 From: Eli Cohen To: , , , , , CC: Subject: [PATCH 1/2] vdpa/mlx5: Fix suspend/resume index restoration Date: Tue, 16 Feb 2021 18:20:01 +0200 Message-ID: <20210216162001.83541-1-elic@nvidia.com> X-Mailer: git-send-email 2.28.0 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1613492406; bh=kc8WQQ00x8KDTal1OlNbrMI+gHFHE5f0MPt31tY4EDc=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:MIME-Version: Content-Transfer-Encoding:Content-Type; b=TQr5ZunPlptIn1FcOetUBfF5zVR0WjNhWUFeVAzcA5BAKo2gDXh5/EP/EJLpOPaYQ vKO+vc7JUOUWCLMtYIlRBtoKr4eVaicgdvw2Kv5ayqMFqctAUCoKXPGGI6z6AJrRXm /v8n5fpkHcrtt+LI8t/U0H8shg9TeykW8iNvCyLlbQgSWJ4wM4cnM7TXZqqCNN8dlH 7Nprp2BaNT/zIFoDTgyYQ+0cpj8SynrYGb4ZzWg7MmWfw/faP3LDSF2eovm3JReVNC 9ez9uPn8FnNAODATGUninHj6tkXBQ+pvrddpETJGulDYlmv7SKegCz/fuii+KNHgH/ Y0vrTwzcojS5Q== Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org When we suspend the VM, the VDPA interface will be reset. When the VM is resumed again, clear_virtqueues() will clear the available and used indices resulting in hardware virqtqueue objects becoming out of sync. We can avoid this function alltogether since qemu will clear them if required, e.g. when the VM went through a reboot. Moreover, since the hw available and used indices should always be identical on query and should be restored to the same value same value for virtqueues that complete in order, we set the single value provided by set_vq_state(). In get_vq_state() we return the value of hardware used index. Fixes: 1a86b377aa21 ("vdpa/mlx5: Add VDPA driver for supported mlx5 devices") Signed-off-by: Eli Cohen Acked-by: Si-Wei Liu --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index b8e9d525d66c..a51b0f86afe2 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -1169,6 +1169,7 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *m return; } mvq->avail_idx = attr.available_index; + mvq->used_idx = attr.used_index; } static void suspend_vqs(struct mlx5_vdpa_net *ndev) @@ -1426,6 +1427,7 @@ static int mlx5_vdpa_set_vq_state(struct vdpa_device *vdev, u16 idx, return -EINVAL; } + mvq->used_idx = state->avail_index; mvq->avail_idx = state->avail_index; return 0; } @@ -1443,7 +1445,7 @@ static int mlx5_vdpa_get_vq_state(struct vdpa_device *vdev, u16 idx, struct vdpa * that cares about emulating the index after vq is stopped. */ if (!mvq->initialized) { - state->avail_index = mvq->avail_idx; + state->avail_index = mvq->used_idx; return 0; } @@ -1452,7 +1454,7 @@ static int mlx5_vdpa_get_vq_state(struct vdpa_device *vdev, u16 idx, struct vdpa mlx5_vdpa_warn(mvdev, "failed to query virtqueue\n"); return err; } - state->avail_index = attr.available_index; + state->avail_index = attr.used_index; return 0; } @@ -1532,16 +1534,6 @@ static void teardown_virtqueues(struct mlx5_vdpa_net *ndev) } } -static void clear_virtqueues(struct mlx5_vdpa_net *ndev) -{ - int i; - - for (i = ndev->mvdev.max_vqs - 1; i >= 0; i--) { - ndev->vqs[i].avail_idx = 0; - ndev->vqs[i].used_idx = 0; - } -} - /* TODO: cross-endian support */ static inline bool mlx5_vdpa_is_little_endian(struct mlx5_vdpa_dev *mvdev) { @@ -1777,7 +1769,6 @@ static void mlx5_vdpa_set_status(struct vdpa_device *vdev, u8 status) if (!status) { mlx5_vdpa_info(mvdev, "performing device reset\n"); teardown_driver(ndev); - clear_virtqueues(ndev); mlx5_vdpa_destroy_mr(&ndev->mvdev); ndev->mvdev.status = 0; ++mvdev->generation; From patchwork Tue Feb 16 16:20:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eli Cohen X-Patchwork-Id: 384297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A963C433E0 for ; Tue, 16 Feb 2021 16:21:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0633A64DE0 for ; Tue, 16 Feb 2021 16:21:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230418AbhBPQVY (ORCPT ); Tue, 16 Feb 2021 11:21:24 -0500 Received: from hqnvemgate26.nvidia.com ([216.228.121.65]:16648 "EHLO hqnvemgate26.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230077AbhBPQVT (ORCPT ); Tue, 16 Feb 2021 11:21:19 -0500 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate26.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 16 Feb 2021 08:20:37 -0800 Received: from HQMAIL109.nvidia.com (172.20.187.15) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 16 Feb 2021 16:20:36 +0000 Received: from vdi.nvidia.com (172.20.145.6) by mail.nvidia.com (172.20.187.15) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Tue, 16 Feb 2021 16:20:34 +0000 From: Eli Cohen To: , , , , , CC: , Parav Pandit Subject: [PATCH 2/2] vdpa/mlx5: Enable user to add/delete vdpa device Date: Tue, 16 Feb 2021 18:20:29 +0200 Message-ID: <20210216162029.83637-1-elic@nvidia.com> X-Mailer: git-send-email 2.28.0 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1613492437; bh=DVT9bjkFDc3JSWqhXK/2NCb6cTnrMC5RutdCMqES0Hs=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:MIME-Version: Content-Transfer-Encoding:Content-Type; b=LT2QJIhnSjQRuc4EqDax3zkHE+adsi2eqCpoPRblV7cj0YgaCYx3JM+rU0ToA5vZz PENEh/CGt7dnySIszTzLv/VkdUyy0RXejpkZ7zn3OrxMhAto2DdIxrkUPP1S/r+Dh2 AkR1J0CkyMSdDSTpP+Q8X7tXLWnW46Xposws3+i7CjCdNTU/dFU7Ao3Hmu1JLlLVVE Vy4TQhH6Cw9l7EUekWnYAkXJi/OJMPMNht0/ck/+1i5x2Z1m5GFhdoccpsL12a+Ufs 8OEDGw1KJjU77FXlu5Y5umwmKIJ38uAyTeUUl2J7JpX4q9/rRZRYgfhoz3UymSVywI PbBAi9YDkLBGA== Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Allow to control vdpa device creation and destruction using the vdpa management tool. Examples: 1. List the management devices $ vdpa mgmtdev show pci/0000:3b:00.1: supported_classes net 2. Create vdpa instance $ vdpa dev add mgmtdev pci/0000:3b:00.1 name vdpa0 3. Show vdpa devices $ vdpa dev show vdpa0: type network mgmtdev pci/0000:3b:00.1 vendor_id 5555 max_vqs 16 \ max_vq_size 256 Signed-off-by: Eli Cohen Reviewed-by: Parav Pandit --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 76 +++++++++++++++++++++++++++---- 1 file changed, 67 insertions(+), 9 deletions(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index a51b0f86afe2..acf7dc61acb0 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -1974,23 +1974,32 @@ static void init_mvqs(struct mlx5_vdpa_net *ndev) } } -static int mlx5v_probe(struct auxiliary_device *adev, - const struct auxiliary_device_id *id) +struct mlx5_vdpa_mgmtdev { + struct vdpa_mgmt_dev mgtdev; + struct mlx5_adev *madev; + struct mlx5_vdpa_net *ndev; +}; + +static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name) { - struct mlx5_adev *madev = container_of(adev, struct mlx5_adev, adev); - struct mlx5_core_dev *mdev = madev->mdev; + struct mlx5_vdpa_mgmtdev *mgtdev = container_of(v_mdev, struct mlx5_vdpa_mgmtdev, mgtdev); struct virtio_net_config *config; struct mlx5_vdpa_dev *mvdev; struct mlx5_vdpa_net *ndev; + struct mlx5_core_dev *mdev; u32 max_vqs; int err; + if (mgtdev->ndev) + return -ENOSPC; + + mdev = mgtdev->madev->mdev; /* we save one virtqueue for control virtqueue should we require it */ max_vqs = MLX5_CAP_DEV_VDPA_EMULATION(mdev, max_num_virtio_queues); max_vqs = min_t(u32, max_vqs, MLX5_MAX_SUPPORTED_VQS); ndev = vdpa_alloc_device(struct mlx5_vdpa_net, mvdev.vdev, mdev->device, &mlx5_vdpa_ops, - 2 * mlx5_vdpa_max_qps(max_vqs), NULL); + 2 * mlx5_vdpa_max_qps(max_vqs), name); if (IS_ERR(ndev)) return PTR_ERR(ndev); @@ -2018,11 +2027,12 @@ static int mlx5v_probe(struct auxiliary_device *adev, if (err) goto err_res; - err = vdpa_register_device(&mvdev->vdev); + mvdev->vdev.mdev = &mgtdev->mgtdev; + err = _vdpa_register_device(&mvdev->vdev); if (err) goto err_reg; - dev_set_drvdata(&adev->dev, ndev); + mgtdev->ndev = ndev; return 0; err_reg: @@ -2035,11 +2045,59 @@ static int mlx5v_probe(struct auxiliary_device *adev, return err; } +static void mlx5_vdpa_dev_del(struct vdpa_mgmt_dev *v_mdev, struct vdpa_device *dev) +{ + _vdpa_unregister_device(dev); +} + +static const struct vdpa_mgmtdev_ops mdev_ops = { + .dev_add = mlx5_vdpa_dev_add, + .dev_del = mlx5_vdpa_dev_del, +}; + +static struct virtio_device_id id_table[] = { + { VIRTIO_ID_NET, VIRTIO_DEV_ANY_ID }, + { 0 }, +}; + +static int mlx5v_probe(struct auxiliary_device *adev, + const struct auxiliary_device_id *id) + +{ + struct mlx5_adev *madev = container_of(adev, struct mlx5_adev, adev); + struct mlx5_core_dev *mdev = madev->mdev; + struct mlx5_vdpa_mgmtdev *mgtdev; + int err; + + mgtdev = kzalloc(sizeof(*mgtdev), GFP_KERNEL); + if (!mgtdev) + return -ENOMEM; + + mgtdev->mgtdev.ops = &mdev_ops; + mgtdev->mgtdev.device = mdev->device; + mgtdev->mgtdev.id_table = id_table; + mgtdev->madev = madev; + + err = vdpa_mgmtdev_register(&mgtdev->mgtdev); + if (err) + goto reg_err; + + dev_set_drvdata(&adev->dev, mgtdev); + + return 0; + +reg_err: + kfree(mdev); + return err; +} + static void mlx5v_remove(struct auxiliary_device *adev) { - struct mlx5_vdpa_dev *mvdev = dev_get_drvdata(&adev->dev); + struct mlx5_vdpa_mgmtdev *mgtdev; - vdpa_unregister_device(&mvdev->vdev); + mgtdev = dev_get_drvdata(&adev->dev); + vdpa_mgmtdev_unregister(&mgtdev->mgtdev); + kfree(mgtdev); } static const struct auxiliary_device_id mlx5v_id_table[] = {