mbox series

[v8,0/9] vfio/hisilicon: add ACC live migration driver

Message ID 20220303230131.2103-1-shameerali.kolothum.thodi@huawei.com
Headers show
Series vfio/hisilicon: add ACC live migration driver | expand

Message

Shameerali Kolothum Thodi March 3, 2022, 11:01 p.m. UTC 
Hi,

This series attempts to add vfio live migration support for HiSilicon
ACC VF devices based on the new v2 migration protocol definition and
mlx5 v9 series discussed here[0].

v7 --> v8
 - Dropped PRE_COPY support and early compatibility checking based on
   the discussion here[1].
 - Addressed comments from John, Jason & Alex (Thanks!).

This is sanity tested on a HiSilicon platform using the Qemu branch
provided here[2].

Please take a look and let me know your feedback.

Thanks,
Shameer
[0] https://lore.kernel.org/kvm/20220224142024.147653-1-yishaih@nvidia.com/
[1] https://lore.kernel.org/kvm/20220302133159.3c803f56.alex.williamson@redhat.com/
[2] https://github.com/jgunthorpe/qemu/commits/vfio_migration_v2

v6 --> v7
 -Renamed MIG_PRECOPY ioctl name and struct name. Updated ioctl descriptions
  regarding ioctl validity (patch #7).
- Adressed comments from Jason and Alex on PRE_COPY read() and ioctl() fns
  (patch #9).
- Moved only VF PCI ids to pci_ids.h(patch #3).

v5 --> v6
 -Report PRE_COPY support and use that for early compatibility check
  between src and dst devices.
 -For generic PRE_COPY support, included patch #7 from Jason(Thanks!).
 -Addressed comments from Alex(Thanks!).
 -Added the QM state register update to QM driver(patch #8) since that
  is being used in migration driver to decide whether the device is
  ready to save the state.

RFCv4 --> v5
  - Dropped RFC tag as v2 migration APIs are more stable now.
  - Addressed review comments from Jason and Alex (Thanks!).

v3 --> RFCv4
-Based on migration v2 protocol and mlx5 v7 series.
-Added RFC tag again as migration v2 protocol is still under discussion.
-Added new patch #6 to retrieve the PF QM data.
-PRE_COPY compatibility check is now done after the migration data
 transfer. This is not ideal and needs discussion.

RFC v2 --> v3
 -Dropped RFC tag as the vfio_pci_core subsystem framework is now
  part of 5.15-rc1.
 -Added override methods for vfio_device_ops read/write/mmap calls
  to limit the access within the functional register space.
 -Patches 1 to 3 are code refactoring to move the common ACC QM
  definitions and header around.

RFCv1 --> RFCv2

 -Adds a new vendor-specific vfio_pci driver(hisi-acc-vfio-pci)
  for HiSilicon ACC VF devices based on the new vfio-pci-core
  framework proposal.

 -Since HiSilicon ACC VF device MMIO space contains both the
  functional register space and migration control register space,
  override the vfio_device_ops ioctl method to report only the
  functional space to VMs.

 -For a successful migration, we still need access to VF dev
  functional register space mainly to read the status registers.
  But accessing these while the Guest vCPUs are running may leave
  a security hole. To avoid any potential security issues, we
  map/unmap the MMIO regions on a need basis and is safe to do so.
  (Please see hisi_acc_vf_ioremap/unmap() fns in patch #4).
 
 -Dropped debugfs support for now.
 -Uses common QM functions for mailbox access(patch #3).

Longfang Liu (3):
  crypto: hisilicon/qm: Move few definitions to common header
  crypto: hisilicon/qm: Set the VF QM state register
  hisi_acc_vfio_pci: Add support for VFIO live migration

Shameer Kolothum (6):
  crypto: hisilicon/qm: Move the QM header to include/linux
  hisi_acc_qm: Move VF PCI device IDs to common header
  hisi_acc_vfio_pci: add new vfio_pci driver for HiSilicon ACC devices
  hisi_acc_vfio_pci: Restrict access to VF dev BAR2 migration region
  hisi_acc_vfio_pci: Add helper to retrieve the struct pci_driver
  hisi_acc_vfio_pci: Use its own PCI reset_done error handler

 drivers/crypto/hisilicon/hpre/hpre.h          |    2 +-
 drivers/crypto/hisilicon/hpre/hpre_main.c     |   19 +-
 drivers/crypto/hisilicon/qm.c                 |   68 +-
 drivers/crypto/hisilicon/sec2/sec.h           |    2 +-
 drivers/crypto/hisilicon/sec2/sec_main.c      |   21 +-
 drivers/crypto/hisilicon/sgl.c                |    2 +-
 drivers/crypto/hisilicon/zip/zip.h            |    2 +-
 drivers/crypto/hisilicon/zip/zip_main.c       |   17 +-
 drivers/vfio/pci/Kconfig                      |    2 +
 drivers/vfio/pci/Makefile                     |    2 +
 drivers/vfio/pci/hisilicon/Kconfig            |   17 +
 drivers/vfio/pci/hisilicon/Makefile           |    4 +
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 1319 +++++++++++++++++
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |  116 ++
 .../qm.h => include/linux/hisi_acc_qm.h       |   49 +
 include/linux/pci_ids.h                       |    3 +
 16 files changed, 1579 insertions(+), 66 deletions(-)
 create mode 100644 drivers/vfio/pci/hisilicon/Kconfig
 create mode 100644 drivers/vfio/pci/hisilicon/Makefile
 create mode 100644 drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
 create mode 100644 drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
 rename drivers/crypto/hisilicon/qm.h => include/linux/hisi_acc_qm.h (87%)

Comments

Shameerali Kolothum Thodi March 4, 2022, 11:33 a.m. UTC | #1 
[+ Zaibo]

> -----Original Message-----
> From: Wangzhou (B)
> Sent: 04 March 2022 09:03
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>;
> kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-crypto@vger.kernel.org
> Cc: linux-pci@vger.kernel.org; alex.williamson@redhat.com; jgg@nvidia.com;
> cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com; Linuxarm
> <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>; Zengtao (B)
> <prime.zeng@hisilicon.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>
> Subject: Re: [PATCH v8 1/9] crypto: hisilicon/qm: Move the QM header to
> include/linux
> 
> > Since we are going to introduce VFIO PCI HiSilicon ACC driver for live
> > migration in subsequent patches, move the ACC QM header file to a
> > common include dir.
> >
> > Signed-off-by: Shameer Kolothum
> <shameerali.kolothum.thodi@huawei.com>
> 
> Hi Shameer,
> 
> It looks good to me for this movement.
> 
> Acked-by: Zhou Wang <wangzhou1@hisilicon.com>

Thanks. [+cc Zaibo] for hpre/sec part.


> 
> > ---
> >  drivers/crypto/hisilicon/hpre/hpre.h                         | 2 +-
> >  drivers/crypto/hisilicon/qm.c                                | 2 +-
> >  drivers/crypto/hisilicon/sec2/sec.h                          | 2 +-
> >  drivers/crypto/hisilicon/sgl.c                               | 2 +-
> >  drivers/crypto/hisilicon/zip/zip.h                           | 2 +-
> >  drivers/crypto/hisilicon/qm.h => include/linux/hisi_acc_qm.h | 0
> >  6 files changed, 5 insertions(+), 5 deletions(-)
> >  rename drivers/crypto/hisilicon/qm.h => include/linux/hisi_acc_qm.h
> (100%)
> >
> > diff --git a/drivers/crypto/hisilicon/hpre/hpre.h
> b/drivers/crypto/hisilicon/hpre/hpre.h
> > index e0b4a1982ee9..9a0558ed82f9 100644
> > --- a/drivers/crypto/hisilicon/hpre/hpre.h
> > +++ b/drivers/crypto/hisilicon/hpre/hpre.h
> > @@ -4,7 +4,7 @@
> >  #define __HISI_HPRE_H
> >
> >  #include <linux/list.h>
> > -#include "../qm.h"
> > +#include <linux/hisi_acc_qm.h>
> >
> >  #define HPRE_SQE_SIZE			sizeof(struct hpre_sqe)
> >  #define HPRE_PF_DEF_Q_NUM		64
> > diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
> > index c5b84a5ea350..ed23e1d3fa27 100644
> > --- a/drivers/crypto/hisilicon/qm.c
> > +++ b/drivers/crypto/hisilicon/qm.c
> > @@ -15,7 +15,7 @@
> >  #include <linux/uacce.h>
> >  #include <linux/uaccess.h>
> >  #include <uapi/misc/uacce/hisi_qm.h>
> > -#include "qm.h"
> > +#include <linux/hisi_acc_qm.h>
> >
> >  /* eq/aeq irq enable */
> >  #define QM_VF_AEQ_INT_SOURCE		0x0
> > diff --git a/drivers/crypto/hisilicon/sec2/sec.h
> b/drivers/crypto/hisilicon/sec2/sec.h
> > index d97cf02b1df7..c2e9b01187a7 100644
> > --- a/drivers/crypto/hisilicon/sec2/sec.h
> > +++ b/drivers/crypto/hisilicon/sec2/sec.h
> > @@ -4,7 +4,7 @@
> >  #ifndef __HISI_SEC_V2_H
> >  #define __HISI_SEC_V2_H
> >
> > -#include "../qm.h"
> > +#include <linux/hisi_acc_qm.h>
> >  #include "sec_crypto.h"
> >
> >  /* Algorithm resource per hardware SEC queue */
> > diff --git a/drivers/crypto/hisilicon/sgl.c b/drivers/crypto/hisilicon/sgl.c
> > index 057273769f26..f7efc02b065f 100644
> > --- a/drivers/crypto/hisilicon/sgl.c
> > +++ b/drivers/crypto/hisilicon/sgl.c
> > @@ -1,9 +1,9 @@
> >  // SPDX-License-Identifier: GPL-2.0
> >  /* Copyright (c) 2019 HiSilicon Limited. */
> >  #include <linux/dma-mapping.h>
> > +#include <linux/hisi_acc_qm.h>
> >  #include <linux/module.h>
> >  #include <linux/slab.h>
> > -#include "qm.h"
> >
> >  #define HISI_ACC_SGL_SGE_NR_MIN		1
> >  #define HISI_ACC_SGL_NR_MAX		256
> > diff --git a/drivers/crypto/hisilicon/zip/zip.h
> b/drivers/crypto/hisilicon/zip/zip.h
> > index 517fdbdff3ea..3dfd3bac5a33 100644
> > --- a/drivers/crypto/hisilicon/zip/zip.h
> > +++ b/drivers/crypto/hisilicon/zip/zip.h
> > @@ -7,7 +7,7 @@
> >  #define pr_fmt(fmt)	"hisi_zip: " fmt
> >
> >  #include <linux/list.h>
> > -#include "../qm.h"
> > +#include <linux/hisi_acc_qm.h>
> >
> >  enum hisi_zip_error_type {
> >  	/* negative compression */
> > diff --git a/drivers/crypto/hisilicon/qm.h b/include/linux/hisi_acc_qm.h
> > similarity index 100%
> > rename from drivers/crypto/hisilicon/qm.h
> > rename to include/linux/hisi_acc_qm.h
> >
Jason Gunthorpe March 4, 2022, 8:54 p.m. UTC | #2 
On Thu, Mar 03, 2022 at 11:01:31PM +0000, Shameer Kolothum wrote:
> Register private handler for pci_error_handlers.reset_done and update
> state accordingly.
> 
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> ---
>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 57 ++++++++++++++++++-
>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |  4 +-
>  2 files changed, 57 insertions(+), 4 deletions(-)

It looks OK to me

Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>

Jason