mbox series

[0/4] RDMA/mlx5: Support DMABUF in umems and enable ATS

Message ID 0-v1-bd147097458e+ede-umem_dmabuf_jgg@nvidia.com
Headers show
Series RDMA/mlx5: Support DMABUF in umems and enable ATS | expand

Message

Jason Gunthorpe Sept. 1, 2022, 2:20 p.m. UTC 
This series adds support for DMABUF when creating a devx umem. devx umems
are quite similar to MR's execpt they cannot be revoked, so this uses the
dmabuf pinned memory flow. Several mlx5dv flows require umem and cannot
work with MR.

The intended use case is primarily for P2P transfers using dmabuf as a
handle to the underlying PCI BAR memory from the exporter. When a PCI
switch is present the P2P transfers can bypass the host bridge completely
and go directly through the switch. ATS allows this bypass to function in
more cases as translated TLPs issued after an ATS query allows the request
redirect setting to be bypassed in the switch.

Have mlx5 automatically use ATS in places where it makes sense.

Jason Gunthorpe (4):
  net/mlx5: Add IFC bits for mkey ATS
  RDMA/core: Add UVERBS_ATTR_RAW_FD
  RDMA/mlx5: Add support for dmabuf to devx umem
  RDMA/mlx5: Enable ATS support for MRs and umems

 drivers/infiniband/core/uverbs_ioctl.c   |  8 ++++
 drivers/infiniband/hw/mlx5/devx.c        | 55 +++++++++++++++++-------
 drivers/infiniband/hw/mlx5/mlx5_ib.h     | 36 ++++++++++++++++
 drivers/infiniband/hw/mlx5/mr.c          |  5 ++-
 include/linux/mlx5/mlx5_ifc.h            | 11 +++--
 include/rdma/uverbs_ioctl.h              | 13 ++++++
 include/uapi/rdma/mlx5_user_ioctl_cmds.h |  1 +
 7 files changed, 109 insertions(+), 20 deletions(-)


base-commit: b90cb1053190353cc30f0fef0ef1f378ccc063c5

Comments

Jason Gunthorpe Sept. 26, 2022, 5:51 p.m. UTC | #1 
On Thu, Sep 01, 2022 at 11:20:52AM -0300, Jason Gunthorpe wrote:
> This series adds support for DMABUF when creating a devx umem. devx umems
> are quite similar to MR's execpt they cannot be revoked, so this uses the
> dmabuf pinned memory flow. Several mlx5dv flows require umem and cannot
> work with MR.
> 
> The intended use case is primarily for P2P transfers using dmabuf as a
> handle to the underlying PCI BAR memory from the exporter. When a PCI
> switch is present the P2P transfers can bypass the host bridge completely
> and go directly through the switch. ATS allows this bypass to function in
> more cases as translated TLPs issued after an ATS query allows the request
> redirect setting to be bypassed in the switch.
> 
> Have mlx5 automatically use ATS in places where it makes sense.
> 
> Jason Gunthorpe (4):
>   net/mlx5: Add IFC bits for mkey ATS
>   RDMA/core: Add UVERBS_ATTR_RAW_FD
>   RDMA/mlx5: Add support for dmabuf to devx umem
>   RDMA/mlx5: Enable ATS support for MRs and umems
> 
>  drivers/infiniband/core/uverbs_ioctl.c   |  8 ++++
>  drivers/infiniband/hw/mlx5/devx.c        | 55 +++++++++++++++++-------
>  drivers/infiniband/hw/mlx5/mlx5_ib.h     | 36 ++++++++++++++++
>  drivers/infiniband/hw/mlx5/mr.c          |  5 ++-
>  include/linux/mlx5/mlx5_ifc.h            | 11 +++--
>  include/rdma/uverbs_ioctl.h              | 13 ++++++
>  include/uapi/rdma/mlx5_user_ioctl_cmds.h |  1 +
>  7 files changed, 109 insertions(+), 20 deletions(-)

Applied to for-next, thanks

Jason