| Message ID | 20250424141341.841734-1-karim.manaouil@linaro.org |
|---|---|
| Headers | show |
| Series | Running Qualcomm's Gunyah Guests via KVM in EL1 | expand |
On Thu, Apr 24, 2025 at 08:34:50AM -0700, Oliver Upton wrote: > On Thu, Apr 24, 2025 at 03:13:07PM +0100, Karim Manaouil wrote: > > This series introduces the capability of running Gunyah guests via KVM on > > Qualcomm SoCs shipped with Gunyah hypervisor [1] (e.g. RB3 Gen2). > > > > The goal of this work is to port the existing Gunyah hypervisor support from a > > standalone driver interface [2] to KVM, with the aim of leveraging as much of the > > existing KVM infrastructure as possible to reduce duplication of effort around > > memory management (e.g. guest_memfd), irqfd, and other core components. > > > > In short, Gunyah is a Type-1 hypervisor, meaning that it runs independently of any > > high-level OS kernel such as Linux and runs in a higher CPU privilege level than VMs. > > Gunyah is shipped as firmware and guests typically talk with Gunyah via hypercalls. > > KVM is designed to run as Type-2 hypervisor. This port allows KVM to run in EL1 and > > serve as the interface for VM lifecycle management,while offloading virtualization > > to Gunyah. > > If you're keen on running your own hypervisor then I'm sorry, you get to > deal with it soup to nuts. Other hypervisors (e.g. mshv) have their own > kernel drivers for managing the host / UAPI parts of driving VMs. > > The KVM arch interface is *internal* to KVM, not something to be > (ab)used for cramming in a non-KVM hypervisor. KVM and other hypervisors > can still share other bits of truly common infrastructure, like > guest_memfd. > > I understand the value in what you're trying to do, but if you want it > to smell like KVM you may as well just let the user run it at EL2. Hi Oliver, Thanks for your reply! I very much expected this take and my discussion with Qauclomm engineers also more or less had the same conclusion. There was a previous effort to go into this direction, but it never got implemented, so this was my (no bullshit) attempt at doing that! I believe that Qcom are more interested in having their own driver, instead (as Trilok pointed out in another thread). But this port could still be interesting for those who wanted to see KVM in EL1. On the positive side, you get all the userspace tools (Qemu, Crosvm, kvmtool, cloud-hypervisor, firecracker) working with minimal effort. Cheers Karim
This series introduces the capability of running Gunyah guests via KVM on Qualcomm SoCs shipped with Gunyah hypervisor [1] (e.g. RB3 Gen2). The goal of this work is to port the existing Gunyah hypervisor support from a standalone driver interface [2] to KVM, with the aim of leveraging as much of the existing KVM infrastructure as possible to reduce duplication of effort around memory management (e.g. guest_memfd), irqfd, and other core components. In short, Gunyah is a Type-1 hypervisor, meaning that it runs independently of any high-level OS kernel such as Linux and runs in a higher CPU privilege level than VMs. Gunyah is shipped as firmware and guests typically talk with Gunyah via hypercalls. KVM is designed to run as Type-2 hypervisor. This port allows KVM to run in EL1 and serve as the interface for VM lifecycle management,while offloading virtualization to Gunyah. This series is heavily based on previous work from Elliot Berman and others, available at: https://lore.kernel.org/lkml/20240222-gunyah-v17-0-1e9da6763d38@quicinc.com/ Many commits in this series are identical to or derived from previous work. To preserve authorship and attribution, original `Author:` and `Signed-off-by:` tags have been retained where appropriate, and occasionally `Reviewed-by:` lines were kept as well. While this series builds on much of the original Gunyah implementation, it drops certain parts in favor of existing upstream features: - `gunyah_memfd` is dropped (currently using pinned anonymous pages — see below). - `gunyah_irqfd` is dropped in favor of KVM's irqfd. - Resource management, vCPU creation, and VM lifecycle are integrated directly into KVM’s architecture hooks. At this stage, the port is functional but still **work in progress**. Notably: - Memory for guests is currently backed by **pinned anonymous pages**. This is a temporary solution: we intend to migrate to `guest_memfd`. - Memory compaction or swap must be avoided for now, as pages donated to Gunyah are no longer accessible from the host once mapped into the guest. - SMP boot is not available at the moment because Gunyah does not yet forward PSCI hypercalls to the host. - Virtio is not supported yet. There is a lot of room for performance improvment. For example, ATM. there is a lot of privilege level switching between EL1, EL2 (and possible EL3). As well as context swicthing between guest kernel space and guest userspace. For those interested, I have a compiled some slides here to summarise the issues: https://docs.google.com/presentation/d/1fL1TM1oxBrWFSL8KKw4jQtMDh7NOvyqbWWVu8iQxvoI/edit?usp=sharing To test the port, a modified version of Qemu is needed. A version can be found here: https://github.com/karim-manaouil/qemu-for-gunyah The series is based on linux-next as of today. A git tree is also available here (gunyah-kvm branch): https://github.com/karim-manaouil/linux-next To test the guest, run (with the modified qemu): ./qemu-system-aarch64 \ -enable-kvm \ -machine virt,highmem=off \ -cpu host \ -smp 1 \ -m 1G \ -nographic \ -kernel Image \ -initrd initrd.img \ -append "console=ttyAMA0 earlycon rdinit=/bin/sh" Feedback Welcome! [1] https://www.qualcomm.com/developer/blog/2024/08/learn-about-gunyah--qualcomm-s-open-source--lightweight-hypervis [2] https://lore.kernel.org/lkml/20240222-gunyah-v17-0-1e9da6763d38@quicinc.com/ Elliot Berman (20): docs: gunyah: Introduce Gunyah Hypervisor dt-bindings: Add binding for gunyah hypervisor gunyah: Common types and error codes for Gunyah hypercalls gunyah: Add hypercalls to identify Gunyah gunyah: Add hypervisor driver gunyah: Add hypercalls to send and receive messages gunyah: Add resource manager RPC core gunyah: Add VM lifecycle RPC gunyah: Translate gh_rm_hyp_resource into gunyah_resource gunyah: Add resource tickets gunyah: Add hypercalls for running a vCPU gunyah: Add hypercalls for demand paging gunyah: Add memory parcel RPC gunyah: Add interfaces to map memory into guest address space gunyah: Add platform ops on mem_lend/mem_reclaim gunyah: Add Qualcomm Gunyah platform ops gunyah: Share guest VM dtb configuration to Gunyah gunyah: Add RPC to enable demand paging gunyah: Add RPC to set VM boot context gunyah: Add hypercalls for sending doorbell Karim Manaouil (14): KVM: Allow arch-specific vCPU allocation and freeing KVM: irqfd: Add architecture hooks for irqfd allocation and initialization KVM: irqfd: Allow KVM backends to override IRQ injection via set_irq callback KVM: Add weak stubs for irqchip-related functions for Gunyah builds KVM: Add KVM_SET_DTB_ADDRESS ioctl to pass guest DTB address from userspace KVM: gunyah: Add initial Gunyah backend support KVM: gunyah: Pin guest memory gunyah: Add basic VM lifecycle management gunyah: add proxy-scheduled vCPUs gunyah: Share memory parcels gunyah: Enable demand paging gunyah: allow userspace to set boot cpu context KVM: gunyah: Implement irqfd interface KVM: gunyah: enable KVM for Gunyah .../bindings/firmware/gunyah-hypervisor.yaml | 82 + Documentation/virt/gunyah/index.rst | 135 ++ Documentation/virt/gunyah/message-queue.rst | 68 + Documentation/virt/index.rst | 1 + arch/arm64/Kbuild | 1 + arch/arm64/gunyah/Makefile | 3 + arch/arm64/gunyah/gunyah_hypercall.c | 279 +++ arch/arm64/include/asm/gunyah.h | 57 + arch/arm64/include/asm/kvm_host.h | 10 +- arch/arm64/include/asm/virt.h | 7 + arch/arm64/kernel/cpufeature.c | 4 + arch/arm64/kernel/image-vars.h | 2 +- arch/arm64/kvm/Kconfig | 22 +- arch/arm64/kvm/Makefile | 14 +- arch/arm64/kvm/gunyah.c | 2085 +++++++++++++++++ drivers/virt/Kconfig | 2 + drivers/virt/Makefile | 1 + drivers/virt/gunyah/Kconfig | 29 + drivers/virt/gunyah/Makefile | 7 + drivers/virt/gunyah/gunyah.c | 55 + drivers/virt/gunyah/gunyah_platform_hooks.c | 117 + drivers/virt/gunyah/gunyah_qcom.c | 220 ++ drivers/virt/gunyah/rsc_mgr.c | 792 +++++++ drivers/virt/gunyah/rsc_mgr_rpc.c | 572 +++++ include/kvm/arm_pmu.h | 2 +- include/linux/gunyah.h | 425 ++++ include/linux/gunyah_rsc_mgr.h | 182 ++ include/linux/irqchip/arm-vgic-info.h | 2 +- include/linux/kvm_host.h | 3 + include/linux/kvm_irqfd.h | 5 + include/linux/perf/arm_pmu.h | 2 +- include/uapi/linux/kvm.h | 14 +- virt/kvm/eventfd.c | 56 +- virt/kvm/kvm_main.c | 24 +- 34 files changed, 5258 insertions(+), 22 deletions(-) create mode 100644 Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml create mode 100644 Documentation/virt/gunyah/index.rst create mode 100644 Documentation/virt/gunyah/message-queue.rst create mode 100644 arch/arm64/gunyah/Makefile create mode 100644 arch/arm64/gunyah/gunyah_hypercall.c create mode 100644 arch/arm64/include/asm/gunyah.h create mode 100644 arch/arm64/kvm/gunyah.c create mode 100644 drivers/virt/gunyah/Kconfig create mode 100644 drivers/virt/gunyah/Makefile create mode 100644 drivers/virt/gunyah/gunyah.c create mode 100644 drivers/virt/gunyah/gunyah_platform_hooks.c create mode 100644 drivers/virt/gunyah/gunyah_qcom.c create mode 100644 drivers/virt/gunyah/rsc_mgr.c create mode 100644 drivers/virt/gunyah/rsc_mgr_rpc.c create mode 100644 include/linux/gunyah.h create mode 100644 include/linux/gunyah_rsc_mgr.h