| Message ID | 20241125195626.856992-2-jean-philippe@linaro.org |
|---|---|
| Headers | show |
| Series | arm: Run Arm CCA VMs with KVM | expand |
Hi Jean, On 11/26/24 5:55 AM, Jean-Philippe Brucker wrote: > This series enables running confidential VMs on Arm CCA. The host KVM > support is progressing but still under discussion [1], so there is no > urgency to upstream this series. I'm sending this new version to give a > status update, and also to discuss remote attestation below. > > Since v2 [2] I addressed comments on the QAPI patches. The support for > running Linux in a Realm will be in Linux v6.13 [3], so the guest-facing > interface is now stable. One important change since v2 is the requirement > to initialize the whole GPA space in RMM before boot, which we do in patch > 9. The 'earlycon' kernel parameter now requires an unprotected address > parameter (see Documentation/arch/arm64/arm-cca.rst in Linux v6.13). > > Documentation to try this series out: > https://linaro.atlassian.net/wiki/spaces/QEMU/pages/29051027459/Building+an+RME+stack+for+QEMU > I followed the instructions, but encountering some issues. First of all, the recommended TF-RMM (branch: cca/v4) fails to be built. After switching to the latest upstream TF-RMM, it fails to be initialized. Note: the combination of the upstream TF-A + upstream TF-RMM + upstream EDK2 worked fine for the 'fvp' emulator. TF-RMM fails to be built ======================== $ git clone https://git.codelinaro.org/linaro/dcap/rmm.git tf-rmm $ cd tf-rmm $ git checkout origin/cca/v4 -b cca/v4 $ git submodule update --init --recursive $ cmake -DCMAKE_BUILD_TYPE=Debug -DRMM_CONFIG=qemu_virt_defcfg -B build-qemu -- The C compiler identification is GNU 11.5.0 -- The CXX compiler identification is GNU 11.5.0 -- The ASM compiler identification is GNU -- Found assembler: /usr/bin/gcc -- Detecting C compiler ABI info -- Detecting C compiler ABI info - failed -- Check for working C compiler: /usr/bin/gcc -- Check for working C compiler: /usr/bin/gcc - broken CMake Error at /usr/share/cmake/Modules/CMakeTestCCompiler.cmake:67 (message): The C compiler "/usr/bin/gcc" is not able to compile a simple test program. It fails with the following output: Change Dir: /home/gshan/sandbox/qemu/host/tf-rmm/build-qemu/CMakeFiles/CMakeScratch/TryCompile-Aab6zP Run Build Command(s):/usr/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile cmTC_87ded/fast && /usr/bin/gmake -f CMakeFiles/cmTC_87ded.dir/build.make CMakeFiles/cmTC_87ded.dir/build gmake[1]: Entering directory '/home/gshan/sandbox/qemu/host/tf-rmm/build-qemu/CMakeFiles/CMakeScratch/TryCompile-Aab6zP' Building C object CMakeFiles/cmTC_87ded.dir/testCCompiler.c.obj /usr/bin/gcc -fno-common -ffunction-sections -fdata-sections -Wall -Werror -gdwarf-4 -ffreestanding -mbranch-protection=standard -mgeneral-regs-only -mstrict-align -fpie -o CMakeFiles/cmTC_87ded.dir/testCCompiler.c.obj -c /home/gshan/sandbox/qemu/host/tf-rmm/build-qemu/CMakeFiles/CMakeScratch/TryCompile-Aab6zP/testCCompiler.c gcc: error: unrecognized command-line option ‘-mbranch-protection=standard’ gcc: error: unrecognized command-line option ‘-mstrict-align’; did you mean ‘-Wstrict-aliasing’? gmake[1]: *** [CMakeFiles/cmTC_87ded.dir/build.make:78: CMakeFiles/cmTC_87ded.dir/testCCompiler.c.obj] Error 1 gmake[1]: Leaving directory '/home/gshan/sandbox/qemu/host/tf-rmm/build-qemu/CMakeFiles/CMakeScratch/TryCompile-Aab6zP' gmake: *** [Makefile:127: cmTC_87ded/fast] Error 2 CMake will not be able to correctly generate this project. Call Stack (most recent call first): CMakeLists.txt:51 (project) -- Configuring incomplete, errors occurred! Upstream TF-RMM fails to be initialized ======================================== $ git clone git@github.com:TF-RMM/tf-rmm.git tf-rmm $ cd tf-rmm $ git submodule update --init --recursive $ export CROSS_COMPILE=aarch64-none-elf- $ cmake -DCMAKE_BUILD_TYPE=Debug -DRMM_CONFIG=qemu_virt_defcfg -B build-qemu $ cmake --build build-qemu $ cd .. $ git clone git@github.com:tianocore/edk2.git edk2 $ cd edk2 $ git submodule update --init --recursive $ source edksetup.sh $ make -j -C BaseTools $ export GCC5_AARCH64_PREFIX=aarch64-linux-gnu- $ build -b RELEASE -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemuKernel.dsc $ cd .. $ git@github.com:ARM-software/arm-trusted-firmware.git tf-a $ make -j CROSS_COMPILE=aarch64-linux-gnu- \ PLAT=qemu ENABLE_RME=1 DEBUG=1 LOG_LEVEL=40 \ QEMU_USE_GIC_DRIVER=QEMU_GICV3 \ RMM=../rmm/build-qemu/Debug/rmm.img \ BL33=../edk2/Build/ArmVirtQemuKernel-AARCH64/RELEASE_GCC5/FV/QEMU_EFI.fd all fip $ dd if=build/qemu/debug/bl1.bin of=flash.bin $ dd if=build/qemu/debug/fip.bin of=flash.bin seek=64 bs=4096 $ ${HOST_PATH}/qemu/build/qemu-system-aarch64 \ -M virt,virtualization=on,secure=on,gic-version=3,acpi=off \ -cpu max,x-rme=on -m 64G -smp 8 \ -serial mon:stdio -monitor none -nographic -nodefaults \ -bios ${HOST_PATH}/tf-a/flash.bin \ -kernel ${HOST_PATH}/linux/arch/arm64/boot/Image \ -initrd ${HOST_PATH}/buildroot/output/images/rootfs.cpio.xz \ -device pcie-root-port,bus=pcie.0,chassis=1,id=pcie.1 \ -device pcie-root-port,bus=pcie.0,chassis=2,id=pcie.2 \ -device pcie-root-port,bus=pcie.0,chassis=3,id=pcie.3 \ -device pcie-root-port,bus=pcie.0,chassis=4,id=pcie.4 \ -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \ -fsdev local,security_model=none,path=${GUEST_PATH},id=shr0 \ -netdev tap,id=tap1,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \ -device virtio-net-pci,bus=pcie.2,netdev=tap1,mac=78:ac:44:2b:43:f0 : : NOTICE: Booting Trusted Firmware NOTICE: BL1: v2.12.0(debug):ad6310a66 NOTICE: BL1: Built : 22:09:22, Dec 7 2024 INFO: BL1: RAM 0xe0ee000 - 0xe0f6000 INFO: BL1: Loading BL2 INFO: Loading image id=1 at address 0xe05b000 INFO: Image id=1 loaded: 0xe05b000 - 0xe0652b9 NOTICE: BL1: Booting BL2 INFO: Entry point address = 0xe05b000 INFO: SPSR = 0x3cd INFO: GPT: Boot Configuration INFO: PPS/T: 0x2/40 INFO: PGS/P: 0x0/12 INFO: L0GPTSZ/S: 0x0/30 INFO: PAS count: 6 INFO: L0 base: 0xeefc000 INFO: Enabling Granule Protection Checks NOTICE: BL2: v2.12.0(debug):ad6310a66 NOTICE: BL2: Built : 22:09:23, Dec 7 2024 INFO: BL2: Doing platform setup INFO: Reserved RMM memory [0x40100000, 0x418fffff] in Device tree INFO: BL2: Loading image id 3 INFO: Loading image id=3 at address 0xe090000 INFO: Image id=3 loaded: 0xe090000 - 0xe0a20c4 INFO: BL2: Loading image id 35 INFO: Loading image id=35 at address 0x40100000 INFO: Image id=35 loaded: 0x40100000 - 0x40303a00 INFO: BL2: Loading image id 5 INFO: Loading image id=5 at address 0x60000000 INFO: Image id=5 loaded: 0x60000000 - 0x60200000 NOTICE: BL2: Booting BL31 INFO: Entry point address = 0xe090000 INFO: SPSR = 0x3cd NOTICE: BL31: v2.12.0(debug):ad6310a66 NOTICE: BL31: Built : 22:09:23, Dec 7 2024 INFO: GICv3 without legacy support detected. INFO: ARM GICv3 driver initialized in EL3 INFO: Maximum SPI INTID supported: 287 INFO: BL31: Initializing runtime services INFO: RMM setup done. INFO: BL31: Initializing RMM INFO: RMM init start. ERROR: RMM init failed: -7 <<<<< RMM initialization failed WARNING: BL31: RMM initialization failed INFO: BL31: Preparing for EL3 exit to normal world INFO: Entry point address = 0x60000000 INFO: SPSR = 0x3c9 UEFI firmware (version built at 22:03:35 on Dec 7 2024) : EFI stub: Booting Linux Kernel... EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path EFI stub: Using DTB from configuration table EFI stub: Exiting boot services... ERROR: RMM Failed to initialize. Ignoring for CPU1 ERROR: RMM Failed to initialize. Ignoring for CPU2 ERROR: RMM Failed to initialize. Ignoring for CPU3 ERROR: RMM Failed to initialize. Ignoring for CPU4 ERROR: RMM Failed to initialize. Ignoring for CPU5 ERROR: RMM Failed to initialize. Ignoring for CPU6 ERROR: RMM Failed to initialize. Ignoring for CPU7 [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510] Thanks, Gavin
Hi Jean, On 12/11/24 1:01 PM, Gavin Shan wrote: > On 11/26/24 5:55 AM, Jean-Philippe Brucker wrote: >> This series enables running confidential VMs on Arm CCA. The host KVM >> support is progressing but still under discussion [1], so there is no >> urgency to upstream this series. I'm sending this new version to give a >> status update, and also to discuss remote attestation below. >> >> Since v2 [2] I addressed comments on the QAPI patches. The support for >> running Linux in a Realm will be in Linux v6.13 [3], so the guest-facing >> interface is now stable. One important change since v2 is the requirement >> to initialize the whole GPA space in RMM before boot, which we do in patch >> 9. The 'earlycon' kernel parameter now requires an unprotected address >> parameter (see Documentation/arch/arm64/arm-cca.rst in Linux v6.13). >> >> Documentation to try this series out: >> https://linaro.atlassian.net/wiki/spaces/QEMU/pages/29051027459/Building+an+RME+stack+for+QEMU >> > > I followed the instructions, but encountering some issues. First of all, the > recommended TF-RMM (branch: cca/v4) fails to be built. After switching to the > latest upstream TF-RMM, it fails to be initialized. > Please ignore the issue. The RMM service can be started successfully with your recommended tf-a/tf-rmm repositories. I don't know why tf-rmm wasn't built successfully and all looks good after I reconnect to my x86-64 server. So it seems the environment was corrupted. https://git.codelinaro.org/linaro/dcap/rmm.git (branch: cca/v4) https://git.codelinaro.org/linaro/dcap/tf-a/trusted-firmware-a.git (branch: cca/v4) Logs from the host side ======================== INFO: BL31: Initializing runtime services INFO: RMM setup done. INFO: BL31: Initializing RMM INFO: RMM init start. Booting RMM v.0.5.0(debug) 4e2e764 Built with GCC 13.3.1 RMM-EL3 Interface v.0.4 Boot Manifest Interface v.0.3 RMI/RSI ABI v.1.0/1.0 built: Dec 11 2024 02:52:28 INFO: RMM init end. Thanks, Gavin
This series enables running confidential VMs on Arm CCA. The host KVM support is progressing but still under discussion [1], so there is no urgency to upstream this series. I'm sending this new version to give a status update, and also to discuss remote attestation below. Since v2 [2] I addressed comments on the QAPI patches. The support for running Linux in a Realm will be in Linux v6.13 [3], so the guest-facing interface is now stable. One important change since v2 is the requirement to initialize the whole GPA space in RMM before boot, which we do in patch 9. The 'earlycon' kernel parameter now requires an unprotected address parameter (see Documentation/arch/arm64/arm-cca.rst in Linux v6.13). Documentation to try this series out: https://linaro.atlassian.net/wiki/spaces/QEMU/pages/29051027459/Building+an+RME+stack+for+QEMU The new RFC patches 21-26 are ideas to make remote attestation easier. They're all optional, and in order to understand which patches should be kept and what should be standardized, we'd like feedback from people who will provide hosting and verification services, and also from people working on other VMMs, who will be faced with the same problems. QEMU and Realm attestation ========================== Confidential computing wouldn't be complete without the ability to attest that a remote VM you own is running your software on the correct hardware. There are many components in enabling remote attestation, but I'd like to discuss specifically how to independently reconstruct the initial state of the VM created by QEMU. We can illustrate this problem using the keybroker attestation demo [4]: ┌──────┬───────────────┐ │ NS │ REALM │ RELYING PARTY ┌───┼──────┼───────────────┤ (b) (you are here) │ │ │ │ evidence ┌────────────┐ │EL0│ QEMU │ keybroker-app ┼────────────► keybroker- │ │ │ │ │ ▲ ◄────────────┼ server │ ├───┼───┼──┼──────┼────────┤ secret └─────▲──────┘ │ │ │ │ │ │ (d) │ (c) │EL1│ │ │ guest kernel │ ┌───▼────┐ │ │ │ │ ▲ │ │veraison│ │ │ │ ├──────┼────────┤ └────────┘ │ │ ▼ │ (a) │ │ VERIFIER │EL2│ KVM ┼───► RMM │ │ │ │ │ └───┴──────┴───────────────┘ ATTESTER QEMU and KVM load data into the VM memory (kernel, initrd, firmware, etc), before activating it and sealing its initial state (a). The initial state is recorded by the RMM as the Realm Initial Measurement (RIM). Additional data may be loaded at runtime, for example a bootloader loading an initrd from untrusted storage. Those are recorded in Realm Extensible Measurements (REM). Now say you want the VM to run an application, keybroker-app, that downloads a secret from your keybroker-server, for example a private key to open an encrypted archive. keybroker-server -e http://10.0.2.2 -vv keybroker-app -e http://10.0.2.2:8088 -v <key> keybroker-app obtains signed evidence using Linux configfs-tsm, sends it to keybroker-server (b). The evidence contains a platform token describing the platform state (host hardware, firmware and RMM), and a realm token. The signature and platform verification, proving that the VM is running in a confidential environment, is offloaded to a trusted third party in this example (c), though you could of course run veraison yourself. keybroker-server then checks the RIM and REMs contained in the realm token, to ensure the Realm is running the correct payload, and sends the secret key (d). Computing the Realm Initial Measurement --------------------------------------- The simplest way to obtain the RIM is to run the payload once on a trusted machine, extract the reference values and provision keybroker-server with them. On the first run the demo tells you how to do this: INFO Known-good RIM values are missing. If you trust the client that submitted evidence for challenge 352278775, you should restart the keybroker-server with the following command-line option to populate it with known-good RIM values: --reference-values <(echo '{ "reference-values": [ "Up9LS4jrkX002..." ] }') This is fine for testing, but does not scale well. For one thing you may not have access to a trusted machine to run the payload. Also, the RIM is relatively unstable: it depends on the host features (eg. SVE vector length), the VM configuration (eg. amount of RAM) and the images loaded into the Realm. Since the DTB/ACPI tables are included in the measurements, changing the number of CPUs or MMIO devices will also produce a different RIM. Some VMMs even have unstable IRQ allocation. To pre-calculate the RIM for a set of VM parameters, rather than running the workload once you can use the realm-measurements tool [5]: realm-measurements --print-b64 -c configs/qemu-max-8.2.conf -c configs/kvm.conf # host caps -k <kernel> -i <initrd> # payload qemu <qemu command-line parameters> RIM: Up9LS4jrkX002... Internally realm-measurements generates a DTB for the given machine configuration and uses it to calculate the RIM. This means that QEMU must be run with the same generated DTB, which can be obtained by running realm-measurements with `--output-dtb qemu.dtb --no-measurements` on the host. See patch 21, which enables loading this DTB. I've also experimented with reproducing the DTB generated by QEMU, from a template obtained with `-M dumpdtb=<tpl.dtb>` [6]. It's rather fragile due to phandle allocation, property names and libfdt internals. You can try this mode by passing `--input-dtb <tpl.dtb>`, in which case there is no need to pass a DTB to QEMU. But this solution will probably remain experimental. An event log from the VMM ------------------------- Reconstructing the RIM this way requires a strict specification of the VMM behavior [7]. To relax this, the VMM could provide an event log, describing the different operations that contributed to the RIM: Realm parameters, images loaded into RAM, initial CPU registers. The Trusted Computing Group (TCG) defines a log format that describes things measured by a TPM. Since this format will probably be reused to describe images measured into the REM, it would make sense to adopt it for the RIM as well. UEFI normally creates this log in RAM and passes it to the OS via firmware table (CCEL for ACPI, TBD [8] for device tree) Patches 23-26 show how we could load such event log into Realm memory. Something to improve is the image identifiers passed in the log, that allow the verifier to locate the image in its store of known-good images. At the moment we use a hash function for this, but it would be useful to either let the host pass the pre-calculated hash, or we could extend RMM to return the hashes that it already computes when measuring data granules. Using a modified keybroker-demo that supports the event log [10], you can pass a list of known-good images loaded into the realm: sha512sum path/to/images/* > checksums.txt keybroker-server -e http://10.0.2.2 -vv --images checksums.txt Now when receiving evidence, the server can reconstruct the RIM and REMs by parsing the event log. To reduce bandwidth use, the server caches the computed reference values, and only requires an event log when reference values are absent. Note that the PoC is incomplete, because it trusts the log too much when constructing the RIM: although data loaded into the Realm is correctly verified, some events aren't properly checked by the policy at the moment. In particular the REC event (vCPU initial state) must have sane initial registers (which can be predicted from the DTB, KERNEL and FIRMWARE log events, so should be easy to fix). The kernel parameters, contained in the DTB, will also need to be validated by the policy. The server reconstructs the DTB using a few VM properties passed in the log (number of CPUs, RAM size). Another possible extension is to send the DTB/ACPI tables in the log, so the tool can verify their content rather than generate them, but I'm not sure it's even worth attempting. Loading IGVM payload -------------------- IGVM [9] packs the payload along with instructions for creating the VM, into a file that is fed to QEMU. We'll likely want to support it on Arm in order to load complex payloads like COCONUT-SVSM, but probably not for lightweight container payloads where firmware would be unnecessary overhead. If the IGVM file describes all operations leading to a RIM, then a strict VM specification isn't needed in this case either. However to get more flexibility it may be useful to combine IGVM with an event log: the host packs the IGVM file including an event log, and the relying party only needs the event log to reconstruct the RIM, it doesn't need to upload a pack for each different configuration. [1] [PATCH v5] arm64: Support for Arm CCA in KVM https://lore.kernel.org/all/20241004152804.72508-1-steven.price@arm.com/ [2] [PATCH v2] arm: Run CCA VMs with KVM https://lore.kernel.org/all/20240419155709.318866-2-jean-philippe@linaro.org/ [3] [PATCH v7] arm64: Support for running as a guest in Arm CCA https://lore.kernel.org/all/172967739783.1412028.8494484908145931121.b4-ty@arm.com/ [4] Veraison keybroker demo https://github.com/veraison/keybroker-demo/ [5] Realm measurements tool https://github.com/veraison/cca-realm-measurements https://docs.rs/cca-realm-measurements/0.1.0/cca_realm_measurements/ [6] DTB modification and libfdt changes https://github.com/veraison/cca-realm-measurements/commit/4b7aa3cbd7ae7ce114726979367329f57b8ef2c6 https://git.codelinaro.org/linaro/dcap/buildroot-external-cca/-/commit/28f4afa6120b467cadab9bc0af67e7bab239797b [7] VM specification https://github.com/veraison/cca-realm-measurements/blob/main/docs/realm-vm.md [8] Confidential computing event log for device-tree https://jpbrucker.net/git/linux/log/?h=cca/ccel-dt [9] IGVM in QEMU, COCONUT-SVSM https://lore.kernel.org/qemu-devel/cover.1727341768.git.roy.hopkins@suse.com/ https://github.com/coconut-svsm/svsm/blob/main/Documentation/docs/installation/INSTALL.md [10] keybroker-demo with event log support https://github.com/jpbrucker/keybroker-demo/commits/event-log/ Cc: Alex Bennée <alex.bennee@linaro.org> Cc: Matias Ezequiel Vara Larsen <mvaralar@redhat.com> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Gonda <pgonda@google.com> Cc: Peter Maydell <peter.maydell@linaro.org> Cc: Philippe Mathieu-Daudé <philmd@linaro.org> Cc: Richard Henderson <richard.henderson@linaro.org> Cc: Stefan Berger <stefanb@linux.vnet.ibm.com> Jean-Philippe Brucker (26): kvm: Merge kvm_check_extension() and kvm_vm_check_extension() target/arm: Add confidential guest support target/arm/kvm: Return immediately on error in kvm_arch_init() target/arm/kvm-rme: Initialize realm target/arm/kvm: Split kvm_arch_get/put_registers target/arm/kvm-rme: Initialize vCPU target/arm/kvm: Create scratch VM as Realm if necessary hw/core/loader: Add ROM loader notifier target/arm/kvm-rme: Initialize Realm memory target/arm/kvm-rme: Add Realm Personalization Value parameter target/arm/kvm-rme: Add measurement algorithm property target/arm/cpu: Set number of breakpoints and watchpoints in KVM target/arm/cpu: Set number of PMU counters in KVM target/arm/cpu: Inform about reading confidential CPU registers hw/arm/virt: Add support for Arm RME hw/arm/virt: Disable DTB randomness for confidential VMs hw/arm/virt: Reserve one bit of guest-physical address for RME hw/arm/boot: Mark all guest memory as RIPAS_RAM. hw/arm/virt: Move virt_flash_create() to machvirt_init() hw/arm/virt: Use RAM instead of flash for confidential guest firmware hw/arm/boot: Load DTB as is for confidential VMs hw/arm/boot: Skip bootloader for confidential guests hw/tpm: Add TPM event log hw/core/loader: Add fields to RomLoaderNotify target/arm/kvm-rme: Add measurement log hw/arm/virt: Add measurement log for confidential boot docs/system/arm/virt.rst | 9 +- docs/system/confidential-guest-support.rst | 1 + qapi/qom.json | 40 + qapi/tpm.json | 14 + include/hw/arm/boot.h | 18 + include/hw/arm/virt.h | 3 +- include/hw/loader.h | 17 + include/hw/tpm/tpm_log.h | 89 +++ include/sysemu/kvm.h | 2 - include/sysemu/kvm_int.h | 1 + target/arm/cpu.h | 10 + target/arm/kvm_arm.h | 98 +++ accel/kvm/kvm-all.c | 41 +- hw/arm/boot.c | 115 ++- hw/arm/virt.c | 143 +++- hw/core/loader.c | 17 + hw/tpm/tpm_log.c | 325 ++++++++ target/arm/arm-qmp-cmds.c | 1 + target/arm/cpu.c | 5 + target/arm/cpu64.c | 118 +++ target/arm/kvm-rme.c | 843 +++++++++++++++++++++ target/arm/kvm.c | 201 ++++- target/i386/kvm/kvm.c | 6 +- target/ppc/kvm.c | 36 +- hw/tpm/Kconfig | 4 + hw/tpm/meson.build | 1 + target/arm/Kconfig | 1 + target/arm/meson.build | 7 +- 28 files changed, 2075 insertions(+), 91 deletions(-) create mode 100644 include/hw/tpm/tpm_log.h create mode 100644 hw/tpm/tpm_log.c create mode 100644 target/arm/kvm-rme.c