mbox series

[v2,00/25] Enable FRED with KVM VMX

Message ID 20240207172646.3981-1-xin3.li@intel.com
Headers show
Series Enable FRED with KVM VMX | expand

Message

Li, Xin3 Feb. 7, 2024, 5:26 p.m. UTC
This patch set enables the Intel flexible return and event delivery
(FRED) architecture with KVM VMX to allow guests to utilize FRED.

The FRED architecture defines simple new transitions that change
privilege level (ring transitions). The FRED architecture was
designed with the following goals:

1) Improve overall performance and response time by replacing event
   delivery through the interrupt descriptor table (IDT event
   delivery) and event return by the IRET instruction with lower
   latency transitions.

2) Improve software robustness by ensuring that event delivery
   establishes the full supervisor context and that event return
   establishes the full user context.

The new transitions defined by the FRED architecture are FRED event
delivery and, for returning from events, two FRED return instructions.
FRED event delivery can effect a transition from ring 3 to ring 0, but
it is used also to deliver events incident to ring 0. One FRED
instruction (ERETU) effects a return from ring 0 to ring 3, while the
other (ERETS) returns while remaining in ring 0. Collectively, FRED
event delivery and the FRED return instructions are FRED transitions.

Intel VMX architecture is extended to run FRED guests, and the major
changes are:

1) New VMCS fields for FRED context management, which includes two new
event data VMCS fields, eight new guest FRED context VMCS fields and
eight new host FRED context VMCS fields.

2) VMX nested-exception support for proper virtualization of stack
levels introduced with FRED architecture.

Search for the latest FRED spec in most search engines with this search
pattern:

  site:intel.com FRED (flexible return and event delivery) specification

As the native FRED patches are committed in the tip tree "x86/fred"
branch:
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/log/?h=x86/fred,
and we have received a good amount of review comments for v1, it's time
to send out v2 based on this branch for further help from the community.

Patch 1-2 are cleanups to VMX basic and misc MSRs, which were sent
out earlier as a preparation for FRED changes:
https://lore.kernel.org/kvm/20240206182032.1596-1-xin3.li@intel.com/T/#u

Patch 3-15 add FRED support to VMX.
Patch 16-21 add FRED support to nested VMX.
Patch 22 exposes FRED and its baseline features to KVM guests.
Patch 23-25 add FRED selftests.

There is also a counterpart qemu patch set for FRED at:
https://lore.kernel.org/qemu-devel/20231109072012.8078-1-xin3.li@intel.com/T/,
which works with this patch set to allow KVM to run FRED guests.


Changes since v1:
* Always load the secondary VM exit controls (Sean Christopherson).
* Remove FRED VM entry/exit controls consistency checks in
  setup_vmcs_config() (Sean Christopherson).
* Clear FRED VM entry/exit controls if FRED is not enumerated (Chao Gao).
* Use guest_can_use() to trace FRED enumeration in a vcpu (Chao Gao).
* Enable FRED MSRs intercept if FRED is no longer enumerated in CPUID
  (Chao Gao).
* Move guest FRED states init into __vmx_vcpu_reset() (Chao Gao).
* Don't use guest_cpuid_has() in vmx_prepare_switch_to_{host,guest}(),
  which are called from IRQ-disabled context (Chao Gao).
* Reset msr_guest_fred_rsp0 in __vmx_vcpu_reset() (Chao Gao).
* Fail host requested FRED MSRs access if KVM cannot virtualize FRED
  (Chao Gao).
* Handle the case FRED MSRs are valid but KVM cannot virtualize FRED
  (Chao Gao).
* Add sanity checks when writing to FRED MSRs.
* Explain why it is ok to only check CR4.FRED in kvm_is_fred_enabled()
  (Chao Gao).
* Document event data should be equal to CR2/DR6/IA32_XFD_ERR instead
  of using WARN_ON() (Chao Gao).
* Zero event data if a #NM was not caused by extended feature disable
  (Chao Gao).
* Set the nested flag when there is an original interrupt (Chao Gao).
* Dump guest FRED states only if guest has FRED enabled (Nikolay Borisov).
* Add a prerequisite to SHADOW_FIELD_R[OW] macros
* Remove hyperv TLFS related changes (Jeremi Piotrowski).
* Use kvm_cpu_cap_has() instead of cpu_feature_enabled() to decouple
  KVM's capability to virtualize a feature and host's enabling of a
  feature (Chao Gao).


Xin Li (25):
  KVM: VMX: Cleanup VMX basic information defines and usages
  KVM: VMX: Cleanup VMX misc information defines and usages
  KVM: VMX: Add support for the secondary VM exit controls
  KVM: x86: Mark CR4.FRED as not reserved
  KVM: VMX: Initialize FRED VM entry/exit controls in vmcs_config
  KVM: VMX: Defer enabling FRED MSRs save/load until after set CPUID
  KVM: VMX: Set intercept for FRED MSRs
  KVM: VMX: Initialize VMCS FRED fields
  KVM: VMX: Switch FRED RSP0 between host and guest
  KVM: VMX: Add support for FRED context save/restore
  KVM: x86: Add kvm_is_fred_enabled()
  KVM: VMX: Handle FRED event data
  KVM: VMX: Handle VMX nested exception for FRED
  KVM: VMX: Disable FRED if FRED consistency checks fail
  KVM: VMX: Dump FRED context in dump_vmcs()
  KVM: VMX: Invoke vmx_set_cpu_caps() before nested setup
  KVM: nVMX: Add support for the secondary VM exit controls
  KVM: nVMX: Add a prerequisite to SHADOW_FIELD_R[OW] macros
  KVM: nVMX: Add FRED VMCS fields
  KVM: nVMX: Add support for VMX FRED controls
  KVM: nVMX: Add VMCS FRED states checking
  KVM: x86: Allow FRED/LKGS/WRMSRNS to be exposed to guests
  KVM: selftests: Run debug_regs test with FRED enabled
  KVM: selftests: Add a new VM guest mode to run user level code
  KVM: selftests: Add fred exception tests

 Documentation/virt/kvm/x86/nested-vmx.rst     |  19 +
 arch/x86/include/asm/kvm_host.h               |   8 +-
 arch/x86/include/asm/msr-index.h              |  15 +-
 arch/x86/include/asm/vmx.h                    |  59 ++-
 arch/x86/kvm/cpuid.c                          |   4 +-
 arch/x86/kvm/governed_features.h              |   1 +
 arch/x86/kvm/kvm_cache_regs.h                 |  17 +
 arch/x86/kvm/svm/svm.c                        |   4 +-
 arch/x86/kvm/vmx/capabilities.h               |  30 +-
 arch/x86/kvm/vmx/nested.c                     | 329 ++++++++++++---
 arch/x86/kvm/vmx/nested.h                     |   2 +-
 arch/x86/kvm/vmx/vmcs.h                       |   1 +
 arch/x86/kvm/vmx/vmcs12.c                     |  19 +
 arch/x86/kvm/vmx/vmcs12.h                     |  38 ++
 arch/x86/kvm/vmx/vmcs_shadow_fields.h         |  80 ++--
 arch/x86/kvm/vmx/vmx.c                        | 385 +++++++++++++++---
 arch/x86/kvm/vmx/vmx.h                        |  15 +-
 arch/x86/kvm/x86.c                            | 103 ++++-
 arch/x86/kvm/x86.h                            |   5 +-
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../selftests/kvm/include/kvm_util_base.h     |   1 +
 .../selftests/kvm/include/x86_64/processor.h  |  36 ++
 tools/testing/selftests/kvm/lib/kvm_util.c    |   5 +-
 .../selftests/kvm/lib/x86_64/processor.c      |  15 +-
 tools/testing/selftests/kvm/lib/x86_64/vmx.c  |   4 +-
 .../testing/selftests/kvm/x86_64/debug_regs.c |  50 ++-
 .../testing/selftests/kvm/x86_64/fred_test.c  | 297 ++++++++++++++
 27 files changed, 1320 insertions(+), 223 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86_64/fred_test.c


base-commit: e13841907b8fda0ae0ce1ec03684665f578416a8

Comments

Kang, Shan March 27, 2024, 8:08 a.m. UTC | #1
On Wed, 2024-02-07 at 09:26 -0800, Xin Li wrote:
> This patch set enables the Intel flexible return and event delivery
> (FRED) architecture with KVM VMX to allow guests to utilize FRED.
> 
We tested this FRED KVM patch set on a 7th Intel(R) Core(TM) CPU and the Intel 
Simics® Simulator with the following four configurations:

The first config is the baseline on bare metal.
The second config is the baseline on Intel Simics® Simulator.
The third config enables host FRED, but disables guest FRED.
The last config enables both host and guest FRED.

Following are the Kselftest results on KVM guests.

+---------------------------------------------+-------+-------+-------+-------+
|                  Config                     |  Pass |  Fail |  Skip |  Hang |
+---------------------------------------------+-------+-------+-------+-------+
|the 7th Intel(R) Core(TM) CPU                |       |       |       |       |
|  L0: 6.8.0-rc3+ w/ FRED native/KVM patch set|  1775 |  526  |  332  |   6   |
|  L1: 6.8.0-rc3+                             |       |       |       |       |
+---------------------------------------------+-------+-------+-------+-------+
|Intel Simics® Simulator w/o FRED model       |       |       |       |       |
|  L0: 6.8.0-rc3+ w/ FRED native/KVM patch set|  1770 |  526  |  331  |   12  |
|  L1: 6.8.0-rc3+ w/ FRED native/KVM patch set|       |       |       |       |
+---------------------------------------------+-------+-------+-------+-------+
|Intel Simics® Simulator w/ FRED model        |       |       |       |       |
|  L0: 6.8.0-rc3+ w/ FRED native/KVM patch set|  1770 |  526  |  331  |   12  |
|  L1: 6.8.0-rc3+ w/ FRED native/KVM patch set|       |       |       |       |
|         but FRED disabled                   |       |       |       |       |
+---------------------------------------------+-------+-------+-------+-------+
|Intel Simics® Simulator w/ FRED model        |       |       |       |       |
|  L0: 6.8.0-rc3+ w/ FRED native/KVM patch set|  1769 |  528  |  330  |   12  |
|  L1: 6.8.0-rc3+ w/ FRED native/KVM patch set|       |       |       |       |
+---------------------------------------------+-------+-------+-------+-------+

First of all we don't see any major issue. One major differences come from
perf tests. Another variances are from timer tests due to Intel Simics®
emulator's slowness.

The tests "x86:sysret_rip_64" and "x86:sigreturn_32" fail on the last config,
because they are not valid tests for FRED:
https://lore.kernel.org/lkml/20230220030959.119222-1-ammarfaizi2@gnuweeb.org/T/#u
https://lore.kernel.org/lkml/20230706052231.2183-1-xin3.li@intel.com/

NB: Some tests pass on Intel Simics® emulator, but not bare metal. If needed,
we can share the data.

We conducted local live migration tests with LKGS/FRED/WRMSRNS enabled/disabled
on Intel Simics® Simulator, and see no problems.

We also tested KVM Kselftest on a nested FRED KVM guest, 48 out of 83 cases
passed and the rest cases failed due to the slowness of Intel Simics® emulator.

Detailed report upon request.
Li, Xin3 April 15, 2024, 5:58 p.m. UTC | #2
> This patch set enables the Intel flexible return and event delivery
> (FRED) architecture with KVM VMX to allow guests to utilize FRED.
> 

<snip>

> 
> Intel VMX architecture is extended to run FRED guests, and the major changes
> are:
> 
> 1) New VMCS fields for FRED context management, which includes two new
> event data VMCS fields, eight new guest FRED context VMCS fields and eight new
> host FRED context VMCS fields.
> 
> 2) VMX nested-exception support for proper virtualization of stack levels
> introduced with FRED architecture.
> 

<snip>

> 
> Patch 1-2 are cleanups to VMX basic and misc MSRs, which were sent out earlier
> as a preparation for FRED changes:
> https://lore.kernel.org/kvm/20240206182032.1596-1-xin3.li@intel.com/T/#u

Obviously I will drop the 2 clean patches in the next iteration.

> Patch 3-15 add FRED support to VMX.
> Patch 16-21 add FRED support to nested VMX.
> Patch 22 exposes FRED and its baseline features to KVM guests.
> Patch 23-25 add FRED selftests.

Please help to review and comment on the FRED KVM/VMX patches.

Thanks!
    Xin
Sean Christopherson June 13, 2024, 6:38 p.m. UTC | #3
On Wed, Mar 27, 2024, Shan Kang wrote:
> On Wed, 2024-02-07 at 09:26 -0800, Xin Li wrote:
> > This patch set enables the Intel flexible return and event delivery
> > (FRED) architecture with KVM VMX to allow guests to utilize FRED.
> > 
> We tested this FRED KVM patch set on a 7th Intel(R) Core(TM) CPU and the Intel 
> Simics® Simulator with the following four configurations:

That likely provides coverage for the happy cases, but I doubt it provides negative
testing, e.g. for VM-Enter consistency checks.  KVM-Unit-Tests are currently the
best choice for concistency checks (unfortunately).

And given the insanity of event re-injection, KVM selftests needs a dedicated test
for that, and another for the interactions with nVMX, e.g. a la svm_nested_soft_inject_test.c.

I haven't looked too closely at the selftest that's already provided, but my
suspicion is that we'll want multiple tests, or alternatively one test that
uses KVM_ONE_VCPU_TEST_SUITE().
Li, Xin3 June 14, 2024, 12:52 a.m. UTC | #4
> On Wed, Mar 27, 2024, Shan Kang wrote:
> > On Wed, 2024-02-07 at 09:26 -0800, Xin Li wrote:
> > > This patch set enables the Intel flexible return and event delivery
> > > (FRED) architecture with KVM VMX to allow guests to utilize FRED.
> > >
> > We tested this FRED KVM patch set on a 7th Intel(R) Core(TM) CPU and
> > the Intel Simics® Simulator with the following four configurations:
> 
> That likely provides coverage for the happy cases, but I doubt it provides negative
> testing, e.g. for VM-Enter consistency checks.  KVM-Unit-Tests are currently the
> best choice for concistency checks (unfortunately).
> 
> And given the insanity of event re-injection, KVM selftests needs a dedicated test
> for that, and another for the interactions with nVMX, e.g. a la
> svm_nested_soft_inject_test.c.
> 
> I haven't looked too closely at the selftest that's already provided, but my
> suspicion is that we'll want multiple tests, or alternatively one test that uses
> KVM_ONE_VCPU_TEST_SUITE().

What about posting the FRED selftests in another new patch set?

Thanks!
    Xin