mbox series

[v11,0/5] Enable Linux guests on Hyper-V on ARM64

Message ID 1626793023-13830-1-git-send-email-mikelley@microsoft.com
Headers show
Series Enable Linux guests on Hyper-V on ARM64 | expand

Message

Michael Kelley July 20, 2021, 2:56 p.m. UTC
This series enables Linux guests running on Hyper-V on ARM64
hardware. New ARM64-specific code in arch/arm64/hyperv initializes
Hyper-V and its hypercall mechanism.  Existing architecture
independent drivers for Hyper-V's VMbus and synthetic devices just
work when built for ARM64. Hyper-V code is built and included in
the image and modules only if CONFIG_HYPERV is enabled.

The five patches are organized as follows:

1) Add definitions and functions for making Hyper-V hypercalls
   and getting/setting virtual processor registers provided by
   Hyper-V

2) Add the function needed by the arch independent VMbus driver
   for reporting a panic to Hyper-V.

3) Add Hyper-V initialization code and utility functions that
   report Hyper-v status.

4) Export screen_info so it may be used by the Hyper-V frame buffer
   driver built as a module. It is already exported for x86,
   powerpc, and alpha architectures.

5) Make CONFIG_HYPERV selectable on ARM64 in addition to x86/x64.

Hyper-V on ARM64 runs with a 4 Kbyte page size, but allows guests
with 4K/16K/64K page size. Linux guests with this patch series
work with all three supported ARM64 page sizes.

The Hyper-V vPCI driver at drivers/pci/host/pci-hyperv.c has
x86/x64-specific code and is not being built for ARM64. Enabling
Hyper-V vPCI devices on ARM64 is in progress via a separate set
of patches.

This patch set is based on the linux-next20210720 code tree.

Changes in v11:
* Drop the previous Patch 1 as the fixes have already been
  separately accepted upstream.
* Drop the previous Patch 3 for enabling Hyper-V enlightened
  clocks/timers.  Hyper-V is now offering the full ARM64
  architectural Generic Timer in guest VMs, so the existing
  arch_arch_timer.c driver just works. [Mark Rutland, Marc
  Zyngier]
* Simplified the previous Patch 4 and Patch 5 (now Patch 2 and 3)
  by sharing more code between the x86 and ARM64 architectures.
* Combined hyperv_early_init() and hyperv_init() into a single
  initialization function that runs as an early_initcall. This
  is possible because the Hyper-V enlightened clocks/timers
  no longer need to be initialized early in the boot sequence.

Changes in v10:
* Drop previous Patch 1 for SMCCC v1.2 HVC calls in favor of
  Sudeep Holla's patch [Mark Rutland]
* Use new helper functions in 5.13 for checking Hyper-V hypercall
  return status
* Set up per-CPU hypercall argument page that is now used by the
  Hyper-V balloon driver
* Rebase on 5.13-RC1

Changes in v9:
* Added Patch 1 to enable making an SMCCC compliant hypercall
  that returns results in other than registers X0 thru X3, per
  version 1.2 and later of the SMCCC spec.
* Using the ability to return results in registers X6 and X7,
  converted hv_get_vpreg_128() to use a "fast" hypercall that
  passes inputs and outputs in registers, and in doing so eliminated
  a lot of memory allocation complexity.
* Cleaned up some extra blank lines and use of spaces in aligning
  local variables. [Sunil Muthuswamy]
* Based on discussion about future directions, reverted the
  population of hv_vp_index array to use a cpuhp state instead
  of a hypercall, which is like it was in v7 and earlier.

Changes in v8:
* Removed a lot of code based on refactoring the boundary between
  arch independent and arch dependent code for Hyper-V, per comments
  from Arnd Bergmann. The removed code was either duplicated on
  the x86 side, or has been folded into architecture independent
  code as not really being architecture dependent.
* Added config dependency on !CONFIG_CPU_BIG_ENDIAN [Arnd Bergmann]
* Reworked the approach to Hyper-V initialization. The functionality
  is the same, but is now structured like the Xen code with an early
  init function called in setup_arch() and an early initcall to
  finish the initialization. [Arnd Bergmann]

Changes in v7:
* Separately upstreamed split of hyperv-tlfs.h into arch dependent
  and independent versions.  In this patch set, update the ARM64
  hyperv-tlfs.h to include architecture independent definitions.
  This approach eliminates a lot of lines of otherwise duplicated
  code on the ARM64 side.
* Break ARM64 mshyperv.h into smaller pieces. Have an initial
  baseline, and add code along with patches for a particular
  functional area. [Marc Zyngier]
* In mshyperv.h, use static inline functions instead of #defines
  where possible. [Arnd Bergmann]
* Use VMbus INTID obtained from ACPI DSDT instead of hardcoding.
  The STIMER INTID is still hardcoded because it is needed
  before Linux has initialized the ACPI subsystem, so it can't
  be obtained from the DSDT.  Wedging it into the GTDT seems
  dubious, so was not done. [Marc Zyngier]
* Update Hyper-V page size allocation functions to use
  alloc_page() if PAGE_SIZE == HV_HYP_PAGE_SIZE [Arnd
  Bergmann]
* Various other minor changes based on feedback and to rebase
  to latest linux-next [Marc Zyngier and Arnd Bergmann]

Changes in v6:
* Use SMCCC hypercall interface instead of direct invocation
  of HVC instruction and the Hyper-V hypercall interface
  [Marc Zyngier]
* Reimplemented functions to alloc/free Hyper-V size pages
  using kmalloc/kfree since kmalloc now guarantees alignment of
  power of 2 size allocations [Marc Zyngier]
* Export screen_info in arm64 architecture so it can be used
  by the Hyper-V buffer driver built as a module
* Renamed source file arch/arm64/hyperv/hv_init.c to hv_core.c
  to better reflect its content
* Fixed the bit position of certain feature flags presented by
  Hyper-V to the guest.  The bit positions on ARM64 don't match
  the position on x86 like originally thought.
* Minor fixups to rebase to 5.6-rc5 linux-next

Changes in v5:
* Minor fixups to rebase to 5.4-rc1 linux-next

Changes in v4:
* Moved clock-related code into an architecture independent
  Hyper-V clocksource driver that is already upstream. Clock
  related code is removed from this patch set except for the
  ARM64 specific interrupt handler. [Marc Zyngier]
* Separately upstreamed the split of mshyperv.h into arch independent
  and arch dependent portions. The arch independent portion has been
  removed from this patch set.
* Divided patch #2 of the series into multiple smaller patches
  [Marc Zyngier]
* Changed a dozen or so smaller things based on feedback
  [Marc Zyngier, Will Deacon]
* Added functions to alloc/free Hyper-V size pages for use by
  drivers for Hyper-V synthetic devices when updated to not assume
  guest page size and Hyper-v page size are the same

Changes in v3:
* Added initialization of hv_vp_index array like was recently
  added on x86 branch [KY Srinivasan]
* Changed Hyper-V ARM64 register symbols to be all uppercase 
  instead of mixed case [KY Srinivasan]
* Separated mshyperv.h into two files, one architecture
  independent and one architecture dependent. After this code
  is upstream, will make changes to the x86 code to use the
  architecture independent file and remove duplication. And
  once we have a multi-architecture Hyper-V TLFS, will do a
  separate patch to split hyperv-tlfs.h in the same way.
  [KY Srinivasan]
* Minor tweaks to rebase to latest linux-next code

Changes in v2:
* Removed patch to implement slow_virt_to_phys() on ARM64.
  Use of slow_virt_to_phys() in arch independent Hyper-V
  drivers has been eliminated by commit 6ba34171bcbd
  ("Drivers: hv: vmbus: Remove use of slow_virt_to_phys()")
* Minor tweaks to rebase to latest linux-next code

Michael Kelley (5):
  arm64: hyperv: Add Hyper-V hypercall and register access utilities
  arm64: hyperv: Add panic handler
  arm64: hyperv: Initialize hypervisor on boot
  arm64: efi: Export screen_info
  Drivers: hv: Enable Hyper-V code to be built on ARM64

 MAINTAINERS                          |   3 +
 arch/arm64/Kbuild                    |   1 +
 arch/arm64/hyperv/Makefile           |   2 +
 arch/arm64/hyperv/hv_core.c          | 181 +++++++++++++++++++++++++++++++++++
 arch/arm64/hyperv/mshyperv.c         |  83 ++++++++++++++++
 arch/arm64/include/asm/hyperv-tlfs.h |  69 +++++++++++++
 arch/arm64/include/asm/mshyperv.h    |  54 +++++++++++
 arch/arm64/kernel/efi.c              |   1 +
 drivers/hv/Kconfig                   |   5 +-
 9 files changed, 397 insertions(+), 2 deletions(-)
 create mode 100644 arch/arm64/hyperv/Makefile
 create mode 100644 arch/arm64/hyperv/hv_core.c
 create mode 100644 arch/arm64/hyperv/mshyperv.c
 create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
 create mode 100644 arch/arm64/include/asm/mshyperv.h

Comments

Michael Kelley Aug. 2, 2021, 3:15 p.m. UTC | #1
From: Michael Kelley <mikelley@microsoft.com> Sent: Tuesday, July 20, 2021 7:57 AM

> 

> This series enables Linux guests running on Hyper-V on ARM64

> hardware. New ARM64-specific code in arch/arm64/hyperv initializes

> Hyper-V and its hypercall mechanism.  Existing architecture

> independent drivers for Hyper-V's VMbus and synthetic devices just

> work when built for ARM64. Hyper-V code is built and included in

> the image and modules only if CONFIG_HYPERV is enabled.


Will and Catalin --

Gentle ping. :-)   The clock/timer issues are now worked out in that
Hyper-V is providing full Generic Timer functionality in guest VMs.
That dropped some code out of the patch set, and making additional
code architecture neutral reduced the ARM64 specific code in this patch
set even further.  

To my knowledge, all issues have been addressed, so I'm looking for
any further review from you as ARM64 maintainers, or your Ack so
this can be ready for the 5.15 merge window.

Thanks!

Michael

> 

> The five patches are organized as follows:

> 

> 1) Add definitions and functions for making Hyper-V hypercalls

>    and getting/setting virtual processor registers provided by

>    Hyper-V

> 

> 2) Add the function needed by the arch independent VMbus driver

>    for reporting a panic to Hyper-V.

> 

> 3) Add Hyper-V initialization code and utility functions that

>    report Hyper-v status.

> 

> 4) Export screen_info so it may be used by the Hyper-V frame buffer

>    driver built as a module. It is already exported for x86,

>    powerpc, and alpha architectures.

> 

> 5) Make CONFIG_HYPERV selectable on ARM64 in addition to x86/x64.

> 

> Hyper-V on ARM64 runs with a 4 Kbyte page size, but allows guests

> with 4K/16K/64K page size. Linux guests with this patch series

> work with all three supported ARM64 page sizes.

> 

> The Hyper-V vPCI driver at drivers/pci/host/pci-hyperv.c has

> x86/x64-specific code and is not being built for ARM64. Enabling

> Hyper-V vPCI devices on ARM64 is in progress via a separate set

> of patches.

> 

> This patch set is based on the linux-next20210720 code tree.

> 

> Changes in v11:

> * Drop the previous Patch 1 as the fixes have already been

>   separately accepted upstream.

> * Drop the previous Patch 3 for enabling Hyper-V enlightened

>   clocks/timers.  Hyper-V is now offering the full ARM64

>   architectural Generic Timer in guest VMs, so the existing

>   arch_arch_timer.c driver just works. [Mark Rutland, Marc

>   Zyngier]

> * Simplified the previous Patch 4 and Patch 5 (now Patch 2 and 3)

>   by sharing more code between the x86 and ARM64 architectures.

> * Combined hyperv_early_init() and hyperv_init() into a single

>   initialization function that runs as an early_initcall. This

>   is possible because the Hyper-V enlightened clocks/timers

>   no longer need to be initialized early in the boot sequence.

> 

> Changes in v10:

> * Drop previous Patch 1 for SMCCC v1.2 HVC calls in favor of

>   Sudeep Holla's patch [Mark Rutland]

> * Use new helper functions in 5.13 for checking Hyper-V hypercall

>   return status

> * Set up per-CPU hypercall argument page that is now used by the

>   Hyper-V balloon driver

> * Rebase on 5.13-RC1

> 

> Changes in v9:

> * Added Patch 1 to enable making an SMCCC compliant hypercall

>   that returns results in other than registers X0 thru X3, per

>   version 1.2 and later of the SMCCC spec.

> * Using the ability to return results in registers X6 and X7,

>   converted hv_get_vpreg_128() to use a "fast" hypercall that

>   passes inputs and outputs in registers, and in doing so eliminated

>   a lot of memory allocation complexity.

> * Cleaned up some extra blank lines and use of spaces in aligning

>   local variables. [Sunil Muthuswamy]

> * Based on discussion about future directions, reverted the

>   population of hv_vp_index array to use a cpuhp state instead

>   of a hypercall, which is like it was in v7 and earlier.

> 

> Changes in v8:

> * Removed a lot of code based on refactoring the boundary between

>   arch independent and arch dependent code for Hyper-V, per comments

>   from Arnd Bergmann. The removed code was either duplicated on

>   the x86 side, or has been folded into architecture independent

>   code as not really being architecture dependent.

> * Added config dependency on !CONFIG_CPU_BIG_ENDIAN [Arnd Bergmann]

> * Reworked the approach to Hyper-V initialization. The functionality

>   is the same, but is now structured like the Xen code with an early

>   init function called in setup_arch() and an early initcall to

>   finish the initialization. [Arnd Bergmann]

> 

> Changes in v7:

> * Separately upstreamed split of hyperv-tlfs.h into arch dependent

>   and independent versions.  In this patch set, update the ARM64

>   hyperv-tlfs.h to include architecture independent definitions.

>   This approach eliminates a lot of lines of otherwise duplicated

>   code on the ARM64 side.

> * Break ARM64 mshyperv.h into smaller pieces. Have an initial

>   baseline, and add code along with patches for a particular

>   functional area. [Marc Zyngier]

> * In mshyperv.h, use static inline functions instead of #defines

>   where possible. [Arnd Bergmann]

> * Use VMbus INTID obtained from ACPI DSDT instead of hardcoding.

>   The STIMER INTID is still hardcoded because it is needed

>   before Linux has initialized the ACPI subsystem, so it can't

>   be obtained from the DSDT.  Wedging it into the GTDT seems

>   dubious, so was not done. [Marc Zyngier]

> * Update Hyper-V page size allocation functions to use

>   alloc_page() if PAGE_SIZE == HV_HYP_PAGE_SIZE [Arnd

>   Bergmann]

> * Various other minor changes based on feedback and to rebase

>   to latest linux-next [Marc Zyngier and Arnd Bergmann]

> 

> Changes in v6:

> * Use SMCCC hypercall interface instead of direct invocation

>   of HVC instruction and the Hyper-V hypercall interface

>   [Marc Zyngier]

> * Reimplemented functions to alloc/free Hyper-V size pages

>   using kmalloc/kfree since kmalloc now guarantees alignment of

>   power of 2 size allocations [Marc Zyngier]

> * Export screen_info in arm64 architecture so it can be used

>   by the Hyper-V buffer driver built as a module

> * Renamed source file arch/arm64/hyperv/hv_init.c to hv_core.c

>   to better reflect its content

> * Fixed the bit position of certain feature flags presented by

>   Hyper-V to the guest.  The bit positions on ARM64 don't match

>   the position on x86 like originally thought.

> * Minor fixups to rebase to 5.6-rc5 linux-next

> 

> Changes in v5:

> * Minor fixups to rebase to 5.4-rc1 linux-next

> 

> Changes in v4:

> * Moved clock-related code into an architecture independent

>   Hyper-V clocksource driver that is already upstream. Clock

>   related code is removed from this patch set except for the

>   ARM64 specific interrupt handler. [Marc Zyngier]

> * Separately upstreamed the split of mshyperv.h into arch independent

>   and arch dependent portions. The arch independent portion has been

>   removed from this patch set.

> * Divided patch #2 of the series into multiple smaller patches

>   [Marc Zyngier]

> * Changed a dozen or so smaller things based on feedback

>   [Marc Zyngier, Will Deacon]

> * Added functions to alloc/free Hyper-V size pages for use by

>   drivers for Hyper-V synthetic devices when updated to not assume

>   guest page size and Hyper-v page size are the same

> 

> Changes in v3:

> * Added initialization of hv_vp_index array like was recently

>   added on x86 branch [KY Srinivasan]

> * Changed Hyper-V ARM64 register symbols to be all uppercase

>   instead of mixed case [KY Srinivasan]

> * Separated mshyperv.h into two files, one architecture

>   independent and one architecture dependent. After this code

>   is upstream, will make changes to the x86 code to use the

>   architecture independent file and remove duplication. And

>   once we have a multi-architecture Hyper-V TLFS, will do a

>   separate patch to split hyperv-tlfs.h in the same way.

>   [KY Srinivasan]

> * Minor tweaks to rebase to latest linux-next code

> 

> Changes in v2:

> * Removed patch to implement slow_virt_to_phys() on ARM64.

>   Use of slow_virt_to_phys() in arch independent Hyper-V

>   drivers has been eliminated by commit 6ba34171bcbd

>   ("Drivers: hv: vmbus: Remove use of slow_virt_to_phys()")

> * Minor tweaks to rebase to latest linux-next code

> 

> Michael Kelley (5):

>   arm64: hyperv: Add Hyper-V hypercall and register access utilities

>   arm64: hyperv: Add panic handler

>   arm64: hyperv: Initialize hypervisor on boot

>   arm64: efi: Export screen_info

>   Drivers: hv: Enable Hyper-V code to be built on ARM64

> 

>  MAINTAINERS                          |   3 +

>  arch/arm64/Kbuild                    |   1 +

>  arch/arm64/hyperv/Makefile           |   2 +

>  arch/arm64/hyperv/hv_core.c          | 181 +++++++++++++++++++++++++++++++++++

>  arch/arm64/hyperv/mshyperv.c         |  83 ++++++++++++++++

>  arch/arm64/include/asm/hyperv-tlfs.h |  69 +++++++++++++

>  arch/arm64/include/asm/mshyperv.h    |  54 +++++++++++

>  arch/arm64/kernel/efi.c              |   1 +

>  drivers/hv/Kconfig                   |   5 +-

>  9 files changed, 397 insertions(+), 2 deletions(-)

>  create mode 100644 arch/arm64/hyperv/Makefile

>  create mode 100644 arch/arm64/hyperv/hv_core.c

>  create mode 100644 arch/arm64/hyperv/mshyperv.c

>  create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h

>  create mode 100644 arch/arm64/include/asm/mshyperv.h

> 

> --

> 1.8.3.1
Marc Zyngier Aug. 2, 2021, 4:01 p.m. UTC | #2
On Tue, 20 Jul 2021 15:56:58 +0100,
Michael Kelley <mikelley@microsoft.com> wrote:
> 

> This series enables Linux guests running on Hyper-V on ARM64

> hardware. New ARM64-specific code in arch/arm64/hyperv initializes

> Hyper-V and its hypercall mechanism.  Existing architecture

> independent drivers for Hyper-V's VMbus and synthetic devices just

> work when built for ARM64. Hyper-V code is built and included in

> the image and modules only if CONFIG_HYPERV is enabled.

> 

> The five patches are organized as follows:

> 

> 1) Add definitions and functions for making Hyper-V hypercalls

>    and getting/setting virtual processor registers provided by

>    Hyper-V

> 

> 2) Add the function needed by the arch independent VMbus driver

>    for reporting a panic to Hyper-V.

> 

> 3) Add Hyper-V initialization code and utility functions that

>    report Hyper-v status.

> 

> 4) Export screen_info so it may be used by the Hyper-V frame buffer

>    driver built as a module. It is already exported for x86,

>    powerpc, and alpha architectures.

> 

> 5) Make CONFIG_HYPERV selectable on ARM64 in addition to x86/x64.

> 

> Hyper-V on ARM64 runs with a 4 Kbyte page size, but allows guests

> with 4K/16K/64K page size. Linux guests with this patch series

> work with all three supported ARM64 page sizes.

> 

> The Hyper-V vPCI driver at drivers/pci/host/pci-hyperv.c has

> x86/x64-specific code and is not being built for ARM64. Enabling

> Hyper-V vPCI devices on ARM64 is in progress via a separate set

> of patches.

> 

> This patch set is based on the linux-next20210720 code tree.

> 

> Changes in v11:

> * Drop the previous Patch 1 as the fixes have already been

>   separately accepted upstream.

> * Drop the previous Patch 3 for enabling Hyper-V enlightened

>   clocks/timers.  Hyper-V is now offering the full ARM64

>   architectural Generic Timer in guest VMs, so the existing

>   arch_arch_timer.c driver just works. [Mark Rutland, Marc

>   Zyngier]


Thanks for doing this. Assuming you fix the issue I mentioned in my
reply to patch #3. FWIW:

Acked-by: Marc Zyngier <maz@kernel.org>


	M.

-- 
Without deviation from the norm, progress is not possible.
Mark Rutland Aug. 2, 2021, 4:28 p.m. UTC | #3
On Mon, Aug 02, 2021 at 05:01:02PM +0100, Marc Zyngier wrote:
> On Tue, 20 Jul 2021 15:56:58 +0100,

> Michael Kelley <mikelley@microsoft.com> wrote:

> > 

> > This series enables Linux guests running on Hyper-V on ARM64

> > hardware. New ARM64-specific code in arch/arm64/hyperv initializes

> > Hyper-V and its hypercall mechanism.  Existing architecture

> > independent drivers for Hyper-V's VMbus and synthetic devices just

> > work when built for ARM64. Hyper-V code is built and included in

> > the image and modules only if CONFIG_HYPERV is enabled.

> > 

> > The five patches are organized as follows:

> > 

> > 1) Add definitions and functions for making Hyper-V hypercalls

> >    and getting/setting virtual processor registers provided by

> >    Hyper-V

> > 

> > 2) Add the function needed by the arch independent VMbus driver

> >    for reporting a panic to Hyper-V.

> > 

> > 3) Add Hyper-V initialization code and utility functions that

> >    report Hyper-v status.

> > 

> > 4) Export screen_info so it may be used by the Hyper-V frame buffer

> >    driver built as a module. It is already exported for x86,

> >    powerpc, and alpha architectures.

> > 

> > 5) Make CONFIG_HYPERV selectable on ARM64 in addition to x86/x64.

> > 

> > Hyper-V on ARM64 runs with a 4 Kbyte page size, but allows guests

> > with 4K/16K/64K page size. Linux guests with this patch series

> > work with all three supported ARM64 page sizes.

> > 

> > The Hyper-V vPCI driver at drivers/pci/host/pci-hyperv.c has

> > x86/x64-specific code and is not being built for ARM64. Enabling

> > Hyper-V vPCI devices on ARM64 is in progress via a separate set

> > of patches.

> > 

> > This patch set is based on the linux-next20210720 code tree.

> > 

> > Changes in v11:

> > * Drop the previous Patch 1 as the fixes have already been

> >   separately accepted upstream.

> > * Drop the previous Patch 3 for enabling Hyper-V enlightened

> >   clocks/timers.  Hyper-V is now offering the full ARM64

> >   architectural Generic Timer in guest VMs, so the existing

> >   arch_arch_timer.c driver just works. [Mark Rutland, Marc

> >   Zyngier]

> 

> Thanks for doing this. Assuming you fix the issue I mentioned in my

> reply to patch #3. FWIW:

> 

> Acked-by: Marc Zyngier <maz@kernel.org>


I've tried to provide concrete options for those. With those fixed up
somehow:

Acked-by: Mark Rutland <mark.rutland@arm.com>


Mark.