[3/7] target-arm: Add minimal KVM AArch64 support

From: "Mian M. Hamayun" <m.hamayun@virtualopensystems.com>

Add the bare minimum set of functions needed for control of an
AArch64 KVM vcpu:
 * CPU initialization
 * minimal get/put register functions which only handle the
   basic state of the CPU

Signed-off-by: Mian M. Hamayun <m.hamayun@virtualopensystems.com>
[PMM: significantly overhauled; most notably:
 * code lives in kvm64.c rather than using #ifdefs
 * support '-cpu host' rather than implicitly using whatever the
   host's CPU is regardless of what the user requests
 * fix bug attempting to get/set nonexistent X[31]
 * fix bug writing 64 bit kernel pstate into uint32_t env field
]
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
 target-arm/Makefile.objs |    1 +
 target-arm/kvm.c         |    4 +
 target-arm/kvm64.c       |  204 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 209 insertions(+)
 create mode 100644 target-arm/kvm64.c

On Thu, Nov 28, 2013 at 01:33:18PM +0000, Peter Maydell wrote:
> From: "Mian M. Hamayun" <m.hamayun@virtualopensystems.com>
> 
> Add the bare minimum set of functions needed for control of an
> AArch64 KVM vcpu:
>  * CPU initialization
>  * minimal get/put register functions which only handle the
>    basic state of the CPU
> 
> Signed-off-by: Mian M. Hamayun <m.hamayun@virtualopensystems.com>
> [PMM: significantly overhauled; most notably:
>  * code lives in kvm64.c rather than using #ifdefs
>  * support '-cpu host' rather than implicitly using whatever the
>    host's CPU is regardless of what the user requests
>  * fix bug attempting to get/set nonexistent X[31]
>  * fix bug writing 64 bit kernel pstate into uint32_t env field
> ]
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

Besides the minor nits/questions below:

Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>

> ---
>  target-arm/Makefile.objs |    1 +
>  target-arm/kvm.c         |    4 +
>  target-arm/kvm64.c       |  204 ++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 209 insertions(+)
>  create mode 100644 target-arm/kvm64.c
> 
> diff --git a/target-arm/Makefile.objs b/target-arm/Makefile.objs
> index d1db77c..5493a4c 100644
> --- a/target-arm/Makefile.objs
> +++ b/target-arm/Makefile.objs
> @@ -7,3 +7,4 @@ obj-y += neon_helper.o iwmmxt_helper.o
>  obj-y += gdbstub.o
>  obj-$(TARGET_AARCH64) += cpu64.o translate-a64.o gdbstub64.o
>  obj-$(call land,$(CONFIG_KVM),$(call lnot,$(TARGET_AARCH64))) += kvm32.o
> +obj-$(call land,$(CONFIG_KVM),$(TARGET_AARCH64)) += kvm64.o
> diff --git a/target-arm/kvm.c b/target-arm/kvm.c
> index 5cdb3b9..1d2688d 100644
> --- a/target-arm/kvm.c
> +++ b/target-arm/kvm.c
> @@ -128,7 +128,11 @@ static void kvm_arm_host_cpu_initfn(Object *obj)
>  
>  static const TypeInfo host_arm_cpu_type_info = {
>      .name = TYPE_ARM_HOST_CPU,
> +#ifdef TARGET_AARCH64
> +    .parent = TYPE_AARCH64_CPU,
> +#else
>      .parent = TYPE_ARM_CPU,
> +#endif
>      .instance_init = kvm_arm_host_cpu_initfn,
>      .class_init = kvm_arm_host_cpu_class_init,
>      .class_size = sizeof(ARMHostCPUClass),
> diff --git a/target-arm/kvm64.c b/target-arm/kvm64.c
> new file mode 100644
> index 0000000..599db5d
> --- /dev/null
> +++ b/target-arm/kvm64.c
> @@ -0,0 +1,204 @@
> +/*
> + * ARM implementation of KVM hooks, 64 bit specific code
> + *
> + * Copyright Mian-M. Hamayun 2013, Virtual Open Systems
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#include <stdio.h>
> +#include <sys/types.h>
> +#include <sys/ioctl.h>
> +#include <sys/mman.h>
> +
> +#include <linux/kvm.h>
> +
> +#include "qemu-common.h"
> +#include "qemu/timer.h"
> +#include "sysemu/sysemu.h"
> +#include "sysemu/kvm.h"
> +#include "kvm_arm.h"
> +#include "cpu.h"
> +#include "hw/arm/arm.h"
> +
> +static inline void set_feature(uint64_t *features, int feature)
> +{
> +    *features |= 1ULL << feature;
> +}
> +
> +bool kvm_arm_get_host_cpu_features(ARMHostCPUClass *ahcc)
> +{
> +    /* Identify the feature bits corresponding to the host CPU, and
> +     * fill out the ARMHostCPUClass fields accordingly. To do this
> +     * we have to create a scratch VM, create a single CPU inside it,
> +     * and then query that CPU for the relevant ID registers.
> +     * For AArch64 we currently don't care about ID registers at
> +     * all; we just want to know the CPU type.
> +     */
> +    int fdarray[3];
> +    uint64_t features = 0;
> +    /* Old kernels may not know about the PREFERRED_TARGET ioctl: however
> +     * we know these will only support creating one kind of guest CPU,
> +     * which is its preferred CPU type. Fortunately these old kernels
> +     * support only a very limited number of CPUs.
> +     */
> +    static const uint32_t cpus_to_try[] = {
> +        KVM_ARM_TARGET_AEM_V8,
> +        KVM_ARM_TARGET_FOUNDATION_V8,
> +        KVM_ARM_TARGET_CORTEX_A57,
> +        QEMU_KVM_ARM_TARGET_NONE
> +    };
> +    struct kvm_vcpu_init init;
> +
> +    if (!kvm_arm_create_scratch_host_vcpu(cpus_to_try, fdarray, &init)) {
> +        return false;
> +    }
> +
> +    ahcc->target = init.target;
> +    ahcc->dtb_compatible = "arm,arm-v7";

arm,arm-v8 ?

> +
> +    kvm_arm_destroy_scratch_host_vcpu(fdarray);
> +
> +   /* We can assume any KVM supporting CPU is at least a v8
> +     * with VFPv4+Neon; this in turn implies most of the other
> +     * feature bits.

not sure I understand the bit about implying other feature bits, the
only other thing we're setting here is AARCH64 and the features bits are
enum values?

> +     */
> +    set_feature(&features, ARM_FEATURE_V8);
> +    set_feature(&features, ARM_FEATURE_VFP4);
> +    set_feature(&features, ARM_FEATURE_NEON);
> +    set_feature(&features, ARM_FEATURE_AARCH64);
> +
> +    ahcc->features = features;
> +
> +    return true;
> +}
> +
> +int kvm_arch_init_vcpu(CPUState *cs)
> +{
> +    ARMCPU *cpu = ARM_CPU(cs);
> +    struct kvm_vcpu_init init;
> +    int ret;
> +
> +    if (cpu->kvm_target == QEMU_KVM_ARM_TARGET_NONE ||
> +        !arm_feature(&cpu->env, ARM_FEATURE_AARCH64)) {
> +        fprintf(stderr, "KVM is not supported for this guest CPU type\n");
> +        return -EINVAL;
> +    }
> +
> +    init.target = cpu->kvm_target;
> +    memset(init.features, 0, sizeof(init.features));
> +    if (cpu->start_powered_off) {
> +        init.features[0] = 1 << KVM_ARM_VCPU_POWER_OFF;
> +    }
> +    ret = kvm_vcpu_ioctl(cs, KVM_ARM_VCPU_INIT, &init);
> +
> +    /* TODO : support for save/restore/reset of system regs via tuple list */
> +
> +    return ret;
> +}
> +
> +#define AARCH64_CORE_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
> +                 KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
> +
> +int kvm_arch_put_registers(CPUState *cs, int level)
> +{
> +    struct kvm_one_reg reg;
> +    uint64_t val;
> +    int i;
> +    int ret;
> +
> +    ARMCPU *cpu = ARM_CPU(cs);
> +    CPUARMState *env = &cpu->env;
> +
> +    for (i = 0; i < 31; i++) {
> +        reg.id = AARCH64_CORE_REG(regs.regs[i]);
> +        reg.addr = (uintptr_t) &env->xregs[i];
> +        ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
> +        if (ret) {
> +            return ret;
> +        }
> +    }
> +
> +    reg.id = AARCH64_CORE_REG(regs.sp);
> +    reg.addr = (uintptr_t) &env->xregs[31];
> +    ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
> +    if (ret) {
> +        return ret;
> +    }
> +
> +    /* Note that KVM thinks pstate is 64 bit but we use a uint32_t */
> +    val = pstate_read(env);
> +    reg.id = AARCH64_CORE_REG(regs.pstate);
> +    reg.addr = (uintptr_t) &val;
> +    ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
> +    if (ret) {
> +        return ret;
> +    }
> +
> +    reg.id = AARCH64_CORE_REG(regs.pc);
> +    reg.addr = (uintptr_t) &env->pc;
> +    ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
> +    if (ret) {
> +        return ret;
> +    }
> +
> +    /* TODO:
> +     * SP_EL1
> +     * ELR_EL1
> +     * SPSR[]
> +     * FP state
> +     * system registers
> +     */
> +    return ret;
> +}
> +
> +int kvm_arch_get_registers(CPUState *cs)
> +{
> +    struct kvm_one_reg reg;
> +    uint64_t val;
> +    int i;
> +    int ret;
> +
> +    ARMCPU *cpu = ARM_CPU(cs);
> +    CPUARMState *env = &cpu->env;
> +
> +    for (i = 0; i < 31; i++) {
> +        reg.id = AARCH64_CORE_REG(regs.regs[i]);
> +        reg.addr = (uintptr_t) &env->xregs[i];
> +        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
> +        if (ret) {
> +            return ret;
> +        }
> +    }
> +
> +    reg.id = AARCH64_CORE_REG(regs.sp);
> +    reg.addr = (uintptr_t) &env->xregs[31];
> +    ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
> +    if (ret) {
> +        return ret;
> +    }
> +
> +    reg.id = AARCH64_CORE_REG(regs.pstate);
> +    reg.addr = (uintptr_t) &val;
> +    ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
> +    if (ret) {
> +        return ret;
> +    }
> +    pstate_write(env, val);
> +
> +    reg.id = AARCH64_CORE_REG(regs.pc);
> +    reg.addr = (uintptr_t) &env->pc;
> +    ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
> +    if (ret) {
> +        return ret;
> +    }
> +
> +    /* TODO: other registers */
> +    return ret;
> +}
> +
> +void kvm_arch_reset_vcpu(CPUState *cs)
> +{
> +}
> -- 
> 1.7.9.5
> 
> _______________________________________________
> kvmarm mailing list
> kvmarm@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/cucslists/listinfo/kvmarm

On 16 December 2013 23:39, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> On Thu, Nov 28, 2013 at 01:33:18PM +0000, Peter Maydell wrote:
>> +    ahcc->target = init.target;
>> +    ahcc->dtb_compatible = "arm,arm-v7";
>
> arm,arm-v8 ?

Oops, yes, cut-n-pasto.


>
>> +
>> +    kvm_arm_destroy_scratch_host_vcpu(fdarray);
>> +
>> +   /* We can assume any KVM supporting CPU is at least a v8
>> +     * with VFPv4+Neon; this in turn implies most of the other
>> +     * feature bits.
>
> not sure I understand the bit about implying other feature bits, the
> only other thing we're setting here is AARCH64 and the features bits are
> enum values?

target-arm/cpu.c:cpu_realize_fn() has a large set of if statements
like

    if (arm_feature(env, ARM_FEATURE_V8)) {
        set_feature(env, ARM_FEATURE_V7);
        set_feature(env, ARM_FEATURE_ARM_DIV);
        set_feature(env, ARM_FEATURE_LPAE);
    }

because architecturally some features or arch versions imply
that you have others (eg above v8 means we always know
we have LPAE and division)...

>> +     */
>> +    set_feature(&features, ARM_FEATURE_V8);
>> +    set_feature(&features, ARM_FEATURE_VFP4);
>> +    set_feature(&features, ARM_FEATURE_NEON);
>> +    set_feature(&features, ARM_FEATURE_AARCH64);

...and because presence of the 'v8', 'vfp4', 'neon' features implies
(as enforced via those if statements) presence of just about every
other feature it means we don't need to have specific
tests for "do the CPU's feature registers say we support
division?" like the v7 KVM code does, because we know
that it's all implied automatically.

thanks
-- PMM

On Tue, Dec 17, 2013 at 12:21:27AM +0000, Peter Maydell wrote:
> On 16 December 2013 23:39, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> > On Thu, Nov 28, 2013 at 01:33:18PM +0000, Peter Maydell wrote:
> >> +    ahcc->target = init.target;
> >> +    ahcc->dtb_compatible = "arm,arm-v7";
> >
> > arm,arm-v8 ?
> 
> Oops, yes, cut-n-pasto.
> 
> 
> >
> >> +
> >> +    kvm_arm_destroy_scratch_host_vcpu(fdarray);
> >> +
> >> +   /* We can assume any KVM supporting CPU is at least a v8
> >> +     * with VFPv4+Neon; this in turn implies most of the other
> >> +     * feature bits.
> >
> > not sure I understand the bit about implying other feature bits, the
> > only other thing we're setting here is AARCH64 and the features bits are
> > enum values?
> 
> target-arm/cpu.c:cpu_realize_fn() has a large set of if statements
> like
> 
>     if (arm_feature(env, ARM_FEATURE_V8)) {
>         set_feature(env, ARM_FEATURE_V7);
>         set_feature(env, ARM_FEATURE_ARM_DIV);
>         set_feature(env, ARM_FEATURE_LPAE);
>     }
> 
> because architecturally some features or arch versions imply
> that you have others (eg above v8 means we always know
> we have LPAE and division)...
> 
> >> +     */
> >> +    set_feature(&features, ARM_FEATURE_V8);
> >> +    set_feature(&features, ARM_FEATURE_VFP4);
> >> +    set_feature(&features, ARM_FEATURE_NEON);
> >> +    set_feature(&features, ARM_FEATURE_AARCH64);
> 
> ...and because presence of the 'v8', 'vfp4', 'neon' features implies
> (as enforced via those if statements) presence of just about every
> other feature it means we don't need to have specific
> tests for "do the CPU's feature registers say we support
> division?" like the v7 KVM code does, because we know
> that it's all implied automatically.
> 
Got it, I was looking for something like that, but somehow missed the
realize function.  Thanks!