From patchwork Tue Jan 9 19:37:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 761171 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 096D43D961; Tue, 9 Jan 2024 19:38:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=quicinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=quicinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=quicinc.com header.i=@quicinc.com header.b="TDAArpDa" Received: from pps.filterd (m0279868.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.24/8.17.1.24) with ESMTP id 409GaJ18010710; Tue, 9 Jan 2024 19:37:57 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h= from:date:subject:mime-version:content-type :content-transfer-encoding:message-id:references:in-reply-to:to :cc; s=qcppdkim1; bh=xWdZbdG7dSafWq3cabzRvyRx9NAZngj+f9YcIbeZBfY =; b=TDAArpDahUKJYT7Ji4NE1rBHgi9T9MX6/TKw57bQQmvVzjuO7uQw0AHQLOS Ru7UbgJaSCXKlurAPEVVoULnJ8dyMtzYsYRJfxwGW1YVhFS+laYSAyr1cEWK1wH7 FnWH/fBjD7+iEyaaUwUeonBaw23vg0oR0ZEkjdz57ZBxnC4Imxor1MArNGW8BoE3 i4Lx7BTIvLf/7bOvorKdKiJggh3qm0CJUl7xYCguAHA0oHGKSyUmTMnluXLj1nKd k/obzOk5EFohZVl1J4FW+JLwuFaLDd/sgPojysn7Q1gIHCKUSMZ+AbAn+ZzSZC2j 39vNCH9hRwOh7UuVwR+iRBmpwdg== Received: from nasanppmta03.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3vh9ta0dw8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 09 Jan 2024 19:37:57 +0000 (GMT) Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA03.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 409JbuQD011419 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 9 Jan 2024 19:37:56 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.40; Tue, 9 Jan 2024 11:37:55 -0800 From: Elliot Berman Date: Tue, 9 Jan 2024 11:37:48 -0800 Subject: [PATCH v16 10/34] gunyah: vm_mgr: Add VM start/stop Precedence: bulk X-Mailing-List: devicetree@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-ID: <20240109-gunyah-v16-10-634904bf4ce9@quicinc.com> References: <20240109-gunyah-v16-0-634904bf4ce9@quicinc.com> In-Reply-To: <20240109-gunyah-v16-0-634904bf4ce9@quicinc.com> To: Alex Elder , Srinivas Kandagatla , Murali Nalajal , Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Philip Derrin , Prakruthi Deepak Heragu , Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Conor Dooley , Catalin Marinas , Will Deacon , Konrad Dybcio , Bjorn Andersson , Dmitry Baryshkov , "Fuad Tabba" , Sean Christopherson , "Andrew Morton" CC: , , , , , , Elliot Berman X-Mailer: b4 0.13-dev X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: xkWEhD3cCcd4_26fH5x9GQrV4OQNq86C X-Proofpoint-ORIG-GUID: xkWEhD3cCcd4_26fH5x9GQrV4OQNq86C X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-09_02,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 impostorscore=0 malwarescore=0 mlxscore=0 adultscore=0 mlxlogscore=999 bulkscore=0 priorityscore=1501 lowpriorityscore=0 clxscore=1015 phishscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2311290000 definitions=main-2401090158 Add ioctl to trigger the start of a Gunyah virtual machine. Subsequent commits will provide memory to the virtual machine and add ability to interact with the resources (capabilities) of the virtual machine. Although start of the virtual machine can be done implicitly on the first vCPU run for proxy-schedule virtual machines, there is a non-trivial number of calls to Gunyah: a more precise error can be given to userspace which calls VM_START without looking at kernel logs because userspace can detect that the VM start failed instead of "couldn't run the vCPU". Co-developed-by: Prakruthi Deepak Heragu Signed-off-by: Prakruthi Deepak Heragu Signed-off-by: Elliot Berman --- drivers/virt/gunyah/vm_mgr.c | 198 +++++++++++++++++++++++++++++++++++++++++++ drivers/virt/gunyah/vm_mgr.h | 19 +++++ include/uapi/linux/gunyah.h | 5 ++ 3 files changed, 222 insertions(+) diff --git a/drivers/virt/gunyah/vm_mgr.c b/drivers/virt/gunyah/vm_mgr.c index e9dff733e35e..f6e6b5669aae 100644 --- a/drivers/virt/gunyah/vm_mgr.c +++ b/drivers/virt/gunyah/vm_mgr.c @@ -15,6 +15,68 @@ #include "rsc_mgr.h" #include "vm_mgr.h" +static int gunyah_vm_rm_notification_status(struct gunyah_vm *ghvm, void *data) +{ + struct gunyah_rm_vm_status_payload *payload = data; + + if (le16_to_cpu(payload->vmid) != ghvm->vmid) + return NOTIFY_OK; + + /* All other state transitions are synchronous to a corresponding RM call */ + if (payload->vm_status == GUNYAH_RM_VM_STATUS_RESET) { + down_write(&ghvm->status_lock); + ghvm->vm_status = payload->vm_status; + up_write(&ghvm->status_lock); + wake_up(&ghvm->vm_status_wait); + } + + return NOTIFY_DONE; +} + +static int gunyah_vm_rm_notification_exited(struct gunyah_vm *ghvm, void *data) +{ + struct gunyah_rm_vm_exited_payload *payload = data; + + if (le16_to_cpu(payload->vmid) != ghvm->vmid) + return NOTIFY_OK; + + down_write(&ghvm->status_lock); + ghvm->vm_status = GUNYAH_RM_VM_STATUS_EXITED; + up_write(&ghvm->status_lock); + wake_up(&ghvm->vm_status_wait); + + return NOTIFY_DONE; +} + +static int gunyah_vm_rm_notification(struct notifier_block *nb, + unsigned long action, void *data) +{ + struct gunyah_vm *ghvm = container_of(nb, struct gunyah_vm, nb); + + switch (action) { + case GUNYAH_RM_NOTIFICATION_VM_STATUS: + return gunyah_vm_rm_notification_status(ghvm, data); + case GUNYAH_RM_NOTIFICATION_VM_EXITED: + return gunyah_vm_rm_notification_exited(ghvm, data); + default: + return NOTIFY_OK; + } +} + +static void gunyah_vm_stop(struct gunyah_vm *ghvm) +{ + int ret; + + if (ghvm->vm_status == GUNYAH_RM_VM_STATUS_RUNNING) { + ret = gunyah_rm_vm_stop(ghvm->rm, ghvm->vmid); + if (ret) + dev_warn(ghvm->parent, "Failed to stop VM: %d\n", ret); + } + + wait_event(ghvm->vm_status_wait, + ghvm->vm_status != GUNYAH_RM_VM_STATUS_RUNNING); +} + static __must_check struct gunyah_vm *gunyah_vm_alloc(struct gunyah_rm *rm) { struct gunyah_vm *ghvm; @@ -24,14 +86,148 @@ static __must_check struct gunyah_vm *gunyah_vm_alloc(struct gunyah_rm *rm) return ERR_PTR(-ENOMEM); ghvm->parent = gunyah_rm_get(rm); + ghvm->vmid = GUNYAH_VMID_INVAL; ghvm->rm = rm; + init_rwsem(&ghvm->status_lock); + init_waitqueue_head(&ghvm->vm_status_wait); + ghvm->vm_status = GUNYAH_RM_VM_STATUS_NO_STATE; + return ghvm; } +static int gunyah_vm_start(struct gunyah_vm *ghvm) +{ + int ret; + + down_write(&ghvm->status_lock); + if (ghvm->vm_status != GUNYAH_RM_VM_STATUS_NO_STATE) { + up_write(&ghvm->status_lock); + return 0; + } + + ghvm->nb.notifier_call = gunyah_vm_rm_notification; + ret = gunyah_rm_notifier_register(ghvm->rm, &ghvm->nb); + if (ret) + goto err; + + ret = gunyah_rm_alloc_vmid(ghvm->rm, 0); + if (ret < 0) { + gunyah_rm_notifier_unregister(ghvm->rm, &ghvm->nb); + goto err; + } + ghvm->vmid = ret; + ghvm->vm_status = GUNYAH_RM_VM_STATUS_LOAD; + + ret = gunyah_rm_vm_configure(ghvm->rm, ghvm->vmid, ghvm->auth, 0, 0, 0, + 0, 0); + if (ret) { + dev_warn(ghvm->parent, "Failed to configure VM: %d\n", ret); + goto err; + } + + ret = gunyah_rm_vm_init(ghvm->rm, ghvm->vmid); + if (ret) { + ghvm->vm_status = GUNYAH_RM_VM_STATUS_INIT_FAILED; + dev_warn(ghvm->parent, "Failed to initialize VM: %d\n", ret); + goto err; + } + ghvm->vm_status = GUNYAH_RM_VM_STATUS_READY; + + ret = gunyah_rm_vm_start(ghvm->rm, ghvm->vmid); + if (ret) { + dev_warn(ghvm->parent, "Failed to start VM: %d\n", ret); + goto err; + } + + ghvm->vm_status = GUNYAH_RM_VM_STATUS_RUNNING; + up_write(&ghvm->status_lock); + return ret; +err: + /* gunyah_vm_free will handle releasing resources and reclaiming memory */ + up_write(&ghvm->status_lock); + return ret; +} + +static int gunyah_vm_ensure_started(struct gunyah_vm *ghvm) +{ + int ret; + + ret = down_read_interruptible(&ghvm->status_lock); + if (ret) + return ret; + + /* Unlikely because VM is typically started */ + if (unlikely(ghvm->vm_status == GUNYAH_RM_VM_STATUS_NO_STATE)) { + up_read(&ghvm->status_lock); + ret = gunyah_vm_start(ghvm); + if (ret) + return ret; + /** gunyah_vm_start() is guaranteed to bring status out of + * GUNYAH_RM_VM_STATUS_LOAD, thus infinitely recursive call is not + * possible + */ + return gunyah_vm_ensure_started(ghvm); + } + + /* Unlikely because VM is typically running */ + if (unlikely(ghvm->vm_status != GUNYAH_RM_VM_STATUS_RUNNING)) + ret = -ENODEV; + + up_read(&ghvm->status_lock); + return ret; +} + +static long gunyah_vm_ioctl(struct file *filp, unsigned int cmd, + unsigned long arg) +{ + struct gunyah_vm *ghvm = filp->private_data; + long r; + + switch (cmd) { + case GUNYAH_VM_START: { + r = gunyah_vm_ensure_started(ghvm); + break; + } + default: + r = -ENOTTY; + break; + } + + return r; +} + static int gunyah_vm_release(struct inode *inode, struct file *filp) { struct gunyah_vm *ghvm = filp->private_data; + int ret; + + /** + * We might race with a VM exit notification, but that's ok: + * gh_rm_vm_stop() will just return right away. + */ + if (ghvm->vm_status == GUNYAH_RM_VM_STATUS_RUNNING) + gunyah_vm_stop(ghvm); + + if (ghvm->vm_status != GUNYAH_RM_VM_STATUS_NO_STATE && + ghvm->vm_status != GUNYAH_RM_VM_STATUS_LOAD && + ghvm->vm_status != GUNYAH_RM_VM_STATUS_RESET) { + ret = gunyah_rm_vm_reset(ghvm->rm, ghvm->vmid); + if (ret) + dev_err(ghvm->parent, "Failed to reset the vm: %d\n", + ret); + wait_event(ghvm->vm_status_wait, + ghvm->vm_status == GUNYAH_RM_VM_STATUS_RESET); + } + + if (ghvm->vm_status > GUNYAH_RM_VM_STATUS_NO_STATE) { + gunyah_rm_notifier_unregister(ghvm->rm, &ghvm->nb); + + ret = gunyah_rm_dealloc_vmid(ghvm->rm, ghvm->vmid); + if (ret) + dev_warn(ghvm->parent, + "Failed to deallocate vmid: %d\n", ret); + } gunyah_rm_put(ghvm->rm); kfree(ghvm); @@ -40,6 +236,8 @@ static int gunyah_vm_release(struct inode *inode, struct file *filp) static const struct file_operations gunyah_vm_fops = { .owner = THIS_MODULE, + .unlocked_ioctl = gunyah_vm_ioctl, + .compat_ioctl = compat_ptr_ioctl, .release = gunyah_vm_release, .llseek = noop_llseek, }; diff --git a/drivers/virt/gunyah/vm_mgr.h b/drivers/virt/gunyah/vm_mgr.h index 50790d402676..e6cc9aead0b6 100644 --- a/drivers/virt/gunyah/vm_mgr.h +++ b/drivers/virt/gunyah/vm_mgr.h @@ -7,6 +7,8 @@ #define _GUNYAH_VM_MGR_PRIV_H #include +#include +#include #include @@ -17,12 +19,29 @@ long gunyah_dev_vm_mgr_ioctl(struct gunyah_rm *rm, unsigned int cmd, /** * struct gunyah_vm - Main representation of a Gunyah Virtual machine + * @vmid: Gunyah's VMID for this virtual machine * @rm: Pointer to the resource manager struct to make RM calls * @parent: For logging + * @nb: Notifier block for RM notifications + * @vm_status: Current state of the VM, as last reported by RM + * @vm_status_wait: Wait queue for status @vm_status changes + * @status_lock: Serializing state transitions + * @auth: Authentication mechanism to be used by resource manager when + * launching the VM + * + * Members are grouped by hot path. */ struct gunyah_vm { + u16 vmid; struct gunyah_rm *rm; + + struct notifier_block nb; + enum gunyah_rm_vm_status vm_status; + wait_queue_head_t vm_status_wait; + struct rw_semaphore status_lock; + struct device *parent; + enum gunyah_rm_vm_auth_mechanism auth; }; #endif diff --git a/include/uapi/linux/gunyah.h b/include/uapi/linux/gunyah.h index ac338ec4b85d..31e7f79a6c39 100644 --- a/include/uapi/linux/gunyah.h +++ b/include/uapi/linux/gunyah.h @@ -20,4 +20,9 @@ */ #define GUNYAH_CREATE_VM _IO(GUNYAH_IOCTL_TYPE, 0x0) /* Returns a Gunyah VM fd */ +/* + * ioctls for gunyah-vm fds (returned by GUNYAH_CREATE_VM) + */ +#define GUNYAH_VM_START _IO(GUNYAH_IOCTL_TYPE, 0x3) + #endif