From patchwork Fri Jan 20 22:46:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 645109 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15E39C38159 for ; Fri, 20 Jan 2023 22:48:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229910AbjATWsC (ORCPT ); Fri, 20 Jan 2023 17:48:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229893AbjATWsA (ORCPT ); Fri, 20 Jan 2023 17:48:00 -0500 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27C509778; Fri, 20 Jan 2023 14:47:46 -0800 (PST) Received: from pps.filterd (m0279866.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KMb24V027326; Fri, 20 Jan 2023 22:47:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=9VxO+yl00zUxXfgp6XmdCqU9vdB2vjmzFe03W70hmXM=; b=k9+ZujqoqEuQFTuuZOLOOMoR3mCWw+sQffyPoD4EAiomqOthvPzw5Nd2AVHNsq4x9yQM ILlrWbi+fPuu8zMy8OjBcTGK97WE4L6+FMeqO/12CLdbnhNEkxEnOMM2ycEwlDbUUxj0 hNcjHPuHIv0jmH6ydvjKGuKADcW1WRZMbvikIzljDKNi2TO5foo2PWFsb+DKnllxFkYZ hxDx0s4MIS1YYBriTyI5jFPoj3V9E47G3RXpwkVP0TLYE17zDxRAavBx3aHtNcBpnnLQ LstqHLIGbpBLVXwrCsLj6R4DAPlZfjpIig3hA8v+DTRhxMrbuXtldnt2zVvqy3imnAJS Xg== Received: from nasanppmta04.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7yc60ggu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:31 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA04.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMlUgt022158 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:30 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:29 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala , Jonathan Corbet CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Bagas Sanjaya , Catalin Marinas , "Will Deacon" , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 01/27] docs: gunyah: Introduce Gunyah Hypervisor Date: Fri, 20 Jan 2023 14:46:00 -0800 Message-ID: <20230120224627.4053418-2-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: VN-ujdXqtakzk-7582zg6dKwFPbGs2wi X-Proofpoint-ORIG-GUID: VN-ujdXqtakzk-7582zg6dKwFPbGs2wi X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 mlxscore=0 priorityscore=1501 spamscore=0 impostorscore=0 phishscore=0 clxscore=1015 mlxlogscore=999 adultscore=0 malwarescore=0 bulkscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Gunyah is an open-source Type-1 hypervisor developed by Qualcomm. It does not depend on any lower-privileged OS/kernel code for its core functionality. This increases its security and can support a smaller trusted computing based when compared to Type-2 hypervisors. Add documentation describing the Gunyah hypervisor and the main components of the Gunyah hypervisor which are of interest to Linux virtualization development. Reviewed-by: Bagas Sanjaya Signed-off-by: Elliot Berman --- Documentation/virt/gunyah/index.rst | 113 ++++++++++++++++++++ Documentation/virt/gunyah/message-queue.rst | 61 +++++++++++ Documentation/virt/index.rst | 1 + MAINTAINERS | 7 ++ 4 files changed, 182 insertions(+) create mode 100644 Documentation/virt/gunyah/index.rst create mode 100644 Documentation/virt/gunyah/message-queue.rst diff --git a/Documentation/virt/gunyah/index.rst b/Documentation/virt/gunyah/index.rst new file mode 100644 index 000000000000..45adbbc311db --- /dev/null +++ b/Documentation/virt/gunyah/index.rst @@ -0,0 +1,113 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================= +Gunyah Hypervisor +================= + +.. toctree:: + :maxdepth: 1 + + message-queue + +Gunyah is a Type-1 hypervisor which is independent of any OS kernel, and runs in +a higher CPU privilege level. It does not depend on any lower-privileged operating system +for its core functionality. This increases its security and can support a much smaller +trusted computing base than a Type-2 hypervisor. + +Gunyah is an open source hypervisor. The source repo is available at +https://github.com/quic/gunyah-hypervisor. + +Gunyah provides these following features. + +- Scheduling: + + A scheduler for virtual CPUs (vCPUs) on physical CPUs enables time-sharing + of the CPUs. Gunyah supports two models of scheduling: + + 1. "Behind the back" scheduling in which Gunyah hypervisor schedules vCPUS on its own. + 2. "Proxy" scheduling in which a delegated VM can donate part of one of its vCPU slice + to another VM's vCPU via a hypercall. + +- Memory Management: + + APIs handling memory, abstracted as objects, limiting direct use of physical + addresses. Memory ownership and usage tracking of all memory under its control. + Memory partitioning between VMs is a fundamental security feature. + +- Interrupt Virtualization: + + Uses CPU hardware interrupt virtualization capabilities. Interrupts are handled + in the hypervisor and routed to the assigned VM. + +- Inter-VM Communication: + + There are several different mechanisms provided for communicating between VMs. + +- Virtual platform: + + Architectural devices such as interrupt controllers and CPU timers are directly provided + by the hypervisor as well as core virtual platform devices and system APIs such as ARM PSCI. + +- Device Virtualization: + + Para-virtualization of devices is supported using inter-VM communication. + +Architectures supported +======================= +AArch64 with a GIC + +Resources and Capabilities +========================== + +Some services or resources provided by the Gunyah hypervisor are described to a virtual machine by +capability IDs. For instance, inter-VM communication is performed with doorbells and message queues. +Gunyah allows access to manipulate that doorbell via the capability ID. These resources are +described in Linux as a struct gunyah_resource. + +High level management of these resources is performed by the resource manager VM. RM informs a +guest VM about resources it can access through either the device tree or via guest-initiated RPC. + +For each virtual machine, Gunyah maintains a table of resources which can be accessed by that VM. +An entry in this table is called a "capability" and VMs can only access resources via this +capability table. Hence, virtual Gunyah resources are referenced by a "capability IDs" and not +"resource IDs". If 2 VMs have access to the same resource, they might not be using the same +capability ID to access that resource since the capability tables are independent per VM. + +Resource Manager +================ + +The resource manager (RM) is a privileged application VM supporting the Gunyah Hypervisor. +It provides policy enforcement aspects of the virtualization system. The resource manager can +be treated as an extension of the Hypervisor but is separated to its own partition to ensure +that the hypervisor layer itself remains small and secure and to maintain a separation of policy +and mechanism in the platform. RM runs at arm64 NS-EL1 similar to other virtual machines. + +Communication with the resource manager from each guest VM happens with message-queue.rst. Details +about the specific messages can be found in drivers/virt/gunyah/rsc_mgr.c + +:: + + +-------+ +--------+ +--------+ + | RM | | VM_A | | VM_B | + +-.-.-.-+ +---.----+ +---.----+ + | | | | + +-.-.-----------.------------.----+ + | | \==========/ | | + | \========================/ | + | Gunyah | + +---------------------------------+ + +The source for the resource manager is available at https://github.com/quic/gunyah-resource-manager. + +The resource manager provides the following features: + +- VM lifecycle management: allocating a VM, starting VMs, destruction of VMs +- VM access control policy, including memory sharing and lending +- Interrupt routing configuration +- Forwarding of system-level events (e.g. VM shutdown) to owner VM + +When booting a virtual machine which uses a devicetree such as Linux, resource manager overlays a +/hypervisor node. This node can let Linux know it is running as a Gunyah guest VM, +how to communicate with resource manager, and basic description and capabilities of +this VM. See Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml for a description +of this node. diff --git a/Documentation/virt/gunyah/message-queue.rst b/Documentation/virt/gunyah/message-queue.rst new file mode 100644 index 000000000000..0667b3eb1ff9 --- /dev/null +++ b/Documentation/virt/gunyah/message-queue.rst @@ -0,0 +1,61 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Message Queues +============== +Message queue is a simple low-capacity IPC channel between two VMs. It is +intended for sending small control and configuration messages. Each message +queue is unidirectional, so a full-duplex IPC channel requires a pair of queues. + +Messages can be up to 240 bytes in length. Longer messages require a further +protocol on top of the message queue messages themselves. For instance, communication +with the resource manager adds a header field for sending longer messages via multiple +message fragments. + +The diagram below shows how message queue works. A typical configuration involves +2 message queues. Message queue 1 allows VM_A to send messages to VM_B. Message +queue 2 allows VM_B to send messages to VM_A. + +1. VM_A sends a message of up to 240 bytes in length. It raises a hypercall + with the message to inform the hypervisor to add the message to + message queue 1's queue. + +2. Gunyah raises the corresponding interrupt for VM_B (Rx vIRQ) when any of + these happens: + + a. gh_msgq_send has PUSH flag. Queue is immediately flushed. This is the typical case. + b. Explicility with gh_msgq_push command from VM_A. + c. Message queue has reached a threshold depth. + +3. VM_B calls gh_msgq_recv and Gunyah copies message to requested buffer. + +4. Gunyah buffers messages in the queue. If the queue became full when VM_A added a message, + the return values for gh_msgq_send() include a flag that indicates the queue is full. + Once VM_B receives the message and, thus, there is space in the queue, Gunyah + will raise the Tx vIRQ on VM_A to indicate it can continue sending messages. + +For VM_B to send a message to VM_A, the process is identical, except that hypercalls +reference message queue 2's capability ID. Each message queue has its own independent +vIRQ: two TX message queues will have two vIRQs (and two capability IDs). + +:: + + +---------------+ +-----------------+ +---------------+ + | VM_A | |Gunyah hypervisor| | VM_B | + | | | | | | + | | | | | | + | | Tx | | | | + | |-------->| | Rx vIRQ | | + |gh_msgq_send() | Tx vIRQ |Message queue 1 |-------->|gh_msgq_recv() | + | |<------- | | | | + | | | | | | + | Message Queue | | | | Message Queue | + | driver | | | | driver | + | | | | | | + | | | | | | + | | | | Tx | | + | | Rx vIRQ | |<--------| | + |gh_msgq_recv() |<--------|Message queue 2 | Tx vIRQ |gh_msgq_send() | + | | | |-------->| | + | | | | | | + | | | | | | + +---------------+ +-----------------+ +---------------+ diff --git a/Documentation/virt/index.rst b/Documentation/virt/index.rst index 56e003ff28ff..e92a9cf4b99d 100644 --- a/Documentation/virt/index.rst +++ b/Documentation/virt/index.rst @@ -16,6 +16,7 @@ Linux Virtualization Support coco/sev-guest coco/tdx-guest hyperv/index + gunyah/index .. only:: html and subproject diff --git a/MAINTAINERS b/MAINTAINERS index 42fc47c6edfd..14fcfc7f91df 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9041,6 +9041,13 @@ L: linux-efi@vger.kernel.org S: Maintained F: block/partitions/efi.* +GUNYAH HYPERVISOR DRIVER +M: Elliot Berman +M: Murali Nalajala +L: linux-arm-msm@vger.kernel.org +S: Supported +F: Documentation/virt/gunyah/ + HABANALABS PCI DRIVER M: Oded Gabbay S: Supported From patchwork Fri Jan 20 22:46:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 644684 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7386C38159 for ; Fri, 20 Jan 2023 22:47:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229875AbjATWr5 (ORCPT ); Fri, 20 Jan 2023 17:47:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55382 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229744AbjATWr4 (ORCPT ); Fri, 20 Jan 2023 17:47:56 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1BB2252BA; Fri, 20 Jan 2023 14:47:45 -0800 (PST) Received: from pps.filterd (m0279868.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KLpBxR001787; Fri, 20 Jan 2023 22:47:33 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=Ij+kymD06vQBfS3DsyzOfY1gb7Js0y5d3S6gxBAzk88=; b=AWJUkSum+vZ6ptmgZPnIIlS7VVcQ6Z9/LLSHfdgvIefD3hhjo2XkK5J9bf8oR/qe+hIY ye87kWD8cifgm8BDCnbe9tj80SZNoTjSzrm/VbgDsBFOLNKi1cGsadF/BxnmhwbqDSAV tDY8L5cYzeM/icALnuj4wx5RdKY/w+K6OTXJ6z+xbXyx/7UiyZHLf+EEBgqaALqrRAyG RtqyKUEJt4u5ePlVDFyLa3zE57RBcBWdyfN7UxpoD5+iQ9sJYPDvLzxMsB10Wz7x+TOb 0PyJkIIl2yFGl9c4Lt6+mPXDdIQLNVonXGkYGUgUwBxY2Xgh79k7kABbfNc/aiRHDAHC aQ== Received: from nasanppmta02.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n75w3kmwf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:33 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA02.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMlV6C027129 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:31 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:30 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala , Rob Herring , "Krzysztof Kozlowski" CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Will Deacon , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , , "Rob Herring" Subject: [PATCH v9 02/27] dt-bindings: Add binding for gunyah hypervisor Date: Fri, 20 Jan 2023 14:46:01 -0800 Message-ID: <20230120224627.4053418-3-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: Pmo8o9kGMJDtmMZhWhQiic-rRd41HPnv X-Proofpoint-GUID: Pmo8o9kGMJDtmMZhWhQiic-rRd41HPnv X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 impostorscore=0 lowpriorityscore=0 adultscore=0 clxscore=1011 mlxscore=0 spamscore=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org When Linux is booted as a guest under the Gunyah hypervisor, the Gunyah Resource Manager applies a devicetree overlay describing the virtual platform configuration of the guest VM, such as the message queue capability IDs for communicating with the Resource Manager. This information is not otherwise discoverable by a VM: the Gunyah hypervisor core does not provide a direct interface to discover capability IDs nor a way to communicate with RM without having already known the corresponding message queue capability ID. Add the DT bindings that Gunyah adheres for the hypervisor node and message queues. Reviewed-by: Rob Herring Signed-off-by: Elliot Berman --- .../bindings/firmware/gunyah-hypervisor.yaml | 82 +++++++++++++++++++ MAINTAINERS | 1 + 2 files changed, 83 insertions(+) create mode 100644 Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml diff --git a/Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml b/Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml new file mode 100644 index 000000000000..9fd1d254b156 --- /dev/null +++ b/Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml @@ -0,0 +1,82 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/firmware/gunyah-hypervisor.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Gunyah Hypervisor + +maintainers: + - Murali Nalajala + - Elliot Berman + +description: |+ + Gunyah virtual machines use this information to determine the capability IDs + of the message queues used to communicate with the Gunyah Resource Manager. + See also: https://github.com/quic/gunyah-resource-manager/blob/develop/src/vm_creation/dto_construct.c + +properties: + compatible: + const: gunyah-hypervisor + + "#address-cells": + description: Number of cells needed to represent 64-bit capability IDs. + const: 2 + + "#size-cells": + description: must be 0, because capability IDs are not memory address + ranges and do not have a size. + const: 0 + +patternProperties: + "^gunyah-resource-mgr(@.*)?": + type: object + description: + Resource Manager node which is required to communicate to Resource + Manager VM using Gunyah Message Queues. + + properties: + compatible: + const: gunyah-resource-manager + + reg: + items: + - description: Gunyah capability ID of the TX message queue + - description: Gunyah capability ID of the RX message queue + + interrupts: + items: + - description: Interrupt for the TX message queue + - description: Interrupt for the RX message queue + + additionalProperties: false + + required: + - compatible + - reg + - interrupts + +additionalProperties: false + +required: + - compatible + - "#address-cells" + - "#size-cells" + +examples: + - | + #include + + hypervisor { + #address-cells = <2>; + #size-cells = <0>; + compatible = "gunyah-hypervisor"; + + gunyah-resource-mgr@0 { + compatible = "gunyah-resource-manager"; + interrupts = , /* TX full IRQ */ + ; /* RX empty IRQ */ + reg = <0x00000000 0x00000000>, <0x00000000 0x00000001>; + /* TX, RX cap ids */ + }; + }; diff --git a/MAINTAINERS b/MAINTAINERS index 14fcfc7f91df..d9205cb9e988 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9046,6 +9046,7 @@ M: Elliot Berman M: Murali Nalajala L: linux-arm-msm@vger.kernel.org S: Supported +F: Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml F: Documentation/virt/gunyah/ HABANALABS PCI DRIVER From patchwork Fri Jan 20 22:46:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 644683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E6A3C27C76 for ; Fri, 20 Jan 2023 22:48:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229934AbjATWsH (ORCPT ); Fri, 20 Jan 2023 17:48:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55526 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229744AbjATWsA (ORCPT ); Fri, 20 Jan 2023 17:48:00 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF56013525; Fri, 20 Jan 2023 14:47:47 -0800 (PST) Received: from pps.filterd (m0279871.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KM03cg024187; Fri, 20 Jan 2023 22:47:34 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=kyuLsPAltQCv9R8fMJCTXGUe/Z7HvJ7ZAS2S9ubamac=; b=VbYEZEP5SGIKWcsXW7QlWAHR9E3qTX47HYJOAg42uaQiuj6GVwc6lh3W55eDVmeND9FE 8Hn9V7Uc4oRxDoEWau3BTiijD5+S94SheoYP1XcOZtl4XV89aL7+GbYZ5fXkGVJlEkcU nLE69+G5rzlC8LrmSnqfVNeAGeEZCveki2xoMBndoQ1xFg/PTTvJ2xkmHbVmcrLfkf06 OGS0chNgEB2N2afaBH91IgfaxCBrgaGaScl+VfmqVoOAvUIwolQjPfUdmwxH0TSgsq2b GD3DZudKUqRRbwKoWXxX9n954iBE7cqlFc7BgoPtlaMnTB8wSdJrURZRS3XPMNX+s+RW hg== Received: from nasanppmta03.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7c28jrek-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:34 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA03.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMlX9T017493 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:33 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:32 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Will Deacon , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 03/27] gunyah: Common types and error codes for Gunyah hypercalls Date: Fri, 20 Jan 2023 14:46:02 -0800 Message-ID: <20230120224627.4053418-4-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: mHdx1hBxo0GmSzimkzOTJNpb6eCL3Tqh X-Proofpoint-ORIG-GUID: mHdx1hBxo0GmSzimkzOTJNpb6eCL3Tqh X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 spamscore=0 priorityscore=1501 adultscore=0 lowpriorityscore=0 mlxlogscore=810 clxscore=1015 mlxscore=0 suspectscore=0 malwarescore=0 bulkscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Add architecture-independent standard error codes, types, and macros for Gunyah hypercalls. Reviewed-by: Dmitry Baryshkov Signed-off-by: Elliot Berman --- MAINTAINERS | 1 + include/linux/gunyah.h | 74 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+) create mode 100644 include/linux/gunyah.h diff --git a/MAINTAINERS b/MAINTAINERS index d9205cb9e988..f1e07e39b2f5 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9048,6 +9048,7 @@ L: linux-arm-msm@vger.kernel.org S: Supported F: Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml F: Documentation/virt/gunyah/ +F: include/linux/gunyah.h HABANALABS PCI DRIVER M: Oded Gabbay diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h new file mode 100644 index 000000000000..985c6086348e --- /dev/null +++ b/include/linux/gunyah.h @@ -0,0 +1,74 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#ifndef _LINUX_GUNYAH_H +#define _LINUX_GUNYAH_H + +#include +#include + +/* Common Gunyah macros */ +#define GH_CAPID_INVAL U64_MAX +#define GH_VMID_ROOT_VM 0xff + +#define GH_ERROR_OK 0 + +#define GH_ERROR_UNIMPLEMENTED -1 +#define GH_ERROR_RETRY -2 + +#define GH_ERROR_ARG_INVAL 1 +#define GH_ERROR_ARG_SIZE 2 +#define GH_ERROR_ARG_ALIGN 3 + +#define GH_ERROR_NOMEM 10 + +#define GH_ERROR_ADDR_OVFL 20 +#define GH_ERROR_ADDR_UNFL 21 +#define GH_ERROR_ADDR_INVAL 22 + +#define GH_ERROR_DENIED 30 +#define GH_ERROR_BUSY 31 +#define GH_ERROR_IDLE 32 + +#define GH_ERROR_IRQ_BOUND 40 +#define GH_ERROR_IRQ_UNBOUND 41 + +#define GH_ERROR_CSPACE_CAP_NULL 50 +#define GH_ERROR_CSPACE_CAP_REVOKED 51 +#define GH_ERROR_CSPACE_WRONG_OBJ_TYPE 52 +#define GH_ERROR_CSPACE_INSUF_RIGHTS 53 +#define GH_ERROR_CSPACE_FULL 54 + +#define GH_ERROR_MSGQUEUE_EMPTY 60 +#define GH_ERROR_MSGQUEUE_FULL 61 + +static inline int gh_remap_error(int gh_error) +{ + switch (gh_error) { + case GH_ERROR_OK: + return 0; + case GH_ERROR_NOMEM: + return -ENOMEM; + case GH_ERROR_DENIED: + case GH_ERROR_CSPACE_CAP_NULL: + case GH_ERROR_CSPACE_CAP_REVOKED: + case GH_ERROR_CSPACE_WRONG_OBJ_TYPE: + case GH_ERROR_CSPACE_INSUF_RIGHTS: + case GH_ERROR_CSPACE_FULL: + return -EACCES; + case GH_ERROR_BUSY: + case GH_ERROR_IDLE: + return -EBUSY; + case GH_ERROR_IRQ_BOUND: + case GH_ERROR_IRQ_UNBOUND: + case GH_ERROR_MSGQUEUE_FULL: + case GH_ERROR_MSGQUEUE_EMPTY: + return -EPERM; + default: + return -EINVAL; + } +} + +#endif From patchwork Fri Jan 20 22:46:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 645108 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6106C5320B for ; Fri, 20 Jan 2023 22:48:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229697AbjATWsI (ORCPT ); Fri, 20 Jan 2023 17:48:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55586 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229679AbjATWsC (ORCPT ); Fri, 20 Jan 2023 17:48:02 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B22423C66; Fri, 20 Jan 2023 14:47:49 -0800 (PST) Received: from pps.filterd (m0279871.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KLa0MV023841; Fri, 20 Jan 2023 22:47:36 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=OO98CIBdly0mmgA1GwNwOM2BQ1GVB1wy57q24H0LVUc=; b=GGH2cd6VDIs5iA0hR0YGeLT8QlWAukaBlSJcbkdX/peWPS2jt/qfKUUptYs8MGsBn8mH brkzzn58k0jvNepEytuOgXQy1mkq4i69FONKZN/nA34wr/DulIcExKyw0h5dFWxWn9Gu 0LkMqvFKvYHe1omTslChoMsLr/AvF/WdpEaLsYlgW3d0noq205EdMFlybL/UzeCAnh5b EFO9tsqSQMlb72/N8AeBVzwB3ObvcbrBOm6Kot2czXdVRZ7zIrTXBeBvwQkFbwMGHw8z ab3TbwgqcjRpkL5FKODyWCi0rZD9diK7vXOWiKl5oQ+yduxh9HWuaKOSOYQfOUmPi9yC YQ== Received: from nasanppmta04.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7c28jren-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:35 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA04.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMlZkq022200 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:35 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:34 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Catalin Marinas , Will Deacon , Elliot Berman , Murali Nalajala CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 04/27] virt: gunyah: Add hypercalls to identify Gunyah Date: Fri, 20 Jan 2023 14:46:03 -0800 Message-ID: <20230120224627.4053418-5-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: pWBNmX3rT8gpCMtAwMmNEaimMZ-Y1FKY X-Proofpoint-ORIG-GUID: pWBNmX3rT8gpCMtAwMmNEaimMZ-Y1FKY X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 spamscore=0 priorityscore=1501 adultscore=0 lowpriorityscore=0 mlxlogscore=999 clxscore=1015 mlxscore=0 suspectscore=0 malwarescore=0 bulkscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Add hypercalls to identify when Linux is running a virtual machine under Gunyah. There are two calls to help identify Gunyah: 1. gh_hypercall_get_uid() returns a UID when running under a Gunyah hypervisor. 2. gh_hypercall_hyp_identify() returns build information and a set of feature flags that are supported by Gunyah. Signed-off-by: Elliot Berman --- MAINTAINERS | 2 + arch/arm64/Kbuild | 1 + arch/arm64/gunyah/Makefile | 3 ++ arch/arm64/gunyah/gunyah_hypercall.c | 75 ++++++++++++++++++++++++++++ drivers/virt/Kconfig | 2 + drivers/virt/gunyah/Kconfig | 14 ++++++ include/linux/gunyah.h | 33 ++++++++++++ 7 files changed, 130 insertions(+) create mode 100644 arch/arm64/gunyah/Makefile create mode 100644 arch/arm64/gunyah/gunyah_hypercall.c create mode 100644 drivers/virt/gunyah/Kconfig diff --git a/MAINTAINERS b/MAINTAINERS index f1e07e39b2f5..fe19e71efc6d 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9048,6 +9048,8 @@ L: linux-arm-msm@vger.kernel.org S: Supported F: Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml F: Documentation/virt/gunyah/ +F: arch/arm64/gunyah/ +F: drivers/virt/gunyah/ F: include/linux/gunyah.h HABANALABS PCI DRIVER diff --git a/arch/arm64/Kbuild b/arch/arm64/Kbuild index 5bfbf7d79c99..e4847ba0e3c9 100644 --- a/arch/arm64/Kbuild +++ b/arch/arm64/Kbuild @@ -3,6 +3,7 @@ obj-y += kernel/ mm/ net/ obj-$(CONFIG_KVM) += kvm/ obj-$(CONFIG_XEN) += xen/ obj-$(subst m,y,$(CONFIG_HYPERV)) += hyperv/ +obj-$(CONFIG_GUNYAH) += gunyah/ obj-$(CONFIG_CRYPTO) += crypto/ # for cleaning diff --git a/arch/arm64/gunyah/Makefile b/arch/arm64/gunyah/Makefile new file mode 100644 index 000000000000..84f1e38cafb1 --- /dev/null +++ b/arch/arm64/gunyah/Makefile @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_GUNYAH) += gunyah_hypercall.o diff --git a/arch/arm64/gunyah/gunyah_hypercall.c b/arch/arm64/gunyah/gunyah_hypercall.c new file mode 100644 index 000000000000..ffed4b71641f --- /dev/null +++ b/arch/arm64/gunyah/gunyah_hypercall.c @@ -0,0 +1,75 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#include +#include +#include + +static const uint32_t gunyah_known_uuids[][4] = { + {0x19bd54bd, 0x0b37571b, 0x946f609b, 0x54539de6}, /* QC_HYP (Qualcomm's build) */ + {0x673d5f14, 0x9265ce36, 0xa4535fdb, 0xc1d58fcd}, /* GUNYAH (open source build) */ +}; + +#define GH_HYPERCALL_HYP_IDENTIFY GH_HYPERCALL(0x0000) + +/** + * gh_hypercall_get_uid() - Returns a UID when running under a Gunyah hypervisor + * @uid: An array of 4 u32's (u32 uid[4];) + * + * Caller should compare the resulting UID to a list of known Gunyah UIDs to + * confirm that Linux is running as a guest of Gunyah. + */ +void gh_hypercall_get_uid(u32 uid[4]) +{ + struct arm_smccc_res res; + u32 uid[4]; + int i; + + arm_smccc_1_1_hvc(ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID, &res); + + uid[0] = lower_32_bits(res.a0); + uid[1] = lower_32_bits(res.a1); + uid[2] = lower_32_bits(res.a2); + uid[3] = lower_32_bits(res.a3); + + for (i = 0; i < ARRAY_SIZE(gunyah_known_uuids); i++) + if (!memcmp(uid, gunyah_known_uuids[i], sizeof(uid))) + break; + + return i != ARRAY_SIZE(gunyah_known_uuids); +} +EXPORT_SYMBOL_GPL(arch_is_gunyah_guest); + +#define GH_HYPERCALL(fn) ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, ARM_SMCCC_SMC_64, \ + ARM_SMCCC_OWNER_VENDOR_HYP, \ + fn) + +#define GH_HYPERCALL_HYP_IDENTIFY GH_HYPERCALL(0x8000) +#define GH_HYPERCALL_DBL_SEND GH_HYPERCALL(0x8012) +#define GH_HYPERCALL_DBL_SET_MASK GH_HYPERCALL(0x8015) +#define GH_HYPERCALL_MSGQ_SEND GH_HYPERCALL(0x801B) +#define GH_HYPERCALL_MSGQ_RECV GH_HYPERCALL(0x801C) +#define GH_HYPERCALL_VCPU_RUN GH_HYPERCALL(0x8065) + +/** + * gh_hypercall_hyp_identify() - Returns build information and feature flags + * supported by Gunyah. + * @hyp_identity: filled by the hypercall with the API info and feature flags. + */ +void gh_hypercall_hyp_identify(struct gh_hypercall_hyp_identify_resp *hyp_identity) +{ + struct arm_smccc_res res; + + arm_smccc_1_1_hvc(GH_HYPERCALL_HYP_IDENTIFY, &res); + + hyp_identity->api_info = res.a0; + hyp_identity->flags[0] = res.a1; + hyp_identity->flags[1] = res.a2; + hyp_identity->flags[2] = res.a3; +} +EXPORT_SYMBOL_GPL(gh_hypercall_hyp_identify); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("Gunyah Hypervisor Hypercalls"); diff --git a/drivers/virt/Kconfig b/drivers/virt/Kconfig index f79ab13a5c28..85bd6626ffc9 100644 --- a/drivers/virt/Kconfig +++ b/drivers/virt/Kconfig @@ -54,4 +54,6 @@ source "drivers/virt/coco/sev-guest/Kconfig" source "drivers/virt/coco/tdx-guest/Kconfig" +source "drivers/virt/gunyah/Kconfig" + endif diff --git a/drivers/virt/gunyah/Kconfig b/drivers/virt/gunyah/Kconfig new file mode 100644 index 000000000000..71c38845bcea --- /dev/null +++ b/drivers/virt/gunyah/Kconfig @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: GPL-2.0-only + +config GUNYAH + tristate "Gunyah Virtualization drivers" + depends on ARM64 + depends on MAILBOX + select GUNYAH_PLATFORM_HOOKS + help + The Gunyah drivers are the helper interfaces that run in a guest VM + such as basic inter-VM IPC and signaling mechanisms, and higher level + services such as memory/device sharing, IRQ sharing, and so on. + + Say Y/M here to enable the drivers needed to interact in a Gunyah + virtual environment. diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h index 985c6086348e..6724d1264d58 100644 --- a/include/linux/gunyah.h +++ b/include/linux/gunyah.h @@ -6,8 +6,10 @@ #ifndef _LINUX_GUNYAH_H #define _LINUX_GUNYAH_H +#include #include #include +#include /* Common Gunyah macros */ #define GH_CAPID_INVAL U64_MAX @@ -71,4 +73,35 @@ static inline int gh_remap_error(int gh_error) } } +enum gh_api_feature { + GH_API_FEATURE_DOORBELL, + GH_API_FEATURE_MSGQUEUE, + GH_API_FEATURE_VCPU, + GH_API_FEATURE_MEMEXTENT, +}; + +bool arch_is_gunyah_guest(void); + +u16 gh_api_version(void); +bool gh_api_has_feature(enum gh_api_feature feature); + +#define GUNYAH_API_V1 1 + +#define GH_API_INFO_API_VERSION_MASK GENMASK_ULL(13, 0) +#define GH_API_INFO_BIG_ENDIAN BIT_ULL(14) +#define GH_API_INFO_IS_64BIT BIT_ULL(15) +#define GH_API_INFO_VARIANT_MASK GENMASK_ULL(63, 56) + +#define GH_IDENTIFY_DOORBELL BIT_ULL(1) +#define GH_IDENTIFY_MSGQUEUE BIT_ULL(2) +#define GH_IDENTIFY_VCPU BIT_ULL(5) +#define GH_IDENTIFY_MEMEXTENT BIT_ULL(6) + +struct gh_hypercall_hyp_identify_resp { + u64 api_info; + u64 flags[3]; +}; + +void gh_hypercall_hyp_identify(struct gh_hypercall_hyp_identify_resp *hyp_identity); + #endif From patchwork Fri Jan 20 22:46:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 645107 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5954AC27C76 for ; Fri, 20 Jan 2023 22:48:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229981AbjATWsK (ORCPT ); Fri, 20 Jan 2023 17:48:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55552 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229893AbjATWsH (ORCPT ); Fri, 20 Jan 2023 17:48:07 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B885329161; Fri, 20 Jan 2023 14:47:49 -0800 (PST) Received: from pps.filterd (m0279873.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KMeJoO014550; Fri, 20 Jan 2023 22:47:37 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=k1t08PAttr9GVLaiQO+z0sFDGMbPRc3GjKBx/B+SyEA=; b=PFO1TmgUcx+Feq8KmwpEYvjgjSoe8TJA8lH+EzFDLw/iGOL6BVWH9NurOU39q1yEJtZS sUZ1zNZa5QWTdblL4NpEF0veusRQa21ulqQx14Ib1RkO/NwnDR+R+A4MG9FHsNkF9CkU s1obgGp/bTmL86RbtfC7/qamMqjJ3+iNKSH0fs7r7sl4n1XyovGLas/2SSvxa5c2s79Y Bgqdz6bsBdzzrQtwRgteX2MnnpzsXMfgeXJOHbvufuyJVBDFf2ybkwO7xMKfdLL+fi9Q g/07vUuZGWt6G5ICIovRsL0U8qm5V/Rx2Hud8MVfdhwkS+MH3cmGak+iXGskFWE8dBBb qw== Received: from nasanppmta05.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7593uqxp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:37 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA05.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMla2X030153 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:36 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:35 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Will Deacon , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 05/27] virt: gunyah: Identify hypervisor version Date: Fri, 20 Jan 2023 14:46:04 -0800 Message-ID: <20230120224627.4053418-6-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: 6Upn7tBCYYsP4iIn8eZYgneASuW-c2Kd X-Proofpoint-ORIG-GUID: 6Upn7tBCYYsP4iIn8eZYgneASuW-c2Kd X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 mlxlogscore=999 spamscore=0 suspectscore=0 mlxscore=0 adultscore=0 clxscore=1015 impostorscore=0 priorityscore=1501 bulkscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Export the version of Gunyah which is reported via the hyp_identify hypercall. Increments of the major API version indicate possibly backwards incompatible changes. Export the hypervisor identity so that Gunyah drivers can act according to the major API version. Signed-off-by: Elliot Berman --- drivers/virt/Makefile | 1 + drivers/virt/gunyah/Makefile | 3 ++ drivers/virt/gunyah/gunyah.c | 54 ++++++++++++++++++++++++++++++++++++ 3 files changed, 58 insertions(+) create mode 100644 drivers/virt/gunyah/Makefile create mode 100644 drivers/virt/gunyah/gunyah.c diff --git a/drivers/virt/Makefile b/drivers/virt/Makefile index e9aa6fc96fab..a5817e2d7d71 100644 --- a/drivers/virt/Makefile +++ b/drivers/virt/Makefile @@ -12,3 +12,4 @@ obj-$(CONFIG_ACRN_HSM) += acrn/ obj-$(CONFIG_EFI_SECRET) += coco/efi_secret/ obj-$(CONFIG_SEV_GUEST) += coco/sev-guest/ obj-$(CONFIG_INTEL_TDX_GUEST) += coco/tdx-guest/ +obj-y += gunyah/ diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile new file mode 100644 index 000000000000..34f32110faf9 --- /dev/null +++ b/drivers/virt/gunyah/Makefile @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_GUNYAH) += gunyah.o diff --git a/drivers/virt/gunyah/gunyah.c b/drivers/virt/gunyah/gunyah.c new file mode 100644 index 000000000000..776e83c6920d --- /dev/null +++ b/drivers/virt/gunyah/gunyah.c @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#define pr_fmt(fmt) "gunyah: " fmt + +#include +#include +#include +#include + +static struct gh_hypercall_hyp_identify_resp gunyah_api; + +u16 gh_api_version(void) +{ + return FIELD_GET(GH_API_INFO_API_VERSION_MASK, gunyah_api.api_info); +} +EXPORT_SYMBOL_GPL(gh_api_version); + +bool gh_api_has_feature(enum gh_api_feature feature) +{ + switch (feature) { + case GH_API_FEATURE_DOORBELL: + return !!(gunyah_api.flags[0] & GH_IDENTIFY_DOORBELL); + case GH_API_FEATURE_MSGQUEUE: + return !!(gunyah_api.flags[0] & GH_IDENTIFY_MSGQUEUE); + case GH_API_FEATURE_VCPU: + return !!(gunyah_api.flags[0] & GH_IDENTIFY_VCPU); + case GH_API_FEATURE_MEMEXTENT: + return !!(gunyah_api.flags[0] & GH_IDENTIFY_MEMEXTENT); + default: + return false; + } +} +EXPORT_SYMBOL_GPL(gh_api_has_feature); + +static int __init gunyah_init(void) +{ + if (!arch_is_gunyah_guest()) + return -ENODEV; + + gh_hypercall_hyp_identify(&gunyah_api); + + pr_info("Running under Gunyah hypervisor %llx/v%u\n", + FIELD_GET(GH_API_INFO_VARIANT_MASK, gunyah_api.api_info), + gh_api_version()); + + return 0; +} +arch_initcall(gunyah_init); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("Gunyah Hypervisor Driver"); From patchwork Fri Jan 20 22:46:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 644681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B6A7C54EB4 for ; Fri, 20 Jan 2023 22:48:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229988AbjATWsL (ORCPT ); Fri, 20 Jan 2023 17:48:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55646 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229929AbjATWsG (ORCPT ); Fri, 20 Jan 2023 17:48:06 -0500 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8509F37B52; Fri, 20 Jan 2023 14:47:50 -0800 (PST) Received: from pps.filterd (m0279865.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KMItSc005715; Fri, 20 Jan 2023 22:47:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=iUyidqzRy3Qp4YlkdAFiUxKftd/XY8QtYwo4Hxv3e4k=; b=FpFgOPArix40pjQMz7VdbXpjnZ9vTHAnGkco0yCKbvCpfZGQZhMby4/z04ejDuV6TCVc boTFttJmlVxaOJE+SlcKcrRHvhtjpp15P2n8lz3p89E1MXMEOuDHIaJfarqEcxqrJ2WD MpTFQxAkXvXO+1epHndMADtSCRePOh/uEC8npuvjFXNsW9ABB5IuokNipD41S8AdS+Vo BSl9azuyEVLTHuoVGMNQAKJmVxy4E3TQpupw24GBnTtz1izs+DfRWRQ5ttpZV1UqE/Vy Hy/QLqbqWVR1Op0Rwy0UvIxWcy5TsXBCIa2YDuo/I6/7i22gqcz8K8Tj5OUpnfiCTRYv nw== Received: from nasanppmta04.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n6r59ddgg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:38 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA04.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMlbM7022220 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:37 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:36 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Jassi Brar , Sudeep Holla CC: Elliot Berman , Murali Nalajala , Trilok Soni , "Srivatsa Vaddagiri" , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , Greg Kroah-Hartman , Rob Herring , "Krzysztof Kozlowski" , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Will Deacon , Marc Zyngier , , , , , , Subject: [PATCH v9 06/27] mailbox: Allow direct registration to a channel Date: Fri, 20 Jan 2023 14:46:05 -0800 Message-ID: <20230120224627.4053418-7-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: GEAqWdmvPK5V_nnmswtn9mA9pTfM5BJ2 X-Proofpoint-GUID: GEAqWdmvPK5V_nnmswtn9mA9pTfM5BJ2 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 clxscore=1015 mlxscore=0 mlxlogscore=999 malwarescore=0 priorityscore=1501 bulkscore=0 suspectscore=0 phishscore=0 adultscore=0 spamscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Support virtual mailbox controllers and clients which are not platform devices or come from the devicetree by allowing them to match client to channel via some other mechanism. Signed-off-by: Elliot Berman --- drivers/mailbox/mailbox.c | 96 ++++++++++++++++++++++++---------- drivers/mailbox/omap-mailbox.c | 19 ++----- drivers/mailbox/pcc.c | 18 ++----- include/linux/mailbox_client.h | 1 + 4 files changed, 76 insertions(+), 58 deletions(-) diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c index 4229b9b5da98..adf36c05fa43 100644 --- a/drivers/mailbox/mailbox.c +++ b/drivers/mailbox/mailbox.c @@ -317,6 +317,71 @@ int mbox_flush(struct mbox_chan *chan, unsigned long timeout) } EXPORT_SYMBOL_GPL(mbox_flush); +static int __mbox_bind_client(struct mbox_chan *chan, struct mbox_client *cl) +{ + struct device *dev = cl->dev; + unsigned long flags; + int ret; + + if (chan->cl || !try_module_get(chan->mbox->dev->driver->owner)) { + dev_dbg(dev, "%s: mailbox not free\n", __func__); + return -EBUSY; + } + + spin_lock_irqsave(&chan->lock, flags); + chan->msg_free = 0; + chan->msg_count = 0; + chan->active_req = NULL; + chan->cl = cl; + init_completion(&chan->tx_complete); + + if (chan->txdone_method == TXDONE_BY_POLL && cl->knows_txdone) + chan->txdone_method = TXDONE_BY_ACK; + + spin_unlock_irqrestore(&chan->lock, flags); + + if (chan->mbox->ops->startup) { + ret = chan->mbox->ops->startup(chan); + + if (ret) { + dev_err(dev, "Unable to startup the chan (%d)\n", ret); + mbox_free_channel(chan); + return ret; + } + } + + return 0; +} + +/** + * mbox_bind_client - Request a mailbox channel. + * @chan: The mailbox channel to bind the client to. + * @cl: Identity of the client requesting the channel. + * + * The Client specifies its requirements and capabilities while asking for + * a mailbox channel. It can't be called from atomic context. + * The channel is exclusively allocated and can't be used by another + * client before the owner calls mbox_free_channel. + * After assignment, any packet received on this channel will be + * handed over to the client via the 'rx_callback'. + * The framework holds reference to the client, so the mbox_client + * structure shouldn't be modified until the mbox_free_channel returns. + * + * Return: 0 if the channel was assigned to the client successfully. + * <0 for request failure. + */ +int mbox_bind_client(struct mbox_chan *chan, struct mbox_client *cl) +{ + int ret; + + mutex_lock(&con_mutex); + ret = __mbox_bind_client(chan, cl); + mutex_unlock(&con_mutex); + + return ret; +} +EXPORT_SYMBOL_GPL(mbox_bind_client); + /** * mbox_request_channel - Request a mailbox channel. * @cl: Identity of the client requesting the channel. @@ -340,7 +405,6 @@ struct mbox_chan *mbox_request_channel(struct mbox_client *cl, int index) struct mbox_controller *mbox; struct of_phandle_args spec; struct mbox_chan *chan; - unsigned long flags; int ret; if (!dev || !dev->of_node) { @@ -372,33 +436,9 @@ struct mbox_chan *mbox_request_channel(struct mbox_client *cl, int index) return chan; } - if (chan->cl || !try_module_get(mbox->dev->driver->owner)) { - dev_dbg(dev, "%s: mailbox not free\n", __func__); - mutex_unlock(&con_mutex); - return ERR_PTR(-EBUSY); - } - - spin_lock_irqsave(&chan->lock, flags); - chan->msg_free = 0; - chan->msg_count = 0; - chan->active_req = NULL; - chan->cl = cl; - init_completion(&chan->tx_complete); - - if (chan->txdone_method == TXDONE_BY_POLL && cl->knows_txdone) - chan->txdone_method = TXDONE_BY_ACK; - - spin_unlock_irqrestore(&chan->lock, flags); - - if (chan->mbox->ops->startup) { - ret = chan->mbox->ops->startup(chan); - - if (ret) { - dev_err(dev, "Unable to startup the chan (%d)\n", ret); - mbox_free_channel(chan); - chan = ERR_PTR(ret); - } - } + ret = __mbox_bind_client(chan, cl); + if (ret) + chan = ERR_PTR(ret); mutex_unlock(&con_mutex); return chan; diff --git a/drivers/mailbox/omap-mailbox.c b/drivers/mailbox/omap-mailbox.c index 098c82d87137..8d2498fec03a 100644 --- a/drivers/mailbox/omap-mailbox.c +++ b/drivers/mailbox/omap-mailbox.c @@ -418,7 +418,6 @@ struct mbox_chan *omap_mbox_request_channel(struct mbox_client *cl, struct omap_mbox *mbox = NULL; struct omap_mbox_device *mdev; struct mbox_chan *chan; - unsigned long flags; int ret; if (!dev) @@ -441,21 +440,9 @@ struct mbox_chan *omap_mbox_request_channel(struct mbox_client *cl, if (!mbox || !mbox->chan) return ERR_PTR(-ENOENT); - chan = mbox->chan; - spin_lock_irqsave(&chan->lock, flags); - chan->msg_free = 0; - chan->msg_count = 0; - chan->active_req = NULL; - chan->cl = cl; - init_completion(&chan->tx_complete); - spin_unlock_irqrestore(&chan->lock, flags); - - ret = chan->mbox->ops->startup(chan); - if (ret) { - pr_err("Unable to startup the chan (%d)\n", ret); - mbox_free_channel(chan); - chan = ERR_PTR(ret); - } + ret = mbox_bind_client(mbox->chan, cl); + if (ret) + return ERR_PTR(ret); return chan; } diff --git a/drivers/mailbox/pcc.c b/drivers/mailbox/pcc.c index 105d46c9801b..3d80c3ac0e7d 100644 --- a/drivers/mailbox/pcc.c +++ b/drivers/mailbox/pcc.c @@ -283,7 +283,7 @@ pcc_mbox_request_channel(struct mbox_client *cl, int subspace_id) struct pcc_chan_info *pchan; struct mbox_chan *chan; struct device *dev; - unsigned long flags; + int rc; if (subspace_id < 0 || subspace_id >= pcc_chan_count) return ERR_PTR(-ENOENT); @@ -296,21 +296,11 @@ pcc_mbox_request_channel(struct mbox_client *cl, int subspace_id) } dev = chan->mbox->dev; - spin_lock_irqsave(&chan->lock, flags); - chan->msg_free = 0; - chan->msg_count = 0; - chan->active_req = NULL; - chan->cl = cl; - init_completion(&chan->tx_complete); - - if (chan->txdone_method == TXDONE_BY_POLL && cl->knows_txdone) - chan->txdone_method = TXDONE_BY_ACK; - - spin_unlock_irqrestore(&chan->lock, flags); + rc = mbox_bind_client(chan, cl); + if (rc) + return ERR_PTR(rc); if (pchan->plat_irq > 0) { - int rc; - rc = devm_request_irq(dev, pchan->plat_irq, pcc_mbox_irq, 0, MBOX_IRQ_NAME, chan); if (unlikely(rc)) { diff --git a/include/linux/mailbox_client.h b/include/linux/mailbox_client.h index 65229a45590f..734694912ef7 100644 --- a/include/linux/mailbox_client.h +++ b/include/linux/mailbox_client.h @@ -37,6 +37,7 @@ struct mbox_client { void (*tx_done)(struct mbox_client *cl, void *mssg, int r); }; +int mbox_bind_client(struct mbox_chan *chan, struct mbox_client *cl); struct mbox_chan *mbox_request_channel_byname(struct mbox_client *cl, const char *name); struct mbox_chan *mbox_request_channel(struct mbox_client *cl, int index); From patchwork Fri Jan 20 22:46:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 644682 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B4D9C52D11 for ; Fri, 20 Jan 2023 22:48:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229967AbjATWsJ (ORCPT ); Fri, 20 Jan 2023 17:48:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229915AbjATWsG (ORCPT ); Fri, 20 Jan 2023 17:48:06 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A37D237F09; Fri, 20 Jan 2023 14:47:50 -0800 (PST) Received: from pps.filterd (m0279871.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KM03ch024187; Fri, 20 Jan 2023 22:47:40 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=ovIk7K18fTf8BAB1Q7TgDJbm1HCEqrh/BQIJHr0vR7Q=; b=aF/evuq0EMXoKA3Yr8jd5gdoFm9yEp3hfyCgGYnB06+oIzRodnukvdfk+8HpKpQLlsxI H/tasgrMptG1vC1JjUSLqQRLNsrez3j/JyiZMAUTSWaWf5l02wUjvrNlVd7JQLmg6Nux +LQ+g8bdaNSGuKP+sIanXo96UWFJUwaWi6y1bjhJpl2kApIGhMNApQs1Qs5z09Bb3HCk hAMaH82ff52MlO+KmlE63+mR1fZK+uLIqQbwg+U80VlbLihjsOVoqS2KRn24zL+LOcU5 wbH5Lt1YEJg9LkmtJugneBrnJbgaSKZtcQqDdHcRwUdEcCSKrsj7k6Svh1Lk6T7xmCTo gA== Received: from nasanppmta05.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7c28jrer-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:39 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA05.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMlcYK030181 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:38 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:38 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala , Catalin Marinas , "Will Deacon" CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 07/27] virt: gunyah: msgq: Add hypercalls to send and receive messages Date: Fri, 20 Jan 2023 14:46:06 -0800 Message-ID: <20230120224627.4053418-8-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: -j1pKOQnNXgVbsFDIKQgxAlfSA6Hzwfc X-Proofpoint-ORIG-GUID: -j1pKOQnNXgVbsFDIKQgxAlfSA6Hzwfc X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 spamscore=0 priorityscore=1501 adultscore=0 lowpriorityscore=0 mlxlogscore=999 clxscore=1015 mlxscore=0 suspectscore=0 malwarescore=0 bulkscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Add hypercalls to send and receive messages on a Gunyah message queue. Signed-off-by: Elliot Berman --- arch/arm64/gunyah/gunyah_hypercall.c | 33 ++++++++++++++++++++++++++++ include/linux/gunyah.h | 5 +++++ 2 files changed, 38 insertions(+) diff --git a/arch/arm64/gunyah/gunyah_hypercall.c b/arch/arm64/gunyah/gunyah_hypercall.c index ffed4b71641f..d93ad2c08479 100644 --- a/arch/arm64/gunyah/gunyah_hypercall.c +++ b/arch/arm64/gunyah/gunyah_hypercall.c @@ -13,6 +13,8 @@ static const uint32_t gunyah_known_uuids[][4] = { }; #define GH_HYPERCALL_HYP_IDENTIFY GH_HYPERCALL(0x0000) +#define GH_HYPERCALL_MSGQ_SEND GH_HYPERCALL(0x001B) +#define GH_HYPERCALL_MSGQ_RECV GH_HYPERCALL(0x001C) /** * gh_hypercall_get_uid() - Returns a UID when running under a Gunyah hypervisor @@ -71,5 +73,36 @@ void gh_hypercall_hyp_identify(struct gh_hypercall_hyp_identify_resp *hyp_identi } EXPORT_SYMBOL_GPL(gh_hypercall_hyp_identify); +int gh_hypercall_msgq_send(u64 capid, size_t size, uintptr_t buff, int tx_flags, bool *ready) +{ + struct arm_smccc_res res; + + arm_smccc_1_1_hvc(GH_HYPERCALL_MSGQ_SEND, capid, size, buff, tx_flags, 0, &res); + + if (res.a0) + return res.a0; + + *ready = res.a1; + + return res.a0; +} +EXPORT_SYMBOL_GPL(gh_hypercall_msgq_send); + +int gh_hypercall_msgq_recv(u64 capid, uintptr_t buff, size_t size, size_t *recv_size, bool *ready) +{ + struct arm_smccc_res res; + + arm_smccc_1_1_hvc(GH_HYPERCALL_MSGQ_RECV, capid, buff, size, 0, &res); + + if (res.a0) + return res.a0; + + *recv_size = res.a1; + *ready = res.a2; + + return res.a0; +} +EXPORT_SYMBOL_GPL(gh_hypercall_msgq_recv); + MODULE_LICENSE("GPL"); MODULE_DESCRIPTION("Gunyah Hypervisor Hypercalls"); diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h index 6724d1264d58..b5f61c14ec1e 100644 --- a/include/linux/gunyah.h +++ b/include/linux/gunyah.h @@ -104,4 +104,9 @@ struct gh_hypercall_hyp_identify_resp { void gh_hypercall_hyp_identify(struct gh_hypercall_hyp_identify_resp *hyp_identity); +#define GH_HYPERCALL_MSGQ_TX_FLAGS_PUSH BIT(0) + +int gh_hypercall_msgq_send(u64 capid, size_t size, uintptr_t buff, int tx_flags, bool *ready); +int gh_hypercall_msgq_recv(u64 capid, uintptr_t buff, size_t size, size_t *recv_size, bool *ready); + #endif From patchwork Fri Jan 20 22:46:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 645106 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B807C38159 for ; Fri, 20 Jan 2023 22:48:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230041AbjATWs2 (ORCPT ); Fri, 20 Jan 2023 17:48:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229969AbjATWsJ (ORCPT ); Fri, 20 Jan 2023 17:48:09 -0500 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0EB1913DC4; Fri, 20 Jan 2023 14:47:56 -0800 (PST) Received: from pps.filterd (m0279864.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KM4WNV032638; Fri, 20 Jan 2023 22:47:42 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=JYHyFSAeNCnz7vydCdyxt3nbDj5JL+Y3zRy0rMWU20c=; b=Wvh6nanF/qjBE1KXxh8qR3EGWCbdOtLv+2/s9/7KM0qUX0WJmpnFipQGf0XcFcEFvmkf 6tQRR4348vMLS8QQ3HvCnH8nNKek8KaRq1YKlKTkAcJEud1p500ta+XOgH1mDAbTTtju +BW8wAaFUx0zw+jCCW2Jh0ADS0vdu2I5/W//i2iPwhnwVcwntt7Mg66wpMk8tCANGge9 17l0IMuyBSZgws0ViiL8vujXUkxINBghs25K2RlvFDDeBlSHNiaoJTy6jbJD5Ha9CTS8 6hBLYzydj28U+aoCf4VukFdI2SxNoQaWBXWrGEVaUWGUmbHnN/erIv+S+wvSnhdnqwPV 4A== Received: from nasanppmta03.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7q859eay-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:41 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA03.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMlen7017541 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:40 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:39 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala , Jonathan Corbet , Jassi Brar CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Bagas Sanjaya , Catalin Marinas , "Will Deacon" , Marc Zyngier , Sudeep Holla , , , , , Subject: [PATCH v9 08/27] mailbox: Add Gunyah message queue mailbox Date: Fri, 20 Jan 2023 14:46:07 -0800 Message-ID: <20230120224627.4053418-9-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: ngfg6vBogh8bIkeqJMiCNRnHo84XLgCA X-Proofpoint-GUID: ngfg6vBogh8bIkeqJMiCNRnHo84XLgCA X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 suspectscore=0 spamscore=0 malwarescore=0 priorityscore=1501 clxscore=1015 adultscore=0 phishscore=0 lowpriorityscore=0 bulkscore=0 mlxlogscore=505 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Gunyah message queues are a unidirectional inter-VM pipe for messages up to 1024 bytes. This driver supports pairing a receiver message queue and a transmitter message queue to expose a single mailbox channel. Signed-off-by: Elliot Berman --- Documentation/virt/gunyah/message-queue.rst | 8 + MAINTAINERS | 1 + drivers/mailbox/Makefile | 2 + drivers/mailbox/gunyah-msgq.c | 214 ++++++++++++++++++++ include/linux/gunyah.h | 59 +++++- 5 files changed, 283 insertions(+), 1 deletion(-) create mode 100644 drivers/mailbox/gunyah-msgq.c diff --git a/Documentation/virt/gunyah/message-queue.rst b/Documentation/virt/gunyah/message-queue.rst index 0667b3eb1ff9..082085e981e0 100644 --- a/Documentation/virt/gunyah/message-queue.rst +++ b/Documentation/virt/gunyah/message-queue.rst @@ -59,3 +59,11 @@ vIRQ: two TX message queues will have two vIRQs (and two capability IDs). | | | | | | | | | | | | +---------------+ +-----------------+ +---------------+ + +Gunyah message queues are exposed as mailboxes. To create the mailbox, create +a mbox_client and call `gh_msgq_init`. On receipt of the RX_READY interrupt, +all messages in the RX message queue are read and pushed via the `rx_callback` +of the registered mbox_client. + +.. kernel-doc:: drivers/mailbox/gunyah-msgq.c + :identifiers: gh_msgq_init diff --git a/MAINTAINERS b/MAINTAINERS index fe19e71efc6d..d02e8abe6457 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9049,6 +9049,7 @@ S: Supported F: Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml F: Documentation/virt/gunyah/ F: arch/arm64/gunyah/ +F: drivers/mailbox/gunyah-msgq.c F: drivers/virt/gunyah/ F: include/linux/gunyah.h diff --git a/drivers/mailbox/Makefile b/drivers/mailbox/Makefile index fc9376117111..5f929bb55e9a 100644 --- a/drivers/mailbox/Makefile +++ b/drivers/mailbox/Makefile @@ -55,6 +55,8 @@ obj-$(CONFIG_MTK_CMDQ_MBOX) += mtk-cmdq-mailbox.o obj-$(CONFIG_ZYNQMP_IPI_MBOX) += zynqmp-ipi-mailbox.o +obj-$(CONFIG_GUNYAH) += gunyah-msgq.o + obj-$(CONFIG_SUN6I_MSGBOX) += sun6i-msgbox.o obj-$(CONFIG_SPRD_MBOX) += sprd-mailbox.o diff --git a/drivers/mailbox/gunyah-msgq.c b/drivers/mailbox/gunyah-msgq.c new file mode 100644 index 000000000000..caa283f7248b --- /dev/null +++ b/drivers/mailbox/gunyah-msgq.c @@ -0,0 +1,214 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +#define mbox_chan_to_msgq(chan) (container_of(chan->mbox, struct gh_msgq, mbox)) + +static irqreturn_t gh_msgq_rx_irq_handler(int irq, void *data) +{ + struct gh_msgq *msgq = data; + struct gh_msgq_rx_data rx_data; + unsigned long gh_err; + bool ready = true; + + while (ready) { + gh_err = gh_hypercall_msgq_recv(msgq->rx_ghrsc->capid, + (uintptr_t)&rx_data.data, sizeof(rx_data.data), + &rx_data.length, &ready); + if (gh_err == GH_ERROR_OK) { + mbox_chan_received_data(gh_msgq_chan(msgq), &rx_data); + } else if (gh_err == GH_ERROR_MSGQUEUE_EMPTY) { + break; + } else { + pr_warn("Failed to receive data from msgq for %s: %zd\n", + msgq->mbox.dev ? dev_name(msgq->mbox.dev) : "", gh_err); + break; + } + } + + return IRQ_HANDLED; +} + +static irqreturn_t gh_msgq_tx_irq_handler(int irq, void *data) +{ + struct gh_msgq *msgq = data; + + mbox_chan_txdone(gh_msgq_chan(msgq), 0); + + return IRQ_HANDLED; +} + +static void gh_msgq_txdone_tasklet(struct tasklet_struct *tasklet) +{ + struct gh_msgq *msgq = container_of(tasklet, struct gh_msgq, txdone_tasklet); + + mbox_chan_txdone(gh_msgq_chan(msgq), msgq->last_ret); +} + +static int gh_msgq_send_data(struct mbox_chan *chan, void *data) +{ + struct gh_msgq *msgq = mbox_chan_to_msgq(chan); + struct gh_msgq_tx_data *msgq_data = data; + u64 tx_flags = 0; + unsigned long ret; + bool ready; + + if (msgq_data->push) + tx_flags |= GH_HYPERCALL_MSGQ_TX_FLAGS_PUSH; + + ret = gh_hypercall_msgq_send(msgq->tx_ghrsc->capid, msgq_data->length, + (uintptr_t)msgq_data->data, tx_flags, &ready); + + /** + * unlikely because Linux tracks state of msgq and should not try to + * send message when msgq is full. + */ + if (unlikely(ret == GH_ERROR_MSGQUEUE_FULL)) + return -EAGAIN; + + /** + * Propagate all other errors to client. If we return error to mailbox + * framework, then no other messages can be sent and nobody will know + * to retry this message. + */ + msgq->last_ret = gh_remap_error(ret); + + /** + * This message was successfully sent, but message queue isn't ready to + * receive more messages because it's now full. Mailbox framework + * requires that we only report that message was transmitted when + * we're ready to transmit another message. We'll get that in the form + * of tx IRQ once the other side starts to drain the msgq. + */ + if (ret == GH_ERROR_OK && !ready) + return 0; + + /** + * We can send more messages. Mailbox framework requires that tx done + * happens asynchronously to sending the message. Gunyah message queues + * tell us right away on the hypercall return whether we can send more + * messages. To work around this, defer the txdone to a tasklet. + */ + tasklet_schedule(&msgq->txdone_tasklet); + + return 0; +} + +static struct mbox_chan_ops gh_msgq_ops = { + .send_data = gh_msgq_send_data, +}; + +/** + * gh_msgq_init() - Initialize a Gunyah message queue with an mbox_client + * @parent: optional, device parent used for the mailbox controller + * @msgq: Pointer to the gh_msgq to initialize + * @cl: A mailbox client to bind to the mailbox channel that the message queue creates + * @tx_ghrsc: optional, the transmission side of the message queue + * @rx_ghrsc: optional, the receiving side of the message queue + * + * At least one of tx_ghrsc and rx_ghrsc should be not NULL. Most message queue use cases come with + * a pair of message queues to facilitate bidirectional communication. When tx_ghrsc is set, + * the client can send messages with mbox_send_message(gh_msgq_chan(msgq), msg). When rx_ghrsc + * is set, the mbox_client should register an .rx_callback() and the message queue driver will + * push all available messages upon receiving the RX ready interrupt. The messages should be + * consumed or copied by the client right away as the gh_msgq_rx_data will be replaced/destroyed + * after the callback. + * + * Returns - 0 on success, negative otherwise + */ +int gh_msgq_init(struct device *parent, struct gh_msgq *msgq, struct mbox_client *cl, + struct gunyah_resource *tx_ghrsc, struct gunyah_resource *rx_ghrsc) +{ + int ret; + + /* Must have at least a tx_ghrsc or rx_ghrsc and that they are the right device types */ + if ((!tx_ghrsc && !rx_ghrsc) || + (tx_ghrsc && tx_ghrsc->type != GUNYAH_RESOURCE_TYPE_MSGQ_TX) || + (rx_ghrsc && rx_ghrsc->type != GUNYAH_RESOURCE_TYPE_MSGQ_RX)) + return -EINVAL; + + if (gh_api_version() != GUNYAH_API_V1) { + pr_warn("Unrecognized gunyah version: %u. Currently supported: %d\n", + gh_api_version(), GUNYAH_API_V1); + return -ENODEV; + } + + if (!gh_api_has_feature(GH_API_FEATURE_MSGQUEUE)) + return -EOPNOTSUPP; + + msgq->tx_ghrsc = tx_ghrsc; + msgq->rx_ghrsc = rx_ghrsc; + + msgq->mbox.dev = parent; + msgq->mbox.ops = &gh_msgq_ops; + msgq->mbox.num_chans = 1; + msgq->mbox.chans = kcalloc(msgq->mbox.num_chans, sizeof(*msgq->mbox.chans), GFP_KERNEL); + if (!msgq->mbox.chans) + return -ENOMEM; + msgq->mbox.txdone_irq = true; + + if (msgq->tx_ghrsc) { + ret = request_irq(msgq->tx_ghrsc->irq, gh_msgq_tx_irq_handler, 0, "gh_msgq_tx", + msgq); + if (ret) + goto err_chans; + } + + if (msgq->rx_ghrsc) { + ret = request_threaded_irq(msgq->rx_ghrsc->irq, NULL, gh_msgq_rx_irq_handler, + IRQF_ONESHOT, "gh_msgq_rx", msgq); + if (ret) + goto err_tx_irq; + } + + tasklet_setup(&msgq->txdone_tasklet, gh_msgq_txdone_tasklet); + + ret = mbox_controller_register(&msgq->mbox); + if (ret) + goto err_rx_irq; + + ret = mbox_bind_client(gh_msgq_chan(msgq), cl); + if (ret) + goto err_mbox; + + return 0; +err_mbox: + mbox_controller_unregister(&msgq->mbox); +err_rx_irq: + if (msgq->rx_ghrsc) + free_irq(msgq->rx_ghrsc->irq, msgq); +err_tx_irq: + if (msgq->tx_ghrsc) + free_irq(msgq->tx_ghrsc->irq, msgq); +err_chans: + kfree(msgq->mbox.chans); + return ret; +} +EXPORT_SYMBOL_GPL(gh_msgq_init); + +void gh_msgq_remove(struct gh_msgq *msgq) +{ + mbox_controller_unregister(&msgq->mbox); + + if (msgq->rx_ghrsc) + free_irq(msgq->rx_ghrsc->irq, msgq); + + if (msgq->tx_ghrsc) + free_irq(msgq->tx_ghrsc->irq, msgq); + + kfree(msgq->mbox.chans); +} +EXPORT_SYMBOL_GPL(gh_msgq_remove); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("Gunyah Message Queue Driver"); diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h index b5f61c14ec1e..5d6fc9f88718 100644 --- a/include/linux/gunyah.h +++ b/include/linux/gunyah.h @@ -8,10 +8,67 @@ #include #include +#include #include +#include +#include #include -/* Common Gunyah macros */ +/* Follows resource manager's resource types for VM_GET_HYP_RESOURCES */ +enum gunyah_resource_type { + GUNYAH_RESOURCE_TYPE_BELL_TX = 0, + GUNYAH_RESOURCE_TYPE_BELL_RX = 1, + GUNYAH_RESOURCE_TYPE_MSGQ_TX = 2, + GUNYAH_RESOURCE_TYPE_MSGQ_RX = 3, + GUNYAH_RESOURCE_TYPE_VCPU = 4, +}; + +struct gunyah_resource { + enum gunyah_resource_type type; + u64 capid; + int irq; +}; + +/** + * Gunyah Message Queues + */ + +#define GH_MSGQ_MAX_MSG_SIZE 240 + +struct gh_msgq_tx_data { + size_t length; + bool push; + char data[]; +}; + +struct gh_msgq_rx_data { + size_t length; + char data[GH_MSGQ_MAX_MSG_SIZE]; +}; + +struct gh_msgq { + struct gunyah_resource *tx_ghrsc; + struct gunyah_resource *rx_ghrsc; + + /* msgq private */ + int last_ret; /* Linux error, not GH_STATUS_* */ + struct mbox_controller mbox; + struct tasklet_struct txdone_tasklet; +}; + + +int gh_msgq_init(struct device *parent, struct gh_msgq *msgq, struct mbox_client *cl, + struct gunyah_resource *tx_ghrsc, struct gunyah_resource *rx_ghrsc); +void gh_msgq_remove(struct gh_msgq *msgq); + +static inline struct mbox_chan *gh_msgq_chan(struct gh_msgq *msgq) +{ + return &msgq->mbox.chans[0]; +} + +/******************************************************************************/ +/* Common arch-independent macros and definitions for Gunyah hypercalls */ + #define GH_CAPID_INVAL U64_MAX #define GH_VMID_ROOT_VM 0xff From patchwork Fri Jan 20 22:46:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 645105 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FBDCC05027 for ; Fri, 20 Jan 2023 22:48:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229969AbjATWsm (ORCPT ); Fri, 20 Jan 2023 17:48:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56566 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230084AbjATWsb (ORCPT ); Fri, 20 Jan 2023 17:48:31 -0500 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A62A66FA0; Fri, 20 Jan 2023 14:48:08 -0800 (PST) Received: from pps.filterd (m0279866.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KMZXXu022939; Fri, 20 Jan 2023 22:47:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=4X74yKIrLrqF+b1pUTzOb8tH+X7zRbJn3cEBM2UgYuE=; b=InuzS6CStbayoHiw4f18cxVgHS0Lz/MHEDamS115sQ2xm05/HwG3ltC6efVD4jVcbL/7 nNjP/2twik6Wc2G4n/ttamgRTe7RA9OQ7hE4HFRdLCLPgau8nf1sBvsqYYC3LjYrC/4h dBJ7lJMJcVN43Ls+Q2b1+g0C6DTmrMjbLSWd+Ghc6rTsFB/VBOujW1a0DKeJ0iMeRNQd Zki5VR7Jrkt8st8CmSeifnKje0e7/5JSGw+3x5O6hwhM47Mjd/fPqXfdHkhZ6ouBL6eY 1bxqHp1db+ktGgehHuGQ+f7Q9tTi9epwJdhsfmUtMJ/bBuPhBd6qXMoqrtZhkRytn3Nr 3w== Received: from nasanppmta02.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7yc60gh1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:43 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA02.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMlhDB027197 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:43 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:42 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Will Deacon , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 09/27] gunyah: rsc_mgr: Add resource manager RPC core Date: Fri, 20 Jan 2023 14:46:08 -0800 Message-ID: <20230120224627.4053418-10-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: Yaarz7o1gMZecAiR7JQ4DD_2dcT_0Cjn X-Proofpoint-ORIG-GUID: Yaarz7o1gMZecAiR7JQ4DD_2dcT_0Cjn X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 mlxscore=0 priorityscore=1501 spamscore=0 impostorscore=0 phishscore=0 clxscore=1015 mlxlogscore=999 adultscore=0 malwarescore=0 bulkscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org The resource manager is a special virtual machine which is always running on a Gunyah system. It provides APIs for creating and destroying VMs, secure memory management, sharing/lending of memory between VMs, and setup of inter-VM communication. Calls to the resource manager are made via message queues. This patch implements the basic probing and RPC mechanism to make those API calls. Request/response calls can be made with gh_rm_call. Drivers can also register to notifications pushed by RM via gh_rm_register_notifier Specific API calls that resource manager supports will be implemented in subsequent patches. Signed-off-by: Elliot Berman --- MAINTAINERS | 2 +- drivers/virt/gunyah/Kconfig | 1 - drivers/virt/gunyah/Makefile | 3 + drivers/virt/gunyah/rsc_mgr.c | 616 +++++++++++++++++++++++++++++++++ drivers/virt/gunyah/rsc_mgr.h | 71 ++++ include/linux/gunyah.h | 3 +- include/linux/gunyah_rsc_mgr.h | 18 + 7 files changed, 710 insertions(+), 4 deletions(-) create mode 100644 drivers/virt/gunyah/rsc_mgr.c create mode 100644 drivers/virt/gunyah/rsc_mgr.h create mode 100644 include/linux/gunyah_rsc_mgr.h diff --git a/MAINTAINERS b/MAINTAINERS index d02e8abe6457..7ab1462577cc 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9051,7 +9051,7 @@ F: Documentation/virt/gunyah/ F: arch/arm64/gunyah/ F: drivers/mailbox/gunyah-msgq.c F: drivers/virt/gunyah/ -F: include/linux/gunyah.h +F: include/linux/gunyah*.h HABANALABS PCI DRIVER M: Oded Gabbay diff --git a/drivers/virt/gunyah/Kconfig b/drivers/virt/gunyah/Kconfig index 71c38845bcea..1a737694c333 100644 --- a/drivers/virt/gunyah/Kconfig +++ b/drivers/virt/gunyah/Kconfig @@ -4,7 +4,6 @@ config GUNYAH tristate "Gunyah Virtualization drivers" depends on ARM64 depends on MAILBOX - select GUNYAH_PLATFORM_HOOKS help The Gunyah drivers are the helper interfaces that run in a guest VM such as basic inter-VM IPC and signaling mechanisms, and higher level diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile index 34f32110faf9..cc864ff5abbb 100644 --- a/drivers/virt/gunyah/Makefile +++ b/drivers/virt/gunyah/Makefile @@ -1,3 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_GUNYAH) += gunyah.o + +gunyah_rsc_mgr-y += rsc_mgr.o +obj-$(CONFIG_GUNYAH) += gunyah_rsc_mgr.o diff --git a/drivers/virt/gunyah/rsc_mgr.c b/drivers/virt/gunyah/rsc_mgr.c new file mode 100644 index 000000000000..382f9943fd31 --- /dev/null +++ b/drivers/virt/gunyah/rsc_mgr.c @@ -0,0 +1,616 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "rsc_mgr.h" + +#define RM_RPC_API_VERSION_MASK GENMASK(3, 0) +#define RM_RPC_HEADER_WORDS_MASK GENMASK(7, 4) +#define RM_RPC_API_VERSION FIELD_PREP(RM_RPC_API_VERSION_MASK, 1) +#define RM_RPC_HEADER_WORDS FIELD_PREP(RM_RPC_HEADER_WORDS_MASK, \ + (sizeof(struct gh_rm_rpc_hdr) / sizeof(u32))) +#define RM_RPC_API (RM_RPC_API_VERSION | RM_RPC_HEADER_WORDS) + +#define RM_RPC_TYPE_CONTINUATION 0x0 +#define RM_RPC_TYPE_REQUEST 0x1 +#define RM_RPC_TYPE_REPLY 0x2 +#define RM_RPC_TYPE_NOTIF 0x3 +#define RM_RPC_TYPE_MASK GENMASK(1, 0) + +#define GH_RM_MAX_NUM_FRAGMENTS 62 +#define RM_RPC_FRAGMENTS_MASK GENMASK(7, 2) + +struct gh_rm_rpc_hdr { + u8 api; + u8 type; + __le16 seq; + __le32 msg_id; +} __packed; + +struct gh_rm_rpc_reply_hdr { + struct gh_rm_rpc_hdr rpc_hdr; + __le32 err_code; /* GH_RM_ERROR_* */ +} __packed; + +#define GH_RM_MAX_MSG_SIZE (GH_MSGQ_MAX_MSG_SIZE - sizeof(struct gh_rm_rpc_hdr)) + +/** + * struct gh_rm_connection - Represents a complete message from resource manager + * @payload: Combined payload of all the fragments (msg headers stripped off). + * @size: Size of the payload received so far. + * @ret: Linux return code, set in case there was an error processing connection + * @msg_id: Message ID from the header. + * @type: RM_RPC_TYPE_REPLY or RM_RPC_TYPE_NOTIF. + * @num_fragments: total number of fragments expected to be received. + * @fragments_received: fragments received so far. + * @rm_error: For request/reply sequences with standard replies. + * @seq: Sequence ID for the main message. + * @seq_done: Signals caller that the RM reply has been received + */ +struct gh_rm_connection { + void *payload; + size_t size; + __le32 msg_id; + u8 type; + + u8 num_fragments; + u8 fragments_received; + + union { + struct { + /* only for req/reply sequence */ + int ret; + u16 seq; + u32 rm_error; /* GH_RM_ERROR_* */ + struct completion seq_done; + } reply; + + struct { + struct work_struct work; + } notification; + }; +}; + +struct gh_rm_notif_complete { + struct gh_rm_connection *conn; + struct work_struct work; +}; + +struct gh_rm { + struct device *dev; + struct gunyah_resource tx_ghrsc, rx_ghrsc; + struct gh_msgq msgq; + struct mbox_client msgq_client; + struct gh_rm_connection *active_rx_connection; + int last_tx_ret; + + struct idr call_idr; + struct mutex call_idr_lock; + + struct kmem_cache *cache; + struct mutex send_lock; + + struct work_struct recv_work; +}; + +static struct gh_rm_connection *gh_rm_alloc_connection(__le32 msg_id, u8 type) +{ + struct gh_rm_connection *connection; + + connection = kzalloc(sizeof(*connection), GFP_KERNEL); + if (!connection) + return NULL; + + connection->type = type; + connection->msg_id = msg_id; + + return connection; +} + +static int gh_rm_init_connection_payload(struct gh_rm_connection *connection, void *msg, + size_t hdr_size, size_t msg_size) +{ + struct gh_rm_rpc_hdr *hdr = msg; + size_t max_buf_size, payload_size; + + if (hdr_size > msg_size) + return -EINVAL; + + payload_size = msg_size - hdr_size; + + connection->num_fragments = FIELD_GET(RM_RPC_FRAGMENTS_MASK, hdr->type); + connection->fragments_received = 0; + + /* There's not going to be any payload, no need to allocate buffer. */ + if (!payload_size && !connection->num_fragments) + return 0; + + if (connection->num_fragments > GH_RM_MAX_NUM_FRAGMENTS) + return -EINVAL; + + max_buf_size = payload_size + (connection->num_fragments * GH_RM_MAX_MSG_SIZE); + + connection->payload = kzalloc(max_buf_size, GFP_KERNEL); + if (!connection->payload) + return -ENOMEM; + + memcpy(connection->payload, msg + hdr_size, payload_size); + connection->size = payload_size; + return 0; +} + +static void gh_rm_notif_work(struct work_struct *work) +{ + struct gh_rm_connection *connection = container_of(work, struct gh_rm_connection, + notification.work); + + /* No users of notifications, yet. */ + + kfree(connection->payload); + kfree(connection); +} + +static struct gh_rm_connection *gh_rm_process_notif(struct gh_rm *rm, void *msg, size_t msg_size) +{ + struct gh_rm_rpc_hdr *hdr = msg; + struct gh_rm_connection *connection; + int ret; + + connection = gh_rm_alloc_connection(hdr->msg_id, RM_RPC_TYPE_NOTIF); + if (!connection) { + dev_err(rm->dev, "Failed to alloc connection for notification, dropping.\n"); + return NULL; + } + + INIT_WORK(&connection->notification.work, gh_rm_notif_work); + + ret = gh_rm_init_connection_payload(connection, msg, sizeof(*hdr), msg_size); + if (ret) { + dev_err(rm->dev, "Failed to initialize connection buffer for notification: %d\n", + ret); + kfree(connection); + return NULL; + } + + return connection; +} + +static struct gh_rm_connection *gh_rm_process_rply(struct gh_rm *rm, void *msg, size_t msg_size) +{ + struct gh_rm_rpc_reply_hdr *reply_hdr = msg; + struct gh_rm_rpc_hdr *hdr = &reply_hdr->rpc_hdr; + struct gh_rm_connection *connection; + u16 seq_id = le16_to_cpu(hdr->seq); + + mutex_lock(&rm->call_idr_lock); + connection = idr_find(&rm->call_idr, seq_id); + mutex_unlock(&rm->call_idr_lock); + + if (!connection) { + dev_err(rm->dev, "Failed to find connection for sequence %u\n", seq_id); + return NULL; + } + if (connection->msg_id != hdr->msg_id) { + dev_err(rm->dev, "Reply for sequence %u expected msg_id: %x but got %x\n", + seq_id, le32_to_cpu(connection->msg_id), le32_to_cpu(hdr->msg_id)); + /* + * Don't complete connection and error the client, maybe + * resource manager will send us the expected reply sequence soon. + */ + return NULL; + } + + if (gh_rm_init_connection_payload(connection, msg, sizeof(*reply_hdr), msg_size)) { + dev_err(rm->dev, "Failed to alloc connection buffer for sequence %d\n", seq_id); + /* Send connection complete and error the client. */ + connection->reply.ret = -ENOMEM; + complete(&connection->reply.seq_done); + return NULL; + } + + connection->reply.rm_error = le32_to_cpu(reply_hdr->err_code); + return connection; +} + +static int gh_rm_process_cont(struct gh_rm *rm, struct gh_rm_connection *connection, + void *msg, size_t msg_size) +{ + struct gh_rm_rpc_hdr *hdr = msg; + size_t payload_size = msg_size - sizeof(*hdr); + + /* + * hdr->fragments and hdr->msg_id preserves the value from first reply + * or notif message. To detect mishandling, check it's still intact. + */ + if (connection->msg_id != hdr->msg_id) { + dev_err(rm->dev, "Appending mismatched continuation with id %d to connection with id %d\n", + le32_to_cpu(hdr->msg_id), le32_to_cpu(connection->msg_id)); + return -EINVAL; + } + if (connection->num_fragments != FIELD_GET(RM_RPC_FRAGMENTS_MASK, hdr->type)) { + dev_err(rm->dev, "Number of fragments mismatch for message %x\n", + le32_to_cpu(hdr->msg_id)); + return -EINVAL; + } + + memcpy(connection->payload + connection->size, msg + sizeof(*hdr), payload_size); + connection->size += payload_size; + connection->fragments_received++; + return 0; +} + +static bool gh_rm_complete_connection(struct gh_rm *rm, struct gh_rm_connection *connection) +{ + if (!connection) + return false; + + if (connection->fragments_received != connection->num_fragments) + return false; + + switch (connection->type) { + case RM_RPC_TYPE_REPLY: + complete(&connection->reply.seq_done); + break; + case RM_RPC_TYPE_NOTIF: + schedule_work(&connection->notification.work); + break; + default: + dev_err(rm->dev, "Invalid message type (%d) received\n", connection->type); + break; + } + + return true; +} + +static void gh_rm_abort_connection(struct gh_rm_connection *connection) +{ + switch (connection->type) { + case RM_RPC_TYPE_REPLY: + connection->reply.ret = -EIO; + complete(&connection->reply.seq_done); + break; + case RM_RPC_TYPE_NOTIF: + fallthrough; + default: + kfree(connection->payload); + kfree(connection); + } +} + +static void gh_rm_msgq_rx_data(struct mbox_client *cl, void *mssg) +{ + struct gh_rm *rm = container_of(cl, struct gh_rm, msgq_client); + struct gh_msgq_rx_data *rx_data = mssg; + void *msg = rx_data->data; + size_t msg_size = rx_data->length; + struct gh_rm_rpc_hdr *hdr; + + if (msg_size <= sizeof(struct gh_rm_rpc_hdr)) { + dev_err(rm->dev, "Incomplete message size: %ld is too small\n", msg_size); + return; + } + + hdr = msg; + if (hdr->api != RM_RPC_API) { + dev_err(rm->dev, "Unknown RM RPC API version: %x\n", hdr->api); + return; + } + + if (msg_size > GH_MSGQ_MAX_MSG_SIZE) { + dev_err(rm->dev, "Unexpected large message received: %ld\n", msg_size); + return; + } + + switch (FIELD_GET(RM_RPC_TYPE_MASK, hdr->type)) { + case RM_RPC_TYPE_NOTIF: + if (rm->active_rx_connection) { + /* Not possible per protocol. Do something better than BUG_ON */ + dev_err(rm->dev, "Received start of new notification without finishing existing message series.\n"); + gh_rm_abort_connection(rm->active_rx_connection); + } + rm->active_rx_connection = gh_rm_process_notif(rm, msg, msg_size); + break; + case RM_RPC_TYPE_REPLY: + if (rm->active_rx_connection) { + /* Not possible per protocol. Do something better than BUG_ON */ + dev_err(rm->dev, "Received start of new reply without finishing existing message series.\n"); + gh_rm_abort_connection(rm->active_rx_connection); + } + rm->active_rx_connection = gh_rm_process_rply(rm, msg, msg_size); + break; + case RM_RPC_TYPE_CONTINUATION: + if (!rm->active_rx_connection) { + dev_err(rm->dev, "Received a continuation message without receiving initial message\n"); + break; + } + if (gh_rm_process_cont(rm, rm->active_rx_connection, msg, msg_size)) { + gh_rm_abort_connection(rm->active_rx_connection); + rm->active_rx_connection = NULL; + } + break; + default: + dev_err(rm->dev, "Invalid message type (%lu) received\n", + FIELD_GET(RM_RPC_TYPE_MASK, hdr->type)); + return; + } + + if (gh_rm_complete_connection(rm, rm->active_rx_connection)) + rm->active_rx_connection = NULL; +} + +static void gh_rm_msgq_tx_done(struct mbox_client *cl, void *mssg, int r) +{ + struct gh_rm *rm = container_of(cl, struct gh_rm, msgq_client); + + kmem_cache_free(rm->cache, mssg); + rm->last_tx_ret = r; +} + +static int gh_rm_send_request(struct gh_rm *rm, u32 message_id, + const void *req_buff, size_t req_buff_size, + struct gh_rm_connection *connection) +{ + size_t buff_size_remaining = req_buff_size; + u32 cont_fragments = 0; + struct gh_rm_rpc_hdr *hdr; + struct gh_msgq_tx_data *msg; + u8 msg_type = FIELD_PREP(RM_RPC_TYPE_MASK, RM_RPC_TYPE_REQUEST); + void *payload; + size_t payload_size; + const void *req_buff_curr = req_buff; + int ret; + + if (req_buff_size) + cont_fragments = (req_buff_size - 1) / GH_RM_MAX_MSG_SIZE; + + if (req_buff_size > GH_RM_MAX_NUM_FRAGMENTS * GH_RM_MAX_MSG_SIZE) { + pr_warn("Limit exceeded for the number of fragments: %u\n", cont_fragments); + dump_stack(); + return -E2BIG; + } + + ret = mutex_lock_interruptible(&rm->send_lock); + if (ret) + return ret; + + /* Consider also the 'request' packet for the loop count */ + do { + msg = kmem_cache_zalloc(rm->cache, GFP_KERNEL); + if (!msg) { + ret = -ENOMEM; + goto out; + } + + /* Fill header */ + hdr = (struct gh_rm_rpc_hdr *)msg->data; + hdr->api = RM_RPC_API; + hdr->type = msg_type | FIELD_PREP(RM_RPC_FRAGMENTS_MASK, cont_fragments); + hdr->seq = cpu_to_le16(connection->reply.seq); + hdr->msg_id = cpu_to_le32(message_id); + + /* Copy payload */ + payload = hdr + 1; + payload_size = min(buff_size_remaining, GH_RM_MAX_MSG_SIZE); + memcpy(payload, req_buff_curr, payload_size); + req_buff_curr += payload_size; + buff_size_remaining -= payload_size; + + /* Force the last fragment to immediately alert the receiver */ + msg->push = !buff_size_remaining; + msg->length = sizeof(*hdr) + payload_size; + + ret = mbox_send_message(gh_msgq_chan(&rm->msgq), msg); + if (ret < 0) { + kmem_cache_free(rm->cache, msg); + break; + } + + if (rm->last_tx_ret) { + ret = rm->last_tx_ret; + break; + } + + msg_type = FIELD_PREP(RM_RPC_TYPE_MASK, RM_RPC_TYPE_CONTINUATION); + } while (buff_size_remaining); + +out: + mutex_unlock(&rm->send_lock); + return ret < 0 ? ret : 0; +} + +/** + * gh_rm_call: Achieve request-response type communication with RPC + * @rm: Pointer to Gunyah resource manager internal data + * @message_id: The RM RPC message-id + * @req_buff: Request buffer that contains the payload + * @req_buff_size: Total size of the payload + * @resp_buf: Pointer to a response buffer + * @resp_buff_size: Size of the response buffer + * + * Make a request to the RM-VM and wait for reply back. For a successful + * response, the function returns the payload. The size of the payload is set in + * resp_buff_size. The resp_buf should be freed by the caller. + * + * req_buff should be not NULL for req_buff_size >0. If req_buff_size == 0, + * req_buff *can* be NULL and no additional payload is sent. + * + * Context: Process context. Will sleep waiting for reply. + * Return: 0 on success. <0 if error. + */ +int gh_rm_call(struct gh_rm *rm, u32 message_id, void *req_buff, size_t req_buff_size, + void **resp_buf, size_t *resp_buff_size) +{ + struct gh_rm_connection *connection; + int ret; + + /* message_id 0 is reserved. req_buff_size implies req_buf is not NULL */ + if (!message_id || (!req_buff && req_buff_size)) + return -EINVAL; + + if (!rm) + return -EPROBE_DEFER; + + connection = gh_rm_alloc_connection(cpu_to_le32(message_id), RM_RPC_TYPE_REPLY); + if (!connection) + return -ENOMEM; + + init_completion(&connection->reply.seq_done); + + /* Allocate a new seq number for this connection */ + mutex_lock(&rm->call_idr_lock); + ret = idr_alloc_cyclic(&rm->call_idr, connection, 0, U16_MAX, + GFP_KERNEL); + mutex_unlock(&rm->call_idr_lock); + if (ret < 0) + goto out; + connection->reply.seq = ret; + + /* Send the request to the Resource Manager */ + ret = gh_rm_send_request(rm, message_id, req_buff, req_buff_size, connection); + if (ret < 0) + goto out; + + /* Wait for response */ + ret = wait_for_completion_interruptible(&connection->reply.seq_done); + if (ret) + goto out; + + /* Check for internal (kernel) error waiting for the response */ + if (connection->reply.ret) { + ret = connection->reply.ret; + kfree(connection->payload); + goto out; + } + + /* Got a response, did resource manager give us an error? */ + if (connection->reply.rm_error) { + pr_warn("RM rejected message %08x. Error: %d\n", message_id, + connection->reply.rm_error); + dump_stack(); + ret = gh_rm_remap_error(connection->reply.rm_error); + kfree(connection->payload); + goto out; + } + + /* Everything looks good, return the payload */ + *resp_buff_size = connection->size; + if (connection->size) + *resp_buf = connection->payload; + else { + /* kfree in case RM sent us multiple fragments but never any data in + * those fragments. We would've allocated memory for it, but connection->size == 0 + */ + kfree(connection->payload); + } + +out: + mutex_lock(&rm->call_idr_lock); + idr_remove(&rm->call_idr, connection->reply.seq); + mutex_unlock(&rm->call_idr_lock); + kfree(connection); + return ret; +} + +static int gh_msgq_platform_probe_direction(struct platform_device *pdev, + bool tx, int idx, struct gunyah_resource *ghrsc) +{ + int ret; + struct device_node *node = pdev->dev.of_node; + + ghrsc->type = tx ? GUNYAH_RESOURCE_TYPE_MSGQ_TX : GUNYAH_RESOURCE_TYPE_MSGQ_RX; + + ghrsc->irq = platform_get_irq(pdev, idx); + if (ghrsc->irq < 0) { + dev_err(&pdev->dev, "Failed to get irq%d: %d\n", idx, ghrsc->irq); + return ghrsc->irq; + } + + ret = of_property_read_u64_index(node, "reg", idx, &ghrsc->capid); + if (ret) { + dev_err(&pdev->dev, "Failed to get capid%d: %d\n", idx, ret); + return ret; + } + + return 0; +} + +static int gh_rm_drv_probe(struct platform_device *pdev) +{ + struct gh_rm *rm; + int ret; + + rm = devm_kzalloc(&pdev->dev, sizeof(*rm), GFP_KERNEL); + if (!rm) + return -ENOMEM; + + platform_set_drvdata(pdev, rm); + rm->dev = &pdev->dev; + + mutex_init(&rm->call_idr_lock); + idr_init(&rm->call_idr); + rm->cache = kmem_cache_create("gh_rm", struct_size(msg, data, GH_MSGQ_MAX_MSG_SIZE), 0, + SLAB_HWCACHE_ALIGN, NULL); + if (!rm->cache) + return -ENOMEM; + mutex_init(&rm->send_lock); + + ret = gh_msgq_platform_probe_direction(pdev, true, 0, &rm->tx_ghrsc); + if (ret) + goto err_cache; + + ret = gh_msgq_platform_probe_direction(pdev, false, 1, &rm->rx_ghrsc); + if (ret) + goto err_cache; + + rm->msgq_client.dev = &pdev->dev; + rm->msgq_client.tx_block = true; + rm->msgq_client.rx_callback = gh_rm_msgq_rx_data; + rm->msgq_client.tx_done = gh_rm_msgq_tx_done; + + return gh_msgq_init(&pdev->dev, &rm->msgq, &rm->msgq_client, &rm->tx_ghrsc, &rm->rx_ghrsc); +} + +static int gh_rm_drv_remove(struct platform_device *pdev) +{ + struct gh_rm *rm = platform_get_drvdata(pdev); + + mbox_free_channel(gh_msgq_chan(&rm->msgq)); + gh_msgq_remove(&rm->msgq); + kmem_cache_destroy(rm->cache); + + return 0; +} + +static const struct of_device_id gh_rm_of_match[] = { + { .compatible = "gunyah-resource-manager" }, + {} +}; +MODULE_DEVICE_TABLE(of, gh_rm_of_match); + +static struct platform_driver gh_rm_driver = { + .probe = gh_rm_drv_probe, + .remove = gh_rm_drv_remove, + .driver = { + .name = "gh_rsc_mgr", + .of_match_table = gh_rm_of_match, + }, +}; +module_platform_driver(gh_rm_driver); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("Gunyah Resource Manager Driver"); diff --git a/drivers/virt/gunyah/rsc_mgr.h b/drivers/virt/gunyah/rsc_mgr.h new file mode 100644 index 000000000000..824749e63a54 --- /dev/null +++ b/drivers/virt/gunyah/rsc_mgr.h @@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ +#ifndef __GH_RSC_MGR_PRIV_H +#define __GH_RSC_MGR_PRIV_H + +#include +#include +#include + +/* RM Error codes */ +#define GH_RM_ERROR_OK 0x0 +#define GH_RM_ERROR_UNIMPLEMENTED 0xFFFFFFFF +#define GH_RM_ERROR_NOMEM 0x1 +#define GH_RM_ERROR_NORESOURCE 0x2 +#define GH_RM_ERROR_DENIED 0x3 +#define GH_RM_ERROR_INVALID 0x4 +#define GH_RM_ERROR_BUSY 0x5 +#define GH_RM_ERROR_ARGUMENT_INVALID 0x6 +#define GH_RM_ERROR_HANDLE_INVALID 0x7 +#define GH_RM_ERROR_VALIDATE_FAILED 0x8 +#define GH_RM_ERROR_MAP_FAILED 0x9 +#define GH_RM_ERROR_MEM_INVALID 0xA +#define GH_RM_ERROR_MEM_INUSE 0xB +#define GH_RM_ERROR_MEM_RELEASED 0xC +#define GH_RM_ERROR_VMID_INVALID 0xD +#define GH_RM_ERROR_LOOKUP_FAILED 0xE +#define GH_RM_ERROR_IRQ_INVALID 0xF +#define GH_RM_ERROR_IRQ_INUSE 0x10 +#define GH_RM_ERROR_IRQ_RELEASED 0x11 + +static inline int gh_rm_remap_error(int rm_error) +{ + switch (rm_error) { + case GH_RM_ERROR_OK: + return 0; + case GH_RM_ERROR_UNIMPLEMENTED: + return -EOPNOTSUPP; + case GH_RM_ERROR_NOMEM: + return -ENOMEM; + case GH_RM_ERROR_NORESOURCE: + return -ENODEV; + case GH_RM_ERROR_DENIED: + return -EPERM; + case GH_RM_ERROR_BUSY: + return -EBUSY; + case GH_RM_ERROR_INVALID: + case GH_RM_ERROR_ARGUMENT_INVALID: + case GH_RM_ERROR_HANDLE_INVALID: + case GH_RM_ERROR_VALIDATE_FAILED: + case GH_RM_ERROR_MAP_FAILED: + case GH_RM_ERROR_MEM_INVALID: + case GH_RM_ERROR_MEM_INUSE: + case GH_RM_ERROR_MEM_RELEASED: + case GH_RM_ERROR_VMID_INVALID: + case GH_RM_ERROR_LOOKUP_FAILED: + case GH_RM_ERROR_IRQ_INVALID: + case GH_RM_ERROR_IRQ_INUSE: + case GH_RM_ERROR_IRQ_RELEASED: + return -EINVAL; + default: + return -EBADMSG; + } +} + +struct gh_rm; +int gh_rm_call(struct gh_rm *rsc_mgr, u32 message_id, void *req_buff, size_t req_buff_size, + void **resp_buf, size_t *resp_buff_size); + +#endif diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h index 5d6fc9f88718..91d2c9f5b2f0 100644 --- a/include/linux/gunyah.h +++ b/include/linux/gunyah.h @@ -119,12 +119,11 @@ static inline int gh_remap_error(int gh_error) return -EACCES; case GH_ERROR_BUSY: case GH_ERROR_IDLE: - return -EBUSY; case GH_ERROR_IRQ_BOUND: case GH_ERROR_IRQ_UNBOUND: case GH_ERROR_MSGQUEUE_FULL: case GH_ERROR_MSGQUEUE_EMPTY: - return -EPERM; + return -EBUSY; default: return -EINVAL; } diff --git a/include/linux/gunyah_rsc_mgr.h b/include/linux/gunyah_rsc_mgr.h new file mode 100644 index 000000000000..959787ad0a3d --- /dev/null +++ b/include/linux/gunyah_rsc_mgr.h @@ -0,0 +1,18 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#ifndef _GUNYAH_RSC_MGR_H +#define _GUNYAH_RSC_MGR_H + +#include +#include +#include + +#define GH_VMID_INVAL U16_MAX + +/* Gunyah recognizes VMID0 as an alias to the current VM's ID */ +#define GH_VMID_SELF 0 + +#endif From patchwork Fri Jan 20 22:46:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 644679 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E55ACC54794 for ; Fri, 20 Jan 2023 22:48:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230109AbjATWsp (ORCPT ); Fri, 20 Jan 2023 17:48:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56492 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230031AbjATWsb (ORCPT ); Fri, 20 Jan 2023 17:48:31 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 703616C110; Fri, 20 Jan 2023 14:48:10 -0800 (PST) Received: from pps.filterd (m0279872.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KMj7Pg012734; Fri, 20 Jan 2023 22:47:46 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=buRBYdqqq4lVajfjuXfcbraNXLbcL/HEObILLeW8jNE=; b=BNXJaDSRbyLUbMnVf3ijSCx4jf9qlw2e4H0u/SrecA83gvrGmFAC09ag/N5Gyj0ZGtyW QZAMPzxNGMJrjZHXxqSYFXMWXeHatqyNPPm3dcII0L7dLvuIbDozKhW2fY9g1yrCPxHR 1Wf0jERLamAne+00Gkj3HrwVoYsL5qW/hpp74SLZ19/4Dys9jfa1iEdm8AKj4JxYZh6B DztMt8sg5l5TsSlH5UXWikueFLpRvmzLscDF8pDqN/o9VITuX+8nUz/wWXd7Od0vSBP6 3eFYERgo435JapcxXMOkDYU+HXv3ySYzNpJXO4Lszh7fCWDHR6ikZkjTHcHEU9lk83Oi lg== Received: from nasanppmta02.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7xm40k8w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:45 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA02.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMliFq027210 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:44 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:43 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Will Deacon , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 10/27] gunyah: rsc_mgr: Add VM lifecycle RPC Date: Fri, 20 Jan 2023 14:46:09 -0800 Message-ID: <20230120224627.4053418-11-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: JS1-phWs0Rucq5eQG4DjwRxi-Ob3IDt4 X-Proofpoint-GUID: JS1-phWs0Rucq5eQG4DjwRxi-Ob3IDt4 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 malwarescore=0 spamscore=0 bulkscore=0 clxscore=1015 priorityscore=1501 mlxlogscore=999 impostorscore=0 mlxscore=0 suspectscore=0 adultscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Add Gunyah Resource Manager RPC to launch an unauthenticated VM. Signed-off-by: Elliot Berman --- drivers/virt/gunyah/Makefile | 2 +- drivers/virt/gunyah/rsc_mgr.h | 36 +++++ drivers/virt/gunyah/rsc_mgr_rpc.c | 238 ++++++++++++++++++++++++++++++ include/linux/gunyah_rsc_mgr.h | 55 +++++++ 4 files changed, 330 insertions(+), 1 deletion(-) create mode 100644 drivers/virt/gunyah/rsc_mgr_rpc.c diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile index cc864ff5abbb..de29769f2f3f 100644 --- a/drivers/virt/gunyah/Makefile +++ b/drivers/virt/gunyah/Makefile @@ -2,5 +2,5 @@ obj-$(CONFIG_GUNYAH) += gunyah.o -gunyah_rsc_mgr-y += rsc_mgr.o +gunyah_rsc_mgr-y += rsc_mgr.o rsc_mgr_rpc.o obj-$(CONFIG_GUNYAH) += gunyah_rsc_mgr.o diff --git a/drivers/virt/gunyah/rsc_mgr.h b/drivers/virt/gunyah/rsc_mgr.h index 824749e63a54..2f12f31a2ea6 100644 --- a/drivers/virt/gunyah/rsc_mgr.h +++ b/drivers/virt/gunyah/rsc_mgr.h @@ -68,4 +68,40 @@ struct gh_rm; int gh_rm_call(struct gh_rm *rsc_mgr, u32 message_id, void *req_buff, size_t req_buff_size, void **resp_buf, size_t *resp_buff_size); +/* Message IDs: VM Management */ +#define GH_RM_RPC_VM_ALLOC_VMID 0x56000001 +#define GH_RM_RPC_VM_DEALLOC_VMID 0x56000002 +#define GH_RM_RPC_VM_START 0x56000004 +#define GH_RM_RPC_VM_STOP 0x56000005 +#define GH_RM_RPC_VM_CONFIG_IMAGE 0x56000009 +#define GH_RM_RPC_VM_INIT 0x5600000B +#define GH_RM_RPC_VM_GET_HYP_RESOURCES 0x56000020 +#define GH_RM_RPC_VM_GET_VMID 0x56000024 + +struct gh_vm_common_vmid_req { + __le16 vmid; + __le16 reserved0; +} __packed; + +/* Call: VM_STOP */ +struct gh_vm_stop_req { + __le16 vmid; + u8 flags; /* currently not used */ + u8 reserved; + __le32 stop_reason; /* currently not used */ +} __packed; + +/* Call: VM_CONFIG_IMAGE */ +struct gh_vm_config_image_req { + __le16 vmid; + __le16 auth_mech; + __le32 mem_handle; + __le64 image_offset; + __le64 image_size; + __le64 dtb_offset; + __le64 dtb_size; +} __packed; + +/* Call: GET_HYP_RESOURCES */ + #endif diff --git a/drivers/virt/gunyah/rsc_mgr_rpc.c b/drivers/virt/gunyah/rsc_mgr_rpc.c new file mode 100644 index 000000000000..b6935dfac1fe --- /dev/null +++ b/drivers/virt/gunyah/rsc_mgr_rpc.c @@ -0,0 +1,238 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#include + +#include "rsc_mgr.h" + +/* + * Several RM calls take only a VMID as a parameter and give only standard + * response back. Deduplicate boilerplate code by using this common call. + */ +static int gh_rm_common_vmid_call(struct gh_rm *rm, u32 message_id, u16 vmid) +{ + void *resp; + struct gh_vm_common_vmid_req req_payload = { + .vmid = cpu_to_le16(vmid), + }; + size_t resp_size; + int ret; + + ret = gh_rm_call(rm, message_id, &req_payload, sizeof(req_payload), &resp, &resp_size); + if (!ret && resp_size) { + pr_warn("Unexpected payload size: %ld Expected: 0", resp_size); + dump_stack(); + kfree(resp); + return -EBADMSG; + } + + return ret; +} + +/** + * gh_rm_alloc_vmid() - Allocate a new VM in Gunyah. Returns the VM identifier. + * @vmid: Use GH_VMID_INVAL or GH_VMID_SELF (0) to dynamically allocate a VM. A reserved VMID can + * be supplied to request allocation of a platform-defined VM. + * + * Returns - the allocated VMID or negative value on error + */ +int gh_rm_alloc_vmid(struct gh_rm *rm, u16 vmid) +{ + void *resp; + struct gh_vm_common_vmid_req req_payload = { + .vmid = cpu_to_le16(vmid), + }; + struct gh_vm_alloc_vmid_resp *resp_payload; + size_t resp_size; + int ret; + + if (vmid == GH_VMID_INVAL) + vmid = 0; + + ret = gh_rm_call(rm, GH_RM_RPC_VM_ALLOC_VMID, &req_payload, sizeof(req_payload), &resp, + &resp_size); + if (ret) + return ret; + + if (!vmid) { + if (resp_size != sizeof(*resp_payload)) { + pr_warn("%s: unexpected payload size: %ld Expected: %ld", __func__, + resp_size, sizeof(*resp_payload)); + ret = -EBADMSG; + } else { + resp_payload = resp; + ret = le16_to_cpu(resp_payload->vmid); + } + } + kfree(resp); + + return ret; +} +EXPORT_SYMBOL_GPL(gh_rm_alloc_vmid); + +/** + * gh_rm_dealloc_vmid() - Dispose the VMID + * @vmid: VM identifier + */ +int gh_rm_dealloc_vmid(struct gh_rm *rm, u16 vmid) +{ + return gh_rm_common_vmid_call(rm, GH_RM_RPC_VM_DEALLOC_VMID, vmid); +} +EXPORT_SYMBOL_GPL(gh_rm_dealloc_vmid); + +/** + * gh_rm_vm_start() - Move the VM into "ready to run" state + * @vmid: VM identifier + * + * On VMs which use proxy scheduling, vcpu_run is needed to actually run the VM. + * On VMs which use Gunyah's scheduling, the vCPUs start executing in accordance with Gunyah + * scheduling policies. + */ +int gh_rm_vm_start(struct gh_rm *rm, u16 vmid) +{ + return gh_rm_common_vmid_call(rm, GH_RM_RPC_VM_START, vmid); +} +EXPORT_SYMBOL_GPL(gh_rm_vm_start); + +/** + * gh_rm_vm_stop() - Send a request to Resource Manager VM to stop a VM. + * @vmid: VM identifier + * + * Returns - 0 on success; negative value on failure + */ +int gh_rm_vm_stop(struct gh_rm *rm, u16 vmid) +{ + struct gh_vm_stop_req req_payload = { + .vmid = cpu_to_le16(vmid), + }; + void *resp; + size_t resp_size; + int ret; + + ret = gh_rm_call(rm, GH_RM_RPC_VM_STOP, &req_payload, sizeof(req_payload), + &resp, &resp_size); + if (!ret && resp_size) { + pr_warn("%s: unexpected payload size: %ld Expected: 0", __func__, resp_size); + kfree(resp); + return -EBADMSG; + } + + return ret; +} +EXPORT_SYMBOL_GPL(gh_rm_vm_stop); + +int gh_rm_vm_configure(struct gh_rm *rm, u16 vmid, enum gh_rm_vm_auth_mechanism auth_mechanism, + u32 mem_handle, u64 image_offset, u64 image_size, u64 dtb_offset, u64 dtb_size) +{ + struct gh_vm_config_image_req req_payload = { 0 }; + void *resp; + size_t resp_size; + int ret; + + req_payload.vmid = cpu_to_le16(vmid); + req_payload.auth_mech = cpu_to_le16(auth_mechanism); + req_payload.mem_handle = cpu_to_le32(mem_handle); + req_payload.image_offset = cpu_to_le64(image_offset); + req_payload.image_size = cpu_to_le64(image_size); + req_payload.dtb_offset = cpu_to_le64(dtb_offset); + req_payload.dtb_size = cpu_to_le64(dtb_size); + + ret = gh_rm_call(rm, GH_RM_RPC_VM_CONFIG_IMAGE, &req_payload, sizeof(req_payload), + &resp, &resp_size); + if (!ret && resp_size) { + pr_warn("%s: unexpected payload size: %ld Expected: 0", __func__, resp_size); + kfree(resp); + return -EBADMSG; + } + + return ret; +} +EXPORT_SYMBOL_GPL(gh_rm_vm_configure); + +/** + * gh_rm_vm_init() - Move the VM to initialized state. + * @vmid: VM identifier + * + * RM will allocate needed resources for the VM. After gh_rm_vm_init, gh_rm_get_hyp_resources() + * can be called to learn of the capabilities we can use with the new VM. + * + * Returns - 0 on success; negative value on failure + */ +int gh_rm_vm_init(struct gh_rm *rm, u16 vmid) +{ + return gh_rm_common_vmid_call(rm, GH_RM_RPC_VM_INIT, vmid); +} +EXPORT_SYMBOL_GPL(gh_rm_vm_init); + +/** + * gh_rm_get_hyp_resources() - Retrieve hypervisor resources (capabilities) associated with a VM + * @vmid: VMID of the other VM to get the resources of + * @resources: Set by gh_rm_get_hyp_resources and contains the returned hypervisor resources. + * + * Returns - 0 on success; negative value on failure + */ +int gh_rm_get_hyp_resources(struct gh_rm *rm, u16 vmid, + struct gh_rm_hyp_resources **resources) +{ + struct gh_rm_hyp_resources *resp; + size_t resp_size; + int ret; + struct gh_vm_common_vmid_req req_payload = { + .vmid = cpu_to_le16(vmid), + }; + + ret = gh_rm_call(rm, GH_RM_RPC_VM_GET_HYP_RESOURCES, + &req_payload, sizeof(req_payload), + (void **)&resp, &resp_size); + if (ret) + return ret; + + if (!resp_size) + return -EBADMSG; + + if (resp_size < struct_size(resp, entries, 0) || + resp_size != struct_size(resp, entries, le32_to_cpu(resp->n_entries))) { + kfree(resp); + return -EBADMSG; + } + + *resources = resp; + return 0; +} +EXPORT_SYMBOL_GPL(gh_rm_get_hyp_resources); + +/** + * gh_rm_get_vmid() - Retrieve VMID of this virtual machine + * @vmid: Filled with the VMID of this VM + */ +int gh_rm_get_vmid(struct gh_rm *rm, u16 *vmid) +{ + static u16 cached_vmid = GH_VMID_INVAL; + __le16 *resp; + size_t resp_size; + int ret; + + if (cached_vmid != GH_VMID_INVAL) { + *vmid = cached_vmid; + return 0; + } + + ret = gh_rm_call(rm, GH_RM_RPC_VM_GET_VMID, NULL, 0, (void **)&resp, &resp_size); + if (ret) + return ret; + + if (resp_size != sizeof(*resp)) { + pr_warn("%s: unexpected payload size: %ld Expected: %ld", __func__, + resp_size, sizeof(*resp)); + ret = -EBADMSG; + goto out; + } + + *vmid = cached_vmid = le16_to_cpu(*resp); +out: + kfree(resp); + return ret; +} +EXPORT_SYMBOL_GPL(gh_rm_get_vmid); diff --git a/include/linux/gunyah_rsc_mgr.h b/include/linux/gunyah_rsc_mgr.h index 959787ad0a3d..be0bce5507b1 100644 --- a/include/linux/gunyah_rsc_mgr.h +++ b/include/linux/gunyah_rsc_mgr.h @@ -15,4 +15,59 @@ /* Gunyah recognizes VMID0 as an alias to the current VM's ID */ #define GH_VMID_SELF 0 +struct gh_rm; + +enum gh_rm_vm_status { + GH_RM_VM_STATUS_NO_STATE = 0, + GH_RM_VM_STATUS_INIT = 1, + GH_RM_VM_STATUS_READY = 2, + GH_RM_VM_STATUS_RUNNING = 3, + GH_RM_VM_STATUS_PAUSED = 4, + GH_RM_VM_STATUS_LOAD = 5, + GH_RM_VM_STATUS_AUTH = 6, + GH_RM_VM_STATUS_INIT_FAILED = 8, + GH_RM_VM_STATUS_EXITED = 9, + GH_RM_VM_STATUS_RESETTING = 10, + GH_RM_VM_STATUS_RESET = 11, +}; + +/* RPC Calls */ +int gh_rm_alloc_vmid(struct gh_rm_rpc *rm, u16 vmid); +int gh_rm_dealloc_vmid(struct gh_rm_rpc *rm, u16 vmid); +int gh_rm_vm_start(struct gh_rm_rpc *rm, u16 vmid); +int gh_rm_vm_stop(struct gh_rm_rpc *rm, u16 vmid); + +enum gh_rm_vm_auth_mechanism { + GH_RM_VM_AUTH_NONE = 0, + GH_RM_VM_AUTH_QCOM_PIL_ELF = 1, + GH_RM_VM_AUTH_QCOM_ANDROID_PVM = 2, +}; + +int gh_rm_vm_configure(struct gh_rm *rm, u16 vmid, enum gh_rm_vm_auth_mechanism auth_mechanism, + u32 mem_handle, u64 image_offset, u64 image_size, + u64 dtb_offset, u64 dtb_size); +int gh_rm_vm_init(struct gh_rm *rm, u16 vmid); + +struct gh_rm_hyp_resource { + u8 type; + u8 reserved; + __le16 partner_vmid; + __le32 resource_handle; + __le32 resource_label; + __le64 cap_id; + __le32 virq_handle; + __le32 virq; + __le64 base; + __le64 size; +} __packed; + +struct gh_rm_hyp_resources { + __le32 n_entries; + struct gh_rm_hyp_resource entries[]; +} __packed; + +int gh_rm_get_hyp_resources(struct gh_rm *rm, u16 vmid, + struct gh_rm_hyp_resources **resources); +int gh_rm_get_vmid(struct gh_rm *rm, u16 *vmid); + #endif From patchwork Fri Jan 20 22:46:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 644680 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CD4AC52D11 for ; Fri, 20 Jan 2023 22:48:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230078AbjATWsp (ORCPT ); Fri, 20 Jan 2023 17:48:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229929AbjATWsb (ORCPT ); Fri, 20 Jan 2023 17:48:31 -0500 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6752863E12; Fri, 20 Jan 2023 14:48:08 -0800 (PST) Received: from pps.filterd (m0279864.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KMc40h023107; Fri, 20 Jan 2023 22:47:47 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=tS0TNGM1FTZCkg//mzom5T+cH/vZ8HjSj+xPMrvN+is=; b=iog6J3vh5PzNKDfF1ZjzB+G2ZBYEhfRnaU0Y1CIKztUgu2EfoD3uWVbLtje5dbDBeuf6 N6MKKV0d4FghmVSSZb04bbUrJBBU9E4sRsB4op6kLyIJw6v8Xp+kNRqLbiuK2G16q6dx NATTGTYdD+gWE+V1rCOLTZewi6kLdakjr7pqOYSPmUqqmk0gjnwk07CH7i4YSAxDE6Dc CMRyn5hjA2p9LCLbjS/jf4Iu+qaQ4y3YQrRwlqgysOERaU0dzzWP1ykihi/S+gRm0dyW ahAi6OkbQZqfJ8D+30NPhJvMJiinp7Gxvjs1a0/IiYtTYXS4lkngIFWAc5M1drsfAhcj vg== Received: from nasanppmta04.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7q859eb3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:47 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA04.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMlkkx022279 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:46 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:45 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Jonathan Corbet , Elliot Berman , Murali Nalajala CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Bagas Sanjaya , Catalin Marinas , "Will Deacon" , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 11/27] gunyah: vm_mgr: Introduce basic VM Manager Date: Fri, 20 Jan 2023 14:46:10 -0800 Message-ID: <20230120224627.4053418-12-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: ORopLD2ENlIml2XjaaGh75_sJQfNjIYO X-Proofpoint-GUID: ORopLD2ENlIml2XjaaGh75_sJQfNjIYO X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 suspectscore=0 spamscore=0 malwarescore=0 priorityscore=1501 clxscore=1015 adultscore=0 phishscore=0 lowpriorityscore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Gunyah VM manager is a kernel moduel which exposes an interface to Gunyah userspace to load, run, and interact with other Gunyah virtual machines. The interface is a character device at /dev/gunyah. Add a basic VM manager driver. Upcoming patches will add more ioctls into this driver. Co-developed-by: Prakruthi Deepak Heragu Signed-off-by: Prakruthi Deepak Heragu Signed-off-by: Elliot Berman --- .../userspace-api/ioctl/ioctl-number.rst | 1 + drivers/virt/gunyah/Makefile | 2 +- drivers/virt/gunyah/rsc_mgr.c | 51 +++++++- drivers/virt/gunyah/vm_mgr.c | 117 ++++++++++++++++++ drivers/virt/gunyah/vm_mgr.h | 20 +++ include/linux/gunyah_rsc_mgr.h | 3 + include/uapi/linux/gunyah.h | 23 ++++ 7 files changed, 215 insertions(+), 2 deletions(-) create mode 100644 drivers/virt/gunyah/vm_mgr.c create mode 100644 drivers/virt/gunyah/vm_mgr.h create mode 100644 include/uapi/linux/gunyah.h diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst index eb045fc495a4..8696dc3cdd83 100644 --- a/Documentation/userspace-api/ioctl/ioctl-number.rst +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst @@ -137,6 +137,7 @@ Code Seq# Include File Comments 'F' DD video/sstfb.h conflict! 'G' 00-3F drivers/misc/sgi-gru/grulib.h conflict! 'G' 00-0F xen/gntalloc.h, xen/gntdev.h conflict! +'G' 00-0f linux/gunyah.h conflict! 'H' 00-7F linux/hiddev.h conflict! 'H' 00-0F linux/hidraw.h conflict! 'H' 01 linux/mei.h conflict! diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile index de29769f2f3f..03951cf82023 100644 --- a/drivers/virt/gunyah/Makefile +++ b/drivers/virt/gunyah/Makefile @@ -2,5 +2,5 @@ obj-$(CONFIG_GUNYAH) += gunyah.o -gunyah_rsc_mgr-y += rsc_mgr.o rsc_mgr_rpc.o +gunyah_rsc_mgr-y += rsc_mgr.o rsc_mgr_rpc.o vm_mgr.o obj-$(CONFIG_GUNYAH) += gunyah_rsc_mgr.o diff --git a/drivers/virt/gunyah/rsc_mgr.c b/drivers/virt/gunyah/rsc_mgr.c index 382f9943fd31..95f4aa928aaf 100644 --- a/drivers/virt/gunyah/rsc_mgr.c +++ b/drivers/virt/gunyah/rsc_mgr.c @@ -16,8 +16,10 @@ #include #include #include +#include #include "rsc_mgr.h" +#include "vm_mgr.h" #define RM_RPC_API_VERSION_MASK GENMASK(3, 0) #define RM_RPC_HEADER_WORDS_MASK GENMASK(7, 4) @@ -105,6 +107,7 @@ struct gh_rm { struct kmem_cache *cache; struct mutex send_lock; + struct miscdevice miscdev; struct work_struct recv_work; }; @@ -526,6 +529,33 @@ int gh_rm_call(struct gh_rm *rm, u32 message_id, void *req_buff, size_t req_buff return ret; } +void get_gh_rm(struct gh_rm *rm) +{ + get_device(rm->dev); +} +EXPORT_SYMBOL_GPL(get_gh_rm); + +void put_gh_rm(struct gh_rm *rm) +{ + put_device(rm->dev); +} +EXPORT_SYMBOL_GPL(put_gh_rm); + +static long gh_dev_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) +{ + struct miscdevice *miscdev = filp->private_data; + struct gh_rm *rm = container_of(miscdev, struct gh_rm, miscdev); + + return gh_dev_vm_mgr_ioctl(rm, cmd, arg); +} + +static const struct file_operations gh_dev_fops = { + .owner = THIS_MODULE, + .unlocked_ioctl = gh_dev_ioctl, + .compat_ioctl = compat_ptr_ioctl, + .llseek = noop_llseek, +}; + static int gh_msgq_platform_probe_direction(struct platform_device *pdev, bool tx, int idx, struct gunyah_resource *ghrsc) { @@ -582,13 +612,32 @@ static int gh_rm_drv_probe(struct platform_device *pdev) rm->msgq_client.rx_callback = gh_rm_msgq_rx_data; rm->msgq_client.tx_done = gh_rm_msgq_tx_done; - return gh_msgq_init(&pdev->dev, &rm->msgq, &rm->msgq_client, &rm->tx_ghrsc, &rm->rx_ghrsc); + ret = gh_msgq_init(&pdev->dev, &rm->msgq, &rm->msgq_client, &rm->tx_ghrsc, &rm->rx_ghrsc); + if (ret) + goto err_cache; + + rm->miscdev.name = "gunyah"; + rm->miscdev.minor = MISC_DYNAMIC_MINOR; + rm->miscdev.fops = &gh_dev_fops; + + ret = misc_register(&rm->miscdev); + if (ret) + goto err_msgq; + + return 0; +err_msgq: + mbox_free_channel(gh_msgq_chan(&rm->msgq)); + gh_msgq_remove(&rm->msgq); +err_cache: + kmem_cache_destroy(rm->cache); + return ret; } static int gh_rm_drv_remove(struct platform_device *pdev) { struct gh_rm *rm = platform_get_drvdata(pdev); + misc_deregister(&rm->miscdev); mbox_free_channel(gh_msgq_chan(&rm->msgq)); gh_msgq_remove(&rm->msgq); kmem_cache_destroy(rm->cache); diff --git a/drivers/virt/gunyah/vm_mgr.c b/drivers/virt/gunyah/vm_mgr.c new file mode 100644 index 000000000000..0864dbd77e28 --- /dev/null +++ b/drivers/virt/gunyah/vm_mgr.c @@ -0,0 +1,117 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#define pr_fmt(fmt) "gh_vm_mgr: " fmt + +#include +#include +#include +#include +#include + +#include + +#include "vm_mgr.h" + +static __must_check struct gunyah_vm *gunyah_vm_alloc(struct gh_rm_rpc *rm) +{ + struct gunyah_vm *ghvm; + int vmid; + + vmid = gh_rm_alloc_vmid(rm, 0); + if (vmid < 0) + return ERR_PTR(vmid); + + ghvm = kzalloc(sizeof(*ghvm), GFP_KERNEL); + if (!ghvm) { + gh_rm_dealloc_vmid(rm, vmid); + return ghvm; + } + + get_gh_rm(rm); + + ghvm->vmid = vmid; + ghvm->rm = rm; + + return ghvm; +} + +static long gh_vm_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) +{ + long r; + + switch (cmd) { + default: + r = -ENOTTY; + break; + } + + return r; +} + +static int gh_vm_release(struct inode *inode, struct file *filp) +{ + struct gunyah_vm *ghvm = filp->private_data; + + put_gh_rm(ghvm->rm); + kfree(ghvm); + return 0; +} + +static const struct file_operations gh_vm_fops = { + .unlocked_ioctl = gh_vm_ioctl, + .release = gh_vm_release, + .compat_ioctl = compat_ptr_ioctl, + .llseek = noop_llseek, +}; + +static long gh_dev_ioctl_create_vm(struct gh_rm *rm, unsigned long arg) +{ + struct gunyah_vm *ghvm; + struct file *file; + int fd, err; + + /* arg reserved for future use. */ + if (arg) + return -EINVAL; + + ghvm = gunyah_vm_alloc(rm); + if (IS_ERR_OR_NULL(ghvm)) + return PTR_ERR(ghvm) ? : -ENOMEM; + + fd = get_unused_fd_flags(O_CLOEXEC); + if (fd < 0) { + err = fd; + goto err_destroy_vm; + } + + file = anon_inode_getfile("gunyah-vm", &gh_vm_fops, ghvm, O_RDWR); + if (IS_ERR(file)) { + err = PTR_ERR(file); + goto err_put_fd; + } + + fd_install(fd, file); + + return fd; + +err_put_fd: + put_unused_fd(fd); +err_destroy_vm: + kfree(ghvm); + return err; +} + +long gh_dev_vm_mgr_ioctl(struct gh_rm *rm, unsigned int cmd, unsigned long arg) +{ + switch (cmd) { + case GH_CREATE_VM: + return gh_dev_ioctl_create_vm(rm, arg); + default: + return -ENOIOCTLCMD; + } +} +EXPORT_SYMBOL_GPL(gh_dev_vm_mgr_ioctl); + diff --git a/drivers/virt/gunyah/vm_mgr.h b/drivers/virt/gunyah/vm_mgr.h new file mode 100644 index 000000000000..e47f34de7f9e --- /dev/null +++ b/drivers/virt/gunyah/vm_mgr.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#ifndef _GH_PRIV_VM_MGR_H +#define _GH_PRIV_VM_MGR_H + +#include + +#include + +long gh_dev_vm_mgr_ioctl(struct gh_rm *rm, unsigned int cmd, unsigned long arg); + +struct gunyah_vm { + u16 vmid; + struct gh_rm_rpc *rm; +}; + +#endif diff --git a/include/linux/gunyah_rsc_mgr.h b/include/linux/gunyah_rsc_mgr.h index be0bce5507b1..9a9a037b349a 100644 --- a/include/linux/gunyah_rsc_mgr.h +++ b/include/linux/gunyah_rsc_mgr.h @@ -17,6 +17,9 @@ struct gh_rm; +void get_gh_rm(struct gh_rm *rm); +void put_gh_rm(struct gh_rm *rm); + enum gh_rm_vm_status { GH_RM_VM_STATUS_NO_STATE = 0, GH_RM_VM_STATUS_INIT = 1, diff --git a/include/uapi/linux/gunyah.h b/include/uapi/linux/gunyah.h new file mode 100644 index 000000000000..88a40d6e0b96 --- /dev/null +++ b/include/uapi/linux/gunyah.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0-only WITH Linux-syscall-note */ +/* + * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#ifndef _UAPI_LINUX_GUNYAH +#define _UAPI_LINUX_GUNYAH + +/* + * Userspace interface for /dev/gunyah - gunyah based virtual machine + */ + +#include +#include + +#define GH_IOCTL_TYPE 'G' + +/* + * ioctls for /dev/gunyah fds: + */ +#define GH_CREATE_VM _IO(GH_IOCTL_TYPE, 0x0) /* Returns a Gunyah VM fd */ + +#endif From patchwork Fri Jan 20 22:46:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 644676 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2896C54737 for ; Fri, 20 Jan 2023 22:49:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229924AbjATWte (ORCPT ); Fri, 20 Jan 2023 17:49:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229963AbjATWt0 (ORCPT ); Fri, 20 Jan 2023 17:49:26 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B8696779E; Fri, 20 Jan 2023 14:48:49 -0800 (PST) Received: from pps.filterd (m0279872.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KMe2fF029889; Fri, 20 Jan 2023 22:47:49 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=YP+69sXYD0uhnHJf+coVLGSsBBt1XgaAsipO/XYv0PU=; b=c0iQVqbcPNBuJJZboD2pXHbq53CxvZZdGgr/tOJZ9/yxrK/Z8tV3LF0Q0oIHVmN2ywWb a5RqrOEGVUvFTt/wQzBC3RhVma/51oZs8JtW9C9lC1VYrOtPX6znafQgvmlScijbubAX KlWULmD4Yzk4vhXZgl7vLwwJyvTjQ+DA0/0Hgw4YsomSemM93DwE7CuwO0KCv6B3XDS5 vwnRLParG3u1/hGwiGACTjZjoNMP0BKAXQkHKRKsEKKXcBkoEelswIrUtjePslzJE8Xc 4ccuoa/CH9eNpJVlD5JWE72IjtCS4+0G0Spri2EH1pHhF3GUrbOSW78UaY9z8svEeT2a XQ== Received: from nasanppmta04.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7xm40k8y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:49 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA04.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMlmq8022649 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:48 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:47 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Will Deacon , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 12/27] gunyah: rsc_mgr: Add RPC for sharing memory Date: Fri, 20 Jan 2023 14:46:11 -0800 Message-ID: <20230120224627.4053418-13-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: h8-LVh-TZkr7OIpKs_1DtbMtZ0mrjCt6 X-Proofpoint-GUID: h8-LVh-TZkr7OIpKs_1DtbMtZ0mrjCt6 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 malwarescore=0 spamscore=0 bulkscore=0 clxscore=1015 priorityscore=1501 mlxlogscore=999 impostorscore=0 mlxscore=0 suspectscore=0 adultscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Gunyah resource manager provides API to manipulate stage 2 page tables. Manipulations are represented as a memory parcel. Memory parcels describe a list of memory regions (intermediate physical address and size), a list of new permissions for VMs, and the memory type (DDR or MMIO). Memory parcels are uniquely identified by a handle allocated by Gunyah. There are a few types of memory parcel sharing which Gunyah supports: - Sharing: the guest and host VM both have access - Lending: only the guest has access; host VM loses access - Donating: Permanently lent (not reclaimed even if guest shuts down) Memory parcels that have been shared or lent can be reclaimed by the host via an additional call. The reclaim operation restores the original access the host VM had to the memory parcel and removes the access to other VM. One point to note that memory parcels don't describe where in the guest VM the memory parcel should reside. The guest VM must accept the memory parcel either explicitly via a "gh_rm_mem_accept" call (not introduced here) or be configured to accept it automatically at boot. As the guest VM accepts the memory parcel, it also mentions the IPA it wants to place memory parcel. Co-developed-by: Prakruthi Deepak Heragu Signed-off-by: Prakruthi Deepak Heragu Signed-off-by: Elliot Berman --- drivers/virt/gunyah/rsc_mgr.h | 50 +++++++ drivers/virt/gunyah/rsc_mgr_rpc.c | 216 ++++++++++++++++++++++++++++++ include/linux/gunyah_rsc_mgr.h | 55 +++++++- 3 files changed, 317 insertions(+), 4 deletions(-) diff --git a/drivers/virt/gunyah/rsc_mgr.h b/drivers/virt/gunyah/rsc_mgr.h index 2f12f31a2ea6..17047afefb28 100644 --- a/drivers/virt/gunyah/rsc_mgr.h +++ b/drivers/virt/gunyah/rsc_mgr.h @@ -68,6 +68,12 @@ struct gh_rm; int gh_rm_call(struct gh_rm *rsc_mgr, u32 message_id, void *req_buff, size_t req_buff_size, void **resp_buf, size_t *resp_buff_size); +/* Message IDs: Memory Management */ +#define GH_RM_RPC_MEM_LEND 0x51000012 +#define GH_RM_RPC_MEM_SHARE 0x51000013 +#define GH_RM_RPC_MEM_RECLAIM 0x51000015 +#define GH_RM_RPC_MEM_APPEND 0x51000018 + /* Message IDs: VM Management */ #define GH_RM_RPC_VM_ALLOC_VMID 0x56000001 #define GH_RM_RPC_VM_DEALLOC_VMID 0x56000002 @@ -83,6 +89,50 @@ struct gh_vm_common_vmid_req { __le16 reserved0; } __packed; +/* Call: MEM_LEND, MEM_SHARE */ +struct gh_mem_share_req_header { + u8 mem_type; + u8 reserved0; +#define GH_MEM_SHARE_REQ_FLAGS_APPEND BIT(1) + u8 flags; + u8 reserved1; + __le32 label; +} __packed; + +struct gh_mem_share_req_acl_section { + __le32 n_entries; + struct gh_rm_mem_acl_entry entries[]; +}; + +struct gh_mem_share_req_mem_section { + __le16 n_entries; + __le16 reserved0; + struct gh_rm_mem_entry entries[]; +}; + +/* Call: MEM_RELEASE */ +struct gh_mem_release_req { + __le32 mem_handle; + u8 flags; /* currently not used */ + __le16 reserved0; + u8 reserved1; +} __packed; + +/* Call: MEM_APPEND */ +struct gh_mem_append_req_header { + __le32 mem_handle; +#define GH_MEM_APPEND_REQ_FLAGS_END BIT(0) + u8 flags; + __le16 reserved0; + u8 reserved1; +} __packed; + +/* Call: VM_ALLOC */ +struct gh_vm_alloc_vmid_resp { + __le16 vmid; + __le16 reserved0; +} __packed; + /* Call: VM_STOP */ struct gh_vm_stop_req { __le16 vmid; diff --git a/drivers/virt/gunyah/rsc_mgr_rpc.c b/drivers/virt/gunyah/rsc_mgr_rpc.c index b6935dfac1fe..a3633df98a38 100644 --- a/drivers/virt/gunyah/rsc_mgr_rpc.c +++ b/drivers/virt/gunyah/rsc_mgr_rpc.c @@ -7,6 +7,8 @@ #include "rsc_mgr.h" +#define GH_RM_MAX_MEM_ENTRIES 512 + /* * Several RM calls take only a VMID as a parameter and give only standard * response back. Deduplicate boilerplate code by using this common call. @@ -31,6 +33,220 @@ static int gh_rm_common_vmid_call(struct gh_rm *rm, u32 message_id, u16 vmid) return ret; } +static int _gh_rm_mem_append(struct gh_rm *rm, u32 mem_handle, bool end_append, + struct gh_rm_mem_entry *mem_entries, size_t n_mem_entries) +{ + size_t msg_size = 0; + void *msg; + struct gh_mem_append_req_header *req_header; + struct gh_mem_share_req_mem_section *mem_section; + void *resp; + size_t resp_size; + int ret; + + msg_size += sizeof(struct gh_mem_append_req_header); + msg_size += offsetof(struct gh_mem_share_req_mem_section, entries[n_mem_entries]); + + msg = kzalloc(msg_size, GFP_KERNEL); + if (!msg) + return -ENOMEM; + + req_header = msg; + mem_section = (void *)req_header + sizeof(struct gh_mem_append_req_header); + + req_header->mem_handle = cpu_to_le32(mem_handle); + if (end_append) + req_header->flags |= GH_MEM_APPEND_REQ_FLAGS_END; + + mem_section->n_entries = cpu_to_le16(n_mem_entries); + memcpy(mem_section->entries, mem_entries, sizeof(*mem_entries) * n_mem_entries); + + ret = gh_rm_call(rm, GH_RM_RPC_MEM_APPEND, msg, msg_size, &resp, &resp_size); + kfree(msg); + if (!ret && resp_size) { + pr_warn("%s: unexpected payload size: %ld Expected: 0", __func__, resp_size); + kfree(resp); + ret = -EBADMSG; + } + + return ret; +} + +static int gh_rm_mem_append(struct gh_rm *rm, u32 mem_handle, + struct gh_rm_mem_entry *mem_entries, size_t n_mem_entries) +{ + bool end_append; + size_t n; + int ret = 0; + + while (n_mem_entries) { + if (n_mem_entries > GH_RM_MAX_MEM_ENTRIES) { + end_append = false; + n = GH_RM_MAX_MEM_ENTRIES; + } else { + end_append = true; + n = n_mem_entries; + } + + ret = _gh_rm_mem_append(rm, mem_handle, end_append, mem_entries, n); + if (ret) + break; + + mem_entries += n; + n_mem_entries -= n; + } + + return ret; +} + +static int gh_rm_mem_lend_common(struct gh_rm *rm, u32 message_id, struct gh_rm_mem_parcel *p) +{ + size_t msg_size = 0, initial_n_mem_entries = p->n_mem_entries; + void *msg; + __le32 *resp; + struct gh_mem_share_req_header *req_header; + struct gh_mem_share_req_acl_section *acl_section; + struct gh_mem_share_req_mem_section *mem_section; + u32 *mem_attr_section; + size_t resp_size; + int ret; + + if (!p->acl_entries || !p->n_acl_entries || !p->mem_entries || !p->n_mem_entries || + p->n_acl_entries > U8_MAX || p->mem_handle != GH_MEM_HANDLE_INVAL) + return -EINVAL; + + if (initial_n_mem_entries > GH_RM_MAX_MEM_ENTRIES) + initial_n_mem_entries = GH_RM_MAX_MEM_ENTRIES; + + /* The format of the message goes: + * request header + * ACL entries (which VMs get what kind of access to this memory parcel) + * Memory entries (list of memory regions to share) + * Memory attributes (currently unused, we'll hard-code the size to 0) + */ + msg_size += sizeof(struct gh_mem_share_req_header); + msg_size += offsetof(struct gh_mem_share_req_acl_section, entries[p->n_acl_entries]); + msg_size += offsetof(struct gh_mem_share_req_mem_section, entries[initial_n_mem_entries]); + msg_size += sizeof(u32); /* for memory attributes, currently unused */ + + msg = kzalloc(msg_size, GFP_KERNEL); + if (!msg) + return -ENOMEM; + + ret = gh_rm_platform_pre_mem_share(rm, p); + if (ret) { + kfree(msg); + return ret; + } + + req_header = msg; + acl_section = (void *)req_header + sizeof(*req_header); + mem_section = (void *)acl_section + offsetof(struct gh_mem_share_req_acl_section, + entries[p->n_acl_entries]); + mem_attr_section = (void *)mem_section + offsetof(struct gh_mem_share_req_mem_section, + entries[initial_n_mem_entries]); + + req_header->mem_type = p->mem_type; + if (initial_n_mem_entries != p->n_mem_entries) + req_header->flags |= GH_MEM_SHARE_REQ_FLAGS_APPEND; + req_header->label = cpu_to_le32(p->label); + + acl_section->n_entries = cpu_to_le32(p->n_acl_entries); + memcpy(acl_section->entries, p->acl_entries, sizeof(*(p->acl_entries)) * p->n_acl_entries); + + mem_section->n_entries = cpu_to_le16(initial_n_mem_entries); + memcpy(mem_section->entries, p->mem_entries, + sizeof(*(p->mem_entries)) * initial_n_mem_entries); + + /* Set n_entries for memory attribute section to 0 */ + *mem_attr_section = 0; + + ret = gh_rm_call(rm, message_id, msg, msg_size, (void **)&resp, &resp_size); + kfree(msg); + if (ret) { + gh_rm_platform_post_mem_reclaim(rm, p); + goto out; + } + + if (resp_size != sizeof(__le32)) { + ret = -EIO; + goto out; + } + + p->mem_handle = le32_to_cpu(*resp); + + if (initial_n_mem_entries != p->n_mem_entries) { + ret = gh_rm_mem_append(rm, p->mem_handle, + &p->mem_entries[initial_n_mem_entries], + p->n_mem_entries - initial_n_mem_entries); + if (ret) { + gh_rm_mem_reclaim(rm, p); + p->mem_handle = GH_MEM_HANDLE_INVAL; + } + } + +out: + kfree(resp); + return ret; +} + +/** + * gh_rm_mem_lend() - Lend memory to other virtual machines. + * @parcel: Package the memory information of the memory to be lent. + * + * Returns - 0 on success; negative value on failure + * + * Lending removes Linux's access to the memory while the memory parcel is lent. + */ +int gh_rm_mem_lend(struct gh_rm *rm, struct gh_rm_mem_parcel *parcel) +{ + return gh_rm_mem_lend_common(rm, GH_RM_RPC_MEM_LEND, parcel); +} +EXPORT_SYMBOL_GPL(gh_rm_mem_lend); + + +/** + * gh_rm_mem_share() - Share memory with other virtual machines. + * @parcel: Package the memory information of the memory to be shared. + * + * Returns - 0 on success; negative value on failure + * + * Sharing keeps Linux's access to the memory while the memory parcel is shared. + */ +int gh_rm_mem_share(struct gh_rm *rm, struct gh_rm_mem_parcel *parcel) +{ + return gh_rm_mem_lend_common(rm, GH_RM_RPC_MEM_SHARE, parcel); +} +EXPORT_SYMBOL_GPL(gh_rm_mem_share); + +/** + * gh_rm_mem_reclaim() - Reclaim a memory parcel + * @parcel: Package the memory information of the memory to be reclaimed. + * + * Returns - 0 on success; negative value on failure + * + * RM maps the associated memory back into the stage-2 page tables of the owner VM. + */ +int gh_rm_mem_reclaim(struct gh_rm *rm, struct gh_rm_mem_parcel *parcel) +{ + struct gh_mem_release_req req = { + .mem_handle = cpu_to_le32(parcel->mem_handle), + }; + size_t resp_size; + void *resp; + int ret; + + ret = gh_rm_call(rm, GH_RM_RPC_MEM_RECLAIM, &req, sizeof(req), &resp, &resp_size); + if (!ret && resp_size) { + pr_warn("%s: unexpected payload size: %ld Expected: 0", __func__, resp_size); + kfree(resp); + return -EBADMSG; + } + + return ret; +} +EXPORT_SYMBOL_GPL(gh_rm_mem_reclaim); + /** * gh_rm_alloc_vmid() - Allocate a new VM in Gunyah. Returns the VM identifier. * @vmid: Use GH_VMID_INVAL or GH_VMID_SELF (0) to dynamically allocate a VM. A reserved VMID can diff --git a/include/linux/gunyah_rsc_mgr.h b/include/linux/gunyah_rsc_mgr.h index 9a9a037b349a..264f054540b8 100644 --- a/include/linux/gunyah_rsc_mgr.h +++ b/include/linux/gunyah_rsc_mgr.h @@ -11,6 +11,7 @@ #include #define GH_VMID_INVAL U16_MAX +#define GH_MEM_HANDLE_INVAL U32_MAX /* Gunyah recognizes VMID0 as an alias to the current VM's ID */ #define GH_VMID_SELF 0 @@ -34,11 +35,57 @@ enum gh_rm_vm_status { GH_RM_VM_STATUS_RESET = 11, }; +struct gh_rm_mem_acl_entry { + __le16 vmid; +#define GH_RM_ACL_X BIT(0) +#define GH_RM_ACL_W BIT(1) +#define GH_RM_ACL_R BIT(2) + u8 perms; + u8 reserved; +} __packed; + +struct gh_rm_mem_entry { + __le64 ipa_base; + __le64 size; +} __packed; + +enum gh_rm_mem_type { + GH_RM_MEM_TYPE_NORMAL = 0, + GH_RM_MEM_TYPE_IO = 1, +}; + +/* + * struct gh_rm_mem_parcel - Package info about memory to be lent/shared/donated/reclaimed + * @mem_type: The type of memory: normal (DDR) or IO + * @label: An client-specified identifier which can be used by the other VMs to identify the purpose + * of the memory parcel. + * @acl_entries: An array of access control entries. Each entry specifies a VM and what access + * is allowed for the memory parcel. + * @n_acl_entries: Count of the number of entries in the `acl_entries` array. + * @mem_entries: An list of regions to be associated with the memory parcel. Addresses should be + * (intermediate) physical addresses from Linux's perspective. + * @n_mem_entries: Count of the number of entries in the `mem_entries` array. + * @mem_handle: On success, filled with memory handle that RM allocates for this memory parcel + */ +struct gh_rm_mem_parcel { + enum gh_rm_mem_type mem_type; + u32 label; + size_t n_acl_entries; + struct gh_rm_mem_acl_entry *acl_entries; + size_t n_mem_entries; + struct gh_rm_mem_entry *mem_entries; + u32 mem_handle; +}; + /* RPC Calls */ -int gh_rm_alloc_vmid(struct gh_rm_rpc *rm, u16 vmid); -int gh_rm_dealloc_vmid(struct gh_rm_rpc *rm, u16 vmid); -int gh_rm_vm_start(struct gh_rm_rpc *rm, u16 vmid); -int gh_rm_vm_stop(struct gh_rm_rpc *rm, u16 vmid); +int gh_rm_mem_lend(struct gh_rm *rm, struct gh_rm_mem_parcel *parcel); +int gh_rm_mem_share(struct gh_rm *rm, struct gh_rm_mem_parcel *parcel); +int gh_rm_mem_reclaim(struct gh_rm *rm, struct gh_rm_mem_parcel *parcel); + +int gh_rm_alloc_vmid(struct gh_rm *rm, u16 vmid); +int gh_rm_dealloc_vmid(struct gh_rm *rm, u16 vmid); +int gh_rm_vm_start(struct gh_rm *rm, u16 vmid); +int gh_rm_vm_stop(struct gh_rm *rm, u16 vmid); enum gh_rm_vm_auth_mechanism { GH_RM_VM_AUTH_NONE = 0, From patchwork Fri Jan 20 22:46:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 645102 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC66BC52D77 for ; Fri, 20 Jan 2023 22:49:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230131AbjATWtd (ORCPT ); Fri, 20 Jan 2023 17:49:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229964AbjATWtW (ORCPT ); Fri, 20 Jan 2023 17:49:22 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 806D14ED1A; Fri, 20 Jan 2023 14:48:47 -0800 (PST) Received: from pps.filterd (m0279870.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KMVwBn028104; Fri, 20 Jan 2023 22:47:51 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=xDhlof3Ycbpydme9S4Ixj3GcrzqAi/zDbWt6czgF1Ic=; b=aT+gWkQ0uufSXC005XzcziL9qhnHBfWxIxijonlqCwaJ+LQOLM6/aaCbFgA6WT/GBnt5 Hr7HrG1qMF3RS3MQD1qiJCK1TvifiVysTruAbtHDpeREl3Hl4FfJuiOTj8EoRUaQq1Fx F15GdXrIsr1wv53dyPH/P5+LGNA8l/Z2b1woJB/+Mv7wlcGbgHe/ic0/1cmD9Dy4kvnB Q39JSTbsjElw4mkc/QxrLFhCrQbEglvCTEnxNT2FF/MKMvJX/dxg9uQg0ePau+NjC4Ua iBOOBf+tDIY48QZ27Eud0POppmu/kAoZJDYoZzRBLcrelq4o30JJAt8u4oAwaW0/SPI8 ig== Received: from nasanppmta01.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7xt7rjw4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:50 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA01.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMlnWq004033 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:49 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:49 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Will Deacon , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 13/27] gunyah: vm_mgr: Add/remove user memory regions Date: Fri, 20 Jan 2023 14:46:12 -0800 Message-ID: <20230120224627.4053418-14-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: YOf3unQSLCOptKQf9vQVsL8olr1Mdwgw X-Proofpoint-ORIG-GUID: YOf3unQSLCOptKQf9vQVsL8olr1Mdwgw X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 mlxscore=0 suspectscore=0 impostorscore=0 spamscore=0 priorityscore=1501 lowpriorityscore=0 mlxlogscore=999 phishscore=0 bulkscore=0 adultscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org When launching a virtual machine, Gunyah userspace allocates memory for the guest and informs Gunyah about these memory regions through SET_USER_MEMORY_REGION ioctl. Co-developed-by: Prakruthi Deepak Heragu Signed-off-by: Prakruthi Deepak Heragu Signed-off-by: Elliot Berman --- drivers/virt/gunyah/Makefile | 2 +- drivers/virt/gunyah/vm_mgr.c | 46 +++++++ drivers/virt/gunyah/vm_mgr.h | 28 +++- drivers/virt/gunyah/vm_mgr_mm.c | 223 ++++++++++++++++++++++++++++++++ include/uapi/linux/gunyah.h | 22 ++++ 5 files changed, 319 insertions(+), 2 deletions(-) create mode 100644 drivers/virt/gunyah/vm_mgr_mm.c diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile index 03951cf82023..ff8bc4925392 100644 --- a/drivers/virt/gunyah/Makefile +++ b/drivers/virt/gunyah/Makefile @@ -2,5 +2,5 @@ obj-$(CONFIG_GUNYAH) += gunyah.o -gunyah_rsc_mgr-y += rsc_mgr.o rsc_mgr_rpc.o vm_mgr.o +gunyah_rsc_mgr-y += rsc_mgr.o rsc_mgr_rpc.o vm_mgr.o vm_mgr_mm.o obj-$(CONFIG_GUNYAH) += gunyah_rsc_mgr.o diff --git a/drivers/virt/gunyah/vm_mgr.c b/drivers/virt/gunyah/vm_mgr.c index 0864dbd77e28..b847fde63333 100644 --- a/drivers/virt/gunyah/vm_mgr.c +++ b/drivers/virt/gunyah/vm_mgr.c @@ -35,14 +35,55 @@ static __must_check struct gunyah_vm *gunyah_vm_alloc(struct gh_rm_rpc *rm) ghvm->vmid = vmid; ghvm->rm = rm; + mutex_init(&ghvm->mm_lock); + INIT_LIST_HEAD(&ghvm->memory_mappings); + return ghvm; } static long gh_vm_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) { + struct gunyah_vm *ghvm = filp->private_data; + void __user *argp = (void __user *)arg; long r; switch (cmd) { + case GH_VM_SET_USER_MEM_REGION: { + struct gunyah_vm_memory_mapping *mapping; + struct gh_userspace_memory_region region; + + r = -EFAULT; + if (copy_from_user(®ion, argp, sizeof(region))) + break; + + r = -EINVAL; + /* All other flag bits are reserved for future use */ + if (region.flags & ~(GH_MEM_ALLOW_READ | GH_MEM_ALLOW_WRITE | GH_MEM_ALLOW_EXEC | + GH_MEM_LENT)) + break; + + + if (region.memory_size) { + r = 0; + mapping = gh_vm_mem_mapping_alloc(ghvm, ®ion); + if (IS_ERR(mapping)) { + r = PTR_ERR(mapping); + break; + } + } else { + mapping = gh_vm_mem_mapping_find(ghvm, region.label); + if (IS_ERR(mapping)) { + r = PTR_ERR(mapping); + break; + } + r = 0; + if (!mapping) + break; + gh_vm_mem_mapping_reclaim(ghvm, mapping); + kfree(mapping); + } + break; + } default: r = -ENOTTY; break; @@ -54,7 +95,12 @@ static long gh_vm_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) static int gh_vm_release(struct inode *inode, struct file *filp) { struct gunyah_vm *ghvm = filp->private_data; + struct gunyah_vm_memory_mapping *mapping, *tmp; + list_for_each_entry_safe(mapping, tmp, &ghvm->memory_mappings, list) { + gh_vm_mem_mapping_reclaim(ghvm, mapping); + kfree(mapping); + } put_gh_rm(ghvm->rm); kfree(ghvm); return 0; diff --git a/drivers/virt/gunyah/vm_mgr.h b/drivers/virt/gunyah/vm_mgr.h index e47f34de7f9e..6b38bf780f76 100644 --- a/drivers/virt/gunyah/vm_mgr.h +++ b/drivers/virt/gunyah/vm_mgr.h @@ -7,14 +7,40 @@ #define _GH_PRIV_VM_MGR_H #include +#include +#include +#include #include long gh_dev_vm_mgr_ioctl(struct gh_rm *rm, unsigned int cmd, unsigned long arg); +enum gunyah_vm_mem_share_type { + VM_MEM_SHARE, + VM_MEM_LEND, +}; + +struct gunyah_vm_memory_mapping { + struct list_head list; + enum gunyah_vm_mem_share_type share_type; + struct gh_rm_mem_parcel parcel; + + __u64 guest_phys_addr; + struct page **pages; + unsigned long npages; +}; + struct gunyah_vm { u16 vmid; - struct gh_rm_rpc *rm; + struct gh_rm *rm; + + struct mutex mm_lock; + struct list_head memory_mappings; }; +struct gunyah_vm_memory_mapping *gh_vm_mem_mapping_alloc(struct gunyah_vm *ghvm, + struct gh_userspace_memory_region *region); +void gh_vm_mem_mapping_reclaim(struct gunyah_vm *ghvm, struct gunyah_vm_memory_mapping *mapping); +struct gunyah_vm_memory_mapping *gh_vm_mem_mapping_find(struct gunyah_vm *ghvm, u32 label); + #endif diff --git a/drivers/virt/gunyah/vm_mgr_mm.c b/drivers/virt/gunyah/vm_mgr_mm.c new file mode 100644 index 000000000000..f2dbdb4ee8ab --- /dev/null +++ b/drivers/virt/gunyah/vm_mgr_mm.c @@ -0,0 +1,223 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#define pr_fmt(fmt) "gh_vm_mgr: " fmt + +#include +#include + +#include + +#include "vm_mgr.h" + +static inline bool page_contiguous(phys_addr_t p, phys_addr_t t) +{ + return t - p == PAGE_SIZE; +} + +static struct gunyah_vm_memory_mapping *__gh_vm_mem_mapping_find(struct gunyah_vm *ghvm, u32 label) +{ + struct gunyah_vm_memory_mapping *mapping; + + + list_for_each_entry(mapping, &ghvm->memory_mappings, list) + if (mapping->parcel.label == label) + return mapping; + + return NULL; +} + +void gh_vm_mem_mapping_reclaim(struct gunyah_vm *ghvm, struct gunyah_vm_memory_mapping *mapping) +{ + int i, ret = 0; + + if (mapping->parcel.mem_handle != GH_MEM_HANDLE_INVAL) { + ret = gh_rm_mem_reclaim(ghvm->rm, &mapping->parcel); + if (ret) + pr_warn("Failed to reclaim memory parcel for label %d: %d\n", + mapping->parcel.label, ret); + } + + if (!ret) + for (i = 0; i < mapping->npages; i++) + unpin_user_page(mapping->pages[i]); + + kfree(mapping->pages); + kfree(mapping->parcel.acl_entries); + kfree(mapping->parcel.mem_entries); + + mutex_lock(&ghvm->mm_lock); + list_del(&mapping->list); + mutex_unlock(&ghvm->mm_lock); +} + +struct gunyah_vm_memory_mapping *gh_vm_mem_mapping_find(struct gunyah_vm *ghvm, u32 label) +{ + struct gunyah_vm_memory_mapping *mapping; + int ret; + + ret = mutex_lock_interruptible(&ghvm->mm_lock); + if (ret) + return ERR_PTR(ret); + mapping = __gh_vm_mem_mapping_find(ghvm, label); + mutex_unlock(&ghvm->mm_lock); + return mapping ? : ERR_PTR(-ENODEV); +} + +struct gunyah_vm_memory_mapping *gh_vm_mem_mapping_alloc(struct gunyah_vm *ghvm, + struct gh_userspace_memory_region *region) +{ + phys_addr_t curr_page, prev_page; + struct gunyah_vm_memory_mapping *mapping, *tmp_mapping; + struct gh_rm_mem_entry *mem_entries; + int i, j, pinned, ret = 0; + struct gh_rm_mem_parcel *parcel; + size_t entry_size; + u16 vmid; + + if (!region->memory_size || !PAGE_ALIGNED(region->memory_size) || + !PAGE_ALIGNED(region->userspace_addr)) + return ERR_PTR(-EINVAL); + + if (!gh_api_has_feature(GH_API_FEATURE_MEMEXTENT)) + return ERR_PTR(-EOPNOTSUPP); + + ret = mutex_lock_interruptible(&ghvm->mm_lock); + if (ret) + return ERR_PTR(ret); + mapping = __gh_vm_mem_mapping_find(ghvm, region->label); + if (mapping) { + mutex_unlock(&ghvm->mm_lock); + return ERR_PTR(-EEXIST); + } + + mapping = kzalloc(sizeof(*mapping), GFP_KERNEL); + if (!mapping) { + ret = -ENOMEM; + goto unlock; + } + + mapping->parcel.label = region->label; + mapping->guest_phys_addr = region->guest_phys_addr; + mapping->npages = region->memory_size >> PAGE_SHIFT; + parcel = &mapping->parcel; + parcel->mem_handle = GH_MEM_HANDLE_INVAL; /* to be filled later by mem_share/mem_lend */ + parcel->mem_type = GH_RM_MEM_TYPE_NORMAL; + + /* Check for overlap */ + list_for_each_entry(tmp_mapping, &ghvm->memory_mappings, list) { + if (!((mapping->guest_phys_addr + (mapping->npages << PAGE_SHIFT) <= + tmp_mapping->guest_phys_addr) || + (mapping->guest_phys_addr >= + tmp_mapping->guest_phys_addr + (tmp_mapping->npages << PAGE_SHIFT)))) { + ret = -EEXIST; + goto unlock; + } + } + + list_add(&mapping->list, &ghvm->memory_mappings); +unlock: + mutex_unlock(&ghvm->mm_lock); + if (ret) + goto free_mapping; + + mapping->pages = kcalloc(mapping->npages, sizeof(*mapping->pages), GFP_KERNEL); + if (!mapping->pages) { + ret = -ENOMEM; + goto reclaim; + } + + pinned = pin_user_pages_fast(region->userspace_addr, mapping->npages, + FOLL_WRITE | FOLL_LONGTERM, mapping->pages); + if (pinned < 0) { + ret = pinned; + goto reclaim; + } else if (pinned != mapping->npages) { + ret = -EFAULT; + mapping->npages = pinned; /* update npages for reclaim */ + goto reclaim; + } + + if (region->flags & GH_MEM_LENT) { + parcel->n_acl_entries = 1; + mapping->share_type = VM_MEM_LEND; + } else { + parcel->n_acl_entries = 2; + mapping->share_type = VM_MEM_SHARE; + } + parcel->acl_entries = kcalloc(parcel->n_acl_entries, + sizeof(*parcel->acl_entries), + GFP_KERNEL); + if (!parcel->acl_entries) { + ret = -ENOMEM; + goto reclaim; + } + + parcel->acl_entries[0].vmid = cpu_to_le16(ghvm->vmid); + if (region->flags & GH_MEM_ALLOW_READ) + parcel->acl_entries[0].perms |= GH_RM_ACL_R; + if (region->flags & GH_MEM_ALLOW_WRITE) + parcel->acl_entries[0].perms |= GH_RM_ACL_W; + if (region->flags & GH_MEM_ALLOW_EXEC) + parcel->acl_entries[0].perms |= GH_RM_ACL_X; + + if (mapping->share_type == VM_MEM_SHARE) { + ret = gh_rm_get_vmid(ghvm->rm, &vmid); + if (ret) { + if (ret > 0) { + pr_warn("RM failed to get this VM's VMID: %d", ret); + ret = -EINVAL; + } + goto reclaim; + } + + parcel->acl_entries[1].vmid = cpu_to_le16(vmid); + /* Host assumed to have all these permissions. Gunyah will not + * grant new permissions if host actually had less than RWX + */ + parcel->acl_entries[1].perms |= GH_RM_ACL_R | GH_RM_ACL_W | GH_RM_ACL_X; + } + + mem_entries = kcalloc(mapping->npages, sizeof(*mem_entries), GFP_KERNEL); + if (!mem_entries) { + ret = -ENOMEM; + goto reclaim; + } + + /* reduce number of entries by combining contiguous pages into single memory entry */ + prev_page = page_to_phys(mapping->pages[0]); + mem_entries[0].ipa_base = cpu_to_le64(prev_page); + entry_size = PAGE_SIZE; + for (i = 1, j = 0; i < mapping->npages; i++) { + curr_page = page_to_phys(mapping->pages[i]); + if (page_contiguous(prev_page, curr_page)) { + entry_size += PAGE_SIZE; + } else { + mem_entries[j].size = cpu_to_le64(entry_size); + j++; + mem_entries[j].ipa_base = cpu_to_le64(curr_page); + entry_size = PAGE_SIZE; + } + + prev_page = curr_page; + } + mem_entries[j].size = cpu_to_le64(entry_size); + + parcel->n_mem_entries = j + 1; + parcel->mem_entries = kmemdup(mem_entries, sizeof(*mem_entries) * parcel->n_mem_entries, + GFP_KERNEL); + kfree(mem_entries); + if (!parcel->mem_entries) { + ret = -ENOMEM; + goto reclaim; + } + + return mapping; +reclaim: + gh_vm_mem_mapping_reclaim(ghvm, mapping); +free_mapping: + kfree(mapping); + return ERR_PTR(ret); +} diff --git a/include/uapi/linux/gunyah.h b/include/uapi/linux/gunyah.h index 88a40d6e0b96..574f33b198d0 100644 --- a/include/uapi/linux/gunyah.h +++ b/include/uapi/linux/gunyah.h @@ -20,4 +20,26 @@ */ #define GH_CREATE_VM _IO(GH_IOCTL_TYPE, 0x0) /* Returns a Gunyah VM fd */ +/* + * ioctls for VM fds + */ +struct gh_userspace_memory_region { + __u32 label; +#define GH_MEM_ALLOW_READ (1UL << 0) +#define GH_MEM_ALLOW_WRITE (1UL << 1) +#define GH_MEM_ALLOW_EXEC (1UL << 2) +/* + * The guest will be lent the memory instead of shared. + * In other words, the guest has exclusive access to the memory region and the host loses access. + */ +#define GH_MEM_LENT (1UL << 3) + __u32 flags; + __u64 guest_phys_addr; + __u64 memory_size; + __u64 userspace_addr; +}; + +#define GH_VM_SET_USER_MEM_REGION _IOW(GH_IOCTL_TYPE, 0x1, \ + struct gh_userspace_memory_region) + #endif From patchwork Fri Jan 20 22:46:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 644678 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C606C5320B for ; Fri, 20 Jan 2023 22:49:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230030AbjATWtH (ORCPT ); Fri, 20 Jan 2023 17:49:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230086AbjATWsp (ORCPT ); Fri, 20 Jan 2023 17:48:45 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5A8E53E4C; Fri, 20 Jan 2023 14:48:15 -0800 (PST) Received: from pps.filterd (m0279868.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KMIc6S001985; Fri, 20 Jan 2023 22:47:52 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=PFpOTHSWciIB3y4OtfTFpmzgQSpOlTb9sWxpLGPZE5M=; b=RrlEb2t4BSrmPITkcZXUbVfoY31h6BV5DRAbRupwfRo3lZ3oVCgPhuRxaej3s0dtFs9t uRgp2DVKMqkvtSKV9/5lwyR4AKcmgKq2S8vquyZuyaWNyB6itbarabzaOUiPUqc+G5Cs KjlF0vlGOpBzz6GEe3k6p/n2zKIicfpGoKR1SZpT3G5uwaT8S4YzJn0e46jAATCOgkjU HfGBNefNYOAOWz4ZtJPuUkv9e1xNuXVSG2H36z3V6NO5rjAEqRYCcyO9wo1awFlyzqhd aXvIebRQzjf7DkNECYApxtRokbFZvl4qp3UMR4CQR2Fjj+MVIvGkbKdRZkAn5wTOQfSC pw== Received: from nasanppmta02.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n75w3kmx0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:52 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA02.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMlpat027695 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:51 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:50 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Will Deacon , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 14/27] gunyah: vm_mgr: Add ioctls to support basic non-proxy VM boot Date: Fri, 20 Jan 2023 14:46:13 -0800 Message-ID: <20230120224627.4053418-15-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: m-UFTBsVdZcKMIEsmkx7jPaAxu9CmTlC X-Proofpoint-GUID: m-UFTBsVdZcKMIEsmkx7jPaAxu9CmTlC X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 impostorscore=0 lowpriorityscore=0 adultscore=0 clxscore=1015 mlxscore=0 spamscore=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Add remaining ioctls to support non-proxy VM boot: - Gunyah Resource Manager uses the VM's devicetree to configure the virtual machine. The location of the devicetree in the guest's virtual memory can be declared via the SET_DTB_CONFIG ioctl. - Trigger start of the virtual machine with VM_START ioctl. Co-developed-by: Prakruthi Deepak Heragu Signed-off-by: Prakruthi Deepak Heragu Signed-off-by: Elliot Berman --- drivers/virt/gunyah/vm_mgr.c | 110 ++++++++++++++++++++++++++++++++ drivers/virt/gunyah/vm_mgr.h | 9 +++ drivers/virt/gunyah/vm_mgr_mm.c | 24 +++++++ include/uapi/linux/gunyah.h | 8 +++ 4 files changed, 151 insertions(+) diff --git a/drivers/virt/gunyah/vm_mgr.c b/drivers/virt/gunyah/vm_mgr.c index b847fde63333..48bd3f06fb6c 100644 --- a/drivers/virt/gunyah/vm_mgr.c +++ b/drivers/virt/gunyah/vm_mgr.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include @@ -37,10 +38,98 @@ static __must_check struct gunyah_vm *gunyah_vm_alloc(struct gh_rm_rpc *rm) mutex_init(&ghvm->mm_lock); INIT_LIST_HEAD(&ghvm->memory_mappings); + init_rwsem(&ghvm->status_lock); return ghvm; } +static int gh_vm_start(struct gunyah_vm *ghvm) +{ + struct gunyah_vm_memory_mapping *mapping; + u64 dtb_offset; + u32 mem_handle; + int ret; + + down_write(&ghvm->status_lock); + if (ghvm->vm_status != GH_RM_VM_STATUS_NO_STATE) { + up_write(&ghvm->status_lock); + return 0; + } + + list_for_each_entry(mapping, &ghvm->memory_mappings, list) { + switch (mapping->share_type) { + case VM_MEM_LEND: + ret = gh_rm_mem_lend(ghvm->rm, &mapping->parcel); + break; + case VM_MEM_SHARE: + ret = gh_rm_mem_share(ghvm->rm, &mapping->parcel); + break; + } + if (ret > 0) + ret = -EINVAL; + if (ret) { + pr_warn("Failed to %s parcel %d: %d\n", + mapping->share_type == VM_MEM_LEND ? "lend" : "share", + mapping->parcel.label, + ret); + goto err; + } + } + + mapping = gh_vm_mem_mapping_find_mapping(ghvm, ghvm->dtb_config.gpa, ghvm->dtb_config.size); + if (!mapping) { + pr_warn("Failed to find the memory_handle for DTB\n"); + ret = -EINVAL; + goto err; + } + + mem_handle = mapping->parcel.mem_handle; + dtb_offset = ghvm->dtb_config.gpa - mapping->guest_phys_addr; + + ret = gh_rm_vm_configure(ghvm->rm, ghvm->vmid, ghvm->auth, mem_handle, + 0, 0, dtb_offset, ghvm->dtb_config.size); + if (ret) { + pr_warn("Failed to configure VM: %d\n", ret); + goto err; + } + + ret = gh_rm_vm_init(ghvm->rm, ghvm->vmid); + if (ret) { + pr_warn("Failed to initialize VM: %d\n", ret); + goto err; + } + + ret = gh_rm_vm_start(ghvm->rm, ghvm->vmid); + if (ret) { + pr_warn("Failed to start VM: %d\n", ret); + goto err; + } + + ghvm->vm_status = GH_RM_VM_STATUS_READY; + + up_write(&ghvm->status_lock); + return ret; +err: + ghvm->vm_status = GH_RM_VM_STATUS_INIT_FAILED; + up_write(&ghvm->status_lock); + return ret; +} + +static void gh_vm_stop(struct gunyah_vm *ghvm) +{ + int ret; + + down_write(&ghvm->status_lock); + if (ghvm->vm_status == GH_RM_VM_STATUS_READY) { + ret = gh_rm_vm_stop(ghvm->rm, ghvm->vmid); + if (ret) + pr_warn("Failed to stop VM: %d\n", ret); + } + + ghvm->vm_status = GH_RM_VM_STATUS_EXITED; + up_write(&ghvm->status_lock); +} + static long gh_vm_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) { struct gunyah_vm *ghvm = filp->private_data; @@ -84,6 +173,25 @@ static long gh_vm_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) } break; } + case GH_VM_SET_DTB_CONFIG: { + struct gh_vm_dtb_config dtb_config; + + r = -EFAULT; + if (copy_from_user(&dtb_config, argp, sizeof(dtb_config))) + break; + + dtb_config.size = PAGE_ALIGN(dtb_config.size); + ghvm->dtb_config = dtb_config; + + r = 0; + break; + } + case GH_VM_START: { + r = gh_vm_start(ghvm); + if (r) + r = -EINVAL; + break; + } default: r = -ENOTTY; break; @@ -97,6 +205,8 @@ static int gh_vm_release(struct inode *inode, struct file *filp) struct gunyah_vm *ghvm = filp->private_data; struct gunyah_vm_memory_mapping *mapping, *tmp; + gh_vm_stop(ghvm); + list_for_each_entry_safe(mapping, tmp, &ghvm->memory_mappings, list) { gh_vm_mem_mapping_reclaim(ghvm, mapping); kfree(mapping); diff --git a/drivers/virt/gunyah/vm_mgr.h b/drivers/virt/gunyah/vm_mgr.h index 6b38bf780f76..5c02fb305893 100644 --- a/drivers/virt/gunyah/vm_mgr.h +++ b/drivers/virt/gunyah/vm_mgr.h @@ -10,6 +10,7 @@ #include #include #include +#include #include @@ -34,6 +35,12 @@ struct gunyah_vm { u16 vmid; struct gh_rm *rm; + enum gh_rm_vm_auth_mechanism auth; + struct gh_vm_dtb_config dtb_config; + + enum gh_rm_vm_status vm_status; + struct rw_semaphore status_lock; + struct mutex mm_lock; struct list_head memory_mappings; }; @@ -42,5 +49,7 @@ struct gunyah_vm_memory_mapping *gh_vm_mem_mapping_alloc(struct gunyah_vm *ghvm, struct gh_userspace_memory_region *region); void gh_vm_mem_mapping_reclaim(struct gunyah_vm *ghvm, struct gunyah_vm_memory_mapping *mapping); struct gunyah_vm_memory_mapping *gh_vm_mem_mapping_find(struct gunyah_vm *ghvm, u32 label); +struct gunyah_vm_memory_mapping *gh_vm_mem_mapping_find_mapping(struct gunyah_vm *ghvm, + u64 gpa, u32 size); #endif diff --git a/drivers/virt/gunyah/vm_mgr_mm.c b/drivers/virt/gunyah/vm_mgr_mm.c index f2dbdb4ee8ab..7fcb9f8a29bf 100644 --- a/drivers/virt/gunyah/vm_mgr_mm.c +++ b/drivers/virt/gunyah/vm_mgr_mm.c @@ -53,6 +53,30 @@ void gh_vm_mem_mapping_reclaim(struct gunyah_vm *ghvm, struct gunyah_vm_memory_m mutex_unlock(&ghvm->mm_lock); } +struct gunyah_vm_memory_mapping *gh_vm_mem_mapping_find_mapping(struct gunyah_vm *ghvm, + u64 gpa, u32 size) +{ + struct gunyah_vm_memory_mapping *mapping = NULL; + int ret; + + ret = mutex_lock_interruptible(&ghvm->mm_lock); + if (ret) + return ERR_PTR(ret); + + list_for_each_entry(mapping, &ghvm->memory_mappings, list) { + if (gpa >= mapping->guest_phys_addr && + (gpa + size <= mapping->guest_phys_addr + + (mapping->npages << PAGE_SHIFT))) { + goto unlock; + } + } + + mapping = NULL; +unlock: + mutex_unlock(&ghvm->mm_lock); + return mapping; +} + struct gunyah_vm_memory_mapping *gh_vm_mem_mapping_find(struct gunyah_vm *ghvm, u32 label) { struct gunyah_vm_memory_mapping *mapping; diff --git a/include/uapi/linux/gunyah.h b/include/uapi/linux/gunyah.h index 574f33b198d0..36359ad2175e 100644 --- a/include/uapi/linux/gunyah.h +++ b/include/uapi/linux/gunyah.h @@ -42,4 +42,12 @@ struct gh_userspace_memory_region { #define GH_VM_SET_USER_MEM_REGION _IOW(GH_IOCTL_TYPE, 0x1, \ struct gh_userspace_memory_region) +struct gh_vm_dtb_config { + __u64 gpa; + __u64 size; +}; +#define GH_VM_SET_DTB_CONFIG _IOW(GH_IOCTL_TYPE, 0x2, struct gh_vm_dtb_config) + +#define GH_VM_START _IO(GH_IOCTL_TYPE, 0x3) + #endif From patchwork Fri Jan 20 22:46:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 645101 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EFA9C54E94 for ; Fri, 20 Jan 2023 22:49:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230026AbjATWtf (ORCPT ); Fri, 20 Jan 2023 17:49:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230079AbjATWt2 (ORCPT ); Fri, 20 Jan 2023 17:49:28 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3927B646B4; Fri, 20 Jan 2023 14:48:52 -0800 (PST) Received: from pps.filterd (m0279871.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KM03ci024187; Fri, 20 Jan 2023 22:47:54 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=7ArPs5g314Fs0dtDwcEUquwnQBn1GZOIYZtz5VzBVwQ=; b=V8VD+NnJhMinvnfOnAvav5MPKD5+NkjY+c9Llx8NmmG6T1XdbdkGReB4xEQwPNWwK51W HkCsYCuzrmA/GvyHjsiVMlBaKAGYGTmDDQWc0rF1tRgTDst8gcK8miW269gwYTrbdCGa At1aY65uCLXNilhrWms9PF5vJdumgwaIQRmvqjGMizc8W+7A3OsdC18e4rh31kQ501Z0 IFZ2qBlvgyhRqyZRBssdKoejUN67fgKmi16byeet1fPvUMllaDyNLUUKm12Lw/hZqh1S sZ68mpSKFh5n4ooXb2AOzxL6ogBfNrk7KZVrfzeAfsRwnzueJhH1KOrOh2HcG/cwxfTc +A== Received: from nasanppmta01.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7c28jrf0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:54 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA01.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMlrOr004044 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:53 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:52 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Will Deacon , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 15/27] samples: Add sample userspace Gunyah VM Manager Date: Fri, 20 Jan 2023 14:46:14 -0800 Message-ID: <20230120224627.4053418-16-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: BczsjI40LI5DMqShMp2LiuzuqQxeF_Jf X-Proofpoint-ORIG-GUID: BczsjI40LI5DMqShMp2LiuzuqQxeF_Jf X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 spamscore=0 priorityscore=1501 adultscore=0 lowpriorityscore=0 mlxlogscore=999 clxscore=1015 mlxscore=0 suspectscore=0 malwarescore=0 bulkscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Add a sample Gunyah VMM capable of launching a non-proxy scheduled VM. Signed-off-by: Elliot Berman --- MAINTAINERS | 1 + samples/Kconfig | 10 ++ samples/Makefile | 1 + samples/gunyah/.gitignore | 2 + samples/gunyah/Makefile | 6 + samples/gunyah/gunyah_vmm.c | 270 +++++++++++++++++++++++++++++++++++ samples/gunyah/sample_vm.dts | 69 +++++++++ 7 files changed, 359 insertions(+) create mode 100644 samples/gunyah/.gitignore create mode 100644 samples/gunyah/Makefile create mode 100644 samples/gunyah/gunyah_vmm.c create mode 100644 samples/gunyah/sample_vm.dts diff --git a/MAINTAINERS b/MAINTAINERS index 7ab1462577cc..10d44ca7e6ce 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9052,6 +9052,7 @@ F: arch/arm64/gunyah/ F: drivers/mailbox/gunyah-msgq.c F: drivers/virt/gunyah/ F: include/linux/gunyah*.h +F: samples/gunyah/ HABANALABS PCI DRIVER M: Oded Gabbay diff --git a/samples/Kconfig b/samples/Kconfig index 0d81c00289ee..7d0903da223c 100644 --- a/samples/Kconfig +++ b/samples/Kconfig @@ -263,6 +263,16 @@ config SAMPLE_CORESIGHT_SYSCFG This demonstrates how a user may create their own CoreSight configurations and easily load them into the system at runtime. +config SAMPLE_GUNYAH + bool "Build example Gunyah Virtual Machine Manager" + depends on CC_CAN_LINK && HEADERS_INSTALL + depends on GUNYAH + help + Build an example Gunyah VMM userspace program capable of launching + a basic virtual machine under the Gunyah hypervisor. + This demonstrates how to create a virtual machine under the Gunyah + hypervisor. + source "samples/rust/Kconfig" endif # SAMPLES diff --git a/samples/Makefile b/samples/Makefile index 9832ef3f8fcb..2600bd4b82f8 100644 --- a/samples/Makefile +++ b/samples/Makefile @@ -36,3 +36,4 @@ obj-$(CONFIG_DEBUG_KMEMLEAK_TEST) += kmemleak/ obj-$(CONFIG_SAMPLE_CORESIGHT_SYSCFG) += coresight/ obj-$(CONFIG_SAMPLE_FPROBE) += fprobe/ obj-$(CONFIG_SAMPLES_RUST) += rust/ +obj-$(CONFIG_SAMPLE_GUNYAH) += gunyah/ diff --git a/samples/gunyah/.gitignore b/samples/gunyah/.gitignore new file mode 100644 index 000000000000..adc7d1589fde --- /dev/null +++ b/samples/gunyah/.gitignore @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0 +/gunyah_vmm diff --git a/samples/gunyah/Makefile b/samples/gunyah/Makefile new file mode 100644 index 000000000000..faf14f9bb337 --- /dev/null +++ b/samples/gunyah/Makefile @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: GPL-2.0-only + +userprogs-always-y += gunyah_vmm +dtb-y += sample_vm.dtb + +userccflags += -I usr/include diff --git a/samples/gunyah/gunyah_vmm.c b/samples/gunyah/gunyah_vmm.c new file mode 100644 index 000000000000..c594b1ec64c9 --- /dev/null +++ b/samples/gunyah/gunyah_vmm.c @@ -0,0 +1,270 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define __USE_GNU +#include + +#include + +struct vm_config { + int image_fd; + int dtb_fd; + int ramdisk_fd; + + uint64_t guest_base; + uint64_t guest_size; + + uint64_t image_offset; + off_t image_size; + uint64_t dtb_offset; + off_t dtb_size; + uint64_t ramdisk_offset; + off_t ramdisk_size; +}; + +static struct option options[] = { + { "help", no_argument, NULL, 'h' }, + { "image", required_argument, NULL, 'i' }, + { "dtb", required_argument, NULL, 'd' }, + { "ramdisk", optional_argument, NULL, 'r' }, + { "base", optional_argument, NULL, 'B' }, + { "size", optional_argument, NULL, 'S' }, + { "image_offset", optional_argument, NULL, 'I' }, + { "dtb_offset", optional_argument, NULL, 'D' }, + { "ramdisk_offset", optional_argument, NULL, 'R' }, + { } +}; + +static void print_help(char *cmd) +{ + printf("gunyah_vmm, a sample tool to launch Gunyah VMs\n" + "Usage: %s \n" + " --help, -h this menu\n" + " --image, -i VM image file to load (e.g. a kernel Image) [Required]\n" + " --dtb, -d Devicetree to load [Required]\n" + " --ramdisk, -r Ramdisk to load\n" + " --base, -B
Set the base address of guest's memory [Default: 0x80000000]\n" + " --size, -S The number of bytes large to make the guest's memory [Default: 0x6400000 (100 MB)]\n" + " --image_offset, -I Offset into guest memory to load the VM image file [Default: 0x10000]\n" + " --dtb_offset, -D Offset into guest memory to load the DTB [Default: 0]\n" + " --ramdisk_offset, -R Offset into guest memory to load a ramdisk [Default: 0x4600000]\n" + , cmd); +} + +int main(int argc, char **argv) +{ + int gunyah_fd, vm_fd, guest_fd; + struct gh_userspace_memory_region guest_mem_desc = { 0 }; + struct gh_vm_dtb_config dtb_config = { 0 }; + char *guest_mem; + struct vm_config config = { + /* Defaults good enough to boot static kernel and a basic ramdisk */ + .ramdisk_fd = -1, + .guest_base = 0x80000000, + .guest_size = 104857600, /* 100 MB */ + .image_offset = 0, + .dtb_offset = 0x45f0000, + .ramdisk_offset = 0x4600000, /* put at +70MB (30MB for ramdisk) */ + }; + struct stat st; + int opt, optidx, ret = 0; + long l; + + while ((opt = getopt_long(argc, argv, "hi:d:r:B:S:I:D:R:c:", options, &optidx)) != -1) { + switch (opt) { + case 'i': + config.image_fd = open(optarg, O_RDONLY | O_CLOEXEC); + if (config.image_fd < 0) { + perror("Failed to open image"); + return -1; + } + if (stat(optarg, &st) < 0) { + perror("Failed to stat image"); + return -1; + } + config.image_size = st.st_size; + break; + case 'd': + config.dtb_fd = open(optarg, O_RDONLY | O_CLOEXEC); + if (config.dtb_fd < 0) { + perror("Failed to open dtb"); + return -1; + } + if (stat(optarg, &st) < 0) { + perror("Failed to stat dtb"); + return -1; + } + config.dtb_size = st.st_size; + break; + case 'r': + config.ramdisk_fd = open(optarg, O_RDONLY | O_CLOEXEC); + if (config.ramdisk_fd < 0) { + perror("Failed to open ramdisk"); + return -1; + } + if (stat(optarg, &st) < 0) { + perror("Failed to stat ramdisk"); + return -1; + } + config.ramdisk_size = st.st_size; + break; + case 'B': + l = strtol(optarg, NULL, 0); + if (l == LONG_MIN) { + perror("Failed to parse base address"); + return -1; + } + config.guest_base = l; + break; + case 'S': + l = strtol(optarg, NULL, 0); + if (l == LONG_MIN) { + perror("Failed to parse memory size"); + return -1; + } + config.guest_size = l; + break; + case 'I': + l = strtol(optarg, NULL, 0); + if (l == LONG_MIN) { + perror("Failed to parse image offset"); + return -1; + } + config.image_offset = l; + break; + case 'D': + l = strtol(optarg, NULL, 0); + if (l == LONG_MIN) { + perror("Failed to parse dtb offset"); + return -1; + } + config.dtb_offset = l; + break; + case 'R': + l = strtol(optarg, NULL, 0); + if (l == LONG_MIN) { + perror("Failed to parse ramdisk offset"); + return -1; + } + config.ramdisk_offset = l; + break; + case 'h': + print_help(argv[0]); + return 0; + default: + print_help(argv[0]); + return -1; + } + } + + if (!config.image_fd || !config.dtb_fd) { + print_help(argv[0]); + return -1; + } + + if (config.image_offset + config.image_size > config.guest_size) { + fprintf(stderr, "Image offset and size puts it outside guest memory. Make image smaller or increase guest memory size.\n"); + return -1; + } + + if (config.dtb_offset + config.dtb_size > config.guest_size) { + fprintf(stderr, "DTB offset and size puts it outside guest memory. Make dtb smaller or increase guest memory size.\n"); + return -1; + } + + if (config.ramdisk_fd == -1 && + config.ramdisk_offset + config.ramdisk_size > config.guest_size) { + fprintf(stderr, "Ramdisk offset and size puts it outside guest memory. Make ramdisk smaller or increase guest memory size.\n"); + return -1; + } + + gunyah_fd = open("/dev/gunyah", O_RDWR | O_CLOEXEC); + if (gunyah_fd < 0) { + perror("Failed to open /dev/gunyah"); + return -1; + } + + vm_fd = ioctl(gunyah_fd, GH_CREATE_VM, 0); + if (vm_fd < 0) { + perror("Failed to create vm"); + return -1; + } + + guest_fd = memfd_create("guest_memory", MFD_CLOEXEC); + if (guest_fd < 0) { + perror("Failed to create guest memfd"); + return -1; + } + + if (ftruncate(guest_fd, config.guest_size) < 0) { + perror("Failed to grow guest memory"); + return -1; + } + + guest_mem = mmap(NULL, config.guest_size, PROT_READ | PROT_WRITE, MAP_SHARED, guest_fd, 0); + if (guest_mem == MAP_FAILED) { + perror("Not enough memory"); + return -1; + } + + if (read(config.image_fd, guest_mem + config.image_offset, config.image_size) < 0) { + perror("Failed to read image into guest memory"); + return -1; + } + + if (read(config.dtb_fd, guest_mem + config.dtb_offset, config.dtb_size) < 0) { + perror("Failed to read dtb into guest memory"); + return -1; + } + + if (config.ramdisk_fd > 0 && + read(config.ramdisk_fd, guest_mem + config.ramdisk_offset, + config.ramdisk_size) < 0) { + perror("Failed to read ramdisk into guest memory"); + return -1; + } + + guest_mem_desc.label = 0; + guest_mem_desc.flags = GH_MEM_ALLOW_READ | GH_MEM_ALLOW_WRITE | GH_MEM_ALLOW_EXEC; + guest_mem_desc.guest_phys_addr = config.guest_base; + guest_mem_desc.memory_size = config.guest_size; + guest_mem_desc.userspace_addr = (__u64)guest_mem; + + if (ioctl(vm_fd, GH_VM_SET_USER_MEM_REGION, &guest_mem_desc) < 0) { + perror("Failed to register guest memory with VM"); + return -1; + } + + dtb_config.gpa = config.guest_base + config.dtb_offset; + dtb_config.size = config.dtb_size; + if (ioctl(vm_fd, GH_VM_SET_DTB_CONFIG, &dtb_config) < 0) { + perror("Failed to set DTB configuration for VM"); + return -1; + } + + ret = ioctl(vm_fd, GH_VM_START); + if (ret) { + perror("GH_VM_START failed"); + return -1; + } + + while (1) + sleep(10); + + return 0; +} diff --git a/samples/gunyah/sample_vm.dts b/samples/gunyah/sample_vm.dts new file mode 100644 index 000000000000..1c410d58c298 --- /dev/null +++ b/samples/gunyah/sample_vm.dts @@ -0,0 +1,69 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +/dts-v1/; + +/ { + #address-cells = <2>; + #size-cells = <2>; + interrupt-parent = <&intc>; + + chosen { + bootargs = "nokaslr"; + }; + + cpus { + #address-cells = <0x2>; + #size-cells = <0>; + + cpu@0 { + device_type = "cpu"; + compatible = "arm,armv8"; + reg = <0 0>; + }; + }; + + intc: interrupt-controller@3FFF0000 { + compatible = "arm,gic-v3"; + #interrupt-cells = <3>; + #address-cells = <2>; + #size-cells = <2>; + interrupt-controller; + reg = <0 0x3FFF0000 0 0x10000>, + <0 0x3FFD0000 0 0x20000>; + }; + + timer { + compatible = "arm,armv8-timer"; + always-on; + interrupts = <1 13 0x108>, + <1 14 0x108>, + <1 11 0x108>, + <1 10 0x108>; + clock-frequency = <19200000>; + }; + + gunyah-vm-config { + image-name = "linux_vm_0"; + os-type = "linux"; + + memory { + #address-cells = <2>; + #size-cells = <2>; + + base-address = <0 0x80000000>; + }; + + interrupts { + config = <&intc>; + }; + + vcpus { + affinity-map = < 0 >; + sched-priority = < (-1) >; + sched-timeslice = < 2000 >; + }; + }; +}; From patchwork Fri Jan 20 22:46:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 645103 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADF33C25B50 for ; Fri, 20 Jan 2023 22:49:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230056AbjATWtX (ORCPT ); Fri, 20 Jan 2023 17:49:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230163AbjATWtD (ORCPT ); Fri, 20 Jan 2023 17:49:03 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3FFAC4F84A; Fri, 20 Jan 2023 14:48:31 -0800 (PST) Received: from pps.filterd (m0279873.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KMgexe021199; Fri, 20 Jan 2023 22:47:56 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=Da8LxfgTfYtKfI/cjus48EkvxBRSYu1soTjigcymveE=; b=XJGW4JiIqbdbOgXxgXj9UOnQTkxAdZiJQS/y729a50brZC0DlaDlX4XjibC8JmtM0Q0Q 083+yREyK4dqHJXIWSRIZh/AGvA3nNsg7TRcrVGaUabkD5QQuwXma4uk6nPMOG91QlPR z0nwy4Kh7gk6TKVNIQcmBJDJOaGeA3EObKf5aZxUPTkHKQXb3rR8t4FXeOZtXprLtbhb kJkBr5/1HAWJvHmEZGL4KLy/5lTLnL9rMFiTUQvMVum5EyyDXYFUP7TRDGKC9qqqWJx/ 8yg5KBxGmAFwfdtT411j6yeVITG9m8D+apfX++scX2apb2N2l9RJ39rc+BAOL7xmp4vM dw== Received: from nasanppmta01.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7593uqy4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:56 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA01.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMltVn004285 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:55 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:54 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Will Deacon , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 16/27] gunyah: rsc_mgr: Add platform ops on mem_lend/mem_reclaim Date: Fri, 20 Jan 2023 14:46:15 -0800 Message-ID: <20230120224627.4053418-17-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: Xrb_VGEsFheP3jOUjhkrXkl2n1EtCD9Y X-Proofpoint-ORIG-GUID: Xrb_VGEsFheP3jOUjhkrXkl2n1EtCD9Y X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 mlxlogscore=999 spamscore=0 suspectscore=0 mlxscore=0 adultscore=0 clxscore=1015 impostorscore=0 priorityscore=1501 bulkscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org On Qualcomm platforms, there is a firmware entity which controls access to physical pages. In order to share memory with another VM, this entity needs to be informed that the guest VM should have access to the memory. Co-developed-by: Prakruthi Deepak Heragu Signed-off-by: Prakruthi Deepak Heragu Signed-off-by: Elliot Berman --- drivers/virt/gunyah/Kconfig | 4 ++ drivers/virt/gunyah/Makefile | 1 + drivers/virt/gunyah/gunyah_platform_hooks.c | 63 +++++++++++++++++++++ drivers/virt/gunyah/rsc_mgr.h | 3 + drivers/virt/gunyah/rsc_mgr_rpc.c | 9 ++- include/linux/gunyah_rsc_mgr.h | 14 +++++ 6 files changed, 91 insertions(+), 3 deletions(-) create mode 100644 drivers/virt/gunyah/gunyah_platform_hooks.c diff --git a/drivers/virt/gunyah/Kconfig b/drivers/virt/gunyah/Kconfig index 1a737694c333..de815189dab6 100644 --- a/drivers/virt/gunyah/Kconfig +++ b/drivers/virt/gunyah/Kconfig @@ -4,6 +4,7 @@ config GUNYAH tristate "Gunyah Virtualization drivers" depends on ARM64 depends on MAILBOX + select GUNYAH_PLATFORM_HOOKS help The Gunyah drivers are the helper interfaces that run in a guest VM such as basic inter-VM IPC and signaling mechanisms, and higher level @@ -11,3 +12,6 @@ config GUNYAH Say Y/M here to enable the drivers needed to interact in a Gunyah virtual environment. + +config GUNYAH_PLATFORM_HOOKS + tristate diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile index ff8bc4925392..6b8f84dbfe0d 100644 --- a/drivers/virt/gunyah/Makefile +++ b/drivers/virt/gunyah/Makefile @@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_GUNYAH) += gunyah.o +obj-$(CONFIG_GUNYAH_PLATFORM_HOOKS) += gunyah_platform_hooks.o gunyah_rsc_mgr-y += rsc_mgr.o rsc_mgr_rpc.o vm_mgr.o vm_mgr_mm.o obj-$(CONFIG_GUNYAH) += gunyah_rsc_mgr.o diff --git a/drivers/virt/gunyah/gunyah_platform_hooks.c b/drivers/virt/gunyah/gunyah_platform_hooks.c new file mode 100644 index 000000000000..73ea229438ca --- /dev/null +++ b/drivers/virt/gunyah/gunyah_platform_hooks.c @@ -0,0 +1,63 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#include +#include +#include + +#include "rsc_mgr.h" + +static struct gunyah_rm_platform_ops *rm_platform_ops; +static DECLARE_RWSEM(rm_platform_ops_lock); + +int gh_rm_platform_pre_mem_share(struct gh_rm *rm, struct gh_rm_mem_parcel *mem_parcel) +{ + int ret = 0; + + down_read(&rm_platform_ops_lock); + if (rm_platform_ops && rm_platform_ops->pre_mem_share) + ret = rm_platform_ops->pre_mem_share(rm, mem_parcel); + up_read(&rm_platform_ops_lock); + return ret; +} +EXPORT_SYMBOL_GPL(gh_rm_platform_pre_mem_share); + +int gh_rm_platform_post_mem_reclaim(struct gh_rm *rm, struct gh_rm_mem_parcel *mem_parcel) +{ + int ret = 0; + + down_read(&rm_platform_ops_lock); + if (rm_platform_ops && rm_platform_ops->post_mem_reclaim) + ret = rm_platform_ops->post_mem_reclaim(rm, mem_parcel); + up_read(&rm_platform_ops_lock); + return ret; +} +EXPORT_SYMBOL_GPL(gh_rm_platform_post_mem_reclaim); + +int gh_rm_register_platform_ops(struct gunyah_rm_platform_ops *platform_ops) +{ + int ret = 0; + + down_write(&rm_platform_ops_lock); + if (!rm_platform_ops) + rm_platform_ops = platform_ops; + else + ret = -EEXIST; + up_write(&rm_platform_ops_lock); + return ret; +} +EXPORT_SYMBOL_GPL(gh_rm_register_platform_ops); + +void gh_rm_unregister_platform_ops(struct gunyah_rm_platform_ops *platform_ops) +{ + down_write(&rm_platform_ops_lock); + if (rm_platform_ops == platform_ops) + rm_platform_ops = NULL; + up_write(&rm_platform_ops_lock); +} +EXPORT_SYMBOL_GPL(gh_rm_unregister_platform_ops); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("Gunyah Platform Hooks"); diff --git a/drivers/virt/gunyah/rsc_mgr.h b/drivers/virt/gunyah/rsc_mgr.h index 17047afefb28..8486f6b7bd50 100644 --- a/drivers/virt/gunyah/rsc_mgr.h +++ b/drivers/virt/gunyah/rsc_mgr.h @@ -68,6 +68,9 @@ struct gh_rm; int gh_rm_call(struct gh_rm *rsc_mgr, u32 message_id, void *req_buff, size_t req_buff_size, void **resp_buf, size_t *resp_buff_size); +int gh_rm_platform_pre_mem_share(struct gh_rm *rm, struct gh_rm_mem_parcel *mem_parcel); +int gh_rm_platform_post_mem_reclaim(struct gh_rm *rm, struct gh_rm_mem_parcel *mem_parcel); + /* Message IDs: Memory Management */ #define GH_RM_RPC_MEM_LEND 0x51000012 #define GH_RM_RPC_MEM_SHARE 0x51000013 diff --git a/drivers/virt/gunyah/rsc_mgr_rpc.c b/drivers/virt/gunyah/rsc_mgr_rpc.c index a3633df98a38..ea4b8be1504b 100644 --- a/drivers/virt/gunyah/rsc_mgr_rpc.c +++ b/drivers/virt/gunyah/rsc_mgr_rpc.c @@ -168,8 +168,11 @@ static int gh_rm_mem_lend_common(struct gh_rm *rm, u32 message_id, struct gh_rm_ goto out; } - if (resp_size != sizeof(__le32)) { - ret = -EIO; + if (resp_size != sizeof(*resp)) { + pr_warn("%s: unexpected payload size: %ld Expected: %ld", __func__, + resp_size, sizeof(*resp)); + ret = -EBADMSG; + gh_rm_platform_post_mem_reclaim(rm, p); goto out; } @@ -243,7 +246,7 @@ int gh_rm_mem_reclaim(struct gh_rm *rm, struct gh_rm_mem_parcel *parcel) return -EBADMSG; } - return ret; + return gh_rm_platform_post_mem_reclaim(rm, parcel); } EXPORT_SYMBOL_GPL(gh_rm_mem_reclaim); diff --git a/include/linux/gunyah_rsc_mgr.h b/include/linux/gunyah_rsc_mgr.h index 264f054540b8..dd94bb3f378b 100644 --- a/include/linux/gunyah_rsc_mgr.h +++ b/include/linux/gunyah_rsc_mgr.h @@ -120,4 +120,18 @@ int gh_rm_get_hyp_resources(struct gh_rm *rm, u16 vmid, struct gh_rm_hyp_resources **resources); int gh_rm_get_vmid(struct gh_rm *rm, u16 *vmid); +struct gunyah_rm_platform_ops { + int (*pre_mem_share)(struct gh_rm *rm, struct gh_rm_mem_parcel *mem_parcel); + int (*post_mem_reclaim)(struct gh_rm *rm, struct gh_rm_mem_parcel *mem_parcel); +}; + +#if IS_ENABLED(CONFIG_GUNYAH_PLATFORM_HOOKS) +int gh_rm_register_platform_ops(struct gunyah_rm_platform_ops *platform_ops); +void gh_rm_unregister_platform_ops(struct gunyah_rm_platform_ops *platform_ops); +#else +static inline int gh_rm_register_platform_ops(struct gunyah_rm_platform_ops *platform_ops) + { return 0; } +static inline void gh_rm_unregister_platform_ops(struct gunyah_rm_platform_ops *platform_ops) { } +#endif + #endif From patchwork Fri Jan 20 22:46:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 645100 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DE5AC38A23 for ; Fri, 20 Jan 2023 22:50:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230184AbjATWuI (ORCPT ); Fri, 20 Jan 2023 17:50:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58470 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229697AbjATWuF (ORCPT ); Fri, 20 Jan 2023 17:50:05 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F08B9DF940; Fri, 20 Jan 2023 14:49:14 -0800 (PST) Received: from pps.filterd (m0279873.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KMgexg021199; Fri, 20 Jan 2023 22:47:58 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=pgm72xy3Fu/eOIqzOX83nJcOfePswS4HnP+yNaRXne0=; b=hnb0eIwH13Q/9z265+nOCEqV44dPOxoyf5oiru8UpxNt5lvaTXIuZiu1ocx+wXux54+p u4DG3i/2DRYyAEbaJj1ep9fYPfLjqybXNh7gk4DmuSh5UCJPQFSTducg5FgopY1TNtNn nmqjYFCgION59iDyI8aSMySRZOK6jPgXgnf1BNslLbXkLXoZE1Ct8VdxGJI39nej7EBc hqaueZndC1fVLfMxcUeIrP6iDgTZS9C6ii07fh4G6jI28dB3I8m3XswwMz9QFvO/CPMl DxcA6gi9TI1cLi5ZBrBa9bkQ+OGe0W2b+0/GQMnWDfV9WXwPhgymRL5+cBYiNtd0b/+q HA== Received: from nasanppmta01.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7593uqya-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:58 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA01.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMlvLG005336 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:57 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:55 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Andy Gross , Bjorn Andersson , Konrad Dybcio , "Srinivas Kandagatla" , Amol Maheshwari , Arnd Bergmann , Greg Kroah-Hartman , Kalle Valo , "David S. Miller" , Eric Dumazet , "Jakub Kicinski" , Paolo Abeni , Mathieu Poirier CC: Elliot Berman , Murali Nalajala , Trilok Soni , "Srivatsa Vaddagiri" , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , "Bagas Sanjaya" , Catalin Marinas , Will Deacon , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , , , , , Subject: [PATCH v9 17/27] firmware: qcom_scm: Use fixed width src vm bitmap Date: Fri, 20 Jan 2023 14:46:16 -0800 Message-ID: <20230120224627.4053418-18-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: Czb8svzwwF-6YkTntGFc2n6bBK9YYHOP X-Proofpoint-ORIG-GUID: Czb8svzwwF-6YkTntGFc2n6bBK9YYHOP X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 mlxlogscore=999 spamscore=0 suspectscore=0 mlxscore=0 adultscore=0 clxscore=1011 impostorscore=0 priorityscore=1501 bulkscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org The maximum VMID for assign_mem is 63. Use a u64 to represent this bitmap instead of architecture-dependent "unsigned int" which varies in size on 32-bit and 64-bit platforms. Acked-by: Kalle Valo [ath10k] Signed-off-by: Elliot Berman --- drivers/firmware/qcom_scm.c | 12 +++++++----- drivers/misc/fastrpc.c | 2 +- drivers/net/wireless/ath/ath10k/qmi.c | 4 ++-- drivers/remoteproc/qcom_q6v5_mss.c | 8 ++++---- drivers/soc/qcom/rmtfs_mem.c | 2 +- include/linux/qcom_scm.h | 2 +- 6 files changed, 16 insertions(+), 14 deletions(-) diff --git a/drivers/firmware/qcom_scm.c b/drivers/firmware/qcom_scm.c index cdbfe54c8146..92763dce6477 100644 --- a/drivers/firmware/qcom_scm.c +++ b/drivers/firmware/qcom_scm.c @@ -898,7 +898,7 @@ static int __qcom_scm_assign_mem(struct device *dev, phys_addr_t mem_region, * Return negative errno on failure or 0 on success with @srcvm updated. */ int qcom_scm_assign_mem(phys_addr_t mem_addr, size_t mem_sz, - unsigned int *srcvm, + u64 *srcvm, const struct qcom_scm_vmperm *newvm, unsigned int dest_cnt) { @@ -915,9 +915,9 @@ int qcom_scm_assign_mem(phys_addr_t mem_addr, size_t mem_sz, __le32 *src; void *ptr; int ret, i, b; - unsigned long srcvm_bits = *srcvm; + u64 srcvm_bits = *srcvm; - src_sz = hweight_long(srcvm_bits) * sizeof(*src); + src_sz = hweight64(srcvm_bits) * sizeof(*src); mem_to_map_sz = sizeof(*mem_to_map); dest_sz = dest_cnt * sizeof(*destvm); ptr_sz = ALIGN(src_sz, SZ_64) + ALIGN(mem_to_map_sz, SZ_64) + @@ -930,8 +930,10 @@ int qcom_scm_assign_mem(phys_addr_t mem_addr, size_t mem_sz, /* Fill source vmid detail */ src = ptr; i = 0; - for_each_set_bit(b, &srcvm_bits, BITS_PER_LONG) - src[i++] = cpu_to_le32(b); + for (b = 0; b < BITS_PER_TYPE(u64); b++) { + if (srcvm_bits & BIT(b)) + src[i++] = cpu_to_le32(b); + } /* Fill details of mem buff to map */ mem_to_map = ptr + ALIGN(src_sz, SZ_64); diff --git a/drivers/misc/fastrpc.c b/drivers/misc/fastrpc.c index 5310606113fe..67c24b4002b8 100644 --- a/drivers/misc/fastrpc.c +++ b/drivers/misc/fastrpc.c @@ -262,7 +262,7 @@ struct fastrpc_channel_ctx { int domain_id; int sesscount; int vmcount; - u32 perms; + u64 perms; struct qcom_scm_vmperm vmperms[FASTRPC_MAX_VMIDS]; struct rpmsg_device *rpdev; struct fastrpc_session_ctx session[FASTRPC_MAX_SESSIONS]; diff --git a/drivers/net/wireless/ath/ath10k/qmi.c b/drivers/net/wireless/ath/ath10k/qmi.c index 3f94fbf83702..a8497a6ea03c 100644 --- a/drivers/net/wireless/ath/ath10k/qmi.c +++ b/drivers/net/wireless/ath/ath10k/qmi.c @@ -33,7 +33,7 @@ static int ath10k_qmi_map_msa_permission(struct ath10k_qmi *qmi, { struct qcom_scm_vmperm dst_perms[3]; struct ath10k *ar = qmi->ar; - unsigned int src_perms; + u64 src_perms; u32 perm_count; int ret; @@ -65,7 +65,7 @@ static int ath10k_qmi_unmap_msa_permission(struct ath10k_qmi *qmi, { struct qcom_scm_vmperm dst_perms; struct ath10k *ar = qmi->ar; - unsigned int src_perms; + u64 src_perms; int ret; src_perms = BIT(QCOM_SCM_VMID_MSS_MSA) | BIT(QCOM_SCM_VMID_WLAN); diff --git a/drivers/remoteproc/qcom_q6v5_mss.c b/drivers/remoteproc/qcom_q6v5_mss.c index fddb63cffee0..9e8bde7a7ec4 100644 --- a/drivers/remoteproc/qcom_q6v5_mss.c +++ b/drivers/remoteproc/qcom_q6v5_mss.c @@ -227,8 +227,8 @@ struct q6v5 { bool has_qaccept_regs; bool has_ext_cntl_regs; bool has_vq6; - int mpss_perm; - int mba_perm; + u64 mpss_perm; + u64 mba_perm; const char *hexagon_mdt_image; int version; }; @@ -404,7 +404,7 @@ static void q6v5_pds_disable(struct q6v5 *qproc, struct device **pds, } } -static int q6v5_xfer_mem_ownership(struct q6v5 *qproc, int *current_perm, +static int q6v5_xfer_mem_ownership(struct q6v5 *qproc, u64 *current_perm, bool local, bool remote, phys_addr_t addr, size_t size) { @@ -939,7 +939,7 @@ static int q6v5_mpss_init_image(struct q6v5 *qproc, const struct firmware *fw, struct page *page; dma_addr_t phys; void *metadata; - int mdata_perm; + u64 mdata_perm; int xferop_ret; size_t size; void *vaddr; diff --git a/drivers/soc/qcom/rmtfs_mem.c b/drivers/soc/qcom/rmtfs_mem.c index 0feaae357821..69991e47aa23 100644 --- a/drivers/soc/qcom/rmtfs_mem.c +++ b/drivers/soc/qcom/rmtfs_mem.c @@ -30,7 +30,7 @@ struct qcom_rmtfs_mem { unsigned int client_id; - unsigned int perms; + u64 perms; }; static ssize_t qcom_rmtfs_mem_show(struct device *dev, diff --git a/include/linux/qcom_scm.h b/include/linux/qcom_scm.h index f8335644a01a..77f7b5837216 100644 --- a/include/linux/qcom_scm.h +++ b/include/linux/qcom_scm.h @@ -96,7 +96,7 @@ extern int qcom_scm_mem_protect_video_var(u32 cp_start, u32 cp_size, u32 cp_nonpixel_start, u32 cp_nonpixel_size); extern int qcom_scm_assign_mem(phys_addr_t mem_addr, size_t mem_sz, - unsigned int *src, + u64 *src, const struct qcom_scm_vmperm *newvm, unsigned int dest_cnt); From patchwork Fri Jan 20 22:46:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 644677 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A9DCC38159 for ; Fri, 20 Jan 2023 22:49:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230106AbjATWtc (ORCPT ); Fri, 20 Jan 2023 17:49:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56982 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230123AbjATWtT (ORCPT ); Fri, 20 Jan 2023 17:49:19 -0500 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 65BDC4ED21; Fri, 20 Jan 2023 14:48:46 -0800 (PST) Received: from pps.filterd (m0279863.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KMfxBx005026; Fri, 20 Jan 2023 22:47:59 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=oPuaCrarO4oyPhQJzJXH1Z0yN/Q+wCwOvUWJSM7iWjA=; b=SABIp335W4srEe86Via4gLsYx6V691HsW9wGiCWg8AaQx06AixO+nUFhpXS8K53mfbPQ n6mOkQrOBcC1LRI4Rk18IqGDmhnQ8Ql9oWSKxddCzpvrhod3fvzP1jndJFEYKUrQIqJJ 0dgonYkuxTIrm17EbDYp7ccwrXWej/WpF2dE4Fs7VX6pAQwxdmOHQqfEYZIW0xl75lw0 DOCmCq99SXgrUj9OW+8nd7HwdOKrwSICBEI8mdprEWVBpFupWEnjlJNkxItrpJsZPva2 KMT/CpKQCUKHzKvK0wVq66yWdh2+OrMObrWq1DiryvnNQIVJw6FdTj5C2g1dYZcPNdwP yg== Received: from nasanppmta04.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n70eymkdv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:59 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA04.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMlwQI024342 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:58 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:57 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Andy Gross , Bjorn Andersson , Konrad Dybcio CC: Elliot Berman , Murali Nalajala , Trilok Soni , "Srivatsa Vaddagiri" , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , Greg Kroah-Hartman , Rob Herring , "Krzysztof Kozlowski" , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Will Deacon , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 18/27] firmware: qcom_scm: Register Gunyah platform ops Date: Fri, 20 Jan 2023 14:46:17 -0800 Message-ID: <20230120224627.4053418-19-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: 1To2etVPKnxLoraKqmvepaxuxWoO0VuE X-Proofpoint-GUID: 1To2etVPKnxLoraKqmvepaxuxWoO0VuE X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 impostorscore=0 spamscore=0 priorityscore=1501 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 clxscore=1015 mlxscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Qualcomm platforms have a firmware entity which performs access control to physical pages. Dynamically started Gunyah virtual machines use the QCOM_SCM_RM_MANAGED_VMID for access. Linux thus needs to assign access to the memory used by guest VMs. Gunyah doesn't do this operation for us since it is the current VM (typically VMID_HLOS) delegating the access and not Gunyah itself. Use the Gunyah platform ops to achieve this so that only Qualcomm platforms attempt to make the needed SCM calls. Co-developed-by: Prakruthi Deepak Heragu Signed-off-by: Prakruthi Deepak Heragu Signed-off-by: Elliot Berman --- drivers/firmware/Kconfig | 2 + drivers/firmware/qcom_scm.c | 100 ++++++++++++++++++++++++++++++++++++ 2 files changed, 102 insertions(+) diff --git a/drivers/firmware/Kconfig b/drivers/firmware/Kconfig index b59e3041fd62..b888068ff6f2 100644 --- a/drivers/firmware/Kconfig +++ b/drivers/firmware/Kconfig @@ -214,6 +214,8 @@ config MTK_ADSP_IPC config QCOM_SCM tristate + select VIRT_DRIVERS + select GUNYAH_PLATFORM_HOOKS config QCOM_SCM_DOWNLOAD_MODE_DEFAULT bool "Qualcomm download mode enabled by default" diff --git a/drivers/firmware/qcom_scm.c b/drivers/firmware/qcom_scm.c index 92763dce6477..20a1434087eb 100644 --- a/drivers/firmware/qcom_scm.c +++ b/drivers/firmware/qcom_scm.c @@ -17,6 +17,7 @@ #include #include #include +#include #include "qcom_scm.h" @@ -27,6 +28,9 @@ module_param(download_mode, bool, 0); #define SCM_HAS_IFACE_CLK BIT(1) #define SCM_HAS_BUS_CLK BIT(2) +#define QCOM_SCM_RM_MANAGED_VMID 0x3A +#define QCOM_SCM_MAX_MANAGED_VMID 0x3F + struct qcom_scm { struct device *dev; struct clk *core_clk; @@ -1292,6 +1296,99 @@ int qcom_scm_lmh_dcvsh(u32 payload_fn, u32 payload_reg, u32 payload_val, } EXPORT_SYMBOL(qcom_scm_lmh_dcvsh); +static int qcom_scm_gh_rm_pre_mem_share(struct gh_rm *rm, struct gh_rm_mem_parcel *mem_parcel) +{ + struct qcom_scm_vmperm *new_perms; + u64 src, src_cpy; + int ret = 0, i, n; + u16 vmid; + + new_perms = kcalloc(mem_parcel->n_acl_entries, sizeof(*new_perms), GFP_KERNEL); + if (!new_perms) + return -ENOMEM; + + for (n = 0; n < mem_parcel->n_acl_entries; n++) { + vmid = le16_to_cpu(mem_parcel->acl_entries[n].vmid); + if (vmid <= QCOM_SCM_MAX_MANAGED_VMID) + new_perms[n].vmid = vmid; + else + new_perms[n].vmid = QCOM_SCM_RM_MANAGED_VMID; + if (mem_parcel->acl_entries[n].perms & GH_RM_ACL_X) + new_perms[n].perm |= QCOM_SCM_PERM_EXEC; + if (mem_parcel->acl_entries[n].perms & GH_RM_ACL_W) + new_perms[n].perm |= QCOM_SCM_PERM_WRITE; + if (mem_parcel->acl_entries[n].perms & GH_RM_ACL_R) + new_perms[n].perm |= QCOM_SCM_PERM_READ; + } + + src = (1ull << QCOM_SCM_VMID_HLOS); + + for (i = 0; i < mem_parcel->n_mem_entries; i++) { + src_cpy = src; + ret = qcom_scm_assign_mem(le64_to_cpu(mem_parcel->mem_entries[i].ipa_base), + le64_to_cpu(mem_parcel->mem_entries[i].size), + &src_cpy, new_perms, mem_parcel->n_acl_entries); + if (ret) { + src = 0; + for (n = 0; n < mem_parcel->n_acl_entries; n++) { + vmid = le16_to_cpu(mem_parcel->acl_entries[n].vmid); + if (vmid <= QCOM_SCM_MAX_MANAGED_VMID) + src |= (1ull << vmid); + else + src |= (1ull << QCOM_SCM_RM_MANAGED_VMID); + } + + new_perms[0].vmid = QCOM_SCM_VMID_HLOS; + + for (i--; i >= 0; i--) { + src_cpy = src; + ret = qcom_scm_assign_mem( + le64_to_cpu(mem_parcel->mem_entries[i].ipa_base), + le64_to_cpu(mem_parcel->mem_entries[i].size), + &src_cpy, new_perms, 1); + WARN_ON_ONCE(ret); + } + break; + } + } + + kfree(new_perms); + return ret; +} + +static int qcom_scm_gh_rm_post_mem_reclaim(struct gh_rm *rm, struct gh_rm_mem_parcel *mem_parcel) +{ + struct qcom_scm_vmperm new_perms; + u64 src = 0; + int ret = 0, i, n; + u16 vmid; + + new_perms.vmid = QCOM_SCM_VMID_HLOS; + new_perms.perm = QCOM_SCM_PERM_EXEC | QCOM_SCM_PERM_WRITE | QCOM_SCM_PERM_READ; + + for (n = 0; n < mem_parcel->n_acl_entries; n++) { + vmid = le16_to_cpu(mem_parcel->acl_entries[n].vmid); + if (vmid <= QCOM_SCM_MAX_MANAGED_VMID) + src |= (1ull << vmid); + else + src |= (1ull << QCOM_SCM_RM_MANAGED_VMID); + } + + for (i = 0; i < mem_parcel->n_mem_entries; i++) { + ret = qcom_scm_assign_mem(le64_to_cpu(mem_parcel->mem_entries[i].ipa_base), + le64_to_cpu(mem_parcel->mem_entries[i].size), + &src, &new_perms, 1); + WARN_ON_ONCE(ret); + } + + return ret; +} + +static struct gunyah_rm_platform_ops qcom_scm_gh_rm_platform_ops = { + .pre_mem_share = qcom_scm_gh_rm_pre_mem_share, + .post_mem_reclaim = qcom_scm_gh_rm_post_mem_reclaim, +}; + static int qcom_scm_find_dload_address(struct device *dev, u64 *addr) { struct device_node *tcsr; @@ -1414,6 +1511,9 @@ static int qcom_scm_probe(struct platform_device *pdev) if (download_mode) qcom_scm_set_download_mode(true); + if (gh_rm_register_platform_ops(&qcom_scm_gh_rm_platform_ops)) + dev_warn(__scm->dev, "Gunyah RM platform ops were already registered\n"); + return 0; } From patchwork Fri Jan 20 22:46:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 645104 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97916C27C76 for ; Fri, 20 Jan 2023 22:49:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229826AbjATWtG (ORCPT ); Fri, 20 Jan 2023 17:49:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56800 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230026AbjATWso (ORCPT ); Fri, 20 Jan 2023 17:48:44 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 56FA8743BB; Fri, 20 Jan 2023 14:48:15 -0800 (PST) Received: from pps.filterd (m0279873.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KLS46r026515; Fri, 20 Jan 2023 22:48:00 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=XFJLx/WXuFDs7JAeLccnha22C3yPfKyzMwCj8d45lFY=; b=Cy9CiLCQbVEvjzMMG3Xu5gouVX+/ExPvfj5z1gR/bJ7EXSJrMvaU1CpzWuLXRBug+d4I jJ5oKL73vlkvSfMJFqWAmlWkT74T6s93g8IMJCKTo0+vN24dOYaxALE4oXyWPf5iqJ8s cfKpOjw/T6X+BqbBalOrS+v9hm0h8KLuMLeqQoJPeLVIybj1xdMMfO5Xd3g2ntieJhFd h0Hpc9tVCPgQazbWlb5aut9ni4wef/dO1gjGOFc7WS2B4b9MT0t5U2dK5Iu935+0qCs7 gYUJZDG7krIsvqLNR9dR23CUg48rHItPipu/bjm5a6DPj8pfKFoiOzDv88kPeunMVlpD gg== Received: from nasanppmta02.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7593uqye-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:48:00 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA02.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMlx00029311 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:47:59 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:47:58 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala , Jonathan Corbet CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Bagas Sanjaya , Catalin Marinas , "Will Deacon" , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 19/27] docs: gunyah: Document Gunyah VM Manager Date: Fri, 20 Jan 2023 14:46:18 -0800 Message-ID: <20230120224627.4053418-20-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: 68mAisyA32viiDKsF17vki1pwGe4-6pL X-Proofpoint-ORIG-GUID: 68mAisyA32viiDKsF17vki1pwGe4-6pL X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 mlxlogscore=999 spamscore=0 suspectscore=0 mlxscore=0 adultscore=0 clxscore=1015 impostorscore=0 priorityscore=1501 bulkscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Document the ioctls and usage of Gunyah VM Manager driver. Signed-off-by: Elliot Berman --- Documentation/virt/gunyah/index.rst | 1 + Documentation/virt/gunyah/vm-manager.rst | 100 +++++++++++++++++++++++ 2 files changed, 101 insertions(+) create mode 100644 Documentation/virt/gunyah/vm-manager.rst diff --git a/Documentation/virt/gunyah/index.rst b/Documentation/virt/gunyah/index.rst index 45adbbc311db..b204b85e86db 100644 --- a/Documentation/virt/gunyah/index.rst +++ b/Documentation/virt/gunyah/index.rst @@ -7,6 +7,7 @@ Gunyah Hypervisor .. toctree:: :maxdepth: 1 + vm-manager message-queue Gunyah is a Type-1 hypervisor which is independent of any OS kernel, and runs in diff --git a/Documentation/virt/gunyah/vm-manager.rst b/Documentation/virt/gunyah/vm-manager.rst new file mode 100644 index 000000000000..62513af09cbf --- /dev/null +++ b/Documentation/virt/gunyah/vm-manager.rst @@ -0,0 +1,100 @@ +.. SPDX-License-Identifier: GPL-2.0 + +======================= +Virtual Machine Manager +======================= + +The Gunyah Virtual Machine Manager is a Linux driver to support launching +virtual machines using Gunyah. It presently supports launching non-proxy +scheduled Linux-like virtual machines. + +Except for some basic information about the location of initial binaries, +most of the configuration about a Gunyah virtual machine is described in the +VM's devicetree. The devicetree is generated by userspace. Interacting with the +virtual machine is still done via the kernel and VM configuration requires some +of the corresponding functionality to be set up in the kernel. For instance, +sharing userspace memory with a VM is done via the GH_VM_SET_USER_MEM_REGION +ioctl. The VM itself is configured to use the memory region via the +devicetree. + +Sample Userspace VMM +==================== + +A sample userspace VMM is included in samples/gunyah/ along with a minimal +devicetree that can be used to launch a VM. To build this sample, enable +CONFIG_SAMPLE_GUNYAH. + +IOCTLs and userspace VMM flows +============================== + +The kernel exposes a char device interface at /dev/gunyah. + +To create a VM, use the GH_CREATE_VM ioctl. A successful call will return a +"Gunyah VM" file descriptor. + +/dev/gunyah API Descriptions +---------------------------- + +GH_CREATE_VM +~~~~~~~~~~~~ + +Creates a Gunyah VM. The argument is reserved for future use and must be 0. + +Gunyah VM API Descriptions +-------------------------- + +GH_VM_SET_USER_MEM_REGION +~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + struct gh_userspace_memory_region { + __u32 label; + __u32 flags; + __u64 guest_phys_addr; + __u64 memory_size; + __u64 userspace_addr; + }; + +This ioctl allows the user to create or delete a memory parcel for a guest +virtual machine. Each memory region is uniquely identified by a label; +attempting to create two regions with the same label is not allowed. + +While VMM is guest-agnostic and allows runtime addition of memory regions, +Linux guest virtual machines do not support accepting memory regions at runtime. +Thus, memory regions should be provided before starting the VM and the VM must +be configured to accept these at boot-up. + +The guest physical address is used by Linux kernel to check that the requested +user regions do not overlap and to help find the corresponding memory region +for calls like GH_VM_SET_DTB_CONFIG. + +To delete a memory region, call GH_VM_SET_USER_MEM_REGION with label set to the +desired region and memory_size set to 0. + +The flags field of gh_userspace_memory_region accepts the following bits. All +other bits must be 0 and are reserved for future use. The ioctl will return +-EINVAL if an unsupported bit is detected. + + - GH_MEM_ALLOW_READ/GH_MEM_ALLOW_WRITE/GH_MEM_ALLOW_EXEC sets read/write/exec + permissions for the guest, respectively. + - GH_MEM_LENT means that the memory will be unmapped from the host and be + unaccessible by the host while the guest has the region. + +GH_VM_SET_DTB_CONFIG +~~~~~~~~~~~~~~~~~~~~ + +:: + + struct gh_vm_dtb_config { + __u64 gpa; + __u64 size; + }; + +This ioctl sets the location of the VM's devicetree blob and is used by Gunyah +Resource Manager to allocate resources. + +GH_VM_START +~~~~~~~~~~~ + +This ioctl starts the VM. From patchwork Fri Jan 20 22:46:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 644675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60B12C05027 for ; Fri, 20 Jan 2023 22:50:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230050AbjATWuH (ORCPT ); Fri, 20 Jan 2023 17:50:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58296 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229902AbjATWt5 (ORCPT ); Fri, 20 Jan 2023 17:49:57 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5115B71794; Fri, 20 Jan 2023 14:49:14 -0800 (PST) Received: from pps.filterd (m0279868.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KMaweS014407; Fri, 20 Jan 2023 22:48:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=BXEnrXwGqTI3T1+o30DwR+yNP1d/WEZFOXygxs9Y0i8=; b=ADCNWxZoWV59UDRYWq4rF7anCGhhgsL/t1sjwmJKC51V3q+lWhHmXHAgK094SCKu26uj q+peRENGy7F6VBZ4Kqiw6UGY3ySQYPie4id/XRrn0ohEVurG5rvuaz+0F7CLS9ZQ6vm6 /v3lVYWvihY7YXw0YYdVgMDypgdWlbdkCiPgrOqx0bxralurseOPXw4/XjwRZu29zN4K PnBj2fRFZahnlQ7+n3LCAM/ELia+RHBx8VJbIIoumP6WTDPmqRLyNLYQRvxwHqrnn2o2 vCUSBBJvqUaqcN8gYHXXVzlltXcQplyr5qZQanbIFjn0XV65umAukq2/32Zx9wHPEqm+ nA== Received: from nasanppmta05.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n75w3kmxb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:48:02 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA05.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMm1S2032328 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:48:01 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:48:00 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Catalin Marinas , Will Deacon , Elliot Berman , Murali Nalajala CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 20/27] virt: gunyah: Translate gh_rm_hyp_resource into gunyah_resource Date: Fri, 20 Jan 2023 14:46:19 -0800 Message-ID: <20230120224627.4053418-21-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: qxB0EkfttoQjpmgn9t4K3XNc4oSlS_wl X-Proofpoint-GUID: qxB0EkfttoQjpmgn9t4K3XNc4oSlS_wl X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=992 impostorscore=0 lowpriorityscore=0 adultscore=0 clxscore=1015 mlxscore=0 spamscore=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org When booting a Gunyah virtual machine, the host VM may gain capabilities to interact with resources for the guest virtual machine. Examples of such resources are vCPUs or message queues. To use those resources, we need to translate the RM response into a gunyah_resource structure which are useful to Linux drivers. Presently, Linux drivers need only to know the type of resource, the capability ID, and an interrupt. On ARM64 systems, the interrupt reported by Gunyah is the GIC interrupt ID number and always a SPI. Signed-off-by: Elliot Berman --- arch/arm64/include/asm/gunyah.h | 23 +++++ drivers/virt/gunyah/rsc_mgr.c | 170 +++++++++++++++++++++++++++++++- include/linux/gunyah_rsc_mgr.h | 4 + 3 files changed, 196 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/include/asm/gunyah.h diff --git a/arch/arm64/include/asm/gunyah.h b/arch/arm64/include/asm/gunyah.h new file mode 100644 index 000000000000..64cfb964efee --- /dev/null +++ b/arch/arm64/include/asm/gunyah.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved. + */ +#ifndef __ASM_GUNYAH_H_ +#define __ASM_GUNYAH_H_ + +#include +#include + +static inline int arch_gh_fill_irq_fwspec_params(u32 virq, struct irq_fwspec *fwspec) +{ + if (virq < 32 || virq > 1019) + return -EINVAL; + + fwspec->param_count = 3; + fwspec->param[0] = GIC_SPI; + fwspec->param[1] = virq - 32; + fwspec->param[2] = IRQ_TYPE_EDGE_RISING; + return 0; +} + +#endif diff --git a/drivers/virt/gunyah/rsc_mgr.c b/drivers/virt/gunyah/rsc_mgr.c index 95f4aa928aaf..b9ee8fa71ccf 100644 --- a/drivers/virt/gunyah/rsc_mgr.c +++ b/drivers/virt/gunyah/rsc_mgr.c @@ -18,6 +18,8 @@ #include #include +#include + #include "rsc_mgr.h" #include "vm_mgr.h" @@ -108,9 +110,146 @@ struct gh_rm { struct mutex send_lock; struct miscdevice miscdev; + struct irq_domain *irq_domain; + struct work_struct recv_work; }; +struct gh_irq_chip_data { + u32 gh_virq; +}; + +static struct irq_chip gh_rm_irq_chip = { + .name = "Gunyah", + .irq_enable = irq_chip_enable_parent, + .irq_disable = irq_chip_disable_parent, + .irq_ack = irq_chip_ack_parent, + .irq_mask = irq_chip_mask_parent, + .irq_mask_ack = irq_chip_mask_ack_parent, + .irq_unmask = irq_chip_unmask_parent, + .irq_eoi = irq_chip_eoi_parent, + .irq_set_affinity = irq_chip_set_affinity_parent, + .irq_set_type = irq_chip_set_type_parent, + .irq_set_wake = irq_chip_set_wake_parent, + .irq_set_vcpu_affinity = irq_chip_set_vcpu_affinity_parent, + .irq_retrigger = irq_chip_retrigger_hierarchy, + .irq_get_irqchip_state = irq_chip_get_parent_state, + .irq_set_irqchip_state = irq_chip_set_parent_state, + .flags = IRQCHIP_SET_TYPE_MASKED | + IRQCHIP_SKIP_SET_WAKE | + IRQCHIP_MASK_ON_SUSPEND, +}; + +static int gh_rm_irq_domain_alloc(struct irq_domain *d, unsigned int virq, unsigned int nr_irqs, + void *arg) +{ + struct irq_fwspec parent_fwspec; + struct gh_irq_chip_data *chip_data, *spec = arg; + u32 gh_virq = spec->gh_virq; + struct gh_rm *rm = d->host_data; + int ret; + + if (nr_irqs != 1 || gh_virq == U32_MAX) + return -EINVAL; + + chip_data = kzalloc(sizeof(*chip_data), GFP_KERNEL); + if (!chip_data) + return -ENOMEM; + + chip_data->gh_virq = gh_virq; + + ret = irq_domain_set_hwirq_and_chip(d, virq, chip_data->gh_virq, &gh_rm_irq_chip, + chip_data); + if (ret) + return ret; + + parent_fwspec.fwnode = d->parent->fwnode; + ret = arch_gh_fill_irq_fwspec_params(chip_data->gh_virq, &parent_fwspec); + if (ret) { + dev_err(rm->dev, "virq translation failed %u: %d\n", chip_data->gh_virq, ret); + goto err_free_irq_data; + } + + ret = irq_domain_alloc_irqs_parent(d, virq, nr_irqs, &parent_fwspec); + if (ret) + goto err_free_irq_data; + + return ret; +err_free_irq_data: + kfree(chip_data); + return ret; +} + +static void gh_rm_irq_domain_free_single(struct irq_domain *d, unsigned int virq) +{ + struct irq_data *irq_data; + struct gh_irq_chip_data *chip_data; + + irq_data = irq_domain_get_irq_data(d, virq); + if (!irq_data) + return; + + chip_data = irq_data->chip_data; + + kfree(chip_data); + irq_data->chip_data = NULL; +} + +static void gh_rm_irq_domain_free(struct irq_domain *d, unsigned int virq, unsigned int nr_irqs) +{ + unsigned int i; + + for (i = 0; i < nr_irqs; i++) + gh_rm_irq_domain_free_single(d, virq); +} + +static const struct irq_domain_ops gh_rm_irq_domain_ops = { + .alloc = gh_rm_irq_domain_alloc, + .free = gh_rm_irq_domain_free, +}; + +struct gunyah_resource *gh_rm_alloc_resource(struct gh_rm *rm, + struct gh_rm_hyp_resource *hyp_resource) +{ + struct gunyah_resource *ghrsc; + + ghrsc = kzalloc(sizeof(*ghrsc), GFP_KERNEL); + if (!ghrsc) + return NULL; + + ghrsc->type = hyp_resource->type; + ghrsc->capid = le64_to_cpu(hyp_resource->cap_id); + ghrsc->irq = IRQ_NOTCONNECTED; + ghrsc->rm_label = le32_to_cpu(hyp_resource->resource_label); + /* Gunyah doesn't send us virq_handle anymore and will always give us an interrupt + * that's ready to be used. Defensively check that it's not set. + */ + if (hyp_resource->virq_handle) { + pr_warn_once("Unexpected virq handle.\n"); + } else if (hyp_resource->virq && le32_to_cpu(hyp_resource->virq) != U32_MAX) { + struct gh_irq_chip_data irq_data = { + .gh_virq = le32_to_cpu(hyp_resource->virq), + }; + + ghrsc->irq = irq_domain_alloc_irqs(rm->irq_domain, 1, NUMA_NO_NODE, &irq_data); + if (ghrsc->irq < 0) { + pr_err("Failed to allocate interrupt for resource %d label: %d: %d\n", + ghrsc->type, ghrsc->rm_label, ghrsc->irq); + ghrsc->irq = IRQ_NOTCONNECTED; + } + } + + return ghrsc; +} +EXPORT_SYMBOL_GPL(gh_rm_alloc_resource); + +void gh_rm_free_resource(struct gunyah_resource *ghrsc) +{ + irq_dispose_mapping(ghrsc->irq); + kfree(ghrsc); +} +EXPORT_SYMBOL_GPL(gh_rm_free_resource); + static struct gh_rm_connection *gh_rm_alloc_connection(__le32 msg_id, u8 type) { struct gh_rm_connection *connection; @@ -582,6 +721,9 @@ static int gh_msgq_platform_probe_direction(struct platform_device *pdev, static int gh_rm_drv_probe(struct platform_device *pdev) { struct gh_rm *rm; + struct device_node *parent_irq_node; + struct irq_domain *parent_irq_domain; + struct gh_msgq_tx_data *msg; int ret; rm = devm_kzalloc(&pdev->dev, sizeof(*rm), GFP_KERNEL); @@ -616,15 +758,40 @@ static int gh_rm_drv_probe(struct platform_device *pdev) if (ret) goto err_cache; + parent_irq_node = of_irq_find_parent(pdev->dev.of_node); + if (!parent_irq_node) { + dev_err(&pdev->dev, "Failed to find interrupt parent of resource manager\n"); + ret = -ENODEV; + goto err_msgq; + } + + parent_irq_domain = irq_find_host(parent_irq_node); + if (!parent_irq_domain) { + dev_err(&pdev->dev, "Failed to find interrupt parent domain of resource manager\n"); + ret = -ENODEV; + goto err_msgq; + } + + rm->irq_domain = irq_domain_add_hierarchy(parent_irq_domain, 0, 0, pdev->dev.of_node, + &gh_rm_irq_domain_ops, NULL); + if (!rm->irq_domain) { + dev_err(&pdev->dev, "Failed to add irq domain\n"); + ret = -ENODEV; + goto err_msgq; + } + rm->irq_domain->host_data = rm; + rm->miscdev.name = "gunyah"; rm->miscdev.minor = MISC_DYNAMIC_MINOR; rm->miscdev.fops = &gh_dev_fops; ret = misc_register(&rm->miscdev); if (ret) - goto err_msgq; + goto err_irq_domain; return 0; +err_irq_domain: + irq_domain_remove(rm->irq_domain); err_msgq: mbox_free_channel(gh_msgq_chan(&rm->msgq)); gh_msgq_remove(&rm->msgq); @@ -638,6 +805,7 @@ static int gh_rm_drv_remove(struct platform_device *pdev) struct gh_rm *rm = platform_get_drvdata(pdev); misc_deregister(&rm->miscdev); + irq_domain_remove(rm->irq_domain); mbox_free_channel(gh_msgq_chan(&rm->msgq)); gh_msgq_remove(&rm->msgq); kmem_cache_destroy(rm->cache); diff --git a/include/linux/gunyah_rsc_mgr.h b/include/linux/gunyah_rsc_mgr.h index dd94bb3f378b..39d0c3d2dbc1 100644 --- a/include/linux/gunyah_rsc_mgr.h +++ b/include/linux/gunyah_rsc_mgr.h @@ -120,6 +120,10 @@ int gh_rm_get_hyp_resources(struct gh_rm *rm, u16 vmid, struct gh_rm_hyp_resources **resources); int gh_rm_get_vmid(struct gh_rm *rm, u16 *vmid); +struct gunyah_resource *gh_rm_alloc_resource(struct gh_rm *rm, + struct gh_rm_hyp_resource *hyp_resource); +void gh_rm_free_resource(struct gunyah_resource *ghrsc); + struct gunyah_rm_platform_ops { int (*pre_mem_share)(struct gh_rm *rm, struct gh_rm_mem_parcel *mem_parcel); int (*post_mem_reclaim)(struct gh_rm *rm, struct gh_rm_mem_parcel *mem_parcel); From patchwork Fri Jan 20 22:46:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 644674 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB62FC54794 for ; Fri, 20 Jan 2023 22:50:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230215AbjATWuI (ORCPT ); Fri, 20 Jan 2023 17:50:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229934AbjATWuG (ORCPT ); Fri, 20 Jan 2023 17:50:06 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C263D4F35D; Fri, 20 Jan 2023 14:49:18 -0800 (PST) Received: from pps.filterd (m0279873.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KMb1WQ007055; Fri, 20 Jan 2023 22:48:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=ytZx091v+Zeou4MbVlh64449R9MyK4MSZfSGoF9kPNA=; b=Hko1lbODhT5XCfdtM4oC/kAUpW2zrdCeVui+M/IO5xUhCpKWhvTCvB1EWypKuDC8+a9z Nodhhsenr+G0/fhv/H+otxPJGv/Mobcat5X2xLpDcjX2UWWxHPnf/31ctlrGzRXUV2iE X6kf7hfIQLTG8hzEbJRgY6fjjPL+ZY1AFcxku0emNv9rITH/3VaYHD5Cwz2CXfkINfal NXwKxEMReWcm+sGN2xGMR1cZVxIrIiR9EOFWlOlzfazGFdL5ZNhBAzfjLQ+bIbhGxGDO Hf/dHPzyqB1CMIKHf9lahIoKtz23xCYw3hJpbZj0KsjGgvZZVk1TfS+QNCJdMB4yOXfe QQ== Received: from nasanppmta04.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7593uqyk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:48:03 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA04.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMm2Zq024408 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:48:02 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:48:02 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala , Jonathan Corbet CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Bagas Sanjaya , Catalin Marinas , "Will Deacon" , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 21/27] gunyah: vm_mgr: Add framework to add VM Functions Date: Fri, 20 Jan 2023 14:46:20 -0800 Message-ID: <20230120224627.4053418-22-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: XI3-3LRiBfYBc8VR1CiCD0_diTlYn1hB X-Proofpoint-ORIG-GUID: XI3-3LRiBfYBc8VR1CiCD0_diTlYn1hB X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 mlxlogscore=999 spamscore=0 suspectscore=0 mlxscore=0 adultscore=0 clxscore=1015 impostorscore=0 priorityscore=1501 bulkscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Introduce a framework for Gunyah userspace to install VM functions. VM functions are optional interfaces to the virtual machine. vCPUs, ioeventfs, and irqfds are examples of such VM functions and are implemented in subsequent patches. A generic framework is implemented instead of individual ioctls to create vCPUs, irqfds, etc., in order to simplify the VM manager core implementation and allow dynamic loading of VM function modules. Signed-off-by: Elliot Berman --- Documentation/virt/gunyah/vm-manager.rst | 18 +++ drivers/virt/gunyah/vm_mgr.c | 187 ++++++++++++++++++++++- drivers/virt/gunyah/vm_mgr.h | 5 + include/linux/gunyah_vm_mgr.h | 68 +++++++++ include/uapi/linux/gunyah.h | 13 ++ 5 files changed, 289 insertions(+), 2 deletions(-) create mode 100644 include/linux/gunyah_vm_mgr.h diff --git a/Documentation/virt/gunyah/vm-manager.rst b/Documentation/virt/gunyah/vm-manager.rst index 62513af09cbf..d929f8f20b3b 100644 --- a/Documentation/virt/gunyah/vm-manager.rst +++ b/Documentation/virt/gunyah/vm-manager.rst @@ -17,6 +17,24 @@ sharing userspace memory with a VM is done via the GH_VM_SET_USER_MEM_REGION ioctl. The VM itself is configured to use the memory region via the devicetree. +Gunyah Functions +================ + +Components of a Gunyah VM's configuration that need kernel configuration are +called "functions" and are built on top of a framework. Functions are identified +by a string and have some argument(s) to configure them. They are typically +created by the `GH_VM_ADD_FUNCTION` ioctl. + +Functions typically will always do at least one of these operations: + +1. Create resource ticket(s). Resource tickets allow a function to register + itself as the client for a Gunyah resource (e.g. doorbell or vCPU) and + the function is given the pointer to the `struct gunyah_resource` when the + VM is starting. + +2. Register IO handler(s). IO handlers allow a function to handle stage-2 faults + from the virtual machine. + Sample Userspace VMM ==================== diff --git a/drivers/virt/gunyah/vm_mgr.c b/drivers/virt/gunyah/vm_mgr.c index 48bd3f06fb6c..1e795f3d19d5 100644 --- a/drivers/virt/gunyah/vm_mgr.c +++ b/drivers/virt/gunyah/vm_mgr.c @@ -16,7 +16,147 @@ #include "vm_mgr.h" -static __must_check struct gunyah_vm *gunyah_vm_alloc(struct gh_rm_rpc *rm) +static DEFINE_MUTEX(functions_lock); +static LIST_HEAD(functions); + +static struct gunyah_vm_function_driver *__find_function(const char name[GUNYAH_FUNCTION_NAME_SIZE]) + __must_hold(functions_lock) +{ + struct gunyah_vm_function_driver *iter, *drv = NULL; + + list_for_each_entry(iter, &functions, list) { + if (!strncmp(iter->name, name, GUNYAH_FUNCTION_NAME_SIZE)) { + drv = iter; + break; + } + } + + return drv; +} + +int gunyah_vm_function_register(struct gunyah_vm_function_driver *drv) +{ + int ret = 0; + + mutex_lock(&functions_lock); + if (__find_function(drv->name)) { + ret = -EEXIST; + goto out; + } + + INIT_LIST_HEAD(&drv->instances); + list_add(&drv->list, &functions); +out: + mutex_unlock(&functions_lock); + return ret; +} +EXPORT_SYMBOL_GPL(gunyah_vm_function_register); + +static void gh_vm_remove_function(struct gunyah_vm_function *f) + __must_hold(functions_lock) +{ + f->drv->release(f); + list_del(&f->vm_list); + list_del(&f->drv_list); + module_put(f->drv->mod); + kfree(f); +} + +void gunyah_vm_function_unregister(struct gunyah_vm_function_driver *drv) +{ + struct gunyah_vm_function *f, *iter; + + mutex_lock(&functions_lock); + list_for_each_entry_safe(f, iter, &drv->instances, drv_list) + gh_vm_remove_function(f); + list_del(&drv->list); + mutex_unlock(&functions_lock); +} +EXPORT_SYMBOL_GPL(gunyah_vm_function_unregister); + +static long gh_vm_add_function(struct gunyah_vm *ghvm, struct gunyah_vm_function *f) +{ + long r = 0; + + mutex_lock(&functions_lock); + f->drv = __find_function(f->fn.name); + if (!f->drv) { + mutex_unlock(&functions_lock); + r = request_module("ghfunc:%s", f->fn.name); + if (r) + return r; + + mutex_lock(&functions_lock); + f->drv = __find_function(f->fn.name); + } + + if (!f->drv) { + r = -ENOENT; + goto out; + } + + if (!try_module_get(f->drv->mod)) { + r = -ENOENT; + f->drv = NULL; + goto out; + } + + f->ghvm = ghvm; + f->rm = ghvm->rm; + + r = f->drv->bind(f); + if (r < 0) { + module_put(f->drv->mod); + goto out; + } + + list_add(&f->vm_list, &ghvm->functions); + list_add(&f->drv_list, &f->drv->instances); +out: + mutex_unlock(&functions_lock); + return r; +} + +static long gh_vm_rm_function(struct gunyah_vm *ghvm, struct gh_vm_function *fn) +{ + long r = 0; + struct gunyah_vm_function *f, *iter; + + r = mutex_lock_interruptible(&functions_lock); + if (r) + return r; + + list_for_each_entry_safe(f, iter, &ghvm->functions, vm_list) { + if (!memcmp(&f->fn, fn, sizeof(*fn))) + gh_vm_remove_function(f); + } + + mutex_unlock(&functions_lock); + return 0; +} + +static void ghvm_put(struct kref *kref) +{ + struct gunyah_vm *ghvm = container_of(kref, struct gunyah_vm, kref); + + gh_rm_dealloc_vmid(ghvm->rm, ghvm->vmid); + put_gh_rm(ghvm->rm); + kfree(ghvm); +} + +int __must_check get_gunyah_vm(struct gunyah_vm *ghvm) +{ + return kref_get_unless_zero(&ghvm->kref); +} +EXPORT_SYMBOL_GPL(get_gunyah_vm); + +void put_gunyah_vm(struct gunyah_vm *ghvm) +{ + kref_put(&ghvm->kref, ghvm_put); +} +EXPORT_SYMBOL_GPL(put_gunyah_vm); + +static __must_check struct gunyah_vm *gunyah_vm_alloc(struct gh_rm *rm) { struct gunyah_vm *ghvm; int vmid; @@ -39,6 +179,8 @@ static __must_check struct gunyah_vm *gunyah_vm_alloc(struct gh_rm_rpc *rm) mutex_init(&ghvm->mm_lock); INIT_LIST_HEAD(&ghvm->memory_mappings); init_rwsem(&ghvm->status_lock); + kref_init(&ghvm->kref); + INIT_LIST_HEAD(&ghvm->functions); return ghvm; } @@ -192,6 +334,39 @@ static long gh_vm_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) r = -EINVAL; break; } + case GH_VM_ADD_FUNCTION: { + struct gunyah_vm_function *f; + + r = -ENOMEM; + f = kzalloc(sizeof(*f), GFP_KERNEL); + if (!f) + break; + + r = -EFAULT; + if (copy_from_user(&f->fn, argp, sizeof(f->fn))) + break; + + r = gh_vm_add_function(ghvm, f); + if (r < 0) + kfree(f); + break; + } + case GH_VM_REMOVE_FUNCTION: { + struct gh_vm_function *fn; + + r = -ENOMEM; + fn = kzalloc(sizeof(*fn), GFP_KERNEL); + if (!fn) + break; + + r = -EFAULT; + if (copy_from_user(fn, argp, sizeof(*fn))) + break; + + r = gh_vm_rm_function(ghvm, fn); + kfree(fn); + break; + } default: r = -ENOTTY; break; @@ -204,15 +379,23 @@ static int gh_vm_release(struct inode *inode, struct file *filp) { struct gunyah_vm *ghvm = filp->private_data; struct gunyah_vm_memory_mapping *mapping, *tmp; + struct gunyah_vm_function *f, *fiter; gh_vm_stop(ghvm); + mutex_lock(&functions_lock); + list_for_each_entry_safe(f, fiter, &ghvm->functions, vm_list) { + gh_vm_remove_function(f); + } + mutex_unlock(&functions_lock); + list_for_each_entry_safe(mapping, tmp, &ghvm->memory_mappings, list) { gh_vm_mem_mapping_reclaim(ghvm, mapping); kfree(mapping); } + put_gh_rm(ghvm->rm); - kfree(ghvm); + put_gunyah_vm(ghvm); return 0; } diff --git a/drivers/virt/gunyah/vm_mgr.h b/drivers/virt/gunyah/vm_mgr.h index 5c02fb305893..8d3b0678fb96 100644 --- a/drivers/virt/gunyah/vm_mgr.h +++ b/drivers/virt/gunyah/vm_mgr.h @@ -8,6 +8,7 @@ #include #include +#include #include #include #include @@ -41,8 +42,12 @@ struct gunyah_vm { enum gh_rm_vm_status vm_status; struct rw_semaphore status_lock; + struct kref kref; + struct mutex mm_lock; struct list_head memory_mappings; + + struct list_head functions; }; struct gunyah_vm_memory_mapping *gh_vm_mem_mapping_alloc(struct gunyah_vm *ghvm, diff --git a/include/linux/gunyah_vm_mgr.h b/include/linux/gunyah_vm_mgr.h new file mode 100644 index 000000000000..69f98eb503e9 --- /dev/null +++ b/include/linux/gunyah_vm_mgr.h @@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#ifndef _GUNYAH_VM_MGR_H +#define _GUNYAH_VM_MGR_H + +#include +#include +#include +#include +#include +#include + +#include + +struct gunyah_vm; + +int __must_check get_gunyah_vm(struct gunyah_vm *ghvm); +void put_gunyah_vm(struct gunyah_vm *ghvm); + +struct gunyah_vm_function; +struct gunyah_vm_function_driver { + const char name[GUNYAH_FUNCTION_NAME_SIZE]; + struct module *mod; + long (*bind)(struct gunyah_vm_function *f); + void (*release)(struct gunyah_vm_function *f); + struct list_head list; + struct list_head instances; +}; + +struct gunyah_vm_function { + struct gh_vm_function fn; + struct gunyah_vm *ghvm; + struct gh_rm *rm; + struct gunyah_vm_function_driver *drv; + void *data; + struct list_head vm_list; + struct list_head drv_list; +}; + +int gunyah_vm_function_register(struct gunyah_vm_function_driver *f); +void gunyah_vm_function_unregister(struct gunyah_vm_function_driver *f); + +#define DECLARE_GUNYAH_VM_FUNCTION(_name, _bind, _release) \ + static struct gunyah_vm_function_driver _name = { \ + .name = __stringify(_name), \ + .mod = THIS_MODULE, \ + .bind = _bind, \ + .release = _release, \ + }; \ + MODULE_ALIAS("ghfunc:"__stringify(_name)) + +#define DECLARE_GUNYAH_VM_FUNCTION_INIT(_name, _bind, _release) \ + DECLARE_GUNYAH_VM_FUNCTION(_name, _bind, _release); \ + static int __init _name##_mod_init(void) \ + { \ + return gunyah_vm_function_register(&(_name)); \ + } \ + module_init(_name##_mod_init); \ + static void __exit _name##_mod_exit(void) \ + { \ + gunyah_vm_function_unregister(&(_name)); \ + } \ + module_exit(_name##_mod_exit) + +#endif diff --git a/include/uapi/linux/gunyah.h b/include/uapi/linux/gunyah.h index 36359ad2175e..ec8da6fde045 100644 --- a/include/uapi/linux/gunyah.h +++ b/include/uapi/linux/gunyah.h @@ -50,4 +50,17 @@ struct gh_vm_dtb_config { #define GH_VM_START _IO(GH_IOCTL_TYPE, 0x3) +#define GUNYAH_FUNCTION_NAME_SIZE 32 +#define GUNYAH_FUNCTION_MAX_ARG_SIZE 1024 + +struct gh_vm_function { + char name[GUNYAH_FUNCTION_NAME_SIZE]; + union { + char data[GUNYAH_FUNCTION_MAX_ARG_SIZE]; + }; +}; + +#define GH_VM_ADD_FUNCTION _IOW(GH_IOCTL_TYPE, 0x4, struct gh_vm_function) +#define GH_VM_REMOVE_FUNCTION _IOW(GH_IOCTL_TYPE, 0x7, struct gh_vm_function) + #endif From patchwork Fri Jan 20 22:46:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 644672 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 846C3C27C76 for ; Fri, 20 Jan 2023 23:12:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229669AbjATXM5 (ORCPT ); Fri, 20 Jan 2023 18:12:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51394 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229591AbjATXM4 (ORCPT ); Fri, 20 Jan 2023 18:12:56 -0500 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 51A0139BB2; Fri, 20 Jan 2023 15:12:55 -0800 (PST) Received: from pps.filterd (m0279862.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KMDFiY020834; Fri, 20 Jan 2023 22:48:05 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=7HbBt3e0a1vAlc9eCKkIzVLn1j5GxGmq0P3U4Vw3h48=; b=ZN9RAm/VAbnLc4zWttEDFDOsbtzD6gDxw7/fXXnQFtwtuktwIxFlQ3T3+MCeeo3l8vsA 4qPXBwfCV5/58CSSGHF3dg9N0kSEgOIBP2bQVk3YMOGLHSH2R+YsT1pyzvR4YYsagHoM tW6cuKX2GXhwldS4bZpQMu9WGEuIf9YNGNJBxAikOrmTfkKWWM9eHzjRpm8C+METidgI eqCKysPkJtNygXf+tYMNrNfbIdQ6TOHkPbKSzESwf112nzwUFlJtTAEDw6ZgIqj9XwJO rEsZq0i8KsJpRIcAfFsSHbmWzfda7MuIiuZ5yy9Rl2oyPGO8oanmSIpzUNYqatad5dYq WQ== Received: from nasanppmta04.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n81yq062k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:48:05 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA04.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMm4MA024424 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:48:04 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:48:03 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Will Deacon , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 22/27] virt: gunyah: Add resource tickets Date: Fri, 20 Jan 2023 14:46:21 -0800 Message-ID: <20230120224627.4053418-23-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: 1B8-ZnTqIwdwSlHPVIzxct3lZ3qbYisu X-Proofpoint-GUID: 1B8-ZnTqIwdwSlHPVIzxct3lZ3qbYisu X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 spamscore=0 priorityscore=1501 phishscore=0 suspectscore=0 lowpriorityscore=0 mlxlogscore=999 impostorscore=0 bulkscore=0 mlxscore=0 malwarescore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Some VM functions need to acquire Gunyah resources. For instance, Gunyah vCPUs are exposed to the host as a resource. The Gunyah vCPU function will register a resource ticket and be able to interact with the hypervisor once the resource ticket is filled. Signed-off-by: Elliot Berman --- drivers/virt/gunyah/vm_mgr.c | 105 +++++++++++++++++++++++++++++++++- drivers/virt/gunyah/vm_mgr.h | 5 ++ include/linux/gunyah.h | 4 ++ include/linux/gunyah_vm_mgr.h | 14 +++++ 4 files changed, 127 insertions(+), 1 deletion(-) diff --git a/drivers/virt/gunyah/vm_mgr.c b/drivers/virt/gunyah/vm_mgr.c index 1e795f3d19d5..ee593cf1b074 100644 --- a/drivers/virt/gunyah/vm_mgr.c +++ b/drivers/virt/gunyah/vm_mgr.c @@ -135,6 +135,56 @@ static long gh_vm_rm_function(struct gunyah_vm *ghvm, struct gh_vm_function *fn) return 0; } +int ghvm_add_resource_ticket(struct gunyah_vm *ghvm, struct gunyah_vm_resource_ticket *ticket) +{ + struct gunyah_vm_resource_ticket *iter; + struct gunyah_resource *ghrsc; + int ret = 0; + + mutex_lock(&ghvm->resources_lock); + list_for_each_entry(iter, &ghvm->resource_tickets, list) { + if (iter->resource_type == ticket->resource_type && iter->label == ticket->label) { + ret = -EEXIST; + goto out; + } + } + + if (!try_module_get(ticket->owner)) { + ret = -ENODEV; + goto out; + } + + list_add(&ticket->list, &ghvm->resource_tickets); + INIT_LIST_HEAD(&ticket->resources); + + list_for_each_entry(ghrsc, &ghvm->resources, list) { + if (ghrsc->type == ticket->resource_type && ghrsc->rm_label == ticket->label) { + if (!ticket->populate(ticket, ghrsc)) + list_move(&ghrsc->list, &ticket->resources); + } + } +out: + mutex_unlock(&ghvm->resources_lock); + return ret; +} +EXPORT_SYMBOL_GPL(ghvm_add_resource_ticket); + +void ghvm_remove_resource_ticket(struct gunyah_vm *ghvm, struct gunyah_vm_resource_ticket *ticket) +{ + struct gunyah_resource *ghrsc, *iter; + + mutex_lock(&ghvm->resources_lock); + list_for_each_entry_safe(ghrsc, iter, &ticket->resources, list) { + ticket->unpopulate(ticket, ghrsc); + list_move(&ghrsc->list, &ghvm->resources); + } + + module_put(ticket->owner); + list_del(&ticket->list); + mutex_unlock(&ghvm->resources_lock); +} +EXPORT_SYMBOL_GPL(ghvm_remove_resource_ticket); + static void ghvm_put(struct kref *kref) { struct gunyah_vm *ghvm = container_of(kref, struct gunyah_vm, kref); @@ -180,17 +230,40 @@ static __must_check struct gunyah_vm *gunyah_vm_alloc(struct gh_rm *rm) INIT_LIST_HEAD(&ghvm->memory_mappings); init_rwsem(&ghvm->status_lock); kref_init(&ghvm->kref); + mutex_init(&ghvm->resources_lock); + INIT_LIST_HEAD(&ghvm->resources); + INIT_LIST_HEAD(&ghvm->resource_tickets); INIT_LIST_HEAD(&ghvm->functions); return ghvm; } +static void gh_vm_add_resource(struct gunyah_vm *ghvm, struct gunyah_resource *ghrsc) +{ + struct gunyah_vm_resource_ticket *ticket; + + mutex_lock(&ghvm->resources_lock); + list_for_each_entry(ticket, &ghvm->resource_tickets, list) { + if (ghrsc->type == ticket->resource_type && ghrsc->rm_label == ticket->label) { + if (!ticket->populate(ticket, ghrsc)) { + list_add(&ghrsc->list, &ticket->resources); + goto found; + } + } + } + list_add(&ghrsc->list, &ghvm->resources); +found: + mutex_unlock(&ghvm->resources_lock); +} + static int gh_vm_start(struct gunyah_vm *ghvm) { struct gunyah_vm_memory_mapping *mapping; u64 dtb_offset; u32 mem_handle; - int ret; + struct gunyah_resource *ghrsc; + struct gh_rm_hyp_resources *resources; + int ret, i; down_write(&ghvm->status_lock); if (ghvm->vm_status != GH_RM_VM_STATUS_NO_STATE) { @@ -241,6 +314,22 @@ static int gh_vm_start(struct gunyah_vm *ghvm) goto err; } + ret = gh_rm_get_hyp_resources(ghvm->rm, ghvm->vmid, &resources); + if (ret) { + pr_warn("Failed to get hypervisor resources for VM: %d\n", ret); + goto err; + } + + for (i = 0; i < le32_to_cpu(resources->n_entries); i++) { + ghrsc = gh_rm_alloc_resource(ghvm->rm, &resources->entries[i]); + if (!ghrsc) { + ret = -ENOMEM; + goto err; + } + + gh_vm_add_resource(ghvm, ghrsc); + } + ret = gh_rm_vm_start(ghvm->rm, ghvm->vmid); if (ret) { pr_warn("Failed to start VM: %d\n", ret); @@ -380,6 +469,8 @@ static int gh_vm_release(struct inode *inode, struct file *filp) struct gunyah_vm *ghvm = filp->private_data; struct gunyah_vm_memory_mapping *mapping, *tmp; struct gunyah_vm_function *f, *fiter; + struct gunyah_vm_resource_ticket *ticket, *titer; + struct gunyah_resource *ghrsc, *riter; gh_vm_stop(ghvm); @@ -394,6 +485,18 @@ static int gh_vm_release(struct inode *inode, struct file *filp) kfree(mapping); } + if (!list_empty(&ghvm->resource_tickets)) { + pr_warn("Dangling resource tickets:\n"); + list_for_each_entry_safe(ticket, titer, &ghvm->resource_tickets, list) { + pr_warn(" %pS\n", ticket->populate); + ghvm_remove_resource_ticket(ghvm, ticket); + } + } + + list_for_each_entry_safe(ghrsc, riter, &ghvm->resources, list) { + gh_rm_free_resource(ghrsc); + } + put_gh_rm(ghvm->rm); put_gunyah_vm(ghvm); return 0; diff --git a/drivers/virt/gunyah/vm_mgr.h b/drivers/virt/gunyah/vm_mgr.h index 8d3b0678fb96..f52350145010 100644 --- a/drivers/virt/gunyah/vm_mgr.h +++ b/drivers/virt/gunyah/vm_mgr.h @@ -7,6 +7,7 @@ #define _GH_PRIV_VM_MGR_H #include +#include #include #include #include @@ -47,6 +48,10 @@ struct gunyah_vm { struct mutex mm_lock; struct list_head memory_mappings; + struct mutex resources_lock; + struct list_head resources; + struct list_head resource_tickets; + struct list_head functions; }; diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h index 91d2c9f5b2f0..d745916cabcc 100644 --- a/include/linux/gunyah.h +++ b/include/linux/gunyah.h @@ -27,6 +27,10 @@ struct gunyah_resource { enum gunyah_resource_type type; u64 capid; int irq; + + /* To help allocator of resource manager */ + struct list_head list; + u32 rm_label; }; /** diff --git a/include/linux/gunyah_vm_mgr.h b/include/linux/gunyah_vm_mgr.h index 69f98eb503e9..995a9b0eb6b7 100644 --- a/include/linux/gunyah_vm_mgr.h +++ b/include/linux/gunyah_vm_mgr.h @@ -65,4 +65,18 @@ void gunyah_vm_function_unregister(struct gunyah_vm_function_driver *f); } \ module_exit(_name##_mod_exit) +struct gunyah_vm_resource_ticket { + struct list_head list; + struct list_head resources; + enum gunyah_resource_type resource_type; + u32 label; + + struct module *owner; + int (*populate)(struct gunyah_vm_resource_ticket *ticket, struct gunyah_resource *ghrsc); + void (*unpopulate)(struct gunyah_vm_resource_ticket *ticket, struct gunyah_resource *ghrsc); +}; + +int ghvm_add_resource_ticket(struct gunyah_vm *ghvm, struct gunyah_vm_resource_ticket *ticket); +void ghvm_remove_resource_ticket(struct gunyah_vm *ghvm, struct gunyah_vm_resource_ticket *ticket); + #endif From patchwork Fri Jan 20 22:46:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 644673 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DF9BC38159 for ; Fri, 20 Jan 2023 22:50:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229967AbjATWuM (ORCPT ); Fri, 20 Jan 2023 17:50:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58484 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230167AbjATWuH (ORCPT ); Fri, 20 Jan 2023 17:50:07 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0296F3C284; Fri, 20 Jan 2023 14:49:25 -0800 (PST) Received: from pps.filterd (m0279873.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KMQZ9F011386; Fri, 20 Jan 2023 22:48:06 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=iCSyjBfqrRmZK0zvAXu/cBR+A/KtRTfe11K5Ky0RinU=; b=eje28ZCyQhZNxdTrjsDv7isomZc2YgoaEDvwacerY6goFuevI0og85GYaSpKgHXqXOc0 PEl/U2NuNNqu9VnKsP5T6A08AeaQDbDTT+IKOyD2dRVC4m9t4IMRSLsCxMdgGsQc2sYO MkKAvpCZV8V3m58HXG6szoqNOmhRlURVK1ruTA1V6Ko/akO7la68vq2K/si1IBBpSoeh Q4IuuEJmSL6Y7JKwnPadMuEkLW8/Ff8npiFen1NWgvLD4wDR2R80LxjbUkTa9NxxM1pY UlW0GvmBkq5dOmbtOSZylsST3aXPkSoU49NV+ojelVYYQQEKuoBqmoFNfCwuWP2gchpG 3g== Received: from nasanppmta02.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7593uqyp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:48:06 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA02.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMm5at029387 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:48:05 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:48:04 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Will Deacon , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 23/27] virt: gunyah: Add IO handlers Date: Fri, 20 Jan 2023 14:46:22 -0800 Message-ID: <20230120224627.4053418-24-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: A87Y_caG1S4FalvA4a7H7nv6B8QXsviN X-Proofpoint-ORIG-GUID: A87Y_caG1S4FalvA4a7H7nv6B8QXsviN X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 mlxlogscore=999 spamscore=0 suspectscore=0 mlxscore=0 adultscore=0 clxscore=1015 impostorscore=0 priorityscore=1501 bulkscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Add framework for VM functions to handle stage-2 write faults from Gunyah guest virtual machines. IO handlers have a range of addresses which they apply to. Optionally, they may apply to only when the value written matches the IO handler's value. Co-developed-by: Prakruthi Deepak Heragu Signed-off-by: Prakruthi Deepak Heragu Signed-off-by: Elliot Berman --- drivers/virt/gunyah/vm_mgr.c | 85 +++++++++++++++++++++++++++++++++++ drivers/virt/gunyah/vm_mgr.h | 4 ++ include/linux/gunyah_vm_mgr.h | 25 +++++++++++ 3 files changed, 114 insertions(+) diff --git a/drivers/virt/gunyah/vm_mgr.c b/drivers/virt/gunyah/vm_mgr.c index ee593cf1b074..1dfe354bcc29 100644 --- a/drivers/virt/gunyah/vm_mgr.c +++ b/drivers/virt/gunyah/vm_mgr.c @@ -185,6 +185,91 @@ void ghvm_remove_resource_ticket(struct gunyah_vm *ghvm, struct gunyah_vm_resour } EXPORT_SYMBOL_GPL(ghvm_remove_resource_ticket); +static inline bool gh_vm_io_handler_matches(struct gunyah_vm_io_handler *io_hdlr, u64 addr, + u64 len, u64 data) +{ + u64 mask = BIT_ULL(io_hdlr->len * BITS_PER_BYTE) - 1; + + if (io_hdlr->addr != addr) + return false; + + if (!io_hdlr->datamatch) + return true; + + if (io_hdlr->len != len) + return false; + + return (data & mask) == (io_hdlr->data & mask); +} + +static struct gunyah_vm_io_handler *gh_vm_mgr_find_io_hdlr(struct gunyah_vm *ghvm, u64 addr, + u64 len, u64 data) +{ + struct gunyah_vm_io_handler *io_hdlr = NULL; + struct rb_node *root = NULL; + + root = ghvm->mmio_handler_root.rb_node; + while (root) { + io_hdlr = rb_entry(root, struct gunyah_vm_io_handler, node); + if (addr < io_hdlr->addr) + root = root->rb_left; + else if (addr > io_hdlr->addr) + root = root->rb_right; + else if (gh_vm_io_handler_matches(io_hdlr, addr, len, data)) + return io_hdlr; + } + return NULL; +} + +int gh_vm_mgr_mmio_write(struct gunyah_vm *ghvm, u64 addr, u32 len, u64 data) +{ + struct gunyah_vm_io_handler *io_hdlr = NULL; + + io_hdlr = gh_vm_mgr_find_io_hdlr(ghvm, addr, len, data); + if (!io_hdlr) + return -ENODEV; + + if (io_hdlr->ops && io_hdlr->ops->write) + return io_hdlr->ops->write(io_hdlr, addr, len, data); + + return -ENODEV; +} +EXPORT_SYMBOL_GPL(gh_vm_mgr_mmio_write); + +int gh_vm_mgr_add_io_handler(struct gunyah_vm *ghvm, struct gunyah_vm_io_handler *io_hdlr) +{ + struct rb_node **root, *parent = NULL; + + if (io_hdlr->datamatch && + (!io_hdlr->len || io_hdlr->len > (sizeof(io_hdlr->data) * BITS_PER_BYTE))) + return -EINVAL; + + root = &ghvm->mmio_handler_root.rb_node; + while (*root) { + struct gunyah_vm_io_handler *curr = rb_entry(*root, struct gunyah_vm_io_handler, + node); + + parent = *root; + if (io_hdlr->addr < curr->addr) + root = &((*root)->rb_left); + else if (io_hdlr->addr > curr->addr) + root = &((*root)->rb_right); + else + return -EEXIST; + } + + rb_link_node(&io_hdlr->node, parent, root); + rb_insert_color(&io_hdlr->node, &ghvm->mmio_handler_root); + return 0; +} +EXPORT_SYMBOL_GPL(gh_vm_mgr_add_io_handler); + +void gh_vm_mgr_remove_io_handler(struct gunyah_vm *ghvm, struct gunyah_vm_io_handler *io_hdlr) +{ + rb_erase(&io_hdlr->node, &ghvm->mmio_handler_root); +} +EXPORT_SYMBOL_GPL(gh_vm_mgr_remove_io_handler); + static void ghvm_put(struct kref *kref) { struct gunyah_vm *ghvm = container_of(kref, struct gunyah_vm, kref); diff --git a/drivers/virt/gunyah/vm_mgr.h b/drivers/virt/gunyah/vm_mgr.h index f52350145010..eb17a2dda2a5 100644 --- a/drivers/virt/gunyah/vm_mgr.h +++ b/drivers/virt/gunyah/vm_mgr.h @@ -52,6 +52,8 @@ struct gunyah_vm { struct list_head resources; struct list_head resource_tickets; + struct rb_root mmio_handler_root; + struct list_head functions; }; @@ -62,4 +64,6 @@ struct gunyah_vm_memory_mapping *gh_vm_mem_mapping_find(struct gunyah_vm *ghvm, struct gunyah_vm_memory_mapping *gh_vm_mem_mapping_find_mapping(struct gunyah_vm *ghvm, u64 gpa, u32 size); +int gh_vm_mgr_mmio_write(struct gunyah_vm *ghvm, u64 addr, u32 len, u64 data); + #endif diff --git a/include/linux/gunyah_vm_mgr.h b/include/linux/gunyah_vm_mgr.h index 995a9b0eb6b7..dd719061f3bf 100644 --- a/include/linux/gunyah_vm_mgr.h +++ b/include/linux/gunyah_vm_mgr.h @@ -79,4 +79,29 @@ struct gunyah_vm_resource_ticket { int ghvm_add_resource_ticket(struct gunyah_vm *ghvm, struct gunyah_vm_resource_ticket *ticket); void ghvm_remove_resource_ticket(struct gunyah_vm *ghvm, struct gunyah_vm_resource_ticket *ticket); +/* + * gunyah_vm_io_handler contains the info about an io device and its associated + * addr and the ops associated with the io device. + */ +struct gunyah_vm_io_handler { + struct rb_node node; + u64 addr; + + bool datamatch; + u8 len; + u64 data; + struct gunyah_vm_io_handler_ops *ops; +}; + +/* + * gunyah_vm_io_handler_ops contains function pointers associated with an iodevice. + */ +struct gunyah_vm_io_handler_ops { + int (*read)(struct gunyah_vm_io_handler *io_dev, u64 addr, u32 len, u64 data); + int (*write)(struct gunyah_vm_io_handler *io_dev, u64 addr, u32 len, u64 data); +}; + +int gh_vm_mgr_add_io_handler(struct gunyah_vm *ghvm, struct gunyah_vm_io_handler *io_dev); +void gh_vm_mgr_remove_io_handler(struct gunyah_vm *ghvm, struct gunyah_vm_io_handler *io_dev); + #endif From patchwork Fri Jan 20 22:46:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 645097 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67F69C38159 for ; Fri, 20 Jan 2023 23:17:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229787AbjATXRs (ORCPT ); Fri, 20 Jan 2023 18:17:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54702 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229682AbjATXRq (ORCPT ); Fri, 20 Jan 2023 18:17:46 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D90A571BC5; Fri, 20 Jan 2023 15:17:43 -0800 (PST) Received: from pps.filterd (m0279872.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KLvY30027585; Fri, 20 Jan 2023 22:48:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=3RJ1kubIWWBDOgXPLNeHnyd+/btuhmzTRg4X4qAaeps=; b=ODQ5sPjo1kb/StN/HUX4vgvwJaF5Vz/c7rlYDyJdNFVKoxdkB/3kOZ8HttkHxgxaFkES J+GTS0b3lSqLJ2rumD2P/q23wwTMP8eot6paejUzOQ1a1WGBQVTkv2e20t4I4rtxMh3k 5b4siXAIZ2QP+g2z/+b+qQIBQ4ZFHfwinmwW/taoRb/Vfi8s8rAW7gnlcST/416suR9I U0mvyO4mQiSUbXzQsIuTQHPPpLXX8NixkMw2fndL87IYvTy7ARG5mdt8fsaDiLhZpbn8 pvUUrevUtnxvvnEG6mOIY1Wn3pQCyFYE/BQjKSTgxpXTPvwL+mcvN1VX2AWJD1dYcr1Z 5g== Received: from nasanppmta03.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7xm40k9j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:48:09 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA03.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMm8v2019836 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:48:08 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:48:07 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala , Jonathan Corbet , "Catalin Marinas" , Will Deacon CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Bagas Sanjaya , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 24/27] virt: gunyah: Add proxy-scheduled vCPUs Date: Fri, 20 Jan 2023 14:46:23 -0800 Message-ID: <20230120224627.4053418-25-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: GHW_nipU7EmjzDU4ciwsoTDiFHliuNm4 X-Proofpoint-GUID: GHW_nipU7EmjzDU4ciwsoTDiFHliuNm4 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 malwarescore=0 spamscore=0 bulkscore=0 clxscore=1015 priorityscore=1501 mlxlogscore=999 impostorscore=0 mlxscore=0 suspectscore=0 adultscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Gunyah allows host virtual machines to schedule guest virtual machines and handle their MMIO accesses. vCPUs are presented to the host as a Gunyah resource and represented to userspace as a Gunyah VM function. Creating the vcpu VM function will create a file descriptor that: - can run an ioctl: GH_VCPU_RUN to schedule the guest vCPU until the next interrupt occurs on the host or when the guest vCPU can no longer be run. - can be mmap'd to share a gh_vcpu_run structure which can look up the reason why GH_VCPU_RUN returned and provide return values for MMIO access. Co-developed-by: Prakruthi Deepak Heragu Signed-off-by: Prakruthi Deepak Heragu Signed-off-by: Elliot Berman --- Documentation/virt/gunyah/vm-manager.rst | 30 +- arch/arm64/gunyah/gunyah_hypercall.c | 28 ++ drivers/virt/gunyah/Kconfig | 11 + drivers/virt/gunyah/Makefile | 2 + drivers/virt/gunyah/gunyah_vcpu.c | 358 +++++++++++++++++++++++ drivers/virt/gunyah/vm_mgr.c | 25 ++ drivers/virt/gunyah/vm_mgr.h | 1 + include/linux/gunyah.h | 7 + include/uapi/linux/gunyah.h | 30 ++ 9 files changed, 490 insertions(+), 2 deletions(-) create mode 100644 drivers/virt/gunyah/gunyah_vcpu.c diff --git a/Documentation/virt/gunyah/vm-manager.rst b/Documentation/virt/gunyah/vm-manager.rst index d929f8f20b3b..d11267d59802 100644 --- a/Documentation/virt/gunyah/vm-manager.rst +++ b/Documentation/virt/gunyah/vm-manager.rst @@ -5,8 +5,7 @@ Virtual Machine Manager ======================= The Gunyah Virtual Machine Manager is a Linux driver to support launching -virtual machines using Gunyah. It presently supports launching non-proxy -scheduled Linux-like virtual machines. +virtual machines using Gunyah. Except for some basic information about the location of initial binaries, most of the configuration about a Gunyah virtual machine is described in the @@ -116,3 +115,30 @@ GH_VM_START ~~~~~~~~~~~ This ioctl starts the VM. + +GH_VM_ADD_FUNCTION +~~~~~~~~~~~~~~~~~~ + +This ioctl registers a Gunyah VM function with the VM manager. The VM function +is described with a `type` string and some arguments for that type. Typically, +the function is added before the VM starts, but the function doesn't "operate" +until the VM starts with GH_VM_START: e.g. vCPU ioclts will all return an error +until the VM starts because the vCPUs don't exist until the VM is started. This +allows the VMM to set up all the kernel functionality needed for the VM *before* +the VM starts. + +The possible types are documented below: + +Type: "vcpu" +^^^^^^^^^^^^ + +:: + + struct gh_fn_vcpu_arg { + __u32 vcpu_id; + }; + +The vcpu type will register with the VM Manager to expect to control +vCPU number `vcpu_id`. It returns a file descriptor allowing interaction with +the vCPU. See the Gunyah vCPU API description sections for interacting with +the Gunyah vCPU file descriptors. diff --git a/arch/arm64/gunyah/gunyah_hypercall.c b/arch/arm64/gunyah/gunyah_hypercall.c index d93ad2c08479..17d87a130a3e 100644 --- a/arch/arm64/gunyah/gunyah_hypercall.c +++ b/arch/arm64/gunyah/gunyah_hypercall.c @@ -15,6 +15,7 @@ static const uint32_t gunyah_known_uuids[][4] = { #define GH_HYPERCALL_HYP_IDENTIFY GH_HYPERCALL(0x0000) #define GH_HYPERCALL_MSGQ_SEND GH_HYPERCALL(0x001B) #define GH_HYPERCALL_MSGQ_RECV GH_HYPERCALL(0x001C) +#define GH_HYPERCALL_VCPU_RUN GH_HYPERCALL(0x0065) /** * gh_hypercall_get_uid() - Returns a UID when running under a Gunyah hypervisor @@ -104,5 +105,32 @@ int gh_hypercall_msgq_recv(u64 capid, uintptr_t buff, size_t size, size_t *recv_ } EXPORT_SYMBOL_GPL(gh_hypercall_msgq_recv); +int gh_hypercall_vcpu_run(u64 capid, u64 *resume_data, struct gh_hypercall_vcpu_run_resp *resp) +{ + struct arm_smccc_1_2_regs args = { + .a0 = GH_HYPERCALL_VCPU_RUN, + .a1 = capid, + .a2 = resume_data[0], + .a3 = resume_data[1], + .a4 = resume_data[2], + /* C language says this will be implictly zero. Gunyah requires 0, so be explicit */ + .a5 = 0, + }; + struct arm_smccc_1_2_regs res; + + arm_smccc_1_2_hvc(&args, &res); + + if (res.a0) + return res.a0; + + resp->state = res.a1; + resp->state_data[0] = res.a2; + resp->state_data[1] = res.a3; + resp->state_data[2] = res.a4; + + return res.a0; +} +EXPORT_SYMBOL_GPL(gh_hypercall_vcpu_run); + MODULE_LICENSE("GPL"); MODULE_DESCRIPTION("Gunyah Hypervisor Hypercalls"); diff --git a/drivers/virt/gunyah/Kconfig b/drivers/virt/gunyah/Kconfig index de815189dab6..4c1c6110b50e 100644 --- a/drivers/virt/gunyah/Kconfig +++ b/drivers/virt/gunyah/Kconfig @@ -15,3 +15,14 @@ config GUNYAH config GUNYAH_PLATFORM_HOOKS tristate + +config GUNYAH_VCPU + tristate "Runnable Gunyah vCPUs" + depends on GUNYAH + help + Enable kernel support for host-scheduled vCPUs running under Gunyah. + When selecting this option, userspace virtual machine managers (VMM) + can schedule the guest VM's vCPUs instead of using Gunyah's scheduler. + VMMs can also handle stage 2 faults of the vCPUs. + + Say Y/M here if unsure and you want to support Gunyah VMMs. diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile index 6b8f84dbfe0d..2d1b604a7b03 100644 --- a/drivers/virt/gunyah/Makefile +++ b/drivers/virt/gunyah/Makefile @@ -5,3 +5,5 @@ obj-$(CONFIG_GUNYAH_PLATFORM_HOOKS) += gunyah_platform_hooks.o gunyah_rsc_mgr-y += rsc_mgr.o rsc_mgr_rpc.o vm_mgr.o vm_mgr_mm.o obj-$(CONFIG_GUNYAH) += gunyah_rsc_mgr.o + +obj-$(CONFIG_GUNYAH_VCPU) += gunyah_vcpu.o diff --git a/drivers/virt/gunyah/gunyah_vcpu.c b/drivers/virt/gunyah/gunyah_vcpu.c new file mode 100644 index 000000000000..5166e7d8638f --- /dev/null +++ b/drivers/virt/gunyah/gunyah_vcpu.c @@ -0,0 +1,358 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vm_mgr.h" + +#include + +#define MAX_VCPU_NAME 20 /* gh-vcpu:u32_max+NUL */ + +struct gunyah_vcpu { + struct gunyah_resource *rsc; + struct gunyah_vm_resource_ticket ticket; + struct gunyah_vm_function *f; + struct gunyah_vm *ghvm; + + bool handle_mmio; + struct gh_vcpu_run *vcpu_run; + + struct kref kref; + struct completion ready; + struct mutex run_lock; +}; + +/* VCPU is ready to run */ +#define GH_VCPU_STATE_READY 0 +/* VCPU is sleeping until an interrupt arrives */ +#define GH_VCPU_STATE_EXPECTS_WAKEUP 1 +/* VCPU is powered off */ +#define GH_VCPU_STATE_POWERED_OFF 2 +/* VCPU is blocked in EL2 for unspecified reason */ +#define GH_VCPU_STATE_BLOCKED 3 +/* VCPU has returned for MMIO READ */ +#define GH_VCPU_ADDRSPACE_VMMIO_READ 4 +/* VCPU has returned for MMIO WRITE */ +#define GH_VCPU_ADDRSPACE_VMMIO_WRITE 5 + +static void vcpu_release(struct kref *kref) +{ + struct gunyah_vcpu *vcpu = container_of(kref, struct gunyah_vcpu, kref); + + kfree(vcpu); +} + +/* + * When hypervisor allows us to schedule vCPU again, it gives us an interrupt + */ +static irqreturn_t gh_vcpu_irq_handler(int irq, void *data) +{ + struct gunyah_vcpu *vcpu = data; + + complete(&vcpu->ready); + return IRQ_HANDLED; +} + +static void gh_handle_mmio_return(struct gunyah_vcpu *vcpu, u64 *state) +{ + if (!vcpu->vcpu_run->mmio.is_write) + memcpy(&state[0], vcpu->vcpu_run->mmio.data, vcpu->vcpu_run->mmio.len); + + vcpu->handle_mmio = false; + vcpu->vcpu_run->exit_reason = GH_VM_EXIT_UNKNOWN; +} + +static bool gh_handle_mmio(struct gunyah_vcpu *vcpu, + struct gh_hypercall_vcpu_run_resp *vcpu_run_resp) +{ + int ret = 0; + + if (vcpu_run_resp->state == GH_VCPU_ADDRSPACE_VMMIO_READ) { + vcpu->vcpu_run->mmio.is_write = 0; + vcpu->vcpu_run->exit_reason = GH_VM_EXIT_MMIO; + } else { /* GH_VCPU_ADDRSPACE_VMMIO_WRITE case */ + ret = gh_vm_mgr_mmio_write(vcpu->f->ghvm, vcpu_run_resp->state_data[0], + vcpu_run_resp->state_data[1], vcpu_run_resp->state_data[2]); + if (!ret) + return true; + + vcpu->vcpu_run->mmio.is_write = 1; + memcpy(vcpu->vcpu_run->mmio.data, &vcpu_run_resp->state_data[2], + vcpu_run_resp->state_data[1]); + } + + vcpu->vcpu_run->mmio.phys_addr = vcpu_run_resp->state_data[0]; + vcpu->vcpu_run->mmio.len = vcpu_run_resp->state_data[1]; + vcpu->vcpu_run->exit_reason = GH_VM_EXIT_MMIO; + + return false; +} + +/** + * gh_vcpu_run() - Request Gunyah to begin scheduling this vCPU. + * @vcpu: The client descriptor that was obtained via gunyah_vcpu_alloc() + */ +static int gh_vcpu_run(struct gunyah_vcpu *vcpu) +{ + struct gh_hypercall_vcpu_run_resp vcpu_run_resp; + u64 state_data[3] = { 0 }; + int ret = 0; + + ret = gh_vm_ensure_started(vcpu->ghvm); + if (ret) + return ret; + + if (mutex_lock_interruptible(&vcpu->run_lock)) + return -ERESTARTSYS; + + if (!vcpu->rsc || !vcpu->f) { + ret = -ENODEV; + goto out; + } + /* Last exit reason was EXIT_MMIO. Userspace has filled in the data, now we need to tell + * Gunyah about the response. + */ + if (vcpu->handle_mmio) + gh_handle_mmio_return(vcpu, state_data); + + while (!ret && !signal_pending(current)) { + if (vcpu->vcpu_run->immediate_exit) { + ret = -EINTR; + goto out; + } + ret = gh_hypercall_vcpu_run(vcpu->rsc->capid, state_data, &vcpu_run_resp); + if (ret == GH_ERROR_OK) { + switch (vcpu_run_resp.state) { + case GH_VCPU_STATE_READY: + if (need_resched()) + schedule(); + break; + case GH_VCPU_STATE_EXPECTS_WAKEUP: + case GH_VCPU_STATE_POWERED_OFF: + ret = wait_for_completion_interruptible(&vcpu->ready); + /* reinitialize completion before next VCPU_RUN. If we reinitialize + * after the VCPU_RUN, interrupt may have already come before + * we can re-initialize and then waiting for an interrupt that + * was already handled. + */ + reinit_completion(&vcpu->ready); + break; + case GH_VCPU_STATE_BLOCKED: + schedule(); + break; + case GH_VCPU_ADDRSPACE_VMMIO_READ: + case GH_VCPU_ADDRSPACE_VMMIO_WRITE: + if (!gh_handle_mmio(vcpu, &vcpu_run_resp)) { + vcpu->handle_mmio = true; + goto out; + } + break; + default: + pr_warn_ratelimited("Unknown vCPU state: %llx\n", + vcpu_run_resp.state); + schedule(); + break; + } + } else if (ret == GH_ERROR_RETRY) { + schedule(); + ret = 0; + } else + ret = gh_remap_error(ret); + } + +out: + mutex_unlock(&vcpu->run_lock); + + if (signal_pending(current)) + return -ERESTARTSYS; + + return ret; +} + +static long gh_vcpu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) +{ + struct gunyah_vcpu *vcpu = filp->private_data; + long ret = -EINVAL; + + switch (cmd) { + case GH_VCPU_RUN: + ret = gh_vcpu_run(vcpu); + break; + case GH_VCPU_MMAP_SIZE: + ret = PAGE_SIZE; + break; + default: + break; + } + return ret; +} + +static int gh_vcpu_release(struct inode *inode, struct file *filp) +{ + struct gunyah_vcpu *vcpu = filp->private_data; + + kref_put(&vcpu->kref, vcpu_release); + return 0; +} + +static vm_fault_t gh_vcpu_fault(struct vm_fault *vmf) +{ + struct gunyah_vcpu *vcpu = vmf->vma->vm_file->private_data; + struct page *page = NULL; + + if (vmf->pgoff == 0) + page = virt_to_page(vcpu->vcpu_run); + + get_page(page); + vmf->page = page; + return 0; +} + +static const struct vm_operations_struct gh_vcpu_ops = { + .fault = gh_vcpu_fault, +}; + +static int gh_vcpu_mmap(struct file *file, struct vm_area_struct *vma) +{ + vma->vm_ops = &gh_vcpu_ops; + return 0; +} + +static const struct file_operations gh_vcpu_fops = { + .unlocked_ioctl = gh_vcpu_ioctl, + .release = gh_vcpu_release, + .llseek = noop_llseek, + .mmap = gh_vcpu_mmap, +}; + +static int gunyah_vcpu_populate(struct gunyah_vm_resource_ticket *ticket, + struct gunyah_resource *ghrsc) +{ + struct gunyah_vcpu *vcpu = container_of(ticket, struct gunyah_vcpu, ticket); + int ret; + + mutex_lock(&vcpu->run_lock); + vcpu->rsc = ghrsc; + + init_completion(&vcpu->ready); + mutex_unlock(&vcpu->run_lock); + + ret = request_irq(vcpu->rsc->irq, gh_vcpu_irq_handler, IRQF_TRIGGER_RISING, "gh_vcpu", + vcpu); + if (ret) + pr_warn("Failed to request vcpu irq %d: %d", vcpu->rsc->irq, ret); + + return ret; +} + +static void gunyah_vcpu_unpopulate(struct gunyah_vm_resource_ticket *ticket, + struct gunyah_resource *ghrsc) +{ + struct gunyah_vcpu *vcpu = container_of(ticket, struct gunyah_vcpu, ticket); + + vcpu->vcpu_run->immediate_exit = true; + mutex_lock(&vcpu->run_lock); + free_irq(vcpu->rsc->irq, vcpu); + vcpu->rsc = NULL; + mutex_unlock(&vcpu->run_lock); +} + +static long gunyah_vcpu_bind(struct gunyah_vm_function *f) +{ + struct gunyah_vcpu *vcpu; + char name[MAX_VCPU_NAME]; + struct file *file; + struct page *page; + int fd; + long r; + + if (!gh_api_has_feature(GH_API_FEATURE_VCPU)) + return -EOPNOTSUPP; + + vcpu = kzalloc(sizeof(*vcpu), GFP_KERNEL); + if (!vcpu) + return -ENOMEM; + + vcpu->f = f; + f->data = vcpu; + mutex_init(&vcpu->run_lock); + kref_init(&vcpu->kref); + + page = alloc_page(GFP_KERNEL | __GFP_ZERO); + if (!page) { + r = -ENOMEM; + goto err_destroy_vcpu; + } + vcpu->vcpu_run = page_address(page); + + vcpu->ticket.resource_type = GUNYAH_RESOURCE_TYPE_VCPU; + vcpu->ticket.label = f->fn.vcpu.vcpu_id; + vcpu->ticket.owner = THIS_MODULE; + vcpu->ticket.populate = gunyah_vcpu_populate; + vcpu->ticket.unpopulate = gunyah_vcpu_unpopulate; + + r = ghvm_add_resource_ticket(f->ghvm, &vcpu->ticket); + if (r) + goto err_destroy_page; + + fd = get_unused_fd_flags(O_CLOEXEC); + if (fd < 0) { + r = fd; + goto err_remove_vcpu; + } + + if (!get_gunyah_vm(f->ghvm)) { + r = -ENODEV; + goto err_put_fd; + } + vcpu->ghvm = f->ghvm; + + kref_get(&vcpu->kref); + snprintf(name, sizeof(name), "gh-vcpu:%d", vcpu->ticket.label); + file = anon_inode_getfile(name, &gh_vcpu_fops, vcpu, O_RDWR); + if (IS_ERR(file)) { + r = PTR_ERR(file); + goto err_put_ghvm; + } + + fd_install(fd, file); + + return fd; +err_put_ghvm: + put_gunyah_vm(vcpu->ghvm); +err_put_fd: + put_unused_fd(fd); +err_remove_vcpu: + ghvm_remove_resource_ticket(f->ghvm, &vcpu->ticket); +err_destroy_page: + free_page((unsigned long)vcpu->vcpu_run); +err_destroy_vcpu: + kfree(vcpu); + return r; +} + +static void gunyah_vcpu_release(struct gunyah_vm_function *f) +{ + struct gunyah_vcpu *vcpu = f->data; + + ghvm_remove_resource_ticket(vcpu->f->ghvm, &vcpu->ticket); + vcpu->f = NULL; + + kref_put(&vcpu->kref, vcpu_release); +} + +DECLARE_GUNYAH_VM_FUNCTION_INIT(vcpu, gunyah_vcpu_bind, gunyah_vcpu_release); +MODULE_DESCRIPTION("Gunyah vCPU Driver"); +MODULE_LICENSE("GPL"); diff --git a/drivers/virt/gunyah/vm_mgr.c b/drivers/virt/gunyah/vm_mgr.c index 1dfe354bcc29..9786de138f7c 100644 --- a/drivers/virt/gunyah/vm_mgr.c +++ b/drivers/virt/gunyah/vm_mgr.c @@ -431,6 +431,31 @@ static int gh_vm_start(struct gunyah_vm *ghvm) return ret; } +int gh_vm_ensure_started(struct gunyah_vm *ghvm) +{ + int ret; + +retry: + ret = down_read_interruptible(&ghvm->status_lock); + if (ret) + return ret; + + if (unlikely(ghvm->vm_status == GH_RM_VM_STATUS_NO_STATE)) { + up_read(&ghvm->status_lock); + ret = gh_vm_start(ghvm); + if (ret) + return ret; + goto retry; + } + + if (unlikely(ghvm->vm_status != GH_RM_VM_STATUS_READY)) + ret = -ENODEV; + + up_read(&ghvm->status_lock); + return ret; +} +EXPORT_SYMBOL_GPL(gh_vm_ensure_started); + static void gh_vm_stop(struct gunyah_vm *ghvm) { int ret; diff --git a/drivers/virt/gunyah/vm_mgr.h b/drivers/virt/gunyah/vm_mgr.h index eb17a2dda2a5..100c457dce14 100644 --- a/drivers/virt/gunyah/vm_mgr.h +++ b/drivers/virt/gunyah/vm_mgr.h @@ -64,6 +64,7 @@ struct gunyah_vm_memory_mapping *gh_vm_mem_mapping_find(struct gunyah_vm *ghvm, struct gunyah_vm_memory_mapping *gh_vm_mem_mapping_find_mapping(struct gunyah_vm *ghvm, u64 gpa, u32 size); +int gh_vm_ensure_started(struct gunyah_vm *ghvm); int gh_vm_mgr_mmio_write(struct gunyah_vm *ghvm, u64 addr, u32 len, u64 data); #endif diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h index d745916cabcc..feeab01bff30 100644 --- a/include/linux/gunyah.h +++ b/include/linux/gunyah.h @@ -169,4 +169,11 @@ void gh_hypercall_hyp_identify(struct gh_hypercall_hyp_identify_resp *hyp_identi int gh_hypercall_msgq_send(u64 capid, size_t size, uintptr_t buff, int tx_flags, bool *ready); int gh_hypercall_msgq_recv(u64 capid, uintptr_t buff, size_t size, size_t *recv_size, bool *ready); +struct gh_hypercall_vcpu_run_resp { + u64 state; + u64 state_data[3]; +}; + +int gh_hypercall_vcpu_run(u64 capid, u64 *resume_data, struct gh_hypercall_vcpu_run_resp *resp); + #endif diff --git a/include/uapi/linux/gunyah.h b/include/uapi/linux/gunyah.h index ec8da6fde045..b4afb11f538a 100644 --- a/include/uapi/linux/gunyah.h +++ b/include/uapi/linux/gunyah.h @@ -53,9 +53,14 @@ struct gh_vm_dtb_config { #define GUNYAH_FUNCTION_NAME_SIZE 32 #define GUNYAH_FUNCTION_MAX_ARG_SIZE 1024 +struct gh_fn_vcpu_arg { + __u32 vcpu_id; +}; + struct gh_vm_function { char name[GUNYAH_FUNCTION_NAME_SIZE]; union { + struct gh_device_vcpu_arg vcpu; char data[GUNYAH_FUNCTION_MAX_ARG_SIZE]; }; }; @@ -63,4 +68,29 @@ struct gh_vm_function { #define GH_VM_ADD_FUNCTION _IOW(GH_IOCTL_TYPE, 0x4, struct gh_vm_function) #define GH_VM_REMOVE_FUNCTION _IOW(GH_IOCTL_TYPE, 0x7, struct gh_vm_function) +/* for GH_VCPU_RUN, returned by mmap(vcpu_fd, offset=0) */ +struct gh_vcpu_run { + /* in */ + __u8 immediate_exit; + __u8 padding1[7]; + + /* out */ +#define GH_VM_EXIT_UNKNOWN 0 +#define GH_VM_EXIT_MMIO 1 + __u32 exit_reason; + + union { + /* GH_VM_EXIT_MMIO */ + struct { + __u64 phys_addr; + __u8 data[8]; + __u32 len; + __u8 is_write; + } mmio; + }; +}; + +#define GH_VCPU_RUN _IO(GH_IOCTL_TYPE, 0x5) +#define GH_VCPU_MMAP_SIZE _IO(GH_IOCTL_TYPE, 0x6) + #endif From patchwork Fri Jan 20 22:46:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 645098 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1D1FC25B50 for ; Fri, 20 Jan 2023 23:09:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229700AbjATXI7 (ORCPT ); Fri, 20 Jan 2023 18:08:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229694AbjATXI6 (ORCPT ); Fri, 20 Jan 2023 18:08:58 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC1085E515; Fri, 20 Jan 2023 15:08:49 -0800 (PST) Received: from pps.filterd (m0279870.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KLAU7O022403; Fri, 20 Jan 2023 22:48:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=+z0xlUlEWhGDHObbmbwPzHrQHX6Yp1OolxLxSjxGCHk=; b=NKEGv+dyCkLx4R4NImhfxqLObWHfEokgkG+G26vuRoQBPbPGdRBBqeECm1mrm01zoaPX uEnkgMcvyDMgI0AI4G7xqtQbJZ3/nVsGrGTCLHQxCPs7N8s+UKXlIGkiR4mM2sZrCYCk ywGHj4sA6luFOsPzuSksUz0IU31zW/SiGop5gELpmJGxVqvam73uUQsdA7ME5977JexD IWQYigACJKTREtvW9O3kyC7bhXQlggnlUw42AoNxZ5o37s0tH3JteVS1ln3282upnwi/ WVXwXN8t+VATmhAghzEvrZBJdgczmWG9hLyz7XFSCib8M/TzRA6B0IW+DjRUzJdjL+G6 0Q== Received: from nasanppmta02.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7xt7rjwu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:48:10 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA02.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMm9w7029470 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:48:09 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:48:08 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala , Catalin Marinas , "Will Deacon" CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 25/27] virt: gunyah: Add hypercalls for sending doorbell Date: Fri, 20 Jan 2023 14:46:24 -0800 Message-ID: <20230120224627.4053418-26-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: PEWX8HQrH9KTkKaoy7akz9laQMWtl8eX X-Proofpoint-ORIG-GUID: PEWX8HQrH9KTkKaoy7akz9laQMWtl8eX X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 mlxscore=0 suspectscore=0 impostorscore=0 spamscore=0 priorityscore=1501 lowpriorityscore=0 mlxlogscore=964 phishscore=0 bulkscore=0 adultscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Gunyah doorbells allow two virtual machines to signal each other using interrupts. Add the hypercalls needed to assert the interrupt. Signed-off-by: Elliot Berman --- arch/arm64/gunyah/gunyah_hypercall.c | 39 ++++++++++++++++++---------- include/linux/gunyah.h | 3 +++ 2 files changed, 29 insertions(+), 13 deletions(-) diff --git a/arch/arm64/gunyah/gunyah_hypercall.c b/arch/arm64/gunyah/gunyah_hypercall.c index 17d87a130a3e..057e127f2b4d 100644 --- a/arch/arm64/gunyah/gunyah_hypercall.c +++ b/arch/arm64/gunyah/gunyah_hypercall.c @@ -12,19 +12,7 @@ static const uint32_t gunyah_known_uuids[][4] = { {0x673d5f14, 0x9265ce36, 0xa4535fdb, 0xc1d58fcd}, /* GUNYAH (open source build) */ }; -#define GH_HYPERCALL_HYP_IDENTIFY GH_HYPERCALL(0x0000) -#define GH_HYPERCALL_MSGQ_SEND GH_HYPERCALL(0x001B) -#define GH_HYPERCALL_MSGQ_RECV GH_HYPERCALL(0x001C) -#define GH_HYPERCALL_VCPU_RUN GH_HYPERCALL(0x0065) - -/** - * gh_hypercall_get_uid() - Returns a UID when running under a Gunyah hypervisor - * @uid: An array of 4 u32's (u32 uid[4];) - * - * Caller should compare the resulting UID to a list of known Gunyah UIDs to - * confirm that Linux is running as a guest of Gunyah. - */ -void gh_hypercall_get_uid(u32 uid[4]) +bool arch_is_gunyah_guest(void) { struct arm_smccc_res res; u32 uid[4]; @@ -74,6 +62,31 @@ void gh_hypercall_hyp_identify(struct gh_hypercall_hyp_identify_resp *hyp_identi } EXPORT_SYMBOL_GPL(gh_hypercall_hyp_identify); +int gh_hypercall_dbl_send(u64 capid, u64 new_flags, u64 *old_flags) +{ + struct arm_smccc_res res; + + arm_smccc_1_1_hvc(GH_HYPERCALL_DBL_SEND, capid, new_flags, 0, &res); + + if (res.a0) + return res.a0; + + *old_flags = res.a1; + + return res.a0; +} +EXPORT_SYMBOL_GPL(gh_hypercall_dbl_send); + +int gh_hypercall_dbl_set_mask(u64 capid, u64 enable_mask, u64 ack_mask) +{ + struct arm_smccc_res res; + + arm_smccc_1_1_hvc(GH_HYPERCALL_DBL_SET_MASK, capid, enable_mask, ack_mask, 0, &res); + + return res.a0; +} +EXPORT_SYMBOL_GPL(gh_hypercall_dbl_set_mask); + int gh_hypercall_msgq_send(u64 capid, size_t size, uintptr_t buff, int tx_flags, bool *ready) { struct arm_smccc_res res; diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h index feeab01bff30..ac4879940c10 100644 --- a/include/linux/gunyah.h +++ b/include/linux/gunyah.h @@ -164,6 +164,9 @@ struct gh_hypercall_hyp_identify_resp { void gh_hypercall_hyp_identify(struct gh_hypercall_hyp_identify_resp *hyp_identity); +int gh_hypercall_dbl_send(u64 capid, u64 new_flags, u64 *old_flags); +int gh_hypercall_dbl_set_mask(u64 capid, u64 enable_mask, u64 ack_mask); + #define GH_HYPERCALL_MSGQ_TX_FLAGS_PUSH BIT(0) int gh_hypercall_msgq_send(u64 capid, size_t size, uintptr_t buff, int tx_flags, bool *ready); From patchwork Fri Jan 20 22:46:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 645096 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7456C38159 for ; Fri, 20 Jan 2023 23:23:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230078AbjATXXr (ORCPT ); Fri, 20 Jan 2023 18:23:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229755AbjATXXr (ORCPT ); Fri, 20 Jan 2023 18:23:47 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8B9673EE2; Fri, 20 Jan 2023 15:23:45 -0800 (PST) Received: from pps.filterd (m0279872.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KMddvj029658; Fri, 20 Jan 2023 22:48:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=fb0yRpVmhMXXS6AyR3FeSmfPN/D1GiSlLNNO1AqAe2I=; b=A3vzg1Up4h19vdn/REubMpgX9Z+o27tXJJOLLJpIYjP67QXoYx+ii3nlmT90cJ7YVnn/ BOHav4N5vQIbO+c1rpkM/OFSHA0a9lRtLjBFGNsZv6ctphci0QYoSq/8lKCZSAu+UT4S lVXn63q3ALX6Fl2Z0+FYQ9aplOL3IfOuBkJ7o/c41x163cV/XmR0Jzg3FpSQ2tCi2ldG DMn4Yw0/I29Ts9eQXeXn68uwJQYcAYyRHHAYHZNxDGQ3AFSt2dECnDS346LUKns7o6ZQ icamG3fnb8ix9Oloea/qa70R+oSoahkoppN7Aler9mOVxwwI6EyZZkSzWU5TQNuu5nOw oA== Received: from nasanppmta02.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7xm40k9n-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:48:11 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA02.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMmA7K029482 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:48:11 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:48:10 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala , Jonathan Corbet CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Bagas Sanjaya , Catalin Marinas , "Will Deacon" , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 26/27] virt: gunyah: Add irqfd interface Date: Fri, 20 Jan 2023 14:46:25 -0800 Message-ID: <20230120224627.4053418-27-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: M1eYgToxgYnRDIieRSUBBmswDMFw9s8A X-Proofpoint-GUID: M1eYgToxgYnRDIieRSUBBmswDMFw9s8A X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 malwarescore=0 spamscore=0 bulkscore=0 clxscore=1015 priorityscore=1501 mlxlogscore=999 impostorscore=0 mlxscore=0 suspectscore=0 adultscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Enable support for creating irqfds which can raise an interrupt on a Gunyah virtual machine. irqfds are exposed to userspace as a Gunyah VM function with the name "irqfd". If the VM devicetree is not configured to create a doorbell with the corresponding label, userspace will still be able to assert the eventfd but no interrupt will be raised on the guest. Co-developed-by: Prakruthi Deepak Heragu Signed-off-by: Prakruthi Deepak Heragu Signed-off-by: Elliot Berman --- Documentation/virt/gunyah/vm-manager.rst | 22 +++ drivers/virt/gunyah/Kconfig | 9 ++ drivers/virt/gunyah/Makefile | 1 + drivers/virt/gunyah/gunyah_irqfd.c | 166 +++++++++++++++++++++++ include/linux/gunyah.h | 5 + include/uapi/linux/gunyah.h | 11 +- 6 files changed, 213 insertions(+), 1 deletion(-) create mode 100644 drivers/virt/gunyah/gunyah_irqfd.c diff --git a/Documentation/virt/gunyah/vm-manager.rst b/Documentation/virt/gunyah/vm-manager.rst index d11267d59802..b6cf8db826b8 100644 --- a/Documentation/virt/gunyah/vm-manager.rst +++ b/Documentation/virt/gunyah/vm-manager.rst @@ -142,3 +142,25 @@ The vcpu type will register with the VM Manager to expect to control vCPU number `vcpu_id`. It returns a file descriptor allowing interaction with the vCPU. See the Gunyah vCPU API description sections for interacting with the Gunyah vCPU file descriptors. + +Type: "irqfd" +^^^^^^^^^^^^^ + +:: + + struct gh_fn_irqfd_arg { + __u32 fd; + __u32 label; + #define GH_IRQFD_LEVEL (1UL << 0) + #define GH_IRQFD_DEASSIGN (1UL << 1) + __u32 flags; + }; + +Allows setting an eventfd to directly trigger a guest interrupt. +irqfd.fd specifies the file descriptor to use as the eventfd. +irqfd.label corresponds to the doorbell label used in the guest VM's devicetree. +The irqfd is removed using the GH_IRQFD_DEASSIGN flag and specifying at least +the irqfd.label. + +GH_IRQFD_LEVEL configures the corresponding doorbell to behave like a level +triggered interrupt. diff --git a/drivers/virt/gunyah/Kconfig b/drivers/virt/gunyah/Kconfig index 4c1c6110b50e..2cde24d429d1 100644 --- a/drivers/virt/gunyah/Kconfig +++ b/drivers/virt/gunyah/Kconfig @@ -26,3 +26,12 @@ config GUNYAH_VCPU VMMs can also handle stage 2 faults of the vCPUs. Say Y/M here if unsure and you want to support Gunyah VMMs. + +config GUNYAH_IRQFD + tristate "Gunyah irqfd interface" + depends on GUNYAH + help + Enable kernel support for creating irqfds which can raise an interrupt + on Gunyah virtual machine. + + Say Y/M here if unsure and you want to support Gunyah VMMs. diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile index 2d1b604a7b03..6cf756bfa3c2 100644 --- a/drivers/virt/gunyah/Makefile +++ b/drivers/virt/gunyah/Makefile @@ -7,3 +7,4 @@ gunyah_rsc_mgr-y += rsc_mgr.o rsc_mgr_rpc.o vm_mgr.o vm_mgr_mm.o obj-$(CONFIG_GUNYAH) += gunyah_rsc_mgr.o obj-$(CONFIG_GUNYAH_VCPU) += gunyah_vcpu.o +obj-$(CONFIG_GUNYAH_IRQFD) += gunyah_irqfd.o diff --git a/drivers/virt/gunyah/gunyah_irqfd.c b/drivers/virt/gunyah/gunyah_irqfd.c new file mode 100644 index 000000000000..a3be9ca2377a --- /dev/null +++ b/drivers/virt/gunyah/gunyah_irqfd.c @@ -0,0 +1,166 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +struct gunyah_irqfd { + struct gunyah_resource *ghrsc; + struct gunyah_vm_resource_ticket ticket; + struct gunyah_vm_function *f; + + struct kref kref; + bool level; + + struct eventfd_ctx *ctx; + wait_queue_entry_t wait; + poll_table pt; + struct fd fd; +}; + +static void gh_irqfd_cleanup(struct kref *kref) +{ + struct gunyah_irqfd *irqfd = container_of(kref, struct gunyah_irqfd, kref); + + kfree(irqfd); +} + +static int irqfd_wakeup(wait_queue_entry_t *wait, unsigned int mode, int sync, void *key) +{ + struct gunyah_irqfd *irqfd = container_of(wait, struct gunyah_irqfd, wait); + __poll_t flags = key_to_poll(key); + u64 enable_mask = GH_DBL_NONBLOCK; + u64 old_flags; + int ret = 0; + + if (flags & EPOLLIN) { + if (irqfd->ghrsc) { + ret = gh_hypercall_dbl_send(irqfd->ghrsc->capid, enable_mask, &old_flags); + if (ret) + pr_err("Failed to assert irq %d\n", irqfd->f->fn.irqfd.label); + } + } + + return 0; +} + +static void irqfd_ptable_queue_proc(struct file *file, wait_queue_head_t *wqh, poll_table *pt) +{ + struct gunyah_irqfd *irq_ctx = container_of(pt, struct gunyah_irqfd, pt); + + add_wait_queue(wqh, &irq_ctx->wait); +} + +static int gunyah_irqfd_populate(struct gunyah_vm_resource_ticket *ticket, + struct gunyah_resource *ghrsc) +{ + struct gunyah_irqfd *irqfd = container_of(ticket, struct gunyah_irqfd, ticket); + u64 enable_mask = GH_DBL_NONBLOCK; + u64 ack_mask = ~0; + int ret = 0; + + irqfd->ghrsc = ghrsc; + if (irqfd->level) { + ret = gh_hypercall_dbl_set_mask(irqfd->ghrsc->capid, enable_mask, ack_mask); + if (ret) + pr_warn("irq %d couldn't be set as level triggered. Might cause IRQ storm if asserted\n", + irqfd->f->fn.irqfd.label); + } + kref_get(&irqfd->kref); + + return 0; +} + +static void gunyah_irqfd_unpopulate(struct gunyah_vm_resource_ticket *ticket, + struct gunyah_resource *ghrsc) +{ + struct gunyah_irqfd *irqfd = container_of(ticket, struct gunyah_irqfd, ticket); + u64 cnt; + + eventfd_ctx_remove_wait_queue(irqfd->ctx, &irqfd->wait, &cnt); + eventfd_ctx_put(irqfd->ctx); + fdput(irqfd->fd); + irqfd->ctx = NULL; + irqfd->fd.file = NULL; + irqfd->ghrsc = NULL; + kref_put(&irqfd->kref, gh_irqfd_cleanup); +} + +static long gunyah_irqfd_bind(struct gunyah_vm_function *f) +{ + __poll_t events; + struct gunyah_irqfd *irqfd; + long r; + + irqfd = kzalloc(sizeof(*irqfd), GFP_KERNEL); + if (!irqfd) + return -ENOMEM; + + irqfd->f = f; + f->data = irqfd; + + irqfd->fd = fdget(f->fn.irqfd.fd); + if (!irqfd->fd.file) { + r = -EBADF; + goto err_free; + } + + irqfd->ctx = eventfd_ctx_fileget(irqfd->fd.file); + if (IS_ERR(irqfd->ctx)) { + r = PTR_ERR(irqfd->ctx); + goto err_fdput; + } + + if (f->fn.irqfd.flags & GH_IRQFD_LEVEL) + irqfd->level = true; + + init_waitqueue_func_entry(&irqfd->wait, irqfd_wakeup); + init_poll_funcptr(&irqfd->pt, irqfd_ptable_queue_proc); + kref_init(&irqfd->kref); + + irqfd->ticket.resource_type = GUNYAH_RESOURCE_TYPE_BELL_TX; + irqfd->ticket.label = f->fn.irqfd.label; + irqfd->ticket.owner = THIS_MODULE; + irqfd->ticket.populate = gunyah_irqfd_populate; + irqfd->ticket.unpopulate = gunyah_irqfd_unpopulate; + + r = ghvm_add_resource_ticket(f->ghvm, &irqfd->ticket); + if (r) + goto err_ctx; + + events = vfs_poll(irqfd->fd.file, &irqfd->pt); + if (events & EPOLLIN) + pr_warn("Premature injection of interrupt\n"); + + return 0; +err_ctx: + eventfd_ctx_put(irqfd->ctx); +err_fdput: + fdput(irqfd->fd); +err_free: + kfree(irqfd); + return r; +} + +static void gunyah_irqfd_release(struct gunyah_vm_function *f) +{ + struct gunyah_irqfd *irqfd = f->data; + + /* unpopulate will trigger clean up of the eventfd */ + ghvm_remove_resource_ticket(irqfd->f->ghvm, &irqfd->ticket); +} + +DECLARE_GUNYAH_VM_FUNCTION_INIT(irqfd, gunyah_irqfd_bind, gunyah_irqfd_release); +MODULE_DESCRIPTION("Gunyah irqfds"); +MODULE_LICENSE("GPL"); diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h index ac4879940c10..6b363707a901 100644 --- a/include/linux/gunyah.h +++ b/include/linux/gunyah.h @@ -33,6 +33,11 @@ struct gunyah_resource { u32 rm_label; }; +/** + * Gunyah Doorbells + */ +#define GH_DBL_NONBLOCK BIT(32) + /** * Gunyah Message Queues */ diff --git a/include/uapi/linux/gunyah.h b/include/uapi/linux/gunyah.h index b4afb11f538a..a947f0317ca9 100644 --- a/include/uapi/linux/gunyah.h +++ b/include/uapi/linux/gunyah.h @@ -57,10 +57,19 @@ struct gh_fn_vcpu_arg { __u32 vcpu_id; }; +struct gh_fn_irqfd_arg { + __u32 fd; + __u32 label; +#define GH_IRQFD_LEVEL (1UL << 0) +#define GH_IRQFD_DEASSIGN (1UL << 1) + __u32 flags; +}; + struct gh_vm_function { char name[GUNYAH_FUNCTION_NAME_SIZE]; union { - struct gh_device_vcpu_arg vcpu; + struct gh_fn_vcpu_arg vcpu; + struct gh_fn_irqfd_arg irqfd; char data[GUNYAH_FUNCTION_MAX_ARG_SIZE]; }; }; From patchwork Fri Jan 20 22:46:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 645099 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20683C54EAA for ; Fri, 20 Jan 2023 22:50:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230283AbjATWuK (ORCPT ); Fri, 20 Jan 2023 17:50:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230228AbjATWuJ (ORCPT ); Fri, 20 Jan 2023 17:50:09 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C359C4DBDF; Fri, 20 Jan 2023 14:49:26 -0800 (PST) Received: from pps.filterd (m0279873.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30KLvekf003343; Fri, 20 Jan 2023 22:48:13 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=w6/aHGaEUJiqoCCNWVVmdNmnpXulOdKevHNVnLS9tfw=; b=hMLgkUDpPUW2CE6uo1tfIsnKGboCUqJpscYvRG5uUbLY2GjdFz035TJpdZWFbyKnALUW f5W6N3BsRhzaPZH6vcAx8o4SsrE280Luobmll+d60qSp8w2l0lk0AGmQzMqRZLG/F8eP 8fXMpd4MgvJVCsKJcHKy384Tx6+TYPzeA/91yU2elqcFo/+C0BBirCSRqBF3jgifKWwy Jt4Y50GYPqBo5AntJyDsajwxIcf5R+buOxEZNLFvZtOCZ9RSiDNC+8CSF8I1g2SB16V9 WtuLtD6x9Q9vOdfSrVK9jYQWhmEnd2s37REYt/bwrhvLSZGLo+nSW7rpI4di0EOeviWL yg== Received: from nasanppmta01.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3n7593uqyw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:48:13 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA01.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 30KMmC59005805 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 22:48:12 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Fri, 20 Jan 2023 14:48:11 -0800 From: Elliot Berman To: Bjorn Andersson , Alex Elder , Elliot Berman , Murali Nalajala , Jonathan Corbet CC: Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Bagas Sanjaya , Catalin Marinas , "Will Deacon" , Marc Zyngier , Jassi Brar , Sudeep Holla , , , , , Subject: [PATCH v9 27/27] virt: gunyah: Add ioeventfd Date: Fri, 20 Jan 2023 14:46:26 -0800 Message-ID: <20230120224627.4053418-28-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230120224627.4053418-1-quic_eberman@quicinc.com> References: <20230120224627.4053418-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: 6VKb8RmDFqQGIsBLbynaHrsPWYv04-50 X-Proofpoint-ORIG-GUID: 6VKb8RmDFqQGIsBLbynaHrsPWYv04-50 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1 definitions=2023-01-20_11,2023-01-20_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 mlxlogscore=999 spamscore=0 suspectscore=0 mlxscore=0 adultscore=0 clxscore=1015 impostorscore=0 priorityscore=1501 bulkscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301200218 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Allow userspace to attach an ioeventfd to an mmio address within the guest. Co-developed-by: Prakruthi Deepak Heragu Signed-off-by: Prakruthi Deepak Heragu Signed-off-by: Elliot Berman --- Documentation/virt/gunyah/vm-manager.rst | 21 +++++ drivers/virt/gunyah/Kconfig | 9 ++ drivers/virt/gunyah/Makefile | 1 + drivers/virt/gunyah/gunyah_ioeventfd.c | 109 +++++++++++++++++++++++ include/uapi/linux/gunyah.h | 10 +++ 5 files changed, 150 insertions(+) create mode 100644 drivers/virt/gunyah/gunyah_ioeventfd.c diff --git a/Documentation/virt/gunyah/vm-manager.rst b/Documentation/virt/gunyah/vm-manager.rst index b6cf8db826b8..19ec859a7912 100644 --- a/Documentation/virt/gunyah/vm-manager.rst +++ b/Documentation/virt/gunyah/vm-manager.rst @@ -164,3 +164,24 @@ the irqfd.label. GH_IRQFD_LEVEL configures the corresponding doorbell to behave like a level triggered interrupt. + +Type: "ioeventfd" +^^^^^^^^^^^^^^^^^ + +:: + + struct gh_fn_ioeventfd_arg { + __u64 datamatch; + __u64 addr; /* legal mmio address */ + __u32 len; /* 1, 2, 4, or 8 bytes */ + __s32 fd; + #define GH_IOEVENTFD_DATAMATCH (1UL << 0) + __u32 flags; + }; + +Attaches an ioeventfd to a legal mmio address within the guest. A guest write +in the registered address will signal the provided event instead of triggering +an exit on the GH_VCPU_RUN ioctl. + +If datamatch flag is set, the event will be signaled only if the written value +to the registered address is equal to datamatch in struct gh_fn_ioeventfd_arg. diff --git a/drivers/virt/gunyah/Kconfig b/drivers/virt/gunyah/Kconfig index 2cde24d429d1..bd8e31184962 100644 --- a/drivers/virt/gunyah/Kconfig +++ b/drivers/virt/gunyah/Kconfig @@ -35,3 +35,12 @@ config GUNYAH_IRQFD on Gunyah virtual machine. Say Y/M here if unsure and you want to support Gunyah VMMs. + +config GUNYAH_IOEVENTFD + tristate "Gunyah ioeventfd interface" + depends on GUNYAH + help + Enable kernel support for creating ioeventfds which can alert userspace + when a Gunyah virtual machine accesses a memory address. + + Say Y/M here if unsure and you want to support Gunyah VMMs. diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile index 6cf756bfa3c2..7347b1470491 100644 --- a/drivers/virt/gunyah/Makefile +++ b/drivers/virt/gunyah/Makefile @@ -8,3 +8,4 @@ obj-$(CONFIG_GUNYAH) += gunyah_rsc_mgr.o obj-$(CONFIG_GUNYAH_VCPU) += gunyah_vcpu.o obj-$(CONFIG_GUNYAH_IRQFD) += gunyah_irqfd.o +obj-$(CONFIG_GUNYAH_IOEVENTFD) += gunyah_ioeventfd.o diff --git a/drivers/virt/gunyah/gunyah_ioeventfd.c b/drivers/virt/gunyah/gunyah_ioeventfd.c new file mode 100644 index 000000000000..c16a6725fe7e --- /dev/null +++ b/drivers/virt/gunyah/gunyah_ioeventfd.c @@ -0,0 +1,109 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include + +struct gunyah_ioeventfd { + struct gunyah_vm_function *f; + struct gunyah_vm_io_handler io_handler; + + struct eventfd_ctx *ctx; +}; + +static int gh_write_ioeventfd(struct gunyah_vm_io_handler *io_dev, u64 addr, u32 len, u64 data) +{ + struct gunyah_ioeventfd *iofd = container_of(io_dev, struct gunyah_ioeventfd, io_handler); + + eventfd_signal(iofd->ctx, 1); + return 0; +} + +static struct gunyah_vm_io_handler_ops io_ops = { + .write = gh_write_ioeventfd, +}; + +static long gunyah_ioeventfd_bind(struct gunyah_vm_function *f) +{ + struct eventfd_ctx *ctx = NULL; + struct gunyah_ioeventfd *iofd; + const struct gh_fn_ioeventfd_arg *args = &f->fn.ioeventfd; + int ret; + + /* must be natural-word sized, or 0 to ignore length */ + switch (args->len) { + case 0: + case 1: + case 2: + case 4: + case 8: + break; + default: + return -EINVAL; + } + + /* check for range overflow */ + if (args->addr + args->len < args->addr) + return -EINVAL; + + /* ioeventfd with no length can't be combined with DATAMATCH */ + if (!args->len && (args->flags & GH_IOEVENTFD_DATAMATCH)) + return -EINVAL; + + ctx = eventfd_ctx_fdget(args->fd); + if (IS_ERR(ctx)) + return PTR_ERR(ctx); + + iofd = kzalloc(sizeof(*iofd), GFP_KERNEL); + if (!iofd) { + ret = -ENOMEM; + goto err_eventfd; + } + + f->data = iofd; + iofd->f = f; + + iofd->ctx = ctx; + + if (args->flags & GH_IOEVENTFD_DATAMATCH) { + iofd->io_handler.datamatch = true; + iofd->io_handler.len = args->len; + iofd->io_handler.data = args->datamatch; + } + iofd->io_handler.addr = args->addr; + iofd->io_handler.ops = &io_ops; + + ret = gh_vm_mgr_add_io_handler(f->ghvm, &iofd->io_handler); + if (ret) + goto err_io_dev_add; + + return 0; + +err_io_dev_add: + kfree(iofd); +err_eventfd: + eventfd_ctx_put(ctx); + return ret; +} + +static void gunyah_ioeventfd_release(struct gunyah_vm_function *f) +{ + struct gunyah_ioeventfd *iofd = f->data; + + eventfd_ctx_put(iofd->ctx); + gh_vm_mgr_remove_io_handler(iofd->f->ghvm, &iofd->io_handler); + kfree(iofd); +} + +DECLARE_GUNYAH_VM_FUNCTION_INIT(ioeventfd, gunyah_ioeventfd_bind, gunyah_ioeventfd_release); +MODULE_DESCRIPTION("Gunyah ioeventfds"); +MODULE_LICENSE("GPL"); diff --git a/include/uapi/linux/gunyah.h b/include/uapi/linux/gunyah.h index a947f0317ca9..3cc387f0831a 100644 --- a/include/uapi/linux/gunyah.h +++ b/include/uapi/linux/gunyah.h @@ -65,11 +65,21 @@ struct gh_fn_irqfd_arg { __u32 flags; }; +struct gh_fn_ioeventfd_arg { + __u64 datamatch; + __u64 addr; /* legal mmio address */ + __u32 len; /* 1, 2, 4, or 8 bytes; or 0 to ignore length */ + __s32 fd; +#define GH_IOEVENTFD_DATAMATCH (1UL << 0) + __u32 flags; +}; + struct gh_vm_function { char name[GUNYAH_FUNCTION_NAME_SIZE]; union { struct gh_fn_vcpu_arg vcpu; struct gh_fn_irqfd_arg irqfd; + struct gh_fn_ioeventfd_arg ioeventfd; char data[GUNYAH_FUNCTION_MAX_ARG_SIZE]; }; };