From patchwork Tue Feb 14 21:12:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 653527 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E609C05027 for ; Tue, 14 Feb 2023 21:13:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232248AbjBNVNu (ORCPT ); Tue, 14 Feb 2023 16:13:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232169AbjBNVNq (ORCPT ); Tue, 14 Feb 2023 16:13:46 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BD4EE2CFEF; Tue, 14 Feb 2023 13:13:41 -0800 (PST) Received: from pps.filterd (m0279868.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 31EIohiQ023146; Tue, 14 Feb 2023 21:13:25 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=BpczvFjXIF4w89r5EuSkN2NIsV/HuXG62UUKR5Nps4Q=; b=Isw2wOmxQZqFdcB9nXyDW/ZNv0oJ813sRIVPxqb73QKWDZZa32/6bcYTAQliU75UYh2N OGatPrAMjQIibY9e5RAGr1mvgKcp4c0rhENAQUaYH0Wwtp7NBofKwSO0b1w9G5roS+xf ozROeU0DSxwbjYnxNaymejP8uEupl63TUcVRm2H9g3qkbafxYDH6uIQwRFhCRjnhppiJ nTjs+TCMoGX5ILh6Dd+nx9a4zTz/COBQL++xucuT9E6jJlsZuOwBdoC0bM69s9+IQluA plwyz5T/pBvUcbAlTfazaBxf5LFEyT8vqLpCSduQ5NP3AMbeqcq6FXUf1ebKl3ADJE0y fA== Received: from nasanppmta05.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3nr6qkhwsy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:13:24 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA05.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 31ELDNRt018557 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:13:23 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Tue, 14 Feb 2023 13:13:22 -0800 From: Elliot Berman To: Alex Elder , Srinivas Kandagatla , Elliot Berman , Prakruthi Deepak Heragu , Rob Herring , Krzysztof Kozlowski CC: Murali Nalajala , Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Dmitry Baryshkov , Bjorn Andersson , "Konrad Dybcio" , Arnd Bergmann , "Greg Kroah-Hartman" , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Jassi Brar , , , , , , Rob Herring Subject: [PATCH v10 02/26] dt-bindings: Add binding for gunyah hypervisor Date: Tue, 14 Feb 2023 13:12:05 -0800 Message-ID: <20230214211229.3239350-3-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230214211229.3239350-1-quic_eberman@quicinc.com> References: <20230214211229.3239350-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01a.na.qualcomm.com (10.47.209.196) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: Z8eJEK1iRqVXd-3h3zgfI0JNLWIRSa9w X-Proofpoint-ORIG-GUID: Z8eJEK1iRqVXd-3h3zgfI0JNLWIRSa9w X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-14_15,2023-02-14_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 adultscore=0 clxscore=1015 bulkscore=0 impostorscore=0 suspectscore=0 lowpriorityscore=0 spamscore=0 priorityscore=1501 mlxscore=0 phishscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2302140183 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org When Linux is booted as a guest under the Gunyah hypervisor, the Gunyah Resource Manager applies a devicetree overlay describing the virtual platform configuration of the guest VM, such as the message queue capability IDs for communicating with the Resource Manager. This information is not otherwise discoverable by a VM: the Gunyah hypervisor core does not provide a direct interface to discover capability IDs nor a way to communicate with RM without having already known the corresponding message queue capability ID. Add the DT bindings that Gunyah adheres for the hypervisor node and message queues. Reviewed-by: Rob Herring Signed-off-by: Elliot Berman --- .../bindings/firmware/gunyah-hypervisor.yaml | 82 +++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml diff --git a/Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml b/Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml new file mode 100644 index 000000000000..3fc0b043ac3c --- /dev/null +++ b/Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml @@ -0,0 +1,82 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/firmware/gunyah-hypervisor.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Gunyah Hypervisor + +maintainers: + - Prakruthi Deepak Heragu + - Elliot Berman + +description: |+ + Gunyah virtual machines use this information to determine the capability IDs + of the message queues used to communicate with the Gunyah Resource Manager. + See also: https://github.com/quic/gunyah-resource-manager/blob/develop/src/vm_creation/dto_construct.c + +properties: + compatible: + const: gunyah-hypervisor + + "#address-cells": + description: Number of cells needed to represent 64-bit capability IDs. + const: 2 + + "#size-cells": + description: must be 0, because capability IDs are not memory address + ranges and do not have a size. + const: 0 + +patternProperties: + "^gunyah-resource-mgr(@.*)?": + type: object + description: + Resource Manager node which is required to communicate to Resource + Manager VM using Gunyah Message Queues. + + properties: + compatible: + const: gunyah-resource-manager + + reg: + items: + - description: Gunyah capability ID of the TX message queue + - description: Gunyah capability ID of the RX message queue + + interrupts: + items: + - description: Interrupt for the TX message queue + - description: Interrupt for the RX message queue + + additionalProperties: false + + required: + - compatible + - reg + - interrupts + +additionalProperties: false + +required: + - compatible + - "#address-cells" + - "#size-cells" + +examples: + - | + #include + + hypervisor { + #address-cells = <2>; + #size-cells = <0>; + compatible = "gunyah-hypervisor"; + + gunyah-resource-mgr@0 { + compatible = "gunyah-resource-manager"; + interrupts = , /* TX full IRQ */ + ; /* RX empty IRQ */ + reg = <0x00000000 0x00000000>, <0x00000000 0x00000001>; + /* TX, RX cap ids */ + }; + }; From patchwork Tue Feb 14 21:18:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 653526 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40F69C05027 for ; Tue, 14 Feb 2023 21:19:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232645AbjBNVTQ (ORCPT ); Tue, 14 Feb 2023 16:19:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42084 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232734AbjBNVTJ (ORCPT ); Tue, 14 Feb 2023 16:19:09 -0500 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C14922068B; Tue, 14 Feb 2023 13:18:57 -0800 (PST) Received: from pps.filterd (m0279866.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 31E7kIuj022302; Tue, 14 Feb 2023 21:18:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=OcZJ9FjJy7jbdYriOCDiXlSBEC0RxeLblvVoPyOxuDY=; b=N6cOSoYOAj0Ft6Z/+D6wCk5+yaKPeg+3XrNCL1xJ7FnLXoabf8vcFN6rT+aPqZBvYbk4 TFBZc+pGiQtzLcht43PL7lRE6a4daFoXXdaBkrR+vOnrWkPfd1WvtATXD3Jt0qhZ0NBo 5q0KVjrIwgH36fLfsiMueQ6HFwVV6YoldFgsVKKOAnpwPuD2qIIse6F5kATIGRf3S28+ zPCtbNSlt+XLsL6Py/8tv57rxS2tbiWhMp+y6Mb6Scnb8iUQC4ZlbJmVgHl8sKqTInhM DS4AKZ2QOH6O62X2fQVaLUXIVTDbM89QxtLarJBfpodLqDWBC/RCwuFgUrwk30VQ9gWK tw== Received: from nasanppmta01.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3nr661a1cx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:18:44 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA01.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 31ELIhv1031236 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:18:43 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Tue, 14 Feb 2023 13:18:43 -0800 From: Elliot Berman To: Alex Elder , Srinivas Kandagatla , Catalin Marinas , Will Deacon , Elliot Berman , Prakruthi Deepak Heragu CC: Murali Nalajala , Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Dmitry Baryshkov , Bjorn Andersson , "Konrad Dybcio" , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Jassi Brar , , , , , Subject: [PATCH v10 04/26] virt: gunyah: Add hypercalls to identify Gunyah Date: Tue, 14 Feb 2023 13:18:27 -0800 Message-ID: <20230214211828.3277821-1-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230214211229.3239350-1-quic_eberman@quicinc.com> References: <20230214211229.3239350-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01a.na.qualcomm.com (10.47.209.196) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: Bz0d_XdsEOrmOzjbrlT5RzDR9yBkxstn X-Proofpoint-GUID: Bz0d_XdsEOrmOzjbrlT5RzDR9yBkxstn X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-14_15,2023-02-14_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 mlxlogscore=999 mlxscore=0 spamscore=0 impostorscore=0 lowpriorityscore=0 malwarescore=0 priorityscore=1501 clxscore=1015 adultscore=0 bulkscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2302140183 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Add hypercalls to identify when Linux is running a virtual machine under Gunyah. There are two calls to help identify Gunyah: 1. gh_hypercall_get_uid() returns a UID when running under a Gunyah hypervisor. 2. gh_hypercall_hyp_identify() returns build information and a set of feature flags that are supported by Gunyah. Signed-off-by: Elliot Berman --- arch/arm64/Kbuild | 1 + arch/arm64/gunyah/Makefile | 3 ++ arch/arm64/gunyah/gunyah_hypercall.c | 61 ++++++++++++++++++++++++++++ drivers/virt/Kconfig | 2 + drivers/virt/gunyah/Kconfig | 13 ++++++ include/linux/gunyah.h | 33 +++++++++++++++ 6 files changed, 113 insertions(+) create mode 100644 arch/arm64/gunyah/Makefile create mode 100644 arch/arm64/gunyah/gunyah_hypercall.c create mode 100644 drivers/virt/gunyah/Kconfig diff --git a/arch/arm64/Kbuild b/arch/arm64/Kbuild index 5bfbf7d79c99..e4847ba0e3c9 100644 --- a/arch/arm64/Kbuild +++ b/arch/arm64/Kbuild @@ -3,6 +3,7 @@ obj-y += kernel/ mm/ net/ obj-$(CONFIG_KVM) += kvm/ obj-$(CONFIG_XEN) += xen/ obj-$(subst m,y,$(CONFIG_HYPERV)) += hyperv/ +obj-$(CONFIG_GUNYAH) += gunyah/ obj-$(CONFIG_CRYPTO) += crypto/ # for cleaning diff --git a/arch/arm64/gunyah/Makefile b/arch/arm64/gunyah/Makefile new file mode 100644 index 000000000000..84f1e38cafb1 --- /dev/null +++ b/arch/arm64/gunyah/Makefile @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_GUNYAH) += gunyah_hypercall.o diff --git a/arch/arm64/gunyah/gunyah_hypercall.c b/arch/arm64/gunyah/gunyah_hypercall.c new file mode 100644 index 000000000000..f30d06ee80cf --- /dev/null +++ b/arch/arm64/gunyah/gunyah_hypercall.c @@ -0,0 +1,61 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#include +#include +#include + +static const uint32_t gunyah_known_uuids[][4] = { + {0x19bd54bd, 0x0b37571b, 0x946f609b, 0x54539de6}, /* QC_HYP (Qualcomm's build) */ + {0x673d5f14, 0x9265ce36, 0xa4535fdb, 0xc1d58fcd}, /* GUNYAH (open source build) */ +}; + +bool arch_is_gunyah_guest(void) +{ + struct arm_smccc_res res; + u32 uid[4]; + int i; + + arm_smccc_1_1_hvc(ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID, &res); + + uid[0] = lower_32_bits(res.a0); + uid[1] = lower_32_bits(res.a1); + uid[2] = lower_32_bits(res.a2); + uid[3] = lower_32_bits(res.a3); + + for (i = 0; i < ARRAY_SIZE(gunyah_known_uuids); i++) + if (!memcmp(uid, gunyah_known_uuids[i], sizeof(uid))) + break; + + return i != ARRAY_SIZE(gunyah_known_uuids); +} +EXPORT_SYMBOL_GPL(arch_is_gunyah_guest); + +#define GH_HYPERCALL(fn) ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, ARM_SMCCC_SMC_64, \ + ARM_SMCCC_OWNER_VENDOR_HYP, \ + fn) + +#define GH_HYPERCALL_HYP_IDENTIFY GH_HYPERCALL(0x8000) + +/** + * gh_hypercall_hyp_identify() - Returns build information and feature flags + * supported by Gunyah. + * @hyp_identity: filled by the hypercall with the API info and feature flags. + */ +void gh_hypercall_hyp_identify(struct gh_hypercall_hyp_identify_resp *hyp_identity) +{ + struct arm_smccc_res res; + + arm_smccc_1_1_hvc(GH_HYPERCALL_HYP_IDENTIFY, &res); + + hyp_identity->api_info = res.a0; + hyp_identity->flags[0] = res.a1; + hyp_identity->flags[1] = res.a2; + hyp_identity->flags[2] = res.a3; +} +EXPORT_SYMBOL_GPL(gh_hypercall_hyp_identify); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("Gunyah Hypervisor Hypercalls"); diff --git a/drivers/virt/Kconfig b/drivers/virt/Kconfig index f79ab13a5c28..85bd6626ffc9 100644 --- a/drivers/virt/Kconfig +++ b/drivers/virt/Kconfig @@ -54,4 +54,6 @@ source "drivers/virt/coco/sev-guest/Kconfig" source "drivers/virt/coco/tdx-guest/Kconfig" +source "drivers/virt/gunyah/Kconfig" + endif diff --git a/drivers/virt/gunyah/Kconfig b/drivers/virt/gunyah/Kconfig new file mode 100644 index 000000000000..1a737694c333 --- /dev/null +++ b/drivers/virt/gunyah/Kconfig @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: GPL-2.0-only + +config GUNYAH + tristate "Gunyah Virtualization drivers" + depends on ARM64 + depends on MAILBOX + help + The Gunyah drivers are the helper interfaces that run in a guest VM + such as basic inter-VM IPC and signaling mechanisms, and higher level + services such as memory/device sharing, IRQ sharing, and so on. + + Say Y/M here to enable the drivers needed to interact in a Gunyah + virtual environment. diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h index 59ef4c735ae8..3fef2854c5e1 100644 --- a/include/linux/gunyah.h +++ b/include/linux/gunyah.h @@ -6,8 +6,10 @@ #ifndef _LINUX_GUNYAH_H #define _LINUX_GUNYAH_H +#include #include #include +#include /******************************************************************************/ /* Common arch-independent definitions for Gunyah hypercalls */ @@ -79,4 +81,35 @@ static inline int gh_remap_error(enum gh_error gh_error) } } +enum gh_api_feature { + GH_API_FEATURE_DOORBELL, + GH_API_FEATURE_MSGQUEUE, + GH_API_FEATURE_VCPU, + GH_API_FEATURE_MEMEXTENT, +}; + +bool arch_is_gunyah_guest(void); + +u16 gh_api_version(void); +bool gh_api_has_feature(enum gh_api_feature feature); + +#define GUNYAH_API_V1 1 + +#define GH_API_INFO_API_VERSION_MASK GENMASK_ULL(13, 0) +#define GH_API_INFO_BIG_ENDIAN BIT_ULL(14) +#define GH_API_INFO_IS_64BIT BIT_ULL(15) +#define GH_API_INFO_VARIANT_MASK GENMASK_ULL(63, 56) + +#define GH_IDENTIFY_DOORBELL BIT_ULL(1) +#define GH_IDENTIFY_MSGQUEUE BIT_ULL(2) +#define GH_IDENTIFY_VCPU BIT_ULL(5) +#define GH_IDENTIFY_MEMEXTENT BIT_ULL(6) + +struct gh_hypercall_hyp_identify_resp { + u64 api_info; + u64 flags[3]; +}; + +void gh_hypercall_hyp_identify(struct gh_hypercall_hyp_identify_resp *hyp_identity); + #endif From patchwork Tue Feb 14 21:21:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 653525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A7AFC6379F for ; Tue, 14 Feb 2023 21:22:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232169AbjBNVWd (ORCPT ); Tue, 14 Feb 2023 16:22:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229648AbjBNVWc (ORCPT ); Tue, 14 Feb 2023 16:22:32 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E38D02CFC0; Tue, 14 Feb 2023 13:22:27 -0800 (PST) Received: from pps.filterd (m0279870.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 31EK5wND023315; Tue, 14 Feb 2023 21:22:13 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=KL7exBajbG5QSl+dA5y0aBsF2CmemKZa9gCGaSpGY1o=; b=O3N49I12FB0caS+QFhtmI22rrB58r8HCcMdFVOCldNEoXHXJG/ADVnrJ0Xo/0dX/vARf mAstJU+qEi7Vt01kKzX+WZwG+MqaHfU3EI/j83f55f625uERnyO2xsmVV4SXiw5535pY mhqsKPf4F9WIgfTEYURLMHDRlgXij/842FuSApLMeYcjDW+M+CuIRSDMipKvtyn7dEGI PTvveLbFj1JI0jSzX+QRKXWWpUm95/R0BJDkx6wevgI5zrAALui8n/Nb8ttZLNYPSluL GF+wuQ3SyTkyTa1xvIK7tdrWIbs6H/Tn07+NYF09VJlxQNwOJD5RE/QtQTNiEy/Im5YO yA== Received: from nasanppmta03.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3nrf7v0dpg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:22:13 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA03.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 31ELMCJ7031805 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:22:12 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Tue, 14 Feb 2023 13:22:11 -0800 From: Elliot Berman To: Alex Elder , Srinivas Kandagatla , Elliot Berman , Prakruthi Deepak Heragu CC: Murali Nalajala , Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Dmitry Baryshkov , Bjorn Andersson , "Konrad Dybcio" , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Jassi Brar , , , , , Subject: [PATCH v10 05/26] virt: gunyah: Identify hypervisor version Date: Tue, 14 Feb 2023 13:21:21 -0800 Message-ID: <20230214212143.3298451-1-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230214211229.3239350-1-quic_eberman@quicinc.com> References: <20230214211229.3239350-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01a.na.qualcomm.com (10.47.209.196) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: F92RGaNwTqTQe-V-hLc3FHPsfaW-kdfu X-Proofpoint-GUID: F92RGaNwTqTQe-V-hLc3FHPsfaW-kdfu X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-14_15,2023-02-14_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 priorityscore=1501 mlxscore=0 bulkscore=0 mlxlogscore=999 suspectscore=0 spamscore=0 phishscore=0 malwarescore=0 impostorscore=0 adultscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2302140183 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Export the version of Gunyah which is reported via the hyp_identify hypercall. Increments of the major API version indicate possibly backwards incompatible changes. Export the hypervisor identity so that Gunyah drivers can act according to the major API version. Signed-off-by: Elliot Berman --- drivers/virt/Makefile | 1 + drivers/virt/gunyah/Makefile | 3 ++ drivers/virt/gunyah/gunyah.c | 54 ++++++++++++++++++++++++++++++++++++ 3 files changed, 58 insertions(+) create mode 100644 drivers/virt/gunyah/Makefile create mode 100644 drivers/virt/gunyah/gunyah.c diff --git a/drivers/virt/Makefile b/drivers/virt/Makefile index e9aa6fc96fab..a5817e2d7d71 100644 --- a/drivers/virt/Makefile +++ b/drivers/virt/Makefile @@ -12,3 +12,4 @@ obj-$(CONFIG_ACRN_HSM) += acrn/ obj-$(CONFIG_EFI_SECRET) += coco/efi_secret/ obj-$(CONFIG_SEV_GUEST) += coco/sev-guest/ obj-$(CONFIG_INTEL_TDX_GUEST) += coco/tdx-guest/ +obj-y += gunyah/ diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile new file mode 100644 index 000000000000..34f32110faf9 --- /dev/null +++ b/drivers/virt/gunyah/Makefile @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_GUNYAH) += gunyah.o diff --git a/drivers/virt/gunyah/gunyah.c b/drivers/virt/gunyah/gunyah.c new file mode 100644 index 000000000000..776e83c6920d --- /dev/null +++ b/drivers/virt/gunyah/gunyah.c @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#define pr_fmt(fmt) "gunyah: " fmt + +#include +#include +#include +#include + +static struct gh_hypercall_hyp_identify_resp gunyah_api; + +u16 gh_api_version(void) +{ + return FIELD_GET(GH_API_INFO_API_VERSION_MASK, gunyah_api.api_info); +} +EXPORT_SYMBOL_GPL(gh_api_version); + +bool gh_api_has_feature(enum gh_api_feature feature) +{ + switch (feature) { + case GH_API_FEATURE_DOORBELL: + return !!(gunyah_api.flags[0] & GH_IDENTIFY_DOORBELL); + case GH_API_FEATURE_MSGQUEUE: + return !!(gunyah_api.flags[0] & GH_IDENTIFY_MSGQUEUE); + case GH_API_FEATURE_VCPU: + return !!(gunyah_api.flags[0] & GH_IDENTIFY_VCPU); + case GH_API_FEATURE_MEMEXTENT: + return !!(gunyah_api.flags[0] & GH_IDENTIFY_MEMEXTENT); + default: + return false; + } +} +EXPORT_SYMBOL_GPL(gh_api_has_feature); + +static int __init gunyah_init(void) +{ + if (!arch_is_gunyah_guest()) + return -ENODEV; + + gh_hypercall_hyp_identify(&gunyah_api); + + pr_info("Running under Gunyah hypervisor %llx/v%u\n", + FIELD_GET(GH_API_INFO_VARIANT_MASK, gunyah_api.api_info), + gh_api_version()); + + return 0; +} +arch_initcall(gunyah_init); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("Gunyah Hypervisor Driver"); From patchwork Tue Feb 14 21:23:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 653524 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA304C6379F for ; Tue, 14 Feb 2023 21:24:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231687AbjBNVY1 (ORCPT ); Tue, 14 Feb 2023 16:24:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47600 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229526AbjBNVY1 (ORCPT ); Tue, 14 Feb 2023 16:24:27 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C3DD9303EB; Tue, 14 Feb 2023 13:24:01 -0800 (PST) Received: from pps.filterd (m0279869.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 31EK6acu013965; Tue, 14 Feb 2023 21:23:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=cJSfHj1gY99JkxgLa/5h8ouZU9zZ/YrbzbUv5+wZGxY=; b=BGy5r49J2l4UqCzNbNXXvDiRHMetEjjO+08se3qAMQQyWXC5LuqJMulNW6kXazLBKFZV 7hNoH/bFoecrbQqLabmQr9R6d3qmwj+MJPlYXvBeO7nQXxBYn2XyRfPQmJ8UtoEcxqVr R3368zlBFvYJQdlRszM1nb7mNv8cmIrb3372/BOAia1QB5OQqSdFI+9D0e4q2l9KS+QF 7Onx+CWAi8QJsaH0CbS2gMpUUAsY0nw0DkVVrSromj1s7mGYVdB6b0oUEEVRsG28uIgx XzTilXtiCrSZy/58iMneWJ340GmFyx/KwBr465mJIWxQeXC5Zk3helMj2/ZxXk4LUd9t xA== Received: from nasanppmta03.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3nqts7ufe6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:23:43 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA03.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 31ELNg50002462 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:23:42 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Tue, 14 Feb 2023 13:23:41 -0800 From: Elliot Berman To: Alex Elder , Srinivas Kandagatla , Elliot Berman , Prakruthi Deepak Heragu CC: Murali Nalajala , Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Dmitry Baryshkov , Bjorn Andersson , "Konrad Dybcio" , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Jassi Brar , , , , , Subject: [PATCH v10 08/26] gunyah: rsc_mgr: Add resource manager RPC core Date: Tue, 14 Feb 2023 13:23:25 -0800 Message-ID: <20230214212327.3310128-1-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230214211229.3239350-1-quic_eberman@quicinc.com> References: <20230214211229.3239350-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01a.na.qualcomm.com (10.47.209.196) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: wE2JoN7dBSnwTI1TC4mU8QISHlYQJVoi X-Proofpoint-ORIG-GUID: wE2JoN7dBSnwTI1TC4mU8QISHlYQJVoi X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-14_15,2023-02-14_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 priorityscore=1501 bulkscore=0 lowpriorityscore=0 suspectscore=0 mlxlogscore=999 mlxscore=0 impostorscore=0 spamscore=0 phishscore=0 clxscore=1015 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2302140183 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org The resource manager is a special virtual machine which is always running on a Gunyah system. It provides APIs for creating and destroying VMs, secure memory management, sharing/lending of memory between VMs, and setup of inter-VM communication. Calls to the resource manager are made via message queues. This patch implements the basic probing and RPC mechanism to make those API calls. Request/response calls can be made with gh_rm_call. Drivers can also register to notifications pushed by RM via gh_rm_register_notifier Specific API calls that resource manager supports will be implemented in subsequent patches. Signed-off-by: Elliot Berman --- drivers/virt/gunyah/Makefile | 3 + drivers/virt/gunyah/rsc_mgr.c | 604 +++++++++++++++++++++++++++++++++ drivers/virt/gunyah/rsc_mgr.h | 77 +++++ include/linux/gunyah_rsc_mgr.h | 24 ++ 4 files changed, 708 insertions(+) create mode 100644 drivers/virt/gunyah/rsc_mgr.c create mode 100644 drivers/virt/gunyah/rsc_mgr.h create mode 100644 include/linux/gunyah_rsc_mgr.h diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile index 34f32110faf9..cc864ff5abbb 100644 --- a/drivers/virt/gunyah/Makefile +++ b/drivers/virt/gunyah/Makefile @@ -1,3 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_GUNYAH) += gunyah.o + +gunyah_rsc_mgr-y += rsc_mgr.o +obj-$(CONFIG_GUNYAH) += gunyah_rsc_mgr.o diff --git a/drivers/virt/gunyah/rsc_mgr.c b/drivers/virt/gunyah/rsc_mgr.c new file mode 100644 index 000000000000..2a47139873a8 --- /dev/null +++ b/drivers/virt/gunyah/rsc_mgr.c @@ -0,0 +1,604 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "rsc_mgr.h" + +#define RM_RPC_API_VERSION_MASK GENMASK(3, 0) +#define RM_RPC_HEADER_WORDS_MASK GENMASK(7, 4) +#define RM_RPC_API_VERSION FIELD_PREP(RM_RPC_API_VERSION_MASK, 1) +#define RM_RPC_HEADER_WORDS FIELD_PREP(RM_RPC_HEADER_WORDS_MASK, \ + (sizeof(struct gh_rm_rpc_hdr) / sizeof(u32))) +#define RM_RPC_API (RM_RPC_API_VERSION | RM_RPC_HEADER_WORDS) + +#define RM_RPC_TYPE_CONTINUATION 0x0 +#define RM_RPC_TYPE_REQUEST 0x1 +#define RM_RPC_TYPE_REPLY 0x2 +#define RM_RPC_TYPE_NOTIF 0x3 +#define RM_RPC_TYPE_MASK GENMASK(1, 0) + +#define GH_RM_MAX_NUM_FRAGMENTS 62 +#define RM_RPC_FRAGMENTS_MASK GENMASK(7, 2) + +struct gh_rm_rpc_hdr { + u8 api; + u8 type; + __le16 seq; + __le32 msg_id; +} __packed; + +struct gh_rm_rpc_reply_hdr { + struct gh_rm_rpc_hdr hdr; + __le32 err_code; /* GH_RM_ERROR_* */ +} __packed; + +#define GH_RM_MAX_MSG_SIZE (GH_MSGQ_MAX_MSG_SIZE - sizeof(struct gh_rm_rpc_hdr)) + +/** + * struct gh_rm_connection - Represents a complete message from resource manager + * @payload: Combined payload of all the fragments (msg headers stripped off). + * @size: Size of the payload received so far. + * @msg_id: Message ID from the header. + * @num_fragments: total number of fragments expected to be received. + * @fragments_received: fragments received so far. + * @reply: Fields used for request/reply sequences + * @ret: Linux return code, set in case there was an error processing connection + * @type: RM_RPC_TYPE_REPLY or RM_RPC_TYPE_NOTIF. + * @rm_error: For request/reply sequences with standard replies. + * @seq: Sequence ID for the main message. + * @seq_done: Signals caller that the RM reply has been received + * @notification: Fields used for notifiations + * @work: Triggered when all fragments of a notification received + */ +struct gh_rm_connection { + void *payload; + size_t size; + __le32 msg_id; + u8 type; + + u8 num_fragments; + u8 fragments_received; + + union { + struct { + int ret; + u16 seq; + enum gh_rm_error rm_error; + struct completion seq_done; + } reply; + + struct { + struct gh_rm *rm; + struct work_struct work; + } notification; + }; +}; + +struct gh_rm { + struct device *dev; + struct gunyah_resource tx_ghrsc, rx_ghrsc; + struct gh_msgq msgq; + struct mbox_client msgq_client; + struct gh_rm_connection *active_rx_connection; + int last_tx_ret; + + struct idr call_idr; + struct mutex call_idr_lock; + + struct kmem_cache *cache; + struct mutex send_lock; + struct blocking_notifier_head nh; +}; + +static struct gh_rm_connection *gh_rm_alloc_connection(__le32 msg_id, u8 type) +{ + struct gh_rm_connection *connection; + + connection = kzalloc(sizeof(*connection), GFP_KERNEL); + if (!connection) + return ERR_PTR(-ENOMEM); + + connection->type = type; + connection->msg_id = msg_id; + + return connection; +} + +static int gh_rm_init_connection_payload(struct gh_rm_connection *connection, void *msg, + size_t hdr_size, size_t msg_size) +{ + size_t max_buf_size, payload_size; + struct gh_rm_rpc_hdr *hdr = msg; + + if (hdr_size > msg_size) + return -EINVAL; + + payload_size = msg_size - hdr_size; + + connection->num_fragments = FIELD_GET(RM_RPC_FRAGMENTS_MASK, hdr->type); + connection->fragments_received = 0; + + /* There's not going to be any payload, no need to allocate buffer. */ + if (!payload_size && !connection->num_fragments) + return 0; + + if (connection->num_fragments > GH_RM_MAX_NUM_FRAGMENTS) + return -EINVAL; + + max_buf_size = payload_size + (connection->num_fragments * GH_RM_MAX_MSG_SIZE); + + connection->payload = kzalloc(max_buf_size, GFP_KERNEL); + if (!connection->payload) + return -ENOMEM; + + memcpy(connection->payload, msg + hdr_size, payload_size); + connection->size = payload_size; + return 0; +} + +static void gh_rm_notif_work(struct work_struct *work) +{ + struct gh_rm_connection *connection = container_of(work, struct gh_rm_connection, + notification.work); + struct gh_rm *rm = connection->notification.rm; + + blocking_notifier_call_chain(&rm->nh, connection->msg_id, connection->payload); + + put_gh_rm(rm); + kfree(connection->payload); + kfree(connection); +} + +static struct gh_rm_connection *gh_rm_process_notif(struct gh_rm *rm, void *msg, size_t msg_size) +{ + struct gh_rm_connection *connection; + struct gh_rm_rpc_hdr *hdr = msg; + int ret; + + connection = gh_rm_alloc_connection(hdr->msg_id, RM_RPC_TYPE_NOTIF); + if (IS_ERR(connection)) { + dev_err(rm->dev, "Failed to alloc connection for notification: %ld, dropping.\n", + PTR_ERR(connection)); + return NULL; + } + + get_gh_rm(rm); + connection->notification.rm = rm; + INIT_WORK(&connection->notification.work, gh_rm_notif_work); + + ret = gh_rm_init_connection_payload(connection, msg, sizeof(*hdr), msg_size); + if (ret) { + dev_err(rm->dev, "Failed to initialize connection buffer for notification: %d\n", + ret); + kfree(connection); + return NULL; + } + + return connection; +} + +static struct gh_rm_connection *gh_rm_process_rply(struct gh_rm *rm, void *msg, size_t msg_size) +{ + struct gh_rm_rpc_reply_hdr *reply_hdr = msg; + struct gh_rm_connection *connection; + u16 seq_id = le16_to_cpu(reply_hdr->hdr.seq); + + mutex_lock(&rm->call_idr_lock); + connection = idr_find(&rm->call_idr, seq_id); + mutex_unlock(&rm->call_idr_lock); + + if (!connection || connection->msg_id != reply_hdr->hdr.msg_id) + return NULL; + + if (gh_rm_init_connection_payload(connection, msg, sizeof(*reply_hdr), msg_size)) { + dev_err(rm->dev, "Failed to alloc connection buffer for sequence %d\n", seq_id); + /* Send connection complete and error the client. */ + connection->reply.ret = -ENOMEM; + complete(&connection->reply.seq_done); + return NULL; + } + + connection->reply.rm_error = le32_to_cpu(reply_hdr->err_code); + return connection; +} + +static int gh_rm_process_cont(struct gh_rm *rm, struct gh_rm_connection *connection, + void *msg, size_t msg_size) +{ + struct gh_rm_rpc_hdr *hdr = msg; + size_t payload_size = msg_size - sizeof(*hdr); + + /* + * hdr->fragments and hdr->msg_id preserves the value from first reply + * or notif message. To detect mishandling, check it's still intact. + */ + if (connection->msg_id != hdr->msg_id || + connection->num_fragments != FIELD_GET(RM_RPC_FRAGMENTS_MASK, hdr->type)) + return -EINVAL; + + memcpy(connection->payload + connection->size, msg + sizeof(*hdr), payload_size); + connection->size += payload_size; + connection->fragments_received++; + return 0; +} + +static void gh_rm_abort_connection(struct gh_rm_connection *connection) +{ + switch (connection->type) { + case RM_RPC_TYPE_REPLY: + connection->reply.ret = -EIO; + complete(&connection->reply.seq_done); + break; + case RM_RPC_TYPE_NOTIF: + fallthrough; + default: + kfree(connection->payload); + kfree(connection); + } +} + +static bool gh_rm_complete_connection(struct gh_rm *rm, struct gh_rm_connection *connection) +{ + if (!connection || connection->fragments_received != connection->num_fragments) + return false; + + switch (connection->type) { + case RM_RPC_TYPE_REPLY: + complete(&connection->reply.seq_done); + break; + case RM_RPC_TYPE_NOTIF: + schedule_work(&connection->notification.work); + break; + default: + dev_err(rm->dev, "Invalid message type (%d) received\n", connection->type); + gh_rm_abort_connection(connection); + break; + } + + return true; +} + +static void gh_rm_msgq_rx_data(struct mbox_client *cl, void *mssg) +{ + struct gh_rm *rm = container_of(cl, struct gh_rm, msgq_client); + struct gh_msgq_rx_data *rx_data = mssg; + size_t msg_size = rx_data->length; + void *msg = rx_data->data; + struct gh_rm_rpc_hdr *hdr; + + if (msg_size <= sizeof(*hdr) || msg_size > GH_MSGQ_MAX_MSG_SIZE) + return; + + hdr = msg; + if (hdr->api != RM_RPC_API) { + dev_err(rm->dev, "Unknown RM RPC API version: %x\n", hdr->api); + return; + } + + switch (FIELD_GET(RM_RPC_TYPE_MASK, hdr->type)) { + case RM_RPC_TYPE_NOTIF: + rm->active_rx_connection = gh_rm_process_notif(rm, msg, msg_size); + break; + case RM_RPC_TYPE_REPLY: + rm->active_rx_connection = gh_rm_process_rply(rm, msg, msg_size); + break; + case RM_RPC_TYPE_CONTINUATION: + if (gh_rm_process_cont(rm, rm->active_rx_connection, msg, msg_size)) { + gh_rm_abort_connection(rm->active_rx_connection); + rm->active_rx_connection = NULL; + } + break; + default: + dev_err(rm->dev, "Invalid message type (%lu) received\n", + FIELD_GET(RM_RPC_TYPE_MASK, hdr->type)); + return; + } + + if (gh_rm_complete_connection(rm, rm->active_rx_connection)) + rm->active_rx_connection = NULL; +} + +static void gh_rm_msgq_tx_done(struct mbox_client *cl, void *mssg, int r) +{ + struct gh_rm *rm = container_of(cl, struct gh_rm, msgq_client); + + kmem_cache_free(rm->cache, mssg); + rm->last_tx_ret = r; +} + +static int gh_rm_send_request(struct gh_rm *rm, u32 message_id, + const void *req_buff, size_t req_buff_size, + struct gh_rm_connection *connection) +{ + u8 msg_type = FIELD_PREP(RM_RPC_TYPE_MASK, RM_RPC_TYPE_REQUEST); + size_t buff_size_remaining = req_buff_size; + const void *req_buff_curr = req_buff; + struct gh_msgq_tx_data *msg; + struct gh_rm_rpc_hdr *hdr; + u32 cont_fragments = 0; + size_t payload_size; + void *payload; + int ret; + + if (req_buff_size) + cont_fragments = (req_buff_size - 1) / GH_RM_MAX_MSG_SIZE; + + if (req_buff_size > GH_RM_MAX_NUM_FRAGMENTS * GH_RM_MAX_MSG_SIZE) { + pr_warn("Limit exceeded for the number of fragments: %u\n", cont_fragments); + dump_stack(); + return -E2BIG; + } + + ret = mutex_lock_interruptible(&rm->send_lock); + if (ret) + return ret; + + /* Consider also the 'request' packet for the loop count */ + do { + msg = kmem_cache_zalloc(rm->cache, GFP_KERNEL); + if (!msg) { + ret = -ENOMEM; + goto out; + } + + /* Fill header */ + hdr = (struct gh_rm_rpc_hdr *)msg->data; + hdr->api = RM_RPC_API; + hdr->type = msg_type | FIELD_PREP(RM_RPC_FRAGMENTS_MASK, cont_fragments); + hdr->seq = cpu_to_le16(connection->reply.seq); + hdr->msg_id = cpu_to_le32(message_id); + + /* Copy payload */ + payload = hdr + 1; + payload_size = min(buff_size_remaining, GH_RM_MAX_MSG_SIZE); + memcpy(payload, req_buff_curr, payload_size); + req_buff_curr += payload_size; + buff_size_remaining -= payload_size; + + /* Force the last fragment to immediately alert the receiver */ + msg->push = !buff_size_remaining; + msg->length = sizeof(*hdr) + payload_size; + + ret = mbox_send_message(gh_msgq_chan(&rm->msgq), msg); + if (ret < 0) { + kmem_cache_free(rm->cache, msg); + break; + } + + if (rm->last_tx_ret) { + ret = rm->last_tx_ret; + break; + } + + msg_type = FIELD_PREP(RM_RPC_TYPE_MASK, RM_RPC_TYPE_CONTINUATION); + } while (buff_size_remaining); + +out: + mutex_unlock(&rm->send_lock); + return ret < 0 ? ret : 0; +} + +/** + * gh_rm_call: Achieve request-response type communication with RPC + * @rm: Pointer to Gunyah resource manager internal data + * @message_id: The RM RPC message-id + * @req_buff: Request buffer that contains the payload + * @req_buff_size: Total size of the payload + * @resp_buf: Pointer to a response buffer + * @resp_buff_size: Size of the response buffer + * + * Make a request to the RM-VM and wait for reply back. For a successful + * response, the function returns the payload. The size of the payload is set in + * resp_buff_size. The resp_buf should be freed by the caller. + * + * req_buff should be not NULL for req_buff_size >0. If req_buff_size == 0, + * req_buff *can* be NULL and no additional payload is sent. + * + * Context: Process context. Will sleep waiting for reply. + * Return: 0 on success. <0 if error. + */ +int gh_rm_call(struct gh_rm *rm, u32 message_id, void *req_buff, size_t req_buff_size, + void **resp_buf, size_t *resp_buff_size) +{ + struct gh_rm_connection *connection; + int ret; + + /* message_id 0 is reserved. req_buff_size implies req_buf is not NULL */ + if (!message_id || (!req_buff && req_buff_size) || !rm) + return -EINVAL; + + connection = gh_rm_alloc_connection(cpu_to_le32(message_id), RM_RPC_TYPE_REPLY); + if (IS_ERR(connection)) + return PTR_ERR(connection); + + init_completion(&connection->reply.seq_done); + + /* Allocate a new seq number for this connection */ + mutex_lock(&rm->call_idr_lock); + ret = idr_alloc_cyclic(&rm->call_idr, connection, 0, U16_MAX, + GFP_KERNEL); + mutex_unlock(&rm->call_idr_lock); + if (ret < 0) + goto out; + connection->reply.seq = ret; + + /* Send the request to the Resource Manager */ + ret = gh_rm_send_request(rm, message_id, req_buff, req_buff_size, connection); + if (ret < 0) + goto out; + + /* Wait for response */ + ret = wait_for_completion_interruptible(&connection->reply.seq_done); + if (ret) + goto out; + + /* Check for internal (kernel) error waiting for the response */ + if (connection->reply.ret) { + ret = connection->reply.ret; + if (ret != -ENOMEM) + kfree(connection->payload); + goto out; + } + + /* Got a response, did resource manager give us an error? */ + if (connection->reply.rm_error != GH_RM_ERROR_OK) { + pr_warn("RM rejected message %08x. Error: %d\n", message_id, + connection->reply.rm_error); + dump_stack(); + ret = gh_rm_remap_error(connection->reply.rm_error); + kfree(connection->payload); + goto out; + } + + /* Everything looks good, return the payload */ + *resp_buff_size = connection->size; + if (connection->size) + *resp_buf = connection->payload; + else { + /* kfree in case RM sent us multiple fragments but never any data in + * those fragments. We would've allocated memory for it, but connection->size == 0 + */ + kfree(connection->payload); + } + +out: + mutex_lock(&rm->call_idr_lock); + idr_remove(&rm->call_idr, connection->reply.seq); + mutex_unlock(&rm->call_idr_lock); + kfree(connection); + return ret; +} + + +int gh_rm_notifier_register(struct gh_rm *rm, struct notifier_block *nb) +{ + return blocking_notifier_chain_register(&rm->nh, nb); +} +EXPORT_SYMBOL_GPL(gh_rm_notifier_register); + +int gh_rm_notifier_unregister(struct gh_rm *rm, struct notifier_block *nb) +{ + return blocking_notifier_chain_unregister(&rm->nh, nb); +} +EXPORT_SYMBOL_GPL(gh_rm_notifier_unregister); + +void get_gh_rm(struct gh_rm *rm) +{ + get_device(rm->dev); +} +EXPORT_SYMBOL_GPL(get_gh_rm); + +void put_gh_rm(struct gh_rm *rm) +{ + put_device(rm->dev); +} +EXPORT_SYMBOL_GPL(put_gh_rm); + +static int gh_msgq_platform_probe_direction(struct platform_device *pdev, + bool tx, int idx, struct gunyah_resource *ghrsc) +{ + struct device_node *node = pdev->dev.of_node; + int ret; + + ghrsc->type = tx ? GUNYAH_RESOURCE_TYPE_MSGQ_TX : GUNYAH_RESOURCE_TYPE_MSGQ_RX; + + ghrsc->irq = platform_get_irq(pdev, idx); + if (ghrsc->irq < 0) { + dev_err(&pdev->dev, "Failed to get irq%d: %d\n", idx, ghrsc->irq); + return ghrsc->irq; + } + + ret = of_property_read_u64_index(node, "reg", idx, &ghrsc->capid); + if (ret) { + dev_err(&pdev->dev, "Failed to get capid%d: %d\n", idx, ret); + return ret; + } + + return 0; +} + +static int gh_rm_drv_probe(struct platform_device *pdev) +{ + struct gh_msgq_tx_data *msg; + struct gh_rm *rm; + int ret; + + rm = devm_kzalloc(&pdev->dev, sizeof(*rm), GFP_KERNEL); + if (!rm) + return -ENOMEM; + + platform_set_drvdata(pdev, rm); + rm->dev = &pdev->dev; + + mutex_init(&rm->call_idr_lock); + idr_init(&rm->call_idr); + rm->cache = kmem_cache_create("gh_rm", struct_size(msg, data, GH_MSGQ_MAX_MSG_SIZE), 0, + SLAB_HWCACHE_ALIGN, NULL); + if (!rm->cache) + return -ENOMEM; + mutex_init(&rm->send_lock); + BLOCKING_INIT_NOTIFIER_HEAD(&rm->nh); + + ret = gh_msgq_platform_probe_direction(pdev, true, 0, &rm->tx_ghrsc); + if (ret) + goto err_cache; + + ret = gh_msgq_platform_probe_direction(pdev, false, 1, &rm->rx_ghrsc); + if (ret) + goto err_cache; + + rm->msgq_client.dev = &pdev->dev; + rm->msgq_client.tx_block = true; + rm->msgq_client.rx_callback = gh_rm_msgq_rx_data; + rm->msgq_client.tx_done = gh_rm_msgq_tx_done; + + return gh_msgq_init(&pdev->dev, &rm->msgq, &rm->msgq_client, &rm->tx_ghrsc, &rm->rx_ghrsc); +err_cache: + kmem_cache_destroy(rm->cache); + return ret; +} + +static int gh_rm_drv_remove(struct platform_device *pdev) +{ + struct gh_rm *rm = platform_get_drvdata(pdev); + + mbox_free_channel(gh_msgq_chan(&rm->msgq)); + gh_msgq_remove(&rm->msgq); + kmem_cache_destroy(rm->cache); + + return 0; +} + +static const struct of_device_id gh_rm_of_match[] = { + { .compatible = "gunyah-resource-manager" }, + {} +}; +MODULE_DEVICE_TABLE(of, gh_rm_of_match); + +static struct platform_driver gh_rm_driver = { + .probe = gh_rm_drv_probe, + .remove = gh_rm_drv_remove, + .driver = { + .name = "gh_rsc_mgr", + .of_match_table = gh_rm_of_match, + }, +}; +module_platform_driver(gh_rm_driver); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("Gunyah Resource Manager Driver"); diff --git a/drivers/virt/gunyah/rsc_mgr.h b/drivers/virt/gunyah/rsc_mgr.h new file mode 100644 index 000000000000..d4e799a7526f --- /dev/null +++ b/drivers/virt/gunyah/rsc_mgr.h @@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ +#ifndef __GH_RSC_MGR_PRIV_H +#define __GH_RSC_MGR_PRIV_H + +#include +#include +#include + +/* RM Error codes */ +enum gh_rm_error { + GH_RM_ERROR_OK = 0x0, + GH_RM_ERROR_UNIMPLEMENTED = 0xFFFFFFFF, + GH_RM_ERROR_NOMEM = 0x1, + GH_RM_ERROR_NORESOURCE = 0x2, + GH_RM_ERROR_DENIED = 0x3, + GH_RM_ERROR_INVALID = 0x4, + GH_RM_ERROR_BUSY = 0x5, + GH_RM_ERROR_ARGUMENT_INVALID = 0x6, + GH_RM_ERROR_HANDLE_INVALID = 0x7, + GH_RM_ERROR_VALIDATE_FAILED = 0x8, + GH_RM_ERROR_MAP_FAILED = 0x9, + GH_RM_ERROR_MEM_INVALID = 0xA, + GH_RM_ERROR_MEM_INUSE = 0xB, + GH_RM_ERROR_MEM_RELEASED = 0xC, + GH_RM_ERROR_VMID_INVALID = 0xD, + GH_RM_ERROR_LOOKUP_FAILED = 0xE, + GH_RM_ERROR_IRQ_INVALID = 0xF, + GH_RM_ERROR_IRQ_INUSE = 0x10, + GH_RM_ERROR_IRQ_RELEASED = 0x11, +}; + +/** + * gh_rm_remap_error() - Remap Gunyah resource manager errors into a Linux error code + * @gh_error: "Standard" return value from Gunyah resource manager + */ +static inline int gh_rm_remap_error(enum gh_rm_error rm_error) +{ + switch (rm_error) { + case GH_RM_ERROR_OK: + return 0; + case GH_RM_ERROR_UNIMPLEMENTED: + return -EOPNOTSUPP; + case GH_RM_ERROR_NOMEM: + return -ENOMEM; + case GH_RM_ERROR_NORESOURCE: + return -ENODEV; + case GH_RM_ERROR_DENIED: + return -EPERM; + case GH_RM_ERROR_BUSY: + return -EBUSY; + case GH_RM_ERROR_INVALID: + case GH_RM_ERROR_ARGUMENT_INVALID: + case GH_RM_ERROR_HANDLE_INVALID: + case GH_RM_ERROR_VALIDATE_FAILED: + case GH_RM_ERROR_MAP_FAILED: + case GH_RM_ERROR_MEM_INVALID: + case GH_RM_ERROR_MEM_INUSE: + case GH_RM_ERROR_MEM_RELEASED: + case GH_RM_ERROR_VMID_INVALID: + case GH_RM_ERROR_LOOKUP_FAILED: + case GH_RM_ERROR_IRQ_INVALID: + case GH_RM_ERROR_IRQ_INUSE: + case GH_RM_ERROR_IRQ_RELEASED: + return -EINVAL; + default: + return -EBADMSG; + } +} + +struct gh_rm; +int gh_rm_call(struct gh_rm *rsc_mgr, u32 message_id, void *req_buff, size_t req_buff_size, + void **resp_buf, size_t *resp_buff_size); + +#endif diff --git a/include/linux/gunyah_rsc_mgr.h b/include/linux/gunyah_rsc_mgr.h new file mode 100644 index 000000000000..c992b3188c8d --- /dev/null +++ b/include/linux/gunyah_rsc_mgr.h @@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#ifndef _GUNYAH_RSC_MGR_H +#define _GUNYAH_RSC_MGR_H + +#include +#include +#include + +#define GH_VMID_INVAL U16_MAX + +/* Gunyah recognizes VMID0 as an alias to the current VM's ID */ +#define GH_VMID_SELF 0 + +struct gh_rm; +int gh_rm_notifier_register(struct gh_rm *rm, struct notifier_block *nb); +int gh_rm_notifier_unregister(struct gh_rm *rm, struct notifier_block *nb); +void get_gh_rm(struct gh_rm *rm); +void put_gh_rm(struct gh_rm *rm); + +#endif From patchwork Tue Feb 14 21:24:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 653523 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 878F9C64ED6 for ; Tue, 14 Feb 2023 21:25:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229765AbjBNVZA (ORCPT ); Tue, 14 Feb 2023 16:25:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47932 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232791AbjBNVYo (ORCPT ); Tue, 14 Feb 2023 16:24:44 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8DB7B303CE; Tue, 14 Feb 2023 13:24:31 -0800 (PST) Received: from pps.filterd (m0279872.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 31E30vQh006087; Tue, 14 Feb 2023 21:24:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=IUmfgykgdtJZSiIWp/vTVJ0uOP6w/38YV3TjDmCTlkk=; b=nwnnZb7/4jqjsYEqbFdltDD1wnKxKzCK2lGI2LQcy9ag76bT0sqZ+wkbyEwfzILMS5N9 6AojctZNZARAbDLi07MgzR+c4ed2duwzzeljnMcpgaVghsATX1ikfjLZuLroMJuRwC2l qKkO5Unw4JmFMNWdmpJbL3w6Ocf07RhAo940/PxfvRY9wSotaZ/KImpphBsibgBZXpr1 l/rKXNP+OVPWPuB/DOGkXsX3jDZuJzmlZsvAMQxl6ZGEqvc2U+dS/VhIEWfeDA1ZApSM jQbXnO2KPVEnJjAXMSSZYCRWU7oqBeee3Rc392VkXJanpHPBNzHAsTv16buk21J/u9AW cg== Received: from nasanppmta05.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3nr20vtkdt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:24:17 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA05.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 31ELOGK3003411 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:24:16 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Tue, 14 Feb 2023 13:24:15 -0800 From: Elliot Berman To: Alex Elder , Srinivas Kandagatla , Elliot Berman , Prakruthi Deepak Heragu CC: Murali Nalajala , Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Dmitry Baryshkov , Bjorn Andersson , "Konrad Dybcio" , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Jassi Brar , , , , , Subject: [PATCH v10 11/26] gunyah: rsc_mgr: Add RPC for sharing memory Date: Tue, 14 Feb 2023 13:24:04 -0800 Message-ID: <20230214212405.3314219-1-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230214211229.3239350-1-quic_eberman@quicinc.com> References: <20230214211229.3239350-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01a.na.qualcomm.com (10.47.209.196) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: m8CqqstFeM_GNcM5H7IV0kTFTZrewUEh X-Proofpoint-GUID: m8CqqstFeM_GNcM5H7IV0kTFTZrewUEh X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-14_15,2023-02-14_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 mlxlogscore=999 priorityscore=1501 malwarescore=0 clxscore=1015 bulkscore=0 impostorscore=0 spamscore=0 adultscore=0 phishscore=0 suspectscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2302140183 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Gunyah resource manager provides API to manipulate stage 2 page tables. Manipulations are represented as a memory parcel. Memory parcels describe a list of memory regions (intermediate physical address and size), a list of new permissions for VMs, and the memory type (DDR or MMIO). Memory parcels are uniquely identified by a handle allocated by Gunyah. There are a few types of memory parcel sharing which Gunyah supports: - Sharing: the guest and host VM both have access - Lending: only the guest has access; host VM loses access - Donating: Permanently lent (not reclaimed even if guest shuts down) Memory parcels that have been shared or lent can be reclaimed by the host via an additional call. The reclaim operation restores the original access the host VM had to the memory parcel and removes the access to other VM. One point to note that memory parcels don't describe where in the guest VM the memory parcel should reside. The guest VM must accept the memory parcel either explicitly via a "gh_rm_mem_accept" call (not introduced here) or be configured to accept it automatically at boot. As the guest VM accepts the memory parcel, it also mentions the IPA it wants to place memory parcel. Co-developed-by: Prakruthi Deepak Heragu Signed-off-by: Prakruthi Deepak Heragu Signed-off-by: Elliot Berman --- drivers/virt/gunyah/rsc_mgr.h | 44 +++++++ drivers/virt/gunyah/rsc_mgr_rpc.c | 185 ++++++++++++++++++++++++++++++ include/linux/gunyah_rsc_mgr.h | 47 ++++++++ 3 files changed, 276 insertions(+) diff --git a/drivers/virt/gunyah/rsc_mgr.h b/drivers/virt/gunyah/rsc_mgr.h index 7406237bc66d..9b23cefe02b0 100644 --- a/drivers/virt/gunyah/rsc_mgr.h +++ b/drivers/virt/gunyah/rsc_mgr.h @@ -74,6 +74,12 @@ struct gh_rm; int gh_rm_call(struct gh_rm *rsc_mgr, u32 message_id, void *req_buff, size_t req_buff_size, void **resp_buf, size_t *resp_buff_size); +/* Message IDs: Memory Management */ +#define GH_RM_RPC_MEM_LEND 0x51000012 +#define GH_RM_RPC_MEM_SHARE 0x51000013 +#define GH_RM_RPC_MEM_RECLAIM 0x51000015 +#define GH_RM_RPC_MEM_APPEND 0x51000018 + /* Message IDs: VM Management */ #define GH_RM_RPC_VM_ALLOC_VMID 0x56000001 #define GH_RM_RPC_VM_DEALLOC_VMID 0x56000002 @@ -90,6 +96,44 @@ struct gh_rm_vm_common_vmid_req { __le16 reserved0; } __packed; +/* Call: MEM_LEND, MEM_SHARE */ +struct gh_rm_mem_share_req_header { + u8 mem_type; + u8 reserved0; +#define GH_MEM_SHARE_REQ_FLAGS_APPEND BIT(1) + u8 flags; + u8 reserved1; + __le32 label; +} __packed; + +struct gh_rm_mem_share_req_acl_section { + __le32 n_entries; + struct gh_rm_mem_acl_entry entries[]; +}; + +struct gh_rm_mem_share_req_mem_section { + __le16 n_entries; + __le16 reserved0; + struct gh_rm_mem_entry entries[]; +}; + +/* Call: MEM_RELEASE */ +struct gh_rm_mem_release_req { + __le32 mem_handle; + u8 flags; /* currently not used */ + __le16 reserved0; + u8 reserved1; +} __packed; + +/* Call: MEM_APPEND */ +struct gh_rm_mem_append_req_header { + __le32 mem_handle; +#define GH_MEM_APPEND_REQ_FLAGS_END BIT(0) + u8 flags; + __le16 reserved0; + u8 reserved1; +} __packed; + /* Call: VM_ALLOC */ struct gh_rm_vm_alloc_vmid_resp { __le16 vmid; diff --git a/drivers/virt/gunyah/rsc_mgr_rpc.c b/drivers/virt/gunyah/rsc_mgr_rpc.c index 4515cdd80106..0c83b097fec9 100644 --- a/drivers/virt/gunyah/rsc_mgr_rpc.c +++ b/drivers/virt/gunyah/rsc_mgr_rpc.c @@ -7,6 +7,8 @@ #include "rsc_mgr.h" +#define GH_RM_MAX_MEM_ENTRIES 512 + /* * Several RM calls take only a VMID as a parameter and give only standard * response back. Deduplicate boilerplate code by using this common call. @@ -22,6 +24,189 @@ static int gh_rm_common_vmid_call(struct gh_rm *rm, u32 message_id, u16 vmid) return gh_rm_call(rm, message_id, &req_payload, sizeof(req_payload), &resp, &resp_size); } +static int _gh_rm_mem_append(struct gh_rm *rm, u32 mem_handle, bool end_append, + struct gh_rm_mem_entry *mem_entries, size_t n_mem_entries) +{ + struct gh_rm_mem_share_req_mem_section *mem_section; + struct gh_rm_mem_append_req_header *req_header; + size_t msg_size = 0, resp_size; + void *msg, *resp; + int ret; + + msg_size += sizeof(struct gh_rm_mem_append_req_header); + msg_size += struct_size(mem_section, entries, n_mem_entries); + + msg = kzalloc(msg_size, GFP_KERNEL); + if (!msg) + return -ENOMEM; + + req_header = msg; + mem_section = (void *)req_header + sizeof(struct gh_rm_mem_append_req_header); + + req_header->mem_handle = cpu_to_le32(mem_handle); + if (end_append) + req_header->flags |= GH_MEM_APPEND_REQ_FLAGS_END; + + mem_section->n_entries = cpu_to_le16(n_mem_entries); + memcpy(mem_section->entries, mem_entries, sizeof(*mem_entries) * n_mem_entries); + + ret = gh_rm_call(rm, GH_RM_RPC_MEM_APPEND, msg, msg_size, &resp, &resp_size); + kfree(msg); + + return ret; +} + +static int gh_rm_mem_append(struct gh_rm *rm, u32 mem_handle, + struct gh_rm_mem_entry *mem_entries, size_t n_mem_entries) +{ + bool end_append; + int ret = 0; + size_t n; + + while (n_mem_entries) { + if (n_mem_entries > GH_RM_MAX_MEM_ENTRIES) { + end_append = false; + n = GH_RM_MAX_MEM_ENTRIES; + } else { + end_append = true; + n = n_mem_entries; + } + + ret = _gh_rm_mem_append(rm, mem_handle, end_append, mem_entries, n); + if (ret) + break; + + mem_entries += n; + n_mem_entries -= n; + } + + return ret; +} + +static int gh_rm_mem_lend_common(struct gh_rm *rm, u32 message_id, struct gh_rm_mem_parcel *p) +{ + size_t msg_size = 0, initial_mem_entries = p->n_mem_entries, resp_size; + struct gh_rm_mem_share_req_acl_section *acl_section; + struct gh_rm_mem_share_req_mem_section *mem_section; + struct gh_rm_mem_share_req_header *req_header; + u32 *attr_section; + __le32 *resp; + void *msg; + int ret; + + if (!p->acl_entries || !p->n_acl_entries || !p->mem_entries || !p->n_mem_entries || + p->n_acl_entries > U8_MAX || p->mem_handle != GH_MEM_HANDLE_INVAL) + return -EINVAL; + + if (initial_mem_entries > GH_RM_MAX_MEM_ENTRIES) + initial_mem_entries = GH_RM_MAX_MEM_ENTRIES; + + /* The format of the message goes: + * request header + * ACL entries (which VMs get what kind of access to this memory parcel) + * Memory entries (list of memory regions to share) + * Memory attributes (currently unused, we'll hard-code the size to 0) + */ + msg_size += sizeof(struct gh_rm_mem_share_req_header); + msg_size += struct_size(acl_section, entries, p->n_acl_entries); + msg_size += struct_size(mem_section, entries, initial_mem_entries); + msg_size += sizeof(u32); /* for memory attributes, currently unused */ + + msg = kzalloc(msg_size, GFP_KERNEL); + if (!msg) + return -ENOMEM; + + req_header = msg; + acl_section = (void *)req_header + sizeof(*req_header); + mem_section = (void *)acl_section + struct_size(acl_section, entries, p->n_acl_entries); + attr_section = (void *)mem_section + struct_size(mem_section, entries, initial_mem_entries); + + req_header->mem_type = p->mem_type; + if (initial_mem_entries != p->n_mem_entries) + req_header->flags |= GH_MEM_SHARE_REQ_FLAGS_APPEND; + req_header->label = cpu_to_le32(p->label); + + acl_section->n_entries = cpu_to_le32(p->n_acl_entries); + memcpy(acl_section->entries, p->acl_entries, sizeof(*(p->acl_entries)) * p->n_acl_entries); + + mem_section->n_entries = cpu_to_le16(initial_mem_entries); + memcpy(mem_section->entries, p->mem_entries, + sizeof(*(p->mem_entries)) * initial_mem_entries); + + /* Set n_entries for memory attribute section to 0 */ + *attr_section = 0; + + ret = gh_rm_call(rm, message_id, msg, msg_size, (void **)&resp, &resp_size); + kfree(msg); + + if (ret) + return ret; + + p->mem_handle = le32_to_cpu(*resp); + + if (initial_mem_entries != p->n_mem_entries) { + ret = gh_rm_mem_append(rm, p->mem_handle, + &p->mem_entries[initial_mem_entries], + p->n_mem_entries - initial_mem_entries); + if (ret) { + gh_rm_mem_reclaim(rm, p); + p->mem_handle = GH_MEM_HANDLE_INVAL; + } + } + + kfree(resp); + return ret; +} + +/** + * gh_rm_mem_lend() - Lend memory to other virtual machines. + * @rm: Handle to a Gunyah resource manager + * @parcel: Package the memory information of the memory to be lent. + * + * Lending removes Linux's access to the memory while the memory parcel is lent. + */ +int gh_rm_mem_lend(struct gh_rm *rm, struct gh_rm_mem_parcel *parcel) +{ + return gh_rm_mem_lend_common(rm, GH_RM_RPC_MEM_LEND, parcel); +} + + +/** + * gh_rm_mem_share() - Share memory with other virtual machines. + * @rm: Handle to a Gunyah resource manager + * @parcel: Package the memory information of the memory to be shared. + * + * Sharing keeps Linux's access to the memory while the memory parcel is shared. + */ +int gh_rm_mem_share(struct gh_rm *rm, struct gh_rm_mem_parcel *parcel) +{ + return gh_rm_mem_lend_common(rm, GH_RM_RPC_MEM_SHARE, parcel); +} + +/** + * gh_rm_mem_reclaim() - Reclaim a memory parcel + * @rm: Handle to a Gunyah resource manager + * @parcel: Package the memory information of the memory to be reclaimed. + * + * RM maps the associated memory back into the stage-2 page tables of the owner VM. + */ +int gh_rm_mem_reclaim(struct gh_rm *rm, struct gh_rm_mem_parcel *parcel) +{ + struct gh_rm_mem_release_req req = { + .mem_handle = cpu_to_le32(parcel->mem_handle), + }; + size_t resp_size; + void *resp; + int ret; + + ret = gh_rm_call(rm, GH_RM_RPC_MEM_RECLAIM, &req, sizeof(req), &resp, &resp_size); + /* Do not call platform mem reclaim hooks: the reclaim didn't happen*/ + if (ret) + return ret; + + return ret; +} + /** * gh_rm_alloc_vmid() - Allocate a new VM in Gunyah. Returns the VM identifier. * @rm: Handle to a Gunyah resource manager diff --git a/include/linux/gunyah_rsc_mgr.h b/include/linux/gunyah_rsc_mgr.h index e7bd29f8be6e..2d8b8b6cc394 100644 --- a/include/linux/gunyah_rsc_mgr.h +++ b/include/linux/gunyah_rsc_mgr.h @@ -11,6 +11,7 @@ #include #define GH_VMID_INVAL U16_MAX +#define GH_MEM_HANDLE_INVAL U32_MAX /* Gunyah recognizes VMID0 as an alias to the current VM's ID */ #define GH_VMID_SELF 0 @@ -54,7 +55,53 @@ struct gh_rm_vm_status_payload { #define GH_RM_NOTIFICATION_VM_STATUS 0x56100008 +struct gh_rm_mem_acl_entry { + __le16 vmid; +#define GH_RM_ACL_X BIT(0) +#define GH_RM_ACL_W BIT(1) +#define GH_RM_ACL_R BIT(2) + u8 perms; + u8 reserved; +} __packed; + +struct gh_rm_mem_entry { + __le64 ipa_base; + __le64 size; +} __packed; + +enum gh_rm_mem_type { + GH_RM_MEM_TYPE_NORMAL = 0, + GH_RM_MEM_TYPE_IO = 1, +}; + +/* + * struct gh_rm_mem_parcel - Package info about memory to be lent/shared/donated/reclaimed + * @mem_type: The type of memory: normal (DDR) or IO + * @label: An client-specified identifier which can be used by the other VMs to identify the purpose + * of the memory parcel. + * @acl_entries: An array of access control entries. Each entry specifies a VM and what access + * is allowed for the memory parcel. + * @n_acl_entries: Count of the number of entries in the `acl_entries` array. + * @mem_entries: An list of regions to be associated with the memory parcel. Addresses should be + * (intermediate) physical addresses from Linux's perspective. + * @n_mem_entries: Count of the number of entries in the `mem_entries` array. + * @mem_handle: On success, filled with memory handle that RM allocates for this memory parcel + */ +struct gh_rm_mem_parcel { + enum gh_rm_mem_type mem_type; + u32 label; + size_t n_acl_entries; + struct gh_rm_mem_acl_entry *acl_entries; + size_t n_mem_entries; + struct gh_rm_mem_entry *mem_entries; + u32 mem_handle; +}; + /* RPC Calls */ +int gh_rm_mem_lend(struct gh_rm *rm, struct gh_rm_mem_parcel *parcel); +int gh_rm_mem_share(struct gh_rm *rm, struct gh_rm_mem_parcel *parcel); +int gh_rm_mem_reclaim(struct gh_rm *rm, struct gh_rm_mem_parcel *parcel); + int gh_rm_alloc_vmid(struct gh_rm *rm, u16 vmid); int gh_rm_dealloc_vmid(struct gh_rm *rm, u16 vmid); int gh_rm_vm_reset(struct gh_rm *rm, u16 vmid); From patchwork Tue Feb 14 21:24:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 653522 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22D72C6FA9D for ; Tue, 14 Feb 2023 21:25:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232911AbjBNVZD (ORCPT ); Tue, 14 Feb 2023 16:25:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233010AbjBNVY7 (ORCPT ); Tue, 14 Feb 2023 16:24:59 -0500 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D493D301A5; Tue, 14 Feb 2023 13:24:40 -0800 (PST) Received: from pps.filterd (m0279866.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 31E82qBe003162; Tue, 14 Feb 2023 21:24:27 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=INcu8f2yaL4UHM5SoCgaYhlSEsIhI1prYjJV/hf8ZKc=; b=USosHtmighiZos/B4fqxBXD8+1qaKsLEpWxFLGMuU1bonkNRo7M1+FpdwGtt5+VL5py5 vBuwCQoR5yj4zknHmtdHKhGQWyweEIJ2DYFkzJlSgmWXKrDQh4oGasd8ojEhiLsidY36 sxDEGTqYj7EY99FIyzKrCRrlYaHeGDrVyNgsRMmLUg1lrfPVnMlCJf8VSfYgD/TNGcAT d9VnDf2lHhEl5XxXD4RVr9Gfma3uy7OKro2E8wwWABYrNaknUK2ddaCB6PTyQRcxLRi+ vO0mNJEm4WpWKmr66XpV7xaV6Sp8i4O/PhADlH1qLwWhUOGxPYNtk+BYczZdMB0O7OCe Sg== Received: from nasanppmta03.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3nr661a1q1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:24:27 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA03.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 31ELOQRF003532 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:24:26 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Tue, 14 Feb 2023 13:24:25 -0800 From: Elliot Berman To: Alex Elder , Srinivas Kandagatla , Elliot Berman , Prakruthi Deepak Heragu CC: Murali Nalajala , Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Dmitry Baryshkov , Bjorn Andersson , "Konrad Dybcio" , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Jassi Brar , , , , , Subject: [PATCH v10 12/26] gunyah: vm_mgr: Add/remove user memory regions Date: Tue, 14 Feb 2023 13:24:16 -0800 Message-ID: <20230214212417.3315422-1-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230214211229.3239350-1-quic_eberman@quicinc.com> References: <20230214211229.3239350-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01a.na.qualcomm.com (10.47.209.196) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: M3xnN_hAHgVPd3kXu-9fmJ6T5o_f977i X-Proofpoint-GUID: M3xnN_hAHgVPd3kXu-9fmJ6T5o_f977i X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-14_15,2023-02-14_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 mlxlogscore=999 mlxscore=0 spamscore=0 impostorscore=0 lowpriorityscore=0 malwarescore=0 priorityscore=1501 clxscore=1015 adultscore=0 bulkscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2302140184 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org When launching a virtual machine, Gunyah userspace allocates memory for the guest and informs Gunyah about these memory regions through SET_USER_MEMORY_REGION ioctl. Co-developed-by: Prakruthi Deepak Heragu Signed-off-by: Prakruthi Deepak Heragu Signed-off-by: Elliot Berman --- drivers/virt/gunyah/Makefile | 2 +- drivers/virt/gunyah/vm_mgr.c | 44 ++++++ drivers/virt/gunyah/vm_mgr.h | 25 ++++ drivers/virt/gunyah/vm_mgr_mm.c | 235 ++++++++++++++++++++++++++++++++ include/uapi/linux/gunyah.h | 33 +++++ 5 files changed, 338 insertions(+), 1 deletion(-) create mode 100644 drivers/virt/gunyah/vm_mgr_mm.c diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile index 03951cf82023..ff8bc4925392 100644 --- a/drivers/virt/gunyah/Makefile +++ b/drivers/virt/gunyah/Makefile @@ -2,5 +2,5 @@ obj-$(CONFIG_GUNYAH) += gunyah.o -gunyah_rsc_mgr-y += rsc_mgr.o rsc_mgr_rpc.o vm_mgr.o +gunyah_rsc_mgr-y += rsc_mgr.o rsc_mgr_rpc.o vm_mgr.o vm_mgr_mm.o obj-$(CONFIG_GUNYAH) += gunyah_rsc_mgr.o diff --git a/drivers/virt/gunyah/vm_mgr.c b/drivers/virt/gunyah/vm_mgr.c index fd890a57172e..84102bac03cc 100644 --- a/drivers/virt/gunyah/vm_mgr.c +++ b/drivers/virt/gunyah/vm_mgr.c @@ -18,8 +18,16 @@ static void gh_vm_free(struct work_struct *work) { struct gh_vm *ghvm = container_of(work, struct gh_vm, free_work); + struct gh_vm_mem *mapping, *tmp; int ret; + mutex_lock(&ghvm->mm_lock); + list_for_each_entry_safe(mapping, tmp, &ghvm->memory_mappings, list) { + gh_vm_mem_reclaim(ghvm, mapping); + kfree(mapping); + } + mutex_unlock(&ghvm->mm_lock); + ret = gh_rm_dealloc_vmid(ghvm->rm, ghvm->vmid); if (ret) pr_warn("Failed to deallocate vmid: %d\n", ret); @@ -48,11 +56,46 @@ static __must_check struct gh_vm *gh_vm_alloc(struct gh_rm *rm) ghvm->vmid = vmid; ghvm->rm = rm; + mutex_init(&ghvm->mm_lock); + INIT_LIST_HEAD(&ghvm->memory_mappings); INIT_WORK(&ghvm->free_work, gh_vm_free); return ghvm; } +static long gh_vm_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) +{ + struct gh_vm *ghvm = filp->private_data; + void __user *argp = (void __user *)arg; + long r; + + switch (cmd) { + case GH_VM_SET_USER_MEM_REGION: { + struct gh_userspace_memory_region region; + + if (copy_from_user(®ion, argp, sizeof(region))) + return -EFAULT; + + /* All other flag bits are reserved for future use */ + if (region.flags & ~(GH_MEM_ALLOW_READ | GH_MEM_ALLOW_WRITE | GH_MEM_ALLOW_EXEC | + GH_MEM_LENT)) + return -EINVAL; + + + if (region.memory_size) + r = gh_vm_mem_alloc(ghvm, ®ion); + else + r = gh_vm_mem_free(ghvm, region.label); + break; + } + default: + r = -ENOTTY; + break; + } + + return r; +} + static int gh_vm_release(struct inode *inode, struct file *filp) { struct gh_vm *ghvm = filp->private_data; @@ -65,6 +108,7 @@ static int gh_vm_release(struct inode *inode, struct file *filp) } static const struct file_operations gh_vm_fops = { + .unlocked_ioctl = gh_vm_ioctl, .release = gh_vm_release, .compat_ioctl = compat_ptr_ioctl, .llseek = noop_llseek, diff --git a/drivers/virt/gunyah/vm_mgr.h b/drivers/virt/gunyah/vm_mgr.h index 76954da706e9..97bc00c34878 100644 --- a/drivers/virt/gunyah/vm_mgr.h +++ b/drivers/virt/gunyah/vm_mgr.h @@ -7,16 +7,41 @@ #define _GH_PRIV_VM_MGR_H #include +#include +#include +#include #include long gh_dev_vm_mgr_ioctl(struct gh_rm *rm, unsigned int cmd, unsigned long arg); +enum gh_vm_mem_share_type { + VM_MEM_SHARE, + VM_MEM_LEND, +}; + +struct gh_vm_mem { + struct list_head list; + enum gh_vm_mem_share_type share_type; + struct gh_rm_mem_parcel parcel; + + __u64 guest_phys_addr; + struct page **pages; + unsigned long npages; +}; + struct gh_vm { u16 vmid; struct gh_rm *rm; struct work_struct free_work; + struct mutex mm_lock; + struct list_head memory_mappings; }; +int gh_vm_mem_alloc(struct gh_vm *ghvm, struct gh_userspace_memory_region *region); +void gh_vm_mem_reclaim(struct gh_vm *ghvm, struct gh_vm_mem *mapping); +int gh_vm_mem_free(struct gh_vm *ghvm, u32 label); +struct gh_vm_mem *gh_vm_mem_find(struct gh_vm *ghvm, u32 label); + #endif diff --git a/drivers/virt/gunyah/vm_mgr_mm.c b/drivers/virt/gunyah/vm_mgr_mm.c new file mode 100644 index 000000000000..03e71a36ea3b --- /dev/null +++ b/drivers/virt/gunyah/vm_mgr_mm.c @@ -0,0 +1,235 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#define pr_fmt(fmt) "gh_vm_mgr: " fmt + +#include +#include + +#include + +#include "vm_mgr.h" + +static inline bool page_contiguous(phys_addr_t p, phys_addr_t t) +{ + return t - p == PAGE_SIZE; +} + +static struct gh_vm_mem *__gh_vm_mem_find(struct gh_vm *ghvm, u32 label) + __must_hold(&ghvm->mm_lock) +{ + struct gh_vm_mem *mapping; + + list_for_each_entry(mapping, &ghvm->memory_mappings, list) + if (mapping->parcel.label == label) + return mapping; + + return NULL; +} + +void gh_vm_mem_reclaim(struct gh_vm *ghvm, struct gh_vm_mem *mapping) + __must_hold(&ghvm->mm_lock) +{ + int i, ret = 0; + + if (mapping->parcel.mem_handle != GH_MEM_HANDLE_INVAL) { + ret = gh_rm_mem_reclaim(ghvm->rm, &mapping->parcel); + if (ret) + pr_warn("Failed to reclaim memory parcel for label %d: %d\n", + mapping->parcel.label, ret); + } + + if (!ret) + for (i = 0; i < mapping->npages; i++) + unpin_user_page(mapping->pages[i]); + + kfree(mapping->pages); + kfree(mapping->parcel.acl_entries); + kfree(mapping->parcel.mem_entries); + + list_del(&mapping->list); +} + +struct gh_vm_mem *gh_vm_mem_find(struct gh_vm *ghvm, u32 label) +{ + struct gh_vm_mem *mapping; + int ret; + + ret = mutex_lock_interruptible(&ghvm->mm_lock); + if (ret) + return ERR_PTR(ret); + mapping = __gh_vm_mem_find(ghvm, label); + mutex_unlock(&ghvm->mm_lock); + return mapping ? : ERR_PTR(-ENODEV); +} + +int gh_vm_mem_alloc(struct gh_vm *ghvm, struct gh_userspace_memory_region *region) +{ + struct gh_vm_mem *mapping, *tmp_mapping; + struct gh_rm_mem_entry *mem_entries; + phys_addr_t curr_page, prev_page; + struct gh_rm_mem_parcel *parcel; + int i, j, pinned, ret = 0; + size_t entry_size; + u16 vmid; + + if (!gh_api_has_feature(GH_API_FEATURE_MEMEXTENT)) + return -EOPNOTSUPP; + + if (!region->memory_size || !PAGE_ALIGNED(region->memory_size) || + !PAGE_ALIGNED(region->userspace_addr) || !PAGE_ALIGNED(region->guest_phys_addr)) + return -EINVAL; + + ret = mutex_lock_interruptible(&ghvm->mm_lock); + if (ret) + return ret; + mapping = __gh_vm_mem_find(ghvm, region->label); + if (mapping) { + mutex_unlock(&ghvm->mm_lock); + return -EEXIST; + } + + mapping = kzalloc(sizeof(*mapping), GFP_KERNEL); + if (!mapping) { + ret = -ENOMEM; + goto free_mapping; + } + + mapping->parcel.label = region->label; + mapping->guest_phys_addr = region->guest_phys_addr; + mapping->npages = region->memory_size >> PAGE_SHIFT; + parcel = &mapping->parcel; + parcel->mem_handle = GH_MEM_HANDLE_INVAL; /* to be filled later by mem_share/mem_lend */ + parcel->mem_type = GH_RM_MEM_TYPE_NORMAL; + + /* Check for overlap */ + list_for_each_entry(tmp_mapping, &ghvm->memory_mappings, list) { + if (!((mapping->guest_phys_addr + (mapping->npages << PAGE_SHIFT) <= + tmp_mapping->guest_phys_addr) || + (mapping->guest_phys_addr >= + tmp_mapping->guest_phys_addr + (tmp_mapping->npages << PAGE_SHIFT)))) { + ret = -EEXIST; + goto free_mapping; + } + } + + list_add(&mapping->list, &ghvm->memory_mappings); + + mapping->pages = kcalloc(mapping->npages, sizeof(*mapping->pages), GFP_KERNEL); + if (!mapping->pages) { + ret = -ENOMEM; + mapping->npages = 0; /* update npages for reclaim */ + goto reclaim; + } + + pinned = pin_user_pages_fast(region->userspace_addr, mapping->npages, + FOLL_WRITE | FOLL_LONGTERM, mapping->pages); + if (pinned < 0) { + ret = pinned; + mapping->npages = 0; /* update npages for reclaim */ + goto reclaim; + } else if (pinned != mapping->npages) { + ret = -EFAULT; + mapping->npages = pinned; /* update npages for reclaim */ + goto reclaim; + } + + if (region->flags & GH_MEM_LENT) { + parcel->n_acl_entries = 1; + mapping->share_type = VM_MEM_LEND; + } else { + parcel->n_acl_entries = 2; + mapping->share_type = VM_MEM_SHARE; + } + parcel->acl_entries = kcalloc(parcel->n_acl_entries, sizeof(*parcel->acl_entries), + GFP_KERNEL); + if (!parcel->acl_entries) { + ret = -ENOMEM; + goto reclaim; + } + + parcel->acl_entries[0].vmid = cpu_to_le16(ghvm->vmid); + if (region->flags & GH_MEM_ALLOW_READ) + parcel->acl_entries[0].perms |= GH_RM_ACL_R; + if (region->flags & GH_MEM_ALLOW_WRITE) + parcel->acl_entries[0].perms |= GH_RM_ACL_W; + if (region->flags & GH_MEM_ALLOW_EXEC) + parcel->acl_entries[0].perms |= GH_RM_ACL_X; + + if (mapping->share_type == VM_MEM_SHARE) { + ret = gh_rm_get_vmid(ghvm->rm, &vmid); + if (ret) + goto reclaim; + + parcel->acl_entries[1].vmid = cpu_to_le16(vmid); + /* Host assumed to have all these permissions. Gunyah will not + * grant new permissions if host actually had less than RWX + */ + parcel->acl_entries[1].perms |= GH_RM_ACL_R | GH_RM_ACL_W | GH_RM_ACL_X; + } + + mem_entries = kcalloc(mapping->npages, sizeof(*mem_entries), GFP_KERNEL); + if (!mem_entries) { + ret = -ENOMEM; + goto reclaim; + } + + /* reduce number of entries by combining contiguous pages into single memory entry */ + prev_page = page_to_phys(mapping->pages[0]); + mem_entries[0].ipa_base = cpu_to_le64(prev_page); + entry_size = PAGE_SIZE; + for (i = 1, j = 0; i < mapping->npages; i++) { + curr_page = page_to_phys(mapping->pages[i]); + if (page_contiguous(prev_page, curr_page)) { + entry_size += PAGE_SIZE; + } else { + mem_entries[j].size = cpu_to_le64(entry_size); + j++; + mem_entries[j].ipa_base = cpu_to_le64(curr_page); + entry_size = PAGE_SIZE; + } + + prev_page = curr_page; + } + mem_entries[j].size = cpu_to_le64(entry_size); + + parcel->n_mem_entries = j + 1; + parcel->mem_entries = kmemdup(mem_entries, sizeof(*mem_entries) * parcel->n_mem_entries, + GFP_KERNEL); + kfree(mem_entries); + if (!parcel->mem_entries) { + ret = -ENOMEM; + goto reclaim; + } + + mutex_unlock(&ghvm->mm_lock); + return 0; +reclaim: + gh_vm_mem_reclaim(ghvm, mapping); +free_mapping: + kfree(mapping); + mutex_unlock(&ghvm->mm_lock); + return ret; +} + +int gh_vm_mem_free(struct gh_vm *ghvm, u32 label) +{ + struct gh_vm_mem *mapping; + int ret; + + ret = mutex_lock_interruptible(&ghvm->mm_lock); + if (ret) + return ret; + + mapping = __gh_vm_mem_find(ghvm, label); + if (!mapping) + goto out; + + gh_vm_mem_reclaim(ghvm, mapping); + kfree(mapping); +out: + mutex_unlock(&ghvm->mm_lock); + return ret; +} diff --git a/include/uapi/linux/gunyah.h b/include/uapi/linux/gunyah.h index 10ba32d2b0a6..d85d12119a48 100644 --- a/include/uapi/linux/gunyah.h +++ b/include/uapi/linux/gunyah.h @@ -20,4 +20,37 @@ */ #define GH_CREATE_VM _IO(GH_IOCTL_TYPE, 0x0) /* Returns a Gunyah VM fd */ +/* + * ioctls for VM fds + */ + +/** + * struct gh_userspace_memory_region - Userspace memory descripion for GH_VM_SET_USER_MEM_REGION + * @label: Unique identifer to the region. + * @flags: Flags for memory parcel behavior + * @guest_phys_addr: Location of the memory region in guest's memory space (page-aligned) + * @memory_size: Size of the region (page-aligned) + * @userspace_addr: Location of the memory region in caller (userspace)'s memory + * + * See Documentation/virt/gunyah/vm-manager.rst for further details. + */ +struct gh_userspace_memory_region { + __u32 label; +#define GH_MEM_ALLOW_READ (1UL << 0) +#define GH_MEM_ALLOW_WRITE (1UL << 1) +#define GH_MEM_ALLOW_EXEC (1UL << 2) +/* + * The guest will be lent the memory instead of shared. + * In other words, the guest has exclusive access to the memory region and the host loses access. + */ +#define GH_MEM_LENT (1UL << 3) + __u32 flags; + __u64 guest_phys_addr; + __u64 memory_size; + __u64 userspace_addr; +}; + +#define GH_VM_SET_USER_MEM_REGION _IOW(GH_IOCTL_TYPE, 0x1, \ + struct gh_userspace_memory_region) + #endif From patchwork Tue Feb 14 21:24:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 653521 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E2AEC05027 for ; Tue, 14 Feb 2023 21:25:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232941AbjBNVZ2 (ORCPT ); Tue, 14 Feb 2023 16:25:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48968 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232927AbjBNVZW (ORCPT ); Tue, 14 Feb 2023 16:25:22 -0500 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 52F84302BE; Tue, 14 Feb 2023 13:25:00 -0800 (PST) Received: from pps.filterd (m0279863.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 31EJa0i3011449; Tue, 14 Feb 2023 21:24:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=Q6yBeStn/9rsfyrBVOoKcRhcNVkTcIVsM7yNq5Gf5/E=; b=iuhFp5GBNRQUjXsEDjuW0pJqXIhT/HFnvGpGpXqAIQDXd1FqcO1BiAbrZU9LYjHRbc4X zfsAoyr0ANVbFDWszirW2TnUju+mfLXLLvgpqh9O/SMZ4kAWuWex8dC8Y92W6aGiReNg /t9FYUKGFQJ/mg7LyBgpsGj4wJ3dhPiu0ZNkJWH7ckOiDVZ5Ed+31PhPDgH1VoPPUTCY YYNNaN+kL/KOMoA6KZbLrDAkpis5RNyleSiKTbutPhkTdtCufO825ounfR9Pds4jvhQ3 yMiiTcm86xY9N5xIK7AKIoBvppxQOcM/9Yizdo2cKVa4rDT4s18o1Spd4pLWsvCmoajZ RQ== Received: from nasanppmta04.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3nr6ps1y26-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:24:48 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA04.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 31ELOliS032498 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:24:47 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Tue, 14 Feb 2023 13:24:47 -0800 From: Elliot Berman To: Alex Elder , Srinivas Kandagatla , Elliot Berman , Prakruthi Deepak Heragu CC: Murali Nalajala , Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Dmitry Baryshkov , Bjorn Andersson , "Konrad Dybcio" , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Jassi Brar , , , , , Subject: [PATCH v10 14/26] samples: Add sample userspace Gunyah VM Manager Date: Tue, 14 Feb 2023 13:24:37 -0800 Message-ID: <20230214212438.3317649-1-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230214211229.3239350-1-quic_eberman@quicinc.com> References: <20230214211229.3239350-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01a.na.qualcomm.com (10.47.209.196) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: 8OTi16qsrC-BVeV3bjFl-6L194yNovTR X-Proofpoint-GUID: 8OTi16qsrC-BVeV3bjFl-6L194yNovTR X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-14_15,2023-02-14_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 clxscore=1015 impostorscore=0 adultscore=0 mlxlogscore=999 malwarescore=0 priorityscore=1501 bulkscore=0 suspectscore=0 lowpriorityscore=0 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2302140184 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Add a sample Gunyah VMM capable of launching a non-proxy scheduled VM. Signed-off-by: Elliot Berman --- samples/Kconfig | 10 ++ samples/Makefile | 1 + samples/gunyah/.gitignore | 2 + samples/gunyah/Makefile | 6 + samples/gunyah/gunyah_vmm.c | 270 +++++++++++++++++++++++++++++++++++ samples/gunyah/sample_vm.dts | 68 +++++++++ 6 files changed, 357 insertions(+) create mode 100644 samples/gunyah/.gitignore create mode 100644 samples/gunyah/Makefile create mode 100644 samples/gunyah/gunyah_vmm.c create mode 100644 samples/gunyah/sample_vm.dts diff --git a/samples/Kconfig b/samples/Kconfig index 30ef8bd48ba3..11070bf02bd7 100644 --- a/samples/Kconfig +++ b/samples/Kconfig @@ -273,6 +273,16 @@ config SAMPLE_CORESIGHT_SYSCFG This demonstrates how a user may create their own CoreSight configurations and easily load them into the system at runtime. +config SAMPLE_GUNYAH + bool "Build example Gunyah Virtual Machine Manager" + depends on CC_CAN_LINK && HEADERS_INSTALL + depends on GUNYAH + help + Build an example Gunyah VMM userspace program capable of launching + a basic virtual machine under the Gunyah hypervisor. + This demonstrates how to create a virtual machine under the Gunyah + hypervisor. + source "samples/rust/Kconfig" endif # SAMPLES diff --git a/samples/Makefile b/samples/Makefile index 7cb632ef88ee..a65555802642 100644 --- a/samples/Makefile +++ b/samples/Makefile @@ -37,3 +37,4 @@ obj-$(CONFIG_DEBUG_KMEMLEAK_TEST) += kmemleak/ obj-$(CONFIG_SAMPLE_CORESIGHT_SYSCFG) += coresight/ obj-$(CONFIG_SAMPLE_FPROBE) += fprobe/ obj-$(CONFIG_SAMPLES_RUST) += rust/ +obj-$(CONFIG_SAMPLE_GUNYAH) += gunyah/ diff --git a/samples/gunyah/.gitignore b/samples/gunyah/.gitignore new file mode 100644 index 000000000000..adc7d1589fde --- /dev/null +++ b/samples/gunyah/.gitignore @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0 +/gunyah_vmm diff --git a/samples/gunyah/Makefile b/samples/gunyah/Makefile new file mode 100644 index 000000000000..faf14f9bb337 --- /dev/null +++ b/samples/gunyah/Makefile @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: GPL-2.0-only + +userprogs-always-y += gunyah_vmm +dtb-y += sample_vm.dtb + +userccflags += -I usr/include diff --git a/samples/gunyah/gunyah_vmm.c b/samples/gunyah/gunyah_vmm.c new file mode 100644 index 000000000000..a72fab7ddc3a --- /dev/null +++ b/samples/gunyah/gunyah_vmm.c @@ -0,0 +1,270 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define __USE_GNU +#include + +#include + +struct vm_config { + int image_fd; + int dtb_fd; + int ramdisk_fd; + + uint64_t guest_base; + uint64_t guest_size; + + uint64_t image_offset; + off_t image_size; + uint64_t dtb_offset; + off_t dtb_size; + uint64_t ramdisk_offset; + off_t ramdisk_size; +}; + +static struct option options[] = { + { "help", no_argument, NULL, 'h' }, + { "image", required_argument, NULL, 'i' }, + { "dtb", required_argument, NULL, 'd' }, + { "ramdisk", optional_argument, NULL, 'r' }, + { "base", optional_argument, NULL, 'B' }, + { "size", optional_argument, NULL, 'S' }, + { "image_offset", optional_argument, NULL, 'I' }, + { "dtb_offset", optional_argument, NULL, 'D' }, + { "ramdisk_offset", optional_argument, NULL, 'R' }, + { } +}; + +static void print_help(char *cmd) +{ + printf("gunyah_vmm, a sample tool to launch Gunyah VMs\n" + "Usage: %s \n" + " --help, -h this menu\n" + " --image, -i VM image file to load (e.g. a kernel Image) [Required]\n" + " --dtb, -d Devicetree to load [Required]\n" + " --ramdisk, -r Ramdisk to load\n" + " --base, -B
Set the base address of guest's memory [Default: 0x80000000]\n" + " --size, -S The number of bytes large to make the guest's memory [Default: 0x6400000 (100 MB)]\n" + " --image_offset, -I Offset into guest memory to load the VM image file [Default: 0x10000]\n" + " --dtb_offset, -D Offset into guest memory to load the DTB [Default: 0]\n" + " --ramdisk_offset, -R Offset into guest memory to load a ramdisk [Default: 0x4600000]\n" + , cmd); +} + +int main(int argc, char **argv) +{ + int gunyah_fd, vm_fd, guest_fd; + struct gh_userspace_memory_region guest_mem_desc = { 0 }; + struct gh_vm_dtb_config dtb_config = { 0 }; + char *guest_mem; + struct vm_config config = { + /* Defaults good enough to boot static kernel and a basic ramdisk */ + .ramdisk_fd = -1, + .guest_base = 0x80000000, + .guest_size = 0x6400000, /* 100 MB */ + .image_offset = 0, + .dtb_offset = 0x45f0000, + .ramdisk_offset = 0x4600000, /* put at +70MB (30MB for ramdisk) */ + }; + struct stat st; + int opt, optidx, ret = 0; + long l; + + while ((opt = getopt_long(argc, argv, "hi:d:r:B:S:I:D:R:c:", options, &optidx)) != -1) { + switch (opt) { + case 'i': + config.image_fd = open(optarg, O_RDONLY | O_CLOEXEC); + if (config.image_fd < 0) { + perror("Failed to open image"); + return -1; + } + if (stat(optarg, &st) < 0) { + perror("Failed to stat image"); + return -1; + } + config.image_size = st.st_size; + break; + case 'd': + config.dtb_fd = open(optarg, O_RDONLY | O_CLOEXEC); + if (config.dtb_fd < 0) { + perror("Failed to open dtb"); + return -1; + } + if (stat(optarg, &st) < 0) { + perror("Failed to stat dtb"); + return -1; + } + config.dtb_size = st.st_size; + break; + case 'r': + config.ramdisk_fd = open(optarg, O_RDONLY | O_CLOEXEC); + if (config.ramdisk_fd < 0) { + perror("Failed to open ramdisk"); + return -1; + } + if (stat(optarg, &st) < 0) { + perror("Failed to stat ramdisk"); + return -1; + } + config.ramdisk_size = st.st_size; + break; + case 'B': + l = strtol(optarg, NULL, 0); + if (l == LONG_MIN) { + perror("Failed to parse base address"); + return -1; + } + config.guest_base = l; + break; + case 'S': + l = strtol(optarg, NULL, 0); + if (l == LONG_MIN) { + perror("Failed to parse memory size"); + return -1; + } + config.guest_size = l; + break; + case 'I': + l = strtol(optarg, NULL, 0); + if (l == LONG_MIN) { + perror("Failed to parse image offset"); + return -1; + } + config.image_offset = l; + break; + case 'D': + l = strtol(optarg, NULL, 0); + if (l == LONG_MIN) { + perror("Failed to parse dtb offset"); + return -1; + } + config.dtb_offset = l; + break; + case 'R': + l = strtol(optarg, NULL, 0); + if (l == LONG_MIN) { + perror("Failed to parse ramdisk offset"); + return -1; + } + config.ramdisk_offset = l; + break; + case 'h': + print_help(argv[0]); + return 0; + default: + print_help(argv[0]); + return -1; + } + } + + if (!config.image_fd || !config.dtb_fd) { + print_help(argv[0]); + return -1; + } + + if (config.image_offset + config.image_size > config.guest_size) { + fprintf(stderr, "Image offset and size puts it outside guest memory. Make image smaller or increase guest memory size.\n"); + return -1; + } + + if (config.dtb_offset + config.dtb_size > config.guest_size) { + fprintf(stderr, "DTB offset and size puts it outside guest memory. Make dtb smaller or increase guest memory size.\n"); + return -1; + } + + if (config.ramdisk_fd == -1 && + config.ramdisk_offset + config.ramdisk_size > config.guest_size) { + fprintf(stderr, "Ramdisk offset and size puts it outside guest memory. Make ramdisk smaller or increase guest memory size.\n"); + return -1; + } + + gunyah_fd = open("/dev/gunyah", O_RDWR | O_CLOEXEC); + if (gunyah_fd < 0) { + perror("Failed to open /dev/gunyah"); + return -1; + } + + vm_fd = ioctl(gunyah_fd, GH_CREATE_VM, 0); + if (vm_fd < 0) { + perror("Failed to create vm"); + return -1; + } + + guest_fd = memfd_create("guest_memory", MFD_CLOEXEC); + if (guest_fd < 0) { + perror("Failed to create guest memfd"); + return -1; + } + + if (ftruncate(guest_fd, config.guest_size) < 0) { + perror("Failed to grow guest memory"); + return -1; + } + + guest_mem = mmap(NULL, config.guest_size, PROT_READ | PROT_WRITE, MAP_SHARED, guest_fd, 0); + if (guest_mem == MAP_FAILED) { + perror("Not enough memory"); + return -1; + } + + if (read(config.image_fd, guest_mem + config.image_offset, config.image_size) < 0) { + perror("Failed to read image into guest memory"); + return -1; + } + + if (read(config.dtb_fd, guest_mem + config.dtb_offset, config.dtb_size) < 0) { + perror("Failed to read dtb into guest memory"); + return -1; + } + + if (config.ramdisk_fd > 0 && + read(config.ramdisk_fd, guest_mem + config.ramdisk_offset, + config.ramdisk_size) < 0) { + perror("Failed to read ramdisk into guest memory"); + return -1; + } + + guest_mem_desc.label = 0; + guest_mem_desc.flags = GH_MEM_ALLOW_READ | GH_MEM_ALLOW_WRITE | GH_MEM_ALLOW_EXEC; + guest_mem_desc.guest_phys_addr = config.guest_base; + guest_mem_desc.memory_size = config.guest_size; + guest_mem_desc.userspace_addr = (__u64)guest_mem; + + if (ioctl(vm_fd, GH_VM_SET_USER_MEM_REGION, &guest_mem_desc) < 0) { + perror("Failed to register guest memory with VM"); + return -1; + } + + dtb_config.gpa = config.guest_base + config.dtb_offset; + dtb_config.size = config.dtb_size; + if (ioctl(vm_fd, GH_VM_SET_DTB_CONFIG, &dtb_config) < 0) { + perror("Failed to set DTB configuration for VM"); + return -1; + } + + ret = ioctl(vm_fd, GH_VM_START); + if (ret) { + perror("GH_VM_START failed"); + return -1; + } + + while (1) + sleep(10); + + return 0; +} diff --git a/samples/gunyah/sample_vm.dts b/samples/gunyah/sample_vm.dts new file mode 100644 index 000000000000..293bbc0469c8 --- /dev/null +++ b/samples/gunyah/sample_vm.dts @@ -0,0 +1,68 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +/dts-v1/; + +/ { + #address-cells = <2>; + #size-cells = <2>; + interrupt-parent = <&intc>; + + chosen { + bootargs = "nokaslr"; + }; + + cpus { + #address-cells = <0x2>; + #size-cells = <0>; + + cpu@0 { + device_type = "cpu"; + compatible = "arm,armv8"; + reg = <0 0>; + }; + }; + + intc: interrupt-controller@3FFF0000 { + compatible = "arm,gic-v3"; + #interrupt-cells = <3>; + #address-cells = <2>; + #size-cells = <2>; + interrupt-controller; + reg = <0 0x3FFF0000 0 0x10000>, + <0 0x3FFD0000 0 0x20000>; + }; + + timer { + compatible = "arm,armv8-timer"; + always-on; + interrupts = <1 13 0x108>, + <1 14 0x108>, + <1 11 0x108>, + <1 10 0x108>; + clock-frequency = <19200000>; + }; + + gunyah-vm-config { + image-name = "linux_vm_0"; + + memory { + #address-cells = <2>; + #size-cells = <2>; + + base-address = <0 0x80000000>; + }; + + interrupts { + config = <&intc>; + }; + + vcpus { + affinity-map = < 0 >; + sched-priority = < (-1) >; + sched-timeslice = < 2000 >; + }; + }; +}; From patchwork Tue Feb 14 21:24:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 653520 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9C8BC6379F for ; Tue, 14 Feb 2023 21:26:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229591AbjBNV0j (ORCPT ); Tue, 14 Feb 2023 16:26:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50380 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229581AbjBNV0i (ORCPT ); Tue, 14 Feb 2023 16:26:38 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B4E87ABD; Tue, 14 Feb 2023 13:26:01 -0800 (PST) Received: from pps.filterd (m0279868.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 31EBLApr014334; Tue, 14 Feb 2023 21:25:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=XPWY8+sNHOreIEvICe2zHJsGaLipAQ2qPjvVkiKc32o=; b=nUPVJaM9ibQXvbDBWhdw4AmUYjki3QNsGa3ySGXDNsGjOej9+Cql0JIvrHLZHlYfeuzX 2v2c8tgHwd7egQfi73zRkzXyTLbPt9JKkXB7EPWLascMK+bORHQ8UXKGL6faL4sPs/Z8 4Vf3Syn+jZVtys7whBMh798vPDlL7crOsqFmwbIf9lIsXwmF3HzTTsMX7dh3BnqMVqtX /z419NhMhL+3X61mzUQZIGFUEC5D5AyCy8RIggMXP/vuT7wwhCXFjOlRyT0byQqFvrxh Pbp1+9YnQSfG/YUiZAb4zr6VVFXsYFNkTmBeCYqVhaTTsM1yuC+HPZumKTqqAK/bLp1D Hg== Received: from nasanppmta04.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3nr6qkhxgs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:25:07 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA04.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 31ELP6TW000552 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:25:06 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Tue, 14 Feb 2023 13:25:05 -0800 From: Elliot Berman To: Alex Elder , Srinivas Kandagatla , Andy Gross , "Bjorn Andersson" , Konrad Dybcio CC: Elliot Berman , Murali Nalajala , Trilok Soni , "Srivatsa Vaddagiri" , Carl van Schaik , Prakruthi Deepak Heragu , Dmitry Baryshkov , Arnd Bergmann , Greg Kroah-Hartman , Rob Herring , "Krzysztof Kozlowski" , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Jassi Brar , , , , , Subject: [PATCH v10 16/26] firmware: qcom_scm: Register Gunyah platform ops Date: Tue, 14 Feb 2023 13:24:57 -0800 Message-ID: <20230214212457.3319814-1-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230214211229.3239350-1-quic_eberman@quicinc.com> References: <20230214211229.3239350-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01a.na.qualcomm.com (10.47.209.196) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: rHalghu-Y0iVd-6YK544BfGcEGYTmKzj X-Proofpoint-ORIG-GUID: rHalghu-Y0iVd-6YK544BfGcEGYTmKzj X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-14_15,2023-02-14_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 adultscore=0 clxscore=1015 bulkscore=0 impostorscore=0 suspectscore=0 lowpriorityscore=0 spamscore=0 priorityscore=1501 mlxscore=0 phishscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2302140183 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Qualcomm platforms have a firmware entity which performs access control to physical pages. Dynamically started Gunyah virtual machines use the QCOM_SCM_RM_MANAGED_VMID for access. Linux thus needs to assign access to the memory used by guest VMs. Gunyah doesn't do this operation for us since it is the current VM (typically VMID_HLOS) delegating the access and not Gunyah itself. Use the Gunyah platform ops to achieve this so that only Qualcomm platforms attempt to make the needed SCM calls. Co-developed-by: Prakruthi Deepak Heragu Signed-off-by: Prakruthi Deepak Heragu Signed-off-by: Elliot Berman --- drivers/firmware/Kconfig | 2 + drivers/firmware/qcom_scm.c | 100 ++++++++++++++++++++++++++++++++++++ 2 files changed, 102 insertions(+) diff --git a/drivers/firmware/Kconfig b/drivers/firmware/Kconfig index b59e3041fd62..b888068ff6f2 100644 --- a/drivers/firmware/Kconfig +++ b/drivers/firmware/Kconfig @@ -214,6 +214,8 @@ config MTK_ADSP_IPC config QCOM_SCM tristate + select VIRT_DRIVERS + select GUNYAH_PLATFORM_HOOKS config QCOM_SCM_DOWNLOAD_MODE_DEFAULT bool "Qualcomm download mode enabled by default" diff --git a/drivers/firmware/qcom_scm.c b/drivers/firmware/qcom_scm.c index 468d4d5ab550..875040982b48 100644 --- a/drivers/firmware/qcom_scm.c +++ b/drivers/firmware/qcom_scm.c @@ -20,6 +20,7 @@ #include #include #include +#include #include "qcom_scm.h" @@ -30,6 +31,9 @@ module_param(download_mode, bool, 0); #define SCM_HAS_IFACE_CLK BIT(1) #define SCM_HAS_BUS_CLK BIT(2) +#define QCOM_SCM_RM_MANAGED_VMID 0x3A +#define QCOM_SCM_MAX_MANAGED_VMID 0x3F + struct qcom_scm { struct device *dev; struct clk *core_clk; @@ -1297,6 +1301,99 @@ int qcom_scm_lmh_dcvsh(u32 payload_fn, u32 payload_reg, u32 payload_val, } EXPORT_SYMBOL(qcom_scm_lmh_dcvsh); +static int qcom_scm_gh_rm_pre_mem_share(struct gh_rm *rm, struct gh_rm_mem_parcel *mem_parcel) +{ + struct qcom_scm_vmperm *new_perms; + u64 src, src_cpy; + int ret = 0, i, n; + u16 vmid; + + new_perms = kcalloc(mem_parcel->n_acl_entries, sizeof(*new_perms), GFP_KERNEL); + if (!new_perms) + return -ENOMEM; + + for (n = 0; n < mem_parcel->n_acl_entries; n++) { + vmid = le16_to_cpu(mem_parcel->acl_entries[n].vmid); + if (vmid <= QCOM_SCM_MAX_MANAGED_VMID) + new_perms[n].vmid = vmid; + else + new_perms[n].vmid = QCOM_SCM_RM_MANAGED_VMID; + if (mem_parcel->acl_entries[n].perms & GH_RM_ACL_X) + new_perms[n].perm |= QCOM_SCM_PERM_EXEC; + if (mem_parcel->acl_entries[n].perms & GH_RM_ACL_W) + new_perms[n].perm |= QCOM_SCM_PERM_WRITE; + if (mem_parcel->acl_entries[n].perms & GH_RM_ACL_R) + new_perms[n].perm |= QCOM_SCM_PERM_READ; + } + + src = (1ull << QCOM_SCM_VMID_HLOS); + + for (i = 0; i < mem_parcel->n_mem_entries; i++) { + src_cpy = src; + ret = qcom_scm_assign_mem(le64_to_cpu(mem_parcel->mem_entries[i].ipa_base), + le64_to_cpu(mem_parcel->mem_entries[i].size), + &src_cpy, new_perms, mem_parcel->n_acl_entries); + if (ret) { + src = 0; + for (n = 0; n < mem_parcel->n_acl_entries; n++) { + vmid = le16_to_cpu(mem_parcel->acl_entries[n].vmid); + if (vmid <= QCOM_SCM_MAX_MANAGED_VMID) + src |= (1ull << vmid); + else + src |= (1ull << QCOM_SCM_RM_MANAGED_VMID); + } + + new_perms[0].vmid = QCOM_SCM_VMID_HLOS; + + for (i--; i >= 0; i--) { + src_cpy = src; + WARN_ON_ONCE(qcom_scm_assign_mem( + le64_to_cpu(mem_parcel->mem_entries[i].ipa_base), + le64_to_cpu(mem_parcel->mem_entries[i].size), + &src_cpy, new_perms, 1)); + } + break; + } + } + + kfree(new_perms); + return ret; +} + +static int qcom_scm_gh_rm_post_mem_reclaim(struct gh_rm *rm, struct gh_rm_mem_parcel *mem_parcel) +{ + struct qcom_scm_vmperm new_perms; + u64 src = 0, src_cpy; + int ret = 0, i, n; + u16 vmid; + + new_perms.vmid = QCOM_SCM_VMID_HLOS; + new_perms.perm = QCOM_SCM_PERM_EXEC | QCOM_SCM_PERM_WRITE | QCOM_SCM_PERM_READ; + + for (n = 0; n < mem_parcel->n_acl_entries; n++) { + vmid = le16_to_cpu(mem_parcel->acl_entries[n].vmid); + if (vmid <= QCOM_SCM_MAX_MANAGED_VMID) + src |= (1ull << vmid); + else + src |= (1ull << QCOM_SCM_RM_MANAGED_VMID); + } + + for (i = 0; i < mem_parcel->n_mem_entries; i++) { + src_cpy = src; + ret = qcom_scm_assign_mem(le64_to_cpu(mem_parcel->mem_entries[i].ipa_base), + le64_to_cpu(mem_parcel->mem_entries[i].size), + &src_cpy, &new_perms, 1); + WARN_ON_ONCE(ret); + } + + return ret; +} + +static struct gunyah_rm_platform_ops qcom_scm_gh_rm_platform_ops = { + .pre_mem_share = qcom_scm_gh_rm_pre_mem_share, + .post_mem_reclaim = qcom_scm_gh_rm_post_mem_reclaim, +}; + static int qcom_scm_find_dload_address(struct device *dev, u64 *addr) { struct device_node *tcsr; @@ -1500,6 +1597,9 @@ static int qcom_scm_probe(struct platform_device *pdev) if (download_mode) qcom_scm_set_download_mode(true); + if (devm_gh_rm_register_platform_ops(&pdev->dev, &qcom_scm_gh_rm_platform_ops)) + dev_warn(__scm->dev, "Gunyah RM platform ops were already registered\n"); + return 0; } From patchwork Tue Feb 14 21:25:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 653516 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0A06C64EC7 for ; Tue, 14 Feb 2023 21:46:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231891AbjBNVqL (ORCPT ); Tue, 14 Feb 2023 16:46:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39548 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230379AbjBNVqK (ORCPT ); Tue, 14 Feb 2023 16:46:10 -0500 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F299029400; Tue, 14 Feb 2023 13:46:08 -0800 (PST) Received: from pps.filterd (m0279862.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 31ELHFtl027210; Tue, 14 Feb 2023 21:25:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=XFIbjyjqBQ0YC71kR7vks/UO90Rz2T2TuIWzalc6tiI=; b=HVcDH1TCvKCeFPTooCj1iwZfuGQ7XtQ22SGOB8nLH0i0pkB02jujmQ6YuNSav8tzUwPA TJ4tzHrnoufGvvDkO6BGU7XE6pJA3tLRo82midoHJYGBH680VmW8egKAgUMTp1dECn1F sToMsYkujDuBRxME0F2mctMCrswOjM9NX/TZnNye1EgK98AKGo5qiZg+hLHk8ncSmkjJ aYbfII0tQC65fsczXXRx5WrzBLLN8KBIoQtrD7ENO/G89gvEXb4WwOxSRHz8RQbzS8mN 3/Z+fSc98nI06owh6dSeHmRPcxBaraVknJkwuGHlYjCi4E4Uk1eH47PbDTjg4N2EXkJh nQ== Received: from nasanppmta04.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3nqyygttx1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:25:31 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA04.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 31ELPUmV001471 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:25:30 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Tue, 14 Feb 2023 13:25:29 -0800 From: Elliot Berman To: Alex Elder , Srinivas Kandagatla , Catalin Marinas , Will Deacon , Elliot Berman , Prakruthi Deepak Heragu CC: Murali Nalajala , Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Dmitry Baryshkov , Bjorn Andersson , "Konrad Dybcio" , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Jassi Brar , , , , , Subject: [PATCH v10 18/26] virt: gunyah: Translate gh_rm_hyp_resource into gunyah_resource Date: Tue, 14 Feb 2023 13:25:20 -0800 Message-ID: <20230214212521.3322247-1-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230214211229.3239350-1-quic_eberman@quicinc.com> References: <20230214211229.3239350-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01a.na.qualcomm.com (10.47.209.196) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: eKEr1g2BVd_PtdDwXMlAEEUR0RkxzYGK X-Proofpoint-GUID: eKEr1g2BVd_PtdDwXMlAEEUR0RkxzYGK X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-14_15,2023-02-14_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 phishscore=0 impostorscore=0 malwarescore=0 adultscore=0 mlxlogscore=947 mlxscore=0 priorityscore=1501 lowpriorityscore=0 spamscore=0 bulkscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2302140184 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org When booting a Gunyah virtual machine, the host VM may gain capabilities to interact with resources for the guest virtual machine. Examples of such resources are vCPUs or message queues. To use those resources, we need to translate the RM response into a gunyah_resource structure which are useful to Linux drivers. Presently, Linux drivers need only to know the type of resource, the capability ID, and an interrupt. On ARM64 systems, the interrupt reported by Gunyah is the GIC interrupt ID number and always a SPI. Signed-off-by: Elliot Berman --- arch/arm64/include/asm/gunyah.h | 23 +++++ drivers/virt/gunyah/rsc_mgr.c | 161 +++++++++++++++++++++++++++++++- include/linux/gunyah.h | 4 + include/linux/gunyah_rsc_mgr.h | 4 + 4 files changed, 191 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/include/asm/gunyah.h diff --git a/arch/arm64/include/asm/gunyah.h b/arch/arm64/include/asm/gunyah.h new file mode 100644 index 000000000000..64cfb964efee --- /dev/null +++ b/arch/arm64/include/asm/gunyah.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved. + */ +#ifndef __ASM_GUNYAH_H_ +#define __ASM_GUNYAH_H_ + +#include +#include + +static inline int arch_gh_fill_irq_fwspec_params(u32 virq, struct irq_fwspec *fwspec) +{ + if (virq < 32 || virq > 1019) + return -EINVAL; + + fwspec->param_count = 3; + fwspec->param[0] = GIC_SPI; + fwspec->param[1] = virq - 32; + fwspec->param[2] = IRQ_TYPE_EDGE_RISING; + return 0; +} + +#endif diff --git a/drivers/virt/gunyah/rsc_mgr.c b/drivers/virt/gunyah/rsc_mgr.c index 73c5a6b7cbbc..eb1bc3f68792 100644 --- a/drivers/virt/gunyah/rsc_mgr.c +++ b/drivers/virt/gunyah/rsc_mgr.c @@ -18,6 +18,8 @@ #include #include +#include + #include "rsc_mgr.h" #include "vm_mgr.h" @@ -107,8 +109,137 @@ struct gh_rm { struct blocking_notifier_head nh; struct miscdevice miscdev; + struct irq_domain *irq_domain; +}; + +struct gh_irq_chip_data { + u32 gh_virq; +}; + +static struct irq_chip gh_rm_irq_chip = { + .name = "Gunyah", + .irq_enable = irq_chip_enable_parent, + .irq_disable = irq_chip_disable_parent, + .irq_ack = irq_chip_ack_parent, + .irq_mask = irq_chip_mask_parent, + .irq_mask_ack = irq_chip_mask_ack_parent, + .irq_unmask = irq_chip_unmask_parent, + .irq_eoi = irq_chip_eoi_parent, + .irq_set_affinity = irq_chip_set_affinity_parent, + .irq_set_type = irq_chip_set_type_parent, + .irq_set_wake = irq_chip_set_wake_parent, + .irq_set_vcpu_affinity = irq_chip_set_vcpu_affinity_parent, + .irq_retrigger = irq_chip_retrigger_hierarchy, + .irq_get_irqchip_state = irq_chip_get_parent_state, + .irq_set_irqchip_state = irq_chip_set_parent_state, + .flags = IRQCHIP_SET_TYPE_MASKED | + IRQCHIP_SKIP_SET_WAKE | + IRQCHIP_MASK_ON_SUSPEND, +}; + +static int gh_rm_irq_domain_alloc(struct irq_domain *d, unsigned int virq, unsigned int nr_irqs, + void *arg) +{ + struct gh_irq_chip_data *chip_data, *spec = arg; + struct irq_fwspec parent_fwspec; + struct gh_rm *rm = d->host_data; + u32 gh_virq = spec->gh_virq; + int ret; + + if (nr_irqs != 1 || gh_virq == U32_MAX) + return -EINVAL; + + chip_data = kzalloc(sizeof(*chip_data), GFP_KERNEL); + if (!chip_data) + return -ENOMEM; + + chip_data->gh_virq = gh_virq; + + ret = irq_domain_set_hwirq_and_chip(d, virq, chip_data->gh_virq, &gh_rm_irq_chip, + chip_data); + if (ret) + return ret; + + parent_fwspec.fwnode = d->parent->fwnode; + ret = arch_gh_fill_irq_fwspec_params(chip_data->gh_virq, &parent_fwspec); + if (ret) { + dev_err(rm->dev, "virq translation failed %u: %d\n", chip_data->gh_virq, ret); + goto err_free_irq_data; + } + + ret = irq_domain_alloc_irqs_parent(d, virq, nr_irqs, &parent_fwspec); + if (ret) + goto err_free_irq_data; + + return ret; +err_free_irq_data: + kfree(chip_data); + return ret; +} + +static void gh_rm_irq_domain_free_single(struct irq_domain *d, unsigned int virq) +{ + struct gh_irq_chip_data *chip_data; + struct irq_data *irq_data; + + irq_data = irq_domain_get_irq_data(d, virq); + if (!irq_data) + return; + + chip_data = irq_data->chip_data; + + kfree(chip_data); + irq_data->chip_data = NULL; +} + +static void gh_rm_irq_domain_free(struct irq_domain *d, unsigned int virq, unsigned int nr_irqs) +{ + unsigned int i; + + for (i = 0; i < nr_irqs; i++) + gh_rm_irq_domain_free_single(d, virq); +} + +static const struct irq_domain_ops gh_rm_irq_domain_ops = { + .alloc = gh_rm_irq_domain_alloc, + .free = gh_rm_irq_domain_free, }; +struct gunyah_resource *gh_rm_alloc_resource(struct gh_rm *rm, + struct gh_rm_hyp_resource *hyp_resource) +{ + struct gunyah_resource *ghrsc; + + ghrsc = kzalloc(sizeof(*ghrsc), GFP_KERNEL); + if (!ghrsc) + return NULL; + + ghrsc->type = hyp_resource->type; + ghrsc->capid = le64_to_cpu(hyp_resource->cap_id); + ghrsc->irq = IRQ_NOTCONNECTED; + ghrsc->rm_label = le32_to_cpu(hyp_resource->resource_label); + if (hyp_resource->virq && le32_to_cpu(hyp_resource->virq) != U32_MAX) { + struct gh_irq_chip_data irq_data = { + .gh_virq = le32_to_cpu(hyp_resource->virq), + }; + + ghrsc->irq = irq_domain_alloc_irqs(rm->irq_domain, 1, NUMA_NO_NODE, &irq_data); + if (ghrsc->irq < 0) { + pr_err("Failed to allocate interrupt for resource %d label: %d: %d\n", + ghrsc->type, ghrsc->rm_label, ghrsc->irq); + ghrsc->irq = IRQ_NOTCONNECTED; + } + } + + return ghrsc; +} + +void gh_rm_free_resource(struct gunyah_resource *ghrsc) +{ + irq_dispose_mapping(ghrsc->irq); + kfree(ghrsc); +} + static struct gh_rm_connection *gh_rm_alloc_connection(__le32 msg_id, u8 type) { struct gh_rm_connection *connection; @@ -553,6 +684,8 @@ static int gh_msgq_platform_probe_direction(struct platform_device *pdev, static int gh_rm_drv_probe(struct platform_device *pdev) { + struct irq_domain *parent_irq_domain; + struct device_node *parent_irq_node; struct gh_msgq_tx_data *msg; struct gh_rm *rm; int ret; @@ -590,15 +723,40 @@ static int gh_rm_drv_probe(struct platform_device *pdev) if (ret) goto err_cache; + parent_irq_node = of_irq_find_parent(pdev->dev.of_node); + if (!parent_irq_node) { + dev_err(&pdev->dev, "Failed to find interrupt parent of resource manager\n"); + ret = -ENODEV; + goto err_msgq; + } + + parent_irq_domain = irq_find_host(parent_irq_node); + if (!parent_irq_domain) { + dev_err(&pdev->dev, "Failed to find interrupt parent domain of resource manager\n"); + ret = -ENODEV; + goto err_msgq; + } + + rm->irq_domain = irq_domain_add_hierarchy(parent_irq_domain, 0, 0, pdev->dev.of_node, + &gh_rm_irq_domain_ops, NULL); + if (!rm->irq_domain) { + dev_err(&pdev->dev, "Failed to add irq domain\n"); + ret = -ENODEV; + goto err_msgq; + } + rm->irq_domain->host_data = rm; + rm->miscdev.name = "gunyah"; rm->miscdev.minor = MISC_DYNAMIC_MINOR; rm->miscdev.fops = &gh_dev_fops; ret = misc_register(&rm->miscdev); if (ret) - goto err_msgq; + goto err_irq_domain; return 0; +err_irq_domain: + irq_domain_remove(rm->irq_domain); err_msgq: mbox_free_channel(gh_msgq_chan(&rm->msgq)); gh_msgq_remove(&rm->msgq); @@ -612,6 +770,7 @@ static int gh_rm_drv_remove(struct platform_device *pdev) struct gh_rm *rm = platform_get_drvdata(pdev); misc_deregister(&rm->miscdev); + irq_domain_remove(rm->irq_domain); mbox_free_channel(gh_msgq_chan(&rm->msgq)); gh_msgq_remove(&rm->msgq); kmem_cache_destroy(rm->cache); diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h index 2e13669c6363..a06d5fa68a65 100644 --- a/include/linux/gunyah.h +++ b/include/linux/gunyah.h @@ -27,6 +27,10 @@ struct gunyah_resource { enum gunyah_resource_type type; u64 capid; int irq; + + /* To help allocator of resource manager */ + struct list_head list; + u32 rm_label; }; /** diff --git a/include/linux/gunyah_rsc_mgr.h b/include/linux/gunyah_rsc_mgr.h index dc05d5b1e1a3..2fb6efbe2f70 100644 --- a/include/linux/gunyah_rsc_mgr.h +++ b/include/linux/gunyah_rsc_mgr.h @@ -147,6 +147,10 @@ int gh_rm_get_hyp_resources(struct gh_rm *rm, u16 vmid, struct gh_rm_hyp_resources **resources); int gh_rm_get_vmid(struct gh_rm *rm, u16 *vmid); +struct gunyah_resource *gh_rm_alloc_resource(struct gh_rm *rm, + struct gh_rm_hyp_resource *hyp_resource); +void gh_rm_free_resource(struct gunyah_resource *ghrsc); + struct gunyah_rm_platform_ops { int (*pre_mem_share)(struct gh_rm *rm, struct gh_rm_mem_parcel *mem_parcel); int (*post_mem_reclaim)(struct gh_rm *rm, struct gh_rm_mem_parcel *mem_parcel); From patchwork Tue Feb 14 21:25:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 653518 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 548C1C05027 for ; Tue, 14 Feb 2023 21:27:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233007AbjBNV1w (ORCPT ); Tue, 14 Feb 2023 16:27:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51146 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233043AbjBNV1o (ORCPT ); Tue, 14 Feb 2023 16:27:44 -0500 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35E5C303F1; Tue, 14 Feb 2023 13:27:21 -0800 (PST) Received: from pps.filterd (m0279865.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 31EFfkIY008029; Tue, 14 Feb 2023 21:25:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=b7G1zihIRPJyCCnYYGc4mIFQvqk2y844xfg2/C9NQv4=; b=DbOShkAyt3l5ADheaYdXVMiQDJHc7OZw5HI7HiZYudCHnH2cjbWJ/nxQeY1//TiEXzSb fqkprEIqjK4RD5Q4ZhheIDBU7zTRhLq99f6S5+Q1qzKhW5uMpjJFaDXp6hRr+bcFZM5r ABnQVSDVYFO/U4p5AhWczw5inZMTaZnHb9LjhlHV8zb5TLit9MhatHWU5vRmjDy8QLZ/ Z+UItv2Y1KP9beE20445fAPfWyIolDZ6dsLiCzdRFQ/3IEFuKmjEpCfliV7NxfzCYLnm MV1YeXZKlxLXapWZSY2ARXieS3hEQ9sXTYpJPlK3PS0s6FSld2wR+/iSoz8wdNT3D/8O ow== Received: from nasanppmta05.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3nr26u2hue-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:25:40 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA05.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 31ELPegC005625 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:25:40 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Tue, 14 Feb 2023 13:25:39 -0800 From: Elliot Berman To: Alex Elder , Srinivas Kandagatla , Elliot Berman , Prakruthi Deepak Heragu , Jonathan Corbet CC: Murali Nalajala , Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Dmitry Baryshkov , Bjorn Andersson , "Konrad Dybcio" , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Bagas Sanjaya , Catalin Marinas , Jassi Brar , , , , , Subject: [PATCH v10 19/26] gunyah: vm_mgr: Add framework to add VM Functions Date: Tue, 14 Feb 2023 13:25:30 -0800 Message-ID: <20230214212531.3323284-1-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230214211229.3239350-1-quic_eberman@quicinc.com> References: <20230214211229.3239350-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01a.na.qualcomm.com (10.47.209.196) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: 8bTHOsjE0C-CGJOteESNWd9xYK6onqlk X-Proofpoint-ORIG-GUID: 8bTHOsjE0C-CGJOteESNWd9xYK6onqlk X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-14_15,2023-02-14_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 adultscore=0 mlxscore=0 spamscore=0 impostorscore=0 malwarescore=0 mlxlogscore=999 clxscore=1015 lowpriorityscore=0 bulkscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2302140184 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Introduce a framework for Gunyah userspace to install VM functions. VM functions are optional interfaces to the virtual machine. vCPUs, ioeventfs, and irqfds are examples of such VM functions and are implemented in subsequent patches. A generic framework is implemented instead of individual ioctls to create vCPUs, irqfds, etc., in order to simplify the VM manager core implementation and allow dynamic loading of VM function modules. Signed-off-by: Elliot Berman --- Documentation/virt/gunyah/vm-manager.rst | 18 ++ drivers/virt/gunyah/vm_mgr.c | 240 ++++++++++++++++++++++- drivers/virt/gunyah/vm_mgr.h | 3 + include/linux/gunyah_vm_mgr.h | 80 ++++++++ include/uapi/linux/gunyah.h | 17 ++ 5 files changed, 353 insertions(+), 5 deletions(-) create mode 100644 include/linux/gunyah_vm_mgr.h diff --git a/Documentation/virt/gunyah/vm-manager.rst b/Documentation/virt/gunyah/vm-manager.rst index c0126cfeadc7..5272a6e9145c 100644 --- a/Documentation/virt/gunyah/vm-manager.rst +++ b/Documentation/virt/gunyah/vm-manager.rst @@ -17,6 +17,24 @@ sharing userspace memory with a VM is done via the GH_VM_SET_USER_MEM_REGION ioctl. The VM itself is configured to use the memory region via the devicetree. +Gunyah Functions +================ + +Components of a Gunyah VM's configuration that need kernel configuration are +called "functions" and are built on top of a framework. Functions are identified +by a string and have some argument(s) to configure them. They are typically +created by the `GH_VM_ADD_FUNCTION` ioctl. + +Functions typically will always do at least one of these operations: + +1. Create resource ticket(s). Resource tickets allow a function to register + itself as the client for a Gunyah resource (e.g. doorbell or vCPU) and + the function is given the pointer to the `struct gunyah_resource` when the + VM is starting. + +2. Register IO handler(s). IO handlers allow a function to handle stage-2 faults + from the virtual machine. + Sample Userspace VMM ==================== diff --git a/drivers/virt/gunyah/vm_mgr.c b/drivers/virt/gunyah/vm_mgr.c index fa324385ade5..e9c55e7dd1b3 100644 --- a/drivers/virt/gunyah/vm_mgr.c +++ b/drivers/virt/gunyah/vm_mgr.c @@ -6,8 +6,10 @@ #define pr_fmt(fmt) "gh_vm_mgr: " fmt #include +#include #include #include +#include #include #include #include @@ -16,6 +18,177 @@ #include "vm_mgr.h" +static DEFINE_MUTEX(functions_lock); +static DEFINE_IDR(functions); + +int gh_vm_function_register(struct gh_vm_function *drv) +{ + int ret = 0; + + if (!drv->bind || !drv->unbind) + return -EINVAL; + + mutex_lock(&functions_lock); + if (idr_find(&functions, drv->type)) { + ret = -EEXIST; + goto out; + } + + INIT_LIST_HEAD(&drv->instances); + ret = idr_alloc(&functions, drv, drv->type, drv->type + 1, GFP_KERNEL); + if (ret > 0) + ret = 0; +out: + mutex_unlock(&functions_lock); + return ret; +} +EXPORT_SYMBOL_GPL(gh_vm_function_register); + +static void gh_vm_remove_function_instance(struct gh_vm_function_instance *inst) + __must_hold(functions_lock) +{ + inst->fn->unbind(inst); + list_del(&inst->vm_list); + list_del(&inst->fn_list); + module_put(inst->fn->mod); + if (inst->arg_size) + kfree(inst->argp); + kfree(inst); +} + +void gh_vm_function_unregister(struct gh_vm_function *fn) +{ + struct gh_vm_function_instance *inst, *iter; + + mutex_lock(&functions_lock); + list_for_each_entry_safe(inst, iter, &fn->instances, fn_list) + gh_vm_remove_function_instance(inst); + idr_remove(&functions, fn->type); + mutex_unlock(&functions_lock); +} +EXPORT_SYMBOL_GPL(gh_vm_function_unregister); + +static long gh_vm_add_function(struct gh_vm *ghvm, struct gh_fn_desc *f) +{ + struct gh_vm_function_instance *inst; + void __user *argp; + long r = 0; + + if (f->arg_size > GH_FN_MAX_ARG_SIZE) + return -EINVAL; + + inst = kzalloc(sizeof(*inst), GFP_KERNEL); + if (!inst) + return -ENOMEM; + + inst->arg_size = f->arg_size; + if (inst->arg_size) { + inst->argp = kzalloc(inst->arg_size, GFP_KERNEL); + if (!inst->arg) { + r = -ENOMEM; + goto free; + } + + argp = is_compat_task() ? compat_ptr(f->arg) : (void __user *) f->arg; + if (copy_from_user(inst->argp, argp, f->arg_size)) { + r = -EFAULT; + goto free_arg; + } + } else { + inst->arg = f->arg; + } + + mutex_lock(&functions_lock); + inst->fn = idr_find(&functions, f->type); + if (!inst->fn) { + mutex_unlock(&functions_lock); + r = request_module("ghfunc:%d", f->type); + if (r) + goto unlock_free; + + mutex_lock(&functions_lock); + inst->fn = idr_find(&functions, f->type); + } + + if (!inst->fn) { + r = -ENOENT; + goto unlock_free; + } + + if (!try_module_get(inst->fn->mod)) { + r = -ENOENT; + inst->fn = NULL; + goto unlock_free; + } + + inst->ghvm = ghvm; + inst->rm = ghvm->rm; + + r = inst->fn->bind(inst); + if (r < 0) { + module_put(inst->fn->mod); + goto unlock_free; + } + + list_add(&inst->vm_list, &ghvm->functions); + list_add(&inst->fn_list, &inst->fn->instances); + mutex_unlock(&functions_lock); + return r; +unlock_free: + mutex_unlock(&functions_lock); +free_arg: + if (inst->arg_size) + kfree(inst->argp); +free: + kfree(inst); + return r; +} + +static long gh_vm_rm_function(struct gh_vm *ghvm, struct gh_fn_desc *f) +{ + struct gh_vm_function_instance *inst, *iter; + void __user *user_argp; + void *argp; + long r = 0; + + r = mutex_lock_interruptible(&functions_lock); + if (r) + return r; + + if (f->arg_size) { + argp = kzalloc(f->arg_size, GFP_KERNEL); + if (!argp) { + r = -ENOMEM; + goto out; + } + + user_argp = is_compat_task() ? compat_ptr(f->arg) : (void __user *) f->arg; + if (copy_from_user(argp, user_argp, f->arg_size)) { + r = -EFAULT; + kfree(argp); + goto out; + } + + list_for_each_entry_safe(inst, iter, &ghvm->functions, vm_list) { + if (inst->fn->type == f->type && + f->arg_size == inst->arg_size && + !memcmp(argp, inst->argp, f->arg_size)) + gh_vm_remove_function_instance(inst); + } + } else { + list_for_each_entry_safe(inst, iter, &ghvm->functions, vm_list) { + if (inst->fn->type == f->type && + f->arg_size == inst->arg_size && + inst->arg == f->arg) + gh_vm_remove_function_instance(inst); + } + } + +out: + mutex_unlock(&functions_lock); + return r; +} + static int gh_vm_rm_notification_status(struct gh_vm *ghvm, void *data) { struct gh_rm_vm_status_payload *payload = data; @@ -80,6 +253,7 @@ static void gh_vm_stop(struct gh_vm *ghvm) static void gh_vm_free(struct work_struct *work) { struct gh_vm *ghvm = container_of(work, struct gh_vm, free_work); + struct gh_vm_function_instance *inst, *iiter; struct gh_vm_mem *mapping, *tmp; int ret; @@ -90,7 +264,13 @@ static void gh_vm_free(struct work_struct *work) fallthrough; case GH_RM_VM_STATUS_INIT_FAILED: case GH_RM_VM_STATUS_LOAD: - case GH_RM_VM_STATUS_LOAD_FAILED: + case GH_RM_VM_STATUS_EXITED: + mutex_lock(&functions_lock); + list_for_each_entry_safe(inst, iiter, &ghvm->functions, vm_list) { + gh_vm_remove_function_instance(inst); + } + mutex_unlock(&functions_lock); + mutex_lock(&ghvm->mm_lock); list_for_each_entry_safe(mapping, tmp, &ghvm->memory_mappings, list) { gh_vm_mem_reclaim(ghvm, mapping); @@ -113,6 +293,28 @@ static void gh_vm_free(struct work_struct *work) } } +static void _gh_vm_put(struct kref *kref) +{ + struct gh_vm *ghvm = container_of(kref, struct gh_vm, kref); + + /* VM will be reset and make RM calls which can interruptible sleep. + * Defer to a work so this thread can receive signal. + */ + schedule_work(&ghvm->free_work); +} + +int __must_check gh_vm_get(struct gh_vm *ghvm) +{ + return kref_get_unless_zero(&ghvm->kref); +} +EXPORT_SYMBOL_GPL(gh_vm_get); + +void gh_vm_put(struct gh_vm *ghvm) +{ + kref_put(&ghvm->kref, _gh_vm_put); +} +EXPORT_SYMBOL_GPL(gh_vm_put); + static __must_check struct gh_vm *gh_vm_alloc(struct gh_rm *rm) { struct gh_vm *ghvm; @@ -147,6 +349,8 @@ static __must_check struct gh_vm *gh_vm_alloc(struct gh_rm *rm) INIT_LIST_HEAD(&ghvm->memory_mappings); init_rwsem(&ghvm->status_lock); INIT_WORK(&ghvm->free_work, gh_vm_free); + kref_init(&ghvm->kref); + INIT_LIST_HEAD(&ghvm->functions); ghvm->vm_status = GH_RM_VM_STATUS_LOAD; return ghvm; @@ -291,6 +495,35 @@ static long gh_vm_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) r = gh_vm_ensure_started(ghvm); break; } + case GH_VM_ADD_FUNCTION: { + struct gh_fn_desc *f; + + f = kzalloc(sizeof(*f), GFP_KERNEL); + if (!f) + return -ENOMEM; + + if (copy_from_user(f, argp, sizeof(*f))) + return -EFAULT; + + r = gh_vm_add_function(ghvm, f); + if (r < 0) + kfree(f); + break; + } + case GH_VM_REMOVE_FUNCTION: { + struct gh_fn_desc *f; + + f = kzalloc(sizeof(*f), GFP_KERNEL); + if (!f) + return -ENOMEM; + + if (copy_from_user(f, argp, sizeof(*f))) + return -EFAULT; + + r = gh_vm_rm_function(ghvm, f); + kfree(f); + break; + } default: r = -ENOTTY; break; @@ -303,10 +536,7 @@ static int gh_vm_release(struct inode *inode, struct file *filp) { struct gh_vm *ghvm = filp->private_data; - /* VM will be reset and make RM calls which can interruptible sleep. - * Defer to a work so this thread can receive signal. - */ - schedule_work(&ghvm->free_work); + gh_vm_put(ghvm); return 0; } diff --git a/drivers/virt/gunyah/vm_mgr.h b/drivers/virt/gunyah/vm_mgr.h index e9cf56647cc2..4750d56c1297 100644 --- a/drivers/virt/gunyah/vm_mgr.h +++ b/drivers/virt/gunyah/vm_mgr.h @@ -8,6 +8,7 @@ #include #include +#include #include #include #include @@ -44,8 +45,10 @@ struct gh_vm { struct rw_semaphore status_lock; struct work_struct free_work; + struct kref kref; struct mutex mm_lock; struct list_head memory_mappings; + struct list_head functions; }; int gh_vm_mem_alloc(struct gh_vm *ghvm, struct gh_userspace_memory_region *region); diff --git a/include/linux/gunyah_vm_mgr.h b/include/linux/gunyah_vm_mgr.h new file mode 100644 index 000000000000..f0a95af50b2e --- /dev/null +++ b/include/linux/gunyah_vm_mgr.h @@ -0,0 +1,80 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#ifndef _GUNYAH_VM_MGR_H +#define _GUNYAH_VM_MGR_H + +#include +#include +#include +#include +#include +#include + +#include + +struct gh_vm; + +int __must_check gh_vm_get(struct gh_vm *ghvm); +void gh_vm_put(struct gh_vm *ghvm); + +struct gh_vm_function_instance; +struct gh_vm_function { + u32 type; + const char *name; + struct module *mod; + long (*bind)(struct gh_vm_function_instance *f); + void (*unbind)(struct gh_vm_function_instance *f); + struct mutex instances_lock; + struct list_head instances; +}; + +/** + * struct gh_vm_function_instance - Represents one function instance + * @arg_size: size of user argument + * @arg: user argument to describe the function instance; arg_size is 0 + * @argp: pointer to user argument + * @ghvm: Pointer to VM instance + * @rm: Pointer to resource manager for the VM instance + * @fn: The ops for the function + * @data: Private data for function + * @vm_list: for gh_vm's functions list + * @fn_list: for gh_vm_function's instances list + */ +struct gh_vm_function_instance { + size_t arg_size; + union { + u64 arg; + void *argp; + }; + struct gh_vm *ghvm; + struct gh_rm *rm; + struct gh_vm_function *fn; + void *data; + struct list_head vm_list; + struct list_head fn_list; +}; + +int gh_vm_function_register(struct gh_vm_function *f); +void gh_vm_function_unregister(struct gh_vm_function *f); + +#define DECLARE_GUNYAH_VM_FUNCTION(_name, _type, _bind, _unbind) \ + static struct gh_vm_function _name = { \ + .type = _type, \ + .name = __stringify(_name), \ + .mod = THIS_MODULE, \ + .bind = _bind, \ + .unbind = _unbind, \ + }; \ + MODULE_ALIAS("ghfunc:"__stringify(_type)) + +#define module_gunyah_vm_function(__gf) \ + module_driver(__gf, gh_vm_function_register, gh_vm_function_unregister) + +#define DECLARE_GUNYAH_VM_FUNCTION_INIT(_name, _type, _bind, _unbind) \ + DECLARE_GUNYAH_VM_FUNCTION(_name, _type, _bind, _unbind); \ + module_gunyah_vm_function(_name) + +#endif diff --git a/include/uapi/linux/gunyah.h b/include/uapi/linux/gunyah.h index d899bba6a4c6..8df455a2a293 100644 --- a/include/uapi/linux/gunyah.h +++ b/include/uapi/linux/gunyah.h @@ -66,4 +66,21 @@ struct gh_vm_dtb_config { #define GH_VM_START _IO(GH_IOCTL_TYPE, 0x3) +#define GH_FN_MAX_ARG_SIZE 256 + +/** + * struct gh_fn_desc - Arguments to create a VM function + * @type: Type of the function. See GH_FN_* macro for supported types + * @arg_size: Size of argument to pass to the function + * @arg: Value or pointer to argument given to the function + */ +struct gh_fn_desc { + __u32 type; + __u32 arg_size; + __u64 arg; +}; + +#define GH_VM_ADD_FUNCTION _IOW(GH_IOCTL_TYPE, 0x4, struct gh_fn_desc) +#define GH_VM_REMOVE_FUNCTION _IOW(GH_IOCTL_TYPE, 0x7, struct gh_fn_desc) + #endif From patchwork Tue Feb 14 21:25:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 653519 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CAE1C6379F for ; Tue, 14 Feb 2023 21:27:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233033AbjBNV1m (ORCPT ); Tue, 14 Feb 2023 16:27:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51468 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233034AbjBNV1d (ORCPT ); Tue, 14 Feb 2023 16:27:33 -0500 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 180122FCE2; Tue, 14 Feb 2023 13:27:05 -0800 (PST) Received: from pps.filterd (m0279864.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 31EKidC4012731; Tue, 14 Feb 2023 21:25:53 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=zqGbikFIIa2m+Gc6+asp0mxiTKZjbfNd8w4T+3IJz6o=; b=f1otfkkXBDniANCdcPdoIBE9ij8/5ag/W2riUvsMSYeezVQZne2nTGvNaoBMHeCED6LA wWeYziLLaKiOI6dGvk//XD/nnIa9b681jVHzPqEHJO0LSlOAuOS0zK9iyUHrIDybsKEW OF5S0sywDLq0XI5iQffAvIYOuAyAb/Z/TP29RENJZM/u/ChgaW3f8GBJVJ+wVxYOO2XY zDzLfbRF6mb5ZrIBIE4gKmRi4x50t6kuVjxR/GO+WnsBfMGpKUiIdB1YisvG/MsgSnZR +pSnGXtawb63qqt6pyWnPwZgvThnNrgZkZT+UZDF70V4zFT0QYoE0MPyknoBCAIOiAM6 2A== Received: from nasanppmta02.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3nqpmmm1yr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:25:53 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA02.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 31ELPqIt028542 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:25:52 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Tue, 14 Feb 2023 13:25:51 -0800 From: Elliot Berman To: Alex Elder , Srinivas Kandagatla , Elliot Berman , Prakruthi Deepak Heragu CC: Murali Nalajala , Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Dmitry Baryshkov , Bjorn Andersson , "Konrad Dybcio" , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Catalin Marinas , Jassi Brar , , , , , Subject: [PATCH v10 20/26] virt: gunyah: Add resource tickets Date: Tue, 14 Feb 2023 13:25:40 -0800 Message-ID: <20230214212540.3324345-1-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230214211229.3239350-1-quic_eberman@quicinc.com> References: <20230214211229.3239350-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01a.na.qualcomm.com (10.47.209.196) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: wVUxHGU_83jb-A6eB3GNYXE34WgMsOsh X-Proofpoint-GUID: wVUxHGU_83jb-A6eB3GNYXE34WgMsOsh X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-14_15,2023-02-14_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 spamscore=0 priorityscore=1501 suspectscore=0 adultscore=0 impostorscore=0 bulkscore=0 clxscore=1015 mlxscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2302140184 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Some VM functions need to acquire Gunyah resources. For instance, Gunyah vCPUs are exposed to the host as a resource. The Gunyah vCPU function will register a resource ticket and be able to interact with the hypervisor once the resource ticket is filled. Resource tickets are the mechanism for functions to acquire ownership of Gunyah resources. Gunyah functions can be created before the VM's resources are created and made available to Linux. A resource ticket identifies a type of resource and a label of a resource which the ticket holder is interested in. Resources are created by Gunyah as configured in the VM's devicetree configuration. Gunyah doesn't process the label and that makes it possible for userspace to create multiple resources with the same label. Resource ticket owners need to be prepared for populate to be called multiple times if userspace created multiple resources with the same label. Signed-off-by: Elliot Berman --- drivers/virt/gunyah/vm_mgr.c | 110 +++++++++++++++++++++++++++++++++- drivers/virt/gunyah/vm_mgr.h | 4 ++ include/linux/gunyah_vm_mgr.h | 14 +++++ 3 files changed, 127 insertions(+), 1 deletion(-) diff --git a/drivers/virt/gunyah/vm_mgr.c b/drivers/virt/gunyah/vm_mgr.c index e9c55e7dd1b3..7190107a6a0d 100644 --- a/drivers/virt/gunyah/vm_mgr.c +++ b/drivers/virt/gunyah/vm_mgr.c @@ -189,6 +189,74 @@ static long gh_vm_rm_function(struct gh_vm *ghvm, struct gh_fn_desc *f) return r; } +int gh_vm_add_resource_ticket(struct gh_vm *ghvm, struct gh_vm_resource_ticket *ticket) +{ + struct gh_vm_resource_ticket *iter; + struct gunyah_resource *ghrsc; + int ret = 0; + + mutex_lock(&ghvm->resources_lock); + list_for_each_entry(iter, &ghvm->resource_tickets, list) { + if (iter->resource_type == ticket->resource_type && iter->label == ticket->label) { + ret = -EEXIST; + goto out; + } + } + + if (!try_module_get(ticket->owner)) { + ret = -ENODEV; + goto out; + } + + list_add(&ticket->list, &ghvm->resource_tickets); + INIT_LIST_HEAD(&ticket->resources); + + list_for_each_entry(ghrsc, &ghvm->resources, list) { + if (ghrsc->type == ticket->resource_type && ghrsc->rm_label == ticket->label) { + if (!ticket->populate(ticket, ghrsc)) + list_move(&ghrsc->list, &ticket->resources); + } + } +out: + mutex_unlock(&ghvm->resources_lock); + return ret; +} +EXPORT_SYMBOL_GPL(gh_vm_add_resource_ticket); + +void gh_vm_remove_resource_ticket(struct gh_vm *ghvm, struct gh_vm_resource_ticket *ticket) +{ + struct gunyah_resource *ghrsc, *iter; + + mutex_lock(&ghvm->resources_lock); + list_for_each_entry_safe(ghrsc, iter, &ticket->resources, list) { + ticket->unpopulate(ticket, ghrsc); + list_move(&ghrsc->list, &ghvm->resources); + } + + module_put(ticket->owner); + list_del(&ticket->list); + mutex_unlock(&ghvm->resources_lock); +} +EXPORT_SYMBOL_GPL(gh_vm_remove_resource_ticket); + +static void gh_vm_add_resource(struct gh_vm *ghvm, struct gunyah_resource *ghrsc) +{ + struct gh_vm_resource_ticket *ticket; + + mutex_lock(&ghvm->resources_lock); + list_for_each_entry(ticket, &ghvm->resource_tickets, list) { + if (ghrsc->type == ticket->resource_type && ghrsc->rm_label == ticket->label) { + if (!ticket->populate(ticket, ghrsc)) { + list_add(&ghrsc->list, &ticket->resources); + goto found; + } + } + } + list_add(&ghrsc->list, &ghvm->resources); +found: + mutex_unlock(&ghvm->resources_lock); +} + static int gh_vm_rm_notification_status(struct gh_vm *ghvm, void *data) { struct gh_rm_vm_status_payload *payload = data; @@ -254,6 +322,8 @@ static void gh_vm_free(struct work_struct *work) { struct gh_vm *ghvm = container_of(work, struct gh_vm, free_work); struct gh_vm_function_instance *inst, *iiter; + struct gh_vm_resource_ticket *ticket, *titer; + struct gunyah_resource *ghrsc, *riter; struct gh_vm_mem *mapping, *tmp; int ret; @@ -271,6 +341,23 @@ static void gh_vm_free(struct work_struct *work) } mutex_unlock(&functions_lock); + if (!list_empty(&ghvm->resource_tickets)) { + pr_warn("Dangling resource tickets:\n"); + list_for_each_entry_safe(ticket, titer, &ghvm->resource_tickets, list) { + pr_warn(" %pS\n", ticket->populate); + gh_vm_remove_resource_ticket(ghvm, ticket); + } + } + + list_for_each_entry_safe(ghrsc, riter, &ghvm->resources, list) { + gh_rm_free_resource(ghrsc); + } + + ret = gh_rm_vm_reset(ghvm->rm, ghvm->vmid); + if (ret) + pr_err("Failed to reset the vm: %d\n", ret); + wait_event(ghvm->vm_status_wait, ghvm->vm_status == GH_RM_VM_STATUS_RESET); + mutex_lock(&ghvm->mm_lock); list_for_each_entry_safe(mapping, tmp, &ghvm->memory_mappings, list) { gh_vm_mem_reclaim(ghvm, mapping); @@ -350,6 +437,9 @@ static __must_check struct gh_vm *gh_vm_alloc(struct gh_rm *rm) init_rwsem(&ghvm->status_lock); INIT_WORK(&ghvm->free_work, gh_vm_free); kref_init(&ghvm->kref); + mutex_init(&ghvm->resources_lock); + INIT_LIST_HEAD(&ghvm->resources); + INIT_LIST_HEAD(&ghvm->resource_tickets); INIT_LIST_HEAD(&ghvm->functions); ghvm->vm_status = GH_RM_VM_STATUS_LOAD; @@ -359,9 +449,11 @@ static __must_check struct gh_vm *gh_vm_alloc(struct gh_rm *rm) static int gh_vm_start(struct gh_vm *ghvm) { struct gh_vm_mem *mapping; + struct gh_rm_hyp_resources *resources; + struct gunyah_resource *ghrsc; u64 dtb_offset; u32 mem_handle; - int ret; + int ret, i, n; down_write(&ghvm->status_lock); if (ghvm->vm_status != GH_RM_VM_STATUS_LOAD) { @@ -412,6 +504,22 @@ static int gh_vm_start(struct gh_vm *ghvm) goto err; } + ret = gh_rm_get_hyp_resources(ghvm->rm, ghvm->vmid, &resources); + if (ret) { + pr_warn("Failed to get hypervisor resources for VM: %d\n", ret); + goto err; + } + + for (i = 0, n = le32_to_cpu(resources->n_entries); i < n; i++) { + ghrsc = gh_rm_alloc_resource(ghvm->rm, &resources->entries[i]); + if (!ghrsc) { + ret = -ENOMEM; + goto err; + } + + gh_vm_add_resource(ghvm, ghrsc); + } + ret = gh_rm_vm_start(ghvm->rm, ghvm->vmid); if (ret) { pr_warn("Failed to start VM: %d\n", ret); diff --git a/drivers/virt/gunyah/vm_mgr.h b/drivers/virt/gunyah/vm_mgr.h index 4750d56c1297..56ae97d752d6 100644 --- a/drivers/virt/gunyah/vm_mgr.h +++ b/drivers/virt/gunyah/vm_mgr.h @@ -7,6 +7,7 @@ #define _GH_PRIV_VM_MGR_H #include +#include #include #include #include @@ -49,6 +50,9 @@ struct gh_vm { struct mutex mm_lock; struct list_head memory_mappings; struct list_head functions; + struct mutex resources_lock; + struct list_head resources; + struct list_head resource_tickets; }; int gh_vm_mem_alloc(struct gh_vm *ghvm, struct gh_userspace_memory_region *region); diff --git a/include/linux/gunyah_vm_mgr.h b/include/linux/gunyah_vm_mgr.h index f0a95af50b2e..84579150d6bc 100644 --- a/include/linux/gunyah_vm_mgr.h +++ b/include/linux/gunyah_vm_mgr.h @@ -77,4 +77,18 @@ void gh_vm_function_unregister(struct gh_vm_function *f); DECLARE_GUNYAH_VM_FUNCTION(_name, _type, _bind, _unbind); \ module_gunyah_vm_function(_name) +struct gh_vm_resource_ticket { + struct list_head list; /* for gh_vm's resources list */ + struct list_head resources; /* for gunyah_resources's list */ + enum gunyah_resource_type resource_type; + u32 label; + + struct module *owner; + int (*populate)(struct gh_vm_resource_ticket *ticket, struct gunyah_resource *ghrsc); + void (*unpopulate)(struct gh_vm_resource_ticket *ticket, struct gunyah_resource *ghrsc); +}; + +int gh_vm_add_resource_ticket(struct gh_vm *ghvm, struct gh_vm_resource_ticket *ticket); +void gh_vm_remove_resource_ticket(struct gh_vm *ghvm, struct gh_vm_resource_ticket *ticket); + #endif From patchwork Tue Feb 14 21:26:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 653517 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7EE4C64ED9 for ; Tue, 14 Feb 2023 21:28:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233141AbjBNV2K (ORCPT ); Tue, 14 Feb 2023 16:28:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51166 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233079AbjBNV1x (ORCPT ); Tue, 14 Feb 2023 16:27:53 -0500 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2897D21968; Tue, 14 Feb 2023 13:27:30 -0800 (PST) Received: from pps.filterd (m0279868.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 31EK7bg0003281; Tue, 14 Feb 2023 21:26:21 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=obyoAyuAjuu3vuNVMV1Ga1m99m6o20IntGSnJs/cUlY=; b=UoqG/nGvpgV6XUM2ExnjVYfVtjH1jjz6+Vq7O+mR6fvsraUd3/qkBGJ3g/u3o77Kz/Io u7WrrbGx+DOQW/cJArxvil9ahJs2+1R/QV9r+CPLsNgJo7DfvOX1tSAHGsU7833LfvuO XBi2QE0Eqz69CFeLxfPca4TP7gDgrvbAEsSZVUAk2zNJdPt8OSGAoEtQabEWBRcPk0WC f6KcnAzykdRHpG/g+i7h/nHxrdgZEUfgcPVzEKykxq+wdUHCyc8wqPRvmo6NHMv4tIWm abDp4vzBBxS/9A6YpO057qkbsD937BuzE9oxsb5WygvMxNiY8YDbdFCAag8qgfgxNtPT tg== Received: from nasanppmta02.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3nr6qkhxjt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:26:21 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA02.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 31ELQKkc028958 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:26:20 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Tue, 14 Feb 2023 13:26:18 -0800 From: Elliot Berman To: Alex Elder , Srinivas Kandagatla , Elliot Berman , Prakruthi Deepak Heragu , Catalin Marinas , Will Deacon CC: Murali Nalajala , Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Dmitry Baryshkov , Bjorn Andersson , "Konrad Dybcio" , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Jonathan Corbet , Bagas Sanjaya , Jassi Brar , , , , , Subject: [PATCH v10 23/26] virt: gunyah: Add hypercalls for sending doorbell Date: Tue, 14 Feb 2023 13:26:10 -0800 Message-ID: <20230214212610.3327589-1-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230214211229.3239350-1-quic_eberman@quicinc.com> References: <20230214211229.3239350-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01a.na.qualcomm.com (10.47.209.196) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: _qahj_dGRK4K2TkFP_ROOyBc7CeNTUB6 X-Proofpoint-ORIG-GUID: _qahj_dGRK4K2TkFP_ROOyBc7CeNTUB6 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-14_15,2023-02-14_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 adultscore=0 clxscore=1015 bulkscore=0 impostorscore=0 suspectscore=0 lowpriorityscore=0 spamscore=0 priorityscore=1501 mlxscore=0 phishscore=0 mlxlogscore=784 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2302140184 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Gunyah doorbells allow two virtual machines to signal each other using interrupts. Add the hypercalls needed to assert the interrupt. Signed-off-by: Elliot Berman --- arch/arm64/gunyah/gunyah_hypercall.c | 25 +++++++++++++++++++++++++ include/linux/gunyah.h | 3 +++ 2 files changed, 28 insertions(+) diff --git a/arch/arm64/gunyah/gunyah_hypercall.c b/arch/arm64/gunyah/gunyah_hypercall.c index 260d416dd006..54e561159ac7 100644 --- a/arch/arm64/gunyah/gunyah_hypercall.c +++ b/arch/arm64/gunyah/gunyah_hypercall.c @@ -38,6 +38,8 @@ EXPORT_SYMBOL_GPL(arch_is_gunyah_guest); fn) #define GH_HYPERCALL_HYP_IDENTIFY GH_HYPERCALL(0x8000) +#define GH_HYPERCALL_BELL_SEND GH_HYPERCALL(0x8012) +#define GH_HYPERCALL_BELL_SET_MASK GH_HYPERCALL(0x8015) #define GH_HYPERCALL_MSGQ_SEND GH_HYPERCALL(0x801B) #define GH_HYPERCALL_MSGQ_RECV GH_HYPERCALL(0x801C) #define GH_HYPERCALL_VCPU_RUN GH_HYPERCALL(0x8065) @@ -60,6 +62,29 @@ void gh_hypercall_hyp_identify(struct gh_hypercall_hyp_identify_resp *hyp_identi } EXPORT_SYMBOL_GPL(gh_hypercall_hyp_identify); +enum gh_error gh_hypercall_bell_send(u64 capid, u64 new_flags, u64 *old_flags) +{ + struct arm_smccc_res res; + + arm_smccc_1_1_hvc(GH_HYPERCALL_BELL_SEND, capid, new_flags, 0, &res); + + if (res.a0 == GH_ERROR_OK) + *old_flags = res.a1; + + return res.a0; +} +EXPORT_SYMBOL_GPL(gh_hypercall_bell_send); + +enum gh_error gh_hypercall_bell_set_mask(u64 capid, u64 enable_mask, u64 ack_mask) +{ + struct arm_smccc_res res; + + arm_smccc_1_1_hvc(GH_HYPERCALL_BELL_SET_MASK, capid, enable_mask, ack_mask, 0, &res); + + return res.a0; +} +EXPORT_SYMBOL_GPL(gh_hypercall_bell_set_mask); + enum gh_error gh_hypercall_msgq_send(u64 capid, size_t size, uintptr_t buff, int tx_flags, bool *ready) { diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h index c819df72d303..bfceaa7000e5 100644 --- a/include/linux/gunyah.h +++ b/include/linux/gunyah.h @@ -172,6 +172,9 @@ struct gh_hypercall_hyp_identify_resp { void gh_hypercall_hyp_identify(struct gh_hypercall_hyp_identify_resp *hyp_identity); +enum gh_error gh_hypercall_bell_send(u64 capid, u64 new_flags, u64 *old_flags); +enum gh_error gh_hypercall_bell_set_mask(u64 capid, u64 enable_mask, u64 ack_mask); + #define GH_HYPERCALL_MSGQ_TX_FLAGS_PUSH BIT(0) enum gh_error gh_hypercall_msgq_send(u64 capid, size_t size, uintptr_t buff, int tx_flags, From patchwork Tue Feb 14 21:26:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 653515 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F57BC05027 for ; Tue, 14 Feb 2023 21:52:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230054AbjBNVwf (ORCPT ); Tue, 14 Feb 2023 16:52:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229510AbjBNVwe (ORCPT ); Tue, 14 Feb 2023 16:52:34 -0500 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 65BB42B083; Tue, 14 Feb 2023 13:52:33 -0800 (PST) Received: from pps.filterd (m0279865.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 31EEfsJ9006217; Tue, 14 Feb 2023 21:26:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=qcppdkim1; bh=LI21iyaD6256tDvZGp/UY35e1DSgHlcFcXiwafqOl98=; b=eGT6/LcLe7+dgJ/jYXMz3BTeKYvMt/E92s4uRXtaHRGDVaQx45WRWRktbX2dvtW5ZfvR ZpcunSM7FTRleO2QMyd4XADZuDoPVxydhPb3jF4ucJEm0ynR5OjqkQBCNGPpBPKIC5vn DpwbtpqSrayJoWJGDKdXeNHs4KucQ1vyupkmXiLJ104GU8Zd4YqATl2nT9Ajk25sObia +xSkR6r3BFgdRHJ+KWTZFCzLtMv9A9pXsoIS7zvBNGvUzf4b7CKmT25tmpmcgeIVXiow T65eyQSBxhmLw650VUftjYml5H1iWpPOZm30vOniVvUPGMwz0lAsX5ynxjLqgKnYLg5N Mg== Received: from nasanppmta04.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3nr26u2hvk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:26:41 +0000 Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA04.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 31ELQeJJ003539 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Feb 2023 21:26:40 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Tue, 14 Feb 2023 13:26:39 -0800 From: Elliot Berman To: Alex Elder , Srinivas Kandagatla , Elliot Berman , Prakruthi Deepak Heragu , Jonathan Corbet CC: Murali Nalajala , Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Dmitry Baryshkov , Bjorn Andersson , "Konrad Dybcio" , Arnd Bergmann , "Greg Kroah-Hartman" , Rob Herring , Krzysztof Kozlowski , Bagas Sanjaya , Catalin Marinas , Jassi Brar , , , , , Subject: [PATCH v10 25/26] virt: gunyah: Add ioeventfd Date: Tue, 14 Feb 2023 13:26:31 -0800 Message-ID: <20230214212632.3329812-1-quic_eberman@quicinc.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230214211229.3239350-1-quic_eberman@quicinc.com> References: <20230214211229.3239350-1-quic_eberman@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01a.na.qualcomm.com (10.47.209.196) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: IUqAugtH67wi5-c5bKUDzgyR-nmm0FcP X-Proofpoint-ORIG-GUID: IUqAugtH67wi5-c5bKUDzgyR-nmm0FcP X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-14_15,2023-02-14_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 adultscore=0 mlxscore=0 spamscore=0 impostorscore=0 malwarescore=0 mlxlogscore=999 clxscore=1015 lowpriorityscore=0 bulkscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2302140184 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Allow userspace to attach an ioeventfd to an mmio address within the guest. Co-developed-by: Prakruthi Deepak Heragu Signed-off-by: Prakruthi Deepak Heragu Signed-off-by: Elliot Berman --- Documentation/virt/gunyah/vm-manager.rst | 21 +++++ drivers/virt/gunyah/Kconfig | 9 ++ drivers/virt/gunyah/Makefile | 1 + drivers/virt/gunyah/gunyah_ioeventfd.c | 113 +++++++++++++++++++++++ include/uapi/linux/gunyah.h | 24 +++++ 5 files changed, 168 insertions(+) create mode 100644 drivers/virt/gunyah/gunyah_ioeventfd.c diff --git a/Documentation/virt/gunyah/vm-manager.rst b/Documentation/virt/gunyah/vm-manager.rst index 55d4fb466644..62d628f810f0 100644 --- a/Documentation/virt/gunyah/vm-manager.rst +++ b/Documentation/virt/gunyah/vm-manager.rst @@ -170,3 +170,24 @@ the irqfd.label. GH_IRQFD_LEVEL configures the corresponding doorbell to behave like a level triggered interrupt. + +Type: "ioeventfd" +^^^^^^^^^^^^^^^^^ + +:: + + struct gh_fn_ioeventfd_arg { + __u64 datamatch; + __u64 addr; /* legal mmio address */ + __u32 len; /* 1, 2, 4, or 8 bytes */ + __s32 fd; + #define GH_IOEVENTFD_DATAMATCH (1UL << 0) + __u32 flags; + }; + +Attaches an ioeventfd to a legal mmio address within the guest. A guest write +in the registered address will signal the provided event instead of triggering +an exit on the GH_VCPU_RUN ioctl. + +If datamatch flag is set, the event will be signaled only if the written value +to the registered address is equal to datamatch in struct gh_fn_ioeventfd_arg. diff --git a/drivers/virt/gunyah/Kconfig b/drivers/virt/gunyah/Kconfig index 2cde24d429d1..bd8e31184962 100644 --- a/drivers/virt/gunyah/Kconfig +++ b/drivers/virt/gunyah/Kconfig @@ -35,3 +35,12 @@ config GUNYAH_IRQFD on Gunyah virtual machine. Say Y/M here if unsure and you want to support Gunyah VMMs. + +config GUNYAH_IOEVENTFD + tristate "Gunyah ioeventfd interface" + depends on GUNYAH + help + Enable kernel support for creating ioeventfds which can alert userspace + when a Gunyah virtual machine accesses a memory address. + + Say Y/M here if unsure and you want to support Gunyah VMMs. diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile index 6cf756bfa3c2..7347b1470491 100644 --- a/drivers/virt/gunyah/Makefile +++ b/drivers/virt/gunyah/Makefile @@ -8,3 +8,4 @@ obj-$(CONFIG_GUNYAH) += gunyah_rsc_mgr.o obj-$(CONFIG_GUNYAH_VCPU) += gunyah_vcpu.o obj-$(CONFIG_GUNYAH_IRQFD) += gunyah_irqfd.o +obj-$(CONFIG_GUNYAH_IOEVENTFD) += gunyah_ioeventfd.o diff --git a/drivers/virt/gunyah/gunyah_ioeventfd.c b/drivers/virt/gunyah/gunyah_ioeventfd.c new file mode 100644 index 000000000000..b1d1e2d80f60 --- /dev/null +++ b/drivers/virt/gunyah/gunyah_ioeventfd.c @@ -0,0 +1,113 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include + +struct gunyah_ioeventfd { + struct gh_vm_function_instance *f; + struct gh_vm_io_handler io_handler; + + struct eventfd_ctx *ctx; +}; + +static int gh_write_ioeventfd(struct gh_vm_io_handler *io_dev, u64 addr, u32 len, u64 data) +{ + struct gunyah_ioeventfd *iofd = container_of(io_dev, struct gunyah_ioeventfd, io_handler); + + eventfd_signal(iofd->ctx, 1); + return 0; +} + +static struct gh_vm_io_handler_ops io_ops = { + .write = gh_write_ioeventfd, +}; + +static long gunyah_ioeventfd_bind(struct gh_vm_function_instance *f) +{ + const struct gh_fn_ioeventfd_arg *args = f->argp; + struct eventfd_ctx *ctx = NULL; + struct gunyah_ioeventfd *iofd; + int ret; + + if (f->arg_size != sizeof(*args)) + return -EINVAL; + + /* must be natural-word sized, or 0 to ignore length */ + switch (args->len) { + case 0: + case 1: + case 2: + case 4: + case 8: + break; + default: + return -EINVAL; + } + + /* check for range overflow */ + if (args->addr + args->len < args->addr) + return -EINVAL; + + /* ioeventfd with no length can't be combined with DATAMATCH */ + if (!args->len && (args->flags & GH_IOEVENTFD_DATAMATCH)) + return -EINVAL; + + ctx = eventfd_ctx_fdget(args->fd); + if (IS_ERR(ctx)) + return PTR_ERR(ctx); + + iofd = kzalloc(sizeof(*iofd), GFP_KERNEL); + if (!iofd) { + ret = -ENOMEM; + goto err_eventfd; + } + + f->data = iofd; + iofd->f = f; + + iofd->ctx = ctx; + + if (args->flags & GH_IOEVENTFD_DATAMATCH) { + iofd->io_handler.datamatch = true; + iofd->io_handler.len = args->len; + iofd->io_handler.data = args->datamatch; + } + iofd->io_handler.addr = args->addr; + iofd->io_handler.ops = &io_ops; + + ret = gh_vm_add_io_handler(f->ghvm, &iofd->io_handler); + if (ret) + goto err_io_dev_add; + + return 0; + +err_io_dev_add: + kfree(iofd); +err_eventfd: + eventfd_ctx_put(ctx); + return ret; +} + +static void gunyah_ioevent_unbind(struct gh_vm_function_instance *f) +{ + struct gunyah_ioeventfd *iofd = f->data; + + eventfd_ctx_put(iofd->ctx); + gh_vm_remove_io_handler(iofd->f->ghvm, &iofd->io_handler); + kfree(iofd); +} + +DECLARE_GUNYAH_VM_FUNCTION_INIT(ioeventfd, GH_FN_IOEVENTFD, + gunyah_ioeventfd_bind, gunyah_ioevent_unbind); +MODULE_DESCRIPTION("Gunyah ioeventfds"); +MODULE_LICENSE("GPL"); diff --git a/include/uapi/linux/gunyah.h b/include/uapi/linux/gunyah.h index ce2ccb71993f..63b6d275a64f 100644 --- a/include/uapi/linux/gunyah.h +++ b/include/uapi/linux/gunyah.h @@ -78,6 +78,12 @@ struct gh_vm_dtb_config { */ #define GH_FN_IRQFD 2 +/** + * GH_FN_IOEVENTFD - register ioeventfd to trigger when VM faults on parameter + * gh_fn_desc usage: fill arg with gh_fn_ioeventfd_arg + */ +#define GH_FN_IOEVENTFD 3 + #define GH_FN_MAX_ARG_SIZE 256 /** @@ -94,6 +100,24 @@ struct gh_fn_irqfd_arg { __u32 reserved; }; +/** + * struct gh_fn_ioeventfd_arg - Arguments to create an ioeventfd function + * @datamatch: data used when GH_IOEVENTFD_DATAMATCH is set + * @addr: Address in guest memory + * @len: Length of access + * @fd: When ioeventfd is matched, this eventfd is written + * @flags: See Documentation/virt/gunyah/vm-manager.rst for flag usage. + */ +struct gh_fn_ioeventfd_arg { + __u64 datamatch; + __u64 addr; /* legal mmio address */ + __u32 len; /* 1, 2, 4, or 8 bytes; or 0 to ignore length */ + __s32 fd; +#define GH_IOEVENTFD_DATAMATCH (1UL << 0) + __u32 flags; + __u32 reserved; +}; + /** * struct gh_fn_desc - Arguments to create a VM function * @type: Type of the function. See GH_FN_* macro for supported types