From patchwork Wed Nov 16 21:00:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 625326 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 778B0C41535 for ; Wed, 16 Nov 2022 21:04:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239136AbiKPVDN (ORCPT ); Wed, 16 Nov 2022 16:03:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44014 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238868AbiKPVBy (ORCPT ); Wed, 16 Nov 2022 16:01:54 -0500 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2060.outbound.protection.outlook.com [40.107.220.60]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 53C2764D3; Wed, 16 Nov 2022 13:00:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=C/nliwOKzUHBcQdGGL89sZAKhO/VfGoBF5yGLnm9wa0wdJz3jSiUOmIAnlo0rQWkGEebEqK66u6R3UQ+bbmqeR0pr+Xk8jFZF5LV6k05cjIFAM0DOgdX6RU/5bLDUbh8x9CVy4XNEBbRUhYZfdkc1sBQjdpNoyHC7y4I1blwGBZBfpxOJQgBThMRXNsufLrb05kG+1NTbJLp508/64Kf2Rc6K8z0qtIcmuDQ1TxIjyyOYmg3MecBlB2lCGLZ18tvBfE4wusVdh5kJZf1OibhrroyOD4J5R4AQSXGjPU8rn/16Uf8CkVhAJNxH+0i5V1NeXwSgUzTQaRHopPw1KTVIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=i/5IoMdzanZlihohIxZq3RVStRxJF7NBwPZO3xVT9Aw=; b=E+RIt2koyx1uIcE1D36njNU4Lg9bXuwwfBgMX9vyeVvjQPymnSZkns47ITr1VJq7QMlsXBIOT6GHtfpexFN5On6nKZcRUYPHUUZBW3n6VRzNimzauNQcWxCZABwHTrxa8iw1pjaNIjIe4bIVgDPlQL142XLiROSKDxuR0oL1X82hdXv2UNFlxdL56FUvThkmcXOtKRZDeekjyTm4aC3TB3IIIHhqu5e4Ov4U/CgOrGvTRRrDOoxuTvTOFAhki7PRtohPtifwyhp3+63qdSXsvPyiQikZANd0mWpRGiA/Um3WbV03ZlNm2/1zzt23WBgW+0Te/OKgYwdHkm/T0mjTiQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=i/5IoMdzanZlihohIxZq3RVStRxJF7NBwPZO3xVT9Aw=; b=J/Xu12puPI4+TtguG1rQOieFSPJ5B/p2XVtHM4h0C71VG7O2qrmOg2rVF3VC2sicJMmauUmsuoyMzQ/upMbyiyzbmROYU0Sil2dU55uIPQUjJpmzE5BClwhDV7lOfCOVzkbpaWkg+4y2+U6RA3sL/imV5yc6rzsOAq3xd33fi0+Zsja7F18d+zrHSAvrjLNkWCxOYcyott1tQDij4rZgkcnNlPnjuCyBQreHEwPr/g2pQhgJ4NsDZ6I2avBxoPeTCR6AtGwI6EGrCwo77LL7osSyfNJ0BkYSnVgSSK2rF0zyblznh85XEa4IyMI4bvhuDFFMHgb7ZyiwAHisayWWFQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by CY8PR12MB7682.namprd12.prod.outlook.com (2603:10b6:930:85::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5813.18; Wed, 16 Nov 2022 21:00:57 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a%7]) with mapi id 15.20.5813.017; Wed, 16 Nov 2022 21:00:56 +0000 From: Jason Gunthorpe To: bpf@vger.kernel.org, Jonathan Corbet , David Woodhouse , iommu@lists.linux.dev, Joerg Roedel , Kevin Tian , linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, llvm@lists.linux.dev, Nathan Chancellor , Nick Desaulniers , Miguel Ojeda , Robin Murphy , Shuah Khan , Suravee Suthikulpanit , Tom Rix , Will Deacon Cc: Anthony Krowiak , Alex Williamson , Bagas Sanjaya , Lu Baolu , Chaitanya Kulkarni , Cornelia Huck , Daniel Jordan , David Gibson , Eric Auger , Eric Farman , Jason Wang , Jean-Philippe Brucker , Jason Herne , Joao Martins , kvm@vger.kernel.org, Lixiao Yang , Matthew Rosato , "Michael S. Tsirkin" , Nicolin Chen , Halil Pasic , Niklas Schnelle , Shameerali Kolothum Thodi , Yi Liu , Keqian Zhu Subject: [PATCH v5 02/19] iommu: Add device-centric DMA ownership interfaces Date: Wed, 16 Nov 2022 17:00:38 -0400 Message-Id: <2-v5-4001c2997bd0+30c-iommufd_jgg@nvidia.com> In-Reply-To: <0-v5-4001c2997bd0+30c-iommufd_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR02CA0116.namprd02.prod.outlook.com (2603:10b6:208:35::21) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|CY8PR12MB7682:EE_ X-MS-Office365-Filtering-Correlation-Id: ec1cd239-51bf-4ef1-6590-08dac815a7ed X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: gqtLpxZckgvsM9KZS0iDDBHLFwvn7ytrIHPxpdOnj0mQ1tUwDR+URItxXpWh1BjYTnC1OrRqO3dhWgOy9Elouhj3yzlfN7u2LJm90fWNZ9jxIRqOHDlJSF8sTWr+cUDXYXEh8jA0Z6jyLYXBLSh80ihev75txP6trbktUXY3Fc7OqO0eZM1jjrJmwrS3C338zVQtZdlWRZ4x5EqsjlKDlmTF06KWfQNnQDMfYv0JwsrMgDXw91YxD0qM7H4Ps7znw15xJnHLBdldIjCgu6TOBaNSFsut7uMDR+zZ+nlFD3mJeI/rIGBM6hHN9AD33mtapKNybs0l+vFPsM+mODIpgWws9hZUHcCytavnnzchSdqd09IzuQH5T8OeJqIFUTzQyVes4BOwQCRAGfcVHt/60NLzxcGoskD4tIYw1/XpbrikyiZoluLHsW4xjufNGs5IiwrkcaUSA5l61jHA9/UUpzP97rbIQ9jTXNWuXDvJ3nFyyVfVhpEA0dlXrDNwwR2/AOl5AO+Clv58eLFm/qNHtEtAAVBGwMxGg+K2P6goFsnzdBQJUP9x8HTnGW8xr24zGt3XvPwoKcIxAIBZDAfEqQvhsxmZyeexgaiWYvued00gi8Vs0JDJtdKZGgZPqAI1eC4Rv84NswxfehOFzO6VBjlrLGBUdZvcdE30CtiiRgMrdD1TelXNJYvAS0IVO95JChGkafpE39q5t2sfj2jUY+p5wawQRmQqTlhQDz7m1B383Udf7I7BqxZHlsPv7Ekm1TDu3LMfqVHKNyx9jvVfhA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:LV2PR12MB5869.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(4636009)(39860400002)(346002)(136003)(396003)(366004)(376002)(451199015)(38100700002)(2616005)(86362001)(921005)(110136005)(7406005)(6666004)(478600001)(6486002)(7416002)(8936002)(66476007)(8676002)(4326008)(54906003)(66556008)(66946007)(316002)(41300700001)(83380400001)(2906002)(6506007)(186003)(5660300002)(6512007)(36756003)(26005)(4216001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: +bfF6etGvMY6tDqd/t6iXbjw7VPCukrjkUNUQjKpNYF3z6te5niHCyCEc7jv+m+m2mrBA0VCgr5mCF3m9ITgxZsV6Z8cYDwxpshzi6fTtMEGVhBoY/6UZxrTJBPSeHb21kNT8XSaLql793f2K2wy5VKrriRvNF9doV1EWtPV0vwHw8A4+pzW6vioNQu7bh26udhmlPBgVaXOLQVKiy9QItvKRAi+A8ZmUD6yE3jTdwvkaWG0VBR4bAMtx5Z1+N2OgXaEfFwRVZbLTK7h8J3B8ix35rtwjNPbPcC3EzjxCkWj83oVBOoasFjTo+EN4Ly1zmwdh2CDfZnsbJKAoz/jjjwUK68XRr/CtmWLBT7BAfrfLup/JcxliIhZqg0vZm1UaoWHfTsDwBriuTfTVGjAceIamavKtbo25+uhnoVyysQxALhNF+ASWHT2O6tK+60X4TMoEcGrpBd9buJOICnPNIZ4r7b4SvOiuwzHIcuz8VXcK2/UAWuxYs29T7uzqAoLC3bY29wTOGO8Fy007uvvYve7L9+Vyeg2q35dQbvCcvlVX2krqOAb3I6ORP/0ddA/atTY38aNpx2Nnytyil8aKQp28VnvSLlOtAMSw8T/GAHfJIw2yKqyYlF4V1Oyh0atqgxDDHV8PNY3alfng67z3ivMLT10qsOxsYvYHMWwzgouHTzzb6ynVCiAbKyO7enARQTasC1TtPwWbJFuKriJjsVaveVx/38rXATlHLnX08jilg8utLkzIZGBuXLAHAgSXHiseMrjpqo0xXjRfHjpRNA2LWV0XSNkpMK/YQcRLtXV5vG9nKpuP7gHF+hZmMQ/7Agwl5y2eywygOSR31CzUPgpdGTXB5TFGm4Q7rtnCSvtOQNHU0bJJOrWSDwXncE+pQI7O8CrYxPjRzV0nE/H+xSoMsHSujrnezAdqEi0B+jApaMgxwgXcDDZmidrgGSLE3gC4fnxa9PxfPuRHGYMd9qT501Oa5CSEMlHVQaRrP8j179USMa7FC/qaaIxKUG6iOFhJ2Fr1AbSaTW5gXz7SAfrPmJbX9vpHOVv7+S6EWeKUQ2pUdUSZi1oiH/WlBX0uRGIIgQXRoRWVYHxQENUV5qivD4KKInn2d6jwz190Fmhitc7mrbkbnm/BkjcZGrrtRAeEWgYpNGPx9UozG5gWMHJwF/DDbYs0O+FaHZuRwIEPklbBm4Y0x8KQLEKA9QKTRQIhZ2hhCjJ8CE2NNa26INYn33FrsSsip8dI5+/KOlrube799GQiw8dRZLKkobN6FiNi9QS9XcqEFbeqODHHyMtApprMz6gDtnnuFsI/yjBt604yX39YMMI+HmXZS/VUngeM++oaGEG3Zto5PiiISYrmX8VL0+9azI2/bW2Qz1xuVRETsle/eaoZLAdL+zWuFG6BEjbC35Sp08FxXxE40ze2jQatuQJvZlY6KWy3QwAKOFlj/3zgneH+GDiJDKN1QfcygLGfiYVbNSWDNhbDE6ZitIsEy4Y5X3Of5kLXR7TbnMFl8ay+PY0rACl6ChMrkfDdfbHcQaTxfqb40RdRK1Y2cEpAYr8jypnCn0tlVQ= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: ec1cd239-51bf-4ef1-6590-08dac815a7ed X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Nov 2022 21:00:56.7004 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: dZh/onY7Q3IyZQxJKknIesshoPTQyiNNpowHXP8agmjDim5vMKSr6EorVl3iQPbn X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7682 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org From: Lu Baolu These complement the group interfaces used by VFIO and are for use by iommufd. The main difference is that multiple devices in the same group can all share the ownership by passing the same ownership pointer. Move the common code into shared functions. Tested-by: Nicolin Chen Tested-by: Yi Liu Tested-by: Lixiao Yang Tested-by: Matthew Rosato Reviewed-by: Kevin Tian Reviewed-by: Eric Auger Signed-off-by: Lu Baolu Signed-off-by: Jason Gunthorpe --- drivers/iommu/iommu.c | 121 +++++++++++++++++++++++++++++++++--------- include/linux/iommu.h | 12 +++++ 2 files changed, 107 insertions(+), 26 deletions(-) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 6ca377f4fbf9e9..d69ebba81bebd8 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -3108,41 +3108,49 @@ static int __iommu_group_alloc_blocking_domain(struct iommu_group *group) return 0; } +static int __iommu_take_dma_ownership(struct iommu_group *group, void *owner) +{ + int ret; + + if ((group->domain && group->domain != group->default_domain) || + !xa_empty(&group->pasid_array)) + return -EBUSY; + + ret = __iommu_group_alloc_blocking_domain(group); + if (ret) + return ret; + ret = __iommu_group_set_domain(group, group->blocking_domain); + if (ret) + return ret; + + group->owner = owner; + group->owner_cnt++; + return 0; +} + /** * iommu_group_claim_dma_owner() - Set DMA ownership of a group * @group: The group. * @owner: Caller specified pointer. Used for exclusive ownership. * - * This is to support backward compatibility for vfio which manages - * the dma ownership in iommu_group level. New invocations on this - * interface should be prohibited. + * This is to support backward compatibility for vfio which manages the dma + * ownership in iommu_group level. New invocations on this interface should be + * prohibited. Only a single owner may exist for a group. */ int iommu_group_claim_dma_owner(struct iommu_group *group, void *owner) { int ret = 0; + if (WARN_ON(!owner)) + return -EINVAL; + mutex_lock(&group->mutex); if (group->owner_cnt) { ret = -EPERM; goto unlock_out; - } else { - if ((group->domain && group->domain != group->default_domain) || - !xa_empty(&group->pasid_array)) { - ret = -EBUSY; - goto unlock_out; - } - - ret = __iommu_group_alloc_blocking_domain(group); - if (ret) - goto unlock_out; - - ret = __iommu_group_set_domain(group, group->blocking_domain); - if (ret) - goto unlock_out; - group->owner = owner; } - group->owner_cnt++; + ret = __iommu_take_dma_ownership(group, owner); unlock_out: mutex_unlock(&group->mutex); @@ -3151,30 +3159,91 @@ int iommu_group_claim_dma_owner(struct iommu_group *group, void *owner) EXPORT_SYMBOL_GPL(iommu_group_claim_dma_owner); /** - * iommu_group_release_dma_owner() - Release DMA ownership of a group - * @group: The group. + * iommu_device_claim_dma_owner() - Set DMA ownership of a device + * @dev: The device. + * @owner: Caller specified pointer. Used for exclusive ownership. * - * Release the DMA ownership claimed by iommu_group_claim_dma_owner(). + * Claim the DMA ownership of a device. Multiple devices in the same group may + * concurrently claim ownership if they present the same owner value. Returns 0 + * on success and error code on failure */ -void iommu_group_release_dma_owner(struct iommu_group *group) +int iommu_device_claim_dma_owner(struct device *dev, void *owner) { - int ret; + struct iommu_group *group = iommu_group_get(dev); + int ret = 0; + + if (!group) + return -ENODEV; + if (WARN_ON(!owner)) + return -EINVAL; mutex_lock(&group->mutex); + if (group->owner_cnt) { + if (group->owner != owner) { + ret = -EPERM; + goto unlock_out; + } + group->owner_cnt++; + goto unlock_out; + } + + ret = __iommu_take_dma_ownership(group, owner); +unlock_out: + mutex_unlock(&group->mutex); + iommu_group_put(group); + + return ret; +} +EXPORT_SYMBOL_GPL(iommu_device_claim_dma_owner); + +static void __iommu_release_dma_ownership(struct iommu_group *group) +{ + int ret; + if (WARN_ON(!group->owner_cnt || !group->owner || !xa_empty(&group->pasid_array))) - goto unlock_out; + return; group->owner_cnt = 0; group->owner = NULL; ret = __iommu_group_set_domain(group, group->default_domain); WARN(ret, "iommu driver failed to attach the default domain"); +} -unlock_out: +/** + * iommu_group_release_dma_owner() - Release DMA ownership of a group + * @dev: The device + * + * Release the DMA ownership claimed by iommu_group_claim_dma_owner(). + */ +void iommu_group_release_dma_owner(struct iommu_group *group) +{ + mutex_lock(&group->mutex); + __iommu_release_dma_ownership(group); mutex_unlock(&group->mutex); } EXPORT_SYMBOL_GPL(iommu_group_release_dma_owner); +/** + * iommu_device_release_dma_owner() - Release DMA ownership of a device + * @group: The device. + * + * Release the DMA ownership claimed by iommu_device_claim_dma_owner(). + */ +void iommu_device_release_dma_owner(struct device *dev) +{ + struct iommu_group *group = iommu_group_get(dev); + + mutex_lock(&group->mutex); + if (group->owner_cnt > 1) + group->owner_cnt--; + else + __iommu_release_dma_ownership(group); + mutex_unlock(&group->mutex); + iommu_group_put(group); +} +EXPORT_SYMBOL_GPL(iommu_device_release_dma_owner); + /** * iommu_group_dma_owner_claimed() - Query group dma ownership status * @group: The group. diff --git a/include/linux/iommu.h b/include/linux/iommu.h index a09fd32d8cc273..1690c334e51631 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -707,6 +707,9 @@ int iommu_group_claim_dma_owner(struct iommu_group *group, void *owner); void iommu_group_release_dma_owner(struct iommu_group *group); bool iommu_group_dma_owner_claimed(struct iommu_group *group); +int iommu_device_claim_dma_owner(struct device *dev, void *owner); +void iommu_device_release_dma_owner(struct device *dev); + struct iommu_domain *iommu_sva_domain_alloc(struct device *dev, struct mm_struct *mm); int iommu_attach_device_pasid(struct iommu_domain *domain, @@ -1064,6 +1067,15 @@ static inline bool iommu_group_dma_owner_claimed(struct iommu_group *group) return false; } +static inline void iommu_device_release_dma_owner(struct device *dev) +{ +} + +static inline int iommu_device_claim_dma_owner(struct device *dev, void *owner) +{ + return -ENODEV; +} + static inline struct iommu_domain * iommu_sva_domain_alloc(struct device *dev, struct mm_struct *mm) {