From patchwork Thu Jun 3 08:50:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sven Peter X-Patchwork-Id: 454328 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC821C47097 for ; Thu, 3 Jun 2021 08:50:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D1E74613F2 for ; Thu, 3 Jun 2021 08:50:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229486AbhFCIwO (ORCPT ); Thu, 3 Jun 2021 04:52:14 -0400 Received: from new2-smtp.messagingengine.com ([66.111.4.224]:47549 "EHLO new2-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229754AbhFCIwO (ORCPT ); Thu, 3 Jun 2021 04:52:14 -0400 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailnew.nyi.internal (Postfix) with ESMTP id BD5D7580EC3; Thu, 3 Jun 2021 04:50:29 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Thu, 03 Jun 2021 04:50:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=6dgROPTG2FgZP QhdtR6kWJqhz8OvQLzcBN89UwKC3T4=; b=YZBoHdD6jqHwLwcR1DxQCnGR6tUwM n4gKQYSAEIf703dsJ7+6lC1pai5tjKhBAKZYUQ6kj2rky5h1+zmDx05jqstt19at tYycYwQazeAHd7zqDX3PV/IPTH+LSWEbHliJEYp98wmgzrZqliHbrHfwtjBdJkJy bgjE086l5oUUsprEFeTXW+3HuF5/rCatU3MctFESh0gwIAg+ZTQEZ81XCSxQaflq P36V8yegkT9X8mNSTs+A6fDOQbLw9DMchsBwMSyhg8NHD8bOg9T50LsNc0XEIqAn mOA7UEYGGiVd3Cj+J4MCq8dv91J854WUGJI9zjNCmLghtUyfjheTB2AJw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=6dgROPTG2FgZPQhdtR6kWJqhz8OvQLzcBN89UwKC3T4=; b=Evd9CJBa 2a338kJdBJd6FeLK4iOlVhZOmY4Jo+WSiBZOmxbE6rq/hxPkDNXNODUBCJr4S5Iw LVglMwZNGN/IO5pABevTIgcgTVN7aBDyx8fIVrMX/IgJLSJ/qxurufNfKNsU60Yf O5U7PUMwINJO5py3uapXisFi3zEc9P5CZspCHdDXkBq75eLLs9citLh/HaMmaByo kl9t9/PnaEkFwGesbJbjmxmi0hPPYI2xlkjVV2psMvW8fW1jj9UNjBiVDimlAgj8 Qc4E3+FEmkLdnZoRd8d5xiSkebZfYiRzSF4dFHBsW8oVIond4JXVV7vYXnz4sieI 8/J6hifWlwlIWg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdelledgtdeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefuvhgvnhcu rfgvthgvrhcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrfgrthhtvg hrnheptedvkeetleeuffffhfekteetffeggffgveehieelueefvddtueffveevlefhfeej necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepshhvvg hnsehsvhgvnhhpvghtvghrrdguvghv X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 3 Jun 2021 04:50:27 -0400 (EDT) From: Sven Peter To: Will Deacon , Robin Murphy , Joerg Roedel , Rob Herring Cc: Sven Peter , Arnd Bergmann , devicetree@vger.kernel.org, Hector Martin , linux-kernel@vger.kernel.org, Marc Zyngier , Mohamed Mediouni , Stan Skowronek , linux-arm-kernel@lists.infradead.org, Mark Kettenis , iommu@lists.linux-foundation.org, Alexander Graf Subject: [PATCH v3 1/3] iommu: io-pgtable: add DART pagetable format Date: Thu, 3 Jun 2021 10:50:01 +0200 Message-Id: <20210603085003.50465-2-sven@svenpeter.dev> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20210603085003.50465-1-sven@svenpeter.dev> References: <20210603085003.50465-1-sven@svenpeter.dev> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org Apple's DART iommu uses a pagetable format that shares some similarities with the ones already implemented by io-pgtable.c. Add a new format variant to support the required differences so that we don't have to duplicate the pagetable handling code. Signed-off-by: Sven Peter --- drivers/iommu/io-pgtable-arm.c | 62 ++++++++++++++++++++++++++++++++++ drivers/iommu/io-pgtable.c | 1 + include/linux/io-pgtable.h | 7 ++++ 3 files changed, 70 insertions(+) diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c index 87def58e79b5..1dd5c45b4b5b 100644 --- a/drivers/iommu/io-pgtable-arm.c +++ b/drivers/iommu/io-pgtable-arm.c @@ -127,6 +127,9 @@ #define ARM_MALI_LPAE_MEMATTR_IMP_DEF 0x88ULL #define ARM_MALI_LPAE_MEMATTR_WRITE_ALLOC 0x8DULL +#define APPLE_DART_PTE_PROT_NO_WRITE (1<<7) +#define APPLE_DART_PTE_PROT_NO_READ (1<<8) + /* IOPTE accessors */ #define iopte_deref(pte,d) __va(iopte_to_paddr(pte, d)) @@ -381,6 +384,15 @@ static arm_lpae_iopte arm_lpae_prot_to_pte(struct arm_lpae_io_pgtable *data, { arm_lpae_iopte pte; + if (data->iop.fmt == ARM_APPLE_DART) { + pte = 0; + if (!(prot & IOMMU_WRITE)) + pte |= APPLE_DART_PTE_PROT_NO_WRITE; + if (!(prot & IOMMU_READ)) + pte |= APPLE_DART_PTE_PROT_NO_READ; + return pte; + } + if (data->iop.fmt == ARM_64_LPAE_S1 || data->iop.fmt == ARM_32_LPAE_S1) { pte = ARM_LPAE_PTE_nG; @@ -1043,6 +1055,51 @@ arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie) return NULL; } +static struct io_pgtable * +apple_dart_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie) +{ + struct arm_lpae_io_pgtable *data; + int i; + + if (cfg->oas > 36) + return NULL; + + data = arm_lpae_alloc_pgtable(cfg); + if (!data) + return NULL; + + /* + * Apple's DART always requires three levels with the first level being + * stored in four MMIO registers. We always concatenate the first and + * second level so that we only have to setup the MMIO registers once. + * This results in an effective two level pagetable. + */ + if (data->start_level < 1) + return NULL; + if (data->start_level == 1 && data->pgd_bits > 2) + return NULL; + if (data->start_level > 1) + data->pgd_bits = 0; + data->start_level = 2; + cfg->apple_dart_cfg.n_ttbrs = 1 << data->pgd_bits; + data->pgd_bits += data->bits_per_level; + + data->pgd = __arm_lpae_alloc_pages(ARM_LPAE_PGD_SIZE(data), GFP_KERNEL, + cfg); + if (!data->pgd) + goto out_free_data; + + for (i = 0; i < cfg->apple_dart_cfg.n_ttbrs; ++i) + cfg->apple_dart_cfg.ttbr[i] = + virt_to_phys(data->pgd + i * ARM_LPAE_GRANULE(data)); + + return &data->iop; + +out_free_data: + kfree(data); + return NULL; +} + struct io_pgtable_init_fns io_pgtable_arm_64_lpae_s1_init_fns = { .alloc = arm_64_lpae_alloc_pgtable_s1, .free = arm_lpae_free_pgtable, @@ -1068,6 +1125,11 @@ struct io_pgtable_init_fns io_pgtable_arm_mali_lpae_init_fns = { .free = arm_lpae_free_pgtable, }; +struct io_pgtable_init_fns io_pgtable_apple_dart_init_fns = { + .alloc = apple_dart_alloc_pgtable, + .free = arm_lpae_free_pgtable, +}; + #ifdef CONFIG_IOMMU_IO_PGTABLE_LPAE_SELFTEST static struct io_pgtable_cfg *cfg_cookie __initdata; diff --git a/drivers/iommu/io-pgtable.c b/drivers/iommu/io-pgtable.c index 6e9917ce980f..fd8e6bd6caf9 100644 --- a/drivers/iommu/io-pgtable.c +++ b/drivers/iommu/io-pgtable.c @@ -20,6 +20,7 @@ io_pgtable_init_table[IO_PGTABLE_NUM_FMTS] = { [ARM_64_LPAE_S1] = &io_pgtable_arm_64_lpae_s1_init_fns, [ARM_64_LPAE_S2] = &io_pgtable_arm_64_lpae_s2_init_fns, [ARM_MALI_LPAE] = &io_pgtable_arm_mali_lpae_init_fns, + [ARM_APPLE_DART] = &io_pgtable_apple_dart_init_fns, #endif #ifdef CONFIG_IOMMU_IO_PGTABLE_ARMV7S [ARM_V7S] = &io_pgtable_arm_v7s_init_fns, diff --git a/include/linux/io-pgtable.h b/include/linux/io-pgtable.h index 4d40dfa75b55..a4bfac7f85f7 100644 --- a/include/linux/io-pgtable.h +++ b/include/linux/io-pgtable.h @@ -16,6 +16,7 @@ enum io_pgtable_fmt { ARM_V7S, ARM_MALI_LPAE, AMD_IOMMU_V1, + ARM_APPLE_DART, IO_PGTABLE_NUM_FMTS, }; @@ -136,6 +137,11 @@ struct io_pgtable_cfg { u64 transtab; u64 memattr; } arm_mali_lpae_cfg; + + struct { + u64 ttbr[4]; + u32 n_ttbrs; + } apple_dart_cfg; }; }; @@ -246,5 +252,6 @@ extern struct io_pgtable_init_fns io_pgtable_arm_64_lpae_s2_init_fns; extern struct io_pgtable_init_fns io_pgtable_arm_v7s_init_fns; extern struct io_pgtable_init_fns io_pgtable_arm_mali_lpae_init_fns; extern struct io_pgtable_init_fns io_pgtable_amd_iommu_v1_init_fns; +extern struct io_pgtable_init_fns io_pgtable_apple_dart_init_fns; #endif /* __IO_PGTABLE_H */ From patchwork Thu Jun 3 08:50:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sven Peter X-Patchwork-Id: 453510 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B3B6C47096 for ; Thu, 3 Jun 2021 08:50:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6D0C9613F2 for ; Thu, 3 Jun 2021 08:50:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229850AbhFCIwR (ORCPT ); Thu, 3 Jun 2021 04:52:17 -0400 Received: from new2-smtp.messagingengine.com ([66.111.4.224]:51425 "EHLO new2-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229814AbhFCIwQ (ORCPT ); Thu, 3 Jun 2021 04:52:16 -0400 Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 1700A580F08; Thu, 3 Jun 2021 04:50:32 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Thu, 03 Jun 2021 04:50:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=gGQl9SRxrhirr YTrU4b/HJo5lcf3YI1dsRlqz4epgR8=; b=GQyTi/dHgPLwCAtvecYogDZ3jMN+u 3P5m5hN5SMDYk76wbSeqZfdCwWX+aSKWc/aiaq1sT6ad0Y9ueNznZl8UAhtEfzJF 1K09MoJsF+ByCTav2nVB40yaoW1siaGgl081YcColXrZzwW3LY/YUDpN1PZfFVFp ke/ky84YeiL3guBP5m4kcLCSw+FCyZx1DkhoqUPTmCHcVSBAaE2U5J9yDgWhE9ek Ig4gJHRNjyarwZjTFmDI1Eymdqc1oWPWVHYzUdJV0tA77RC3w+b3jMC+7Nkn5F4C lCxTHVTlZJUj7qiGpMwWJlf4EW+LSX2Px53jwOQpBp4WU+LIlZYQAALDw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=gGQl9SRxrhirrYTrU4b/HJo5lcf3YI1dsRlqz4epgR8=; b=qpHKcacl OC+UXWdxt2zLAB+FZjA60l5fIOqL7BpEqnVSdeUWTvIGkz+jz9yS97w6JEaJgTCO hrPffxQSIRafemdLJUuVauhqXws1NqbU00le9or//JLXo2aPNhdsRSftRY7mjpom Hd1aTqsEyIE6JBHMh2T4P1Xgyrwg1QowVi0fzdMNLfBVCe3+BAz85EzNXlYZbGnq LRjBCbuPqPW3s2hjQ5XPpNZNhadDe/Qt5pG/7w8U8ODmCNp+t06D4hW8EI5CdiqD Z4cYMdpnNTRE+1nr/EmaUdaDDXYWyFfk0iKn9HpWwD1IMdThQGVe0ADKg30HWbZS /HTXhXN0OyUHxQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdelledgtdejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefuvhgvnhcu rfgvthgvrhcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrfgrthhtvg hrnhephfefffejvdekleeitdffiefhhfffveffieejteefhedutdfgffeftdejhedtteeh necuffhomhgrihhnpeguvghvihgtvghtrhgvvgdrohhrghenucevlhhushhtvghrufhiii gvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsvhgvnhesshhvvghnphgvthgvrhdr uggvvh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 3 Jun 2021 04:50:29 -0400 (EDT) From: Sven Peter To: Will Deacon , Robin Murphy , Joerg Roedel , Rob Herring Cc: Sven Peter , Arnd Bergmann , devicetree@vger.kernel.org, Hector Martin , linux-kernel@vger.kernel.org, Marc Zyngier , Mohamed Mediouni , Stan Skowronek , linux-arm-kernel@lists.infradead.org, Mark Kettenis , iommu@lists.linux-foundation.org, Alexander Graf Subject: [PATCH v3 2/3] dt-bindings: iommu: add DART iommu bindings Date: Thu, 3 Jun 2021 10:50:02 +0200 Message-Id: <20210603085003.50465-3-sven@svenpeter.dev> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20210603085003.50465-1-sven@svenpeter.dev> References: <20210603085003.50465-1-sven@svenpeter.dev> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org DART (Device Address Resolution Table) is the iommu found on Apple ARM SoCs such as the M1. Signed-off-by: Sven Peter Reviewed-by: Rob Herring --- .../devicetree/bindings/iommu/apple,dart.yaml | 81 +++++++++++++++++++ MAINTAINERS | 6 ++ 2 files changed, 87 insertions(+) create mode 100644 Documentation/devicetree/bindings/iommu/apple,dart.yaml diff --git a/Documentation/devicetree/bindings/iommu/apple,dart.yaml b/Documentation/devicetree/bindings/iommu/apple,dart.yaml new file mode 100644 index 000000000000..db21ca07d121 --- /dev/null +++ b/Documentation/devicetree/bindings/iommu/apple,dart.yaml @@ -0,0 +1,81 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/iommu/apple,dart.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Apple DART IOMMU + +maintainers: + - Sven Peter + +description: |+ + Apple SoCs may contain an implementation of their Device Address + Resolution Table which provides a mandatory layer of address + translations for various masters. + + Each DART instance is capable of handling up to 16 different streams + with individual pagetables and page-level read/write protection flags. + + This DART IOMMU also raises interrupts in response to various + fault conditions. + +properties: + compatible: + const: apple,t8103-dart + + reg: + maxItems: 1 + + interrupts: + maxItems: 1 + + clocks: + description: + Reference to the gate clock phandle if required for this IOMMU. + Optional since not all IOMMUs are attached to a clock gate. + + '#iommu-cells': + const: 1 + description: + Has to be one. The single cell describes the stream id emitted by + a master to the IOMMU. + +required: + - compatible + - reg + - '#iommu-cells' + - interrupts + +additionalProperties: false + +examples: + - |+ + dart1: iommu@82f80000 { + compatible = "apple,t8103-dart"; + reg = <0x82f80000 0x4000>; + interrupts = <1 781 4>; + #iommu-cells = <1>; + }; + + master1 { + iommus = <&{/dart1} 0>; + }; + + - |+ + dart2a: iommu@82f00000 { + compatible = "apple,t8103-dart"; + reg = <0x82f00000 0x4000>; + interrupts = <1 781 4>; + #iommu-cells = <1>; + }; + dart2b: iommu@82f80000 { + compatible = "apple,t8103-dart"; + reg = <0x82f80000 0x4000>; + interrupts = <1 781 4>; + #iommu-cells = <1>; + }; + + master2 { + iommus = <&{/dart2a} 0>, <&{/dart2b} 1>; + }; diff --git a/MAINTAINERS b/MAINTAINERS index 673cadd5107a..4373d63f9ccf 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1240,6 +1240,12 @@ L: linux-input@vger.kernel.org S: Odd fixes F: drivers/input/mouse/bcm5974.c +APPLE DART IOMMU DRIVER +M: Sven Peter +L: iommu@lists.linux-foundation.org +S: Maintained +F: Documentation/devicetree/bindings/iommu/apple,dart.yaml + APPLE SMC DRIVER M: Henrik Rydberg L: linux-hwmon@vger.kernel.org From patchwork Thu Jun 3 08:50:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sven Peter X-Patchwork-Id: 454327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A9F6C47082 for ; Thu, 3 Jun 2021 08:50:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0FDF4613E3 for ; Thu, 3 Jun 2021 08:50:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229922AbhFCIwV (ORCPT ); Thu, 3 Jun 2021 04:52:21 -0400 Received: from new2-smtp.messagingengine.com ([66.111.4.224]:51709 "EHLO new2-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229754AbhFCIwT (ORCPT ); Thu, 3 Jun 2021 04:52:19 -0400 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailnew.nyi.internal (Postfix) with ESMTP id 649F1580F12; Thu, 3 Jun 2021 04:50:34 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Thu, 03 Jun 2021 04:50:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=aHTLORIJnWP6g XjyjoxcNazEa1yBNVAq2jyuIskc8tU=; b=cXnbugboHoNYufAxIMD0Arnq4BzXp 5MfcpmFbttJdy+7JpIcv3d6shl5uxdcdMK7ZOMR1UsRYYmYq+pqxiQ5usLD/y77o DxoLmjEKDiObMGqc7viI7HjstAK4q7yoADz2mwyytey5vXio/NgP7ec+xQ/ZRedj n22H8std4yv4PuOEib5AYnmawkFcWUS+fTGl9G1lhNwWjKhbVbXj1Z831ddace6E Lb7F9P6HJq4gOdoTg+r5h8y4svqOwCAFkWVbcIUhJzH3cRucEnJGsRgSmph6gmDH /2znWSMrhi+Ft4MhaZ2fd6DchBLuI5PJmS38QTbN98y6ryWlRDbD+B2qQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=aHTLORIJnWP6gXjyjoxcNazEa1yBNVAq2jyuIskc8tU=; b=B5KA27Ho Zdrsv8ruYaSUj/WvvaFHFx6WL8TfK1NwiJdZM0rW6+wsolM/Lx9tbuZV6iH8MYLz fi5hSDzc5seijFldTL06UZAt+AZsnqGA/jjB0zcvH2byYd6r4h/T+6S1ckf8ovNY zOaNTzSQJX7g1AjUzei4D/pTUqufb5A3i+H5QESnq2IL3GOu+m1SOab3wDq6PZJc PbE0aXNebvMPWq0fsQs0aGi+l8UsCfjlNNCfqgB+6uwzw9vv0XinH9u8Rg6YXTMW 23omFf0GBLKGXxCFA7wP6OM49fpobMXHeOnb+LBMVQriNQUo99Wde6wwYvxa5oSo Ptvvy2fYjlL7BA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdelledgtdeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefuvhgvnhcu rfgvthgvrhcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrfgrthhtvg hrnheptedvkeetleeuffffhfekteetffeggffgveehieelueefvddtueffveevlefhfeej necuvehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepshhvvg hnsehsvhgvnhhpvghtvghrrdguvghv X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 3 Jun 2021 04:50:32 -0400 (EDT) From: Sven Peter To: Will Deacon , Robin Murphy , Joerg Roedel , Rob Herring Cc: Sven Peter , Arnd Bergmann , devicetree@vger.kernel.org, Hector Martin , linux-kernel@vger.kernel.org, Marc Zyngier , Mohamed Mediouni , Stan Skowronek , linux-arm-kernel@lists.infradead.org, Mark Kettenis , iommu@lists.linux-foundation.org, Alexander Graf Subject: [PATCH v3 3/3] iommu: dart: Add DART iommu driver Date: Thu, 3 Jun 2021 10:50:03 +0200 Message-Id: <20210603085003.50465-4-sven@svenpeter.dev> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20210603085003.50465-1-sven@svenpeter.dev> References: <20210603085003.50465-1-sven@svenpeter.dev> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org Apple's new SoCs use iommus for almost all peripherals. These Device Address Resolution Tables must be setup before these peripherals can act as DMA masters. Signed-off-by: Sven Peter --- MAINTAINERS | 1 + drivers/iommu/Kconfig | 15 + drivers/iommu/Makefile | 1 + drivers/iommu/apple-dart-iommu.c | 966 +++++++++++++++++++++++++++++++ 4 files changed, 983 insertions(+) create mode 100644 drivers/iommu/apple-dart-iommu.c diff --git a/MAINTAINERS b/MAINTAINERS index 4373d63f9ccf..cb9200ad05fe 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1245,6 +1245,7 @@ M: Sven Peter L: iommu@lists.linux-foundation.org S: Maintained F: Documentation/devicetree/bindings/iommu/apple,dart.yaml +F: drivers/iommu/apple-dart-iommu.c APPLE SMC DRIVER M: Henrik Rydberg diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index 1f111b399bca..87882c628b46 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -249,6 +249,21 @@ config SPAPR_TCE_IOMMU Enables bits of IOMMU API required by VFIO. The iommu_ops is not implemented as it is not necessary for VFIO. +config IOMMU_APPLE_DART + tristate "Apple DART IOMMU Support" + depends on ARM64 || (COMPILE_TEST && !GENERIC_ATOMIC64) + select IOMMU_API + select IOMMU_IO_PGTABLE + select IOMMU_IO_PGTABLE_LPAE + default ARCH_APPLE + help + Support for Apple DART (Device Address Resolution Table) IOMMUs + found in Apple ARM SoCs like the M1. + This IOMMU is required for most peripherals using DMA to access + the main memory. + + Say Y here if you are using an Apple SoC with a DART IOMMU. + # ARM IOMMU support config ARM_SMMU tristate "ARM Ltd. System MMU (SMMU) Support" diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile index c0fb0ba88143..8c813f0ebc54 100644 --- a/drivers/iommu/Makefile +++ b/drivers/iommu/Makefile @@ -29,3 +29,4 @@ obj-$(CONFIG_HYPERV_IOMMU) += hyperv-iommu.o obj-$(CONFIG_VIRTIO_IOMMU) += virtio-iommu.o obj-$(CONFIG_IOMMU_SVA_LIB) += iommu-sva-lib.o io-pgfault.o obj-$(CONFIG_SPRD_IOMMU) += sprd-iommu.o +obj-$(CONFIG_IOMMU_APPLE_DART) += apple-dart-iommu.o diff --git a/drivers/iommu/apple-dart-iommu.c b/drivers/iommu/apple-dart-iommu.c new file mode 100644 index 000000000000..2777852498de --- /dev/null +++ b/drivers/iommu/apple-dart-iommu.c @@ -0,0 +1,966 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Apple DART (Device Address Resolution Table) IOMMU driver + * + * Copyright (C) 2021 The Asahi Linux Contributors + * + * Based on arm/arm-smmu/arm-ssmu.c and arm/arm-smmu-v3/arm-smmu-v3.c + * Copyright (C) 2013 ARM Limited + * Copyright (C) 2015 ARM Limited + * and on exynos-iommu.c + * Copyright (c) 2011,2016 Samsung Electronics Co., Ltd. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define DART_MAX_STREAMS 16 +#define DART_MAX_TTBR 4 + +#define DART_STREAM_ALL 0xffff + +#define DART_PARAMS1 0x00 +#define DART_PARAMS_PAGE_SHIFT GENMASK(27, 24) + +#define DART_PARAMS2 0x04 +#define DART_PARAMS_BYPASS_SUPPORT BIT(0) + +#define DART_STREAM_COMMAND 0x20 +#define DART_STREAM_COMMAND_BUSY BIT(2) +#define DART_STREAM_COMMAND_INVALIDATE BIT(20) + +#define DART_STREAM_SELECT 0x34 + +#define DART_ERROR 0x40 +#define DART_ERROR_STREAM GENMASK(27, 24) +#define DART_ERROR_CODE GENMASK(23, 0) +#define DART_ERROR_FLAG BIT(31) +#define DART_ERROR_READ_FAULT BIT(4) +#define DART_ERROR_WRITE_FAULT BIT(3) +#define DART_ERROR_NO_PTE BIT(2) +#define DART_ERROR_NO_PMD BIT(1) +#define DART_ERROR_NO_TTBR BIT(0) + +#define DART_CONFIG 0x60 +#define DART_CONFIG_LOCK BIT(15) + +#define DART_STREAM_COMMAND_BUSY_TIMEOUT 100 + +#define DART_STREAM_REMAP 0x80 + +#define DART_ERROR_ADDR_HI 0x54 +#define DART_ERROR_ADDR_LO 0x50 + +#define DART_TCR(sid) (0x100 + 4 * (sid)) +#define DART_TCR_TRANSLATE_ENABLE BIT(7) +#define DART_TCR_BYPASS0_ENABLE BIT(8) +#define DART_TCR_BYPASS1_ENABLE BIT(12) + +#define DART_TTBR(sid, idx) (0x200 + 16 * (sid) + 4 * (idx)) +#define DART_TTBR_VALID BIT(31) +#define DART_TTBR_SHIFT 12 + +/* + * Private structure associated with each DART device. + * + * @dev: device struct + * @regs: mapped MMIO region + * @irq: interrupt number, can be shared with other DARTs + * @clks: clocks associated with this DART + * @num_clks: number of @clks + * @lock: lock for @used_sids and hardware operations involving this dart + * @used_sids: bitmap of streams attached to a domain + * @pgsize: pagesize supported by this DART + * @supports_bypass: indicates if this DART supports bypass mode + * @force_bypass: force bypass mode due to pagesize mismatch? + * @iommu: iommu core device + */ +struct apple_dart { + struct device *dev; + + void __iomem *regs; + + int irq; + struct clk_bulk_data *clks; + int num_clks; + + spinlock_t lock; + + u32 used_sids; + u32 pgsize; + + u32 supports_bypass : 1; + u32 force_bypass : 1; + u64 force_bypass_offset; + u64 force_bypass_len; + + struct iommu_device iommu; +}; + +/* + * This structure is used to identify a single stream attached to a domain. + * It's used as a list inside that domain to be able to attach multiple + * streams to a single domain. Since multiple devices can use a single stream + * it additionally keeps track of how many devices are represented by this + * stream. Once that number reaches zero it is detached from the IOMMU domain + * and all translations from this stream are disabled. + * + * @dart: DART instance to which this stream belongs + * @sid: stream id within the DART instance + * @num_devices: count of devices attached to this stream + * @stream_head: list head for the next stream + */ +struct apple_dart_stream { + struct apple_dart *dart; + u32 sid; + + u32 num_devices; + + struct list_head stream_head; +}; + +/* + * This structure is attached to each iommu domain handled by a DART. + * A single domain is used to represent a single virtual address spaces. + * It is always allocated together with a page table. + * + * Streams are the smallest units the DART hardware can differentiate. + * These are pointed to the page table of a domain whenever a device is + * attached to it. A single stream can only be assigned to a single domain. + * + * Devices are assigned to at least a single and sometimes multiple individual + * streams (using the iommus property in the device tree). Multiple devices + * can theoretically be represented by the same stream, though this is usually + * not the case. + * + * We only keep track of streams here and just count how many devices are + * represented by each stream. When the last device is removed the whole stream + * is removed from the domain. + * + * @dart: pointer to the DART instance + * @pgtbl_ops: pagetable ops allocated by io-pgtable + * @type: domain type IOMMU_DOMAIN_IDENTITY_{IDENTITY,DMA,UNMANAGED,BLOCKED} + * @streams: list of streams attached to this domain + * @lock: spinlock for operations involving the list of streams + * @domain: core iommu domain pointer + */ +struct apple_dart_domain { + struct apple_dart *dart; + struct io_pgtable_ops *pgtbl_ops; + + unsigned int type; + + struct list_head streams; + + spinlock_t lock; + + struct iommu_domain domain; +}; + +/* + * This structure is attached to devices with dev_iommu_priv_set() on of_xlate + * and contains a list of streams bound to this device as defined in the + * device tree. Multiple DART instances can be attached to a single device + * and each stream is identified by its stream id. + * It's usually reference by a pointer called *cfg. + * + * A dynamic array instead of a linked list is used here since in almost + * all cases a device will just be attached to a single stream and streams + * are never removed after they have been added. + * + * @num_streams: number of streams attached + * @streams: array of structs to identify attached streams and the device link + * to the iommu + */ +struct apple_dart_master_cfg { + int num_streams; + struct { + struct apple_dart *dart; + u32 sid; + + struct device_link *link; + } streams[]; +}; + +static struct platform_driver apple_dart_driver; +static const struct iommu_ops apple_dart_iommu_ops; +static const struct iommu_flush_ops apple_dart_tlb_ops; + +static struct apple_dart_domain *to_dart_domain(struct iommu_domain *dom) +{ + return container_of(dom, struct apple_dart_domain, domain); +} + +static void apple_dart_hw_enable_translation(struct apple_dart *dart, u16 sid) +{ + writel(DART_TCR_TRANSLATE_ENABLE, dart->regs + DART_TCR(sid)); +} + +static void apple_dart_hw_disable_dma(struct apple_dart *dart, u16 sid) +{ + writel(0, dart->regs + DART_TCR(sid)); +} + +static void apple_dart_hw_enable_bypass(struct apple_dart *dart, u16 sid) +{ + WARN_ON(!dart->supports_bypass); + writel(DART_TCR_BYPASS0_ENABLE | DART_TCR_BYPASS1_ENABLE, + dart->regs + DART_TCR(sid)); +} + +static void apple_dart_hw_set_ttbr(struct apple_dart *dart, u16 sid, u16 idx, + phys_addr_t paddr) +{ + writel(DART_TTBR_VALID | (paddr >> DART_TTBR_SHIFT), + dart->regs + DART_TTBR(sid, idx)); +} + +static void apple_dart_hw_clear_ttbr(struct apple_dart *dart, u16 sid, u16 idx) +{ + writel(0, dart->regs + DART_TTBR(sid, idx)); +} + +static void apple_dart_hw_clear_all_ttbrs(struct apple_dart *dart, u16 sid) +{ + int i; + + for (i = 0; i < 4; ++i) + apple_dart_hw_clear_ttbr(dart, sid, i); +} + +static int apple_dart_hw_stream_command(struct apple_dart *dart, u16 sid_bitmap, + u32 command) +{ + unsigned long flags; + int ret; + u32 command_reg; + + spin_lock_irqsave(&dart->lock, flags); + + writel(sid_bitmap, dart->regs + DART_STREAM_SELECT); + writel(command, dart->regs + DART_STREAM_COMMAND); + + ret = readl_poll_timeout_atomic( + dart->regs + DART_STREAM_COMMAND, command_reg, + !(command_reg & DART_STREAM_COMMAND_BUSY), 1, + DART_STREAM_COMMAND_BUSY_TIMEOUT); + + spin_unlock_irqrestore(&dart->lock, flags); + + if (ret) { + dev_err(dart->dev, + "busy bit did not clear after command %x for streams %x\n", + command, sid_bitmap); + return ret; + } + + return 0; +} + +static int apple_dart_hw_invalidate_tlb_global(struct apple_dart *dart) +{ + return apple_dart_hw_stream_command(dart, DART_STREAM_ALL, + DART_STREAM_COMMAND_INVALIDATE); +} + +static int apple_dart_hw_invalidate_tlb_stream(struct apple_dart *dart, u16 sid) +{ + return apple_dart_hw_stream_command(dart, 1 << sid, + DART_STREAM_COMMAND_INVALIDATE); +} + +static int apple_dart_hw_reset(struct apple_dart *dart) +{ + int sid; + u32 config; + + config = readl(dart->regs + DART_CONFIG); + if (config & DART_CONFIG_LOCK) { + dev_err(dart->dev, "DART is locked down until reboot: %08x\n", + config); + return -EINVAL; + } + + for (sid = 0; sid < DART_MAX_STREAMS; ++sid) { + apple_dart_hw_disable_dma(dart, sid); + apple_dart_hw_clear_all_ttbrs(dart, sid); + apple_dart_hw_enable_translation(dart, sid); + } + + /* restore stream identity map */ + writel(0x03020100, dart->regs + DART_STREAM_REMAP); + writel(0x07060504, dart->regs + DART_STREAM_REMAP + 4); + writel(0x0b0a0908, dart->regs + DART_STREAM_REMAP + 8); + writel(0x0f0e0d0c, dart->regs + DART_STREAM_REMAP + 12); + + /* clear any pending errors before the interrupt is unmasked */ + writel(readl(dart->regs + DART_ERROR), dart->regs + DART_ERROR); + + return apple_dart_hw_invalidate_tlb_global(dart); +} + +static void apple_dart_domain_flush_tlb(struct apple_dart_domain *domain) +{ + unsigned long flags; + struct apple_dart_stream *stream; + struct apple_dart *dart = domain->dart; + + if (!dart) + return; + + spin_lock_irqsave(&domain->lock, flags); + list_for_each_entry(stream, &domain->streams, stream_head) { + apple_dart_hw_invalidate_tlb_stream(stream->dart, stream->sid); + } + spin_unlock_irqrestore(&domain->lock, flags); +} + +static void apple_dart_flush_iotlb_all(struct iommu_domain *domain) +{ + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + + apple_dart_domain_flush_tlb(dart_domain); +} + +static void apple_dart_iotlb_sync(struct iommu_domain *domain, + struct iommu_iotlb_gather *gather) +{ + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + + apple_dart_domain_flush_tlb(dart_domain); +} + +static void apple_dart_iotlb_sync_map(struct iommu_domain *domain, + unsigned long iova, size_t size) +{ + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + + apple_dart_domain_flush_tlb(dart_domain); +} + +static void apple_dart_tlb_flush_all(void *cookie) +{ + struct apple_dart_domain *domain = cookie; + + apple_dart_domain_flush_tlb(domain); +} + +static void apple_dart_tlb_flush_walk(unsigned long iova, size_t size, + size_t granule, void *cookie) +{ + struct apple_dart_domain *domain = cookie; + + apple_dart_domain_flush_tlb(domain); +} + +static const struct iommu_flush_ops apple_dart_tlb_ops = { + .tlb_flush_all = apple_dart_tlb_flush_all, + .tlb_flush_walk = apple_dart_tlb_flush_walk, + .tlb_add_page = NULL, +}; + +static phys_addr_t apple_dart_iova_to_phys(struct iommu_domain *domain, + dma_addr_t iova) +{ + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + struct io_pgtable_ops *ops = dart_domain->pgtbl_ops; + + if (domain->type == IOMMU_DOMAIN_IDENTITY) + return iova; + if (!ops) + return -ENODEV; + + return ops->iova_to_phys(ops, iova); +} + +static int apple_dart_map(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) +{ + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + struct io_pgtable_ops *ops = dart_domain->pgtbl_ops; + + if (!ops) + return -ENODEV; + if (prot & IOMMU_MMIO) + return -EINVAL; + if (prot & IOMMU_NOEXEC) + return -EINVAL; + + return ops->map(ops, iova, paddr, size, prot, gfp); +} + +static size_t apple_dart_unmap(struct iommu_domain *domain, unsigned long iova, + size_t size, struct iommu_iotlb_gather *gather) +{ + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + struct io_pgtable_ops *ops = dart_domain->pgtbl_ops; + + if (!ops) + return 0; + + return ops->unmap(ops, iova, size, gather); +} + +static int apple_dart_domain_needs_pgtbl_ops(struct apple_dart *dart, + struct iommu_domain *domain) +{ + if (domain->type == IOMMU_DOMAIN_DMA) + return 1; + if (domain->type == IOMMU_DOMAIN_UNMANAGED) + return 1; + if (!dart->supports_bypass && domain->type == IOMMU_DOMAIN_IDENTITY) + return 1; + return 0; +} + +/* must be called with held dart_domain->lock */ +static int apple_dart_finalize_domain(struct iommu_domain *domain) +{ + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + struct apple_dart *dart = dart_domain->dart; + struct io_pgtable_cfg pgtbl_cfg; + + if (dart_domain->pgtbl_ops) + return 0; + if (!apple_dart_domain_needs_pgtbl_ops(dart, domain)) + return 0; + + pgtbl_cfg = (struct io_pgtable_cfg){ + .pgsize_bitmap = dart->pgsize, + .ias = 32, + .oas = 36, + .coherent_walk = 1, + .tlb = &apple_dart_tlb_ops, + .iommu_dev = dart->dev, + }; + + dart_domain->pgtbl_ops = + alloc_io_pgtable_ops(ARM_APPLE_DART, &pgtbl_cfg, domain); + if (!dart_domain->pgtbl_ops) + return -ENOMEM; + + domain->pgsize_bitmap = pgtbl_cfg.pgsize_bitmap; + domain->geometry.aperture_start = 0; + domain->geometry.aperture_end = DMA_BIT_MASK(32); + domain->geometry.force_aperture = true; + + // HACK: create a static pagetable to fake bypass mode + if (domain->type == IOMMU_DOMAIN_IDENTITY) { + u64 addr; + + for (addr = 0; addr < dart->force_bypass_len; + addr += dart->pgsize) + dart_domain->pgtbl_ops->map( + dart_domain->pgtbl_ops, addr, + dart->force_bypass_offset + addr, dart->pgsize, + IOMMU_READ | IOMMU_WRITE, GFP_ATOMIC); + } + + return 0; +} + +static void apple_dart_stream_setup_ttbrs(struct apple_dart_domain *domain, + struct apple_dart *dart, u32 sid) +{ + int i; + struct io_pgtable_cfg *pgtbl_cfg = + &io_pgtable_ops_to_pgtable(domain->pgtbl_ops)->cfg; + + for (i = 0; i < pgtbl_cfg->apple_dart_cfg.n_ttbrs; ++i) + apple_dart_hw_set_ttbr(dart, sid, i, + pgtbl_cfg->apple_dart_cfg.ttbr[i]); + for (; i < DART_MAX_TTBR; ++i) + apple_dart_hw_clear_ttbr(dart, sid, i); +} + +/* must be called with held domain->lock */ +static int apple_dart_attach_stream(struct apple_dart_domain *domain, + struct apple_dart *dart, u32 sid) +{ + unsigned long flags; + struct apple_dart_stream *stream; + int ret; + + if (WARN_ON(dart->force_bypass && + domain->type != IOMMU_DOMAIN_IDENTITY)) + return -EINVAL; + + /* + * we can't mix and match DARTs that support bypass mode with those who don't + * because the iova space in fake bypass mode generally has an offset + */ + if (WARN_ON(domain->type == IOMMU_DOMAIN_IDENTITY && + (domain->dart->supports_bypass != dart->supports_bypass))) + return -EINVAL; + + list_for_each_entry(stream, &domain->streams, stream_head) { + if (stream->dart == dart && stream->sid == sid) { + stream->num_devices++; + return 0; + } + } + + spin_lock_irqsave(&dart->lock, flags); + + if (WARN_ON(dart->used_sids & BIT(sid))) { + ret = -EINVAL; + goto error; + } + + stream = kzalloc(sizeof(*stream), GFP_ATOMIC); + if (!stream) { + ret = -ENOMEM; + goto error; + } + + stream->dart = dart; + stream->sid = sid; + stream->num_devices = 1; + list_add(&stream->stream_head, &domain->streams); + + dart->used_sids |= BIT(sid); + spin_unlock_irqrestore(&dart->lock, flags); + + apple_dart_hw_clear_all_ttbrs(stream->dart, stream->sid); + + switch (domain->type) { + case IOMMU_DOMAIN_IDENTITY: + if (stream->dart->supports_bypass) { + apple_dart_hw_enable_bypass(stream->dart, stream->sid); + } else { + apple_dart_stream_setup_ttbrs(domain, stream->dart, + stream->sid); + apple_dart_hw_enable_translation(stream->dart, + stream->sid); + apple_dart_hw_invalidate_tlb_stream(stream->dart, + stream->sid); + } + break; + case IOMMU_DOMAIN_BLOCKED: + apple_dart_hw_disable_dma(stream->dart, stream->sid); + break; + case IOMMU_DOMAIN_UNMANAGED: + case IOMMU_DOMAIN_DMA: + apple_dart_stream_setup_ttbrs(domain, stream->dart, + stream->sid); + apple_dart_hw_enable_translation(stream->dart, stream->sid); + apple_dart_hw_invalidate_tlb_stream(stream->dart, stream->sid); + break; + } + + return 0; + +error: + spin_unlock_irqrestore(&dart->lock, flags); + return ret; +} + +static void apple_dart_disable_stream(struct apple_dart *dart, u32 sid) +{ + unsigned long flags; + + apple_dart_hw_disable_dma(dart, sid); + apple_dart_hw_clear_all_ttbrs(dart, sid); + apple_dart_hw_invalidate_tlb_stream(dart, sid); + + spin_lock_irqsave(&dart->lock, flags); + dart->used_sids &= ~BIT(sid); + spin_unlock_irqrestore(&dart->lock, flags); +} + +/* must be called with held domain->lock */ +static void apple_dart_detach_stream(struct apple_dart_domain *domain, + struct apple_dart *dart, u32 sid) +{ + struct apple_dart_stream *stream; + + list_for_each_entry(stream, &domain->streams, stream_head) { + if (stream->dart == dart && stream->sid == sid) { + stream->num_devices--; + + if (stream->num_devices == 0) { + apple_dart_disable_stream(dart, sid); + list_del(&stream->stream_head); + kfree(stream); + } + return; + } + } +} + +static int apple_dart_attach_dev(struct iommu_domain *domain, + struct device *dev) +{ + int ret; + int i, j; + unsigned long flags; + struct apple_dart_master_cfg *cfg = dev_iommu_priv_get(dev); + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + struct apple_dart *dart = cfg->streams[0].dart; + + spin_lock_irqsave(&dart_domain->lock, flags); + + if (!dart_domain->dart) + dart_domain->dart = dart; + + if (WARN_ON(dart->force_bypass && + dart_domain->type != IOMMU_DOMAIN_IDENTITY)) { + dev_warn( + dev, + "IOMMU must be in bypass mode but trying to attach to translated domain.\n"); + ret = -EINVAL; + goto out; + } + + ret = apple_dart_finalize_domain(domain); + if (ret) + goto out; + + for (i = 0; i < cfg->num_streams; ++i) { + ret = apple_dart_attach_stream( + dart_domain, cfg->streams[i].dart, cfg->streams[i].sid); + if (ret) { + /* try to undo what we did before returning */ + for (j = 0; j < i; ++j) + apple_dart_detach_stream(dart_domain, + cfg->streams[j].dart, + cfg->streams[j].sid); + + goto out; + } + } + + ret = 0; + +out: + spin_unlock_irqrestore(&dart_domain->lock, flags); + return ret; +} + +static void apple_dart_detach_dev(struct iommu_domain *domain, + struct device *dev) +{ + int i; + unsigned long flags; + struct apple_dart_master_cfg *cfg = dev_iommu_priv_get(dev); + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + + spin_lock_irqsave(&dart_domain->lock, flags); + + for (i = 0; i < cfg->num_streams; ++i) + apple_dart_detach_stream(dart_domain, cfg->streams[i].dart, + cfg->streams[i].sid); + + spin_unlock_irqrestore(&dart_domain->lock, flags); +} + +static struct iommu_device *apple_dart_probe_device(struct device *dev) +{ + struct apple_dart_master_cfg *cfg = dev_iommu_priv_get(dev); + int i; + + if (!cfg) + return ERR_PTR(-ENODEV); + + for (i = 0; i < cfg->num_streams; ++i) { + cfg->streams[i].link = + device_link_add(dev, cfg->streams[i].dart->dev, + DL_FLAG_PM_RUNTIME | DL_FLAG_STATELESS); + } + + return &cfg->streams[0].dart->iommu; +} + +static void apple_dart_release_device(struct device *dev) +{ + struct apple_dart_master_cfg *cfg = dev_iommu_priv_get(dev); + int i; + + if (!cfg) + return; + + for (i = 0; i < cfg->num_streams; ++i) + device_link_del(cfg->streams[i].link); + + dev_iommu_priv_set(dev, NULL); + kfree(cfg); +} + +static struct iommu_domain *apple_dart_domain_alloc(unsigned int type) +{ + struct apple_dart_domain *dart_domain; + + if (type != IOMMU_DOMAIN_DMA && type != IOMMU_DOMAIN_UNMANAGED && + type != IOMMU_DOMAIN_IDENTITY && type != IOMMU_DOMAIN_BLOCKED) + return NULL; + + dart_domain = kzalloc(sizeof(*dart_domain), GFP_KERNEL); + if (!dart_domain) + return NULL; + + INIT_LIST_HEAD(&dart_domain->streams); + spin_lock_init(&dart_domain->lock); + iommu_get_dma_cookie(&dart_domain->domain); + dart_domain->type = type; + + return &dart_domain->domain; +} + +static void apple_dart_domain_free(struct iommu_domain *domain) +{ + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + + WARN_ON(!list_empty(&dart_domain->streams)); + + kfree(dart_domain); +} + +static int apple_dart_of_xlate(struct device *dev, struct of_phandle_args *args) +{ + struct platform_device *iommu_pdev = of_find_device_by_node(args->np); + struct apple_dart_master_cfg *cfg = dev_iommu_priv_get(dev); + unsigned int num_streams = cfg ? cfg->num_streams : 0; + struct apple_dart_master_cfg *cfg_new; + struct apple_dart *dart = platform_get_drvdata(iommu_pdev); + + if (args->args_count != 1) + return -EINVAL; + + cfg_new = krealloc(cfg, struct_size(cfg, streams, num_streams + 1), + GFP_KERNEL); + if (!cfg_new) + return -ENOMEM; + + cfg = cfg_new; + dev_iommu_priv_set(dev, cfg); + + cfg->num_streams = num_streams; + cfg->streams[cfg->num_streams].dart = dart; + cfg->streams[cfg->num_streams].sid = args->args[0]; + cfg->num_streams++; + + return 0; +} + +static struct iommu_group *apple_dart_device_group(struct device *dev) +{ +#ifdef CONFIG_PCI + struct iommu_group *group; + + if (dev_is_pci(dev)) + group = pci_device_group(dev); + else + group = generic_device_group(dev); + + return group; +#else + return generic_device_group(dev); +#endif +} + +static const struct iommu_ops apple_dart_iommu_ops = { + .domain_alloc = apple_dart_domain_alloc, + .domain_free = apple_dart_domain_free, + .attach_dev = apple_dart_attach_dev, + .detach_dev = apple_dart_detach_dev, + .map = apple_dart_map, + .unmap = apple_dart_unmap, + .flush_iotlb_all = apple_dart_flush_iotlb_all, + .iotlb_sync = apple_dart_iotlb_sync, + .iotlb_sync_map = apple_dart_iotlb_sync_map, + .iova_to_phys = apple_dart_iova_to_phys, + .probe_device = apple_dart_probe_device, + .release_device = apple_dart_release_device, + .device_group = apple_dart_device_group, + .of_xlate = apple_dart_of_xlate, + .pgsize_bitmap = -1UL, /* Restricted during dart probe */ +}; + +static irqreturn_t apple_dart_irq(int irq, void *dev) +{ + struct apple_dart *dart = dev; + static DEFINE_RATELIMIT_STATE(rs, DEFAULT_RATELIMIT_INTERVAL, + DEFAULT_RATELIMIT_BURST); + const char *fault_name = NULL; + u32 error = readl(dart->regs + DART_ERROR); + u32 error_code = FIELD_GET(DART_ERROR_CODE, error); + u32 addr_lo = readl(dart->regs + DART_ERROR_ADDR_LO); + u32 addr_hi = readl(dart->regs + DART_ERROR_ADDR_HI); + u64 addr = addr_lo | (((u64)addr_hi) << 32); + u8 stream_idx = FIELD_GET(DART_ERROR_STREAM, error); + + if (!(error & DART_ERROR_FLAG)) + return IRQ_NONE; + + if (error_code & DART_ERROR_READ_FAULT) + fault_name = "READ FAULT"; + else if (error_code & DART_ERROR_WRITE_FAULT) + fault_name = "WRITE FAULT"; + else if (error_code & DART_ERROR_NO_PTE) + fault_name = "NO PTE FOR IOVA"; + else if (error_code & DART_ERROR_NO_PMD) + fault_name = "NO PMD FOR IOVA"; + else if (error_code & DART_ERROR_NO_TTBR) + fault_name = "NO TTBR FOR IOVA"; + + if (WARN_ON(fault_name == NULL)) + fault_name = "unknown"; + + if (__ratelimit(&rs)) { + dev_err(dart->dev, + "translation fault: status:0x%x stream:%d code:0x%x (%s) at 0x%llx", + error, stream_idx, error_code, fault_name, addr); + } + + writel(error, dart->regs + DART_ERROR); + return IRQ_HANDLED; +} + +static int apple_dart_probe(struct platform_device *pdev) +{ + int ret; + u32 dart_params[2]; + struct resource *res; + struct apple_dart *dart; + struct device *dev = &pdev->dev; + + dart = devm_kzalloc(dev, sizeof(*dart), GFP_KERNEL); + if (!dart) + return -ENOMEM; + + dart->dev = dev; + spin_lock_init(&dart->lock); + + if (pdev->num_resources < 1) + return -ENODEV; + + res = platform_get_resource(pdev, IORESOURCE_MEM, 0); + if (resource_size(res) < 0x4000) { + dev_err(dev, "MMIO region too small (%pr)\n", res); + return -EINVAL; + } + + dart->regs = devm_ioremap_resource(dev, res); + if (IS_ERR(dart->regs)) + return PTR_ERR(dart->regs); + + ret = devm_clk_bulk_get_all(dev, &dart->clks); + if (ret < 0) + return ret; + dart->num_clks = ret; + + ret = clk_bulk_prepare_enable(dart->num_clks, dart->clks); + if (ret) + return ret; + + ret = apple_dart_hw_reset(dart); + if (ret) + return ret; + + dart_params[0] = readl(dart->regs + DART_PARAMS1); + dart_params[1] = readl(dart->regs + DART_PARAMS2); + dart->pgsize = 1 << FIELD_GET(DART_PARAMS_PAGE_SHIFT, dart_params[0]); + dart->supports_bypass = dart_params[1] & DART_PARAMS_BYPASS_SUPPORT; + dart->force_bypass = dart->pgsize > PAGE_SIZE; + + // HACK: these shouldn't be hardcoded here but come from the device tree + dart->force_bypass_offset = 0x0800000000; + dart->force_bypass_len = 0x0100000000; + + dart->irq = platform_get_irq(pdev, 0); + if (dart->irq < 0) + return -ENODEV; + + ret = devm_request_irq(dart->dev, dart->irq, apple_dart_irq, + IRQF_SHARED, "apple-dart fault handler", dart); + if (ret) + return ret; + + platform_set_drvdata(pdev, dart); + + ret = iommu_device_sysfs_add(&dart->iommu, dev, NULL, "apple-dart.%s", + dev_name(&pdev->dev)); + if (ret) + return ret; + + ret = iommu_device_register(&dart->iommu, &apple_dart_iommu_ops, dev); + if (ret) + return ret; + + if (dev->bus->iommu_ops != &apple_dart_iommu_ops) { + ret = bus_set_iommu(dev->bus, &apple_dart_iommu_ops); + if (ret) + return ret; + } +#ifdef CONFIG_PCI + if (dev->bus->iommu_ops != pci_bus_type.iommu_ops) { + ret = bus_set_iommu(&pci_bus_type, &apple_dart_iommu_ops); + if (ret) + return ret; + } +#endif + + dev_info( + &pdev->dev, + "DART [pagesize %x, bypass support: %d, bypass forced: %d] initialized\n", + dart->pgsize, dart->supports_bypass, dart->force_bypass); + return 0; +} + +static int apple_dart_remove(struct platform_device *pdev) +{ + struct apple_dart *dart = platform_get_drvdata(pdev); + + devm_free_irq(dart->dev, dart->irq, dart); + + iommu_device_unregister(&dart->iommu); + iommu_device_sysfs_remove(&dart->iommu); + + clk_bulk_disable(dart->num_clks, dart->clks); + clk_bulk_unprepare(dart->num_clks, dart->clks); + + return 0; +} + +static void apple_dart_shutdown(struct platform_device *pdev) +{ + apple_dart_remove(pdev); +} + +static const struct of_device_id apple_dart_of_match[] = { + { .compatible = "apple,t8103-dart", .data = NULL }, + {}, +}; +MODULE_DEVICE_TABLE(of, apple_dart_of_match); + +static struct platform_driver apple_dart_driver = { + .driver = { + .name = "apple-dart", + .of_match_table = apple_dart_of_match, + }, + .probe = apple_dart_probe, + .remove = apple_dart_remove, + .shutdown = apple_dart_shutdown, +}; +module_platform_driver(apple_dart_driver); + +MODULE_DESCRIPTION("IOMMU API for Apple's DART"); +MODULE_AUTHOR("Sven Peter "); +MODULE_LICENSE("GPL v2");