From patchwork Mon Nov 11 13:53:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Peter Ujfalusi X-Patchwork-Id: 179068 Delivered-To: patch@linaro.org Received: by 2002:ac9:3c86:0:0:0:0:0 with SMTP id w6csp6372304ocf; Mon, 11 Nov 2019 05:52:45 -0800 (PST) X-Google-Smtp-Source: APXvYqyhsUfQlWQHUHG98QXm32Xd/Nw+gf1kCMXrVMr4S14hsT5Rnmd45791GlK4Cac+nAGzcaqm X-Received: by 2002:a17:906:fac7:: with SMTP id lu7mr11595350ejb.5.1573480364986; Mon, 11 Nov 2019 05:52:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573480364; cv=none; d=google.com; s=arc-20160816; b=xGnmpAqI9aX1g19Gn5Dru8UegG0QEXci5cWZtFz+tS8HaZwKYrBLLvrqZDeOlLk0yS 3BXz2+jtprEpHZFp7/cKItpAh60eNrX9beNS9gVo7o0HJZyKowki+6uVmvxA3M4A/6eF 5N481rdKFB3HRPs1ZvRtrjLoKwRsnuzaYe9rf1PKQeYz/mOEmcwRE78vMIuz5iJklTbM gaxij+B+Zt9xCfcuvkHcHP86YvjgV8CyRFVVscWzZy1zqTQK5HqXWc7QPn7lzDZF9QRl XIG8pQ3eafbyMZdVf3lGGeOUPIKpB+qfJfOmEE1A8/LzPEQKT1bSYWWTlFLV9LpBDNJ8 FGJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=sQPYIOUSsT6DDcopQ7EaFDSZEh5w1AAUWnXYoq0RNFA=; b=LOe4GRCkhT6MCHzEphUqcoDz1BjLINyy2U0oCzhJi1hwa0rU1b1Hp+sLY4leTa0Bdu LsgR69EHhj8b6Rj+9XS6dIN4uGnrSvn+aVv7kkKX8RtA9d9wSWd7n3TDDD0kFjbP6QWS WxcdPLvEwuhrEj79ZCQaEUXJVRF0qrMnmgdjgO+RFZhB68PegeBkwYEvV28hPS0nKUNx Kkx2SYy1TDdPvKXWHDjUuNlGN5HGGX8ST41X/RsUaJGS/OCL1Yp879N/sNlEiXoBQnU9 tlJaq+enX8zpFppI9bSqz/Pm1GIsqFVManY9KEOsLR5RlO2nMYNcQ0gPxtmLrxF0cf1U tuNg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ti.com header.s=ti-com-17Q1 header.b=mpswwmWw; spf=pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=dmaengine-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=ti.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q15si9555555ejm.31.2019.11.11.05.52.44; Mon, 11 Nov 2019 05:52:44 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@ti.com header.s=ti-com-17Q1 header.b=mpswwmWw; spf=pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=dmaengine-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=ti.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727083AbfKKNwn (ORCPT + 3 others); Mon, 11 Nov 2019 08:52:43 -0500 Received: from lelv0143.ext.ti.com ([198.47.23.248]:43398 "EHLO lelv0143.ext.ti.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726902AbfKKNwm (ORCPT ); Mon, 11 Nov 2019 08:52:42 -0500 Received: from lelv0266.itg.ti.com ([10.180.67.225]) by lelv0143.ext.ti.com (8.15.2/8.15.2) with ESMTP id xABDqQsK013794; Mon, 11 Nov 2019 07:52:26 -0600 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1573480346; bh=sQPYIOUSsT6DDcopQ7EaFDSZEh5w1AAUWnXYoq0RNFA=; h=From:To:CC:Subject:Date:In-Reply-To:References; b=mpswwmWwuTm3oimzL6ud5OhlKtTFwS6DHvYDAzZl9qRktZ9o08J/Sq6fCqVTmIyqA 9h6H7c3KNWUp/kGb/e7d3mBGkmdI3UZZIHYDJXDZZ0Q2rS6mfnVTYs7eQUnHa2lm6U CeqzNOrY6lcLboMevJQ+83rCvoezcuE9q7xtqQU8= Received: from DFLE104.ent.ti.com (dfle104.ent.ti.com [10.64.6.25]) by lelv0266.itg.ti.com (8.15.2/8.15.2) with ESMTPS id xABDqQ3A079433 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 11 Nov 2019 07:52:26 -0600 Received: from DFLE112.ent.ti.com (10.64.6.33) by DFLE104.ent.ti.com (10.64.6.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1847.3; Mon, 11 Nov 2019 07:52:09 -0600 Received: from fllv0040.itg.ti.com (10.64.41.20) by DFLE112.ent.ti.com (10.64.6.33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1847.3 via Frontend Transport; Mon, 11 Nov 2019 07:52:09 -0600 Received: from feketebors.ti.com (ileax41-snat.itg.ti.com [10.172.224.153]) by fllv0040.itg.ti.com (8.15.2/8.15.2) with ESMTP id xABDqE8n097668; Mon, 11 Nov 2019 07:52:22 -0600 From: Peter Ujfalusi To: , , , CC: , , , , , , , , , Subject: [PATCH v5 02/15] soc: ti: k3: add navss ringacc driver Date: Mon, 11 Nov 2019 15:53:17 +0200 Message-ID: <20191111135330.8235-3-peter.ujfalusi@ti.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: <20191111135330.8235-1-peter.ujfalusi@ti.com> References: <20191111135330.8235-1-peter.ujfalusi@ti.com> MIME-Version: 1.0 X-EXCLAIMER-MD-CONFIG: e1e8a2fd-e40a-4ac6-ac9b-f7e9cc9ee180 Sender: dmaengine-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org From: Grygorii Strashko The Ring Accelerator (RINGACC or RA) provides hardware acceleration to enable straightforward passing of work between a producer and a consumer. There is one RINGACC module per NAVSS on TI AM65x SoCs. The RINGACC converts constant-address read and write accesses to equivalent read or write accesses to a circular data structure in memory. The RINGACC eliminates the need for each DMA controller which needs to access ring elements from having to know the current state of the ring (base address, current offset). The DMA controller performs a read or write access to a specific address range (which maps to the source interface on the RINGACC) and the RINGACC replaces the address for the transaction with a new address which corresponds to the head or tail element of the ring (head for reads, tail for writes). Since the RINGACC maintains the state, multiple DMA controllers or channels are allowed to coherently share the same rings as applicable. The RINGACC is able to place data which is destined towards software into cached memory directly. Supported ring modes: - Ring Mode - Messaging Mode - Credentials Mode - Queue Manager Mode TI-SCI integration: Texas Instrument's System Control Interface (TI-SCI) Message Protocol now has control over Ringacc module resources management (RM) and Rings configuration. The corresponding support of TI-SCI Ringacc module RM protocol introduced as option through DT parameters: - ti,sci: phandle on TI-SCI firmware controller DT node - ti,sci-dev-id: TI-SCI device identifier as per TI-SCI firmware spec if both parameters present - Ringacc driver will configure/free/reset Rings using TI-SCI Message Ringacc RM Protocol. The Ringacc driver manages Rings allocation by itself now and requests TI-SCI firmware to allocate and configure specific Rings only. It's done this way because, Linux driver implements two stage Rings allocation and configuration (allocate ring and configure ring) while TI-SCI Message Protocol supports only one combined operation (allocate+configure). Signed-off-by: Grygorii Strashko Signed-off-by: Peter Ujfalusi Reviewed-by: Tero Kristo --- drivers/soc/ti/Kconfig | 11 + drivers/soc/ti/Makefile | 1 + drivers/soc/ti/k3-ringacc.c | 1180 +++++++++++++++++++++++++++++ include/linux/soc/ti/k3-ringacc.h | 244 ++++++ 4 files changed, 1436 insertions(+) create mode 100644 drivers/soc/ti/k3-ringacc.c create mode 100644 include/linux/soc/ti/k3-ringacc.h -- Peter Texas Instruments Finland Oy, Porkkalankatu 22, 00180 Helsinki. Y-tunnus/Business ID: 0615521-4. Kotipaikka/Domicile: Helsinki diff --git a/drivers/soc/ti/Kconfig b/drivers/soc/ti/Kconfig index cf545f428d03..bdce98f68a3e 100644 --- a/drivers/soc/ti/Kconfig +++ b/drivers/soc/ti/Kconfig @@ -80,6 +80,17 @@ config TI_SCI_PM_DOMAINS called ti_sci_pm_domains. Note this is needed early in boot before rootfs may be available. +config TI_K3_RINGACC + tristate "K3 Ring accelerator Sub System" + depends on ARCH_K3 || COMPILE_TEST + depends on TI_SCI_INTA_IRQCHIP + help + Say y here to support the K3 Ring accelerator module. + The Ring Accelerator (RINGACC or RA) provides hardware acceleration + to enable straightforward passing of work between a producer + and a consumer. There is one RINGACC module per NAVSS on TI AM65x SoCs + If unsure, say N. + endif # SOC_TI config TI_SCI_INTA_MSI_DOMAIN diff --git a/drivers/soc/ti/Makefile b/drivers/soc/ti/Makefile index b3868d392d4f..cc4bc8b08bf5 100644 --- a/drivers/soc/ti/Makefile +++ b/drivers/soc/ti/Makefile @@ -9,3 +9,4 @@ obj-$(CONFIG_AMX3_PM) += pm33xx.o obj-$(CONFIG_WKUP_M3_IPC) += wkup_m3_ipc.o obj-$(CONFIG_TI_SCI_PM_DOMAINS) += ti_sci_pm_domains.o obj-$(CONFIG_TI_SCI_INTA_MSI_DOMAIN) += ti_sci_inta_msi.o +obj-$(CONFIG_TI_K3_RINGACC) += k3-ringacc.o diff --git a/drivers/soc/ti/k3-ringacc.c b/drivers/soc/ti/k3-ringacc.c new file mode 100644 index 000000000000..fd9f35b7c9a6 --- /dev/null +++ b/drivers/soc/ti/k3-ringacc.c @@ -0,0 +1,1180 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * TI K3 NAVSS Ring Accelerator subsystem driver + * + * Copyright (C) 2019 Texas Instruments Incorporated - http://www.ti.com + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static LIST_HEAD(k3_ringacc_list); +static DEFINE_MUTEX(k3_ringacc_list_lock); + +#define K3_RINGACC_CFG_RING_SIZE_ELCNT_MASK GENMASK(19, 0) + +/** + * struct k3_ring_rt_regs - The RA realtime Control/Status Registers region + * + * @resv_16: Reserved + * @db: Ring Doorbell Register + * @resv_4: Reserved + * @occ: Ring Occupancy Register + * @indx: Ring Current Index Register + * @hwocc: Ring Hardware Occupancy Register + * @hwindx: Ring Hardware Current Index Register + */ +struct k3_ring_rt_regs { + u32 resv_16[4]; + u32 db; + u32 resv_4[1]; + u32 occ; + u32 indx; + u32 hwocc; + u32 hwindx; +}; + +#define K3_RINGACC_RT_REGS_STEP 0x1000 + +/** + * struct k3_ring_fifo_regs - The Ring Accelerator Queues Registers region + * + * @head_data: Ring Head Entry Data Registers + * @tail_data: Ring Tail Entry Data Registers + * @peek_head_data: Ring Peek Head Entry Data Regs + * @peek_tail_data: Ring Peek Tail Entry Data Regs + */ +struct k3_ring_fifo_regs { + u32 head_data[128]; + u32 tail_data[128]; + u32 peek_head_data[128]; + u32 peek_tail_data[128]; +}; + +/** + * struct k3_ringacc_proxy_gcfg_regs - RA Proxy Global Config MMIO Region + * + * @revision: Revision Register + * @config: Config Register + */ +struct k3_ringacc_proxy_gcfg_regs { + u32 revision; + u32 config; +}; + +#define K3_RINGACC_PROXY_CFG_THREADS_MASK GENMASK(15, 0) + +/** + * struct k3_ringacc_proxy_target_regs - Proxy Datapath MMIO Region + * + * @control: Proxy Control Register + * @status: Proxy Status Register + * @resv_512: Reserved + * @data: Proxy Data Register + */ +struct k3_ringacc_proxy_target_regs { + u32 control; + u32 status; + u8 resv_512[504]; + u32 data[128]; +}; + +#define K3_RINGACC_PROXY_TARGET_STEP 0x1000 +#define K3_RINGACC_PROXY_NOT_USED (-1) + +enum k3_ringacc_proxy_access_mode { + PROXY_ACCESS_MODE_HEAD = 0, + PROXY_ACCESS_MODE_TAIL = 1, + PROXY_ACCESS_MODE_PEEK_HEAD = 2, + PROXY_ACCESS_MODE_PEEK_TAIL = 3, +}; + +#define K3_RINGACC_FIFO_WINDOW_SIZE_BYTES (512U) +#define K3_RINGACC_FIFO_REGS_STEP 0x1000 +#define K3_RINGACC_MAX_DB_RING_CNT (127U) + +struct k3_ring_ops { + int (*push_tail)(struct k3_ring *ring, void *elm); + int (*push_head)(struct k3_ring *ring, void *elm); + int (*pop_tail)(struct k3_ring *ring, void *elm); + int (*pop_head)(struct k3_ring *ring, void *elm); +}; + +/** + * struct k3_ring - RA Ring descriptor + * + * @rt: Ring control/status registers + * @fifos: Ring queues registers + * @proxy: Ring Proxy Datapath registers + * @ring_mem_dma: Ring buffer dma address + * @ring_mem_virt: Ring buffer virt address + * @ops: Ring operations + * @size: Ring size in elements + * @elm_size: Size of the ring element + * @mode: Ring mode + * @flags: flags + * @free: Number of free elements + * @occ: Ring occupancy + * @windex: Write index (only for @K3_RINGACC_RING_MODE_RING) + * @rindex: Read index (only for @K3_RINGACC_RING_MODE_RING) + * @ring_id: Ring Id + * @parent: Pointer on struct @k3_ringacc + * @use_count: Use count for shared rings + * @proxy_id: RA Ring Proxy Id (only if @K3_RINGACC_RING_USE_PROXY) + */ +struct k3_ring { + struct k3_ring_rt_regs __iomem *rt; + struct k3_ring_fifo_regs __iomem *fifos; + struct k3_ringacc_proxy_target_regs __iomem *proxy; + dma_addr_t ring_mem_dma; + void *ring_mem_virt; + struct k3_ring_ops *ops; + u32 size; + enum k3_ring_size elm_size; + enum k3_ring_mode mode; + u32 flags; +#define K3_RING_FLAG_BUSY BIT(1) +#define K3_RING_FLAG_SHARED BIT(2) + u32 free; + u32 occ; + u32 windex; + u32 rindex; + u32 ring_id; + struct k3_ringacc *parent; + u32 use_count; + int proxy_id; +}; + +/** + * struct k3_ringacc - Rings accelerator descriptor + * + * @dev: pointer on RA device + * @proxy_gcfg: RA proxy global config registers + * @proxy_target_base: RA proxy datapath region + * @num_rings: number of ring in RA + * @rings_inuse: bitfield for ring usage tracking + * @rm_gp_range: general purpose rings range from tisci + * @dma_ring_reset_quirk: DMA reset w/a enable + * @num_proxies: number of RA proxies + * @proxy_inuse: bitfield for proxy usage tracking + * @rings: array of rings descriptors (struct @k3_ring) + * @list: list of RAs in the system + * @req_lock: protect rings allocation + * @tisci: pointer ti-sci handle + * @tisci_ring_ops: ti-sci rings ops + * @tisci_dev_id: ti-sci device id + */ +struct k3_ringacc { + struct device *dev; + struct k3_ringacc_proxy_gcfg_regs __iomem *proxy_gcfg; + void __iomem *proxy_target_base; + u32 num_rings; /* number of rings in Ringacc module */ + unsigned long *rings_inuse; + struct ti_sci_resource *rm_gp_range; + + bool dma_ring_reset_quirk; + u32 num_proxies; + unsigned long *proxy_inuse; + + struct k3_ring *rings; + struct list_head list; + struct mutex req_lock; /* protect rings allocation */ + + const struct ti_sci_handle *tisci; + const struct ti_sci_rm_ringacc_ops *tisci_ring_ops; + u32 tisci_dev_id; +}; + +static long k3_ringacc_ring_get_fifo_pos(struct k3_ring *ring) +{ + return K3_RINGACC_FIFO_WINDOW_SIZE_BYTES - + (4 << ring->elm_size); +} + +static void *k3_ringacc_get_elm_addr(struct k3_ring *ring, u32 idx) +{ + return (ring->ring_mem_virt + idx * (4 << ring->elm_size)); +} + +static int k3_ringacc_ring_push_mem(struct k3_ring *ring, void *elem); +static int k3_ringacc_ring_pop_mem(struct k3_ring *ring, void *elem); + +static struct k3_ring_ops k3_ring_mode_ring_ops = { + .push_tail = k3_ringacc_ring_push_mem, + .pop_head = k3_ringacc_ring_pop_mem, +}; + +static int k3_ringacc_ring_push_io(struct k3_ring *ring, void *elem); +static int k3_ringacc_ring_pop_io(struct k3_ring *ring, void *elem); +static int k3_ringacc_ring_push_head_io(struct k3_ring *ring, void *elem); +static int k3_ringacc_ring_pop_tail_io(struct k3_ring *ring, void *elem); + +static struct k3_ring_ops k3_ring_mode_msg_ops = { + .push_tail = k3_ringacc_ring_push_io, + .push_head = k3_ringacc_ring_push_head_io, + .pop_tail = k3_ringacc_ring_pop_tail_io, + .pop_head = k3_ringacc_ring_pop_io, +}; + +static int k3_ringacc_ring_push_head_proxy(struct k3_ring *ring, void *elem); +static int k3_ringacc_ring_push_tail_proxy(struct k3_ring *ring, void *elem); +static int k3_ringacc_ring_pop_head_proxy(struct k3_ring *ring, void *elem); +static int k3_ringacc_ring_pop_tail_proxy(struct k3_ring *ring, void *elem); + +static struct k3_ring_ops k3_ring_mode_proxy_ops = { + .push_tail = k3_ringacc_ring_push_tail_proxy, + .push_head = k3_ringacc_ring_push_head_proxy, + .pop_tail = k3_ringacc_ring_pop_tail_proxy, + .pop_head = k3_ringacc_ring_pop_head_proxy, +}; + +static void k3_ringacc_ring_dump(struct k3_ring *ring) +{ + struct device *dev = ring->parent->dev; + + dev_dbg(dev, "dump ring: %d\n", ring->ring_id); + dev_dbg(dev, "dump mem virt %p, dma %pad\n", ring->ring_mem_virt, + &ring->ring_mem_dma); + dev_dbg(dev, "dump elmsize %d, size %d, mode %d, proxy_id %d\n", + ring->elm_size, ring->size, ring->mode, ring->proxy_id); + + dev_dbg(dev, "dump ring_rt_regs: db%08x\n", readl(&ring->rt->db)); + dev_dbg(dev, "dump occ%08x\n", readl(&ring->rt->occ)); + dev_dbg(dev, "dump indx%08x\n", readl(&ring->rt->indx)); + dev_dbg(dev, "dump hwocc%08x\n", readl(&ring->rt->hwocc)); + dev_dbg(dev, "dump hwindx%08x\n", readl(&ring->rt->hwindx)); + + if (ring->ring_mem_virt) + print_hex_dump_debug("dump ring_mem_virt ", DUMP_PREFIX_NONE, + 16, 1, ring->ring_mem_virt, 16 * 8, false); +} + +struct k3_ring *k3_ringacc_request_ring(struct k3_ringacc *ringacc, + int id, u32 flags) +{ + int proxy_id = K3_RINGACC_PROXY_NOT_USED; + + mutex_lock(&ringacc->req_lock); + + if (!try_module_get(ringacc->dev->driver->owner)) { + mutex_unlock(&ringacc->req_lock); + return NULL; + } + + if (id == K3_RINGACC_RING_ID_ANY) { + /* Request for any general purpose ring */ + struct ti_sci_resource_desc *gp_rings = + &ringacc->rm_gp_range->desc[0]; + unsigned long size; + + size = gp_rings->start + gp_rings->num; + id = find_next_zero_bit(ringacc->rings_inuse, size, + gp_rings->start); + if (id == size) + goto error; + } else if (id < 0) { + goto error; + } + + if (test_bit(id, ringacc->rings_inuse) && + !(ringacc->rings[id].flags & K3_RING_FLAG_SHARED)) + goto error; + else if (ringacc->rings[id].flags & K3_RING_FLAG_SHARED) + goto out; + + if (flags & K3_RINGACC_RING_USE_PROXY) { + proxy_id = find_next_zero_bit(ringacc->proxy_inuse, + ringacc->num_proxies, 0); + if (proxy_id == ringacc->num_proxies) + goto error; + } + + if (proxy_id != K3_RINGACC_PROXY_NOT_USED) { + set_bit(proxy_id, ringacc->proxy_inuse); + ringacc->rings[id].proxy_id = proxy_id; + dev_dbg(ringacc->dev, "Giving ring#%d proxy#%d\n", id, + proxy_id); + } else { + dev_dbg(ringacc->dev, "Giving ring#%d\n", id); + } + + set_bit(id, ringacc->rings_inuse); +out: + ringacc->rings[id].use_count++; + mutex_unlock(&ringacc->req_lock); + return &ringacc->rings[id]; + +error: + module_put(ringacc->dev->driver->owner); + mutex_unlock(&ringacc->req_lock); + return NULL; +} +EXPORT_SYMBOL_GPL(k3_ringacc_request_ring); + +static void k3_ringacc_ring_reset_sci(struct k3_ring *ring) +{ + struct k3_ringacc *ringacc = ring->parent; + int ret; + + ret = ringacc->tisci_ring_ops->config( + ringacc->tisci, + TI_SCI_MSG_VALUE_RM_RING_COUNT_VALID, + ringacc->tisci_dev_id, + ring->ring_id, + 0, + 0, + ring->size, + 0, + 0, + 0); + if (ret) + dev_err(ringacc->dev, "TISCI reset ring fail (%d) ring_idx %d\n", + ret, ring->ring_id); +} + +void k3_ringacc_ring_reset(struct k3_ring *ring) +{ + if (!ring || !(ring->flags & K3_RING_FLAG_BUSY)) + return; + + ring->occ = 0; + ring->free = 0; + ring->rindex = 0; + ring->windex = 0; + + k3_ringacc_ring_reset_sci(ring); +} +EXPORT_SYMBOL_GPL(k3_ringacc_ring_reset); + +static void k3_ringacc_ring_reconfig_qmode_sci(struct k3_ring *ring, + enum k3_ring_mode mode) +{ + struct k3_ringacc *ringacc = ring->parent; + int ret; + + ret = ringacc->tisci_ring_ops->config( + ringacc->tisci, + TI_SCI_MSG_VALUE_RM_RING_MODE_VALID, + ringacc->tisci_dev_id, + ring->ring_id, + 0, + 0, + 0, + mode, + 0, + 0); + if (ret) + dev_err(ringacc->dev, "TISCI reconf qmode fail (%d) ring_idx %d\n", + ret, ring->ring_id); +} + +void k3_ringacc_ring_reset_dma(struct k3_ring *ring, u32 occ) +{ + if (!ring || !(ring->flags & K3_RING_FLAG_BUSY)) + return; + + if (!ring->parent->dma_ring_reset_quirk) + goto reset; + + if (!occ) + occ = readl(&ring->rt->occ); + + if (occ) { + u32 db_ring_cnt, db_ring_cnt_cur; + + dev_dbg(ring->parent->dev, "%s %u occ: %u\n", __func__, + ring->ring_id, occ); + /* TI-SCI ring reset */ + k3_ringacc_ring_reset_sci(ring); + + /* + * Setup the ring in ring/doorbell mode (if not already in this + * mode) + */ + if (ring->mode != K3_RINGACC_RING_MODE_RING) + k3_ringacc_ring_reconfig_qmode_sci( + ring, K3_RINGACC_RING_MODE_RING); + /* + * Ring the doorbell 2**22 – ringOcc times. + * This will wrap the internal UDMAP ring state occupancy + * counter (which is 21-bits wide) to 0. + */ + db_ring_cnt = (1U << 22) - occ; + + while (db_ring_cnt != 0) { + /* + * Ring the doorbell with the maximum count each + * iteration if possible to minimize the total + * of writes + */ + if (db_ring_cnt > K3_RINGACC_MAX_DB_RING_CNT) + db_ring_cnt_cur = K3_RINGACC_MAX_DB_RING_CNT; + else + db_ring_cnt_cur = db_ring_cnt; + + writel(db_ring_cnt_cur, &ring->rt->db); + db_ring_cnt -= db_ring_cnt_cur; + } + + /* Restore the original ring mode (if not ring mode) */ + if (ring->mode != K3_RINGACC_RING_MODE_RING) + k3_ringacc_ring_reconfig_qmode_sci(ring, ring->mode); + } + +reset: + /* Reset the ring */ + k3_ringacc_ring_reset(ring); +} +EXPORT_SYMBOL_GPL(k3_ringacc_ring_reset_dma); + +static void k3_ringacc_ring_free_sci(struct k3_ring *ring) +{ + struct k3_ringacc *ringacc = ring->parent; + int ret; + + ret = ringacc->tisci_ring_ops->config( + ringacc->tisci, + TI_SCI_MSG_VALUE_RM_ALL_NO_ORDER, + ringacc->tisci_dev_id, + ring->ring_id, + 0, + 0, + 0, + 0, + 0, + 0); + if (ret) + dev_err(ringacc->dev, "TISCI ring free fail (%d) ring_idx %d\n", + ret, ring->ring_id); +} + +int k3_ringacc_ring_free(struct k3_ring *ring) +{ + struct k3_ringacc *ringacc; + + if (!ring) + return -EINVAL; + + ringacc = ring->parent; + + dev_dbg(ring->parent->dev, "flags: 0x%08x\n", ring->flags); + + if (!test_bit(ring->ring_id, ringacc->rings_inuse)) + return -EINVAL; + + mutex_lock(&ringacc->req_lock); + + if (--ring->use_count) + goto out; + + if (!(ring->flags & K3_RING_FLAG_BUSY)) + goto no_init; + + k3_ringacc_ring_free_sci(ring); + + dma_free_coherent(ringacc->dev, + ring->size * (4 << ring->elm_size), + ring->ring_mem_virt, ring->ring_mem_dma); + ring->flags = 0; + ring->ops = NULL; + if (ring->proxy_id != K3_RINGACC_PROXY_NOT_USED) { + clear_bit(ring->proxy_id, ringacc->proxy_inuse); + ring->proxy = NULL; + ring->proxy_id = K3_RINGACC_PROXY_NOT_USED; + } + +no_init: + clear_bit(ring->ring_id, ringacc->rings_inuse); + + module_put(ringacc->dev->driver->owner); + +out: + mutex_unlock(&ringacc->req_lock); + return 0; +} +EXPORT_SYMBOL_GPL(k3_ringacc_ring_free); + +u32 k3_ringacc_get_ring_id(struct k3_ring *ring) +{ + if (!ring) + return -EINVAL; + + return ring->ring_id; +} +EXPORT_SYMBOL_GPL(k3_ringacc_get_ring_id); + +u32 k3_ringacc_get_tisci_dev_id(struct k3_ring *ring) +{ + if (!ring) + return -EINVAL; + + return ring->parent->tisci_dev_id; +} +EXPORT_SYMBOL_GPL(k3_ringacc_get_tisci_dev_id); + +int k3_ringacc_get_ring_irq_num(struct k3_ring *ring) +{ + int irq_num; + + if (!ring) + return -EINVAL; + + irq_num = ti_sci_inta_msi_get_virq(ring->parent->dev, ring->ring_id); + if (irq_num <= 0) + irq_num = -EINVAL; + return irq_num; +} +EXPORT_SYMBOL_GPL(k3_ringacc_get_ring_irq_num); + +static int k3_ringacc_ring_cfg_sci(struct k3_ring *ring) +{ + struct k3_ringacc *ringacc = ring->parent; + u32 ring_idx; + int ret; + + if (!ringacc->tisci) + return -EINVAL; + + ring_idx = ring->ring_id; + ret = ringacc->tisci_ring_ops->config( + ringacc->tisci, + TI_SCI_MSG_VALUE_RM_ALL_NO_ORDER, + ringacc->tisci_dev_id, + ring_idx, + lower_32_bits(ring->ring_mem_dma), + upper_32_bits(ring->ring_mem_dma), + ring->size, + ring->mode, + ring->elm_size, + 0); + if (ret) + dev_err(ringacc->dev, "TISCI config ring fail (%d) ring_idx %d\n", + ret, ring_idx); + + return ret; +} + +int k3_ringacc_ring_cfg(struct k3_ring *ring, struct k3_ring_cfg *cfg) +{ + struct k3_ringacc *ringacc = ring->parent; + int ret = 0; + + if (!ring || !cfg) + return -EINVAL; + if (cfg->elm_size > K3_RINGACC_RING_ELSIZE_256 || + cfg->mode >= K3_RINGACC_RING_MODE_INVALID || + cfg->size & ~K3_RINGACC_CFG_RING_SIZE_ELCNT_MASK || + !test_bit(ring->ring_id, ringacc->rings_inuse)) + return -EINVAL; + + if (cfg->mode == K3_RINGACC_RING_MODE_MESSAGE && + ring->proxy_id == K3_RINGACC_PROXY_NOT_USED && + cfg->elm_size > K3_RINGACC_RING_ELSIZE_8) { + dev_err(ringacc->dev, + "Message mode must use proxy for %u element size\n", + 4 << ring->elm_size); + return -EINVAL; + } + + /* + * In case of shared ring only the first user (master user) can + * configure the ring. The sequence should be by the client: + * ring = k3_ringacc_request_ring(ringacc, ring_id, 0); # master user + * k3_ringacc_ring_cfg(ring, cfg); # master configuration + * k3_ringacc_request_ring(ringacc, ring_id, K3_RING_FLAG_SHARED); + * k3_ringacc_request_ring(ringacc, ring_id, K3_RING_FLAG_SHARED); + */ + if (ring->use_count != 1) + return 0; + + ring->size = cfg->size; + ring->elm_size = cfg->elm_size; + ring->mode = cfg->mode; + ring->occ = 0; + ring->free = 0; + ring->rindex = 0; + ring->windex = 0; + + if (ring->proxy_id != K3_RINGACC_PROXY_NOT_USED) + ring->proxy = ringacc->proxy_target_base + + ring->proxy_id * K3_RINGACC_PROXY_TARGET_STEP; + + switch (ring->mode) { + case K3_RINGACC_RING_MODE_RING: + ring->ops = &k3_ring_mode_ring_ops; + break; + case K3_RINGACC_RING_MODE_MESSAGE: + if (ring->proxy) + ring->ops = &k3_ring_mode_proxy_ops; + else + ring->ops = &k3_ring_mode_msg_ops; + break; + default: + ring->ops = NULL; + ret = -EINVAL; + goto err_free_proxy; + }; + + ring->ring_mem_virt = dma_alloc_coherent(ringacc->dev, + ring->size * (4 << ring->elm_size), + &ring->ring_mem_dma, GFP_KERNEL); + if (!ring->ring_mem_virt) { + dev_err(ringacc->dev, "Failed to alloc ring mem\n"); + ret = -ENOMEM; + goto err_free_ops; + } + + ret = k3_ringacc_ring_cfg_sci(ring); + + if (ret) + goto err_free_mem; + + ring->flags |= K3_RING_FLAG_BUSY; + ring->flags |= (cfg->flags & K3_RINGACC_RING_SHARED) ? + K3_RING_FLAG_SHARED : 0; + + k3_ringacc_ring_dump(ring); + + return 0; + +err_free_mem: + dma_free_coherent(ringacc->dev, + ring->size * (4 << ring->elm_size), + ring->ring_mem_virt, + ring->ring_mem_dma); +err_free_ops: + ring->ops = NULL; +err_free_proxy: + ring->proxy = NULL; + return ret; +} +EXPORT_SYMBOL_GPL(k3_ringacc_ring_cfg); + +u32 k3_ringacc_ring_get_size(struct k3_ring *ring) +{ + if (!ring || !(ring->flags & K3_RING_FLAG_BUSY)) + return -EINVAL; + + return ring->size; +} +EXPORT_SYMBOL_GPL(k3_ringacc_ring_get_size); + +u32 k3_ringacc_ring_get_free(struct k3_ring *ring) +{ + if (!ring || !(ring->flags & K3_RING_FLAG_BUSY)) + return -EINVAL; + + if (!ring->free) + ring->free = ring->size - readl(&ring->rt->occ); + + return ring->free; +} +EXPORT_SYMBOL_GPL(k3_ringacc_ring_get_free); + +u32 k3_ringacc_ring_get_occ(struct k3_ring *ring) +{ + if (!ring || !(ring->flags & K3_RING_FLAG_BUSY)) + return -EINVAL; + + return readl(&ring->rt->occ); +} +EXPORT_SYMBOL_GPL(k3_ringacc_ring_get_occ); + +u32 k3_ringacc_ring_is_full(struct k3_ring *ring) +{ + return !k3_ringacc_ring_get_free(ring); +} +EXPORT_SYMBOL_GPL(k3_ringacc_ring_is_full); + +enum k3_ringacc_access_mode { + K3_RINGACC_ACCESS_MODE_PUSH_HEAD, + K3_RINGACC_ACCESS_MODE_POP_HEAD, + K3_RINGACC_ACCESS_MODE_PUSH_TAIL, + K3_RINGACC_ACCESS_MODE_POP_TAIL, + K3_RINGACC_ACCESS_MODE_PEEK_HEAD, + K3_RINGACC_ACCESS_MODE_PEEK_TAIL, +}; + +#define K3_RINGACC_PROXY_MODE(x) (((x) & 0x3) << 16) +#define K3_RINGACC_PROXY_ELSIZE(x) (((x) & 0x7) << 24) +static int k3_ringacc_ring_cfg_proxy(struct k3_ring *ring, + enum k3_ringacc_proxy_access_mode mode) +{ + u32 val; + + val = ring->ring_id; + val |= K3_RINGACC_PROXY_MODE(mode); + val |= K3_RINGACC_PROXY_ELSIZE(ring->elm_size); + writel(val, &ring->proxy->control); + return 0; +} + +static int k3_ringacc_ring_access_proxy(struct k3_ring *ring, void *elem, + enum k3_ringacc_access_mode access_mode) +{ + void __iomem *ptr; + + ptr = (void __iomem *)&ring->proxy->data; + + switch (access_mode) { + case K3_RINGACC_ACCESS_MODE_PUSH_HEAD: + case K3_RINGACC_ACCESS_MODE_POP_HEAD: + k3_ringacc_ring_cfg_proxy(ring, PROXY_ACCESS_MODE_HEAD); + break; + case K3_RINGACC_ACCESS_MODE_PUSH_TAIL: + case K3_RINGACC_ACCESS_MODE_POP_TAIL: + k3_ringacc_ring_cfg_proxy(ring, PROXY_ACCESS_MODE_TAIL); + break; + default: + return -EINVAL; + } + + ptr += k3_ringacc_ring_get_fifo_pos(ring); + + switch (access_mode) { + case K3_RINGACC_ACCESS_MODE_POP_HEAD: + case K3_RINGACC_ACCESS_MODE_POP_TAIL: + dev_dbg(ring->parent->dev, + "proxy:memcpy_fromio(x): --> ptr(%p), mode:%d\n", ptr, + access_mode); + memcpy_fromio(elem, ptr, (4 << ring->elm_size)); + ring->occ--; + break; + case K3_RINGACC_ACCESS_MODE_PUSH_TAIL: + case K3_RINGACC_ACCESS_MODE_PUSH_HEAD: + dev_dbg(ring->parent->dev, + "proxy:memcpy_toio(x): --> ptr(%p), mode:%d\n", ptr, + access_mode); + memcpy_toio(ptr, elem, (4 << ring->elm_size)); + ring->free--; + break; + default: + return -EINVAL; + } + + dev_dbg(ring->parent->dev, "proxy: free%d occ%d\n", ring->free, + ring->occ); + return 0; +} + +static int k3_ringacc_ring_push_head_proxy(struct k3_ring *ring, void *elem) +{ + return k3_ringacc_ring_access_proxy(ring, elem, + K3_RINGACC_ACCESS_MODE_PUSH_HEAD); +} + +static int k3_ringacc_ring_push_tail_proxy(struct k3_ring *ring, void *elem) +{ + return k3_ringacc_ring_access_proxy(ring, elem, + K3_RINGACC_ACCESS_MODE_PUSH_TAIL); +} + +static int k3_ringacc_ring_pop_head_proxy(struct k3_ring *ring, void *elem) +{ + return k3_ringacc_ring_access_proxy(ring, elem, + K3_RINGACC_ACCESS_MODE_POP_HEAD); +} + +static int k3_ringacc_ring_pop_tail_proxy(struct k3_ring *ring, void *elem) +{ + return k3_ringacc_ring_access_proxy(ring, elem, + K3_RINGACC_ACCESS_MODE_POP_HEAD); +} + +static int k3_ringacc_ring_access_io(struct k3_ring *ring, void *elem, + enum k3_ringacc_access_mode access_mode) +{ + void __iomem *ptr; + + switch (access_mode) { + case K3_RINGACC_ACCESS_MODE_PUSH_HEAD: + case K3_RINGACC_ACCESS_MODE_POP_HEAD: + ptr = (void __iomem *)&ring->fifos->head_data; + break; + case K3_RINGACC_ACCESS_MODE_PUSH_TAIL: + case K3_RINGACC_ACCESS_MODE_POP_TAIL: + ptr = (void __iomem *)&ring->fifos->tail_data; + break; + default: + return -EINVAL; + } + + ptr += k3_ringacc_ring_get_fifo_pos(ring); + + switch (access_mode) { + case K3_RINGACC_ACCESS_MODE_POP_HEAD: + case K3_RINGACC_ACCESS_MODE_POP_TAIL: + dev_dbg(ring->parent->dev, + "memcpy_fromio(x): --> ptr(%p), mode:%d\n", ptr, + access_mode); + memcpy_fromio(elem, ptr, (4 << ring->elm_size)); + ring->occ--; + break; + case K3_RINGACC_ACCESS_MODE_PUSH_TAIL: + case K3_RINGACC_ACCESS_MODE_PUSH_HEAD: + dev_dbg(ring->parent->dev, + "memcpy_toio(x): --> ptr(%p), mode:%d\n", ptr, + access_mode); + memcpy_toio(ptr, elem, (4 << ring->elm_size)); + ring->free--; + break; + default: + return -EINVAL; + } + + dev_dbg(ring->parent->dev, "free%d index%d occ%d index%d\n", ring->free, + ring->windex, ring->occ, ring->rindex); + return 0; +} + +static int k3_ringacc_ring_push_head_io(struct k3_ring *ring, void *elem) +{ + return k3_ringacc_ring_access_io(ring, elem, + K3_RINGACC_ACCESS_MODE_PUSH_HEAD); +} + +static int k3_ringacc_ring_push_io(struct k3_ring *ring, void *elem) +{ + return k3_ringacc_ring_access_io(ring, elem, + K3_RINGACC_ACCESS_MODE_PUSH_TAIL); +} + +static int k3_ringacc_ring_pop_io(struct k3_ring *ring, void *elem) +{ + return k3_ringacc_ring_access_io(ring, elem, + K3_RINGACC_ACCESS_MODE_POP_HEAD); +} + +static int k3_ringacc_ring_pop_tail_io(struct k3_ring *ring, void *elem) +{ + return k3_ringacc_ring_access_io(ring, elem, + K3_RINGACC_ACCESS_MODE_POP_HEAD); +} + +static int k3_ringacc_ring_push_mem(struct k3_ring *ring, void *elem) +{ + void *elem_ptr; + + elem_ptr = k3_ringacc_get_elm_addr(ring, ring->windex); + + memcpy(elem_ptr, elem, (4 << ring->elm_size)); + + ring->windex = (ring->windex + 1) % ring->size; + ring->free--; + writel(1, &ring->rt->db); + + dev_dbg(ring->parent->dev, "ring_push_mem: free%d index%d\n", + ring->free, ring->windex); + + return 0; +} + +static int k3_ringacc_ring_pop_mem(struct k3_ring *ring, void *elem) +{ + void *elem_ptr; + + elem_ptr = k3_ringacc_get_elm_addr(ring, ring->rindex); + + memcpy(elem, elem_ptr, (4 << ring->elm_size)); + + ring->rindex = (ring->rindex + 1) % ring->size; + ring->occ--; + writel(-1, &ring->rt->db); + + dev_dbg(ring->parent->dev, "ring_pop_mem: occ%d index%d pos_ptr%p\n", + ring->occ, ring->rindex, elem_ptr); + return 0; +} + +int k3_ringacc_ring_push(struct k3_ring *ring, void *elem) +{ + int ret = -EOPNOTSUPP; + + if (!ring || !(ring->flags & K3_RING_FLAG_BUSY)) + return -EINVAL; + + dev_dbg(ring->parent->dev, "ring_push: free%d index%d\n", ring->free, + ring->windex); + + if (k3_ringacc_ring_is_full(ring)) + return -ENOMEM; + + if (ring->ops && ring->ops->push_tail) + ret = ring->ops->push_tail(ring, elem); + + return ret; +} +EXPORT_SYMBOL_GPL(k3_ringacc_ring_push); + +int k3_ringacc_ring_push_head(struct k3_ring *ring, void *elem) +{ + int ret = -EOPNOTSUPP; + + if (!ring || !(ring->flags & K3_RING_FLAG_BUSY)) + return -EINVAL; + + dev_dbg(ring->parent->dev, "ring_push_head: free%d index%d\n", + ring->free, ring->windex); + + if (k3_ringacc_ring_is_full(ring)) + return -ENOMEM; + + if (ring->ops && ring->ops->push_head) + ret = ring->ops->push_head(ring, elem); + + return ret; +} +EXPORT_SYMBOL_GPL(k3_ringacc_ring_push_head); + +int k3_ringacc_ring_pop(struct k3_ring *ring, void *elem) +{ + int ret = -EOPNOTSUPP; + + if (!ring || !(ring->flags & K3_RING_FLAG_BUSY)) + return -EINVAL; + + if (!ring->occ) + ring->occ = k3_ringacc_ring_get_occ(ring); + + dev_dbg(ring->parent->dev, "ring_pop: occ%d index%d\n", ring->occ, + ring->rindex); + + if (!ring->occ) + return -ENODATA; + + if (ring->ops && ring->ops->pop_head) + ret = ring->ops->pop_head(ring, elem); + + return ret; +} +EXPORT_SYMBOL_GPL(k3_ringacc_ring_pop); + +int k3_ringacc_ring_pop_tail(struct k3_ring *ring, void *elem) +{ + int ret = -EOPNOTSUPP; + + if (!ring || !(ring->flags & K3_RING_FLAG_BUSY)) + return -EINVAL; + + if (!ring->occ) + ring->occ = k3_ringacc_ring_get_occ(ring); + + dev_dbg(ring->parent->dev, "ring_pop_tail: occ%d index%d\n", ring->occ, + ring->rindex); + + if (!ring->occ) + return -ENODATA; + + if (ring->ops && ring->ops->pop_tail) + ret = ring->ops->pop_tail(ring, elem); + + return ret; +} +EXPORT_SYMBOL_GPL(k3_ringacc_ring_pop_tail); + +struct k3_ringacc *of_k3_ringacc_get_by_phandle(struct device_node *np, + const char *property) +{ + struct device_node *ringacc_np; + struct k3_ringacc *ringacc = ERR_PTR(-EPROBE_DEFER); + struct k3_ringacc *entry; + + ringacc_np = of_parse_phandle(np, property, 0); + if (!ringacc_np) + return ERR_PTR(-ENODEV); + + mutex_lock(&k3_ringacc_list_lock); + list_for_each_entry(entry, &k3_ringacc_list, list) + if (entry->dev->of_node == ringacc_np) { + ringacc = entry; + break; + } + mutex_unlock(&k3_ringacc_list_lock); + of_node_put(ringacc_np); + + return ringacc; +} +EXPORT_SYMBOL_GPL(of_k3_ringacc_get_by_phandle); + +static int k3_ringacc_probe_dt(struct k3_ringacc *ringacc) +{ + struct device_node *node = ringacc->dev->of_node; + struct device *dev = ringacc->dev; + struct platform_device *pdev = to_platform_device(dev); + int ret; + + if (!node) { + dev_err(dev, "device tree info unavailable\n"); + return -ENODEV; + } + + ret = of_property_read_u32(node, "ti,num-rings", &ringacc->num_rings); + if (ret) { + dev_err(dev, "ti,num-rings read failure %d\n", ret); + return ret; + } + + ringacc->dma_ring_reset_quirk = + of_property_read_bool(node, "ti,dma-ring-reset-quirk"); + + ringacc->tisci = ti_sci_get_by_phandle(node, "ti,sci"); + if (IS_ERR(ringacc->tisci)) { + ret = PTR_ERR(ringacc->tisci); + if (ret != -EPROBE_DEFER) + dev_err(dev, "ti,sci read fail %d\n", ret); + ringacc->tisci = NULL; + return ret; + } + + ret = of_property_read_u32(node, "ti,sci-dev-id", + &ringacc->tisci_dev_id); + if (ret) { + dev_err(dev, "ti,sci-dev-id read fail %d\n", ret); + return ret; + } + + pdev->id = ringacc->tisci_dev_id; + + ringacc->rm_gp_range = devm_ti_sci_get_of_resource(ringacc->tisci, dev, + ringacc->tisci_dev_id, + "ti,sci-rm-range-gp-rings"); + if (IS_ERR(ringacc->rm_gp_range)) { + dev_err(dev, "Failed to allocate MSI interrupts\n"); + return PTR_ERR(ringacc->rm_gp_range); + } + + return ti_sci_inta_msi_domain_alloc_irqs(ringacc->dev, + ringacc->rm_gp_range); +} + +static int k3_ringacc_probe(struct platform_device *pdev) +{ + struct k3_ringacc *ringacc; + void __iomem *base_fifo, *base_rt; + struct device *dev = &pdev->dev; + struct resource *res; + int ret, i; + + ringacc = devm_kzalloc(dev, sizeof(*ringacc), GFP_KERNEL); + if (!ringacc) + return -ENOMEM; + + ringacc->dev = dev; + mutex_init(&ringacc->req_lock); + + dev->msi_domain = of_msi_get_domain(dev, dev->of_node, + DOMAIN_BUS_TI_SCI_INTA_MSI); + if (!dev->msi_domain) { + dev_err(dev, "Failed to get MSI domain\n"); + return -EPROBE_DEFER; + } + + ret = k3_ringacc_probe_dt(ringacc); + if (ret) + return ret; + + res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "rt"); + base_rt = devm_ioremap_resource(dev, res); + if (IS_ERR(base_rt)) + return PTR_ERR(base_rt); + + res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "fifos"); + base_fifo = devm_ioremap_resource(dev, res); + if (IS_ERR(base_fifo)) + return PTR_ERR(base_fifo); + + res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "proxy_gcfg"); + ringacc->proxy_gcfg = devm_ioremap_resource(dev, res); + if (IS_ERR(ringacc->proxy_gcfg)) + return PTR_ERR(ringacc->proxy_gcfg); + + res = platform_get_resource_byname(pdev, IORESOURCE_MEM, + "proxy_target"); + ringacc->proxy_target_base = devm_ioremap_resource(dev, res); + if (IS_ERR(ringacc->proxy_target_base)) + return PTR_ERR(ringacc->proxy_target_base); + + ringacc->num_proxies = readl(&ringacc->proxy_gcfg->config) & + K3_RINGACC_PROXY_CFG_THREADS_MASK; + + ringacc->rings = devm_kzalloc(dev, + sizeof(*ringacc->rings) * + ringacc->num_rings, + GFP_KERNEL); + ringacc->rings_inuse = devm_kcalloc(dev, + BITS_TO_LONGS(ringacc->num_rings), + sizeof(unsigned long), GFP_KERNEL); + ringacc->proxy_inuse = devm_kcalloc(dev, + BITS_TO_LONGS(ringacc->num_proxies), + sizeof(unsigned long), GFP_KERNEL); + + if (!ringacc->rings || !ringacc->rings_inuse || !ringacc->proxy_inuse) + return -ENOMEM; + + for (i = 0; i < ringacc->num_rings; i++) { + ringacc->rings[i].rt = base_rt + + K3_RINGACC_RT_REGS_STEP * i; + ringacc->rings[i].fifos = base_fifo + + K3_RINGACC_FIFO_REGS_STEP * i; + ringacc->rings[i].parent = ringacc; + ringacc->rings[i].ring_id = i; + ringacc->rings[i].proxy_id = K3_RINGACC_PROXY_NOT_USED; + } + dev_set_drvdata(dev, ringacc); + + ringacc->tisci_ring_ops = &ringacc->tisci->ops.rm_ring_ops; + + mutex_lock(&k3_ringacc_list_lock); + list_add_tail(&ringacc->list, &k3_ringacc_list); + mutex_unlock(&k3_ringacc_list_lock); + + dev_info(dev, "Ring Accelerator probed rings:%u, gp-rings[%u,%u] sci-dev-id:%u\n", + ringacc->num_rings, + ringacc->rm_gp_range->desc[0].start, + ringacc->rm_gp_range->desc[0].num, + ringacc->tisci_dev_id); + dev_info(dev, "dma-ring-reset-quirk: %s\n", + ringacc->dma_ring_reset_quirk ? "enabled" : "disabled"); + dev_info(dev, "RA Proxy rev. %08x, num_proxies:%u\n", + readl(&ringacc->proxy_gcfg->revision), ringacc->num_proxies); + return 0; +} + +static int k3_ringacc_remove(struct platform_device *pdev) +{ + struct k3_ringacc *ringacc = dev_get_drvdata(&pdev->dev); + + mutex_lock(&k3_ringacc_list_lock); + list_del(&ringacc->list); + mutex_unlock(&k3_ringacc_list_lock); + return 0; +} + +/* Match table for of_platform binding */ +static const struct of_device_id k3_ringacc_of_match[] = { + { .compatible = "ti,am654-navss-ringacc", }, + {}, +}; +MODULE_DEVICE_TABLE(of, k3_ringacc_of_match); + +static struct platform_driver k3_ringacc_driver = { + .probe = k3_ringacc_probe, + .remove = k3_ringacc_remove, + .driver = { + .name = "k3-ringacc", + .of_match_table = k3_ringacc_of_match, + }, +}; +module_platform_driver(k3_ringacc_driver); + +MODULE_LICENSE("GPL v2"); +MODULE_DESCRIPTION("TI Ringacc driver for K3 SOCs"); +MODULE_AUTHOR("Grygorii Strashko "); diff --git a/include/linux/soc/ti/k3-ringacc.h b/include/linux/soc/ti/k3-ringacc.h new file mode 100644 index 000000000000..26f73df0a524 --- /dev/null +++ b/include/linux/soc/ti/k3-ringacc.h @@ -0,0 +1,244 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * K3 Ring Accelerator (RA) subsystem interface + * + * Copyright (C) 2019 Texas Instruments Incorporated - http://www.ti.com + */ + +#ifndef __SOC_TI_K3_RINGACC_API_H_ +#define __SOC_TI_K3_RINGACC_API_H_ + +#include + +struct device_node; + +/** + * enum k3_ring_mode - &struct k3_ring_cfg mode + * + * RA ring operational modes + * + * @K3_RINGACC_RING_MODE_RING: Exposed Ring mode for SW direct access + * @K3_RINGACC_RING_MODE_MESSAGE: Messaging mode. Messaging mode requires + * that all accesses to the queue must go through this IP so that all + * accesses to the memory are controlled and ordered. This IP then + * controls the entire state of the queue, and SW has no directly control, + * such as through doorbells and cannot access the storage memory directly. + * This is particularly useful when more than one SW or HW entity can be + * the producer and/or consumer at the same time + * @K3_RINGACC_RING_MODE_CREDENTIALS: Credentials mode is message mode plus + * stores credentials with each message, requiring the element size to be + * doubled to fit the credentials. Any exposed memory should be protected + * by a firewall from unwanted access + */ +enum k3_ring_mode { + K3_RINGACC_RING_MODE_RING = 0, + K3_RINGACC_RING_MODE_MESSAGE, + K3_RINGACC_RING_MODE_CREDENTIALS, + K3_RINGACC_RING_MODE_INVALID +}; + +/** + * enum k3_ring_size - &struct k3_ring_cfg elm_size + * + * RA ring element's sizes in bytes. + */ +enum k3_ring_size { + K3_RINGACC_RING_ELSIZE_4 = 0, + K3_RINGACC_RING_ELSIZE_8, + K3_RINGACC_RING_ELSIZE_16, + K3_RINGACC_RING_ELSIZE_32, + K3_RINGACC_RING_ELSIZE_64, + K3_RINGACC_RING_ELSIZE_128, + K3_RINGACC_RING_ELSIZE_256, + K3_RINGACC_RING_ELSIZE_INVALID +}; + +struct k3_ringacc; +struct k3_ring; + +/** + * enum k3_ring_cfg - RA ring configuration structure + * + * @size: Ring size, number of elements + * @elm_size: Ring element size + * @mode: Ring operational mode + * @flags: Ring configuration flags. Possible values: + * @K3_RINGACC_RING_SHARED: when set allows to request the same ring + * few times. It's usable when the same ring is used as Free Host PD ring + * for different flows, for example. + * Note: Locking should be done by consumer if required + */ +struct k3_ring_cfg { + u32 size; + enum k3_ring_size elm_size; + enum k3_ring_mode mode; +#define K3_RINGACC_RING_SHARED BIT(1) + u32 flags; +}; + +#define K3_RINGACC_RING_ID_ANY (-1) + +/** + * of_k3_ringacc_get_by_phandle - find a RA by phandle property + * @np: device node + * @propname: property name containing phandle on RA node + * + * Returns pointer on the RA - struct k3_ringacc + * or -ENODEV if not found, + * or -EPROBE_DEFER if not yet registered + */ +struct k3_ringacc *of_k3_ringacc_get_by_phandle(struct device_node *np, + const char *property); + +#define K3_RINGACC_RING_USE_PROXY BIT(1) + +/** + * k3_ringacc_request_ring - request ring from ringacc + * @ringacc: pointer on ringacc + * @id: ring id or K3_RINGACC_RING_ID_ANY for any general purpose ring + * @flags: + * @K3_RINGACC_RING_USE_PROXY: if set - proxy will be allocated and + * used to access ring memory. Sopported only for rings in + * Message/Credentials/Queue mode. + * + * Returns pointer on the Ring - struct k3_ring + * or NULL in case of failure. + */ +struct k3_ring *k3_ringacc_request_ring(struct k3_ringacc *ringacc, + int id, u32 flags); + +/** + * k3_ringacc_ring_reset - ring reset + * @ring: pointer on Ring + * + * Resets ring internal state ((hw)occ, (hw)idx). + */ +void k3_ringacc_ring_reset(struct k3_ring *ring); +/** + * k3_ringacc_ring_reset - ring reset for DMA rings + * @ring: pointer on Ring + * + * Resets ring internal state ((hw)occ, (hw)idx). Should be used for rings + * which are read by K3 UDMA, like TX or Free Host PD rings. + */ +void k3_ringacc_ring_reset_dma(struct k3_ring *ring, u32 occ); + +/** + * k3_ringacc_ring_free - ring free + * @ring: pointer on Ring + * + * Resets ring and free all alocated resources. + */ +int k3_ringacc_ring_free(struct k3_ring *ring); + +/** + * k3_ringacc_get_ring_id - Get the Ring ID + * @ring: pointer on ring + * + * Returns the Ring ID + */ +u32 k3_ringacc_get_ring_id(struct k3_ring *ring); + +/** + * k3_ringacc_get_ring_irq_num - Get the irq number for the ring + * @ring: pointer on ring + * + * Returns the interrupt number which can be used to request the interrupt + */ +int k3_ringacc_get_ring_irq_num(struct k3_ring *ring); + +/** + * k3_ringacc_ring_cfg - ring configure + * @ring: pointer on ring + * @cfg: Ring configuration parameters (see &struct k3_ring_cfg) + * + * Configures ring, including ring memory allocation. + * Returns 0 on success, errno otherwise. + */ +int k3_ringacc_ring_cfg(struct k3_ring *ring, struct k3_ring_cfg *cfg); + +/** + * k3_ringacc_ring_get_size - get ring size + * @ring: pointer on ring + * + * Returns ring size in number of elements. + */ +u32 k3_ringacc_ring_get_size(struct k3_ring *ring); + +/** + * k3_ringacc_ring_get_free - get free elements + * @ring: pointer on ring + * + * Returns number of free elements in the ring. + */ +u32 k3_ringacc_ring_get_free(struct k3_ring *ring); + +/** + * k3_ringacc_ring_get_occ - get ring occupancy + * @ring: pointer on ring + * + * Returns total number of valid entries on the ring + */ +u32 k3_ringacc_ring_get_occ(struct k3_ring *ring); + +/** + * k3_ringacc_ring_is_full - checks if ring is full + * @ring: pointer on ring + * + * Returns true if the ring is full + */ +u32 k3_ringacc_ring_is_full(struct k3_ring *ring); + +/** + * k3_ringacc_ring_push - push element to the ring tail + * @ring: pointer on ring + * @elem: pointer on ring element buffer + * + * Push one ring element to the ring tail. Size of the ring element is + * determined by ring configuration &struct k3_ring_cfg elm_size. + * + * Returns 0 on success, errno otherwise. + */ +int k3_ringacc_ring_push(struct k3_ring *ring, void *elem); + +/** + * k3_ringacc_ring_pop - pop element from the ring head + * @ring: pointer on ring + * @elem: pointer on ring element buffer + * + * Push one ring element from the ring head. Size of the ring element is + * determined by ring configuration &struct k3_ring_cfg elm_size.. + * + * Returns 0 on success, errno otherwise. + */ +int k3_ringacc_ring_pop(struct k3_ring *ring, void *elem); + +/** + * k3_ringacc_ring_push_head - push element to the ring head + * @ring: pointer on ring + * @elem: pointer on ring element buffer + * + * Push one ring element to the ring head. Size of the ring element is + * determined by ring configuration &struct k3_ring_cfg elm_size. + * + * Returns 0 on success, errno otherwise. + * Not Supported by ring modes: K3_RINGACC_RING_MODE_RING + */ +int k3_ringacc_ring_push_head(struct k3_ring *ring, void *elem); + +/** + * k3_ringacc_ring_pop_tail - pop element from the ring tail + * @ring: pointer on ring + * @elem: pointer on ring element buffer + * + * Push one ring element from the ring tail. Size of the ring element is + * determined by ring configuration &struct k3_ring_cfg elm_size. + * + * Returns 0 on success, errno otherwise. + * Not Supported by ring modes: K3_RINGACC_RING_MODE_RING + */ +int k3_ringacc_ring_pop_tail(struct k3_ring *ring, void *elem); + +u32 k3_ringacc_get_tisci_dev_id(struct k3_ring *ring); + +#endif /* __SOC_TI_K3_RINGACC_API_H_ */ From patchwork Mon Nov 11 13:53:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Ujfalusi X-Patchwork-Id: 179071 Delivered-To: patch@linaro.org Received: by 2002:ac9:3c86:0:0:0:0:0 with SMTP id w6csp6372581ocf; Mon, 11 Nov 2019 05:53:01 -0800 (PST) X-Google-Smtp-Source: APXvYqzlzHcte06u4Xzp+XYf0eCjuUVCrmcRdUMPaof+IN5cNSBpNo9YW+zkh9J/6U05fq17D2sb X-Received: by 2002:a17:906:7c5:: with SMTP id m5mr20007861ejc.231.1573480381620; Mon, 11 Nov 2019 05:53:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573480381; cv=none; d=google.com; s=arc-20160816; b=XeVG+mco2yIdkgdVwU2nhbTC/3uGOSUe+yM5FvG61LijE8B3jHp5E71kh51E4EZhCZ MH5xudYwFCyB6fiDhb4F9WzrS/zA60nfKjVD3oWECziSS3jyggGngSTShjg+Gr8oj6Sj kfDqkSabgiBquYkSN9o4YbvqNUWqg5FCtdl17i9jGvRCH0TBDaSbo7YzKXd4tftcjNgC Q45Es76KjcEDR3Zey18py3XfPjQ/Fe2K886/bLaJTpfUVi1tmeucs2IR9uuKuB1R5tcM ICIdPuxnIoA68Hi2ujGFzFiVbJiytvqsXpmF52ZbcUr50xJfjyUw22fHYc1V2f6A+vGI 0Pww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=uSViP+QvJDzIv6c8dhx5jMKNUxJQ+VwAKiFTRE1e4co=; b=r19fFGc9ZQfv69sU91IWqBWeY7oEuiLXugzBEswbmyYyQC4Jns95P/3DUmgNW16qTL IOdq8yDI5SB3iln3ugdTluTHH32ejYOtB2hzi5AxmE3/fHlABuVbh/2wpANoLh43kFcD bItNdYJQ210QTafItmSSys3Cfa70JCpZczduRh/7UOj11LBmMYB7D0qm6E3CluJaw7pW XXiU6kCAeqRBbGZwnvggNLGD8oWEdcrvQRFcKtXkByDJ1RZY5x7Ixv8HhmkxIjZDA1sg cTnYiqIEtmhwJWUVGahdp2PVkBCNmd9Pjh3gA/12DzmtZcjDJDUXJVr9bdX4kz+M8kU7 KhbQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ti.com header.s=ti-com-17Q1 header.b="v/6XRVI8"; spf=pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=dmaengine-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=ti.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id dt19si9220660ejb.206.2019.11.11.05.53.01; Mon, 11 Nov 2019 05:53:01 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@ti.com header.s=ti-com-17Q1 header.b="v/6XRVI8"; spf=pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=dmaengine-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=ti.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727192AbfKKNw6 (ORCPT + 3 others); Mon, 11 Nov 2019 08:52:58 -0500 Received: from lelv0142.ext.ti.com ([198.47.23.249]:52062 "EHLO lelv0142.ext.ti.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727186AbfKKNw5 (ORCPT ); Mon, 11 Nov 2019 08:52:57 -0500 Received: from fllv0034.itg.ti.com ([10.64.40.246]) by lelv0142.ext.ti.com (8.15.2/8.15.2) with ESMTP id xABDqULr010446; Mon, 11 Nov 2019 07:52:30 -0600 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1573480350; bh=uSViP+QvJDzIv6c8dhx5jMKNUxJQ+VwAKiFTRE1e4co=; h=From:To:CC:Subject:Date:In-Reply-To:References; b=v/6XRVI80Ry1WPxwcUMcx9u+u467KI8tzpc+m8yFipmN+ynHn8V1UHqslX9s1AWDs MNsR+F7ZSnboEWtW6Fy5OEB2302/+ajmk96hmHRaJ70LpunEgrkQzw0HjHdUQy8T1E qpI7X9vJrQVFW7RrrLWfdID6vkmCKuDMqgxyTP1E= Received: from DLEE100.ent.ti.com (dlee100.ent.ti.com [157.170.170.30]) by fllv0034.itg.ti.com (8.15.2/8.15.2) with ESMTPS id xABDqU0l111142 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 11 Nov 2019 07:52:30 -0600 Received: from DLEE105.ent.ti.com (157.170.170.35) by DLEE100.ent.ti.com (157.170.170.30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1847.3; Mon, 11 Nov 2019 07:52:12 -0600 Received: from fllv0040.itg.ti.com (10.64.41.20) by DLEE105.ent.ti.com (157.170.170.35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1847.3 via Frontend Transport; Mon, 11 Nov 2019 07:52:12 -0600 Received: from feketebors.ti.com (ileax41-snat.itg.ti.com [10.172.224.153]) by fllv0040.itg.ti.com (8.15.2/8.15.2) with ESMTP id xABDqE8o097668; Mon, 11 Nov 2019 07:52:26 -0600 From: Peter Ujfalusi To: , , , CC: , , , , , , , , , Subject: [PATCH v5 03/15] dmaengine: doc: Add sections for per descriptor metadata support Date: Mon, 11 Nov 2019 15:53:18 +0200 Message-ID: <20191111135330.8235-4-peter.ujfalusi@ti.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: <20191111135330.8235-1-peter.ujfalusi@ti.com> References: <20191111135330.8235-1-peter.ujfalusi@ti.com> MIME-Version: 1.0 X-EXCLAIMER-MD-CONFIG: e1e8a2fd-e40a-4ac6-ac9b-f7e9cc9ee180 Sender: dmaengine-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org Update the provider and client documentation with details about the metadata support. Signed-off-by: Peter Ujfalusi Reviewed-by: Tero Kristo --- Documentation/driver-api/dmaengine/client.rst | 75 +++++++++++++++++++ .../driver-api/dmaengine/provider.rst | 46 ++++++++++++ 2 files changed, 121 insertions(+) -- Peter Texas Instruments Finland Oy, Porkkalankatu 22, 00180 Helsinki. Y-tunnus/Business ID: 0615521-4. Kotipaikka/Domicile: Helsinki diff --git a/Documentation/driver-api/dmaengine/client.rst b/Documentation/driver-api/dmaengine/client.rst index 45953f171500..41309c176df4 100644 --- a/Documentation/driver-api/dmaengine/client.rst +++ b/Documentation/driver-api/dmaengine/client.rst @@ -151,6 +151,81 @@ The details of these operations are: Note that callbacks will always be invoked from the DMA engines tasklet, never from interrupt context. + Optional: per descriptor metadata + --------------------------------- + DMAengine provides two ways for metadata support. + + DESC_METADATA_CLIENT + + The metadata buffer is allocated/provided by the client driver and it is + attached to the descriptor. + + .. code-block:: c + + int dmaengine_desc_attach_metadata(struct dma_async_tx_descriptor *desc, + void *data, size_t len); + + DESC_METADATA_ENGINE + + The metadata buffer is allocated/managed by the DMA driver. The client + driver can ask for the pointer, maximum size and the currently used size of + the metadata and can directly update or read it. + + .. code-block:: c + + void *dmaengine_desc_get_metadata_ptr(struct dma_async_tx_descriptor *desc, + size_t *payload_len, size_t *max_len); + + int dmaengine_desc_set_metadata_len(struct dma_async_tx_descriptor *desc, + size_t payload_len); + + Client drivers can query if a given mode is supported with: + + .. code-block:: c + + bool dmaengine_is_metadata_mode_supported(struct dma_chan *chan, + enum dma_desc_metadata_mode mode); + + Depending on the used mode client drivers must follow different flow. + + DESC_METADATA_CLIENT + + - DMA_MEM_TO_DEV / DEV_MEM_TO_MEM: + 1. prepare the descriptor (dmaengine_prep_*) + construct the metadata in the client's buffer + 2. use dmaengine_desc_attach_metadata() to attach the buffer to the + descriptor + 3. submit the transfer + - DMA_DEV_TO_MEM: + 1. prepare the descriptor (dmaengine_prep_*) + 2. use dmaengine_desc_attach_metadata() to attach the buffer to the + descriptor + 3. submit the transfer + 4. when the transfer is completed, the metadata should be available in the + attached buffer + + DESC_METADATA_ENGINE + + - DMA_MEM_TO_DEV / DEV_MEM_TO_MEM: + 1. prepare the descriptor (dmaengine_prep_*) + 2. use dmaengine_desc_get_metadata_ptr() to get the pointer to the + engine's metadata area + 3. update the metadata at the pointer + 4. use dmaengine_desc_set_metadata_len() to tell the DMA engine the + amount of data the client has placed into the metadata buffer + 5. submit the transfer + - DMA_DEV_TO_MEM: + 1. prepare the descriptor (dmaengine_prep_*) + 2. submit the transfer + 3. on transfer completion, use dmaengine_desc_get_metadata_ptr() to get the + pointer to the engine's metadata area + 4. Read out the metadata from the pointer + + .. note:: + + Mixed use of DESC_METADATA_CLIENT / DESC_METADATA_ENGINE is not allowed, + client drivers must use either of the modes per descriptor. + 4. Submit the transaction Once the descriptor has been prepared and the callback information diff --git a/Documentation/driver-api/dmaengine/provider.rst b/Documentation/driver-api/dmaengine/provider.rst index dfc4486b5743..da557c21c619 100644 --- a/Documentation/driver-api/dmaengine/provider.rst +++ b/Documentation/driver-api/dmaengine/provider.rst @@ -247,6 +247,52 @@ after each transfer. In case of a ring buffer, they may loop (DMA_CYCLIC). Addresses pointing to a device's register (e.g. a FIFO) are typically fixed. +Per descriptor metadata support +------------------------------- +Some data movement architecture (DMA controller and peripherals) uses metadata +associated with a transaction. The DMA controller role is to transfer the +payload and the metadata alongside. +The metadata itself is not used by the DMA engine itself, but it contains +parameters, keys, vectors, etc for peripheral or from the peripheral. + +The DMAengine framework provides a generic ways to facilitate the metadata for +descriptors. Depending on the architecture the DMA driver can implement either +or both of the methods and it is up to the client driver to choose which one +to use. + +- DESC_METADATA_CLIENT + + The metadata buffer is allocated/provided by the client driver and it is + attached (via the dmaengine_desc_attach_metadata() helper to the descriptor. + + From the DMA driver the following is expected for this mode: + - DMA_MEM_TO_DEV / DEV_MEM_TO_MEM + The data from the provided metadata buffer should be prepared for the DMA + controller to be sent alongside of the payload data. Either by copying to a + hardware descriptor, or highly coupled packet. + - DMA_DEV_TO_MEM + On transfer completion the DMA driver must copy the metadata to the client + provided metadata buffer. + +- DESC_METADATA_ENGINE + + The metadata buffer is allocated/managed by the DMA driver. The client driver + can ask for the pointer, maximum size and the currently used size of the + metadata and can directly update or read it. dmaengine_desc_get_metadata_ptr() + and dmaengine_desc_set_metadata_len() is provided as helper functions. + + From the DMA driver the following is expected for this mode: + - get_metadata_ptr + Should return a pointer for the metadata buffer, the maximum size of the + metadata buffer and the currently used / valid (if any) bytes in the buffer. + - set_metadata_len + It is called by the clients after it have placed the metadata to the buffer + to let the DMA driver know the number of valid bytes provided. + + Note: since the client will ask for the metadata pointer in the completion + callback (in DMA_DEV_TO_MEM case) the DMA driver must ensure that the + descriptor is not freed up prior the callback is called. + Device operations ----------------- From patchwork Mon Nov 11 13:53:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Ujfalusi X-Patchwork-Id: 179069 Delivered-To: patch@linaro.org Received: by 2002:ac9:3c86:0:0:0:0:0 with SMTP id w6csp6372383ocf; Mon, 11 Nov 2019 05:52:50 -0800 (PST) X-Google-Smtp-Source: APXvYqxdvoYqoJMcJobps7y2wNaEcMQxvgqpLiDUTJsaAE36d4Zn4whwt2mnN9DQanD/63eblLbd X-Received: by 2002:a50:fa4a:: with SMTP id c10mr26290006edq.51.1573480370364; Mon, 11 Nov 2019 05:52:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573480370; cv=none; d=google.com; s=arc-20160816; b=AWJwSbqbOo5CHTUUJAqTGRiPAP7rNwfnyNbl3Li9zm36DTp0d3hjgO9ZZ4w7O1JDlm Fn5oqKm5dTcP3kjFeSDyHr5sftXsxF7nYPlgffkKQ3NLzowTrf33uXG5ORlYPB0uhROX l2cF2s7+mYpaXz8asm0Nr88IlBfA8Dut7Ve2QgSUWD0KXBm1ZvbxGt3rgcwFfX2MTxvd Tn6vaGkiUkBlyvyU3oKoTX29oLozi16hHh3GFdAHxihim0Ps6JN+dPjXUxrWhEJ8cpOF dZn6FWKRgq38LH3haABpi/9WkcPkM6kmBQhwDYGkSucTonZ7X1pvN2iDGipJZm5nZXyu /pWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0nliBJ0aS+ZPJD5/Er1M0lCPZp+R7mh7eBej+Gho4Bw=; b=QV2SQDGFkeqCmdnXoXneTO5GVasLqShTVXZc0geCBnu5+Ip+/Tgs4RpH98SEQ66szh IOS8qzBqiRShYkexMCmCxtjUZSVetSie2qLBzHUu9hrkEHd+SDIb4Y5kad9vE70m7O/o hvAjsKDgC907rcV48jGGM1rsc07PDxHWIBjssArjYGWanJP7A+DvSp49kxnJm8RGH4FW BNwXFvweiX9H8JID26UJ0H9BMaPQO6VaLwSi83TqXXpZJZDVYBjsDySYKCycLh8rnItj MZiZGorRGk2H6I6jBkneB0/vE/ZSKxDWaV6WARhlLCp9ihyZe9VOkdbXlE+M22/3Y7NU CvvQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ti.com header.s=ti-com-17Q1 header.b=Qjva9kUH; spf=pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=dmaengine-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=ti.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h1si11233672edt.144.2019.11.11.05.52.50; Mon, 11 Nov 2019 05:52:50 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@ti.com header.s=ti-com-17Q1 header.b=Qjva9kUH; spf=pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=dmaengine-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=ti.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727124AbfKKNws (ORCPT + 3 others); Mon, 11 Nov 2019 08:52:48 -0500 Received: from lelv0143.ext.ti.com ([198.47.23.248]:43408 "EHLO lelv0143.ext.ti.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726902AbfKKNwr (ORCPT ); Mon, 11 Nov 2019 08:52:47 -0500 Received: from lelv0266.itg.ti.com ([10.180.67.225]) by lelv0143.ext.ti.com (8.15.2/8.15.2) with ESMTP id xABDqXG9013824; Mon, 11 Nov 2019 07:52:33 -0600 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1573480353; bh=0nliBJ0aS+ZPJD5/Er1M0lCPZp+R7mh7eBej+Gho4Bw=; h=From:To:CC:Subject:Date:In-Reply-To:References; b=Qjva9kUHgKEI6lSKAmcOxD87sNFznM2JC0kfyWEtjwydFVzq8neWzXBDNsfkCtsv6 Ad8gZ1RIl83t+iKJ4JKWEi75XnuSOyR+QgjrnQQI00zlDUz6ans2CTG4eU89kYzgVk KYp06rr/N6JJ10A8bldI+MRSfj7Y4HoZmaRXasQA= Received: from DFLE104.ent.ti.com (dfle104.ent.ti.com [10.64.6.25]) by lelv0266.itg.ti.com (8.15.2/8.15.2) with ESMTPS id xABDqX2r079536 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 11 Nov 2019 07:52:33 -0600 Received: from DFLE114.ent.ti.com (10.64.6.35) by DFLE104.ent.ti.com (10.64.6.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1847.3; Mon, 11 Nov 2019 07:52:16 -0600 Received: from fllv0040.itg.ti.com (10.64.41.20) by DFLE114.ent.ti.com (10.64.6.35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1847.3 via Frontend Transport; Mon, 11 Nov 2019 07:52:16 -0600 Received: from feketebors.ti.com (ileax41-snat.itg.ti.com [10.172.224.153]) by fllv0040.itg.ti.com (8.15.2/8.15.2) with ESMTP id xABDqE8p097668; Mon, 11 Nov 2019 07:52:30 -0600 From: Peter Ujfalusi To: , , , CC: , , , , , , , , , Subject: [PATCH v5 04/15] dmaengine: Add metadata_ops for dma_async_tx_descriptor Date: Mon, 11 Nov 2019 15:53:19 +0200 Message-ID: <20191111135330.8235-5-peter.ujfalusi@ti.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: <20191111135330.8235-1-peter.ujfalusi@ti.com> References: <20191111135330.8235-1-peter.ujfalusi@ti.com> MIME-Version: 1.0 X-EXCLAIMER-MD-CONFIG: e1e8a2fd-e40a-4ac6-ac9b-f7e9cc9ee180 Sender: dmaengine-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org The metadata is best described as side band data or parameters traveling alongside the data DMAd by the DMA engine. It is data which is understood by the peripheral and the peripheral driver only, the DMA engine see it only as data block and it is not interpreting it in any way. The metadata can be different per descriptor as it is a parameter for the data being transferred. If the DMA supports per descriptor metadata it can implement the attach, get_ptr/set_len callbacks. Client drivers must only use either attach or get_ptr/set_len to avoid misconfiguration. Client driver can check if a given metadata mode is supported by the channel during probe time with dmaengine_is_metadata_mode_supported(chan, DESC_METADATA_CLIENT); dmaengine_is_metadata_mode_supported(chan, DESC_METADATA_ENGINE); and based on this information can use either mode. Wrappers are also added for the metadata_ops. To be used in DESC_METADATA_CLIENT mode: dmaengine_desc_attach_metadata() To be used in DESC_METADATA_ENGINE mode: dmaengine_desc_get_metadata_ptr() dmaengine_desc_set_metadata_len() Signed-off-by: Peter Ujfalusi Reviewed-by: Tero Kristo --- drivers/dma/dmaengine.c | 73 ++++++++++++++++++++++++++ include/linux/dmaengine.h | 108 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 181 insertions(+) -- Peter Texas Instruments Finland Oy, Porkkalankatu 22, 00180 Helsinki. Y-tunnus/Business ID: 0615521-4. Kotipaikka/Domicile: Helsinki diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c index 03ac4b96117c..6baddf7dcbfd 100644 --- a/drivers/dma/dmaengine.c +++ b/drivers/dma/dmaengine.c @@ -1302,6 +1302,79 @@ void dma_async_tx_descriptor_init(struct dma_async_tx_descriptor *tx, } EXPORT_SYMBOL(dma_async_tx_descriptor_init); +static inline int desc_check_and_set_metadata_mode( + struct dma_async_tx_descriptor *desc, enum dma_desc_metadata_mode mode) +{ + /* Make sure that the metadata mode is not mixed */ + if (!desc->desc_metadata_mode) { + if (dmaengine_is_metadata_mode_supported(desc->chan, mode)) + desc->desc_metadata_mode = mode; + else + return -ENOTSUPP; + } else if (desc->desc_metadata_mode != mode) { + return -EINVAL; + } + + return 0; +} + +int dmaengine_desc_attach_metadata(struct dma_async_tx_descriptor *desc, + void *data, size_t len) +{ + int ret; + + if (!desc) + return -EINVAL; + + ret = desc_check_and_set_metadata_mode(desc, DESC_METADATA_CLIENT); + if (ret) + return ret; + + if (!desc->metadata_ops || !desc->metadata_ops->attach) + return -ENOTSUPP; + + return desc->metadata_ops->attach(desc, data, len); +} +EXPORT_SYMBOL_GPL(dmaengine_desc_attach_metadata); + +void *dmaengine_desc_get_metadata_ptr(struct dma_async_tx_descriptor *desc, + size_t *payload_len, size_t *max_len) +{ + int ret; + + if (!desc) + return ERR_PTR(-EINVAL); + + ret = desc_check_and_set_metadata_mode(desc, DESC_METADATA_ENGINE); + if (ret) + return ERR_PTR(ret); + + if (!desc->metadata_ops || !desc->metadata_ops->get_ptr) + return ERR_PTR(-ENOTSUPP); + + return desc->metadata_ops->get_ptr(desc, payload_len, max_len); +} +EXPORT_SYMBOL_GPL(dmaengine_desc_get_metadata_ptr); + +int dmaengine_desc_set_metadata_len(struct dma_async_tx_descriptor *desc, + size_t payload_len) +{ + int ret; + + if (!desc) + return -EINVAL; + + ret = desc_check_and_set_metadata_mode(desc, DESC_METADATA_ENGINE); + if (ret) + return ret; + + if (!desc->metadata_ops || !desc->metadata_ops->set_len) + return -ENOTSUPP; + + return desc->metadata_ops->set_len(desc, payload_len); +} +EXPORT_SYMBOL_GPL(dmaengine_desc_set_metadata_len); + /* dma_wait_for_async_tx - spin wait for a transaction to complete * @tx: in-flight transaction to wait on */ diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h index 8fcdee1c0cf9..0e8b426bbde9 100644 --- a/include/linux/dmaengine.h +++ b/include/linux/dmaengine.h @@ -219,6 +219,58 @@ typedef struct { DECLARE_BITMAP(bits, DMA_TX_TYPE_END); } dma_cap_mask_t; * @bytes_transferred: byte counter */ +/** + * enum dma_desc_metadata_mode - per descriptor metadata mode types supported + * @DESC_METADATA_CLIENT - the metadata buffer is allocated/provided by the + * client driver and it is attached (via the dmaengine_desc_attach_metadata() + * helper) to the descriptor. + * + * Client drivers interested to use this mode can follow: + * - DMA_MEM_TO_DEV / DEV_MEM_TO_MEM: + * 1. prepare the descriptor (dmaengine_prep_*) + * construct the metadata in the client's buffer + * 2. use dmaengine_desc_attach_metadata() to attach the buffer to the + * descriptor + * 3. submit the transfer + * - DMA_DEV_TO_MEM: + * 1. prepare the descriptor (dmaengine_prep_*) + * 2. use dmaengine_desc_attach_metadata() to attach the buffer to the + * descriptor + * 3. submit the transfer + * 4. when the transfer is completed, the metadata should be available in the + * attached buffer + * + * @DESC_METADATA_ENGINE - the metadata buffer is allocated/managed by the DMA + * driver. The client driver can ask for the pointer, maximum size and the + * currently used size of the metadata and can directly update or read it. + * dmaengine_desc_get_metadata_ptr() and dmaengine_desc_set_metadata_len() is + * provided as helper functions. + * + * Client drivers interested to use this mode can follow: + * - DMA_MEM_TO_DEV / DEV_MEM_TO_MEM: + * 1. prepare the descriptor (dmaengine_prep_*) + * 2. use dmaengine_desc_get_metadata_ptr() to get the pointer to the engine's + * metadata area + * 3. update the metadata at the pointer + * 4. use dmaengine_desc_set_metadata_len() to tell the DMA engine the amount + * of data the client has placed into the metadata buffer + * 5. submit the transfer + * - DMA_DEV_TO_MEM: + * 1. prepare the descriptor (dmaengine_prep_*) + * 2. submit the transfer + * 3. on transfer completion, use dmaengine_desc_get_metadata_ptr() to get the + * pointer to the engine's metadata area + * 4. Read out the metadata from the pointer + * + * Note: the two mode is not compatible and clients must use one mode for a + * descriptor. + */ +enum dma_desc_metadata_mode { + DESC_METADATA_NONE = 0, + DESC_METADATA_CLIENT = BIT(0), + DESC_METADATA_ENGINE = BIT(1), +}; + struct dma_chan_percpu { /* stats */ unsigned long memcpy_count; @@ -475,6 +527,18 @@ struct dmaengine_unmap_data { dma_addr_t addr[0]; }; +struct dma_async_tx_descriptor; + +struct dma_descriptor_metadata_ops { + int (*attach)(struct dma_async_tx_descriptor *desc, void *data, + size_t len); + + void *(*get_ptr)(struct dma_async_tx_descriptor *desc, + size_t *payload_len, size_t *max_len); + int (*set_len)(struct dma_async_tx_descriptor *desc, + size_t payload_len); +}; + /** * struct dma_async_tx_descriptor - async transaction descriptor * ---dma generic offload fields--- @@ -488,6 +552,11 @@ struct dmaengine_unmap_data { * descriptor pending. To be pushed on .issue_pending() call * @callback: routine to call after this operation is complete * @callback_param: general parameter to pass to the callback routine + * @desc_metadata_mode: core managed metadata mode to protect mixed use of + * DESC_METADATA_CLIENT or DESC_METADATA_ENGINE. Otherwise + * DESC_METADATA_NONE + * @metadata_ops: DMA driver provided metadata mode ops, need to be set by the + * DMA driver if metadata mode is supported with the descriptor * ---async_tx api specific fields--- * @next: at completion submit this descriptor * @parent: pointer to the next level up in the dependency chain @@ -504,6 +573,8 @@ struct dma_async_tx_descriptor { dma_async_tx_callback_result callback_result; void *callback_param; struct dmaengine_unmap_data *unmap; + enum dma_desc_metadata_mode desc_metadata_mode; + struct dma_descriptor_metadata_ops *metadata_ops; #ifdef CONFIG_ASYNC_TX_ENABLE_CHANNEL_SWITCH struct dma_async_tx_descriptor *next; struct dma_async_tx_descriptor *parent; @@ -666,6 +737,7 @@ struct dma_filter { * @global_node: list_head for global dma_device_list * @filter: information for device/slave to filter function/param mapping * @cap_mask: one or more dma_capability flags + * @desc_metadata_modes: supported metadata modes by the DMA device * @max_xor: maximum number of xor sources, 0 if no capability * @max_pq: maximum number of PQ sources and PQ-continue capability * @copy_align: alignment shift for memcpy operations @@ -727,6 +799,7 @@ struct dma_device { struct list_head global_node; struct dma_filter filter; dma_cap_mask_t cap_mask; + enum dma_desc_metadata_mode desc_metadata_modes; unsigned short max_xor; unsigned short max_pq; enum dmaengine_alignment copy_align; @@ -902,6 +975,41 @@ static inline struct dma_async_tx_descriptor *dmaengine_prep_dma_memcpy( len, flags); } +static inline bool dmaengine_is_metadata_mode_supported(struct dma_chan *chan, + enum dma_desc_metadata_mode mode) +{ + if (!chan) + return false; + + return !!(chan->device->desc_metadata_modes & mode); +} + +#ifdef CONFIG_DMA_ENGINE +int dmaengine_desc_attach_metadata(struct dma_async_tx_descriptor *desc, + void *data, size_t len); +void *dmaengine_desc_get_metadata_ptr(struct dma_async_tx_descriptor *desc, + size_t *payload_len, size_t *max_len); +int dmaengine_desc_set_metadata_len(struct dma_async_tx_descriptor *desc, + size_t payload_len); +#else /* CONFIG_DMA_ENGINE */ +static inline int dmaengine_desc_attach_metadata( + struct dma_async_tx_descriptor *desc, void *data, size_t len) +{ + return -EINVAL; +} +static inline void *dmaengine_desc_get_metadata_ptr( + struct dma_async_tx_descriptor *desc, size_t *payload_len, + size_t *max_len) +{ + return NULL; +} +static inline int dmaengine_desc_set_metadata_len( + struct dma_async_tx_descriptor *desc, size_t payload_len) +{ + return -EINVAL; +} +#endif /* CONFIG_DMA_ENGINE */ + /** * dmaengine_terminate_all() - Terminate all active DMA transfers * @chan: The channel for which to terminate the transfers From patchwork Mon Nov 11 13:53:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Ujfalusi X-Patchwork-Id: 179070 Delivered-To: patch@linaro.org Received: by 2002:ac9:3c86:0:0:0:0:0 with SMTP id w6csp6372478ocf; Mon, 11 Nov 2019 05:52:55 -0800 (PST) X-Google-Smtp-Source: APXvYqxLneNaBVaEizEZ6pGZCFppOMsfNr5/QoS58BwDc+FG/YYDMNkIlIM3jmp0JS79GGGBuJin X-Received: by 2002:a50:b558:: with SMTP id z24mr26637629edd.67.1573480375516; Mon, 11 Nov 2019 05:52:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573480375; cv=none; d=google.com; s=arc-20160816; b=ct5+24QVFN5jlrU7eAY9tKm2MTs38QAorgdmSxrT6xX+Yq4XfTpuBQO1EDrXPE9vEJ tYXfg3kwr7iBHq6G8gyxA+IjnxazbPMNWt9fj+b+fXHUM7SMSXEgYZGMxWySSeXcLCE5 PFoTAzMhzTt08ev8ev4ECmis4I+HIo7p5tWboYzD9urfwxggN2XHAm525sAdOCCSV9Ho V62vUto2XBcCZICwt4syWSGNbfv1qGVrmmWb0IpDz5Q8DgneUPk9rxrhQnfAewdYBjCK Y7S/VvjHJ+w8e7IOI8th9z27UyNw8KJ73WpQdcp5CkvkEsEkyejqKQvoMcQH7qlImxY4 mOYg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=rTdWRsMQAqUSYWttDpcgbgQCSwD5mLTBJmv5WLQ/N7A=; b=Ay236LHJd6JYuwRp0VV4e3XDF8s63J86AfeRc2T508jkSlfrhMEcxnz3VcULws7uuZ H+aSHUwa36M1ZNpHi412d7WzHEwUDou90GrIPbwrhZMPahwEgqBHqoQ+gRo9SH/6Efvz 1dNaoGEwNtitGvoOnsShYuXE3vU8zt/CtTWbj1E4LAZjmXB/x9BhkioaJKAMmFzvkhhD JpkqCyBDV5BIQmvBsrhUya43Cr6TrVb7DFL24TmAuPavvN74bBtRNm3BDEgKd1JZEQ3t ym2r3BBv186FBNCgIpVGDSswqrqMCcRZzKr3bm1oZpDWZ/+ECb4kNMeHB/iR5l5RgYB8 /QAA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ti.com header.s=ti-com-17Q1 header.b=UMvMTJmi; spf=pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=dmaengine-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=ti.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h1si11233672edt.144.2019.11.11.05.52.55; Mon, 11 Nov 2019 05:52:55 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@ti.com header.s=ti-com-17Q1 header.b=UMvMTJmi; spf=pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=dmaengine-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=ti.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727166AbfKKNwy (ORCPT + 3 others); Mon, 11 Nov 2019 08:52:54 -0500 Received: from fllv0016.ext.ti.com ([198.47.19.142]:54242 "EHLO fllv0016.ext.ti.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726902AbfKKNwy (ORCPT ); Mon, 11 Nov 2019 08:52:54 -0500 Received: from fllv0035.itg.ti.com ([10.64.41.0]) by fllv0016.ext.ti.com (8.15.2/8.15.2) with ESMTP id xABDqfla059656; Mon, 11 Nov 2019 07:52:41 -0600 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1573480361; bh=rTdWRsMQAqUSYWttDpcgbgQCSwD5mLTBJmv5WLQ/N7A=; h=From:To:CC:Subject:Date:In-Reply-To:References; b=UMvMTJmi5L7S1Kk3BnE15gkoJMQh7Q9Y/UzR4P0TXWayloJaLLy9ou7zToPxOje2m 6bJU8ATHZK33D/e09K08MKanwJvYT4HM/3HFkdDOvueQXbi4SZC/tS/BF8qUUBI0G7 +dWUfD9oGwX8Bd7GiMXUopgFgzEEPA+KfNh3LuBE= Received: from DFLE105.ent.ti.com (dfle105.ent.ti.com [10.64.6.26]) by fllv0035.itg.ti.com (8.15.2/8.15.2) with ESMTP id xABDqfD3045730; Mon, 11 Nov 2019 07:52:41 -0600 Received: from DFLE105.ent.ti.com (10.64.6.26) by DFLE105.ent.ti.com (10.64.6.26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1847.3; Mon, 11 Nov 2019 07:52:24 -0600 Received: from fllv0040.itg.ti.com (10.64.41.20) by DFLE105.ent.ti.com (10.64.6.26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1847.3 via Frontend Transport; Mon, 11 Nov 2019 07:52:24 -0600 Received: from feketebors.ti.com (ileax41-snat.itg.ti.com [10.172.224.153]) by fllv0040.itg.ti.com (8.15.2/8.15.2) with ESMTP id xABDqE8r097668; Mon, 11 Nov 2019 07:52:37 -0600 From: Peter Ujfalusi To: , , , CC: , , , , , , , , , Subject: [PATCH v5 06/15] dmaengine: ti: Add cppi5 header for K3 NAVSS/UDMA Date: Mon, 11 Nov 2019 15:53:21 +0200 Message-ID: <20191111135330.8235-7-peter.ujfalusi@ti.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: <20191111135330.8235-1-peter.ujfalusi@ti.com> References: <20191111135330.8235-1-peter.ujfalusi@ti.com> MIME-Version: 1.0 X-EXCLAIMER-MD-CONFIG: e1e8a2fd-e40a-4ac6-ac9b-f7e9cc9ee180 Sender: dmaengine-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org The K3 DMA architecture uses CPPI5 (Communications Port Programming Interface) specified descriptors over PSI-L bus within NAVSS. The header provides helpers, macros to work with these descriptors in a consistent way. Signed-off-by: Peter Ujfalusi --- include/linux/dma/ti-cppi5.h | 1049 ++++++++++++++++++++++++++++++++++ 1 file changed, 1049 insertions(+) create mode 100644 include/linux/dma/ti-cppi5.h -- Peter Texas Instruments Finland Oy, Porkkalankatu 22, 00180 Helsinki. Y-tunnus/Business ID: 0615521-4. Kotipaikka/Domicile: Helsinki diff --git a/include/linux/dma/ti-cppi5.h b/include/linux/dma/ti-cppi5.h new file mode 100644 index 000000000000..f795f8cb7cc5 --- /dev/null +++ b/include/linux/dma/ti-cppi5.h @@ -0,0 +1,1049 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * CPPI5 descriptors interface + * + * Copyright (C) 2019 Texas Instruments Incorporated - http://www.ti.com + */ + +#ifndef __TI_CPPI5_H__ +#define __TI_CPPI5_H__ + +#include +#include +#include + +/** + * struct cppi5_desc_hdr_t - Descriptor header, present in all types of + * descriptors + * @pkt_info0: Packet info word 0 (n/a in Buffer desc) + * @pkt_info0: Packet info word 1 (n/a in Buffer desc) + * @pkt_info0: Packet info word 2 (n/a in Buffer desc) + * @src_dst_tag: Packet info word 3 (n/a in Buffer desc) + */ +struct cppi5_desc_hdr_t { + u32 pkt_info0; + u32 pkt_info1; + u32 pkt_info2; + u32 src_dst_tag; +} __packed; + +/** + * struct cppi5_host_desc_t - Host-mode packet and buffer descriptor definition + * @hdr: Descriptor header + * @next_desc: word 4/5: Linking word + * @buf_ptr: word 6/7: Buffer pointer + * @buf_info1: word 8: Buffer valid data length + * @org_buf_len: word 9: Original buffer length + * @org_buf_ptr: word 10/11: Original buffer pointer + * @epib[0]: Extended Packet Info Data (optional, 4 words), and/or + * Protocol Specific Data (optional, 0-128 bytes in + * multiples of 4), and/or + * Other Software Data (0-N bytes, optional) + */ +struct cppi5_host_desc_t { + struct cppi5_desc_hdr_t hdr; + u64 next_desc; + u64 buf_ptr; + u32 buf_info1; + u32 org_buf_len; + u64 org_buf_ptr; + u32 epib[0]; +} __packed; + +#define CPPI5_DESC_MIN_ALIGN (16U) + +#define CPPI5_INFO0_HDESC_EPIB_SIZE (16U) +#define CPPI5_INFO0_HDESC_PSDATA_MAX_SIZE (128U) + +#define CPPI5_INFO0_HDESC_TYPE_SHIFT (30U) +#define CPPI5_INFO0_HDESC_TYPE_MASK GENMASK(31, 30) +#define CPPI5_INFO0_DESC_TYPE_VAL_HOST (1U) +#define CPPI5_INFO0_DESC_TYPE_VAL_MONO (2U) +#define CPPI5_INFO0_DESC_TYPE_VAL_TR (3U) +#define CPPI5_INFO0_HDESC_EPIB_PRESENT BIT(29) +/* + * Protocol Specific Words location: + * 0 - located in the descriptor, + * 1 = located in the SOP Buffer immediately prior to the data. + */ +#define CPPI5_INFO0_HDESC_PSINFO_LOCATION BIT(28) +#define CPPI5_INFO0_HDESC_PSINFO_SIZE_SHIFT (22U) +#define CPPI5_INFO0_HDESC_PSINFO_SIZE_MASK GENMASK(27, 22) +#define CPPI5_INFO0_HDESC_PKTLEN_SHIFT (0) +#define CPPI5_INFO0_HDESC_PKTLEN_MASK GENMASK(21, 0) + +#define CPPI5_INFO1_DESC_PKTERROR_SHIFT (28U) +#define CPPI5_INFO1_DESC_PKTERROR_MASK GENMASK(31, 28) +#define CPPI5_INFO1_HDESC_PSFLGS_SHIFT (24U) +#define CPPI5_INFO1_HDESC_PSFLGS_MASK GENMASK(27, 24) +#define CPPI5_INFO1_DESC_PKTID_SHIFT (14U) +#define CPPI5_INFO1_DESC_PKTID_MASK GENMASK(23, 14) +#define CPPI5_INFO1_DESC_FLOWID_SHIFT (0) +#define CPPI5_INFO1_DESC_FLOWID_MASK GENMASK(13, 0) + +#define CPPI5_INFO2_HDESC_PKTTYPE_SHIFT (27U) +#define CPPI5_INFO2_HDESC_PKTTYPE_MASK GENMASK(31, 27) +/* Return Policy: 0 - Entire packet 1 - Each buffer */ +#define CPPI5_INFO2_HDESC_RETPOLICY BIT(18) +/* + * Early Return: + * 0 = desc pointers should be returned after all reads have been completed + * 1 = desc pointers should be returned immediately upon fetching + * the descriptor and beginning to transfer data. + */ +#define CPPI5_INFO2_HDESC_EARLYRET BIT(17) +/* + * Return Push Policy: + * 0 = Descriptor must be returned to tail of queue + * 1 = Descriptor must be returned to head of queue + */ +#define CPPI5_INFO2_DESC_RETPUSHPOLICY BIT(16) +#define CPPI5_INFO2_DESC_RETQ_SHIFT (0) +#define CPPI5_INFO2_DESC_RETQ_MASK GENMASK(15, 0) + +#define CPPI5_INFO3_DESC_SRCTAG_SHIFT (16U) +#define CPPI5_INFO3_DESC_SRCTAG_MASK GENMASK(31, 16) +#define CPPI5_INFO3_DESC_DSTTAG_SHIFT (0) +#define CPPI5_INFO3_DESC_DSTTAG_MASK GENMASK(15, 0) + +#define CPPI5_BUFINFO1_HDESC_DATA_LEN_SHIFT (0) +#define CPPI5_BUFINFO1_HDESC_DATA_LEN_MASK GENMASK(27, 0) + +#define CPPI5_OBUFINFO0_HDESC_BUF_LEN_SHIFT (0) +#define CPPI5_OBUFINFO0_HDESC_BUF_LEN_MASK GENMASK(27, 0) + +/** + * struct cppi5_desc_epib_t - Host Packet Descriptor Extended Packet Info Block + * @timestamp: word 0: application specific timestamp + * @sw_info0: word 1: Software Info 0 + * @sw_info1: word 1: Software Info 1 + * @sw_info2: word 1: Software Info 2 + */ +struct cppi5_desc_epib_t { + u32 timestamp; /* w0: application specific timestamp */ + u32 sw_info0; /* w1: Software Info 0 */ + u32 sw_info1; /* w2: Software Info 1 */ + u32 sw_info2; /* w3: Software Info 2 */ +}; + +/** + * struct cppi5_monolithic_desc_t - Monolithic-mode packet descriptor + * @hdr: Descriptor header + * @epib[0]: Extended Packet Info Data (optional, 4 words), and/or + * Protocol Specific Data (optional, 0-128 bytes in + * multiples of 4), and/or + * Other Software Data (0-N bytes, optional) + */ +struct cppi5_monolithic_desc_t { + struct cppi5_desc_hdr_t hdr; + u32 epib[0]; +}; + +#define CPPI5_INFO2_MDESC_DATA_OFFSET_SHIFT (18U) +#define CPPI5_INFO2_MDESC_DATA_OFFSET_MASK GENMASK(26, 18) + +/* + * Reload Count: + * 0 = Finish the packet and place the descriptor back on the return queue + * 1-0x1ff = Vector to the Reload Index and resume processing + * 0x1ff indicates perpetual loop, infinite reload until the channel is stopped + */ +#define CPPI5_INFO0_TRDESC_RLDCNT_SHIFT (20U) +#define CPPI5_INFO0_TRDESC_RLDCNT_MASK GENMASK(28, 20) +#define CPPI5_INFO0_TRDESC_RLDCNT_MAX (0x1ff) +#define CPPI5_INFO0_TRDESC_RLDCNT_INFINITE CPPI5_INFO0_TRDESC_RLDCNT_MAX +#define CPPI5_INFO0_TRDESC_RLDIDX_SHIFT (14U) +#define CPPI5_INFO0_TRDESC_RLDIDX_MASK GENMASK(19, 14) +#define CPPI5_INFO0_TRDESC_RLDIDX_MAX (0x3f) +#define CPPI5_INFO0_TRDESC_LASTIDX_SHIFT (0) +#define CPPI5_INFO0_TRDESC_LASTIDX_MASK GENMASK(13, 0) + +#define CPPI5_INFO1_TRDESC_RECSIZE_SHIFT (24U) +#define CPPI5_INFO1_TRDESC_RECSIZE_MASK GENMASK(26, 24) +#define CPPI5_INFO1_TRDESC_RECSIZE_VAL_16B (0) +#define CPPI5_INFO1_TRDESC_RECSIZE_VAL_32B (1U) +#define CPPI5_INFO1_TRDESC_RECSIZE_VAL_64B (2U) +#define CPPI5_INFO1_TRDESC_RECSIZE_VAL_128B (3U) + +static inline void cppi5_desc_dump(void *desc, u32 size) +{ + print_hex_dump(KERN_ERR, "dump udmap_desc: ", DUMP_PREFIX_NONE, + 32, 4, desc, size, false); +} + +#define CPPI5_TDCM_MARKER (0x1) +/** + * cppi5_desc_is_tdcm - check if the paddr indicates Teardown Complete Message + * @paddr: Physical address of the packet popped from the ring + * + * Returns true if the address indicates TDCM + */ +static inline bool cppi5_desc_is_tdcm(dma_addr_t paddr) +{ + return (paddr & CPPI5_TDCM_MARKER) ? true : false; +} + +/** + * cppi5_desc_get_type - get descriptor type + * @desc_hdr: packet descriptor/TR header + * + * Returns descriptor type: + * CPPI5_INFO0_DESC_TYPE_VAL_HOST + * CPPI5_INFO0_DESC_TYPE_VAL_MONO + * CPPI5_INFO0_DESC_TYPE_VAL_TR + */ +static inline u32 cppi5_desc_get_type(struct cppi5_desc_hdr_t *desc_hdr) +{ + return (desc_hdr->pkt_info0 & CPPI5_INFO0_HDESC_TYPE_MASK) >> + CPPI5_INFO0_HDESC_TYPE_SHIFT; +} + +/** + * cppi5_desc_get_errflags - get Error Flags from Desc + * @desc_hdr: packet/TR descriptor header + * + * Returns Error Flags from Packet/TR Descriptor + */ +static inline u32 cppi5_desc_get_errflags(struct cppi5_desc_hdr_t *desc_hdr) +{ + return (desc_hdr->pkt_info1 & CPPI5_INFO1_DESC_PKTERROR_MASK) >> + CPPI5_INFO1_DESC_PKTERROR_SHIFT; +} + +/** + * cppi5_desc_get_pktids - get Packet and Flow ids from Desc + * @desc_hdr: packet/TR descriptor header + * @pkt_id: Packet ID + * @flow_id: Flow ID + * + * Returns Packet and Flow ids from packet/TR descriptor + */ +static inline void cppi5_desc_get_pktids(struct cppi5_desc_hdr_t *desc_hdr, + u32 *pkt_id, u32 *flow_id) +{ + *pkt_id = (desc_hdr->pkt_info1 & CPPI5_INFO1_DESC_PKTID_MASK) >> + CPPI5_INFO1_DESC_PKTID_SHIFT; + *flow_id = (desc_hdr->pkt_info1 & CPPI5_INFO1_DESC_FLOWID_MASK) >> + CPPI5_INFO1_DESC_FLOWID_SHIFT; +} + +/** + * cppi5_desc_set_pktids - set Packet and Flow ids in Desc + * @desc_hdr: packet/TR descriptor header + * @pkt_id: Packet ID + * @flow_id: Flow ID + */ +static inline void cppi5_desc_set_pktids(struct cppi5_desc_hdr_t *desc_hdr, + u32 pkt_id, u32 flow_id) +{ + desc_hdr->pkt_info1 |= (pkt_id << CPPI5_INFO1_DESC_PKTID_SHIFT) & + CPPI5_INFO1_DESC_PKTID_MASK; + desc_hdr->pkt_info1 |= (flow_id << CPPI5_INFO1_DESC_FLOWID_SHIFT) & + CPPI5_INFO1_DESC_FLOWID_MASK; +} + +/** + * cppi5_desc_set_retpolicy - set Packet Return Policy in Desc + * @desc_hdr: packet/TR descriptor header + * @flags: fags, supported values + * CPPI5_INFO2_HDESC_RETPOLICY + * CPPI5_INFO2_HDESC_EARLYRET + * CPPI5_INFO2_DESC_RETPUSHPOLICY + * @return_ring_id: Packet Return Queue/Ring id, value 0xFFFF reserved + */ +static inline void cppi5_desc_set_retpolicy(struct cppi5_desc_hdr_t *desc_hdr, + u32 flags, u32 return_ring_id) +{ + desc_hdr->pkt_info2 |= flags; + desc_hdr->pkt_info2 |= return_ring_id & CPPI5_INFO2_DESC_RETQ_MASK; +} + +/** + * cppi5_desc_get_tags_ids - get Packet Src/Dst Tags from Desc + * @desc_hdr: packet/TR descriptor header + * @src_tag_id: Source Tag + * @dst_tag_id: Dest Tag + * + * Returns Packet Src/Dst Tags from packet/TR descriptor + */ +static inline void cppi5_desc_get_tags_ids(struct cppi5_desc_hdr_t *desc_hdr, + u32 *src_tag_id, u32 *dst_tag_id) +{ + if (src_tag_id) + *src_tag_id = (desc_hdr->src_dst_tag & + CPPI5_INFO3_DESC_SRCTAG_MASK) >> + CPPI5_INFO3_DESC_SRCTAG_SHIFT; + if (dst_tag_id) + *dst_tag_id = desc_hdr->src_dst_tag & + CPPI5_INFO3_DESC_DSTTAG_MASK; +} + +/** + * cppi5_desc_set_tags_ids - set Packet Src/Dst Tags in HDesc + * @desc_hdr: packet/TR descriptor header + * @src_tag_id: Source Tag + * @dst_tag_id: Dest Tag + * + * Returns Packet Src/Dst Tags from packet/TR descriptor + */ +static inline void cppi5_desc_set_tags_ids(struct cppi5_desc_hdr_t *desc_hdr, + u32 src_tag_id, u32 dst_tag_id) +{ + desc_hdr->src_dst_tag = (src_tag_id << CPPI5_INFO3_DESC_SRCTAG_SHIFT) & + CPPI5_INFO3_DESC_SRCTAG_MASK; + desc_hdr->src_dst_tag |= dst_tag_id & CPPI5_INFO3_DESC_DSTTAG_MASK; +} + +/** + * cppi5_hdesc_calc_size - Calculate Host Packet Descriptor size + * @epib: is EPIB present + * @psdata_size: PSDATA size + * @sw_data_size: SWDATA size + * + * Returns required Host Packet Descriptor size + * 0 - if PSDATA > CPPI5_INFO0_HDESC_PSDATA_MAX_SIZE + */ +static inline u32 cppi5_hdesc_calc_size(bool epib, u32 psdata_size, + u32 sw_data_size) +{ + u32 desc_size; + + if (psdata_size > CPPI5_INFO0_HDESC_PSDATA_MAX_SIZE) + return 0; + + desc_size = sizeof(struct cppi5_host_desc_t) + psdata_size + + sw_data_size; + + if (epib) + desc_size += CPPI5_INFO0_HDESC_EPIB_SIZE; + + return ALIGN(desc_size, CPPI5_DESC_MIN_ALIGN); +} + +/** + * cppi5_hdesc_init - Init Host Packet Descriptor size + * @desc: Host packet descriptor + * @flags: supported values + * CPPI5_INFO0_HDESC_EPIB_PRESENT + * CPPI5_INFO0_HDESC_PSINFO_LOCATION + * @psdata_size: PSDATA size + * + * Returns required Host Packet Descriptor size + * 0 - if PSDATA > CPPI5_INFO0_HDESC_PSDATA_MAX_SIZE + */ +static inline void cppi5_hdesc_init(struct cppi5_host_desc_t *desc, u32 flags, + u32 psdata_size) +{ + desc->hdr.pkt_info0 = (CPPI5_INFO0_DESC_TYPE_VAL_HOST << + CPPI5_INFO0_HDESC_TYPE_SHIFT) | (flags); + desc->hdr.pkt_info0 |= ((psdata_size >> 2) << + CPPI5_INFO0_HDESC_PSINFO_SIZE_SHIFT) & + CPPI5_INFO0_HDESC_PSINFO_SIZE_MASK; + desc->next_desc = 0; +} + +/** + * cppi5_hdesc_update_flags - Replace descriptor flags + * @desc: Host packet descriptor + * @flags: supported values + * CPPI5_INFO0_HDESC_EPIB_PRESENT + * CPPI5_INFO0_HDESC_PSINFO_LOCATION + */ +static inline void cppi5_hdesc_update_flags(struct cppi5_host_desc_t *desc, + u32 flags) +{ + desc->hdr.pkt_info0 &= ~(CPPI5_INFO0_HDESC_EPIB_PRESENT | + CPPI5_INFO0_HDESC_PSINFO_LOCATION); + desc->hdr.pkt_info0 |= flags; +} + +/** + * cppi5_hdesc_update_psdata_size - Replace PSdata size + * @desc: Host packet descriptor + * @psdata_size: PSDATA size + */ +static inline void cppi5_hdesc_update_psdata_size( + struct cppi5_host_desc_t *desc, u32 psdata_size) +{ + desc->hdr.pkt_info0 &= ~CPPI5_INFO0_HDESC_PSINFO_SIZE_MASK; + desc->hdr.pkt_info0 |= ((psdata_size >> 2) << + CPPI5_INFO0_HDESC_PSINFO_SIZE_SHIFT) & + CPPI5_INFO0_HDESC_PSINFO_SIZE_MASK; +} + +/** + * cppi5_hdesc_get_psdata_size - get PSdata size in bytes + * @desc: Host packet descriptor + */ +static inline u32 cppi5_hdesc_get_psdata_size(struct cppi5_host_desc_t *desc) +{ + u32 psdata_size = 0; + + if (!(desc->hdr.pkt_info0 & CPPI5_INFO0_HDESC_PSINFO_LOCATION)) + psdata_size = (desc->hdr.pkt_info0 & + CPPI5_INFO0_HDESC_PSINFO_SIZE_MASK) >> + CPPI5_INFO0_HDESC_PSINFO_SIZE_SHIFT; + + return (psdata_size << 2); +} + +/** + * cppi5_hdesc_get_pktlen - get Packet Length from HDesc + * @desc: Host packet descriptor + * + * Returns Packet Length from Host Packet Descriptor + */ +static inline u32 cppi5_hdesc_get_pktlen(struct cppi5_host_desc_t *desc) +{ + return (desc->hdr.pkt_info0 & CPPI5_INFO0_HDESC_PKTLEN_MASK); +} + +/** + * cppi5_hdesc_set_pktlen - set Packet Length in HDesc + * @desc: Host packet descriptor + */ +static inline void cppi5_hdesc_set_pktlen(struct cppi5_host_desc_t *desc, + u32 pkt_len) +{ + desc->hdr.pkt_info0 |= (pkt_len & CPPI5_INFO0_HDESC_PKTLEN_MASK); +} + +/** + * cppi5_hdesc_get_psflags - get Protocol Specific Flags from HDesc + * @desc: Host packet descriptor + * + * Returns Protocol Specific Flags from Host Packet Descriptor + */ +static inline u32 cppi5_hdesc_get_psflags(struct cppi5_host_desc_t *desc) +{ + return (desc->hdr.pkt_info1 & CPPI5_INFO1_HDESC_PSFLGS_MASK) >> + CPPI5_INFO1_HDESC_PSFLGS_SHIFT; +} + +/** + * cppi5_hdesc_set_psflags - set Protocol Specific Flags in HDesc + * @desc: Host packet descriptor + */ +static inline void cppi5_hdesc_set_psflags(struct cppi5_host_desc_t *desc, + u32 ps_flags) +{ + desc->hdr.pkt_info1 |= (ps_flags << + CPPI5_INFO1_HDESC_PSFLGS_SHIFT) & + CPPI5_INFO1_HDESC_PSFLGS_MASK; +} + +/** + * cppi5_hdesc_get_errflags - get Packet Type from HDesc + * @desc: Host packet descriptor + */ +static inline u32 cppi5_hdesc_get_pkttype(struct cppi5_host_desc_t *desc) +{ + return (desc->hdr.pkt_info2 & CPPI5_INFO2_HDESC_PKTTYPE_MASK) >> + CPPI5_INFO2_HDESC_PKTTYPE_SHIFT; +} + +/** + * cppi5_hdesc_get_errflags - set Packet Type in HDesc + * @desc: Host packet descriptor + * @pkt_type: Packet Type + */ +static inline void cppi5_hdesc_set_pkttype(struct cppi5_host_desc_t *desc, + u32 pkt_type) +{ + desc->hdr.pkt_info2 |= + (pkt_type << CPPI5_INFO2_HDESC_PKTTYPE_SHIFT) & + CPPI5_INFO2_HDESC_PKTTYPE_MASK; +} + +/** + * cppi5_hdesc_attach_buf - attach buffer to HDesc + * @desc: Host packet descriptor + * @buf: Buffer physical address + * @buf_data_len: Buffer length + * @obuf: Original Buffer physical address + * @obuf_len: Original Buffer length + * + * Attaches buffer to Host Packet Descriptor + */ +static inline void cppi5_hdesc_attach_buf(struct cppi5_host_desc_t *desc, + dma_addr_t buf, u32 buf_data_len, + dma_addr_t obuf, u32 obuf_len) +{ + desc->buf_ptr = buf; + desc->buf_info1 = buf_data_len & CPPI5_BUFINFO1_HDESC_DATA_LEN_MASK; + desc->org_buf_ptr = obuf; + desc->org_buf_len = obuf_len & CPPI5_OBUFINFO0_HDESC_BUF_LEN_MASK; +} + +static inline void cppi5_hdesc_get_obuf(struct cppi5_host_desc_t *desc, + dma_addr_t *obuf, u32 *obuf_len) +{ + *obuf = desc->org_buf_ptr; + *obuf_len = desc->org_buf_len & CPPI5_OBUFINFO0_HDESC_BUF_LEN_MASK; +} + +static inline void cppi5_hdesc_reset_to_original(struct cppi5_host_desc_t *desc) +{ + desc->buf_ptr = desc->org_buf_ptr; + desc->buf_info1 = desc->org_buf_len; +} + +/** + * cppi5_hdesc_link_hbdesc - link Host Buffer Descriptor to HDesc + * @desc: Host Packet Descriptor + * @buf_desc: Host Buffer Descriptor physical address + * + * add and link Host Buffer Descriptor to HDesc + */ +static inline void cppi5_hdesc_link_hbdesc(struct cppi5_host_desc_t *desc, + dma_addr_t hbuf_desc) +{ + desc->next_desc = hbuf_desc; +} + +static inline dma_addr_t cppi5_hdesc_get_next_hbdesc( + struct cppi5_host_desc_t *desc) +{ + return (dma_addr_t)desc->next_desc; +} + +static inline void cppi5_hdesc_reset_hbdesc(struct cppi5_host_desc_t *desc) +{ + desc->hdr = (struct cppi5_desc_hdr_t) { 0 }; + desc->next_desc = 0; +} + +/** + * cppi5_hdesc_epib_present - check if EPIB present + * @desc_hdr: packet descriptor/TR header + * + * Returns true if EPIB present in the packet + */ +static inline bool cppi5_hdesc_epib_present(struct cppi5_desc_hdr_t *desc_hdr) +{ + return !!(desc_hdr->pkt_info0 & CPPI5_INFO0_HDESC_EPIB_PRESENT); +} + +/** + * cppi5_hdesc_get_psdata - Get pointer on PSDATA + * @desc: Host packet descriptor + * + * Returns pointer on PSDATA in HDesc. + * NULL - if ps_data placed at the start of data buffer. + */ +static inline void *cppi5_hdesc_get_psdata(struct cppi5_host_desc_t *desc) +{ + u32 psdata_size; + void *psdata; + + if (desc->hdr.pkt_info0 & CPPI5_INFO0_HDESC_PSINFO_LOCATION) + return NULL; + + psdata_size = (desc->hdr.pkt_info0 & + CPPI5_INFO0_HDESC_PSINFO_SIZE_MASK) >> + CPPI5_INFO0_HDESC_PSINFO_SIZE_SHIFT; + + if (!psdata_size) + return NULL; + + psdata = &desc->epib; + + if (cppi5_hdesc_epib_present(&desc->hdr)) + psdata += CPPI5_INFO0_HDESC_EPIB_SIZE; + + return psdata; +} + +static inline u32 *cppi5_hdesc_get_psdata32(struct cppi5_host_desc_t *desc) +{ + return (u32 *)cppi5_hdesc_get_psdata(desc); +} + +/** + * cppi5_hdesc_get_swdata - Get pointer on swdata + * @desc: Host packet descriptor + * + * Returns pointer on SWDATA in HDesc. + * NOTE. It's caller responsibility to be sure hdesc actually has swdata. + */ +static inline void *cppi5_hdesc_get_swdata(struct cppi5_host_desc_t *desc) +{ + u32 psdata_size = 0; + void *swdata; + + if (!(desc->hdr.pkt_info0 & CPPI5_INFO0_HDESC_PSINFO_LOCATION)) + psdata_size = (desc->hdr.pkt_info0 & + CPPI5_INFO0_HDESC_PSINFO_SIZE_MASK) >> + CPPI5_INFO0_HDESC_PSINFO_SIZE_SHIFT; + + swdata = &desc->epib; + + if (cppi5_hdesc_epib_present(&desc->hdr)) + swdata += CPPI5_INFO0_HDESC_EPIB_SIZE; + + swdata += (psdata_size << 2); + + return swdata; +} + +/* ================================== TR ================================== */ + +#define CPPI5_TR_TYPE_SHIFT (0U) +#define CPPI5_TR_TYPE_MASK GENMASK(3, 0) +#define CPPI5_TR_STATIC BIT(4) +#define CPPI5_TR_WAIT BIT(5) +#define CPPI5_TR_EVENT_SIZE_SHIFT (6U) +#define CPPI5_TR_EVENT_SIZE_MASK GENMASK(7, 6) +#define CPPI5_TR_TRIGGER0_SHIFT (8U) +#define CPPI5_TR_TRIGGER0_MASK GENMASK(9, 8) +#define CPPI5_TR_TRIGGER0_TYPE_SHIFT (10U) +#define CPPI5_TR_TRIGGER0_TYPE_MASK GENMASK(11, 10) +#define CPPI5_TR_TRIGGER1_SHIFT (12U) +#define CPPI5_TR_TRIGGER1_MASK GENMASK(13, 12) +#define CPPI5_TR_TRIGGER1_TYPE_SHIFT (14U) +#define CPPI5_TR_TRIGGER1_TYPE_MASK GENMASK(15, 14) +#define CPPI5_TR_CMD_ID_SHIFT (16U) +#define CPPI5_TR_CMD_ID_MASK GENMASK(23, 16) +#define CPPI5_TR_CSF_FLAGS_SHIFT (24U) +#define CPPI5_TR_CSF_FLAGS_MASK GENMASK(31, 24) +#define CPPI5_TR_CSF_SA_INDIRECT BIT(0) +#define CPPI5_TR_CSF_DA_INDIRECT BIT(1) +#define CPPI5_TR_CSF_SUPR_EVT BIT(2) +#define CPPI5_TR_CSF_EOL_ADV_SHIFT (4U) +#define CPPI5_TR_CSF_EOL_ADV_MASK GENMASK(6, 4) +#define CPPI5_TR_CSF_EOP BIT(7) + +/** + * enum cppi5_tr_types - TR types + * @CPPI5_TR_TYPE0: One dimensional data move + * @CPPI5_TR_TYPE1: Two dimensional data move + * @CPPI5_TR_TYPE2: Three dimensional data move + * @CPPI5_TR_TYPE3: Four dimensional data move + * @CPPI5_TR_TYPE4: Four dimensional data move with data formatting + * @CPPI5_TR_TYPE5: Four dimensional Cache Warm + * @CPPI5_TR_TYPE8: Four Dimensional Block Move + * @CPPI5_TR_TYPE9: Four Dimensional Block Move with Repacking + * @CPPI5_TR_TYPE10: Two Dimensional Block Move + * @CPPI5_TR_TYPE11: Two Dimensional Block Move with Repacking + * @CPPI5_TR_TYPE15: Four Dimensional Block Move with Repacking and + * Indirection + */ +enum cppi5_tr_types { + CPPI5_TR_TYPE0 = 0, + CPPI5_TR_TYPE1, + CPPI5_TR_TYPE2, + CPPI5_TR_TYPE3, + CPPI5_TR_TYPE4, + CPPI5_TR_TYPE5, + /* type6-7: Reserved */ + CPPI5_TR_TYPE8 = 8, + CPPI5_TR_TYPE9, + CPPI5_TR_TYPE10, + CPPI5_TR_TYPE11, + /* type12-14: Reserved */ + CPPI5_TR_TYPE15 = 15, + CPPI5_TR_TYPE_MAX +}; + +/** + * enum cppi5_tr_event_size - TR Flags EVENT_SIZE field specifies when an event + * is generated for each TR. + * @CPPI5_TR_EVENT_SIZE_COMPLETION: When TR is complete and all status for + * the TR has been received + * @CPPI5_TR_EVENT_SIZE_ICNT1_DEC: Type 0: when the last data transaction + * is sent for the TR + * Type 1-11: when ICNT1 is decremented + * @CPPI5_TR_EVENT_SIZE_ICNT2_DEC: Type 0-1,10-11: when the last data + * transaction is sent for the TR + * All other types: when ICNT2 is + * decremented + * @CPPI5_TR_EVENT_SIZE_ICNT3_DEC: Type 0-2,10-11: when the last data + * transaction is sent for the TR + * All other types: when ICNT3 is + * decremented + */ +enum cppi5_tr_event_size { + CPPI5_TR_EVENT_SIZE_COMPLETION, + CPPI5_TR_EVENT_SIZE_ICNT1_DEC, + CPPI5_TR_EVENT_SIZE_ICNT2_DEC, + CPPI5_TR_EVENT_SIZE_ICNT3_DEC, + CPPI5_TR_EVENT_SIZE_MAX +}; + +/** + * enum cppi5_tr_trigger - TR Flags TRIGGERx field specifies the type of trigger + * used to enable the TR to transfer data as specified + * by TRIGGERx_TYPE field. + * @CPPI5_TR_TRIGGER_NONE: No trigger + * @CPPI5_TR_TRIGGER_GLOBAL0: Global trigger 0 + * @CPPI5_TR_TRIGGER_GLOBAL1: Global trigger 1 + * @CPPI5_TR_TRIGGER_LOCAL_EVENT: Local Event + */ +enum cppi5_tr_trigger { + CPPI5_TR_TRIGGER_NONE, + CPPI5_TR_TRIGGER_GLOBAL0, + CPPI5_TR_TRIGGER_GLOBAL1, + CPPI5_TR_TRIGGER_LOCAL_EVENT, + CPPI5_TR_TRIGGER_MAX +}; + +/** + * enum cppi5_tr_trigger_type - TR Flags TRIGGERx_TYPE field specifies the type + * of data transfer that will be enabled by + * receiving a trigger as specified by TRIGGERx. + * @CPPI5_TR_TRIGGER_TYPE_ICNT1_DEC: The second inner most loop (ICNT1) will + * be decremented by 1 + * @CPPI5_TR_TRIGGER_TYPE_ICNT2_DEC: The third inner most loop (ICNT2) will + * be decremented by 1 + * @CPPI5_TR_TRIGGER_TYPE_ICNT3_DEC: The outer most loop (ICNT3) will be + * decremented by 1 + * @CPPI5_TR_TRIGGER_TYPE_ALL: The entire TR will be allowed to + * complete + */ +enum cppi5_tr_trigger_type { + CPPI5_TR_TRIGGER_TYPE_ICNT1_DEC, + CPPI5_TR_TRIGGER_TYPE_ICNT2_DEC, + CPPI5_TR_TRIGGER_TYPE_ICNT3_DEC, + CPPI5_TR_TRIGGER_TYPE_ALL, + CPPI5_TR_TRIGGER_TYPE_MAX +}; + +typedef u32 cppi5_tr_flags_t; + +/** + * struct cppi5_tr_type0_t - Type 0 (One dimensional data move) TR (16 byte) + * @flags: TR flags (type, triggers, event, configuration) + * @icnt0: Total loop iteration count for level 0 (innermost) + * @_reserved: Not used + * @addr: Starting address for the source data or destination data + */ +struct cppi5_tr_type0_t { + cppi5_tr_flags_t flags; + u16 icnt0; + u16 _reserved; + u64 addr; +} __aligned(16) __packed; + +/** + * struct cppi5_tr_type1_t - Type 1 (Two dimensional data move) TR (32 byte) + * @flags: TR flags (type, triggers, event, configuration) + * @icnt0: Total loop iteration count for level 0 (innermost) + * @icnt1: Total loop iteration count for level 1 + * @addr: Starting address for the source data or destination data + * @dim1: Signed dimension for loop level 1 + */ +struct cppi5_tr_type1_t { + cppi5_tr_flags_t flags; + u16 icnt0; + u16 icnt1; + u64 addr; + s32 dim1; +} __aligned(32) __packed; + +/** + * struct cppi5_tr_type2_t - Type 2 (Three dimensional data move) TR (32 byte) + * @flags: TR flags (type, triggers, event, configuration) + * @icnt0: Total loop iteration count for level 0 (innermost) + * @icnt1: Total loop iteration count for level 1 + * @addr: Starting address for the source data or destination data + * @dim1: Signed dimension for loop level 1 + * @icnt2: Total loop iteration count for level 2 + * @_reserved: Not used + * @dim2: Signed dimension for loop level 2 + */ +struct cppi5_tr_type2_t { + cppi5_tr_flags_t flags; + u16 icnt0; + u16 icnt1; + u64 addr; + s32 dim1; + u16 icnt2; + u16 _reserved; + s32 dim2; +} __aligned(32) __packed; + +/** + * struct cppi5_tr_type3_t - Type 3 (Four dimensional data move) TR (32 byte) + * @flags: TR flags (type, triggers, event, configuration) + * @icnt0: Total loop iteration count for level 0 (innermost) + * @icnt1: Total loop iteration count for level 1 + * @addr: Starting address for the source data or destination data + * @dim1: Signed dimension for loop level 1 + * @icnt2: Total loop iteration count for level 2 + * @icnt3: Total loop iteration count for level 3 (outermost) + * @dim2: Signed dimension for loop level 2 + * @dim3: Signed dimension for loop level 3 + */ +struct cppi5_tr_type3_t { + cppi5_tr_flags_t flags; + u16 icnt0; + u16 icnt1; + u64 addr; + s32 dim1; + u16 icnt2; + u16 icnt3; + s32 dim2; + s32 dim3; +} __aligned(32) __packed; + +/** + * struct cppi5_tr_type15_t - Type 15 (Four Dimensional Block Copy with + * Repacking and Indirection Support) TR (64 byte) + * @flags: TR flags (type, triggers, event, configuration) + * @icnt0: Total loop iteration count for level 0 (innermost) for + * source + * @icnt1: Total loop iteration count for level 1 for source + * @addr: Starting address for the source data + * @dim1: Signed dimension for loop level 1 for source + * @icnt2: Total loop iteration count for level 2 for source + * @icnt3: Total loop iteration count for level 3 (outermost) for + * source + * @dim2: Signed dimension for loop level 2 for source + * @dim3: Signed dimension for loop level 3 for source + * @_reserved: Not used + * @ddim1: Signed dimension for loop level 1 for destination + * @daddr: Starting address for the destination data + * @ddim2: Signed dimension for loop level 2 for destination + * @ddim3: Signed dimension for loop level 3 for destination + * @dicnt0: Total loop iteration count for level 0 (innermost) for + * destination + * @dicnt1: Total loop iteration count for level 1 for destination + * @dicnt2: Total loop iteration count for level 2 for destination + * @sicnt3: Total loop iteration count for level 3 (outermost) for + * destination + */ +struct cppi5_tr_type15_t { + cppi5_tr_flags_t flags; + u16 icnt0; + u16 icnt1; + u64 addr; + s32 dim1; + u16 icnt2; + u16 icnt3; + s32 dim2; + s32 dim3; + u32 _reserved; + s32 ddim1; + u64 daddr; + s32 ddim2; + s32 ddim3; + u16 dicnt0; + u16 dicnt1; + u16 dicnt2; + u16 dicnt3; +} __aligned(64) __packed; + +/** + * struct cppi5_tr_resp_t - TR response record + * @status: Status type and info + * @_reserved: Not used + * @cmd_id: Command ID for the TR for TR identification + * @flags: Configuration Specific Flags + */ +struct cppi5_tr_resp_t { + u8 status; + u8 _reserved; + u8 cmd_id; + u8 flags; +} __packed; + +#define CPPI5_TR_RESPONSE_STATUS_TYPE_SHIFT (0U) +#define CPPI5_TR_RESPONSE_STATUS_TYPE_MASK GENMASK(3, 0) +#define CPPI5_TR_RESPONSE_STATUS_INFO_SHIFT (4U) +#define CPPI5_TR_RESPONSE_STATUS_INFO_MASK GENMASK(7, 4) +#define CPPI5_TR_RESPONSE_CMDID_SHIFT (16U) +#define CPPI5_TR_RESPONSE_CMDID_MASK GENMASK(23, 16) +#define CPPI5_TR_RESPONSE_CFG_SPECIFIC_SHIFT (24U) +#define CPPI5_TR_RESPONSE_CFG_SPECIFIC_MASK GENMASK(31, 24) + +/** + * enum cppi5_tr_resp_status_type - TR Response Status Type field is used to + * determine what type of status is being + * returned. + * @CPPI5_TR_RESPONSE_STATUS_NONE: No error, completion: completed + * @CPPI5_TR_RESPONSE_STATUS_TRANSFER_ERR: Transfer Error, completion: none + * or partially completed + * @CPPI5_TR_RESPONSE_STATUS_ABORTED_ERR: Aborted Error, completion: none + * or partially completed + * @CPPI5_TR_RESPONSE_STATUS_SUBMISSION_ERR: Submission Error, completion: + * none + * @CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_ERR: Unsupported Error, completion: + * none + * @CPPI5_TR_RESPONSE_STATUS_TRANSFER_EXCEPTION: Transfer Exception, completion: + * partially completed + * @CPPI5_TR_RESPONSE_STATUS__TEARDOWN_FLUSH: Teardown Flush, completion: none + */ +enum cppi5_tr_resp_status_type { + CPPI5_TR_RESPONSE_STATUS_NONE, + CPPI5_TR_RESPONSE_STATUS_TRANSFER_ERR, + CPPI5_TR_RESPONSE_STATUS_ABORTED_ERR, + CPPI5_TR_RESPONSE_STATUS_SUBMISSION_ERR, + CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_ERR, + CPPI5_TR_RESPONSE_STATUS_TRANSFER_EXCEPTION, + CPPI5_TR_RESPONSE_STATUS__TEARDOWN_FLUSH, + CPPI5_TR_RESPONSE_STATUS_MAX +}; + +/** + * enum cppi5_tr_resp_status_submission - TR Response Status field values which + * corresponds Submission Error + * @CPPI5_TR_RESPONSE_STATUS_SUBMISSION_ICNT0: ICNT0 was 0 + * @CPPI5_TR_RESPONSE_STATUS_SUBMISSION_FIFO_FULL: Channel FIFO was full when TR + * received + * @CPPI5_TR_RESPONSE_STATUS_SUBMISSION_OWN: Channel is not owned by the + * submitter + */ +enum cppi5_tr_resp_status_submission { + CPPI5_TR_RESPONSE_STATUS_SUBMISSION_ICNT0, + CPPI5_TR_RESPONSE_STATUS_SUBMISSION_FIFO_FULL, + CPPI5_TR_RESPONSE_STATUS_SUBMISSION_OWN, + CPPI5_TR_RESPONSE_STATUS_SUBMISSION_MAX +}; + +/** + * enum cppi5_tr_resp_status_unsupported - TR Response Status field values which + * corresponds Unsupported Error + * @CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_TR_TYPE: TR Type not supported + * @CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_STATIC: STATIC not supported + * @CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_EOL: EOL not supported + * @CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_CFG_SPECIFIC: CONFIGURATION SPECIFIC + * not supported + * @CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_AMODE: AMODE not supported + * @CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_ELTYPE: ELTYPE not supported + * @CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_DFMT: DFMT not supported + * @CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_SECTR: SECTR not supported + * @CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_AMODE_SPECIFIC: AMODE SPECIFIC field + * not supported + */ +enum cppi5_tr_resp_status_unsupported { + CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_TR_TYPE, + CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_STATIC, + CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_EOL, + CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_CFG_SPECIFIC, + CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_AMODE, + CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_ELTYPE, + CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_DFMT, + CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_SECTR, + CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_AMODE_SPECIFIC, + CPPI5_TR_RESPONSE_STATUS_UNSUPPORTED_MAX +}; + +/** + * cppi5_trdesc_calc_size - Calculate TR Descriptor size + * @tr_count: number of TR records + * @tr_size: Nominal size of TR record (max) [16, 32, 64, 128] + * + * Returns required TR Descriptor size + */ +static inline size_t cppi5_trdesc_calc_size(u32 tr_count, u32 tr_size) +{ + /* + * The Size of a TR descriptor is: + * 1 x tr_size : the first 16 bytes is used by the packet info block + + * tr_count x tr_size : Transfer Request Records + + * tr_count x sizeof(struct cppi5_tr_resp_t) : Transfer Response Records + */ + return tr_size * (tr_count + 1) + + sizeof(struct cppi5_tr_resp_t) * tr_count; +} + +/** + * cppi5_trdesc_init - Init TR Descriptor + * @desc: TR Descriptor + * @tr_count: number of TR records + * @tr_size: Nominal size of TR record (max) [16, 32, 64, 128] + * @reload_idx: Absolute index to jump to on the 2nd and following passes + * through the TR packet. + * @reload_count: Number of times to jump from last entry to reload_idx. 0x1ff + * indicates infinite looping. + * + * Init TR Descriptor + */ +static inline void cppi5_trdesc_init(struct cppi5_desc_hdr_t *desc_hdr, + u32 tr_count, u32 tr_size, u32 reload_idx, + u32 reload_count) +{ + desc_hdr->pkt_info0 = CPPI5_INFO0_DESC_TYPE_VAL_TR << + CPPI5_INFO0_HDESC_TYPE_SHIFT; + desc_hdr->pkt_info0 |= (reload_count << CPPI5_INFO0_TRDESC_RLDCNT_SHIFT) & + CPPI5_INFO0_TRDESC_RLDCNT_MASK; + desc_hdr->pkt_info0 |= (reload_idx << CPPI5_INFO0_TRDESC_RLDIDX_SHIFT) & + CPPI5_INFO0_TRDESC_RLDIDX_MASK; + desc_hdr->pkt_info0 |= (tr_count - 1) & CPPI5_INFO0_TRDESC_LASTIDX_MASK; + + desc_hdr->pkt_info1 |= ((ffs(tr_size >> 4) - 1) << + CPPI5_INFO1_TRDESC_RECSIZE_SHIFT) & + CPPI5_INFO1_TRDESC_RECSIZE_MASK; +} + +/** + * cppi5_tr_init - Init TR record + * @flags: Pointer to the TR's flags + * @type: TR type + * @static_tr: TR is static + * @wait: Wait for TR completion before allow the next TR to start + * @event_size: output event generation cfg + * @cmd_id: TR identifier (application specifics) + * + * Init TR record + */ +static inline void cppi5_tr_init(cppi5_tr_flags_t *flags, + enum cppi5_tr_types type, bool static_tr, + bool wait, enum cppi5_tr_event_size event_size, + u32 cmd_id) +{ + *flags = type; + *flags |= (event_size << CPPI5_TR_EVENT_SIZE_SHIFT) & + CPPI5_TR_EVENT_SIZE_MASK; + + *flags |= (cmd_id << CPPI5_TR_CMD_ID_SHIFT) & + CPPI5_TR_CMD_ID_MASK; + + if (static_tr && (type == CPPI5_TR_TYPE8 || type == CPPI5_TR_TYPE9)) + *flags |= CPPI5_TR_STATIC; + + if (wait) + *flags |= CPPI5_TR_WAIT; +} + +/** + * cppi5_tr_set_trigger - Configure trigger0/1 and trigger0/1_type + * @flags: Pointer to the TR's flags + * @trigger0: trigger0 selection + * @trigger0_type: type of data transfer that will be enabled by trigger0 + * @trigger1: trigger1 selection + * @trigger1_type: type of data transfer that will be enabled by trigger1 + * + * Configure the triggers for the TR + */ +static inline void cppi5_tr_set_trigger(cppi5_tr_flags_t *flags, + enum cppi5_tr_trigger trigger0, + enum cppi5_tr_trigger_type trigger0_type, + enum cppi5_tr_trigger trigger1, + enum cppi5_tr_trigger_type trigger1_type) +{ + *flags |= (trigger0 << CPPI5_TR_TRIGGER0_SHIFT) & + CPPI5_TR_TRIGGER0_MASK; + *flags |= (trigger0_type << CPPI5_TR_TRIGGER0_TYPE_SHIFT) & + CPPI5_TR_TRIGGER0_TYPE_MASK; + + *flags |= (trigger1 << CPPI5_TR_TRIGGER1_SHIFT) & + CPPI5_TR_TRIGGER1_MASK; + *flags |= (trigger1_type << CPPI5_TR_TRIGGER1_TYPE_SHIFT) & + CPPI5_TR_TRIGGER1_TYPE_MASK; +} + +/** + * cppi5_tr_cflag_set - Update the Configuration specific flags + * @flags: Pointer to the TR's flags + * @csf: Configuration specific flags + * + * Set a bit in Configuration Specific Flags section of the TR flags. + */ +static inline void cppi5_tr_csf_set(cppi5_tr_flags_t *flags, u32 csf) +{ + *flags |= (csf << CPPI5_TR_CSF_FLAGS_SHIFT) & + CPPI5_TR_CSF_FLAGS_MASK; +} + +#endif /* __TI_CPPI5_H__ */ From patchwork Mon Nov 11 13:53:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Peter Ujfalusi X-Patchwork-Id: 179078 Delivered-To: patch@linaro.org Received: by 2002:ac9:3c86:0:0:0:0:0 with SMTP id w6csp6372954ocf; Mon, 11 Nov 2019 05:53:23 -0800 (PST) X-Google-Smtp-Source: APXvYqyi3HoNP/w98iBVm7gf91tafOy8n9FuiPvCdWH1nG0f+NAuJYPgYzuG8cyMhJQn5pUlHp4M X-Received: by 2002:a17:906:4c97:: with SMTP id q23mr22150951eju.78.1573480403491; Mon, 11 Nov 2019 05:53:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573480403; cv=none; d=google.com; s=arc-20160816; b=PwvQO1UoG0ucHmgn1Cxk5KYSLuPo9AzzBDQ0Mo3NmazqFgZHLSs5gCM4do3M8zWdz5 VQg3Wr/08ZMe36uCgFFBHlwHSb6+azW7E8VI3Mu1ZKQ/GdYQ4BHdgmr63GdDmjIQ7KgZ HHWjki7DZ1X2tEVyxk4lRow0CLM2LyhHHGLDzmEEth46dYcXm1bEU2Bs6dzOFWHzmynN oud/Kph+53hj187QSGBet74cIZt1T0w4bTFkjiLEVIatVncRznDkTBaQnEDokUbbpJcN Y+ZgWopnEM6IbYCvJTQ1+ULJEGagHi/rJsUR4OKEzBmI8WE2SAxeaxhtofnFWDK9PoE5 WndQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ohRHfztl/6cwF/ARblvPlIwfCKujALnmx6Fu/d0o7cU=; b=lReas7VQBR56jbK54wXvRTvoybrII3S8btYS+O7BIpQpARttlOIn/qR5JDELgpiSs2 KvDIXSIVQYJKJm901mGJWsKpPbS2zw2lgyiBijAEdERr0G63mhG3xHlMaMtUL3Lo3Nio UkedOnTgZyxRJyaSAA9Kp4Gif8pqj09Y68tymdiid7K4b6pFQx3xEyak/gjU00UPTwHN p6js+a2xPIGtMk1cgIbaD/rtp24Qc6PrUecNJHpY9YEN5FyZGekkjok4janD0zIvcBBI pwAp4Q71gWVhChojxXVQ67CF/HXXjDT/xKNFYwBiACnLQkvA45FLQrz34D1fkgck1bHK vfyg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ti.com header.s=ti-com-17Q1 header.b=JaU2tcB3; spf=pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=dmaengine-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=ti.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id d1si9100393ejh.281.2019.11.11.05.53.23; Mon, 11 Nov 2019 05:53:23 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@ti.com header.s=ti-com-17Q1 header.b=JaU2tcB3; spf=pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=dmaengine-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=ti.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727389AbfKKNxS (ORCPT + 3 others); Mon, 11 Nov 2019 08:53:18 -0500 Received: from fllv0015.ext.ti.com ([198.47.19.141]:56722 "EHLO fllv0015.ext.ti.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727365AbfKKNxR (ORCPT ); Mon, 11 Nov 2019 08:53:17 -0500 Received: from fllv0035.itg.ti.com ([10.64.41.0]) by fllv0015.ext.ti.com (8.15.2/8.15.2) with ESMTP id xABDr6be090695; Mon, 11 Nov 2019 07:53:06 -0600 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1573480386; bh=ohRHfztl/6cwF/ARblvPlIwfCKujALnmx6Fu/d0o7cU=; h=From:To:CC:Subject:Date:In-Reply-To:References; b=JaU2tcB3WN+oVX9nAefMGll10oCv8xzvrHGI6fxhCXyoYinufZUOkG++nUWIPhzqJ X70LtqTIAZ2Zo8A6JFFeuPA2ThL2s2LFqd4HUXcvz8mWVjcAwORj2Zb6oY9yVOoKDA OtDTg2rhc3KjPHmp+JE2QVF+DlRjtZ5O+1PRBbrA= Received: from DFLE103.ent.ti.com (dfle103.ent.ti.com [10.64.6.24]) by fllv0035.itg.ti.com (8.15.2/8.15.2) with ESMTP id xABDr633046552; Mon, 11 Nov 2019 07:53:06 -0600 Received: from DFLE113.ent.ti.com (10.64.6.34) by DFLE103.ent.ti.com (10.64.6.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1847.3; Mon, 11 Nov 2019 07:52:49 -0600 Received: from fllv0040.itg.ti.com (10.64.41.20) by DFLE113.ent.ti.com (10.64.6.34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1847.3 via Frontend Transport; Mon, 11 Nov 2019 07:52:49 -0600 Received: from feketebors.ti.com (ileax41-snat.itg.ti.com [10.172.224.153]) by fllv0040.itg.ti.com (8.15.2/8.15.2) with ESMTP id xABDqE90097668; Mon, 11 Nov 2019 07:53:03 -0600 From: Peter Ujfalusi To: , , , CC: , , , , , , , , , Subject: [PATCH v5 13/15] dmaengine: ti: New driver for K3 UDMA - split#5: dma_device callbacks 2 Date: Mon, 11 Nov 2019 15:53:28 +0200 Message-ID: <20191111135330.8235-14-peter.ujfalusi@ti.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: <20191111135330.8235-1-peter.ujfalusi@ti.com> References: <20191111135330.8235-1-peter.ujfalusi@ti.com> MIME-Version: 1.0 X-EXCLAIMER-MD-CONFIG: e1e8a2fd-e40a-4ac6-ac9b-f7e9cc9ee180 Sender: dmaengine-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org Split patch for review containing: device_prep_slave_sg and device_prep_dma_cyclic implementation supporting packet and TR channels. DMA driver for Texas Instruments K3 NAVSS Unified DMA – Peripheral Root Complex (UDMA-P) The UDMA-P is intended to perform similar (but significantly upgraded) functions as the packet-oriented DMA used on previous SoC devices. The UDMA-P module supports the transmission and reception of various packet types. The UDMA-P is architected to facilitate the segmentation and reassembly of SoC DMA data structure compliant packets to/from smaller data blocks that are natively compatible with the specific requirements of each connected peripheral. Multiple Tx and Rx channels are provided within the DMA which allow multiple segmentation or reassembly operations to be ongoing. The DMA controller maintains state information for each of the channels which allows packet segmentation and reassembly operations to be time division multiplexed between channels in order to share the underlying DMA hardware. An external DMA scheduler is used to control the ordering and rate at which this multiplexing occurs for Transmit operations. The ordering and rate of Receive operations is indirectly controlled by the order in which blocks are pushed into the DMA on the Rx PSI-L interface. The UDMA-P also supports acting as both a UTC and UDMA-C for its internal channels. Channels in the UDMA-P can be configured to be either Packet-Based or Third-Party channels on a channel by channel basis. The initial driver supports: - MEM_TO_MEM (TR mode) - DEV_TO_MEM (Packet / TR mode) - MEM_TO_DEV (Packet / TR mode) - Cyclic (Packet / TR mode) - Metadata for descriptors Signed-off-by: Peter Ujfalusi --- drivers/dma/ti/k3-udma.c | 702 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 702 insertions(+) -- Peter Texas Instruments Finland Oy, Porkkalankatu 22, 00180 Helsinki. Y-tunnus/Business ID: 0615521-4. Kotipaikka/Domicile: Helsinki diff --git a/drivers/dma/ti/k3-udma.c b/drivers/dma/ti/k3-udma.c index 4b113c4eb3d9..8cffc41b9d38 100644 --- a/drivers/dma/ti/k3-udma.c +++ b/drivers/dma/ti/k3-udma.c @@ -1790,6 +1790,708 @@ static int udma_slave_config(struct dma_chan *chan, return 0; } +static struct udma_desc *udma_alloc_tr_desc(struct udma_chan *uc, + size_t tr_size, int tr_count, + enum dma_transfer_direction dir) +{ + struct udma_hwdesc *hwdesc; + struct cppi5_desc_hdr_t *tr_desc; + struct udma_desc *d; + u32 reload_count = 0; + u32 ring_id; + + switch (tr_size) { + case 16: + case 32: + case 64: + case 128: + break; + default: + dev_err(uc->ud->dev, "Unsupported TR size of %zu\n", tr_size); + return NULL; + } + + /* We have only one descriptor containing multiple TRs */ + d = kzalloc(sizeof(*d) + sizeof(d->hwdesc[0]), GFP_ATOMIC); + if (!d) + return NULL; + + d->sglen = tr_count; + + d->hwdesc_count = 1; + hwdesc = &d->hwdesc[0]; + + /* Allocate memory for DMA ring descriptor */ + if (uc->use_dma_pool) { + hwdesc->cppi5_desc_size = uc->hdesc_size; + hwdesc->cppi5_desc_vaddr = dma_pool_zalloc(uc->hdesc_pool, + GFP_ATOMIC, + &hwdesc->cppi5_desc_paddr); + } else { + hwdesc->cppi5_desc_size = cppi5_trdesc_calc_size(tr_size, + tr_count); + hwdesc->cppi5_desc_size = ALIGN(hwdesc->cppi5_desc_size, + uc->ud->desc_align); + hwdesc->cppi5_desc_vaddr = dma_alloc_coherent(uc->ud->dev, + hwdesc->cppi5_desc_size, + &hwdesc->cppi5_desc_paddr, + GFP_ATOMIC); + } + + if (!hwdesc->cppi5_desc_vaddr) { + kfree(d); + return NULL; + } + + /* Start of the TR req records */ + hwdesc->tr_req_base = hwdesc->cppi5_desc_vaddr + tr_size; + /* Start address of the TR response array */ + hwdesc->tr_resp_base = hwdesc->tr_req_base + tr_size * tr_count; + + tr_desc = hwdesc->cppi5_desc_vaddr; + + if (uc->cyclic) + reload_count = CPPI5_INFO0_TRDESC_RLDCNT_INFINITE; + + if (dir == DMA_DEV_TO_MEM) + ring_id = k3_ringacc_get_ring_id(uc->rflow->r_ring); + else + ring_id = k3_ringacc_get_ring_id(uc->tchan->tc_ring); + + cppi5_trdesc_init(tr_desc, tr_count, tr_size, 0, reload_count); + cppi5_desc_set_pktids(tr_desc, uc->id, 0x3fff); + cppi5_desc_set_retpolicy(tr_desc, 0, ring_id); + + return d; +} + +static struct udma_desc * +udma_prep_slave_sg_tr(struct udma_chan *uc, struct scatterlist *sgl, + unsigned int sglen, enum dma_transfer_direction dir, + unsigned long tx_flags, void *context) +{ + enum dma_slave_buswidth dev_width; + struct scatterlist *sgent; + struct udma_desc *d; + size_t tr_size; + struct cppi5_tr_type1_t *tr_req = NULL; + unsigned int i; + u32 burst; + + if (dir == DMA_DEV_TO_MEM) { + dev_width = uc->cfg.src_addr_width; + burst = uc->cfg.src_maxburst; + } else if (dir == DMA_MEM_TO_DEV) { + dev_width = uc->cfg.dst_addr_width; + burst = uc->cfg.dst_maxburst; + } else { + dev_err(uc->ud->dev, "%s: bad direction?\n", __func__); + return NULL; + } + + if (!burst) + burst = 1; + + /* Now allocate and setup the descriptor. */ + tr_size = sizeof(struct cppi5_tr_type1_t); + d = udma_alloc_tr_desc(uc, tr_size, sglen, dir); + if (!d) + return NULL; + + d->sglen = sglen; + + tr_req = (struct cppi5_tr_type1_t *)d->hwdesc[0].tr_req_base; + for_each_sg(sgl, sgent, sglen, i) { + d->residue += sg_dma_len(sgent); + + cppi5_tr_init(&tr_req[i].flags, CPPI5_TR_TYPE1, false, false, + CPPI5_TR_EVENT_SIZE_COMPLETION, 0); + cppi5_tr_csf_set(&tr_req[i].flags, CPPI5_TR_CSF_SUPR_EVT); + + tr_req[i].addr = sg_dma_address(sgent); + tr_req[i].icnt0 = burst * dev_width; + tr_req[i].dim1 = burst * dev_width; + tr_req[i].icnt1 = sg_dma_len(sgent) / tr_req[i].icnt0; + } + + cppi5_tr_csf_set(&tr_req[i - 1].flags, CPPI5_TR_CSF_EOP); + + return d; +} + +static int udma_configure_statictr(struct udma_chan *uc, struct udma_desc *d, + enum dma_slave_buswidth dev_width, + u16 elcnt) +{ + if (uc->ep_type != PSIL_EP_PDMA_XY) + return 0; + + /* Bus width translates to the element size (ES) */ + switch (dev_width) { + case DMA_SLAVE_BUSWIDTH_1_BYTE: + d->static_tr.elsize = 0; + break; + case DMA_SLAVE_BUSWIDTH_2_BYTES: + d->static_tr.elsize = 1; + break; + case DMA_SLAVE_BUSWIDTH_3_BYTES: + d->static_tr.elsize = 2; + break; + case DMA_SLAVE_BUSWIDTH_4_BYTES: + d->static_tr.elsize = 3; + break; + case DMA_SLAVE_BUSWIDTH_8_BYTES: + d->static_tr.elsize = 4; + break; + default: /* not reached */ + return -EINVAL; + } + + d->static_tr.elcnt = elcnt; + + /* + * PDMA must to close the packet when the channel is in packet mode. + * For TR mode when the channel is not cyclic we also need PDMA to close + * the packet otherwise the transfer will stall because PDMA holds on + * the data it has received from the peripheral. + */ + if (uc->pkt_mode || !uc->cyclic) { + unsigned int div = dev_width * elcnt; + + if (uc->cyclic) + d->static_tr.bstcnt = d->residue / d->sglen / div; + else + d->static_tr.bstcnt = d->residue / div; + + if (uc->dir == DMA_DEV_TO_MEM && + d->static_tr.bstcnt > uc->ud->match_data->statictr_z_mask) + return -EINVAL; + } else { + d->static_tr.bstcnt = 0; + } + + return 0; +} + +static struct udma_desc * +udma_prep_slave_sg_pkt(struct udma_chan *uc, struct scatterlist *sgl, + unsigned int sglen, enum dma_transfer_direction dir, + unsigned long tx_flags, void *context) +{ + struct scatterlist *sgent; + struct cppi5_host_desc_t *h_desc = NULL; + struct udma_desc *d; + u32 ring_id; + unsigned int i; + + d = kzalloc(sizeof(*d) + sglen * sizeof(d->hwdesc[0]), GFP_ATOMIC); + if (!d) + return NULL; + + d->sglen = sglen; + d->hwdesc_count = sglen; + + if (dir == DMA_DEV_TO_MEM) + ring_id = k3_ringacc_get_ring_id(uc->rflow->r_ring); + else + ring_id = k3_ringacc_get_ring_id(uc->tchan->tc_ring); + + for_each_sg(sgl, sgent, sglen, i) { + struct udma_hwdesc *hwdesc = &d->hwdesc[i]; + dma_addr_t sg_addr = sg_dma_address(sgent); + struct cppi5_host_desc_t *desc; + size_t sg_len = sg_dma_len(sgent); + + hwdesc->cppi5_desc_vaddr = dma_pool_zalloc(uc->hdesc_pool, + GFP_ATOMIC, + &hwdesc->cppi5_desc_paddr); + if (!hwdesc->cppi5_desc_vaddr) { + dev_err(uc->ud->dev, + "descriptor%d allocation failed\n", i); + + udma_free_hwdesc(uc, d); + kfree(d); + return NULL; + } + + d->residue += sg_len; + hwdesc->cppi5_desc_size = uc->hdesc_size; + desc = hwdesc->cppi5_desc_vaddr; + + if (i == 0) { + cppi5_hdesc_init(desc, 0, 0); + /* Flow and Packed ID */ + cppi5_desc_set_pktids(&desc->hdr, uc->id, 0x3fff); + cppi5_desc_set_retpolicy(&desc->hdr, 0, ring_id); + } else { + cppi5_hdesc_reset_hbdesc(desc); + cppi5_desc_set_retpolicy(&desc->hdr, 0, 0xffff); + } + + /* attach the sg buffer to the descriptor */ + cppi5_hdesc_attach_buf(desc, sg_addr, sg_len, sg_addr, sg_len); + + /* Attach link as host buffer descriptor */ + if (h_desc) + cppi5_hdesc_link_hbdesc(h_desc, + hwdesc->cppi5_desc_paddr); + + if (dir == DMA_MEM_TO_DEV) + h_desc = desc; + } + + if (d->residue >= SZ_4M) { + dev_err(uc->ud->dev, + "%s: Transfer size %u is over the supported 4M range\n", + __func__, d->residue); + udma_free_hwdesc(uc, d); + kfree(d); + return NULL; + } + + h_desc = d->hwdesc[0].cppi5_desc_vaddr; + cppi5_hdesc_set_pktlen(h_desc, d->residue); + + return d; +} + +static int udma_attach_metadata(struct dma_async_tx_descriptor *desc, + void *data, size_t len) +{ + struct udma_desc *d = to_udma_desc(desc); + struct udma_chan *uc = to_udma_chan(desc->chan); + struct cppi5_host_desc_t *h_desc; + u32 psd_size = len; + u32 flags = 0; + + if (!uc->pkt_mode || !uc->metadata_size) + return -ENOTSUPP; + + if (!data || len > uc->metadata_size) + return -EINVAL; + + if (uc->needs_epib && len < CPPI5_INFO0_HDESC_EPIB_SIZE) + return -EINVAL; + + h_desc = d->hwdesc[0].cppi5_desc_vaddr; + if (d->dir == DMA_MEM_TO_DEV) + memcpy(h_desc->epib, data, len); + + if (uc->needs_epib) + psd_size -= CPPI5_INFO0_HDESC_EPIB_SIZE; + + d->metadata = data; + d->metadata_size = len; + if (uc->needs_epib) + flags |= CPPI5_INFO0_HDESC_EPIB_PRESENT; + + cppi5_hdesc_update_flags(h_desc, flags); + cppi5_hdesc_update_psdata_size(h_desc, psd_size); + + return 0; +} + +static void *udma_get_metadata_ptr(struct dma_async_tx_descriptor *desc, + size_t *payload_len, size_t *max_len) +{ + struct udma_desc *d = to_udma_desc(desc); + struct udma_chan *uc = to_udma_chan(desc->chan); + struct cppi5_host_desc_t *h_desc; + + if (!uc->pkt_mode || !uc->metadata_size) + return ERR_PTR(-ENOTSUPP); + + h_desc = d->hwdesc[0].cppi5_desc_vaddr; + + *max_len = uc->metadata_size; + + *payload_len = cppi5_hdesc_epib_present(&h_desc->hdr) ? + CPPI5_INFO0_HDESC_EPIB_SIZE : 0; + *payload_len += cppi5_hdesc_get_psdata_size(h_desc); + + return h_desc->epib; +} + +static int udma_set_metadata_len(struct dma_async_tx_descriptor *desc, + size_t payload_len) +{ + struct udma_desc *d = to_udma_desc(desc); + struct udma_chan *uc = to_udma_chan(desc->chan); + struct cppi5_host_desc_t *h_desc; + u32 psd_size = payload_len; + u32 flags = 0; + + if (!uc->pkt_mode || !uc->metadata_size) + return -ENOTSUPP; + + if (payload_len > uc->metadata_size) + return -EINVAL; + + if (uc->needs_epib && payload_len < CPPI5_INFO0_HDESC_EPIB_SIZE) + return -EINVAL; + + h_desc = d->hwdesc[0].cppi5_desc_vaddr; + + if (uc->needs_epib) { + psd_size -= CPPI5_INFO0_HDESC_EPIB_SIZE; + flags |= CPPI5_INFO0_HDESC_EPIB_PRESENT; + } + + cppi5_hdesc_update_flags(h_desc, flags); + cppi5_hdesc_update_psdata_size(h_desc, psd_size); + + return 0; +} + +static struct dma_descriptor_metadata_ops metadata_ops = { + .attach = udma_attach_metadata, + .get_ptr = udma_get_metadata_ptr, + .set_len = udma_set_metadata_len, +}; + +static struct dma_async_tx_descriptor * +udma_prep_slave_sg(struct dma_chan *chan, struct scatterlist *sgl, + unsigned int sglen, enum dma_transfer_direction dir, + unsigned long tx_flags, void *context) +{ + struct udma_chan *uc = to_udma_chan(chan); + enum dma_slave_buswidth dev_width; + struct udma_desc *d; + u32 burst; + + if (dir != uc->dir) { + dev_err(chan->device->dev, + "%s: chan%d is for %s, not supporting %s\n", + __func__, uc->id, udma_get_dir_text(uc->dir), + udma_get_dir_text(dir)); + return NULL; + } + + if (dir == DMA_DEV_TO_MEM) { + dev_width = uc->cfg.src_addr_width; + burst = uc->cfg.src_maxburst; + } else if (dir == DMA_MEM_TO_DEV) { + dev_width = uc->cfg.dst_addr_width; + burst = uc->cfg.dst_maxburst; + } else { + dev_err(chan->device->dev, "%s: bad direction?\n", __func__); + return NULL; + } + + if (!burst) + burst = 1; + + if (uc->pkt_mode) + d = udma_prep_slave_sg_pkt(uc, sgl, sglen, dir, tx_flags, + context); + else + d = udma_prep_slave_sg_tr(uc, sgl, sglen, dir, tx_flags, + context); + + if (!d) + return NULL; + + d->dir = dir; + d->desc_idx = 0; + d->tr_idx = 0; + + /* static TR for remote PDMA */ + if (udma_configure_statictr(uc, d, dev_width, burst)) { + dev_err(uc->ud->dev, + "%s: StaticTR Z is limted to maximum 4095 (%u)\n", + __func__, d->static_tr.bstcnt); + + udma_free_hwdesc(uc, d); + kfree(d); + return NULL; + } + + if (uc->metadata_size) + d->vd.tx.metadata_ops = &metadata_ops; + + return vchan_tx_prep(&uc->vc, &d->vd, tx_flags); +} + +static struct udma_desc * +udma_prep_dma_cyclic_tr(struct udma_chan *uc, dma_addr_t buf_addr, + size_t buf_len, size_t period_len, + enum dma_transfer_direction dir, unsigned long flags) +{ + enum dma_slave_buswidth dev_width; + struct udma_desc *d; + size_t tr_size; + struct cppi5_tr_type1_t *tr_req; + unsigned int i; + unsigned int periods = buf_len / period_len; + u32 burst; + + if (dir == DMA_DEV_TO_MEM) { + dev_width = uc->cfg.src_addr_width; + burst = uc->cfg.src_maxburst; + } else if (dir == DMA_MEM_TO_DEV) { + dev_width = uc->cfg.dst_addr_width; + burst = uc->cfg.dst_maxburst; + } else { + dev_err(uc->ud->dev, "%s: bad direction?\n", __func__); + return NULL; + } + + if (!burst) + burst = 1; + + /* Now allocate and setup the descriptor. */ + tr_size = sizeof(struct cppi5_tr_type1_t); + d = udma_alloc_tr_desc(uc, tr_size, periods, dir); + if (!d) + return NULL; + + tr_req = (struct cppi5_tr_type1_t *)d->hwdesc[0].tr_req_base; + for (i = 0; i < periods; i++) { + cppi5_tr_init(&tr_req[i].flags, CPPI5_TR_TYPE1, false, false, + CPPI5_TR_EVENT_SIZE_COMPLETION, 0); + + tr_req[i].addr = buf_addr + period_len * i; + tr_req[i].icnt0 = dev_width; + tr_req[i].icnt1 = period_len / dev_width; + tr_req[i].dim1 = dev_width; + + if (!(flags & DMA_PREP_INTERRUPT)) + cppi5_tr_csf_set(&tr_req[i].flags, + CPPI5_TR_CSF_SUPR_EVT); + } + + return d; +} + +static struct udma_desc * +udma_prep_dma_cyclic_pkt(struct udma_chan *uc, dma_addr_t buf_addr, + size_t buf_len, size_t period_len, + enum dma_transfer_direction dir, unsigned long flags) +{ + struct udma_desc *d; + u32 ring_id; + int i; + int periods = buf_len / period_len; + + if (periods > (K3_UDMA_DEFAULT_RING_SIZE - 1)) + return NULL; + + if (period_len > 0x3FFFFF) + return NULL; + + d = kzalloc(sizeof(*d) + periods * sizeof(d->hwdesc[0]), GFP_ATOMIC); + if (!d) + return NULL; + + d->hwdesc_count = periods; + + /* TODO: re-check this... */ + if (dir == DMA_DEV_TO_MEM) + ring_id = k3_ringacc_get_ring_id(uc->rflow->r_ring); + else + ring_id = k3_ringacc_get_ring_id(uc->tchan->tc_ring); + + for (i = 0; i < periods; i++) { + struct udma_hwdesc *hwdesc = &d->hwdesc[i]; + dma_addr_t period_addr = buf_addr + (period_len * i); + struct cppi5_host_desc_t *h_desc; + + hwdesc->cppi5_desc_vaddr = dma_pool_zalloc(uc->hdesc_pool, + GFP_ATOMIC, + &hwdesc->cppi5_desc_paddr); + if (!hwdesc->cppi5_desc_vaddr) { + dev_err(uc->ud->dev, + "descriptor%d allocation failed\n", i); + + udma_free_hwdesc(uc, d); + kfree(d); + return NULL; + } + + hwdesc->cppi5_desc_size = uc->hdesc_size; + h_desc = hwdesc->cppi5_desc_vaddr; + + cppi5_hdesc_init(h_desc, 0, 0); + cppi5_hdesc_set_pktlen(h_desc, period_len); + + /* Flow and Packed ID */ + cppi5_desc_set_pktids(&h_desc->hdr, uc->id, 0x3fff); + cppi5_desc_set_retpolicy(&h_desc->hdr, 0, ring_id); + + /* attach each period to a new descriptor */ + cppi5_hdesc_attach_buf(h_desc, + period_addr, period_len, + period_addr, period_len); + } + + return d; +} + +static struct dma_async_tx_descriptor * +udma_prep_dma_cyclic(struct dma_chan *chan, dma_addr_t buf_addr, size_t buf_len, + size_t period_len, enum dma_transfer_direction dir, + unsigned long flags) +{ + struct udma_chan *uc = to_udma_chan(chan); + enum dma_slave_buswidth dev_width; + struct udma_desc *d; + u32 burst; + + if (dir != uc->dir) { + dev_err(chan->device->dev, + "%s: chan%d is for %s, not supporting %s\n", + __func__, uc->id, udma_get_dir_text(uc->dir), + udma_get_dir_text(dir)); + return NULL; + } + + uc->cyclic = true; + + if (dir == DMA_DEV_TO_MEM) { + dev_width = uc->cfg.src_addr_width; + burst = uc->cfg.src_maxburst; + } else if (dir == DMA_MEM_TO_DEV) { + dev_width = uc->cfg.dst_addr_width; + burst = uc->cfg.dst_maxburst; + } else { + dev_err(uc->ud->dev, "%s: bad direction?\n", __func__); + return NULL; + } + + if (!burst) + burst = 1; + + if (uc->pkt_mode) + d = udma_prep_dma_cyclic_pkt(uc, buf_addr, buf_len, period_len, + dir, flags); + else + d = udma_prep_dma_cyclic_tr(uc, buf_addr, buf_len, period_len, + dir, flags); + + if (!d) + return NULL; + + d->sglen = buf_len / period_len; + + d->dir = dir; + d->residue = buf_len; + + /* static TR for remote PDMA */ + if (udma_configure_statictr(uc, d, dev_width, burst)) { + dev_err(uc->ud->dev, + "%s: StaticTR Z is limted to maximum 4095 (%u)\n", + __func__, d->static_tr.bstcnt); + + udma_free_hwdesc(uc, d); + kfree(d); + return NULL; + } + + if (uc->metadata_size) + d->vd.tx.metadata_ops = &metadata_ops; + + return vchan_tx_prep(&uc->vc, &d->vd, flags); +} + +static struct dma_async_tx_descriptor * +udma_prep_dma_memcpy(struct dma_chan *chan, dma_addr_t dest, dma_addr_t src, + size_t len, unsigned long tx_flags) +{ + struct udma_chan *uc = to_udma_chan(chan); + struct udma_desc *d; + struct cppi5_tr_type15_t *tr_req; + int num_tr; + size_t tr_size = sizeof(struct cppi5_tr_type15_t); + u16 tr0_cnt0, tr0_cnt1, tr1_cnt0; + + if (uc->dir != DMA_MEM_TO_MEM) { + dev_err(chan->device->dev, + "%s: chan%d is for %s, not supporting %s\n", + __func__, uc->id, udma_get_dir_text(uc->dir), + udma_get_dir_text(DMA_MEM_TO_MEM)); + return NULL; + } + + if (len < SZ_64K) { + num_tr = 1; + tr0_cnt0 = len; + tr0_cnt1 = 1; + } else { + unsigned long align_to = __ffs(src | dest); + + if (align_to > 3) + align_to = 3; + /* + * Keep simple: tr0: SZ_64K-alignment blocks, + * tr1: the remaining + */ + num_tr = 2; + tr0_cnt0 = (SZ_64K - BIT(align_to)); + if (len / tr0_cnt0 >= SZ_64K) { + dev_err(uc->ud->dev, "size %zu is not supported\n", + len); + return NULL; + } + + tr0_cnt1 = len / tr0_cnt0; + tr1_cnt0 = len % tr0_cnt0; + } + + d = udma_alloc_tr_desc(uc, tr_size, num_tr, DMA_MEM_TO_MEM); + if (!d) + return NULL; + + d->dir = DMA_MEM_TO_MEM; + d->desc_idx = 0; + d->tr_idx = 0; + d->residue = len; + + tr_req = (struct cppi5_tr_type15_t *)d->hwdesc[0].tr_req_base; + + cppi5_tr_init(&tr_req[0].flags, CPPI5_TR_TYPE15, false, true, + CPPI5_TR_EVENT_SIZE_COMPLETION, 0); + cppi5_tr_csf_set(&tr_req[0].flags, CPPI5_TR_CSF_SUPR_EVT); + + tr_req[0].addr = src; + tr_req[0].icnt0 = tr0_cnt0; + tr_req[0].icnt1 = tr0_cnt1; + tr_req[0].icnt2 = 1; + tr_req[0].icnt3 = 1; + tr_req[0].dim1 = tr0_cnt0; + + tr_req[0].daddr = dest; + tr_req[0].dicnt0 = tr0_cnt0; + tr_req[0].dicnt1 = tr0_cnt1; + tr_req[0].dicnt2 = 1; + tr_req[0].dicnt3 = 1; + tr_req[0].ddim1 = tr0_cnt0; + + if (num_tr == 2) { + cppi5_tr_init(&tr_req[1].flags, CPPI5_TR_TYPE15, false, true, + CPPI5_TR_EVENT_SIZE_COMPLETION, 0); + cppi5_tr_csf_set(&tr_req[1].flags, CPPI5_TR_CSF_SUPR_EVT); + + tr_req[1].addr = src + tr0_cnt1 * tr0_cnt0; + tr_req[1].icnt0 = tr1_cnt0; + tr_req[1].icnt1 = 1; + tr_req[1].icnt2 = 1; + tr_req[1].icnt3 = 1; + + tr_req[1].daddr = dest + tr0_cnt1 * tr0_cnt0; + tr_req[1].dicnt0 = tr1_cnt0; + tr_req[1].dicnt1 = 1; + tr_req[1].dicnt2 = 1; + tr_req[1].dicnt3 = 1; + } + + cppi5_tr_csf_set(&tr_req[num_tr - 1].flags, CPPI5_TR_CSF_EOP); + + if (uc->metadata_size) + d->vd.tx.metadata_ops = &metadata_ops; + + return vchan_tx_prep(&uc->vc, &d->vd, tx_flags); +} + static void udma_issue_pending(struct dma_chan *chan) { struct udma_chan *uc = to_udma_chan(chan); From patchwork Mon Nov 11 13:53:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Peter Ujfalusi X-Patchwork-Id: 179076 Delivered-To: patch@linaro.org Received: by 2002:ac9:3c86:0:0:0:0:0 with SMTP id w6csp6372943ocf; Mon, 11 Nov 2019 05:53:23 -0800 (PST) X-Google-Smtp-Source: APXvYqzhRwj+aEA5ESIStZ/LFCABfvQhFsI3wBYg75uWvxxdU6ZOLXci/3tvhIU/rnLASc6GNTFH X-Received: by 2002:aa7:c159:: with SMTP id r25mr11005158edp.118.1573480402998; Mon, 11 Nov 2019 05:53:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573480402; cv=none; d=google.com; s=arc-20160816; b=y97zz1rpP+QDIbs2SGnVfMKoNRHbyGq0JCWx5sJ6EYnR2Hcjg02Ayl0ssFjiTgbRuB ytDVIYr4BvgygIfBWUB7hsN64kZx6D7VvE6ob+NwaDBH8h3QrUdP49MbA9pGgMIEQWHs px41XlKRc2JGLebPnMJTqRVZPSDiuD8N2uUXSBsPx7VYsasahfXT0qkf2tX6U95Blxne yv6RACwo00poqhUdwsiTu3EOKQ1hc6ylu0XgnkSDU91R7+u7cbLAaZUSW7+0CXVTS+mD A68Ldw05aVqVm+fex5yvh5LcErflZ/3COs0J/FiuRkLDn+efZHDkqkBs/p6JEQ/VoIa/ Q7dQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=EOL7+ILAJKMK9C2xGDp+0K1qLoucOFWJ/CJCGCWfatY=; b=t9fYstfCJqa6OGWurYt10q+mlWFS3N8bsPDUGrKi2j7677OCgTp+9ji4PbyZr8fI57 ItRdHkykB7KY9h06jknuL8WQ9dBKiTRD0hiNM74FSCHuB+GM9RdzABdWH3R8rNTWFjBS uRZ6V1xcV+Mt3cUKMZPbfk8uayygXyZfOv0PLXJOtnCh8UUXHUuWbN9XHMdDRMb1+ePK fyyTQOjAjzZurfhPJMBSSxuQyJUhpYKAmLoni5Uks5gDOQ3gNOmAv9aIEwBo9ZYloUlO GkXT+POw6+wm6XMA8M1BOIm0TyJOLZ3VSsWcs4uSO3yVukHuPTyjUo5FYbkmE6QF5IRB Zsww== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ti.com header.s=ti-com-17Q1 header.b=C74CsEKK; spf=pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=dmaengine-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=ti.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id d1si9100393ejh.281.2019.11.11.05.53.22; Mon, 11 Nov 2019 05:53:22 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@ti.com header.s=ti-com-17Q1 header.b=C74CsEKK; spf=pass (google.com: best guess record for domain of dmaengine-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=dmaengine-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=ti.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726879AbfKKNxS (ORCPT + 3 others); Mon, 11 Nov 2019 08:53:18 -0500 Received: from fllv0015.ext.ti.com ([198.47.19.141]:56726 "EHLO fllv0015.ext.ti.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727372AbfKKNxR (ORCPT ); Mon, 11 Nov 2019 08:53:17 -0500 Received: from lelv0265.itg.ti.com ([10.180.67.224]) by fllv0015.ext.ti.com (8.15.2/8.15.2) with ESMTP id xABDrA8p090734; Mon, 11 Nov 2019 07:53:10 -0600 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1573480391; bh=EOL7+ILAJKMK9C2xGDp+0K1qLoucOFWJ/CJCGCWfatY=; h=From:To:CC:Subject:Date:In-Reply-To:References; b=C74CsEKKdE4xkIUJfcDlndtfyjSzVetlyShTs0xbfJCR37wviX65PBuH4tXgRTmPm x8l9JAQrHuOsqqA+jkgpWKYDrcrPmDRGZzU5ZfcTEgh1qWuQWGmEPRm77syOfve3Eq JxbOUtVMSvrOCBHvmVUQpN4a/ZafP/4hiMatace8= Received: from DLEE113.ent.ti.com (dlee113.ent.ti.com [157.170.170.24]) by lelv0265.itg.ti.com (8.15.2/8.15.2) with ESMTPS id xABDrAxK002742 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 11 Nov 2019 07:53:10 -0600 Received: from DLEE100.ent.ti.com (157.170.170.30) by DLEE113.ent.ti.com (157.170.170.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1847.3; Mon, 11 Nov 2019 07:53:10 -0600 Received: from fllv0040.itg.ti.com (10.64.41.20) by DLEE100.ent.ti.com (157.170.170.30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1847.3 via Frontend Transport; Mon, 11 Nov 2019 07:52:52 -0600 Received: from feketebors.ti.com (ileax41-snat.itg.ti.com [10.172.224.153]) by fllv0040.itg.ti.com (8.15.2/8.15.2) with ESMTP id xABDqE91097668; Mon, 11 Nov 2019 07:53:06 -0600 From: Peter Ujfalusi To: , , , CC: , , , , , , , , , Subject: [PATCH v5 14/15] dmaengine: ti: New driver for K3 UDMA - split#6: Kconfig and Makefile Date: Mon, 11 Nov 2019 15:53:29 +0200 Message-ID: <20191111135330.8235-15-peter.ujfalusi@ti.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: <20191111135330.8235-1-peter.ujfalusi@ti.com> References: <20191111135330.8235-1-peter.ujfalusi@ti.com> MIME-Version: 1.0 X-EXCLAIMER-MD-CONFIG: e1e8a2fd-e40a-4ac6-ac9b-f7e9cc9ee180 Sender: dmaengine-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org Split patch for review containing: Kconfig and Makefile changes DMA driver for Texas Instruments K3 NAVSS Unified DMA – Peripheral Root Complex (UDMA-P) The UDMA-P is intended to perform similar (but significantly upgraded) functions as the packet-oriented DMA used on previous SoC devices. The UDMA-P module supports the transmission and reception of various packet types. The UDMA-P is architected to facilitate the segmentation and reassembly of SoC DMA data structure compliant packets to/from smaller data blocks that are natively compatible with the specific requirements of each connected peripheral. Multiple Tx and Rx channels are provided within the DMA which allow multiple segmentation or reassembly operations to be ongoing. The DMA controller maintains state information for each of the channels which allows packet segmentation and reassembly operations to be time division multiplexed between channels in order to share the underlying DMA hardware. An external DMA scheduler is used to control the ordering and rate at which this multiplexing occurs for Transmit operations. The ordering and rate of Receive operations is indirectly controlled by the order in which blocks are pushed into the DMA on the Rx PSI-L interface. The UDMA-P also supports acting as both a UTC and UDMA-C for its internal channels. Channels in the UDMA-P can be configured to be either Packet-Based or Third-Party channels on a channel by channel basis. The initial driver supports: - MEM_TO_MEM (TR mode) - DEV_TO_MEM (Packet / TR mode) - MEM_TO_DEV (Packet / TR mode) - Cyclic (Packet / TR mode) - Metadata for descriptors Signed-off-by: Peter Ujfalusi --- drivers/dma/ti/Kconfig | 13 +++++++++++++ drivers/dma/ti/Makefile | 1 + 2 files changed, 14 insertions(+) -- Peter Texas Instruments Finland Oy, Porkkalankatu 22, 00180 Helsinki. Y-tunnus/Business ID: 0615521-4. Kotipaikka/Domicile: Helsinki diff --git a/drivers/dma/ti/Kconfig b/drivers/dma/ti/Kconfig index 72f3d2728178..a096f0ec3998 100644 --- a/drivers/dma/ti/Kconfig +++ b/drivers/dma/ti/Kconfig @@ -34,6 +34,19 @@ config DMA_OMAP Enable support for the TI sDMA (System DMA or DMA4) controller. This DMA engine is found on OMAP and DRA7xx parts. +config TI_K3_UDMA + tristate "Texas Instruments UDMA support" + depends on ARCH_K3 || COMPILE_TEST + depends on TI_SCI_PROTOCOL + depends on TI_SCI_INTA_IRQCHIP + select DMA_ENGINE + select DMA_VIRTUAL_CHANNELS + select TI_K3_RINGACC + select TI_K3_PSIL + help + Enable support for the TI UDMA (Unified DMA) controller. This + DMA engine is used in AM65x. + config TI_K3_PSIL bool diff --git a/drivers/dma/ti/Makefile b/drivers/dma/ti/Makefile index f8d912ad7eaf..9d787f009195 100644 --- a/drivers/dma/ti/Makefile +++ b/drivers/dma/ti/Makefile @@ -2,5 +2,6 @@ obj-$(CONFIG_TI_CPPI41) += cppi41.o obj-$(CONFIG_TI_EDMA) += edma.o obj-$(CONFIG_DMA_OMAP) += omap-dma.o +obj-$(CONFIG_TI_K3_UDMA) += k3-udma.o obj-$(CONFIG_TI_K3_PSIL) += k3-psil.o k3-psil-am654.o k3-psil-j721e.o obj-$(CONFIG_TI_DMA_CROSSBAR) += dma-crossbar.o