From patchwork Fri May 8 09:57:48 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Uvarov X-Patchwork-Id: 48165 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-wi0-f197.google.com (mail-wi0-f197.google.com [209.85.212.197]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 5EBE12121F for ; Fri, 8 May 2015 09:59:22 +0000 (UTC) Received: by wizk4 with SMTP id k4sf5710021wiz.2 for ; Fri, 08 May 2015 02:59:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:delivered-to:from:to:date :message-id:in-reply-to:references:subject:precedence:list-id :list-unsubscribe:list-archive:list-post:list-help:list-subscribe :mime-version:content-type:content-transfer-encoding:errors-to :sender:x-original-sender:x-original-authentication-results :mailing-list; bh=40nRp9c5a40xKC0FxoaJOurW/Hq1KdtVgT5dVQQk8ds=; b=FoCTE74oTwWDZ8vKWmnmMVQEEF20ESgRp5MRhYXV+kURD3kvPO6r+LAViFcLn1rkY8 HvTrMIqIkkUvuFgJhjPhyfxRShgxEv4/fKWXhk35GQ61TmA+Q+NAx0Sz+z5R39w4htLA Gy1VftGz/JsnQTW3/+t0G9ygmfcHtoqWM5VDeNfP3ZwDnxg8aGFsx5+KHa//uFb387Y+ +5pZCWaIv1F193w0GYJ9Z0zeUVTsz0UJ4qBFtUPClELx0oJVUBEtIpxBVfD8NP8klqPu +0MMeFPTmNAWBhHyha20wNQPwEPJzd9C+WUQrbE6w+wXEZtAAT4P7NByYmXPazDttSsB b8zw== X-Gm-Message-State: ALoCoQngdeqfEvGi49USGNU1GHFU84vowduPGeIjZVjxHZbEt/dT6Qla8F/3lAiOM0q653jT6uJs X-Received: by 10.112.219.200 with SMTP id pq8mr1654523lbc.7.1431079161663; Fri, 08 May 2015 02:59:21 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.4.162 with SMTP id l2ls330286lal.93.gmail; Fri, 08 May 2015 02:59:21 -0700 (PDT) X-Received: by 10.152.9.66 with SMTP id x2mr2304868laa.36.1431079161352; Fri, 08 May 2015 02:59:21 -0700 (PDT) Received: from mail-lb0-f172.google.com (mail-lb0-f172.google.com. [209.85.217.172]) by mx.google.com with ESMTPS id bq3si2978608lbb.128.2015.05.08.02.59.21 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 08 May 2015 02:59:21 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.172 as permitted sender) client-ip=209.85.217.172; Received: by lbbuc2 with SMTP id uc2so49549586lbb.2 for ; Fri, 08 May 2015 02:59:21 -0700 (PDT) X-Received: by 10.112.140.231 with SMTP id rj7mr2242350lbb.76.1431079161173; Fri, 08 May 2015 02:59:21 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.108.230 with SMTP id hn6csp349373lbb; Fri, 8 May 2015 02:59:19 -0700 (PDT) X-Received: by 10.140.165.150 with SMTP id l144mr3756973qhl.99.1431079158742; Fri, 08 May 2015 02:59:18 -0700 (PDT) Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id ft1si4811202qcb.47.2015.05.08.02.59.18; Fri, 08 May 2015 02:59:18 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Received: by lists.linaro.org (Postfix, from userid 109) id 043DE61D9C; Fri, 8 May 2015 09:59:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252.ec2.internal X-Spam-Level: X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, RCVD_IN_SORBS_WEB, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from ip-10-142-244-252.ec2.internal (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id CFC9F61D91; Fri, 8 May 2015 09:58:22 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id D62C561D9C; Fri, 8 May 2015 09:58:04 +0000 (UTC) Received: from mail-lb0-f178.google.com (mail-lb0-f178.google.com [209.85.217.178]) by lists.linaro.org (Postfix) with ESMTPS id 95BC161D92 for ; Fri, 8 May 2015 09:58:00 +0000 (UTC) Received: by lbbuc2 with SMTP id uc2so49528442lbb.2 for ; Fri, 08 May 2015 02:57:59 -0700 (PDT) X-Received: by 10.112.161.40 with SMTP id xp8mr2241347lbb.71.1431079079519; Fri, 08 May 2015 02:57:59 -0700 (PDT) Received: from localhost.localdomain (ppp91-76-165-85.pppoe.mtu-net.ru. [91.76.165.85]) by mx.google.com with ESMTPSA id m1sm1045395lbg.36.2015.05.08.02.57.57 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 May 2015 02:57:58 -0700 (PDT) From: Maxim Uvarov To: lng-odp@lists.linaro.org Date: Fri, 8 May 2015 12:57:48 +0300 Message-Id: <1431079069-9702-4-git-send-email-maxim.uvarov@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1431079069-9702-1-git-send-email-maxim.uvarov@linaro.org> References: <1431079069-9702-1-git-send-email-maxim.uvarov@linaro.org> X-Topics: patch Subject: [lng-odp] [PATCH 3/4] linux-generic: add ipc pktio support X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: maxim.uvarov@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.172 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 Signed-off-by: Maxim Uvarov --- helper/include/odp/helper/ring.h | 7 +- helper/ring.c | 3 +- platform/linux-generic/Makefile.am | 3 + .../linux-generic/include/odp_buffer_internal.h | 3 + .../linux-generic/include/odp_packet_io_internal.h | 16 + .../include/odp_packet_io_ipc_internal.h | 48 ++ platform/linux-generic/include/odp_shm_internal.h | 22 + platform/linux-generic/odp_packet_io.c | 19 +- platform/linux-generic/odp_packet_io_ipc.c | 603 +++++++++++++++++++++ platform/linux-generic/odp_pool.c | 23 +- platform/linux-generic/odp_shared_memory.c | 10 +- 11 files changed, 745 insertions(+), 12 deletions(-) create mode 100644 platform/linux-generic/include/odp_packet_io_ipc_internal.h create mode 100644 platform/linux-generic/include/odp_shm_internal.h create mode 100644 platform/linux-generic/odp_packet_io_ipc.c diff --git a/helper/include/odp/helper/ring.h b/helper/include/odp/helper/ring.h index 5e640a7..c3c2f6a 100644 --- a/helper/include/odp/helper/ring.h +++ b/helper/include/odp/helper/ring.h @@ -156,10 +156,11 @@ typedef struct odph_ring { } odph_ring_t; -#define ODPH_RING_F_SP_ENQ 0x0001 /* The default enqueue is "single-producer".*/ -#define ODPH_RING_F_SC_DEQ 0x0002 /* The default dequeue is "single-consumer".*/ -#define ODPH_RING_SHM_PROC 0x0004 /* If set - ring is visible from different +#define ODPH_RING_F_SP_ENQ (1 << 0) /* The default enqueue is "single-producer".*/ +#define ODPH_RING_F_SC_DEQ (1 << 1) /* The default dequeue is "single-consumer".*/ +#define ODPH_RING_SHM_PROC (1 << 2) /* If set - ring is visible from different processes. Default is thread visible. */ +#define ODPH_RING_NO_LIST (1 << 3) /* Do not link ring to linked list. */ #define ODPH_RING_QUOT_EXCEED (1 << 31) /* Quota exceed for burst ops */ #define ODPH_RING_SZ_MASK (unsigned)(0x0fffffff) /* Ring size mask */ diff --git a/helper/ring.c b/helper/ring.c index 0927a6c..54c40cf 100644 --- a/helper/ring.c +++ b/helper/ring.c @@ -200,7 +200,8 @@ odph_ring_create(const char *name, unsigned count, unsigned flags) r->prod.tail = 0; r->cons.tail = 0; - TAILQ_INSERT_TAIL(&odp_ring_list, r, next); + if (!(flags & ODPH_RING_NO_LIST)) + TAILQ_INSERT_TAIL(&odp_ring_list, r, next); } else { ODP_ERR("Cannot reserve memory\n"); } diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 66f0474..4843386 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -120,11 +120,13 @@ noinst_HEADERS = \ ${top_srcdir}/platform/linux-generic/include/odp_internal.h \ ${top_srcdir}/platform/linux-generic/include/odp_packet_internal.h \ ${top_srcdir}/platform/linux-generic/include/odp_packet_io_internal.h \ + ${top_srcdir}/platform/linux-generic/include/odp_packet_io_ipc_internal.h \ ${top_srcdir}/platform/linux-generic/include/odp_packet_io_queue.h \ ${top_srcdir}/platform/linux-generic/include/odp_packet_socket.h \ ${top_srcdir}/platform/linux-generic/include/odp_pool_internal.h \ ${top_srcdir}/platform/linux-generic/include/odp_queue_internal.h \ ${top_srcdir}/platform/linux-generic/include/odp_schedule_internal.h \ + ${top_srcdir}/platform/linux-generic/include/odp_shm_internal.h \ ${top_srcdir}/platform/linux-generic/include/odp_spin_internal.h \ ${top_srcdir}/platform/linux-generic/include/odp_timer_internal.h \ ${top_srcdir}/platform/linux-generic/Makefile.inc @@ -155,6 +157,7 @@ __LIB__libodp_la_SOURCES = \ odp_packet.c \ odp_packet_flags.c \ odp_packet_io.c \ + odp_packet_io_ipc.c \ odp_packet_socket.c \ odp_pool.c \ odp_queue.c \ diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index 3a3d2a2..4ea7c62 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -129,6 +129,9 @@ typedef struct odp_buffer_hdr_t { size_t udata_size; /* size of user metadata */ uint32_t segcount; /* segment count */ uint32_t segsize; /* segment size */ + /* ipc mapped process can not walk over pointers, + * offset has to be used */ + uint64_t ipc_addr_offset[ODP_BUFFER_MAX_SEG]; void *addr[ODP_BUFFER_MAX_SEG]; /* block addrs */ } odp_buffer_hdr_t; diff --git a/platform/linux-generic/include/odp_packet_io_internal.h b/platform/linux-generic/include/odp_packet_io_internal.h index 18b59ef..d12a1fa 100644 --- a/platform/linux-generic/include/odp_packet_io_internal.h +++ b/platform/linux-generic/include/odp_packet_io_internal.h @@ -23,6 +23,7 @@ extern "C" { #include #include #include +#include #include #include @@ -36,6 +37,8 @@ typedef enum { ODP_PKTIO_TYPE_SOCKET_MMSG, ODP_PKTIO_TYPE_SOCKET_MMAP, ODP_PKTIO_TYPE_LOOPBACK, + ODP_PKTIO_TYPE_IPC, + ODP_PKTIO_TYPE_IPC_SLAVE, } odp_pktio_type_t; struct pktio_entry { @@ -53,6 +56,19 @@ struct pktio_entry { char name[IFNAMSIZ]; /**< name of pktio provided to pktio_open() */ odp_bool_t promisc; /**< promiscuous mode state */ + odph_ring_t *ipc_r; /**< ODP ring for IPC msg packets + indexes transmitted to shared memory */ + odph_ring_t *ipc_p; /**< ODP ring for IPC msg packets + indexes already processed by remote process */ + void *ipc_pool_base; /**< IPC Remote pool base addr */ + void *ipc_pool_mdata_base; /**< IPC Remote pool mdata base addr */ + uint64_t ipc_pkt_size; /**< IPC: packet size in remote pool */ + + odph_ring_t *ipc_r_slave; + odph_ring_t *ipc_p_slave; + + odp_pool_t ipc_pool; /**< IPC: Pool of main process */ + odp_shm_t ipc_pool_shm; /**< IPC: Shm memory for remote pool */ }; typedef union { diff --git a/platform/linux-generic/include/odp_packet_io_ipc_internal.h b/platform/linux-generic/include/odp_packet_io_ipc_internal.h new file mode 100644 index 0000000..e11ec35 --- /dev/null +++ b/platform/linux-generic/include/odp_packet_io_ipc_internal.h @@ -0,0 +1,48 @@ +/* Copyright (c) 2015, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +/* IPC packet I/O over odph_ring */ +#include + +#define PKTIO_IPC_ENTRIES 4096 /**< number of odp buffers in + odp ring queue */ + +/* that struct is exported to shared memory, so that 2 processes can find + * each other. + */ +struct pktio_info { + char remote_pool_name[ODP_POOL_NAME_LEN]; + int shm_pool_num; + size_t shm_pkt_pool_size; + uint32_t shm_pkt_size; + odp_shm_t shm; /*< current structure stored in this shm */ + size_t mdata_offset; /*< offset from shared memory block start + *to pool_mdata_addr */ + struct { + size_t mdata_offset; + char pool_name[ODP_POOL_NAME_LEN]; + } slave; +} __packed; + +int ipc_pktio_init(pktio_entry_t *pktio_entry, const char *dev, + odp_pool_t pool); + +int ipc_pktio_recv(pktio_entry_t *pktio_entry, odp_packet_t pkt_table[], + unsigned len); + +int ipc_pktio_send(pktio_entry_t *pktio_entry, odp_packet_t pkt_table[], + unsigned len); diff --git a/platform/linux-generic/include/odp_shm_internal.h b/platform/linux-generic/include/odp_shm_internal.h new file mode 100644 index 0000000..e6b37fa --- /dev/null +++ b/platform/linux-generic/include/odp_shm_internal.h @@ -0,0 +1,22 @@ +/* Copyright (c) 2013, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + + + +#ifndef ODP_SHM_INTERNAL_H_ +#define ODP_SHM_INTERNAL_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#define _ODP_SHM_PROC_NOCREAT 0x4 /**< Do not create shm if not exist */ + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/odp_packet_io.c b/platform/linux-generic/odp_packet_io.c index cfe5b71..0be4cce 100644 --- a/platform/linux-generic/odp_packet_io.c +++ b/platform/linux-generic/odp_packet_io.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include @@ -25,6 +26,9 @@ #include #include +#include +#include + /* MTU to be reported for the "loop" interface */ #define PKTIO_LOOP_MTU 1500 /* MAC address for the "loop" interface */ @@ -263,7 +267,12 @@ static odp_pktio_t setup_pktio_entry(const char *dev, odp_pool_t pool) if (strcmp(dev, "loop") == 0) ret = init_loop(pktio_entry, id); - else + else if (!strncmp(dev, "ipc", 3)) { + ret = ipc_pktio_init(pktio_entry, dev, pool); + if (ret != 0) + ODP_ABORT("unable to init ipc for %s, pool %" PRIu64 "\n", + dev, pool); + } else ret = init_socket(pktio_entry, dev, pool); if (ret != 0) { @@ -408,6 +417,10 @@ int odp_pktio_recv(odp_pktio_t id, odp_packet_t pkt_table[], int len) case ODP_PKTIO_TYPE_LOOPBACK: pkts = deq_loopback(pktio_entry, pkt_table, len); break; + case ODP_PKTIO_TYPE_IPC_SLAVE: + case ODP_PKTIO_TYPE_IPC: + pkts = ipc_pktio_recv(pktio_entry, pkt_table, len); + break; default: pkts = -1; break; @@ -462,6 +475,10 @@ int odp_pktio_send(odp_pktio_t id, odp_packet_t pkt_table[], int len) case ODP_PKTIO_TYPE_LOOPBACK: pkts = enq_loopback(pktio_entry, pkt_table, len); break; + case ODP_PKTIO_TYPE_IPC: + case ODP_PKTIO_TYPE_IPC_SLAVE: + pkts = ipc_pktio_send(pktio_entry, pkt_table, len); + break; default: pkts = -1; } diff --git a/platform/linux-generic/odp_packet_io_ipc.c b/platform/linux-generic/odp_packet_io_ipc.c new file mode 100644 index 0000000..f9ae87f --- /dev/null +++ b/platform/linux-generic/odp_packet_io_ipc.c @@ -0,0 +1,603 @@ +/* Copyright (c) 2015, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +static void *_ipc_map_remote_pool(const char *name, size_t size); + +static const char *_ipc_odp_buffer_pool_shm_name(odp_pool_t pool_hdl) +{ + pool_entry_t *pool; + uint32_t pool_id; + odp_shm_t shm; + odp_shm_info_t info; + + pool_id = pool_handle_to_index(pool_hdl); + pool = get_pool_entry(pool_id); + shm = pool->s.pool_shm; + + odp_shm_info(shm, &info); + + return info.name; +} + +/** +* Look up for shared memory object. +* +* @param name name of shm object +* +* @return 0 on success, otherwise non-zero +*/ +static int _odp_shm_lookup_ipc(const char *name) +{ + int shm; + + shm = shm_open(name, O_RDWR, S_IRUSR | S_IWUSR); + if (shm == -1) { + if (errno == ENOENT) + return -1; + else + ODP_ABORT("shm_open for %s err %s\n", + name, strerror(errno)); + } + close(shm); + return 0; +} + +static struct pktio_info *_ipc_map_pool_info(pktio_entry_t *pktio_entry, + const char *pool_name, + int flag) +{ + struct pktio_info *pinfo; + char name[ODP_POOL_NAME_LEN + sizeof("_info")]; + + /* Create info about remote pktio */ + snprintf(name, sizeof(name), "%s_info", pool_name); + odp_shm_t shm = odp_shm_reserve(name, sizeof(struct pktio_info), + ODP_CACHE_LINE_SIZE, + flag); + if (ODP_SHM_INVALID == shm) + ODP_ABORT("unable to reserve memory for shm info"); + pinfo = odp_shm_addr(shm); + if (flag != _ODP_SHM_PROC_NOCREAT) + pinfo->remote_pool_name[0] = 0; + + pktio_entry->s.ipc_pool_shm = shm; + return pinfo; +} + +static int _ipc_pktio_init_master(pktio_entry_t *pktio_entry, const char *dev, + odp_pool_t pool) +{ + char ipc_shm_name[ODP_POOL_NAME_LEN + sizeof("_slave_r")]; + pool_entry_t *pool_entry; + uint32_t pool_id; + void *ipc_pool_base; + struct pktio_info *pinfo; + const char *pool_name; + odp_shm_t shm; + + pool_id = pool_handle_to_index(pool); + pool_entry = get_pool_entry(pool_id); + + if (ODP_POOL_NAME_LEN != ODPH_RING_NAMESIZE) + ODP_ABORT(""); + + if (strlen(dev) > (ODP_POOL_NAME_LEN - sizeof("_slave_r"))) { + ODP_DBG("too big ipc name\n"); + return -1; + } + + /* generate name in shm like ipc_pktio_r for + * to be processed packets ring. + */ + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_r", dev); + pktio_entry->s.ipc_r = odph_ring_create(ipc_shm_name, + PKTIO_IPC_ENTRIES, + ODPH_RING_SHM_PROC | ODPH_RING_NO_LIST); + if (!pktio_entry->s.ipc_r) { + ODP_DBG("pid %d unable to create ipc ring %s name\n", + getpid(), ipc_shm_name); + return -1; + } + ODP_DBG("Created IPC ring: %s, count %d, free %d\n", + ipc_shm_name, odph_ring_count(pktio_entry->s.ipc_r), + odph_ring_free_count(pktio_entry->s.ipc_r)); + + /* generate name in shm like ipc_pktio_p for + * already processed packets + */ + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_p", dev); + pktio_entry->s.ipc_p = odph_ring_create(ipc_shm_name, + PKTIO_IPC_ENTRIES, + ODPH_RING_SHM_PROC | ODPH_RING_NO_LIST); + if (!pktio_entry->s.ipc_p) { + ODP_DBG("pid %d unable to create ipc ring %s name\n", + getpid(), ipc_shm_name); + goto free_r; + } + ODP_DBG("Created IPC ring: %s, count %d, free %d\n", + ipc_shm_name, odph_ring_count(pktio_entry->s.ipc_p), + odph_ring_free_count(pktio_entry->s.ipc_p)); + + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_slave_r", dev); + pktio_entry->s.ipc_r_slave = odph_ring_create(ipc_shm_name, + PKTIO_IPC_ENTRIES, + ODPH_RING_SHM_PROC | ODPH_RING_NO_LIST); + if (!pktio_entry->s.ipc_r_slave) { + ODP_DBG("pid %d unable to create ipc ring %s name\n", + getpid(), ipc_shm_name); + goto free_p; + } + ODP_DBG("Created IPC ring: %s, count %d, free %d\n", + ipc_shm_name, odph_ring_count(pktio_entry->s.ipc_r_slave), + odph_ring_free_count(pktio_entry->s.ipc_r_slave)); + + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_slave_p", dev); + pktio_entry->s.ipc_p_slave = odph_ring_create(ipc_shm_name, + PKTIO_IPC_ENTRIES, + ODPH_RING_SHM_PROC | ODPH_RING_NO_LIST); + if (!pktio_entry->s.ipc_p_slave) { + ODP_DBG("pid %d unable to create ipc ring %s name\n", + getpid(), ipc_shm_name); + goto free_slave_r; + } + ODP_DBG("Created IPC ring: %s, count %d, free %d\n", + ipc_shm_name, odph_ring_count(pktio_entry->s.ipc_p_slave), + odph_ring_free_count(pktio_entry->s.ipc_p_slave)); + + /* Memory to store information about exported pool */ + pinfo = _ipc_map_pool_info(pktio_entry, dev, ODP_SHM_PROC); + + /* Set up pool name for remote info */ + pool_name = _ipc_odp_buffer_pool_shm_name(pool); + memcpy(pinfo->remote_pool_name, pool_name, strlen(pool_name)); + pinfo->shm_pkt_pool_size = pool_entry->s.pool_size; + pinfo->shm_pool_num = pool_entry->s.buf_num; + pinfo->shm_pkt_size = pool_entry->s.seg_size; + pinfo->mdata_offset = pool_entry->s.pool_mdata_addr - + pool_entry->s.pool_base_addr; + pinfo->slave.mdata_offset = 0; + ODP_DBG("Master waiting for slave to be connected now..\n"); + + /* Wait for remote process to export his pool. */ + ODP_DBG("Wait for second process set mdata_offset...\n"); + while (pinfo->slave.mdata_offset == 0) + odp_spin(); + + ODP_DBG("Wait for second process set mdata_offset... DONE.\n"); + + while (1) { + int ret = _odp_shm_lookup_ipc(pinfo->slave.pool_name); + if (!ret) + break; + ODP_DBG("Master looking for %s\n", pinfo->slave.pool_name); + sleep(1); + } + + ipc_pool_base = _ipc_map_remote_pool(pinfo->slave.pool_name, + pinfo->shm_pkt_pool_size); + pktio_entry->s.ipc_pool_mdata_base = (char *)ipc_pool_base + + pinfo->slave.mdata_offset; + pktio_entry->s.ipc_pool = pool; + + return 0; + +free_slave_r: + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_slave_r", dev); + shm = odp_shm_lookup(ipc_shm_name); + odp_shm_free(shm); +free_p: + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_p", dev); + shm = odp_shm_lookup(ipc_shm_name); + odp_shm_free(shm); +free_r: + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_r", dev); + shm = odp_shm_lookup(ipc_shm_name); + odp_shm_free(shm); + return -1; +} + +static odp_pool_t _ipc_odp_alloc_and_create_pool_slave(struct pktio_info *pinfo) +{ + odp_pool_t pool; + char *pool_name; + odp_pool_param_t params; + int num = pinfo->shm_pool_num; + uint32_t buf_size = pinfo->shm_pkt_size; + pool_entry_t *pool_entry; + + pool_name = calloc(1, strlen(pinfo->remote_pool_name) + + sizeof("ipc_pool_slave_")); + sprintf(pool_name, "ipc_pool_slave_%s", pinfo->remote_pool_name); + + ODP_DBG("slave uses pool %s\n", pool_name); + + memset(¶ms, 0, sizeof(params)); + params.pkt.num = num; + params.pkt.len = buf_size; + params.pkt.seg_len = buf_size; + params.type = ODP_POOL_PACKET; + + pool = odp_pool_create(pool_name, ODP_SHM_NULL, ¶ms); + if (pool == ODP_POOL_INVALID) + ODP_ABORT("Error: packet pool create failed.\n" + "num %d, len %d, seg_len %d\n", + params.pkt.num, params.pkt.len, params.pkt.seg_len); + + /* Export info so that master can connect to that pool*/ + snprintf(pinfo->slave.pool_name, 30, "%s", pool_name); + pool_entry = odp_pool_to_entry(pool); + pinfo->slave.mdata_offset = pool_entry->s.pool_mdata_addr - + pool_entry->s.pool_base_addr; + free(pool_name); + + return pool; +} + +static void *_ipc_map_remote_pool(const char *name, size_t size) +{ + odp_shm_t shm; + + ODP_DBG("Mapping remote pool %s, size %ld\n", name, size); + shm = odp_shm_reserve(name, + size, + ODP_CACHE_LINE_SIZE, + _ODP_SHM_PROC_NOCREAT); + if (shm == ODP_SHM_INVALID) + ODP_ABORT("unable map %s\n", name); + return odp_shm_addr(shm); +} + +static void *_ipc_shm_map(char *name, size_t size, int timeout) +{ + odp_shm_t shm; + int ret; + + while (1) { + ret = _odp_shm_lookup_ipc(name); + if (!ret) + break; + ODP_DBG("Waiting for %s\n", name); + if (timeout <= 0) + return NULL; + timeout--; + sleep(1); + } + + shm = odp_shm_reserve(name, size, + ODP_CACHE_LINE_SIZE, + _ODP_SHM_PROC_NOCREAT); + if (ODP_SHM_INVALID == shm) + ODP_ABORT("unable to map: %s\n", name); + + return odp_shm_addr(shm); +} + +static int _ipc_pktio_init_slave(const char *dev, pktio_entry_t *pktio_entry) +{ + char ipc_shm_name[ODP_POOL_NAME_LEN + sizeof("_slave_r")]; + size_t ring_size = PKTIO_IPC_ENTRIES * sizeof(void *) + + sizeof(odph_ring_t); + struct pktio_info *pinfo; + void *ipc_pool_base; + odp_shm_t shm; + + if (ODP_POOL_NAME_LEN != ODPH_RING_NAMESIZE) + ODP_ABORT(""); + + if (strlen(dev) > (ODP_POOL_NAME_LEN - sizeof("_slave_r"))) { + ODP_DBG("too big ipc name\n"); + return -1; + } + + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_r", dev); + pktio_entry->s.ipc_r = _ipc_shm_map(ipc_shm_name, ring_size, 10); + if (!pktio_entry->s.ipc_r) { + ODP_DBG("pid %d unable to find ipc ring %s name\n", + getpid(), dev); + return -1; + } + ODP_DBG("Connected IPC ring: %s, count %d, free %d\n", + ipc_shm_name, odph_ring_count(pktio_entry->s.ipc_r), + odph_ring_free_count(pktio_entry->s.ipc_r)); + + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_p", dev); + pktio_entry->s.ipc_p = _ipc_shm_map(ipc_shm_name, ring_size, 10); + if (!pktio_entry->s.ipc_p) { + ODP_DBG("pid %d unable to find ipc ring %s name\n", + getpid(), dev); + goto free_r; + } + ODP_DBG("Connected IPC ring: %s, count %d, free %d\n", + ipc_shm_name, odph_ring_count(pktio_entry->s.ipc_p), + odph_ring_free_count(pktio_entry->s.ipc_p)); + + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_slave_r", dev); + pktio_entry->s.ipc_r_slave = _ipc_shm_map(ipc_shm_name, ring_size, 10); + if (!pktio_entry->s.ipc_r_slave) { + ODP_DBG("pid %d unable to find ipc ring %s name\n", + getpid(), dev); + goto free_p; + } + ODP_DBG("Connected IPC ring: %s, count %d, free %d\n", + ipc_shm_name, odph_ring_count(pktio_entry->s.ipc_r_slave), + odph_ring_free_count(pktio_entry->s.ipc_r_slave)); + + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_slave_p", dev); + pktio_entry->s.ipc_p_slave = _ipc_shm_map(ipc_shm_name, ring_size, 10); + if (!pktio_entry->s.ipc_p_slave) { + ODP_DBG("pid %d unable to find ipc ring %s name\n", + getpid(), dev); + goto free_slave_r; + } + ODP_DBG("Connected IPC ring: %s, count %d, free %d\n", + ipc_shm_name, odph_ring_count(pktio_entry->s.ipc_p_slave), + odph_ring_free_count(pktio_entry->s.ipc_p_slave)); + + + /* Get info about remote pool */ + pinfo = _ipc_map_pool_info(pktio_entry, dev, _ODP_SHM_PROC_NOCREAT); + + ipc_pool_base = _ipc_map_remote_pool(pinfo->remote_pool_name, + pinfo->shm_pkt_pool_size); + pktio_entry->s.ipc_pool_mdata_base = (char *)ipc_pool_base + + pinfo->mdata_offset; + pktio_entry->s.ipc_pkt_size = pinfo->shm_pkt_size; + + /* @todo: to simplify in linux-generic implementation we create pool for + * packets from IPC queue. On receive implementation copies packets to + * that pool. Later we can try to reuse original pool without packets + * copying. + */ + pktio_entry->s.ipc_pool = _ipc_odp_alloc_and_create_pool_slave(pinfo); + ODP_DBG("%s OK.\n", __func__); + return 0; + +free_slave_r: + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_slave_r", dev); + shm = odp_shm_lookup(ipc_shm_name); + odp_shm_free(shm); +free_p: + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_p", dev); + shm = odp_shm_lookup(ipc_shm_name); + odp_shm_free(shm); +free_r: + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_r", dev); + shm = odp_shm_lookup(ipc_shm_name); + odp_shm_free(shm); + return -1; +} + +int ipc_pktio_init(pktio_entry_t *pktio_entry, const char *dev, + odp_pool_t pool) +{ + int ret; + + /* if pool is zero we assume that it's slave process connects + * to shared memory already created by main process. + */ + if (ODP_POOL_INVALID != pool) { + pktio_entry->s.type = ODP_PKTIO_TYPE_IPC; + ret = _ipc_pktio_init_master(pktio_entry, dev, pool); + } else { + pktio_entry->s.type = ODP_PKTIO_TYPE_IPC_SLAVE; + ret = _ipc_pktio_init_slave(dev, pktio_entry); + } + + return ret; +} + + +static inline void *_ipc_buffer_map(odp_buffer_hdr_t *buf, + uint32_t offset, + uint32_t *seglen, + uint32_t limit) +{ + int seg_index = offset / buf->segsize; + int seg_offset = offset % buf->segsize; + void *addr = (char *)buf - buf->ipc_addr_offset[seg_index]; + + if (seglen != NULL) { + uint32_t buf_left = limit - offset; + *seglen = seg_offset + buf_left <= buf->segsize ? + buf_left : buf->segsize - seg_offset; + } + + return (void *)(seg_offset + (uint8_t *)addr); +} + + +static inline void *_ipc_packet_map(odp_packet_hdr_t *pkt_hdr, + uint32_t offset, uint32_t *seglen) +{ + if (offset > pkt_hdr->frame_len) + return NULL; + + return _ipc_buffer_map(&pkt_hdr->buf_hdr, + pkt_hdr->headroom + offset, seglen, + pkt_hdr->headroom + pkt_hdr->frame_len); +} + +int ipc_pktio_recv(pktio_entry_t *pktio_entry, + odp_packet_t pkt_table[], unsigned len) +{ + int pkts = 0; + int i; + odph_ring_t *r; + odph_ring_t *r_p; + odp_packet_t remote_pkts[PKTIO_IPC_ENTRIES]; + void **ipcbufs_p = (void *)&remote_pkts; + + if (pktio_entry->s.type == ODP_PKTIO_TYPE_IPC) { + r = pktio_entry->s.ipc_r_slave; + r_p = pktio_entry->s.ipc_p_slave; + } else if (pktio_entry->s.type == ODP_PKTIO_TYPE_IPC_SLAVE) { + r = pktio_entry->s.ipc_r; + r_p = pktio_entry->s.ipc_p; + } else { + ODP_ABORT("wrong type: %d\n", pktio_entry->s.type); + } + + pkts = odph_ring_mc_dequeue_burst(r, ipcbufs_p, len); + if (odp_unlikely(pkts < 0)) + ODP_ABORT("error to dequeue no packets\n"); + + /* fast path */ + if (odp_likely(0 == pkts)) + return 0; + + for (i = 0; i < pkts; i++) { + odp_pool_t pool; + odp_packet_t pkt; + odp_packet_hdr_t *phdr; + odp_buffer_bits_t handle; + int idx; /* Remote packet has coded pool and index. + * We need only index.*/ + void *pkt_data; + void *remote_pkt_data; + + handle.handle = _odp_packet_to_buffer(remote_pkts[i]); + idx = handle.index; + + /* Link to packed data. To this line we have Zero-Copy between + * processes, to simplify use packet copy in that version which + * can be removed later with more advance buffer management + * (ref counters). + */ + /* reverse odp_buf_to_hdr() */ + phdr = (odp_packet_hdr_t *)( + (char *)pktio_entry->s.ipc_pool_mdata_base + + (idx * ODP_CACHE_LINE_SIZE)); + + /* Allocate new packet. Select*/ + pool = pktio_entry->s.ipc_pool; + if (odp_unlikely(pool == ODP_POOL_INVALID)) + ODP_ABORT("invalid pool"); + + pkt = odp_packet_alloc(pool, phdr->frame_len); + if (odp_unlikely(pkt == ODP_PACKET_INVALID)) { + /* Original pool might be smaller then + * PKTIO_IPC_ENTRIES. If packet can not be + * allocated from pool at this time, + * simple get in on next recv() call. + */ + pkts = i - 1; + break; + } + + /* Copy packet data. */ + pkt_data = odp_packet_data(pkt); + if (odp_unlikely(pkt_data == NULL)) + ODP_ABORT("unable to map pkt_data ipc_slave %d\n", + (ODP_PKTIO_TYPE_IPC_SLAVE == + pktio_entry->s.type)); + + remote_pkt_data = _ipc_packet_map(phdr, 0, NULL); + if (odp_unlikely(remote_pkt_data == NULL)) + ODP_ABORT("unable to map remote_pkt_data, ipc_slave %d\n", + (ODP_PKTIO_TYPE_IPC_SLAVE == + pktio_entry->s.type)); + + /* @todo fix copy packet!!! */ + memcpy(pkt_data, remote_pkt_data, phdr->frame_len); + + /* Copy packets L2, L3 parsed offsets and size */ + odp_packet_hdr(pkt)->l2_offset = phdr->l2_offset; + odp_packet_hdr(pkt)->l3_offset = phdr->l3_offset; + odp_packet_hdr(pkt)->l4_offset = phdr->l4_offset; + odp_packet_hdr(pkt)->payload_offset = phdr->payload_offset; + + odp_packet_hdr(pkt)->vlan_s_tag = phdr->vlan_s_tag; + odp_packet_hdr(pkt)->vlan_c_tag = phdr->vlan_c_tag; + odp_packet_hdr(pkt)->l3_protocol = phdr->l3_protocol; + odp_packet_hdr(pkt)->l3_len = phdr->l3_len; + + odp_packet_hdr(pkt)->frame_len = phdr->frame_len; + odp_packet_hdr(pkt)->headroom = phdr->headroom; + odp_packet_hdr(pkt)->tailroom = phdr->tailroom; + pkt_table[i] = pkt; + } + + /* Now tell other process that we no longer need that buffers.*/ + pkts = odph_ring_mp_enqueue_burst(r_p, ipcbufs_p, pkts); + if (odp_unlikely(pkts < 0)) + ODP_ABORT("ipc: odp_ring_mp_enqueue_bulk r_p fail\n"); + + return pkts; +} + +int ipc_pktio_send(pktio_entry_t *pktio_entry, odp_packet_t pkt_table[], + unsigned len) +{ + odph_ring_t *r; + odph_ring_t *r_p; + void **rbuf_p; + int ret; + unsigned i; + + if (pktio_entry->s.type == ODP_PKTIO_TYPE_IPC_SLAVE) { + r = pktio_entry->s.ipc_r_slave; + r_p = pktio_entry->s.ipc_p_slave; + } else if (pktio_entry->s.type == ODP_PKTIO_TYPE_IPC) { + r = pktio_entry->s.ipc_r; + r_p = pktio_entry->s.ipc_p; + } else { + ODP_ABORT("wrong type: %d\n", pktio_entry->s.type); + } + + /* Free already processed packets, if any */ + while (1) { + odp_packet_t r_p_pkts[PKTIO_IPC_ENTRIES]; + rbuf_p = (void *)&r_p_pkts; + + ret = odph_ring_mc_dequeue_burst(r_p, rbuf_p, + PKTIO_IPC_ENTRIES); + if (0 == ret) + break; + for (i = 0; i < (unsigned)ret; i++) + odp_packet_free(r_p_pkts[i]); + } + + /* Prepare packets: calculate offset from address. */ + for (i = 0; i < len; i++) { + int j; + odp_packet_t pkt = pkt_table[i]; + rbuf_p = (void *)&pkt; + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); + + /* buf_hdr.addr can not be used directly in remote process, + * convert it to offset + */ + for (j = 0; j < ODP_BUFFER_MAX_SEG; j++) + pkt_hdr->buf_hdr.ipc_addr_offset[j] = (char *)pkt_hdr - + (char *)pkt_hdr->buf_hdr.addr[j]; + } + + /* Put packets to ring to be processed in other process. */ + rbuf_p = (void *)&pkt_table[0]; + ret = odph_ring_mp_enqueue_burst(r, rbuf_p, len); + if (odp_unlikely(ret < 0)) { + ODP_ERR("pid %d odp_ring_mp_enqueue_bulk fail, ipc_slave %d, ret %d\n", + getpid(), + (ODP_PKTIO_TYPE_IPC_SLAVE == pktio_entry->s.type), + ret); + ODP_ERR("odp_ring_full: %d, odp_ring_count %d, odph_ring_free_count %d\n", + odph_ring_full(r), odph_ring_count(r), + odph_ring_free_count(r)); + } + + return ret; +} diff --git a/platform/linux-generic/odp_pool.c b/platform/linux-generic/odp_pool.c index a3d80b5..78153d1 100644 --- a/platform/linux-generic/odp_pool.c +++ b/platform/linux-generic/odp_pool.c @@ -231,8 +231,11 @@ odp_pool_t odp_pool_create(const char *name, ODP_ALIGN_ROUNDUP(params->pkt.len, seg_len); /* Reject create if pkt.len needs too many segments */ - if (blk_size / seg_len > ODP_BUFFER_MAX_SEG) + if (blk_size / seg_len > ODP_BUFFER_MAX_SEG) { + ODP_ERR("ODP_BUFFER_MAX_SEG exceed %d(%d)\n", + blk_size / seg_len, ODP_BUFFER_MAX_SEG); return ODP_POOL_INVALID; + } buf_stride = sizeof(odp_packet_hdr_stride); break; @@ -249,8 +252,12 @@ odp_pool_t odp_pool_create(const char *name, /* Validate requested number of buffers against addressable limits */ if (buf_num > - (ODP_BUFFER_MAX_BUFFERS / (buf_stride / ODP_CACHE_LINE_SIZE))) + (ODP_BUFFER_MAX_BUFFERS / (buf_stride / ODP_CACHE_LINE_SIZE))) { + ODP_ERR("buf_num %d > then expected %d\n", + buf_num, ODP_BUFFER_MAX_BUFFERS / + (buf_stride / ODP_CACHE_LINE_SIZE)); return ODP_POOL_INVALID; + } /* Find an unused buffer pool slot and iniitalize it as requested */ for (i = 0; i < ODP_CONFIG_POOLS; i++) { @@ -302,7 +309,8 @@ odp_pool_t odp_pool_create(const char *name, if (shm == ODP_SHM_NULL) { shm = odp_shm_reserve(pool->s.name, pool->s.pool_size, - ODP_PAGE_SIZE, 0); + ODP_PAGE_SIZE, + ODP_SHM_PROC); if (shm == ODP_SHM_INVALID) { POOL_UNLOCK(&pool->s.lock); return ODP_POOL_INVALID; @@ -310,8 +318,12 @@ odp_pool_t odp_pool_create(const char *name, pool->s.pool_base_addr = odp_shm_addr(shm); } else { odp_shm_info_t info; - if (odp_shm_info(shm, &info) != 0 || - info.size < pool->s.pool_size) { + int ret; + + ret = odp_shm_info(shm, &info); + if (ret != 0 || info.size < pool->s.pool_size) { + ODP_ERR("shm info %d, info size %ld, pool size %ld\n", + ret, info.size, pool->s.pool_size); POOL_UNLOCK(&pool->s.lock); return ODP_POOL_INVALID; } @@ -324,6 +336,7 @@ odp_pool_t odp_pool_create(const char *name, ((size_t)page_addr - (size_t)pool->s.pool_base_addr)) { POOL_UNLOCK(&pool->s.lock); + ODP_ERR("small shm size\n"); return ODP_POOL_INVALID; } pool->s.pool_base_addr = page_addr; diff --git a/platform/linux-generic/odp_shared_memory.c b/platform/linux-generic/odp_shared_memory.c index ab48dda..5de48d3 100644 --- a/platform/linux-generic/odp_shared_memory.c +++ b/platform/linux-generic/odp_shared_memory.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include @@ -189,7 +190,7 @@ odp_shm_t odp_shm_reserve(const char *name, uint64_t size, uint64_t align, int fd = -1; int map_flag = MAP_SHARED; /* If already exists: O_EXCL: error, O_TRUNC: truncate to zero */ - int oflag = O_RDWR | O_CREAT | O_TRUNC; + int oflag = O_RDWR; uint64_t alloc_size; uint64_t page_sz, huge_sz; #ifdef MAP_HUGETLB @@ -207,7 +208,12 @@ odp_shm_t odp_shm_reserve(const char *name, uint64_t size, uint64_t align, alloc_hp_size = (size + align + (huge_sz - 1)) & (-huge_sz); #endif - if (flags & ODP_SHM_PROC) { + if (flags & ODP_SHM_PROC) + oflag |= O_CREAT | O_TRUNC; + + if (flags & (ODP_SHM_PROC | _ODP_SHM_PROC_NOCREAT)) { + need_huge_page = 0; + /* Creates a file to /dev/shm */ fd = shm_open(name, oflag, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);