From patchwork Wed Nov 5 19:07:38 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Uvarov X-Patchwork-Id: 40219 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-wi0-f198.google.com (mail-wi0-f198.google.com [209.85.212.198]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 40A86240A6 for ; Wed, 5 Nov 2014 19:08:16 +0000 (UTC) Received: by mail-wi0-f198.google.com with SMTP id n3sf1257460wiv.9 for ; Wed, 05 Nov 2014 11:08:15 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:date:message-id:subject :precedence:list-id:list-unsubscribe:list-archive:list-post :list-help:list-subscribe:mime-version:errors-to:sender :x-original-sender:x-original-authentication-results:mailing-list :content-type:content-transfer-encoding; bh=MieFuyST09kMU862GdxDKwGgT+XSdfsW4oGQK7gfKvI=; b=ZbgAgAEWMLW+5ywIQ5OZ34DMJxjVR1sW3qH5dTUIVrTMFWNltFYECbPFE8/P1AEOeE ZT20AJ550YI2QVds/B+eXD026DVayFmH60x9qYb97UocGzqAvD9AcpYRV3NiI5Stl0iW 4HWUI4GOJjGJbXjoLH/CxXWghmdb7e9jf8E7fF7Vy/iQQHjlSbV91pVBZLOzNVZWvXBr SO2LjizVjKo2TZZfUnQhLx0VPYeckPe0dqF8TTtzrwjwnM0aV9ZHaZHRGH/CXURsoilH pQjK1bYWzzmSHQ/8cBIP0CYAEjomuJ3Y59Bvybq5iH2I9ax1vJrdICrx4U7H1KwD4s9N 1L1Q== X-Gm-Message-State: ALoCoQkhIJGduipmhdTcTe+mftucUVTYyal5XUPnAaX2n1fFZArS+kLFNekK03SiC6JsPdVXGPVZ X-Received: by 10.112.159.199 with SMTP id xe7mr11420lbb.21.1415214495419; Wed, 05 Nov 2014 11:08:15 -0800 (PST) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.23.103 with SMTP id l7ls788567laf.6.gmail; Wed, 05 Nov 2014 11:08:15 -0800 (PST) X-Received: by 10.112.77.74 with SMTP id q10mr69502681lbw.66.1415214495125; Wed, 05 Nov 2014 11:08:15 -0800 (PST) Received: from mail-la0-f51.google.com (mail-la0-f51.google.com. [209.85.215.51]) by mx.google.com with ESMTPS id m10si7795449laj.61.2014.11.05.11.08.14 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 05 Nov 2014 11:08:14 -0800 (PST) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.51 as permitted sender) client-ip=209.85.215.51; Received: by mail-la0-f51.google.com with SMTP id q1so1298823lam.10 for ; Wed, 05 Nov 2014 11:08:14 -0800 (PST) X-Received: by 10.152.87.100 with SMTP id w4mr7292243laz.27.1415214494662; Wed, 05 Nov 2014 11:08:14 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.184.201 with SMTP id ew9csp338072lbc; Wed, 5 Nov 2014 11:08:12 -0800 (PST) X-Received: by 10.236.60.195 with SMTP id u43mr16115522yhc.142.1415214491426; Wed, 05 Nov 2014 11:08:11 -0800 (PST) Received: from ip-10-35-177-41.ec2.internal (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTPS id 50si7897564qgc.95.2014.11.05.11.08.10 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Wed, 05 Nov 2014 11:08:11 -0800 (PST) Received-SPF: none (google.com: lng-odp-bounces@lists.linaro.org does not designate permitted sender hosts) client-ip=54.225.227.206; Received: from localhost ([127.0.0.1] helo=ip-10-35-177-41.ec2.internal) by ip-10-35-177-41.ec2.internal with esmtp (Exim 4.76) (envelope-from ) id 1Xm5w8-00062n-5r; Wed, 05 Nov 2014 19:08:08 +0000 Received: from mail-lb0-f175.google.com ([209.85.217.175]) by ip-10-35-177-41.ec2.internal with esmtp (Exim 4.76) (envelope-from ) id 1Xm5vw-00062U-Ll for lng-odp@lists.linaro.org; Wed, 05 Nov 2014 19:07:57 +0000 Received: by mail-lb0-f175.google.com with SMTP id n15so1262807lbi.34 for ; Wed, 05 Nov 2014 11:07:50 -0800 (PST) X-Received: by 10.112.182.1 with SMTP id ea1mr69417666lbc.16.1415214470349; Wed, 05 Nov 2014 11:07:50 -0800 (PST) Received: from localhost.localdomain (ppp91-76-163-205.pppoe.mtu-net.ru. [91.76.163.205]) by mx.google.com with ESMTPSA id r4sm1631138lar.3.2014.11.05.11.07.47 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 05 Nov 2014 11:07:49 -0800 (PST) From: Maxim Uvarov To: lng-odp@lists.linaro.org Date: Wed, 5 Nov 2014 22:07:38 +0300 Message-Id: <1415214459-14179-1-git-send-email-maxim.uvarov@linaro.org> X-Mailer: git-send-email 1.8.5.1.163.gd7aced9 X-Topics: patch Subject: [lng-odp] [PATCHv3] ipc linux-generic implementation based on pktio X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Errors-To: lng-odp-bounces@lists.linaro.org Sender: lng-odp-bounces@lists.linaro.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: maxim.uvarov@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.51 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 Signed-off-by: Maxim Uvarov --- v3: - add fixed proposed by Jerin Jacob (pkt_lookup should be more portable and for linux-generic it has to create pool if needed). I.e.: odp_pktio_lookup("ipc_pktio", pool, pool_base) now goes to simple odp_pktio_lookup("ipc_pktio"); - put IPC linux-generic implementation to different files; - removed bunch of preset defines @todo: One function has predefined value due to need new pool API to get shared memory block name from pool handler. Added placeholder in: const char *odp_buffer_pool_shm_name(odp_buffer_pool_t pool_hdl). v2: - remove out2_setdef; - add ipc to configure.ac which somehow was missed in first patch; configure.ac | 1 + example/Makefile.am | 2 +- example/ipc/Makefile.am | 6 + example/ipc/odp_pktio.c | 717 +++++++++++++++++++++ helper/include/odph_ring.h | 3 + platform/linux-generic/Makefile.am | 1 + platform/linux-generic/include/api/odp_packet_io.h | 10 + .../linux-generic/include/api/odp_shared_memory.h | 11 + .../include/odp_buffer_pool_internal.h | 17 + platform/linux-generic/include/odp_ipc.h | 42 ++ .../linux-generic/include/odp_packet_io_internal.h | 8 + platform/linux-generic/odp_buffer_pool.c | 41 +- platform/linux-generic/odp_init.c | 6 + platform/linux-generic/odp_ipc.c | 319 +++++++++ platform/linux-generic/odp_packet_io.c | 52 ++ platform/linux-generic/odp_ring.c | 9 +- platform/linux-generic/odp_shared_memory.c | 26 +- 17 files changed, 1255 insertions(+), 16 deletions(-) create mode 100644 example/ipc/Makefile.am create mode 100644 example/ipc/odp_pktio.c create mode 100644 platform/linux-generic/include/odp_ipc.h create mode 100644 platform/linux-generic/odp_ipc.c diff --git a/configure.ac b/configure.ac index 46eaec1..8a2fa0e 100644 --- a/configure.ac +++ b/configure.ac @@ -131,6 +131,7 @@ AC_CONFIG_FILES([Makefile platform/linux-generic/Makefile example/Makefile example/generator/Makefile + example/ipc/Makefile example/ipsec/Makefile example/l2fwd/Makefile example/odp_example/Makefile diff --git a/example/Makefile.am b/example/Makefile.am index b2a22a3..7911069 100644 --- a/example/Makefile.am +++ b/example/Makefile.am @@ -1 +1 @@ -SUBDIRS = generator ipsec l2fwd odp_example packet timer +SUBDIRS = generator ipsec l2fwd odp_example packet timer ipc diff --git a/example/ipc/Makefile.am b/example/ipc/Makefile.am new file mode 100644 index 0000000..603a1ab --- /dev/null +++ b/example/ipc/Makefile.am @@ -0,0 +1,6 @@ +include $(top_srcdir)/example/Makefile.inc + +bin_PROGRAMS = odp_pktio +odp_pktio_LDFLAGS = $(AM_LDFLAGS) -static + +dist_odp_pktio_SOURCES = odp_pktio.c diff --git a/example/ipc/odp_pktio.c b/example/ipc/odp_pktio.c new file mode 100644 index 0000000..bbb2621 --- /dev/null +++ b/example/ipc/odp_pktio.c @@ -0,0 +1,717 @@ +/* Copyright (c) 2014, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/** + * @file + * + * @example odp_pktio.c ODP basic packet IO loopback test application + */ + +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +/** @def MAX_WORKERS + * @brief Maximum number of worker threads + */ +#define MAX_WORKERS 32 + +/** @def SHM_PKT_POOL_SIZE + * @brief Size of the shared memory block + */ +#define SHM_PKT_POOL_SIZE (512*2048) + +/** @def SHM_PKT_POOL_BUF_SIZE + * @brief Buffer size of the packet pool buffer + */ +#define SHM_PKT_POOL_BUF_SIZE 1856 + +/** @def MAX_PKT_BURST + * @brief Maximum number of packet bursts + */ +#define MAX_PKT_BURST 16 + +/** @def APPL_MODE_PKT_BURST + * @brief The application will handle pakcets in bursts + */ +#define APPL_MODE_PKT_BURST 0 + +/** @def APPL_MODE_PKT_QUEUE + * @brief The application will handle packets in queues + */ +#define APPL_MODE_PKT_QUEUE 1 + +/** @def PRINT_APPL_MODE(x) + * @brief Macro to print the current status of how the application handles + * packets. + */ +#define PRINT_APPL_MODE(x) printf("%s(%i)\n", #x, (x)) + +/** Get rid of path in filename - only for unix-type paths using '/' */ +#define NO_PATH(file_name) (strrchr((file_name), '/') ? \ + strrchr((file_name), '/') + 1 : (file_name)) +/** + * Parsed command line application arguments + */ +typedef struct { + int core_count; + int if_count; /**< Number of interfaces to be used */ + char **if_names; /**< Array of pointers to interface names */ + int mode; /**< Packet IO mode */ + odp_buffer_pool_t pool; /**< Buffer pool for packet IO */ +} appl_args_t; + +/** + * Thread specific arguments + */ +typedef struct { + char *pktio_dev; /**< Interface name to use */ + odp_buffer_pool_t pool; /**< Buffer pool for packet IO */ + int mode; /**< Thread mode */ +} thread_args_t; + +/** + * Grouping of both parsed CL args and thread specific args - alloc together + */ +typedef struct { + /** Application (parsed) arguments */ + appl_args_t appl; + /** Thread specific arguments */ + thread_args_t thread[MAX_WORKERS]; +} args_t; + +/** Global pointer to args */ +static args_t *args; + +/* helper funcs */ +static int drop_err_pkts(odp_packet_t pkt_tbl[], unsigned len); +static void swap_pkt_addrs(odp_packet_t pkt_tbl[], unsigned len); +static void parse_args(int argc, char *argv[], appl_args_t *appl_args); +static void print_info(char *progname, appl_args_t *appl_args); +static void usage(char *progname); + +/** + * Packet IO loopback worker thread using ODP queues + * + * @param arg thread arguments of type 'thread_args_t *' + */ +static void *pktio_queue_thread(void *arg) +{ + int thr; + odp_buffer_pool_t pkt_pool; + odp_pktio_t pktio; + odp_pktio_t ipc_pktio; + thread_args_t *thr_args; + odp_queue_t ipcq; + odp_queue_t inq_def; + char inq_name[ODP_QUEUE_NAME_LEN]; + odp_queue_param_t qparam; + odp_packet_t pkt; + odp_buffer_t buf; + int ret; + unsigned long pkt_cnt = 0; + unsigned long err_cnt = 0; + + thr = odp_thread_id(); + thr_args = arg; + + printf("Pktio thread [%02i] starts, pktio_dev:%s\n", thr, + thr_args->pktio_dev); + + /* Lookup the packet pool */ + pkt_pool = odp_buffer_pool_lookup("packet_pool"); + if (pkt_pool == ODP_BUFFER_POOL_INVALID || pkt_pool != thr_args->pool) { + ODP_ERR(" [%02i] Error: pkt_pool not found\n", thr); + return NULL; + } + + /* Open a packet IO instance for this thread */ + pktio = odp_pktio_open(thr_args->pktio_dev, pkt_pool); + if (pktio == ODP_PKTIO_INVALID) { + ODP_ERR(" [%02i] Error: pktio create failed\n", thr); + return NULL; + } + + /* + * Create and set the default INPUT queue associated with the 'pktio' + * resource + */ + qparam.sched.prio = ODP_SCHED_PRIO_DEFAULT; + qparam.sched.sync = ODP_SCHED_SYNC_ATOMIC; + qparam.sched.group = ODP_SCHED_GROUP_DEFAULT; + snprintf(inq_name, sizeof(inq_name), "%i-pktio_inq_def", (int)pktio); + inq_name[ODP_QUEUE_NAME_LEN - 1] = '\0'; + + inq_def = odp_queue_create(inq_name, ODP_QUEUE_TYPE_PKTIN, &qparam); + if (inq_def == ODP_QUEUE_INVALID) { + ODP_ERR(" [%02i] Error: pktio queue creation failed\n", thr); + return NULL; + } + + ret = odp_pktio_inq_setdef(pktio, inq_def); + if (ret != 0) { + ODP_ERR(" [%02i] Error: default input-Q setup\n", thr); + return NULL; + } + + /* IPC pktio */ + ipc_pktio = odp_pktio_open("ipc_pktio", pkt_pool); + if (ipc_pktio == ODP_PKTIO_INVALID) { + ODP_ERR(" [%02i] Error: ipc pktio create failed.\n", thr); + return NULL; + } + + /* Get reference to output IPC queue */ + ipcq = odp_pktio_outq_getdef(ipc_pktio); + if (ipcq == ODP_QUEUE_INVALID) { + ODP_ERR(" [%02i] Error: get output queue for ipc_pktio\n", + thr); + return NULL; + } + + printf(" [%02i] created pktio:%02i, queue mode (ATOMIC queues)\n" + " default pktio%02i-INPUT queue:%u\n", + thr, pktio, pktio, inq_def); + + /* Loop packets */ + for (;;) { + /* Use schedule to get buf from any input queue */ + buf = odp_schedule(NULL, ODP_SCHED_WAIT); + + pkt = odp_packet_from_buffer(buf); + + /* Drop packets with errors */ + if (odp_unlikely(drop_err_pkts(&pkt, 1) == 0)) { + ODP_ERR("Drop frame - err_cnt:%lu\n", ++err_cnt); + continue; + } + + /* Swap Eth MACs and possibly IP-addrs before sending back */ + swap_pkt_addrs(&pkt, 1); + + /* Enqueue the packet for output */ + odp_queue_enq(ipcq, buf); + + /* Print packet counts every once in a while */ + if (odp_unlikely(pkt_cnt++ % 100000 == 0)) { + printf(" [%02i] pkt_cnt:%lu\n", thr, pkt_cnt); + fflush(NULL); + } + } + + ODP_ABORT("unreachable code"); +} + +/** + * Packet IO loopback worker thread using bursts from/to IO resources + * + * @param arg thread arguments of type 'thread_args_t *' + */ +static void *pktio_ifburst_thread(void *arg) +{ + int thr; + odp_buffer_pool_t pkt_pool; + odp_pktio_t pktio; + odp_pktio_t ipc_pktio; + thread_args_t *thr_args; + int pkts, pkts_ok; + odp_packet_t pkt_tbl[MAX_PKT_BURST]; + unsigned long pkt_cnt = 0; + unsigned long err_cnt = 0; + unsigned long tmp = 0; + + thr = odp_thread_id(); + thr_args = arg; + + printf("Pktio thread [%02i] starts, pktio_dev:%s\n", thr, + thr_args->pktio_dev); + + /* Lookup the packet pool */ + pkt_pool = odp_buffer_pool_lookup("packet_pool"); + if (pkt_pool == ODP_BUFFER_POOL_INVALID || pkt_pool != thr_args->pool) { + ODP_ERR(" [%02i] Error: pkt_pool not found\n", thr); + return NULL; + } + + /* Open a packet IO instance for this thread */ + pktio = odp_pktio_open(thr_args->pktio_dev, pkt_pool); + if (pktio == ODP_PKTIO_INVALID) { + ODP_ERR(" [%02i] Error: pktio create failed.\n", thr); + return NULL; + } + + printf(" [%02i] created pktio:%02i, burst mode\n", + thr, pktio); + + printf("pid: %d, create IPC pktio\n", getpid()); + + ipc_pktio = odp_pktio_open("ipc_pktio", pkt_pool); + if (ipc_pktio == ODP_PKTIO_INVALID) { + ODP_ERR(" [%02i] Error: ipc pktio create failed.\n", thr); + return NULL; + } + + /* Loop packets */ + for (;;) { + pkts = odp_pktio_recv(pktio, pkt_tbl, MAX_PKT_BURST); + if (pkts > 0) { + /* Drop packets with errors */ + pkts_ok = drop_err_pkts(pkt_tbl, pkts); + if (pkts_ok > 0) { + /* Swap Eth MACs and IP-addrs */ + swap_pkt_addrs(pkt_tbl, pkts_ok); + odp_pktio_send(ipc_pktio, pkt_tbl, pkts_ok); + } + + if (odp_unlikely(pkts_ok != pkts)) + ODP_ERR("Dropped frames:%u - err_cnt:%lu\n", + pkts-pkts_ok, ++err_cnt); + + /* Print packet counts every once in a while */ + tmp += pkts_ok; + if (odp_unlikely((tmp >= 100000) || /* OR first print:*/ + ((pkt_cnt == 0) && ((tmp-1) < MAX_PKT_BURST)))) { + pkt_cnt += tmp; + printf(" [%02i] pkt_cnt:%lu\n", thr, pkt_cnt); + fflush(NULL); + tmp = 0; + } + } + } + +/* unreachable */ +} + + +static int ipc_second_process(void) +{ + odp_pktio_t pktio; + odp_packet_t pkt_tbl[MAX_PKT_BURST]; + int i; + int pkts; + + sleep(3); + + /* Do lookup packet I/O in IPC shared memory, + * and link it to local pool. */ + while (1) { + pktio = odp_pktio_lookup("ipc_pktio"); + if (pktio == ODP_PKTIO_INVALID) { + sleep(1); + printf("pid %d: looking for ipc_pktio\n", getpid()); + continue; + } + break; + } + + for (;;) { + pkts = odp_pktio_recv(pktio, pkt_tbl, MAX_PKT_BURST); + if (pkts > 0) { + for (i = 0; i < pkts; i++) { + ODP_DBG("%s: pid %d, got packet %d, size %ld\n", + __func__, getpid(), pkt_tbl[i], + odp_packet_get_len(pkt_tbl[i])); + odp_buffer_free(pkt_tbl[i]); + } + } else { + /* No need to load cpu in example app.*/ + sleep(1); + } + } + + ODP_ABORT("Unexpected close."); + return 0; +} + + +/** + * ODP packet example main function + */ +int main(int argc, char *argv[]) +{ + odph_linux_pthread_t thread_tbl[MAX_WORKERS]; + odp_buffer_pool_t pool; + int num_workers; + void *pool_base; + int i; + int first_core; + int core_count; + odp_shm_t shm; + int f; + + + f = fork(); + if (f) { + printf("Process one pid: %d\n", getpid()); + /* Init ODP before calling anything else */ + if (odp_init_global()) { + ODP_ERR("Error: ODP global init failed.\n"); + exit(EXIT_FAILURE); + } + + /* Init this thread */ + if (odp_init_local()) { + ODP_ERR("Error: ODP local init failed.\n"); + exit(EXIT_FAILURE); + } + + ipc_second_process(); + } else { + printf("Process two pid: %d\n", getpid()); + } + + + /* Init ODP before calling anything else */ + if (odp_init_global()) { + ODP_ERR("Error: ODP global init failed.\n"); + exit(EXIT_FAILURE); + } + + /* Init this thread */ + if (odp_init_local()) { + ODP_ERR("Error: ODP local init failed.\n"); + exit(EXIT_FAILURE); + } + + /* At early stage fork to 2 separate processes */ + /* Reserve memory for args from shared mem */ + shm = odp_shm_reserve("shm_args", sizeof(args_t), + ODP_CACHE_LINE_SIZE, 0); + args = odp_shm_addr(shm); + + if (args == NULL) { + ODP_ERR("Error: shared mem alloc failed.\n"); + exit(EXIT_FAILURE); + } + memset(args, 0, sizeof(*args)); + + /* Parse and store the application arguments */ + parse_args(argc, argv, &args->appl); + + /* Print both system and application information */ + print_info(NO_PATH(argv[0]), &args->appl); + + core_count = odp_sys_core_count(); + num_workers = core_count; + + if (args->appl.core_count) + num_workers = args->appl.core_count; + + if (num_workers > MAX_WORKERS) + num_workers = MAX_WORKERS; + + printf("Num worker threads: %i\n", num_workers); + + /* + * By default core #0 runs Linux kernel background tasks. + * Start mapping thread from core #1 + */ + first_core = 1; + if (core_count == 1) + first_core = 0; + + printf("First core: %i\n\n", first_core); + + /* Create packet pool in shared memory */ + shm = odp_shm_reserve("shm_packet_pool", + SHM_PKT_POOL_SIZE, + ODP_CACHE_LINE_SIZE, + ODP_SHM_PROC); + pool_base = odp_shm_addr(shm); + + if (pool_base == NULL) { + ODP_ERR("Error: packet pool mem alloc failed.\n"); + exit(EXIT_FAILURE); + } + + pool = odp_buffer_pool_create("packet_pool", pool_base, + SHM_PKT_POOL_SIZE, + SHM_PKT_POOL_BUF_SIZE, + ODP_CACHE_LINE_SIZE, + ODP_BUFFER_TYPE_PACKET); + if (pool == ODP_BUFFER_POOL_INVALID) { + ODP_ERR("Error: packet pool create failed.\n"); + exit(EXIT_FAILURE); + } + odp_buffer_pool_print(pool); + + /* Create and init worker threads */ + memset(thread_tbl, 0, sizeof(thread_tbl)); + for (i = 0; i < num_workers; ++i) { + void *(*thr_run_func) (void *); + int core; + int if_idx; + + core = (first_core + i) % core_count; + + if_idx = i % args->appl.if_count; + + args->thread[i].pktio_dev = args->appl.if_names[if_idx]; + args->thread[i].pool = pool; + args->thread[i].mode = args->appl.mode; + + if (args->appl.mode == APPL_MODE_PKT_BURST) + thr_run_func = pktio_ifburst_thread; + else /* APPL_MODE_PKT_QUEUE */ + thr_run_func = pktio_queue_thread; + /* + * Create threads one-by-one instead of all-at-once, + * because each thread might get different arguments. + * Calls odp_thread_create(cpu) for each thread + */ + odph_linux_pthread_create(&thread_tbl[i], 1, core, thr_run_func, + &args->thread[i]); + } + + /* Master thread waits for other threads to exit */ + odph_linux_pthread_join(thread_tbl, num_workers); + + printf("Exit\n\n"); + + return 0; +} + +/** + * Drop packets which input parsing marked as containing errors. + * + * Frees packets with error and modifies pkt_tbl[] to only contain packets with + * no detected errors. + * + * @param pkt_tbl Array of packet + * @param len Length of pkt_tbl[] + * + * @return Number of packets with no detected error + */ +static int drop_err_pkts(odp_packet_t pkt_tbl[], unsigned len) +{ + odp_packet_t pkt; + unsigned pkt_cnt = len; + unsigned i, j; + + for (i = 0, j = 0; i < len; ++i) { + pkt = pkt_tbl[i]; + + if (odp_unlikely(odp_packet_error(pkt))) { + odph_packet_free(pkt); /* Drop */ + pkt_cnt--; + } else if (odp_unlikely(i != j++)) { + pkt_tbl[j-1] = pkt; + } + } + + return pkt_cnt; +} + +/** + * Swap eth src<->dst and IP src<->dst addresses + * + * @param pkt_tbl Array of packets + * @param len Length of pkt_tbl[] + */ + +static void swap_pkt_addrs(odp_packet_t pkt_tbl[], unsigned len) +{ + odp_packet_t pkt; + odph_ethhdr_t *eth; + odph_ethaddr_t tmp_addr; + odph_ipv4hdr_t *ip; + uint32be_t ip_tmp_addr; /* tmp ip addr */ + unsigned i; + + for (i = 0; i < len; ++i) { + pkt = pkt_tbl[i]; + if (odp_packet_inflag_eth(pkt)) { + eth = (odph_ethhdr_t *)odp_packet_l2(pkt); + + tmp_addr = eth->dst; + eth->dst = eth->src; + eth->src = tmp_addr; + + if (odp_packet_inflag_ipv4(pkt)) { + /* IPv4 */ + ip = (odph_ipv4hdr_t *)odp_packet_l3(pkt); + + ip_tmp_addr = ip->src_addr; + ip->src_addr = ip->dst_addr; + ip->dst_addr = ip_tmp_addr; + } + } + } +} + +/** + * Parse and store the command line arguments + * + * @param argc argument count + * @param argv[] argument vector + * @param appl_args Store application arguments here + */ +static void parse_args(int argc, char *argv[], appl_args_t *appl_args) +{ + int opt; + int long_index; + char *names, *str, *token, *save; + size_t len; + int i; + static struct option longopts[] = { + {"count", required_argument, NULL, 'c'}, + {"interface", required_argument, NULL, 'i'}, /* return 'i' */ + {"mode", required_argument, NULL, 'm'}, /* return 'm' */ + {"help", no_argument, NULL, 'h'}, /* return 'h' */ + {NULL, 0, NULL, 0} + }; + + appl_args->mode = -1; /* Invalid, must be changed by parsing */ + + while (1) { + opt = getopt_long(argc, argv, "+c:i:m:h", + longopts, &long_index); + + if (opt == -1) + break; /* No more options */ + + switch (opt) { + case 'c': + appl_args->core_count = atoi(optarg); + break; + /* parse packet-io interface names */ + case 'i': + len = strlen(optarg); + if (len == 0) { + usage(argv[0]); + exit(EXIT_FAILURE); + } + len += 1; /* add room for '\0' */ + + names = malloc(len); + if (names == NULL) { + usage(argv[0]); + exit(EXIT_FAILURE); + } + + /* count the number of tokens separated by ',' */ + strcpy(names, optarg); + for (str = names, i = 0;; str = NULL, i++) { + token = strtok_r(str, ",", &save); + if (token == NULL) + break; + } + appl_args->if_count = i; + + if (appl_args->if_count == 0) { + usage(argv[0]); + exit(EXIT_FAILURE); + } + + /* allocate storage for the if names */ + appl_args->if_names = + calloc(appl_args->if_count, sizeof(char *)); + + /* store the if names (reset names string) */ + strcpy(names, optarg); + for (str = names, i = 0;; str = NULL, i++) { + token = strtok_r(str, ",", &save); + if (token == NULL) + break; + appl_args->if_names[i] = token; + } + break; + + case 'm': + i = atoi(optarg); + if (i == 0) + appl_args->mode = APPL_MODE_PKT_BURST; + else + appl_args->mode = APPL_MODE_PKT_QUEUE; + break; + + case 'h': + usage(argv[0]); + exit(EXIT_SUCCESS); + break; + + default: + break; + } + } + + if (appl_args->if_count == 0 || appl_args->mode == -1) { + usage(argv[0]); + exit(EXIT_FAILURE); + } + + optind = 1; /* reset 'extern optind' from the getopt lib */ +} + +/** + * Print system and application info + */ +static void print_info(char *progname, appl_args_t *appl_args) +{ + int i; + + printf("\n" + "ODP system info\n" + "---------------\n" + "ODP API version: %s\n" + "CPU model: %s\n" + "CPU freq (hz): %"PRIu64"\n" + "Cache line size: %i\n" + "Core count: %i\n" + "\n", + odp_version_api_str(), odp_sys_cpu_model_str(), odp_sys_cpu_hz(), + odp_sys_cache_line_size(), odp_sys_core_count()); + + printf("Running ODP appl: \"%s\"\n" + "-----------------\n" + "IF-count: %i\n" + "Using IFs: ", + progname, appl_args->if_count); + for (i = 0; i < appl_args->if_count; ++i) + printf(" %s", appl_args->if_names[i]); + printf("\n" + "Mode: "); + if (appl_args->mode == APPL_MODE_PKT_BURST) + PRINT_APPL_MODE(APPL_MODE_PKT_BURST); + else + PRINT_APPL_MODE(APPL_MODE_PKT_QUEUE); + printf("\n\n"); + fflush(NULL); +} + +/** + * Prinf usage information + */ +static void usage(char *progname) +{ + printf("\n" + "Usage: %s OPTIONS\n" + " E.g. %s -i eth1,eth2,eth3 -m 0\n" + "\n" + "OpenDataPlane example application.\n" + "\n" + "Mandatory OPTIONS:\n" + " -i, --interface Eth interfaces (comma-separated, no spaces)\n" + " -m, --mode 0: Burst send&receive packets (no queues)\n" + " 1: Send&receive packets through ODP queues.\n" + "\n" + "Optional OPTIONS\n" + " -c, --count Core count.\n" + " -h, --help Display help and exit.\n" + " environment variables: ODP_PKTIO_DISABLE_SOCKET_MMAP\n" + " ODP_PKTIO_DISABLE_SOCKET_MMSG\n" + " ODP_PKTIO_DISABLE_SOCKET_BASIC\n" + " can be used to advanced pkt I/O selection for linux-generic\n" + "\n", NO_PATH(progname), NO_PATH(progname) + ); +} diff --git a/helper/include/odph_ring.h b/helper/include/odph_ring.h index 76c1db8..6240646 100644 --- a/helper/include/odph_ring.h +++ b/helper/include/odph_ring.h @@ -100,6 +100,7 @@ extern "C" { #include #include #include +#include #include #include @@ -158,6 +159,8 @@ typedef struct odph_ring { #define ODPH_RING_F_SP_ENQ 0x0001 /* The default enqueue is "single-producer".*/ #define ODPH_RING_F_SC_DEQ 0x0002 /* The default dequeue is "single-consumer".*/ +#define ODPH_RING_SHM_PROC 0x0004 /* If set - ring is visible from different + processes. Default is thread visible. */ #define ODPH_RING_QUOT_EXCEED (1 << 31) /* Quota exceed for burst ops */ #define ODPH_RING_SZ_MASK (unsigned)(0x0fffffff) /* Ring size mask */ diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index d076d50..3e45f84 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -54,6 +54,7 @@ __LIB__libodp_la_SOURCES = \ odp_coremask.c \ odp_crypto.c \ odp_init.c \ + odp_ipc.c \ odp_linux.c \ odp_packet.c \ odp_packet_flags.c \ diff --git a/platform/linux-generic/include/api/odp_packet_io.h b/platform/linux-generic/include/api/odp_packet_io.h index 29fd105..6d2e39c 100644 --- a/platform/linux-generic/include/api/odp_packet_io.h +++ b/platform/linux-generic/include/api/odp_packet_io.h @@ -30,6 +30,16 @@ typedef uint32_t odp_pktio_t; #define ODP_PKTIO_INVALID 0 /** + * Lookup already existance ODP packet IO instance + * + * @param dev Packet IO device name + * + * @return ODP packet IO handle or ODP_PKTIO_INVALID on error + */ +odp_pktio_t odp_pktio_lookup(const char *dev); + + +/** * Open an ODP packet IO instance * * @param dev Packet IO device diff --git a/platform/linux-generic/include/api/odp_shared_memory.h b/platform/linux-generic/include/api/odp_shared_memory.h index 7ad29c3..46b6e18 100644 --- a/platform/linux-generic/include/api/odp_shared_memory.h +++ b/platform/linux-generic/include/api/odp_shared_memory.h @@ -31,6 +31,7 @@ extern "C" { /* Share level */ #define ODP_SHM_SW_ONLY 0x1 /**< Application SW only, no HW access */ #define ODP_SHM_PROC 0x2 /**< Share with external processes */ +#define ODP_SHM_PROC_NOCREAT 0x4 /** * ODP shared memory block @@ -98,6 +99,16 @@ int odp_shm_info(odp_shm_t shm, odp_shm_info_t *info); /** + * Look up for shared memory object. + * + * @param name name of shm object + * + * @return 0 on success, otherwise non-zero + */ + +int odp_shm_lookup_ipc(const char *name); + +/** * Print all shared memory blocks */ void odp_shm_print_all(void); diff --git a/platform/linux-generic/include/odp_buffer_pool_internal.h b/platform/linux-generic/include/odp_buffer_pool_internal.h index e0210bd..b9eb521 100644 --- a/platform/linux-generic/include/odp_buffer_pool_internal.h +++ b/platform/linux-generic/include/odp_buffer_pool_internal.h @@ -73,6 +73,12 @@ static inline void *get_pool_entry(uint32_t pool_id) return pool_entry_ptr[pool_id]; } +typedef union { + struct pool_entry_s s; + + uint8_t pad[ODP_CACHE_LINE_SIZE_ROUNDUP(sizeof(struct pool_entry_s))]; + +} pool_entry_t; static inline odp_buffer_hdr_t *odp_buf_to_hdr(odp_buffer_t buf) { @@ -107,6 +113,17 @@ static inline odp_buffer_hdr_t *odp_buf_to_hdr(odp_buffer_t buf) return hdr; } +/** + * Get buffer pool shared memory name by handler + * + * @param poll_hdl Pool handle + * + * @return Name of the pull + */ +const char *odp_buffer_pool_shm_name(odp_buffer_pool_t pool_hdl); + +uint64_t odp_buffer_pool_get_pktsize(odp_buffer_pool_t pool_hdl); +uint64_t odp_buffer_pool_get_size(odp_buffer_pool_t pool_hdl); #ifdef __cplusplus } diff --git a/platform/linux-generic/include/odp_ipc.h b/platform/linux-generic/include/odp_ipc.h new file mode 100644 index 0000000..5f055b1 --- /dev/null +++ b/platform/linux-generic/include/odp_ipc.h @@ -0,0 +1,42 @@ +/* Copyright (c) 2014, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include +#include +#include +#include +#include +#include + +#include +#include + +/* IPC packet I/O over odph_ring */ +#include + +#define PKTIO_IPC_ENTRIES 4096 /**< number of odp buffers in + odp ring queue */ + +struct pktio_info { + char remote_pool_name[30]; + /* values bellow are subject for + * pool info struct. + */ + size_t shm_pkt_pool_size; + size_t shm_pkt_size; +} __packed; + + +int lg_setup_ipc_pktio(pktio_entry_t *pktio_entry, const char *dev, + odp_buffer_pool_t pool); + +int lg_odp_pktio_lookup(const char *dev, pktio_entry_t *pktio_entry); + +int lg_ipc_pktio_recv(pktio_entry_t *pktio_entry, odp_packet_t pkt_table[], + unsigned len); + +int lg_ipc_pktio_send(pktio_entry_t *pktio_entry, odp_packet_t pkt_table[], + unsigned len); diff --git a/platform/linux-generic/include/odp_packet_io_internal.h b/platform/linux-generic/include/odp_packet_io_internal.h index 23633ed..5a7ea09 100644 --- a/platform/linux-generic/include/odp_packet_io_internal.h +++ b/platform/linux-generic/include/odp_packet_io_internal.h @@ -20,6 +20,7 @@ extern "C" { #include #include +#include /** * Packet IO types @@ -28,6 +29,7 @@ typedef enum { ODP_PKTIO_TYPE_SOCKET_BASIC = 0x1, ODP_PKTIO_TYPE_SOCKET_MMSG, ODP_PKTIO_TYPE_SOCKET_MMAP, + ODP_PKTIO_TYPE_IPC, } odp_pktio_type_t; struct pktio_entry { @@ -38,6 +40,12 @@ struct pktio_entry { odp_pktio_type_t type; /**< pktio type */ pkt_sock_t pkt_sock; /**< using socket API for IO */ pkt_sock_mmap_t pkt_sock_mmap; /**< using socket mmap API for IO */ + odph_ring_t *ipc_r; /**< ODP ring for IPC mgs packets + indexes transmitted to shared memory */ + odph_ring_t *ipc_p; /**< ODP ring for IPC msg packets + indexes already processed by remote process */ + void *ipc_pool_base; /**< IPC Remote pool base addr */ + uint64_t ipc_pkt_size; /**< IPC: packet size in remote pool */ }; typedef union { diff --git a/platform/linux-generic/odp_buffer_pool.c b/platform/linux-generic/odp_buffer_pool.c index a48d7d6..2f0d825 100644 --- a/platform/linux-generic/odp_buffer_pool.c +++ b/platform/linux-generic/odp_buffer_pool.c @@ -55,15 +55,6 @@ typedef struct { uint8_t buf_data[]; /* start of buffer data area */ } odp_any_buffer_hdr_t; - -typedef union pool_entry_u { - struct pool_entry_s s; - - uint8_t pad[ODP_CACHE_LINE_SIZE_ROUNDUP(sizeof(struct pool_entry_s))]; - -} pool_entry_t; - - typedef struct pool_table_t { pool_entry_t pool[ODP_CONFIG_BUFFER_POOLS]; @@ -572,3 +563,35 @@ void odp_buffer_pool_print(odp_buffer_pool_t pool_hdl) printf("\n"); } + +const char *odp_buffer_pool_shm_name(odp_buffer_pool_t pool_hdl ODP_UNUSED) +{ + /* @todo + * Pending on Petris patch to provide shm handler to pool create, + * so that I can get shared memory block name instead of pool name. + * For now return the same name what example/ipc/ expects. + */ + return "shm_packet_pool"; +} + +uint64_t odp_buffer_pool_get_pktsize(odp_buffer_pool_t pool_hdl) +{ + pool_entry_t *pool; + uint32_t pool_id; + + pool_id = pool_handle_to_index(pool_hdl); + pool = get_pool_entry(pool_id); + + return pool->s.buf_size; +} + +uint64_t odp_buffer_pool_get_size(odp_buffer_pool_t pool_hdl) +{ + pool_entry_t *pool; + uint32_t pool_id; + + pool_id = pool_handle_to_index(pool_hdl); + pool = get_pool_entry(pool_id); + + return pool->s.pool_size; +} diff --git a/platform/linux-generic/odp_init.c b/platform/linux-generic/odp_init.c index 55fa53a..fa41ce3 100644 --- a/platform/linux-generic/odp_init.c +++ b/platform/linux-generic/odp_init.c @@ -8,6 +8,7 @@ #include #include +#include int odp_init_global(void) { @@ -53,6 +54,11 @@ int odp_init_global(void) return -1; } + /* for linux-generic IPC queue implemented totaly in + * software using odp_ring. + */ + odph_ring_tailq_init(); + return 0; } diff --git a/platform/linux-generic/odp_ipc.c b/platform/linux-generic/odp_ipc.c new file mode 100644 index 0000000..4106dda --- /dev/null +++ b/platform/linux-generic/odp_ipc.c @@ -0,0 +1,319 @@ +/* Copyright (c) 2014, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include + +static struct pktio_info *lng_map_shm_pool_info(const char *dev, int flag) +{ + /* Create info about remote pktio */ + char *name = (char *)malloc(strlen(dev) + sizeof("_info")); + memcpy(name, dev, strlen(dev)); + memcpy(name + strlen(dev), "_info", sizeof("_info")); + odp_shm_t shm = odp_shm_reserve(name, sizeof(struct pktio_info), + ODP_CACHE_LINE_SIZE, + flag); + free(name); + struct pktio_info *pinfo = odp_shm_addr(shm); + if (flag != ODP_SHM_PROC_NOCREAT) + memset(pinfo->remote_pool_name, 0, 30); + return pinfo; +} + +int lg_setup_ipc_pktio(pktio_entry_t *pktio_entry, const char *dev, + odp_buffer_pool_t pool) +{ + char ipc_shm_name[ODPH_RING_NAMESIZE]; + + /* generate name in shm like ipc_pktio_r for + * to be processed packets ring. + */ + memset(ipc_shm_name, 0, ODPH_RING_NAMESIZE); + memcpy(ipc_shm_name, dev, strlen(dev)); + memcpy(ipc_shm_name + strlen(dev), "_r", 2); + + pktio_entry->s.ipc_r = odph_ring_create(ipc_shm_name, + PKTIO_IPC_ENTRIES, + ODPH_RING_SHM_PROC); + if (!pktio_entry->s.ipc_r) { + ODP_DBG("pid %d unable to create ipc ring %s name\n", + getpid(), ipc_shm_name); + return -1; + } + ODP_DBG("Created IPC ring: %s\n", ipc_shm_name); + + /* generate name in shm like ipc_pktio_p for + * already processed packets + */ + memcpy(ipc_shm_name + strlen(dev), "_p", 2); + + pktio_entry->s.ipc_p = odph_ring_create(ipc_shm_name, + PKTIO_IPC_ENTRIES, + ODPH_RING_SHM_PROC); + if (!pktio_entry->s.ipc_p) { + ODP_DBG("pid %d unable to create ipc ring %s name\n", + getpid(), ipc_shm_name); + return -1; + } + + ODP_DBG("Created IPC ring: %s\n", ipc_shm_name); + + struct pktio_info *pinfo = lng_map_shm_pool_info(dev, ODP_SHM_PROC); + + /* Set up pool name for remote info */ + const char *pool_name = odp_buffer_pool_shm_name(pool); + memcpy(pinfo->remote_pool_name, pool_name, strlen(pool_name)); + pinfo->shm_pkt_pool_size = odp_buffer_pool_get_size(pool); + pinfo->shm_pkt_size = odp_buffer_pool_get_pktsize(pool); + + ODP_DBG("Created IPC ring: %s\n", ipc_shm_name); + return 0; +} + +static int lg_odp_shm_lookup_pktio(const char *name) +{ + /* In linux generic pktio is impmeneted with ring with _r _p prefixes + * for delivered and produced packets. So just call them here. + */ + char ipc_shm_name[ODPH_RING_NAMESIZE]; + int ret; + + memset(ipc_shm_name, 0, ODPH_RING_NAMESIZE); + memcpy(ipc_shm_name, name, strlen(name)); + memcpy(ipc_shm_name + strlen(name), "_r", 2); + + ret = odp_shm_lookup_ipc(ipc_shm_name); + if (ret != 0) + return ret; + + memcpy(ipc_shm_name + strlen(name), "_p", 2); + ret = odp_shm_lookup_ipc(ipc_shm_name); + + return ret; +} + +static odp_buffer_pool_t lg_odp_alloc_and_create_pool(uint64_t pool_size, + uint64_t buf_size) +{ + odp_shm_t shm; + odp_buffer_pool_t pool; + void *pool_base; + char rnd_name[15]; + + memcpy(rnd_name, "odp_shm_pool_", 13); + rnd_name[14] = 0; + shm = odp_shm_reserve(rnd_name, + pool_size, ODP_CACHE_LINE_SIZE, 0); + + pool_base = odp_shm_addr(shm); + if (pool_base == NULL) { + ODP_ERR("Error: packet pool mem alloc failed.\n"); + exit(EXIT_FAILURE); + } + + memcpy(rnd_name, "odp_pool_", 9); + pool = odp_buffer_pool_create(rnd_name, pool_base, + pool_size, + buf_size, + ODP_CACHE_LINE_SIZE, + ODP_BUFFER_TYPE_PACKET); + if (pool == ODP_BUFFER_POOL_INVALID) { + ODP_ERR("Error: packet pool create failed.\n"); + exit(EXIT_FAILURE); + } + + return pool; +} + +static void *map_remote_pool(struct pktio_info *pinfo) +{ + void *ext_pool_base; + odp_shm_t shm; + const char *name = pinfo->remote_pool_name; + + /* Try to find and map remote pool. */ + shm = odp_shm_reserve(name, + pinfo->shm_pkt_pool_size, + ODP_CACHE_LINE_SIZE, + ODP_SHM_PROC_NOCREAT); + ext_pool_base = odp_shm_addr(shm); + return ext_pool_base; +} + +int lg_odp_pktio_lookup(const char *dev, pktio_entry_t *pktio_entry) +{ + int ret = -1; + char ipc_shm_name[ODPH_RING_NAMESIZE]; + size_t ring_size; + + if (!(lg_odp_shm_lookup_pktio(dev) == 0 && + odp_shm_lookup_ipc("shm_packet_pool") == 0)) { + ODP_DBG("pid %d unable to find ipc object: %s.\n", + getpid(), dev); + goto error; + } + + ODP_DBG("pid %d odp_shm_lookup_ipc found shared object\n", + getpid()); + ring_size = PKTIO_IPC_ENTRIES * sizeof(void *) + + sizeof(odph_ring_t); + + memset(ipc_shm_name, 0, ODPH_RING_NAMESIZE); + memcpy(ipc_shm_name, dev, strlen(dev)); + memcpy(ipc_shm_name + strlen(dev), "_r", 2); + + /* allocate shared memory for buffers needed to be produced */ + odp_shm_t shm = odp_shm_reserve(ipc_shm_name, ring_size, + ODP_CACHE_LINE_SIZE, + ODP_SHM_PROC_NOCREAT); + + pktio_entry->s.ipc_r = odp_shm_addr(shm); + if (!pktio_entry->s.ipc_r) { + ODP_DBG("pid %d unable to find ipc ring %s name\n", + getpid(), dev); + goto error; + } + + memcpy(ipc_shm_name + strlen(dev), "_p", 2); + /* allocate shared memory for produced ring buffer handlers. That + * buffers will be cleaned up after they are produced by other process. + */ + shm = odp_shm_reserve(ipc_shm_name, ring_size, + ODP_CACHE_LINE_SIZE, + ODP_SHM_PROC_NOCREAT); + + pktio_entry->s.ipc_p = odp_shm_addr(shm); + if (!pktio_entry->s.ipc_p) { + ODP_DBG("pid %d unable to find ipc ring %s name\n", + getpid(), dev); + goto error; + } + + pktio_entry->s.type = ODP_PKTIO_TYPE_IPC; + + /* Get info about remote pool */ + struct pktio_info *pinfo = lng_map_shm_pool_info(dev, + ODP_SHM_PROC_NOCREAT); + pktio_entry->s.ipc_pool_base = map_remote_pool(pinfo); + pktio_entry->s.ipc_pkt_size = pinfo->shm_pkt_size; + + /* @todo: to simplify in linux-generic implementation we create pool for + * packets from IPC queue. On receive implementation copies packets to + * that pool. Later we can try to reuse original tool without packets + * copying. + */ + pktio_entry->s.pkt_sock.pool = lg_odp_alloc_and_create_pool( + pinfo->shm_pkt_pool_size, pinfo->shm_pkt_size); + + ret = 0; + ODP_DBG("%s OK.\n", __func__); +error: + /* @todo free shm on error (api not impemented yet) */ + return ret; +} + +int lg_ipc_pktio_recv(pktio_entry_t *pktio_entry, + odp_packet_t pkt_table[], unsigned len) +{ + int pkts = 0; + int ret; + int i; + odph_ring_t *r = pktio_entry->s.ipc_r; + odph_ring_t *r_p = pktio_entry->s.ipc_p; + odp_packet_t remote_pkts[PKTIO_IPC_ENTRIES]; + void **ipcbufs_p = (void *)&remote_pkts; + unsigned ring_len = odph_ring_count(r); + int idx; + + pkts = len; + if (len > ring_len) + pkts = ring_len; + + ret = odph_ring_mc_dequeue_bulk(r, ipcbufs_p, pkts); + if (ret != 0) { + ODP_DBG("dequeue no packets\n"); + pkts = -1; + return pkts; + } + + for (i = 0; i < pkts; i++) { + /* Remote packet has coded pool and index. We need only index.*/ + odp_buffer_bits_t handle; + handle.u32 = remote_pkts[i]; + idx = handle.index; + + /* Link to packed data. To this line we have Zero-Copy between + * processes, to simplify use packet copy in that version which + * can be removed later with more advance buffer management + * (ref counters). + */ + odp_packet_hdr_t *phdr; + phdr = (odp_packet_hdr_t *)((char *)pktio_entry->s.ipc_pool_base + + (idx * pktio_entry->s.ipc_pkt_size)); + + /* Allocate new packet.*/ + odp_buffer_pool_t pool = pktio_entry->s.pkt_sock.pool; + odp_packet_t pkt = odp_buffer_alloc(pool); + if (odp_unlikely(pkt == ODP_PACKET_INVALID)) + ODP_ABORT("unable to allocate memory for pool"); + + /* Copy packet data. */ + uint8_t *pkt_buf = odp_packet_addr(pkt); + uint8_t *l2_hdr = pkt_buf + + pktio_entry->s.pkt_sock.frame_offset; + memcpy(l2_hdr, phdr->buf_data, phdr->frame_len); + + /* Copy packets L2, L3 parsed offsets and size */ + odp_packet_hdr(pkt)->l2_offset = phdr->l2_offset; + odp_packet_hdr(pkt)->l3_offset = phdr->l3_offset; + odp_packet_hdr(pkt)->l4_offset = phdr->l4_offset; + odp_packet_hdr(pkt)->frame_len = phdr->frame_len; + odp_packet_hdr(pkt)->user_ctx = phdr->user_ctx; + + pkt_table[i] = pkt; + } + + /* Now tell other process that we no longer need that buffers.*/ + ret = odph_ring_mp_enqueue_bulk(r_p, ipcbufs_p, pkts); + if (ret != 0) + ODP_ABORT("ipc: odp_ring_mp_enqueue_bulk r_p fail\n"); + + return pkts; +} + +int lg_ipc_pktio_send(pktio_entry_t *pktio_entry, odp_packet_t pkt_table[], + unsigned len) +{ + odph_ring_t *r = pktio_entry->s.ipc_r; + void **rbuf_p; + int ret; + unsigned i; + + /* Free already processed packets, if any */ + { + odph_ring_t *r_p = pktio_entry->s.ipc_p; + unsigned complete_packets = odph_ring_count(r_p); + odp_packet_t r_p_pkts[PKTIO_IPC_ENTRIES]; + if (complete_packets > 0) { + rbuf_p = (void *)&r_p_pkts; + ret = odph_ring_mc_dequeue_bulk(r_p, rbuf_p, + complete_packets); + if (ret == 0) { + for (i = 0; i < complete_packets; i++) + odp_buffer_free(r_p_pkts[i]); + } + } + } + + /* Put packets to ring to be processed in other process. */ + for (i = 0; i < len; i++) { + odp_packet_t pkt = pkt_table[i]; + rbuf_p = (void *)&pkt; + ret = odph_ring_mp_enqueue_bulk(r, rbuf_p, 1); + if (ret != 0) + ODP_ERR("odp_ring_mp_enqueue_bulk fail\n"); + } + return len; +} diff --git a/platform/linux-generic/odp_packet_io.c b/platform/linux-generic/odp_packet_io.c index 0c30f0f..24d061b 100644 --- a/platform/linux-generic/odp_packet_io.c +++ b/platform/linux-generic/odp_packet_io.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -20,6 +21,14 @@ #include #include +#include +#include + +/* IPC packet I/O over odph_ring */ +#include + +#define PKTIO_IPC_ENTRIES 4096 /**< number of odp buffers in + odp ring queue */ typedef struct { pktio_entry_t entries[ODP_CONFIG_PKTIO_ENTRIES]; @@ -149,6 +158,31 @@ static int free_pktio_entry(odp_pktio_t id) return 0; } +odp_pktio_t odp_pktio_lookup(const char *dev) +{ + odp_pktio_t id; + pktio_entry_t *pktio_entry; + int ret; + + id = alloc_lock_pktio_entry(); + if (id == ODP_PKTIO_INVALID) { + ODP_ERR("No resources available.\n"); + return ODP_PKTIO_INVALID; + } + /* if successful, alloc_pktio_entry() returns with the entry locked */ + + pktio_entry = get_entry(id); + + ret = lg_odp_pktio_lookup(dev, pktio_entry); + if (ret != 0) + id = ODP_PKTIO_INVALID; + unlock_entry(pktio_entry); + + if (id == ODP_PKTIO_INVALID) + free_pktio_entry(id); + return id; +} + odp_pktio_t odp_pktio_open(const char *dev, odp_buffer_pool_t pool) { odp_pktio_t id; @@ -165,6 +199,18 @@ odp_pktio_t odp_pktio_open(const char *dev, odp_buffer_pool_t pool) pktio_entry = get_entry(id); + /* if name begins with ipc, then we assume that queue is IPC, I.e. + * it's software packet I/O communicating to different process. + */ + if (!memcmp(dev, "ipc", 3)) { + pktio_entry->s.type = ODP_PKTIO_TYPE_IPC; + res = lg_setup_ipc_pktio(pktio_entry, dev, pool); + if (res != -1) { + ODP_DBG("IO type: ODP_PKTIO_TYPE_IPC\n"); + goto done; + } + } + ODP_DBG("ODP_PKTIO_USE_FANOUT: %d\n", fanout); if (getenv("ODP_PKTIO_DISABLE_SOCKET_MMAP") == NULL) { pktio_entry->s.type = ODP_PKTIO_TYPE_SOCKET_MMAP; @@ -272,6 +318,9 @@ int odp_pktio_recv(odp_pktio_t id, odp_packet_t pkt_table[], unsigned len) pkts = recv_pkt_sock_mmap(&pktio_entry->s.pkt_sock_mmap, pkt_table, len); break; + case ODP_PKTIO_TYPE_IPC: + pkts = lg_ipc_pktio_recv(pktio_entry, pkt_table, len); + break; default: pkts = -1; break; @@ -309,6 +358,9 @@ int odp_pktio_send(odp_pktio_t id, odp_packet_t pkt_table[], unsigned len) pkts = send_pkt_sock_mmap(&pktio_entry->s.pkt_sock_mmap, pkt_table, len); break; + case ODP_PKTIO_TYPE_IPC: + pkts = lg_ipc_pktio_send(pktio_entry, pkt_table, len); + break; default: pkts = -1; } diff --git a/platform/linux-generic/odp_ring.c b/platform/linux-generic/odp_ring.c index 632aa66..7f6eaad 100644 --- a/platform/linux-generic/odp_ring.c +++ b/platform/linux-generic/odp_ring.c @@ -158,8 +158,14 @@ odph_ring_create(const char *name, unsigned count, unsigned flags) char ring_name[ODPH_RING_NAMESIZE]; odph_ring_t *r; size_t ring_size; + uint32_t shm_flag; odp_shm_t shm; + if (flags & ODPH_RING_SHM_PROC) + shm_flag = ODP_SHM_PROC; + else + shm_flag = 0; + /* count must be a power of 2 */ if (!ODP_VAL_IS_POWER_2(count) || (count > ODPH_RING_SZ_MASK)) { ODP_ERR("Requested size is invalid, must be power of 2, and do not exceed the size limit %u\n", @@ -172,7 +178,8 @@ odph_ring_create(const char *name, unsigned count, unsigned flags) odp_rwlock_write_lock(&qlock); /* reserve a memory zone for this ring.*/ - shm = odp_shm_reserve(ring_name, ring_size, ODP_CACHE_LINE_SIZE, 0); + shm = odp_shm_reserve(ring_name, ring_size, ODP_CACHE_LINE_SIZE, + shm_flag); r = odp_shm_addr(shm); diff --git a/platform/linux-generic/odp_shared_memory.c b/platform/linux-generic/odp_shared_memory.c index 1898a34..b3635ab 100644 --- a/platform/linux-generic/odp_shared_memory.c +++ b/platform/linux-generic/odp_shared_memory.c @@ -20,6 +20,7 @@ #include +#include #define ODP_SHM_NUM_BLOCKS 32 @@ -112,7 +113,6 @@ static int find_block(const char *name, uint32_t *index) return 0; } - odp_shm_t odp_shm_reserve(const char *name, uint64_t size, uint64_t align, uint32_t flags) { @@ -122,20 +122,24 @@ odp_shm_t odp_shm_reserve(const char *name, uint64_t size, uint64_t align, int fd = -1; int map_flag = MAP_SHARED; /* If already exists: O_EXCL: error, O_TRUNC: truncate to zero */ - int oflag = O_RDWR | O_CREAT | O_TRUNC; + int oflag = O_RDWR; uint64_t alloc_size = size + align; uint64_t page_sz, huge_sz; huge_sz = odp_sys_huge_page_size(); page_sz = odp_sys_page_size(); - if (flags & ODP_SHM_PROC) { + if (flags & ODP_SHM_PROC) + oflag |= O_CREAT | O_TRUNC; + + if (flags & (ODP_SHM_PROC | ODP_SHM_PROC_NOCREAT)) { /* Creates a file to /dev/shm */ fd = shm_open(name, oflag, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH); if (fd == -1) { - ODP_DBG("odp_shm_reserve: shm_open failed\n"); + ODP_DBG("odp_shm_reserve: shm_open: name:\"%s\" flags: %d failed\n", + name, oflag); return ODP_SHM_INVALID; } @@ -153,7 +157,7 @@ odp_shm_t odp_shm_reserve(const char *name, uint64_t size, uint64_t align, if (find_block(name, NULL)) { /* Found a block with the same name */ odp_spinlock_unlock(&odp_shm_tbl->lock); - ODP_DBG("odp_shm_reserve: name already used\n"); + ODP_DBG("odp_shm_reserve: name: \"%s\" already used\n", name); return ODP_SHM_INVALID; } @@ -273,6 +277,18 @@ int odp_shm_info(odp_shm_t shm, odp_shm_info_t *info) return 0; } +int odp_shm_lookup_ipc(const char *name) +{ + int shm; + + shm = shm_open(name, O_RDWR, S_IRUSR | S_IWUSR); + if (shm == -1) { + ODP_DBG("IPC shm_open for %s not found\n", name); + return -1; + } + close(shm); + return 0; +} void odp_shm_print_all(void) {