From patchwork Thu Feb 19 11:26:15 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Venkatesh Vivekanandan X-Patchwork-Id: 44810 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-ee0-f70.google.com (mail-ee0-f70.google.com [74.125.83.70]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 3924621554 for ; Thu, 19 Feb 2015 11:30:43 +0000 (UTC) Received: by mail-ee0-f70.google.com with SMTP id c41sf5100212eek.1 for ; Thu, 19 Feb 2015 03:30:42 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:date:message-id :mime-version:subject:precedence:list-id:list-unsubscribe :list-archive:list-post:list-help:list-subscribe:content-type :content-transfer-encoding:errors-to:sender:x-original-sender :x-original-authentication-results:mailing-list; bh=yw52Z7meerTS6y0xqr0DOQ3IgWxZ6jwv3Up3dM9b3Wc=; b=lYM5lg69ChCWHMmj2SOJTC011l8Jvjdz9YjX7CZo8CFvqB5Aeb8VbzkY9HEjRispOu BkyVH9+3ShAPIO9Lp4vUDwubB3nk7BdcpoBaCL4HNME8J8+AowXj6jOTUW+IwZRUmzIJ y2MNxSNumjqn4hbcJwfwIFW7mC4BPmOV3krpVpczfn3IDlzc0jQ1EqcB8IqLu0+Ojuvu l/HoUPq2RMMUWpnlNiStUbwiogQpdFOOU7n193kQ3RxciOVxJyajBQNAntinmmx3ZM8F CEqHXW+4HStqvnZSP6xLy8GDcY0hAljSiut9S6D+zSGU9f1fmUXCPvIQFDOsCXvQewEz y2Vw== X-Gm-Message-State: ALoCoQnATUTShc/dMSG3aBlb9EnuT/ftw73HfV37UZwD2I2Vp0CdR5UnyurVq9RquFwWQTc4PEnK X-Received: by 10.112.124.142 with SMTP id mi14mr503038lbb.1.1424345442405; Thu, 19 Feb 2015 03:30:42 -0800 (PST) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.21.130 with SMTP id v2ls112156lae.98.gmail; Thu, 19 Feb 2015 03:30:42 -0800 (PST) X-Received: by 10.152.178.197 with SMTP id da5mr3383381lac.87.1424345442027; Thu, 19 Feb 2015 03:30:42 -0800 (PST) Received: from mail-lb0-f179.google.com (mail-lb0-f179.google.com. [209.85.217.179]) by mx.google.com with ESMTPS id b6si9845771lbc.116.2015.02.19.03.30.41 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 19 Feb 2015 03:30:41 -0800 (PST) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.179 as permitted sender) client-ip=209.85.217.179; Received: by lbiw7 with SMTP id w7so6831869lbi.9 for ; Thu, 19 Feb 2015 03:30:41 -0800 (PST) X-Received: by 10.112.211.200 with SMTP id ne8mr793564lbc.73.1424345441865; Thu, 19 Feb 2015 03:30:41 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.35.133 with SMTP id h5csp429968lbj; Thu, 19 Feb 2015 03:30:40 -0800 (PST) X-Received: by 10.140.151.145 with SMTP id 139mr11087042qhx.72.1424345439736; Thu, 19 Feb 2015 03:30:39 -0800 (PST) Received: from ip-10-35-177-41.ec2.internal (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTPS id p16si24493687qax.38.2015.02.19.03.30.38 (version=TLSv1 cipher=RC4-SHA bits=128/128); Thu, 19 Feb 2015 03:30:39 -0800 (PST) Received-SPF: none (google.com: lng-odp-bounces@lists.linaro.org does not designate permitted sender hosts) client-ip=54.225.227.206; Received: from localhost ([127.0.0.1] helo=ip-10-35-177-41.ec2.internal) by ip-10-35-177-41.ec2.internal with esmtp (Exim 4.76) (envelope-from ) id 1YOPJT-0005Rw-3k; Thu, 19 Feb 2015 11:30:35 +0000 Received: from mail-gw1-out.broadcom.com ([216.31.210.62]) by ip-10-35-177-41.ec2.internal with esmtp (Exim 4.76) (envelope-from ) id 1YOPJO-0005Rq-1c for lng-odp@lists.linaro.org; Thu, 19 Feb 2015 11:30:30 +0000 X-IronPort-AV: E=Sophos;i="5.09,512,1418112000"; d="scan'208";a="57648881" Received: from irvexchcas08.broadcom.com (HELO IRVEXCHCAS08.corp.ad.broadcom.com) ([10.9.208.57]) by mail-gw1-out.broadcom.com with ESMTP; 19 Feb 2015 05:56:57 -0800 Received: from IRVEXCHSMTP1.corp.ad.broadcom.com (10.9.207.51) by IRVEXCHCAS08.corp.ad.broadcom.com (10.9.208.57) with Microsoft SMTP Server (TLS) id 14.3.174.1; Thu, 19 Feb 2015 03:30:24 -0800 Received: from mail-irva-13.broadcom.com (10.10.10.20) by IRVEXCHSMTP1.corp.ad.broadcom.com (10.9.207.51) with Microsoft SMTP Server id 14.3.174.1; Thu, 19 Feb 2015 03:30:31 -0800 Received: from localhost.localdomain (unknown [10.131.60.56]) by mail-irva-13.broadcom.com (Postfix) with ESMTP id 216C041048; Thu, 19 Feb 2015 03:29:12 -0800 (PST) From: To: Date: Thu, 19 Feb 2015 16:56:15 +0530 Message-ID: <1424345175-8763-1-git-send-email-venkatesh.vivekanandan@linaro.org> X-Mailer: git-send-email 1.9.1 MIME-Version: 1.0 X-Topics: patch Subject: [lng-odp] [PATCHv4 DPDK 1/6] Generic changes to sync with ODP v0.5 X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: lng-odp-bounces@lists.linaro.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: venkatesh.vivekanandan@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.179 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 From: Venkatesh Vivekanandan - Removed odp_schedule.c and odp_pktio_types.h - Made necessary modifications in Makefile - Modified odp_debug.h to sync with linux-generic - Added necessary headers to odp_queue.c - Added parameters to odp_init_global - Added odp_thread_init_global/local and odp_classification_init_global - Modified pthread_create to follow linux-generic implementation Signed-off-by: Venkatesh Vivekanandan --- -> This patch must be applied on odp v0.5.0 ontop of odp-dpdk -> odp_l2fwd, odp_pktio were tested both in burst and queue mode which worked fine. -> ipsec app was tested and working fine. For this to work, USE_MAC_ADDR_HACK has to be set to 0 for now. Will send out a patch for MAC address fix once this patch gets in. Checkpatch warning for camel case could not be avoided. WARNING: Avoid CamelCase: <_Static_assert> #373: FILE: platform/linux-dpdk/include/api/odp_debug.h:34: +#define _Static_assert(e, s) extern int (*static_assert_checker(void)) \ WARNING: storage class should be at the beginning of the declaration #373: FILE: platform/linux-dpdk/include/api/odp_debug.h:34: +#define _Static_assert(e, s) extern int (*static_assert_checker(void)) \ v2: Fixed odp_flush_caches compilation error v3: Removed debug code that slipped in v4: Split a big patch into 6 different logical chunks platform/linux-dpdk/Makefile.am | 13 +- platform/linux-dpdk/include/api/odp_debug.h | 77 ++-- platform/linux-dpdk/include/api/odp_pktio_types.h | 45 --- platform/linux-dpdk/odp_init.c | 35 +- platform/linux-dpdk/odp_linux.c | 117 +++++- platform/linux-dpdk/odp_queue.c | 3 +- platform/linux-dpdk/odp_schedule.c | 421 ---------------------- 7 files changed, 189 insertions(+), 522 deletions(-) delete mode 100644 platform/linux-dpdk/include/api/odp_pktio_types.h delete mode 100644 platform/linux-dpdk/odp_schedule.c diff --git a/platform/linux-dpdk/Makefile.am b/platform/linux-dpdk/Makefile.am index db11e0a..6491d3a 100644 --- a/platform/linux-dpdk/Makefile.am +++ b/platform/linux-dpdk/Makefile.am @@ -29,6 +29,7 @@ include_HEADERS = \ $(top_srcdir)/platform/linux-generic/include/api/odp_barrier.h \ $(srcdir)/include/api/odp_buffer_pool.h \ $(top_srcdir)/platform/linux-generic/include/api/odp_byteorder.h \ + $(top_srcdir)/platform/linux-generic/include/api/odp_classification.h \ $(top_srcdir)/platform/linux-generic/include/api/odp_compiler.h \ $(top_srcdir)/platform/linux-generic/include/api/odp_config.h \ $(top_srcdir)/platform/linux-generic/include/api/odp_coremask.h \ @@ -52,15 +53,16 @@ include_HEADERS = \ $(top_srcdir)/platform/linux-generic/include/api/odp_time.h \ $(top_srcdir)/platform/linux-generic/include/api/odp_timer.h \ $(top_srcdir)/platform/linux-generic/include/api/odp_version.h \ - $(srcdir)/include/api/odp_pktio_types.h + $(srcdir)/include/api/odp_platform_types.h subdirheadersdir = $(includedir)/helper subdirheaders_HEADERS = \ $(top_srcdir)/helper/include/odph_chksum.h \ $(top_srcdir)/helper/include/odph_eth.h \ + $(top_srcdir)/helper/include/odph_icmp.h \ $(top_srcdir)/helper/include/odph_ip.h \ + $(top_srcdir)/helper/include/odph_ipsec.h \ $(top_srcdir)/helper/include/odph_linux.h \ - $(top_srcdir)/helper/include/odph_packet.h \ $(top_srcdir)/helper/include/odph_ring.h \ $(top_srcdir)/helper/include/odph_udp.h @@ -68,9 +70,11 @@ __LIB__libodp_la_SOURCES = \ ../linux-generic/odp_barrier.c \ odp_buffer.c \ odp_buffer_pool.c \ + ../linux-generic/odp_classification.c \ ../linux-generic/odp_coremask.c \ ../linux-generic/odp_crypto.c \ odp_init.c \ + ../linux-generic/odp_impl.c \ odp_linux.c \ odp_packet.c \ odp_packet_dpdk.c \ @@ -80,11 +84,12 @@ __LIB__libodp_la_SOURCES = \ odp_queue.c \ ../linux-generic/odp_ring.c \ ../linux-generic/odp_rwlock.c \ - odp_schedule.c \ + ../linux-generic/odp_schedule.c \ ../linux-generic/odp_shared_memory.c \ ../linux-generic/odp_spinlock.c \ ../linux-generic/odp_system_info.c \ ../linux-generic/odp_thread.c \ ../linux-generic/odp_ticketlock.c \ ../linux-generic/odp_time.c \ - ../linux-generic/odp_timer.c + ../linux-generic/odp_timer.c \ + ../linux-generic/odp_weak.c diff --git a/platform/linux-dpdk/include/api/odp_debug.h b/platform/linux-dpdk/include/api/odp_debug.h index 6b145b5..aed8a5f 100644 --- a/platform/linux-dpdk/include/api/odp_debug.h +++ b/platform/linux-dpdk/include/api/odp_debug.h @@ -12,73 +12,74 @@ #ifndef ODP_DEBUG_H_ #define ODP_DEBUG_H_ -#include -#include #ifdef __cplusplus extern "C" { #endif +/** @addtogroup odp_ver_abt_log_dbg + * Macros that allows different messages. + * @{ + */ + #ifdef __GNUC__ -/** - * Indicate deprecated variables, functions or types - */ -#define ODP_DEPRECATED __attribute__((__deprecated__)) + +#if __GNUC__ < 4 || (__GNUC__ == 4 && (__GNUC_MINOR__ < 6)) /** - * Intentionally unused variables ot functions + * _Static_assert was only added in GCC 4.6. Provide a weak replacement + * for previous versions. */ -#define ODP_UNUSED __attribute__((__unused__)) +#define _Static_assert(e, s) extern int (*static_assert_checker(void)) \ + [sizeof(struct { unsigned int error_if_negative:(e) ? 1 : -1; })] -#else +#endif -#define ODP_DEPRECATED -#define ODP_UNUSED -#endif -/** - * Runtime assertion-macro - aborts if 'cond' is false. - */ -#ifndef ODP_NO_DEBUG -#define ODP_ASSERT(cond, msg) \ - do { if (!(cond)) {ODP_ERR("%s\n", msg); abort(); } } while (0) -#else -#define ODP_ASSERT(cond, msg) #endif + /** * Compile time assertion-macro - fail compilation if cond is false. * @note This macro has zero runtime overhead */ -#define ODP_STATIC_ASSERT(cond, msg) _Static_assert(1, msg) +#define _ODP_STATIC_ASSERT(cond, msg) _Static_assert(1, msg) /** - * Debug printing macro, which prints output when DEBUG flag is set. + * ODP log level. */ -#define ODP_DBG(fmt, ...) \ - do { if (ODP_DEBUG_PRINT == 1) \ - printf(fmt, ##__VA_ARGS__); \ - } while (0) +typedef enum odp_log_level { + ODP_LOG_DBG, + ODP_LOG_ERR, + ODP_LOG_UNIMPLEMENTED, + ODP_LOG_ABORT, + ODP_LOG_PRINT +} odp_log_level_e; /** - * Print output to stderr (file, line and function). + * ODP log function + * + * Instead of direct prints to stdout/stderr all logging in ODP implementation + * should be done via this function or its wrappers. + * ODP platform MUST provide a default *weak* implementation of this function. + * Application MAY override the function if needed by providing a strong + * function. + * + * @param[in] level Log level + * @param[in] fmt printf-style message format + * + * @return The number of characters logged if succeeded. Otherwise returns + * a negative number. */ -#define ODP_ERR(fmt, ...) \ -do { fprintf(stderr, "%s:%d:%s(): " fmt, __FILE__, \ - __LINE__, __func__, ##__VA_ARGS__); \ -} while (0) +extern int odp_override_log(odp_log_level_e level, const char *fmt, ...); + + /** - * Print output to stderr (file, line and function), - * then abort. + * @} */ -#define ODP_ABORT(fmt, ...) \ -do { fprintf(stderr, "%s:%d:%s(): " fmt, __FILE__, \ - __LINE__, __func__, ##__VA_ARGS__); \ - abort(); \ -} while (0) #ifdef __cplusplus } diff --git a/platform/linux-dpdk/include/api/odp_pktio_types.h b/platform/linux-dpdk/include/api/odp_pktio_types.h deleted file mode 100644 index b23e6da..0000000 --- a/platform/linux-dpdk/include/api/odp_pktio_types.h +++ /dev/null @@ -1,45 +0,0 @@ - -/* Copyright (c) 2013, Linaro Limited - * All rights reserved. - * - * SPDX-License-Identifier: BSD-3-Clause - */ - -#ifndef ODP_PKTIO_TYPES_H -#define ODP_PKTIO_TYPES_H - -#ifdef __cplusplus -extern "C" { -#endif - -/* We should ensure that future enum values will never overlap, otherwise - * applications that want netmap suport might get in trouble if the odp lib - * was not built with netmap support and there are more types define below - */ - -typedef enum { - ODP_PKTIO_TYPE_SOCKET_BASIC = 0x1, - ODP_PKTIO_TYPE_SOCKET_MMSG, - ODP_PKTIO_TYPE_SOCKET_MMAP, - ODP_PKTIO_TYPE_NETMAP, - ODP_PKTIO_TYPE_DPDK, -} odp_pktio_type_t; - -#include -#ifdef ODP_HAVE_NETMAP -#include -#endif - -typedef union odp_pktio_params_t { - odp_pktio_type_t type; - socket_params_t sock_params; -#ifdef ODP_HAVE_NETMAP - netmap_params_t nm_params; -#endif -} odp_pktio_params_t; - -#ifdef __cplusplus -} -#endif - -#endif diff --git a/platform/linux-dpdk/odp_init.c b/platform/linux-dpdk/odp_init.c index fa10022..7c8763d 100644 --- a/platform/linux-dpdk/odp_init.c +++ b/platform/linux-dpdk/odp_init.c @@ -7,6 +7,8 @@ #include #include #include +#include +#include #include int odp_init_dpdk(void) @@ -45,10 +47,9 @@ int odp_init_dpdk(void) return 0; } -int odp_init_global(void) +int odp_init_global(odp_init_t *params ODP_UNUSED, + odp_platform_init_t *platform_params ODP_UNUSED) { - odp_thread_init_global(); - odp_system_info_init(); if (odp_init_dpdk()) { @@ -61,6 +62,11 @@ int odp_init_global(void) return -1; } + if (odp_thread_init_global()) { + ODP_ERR("ODP thread init failed.\n"); + return -1; + } + if (odp_buffer_pool_init_global()) { ODP_ERR("ODP buffer pool init failed.\n"); return -1; @@ -91,13 +97,26 @@ int odp_init_global(void) return -1; } + if (odp_classification_init_global()) { + ODP_ERR("ODP classification init failed.\n"); + return -1; + } + return 0; } +int odp_term_global(void) +{ + ODP_UNIMPLEMENTED(); + return 0; +} -int odp_init_local(int thr_id) +int odp_init_local(void) { - odp_thread_init_local(thr_id); + if (odp_thread_init_local()) { + ODP_ERR("ODP thread local init failed.\n"); + return -1; + } if (odp_pktio_init_local()) { ODP_ERR("ODP packet io local init failed.\n"); @@ -111,3 +130,9 @@ int odp_init_local(int thr_id) return 0; } + +int odp_term_local(void) +{ + ODP_UNIMPLEMENTED(); + return 0; +} diff --git a/platform/linux-dpdk/odp_linux.c b/platform/linux-dpdk/odp_linux.c index 96c91a5..2b68c13 100644 --- a/platform/linux-dpdk/odp_linux.c +++ b/platform/linux-dpdk/odp_linux.c @@ -8,6 +8,9 @@ #define _GNU_SOURCE #endif #include +#include +#include +#include #include #include @@ -19,28 +22,30 @@ #include #include #include +#include #include typedef struct { - int thr_id; void *(*start_routine) (void *); void *arg; - } odp_start_args_t; - static void *odp_run_start_routine(void *arg) { odp_start_args_t *start_args = arg; /* ODP thread local init */ - odp_init_local(start_args->thr_id); + if (odp_init_local()) { + ODP_ERR("Local init failed\n"); + return NULL; + } - return start_args->start_routine(start_args->arg); + void *ret = start_args->start_routine(start_args->arg); + _odp_flush_caches(); + return ret; } - void odph_linux_pthread_create(odph_linux_pthread_t *thread_tbl, int num, int first_core, void *(*start_routine) (void *), void *arg) { @@ -62,12 +67,12 @@ void odph_linux_pthread_create(odph_linux_pthread_t *thread_tbl, int num, cpu = (first_core + i) % core_count; start_args = malloc(sizeof(odp_start_args_t)); + if (start_args == NULL) + ODP_ABORT("Malloc failed"); memset(start_args, 0, sizeof(odp_start_args_t)); start_args->start_routine = start_routine; start_args->arg = arg; - odp_thread_create(cpu); - start_args->thr_id = cpu; /* If not master core */ if (cpu != 0) { rte_eal_remote_launch( @@ -94,3 +99,99 @@ void odph_linux_pthread_join(odph_linux_pthread_t *thread_tbl, int num) return; } } + +int odph_linux_process_fork_n(odph_linux_process_t *proc_tbl, + int num, int first_core) +{ + cpu_set_t cpu_set; + pid_t pid; + int core_count; + int cpu; + int i; + + memset(proc_tbl, 0, num*sizeof(odph_linux_process_t)); + + core_count = odp_sys_core_count(); + + if (first_core < 0 || first_core >= core_count) { + ODP_ERR("Bad first_core\n"); + return -1; + } + + if (num < 0 || num > core_count) { + ODP_ERR("Bad num\n"); + return -1; + } + + for (i = 0; i < num; i++) { + cpu = (first_core + i) % core_count; + pid = fork(); + + if (pid < 0) { + ODP_ERR("fork() failed\n"); + return -1; + } + + /* Parent continues to fork */ + if (pid > 0) { + proc_tbl[i].pid = pid; + proc_tbl[i].core = cpu; + continue; + } + + /* Child process */ + CPU_ZERO(&cpu_set); + CPU_SET(cpu, &cpu_set); + + if (sched_setaffinity(0, sizeof(cpu_set_t), &cpu_set)) { + ODP_ERR("sched_setaffinity() failed\n"); + return -2; + } + + if (odp_init_local()) { + ODP_ERR("Local init failed\n"); + return -2; + } + + return 0; + } + + return 1; +} + + +int odph_linux_process_fork(odph_linux_process_t *proc, int core) +{ + return odph_linux_process_fork_n(proc, 1, core); +} + + +int odph_linux_process_wait_n(odph_linux_process_t *proc_tbl, int num) +{ + pid_t pid; + int i, j; + int status; + + for (i = 0; i < num; i++) { + pid = wait(&status); + + if (pid < 0) { + ODP_ERR("wait() failed\n"); + return -1; + } + + for (j = 0; j < num; j++) { + if (proc_tbl[j].pid == pid) { + proc_tbl[j].status = status; + break; + } + } + + if (j == num) { + ODP_ERR("Bad pid\n"); + return -1; + } + } + + return 0; +} diff --git a/platform/linux-dpdk/odp_queue.c b/platform/linux-dpdk/odp_queue.c index 03f7c2c..9e906a3 100644 --- a/platform/linux-dpdk/odp_queue.c +++ b/platform/linux-dpdk/odp_queue.c @@ -11,13 +11,14 @@ #include #include #include +#include #include #include #include #include #include #include -#include +#include #include #ifdef USE_TICKETLOCK diff --git a/platform/linux-dpdk/odp_schedule.c b/platform/linux-dpdk/odp_schedule.c deleted file mode 100644 index cb559e4..0000000 --- a/platform/linux-dpdk/odp_schedule.c +++ /dev/null @@ -1,421 +0,0 @@ -/* Copyright (c) 2013, Linaro Limited - * All rights reserved. - * - * SPDX-License-Identifier: BSD-3-Clause - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include - - -/* Limits to number of scheduled queues */ -#define SCHED_POOL_SIZE (256*1024) - -/* Scheduler sub queues */ -#define QUEUES_PER_PRIO 4 - -/* TODO: random or queue based selection */ -#define SEL_PRI_QUEUE(x) ((QUEUES_PER_PRIO-1) & (queue_to_id(x))) - -/* Maximum number of dequeues */ -#define MAX_DEQ 4 - - -/* Mask of queues per priority */ -typedef uint8_t pri_mask_t; - -ODP_STATIC_ASSERT((8 * sizeof(pri_mask_t)) >= - QUEUES_PER_PRIO, "pri_mask_t_is_too_small"); - - -typedef struct { - odp_queue_t pri_queue[ODP_CONFIG_SCHED_PRIOS][QUEUES_PER_PRIO]; - pri_mask_t pri_mask[ODP_CONFIG_SCHED_PRIOS]; - odp_spinlock_t mask_lock; - odp_buffer_pool_t pool; -} sched_t; - -typedef struct { - odp_queue_t queue; - -} queue_desc_t; - -typedef struct { - odp_queue_t pri_queue; - odp_buffer_t desc_buf; - - odp_buffer_t buf[MAX_DEQ]; - int num; - int index; - odp_queue_t queue; - int pause; - -} sched_local_t; - -/* Global scheduler context */ -static sched_t *sched; - -/* Thread local scheduler context */ -static __thread sched_local_t sched_local; - - -static inline odp_queue_t select_pri_queue(odp_queue_t queue, int prio) -{ - int id = SEL_PRI_QUEUE(queue); - return sched->pri_queue[prio][id]; -} - - -int odp_schedule_init_global(void) -{ - odp_shm_t shm; - odp_buffer_pool_t pool; - int i, j; - - ODP_DBG("Schedule init ... "); - - shm = odp_shm_reserve("odp_scheduler", - sizeof(sched_t), - ODP_CACHE_LINE_SIZE, 0); - - sched = odp_shm_addr(shm); - - if (sched == NULL) { - ODP_ERR("Schedule init: Shm reserve failed.\n"); - return -1; - } - - pool = odp_buffer_pool_create("odp_sched_pool", NULL, - SCHED_POOL_SIZE, sizeof(queue_desc_t), - ODP_CACHE_LINE_SIZE, - ODP_BUFFER_TYPE_RAW); - - if (pool == ODP_BUFFER_POOL_INVALID) { - ODP_ERR("Schedule init: Pool create failed.\n"); - return -1; - } - - sched->pool = pool; - odp_spinlock_init(&sched->mask_lock); - - for (i = 0; i < ODP_CONFIG_SCHED_PRIOS; i++) { - odp_queue_t queue; - char name[] = "odp_priXX_YY"; - - name[7] = '0' + i / 10; - name[8] = '0' + i - 10*(i / 10); - - for (j = 0; j < QUEUES_PER_PRIO; j++) { - name[10] = '0' + j / 10; - name[11] = '0' + j - 10*(j / 10); - - queue = odp_queue_create(name, - ODP_QUEUE_TYPE_POLL, NULL); - - if (queue == ODP_QUEUE_INVALID) { - ODP_ERR("Sched init: Queue create failed.\n"); - return -1; - } - - sched->pri_queue[i][j] = queue; - sched->pri_mask[i] = 0; - } - } - - ODP_DBG("done\n"); - - return 0; -} - - -int odp_schedule_init_local(void) -{ - int i; - - sched_local.pri_queue = ODP_QUEUE_INVALID; - sched_local.desc_buf = ODP_BUFFER_INVALID; - - for (i = 0; i < MAX_DEQ; i++) - sched_local.buf[i] = ODP_BUFFER_INVALID; - - sched_local.num = 0; - sched_local.index = 0; - sched_local.queue = ODP_QUEUE_INVALID; - sched_local.pause = 0; - - return 0; -} - - -void odp_schedule_mask_set(odp_queue_t queue, int prio) -{ - int id = SEL_PRI_QUEUE(queue); - - odp_spinlock_lock(&sched->mask_lock); - sched->pri_mask[prio] |= 1 << id; - odp_spinlock_unlock(&sched->mask_lock); -} - - -odp_buffer_t odp_schedule_buffer_alloc(odp_queue_t queue) -{ - odp_buffer_t buf; - - buf = odp_buffer_alloc(sched->pool); - - if (buf != ODP_BUFFER_INVALID) { - queue_desc_t *desc; - desc = odp_buffer_addr(buf); - desc->queue = queue; - } - - return buf; -} - - -void odp_schedule_queue(odp_queue_t queue, int prio) -{ - odp_buffer_t desc_buf; - odp_queue_t pri_queue; - - pri_queue = select_pri_queue(queue, prio); - desc_buf = queue_sched_buf(queue); - - odp_queue_enq(pri_queue, desc_buf); -} - - -void odp_schedule_release_atomic(void) -{ - if (sched_local.pri_queue != ODP_QUEUE_INVALID && - sched_local.num == 0) { - /* Release current atomic queue */ - odp_queue_enq(sched_local.pri_queue, sched_local.desc_buf); - sched_local.pri_queue = ODP_QUEUE_INVALID; - } -} - - -static inline int copy_bufs(odp_buffer_t out_buf[], unsigned int max) -{ - int i = 0; - - while (sched_local.num && max) { - out_buf[i] = sched_local.buf[sched_local.index]; - sched_local.index++; - sched_local.num--; - max--; - i++; - } - - return i; -} - - -/* - * Schedule queues - * - * TODO: SYNC_ORDERED not implemented yet - */ -static int schedule(odp_queue_t *out_queue, odp_buffer_t out_buf[], - unsigned int max_num, unsigned int max_deq) -{ - int i, j; - int thr; - int ret; - - if (sched_local.num) { - ret = copy_bufs(out_buf, max_num); - - if (out_queue) - *out_queue = sched_local.queue; - - return ret; - } - - odp_schedule_release_atomic(); - - if (odp_unlikely(sched_local.pause)) - return 0; - - thr = odp_thread_id(); - - for (i = 0; i < ODP_CONFIG_SCHED_PRIOS; i++) { - int id; - - if (sched->pri_mask[i] == 0) - continue; - - id = thr & (QUEUES_PER_PRIO-1); - - for (j = 0; j < QUEUES_PER_PRIO; j++, id++) { - odp_queue_t pri_q; - odp_buffer_t desc_buf; - - if (id >= QUEUES_PER_PRIO) - id = 0; - - if (odp_unlikely((sched->pri_mask[i] & (1 << id)) == 0)) - continue; - - pri_q = sched->pri_queue[i][id]; - desc_buf = odp_queue_deq(pri_q); - - if (desc_buf != ODP_BUFFER_INVALID) { - queue_desc_t *desc; - odp_queue_t queue; - int num; - - desc = odp_buffer_addr(desc_buf); - queue = desc->queue; - - num = odp_queue_deq_multi(queue, - sched_local.buf, - max_deq); - - if (num == 0) { - /* Remove empty queue from scheduling, - * except packet input queues - */ - if (odp_queue_type(queue) == - ODP_QUEUE_TYPE_PKTIN) - odp_queue_enq(pri_q, desc_buf); - - continue; - } - - sched_local.num = num; - sched_local.index = 0; - ret = copy_bufs(out_buf, max_num); - - sched_local.queue = queue; - - if (queue_sched_atomic(queue)) { - /* Hold queue during atomic access */ - sched_local.pri_queue = pri_q; - sched_local.desc_buf = desc_buf; - } else { - /* Continue scheduling the queue */ - odp_queue_enq(pri_q, desc_buf); - } - - /* Output the source queue handle */ - if (out_queue) - *out_queue = queue; - - return ret; - } - } - } - - return 0; -} - - -static int schedule_loop(odp_queue_t *out_queue, uint64_t wait, - odp_buffer_t out_buf[], - unsigned int max_num, unsigned int max_deq) -{ - uint64_t start_cycle, cycle, diff; - int ret; - - start_cycle = 0; - - while (1) { - ret = schedule(out_queue, out_buf, max_num, max_deq); - - if (ret) - break; - - if (wait == ODP_SCHED_WAIT) - continue; - - if (wait == ODP_SCHED_NO_WAIT) - break; - - if (start_cycle == 0) { - start_cycle = odp_time_get_cycles(); - continue; - } - - cycle = odp_time_get_cycles(); - diff = odp_time_diff_cycles(start_cycle, cycle); - - if (wait < diff) - break; - } - - return ret; -} - - -odp_buffer_t odp_schedule(odp_queue_t *out_queue, uint64_t wait) -{ - odp_buffer_t buf; - - buf = ODP_BUFFER_INVALID; - - schedule_loop(out_queue, wait, &buf, 1, MAX_DEQ); - - return buf; -} - - -odp_buffer_t odp_schedule_one(odp_queue_t *out_queue, uint64_t wait) -{ - odp_buffer_t buf; - - buf = ODP_BUFFER_INVALID; - - schedule_loop(out_queue, wait, &buf, 1, 1); - - return buf; -} - - -int odp_schedule_multi(odp_queue_t *out_queue, uint64_t wait, - odp_buffer_t out_buf[], unsigned int num) -{ - return schedule_loop(out_queue, wait, out_buf, num, MAX_DEQ); -} - - -void odp_schedule_pause(void) -{ - sched_local.pause = 1; -} - - -void odp_schedule_resume(void) -{ - sched_local.pause = 0; -} - - -uint64_t odp_schedule_wait_time(uint64_t ns) -{ - if (ns <= ODP_SCHED_NO_WAIT) - ns = ODP_SCHED_NO_WAIT + 1; - - return odp_time_ns_to_cycles(ns); -} - - -int odp_schedule_num_prio(void) -{ - return ODP_CONFIG_SCHED_PRIOS; -}