From patchwork Fri Mar 2 15:00:10 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 130555 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp1379890lja; Fri, 2 Mar 2018 07:12:21 -0800 (PST) X-Google-Smtp-Source: AG47ELtpfI1KAMg+5TrW25wBj4PyuxpRK3m59HMlaqG+K2GZLNIvvllvmEzMpBFbQvDdyzXK0ulx X-Received: by 10.55.12.19 with SMTP id 19mr8697031qkm.277.1520003541393; Fri, 02 Mar 2018 07:12:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520003541; cv=none; d=google.com; s=arc-20160816; b=tcOjQte+kqYn7BQa+tPSMU740u0W4HB+82faajxyJyzPjdd0vJ345ctbTgdUK95E4K f9HfzgZxeir24PzePjCP8OMXxDgBf99kWfbo01tLnE7hC9/HgsH+7kkqxJMMMlm3WzfO gd6G+YhGZPSzkeGX8Kvu9NpbGP/oPY+/P5bogV0hN4OuohXCdqt69pjpWgq/9DwwNLQI xi/vOnuezrkBfzooRqrMNr/kImE6Gc6Y/dmaSh2rDOzYhHiMk26IaQT6QwLk7QEA3yaY oE36zpaJZoZTSARvrXGyPvO50FJ2G9VIvHJrRtpipIb4hrBundTKrX6eOEynkXr+Ij4L E+aA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=9eFvDmbemaFYCZpw3Nm22GsXjFVEtySezVbcjALYink=; b=t2vjNIPb7pOCfZJmCkDwqAeMxr61C122w91ddh7zjnz/Y6ZYzyTtsXXOuQgTxQha2J N5EVtzhEYgSsVeSBjcZGtbXV1fIRf9TXn+s6p59QOOxtD1Rw1hansjcGofxSg/FATTuK xINyLqNn6/ue+swphmAj5k2Tsrtj0BLyicnk4YlBhUgV1cwCfEJ70Nc6MPpFIMVHCrlE DMRfJDzTeP7wI7zu1Vw4tNV2GU6gQy2HZh/VgkyibQPP0l1ywyvPgN8Q4XTjljoff6iG EPFadPZLGeaMK+04kpf21spuV9/zImXHmwZfuuaoNsyNBjKQjk21X/BgBeCBAR4JE9kJ gPgw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id 27si7354894qkz.45.2018.03.02.07.12.21; Fri, 02 Mar 2018 07:12:21 -0800 (PST) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 0A7DD61782; Fri, 2 Mar 2018 15:12:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H2 autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 40BD1609D9; Fri, 2 Mar 2018 15:01:30 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id BFF6260744; Fri, 2 Mar 2018 15:00:35 +0000 (UTC) Received: from forward100j.mail.yandex.net (forward100j.mail.yandex.net [5.45.198.240]) by lists.linaro.org (Postfix) with ESMTPS id C2AB060CBD for ; Fri, 2 Mar 2018 15:00:18 +0000 (UTC) Received: from mxback3o.mail.yandex.net (mxback3o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::1d]) by forward100j.mail.yandex.net (Yandex) with ESMTP id A1EF75D81F8E for ; Fri, 2 Mar 2018 18:00:17 +0300 (MSK) Received: from smtp2j.mail.yandex.net (smtp2j.mail.yandex.net [2a02:6b8:0:801::ac]) by mxback3o.mail.yandex.net (nwsmtp/Yandex) with ESMTP id ZMBTuJuLNZ-0HwuP4bX; Fri, 02 Mar 2018 18:00:17 +0300 Received: by smtp2j.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id ChbgkO0UiZ-0G2iP0vd; Fri, 02 Mar 2018 18:00:16 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 2 Mar 2018 18:00:10 +0300 Message-Id: <1520002815-30682-2-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520002815-30682-1-git-send-email-odpbot@yandex.ru> References: <1520002815-30682-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 504 Subject: [lng-odp] [PATCH v2 1/6] linux-gen: sched: optimize packet input polling X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Optimize scheduler throughput with packet IO interfaces. Special pktio poll commands are removed and event queue is used instead to trigger packet input polling. Packet input is polled when those queues are empty. Thus, these queues connected to packet input are not removed from scheduling when empty. Signed-off-by: Petri Savolainen --- /** Email created from pull request 504 (psavol:master-sched-optim-2) ** https://github.com/Linaro/odp/pull/504 ** Patch: https://github.com/Linaro/odp/pull/504.patch ** Base sha: e1c0e4570a45d05dd9f2e8e052ce71164209d112 ** Merge commit sha: 964132736e0785222be184065d6ac73121cd46ac **/ platform/linux-generic/include/odp_schedule_if.h | 4 +- platform/linux-generic/odp_queue_basic.c | 24 +- platform/linux-generic/odp_schedule_basic.c | 331 +++++++---------------- platform/linux-generic/odp_schedule_iquery.c | 4 +- platform/linux-generic/odp_schedule_sp.c | 2 +- 5 files changed, 129 insertions(+), 236 deletions(-) diff --git a/platform/linux-generic/include/odp_schedule_if.h b/platform/linux-generic/include/odp_schedule_if.h index 66e050438..dd2097bfb 100644 --- a/platform/linux-generic/include/odp_schedule_if.h +++ b/platform/linux-generic/include/odp_schedule_if.h @@ -82,7 +82,9 @@ int sched_cb_pktin_poll_one(int pktio_index, int rx_queue, odp_event_t evts[]); void sched_cb_pktio_stop_finalize(int pktio_index); odp_queue_t sched_cb_queue_handle(uint32_t queue_index); void sched_cb_queue_destroy_finalize(uint32_t queue_index); -int sched_cb_queue_deq_multi(uint32_t queue_index, odp_event_t ev[], int num); +void sched_cb_queue_set_status(uint32_t queue_index, int status); +int sched_cb_queue_deq_multi(uint32_t queue_index, odp_event_t ev[], int num, + int update_status); int sched_cb_queue_empty(uint32_t queue_index); /* API functions */ diff --git a/platform/linux-generic/odp_queue_basic.c b/platform/linux-generic/odp_queue_basic.c index e4f6fd820..d9e5fdcea 100644 --- a/platform/linux-generic/odp_queue_basic.c +++ b/platform/linux-generic/odp_queue_basic.c @@ -299,6 +299,17 @@ void sched_cb_queue_destroy_finalize(uint32_t queue_index) UNLOCK(queue); } +void sched_cb_queue_set_status(uint32_t queue_index, int status) +{ + queue_entry_t *queue = get_qentry(queue_index); + + LOCK(queue); + + queue->s.status = status; + + UNLOCK(queue); +} + static int queue_destroy(odp_queue_t handle) { queue_entry_t *queue; @@ -493,7 +504,7 @@ static int queue_enq(odp_queue_t handle, odp_event_t ev) } static inline int deq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], - int num) + int num, int update_status) { int status_sync = sched_fn->status_sync; int num_deq; @@ -515,7 +526,7 @@ static inline int deq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], if (num_deq == 0) { /* Already empty queue */ - if (queue->s.status == QUEUE_STATUS_SCHED) { + if (update_status && queue->s.status == QUEUE_STATUS_SCHED) { queue->s.status = QUEUE_STATUS_NOTSCHED; if (status_sync) @@ -542,7 +553,7 @@ static int queue_int_deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], { queue_entry_t *queue = qentry_from_int(q_int); - return deq_multi(queue, buf_hdr, num); + return deq_multi(queue, buf_hdr, num, 0); } static odp_buffer_hdr_t *queue_int_deq(queue_t q_int) @@ -551,7 +562,7 @@ static odp_buffer_hdr_t *queue_int_deq(queue_t q_int) odp_buffer_hdr_t *buf_hdr = NULL; int ret; - ret = deq_multi(queue, &buf_hdr, 1); + ret = deq_multi(queue, &buf_hdr, 1, 0); if (ret == 1) return buf_hdr; @@ -661,11 +672,12 @@ static int queue_info(odp_queue_t handle, odp_queue_info_t *info) return 0; } -int sched_cb_queue_deq_multi(uint32_t queue_index, odp_event_t ev[], int num) +int sched_cb_queue_deq_multi(uint32_t queue_index, odp_event_t ev[], int num, + int update_status) { queue_entry_t *qe = get_qentry(queue_index); - return deq_multi(qe, (odp_buffer_hdr_t **)ev, num); + return deq_multi(qe, (odp_buffer_hdr_t **)ev, num, update_status); } int sched_cb_queue_empty(uint32_t queue_index) diff --git a/platform/linux-generic/odp_schedule_basic.c b/platform/linux-generic/odp_schedule_basic.c index e5e1fe60e..b3847ab91 100644 --- a/platform/linux-generic/odp_schedule_basic.c +++ b/platform/linux-generic/odp_schedule_basic.c @@ -50,39 +50,15 @@ ODP_STATIC_ASSERT((ODP_SCHED_PRIO_NORMAL > 0) && /* Size of poll weight table */ #define WEIGHT_TBL_SIZE ((QUEUES_PER_PRIO - 1) * PREFER_RATIO) -/* Packet input poll cmd queues */ -#define PKTIO_CMD_QUEUES 4 - -/* Mask for wrapping command queues */ -#define PKTIO_CMD_QUEUE_MASK (PKTIO_CMD_QUEUES - 1) - -/* Maximum number of packet input queues per command */ -#define MAX_PKTIN 16 - /* Maximum number of packet IO interfaces */ #define NUM_PKTIO ODP_CONFIG_PKTIO_ENTRIES -/* Maximum number of pktio poll commands */ -#define NUM_PKTIO_CMD (MAX_PKTIN * NUM_PKTIO) +/* Maximum pktin index. Needs to fit into 8 bits. */ +#define MAX_PKTIN_INDEX 255 /* Not a valid index */ #define NULL_INDEX ((uint32_t)-1) -/* Not a valid poll command */ -#define PKTIO_CMD_INVALID NULL_INDEX - -/* Pktio command is free */ -#define PKTIO_CMD_FREE PKTIO_CMD_INVALID - -/* Packet IO poll queue ring size. In worst case, all pktios have all pktins - * enabled and one poll command is created per pktin queue. The ring size must - * be larger than or equal to NUM_PKTIO_CMD / PKTIO_CMD_QUEUES, so that it can - * hold all poll commands in the worst case. */ -#define PKTIO_RING_SIZE (NUM_PKTIO_CMD / PKTIO_CMD_QUEUES) - -/* Mask for wrapping around pktio poll command index */ -#define PKTIO_RING_MASK (PKTIO_RING_SIZE - 1) - /* Priority queue ring size. In worst case, all event queues are scheduled * queues and have the same priority. The ring size must be larger than or * equal to ODP_CONFIG_QUEUES / QUEUES_PER_PRIO, so that it can hold all @@ -90,7 +66,7 @@ ODP_STATIC_ASSERT((ODP_SCHED_PRIO_NORMAL > 0) && #define PRIO_QUEUE_RING_SIZE (ODP_CONFIG_QUEUES / QUEUES_PER_PRIO) /* Mask for wrapping around priority queue index */ -#define PRIO_QUEUE_MASK (PRIO_QUEUE_RING_SIZE - 1) +#define RING_MASK (PRIO_QUEUE_RING_SIZE - 1) /* Priority queue empty, not a valid queue index. */ #define PRIO_QUEUE_EMPTY NULL_INDEX @@ -103,15 +79,6 @@ ODP_STATIC_ASSERT(CHECK_IS_POWER2(ODP_CONFIG_QUEUES), ODP_STATIC_ASSERT(CHECK_IS_POWER2(PRIO_QUEUE_RING_SIZE), "Ring_size_is_not_power_of_two"); -/* Ring size must be power of two, so that PKTIO_RING_MASK can be used. */ -ODP_STATIC_ASSERT(CHECK_IS_POWER2(PKTIO_RING_SIZE), - "pktio_ring_size_is_not_power_of_two"); - -/* Number of commands queues must be power of two, so that PKTIO_CMD_QUEUE_MASK - * can be used. */ -ODP_STATIC_ASSERT(CHECK_IS_POWER2(PKTIO_CMD_QUEUES), - "pktio_cmd_queues_is_not_power_of_two"); - /* Mask of queues per priority */ typedef uint8_t pri_mask_t; @@ -150,7 +117,6 @@ typedef struct { int index; int pause; uint16_t round; - uint16_t pktin_polls; uint32_t queue_index; odp_queue_t queue; odp_event_t ev_stash[MAX_DEQ]; @@ -183,24 +149,6 @@ typedef struct ODP_ALIGNED_CACHE { } prio_queue_t; -/* Packet IO queue */ -typedef struct ODP_ALIGNED_CACHE { - /* Ring header */ - ring_t ring; - - /* Ring data: pktio poll command indexes */ - uint32_t cmd_index[PKTIO_RING_SIZE]; - -} pktio_queue_t; - -/* Packet IO poll command */ -typedef struct { - int pktio_index; - int num_pktin; - int pktin[MAX_PKTIN]; - uint32_t cmd_index; -} pktio_cmd_t; - /* Order context of a queue */ typedef struct ODP_ALIGNED_CACHE { /* Current ordered context id */ @@ -220,16 +168,6 @@ typedef struct { prio_queue_t prio_q[NUM_SCHED_GRPS][NUM_PRIO][QUEUES_PER_PRIO]; - odp_spinlock_t poll_cmd_lock; - /* Number of commands in a command queue */ - uint16_t num_pktio_cmd[PKTIO_CMD_QUEUES]; - - /* Packet IO command queues */ - pktio_queue_t pktio_q[PKTIO_CMD_QUEUES]; - - /* Packet IO poll commands */ - pktio_cmd_t pktio_cmd[NUM_PKTIO_CMD]; - odp_shm_t shm; uint32_t pri_count[NUM_PRIO][QUEUES_PER_PRIO]; @@ -244,22 +182,34 @@ typedef struct { } sched_grp[NUM_SCHED_GRPS]; struct { - int grp; - int prio; - int queue_per_prio; - int sync; - uint32_t order_lock_count; + uint8_t grp; + uint8_t prio; + uint8_t queue_per_prio; + uint8_t sync; + uint8_t order_lock_count; + uint8_t poll_pktin; + uint8_t pktio_index; + uint8_t pktin_index; } queue[ODP_CONFIG_QUEUES]; struct { - /* Number of active commands for a pktio interface */ - int num_cmd; + int num_pktin; } pktio[NUM_PKTIO]; + odp_spinlock_t pktio_lock; order_context_t order[ODP_CONFIG_QUEUES]; } sched_global_t; +/* Check that queue[] variables are large enough */ +ODP_STATIC_ASSERT(NUM_SCHED_GRPS <= 256, "Group_does_not_fit_8_bits"); +ODP_STATIC_ASSERT(NUM_PRIO <= 256, "Prio_does_not_fit_8_bits"); +ODP_STATIC_ASSERT(QUEUES_PER_PRIO <= 256, + "Queues_per_prio_does_not_fit_8_bits"); +ODP_STATIC_ASSERT(CONFIG_QUEUE_MAX_ORD_LOCKS <= 256, + "Ordered_lock_count_does_not_fit_8_bits"); +ODP_STATIC_ASSERT(NUM_PKTIO <= 256, "Pktio_index_does_not_fit_8_bits"); + /* Global scheduler context */ static sched_global_t *sched; @@ -337,16 +287,9 @@ static int schedule_init_global(void) } } - odp_spinlock_init(&sched->poll_cmd_lock); - for (i = 0; i < PKTIO_CMD_QUEUES; i++) { - ring_init(&sched->pktio_q[i].ring); - - for (j = 0; j < PKTIO_RING_SIZE; j++) - sched->pktio_q[i].cmd_index[j] = PKTIO_CMD_INVALID; - } - - for (i = 0; i < NUM_PKTIO_CMD; i++) - sched->pktio_cmd[i].cmd_index = PKTIO_CMD_FREE; + odp_spinlock_init(&sched->pktio_lock); + for (i = 0; i < NUM_PKTIO; i++) + sched->pktio[i].num_pktin = 0; odp_spinlock_init(&sched->grp_lock); odp_atomic_init_u32(&sched->grp_epoch, 0); @@ -384,14 +327,14 @@ static int schedule_term_global(void) ring_t *ring = &sched->prio_q[grp][i][j].ring; uint32_t qi; - while ((qi = ring_deq(ring, PRIO_QUEUE_MASK)) != + while ((qi = ring_deq(ring, RING_MASK)) != RING_EMPTY) { odp_event_t events[1]; int num; num = sched_cb_queue_deq_multi(qi, events, - 1); + 1, 1); if (num < 0) queue_destroy_finalize(qi); @@ -519,6 +462,9 @@ static int schedule_init_queue(uint32_t queue_index, sched->queue[queue_index].queue_per_prio = queue_per_prio(queue_index); sched->queue[queue_index].sync = sched_param->sync; sched->queue[queue_index].order_lock_count = sched_param->lock_count; + sched->queue[queue_index].poll_pktin = 0; + sched->queue[queue_index].pktio_index = 0; + sched->queue[queue_index].pktin_index = 0; odp_atomic_init_u64(&sched->order[queue_index].ctx, 0); odp_atomic_init_u64(&sched->order[queue_index].next_ctx, 0); @@ -554,88 +500,39 @@ static void schedule_destroy_queue(uint32_t queue_index) ODP_ERR("queue reorder incomplete\n"); } -static int poll_cmd_queue_idx(int pktio_index, int pktin_idx) -{ - return PKTIO_CMD_QUEUE_MASK & (pktio_index ^ pktin_idx); -} - -static inline pktio_cmd_t *alloc_pktio_cmd(void) -{ - int i; - pktio_cmd_t *cmd = NULL; - - odp_spinlock_lock(&sched->poll_cmd_lock); - - /* Find next free command */ - for (i = 0; i < NUM_PKTIO_CMD; i++) { - if (sched->pktio_cmd[i].cmd_index == PKTIO_CMD_FREE) { - cmd = &sched->pktio_cmd[i]; - cmd->cmd_index = i; - break; - } - } - - odp_spinlock_unlock(&sched->poll_cmd_lock); - - return cmd; -} - -static inline void free_pktio_cmd(pktio_cmd_t *cmd) +static int schedule_sched_queue(uint32_t queue_index) { - odp_spinlock_lock(&sched->poll_cmd_lock); - - cmd->cmd_index = PKTIO_CMD_FREE; + int grp = sched->queue[queue_index].grp; + int prio = sched->queue[queue_index].prio; + int queue_per_prio = sched->queue[queue_index].queue_per_prio; + ring_t *ring = &sched->prio_q[grp][prio][queue_per_prio].ring; - odp_spinlock_unlock(&sched->poll_cmd_lock); + ring_enq(ring, RING_MASK, queue_index); + return 0; } static void schedule_pktio_start(int pktio_index, int num_pktin, - int pktin_idx[], odp_queue_t odpq[] ODP_UNUSED) + int pktin_idx[], odp_queue_t queue[]) { - int i, idx; - pktio_cmd_t *cmd; - - if (num_pktin > MAX_PKTIN) - ODP_ABORT("Too many input queues for scheduler\n"); + int i; + uint32_t qi; - sched->pktio[pktio_index].num_cmd = num_pktin; + sched->pktio[pktio_index].num_pktin = num_pktin; - /* Create a pktio poll command per queue */ for (i = 0; i < num_pktin; i++) { + qi = queue_to_index(queue[i]); + sched->queue[qi].poll_pktin = 1; + sched->queue[qi].pktio_index = pktio_index; + sched->queue[qi].pktin_index = pktin_idx[i]; - cmd = alloc_pktio_cmd(); - - if (cmd == NULL) - ODP_ABORT("Scheduler out of pktio commands\n"); - - idx = poll_cmd_queue_idx(pktio_index, pktin_idx[i]); + ODP_ASSERT(pktin_idx[i] <= MAX_PKTIN_INDEX); - odp_spinlock_lock(&sched->poll_cmd_lock); - sched->num_pktio_cmd[idx]++; - odp_spinlock_unlock(&sched->poll_cmd_lock); - - cmd->pktio_index = pktio_index; - cmd->num_pktin = 1; - cmd->pktin[0] = pktin_idx[i]; - ring_enq(&sched->pktio_q[idx].ring, PKTIO_RING_MASK, - cmd->cmd_index); + /* Start polling */ + sched_cb_queue_set_status(qi, QUEUE_STATUS_SCHED); + schedule_sched_queue(qi); } } -static int schedule_pktio_stop(int pktio_index, int first_pktin) -{ - int num; - int idx = poll_cmd_queue_idx(pktio_index, first_pktin); - - odp_spinlock_lock(&sched->poll_cmd_lock); - sched->num_pktio_cmd[idx]--; - sched->pktio[pktio_index].num_cmd--; - num = sched->pktio[pktio_index].num_cmd; - odp_spinlock_unlock(&sched->poll_cmd_lock); - - return num; -} - static void schedule_release_atomic(void) { uint32_t qi = sched_local.queue_index; @@ -647,7 +544,7 @@ static void schedule_release_atomic(void) ring_t *ring = &sched->prio_q[grp][prio][queue_per_prio].ring; /* Release current atomic queue */ - ring_enq(ring, PRIO_QUEUE_MASK, qi); + ring_enq(ring, RING_MASK, qi); sched_local.queue_index = PRIO_QUEUE_EMPTY; } } @@ -797,6 +694,37 @@ static int schedule_ord_enq_multi(queue_t q_int, void *buf_hdr[], return 1; } +static inline int queue_is_pktin(uint32_t queue_index) +{ + return sched->queue[queue_index].poll_pktin; +} + +static inline int poll_pktin(uint32_t qi) +{ + int pktio_index, pktin_index, num, num_pktin; + + pktio_index = sched->queue[qi].pktio_index; + pktin_index = sched->queue[qi].pktin_index; + + num = sched_cb_pktin_poll(pktio_index, 1, &pktin_index); + + /* Pktio stopped or closed. Call stop_finalize when we have stopped + * polling all pktin queues of the pktio. */ + if (odp_unlikely(num < 0)) { + odp_spinlock_lock(&sched->pktio_lock); + sched->pktio[pktio_index].num_pktin--; + num_pktin = sched->pktio[pktio_index].num_pktin; + odp_spinlock_unlock(&sched->pktio_lock); + + sched_cb_queue_set_status(qi, QUEUE_STATUS_NOTSCHED); + + if (num_pktin == 0) + sched_cb_pktio_stop_finalize(pktio_index); + } + + return num; +} + static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], unsigned int max_num, int grp, int first) { @@ -820,6 +748,7 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], int ordered; odp_queue_t handle; ring_t *ring; + int pktin; if (id >= QUEUES_PER_PRIO) id = 0; @@ -834,7 +763,7 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], /* Get queue index from the priority queue */ ring = &sched->prio_q[grp][prio][id].ring; - qi = ring_deq(ring, PRIO_QUEUE_MASK); + qi = ring_deq(ring, RING_MASK); /* Priority queue empty */ if (qi == RING_EMPTY) { @@ -857,8 +786,10 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], if (ordered && max_num < MAX_DEQ) max_deq = max_num; + pktin = queue_is_pktin(qi); + num = sched_cb_queue_deq_multi(qi, sched_local.ev_stash, - max_deq); + max_deq, !pktin); if (num < 0) { /* Destroyed queue. Continue scheduling the same @@ -868,6 +799,20 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], } if (num == 0) { + /* Poll packet input. Continue scheduling queue + * connected to a packet input. Move to the next + * priority to avoid starvation of other + * priorities. Stop scheduling queue when pktio + * has been stopped. */ + if (pktin) { + int num_pkt = poll_pktin(qi); + + if (odp_likely(num_pkt >= 0)) { + ring_enq(ring, RING_MASK, qi); + break; + } + } + /* Remove empty queue from scheduling. Continue * scheduling the same priority queue. */ continue; @@ -890,14 +835,14 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], sched_local.ordered.src_queue = qi; /* Continue scheduling ordered queues */ - ring_enq(ring, PRIO_QUEUE_MASK, qi); + ring_enq(ring, RING_MASK, qi); } else if (queue_is_atomic(qi)) { /* Hold queue during atomic access */ sched_local.queue_index = qi; } else { /* Continue scheduling the queue */ - ring_enq(ring, PRIO_QUEUE_MASK, qi); + ring_enq(ring, RING_MASK, qi); } /* Output the source queue handle */ @@ -919,7 +864,7 @@ static inline int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], { int i, num_grp; int ret; - int id, first, grp_id; + int first, grp_id; uint16_t round; uint32_t epoch; @@ -972,66 +917,11 @@ static inline int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], grp_id = 0; } - /* - * Poll packet input when there are no events - * * Each thread starts the search for a poll command from its - * preferred command queue. If the queue is empty, it moves to other - * queues. - * * Most of the times, the search stops on the first command found to - * optimize multi-threaded performance. A small portion of polls - * have to do full iteration to avoid packet input starvation when - * there are less threads than command queues. - */ - id = sched_local.thr & PKTIO_CMD_QUEUE_MASK; - - for (i = 0; i < PKTIO_CMD_QUEUES; i++, id = ((id + 1) & - PKTIO_CMD_QUEUE_MASK)) { - ring_t *ring; - uint32_t cmd_index; - pktio_cmd_t *cmd; - - if (odp_unlikely(sched->num_pktio_cmd[id] == 0)) - continue; - - ring = &sched->pktio_q[id].ring; - cmd_index = ring_deq(ring, PKTIO_RING_MASK); - - if (odp_unlikely(cmd_index == RING_EMPTY)) - continue; - - cmd = &sched->pktio_cmd[cmd_index]; - - /* Poll packet input */ - if (odp_unlikely(sched_cb_pktin_poll(cmd->pktio_index, - cmd->num_pktin, - cmd->pktin))){ - /* Pktio stopped or closed. Remove poll command and call - * stop_finalize when all commands of the pktio has - * been removed. */ - if (schedule_pktio_stop(cmd->pktio_index, - cmd->pktin[0]) == 0) - sched_cb_pktio_stop_finalize(cmd->pktio_index); - - free_pktio_cmd(cmd); - } else { - /* Continue scheduling the pktio */ - ring_enq(ring, PKTIO_RING_MASK, cmd_index); - - /* Do not iterate through all pktin poll command queues - * every time. */ - if (odp_likely(sched_local.pktin_polls & 0xf)) - break; - } - } - - sched_local.pktin_polls++; return 0; } - -static int schedule_loop(odp_queue_t *out_queue, uint64_t wait, - odp_event_t out_ev[], - unsigned int max_num) +static inline int schedule_loop(odp_queue_t *out_queue, uint64_t wait, + odp_event_t out_ev[], unsigned int max_num) { odp_time_t next, wtime; int first = 1; @@ -1387,17 +1277,6 @@ static void schedule_prefetch(int num ODP_UNUSED) { } -static int schedule_sched_queue(uint32_t queue_index) -{ - int grp = sched->queue[queue_index].grp; - int prio = sched->queue[queue_index].prio; - int queue_per_prio = sched->queue[queue_index].queue_per_prio; - ring_t *ring = &sched->prio_q[grp][prio][queue_per_prio].ring; - - ring_enq(ring, PRIO_QUEUE_MASK, queue_index); - return 0; -} - static int schedule_num_grps(void) { return NUM_SCHED_GRPS; diff --git a/platform/linux-generic/odp_schedule_iquery.c b/platform/linux-generic/odp_schedule_iquery.c index ea62c3645..1a82f48da 100644 --- a/platform/linux-generic/odp_schedule_iquery.c +++ b/platform/linux-generic/odp_schedule_iquery.c @@ -291,7 +291,7 @@ static int schedule_term_global(void) odp_event_t events[1]; if (sched->availables[i]) - count = sched_cb_queue_deq_multi(i, events, 1); + count = sched_cb_queue_deq_multi(i, events, 1, 1); if (count < 0) sched_cb_queue_destroy_finalize(i); @@ -1527,7 +1527,7 @@ static inline int consume_queue(int prio, unsigned int queue_index) max = 1; count = sched_cb_queue_deq_multi( - queue_index, cache->stash, max); + queue_index, cache->stash, max, 1); if (count < 0) { DO_SCHED_UNLOCK(); diff --git a/platform/linux-generic/odp_schedule_sp.c b/platform/linux-generic/odp_schedule_sp.c index 007d673f0..50390274b 100644 --- a/platform/linux-generic/odp_schedule_sp.c +++ b/platform/linux-generic/odp_schedule_sp.c @@ -559,7 +559,7 @@ static int schedule_multi(odp_queue_t *from, uint64_t wait, } qi = cmd->s.index; - num = sched_cb_queue_deq_multi(qi, events, 1); + num = sched_cb_queue_deq_multi(qi, events, 1, 1); if (num > 0) { sched_local.cmd = cmd; From patchwork Fri Mar 2 15:00:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 130554 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp1378489lja; Fri, 2 Mar 2018 07:10:59 -0800 (PST) X-Google-Smtp-Source: AG47ELv4tSEOkNnuDEDG8KCVo7WEwEzhOYwCRvsvFP/Dpi3Strq15kvj4GIfujy1MUq2GS1F0aZF X-Received: by 10.55.25.134 with SMTP id 6mr8290465qkz.196.1520003458981; Fri, 02 Mar 2018 07:10:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520003458; cv=none; d=google.com; s=arc-20160816; b=wPn4EG8To5mfa/2QUtXYLnbJqo6oPaUmcLdbXhLIOo793wu8o8fCVKhgq/z7DQghpx eO5+UrTBuKy6Zzhb25UI/dtz79f3qlWoqKwU65iVNb+GosbG06zm/5P3VP+w6qLKdlQM zP8Q4umhRBLt8YmRzFbA0JWbz3+W2j3lqwWMPXmN0sZ+BpKndHYLrtvq+bw4t8b8MeAT 8Zv6MqY+IdRH15C15+n0WpJJ6rcoCKaRQwvyzuBuxIFEIQFXbfnrGn+xqB9no87K2Ugs qPUqbZuTioJZnSzZoBMjgfzTLGqEXeOEganYRoYOyKD7KlOublL3ih+9WnU112m8v2FS aTMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=+mMOzBd5g96WBfe4JxJbsqzVPlY9Mj6n/xf5Ge2PwbM=; b=UYiDGaRCZyg1XdDZvqZP4MDhfYaYtb0Vz0CFFLSTPyXXsJNpDj23rL8wsnQ8c9/XQA cMGRtuCm1eCFzvQvOcfmEOO/UF13DKBS6tP1/kUfwA0rID0PAGvjte33n0cPLAv6RS6P QTjGnpbtxLXgwTYd1QktPsaa4NCg/wQpG+TvnWPV4J+w8gzLJTWzmSlNjEGGLSmO6/iP fQl3fGgSsflpDvEd86sgtpDOIXzl8ZpqROFjo0JV/3KX8UqePX7iFnWz6gZu3L4Hh8Rh AmkSu69G0V+pfxxMpFdNmTdHls5ySXL3UjiAt5LU+NsAGCugONCBk1z7ns3pLzaNQ5hT GLpw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id z15si7021344qth.434.2018.03.02.07.10.58; Fri, 02 Mar 2018 07:10:58 -0800 (PST) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 8D6B26175E; Fri, 2 Mar 2018 15:10:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id BFD82607DC; Fri, 2 Mar 2018 15:01:20 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 369216178E; Fri, 2 Mar 2018 15:00:30 +0000 (UTC) Received: from forward101j.mail.yandex.net (forward101j.mail.yandex.net [5.45.198.241]) by lists.linaro.org (Postfix) with ESMTPS id 26B1D61750 for ; Fri, 2 Mar 2018 15:00:20 +0000 (UTC) Received: from mxback4o.mail.yandex.net (mxback4o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::1e]) by forward101j.mail.yandex.net (Yandex) with ESMTP id CAB2512453C5 for ; Fri, 2 Mar 2018 18:00:18 +0300 (MSK) Received: from smtp2j.mail.yandex.net (smtp2j.mail.yandex.net [2a02:6b8:0:801::ac]) by mxback4o.mail.yandex.net (nwsmtp/Yandex) with ESMTP id NGTvmCuKb8-0IJiqZde; Fri, 02 Mar 2018 18:00:18 +0300 Received: by smtp2j.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id ChbgkO0UiZ-0H2qRvko; Fri, 02 Mar 2018 18:00:17 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 2 Mar 2018 18:00:11 +0300 Message-Id: <1520002815-30682-3-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520002815-30682-1-git-send-email-odpbot@yandex.ru> References: <1520002815-30682-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 504 Subject: [lng-odp] [PATCH v2 2/6] linux-gen: sched: optimize atomic packet input queue throughput X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen When packet input queue is atomic, packets received from packet input are passed directly to the application. Other queue types may have events stashed on other threads, so for those incoming packets are always enqueued (to maintain packet order). Signed-off-by: Petri Savolainen --- /** Email created from pull request 504 (psavol:master-sched-optim-2) ** https://github.com/Linaro/odp/pull/504 ** Patch: https://github.com/Linaro/odp/pull/504.patch ** Base sha: e1c0e4570a45d05dd9f2e8e052ce71164209d112 ** Merge commit sha: 964132736e0785222be184065d6ac73121cd46ac **/ .../linux-generic/include/odp_buffer_inlines.h | 5 ++ .../linux-generic/include/odp_queue_internal.h | 22 ++++++++ platform/linux-generic/include/odp_queue_lf.h | 1 - platform/linux-generic/include/odp_schedule_if.h | 4 +- platform/linux-generic/odp_packet_io.c | 46 +++++++++++++---- platform/linux-generic/odp_queue_basic.c | 17 +----- platform/linux-generic/odp_queue_lf.c | 2 +- platform/linux-generic/odp_schedule_basic.c | 60 ++++++++++++++++++---- platform/linux-generic/odp_schedule_iquery.c | 6 +-- platform/linux-generic/odp_schedule_sp.c | 5 +- 10 files changed, 123 insertions(+), 45 deletions(-) diff --git a/platform/linux-generic/include/odp_buffer_inlines.h b/platform/linux-generic/include/odp_buffer_inlines.h index 9abe79fc7..9e3b6ef70 100644 --- a/platform/linux-generic/include/odp_buffer_inlines.h +++ b/platform/linux-generic/include/odp_buffer_inlines.h @@ -28,6 +28,11 @@ static inline odp_buffer_t buf_from_buf_hdr(odp_buffer_hdr_t *hdr) return (odp_buffer_t)hdr; } +static inline odp_event_t event_from_buf_hdr(odp_buffer_hdr_t *hdr) +{ + return (odp_event_t)hdr; +} + #ifdef __cplusplus } #endif diff --git a/platform/linux-generic/include/odp_queue_internal.h b/platform/linux-generic/include/odp_queue_internal.h index 0540bf72d..3aec3fe9d 100644 --- a/platform/linux-generic/include/odp_queue_internal.h +++ b/platform/linux-generic/include/odp_queue_internal.h @@ -30,6 +30,7 @@ extern "C" { #include #include #include +#include #define QUEUE_STATUS_FREE 0 #define QUEUE_STATUS_DESTROYED 1 @@ -62,6 +63,27 @@ union queue_entry_u { uint8_t pad[ROUNDUP_CACHE_LINE(sizeof(struct queue_entry_s))]; }; +typedef struct ODP_ALIGNED_CACHE { + /* Storage space for ring data */ + uint32_t data[CONFIG_QUEUE_SIZE]; +} queue_ring_data_t; + +typedef struct queue_global_t { + queue_entry_t queue[ODP_CONFIG_QUEUES]; + queue_ring_data_t ring_data[ODP_CONFIG_QUEUES]; + uint32_t queue_lf_num; + uint32_t queue_lf_size; + queue_lf_func_t queue_lf_func; + +} queue_global_t; + +extern queue_global_t *queue_glb; + +static inline queue_t queue_index_to_qint(uint32_t queue_id) +{ + return (queue_t)&queue_glb->queue[queue_id]; +} + static inline uint32_t queue_to_index(odp_queue_t handle) { return _odp_typeval(handle) - 1; diff --git a/platform/linux-generic/include/odp_queue_lf.h b/platform/linux-generic/include/odp_queue_lf.h index ef178e07b..8a973a859 100644 --- a/platform/linux-generic/include/odp_queue_lf.h +++ b/platform/linux-generic/include/odp_queue_lf.h @@ -12,7 +12,6 @@ extern "C" { #endif #include -#include /* Lock-free queue functions */ typedef struct { diff --git a/platform/linux-generic/include/odp_schedule_if.h b/platform/linux-generic/include/odp_schedule_if.h index dd2097bfb..4bd127a42 100644 --- a/platform/linux-generic/include/odp_schedule_if.h +++ b/platform/linux-generic/include/odp_schedule_if.h @@ -77,7 +77,9 @@ typedef struct schedule_fn_t { extern const schedule_fn_t *sched_fn; /* Interface for the scheduler */ -int sched_cb_pktin_poll(int pktio_index, int num_queue, int index[]); +int sched_cb_pktin_poll(int pktio_index, int pktin_index, + odp_buffer_hdr_t *hdr_tbl[], int num); +int sched_cb_pktin_poll_old(int pktio_index, int num_queue, int index[]); int sched_cb_pktin_poll_one(int pktio_index, int rx_queue, odp_event_t evts[]); void sched_cb_pktio_stop_finalize(int pktio_index); odp_queue_t sched_cb_queue_handle(uint32_t queue_index); diff --git a/platform/linux-generic/odp_packet_io.c b/platform/linux-generic/odp_packet_io.c index ae8e390b1..e76ff8140 100644 --- a/platform/linux-generic/odp_packet_io.c +++ b/platform/linux-generic/odp_packet_io.c @@ -559,7 +559,7 @@ odp_pktio_t odp_pktio_lookup(const char *name) return hdl; } -static inline int pktin_recv_buf(odp_pktin_queue_t queue, +static inline int pktin_recv_buf(pktio_entry_t *entry, int pktin_index, odp_buffer_hdr_t *buffer_hdrs[], int num) { odp_packet_t pkt; @@ -570,7 +570,7 @@ static inline int pktin_recv_buf(odp_pktin_queue_t queue, int pkts; int num_rx = 0; - pkts = odp_pktin_recv(queue, packets, num); + pkts = entry->s.ops->recv(entry, pktin_index, packets, num); for (i = 0; i < pkts; i++) { pkt = packets[i]; @@ -624,13 +624,16 @@ static odp_buffer_hdr_t *pktin_dequeue(queue_t q_int) odp_buffer_hdr_t *buf_hdr; odp_buffer_hdr_t *hdr_tbl[QUEUE_MULTI_MAX]; int pkts; + odp_pktin_queue_t pktin_queue = queue_fn->get_pktin(q_int); + odp_pktio_t pktio = pktin_queue.pktio; + int pktin_index = pktin_queue.index; + pktio_entry_t *entry = get_pktio_entry(pktio); buf_hdr = queue_fn->deq(q_int); if (buf_hdr != NULL) return buf_hdr; - pkts = pktin_recv_buf(queue_fn->get_pktin(q_int), - hdr_tbl, QUEUE_MULTI_MAX); + pkts = pktin_recv_buf(entry, pktin_index, hdr_tbl, QUEUE_MULTI_MAX); if (pkts <= 0) return NULL; @@ -646,6 +649,10 @@ static int pktin_deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) int nbr; odp_buffer_hdr_t *hdr_tbl[QUEUE_MULTI_MAX]; int pkts, i, j; + odp_pktin_queue_t pktin_queue = queue_fn->get_pktin(q_int); + odp_pktio_t pktio = pktin_queue.pktio; + int pktin_index = pktin_queue.index; + pktio_entry_t *entry = get_pktio_entry(pktio); nbr = queue_fn->deq_multi(q_int, buf_hdr, num); if (odp_unlikely(nbr > num)) @@ -657,8 +664,8 @@ static int pktin_deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) if (nbr == num) return nbr; - pkts = pktin_recv_buf(queue_fn->get_pktin(q_int), - hdr_tbl, QUEUE_MULTI_MAX); + pkts = pktin_recv_buf(entry, pktin_index, hdr_tbl, QUEUE_MULTI_MAX); + if (pkts <= 0) return nbr; @@ -720,12 +727,29 @@ int sched_cb_pktin_poll_one(int pktio_index, return num_rx; } -int sched_cb_pktin_poll(int pktio_index, int num_queue, int index[]) +int sched_cb_pktin_poll(int pktio_index, int pktin_index, + odp_buffer_hdr_t *hdr_tbl[], int num) +{ + pktio_entry_t *entry = pktio_entry_by_index(pktio_index); + int state = entry->s.state; + + if (odp_unlikely(state != PKTIO_STATE_STARTED)) { + if (state < PKTIO_STATE_ACTIVE || + state == PKTIO_STATE_STOP_PENDING) + return -1; + + ODP_DBG("Interface %s not started\n", entry->s.name); + return 0; + } + + return pktin_recv_buf(entry, pktin_index, hdr_tbl, num); +} + +int sched_cb_pktin_poll_old(int pktio_index, int num_queue, int index[]) { odp_buffer_hdr_t *hdr_tbl[QUEUE_MULTI_MAX]; int num, idx; - pktio_entry_t *entry; - entry = pktio_entry_by_index(pktio_index); + pktio_entry_t *entry = pktio_entry_by_index(pktio_index); int state = entry->s.state; if (odp_unlikely(state != PKTIO_STATE_STARTED)) { @@ -739,9 +763,9 @@ int sched_cb_pktin_poll(int pktio_index, int num_queue, int index[]) for (idx = 0; idx < num_queue; idx++) { queue_t q_int; - odp_pktin_queue_t pktin = entry->s.in_queue[index[idx]].pktin; - num = pktin_recv_buf(pktin, hdr_tbl, QUEUE_MULTI_MAX); + num = pktin_recv_buf(entry, index[idx], hdr_tbl, + QUEUE_MULTI_MAX); if (num == 0) continue; diff --git a/platform/linux-generic/odp_queue_basic.c b/platform/linux-generic/odp_queue_basic.c index d9e5fdcea..399c86ed8 100644 --- a/platform/linux-generic/odp_queue_basic.c +++ b/platform/linux-generic/odp_queue_basic.c @@ -8,7 +8,6 @@ #include #include -#include #include #include #include @@ -40,21 +39,7 @@ static int queue_init(queue_entry_t *queue, const char *name, const odp_queue_param_t *param); -typedef struct ODP_ALIGNED_CACHE { - /* Storage space for ring data */ - uint32_t data[CONFIG_QUEUE_SIZE]; -} ring_data_t; - -typedef struct queue_global_t { - queue_entry_t queue[ODP_CONFIG_QUEUES]; - ring_data_t ring_data[ODP_CONFIG_QUEUES]; - uint32_t queue_lf_num; - uint32_t queue_lf_size; - queue_lf_func_t queue_lf_func; - -} queue_global_t; - -static queue_global_t *queue_glb; +queue_global_t *queue_glb; static inline queue_entry_t *get_qentry(uint32_t queue_id) { diff --git a/platform/linux-generic/odp_queue_lf.c b/platform/linux-generic/odp_queue_lf.c index 74f529469..066e6a674 100644 --- a/platform/linux-generic/odp_queue_lf.c +++ b/platform/linux-generic/odp_queue_lf.c @@ -7,7 +7,7 @@ #include #include #include -#include +#include #include #include diff --git a/platform/linux-generic/odp_schedule_basic.c b/platform/linux-generic/odp_schedule_basic.c index b3847ab91..b251bdeef 100644 --- a/platform/linux-generic/odp_schedule_basic.c +++ b/platform/linux-generic/odp_schedule_basic.c @@ -26,6 +26,7 @@ #include #include #include +#include /* Number of priority levels */ #define NUM_PRIO 8 @@ -699,14 +700,20 @@ static inline int queue_is_pktin(uint32_t queue_index) return sched->queue[queue_index].poll_pktin; } -static inline int poll_pktin(uint32_t qi) +static inline int poll_pktin(uint32_t qi, int stash) { - int pktio_index, pktin_index, num, num_pktin; + odp_buffer_hdr_t *b_hdr[MAX_DEQ]; + int pktio_index, pktin_index, num, num_pktin, i; + int ret; + queue_t qint; pktio_index = sched->queue[qi].pktio_index; pktin_index = sched->queue[qi].pktin_index; - num = sched_cb_pktin_poll(pktio_index, 1, &pktin_index); + num = sched_cb_pktin_poll(pktio_index, pktin_index, b_hdr, MAX_DEQ); + + if (num == 0) + return 0; /* Pktio stopped or closed. Call stop_finalize when we have stopped * polling all pktin queues of the pktio. */ @@ -720,9 +727,33 @@ static inline int poll_pktin(uint32_t qi) if (num_pktin == 0) sched_cb_pktio_stop_finalize(pktio_index); + + return num; } - return num; + if (stash) { + for (i = 0; i < num; i++) + sched_local.ev_stash[i] = event_from_buf_hdr(b_hdr[i]); + + return num; + } + + qint = queue_index_to_qint(qi); + + ret = queue_fn->enq_multi(qint, b_hdr, num); + + /* Drop packets that were not enqueued */ + if (odp_unlikely(ret < num)) { + int num_enq = ret; + + if (odp_unlikely(ret < 0)) + num_enq = 0; + + ODP_DBG("Dropped %i packets\n", num - num_enq); + buffer_free_multi(&b_hdr[num_enq], num - num_enq); + } + + return ret; } static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], @@ -805,17 +836,26 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], * priorities. Stop scheduling queue when pktio * has been stopped. */ if (pktin) { - int num_pkt = poll_pktin(qi); + int atomic = queue_is_atomic(qi); + int num_pkt = poll_pktin(qi, atomic); - if (odp_likely(num_pkt >= 0)) { + if (odp_unlikely(num_pkt < 0)) + continue; + + if (num_pkt == 0 || !atomic) { ring_enq(ring, RING_MASK, qi); break; } - } - /* Remove empty queue from scheduling. Continue - * scheduling the same priority queue. */ - continue; + /* Process packets from an atomic queue + * right away */ + num = num_pkt; + } else { + /* Remove empty queue from scheduling. + * Continue scheduling the same priority + * queue. */ + continue; + } } handle = queue_from_index(qi); diff --git a/platform/linux-generic/odp_schedule_iquery.c b/platform/linux-generic/odp_schedule_iquery.c index 1a82f48da..ddd97bea7 100644 --- a/platform/linux-generic/odp_schedule_iquery.c +++ b/platform/linux-generic/odp_schedule_iquery.c @@ -674,9 +674,9 @@ static inline void pktio_poll_input(void) cmd = &sched->pktio_poll.commands[index]; /* Poll packet input */ - if (odp_unlikely(sched_cb_pktin_poll(cmd->pktio, - cmd->count, - cmd->pktin))) { + if (odp_unlikely(sched_cb_pktin_poll_old(cmd->pktio, + cmd->count, + cmd->pktin))) { /* Pktio stopped or closed. Remove poll * command and call stop_finalize when all * commands of the pktio has been removed. diff --git a/platform/linux-generic/odp_schedule_sp.c b/platform/linux-generic/odp_schedule_sp.c index 50390274b..84d16d3c1 100644 --- a/platform/linux-generic/odp_schedule_sp.c +++ b/platform/linux-generic/odp_schedule_sp.c @@ -524,8 +524,9 @@ static int schedule_multi(odp_queue_t *from, uint64_t wait, cmd = sched_cmd(); if (cmd && cmd->s.type == CMD_PKTIO) { - if (sched_cb_pktin_poll(cmd->s.index, cmd->s.num_pktin, - cmd->s.pktin_idx)) { + if (sched_cb_pktin_poll_old(cmd->s.index, + cmd->s.num_pktin, + cmd->s.pktin_idx)) { /* Pktio stopped or closed. */ sched_cb_pktio_stop_finalize(cmd->s.index); } else { From patchwork Fri Mar 2 15:00:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 130557 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp1382423lja; Fri, 2 Mar 2018 07:14:40 -0800 (PST) X-Google-Smtp-Source: AG47ELt94cz4BmOP4f3H1xbKImLy3MyHg61WnZPvB9Xmlw6e9jlVwx1nAivZKGxBr2mqoRNDEGFs X-Received: by 10.200.6.74 with SMTP id e10mr8535930qth.166.1520003680265; Fri, 02 Mar 2018 07:14:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520003680; cv=none; d=google.com; s=arc-20160816; b=QHNVjjSpHNQzSSVz47YEA16USfcYuyhZX/W9FLUcn5Bwr7b2JurMIGoK+15eL05rLe RABhmbc81azWJRRS+Wo5+RNnOG953Q6RnWiL6jixkOBd47NCM/+jrhzvm4TImaw1Mgse BOKShSEPKXZD00k/bttQAGDpKzszSoHpSlWRnXNYbkhE6HjZqDZ7AJb5a+3bCR56Tl42 mrqjirEAUZsqQsSmnPUGOLEP10xfaXoQj9p5BWsssXKyKgtzhhe5BtvT5VNhUc42xW9t gx04rxrSQVmQTCDithalqUF7QHmHDkZJbMLIH+3yAJAeJm80s18nNdInBucVvP9isb5i URMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=w+byxeHrWBN2tRObBwevJXKH3SGVDvLbccNy68/kIOs=; b=PNN9lBEsubwubsRkLXJtRdGBpFkkqdQaf8N9INWyWVqQheiH71e6fD5vHJ2QHZ4F5M pqZa0DxkuZAATPsNhTO2vlw2pJceMZ6WlvMhYXaBkiXrpnFoKAaIjaV/8sP+KpdCIcpa +KsjvuFaYXGgEDgnaP//dj+Ly3Q4Ijo0qDEWye7VsNA0x5BVfYS5lgtLUxLmlnfV4RvV m1w5EkLQVTIuBdZz/3d0YTNQKSD50C8JZLoW1YDPS6yDT6lAF1utWm8x1MCqxvVcOveK lFX0+ia18EKEjRn31afOkOn/H+dt8IfFdRG8262cX0RD3az5IR8WeE7JqDJdk0LYLoQ4 UdnQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id w193si6445493qka.157.2018.03.02.07.14.40; Fri, 02 Mar 2018 07:14:40 -0800 (PST) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id EA0AC627AA; Fri, 2 Mar 2018 15:14:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id EF56E617C9; Fri, 2 Mar 2018 15:01:40 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 029C26077E; Fri, 2 Mar 2018 15:00:37 +0000 (UTC) Received: from forward102p.mail.yandex.net (forward102p.mail.yandex.net [77.88.28.102]) by lists.linaro.org (Postfix) with ESMTPS id 96B7561778 for ; Fri, 2 Mar 2018 15:00:21 +0000 (UTC) Received: from mxback15j.mail.yandex.net (mxback15j.mail.yandex.net [IPv6:2a02:6b8:0:1619::91]) by forward102p.mail.yandex.net (Yandex) with ESMTP id 9B4FB43014EE for ; Fri, 2 Mar 2018 18:00:19 +0300 (MSK) Received: from smtp2j.mail.yandex.net (smtp2j.mail.yandex.net [2a02:6b8:0:801::ac]) by mxback15j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id yqA50JH0Lq-0Jmmm9vj; Fri, 02 Mar 2018 18:00:19 +0300 Received: by smtp2j.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id ChbgkO0UiZ-0I2GMw9V; Fri, 02 Mar 2018 18:00:18 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 2 Mar 2018 18:00:12 +0300 Message-Id: <1520002815-30682-4-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520002815-30682-1-git-send-email-odpbot@yandex.ru> References: <1520002815-30682-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 504 Subject: [lng-odp] [PATCH v2 3/6] linux-gen: queue: enqueue may fail X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Drop events when queue enqueue fails. Enqueue failure is more likely now when queue has limited size. Signed-off-by: Petri Savolainen --- /** Email created from pull request 504 (psavol:master-sched-optim-2) ** https://github.com/Linaro/odp/pull/504 ** Patch: https://github.com/Linaro/odp/pull/504.patch ** Base sha: e1c0e4570a45d05dd9f2e8e052ce71164209d112 ** Merge commit sha: 964132736e0785222be184065d6ac73121cd46ac **/ platform/linux-generic/odp_packet_io.c | 47 +++++++++++++++++++++++++--- platform/linux-generic/odp_schedule_basic.c | 14 +++++++-- platform/linux-generic/odp_schedule_iquery.c | 13 ++++++-- 3 files changed, 65 insertions(+), 9 deletions(-) diff --git a/platform/linux-generic/odp_packet_io.c b/platform/linux-generic/odp_packet_io.c index e76ff8140..f67181576 100644 --- a/platform/linux-generic/odp_packet_io.c +++ b/platform/linux-generic/odp_packet_io.c @@ -638,8 +638,22 @@ static odp_buffer_hdr_t *pktin_dequeue(queue_t q_int) if (pkts <= 0) return NULL; - if (pkts > 1) - queue_fn->enq_multi(q_int, &hdr_tbl[1], pkts - 1); + if (pkts > 1) { + int num_enq; + int num = pkts - 1; + + num_enq = queue_fn->enq_multi(q_int, &hdr_tbl[1], num); + + if (odp_unlikely(num_enq < num)) { + if (odp_unlikely(num_enq < 0)) + num_enq = 0; + + ODP_DBG("Interface %s dropped %i packets\n", + entry->s.name, num - num_enq); + buffer_free_multi(&hdr_tbl[num_enq + 1], num - num_enq); + } + } + buf_hdr = hdr_tbl[0]; return buf_hdr; } @@ -676,8 +690,21 @@ static int pktin_deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) for (j = 0; i < pkts; i++, j++) hdr_tbl[j] = hdr_tbl[i]; - if (j) - queue_fn->enq_multi(q_int, hdr_tbl, j); + if (j) { + int num_enq; + + num_enq = queue_fn->enq_multi(q_int, hdr_tbl, j); + + if (odp_unlikely(num_enq < j)) { + if (odp_unlikely(num_enq < 0)) + num_enq = 0; + + ODP_DBG("Interface %s dropped %i packets\n", + entry->s.name, j - num_enq); + buffer_free_multi(&buf_hdr[num_enq], j - num_enq); + } + } + return nbr; } @@ -763,6 +790,7 @@ int sched_cb_pktin_poll_old(int pktio_index, int num_queue, int index[]) for (idx = 0; idx < num_queue; idx++) { queue_t q_int; + int num_enq; num = pktin_recv_buf(entry, index[idx], hdr_tbl, QUEUE_MULTI_MAX); @@ -776,7 +804,16 @@ int sched_cb_pktin_poll_old(int pktio_index, int num_queue, int index[]) } q_int = entry->s.in_queue[index[idx]].queue_int; - queue_fn->enq_multi(q_int, hdr_tbl, num); + num_enq = queue_fn->enq_multi(q_int, hdr_tbl, num); + + if (odp_unlikely(num_enq < num)) { + if (odp_unlikely(num_enq < 0)) + num_enq = 0; + + ODP_DBG("Interface %s dropped %i packets\n", + entry->s.name, num - num_enq); + buffer_free_multi(&hdr_tbl[num_enq], num - num_enq); + } } return 0; diff --git a/platform/linux-generic/odp_schedule_basic.c b/platform/linux-generic/odp_schedule_basic.c index b251bdeef..3ea261fae 100644 --- a/platform/linux-generic/odp_schedule_basic.c +++ b/platform/linux-generic/odp_schedule_basic.c @@ -581,13 +581,23 @@ static inline void ordered_stash_release(void) for (i = 0; i < sched_local.ordered.stash_num; i++) { queue_entry_t *queue_entry; odp_buffer_hdr_t **buf_hdr; - int num; + int num, num_enq; queue_entry = sched_local.ordered.stash[i].queue_entry; buf_hdr = sched_local.ordered.stash[i].buf_hdr; num = sched_local.ordered.stash[i].num; - queue_fn->enq_multi(qentry_to_int(queue_entry), buf_hdr, num); + num_enq = queue_fn->enq_multi(qentry_to_int(queue_entry), + buf_hdr, num); + + /* Drop packets that were not enqueued */ + if (odp_unlikely(num_enq < num)) { + if (odp_unlikely(num_enq < 0)) + num_enq = 0; + + ODP_DBG("Dropped %i packets\n", num - num_enq); + buffer_free_multi(&buf_hdr[num_enq], num - num_enq); + } } sched_local.ordered.stash_num = 0; } diff --git a/platform/linux-generic/odp_schedule_iquery.c b/platform/linux-generic/odp_schedule_iquery.c index ddd97bea7..40f2e9fca 100644 --- a/platform/linux-generic/odp_schedule_iquery.c +++ b/platform/linux-generic/odp_schedule_iquery.c @@ -1136,13 +1136,22 @@ static inline void ordered_stash_release(void) for (i = 0; i < thread_local.ordered.stash_num; i++) { queue_entry_t *queue_entry; odp_buffer_hdr_t **buf_hdr; - int num; + int num, num_enq; queue_entry = thread_local.ordered.stash[i].queue_entry; buf_hdr = thread_local.ordered.stash[i].buf_hdr; num = thread_local.ordered.stash[i].num; - queue_fn->enq_multi(qentry_to_int(queue_entry), buf_hdr, num); + num_enq = queue_fn->enq_multi(qentry_to_int(queue_entry), + buf_hdr, num); + + if (odp_unlikely(num_enq < num)) { + if (odp_unlikely(num_enq < 0)) + num_enq = 0; + + ODP_DBG("Dropped %i packets\n", num - num_enq); + buffer_free_multi(&buf_hdr[num_enq], num - num_enq); + } } thread_local.ordered.stash_num = 0; } From patchwork Fri Mar 2 15:00:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 130551 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp1377272lja; Fri, 2 Mar 2018 07:09:50 -0800 (PST) X-Google-Smtp-Source: AG47ELsiBnsnQCYXSy2LDqotkaMR3EDLGy3bfSoKUEXaCV2Zqnaip53DkbF9/USGcTlXVLJgR7RN X-Received: by 10.55.22.24 with SMTP id g24mr8950328qkh.302.1520003390699; Fri, 02 Mar 2018 07:09:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520003390; cv=none; d=google.com; s=arc-20160816; b=RTOQkJ6sdtXyZms/N9mK3N2rVKTB+9xrJte3xG38UzBf/25MDm1J4tPq9B8MQ7RbaQ wdDapFbyQcyo3GNBCFBGXUgt0Md9aBNb8rblbcXpkJ3Dkty1pwT4ERyQmmJsyOMy0ULn Vle3X6/AB8yQ7e51tU/eriDmOhQu7SpjiogPzun4xZCOkI65qwwpcsoap6vbhKqb4Ljn O0VbdzGGkj98xLGwT2Shr5v0HVNsxGPgNLB9+UfV7bzqfmQoJqPcQp9WroBqTHTvF6Fd MK+Zg2Isx3ErfSbdsYGOh73bmdDbrNacsQe8xgCtk1kMTFfS180oJ8hzTTFPC79Bmta1 eO0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=YeHllEv3/aDSTppwrYu6J+kLSDNXbOZF99XejXI4r60=; b=G6XiC3sL3LygEdvdm7mbj2yCqB6Xj+8XNSIETeJZrrSJgD9hY0FRt/+f36EELSHvRV L6N/cUdYgkjW+N7yGaBa+X52h7D2NEfMvRltc9FCX6Xy6rbqvR6i/7anJqmbIfw8LjbX PJ+Ao8qqagnb7MRwoUiupWvtS3bikOmjYCyyNmdj39MYLAfPsT43jcryw7No8hR8JoYn XMhGJ+xIHGaRUNYiM2WaJN/oMu1CbtapfTgGIrl1CElXpQ3MINgwzrp4CTkVbHBZcStz Z3BfYae+grgGfQjzEwNw1PxQkOu09HWdB9E6xkLZd7OLnGd/NKSrQxqvIyPJEuto68k0 P+GQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id g15si6988683qtk.283.2018.03.02.07.09.50; Fri, 02 Mar 2018 07:09:50 -0800 (PST) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 5632B62791; Fri, 2 Mar 2018 15:09:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H2 autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 53A2861103; Fri, 2 Mar 2018 15:01:09 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 7A0EF6178F; Fri, 2 Mar 2018 15:00:29 +0000 (UTC) Received: from forward104j.mail.yandex.net (forward104j.mail.yandex.net [5.45.198.247]) by lists.linaro.org (Postfix) with ESMTPS id 6A2A061768 for ; Fri, 2 Mar 2018 15:00:21 +0000 (UTC) Received: from mxback3g.mail.yandex.net (mxback3g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:164]) by forward104j.mail.yandex.net (Yandex) with ESMTP id 56B154514A for ; Fri, 2 Mar 2018 18:00:20 +0300 (MSK) Received: from smtp2j.mail.yandex.net (smtp2j.mail.yandex.net [2a02:6b8:0:801::ac]) by mxback3g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id DXsEzutEFI-0Ksu2G2q; Fri, 02 Mar 2018 18:00:20 +0300 Received: by smtp2j.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id ChbgkO0UiZ-0J2K3euL; Fri, 02 Mar 2018 18:00:19 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 2 Mar 2018 18:00:13 +0300 Message-Id: <1520002815-30682-5-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520002815-30682-1-git-send-email-odpbot@yandex.ru> References: <1520002815-30682-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 504 Subject: [lng-odp] [PATCH v2 4/6] linux-gen: sched: optimize parallel packet input queue throughput X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Extend direct packet input processing to parallel queues. Parallel queues do not quarantee ordering, so also those can pass packets directly to application and (potentially) stash some per thread. Signed-off-by: Petri Savolainen --- /** Email created from pull request 504 (psavol:master-sched-optim-2) ** https://github.com/Linaro/odp/pull/504 ** Patch: https://github.com/Linaro/odp/pull/504.patch ** Base sha: e1c0e4570a45d05dd9f2e8e052ce71164209d112 ** Merge commit sha: 964132736e0785222be184065d6ac73121cd46ac **/ platform/linux-generic/odp_schedule_basic.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/platform/linux-generic/odp_schedule_basic.c b/platform/linux-generic/odp_schedule_basic.c index 3ea261fae..d662bd6a4 100644 --- a/platform/linux-generic/odp_schedule_basic.c +++ b/platform/linux-generic/odp_schedule_basic.c @@ -846,19 +846,19 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], * priorities. Stop scheduling queue when pktio * has been stopped. */ if (pktin) { - int atomic = queue_is_atomic(qi); - int num_pkt = poll_pktin(qi, atomic); + int stash = !ordered; + int num_pkt = poll_pktin(qi, stash); if (odp_unlikely(num_pkt < 0)) continue; - if (num_pkt == 0 || !atomic) { + if (num_pkt == 0 || !stash) { ring_enq(ring, RING_MASK, qi); break; } - /* Process packets from an atomic queue - * right away */ + /* Process packets from an atomic or + * parallel queue right away. */ num = num_pkt; } else { /* Remove empty queue from scheduling. @@ -868,12 +868,6 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], } } - handle = queue_from_index(qi); - sched_local.num = num; - sched_local.index = 0; - sched_local.queue = handle; - ret = copy_events(out_ev, max_num); - if (ordered) { uint64_t ctx; odp_atomic_u64_t *next_ctx; @@ -895,6 +889,12 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], ring_enq(ring, RING_MASK, qi); } + handle = queue_from_index(qi); + sched_local.num = num; + sched_local.index = 0; + sched_local.queue = handle; + ret = copy_events(out_ev, max_num); + /* Output the source queue handle */ if (out_queue) *out_queue = handle; From patchwork Fri Mar 2 15:00:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 130558 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp1387481lja; Fri, 2 Mar 2018 07:19:36 -0800 (PST) X-Google-Smtp-Source: AG47ELtQnsc1T/yZWvKn/Fh7zBVTPQoipwbw+QqWBYQLi1RH/BwuqkPupkRQYTmePYiYpFFybABw X-Received: by 10.13.202.1 with SMTP id m1mr3798969ywd.67.1520003976050; Fri, 02 Mar 2018 07:19:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520003976; cv=none; d=google.com; s=arc-20160816; b=pN9yHlSNnMx2rWWK17USmVjK+4x7gb7f3nwaKu4e1gIezuQgm5uQmfsIVDIdroGoYN MoI8QpBRmhVG4zYmbqLW3nJYtdZBrrc+IPKFBoRaYfZlxP27+2KwyktYxIC1laZEMuV4 8zMQdfmA9tpUkz7OwfhG0z42R/53x8AR5N83bgsXH9xYvQ742Nc1aU8PDjzvF6FuMRlr fQBIzYIUI0cHdIhgPrzbI3GOiB7o/JihziEelGCW2T50t/UzsGogzXurW+WVgHts5mk3 srmq5/JAhaihreSDuxYFbQXwxICV6b7QgEE38zdH07ZY4tkK/0O94951COtC/sXbUYjb aKIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=W0KT2esX8OdVd3dT8AyQoWe3RE1HupNVgMiKuMCtnKA=; b=vnYbj9SZSqWMxpOR1/2cQnj9WAsSL/kkvK1NphMLyi8Grw2puoI7w+MdRorAyVB5fJ m7a7a8hjjacRaXAEt/X4IZzQ5Bh59zYwIlrRBtItsY7B4+D2/8PAkvi0qGOqAFB+u9c7 EUE17Fp8hePMhHar/Nf0m2QmbbFT/yBHD2MA778Ce6zPKWad40PIUfacMcg0O/NYaHEU +PD9/Joyny2IxIJnf1HqSOZSbycUeSSpwC3TYm277G42qCjtnw5Qr/Hw9l6egJcjbzUN 1m2xJEZo9AqwGqfb61t7VV9k+MoLJpru1W2wtqvDfvTAAmxAemze09dasgtxDx1VTKTQ TSjw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id p7si115518qkp.395.2018.03.02.07.19.35; Fri, 02 Mar 2018 07:19:36 -0800 (PST) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 8F6B0627CF; Fri, 2 Mar 2018 15:19:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H2 autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 0DC296276F; Fri, 2 Mar 2018 15:04:26 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 6BF3A6276F; Fri, 2 Mar 2018 15:04:08 +0000 (UTC) Received: from forward105p.mail.yandex.net (forward105p.mail.yandex.net [77.88.28.108]) by lists.linaro.org (Postfix) with ESMTPS id 181A8607C4 for ; Fri, 2 Mar 2018 15:00:35 +0000 (UTC) Received: from mxback1g.mail.yandex.net (mxback1g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:162]) by forward105p.mail.yandex.net (Yandex) with ESMTP id 38C9140815FF for ; Fri, 2 Mar 2018 18:00:21 +0300 (MSK) Received: from smtp2j.mail.yandex.net (smtp2j.mail.yandex.net [2a02:6b8:0:801::ac]) by mxback1g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id BZFp7SbrJS-0LKGANjp; Fri, 02 Mar 2018 18:00:21 +0300 Received: by smtp2j.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id ChbgkO0UiZ-0K2mhhZ6; Fri, 02 Mar 2018 18:00:20 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 2 Mar 2018 18:00:14 +0300 Message-Id: <1520002815-30682-6-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520002815-30682-1-git-send-email-odpbot@yandex.ru> References: <1520002815-30682-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 504 Subject: [lng-odp] [PATCH v2 5/6] linux-gen: sched: use stash prefix X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Use consistently stash_ prefix for thread local variables related to event stashing. Signed-off-by: Petri Savolainen --- /** Email created from pull request 504 (psavol:master-sched-optim-2) ** https://github.com/Linaro/odp/pull/504 ** Patch: https://github.com/Linaro/odp/pull/504.patch ** Base sha: e1c0e4570a45d05dd9f2e8e052ce71164209d112 ** Merge commit sha: 964132736e0785222be184065d6ac73121cd46ac **/ platform/linux-generic/odp_schedule_basic.c | 59 +++++++++++++++-------------- 1 file changed, 30 insertions(+), 29 deletions(-) diff --git a/platform/linux-generic/odp_schedule_basic.c b/platform/linux-generic/odp_schedule_basic.c index d662bd6a4..b3c913bb6 100644 --- a/platform/linux-generic/odp_schedule_basic.c +++ b/platform/linux-generic/odp_schedule_basic.c @@ -114,13 +114,14 @@ ODP_STATIC_ASSERT(sizeof(lock_called_t) == sizeof(uint32_t), /* Scheduler local data */ typedef struct { int thr; - int num; - int index; + int stash_num; + int stash_index; int pause; uint16_t round; - uint32_t queue_index; - odp_queue_t queue; - odp_event_t ev_stash[MAX_DEQ]; + uint32_t stash_qi; + odp_queue_t stash_queue; + odp_event_t stash_ev[MAX_DEQ]; + struct { /* Source queue index */ uint32_t src_queue; @@ -228,9 +229,9 @@ static void sched_local_init(void) memset(&sched_local, 0, sizeof(sched_local_t)); - sched_local.thr = odp_thread_id(); - sched_local.queue = ODP_QUEUE_INVALID; - sched_local.queue_index = PRIO_QUEUE_EMPTY; + sched_local.thr = odp_thread_id(); + sched_local.stash_queue = ODP_QUEUE_INVALID; + sched_local.stash_qi = PRIO_QUEUE_EMPTY; sched_local.ordered.src_queue = NULL_INDEX; id = sched_local.thr & (QUEUES_PER_PRIO - 1); @@ -364,7 +365,7 @@ static int schedule_init_local(void) static int schedule_term_local(void) { - if (sched_local.num) { + if (sched_local.stash_num) { ODP_ERR("Locally pre-scheduled events exist.\n"); return -1; } @@ -536,9 +537,9 @@ static void schedule_pktio_start(int pktio_index, int num_pktin, static void schedule_release_atomic(void) { - uint32_t qi = sched_local.queue_index; + uint32_t qi = sched_local.stash_qi; - if (qi != PRIO_QUEUE_EMPTY && sched_local.num == 0) { + if (qi != PRIO_QUEUE_EMPTY && sched_local.stash_num == 0) { int grp = sched->queue[qi].grp; int prio = sched->queue[qi].prio; int queue_per_prio = sched->queue[qi].queue_per_prio; @@ -546,7 +547,7 @@ static void schedule_release_atomic(void) /* Release current atomic queue */ ring_enq(ring, RING_MASK, qi); - sched_local.queue_index = PRIO_QUEUE_EMPTY; + sched_local.stash_qi = PRIO_QUEUE_EMPTY; } } @@ -634,7 +635,7 @@ static void schedule_release_ordered(void) queue_index = sched_local.ordered.src_queue; - if (odp_unlikely((queue_index == NULL_INDEX) || sched_local.num)) + if (odp_unlikely((queue_index == NULL_INDEX) || sched_local.stash_num)) return; release_ordered(); @@ -648,14 +649,14 @@ static inline void schedule_release_context(void) schedule_release_atomic(); } -static inline int copy_events(odp_event_t out_ev[], unsigned int max) +static inline int copy_from_stash(odp_event_t out_ev[], unsigned int max) { int i = 0; - while (sched_local.num && max) { - out_ev[i] = sched_local.ev_stash[sched_local.index]; - sched_local.index++; - sched_local.num--; + while (sched_local.stash_num && max) { + out_ev[i] = sched_local.stash_ev[sched_local.stash_index]; + sched_local.stash_index++; + sched_local.stash_num--; max--; i++; } @@ -743,7 +744,7 @@ static inline int poll_pktin(uint32_t qi, int stash) if (stash) { for (i = 0; i < num; i++) - sched_local.ev_stash[i] = event_from_buf_hdr(b_hdr[i]); + sched_local.stash_ev[i] = event_from_buf_hdr(b_hdr[i]); return num; } @@ -829,7 +830,7 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], pktin = queue_is_pktin(qi); - num = sched_cb_queue_deq_multi(qi, sched_local.ev_stash, + num = sched_cb_queue_deq_multi(qi, sched_local.stash_ev, max_deq, !pktin); if (num < 0) { @@ -883,17 +884,17 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], } else if (queue_is_atomic(qi)) { /* Hold queue during atomic access */ - sched_local.queue_index = qi; + sched_local.stash_qi = qi; } else { /* Continue scheduling the queue */ ring_enq(ring, RING_MASK, qi); } - handle = queue_from_index(qi); - sched_local.num = num; - sched_local.index = 0; - sched_local.queue = handle; - ret = copy_events(out_ev, max_num); + handle = queue_from_index(qi); + sched_local.stash_num = num; + sched_local.stash_index = 0; + sched_local.stash_queue = handle; + ret = copy_from_stash(out_ev, max_num); /* Output the source queue handle */ if (out_queue) @@ -918,11 +919,11 @@ static inline int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], uint16_t round; uint32_t epoch; - if (sched_local.num) { - ret = copy_events(out_ev, max_num); + if (sched_local.stash_num) { + ret = copy_from_stash(out_ev, max_num); if (out_queue) - *out_queue = sched_local.queue; + *out_queue = sched_local.stash_queue; return ret; } From patchwork Fri Mar 2 15:00:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 130556 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp1381591lja; Fri, 2 Mar 2018 07:14:00 -0800 (PST) X-Google-Smtp-Source: AG47ELtg16iB2ibEuCROkmDyeH1okdy/5iXhOPglgDXbpvxuLxgkA+is7d+R1r3SZ023FC33ksfF X-Received: by 10.237.47.69 with SMTP id l63mr9022506qtd.68.1520003640413; Fri, 02 Mar 2018 07:14:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520003640; cv=none; d=google.com; s=arc-20160816; b=MtV9/yTe23x7qy0zXtBJi2AMLPOKb9cWQa6NcgS8qeumtrfqLi1nWJ7EsDQlsfkkpT PQEdYiOuWf3Y5pHCCHQGVfE5EgGJda9+gG3ffVqwgCh4ku1kbXKf7iNJ+vaGwKDRVPiw 4mQaI+d4s0m7YCBC9jSoLkWXpUf9hFTI1dC3eCqGroqG3HSmHNxAEHIjklgyb3gax0Vv 5w8iUsM/hFmR/GethjIIi/6WcSZLDcB1xqqeokxF00/gEzTZaKfGn8u1/1zz+UuYS7XT WZZk5NDVC5yg8YbOeB2CClLZgsLcS4HuMYsOwEoJhpmu3ywGmEnlpgL6hBdleIPdAbgq +wJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=zotSvdeDIAkK/lrAOQd8uTTuQ6ivuiCO92gLA365R60=; b=JsE09ro5UkIAgOLNXme+lvJ2fSAhTdnFksMUJQN6yYU3GBmixoWKhgYVboWufJis7u t0V/izVp8uL+mWtMuzU4tv0nbhyDl/RQ3My+Ro3+gC2VpAzJV5tQNqXcTBG6MCKn25JI h+M8VQmtu2kgndpy6nn3yuU5YGvajBL03+8ISHEQXmEQQAWnJ32XuM8f8CWkwjgBuNz1 9geKZqFd5u+D4F4ZXShiQ1SwcV6EhC6AuJVEGpoQfBjmN9lGTsY2daa6xWzMMKnLufsv oOaKwubVF9rOdd6VLAEdlQWiNF5Ci9zzbUl0mZdbvzUopMJHbN4DqXe9YIBBTeprJGOW fT1A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id h126si6541526qkf.282.2018.03.02.07.14.00; Fri, 02 Mar 2018 07:14:00 -0800 (PST) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 0AA0C627AD; Fri, 2 Mar 2018 15:14:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 9F3D0617B1; Fri, 2 Mar 2018 15:01:37 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 03F6B607E0; Fri, 2 Mar 2018 15:00:35 +0000 (UTC) Received: from forward106p.mail.yandex.net (forward106p.mail.yandex.net [77.88.28.109]) by lists.linaro.org (Postfix) with ESMTPS id 0AD146177D for ; Fri, 2 Mar 2018 15:00:24 +0000 (UTC) Received: from mxback3o.mail.yandex.net (mxback3o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::1d]) by forward106p.mail.yandex.net (Yandex) with ESMTP id E1C3C2D82EB0 for ; Fri, 2 Mar 2018 18:00:21 +0300 (MSK) Received: from smtp2j.mail.yandex.net (smtp2j.mail.yandex.net [2a02:6b8:0:801::ac]) by mxback3o.mail.yandex.net (nwsmtp/Yandex) with ESMTP id 3j7gwLfIjP-0Lw8ef0m; Fri, 02 Mar 2018 18:00:21 +0300 Received: by smtp2j.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id ChbgkO0UiZ-0L2mhvoS; Fri, 02 Mar 2018 18:00:21 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 2 Mar 2018 18:00:15 +0300 Message-Id: <1520002815-30682-7-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520002815-30682-1-git-send-email-odpbot@yandex.ru> References: <1520002815-30682-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 504 Subject: [lng-odp] [PATCH v2 6/6] linux-gen: sched: optimize local variable layout X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Pack local variables struct and bring commonly used variables into the head of the struct. Signed-off-by: Petri Savolainen --- /** Email created from pull request 504 (psavol:master-sched-optim-2) ** https://github.com/Linaro/odp/pull/504 ** Patch: https://github.com/Linaro/odp/pull/504.patch ** Base sha: e1c0e4570a45d05dd9f2e8e052ce71164209d112 ** Merge commit sha: 964132736e0785222be184065d6ac73121cd46ac **/ platform/linux-generic/odp_schedule_basic.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/platform/linux-generic/odp_schedule_basic.c b/platform/linux-generic/odp_schedule_basic.c index b3c913bb6..cd20b39dd 100644 --- a/platform/linux-generic/odp_schedule_basic.c +++ b/platform/linux-generic/odp_schedule_basic.c @@ -114,14 +114,20 @@ ODP_STATIC_ASSERT(sizeof(lock_called_t) == sizeof(uint32_t), /* Scheduler local data */ typedef struct { int thr; - int stash_num; - int stash_index; - int pause; + uint16_t stash_num; + uint16_t stash_index; + uint16_t pause; uint16_t round; uint32_t stash_qi; odp_queue_t stash_queue; odp_event_t stash_ev[MAX_DEQ]; + uint32_t grp_epoch; + int num_grp; + uint8_t grp[NUM_SCHED_GRPS]; + uint8_t weight_tbl[WEIGHT_TBL_SIZE]; + uint8_t grp_weight[WEIGHT_TBL_SIZE]; + struct { /* Source queue index */ uint32_t src_queue; @@ -133,12 +139,6 @@ typedef struct { ordered_stash_t stash[MAX_ORDERED_STASH]; } ordered; - uint32_t grp_epoch; - int num_grp; - uint8_t grp[NUM_SCHED_GRPS]; - uint8_t weight_tbl[WEIGHT_TBL_SIZE]; - uint8_t grp_weight[WEIGHT_TBL_SIZE]; - } sched_local_t; /* Priority queue */