From patchwork Thu Sep 14 21:00:04 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 112659 Delivered-To: patch@linaro.org Received: by 10.80.163.150 with SMTP id s22csp1050485edb; Thu, 14 Sep 2017 14:17:44 -0700 (PDT) X-Google-Smtp-Source: AOwi7QAxv5SHpi288bGfSj4SczGJ+22m4WDsdkdVlJ4E81neUhBOCUl8HDeWJRKRERRKeMHmQN/Z X-Received: by 10.200.27.235 with SMTP id m40mr34869179qtk.228.1505423864173; Thu, 14 Sep 2017 14:17:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1505423864; cv=none; d=google.com; s=arc-20160816; b=aZhx+szs4zScVc+hJ+rjc2MtQuXO9NB0s/8s8cPkApBIsg6GMpRmRc1FEpUixTkh9h 2vXZ6uCRYBVnIL1DN5SW+DlYgDsUkL6dfdmkbqGPhgYmPZNUS6bYNKLB0n6hJpvY/fuJ do8asOZea7k2YoSSLGxgFp2V7b4YnUMIrpcG4ddze6oAf4Y5zXZXojbetIeWaa3XBAMG hmc+FOb5XqGnO4MmSkL5KY7pplNORIPEhKD+Lrd5seLnttxrPdY8KOk2B7/niD3MB43A 5+3F2SP/MK0q1x0C3wpJ2ou2Os+6ji6AZyz6P2qs1LYP0qf3Tf8pEOARR5CvIqxkjGrk UJRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=vLJGIoL7AVimL7uCqWyU9bzErsB81IzZPqP45DdqhCk=; b=ZU9zQ8mg2yBNHs2y/qbjK60haTuW8yLyeuOPh+yxBUM5OocnGb7hZrvonEoQz+WBgb /Io73QEt/M5r+UpdjPbJbiHFsi/uI+2TVocc+FdfMrcI6QhNvhRINrmBPohnQ86uDQeE EH+XaRKYspMOb8N+RyAvXVsfBZOHbVluasmCe9hQp11W1jy/s5J4pb4B6uC5lUt5z5n8 SRYlYWre1r+Y7phE5hV7UaxWSEQIWLB3aKSmUC25gA7k49lQUISsJ3uI4aIBMAt33qfR dvK4NiXg89Moy8tZVmEqWyZkVm90yuWBbZ3SBwXRiYG6Q2FoY+f8n1Jta2++ek2oNmoP 38bg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id u90si18446301qku.129.2017.09.14.14.17.43; Thu, 14 Sep 2017 14:17:44 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id AD0BB608E5; Thu, 14 Sep 2017 21:17:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW,URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 4687660C60; Thu, 14 Sep 2017 21:03:09 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 0EA0A60C5E; Thu, 14 Sep 2017 21:02:49 +0000 (UTC) Received: from forward102p.mail.yandex.net (forward102p.mail.yandex.net [77.88.28.102]) by lists.linaro.org (Postfix) with ESMTPS id 0961960966 for ; Thu, 14 Sep 2017 21:00:31 +0000 (UTC) Received: from mxback16j.mail.yandex.net (mxback16j.mail.yandex.net [IPv6:2a02:6b8:0:1619::92]) by forward102p.mail.yandex.net (Yandex) with ESMTP id 96C144301665 for ; Fri, 15 Sep 2017 00:00:29 +0300 (MSK) Received: from smtp1j.mail.yandex.net (smtp1j.mail.yandex.net [2a02:6b8:0:801::ab]) by mxback16j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id 9dcnG91Bq0-0TxenQE2; Fri, 15 Sep 2017 00:00:29 +0300 Received: by smtp1j.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id wNwc7KZcGv-0Tm0SksX; Fri, 15 Sep 2017 00:00:29 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 15 Sep 2017 00:00:04 +0300 Message-Id: <1505422809-5632-15-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1505422809-5632-1-git-send-email-odpbot@yandex.ru> References: <1505422809-5632-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 179 Subject: [lng-odp] [PATCH API-NEXT v1 14/19] linux-gen: packet: implement static references X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Implemented static references with a reference counter. Counter is zero when not used (reference API not used). Signed-off-by: Petri Savolainen Reviewed-by: Bill Fischofer Signed-off-by: Maxim Uvarov --- /** Email created from pull request 179 (muvarov:api-next) ** https://github.com/Linaro/odp/pull/179 ** Patch: https://github.com/Linaro/odp/pull/179.patch ** Base sha: 6b6253c30f88c80bf632436ff06c1b000860a2f1 ** Merge commit sha: ada61f5ba5f940d03a95893940c21028d4c75d19 **/ .../linux-generic/include/odp_buffer_internal.h | 21 ++-- .../linux-generic/include/odp_packet_internal.h | 5 +- platform/linux-generic/odp_packet.c | 111 +++++++++++++++++++-- platform/linux-generic/odp_pool.c | 2 + 4 files changed, 121 insertions(+), 18 deletions(-) diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index 0c873d2d2..cd067a08b 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -62,11 +62,16 @@ struct odp_buffer_hdr_t { /* Last header of the segment list */ void *last_seg; - /* Initial buffer data pointer and length */ + /* Initial buffer data pointer */ uint8_t *base_data; - uint8_t *buf_end; - /* --- 40 bytes --- */ + /* Reference count */ + odp_atomic_u32_t ref_cnt; + + /* Event type. Maybe different than pool type (crypto compl event) */ + int8_t event_type; + + /* --- 37 bytes --- */ /* Segments */ seg_entry_t seg[CONFIG_PACKET_MAX_SEGS]; @@ -93,14 +98,17 @@ struct odp_buffer_hdr_t { /* Pool pointer */ void *pool_ptr; + /* Initial buffer tail pointer */ + uint8_t *buf_end; + /* User area pointer */ void *uarea_addr; /* User area size */ uint32_t uarea_size; - /* Event type. Maybe different than pool type (crypto compl event) */ - int8_t event_type; + /* Max data size */ + uint32_t size; /* ipc mapped process can not walk over pointers, * offset has to be used */ @@ -110,9 +118,6 @@ struct odp_buffer_hdr_t { * inlining */ odp_pool_t pool_hdl; - /* Max data size */ - uint32_t size; - /* Data or next header */ uint8_t data[0]; } ODP_ALIGNED_CACHE; diff --git a/platform/linux-generic/include/odp_packet_internal.h b/platform/linux-generic/include/odp_packet_internal.h index d8e5766ce..d8d4584a7 100644 --- a/platform/linux-generic/include/odp_packet_internal.h +++ b/platform/linux-generic/include/odp_packet_internal.h @@ -116,11 +116,13 @@ typedef struct { packet_parser_t p; + uint32_t frame_len; + odp_pktio_t input; - uint32_t frame_len; uint32_t headroom; uint32_t tailroom; + uint32_t shared_len; /* * Members below are not initialized by packet_init() @@ -217,6 +219,7 @@ static inline void packet_init(odp_packet_hdr_t *pkt_hdr, uint32_t len) * segment occupied by the allocated length. */ pkt_hdr->frame_len = len; + pkt_hdr->shared_len = 0; pkt_hdr->headroom = CONFIG_PACKET_HEADROOM; pkt_hdr->tailroom = CONFIG_PACKET_MAX_SEG_LEN - seg_len + CONFIG_PACKET_TAILROOM; diff --git a/platform/linux-generic/odp_packet.c b/platform/linux-generic/odp_packet.c index a43307374..1a21ab0c6 100644 --- a/platform/linux-generic/odp_packet.c +++ b/platform/linux-generic/odp_packet.c @@ -494,6 +494,76 @@ static inline void copy_buf_hdr(odp_packet_hdr_t *pkt_hdr, int first, int num, } } +static inline void buffer_ref_inc(odp_buffer_hdr_t *buf_hdr) +{ + uint32_t ref_cnt = odp_atomic_load_u32(&buf_hdr->ref_cnt); + + /* First count increment after alloc */ + if (odp_likely(ref_cnt) == 0) + odp_atomic_store_u32(&buf_hdr->ref_cnt, 2); + else + odp_atomic_inc_u32(&buf_hdr->ref_cnt); +} + +static inline uint32_t buffer_ref_dec(odp_buffer_hdr_t *buf_hdr) +{ + return odp_atomic_fetch_dec_u32(&buf_hdr->ref_cnt); +} + +static inline uint32_t buffer_ref(odp_buffer_hdr_t *buf_hdr) +{ + return odp_atomic_load_u32(&buf_hdr->ref_cnt); +} + +static inline int is_multi_ref(uint32_t ref_cnt) +{ + return (ref_cnt > 1); +} + +static inline void packet_ref_inc(odp_packet_hdr_t *pkt_hdr) +{ + seg_entry_t *seg; + int i; + int seg_count = pkt_hdr->buf_hdr.segcount; + odp_packet_hdr_t *hdr = pkt_hdr; + uint8_t idx = 0; + + for (i = 0; i < seg_count; i++) { + seg = seg_entry_next(&hdr, &idx); + buffer_ref_inc(seg->hdr); + } +} + +static inline void packet_free_multi(odp_buffer_hdr_t *hdr[], int num) +{ + int i; + uint32_t ref_cnt; + int num_ref = 0; + + for (i = 0; i < num; i++) { + /* Zero when reference API has not been used */ + ref_cnt = buffer_ref(hdr[i]); + + if (odp_unlikely(ref_cnt)) { + ref_cnt = buffer_ref_dec(hdr[i]); + + if (is_multi_ref(ref_cnt)) { + num_ref++; + continue; + } + } + + /* Skip references and pack to be freed headers to array head */ + if (odp_unlikely(num_ref)) + hdr[i - num_ref] = hdr[i]; + } + + num -= num_ref; + + if (odp_likely(num)) + buffer_free_multi(hdr, num); +} + static inline void free_all_segments(odp_packet_hdr_t *pkt_hdr, int num) { seg_entry_t *seg; @@ -506,7 +576,7 @@ static inline void free_all_segments(odp_packet_hdr_t *pkt_hdr, int num) buf_hdr[i] = seg->hdr; } - buffer_free_multi(buf_hdr, num); + packet_free_multi(buf_hdr, num); } static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, @@ -563,7 +633,7 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, pkt_hdr = new_hdr; - buffer_free_multi(buf_hdr, num); + packet_free_multi(buf_hdr, num); } else { /* Free last 'num' bufs. * First, find the last remaining header. */ @@ -578,7 +648,7 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, buf_hdr[i] = seg->hdr; } - buffer_free_multi(buf_hdr, num); + packet_free_multi(buf_hdr, num); /* Head segment remains, no need to copy or update majority * of the metadata. */ @@ -702,7 +772,7 @@ void odp_packet_free(odp_packet_t pkt) int num_seg = pkt_hdr->buf_hdr.segcount; if (odp_likely(CONFIG_PACKET_MAX_SEGS == 1 || num_seg == 1)) - buffer_free_multi((odp_buffer_hdr_t **)&hdl, 1); + packet_free_multi((odp_buffer_hdr_t **)&hdl, 1); else free_all_segments(pkt_hdr, num_seg); } @@ -710,7 +780,7 @@ void odp_packet_free(odp_packet_t pkt) void odp_packet_free_multi(const odp_packet_t pkt[], int num) { if (CONFIG_PACKET_MAX_SEGS == 1) { - buffer_free_multi((odp_buffer_hdr_t **)(uintptr_t)pkt, num); + packet_free_multi((odp_buffer_hdr_t **)(uintptr_t)pkt, num); } else { odp_buffer_hdr_t *buf_hdr[num * CONFIG_PACKET_MAX_SEGS]; int i; @@ -731,7 +801,7 @@ void odp_packet_free_multi(const odp_packet_t pkt[], int num) bufs += num_seg - 1; } - buffer_free_multi(buf_hdr, bufs); + packet_free_multi(buf_hdr, bufs); } } @@ -1942,7 +2012,12 @@ uint64_t odp_packet_seg_to_u64(odp_packet_seg_t hdl) odp_packet_t odp_packet_ref_static(odp_packet_t pkt) { - return odp_packet_copy(pkt, odp_packet_pool(pkt)); + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); + + packet_ref_inc(pkt_hdr); + pkt_hdr->shared_len = pkt_hdr->frame_len; + + return pkt; } odp_packet_t odp_packet_ref(odp_packet_t pkt, uint32_t offset) @@ -2004,14 +2079,32 @@ odp_packet_t odp_packet_ref_pkt(odp_packet_t pkt, uint32_t offset, int odp_packet_has_ref(odp_packet_t pkt) { - (void)pkt; + odp_buffer_hdr_t *buf_hdr; + seg_entry_t *seg; + int i; + uint32_t ref_cnt; + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); + int seg_count = pkt_hdr->buf_hdr.segcount; + odp_packet_hdr_t *hdr = pkt_hdr; + uint8_t idx = 0; + + for (i = 0; i < seg_count; i++) { + seg = seg_entry_next(&hdr, &idx); + buf_hdr = seg->hdr; + ref_cnt = buffer_ref(buf_hdr); + + if (is_multi_ref(ref_cnt)) + return 1; + } return 0; } uint32_t odp_packet_unshared_len(odp_packet_t pkt) { - return odp_packet_len(pkt); + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); + + return packet_len(pkt_hdr) - pkt_hdr->shared_len; } /* Include non-inlined versions of API functions */ diff --git a/platform/linux-generic/odp_pool.c b/platform/linux-generic/odp_pool.c index 47a39f5b5..2f65cb20c 100644 --- a/platform/linux-generic/odp_pool.c +++ b/platform/linux-generic/odp_pool.c @@ -277,6 +277,8 @@ static void init_buffers(pool_t *pool) buf_hdr->seg[0].data = &data[offset]; buf_hdr->seg[0].len = pool->data_size; + odp_atomic_init_u32(&buf_hdr->ref_cnt, 0); + /* Store base values for fast init */ buf_hdr->base_data = buf_hdr->seg[0].data; buf_hdr->buf_end = &data[offset + pool->data_size +