From patchwork Fri Sep 8 14:00:08 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 112089 Delivered-To: patch@linaro.org Received: by 10.37.128.210 with SMTP id c18csp1675645ybm; Fri, 8 Sep 2017 07:11:53 -0700 (PDT) X-Google-Smtp-Source: ADKCNb5BpsCEbhbgAopmEl0BePAxybg6sDuLhRrMVJmwoKYPU1baW9Iy6b9T1Th7jLDmLmKsT1JD X-Received: by 10.55.111.1 with SMTP id k1mr1284338qkc.15.1504879913585; Fri, 08 Sep 2017 07:11:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504879913; cv=none; d=google.com; s=arc-20160816; b=0QGwU34EPiHkY9so/wTFpfK/OcNzOMXU+hdaDoIBDI55AiL3+nXJfe433pjC67C+z9 ukmsn6a0IWjYvMHEJtKzDYL9hrFRTZLqhZFG+iuuQWlKIi9p0cZygYRheCROumx/bAai CIyuOuGnvXLmLtZdUrWhnRqGuj5RXdFC5/Wdhz46bS5QxWZU6JuOjBGZW2tuO/Iqtlrn tJpvtdgIMM2nqYAwkVFBJgVHIpddbbVF2bbwF2URUDqiNckB7ytHmeQAaScW+d6vvnsT 8GN0VCHGIqGflp2BJbM1mL5XHhCCaID03hyUdC0aITTK4wiFJLg1ppUFMDLE1tODLcmM yueg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=oClz0WnbpPB8IhaUjSNABgelCA+rv4rRkVg1p5Zyj7w=; b=dNi5s3sNTtHUHV7PR8/VlnuVq5h7IdsktlG3g1MmbJufPQZKeHntZ2fVFSGRA4UWNV iPUabdOj5Hq4OSUWCJBOnQLB7qnC6kooZtO+iPnqp3I2yQH8sAZYLVukutRt20bKisyi FYq0m48p3uauO3J6hbTH14YuaVbPMpxF/qTxGKJQlKrLAkrPudm1tNR+KQCKTNsSxn46 f2+kwxZsSIPEFJ5xamidsoqUC6IQ3CinIk6iwb/cW4N5NVWoucNeX2W6b3Jw5mq0l7cN dU/YUFPfBh5F1kybqyhdbY/PGrjpU8yWYPZl3MFDos1xnW6P989DFaFjcuH/iOE3bU1Y FHUA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id o19si2209590qtc.69.2017.09.08.07.11.53; Fri, 08 Sep 2017 07:11:53 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 3DA8260903; Fri, 8 Sep 2017 14:11:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 8ACE760C2D; Fri, 8 Sep 2017 14:01:23 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 1E5E4644B0; Fri, 8 Sep 2017 14:00:37 +0000 (UTC) Received: from forward101j.mail.yandex.net (forward101j.mail.yandex.net [5.45.198.241]) by lists.linaro.org (Postfix) with ESMTPS id AE6EF60A4D for ; Fri, 8 Sep 2017 14:00:24 +0000 (UTC) Received: from mxback10g.mail.yandex.net (mxback10g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:171]) by forward101j.mail.yandex.net (Yandex) with ESMTP id 638101243241 for ; Fri, 8 Sep 2017 17:00:23 +0300 (MSK) Received: from smtp2p.mail.yandex.net (smtp2p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:7]) by mxback10g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id A8OIoLaJra-0NamURvZ; Fri, 08 Sep 2017 17:00:23 +0300 Received: by smtp2p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id IwYlQdvFwh-0MKePN9G; Fri, 08 Sep 2017 17:00:22 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 8 Sep 2017 17:00:08 +0300 Message-Id: <1504879212-3689-8-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> References: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 170 Subject: [lng-odp] [PATCH v1 7/11] linux-gen: packet: implement static references X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Implemented static references with a reference counter. Counter is zero when not used (reference API not used). Signed-off-by: Petri Savolainen --- /** Email created from pull request 170 (psavol:master-packet-ref-rework) ** https://github.com/Linaro/odp/pull/170 ** Patch: https://github.com/Linaro/odp/pull/170.patch ** Base sha: 360c1d34b52a2356619b9290811862b9de41de00 ** Merge commit sha: e19cce86ab00ccd959d2c3c82c0786ec6da19ba7 **/ .../linux-generic/include/odp_buffer_internal.h | 21 +++-- .../linux-generic/include/odp_packet_internal.h | 5 +- platform/linux-generic/odp_packet.c | 92 +++++++++++++++++++--- platform/linux-generic/odp_pool.c | 2 + 4 files changed, 102 insertions(+), 18 deletions(-) diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index 0c873d2d2..cd067a08b 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -62,11 +62,16 @@ struct odp_buffer_hdr_t { /* Last header of the segment list */ void *last_seg; - /* Initial buffer data pointer and length */ + /* Initial buffer data pointer */ uint8_t *base_data; - uint8_t *buf_end; - /* --- 40 bytes --- */ + /* Reference count */ + odp_atomic_u32_t ref_cnt; + + /* Event type. Maybe different than pool type (crypto compl event) */ + int8_t event_type; + + /* --- 37 bytes --- */ /* Segments */ seg_entry_t seg[CONFIG_PACKET_MAX_SEGS]; @@ -93,14 +98,17 @@ struct odp_buffer_hdr_t { /* Pool pointer */ void *pool_ptr; + /* Initial buffer tail pointer */ + uint8_t *buf_end; + /* User area pointer */ void *uarea_addr; /* User area size */ uint32_t uarea_size; - /* Event type. Maybe different than pool type (crypto compl event) */ - int8_t event_type; + /* Max data size */ + uint32_t size; /* ipc mapped process can not walk over pointers, * offset has to be used */ @@ -110,9 +118,6 @@ struct odp_buffer_hdr_t { * inlining */ odp_pool_t pool_hdl; - /* Max data size */ - uint32_t size; - /* Data or next header */ uint8_t data[0]; } ODP_ALIGNED_CACHE; diff --git a/platform/linux-generic/include/odp_packet_internal.h b/platform/linux-generic/include/odp_packet_internal.h index d8e5766ce..d8d4584a7 100644 --- a/platform/linux-generic/include/odp_packet_internal.h +++ b/platform/linux-generic/include/odp_packet_internal.h @@ -116,11 +116,13 @@ typedef struct { packet_parser_t p; + uint32_t frame_len; + odp_pktio_t input; - uint32_t frame_len; uint32_t headroom; uint32_t tailroom; + uint32_t shared_len; /* * Members below are not initialized by packet_init() @@ -217,6 +219,7 @@ static inline void packet_init(odp_packet_hdr_t *pkt_hdr, uint32_t len) * segment occupied by the allocated length. */ pkt_hdr->frame_len = len; + pkt_hdr->shared_len = 0; pkt_hdr->headroom = CONFIG_PACKET_HEADROOM; pkt_hdr->tailroom = CONFIG_PACKET_MAX_SEG_LEN - seg_len + CONFIG_PACKET_TAILROOM; diff --git a/platform/linux-generic/odp_packet.c b/platform/linux-generic/odp_packet.c index 94fdffab3..cde77bb5f 100644 --- a/platform/linux-generic/odp_packet.c +++ b/platform/linux-generic/odp_packet.c @@ -491,6 +491,36 @@ static inline void copy_buf_hdr(odp_packet_hdr_t *pkt_hdr, int first, int num, } } +static inline void packet_free_multi(odp_buffer_hdr_t *hdr[], int num) +{ + int i; + uint32_t ref_cnt; + int num_ref = 0; + + for (i = 0; i < num; i++) { + /* Zero when reference API has not been used */ + ref_cnt = odp_atomic_load_u32(&hdr[i]->ref_cnt); + + if (odp_unlikely(ref_cnt)) { + ref_cnt = odp_atomic_fetch_dec_u32(&hdr[i]->ref_cnt); + + if (ref_cnt > 1) { + num_ref++; + continue; + } + } + + /* Skip references and pack to be freed headers to array head */ + if (odp_unlikely(num_ref)) + hdr[i - num_ref] = hdr[i]; + } + + num -= num_ref; + + if (odp_likely(num)) + buffer_free_multi(hdr, num); +} + static inline void free_all_segments(odp_packet_hdr_t *pkt_hdr, int num) { seg_entry_t *seg; @@ -503,7 +533,7 @@ static inline void free_all_segments(odp_packet_hdr_t *pkt_hdr, int num) buf_hdr[i] = seg->hdr; } - buffer_free_multi(buf_hdr, num); + packet_free_multi(buf_hdr, num); } static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, @@ -560,7 +590,7 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, pkt_hdr = new_hdr; - buffer_free_multi(buf_hdr, num); + packet_free_multi(buf_hdr, num); } else { /* Free last 'num' bufs. * First, find the last remaining header. */ @@ -575,7 +605,7 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, buf_hdr[i] = seg->hdr; } - buffer_free_multi(buf_hdr, num); + packet_free_multi(buf_hdr, num); /* Head segment remains, no need to copy or update majority * of the metadata. */ @@ -699,7 +729,7 @@ void odp_packet_free(odp_packet_t pkt) int num_seg = pkt_hdr->buf_hdr.segcount; if (odp_likely(CONFIG_PACKET_MAX_SEGS == 1 || num_seg == 1)) - buffer_free_multi((odp_buffer_hdr_t **)&hdl, 1); + packet_free_multi((odp_buffer_hdr_t **)&hdl, 1); else free_all_segments(pkt_hdr, num_seg); } @@ -707,7 +737,7 @@ void odp_packet_free(odp_packet_t pkt) void odp_packet_free_multi(const odp_packet_t pkt[], int num) { if (CONFIG_PACKET_MAX_SEGS == 1) { - buffer_free_multi((odp_buffer_hdr_t **)(uintptr_t)pkt, num); + packet_free_multi((odp_buffer_hdr_t **)(uintptr_t)pkt, num); } else { odp_buffer_hdr_t *buf_hdr[num * CONFIG_PACKET_MAX_SEGS]; int i; @@ -728,7 +758,7 @@ void odp_packet_free_multi(const odp_packet_t pkt[], int num) bufs += num_seg - 1; } - buffer_free_multi(buf_hdr, bufs); + packet_free_multi(buf_hdr, bufs); } } @@ -1937,9 +1967,37 @@ uint64_t odp_packet_seg_to_u64(odp_packet_seg_t hdl) return _odp_pri(hdl); } +static inline void buffer_ref_inc(odp_buffer_hdr_t *buf_hdr) +{ + /* First count increment after alloc */ + if (odp_atomic_load_u32(&buf_hdr->ref_cnt) == 0) + odp_atomic_store_u32(&buf_hdr->ref_cnt, 2); + else + odp_atomic_inc_u32(&buf_hdr->ref_cnt); +} + +static inline void packet_ref_inc(odp_packet_hdr_t *pkt_hdr) +{ + seg_entry_t *seg; + int i; + int seg_count = pkt_hdr->buf_hdr.segcount; + odp_packet_hdr_t *hdr = pkt_hdr; + uint8_t idx = 0; + + for (i = 0; i < seg_count; i++) { + seg = seg_entry_next(&hdr, &idx); + buffer_ref_inc(seg->hdr); + } +} + odp_packet_t odp_packet_ref_static(odp_packet_t pkt) { - return odp_packet_copy(pkt, odp_packet_pool(pkt)); + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); + + packet_ref_inc(pkt_hdr); + pkt_hdr->shared_len = packet_len(pkt_hdr); + + return pkt; } odp_packet_t odp_packet_ref(odp_packet_t pkt, uint32_t offset) @@ -2001,14 +2059,30 @@ odp_packet_t odp_packet_ref_pkt(odp_packet_t pkt, uint32_t offset, int odp_packet_has_ref(odp_packet_t pkt) { - (void)pkt; + odp_buffer_hdr_t *buf_hdr; + seg_entry_t *seg; + int i; + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); + int seg_count = pkt_hdr->buf_hdr.segcount; + odp_packet_hdr_t *hdr = pkt_hdr; + uint8_t idx = 0; + + for (i = 0; i < seg_count; i++) { + seg = seg_entry_next(&hdr, &idx); + buf_hdr = seg->hdr; + + if (odp_atomic_load_u32(&buf_hdr->ref_cnt) > 1) + return 1; + } return 0; } uint32_t odp_packet_unshared_len(odp_packet_t pkt) { - return odp_packet_len(pkt); + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); + + return packet_len(pkt_hdr) - pkt_hdr->shared_len; } /* Include non-inlined versions of API functions */ diff --git a/platform/linux-generic/odp_pool.c b/platform/linux-generic/odp_pool.c index 47a39f5b5..2f65cb20c 100644 --- a/platform/linux-generic/odp_pool.c +++ b/platform/linux-generic/odp_pool.c @@ -277,6 +277,8 @@ static void init_buffers(pool_t *pool) buf_hdr->seg[0].data = &data[offset]; buf_hdr->seg[0].len = pool->data_size; + odp_atomic_init_u32(&buf_hdr->ref_cnt, 0); + /* Store base values for fast init */ buf_hdr->base_data = buf_hdr->seg[0].data; buf_hdr->buf_end = &data[offset + pool->data_size +