From patchwork Mon Dec 26 23:27:55 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bill Fischofer X-Patchwork-Id: 89045 Delivered-To: patch@linaro.org Received: by 10.140.20.101 with SMTP id 92csp4740041qgi; Mon, 26 Dec 2016 15:29:57 -0800 (PST) X-Received: by 10.55.19.224 with SMTP id 93mr31542737qkt.198.1482794997381; Mon, 26 Dec 2016 15:29:57 -0800 (PST) Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id n61si16950967qtd.168.2016.12.26.15.29.57; Mon, 26 Dec 2016 15:29:57 -0800 (PST) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=pass (p=NONE dis=NONE) header.from=linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 088D560DB8; Mon, 26 Dec 2016 23:29:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 7C45660670; Mon, 26 Dec 2016 23:29:43 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id B775B60D16; Mon, 26 Dec 2016 23:29:00 +0000 (UTC) Received: from mail-oi0-f43.google.com (mail-oi0-f43.google.com [209.85.218.43]) by lists.linaro.org (Postfix) with ESMTPS id 63B1C60D32 for ; Mon, 26 Dec 2016 23:27:59 +0000 (UTC) Received: by mail-oi0-f43.google.com with SMTP id 128so138544041oig.0 for ; Mon, 26 Dec 2016 15:27:59 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=exq2FEVCA7EzmpHbEUMGcV6bZjWlKGW+biNTq8paeR4=; b=TiEhhAwC8dAthRVWjS7AwvgA7O/Uyin5466WUxoe0+ZSsftfQSF6c8u007nMvhJdRS Jt6UmoJy4AazvPUKN1lVbKF/cdYh+PrukvgJ7GPtG02ZLBw2/FR3e9QwIS8+3UHFNs9b EHcLrusrl0pXuHmAG10sd/1/EwoZdxX8/8LpOzCUVwBHxg6/3+Q3QsW7mIUbLOrgw3HQ caGpESQOOic5G8Kt2br/VcnmGiYdm2M1wWQG4Ns62m8iA4gEoBKEjmA3RFh84Q3u15h7 fdhpOpGTiVWN6u2qwYrLd+ekBMreqaeTCauDhJmewTHLSaxudQkPgk+md4nBZsykwixe Jipw== X-Gm-Message-State: AIkVDXIOoFosiyKyMHbzY93ngs2sJDwS+R20xN51/FB555jm7gG+XbDFy/FOBdqcMsmi82HWzbo= X-Received: by 10.157.26.60 with SMTP id a57mr13599557ote.17.1482794878667; Mon, 26 Dec 2016 15:27:58 -0800 (PST) Received: from Ubuntu15.localdomain (cpe-70-121-83-241.austin.res.rr.com. [70.121.83.241]) by smtp.gmail.com with ESMTPSA id h74sm21726783oic.9.2016.12.26.15.27.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 26 Dec 2016 15:27:58 -0800 (PST) From: Bill Fischofer To: lng-odp@lists.linaro.org Date: Mon, 26 Dec 2016 17:27:55 -0600 Message-Id: <1482794875-23723-1-git-send-email-bill.fischofer@linaro.org> X-Mailer: git-send-email 2.7.4 Subject: [lng-odp] [API-NEXT PATCHv3 5/5] doc: userguide: add user documentation for packet references X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" Signed-off-by: Bill Fischofer --- doc/users-guide/users-guide-packet.adoc | 269 +++++++++++++++++++++++++++++++- 1 file changed, 268 insertions(+), 1 deletion(-) -- 2.7.4 diff --git a/doc/users-guide/users-guide-packet.adoc b/doc/users-guide/users-guide-packet.adoc index e3be23c..f57f34d 100644 --- a/doc/users-guide/users-guide-packet.adoc +++ b/doc/users-guide/users-guide-packet.adoc @@ -246,7 +246,7 @@ packet pool as the original packet. The opposite operation is performed by the `odp_packet_concat()` API. This API takes a destination and source packet as arguments and the result is that the source packet is concatenated to the destination packet and ceases to -have any separete identity. Note that it is legal to concatenate a packet to +have any separate identity. Note that it is legal to concatenate a packet to itself, in which case the result is a packet with double the length of the original packet. @@ -282,3 +282,270 @@ larger than the underlying segment size. The call may also fail if the requested alignment is too high. Alignment limits will vary among different ODP implementations, however ODP requires that all implementations support requested alignments of at least 32 bytes. + +=== Packet References +To support efficient multicast, retransmit, and related processing, ODP +supports two additional types of packet manipulation: static and dynamic +_references_. + +==== Static References +The simplest type of reference is the _static reference_. A static reference is +created by the call: + +[source,c] +----- +ref_pkt = odp_packet_ref_static(pkt); +----- + +If the reference fails, `ODP_PACKET_INVALID` is returned and `pkt` +remains unchanged. + +The effect of this call is shown below: + +.Static Packet Reference +image::refstatic.svg[align="center"] + +A static reference provides a simple and efficient means of creating an alias +for a packet handle that prevents the packet itself from being freed until all +references to it have been released via `odp_packet_free()` calls. This is +useful, for example, to support retransmission processing, since as part of +packet TX processing, `odp_pktout_send()` or `odp_tm_enq()` will free +the packet after it has been transmitted. + +`odp_packet_ref_static()` might be used in a transmit routine wrapper +function like: + +[source,c] +----- +int xmit_pkt(odp_pktout_queue_t queue, odp_packet_t pkt) +{ + odp_packet_t ref = odp_packet_ref_static(pkt); + return ref == ODP_PACKET_INVALID ? -1 : odp_pktout_send(queue, ref, 1); +} +----- + +This transmits a reference to `pkt` so that `pkt` is retained by the caller, +which means that the caller is free to retransmit it if needed at a later +time. When a higher level protocol (_e.g.,_ receipt of a TCP ACK packet) +confirms that the transmission was successful, `pkt` can then be discarded via +an `odp_packet_free()` call. + +The key characteristic of a static reference is that because there are +multiple independent handles that refer to the same packet, the caller should +treat the packet as read only following the creation of a static reference +until all other references to it are freed. This is because all static +references are simply aliases of the same packet, so if multiple threads were +independently manipulating the packet this would lead to unpredictable race +conditions. + +To assist in determining whether there are other references to a packet, ODP +provides the API: + +[source,c] +----- +int odp_packet_has_ref(odp_packet_t pkt); +----- + +that indicates how many other references to `pkt` exist. If this routine +returns 0 then `pkt` has no references and hence the caller can be assured +that it is safe to modify it. Otherwise for best safety and portability, the +caller should treat `pkt` as read only. + +==== Dynamic References +While static references are convenient and efficient, they are limited by the +need to be treated as read only. For example, consider an application that +needs to _multicast_ a packet. Here the same packet needs to be sent to two or +more different destinations. While the packet payload may be the same, each +sent copy of the packet requires its own unique header to specify the +destination that is to receive the packet. + +To address this need, ODP provides _dynamic references_. These are created +by the call: + +[source,c] +----- +ref_pkt = odp_packet_ref(pkt, offset); +----- + +The `offset` parameter specifies the byte offset into `pkt` at which the +reference is to begin. This must be in the range +0..`odp_packet_len(pkt)`-1. As before, if the reference is unable to be +created `ODP_PACKET_INVALID` is returned and `pkt` is unchanged, otherwise the +result is as shown below: + +.Dynamic Packet Reference +image::ref.svg[align="center"] + +Following a successful reference creation, the bytes of `pkt` beginning at +offset `offset` are shared with the created reference. These bytes should be +treated as read only since multiple references point to them. Each reference, +however still retains its own individual headroom that is not shared with any +other reference. This allows unique headers to be created by calling +`odp_packet_push_head()` or `odp_packet_extend_head()` on the reference. This +allows multiple references to the same packet to prefix unique headers onto +common shared data it so that they can be properly multicast using code such +as: + +[source,c] +----- +int pkt_fanout(odp_packet_t payload, odp_queue_t fanout_queue[], int num_queues) +{ + int i; + + for (i = 0, i < num_queues, i++) + odp_queue_enq(fanout_queue[i], odp_packet_ref(payload, 0)); +} +----- + +Receiver worker threads can then operate on each reference to the packet in +parallel to prefix a unique transmit header onto it and send it out. + +==== Dynamic References with Headers +The dynamic references discussed so far have one drawback in that the headers +needed to make each reference unique must be constructed individually after +the reference is created. To address this problem, ODP allows these headers +to be created in advance and then simply prefixed to a base packet as part +of reference creation: + +[source,c] +----- +ref_pkt = odp_packet_ref_pkt(pkt, offset, hdr_pkt); +----- + +Here rather than creating a reference with a null header, a _header packet_ +is supplied that is prefixed onto the reference. The result looks like this: + +.Packet Reference using a Header Packet +image::refpktsingle.svg[align="center"] + +So now multicasting can be more efficient using code such as: + +[source,c] +----- +int pkt_fanout_hdr(odp_packet_t payload, odp_queue_q fanout_queue[], + odp_packet_t hdr[], int num_queues) +{ + int i; + + for (i = 0; i < num_queues, i++) + odp_queue_enq(fanout_queue[i], + odp_packet_ref_pkt(payload, 0, hdr[i])); +} +----- + +Now each individual reference has its own header already prefixed to +it ready for transmission. + +Note that when multiple references like this are made they can each have +their own offset. So if the following code is executed: + +[source,c] +----- +ref_pkt1 = odp_packet_ref_pkt(pkt, offset1, hdr_pkt1); +ref_pkt2 = odp_packet_ref_pkt(pkt, offset2, hdr_pkt2); +----- + +the result will look like: + +image::refpkt1.svg[align="center"] +image::refpktmulti.svg[align="center"] +.Multiple Packet References with Different Offsets +image::refpkt2.svg[align="center"] + +Here two separate header packets are prefixed onto the same shared packet, each +at their own specified offset, which may or may not be the same. The result is +three packets visible to the application: + +* The original `pkt`, which can still be accessed and manipulated directly. +* The first reference, which consists of `hdr_pkt1` followed by bytes +contained in `pkt` starting at `offset1`. +* The second reference, which consists of `hdr_pkt2` followed by bytes +contained in `pkt` starting at `offset2`. + +Only a single copy of the bytes in `pkt` that are common to the +references exist. + +===== Data Sharing with References +Because a `pkt` is a shared object when referenced, applications must observe +certain disciplines when working with them. In particular, any change to +shared data in `pkt` following the creation of a reference will be visible +through any reference handles to it, and this can lead to unpredictable +behavior. For best portability and reliability, the shared data contained in +any packet referred to by references should be treated as read only once it +has been successfully referenced until it is known that all references to it +have been freed. + +To assist applications in working with references, ODP provides three additional +APIs: + +[source,c] +----- +int odp_packet_is_ref(odp_packet_t pkt); + +int odp_packet_has_ref(odp_packet_t pkt); + +uint32_t odp_packet_unshared_len(odp_packet_t pkt); +----- +The `odp_packet_is_ref()` API says whether this packet was created via a call +to `odp_packet_ref()` or `odp_packet_ref_pkt()`. + +The `odp_packet_has_ref()` API says whether this packet has had a reference +created to it. If `odp_packet_has_ref()` returns 0, then it is safe to modify +this packet. If in addition `odp_packet_is_ref()` is greater than 0, then this +means that this packet is both a reference and has had a been used as a packet +for some other reference. Such _compound references_ may or may not be +supported by a given ODP implementation. + +Finally, because references and referenced packets consist of an unshared +prefix, that is modifiable, followed by a shared body that should not be +modified, the `odp_packet_unshared_len()` API is available that operates as +shown here: + +.Packet Reference Lengths +image::reflen.svg[align="center"] + +`odp_packet_unshared_len()` returns the same value as `odp_packet_len()` for +normal packets, but for packets that for which `odp_packet_is_ref()` or +`odp_packet_has_ref()` is non-zero, only returns the number of unshared bytes +prefixed to them. To ensure portability and reliability, only offsets +0..`odp_packet_unshared_len()`-1 should be modified by the caller. + +The relationship between these APIs is seen in the following table: + +.Summary of `is_ref` and `has_ref` attributes +[cols="3*", options="header", width="auto"] +|=== +| +|`odp_packet_is_ref() == 0` +|`odp_packet_is_ref() > 0` + +|`odp_packet_has_ref() == 0` +|Normal packet that has no shared segments, +`odp_packet_unshared_len() == odp_packet_len()` +|Normal reference. Simple reference if `odp_packet_is_ref() == 1`, compound + reference if `odp_packet_is_ref() > 1`. In any case, + `odp_packet_unshared_len() \<= odp_packet_len()` + +|`odp_packet_has_ref() > 0` +|Packet has one or more references that refer to +it. `odp_packet_unshared_len()` is smallest `offset` used by these +references plus any bytes pushed onto this packet by `odp_packet_push_head()` +or `odp_packet_extend_head()`. `odp_packet_unshared_len() \<= odp_packet_len()` + +|Packet is non-final reference in a compound reference, +`odp_packet_unshared_len()` is smallest `offset` used by packets refererencing +it plus any bytes pushed on this this packet by `odp_packet_push_head()` or +`odp_packet_extend_head()`. `odp_packet_unshared_len() \<= odp_packet_len()` + +|=== + +Note that architecturally, ODP does not limit referencing and so it is +possible that a reference may be used as a basis for creating another +reference. Such _compound references_ may or may not be supported by +individual ODP implementations so for best portability it is recommended that +applications restrict themselves to using simple references. + +Note also that a packet may not reference itself, nor may circular reference +relationships be formed, _e.g.,_ packet A is used as a header for a reference +to packet B and B is used as a header for a reference to packet A. Results +are undefined if such circular references are attempted.