[API-NEXT,RFC] api: packet: add optional persistent metadata

This requirement came from ODP-OVS: it handles buffers through 'struct ofpbuf'
(in upstream it became 'struct dp_packet'), and they can be non-ODP buffers as
well. Therefore after receiving a packet currently we have to reset three
fields in it: 'source' (to ODP), 'allocated' (segment size, OVS doesn't support
multiple segments in this struct) and 'odp_pkt' (points to the containing
odp_packet_t)
The native DPDK implementation takes advantage of the fact that the buffer and
user metadata area, where this struct is stored is allocated during pool
creation time, therefore it initializes it to static values at startup. When
I've tried this behaviour with ODP-DPDK, the performance increased from ~12.7
to ~12.9 Mpp. That's only ~%1.6 increase, but in other words it's ~%17 of the
gap between the current and OVS-DPDK performance (13.9 Mpp)
I would like a propose an optional feature where the implementation can signal
if it guarantees this behaviour. I think most implementations would
preallocate these areas to achieve good performance, but e.g. linux-generic
does not AFAIK.

Signed-off-by: Zoltan Kiss <zoltan.kiss@linaro.org>
---
 include/odp/api/config.h | 10 ++++++++++
 include/odp/api/packet.h |  4 +++-
 2 files changed, 13 insertions(+), 1 deletion(-)

The original design for the user metadata feature had persistence as an
odp_pool_param option at pool creation.  We should revisit this in that
context as introducing more hardcoded #defines into odp_config.h doesn't
seem the right way to go for these things.

On Thu, Jul 2, 2015 at 11:42 AM, Zoltan Kiss <zoltan.kiss@linaro.org> wrote:

> This requirement came from ODP-OVS: it handles buffers through 'struct
> ofpbuf'
> (in upstream it became 'struct dp_packet'), and they can be non-ODP
> buffers as
> well. Therefore after receiving a packet currently we have to reset three
> fields in it: 'source' (to ODP), 'allocated' (segment size, OVS doesn't
> support
> multiple segments in this struct) and 'odp_pkt' (points to the containing
> odp_packet_t)
> The native DPDK implementation takes advantage of the fact that the buffer
> and
> user metadata area, where this struct is stored is allocated during pool
> creation time, therefore it initializes it to static values at startup.
> When
> I've tried this behaviour with ODP-DPDK, the performance increased from
> ~12.7
> to ~12.9 Mpp. That's only ~%1.6 increase, but in other words it's ~%17 of
> the
> gap between the current and OVS-DPDK performance (13.9 Mpp)
> I would like a propose an optional feature where the implementation can
> signal
> if it guarantees this behaviour. I think most implementations would
> preallocate these areas to achieve good performance, but e.g. linux-generic
> does not AFAIK.
>
> Signed-off-by: Zoltan Kiss <zoltan.kiss@linaro.org>
> ---
>  include/odp/api/config.h | 10 ++++++++++
>  include/odp/api/packet.h |  4 +++-
>  2 files changed, 13 insertions(+), 1 deletion(-)
>
> diff --git a/include/odp/api/config.h b/include/odp/api/config.h
> index 91ea34e..d48030e 100644
> --- a/include/odp/api/config.h
> +++ b/include/odp/api/config.h
> @@ -121,6 +121,16 @@ extern "C" {
>   */
>  #define ODP_CONFIG_PACKET_BUF_LEN_MAX (ODP_CONFIG_PACKET_SEG_LEN_MIN*6)
>
> +/**
> + * Guarantee that user metadata is allocated during pool creation
> + *
> + * If this parameter is 1, applications can avoid reseting persistent
> parts of
> + * the area returned by odp_packet_user_area() by setting them once,
> after pool
> + * creation. It will keep its content between odp_packet_free() and
> + * odp_packet_alloc().
> + */
> +#define ODP_CONFIG_PACKET_PERSISTENT_MD 0
> +
>  /** Maximum number of shared memory blocks.
>   *
>   * This the the number of separate SHM areas that can be reserved
> concurrently
> diff --git a/include/odp/api/packet.h b/include/odp/api/packet.h
> index 91a124a..d7aeb95 100644
> --- a/include/odp/api/packet.h
> +++ b/include/odp/api/packet.h
> @@ -448,7 +448,9 @@ void odp_packet_user_ptr_set(odp_packet_t pkt, const
> void *ctx);
>   * User area address
>   *
>   * Each packet has an area for user data. Size of the area is fixed and
> defined
> - * in packet pool parameters.
> + * in packet pool parameters. If ODP_CONFIG_PACKET_PERSISTENT_MD is 1,
> this
> + * area is allocated during pool creation, and keep its content between
> + * odp_packet_free() and odp_packet_alloc().
>   *
>   * @param pkt  Packet handle
>   *
> --
> 1.9.1
>
> _______________________________________________
> lng-odp-dpdk mailing list
> lng-odp-dpdk@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/lng-odp-dpdk
>

The issue of callbacks was discussed last year when we were first looking
at this.  The issue is needing to be able to initialize the persistent data
at pool creation time.  There are really only two viable approaches to
this: Some sort of callback function, or else the application needs to
allocate each entry in the pool and do the initialization.  The problem of
course, is that the latter is very awkward.  It should be noted that DPDK
uses the callback approach.

Here's what we said about this last year.  We should discuss what we want
to revisit of this and what we want to change, but it seems clear that this
feature is needed for best performance.

User Meta Data and Persistence

In addition to system meta data associated with each buffer in a pool, ODP
enables the application to reserve a number of bytes of user meta data for
its own use.  While ODP implementations MUST support user meta data,  this
data MAY or MAY NOT be persistent.  Persistent means that it has a fixed
implementation-assigned address that remains addressable to the application
and retains its contents across allocates and frees of the buffer it is
associated with.  In this context persistence means that whatever was in
the user meta data at the time the buffer was freed is still there when it
is subsequently reallocated.  By contrast, user meta data that is not
persistent cannot be relied upon to survive a buffer free operation.  In
this case it is an application responsibility to ensure that if the user
meta data itself contains pointers to other storage areas that they are
appropriately freed or reference-counted prior to freeing the buffer and
that the user meta data is re-initialized whenever a buffer is allocated
from its containing buffer pool. The ODP_BUFFER_OPTS_UDATA_PERSIST option
is used to request that user meta data be persistent.  Note that
implementations MAY provide persistent user meta data by default if they
choose.  This option is used by the application to indicate that it
requires persistence.

User meta data is meta data that the application will use in conjunction
with the buffer, and is expected to be relatively small, though ODP does
not specify a minimum or maximum size of user meta data that can be
requested by an application.  It is understood, however, that there MAY be
performance penalties for large user meta data size requests, and
implementations SHOULD document these so that applications may make
appropriate trade-offs in deciding how much user meta data they can
reasonably expect to use with a given ODP implementation.   For example, an
ODP implementation may be able to store up to X bytes of user meta data in
a very efficient manner.  If an application request a larger amount of user
meta data this will result in the ODP implementation having to allocate a
block of storage of the requested size and store a pointer to this block in
an efficient manner.  The result would be extra storage references to
access the “extended” area and this MAY have a performance impact that the
application SHOULD take into consideration in its design.

When user meta data is requested, ODP will reserve the requested number of
bytes of user meta data for each buffer in the pool that makes use of user
meta data. From then the management of user meta data contents is an
application responsibility.

Applications that employ user meta data typically need to initialize it
prior to use.  There are two ways to do this: as part of buffer pool
creation or as part of buffer processing following buffer allocation from
the pool.

User meta data initialization is accomplished by allowing the application
to specify a user-supplied buffer initialization routine via the buf_init
member of the odp_buffer_pool_init_t structure passed to  to
odp_buffer_pool_create() that has the type signature:

typedef void (odp_buf_init_t)(odp_buffer_t buf, void *buf_init_arg);

When this routine is called depends on whether the user meta data is
persistent or not.  for persistent user meta data, this routine is called
for each buffer in the pool at the time the pool is created.  At entry to
buf_init(), the buffer has been reserved and any user meta data associated
with in it has been cleared to zeros.  This intent of this routine is to
initialize buffer user meta data to other values as desired.  If no buf_init
routine is provided (i.e., the field is specified as NULL), user meta data
will simply be cleared to zeros as part of default buffer pool
initialization.

If the user meta data is not persistent, then the supplied buf_init()
routine is called as part of the processing of the odp_buffer_alloc() routine
to initialize the user meta data for the buffer at the time it is allocated
from the pool.  For non-persistent user meta data, if no buf_init() routine
is supplied then no initialization of the user meta data is performed at
allocation time and the initialization and management of this data is
entirely an application responsibility.

Note that for predefined buffer pools, since these are created by the
implementation during ODP initialization, the odp_buffer_pool_init_t used
to control their initialization is passed as an argument to
odp_init_global().

User meta data management is an application responsibility with the
following provision.  If as part of an ODP API call an ODP implementation
replaces a buffer with an equivalent output buffer as part of its
processing, then any non-persistent user meta data associated with the
input buffer MUST be copied to the new buffer as part of the operation of
the API.  If the user meta data is persistent, then any such copy operation
required is the responsibility of the application.   For example, if an
application queues a buffer to a crypto engine that returns a different
buffer from the one supplied to it, then as part of the processing of that
call, the underlying ODP implementation MUST copy the input buffer user
meta data to the output buffer as part of its processing if the user meta
data is non-persistent.

The reason for this provision is that the application always has
addressability to persistent user meta data even if a buffer is freed and
so can perform whatever copying it needs from the original buffer upon
receiving the replacement buffer.  Applications can use the
odp_buffer_is_same() routine to determine if two odp_buffer_t objects refer
to the same buffer.

On Fri, Jul 3, 2015 at 3:05 AM, Savolainen, Petri (Nokia - FI/Espoo) <
petri.savolainen@nokia.com> wrote:

> Maybe it's better to add this into packet pool parameters. Usage of build
> time config macros should be minimized in the future.
>
>         /** User area size in bytes. Specify as 0 if no user
>             area is needed. */
>         uint32_t uarea_size;
>
>         /** Static user area data.
>           * 0: User area data can be overwritten while in pool (between
> free and alloc calls).
>           * 1: User area data is static. System will never overwrite the
> area.
>         *    Entire area is initialized to zero in pool creation.
>           * Default is 0.
>           */
>         int uarea_static;
>
>
> Every platform should be able to provided static area, but the performance
> may degrade due to extra indirection etc. Zero initialization would help
> user to tell if the area has valid data or not. I'd like to avoid callbacks
> from pool create to keep it simple.
>
> -Petri
>
>
> > -----Original Message-----
> > From: ext Zoltan Kiss [mailto:zoltan.kiss@linaro.org]
> > Sent: Thursday, July 02, 2015 7:43 PM
> > To: lng-odp@lists.linaro.org
> > Cc: lng-odp-dpdk@lists.linaro.org; lng-odp-ovs@lists.linaro.org;
> > Savolainen, Petri (Nokia - FI/Espoo)
> > Subject: [API-NEXT PATCH RFC] api: packet: add optional persistent
> > metadata
> >
> > This requirement came from ODP-OVS: it handles buffers through 'struct
> > ofpbuf'
> > (in upstream it became 'struct dp_packet'), and they can be non-ODP
> > buffers as
> > well. Therefore after receiving a packet currently we have to reset three
> > fields in it: 'source' (to ODP), 'allocated' (segment size, OVS doesn't
> > support
> > multiple segments in this struct) and 'odp_pkt' (points to the containing
> > odp_packet_t)
> > The native DPDK implementation takes advantage of the fact that the
> buffer
> > and
> > user metadata area, where this struct is stored is allocated during pool
> > creation time, therefore it initializes it to static values at startup.
> > When
> > I've tried this behaviour with ODP-DPDK, the performance increased from
> > ~12.7
> > to ~12.9 Mpp. That's only ~%1.6 increase, but in other words it's ~%17 of
> > the
> > gap between the current and OVS-DPDK performance (13.9 Mpp)
> > I would like a propose an optional feature where the implementation can
> > signal
> > if it guarantees this behaviour. I think most implementations would
> > preallocate these areas to achieve good performance, but e.g. linux-
> > generic
> > does not AFAIK.
> >
> > Signed-off-by: Zoltan Kiss <zoltan.kiss@linaro.org>
> > ---
> >  include/odp/api/config.h | 10 ++++++++++
> >  include/odp/api/packet.h |  4 +++-
> >  2 files changed, 13 insertions(+), 1 deletion(-)
> >
> > diff --git a/include/odp/api/config.h b/include/odp/api/config.h
> > index 91ea34e..d48030e 100644
> > --- a/include/odp/api/config.h
> > +++ b/include/odp/api/config.h
> > @@ -121,6 +121,16 @@ extern "C" {
> >   */
> >  #define ODP_CONFIG_PACKET_BUF_LEN_MAX (ODP_CONFIG_PACKET_SEG_LEN_MIN*6)
> >
> > +/**
> > + * Guarantee that user metadata is allocated during pool creation
> > + *
> > + * If this parameter is 1, applications can avoid reseting persistent
> > parts of
> > + * the area returned by odp_packet_user_area() by setting them once,
> > after pool
> > + * creation. It will keep its content between odp_packet_free() and
> > + * odp_packet_alloc().
> > + */
> > +#define ODP_CONFIG_PACKET_PERSISTENT_MD 0
> > +
> >  /** Maximum number of shared memory blocks.
> >   *
> >   * This the the number of separate SHM areas that can be reserved
> > concurrently
> > diff --git a/include/odp/api/packet.h b/include/odp/api/packet.h
> > index 91a124a..d7aeb95 100644
> > --- a/include/odp/api/packet.h
> > +++ b/include/odp/api/packet.h
> > @@ -448,7 +448,9 @@ void odp_packet_user_ptr_set(odp_packet_t pkt, const
> > void *ctx);
> >   * User area address
> >   *
> >   * Each packet has an area for user data. Size of the area is fixed and
> > defined
> > - * in packet pool parameters.
> > + * in packet pool parameters. If ODP_CONFIG_PACKET_PERSISTENT_MD is 1,
> > this
> > + * area is allocated during pool creation, and keep its content between
> > + * odp_packet_free() and odp_packet_alloc().
> >   *
> >   * @param pkt  Packet handle
> >   *
> > --
> > 1.9.1
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/lng-odp
>

On 03/07/15 09:05, Savolainen, Petri (Nokia - FI/Espoo) wrote:
> Maybe it's better to add this into packet pool parameters. Usage of build time config macros should be minimized in the future.

Sounds OK, but we'll need an accessor which tells this based on packet, 
so it could be easily inlined.

odp_bool_t odp_packet_udata_persistent(odp_packet_t pkt)
>
> 	/** User area size in bytes. Specify as 0 if no user
> 	    area is needed. */
> 	uint32_t uarea_size;
>
> 	/** Static user area data.
> 	  * 0: User area data can be overwritten while in pool (between free and alloc calls).
> 	  * 1: User area data is static. System will never overwrite the area.
>          *    Entire area is initialized to zero in pool creation.
> 	  * Default is 0.
> 	  */
> 	int uarea_static;
it should be odp_bool_t as well, and I would use the word 'persistent' 
rather than 'static'.
>
>
> Every platform should be able to provided static area, but the performance may degrade due to extra indirection etc.
I'm not just worried about performance, but how would e.g. linux-generic 
do that? As the packet and its udata comes from a completely new chunk 
of memory every time, how does it know what contents should be in the 
next allocated buffer? Just keep the copy of the udata of an packet 
previously allocated? How does it know if it was initialized properly? 
What happens when the app decides to change the initialization during 
runtime?
I think this should be rather part of odp_pool_info_t, as an output 
parameter from a pool, and let the application deal with the two cases.


> Zero initialization would help user to tell if the area has valid data or not.
> I'd like to avoid callbacks from pool create to keep it simple.

I think a callback is still more simple than allocating all the buffers 
and initializing them one by one in the application.
I like the idea in the text sent by Bill:
- if uarea_static == 0, then init callback is called during alloc, if 
any. If no callback, uarea is not inited, obviously
- if uarea_static == 1, then callback runs during pool creattion, or if 
no callback, then init to zero

>
> -Petri
>
>
>> -----Original Message-----
>> From: ext Zoltan Kiss [mailto:zoltan.kiss@linaro.org]
>> Sent: Thursday, July 02, 2015 7:43 PM
>> To: lng-odp@lists.linaro.org
>> Cc: lng-odp-dpdk@lists.linaro.org; lng-odp-ovs@lists.linaro.org;
>> Savolainen, Petri (Nokia - FI/Espoo)
>> Subject: [API-NEXT PATCH RFC] api: packet: add optional persistent
>> metadata
>>
>> This requirement came from ODP-OVS: it handles buffers through 'struct
>> ofpbuf'
>> (in upstream it became 'struct dp_packet'), and they can be non-ODP
>> buffers as
>> well. Therefore after receiving a packet currently we have to reset three
>> fields in it: 'source' (to ODP), 'allocated' (segment size, OVS doesn't
>> support
>> multiple segments in this struct) and 'odp_pkt' (points to the containing
>> odp_packet_t)
>> The native DPDK implementation takes advantage of the fact that the buffer
>> and
>> user metadata area, where this struct is stored is allocated during pool
>> creation time, therefore it initializes it to static values at startup.
>> When
>> I've tried this behaviour with ODP-DPDK, the performance increased from
>> ~12.7
>> to ~12.9 Mpp. That's only ~%1.6 increase, but in other words it's ~%17 of
>> the
>> gap between the current and OVS-DPDK performance (13.9 Mpp)
>> I would like a propose an optional feature where the implementation can
>> signal
>> if it guarantees this behaviour. I think most implementations would
>> preallocate these areas to achieve good performance, but e.g. linux-
>> generic
>> does not AFAIK.
>>
>> Signed-off-by: Zoltan Kiss <zoltan.kiss@linaro.org>
>> ---
>>   include/odp/api/config.h | 10 ++++++++++
>>   include/odp/api/packet.h |  4 +++-
>>   2 files changed, 13 insertions(+), 1 deletion(-)
>>
>> diff --git a/include/odp/api/config.h b/include/odp/api/config.h
>> index 91ea34e..d48030e 100644
>> --- a/include/odp/api/config.h
>> +++ b/include/odp/api/config.h
>> @@ -121,6 +121,16 @@ extern "C" {
>>    */
>>   #define ODP_CONFIG_PACKET_BUF_LEN_MAX (ODP_CONFIG_PACKET_SEG_LEN_MIN*6)
>>
>> +/**
>> + * Guarantee that user metadata is allocated during pool creation
>> + *
>> + * If this parameter is 1, applications can avoid reseting persistent
>> parts of
>> + * the area returned by odp_packet_user_area() by setting them once,
>> after pool
>> + * creation. It will keep its content between odp_packet_free() and
>> + * odp_packet_alloc().
>> + */
>> +#define ODP_CONFIG_PACKET_PERSISTENT_MD 0
>> +
>>   /** Maximum number of shared memory blocks.
>>    *
>>    * This the the number of separate SHM areas that can be reserved
>> concurrently
>> diff --git a/include/odp/api/packet.h b/include/odp/api/packet.h
>> index 91a124a..d7aeb95 100644
>> --- a/include/odp/api/packet.h
>> +++ b/include/odp/api/packet.h
>> @@ -448,7 +448,9 @@ void odp_packet_user_ptr_set(odp_packet_t pkt, const
>> void *ctx);
>>    * User area address
>>    *
>>    * Each packet has an area for user data. Size of the area is fixed and
>> defined
>> - * in packet pool parameters.
>> + * in packet pool parameters. If ODP_CONFIG_PACKET_PERSISTENT_MD is 1,
>> this
>> + * area is allocated during pool creation, and keep its content between
>> + * odp_packet_free() and odp_packet_alloc().
>>    *
>>    * @param pkt  Packet handle
>>    *
>> --
>> 1.9.1
>

Ping

On 08/07/15 19:44, Zoltan Kiss wrote:
>
>
> On 03/07/15 09:05, Savolainen, Petri (Nokia - FI/Espoo) wrote:
>> Maybe it's better to add this into packet pool parameters. Usage of
>> build time config macros should be minimized in the future.
>
> Sounds OK, but we'll need an accessor which tells this based on packet,
> so it could be easily inlined.
>
> odp_bool_t odp_packet_udata_persistent(odp_packet_t pkt)
>>
>>     /** User area size in bytes. Specify as 0 if no user
>>         area is needed. */
>>     uint32_t uarea_size;
>>
>>     /** Static user area data.
>>       * 0: User area data can be overwritten while in pool (between
>> free and alloc calls).
>>       * 1: User area data is static. System will never overwrite the
>> area.
>>          *    Entire area is initialized to zero in pool creation.
>>       * Default is 0.
>>       */
>>     int uarea_static;
> it should be odp_bool_t as well, and I would use the word 'persistent'
> rather than 'static'.
>>
>>
>> Every platform should be able to provided static area, but the
>> performance may degrade due to extra indirection etc.
> I'm not just worried about performance, but how would e.g. linux-generic
> do that? As the packet and its udata comes from a completely new chunk
> of memory every time, how does it know what contents should be in the
> next allocated buffer? Just keep the copy of the udata of an packet
> previously allocated? How does it know if it was initialized properly?
> What happens when the app decides to change the initialization during
> runtime?
> I think this should be rather part of odp_pool_info_t, as an output
> parameter from a pool, and let the application deal with the two cases.
>
>
>> Zero initialization would help user to tell if the area has valid data
>> or not.
>> I'd like to avoid callbacks from pool create to keep it simple.
>
> I think a callback is still more simple than allocating all the buffers
> and initializing them one by one in the application.
> I like the idea in the text sent by Bill:
> - if uarea_static == 0, then init callback is called during alloc, if
> any. If no callback, uarea is not inited, obviously
> - if uarea_static == 1, then callback runs during pool creattion, or if
> no callback, then init to zero
>
>>
>> -Petri
>>
>>
>>> -----Original Message-----
>>> From: ext Zoltan Kiss [mailto:zoltan.kiss@linaro.org]
>>> Sent: Thursday, July 02, 2015 7:43 PM
>>> To: lng-odp@lists.linaro.org
>>> Cc: lng-odp-dpdk@lists.linaro.org; lng-odp-ovs@lists.linaro.org;
>>> Savolainen, Petri (Nokia - FI/Espoo)
>>> Subject: [API-NEXT PATCH RFC] api: packet: add optional persistent
>>> metadata
>>>
>>> This requirement came from ODP-OVS: it handles buffers through 'struct
>>> ofpbuf'
>>> (in upstream it became 'struct dp_packet'), and they can be non-ODP
>>> buffers as
>>> well. Therefore after receiving a packet currently we have to reset
>>> three
>>> fields in it: 'source' (to ODP), 'allocated' (segment size, OVS doesn't
>>> support
>>> multiple segments in this struct) and 'odp_pkt' (points to the
>>> containing
>>> odp_packet_t)
>>> The native DPDK implementation takes advantage of the fact that the
>>> buffer
>>> and
>>> user metadata area, where this struct is stored is allocated during pool
>>> creation time, therefore it initializes it to static values at startup.
>>> When
>>> I've tried this behaviour with ODP-DPDK, the performance increased from
>>> ~12.7
>>> to ~12.9 Mpp. That's only ~%1.6 increase, but in other words it's
>>> ~%17 of
>>> the
>>> gap between the current and OVS-DPDK performance (13.9 Mpp)
>>> I would like a propose an optional feature where the implementation can
>>> signal
>>> if it guarantees this behaviour. I think most implementations would
>>> preallocate these areas to achieve good performance, but e.g. linux-
>>> generic
>>> does not AFAIK.
>>>
>>> Signed-off-by: Zoltan Kiss <zoltan.kiss@linaro.org>
>>> ---
>>>   include/odp/api/config.h | 10 ++++++++++
>>>   include/odp/api/packet.h |  4 +++-
>>>   2 files changed, 13 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/include/odp/api/config.h b/include/odp/api/config.h
>>> index 91ea34e..d48030e 100644
>>> --- a/include/odp/api/config.h
>>> +++ b/include/odp/api/config.h
>>> @@ -121,6 +121,16 @@ extern "C" {
>>>    */
>>>   #define ODP_CONFIG_PACKET_BUF_LEN_MAX
>>> (ODP_CONFIG_PACKET_SEG_LEN_MIN*6)
>>>
>>> +/**
>>> + * Guarantee that user metadata is allocated during pool creation
>>> + *
>>> + * If this parameter is 1, applications can avoid reseting persistent
>>> parts of
>>> + * the area returned by odp_packet_user_area() by setting them once,
>>> after pool
>>> + * creation. It will keep its content between odp_packet_free() and
>>> + * odp_packet_alloc().
>>> + */
>>> +#define ODP_CONFIG_PACKET_PERSISTENT_MD 0
>>> +
>>>   /** Maximum number of shared memory blocks.
>>>    *
>>>    * This the the number of separate SHM areas that can be reserved
>>> concurrently
>>> diff --git a/include/odp/api/packet.h b/include/odp/api/packet.h
>>> index 91a124a..d7aeb95 100644
>>> --- a/include/odp/api/packet.h
>>> +++ b/include/odp/api/packet.h
>>> @@ -448,7 +448,9 @@ void odp_packet_user_ptr_set(odp_packet_t pkt, const
>>> void *ctx);
>>>    * User area address
>>>    *
>>>    * Each packet has an area for user data. Size of the area is fixed
>>> and
>>> defined
>>> - * in packet pool parameters.
>>> + * in packet pool parameters. If ODP_CONFIG_PACKET_PERSISTENT_MD is 1,
>>> this
>>> + * area is allocated during pool creation, and keep its content between
>>> + * odp_packet_free() and odp_packet_alloc().
>>>    *
>>>    * @param pkt  Packet handle
>>>    *
>>> --
>>> 1.9.1
>>