[RFCv3] Add message I/O (MSGIO) API's

Here is my third attempt at a ODP API for MSGIO - asynchronous message passing
based IPC for a shared nothing architecture.

IPC/message passing can be separated into a number of different layers.
1) Message formats and protocols (as seen by the user).
2) API - defines syntax and semantics of MSGIO operations.
3) Transport interface - defines a binary packet format used by MSGIO
   endpoints/implementations. Cf. OSI network layer.
4) Transport layer - implements the transport of message packet between
   different endpoints. Cf. OSI link layer.

#1 is application specific and there are many different standards (e.g. Linux
Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
#3 may be implementation specific but is important for interoperatibility
(binary compatibility). If the user message (payload), operation and
parameters are encoded in a well-defined way (e.g. some kind of packet format
which includes operation, sender and receiver addresses, payload etc),
inter-operatibility between different ODP/MSGIO implementations may be
achieved or simplified.
#4 is implementation (platform) specific.

This proposal focuses on the API which is fundamental for source code
compatibility between different platforms (ODP implementations) - this being
the primary purpose of ODP. A packet format for messages will be specified as
part of the reference implementation and may be seen as a proposal for a
standard packet format for binary compatibility between different ODP/MSGIO
implementations.

All MSGIO calls are non-blocking (with the possible exception of
odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
transfer is asynchronous so it is not possible to return any status
operation concerning the eventual (or remote) result of an operation.

The semantics of MSGIO message passing is that sending a message to an endpoint
will always look like it succeeds. The appearance of endpoints is explicitly
notified through user-defined messages specified in the odp_msgio_lookup()
call. Similarly, the disappearance (e.g. death of or otherwise lost connection
to endpoint) is also explicitly notified through user-defined messages
specified in the odp_msgio_monitor() call. The send call does not fail because
the addressed endpoint has disappeared, the message is just silently discarded.

Message delivery into the recipient address space is ordered (per priority)
and reliable. Delivery of message N implies delivery of all messages <N
(of the same priority). All messages (accepted by MSGIO send operation) will
be delivered up to the point of endpoint termination or lost connection
where no more messages will be delivered. Actual reception (dequeueing) and
processing by the recipient is not guaranteed (use end-to-end acknowledgements
for that).

MSGIO endpoints can be seen as interfaces (taps) to an internal reliable
multidrop network where each endpoint has a unique address which is only
valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
and then recreated (with the same name), the new endpoint will have a
new unique address (eventually endpoints addresses will have to be recycled
but not for a very long time). Endpoints names do not necessarily have to be
unique. The scope of message network is not defined but it is expected that
the scope by default corresponds to an OS instance (e.g. virtual machine).

Proposed transport packet for message I/O (not visible to the user):
uint8_t dest_addr[6]; /* Unique address of destination endpoint */
uint8_t src_addr[6]; /* Unique address of source endpoint */
uint16_t frametype_o; /* Outer frametype, 0x8100 */
uint16_t pcp_vid; /* 3 bits of Priority Code Point, 5 bits reserved */
uint16_t frametype_i; /* Inner frametype, TBD */
uint8_t payload[]; /* User message */
Looks very much like a VLAN-tagged Ethernet frame so should easily be
transported by and processed by Ethernet HW & SW.
Encoding of publish/lookup/monitor operations TBD.

v3:
Renamed Message Bus (MBUS) to Message I/O (MSGIO).
Changed all definitions to use "msgio" instead of "mbus" prefix/infix.
odp_msgio_create(): removed default input queue parameter.
Removed odp_mbus_inq_setdef() and odp_mbus_inq_remdef().
All input queues must now be specified using odp_msgio_inq_set() before
the MSGIO endpoint is activated.
Added odp_msgio_activate().
Added odp_message_push_head() and odp_message_pull_head().
Updated some function descriptions in msgio.h.

v2:
Split off all message definitions in a separate file message.h
Renamed Inter Process Communication (IPC) to Message Bus (MBUS).
Changed all definitions to use "mbus" instead of "ipc" prefix/infix.
Renamed odp_ipc_msg_t to odp_message_t.
odp_mbus_create(): Added parameter for default input queue. Explicitly state
that the pool must be use type ODP_EVENT_MESSAGE.
Renamed odp_ipc_resolve() to odp_mbus_lookup().
odp_mbus_send(): Added priority parameter.
Renamed odp_ipc_sender() to odp_message_sender().
Renamed odp_ipc_data() to odp_message_data().
Renamed odp_ipc_length() to odp_message_length().
Renamed odp_ipc_reset() to odp_message_length_set().
Renamed odp_ipc_alloc() to odp_message_alloc().
Renamed odp_ipc_free() to odp_message_free().
odp_message_alloc(): Corrected name of invalid message handle.
Added message priorities and calls to set and remove input queues for
specific priorities: odp_mbus_inq_set(), odp_mbus_inq_rem().

Signed-off-by: Ola Liljedahl <ola.liljedahl@linaro.org>
---
(This document/code contribution attached is provided under the terms of
agreement LES-LTM-21309)

 include/odp/api/message.h                          | 199 ++++++++++++++++++
 include/odp/api/msgio.h                            | 222 +++++++++++++++++++++
 platform/linux-generic/include/odp/message.h       |  39 ++++
 platform/linux-generic/include/odp/msgio.h         |  40 ++++
 .../linux-generic/include/odp/plat/message_types.h |  47 +++++
 .../linux-generic/include/odp/plat/msgio_types.h   |  59 ++++++
 6 files changed, 606 insertions(+)
 create mode 100644 include/odp/api/message.h
 create mode 100644 include/odp/api/msgio.h
 create mode 100644 platform/linux-generic/include/odp/message.h
 create mode 100644 platform/linux-generic/include/odp/msgio.h
 create mode 100644 platform/linux-generic/include/odp/plat/message_types.h
 create mode 100644 platform/linux-generic/include/odp/plat/msgio_types.h

On 27 May 2015 at 11:08, Ola Liljedahl <ola.liljedahl@linaro.org> wrote:

> Here is my third attempt at a ODP API for MSGIO - asynchronous message
> passing
> based IPC for a shared nothing architecture.
>
> IPC/message passing can be separated into a number of different layers.
> 1) Message formats and protocols (as seen by the user).
> 2) API - defines syntax and semantics of MSGIO operations.
> 3) Transport interface - defines a binary packet format used by MSGIO
>    endpoints/implementations. Cf. OSI network layer.
> 4) Transport layer - implements the transport of message packet between
>    different endpoints. Cf. OSI link layer.
>
> #1 is application specific and there are many different standards (e.g.
> Linux
> Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
> #3 may be implementation specific but is important for interoperatibility
> (binary compatibility). If the user message (payload), operation and
> parameters are encoded in a well-defined way (e.g. some kind of packet
> format
> which includes operation, sender and receiver addresses, payload etc),
> inter-operatibility between different ODP/MSGIO implementations may be
> achieved or simplified.
> #4 is implementation (platform) specific.
>
> This proposal focuses on the API which is fundamental for source code
> compatibility between different platforms (ODP implementations) - this
> being
> the primary purpose of ODP. A packet format for messages will be specified
> as
> part of the reference implementation and may be seen as a proposal for a
> standard packet format for binary compatibility between different ODP/MSGIO
> implementations.
>
> All MSGIO calls are non-blocking (with the possible exception of
> odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
> transfer is asynchronous so it is not possible to return any status
> operation concerning the eventual (or remote) result of an operation.
>
> The semantics of MSGIO message passing is that sending a message to an
> endpoint
> will always look like it succeeds. The appearance of endpoints is
> explicitly
> notified through user-defined messages specified in the odp_msgio_lookup()
> call. Similarly, the disappearance (e.g. death of or otherwise lost
> connection
> to endpoint) is also explicitly notified through user-defined messages
> specified in the odp_msgio_monitor() call. The send call does not fail
> because
> the addressed endpoint has disappeared, the message is just silently
> discarded.
>
> Message delivery into the recipient address space is ordered (per priority)
> and reliable. Delivery of message N implies delivery of all messages <N
> (of the same priority). All messages (accepted by MSGIO send operation)
> will
> be delivered up to the point of endpoint termination or lost connection
> where no more messages will be delivered. Actual reception (dequeueing) and
> processing by the recipient is not guaranteed (use end-to-end
> acknowledgements
> for that).
>
> MSGIO endpoints can be seen as interfaces (taps) to an internal reliable
> multidrop network where each endpoint has a unique address which is only
> valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
> and then recreated (with the same name), the new endpoint will have a
> new unique address (eventually endpoints addresses will have to be recycled
> but not for a very long time). Endpoints names do not necessarily have to
> be
> unique. The scope of message network is not defined but it is expected that
> the scope by default corresponds to an OS instance (e.g. virtual machine).
>
> Proposed transport packet for message I/O (not visible to the user):
> uint8_t dest_addr[6]; /* Unique address of destination endpoint */
> uint8_t src_addr[6]; /* Unique address of source endpoint */
> uint16_t frametype_o; /* Outer frametype, 0x8100 */
> uint16_t pcp_vid; /* 3 bits of Priority Code Point, 5 bits reserved */
> uint16_t frametype_i; /* Inner frametype, TBD */
> uint8_t payload[]; /* User message */
> Looks very much like a VLAN-tagged Ethernet frame so should easily be
> transported by and processed by Ethernet HW & SW

Why to use Ethernet HW & SW to transport messages inside an OS instance?
Usually Ethernet HW & SW (drivers?) deal with sending/receiving to/from the
wire.
An above requirement was referring to ordering which must be defined by the
transport layer - how this format ensures ordered delivery at the
destination?

> .
> Encoding of publish/lookup/monitor operations TBD.
>
> v3:
> Renamed Message Bus (MBUS) to Message I/O (MSGIO).
> Changed all definitions to use "msgio" instead of "mbus" prefix/infix.
> odp_msgio_create(): removed default input queue parameter.
> Removed odp_mbus_inq_setdef() and odp_mbus_inq_remdef().
> All input queues must now be specified using odp_msgio_inq_set() before
> the MSGIO endpoint is activated.
> Added odp_msgio_activate().
> Added odp_message_push_head() and odp_message_pull_head().
> Updated some function descriptions in msgio.h.
>
> v2:
> Split off all message definitions in a separate file message.h
> Renamed Inter Process Communication (IPC) to Message Bus (MBUS).
> Changed all definitions to use "mbus" instead of "ipc" prefix/infix.
> Renamed odp_ipc_msg_t to odp_message_t.
> odp_mbus_create(): Added parameter for default input queue. Explicitly
> state
> that the pool must be use type ODP_EVENT_MESSAGE.
> Renamed odp_ipc_resolve() to odp_mbus_lookup().
> odp_mbus_send(): Added priority parameter.
> Renamed odp_ipc_sender() to odp_message_sender().
> Renamed odp_ipc_data() to odp_message_data().
> Renamed odp_ipc_length() to odp_message_length().
> Renamed odp_ipc_reset() to odp_message_length_set().
> Renamed odp_ipc_alloc() to odp_message_alloc().
> Renamed odp_ipc_free() to odp_message_free().
> odp_message_alloc(): Corrected name of invalid message handle.
> Added message priorities and calls to set and remove input queues for
> specific priorities: odp_mbus_inq_set(), odp_mbus_inq_rem().
>
> Signed-off-by: Ola Liljedahl <ola.liljedahl@linaro.org>
> ---
> (This document/code contribution attached is provided under the terms of
> agreement LES-LTM-21309)
>
>  include/odp/api/message.h                          | 199
> ++++++++++++++++++
>  include/odp/api/msgio.h                            | 222
> +++++++++++++++++++++
>  platform/linux-generic/include/odp/message.h       |  39 ++++
>  platform/linux-generic/include/odp/msgio.h         |  40 ++++
>  .../linux-generic/include/odp/plat/message_types.h |  47 +++++
>  .../linux-generic/include/odp/plat/msgio_types.h   |  59 ++++++
>  6 files changed, 606 insertions(+)
>  create mode 100644 include/odp/api/message.h
>  create mode 100644 include/odp/api/msgio.h
>  create mode 100644 platform/linux-generic/include/odp/message.h
>  create mode 100644 platform/linux-generic/include/odp/msgio.h
>  create mode 100644 platform/linux-generic/include/odp/plat/message_types.h
>  create mode 100644 platform/linux-generic/include/odp/plat/msgio_types.h
>
> diff --git a/include/odp/api/message.h b/include/odp/api/message.h
> new file mode 100644
> index 0000000..76697fb
> --- /dev/null
> +++ b/include/odp/api/message.h
> @@ -0,0 +1,199 @@
> +/* Copyright (c) 2015, Linaro Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier:     BSD-3-Clause
> + */
> +
> +
> +/**
> + * @file
> + *
> + * ODP Message event type
> + */
> +
> +#ifndef ODP_API_MESSAGE_H_
> +#define ODP_API_MESSAGE_H_
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +/** @defgroup odp_message ODP MESSAGE
> + *  @{
> + */
> +
> +/**
> + * @typedef odp_message_t
> + * ODP message handle
> + */
> +
> +
> +/**
> + * Get address of sender (source) of message
> + *
> + * @param      msg  Message handle
> + * @param[out] addr Buffer to hold sender address
> + */
> +void odp_message_sender(odp_message_t msg,
> +                       uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
> +
> +/**
> + * Message data pointer
> + *
> + * Return a pointer to the message data
> + *
> + * @param msg Message handle
> + *
> + * @return Pointer to the message data
> + */
> +void *odp_message_data(odp_message_t msg);
> +
> +/**
> + * Message data length
> + *
> + * Return length of the message data.
> + *
> + * @param msg Message handle
> + *
> + * @return Message length
> + */
> +uint32_t odp_message_length(const odp_message_t msg);
> +
> +/**
> + * Set message length
> + *
> + * Set length of the message data.
> + * Increase message data length by moving message tail into message
> tailroom.
> + * The actual message data is not modified.
> + *
> + * @param msg Message handle
> + * @param len New length
> + *
> + * @retval 0 on success
> + * @retval <0 on error (e.g. no enough tailroom)
> + */
> +int odp_message_length_set(const odp_message_t msg, uint32_t len);
> +
> +/**
> + * Message headroom length
> + *
> + * Return length of the message headroom
> + *
> + * @param msg Message handle
> + *
> + * @return Headroom length
> + */
> +uint32_t odp_message_headroom(const odp_message_t msg);
> +
> +/**
> + * Push out message head
> + *
> + * Increase message data length by moving message head into message
> headroom.
> + * Message headroom is decreased with the same amount. The message head
> may be
> + * pushed out up to 'headroom' bytes. Message is not modified if there's
> not
> + * enough headroom space.
> + *
> + * odp_message_xxx:
> + * length   += len
> + * headroom -= len
> + * data     -= len
> + *
> + * @param msg  Message handle
> + * @param len  Number of bytes to push the head (0 ... headroom)
> + *
> + * @return The new data pointer
> + * @retval NULL  Requested offset exceeds available headroom
> + *
> + * @see odp_message_headroom(), odp_message_pull_head()
> + */
> +void *odp_message_push_head(odp_message_t msg, uint32_t len);
> +
> +/**
> + * Pull in message head
> + *
> + * Decrease message data length by moving message head into message data.
> + * Message headroom is increased with the same amount. The message head
> may be
> + * pulled in up to 'length' bytes. Message is not modified if there's not
> + * enough data.
> + *
> + * odp_message_xxx:
> + * length   -= len
> + * headroom += len
> + * data     += len
> + *
> + * @param msg  Message handle
> + * @param len  Number of bytes to pull the head (0 ... length)
> + *
> + * @return The new data pointer
> + * @retval NULL  Requested offset exceeds available data
> + *
> + * @see odp_message_headroom(), odp_message_push_head()
> + */
> +void *odp_message_pull_head(odp_message_t msg, uint32_t len);
> +
> +/**
> + * Allocate message
> + *
> + * Allocate a message of a specific length.
> + *
> + * @param pool Message pool to allocate message from
> + * @param len Length of the allocated message
> + *
> + * @return Message handle on success
> + * @retval ODP_MESSAGE_INVALID on failure and errno set
> + */
> +odp_message_t odp_message_alloc(odp_pool_t pool, uint32_t len);
> +
> +/**
> + * Free message
> + *
> + * Free message back to the message pool it was allocated from.
> + *
> + * @param msg Handle of message to free
> + */
> +void odp_message_free(odp_message_t msg);
> +
> +/**
> + * Get message handle from event
> + *
> + * Converts an ODP_EVENT_MESSAGE type event to a message.
> + *
> + * @param ev   Event handle representing a message.
> + *
> + * @return Message handle
> + *
> + * @see odp_event_type()
> + */
> +odp_message_t odp_message_from_event(odp_event_t ev);
> +
> +/**
> + * Convert message handle to event
> + *
> + * @param msg  Message handle
> + *
> + * @return Event handle
> + */
> +odp_event_t odp_message_to_event(odp_message_t msg);
> +
> +/**
> + * Get printable value for an odp_message_t
> + *
> + * @param msg  Message handle to be printed
> + * @return     uint64_t value that can be used to print/display this
> + *            handle
> + *
> + * @note This routine is intended to be used for diagnostic purposes
> + * to enable applications to generate a printable value that represents
> + * an odp_message_t handle.
> + */
> +uint64_t odp_message_to_u64(odp_message_t msg);
> +
> +/**
> + * @}
> + */
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif
> diff --git a/include/odp/api/msgio.h b/include/odp/api/msgio.h
> new file mode 100644
> index 0000000..8d73021
> --- /dev/null
> +++ b/include/odp/api/msgio.h
> @@ -0,0 +1,222 @@
> +/* Copyright (c) 2015, Linaro Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier:     BSD-3-Clause
> + */
> +
> +
> +/**
> + * @file
> + *
> + * ODP Message I/O API
> + */
> +
> +#ifndef ODP_API_MSGIO_H_
> +#define ODP_API_MSGIO_H_
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +/** @defgroup odp_msgio ODP MSGIO
> + *  @{
> + */
> +
> +/**
> + * @typedef odp_msgio_t
> + * ODP message I/O handle
> + */
> +
> +/**
> + * @def ODP_MSGIO_ADDR_SIZE
> + * Size of the address of a message I/O endpoint
> + */
> +
> +/**
> + * @typedef odp_msgio_prio_t
> + * ODP MSGIO message priority
> + */
> +
> +/**
> + * @def ODP_MSGIO_PRIO_HIGHEST
> + * Highest MSGIO message priority
> + */
> +
> +/**
> + * @def ODP_MSGIO_PRIO_NORMAL
> + * Normal MSGIO message priority
> + */
> +
> +/**
> + * @def ODP_MSGIO_PRIO_LOWEST
> + * Lowest MSGIO message priority
> + */
> +
> +/**
> + * @def ODP_MSGIO_PRIO_DEFAULT
> + * Default MSGIO message priority
> + */
> +
> +
> +/**
> + * Create message I/O endpoint
> + *
> + * Create an message I/O endpoint. The scope of the message bus is
> + * not defined but it is expected that it by default encompasses the OS
> + * instance but no more.
> + *
> + * A unique address for the endpoint is created. The addresses for other
> + * endpoints can be obtained using the odp_msgio_lookup() call and
> specified
> + * in the odp_msgio_send() call.
> + *
> + * @param name Name of our endpoint
> + * @param pool Pool (of type ODP_EVENT_MESSAGE) for incoming messages
> + *
> + * @return Message I/O handle on success
> + * @retval ODP_MSGIO_INVALID on failure and errno set
> + */
> +odp_msgio_t odp_msgio_create(const char *name,
> +                            odp_pool_t pool);
> +
> +/**
> + * Set the input queue for a specific message priority
> + *
> + * @param msgio  Message I/O handle
> + * @param prio  Message priority
> + * @param queue Queue handle
> + *
> + * @retval  0 on success
> + * @retval <0 on failure
> + */
> +int odp_msgio_inq_set(odp_msgio_t msgio,
> +                     odp_msgio_prio_t prio,
> +                     odp_queue_t queue);
> +
> +/**
> + * Activate message I/O endpoint
> + *
> + * Make this endpoint visible (odp_msgio_lookup() messages will be
> returned).
> + * Enable transmission and reception of messages through the endpoint.
> + *
> + * @param msgio Message I/O handle
> + *
> + * @retval 0 on success
> + * @retval <0 on failure
> + */
> +int odp_msgio_activate(odp_msgio_t msgio);
> +
> +/**
> + * Destroy message I/O endpoint
> + *
> + * @param msgio Message I/O handle
> + *
> + * @retval 0 on success
> + * @retval <0 on failure
> + */
> +int odp_msgio_destroy(odp_msgio_t msgio);
> +
> +/**
> + * Remove the input queue for a specific message priority
> + *
> + * Remove (disassociate) the matching input queue from a message I/O
> endpoint.
> + * The queue itself is not touched.
> + *
> + * @param msgio  Message I/O handle
> + * @param prio  Message priority
> + *
> + * @retval 0 on success
> + * @retval <0 on failure
> + */
> +int odp_msgio_inq_rem(odp_msgio_t msgio,
> +                     odp_msgio_prio_t prio);
> +
> +/**
> + * Lookup endpoint by name
> + *
> + * Look up a current or future endpoint by name.
> + * When the endpoint exists, return the specified message (not
> necessarily the
> + * same message handle) with the endpoint as the sender.
> + *
> + * @param msgio Message I/O handle
> + * @param name Name to look up
> + * @param msg Message to return
> + */
> +void odp_msgio_lookup(odp_msgio_t msgio,
> +                     const char *name,
> +                     odp_message_t msg);
> +
> +/**
> + * Monitor endpoint
> + *
> + * Monitor an endpoint (which may already have disappeared).
> + * When the endpoint disappears, return the specified message (not
> necessarily
> + * the same message handle) with the endpoint as the sender.
> + *
> + * Unrecognized or invalid endpoint addresses are treated as non-existing
> + * endpoints.
> + *
> + * @param     msgio Message I/O handle
> + * @param[in] addr  Address of monitored endpoint
> + * @param     msg   Message to return
> + */
> +void odp_msgio_monitor(odp_msgio_t msgio,
> +                      const uint8_t addr[ODP_MSGIO_ADDR_SIZE],
> +                      odp_message_t msg);
> +
> +/**
> + * Send message
> + *
> + * Send a message to an endpoint (which may no longer exist).
> + *
> + * Message delivery into the recipient address space is ordered (per
> priority)
> + * and reliable. Delivery of message N implies delivery of all messages <N
> + * (of the same priority).
> + * All messages (accepted by MSGIO) will be delivered up to the point of
> + * recipient endpoint termination or lost connection where no more
> messages
> + * will be delivered.
> + *
> + * Actual reception (dequeueing) and processing by the recipient is not
> + * guaranteed (use user level acknowledgements for that).
> + *
> + * Monitor the remote endpoint to detect the disappearance or
> disconnection
> + * of the endpoint.
> + *
> + * A message may temporarily not be accepted for transmission (e.g. due
> + * to out of local buffer space), -1 will be returned and ODP errno set
> + * to EAGAIN.
> + *
> + * @param     msgio Message I/O handle
> + * @param     msg   Message to send
> + * @param     prio  Priority of message
> + * @param[in] addr  Address of recipient endpoint
> + *
> + * @retval 0 on success
> + * @retval <0 on error
> + */
> +int odp_msgio_send(odp_msgio_t msgio,
> +                  odp_message_t msg,
> +                  odp_msgio_prio_t prio,
> +                  const uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
> +
> +/**
> + * Get printable value for an odp_msgio_t
> + *
> + * @param msgio  Message I/O handle to be printed
> + * @return       uint64_t value that can be used to print/display this
> + *              handle
> + *
> + * @note This routine is intended to be used for diagnostic purposes
> + * to enable applications to generate a printable value that represents
> + * an odp_msgio_t handle.
> + */
> +uint64_t odp_msgio_to_u64(odp_msgio_t msgio);
> +
> +/**
> + * @}
> + */
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif
> diff --git a/platform/linux-generic/include/odp/message.h
> b/platform/linux-generic/include/odp/message.h
> new file mode 100644
> index 0000000..2a29f59
> --- /dev/null
> +++ b/platform/linux-generic/include/odp/message.h
> @@ -0,0 +1,39 @@
> +/* Copyright (c) 2013, Linaro Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier:     BSD-3-Clause
> + */
> +
> +/**
> + * @file
> + *
> + * ODP Message Bus
> + */
> +
> +#ifndef ODP_PLAT_MESSAGE_H_
> +#define ODP_PLAT_MESSAGE_H_
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +#include <odp/std_types.h>
> +#include <odp/plat/pool_types.h>
> +#include <odp/plat/message_types.h>
> +#include <odp/plat/queue_types.h>
> +
> +/** @ingroup odp_message
> + *  @{
> + */
> +
> +/**
> + * @}
> + */
> +
> +#include <odp/api/message.h>
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif
> diff --git a/platform/linux-generic/include/odp/msgio.h
> b/platform/linux-generic/include/odp/msgio.h
> new file mode 100644
> index 0000000..f45b3a2
> --- /dev/null
> +++ b/platform/linux-generic/include/odp/msgio.h
> @@ -0,0 +1,40 @@
> +/* Copyright (c) 2013, Linaro Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier:     BSD-3-Clause
> + */
> +
> +/**
> + * @file
> + *
> + * ODP Message Bus
> + */
> +
> +#ifndef ODP_PLAT_MSGIO_H_
> +#define ODP_PLAT_MSGIO_H_
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +#include <odp/std_types.h>
> +#include <odp/plat/pool_types.h>
> +#include <odp/plat/msgio_types.h>
> +#include <odp/plat/message_types.h>
> +#include <odp/plat/queue_types.h>
> +
> +/** @ingroup odp_msgio
> + *  @{
> + */
> +
> +/**
> + * @}
> + */
> +
> +#include <odp/api/msgio.h>
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif
> diff --git a/platform/linux-generic/include/odp/plat/message_types.h
> b/platform/linux-generic/include/odp/plat/message_types.h
> new file mode 100644
> index 0000000..3a42064
> --- /dev/null
> +++ b/platform/linux-generic/include/odp/plat/message_types.h
> @@ -0,0 +1,47 @@
> +/* Copyright (c) 2015, Linaro Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier: BSD-3-Clause
> + */
> +
> +
> +/**
> + * @file
> + *
> + * ODP Message message type
> + */
> +
> +#ifndef ODP_MESSAGE_TYPES_H_
> +#define ODP_MESSAGE_TYPES_H_
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +#include <odp/std_types.h>
> +#include <odp/plat/strong_types.h>
> +
> +/** @addtogroup odp_message ODP message
> + *  Operations on a message.
> + *  @{
> + */
> +
> +typedef ODP_HANDLE_T(odp_message_t);
> +
> +#define ODP_MESSAGE_INVALID _odp_cast_scalar(odp_message_t, 0xffffffff)
> +
> +/** Get printable format of odp_message_t */
> +static inline uint64_t odp_message_to_u64(odp_message_t hdl)
> +{
> +       return _odp_pri(hdl);
> +}
> +
> +/**
> + * @}
> + */
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif
> diff --git a/platform/linux-generic/include/odp/plat/msgio_types.h
> b/platform/linux-generic/include/odp/plat/msgio_types.h
> new file mode 100644
> index 0000000..9d9f9da
> --- /dev/null
> +++ b/platform/linux-generic/include/odp/plat/msgio_types.h
> @@ -0,0 +1,59 @@
> +/* Copyright (c) 2015, Linaro Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier: BSD-3-Clause
> + */
> +
> +
> +/**
> + * @file
> + *
> + * ODP Message bus types
> + */
> +
> +#ifndef ODP_MSGIO_TYPES_H_
> +#define ODP_MSGIO_TYPES_H_
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +#include <odp/std_types.h>
> +#include <odp/plat/strong_types.h>
> +
> +/** @addtogroup odp_msgio ODP message bus
> + *  Operations on a message bus.
> + *  @{
> + */
> +
> +typedef int odp_msgio_prio_t;
> +
> +#define ODP_MSGIO_PRIO_HIGHEST  0
> +
> +#define ODP_MSGIO_PRIO_NORMAL   2
> +
> +#define ODP_MSGIO_PRIO_LOWEST   3
> +
> +#define ODP_MSGIO_PRIO_DEFAULT  ODP_MSGIO_PRIO_NORMAL
> +
> +typedef ODP_HANDLE_T(odp_msgio_t);
> +
> +#define ODP_MSGIO_INVALID _odp_cast_scalar(odp_msgio_t, 0xffffffff)
> +
> +#define ODP_MSGIO_ADDR_SIZE 6
> +
> +/** Get printable format of odp_msgio_t */
> +static inline uint64_t odp_msgio_to_u64(odp_msgio_t hdl)
> +{
> +       return _odp_pri(hdl);
> +}
> +
> +/**
> + * @}
> + */
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif
> --
> 1.9.1
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/lng-odp
>

On 27 May 2015 at 11:10, Alexandru Badicioiu <alexandru.badicioiu@linaro.org
> wrote:

>
>
> On 27 May 2015 at 11:08, Ola Liljedahl <ola.liljedahl@linaro.org> wrote:
>
>> Here is my third attempt at a ODP API for MSGIO - asynchronous message
>> passing
>> based IPC for a shared nothing architecture.
>>
>> IPC/message passing can be separated into a number of different layers.
>> 1) Message formats and protocols (as seen by the user).
>> 2) API - defines syntax and semantics of MSGIO operations.
>> 3) Transport interface - defines a binary packet format used by MSGIO
>>    endpoints/implementations. Cf. OSI network layer.
>> 4) Transport layer - implements the transport of message packet between
>>    different endpoints. Cf. OSI link layer.
>>
>> #1 is application specific and there are many different standards (e.g.
>> Linux
>> Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
>> #3 may be implementation specific but is important for interoperatibility
>> (binary compatibility). If the user message (payload), operation and
>> parameters are encoded in a well-defined way (e.g. some kind of packet
>> format
>> which includes operation, sender and receiver addresses, payload etc),
>> inter-operatibility between different ODP/MSGIO implementations may be
>> achieved or simplified.
>> #4 is implementation (platform) specific.
>>
>> This proposal focuses on the API which is fundamental for source code
>> compatibility between different platforms (ODP implementations) - this
>> being
>> the primary purpose of ODP. A packet format for messages will be
>> specified as
>> part of the reference implementation and may be seen as a proposal for a
>> standard packet format for binary compatibility between different
>> ODP/MSGIO
>> implementations.
>>
>> All MSGIO calls are non-blocking (with the possible exception of
>> odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
>> transfer is asynchronous so it is not possible to return any status
>> operation concerning the eventual (or remote) result of an operation.
>>
>> The semantics of MSGIO message passing is that sending a message to an
>> endpoint
>> will always look like it succeeds. The appearance of endpoints is
>> explicitly
>> notified through user-defined messages specified in the odp_msgio_lookup()
>> call. Similarly, the disappearance (e.g. death of or otherwise lost
>> connection
>> to endpoint) is also explicitly notified through user-defined messages
>> specified in the odp_msgio_monitor() call. The send call does not fail
>> because
>> the addressed endpoint has disappeared, the message is just silently
>> discarded.
>>
>> Message delivery into the recipient address space is ordered (per
>> priority)
>> and reliable. Delivery of message N implies delivery of all messages <N
>> (of the same priority). All messages (accepted by MSGIO send operation)
>> will
>> be delivered up to the point of endpoint termination or lost connection
>> where no more messages will be delivered. Actual reception (dequeueing)
>> and
>> processing by the recipient is not guaranteed (use end-to-end
>> acknowledgements
>> for that).
>>
>> MSGIO endpoints can be seen as interfaces (taps) to an internal reliable
>> multidrop network where each endpoint has a unique address which is only
>> valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
>> and then recreated (with the same name), the new endpoint will have a
>> new unique address (eventually endpoints addresses will have to be
>> recycled
>> but not for a very long time). Endpoints names do not necessarily have to
>> be
>> unique. The scope of message network is not defined but it is expected
>> that
>> the scope by default corresponds to an OS instance (e.g. virtual machine).
>>
>> Proposed transport packet for message I/O (not visible to the user):
>> uint8_t dest_addr[6]; /* Unique address of destination endpoint */
>> uint8_t src_addr[6]; /* Unique address of source endpoint */
>> uint16_t frametype_o; /* Outer frametype, 0x8100 */
>> uint16_t pcp_vid; /* 3 bits of Priority Code Point, 5 bits reserved */
>> uint16_t frametype_i; /* Inner frametype, TBD */
>> uint8_t payload[]; /* User message */
>> Looks very much like a VLAN-tagged Ethernet frame so should easily be
>> transported by and processed by Ethernet HW & SW
>
> Why to use Ethernet HW & SW to transport messages inside an OS instance?
> Usually Ethernet HW & SW (drivers?) deal with sending/receiving to/from the
> wire.
>
I don't say you have to. I am just suggesting that *if* we standardize the
"packet" (message payload + metadata) format, we might as well use a
well-known format that likely can be processed by preexisting HW and SW
blocks. Possibly this will simplify classification, queueing and scheduling
of messages.



> An above requirement was referring to ordering which must be defined by
> the transport layer - how this format ensures ordered delivery at the
> destination?
>
The packet format is just the format of the message payload + metadata as
passed to and from the transport layer.

The transport layer can (if necessary) add its own encapsulation and state
machine that allows for ordered reliable delivery. That's nothing that has
to be seen by the user of the transport layer. The users of TCP don't see
the TCP header or state machine.


.
>> Encoding of publish/lookup/monitor operations TBD.
>>
>> v3:
>> Renamed Message Bus (MBUS) to Message I/O (MSGIO).
>> Changed all definitions to use "msgio" instead of "mbus" prefix/infix.
>> odp_msgio_create(): removed default input queue parameter.
>> Removed odp_mbus_inq_setdef() and odp_mbus_inq_remdef().
>> All input queues must now be specified using odp_msgio_inq_set() before
>> the MSGIO endpoint is activated.
>> Added odp_msgio_activate().
>> Added odp_message_push_head() and odp_message_pull_head().
>> Updated some function descriptions in msgio.h.
>>
>> v2:
>> Split off all message definitions in a separate file message.h
>> Renamed Inter Process Communication (IPC) to Message Bus (MBUS).
>> Changed all definitions to use "mbus" instead of "ipc" prefix/infix.
>> Renamed odp_ipc_msg_t to odp_message_t.
>> odp_mbus_create(): Added parameter for default input queue. Explicitly
>> state
>> that the pool must be use type ODP_EVENT_MESSAGE.
>> Renamed odp_ipc_resolve() to odp_mbus_lookup().
>> odp_mbus_send(): Added priority parameter.
>> Renamed odp_ipc_sender() to odp_message_sender().
>> Renamed odp_ipc_data() to odp_message_data().
>> Renamed odp_ipc_length() to odp_message_length().
>> Renamed odp_ipc_reset() to odp_message_length_set().
>> Renamed odp_ipc_alloc() to odp_message_alloc().
>> Renamed odp_ipc_free() to odp_message_free().
>> odp_message_alloc(): Corrected name of invalid message handle.
>> Added message priorities and calls to set and remove input queues for
>> specific priorities: odp_mbus_inq_set(), odp_mbus_inq_rem().
>>
>> Signed-off-by: Ola Liljedahl <ola.liljedahl@linaro.org>
>> ---
>> (This document/code contribution attached is provided under the terms of
>> agreement LES-LTM-21309)
>>
>>  include/odp/api/message.h                          | 199
>> ++++++++++++++++++
>>  include/odp/api/msgio.h                            | 222
>> +++++++++++++++++++++
>>  platform/linux-generic/include/odp/message.h       |  39 ++++
>>  platform/linux-generic/include/odp/msgio.h         |  40 ++++
>>  .../linux-generic/include/odp/plat/message_types.h |  47 +++++
>>  .../linux-generic/include/odp/plat/msgio_types.h   |  59 ++++++
>>  6 files changed, 606 insertions(+)
>>  create mode 100644 include/odp/api/message.h
>>  create mode 100644 include/odp/api/msgio.h
>>  create mode 100644 platform/linux-generic/include/odp/message.h
>>  create mode 100644 platform/linux-generic/include/odp/msgio.h
>>  create mode 100644
>> platform/linux-generic/include/odp/plat/message_types.h
>>  create mode 100644 platform/linux-generic/include/odp/plat/msgio_types.h
>>
>> diff --git a/include/odp/api/message.h b/include/odp/api/message.h
>> new file mode 100644
>> index 0000000..76697fb
>> --- /dev/null
>> +++ b/include/odp/api/message.h
>> @@ -0,0 +1,199 @@
>> +/* Copyright (c) 2015, Linaro Limited
>> + * All rights reserved.
>> + *
>> + * SPDX-License-Identifier:     BSD-3-Clause
>> + */
>> +
>> +
>> +/**
>> + * @file
>> + *
>> + * ODP Message event type
>> + */
>> +
>> +#ifndef ODP_API_MESSAGE_H_
>> +#define ODP_API_MESSAGE_H_
>> +
>> +#ifdef __cplusplus
>> +extern "C" {
>> +#endif
>> +
>> +/** @defgroup odp_message ODP MESSAGE
>> + *  @{
>> + */
>> +
>> +/**
>> + * @typedef odp_message_t
>> + * ODP message handle
>> + */
>> +
>> +
>> +/**
>> + * Get address of sender (source) of message
>> + *
>> + * @param      msg  Message handle
>> + * @param[out] addr Buffer to hold sender address
>> + */
>> +void odp_message_sender(odp_message_t msg,
>> +                       uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
>> +
>> +/**
>> + * Message data pointer
>> + *
>> + * Return a pointer to the message data
>> + *
>> + * @param msg Message handle
>> + *
>> + * @return Pointer to the message data
>> + */
>> +void *odp_message_data(odp_message_t msg);
>> +
>> +/**
>> + * Message data length
>> + *
>> + * Return length of the message data.
>> + *
>> + * @param msg Message handle
>> + *
>> + * @return Message length
>> + */
>> +uint32_t odp_message_length(const odp_message_t msg);
>> +
>> +/**
>> + * Set message length
>> + *
>> + * Set length of the message data.
>> + * Increase message data length by moving message tail into message
>> tailroom.
>> + * The actual message data is not modified.
>> + *
>> + * @param msg Message handle
>> + * @param len New length
>> + *
>> + * @retval 0 on success
>> + * @retval <0 on error (e.g. no enough tailroom)
>> + */
>> +int odp_message_length_set(const odp_message_t msg, uint32_t len);
>> +
>> +/**
>> + * Message headroom length
>> + *
>> + * Return length of the message headroom
>> + *
>> + * @param msg Message handle
>> + *
>> + * @return Headroom length
>> + */
>> +uint32_t odp_message_headroom(const odp_message_t msg);
>> +
>> +/**
>> + * Push out message head
>> + *
>> + * Increase message data length by moving message head into message
>> headroom.
>> + * Message headroom is decreased with the same amount. The message head
>> may be
>> + * pushed out up to 'headroom' bytes. Message is not modified if there's
>> not
>> + * enough headroom space.
>> + *
>> + * odp_message_xxx:
>> + * length   += len
>> + * headroom -= len
>> + * data     -= len
>> + *
>> + * @param msg  Message handle
>> + * @param len  Number of bytes to push the head (0 ... headroom)
>> + *
>> + * @return The new data pointer
>> + * @retval NULL  Requested offset exceeds available headroom
>> + *
>> + * @see odp_message_headroom(), odp_message_pull_head()
>> + */
>> +void *odp_message_push_head(odp_message_t msg, uint32_t len);
>> +
>> +/**
>> + * Pull in message head
>> + *
>> + * Decrease message data length by moving message head into message data.
>> + * Message headroom is increased with the same amount. The message head
>> may be
>> + * pulled in up to 'length' bytes. Message is not modified if there's not
>> + * enough data.
>> + *
>> + * odp_message_xxx:
>> + * length   -= len
>> + * headroom += len
>> + * data     += len
>> + *
>> + * @param msg  Message handle
>> + * @param len  Number of bytes to pull the head (0 ... length)
>> + *
>> + * @return The new data pointer
>> + * @retval NULL  Requested offset exceeds available data
>> + *
>> + * @see odp_message_headroom(), odp_message_push_head()
>> + */
>> +void *odp_message_pull_head(odp_message_t msg, uint32_t len);
>> +
>> +/**
>> + * Allocate message
>> + *
>> + * Allocate a message of a specific length.
>> + *
>> + * @param pool Message pool to allocate message from
>> + * @param len Length of the allocated message
>> + *
>> + * @return Message handle on success
>> + * @retval ODP_MESSAGE_INVALID on failure and errno set
>> + */
>> +odp_message_t odp_message_alloc(odp_pool_t pool, uint32_t len);
>> +
>> +/**
>> + * Free message
>> + *
>> + * Free message back to the message pool it was allocated from.
>> + *
>> + * @param msg Handle of message to free
>> + */
>> +void odp_message_free(odp_message_t msg);
>> +
>> +/**
>> + * Get message handle from event
>> + *
>> + * Converts an ODP_EVENT_MESSAGE type event to a message.
>> + *
>> + * @param ev   Event handle representing a message.
>> + *
>> + * @return Message handle
>> + *
>> + * @see odp_event_type()
>> + */
>> +odp_message_t odp_message_from_event(odp_event_t ev);
>> +
>> +/**
>> + * Convert message handle to event
>> + *
>> + * @param msg  Message handle
>> + *
>> + * @return Event handle
>> + */
>> +odp_event_t odp_message_to_event(odp_message_t msg);
>> +
>> +/**
>> + * Get printable value for an odp_message_t
>> + *
>> + * @param msg  Message handle to be printed
>> + * @return     uint64_t value that can be used to print/display this
>> + *            handle
>> + *
>> + * @note This routine is intended to be used for diagnostic purposes
>> + * to enable applications to generate a printable value that represents
>> + * an odp_message_t handle.
>> + */
>> +uint64_t odp_message_to_u64(odp_message_t msg);
>> +
>> +/**
>> + * @}
>> + */
>> +
>> +#ifdef __cplusplus
>> +}
>> +#endif
>> +
>> +#endif
>> diff --git a/include/odp/api/msgio.h b/include/odp/api/msgio.h
>> new file mode 100644
>> index 0000000..8d73021
>> --- /dev/null
>> +++ b/include/odp/api/msgio.h
>> @@ -0,0 +1,222 @@
>> +/* Copyright (c) 2015, Linaro Limited
>> + * All rights reserved.
>> + *
>> + * SPDX-License-Identifier:     BSD-3-Clause
>> + */
>> +
>> +
>> +/**
>> + * @file
>> + *
>> + * ODP Message I/O API
>> + */
>> +
>> +#ifndef ODP_API_MSGIO_H_
>> +#define ODP_API_MSGIO_H_
>> +
>> +#ifdef __cplusplus
>> +extern "C" {
>> +#endif
>> +
>> +/** @defgroup odp_msgio ODP MSGIO
>> + *  @{
>> + */
>> +
>> +/**
>> + * @typedef odp_msgio_t
>> + * ODP message I/O handle
>> + */
>> +
>> +/**
>> + * @def ODP_MSGIO_ADDR_SIZE
>> + * Size of the address of a message I/O endpoint
>> + */
>> +
>> +/**
>> + * @typedef odp_msgio_prio_t
>> + * ODP MSGIO message priority
>> + */
>> +
>> +/**
>> + * @def ODP_MSGIO_PRIO_HIGHEST
>> + * Highest MSGIO message priority
>> + */
>> +
>> +/**
>> + * @def ODP_MSGIO_PRIO_NORMAL
>> + * Normal MSGIO message priority
>> + */
>> +
>> +/**
>> + * @def ODP_MSGIO_PRIO_LOWEST
>> + * Lowest MSGIO message priority
>> + */
>> +
>> +/**
>> + * @def ODP_MSGIO_PRIO_DEFAULT
>> + * Default MSGIO message priority
>> + */
>> +
>> +
>> +/**
>> + * Create message I/O endpoint
>> + *
>> + * Create an message I/O endpoint. The scope of the message bus is
>> + * not defined but it is expected that it by default encompasses the OS
>> + * instance but no more.
>> + *
>> + * A unique address for the endpoint is created. The addresses for other
>> + * endpoints can be obtained using the odp_msgio_lookup() call and
>> specified
>> + * in the odp_msgio_send() call.
>> + *
>> + * @param name Name of our endpoint
>> + * @param pool Pool (of type ODP_EVENT_MESSAGE) for incoming messages
>> + *
>> + * @return Message I/O handle on success
>> + * @retval ODP_MSGIO_INVALID on failure and errno set
>> + */
>> +odp_msgio_t odp_msgio_create(const char *name,
>> +                            odp_pool_t pool);
>> +
>> +/**
>> + * Set the input queue for a specific message priority
>> + *
>> + * @param msgio  Message I/O handle
>> + * @param prio  Message priority
>> + * @param queue Queue handle
>> + *
>> + * @retval  0 on success
>> + * @retval <0 on failure
>> + */
>> +int odp_msgio_inq_set(odp_msgio_t msgio,
>> +                     odp_msgio_prio_t prio,
>> +                     odp_queue_t queue);
>> +
>> +/**
>> + * Activate message I/O endpoint
>> + *
>> + * Make this endpoint visible (odp_msgio_lookup() messages will be
>> returned).
>> + * Enable transmission and reception of messages through the endpoint.
>> + *
>> + * @param msgio Message I/O handle
>> + *
>> + * @retval 0 on success
>> + * @retval <0 on failure
>> + */
>> +int odp_msgio_activate(odp_msgio_t msgio);
>> +
>> +/**
>> + * Destroy message I/O endpoint
>> + *
>> + * @param msgio Message I/O handle
>> + *
>> + * @retval 0 on success
>> + * @retval <0 on failure
>> + */
>> +int odp_msgio_destroy(odp_msgio_t msgio);
>> +
>> +/**
>> + * Remove the input queue for a specific message priority
>> + *
>> + * Remove (disassociate) the matching input queue from a message I/O
>> endpoint.
>> + * The queue itself is not touched.
>> + *
>> + * @param msgio  Message I/O handle
>> + * @param prio  Message priority
>> + *
>> + * @retval 0 on success
>> + * @retval <0 on failure
>> + */
>> +int odp_msgio_inq_rem(odp_msgio_t msgio,
>> +                     odp_msgio_prio_t prio);
>> +
>> +/**
>> + * Lookup endpoint by name
>> + *
>> + * Look up a current or future endpoint by name.
>> + * When the endpoint exists, return the specified message (not
>> necessarily the
>> + * same message handle) with the endpoint as the sender.
>> + *
>> + * @param msgio Message I/O handle
>> + * @param name Name to look up
>> + * @param msg Message to return
>> + */
>> +void odp_msgio_lookup(odp_msgio_t msgio,
>> +                     const char *name,
>> +                     odp_message_t msg);
>> +
>> +/**
>> + * Monitor endpoint
>> + *
>> + * Monitor an endpoint (which may already have disappeared).
>> + * When the endpoint disappears, return the specified message (not
>> necessarily
>> + * the same message handle) with the endpoint as the sender.
>> + *
>> + * Unrecognized or invalid endpoint addresses are treated as non-existing
>> + * endpoints.
>> + *
>> + * @param     msgio Message I/O handle
>> + * @param[in] addr  Address of monitored endpoint
>> + * @param     msg   Message to return
>> + */
>> +void odp_msgio_monitor(odp_msgio_t msgio,
>> +                      const uint8_t addr[ODP_MSGIO_ADDR_SIZE],
>> +                      odp_message_t msg);
>> +
>> +/**
>> + * Send message
>> + *
>> + * Send a message to an endpoint (which may no longer exist).
>> + *
>> + * Message delivery into the recipient address space is ordered (per
>> priority)
>> + * and reliable. Delivery of message N implies delivery of all messages
>> <N
>> + * (of the same priority).
>> + * All messages (accepted by MSGIO) will be delivered up to the point of
>> + * recipient endpoint termination or lost connection where no more
>> messages
>> + * will be delivered.
>> + *
>> + * Actual reception (dequeueing) and processing by the recipient is not
>> + * guaranteed (use user level acknowledgements for that).
>> + *
>> + * Monitor the remote endpoint to detect the disappearance or
>> disconnection
>> + * of the endpoint.
>> + *
>> + * A message may temporarily not be accepted for transmission (e.g. due
>> + * to out of local buffer space), -1 will be returned and ODP errno set
>> + * to EAGAIN.
>> + *
>> + * @param     msgio Message I/O handle
>> + * @param     msg   Message to send
>> + * @param     prio  Priority of message
>> + * @param[in] addr  Address of recipient endpoint
>> + *
>> + * @retval 0 on success
>> + * @retval <0 on error
>> + */
>> +int odp_msgio_send(odp_msgio_t msgio,
>> +                  odp_message_t msg,
>> +                  odp_msgio_prio_t prio,
>> +                  const uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
>> +
>> +/**
>> + * Get printable value for an odp_msgio_t
>> + *
>> + * @param msgio  Message I/O handle to be printed
>> + * @return       uint64_t value that can be used to print/display this
>> + *              handle
>> + *
>> + * @note This routine is intended to be used for diagnostic purposes
>> + * to enable applications to generate a printable value that represents
>> + * an odp_msgio_t handle.
>> + */
>> +uint64_t odp_msgio_to_u64(odp_msgio_t msgio);
>> +
>> +/**
>> + * @}
>> + */
>> +
>> +#ifdef __cplusplus
>> +}
>> +#endif
>> +
>> +#endif
>> diff --git a/platform/linux-generic/include/odp/message.h
>> b/platform/linux-generic/include/odp/message.h
>> new file mode 100644
>> index 0000000..2a29f59
>> --- /dev/null
>> +++ b/platform/linux-generic/include/odp/message.h
>> @@ -0,0 +1,39 @@
>> +/* Copyright (c) 2013, Linaro Limited
>> + * All rights reserved.
>> + *
>> + * SPDX-License-Identifier:     BSD-3-Clause
>> + */
>> +
>> +/**
>> + * @file
>> + *
>> + * ODP Message Bus
>> + */
>> +
>> +#ifndef ODP_PLAT_MESSAGE_H_
>> +#define ODP_PLAT_MESSAGE_H_
>> +
>> +#ifdef __cplusplus
>> +extern "C" {
>> +#endif
>> +
>> +#include <odp/std_types.h>
>> +#include <odp/plat/pool_types.h>
>> +#include <odp/plat/message_types.h>
>> +#include <odp/plat/queue_types.h>
>> +
>> +/** @ingroup odp_message
>> + *  @{
>> + */
>> +
>> +/**
>> + * @}
>> + */
>> +
>> +#include <odp/api/message.h>
>> +
>> +#ifdef __cplusplus
>> +}
>> +#endif
>> +
>> +#endif
>> diff --git a/platform/linux-generic/include/odp/msgio.h
>> b/platform/linux-generic/include/odp/msgio.h
>> new file mode 100644
>> index 0000000..f45b3a2
>> --- /dev/null
>> +++ b/platform/linux-generic/include/odp/msgio.h
>> @@ -0,0 +1,40 @@
>> +/* Copyright (c) 2013, Linaro Limited
>> + * All rights reserved.
>> + *
>> + * SPDX-License-Identifier:     BSD-3-Clause
>> + */
>> +
>> +/**
>> + * @file
>> + *
>> + * ODP Message Bus
>> + */
>> +
>> +#ifndef ODP_PLAT_MSGIO_H_
>> +#define ODP_PLAT_MSGIO_H_
>> +
>> +#ifdef __cplusplus
>> +extern "C" {
>> +#endif
>> +
>> +#include <odp/std_types.h>
>> +#include <odp/plat/pool_types.h>
>> +#include <odp/plat/msgio_types.h>
>> +#include <odp/plat/message_types.h>
>> +#include <odp/plat/queue_types.h>
>> +
>> +/** @ingroup odp_msgio
>> + *  @{
>> + */
>> +
>> +/**
>> + * @}
>> + */
>> +
>> +#include <odp/api/msgio.h>
>> +
>> +#ifdef __cplusplus
>> +}
>> +#endif
>> +
>> +#endif
>> diff --git a/platform/linux-generic/include/odp/plat/message_types.h
>> b/platform/linux-generic/include/odp/plat/message_types.h
>> new file mode 100644
>> index 0000000..3a42064
>> --- /dev/null
>> +++ b/platform/linux-generic/include/odp/plat/message_types.h
>> @@ -0,0 +1,47 @@
>> +/* Copyright (c) 2015, Linaro Limited
>> + * All rights reserved.
>> + *
>> + * SPDX-License-Identifier: BSD-3-Clause
>> + */
>> +
>> +
>> +/**
>> + * @file
>> + *
>> + * ODP Message message type
>> + */
>> +
>> +#ifndef ODP_MESSAGE_TYPES_H_
>> +#define ODP_MESSAGE_TYPES_H_
>> +
>> +#ifdef __cplusplus
>> +extern "C" {
>> +#endif
>> +
>> +#include <odp/std_types.h>
>> +#include <odp/plat/strong_types.h>
>> +
>> +/** @addtogroup odp_message ODP message
>> + *  Operations on a message.
>> + *  @{
>> + */
>> +
>> +typedef ODP_HANDLE_T(odp_message_t);
>> +
>> +#define ODP_MESSAGE_INVALID _odp_cast_scalar(odp_message_t, 0xffffffff)
>> +
>> +/** Get printable format of odp_message_t */
>> +static inline uint64_t odp_message_to_u64(odp_message_t hdl)
>> +{
>> +       return _odp_pri(hdl);
>> +}
>> +
>> +/**
>> + * @}
>> + */
>> +
>> +#ifdef __cplusplus
>> +}
>> +#endif
>> +
>> +#endif
>> diff --git a/platform/linux-generic/include/odp/plat/msgio_types.h
>> b/platform/linux-generic/include/odp/plat/msgio_types.h
>> new file mode 100644
>> index 0000000..9d9f9da
>> --- /dev/null
>> +++ b/platform/linux-generic/include/odp/plat/msgio_types.h
>> @@ -0,0 +1,59 @@
>> +/* Copyright (c) 2015, Linaro Limited
>> + * All rights reserved.
>> + *
>> + * SPDX-License-Identifier: BSD-3-Clause
>> + */
>> +
>> +
>> +/**
>> + * @file
>> + *
>> + * ODP Message bus types
>> + */
>> +
>> +#ifndef ODP_MSGIO_TYPES_H_
>> +#define ODP_MSGIO_TYPES_H_
>> +
>> +#ifdef __cplusplus
>> +extern "C" {
>> +#endif
>> +
>> +#include <odp/std_types.h>
>> +#include <odp/plat/strong_types.h>
>> +
>> +/** @addtogroup odp_msgio ODP message bus
>> + *  Operations on a message bus.
>> + *  @{
>> + */
>> +
>> +typedef int odp_msgio_prio_t;
>> +
>> +#define ODP_MSGIO_PRIO_HIGHEST  0
>> +
>> +#define ODP_MSGIO_PRIO_NORMAL   2
>> +
>> +#define ODP_MSGIO_PRIO_LOWEST   3
>> +
>> +#define ODP_MSGIO_PRIO_DEFAULT  ODP_MSGIO_PRIO_NORMAL
>> +
>> +typedef ODP_HANDLE_T(odp_msgio_t);
>> +
>> +#define ODP_MSGIO_INVALID _odp_cast_scalar(odp_msgio_t, 0xffffffff)
>> +
>> +#define ODP_MSGIO_ADDR_SIZE 6
>> +
>> +/** Get printable format of odp_msgio_t */
>> +static inline uint64_t odp_msgio_to_u64(odp_msgio_t hdl)
>> +{
>> +       return _odp_pri(hdl);
>> +}
>> +
>> +/**
>> + * @}
>> + */
>> +
>> +#ifdef __cplusplus
>> +}
>> +#endif
>> +
>> +#endif
>> --
>> 1.9.1
>>
>> _______________________________________________
>> lng-odp mailing list
>> lng-odp@lists.linaro.org
>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>
>
>

If you want to use an Ethernet-like internal transport, you might was well
use a standard one like TIPC <http://tipc.sourceforge.net/> rather than
reinventing this particular wheel.

On Wed, May 27, 2015 at 4:49 AM, Ola Liljedahl <ola.liljedahl@linaro.org>
wrote:

> On 27 May 2015 at 11:10, Alexandru Badicioiu <
> alexandru.badicioiu@linaro.org> wrote:
>
>>
>>
>> On 27 May 2015 at 11:08, Ola Liljedahl <ola.liljedahl@linaro.org> wrote:
>>
>>> Here is my third attempt at a ODP API for MSGIO - asynchronous message
>>> passing
>>> based IPC for a shared nothing architecture.
>>>
>>> IPC/message passing can be separated into a number of different layers.
>>> 1) Message formats and protocols (as seen by the user).
>>> 2) API - defines syntax and semantics of MSGIO operations.
>>> 3) Transport interface - defines a binary packet format used by MSGIO
>>>    endpoints/implementations. Cf. OSI network layer.
>>> 4) Transport layer - implements the transport of message packet between
>>>    different endpoints. Cf. OSI link layer.
>>>
>>> #1 is application specific and there are many different standards (e.g.
>>> Linux
>>> Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
>>> #3 may be implementation specific but is important for interoperatibility
>>> (binary compatibility). If the user message (payload), operation and
>>> parameters are encoded in a well-defined way (e.g. some kind of packet
>>> format
>>> which includes operation, sender and receiver addresses, payload etc),
>>> inter-operatibility between different ODP/MSGIO implementations may be
>>> achieved or simplified.
>>> #4 is implementation (platform) specific.
>>>
>>> This proposal focuses on the API which is fundamental for source code
>>> compatibility between different platforms (ODP implementations) - this
>>> being
>>> the primary purpose of ODP. A packet format for messages will be
>>> specified as
>>> part of the reference implementation and may be seen as a proposal for a
>>> standard packet format for binary compatibility between different
>>> ODP/MSGIO
>>> implementations.
>>>
>>> All MSGIO calls are non-blocking (with the possible exception of
>>> odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
>>> transfer is asynchronous so it is not possible to return any status
>>> operation concerning the eventual (or remote) result of an operation.
>>>
>>> The semantics of MSGIO message passing is that sending a message to an
>>> endpoint
>>> will always look like it succeeds. The appearance of endpoints is
>>> explicitly
>>> notified through user-defined messages specified in the
>>> odp_msgio_lookup()
>>> call. Similarly, the disappearance (e.g. death of or otherwise lost
>>> connection
>>> to endpoint) is also explicitly notified through user-defined messages
>>> specified in the odp_msgio_monitor() call. The send call does not fail
>>> because
>>> the addressed endpoint has disappeared, the message is just silently
>>> discarded.
>>>
>>> Message delivery into the recipient address space is ordered (per
>>> priority)
>>> and reliable. Delivery of message N implies delivery of all messages <N
>>> (of the same priority). All messages (accepted by MSGIO send operation)
>>> will
>>> be delivered up to the point of endpoint termination or lost connection
>>> where no more messages will be delivered. Actual reception (dequeueing)
>>> and
>>> processing by the recipient is not guaranteed (use end-to-end
>>> acknowledgements
>>> for that).
>>>
>>> MSGIO endpoints can be seen as interfaces (taps) to an internal reliable
>>> multidrop network where each endpoint has a unique address which is only
>>> valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
>>> and then recreated (with the same name), the new endpoint will have a
>>> new unique address (eventually endpoints addresses will have to be
>>> recycled
>>> but not for a very long time). Endpoints names do not necessarily have
>>> to be
>>> unique. The scope of message network is not defined but it is expected
>>> that
>>> the scope by default corresponds to an OS instance (e.g. virtual
>>> machine).
>>>
>>> Proposed transport packet for message I/O (not visible to the user):
>>> uint8_t dest_addr[6]; /* Unique address of destination endpoint */
>>> uint8_t src_addr[6]; /* Unique address of source endpoint */
>>> uint16_t frametype_o; /* Outer frametype, 0x8100 */
>>> uint16_t pcp_vid; /* 3 bits of Priority Code Point, 5 bits reserved */
>>> uint16_t frametype_i; /* Inner frametype, TBD */
>>> uint8_t payload[]; /* User message */
>>> Looks very much like a VLAN-tagged Ethernet frame so should easily be
>>> transported by and processed by Ethernet HW & SW
>>
>> Why to use Ethernet HW & SW to transport messages inside an OS instance?
>> Usually Ethernet HW & SW (drivers?) deal with sending/receiving to/from the
>> wire.
>>
> I don't say you have to. I am just suggesting that *if* we standardize the
> "packet" (message payload + metadata) format, we might as well use a
> well-known format that likely can be processed by preexisting HW and SW
> blocks. Possibly this will simplify classification, queueing and scheduling
> of messages.
>
>
>
>> An above requirement was referring to ordering which must be defined by
>> the transport layer - how this format ensures ordered delivery at the
>> destination?
>>
> The packet format is just the format of the message payload + metadata as
> passed to and from the transport layer.
>
> The transport layer can (if necessary) add its own encapsulation and state
> machine that allows for ordered reliable delivery. That's nothing that has
> to be seen by the user of the transport layer. The users of TCP don't see
> the TCP header or state machine.
>
>
> .
>>> Encoding of publish/lookup/monitor operations TBD.
>>>
>>> v3:
>>> Renamed Message Bus (MBUS) to Message I/O (MSGIO).
>>> Changed all definitions to use "msgio" instead of "mbus" prefix/infix.
>>> odp_msgio_create(): removed default input queue parameter.
>>> Removed odp_mbus_inq_setdef() and odp_mbus_inq_remdef().
>>> All input queues must now be specified using odp_msgio_inq_set() before
>>> the MSGIO endpoint is activated.
>>> Added odp_msgio_activate().
>>> Added odp_message_push_head() and odp_message_pull_head().
>>> Updated some function descriptions in msgio.h.
>>>
>>> v2:
>>> Split off all message definitions in a separate file message.h
>>> Renamed Inter Process Communication (IPC) to Message Bus (MBUS).
>>> Changed all definitions to use "mbus" instead of "ipc" prefix/infix.
>>> Renamed odp_ipc_msg_t to odp_message_t.
>>> odp_mbus_create(): Added parameter for default input queue. Explicitly
>>> state
>>> that the pool must be use type ODP_EVENT_MESSAGE.
>>> Renamed odp_ipc_resolve() to odp_mbus_lookup().
>>> odp_mbus_send(): Added priority parameter.
>>> Renamed odp_ipc_sender() to odp_message_sender().
>>> Renamed odp_ipc_data() to odp_message_data().
>>> Renamed odp_ipc_length() to odp_message_length().
>>> Renamed odp_ipc_reset() to odp_message_length_set().
>>> Renamed odp_ipc_alloc() to odp_message_alloc().
>>> Renamed odp_ipc_free() to odp_message_free().
>>> odp_message_alloc(): Corrected name of invalid message handle.
>>> Added message priorities and calls to set and remove input queues for
>>> specific priorities: odp_mbus_inq_set(), odp_mbus_inq_rem().
>>>
>>> Signed-off-by: Ola Liljedahl <ola.liljedahl@linaro.org>
>>> ---
>>> (This document/code contribution attached is provided under the terms of
>>> agreement LES-LTM-21309)
>>>
>>>  include/odp/api/message.h                          | 199
>>> ++++++++++++++++++
>>>  include/odp/api/msgio.h                            | 222
>>> +++++++++++++++++++++
>>>  platform/linux-generic/include/odp/message.h       |  39 ++++
>>>  platform/linux-generic/include/odp/msgio.h         |  40 ++++
>>>  .../linux-generic/include/odp/plat/message_types.h |  47 +++++
>>>  .../linux-generic/include/odp/plat/msgio_types.h   |  59 ++++++
>>>  6 files changed, 606 insertions(+)
>>>  create mode 100644 include/odp/api/message.h
>>>  create mode 100644 include/odp/api/msgio.h
>>>  create mode 100644 platform/linux-generic/include/odp/message.h
>>>  create mode 100644 platform/linux-generic/include/odp/msgio.h
>>>  create mode 100644
>>> platform/linux-generic/include/odp/plat/message_types.h
>>>  create mode 100644 platform/linux-generic/include/odp/plat/msgio_types.h
>>>
>>> diff --git a/include/odp/api/message.h b/include/odp/api/message.h
>>> new file mode 100644
>>> index 0000000..76697fb
>>> --- /dev/null
>>> +++ b/include/odp/api/message.h
>>> @@ -0,0 +1,199 @@
>>> +/* Copyright (c) 2015, Linaro Limited
>>> + * All rights reserved.
>>> + *
>>> + * SPDX-License-Identifier:     BSD-3-Clause
>>> + */
>>> +
>>> +
>>> +/**
>>> + * @file
>>> + *
>>> + * ODP Message event type
>>> + */
>>> +
>>> +#ifndef ODP_API_MESSAGE_H_
>>> +#define ODP_API_MESSAGE_H_
>>> +
>>> +#ifdef __cplusplus
>>> +extern "C" {
>>> +#endif
>>> +
>>> +/** @defgroup odp_message ODP MESSAGE
>>> + *  @{
>>> + */
>>> +
>>> +/**
>>> + * @typedef odp_message_t
>>> + * ODP message handle
>>> + */
>>> +
>>> +
>>> +/**
>>> + * Get address of sender (source) of message
>>> + *
>>> + * @param      msg  Message handle
>>> + * @param[out] addr Buffer to hold sender address
>>> + */
>>> +void odp_message_sender(odp_message_t msg,
>>> +                       uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
>>> +
>>> +/**
>>> + * Message data pointer
>>> + *
>>> + * Return a pointer to the message data
>>> + *
>>> + * @param msg Message handle
>>> + *
>>> + * @return Pointer to the message data
>>> + */
>>> +void *odp_message_data(odp_message_t msg);
>>> +
>>> +/**
>>> + * Message data length
>>> + *
>>> + * Return length of the message data.
>>> + *
>>> + * @param msg Message handle
>>> + *
>>> + * @return Message length
>>> + */
>>> +uint32_t odp_message_length(const odp_message_t msg);
>>> +
>>> +/**
>>> + * Set message length
>>> + *
>>> + * Set length of the message data.
>>> + * Increase message data length by moving message tail into message
>>> tailroom.
>>> + * The actual message data is not modified.
>>> + *
>>> + * @param msg Message handle
>>> + * @param len New length
>>> + *
>>> + * @retval 0 on success
>>> + * @retval <0 on error (e.g. no enough tailroom)
>>> + */
>>> +int odp_message_length_set(const odp_message_t msg, uint32_t len);
>>> +
>>> +/**
>>> + * Message headroom length
>>> + *
>>> + * Return length of the message headroom
>>> + *
>>> + * @param msg Message handle
>>> + *
>>> + * @return Headroom length
>>> + */
>>> +uint32_t odp_message_headroom(const odp_message_t msg);
>>> +
>>> +/**
>>> + * Push out message head
>>> + *
>>> + * Increase message data length by moving message head into message
>>> headroom.
>>> + * Message headroom is decreased with the same amount. The message head
>>> may be
>>> + * pushed out up to 'headroom' bytes. Message is not modified if
>>> there's not
>>> + * enough headroom space.
>>> + *
>>> + * odp_message_xxx:
>>> + * length   += len
>>> + * headroom -= len
>>> + * data     -= len
>>> + *
>>> + * @param msg  Message handle
>>> + * @param len  Number of bytes to push the head (0 ... headroom)
>>> + *
>>> + * @return The new data pointer
>>> + * @retval NULL  Requested offset exceeds available headroom
>>> + *
>>> + * @see odp_message_headroom(), odp_message_pull_head()
>>> + */
>>> +void *odp_message_push_head(odp_message_t msg, uint32_t len);
>>> +
>>> +/**
>>> + * Pull in message head
>>> + *
>>> + * Decrease message data length by moving message head into message
>>> data.
>>> + * Message headroom is increased with the same amount. The message head
>>> may be
>>> + * pulled in up to 'length' bytes. Message is not modified if there's
>>> not
>>> + * enough data.
>>> + *
>>> + * odp_message_xxx:
>>> + * length   -= len
>>> + * headroom += len
>>> + * data     += len
>>> + *
>>> + * @param msg  Message handle
>>> + * @param len  Number of bytes to pull the head (0 ... length)
>>> + *
>>> + * @return The new data pointer
>>> + * @retval NULL  Requested offset exceeds available data
>>> + *
>>> + * @see odp_message_headroom(), odp_message_push_head()
>>> + */
>>> +void *odp_message_pull_head(odp_message_t msg, uint32_t len);
>>> +
>>> +/**
>>> + * Allocate message
>>> + *
>>> + * Allocate a message of a specific length.
>>> + *
>>> + * @param pool Message pool to allocate message from
>>> + * @param len Length of the allocated message
>>> + *
>>> + * @return Message handle on success
>>> + * @retval ODP_MESSAGE_INVALID on failure and errno set
>>> + */
>>> +odp_message_t odp_message_alloc(odp_pool_t pool, uint32_t len);
>>> +
>>> +/**
>>> + * Free message
>>> + *
>>> + * Free message back to the message pool it was allocated from.
>>> + *
>>> + * @param msg Handle of message to free
>>> + */
>>> +void odp_message_free(odp_message_t msg);
>>> +
>>> +/**
>>> + * Get message handle from event
>>> + *
>>> + * Converts an ODP_EVENT_MESSAGE type event to a message.
>>> + *
>>> + * @param ev   Event handle representing a message.
>>> + *
>>> + * @return Message handle
>>> + *
>>> + * @see odp_event_type()
>>> + */
>>> +odp_message_t odp_message_from_event(odp_event_t ev);
>>> +
>>> +/**
>>> + * Convert message handle to event
>>> + *
>>> + * @param msg  Message handle
>>> + *
>>> + * @return Event handle
>>> + */
>>> +odp_event_t odp_message_to_event(odp_message_t msg);
>>> +
>>> +/**
>>> + * Get printable value for an odp_message_t
>>> + *
>>> + * @param msg  Message handle to be printed
>>> + * @return     uint64_t value that can be used to print/display this
>>> + *            handle
>>> + *
>>> + * @note This routine is intended to be used for diagnostic purposes
>>> + * to enable applications to generate a printable value that represents
>>> + * an odp_message_t handle.
>>> + */
>>> +uint64_t odp_message_to_u64(odp_message_t msg);
>>> +
>>> +/**
>>> + * @}
>>> + */
>>> +
>>> +#ifdef __cplusplus
>>> +}
>>> +#endif
>>> +
>>> +#endif
>>> diff --git a/include/odp/api/msgio.h b/include/odp/api/msgio.h
>>> new file mode 100644
>>> index 0000000..8d73021
>>> --- /dev/null
>>> +++ b/include/odp/api/msgio.h
>>> @@ -0,0 +1,222 @@
>>> +/* Copyright (c) 2015, Linaro Limited
>>> + * All rights reserved.
>>> + *
>>> + * SPDX-License-Identifier:     BSD-3-Clause
>>> + */
>>> +
>>> +
>>> +/**
>>> + * @file
>>> + *
>>> + * ODP Message I/O API
>>> + */
>>> +
>>> +#ifndef ODP_API_MSGIO_H_
>>> +#define ODP_API_MSGIO_H_
>>> +
>>> +#ifdef __cplusplus
>>> +extern "C" {
>>> +#endif
>>> +
>>> +/** @defgroup odp_msgio ODP MSGIO
>>> + *  @{
>>> + */
>>> +
>>> +/**
>>> + * @typedef odp_msgio_t
>>> + * ODP message I/O handle
>>> + */
>>> +
>>> +/**
>>> + * @def ODP_MSGIO_ADDR_SIZE
>>> + * Size of the address of a message I/O endpoint
>>> + */
>>> +
>>> +/**
>>> + * @typedef odp_msgio_prio_t
>>> + * ODP MSGIO message priority
>>> + */
>>> +
>>> +/**
>>> + * @def ODP_MSGIO_PRIO_HIGHEST
>>> + * Highest MSGIO message priority
>>> + */
>>> +
>>> +/**
>>> + * @def ODP_MSGIO_PRIO_NORMAL
>>> + * Normal MSGIO message priority
>>> + */
>>> +
>>> +/**
>>> + * @def ODP_MSGIO_PRIO_LOWEST
>>> + * Lowest MSGIO message priority
>>> + */
>>> +
>>> +/**
>>> + * @def ODP_MSGIO_PRIO_DEFAULT
>>> + * Default MSGIO message priority
>>> + */
>>> +
>>> +
>>> +/**
>>> + * Create message I/O endpoint
>>> + *
>>> + * Create an message I/O endpoint. The scope of the message bus is
>>> + * not defined but it is expected that it by default encompasses the OS
>>> + * instance but no more.
>>> + *
>>> + * A unique address for the endpoint is created. The addresses for other
>>> + * endpoints can be obtained using the odp_msgio_lookup() call and
>>> specified
>>> + * in the odp_msgio_send() call.
>>> + *
>>> + * @param name Name of our endpoint
>>> + * @param pool Pool (of type ODP_EVENT_MESSAGE) for incoming messages
>>> + *
>>> + * @return Message I/O handle on success
>>> + * @retval ODP_MSGIO_INVALID on failure and errno set
>>> + */
>>> +odp_msgio_t odp_msgio_create(const char *name,
>>> +                            odp_pool_t pool);
>>> +
>>> +/**
>>> + * Set the input queue for a specific message priority
>>> + *
>>> + * @param msgio  Message I/O handle
>>> + * @param prio  Message priority
>>> + * @param queue Queue handle
>>> + *
>>> + * @retval  0 on success
>>> + * @retval <0 on failure
>>> + */
>>> +int odp_msgio_inq_set(odp_msgio_t msgio,
>>> +                     odp_msgio_prio_t prio,
>>> +                     odp_queue_t queue);
>>> +
>>> +/**
>>> + * Activate message I/O endpoint
>>> + *
>>> + * Make this endpoint visible (odp_msgio_lookup() messages will be
>>> returned).
>>> + * Enable transmission and reception of messages through the endpoint.
>>> + *
>>> + * @param msgio Message I/O handle
>>> + *
>>> + * @retval 0 on success
>>> + * @retval <0 on failure
>>> + */
>>> +int odp_msgio_activate(odp_msgio_t msgio);
>>> +
>>> +/**
>>> + * Destroy message I/O endpoint
>>> + *
>>> + * @param msgio Message I/O handle
>>> + *
>>> + * @retval 0 on success
>>> + * @retval <0 on failure
>>> + */
>>> +int odp_msgio_destroy(odp_msgio_t msgio);
>>> +
>>> +/**
>>> + * Remove the input queue for a specific message priority
>>> + *
>>> + * Remove (disassociate) the matching input queue from a message I/O
>>> endpoint.
>>> + * The queue itself is not touched.
>>> + *
>>> + * @param msgio  Message I/O handle
>>> + * @param prio  Message priority
>>> + *
>>> + * @retval 0 on success
>>> + * @retval <0 on failure
>>> + */
>>> +int odp_msgio_inq_rem(odp_msgio_t msgio,
>>> +                     odp_msgio_prio_t prio);
>>> +
>>> +/**
>>> + * Lookup endpoint by name
>>> + *
>>> + * Look up a current or future endpoint by name.
>>> + * When the endpoint exists, return the specified message (not
>>> necessarily the
>>> + * same message handle) with the endpoint as the sender.
>>> + *
>>> + * @param msgio Message I/O handle
>>> + * @param name Name to look up
>>> + * @param msg Message to return
>>> + */
>>> +void odp_msgio_lookup(odp_msgio_t msgio,
>>> +                     const char *name,
>>> +                     odp_message_t msg);
>>> +
>>> +/**
>>> + * Monitor endpoint
>>> + *
>>> + * Monitor an endpoint (which may already have disappeared).
>>> + * When the endpoint disappears, return the specified message (not
>>> necessarily
>>> + * the same message handle) with the endpoint as the sender.
>>> + *
>>> + * Unrecognized or invalid endpoint addresses are treated as
>>> non-existing
>>> + * endpoints.
>>> + *
>>> + * @param     msgio Message I/O handle
>>> + * @param[in] addr  Address of monitored endpoint
>>> + * @param     msg   Message to return
>>> + */
>>> +void odp_msgio_monitor(odp_msgio_t msgio,
>>> +                      const uint8_t addr[ODP_MSGIO_ADDR_SIZE],
>>> +                      odp_message_t msg);
>>> +
>>> +/**
>>> + * Send message
>>> + *
>>> + * Send a message to an endpoint (which may no longer exist).
>>> + *
>>> + * Message delivery into the recipient address space is ordered (per
>>> priority)
>>> + * and reliable. Delivery of message N implies delivery of all messages
>>> <N
>>> + * (of the same priority).
>>> + * All messages (accepted by MSGIO) will be delivered up to the point of
>>> + * recipient endpoint termination or lost connection where no more
>>> messages
>>> + * will be delivered.
>>> + *
>>> + * Actual reception (dequeueing) and processing by the recipient is not
>>> + * guaranteed (use user level acknowledgements for that).
>>> + *
>>> + * Monitor the remote endpoint to detect the disappearance or
>>> disconnection
>>> + * of the endpoint.
>>> + *
>>> + * A message may temporarily not be accepted for transmission (e.g. due
>>> + * to out of local buffer space), -1 will be returned and ODP errno set
>>> + * to EAGAIN.
>>> + *
>>> + * @param     msgio Message I/O handle
>>> + * @param     msg   Message to send
>>> + * @param     prio  Priority of message
>>> + * @param[in] addr  Address of recipient endpoint
>>> + *
>>> + * @retval 0 on success
>>> + * @retval <0 on error
>>> + */
>>> +int odp_msgio_send(odp_msgio_t msgio,
>>> +                  odp_message_t msg,
>>> +                  odp_msgio_prio_t prio,
>>> +                  const uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
>>> +
>>> +/**
>>> + * Get printable value for an odp_msgio_t
>>> + *
>>> + * @param msgio  Message I/O handle to be printed
>>> + * @return       uint64_t value that can be used to print/display this
>>> + *              handle
>>> + *
>>> + * @note This routine is intended to be used for diagnostic purposes
>>> + * to enable applications to generate a printable value that represents
>>> + * an odp_msgio_t handle.
>>> + */
>>> +uint64_t odp_msgio_to_u64(odp_msgio_t msgio);
>>> +
>>> +/**
>>> + * @}
>>> + */
>>> +
>>> +#ifdef __cplusplus
>>> +}
>>> +#endif
>>> +
>>> +#endif
>>> diff --git a/platform/linux-generic/include/odp/message.h
>>> b/platform/linux-generic/include/odp/message.h
>>> new file mode 100644
>>> index 0000000..2a29f59
>>> --- /dev/null
>>> +++ b/platform/linux-generic/include/odp/message.h
>>> @@ -0,0 +1,39 @@
>>> +/* Copyright (c) 2013, Linaro Limited
>>> + * All rights reserved.
>>> + *
>>> + * SPDX-License-Identifier:     BSD-3-Clause
>>> + */
>>> +
>>> +/**
>>> + * @file
>>> + *
>>> + * ODP Message Bus
>>> + */
>>> +
>>> +#ifndef ODP_PLAT_MESSAGE_H_
>>> +#define ODP_PLAT_MESSAGE_H_
>>> +
>>> +#ifdef __cplusplus
>>> +extern "C" {
>>> +#endif
>>> +
>>> +#include <odp/std_types.h>
>>> +#include <odp/plat/pool_types.h>
>>> +#include <odp/plat/message_types.h>
>>> +#include <odp/plat/queue_types.h>
>>> +
>>> +/** @ingroup odp_message
>>> + *  @{
>>> + */
>>> +
>>> +/**
>>> + * @}
>>> + */
>>> +
>>> +#include <odp/api/message.h>
>>> +
>>> +#ifdef __cplusplus
>>> +}
>>> +#endif
>>> +
>>> +#endif
>>> diff --git a/platform/linux-generic/include/odp/msgio.h
>>> b/platform/linux-generic/include/odp/msgio.h
>>> new file mode 100644
>>> index 0000000..f45b3a2
>>> --- /dev/null
>>> +++ b/platform/linux-generic/include/odp/msgio.h
>>> @@ -0,0 +1,40 @@
>>> +/* Copyright (c) 2013, Linaro Limited
>>> + * All rights reserved.
>>> + *
>>> + * SPDX-License-Identifier:     BSD-3-Clause
>>> + */
>>> +
>>> +/**
>>> + * @file
>>> + *
>>> + * ODP Message Bus
>>> + */
>>> +
>>> +#ifndef ODP_PLAT_MSGIO_H_
>>> +#define ODP_PLAT_MSGIO_H_
>>> +
>>> +#ifdef __cplusplus
>>> +extern "C" {
>>> +#endif
>>> +
>>> +#include <odp/std_types.h>
>>> +#include <odp/plat/pool_types.h>
>>> +#include <odp/plat/msgio_types.h>
>>> +#include <odp/plat/message_types.h>
>>> +#include <odp/plat/queue_types.h>
>>> +
>>> +/** @ingroup odp_msgio
>>> + *  @{
>>> + */
>>> +
>>> +/**
>>> + * @}
>>> + */
>>> +
>>> +#include <odp/api/msgio.h>
>>> +
>>> +#ifdef __cplusplus
>>> +}
>>> +#endif
>>> +
>>> +#endif
>>> diff --git a/platform/linux-generic/include/odp/plat/message_types.h
>>> b/platform/linux-generic/include/odp/plat/message_types.h
>>> new file mode 100644
>>> index 0000000..3a42064
>>> --- /dev/null
>>> +++ b/platform/linux-generic/include/odp/plat/message_types.h
>>> @@ -0,0 +1,47 @@
>>> +/* Copyright (c) 2015, Linaro Limited
>>> + * All rights reserved.
>>> + *
>>> + * SPDX-License-Identifier: BSD-3-Clause
>>> + */
>>> +
>>> +
>>> +/**
>>> + * @file
>>> + *
>>> + * ODP Message message type
>>> + */
>>> +
>>> +#ifndef ODP_MESSAGE_TYPES_H_
>>> +#define ODP_MESSAGE_TYPES_H_
>>> +
>>> +#ifdef __cplusplus
>>> +extern "C" {
>>> +#endif
>>> +
>>> +#include <odp/std_types.h>
>>> +#include <odp/plat/strong_types.h>
>>> +
>>> +/** @addtogroup odp_message ODP message
>>> + *  Operations on a message.
>>> + *  @{
>>> + */
>>> +
>>> +typedef ODP_HANDLE_T(odp_message_t);
>>> +
>>> +#define ODP_MESSAGE_INVALID _odp_cast_scalar(odp_message_t, 0xffffffff)
>>> +
>>> +/** Get printable format of odp_message_t */
>>> +static inline uint64_t odp_message_to_u64(odp_message_t hdl)
>>> +{
>>> +       return _odp_pri(hdl);
>>> +}
>>> +
>>> +/**
>>> + * @}
>>> + */
>>> +
>>> +#ifdef __cplusplus
>>> +}
>>> +#endif
>>> +
>>> +#endif
>>> diff --git a/platform/linux-generic/include/odp/plat/msgio_types.h
>>> b/platform/linux-generic/include/odp/plat/msgio_types.h
>>> new file mode 100644
>>> index 0000000..9d9f9da
>>> --- /dev/null
>>> +++ b/platform/linux-generic/include/odp/plat/msgio_types.h
>>> @@ -0,0 +1,59 @@
>>> +/* Copyright (c) 2015, Linaro Limited
>>> + * All rights reserved.
>>> + *
>>> + * SPDX-License-Identifier: BSD-3-Clause
>>> + */
>>> +
>>> +
>>> +/**
>>> + * @file
>>> + *
>>> + * ODP Message bus types
>>> + */
>>> +
>>> +#ifndef ODP_MSGIO_TYPES_H_
>>> +#define ODP_MSGIO_TYPES_H_
>>> +
>>> +#ifdef __cplusplus
>>> +extern "C" {
>>> +#endif
>>> +
>>> +#include <odp/std_types.h>
>>> +#include <odp/plat/strong_types.h>
>>> +
>>> +/** @addtogroup odp_msgio ODP message bus
>>> + *  Operations on a message bus.
>>> + *  @{
>>> + */
>>> +
>>> +typedef int odp_msgio_prio_t;
>>> +
>>> +#define ODP_MSGIO_PRIO_HIGHEST  0
>>> +
>>> +#define ODP_MSGIO_PRIO_NORMAL   2
>>> +
>>> +#define ODP_MSGIO_PRIO_LOWEST   3
>>> +
>>> +#define ODP_MSGIO_PRIO_DEFAULT  ODP_MSGIO_PRIO_NORMAL
>>> +
>>> +typedef ODP_HANDLE_T(odp_msgio_t);
>>> +
>>> +#define ODP_MSGIO_INVALID _odp_cast_scalar(odp_msgio_t, 0xffffffff)
>>> +
>>> +#define ODP_MSGIO_ADDR_SIZE 6
>>> +
>>> +/** Get printable format of odp_msgio_t */
>>> +static inline uint64_t odp_msgio_to_u64(odp_msgio_t hdl)
>>> +{
>>> +       return _odp_pri(hdl);
>>> +}
>>> +
>>> +/**
>>> + * @}
>>> + */
>>> +
>>> +#ifdef __cplusplus
>>> +}
>>> +#endif
>>> +
>>> +#endif
>>> --
>>> 1.9.1
>>>
>>> _______________________________________________
>>> lng-odp mailing list
>>> lng-odp@lists.linaro.org
>>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>>
>>
>>
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/lng-odp
>
>

On 27 May 2015 at 14:39, Bill Fischofer <bill.fischofer@linaro.org> wrote:

> If you want to use an Ethernet-like internal transport, you might was well
> use a standard one like TIPC <http://tipc.sourceforge.net/> rather than
> reinventing this particular wheel.
>
I don't want to use or enforce an Ethernet-like internal transport. I am
just proposing a standardized message format (user payload + metadata) for
connecting to the message network/bus so that ODP applications can be
binary interoperable regardless of which ODP implementation they are using.
Everyone seems to be misunderstanding my intention here. I could define
this standard format without trying to emulate any known packet format,
would that be better?

I am aware of TIPC on a high level but isn't the semantics a bit different
from what I propose? E.g. it uses well-known (numeric) ports for
applications to listen to? Does it have a suitable packet format that can
be reused?



>
> On Wed, May 27, 2015 at 4:49 AM, Ola Liljedahl <ola.liljedahl@linaro.org>
> wrote:
>
>> On 27 May 2015 at 11:10, Alexandru Badicioiu <
>> alexandru.badicioiu@linaro.org> wrote:
>>
>>>
>>>
>>> On 27 May 2015 at 11:08, Ola Liljedahl <ola.liljedahl@linaro.org> wrote:
>>>
>>>> Here is my third attempt at a ODP API for MSGIO - asynchronous message
>>>> passing
>>>> based IPC for a shared nothing architecture.
>>>>
>>>> IPC/message passing can be separated into a number of different layers.
>>>> 1) Message formats and protocols (as seen by the user).
>>>> 2) API - defines syntax and semantics of MSGIO operations.
>>>> 3) Transport interface - defines a binary packet format used by MSGIO
>>>>    endpoints/implementations. Cf. OSI network layer.
>>>> 4) Transport layer - implements the transport of message packet between
>>>>    different endpoints. Cf. OSI link layer.
>>>>
>>>> #1 is application specific and there are many different standards (e.g.
>>>> Linux
>>>> Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
>>>> #3 may be implementation specific but is important for
>>>> interoperatibility
>>>> (binary compatibility). If the user message (payload), operation and
>>>> parameters are encoded in a well-defined way (e.g. some kind of packet
>>>> format
>>>> which includes operation, sender and receiver addresses, payload etc),
>>>> inter-operatibility between different ODP/MSGIO implementations may be
>>>> achieved or simplified.
>>>> #4 is implementation (platform) specific.
>>>>
>>>> This proposal focuses on the API which is fundamental for source code
>>>> compatibility between different platforms (ODP implementations) - this
>>>> being
>>>> the primary purpose of ODP. A packet format for messages will be
>>>> specified as
>>>> part of the reference implementation and may be seen as a proposal for a
>>>> standard packet format for binary compatibility between different
>>>> ODP/MSGIO
>>>> implementations.
>>>>
>>>> All MSGIO calls are non-blocking (with the possible exception of
>>>> odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
>>>> transfer is asynchronous so it is not possible to return any status
>>>> operation concerning the eventual (or remote) result of an operation.
>>>>
>>>> The semantics of MSGIO message passing is that sending a message to an
>>>> endpoint
>>>> will always look like it succeeds. The appearance of endpoints is
>>>> explicitly
>>>> notified through user-defined messages specified in the
>>>> odp_msgio_lookup()
>>>> call. Similarly, the disappearance (e.g. death of or otherwise lost
>>>> connection
>>>> to endpoint) is also explicitly notified through user-defined messages
>>>> specified in the odp_msgio_monitor() call. The send call does not fail
>>>> because
>>>> the addressed endpoint has disappeared, the message is just silently
>>>> discarded.
>>>>
>>>> Message delivery into the recipient address space is ordered (per
>>>> priority)
>>>> and reliable. Delivery of message N implies delivery of all messages <N
>>>> (of the same priority). All messages (accepted by MSGIO send operation)
>>>> will
>>>> be delivered up to the point of endpoint termination or lost connection
>>>> where no more messages will be delivered. Actual reception (dequeueing)
>>>> and
>>>> processing by the recipient is not guaranteed (use end-to-end
>>>> acknowledgements
>>>> for that).
>>>>
>>>> MSGIO endpoints can be seen as interfaces (taps) to an internal reliable
>>>> multidrop network where each endpoint has a unique address which is only
>>>> valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
>>>> and then recreated (with the same name), the new endpoint will have a
>>>> new unique address (eventually endpoints addresses will have to be
>>>> recycled
>>>> but not for a very long time). Endpoints names do not necessarily have
>>>> to be
>>>> unique. The scope of message network is not defined but it is expected
>>>> that
>>>> the scope by default corresponds to an OS instance (e.g. virtual
>>>> machine).
>>>>
>>>> Proposed transport packet for message I/O (not visible to the user):
>>>> uint8_t dest_addr[6]; /* Unique address of destination endpoint */
>>>> uint8_t src_addr[6]; /* Unique address of source endpoint */
>>>> uint16_t frametype_o; /* Outer frametype, 0x8100 */
>>>> uint16_t pcp_vid; /* 3 bits of Priority Code Point, 5 bits reserved */
>>>> uint16_t frametype_i; /* Inner frametype, TBD */
>>>> uint8_t payload[]; /* User message */
>>>> Looks very much like a VLAN-tagged Ethernet frame so should easily be
>>>> transported by and processed by Ethernet HW & SW
>>>
>>> Why to use Ethernet HW & SW to transport messages inside an OS instance?
>>> Usually Ethernet HW & SW (drivers?) deal with sending/receiving to/from the
>>> wire.
>>>
>> I don't say you have to. I am just suggesting that *if* we standardize
>> the "packet" (message payload + metadata) format, we might as well use a
>> well-known format that likely can be processed by preexisting HW and SW
>> blocks. Possibly this will simplify classification, queueing and scheduling
>> of messages.
>>
>>
>>
>>> An above requirement was referring to ordering which must be defined by
>>> the transport layer - how this format ensures ordered delivery at the
>>> destination?
>>>
>> The packet format is just the format of the message payload + metadata as
>> passed to and from the transport layer.
>>
>> The transport layer can (if necessary) add its own encapsulation and
>> state machine that allows for ordered reliable delivery. That's nothing
>> that has to be seen by the user of the transport layer. The users of TCP
>> don't see the TCP header or state machine.
>>
>>
>> .
>>>> Encoding of publish/lookup/monitor operations TBD.
>>>>
>>>> v3:
>>>> Renamed Message Bus (MBUS) to Message I/O (MSGIO).
>>>> Changed all definitions to use "msgio" instead of "mbus" prefix/infix.
>>>> odp_msgio_create(): removed default input queue parameter.
>>>> Removed odp_mbus_inq_setdef() and odp_mbus_inq_remdef().
>>>> All input queues must now be specified using odp_msgio_inq_set() before
>>>> the MSGIO endpoint is activated.
>>>> Added odp_msgio_activate().
>>>> Added odp_message_push_head() and odp_message_pull_head().
>>>> Updated some function descriptions in msgio.h.
>>>>
>>>> v2:
>>>> Split off all message definitions in a separate file message.h
>>>> Renamed Inter Process Communication (IPC) to Message Bus (MBUS).
>>>> Changed all definitions to use "mbus" instead of "ipc" prefix/infix.
>>>> Renamed odp_ipc_msg_t to odp_message_t.
>>>> odp_mbus_create(): Added parameter for default input queue. Explicitly
>>>> state
>>>> that the pool must be use type ODP_EVENT_MESSAGE.
>>>> Renamed odp_ipc_resolve() to odp_mbus_lookup().
>>>> odp_mbus_send(): Added priority parameter.
>>>> Renamed odp_ipc_sender() to odp_message_sender().
>>>> Renamed odp_ipc_data() to odp_message_data().
>>>> Renamed odp_ipc_length() to odp_message_length().
>>>> Renamed odp_ipc_reset() to odp_message_length_set().
>>>> Renamed odp_ipc_alloc() to odp_message_alloc().
>>>> Renamed odp_ipc_free() to odp_message_free().
>>>> odp_message_alloc(): Corrected name of invalid message handle.
>>>> Added message priorities and calls to set and remove input queues for
>>>> specific priorities: odp_mbus_inq_set(), odp_mbus_inq_rem().
>>>>
>>>> Signed-off-by: Ola Liljedahl <ola.liljedahl@linaro.org>
>>>> ---
>>>> (This document/code contribution attached is provided under the terms of
>>>> agreement LES-LTM-21309)
>>>>
>>>>  include/odp/api/message.h                          | 199
>>>> ++++++++++++++++++
>>>>  include/odp/api/msgio.h                            | 222
>>>> +++++++++++++++++++++
>>>>  platform/linux-generic/include/odp/message.h       |  39 ++++
>>>>  platform/linux-generic/include/odp/msgio.h         |  40 ++++
>>>>  .../linux-generic/include/odp/plat/message_types.h |  47 +++++
>>>>  .../linux-generic/include/odp/plat/msgio_types.h   |  59 ++++++
>>>>  6 files changed, 606 insertions(+)
>>>>  create mode 100644 include/odp/api/message.h
>>>>  create mode 100644 include/odp/api/msgio.h
>>>>  create mode 100644 platform/linux-generic/include/odp/message.h
>>>>  create mode 100644 platform/linux-generic/include/odp/msgio.h
>>>>  create mode 100644
>>>> platform/linux-generic/include/odp/plat/message_types.h
>>>>  create mode 100644
>>>> platform/linux-generic/include/odp/plat/msgio_types.h
>>>>
>>>> diff --git a/include/odp/api/message.h b/include/odp/api/message.h
>>>> new file mode 100644
>>>> index 0000000..76697fb
>>>> --- /dev/null
>>>> +++ b/include/odp/api/message.h
>>>> @@ -0,0 +1,199 @@
>>>> +/* Copyright (c) 2015, Linaro Limited
>>>> + * All rights reserved.
>>>> + *
>>>> + * SPDX-License-Identifier:     BSD-3-Clause
>>>> + */
>>>> +
>>>> +
>>>> +/**
>>>> + * @file
>>>> + *
>>>> + * ODP Message event type
>>>> + */
>>>> +
>>>> +#ifndef ODP_API_MESSAGE_H_
>>>> +#define ODP_API_MESSAGE_H_
>>>> +
>>>> +#ifdef __cplusplus
>>>> +extern "C" {
>>>> +#endif
>>>> +
>>>> +/** @defgroup odp_message ODP MESSAGE
>>>> + *  @{
>>>> + */
>>>> +
>>>> +/**
>>>> + * @typedef odp_message_t
>>>> + * ODP message handle
>>>> + */
>>>> +
>>>> +
>>>> +/**
>>>> + * Get address of sender (source) of message
>>>> + *
>>>> + * @param      msg  Message handle
>>>> + * @param[out] addr Buffer to hold sender address
>>>> + */
>>>> +void odp_message_sender(odp_message_t msg,
>>>> +                       uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
>>>> +
>>>> +/**
>>>> + * Message data pointer
>>>> + *
>>>> + * Return a pointer to the message data
>>>> + *
>>>> + * @param msg Message handle
>>>> + *
>>>> + * @return Pointer to the message data
>>>> + */
>>>> +void *odp_message_data(odp_message_t msg);
>>>> +
>>>> +/**
>>>> + * Message data length
>>>> + *
>>>> + * Return length of the message data.
>>>> + *
>>>> + * @param msg Message handle
>>>> + *
>>>> + * @return Message length
>>>> + */
>>>> +uint32_t odp_message_length(const odp_message_t msg);
>>>> +
>>>> +/**
>>>> + * Set message length
>>>> + *
>>>> + * Set length of the message data.
>>>> + * Increase message data length by moving message tail into message
>>>> tailroom.
>>>> + * The actual message data is not modified.
>>>> + *
>>>> + * @param msg Message handle
>>>> + * @param len New length
>>>> + *
>>>> + * @retval 0 on success
>>>> + * @retval <0 on error (e.g. no enough tailroom)
>>>> + */
>>>> +int odp_message_length_set(const odp_message_t msg, uint32_t len);
>>>> +
>>>> +/**
>>>> + * Message headroom length
>>>> + *
>>>> + * Return length of the message headroom
>>>> + *
>>>> + * @param msg Message handle
>>>> + *
>>>> + * @return Headroom length
>>>> + */
>>>> +uint32_t odp_message_headroom(const odp_message_t msg);
>>>> +
>>>> +/**
>>>> + * Push out message head
>>>> + *
>>>> + * Increase message data length by moving message head into message
>>>> headroom.
>>>> + * Message headroom is decreased with the same amount. The message
>>>> head may be
>>>> + * pushed out up to 'headroom' bytes. Message is not modified if
>>>> there's not
>>>> + * enough headroom space.
>>>> + *
>>>> + * odp_message_xxx:
>>>> + * length   += len
>>>> + * headroom -= len
>>>> + * data     -= len
>>>> + *
>>>> + * @param msg  Message handle
>>>> + * @param len  Number of bytes to push the head (0 ... headroom)
>>>> + *
>>>> + * @return The new data pointer
>>>> + * @retval NULL  Requested offset exceeds available headroom
>>>> + *
>>>> + * @see odp_message_headroom(), odp_message_pull_head()
>>>> + */
>>>> +void *odp_message_push_head(odp_message_t msg, uint32_t len);
>>>> +
>>>> +/**
>>>> + * Pull in message head
>>>> + *
>>>> + * Decrease message data length by moving message head into message
>>>> data.
>>>> + * Message headroom is increased with the same amount. The message
>>>> head may be
>>>> + * pulled in up to 'length' bytes. Message is not modified if there's
>>>> not
>>>> + * enough data.
>>>> + *
>>>> + * odp_message_xxx:
>>>> + * length   -= len
>>>> + * headroom += len
>>>> + * data     += len
>>>> + *
>>>> + * @param msg  Message handle
>>>> + * @param len  Number of bytes to pull the head (0 ... length)
>>>> + *
>>>> + * @return The new data pointer
>>>> + * @retval NULL  Requested offset exceeds available data
>>>> + *
>>>> + * @see odp_message_headroom(), odp_message_push_head()
>>>> + */
>>>> +void *odp_message_pull_head(odp_message_t msg, uint32_t len);
>>>> +
>>>> +/**
>>>> + * Allocate message
>>>> + *
>>>> + * Allocate a message of a specific length.
>>>> + *
>>>> + * @param pool Message pool to allocate message from
>>>> + * @param len Length of the allocated message
>>>> + *
>>>> + * @return Message handle on success
>>>> + * @retval ODP_MESSAGE_INVALID on failure and errno set
>>>> + */
>>>> +odp_message_t odp_message_alloc(odp_pool_t pool, uint32_t len);
>>>> +
>>>> +/**
>>>> + * Free message
>>>> + *
>>>> + * Free message back to the message pool it was allocated from.
>>>> + *
>>>> + * @param msg Handle of message to free
>>>> + */
>>>> +void odp_message_free(odp_message_t msg);
>>>> +
>>>> +/**
>>>> + * Get message handle from event
>>>> + *
>>>> + * Converts an ODP_EVENT_MESSAGE type event to a message.
>>>> + *
>>>> + * @param ev   Event handle representing a message.
>>>> + *
>>>> + * @return Message handle
>>>> + *
>>>> + * @see odp_event_type()
>>>> + */
>>>> +odp_message_t odp_message_from_event(odp_event_t ev);
>>>> +
>>>> +/**
>>>> + * Convert message handle to event
>>>> + *
>>>> + * @param msg  Message handle
>>>> + *
>>>> + * @return Event handle
>>>> + */
>>>> +odp_event_t odp_message_to_event(odp_message_t msg);
>>>> +
>>>> +/**
>>>> + * Get printable value for an odp_message_t
>>>> + *
>>>> + * @param msg  Message handle to be printed
>>>> + * @return     uint64_t value that can be used to print/display this
>>>> + *            handle
>>>> + *
>>>> + * @note This routine is intended to be used for diagnostic purposes
>>>> + * to enable applications to generate a printable value that represents
>>>> + * an odp_message_t handle.
>>>> + */
>>>> +uint64_t odp_message_to_u64(odp_message_t msg);
>>>> +
>>>> +/**
>>>> + * @}
>>>> + */
>>>> +
>>>> +#ifdef __cplusplus
>>>> +}
>>>> +#endif
>>>> +
>>>> +#endif
>>>> diff --git a/include/odp/api/msgio.h b/include/odp/api/msgio.h
>>>> new file mode 100644
>>>> index 0000000..8d73021
>>>> --- /dev/null
>>>> +++ b/include/odp/api/msgio.h
>>>> @@ -0,0 +1,222 @@
>>>> +/* Copyright (c) 2015, Linaro Limited
>>>> + * All rights reserved.
>>>> + *
>>>> + * SPDX-License-Identifier:     BSD-3-Clause
>>>> + */
>>>> +
>>>> +
>>>> +/**
>>>> + * @file
>>>> + *
>>>> + * ODP Message I/O API
>>>> + */
>>>> +
>>>> +#ifndef ODP_API_MSGIO_H_
>>>> +#define ODP_API_MSGIO_H_
>>>> +
>>>> +#ifdef __cplusplus
>>>> +extern "C" {
>>>> +#endif
>>>> +
>>>> +/** @defgroup odp_msgio ODP MSGIO
>>>> + *  @{
>>>> + */
>>>> +
>>>> +/**
>>>> + * @typedef odp_msgio_t
>>>> + * ODP message I/O handle
>>>> + */
>>>> +
>>>> +/**
>>>> + * @def ODP_MSGIO_ADDR_SIZE
>>>> + * Size of the address of a message I/O endpoint
>>>> + */
>>>> +
>>>> +/**
>>>> + * @typedef odp_msgio_prio_t
>>>> + * ODP MSGIO message priority
>>>> + */
>>>> +
>>>> +/**
>>>> + * @def ODP_MSGIO_PRIO_HIGHEST
>>>> + * Highest MSGIO message priority
>>>> + */
>>>> +
>>>> +/**
>>>> + * @def ODP_MSGIO_PRIO_NORMAL
>>>> + * Normal MSGIO message priority
>>>> + */
>>>> +
>>>> +/**
>>>> + * @def ODP_MSGIO_PRIO_LOWEST
>>>> + * Lowest MSGIO message priority
>>>> + */
>>>> +
>>>> +/**
>>>> + * @def ODP_MSGIO_PRIO_DEFAULT
>>>> + * Default MSGIO message priority
>>>> + */
>>>> +
>>>> +
>>>> +/**
>>>> + * Create message I/O endpoint
>>>> + *
>>>> + * Create an message I/O endpoint. The scope of the message bus is
>>>> + * not defined but it is expected that it by default encompasses the OS
>>>> + * instance but no more.
>>>> + *
>>>> + * A unique address for the endpoint is created. The addresses for
>>>> other
>>>> + * endpoints can be obtained using the odp_msgio_lookup() call and
>>>> specified
>>>> + * in the odp_msgio_send() call.
>>>> + *
>>>> + * @param name Name of our endpoint
>>>> + * @param pool Pool (of type ODP_EVENT_MESSAGE) for incoming messages
>>>> + *
>>>> + * @return Message I/O handle on success
>>>> + * @retval ODP_MSGIO_INVALID on failure and errno set
>>>> + */
>>>> +odp_msgio_t odp_msgio_create(const char *name,
>>>> +                            odp_pool_t pool);
>>>> +
>>>> +/**
>>>> + * Set the input queue for a specific message priority
>>>> + *
>>>> + * @param msgio  Message I/O handle
>>>> + * @param prio  Message priority
>>>> + * @param queue Queue handle
>>>> + *
>>>> + * @retval  0 on success
>>>> + * @retval <0 on failure
>>>> + */
>>>> +int odp_msgio_inq_set(odp_msgio_t msgio,
>>>> +                     odp_msgio_prio_t prio,
>>>> +                     odp_queue_t queue);
>>>> +
>>>> +/**
>>>> + * Activate message I/O endpoint
>>>> + *
>>>> + * Make this endpoint visible (odp_msgio_lookup() messages will be
>>>> returned).
>>>> + * Enable transmission and reception of messages through the endpoint.
>>>> + *
>>>> + * @param msgio Message I/O handle
>>>> + *
>>>> + * @retval 0 on success
>>>> + * @retval <0 on failure
>>>> + */
>>>> +int odp_msgio_activate(odp_msgio_t msgio);
>>>> +
>>>> +/**
>>>> + * Destroy message I/O endpoint
>>>> + *
>>>> + * @param msgio Message I/O handle
>>>> + *
>>>> + * @retval 0 on success
>>>> + * @retval <0 on failure
>>>> + */
>>>> +int odp_msgio_destroy(odp_msgio_t msgio);
>>>> +
>>>> +/**
>>>> + * Remove the input queue for a specific message priority
>>>> + *
>>>> + * Remove (disassociate) the matching input queue from a message I/O
>>>> endpoint.
>>>> + * The queue itself is not touched.
>>>> + *
>>>> + * @param msgio  Message I/O handle
>>>> + * @param prio  Message priority
>>>> + *
>>>> + * @retval 0 on success
>>>> + * @retval <0 on failure
>>>> + */
>>>> +int odp_msgio_inq_rem(odp_msgio_t msgio,
>>>> +                     odp_msgio_prio_t prio);
>>>> +
>>>> +/**
>>>> + * Lookup endpoint by name
>>>> + *
>>>> + * Look up a current or future endpoint by name.
>>>> + * When the endpoint exists, return the specified message (not
>>>> necessarily the
>>>> + * same message handle) with the endpoint as the sender.
>>>> + *
>>>> + * @param msgio Message I/O handle
>>>> + * @param name Name to look up
>>>> + * @param msg Message to return
>>>> + */
>>>> +void odp_msgio_lookup(odp_msgio_t msgio,
>>>> +                     const char *name,
>>>> +                     odp_message_t msg);
>>>> +
>>>> +/**
>>>> + * Monitor endpoint
>>>> + *
>>>> + * Monitor an endpoint (which may already have disappeared).
>>>> + * When the endpoint disappears, return the specified message (not
>>>> necessarily
>>>> + * the same message handle) with the endpoint as the sender.
>>>> + *
>>>> + * Unrecognized or invalid endpoint addresses are treated as
>>>> non-existing
>>>> + * endpoints.
>>>> + *
>>>> + * @param     msgio Message I/O handle
>>>> + * @param[in] addr  Address of monitored endpoint
>>>> + * @param     msg   Message to return
>>>> + */
>>>> +void odp_msgio_monitor(odp_msgio_t msgio,
>>>> +                      const uint8_t addr[ODP_MSGIO_ADDR_SIZE],
>>>> +                      odp_message_t msg);
>>>> +
>>>> +/**
>>>> + * Send message
>>>> + *
>>>> + * Send a message to an endpoint (which may no longer exist).
>>>> + *
>>>> + * Message delivery into the recipient address space is ordered (per
>>>> priority)
>>>> + * and reliable. Delivery of message N implies delivery of all
>>>> messages <N
>>>> + * (of the same priority).
>>>> + * All messages (accepted by MSGIO) will be delivered up to the point
>>>> of
>>>> + * recipient endpoint termination or lost connection where no more
>>>> messages
>>>> + * will be delivered.
>>>> + *
>>>> + * Actual reception (dequeueing) and processing by the recipient is not
>>>> + * guaranteed (use user level acknowledgements for that).
>>>> + *
>>>> + * Monitor the remote endpoint to detect the disappearance or
>>>> disconnection
>>>> + * of the endpoint.
>>>> + *
>>>> + * A message may temporarily not be accepted for transmission (e.g. due
>>>> + * to out of local buffer space), -1 will be returned and ODP errno set
>>>> + * to EAGAIN.
>>>> + *
>>>> + * @param     msgio Message I/O handle
>>>> + * @param     msg   Message to send
>>>> + * @param     prio  Priority of message
>>>> + * @param[in] addr  Address of recipient endpoint
>>>> + *
>>>> + * @retval 0 on success
>>>> + * @retval <0 on error
>>>> + */
>>>> +int odp_msgio_send(odp_msgio_t msgio,
>>>> +                  odp_message_t msg,
>>>> +                  odp_msgio_prio_t prio,
>>>> +                  const uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
>>>> +
>>>> +/**
>>>> + * Get printable value for an odp_msgio_t
>>>> + *
>>>> + * @param msgio  Message I/O handle to be printed
>>>> + * @return       uint64_t value that can be used to print/display this
>>>> + *              handle
>>>> + *
>>>> + * @note This routine is intended to be used for diagnostic purposes
>>>> + * to enable applications to generate a printable value that represents
>>>> + * an odp_msgio_t handle.
>>>> + */
>>>> +uint64_t odp_msgio_to_u64(odp_msgio_t msgio);
>>>> +
>>>> +/**
>>>> + * @}
>>>> + */
>>>> +
>>>> +#ifdef __cplusplus
>>>> +}
>>>> +#endif
>>>> +
>>>> +#endif
>>>> diff --git a/platform/linux-generic/include/odp/message.h
>>>> b/platform/linux-generic/include/odp/message.h
>>>> new file mode 100644
>>>> index 0000000..2a29f59
>>>> --- /dev/null
>>>> +++ b/platform/linux-generic/include/odp/message.h
>>>> @@ -0,0 +1,39 @@
>>>> +/* Copyright (c) 2013, Linaro Limited
>>>> + * All rights reserved.
>>>> + *
>>>> + * SPDX-License-Identifier:     BSD-3-Clause
>>>> + */
>>>> +
>>>> +/**
>>>> + * @file
>>>> + *
>>>> + * ODP Message Bus
>>>> + */
>>>> +
>>>> +#ifndef ODP_PLAT_MESSAGE_H_
>>>> +#define ODP_PLAT_MESSAGE_H_
>>>> +
>>>> +#ifdef __cplusplus
>>>> +extern "C" {
>>>> +#endif
>>>> +
>>>> +#include <odp/std_types.h>
>>>> +#include <odp/plat/pool_types.h>
>>>> +#include <odp/plat/message_types.h>
>>>> +#include <odp/plat/queue_types.h>
>>>> +
>>>> +/** @ingroup odp_message
>>>> + *  @{
>>>> + */
>>>> +
>>>> +/**
>>>> + * @}
>>>> + */
>>>> +
>>>> +#include <odp/api/message.h>
>>>> +
>>>> +#ifdef __cplusplus
>>>> +}
>>>> +#endif
>>>> +
>>>> +#endif
>>>> diff --git a/platform/linux-generic/include/odp/msgio.h
>>>> b/platform/linux-generic/include/odp/msgio.h
>>>> new file mode 100644
>>>> index 0000000..f45b3a2
>>>> --- /dev/null
>>>> +++ b/platform/linux-generic/include/odp/msgio.h
>>>> @@ -0,0 +1,40 @@
>>>> +/* Copyright (c) 2013, Linaro Limited
>>>> + * All rights reserved.
>>>> + *
>>>> + * SPDX-License-Identifier:     BSD-3-Clause
>>>> + */
>>>> +
>>>> +/**
>>>> + * @file
>>>> + *
>>>> + * ODP Message Bus
>>>> + */
>>>> +
>>>> +#ifndef ODP_PLAT_MSGIO_H_
>>>> +#define ODP_PLAT_MSGIO_H_
>>>> +
>>>> +#ifdef __cplusplus
>>>> +extern "C" {
>>>> +#endif
>>>> +
>>>> +#include <odp/std_types.h>
>>>> +#include <odp/plat/pool_types.h>
>>>> +#include <odp/plat/msgio_types.h>
>>>> +#include <odp/plat/message_types.h>
>>>> +#include <odp/plat/queue_types.h>
>>>> +
>>>> +/** @ingroup odp_msgio
>>>> + *  @{
>>>> + */
>>>> +
>>>> +/**
>>>> + * @}
>>>> + */
>>>> +
>>>> +#include <odp/api/msgio.h>
>>>> +
>>>> +#ifdef __cplusplus
>>>> +}
>>>> +#endif
>>>> +
>>>> +#endif
>>>> diff --git a/platform/linux-generic/include/odp/plat/message_types.h
>>>> b/platform/linux-generic/include/odp/plat/message_types.h
>>>> new file mode 100644
>>>> index 0000000..3a42064
>>>> --- /dev/null
>>>> +++ b/platform/linux-generic/include/odp/plat/message_types.h
>>>> @@ -0,0 +1,47 @@
>>>> +/* Copyright (c) 2015, Linaro Limited
>>>> + * All rights reserved.
>>>> + *
>>>> + * SPDX-License-Identifier: BSD-3-Clause
>>>> + */
>>>> +
>>>> +
>>>> +/**
>>>> + * @file
>>>> + *
>>>> + * ODP Message message type
>>>> + */
>>>> +
>>>> +#ifndef ODP_MESSAGE_TYPES_H_
>>>> +#define ODP_MESSAGE_TYPES_H_
>>>> +
>>>> +#ifdef __cplusplus
>>>> +extern "C" {
>>>> +#endif
>>>> +
>>>> +#include <odp/std_types.h>
>>>> +#include <odp/plat/strong_types.h>
>>>> +
>>>> +/** @addtogroup odp_message ODP message
>>>> + *  Operations on a message.
>>>> + *  @{
>>>> + */
>>>> +
>>>> +typedef ODP_HANDLE_T(odp_message_t);
>>>> +
>>>> +#define ODP_MESSAGE_INVALID _odp_cast_scalar(odp_message_t, 0xffffffff)
>>>> +
>>>> +/** Get printable format of odp_message_t */
>>>> +static inline uint64_t odp_message_to_u64(odp_message_t hdl)
>>>> +{
>>>> +       return _odp_pri(hdl);
>>>> +}
>>>> +
>>>> +/**
>>>> + * @}
>>>> + */
>>>> +
>>>> +#ifdef __cplusplus
>>>> +}
>>>> +#endif
>>>> +
>>>> +#endif
>>>> diff --git a/platform/linux-generic/include/odp/plat/msgio_types.h
>>>> b/platform/linux-generic/include/odp/plat/msgio_types.h
>>>> new file mode 100644
>>>> index 0000000..9d9f9da
>>>> --- /dev/null
>>>> +++ b/platform/linux-generic/include/odp/plat/msgio_types.h
>>>> @@ -0,0 +1,59 @@
>>>> +/* Copyright (c) 2015, Linaro Limited
>>>> + * All rights reserved.
>>>> + *
>>>> + * SPDX-License-Identifier: BSD-3-Clause
>>>> + */
>>>> +
>>>> +
>>>> +/**
>>>> + * @file
>>>> + *
>>>> + * ODP Message bus types
>>>> + */
>>>> +
>>>> +#ifndef ODP_MSGIO_TYPES_H_
>>>> +#define ODP_MSGIO_TYPES_H_
>>>> +
>>>> +#ifdef __cplusplus
>>>> +extern "C" {
>>>> +#endif
>>>> +
>>>> +#include <odp/std_types.h>
>>>> +#include <odp/plat/strong_types.h>
>>>> +
>>>> +/** @addtogroup odp_msgio ODP message bus
>>>> + *  Operations on a message bus.
>>>> + *  @{
>>>> + */
>>>> +
>>>> +typedef int odp_msgio_prio_t;
>>>> +
>>>> +#define ODP_MSGIO_PRIO_HIGHEST  0
>>>> +
>>>> +#define ODP_MSGIO_PRIO_NORMAL   2
>>>> +
>>>> +#define ODP_MSGIO_PRIO_LOWEST   3
>>>> +
>>>> +#define ODP_MSGIO_PRIO_DEFAULT  ODP_MSGIO_PRIO_NORMAL
>>>> +
>>>> +typedef ODP_HANDLE_T(odp_msgio_t);
>>>> +
>>>> +#define ODP_MSGIO_INVALID _odp_cast_scalar(odp_msgio_t, 0xffffffff)
>>>> +
>>>> +#define ODP_MSGIO_ADDR_SIZE 6
>>>> +
>>>> +/** Get printable format of odp_msgio_t */
>>>> +static inline uint64_t odp_msgio_to_u64(odp_msgio_t hdl)
>>>> +{
>>>> +       return _odp_pri(hdl);
>>>> +}
>>>> +
>>>> +/**
>>>> + * @}
>>>> + */
>>>> +
>>>> +#ifdef __cplusplus
>>>> +}
>>>> +#endif
>>>> +
>>>> +#endif
>>>> --
>>>> 1.9.1
>>>>
>>>> _______________________________________________
>>>> lng-odp mailing list
>>>> lng-odp@lists.linaro.org
>>>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>>>
>>>
>>>
>>
>> _______________________________________________
>> lng-odp mailing list
>> lng-odp@lists.linaro.org
>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>
>>
>

On 2 June 2015 at 08:53, Jerin Jacob <jerin.jacob@caviumnetworks.com> wrote:

> On Tue, Jun 02, 2015 at 02:36:47PM +0200, Benoît Ganne wrote:
> > Hi Ola, all,
> >
> > On 05/27/2015 10:08 AM, Ola Liljedahl wrote:
> > >Here is my third attempt at a ODP API for MSGIO - asynchronous message
> passing
> > >based IPC for a shared nothing architecture.
> >
> > I understand the value of this, but I feel I need to ask the question:
> does
> > it really belong to ODP ? My understanding so far was that ODP should not
> > try to replace OS services: for example, thread or process related API
> were
> > ruled out. This looks to me a lot like D-BUS or ZeroMQ (or <name your
> > favorite solution here>). Wouldn't it be better to let the application
> > select what they want to use instead of providing our own?
>
> +1 and I am not sure how much value addition it has in hardware
> abstraction perspective.
>

Just a thought, but if it is not a HW abstraction that makes it a candidate
for a helper, assuming the API can support capabilities the helper will
need.


> Most of the HW accelerated platforms will map odp_msg_t as packet and
> odp_msgio_t as pktio
> then it will boils down to a helper function that creates the
> vlan packet/message from given message(uint8_t *), src, dst 48 bit address
>
> >
> > ben
> >
> > >IPC/message passing can be separated into a number of different layers.
> > >1) Message formats and protocols (as seen by the user).
> > >2) API - defines syntax and semantics of MSGIO operations.
> > >3) Transport interface - defines a binary packet format used by MSGIO
> > >    endpoints/implementations. Cf. OSI network layer.
> > >4) Transport layer - implements the transport of message packet between
> > >    different endpoints. Cf. OSI link layer.
> > >
> > >#1 is application specific and there are many different standards (e.g.
> Linux
> > >Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
> > >#3 may be implementation specific but is important for
> interoperatibility
> > >(binary compatibility). If the user message (payload), operation and
> > >parameters are encoded in a well-defined way (e.g. some kind of packet
> format
> > >which includes operation, sender and receiver addresses, payload etc),
> > >inter-operatibility between different ODP/MSGIO implementations may be
> > >achieved or simplified.
> > >#4 is implementation (platform) specific.
> > >
> > >This proposal focuses on the API which is fundamental for source code
> > >compatibility between different platforms (ODP implementations) - this
> being
> > >the primary purpose of ODP. A packet format for messages will be
> specified as
> > >part of the reference implementation and may be seen as a proposal for a
> > >standard packet format for binary compatibility between different
> ODP/MSGIO
> > >implementations.
> > >
> > >All MSGIO calls are non-blocking (with the possible exception of
> > >odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
> > >transfer is asynchronous so it is not possible to return any status
> > >operation concerning the eventual (or remote) result of an operation.
> > >
> > >The semantics of MSGIO message passing is that sending a message to an
> endpoint
> > >will always look like it succeeds. The appearance of endpoints is
> explicitly
> > >notified through user-defined messages specified in the
> odp_msgio_lookup()
> > >call. Similarly, the disappearance (e.g. death of or otherwise lost
> connection
> > >to endpoint) is also explicitly notified through user-defined messages
> > >specified in the odp_msgio_monitor() call. The send call does not fail
> because
> > >the addressed endpoint has disappeared, the message is just silently
> discarded.
> > >
> > >Message delivery into the recipient address space is ordered (per
> priority)
> > >and reliable. Delivery of message N implies delivery of all messages <N
> > >(of the same priority). All messages (accepted by MSGIO send operation)
> will
> > >be delivered up to the point of endpoint termination or lost connection
> > >where no more messages will be delivered. Actual reception (dequeueing)
> and
> > >processing by the recipient is not guaranteed (use end-to-end
> acknowledgements
> > >for that).
> > >
> > >MSGIO endpoints can be seen as interfaces (taps) to an internal reliable
> > >multidrop network where each endpoint has a unique address which is only
> > >valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
> > >and then recreated (with the same name), the new endpoint will have a
> > >new unique address (eventually endpoints addresses will have to be
> recycled
> > >but not for a very long time). Endpoints names do not necessarily have
> to be
> > >unique. The scope of message network is not defined but it is expected
> that
> > >the scope by default corresponds to an OS instance (e.g. virtual
> machine).
> > >
> > >Proposed transport packet for message I/O (not visible to the user):
> > >uint8_t dest_addr[6]; /* Unique address of destination endpoint */
> > >uint8_t src_addr[6]; /* Unique address of source endpoint */
> > >uint16_t frametype_o; /* Outer frametype, 0x8100 */
> > >uint16_t pcp_vid; /* 3 bits of Priority Code Point, 5 bits reserved */
> > >uint16_t frametype_i; /* Inner frametype, TBD */
> > >uint8_t payload[]; /* User message */
> > >Looks very much like a VLAN-tagged Ethernet frame so should easily be
> > >transported by and processed by Ethernet HW & SW.
> > >Encoding of publish/lookup/monitor operations TBD.
> > >
> > >v3:
> > >Renamed Message Bus (MBUS) to Message I/O (MSGIO).
> > >Changed all definitions to use "msgio" instead of "mbus" prefix/infix.
> > >odp_msgio_create(): removed default input queue parameter.
> > >Removed odp_mbus_inq_setdef() and odp_mbus_inq_remdef().
> > >All input queues must now be specified using odp_msgio_inq_set() before
> > >the MSGIO endpoint is activated.
> > >Added odp_msgio_activate().
> > >Added odp_message_push_head() and odp_message_pull_head().
> > >Updated some function descriptions in msgio.h.
> > >
> > >v2:
> > >Split off all message definitions in a separate file message.h
> > >Renamed Inter Process Communication (IPC) to Message Bus (MBUS).
> > >Changed all definitions to use "mbus" instead of "ipc" prefix/infix.
> > >Renamed odp_ipc_msg_t to odp_message_t.
> > >odp_mbus_create(): Added parameter for default input queue. Explicitly
> state
> > >that the pool must be use type ODP_EVENT_MESSAGE.
> > >Renamed odp_ipc_resolve() to odp_mbus_lookup().
> > >odp_mbus_send(): Added priority parameter.
> > >Renamed odp_ipc_sender() to odp_message_sender().
> > >Renamed odp_ipc_data() to odp_message_data().
> > >Renamed odp_ipc_length() to odp_message_length().
> > >Renamed odp_ipc_reset() to odp_message_length_set().
> > >Renamed odp_ipc_alloc() to odp_message_alloc().
> > >Renamed odp_ipc_free() to odp_message_free().
> > >odp_message_alloc(): Corrected name of invalid message handle.
> > >Added message priorities and calls to set and remove input queues for
> > >specific priorities: odp_mbus_inq_set(), odp_mbus_inq_rem().
> > >
> > >Signed-off-by: Ola Liljedahl <ola.liljedahl@linaro.org>
> > >---
> > >(This document/code contribution attached is provided under the terms of
> > >agreement LES-LTM-21309)
> > >
> > >  include/odp/api/message.h                          | 199
> ++++++++++++++++++
> > >  include/odp/api/msgio.h                            | 222
> +++++++++++++++++++++
> > >  platform/linux-generic/include/odp/message.h       |  39 ++++
> > >  platform/linux-generic/include/odp/msgio.h         |  40 ++++
> > >  .../linux-generic/include/odp/plat/message_types.h |  47 +++++
> > >  .../linux-generic/include/odp/plat/msgio_types.h   |  59 ++++++
> > >  6 files changed, 606 insertions(+)
> > >  create mode 100644 include/odp/api/message.h
> > >  create mode 100644 include/odp/api/msgio.h
> > >  create mode 100644 platform/linux-generic/include/odp/message.h
> > >  create mode 100644 platform/linux-generic/include/odp/msgio.h
> > >  create mode 100644
> platform/linux-generic/include/odp/plat/message_types.h
> > >  create mode 100644
> platform/linux-generic/include/odp/plat/msgio_types.h
> > >
> > >diff --git a/include/odp/api/message.h b/include/odp/api/message.h
> > >new file mode 100644
> > >index 0000000..76697fb
> > >--- /dev/null
> > >+++ b/include/odp/api/message.h
> > >@@ -0,0 +1,199 @@
> > >+/* Copyright (c) 2015, Linaro Limited
> > >+ * All rights reserved.
> > >+ *
> > >+ * SPDX-License-Identifier:     BSD-3-Clause
> > >+ */
> > >+
> > >+
> > >+/**
> > >+ * @file
> > >+ *
> > >+ * ODP Message event type
> > >+ */
> > >+
> > >+#ifndef ODP_API_MESSAGE_H_
> > >+#define ODP_API_MESSAGE_H_
> > >+
> > >+#ifdef __cplusplus
> > >+extern "C" {
> > >+#endif
> > >+
> > >+/** @defgroup odp_message ODP MESSAGE
> > >+ *  @{
> > >+ */
> > >+
> > >+/**
> > >+ * @typedef odp_message_t
> > >+ * ODP message handle
> > >+ */
> > >+
> > >+
> > >+/**
> > >+ * Get address of sender (source) of message
> > >+ *
> > >+ * @param      msg  Message handle
> > >+ * @param[out] addr Buffer to hold sender address
> > >+ */
> > >+void odp_message_sender(odp_message_t msg,
> > >+                    uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
> > >+
> > >+/**
> > >+ * Message data pointer
> > >+ *
> > >+ * Return a pointer to the message data
> > >+ *
> > >+ * @param msg Message handle
> > >+ *
> > >+ * @return Pointer to the message data
> > >+ */
> > >+void *odp_message_data(odp_message_t msg);
> > >+
> > >+/**
> > >+ * Message data length
> > >+ *
> > >+ * Return length of the message data.
> > >+ *
> > >+ * @param msg Message handle
> > >+ *
> > >+ * @return Message length
> > >+ */
> > >+uint32_t odp_message_length(const odp_message_t msg);
> > >+
> > >+/**
> > >+ * Set message length
> > >+ *
> > >+ * Set length of the message data.
> > >+ * Increase message data length by moving message tail into message
> tailroom.
> > >+ * The actual message data is not modified.
> > >+ *
> > >+ * @param msg Message handle
> > >+ * @param len New length
> > >+ *
> > >+ * @retval 0 on success
> > >+ * @retval <0 on error (e.g. no enough tailroom)
> > >+ */
> > >+int odp_message_length_set(const odp_message_t msg, uint32_t len);
> > >+
> > >+/**
> > >+ * Message headroom length
> > >+ *
> > >+ * Return length of the message headroom
> > >+ *
> > >+ * @param msg Message handle
> > >+ *
> > >+ * @return Headroom length
> > >+ */
> > >+uint32_t odp_message_headroom(const odp_message_t msg);
> > >+
> > >+/**
> > >+ * Push out message head
> > >+ *
> > >+ * Increase message data length by moving message head into message
> headroom.
> > >+ * Message headroom is decreased with the same amount. The message
> head may be
> > >+ * pushed out up to 'headroom' bytes. Message is not modified if
> there's not
> > >+ * enough headroom space.
> > >+ *
> > >+ * odp_message_xxx:
> > >+ * length   += len
> > >+ * headroom -= len
> > >+ * data     -= len
> > >+ *
> > >+ * @param msg  Message handle
> > >+ * @param len  Number of bytes to push the head (0 ... headroom)
> > >+ *
> > >+ * @return The new data pointer
> > >+ * @retval NULL  Requested offset exceeds available headroom
> > >+ *
> > >+ * @see odp_message_headroom(), odp_message_pull_head()
> > >+ */
> > >+void *odp_message_push_head(odp_message_t msg, uint32_t len);
> > >+
> > >+/**
> > >+ * Pull in message head
> > >+ *
> > >+ * Decrease message data length by moving message head into message
> data.
> > >+ * Message headroom is increased with the same amount. The message
> head may be
> > >+ * pulled in up to 'length' bytes. Message is not modified if there's
> not
> > >+ * enough data.
> > >+ *
> > >+ * odp_message_xxx:
> > >+ * length   -= len
> > >+ * headroom += len
> > >+ * data     += len
> > >+ *
> > >+ * @param msg  Message handle
> > >+ * @param len  Number of bytes to pull the head (0 ... length)
> > >+ *
> > >+ * @return The new data pointer
> > >+ * @retval NULL  Requested offset exceeds available data
> > >+ *
> > >+ * @see odp_message_headroom(), odp_message_push_head()
> > >+ */
> > >+void *odp_message_pull_head(odp_message_t msg, uint32_t len);
> > >+
> > >+/**
> > >+ * Allocate message
> > >+ *
> > >+ * Allocate a message of a specific length.
> > >+ *
> > >+ * @param pool Message pool to allocate message from
> > >+ * @param len Length of the allocated message
> > >+ *
> > >+ * @return Message handle on success
> > >+ * @retval ODP_MESSAGE_INVALID on failure and errno set
> > >+ */
> > >+odp_message_t odp_message_alloc(odp_pool_t pool, uint32_t len);
> > >+
> > >+/**
> > >+ * Free message
> > >+ *
> > >+ * Free message back to the message pool it was allocated from.
> > >+ *
> > >+ * @param msg Handle of message to free
> > >+ */
> > >+void odp_message_free(odp_message_t msg);
> > >+
> > >+/**
> > >+ * Get message handle from event
> > >+ *
> > >+ * Converts an ODP_EVENT_MESSAGE type event to a message.
> > >+ *
> > >+ * @param ev   Event handle representing a message.
> > >+ *
> > >+ * @return Message handle
> > >+ *
> > >+ * @see odp_event_type()
> > >+ */
> > >+odp_message_t odp_message_from_event(odp_event_t ev);
> > >+
> > >+/**
> > >+ * Convert message handle to event
> > >+ *
> > >+ * @param msg  Message handle
> > >+ *
> > >+ * @return Event handle
> > >+ */
> > >+odp_event_t odp_message_to_event(odp_message_t msg);
> > >+
> > >+/**
> > >+ * Get printable value for an odp_message_t
> > >+ *
> > >+ * @param msg  Message handle to be printed
> > >+ * @return     uint64_t value that can be used to print/display this
> > >+ *         handle
> > >+ *
> > >+ * @note This routine is intended to be used for diagnostic purposes
> > >+ * to enable applications to generate a printable value that represents
> > >+ * an odp_message_t handle.
> > >+ */
> > >+uint64_t odp_message_to_u64(odp_message_t msg);
> > >+
> > >+/**
> > >+ * @}
> > >+ */
> > >+
> > >+#ifdef __cplusplus
> > >+}
> > >+#endif
> > >+
> > >+#endif
> > >diff --git a/include/odp/api/msgio.h b/include/odp/api/msgio.h
> > >new file mode 100644
> > >index 0000000..8d73021
> > >--- /dev/null
> > >+++ b/include/odp/api/msgio.h
> > >@@ -0,0 +1,222 @@
> > >+/* Copyright (c) 2015, Linaro Limited
> > >+ * All rights reserved.
> > >+ *
> > >+ * SPDX-License-Identifier:     BSD-3-Clause
> > >+ */
> > >+
> > >+
> > >+/**
> > >+ * @file
> > >+ *
> > >+ * ODP Message I/O API
> > >+ */
> > >+
> > >+#ifndef ODP_API_MSGIO_H_
> > >+#define ODP_API_MSGIO_H_
> > >+
> > >+#ifdef __cplusplus
> > >+extern "C" {
> > >+#endif
> > >+
> > >+/** @defgroup odp_msgio ODP MSGIO
> > >+ *  @{
> > >+ */
> > >+
> > >+/**
> > >+ * @typedef odp_msgio_t
> > >+ * ODP message I/O handle
> > >+ */
> > >+
> > >+/**
> > >+ * @def ODP_MSGIO_ADDR_SIZE
> > >+ * Size of the address of a message I/O endpoint
> > >+ */
> > >+
> > >+/**
> > >+ * @typedef odp_msgio_prio_t
> > >+ * ODP MSGIO message priority
> > >+ */
> > >+
> > >+/**
> > >+ * @def ODP_MSGIO_PRIO_HIGHEST
> > >+ * Highest MSGIO message priority
> > >+ */
> > >+
> > >+/**
> > >+ * @def ODP_MSGIO_PRIO_NORMAL
> > >+ * Normal MSGIO message priority
> > >+ */
> > >+
> > >+/**
> > >+ * @def ODP_MSGIO_PRIO_LOWEST
> > >+ * Lowest MSGIO message priority
> > >+ */
> > >+
> > >+/**
> > >+ * @def ODP_MSGIO_PRIO_DEFAULT
> > >+ * Default MSGIO message priority
> > >+ */
> > >+
> > >+
> > >+/**
> > >+ * Create message I/O endpoint
> > >+ *
> > >+ * Create an message I/O endpoint. The scope of the message bus is
> > >+ * not defined but it is expected that it by default encompasses the OS
> > >+ * instance but no more.
> > >+ *
> > >+ * A unique address for the endpoint is created. The addresses for
> other
> > >+ * endpoints can be obtained using the odp_msgio_lookup() call and
> specified
> > >+ * in the odp_msgio_send() call.
> > >+ *
> > >+ * @param name Name of our endpoint
> > >+ * @param pool Pool (of type ODP_EVENT_MESSAGE) for incoming messages
> > >+ *
> > >+ * @return Message I/O handle on success
> > >+ * @retval ODP_MSGIO_INVALID on failure and errno set
> > >+ */
> > >+odp_msgio_t odp_msgio_create(const char *name,
> > >+                         odp_pool_t pool);
> > >+
> > >+/**
> > >+ * Set the input queue for a specific message priority
> > >+ *
> > >+ * @param msgio  Message I/O handle
> > >+ * @param prio  Message priority
> > >+ * @param queue Queue handle
> > >+ *
> > >+ * @retval  0 on success
> > >+ * @retval <0 on failure
> > >+ */
> > >+int odp_msgio_inq_set(odp_msgio_t msgio,
> > >+                  odp_msgio_prio_t prio,
> > >+                  odp_queue_t queue);
> > >+
> > >+/**
> > >+ * Activate message I/O endpoint
> > >+ *
> > >+ * Make this endpoint visible (odp_msgio_lookup() messages will be
> returned).
> > >+ * Enable transmission and reception of messages through the endpoint.
> > >+ *
> > >+ * @param msgio Message I/O handle
> > >+ *
> > >+ * @retval 0 on success
> > >+ * @retval <0 on failure
> > >+ */
> > >+int odp_msgio_activate(odp_msgio_t msgio);
> > >+
> > >+/**
> > >+ * Destroy message I/O endpoint
> > >+ *
> > >+ * @param msgio Message I/O handle
> > >+ *
> > >+ * @retval 0 on success
> > >+ * @retval <0 on failure
> > >+ */
> > >+int odp_msgio_destroy(odp_msgio_t msgio);
> > >+
> > >+/**
> > >+ * Remove the input queue for a specific message priority
> > >+ *
> > >+ * Remove (disassociate) the matching input queue from a message I/O
> endpoint.
> > >+ * The queue itself is not touched.
> > >+ *
> > >+ * @param msgio  Message I/O handle
> > >+ * @param prio  Message priority
> > >+ *
> > >+ * @retval 0 on success
> > >+ * @retval <0 on failure
> > >+ */
> > >+int odp_msgio_inq_rem(odp_msgio_t msgio,
> > >+                  odp_msgio_prio_t prio);
> > >+
> > >+/**
> > >+ * Lookup endpoint by name
> > >+ *
> > >+ * Look up a current or future endpoint by name.
> > >+ * When the endpoint exists, return the specified message (not
> necessarily the
> > >+ * same message handle) with the endpoint as the sender.
> > >+ *
> > >+ * @param msgio Message I/O handle
> > >+ * @param name Name to look up
> > >+ * @param msg Message to return
> > >+ */
> > >+void odp_msgio_lookup(odp_msgio_t msgio,
> > >+                  const char *name,
> > >+                  odp_message_t msg);
> > >+
> > >+/**
> > >+ * Monitor endpoint
> > >+ *
> > >+ * Monitor an endpoint (which may already have disappeared).
> > >+ * When the endpoint disappears, return the specified message (not
> necessarily
> > >+ * the same message handle) with the endpoint as the sender.
> > >+ *
> > >+ * Unrecognized or invalid endpoint addresses are treated as
> non-existing
> > >+ * endpoints.
> > >+ *
> > >+ * @param     msgio Message I/O handle
> > >+ * @param[in] addr  Address of monitored endpoint
> > >+ * @param     msg   Message to return
> > >+ */
> > >+void odp_msgio_monitor(odp_msgio_t msgio,
> > >+                   const uint8_t addr[ODP_MSGIO_ADDR_SIZE],
> > >+                   odp_message_t msg);
> > >+
> > >+/**
> > >+ * Send message
> > >+ *
> > >+ * Send a message to an endpoint (which may no longer exist).
> > >+ *
> > >+ * Message delivery into the recipient address space is ordered (per
> priority)
> > >+ * and reliable. Delivery of message N implies delivery of all
> messages <N
> > >+ * (of the same priority).
> > >+ * All messages (accepted by MSGIO) will be delivered up to the point
> of
> > >+ * recipient endpoint termination or lost connection where no more
> messages
> > >+ * will be delivered.
> > >+ *
> > >+ * Actual reception (dequeueing) and processing by the recipient is not
> > >+ * guaranteed (use user level acknowledgements for that).
> > >+ *
> > >+ * Monitor the remote endpoint to detect the disappearance or
> disconnection
> > >+ * of the endpoint.
> > >+ *
> > >+ * A message may temporarily not be accepted for transmission (e.g. due
> > >+ * to out of local buffer space), -1 will be returned and ODP errno set
> > >+ * to EAGAIN.
> > >+ *
> > >+ * @param     msgio Message I/O handle
> > >+ * @param     msg   Message to send
> > >+ * @param     prio  Priority of message
> > >+ * @param[in] addr  Address of recipient endpoint
> > >+ *
> > >+ * @retval 0 on success
> > >+ * @retval <0 on error
> > >+ */
> > >+int odp_msgio_send(odp_msgio_t msgio,
> > >+               odp_message_t msg,
> > >+               odp_msgio_prio_t prio,
> > >+               const uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
> > >+
> > >+/**
> > >+ * Get printable value for an odp_msgio_t
> > >+ *
> > >+ * @param msgio  Message I/O handle to be printed
> > >+ * @return       uint64_t value that can be used to print/display this
> > >+ *           handle
> > >+ *
> > >+ * @note This routine is intended to be used for diagnostic purposes
> > >+ * to enable applications to generate a printable value that represents
> > >+ * an odp_msgio_t handle.
> > >+ */
> > >+uint64_t odp_msgio_to_u64(odp_msgio_t msgio);
> > >+
> > >+/**
> > >+ * @}
> > >+ */
> > >+
> > >+#ifdef __cplusplus
> > >+}
> > >+#endif
> > >+
> > >+#endif
> > >diff --git a/platform/linux-generic/include/odp/message.h
> b/platform/linux-generic/include/odp/message.h
> > >new file mode 100644
> > >index 0000000..2a29f59
> > >--- /dev/null
> > >+++ b/platform/linux-generic/include/odp/message.h
> > >@@ -0,0 +1,39 @@
> > >+/* Copyright (c) 2013, Linaro Limited
> > >+ * All rights reserved.
> > >+ *
> > >+ * SPDX-License-Identifier:     BSD-3-Clause
> > >+ */
> > >+
> > >+/**
> > >+ * @file
> > >+ *
> > >+ * ODP Message Bus
> > >+ */
> > >+
> > >+#ifndef ODP_PLAT_MESSAGE_H_
> > >+#define ODP_PLAT_MESSAGE_H_
> > >+
> > >+#ifdef __cplusplus
> > >+extern "C" {
> > >+#endif
> > >+
> > >+#include <odp/std_types.h>
> > >+#include <odp/plat/pool_types.h>
> > >+#include <odp/plat/message_types.h>
> > >+#include <odp/plat/queue_types.h>
> > >+
> > >+/** @ingroup odp_message
> > >+ *  @{
> > >+ */
> > >+
> > >+/**
> > >+ * @}
> > >+ */
> > >+
> > >+#include <odp/api/message.h>
> > >+
> > >+#ifdef __cplusplus
> > >+}
> > >+#endif
> > >+
> > >+#endif
> > >diff --git a/platform/linux-generic/include/odp/msgio.h
> b/platform/linux-generic/include/odp/msgio.h
> > >new file mode 100644
> > >index 0000000..f45b3a2
> > >--- /dev/null
> > >+++ b/platform/linux-generic/include/odp/msgio.h
> > >@@ -0,0 +1,40 @@
> > >+/* Copyright (c) 2013, Linaro Limited
> > >+ * All rights reserved.
> > >+ *
> > >+ * SPDX-License-Identifier:     BSD-3-Clause
> > >+ */
> > >+
> > >+/**
> > >+ * @file
> > >+ *
> > >+ * ODP Message Bus
> > >+ */
> > >+
> > >+#ifndef ODP_PLAT_MSGIO_H_
> > >+#define ODP_PLAT_MSGIO_H_
> > >+
> > >+#ifdef __cplusplus
> > >+extern "C" {
> > >+#endif
> > >+
> > >+#include <odp/std_types.h>
> > >+#include <odp/plat/pool_types.h>
> > >+#include <odp/plat/msgio_types.h>
> > >+#include <odp/plat/message_types.h>
> > >+#include <odp/plat/queue_types.h>
> > >+
> > >+/** @ingroup odp_msgio
> > >+ *  @{
> > >+ */
> > >+
> > >+/**
> > >+ * @}
> > >+ */
> > >+
> > >+#include <odp/api/msgio.h>
> > >+
> > >+#ifdef __cplusplus
> > >+}
> > >+#endif
> > >+
> > >+#endif
> > >diff --git a/platform/linux-generic/include/odp/plat/message_types.h
> b/platform/linux-generic/include/odp/plat/message_types.h
> > >new file mode 100644
> > >index 0000000..3a42064
> > >--- /dev/null
> > >+++ b/platform/linux-generic/include/odp/plat/message_types.h
> > >@@ -0,0 +1,47 @@
> > >+/* Copyright (c) 2015, Linaro Limited
> > >+ * All rights reserved.
> > >+ *
> > >+ * SPDX-License-Identifier: BSD-3-Clause
> > >+ */
> > >+
> > >+
> > >+/**
> > >+ * @file
> > >+ *
> > >+ * ODP Message message type
> > >+ */
> > >+
> > >+#ifndef ODP_MESSAGE_TYPES_H_
> > >+#define ODP_MESSAGE_TYPES_H_
> > >+
> > >+#ifdef __cplusplus
> > >+extern "C" {
> > >+#endif
> > >+
> > >+#include <odp/std_types.h>
> > >+#include <odp/plat/strong_types.h>
> > >+
> > >+/** @addtogroup odp_message ODP message
> > >+ *  Operations on a message.
> > >+ *  @{
> > >+ */
> > >+
> > >+typedef ODP_HANDLE_T(odp_message_t);
> > >+
> > >+#define ODP_MESSAGE_INVALID _odp_cast_scalar(odp_message_t, 0xffffffff)
> > >+
> > >+/** Get printable format of odp_message_t */
> > >+static inline uint64_t odp_message_to_u64(odp_message_t hdl)
> > >+{
> > >+    return _odp_pri(hdl);
> > >+}
> > >+
> > >+/**
> > >+ * @}
> > >+ */
> > >+
> > >+#ifdef __cplusplus
> > >+}
> > >+#endif
> > >+
> > >+#endif
> > >diff --git a/platform/linux-generic/include/odp/plat/msgio_types.h
> b/platform/linux-generic/include/odp/plat/msgio_types.h
> > >new file mode 100644
> > >index 0000000..9d9f9da
> > >--- /dev/null
> > >+++ b/platform/linux-generic/include/odp/plat/msgio_types.h
> > >@@ -0,0 +1,59 @@
> > >+/* Copyright (c) 2015, Linaro Limited
> > >+ * All rights reserved.
> > >+ *
> > >+ * SPDX-License-Identifier: BSD-3-Clause
> > >+ */
> > >+
> > >+
> > >+/**
> > >+ * @file
> > >+ *
> > >+ * ODP Message bus types
> > >+ */
> > >+
> > >+#ifndef ODP_MSGIO_TYPES_H_
> > >+#define ODP_MSGIO_TYPES_H_
> > >+
> > >+#ifdef __cplusplus
> > >+extern "C" {
> > >+#endif
> > >+
> > >+#include <odp/std_types.h>
> > >+#include <odp/plat/strong_types.h>
> > >+
> > >+/** @addtogroup odp_msgio ODP message bus
> > >+ *  Operations on a message bus.
> > >+ *  @{
> > >+ */
> > >+
> > >+typedef int odp_msgio_prio_t;
> > >+
> > >+#define ODP_MSGIO_PRIO_HIGHEST  0
> > >+
> > >+#define ODP_MSGIO_PRIO_NORMAL   2
> > >+
> > >+#define ODP_MSGIO_PRIO_LOWEST   3
> > >+
> > >+#define ODP_MSGIO_PRIO_DEFAULT  ODP_MSGIO_PRIO_NORMAL
> > >+
> > >+typedef ODP_HANDLE_T(odp_msgio_t);
> > >+
> > >+#define ODP_MSGIO_INVALID _odp_cast_scalar(odp_msgio_t, 0xffffffff)
> > >+
> > >+#define ODP_MSGIO_ADDR_SIZE 6
> > >+
> > >+/** Get printable format of odp_msgio_t */
> > >+static inline uint64_t odp_msgio_to_u64(odp_msgio_t hdl)
> > >+{
> > >+    return _odp_pri(hdl);
> > >+}
> > >+
> > >+/**
> > >+ * @}
> > >+ */
> > >+
> > >+#ifdef __cplusplus
> > >+}
> > >+#endif
> > >+
> > >+#endif
> > >
> >
> >
> > --
> > Benoît GANNE
> > Field Application Engineer, Kalray
> > +33 (0)648 125 843
> > _______________________________________________
> > lng-odp mailing list
> > lng-odp@lists.linaro.org
> > https://lists.linaro.org/mailman/listinfo/lng-odp
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/lng-odp
>

On 2 June 2015 at 14:36, Benoît Ganne <bganne@kalray.eu> wrote:

> Hi Ola, all,
>
> On 05/27/2015 10:08 AM, Ola Liljedahl wrote:
>
>> Here is my third attempt at a ODP API for MSGIO - asynchronous message
>> passing
>> based IPC for a shared nothing architecture.
>>
>
> I understand the value of this, but I feel I need to ask the question:
> does it really belong to ODP ? My understanding so far was that ODP should
> not try to replace OS services: for example, thread or process related API
> were ruled out. This looks to me a lot like D-BUS or ZeroMQ (or <name your
> favorite solution here>). Wouldn't it be better to let the application
> select what they want to use instead of providing our own?
>
Yes but I want MSGIO IPC to be part of the ODP functionality such as
scheduling, not something completely separate. So messages need to be some
type of events (or at least incoming messages notified using events) and
when you set up your message IO, input messages (or notifications) are put
on some queue. Perhaps this can be done differently. It would be great of
there was a generic way of including alien functionality (e.g. D-BUS or
ZeroMQ as you mention) in the ODP framework. Any suggestions how this could
be done?


> ben
>
>
>  IPC/message passing can be separated into a number of different layers.
>> 1) Message formats and protocols (as seen by the user).
>> 2) API - defines syntax and semantics of MSGIO operations.
>> 3) Transport interface - defines a binary packet format used by MSGIO
>>     endpoints/implementations. Cf. OSI network layer.
>> 4) Transport layer - implements the transport of message packet between
>>     different endpoints. Cf. OSI link layer.
>>
>> #1 is application specific and there are many different standards (e.g.
>> Linux
>> Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
>> #3 may be implementation specific but is important for interoperatibility
>> (binary compatibility). If the user message (payload), operation and
>> parameters are encoded in a well-defined way (e.g. some kind of packet
>> format
>> which includes operation, sender and receiver addresses, payload etc),
>> inter-operatibility between different ODP/MSGIO implementations may be
>> achieved or simplified.
>> #4 is implementation (platform) specific.
>>
>> This proposal focuses on the API which is fundamental for source code
>> compatibility between different platforms (ODP implementations) - this
>> being
>> the primary purpose of ODP. A packet format for messages will be
>> specified as
>> part of the reference implementation and may be seen as a proposal for a
>> standard packet format for binary compatibility between different
>> ODP/MSGIO
>> implementations.
>>
>> All MSGIO calls are non-blocking (with the possible exception of
>> odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
>> transfer is asynchronous so it is not possible to return any status
>> operation concerning the eventual (or remote) result of an operation.
>>
>> The semantics of MSGIO message passing is that sending a message to an
>> endpoint
>> will always look like it succeeds. The appearance of endpoints is
>> explicitly
>> notified through user-defined messages specified in the odp_msgio_lookup()
>> call. Similarly, the disappearance (e.g. death of or otherwise lost
>> connection
>> to endpoint) is also explicitly notified through user-defined messages
>> specified in the odp_msgio_monitor() call. The send call does not fail
>> because
>> the addressed endpoint has disappeared, the message is just silently
>> discarded.
>>
>> Message delivery into the recipient address space is ordered (per
>> priority)
>> and reliable. Delivery of message N implies delivery of all messages <N
>> (of the same priority). All messages (accepted by MSGIO send operation)
>> will
>> be delivered up to the point of endpoint termination or lost connection
>> where no more messages will be delivered. Actual reception (dequeueing)
>> and
>> processing by the recipient is not guaranteed (use end-to-end
>> acknowledgements
>> for that).
>>
>> MSGIO endpoints can be seen as interfaces (taps) to an internal reliable
>> multidrop network where each endpoint has a unique address which is only
>> valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
>> and then recreated (with the same name), the new endpoint will have a
>> new unique address (eventually endpoints addresses will have to be
>> recycled
>> but not for a very long time). Endpoints names do not necessarily have to
>> be
>> unique. The scope of message network is not defined but it is expected
>> that
>> the scope by default corresponds to an OS instance (e.g. virtual machine).
>>
>> Proposed transport packet for message I/O (not visible to the user):
>> uint8_t dest_addr[6]; /* Unique address of destination endpoint */
>> uint8_t src_addr[6]; /* Unique address of source endpoint */
>> uint16_t frametype_o; /* Outer frametype, 0x8100 */
>> uint16_t pcp_vid; /* 3 bits of Priority Code Point, 5 bits reserved */
>> uint16_t frametype_i; /* Inner frametype, TBD */
>> uint8_t payload[]; /* User message */
>> Looks very much like a VLAN-tagged Ethernet frame so should easily be
>> transported by and processed by Ethernet HW & SW.
>> Encoding of publish/lookup/monitor operations TBD.
>>
>> v3:
>> Renamed Message Bus (MBUS) to Message I/O (MSGIO).
>> Changed all definitions to use "msgio" instead of "mbus" prefix/infix.
>> odp_msgio_create(): removed default input queue parameter.
>> Removed odp_mbus_inq_setdef() and odp_mbus_inq_remdef().
>> All input queues must now be specified using odp_msgio_inq_set() before
>> the MSGIO endpoint is activated.
>> Added odp_msgio_activate().
>> Added odp_message_push_head() and odp_message_pull_head().
>> Updated some function descriptions in msgio.h.
>>
>> v2:
>> Split off all message definitions in a separate file message.h
>> Renamed Inter Process Communication (IPC) to Message Bus (MBUS).
>> Changed all definitions to use "mbus" instead of "ipc" prefix/infix.
>> Renamed odp_ipc_msg_t to odp_message_t.
>> odp_mbus_create(): Added parameter for default input queue. Explicitly
>> state
>> that the pool must be use type ODP_EVENT_MESSAGE.
>> Renamed odp_ipc_resolve() to odp_mbus_lookup().
>> odp_mbus_send(): Added priority parameter.
>> Renamed odp_ipc_sender() to odp_message_sender().
>> Renamed odp_ipc_data() to odp_message_data().
>> Renamed odp_ipc_length() to odp_message_length().
>> Renamed odp_ipc_reset() to odp_message_length_set().
>> Renamed odp_ipc_alloc() to odp_message_alloc().
>> Renamed odp_ipc_free() to odp_message_free().
>> odp_message_alloc(): Corrected name of invalid message handle.
>> Added message priorities and calls to set and remove input queues for
>> specific priorities: odp_mbus_inq_set(), odp_mbus_inq_rem().
>>
>> Signed-off-by: Ola Liljedahl <ola.liljedahl@linaro.org>
>> ---
>> (This document/code contribution attached is provided under the terms of
>> agreement LES-LTM-21309)
>>
>>   include/odp/api/message.h                          | 199
>> ++++++++++++++++++
>>   include/odp/api/msgio.h                            | 222
>> +++++++++++++++++++++
>>   platform/linux-generic/include/odp/message.h       |  39 ++++
>>   platform/linux-generic/include/odp/msgio.h         |  40 ++++
>>   .../linux-generic/include/odp/plat/message_types.h |  47 +++++
>>   .../linux-generic/include/odp/plat/msgio_types.h   |  59 ++++++
>>   6 files changed, 606 insertions(+)
>>   create mode 100644 include/odp/api/message.h
>>   create mode 100644 include/odp/api/msgio.h
>>   create mode 100644 platform/linux-generic/include/odp/message.h
>>   create mode 100644 platform/linux-generic/include/odp/msgio.h
>>   create mode 100644
>> platform/linux-generic/include/odp/plat/message_types.h
>>   create mode 100644 platform/linux-generic/include/odp/plat/msgio_types.h
>>
>> diff --git a/include/odp/api/message.h b/include/odp/api/message.h
>> new file mode 100644
>> index 0000000..76697fb
>> --- /dev/null
>> +++ b/include/odp/api/message.h
>> @@ -0,0 +1,199 @@
>> +/* Copyright (c) 2015, Linaro Limited
>> + * All rights reserved.
>> + *
>> + * SPDX-License-Identifier:     BSD-3-Clause
>> + */
>> +
>> +
>> +/**
>> + * @file
>> + *
>> + * ODP Message event type
>> + */
>> +
>> +#ifndef ODP_API_MESSAGE_H_
>> +#define ODP_API_MESSAGE_H_
>> +
>> +#ifdef __cplusplus
>> +extern "C" {
>> +#endif
>> +
>> +/** @defgroup odp_message ODP MESSAGE
>> + *  @{
>> + */
>> +
>> +/**
>> + * @typedef odp_message_t
>> + * ODP message handle
>> + */
>> +
>> +
>> +/**
>> + * Get address of sender (source) of message
>> + *
>> + * @param      msg  Message handle
>> + * @param[out] addr Buffer to hold sender address
>> + */
>> +void odp_message_sender(odp_message_t msg,
>> +                       uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
>> +
>> +/**
>> + * Message data pointer
>> + *
>> + * Return a pointer to the message data
>> + *
>> + * @param msg Message handle
>> + *
>> + * @return Pointer to the message data
>> + */
>> +void *odp_message_data(odp_message_t msg);
>> +
>> +/**
>> + * Message data length
>> + *
>> + * Return length of the message data.
>> + *
>> + * @param msg Message handle
>> + *
>> + * @return Message length
>> + */
>> +uint32_t odp_message_length(const odp_message_t msg);
>> +
>> +/**
>> + * Set message length
>> + *
>> + * Set length of the message data.
>> + * Increase message data length by moving message tail into message
>> tailroom.
>> + * The actual message data is not modified.
>> + *
>> + * @param msg Message handle
>> + * @param len New length
>> + *
>> + * @retval 0 on success
>> + * @retval <0 on error (e.g. no enough tailroom)
>> + */
>> +int odp_message_length_set(const odp_message_t msg, uint32_t len);
>> +
>> +/**
>> + * Message headroom length
>> + *
>> + * Return length of the message headroom
>> + *
>> + * @param msg Message handle
>> + *
>> + * @return Headroom length
>> + */
>> +uint32_t odp_message_headroom(const odp_message_t msg);
>> +
>> +/**
>> + * Push out message head
>> + *
>> + * Increase message data length by moving message head into message
>> headroom.
>> + * Message headroom is decreased with the same amount. The message head
>> may be
>> + * pushed out up to 'headroom' bytes. Message is not modified if there's
>> not
>> + * enough headroom space.
>> + *
>> + * odp_message_xxx:
>> + * length   += len
>> + * headroom -= len
>> + * data     -= len
>> + *
>> + * @param msg  Message handle
>> + * @param len  Number of bytes to push the head (0 ... headroom)
>> + *
>> + * @return The new data pointer
>> + * @retval NULL  Requested offset exceeds available headroom
>> + *
>> + * @see odp_message_headroom(), odp_message_pull_head()
>> + */
>> +void *odp_message_push_head(odp_message_t msg, uint32_t len);
>> +
>> +/**
>> + * Pull in message head
>> + *
>> + * Decrease message data length by moving message head into message data.
>> + * Message headroom is increased with the same amount. The message head
>> may be
>> + * pulled in up to 'length' bytes. Message is not modified if there's not
>> + * enough data.
>> + *
>> + * odp_message_xxx:
>> + * length   -= len
>> + * headroom += len
>> + * data     += len
>> + *
>> + * @param msg  Message handle
>> + * @param len  Number of bytes to pull the head (0 ... length)
>> + *
>> + * @return The new data pointer
>> + * @retval NULL  Requested offset exceeds available data
>> + *
>> + * @see odp_message_headroom(), odp_message_push_head()
>> + */
>> +void *odp_message_pull_head(odp_message_t msg, uint32_t len);
>> +
>> +/**
>> + * Allocate message
>> + *
>> + * Allocate a message of a specific length.
>> + *
>> + * @param pool Message pool to allocate message from
>> + * @param len Length of the allocated message
>> + *
>> + * @return Message handle on success
>> + * @retval ODP_MESSAGE_INVALID on failure and errno set
>> + */
>> +odp_message_t odp_message_alloc(odp_pool_t pool, uint32_t len);
>> +
>> +/**
>> + * Free message
>> + *
>> + * Free message back to the message pool it was allocated from.
>> + *
>> + * @param msg Handle of message to free
>> + */
>> +void odp_message_free(odp_message_t msg);
>> +
>> +/**
>> + * Get message handle from event
>> + *
>> + * Converts an ODP_EVENT_MESSAGE type event to a message.
>> + *
>> + * @param ev   Event handle representing a message.
>> + *
>> + * @return Message handle
>> + *
>> + * @see odp_event_type()
>> + */
>> +odp_message_t odp_message_from_event(odp_event_t ev);
>> +
>> +/**
>> + * Convert message handle to event
>> + *
>> + * @param msg  Message handle
>> + *
>> + * @return Event handle
>> + */
>> +odp_event_t odp_message_to_event(odp_message_t msg);
>> +
>> +/**
>> + * Get printable value for an odp_message_t
>> + *
>> + * @param msg  Message handle to be printed
>> + * @return     uint64_t value that can be used to print/display this
>> + *            handle
>> + *
>> + * @note This routine is intended to be used for diagnostic purposes
>> + * to enable applications to generate a printable value that represents
>> + * an odp_message_t handle.
>> + */
>> +uint64_t odp_message_to_u64(odp_message_t msg);
>> +
>> +/**
>> + * @}
>> + */
>> +
>> +#ifdef __cplusplus
>> +}
>> +#endif
>> +
>> +#endif
>> diff --git a/include/odp/api/msgio.h b/include/odp/api/msgio.h
>> new file mode 100644
>> index 0000000..8d73021
>> --- /dev/null
>> +++ b/include/odp/api/msgio.h
>> @@ -0,0 +1,222 @@
>> +/* Copyright (c) 2015, Linaro Limited
>> + * All rights reserved.
>> + *
>> + * SPDX-License-Identifier:     BSD-3-Clause
>> + */
>> +
>> +
>> +/**
>> + * @file
>> + *
>> + * ODP Message I/O API
>> + */
>> +
>> +#ifndef ODP_API_MSGIO_H_
>> +#define ODP_API_MSGIO_H_
>> +
>> +#ifdef __cplusplus
>> +extern "C" {
>> +#endif
>> +
>> +/** @defgroup odp_msgio ODP MSGIO
>> + *  @{
>> + */
>> +
>> +/**
>> + * @typedef odp_msgio_t
>> + * ODP message I/O handle
>> + */
>> +
>> +/**
>> + * @def ODP_MSGIO_ADDR_SIZE
>> + * Size of the address of a message I/O endpoint
>> + */
>> +
>> +/**
>> + * @typedef odp_msgio_prio_t
>> + * ODP MSGIO message priority
>> + */
>> +
>> +/**
>> + * @def ODP_MSGIO_PRIO_HIGHEST
>> + * Highest MSGIO message priority
>> + */
>> +
>> +/**
>> + * @def ODP_MSGIO_PRIO_NORMAL
>> + * Normal MSGIO message priority
>> + */
>> +
>> +/**
>> + * @def ODP_MSGIO_PRIO_LOWEST
>> + * Lowest MSGIO message priority
>> + */
>> +
>> +/**
>> + * @def ODP_MSGIO_PRIO_DEFAULT
>> + * Default MSGIO message priority
>> + */
>> +
>> +
>> +/**
>> + * Create message I/O endpoint
>> + *
>> + * Create an message I/O endpoint. The scope of the message bus is
>> + * not defined but it is expected that it by default encompasses the OS
>> + * instance but no more.
>> + *
>> + * A unique address for the endpoint is created. The addresses for other
>> + * endpoints can be obtained using the odp_msgio_lookup() call and
>> specified
>> + * in the odp_msgio_send() call.
>> + *
>> + * @param name Name of our endpoint
>> + * @param pool Pool (of type ODP_EVENT_MESSAGE) for incoming messages
>> + *
>> + * @return Message I/O handle on success
>> + * @retval ODP_MSGIO_INVALID on failure and errno set
>> + */
>> +odp_msgio_t odp_msgio_create(const char *name,
>> +                            odp_pool_t pool);
>> +
>> +/**
>> + * Set the input queue for a specific message priority
>> + *
>> + * @param msgio  Message I/O handle
>> + * @param prio  Message priority
>> + * @param queue Queue handle
>> + *
>> + * @retval  0 on success
>> + * @retval <0 on failure
>> + */
>> +int odp_msgio_inq_set(odp_msgio_t msgio,
>> +                     odp_msgio_prio_t prio,
>> +                     odp_queue_t queue);
>> +
>> +/**
>> + * Activate message I/O endpoint
>> + *
>> + * Make this endpoint visible (odp_msgio_lookup() messages will be
>> returned).
>> + * Enable transmission and reception of messages through the endpoint.
>> + *
>> + * @param msgio Message I/O handle
>> + *
>> + * @retval 0 on success
>> + * @retval <0 on failure
>> + */
>> +int odp_msgio_activate(odp_msgio_t msgio);
>> +
>> +/**
>> + * Destroy message I/O endpoint
>> + *
>> + * @param msgio Message I/O handle
>> + *
>> + * @retval 0 on success
>> + * @retval <0 on failure
>> + */
>> +int odp_msgio_destroy(odp_msgio_t msgio);
>> +
>> +/**
>> + * Remove the input queue for a specific message priority
>> + *
>> + * Remove (disassociate) the matching input queue from a message I/O
>> endpoint.
>> + * The queue itself is not touched.
>> + *
>> + * @param msgio  Message I/O handle
>> + * @param prio  Message priority
>> + *
>> + * @retval 0 on success
>> + * @retval <0 on failure
>> + */
>> +int odp_msgio_inq_rem(odp_msgio_t msgio,
>> +                     odp_msgio_prio_t prio);
>> +
>> +/**
>> + * Lookup endpoint by name
>> + *
>> + * Look up a current or future endpoint by name.
>> + * When the endpoint exists, return the specified message (not
>> necessarily the
>> + * same message handle) with the endpoint as the sender.
>> + *
>> + * @param msgio Message I/O handle
>> + * @param name Name to look up
>> + * @param msg Message to return
>> + */
>> +void odp_msgio_lookup(odp_msgio_t msgio,
>> +                     const char *name,
>> +                     odp_message_t msg);
>> +
>> +/**
>> + * Monitor endpoint
>> + *
>> + * Monitor an endpoint (which may already have disappeared).
>> + * When the endpoint disappears, return the specified message (not
>> necessarily
>> + * the same message handle) with the endpoint as the sender.
>> + *
>> + * Unrecognized or invalid endpoint addresses are treated as non-existing
>> + * endpoints.
>> + *
>> + * @param     msgio Message I/O handle
>> + * @param[in] addr  Address of monitored endpoint
>> + * @param     msg   Message to return
>> + */
>> +void odp_msgio_monitor(odp_msgio_t msgio,
>> +                      const uint8_t addr[ODP_MSGIO_ADDR_SIZE],
>> +                      odp_message_t msg);
>> +
>> +/**
>> + * Send message
>> + *
>> + * Send a message to an endpoint (which may no longer exist).
>> + *
>> + * Message delivery into the recipient address space is ordered (per
>> priority)
>> + * and reliable. Delivery of message N implies delivery of all messages
>> <N
>> + * (of the same priority).
>> + * All messages (accepted by MSGIO) will be delivered up to the point of
>> + * recipient endpoint termination or lost connection where no more
>> messages
>> + * will be delivered.
>> + *
>> + * Actual reception (dequeueing) and processing by the recipient is not
>> + * guaranteed (use user level acknowledgements for that).
>> + *
>> + * Monitor the remote endpoint to detect the disappearance or
>> disconnection
>> + * of the endpoint.
>> + *
>> + * A message may temporarily not be accepted for transmission (e.g. due
>> + * to out of local buffer space), -1 will be returned and ODP errno set
>> + * to EAGAIN.
>> + *
>> + * @param     msgio Message I/O handle
>> + * @param     msg   Message to send
>> + * @param     prio  Priority of message
>> + * @param[in] addr  Address of recipient endpoint
>> + *
>> + * @retval 0 on success
>> + * @retval <0 on error
>> + */
>> +int odp_msgio_send(odp_msgio_t msgio,
>> +                  odp_message_t msg,
>> +                  odp_msgio_prio_t prio,
>> +                  const uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
>> +
>> +/**
>> + * Get printable value for an odp_msgio_t
>> + *
>> + * @param msgio  Message I/O handle to be printed
>> + * @return       uint64_t value that can be used to print/display this
>> + *              handle
>> + *
>> + * @note This routine is intended to be used for diagnostic purposes
>> + * to enable applications to generate a printable value that represents
>> + * an odp_msgio_t handle.
>> + */
>> +uint64_t odp_msgio_to_u64(odp_msgio_t msgio);
>> +
>> +/**
>> + * @}
>> + */
>> +
>> +#ifdef __cplusplus
>> +}
>> +#endif
>> +
>> +#endif
>> diff --git a/platform/linux-generic/include/odp/message.h
>> b/platform/linux-generic/include/odp/message.h
>> new file mode 100644
>> index 0000000..2a29f59
>> --- /dev/null
>> +++ b/platform/linux-generic/include/odp/message.h
>> @@ -0,0 +1,39 @@
>> +/* Copyright (c) 2013, Linaro Limited
>> + * All rights reserved.
>> + *
>> + * SPDX-License-Identifier:     BSD-3-Clause
>> + */
>> +
>> +/**
>> + * @file
>> + *
>> + * ODP Message Bus
>> + */
>> +
>> +#ifndef ODP_PLAT_MESSAGE_H_
>> +#define ODP_PLAT_MESSAGE_H_
>> +
>> +#ifdef __cplusplus
>> +extern "C" {
>> +#endif
>> +
>> +#include <odp/std_types.h>
>> +#include <odp/plat/pool_types.h>
>> +#include <odp/plat/message_types.h>
>> +#include <odp/plat/queue_types.h>
>> +
>> +/** @ingroup odp_message
>> + *  @{
>> + */
>> +
>> +/**
>> + * @}
>> + */
>> +
>> +#include <odp/api/message.h>
>> +
>> +#ifdef __cplusplus
>> +}
>> +#endif
>> +
>> +#endif
>> diff --git a/platform/linux-generic/include/odp/msgio.h
>> b/platform/linux-generic/include/odp/msgio.h
>> new file mode 100644
>> index 0000000..f45b3a2
>> --- /dev/null
>> +++ b/platform/linux-generic/include/odp/msgio.h
>> @@ -0,0 +1,40 @@
>> +/* Copyright (c) 2013, Linaro Limited
>> + * All rights reserved.
>> + *
>> + * SPDX-License-Identifier:     BSD-3-Clause
>> + */
>> +
>> +/**
>> + * @file
>> + *
>> + * ODP Message Bus
>> + */
>> +
>> +#ifndef ODP_PLAT_MSGIO_H_
>> +#define ODP_PLAT_MSGIO_H_
>> +
>> +#ifdef __cplusplus
>> +extern "C" {
>> +#endif
>> +
>> +#include <odp/std_types.h>
>> +#include <odp/plat/pool_types.h>
>> +#include <odp/plat/msgio_types.h>
>> +#include <odp/plat/message_types.h>
>> +#include <odp/plat/queue_types.h>
>> +
>> +/** @ingroup odp_msgio
>> + *  @{
>> + */
>> +
>> +/**
>> + * @}
>> + */
>> +
>> +#include <odp/api/msgio.h>
>> +
>> +#ifdef __cplusplus
>> +}
>> +#endif
>> +
>> +#endif
>> diff --git a/platform/linux-generic/include/odp/plat/message_types.h
>> b/platform/linux-generic/include/odp/plat/message_types.h
>> new file mode 100644
>> index 0000000..3a42064
>> --- /dev/null
>> +++ b/platform/linux-generic/include/odp/plat/message_types.h
>> @@ -0,0 +1,47 @@
>> +/* Copyright (c) 2015, Linaro Limited
>> + * All rights reserved.
>> + *
>> + * SPDX-License-Identifier: BSD-3-Clause
>> + */
>> +
>> +
>> +/**
>> + * @file
>> + *
>> + * ODP Message message type
>> + */
>> +
>> +#ifndef ODP_MESSAGE_TYPES_H_
>> +#define ODP_MESSAGE_TYPES_H_
>> +
>> +#ifdef __cplusplus
>> +extern "C" {
>> +#endif
>> +
>> +#include <odp/std_types.h>
>> +#include <odp/plat/strong_types.h>
>> +
>> +/** @addtogroup odp_message ODP message
>> + *  Operations on a message.
>> + *  @{
>> + */
>> +
>> +typedef ODP_HANDLE_T(odp_message_t);
>> +
>> +#define ODP_MESSAGE_INVALID _odp_cast_scalar(odp_message_t, 0xffffffff)
>> +
>> +/** Get printable format of odp_message_t */
>> +static inline uint64_t odp_message_to_u64(odp_message_t hdl)
>> +{
>> +       return _odp_pri(hdl);
>> +}
>> +
>> +/**
>> + * @}
>> + */
>> +
>> +#ifdef __cplusplus
>> +}
>> +#endif
>> +
>> +#endif
>> diff --git a/platform/linux-generic/include/odp/plat/msgio_types.h
>> b/platform/linux-generic/include/odp/plat/msgio_types.h
>> new file mode 100644
>> index 0000000..9d9f9da
>> --- /dev/null
>> +++ b/platform/linux-generic/include/odp/plat/msgio_types.h
>> @@ -0,0 +1,59 @@
>> +/* Copyright (c) 2015, Linaro Limited
>> + * All rights reserved.
>> + *
>> + * SPDX-License-Identifier: BSD-3-Clause
>> + */
>> +
>> +
>> +/**
>> + * @file
>> + *
>> + * ODP Message bus types
>> + */
>> +
>> +#ifndef ODP_MSGIO_TYPES_H_
>> +#define ODP_MSGIO_TYPES_H_
>> +
>> +#ifdef __cplusplus
>> +extern "C" {
>> +#endif
>> +
>> +#include <odp/std_types.h>
>> +#include <odp/plat/strong_types.h>
>> +
>> +/** @addtogroup odp_msgio ODP message bus
>> + *  Operations on a message bus.
>> + *  @{
>> + */
>> +
>> +typedef int odp_msgio_prio_t;
>> +
>> +#define ODP_MSGIO_PRIO_HIGHEST  0
>> +
>> +#define ODP_MSGIO_PRIO_NORMAL   2
>> +
>> +#define ODP_MSGIO_PRIO_LOWEST   3
>> +
>> +#define ODP_MSGIO_PRIO_DEFAULT  ODP_MSGIO_PRIO_NORMAL
>> +
>> +typedef ODP_HANDLE_T(odp_msgio_t);
>> +
>> +#define ODP_MSGIO_INVALID _odp_cast_scalar(odp_msgio_t, 0xffffffff)
>> +
>> +#define ODP_MSGIO_ADDR_SIZE 6
>> +
>> +/** Get printable format of odp_msgio_t */
>> +static inline uint64_t odp_msgio_to_u64(odp_msgio_t hdl)
>> +{
>> +       return _odp_pri(hdl);
>> +}
>> +
>> +/**
>> + * @}
>> + */
>> +
>> +#ifdef __cplusplus
>> +}
>> +#endif
>> +
>> +#endif
>>
>>
>
> --
> Benoît GANNE
> Field Application Engineer, Kalray
> +33 (0)648 125 843
>

On 2 June 2015 at 14:53, Jerin Jacob <jerin.jacob@caviumnetworks.com> wrote:

> On Tue, Jun 02, 2015 at 02:36:47PM +0200, Benoît Ganne wrote:
> > Hi Ola, all,
> >
> > On 05/27/2015 10:08 AM, Ola Liljedahl wrote:
> > >Here is my third attempt at a ODP API for MSGIO - asynchronous message
> passing
> > >based IPC for a shared nothing architecture.
> >
> > I understand the value of this, but I feel I need to ask the question:
> does
> > it really belong to ODP ? My understanding so far was that ODP should not
> > try to replace OS services: for example, thread or process related API
> were
> > ruled out. This looks to me a lot like D-BUS or ZeroMQ (or <name your
> > favorite solution here>). Wouldn't it be better to let the application
> > select what they want to use instead of providing our own?
>
> +1 and I am not sure how much value addition it has in hardware
> abstraction perspective.
> Most of the HW accelerated platforms will map odp_msg_t as packet and
> odp_msgio_t as pktio
> then it will boils down to a helper function that creates the
> vlan packet/message from given message(uint8_t *), src, dst 48 bit address
>
That's a perfectly valid implementation strategy. But as I wanted to make
the API independent of the underlying HW and ODP implementation I didn't
want to mandate that specific design.

Bala suggested that we actually use odp_packet_t as the message type and
that could work from an API perspective. I don't know if this would
introduce some limitations (packet pools may be special HW constructs (e.g.
limited in number or other characteristics) on some target, can they then
be used for MSGIO as well?).


> >
> > ben
> >
> > >IPC/message passing can be separated into a number of different layers.
> > >1) Message formats and protocols (as seen by the user).
> > >2) API - defines syntax and semantics of MSGIO operations.
> > >3) Transport interface - defines a binary packet format used by MSGIO
> > >    endpoints/implementations. Cf. OSI network layer.
> > >4) Transport layer - implements the transport of message packet between
> > >    different endpoints. Cf. OSI link layer.
> > >
> > >#1 is application specific and there are many different standards (e.g.
> Linux
> > >Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
> > >#3 may be implementation specific but is important for
> interoperatibility
> > >(binary compatibility). If the user message (payload), operation and
> > >parameters are encoded in a well-defined way (e.g. some kind of packet
> format
> > >which includes operation, sender and receiver addresses, payload etc),
> > >inter-operatibility between different ODP/MSGIO implementations may be
> > >achieved or simplified.
> > >#4 is implementation (platform) specific.
> > >
> > >This proposal focuses on the API which is fundamental for source code
> > >compatibility between different platforms (ODP implementations) - this
> being
> > >the primary purpose of ODP. A packet format for messages will be
> specified as
> > >part of the reference implementation and may be seen as a proposal for a
> > >standard packet format for binary compatibility between different
> ODP/MSGIO
> > >implementations.
> > >
> > >All MSGIO calls are non-blocking (with the possible exception of
> > >odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
> > >transfer is asynchronous so it is not possible to return any status
> > >operation concerning the eventual (or remote) result of an operation.
> > >
> > >The semantics of MSGIO message passing is that sending a message to an
> endpoint
> > >will always look like it succeeds. The appearance of endpoints is
> explicitly
> > >notified through user-defined messages specified in the
> odp_msgio_lookup()
> > >call. Similarly, the disappearance (e.g. death of or otherwise lost
> connection
> > >to endpoint) is also explicitly notified through user-defined messages
> > >specified in the odp_msgio_monitor() call. The send call does not fail
> because
> > >the addressed endpoint has disappeared, the message is just silently
> discarded.
> > >
> > >Message delivery into the recipient address space is ordered (per
> priority)
> > >and reliable. Delivery of message N implies delivery of all messages <N
> > >(of the same priority). All messages (accepted by MSGIO send operation)
> will
> > >be delivered up to the point of endpoint termination or lost connection
> > >where no more messages will be delivered. Actual reception (dequeueing)
> and
> > >processing by the recipient is not guaranteed (use end-to-end
> acknowledgements
> > >for that).
> > >
> > >MSGIO endpoints can be seen as interfaces (taps) to an internal reliable
> > >multidrop network where each endpoint has a unique address which is only
> > >valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
> > >and then recreated (with the same name), the new endpoint will have a
> > >new unique address (eventually endpoints addresses will have to be
> recycled
> > >but not for a very long time). Endpoints names do not necessarily have
> to be
> > >unique. The scope of message network is not defined but it is expected
> that
> > >the scope by default corresponds to an OS instance (e.g. virtual
> machine).
> > >
> > >Proposed transport packet for message I/O (not visible to the user):
> > >uint8_t dest_addr[6]; /* Unique address of destination endpoint */
> > >uint8_t src_addr[6]; /* Unique address of source endpoint */
> > >uint16_t frametype_o; /* Outer frametype, 0x8100 */
> > >uint16_t pcp_vid; /* 3 bits of Priority Code Point, 5 bits reserved */
> > >uint16_t frametype_i; /* Inner frametype, TBD */
> > >uint8_t payload[]; /* User message */
> > >Looks very much like a VLAN-tagged Ethernet frame so should easily be
> > >transported by and processed by Ethernet HW & SW.
> > >Encoding of publish/lookup/monitor operations TBD.
> > >
> > >v3:
> > >Renamed Message Bus (MBUS) to Message I/O (MSGIO).
> > >Changed all definitions to use "msgio" instead of "mbus" prefix/infix.
> > >odp_msgio_create(): removed default input queue parameter.
> > >Removed odp_mbus_inq_setdef() and odp_mbus_inq_remdef().
> > >All input queues must now be specified using odp_msgio_inq_set() before
> > >the MSGIO endpoint is activated.
> > >Added odp_msgio_activate().
> > >Added odp_message_push_head() and odp_message_pull_head().
> > >Updated some function descriptions in msgio.h.
> > >
> > >v2:
> > >Split off all message definitions in a separate file message.h
> > >Renamed Inter Process Communication (IPC) to Message Bus (MBUS).
> > >Changed all definitions to use "mbus" instead of "ipc" prefix/infix.
> > >Renamed odp_ipc_msg_t to odp_message_t.
> > >odp_mbus_create(): Added parameter for default input queue. Explicitly
> state
> > >that the pool must be use type ODP_EVENT_MESSAGE.
> > >Renamed odp_ipc_resolve() to odp_mbus_lookup().
> > >odp_mbus_send(): Added priority parameter.
> > >Renamed odp_ipc_sender() to odp_message_sender().
> > >Renamed odp_ipc_data() to odp_message_data().
> > >Renamed odp_ipc_length() to odp_message_length().
> > >Renamed odp_ipc_reset() to odp_message_length_set().
> > >Renamed odp_ipc_alloc() to odp_message_alloc().
> > >Renamed odp_ipc_free() to odp_message_free().
> > >odp_message_alloc(): Corrected name of invalid message handle.
> > >Added message priorities and calls to set and remove input queues for
> > >specific priorities: odp_mbus_inq_set(), odp_mbus_inq_rem().
> > >
> > >Signed-off-by: Ola Liljedahl <ola.liljedahl@linaro.org>
> > >---
> > >(This document/code contribution attached is provided under the terms of
> > >agreement LES-LTM-21309)
> > >
> > >  include/odp/api/message.h                          | 199
> ++++++++++++++++++
> > >  include/odp/api/msgio.h                            | 222
> +++++++++++++++++++++
> > >  platform/linux-generic/include/odp/message.h       |  39 ++++
> > >  platform/linux-generic/include/odp/msgio.h         |  40 ++++
> > >  .../linux-generic/include/odp/plat/message_types.h |  47 +++++
> > >  .../linux-generic/include/odp/plat/msgio_types.h   |  59 ++++++
> > >  6 files changed, 606 insertions(+)
> > >  create mode 100644 include/odp/api/message.h
> > >  create mode 100644 include/odp/api/msgio.h
> > >  create mode 100644 platform/linux-generic/include/odp/message.h
> > >  create mode 100644 platform/linux-generic/include/odp/msgio.h
> > >  create mode 100644
> platform/linux-generic/include/odp/plat/message_types.h
> > >  create mode 100644
> platform/linux-generic/include/odp/plat/msgio_types.h
> > >
> > >diff --git a/include/odp/api/message.h b/include/odp/api/message.h
> > >new file mode 100644
> > >index 0000000..76697fb
> > >--- /dev/null
> > >+++ b/include/odp/api/message.h
> > >@@ -0,0 +1,199 @@
> > >+/* Copyright (c) 2015, Linaro Limited
> > >+ * All rights reserved.
> > >+ *
> > >+ * SPDX-License-Identifier:     BSD-3-Clause
> > >+ */
> > >+
> > >+
> > >+/**
> > >+ * @file
> > >+ *
> > >+ * ODP Message event type
> > >+ */
> > >+
> > >+#ifndef ODP_API_MESSAGE_H_
> > >+#define ODP_API_MESSAGE_H_
> > >+
> > >+#ifdef __cplusplus
> > >+extern "C" {
> > >+#endif
> > >+
> > >+/** @defgroup odp_message ODP MESSAGE
> > >+ *  @{
> > >+ */
> > >+
> > >+/**
> > >+ * @typedef odp_message_t
> > >+ * ODP message handle
> > >+ */
> > >+
> > >+
> > >+/**
> > >+ * Get address of sender (source) of message
> > >+ *
> > >+ * @param      msg  Message handle
> > >+ * @param[out] addr Buffer to hold sender address
> > >+ */
> > >+void odp_message_sender(odp_message_t msg,
> > >+                    uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
> > >+
> > >+/**
> > >+ * Message data pointer
> > >+ *
> > >+ * Return a pointer to the message data
> > >+ *
> > >+ * @param msg Message handle
> > >+ *
> > >+ * @return Pointer to the message data
> > >+ */
> > >+void *odp_message_data(odp_message_t msg);
> > >+
> > >+/**
> > >+ * Message data length
> > >+ *
> > >+ * Return length of the message data.
> > >+ *
> > >+ * @param msg Message handle
> > >+ *
> > >+ * @return Message length
> > >+ */
> > >+uint32_t odp_message_length(const odp_message_t msg);
> > >+
> > >+/**
> > >+ * Set message length
> > >+ *
> > >+ * Set length of the message data.
> > >+ * Increase message data length by moving message tail into message
> tailroom.
> > >+ * The actual message data is not modified.
> > >+ *
> > >+ * @param msg Message handle
> > >+ * @param len New length
> > >+ *
> > >+ * @retval 0 on success
> > >+ * @retval <0 on error (e.g. no enough tailroom)
> > >+ */
> > >+int odp_message_length_set(const odp_message_t msg, uint32_t len);
> > >+
> > >+/**
> > >+ * Message headroom length
> > >+ *
> > >+ * Return length of the message headroom
> > >+ *
> > >+ * @param msg Message handle
> > >+ *
> > >+ * @return Headroom length
> > >+ */
> > >+uint32_t odp_message_headroom(const odp_message_t msg);
> > >+
> > >+/**
> > >+ * Push out message head
> > >+ *
> > >+ * Increase message data length by moving message head into message
> headroom.
> > >+ * Message headroom is decreased with the same amount. The message
> head may be
> > >+ * pushed out up to 'headroom' bytes. Message is not modified if
> there's not
> > >+ * enough headroom space.
> > >+ *
> > >+ * odp_message_xxx:
> > >+ * length   += len
> > >+ * headroom -= len
> > >+ * data     -= len
> > >+ *
> > >+ * @param msg  Message handle
> > >+ * @param len  Number of bytes to push the head (0 ... headroom)
> > >+ *
> > >+ * @return The new data pointer
> > >+ * @retval NULL  Requested offset exceeds available headroom
> > >+ *
> > >+ * @see odp_message_headroom(), odp_message_pull_head()
> > >+ */
> > >+void *odp_message_push_head(odp_message_t msg, uint32_t len);
> > >+
> > >+/**
> > >+ * Pull in message head
> > >+ *
> > >+ * Decrease message data length by moving message head into message
> data.
> > >+ * Message headroom is increased with the same amount. The message
> head may be
> > >+ * pulled in up to 'length' bytes. Message is not modified if there's
> not
> > >+ * enough data.
> > >+ *
> > >+ * odp_message_xxx:
> > >+ * length   -= len
> > >+ * headroom += len
> > >+ * data     += len
> > >+ *
> > >+ * @param msg  Message handle
> > >+ * @param len  Number of bytes to pull the head (0 ... length)
> > >+ *
> > >+ * @return The new data pointer
> > >+ * @retval NULL  Requested offset exceeds available data
> > >+ *
> > >+ * @see odp_message_headroom(), odp_message_push_head()
> > >+ */
> > >+void *odp_message_pull_head(odp_message_t msg, uint32_t len);
> > >+
> > >+/**
> > >+ * Allocate message
> > >+ *
> > >+ * Allocate a message of a specific length.
> > >+ *
> > >+ * @param pool Message pool to allocate message from
> > >+ * @param len Length of the allocated message
> > >+ *
> > >+ * @return Message handle on success
> > >+ * @retval ODP_MESSAGE_INVALID on failure and errno set
> > >+ */
> > >+odp_message_t odp_message_alloc(odp_pool_t pool, uint32_t len);
> > >+
> > >+/**
> > >+ * Free message
> > >+ *
> > >+ * Free message back to the message pool it was allocated from.
> > >+ *
> > >+ * @param msg Handle of message to free
> > >+ */
> > >+void odp_message_free(odp_message_t msg);
> > >+
> > >+/**
> > >+ * Get message handle from event
> > >+ *
> > >+ * Converts an ODP_EVENT_MESSAGE type event to a message.
> > >+ *
> > >+ * @param ev   Event handle representing a message.
> > >+ *
> > >+ * @return Message handle
> > >+ *
> > >+ * @see odp_event_type()
> > >+ */
> > >+odp_message_t odp_message_from_event(odp_event_t ev);
> > >+
> > >+/**
> > >+ * Convert message handle to event
> > >+ *
> > >+ * @param msg  Message handle
> > >+ *
> > >+ * @return Event handle
> > >+ */
> > >+odp_event_t odp_message_to_event(odp_message_t msg);
> > >+
> > >+/**
> > >+ * Get printable value for an odp_message_t
> > >+ *
> > >+ * @param msg  Message handle to be printed
> > >+ * @return     uint64_t value that can be used to print/display this
> > >+ *         handle
> > >+ *
> > >+ * @note This routine is intended to be used for diagnostic purposes
> > >+ * to enable applications to generate a printable value that represents
> > >+ * an odp_message_t handle.
> > >+ */
> > >+uint64_t odp_message_to_u64(odp_message_t msg);
> > >+
> > >+/**
> > >+ * @}
> > >+ */
> > >+
> > >+#ifdef __cplusplus
> > >+}
> > >+#endif
> > >+
> > >+#endif
> > >diff --git a/include/odp/api/msgio.h b/include/odp/api/msgio.h
> > >new file mode 100644
> > >index 0000000..8d73021
> > >--- /dev/null
> > >+++ b/include/odp/api/msgio.h
> > >@@ -0,0 +1,222 @@
> > >+/* Copyright (c) 2015, Linaro Limited
> > >+ * All rights reserved.
> > >+ *
> > >+ * SPDX-License-Identifier:     BSD-3-Clause
> > >+ */
> > >+
> > >+
> > >+/**
> > >+ * @file
> > >+ *
> > >+ * ODP Message I/O API
> > >+ */
> > >+
> > >+#ifndef ODP_API_MSGIO_H_
> > >+#define ODP_API_MSGIO_H_
> > >+
> > >+#ifdef __cplusplus
> > >+extern "C" {
> > >+#endif
> > >+
> > >+/** @defgroup odp_msgio ODP MSGIO
> > >+ *  @{
> > >+ */
> > >+
> > >+/**
> > >+ * @typedef odp_msgio_t
> > >+ * ODP message I/O handle
> > >+ */
> > >+
> > >+/**
> > >+ * @def ODP_MSGIO_ADDR_SIZE
> > >+ * Size of the address of a message I/O endpoint
> > >+ */
> > >+
> > >+/**
> > >+ * @typedef odp_msgio_prio_t
> > >+ * ODP MSGIO message priority
> > >+ */
> > >+
> > >+/**
> > >+ * @def ODP_MSGIO_PRIO_HIGHEST
> > >+ * Highest MSGIO message priority
> > >+ */
> > >+
> > >+/**
> > >+ * @def ODP_MSGIO_PRIO_NORMAL
> > >+ * Normal MSGIO message priority
> > >+ */
> > >+
> > >+/**
> > >+ * @def ODP_MSGIO_PRIO_LOWEST
> > >+ * Lowest MSGIO message priority
> > >+ */
> > >+
> > >+/**
> > >+ * @def ODP_MSGIO_PRIO_DEFAULT
> > >+ * Default MSGIO message priority
> > >+ */
> > >+
> > >+
> > >+/**
> > >+ * Create message I/O endpoint
> > >+ *
> > >+ * Create an message I/O endpoint. The scope of the message bus is
> > >+ * not defined but it is expected that it by default encompasses the OS
> > >+ * instance but no more.
> > >+ *
> > >+ * A unique address for the endpoint is created. The addresses for
> other
> > >+ * endpoints can be obtained using the odp_msgio_lookup() call and
> specified
> > >+ * in the odp_msgio_send() call.
> > >+ *
> > >+ * @param name Name of our endpoint
> > >+ * @param pool Pool (of type ODP_EVENT_MESSAGE) for incoming messages
> > >+ *
> > >+ * @return Message I/O handle on success
> > >+ * @retval ODP_MSGIO_INVALID on failure and errno set
> > >+ */
> > >+odp_msgio_t odp_msgio_create(const char *name,
> > >+                         odp_pool_t pool);
> > >+
> > >+/**
> > >+ * Set the input queue for a specific message priority
> > >+ *
> > >+ * @param msgio  Message I/O handle
> > >+ * @param prio  Message priority
> > >+ * @param queue Queue handle
> > >+ *
> > >+ * @retval  0 on success
> > >+ * @retval <0 on failure
> > >+ */
> > >+int odp_msgio_inq_set(odp_msgio_t msgio,
> > >+                  odp_msgio_prio_t prio,
> > >+                  odp_queue_t queue);
> > >+
> > >+/**
> > >+ * Activate message I/O endpoint
> > >+ *
> > >+ * Make this endpoint visible (odp_msgio_lookup() messages will be
> returned).
> > >+ * Enable transmission and reception of messages through the endpoint.
> > >+ *
> > >+ * @param msgio Message I/O handle
> > >+ *
> > >+ * @retval 0 on success
> > >+ * @retval <0 on failure
> > >+ */
> > >+int odp_msgio_activate(odp_msgio_t msgio);
> > >+
> > >+/**
> > >+ * Destroy message I/O endpoint
> > >+ *
> > >+ * @param msgio Message I/O handle
> > >+ *
> > >+ * @retval 0 on success
> > >+ * @retval <0 on failure
> > >+ */
> > >+int odp_msgio_destroy(odp_msgio_t msgio);
> > >+
> > >+/**
> > >+ * Remove the input queue for a specific message priority
> > >+ *
> > >+ * Remove (disassociate) the matching input queue from a message I/O
> endpoint.
> > >+ * The queue itself is not touched.
> > >+ *
> > >+ * @param msgio  Message I/O handle
> > >+ * @param prio  Message priority
> > >+ *
> > >+ * @retval 0 on success
> > >+ * @retval <0 on failure
> > >+ */
> > >+int odp_msgio_inq_rem(odp_msgio_t msgio,
> > >+                  odp_msgio_prio_t prio);
> > >+
> > >+/**
> > >+ * Lookup endpoint by name
> > >+ *
> > >+ * Look up a current or future endpoint by name.
> > >+ * When the endpoint exists, return the specified message (not
> necessarily the
> > >+ * same message handle) with the endpoint as the sender.
> > >+ *
> > >+ * @param msgio Message I/O handle
> > >+ * @param name Name to look up
> > >+ * @param msg Message to return
> > >+ */
> > >+void odp_msgio_lookup(odp_msgio_t msgio,
> > >+                  const char *name,
> > >+                  odp_message_t msg);
> > >+
> > >+/**
> > >+ * Monitor endpoint
> > >+ *
> > >+ * Monitor an endpoint (which may already have disappeared).
> > >+ * When the endpoint disappears, return the specified message (not
> necessarily
> > >+ * the same message handle) with the endpoint as the sender.
> > >+ *
> > >+ * Unrecognized or invalid endpoint addresses are treated as
> non-existing
> > >+ * endpoints.
> > >+ *
> > >+ * @param     msgio Message I/O handle
> > >+ * @param[in] addr  Address of monitored endpoint
> > >+ * @param     msg   Message to return
> > >+ */
> > >+void odp_msgio_monitor(odp_msgio_t msgio,
> > >+                   const uint8_t addr[ODP_MSGIO_ADDR_SIZE],
> > >+                   odp_message_t msg);
> > >+
> > >+/**
> > >+ * Send message
> > >+ *
> > >+ * Send a message to an endpoint (which may no longer exist).
> > >+ *
> > >+ * Message delivery into the recipient address space is ordered (per
> priority)
> > >+ * and reliable. Delivery of message N implies delivery of all
> messages <N
> > >+ * (of the same priority).
> > >+ * All messages (accepted by MSGIO) will be delivered up to the point
> of
> > >+ * recipient endpoint termination or lost connection where no more
> messages
> > >+ * will be delivered.
> > >+ *
> > >+ * Actual reception (dequeueing) and processing by the recipient is not
> > >+ * guaranteed (use user level acknowledgements for that).
> > >+ *
> > >+ * Monitor the remote endpoint to detect the disappearance or
> disconnection
> > >+ * of the endpoint.
> > >+ *
> > >+ * A message may temporarily not be accepted for transmission (e.g. due
> > >+ * to out of local buffer space), -1 will be returned and ODP errno set
> > >+ * to EAGAIN.
> > >+ *
> > >+ * @param     msgio Message I/O handle
> > >+ * @param     msg   Message to send
> > >+ * @param     prio  Priority of message
> > >+ * @param[in] addr  Address of recipient endpoint
> > >+ *
> > >+ * @retval 0 on success
> > >+ * @retval <0 on error
> > >+ */
> > >+int odp_msgio_send(odp_msgio_t msgio,
> > >+               odp_message_t msg,
> > >+               odp_msgio_prio_t prio,
> > >+               const uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
> > >+
> > >+/**
> > >+ * Get printable value for an odp_msgio_t
> > >+ *
> > >+ * @param msgio  Message I/O handle to be printed
> > >+ * @return       uint64_t value that can be used to print/display this
> > >+ *           handle
> > >+ *
> > >+ * @note This routine is intended to be used for diagnostic purposes
> > >+ * to enable applications to generate a printable value that represents
> > >+ * an odp_msgio_t handle.
> > >+ */
> > >+uint64_t odp_msgio_to_u64(odp_msgio_t msgio);
> > >+
> > >+/**
> > >+ * @}
> > >+ */
> > >+
> > >+#ifdef __cplusplus
> > >+}
> > >+#endif
> > >+
> > >+#endif
> > >diff --git a/platform/linux-generic/include/odp/message.h
> b/platform/linux-generic/include/odp/message.h
> > >new file mode 100644
> > >index 0000000..2a29f59
> > >--- /dev/null
> > >+++ b/platform/linux-generic/include/odp/message.h
> > >@@ -0,0 +1,39 @@
> > >+/* Copyright (c) 2013, Linaro Limited
> > >+ * All rights reserved.
> > >+ *
> > >+ * SPDX-License-Identifier:     BSD-3-Clause
> > >+ */
> > >+
> > >+/**
> > >+ * @file
> > >+ *
> > >+ * ODP Message Bus
> > >+ */
> > >+
> > >+#ifndef ODP_PLAT_MESSAGE_H_
> > >+#define ODP_PLAT_MESSAGE_H_
> > >+
> > >+#ifdef __cplusplus
> > >+extern "C" {
> > >+#endif
> > >+
> > >+#include <odp/std_types.h>
> > >+#include <odp/plat/pool_types.h>
> > >+#include <odp/plat/message_types.h>
> > >+#include <odp/plat/queue_types.h>
> > >+
> > >+/** @ingroup odp_message
> > >+ *  @{
> > >+ */
> > >+
> > >+/**
> > >+ * @}
> > >+ */
> > >+
> > >+#include <odp/api/message.h>
> > >+
> > >+#ifdef __cplusplus
> > >+}
> > >+#endif
> > >+
> > >+#endif
> > >diff --git a/platform/linux-generic/include/odp/msgio.h
> b/platform/linux-generic/include/odp/msgio.h
> > >new file mode 100644
> > >index 0000000..f45b3a2
> > >--- /dev/null
> > >+++ b/platform/linux-generic/include/odp/msgio.h
> > >@@ -0,0 +1,40 @@
> > >+/* Copyright (c) 2013, Linaro Limited
> > >+ * All rights reserved.
> > >+ *
> > >+ * SPDX-License-Identifier:     BSD-3-Clause
> > >+ */
> > >+
> > >+/**
> > >+ * @file
> > >+ *
> > >+ * ODP Message Bus
> > >+ */
> > >+
> > >+#ifndef ODP_PLAT_MSGIO_H_
> > >+#define ODP_PLAT_MSGIO_H_
> > >+
> > >+#ifdef __cplusplus
> > >+extern "C" {
> > >+#endif
> > >+
> > >+#include <odp/std_types.h>
> > >+#include <odp/plat/pool_types.h>
> > >+#include <odp/plat/msgio_types.h>
> > >+#include <odp/plat/message_types.h>
> > >+#include <odp/plat/queue_types.h>
> > >+
> > >+/** @ingroup odp_msgio
> > >+ *  @{
> > >+ */
> > >+
> > >+/**
> > >+ * @}
> > >+ */
> > >+
> > >+#include <odp/api/msgio.h>
> > >+
> > >+#ifdef __cplusplus
> > >+}
> > >+#endif
> > >+
> > >+#endif
> > >diff --git a/platform/linux-generic/include/odp/plat/message_types.h
> b/platform/linux-generic/include/odp/plat/message_types.h
> > >new file mode 100644
> > >index 0000000..3a42064
> > >--- /dev/null
> > >+++ b/platform/linux-generic/include/odp/plat/message_types.h
> > >@@ -0,0 +1,47 @@
> > >+/* Copyright (c) 2015, Linaro Limited
> > >+ * All rights reserved.
> > >+ *
> > >+ * SPDX-License-Identifier: BSD-3-Clause
> > >+ */
> > >+
> > >+
> > >+/**
> > >+ * @file
> > >+ *
> > >+ * ODP Message message type
> > >+ */
> > >+
> > >+#ifndef ODP_MESSAGE_TYPES_H_
> > >+#define ODP_MESSAGE_TYPES_H_
> > >+
> > >+#ifdef __cplusplus
> > >+extern "C" {
> > >+#endif
> > >+
> > >+#include <odp/std_types.h>
> > >+#include <odp/plat/strong_types.h>
> > >+
> > >+/** @addtogroup odp_message ODP message
> > >+ *  Operations on a message.
> > >+ *  @{
> > >+ */
> > >+
> > >+typedef ODP_HANDLE_T(odp_message_t);
> > >+
> > >+#define ODP_MESSAGE_INVALID _odp_cast_scalar(odp_message_t, 0xffffffff)
> > >+
> > >+/** Get printable format of odp_message_t */
> > >+static inline uint64_t odp_message_to_u64(odp_message_t hdl)
> > >+{
> > >+    return _odp_pri(hdl);
> > >+}
> > >+
> > >+/**
> > >+ * @}
> > >+ */
> > >+
> > >+#ifdef __cplusplus
> > >+}
> > >+#endif
> > >+
> > >+#endif
> > >diff --git a/platform/linux-generic/include/odp/plat/msgio_types.h
> b/platform/linux-generic/include/odp/plat/msgio_types.h
> > >new file mode 100644
> > >index 0000000..9d9f9da
> > >--- /dev/null
> > >+++ b/platform/linux-generic/include/odp/plat/msgio_types.h
> > >@@ -0,0 +1,59 @@
> > >+/* Copyright (c) 2015, Linaro Limited
> > >+ * All rights reserved.
> > >+ *
> > >+ * SPDX-License-Identifier: BSD-3-Clause
> > >+ */
> > >+
> > >+
> > >+/**
> > >+ * @file
> > >+ *
> > >+ * ODP Message bus types
> > >+ */
> > >+
> > >+#ifndef ODP_MSGIO_TYPES_H_
> > >+#define ODP_MSGIO_TYPES_H_
> > >+
> > >+#ifdef __cplusplus
> > >+extern "C" {
> > >+#endif
> > >+
> > >+#include <odp/std_types.h>
> > >+#include <odp/plat/strong_types.h>
> > >+
> > >+/** @addtogroup odp_msgio ODP message bus
> > >+ *  Operations on a message bus.
> > >+ *  @{
> > >+ */
> > >+
> > >+typedef int odp_msgio_prio_t;
> > >+
> > >+#define ODP_MSGIO_PRIO_HIGHEST  0
> > >+
> > >+#define ODP_MSGIO_PRIO_NORMAL   2
> > >+
> > >+#define ODP_MSGIO_PRIO_LOWEST   3
> > >+
> > >+#define ODP_MSGIO_PRIO_DEFAULT  ODP_MSGIO_PRIO_NORMAL
> > >+
> > >+typedef ODP_HANDLE_T(odp_msgio_t);
> > >+
> > >+#define ODP_MSGIO_INVALID _odp_cast_scalar(odp_msgio_t, 0xffffffff)
> > >+
> > >+#define ODP_MSGIO_ADDR_SIZE 6
> > >+
> > >+/** Get printable format of odp_msgio_t */
> > >+static inline uint64_t odp_msgio_to_u64(odp_msgio_t hdl)
> > >+{
> > >+    return _odp_pri(hdl);
> > >+}
> > >+
> > >+/**
> > >+ * @}
> > >+ */
> > >+
> > >+#ifdef __cplusplus
> > >+}
> > >+#endif
> > >+
> > >+#endif
> > >
> >
> >
> > --
> > Benoît GANNE
> > Field Application Engineer, Kalray
> > +33 (0)648 125 843
> > _______________________________________________
> > lng-odp mailing list
> > lng-odp@lists.linaro.org
> > https://lists.linaro.org/mailman/listinfo/lng-odp
>

On 2 June 2015 at 15:35, Ola Liljedahl <ola.liljedahl@linaro.org> wrote:

>
>
> On 2 June 2015 at 14:36, Benoît Ganne <bganne@kalray.eu> wrote:
>
>> Hi Ola, all,
>>
>> On 05/27/2015 10:08 AM, Ola Liljedahl wrote:
>>
>>> Here is my third attempt at a ODP API for MSGIO - asynchronous message
>>> passing
>>> based IPC for a shared nothing architecture.
>>>
>>
>> I understand the value of this, but I feel I need to ask the question:
>> does it really belong to ODP ? My understanding so far was that ODP should
>> not try to replace OS services: for example, thread or process related API
>> were ruled out. This looks to me a lot like D-BUS or ZeroMQ (or <name your
>> favorite solution here>). Wouldn't it be better to let the application
>> select what they want to use instead of providing our own?
>>
> Yes but I want MSGIO IPC to be part of the ODP functionality such as
> scheduling, not something completely separate. So messages need to be some
> type of events (or at least incoming messages notified using events) and
> when you set up your message IO, input messages (or notifications) are put
> on some queue. Perhaps this can be done differently. It would be great of
> there was a generic way of including alien functionality (e.g. D-BUS or
> ZeroMQ as you mention) in the ODP framework. Any suggestions how this could
> be done?
>

Thinking about using notification events instead of the messages
themselves...

If I use D-BUS or ZeroMQ, I get a socket descriptor. I don't want to block
on this descriptor, my application is likely blocking in odp_schedule() or
polling some queues. I could allocate an event (of whatever type) and
associate that with the socket descriptor. When the socket can read or
written, the event will be enqueued on some queue (also specified). My
application will receive the notification event and will perform all
message operations on the non-blocking socket. When no more messages to
read or message write would block, the application re-registers the event
and queue.

The only API we would need is:
void odp_notification_register(int sd, odp_event_t evt, odp_queue_t queue);
Perhaps also a call to unregister the notification, to be used at
termination.

This would be a nice and generic design (not only for this specific use
case). But it also seems limited to e.g. Linux and similar operating
systems. What about other operating environments (e.g. bare metal)? Would
those environments not implement this API? Can we have an ODP API which
only is implemented on some platforms/environments?


>
>> ben
>>
>>
>>  IPC/message passing can be separated into a number of different layers.
>>> 1) Message formats and protocols (as seen by the user).
>>> 2) API - defines syntax and semantics of MSGIO operations.
>>> 3) Transport interface - defines a binary packet format used by MSGIO
>>>     endpoints/implementations. Cf. OSI network layer.
>>> 4) Transport layer - implements the transport of message packet between
>>>     different endpoints. Cf. OSI link layer.
>>>
>>> #1 is application specific and there are many different standards (e.g.
>>> Linux
>>> Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
>>> #3 may be implementation specific but is important for interoperatibility
>>> (binary compatibility). If the user message (payload), operation and
>>> parameters are encoded in a well-defined way (e.g. some kind of packet
>>> format
>>> which includes operation, sender and receiver addresses, payload etc),
>>> inter-operatibility between different ODP/MSGIO implementations may be
>>> achieved or simplified.
>>> #4 is implementation (platform) specific.
>>>
>>> This proposal focuses on the API which is fundamental for source code
>>> compatibility between different platforms (ODP implementations) - this
>>> being
>>> the primary purpose of ODP. A packet format for messages will be
>>> specified as
>>> part of the reference implementation and may be seen as a proposal for a
>>> standard packet format for binary compatibility between different
>>> ODP/MSGIO
>>> implementations.
>>>
>>> All MSGIO calls are non-blocking (with the possible exception of
>>> odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
>>> transfer is asynchronous so it is not possible to return any status
>>> operation concerning the eventual (or remote) result of an operation.
>>>
>>> The semantics of MSGIO message passing is that sending a message to an
>>> endpoint
>>> will always look like it succeeds. The appearance of endpoints is
>>> explicitly
>>> notified through user-defined messages specified in the
>>> odp_msgio_lookup()
>>> call. Similarly, the disappearance (e.g. death of or otherwise lost
>>> connection
>>> to endpoint) is also explicitly notified through user-defined messages
>>> specified in the odp_msgio_monitor() call. The send call does not fail
>>> because
>>> the addressed endpoint has disappeared, the message is just silently
>>> discarded.
>>>
>>> Message delivery into the recipient address space is ordered (per
>>> priority)
>>> and reliable. Delivery of message N implies delivery of all messages <N
>>> (of the same priority). All messages (accepted by MSGIO send operation)
>>> will
>>> be delivered up to the point of endpoint termination or lost connection
>>> where no more messages will be delivered. Actual reception (dequeueing)
>>> and
>>> processing by the recipient is not guaranteed (use end-to-end
>>> acknowledgements
>>> for that).
>>>
>>> MSGIO endpoints can be seen as interfaces (taps) to an internal reliable
>>> multidrop network where each endpoint has a unique address which is only
>>> valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
>>> and then recreated (with the same name), the new endpoint will have a
>>> new unique address (eventually endpoints addresses will have to be
>>> recycled
>>> but not for a very long time). Endpoints names do not necessarily have
>>> to be
>>> unique. The scope of message network is not defined but it is expected
>>> that
>>> the scope by default corresponds to an OS instance (e.g. virtual
>>> machine).
>>>
>>> Proposed transport packet for message I/O (not visible to the user):
>>> uint8_t dest_addr[6]; /* Unique address of destination endpoint */
>>> uint8_t src_addr[6]; /* Unique address of source endpoint */
>>> uint16_t frametype_o; /* Outer frametype, 0x8100 */
>>> uint16_t pcp_vid; /* 3 bits of Priority Code Point, 5 bits reserved */
>>> uint16_t frametype_i; /* Inner frametype, TBD */
>>> uint8_t payload[]; /* User message */
>>> Looks very much like a VLAN-tagged Ethernet frame so should easily be
>>> transported by and processed by Ethernet HW & SW.
>>> Encoding of publish/lookup/monitor operations TBD.
>>>
>>> v3:
>>> Renamed Message Bus (MBUS) to Message I/O (MSGIO).
>>> Changed all definitions to use "msgio" instead of "mbus" prefix/infix.
>>> odp_msgio_create(): removed default input queue parameter.
>>> Removed odp_mbus_inq_setdef() and odp_mbus_inq_remdef().
>>> All input queues must now be specified using odp_msgio_inq_set() before
>>> the MSGIO endpoint is activated.
>>> Added odp_msgio_activate().
>>> Added odp_message_push_head() and odp_message_pull_head().
>>> Updated some function descriptions in msgio.h.
>>>
>>> v2:
>>> Split off all message definitions in a separate file message.h
>>> Renamed Inter Process Communication (IPC) to Message Bus (MBUS).
>>> Changed all definitions to use "mbus" instead of "ipc" prefix/infix.
>>> Renamed odp_ipc_msg_t to odp_message_t.
>>> odp_mbus_create(): Added parameter for default input queue. Explicitly
>>> state
>>> that the pool must be use type ODP_EVENT_MESSAGE.
>>> Renamed odp_ipc_resolve() to odp_mbus_lookup().
>>> odp_mbus_send(): Added priority parameter.
>>> Renamed odp_ipc_sender() to odp_message_sender().
>>> Renamed odp_ipc_data() to odp_message_data().
>>> Renamed odp_ipc_length() to odp_message_length().
>>> Renamed odp_ipc_reset() to odp_message_length_set().
>>> Renamed odp_ipc_alloc() to odp_message_alloc().
>>> Renamed odp_ipc_free() to odp_message_free().
>>> odp_message_alloc(): Corrected name of invalid message handle.
>>> Added message priorities and calls to set and remove input queues for
>>> specific priorities: odp_mbus_inq_set(), odp_mbus_inq_rem().
>>>
>>> Signed-off-by: Ola Liljedahl <ola.liljedahl@linaro.org>
>>> ---
>>> (This document/code contribution attached is provided under the terms of
>>> agreement LES-LTM-21309)
>>>
>>>   include/odp/api/message.h                          | 199
>>> ++++++++++++++++++
>>>   include/odp/api/msgio.h                            | 222
>>> +++++++++++++++++++++
>>>   platform/linux-generic/include/odp/message.h       |  39 ++++
>>>   platform/linux-generic/include/odp/msgio.h         |  40 ++++
>>>   .../linux-generic/include/odp/plat/message_types.h |  47 +++++
>>>   .../linux-generic/include/odp/plat/msgio_types.h   |  59 ++++++
>>>   6 files changed, 606 insertions(+)
>>>   create mode 100644 include/odp/api/message.h
>>>   create mode 100644 include/odp/api/msgio.h
>>>   create mode 100644 platform/linux-generic/include/odp/message.h
>>>   create mode 100644 platform/linux-generic/include/odp/msgio.h
>>>   create mode 100644
>>> platform/linux-generic/include/odp/plat/message_types.h
>>>   create mode 100644
>>> platform/linux-generic/include/odp/plat/msgio_types.h
>>>
>>> diff --git a/include/odp/api/message.h b/include/odp/api/message.h
>>> new file mode 100644
>>> index 0000000..76697fb
>>> --- /dev/null
>>> +++ b/include/odp/api/message.h
>>> @@ -0,0 +1,199 @@
>>> +/* Copyright (c) 2015, Linaro Limited
>>> + * All rights reserved.
>>> + *
>>> + * SPDX-License-Identifier:     BSD-3-Clause
>>> + */
>>> +
>>> +
>>> +/**
>>> + * @file
>>> + *
>>> + * ODP Message event type
>>> + */
>>> +
>>> +#ifndef ODP_API_MESSAGE_H_
>>> +#define ODP_API_MESSAGE_H_
>>> +
>>> +#ifdef __cplusplus
>>> +extern "C" {
>>> +#endif
>>> +
>>> +/** @defgroup odp_message ODP MESSAGE
>>> + *  @{
>>> + */
>>> +
>>> +/**
>>> + * @typedef odp_message_t
>>> + * ODP message handle
>>> + */
>>> +
>>> +
>>> +/**
>>> + * Get address of sender (source) of message
>>> + *
>>> + * @param      msg  Message handle
>>> + * @param[out] addr Buffer to hold sender address
>>> + */
>>> +void odp_message_sender(odp_message_t msg,
>>> +                       uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
>>> +
>>> +/**
>>> + * Message data pointer
>>> + *
>>> + * Return a pointer to the message data
>>> + *
>>> + * @param msg Message handle
>>> + *
>>> + * @return Pointer to the message data
>>> + */
>>> +void *odp_message_data(odp_message_t msg);
>>> +
>>> +/**
>>> + * Message data length
>>> + *
>>> + * Return length of the message data.
>>> + *
>>> + * @param msg Message handle
>>> + *
>>> + * @return Message length
>>> + */
>>> +uint32_t odp_message_length(const odp_message_t msg);
>>> +
>>> +/**
>>> + * Set message length
>>> + *
>>> + * Set length of the message data.
>>> + * Increase message data length by moving message tail into message
>>> tailroom.
>>> + * The actual message data is not modified.
>>> + *
>>> + * @param msg Message handle
>>> + * @param len New length
>>> + *
>>> + * @retval 0 on success
>>> + * @retval <0 on error (e.g. no enough tailroom)
>>> + */
>>> +int odp_message_length_set(const odp_message_t msg, uint32_t len);
>>> +
>>> +/**
>>> + * Message headroom length
>>> + *
>>> + * Return length of the message headroom
>>> + *
>>> + * @param msg Message handle
>>> + *
>>> + * @return Headroom length
>>> + */
>>> +uint32_t odp_message_headroom(const odp_message_t msg);
>>> +
>>> +/**
>>> + * Push out message head
>>> + *
>>> + * Increase message data length by moving message head into message
>>> headroom.
>>> + * Message headroom is decreased with the same amount. The message head
>>> may be
>>> + * pushed out up to 'headroom' bytes. Message is not modified if
>>> there's not
>>> + * enough headroom space.
>>> + *
>>> + * odp_message_xxx:
>>> + * length   += len
>>> + * headroom -= len
>>> + * data     -= len
>>> + *
>>> + * @param msg  Message handle
>>> + * @param len  Number of bytes to push the head (0 ... headroom)
>>> + *
>>> + * @return The new data pointer
>>> + * @retval NULL  Requested offset exceeds available headroom
>>> + *
>>> + * @see odp_message_headroom(), odp_message_pull_head()
>>> + */
>>> +void *odp_message_push_head(odp_message_t msg, uint32_t len);
>>> +
>>> +/**
>>> + * Pull in message head
>>> + *
>>> + * Decrease message data length by moving message head into message
>>> data.
>>> + * Message headroom is increased with the same amount. The message head
>>> may be
>>> + * pulled in up to 'length' bytes. Message is not modified if there's
>>> not
>>> + * enough data.
>>> + *
>>> + * odp_message_xxx:
>>> + * length   -= len
>>> + * headroom += len
>>> + * data     += len
>>> + *
>>> + * @param msg  Message handle
>>> + * @param len  Number of bytes to pull the head (0 ... length)
>>> + *
>>> + * @return The new data pointer
>>> + * @retval NULL  Requested offset exceeds available data
>>> + *
>>> + * @see odp_message_headroom(), odp_message_push_head()
>>> + */
>>> +void *odp_message_pull_head(odp_message_t msg, uint32_t len);
>>> +
>>> +/**
>>> + * Allocate message
>>> + *
>>> + * Allocate a message of a specific length.
>>> + *
>>> + * @param pool Message pool to allocate message from
>>> + * @param len Length of the allocated message
>>> + *
>>> + * @return Message handle on success
>>> + * @retval ODP_MESSAGE_INVALID on failure and errno set
>>> + */
>>> +odp_message_t odp_message_alloc(odp_pool_t pool, uint32_t len);
>>> +
>>> +/**
>>> + * Free message
>>> + *
>>> + * Free message back to the message pool it was allocated from.
>>> + *
>>> + * @param msg Handle of message to free
>>> + */
>>> +void odp_message_free(odp_message_t msg);
>>> +
>>> +/**
>>> + * Get message handle from event
>>> + *
>>> + * Converts an ODP_EVENT_MESSAGE type event to a message.
>>> + *
>>> + * @param ev   Event handle representing a message.
>>> + *
>>> + * @return Message handle
>>> + *
>>> + * @see odp_event_type()
>>> + */
>>> +odp_message_t odp_message_from_event(odp_event_t ev);
>>> +
>>> +/**
>>> + * Convert message handle to event
>>> + *
>>> + * @param msg  Message handle
>>> + *
>>> + * @return Event handle
>>> + */
>>> +odp_event_t odp_message_to_event(odp_message_t msg);
>>> +
>>> +/**
>>> + * Get printable value for an odp_message_t
>>> + *
>>> + * @param msg  Message handle to be printed
>>> + * @return     uint64_t value that can be used to print/display this
>>> + *            handle
>>> + *
>>> + * @note This routine is intended to be used for diagnostic purposes
>>> + * to enable applications to generate a printable value that represents
>>> + * an odp_message_t handle.
>>> + */
>>> +uint64_t odp_message_to_u64(odp_message_t msg);
>>> +
>>> +/**
>>> + * @}
>>> + */
>>> +
>>> +#ifdef __cplusplus
>>> +}
>>> +#endif
>>> +
>>> +#endif
>>> diff --git a/include/odp/api/msgio.h b/include/odp/api/msgio.h
>>> new file mode 100644
>>> index 0000000..8d73021
>>> --- /dev/null
>>> +++ b/include/odp/api/msgio.h
>>> @@ -0,0 +1,222 @@
>>> +/* Copyright (c) 2015, Linaro Limited
>>> + * All rights reserved.
>>> + *
>>> + * SPDX-License-Identifier:     BSD-3-Clause
>>> + */
>>> +
>>> +
>>> +/**
>>> + * @file
>>> + *
>>> + * ODP Message I/O API
>>> + */
>>> +
>>> +#ifndef ODP_API_MSGIO_H_
>>> +#define ODP_API_MSGIO_H_
>>> +
>>> +#ifdef __cplusplus
>>> +extern "C" {
>>> +#endif
>>> +
>>> +/** @defgroup odp_msgio ODP MSGIO
>>> + *  @{
>>> + */
>>> +
>>> +/**
>>> + * @typedef odp_msgio_t
>>> + * ODP message I/O handle
>>> + */
>>> +
>>> +/**
>>> + * @def ODP_MSGIO_ADDR_SIZE
>>> + * Size of the address of a message I/O endpoint
>>> + */
>>> +
>>> +/**
>>> + * @typedef odp_msgio_prio_t
>>> + * ODP MSGIO message priority
>>> + */
>>> +
>>> +/**
>>> + * @def ODP_MSGIO_PRIO_HIGHEST
>>> + * Highest MSGIO message priority
>>> + */
>>> +
>>> +/**
>>> + * @def ODP_MSGIO_PRIO_NORMAL
>>> + * Normal MSGIO message priority
>>> + */
>>> +
>>> +/**
>>> + * @def ODP_MSGIO_PRIO_LOWEST
>>> + * Lowest MSGIO message priority
>>> + */
>>> +
>>> +/**
>>> + * @def ODP_MSGIO_PRIO_DEFAULT
>>> + * Default MSGIO message priority
>>> + */
>>> +
>>> +
>>> +/**
>>> + * Create message I/O endpoint
>>> + *
>>> + * Create an message I/O endpoint. The scope of the message bus is
>>> + * not defined but it is expected that it by default encompasses the OS
>>> + * instance but no more.
>>> + *
>>> + * A unique address for the endpoint is created. The addresses for other
>>> + * endpoints can be obtained using the odp_msgio_lookup() call and
>>> specified
>>> + * in the odp_msgio_send() call.
>>> + *
>>> + * @param name Name of our endpoint
>>> + * @param pool Pool (of type ODP_EVENT_MESSAGE) for incoming messages
>>> + *
>>> + * @return Message I/O handle on success
>>> + * @retval ODP_MSGIO_INVALID on failure and errno set
>>> + */
>>> +odp_msgio_t odp_msgio_create(const char *name,
>>> +                            odp_pool_t pool);
>>> +
>>> +/**
>>> + * Set the input queue for a specific message priority
>>> + *
>>> + * @param msgio  Message I/O handle
>>> + * @param prio  Message priority
>>> + * @param queue Queue handle
>>> + *
>>> + * @retval  0 on success
>>> + * @retval <0 on failure
>>> + */
>>> +int odp_msgio_inq_set(odp_msgio_t msgio,
>>> +                     odp_msgio_prio_t prio,
>>> +                     odp_queue_t queue);
>>> +
>>> +/**
>>> + * Activate message I/O endpoint
>>> + *
>>> + * Make this endpoint visible (odp_msgio_lookup() messages will be
>>> returned).
>>> + * Enable transmission and reception of messages through the endpoint.
>>> + *
>>> + * @param msgio Message I/O handle
>>> + *
>>> + * @retval 0 on success
>>> + * @retval <0 on failure
>>> + */
>>> +int odp_msgio_activate(odp_msgio_t msgio);
>>> +
>>> +/**
>>> + * Destroy message I/O endpoint
>>> + *
>>> + * @param msgio Message I/O handle
>>> + *
>>> + * @retval 0 on success
>>> + * @retval <0 on failure
>>> + */
>>> +int odp_msgio_destroy(odp_msgio_t msgio);
>>> +
>>> +/**
>>> + * Remove the input queue for a specific message priority
>>> + *
>>> + * Remove (disassociate) the matching input queue from a message I/O
>>> endpoint.
>>> + * The queue itself is not touched.
>>> + *
>>> + * @param msgio  Message I/O handle
>>> + * @param prio  Message priority
>>> + *
>>> + * @retval 0 on success
>>> + * @retval <0 on failure
>>> + */
>>> +int odp_msgio_inq_rem(odp_msgio_t msgio,
>>> +                     odp_msgio_prio_t prio);
>>> +
>>> +/**
>>> + * Lookup endpoint by name
>>> + *
>>> + * Look up a current or future endpoint by name.
>>> + * When the endpoint exists, return the specified message (not
>>> necessarily the
>>> + * same message handle) with the endpoint as the sender.
>>> + *
>>> + * @param msgio Message I/O handle
>>> + * @param name Name to look up
>>> + * @param msg Message to return
>>> + */
>>> +void odp_msgio_lookup(odp_msgio_t msgio,
>>> +                     const char *name,
>>> +                     odp_message_t msg);
>>> +
>>> +/**
>>> + * Monitor endpoint
>>> + *
>>> + * Monitor an endpoint (which may already have disappeared).
>>> + * When the endpoint disappears, return the specified message (not
>>> necessarily
>>> + * the same message handle) with the endpoint as the sender.
>>> + *
>>> + * Unrecognized or invalid endpoint addresses are treated as
>>> non-existing
>>> + * endpoints.
>>> + *
>>> + * @param     msgio Message I/O handle
>>> + * @param[in] addr  Address of monitored endpoint
>>> + * @param     msg   Message to return
>>> + */
>>> +void odp_msgio_monitor(odp_msgio_t msgio,
>>> +                      const uint8_t addr[ODP_MSGIO_ADDR_SIZE],
>>> +                      odp_message_t msg);
>>> +
>>> +/**
>>> + * Send message
>>> + *
>>> + * Send a message to an endpoint (which may no longer exist).
>>> + *
>>> + * Message delivery into the recipient address space is ordered (per
>>> priority)
>>> + * and reliable. Delivery of message N implies delivery of all messages
>>> <N
>>> + * (of the same priority).
>>> + * All messages (accepted by MSGIO) will be delivered up to the point of
>>> + * recipient endpoint termination or lost connection where no more
>>> messages
>>> + * will be delivered.
>>> + *
>>> + * Actual reception (dequeueing) and processing by the recipient is not
>>> + * guaranteed (use user level acknowledgements for that).
>>> + *
>>> + * Monitor the remote endpoint to detect the disappearance or
>>> disconnection
>>> + * of the endpoint.
>>> + *
>>> + * A message may temporarily not be accepted for transmission (e.g. due
>>> + * to out of local buffer space), -1 will be returned and ODP errno set
>>> + * to EAGAIN.
>>> + *
>>> + * @param     msgio Message I/O handle
>>> + * @param     msg   Message to send
>>> + * @param     prio  Priority of message
>>> + * @param[in] addr  Address of recipient endpoint
>>> + *
>>> + * @retval 0 on success
>>> + * @retval <0 on error
>>> + */
>>> +int odp_msgio_send(odp_msgio_t msgio,
>>> +                  odp_message_t msg,
>>> +                  odp_msgio_prio_t prio,
>>> +                  const uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
>>> +
>>> +/**
>>> + * Get printable value for an odp_msgio_t
>>> + *
>>> + * @param msgio  Message I/O handle to be printed
>>> + * @return       uint64_t value that can be used to print/display this
>>> + *              handle
>>> + *
>>> + * @note This routine is intended to be used for diagnostic purposes
>>> + * to enable applications to generate a printable value that represents
>>> + * an odp_msgio_t handle.
>>> + */
>>> +uint64_t odp_msgio_to_u64(odp_msgio_t msgio);
>>> +
>>> +/**
>>> + * @}
>>> + */
>>> +
>>> +#ifdef __cplusplus
>>> +}
>>> +#endif
>>> +
>>> +#endif
>>> diff --git a/platform/linux-generic/include/odp/message.h
>>> b/platform/linux-generic/include/odp/message.h
>>> new file mode 100644
>>> index 0000000..2a29f59
>>> --- /dev/null
>>> +++ b/platform/linux-generic/include/odp/message.h
>>> @@ -0,0 +1,39 @@
>>> +/* Copyright (c) 2013, Linaro Limited
>>> + * All rights reserved.
>>> + *
>>> + * SPDX-License-Identifier:     BSD-3-Clause
>>> + */
>>> +
>>> +/**
>>> + * @file
>>> + *
>>> + * ODP Message Bus
>>> + */
>>> +
>>> +#ifndef ODP_PLAT_MESSAGE_H_
>>> +#define ODP_PLAT_MESSAGE_H_
>>> +
>>> +#ifdef __cplusplus
>>> +extern "C" {
>>> +#endif
>>> +
>>> +#include <odp/std_types.h>
>>> +#include <odp/plat/pool_types.h>
>>> +#include <odp/plat/message_types.h>
>>> +#include <odp/plat/queue_types.h>
>>> +
>>> +/** @ingroup odp_message
>>> + *  @{
>>> + */
>>> +
>>> +/**
>>> + * @}
>>> + */
>>> +
>>> +#include <odp/api/message.h>
>>> +
>>> +#ifdef __cplusplus
>>> +}
>>> +#endif
>>> +
>>> +#endif
>>> diff --git a/platform/linux-generic/include/odp/msgio.h
>>> b/platform/linux-generic/include/odp/msgio.h
>>> new file mode 100644
>>> index 0000000..f45b3a2
>>> --- /dev/null
>>> +++ b/platform/linux-generic/include/odp/msgio.h
>>> @@ -0,0 +1,40 @@
>>> +/* Copyright (c) 2013, Linaro Limited
>>> + * All rights reserved.
>>> + *
>>> + * SPDX-License-Identifier:     BSD-3-Clause
>>> + */
>>> +
>>> +/**
>>> + * @file
>>> + *
>>> + * ODP Message Bus
>>> + */
>>> +
>>> +#ifndef ODP_PLAT_MSGIO_H_
>>> +#define ODP_PLAT_MSGIO_H_
>>> +
>>> +#ifdef __cplusplus
>>> +extern "C" {
>>> +#endif
>>> +
>>> +#include <odp/std_types.h>
>>> +#include <odp/plat/pool_types.h>
>>> +#include <odp/plat/msgio_types.h>
>>> +#include <odp/plat/message_types.h>
>>> +#include <odp/plat/queue_types.h>
>>> +
>>> +/** @ingroup odp_msgio
>>> + *  @{
>>> + */
>>> +
>>> +/**
>>> + * @}
>>> + */
>>> +
>>> +#include <odp/api/msgio.h>
>>> +
>>> +#ifdef __cplusplus
>>> +}
>>> +#endif
>>> +
>>> +#endif
>>> diff --git a/platform/linux-generic/include/odp/plat/message_types.h
>>> b/platform/linux-generic/include/odp/plat/message_types.h
>>> new file mode 100644
>>> index 0000000..3a42064
>>> --- /dev/null
>>> +++ b/platform/linux-generic/include/odp/plat/message_types.h
>>> @@ -0,0 +1,47 @@
>>> +/* Copyright (c) 2015, Linaro Limited
>>> + * All rights reserved.
>>> + *
>>> + * SPDX-License-Identifier: BSD-3-Clause
>>> + */
>>> +
>>> +
>>> +/**
>>> + * @file
>>> + *
>>> + * ODP Message message type
>>> + */
>>> +
>>> +#ifndef ODP_MESSAGE_TYPES_H_
>>> +#define ODP_MESSAGE_TYPES_H_
>>> +
>>> +#ifdef __cplusplus
>>> +extern "C" {
>>> +#endif
>>> +
>>> +#include <odp/std_types.h>
>>> +#include <odp/plat/strong_types.h>
>>> +
>>> +/** @addtogroup odp_message ODP message
>>> + *  Operations on a message.
>>> + *  @{
>>> + */
>>> +
>>> +typedef ODP_HANDLE_T(odp_message_t);
>>> +
>>> +#define ODP_MESSAGE_INVALID _odp_cast_scalar(odp_message_t, 0xffffffff)
>>> +
>>> +/** Get printable format of odp_message_t */
>>> +static inline uint64_t odp_message_to_u64(odp_message_t hdl)
>>> +{
>>> +       return _odp_pri(hdl);
>>> +}
>>> +
>>> +/**
>>> + * @}
>>> + */
>>> +
>>> +#ifdef __cplusplus
>>> +}
>>> +#endif
>>> +
>>> +#endif
>>> diff --git a/platform/linux-generic/include/odp/plat/msgio_types.h
>>> b/platform/linux-generic/include/odp/plat/msgio_types.h
>>> new file mode 100644
>>> index 0000000..9d9f9da
>>> --- /dev/null
>>> +++ b/platform/linux-generic/include/odp/plat/msgio_types.h
>>> @@ -0,0 +1,59 @@
>>> +/* Copyright (c) 2015, Linaro Limited
>>> + * All rights reserved.
>>> + *
>>> + * SPDX-License-Identifier: BSD-3-Clause
>>> + */
>>> +
>>> +
>>> +/**
>>> + * @file
>>> + *
>>> + * ODP Message bus types
>>> + */
>>> +
>>> +#ifndef ODP_MSGIO_TYPES_H_
>>> +#define ODP_MSGIO_TYPES_H_
>>> +
>>> +#ifdef __cplusplus
>>> +extern "C" {
>>> +#endif
>>> +
>>> +#include <odp/std_types.h>
>>> +#include <odp/plat/strong_types.h>
>>> +
>>> +/** @addtogroup odp_msgio ODP message bus
>>> + *  Operations on a message bus.
>>> + *  @{
>>> + */
>>> +
>>> +typedef int odp_msgio_prio_t;
>>> +
>>> +#define ODP_MSGIO_PRIO_HIGHEST  0
>>> +
>>> +#define ODP_MSGIO_PRIO_NORMAL   2
>>> +
>>> +#define ODP_MSGIO_PRIO_LOWEST   3
>>> +
>>> +#define ODP_MSGIO_PRIO_DEFAULT  ODP_MSGIO_PRIO_NORMAL
>>> +
>>> +typedef ODP_HANDLE_T(odp_msgio_t);
>>> +
>>> +#define ODP_MSGIO_INVALID _odp_cast_scalar(odp_msgio_t, 0xffffffff)
>>> +
>>> +#define ODP_MSGIO_ADDR_SIZE 6
>>> +
>>> +/** Get printable format of odp_msgio_t */
>>> +static inline uint64_t odp_msgio_to_u64(odp_msgio_t hdl)
>>> +{
>>> +       return _odp_pri(hdl);
>>> +}
>>> +
>>> +/**
>>> + * @}
>>> + */
>>> +
>>> +#ifdef __cplusplus
>>> +}
>>> +#endif
>>> +
>>> +#endif
>>>
>>>
>>
>> --
>> Benoît GANNE
>> Field Application Engineer, Kalray
>> +33 (0)648 125 843
>>
>
>

On 2 June 2015 at 16:46, Benoît Ganne <bganne@kalray.eu> wrote:

> Thinking about using notification events instead of the messages
>> themselves...
>> If I use D-BUS or ZeroMQ, I get a socket descriptor. I don't want to
>> block on this descriptor, my application is likely blocking in
>> odp_schedule() or polling some queues. I could allocate an event (of
>> whatever type) and associate that with the socket descriptor. When the
>> socket can read or written, the event will be enqueued on some queue
>> (also specified). My application will receive the notification event and
>> will perform all message operations on the non-blocking socket. When no
>> more messages to read or message write would block, the application
>> re-registers the event and queue.
>>
>> The only API we would need is:
>> void odp_notification_register(int sd, odp_event_t evt, odp_queue_t
>> queue);
>> Perhaps also a call to unregister the notification, to be used at
>> termination.
>>
>> This would be a nice and generic design (not only for this specific use
>> case). But it also seems limited to e.g. Linux and similar operating
>> systems. What about other operating environments (e.g. bare metal)?
>> Would those environments not implement this API? Can we have an ODP API
>> which only is implemented on some platforms/environments?
>>
>
> Why not using standard odp_queue_enq() then?
> I mean, you could create a thread which spin over recv() and enqueue an
> event with the received message to your queue. In that case your datapath
> threads would just have to deal with queues the usual way.
> And this pattern would work with the current API and for any notification
> mechanism, socket, ZeroMQ, POSIX signals.
> Any thoughts?
>
I want my worker threads to have full and exclusive control of the
dataplane CPU's, no time slicing here, no OS invocation at all. Any
additional threads would have to run on some additional CPU which shares
memory and queues with the dataplane CPU's. (I suddenly see a use for
Petri's thread type proposal). Are such CPU's always going to be available?

So I can see other (simpler) solutions but no guaranteed portability
between different ODP platforms. You people (specifically you users) think
this is a good trade-off?

-- Ola

>
> ben
>

On 3 June 2015 at 14:31, Benoît Ganne <bganne@kalray.eu> wrote:

> If I use D-BUS or ZeroMQ, I get a socket descriptor. I don't want to
>>>> block on this descriptor, my application is likely blocking in
>>>> odp_schedule() or polling some queues. I could allocate an event (of
>>>> whatever type) and associate that with the socket descriptor.
>>>> When the socket can read or written, the event will be enqueued
>>>> on some queue (also specified). My application will receive the
>>>> notification event and will perform all message operations on the
>>>> non-blocking socket.
>>>> When no more messages to read or message write would block, the
>>>> application re-registers the event and queue.
>>>> The only API we would need is:
>>>> void odp_notification_register(int sd, odp_event_t evt,
>>>> odp_queue_t queue);
>>>> Perhaps also a call to unregister the notification, to be used
>>>> at termination. This would be a nice and generic design (not only
>>>> for this specific use case). But it also seems limited to e.g.
>>>> Linux and similar operating systems. What about other operating
>>>> environments (e.g. bare metal)?
>>>> Would those environments not implement this API? Can we have an
>>>> ODP API which only is implemented on some platforms/environments?
>>>>
>>>
>  Why not using standard odp_queue_enq() then?
>>> I mean, you could create a thread which spin over recv() and enqueue
>>> an event with the received message to your queue. In that case your
>>> datapath threads would just have to deal with queues the usual way.
>>> And this pattern would work with the current API and for any
>>> notification mechanism, socket, ZeroMQ, POSIX signals.
>>> Any thoughts?
>>>
>>
>  I want my worker threads to have full and exclusive control of the
>> dataplane CPU's, no time slicing here, no OS invocation at all. Any
>> additional threads would have to run on some additional CPU which shares
>> memory and queues with the dataplane CPU's. (I suddenly see a use for
>> Petri's thread type proposal). Are such CPU's always going to be
>> available?
>> So I can see other (simpler) solutions but no guaranteed portability
>> between different ODP platforms. You people (specifically you users)
>> think this is a good trade-off?
>>
>
> I think Petri's proposal could take care of this, along with its new
> addition about cpumasks we could have a nice solution.
> Regarding portability, you won't necessarily have sockets in bare metal
> environment anyway.
> In our solution for example, we have those extra CPUs for control-plane,
> but you don't have sockets on bare metal (at least for now).



> The only "ODP portable" communication so far is pktio.
>
Which can always be used as the implementation for a MSGIO API.

But for now I will continue my prototyping using a non-worker thread in the
dataplane application that interacts with the environment (OS/kernel) and
performs any blocking system calls. Incoming messages can be passed as
events on queues to the worker threads. I will have to see what works for
outgoing messages (sent by the worker threads), possibly the non-worker
thread will have to act as an intermediary here as well.


>
> ben
>