| Message ID | 1447780514-19036-2-git-send-email-bill.fischofer@linaro.org |
|---|---|
| State | Superseded |
| Headers | show |
This is the rendered doc http://people.linaro.org/~mike.holmes/output/users-guide.html On 17 November 2015 at 12:15, Bill Fischofer <bill.fischofer@linaro.org> wrote: > Add a basic ODP overview to the User's Guide, providing > an overview of ODP concepts, components, etc. Included > are a number of diagrams covering component structure > as well as packet RX, event scheduling, and packet TX > processing. > > Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> > --- > doc/users-guide/users-guide.adoc | 400 > ++++++++++++++++++++++++++++++++++++++- > 1 file changed, 398 insertions(+), 2 deletions(-) > > diff --git a/doc/users-guide/users-guide.adoc > b/doc/users-guide/users-guide.adoc > index 7f70046..cf77fa0 100644 > --- a/doc/users-guide/users-guide.adoc > +++ b/doc/users-guide/users-guide.adoc > @@ -8,7 +8,7 @@ OpenDataPlane (ODP) Users-Guide > Abstract > -------- > This document is intended to guide a new ODP application developer. > -Further details about ODP may be found at then http://opendataplane.org[ODP] > home page. > +Further details about ODP may be found at the http://opendataplane.org[ODP] > home page. > > .Overview of a system running ODP applications > image::../images/overview.png[align="center"] > @@ -16,6 +16,403 @@ image::../images/overview.png[align="center"] > ODP is an API specification that allows many implementations to provide > platform independence, automatic hardware acceleration and CPU scaling to > high performance networking applications. > This document describes how to write an application that can successfully > take advantage of the API. > > +:numbered: > +== Introduction == > +.OpenDataPlane Components > +image::../images/odp_components.png[align="center"] > + > +.The ODP API Specification > +ODP consists of three separate but related component parts. First, ODP is > an > +abstract API specification that describes a functional model for > +data plane applications. This specification covers many common data plane > +application programming needs, such as the ability to receive, > manipulate, and > +transmit packet data, without specifying how these functions are > performed. This > +is quite intentional. It is precisely because ODP APIs do not have a > preferred > +embodiment that they permit innovation in how these functions can > +be realized on various platforms that offer implementations of ODP. To > achieve > +this goal, ODP APIs are described using abstract data types whose > definition > +is left up to the ODP implementer. For example, in ODP packets are > referenced > +by abstract handles of type +odp_packet_t+, and packet-related APIs take > +arguments of this type. What an +odp_packet_t+ actually is is not part of > the > +ODP API specification--that is the responsibility of each ODP > implementation. > + > +.Summary: ODP API attributes: > +* Open Source, open contribution, BSD-3 licensed. > +* Vendor and platform neutral. > +* Application-centric. Covers functional needs of data plane > applications. > +* Ensures portability by specifying the functional behavior of ODP. > +* Defined jointly and openly by application writers and platform > implementers. > +* Archiected to be implementable on a wide range of platforms efficiently > +* Sponsored, governed, and maintained by the Linaro Networking Group (LNG) > + > +.ODP Implementations > +Second, ODP consists of multiple implementations of this API > specification, > +each tailored to a specific target platform. ODP implementations determine > +how each ODP abstract type is represented on that platform and how each > ODP > +API is realized. On some platforms, ODP APIs will > +be realized using specialized instructions that accelerate the functional > +behavior specified by the API. On others, hardware co-processing engines > may > +completely offload the API so that again it can be performed with little > or no > +involvement by a CPU. In all cases, the application sees the same > +functional behavior independent of how a given platform has chosen to > realize > +it. By allowing each platform the freedom to determine how best to > realize each > +API's specified functional behavior in an optimal manner, ODP permits > +applications written to its APIs to take full advantage of the unique > +capabilities of each platform without the application programmer needing > to > +have specialist knowledge of that platform or to be concerned with how > best > +to tune the application to a particular platform. This latter > consideration is > +particularly important in Network Function Virtualization (NFV) > environments > +where the application will run on a target platform chosen by someone > else. > + > +.Summary: ODP Implementation Characteristics > +* One size does not fit all--supporting multiple implementations allows > ODP > +to adapt to widely differing internals among platforms. > +* Anyone can create an ODP implementation tailored to their platform > +* Distribution and mainteinance of each implementation is as owner wishes > + - Open source or closed source as business needs determine > + - Have independent release cycles and service streams > +* Allows HW and SW innovation in how ODP APIs are implemented on each > platform. > + > +.Reference Implementations > +To make it easy to get started with implementing ODP on a new platform, > ODP > +supplies a number of _reference implementations_ that can serve as a > +starting point. The two primary references implementations supplied by > ODP are > +*odp-linux* and *odp-dpdk* > + > +.odp-linux > +The *odp-linux* reference implementation is a pure SW implementation of > the > +ODP API that relies only on the Linux programming API. As a functional > model > +for ODP, it enables ODP to be bootstrapped easily to any platform that > +supports a Linux kernel. > + > +.odp-dpdk > +The *odp-dpdk* reference implementation is a pure SW implementation of the > +ODP API that uses http://dpdk.org[DPDK] as a SW accelerator. In > particular, > +*odp-dpdk* offers superior I/O performance for systems that use NICs, > allowing > +ODP applications to take immediate full advantage of the various NIC > device > +drivers supported by DPDK. > + > +.Summary: ODP Reference Implementations > +* Open source, open contribution, BSD-3 licensed. > +* Provide easy bootstrapping of ODP onto new platforms > +* Implementers free to borrow or tailor code as needed for their platform > +* Implementers retain full control over their implementations whether or > not > +they are derived from a reference implementation. > + > +.ODP Validation Test Suite > +Third, to enure consistency between different ODP implementations, ODP > +consists of a validation suite that verifies that any given > implementation of > +ODP faithfully provides the specified functional behavior of each ODP API. > +As a separate open source component, the validation suite may be used by > +application writers, system integrators, and platform providers alike to > +confirm that any purported implementation of ODP does indeed conform to > the > +ODP API specification. > + > +.Summary: ODP Validation Test Suite > +* Synchronized with ODP API specification > +* Maintained and distributed by LNG > +* Open source, open contribution, BSD-3 licensed. > +* Key to ensuring application portability across all ODP implementations > +* Tests that ODP implementations conform to the specified functional > behavior > +of ODP APIs. > +* Can be run at any time by users and vendors to validat implementations > +od ODP. > + > +=== ODP API Specification Versioning === > +As an evolving standard, the ODP API specification is released under an > +incrementing version number, and corresponding implementations of ODP, as > well > +as the validation suite that verifies API conformance, are linked to this > +version number. ODP versions are specified using a stanard three-level > +number (major.minor.fixlevel) that are incremented according to the > degree of > +change the level represents. Increments to the fixlevel represent > clarification > +of the specification or other minor changes that do not affect either the > +syntax or semantics of the specification. Such changes in the API > specification > +are expected to be rare. Increments to the minor level > +represent the introduction of new APIs or functional capabilities, or > changes > +to he specified syntax or functional behavior of APIs and thus may require > +application source code changes. Such changes are well documented in the > +release notes for each revision of the specification. Finally, increments > to > +the major level represent significant structural changes that most likely > +require some level of application source code change, again as documented > in > +the release notes for that version. > + > +=== ODP Implementation Versioning === > +ODP implementations are free to use whatever release naming/numbering > +conventions they wish, as long as it is clear what level of the ODP API a > given > +release implements. A recommended convention is to use the same three > level > +numbering scheme where the major and minor numbers correspond to the ODP > API > +level and the fixlevel represents an implementation-defined service level > +associated with that API level implementation. The LNG-supplied ODP > reference > +implementations follow this convention. > + > +=== ODP Validation Test Suite Versioning === > +The ODP validation test suite follows these same naming conventions. The > major > +and minor release numbers correspond to the ODP API level that the suite > +validates and the fixlevel represents the service level of the validation > +suite itself for that API level. > + > +=== ODP Design Goals === > +ODP has three primary goals that follow from its component structure. The > first > +is application portability across a wide range of platforms. These > platforms > +differ in terms of processor instruction set architecture, number and > types of > +application processing cores, memory oranization, as well as the number > and > +type of platform specific hardware acceleration and offload features that > +are available. ODP applications can move from one conforming > implementation > +to another with at most a recompile. > + > +Second, ODP is designed to permit data plane applications to avail > themselves > +of platform-specific features, including specialized hardware > accelerators, > +without specialized programming. This is achieved by separating the API > +specification from their implementation on individual platforms. Since > each > +platform implements each ODP API in a manner optimal to that platform, > +applications automatically gain the benefit of such optimizations without > the > +need for explicit programming. > + > +Third, ODP is designed to allow applications to scale out automatically to > +support many core architectures. This is done using an event based > programming > +model that permits applications to be written to be independent of the > number > +of processing cores that are available to realize application function. > The > +result is that an application written to this model does not require > redesign > +as it scales from 4, to 40, to 400 cores. > + > +== Organization of this Document == > +This document is organized into several sections. The first presents a > high > +level overview of the ODP API component areas and their associated > abstract > +data types. This section introduces ODP APIs at a conceptual level. > +The second provides a tutorial on the programming model(s) > +supported by ODP, paying particular attention to the event model as this > +represents the preferred structure for most ODP applications. This section > +builds on the concepts introduced in the first section and shows how ODP > +applications are structured to best realize the three ODP design goals > +mentioned earlier. The third section provides a more detailed overview of > +the major ODP API components and is designed to serve as a companion to > the > +full reference specification for each API. The latter is intended to be > used > +by ODP application programmers, as well as implementers, to understand the > +precise syntax and semantics of each API. > + > +== ODP API Concepts == > +ODP programs are built around several conceptual structures that every > +appliation programmer needs to be familiar with to use ODP effectively. > The > +main ODP concepts are: > +Thread, Event, Queue, Pool, Shared Memory, Buffer, Packet, PktIO, Timer, > +and Synchronizer. > + > +=== Thread === > +The thread is the fundamental programming unit in ODP. ODP applications > are > +organized into a collection of threads that perform the work that the > +application is designed to do. ODP threads may or may not share memory > with > +other threads--that is up to the implementation. Threads come in two > "flavors": > +control and worker, that are represented by the abstract type > ++odp_thread_type_t+. > + > +A control thread is a supervisory thread that organizes > +the operation of worker threads. Worker threads, by contrast, exist to > +perform the main processing logic of the application and employ a run to > +completion model. Worker threads, in particular, are intended to operate > on > +dedicated processing cores, especially in many core proessing > environments, > +however a given implementation may multitask multiple threads on a single > +core if desired (typically on smaller and lower performance target > +environments). > + > +In addition to thread types, threads have associated _attributes_ such as > +_thread mask_ and _scheduler group_ that determine where they can run and > +the type of work that they can handle. These will be discussed in greater > +detail later. > + > +=== Event === > +Events are what threads process to perform their work. Events can > represent > +new work, such as the arrival of a packet that needs to be processed, or > they > +can represent the completion of requests that have executed > asynchronously. > +Events can also represent notifications of the passage of time, or of > status > +changes in various components of interest to the application. Events have > an > +event type that describes what it represents. Threads can create new > events > +or consume events processed by them, or they can perform some processing > on > +an event and then pass it along to another component for further > processing. > +References to events are via handles of abstract type +odp_event_t+. Cast > +functions are provided to convert these into specific handles of the > +appropriate type represented by the event. > + > +=== Queue === > +A queue is a message passing channel that holds events. Events can be > +added to a queue via enqueue operations or removed from a queue via > dequeue > +operations. The endpoints of a queue will vary depending on how it is > used. > +Queues come in two major types: polled and scheduled, which will be > +discussed in more detail when the event model is introduced. Queues may > also > +have an associated context, which represents a persistent state for all > +events that make use of it. These states are what permit threads to > perform > +stateful processing on events as well as stateless processing. > + > +Queues are represented by handles of abstract type +odp_queue_t+. > + > +=== Pool === > +A pool is a shared memory area from which elements may be drawn. Pools > +represent the backing store for events, among other things. Pools are > +typically created and destroyed by the application during initialization > and > +termination, respectively, and then used during processing. Pools may be > +used by ODP components exclusively, by applications exclusively, or their > +use may be shared between the two. Pools have an associated type that > +characterizes the elements that they contain. The two most important pool > types > +are Buffer and Packet. > + > +Pools are represented by handles of abstract type +odp_pool_t+. > + > +=== Shared Memory === > +Shared memory represents raw blocks of storage that are sharable between > +threads. They are the building blocks of pools but can be used directly by > +ODP applications if desired. > + > +Shared memory is represented by handles of abstract type +odp_shm_t+. > + > +=== Buffer === > +A buffer is a fixed sized block of shared storage that is used by ODP > +components and/or applications to realize their function. Buffers contain > +zero or more bytes of application data as well as system maintained > +metadata that provide information about the buffer, such as its size or > the > +pool it was allocated from. Metadata is an important ODP concept because > it > +allows for arbitrary amounts of side information to be associated with an > +ODP object. Most ODP objects have assocaited metadata and this metadata is > +manipulated via accessor functions that act as getters and setters for > +this information. Getter acces functions permit an application to read > +a metadata item, while setter access functions permit an application to > write > +a metadata item. Note that some metadata is inherently read only and thus > +no setter is provided to manipulate it. When object have multiple > metadata > +items, each has its own associated getter and/or setter access function to > +inspect or manipulate it. > + > +Buffers are represened by handles of abstract type +odp_buffer_t+. > + > +=== Packet === > +Packets are received and transmitted via I/O interfaces and represent > +the basic data that data plane applications manipulate. > +Packets are drawn from pools of type +ODP_POOL_PACKET+. > +Unlike buffers, which are simple objects, > +ODP packets have a rich set of semantics that permit their inspection > +and manipulation in complex ways to be described later. Packets also > support > +a rich set of metadata as well as user metadata. User metadata permits > +applications to associate an application-determined amount of side > information > +with each packet for its own use. > + > +Packets are represented by handles of abstract type +odp_packet_t+. > + > +=== PktIO === > +PktIO is how ODP represents I/O interfaces. A pktio object is a logical > +port capable of receiving and/or transmitting packets. This may be > directly > +supported by the underlying platform as an integrated feature, > +or may represent a device attached via a PCIE or other bus. > + > +PktIOs are represented by handles of abstract type +odp_pktio_t+. > + > +=== Timer === > +Timers are how ODP applications measure and respond to the passage of > time. > +Timers are drawn from specialized pools called timer pools that have their > +own abstract type (+odp_timer_pool_t+). Applications may have many timers > +active at the same time and can set them to use either relative or > absolute > +time. When timers expire they create events of type +odp_timeout_t+, which > +serve as notifications of timer expiration. > + > +=== Synchronizer === > +Multiple threads operating in parallel typically require various > +synchronization services to permit them to operate in a reliable and > +coordinated manner. ODP provides a rich set of locks, barriers, and > similar > +synchronization primitives, as well as abstract types for representing > various > +types of atomic variables. The ODP event model also makes use of queues to > +avoid the need for explicit locking in many cases. This will be discussed > +in the next section. > + > +== ODP Components == > +Building on ODP concepts, ODP offers several components that relate to the > +flow of work through an ODP application. These include the Classifier, > +Scheduler, and Traffic Manager. These components relate to the three > +main stages of packet processing: Receive, Process, and Transmit. > + > +=== Classifier === > +The *Classifier* provides a suite of APIs that control packet receive (RX) > +processing. > + > +.ODP Receive Processing with Classifier > +image::../images/odp_rx_processing.png[align="center"] > + > +The classifier provides two logically related services: > +[horizontal] > +Packet parsing:: Verifying and extracting structural information from a > +received packet. > + > +Packet classification:: Applying *Pattern Matching Rules (PMRs)* to the > +parsed results to assign an incoming packet to a *Class of Service (CoS)*. > + > +Combined, these permit incoming packets to be sorted into *flows*, which > are > +logically related sequences of packets that share common processing > +requirements. While many data plane applications perform stateless packet > +processing (_e.g.,_ for simple forwarding) others perform stateful packet > +processing. Flows anchor state information relating to these groups of > +packets. > + > +A CoS determines two variables for packets belonging to a flow: > +[list] > +* The pool that they will be stored in on receipt > +* The queue that they will be added to for processing > + > +The PMRs supported by ODP permit flow determination based on combinations > of > +packet field values (tuples). The main advantage of classification is > that on > +many platforms these functions are performed in hardware, meaning that > +classification occurs at line rate as packets are being received without > +any explicit processing by the ODP application. > + > +Note that the use of the classifier is optional. Applications may > directly > +receive packets from a corresponding PktIO input queue via direct polling > +if they choose. > + > +=== Scheduler === > +The *Scheduler* provides a suite of APIs that control scalabable event > +processing. > + > +.ODP Scheduler and Event Processing > +image::../images/odp_scheduling.png[align="center"] > + > +The Scheduler is responsible for selecting and dispatching one or more > events > +to a requesting thread. Event selection is based on several factors > involving > +both the queues containing schedulable events and the thread making an > ++odp_schedule()+ or +odp_schedule_multi()+ call. > + > +ODP queues have a _scheduling priority_ that determines how urgently > events > +on them should be processed relative to events contained in other queues. > +Queues also have a _scheduler group id_ associated with them that must > match > +the associated scheduler group _thread mask_ of the thread calling the > +scheduler. This permits events to be grouped for processing into classes > and > +have threads that are dedicated to processing events from specified > classes. > +Threads can join and leave scheduler groups dynamically, permitting easy > +application response to increases in demand. > + > +When a thread receives an event from the scheduler, it in turn can invoke > +other processing engines via ODP APIs (_e.g.,_ crypto processing) that > +can operate asynchronously. When such processing is complete, the result > is > +that a *completion event* is added to a schedulable queue where it can be > +scheduled back to a thread to continue processing with the results of the > +requested asynchronous operation. > + > +Threads themselves can enqueue events to queues for downstream processing > +by other threads, permitting flexibility in how applicaitions structure > +themselves to maximize concurrency. > + > +=== Traffic Manager === > +The *Traffic Manager* provides a suite of APIs that control traffic > shaping and > +Quality of Service (QoS) processing for packet output. > + > +.ODP Transmit processing with Traffic Manager > +image::../images/odp_traffic_manager.png[align="center"] > + > +The final stage of packet processing is to transmit it. Here, > applications have > +several choices. As with RX processing, applications may send packets > +directly to PktIO TX queues for direct transmission. Often, however, > +applications need to perform traffic shaping and related > +*Quality of Service (QoS)* processing on the packets comprising a flow as > part > +of transmit processing. To handle this need, ODP provides a suite of > +*Traffic Manager* APIs that permit programmatic establishment of arbiters, > +shapers, etc. that control output packet processing to achieve desired QoS > +goals. Again, the advantage here is that on many platforms traffic > management > +functions are implemented in hardware, permitting transparent offload of > +this work. > + > Glossary > -------- > [glossary] > @@ -28,7 +425,6 @@ odp_thread:: > event:: > An event is a notification that can be placed in a queue. > > -:numbered: > The include structure > --------------------- > Applications only include the 'include/odp.h file which includes the > 'platform/<implementation name>/include/plat' files to provide a complete > definition of the API on that platform. > -- > 2.1.4 > > _______________________________________________ > lng-odp mailing list > lng-odp@lists.linaro.org > https://lists.linaro.org/mailman/listinfo/lng-odp > -- Mike Holmes Technical Manager - Linaro Networking Group Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM SoCs
Is that a review comment? Is there a problem with the rendering? On Tue, Nov 17, 2015 at 11:51 AM, Mike Holmes <mike.holmes@linaro.org> wrote: > This is the rendered doc > > http://people.linaro.org/~mike.holmes/output/users-guide.html > > On 17 November 2015 at 12:15, Bill Fischofer <bill.fischofer@linaro.org> > wrote: > >> Add a basic ODP overview to the User's Guide, providing >> an overview of ODP concepts, components, etc. Included >> are a number of diagrams covering component structure >> as well as packet RX, event scheduling, and packet TX >> processing. >> >> Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> >> --- >> doc/users-guide/users-guide.adoc | 400 >> ++++++++++++++++++++++++++++++++++++++- >> 1 file changed, 398 insertions(+), 2 deletions(-) >> >> diff --git a/doc/users-guide/users-guide.adoc >> b/doc/users-guide/users-guide.adoc >> index 7f70046..cf77fa0 100644 >> --- a/doc/users-guide/users-guide.adoc >> +++ b/doc/users-guide/users-guide.adoc >> @@ -8,7 +8,7 @@ OpenDataPlane (ODP) Users-Guide >> Abstract >> -------- >> This document is intended to guide a new ODP application developer. >> -Further details about ODP may be found at then http://opendataplane.org[ODP] >> home page. >> +Further details about ODP may be found at the http://opendataplane.org[ODP] >> home page. >> >> .Overview of a system running ODP applications >> image::../images/overview.png[align="center"] >> @@ -16,6 +16,403 @@ image::../images/overview.png[align="center"] >> ODP is an API specification that allows many implementations to provide >> platform independence, automatic hardware acceleration and CPU scaling to >> high performance networking applications. >> This document describes how to write an application that can >> successfully take advantage of the API. >> >> +:numbered: >> +== Introduction == >> +.OpenDataPlane Components >> +image::../images/odp_components.png[align="center"] >> + >> +.The ODP API Specification >> +ODP consists of three separate but related component parts. First, ODP >> is an >> +abstract API specification that describes a functional model for >> +data plane applications. This specification covers many common data plane >> +application programming needs, such as the ability to receive, >> manipulate, and >> +transmit packet data, without specifying how these functions are >> performed. This >> +is quite intentional. It is precisely because ODP APIs do not have a >> preferred >> +embodiment that they permit innovation in how these functions can >> +be realized on various platforms that offer implementations of ODP. To >> achieve >> +this goal, ODP APIs are described using abstract data types whose >> definition >> +is left up to the ODP implementer. For example, in ODP packets are >> referenced >> +by abstract handles of type +odp_packet_t+, and packet-related APIs take >> +arguments of this type. What an +odp_packet_t+ actually is is not part >> of the >> +ODP API specification--that is the responsibility of each ODP >> implementation. >> + >> +.Summary: ODP API attributes: >> +* Open Source, open contribution, BSD-3 licensed. >> +* Vendor and platform neutral. >> +* Application-centric. Covers functional needs of data plane >> applications. >> +* Ensures portability by specifying the functional behavior of ODP. >> +* Defined jointly and openly by application writers and platform >> implementers. >> +* Archiected to be implementable on a wide range of platforms efficiently >> +* Sponsored, governed, and maintained by the Linaro Networking Group >> (LNG) >> + >> +.ODP Implementations >> +Second, ODP consists of multiple implementations of this API >> specification, >> +each tailored to a specific target platform. ODP implementations >> determine >> +how each ODP abstract type is represented on that platform and how each >> ODP >> +API is realized. On some platforms, ODP APIs will >> +be realized using specialized instructions that accelerate the functional >> +behavior specified by the API. On others, hardware co-processing engines >> may >> +completely offload the API so that again it can be performed with little >> or no >> +involvement by a CPU. In all cases, the application sees the same >> +functional behavior independent of how a given platform has chosen to >> realize >> +it. By allowing each platform the freedom to determine how best to >> realize each >> +API's specified functional behavior in an optimal manner, ODP permits >> +applications written to its APIs to take full advantage of the unique >> +capabilities of each platform without the application programmer needing >> to >> +have specialist knowledge of that platform or to be concerned with how >> best >> +to tune the application to a particular platform. This latter >> consideration is >> +particularly important in Network Function Virtualization (NFV) >> environments >> +where the application will run on a target platform chosen by someone >> else. >> + >> +.Summary: ODP Implementation Characteristics >> +* One size does not fit all--supporting multiple implementations allows >> ODP >> +to adapt to widely differing internals among platforms. >> +* Anyone can create an ODP implementation tailored to their platform >> +* Distribution and mainteinance of each implementation is as owner wishes >> + - Open source or closed source as business needs determine >> + - Have independent release cycles and service streams >> +* Allows HW and SW innovation in how ODP APIs are implemented on each >> platform. >> + >> +.Reference Implementations >> +To make it easy to get started with implementing ODP on a new platform, >> ODP >> +supplies a number of _reference implementations_ that can serve as a >> +starting point. The two primary references implementations supplied by >> ODP are >> +*odp-linux* and *odp-dpdk* >> + >> +.odp-linux >> +The *odp-linux* reference implementation is a pure SW implementation of >> the >> +ODP API that relies only on the Linux programming API. As a functional >> model >> +for ODP, it enables ODP to be bootstrapped easily to any platform that >> +supports a Linux kernel. >> + >> +.odp-dpdk >> +The *odp-dpdk* reference implementation is a pure SW implementation of >> the >> +ODP API that uses http://dpdk.org[DPDK] as a SW accelerator. In >> particular, >> +*odp-dpdk* offers superior I/O performance for systems that use NICs, >> allowing >> +ODP applications to take immediate full advantage of the various NIC >> device >> +drivers supported by DPDK. >> + >> +.Summary: ODP Reference Implementations >> +* Open source, open contribution, BSD-3 licensed. >> +* Provide easy bootstrapping of ODP onto new platforms >> +* Implementers free to borrow or tailor code as needed for their platform >> +* Implementers retain full control over their implementations whether or >> not >> +they are derived from a reference implementation. >> + >> +.ODP Validation Test Suite >> +Third, to enure consistency between different ODP implementations, ODP >> +consists of a validation suite that verifies that any given >> implementation of >> +ODP faithfully provides the specified functional behavior of each ODP >> API. >> +As a separate open source component, the validation suite may be used by >> +application writers, system integrators, and platform providers alike to >> +confirm that any purported implementation of ODP does indeed conform to >> the >> +ODP API specification. >> + >> +.Summary: ODP Validation Test Suite >> +* Synchronized with ODP API specification >> +* Maintained and distributed by LNG >> +* Open source, open contribution, BSD-3 licensed. >> +* Key to ensuring application portability across all ODP implementations >> +* Tests that ODP implementations conform to the specified functional >> behavior >> +of ODP APIs. >> +* Can be run at any time by users and vendors to validat implementations >> +od ODP. >> + >> +=== ODP API Specification Versioning === >> +As an evolving standard, the ODP API specification is released under an >> +incrementing version number, and corresponding implementations of ODP, >> as well >> +as the validation suite that verifies API conformance, are linked to this >> +version number. ODP versions are specified using a stanard three-level >> +number (major.minor.fixlevel) that are incremented according to the >> degree of >> +change the level represents. Increments to the fixlevel represent >> clarification >> +of the specification or other minor changes that do not affect either the >> +syntax or semantics of the specification. Such changes in the API >> specification >> +are expected to be rare. Increments to the minor level >> +represent the introduction of new APIs or functional capabilities, or >> changes >> +to he specified syntax or functional behavior of APIs and thus may >> require >> +application source code changes. Such changes are well documented in the >> +release notes for each revision of the specification. Finally, >> increments to >> +the major level represent significant structural changes that most likely >> +require some level of application source code change, again as >> documented in >> +the release notes for that version. >> + >> +=== ODP Implementation Versioning === >> +ODP implementations are free to use whatever release naming/numbering >> +conventions they wish, as long as it is clear what level of the ODP API >> a given >> +release implements. A recommended convention is to use the same three >> level >> +numbering scheme where the major and minor numbers correspond to the ODP >> API >> +level and the fixlevel represents an implementation-defined service level >> +associated with that API level implementation. The LNG-supplied ODP >> reference >> +implementations follow this convention. >> + >> +=== ODP Validation Test Suite Versioning === >> +The ODP validation test suite follows these same naming conventions. The >> major >> +and minor release numbers correspond to the ODP API level that the suite >> +validates and the fixlevel represents the service level of the validation >> +suite itself for that API level. >> + >> +=== ODP Design Goals === >> +ODP has three primary goals that follow from its component structure. >> The first >> +is application portability across a wide range of platforms. These >> platforms >> +differ in terms of processor instruction set architecture, number and >> types of >> +application processing cores, memory oranization, as well as the number >> and >> +type of platform specific hardware acceleration and offload features that >> +are available. ODP applications can move from one conforming >> implementation >> +to another with at most a recompile. >> + >> +Second, ODP is designed to permit data plane applications to avail >> themselves >> +of platform-specific features, including specialized hardware >> accelerators, >> +without specialized programming. This is achieved by separating the API >> +specification from their implementation on individual platforms. Since >> each >> +platform implements each ODP API in a manner optimal to that platform, >> +applications automatically gain the benefit of such optimizations >> without the >> +need for explicit programming. >> + >> +Third, ODP is designed to allow applications to scale out automatically >> to >> +support many core architectures. This is done using an event based >> programming >> +model that permits applications to be written to be independent of the >> number >> +of processing cores that are available to realize application function. >> The >> +result is that an application written to this model does not require >> redesign >> +as it scales from 4, to 40, to 400 cores. >> + >> +== Organization of this Document == >> +This document is organized into several sections. The first presents a >> high >> +level overview of the ODP API component areas and their associated >> abstract >> +data types. This section introduces ODP APIs at a conceptual level. >> +The second provides a tutorial on the programming model(s) >> +supported by ODP, paying particular attention to the event model as this >> +represents the preferred structure for most ODP applications. This >> section >> +builds on the concepts introduced in the first section and shows how ODP >> +applications are structured to best realize the three ODP design goals >> +mentioned earlier. The third section provides a more detailed overview of >> +the major ODP API components and is designed to serve as a companion to >> the >> +full reference specification for each API. The latter is intended to be >> used >> +by ODP application programmers, as well as implementers, to understand >> the >> +precise syntax and semantics of each API. >> + >> +== ODP API Concepts == >> +ODP programs are built around several conceptual structures that every >> +appliation programmer needs to be familiar with to use ODP effectively. >> The >> +main ODP concepts are: >> +Thread, Event, Queue, Pool, Shared Memory, Buffer, Packet, PktIO, Timer, >> +and Synchronizer. >> + >> +=== Thread === >> +The thread is the fundamental programming unit in ODP. ODP applications >> are >> +organized into a collection of threads that perform the work that the >> +application is designed to do. ODP threads may or may not share memory >> with >> +other threads--that is up to the implementation. Threads come in two >> "flavors": >> +control and worker, that are represented by the abstract type >> ++odp_thread_type_t+. >> + >> +A control thread is a supervisory thread that organizes >> +the operation of worker threads. Worker threads, by contrast, exist to >> +perform the main processing logic of the application and employ a run to >> +completion model. Worker threads, in particular, are intended to operate >> on >> +dedicated processing cores, especially in many core proessing >> environments, >> +however a given implementation may multitask multiple threads on a single >> +core if desired (typically on smaller and lower performance target >> +environments). >> + >> +In addition to thread types, threads have associated _attributes_ such as >> +_thread mask_ and _scheduler group_ that determine where they can run and >> +the type of work that they can handle. These will be discussed in greater >> +detail later. >> + >> +=== Event === >> +Events are what threads process to perform their work. Events can >> represent >> +new work, such as the arrival of a packet that needs to be processed, or >> they >> +can represent the completion of requests that have executed >> asynchronously. >> +Events can also represent notifications of the passage of time, or of >> status >> +changes in various components of interest to the application. Events >> have an >> +event type that describes what it represents. Threads can create new >> events >> +or consume events processed by them, or they can perform some processing >> on >> +an event and then pass it along to another component for further >> processing. >> +References to events are via handles of abstract type +odp_event_t+. Cast >> +functions are provided to convert these into specific handles of the >> +appropriate type represented by the event. >> + >> +=== Queue === >> +A queue is a message passing channel that holds events. Events can be >> +added to a queue via enqueue operations or removed from a queue via >> dequeue >> +operations. The endpoints of a queue will vary depending on how it is >> used. >> +Queues come in two major types: polled and scheduled, which will be >> +discussed in more detail when the event model is introduced. Queues may >> also >> +have an associated context, which represents a persistent state for all >> +events that make use of it. These states are what permit threads to >> perform >> +stateful processing on events as well as stateless processing. >> + >> +Queues are represented by handles of abstract type +odp_queue_t+. >> + >> +=== Pool === >> +A pool is a shared memory area from which elements may be drawn. Pools >> +represent the backing store for events, among other things. Pools are >> +typically created and destroyed by the application during initialization >> and >> +termination, respectively, and then used during processing. Pools may be >> +used by ODP components exclusively, by applications exclusively, or their >> +use may be shared between the two. Pools have an associated type that >> +characterizes the elements that they contain. The two most important >> pool types >> +are Buffer and Packet. >> + >> +Pools are represented by handles of abstract type +odp_pool_t+. >> + >> +=== Shared Memory === >> +Shared memory represents raw blocks of storage that are sharable between >> +threads. They are the building blocks of pools but can be used directly >> by >> +ODP applications if desired. >> + >> +Shared memory is represented by handles of abstract type +odp_shm_t+. >> + >> +=== Buffer === >> +A buffer is a fixed sized block of shared storage that is used by ODP >> +components and/or applications to realize their function. Buffers contain >> +zero or more bytes of application data as well as system maintained >> +metadata that provide information about the buffer, such as its size or >> the >> +pool it was allocated from. Metadata is an important ODP concept because >> it >> +allows for arbitrary amounts of side information to be associated with an >> +ODP object. Most ODP objects have assocaited metadata and this metadata >> is >> +manipulated via accessor functions that act as getters and setters for >> +this information. Getter acces functions permit an application to read >> +a metadata item, while setter access functions permit an application to >> write >> +a metadata item. Note that some metadata is inherently read only and thus >> +no setter is provided to manipulate it. When object have multiple >> metadata >> +items, each has its own associated getter and/or setter access function >> to >> +inspect or manipulate it. >> + >> +Buffers are represened by handles of abstract type +odp_buffer_t+. >> + >> +=== Packet === >> +Packets are received and transmitted via I/O interfaces and represent >> +the basic data that data plane applications manipulate. >> +Packets are drawn from pools of type +ODP_POOL_PACKET+. >> +Unlike buffers, which are simple objects, >> +ODP packets have a rich set of semantics that permit their inspection >> +and manipulation in complex ways to be described later. Packets also >> support >> +a rich set of metadata as well as user metadata. User metadata permits >> +applications to associate an application-determined amount of side >> information >> +with each packet for its own use. >> + >> +Packets are represented by handles of abstract type +odp_packet_t+. >> + >> +=== PktIO === >> +PktIO is how ODP represents I/O interfaces. A pktio object is a logical >> +port capable of receiving and/or transmitting packets. This may be >> directly >> +supported by the underlying platform as an integrated feature, >> +or may represent a device attached via a PCIE or other bus. >> + >> +PktIOs are represented by handles of abstract type +odp_pktio_t+. >> + >> +=== Timer === >> +Timers are how ODP applications measure and respond to the passage of >> time. >> +Timers are drawn from specialized pools called timer pools that have >> their >> +own abstract type (+odp_timer_pool_t+). Applications may have many timers >> +active at the same time and can set them to use either relative or >> absolute >> +time. When timers expire they create events of type +odp_timeout_t+, >> which >> +serve as notifications of timer expiration. >> + >> +=== Synchronizer === >> +Multiple threads operating in parallel typically require various >> +synchronization services to permit them to operate in a reliable and >> +coordinated manner. ODP provides a rich set of locks, barriers, and >> similar >> +synchronization primitives, as well as abstract types for representing >> various >> +types of atomic variables. The ODP event model also makes use of queues >> to >> +avoid the need for explicit locking in many cases. This will be discussed >> +in the next section. >> + >> +== ODP Components == >> +Building on ODP concepts, ODP offers several components that relate to >> the >> +flow of work through an ODP application. These include the Classifier, >> +Scheduler, and Traffic Manager. These components relate to the three >> +main stages of packet processing: Receive, Process, and Transmit. >> + >> +=== Classifier === >> +The *Classifier* provides a suite of APIs that control packet receive >> (RX) >> +processing. >> + >> +.ODP Receive Processing with Classifier >> +image::../images/odp_rx_processing.png[align="center"] >> + >> +The classifier provides two logically related services: >> +[horizontal] >> +Packet parsing:: Verifying and extracting structural information from a >> +received packet. >> + >> +Packet classification:: Applying *Pattern Matching Rules (PMRs)* to the >> +parsed results to assign an incoming packet to a *Class of Service >> (CoS)*. >> + >> +Combined, these permit incoming packets to be sorted into *flows*, which >> are >> +logically related sequences of packets that share common processing >> +requirements. While many data plane applications perform stateless packet >> +processing (_e.g.,_ for simple forwarding) others perform stateful packet >> +processing. Flows anchor state information relating to these groups of >> +packets. >> + >> +A CoS determines two variables for packets belonging to a flow: >> +[list] >> +* The pool that they will be stored in on receipt >> +* The queue that they will be added to for processing >> + >> +The PMRs supported by ODP permit flow determination based on >> combinations of >> +packet field values (tuples). The main advantage of classification is >> that on >> +many platforms these functions are performed in hardware, meaning that >> +classification occurs at line rate as packets are being received without >> +any explicit processing by the ODP application. >> + >> +Note that the use of the classifier is optional. Applications may >> directly >> +receive packets from a corresponding PktIO input queue via direct polling >> +if they choose. >> + >> +=== Scheduler === >> +The *Scheduler* provides a suite of APIs that control scalabable event >> +processing. >> + >> +.ODP Scheduler and Event Processing >> +image::../images/odp_scheduling.png[align="center"] >> + >> +The Scheduler is responsible for selecting and dispatching one or more >> events >> +to a requesting thread. Event selection is based on several factors >> involving >> +both the queues containing schedulable events and the thread making an >> ++odp_schedule()+ or +odp_schedule_multi()+ call. >> + >> +ODP queues have a _scheduling priority_ that determines how urgently >> events >> +on them should be processed relative to events contained in other queues. >> +Queues also have a _scheduler group id_ associated with them that must >> match >> +the associated scheduler group _thread mask_ of the thread calling the >> +scheduler. This permits events to be grouped for processing into classes >> and >> +have threads that are dedicated to processing events from specified >> classes. >> +Threads can join and leave scheduler groups dynamically, permitting easy >> +application response to increases in demand. >> + >> +When a thread receives an event from the scheduler, it in turn can invoke >> +other processing engines via ODP APIs (_e.g.,_ crypto processing) that >> +can operate asynchronously. When such processing is complete, the result >> is >> +that a *completion event* is added to a schedulable queue where it can be >> +scheduled back to a thread to continue processing with the results of the >> +requested asynchronous operation. >> + >> +Threads themselves can enqueue events to queues for downstream processing >> +by other threads, permitting flexibility in how applicaitions structure >> +themselves to maximize concurrency. >> + >> +=== Traffic Manager === >> +The *Traffic Manager* provides a suite of APIs that control traffic >> shaping and >> +Quality of Service (QoS) processing for packet output. >> + >> +.ODP Transmit processing with Traffic Manager >> +image::../images/odp_traffic_manager.png[align="center"] >> + >> +The final stage of packet processing is to transmit it. Here, >> applications have >> +several choices. As with RX processing, applications may send packets >> +directly to PktIO TX queues for direct transmission. Often, however, >> +applications need to perform traffic shaping and related >> +*Quality of Service (QoS)* processing on the packets comprising a flow >> as part >> +of transmit processing. To handle this need, ODP provides a suite of >> +*Traffic Manager* APIs that permit programmatic establishment of >> arbiters, >> +shapers, etc. that control output packet processing to achieve desired >> QoS >> +goals. Again, the advantage here is that on many platforms traffic >> management >> +functions are implemented in hardware, permitting transparent offload of >> +this work. >> + >> Glossary >> -------- >> [glossary] >> @@ -28,7 +425,6 @@ odp_thread:: >> event:: >> An event is a notification that can be placed in a queue. >> >> -:numbered: >> The include structure >> --------------------- >> Applications only include the 'include/odp.h file which includes the >> 'platform/<implementation name>/include/plat' files to provide a complete >> definition of the API on that platform. >> -- >> 2.1.4 >> >> _______________________________________________ >> lng-odp mailing list >> lng-odp@lists.linaro.org >> https://lists.linaro.org/mailman/listinfo/lng-odp >> > > > > -- > Mike Holmes > Technical Manager - Linaro Networking Group > Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM SoCs > > >
On 17 November 2015 at 12:54, Bill Fischofer <bill.fischofer@linaro.org> wrote: > Is that a review comment? Is there a problem with the rendering? > Not a review comment just to make it easier for folks to read initially. > On Tue, Nov 17, 2015 at 11:51 AM, Mike Holmes <mike.holmes@linaro.org> > wrote: > >> This is the rendered doc >> >> http://people.linaro.org/~mike.holmes/output/users-guide.html >> >> On 17 November 2015 at 12:15, Bill Fischofer <bill.fischofer@linaro.org> >> wrote: >> >>> Add a basic ODP overview to the User's Guide, providing >>> an overview of ODP concepts, components, etc. Included >>> are a number of diagrams covering component structure >>> as well as packet RX, event scheduling, and packet TX >>> processing. >>> >>> Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> >>> --- >>> doc/users-guide/users-guide.adoc | 400 >>> ++++++++++++++++++++++++++++++++++++++- >>> 1 file changed, 398 insertions(+), 2 deletions(-) >>> >>> diff --git a/doc/users-guide/users-guide.adoc >>> b/doc/users-guide/users-guide.adoc >>> index 7f70046..cf77fa0 100644 >>> --- a/doc/users-guide/users-guide.adoc >>> +++ b/doc/users-guide/users-guide.adoc >>> @@ -8,7 +8,7 @@ OpenDataPlane (ODP) Users-Guide >>> Abstract >>> -------- >>> This document is intended to guide a new ODP application developer. >>> -Further details about ODP may be found at then http://opendataplane.org[ODP] >>> home page. >>> +Further details about ODP may be found at the http://opendataplane.org[ODP] >>> home page. >>> >>> .Overview of a system running ODP applications >>> image::../images/overview.png[align="center"] >>> @@ -16,6 +16,403 @@ image::../images/overview.png[align="center"] >>> ODP is an API specification that allows many implementations to provide >>> platform independence, automatic hardware acceleration and CPU scaling to >>> high performance networking applications. >>> This document describes how to write an application that can >>> successfully take advantage of the API. >>> >>> +:numbered: >>> +== Introduction == >>> +.OpenDataPlane Components >>> +image::../images/odp_components.png[align="center"] >>> + >>> +.The ODP API Specification >>> +ODP consists of three separate but related component parts. First, ODP >>> is an >>> +abstract API specification that describes a functional model for >>> +data plane applications. This specification covers many common data >>> plane >>> +application programming needs, such as the ability to receive, >>> manipulate, and >>> +transmit packet data, without specifying how these functions are >>> performed. This >>> +is quite intentional. It is precisely because ODP APIs do not have a >>> preferred >>> +embodiment that they permit innovation in how these functions can >>> +be realized on various platforms that offer implementations of ODP. To >>> achieve >>> +this goal, ODP APIs are described using abstract data types whose >>> definition >>> +is left up to the ODP implementer. For example, in ODP packets are >>> referenced >>> +by abstract handles of type +odp_packet_t+, and packet-related APIs take >>> +arguments of this type. What an +odp_packet_t+ actually is is not part >>> of the >>> +ODP API specification--that is the responsibility of each ODP >>> implementation. >>> + >>> +.Summary: ODP API attributes: >>> +* Open Source, open contribution, BSD-3 licensed. >>> +* Vendor and platform neutral. >>> +* Application-centric. Covers functional needs of data plane >>> applications. >>> +* Ensures portability by specifying the functional behavior of ODP. >>> +* Defined jointly and openly by application writers and platform >>> implementers. >>> +* Archiected to be implementable on a wide range of platforms >>> efficiently >>> +* Sponsored, governed, and maintained by the Linaro Networking Group >>> (LNG) >>> + >>> +.ODP Implementations >>> +Second, ODP consists of multiple implementations of this API >>> specification, >>> +each tailored to a specific target platform. ODP implementations >>> determine >>> +how each ODP abstract type is represented on that platform and how each >>> ODP >>> +API is realized. On some platforms, ODP APIs will >>> +be realized using specialized instructions that accelerate the >>> functional >>> +behavior specified by the API. On others, hardware co-processing >>> engines may >>> +completely offload the API so that again it can be performed with >>> little or no >>> +involvement by a CPU. In all cases, the application sees the same >>> +functional behavior independent of how a given platform has chosen to >>> realize >>> +it. By allowing each platform the freedom to determine how best to >>> realize each >>> +API's specified functional behavior in an optimal manner, ODP permits >>> +applications written to its APIs to take full advantage of the unique >>> +capabilities of each platform without the application programmer >>> needing to >>> +have specialist knowledge of that platform or to be concerned with how >>> best >>> +to tune the application to a particular platform. This latter >>> consideration is >>> +particularly important in Network Function Virtualization (NFV) >>> environments >>> +where the application will run on a target platform chosen by someone >>> else. >>> + >>> +.Summary: ODP Implementation Characteristics >>> +* One size does not fit all--supporting multiple implementations allows >>> ODP >>> +to adapt to widely differing internals among platforms. >>> +* Anyone can create an ODP implementation tailored to their platform >>> +* Distribution and mainteinance of each implementation is as owner >>> wishes >>> + - Open source or closed source as business needs determine >>> + - Have independent release cycles and service streams >>> +* Allows HW and SW innovation in how ODP APIs are implemented on each >>> platform. >>> + >>> +.Reference Implementations >>> +To make it easy to get started with implementing ODP on a new platform, >>> ODP >>> +supplies a number of _reference implementations_ that can serve as a >>> +starting point. The two primary references implementations supplied by >>> ODP are >>> +*odp-linux* and *odp-dpdk* >>> + >>> +.odp-linux >>> +The *odp-linux* reference implementation is a pure SW implementation of >>> the >>> +ODP API that relies only on the Linux programming API. As a functional >>> model >>> +for ODP, it enables ODP to be bootstrapped easily to any platform that >>> +supports a Linux kernel. >>> + >>> +.odp-dpdk >>> +The *odp-dpdk* reference implementation is a pure SW implementation of >>> the >>> +ODP API that uses http://dpdk.org[DPDK] as a SW accelerator. In >>> particular, >>> +*odp-dpdk* offers superior I/O performance for systems that use NICs, >>> allowing >>> +ODP applications to take immediate full advantage of the various NIC >>> device >>> +drivers supported by DPDK. >>> + >>> +.Summary: ODP Reference Implementations >>> +* Open source, open contribution, BSD-3 licensed. >>> +* Provide easy bootstrapping of ODP onto new platforms >>> +* Implementers free to borrow or tailor code as needed for their >>> platform >>> +* Implementers retain full control over their implementations whether >>> or not >>> +they are derived from a reference implementation. >>> + >>> +.ODP Validation Test Suite >>> +Third, to enure consistency between different ODP implementations, ODP >>> +consists of a validation suite that verifies that any given >>> implementation of >>> +ODP faithfully provides the specified functional behavior of each ODP >>> API. >>> +As a separate open source component, the validation suite may be used by >>> +application writers, system integrators, and platform providers alike to >>> +confirm that any purported implementation of ODP does indeed conform to >>> the >>> +ODP API specification. >>> + >>> +.Summary: ODP Validation Test Suite >>> +* Synchronized with ODP API specification >>> +* Maintained and distributed by LNG >>> +* Open source, open contribution, BSD-3 licensed. >>> +* Key to ensuring application portability across all ODP implementations >>> +* Tests that ODP implementations conform to the specified functional >>> behavior >>> +of ODP APIs. >>> +* Can be run at any time by users and vendors to validat implementations >>> +od ODP. >>> + >>> +=== ODP API Specification Versioning === >>> +As an evolving standard, the ODP API specification is released under an >>> +incrementing version number, and corresponding implementations of ODP, >>> as well >>> +as the validation suite that verifies API conformance, are linked to >>> this >>> +version number. ODP versions are specified using a stanard three-level >>> +number (major.minor.fixlevel) that are incremented according to the >>> degree of >>> +change the level represents. Increments to the fixlevel represent >>> clarification >>> +of the specification or other minor changes that do not affect either >>> the >>> +syntax or semantics of the specification. Such changes in the API >>> specification >>> +are expected to be rare. Increments to the minor level >>> +represent the introduction of new APIs or functional capabilities, or >>> changes >>> +to he specified syntax or functional behavior of APIs and thus may >>> require >>> +application source code changes. Such changes are well documented in the >>> +release notes for each revision of the specification. Finally, >>> increments to >>> +the major level represent significant structural changes that most >>> likely >>> +require some level of application source code change, again as >>> documented in >>> +the release notes for that version. >>> + >>> +=== ODP Implementation Versioning === >>> +ODP implementations are free to use whatever release naming/numbering >>> +conventions they wish, as long as it is clear what level of the ODP API >>> a given >>> +release implements. A recommended convention is to use the same three >>> level >>> +numbering scheme where the major and minor numbers correspond to the >>> ODP API >>> +level and the fixlevel represents an implementation-defined service >>> level >>> +associated with that API level implementation. The LNG-supplied ODP >>> reference >>> +implementations follow this convention. >>> + >>> +=== ODP Validation Test Suite Versioning === >>> +The ODP validation test suite follows these same naming conventions. >>> The major >>> +and minor release numbers correspond to the ODP API level that the suite >>> +validates and the fixlevel represents the service level of the >>> validation >>> +suite itself for that API level. >>> + >>> +=== ODP Design Goals === >>> +ODP has three primary goals that follow from its component structure. >>> The first >>> +is application portability across a wide range of platforms. These >>> platforms >>> +differ in terms of processor instruction set architecture, number and >>> types of >>> +application processing cores, memory oranization, as well as the number >>> and >>> +type of platform specific hardware acceleration and offload features >>> that >>> +are available. ODP applications can move from one conforming >>> implementation >>> +to another with at most a recompile. >>> + >>> +Second, ODP is designed to permit data plane applications to avail >>> themselves >>> +of platform-specific features, including specialized hardware >>> accelerators, >>> +without specialized programming. This is achieved by separating the API >>> +specification from their implementation on individual platforms. Since >>> each >>> +platform implements each ODP API in a manner optimal to that platform, >>> +applications automatically gain the benefit of such optimizations >>> without the >>> +need for explicit programming. >>> + >>> +Third, ODP is designed to allow applications to scale out automatically >>> to >>> +support many core architectures. This is done using an event based >>> programming >>> +model that permits applications to be written to be independent of the >>> number >>> +of processing cores that are available to realize application function. >>> The >>> +result is that an application written to this model does not require >>> redesign >>> +as it scales from 4, to 40, to 400 cores. >>> + >>> +== Organization of this Document == >>> +This document is organized into several sections. The first presents a >>> high >>> +level overview of the ODP API component areas and their associated >>> abstract >>> +data types. This section introduces ODP APIs at a conceptual level. >>> +The second provides a tutorial on the programming model(s) >>> +supported by ODP, paying particular attention to the event model as this >>> +represents the preferred structure for most ODP applications. This >>> section >>> +builds on the concepts introduced in the first section and shows how ODP >>> +applications are structured to best realize the three ODP design goals >>> +mentioned earlier. The third section provides a more detailed overview >>> of >>> +the major ODP API components and is designed to serve as a companion to >>> the >>> +full reference specification for each API. The latter is intended to be >>> used >>> +by ODP application programmers, as well as implementers, to understand >>> the >>> +precise syntax and semantics of each API. >>> + >>> +== ODP API Concepts == >>> +ODP programs are built around several conceptual structures that every >>> +appliation programmer needs to be familiar with to use ODP effectively. >>> The >>> +main ODP concepts are: >>> +Thread, Event, Queue, Pool, Shared Memory, Buffer, Packet, PktIO, Timer, >>> +and Synchronizer. >>> + >>> +=== Thread === >>> +The thread is the fundamental programming unit in ODP. ODP >>> applications are >>> +organized into a collection of threads that perform the work that the >>> +application is designed to do. ODP threads may or may not share memory >>> with >>> +other threads--that is up to the implementation. Threads come in two >>> "flavors": >>> +control and worker, that are represented by the abstract type >>> ++odp_thread_type_t+. >>> + >>> +A control thread is a supervisory thread that organizes >>> +the operation of worker threads. Worker threads, by contrast, exist to >>> +perform the main processing logic of the application and employ a run to >>> +completion model. Worker threads, in particular, are intended to >>> operate on >>> +dedicated processing cores, especially in many core proessing >>> environments, >>> +however a given implementation may multitask multiple threads on a >>> single >>> +core if desired (typically on smaller and lower performance target >>> +environments). >>> + >>> +In addition to thread types, threads have associated _attributes_ such >>> as >>> +_thread mask_ and _scheduler group_ that determine where they can run >>> and >>> +the type of work that they can handle. These will be discussed in >>> greater >>> +detail later. >>> + >>> +=== Event === >>> +Events are what threads process to perform their work. Events can >>> represent >>> +new work, such as the arrival of a packet that needs to be processed, >>> or they >>> +can represent the completion of requests that have executed >>> asynchronously. >>> +Events can also represent notifications of the passage of time, or of >>> status >>> +changes in various components of interest to the application. Events >>> have an >>> +event type that describes what it represents. Threads can create new >>> events >>> +or consume events processed by them, or they can perform some >>> processing on >>> +an event and then pass it along to another component for further >>> processing. >>> +References to events are via handles of abstract type +odp_event_t+. >>> Cast >>> +functions are provided to convert these into specific handles of the >>> +appropriate type represented by the event. >>> + >>> +=== Queue === >>> +A queue is a message passing channel that holds events. Events can be >>> +added to a queue via enqueue operations or removed from a queue via >>> dequeue >>> +operations. The endpoints of a queue will vary depending on how it is >>> used. >>> +Queues come in two major types: polled and scheduled, which will be >>> +discussed in more detail when the event model is introduced. Queues may >>> also >>> +have an associated context, which represents a persistent state for all >>> +events that make use of it. These states are what permit threads to >>> perform >>> +stateful processing on events as well as stateless processing. >>> + >>> +Queues are represented by handles of abstract type +odp_queue_t+. >>> + >>> +=== Pool === >>> +A pool is a shared memory area from which elements may be drawn. Pools >>> +represent the backing store for events, among other things. Pools are >>> +typically created and destroyed by the application during >>> initialization and >>> +termination, respectively, and then used during processing. Pools may be >>> +used by ODP components exclusively, by applications exclusively, or >>> their >>> +use may be shared between the two. Pools have an associated type that >>> +characterizes the elements that they contain. The two most important >>> pool types >>> +are Buffer and Packet. >>> + >>> +Pools are represented by handles of abstract type +odp_pool_t+. >>> + >>> +=== Shared Memory === >>> +Shared memory represents raw blocks of storage that are sharable between >>> +threads. They are the building blocks of pools but can be used directly >>> by >>> +ODP applications if desired. >>> + >>> +Shared memory is represented by handles of abstract type +odp_shm_t+. >>> + >>> +=== Buffer === >>> +A buffer is a fixed sized block of shared storage that is used by ODP >>> +components and/or applications to realize their function. Buffers >>> contain >>> +zero or more bytes of application data as well as system maintained >>> +metadata that provide information about the buffer, such as its size or >>> the >>> +pool it was allocated from. Metadata is an important ODP concept >>> because it >>> +allows for arbitrary amounts of side information to be associated with >>> an >>> +ODP object. Most ODP objects have assocaited metadata and this metadata >>> is >>> +manipulated via accessor functions that act as getters and setters for >>> +this information. Getter acces functions permit an application to read >>> +a metadata item, while setter access functions permit an application to >>> write >>> +a metadata item. Note that some metadata is inherently read only and >>> thus >>> +no setter is provided to manipulate it. When object have multiple >>> metadata >>> +items, each has its own associated getter and/or setter access function >>> to >>> +inspect or manipulate it. >>> + >>> +Buffers are represened by handles of abstract type +odp_buffer_t+. >>> + >>> +=== Packet === >>> +Packets are received and transmitted via I/O interfaces and represent >>> +the basic data that data plane applications manipulate. >>> +Packets are drawn from pools of type +ODP_POOL_PACKET+. >>> +Unlike buffers, which are simple objects, >>> +ODP packets have a rich set of semantics that permit their inspection >>> +and manipulation in complex ways to be described later. Packets also >>> support >>> +a rich set of metadata as well as user metadata. User metadata permits >>> +applications to associate an application-determined amount of side >>> information >>> +with each packet for its own use. >>> + >>> +Packets are represented by handles of abstract type +odp_packet_t+. >>> + >>> +=== PktIO === >>> +PktIO is how ODP represents I/O interfaces. A pktio object is a logical >>> +port capable of receiving and/or transmitting packets. This may be >>> directly >>> +supported by the underlying platform as an integrated feature, >>> +or may represent a device attached via a PCIE or other bus. >>> + >>> +PktIOs are represented by handles of abstract type +odp_pktio_t+. >>> + >>> +=== Timer === >>> +Timers are how ODP applications measure and respond to the passage of >>> time. >>> +Timers are drawn from specialized pools called timer pools that have >>> their >>> +own abstract type (+odp_timer_pool_t+). Applications may have many >>> timers >>> +active at the same time and can set them to use either relative or >>> absolute >>> +time. When timers expire they create events of type +odp_timeout_t+, >>> which >>> +serve as notifications of timer expiration. >>> + >>> +=== Synchronizer === >>> +Multiple threads operating in parallel typically require various >>> +synchronization services to permit them to operate in a reliable and >>> +coordinated manner. ODP provides a rich set of locks, barriers, and >>> similar >>> +synchronization primitives, as well as abstract types for representing >>> various >>> +types of atomic variables. The ODP event model also makes use of queues >>> to >>> +avoid the need for explicit locking in many cases. This will be >>> discussed >>> +in the next section. >>> + >>> +== ODP Components == >>> +Building on ODP concepts, ODP offers several components that relate to >>> the >>> +flow of work through an ODP application. These include the Classifier, >>> +Scheduler, and Traffic Manager. These components relate to the three >>> +main stages of packet processing: Receive, Process, and Transmit. >>> + >>> +=== Classifier === >>> +The *Classifier* provides a suite of APIs that control packet receive >>> (RX) >>> +processing. >>> + >>> +.ODP Receive Processing with Classifier >>> +image::../images/odp_rx_processing.png[align="center"] >>> + >>> +The classifier provides two logically related services: >>> +[horizontal] >>> +Packet parsing:: Verifying and extracting structural information from a >>> +received packet. >>> + >>> +Packet classification:: Applying *Pattern Matching Rules (PMRs)* to the >>> +parsed results to assign an incoming packet to a *Class of Service >>> (CoS)*. >>> + >>> +Combined, these permit incoming packets to be sorted into *flows*, >>> which are >>> +logically related sequences of packets that share common processing >>> +requirements. While many data plane applications perform stateless >>> packet >>> +processing (_e.g.,_ for simple forwarding) others perform stateful >>> packet >>> +processing. Flows anchor state information relating to these groups of >>> +packets. >>> + >>> +A CoS determines two variables for packets belonging to a flow: >>> +[list] >>> +* The pool that they will be stored in on receipt >>> +* The queue that they will be added to for processing >>> + >>> +The PMRs supported by ODP permit flow determination based on >>> combinations of >>> +packet field values (tuples). The main advantage of classification is >>> that on >>> +many platforms these functions are performed in hardware, meaning that >>> +classification occurs at line rate as packets are being received without >>> +any explicit processing by the ODP application. >>> + >>> +Note that the use of the classifier is optional. Applications may >>> directly >>> +receive packets from a corresponding PktIO input queue via direct >>> polling >>> +if they choose. >>> + >>> +=== Scheduler === >>> +The *Scheduler* provides a suite of APIs that control scalabable event >>> +processing. >>> + >>> +.ODP Scheduler and Event Processing >>> +image::../images/odp_scheduling.png[align="center"] >>> + >>> +The Scheduler is responsible for selecting and dispatching one or more >>> events >>> +to a requesting thread. Event selection is based on several factors >>> involving >>> +both the queues containing schedulable events and the thread making an >>> ++odp_schedule()+ or +odp_schedule_multi()+ call. >>> + >>> +ODP queues have a _scheduling priority_ that determines how urgently >>> events >>> +on them should be processed relative to events contained in other >>> queues. >>> +Queues also have a _scheduler group id_ associated with them that must >>> match >>> +the associated scheduler group _thread mask_ of the thread calling the >>> +scheduler. This permits events to be grouped for processing into >>> classes and >>> +have threads that are dedicated to processing events from specified >>> classes. >>> +Threads can join and leave scheduler groups dynamically, permitting easy >>> +application response to increases in demand. >>> + >>> +When a thread receives an event from the scheduler, it in turn can >>> invoke >>> +other processing engines via ODP APIs (_e.g.,_ crypto processing) that >>> +can operate asynchronously. When such processing is complete, the >>> result is >>> +that a *completion event* is added to a schedulable queue where it can >>> be >>> +scheduled back to a thread to continue processing with the results of >>> the >>> +requested asynchronous operation. >>> + >>> +Threads themselves can enqueue events to queues for downstream >>> processing >>> +by other threads, permitting flexibility in how applicaitions structure >>> +themselves to maximize concurrency. >>> + >>> +=== Traffic Manager === >>> +The *Traffic Manager* provides a suite of APIs that control traffic >>> shaping and >>> +Quality of Service (QoS) processing for packet output. >>> + >>> +.ODP Transmit processing with Traffic Manager >>> +image::../images/odp_traffic_manager.png[align="center"] >>> + >>> +The final stage of packet processing is to transmit it. Here, >>> applications have >>> +several choices. As with RX processing, applications may send packets >>> +directly to PktIO TX queues for direct transmission. Often, however, >>> +applications need to perform traffic shaping and related >>> +*Quality of Service (QoS)* processing on the packets comprising a flow >>> as part >>> +of transmit processing. To handle this need, ODP provides a suite of >>> +*Traffic Manager* APIs that permit programmatic establishment of >>> arbiters, >>> +shapers, etc. that control output packet processing to achieve desired >>> QoS >>> +goals. Again, the advantage here is that on many platforms traffic >>> management >>> +functions are implemented in hardware, permitting transparent offload of >>> +this work. >>> + >>> Glossary >>> -------- >>> [glossary] >>> @@ -28,7 +425,6 @@ odp_thread:: >>> event:: >>> An event is a notification that can be placed in a queue. >>> >>> -:numbered: >>> The include structure >>> --------------------- >>> Applications only include the 'include/odp.h file which includes the >>> 'platform/<implementation name>/include/plat' files to provide a complete >>> definition of the API on that platform. >>> -- >>> 2.1.4 >>> >>> _______________________________________________ >>> lng-odp mailing list >>> lng-odp@lists.linaro.org >>> https://lists.linaro.org/mailman/listinfo/lng-odp >>> >> >> >> >> -- >> Mike Holmes >> Technical Manager - Linaro Networking Group >> Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM SoCs >> >> >> > -- Mike Holmes Technical Manager - Linaro Networking Group Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM SoCs
That's what make does :) On Tue, Nov 17, 2015 at 11:57 AM, Mike Holmes <mike.holmes@linaro.org> wrote: > > > On 17 November 2015 at 12:54, Bill Fischofer <bill.fischofer@linaro.org> > wrote: > >> Is that a review comment? Is there a problem with the rendering? >> > > Not a review comment just to make it easier for folks to read initially. > > >> On Tue, Nov 17, 2015 at 11:51 AM, Mike Holmes <mike.holmes@linaro.org> >> wrote: >> >>> This is the rendered doc >>> >>> http://people.linaro.org/~mike.holmes/output/users-guide.html >>> >>> On 17 November 2015 at 12:15, Bill Fischofer <bill.fischofer@linaro.org> >>> wrote: >>> >>>> Add a basic ODP overview to the User's Guide, providing >>>> an overview of ODP concepts, components, etc. Included >>>> are a number of diagrams covering component structure >>>> as well as packet RX, event scheduling, and packet TX >>>> processing. >>>> >>>> Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> >>>> --- >>>> doc/users-guide/users-guide.adoc | 400 >>>> ++++++++++++++++++++++++++++++++++++++- >>>> 1 file changed, 398 insertions(+), 2 deletions(-) >>>> >>>> diff --git a/doc/users-guide/users-guide.adoc >>>> b/doc/users-guide/users-guide.adoc >>>> index 7f70046..cf77fa0 100644 >>>> --- a/doc/users-guide/users-guide.adoc >>>> +++ b/doc/users-guide/users-guide.adoc >>>> @@ -8,7 +8,7 @@ OpenDataPlane (ODP) Users-Guide >>>> Abstract >>>> -------- >>>> This document is intended to guide a new ODP application developer. >>>> -Further details about ODP may be found at then >>>> http://opendataplane.org[ODP] home page. >>>> +Further details about ODP may be found at the http://opendataplane.org[ODP] >>>> home page. >>>> >>>> .Overview of a system running ODP applications >>>> image::../images/overview.png[align="center"] >>>> @@ -16,6 +16,403 @@ image::../images/overview.png[align="center"] >>>> ODP is an API specification that allows many implementations to >>>> provide platform independence, automatic hardware acceleration and CPU >>>> scaling to high performance networking applications. >>>> This document describes how to write an application that can >>>> successfully take advantage of the API. >>>> >>>> +:numbered: >>>> +== Introduction == >>>> +.OpenDataPlane Components >>>> +image::../images/odp_components.png[align="center"] >>>> + >>>> +.The ODP API Specification >>>> +ODP consists of three separate but related component parts. First, ODP >>>> is an >>>> +abstract API specification that describes a functional model for >>>> +data plane applications. This specification covers many common data >>>> plane >>>> +application programming needs, such as the ability to receive, >>>> manipulate, and >>>> +transmit packet data, without specifying how these functions are >>>> performed. This >>>> +is quite intentional. It is precisely because ODP APIs do not have a >>>> preferred >>>> +embodiment that they permit innovation in how these functions can >>>> +be realized on various platforms that offer implementations of ODP. To >>>> achieve >>>> +this goal, ODP APIs are described using abstract data types whose >>>> definition >>>> +is left up to the ODP implementer. For example, in ODP packets are >>>> referenced >>>> +by abstract handles of type +odp_packet_t+, and packet-related APIs >>>> take >>>> +arguments of this type. What an +odp_packet_t+ actually is is not part >>>> of the >>>> +ODP API specification--that is the responsibility of each ODP >>>> implementation. >>>> + >>>> +.Summary: ODP API attributes: >>>> +* Open Source, open contribution, BSD-3 licensed. >>>> +* Vendor and platform neutral. >>>> +* Application-centric. Covers functional needs of data plane >>>> applications. >>>> +* Ensures portability by specifying the functional behavior of ODP. >>>> +* Defined jointly and openly by application writers and platform >>>> implementers. >>>> +* Archiected to be implementable on a wide range of platforms >>>> efficiently >>>> +* Sponsored, governed, and maintained by the Linaro Networking Group >>>> (LNG) >>>> + >>>> +.ODP Implementations >>>> +Second, ODP consists of multiple implementations of this API >>>> specification, >>>> +each tailored to a specific target platform. ODP implementations >>>> determine >>>> +how each ODP abstract type is represented on that platform and how >>>> each ODP >>>> +API is realized. On some platforms, ODP APIs will >>>> +be realized using specialized instructions that accelerate the >>>> functional >>>> +behavior specified by the API. On others, hardware co-processing >>>> engines may >>>> +completely offload the API so that again it can be performed with >>>> little or no >>>> +involvement by a CPU. In all cases, the application sees the same >>>> +functional behavior independent of how a given platform has chosen to >>>> realize >>>> +it. By allowing each platform the freedom to determine how best to >>>> realize each >>>> +API's specified functional behavior in an optimal manner, ODP permits >>>> +applications written to its APIs to take full advantage of the unique >>>> +capabilities of each platform without the application programmer >>>> needing to >>>> +have specialist knowledge of that platform or to be concerned with how >>>> best >>>> +to tune the application to a particular platform. This latter >>>> consideration is >>>> +particularly important in Network Function Virtualization (NFV) >>>> environments >>>> +where the application will run on a target platform chosen by someone >>>> else. >>>> + >>>> +.Summary: ODP Implementation Characteristics >>>> +* One size does not fit all--supporting multiple implementations >>>> allows ODP >>>> +to adapt to widely differing internals among platforms. >>>> +* Anyone can create an ODP implementation tailored to their platform >>>> +* Distribution and mainteinance of each implementation is as owner >>>> wishes >>>> + - Open source or closed source as business needs determine >>>> + - Have independent release cycles and service streams >>>> +* Allows HW and SW innovation in how ODP APIs are implemented on each >>>> platform. >>>> + >>>> +.Reference Implementations >>>> +To make it easy to get started with implementing ODP on a new >>>> platform, ODP >>>> +supplies a number of _reference implementations_ that can serve as a >>>> +starting point. The two primary references implementations supplied >>>> by ODP are >>>> +*odp-linux* and *odp-dpdk* >>>> + >>>> +.odp-linux >>>> +The *odp-linux* reference implementation is a pure SW implementation >>>> of the >>>> +ODP API that relies only on the Linux programming API. As a functional >>>> model >>>> +for ODP, it enables ODP to be bootstrapped easily to any platform that >>>> +supports a Linux kernel. >>>> + >>>> +.odp-dpdk >>>> +The *odp-dpdk* reference implementation is a pure SW implementation of >>>> the >>>> +ODP API that uses http://dpdk.org[DPDK] as a SW accelerator. In >>>> particular, >>>> +*odp-dpdk* offers superior I/O performance for systems that use NICs, >>>> allowing >>>> +ODP applications to take immediate full advantage of the various NIC >>>> device >>>> +drivers supported by DPDK. >>>> + >>>> +.Summary: ODP Reference Implementations >>>> +* Open source, open contribution, BSD-3 licensed. >>>> +* Provide easy bootstrapping of ODP onto new platforms >>>> +* Implementers free to borrow or tailor code as needed for their >>>> platform >>>> +* Implementers retain full control over their implementations whether >>>> or not >>>> +they are derived from a reference implementation. >>>> + >>>> +.ODP Validation Test Suite >>>> +Third, to enure consistency between different ODP implementations, ODP >>>> +consists of a validation suite that verifies that any given >>>> implementation of >>>> +ODP faithfully provides the specified functional behavior of each ODP >>>> API. >>>> +As a separate open source component, the validation suite may be used >>>> by >>>> +application writers, system integrators, and platform providers alike >>>> to >>>> +confirm that any purported implementation of ODP does indeed conform >>>> to the >>>> +ODP API specification. >>>> + >>>> +.Summary: ODP Validation Test Suite >>>> +* Synchronized with ODP API specification >>>> +* Maintained and distributed by LNG >>>> +* Open source, open contribution, BSD-3 licensed. >>>> +* Key to ensuring application portability across all ODP >>>> implementations >>>> +* Tests that ODP implementations conform to the specified functional >>>> behavior >>>> +of ODP APIs. >>>> +* Can be run at any time by users and vendors to validat >>>> implementations >>>> +od ODP. >>>> + >>>> +=== ODP API Specification Versioning === >>>> +As an evolving standard, the ODP API specification is released under an >>>> +incrementing version number, and corresponding implementations of ODP, >>>> as well >>>> +as the validation suite that verifies API conformance, are linked to >>>> this >>>> +version number. ODP versions are specified using a stanard three-level >>>> +number (major.minor.fixlevel) that are incremented according to the >>>> degree of >>>> +change the level represents. Increments to the fixlevel represent >>>> clarification >>>> +of the specification or other minor changes that do not affect either >>>> the >>>> +syntax or semantics of the specification. Such changes in the API >>>> specification >>>> +are expected to be rare. Increments to the minor level >>>> +represent the introduction of new APIs or functional capabilities, or >>>> changes >>>> +to he specified syntax or functional behavior of APIs and thus may >>>> require >>>> +application source code changes. Such changes are well documented in >>>> the >>>> +release notes for each revision of the specification. Finally, >>>> increments to >>>> +the major level represent significant structural changes that most >>>> likely >>>> +require some level of application source code change, again as >>>> documented in >>>> +the release notes for that version. >>>> + >>>> +=== ODP Implementation Versioning === >>>> +ODP implementations are free to use whatever release naming/numbering >>>> +conventions they wish, as long as it is clear what level of the ODP >>>> API a given >>>> +release implements. A recommended convention is to use the same three >>>> level >>>> +numbering scheme where the major and minor numbers correspond to the >>>> ODP API >>>> +level and the fixlevel represents an implementation-defined service >>>> level >>>> +associated with that API level implementation. The LNG-supplied ODP >>>> reference >>>> +implementations follow this convention. >>>> + >>>> +=== ODP Validation Test Suite Versioning === >>>> +The ODP validation test suite follows these same naming conventions. >>>> The major >>>> +and minor release numbers correspond to the ODP API level that the >>>> suite >>>> +validates and the fixlevel represents the service level of the >>>> validation >>>> +suite itself for that API level. >>>> + >>>> +=== ODP Design Goals === >>>> +ODP has three primary goals that follow from its component structure. >>>> The first >>>> +is application portability across a wide range of platforms. These >>>> platforms >>>> +differ in terms of processor instruction set architecture, number and >>>> types of >>>> +application processing cores, memory oranization, as well as the >>>> number and >>>> +type of platform specific hardware acceleration and offload features >>>> that >>>> +are available. ODP applications can move from one conforming >>>> implementation >>>> +to another with at most a recompile. >>>> + >>>> +Second, ODP is designed to permit data plane applications to avail >>>> themselves >>>> +of platform-specific features, including specialized hardware >>>> accelerators, >>>> +without specialized programming. This is achieved by separating the API >>>> +specification from their implementation on individual platforms. Since >>>> each >>>> +platform implements each ODP API in a manner optimal to that platform, >>>> +applications automatically gain the benefit of such optimizations >>>> without the >>>> +need for explicit programming. >>>> + >>>> +Third, ODP is designed to allow applications to scale out >>>> automatically to >>>> +support many core architectures. This is done using an event based >>>> programming >>>> +model that permits applications to be written to be independent of the >>>> number >>>> +of processing cores that are available to realize application >>>> function. The >>>> +result is that an application written to this model does not require >>>> redesign >>>> +as it scales from 4, to 40, to 400 cores. >>>> + >>>> +== Organization of this Document == >>>> +This document is organized into several sections. The first presents a >>>> high >>>> +level overview of the ODP API component areas and their associated >>>> abstract >>>> +data types. This section introduces ODP APIs at a conceptual level. >>>> +The second provides a tutorial on the programming model(s) >>>> +supported by ODP, paying particular attention to the event model as >>>> this >>>> +represents the preferred structure for most ODP applications. This >>>> section >>>> +builds on the concepts introduced in the first section and shows how >>>> ODP >>>> +applications are structured to best realize the three ODP design goals >>>> +mentioned earlier. The third section provides a more detailed overview >>>> of >>>> +the major ODP API components and is designed to serve as a companion >>>> to the >>>> +full reference specification for each API. The latter is intended to >>>> be used >>>> +by ODP application programmers, as well as implementers, to understand >>>> the >>>> +precise syntax and semantics of each API. >>>> + >>>> +== ODP API Concepts == >>>> +ODP programs are built around several conceptual structures that every >>>> +appliation programmer needs to be familiar with to use ODP >>>> effectively. The >>>> +main ODP concepts are: >>>> +Thread, Event, Queue, Pool, Shared Memory, Buffer, Packet, PktIO, >>>> Timer, >>>> +and Synchronizer. >>>> + >>>> +=== Thread === >>>> +The thread is the fundamental programming unit in ODP. ODP >>>> applications are >>>> +organized into a collection of threads that perform the work that the >>>> +application is designed to do. ODP threads may or may not share memory >>>> with >>>> +other threads--that is up to the implementation. Threads come in two >>>> "flavors": >>>> +control and worker, that are represented by the abstract type >>>> ++odp_thread_type_t+. >>>> + >>>> +A control thread is a supervisory thread that organizes >>>> +the operation of worker threads. Worker threads, by contrast, exist to >>>> +perform the main processing logic of the application and employ a run >>>> to >>>> +completion model. Worker threads, in particular, are intended to >>>> operate on >>>> +dedicated processing cores, especially in many core proessing >>>> environments, >>>> +however a given implementation may multitask multiple threads on a >>>> single >>>> +core if desired (typically on smaller and lower performance target >>>> +environments). >>>> + >>>> +In addition to thread types, threads have associated _attributes_ such >>>> as >>>> +_thread mask_ and _scheduler group_ that determine where they can run >>>> and >>>> +the type of work that they can handle. These will be discussed in >>>> greater >>>> +detail later. >>>> + >>>> +=== Event === >>>> +Events are what threads process to perform their work. Events can >>>> represent >>>> +new work, such as the arrival of a packet that needs to be processed, >>>> or they >>>> +can represent the completion of requests that have executed >>>> asynchronously. >>>> +Events can also represent notifications of the passage of time, or of >>>> status >>>> +changes in various components of interest to the application. Events >>>> have an >>>> +event type that describes what it represents. Threads can create new >>>> events >>>> +or consume events processed by them, or they can perform some >>>> processing on >>>> +an event and then pass it along to another component for further >>>> processing. >>>> +References to events are via handles of abstract type +odp_event_t+. >>>> Cast >>>> +functions are provided to convert these into specific handles of the >>>> +appropriate type represented by the event. >>>> + >>>> +=== Queue === >>>> +A queue is a message passing channel that holds events. Events can be >>>> +added to a queue via enqueue operations or removed from a queue via >>>> dequeue >>>> +operations. The endpoints of a queue will vary depending on how it is >>>> used. >>>> +Queues come in two major types: polled and scheduled, which will be >>>> +discussed in more detail when the event model is introduced. Queues >>>> may also >>>> +have an associated context, which represents a persistent state for all >>>> +events that make use of it. These states are what permit threads to >>>> perform >>>> +stateful processing on events as well as stateless processing. >>>> + >>>> +Queues are represented by handles of abstract type +odp_queue_t+. >>>> + >>>> +=== Pool === >>>> +A pool is a shared memory area from which elements may be drawn. Pools >>>> +represent the backing store for events, among other things. Pools are >>>> +typically created and destroyed by the application during >>>> initialization and >>>> +termination, respectively, and then used during processing. Pools may >>>> be >>>> +used by ODP components exclusively, by applications exclusively, or >>>> their >>>> +use may be shared between the two. Pools have an associated type that >>>> +characterizes the elements that they contain. The two most important >>>> pool types >>>> +are Buffer and Packet. >>>> + >>>> +Pools are represented by handles of abstract type +odp_pool_t+. >>>> + >>>> +=== Shared Memory === >>>> +Shared memory represents raw blocks of storage that are sharable >>>> between >>>> +threads. They are the building blocks of pools but can be used >>>> directly by >>>> +ODP applications if desired. >>>> + >>>> +Shared memory is represented by handles of abstract type +odp_shm_t+. >>>> + >>>> +=== Buffer === >>>> +A buffer is a fixed sized block of shared storage that is used by ODP >>>> +components and/or applications to realize their function. Buffers >>>> contain >>>> +zero or more bytes of application data as well as system maintained >>>> +metadata that provide information about the buffer, such as its size >>>> or the >>>> +pool it was allocated from. Metadata is an important ODP concept >>>> because it >>>> +allows for arbitrary amounts of side information to be associated with >>>> an >>>> +ODP object. Most ODP objects have assocaited metadata and this >>>> metadata is >>>> +manipulated via accessor functions that act as getters and setters for >>>> +this information. Getter acces functions permit an application to read >>>> +a metadata item, while setter access functions permit an application >>>> to write >>>> +a metadata item. Note that some metadata is inherently read only and >>>> thus >>>> +no setter is provided to manipulate it. When object have multiple >>>> metadata >>>> +items, each has its own associated getter and/or setter access >>>> function to >>>> +inspect or manipulate it. >>>> + >>>> +Buffers are represened by handles of abstract type +odp_buffer_t+. >>>> + >>>> +=== Packet === >>>> +Packets are received and transmitted via I/O interfaces and represent >>>> +the basic data that data plane applications manipulate. >>>> +Packets are drawn from pools of type +ODP_POOL_PACKET+. >>>> +Unlike buffers, which are simple objects, >>>> +ODP packets have a rich set of semantics that permit their inspection >>>> +and manipulation in complex ways to be described later. Packets also >>>> support >>>> +a rich set of metadata as well as user metadata. User metadata permits >>>> +applications to associate an application-determined amount of side >>>> information >>>> +with each packet for its own use. >>>> + >>>> +Packets are represented by handles of abstract type +odp_packet_t+. >>>> + >>>> +=== PktIO === >>>> +PktIO is how ODP represents I/O interfaces. A pktio object is a logical >>>> +port capable of receiving and/or transmitting packets. This may be >>>> directly >>>> +supported by the underlying platform as an integrated feature, >>>> +or may represent a device attached via a PCIE or other bus. >>>> + >>>> +PktIOs are represented by handles of abstract type +odp_pktio_t+. >>>> + >>>> +=== Timer === >>>> +Timers are how ODP applications measure and respond to the passage of >>>> time. >>>> +Timers are drawn from specialized pools called timer pools that have >>>> their >>>> +own abstract type (+odp_timer_pool_t+). Applications may have many >>>> timers >>>> +active at the same time and can set them to use either relative or >>>> absolute >>>> +time. When timers expire they create events of type +odp_timeout_t+, >>>> which >>>> +serve as notifications of timer expiration. >>>> + >>>> +=== Synchronizer === >>>> +Multiple threads operating in parallel typically require various >>>> +synchronization services to permit them to operate in a reliable and >>>> +coordinated manner. ODP provides a rich set of locks, barriers, and >>>> similar >>>> +synchronization primitives, as well as abstract types for representing >>>> various >>>> +types of atomic variables. The ODP event model also makes use of >>>> queues to >>>> +avoid the need for explicit locking in many cases. This will be >>>> discussed >>>> +in the next section. >>>> + >>>> +== ODP Components == >>>> +Building on ODP concepts, ODP offers several components that relate to >>>> the >>>> +flow of work through an ODP application. These include the Classifier, >>>> +Scheduler, and Traffic Manager. These components relate to the three >>>> +main stages of packet processing: Receive, Process, and Transmit. >>>> + >>>> +=== Classifier === >>>> +The *Classifier* provides a suite of APIs that control packet receive >>>> (RX) >>>> +processing. >>>> + >>>> +.ODP Receive Processing with Classifier >>>> +image::../images/odp_rx_processing.png[align="center"] >>>> + >>>> +The classifier provides two logically related services: >>>> +[horizontal] >>>> +Packet parsing:: Verifying and extracting structural information from a >>>> +received packet. >>>> + >>>> +Packet classification:: Applying *Pattern Matching Rules (PMRs)* to the >>>> +parsed results to assign an incoming packet to a *Class of Service >>>> (CoS)*. >>>> + >>>> +Combined, these permit incoming packets to be sorted into *flows*, >>>> which are >>>> +logically related sequences of packets that share common processing >>>> +requirements. While many data plane applications perform stateless >>>> packet >>>> +processing (_e.g.,_ for simple forwarding) others perform stateful >>>> packet >>>> +processing. Flows anchor state information relating to these groups of >>>> +packets. >>>> + >>>> +A CoS determines two variables for packets belonging to a flow: >>>> +[list] >>>> +* The pool that they will be stored in on receipt >>>> +* The queue that they will be added to for processing >>>> + >>>> +The PMRs supported by ODP permit flow determination based on >>>> combinations of >>>> +packet field values (tuples). The main advantage of classification is >>>> that on >>>> +many platforms these functions are performed in hardware, meaning that >>>> +classification occurs at line rate as packets are being received >>>> without >>>> +any explicit processing by the ODP application. >>>> + >>>> +Note that the use of the classifier is optional. Applications may >>>> directly >>>> +receive packets from a corresponding PktIO input queue via direct >>>> polling >>>> +if they choose. >>>> + >>>> +=== Scheduler === >>>> +The *Scheduler* provides a suite of APIs that control scalabable event >>>> +processing. >>>> + >>>> +.ODP Scheduler and Event Processing >>>> +image::../images/odp_scheduling.png[align="center"] >>>> + >>>> +The Scheduler is responsible for selecting and dispatching one or more >>>> events >>>> +to a requesting thread. Event selection is based on several factors >>>> involving >>>> +both the queues containing schedulable events and the thread making an >>>> ++odp_schedule()+ or +odp_schedule_multi()+ call. >>>> + >>>> +ODP queues have a _scheduling priority_ that determines how urgently >>>> events >>>> +on them should be processed relative to events contained in other >>>> queues. >>>> +Queues also have a _scheduler group id_ associated with them that must >>>> match >>>> +the associated scheduler group _thread mask_ of the thread calling the >>>> +scheduler. This permits events to be grouped for processing into >>>> classes and >>>> +have threads that are dedicated to processing events from specified >>>> classes. >>>> +Threads can join and leave scheduler groups dynamically, permitting >>>> easy >>>> +application response to increases in demand. >>>> + >>>> +When a thread receives an event from the scheduler, it in turn can >>>> invoke >>>> +other processing engines via ODP APIs (_e.g.,_ crypto processing) that >>>> +can operate asynchronously. When such processing is complete, the >>>> result is >>>> +that a *completion event* is added to a schedulable queue where it can >>>> be >>>> +scheduled back to a thread to continue processing with the results of >>>> the >>>> +requested asynchronous operation. >>>> + >>>> +Threads themselves can enqueue events to queues for downstream >>>> processing >>>> +by other threads, permitting flexibility in how applicaitions structure >>>> +themselves to maximize concurrency. >>>> + >>>> +=== Traffic Manager === >>>> +The *Traffic Manager* provides a suite of APIs that control traffic >>>> shaping and >>>> +Quality of Service (QoS) processing for packet output. >>>> + >>>> +.ODP Transmit processing with Traffic Manager >>>> +image::../images/odp_traffic_manager.png[align="center"] >>>> + >>>> +The final stage of packet processing is to transmit it. Here, >>>> applications have >>>> +several choices. As with RX processing, applications may send packets >>>> +directly to PktIO TX queues for direct transmission. Often, however, >>>> +applications need to perform traffic shaping and related >>>> +*Quality of Service (QoS)* processing on the packets comprising a flow >>>> as part >>>> +of transmit processing. To handle this need, ODP provides a suite of >>>> +*Traffic Manager* APIs that permit programmatic establishment of >>>> arbiters, >>>> +shapers, etc. that control output packet processing to achieve desired >>>> QoS >>>> +goals. Again, the advantage here is that on many platforms traffic >>>> management >>>> +functions are implemented in hardware, permitting transparent offload >>>> of >>>> +this work. >>>> + >>>> Glossary >>>> -------- >>>> [glossary] >>>> @@ -28,7 +425,6 @@ odp_thread:: >>>> event:: >>>> An event is a notification that can be placed in a queue. >>>> >>>> -:numbered: >>>> The include structure >>>> --------------------- >>>> Applications only include the 'include/odp.h file which includes the >>>> 'platform/<implementation name>/include/plat' files to provide a complete >>>> definition of the API on that platform. >>>> -- >>>> 2.1.4 >>>> >>>> _______________________________________________ >>>> lng-odp mailing list >>>> lng-odp@lists.linaro.org >>>> https://lists.linaro.org/mailman/listinfo/lng-odp >>>> >>> >>> >>> >>> -- >>> Mike Holmes >>> Technical Manager - Linaro Networking Group >>> Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM >>> SoCs >>> >>> >>> >> > > > -- > Mike Holmes > Technical Manager - Linaro Networking Group > Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM SoCs > > >
On 17 November 2015 at 13:15, Bill Fischofer <bill.fischofer@linaro.org> wrote: > That's what make does :) > Indeed - but reviews on docs are very hard to get, hopefully some one reads it and objects who would normally ignore it! > > On Tue, Nov 17, 2015 at 11:57 AM, Mike Holmes <mike.holmes@linaro.org> > wrote: > >> >> >> On 17 November 2015 at 12:54, Bill Fischofer <bill.fischofer@linaro.org> >> wrote: >> >>> Is that a review comment? Is there a problem with the rendering? >>> >> >> Not a review comment just to make it easier for folks to read initially. >> >> >>> On Tue, Nov 17, 2015 at 11:51 AM, Mike Holmes <mike.holmes@linaro.org> >>> wrote: >>> >>>> This is the rendered doc >>>> >>>> http://people.linaro.org/~mike.holmes/output/users-guide.html >>>> >>>> On 17 November 2015 at 12:15, Bill Fischofer <bill.fischofer@linaro.org >>>> > wrote: >>>> >>>>> Add a basic ODP overview to the User's Guide, providing >>>>> an overview of ODP concepts, components, etc. Included >>>>> are a number of diagrams covering component structure >>>>> as well as packet RX, event scheduling, and packet TX >>>>> processing. >>>>> >>>>> Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> >>>>> --- >>>>> doc/users-guide/users-guide.adoc | 400 >>>>> ++++++++++++++++++++++++++++++++++++++- >>>>> 1 file changed, 398 insertions(+), 2 deletions(-) >>>>> >>>>> diff --git a/doc/users-guide/users-guide.adoc >>>>> b/doc/users-guide/users-guide.adoc >>>>> index 7f70046..cf77fa0 100644 >>>>> --- a/doc/users-guide/users-guide.adoc >>>>> +++ b/doc/users-guide/users-guide.adoc >>>>> @@ -8,7 +8,7 @@ OpenDataPlane (ODP) Users-Guide >>>>> Abstract >>>>> -------- >>>>> This document is intended to guide a new ODP application developer. >>>>> -Further details about ODP may be found at then >>>>> http://opendataplane.org[ODP] home page. >>>>> +Further details about ODP may be found at the >>>>> http://opendataplane.org[ODP] home page. >>>>> >>>>> .Overview of a system running ODP applications >>>>> image::../images/overview.png[align="center"] >>>>> @@ -16,6 +16,403 @@ image::../images/overview.png[align="center"] >>>>> ODP is an API specification that allows many implementations to >>>>> provide platform independence, automatic hardware acceleration and CPU >>>>> scaling to high performance networking applications. >>>>> This document describes how to write an application that can >>>>> successfully take advantage of the API. >>>>> >>>>> +:numbered: >>>>> +== Introduction == >>>>> +.OpenDataPlane Components >>>>> +image::../images/odp_components.png[align="center"] >>>>> + >>>>> +.The ODP API Specification >>>>> +ODP consists of three separate but related component parts. First, >>>>> ODP is an >>>>> +abstract API specification that describes a functional model for >>>>> +data plane applications. This specification covers many common data >>>>> plane >>>>> +application programming needs, such as the ability to receive, >>>>> manipulate, and >>>>> +transmit packet data, without specifying how these functions are >>>>> performed. This >>>>> +is quite intentional. It is precisely because ODP APIs do not have a >>>>> preferred >>>>> +embodiment that they permit innovation in how these functions can >>>>> +be realized on various platforms that offer implementations of ODP. >>>>> To achieve >>>>> +this goal, ODP APIs are described using abstract data types whose >>>>> definition >>>>> +is left up to the ODP implementer. For example, in ODP packets are >>>>> referenced >>>>> +by abstract handles of type +odp_packet_t+, and packet-related APIs >>>>> take >>>>> +arguments of this type. What an +odp_packet_t+ actually is is not >>>>> part of the >>>>> +ODP API specification--that is the responsibility of each ODP >>>>> implementation. >>>>> + >>>>> +.Summary: ODP API attributes: >>>>> +* Open Source, open contribution, BSD-3 licensed. >>>>> +* Vendor and platform neutral. >>>>> +* Application-centric. Covers functional needs of data plane >>>>> applications. >>>>> +* Ensures portability by specifying the functional behavior of ODP. >>>>> +* Defined jointly and openly by application writers and platform >>>>> implementers. >>>>> +* Archiected to be implementable on a wide range of platforms >>>>> efficiently >>>>> +* Sponsored, governed, and maintained by the Linaro Networking Group >>>>> (LNG) >>>>> + >>>>> +.ODP Implementations >>>>> +Second, ODP consists of multiple implementations of this API >>>>> specification, >>>>> +each tailored to a specific target platform. ODP implementations >>>>> determine >>>>> +how each ODP abstract type is represented on that platform and how >>>>> each ODP >>>>> +API is realized. On some platforms, ODP APIs will >>>>> +be realized using specialized instructions that accelerate the >>>>> functional >>>>> +behavior specified by the API. On others, hardware co-processing >>>>> engines may >>>>> +completely offload the API so that again it can be performed with >>>>> little or no >>>>> +involvement by a CPU. In all cases, the application sees the same >>>>> +functional behavior independent of how a given platform has chosen to >>>>> realize >>>>> +it. By allowing each platform the freedom to determine how best to >>>>> realize each >>>>> +API's specified functional behavior in an optimal manner, ODP permits >>>>> +applications written to its APIs to take full advantage of the unique >>>>> +capabilities of each platform without the application programmer >>>>> needing to >>>>> +have specialist knowledge of that platform or to be concerned with >>>>> how best >>>>> +to tune the application to a particular platform. This latter >>>>> consideration is >>>>> +particularly important in Network Function Virtualization (NFV) >>>>> environments >>>>> +where the application will run on a target platform chosen by someone >>>>> else. >>>>> + >>>>> +.Summary: ODP Implementation Characteristics >>>>> +* One size does not fit all--supporting multiple implementations >>>>> allows ODP >>>>> +to adapt to widely differing internals among platforms. >>>>> +* Anyone can create an ODP implementation tailored to their platform >>>>> +* Distribution and mainteinance of each implementation is as owner >>>>> wishes >>>>> + - Open source or closed source as business needs determine >>>>> + - Have independent release cycles and service streams >>>>> +* Allows HW and SW innovation in how ODP APIs are implemented on each >>>>> platform. >>>>> + >>>>> +.Reference Implementations >>>>> +To make it easy to get started with implementing ODP on a new >>>>> platform, ODP >>>>> +supplies a number of _reference implementations_ that can serve as a >>>>> +starting point. The two primary references implementations supplied >>>>> by ODP are >>>>> +*odp-linux* and *odp-dpdk* >>>>> + >>>>> +.odp-linux >>>>> +The *odp-linux* reference implementation is a pure SW implementation >>>>> of the >>>>> +ODP API that relies only on the Linux programming API. As a >>>>> functional model >>>>> +for ODP, it enables ODP to be bootstrapped easily to any platform that >>>>> +supports a Linux kernel. >>>>> + >>>>> +.odp-dpdk >>>>> +The *odp-dpdk* reference implementation is a pure SW implementation >>>>> of the >>>>> +ODP API that uses http://dpdk.org[DPDK] as a SW accelerator. In >>>>> particular, >>>>> +*odp-dpdk* offers superior I/O performance for systems that use NICs, >>>>> allowing >>>>> +ODP applications to take immediate full advantage of the various NIC >>>>> device >>>>> +drivers supported by DPDK. >>>>> + >>>>> +.Summary: ODP Reference Implementations >>>>> +* Open source, open contribution, BSD-3 licensed. >>>>> +* Provide easy bootstrapping of ODP onto new platforms >>>>> +* Implementers free to borrow or tailor code as needed for their >>>>> platform >>>>> +* Implementers retain full control over their implementations whether >>>>> or not >>>>> +they are derived from a reference implementation. >>>>> + >>>>> +.ODP Validation Test Suite >>>>> +Third, to enure consistency between different ODP implementations, ODP >>>>> +consists of a validation suite that verifies that any given >>>>> implementation of >>>>> +ODP faithfully provides the specified functional behavior of each ODP >>>>> API. >>>>> +As a separate open source component, the validation suite may be used >>>>> by >>>>> +application writers, system integrators, and platform providers alike >>>>> to >>>>> +confirm that any purported implementation of ODP does indeed conform >>>>> to the >>>>> +ODP API specification. >>>>> + >>>>> +.Summary: ODP Validation Test Suite >>>>> +* Synchronized with ODP API specification >>>>> +* Maintained and distributed by LNG >>>>> +* Open source, open contribution, BSD-3 licensed. >>>>> +* Key to ensuring application portability across all ODP >>>>> implementations >>>>> +* Tests that ODP implementations conform to the specified functional >>>>> behavior >>>>> +of ODP APIs. >>>>> +* Can be run at any time by users and vendors to validat >>>>> implementations >>>>> +od ODP. >>>>> + >>>>> +=== ODP API Specification Versioning === >>>>> +As an evolving standard, the ODP API specification is released under >>>>> an >>>>> +incrementing version number, and corresponding implementations of >>>>> ODP, as well >>>>> +as the validation suite that verifies API conformance, are linked to >>>>> this >>>>> +version number. ODP versions are specified using a stanard three-level >>>>> +number (major.minor.fixlevel) that are incremented according to the >>>>> degree of >>>>> +change the level represents. Increments to the fixlevel represent >>>>> clarification >>>>> +of the specification or other minor changes that do not affect either >>>>> the >>>>> +syntax or semantics of the specification. Such changes in the API >>>>> specification >>>>> +are expected to be rare. Increments to the minor level >>>>> +represent the introduction of new APIs or functional capabilities, or >>>>> changes >>>>> +to he specified syntax or functional behavior of APIs and thus may >>>>> require >>>>> +application source code changes. Such changes are well documented in >>>>> the >>>>> +release notes for each revision of the specification. Finally, >>>>> increments to >>>>> +the major level represent significant structural changes that most >>>>> likely >>>>> +require some level of application source code change, again as >>>>> documented in >>>>> +the release notes for that version. >>>>> + >>>>> +=== ODP Implementation Versioning === >>>>> +ODP implementations are free to use whatever release naming/numbering >>>>> +conventions they wish, as long as it is clear what level of the ODP >>>>> API a given >>>>> +release implements. A recommended convention is to use the same three >>>>> level >>>>> +numbering scheme where the major and minor numbers correspond to the >>>>> ODP API >>>>> +level and the fixlevel represents an implementation-defined service >>>>> level >>>>> +associated with that API level implementation. The LNG-supplied ODP >>>>> reference >>>>> +implementations follow this convention. >>>>> + >>>>> +=== ODP Validation Test Suite Versioning === >>>>> +The ODP validation test suite follows these same naming conventions. >>>>> The major >>>>> +and minor release numbers correspond to the ODP API level that the >>>>> suite >>>>> +validates and the fixlevel represents the service level of the >>>>> validation >>>>> +suite itself for that API level. >>>>> + >>>>> +=== ODP Design Goals === >>>>> +ODP has three primary goals that follow from its component structure. >>>>> The first >>>>> +is application portability across a wide range of platforms. These >>>>> platforms >>>>> +differ in terms of processor instruction set architecture, number and >>>>> types of >>>>> +application processing cores, memory oranization, as well as the >>>>> number and >>>>> +type of platform specific hardware acceleration and offload features >>>>> that >>>>> +are available. ODP applications can move from one conforming >>>>> implementation >>>>> +to another with at most a recompile. >>>>> + >>>>> +Second, ODP is designed to permit data plane applications to avail >>>>> themselves >>>>> +of platform-specific features, including specialized hardware >>>>> accelerators, >>>>> +without specialized programming. This is achieved by separating the >>>>> API >>>>> +specification from their implementation on individual platforms. >>>>> Since each >>>>> +platform implements each ODP API in a manner optimal to that platform, >>>>> +applications automatically gain the benefit of such optimizations >>>>> without the >>>>> +need for explicit programming. >>>>> + >>>>> +Third, ODP is designed to allow applications to scale out >>>>> automatically to >>>>> +support many core architectures. This is done using an event based >>>>> programming >>>>> +model that permits applications to be written to be independent of >>>>> the number >>>>> +of processing cores that are available to realize application >>>>> function. The >>>>> +result is that an application written to this model does not require >>>>> redesign >>>>> +as it scales from 4, to 40, to 400 cores. >>>>> + >>>>> +== Organization of this Document == >>>>> +This document is organized into several sections. The first presents >>>>> a high >>>>> +level overview of the ODP API component areas and their associated >>>>> abstract >>>>> +data types. This section introduces ODP APIs at a conceptual level. >>>>> +The second provides a tutorial on the programming model(s) >>>>> +supported by ODP, paying particular attention to the event model as >>>>> this >>>>> +represents the preferred structure for most ODP applications. This >>>>> section >>>>> +builds on the concepts introduced in the first section and shows how >>>>> ODP >>>>> +applications are structured to best realize the three ODP design goals >>>>> +mentioned earlier. The third section provides a more detailed >>>>> overview of >>>>> +the major ODP API components and is designed to serve as a companion >>>>> to the >>>>> +full reference specification for each API. The latter is intended to >>>>> be used >>>>> +by ODP application programmers, as well as implementers, to >>>>> understand the >>>>> +precise syntax and semantics of each API. >>>>> + >>>>> +== ODP API Concepts == >>>>> +ODP programs are built around several conceptual structures that every >>>>> +appliation programmer needs to be familiar with to use ODP >>>>> effectively. The >>>>> +main ODP concepts are: >>>>> +Thread, Event, Queue, Pool, Shared Memory, Buffer, Packet, PktIO, >>>>> Timer, >>>>> +and Synchronizer. >>>>> + >>>>> +=== Thread === >>>>> +The thread is the fundamental programming unit in ODP. ODP >>>>> applications are >>>>> +organized into a collection of threads that perform the work that the >>>>> +application is designed to do. ODP threads may or may not share >>>>> memory with >>>>> +other threads--that is up to the implementation. Threads come in two >>>>> "flavors": >>>>> +control and worker, that are represented by the abstract type >>>>> ++odp_thread_type_t+. >>>>> + >>>>> +A control thread is a supervisory thread that organizes >>>>> +the operation of worker threads. Worker threads, by contrast, exist to >>>>> +perform the main processing logic of the application and employ a run >>>>> to >>>>> +completion model. Worker threads, in particular, are intended to >>>>> operate on >>>>> +dedicated processing cores, especially in many core proessing >>>>> environments, >>>>> +however a given implementation may multitask multiple threads on a >>>>> single >>>>> +core if desired (typically on smaller and lower performance target >>>>> +environments). >>>>> + >>>>> +In addition to thread types, threads have associated _attributes_ >>>>> such as >>>>> +_thread mask_ and _scheduler group_ that determine where they can run >>>>> and >>>>> +the type of work that they can handle. These will be discussed in >>>>> greater >>>>> +detail later. >>>>> + >>>>> +=== Event === >>>>> +Events are what threads process to perform their work. Events can >>>>> represent >>>>> +new work, such as the arrival of a packet that needs to be processed, >>>>> or they >>>>> +can represent the completion of requests that have executed >>>>> asynchronously. >>>>> +Events can also represent notifications of the passage of time, or of >>>>> status >>>>> +changes in various components of interest to the application. Events >>>>> have an >>>>> +event type that describes what it represents. Threads can create new >>>>> events >>>>> +or consume events processed by them, or they can perform some >>>>> processing on >>>>> +an event and then pass it along to another component for further >>>>> processing. >>>>> +References to events are via handles of abstract type +odp_event_t+. >>>>> Cast >>>>> +functions are provided to convert these into specific handles of the >>>>> +appropriate type represented by the event. >>>>> + >>>>> +=== Queue === >>>>> +A queue is a message passing channel that holds events. Events can be >>>>> +added to a queue via enqueue operations or removed from a queue via >>>>> dequeue >>>>> +operations. The endpoints of a queue will vary depending on how it is >>>>> used. >>>>> +Queues come in two major types: polled and scheduled, which will be >>>>> +discussed in more detail when the event model is introduced. Queues >>>>> may also >>>>> +have an associated context, which represents a persistent state for >>>>> all >>>>> +events that make use of it. These states are what permit threads to >>>>> perform >>>>> +stateful processing on events as well as stateless processing. >>>>> + >>>>> +Queues are represented by handles of abstract type +odp_queue_t+. >>>>> + >>>>> +=== Pool === >>>>> +A pool is a shared memory area from which elements may be drawn. Pools >>>>> +represent the backing store for events, among other things. Pools are >>>>> +typically created and destroyed by the application during >>>>> initialization and >>>>> +termination, respectively, and then used during processing. Pools may >>>>> be >>>>> +used by ODP components exclusively, by applications exclusively, or >>>>> their >>>>> +use may be shared between the two. Pools have an associated type that >>>>> +characterizes the elements that they contain. The two most important >>>>> pool types >>>>> +are Buffer and Packet. >>>>> + >>>>> +Pools are represented by handles of abstract type +odp_pool_t+. >>>>> + >>>>> +=== Shared Memory === >>>>> +Shared memory represents raw blocks of storage that are sharable >>>>> between >>>>> +threads. They are the building blocks of pools but can be used >>>>> directly by >>>>> +ODP applications if desired. >>>>> + >>>>> +Shared memory is represented by handles of abstract type +odp_shm_t+. >>>>> + >>>>> +=== Buffer === >>>>> +A buffer is a fixed sized block of shared storage that is used by ODP >>>>> +components and/or applications to realize their function. Buffers >>>>> contain >>>>> +zero or more bytes of application data as well as system maintained >>>>> +metadata that provide information about the buffer, such as its size >>>>> or the >>>>> +pool it was allocated from. Metadata is an important ODP concept >>>>> because it >>>>> +allows for arbitrary amounts of side information to be associated >>>>> with an >>>>> +ODP object. Most ODP objects have assocaited metadata and this >>>>> metadata is >>>>> +manipulated via accessor functions that act as getters and setters for >>>>> +this information. Getter acces functions permit an application to read >>>>> +a metadata item, while setter access functions permit an application >>>>> to write >>>>> +a metadata item. Note that some metadata is inherently read only and >>>>> thus >>>>> +no setter is provided to manipulate it. When object have multiple >>>>> metadata >>>>> +items, each has its own associated getter and/or setter access >>>>> function to >>>>> +inspect or manipulate it. >>>>> + >>>>> +Buffers are represened by handles of abstract type +odp_buffer_t+. >>>>> + >>>>> +=== Packet === >>>>> +Packets are received and transmitted via I/O interfaces and represent >>>>> +the basic data that data plane applications manipulate. >>>>> +Packets are drawn from pools of type +ODP_POOL_PACKET+. >>>>> +Unlike buffers, which are simple objects, >>>>> +ODP packets have a rich set of semantics that permit their inspection >>>>> +and manipulation in complex ways to be described later. Packets also >>>>> support >>>>> +a rich set of metadata as well as user metadata. User metadata permits >>>>> +applications to associate an application-determined amount of side >>>>> information >>>>> +with each packet for its own use. >>>>> + >>>>> +Packets are represented by handles of abstract type +odp_packet_t+. >>>>> + >>>>> +=== PktIO === >>>>> +PktIO is how ODP represents I/O interfaces. A pktio object is a >>>>> logical >>>>> +port capable of receiving and/or transmitting packets. This may be >>>>> directly >>>>> +supported by the underlying platform as an integrated feature, >>>>> +or may represent a device attached via a PCIE or other bus. >>>>> + >>>>> +PktIOs are represented by handles of abstract type +odp_pktio_t+. >>>>> + >>>>> +=== Timer === >>>>> +Timers are how ODP applications measure and respond to the passage of >>>>> time. >>>>> +Timers are drawn from specialized pools called timer pools that have >>>>> their >>>>> +own abstract type (+odp_timer_pool_t+). Applications may have many >>>>> timers >>>>> +active at the same time and can set them to use either relative or >>>>> absolute >>>>> +time. When timers expire they create events of type +odp_timeout_t+, >>>>> which >>>>> +serve as notifications of timer expiration. >>>>> + >>>>> +=== Synchronizer === >>>>> +Multiple threads operating in parallel typically require various >>>>> +synchronization services to permit them to operate in a reliable and >>>>> +coordinated manner. ODP provides a rich set of locks, barriers, and >>>>> similar >>>>> +synchronization primitives, as well as abstract types for >>>>> representing various >>>>> +types of atomic variables. The ODP event model also makes use of >>>>> queues to >>>>> +avoid the need for explicit locking in many cases. This will be >>>>> discussed >>>>> +in the next section. >>>>> + >>>>> +== ODP Components == >>>>> +Building on ODP concepts, ODP offers several components that relate >>>>> to the >>>>> +flow of work through an ODP application. These include the Classifier, >>>>> +Scheduler, and Traffic Manager. These components relate to the three >>>>> +main stages of packet processing: Receive, Process, and Transmit. >>>>> + >>>>> +=== Classifier === >>>>> +The *Classifier* provides a suite of APIs that control packet receive >>>>> (RX) >>>>> +processing. >>>>> + >>>>> +.ODP Receive Processing with Classifier >>>>> +image::../images/odp_rx_processing.png[align="center"] >>>>> + >>>>> +The classifier provides two logically related services: >>>>> +[horizontal] >>>>> +Packet parsing:: Verifying and extracting structural information from >>>>> a >>>>> +received packet. >>>>> + >>>>> +Packet classification:: Applying *Pattern Matching Rules (PMRs)* to >>>>> the >>>>> +parsed results to assign an incoming packet to a *Class of Service >>>>> (CoS)*. >>>>> + >>>>> +Combined, these permit incoming packets to be sorted into *flows*, >>>>> which are >>>>> +logically related sequences of packets that share common processing >>>>> +requirements. While many data plane applications perform stateless >>>>> packet >>>>> +processing (_e.g.,_ for simple forwarding) others perform stateful >>>>> packet >>>>> +processing. Flows anchor state information relating to these groups >>>>> of >>>>> +packets. >>>>> + >>>>> +A CoS determines two variables for packets belonging to a flow: >>>>> +[list] >>>>> +* The pool that they will be stored in on receipt >>>>> +* The queue that they will be added to for processing >>>>> + >>>>> +The PMRs supported by ODP permit flow determination based on >>>>> combinations of >>>>> +packet field values (tuples). The main advantage of classification is >>>>> that on >>>>> +many platforms these functions are performed in hardware, meaning that >>>>> +classification occurs at line rate as packets are being received >>>>> without >>>>> +any explicit processing by the ODP application. >>>>> + >>>>> +Note that the use of the classifier is optional. Applications may >>>>> directly >>>>> +receive packets from a corresponding PktIO input queue via direct >>>>> polling >>>>> +if they choose. >>>>> + >>>>> +=== Scheduler === >>>>> +The *Scheduler* provides a suite of APIs that control scalabable event >>>>> +processing. >>>>> + >>>>> +.ODP Scheduler and Event Processing >>>>> +image::../images/odp_scheduling.png[align="center"] >>>>> + >>>>> +The Scheduler is responsible for selecting and dispatching one or >>>>> more events >>>>> +to a requesting thread. Event selection is based on several factors >>>>> involving >>>>> +both the queues containing schedulable events and the thread making an >>>>> ++odp_schedule()+ or +odp_schedule_multi()+ call. >>>>> + >>>>> +ODP queues have a _scheduling priority_ that determines how urgently >>>>> events >>>>> +on them should be processed relative to events contained in other >>>>> queues. >>>>> +Queues also have a _scheduler group id_ associated with them that >>>>> must match >>>>> +the associated scheduler group _thread mask_ of the thread calling the >>>>> +scheduler. This permits events to be grouped for processing into >>>>> classes and >>>>> +have threads that are dedicated to processing events from specified >>>>> classes. >>>>> +Threads can join and leave scheduler groups dynamically, permitting >>>>> easy >>>>> +application response to increases in demand. >>>>> + >>>>> +When a thread receives an event from the scheduler, it in turn can >>>>> invoke >>>>> +other processing engines via ODP APIs (_e.g.,_ crypto processing) that >>>>> +can operate asynchronously. When such processing is complete, the >>>>> result is >>>>> +that a *completion event* is added to a schedulable queue where it >>>>> can be >>>>> +scheduled back to a thread to continue processing with the results of >>>>> the >>>>> +requested asynchronous operation. >>>>> + >>>>> +Threads themselves can enqueue events to queues for downstream >>>>> processing >>>>> +by other threads, permitting flexibility in how applicaitions >>>>> structure >>>>> +themselves to maximize concurrency. >>>>> + >>>>> +=== Traffic Manager === >>>>> +The *Traffic Manager* provides a suite of APIs that control traffic >>>>> shaping and >>>>> +Quality of Service (QoS) processing for packet output. >>>>> + >>>>> +.ODP Transmit processing with Traffic Manager >>>>> +image::../images/odp_traffic_manager.png[align="center"] >>>>> + >>>>> +The final stage of packet processing is to transmit it. Here, >>>>> applications have >>>>> +several choices. As with RX processing, applications may send packets >>>>> +directly to PktIO TX queues for direct transmission. Often, however, >>>>> +applications need to perform traffic shaping and related >>>>> +*Quality of Service (QoS)* processing on the packets comprising a >>>>> flow as part >>>>> +of transmit processing. To handle this need, ODP provides a suite of >>>>> +*Traffic Manager* APIs that permit programmatic establishment of >>>>> arbiters, >>>>> +shapers, etc. that control output packet processing to achieve >>>>> desired QoS >>>>> +goals. Again, the advantage here is that on many platforms traffic >>>>> management >>>>> +functions are implemented in hardware, permitting transparent offload >>>>> of >>>>> +this work. >>>>> + >>>>> Glossary >>>>> -------- >>>>> [glossary] >>>>> @@ -28,7 +425,6 @@ odp_thread:: >>>>> event:: >>>>> An event is a notification that can be placed in a queue. >>>>> >>>>> -:numbered: >>>>> The include structure >>>>> --------------------- >>>>> Applications only include the 'include/odp.h file which includes the >>>>> 'platform/<implementation name>/include/plat' files to provide a complete >>>>> definition of the API on that platform. >>>>> -- >>>>> 2.1.4 >>>>> >>>>> _______________________________________________ >>>>> lng-odp mailing list >>>>> lng-odp@lists.linaro.org >>>>> https://lists.linaro.org/mailman/listinfo/lng-odp >>>>> >>>> >>>> >>>> >>>> -- >>>> Mike Holmes >>>> Technical Manager - Linaro Networking Group >>>> Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM >>>> SoCs >>>> >>>> >>>> >>> >> >> >> -- >> Mike Holmes >> Technical Manager - Linaro Networking Group >> Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM SoCs >> >> >> > -- Mike Holmes Technical Manager - Linaro Networking Group Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM SoCs
General comments to hash as we get docs moving. - I think this would be a lot easier to review in smaller pieces, a section at a time, for future patches I think we should do that. Images should go in first and it occurs to me we never add an image that is not used in a doc - maybe images and their use in the doc are a single patch ? If there are not too many objections I can live with these two patches as they are though. - We should pick one heading tag style, I have no preference but you and I differ currently in this doc so we should convert them all to the same style - We should pick a line style, again we differ. I put one sentence per line as we try to do with doxygen so that git diffs are per logical sentence rather than being slightly more random diffs of clumps of chars. The down side it in raw form the doc looks a little longer - but I think the diff ability makes it worth it. - Also this patch has introduced highlighting for ODP types and enums using "+" which I like, but is it not used consistently with the initial doc, If we are going to use this I think we should update the existing doc to use them also. On 17 November 2015 at 12:15, Bill Fischofer <bill.fischofer@linaro.org> wrote: > Add a basic ODP overview to the User's Guide, providing > an overview of ODP concepts, components, etc. Included > are a number of diagrams covering component structure > as well as packet RX, event scheduling, and packet TX > processing. > > Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> > --- > doc/users-guide/users-guide.adoc | 400 > ++++++++++++++++++++++++++++++++++++++- > 1 file changed, 398 insertions(+), 2 deletions(-) > > diff --git a/doc/users-guide/users-guide.adoc > b/doc/users-guide/users-guide.adoc > index 7f70046..cf77fa0 100644 > --- a/doc/users-guide/users-guide.adoc > +++ b/doc/users-guide/users-guide.adoc > @@ -8,7 +8,7 @@ OpenDataPlane (ODP) Users-Guide > Abstract > -------- > This document is intended to guide a new ODP application developer. > -Further details about ODP may be found at then http://opendataplane.org[ODP] > home page. > +Further details about ODP may be found at the http://opendataplane.org[ODP] > home page. > > .Overview of a system running ODP applications > image::../images/overview.png[align="center"] > @@ -16,6 +16,403 @@ image::../images/overview.png[align="center"] > ODP is an API specification that allows many implementations to provide > platform independence, automatic hardware acceleration and CPU scaling to > high performance networking applications. > This document describes how to write an application that can successfully > take advantage of the API. > > +:numbered: > +== Introduction == > +.OpenDataPlane Components > +image::../images/odp_components.png[align="center"] > + > +.The ODP API Specification > +ODP consists of three separate but related component parts. First, ODP is > an > +abstract API specification that describes a functional model for > +data plane applications. This specification covers many common data plane > +application programming needs, such as the ability to receive, > manipulate, and > +transmit packet data, without specifying how these functions are > performed. This > +is quite intentional. It is precisely because ODP APIs do not have a > preferred > +embodiment that they permit innovation in how these functions can > +be realized on various platforms that offer implementations of ODP. To > achieve > +this goal, ODP APIs are described using abstract data types whose > definition > +is left up to the ODP implementer. For example, in ODP packets are > referenced > +by abstract handles of type +odp_packet_t+, and packet-related APIs take > +arguments of this type. What an +odp_packet_t+ actually is is not part of > the > +ODP API specification--that is the responsibility of each ODP > implementation. > + > +.Summary: ODP API attributes: > +* Open Source, open contribution, BSD-3 licensed. > +* Vendor and platform neutral. > +* Application-centric. Covers functional needs of data plane > applications. > +* Ensures portability by specifying the functional behavior of ODP. > +* Defined jointly and openly by application writers and platform > implementers. > +* Archiected to be implementable on a wide range of platforms efficiently > +* Sponsored, governed, and maintained by the Linaro Networking Group (LNG) > + > +.ODP Implementations > +Second, ODP consists of multiple implementations of this API > specification, > +each tailored to a specific target platform. ODP implementations determine > +how each ODP abstract type is represented on that platform and how each > ODP > +API is realized. On some platforms, ODP APIs will > +be realized using specialized instructions that accelerate the functional > +behavior specified by the API. On others, hardware co-processing engines > may > +completely offload the API so that again it can be performed with little > or no > +involvement by a CPU. In all cases, the application sees the same > +functional behavior independent of how a given platform has chosen to > realize > +it. By allowing each platform the freedom to determine how best to > realize each > +API's specified functional behavior in an optimal manner, ODP permits > +applications written to its APIs to take full advantage of the unique > +capabilities of each platform without the application programmer needing > to > +have specialist knowledge of that platform or to be concerned with how > best > +to tune the application to a particular platform. This latter > consideration is > +particularly important in Network Function Virtualization (NFV) > environments > +where the application will run on a target platform chosen by someone > else. > + > +.Summary: ODP Implementation Characteristics > +* One size does not fit all--supporting multiple implementations allows > ODP > +to adapt to widely differing internals among platforms. > +* Anyone can create an ODP implementation tailored to their platform > +* Distribution and mainteinance maintenance of each implementation is as owner wishes > + - Open source or closed source as business needs determine > + - Have independent release cycles and service streams > +* Allows HW and SW innovation in how ODP APIs are implemented on each > platform. > + > +.Reference Implementations > +To make it easy to get started with implementing ODP on a new platform, > ODP > +supplies a number of _reference implementations_ that can serve as a > +starting point. The two primary references implementations supplied by > ODP are > +*odp-linux* and *odp-dpdk* > + > +.odp-linux > +The *odp-linux* reference implementation is a pure SW implementation of > the > +ODP API that relies only on the Linux programming API. As a functional > model > +for ODP, it enables ODP to be bootstrapped easily to any platform that > +supports a Linux kernel. > + > +.odp-dpdk > +The *odp-dpdk* reference implementation is a pure SW implementation of the > +ODP API that uses http://dpdk.org[DPDK] as a SW accelerator. In > particular, > +*odp-dpdk* offers superior I/O performance for systems that use NICs, > allowing > +ODP applications to take immediate full advantage of the various NIC > device > +drivers supported by DPDK. > + > +.Summary: ODP Reference Implementations > +* Open source, open contribution, BSD-3 licensed. > +* Provide easy bootstrapping of ODP onto new platforms > +* Implementers free to borrow or tailor code as needed for their platform > +* Implementers retain full control over their implementations whether or > not > +they are derived from a reference implementation. > + > +.ODP Validation Test Suite > +Third, to enure ensure > consistency between different ODP implementations, ODP > +consists of a validation suite that verifies that any given > implementation of > +ODP faithfully provides the specified functional behavior of each ODP API. > +As a separate open source component, the validation suite may be used by > +application writers, system integrators, and platform providers alike to > +confirm that any purported implementation of ODP does indeed conform to > the > +ODP API specification. > + > +.Summary: ODP Validation Test Suite > +* Synchronized with ODP API specification > +* Maintained and distributed by LNG > +* Open source, open contribution, BSD-3 licensed. > +* Key to ensuring application portability across all ODP implementations > +* Tests that ODP implementations conform to the specified functional > behavior > +of ODP APIs. > +* Can be run at any time by users and vendors to validat implementations > +od ODP. > + > +=== ODP API Specification Versioning === > +As an evolving standard, the ODP API specification is released under an > +incrementing version number, and corresponding implementations of ODP, as > well > +as the validation suite that verifies API conformance, are linked to this > +version number. ODP versions are specified using a stanard standard > three-level > +number (major.minor.fixlevel change to fix-level every where since it is a label we are making up and not a real word > ) that are incremented according to the degree of > +change the level represents. Increments to the fixlevel represent clarification > +of the specification or other minor changes that do not affect either the > +syntax or semantics of the specification. Such changes in the API > specification > +are expected to be rare. Increments to the minor level > +represent the introduction of new APIs or functional capabilities, or > changes > +to he specified syntax or functional behavior of APIs and thus may require > +application source code changes. Such changes are well documented in the > +release notes for each revision of the specification. Finally, increments > to > +the major level represent significant structural changes that most likely > +require some level of application source code change, again as documented > in > +the release notes for that version. > + > +=== ODP Implementation Versioning === > +ODP implementations are free to use whatever release naming/numbering > +conventions they wish, as long as it is clear what level of the ODP API a > given > +release implements. A recommended convention is to use the same three > level > +numbering scheme where the major and minor numbers correspond to the ODP > API > +level and the fixlevel represents an implementation-defined service level > +associated with that API level implementation. The LNG-supplied ODP > reference > +implementations follow this convention. > + > +=== ODP Validation Test Suite Versioning === > +The ODP validation test suite follows these same naming conventions. The > major > +and minor release numbers correspond to the ODP API level that the suite > +validates and the fixlevel represents the service level of the validation > +suite itself for that API level. > + > +=== ODP Design Goals === > +ODP has three primary goals that follow from its component structure. The > first > +is application portability across a wide range of platforms. These > platforms > +differ in terms of processor instruction set architecture, number and > types of > +application processing cores, memory oranization organization > , as well as the number and > +type of platform specific hardware acceleration and offload features that > +are available. ODP applications can move from one conforming > implementation > +to another with at most a recompile. > + > +Second, ODP is designed to permit data plane applications to avail > themselves > +of platform-specific features, including specialized hardware > accelerators, > +without specialized programming. This is achieved by separating the API > +specification from their implementation on individual platforms. Since > each > +platform implements each ODP API in a manner optimal to that platform, > +applications automatically gain the benefit of such optimizations without > the > +need for explicit programming. > + > +Third, ODP is designed to allow applications to scale out automatically to > +support many core architectures. This is done using an event based > programming > +model that permits applications to be written to be independent of the > number > +of processing cores that are available to realize application function. > The > +result is that an application written to this model does not require > redesign > +as it scales from 4, to 40, to 400 cores. > + > +== Organization of this Document == > +This document is organized into several sections. The first presents a > high > +level overview of the ODP API component areas and their associated > abstract > +data types. This section introduces ODP APIs at a conceptual level. > +The second provides a tutorial on the programming model(s) > +supported by ODP, paying particular attention to the event model as this > +represents the preferred structure for most ODP applications. This section > +builds on the concepts introduced in the first section and shows how ODP > +applications are structured to best realize the three ODP design goals > +mentioned earlier. The third section provides a more detailed overview of > +the major ODP API components and is designed to serve as a companion to > the > +full reference specification for each API. The latter is intended to be > used > +by ODP application programmers, as well as implementers, to understand the > +precise syntax and semantics of each API. > + > +== ODP API Concepts == > +ODP programs are built around several conceptual structures that every > +appliation application > programmer needs to be familiar with to use ODP effectively. The > +main ODP concepts are: > +Thread, Event, Queue, Pool, Shared Memory, Buffer, Packet, PktIO, Timer, > +and Synchronizer. > Would be better as a bullet list when viewed to get your head around the "big picture" of what we mean by concepts becasue we define components later on and the lists are different but the ideas are very similar In the API doc and in the tests these are called Modules, I think they should correlate 1:1 to headings here > + > +=== Thread === > +The thread is the fundamental programming unit in ODP. ODP applications > are > +organized into a collection of threads that perform the work that the > +application is designed to do. ODP threads may or may not share memory > with > +other threads--that is up to the implementation. Threads come in two > "flavors": > +control and worker, that are represented by the abstract type > ++odp_thread_type_t+. > + > +A control thread is a supervisory thread that organizes > +the operation of worker threads. Worker threads, by contrast, exist to > +perform the main processing logic of the application and employ a run to > +completion model. Worker threads, in particular, are intended to operate > on > +dedicated processing cores, especially in many core proessing processing > environments, > +however a given implementation may multitask multiple threads on a single > +core if desired (typically on smaller and lower performance target > +environments). > + > +In addition to thread types, threads have associated _attributes_ such as > +_thread mask_ and _scheduler group_ that determine where they can run and > +the type of work that they can handle. These will be discussed in greater > +detail later. > + > +=== Event === > +Events are what threads process to perform their work. Events can > represent > +new work, such as the arrival of a packet that needs to be processed, or > they > +can represent the completion of requests that have executed > asynchronously. > +Events can also represent notifications of the passage of time, or of > status > +changes in various components of interest to the application. Events have > an > +event type that describes what it represents. Threads can create new > events > +or consume events processed by them, or they can perform some processing > on > +an event and then pass it along to another component for further > processing. > +References to events are via handles of abstract type +odp_event_t+. Cast > +functions are provided to convert these into specific handles of the > +appropriate type represented by the event. > + > +=== Queue === > +A queue is a message passing channel that holds events. Events can be > +added to a queue via enqueue operations or removed from a queue via > dequeue > +operations. The endpoints of a queue will vary depending on how it is > used. > +Queues come in two major types: polled and scheduled, which will be > +discussed in more detail when the event model is introduced. Queues may > also > +have an associated context, which represents a persistent state for all > +events that make use of it. These states are what permit threads to > perform > +stateful processing on events as well as stateless processing. > + > +Queues are represented by handles of abstract type +odp_queue_t+. > + > +=== Pool === > +A pool is a shared memory area from which elements may be drawn. Pools > +represent the backing store for events, among other things. Pools are > +typically created and destroyed by the application during initialization > and > +termination, respectively, and then used during processing. Pools may be > +used by ODP components exclusively, by applications exclusively, or their > +use may be shared between the two. Pools have an associated type that > +characterizes the elements that they contain. The two most important pool > types > +are Buffer and Packet. > + > +Pools are represented by handles of abstract type +odp_pool_t+. > + > +=== Shared Memory === > +Shared memory represents raw blocks of storage that are sharable between > +threads. They are the building blocks of pools but can be used directly by > +ODP applications if desired. > + > +Shared memory is represented by handles of abstract type +odp_shm_t+. > + > +=== Buffer === > +A buffer is a fixed sized block of shared storage that is used by ODP > +components and/or applications to realize their function. Buffers contain > +zero or more bytes of application data as well as system maintained > +metadata that provide information about the buffer, such as its size or > the > +pool it was allocated from. Metadata is an important ODP concept because > it > +allows for arbitrary amounts of side information to be associated with an > +ODP object. Most ODP objects have assocaited metadata and this metadata is > +manipulated via accessor functions that act as getters and setters for > +this information. Getter acces functions permit an application to read > +a metadata item, while setter access functions permit an application to > write > +a metadata item. Note that some metadata is inherently read only and thus > +no setter is provided to manipulate it. When object have multiple > metadata > +items, each has its own associated getter and/or setter access function to +inspect or manipulate it. > + > +Buffers are represened represented > by handles of abstract type +odp_buffer_t+. > + > +=== Packet === > +Packets are received and transmitted via I/O interfaces and represent > +the basic data that data plane applications manipulate. > +Packets are drawn from pools of type +ODP_POOL_PACKET+. > +Unlike buffers, which are simple objects, > +ODP packets have a rich set of semantics that permit their inspection > +and manipulation in complex ways to be described later. Packets also > support > +a rich set of metadata as well as user metadata. User metadata permits > +applications to associate an application-determined amount of side > information > +with each packet for its own use. > + > +Packets are represented by handles of abstract type +odp_packet_t+. > + > +=== PktIO === > +PktIO is how ODP represents I/O interfaces. A pktio object is a logical > +port capable of receiving and/or transmitting packets. This may be > directly > +supported by the underlying platform as an integrated feature, > +or may represent a device attached via a PCIE or other bus. > + > +PktIOs are represented by handles of abstract type +odp_pktio_t+. > + > +=== Timer === > +Timers are how ODP applications measure and respond to the passage of > time. > +Timers are drawn from specialized pools called timer pools that have their > +own abstract type (+odp_timer_pool_t+). Applications may have many timers > +active at the same time and can set them to use either relative or > absolute > +time. When timers expire they create events of type +odp_timeout_t+, which > +serve as notifications of timer expiration. > + > +=== Synchronizer === > +Multiple threads operating in parallel typically require various > +synchronization services to permit them to operate in a reliable and > +coordinated manner. ODP provides a rich set of locks, barriers, and > similar > +synchronization primitives, as well as abstract types for representing > various > +types of atomic variables. The ODP event model also makes use of queues to > +avoid the need for explicit locking in many cases. This will be discussed > +in the next section. > + > +== ODP Components == +Building on ODP concepts, ODP offers several components that relate to the > +flow of work through an ODP application. These include the Classifier, > +Scheduler, and Traffic Manager. These components relate to the three > +main stages of packet processing: Receive, Process, and Transmit. > I think a diagram showing the end to end overview with PKTIO->classifier->scheduler->TM->Pktio + loopback might help establish what we define as the set of flow components before breaking to to the details of a specific component. + > +=== Classifier === > +The *Classifier* provides a suite of APIs that control packet receive (RX) > +processing. > + > +.ODP Receive Processing with Classifier > +image::../images/odp_rx_processing.png[align="center"] > + > +The classifier provides two logically related services: > +[horizontal] > +Packet parsing:: Verifying and extracting structural information from a > +received packet. > + > +Packet classification:: Applying *Pattern Matching Rules (PMRs)* to the > +parsed results to assign an incoming packet to a *Class of Service (CoS)*. > + > +Combined, these permit incoming packets to be sorted into *flows*, which > are > +logically related sequences of packets that share common processing > +requirements. While many data plane applications perform stateless packet > +processing (_e.g.,_ for simple forwarding) others perform stateful packet > +processing. Flows anchor state information relating to these groups of > +packets. > + > +A CoS determines two variables for packets belonging to a flow: > +[list] > +* The pool that they will be stored in on receipt > +* The queue that they will be added to for processing > + > +The PMRs supported by ODP permit flow determination based on combinations > of > +packet field values (tuples). The main advantage of classification is > that on > +many platforms these functions are performed in hardware, meaning that > +classification occurs at line rate as packets are being received without > +any explicit processing by the ODP application. > + > +Note that the use of the classifier is optional. Applications may > directly > +receive packets from a corresponding PktIO input queue via direct polling > +if they choose. > + > +=== Scheduler === > +The *Scheduler* provides a suite of APIs that control scalabable event > +processing. > + > +.ODP Scheduler and Event Processing > +image::../images/odp_scheduling.png[align="center"] > + > +The Scheduler is responsible for selecting and dispatching one or more > events > +to a requesting thread. Event selection is based on several factors > involving > +both the queues containing schedulable events and the thread making an > ++odp_schedule()+ or +odp_schedule_multi()+ call. > + > +ODP queues have a _scheduling priority_ that determines how urgently > events > +on them should be processed relative to events contained in other queues. > +Queues also have a _scheduler group id_ associated with them that must > match > +the associated scheduler group _thread mask_ of the thread calling the > +scheduler. This permits events to be grouped for processing into classes > and > +have threads that are dedicated to processing events from specified > classes. > +Threads can join and leave scheduler groups dynamically, permitting easy > +application response to increases in demand. > + > +When a thread receives an event from the scheduler, it in turn can invoke > +other processing engines via ODP APIs (_e.g.,_ crypto processing) that > +can operate asynchronously. When such processing is complete, the result > is > +that a *completion event* is added to a schedulable queue where it can be > +scheduled back to a thread to continue processing with the results of the > +requested asynchronous operation. > + > +Threads themselves can enqueue en-queue ? my checker says this > events to queues for downstream processing > +by other threads, permitting flexibility in how applicaitions applications > structure > +themselves to maximize concurrency. > + > +=== Traffic Manager === > +The *Traffic Manager* provides a suite of APIs that control traffic > shaping and > +Quality of Service (QoS) processing for packet output. > + > +.ODP Transmit processing with Traffic Manager > +image::../images/odp_traffic_manager.png[align="center"] > + > +The final stage of packet processing is to transmit it. Here, > applications have > +several choices. As with RX processing, applications may send packets > +directly to PktIO TX queues for direct transmission. Often, however, > +applications need to perform traffic shaping and related > +*Quality of Service (QoS)* processing on the packets comprising a flow as > part > +of transmit processing. To handle this need, ODP provides a suite of > +*Traffic Manager* APIs that permit programmatic establishment of arbiters, > +shapers, etc. that control output packet processing to achieve desired QoS > +goals. Again, the advantage here is that on many platforms traffic > management > +functions are implemented in hardware, permitting transparent offload of > +this work. > + > Glossary > -------- > [glossary] > @@ -28,7 +425,6 @@ odp_thread:: > event:: > An event is a notification that can be placed in a queue. > I think we should add some of the terms introduced here, not sure if all of these should be considered domain knowledge or not in a user-guide, some are ODP specific in meaning scheduler engines ( co-processing ) completion event platform flows components concurrency traffic arbiter shaper forwarding stateful parsing metadata getter setter > -:numbered: > The include structure > --------------------- > Applications only include the 'include/odp.h file which includes the > 'platform/<implementation name>/include/plat' files to provide a complete > definition of the API on that platform. > -- > 2.1.4 > > _______________________________________________ > lng-odp mailing list > lng-odp@lists.linaro.org > https://lists.linaro.org/mailman/listinfo/lng-odp > Apply-and build caught a couple of things PATCH_DIR=~/incoming/bill ./apply-and-build.sh Using patch: lng-odp_PATCHv2_2-2_doc_userguide_add_baseline_overview_to_document.mbox Trying to apply patch Patch applied WARNING: 'assocaited' may be misspelled - perhaps 'associated'? #288: FILE: doc/users-guide/users-guide.adoc:273: +ODP object. Most ODP objects have assocaited metadata and this metadata is WARNING: 'acces' may be misspelled - perhaps 'access'? #290: FILE: doc/users-guide/users-guide.adoc:275: +this information. Getter acces functions permit an application to read -- Mike Holmes Technical Manager - Linaro Networking Group Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM SoCs
Thanks for the spelling corrections. v4 submitted (v3 still had a typo) that corrects these and others and adds the list you suggested. Other comments inline: On Tue, Nov 17, 2015 at 12:52 PM, Mike Holmes <mike.holmes@linaro.org> wrote: > General comments to hash as we get docs moving. > > - I think this would be a lot easier to review in smaller pieces, a > section at a time, for future patches I think we should do that. Images > should go in first and it occurs to me we never add an image that is not > used in a doc - maybe images and their use in the doc are a single patch ? > If there are not too many objections I can live with these two patches as > they are though. > > In general I agree, which is why I stopped here. I don't think we want to be too strict here, however. This isn't code and isn't subject to bisection. Images should be in a separate patch because they are large and non-reviewable in unrendered form since they are binary. Putting them in with the text portions just adds clutter for reviewers. > - > - We should pick one heading tag style, I have no preference but you > and I differ currently in this doc so we should convert them all to the > same style > > Agreed. I prefer the == style since it's very clear that = is 1st level header == is 2nd etc. Using -, +, ~, etc. it's not obvious which is what level. I can submit a patch to standardize on this style. > > - > - We should pick a line style, again we differ. I put one sentence per > line as we try to do with doxygen so that git diffs are per logical > sentence rather than being slightly more random diffs of clumps of chars. > The down side it in raw form the doc looks a little longer - but I think > the diff ability makes it worth it. > > Strongly disagree. The goal here is to make things readable for the human writer and reviewer. Keeping to natural line breaks and our checkpatch 80 char length guidelines do that. Changes are to improve readability, not to minimize the number of characters changed in some diff, which is a completely arbitrary criteria that bears little relevance to how documentation is intended to be used. The whole point of using asciidoc is to make the document source humanly readable. That trumps all else. > > - > - Also this patch has introduced highlighting for ODP types and enums > using "+" which I like, but is it not used consistently with the initial > doc, If we are going to use this I think we should update the existing doc > to use them also. > > Again, additional patches can revise the existing portions of the doc to standardize. > > - > > On 17 November 2015 at 12:15, Bill Fischofer <bill.fischofer@linaro.org> > wrote: > >> Add a basic ODP overview to the User's Guide, providing >> an overview of ODP concepts, components, etc. Included >> are a number of diagrams covering component structure >> as well as packet RX, event scheduling, and packet TX >> processing. >> >> Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> >> --- >> doc/users-guide/users-guide.adoc | 400 >> ++++++++++++++++++++++++++++++++++++++- >> 1 file changed, 398 insertions(+), 2 deletions(-) >> >> diff --git a/doc/users-guide/users-guide.adoc >> b/doc/users-guide/users-guide.adoc >> index 7f70046..cf77fa0 100644 >> --- a/doc/users-guide/users-guide.adoc >> +++ b/doc/users-guide/users-guide.adoc >> @@ -8,7 +8,7 @@ OpenDataPlane (ODP) Users-Guide >> Abstract >> -------- >> This document is intended to guide a new ODP application developer. >> -Further details about ODP may be found at then http://opendataplane.org[ODP] >> home page. >> +Further details about ODP may be found at the http://opendataplane.org[ODP] >> home page. >> >> .Overview of a system running ODP applications >> image::../images/overview.png[align="center"] >> @@ -16,6 +16,403 @@ image::../images/overview.png[align="center"] >> ODP is an API specification that allows many implementations to provide >> platform independence, automatic hardware acceleration and CPU scaling to >> high performance networking applications. >> This document describes how to write an application that can >> successfully take advantage of the API. >> >> +:numbered: >> +== Introduction == >> +.OpenDataPlane Components >> +image::../images/odp_components.png[align="center"] >> + >> +.The ODP API Specification >> +ODP consists of three separate but related component parts. First, ODP >> is an >> +abstract API specification that describes a functional model for >> +data plane applications. This specification covers many common data plane >> +application programming needs, such as the ability to receive, >> manipulate, and >> +transmit packet data, without specifying how these functions are >> performed. This >> +is quite intentional. It is precisely because ODP APIs do not have a >> preferred >> +embodiment that they permit innovation in how these functions can >> +be realized on various platforms that offer implementations of ODP. To >> achieve >> +this goal, ODP APIs are described using abstract data types whose >> definition >> +is left up to the ODP implementer. For example, in ODP packets are >> referenced >> +by abstract handles of type +odp_packet_t+, and packet-related APIs take >> +arguments of this type. What an +odp_packet_t+ actually is is not part >> of the >> +ODP API specification--that is the responsibility of each ODP >> implementation. >> + >> +.Summary: ODP API attributes: >> +* Open Source, open contribution, BSD-3 licensed. >> +* Vendor and platform neutral. >> +* Application-centric. Covers functional needs of data plane >> applications. >> +* Ensures portability by specifying the functional behavior of ODP. >> +* Defined jointly and openly by application writers and platform >> implementers. >> +* Archiected > > to be implementable on a wide range of platforms efficiently >> +* Sponsored, governed, and maintained by the Linaro Networking Group >> (LNG) >> + >> +.ODP Implementations >> +Second, ODP consists of multiple implementations of this API >> specification, >> +each tailored to a specific target platform. ODP implementations >> determine >> +how each ODP abstract type is represented on that platform and how each >> ODP >> +API is realized. On some platforms, ODP APIs will >> +be realized using specialized instructions that accelerate the functional >> +behavior specified by the API. On others, hardware co-processing engines >> may >> +completely offload the API so that again it can be performed with little >> or no >> +involvement by a CPU. In all cases, the application sees the same >> +functional behavior independent of how a given platform has chosen to >> realize >> +it. By allowing each platform the freedom to determine how best to >> realize each >> +API's specified functional behavior in an optimal manner, ODP permits >> +applications written to its APIs to take full advantage of the unique >> +capabilities of each platform without the application programmer needing >> to >> +have specialist knowledge of that platform or to be concerned with how >> best >> +to tune the application to a particular platform. This latter >> consideration is >> +particularly important in Network Function Virtualization (NFV) >> environments >> +where the application will run on a target platform chosen by someone >> else. >> + >> +.Summary: ODP Implementation Characteristics >> +* One size does not fit all--supporting multiple implementations allows >> ODP >> +to adapt to widely differing internals among platforms. >> +* Anyone can create an ODP implementation tailored to their platform >> +* Distribution and mainteinance > > > maintenance > > of each implementation is as owner wishes >> + - Open source or closed source as business needs determine >> + - Have independent release cycles and service streams >> +* Allows HW and SW innovation in how ODP APIs are implemented on each >> platform. >> + >> +.Reference Implementations >> +To make it easy to get started with implementing ODP on a new platform, >> ODP >> +supplies a number of _reference implementations_ that can serve as a >> +starting point. The two primary references implementations supplied by >> ODP are >> +*odp-linux* and *odp-dpdk* >> + >> +.odp-linux >> +The *odp-linux* reference implementation is a pure SW implementation of >> the >> +ODP API that relies only on the Linux programming API. As a functional >> model >> +for ODP, it enables ODP to be bootstrapped easily to any platform that >> +supports a Linux kernel. >> + >> +.odp-dpdk >> +The *odp-dpdk* reference implementation is a pure SW implementation of >> the >> +ODP API that uses http://dpdk.org[DPDK] as a SW accelerator. In >> particular, >> +*odp-dpdk* offers superior I/O performance for systems that use NICs, >> allowing >> +ODP applications to take immediate full advantage of the various NIC >> device >> +drivers supported by DPDK. >> + >> +.Summary: ODP Reference Implementations >> +* Open source, open contribution, BSD-3 licensed. >> +* Provide easy bootstrapping of ODP onto new platforms >> +* Implementers free to borrow or tailor code as needed for their platform >> +* Implementers retain full control over their implementations whether or >> not >> +they are derived from a reference implementation. >> + >> +.ODP Validation Test Suite >> +Third, to enure > > > ensure > > >> consistency between different ODP implementations, ODP >> +consists of a validation suite that verifies that any given >> implementation of >> +ODP faithfully provides the specified functional behavior of each ODP >> API. >> +As a separate open source component, the validation suite may be used by >> +application writers, system integrators, > > and platform providers alike to >> +confirm that any purported implementation of ODP does indeed conform to >> the >> +ODP API specification. >> + >> +.Summary: ODP Validation Test Suite >> +* Synchronized with ODP API specification >> +* Maintained and distributed by LNG >> +* Open source, open contribution, BSD-3 licensed. >> +* Key to ensuring application portability across all ODP implementations >> +* Tests that ODP implementations conform to the specified functional >> behavior >> +of ODP APIs. >> +* Can be run at any time by users and vendors to validat implementations >> +od ODP. >> + >> +=== ODP API Specification Versioning === >> +As an evolving standard, the ODP API specification is released under an >> +incrementing version number, and corresponding implementations of ODP, >> as well >> +as the validation suite that verifies API conformance, are linked to this >> +version number. ODP versions are specified using a stanard > > > standard > > >> three-level >> +number (major.minor.fixlevel > > > change to fix-level every where since it is a label we are making up and > not a real word > > > >> ) that are incremented according to the degree of >> +change the level represents. Increments to the fixlevel > > represent clarification >> +of the specification or other minor changes that do not affect either the >> +syntax or semantics of the specification. Such changes in the API >> specification >> +are expected to be rare. Increments to the minor level >> +represent the introduction of new APIs or functional capabilities, or >> changes >> +to he specified syntax or functional behavior of APIs and thus may >> require >> +application source code changes. Such changes are well documented in the >> +release notes for each revision of the specification. Finally, >> increments to >> +the major level represent significant structural changes that most likely >> +require some level of application source code change, again as >> documented in >> +the release notes for that version. >> + >> +=== ODP Implementation Versioning === >> +ODP implementations are free to use whatever release naming/numbering >> +conventions they wish, as long as it is clear what level of the ODP API >> a given >> +release implements. A recommended convention is to use the same three >> level >> +numbering scheme where the major and minor numbers correspond to the ODP >> API >> +level and the fixlevel represents an implementation-defined service level >> +associated with that API level implementation. The LNG-supplied ODP >> reference >> +implementations follow this convention. >> + >> +=== ODP Validation Test Suite Versioning === >> +The ODP validation test suite follows these same naming conventions. The >> major >> +and minor release numbers correspond to the ODP API level that the suite >> +validates and the fixlevel represents the service level of the validation >> +suite itself for that API level. >> + >> +=== ODP Design Goals === >> +ODP has three primary goals that follow from its component structure. >> The first >> +is application portability across a wide range of platforms. These >> platforms >> +differ in terms of processor instruction set architecture, number and >> types of >> +application processing cores, memory oranization > > > organization > > >> , as well as the number and >> +type of platform specific hardware acceleration and offload features that >> +are available. ODP applications can move from one conforming >> implementation >> +to another with at most a recompile. >> + >> +Second, ODP is designed to permit data plane applications to avail >> themselves >> +of platform-specific features, including specialized hardware >> accelerators, >> +without specialized programming. This is achieved by separating the API >> +specification from their implementation on individual platforms. Since >> each >> +platform implements each ODP API in a manner optimal to that platform, >> +applications automatically gain the benefit of such optimizations >> without the >> +need for explicit programming. >> + >> +Third, ODP is designed to allow applications to scale out automatically >> to >> +support many core architectures. This is done using an event based >> programming >> +model that permits applications to be written to be independent of the >> number >> +of processing cores that are available to realize application function. >> The >> +result is that an application written to this model does not require >> redesign >> +as it scales from 4, to 40, to 400 cores. >> + >> +== Organization of this Document == >> +This document is organized into several sections. The first presents a >> high >> +level overview of the ODP API component areas and their associated >> abstract >> +data types. This section introduces ODP APIs at a conceptual level. >> +The second provides a tutorial on the programming model(s) >> +supported by ODP, paying particular attention to the event model as this >> +represents the preferred structure for most ODP applications. This >> section >> +builds on the concepts introduced in the first section and shows how ODP >> +applications are structured to best realize the three ODP design goals >> +mentioned earlier. The third section provides a more detailed overview of >> +the major ODP API components and is designed to serve as a companion to >> the >> +full reference specification for each API. The latter is intended to be >> used >> +by ODP application programmers, as well as implementers, to understand >> the >> +precise syntax and semantics of each API. >> + >> +== ODP API Concepts == >> +ODP programs are built around several conceptual structures that every >> +appliation > > > application > > >> programmer needs to be familiar with to use ODP effectively. The >> +main ODP concepts are: >> +Thread, Event, Queue, Pool, Shared Memory, Buffer, Packet, PktIO, Timer, >> +and Synchronizer. >> > > Would be better as a bullet list when viewed to get your head around the > "big picture" of what we mean by concepts becasue we define components > later on and the lists are different but the ideas are very similar > > In the API doc and in the tests these are called Modules, I think they > should correlate 1:1 to headings here > > >> + >> +=== Thread === >> +The thread is the fundamental programming unit in ODP. ODP applications >> are >> +organized into a collection of threads that perform the work that the >> +application is designed to do. ODP threads may or may not share memory >> with >> +other threads--that is up to the implementation. Threads come in two >> "flavors": >> +control and worker, that are represented by the abstract type >> ++odp_thread_type_t+. >> + >> +A control thread is a supervisory thread that organizes >> +the operation of worker threads. Worker threads, by contrast, exist to >> +perform the main processing logic of the application and employ a run to >> +completion model. Worker threads, in particular, are intended to operate >> on >> +dedicated processing cores, especially in many core proessing > > > processing > > >> environments, >> +however a given implementation may multitask multiple threads on a single >> +core if desired (typically on smaller and lower performance target >> +environments). >> + >> +In addition to thread types, threads have associated _attributes_ such as >> +_thread mask_ and _scheduler group_ that determine where they can run and >> +the type of work that they can handle. These will be discussed in greater >> +detail later. >> + >> +=== Event === >> +Events are what threads process to perform their work. Events can >> represent >> +new work, such as the arrival of a packet that needs to be processed, or >> they >> +can represent the completion of requests that have executed >> asynchronously. >> +Events can also represent notifications of the passage of time, or of >> status >> +changes in various components of interest to the application. Events >> have an >> +event type that describes what it represents. Threads can create new >> events >> +or consume events processed by them, or they can perform some processing >> on >> +an event and then pass it along to another component for further >> processing. >> +References to events are via handles of abstract type +odp_event_t+. Cast >> +functions are provided to convert these into specific handles of the >> +appropriate type represented by the event. >> + >> +=== Queue === >> +A queue is a message passing channel that holds events. Events can be >> +added to a queue via enqueue operations or removed from a queue via >> dequeue >> +operations. The endpoints of a queue will vary depending on how it is >> used. >> +Queues come in two major types: polled and scheduled, which will be >> +discussed in more detail when the event model is introduced. Queues may >> also >> +have an associated context, which represents a persistent state for all >> +events that make use of it. These states are what permit threads to >> perform >> +stateful processing on events as well as stateless processing. >> + >> +Queues are represented by handles of abstract type +odp_queue_t+. >> + >> +=== Pool === >> +A pool is a shared memory area from which elements may be drawn. Pools >> +represent the backing store for events, among other things. Pools are >> +typically created and destroyed by the application during initialization >> and >> +termination, respectively, and then used during processing. Pools may be >> +used by ODP components exclusively, by applications exclusively, or their >> +use may be shared between the two. Pools have an associated type that >> +characterizes the elements that they contain. The two most important >> pool types >> +are Buffer and Packet. >> + >> +Pools are represented by handles of abstract type +odp_pool_t+. >> + >> +=== Shared Memory === >> +Shared memory represents raw blocks of storage that are sharable between >> +threads. They are the building blocks of pools but can be used directly >> by >> +ODP applications if desired. >> + >> +Shared memory is represented by handles of abstract type +odp_shm_t+. >> + >> +=== Buffer === >> +A buffer is a fixed sized block of shared storage that is used by ODP >> +components and/or applications to realize their function. Buffers contain >> +zero or more bytes of application data as well as system maintained >> +metadata that provide information about the buffer, such as its size or >> the >> +pool it was allocated from. Metadata is an important ODP concept because >> it >> +allows for arbitrary amounts of side information to be associated with an >> +ODP object. Most ODP objects have assocaited metadata and this metadata >> is >> +manipulated via accessor functions that act as getters and setters for >> +this information. Getter acces functions permit an application to read >> +a metadata item, while setter access functions permit an application to >> write >> +a metadata item. Note that some metadata is inherently read only and thus >> +no setter is provided to manipulate it. When object have multiple >> metadata >> +items, each has its own associated getter and/or setter access function >> to > > +inspect or manipulate it. >> + >> +Buffers are represened > > > represented > > > >> by handles of abstract type +odp_buffer_t+. >> + >> +=== Packet === >> +Packets are received and transmitted via I/O interfaces and represent >> +the basic data that data plane applications manipulate. >> +Packets are drawn from pools of type +ODP_POOL_PACKET+. >> +Unlike buffers, which are simple objects, >> +ODP packets have a rich set of semantics that permit their inspection >> +and manipulation in complex ways to be described later. Packets also >> support >> +a rich set of metadata as well as user metadata. User metadata permits >> +applications to associate an application-determined amount of side >> information >> +with each packet for its own use. >> + >> +Packets are represented by handles of abstract type +odp_packet_t+. >> + >> +=== PktIO === >> +PktIO is how ODP represents I/O interfaces. A pktio object is a logical >> +port capable of receiving and/or transmitting packets. This may be >> directly >> +supported by the underlying platform as an integrated feature, >> +or may represent a device attached via a PCIE or other bus. >> + >> +PktIOs are represented by handles of abstract type +odp_pktio_t+. >> + >> +=== Timer === >> +Timers are how ODP applications measure and respond to the passage of >> time. >> +Timers are drawn from specialized pools called timer pools that have >> their >> +own abstract type (+odp_timer_pool_t+). Applications may have many timers >> +active at the same time and can set them to use either relative or >> absolute >> +time. When timers expire they create events of type +odp_timeout_t+, >> which >> +serve as notifications of timer expiration. >> + >> +=== Synchronizer === >> +Multiple threads operating in parallel typically require various >> +synchronization services to permit them to operate in a reliable and >> +coordinated manner. ODP provides a rich set of locks, barriers, and >> similar >> +synchronization primitives, as well as abstract types for representing >> various >> +types of atomic variables. The ODP event model also makes use of queues >> to >> +avoid the need for explicit locking in many cases. This will be discussed >> +in the next section. >> + >> +== ODP Components == > > +Building on ODP concepts, ODP offers several components that relate to the >> +flow of work through an ODP application. These include the Classifier, >> +Scheduler, and Traffic Manager. These components relate to the three >> +main stages of packet processing: Receive, Process, and Transmit. >> > > I think a diagram showing the end to end overview with > PKTIO->classifier->scheduler->TM->Pktio + loopback might help establish > what we define as the set of flow components before breaking to to the > details of a specific component. > > + >> +=== Classifier === >> +The *Classifier* provides a suite of APIs that control packet receive >> (RX) >> +processing. >> + >> +.ODP Receive Processing with Classifier >> +image::../images/odp_rx_processing.png[align="center"] >> + >> +The classifier provides two logically related services: >> +[horizontal] >> +Packet parsing:: Verifying and extracting structural information from a >> +received packet. >> + >> +Packet classification:: Applying *Pattern Matching Rules (PMRs)* to the >> +parsed results to assign an incoming packet to a *Class of Service >> (CoS)*. >> + >> +Combined, these permit incoming packets to be sorted into *flows*, which >> are >> +logically related sequences of packets that share common processing >> +requirements. While many data plane applications perform stateless packet >> +processing (_e.g.,_ for simple forwarding) others perform stateful packet >> +processing. Flows anchor state information relating to these groups of >> +packets. >> + >> +A CoS determines two variables for packets belonging to a flow: >> +[list] >> +* The pool that they will be stored in on receipt >> +* The queue that they will be added to for processing >> + >> +The PMRs supported by ODP permit flow determination based on >> combinations of >> +packet field values (tuples). The main advantage of classification is >> that on >> +many platforms these functions are performed in hardware, meaning that >> +classification occurs at line rate as packets are being received without >> +any explicit processing by the ODP application. >> + >> +Note that the use of the classifier is optional. Applications may >> directly >> +receive packets from a corresponding PktIO input queue via direct polling >> +if they choose. >> + >> +=== Scheduler === >> +The *Scheduler* provides a suite of APIs that control scalabable > > event >> +processing. >> + >> +.ODP Scheduler and Event Processing >> +image::../images/odp_scheduling.png[align="center"] >> + >> +The Scheduler is responsible for selecting and dispatching one or more >> events >> +to a requesting thread. Event selection is based on several factors >> involving >> +both the queues containing schedulable events and the thread making an >> ++odp_schedule()+ or +odp_schedule_multi()+ call. >> + >> +ODP queues have a _scheduling priority_ that determines how urgently >> events >> +on them should be processed relative to events contained in other queues. >> +Queues also have a _scheduler group id_ associated with them that must >> match >> +the associated scheduler group _thread mask_ of the thread calling the >> +scheduler. This permits events to be grouped for processing into classes >> and >> +have threads that are dedicated to processing events from specified >> classes. >> +Threads can join and leave scheduler groups dynamically, permitting easy >> +application response to increases in demand. >> + >> +When a thread receives an event from the scheduler, it in turn can invoke >> +other processing engines via ODP APIs (_e.g.,_ crypto processing) that >> +can operate asynchronously. When such processing is complete, the result >> is >> +that a *completion event* is added to a schedulable queue where it can be >> +scheduled back to a thread to continue processing with the results of the >> +requested asynchronous operation. >> + >> +Threads themselves can enqueue > > > en-queue ? my checker says this > > >> events to queues for downstream processing >> +by other threads, permitting flexibility in how applicaitions > > > applications > > >> structure >> +themselves to maximize concurrency. >> + >> +=== Traffic Manager === >> +The *Traffic Manager* provides a suite of APIs that control traffic >> shaping and >> +Quality of Service (QoS) processing for packet output. >> + >> +.ODP Transmit processing with Traffic Manager >> +image::../images/odp_traffic_manager.png[align="center"] >> + >> +The final stage of packet processing is to transmit it. Here, >> applications have >> +several choices. As with RX processing, applications may send packets >> +directly to PktIO TX queues for direct transmission. Often, however, >> +applications need to perform traffic shaping and related >> +*Quality of Service (QoS)* processing on the packets comprising a flow >> as part >> +of transmit processing. To handle this need, ODP provides a suite of >> +*Traffic Manager* APIs that permit programmatic establishment of >> arbiters, >> +shapers, etc. that control output packet processing to achieve desired >> QoS >> +goals. Again, the advantage here is that on many platforms traffic >> management >> +functions are implemented in hardware, permitting transparent offload of >> +this work. >> + >> Glossary >> -------- >> [glossary] >> @@ -28,7 +425,6 @@ odp_thread:: >> event:: >> An event is a notification that can be placed in a queue. >> > > > I think we should add some of the terms introduced here, not sure if all > of these should be considered domain knowledge or not in a user-guide, some > are ODP specific in meaning > > scheduler > engines ( co-processing ) > completion event > platform > flows > components > concurrency > traffic > arbiter > shaper > forwarding > stateful > parsing > metadata > getter > setter > > > >> -:numbered: >> The include structure >> --------------------- >> Applications only include the 'include/odp.h file which includes the >> 'platform/<implementation name>/include/plat' files to provide a complete >> definition of the API on that platform. >> -- >> 2.1.4 >> >> _______________________________________________ >> lng-odp mailing list >> lng-odp@lists.linaro.org >> https://lists.linaro.org/mailman/listinfo/lng-odp >> > > > Apply-and build caught a couple of things > > PATCH_DIR=~/incoming/bill ./apply-and-build.sh > Using patch: > lng-odp_PATCHv2_2-2_doc_userguide_add_baseline_overview_to_document.mbox > Trying to apply patch > Patch applied > WARNING: 'assocaited' may be misspelled - perhaps 'associated'? > #288: FILE: doc/users-guide/users-guide.adoc:273: > +ODP object. Most ODP objects have assocaited metadata and this metadata is > > WARNING: 'acces' may be misspelled - perhaps 'access'? > #290: FILE: doc/users-guide/users-guide.adoc:275: > +this information. Getter acces functions permit an application to read > > > -- > Mike Holmes > Technical Manager - Linaro Networking Group > Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM SoCs > > >
On Tue, Nov 17, 2015 at 11:15:14AM -0600, Bill Fischofer wrote: > Add a basic ODP overview to the User's Guide, providing > an overview of ODP concepts, components, etc. Included > are a number of diagrams covering component structure > as well as packet RX, event scheduling, and packet TX > processing. > > Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> Looks good to me. Mostly just typo comments below. > --- > doc/users-guide/users-guide.adoc | 400 ++++++++++++++++++++++++++++++++++++++- > 1 file changed, 398 insertions(+), 2 deletions(-) > > diff --git a/doc/users-guide/users-guide.adoc b/doc/users-guide/users-guide.adoc > index 7f70046..cf77fa0 100644 > --- a/doc/users-guide/users-guide.adoc > +++ b/doc/users-guide/users-guide.adoc > @@ -8,7 +8,7 @@ OpenDataPlane (ODP) Users-Guide > Abstract > -------- > This document is intended to guide a new ODP application developer. > -Further details about ODP may be found at then http://opendataplane.org[ODP] home page. > +Further details about ODP may be found at the http://opendataplane.org[ODP] home page. > > .Overview of a system running ODP applications > image::../images/overview.png[align="center"] > @@ -16,6 +16,403 @@ image::../images/overview.png[align="center"] > ODP is an API specification that allows many implementations to provide platform independence, automatic hardware acceleration and CPU scaling to high performance networking applications. > This document describes how to write an application that can successfully take advantage of the API. > > +:numbered: > +== Introduction == > +.OpenDataPlane Components > +image::../images/odp_components.png[align="center"] > + > +.The ODP API Specification > +ODP consists of three separate but related component parts. First, ODP is an > +abstract API specification that describes a functional model for > +data plane applications. This specification covers many common data plane > +application programming needs, such as the ability to receive, manipulate, and > +transmit packet data, without specifying how these functions are performed. This > +is quite intentional. It is precisely because ODP APIs do not have a preferred > +embodiment that they permit innovation in how these functions can > +be realized on various platforms that offer implementations of ODP. To achieve > +this goal, ODP APIs are described using abstract data types whose definition > +is left up to the ODP implementer. For example, in ODP packets are referenced > +by abstract handles of type +odp_packet_t+, and packet-related APIs take > +arguments of this type. What an +odp_packet_t+ actually is is not part of the > +ODP API specification--that is the responsibility of each ODP implementation. > + > +.Summary: ODP API attributes: > +* Open Source, open contribution, BSD-3 licensed. > +* Vendor and platform neutral. > +* Application-centric. Covers functional needs of data plane applications. > +* Ensures portability by specifying the functional behavior of ODP. > +* Defined jointly and openly by application writers and platform implementers. > +* Archiected to be implementable on a wide range of platforms efficiently Architected > +* Sponsored, governed, and maintained by the Linaro Networking Group (LNG) > + > +.ODP Implementations > +Second, ODP consists of multiple implementations of this API specification, > +each tailored to a specific target platform. ODP implementations determine > +how each ODP abstract type is represented on that platform and how each ODP > +API is realized. On some platforms, ODP APIs will > +be realized using specialized instructions that accelerate the functional > +behavior specified by the API. On others, hardware co-processing engines may > +completely offload the API so that again it can be performed with little or no > +involvement by a CPU. In all cases, the application sees the same > +functional behavior independent of how a given platform has chosen to realize > +it. By allowing each platform the freedom to determine how best to realize each > +API's specified functional behavior in an optimal manner, ODP permits > +applications written to its APIs to take full advantage of the unique > +capabilities of each platform without the application programmer needing to > +have specialist knowledge of that platform or to be concerned with how best > +to tune the application to a particular platform. This latter consideration is > +particularly important in Network Function Virtualization (NFV) environments > +where the application will run on a target platform chosen by someone else. > + > +.Summary: ODP Implementation Characteristics > +* One size does not fit all--supporting multiple implementations allows ODP > +to adapt to widely differing internals among platforms. > +* Anyone can create an ODP implementation tailored to their platform > +* Distribution and mainteinance of each implementation is as owner wishes maintenance > + - Open source or closed source as business needs determine > + - Have independent release cycles and service streams > +* Allows HW and SW innovation in how ODP APIs are implemented on each platform. > + > +.Reference Implementations > +To make it easy to get started with implementing ODP on a new platform, ODP > +supplies a number of _reference implementations_ that can serve as a > +starting point. The two primary references implementations supplied by ODP are > +*odp-linux* and *odp-dpdk* > + > +.odp-linux > +The *odp-linux* reference implementation is a pure SW implementation of the > +ODP API that relies only on the Linux programming API. As a functional model > +for ODP, it enables ODP to be bootstrapped easily to any platform that > +supports a Linux kernel. > + > +.odp-dpdk > +The *odp-dpdk* reference implementation is a pure SW implementation of the > +ODP API that uses http://dpdk.org[DPDK] as a SW accelerator. In particular, > +*odp-dpdk* offers superior I/O performance for systems that use NICs, allowing > +ODP applications to take immediate full advantage of the various NIC device > +drivers supported by DPDK. > + > +.Summary: ODP Reference Implementations > +* Open source, open contribution, BSD-3 licensed. > +* Provide easy bootstrapping of ODP onto new platforms > +* Implementers free to borrow or tailor code as needed for their platform > +* Implementers retain full control over their implementations whether or not > +they are derived from a reference implementation. > + > +.ODP Validation Test Suite > +Third, to enure consistency between different ODP implementations, ODP > +consists of a validation suite that verifies that any given implementation of > +ODP faithfully provides the specified functional behavior of each ODP API. > +As a separate open source component, the validation suite may be used by > +application writers, system integrators, and platform providers alike to > +confirm that any purported implementation of ODP does indeed conform to the > +ODP API specification. > + > +.Summary: ODP Validation Test Suite > +* Synchronized with ODP API specification > +* Maintained and distributed by LNG > +* Open source, open contribution, BSD-3 licensed. > +* Key to ensuring application portability across all ODP implementations > +* Tests that ODP implementations conform to the specified functional behavior > +of ODP APIs. > +* Can be run at any time by users and vendors to validat implementations validate > +od ODP. of > + > +=== ODP API Specification Versioning === > +As an evolving standard, the ODP API specification is released under an > +incrementing version number, and corresponding implementations of ODP, as well > +as the validation suite that verifies API conformance, are linked to this > +version number. ODP versions are specified using a stanard three-level standard > +number (major.minor.fixlevel) that are incremented according to the degree of > +change the level represents. Increments to the fixlevel represent clarification > +of the specification or other minor changes that do not affect either the > +syntax or semantics of the specification. Such changes in the API specification > +are expected to be rare. Increments to the minor level > +represent the introduction of new APIs or functional capabilities, or changes > +to he specified syntax or functional behavior of APIs and thus may require the > +application source code changes. Such changes are well documented in the > +release notes for each revision of the specification. Finally, increments to > +the major level represent significant structural changes that most likely > +require some level of application source code change, again as documented in > +the release notes for that version. > + > +=== ODP Implementation Versioning === > +ODP implementations are free to use whatever release naming/numbering > +conventions they wish, as long as it is clear what level of the ODP API a given > +release implements. A recommended convention is to use the same three level > +numbering scheme where the major and minor numbers correspond to the ODP API > +level and the fixlevel represents an implementation-defined service level > +associated with that API level implementation. The LNG-supplied ODP reference > +implementations follow this convention. > + > +=== ODP Validation Test Suite Versioning === > +The ODP validation test suite follows these same naming conventions. The major > +and minor release numbers correspond to the ODP API level that the suite > +validates and the fixlevel represents the service level of the validation > +suite itself for that API level. > + > +=== ODP Design Goals === > +ODP has three primary goals that follow from its component structure. The first > +is application portability across a wide range of platforms. These platforms > +differ in terms of processor instruction set architecture, number and types of > +application processing cores, memory oranization, as well as the number and organization (or organisation.. :) > +type of platform specific hardware acceleration and offload features that > +are available. ODP applications can move from one conforming implementation > +to another with at most a recompile. > + > +Second, ODP is designed to permit data plane applications to avail themselves > +of platform-specific features, including specialized hardware accelerators, > +without specialized programming. This is achieved by separating the API > +specification from their implementation on individual platforms. Since each > +platform implements each ODP API in a manner optimal to that platform, > +applications automatically gain the benefit of such optimizations without the > +need for explicit programming. > + > +Third, ODP is designed to allow applications to scale out automatically to > +support many core architectures. This is done using an event based programming > +model that permits applications to be written to be independent of the number > +of processing cores that are available to realize application function. The > +result is that an application written to this model does not require redesign > +as it scales from 4, to 40, to 400 cores. > + > +== Organization of this Document == > +This document is organized into several sections. The first presents a high > +level overview of the ODP API component areas and their associated abstract > +data types. This section introduces ODP APIs at a conceptual level. > +The second provides a tutorial on the programming model(s) > +supported by ODP, paying particular attention to the event model as this > +represents the preferred structure for most ODP applications. This section > +builds on the concepts introduced in the first section and shows how ODP > +applications are structured to best realize the three ODP design goals > +mentioned earlier. The third section provides a more detailed overview of > +the major ODP API components and is designed to serve as a companion to the > +full reference specification for each API. The latter is intended to be used > +by ODP application programmers, as well as implementers, to understand the > +precise syntax and semantics of each API. > + > +== ODP API Concepts == > +ODP programs are built around several conceptual structures that every > +appliation programmer needs to be familiar with to use ODP effectively. The application > +main ODP concepts are: > +Thread, Event, Queue, Pool, Shared Memory, Buffer, Packet, PktIO, Timer, > +and Synchronizer. > + > +=== Thread === > +The thread is the fundamental programming unit in ODP. ODP applications are > +organized into a collection of threads that perform the work that the > +application is designed to do. ODP threads may or may not share memory with > +other threads--that is up to the implementation. Threads come in two "flavors": > +control and worker, that are represented by the abstract type > ++odp_thread_type_t+. > + > +A control thread is a supervisory thread that organizes > +the operation of worker threads. Worker threads, by contrast, exist to > +perform the main processing logic of the application and employ a run to > +completion model. Worker threads, in particular, are intended to operate on > +dedicated processing cores, especially in many core proessing environments, processing > +however a given implementation may multitask multiple threads on a single > +core if desired (typically on smaller and lower performance target > +environments). > + > +In addition to thread types, threads have associated _attributes_ such as > +_thread mask_ and _scheduler group_ that determine where they can run and > +the type of work that they can handle. These will be discussed in greater > +detail later. > + > +=== Event === > +Events are what threads process to perform their work. Events can represent > +new work, such as the arrival of a packet that needs to be processed, or they > +can represent the completion of requests that have executed asynchronously. > +Events can also represent notifications of the passage of time, or of status > +changes in various components of interest to the application. Events have an > +event type that describes what it represents. Threads can create new events > +or consume events processed by them, or they can perform some processing on > +an event and then pass it along to another component for further processing. > +References to events are via handles of abstract type +odp_event_t+. Cast > +functions are provided to convert these into specific handles of the > +appropriate type represented by the event. > + > +=== Queue === > +A queue is a message passing channel that holds events. Events can be > +added to a queue via enqueue operations or removed from a queue via dequeue > +operations. The endpoints of a queue will vary depending on how it is used. > +Queues come in two major types: polled and scheduled, which will be > +discussed in more detail when the event model is introduced. Queues may also > +have an associated context, which represents a persistent state for all > +events that make use of it. These states are what permit threads to perform > +stateful processing on events as well as stateless processing. > + > +Queues are represented by handles of abstract type +odp_queue_t+. > + > +=== Pool === > +A pool is a shared memory area from which elements may be drawn. Pools > +represent the backing store for events, among other things. Pools are > +typically created and destroyed by the application during initialization and > +termination, respectively, and then used during processing. Pools may be > +used by ODP components exclusively, by applications exclusively, or their > +use may be shared between the two. Pools have an associated type that > +characterizes the elements that they contain. The two most important pool types > +are Buffer and Packet. > + > +Pools are represented by handles of abstract type +odp_pool_t+. > + > +=== Shared Memory === > +Shared memory represents raw blocks of storage that are sharable between > +threads. They are the building blocks of pools but can be used directly by > +ODP applications if desired. > + > +Shared memory is represented by handles of abstract type +odp_shm_t+. > + > +=== Buffer === > +A buffer is a fixed sized block of shared storage that is used by ODP > +components and/or applications to realize their function. Buffers contain > +zero or more bytes of application data as well as system maintained > +metadata that provide information about the buffer, such as its size or the > +pool it was allocated from. Metadata is an important ODP concept because it > +allows for arbitrary amounts of side information to be associated with an > +ODP object. Most ODP objects have assocaited metadata and this metadata is associated > +manipulated via accessor functions that act as getters and setters for > +this information. Getter acces functions permit an application to read access > +a metadata item, while setter access functions permit an application to write > +a metadata item. Note that some metadata is inherently read only and thus > +no setter is provided to manipulate it. When object have multiple metadata "When an object has"? > +items, each has its own associated getter and/or setter access function to > +inspect or manipulate it. > + > +Buffers are represened by handles of abstract type +odp_buffer_t+. represented > + > +=== Packet === > +Packets are received and transmitted via I/O interfaces and represent > +the basic data that data plane applications manipulate. > +Packets are drawn from pools of type +ODP_POOL_PACKET+. > +Unlike buffers, which are simple objects, > +ODP packets have a rich set of semantics that permit their inspection > +and manipulation in complex ways to be described later. Packets also support > +a rich set of metadata as well as user metadata. User metadata permits > +applications to associate an application-determined amount of side information > +with each packet for its own use. > + > +Packets are represented by handles of abstract type +odp_packet_t+. > + > +=== PktIO === > +PktIO is how ODP represents I/O interfaces. A pktio object is a logical > +port capable of receiving and/or transmitting packets. This may be directly > +supported by the underlying platform as an integrated feature, > +or may represent a device attached via a PCIE or other bus. > + > +PktIOs are represented by handles of abstract type +odp_pktio_t+. > + > +=== Timer === > +Timers are how ODP applications measure and respond to the passage of time. > +Timers are drawn from specialized pools called timer pools that have their > +own abstract type (+odp_timer_pool_t+). Applications may have many timers > +active at the same time and can set them to use either relative or absolute > +time. When timers expire they create events of type +odp_timeout_t+, which > +serve as notifications of timer expiration. > + > +=== Synchronizer === > +Multiple threads operating in parallel typically require various > +synchronization services to permit them to operate in a reliable and > +coordinated manner. ODP provides a rich set of locks, barriers, and similar > +synchronization primitives, as well as abstract types for representing various > +types of atomic variables. The ODP event model also makes use of queues to > +avoid the need for explicit locking in many cases. This will be discussed > +in the next section. > + > +== ODP Components == > +Building on ODP concepts, ODP offers several components that relate to the > +flow of work through an ODP application. These include the Classifier, > +Scheduler, and Traffic Manager. These components relate to the three > +main stages of packet processing: Receive, Process, and Transmit. > + > +=== Classifier === > +The *Classifier* provides a suite of APIs that control packet receive (RX) > +processing. > + > +.ODP Receive Processing with Classifier > +image::../images/odp_rx_processing.png[align="center"] > + > +The classifier provides two logically related services: > +[horizontal] > +Packet parsing:: Verifying and extracting structural information from a > +received packet. > + > +Packet classification:: Applying *Pattern Matching Rules (PMRs)* to the > +parsed results to assign an incoming packet to a *Class of Service (CoS)*. > + > +Combined, these permit incoming packets to be sorted into *flows*, which are > +logically related sequences of packets that share common processing > +requirements. While many data plane applications perform stateless packet > +processing (_e.g.,_ for simple forwarding) others perform stateful packet > +processing. Flows anchor state information relating to these groups of > +packets. > + > +A CoS determines two variables for packets belonging to a flow: > +[list] > +* The pool that they will be stored in on receipt > +* The queue that they will be added to for processing > + > +The PMRs supported by ODP permit flow determination based on combinations of > +packet field values (tuples). The main advantage of classification is that on > +many platforms these functions are performed in hardware, meaning that > +classification occurs at line rate as packets are being received without > +any explicit processing by the ODP application. > + > +Note that the use of the classifier is optional. Applications may directly > +receive packets from a corresponding PktIO input queue via direct polling > +if they choose. > + > +=== Scheduler === > +The *Scheduler* provides a suite of APIs that control scalabable event scalable > +processing. > + > +.ODP Scheduler and Event Processing > +image::../images/odp_scheduling.png[align="center"] > + > +The Scheduler is responsible for selecting and dispatching one or more events > +to a requesting thread. Event selection is based on several factors involving > +both the queues containing schedulable events and the thread making an > ++odp_schedule()+ or +odp_schedule_multi()+ call. > + > +ODP queues have a _scheduling priority_ that determines how urgently events > +on them should be processed relative to events contained in other queues. > +Queues also have a _scheduler group id_ associated with them that must match > +the associated scheduler group _thread mask_ of the thread calling the > +scheduler. This permits events to be grouped for processing into classes and > +have threads that are dedicated to processing events from specified classes. > +Threads can join and leave scheduler groups dynamically, permitting easy > +application response to increases in demand. > + > +When a thread receives an event from the scheduler, it in turn can invoke > +other processing engines via ODP APIs (_e.g.,_ crypto processing) that > +can operate asynchronously. When such processing is complete, the result is > +that a *completion event* is added to a schedulable queue where it can be > +scheduled back to a thread to continue processing with the results of the > +requested asynchronous operation. > + > +Threads themselves can enqueue events to queues for downstream processing > +by other threads, permitting flexibility in how applicaitions structure applications > +themselves to maximize concurrency. > + > +=== Traffic Manager === > +The *Traffic Manager* provides a suite of APIs that control traffic shaping and > +Quality of Service (QoS) processing for packet output. > + > +.ODP Transmit processing with Traffic Manager > +image::../images/odp_traffic_manager.png[align="center"] > + > +The final stage of packet processing is to transmit it. Here, applications have > +several choices. As with RX processing, applications may send packets > +directly to PktIO TX queues for direct transmission. Often, however, > +applications need to perform traffic shaping and related > +*Quality of Service (QoS)* processing on the packets comprising a flow as part > +of transmit processing. To handle this need, ODP provides a suite of > +*Traffic Manager* APIs that permit programmatic establishment of arbiters, > +shapers, etc. that control output packet processing to achieve desired QoS > +goals. Again, the advantage here is that on many platforms traffic management > +functions are implemented in hardware, permitting transparent offload of > +this work. > + > Glossary > -------- > [glossary] This glossary now seems a bit redundant, maybe some of the text could be merged into the ODP API Concepts -> Thread section. > @@ -28,7 +425,6 @@ odp_thread:: > event:: > An event is a notification that can be placed in a queue. > > -:numbered: > The include structure > --------------------- > Applications only include the 'include/odp.h file which includes the 'platform/<implementation name>/include/plat' files to provide a complete definition of the API on that platform. > -- > 2.1.4 >
On Wed, Nov 18, 2015 at 03:30:31PM +0000, Stuart Haslam wrote: > On Tue, Nov 17, 2015 at 11:15:14AM -0600, Bill Fischofer wrote: > > Add a basic ODP overview to the User's Guide, providing > > an overview of ODP concepts, components, etc. Included > > are a number of diagrams covering component structure > > as well as packet RX, event scheduling, and packet TX > > processing. > > > > Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> > > Looks good to me. Mostly just typo comments below. > Sorry just realised I reviewed v2 and you've sent v4 with the typos fixed. The comment about the glossary is still valid though. -- Stuart.
Thanks. The glossary is part of the base document and not covered by this patch. The intent is to start improving the document on an incremental basis to keep the size of individual patches manageable. Much more will follow as we restructure and fill in this document. On Wed, Nov 18, 2015 at 9:38 AM, Stuart Haslam <stuart.haslam@linaro.org> wrote: > On Wed, Nov 18, 2015 at 03:30:31PM +0000, Stuart Haslam wrote: > > On Tue, Nov 17, 2015 at 11:15:14AM -0600, Bill Fischofer wrote: > > > Add a basic ODP overview to the User's Guide, providing > > > an overview of ODP concepts, components, etc. Included > > > are a number of diagrams covering component structure > > > as well as packet RX, event scheduling, and packet TX > > > processing. > > > > > > Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> > > > > Looks good to me. Mostly just typo comments below. > > > > Sorry just realised I reviewed v2 and you've sent v4 with the typos > fixed. The comment about the glossary is still valid though. > > -- > Stuart. >
On that subject, I was just discussing with Mike this morning how Asciidoc support include structures, so it's probably reasonable to put things like a glossary into its own file that can be maintained separately and included in the main doc. On Wed, Nov 18, 2015 at 9:53 AM, Bill Fischofer <bill.fischofer@linaro.org> wrote: > Thanks. The glossary is part of the base document and not covered by this > patch. The intent is to start improving the document on an incremental > basis to keep the size of individual patches manageable. Much more will > follow as we restructure and fill in this document. > > On Wed, Nov 18, 2015 at 9:38 AM, Stuart Haslam <stuart.haslam@linaro.org> > wrote: > >> On Wed, Nov 18, 2015 at 03:30:31PM +0000, Stuart Haslam wrote: >> > On Tue, Nov 17, 2015 at 11:15:14AM -0600, Bill Fischofer wrote: >> > > Add a basic ODP overview to the User's Guide, providing >> > > an overview of ODP concepts, components, etc. Included >> > > are a number of diagrams covering component structure >> > > as well as packet RX, event scheduling, and packet TX >> > > processing. >> > > >> > > Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> >> > >> > Looks good to me. Mostly just typo comments below. >> > >> >> Sorry just realised I reviewed v2 and you've sent v4 with the typos >> fixed. The comment about the glossary is still valid though. >> >> -- >> Stuart. >> > >
Reviewed-by: Mike Holmes <mike.holmes@linaro.org Caveat that issues such as the glossary etc are solved before Monarch On 18 November 2015 at 10:54, Bill Fischofer <bill.fischofer@linaro.org> wrote: > On that subject, I was just discussing with Mike this morning how Asciidoc > support include structures, so it's probably reasonable to put things like > a glossary into its own file that can be maintained separately and included > in the main > doc. > > On Wed, Nov 18, 2015 at 9:53 AM, Bill Fischofer <bill.fischofer@linaro.org > > wrote: > >> Thanks. The glossary is part of the base document and not covered by >> this patch. The intent is to start improving the document on an >> incremental basis to keep the size of individual patches manageable. Much >> more will follow as we restructure and fill in this document. >> >> On Wed, Nov 18, 2015 at 9:38 AM, Stuart Haslam <stuart.haslam@linaro.org> >> wrote: >> >>> On Wed, Nov 18, 2015 at 03:30:31PM +0000, Stuart Haslam wrote: >>> > On Tue, Nov 17, 2015 at 11:15:14AM -0600, Bill Fischofer wrote: >>> > > Add a basic ODP overview to the User's Guide, providing >>> > > an overview of ODP concepts, components, etc. Included >>> > > are a number of diagrams covering component structure >>> > > as well as packet RX, event scheduling, and packet TX >>> > > processing. >>> > > >>> > > Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> >>> > >>> > Looks good to me. Mostly just typo comments below. >>> > >>> >>> Sorry just realised I reviewed v2 and you've sent v4 with the typos >>> fixed. The comment about the glossary is still valid though. >>> >>> -- >>> Stuart. >>> >> >> > > _______________________________________________ > lng-odp mailing list > lng-odp@lists.linaro.org > https://lists.linaro.org/mailman/listinfo/lng-odp > > -- Mike Holmes Technical Manager - Linaro Networking Group Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM SoCs
ping. These doc updates need to be merged so we can continue to add to them. On Mon, Nov 23, 2015 at 10:37 AM, Mike Holmes <mike.holmes@linaro.org> wrote: > Reviewed-by: Mike Holmes <mike.holmes@linaro.org > > Caveat that issues such as the glossary etc are solved before Monarch > > > On 18 November 2015 at 10:54, Bill Fischofer <bill.fischofer@linaro.org> > wrote: > >> On that subject, I was just discussing with Mike this morning how >> Asciidoc support include structures, so it's probably reasonable to put >> things like a glossary into its own file that can be maintained separately >> and included in the main >> doc. >> >> On Wed, Nov 18, 2015 at 9:53 AM, Bill Fischofer < >> bill.fischofer@linaro.org> wrote: >> >>> Thanks. The glossary is part of the base document and not covered by >>> this patch. The intent is to start improving the document on an >>> incremental basis to keep the size of individual patches manageable. Much >>> more will follow as we restructure and fill in this document. >>> >>> On Wed, Nov 18, 2015 at 9:38 AM, Stuart Haslam <stuart.haslam@linaro.org >>> > wrote: >>> >>>> On Wed, Nov 18, 2015 at 03:30:31PM +0000, Stuart Haslam wrote: >>>> > On Tue, Nov 17, 2015 at 11:15:14AM -0600, Bill Fischofer wrote: >>>> > > Add a basic ODP overview to the User's Guide, providing >>>> > > an overview of ODP concepts, components, etc. Included >>>> > > are a number of diagrams covering component structure >>>> > > as well as packet RX, event scheduling, and packet TX >>>> > > processing. >>>> > > >>>> > > Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> >>>> > >>>> > Looks good to me. Mostly just typo comments below. >>>> > >>>> >>>> Sorry just realised I reviewed v2 and you've sent v4 with the typos >>>> fixed. The comment about the glossary is still valid though. >>>> >>>> -- >>>> Stuart. >>>> >>> >>> >> >> _______________________________________________ >> lng-odp mailing list >> lng-odp@lists.linaro.org >> https://lists.linaro.org/mailman/listinfo/lng-odp >> >> > > > -- > Mike Holmes > Technical Manager - Linaro Networking Group > Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM SoCs > > >
Merged, Maxim. On 11/29/2015 19:13, Bill Fischofer wrote: > ping. These doc updates need to be merged so we can continue to add > to them. > > On Mon, Nov 23, 2015 at 10:37 AM, Mike Holmes <mike.holmes@linaro.org > <mailto:mike.holmes@linaro.org>> wrote: > > Reviewed-by: Mike Holmes <mike.holmes@linaro.org > <mailto:mike.holmes@linaro.org> > > Caveat that issues such as the glossary etc are solved before Monarch > > > On 18 November 2015 at 10:54, Bill Fischofer > <bill.fischofer@linaro.org <mailto:bill.fischofer@linaro.org>> wrote: > > On that subject, I was just discussing with Mike this morning > how Asciidoc support include structures, so it's probably > reasonable to put things like a glossary into its own file > that can be maintained separately and included in the main > doc. > > On Wed, Nov 18, 2015 at 9:53 AM, Bill Fischofer > <bill.fischofer@linaro.org <mailto:bill.fischofer@linaro.org>> > wrote: > > Thanks. The glossary is part of the base document and not > covered by this patch. The intent is to start improving > the document on an incremental basis to keep the size of > individual patches manageable. Much more will follow as > we restructure and fill in this document. > > On Wed, Nov 18, 2015 at 9:38 AM, Stuart Haslam > <stuart.haslam@linaro.org > <mailto:stuart.haslam@linaro.org>> wrote: > > On Wed, Nov 18, 2015 at 03:30:31PM +0000, Stuart > Haslam wrote: > > On Tue, Nov 17, 2015 at 11:15:14AM -0600, Bill > Fischofer wrote: > > > Add a basic ODP overview to the User's Guide, > providing > > > an overview of ODP concepts, components, etc. Included > > > are a number of diagrams covering component structure > > > as well as packet RX, event scheduling, and packet TX > > > processing. > > > > > > Signed-off-by: Bill Fischofer > <bill.fischofer@linaro.org > <mailto:bill.fischofer@linaro.org>> > > > > Looks good to me. Mostly just typo comments below. > > > > Sorry just realised I reviewed v2 and you've sent v4 > with the typos > fixed. The comment about the glossary is still valid > though. > > -- > Stuart. > > > > > _______________________________________________ > lng-odp mailing list > lng-odp@lists.linaro.org <mailto:lng-odp@lists.linaro.org> > https://lists.linaro.org/mailman/listinfo/lng-odp > > > > > -- > Mike Holmes > Technical Manager - Linaro Networking Group > Linaro.org <http://www.linaro.org/>***│ *Open source software for > ARM SoCs > > > > > _______________________________________________ > lng-odp mailing list > lng-odp@lists.linaro.org > https://lists.linaro.org/mailman/listinfo/lng-odp
diff --git a/doc/users-guide/users-guide.adoc b/doc/users-guide/users-guide.adoc index 7f70046..cf77fa0 100644 --- a/doc/users-guide/users-guide.adoc +++ b/doc/users-guide/users-guide.adoc @@ -8,7 +8,7 @@ OpenDataPlane (ODP) Users-Guide Abstract -------- This document is intended to guide a new ODP application developer. -Further details about ODP may be found at then http://opendataplane.org[ODP] home page. +Further details about ODP may be found at the http://opendataplane.org[ODP] home page. .Overview of a system running ODP applications image::../images/overview.png[align="center"] @@ -16,6 +16,403 @@ image::../images/overview.png[align="center"] ODP is an API specification that allows many implementations to provide platform independence, automatic hardware acceleration and CPU scaling to high performance networking applications. This document describes how to write an application that can successfully take advantage of the API. +:numbered: +== Introduction == +.OpenDataPlane Components +image::../images/odp_components.png[align="center"] + +.The ODP API Specification +ODP consists of three separate but related component parts. First, ODP is an +abstract API specification that describes a functional model for +data plane applications. This specification covers many common data plane +application programming needs, such as the ability to receive, manipulate, and +transmit packet data, without specifying how these functions are performed. This +is quite intentional. It is precisely because ODP APIs do not have a preferred +embodiment that they permit innovation in how these functions can +be realized on various platforms that offer implementations of ODP. To achieve +this goal, ODP APIs are described using abstract data types whose definition +is left up to the ODP implementer. For example, in ODP packets are referenced +by abstract handles of type +odp_packet_t+, and packet-related APIs take +arguments of this type. What an +odp_packet_t+ actually is is not part of the +ODP API specification--that is the responsibility of each ODP implementation. + +.Summary: ODP API attributes: +* Open Source, open contribution, BSD-3 licensed. +* Vendor and platform neutral. +* Application-centric. Covers functional needs of data plane applications. +* Ensures portability by specifying the functional behavior of ODP. +* Defined jointly and openly by application writers and platform implementers. +* Archiected to be implementable on a wide range of platforms efficiently +* Sponsored, governed, and maintained by the Linaro Networking Group (LNG) + +.ODP Implementations +Second, ODP consists of multiple implementations of this API specification, +each tailored to a specific target platform. ODP implementations determine +how each ODP abstract type is represented on that platform and how each ODP +API is realized. On some platforms, ODP APIs will +be realized using specialized instructions that accelerate the functional +behavior specified by the API. On others, hardware co-processing engines may +completely offload the API so that again it can be performed with little or no +involvement by a CPU. In all cases, the application sees the same +functional behavior independent of how a given platform has chosen to realize +it. By allowing each platform the freedom to determine how best to realize each +API's specified functional behavior in an optimal manner, ODP permits +applications written to its APIs to take full advantage of the unique +capabilities of each platform without the application programmer needing to +have specialist knowledge of that platform or to be concerned with how best +to tune the application to a particular platform. This latter consideration is +particularly important in Network Function Virtualization (NFV) environments +where the application will run on a target platform chosen by someone else. + +.Summary: ODP Implementation Characteristics +* One size does not fit all--supporting multiple implementations allows ODP +to adapt to widely differing internals among platforms. +* Anyone can create an ODP implementation tailored to their platform +* Distribution and mainteinance of each implementation is as owner wishes + - Open source or closed source as business needs determine + - Have independent release cycles and service streams +* Allows HW and SW innovation in how ODP APIs are implemented on each platform. + +.Reference Implementations +To make it easy to get started with implementing ODP on a new platform, ODP +supplies a number of _reference implementations_ that can serve as a +starting point. The two primary references implementations supplied by ODP are +*odp-linux* and *odp-dpdk* + +.odp-linux +The *odp-linux* reference implementation is a pure SW implementation of the +ODP API that relies only on the Linux programming API. As a functional model +for ODP, it enables ODP to be bootstrapped easily to any platform that +supports a Linux kernel. + +.odp-dpdk +The *odp-dpdk* reference implementation is a pure SW implementation of the +ODP API that uses http://dpdk.org[DPDK] as a SW accelerator. In particular, +*odp-dpdk* offers superior I/O performance for systems that use NICs, allowing +ODP applications to take immediate full advantage of the various NIC device +drivers supported by DPDK. + +.Summary: ODP Reference Implementations +* Open source, open contribution, BSD-3 licensed. +* Provide easy bootstrapping of ODP onto new platforms +* Implementers free to borrow or tailor code as needed for their platform +* Implementers retain full control over their implementations whether or not +they are derived from a reference implementation. + +.ODP Validation Test Suite +Third, to enure consistency between different ODP implementations, ODP +consists of a validation suite that verifies that any given implementation of +ODP faithfully provides the specified functional behavior of each ODP API. +As a separate open source component, the validation suite may be used by +application writers, system integrators, and platform providers alike to +confirm that any purported implementation of ODP does indeed conform to the +ODP API specification. + +.Summary: ODP Validation Test Suite +* Synchronized with ODP API specification +* Maintained and distributed by LNG +* Open source, open contribution, BSD-3 licensed. +* Key to ensuring application portability across all ODP implementations +* Tests that ODP implementations conform to the specified functional behavior +of ODP APIs. +* Can be run at any time by users and vendors to validat implementations +od ODP. + +=== ODP API Specification Versioning === +As an evolving standard, the ODP API specification is released under an +incrementing version number, and corresponding implementations of ODP, as well +as the validation suite that verifies API conformance, are linked to this +version number. ODP versions are specified using a stanard three-level +number (major.minor.fixlevel) that are incremented according to the degree of +change the level represents. Increments to the fixlevel represent clarification +of the specification or other minor changes that do not affect either the +syntax or semantics of the specification. Such changes in the API specification +are expected to be rare. Increments to the minor level +represent the introduction of new APIs or functional capabilities, or changes +to he specified syntax or functional behavior of APIs and thus may require +application source code changes. Such changes are well documented in the +release notes for each revision of the specification. Finally, increments to +the major level represent significant structural changes that most likely +require some level of application source code change, again as documented in +the release notes for that version. + +=== ODP Implementation Versioning === +ODP implementations are free to use whatever release naming/numbering +conventions they wish, as long as it is clear what level of the ODP API a given +release implements. A recommended convention is to use the same three level +numbering scheme where the major and minor numbers correspond to the ODP API +level and the fixlevel represents an implementation-defined service level +associated with that API level implementation. The LNG-supplied ODP reference +implementations follow this convention. + +=== ODP Validation Test Suite Versioning === +The ODP validation test suite follows these same naming conventions. The major +and minor release numbers correspond to the ODP API level that the suite +validates and the fixlevel represents the service level of the validation +suite itself for that API level. + +=== ODP Design Goals === +ODP has three primary goals that follow from its component structure. The first +is application portability across a wide range of platforms. These platforms +differ in terms of processor instruction set architecture, number and types of +application processing cores, memory oranization, as well as the number and +type of platform specific hardware acceleration and offload features that +are available. ODP applications can move from one conforming implementation +to another with at most a recompile. + +Second, ODP is designed to permit data plane applications to avail themselves +of platform-specific features, including specialized hardware accelerators, +without specialized programming. This is achieved by separating the API +specification from their implementation on individual platforms. Since each +platform implements each ODP API in a manner optimal to that platform, +applications automatically gain the benefit of such optimizations without the +need for explicit programming. + +Third, ODP is designed to allow applications to scale out automatically to +support many core architectures. This is done using an event based programming +model that permits applications to be written to be independent of the number +of processing cores that are available to realize application function. The +result is that an application written to this model does not require redesign +as it scales from 4, to 40, to 400 cores. + +== Organization of this Document == +This document is organized into several sections. The first presents a high +level overview of the ODP API component areas and their associated abstract +data types. This section introduces ODP APIs at a conceptual level. +The second provides a tutorial on the programming model(s) +supported by ODP, paying particular attention to the event model as this +represents the preferred structure for most ODP applications. This section +builds on the concepts introduced in the first section and shows how ODP +applications are structured to best realize the three ODP design goals +mentioned earlier. The third section provides a more detailed overview of +the major ODP API components and is designed to serve as a companion to the +full reference specification for each API. The latter is intended to be used +by ODP application programmers, as well as implementers, to understand the +precise syntax and semantics of each API. + +== ODP API Concepts == +ODP programs are built around several conceptual structures that every +appliation programmer needs to be familiar with to use ODP effectively. The +main ODP concepts are: +Thread, Event, Queue, Pool, Shared Memory, Buffer, Packet, PktIO, Timer, +and Synchronizer. + +=== Thread === +The thread is the fundamental programming unit in ODP. ODP applications are +organized into a collection of threads that perform the work that the +application is designed to do. ODP threads may or may not share memory with +other threads--that is up to the implementation. Threads come in two "flavors": +control and worker, that are represented by the abstract type ++odp_thread_type_t+. + +A control thread is a supervisory thread that organizes +the operation of worker threads. Worker threads, by contrast, exist to +perform the main processing logic of the application and employ a run to +completion model. Worker threads, in particular, are intended to operate on +dedicated processing cores, especially in many core proessing environments, +however a given implementation may multitask multiple threads on a single +core if desired (typically on smaller and lower performance target +environments). + +In addition to thread types, threads have associated _attributes_ such as +_thread mask_ and _scheduler group_ that determine where they can run and +the type of work that they can handle. These will be discussed in greater +detail later. + +=== Event === +Events are what threads process to perform their work. Events can represent +new work, such as the arrival of a packet that needs to be processed, or they +can represent the completion of requests that have executed asynchronously. +Events can also represent notifications of the passage of time, or of status +changes in various components of interest to the application. Events have an +event type that describes what it represents. Threads can create new events +or consume events processed by them, or they can perform some processing on +an event and then pass it along to another component for further processing. +References to events are via handles of abstract type +odp_event_t+. Cast +functions are provided to convert these into specific handles of the +appropriate type represented by the event. + +=== Queue === +A queue is a message passing channel that holds events. Events can be +added to a queue via enqueue operations or removed from a queue via dequeue +operations. The endpoints of a queue will vary depending on how it is used. +Queues come in two major types: polled and scheduled, which will be +discussed in more detail when the event model is introduced. Queues may also +have an associated context, which represents a persistent state for all +events that make use of it. These states are what permit threads to perform +stateful processing on events as well as stateless processing. + +Queues are represented by handles of abstract type +odp_queue_t+. + +=== Pool === +A pool is a shared memory area from which elements may be drawn. Pools +represent the backing store for events, among other things. Pools are +typically created and destroyed by the application during initialization and +termination, respectively, and then used during processing. Pools may be +used by ODP components exclusively, by applications exclusively, or their +use may be shared between the two. Pools have an associated type that +characterizes the elements that they contain. The two most important pool types +are Buffer and Packet. + +Pools are represented by handles of abstract type +odp_pool_t+. + +=== Shared Memory === +Shared memory represents raw blocks of storage that are sharable between +threads. They are the building blocks of pools but can be used directly by +ODP applications if desired. + +Shared memory is represented by handles of abstract type +odp_shm_t+. + +=== Buffer === +A buffer is a fixed sized block of shared storage that is used by ODP +components and/or applications to realize their function. Buffers contain +zero or more bytes of application data as well as system maintained +metadata that provide information about the buffer, such as its size or the +pool it was allocated from. Metadata is an important ODP concept because it +allows for arbitrary amounts of side information to be associated with an +ODP object. Most ODP objects have assocaited metadata and this metadata is +manipulated via accessor functions that act as getters and setters for +this information. Getter acces functions permit an application to read +a metadata item, while setter access functions permit an application to write +a metadata item. Note that some metadata is inherently read only and thus +no setter is provided to manipulate it. When object have multiple metadata +items, each has its own associated getter and/or setter access function to +inspect or manipulate it. + +Buffers are represened by handles of abstract type +odp_buffer_t+. + +=== Packet === +Packets are received and transmitted via I/O interfaces and represent +the basic data that data plane applications manipulate. +Packets are drawn from pools of type +ODP_POOL_PACKET+. +Unlike buffers, which are simple objects, +ODP packets have a rich set of semantics that permit their inspection +and manipulation in complex ways to be described later. Packets also support +a rich set of metadata as well as user metadata. User metadata permits +applications to associate an application-determined amount of side information +with each packet for its own use. + +Packets are represented by handles of abstract type +odp_packet_t+. + +=== PktIO === +PktIO is how ODP represents I/O interfaces. A pktio object is a logical +port capable of receiving and/or transmitting packets. This may be directly +supported by the underlying platform as an integrated feature, +or may represent a device attached via a PCIE or other bus. + +PktIOs are represented by handles of abstract type +odp_pktio_t+. + +=== Timer === +Timers are how ODP applications measure and respond to the passage of time. +Timers are drawn from specialized pools called timer pools that have their +own abstract type (+odp_timer_pool_t+). Applications may have many timers +active at the same time and can set them to use either relative or absolute +time. When timers expire they create events of type +odp_timeout_t+, which +serve as notifications of timer expiration. + +=== Synchronizer === +Multiple threads operating in parallel typically require various +synchronization services to permit them to operate in a reliable and +coordinated manner. ODP provides a rich set of locks, barriers, and similar +synchronization primitives, as well as abstract types for representing various +types of atomic variables. The ODP event model also makes use of queues to +avoid the need for explicit locking in many cases. This will be discussed +in the next section. + +== ODP Components == +Building on ODP concepts, ODP offers several components that relate to the +flow of work through an ODP application. These include the Classifier, +Scheduler, and Traffic Manager. These components relate to the three +main stages of packet processing: Receive, Process, and Transmit. + +=== Classifier === +The *Classifier* provides a suite of APIs that control packet receive (RX) +processing. + +.ODP Receive Processing with Classifier +image::../images/odp_rx_processing.png[align="center"] + +The classifier provides two logically related services: +[horizontal] +Packet parsing:: Verifying and extracting structural information from a +received packet. + +Packet classification:: Applying *Pattern Matching Rules (PMRs)* to the +parsed results to assign an incoming packet to a *Class of Service (CoS)*. + +Combined, these permit incoming packets to be sorted into *flows*, which are +logically related sequences of packets that share common processing +requirements. While many data plane applications perform stateless packet +processing (_e.g.,_ for simple forwarding) others perform stateful packet +processing. Flows anchor state information relating to these groups of +packets. + +A CoS determines two variables for packets belonging to a flow: +[list] +* The pool that they will be stored in on receipt +* The queue that they will be added to for processing + +The PMRs supported by ODP permit flow determination based on combinations of +packet field values (tuples). The main advantage of classification is that on +many platforms these functions are performed in hardware, meaning that +classification occurs at line rate as packets are being received without +any explicit processing by the ODP application. + +Note that the use of the classifier is optional. Applications may directly +receive packets from a corresponding PktIO input queue via direct polling +if they choose. + +=== Scheduler === +The *Scheduler* provides a suite of APIs that control scalabable event +processing. + +.ODP Scheduler and Event Processing +image::../images/odp_scheduling.png[align="center"] + +The Scheduler is responsible for selecting and dispatching one or more events +to a requesting thread. Event selection is based on several factors involving +both the queues containing schedulable events and the thread making an ++odp_schedule()+ or +odp_schedule_multi()+ call. + +ODP queues have a _scheduling priority_ that determines how urgently events +on them should be processed relative to events contained in other queues. +Queues also have a _scheduler group id_ associated with them that must match +the associated scheduler group _thread mask_ of the thread calling the +scheduler. This permits events to be grouped for processing into classes and +have threads that are dedicated to processing events from specified classes. +Threads can join and leave scheduler groups dynamically, permitting easy +application response to increases in demand. + +When a thread receives an event from the scheduler, it in turn can invoke +other processing engines via ODP APIs (_e.g.,_ crypto processing) that +can operate asynchronously. When such processing is complete, the result is +that a *completion event* is added to a schedulable queue where it can be +scheduled back to a thread to continue processing with the results of the +requested asynchronous operation. + +Threads themselves can enqueue events to queues for downstream processing +by other threads, permitting flexibility in how applicaitions structure +themselves to maximize concurrency. + +=== Traffic Manager === +The *Traffic Manager* provides a suite of APIs that control traffic shaping and +Quality of Service (QoS) processing for packet output. + +.ODP Transmit processing with Traffic Manager +image::../images/odp_traffic_manager.png[align="center"] + +The final stage of packet processing is to transmit it. Here, applications have +several choices. As with RX processing, applications may send packets +directly to PktIO TX queues for direct transmission. Often, however, +applications need to perform traffic shaping and related +*Quality of Service (QoS)* processing on the packets comprising a flow as part +of transmit processing. To handle this need, ODP provides a suite of +*Traffic Manager* APIs that permit programmatic establishment of arbiters, +shapers, etc. that control output packet processing to achieve desired QoS +goals. Again, the advantage here is that on many platforms traffic management +functions are implemented in hardware, permitting transparent offload of +this work. + Glossary -------- [glossary] @@ -28,7 +425,6 @@ odp_thread:: event:: An event is a notification that can be placed in a queue. -:numbered: The include structure --------------------- Applications only include the 'include/odp.h file which includes the 'platform/<implementation name>/include/plat' files to provide a complete definition of the API on that platform.
Add a basic ODP overview to the User's Guide, providing an overview of ODP concepts, components, etc. Included are a number of diagrams covering component structure as well as packet RX, event scheduling, and packet TX processing. Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> --- doc/users-guide/users-guide.adoc | 400 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 398 insertions(+), 2 deletions(-)