| Message ID | 1422623405-11667-1-git-send-email-petri.savolainen@linaro.org |
|---|---|
| State | New |
| Headers | show |
I really can't concur with this proposed design. The fundamental issue here is the proper relationship between applications and implementations with respect to packet storage. Packets have an overall length, and this is within the purview of the application, since it understands the type of traffic it is looking to process. Length is a property that packets have "on the wire" and is independent of how packets may be stored within a processing node. Segments are always an implementation construct and exist for the convenience of an implementation. Segments do not exist on the wire and are not part of any inherent (i.e., platform-independent) packet structure. To assert application control over packet segmentation is to say that the application is controlling the implementation of packet storage. This is a fundamental departure from the API/Implementation separation paradigm that ODP is promoting. If an application wishes to do this it is leaving no room for HW offload or innovation in this area--it's just using the HW as a raw block manager and doing everything itself in SW. It is understood that the existence of segmentation imposes some processing overhead on SW to the extent that SW must deal with the "seams" in a packet addressability that results from segmentation. There are two ways to address this. The first is to recognize that in the data plane the vast bulk of processing is on packet headers rather than payload and that HW is aware of this fact, which is why HW designed for packet processing invariably uses segment sizes large enough to contain all of the packet headers within the first packet segment for the vast majority of packets of interest. In the spec we worked on last year we stated that ODP would require a minimum segment size of 256 bytes so that applications would have assurance regarding this, and no surveyed platforms had issues with that. The second means of addressing this problem is to allow applications to explicitly request unsegmented pools. While recognizing that not all platforms can provide unsegmented pools efficiently, the idea behind unsegmented pools was that this would aid applications that for whatever reason could not deal with packet segments. So in this model the application has two choices. It can either work with an implementation-chosen segment size, understanding that that size will be large enough so that it need not worry about segment boundaries in packet headers for almost all packets of interest, or it can request that the pool be unsegmented so that the entire packet is always a single segment. If you believe that this model is insufficient, I would like to understand, with use cases, why that is so. I would also like to hear from SoC vendors looking to implement ODP whether they can efficiently support arbitrary application-specified segment sizes for packet processing. On Fri, Jan 30, 2015 at 7:10 AM, Petri Savolainen < petri.savolainen@linaro.org> wrote: > Completed odp_pool_param_t definition with packet pool parameters. > Parameter definition is close to what we are using already. Segment > min length, segment min alignment and number of segments. > > Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> > --- > include/odp/api/pool.h | 20 +++++++++++++++----- > 1 file changed, 15 insertions(+), 5 deletions(-) > > diff --git a/include/odp/api/pool.h b/include/odp/api/pool.h > index 1582102..e407704 100644 > --- a/include/odp/api/pool.h > +++ b/include/odp/api/pool.h > @@ -61,13 +61,23 @@ typedef struct odp_pool_param_t { > of 8. */ > uint32_t num; /**< Number of buffers in the pool > */ > } buf; > -/* Reserved for packet and timeout specific params > struct { > - uint32_t seg_size; > - uint32_t seg_align; > - uint32_t num; > + uint32_t seg_len; /**< Minimum packet segment > buffer > + length in bytes. It > includes > + possible head-/tailroom > bytes. > + Use 0 for default length. > */ > + uint32_t seg_align; /**< Minimum packet segment > buffer > + alignment in bytes. Valid > + values are powers of two. > Use 0 > + for default alignment. > Default > + will always be a multiple > of 8. > + */ > + uint32_t seg_num; /**< Number of packet segments > in > + the pool. It's also the > maximum > + number of packets, since > each > + packet consist of at > least one > + segment. */ > } pkt; > -*/ > struct { > uint32_t __res1; /* Keep struct identical to buf, > */ > uint32_t __res2; /* until pool implementation is > fixed*/ > -- > 2.2.2 > > > _______________________________________________ > lng-odp mailing list > lng-odp@lists.linaro.org > http://lists.linaro.org/mailman/listinfo/lng-odp >
One important aspect of ODP is a hardware abstraction, the ODP API is supposed to hide implementations details such as how buffers are managed (segmentation is one implementation detail that we allow to leak through as this is common and very expensive to hide from the application). I recently heard of a 40G NIC which doesn't use (user visible) buffer pools at all. You just pass a large (shared) memory region to the NIC and it carves up suitable buffers as needed. Needless to say, DPDK has problem with that. But ODP shouldn't. On 31 January 2015 at 01:22, Bill Fischofer <bill.fischofer@linaro.org> wrote: > I really can't concur with this proposed design. The fundamental issue here > is the proper relationship between applications and implementations with > respect to packet storage. Packets have an overall length, and this is > within the purview of the application, since it understands the type of > traffic it is looking to process. Length is a property that packets have > "on the wire" and is independent of how packets may be stored within a > processing node. > > Segments are always an implementation construct and exist for the > convenience of an implementation. Segments do not exist on the wire and are > not part of any inherent (i.e., platform-independent) packet structure. To > assert application control over packet segmentation is to say that the > application is controlling the implementation of packet storage. This is a > fundamental departure from the API/Implementation separation paradigm that > ODP is promoting. If an application wishes to do this it is leaving no > room for HW offload or innovation in this area--it's just using the HW as a > raw block manager and doing everything itself in SW. > > It is understood that the existence of segmentation imposes some processing > overhead on SW to the extent that SW must deal with the "seams" in a packet > addressability that results from segmentation. There are two ways to > address this. > > The first is to recognize that in the data plane the vast bulk of processing > is on packet headers rather than payload and that HW is aware of this fact, > which is why HW designed for packet processing invariably uses segment sizes > large enough to contain all of the packet headers within the first packet > segment for the vast majority of packets of interest. In the spec we worked > on last year we stated that ODP would require a minimum segment size of 256 > bytes so that applications would have assurance regarding this, and no > surveyed platforms had issues with that. > > The second means of addressing this problem is to allow applications to > explicitly request unsegmented pools. While recognizing that not all > platforms can provide unsegmented pools efficiently, the idea behind > unsegmented pools was that this would aid applications that for whatever > reason could not deal with packet segments. > > So in this model the application has two choices. It can either work with > an implementation-chosen segment size, understanding that that size will be > large enough so that it need not worry about segment boundaries in packet > headers for almost all packets of interest, or it can request that the pool > be unsegmented so that the entire packet is always a single segment. > > If you believe that this model is insufficient, I would like to understand, > with use cases, why that is so. I would also like to hear from SoC vendors > looking to implement ODP whether they can efficiently support arbitrary > application-specified segment sizes for packet processing. > > > On Fri, Jan 30, 2015 at 7:10 AM, Petri Savolainen > <petri.savolainen@linaro.org> wrote: >> >> Completed odp_pool_param_t definition with packet pool parameters. >> Parameter definition is close to what we are using already. Segment >> min length, segment min alignment and number of segments. >> >> Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> >> --- >> include/odp/api/pool.h | 20 +++++++++++++++----- >> 1 file changed, 15 insertions(+), 5 deletions(-) >> >> diff --git a/include/odp/api/pool.h b/include/odp/api/pool.h >> index 1582102..e407704 100644 >> --- a/include/odp/api/pool.h >> +++ b/include/odp/api/pool.h >> @@ -61,13 +61,23 @@ typedef struct odp_pool_param_t { >> of 8. */ >> uint32_t num; /**< Number of buffers in the pool >> */ >> } buf; >> -/* Reserved for packet and timeout specific params >> struct { >> - uint32_t seg_size; >> - uint32_t seg_align; >> - uint32_t num; >> + uint32_t seg_len; /**< Minimum packet segment >> buffer >> + length in bytes. It >> includes >> + possible head-/tailroom >> bytes. >> + Use 0 for default length. >> */ >> + uint32_t seg_align; /**< Minimum packet segment >> buffer >> + alignment in bytes. Valid >> + values are powers of two. >> Use 0 >> + for default alignment. >> Default >> + will always be a multiple >> of 8. >> + */ >> + uint32_t seg_num; /**< Number of packet segments >> in >> + the pool. It's also the >> maximum >> + number of packets, since >> each >> + packet consist of at >> least one >> + segment. */ >> } pkt; >> -*/ >> struct { >> uint32_t __res1; /* Keep struct identical to buf, >> */ >> uint32_t __res2; /* until pool implementation is >> fixed*/ >> -- >> 2.2.2 >> >> >> _______________________________________________ >> lng-odp mailing list >> lng-odp@lists.linaro.org >> http://lists.linaro.org/mailman/listinfo/lng-odp > > > > _______________________________________________ > lng-odp mailing list > lng-odp@lists.linaro.org > http://lists.linaro.org/mailman/listinfo/lng-odp >
On 2 February 2015 at 10:41, Savolainen, Petri (NSN - FI/Espoo) <petri.savolainen@nsn.com> wrote: > Hi, > > An ODP application is SW running on cores (not HW or firmware). If there's no SW, there's no need for ODP API. Has anyone disputed this? > Application receives odp_packets, which consist of odp_packet_segs. Packets are a primary concept in networking. Segments are not. Segments exists in specific packet processing implementations (hardware+drivers+application). In some implementations, each packet may always be stored in one consecutive buffer (segment?). In order implementations, packets may have to be stored in multiple segments. It may depend on the size of each specific packet as well. Etc. > From application point of view, each segment is a block of contiguous memory (implementation can be anything as long as application can access it contiguously). Application has the knowledge on incoming packets and SW processing of those. Segmentation is a trade-off between linear memory processing (good performance) and memory usage. Now you are inventing a new reason for ODP to support segmentation. Until this point, support for segmentation has been to handle hardware designs which cannot (efficiently) support contiguous packets for all packet sizes. >These parameters give application the change to tune the trade-off in each use case. > > For example, application may > - receive 49% 64 byte packets, 49% 1500 byte packets and 2% 9kB packets > - read up to 300 bytes into a packet > - add upto 54 bytes of tunnel headers in front of the packet > => ask first min seg len 354 bytes, that would fit all accesses. Implementation rounds that up to 384 bytes (full cache line) > => run tests and notice that performance is good, but we need to lower the memory usage of this pool > => drop min seg len to 200 bytes, implementation rounds that to 256 > => run tests, performance is still good enough and pool uses less memory (which can be then used for something else) > > How implementation could do this trade-off analysis behalf of the user? The application can provide the expected packet size distribution to the ODP implementation as hints for optimal memory utilization (and optimal performance). I don't see this as equivalent to specifying which segment sizes the ODP implementation *must* use. I also thought that the class-of-service support in ODP classifier API had the support for selecting the relevant pool based on e.g. ingress packet size. Is this solution not complete? What is missing? The application should not specify segment sizes as requirements. Actual segment sizes used are selected by the ODP implementation (possibly pre-selected by the hardware design itself). If you are pushing for a solution, you need to specify the problem first. There could be other (better) solutions to the problem. The power of collaboration is that many brains can work together on a problem. > > > + uint32_t seg_len; /**< Minimum packet segment buffer > + length in bytes. It includes > + possible head-/tailroom bytes. > + Use 0 for default length. */ > > In addition to this, we have lower and upper config limits (ODP_CONFIG_PACKET_BUF_LEN_XXX need to be updated also). If implementation can support only one segment len, it will be documented by those min/max limits being the same. > > > -Petri > > > >> -----Original Message----- >> From: ext Ola Liljedahl [mailto:ola.liljedahl@linaro.org] >> Sent: Sunday, February 01, 2015 12:06 AM >> To: Bill Fischofer >> Cc: Petri Savolainen; LNG ODP Mailman List >> Subject: Re: [lng-odp] [PATCH 1/2] api: pool: Added packet pool parameters >> >> One important aspect of ODP is a hardware abstraction, the ODP API is >> supposed to hide implementations details such as how buffers are >> managed (segmentation is one implementation detail that we allow to >> leak through as this is common and very expensive to hide from the >> application). I recently heard of a 40G NIC which doesn't use (user >> visible) buffer pools at all. You just pass a large (shared) memory >> region to the NIC and it carves up suitable buffers as needed. >> Needless to say, DPDK has problem with that. But ODP shouldn't. >> >> On 31 January 2015 at 01:22, Bill Fischofer <bill.fischofer@linaro.org> >> wrote: >> > I really can't concur with this proposed design. The fundamental issue >> here >> > is the proper relationship between applications and implementations with >> > respect to packet storage. Packets have an overall length, and this is >> > within the purview of the application, since it understands the type of >> > traffic it is looking to process. Length is a property that packets >> have >> > "on the wire" and is independent of how packets may be stored within a >> > processing node. >> > >> > Segments are always an implementation construct and exist for the >> > convenience of an implementation. Segments do not exist on the wire and >> are >> > not part of any inherent (i.e., platform-independent) packet structure. >> To >> > assert application control over packet segmentation is to say that the >> > application is controlling the implementation of packet storage. This is >> a >> > fundamental departure from the API/Implementation separation paradigm >> that >> > ODP is promoting. If an application wishes to do this it is leaving no >> > room for HW offload or innovation in this area--it's just using the HW >> as a >> > raw block manager and doing everything itself in SW. >> > >> > It is understood that the existence of segmentation imposes some >> processing >> > overhead on SW to the extent that SW must deal with the "seams" in a >> packet >> > addressability that results from segmentation. There are two ways to >> > address this. >> > >> > The first is to recognize that in the data plane the vast bulk of >> processing >> > is on packet headers rather than payload and that HW is aware of this >> fact, >> > which is why HW designed for packet processing invariably uses segment >> sizes >> > large enough to contain all of the packet headers within the first >> packet >> > segment for the vast majority of packets of interest. In the spec we >> worked >> > on last year we stated that ODP would require a minimum segment size of >> 256 >> > bytes so that applications would have assurance regarding this, and no >> > surveyed platforms had issues with that. >> > >> > The second means of addressing this problem is to allow applications to >> > explicitly request unsegmented pools. While recognizing that not all >> > platforms can provide unsegmented pools efficiently, the idea behind >> > unsegmented pools was that this would aid applications that for whatever >> > reason could not deal with packet segments. >> > >> > So in this model the application has two choices. It can either work >> with >> > an implementation-chosen segment size, understanding that that size will >> be >> > large enough so that it need not worry about segment boundaries in >> packet >> > headers for almost all packets of interest, or it can request that the >> pool >> > be unsegmented so that the entire packet is always a single segment. >> > >> > If you believe that this model is insufficient, I would like to >> understand, >> > with use cases, why that is so. I would also like to hear from SoC >> vendors >> > looking to implement ODP whether they can efficiently support arbitrary >> > application-specified segment sizes for packet processing. >> > >> > >> > On Fri, Jan 30, 2015 at 7:10 AM, Petri Savolainen >> > <petri.savolainen@linaro.org> wrote: >> >> >> >> Completed odp_pool_param_t definition with packet pool parameters. >> >> Parameter definition is close to what we are using already. Segment >> >> min length, segment min alignment and number of segments. >> >> >> >> Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> >> >> --- >> >> include/odp/api/pool.h | 20 +++++++++++++++----- >> >> 1 file changed, 15 insertions(+), 5 deletions(-) >> >> >> >> diff --git a/include/odp/api/pool.h b/include/odp/api/pool.h >> >> index 1582102..e407704 100644 >> >> --- a/include/odp/api/pool.h >> >> +++ b/include/odp/api/pool.h >> >> @@ -61,13 +61,23 @@ typedef struct odp_pool_param_t { >> >> of 8. */ >> >> uint32_t num; /**< Number of buffers in the >> pool >> >> */ >> >> } buf; >> >> -/* Reserved for packet and timeout specific params >> >> struct { >> >> - uint32_t seg_size; >> >> - uint32_t seg_align; >> >> - uint32_t num; >> >> + uint32_t seg_len; /**< Minimum packet segment >> >> buffer >> >> + length in bytes. It >> >> includes >> >> + possible head- >> /tailroom >> >> bytes. >> >> + Use 0 for default >> length. >> >> */ >> >> + uint32_t seg_align; /**< Minimum packet segment >> >> buffer >> >> + alignment in bytes. >> Valid >> >> + values are powers of >> two. >> >> Use 0 >> >> + for default alignment. >> >> Default >> >> + will always be a >> multiple >> >> of 8. >> >> + */ >> >> + uint32_t seg_num; /**< Number of packet >> segments >> >> in >> >> + the pool. It's also >> the >> >> maximum >> >> + number of packets, >> since >> >> each >> >> + packet consist of at >> >> least one >> >> + segment. */ >> >> } pkt; >> >> -*/ >> >> struct { >> >> uint32_t __res1; /* Keep struct identical to >> buf, >> >> */ >> >> uint32_t __res2; /* until pool implementation >> is >> >> fixed*/ >> >> -- >> >> 2.2.2 >> >> >> >> >> >> _______________________________________________ >> >> lng-odp mailing list >> >> lng-odp@lists.linaro.org >> >> http://lists.linaro.org/mailman/listinfo/lng-odp >> > >> > >> > >> > _______________________________________________ >> > lng-odp mailing list >> > lng-odp@lists.linaro.org >> > http://lists.linaro.org/mailman/listinfo/lng-odp >> >
The model referred to here involves the use of unsegmented pools combined with classification rules based on packet length, as originally suggested by Barry as what Tilera does. It's why we added the pkt_len PMR term. If an application wishes it can create multiple unsegmented pools of different sizes and use classification to sort arriving packets into them based on their length. This give you exactly what you want: Application does not have to deal with segments and good memory utilization. This model is easy to understand, and independent of the underlying platform. Some platforms may not be able to provide this level of control, but ODP doesn't say that every platform must support every ODP application or API in an optimal manner. For platforms that do segmentation in HW, this is properly the domain of the implementation. There's no point in SW asking HW to do something it cannot do, so you'd be left with the paradigm that SW queries HW limits and then tells HW to do what it's going to do anyway, which isn't terribly useful. The pool parameters tell the implementation two key things: How many packets the pool should contain, and how large they are expected to be (on average). That's what allows the implementation to calculate how much storage needs to be reserved. Other items such as alignment just refine this essential information. The number of segments that these packets may be broken into is again something that the implementation knows, not the application, because its the implementation that knows how packets are actually stored on its platform. On Mon, Feb 2, 2015 at 4:55 AM, Ola Liljedahl <ola.liljedahl@linaro.org> wrote: > On 2 February 2015 at 10:41, Savolainen, Petri (NSN - FI/Espoo) > <petri.savolainen@nsn.com> wrote: > > Hi, > > > > An ODP application is SW running on cores (not HW or firmware). If > there's no SW, there's no need for ODP API. > Has anyone disputed this? > > > Application receives odp_packets, which consist of odp_packet_segs. > Packets are a primary concept in networking. Segments are not. > Segments exists in specific packet processing implementations > (hardware+drivers+application). In some implementations, each packet > may always be stored in one consecutive buffer (segment?). In order > implementations, packets may have to be stored in multiple segments. > It may depend on the size of each specific packet as well. Etc. > > > From application point of view, each segment is a block of contiguous > memory (implementation can be anything as long as application can access it > contiguously). Application has the knowledge on incoming packets and SW > processing of those. Segmentation is a trade-off between linear memory > processing (good performance) and memory usage. > Now you are inventing a new reason for ODP to support segmentation. > Until this point, support for segmentation has been to handle hardware > designs which cannot (efficiently) support contiguous packets for all > packet sizes. > > >These parameters give application the change to tune the trade-off in > each use case. > > > > For example, application may > > - receive 49% 64 byte packets, 49% 1500 byte packets and 2% 9kB packets > > - read up to 300 bytes into a packet > > - add upto 54 bytes of tunnel headers in front of the packet > > => ask first min seg len 354 bytes, that would fit all accesses. > Implementation rounds that up to 384 bytes (full cache line) > > => run tests and notice that performance is good, but we need to lower > the memory usage of this pool > > => drop min seg len to 200 bytes, implementation rounds that to 256 > > => run tests, performance is still good enough and pool uses less memory > (which can be then used for something else) > > > > How implementation could do this trade-off analysis behalf of the user? > The application can provide the expected packet size distribution to > the ODP implementation as hints for optimal memory utilization (and > optimal performance). I don't see this as equivalent to specifying > which segment sizes the ODP implementation *must* use. I also thought > that the class-of-service support in ODP classifier API had the > support for selecting the relevant pool based on e.g. ingress packet > size. Is this solution not complete? What is missing? > > The application should not specify segment sizes as requirements. > Actual segment sizes used are selected by the ODP implementation > (possibly pre-selected by the hardware design itself). > > If you are pushing for a solution, you need to specify the problem > first. There could be other (better) solutions to the problem. The > power of collaboration is that many brains can work together on a > problem. > > > > > > > + uint32_t seg_len; /**< Minimum packet segment > buffer > > + length in bytes. It > includes > > + possible head-/tailroom > bytes. > > + Use 0 for default > length. */ > > > > In addition to this, we have lower and upper config limits > (ODP_CONFIG_PACKET_BUF_LEN_XXX need to be updated also). If implementation > can support only one segment len, it will be documented by those min/max > limits being the same. > > > > > > -Petri > > > > > > > >> -----Original Message----- > >> From: ext Ola Liljedahl [mailto:ola.liljedahl@linaro.org] > >> Sent: Sunday, February 01, 2015 12:06 AM > >> To: Bill Fischofer > >> Cc: Petri Savolainen; LNG ODP Mailman List > >> Subject: Re: [lng-odp] [PATCH 1/2] api: pool: Added packet pool > parameters > >> > >> One important aspect of ODP is a hardware abstraction, the ODP API is > >> supposed to hide implementations details such as how buffers are > >> managed (segmentation is one implementation detail that we allow to > >> leak through as this is common and very expensive to hide from the > >> application). I recently heard of a 40G NIC which doesn't use (user > >> visible) buffer pools at all. You just pass a large (shared) memory > >> region to the NIC and it carves up suitable buffers as needed. > >> Needless to say, DPDK has problem with that. But ODP shouldn't. > >> > >> On 31 January 2015 at 01:22, Bill Fischofer <bill.fischofer@linaro.org> > >> wrote: > >> > I really can't concur with this proposed design. The fundamental > issue > >> here > >> > is the proper relationship between applications and implementations > with > >> > respect to packet storage. Packets have an overall length, and this > is > >> > within the purview of the application, since it understands the type > of > >> > traffic it is looking to process. Length is a property that packets > >> have > >> > "on the wire" and is independent of how packets may be stored within a > >> > processing node. > >> > > >> > Segments are always an implementation construct and exist for the > >> > convenience of an implementation. Segments do not exist on the wire > and > >> are > >> > not part of any inherent (i.e., platform-independent) packet > structure. > >> To > >> > assert application control over packet segmentation is to say that the > >> > application is controlling the implementation of packet storage. This > is > >> a > >> > fundamental departure from the API/Implementation separation paradigm > >> that > >> > ODP is promoting. If an application wishes to do this it is leaving > no > >> > room for HW offload or innovation in this area--it's just using the HW > >> as a > >> > raw block manager and doing everything itself in SW. > >> > > >> > It is understood that the existence of segmentation imposes some > >> processing > >> > overhead on SW to the extent that SW must deal with the "seams" in a > >> packet > >> > addressability that results from segmentation. There are two ways to > >> > address this. > >> > > >> > The first is to recognize that in the data plane the vast bulk of > >> processing > >> > is on packet headers rather than payload and that HW is aware of this > >> fact, > >> > which is why HW designed for packet processing invariably uses segment > >> sizes > >> > large enough to contain all of the packet headers within the first > >> packet > >> > segment for the vast majority of packets of interest. In the spec we > >> worked > >> > on last year we stated that ODP would require a minimum segment size > of > >> 256 > >> > bytes so that applications would have assurance regarding this, and no > >> > surveyed platforms had issues with that. > >> > > >> > The second means of addressing this problem is to allow applications > to > >> > explicitly request unsegmented pools. While recognizing that not all > >> > platforms can provide unsegmented pools efficiently, the idea behind > >> > unsegmented pools was that this would aid applications that for > whatever > >> > reason could not deal with packet segments. > >> > > >> > So in this model the application has two choices. It can either work > >> with > >> > an implementation-chosen segment size, understanding that that size > will > >> be > >> > large enough so that it need not worry about segment boundaries in > >> packet > >> > headers for almost all packets of interest, or it can request that the > >> pool > >> > be unsegmented so that the entire packet is always a single segment. > >> > > >> > If you believe that this model is insufficient, I would like to > >> understand, > >> > with use cases, why that is so. I would also like to hear from SoC > >> vendors > >> > looking to implement ODP whether they can efficiently support > arbitrary > >> > application-specified segment sizes for packet processing. > >> > > >> > > >> > On Fri, Jan 30, 2015 at 7:10 AM, Petri Savolainen > >> > <petri.savolainen@linaro.org> wrote: > >> >> > >> >> Completed odp_pool_param_t definition with packet pool parameters. > >> >> Parameter definition is close to what we are using already. Segment > >> >> min length, segment min alignment and number of segments. > >> >> > >> >> Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> > >> >> --- > >> >> include/odp/api/pool.h | 20 +++++++++++++++----- > >> >> 1 file changed, 15 insertions(+), 5 deletions(-) > >> >> > >> >> diff --git a/include/odp/api/pool.h b/include/odp/api/pool.h > >> >> index 1582102..e407704 100644 > >> >> --- a/include/odp/api/pool.h > >> >> +++ b/include/odp/api/pool.h > >> >> @@ -61,13 +61,23 @@ typedef struct odp_pool_param_t { > >> >> of 8. */ > >> >> uint32_t num; /**< Number of buffers in the > >> pool > >> >> */ > >> >> } buf; > >> >> -/* Reserved for packet and timeout specific params > >> >> struct { > >> >> - uint32_t seg_size; > >> >> - uint32_t seg_align; > >> >> - uint32_t num; > >> >> + uint32_t seg_len; /**< Minimum packet > segment > >> >> buffer > >> >> + length in bytes. It > >> >> includes > >> >> + possible head- > >> /tailroom > >> >> bytes. > >> >> + Use 0 for default > >> length. > >> >> */ > >> >> + uint32_t seg_align; /**< Minimum packet > segment > >> >> buffer > >> >> + alignment in bytes. > >> Valid > >> >> + values are powers of > >> two. > >> >> Use 0 > >> >> + for default > alignment. > >> >> Default > >> >> + will always be a > >> multiple > >> >> of 8. > >> >> + */ > >> >> + uint32_t seg_num; /**< Number of packet > >> segments > >> >> in > >> >> + the pool. It's also > >> the > >> >> maximum > >> >> + number of packets, > >> since > >> >> each > >> >> + packet consist of at > >> >> least one > >> >> + segment. */ > >> >> } pkt; > >> >> -*/ > >> >> struct { > >> >> uint32_t __res1; /* Keep struct identical to > >> buf, > >> >> */ > >> >> uint32_t __res2; /* until pool implementation > >> is > >> >> fixed*/ > >> >> -- > >> >> 2.2.2 > >> >> > >> >> > >> >> _______________________________________________ > >> >> lng-odp mailing list > >> >> lng-odp@lists.linaro.org > >> >> http://lists.linaro.org/mailman/listinfo/lng-odp > >> > > >> > > >> > > >> > _______________________________________________ > >> > lng-odp mailing list > >> > lng-odp@lists.linaro.org > >> > http://lists.linaro.org/mailman/listinfo/lng-odp > >> > >
While ODP doesn't say that every platform must support the API in an optimal manner I think the APIs should target to be possible to implement on a broad range of platforms . "Classification" on packet length, as defined by the ODP API, is not supported on current FSL platforms. However, the HW has this feature of selecting the buffer pool depending on the frame size and this feature was used when odp_pktio_open(dev, pool) was possible multiple times for a given device, but now this model is dropped. I think there is a need to align the API here with the currently involved platforms. Thanks, Alex On 2 February 2015 at 14:13, Bill Fischofer <bill.fischofer@linaro.org> wrote: > The model referred to here involves the use of unsegmented pools combined > with classification rules based on packet length, as originally suggested > by Barry as what Tilera does. It's why we added the pkt_len PMR term. If > an application wishes it can create multiple unsegmented pools of different > sizes and use classification to sort arriving packets into them based on > their length. This give you exactly what you want: Application does not > have to deal with segments and good memory utilization. This model is easy > to understand, and independent of the underlying platform. Some platforms > may not be able to provide this level of control, but ODP doesn't say that > every platform must support every ODP application or API in an optimal > manner. > > For platforms that do segmentation in HW, this is properly the domain of > the implementation. There's no point in SW asking HW to do something it > cannot do, so you'd be left with the paradigm that SW queries HW limits and > then tells HW to do what it's going to do anyway, which isn't terribly > useful. > > The pool parameters tell the implementation two key things: How many > packets the pool should contain, and how large they are expected to be (on > average). That's what allows the implementation to calculate how much > storage needs to be reserved. Other items such as alignment just refine > this essential information. The number of segments that these packets may > be broken into is again something that the implementation knows, not the > application, because its the implementation that knows how packets are > actually stored on its platform. > > > > On Mon, Feb 2, 2015 at 4:55 AM, Ola Liljedahl <ola.liljedahl@linaro.org> > wrote: > >> On 2 February 2015 at 10:41, Savolainen, Petri (NSN - FI/Espoo) >> <petri.savolainen@nsn.com> wrote: >> > Hi, >> > >> > An ODP application is SW running on cores (not HW or firmware). If >> there's no SW, there's no need for ODP API. >> Has anyone disputed this? >> >> > Application receives odp_packets, which consist of odp_packet_segs. >> Packets are a primary concept in networking. Segments are not. >> Segments exists in specific packet processing implementations >> (hardware+drivers+application). In some implementations, each packet >> may always be stored in one consecutive buffer (segment?). In order >> implementations, packets may have to be stored in multiple segments. >> It may depend on the size of each specific packet as well. Etc. >> >> > From application point of view, each segment is a block of contiguous >> memory (implementation can be anything as long as application can access it >> contiguously). Application has the knowledge on incoming packets and SW >> processing of those. Segmentation is a trade-off between linear memory >> processing (good performance) and memory usage. >> Now you are inventing a new reason for ODP to support segmentation. >> Until this point, support for segmentation has been to handle hardware >> designs which cannot (efficiently) support contiguous packets for all >> packet sizes. >> >> >These parameters give application the change to tune the trade-off in >> each use case. >> > >> > For example, application may >> > - receive 49% 64 byte packets, 49% 1500 byte packets and 2% 9kB packets >> > - read up to 300 bytes into a packet >> > - add upto 54 bytes of tunnel headers in front of the packet >> > => ask first min seg len 354 bytes, that would fit all accesses. >> Implementation rounds that up to 384 bytes (full cache line) >> > => run tests and notice that performance is good, but we need to lower >> the memory usage of this pool >> > => drop min seg len to 200 bytes, implementation rounds that to 256 >> > => run tests, performance is still good enough and pool uses less >> memory (which can be then used for something else) >> > >> > How implementation could do this trade-off analysis behalf of the user? >> The application can provide the expected packet size distribution to >> the ODP implementation as hints for optimal memory utilization (and >> optimal performance). I don't see this as equivalent to specifying >> which segment sizes the ODP implementation *must* use. I also thought >> that the class-of-service support in ODP classifier API had the >> support for selecting the relevant pool based on e.g. ingress packet >> size. Is this solution not complete? What is missing? >> >> The application should not specify segment sizes as requirements. >> Actual segment sizes used are selected by the ODP implementation >> (possibly pre-selected by the hardware design itself). >> >> If you are pushing for a solution, you need to specify the problem >> first. There could be other (better) solutions to the problem. The >> power of collaboration is that many brains can work together on a >> problem. >> >> > >> > >> > + uint32_t seg_len; /**< Minimum packet segment >> buffer >> > + length in bytes. It >> includes >> > + possible >> head-/tailroom bytes. >> > + Use 0 for default >> length. */ >> > >> > In addition to this, we have lower and upper config limits >> (ODP_CONFIG_PACKET_BUF_LEN_XXX need to be updated also). If implementation >> can support only one segment len, it will be documented by those min/max >> limits being the same. >> > >> > >> > -Petri >> > >> > >> > >> >> -----Original Message----- >> >> From: ext Ola Liljedahl [mailto:ola.liljedahl@linaro.org] >> >> Sent: Sunday, February 01, 2015 12:06 AM >> >> To: Bill Fischofer >> >> Cc: Petri Savolainen; LNG ODP Mailman List >> >> Subject: Re: [lng-odp] [PATCH 1/2] api: pool: Added packet pool >> parameters >> >> >> >> One important aspect of ODP is a hardware abstraction, the ODP API is >> >> supposed to hide implementations details such as how buffers are >> >> managed (segmentation is one implementation detail that we allow to >> >> leak through as this is common and very expensive to hide from the >> >> application). I recently heard of a 40G NIC which doesn't use (user >> >> visible) buffer pools at all. You just pass a large (shared) memory >> >> region to the NIC and it carves up suitable buffers as needed. >> >> Needless to say, DPDK has problem with that. But ODP shouldn't. >> >> >> >> On 31 January 2015 at 01:22, Bill Fischofer <bill.fischofer@linaro.org >> > >> >> wrote: >> >> > I really can't concur with this proposed design. The fundamental >> issue >> >> here >> >> > is the proper relationship between applications and implementations >> with >> >> > respect to packet storage. Packets have an overall length, and this >> is >> >> > within the purview of the application, since it understands the type >> of >> >> > traffic it is looking to process. Length is a property that packets >> >> have >> >> > "on the wire" and is independent of how packets may be stored within >> a >> >> > processing node. >> >> > >> >> > Segments are always an implementation construct and exist for the >> >> > convenience of an implementation. Segments do not exist on the wire >> and >> >> are >> >> > not part of any inherent (i.e., platform-independent) packet >> structure. >> >> To >> >> > assert application control over packet segmentation is to say that >> the >> >> > application is controlling the implementation of packet storage. >> This is >> >> a >> >> > fundamental departure from the API/Implementation separation paradigm >> >> that >> >> > ODP is promoting. If an application wishes to do this it is >> leaving no >> >> > room for HW offload or innovation in this area--it's just using the >> HW >> >> as a >> >> > raw block manager and doing everything itself in SW. >> >> > >> >> > It is understood that the existence of segmentation imposes some >> >> processing >> >> > overhead on SW to the extent that SW must deal with the "seams" in a >> >> packet >> >> > addressability that results from segmentation. There are two ways to >> >> > address this. >> >> > >> >> > The first is to recognize that in the data plane the vast bulk of >> >> processing >> >> > is on packet headers rather than payload and that HW is aware of this >> >> fact, >> >> > which is why HW designed for packet processing invariably uses >> segment >> >> sizes >> >> > large enough to contain all of the packet headers within the first >> >> packet >> >> > segment for the vast majority of packets of interest. In the spec we >> >> worked >> >> > on last year we stated that ODP would require a minimum segment size >> of >> >> 256 >> >> > bytes so that applications would have assurance regarding this, and >> no >> >> > surveyed platforms had issues with that. >> >> > >> >> > The second means of addressing this problem is to allow applications >> to >> >> > explicitly request unsegmented pools. While recognizing that not all >> >> > platforms can provide unsegmented pools efficiently, the idea behind >> >> > unsegmented pools was that this would aid applications that for >> whatever >> >> > reason could not deal with packet segments. >> >> > >> >> > So in this model the application has two choices. It can either work >> >> with >> >> > an implementation-chosen segment size, understanding that that size >> will >> >> be >> >> > large enough so that it need not worry about segment boundaries in >> >> packet >> >> > headers for almost all packets of interest, or it can request that >> the >> >> pool >> >> > be unsegmented so that the entire packet is always a single segment. >> >> > >> >> > If you believe that this model is insufficient, I would like to >> >> understand, >> >> > with use cases, why that is so. I would also like to hear from SoC >> >> vendors >> >> > looking to implement ODP whether they can efficiently support >> arbitrary >> >> > application-specified segment sizes for packet processing. >> >> > >> >> > >> >> > On Fri, Jan 30, 2015 at 7:10 AM, Petri Savolainen >> >> > <petri.savolainen@linaro.org> wrote: >> >> >> >> >> >> Completed odp_pool_param_t definition with packet pool parameters. >> >> >> Parameter definition is close to what we are using already. Segment >> >> >> min length, segment min alignment and number of segments. >> >> >> >> >> >> Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> >> >> >> --- >> >> >> include/odp/api/pool.h | 20 +++++++++++++++----- >> >> >> 1 file changed, 15 insertions(+), 5 deletions(-) >> >> >> >> >> >> diff --git a/include/odp/api/pool.h b/include/odp/api/pool.h >> >> >> index 1582102..e407704 100644 >> >> >> --- a/include/odp/api/pool.h >> >> >> +++ b/include/odp/api/pool.h >> >> >> @@ -61,13 +61,23 @@ typedef struct odp_pool_param_t { >> >> >> of 8. */ >> >> >> uint32_t num; /**< Number of buffers in >> the >> >> pool >> >> >> */ >> >> >> } buf; >> >> >> -/* Reserved for packet and timeout specific params >> >> >> struct { >> >> >> - uint32_t seg_size; >> >> >> - uint32_t seg_align; >> >> >> - uint32_t num; >> >> >> + uint32_t seg_len; /**< Minimum packet >> segment >> >> >> buffer >> >> >> + length in bytes. It >> >> >> includes >> >> >> + possible head- >> >> /tailroom >> >> >> bytes. >> >> >> + Use 0 for default >> >> length. >> >> >> */ >> >> >> + uint32_t seg_align; /**< Minimum packet >> segment >> >> >> buffer >> >> >> + alignment in bytes. >> >> Valid >> >> >> + values are powers >> of >> >> two. >> >> >> Use 0 >> >> >> + for default >> alignment. >> >> >> Default >> >> >> + will always be a >> >> multiple >> >> >> of 8. >> >> >> + */ >> >> >> + uint32_t seg_num; /**< Number of packet >> >> segments >> >> >> in >> >> >> + the pool. It's also >> >> the >> >> >> maximum >> >> >> + number of packets, >> >> since >> >> >> each >> >> >> + packet consist of >> at >> >> >> least one >> >> >> + segment. */ >> >> >> } pkt; >> >> >> -*/ >> >> >> struct { >> >> >> uint32_t __res1; /* Keep struct identical to >> >> buf, >> >> >> */ >> >> >> uint32_t __res2; /* until pool >> implementation >> >> is >> >> >> fixed*/ >> >> >> -- >> >> >> 2.2.2 >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> lng-odp mailing list >> >> >> lng-odp@lists.linaro.org >> >> >> http://lists.linaro.org/mailman/listinfo/lng-odp >> >> > >> >> > >> >> > >> >> > _______________________________________________ >> >> > lng-odp mailing list >> >> > lng-odp@lists.linaro.org >> >> > http://lists.linaro.org/mailman/listinfo/lng-odp >> >> > >> > > > _______________________________________________ > lng-odp mailing list > lng-odp@lists.linaro.org > http://lists.linaro.org/mailman/listinfo/lng-odp > >
If your HW can select a pool based on frame length then it sounds like you do support this feature. The PMR syntax is just the way ODP applications would access this feature. It's reasonable for implementations to impose restrictions on how "complex" classification rules can be. So, for example, if you have a pmr_len rule then that's it and you can't further refine the classification beyond that. Would that work for you? On Mon, Feb 2, 2015 at 7:03 AM, Alexandru Badicioiu < alexandru.badicioiu@linaro.org> wrote: > While ODP doesn't say that every platform must support the API in an > optimal manner I think the APIs should target to be possible to implement > on a broad range of platforms . "Classification" on packet length, as > defined by the ODP API, is not supported on current FSL platforms. However, > the HW has this feature of selecting the buffer pool depending on the frame > size and this feature was used when odp_pktio_open(dev, pool) was possible > multiple times for a given device, but now this model is dropped. > I think there is a need to align the API here with the currently involved > platforms. > > Thanks, > Alex > > > On 2 February 2015 at 14:13, Bill Fischofer <bill.fischofer@linaro.org> > wrote: > >> The model referred to here involves the use of unsegmented pools combined >> with classification rules based on packet length, as originally suggested >> by Barry as what Tilera does. It's why we added the pkt_len PMR term. If >> an application wishes it can create multiple unsegmented pools of different >> sizes and use classification to sort arriving packets into them based on >> their length. This give you exactly what you want: Application does not >> have to deal with segments and good memory utilization. This model is easy >> to understand, and independent of the underlying platform. Some platforms >> may not be able to provide this level of control, but ODP doesn't say that >> every platform must support every ODP application or API in an optimal >> manner. >> >> For platforms that do segmentation in HW, this is properly the domain of >> the implementation. There's no point in SW asking HW to do something it >> cannot do, so you'd be left with the paradigm that SW queries HW limits and >> then tells HW to do what it's going to do anyway, which isn't terribly >> useful. >> >> The pool parameters tell the implementation two key things: How many >> packets the pool should contain, and how large they are expected to be (on >> average). That's what allows the implementation to calculate how much >> storage needs to be reserved. Other items such as alignment just refine >> this essential information. The number of segments that these packets may >> be broken into is again something that the implementation knows, not the >> application, because its the implementation that knows how packets are >> actually stored on its platform. >> >> >> >> On Mon, Feb 2, 2015 at 4:55 AM, Ola Liljedahl <ola.liljedahl@linaro.org> >> wrote: >> >>> On 2 February 2015 at 10:41, Savolainen, Petri (NSN - FI/Espoo) >>> <petri.savolainen@nsn.com> wrote: >>> > Hi, >>> > >>> > An ODP application is SW running on cores (not HW or firmware). If >>> there's no SW, there's no need for ODP API. >>> Has anyone disputed this? >>> >>> > Application receives odp_packets, which consist of odp_packet_segs. >>> Packets are a primary concept in networking. Segments are not. >>> Segments exists in specific packet processing implementations >>> (hardware+drivers+application). In some implementations, each packet >>> may always be stored in one consecutive buffer (segment?). In order >>> implementations, packets may have to be stored in multiple segments. >>> It may depend on the size of each specific packet as well. Etc. >>> >>> > From application point of view, each segment is a block of contiguous >>> memory (implementation can be anything as long as application can access it >>> contiguously). Application has the knowledge on incoming packets and SW >>> processing of those. Segmentation is a trade-off between linear memory >>> processing (good performance) and memory usage. >>> Now you are inventing a new reason for ODP to support segmentation. >>> Until this point, support for segmentation has been to handle hardware >>> designs which cannot (efficiently) support contiguous packets for all >>> packet sizes. >>> >>> >These parameters give application the change to tune the trade-off in >>> each use case. >>> > >>> > For example, application may >>> > - receive 49% 64 byte packets, 49% 1500 byte packets and 2% 9kB packets >>> > - read up to 300 bytes into a packet >>> > - add upto 54 bytes of tunnel headers in front of the packet >>> > => ask first min seg len 354 bytes, that would fit all accesses. >>> Implementation rounds that up to 384 bytes (full cache line) >>> > => run tests and notice that performance is good, but we need to lower >>> the memory usage of this pool >>> > => drop min seg len to 200 bytes, implementation rounds that to 256 >>> > => run tests, performance is still good enough and pool uses less >>> memory (which can be then used for something else) >>> > >>> > How implementation could do this trade-off analysis behalf of the user? >>> The application can provide the expected packet size distribution to >>> the ODP implementation as hints for optimal memory utilization (and >>> optimal performance). I don't see this as equivalent to specifying >>> which segment sizes the ODP implementation *must* use. I also thought >>> that the class-of-service support in ODP classifier API had the >>> support for selecting the relevant pool based on e.g. ingress packet >>> size. Is this solution not complete? What is missing? >>> >>> The application should not specify segment sizes as requirements. >>> Actual segment sizes used are selected by the ODP implementation >>> (possibly pre-selected by the hardware design itself). >>> >>> If you are pushing for a solution, you need to specify the problem >>> first. There could be other (better) solutions to the problem. The >>> power of collaboration is that many brains can work together on a >>> problem. >>> >>> > >>> > >>> > + uint32_t seg_len; /**< Minimum packet >>> segment buffer >>> > + length in bytes. It >>> includes >>> > + possible >>> head-/tailroom bytes. >>> > + Use 0 for default >>> length. */ >>> > >>> > In addition to this, we have lower and upper config limits >>> (ODP_CONFIG_PACKET_BUF_LEN_XXX need to be updated also). If implementation >>> can support only one segment len, it will be documented by those min/max >>> limits being the same. >>> > >>> > >>> > -Petri >>> > >>> > >>> > >>> >> -----Original Message----- >>> >> From: ext Ola Liljedahl [mailto:ola.liljedahl@linaro.org] >>> >> Sent: Sunday, February 01, 2015 12:06 AM >>> >> To: Bill Fischofer >>> >> Cc: Petri Savolainen; LNG ODP Mailman List >>> >> Subject: Re: [lng-odp] [PATCH 1/2] api: pool: Added packet pool >>> parameters >>> >> >>> >> One important aspect of ODP is a hardware abstraction, the ODP API is >>> >> supposed to hide implementations details such as how buffers are >>> >> managed (segmentation is one implementation detail that we allow to >>> >> leak through as this is common and very expensive to hide from the >>> >> application). I recently heard of a 40G NIC which doesn't use (user >>> >> visible) buffer pools at all. You just pass a large (shared) memory >>> >> region to the NIC and it carves up suitable buffers as needed. >>> >> Needless to say, DPDK has problem with that. But ODP shouldn't. >>> >> >>> >> On 31 January 2015 at 01:22, Bill Fischofer < >>> bill.fischofer@linaro.org> >>> >> wrote: >>> >> > I really can't concur with this proposed design. The fundamental >>> issue >>> >> here >>> >> > is the proper relationship between applications and implementations >>> with >>> >> > respect to packet storage. Packets have an overall length, and >>> this is >>> >> > within the purview of the application, since it understands the >>> type of >>> >> > traffic it is looking to process. Length is a property that packets >>> >> have >>> >> > "on the wire" and is independent of how packets may be stored >>> within a >>> >> > processing node. >>> >> > >>> >> > Segments are always an implementation construct and exist for the >>> >> > convenience of an implementation. Segments do not exist on the wire >>> and >>> >> are >>> >> > not part of any inherent (i.e., platform-independent) packet >>> structure. >>> >> To >>> >> > assert application control over packet segmentation is to say that >>> the >>> >> > application is controlling the implementation of packet storage. >>> This is >>> >> a >>> >> > fundamental departure from the API/Implementation separation >>> paradigm >>> >> that >>> >> > ODP is promoting. If an application wishes to do this it is >>> leaving no >>> >> > room for HW offload or innovation in this area--it's just using the >>> HW >>> >> as a >>> >> > raw block manager and doing everything itself in SW. >>> >> > >>> >> > It is understood that the existence of segmentation imposes some >>> >> processing >>> >> > overhead on SW to the extent that SW must deal with the "seams" in a >>> >> packet >>> >> > addressability that results from segmentation. There are two ways >>> to >>> >> > address this. >>> >> > >>> >> > The first is to recognize that in the data plane the vast bulk of >>> >> processing >>> >> > is on packet headers rather than payload and that HW is aware of >>> this >>> >> fact, >>> >> > which is why HW designed for packet processing invariably uses >>> segment >>> >> sizes >>> >> > large enough to contain all of the packet headers within the first >>> >> packet >>> >> > segment for the vast majority of packets of interest. In the spec we >>> >> worked >>> >> > on last year we stated that ODP would require a minimum segment >>> size of >>> >> 256 >>> >> > bytes so that applications would have assurance regarding this, and >>> no >>> >> > surveyed platforms had issues with that. >>> >> > >>> >> > The second means of addressing this problem is to allow >>> applications to >>> >> > explicitly request unsegmented pools. While recognizing that not >>> all >>> >> > platforms can provide unsegmented pools efficiently, the idea behind >>> >> > unsegmented pools was that this would aid applications that for >>> whatever >>> >> > reason could not deal with packet segments. >>> >> > >>> >> > So in this model the application has two choices. It can either >>> work >>> >> with >>> >> > an implementation-chosen segment size, understanding that that size >>> will >>> >> be >>> >> > large enough so that it need not worry about segment boundaries in >>> >> packet >>> >> > headers for almost all packets of interest, or it can request that >>> the >>> >> pool >>> >> > be unsegmented so that the entire packet is always a single segment. >>> >> > >>> >> > If you believe that this model is insufficient, I would like to >>> >> understand, >>> >> > with use cases, why that is so. I would also like to hear from SoC >>> >> vendors >>> >> > looking to implement ODP whether they can efficiently support >>> arbitrary >>> >> > application-specified segment sizes for packet processing. >>> >> > >>> >> > >>> >> > On Fri, Jan 30, 2015 at 7:10 AM, Petri Savolainen >>> >> > <petri.savolainen@linaro.org> wrote: >>> >> >> >>> >> >> Completed odp_pool_param_t definition with packet pool parameters. >>> >> >> Parameter definition is close to what we are using already. Segment >>> >> >> min length, segment min alignment and number of segments. >>> >> >> >>> >> >> Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> >>> >> >> --- >>> >> >> include/odp/api/pool.h | 20 +++++++++++++++----- >>> >> >> 1 file changed, 15 insertions(+), 5 deletions(-) >>> >> >> >>> >> >> diff --git a/include/odp/api/pool.h b/include/odp/api/pool.h >>> >> >> index 1582102..e407704 100644 >>> >> >> --- a/include/odp/api/pool.h >>> >> >> +++ b/include/odp/api/pool.h >>> >> >> @@ -61,13 +61,23 @@ typedef struct odp_pool_param_t { >>> >> >> of 8. */ >>> >> >> uint32_t num; /**< Number of buffers in >>> the >>> >> pool >>> >> >> */ >>> >> >> } buf; >>> >> >> -/* Reserved for packet and timeout specific params >>> >> >> struct { >>> >> >> - uint32_t seg_size; >>> >> >> - uint32_t seg_align; >>> >> >> - uint32_t num; >>> >> >> + uint32_t seg_len; /**< Minimum packet >>> segment >>> >> >> buffer >>> >> >> + length in bytes. >>> It >>> >> >> includes >>> >> >> + possible head- >>> >> /tailroom >>> >> >> bytes. >>> >> >> + Use 0 for default >>> >> length. >>> >> >> */ >>> >> >> + uint32_t seg_align; /**< Minimum packet >>> segment >>> >> >> buffer >>> >> >> + alignment in >>> bytes. >>> >> Valid >>> >> >> + values are powers >>> of >>> >> two. >>> >> >> Use 0 >>> >> >> + for default >>> alignment. >>> >> >> Default >>> >> >> + will always be a >>> >> multiple >>> >> >> of 8. >>> >> >> + */ >>> >> >> + uint32_t seg_num; /**< Number of packet >>> >> segments >>> >> >> in >>> >> >> + the pool. It's >>> also >>> >> the >>> >> >> maximum >>> >> >> + number of packets, >>> >> since >>> >> >> each >>> >> >> + packet consist of >>> at >>> >> >> least one >>> >> >> + segment. */ >>> >> >> } pkt; >>> >> >> -*/ >>> >> >> struct { >>> >> >> uint32_t __res1; /* Keep struct identical >>> to >>> >> buf, >>> >> >> */ >>> >> >> uint32_t __res2; /* until pool >>> implementation >>> >> is >>> >> >> fixed*/ >>> >> >> -- >>> >> >> 2.2.2 >>> >> >> >>> >> >> >>> >> >> _______________________________________________ >>> >> >> lng-odp mailing list >>> >> >> lng-odp@lists.linaro.org >>> >> >> http://lists.linaro.org/mailman/listinfo/lng-odp >>> >> > >>> >> > >>> >> > >>> >> > _______________________________________________ >>> >> > lng-odp mailing list >>> >> > lng-odp@lists.linaro.org >>> >> > http://lists.linaro.org/mailman/listinfo/lng-odp >>> >> > >>> >> >> >> _______________________________________________ >> lng-odp mailing list >> lng-odp@lists.linaro.org >> http://lists.linaro.org/mailman/listinfo/lng-odp >> >> >
pkt_len PMR API requires a CoS made up of a queue or a group of queues and a buffer pool. My HW does not support selecting the queue based on the packet length (solely), only the buffer pool. The queue is selected based on the packet content (not length) and after the buffer pool was determined. This last step is where platform classification capabilities are involved. Buffer pool selection is more a port (pktio) feature. Alex On 2 February 2015 at 15:06, Bill Fischofer <bill.fischofer@linaro.org> wrote: > If your HW can select a pool based on frame length then it sounds like you > do support this feature. The PMR syntax is just the way ODP applications > would access this feature. > > It's reasonable for implementations to impose restrictions on how > "complex" classification rules can be. So, for example, if you have a > pmr_len rule then that's it and you can't further refine the classification > beyond that. > > Would that work for you? > > On Mon, Feb 2, 2015 at 7:03 AM, Alexandru Badicioiu < > alexandru.badicioiu@linaro.org> wrote: > >> While ODP doesn't say that every platform must support the API in an >> optimal manner I think the APIs should target to be possible to implement >> on a broad range of platforms . "Classification" on packet length, as >> defined by the ODP API, is not supported on current FSL platforms. However, >> the HW has this feature of selecting the buffer pool depending on the frame >> size and this feature was used when odp_pktio_open(dev, pool) was possible >> multiple times for a given device, but now this model is dropped. >> I think there is a need to align the API here with the currently involved >> platforms. >> >> Thanks, >> Alex >> >> >> On 2 February 2015 at 14:13, Bill Fischofer <bill.fischofer@linaro.org> >> wrote: >> >>> The model referred to here involves the use of unsegmented pools >>> combined with classification rules based on packet length, as originally >>> suggested by Barry as what Tilera does. It's why we added the pkt_len PMR >>> term. If an application wishes it can create multiple unsegmented pools of >>> different sizes and use classification to sort arriving packets into them >>> based on their length. This give you exactly what you want: Application >>> does not have to deal with segments and good memory utilization. This >>> model is easy to understand, and independent of the underlying platform. >>> Some platforms may not be able to provide this level of control, but ODP >>> doesn't say that every platform must support every ODP application or API >>> in an optimal manner. >>> >>> For platforms that do segmentation in HW, this is properly the domain of >>> the implementation. There's no point in SW asking HW to do something it >>> cannot do, so you'd be left with the paradigm that SW queries HW limits and >>> then tells HW to do what it's going to do anyway, which isn't terribly >>> useful. >>> >>> The pool parameters tell the implementation two key things: How many >>> packets the pool should contain, and how large they are expected to be (on >>> average). That's what allows the implementation to calculate how much >>> storage needs to be reserved. Other items such as alignment just refine >>> this essential information. The number of segments that these packets may >>> be broken into is again something that the implementation knows, not the >>> application, because its the implementation that knows how packets are >>> actually stored on its platform. >>> >>> >>> >>> On Mon, Feb 2, 2015 at 4:55 AM, Ola Liljedahl <ola.liljedahl@linaro.org> >>> wrote: >>> >>>> On 2 February 2015 at 10:41, Savolainen, Petri (NSN - FI/Espoo) >>>> <petri.savolainen@nsn.com> wrote: >>>> > Hi, >>>> > >>>> > An ODP application is SW running on cores (not HW or firmware). If >>>> there's no SW, there's no need for ODP API. >>>> Has anyone disputed this? >>>> >>>> > Application receives odp_packets, which consist of odp_packet_segs. >>>> Packets are a primary concept in networking. Segments are not. >>>> Segments exists in specific packet processing implementations >>>> (hardware+drivers+application). In some implementations, each packet >>>> may always be stored in one consecutive buffer (segment?). In order >>>> implementations, packets may have to be stored in multiple segments. >>>> It may depend on the size of each specific packet as well. Etc. >>>> >>>> > From application point of view, each segment is a block of contiguous >>>> memory (implementation can be anything as long as application can access it >>>> contiguously). Application has the knowledge on incoming packets and SW >>>> processing of those. Segmentation is a trade-off between linear memory >>>> processing (good performance) and memory usage. >>>> Now you are inventing a new reason for ODP to support segmentation. >>>> Until this point, support for segmentation has been to handle hardware >>>> designs which cannot (efficiently) support contiguous packets for all >>>> packet sizes. >>>> >>>> >These parameters give application the change to tune the trade-off in >>>> each use case. >>>> > >>>> > For example, application may >>>> > - receive 49% 64 byte packets, 49% 1500 byte packets and 2% 9kB >>>> packets >>>> > - read up to 300 bytes into a packet >>>> > - add upto 54 bytes of tunnel headers in front of the packet >>>> > => ask first min seg len 354 bytes, that would fit all accesses. >>>> Implementation rounds that up to 384 bytes (full cache line) >>>> > => run tests and notice that performance is good, but we need to >>>> lower the memory usage of this pool >>>> > => drop min seg len to 200 bytes, implementation rounds that to 256 >>>> > => run tests, performance is still good enough and pool uses less >>>> memory (which can be then used for something else) >>>> > >>>> > How implementation could do this trade-off analysis behalf of the >>>> user? >>>> The application can provide the expected packet size distribution to >>>> the ODP implementation as hints for optimal memory utilization (and >>>> optimal performance). I don't see this as equivalent to specifying >>>> which segment sizes the ODP implementation *must* use. I also thought >>>> that the class-of-service support in ODP classifier API had the >>>> support for selecting the relevant pool based on e.g. ingress packet >>>> size. Is this solution not complete? What is missing? >>>> >>>> The application should not specify segment sizes as requirements. >>>> Actual segment sizes used are selected by the ODP implementation >>>> (possibly pre-selected by the hardware design itself). >>>> >>>> If you are pushing for a solution, you need to specify the problem >>>> first. There could be other (better) solutions to the problem. The >>>> power of collaboration is that many brains can work together on a >>>> problem. >>>> >>>> > >>>> > >>>> > + uint32_t seg_len; /**< Minimum packet >>>> segment buffer >>>> > + length in bytes. It >>>> includes >>>> > + possible >>>> head-/tailroom bytes. >>>> > + Use 0 for default >>>> length. */ >>>> > >>>> > In addition to this, we have lower and upper config limits >>>> (ODP_CONFIG_PACKET_BUF_LEN_XXX need to be updated also). If implementation >>>> can support only one segment len, it will be documented by those min/max >>>> limits being the same. >>>> > >>>> > >>>> > -Petri >>>> > >>>> > >>>> > >>>> >> -----Original Message----- >>>> >> From: ext Ola Liljedahl [mailto:ola.liljedahl@linaro.org] >>>> >> Sent: Sunday, February 01, 2015 12:06 AM >>>> >> To: Bill Fischofer >>>> >> Cc: Petri Savolainen; LNG ODP Mailman List >>>> >> Subject: Re: [lng-odp] [PATCH 1/2] api: pool: Added packet pool >>>> parameters >>>> >> >>>> >> One important aspect of ODP is a hardware abstraction, the ODP API is >>>> >> supposed to hide implementations details such as how buffers are >>>> >> managed (segmentation is one implementation detail that we allow to >>>> >> leak through as this is common and very expensive to hide from the >>>> >> application). I recently heard of a 40G NIC which doesn't use (user >>>> >> visible) buffer pools at all. You just pass a large (shared) memory >>>> >> region to the NIC and it carves up suitable buffers as needed. >>>> >> Needless to say, DPDK has problem with that. But ODP shouldn't. >>>> >> >>>> >> On 31 January 2015 at 01:22, Bill Fischofer < >>>> bill.fischofer@linaro.org> >>>> >> wrote: >>>> >> > I really can't concur with this proposed design. The fundamental >>>> issue >>>> >> here >>>> >> > is the proper relationship between applications and >>>> implementations with >>>> >> > respect to packet storage. Packets have an overall length, and >>>> this is >>>> >> > within the purview of the application, since it understands the >>>> type of >>>> >> > traffic it is looking to process. Length is a property that >>>> packets >>>> >> have >>>> >> > "on the wire" and is independent of how packets may be stored >>>> within a >>>> >> > processing node. >>>> >> > >>>> >> > Segments are always an implementation construct and exist for the >>>> >> > convenience of an implementation. Segments do not exist on the >>>> wire and >>>> >> are >>>> >> > not part of any inherent (i.e., platform-independent) packet >>>> structure. >>>> >> To >>>> >> > assert application control over packet segmentation is to say that >>>> the >>>> >> > application is controlling the implementation of packet storage. >>>> This is >>>> >> a >>>> >> > fundamental departure from the API/Implementation separation >>>> paradigm >>>> >> that >>>> >> > ODP is promoting. If an application wishes to do this it is >>>> leaving no >>>> >> > room for HW offload or innovation in this area--it's just using >>>> the HW >>>> >> as a >>>> >> > raw block manager and doing everything itself in SW. >>>> >> > >>>> >> > It is understood that the existence of segmentation imposes some >>>> >> processing >>>> >> > overhead on SW to the extent that SW must deal with the "seams" in >>>> a >>>> >> packet >>>> >> > addressability that results from segmentation. There are two ways >>>> to >>>> >> > address this. >>>> >> > >>>> >> > The first is to recognize that in the data plane the vast bulk of >>>> >> processing >>>> >> > is on packet headers rather than payload and that HW is aware of >>>> this >>>> >> fact, >>>> >> > which is why HW designed for packet processing invariably uses >>>> segment >>>> >> sizes >>>> >> > large enough to contain all of the packet headers within the first >>>> >> packet >>>> >> > segment for the vast majority of packets of interest. In the spec >>>> we >>>> >> worked >>>> >> > on last year we stated that ODP would require a minimum segment >>>> size of >>>> >> 256 >>>> >> > bytes so that applications would have assurance regarding this, >>>> and no >>>> >> > surveyed platforms had issues with that. >>>> >> > >>>> >> > The second means of addressing this problem is to allow >>>> applications to >>>> >> > explicitly request unsegmented pools. While recognizing that not >>>> all >>>> >> > platforms can provide unsegmented pools efficiently, the idea >>>> behind >>>> >> > unsegmented pools was that this would aid applications that for >>>> whatever >>>> >> > reason could not deal with packet segments. >>>> >> > >>>> >> > So in this model the application has two choices. It can either >>>> work >>>> >> with >>>> >> > an implementation-chosen segment size, understanding that that >>>> size will >>>> >> be >>>> >> > large enough so that it need not worry about segment boundaries in >>>> >> packet >>>> >> > headers for almost all packets of interest, or it can request that >>>> the >>>> >> pool >>>> >> > be unsegmented so that the entire packet is always a single >>>> segment. >>>> >> > >>>> >> > If you believe that this model is insufficient, I would like to >>>> >> understand, >>>> >> > with use cases, why that is so. I would also like to hear from SoC >>>> >> vendors >>>> >> > looking to implement ODP whether they can efficiently support >>>> arbitrary >>>> >> > application-specified segment sizes for packet processing. >>>> >> > >>>> >> > >>>> >> > On Fri, Jan 30, 2015 at 7:10 AM, Petri Savolainen >>>> >> > <petri.savolainen@linaro.org> wrote: >>>> >> >> >>>> >> >> Completed odp_pool_param_t definition with packet pool parameters. >>>> >> >> Parameter definition is close to what we are using already. >>>> Segment >>>> >> >> min length, segment min alignment and number of segments. >>>> >> >> >>>> >> >> Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> >>>> >> >> --- >>>> >> >> include/odp/api/pool.h | 20 +++++++++++++++----- >>>> >> >> 1 file changed, 15 insertions(+), 5 deletions(-) >>>> >> >> >>>> >> >> diff --git a/include/odp/api/pool.h b/include/odp/api/pool.h >>>> >> >> index 1582102..e407704 100644 >>>> >> >> --- a/include/odp/api/pool.h >>>> >> >> +++ b/include/odp/api/pool.h >>>> >> >> @@ -61,13 +61,23 @@ typedef struct odp_pool_param_t { >>>> >> >> of 8. */ >>>> >> >> uint32_t num; /**< Number of buffers in >>>> the >>>> >> pool >>>> >> >> */ >>>> >> >> } buf; >>>> >> >> -/* Reserved for packet and timeout specific params >>>> >> >> struct { >>>> >> >> - uint32_t seg_size; >>>> >> >> - uint32_t seg_align; >>>> >> >> - uint32_t num; >>>> >> >> + uint32_t seg_len; /**< Minimum packet >>>> segment >>>> >> >> buffer >>>> >> >> + length in bytes. >>>> It >>>> >> >> includes >>>> >> >> + possible head- >>>> >> /tailroom >>>> >> >> bytes. >>>> >> >> + Use 0 for default >>>> >> length. >>>> >> >> */ >>>> >> >> + uint32_t seg_align; /**< Minimum packet >>>> segment >>>> >> >> buffer >>>> >> >> + alignment in >>>> bytes. >>>> >> Valid >>>> >> >> + values are >>>> powers of >>>> >> two. >>>> >> >> Use 0 >>>> >> >> + for default >>>> alignment. >>>> >> >> Default >>>> >> >> + will always be a >>>> >> multiple >>>> >> >> of 8. >>>> >> >> + */ >>>> >> >> + uint32_t seg_num; /**< Number of packet >>>> >> segments >>>> >> >> in >>>> >> >> + the pool. It's >>>> also >>>> >> the >>>> >> >> maximum >>>> >> >> + number of >>>> packets, >>>> >> since >>>> >> >> each >>>> >> >> + packet consist >>>> of at >>>> >> >> least one >>>> >> >> + segment. */ >>>> >> >> } pkt; >>>> >> >> -*/ >>>> >> >> struct { >>>> >> >> uint32_t __res1; /* Keep struct identical >>>> to >>>> >> buf, >>>> >> >> */ >>>> >> >> uint32_t __res2; /* until pool >>>> implementation >>>> >> is >>>> >> >> fixed*/ >>>> >> >> -- >>>> >> >> 2.2.2 >>>> >> >> >>>> >> >> >>>> >> >> _______________________________________________ >>>> >> >> lng-odp mailing list >>>> >> >> lng-odp@lists.linaro.org >>>> >> >> http://lists.linaro.org/mailman/listinfo/lng-odp >>>> >> > >>>> >> > >>>> >> > >>>> >> > _______________________________________________ >>>> >> > lng-odp mailing list >>>> >> > lng-odp@lists.linaro.org >>>> >> > http://lists.linaro.org/mailman/listinfo/lng-odp >>>> >> > >>>> >>> >>> >>> _______________________________________________ >>> lng-odp mailing list >>> lng-odp@lists.linaro.org >>> http://lists.linaro.org/mailman/listinfo/lng-odp >>> >>> >> >
Let's discuss this during today's sync call with Petri. On Mon, Feb 2, 2015 at 7:19 AM, Alexandru Badicioiu < alexandru.badicioiu@linaro.org> wrote: > pkt_len PMR API requires a CoS made up of a queue or a group of queues and > a buffer pool. My HW does not support selecting the queue based on the > packet length (solely), only the buffer pool. The queue is selected based > on the packet content (not length) and after the buffer pool was > determined. This last step is where platform classification capabilities > are involved. Buffer pool selection is more a port (pktio) feature. > > Alex > > On 2 February 2015 at 15:06, Bill Fischofer <bill.fischofer@linaro.org> > wrote: > >> If your HW can select a pool based on frame length then it sounds like >> you do support this feature. The PMR syntax is just the way ODP >> applications would access this feature. >> >> It's reasonable for implementations to impose restrictions on how >> "complex" classification rules can be. So, for example, if you have a >> pmr_len rule then that's it and you can't further refine the classification >> beyond that. >> >> Would that work for you? >> >> On Mon, Feb 2, 2015 at 7:03 AM, Alexandru Badicioiu < >> alexandru.badicioiu@linaro.org> wrote: >> >>> While ODP doesn't say that every platform must support the API in an >>> optimal manner I think the APIs should target to be possible to implement >>> on a broad range of platforms . "Classification" on packet length, as >>> defined by the ODP API, is not supported on current FSL platforms. However, >>> the HW has this feature of selecting the buffer pool depending on the frame >>> size and this feature was used when odp_pktio_open(dev, pool) was possible >>> multiple times for a given device, but now this model is dropped. >>> I think there is a need to align the API here with the currently >>> involved platforms. >>> >>> Thanks, >>> Alex >>> >>> >>> On 2 February 2015 at 14:13, Bill Fischofer <bill.fischofer@linaro.org> >>> wrote: >>> >>>> The model referred to here involves the use of unsegmented pools >>>> combined with classification rules based on packet length, as originally >>>> suggested by Barry as what Tilera does. It's why we added the pkt_len PMR >>>> term. If an application wishes it can create multiple unsegmented pools of >>>> different sizes and use classification to sort arriving packets into them >>>> based on their length. This give you exactly what you want: Application >>>> does not have to deal with segments and good memory utilization. This >>>> model is easy to understand, and independent of the underlying platform. >>>> Some platforms may not be able to provide this level of control, but ODP >>>> doesn't say that every platform must support every ODP application or API >>>> in an optimal manner. >>>> >>>> For platforms that do segmentation in HW, this is properly the domain >>>> of the implementation. There's no point in SW asking HW to do something it >>>> cannot do, so you'd be left with the paradigm that SW queries HW limits and >>>> then tells HW to do what it's going to do anyway, which isn't terribly >>>> useful. >>>> >>>> The pool parameters tell the implementation two key things: How many >>>> packets the pool should contain, and how large they are expected to be (on >>>> average). That's what allows the implementation to calculate how much >>>> storage needs to be reserved. Other items such as alignment just refine >>>> this essential information. The number of segments that these packets may >>>> be broken into is again something that the implementation knows, not the >>>> application, because its the implementation that knows how packets are >>>> actually stored on its platform. >>>> >>>> >>>> >>>> On Mon, Feb 2, 2015 at 4:55 AM, Ola Liljedahl <ola.liljedahl@linaro.org >>>> > wrote: >>>> >>>>> On 2 February 2015 at 10:41, Savolainen, Petri (NSN - FI/Espoo) >>>>> <petri.savolainen@nsn.com> wrote: >>>>> > Hi, >>>>> > >>>>> > An ODP application is SW running on cores (not HW or firmware). If >>>>> there's no SW, there's no need for ODP API. >>>>> Has anyone disputed this? >>>>> >>>>> > Application receives odp_packets, which consist of odp_packet_segs. >>>>> Packets are a primary concept in networking. Segments are not. >>>>> Segments exists in specific packet processing implementations >>>>> (hardware+drivers+application). In some implementations, each packet >>>>> may always be stored in one consecutive buffer (segment?). In order >>>>> implementations, packets may have to be stored in multiple segments. >>>>> It may depend on the size of each specific packet as well. Etc. >>>>> >>>>> > From application point of view, each segment is a block of >>>>> contiguous memory (implementation can be anything as long as application >>>>> can access it contiguously). Application has the knowledge on incoming >>>>> packets and SW processing of those. Segmentation is a trade-off between >>>>> linear memory processing (good performance) and memory usage. >>>>> Now you are inventing a new reason for ODP to support segmentation. >>>>> Until this point, support for segmentation has been to handle hardware >>>>> designs which cannot (efficiently) support contiguous packets for all >>>>> packet sizes. >>>>> >>>>> >These parameters give application the change to tune the trade-off in >>>>> each use case. >>>>> > >>>>> > For example, application may >>>>> > - receive 49% 64 byte packets, 49% 1500 byte packets and 2% 9kB >>>>> packets >>>>> > - read up to 300 bytes into a packet >>>>> > - add upto 54 bytes of tunnel headers in front of the packet >>>>> > => ask first min seg len 354 bytes, that would fit all accesses. >>>>> Implementation rounds that up to 384 bytes (full cache line) >>>>> > => run tests and notice that performance is good, but we need to >>>>> lower the memory usage of this pool >>>>> > => drop min seg len to 200 bytes, implementation rounds that to 256 >>>>> > => run tests, performance is still good enough and pool uses less >>>>> memory (which can be then used for something else) >>>>> > >>>>> > How implementation could do this trade-off analysis behalf of the >>>>> user? >>>>> The application can provide the expected packet size distribution to >>>>> the ODP implementation as hints for optimal memory utilization (and >>>>> optimal performance). I don't see this as equivalent to specifying >>>>> which segment sizes the ODP implementation *must* use. I also thought >>>>> that the class-of-service support in ODP classifier API had the >>>>> support for selecting the relevant pool based on e.g. ingress packet >>>>> size. Is this solution not complete? What is missing? >>>>> >>>>> The application should not specify segment sizes as requirements. >>>>> Actual segment sizes used are selected by the ODP implementation >>>>> (possibly pre-selected by the hardware design itself). >>>>> >>>>> If you are pushing for a solution, you need to specify the problem >>>>> first. There could be other (better) solutions to the problem. The >>>>> power of collaboration is that many brains can work together on a >>>>> problem. >>>>> >>>>> > >>>>> > >>>>> > + uint32_t seg_len; /**< Minimum packet >>>>> segment buffer >>>>> > + length in bytes. It >>>>> includes >>>>> > + possible >>>>> head-/tailroom bytes. >>>>> > + Use 0 for default >>>>> length. */ >>>>> > >>>>> > In addition to this, we have lower and upper config limits >>>>> (ODP_CONFIG_PACKET_BUF_LEN_XXX need to be updated also). If implementation >>>>> can support only one segment len, it will be documented by those min/max >>>>> limits being the same. >>>>> > >>>>> > >>>>> > -Petri >>>>> > >>>>> > >>>>> > >>>>> >> -----Original Message----- >>>>> >> From: ext Ola Liljedahl [mailto:ola.liljedahl@linaro.org] >>>>> >> Sent: Sunday, February 01, 2015 12:06 AM >>>>> >> To: Bill Fischofer >>>>> >> Cc: Petri Savolainen; LNG ODP Mailman List >>>>> >> Subject: Re: [lng-odp] [PATCH 1/2] api: pool: Added packet pool >>>>> parameters >>>>> >> >>>>> >> One important aspect of ODP is a hardware abstraction, the ODP API >>>>> is >>>>> >> supposed to hide implementations details such as how buffers are >>>>> >> managed (segmentation is one implementation detail that we allow to >>>>> >> leak through as this is common and very expensive to hide from the >>>>> >> application). I recently heard of a 40G NIC which doesn't use (user >>>>> >> visible) buffer pools at all. You just pass a large (shared) memory >>>>> >> region to the NIC and it carves up suitable buffers as needed. >>>>> >> Needless to say, DPDK has problem with that. But ODP shouldn't. >>>>> >> >>>>> >> On 31 January 2015 at 01:22, Bill Fischofer < >>>>> bill.fischofer@linaro.org> >>>>> >> wrote: >>>>> >> > I really can't concur with this proposed design. The fundamental >>>>> issue >>>>> >> here >>>>> >> > is the proper relationship between applications and >>>>> implementations with >>>>> >> > respect to packet storage. Packets have an overall length, and >>>>> this is >>>>> >> > within the purview of the application, since it understands the >>>>> type of >>>>> >> > traffic it is looking to process. Length is a property that >>>>> packets >>>>> >> have >>>>> >> > "on the wire" and is independent of how packets may be stored >>>>> within a >>>>> >> > processing node. >>>>> >> > >>>>> >> > Segments are always an implementation construct and exist for the >>>>> >> > convenience of an implementation. Segments do not exist on the >>>>> wire and >>>>> >> are >>>>> >> > not part of any inherent (i.e., platform-independent) packet >>>>> structure. >>>>> >> To >>>>> >> > assert application control over packet segmentation is to say >>>>> that the >>>>> >> > application is controlling the implementation of packet storage. >>>>> This is >>>>> >> a >>>>> >> > fundamental departure from the API/Implementation separation >>>>> paradigm >>>>> >> that >>>>> >> > ODP is promoting. If an application wishes to do this it is >>>>> leaving no >>>>> >> > room for HW offload or innovation in this area--it's just using >>>>> the HW >>>>> >> as a >>>>> >> > raw block manager and doing everything itself in SW. >>>>> >> > >>>>> >> > It is understood that the existence of segmentation imposes some >>>>> >> processing >>>>> >> > overhead on SW to the extent that SW must deal with the "seams" >>>>> in a >>>>> >> packet >>>>> >> > addressability that results from segmentation. There are two >>>>> ways to >>>>> >> > address this. >>>>> >> > >>>>> >> > The first is to recognize that in the data plane the vast bulk of >>>>> >> processing >>>>> >> > is on packet headers rather than payload and that HW is aware of >>>>> this >>>>> >> fact, >>>>> >> > which is why HW designed for packet processing invariably uses >>>>> segment >>>>> >> sizes >>>>> >> > large enough to contain all of the packet headers within the first >>>>> >> packet >>>>> >> > segment for the vast majority of packets of interest. In the spec >>>>> we >>>>> >> worked >>>>> >> > on last year we stated that ODP would require a minimum segment >>>>> size of >>>>> >> 256 >>>>> >> > bytes so that applications would have assurance regarding this, >>>>> and no >>>>> >> > surveyed platforms had issues with that. >>>>> >> > >>>>> >> > The second means of addressing this problem is to allow >>>>> applications to >>>>> >> > explicitly request unsegmented pools. While recognizing that not >>>>> all >>>>> >> > platforms can provide unsegmented pools efficiently, the idea >>>>> behind >>>>> >> > unsegmented pools was that this would aid applications that for >>>>> whatever >>>>> >> > reason could not deal with packet segments. >>>>> >> > >>>>> >> > So in this model the application has two choices. It can either >>>>> work >>>>> >> with >>>>> >> > an implementation-chosen segment size, understanding that that >>>>> size will >>>>> >> be >>>>> >> > large enough so that it need not worry about segment boundaries in >>>>> >> packet >>>>> >> > headers for almost all packets of interest, or it can request >>>>> that the >>>>> >> pool >>>>> >> > be unsegmented so that the entire packet is always a single >>>>> segment. >>>>> >> > >>>>> >> > If you believe that this model is insufficient, I would like to >>>>> >> understand, >>>>> >> > with use cases, why that is so. I would also like to hear from >>>>> SoC >>>>> >> vendors >>>>> >> > looking to implement ODP whether they can efficiently support >>>>> arbitrary >>>>> >> > application-specified segment sizes for packet processing. >>>>> >> > >>>>> >> > >>>>> >> > On Fri, Jan 30, 2015 at 7:10 AM, Petri Savolainen >>>>> >> > <petri.savolainen@linaro.org> wrote: >>>>> >> >> >>>>> >> >> Completed odp_pool_param_t definition with packet pool >>>>> parameters. >>>>> >> >> Parameter definition is close to what we are using already. >>>>> Segment >>>>> >> >> min length, segment min alignment and number of segments. >>>>> >> >> >>>>> >> >> Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> >>>>> >> >> --- >>>>> >> >> include/odp/api/pool.h | 20 +++++++++++++++----- >>>>> >> >> 1 file changed, 15 insertions(+), 5 deletions(-) >>>>> >> >> >>>>> >> >> diff --git a/include/odp/api/pool.h b/include/odp/api/pool.h >>>>> >> >> index 1582102..e407704 100644 >>>>> >> >> --- a/include/odp/api/pool.h >>>>> >> >> +++ b/include/odp/api/pool.h >>>>> >> >> @@ -61,13 +61,23 @@ typedef struct odp_pool_param_t { >>>>> >> >> of 8. */ >>>>> >> >> uint32_t num; /**< Number of buffers >>>>> in the >>>>> >> pool >>>>> >> >> */ >>>>> >> >> } buf; >>>>> >> >> -/* Reserved for packet and timeout specific params >>>>> >> >> struct { >>>>> >> >> - uint32_t seg_size; >>>>> >> >> - uint32_t seg_align; >>>>> >> >> - uint32_t num; >>>>> >> >> + uint32_t seg_len; /**< Minimum packet >>>>> segment >>>>> >> >> buffer >>>>> >> >> + length in >>>>> bytes. It >>>>> >> >> includes >>>>> >> >> + possible head- >>>>> >> /tailroom >>>>> >> >> bytes. >>>>> >> >> + Use 0 for >>>>> default >>>>> >> length. >>>>> >> >> */ >>>>> >> >> + uint32_t seg_align; /**< Minimum packet >>>>> segment >>>>> >> >> buffer >>>>> >> >> + alignment in >>>>> bytes. >>>>> >> Valid >>>>> >> >> + values are >>>>> powers of >>>>> >> two. >>>>> >> >> Use 0 >>>>> >> >> + for default >>>>> alignment. >>>>> >> >> Default >>>>> >> >> + will always be a >>>>> >> multiple >>>>> >> >> of 8. >>>>> >> >> + */ >>>>> >> >> + uint32_t seg_num; /**< Number of packet >>>>> >> segments >>>>> >> >> in >>>>> >> >> + the pool. It's >>>>> also >>>>> >> the >>>>> >> >> maximum >>>>> >> >> + number of >>>>> packets, >>>>> >> since >>>>> >> >> each >>>>> >> >> + packet consist >>>>> of at >>>>> >> >> least one >>>>> >> >> + segment. */ >>>>> >> >> } pkt; >>>>> >> >> -*/ >>>>> >> >> struct { >>>>> >> >> uint32_t __res1; /* Keep struct >>>>> identical to >>>>> >> buf, >>>>> >> >> */ >>>>> >> >> uint32_t __res2; /* until pool >>>>> implementation >>>>> >> is >>>>> >> >> fixed*/ >>>>> >> >> -- >>>>> >> >> 2.2.2 >>>>> >> >> >>>>> >> >> >>>>> >> >> _______________________________________________ >>>>> >> >> lng-odp mailing list >>>>> >> >> lng-odp@lists.linaro.org >>>>> >> >> http://lists.linaro.org/mailman/listinfo/lng-odp >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > _______________________________________________ >>>>> >> > lng-odp mailing list >>>>> >> > lng-odp@lists.linaro.org >>>>> >> > http://lists.linaro.org/mailman/listinfo/lng-odp >>>>> >> > >>>>> >>>> >>>> >>>> _______________________________________________ >>>> lng-odp mailing list >>>> lng-odp@lists.linaro.org >>>> http://lists.linaro.org/mailman/listinfo/lng-odp >>>> >>>> >>> >> >
diff --git a/include/odp/api/pool.h b/include/odp/api/pool.h index 1582102..e407704 100644 --- a/include/odp/api/pool.h +++ b/include/odp/api/pool.h @@ -61,13 +61,23 @@ typedef struct odp_pool_param_t { of 8. */ uint32_t num; /**< Number of buffers in the pool */ } buf; -/* Reserved for packet and timeout specific params struct { - uint32_t seg_size; - uint32_t seg_align; - uint32_t num; + uint32_t seg_len; /**< Minimum packet segment buffer + length in bytes. It includes + possible head-/tailroom bytes. + Use 0 for default length. */ + uint32_t seg_align; /**< Minimum packet segment buffer + alignment in bytes. Valid + values are powers of two. Use 0 + for default alignment. Default + will always be a multiple of 8. + */ + uint32_t seg_num; /**< Number of packet segments in + the pool. It's also the maximum + number of packets, since each + packet consist of at least one + segment. */ } pkt; -*/ struct { uint32_t __res1; /* Keep struct identical to buf, */ uint32_t __res2; /* until pool implementation is fixed*/
Completed odp_pool_param_t definition with packet pool parameters. Parameter definition is close to what we are using already. Segment min length, segment min alignment and number of segments. Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> --- include/odp/api/pool.h | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)