[v2] Add testing.dox

Signed-off-by: Mike Holmes <mike.holmes@linaro.org>
---
 testing.dox | 313 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 313 insertions(+)
 create mode 100644 testing.dox

Hello Mike,

Please find some review comments bellow.

BR,
Maxim.

On 11/14/2014 12:37 AM, Mike Holmes wrote:
> Signed-off-by: Mike Holmes <mike.holmes@linaro.org>
> ---
>   testing.dox | 313 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 313 insertions(+)
>   create mode 100644 testing.dox
>
> diff --git a/testing.dox b/testing.dox
> new file mode 100644
> index 0000000..627fb38
> --- /dev/null
> +++ b/testing.dox
> @@ -0,0 +1,313 @@
> +/* Copyright (c) 2014, Linaro Limited
> + * All rights reserved
> + *
> + * SPDX-License-Identifier BSD-3-Clause
> + */
> +
> +/**
> +
> +@page testing Testing, Verification & Validation
> +
> +@tableofcontents
> +
> +@sectiooverview Overview
> +The goals for ODP API testing are:
> + - Ensure that all platform implementations operate identically from the perspective of the application, except for optional APIs or optional features of an API

I think we need to remove "except for optional APIs or optional features 
of an API".  Optional API have to operate identically also. And most 
probably
we will have tests for that.

> + - Ensure that a given API performs to the specification defined in the published ODP Architecture & API document in source code control.

After implementation it may also vary. The main document for API is 
doxygen generated document for odp.git. Tests are checking 
functionality, they do not compare Arch and API documents.

> + - Catch differences in the API definition between the version of the test suite and the implementation
> + - GCOV or similar tool be used to ensure complete coverage of the API, note that the implementation may have less coverage than 100% but the API must be covered 100%.
> +Both branch and line coverage will be reported.
> + - New APIs are considered delivered when there is a test suite to compliment the API definition.
> + - Framework should be very light and configurable.
> + - Provide a mechanism to publish compliance to a released API version
> +
> +There are three elements to the overall test strategy.
> +
> +- @ref compliance
> +The simplest is the ODP Compliance test suite, this is updated with every commit that adds or modifies an ODP API.
> +It is intended to be very lightweight and is implemented using CUnit for the APIs, with simple scripting for the documentation.
> +
> +- @ref functional
> +Functional tests that attempt to perform some more in depth processing which has its output verified, for example encrypt and then decrypt a packet, checking that the result matches the input (similar to LTP).
> +These may require external equipment.
> +
> +- @ref benchmarking
> +Provides no pass fail indication but instead returns a metric on the performance of some portion of the implementation of the API
> +
> +@section logical_groups Test suite API groups
> +
> +The following logical areas for the API need to be addressed for the 1.0 release cycle.
> +The API logical groups will be used in the test directory structure and in defining the unit test suites
> +to gather the tests in to groups of related functionality.
> +
> +API logical group | Headers
> + :--------------- | :---
> +Buffer         | odp_buffer.h odp_buffer_pool.h
> +Packet         | odp_packet_flags.h odp_packet.h
> +Classificatio  | odp_classification.h
Classification

> +Crypt          | odp_crypto.h
> +IP             | not defined

what is IP?

> +Pkti           | not defined

Packet I/O - odp_packet_io.h

> +Queue          | odp_queue.h
> +Schedule       | odp_schedule.h odp_coremask.h
> +Shared Memor   | odp_shared_memory.h

y

> +Synchronizers / Barriers | odp_rwlock.h odp_sync.h odp_atomic.h odp_barrier.h odp_spinlock.h odp_ticketlock.h
> +Thread         | odp_thread.h
> +Timer          | odp_timer.h
> +Syste          | odp_time.h odp_perf.h

System
> +Logging / Abor | odp_version.h odp_debug.h odp_system_info.h
> +Initializatio  | odp_init.h
n

> +
> +@section directory Directory structure
> +
> +Test contains all the tests that must be passed for a given revision of the API to be declared compliant.
> +The tests cover:
> + - the unit tests
> + - documentation tests

what is documentation test case?

> + - functional & benchmark tests
> +
> +@verbatim
> +test
> +|
> +|-- functional
> +|  |-- <test executable>
> +|  |-- <API group>/<test executable>
> +|
> +|-- documentation
> +|  |--<test case>
> +|
> +|-- cunit
> +   |-- <test executable>
> +   |-- <API group>/<test executable>
> +
> +@endverbatim
> +
> +@section compliance ODP-Compliance test suite
> +Because this suite is extended with every API extension it needs to be very light weight.
> +This suite would be run by any new platform attempting to be API compliant, it will try to pass in the obvious combinations of arguments that arise from reading the API definition.
> +For example a unit test is of the following form.
> +
> +@code
> +/* a is in the range 0 to 10
> +returns a *2 or -1 if a is out of range */



> +int foo (unsigned int a)
> +@endcode
> +
> +Tests for foo arguments would be
> + - a=-check that  return = -1
> + - a=check that  return = 0
> + - a=check that  return = 10
> + - a=1check that  return = 20
> + - a=11 check that return = -1
> +
> +The frame work to support generating these test cases is CUnit which provides batch and interactive operation.
> +Unit test cases do not use any external equipment.
> +
> +@section cunit CUnit
> +CUnit http://cunit.sourceforge.net/doc/index.html is a system for writing, administering, and running unit tests in C.
> +It is built as a static library which is linked with the user's testing code.
> +CUnit uses a simple framework for building test structures, and provides a rich set of assertions for testing common data types.
> +In addition, several different interfaces are provided for running tests and reporting results.
> +These include automated interfaces for code-controlled testing and reporting, as well as interactive interfaces allowing the user to run tests and view results dynamically.
> +
> +@subsection structure Structure
> +CUnit is a combination of a platform-independent framework with various user interfaces.
> +The core framework provides basic support for managing a test registry, suites, and test cases.
> +The user interfaces facilitate interaction with the framework to run tests and view results.
> +CUnit is organized like a conventional unit testing framework:
> +
> +@verbatim
> +                  Test Registry
> +                         |
> +          ------------------------------
> +          |                            |
> +       Suite '1'      . . . .       Suite 'N'
> +          |                            |
> +    ---------------             ---------------
> +    |             |             |             |
> + Test '11' ... Test '1M'     Test 'N1' ... Test 'NM'
> +@endverbatim
> +
> +Individual test cases are packaged into suites, which are registered with the active test registry.
> +Suites can have setup and tear down functions which are automatically called before and after running the suite's tests.
> +All suites/tests in the registry may be run using a single function call, or selected suites or tests can be run.
> +
> +@subsection general_usage General Usage
> + - A typical sequence of steps for using the CUnit framework is:
> + - Write functions for tests (and suite init/cleanup if necessary).
> + - Initialize the test registry - CU_initialize_registry()
> + - Add suites to the test registry - CU_add_suite()
> + - Add tests to the suites - CU_add_test()
> + - Run tests using an appropriate interface, e.g. CU_console_run_tests
> + - Cleanup the test registry - CU_cleanup_registry
> +
> +@code
> +/* Copyright (c) 2014, Linaro Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier BSD-3-Clause
> + */
> +
> +#include "odp.h"
> +#include "CUnit/Basic.h"
> +
> +#define DEFAULT_MSG_POOL_SIZE (4*1024*1024)
> +#define DEFAULT_MSG_SIZE (8)
> +
> +static void test_odp_init_global(void)
> +{
> +
> +     int status;
> +     status = odp_init_global();
> +     CU_ASSERT (status == 0) ;
> +}
> +
> +static int init(void)
> +{
> +     printf("ODP version:%s\n",odp_version_api_str());
> +     return 0;
> +}
> +
> +
> +int main(void)
> +{
> +     CU_pSuite pSuite = NULL;

No need to set it to NULL here. Add empty line after variable definition.

> +     /* initialize the CUnit test registry */
> +     if (CUE_SUCCESS != CU_initialize_registry())
> +             return CU_get_error();
> +     /* add a suite to the registry */
> +     pSuite = CU_add_suite("odp_init", init, NULL);
> +     if (NULL == pSuite) {
if (!pSuite) ?

> +             CU_cleanup_registry();
> +             return CU_get_error();
> +     }
> +     /* add the tests to the suite */
> +     if (NULL == CU_ADD_TEST(pSuite, test_odp_init_global)) {

It's better for follow our odp style. Like:


ret =  CU_ADD_TEST(pSuite, test_odp_init_global);
if (!ret) {

}

> +             CU_cleanup_registry();
> +             return CU_get_error();
> +     }
> +     /* Run all tests using the CUnit Basic interface */
> +     CU_basic_set_mode(CU_BRM_VERBOSE);
> +     CU_basic_run_tests();
> +     CU_cleanup_registry();
> +     return CU_get_error();
> +}
> +@endcode
> +
> +@subsection results Normal results
> +
> +@code
> +  CUnit - A unit testing framework for C - Version 2.1-2
> +  http://cunit.sourceforge.net/
> +
> +ODP version:0.0.1
> +
> +Suite: odp_init
> +  Test: test_odp_init_global ...
> +Buffer pool init global
> +  pool_entry_s siz   192
> +  pool_entry_t siz   192
> +  odp_buffer_hdr_t size 120
> +
> +Queue init ... done
> +Queue init global
> +  struct queue_entry_s size 192
> +  queue_entry_t siz      192
> +
> +Schedule init ... done
> +Timer init ...done
> +passed
> +
> +Run Summary  Type  Total    Ran Passed Failed Inactive
> +           suites      1      1    n/a      0        0
> +            tests      1      1      1      0        0
> +          asserts      1      1      1      0      n/a
> +@endcode
> +
> +
> +@subsection compliance_xml Compliance results (XML)
> +The CUnit results may also be gathered in XML format, this format may be use to submit an implementations CUnit test case completion for inclusion on the ODP website as compliant to a given release.
> +
> +~~~{.xml}
> +<?xml version="1.0" ?>.
> +<?xml-stylesheet type="text/xsl" href="CUnit-Run.xsl" ?>.
> +<!DOCTYPE CUNIT_TEST_RUN_REPORT SYSTEM "CUnit-Run.dtd">.
> +<CUNIT_TEST_RUN_REPORT>.
> +  <CUNIT_HEADER/>.
> +  <CUNIT_RESULT_LISTING>.
> + <CUNIT_RUN_SUITE>.
> +   <CUNIT_RUN_SUITE_SUCCESS>.
> +     <SUITE_NAME> odp initialization </SUITE_NAME>.
> +     <CUNIT_RUN_TEST_RECORD>.
> +       <CUNIT_RUN_TEST_SUCCESS>.
> +         <TEST_NAME> test_odp_init_global </TEST_NAME>.
> +       </CUNIT_RUN_TEST_SUCCESS>.
> +     </CUNIT_RUN_TEST_RECORD>.
> +   </CUNIT_RUN_SUITE_SUCCESS>.
> + </CUNIT_RUN_SUITE>.
> +  </CUNIT_RESULT_LISTING>
> +  <CUNIT_RUN_SUMMARY>.
> + <CUNIT_RUN_SUMMARY_RECORD>.
> +   <TYPE> Suites </TYPE>.
> +   <TOTAL> 1 </TOTAL>.
> +   <RUN> 1 </RUN>.
> +   <SUCCEEDED> - NA - </SUCCEEDED>.
> +   <FAILED> 0 </FAILED>.
> +   <INACTIVE> 0 </INACTIVE>.
> + </CUNIT_RUN_SUMMARY_RECORD>.
> + <CUNIT_RUN_SUMMARY_RECORD>.
> +   <TYPE> Test Cases </TYPE>.
> +   <TOTAL> 1 </TOTAL>.
> +   <RUN> 1 </RUN>.
> +   <SUCCEEDED> 1 </SUCCEEDED>.
> +   <FAILED> 0 </FAILED>.
> +   <INACTIVE> 0 </INACTIVE>.
> + </CUNIT_RUN_SUMMARY_RECORD>.
> + <CUNIT_RUN_SUMMARY_RECORD>.
> +   <TYPE> Assertions </TYPE>.
> +   <TOTAL> 1 </TOTAL>.
> +   <RUN> 1 </RUN>.
> +   <SUCCEEDED> 1 </SUCCEEDED>.
> +   <FAILED> 0 </FAILED>.
> +   <INACTIVE> n/a </INACTIVE>.
> + </CUNIT_RUN_SUMMARY_RECORD>.
> +  </CUNIT_RUN_SUMMARY>.
> +  <CUNIT_FOOTER> File Generated By CUnit v2.1-2 - Thu Oct 23 15:02:46 2014
> + </CUNIT_FOOTER>.
> +</CUNIT_TEST_RUN_REPORT>
> +~~~
> +
> +@section functional Functional tests
> +
> +These tests are generally more elaborate with a goal of proving a complex functionality.
> +@verbatim
> +test
> +.
> +|-- functional
> +  |-- <test executable>
> +  |-- <API group>/<test executable>
> +@endverbatim
> + - There is no framework for these tests and they generally take command line arguments to modify their behaviour.
> + - These tests do provide a pass fail indication.
> + - These tests may require network interfaces and external equipment.
> +
> +@section benchmarking Benchmarks
> +
> +Benchmarks are currently also stored with the functional tests, this may be re assessed depending on the test volume.
> +@verbatim
> +test
> +.
> +|-- functional
> +  |-- <test executable>
> +  |-- <API group>/<test executable>
> +@endverbatim
> + - These tests are generally more elaborate with a goal of quantifying the capacity of a complex functionality.
> + - There is no framework for these benchmarks and they generally take command line arguments to modify their behaviour.
> + - They do not provide a pass fail indication.
> +
> +@section test_versioning Test versioning relationship to ODP and versioning
> +
> +For ODP to declare for example the 1.0 tag in its git repository it must have passed the corresponding test suit.
v1.0
> +When ODP API v3.0 is released it will be validated against the test suite version tagged ODP v3.0
> +It is possible that a version of the ODP API is released and that the test suite used has a flaw, in this case the test suite will be re released as v3.0-1
> +*/

On 16 November 2014 14:26, Maxim Uvarov <maxim.uvarov@linaro.org> wrote:

> Hello Mike,
>
> Please find some review comments bellow.
>
> BR,
> Maxim.
>
>
> On 11/14/2014 12:37 AM, Mike Holmes wrote:
>
>> Signed-off-by: Mike Holmes <mike.holmes@linaro.org>
>> ---
>>   testing.dox | 313 ++++++++++++++++++++++++++++++
>> ++++++++++++++++++++++++++++++
>>   1 file changed, 313 insertions(+)
>>   create mode 100644 testing.dox
>>
>> diff --git a/testing.dox b/testing.dox
>> new file mode 100644
>> index 0000000..627fb38
>> --- /dev/null
>> +++ b/testing.dox
>> @@ -0,0 +1,313 @@
>> +/* Copyright (c) 2014, Linaro Limited
>> + * All rights reserved
>> + *
>> + * SPDX-License-Identifier BSD-3-Clause
>> + */
>> +
>> +/**
>> +
>> +@page testing Testing, Verification & Validation
>> +
>> +@tableofcontents
>> +
>> +@sectiooverview Overview
>> +The goals for ODP API testing are:
>> + - Ensure that all platform implementations operate identically from the
>> perspective of the application, except for optional APIs or optional
>> features of an API
>>
>
> I think we need to remove "except for optional APIs or optional features
> of an API".  Optional API have to operate identically also. And most
> probably
> we will have tests for that.


Optional API may not function at all but you are correct they must work
identically, in this case by retuning the specified error and calling
ODP_UNIMPLIMENTED.  I will address this


>
>
>  + - Ensure that a given API performs to the specification defined in the
>> published ODP Architecture & API document in source code control.
>>
>
> After implementation it may also vary. The main document for API is
> doxygen generated document for odp.git. Tests are checking functionality,
> they do not compare Arch and API documents.


The arch doc is generated by linking it to the API doc, and the API doc is
the header where is must define the  behavior fully - I can try to
elaborate but I think the text is correct.

The definitive guide is the Arch doc because it contains all the
information, although the API doc as sub section should be all that is
needed.
As Bill has found we have not closed the loop on ensuring that the API doc
translated the implementation fully from the ARCH doc in some cases.
The tests by definition must check that the implementation does what the
API doc said that API would do. To Balas point earlier the problem is that
our API doc is not testable yet, there is no specification of what the API
will do in many cases.

So I 100% agree we need to clean the path from the ARCH->API DOC ->
Implementation ->Test so that test closes the loop on what the ARCH/API
docs said.



>
>  + - Catch differences in the API definition between the version of the
>> test suite and the implementation
>> + - GCOV or similar tool be used to ensure complete coverage of the API,
>> note that the implementation may have less coverage than 100% but the API
>> must be covered 100%.
>> +Both branch and line coverage will be reported.
>> + - New APIs are considered delivered when there is a test suite to
>> compliment the API definition.
>> + - Framework should be very light and configurable.
>> + - Provide a mechanism to publish compliance to a released API version
>> +
>> +There are three elements to the overall test strategy.
>> +
>> +- @ref compliance
>> +The simplest is the ODP Compliance test suite, this is updated with
>> every commit that adds or modifies an ODP API.
>> +It is intended to be very lightweight and is implemented using CUnit for
>> the APIs, with simple scripting for the documentation.
>> +
>> +- @ref functional
>> +Functional tests that attempt to perform some more in depth processing
>> which has its output verified, for example encrypt and then decrypt a
>> packet, checking that the result matches the input (similar to LTP).
>> +These may require external equipment.
>> +
>> +- @ref benchmarking
>> +Provides no pass fail indication but instead returns a metric on the
>> performance of some portion of the implementation of the API
>> +
>> +@section logical_groups Test suite API groups
>> +
>> +The following logical areas for the API need to be addressed for the 1.0
>> release cycle.
>> +The API logical groups will be used in the test directory structure and
>> in defining the unit test suites
>> +to gather the tests in to groups of related functionality.
>> +
>> +API logical group | Headers
>> + :--------------- | :---
>> +Buffer         | odp_buffer.h odp_buffer_pool.h
>> +Packet         | odp_packet_flags.h odp_packet.h
>> +Classificatio  | odp_classification.h
>>
> Classification


Will fix


>
>
>  +Crypt          | odp_crypto.h
>> +IP             | not defined
>>
>
> what is IP?
>


Will fix


>
>  +Pkti           | not defined
>>
>
> Packet I/O - odp_packet_io.h



Will fix


>
>
>  +Queue          | odp_queue.h
>> +Schedule       | odp_schedule.h odp_coremask.h
>> +Shared Memor   | odp_shared_memory.h
>>
>
> y
>


Will fix



>
>  +Synchronizers / Barriers | odp_rwlock.h odp_sync.h odp_atomic.h
>> odp_barrier.h odp_spinlock.h odp_ticketlock.h
>> +Thread         | odp_thread.h
>> +Timer          | odp_timer.h
>> +Syste          | odp_time.h odp_perf.h
>>
>
> System
>
>> +Logging / Abor | odp_version.h odp_debug.h odp_system_info.h
>> +Initializatio  | odp_init.h
>>
> n
>
>
Will fix



>  +
>> +@section directory Directory structure
>> +
>> +Test contains all the tests that must be passed for a given revision of
>> the API to be declared compliant.
>> +The tests cover:
>> + - the unit tests
>> + - documentation tests
>>
>
> what is documentation test case?
>
> Currently running make doxygen-html & doxygen-pdf and proving that a
document is created without warnings

>
>  + - functional & benchmark tests
>> +
>> +@verbatim
>> +test
>> +|
>> +|-- functional
>> +|  |-- <test executable>
>> +|  |-- <API group>/<test executable>
>> +|
>> +|-- documentation
>> +|  |--<test case>
>> +|
>> +|-- cunit
>> +   |-- <test executable>
>> +   |-- <API group>/<test executable>
>> +
>> +@endverbatim
>> +
>> +@section compliance ODP-Compliance test suite
>> +Because this suite is extended with every API extension it needs to be
>> very light weight.
>> +This suite would be run by any new platform attempting to be API
>> compliant, it will try to pass in the obvious combinations of arguments
>> that arise from reading the API definition.
>> +For example a unit test is of the following form.
>> +
>> +@code
>> +/* a is in the range 0 to 10
>> +returns a *2 or -1 if a is out of range */
>>
>
>
>
>  +int foo (unsigned int a)
>> +@endcode
>> +
>> +Tests for foo arguments would be
>> + - a=-check that  return = -1
>> + - a=check that  return = 0
>> + - a=check that  return = 10
>> + - a=1check that  return = 20
>> + - a=11 check that return = -1
>> +
>> +The frame work to support generating these test cases is CUnit which
>> provides batch and interactive operation.
>> +Unit test cases do not use any external equipment.
>> +
>> +@section cunit CUnit
>> +CUnit http://cunit.sourceforge.net/doc/index.html is a system for
>> writing, administering, and running unit tests in C.
>> +It is built as a static library which is linked with the user's testing
>> code.
>> +CUnit uses a simple framework for building test structures, and provides
>> a rich set of assertions for testing common data types.
>> +In addition, several different interfaces are provided for running tests
>> and reporting results.
>> +These include automated interfaces for code-controlled testing and
>> reporting, as well as interactive interfaces allowing the user to run tests
>> and view results dynamically.
>> +
>> +@subsection structure Structure
>> +CUnit is a combination of a platform-independent framework with various
>> user interfaces.
>> +The core framework provides basic support for managing a test registry,
>> suites, and test cases.
>> +The user interfaces facilitate interaction with the framework to run
>> tests and view results.
>> +CUnit is organized like a conventional unit testing framework:
>> +
>> +@verbatim
>> +                  Test Registry
>> +                         |
>> +          ------------------------------
>> +          |                            |
>> +       Suite '1'      . . . .       Suite 'N'
>> +          |                            |
>> +    ---------------             ---------------
>> +    |             |             |             |
>> + Test '11' ... Test '1M'     Test 'N1' ... Test 'NM'
>> +@endverbatim
>> +
>> +Individual test cases are packaged into suites, which are registered
>> with the active test registry.
>> +Suites can have setup and tear down functions which are automatically
>> called before and after running the suite's tests.
>> +All suites/tests in the registry may be run using a single function
>> call, or selected suites or tests can be run.
>> +
>> +@subsection general_usage General Usage
>> + - A typical sequence of steps for using the CUnit framework is:
>> + - Write functions for tests (and suite init/cleanup if necessary).
>> + - Initialize the test registry - CU_initialize_registry()
>> + - Add suites to the test registry - CU_add_suite()
>> + - Add tests to the suites - CU_add_test()
>> + - Run tests using an appropriate interface, e.g. CU_console_run_tests
>> + - Cleanup the test registry - CU_cleanup_registry
>> +
>> +@code
>> +/* Copyright (c) 2014, Linaro Limited
>> + * All rights reserved.
>> + *
>> + * SPDX-License-Identifier BSD-3-Clause
>> + */
>> +
>> +#include "odp.h"
>> +#include "CUnit/Basic.h"
>> +
>> +#define DEFAULT_MSG_POOL_SIZE (4*1024*1024)
>> +#define DEFAULT_MSG_SIZE (8)
>> +
>> +static void test_odp_init_global(void)
>> +{
>> +
>> +     int status;
>> +     status = odp_init_global();
>> +     CU_ASSERT (status == 0) ;
>> +}
>> +
>> +static int init(void)
>> +{
>> +     printf("ODP version:%s\n",odp_version_api_str());
>> +     return 0;
>> +}
>> +
>> +
>> +int main(void)
>> +{
>> +     CU_pSuite pSuite = NULL;
>>
>
> No need to set it to NULL here. Add empty line after variable definition.



>
>
>  +     /* initialize the CUnit test registry */
>> +     if (CUE_SUCCESS != CU_initialize_registry())
>> +             return CU_get_error();
>> +     /* add a suite to the registry */
>> +     pSuite = CU_add_suite("odp_init", init, NULL);
>> +     if (NULL == pSuite) {
>>
> if (!pSuite) ?




>
>  +             CU_cleanup_registry();
>> +             return CU_get_error();
>> +     }
>> +     /* add the tests to the suite */
>> +     if (NULL == CU_ADD_TEST(pSuite, test_odp_init_global)) {
>>
>
> It's better for follow our odp style. Like:
>
>
> ret =  CU_ADD_TEST(pSuite, test_odp_init_global);
> if (!ret) {
>
>
> }
>
>  +             CU_cleanup_registry();
>> +             return CU_get_error();
>> +     }
>> +     /* Run all tests using the CUnit Basic interface */
>> +     CU_basic_set_mode(CU_BRM_VERBOSE);
>> +     CU_basic_run_tests();
>> +     CU_cleanup_registry();
>> +     return CU_get_error();
>> +}
>> +@endcode
>> +
>> +@subsection results Normal results
>> +
>> +@code
>> +  CUnit - A unit testing framework for C - Version 2.1-2
>> +  http://cunit.sourceforge.net/
>> +
>> +ODP version:0.0.1
>> +
>> +Suite: odp_init
>> +  Test: test_odp_init_global ...
>> +Buffer pool init global
>> +  pool_entry_s siz   192
>> +  pool_entry_t siz   192
>> +  odp_buffer_hdr_t size 120
>> +
>> +Queue init ... done
>> +Queue init global
>> +  struct queue_entry_s size 192
>> +  queue_entry_t siz      192
>> +
>> +Schedule init ... done
>> +Timer init ...done
>> +passed
>> +
>> +Run Summary  Type  Total    Ran Passed Failed Inactive
>> +           suites      1      1    n/a      0        0
>> +            tests      1      1      1      0        0
>> +          asserts      1      1      1      0      n/a
>> +@endcode
>> +
>> +
>> +@subsection compliance_xml Compliance results (XML)
>> +The CUnit results may also be gathered in XML format, this format may be
>> use to submit an implementations CUnit test case completion for inclusion
>> on the ODP website as compliant to a given release.
>> +
>> +~~~{.xml}
>> +<?xml version="1.0" ?>.
>> +<?xml-stylesheet type="text/xsl" href="CUnit-Run.xsl" ?>.
>> +<!DOCTYPE CUNIT_TEST_RUN_REPORT SYSTEM "CUnit-Run.dtd">.
>> +<CUNIT_TEST_RUN_REPORT>.
>> +  <CUNIT_HEADER/>.
>> +  <CUNIT_RESULT_LISTING>.
>> + <CUNIT_RUN_SUITE>.
>> +   <CUNIT_RUN_SUITE_SUCCESS>.
>> +     <SUITE_NAME> odp initialization </SUITE_NAME>.
>> +     <CUNIT_RUN_TEST_RECORD>.
>> +       <CUNIT_RUN_TEST_SUCCESS>.
>> +         <TEST_NAME> test_odp_init_global </TEST_NAME>.
>> +       </CUNIT_RUN_TEST_SUCCESS>.
>> +     </CUNIT_RUN_TEST_RECORD>.
>> +   </CUNIT_RUN_SUITE_SUCCESS>.
>> + </CUNIT_RUN_SUITE>.
>> +  </CUNIT_RESULT_LISTING>
>> +  <CUNIT_RUN_SUMMARY>.
>> + <CUNIT_RUN_SUMMARY_RECORD>.
>> +   <TYPE> Suites </TYPE>.
>> +   <TOTAL> 1 </TOTAL>.
>> +   <RUN> 1 </RUN>.
>> +   <SUCCEEDED> - NA - </SUCCEEDED>.
>> +   <FAILED> 0 </FAILED>.
>> +   <INACTIVE> 0 </INACTIVE>.
>> + </CUNIT_RUN_SUMMARY_RECORD>.
>> + <CUNIT_RUN_SUMMARY_RECORD>.
>> +   <TYPE> Test Cases </TYPE>.
>> +   <TOTAL> 1 </TOTAL>.
>> +   <RUN> 1 </RUN>.
>> +   <SUCCEEDED> 1 </SUCCEEDED>.
>> +   <FAILED> 0 </FAILED>.
>> +   <INACTIVE> 0 </INACTIVE>.
>> + </CUNIT_RUN_SUMMARY_RECORD>.
>> + <CUNIT_RUN_SUMMARY_RECORD>.
>> +   <TYPE> Assertions </TYPE>.
>> +   <TOTAL> 1 </TOTAL>.
>> +   <RUN> 1 </RUN>.
>> +   <SUCCEEDED> 1 </SUCCEEDED>.
>> +   <FAILED> 0 </FAILED>.
>> +   <INACTIVE> n/a </INACTIVE>.
>> + </CUNIT_RUN_SUMMARY_RECORD>.
>> +  </CUNIT_RUN_SUMMARY>.
>> +  <CUNIT_FOOTER> File Generated By CUnit v2.1-2 - Thu Oct 23 15:02:46
>> 2014
>> + </CUNIT_FOOTER>.
>> +</CUNIT_TEST_RUN_REPORT>
>> +~~~
>> +
>> +@section functional Functional tests
>> +
>> +These tests are generally more elaborate with a goal of proving a
>> complex functionality.
>> +@verbatim
>> +test
>> +.
>> +|-- functional
>> +  |-- <test executable>
>> +  |-- <API group>/<test executable>
>> +@endverbatim
>> + - There is no framework for these tests and they generally take command
>> line arguments to modify their behaviour.
>> + - These tests do provide a pass fail indication.
>> + - These tests may require network interfaces and external equipment.
>> +
>> +@section benchmarking Benchmarks
>> +
>> +Benchmarks are currently also stored with the functional tests, this may
>> be re assessed depending on the test volume.
>> +@verbatim
>> +test
>> +.
>> +|-- functional
>> +  |-- <test executable>
>> +  |-- <API group>/<test executable>
>> +@endverbatim
>> + - These tests are generally more elaborate with a goal of quantifying
>> the capacity of a complex functionality.
>> + - There is no framework for these benchmarks and they generally take
>> command line arguments to modify their behaviour.
>> + - They do not provide a pass fail indication.
>> +
>> +@section test_versioning Test versioning relationship to ODP and
>> versioning
>> +
>> +For ODP to declare for example the 1.0 tag in its git repository it must
>> have passed the corresponding test suit.
>>
> v1.0
>
>> +When ODP API v3.0 is released it will be validated against the test
>> suite version tagged ODP v3.0
>> +It is possible that a version of the ODP API is released and that the
>> test suite used has a flaw, in this case the test suite will be re released
>> as v3.0-1
>> +*/
>>
>
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> http://lists.linaro.org/mailman/listinfo/lng-odp
>