| Message ID | 1444814900-14384-5-git-send-email-stuart.haslam@linaro.org |
|---|---|
| State | Accepted |
| Commit | 2143920e6a98a88f9a1e2c31d7dca052dadd2a9d |
| Headers | show |
On 2015-10-14 10:28, Stuart Haslam wrote: > Update sections describing how a specific platform may skip tests by > marking them as inactive. > > Signed-off-by: Stuart Haslam <stuart.haslam@linaro.org> Reviewed-by: Christophe Milard <christophe.milard@linaro.org> > --- > doc/implementers-guide/implementers-guide.adoc | 104 +++++++++++++++++-------- > 1 file changed, 71 insertions(+), 33 deletions(-) > > diff --git a/doc/implementers-guide/implementers-guide.adoc b/doc/implementers-guide/implementers-guide.adoc > index 090d4e5..0033ba3 100644 > --- a/doc/implementers-guide/implementers-guide.adoc > +++ b/doc/implementers-guide/implementers-guide.adoc > @@ -1,4 +1,4 @@ > -OpenDataPlane (ODP) Implementers-Guide > +OpenDataPlane (ODP) Implementers Guide > ======================================= > :toc: > > @@ -70,12 +70,12 @@ This grouping defines tests that are expected to be executable and succeed on an > They are written in plain C code, and may only use functions defined in the standard libC library (besides the ODP functions being tested, of course). > No other languages (like scripting) are allowed as their usage would make assumptions on the platform capability. > > -This area is located at: '<ODP_ROOT>/test/validation/' > +This area is located at: 'test/validation/' > > The ODP API itself is ordered by module, where each module groups the set of ODP API functions related to the same "topic". > Examples of modules includes "classification" (API functions dealing with ingres packets classification), time (functions dealing with time, excluding timers which have their own module), timer,... > The complete module list can be seen at: http://docs.opendataplane.org/linux-generic-doxygen-html/modules.html[ODP Modules] + > -Within the platform agnostic area, the tests are also grouped by modules, matching the ODP API modules: '<ODP_ROOT>/test/validation/' mainly contains a list of directories matching each module name (as defined by the doxygen "@defgroup" or "@ingroup" statement present in each API ".h" file). > +Within the platform agnostic area, the tests are also grouped by modules, matching the ODP API modules: 'test/validation/' mainly contains a list of directories matching each module name (as defined by the doxygen "@defgroup" or "@ingroup" statement present in each API ".h" file). > > Within each of these directories, a library (called "libtest<module>.la") and its associated ".h" file (called "<module>.h") defines all the test functions for this module as well as few other functions to initialize, terminate, and group the tests. > An executable called "<module>_main*", is also built. It is permissible to generate more than one executable to cover the functionality in the test library for the module. > @@ -87,38 +87,36 @@ The obvious illustration of this is for module "init" whose functions are requir > > There is a "Makefile.am" located at the top of the platform agnostic area. Its role is limited to the construction of the different test libraries and the "<module>_main*" executables. No tests are run from this area when "make check" is performed. > > -A convenience library '.../test/validation/libodptests.la' (and its associated .h file, '.../test/validation/odptest.h') regrouping all tests symbols of all modules may be built in the future. (The superlib) > - > -C_UNIT > +CUnit > ^^^^^^ > -Within a given test executable C_UNIT is used to run the different tests. The usage of C_UNIT implies the following structure: > +Within a given test executable CUnit is used to run the different tests. The usage of CUnit implies the following structure: > > * Tests are simple C functions. > -* Tests are grouped in arrays called test suites. Each test suite can be associated with a suite initialization/termination function(s), called by C_UNIT before and after the whole suite is ran. > -* An array of test suites (and associated init/term functions) defines the test registry ran by the test executable. > +* Tests are grouped in arrays called test suites. Each test suite can be associated with a suite initialization/termination function(s), called by CUnit before and after the whole suite is run. > +* An array of test suites (and associated init/term functions) defines the test registry run by the test executable. > > -Moreover, two extra functions can be used to initialize/terminate the test executable (these are not part of C_UNIT). + > -A test executable return success (0) if every tests of each suite succeed. > +Moreover, two extra functions can be used to initialize/terminate the test executable (these are not part of CUnit). + > +A test executable return success (0) if every test of each suite succeed. > > -More details about http://cunit.sourceforge.net/doc/index.html[C_Unit users guide] > +More details about http://cunit.sourceforge.net/doc/index.html[CUnit users guide] > > [[anchor-1]] > Module test and naming convention > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > -* Tests, i.e. C functions which are used in CUNIT test suites are named: > +* Tests, i.e. C functions which are used in CUnit test suites are named: > *<Module>_test_+++*+++* + > where the suffix idendify the test. > > -* Test arrays, i.e. arrays of CU_TestInfo, listing the test functions belonging to a suite, are called: > +* Test arrays, i.e. arrays of odp_testinfo_t, listing the test functions belonging to a suite, are called: > *<Module>_suite+++[_*]+++* + > where the possible suffix can be used if many suites are declared. > > -* CUNIT suite init and termination functions are called: > +* CUnit suite init and termination functions are called: > *<Module>+++_suite[_*]_init()+++* and *<Module>+++_suite[_*]_term()+++* respectively. + > where the possible extra middle pattern can be used if many suites are declared. > > -* Suite arrays, i.e. arrays of CU_SuiteInfo used in executables (C_UNIT registry) are called: > +* Suite arrays, i.e. arrays of odp_suiteinfo_t used in executables (CUnit registry) are called: > *<Module>+++_suites[_*]+++* + > where the possible suffix identifies the executable using it, if many. > > @@ -134,14 +132,14 @@ All the above symbols are part of the generated libtest<Module>.la libraries. Th > > Platform specific > ~~~~~~~~~~~~~~~~~ > -These tests are located under '<ODP_ROOT>/platform/<platform>/test'. There is one such area for each platform implementing ODP. > +These tests are located under 'platform/<platform>/test'. There is one such area for each platform implementing ODP. > This location will be referred as <PLATFORM_SPECIFIC> in the rest of this document. > > The normal case > ^^^^^^^^^^^^^^^ > -If the considered platform needs nothing specific to be tested this directory will contain a single "Makefile.am". > -This "Makefile.am" then only lists the executables to be run on "make check" (in the automake TEST variable): when the platform has nothing specific to it, this just list the "<module>_main+++[_*]+++" executables, picked from the platform agnostic area. > -For the linux-generic platform, most tested modules fall into this category: currently, the '<ODP_ROOT>/platform/linux-generic/test/Makefile.am' looks as follows: > +If the considered platform needs no platform specific tests, this directory simply needs to contain a single Makefile.am listing each of the executables (named <module>_main) built from the platform agnostic area. The executables are listed in the automake TEST variable and will therefore be run on "make check". > + > +For the linux-generic platform, most tested modules fall into this category: currently, the 'platform/linux-generic/test/Makefile.am' looks as follows: > > [source,am] > ---- > @@ -179,13 +177,11 @@ endif > > With the exception for module pktio, all other modules testing just involves calling the platform agnostic <module>_main executables (in test/validation). > > -When no platform specific testing is required, the '<PLATFORM_SPECIFIC>/Makefile.am' is used to list the tests executables to be run only, as these tests are actually built from the platform agnostic side by the 'test/validation/Makefile.am' (and subdirectories). '<PLATFORM_SPECIFIC>/Makefile.am' is involved in building only when platform specific tests exists, as discussed below. > - > Using other languages > ^^^^^^^^^^^^^^^^^^^^^ > -The pktio module, above, is actually tested using a bash script. This script is needed to set up the interfaces used by the tests. The pktio_run script actually eventually calls the platform agnostic 'test/validation/pktio/pktio_main' after setting up the interfaces needed by the tests. > -Notice that the path to the script is '<PLATFORM_SPECIFIC>/pktio/pktio_run', i.e. it is private to this platform. Any languages supported by the tested platform can be used there, as it will not impact other platforms. > -The platform "private" executables (such as this script), of course, must also return one of the return code expected by the automake test harness (0 for success, 77 for inconclusive, other values for errors). > +The pktio module, above, is actually tested using a bash script. This script is needed to set up the interfaces used by the tests. The pktio_run script eventually calls the platform agnostic 'test/validation/pktio/pktio_main' after setting up the interfaces needed by the tests. > +Notice that the path to the script, 'pktio/pktio_run', is pointing to a file within the <PLATFORM_SPECIFIC> tree so is private to this platform. Any languages supported by the tested platform can be used there, as it will not impact other platforms. > +The platform "private" executables (such as this script), of course, must also return one of the return code expected by the automake test harness (0 for success, 77 for skipped, other values for errors). > > Defining test wrappers > ^^^^^^^^^^^^^^^^^^^^^^ > @@ -232,14 +228,56 @@ Defining platform specific tests > Sometimes, it may be necessary to call platform specific system calls to check some functionality: For instance, testing odp_cpumask_* could involve checking the underlying system CPU mask. On linux, such a test would require using the CPU_ISSET macro, which is linux specific. Such a test would be written in '<PLATFORM_SPECIFIC>/cpumask/...' The contents of this directory would be very similar to the contents of the platform agnostic side cpu_mask tests (including a Makefile.am...), but platform specific test would be written there. > '<PLATFORM_SPECIFIC>/Makefile.am' would then trigger the building of the platform specific tests (by listing their module name in SUBDIRS and therefore calling the appropriate Makefile.am) and then it would call both the platform agnostic executable(s) and the platform specific test executable. > > -Skipping tests during development > -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > -During ODP development, it may be useful to skip some test. This can be achieved by creating a new test executable (still on the platform side), picking up the required tests from the platform agnostic libtest<module>.la. > +Marking tests as inactive > +^^^^^^^^^^^^^^^^^^^^^^^^^ > +The general policy is that a full run of the validation suite (a "make check") must pass at all times. However a particular platform may have one or more test cases that are known to be unimplemented either during development or permanently, so to avoid these test cases being reported as failures it's useful to be able to skip them. This can be achieved by creating a new test executable (still on the platform side), giving the platform specific initialisation code the opportunity to modify the registered tests in order to mark unwanted tests as inactive while leaving the remaining tests active. It's important that the unwanted tests are still registered with the test framework to allow the fact that they're not being tested to be recorded. > + > +The odp_cunit_update() function is intended for this purpose, it is used to modify the properties of previously registered tests, for example to mark them as inactive. Inactive tests are registered with the test framework but aren't executed and will be recorded as inactive in test reports. > + > +In 'test/validation/foo/foo.c', define all tests for the 'foo' module: > + > +[source,c] > +------------------ > +odp_testinfo_t foo_tests[] = { > + ODP_TEST_INFO(foo_test_a), > + ODP_TEST_INFO(foo_test_b), > + ODP_TEST_INFO_NULL > +}; > + > +odp_suiteinfo_t foo_suites[] = { > + {"Foo", foo_suite_init, foo_suite_term, foo_tests}, > + ODP_SUITE_INFO_NULL > +}; > +------------------ > + > +In 'platform/<platform>/test/foo/foo_main.c', register all the tests defined in the 'foo' module, then mark a single specific test case as inactive: > + > +[source,c] > +------------------ > +static odp_testinfo_t foo_tests_updates[] = { > + ODP_TEST_INFO_INACTIVE(foo_test_b), > + ODP_TEST_INFO_NULL > +}; > + > +static odp_suiteinfo_t foo_suites_updates[] = { > + {"Foo", foo_suite_init, foo_suite_term, foo_tests_updates}, > + ODP_SUITE_INFO_NULL > +}; > + > +int foo_main(void) > +{ > + int ret = odp_cunit_register(foo_suites); > + > + if (ret == 0) > + ret = odp_cunit_update(foo_suites_updates); > + > + if (ret == 0) > + ret = odp_cunit_run(); > > -The top Makefile would then call only the platform specific executable, hence skipping the tests which have been omitted. > + return ret; > +} > +------------------ > > -TIP: You probably want to copy the platform-agnostic module main function and prune it from the undesired tests when writing your own platform specific main, for a given module. > +So 'foo_test_a' will be executed and 'foo_test_b' is inactive. > > -Permanently skipping test > -^^^^^^^^^^^^^^^^^^^^^^^^^^ > -If a platform wants to permanently skip a test (i.e. a part of the ODP API is and will not be supported on that platform), it is recommended to use the function odp_cunit_TBD() to removed the tests or suite from the list of tests. This gives a chance to the test environment to trace this removal. > +It's expected that early in the development cycle of a new implementation the inactive list will be quite long, but it should shrink over time as more parts of the API are implemented. > -- > 2.1.1 >
diff --git a/doc/implementers-guide/implementers-guide.adoc b/doc/implementers-guide/implementers-guide.adoc index 090d4e5..0033ba3 100644 --- a/doc/implementers-guide/implementers-guide.adoc +++ b/doc/implementers-guide/implementers-guide.adoc @@ -1,4 +1,4 @@ -OpenDataPlane (ODP) Implementers-Guide +OpenDataPlane (ODP) Implementers Guide ======================================= :toc: @@ -70,12 +70,12 @@ This grouping defines tests that are expected to be executable and succeed on an They are written in plain C code, and may only use functions defined in the standard libC library (besides the ODP functions being tested, of course). No other languages (like scripting) are allowed as their usage would make assumptions on the platform capability. -This area is located at: '<ODP_ROOT>/test/validation/' +This area is located at: 'test/validation/' The ODP API itself is ordered by module, where each module groups the set of ODP API functions related to the same "topic". Examples of modules includes "classification" (API functions dealing with ingres packets classification), time (functions dealing with time, excluding timers which have their own module), timer,... The complete module list can be seen at: http://docs.opendataplane.org/linux-generic-doxygen-html/modules.html[ODP Modules] + -Within the platform agnostic area, the tests are also grouped by modules, matching the ODP API modules: '<ODP_ROOT>/test/validation/' mainly contains a list of directories matching each module name (as defined by the doxygen "@defgroup" or "@ingroup" statement present in each API ".h" file). +Within the platform agnostic area, the tests are also grouped by modules, matching the ODP API modules: 'test/validation/' mainly contains a list of directories matching each module name (as defined by the doxygen "@defgroup" or "@ingroup" statement present in each API ".h" file). Within each of these directories, a library (called "libtest<module>.la") and its associated ".h" file (called "<module>.h") defines all the test functions for this module as well as few other functions to initialize, terminate, and group the tests. An executable called "<module>_main*", is also built. It is permissible to generate more than one executable to cover the functionality in the test library for the module. @@ -87,38 +87,36 @@ The obvious illustration of this is for module "init" whose functions are requir There is a "Makefile.am" located at the top of the platform agnostic area. Its role is limited to the construction of the different test libraries and the "<module>_main*" executables. No tests are run from this area when "make check" is performed. -A convenience library '.../test/validation/libodptests.la' (and its associated .h file, '.../test/validation/odptest.h') regrouping all tests symbols of all modules may be built in the future. (The superlib) - -C_UNIT +CUnit ^^^^^^ -Within a given test executable C_UNIT is used to run the different tests. The usage of C_UNIT implies the following structure: +Within a given test executable CUnit is used to run the different tests. The usage of CUnit implies the following structure: * Tests are simple C functions. -* Tests are grouped in arrays called test suites. Each test suite can be associated with a suite initialization/termination function(s), called by C_UNIT before and after the whole suite is ran. -* An array of test suites (and associated init/term functions) defines the test registry ran by the test executable. +* Tests are grouped in arrays called test suites. Each test suite can be associated with a suite initialization/termination function(s), called by CUnit before and after the whole suite is run. +* An array of test suites (and associated init/term functions) defines the test registry run by the test executable. -Moreover, two extra functions can be used to initialize/terminate the test executable (these are not part of C_UNIT). + -A test executable return success (0) if every tests of each suite succeed. +Moreover, two extra functions can be used to initialize/terminate the test executable (these are not part of CUnit). + +A test executable return success (0) if every test of each suite succeed. -More details about http://cunit.sourceforge.net/doc/index.html[C_Unit users guide] +More details about http://cunit.sourceforge.net/doc/index.html[CUnit users guide] [[anchor-1]] Module test and naming convention ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -* Tests, i.e. C functions which are used in CUNIT test suites are named: +* Tests, i.e. C functions which are used in CUnit test suites are named: *<Module>_test_+++*+++* + where the suffix idendify the test. -* Test arrays, i.e. arrays of CU_TestInfo, listing the test functions belonging to a suite, are called: +* Test arrays, i.e. arrays of odp_testinfo_t, listing the test functions belonging to a suite, are called: *<Module>_suite+++[_*]+++* + where the possible suffix can be used if many suites are declared. -* CUNIT suite init and termination functions are called: +* CUnit suite init and termination functions are called: *<Module>+++_suite[_*]_init()+++* and *<Module>+++_suite[_*]_term()+++* respectively. + where the possible extra middle pattern can be used if many suites are declared. -* Suite arrays, i.e. arrays of CU_SuiteInfo used in executables (C_UNIT registry) are called: +* Suite arrays, i.e. arrays of odp_suiteinfo_t used in executables (CUnit registry) are called: *<Module>+++_suites[_*]+++* + where the possible suffix identifies the executable using it, if many. @@ -134,14 +132,14 @@ All the above symbols are part of the generated libtest<Module>.la libraries. Th Platform specific ~~~~~~~~~~~~~~~~~ -These tests are located under '<ODP_ROOT>/platform/<platform>/test'. There is one such area for each platform implementing ODP. +These tests are located under 'platform/<platform>/test'. There is one such area for each platform implementing ODP. This location will be referred as <PLATFORM_SPECIFIC> in the rest of this document. The normal case ^^^^^^^^^^^^^^^ -If the considered platform needs nothing specific to be tested this directory will contain a single "Makefile.am". -This "Makefile.am" then only lists the executables to be run on "make check" (in the automake TEST variable): when the platform has nothing specific to it, this just list the "<module>_main+++[_*]+++" executables, picked from the platform agnostic area. -For the linux-generic platform, most tested modules fall into this category: currently, the '<ODP_ROOT>/platform/linux-generic/test/Makefile.am' looks as follows: +If the considered platform needs no platform specific tests, this directory simply needs to contain a single Makefile.am listing each of the executables (named <module>_main) built from the platform agnostic area. The executables are listed in the automake TEST variable and will therefore be run on "make check". + +For the linux-generic platform, most tested modules fall into this category: currently, the 'platform/linux-generic/test/Makefile.am' looks as follows: [source,am] ---- @@ -179,13 +177,11 @@ endif With the exception for module pktio, all other modules testing just involves calling the platform agnostic <module>_main executables (in test/validation). -When no platform specific testing is required, the '<PLATFORM_SPECIFIC>/Makefile.am' is used to list the tests executables to be run only, as these tests are actually built from the platform agnostic side by the 'test/validation/Makefile.am' (and subdirectories). '<PLATFORM_SPECIFIC>/Makefile.am' is involved in building only when platform specific tests exists, as discussed below. - Using other languages ^^^^^^^^^^^^^^^^^^^^^ -The pktio module, above, is actually tested using a bash script. This script is needed to set up the interfaces used by the tests. The pktio_run script actually eventually calls the platform agnostic 'test/validation/pktio/pktio_main' after setting up the interfaces needed by the tests. -Notice that the path to the script is '<PLATFORM_SPECIFIC>/pktio/pktio_run', i.e. it is private to this platform. Any languages supported by the tested platform can be used there, as it will not impact other platforms. -The platform "private" executables (such as this script), of course, must also return one of the return code expected by the automake test harness (0 for success, 77 for inconclusive, other values for errors). +The pktio module, above, is actually tested using a bash script. This script is needed to set up the interfaces used by the tests. The pktio_run script eventually calls the platform agnostic 'test/validation/pktio/pktio_main' after setting up the interfaces needed by the tests. +Notice that the path to the script, 'pktio/pktio_run', is pointing to a file within the <PLATFORM_SPECIFIC> tree so is private to this platform. Any languages supported by the tested platform can be used there, as it will not impact other platforms. +The platform "private" executables (such as this script), of course, must also return one of the return code expected by the automake test harness (0 for success, 77 for skipped, other values for errors). Defining test wrappers ^^^^^^^^^^^^^^^^^^^^^^ @@ -232,14 +228,56 @@ Defining platform specific tests Sometimes, it may be necessary to call platform specific system calls to check some functionality: For instance, testing odp_cpumask_* could involve checking the underlying system CPU mask. On linux, such a test would require using the CPU_ISSET macro, which is linux specific. Such a test would be written in '<PLATFORM_SPECIFIC>/cpumask/...' The contents of this directory would be very similar to the contents of the platform agnostic side cpu_mask tests (including a Makefile.am...), but platform specific test would be written there. '<PLATFORM_SPECIFIC>/Makefile.am' would then trigger the building of the platform specific tests (by listing their module name in SUBDIRS and therefore calling the appropriate Makefile.am) and then it would call both the platform agnostic executable(s) and the platform specific test executable. -Skipping tests during development -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -During ODP development, it may be useful to skip some test. This can be achieved by creating a new test executable (still on the platform side), picking up the required tests from the platform agnostic libtest<module>.la. +Marking tests as inactive +^^^^^^^^^^^^^^^^^^^^^^^^^ +The general policy is that a full run of the validation suite (a "make check") must pass at all times. However a particular platform may have one or more test cases that are known to be unimplemented either during development or permanently, so to avoid these test cases being reported as failures it's useful to be able to skip them. This can be achieved by creating a new test executable (still on the platform side), giving the platform specific initialisation code the opportunity to modify the registered tests in order to mark unwanted tests as inactive while leaving the remaining tests active. It's important that the unwanted tests are still registered with the test framework to allow the fact that they're not being tested to be recorded. + +The odp_cunit_update() function is intended for this purpose, it is used to modify the properties of previously registered tests, for example to mark them as inactive. Inactive tests are registered with the test framework but aren't executed and will be recorded as inactive in test reports. + +In 'test/validation/foo/foo.c', define all tests for the 'foo' module: + +[source,c] +------------------ +odp_testinfo_t foo_tests[] = { + ODP_TEST_INFO(foo_test_a), + ODP_TEST_INFO(foo_test_b), + ODP_TEST_INFO_NULL +}; + +odp_suiteinfo_t foo_suites[] = { + {"Foo", foo_suite_init, foo_suite_term, foo_tests}, + ODP_SUITE_INFO_NULL +}; +------------------ + +In 'platform/<platform>/test/foo/foo_main.c', register all the tests defined in the 'foo' module, then mark a single specific test case as inactive: + +[source,c] +------------------ +static odp_testinfo_t foo_tests_updates[] = { + ODP_TEST_INFO_INACTIVE(foo_test_b), + ODP_TEST_INFO_NULL +}; + +static odp_suiteinfo_t foo_suites_updates[] = { + {"Foo", foo_suite_init, foo_suite_term, foo_tests_updates}, + ODP_SUITE_INFO_NULL +}; + +int foo_main(void) +{ + int ret = odp_cunit_register(foo_suites); + + if (ret == 0) + ret = odp_cunit_update(foo_suites_updates); + + if (ret == 0) + ret = odp_cunit_run(); -The top Makefile would then call only the platform specific executable, hence skipping the tests which have been omitted. + return ret; +} +------------------ -TIP: You probably want to copy the platform-agnostic module main function and prune it from the undesired tests when writing your own platform specific main, for a given module. +So 'foo_test_a' will be executed and 'foo_test_b' is inactive. -Permanently skipping test -^^^^^^^^^^^^^^^^^^^^^^^^^^ -If a platform wants to permanently skip a test (i.e. a part of the ODP API is and will not be supported on that platform), it is recommended to use the function odp_cunit_TBD() to removed the tests or suite from the list of tests. This gives a chance to the test environment to trace this removal. +It's expected that early in the development cycle of a new implementation the inactive list will be quite long, but it should shrink over time as more parts of the API are implemented.
Update sections describing how a specific platform may skip tests by marking them as inactive. Signed-off-by: Stuart Haslam <stuart.haslam@linaro.org> --- doc/implementers-guide/implementers-guide.adoc | 104 +++++++++++++++++-------- 1 file changed, 71 insertions(+), 33 deletions(-)