| Message ID | 1407274197-44498-1-git-send-email-mike.holmes@linaro.org |
|---|---|
| State | Rejected |
| Headers | show |
On Tue, Aug 05, 2014 at 10:29:53PM +0100, Mike Holmes wrote: > Signed-off-by: Mike Holmes <mike.holmes@linaro.org> > --- > .gitignore | 1 + > test/CONTRIBUTING | 61 +++++++++++++++++++++++++++++++ > test/README | 27 ++++++++++++++ > test/api_test/generic_template.sh | 12 +++++++ > test/api_test/odp_atomic.sh | 21 +++++++++++ > test/api_test/odp_ring.sh | 12 +++++++ > test/api_test/odp_shm.sh | 13 +++++++ > test/api_test/odp_timer_ping_root.sh | 13 +++++++ > test/run_all.sh | 69 ++++++++++++++++++++++++++++++++++++ > 9 files changed, 229 insertions(+) > create mode 100644 test/CONTRIBUTING > create mode 100755 test/api_test/generic_template.sh > create mode 100755 test/api_test/odp_atomic.sh > create mode 100755 test/api_test/odp_ring.sh > create mode 100755 test/api_test/odp_shm.sh > create mode 100755 test/api_test/odp_timer_ping_root.sh > create mode 100755 test/run_all.sh > > diff --git a/.gitignore b/.gitignore > index 39c8d77..9fa2ac9 100644 > --- a/.gitignore > +++ b/.gitignore > @@ -43,3 +43,4 @@ odp_timer_test > odp_generator > odp_l2fwd > doxygen-doc > +results.txt Should be results*.txt > diff --git a/test/CONTRIBUTING b/test/CONTRIBUTING > new file mode 100644 > index 0000000..a3b384a > --- /dev/null > +++ b/test/CONTRIBUTING > @@ -0,0 +1,61 @@ > +There are three elements to this test suite. > + > +1. The executable tests & examples built and installed by ODP > +2. The run_all.sh top level script in the odp/test directory that discovers and > +executes the tests > +3. The wrapper scripts located along with each of the test source files. > + > + > +Thus the test structure looks approximately like this > + > +test > +├── api_test > +│ ├── generic_template.sh > +│ ├── Makefile > +│ ├── Makefile.am > +│ ├── Makefile.in > +│ ├── minunit.h > +│ ├── odp_atomic.sh > +│ ├── odp_atomic_test.c > +│ ├── odp_atomic_test.h > +│ ├── odp_buffer > +│ ├── odp_buffer.c > +│ ├── odp_buffer.sh > +│ ├── odp_common.c > +│ ├── odp_common.h > +│ ├── odp_ring.sh > +│ ├── odp_ring_test.c > +│ ├── odp_shm.sh > +│ ├── odp_shm_test.c > +│ ├── odp_shm_test.h > +│ ├── odp_timer_ping.c > +│ └── odp_timer_ping_root.sh > +├── Makefile > +├── Makefile.am > +├── Makefile.in > +├── Makefile.inc > +├── README > +└── run_all.sh > + > +To add a new test its C source file should be added, if it is a pure API test > +that just attempts to validate the functional interface or the members of a > +struct then it should be added in /api_test, if it is a functional test such as > +for crypto it would have its own sub directory under test. > + > +Once this compiles and can be installed it needs a wrapper to call it with all > +its cmd line argument permutations. This is done by crating a bash script with > +the samename as the executable with a .sh suffix. > + > +If there are no comandline permuations then the generic template can be copied > +and renamed. > +For example a new test that compiled and is installed with the name new_test > +would need to have the wrapper created with "cp generic_template.sh new_test.sh" > +and this new file should be present along side new_test.c > + > +If the new test requires root permissions to execute a further rename indicates > +this fact. > +For example "mv new_test.sh new_test_root.sh" > + > I think requiring the wrapper scripts makes it more complicated than it needs to be and will end up cluttering up the api_test directory. IMO it would be better to modify the C source to run all the permutations by default. You could then control which binaries are run by run_all.sh with a text file that lists the binaries, e.g. create a file called tests_enabled.sh with the following; TEST_SUITE_LIST="\ api_test/odp_buffer \ api_test/odp_ring \ api_test/odp_atomic \ api_test/odp_shm" TEST_SUITE_LIST_ROOT="\ api_test/odp_timer_ping" Then just "source" this from run_all.sh. You could still allow these to be overridden; if [ -z "$TEST_SUITE_LIST" ]; then source tests_enabled.sh fi If you did *need* a wrapper script for a particular test case then you could just add that script to the TEST_SUITE_LIST rather than the name of the binary itself. > + > + > diff --git a/test/README b/test/README > index 045f88b..d7fa8c0 100644 > --- a/test/README > +++ b/test/README > @@ -1,2 +1,29 @@ > Files in this directory are intended to be terse checks that help ensure > that the ODP API Implementations all perform identically and to specification. > + > +To run all the tests the path to the executables must be specified. > +To build and install odp to a location other than the normal system install > +location the the path prefix needs to be specified to the configure script > +when building. > + > +To prepare the odp lib and the test binaries:- > +odp/bootstrap > +odp/.configure --prefix=~/install This doesn't work for me.. $ ./configure --prefix=~/install configure: error: expected an absolute directory name for --prefix: ~/install Because the ~ only gets expanded if it's the first character in a word (at least in bash..), also the paths are wrong, how about this.. ./bootstrap ./configure --prefix=$HOME/install > +make install > + > +The binaries should now be in ~/install/bin > + > +In the odp/test directory all the tests can be executed with the run_all.sh > +script, passing it the location of the installed executables. If you are root > +additional tests can be run, the script will inform you. > + > +cd odp/test > +TEST_PATH=/home/mike/install/bin/ ./run_all.sh > + > +The list of locations that will be searched for test scripts can be controlled > +with TEST_DIR_LIST which defaults to TEST_DIR_LIST="./api_test ../example". Better to say it defaults to running all available tests. > + > +The resuts will be saved in odp/test and the format is a file named > +results_<date & time stamp>.txt > + > + > diff --git a/test/api_test/generic_template.sh b/test/api_test/generic_template.sh > new file mode 100755 > index 0000000..afebb15 > --- /dev/null > +++ b/test/api_test/generic_template.sh > @@ -0,0 +1,12 @@ > +#!/bin/bash > +EXECUTABLE=$(basename $0 .sh) > +RESULT_FILE=$1 > + > +if [ ! -n "$TEST_PATH$EXECUTABLE" ] > +then > + echo Result:fail Have you installed the EXECUTABLE "$TEST_PATH$EXECUTABLE" > +fi > + > +FULL_CASE="$TEST_PATH$EXECUTABLE" > +echo "Calling $FULL_CASE from $0" > +$FULL_CASE >> $RESULT_FILE > diff --git a/test/api_test/odp_atomic.sh b/test/api_test/odp_atomic.sh > new file mode 100755 > index 0000000..deef32b > --- /dev/null > +++ b/test/api_test/odp_atomic.sh > @@ -0,0 +1,21 @@ > +#!/bin/bash > +EXECUTABLE=$(basename $0 .sh) > +RESULT_FILE=$1 > + > +if [[ ! -x "$TEST_PATH$EXECUTABLE" ]] > +then > + echo "$EXECUTABLE" Result:fail >> $RESULT_FILE > +fi > + > +NUM_CORES=$(grep -c ^processor /proc/cpuinfo) > + > +for CORES in $(seq 1 $NUM_CORES) > +do > + for TEST_CASE in 1 2 3 4 5 6 7 > + do > + FULL_CASE="$TEST_PATH$EXECUTABLE -t $TEST_CASE -n $CORES" > + echo "Calling $FULL_CASE from $0" > + $FULL_CASE >> $RESULT_FILE > + > + done > +done It would be good to be able to execute this script directly in the api_test directory, currently it fails; $ ./odp_ring.sh Calling odp_ring from ./odp_ring.sh ./odp_ring.sh: line 12: $RESULT_FILE: ambiguous redirect > diff --git a/test/run_all.sh b/test/run_all.sh > new file mode 100755 > index 0000000..0f272fd > --- /dev/null > +++ b/test/run_all.sh > @@ -0,0 +1,69 @@ > +#!/bin/bash > + > +NOW=$(date +"%m_%d_%Y:%H:%M:%S") > +TEST_DIR_LIST="./api_test ../example" > + > +#discover where the test executables are > +if [ -z "$TEST_PATH" ] > +then > + echo "Assuming the exeutables are on the PATH" > + echo "TEST_PATH=<your path> to override" > +else > + > + if [ ! -d "$TEST_PATH" ] > + then > + echo "TEST_PATH spcified as '$TEST_PATH' which does not exist" > + exit -1 > + fi > + TEST_PATH=$(readlink -f "$TEST_PATH") > + TEST_PATH="$TEST_PATH/" > + echo "Tests binary install path '$TEST_PATH'" > +fi > + > +#find test scripts to run by conctentatng for each TEST_DIR > +echo "Looking for test scripts in $TEST_DIR_LIST" > +for TEST_DIR in $TEST_DIR_LIST > +do > + TEST_SUITE_LIST=$TEST_SUITE_LIST$(find -L $TEST_DIR -executable -type f | \ > + egrep "\.sh$" | egrep -v "root.sh$" | grep -v template) > + TEST_SUITE_LIST=$(tr '\n' ' '<<< $TEST_SUITE_LIST) > + > + ROOT_TEST_SUITE_LIST=$ROOT_TEST_SUITE_LIST$(find -L $TEST_DIR -executable -type f | \ > + egrep ".*_root.sh$" | grep -v template) > + ROOT_TEST_SUITE_LIST=$(tr '\n' ' '<<< $ROOT_TEST_SUITE_LIST) > +done > + > +echo "Found $TEST_SUITE_LIST" | tee -a results_$NOW.txt > + > +#Inform the users about the list of tests that require root priveldge > +if [ ! -z "$ROOT_TEST_SUITE_LIST" ] > +then > + if [ "$UID" -ne 0 ] > + then > + echo "Your are NOT root; NOT adding these tests: $ROOT_TEST_SUITE_LIST" > + else > + echo "Your are root; adding these tests: $ROOT_TEST_SUITE_LIST" > + TEST_SUITE_LIST=$TEST_SUITE_LIST $ROOT_TEST_SUITE_LIST > + fi > +fi > + > +#Start running the tests > +for TEST_SUITE in $TEST_SUITE_LIST > +do > + echo "Calling $TEST_SUITE from $0" | tee -a results_$NOW.txt > + $TEST_SUITE results_$NOW.txt > +done > + > +#parse the results for pass/fail > +echo > +echo RESULT SUMMARY > +grep '^trying\b' results_$NOW.txt > +grep 'pass\b\|fail\b' results_$NOW.txt > + > +grep 'error\b\|warning\b' results_$NOW.txt > + > +#summary of the results > +COUNT_ARRAY=( $TEST_SUITE_LIST ) > +echo Attempted ${#COUNT_ARRAY[@]} test scripts, \ > + each may have had multiple test cases. > + It would be good to get a summary of the number of tests that executed/passed/failed, even better if they could be split by test binary. This would be easy if each binary output into a different log file, then just "grep -c" them for pass/fail. All of the files could be tar'd up (or concatenated) to make a single results file.
On 6 August 2014 07:53, Stuart Haslam <stuart.haslam@arm.com> wrote: > On Tue, Aug 05, 2014 at 10:29:53PM +0100, Mike Holmes wrote: > > Signed-off-by: Mike Holmes <mike.holmes@linaro.org> > > --- > > .gitignore | 1 + > > test/CONTRIBUTING | 61 > +++++++++++++++++++++++++++++++ > > test/README | 27 ++++++++++++++ > > test/api_test/generic_template.sh | 12 +++++++ > > test/api_test/odp_atomic.sh | 21 +++++++++++ > > test/api_test/odp_ring.sh | 12 +++++++ > > test/api_test/odp_shm.sh | 13 +++++++ > > test/api_test/odp_timer_ping_root.sh | 13 +++++++ > > test/run_all.sh | 69 > ++++++++++++++++++++++++++++++++++++ > > 9 files changed, 229 insertions(+) > > create mode 100644 test/CONTRIBUTING > > create mode 100755 test/api_test/generic_template.sh > > create mode 100755 test/api_test/odp_atomic.sh > > create mode 100755 test/api_test/odp_ring.sh > > create mode 100755 test/api_test/odp_shm.sh > > create mode 100755 test/api_test/odp_timer_ping_root.sh > > create mode 100755 test/run_all.sh > > > > diff --git a/.gitignore b/.gitignore > > index 39c8d77..9fa2ac9 100644 > > --- a/.gitignore > > +++ b/.gitignore > > @@ -43,3 +43,4 @@ odp_timer_test > > odp_generator > > odp_l2fwd > > doxygen-doc > > +results.txt > > Should be results*.txt > Thanks - I will fix that > > > diff --git a/test/CONTRIBUTING b/test/CONTRIBUTING > > new file mode 100644 > > index 0000000..a3b384a > > --- /dev/null > > +++ b/test/CONTRIBUTING > > @@ -0,0 +1,61 @@ > > +There are three elements to this test suite. > > + > > +1. The executable tests & examples built and installed by ODP > > +2. The run_all.sh top level script in the odp/test directory that > discovers and > > +executes the tests > > +3. The wrapper scripts located along with each of the test source files. > > + > > + > > +Thus the test structure looks approximately like this > > + > > +test > > +├── api_test > > +│ ├── generic_template.sh > > +│ ├── Makefile > > +│ ├── Makefile.am > > +│ ├── Makefile.in > > +│ ├── minunit.h > > +│ ├── odp_atomic.sh > > +│ ├── odp_atomic_test.c > > +│ ├── odp_atomic_test.h > > +│ ├── odp_buffer > > +│ ├── odp_buffer.c > > +│ ├── odp_buffer.sh > > +│ ├── odp_common.c > > +│ ├── odp_common.h > > +│ ├── odp_ring.sh > > +│ ├── odp_ring_test.c > > +│ ├── odp_shm.sh > > +│ ├── odp_shm_test.c > > +│ ├── odp_shm_test.h > > +│ ├── odp_timer_ping.c > > +│ └── odp_timer_ping_root.sh > > +├── Makefile > > +├── Makefile.am > > +├── Makefile.in > > +├── Makefile.inc > > +├── README > > +└── run_all.sh > > + > > +To add a new test its C source file should be added, if it is a pure > API test > > +that just attempts to validate the functional interface or the members > of a > > +struct then it should be added in /api_test, if it is a functional test > such as > > +for crypto it would have its own sub directory under test. > > + > > +Once this compiles and can be installed it needs a wrapper to call it > with all > > +its cmd line argument permutations. This is done by crating a bash > script with > > +the samename as the executable with a .sh suffix. > > + > > +If there are no comandline permuations then the generic template can be > copied > > +and renamed. > > +For example a new test that compiled and is installed with the name > new_test > > +would need to have the wrapper created with "cp generic_template.sh > new_test.sh" > > +and this new file should be present along side new_test.c > > + > > +If the new test requires root permissions to execute a further rename > indicates > > +this fact. > > +For example "mv new_test.sh new_test_root.sh" > > + > > > > I think requiring the wrapper scripts makes it more complicated than it > needs to be and will end up cluttering up the api_test directory. IMO it > would be better to modify the C source to run all the permutations by > default. > I wanted to be able to support "normal" applications that need wrapping as they cant be modified such as also running things from the examples directory. > > You could then control which binaries are run by run_all.sh with a text > file that lists the binaries, e.g. create a file called tests_enabled.sh > with the following; > I thought about that but wanted to make it more automated by default, how about optionally you can pass a list ? > > TEST_SUITE_LIST="\ > api_test/odp_buffer \ > api_test/odp_ring \ > api_test/odp_atomic \ > api_test/odp_shm" > > TEST_SUITE_LIST_ROOT="\ > api_test/odp_timer_ping" > > Then just "source" this from run_all.sh. You could still allow these to > be overridden; > > if [ -z "$TEST_SUITE_LIST" ]; then > source tests_enabled.sh > fi > > If you did *need* a wrapper script for a particular test case then you > could just add that script to the TEST_SUITE_LIST rather than the name > of the binary itself. > I could make it look for the binary and call it if there is no script overriding it > > > + > > + > > diff --git a/test/README b/test/README > > index 045f88b..d7fa8c0 100644 > > --- a/test/README > > +++ b/test/README > > @@ -1,2 +1,29 @@ > > Files in this directory are intended to be terse checks that help ensure > > that the ODP API Implementations all perform identically and to > specification. > > + > > +To run all the tests the path to the executables must be specified. > > +To build and install odp to a location other than the normal system > install > > +location the the path prefix needs to be specified to the configure > script > > +when building. > > + > > +To prepare the odp lib and the test binaries:- > > +odp/bootstrap > > +odp/.configure --prefix=~/install > > This doesn't work for me.. > Ok I tried it a few times and convinced myself it was working, I will investigate > > $ ./configure --prefix=~/install > configure: error: expected an absolute directory name for --prefix: > ~/install > > Because the ~ only gets expanded if it's the first character in a word > (at least in bash..), also the paths are wrong, how about this.. > > ./bootstrap > ./configure --prefix=$HOME/install > > > +make install > > + > > +The binaries should now be in ~/install/bin > > + > > +In the odp/test directory all the tests can be executed with the > run_all.sh > > +script, passing it the location of the installed executables. If you > are root > > +additional tests can be run, the script will inform you. > > + > > +cd odp/test > > +TEST_PATH=/home/mike/install/bin/ ./run_all.sh > > + > > +The list of locations that will be searched for test scripts can be > controlled > > +with TEST_DIR_LIST which defaults to TEST_DIR_LIST="./api_test > ../example". > > Better to say it defaults to running all available tests. > Ok, I will update the text > > > + > > +The resuts will be saved in odp/test and the format is a file named > > +results_<date & time stamp>.txt > > + > > + > > diff --git a/test/api_test/generic_template.sh > b/test/api_test/generic_template.sh > > new file mode 100755 > > index 0000000..afebb15 > > --- /dev/null > > +++ b/test/api_test/generic_template.sh > > @@ -0,0 +1,12 @@ > > +#!/bin/bash > > +EXECUTABLE=$(basename $0 .sh) > > +RESULT_FILE=$1 > > + > > +if [ ! -n "$TEST_PATH$EXECUTABLE" ] > > +then > > + echo Result:fail Have you installed the EXECUTABLE > "$TEST_PATH$EXECUTABLE" > > +fi > > + > > +FULL_CASE="$TEST_PATH$EXECUTABLE" > > +echo "Calling $FULL_CASE from $0" > > +$FULL_CASE >> $RESULT_FILE > > diff --git a/test/api_test/odp_atomic.sh b/test/api_test/odp_atomic.sh > > new file mode 100755 > > index 0000000..deef32b > > --- /dev/null > > +++ b/test/api_test/odp_atomic.sh > > @@ -0,0 +1,21 @@ > > +#!/bin/bash > > +EXECUTABLE=$(basename $0 .sh) > > +RESULT_FILE=$1 > > + > > +if [[ ! -x "$TEST_PATH$EXECUTABLE" ]] > > +then > > + echo "$EXECUTABLE" Result:fail >> $RESULT_FILE > > +fi > > + > > +NUM_CORES=$(grep -c ^processor /proc/cpuinfo) > > + > > +for CORES in $(seq 1 $NUM_CORES) > > +do > > + for TEST_CASE in 1 2 3 4 5 6 7 > > + do > > + FULL_CASE="$TEST_PATH$EXECUTABLE -t $TEST_CASE -n $CORES" > > + echo "Calling $FULL_CASE from $0" > > + $FULL_CASE >> $RESULT_FILE > > + > > + done > > +done > > It would be good to be able to execute this script directly in the > api_test directory, currently it fails; > Ok, sounds reasonable, it needs to default to "./" I suspect > > $ ./odp_ring.sh > Calling odp_ring from ./odp_ring.sh > ./odp_ring.sh: line 12: $RESULT_FILE: ambiguous redirect > > > diff --git a/test/run_all.sh b/test/run_all.sh > > new file mode 100755 > > index 0000000..0f272fd > > --- /dev/null > > +++ b/test/run_all.sh > > @@ -0,0 +1,69 @@ > > +#!/bin/bash > > + > > +NOW=$(date +"%m_%d_%Y:%H:%M:%S") > > +TEST_DIR_LIST="./api_test ../example" > > + > > +#discover where the test executables are > > +if [ -z "$TEST_PATH" ] > > +then > > + echo "Assuming the exeutables are on the PATH" > > + echo "TEST_PATH=<your path> to override" > > +else > > + > > + if [ ! -d "$TEST_PATH" ] > > + then > > + echo "TEST_PATH spcified as '$TEST_PATH' which does not exist" > > + exit -1 > > + fi > > + TEST_PATH=$(readlink -f "$TEST_PATH") > > + TEST_PATH="$TEST_PATH/" > > + echo "Tests binary install path '$TEST_PATH'" > > +fi > > + > > +#find test scripts to run by conctentatng for each TEST_DIR > > +echo "Looking for test scripts in $TEST_DIR_LIST" > > +for TEST_DIR in $TEST_DIR_LIST > > +do > > + TEST_SUITE_LIST=$TEST_SUITE_LIST$(find -L $TEST_DIR -executable > -type f | \ > > + egrep "\.sh$" | egrep -v "root.sh$" | grep -v template) > > + TEST_SUITE_LIST=$(tr '\n' ' '<<< $TEST_SUITE_LIST) > > + > > + ROOT_TEST_SUITE_LIST=$ROOT_TEST_SUITE_LIST$(find -L $TEST_DIR > -executable -type f | \ > > + egrep ".*_root.sh$" | grep -v template) > > + ROOT_TEST_SUITE_LIST=$(tr '\n' ' '<<< $ROOT_TEST_SUITE_LIST) > > +done > > + > > +echo "Found $TEST_SUITE_LIST" | tee -a results_$NOW.txt > > + > > +#Inform the users about the list of tests that require root priveldge > > +if [ ! -z "$ROOT_TEST_SUITE_LIST" ] > > +then > > + if [ "$UID" -ne 0 ] > > + then > > + echo "Your are NOT root; NOT adding these tests: > $ROOT_TEST_SUITE_LIST" > > + else > > + echo "Your are root; adding these tests: $ROOT_TEST_SUITE_LIST" > > + TEST_SUITE_LIST=$TEST_SUITE_LIST $ROOT_TEST_SUITE_LIST > > + fi > > +fi > > + > > +#Start running the tests > > +for TEST_SUITE in $TEST_SUITE_LIST > > +do > > + echo "Calling $TEST_SUITE from $0" | tee -a results_$NOW.txt > > + $TEST_SUITE results_$NOW.txt > > +done > > + > > +#parse the results for pass/fail > > +echo > > +echo RESULT SUMMARY > > +grep '^trying\b' results_$NOW.txt > > +grep 'pass\b\|fail\b' results_$NOW.txt > > + > > +grep 'error\b\|warning\b' results_$NOW.txt > > + > > +#summary of the results > > +COUNT_ARRAY=( $TEST_SUITE_LIST ) > > +echo Attempted ${#COUNT_ARRAY[@]} test scripts, \ > > + each may have had multiple test cases. > > + > > It would be good to get a summary of the number of tests that > executed/passed/failed, even better if they could be split by test > binary. This would be easy if each binary output into a different log > file, then just "grep -c" them for pass/fail. All of the files could be > tar'd up (or concatenated) to make a single results file. > > There is a complete pass/fail result produced, but I don't summarise by test suite. I can split into multiple results files, I was trying to keep it all in one simple file Let me check on expanding the summary, or do you expressly want multiple output files ? > -- > Stuart. > >
On Wed, Aug 06, 2014 at 01:40:20PM +0100, Mike Holmes wrote: >> On 6 August 2014 07:53, Stuart Haslam <stuart.haslam@arm.com<mailto:stuart.haslam@arm.com>> wrote: >>> On Tue, Aug 05, 2014 at 10:29:53PM +0100, Mike Holmes wrote: Forgot to mention in the last mail, but I got some whitespace errors when applying this patch due to the use of spaces for indentation rather than tabs in the bash scripts (I have git configured to warn on indent-with-non-tab). I don't think the style policy explicitly mentions anything other than C, but to me it would make sense to use tabs here too for consistency. >> I think requiring the wrapper scripts makes it more complicated than it >> needs to be and will end up cluttering up the api_test directory. IMO it >> would be better to modify the C source to run all the permutations by >> default. > > I wanted to be able to support "normal" applications that need wrapping as they cant be modified > such as also running things from the examples directory. OK fair enough, it's mostly the wrapper scripts which do nothing but call the application without parameters that seem pointless. >> >> You could then control which binaries are run by run_all.sh with a text >> file that lists the binaries, e.g. create a file called tests_enabled.sh >> with the following; > > I thought about that but wanted to make it more automated by default, how about optionally > you can pass a list ? > I think it's better to have an explicit list of enabled test cases/applications. What if we need to add a *.sh script that isn't supposed to be run from run_all.sh? there already needs to be exception for the template.. also how would you temporarily disable a specific test? >> TEST_SUITE_LIST="\ >> api_test/odp_buffer \ >> api_test/odp_ring \ >> api_test/odp_atomic \ >> api_test/odp_shm" >> >> TEST_SUITE_LIST_ROOT="\ >> api_test/odp_timer_ping" >> >> Then just "source" this from run_all.sh. You could still allow these to >> be overridden; >> >> if [ -z "$TEST_SUITE_LIST" ]; then >> source tests_enabled.sh >> fi >> >> If you did *need* a wrapper script for a particular test case then you >> could just add that script to the TEST_SUITE_LIST rather than the name >> of the binary itself. > > I could make it look for the binary and call it if there is no script overriding it > That would be an improvement, but I still think it's better to have an explicit list. >> It would be good to get a summary of the number of tests that >> executed/passed/failed, even better if they could be split by test >> binary. This would be easy if each binary output into a different log >> file, then just "grep -c" them for pass/fail. All of the files could be >> tar'd up (or concatenated) to make a single results file. > > There is a complete pass/fail result produced, but I don't summarise by test suite. > I can split into multiple results files, I was trying to keep it all in one simple file > Let me check on expanding the summary, or do you expressly want multiple output files ? I was just after something like; executed passed failed odp_buffer 12 12 0 odp_ring 3 3 0 total 15 15 0 I don't mind if results are in one or multiple files, I just suggested using multiple files as an easy way to report numbers separately for each binary/API.
diff --git a/.gitignore b/.gitignore index 39c8d77..9fa2ac9 100644 --- a/.gitignore +++ b/.gitignore @@ -43,3 +43,4 @@ odp_timer_test odp_generator odp_l2fwd doxygen-doc +results.txt diff --git a/test/CONTRIBUTING b/test/CONTRIBUTING new file mode 100644 index 0000000..a3b384a --- /dev/null +++ b/test/CONTRIBUTING @@ -0,0 +1,61 @@ +There are three elements to this test suite. + +1. The executable tests & examples built and installed by ODP +2. The run_all.sh top level script in the odp/test directory that discovers and +executes the tests +3. The wrapper scripts located along with each of the test source files. + + +Thus the test structure looks approximately like this + +test +├── api_test +│ ├── generic_template.sh +│ ├── Makefile +│ ├── Makefile.am +│ ├── Makefile.in +│ ├── minunit.h +│ ├── odp_atomic.sh +│ ├── odp_atomic_test.c +│ ├── odp_atomic_test.h +│ ├── odp_buffer +│ ├── odp_buffer.c +│ ├── odp_buffer.sh +│ ├── odp_common.c +│ ├── odp_common.h +│ ├── odp_ring.sh +│ ├── odp_ring_test.c +│ ├── odp_shm.sh +│ ├── odp_shm_test.c +│ ├── odp_shm_test.h +│ ├── odp_timer_ping.c +│ └── odp_timer_ping_root.sh +├── Makefile +├── Makefile.am +├── Makefile.in +├── Makefile.inc +├── README +└── run_all.sh + +To add a new test its C source file should be added, if it is a pure API test +that just attempts to validate the functional interface or the members of a +struct then it should be added in /api_test, if it is a functional test such as +for crypto it would have its own sub directory under test. + +Once this compiles and can be installed it needs a wrapper to call it with all +its cmd line argument permutations. This is done by crating a bash script with +the samename as the executable with a .sh suffix. + +If there are no comandline permuations then the generic template can be copied +and renamed. +For example a new test that compiled and is installed with the name new_test +would need to have the wrapper created with "cp generic_template.sh new_test.sh" +and this new file should be present along side new_test.c + +If the new test requires root permissions to execute a further rename indicates +this fact. +For example "mv new_test.sh new_test_root.sh" + + + + diff --git a/test/README b/test/README index 045f88b..d7fa8c0 100644 --- a/test/README +++ b/test/README @@ -1,2 +1,29 @@ Files in this directory are intended to be terse checks that help ensure that the ODP API Implementations all perform identically and to specification. + +To run all the tests the path to the executables must be specified. +To build and install odp to a location other than the normal system install +location the the path prefix needs to be specified to the configure script +when building. + +To prepare the odp lib and the test binaries:- +odp/bootstrap +odp/.configure --prefix=~/install +make install + +The binaries should now be in ~/install/bin + +In the odp/test directory all the tests can be executed with the run_all.sh +script, passing it the location of the installed executables. If you are root +additional tests can be run, the script will inform you. + +cd odp/test +TEST_PATH=/home/mike/install/bin/ ./run_all.sh + +The list of locations that will be searched for test scripts can be controlled +with TEST_DIR_LIST which defaults to TEST_DIR_LIST="./api_test ../example". + +The resuts will be saved in odp/test and the format is a file named +results_<date & time stamp>.txt + + diff --git a/test/api_test/generic_template.sh b/test/api_test/generic_template.sh new file mode 100755 index 0000000..afebb15 --- /dev/null +++ b/test/api_test/generic_template.sh @@ -0,0 +1,12 @@ +#!/bin/bash +EXECUTABLE=$(basename $0 .sh) +RESULT_FILE=$1 + +if [ ! -n "$TEST_PATH$EXECUTABLE" ] +then + echo Result:fail Have you installed the EXECUTABLE "$TEST_PATH$EXECUTABLE" +fi + +FULL_CASE="$TEST_PATH$EXECUTABLE" +echo "Calling $FULL_CASE from $0" +$FULL_CASE >> $RESULT_FILE diff --git a/test/api_test/odp_atomic.sh b/test/api_test/odp_atomic.sh new file mode 100755 index 0000000..deef32b --- /dev/null +++ b/test/api_test/odp_atomic.sh @@ -0,0 +1,21 @@ +#!/bin/bash +EXECUTABLE=$(basename $0 .sh) +RESULT_FILE=$1 + +if [[ ! -x "$TEST_PATH$EXECUTABLE" ]] +then + echo "$EXECUTABLE" Result:fail >> $RESULT_FILE +fi + +NUM_CORES=$(grep -c ^processor /proc/cpuinfo) + +for CORES in $(seq 1 $NUM_CORES) +do + for TEST_CASE in 1 2 3 4 5 6 7 + do + FULL_CASE="$TEST_PATH$EXECUTABLE -t $TEST_CASE -n $CORES" + echo "Calling $FULL_CASE from $0" + $FULL_CASE >> $RESULT_FILE + + done +done diff --git a/test/api_test/odp_ring.sh b/test/api_test/odp_ring.sh new file mode 100755 index 0000000..5dc4537 --- /dev/null +++ b/test/api_test/odp_ring.sh @@ -0,0 +1,12 @@ +#!/bin/bash +EXECUTABLE=$(basename $0 .sh) +RESULT_FILE=$1 + +if [[ ! -x "$TEST_PATH$EXECUTABLE" ]] +then + echo "$EXECUTABLE" Result:fail >> $RESULT_FILE +fi + +FULL_CASE="$TEST_PATH$EXECUTABLE" +echo "Calling $FULL_CASE from $0" +$FULL_CASE >> $RESULT_FILE diff --git a/test/api_test/odp_shm.sh b/test/api_test/odp_shm.sh new file mode 100755 index 0000000..82dd56b --- /dev/null +++ b/test/api_test/odp_shm.sh @@ -0,0 +1,13 @@ +#!/bin/bash +EXECUTABLE=$(basename $0 .sh) +RESULT_FILE=$1 + +if [[ ! -x "$TEST_PATH$EXECUTABLE" ]] +then + echo "$EXECUTABLE" Result:fail >> $RESULT_FILE +fi + +FULL_CASE="$TEST_PATH$EXECUTABLE" +echo "Calling $FULL_CASE from $0" +$FULL_CASE >> $RESULT_FILE + diff --git a/test/api_test/odp_timer_ping_root.sh b/test/api_test/odp_timer_ping_root.sh new file mode 100755 index 0000000..d566768 --- /dev/null +++ b/test/api_test/odp_timer_ping_root.sh @@ -0,0 +1,13 @@ +#!/bin/bash +EXECUTABLE=$(basename $0 .sh) +RESULT_FILE=$1 + +if [[ ! -x "$TEST_PATH$EXECUTABLE" ]] +then + echo "$EXECUTABLE" Result:fail >> $RESULT_FILE +fi + +EXECUTABLE=${EXECUTABLE/_root} +FULL_CASE="$INSTALL$EXECUTABLE" +echo "Calling $FULL_CASE from $0" +$FULL_CASE 127.0.0.1 >> $RESULT_FILE diff --git a/test/run_all.sh b/test/run_all.sh new file mode 100755 index 0000000..0f272fd --- /dev/null +++ b/test/run_all.sh @@ -0,0 +1,69 @@ +#!/bin/bash + +NOW=$(date +"%m_%d_%Y:%H:%M:%S") +TEST_DIR_LIST="./api_test ../example" + +#discover where the test executables are +if [ -z "$TEST_PATH" ] +then + echo "Assuming the exeutables are on the PATH" + echo "TEST_PATH=<your path> to override" +else + + if [ ! -d "$TEST_PATH" ] + then + echo "TEST_PATH spcified as '$TEST_PATH' which does not exist" + exit -1 + fi + TEST_PATH=$(readlink -f "$TEST_PATH") + TEST_PATH="$TEST_PATH/" + echo "Tests binary install path '$TEST_PATH'" +fi + +#find test scripts to run by conctentatng for each TEST_DIR +echo "Looking for test scripts in $TEST_DIR_LIST" +for TEST_DIR in $TEST_DIR_LIST +do + TEST_SUITE_LIST=$TEST_SUITE_LIST$(find -L $TEST_DIR -executable -type f | \ + egrep "\.sh$" | egrep -v "root.sh$" | grep -v template) + TEST_SUITE_LIST=$(tr '\n' ' '<<< $TEST_SUITE_LIST) + + ROOT_TEST_SUITE_LIST=$ROOT_TEST_SUITE_LIST$(find -L $TEST_DIR -executable -type f | \ + egrep ".*_root.sh$" | grep -v template) + ROOT_TEST_SUITE_LIST=$(tr '\n' ' '<<< $ROOT_TEST_SUITE_LIST) +done + +echo "Found $TEST_SUITE_LIST" | tee -a results_$NOW.txt + +#Inform the users about the list of tests that require root priveldge +if [ ! -z "$ROOT_TEST_SUITE_LIST" ] +then + if [ "$UID" -ne 0 ] + then + echo "Your are NOT root; NOT adding these tests: $ROOT_TEST_SUITE_LIST" + else + echo "Your are root; adding these tests: $ROOT_TEST_SUITE_LIST" + TEST_SUITE_LIST=$TEST_SUITE_LIST $ROOT_TEST_SUITE_LIST + fi +fi + +#Start running the tests +for TEST_SUITE in $TEST_SUITE_LIST +do + echo "Calling $TEST_SUITE from $0" | tee -a results_$NOW.txt + $TEST_SUITE results_$NOW.txt +done + +#parse the results for pass/fail +echo +echo RESULT SUMMARY +grep '^trying\b' results_$NOW.txt +grep 'pass\b\|fail\b' results_$NOW.txt + +grep 'error\b\|warning\b' results_$NOW.txt + +#summary of the results +COUNT_ARRAY=( $TEST_SUITE_LIST ) +echo Attempted ${#COUNT_ARRAY[@]} test scripts, \ + each may have had multiple test cases. +
Signed-off-by: Mike Holmes <mike.holmes@linaro.org> --- .gitignore | 1 + test/CONTRIBUTING | 61 +++++++++++++++++++++++++++++++ test/README | 27 ++++++++++++++ test/api_test/generic_template.sh | 12 +++++++ test/api_test/odp_atomic.sh | 21 +++++++++++ test/api_test/odp_ring.sh | 12 +++++++ test/api_test/odp_shm.sh | 13 +++++++ test/api_test/odp_timer_ping_root.sh | 13 +++++++ test/run_all.sh | 69 ++++++++++++++++++++++++++++++++++++ 9 files changed, 229 insertions(+) create mode 100644 test/CONTRIBUTING create mode 100755 test/api_test/generic_template.sh create mode 100755 test/api_test/odp_atomic.sh create mode 100755 test/api_test/odp_ring.sh create mode 100755 test/api_test/odp_shm.sh create mode 100755 test/api_test/odp_timer_ping_root.sh create mode 100755 test/run_all.sh