From patchwork Tue Aug 5 21:29:53 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Mike Holmes X-Patchwork-Id: 34963 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-ie0-f200.google.com (mail-ie0-f200.google.com [209.85.223.200]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 5D3FF20523 for ; Tue, 5 Aug 2014 21:30:22 +0000 (UTC) Received: by mail-ie0-f200.google.com with SMTP id at20sf6358548iec.7 for ; Tue, 05 Aug 2014 14:30:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:date:message-id :mime-version:subject:precedence:list-id:list-unsubscribe :list-archive:list-post:list-help:list-subscribe:errors-to:sender :x-original-sender:x-original-authentication-results:mailing-list :content-type:content-transfer-encoding; bh=id6/sQyV8nycwaMQByP2QGyHpKrj2EjjaTALYNbe5k0=; b=Wm8TklAA4noe09E+scbvmce7kWT+IJmxTkELZA26o9RXSG+eeRV9ee6SMaEW0l20eQ prxfNdc082nOvN16YHV8MrrI+ej2yqGwTTxUVGnrM5JW1NdLsYjjOEftomitvWOUy/kE +VG+s8SRen9GBNKLsxOiQPa8VorQk0HxthCCkWaFOf/WZWMi6GlRMLARbpLVgnuTkiUD XV6VuIX4SqJynZ/ry447YZOmzew/EhGtwA7icWujkCs4qC1tYWA5QLWJS8k4Cx+G9sY9 dllsNrAwKdfh6sKGq1lcyEG/Ja/cUAn6ZR0UwjyvuHUoo0AaOvVpIdUnj57TD7xE5oND onoQ== X-Gm-Message-State: ALoCoQmPrB9T6baFqqi6ao/N3Ucvd5cX8FeCFWZ9Y3treJTtbygvdJnpFR3em/sAA7WB2bJ4vM0C X-Received: by 10.43.1.133 with SMTP id nq5mr4005196icb.21.1407274221860; Tue, 05 Aug 2014 14:30:21 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.81.48 with SMTP id e45ls452806qgd.33.gmail; Tue, 05 Aug 2014 14:30:21 -0700 (PDT) X-Received: by 10.220.189.3 with SMTP id dc3mr5058278vcb.64.1407274221706; Tue, 05 Aug 2014 14:30:21 -0700 (PDT) Received: from mail-vc0-f182.google.com (mail-vc0-f182.google.com [209.85.220.182]) by mx.google.com with ESMTPS id d8si2083091vee.90.2014.08.05.14.30.21 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 05 Aug 2014 14:30:21 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.182 as permitted sender) client-ip=209.85.220.182; Received: by mail-vc0-f182.google.com with SMTP id hy4so2719513vcb.27 for ; Tue, 05 Aug 2014 14:30:21 -0700 (PDT) X-Received: by 10.52.120.38 with SMTP id kz6mr4124035vdb.86.1407274221570; Tue, 05 Aug 2014 14:30:21 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.221.37.5 with SMTP id tc5csp422347vcb; Tue, 5 Aug 2014 14:30:20 -0700 (PDT) X-Received: by 10.224.136.200 with SMTP id s8mr10305143qat.44.1407274220767; Tue, 05 Aug 2014 14:30:20 -0700 (PDT) Received: from ip-10-141-164-156.ec2.internal (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTPS id gs9si5170373qcb.21.2014.08.05.14.30.20 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 05 Aug 2014 14:30:20 -0700 (PDT) Received-SPF: none (google.com: lng-odp-bounces@lists.linaro.org does not designate permitted sender hosts) client-ip=54.225.227.206; Received: from localhost ([127.0.0.1] helo=ip-10-141-164-156.ec2.internal) by ip-10-141-164-156.ec2.internal with esmtp (Exim 4.76) (envelope-from ) id 1XEmJH-0000Ur-Of; Tue, 05 Aug 2014 21:30:19 +0000 Received: from mail-qa0-f44.google.com ([209.85.216.44]) by ip-10-141-164-156.ec2.internal with esmtp (Exim 4.76) (envelope-from ) id 1XEmJC-0000U7-BK for lng-odp@lists.linaro.org; Tue, 05 Aug 2014 21:30:14 +0000 Received: by mail-qa0-f44.google.com with SMTP id f12so1613764qad.31 for ; Tue, 05 Aug 2014 14:30:09 -0700 (PDT) X-Received: by 10.224.104.1 with SMTP id m1mr10185916qao.81.1407274209107; Tue, 05 Aug 2014 14:30:09 -0700 (PDT) Received: from fedora1.holmesfamily.ws (c-98-221-136-245.hsd1.nj.comcast.net. [98.221.136.245]) by mx.google.com with ESMTPSA id h7sm3781428qai.20.2014.08.05.14.30.07 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 05 Aug 2014 14:30:08 -0700 (PDT) From: Mike Holmes To: lng-odp@lists.linaro.org Date: Tue, 5 Aug 2014 17:29:53 -0400 Message-Id: <1407274197-44498-1-git-send-email-mike.holmes@linaro.org> X-Mailer: git-send-email 1.9.1 MIME-Version: 1.0 X-Topics: patch Subject: [lng-odp] [PATCH 1/5] test: Add scripts to exercise all tests X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: lng-odp-bounces@lists.linaro.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: mike.holmes@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.182 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 Signed-off-by: Mike Holmes --- .gitignore | 1 + test/CONTRIBUTING | 61 +++++++++++++++++++++++++++++++ test/README | 27 ++++++++++++++ test/api_test/generic_template.sh | 12 +++++++ test/api_test/odp_atomic.sh | 21 +++++++++++ test/api_test/odp_ring.sh | 12 +++++++ test/api_test/odp_shm.sh | 13 +++++++ test/api_test/odp_timer_ping_root.sh | 13 +++++++ test/run_all.sh | 69 ++++++++++++++++++++++++++++++++++++ 9 files changed, 229 insertions(+) create mode 100644 test/CONTRIBUTING create mode 100755 test/api_test/generic_template.sh create mode 100755 test/api_test/odp_atomic.sh create mode 100755 test/api_test/odp_ring.sh create mode 100755 test/api_test/odp_shm.sh create mode 100755 test/api_test/odp_timer_ping_root.sh create mode 100755 test/run_all.sh diff --git a/.gitignore b/.gitignore index 39c8d77..9fa2ac9 100644 --- a/.gitignore +++ b/.gitignore @@ -43,3 +43,4 @@ odp_timer_test odp_generator odp_l2fwd doxygen-doc +results.txt diff --git a/test/CONTRIBUTING b/test/CONTRIBUTING new file mode 100644 index 0000000..a3b384a --- /dev/null +++ b/test/CONTRIBUTING @@ -0,0 +1,61 @@ +There are three elements to this test suite. + +1. The executable tests & examples built and installed by ODP +2. The run_all.sh top level script in the odp/test directory that discovers and +executes the tests +3. The wrapper scripts located along with each of the test source files. + + +Thus the test structure looks approximately like this + +test +├── api_test +│   ├── generic_template.sh +│   ├── Makefile +│   ├── Makefile.am +│   ├── Makefile.in +│   ├── minunit.h +│   ├── odp_atomic.sh +│   ├── odp_atomic_test.c +│   ├── odp_atomic_test.h +│   ├── odp_buffer +│   ├── odp_buffer.c +│   ├── odp_buffer.sh +│   ├── odp_common.c +│   ├── odp_common.h +│   ├── odp_ring.sh +│   ├── odp_ring_test.c +│   ├── odp_shm.sh +│   ├── odp_shm_test.c +│   ├── odp_shm_test.h +│   ├── odp_timer_ping.c +│   └── odp_timer_ping_root.sh +├── Makefile +├── Makefile.am +├── Makefile.in +├── Makefile.inc +├── README +└── run_all.sh + +To add a new test its C source file should be added, if it is a pure API test +that just attempts to validate the functional interface or the members of a +struct then it should be added in /api_test, if it is a functional test such as +for crypto it would have its own sub directory under test. + +Once this compiles and can be installed it needs a wrapper to call it with all +its cmd line argument permutations. This is done by crating a bash script with +the samename as the executable with a .sh suffix. + +If there are no comandline permuations then the generic template can be copied +and renamed. +For example a new test that compiled and is installed with the name new_test +would need to have the wrapper created with "cp generic_template.sh new_test.sh" +and this new file should be present along side new_test.c + +If the new test requires root permissions to execute a further rename indicates +this fact. +For example "mv new_test.sh new_test_root.sh" + + + + diff --git a/test/README b/test/README index 045f88b..d7fa8c0 100644 --- a/test/README +++ b/test/README @@ -1,2 +1,29 @@ Files in this directory are intended to be terse checks that help ensure that the ODP API Implementations all perform identically and to specification. + +To run all the tests the path to the executables must be specified. +To build and install odp to a location other than the normal system install +location the the path prefix needs to be specified to the configure script +when building. + +To prepare the odp lib and the test binaries:- +odp/bootstrap +odp/.configure --prefix=~/install +make install + +The binaries should now be in ~/install/bin + +In the odp/test directory all the tests can be executed with the run_all.sh +script, passing it the location of the installed executables. If you are root +additional tests can be run, the script will inform you. + +cd odp/test +TEST_PATH=/home/mike/install/bin/ ./run_all.sh + +The list of locations that will be searched for test scripts can be controlled +with TEST_DIR_LIST which defaults to TEST_DIR_LIST="./api_test ../example". + +The resuts will be saved in odp/test and the format is a file named +results_.txt + + diff --git a/test/api_test/generic_template.sh b/test/api_test/generic_template.sh new file mode 100755 index 0000000..afebb15 --- /dev/null +++ b/test/api_test/generic_template.sh @@ -0,0 +1,12 @@ +#!/bin/bash +EXECUTABLE=$(basename $0 .sh) +RESULT_FILE=$1 + +if [ ! -n "$TEST_PATH$EXECUTABLE" ] +then + echo Result:fail Have you installed the EXECUTABLE "$TEST_PATH$EXECUTABLE" +fi + +FULL_CASE="$TEST_PATH$EXECUTABLE" +echo "Calling $FULL_CASE from $0" +$FULL_CASE >> $RESULT_FILE diff --git a/test/api_test/odp_atomic.sh b/test/api_test/odp_atomic.sh new file mode 100755 index 0000000..deef32b --- /dev/null +++ b/test/api_test/odp_atomic.sh @@ -0,0 +1,21 @@ +#!/bin/bash +EXECUTABLE=$(basename $0 .sh) +RESULT_FILE=$1 + +if [[ ! -x "$TEST_PATH$EXECUTABLE" ]] +then + echo "$EXECUTABLE" Result:fail >> $RESULT_FILE +fi + +NUM_CORES=$(grep -c ^processor /proc/cpuinfo) + +for CORES in $(seq 1 $NUM_CORES) +do + for TEST_CASE in 1 2 3 4 5 6 7 + do + FULL_CASE="$TEST_PATH$EXECUTABLE -t $TEST_CASE -n $CORES" + echo "Calling $FULL_CASE from $0" + $FULL_CASE >> $RESULT_FILE + + done +done diff --git a/test/api_test/odp_ring.sh b/test/api_test/odp_ring.sh new file mode 100755 index 0000000..5dc4537 --- /dev/null +++ b/test/api_test/odp_ring.sh @@ -0,0 +1,12 @@ +#!/bin/bash +EXECUTABLE=$(basename $0 .sh) +RESULT_FILE=$1 + +if [[ ! -x "$TEST_PATH$EXECUTABLE" ]] +then + echo "$EXECUTABLE" Result:fail >> $RESULT_FILE +fi + +FULL_CASE="$TEST_PATH$EXECUTABLE" +echo "Calling $FULL_CASE from $0" +$FULL_CASE >> $RESULT_FILE diff --git a/test/api_test/odp_shm.sh b/test/api_test/odp_shm.sh new file mode 100755 index 0000000..82dd56b --- /dev/null +++ b/test/api_test/odp_shm.sh @@ -0,0 +1,13 @@ +#!/bin/bash +EXECUTABLE=$(basename $0 .sh) +RESULT_FILE=$1 + +if [[ ! -x "$TEST_PATH$EXECUTABLE" ]] +then + echo "$EXECUTABLE" Result:fail >> $RESULT_FILE +fi + +FULL_CASE="$TEST_PATH$EXECUTABLE" +echo "Calling $FULL_CASE from $0" +$FULL_CASE >> $RESULT_FILE + diff --git a/test/api_test/odp_timer_ping_root.sh b/test/api_test/odp_timer_ping_root.sh new file mode 100755 index 0000000..d566768 --- /dev/null +++ b/test/api_test/odp_timer_ping_root.sh @@ -0,0 +1,13 @@ +#!/bin/bash +EXECUTABLE=$(basename $0 .sh) +RESULT_FILE=$1 + +if [[ ! -x "$TEST_PATH$EXECUTABLE" ]] +then + echo "$EXECUTABLE" Result:fail >> $RESULT_FILE +fi + +EXECUTABLE=${EXECUTABLE/_root} +FULL_CASE="$INSTALL$EXECUTABLE" +echo "Calling $FULL_CASE from $0" +$FULL_CASE 127.0.0.1 >> $RESULT_FILE diff --git a/test/run_all.sh b/test/run_all.sh new file mode 100755 index 0000000..0f272fd --- /dev/null +++ b/test/run_all.sh @@ -0,0 +1,69 @@ +#!/bin/bash + +NOW=$(date +"%m_%d_%Y:%H:%M:%S") +TEST_DIR_LIST="./api_test ../example" + +#discover where the test executables are +if [ -z "$TEST_PATH" ] +then + echo "Assuming the exeutables are on the PATH" + echo "TEST_PATH= to override" +else + + if [ ! -d "$TEST_PATH" ] + then + echo "TEST_PATH spcified as '$TEST_PATH' which does not exist" + exit -1 + fi + TEST_PATH=$(readlink -f "$TEST_PATH") + TEST_PATH="$TEST_PATH/" + echo "Tests binary install path '$TEST_PATH'" +fi + +#find test scripts to run by conctentatng for each TEST_DIR +echo "Looking for test scripts in $TEST_DIR_LIST" +for TEST_DIR in $TEST_DIR_LIST +do + TEST_SUITE_LIST=$TEST_SUITE_LIST$(find -L $TEST_DIR -executable -type f | \ + egrep "\.sh$" | egrep -v "root.sh$" | grep -v template) + TEST_SUITE_LIST=$(tr '\n' ' '<<< $TEST_SUITE_LIST) + + ROOT_TEST_SUITE_LIST=$ROOT_TEST_SUITE_LIST$(find -L $TEST_DIR -executable -type f | \ + egrep ".*_root.sh$" | grep -v template) + ROOT_TEST_SUITE_LIST=$(tr '\n' ' '<<< $ROOT_TEST_SUITE_LIST) +done + +echo "Found $TEST_SUITE_LIST" | tee -a results_$NOW.txt + +#Inform the users about the list of tests that require root priveldge +if [ ! -z "$ROOT_TEST_SUITE_LIST" ] +then + if [ "$UID" -ne 0 ] + then + echo "Your are NOT root; NOT adding these tests: $ROOT_TEST_SUITE_LIST" + else + echo "Your are root; adding these tests: $ROOT_TEST_SUITE_LIST" + TEST_SUITE_LIST=$TEST_SUITE_LIST $ROOT_TEST_SUITE_LIST + fi +fi + +#Start running the tests +for TEST_SUITE in $TEST_SUITE_LIST +do + echo "Calling $TEST_SUITE from $0" | tee -a results_$NOW.txt + $TEST_SUITE results_$NOW.txt +done + +#parse the results for pass/fail +echo +echo RESULT SUMMARY +grep '^trying\b' results_$NOW.txt +grep 'pass\b\|fail\b' results_$NOW.txt + +grep 'error\b\|warning\b' results_$NOW.txt + +#summary of the results +COUNT_ARRAY=( $TEST_SUITE_LIST ) +echo Attempted ${#COUNT_ARRAY[@]} test scripts, \ + each may have had multiple test cases. +