From patchwork Fri Jan 4 13:19:14 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Antonio Terceiro X-Patchwork-Id: 13801 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 41F3723ED1 for ; Fri, 4 Jan 2013 13:19:18 +0000 (UTC) Received: from mail-vb0-f43.google.com (mail-vb0-f43.google.com [209.85.212.43]) by fiordland.canonical.com (Postfix) with ESMTP id D3AE1A18F8E for ; Fri, 4 Jan 2013 13:19:17 +0000 (UTC) Received: by mail-vb0-f43.google.com with SMTP id fs19so16283298vbb.16 for ; Fri, 04 Jan 2013 05:19:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:x-forwarded-to:x-forwarded-for:delivered-to:x-received :received-spf:content-type:mime-version:x-launchpad-project :x-launchpad-branch:x-launchpad-message-rationale :x-launchpad-branch-revision-number:x-launchpad-notification-type:to :from:subject:message-id:date:reply-to:sender:errors-to:precedence :x-generated-by:x-launchpad-hash:x-gm-message-state; bh=NDzcXB0RbC6rVRZdjYeLH/wiAPnyHP51/BI+Mt2gY7w=; b=Onvd6scODSHxXn0JIm0VJ4HOnfQo5FpDGHpUbKy0LTCsZ9vaZeayPlA3eGGK6YlVKK uGjVgZW9j3AjTQCGFRvnjrnAJjK361cDyCqxK/u67FwlPNhJBREBIhBIQiTf5sSrMTsF HnNAzU3kw6oQfbQMGzBtouDmPEH/siuIhbzVN5XXG4/FnnI1WKvqwLXzlxlq16dsbJw0 raYGBLY+OaVJsf6ENCY9DAOhK3lYAIpWnrjMUqf+57X0n6cbB+KOyN+dqTDYM1iIU7i8 OH/9tv8rQxUeFmoVvPeH8tunoGArNOtiPleu645UvltTP/VeeAWGR855Ju2W7t0rDNjN mKfw== X-Received: by 10.52.18.147 with SMTP id w19mr67105024vdd.94.1357305557213; Fri, 04 Jan 2013 05:19:17 -0800 (PST) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.58.145.101 with SMTP id st5csp136488veb; Fri, 4 Jan 2013 05:19:16 -0800 (PST) X-Received: by 10.194.174.196 with SMTP id bu4mr83464305wjc.35.1357305555469; Fri, 04 Jan 2013 05:19:15 -0800 (PST) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id wp3si75908311wjc.77.2013.01.04.05.19.14 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 04 Jan 2013 05:19:15 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) smtp.mail=bounces@canonical.com Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1Tr7B4-0002QM-BS for ; Fri, 04 Jan 2013 13:19:14 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id 470F9E0011 for ; Fri, 4 Jan 2013 13:19:14 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-dispatcher X-Launchpad-Branch: ~linaro-validation/lava-dispatcher/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 517 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-dispatcher/trunk] Rev 517: Make lava-test-shell work with the Galaxy Nexus + AOSP Message-Id: <20130104131914.12887.76053.launchpad@ackee.canonical.com> Date: Fri, 04 Jan 2013 13:19:14 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="16393"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: 603a11ac3bd7b66e9b5a5523ed609abd09a096c5 X-Gm-Message-State: ALoCoQmUFXs84GjZPymuVsOrK+ko9DexjRJmLxlOBSrt9G3hEL2bRoUOIYGj/rwXUHAwe8nP1rPU Merge authors: Antonio Terceiro (terceiro) Related merge proposals: https://code.launchpad.net/~terceiro/lava-dispatcher/lava-test-shell-nexus-aosp/+merge/139796 proposed by: Antonio Terceiro (terceiro) review: Approve - Michael Hudson-Doyle (mwhudson) ------------------------------------------------------------ revno: 517 [merge] committer: Antonio Terceiro branch nick: trunk timestamp: Fri 2013-01-04 10:12:18 -0300 message: Make lava-test-shell work with the Galaxy Nexus + AOSP This involves dropping the assumption that the connection to the device is a serial connection that can read from /dev/console. In the Nexus, and in other consumer devices, the only connection available to the device without hardware tinkering is `adb shell`. Because of this we cannot, we cannot run the lava-test-shell command in the background and redirect its output to /dev/console; instead, we have to run it in the foreground and make sure that stdin/stdout are properly handled. Additionally, some points where the lava-test-shell scripts were assuming a shell more capable than the one in AOSP had to be reworked. Two notable cases where the usage of `read -t` (-t is not in POSIX, and not so widely implemented) and of mkfifo (AOSP does not include a mkfifo binary). removed: lava_test_shell/lava-test-runner.conf lava_test_shell/lava-test-runner.init.d modified: lava_dispatcher/actions/lava_test_shell.py lava_test_shell/lava-test-case lava_test_shell/lava-test-case-attach lava_test_shell/lava-test-run-attach lava_test_shell/lava-test-runner-android lava_test_shell/lava-test-runner-ubuntu lava_test_shell/lava-test-shell --- lp:lava-dispatcher https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk You are subscribed to branch lp:lava-dispatcher. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk/+edit-subscription === modified file 'lava_dispatcher/actions/lava_test_shell.py' --- lava_dispatcher/actions/lava_test_shell.py 2013-01-02 22:30:20 +0000 +++ lava_dispatcher/actions/lava_test_shell.py 2013-01-04 13:09:27 +0000 @@ -103,6 +103,7 @@ # After the test run has completed, the /lava/results directory is pulled over # to the host and turned into a bundle for submission to the dashboard. +import glob import logging import os import pexpect @@ -126,11 +127,6 @@ from lava_dispatcher.device.target import Target from lava_dispatcher.downloader import download_image -# Reading from STDIN in the lava-test-shell doesn't work well because its -# STDIN is /dev/console which we are doing echo's on in our scripts. This -# just makes a well known fifo we can read the ACK's with -ACK_FIFO = '/lava_ack.fifo' - LAVA_TEST_DIR = '%s/../../lava_test_shell' % os.path.dirname(__file__) LAVA_TEST_ANDROID = '%s/lava-test-runner-android' % LAVA_TEST_DIR LAVA_TEST_UBUNTU = '%s/lava-test-runner-ubuntu' % LAVA_TEST_DIR @@ -171,36 +167,6 @@ # 755 file permissions XMOD = stat.S_IRWXU | stat.S_IXGRP | stat.S_IRGRP | stat.S_IXOTH | stat.S_IROTH - -def _configure_ubuntu_startup(etcdir): - logging.info('adding ubuntu upstart job') - shutil.copy(LAVA_TEST_UPSTART, '%s/init/' % etcdir) - -Target.ubuntu_deployment_data['lava_test_configure_startup'] = \ - _configure_ubuntu_startup - - -def _configure_oe_startup(etcdir): - logging.info('adding init.d script') - initd_file = '%s/init.d/lava-test-runner' % etcdir - shutil.copy(LAVA_TEST_INITD, initd_file) - os.chmod(initd_file, XMOD) - shutil.copy(initd_file, '%s/rc5.d/S50lava-test-runner' % etcdir) - shutil.copy(initd_file, '%s/rc6.d/K50lava-test-runner' % etcdir) - -Target.oe_deployment_data['lava_test_configure_startup'] = \ - _configure_oe_startup - - -def _configure_android_startup(etcdir): - logging.info('hacking android start up job') - with open('%s/mkshrc' % etcdir, 'a') as f: - f.write('\n/data/lava/bin/lava-test-runner\n') - -Target.android_deployment_data['lava_test_configure_startup'] = \ - _configure_android_startup - - def _get_testdef_git_repo(testdef_repo, tmpdir, revision): cwd = os.getcwd() gitdir = os.path.join(tmpdir, 'gittestrepo') @@ -417,20 +383,18 @@ with open('%s/run.sh' % hostdir, 'w') as f: f.write('set -e\n') f.write('export TESTRUN_ID=%s\n' % self.test_run_id) - f.write('[ -p %s ] && rm %s\n' % (ACK_FIFO, ACK_FIFO)) - f.write('mkfifo %s\n' % ACK_FIFO) f.write('cd %s\n' % targetdir) f.write('UUID=`cat uuid`\n') f.write('echo ""\n') - f.write('#wait up to 10 minutes for an ack from the dispatcher\n') - f.write('read -t 600 < %s\n' % ACK_FIFO) + f.write('#wait for an ack from the dispatcher\n') + f.write('read\n') steps = self.testdef['run'].get('steps', []) if steps: for cmd in steps: f.write('%s\n' % cmd) f.write('echo ""\n') - f.write('#wait up to 10 minutes for an ack from the dispatcher\n') - f.write('read -t 600 < %s\n' % ACK_FIFO) + f.write('#wait for an ack from the dispatcher\n') + f.write('read\n') class RepoTestDefinition(URLTestDefinition): @@ -489,6 +453,8 @@ signal_director = SignalDirector(self.client, testdefs_by_uuid) with target.runner() as runner: + runner.run("") # make sure we have a shell prompt + runner._connection.sendline("%s/bin/lava-test-runner" % target.deployment_data['lava_test_dir']) start = time.time() if timeout == -1: timeout = runner._connection.timeout @@ -516,12 +482,13 @@ logging.warn('lava_test_shell has timed out') elif idx == 3: name, params = runner._connection.match.groups() + logging.debug("Received signal <%s>" % name) params = params.split() try: signal_director.signal(name, params) except: logging.exception("on_signal failed") - runner._connection.sendline('echo LAVA_ACK > %s' % ACK_FIFO) + runner._connection.sendline('echo LAVA_ACK') return True return False @@ -545,7 +512,6 @@ with open(tc, 'r') as fin: with open('%s/bin/lava-test-case' % mntdir, 'w') as fout: fout.write('#!%s\n\n' % shcmd) - fout.write('ACK_FIFO=%s\n' % ACK_FIFO) fout.write(fin.read()) os.fchmod(fout.fileno(), XMOD) @@ -588,9 +554,6 @@ for testdir in tdirs: f.write('%s\n' % testdir) - with target.file_system(target.config.root_part, 'etc') as d: - target.deployment_data['lava_test_configure_startup'](d) - return testdef_loader.testdefs_by_uuid def _bundle_results(self, target, signal_director, testdefs_by_uuid): @@ -620,7 +583,7 @@ raise RuntimeError('Target includes no deployment_data') keys = ['lava_test_runner', 'lava_test_shell', 'lava_test_dir', - 'lava_test_configure_startup', 'lava_test_sh_cmd'] + 'lava_test_sh_cmd'] for k in keys: if k not in target.deployment_data: raise RuntimeError('Target deployment_data missing %s' % k) === modified file 'lava_test_shell/lava-test-case' --- lava_test_shell/lava-test-case 2012-12-18 13:40:59 +0000 +++ lava_test_shell/lava-test-case 2012-12-19 20:54:43 +0000 @@ -19,12 +19,12 @@ fi if [ "$1" = "--shell" ]; then shift - echo "" > /dev/console - read -t 600 < $ACK_FIFO + echo "" + read $* rc=$? - echo "" > /dev/console - read -t 600 < $ACK_FIFO + echo "" + read if [ $rc -eq 0 ]; then RESULT=pass else === modified file 'lava_test_shell/lava-test-case-attach' --- lava_test_shell/lava-test-case-attach 2012-12-11 19:49:25 +0000 +++ lava_test_shell/lava-test-case-attach 2012-12-19 20:54:43 +0000 @@ -4,7 +4,7 @@ # basename is not present on AOSP builds, but the /*\// thing does not # work with dash (Ubuntu builds) or busybox (OpenEmbedded). Both of # those have basename though. -which basename > /dev/null || basename () { echo ${1/*\//}; } +type basename > /dev/null || basename () { echo ${1/*\//}; } usage () { echo "Usage: lava-test-case-attach TEST_CASE_ID FILE [MIME_TYPE]" @@ -12,17 +12,18 @@ echo "Attach FILE to the test case TEST_CASE_ID." } +if [ $# -ne 2 -a $# -ne 3 ]; then + usage + exit 1 +fi + TEST_CASE_ID="$1" shift FILE="$1" shift MIMETYPE="$1" -shift -if [ $# -gt 0 ]; then - usage - exit 1 -fi -if [ ! -z "$FILE" ]; then + +if [ -z "$FILE" ]; then usage exit 1 fi === modified file 'lava_test_shell/lava-test-run-attach' --- lava_test_shell/lava-test-run-attach 2012-12-11 19:49:25 +0000 +++ lava_test_shell/lava-test-run-attach 2012-12-19 20:54:43 +0000 @@ -4,23 +4,24 @@ # basename is not present on AOSP builds, but the /*\// thing does not # work with dash (Ubuntu builds) or busybox (OpenEmbedded). Both of # those have basename though. -which basename > /dev/null || basename () { echo ${1/*\//}; } +type basename > /dev/null || basename () { echo ${1/*\//}; } usage () { - echo "Usage: lava-test--attach FILE [MIME_TYPE]" + echo "Usage: lava-test-run-attach FILE [MIME_TYPE]" echo "" echo "Attach FILE to the current test run." } +if [ $# -ne 1 -a $# -ne 2 ]; then + usage + exit 1 +fi + FILE="$1" shift MIMETYPE="$1" -shift -if [ $# -gt 0 ]; then - usage - exit 1 -fi -if [ ! -z "$FILE" ]; then + +if [ -z "$FILE" ]; then usage exit 1 fi === modified file 'lava_test_shell/lava-test-runner-android' --- lava_test_shell/lava-test-runner-android 2012-12-03 19:46:40 +0000 +++ lava_test_shell/lava-test-runner-android 2012-12-19 20:54:43 +0000 @@ -9,14 +9,6 @@ exit 0 fi -# make sure we log to serial console -exec >/dev/console - -# This is a total hack to make sure we wait until the shell prompt has -# appeared before sending any signals. -sleep 15 - - PREFIX=":" WORKFILE="/data/lava/lava-test-runner.conf" RESULTSDIR="/data/lava/results" @@ -39,7 +31,7 @@ pkgs=${RESULTSDIR}/swcontext/pkgs.txt [ -f ${build} ] || getprop ro.build.display.id > ${build} - [ -f ${pkgs} ] || pm list packages -v > ${pkgs} + [ -f ${pkgs} ] || pm list packages > ${pkgs} } cleanup() @@ -51,7 +43,6 @@ echo "${PREFIX} exiting" } -# in background, since we don't have this working as a proper android service { trap cleanup INT TERM EXIT @@ -69,11 +60,12 @@ WORKFILE=${WORKFILE}-${TS} echo "${PREFIX} looking for installation work in ${WORKFILE}" - while read line ; do - test=`basename $line` + for line in $(cat ${WORKFILE}); do + # we don't have "basename" on android, but this does the + # equivalent under mksh + testdir=${line%/} # trim off trailing slash iff it exists + test=${testdir/*\//} if [ -f ${line}/install.sh ] ; then - testdir=${line%/} # trim off trailing slash iff it exists - test=${testdir/*\//} echo "${PREFIX} running ${test} installer ..." /system/bin/sh ${line}/install.sh if [ $? -ne 0 ] ; then @@ -83,16 +75,17 @@ exit 1 fi fi - done < ${WORKFILE} + done echo "${PREFIX} save hardware/software context info..." hwcontext swcontext echo "${PREFIX} looking for work in ${WORKFILE}" - while read line ; do - # we don't have "basename" on android, but this is does the + for line in $(cat ${WORKFILE}); do + # we don't have "basename" on android, but this does the # equivalent under mksh + testdir=${line%/} # trim off trailing slash iff it exists test=${testdir/*\//} echo "${PREFIX} running ${test} under lava-test-shell..." odir=${RESULTSDIR}/${test}-`date +%s` @@ -108,6 +101,6 @@ fi lava-test-shell --output_dir ${odir} /system/bin/sh -e "${line}/run.sh" echo "${PREFIX} ${test} exited with: `cat ${odir}/return_code`" - done < ${WORKFILE} -} & + done +} === modified file 'lava_test_shell/lava-test-runner-ubuntu' --- lava_test_shell/lava-test-runner-ubuntu 2012-12-04 22:20:34 +0000 +++ lava_test_shell/lava-test-runner-ubuntu 2012-12-19 20:54:43 +0000 @@ -1,9 +1,5 @@ #!/bin/bash -# This is a total hack to make sure we wait until the shell prompt has -# appeared before sending any signals. -sleep 15 - PREFIX=":" WORKFILE="/lava/lava-test-runner.conf" RESULTSDIR="/lava/results" @@ -72,7 +68,7 @@ swcontext echo "${PREFIX} looking for work in ${WORKFILE}" -while read -u 9 line ; do +for line in $(cat ${WORKFILE}); do test=`basename $line` echo "${PREFIX} running ${test} under lava-test-shell..." odir=${RESULTSDIR}/${test}-`date +%s` @@ -90,5 +86,5 @@ # so be sure to use bash lava-test-shell --output_dir ${odir} /bin/bash -e "${line}/run.sh" echo "${PREFIX} ${test} exited with: `cat ${odir}/return_code`" -done 9< ${WORKFILE} +done === removed file 'lava_test_shell/lava-test-runner.conf' --- lava_test_shell/lava-test-runner.conf 2012-11-20 22:16:17 +0000 +++ lava_test_shell/lava-test-runner.conf 1970-01-01 00:00:00 +0000 @@ -1,12 +0,0 @@ -# lava-test-runner -# -# This is an upstart job to start our lava-test-runner once the system is ready - -description "Launches the lava-test-runner at system start" -author "Linaro Validation Team" - -start on (filesystem and runlevel [2]) - -console output - -exec /lava/bin/lava-test-runner === removed file 'lava_test_shell/lava-test-runner.init.d' --- lava_test_shell/lava-test-runner.init.d 2012-10-23 03:21:33 +0000 +++ lava_test_shell/lava-test-runner.init.d 1970-01-01 00:00:00 +0000 @@ -1,29 +0,0 @@ -#!/bin/sh - -PATH=/lava/bin:/sbin:/bin:/usr/sbin:/usr/bin -DAEMON=/lava/bin/lava-test-runner -NAME="lava-test-runner" - -case "$1" in - start) - echo -n "Starting $NAME: " - start-stop-daemon -S -b -n $NAME --exec $DAEMON - echo "done" - ;; - stop) - echo -n "Stopping $NAME: " - start-stop-daemon -K -n $NAME - echo "done" - ;; - restart) - $0 stop - $0 start - ;; - *) - echo "Usage: $NAME { start | stop | restart }" >&2 - exit 1 - ;; -esac - -exit 0 - === modified file 'lava_test_shell/lava-test-shell' --- lava_test_shell/lava-test-shell 2012-11-18 22:20:57 +0000 +++ lava_test_shell/lava-test-shell 2012-12-13 21:19:18 +0000 @@ -9,4 +9,7 @@ { $TEST echo $? > ${ODIR}/return_code -} 2>&1 | tee ${ODIR}/stdout.log +} 2>&1 | while read line; do + echo "$line" + echo "$line" >> ${ODIR}/stdout.log +done