From patchwork Mon Mar 19 07:01:13 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yongqin Liu X-Patchwork-Id: 7350 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 53B0D23E00 for ; Mon, 19 Mar 2012 07:01:15 +0000 (UTC) Received: from mail-iy0-f180.google.com (mail-iy0-f180.google.com [209.85.210.180]) by fiordland.canonical.com (Postfix) with ESMTP id 03F5DA186BF for ; Mon, 19 Mar 2012 07:01:14 +0000 (UTC) Received: by mail-iy0-f180.google.com with SMTP id e36so12054887iag.11 for ; Mon, 19 Mar 2012 00:01:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf :content-type:mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state; bh=q6pr+Q56svq0MU6Ah2oTEiXSQHNVF7EAClDaFfOvuFo=; b=Zc+UBk1cbD/p1D6zTJ1IkFucOhNkh0QBsJcas7GZRlxV1lGmQeRXpWZm2q8lM9dHP+ Co7HprROcajhleXR+6/vSvdbCDKF52KxwRRK1IeLNjCfAlMddaf/npOm3VIuls88Nfp+ aniAsWrX4y4MMng+8b8/aqQ9MuCP5QDGbJoYRGjyHFHeT436lCSzr5rKq8yUXyLK52Nl yRY9g3lgGSLCVWNNGQFZAO7QgG8+AWFZvXxHZnrZcdh31MTQe3nBbVmv3R8mRESgxKh2 Bg7bevISbF7CugR45U0vQ+gUDbq48zrUsv+jbnJUTNqFIXYxIlfV2luJkf0c3GNKsz50 X19A== Received: by 10.50.153.193 with SMTP id vi1mr5250886igb.2.1332140474788; Mon, 19 Mar 2012 00:01:14 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.231.203.79 with SMTP id fh15csp65236ibb; Mon, 19 Mar 2012 00:01:14 -0700 (PDT) Received: by 10.180.91.10 with SMTP id ca10mr17237353wib.17.1332140473601; Mon, 19 Mar 2012 00:01:13 -0700 (PDT) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id 8si14227984wel.48.2012.03.19.00.01.13 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 19 Mar 2012 00:01:13 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) smtp.mail=bounces@canonical.com Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1S9Waf-0007aX-3q for ; Mon, 19 Mar 2012 07:01:13 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id 10B50E0768 for ; Mon, 19 Mar 2012 07:01:13 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-dispatcher X-Launchpad-Branch: ~linaro-validation/lava-dispatcher/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 257 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-dispatcher/trunk] Rev 257: merge with the support custom android command branch Message-Id: <20120319070113.2776.82531.launchpad@ackee.canonical.com> Date: Mon, 19 Mar 2012 07:01:13 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="14969"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: bd1fd5b94c22f169b0b2fddc1370eba4ba5fd8fd X-Gm-Message-State: ALoCoQlLPe+ZI29lNxHxTQHV742im+T9KNH+sS/XUC/Fx1xyjHY/1q1ET2s2Uf0Wd2AodI6zJ4uB Merge authors: Yongqin Liu (liuyq0307) Related merge proposals: https://code.launchpad.net/~liuyq0307/lava-dispatcher/support-custom-command/+merge/98093 proposed by: Yongqin Liu (liuyq0307) review: Approve - Michael Hudson-Doyle (mwhudson) ------------------------------------------------------------ revno: 257 [merge] committer: Yongqin Liu branch nick: lava-dispatcher timestamp: Mon 2012-03-19 14:59:17 +0800 message: merge with the support custom android command branch renamed: doc/examples/jobs/lava-android-test-leb-panda.json => doc/examples/jobs/lava-android-test.json modified: lava_dispatcher/actions/lava-android-test.py lava_dispatcher/actions/lava-test.py lava_dispatcher/utils.py doc/examples/jobs/lava-android-test.json --- lp:lava-dispatcher https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk You are subscribed to branch lp:lava-dispatcher. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk/+edit-subscription === renamed file 'doc/examples/jobs/lava-android-test-leb-panda.json' => 'doc/examples/jobs/lava-android-test.json' --- doc/examples/jobs/lava-android-test-leb-panda.json 2012-03-09 01:18:19 +0000 +++ doc/examples/jobs/lava-android-test.json 2012-03-15 09:16:09 +0000 @@ -1,5 +1,5 @@ { - "job_name": "android_monkey_test2", + "job_name": "lava_android_test", "target": "panda01", "timeout": 18000, "actions": [ @@ -7,9 +7,9 @@ "command": "deploy_linaro_android_image", "parameters": { - "boot": "http://releases.linaro.org/11.10/android/leb-panda/boot.tar.bz2", - "system": "http://releases.linaro.org/11.10/android/leb-panda/system.tar.bz2", - "data": "http://releases.linaro.org/11.10/android/leb-panda/userdata.tar.bz2" + "boot": "http://snapshots.linaro.org/android/~linaro-android/panda-ics-gcc46-kwg-upstream-open/540/target/product/pandaboard/boot.tar.bz2", + "system": "http://snapshots.linaro.org/android/~linaro-android/panda-ics-gcc46-kwg-upstream-open/540/target/product/pandaboard/system.tar.bz2", + "data": "http://snapshots.linaro.org/android/~linaro-android/panda-ics-gcc46-kwg-upstream-open/540/target/product/pandaboard/userdata.tar.bz2" }, "metadata": { @@ -42,6 +42,22 @@ } }, { + "command": "lava_android_test_run_custom", + "parameters": + { + "command_file": "http://bazaar.launchpad.net/~linaro-validation/lava-android-test/trunk/download/head:/busybox_test.sh-20110927085925-2fxzf7wrrtq4gci0-3/busybox_test.sh", + "parser": "^\\s*(?P\\w+)=(?P\\w+)\\s*$" + } + }, + { + "command": "lava_android_test_run_custom", + "parameters": + { + "commands": ["tjunittest"], + "parser": "^\\s*(?P.+)\\s+\\.\\.\\.\\s+(?P\\w+)\\.\\s+(?P[\\d\\.]+)\\s+(?P\\w+)\\s*$" + } + }, + { "command": "lava_android_test_run", "parameters": { === modified file 'lava_dispatcher/actions/lava-android-test.py' --- lava_dispatcher/actions/lava-android-test.py 2012-03-08 22:29:23 +0000 +++ lava_dispatcher/actions/lava-android-test.py 2012-03-19 06:52:06 +0000 @@ -1,6 +1,6 @@ #!/usr/bin/python -# Copyright (C) 2011 Linaro Limited +# Copyright (C) 2011-2012 Linaro Limited # # Author: Linaro Validation Team # @@ -20,10 +20,12 @@ # along with this program; if not, see . import os +import subprocess import logging from datetime import datetime from lava_dispatcher.actions import BaseAction from lava_dispatcher.client.base import OperationFailed +from lava_dispatcher.utils import generate_bundle_file_name class AndroidTestAction(BaseAction): @@ -40,6 +42,7 @@ 'type': 'object', 'properties': { 'test_name': {'type': 'string'}, + 'option': {'type': 'string', 'optional': True}, 'timeout': {'type': 'integer', 'optional': True}, }, 'additionalProperties': False, @@ -49,23 +52,80 @@ return super(cmd_lava_android_test_run, self).test_name() + \ ' (%s)' % test_name - def run(self, test_name, timeout=-1): + def run(self, test_name, option=None, timeout=-1): #Make sure in test image now self.check_lava_android_test_installed() with self.client.android_tester_session() as session: - bundle_name = test_name + "-" + datetime.now().strftime("%H%M%S") - cmd = 'lava-android-test run %s -s %s -o %s/%s.bundle' % ( - test_name, session.dev_name, self.context.host_result_dir, - bundle_name) - - logging.info("Execute command on host: %s" % cmd) - rc = os.system(cmd) + bundle_name = generate_bundle_file_name(test_name) + cmds = ["lava-android-test", 'run', test_name, + '-s', session.dev_name, + '-o', '%s/%s.bundle' % (self.context.host_result_dir, + bundle_name)] + if option is not None: + cmds.extend(['-O', option]) + logging.info("Execute command on host: %s" % (' '.join(cmds))) + rc = subprocess.call(cmds) if rc != 0: raise OperationFailed( "Failed to run test case(%s) on device(%s) with return " "value: %s" % (test_name, session.dev_name, rc)) +class cmd_lava_android_test_run_custom(AndroidTestAction): + + parameters_schema = { + 'type': 'object', + 'properties': { + 'commands': {'type': 'array', 'items': {'type': 'string'}, + 'optional': True}, + 'command_file': {'type': 'string', 'optional': True}, + 'parser': {'type': 'string', 'optional': True}, + 'timeout': {'type': 'integer', 'optional': True}, + }, + 'additionalProperties': False, + } + + def test_name(self, commands=[], command_file=None, parser=None, + timeout=-1): + if commands: + return '%s (commands=[%s])' % ( + super(cmd_lava_android_test_run_custom, self).test_name(), + ','.join(commands)) + elif command_file: + return '%s (command-file=%s)' % ( + super(cmd_lava_android_test_run_custom, self).test_name(), + command_file) + + def run(self, commands=[], command_file=None, parser=None, timeout=-1): + #Make sure in test image now + self.check_lava_android_test_installed() + if commands or command_file: + with self.client.android_tester_session() as session: + bundle_name = generate_bundle_file_name('custom') + cmds = ["lava-android-test", 'run-custom'] + if commands: + for command in commands: + cmds.extend(['-c', command]) + elif command_file: + cmds.extend(['-f', command_file]) + else: + raise OperationFailed( + "Only one of the -c and -f option can be specified" + " for lava_android_test_run_custom action") + cmds.extend(['-s', session.dev_name, '-o', + '%s/%s.bundle' % (self.context.host_result_dir, + bundle_name)]) + if parser is not None: + cmds.extend(['-p', parser]) + logging.info("Execute command on host: %s" % (' '.join(cmds))) + rc = subprocess.call(cmds) + if rc != 0: + raise OperationFailed( + "Failed to run test custom case[%s] on device(%s)" + " with return value: %s" % (' '.join(cmds), + session.dev_name, rc)) + + class cmd_lava_android_test_install(AndroidTestAction): """ lava-test deployment to test image rootfs by chroot @@ -85,12 +145,13 @@ self.check_lava_android_test_installed() with self.client.android_tester_session() as session: for test in tests: - cmd = 'lava-android-test install %s -s %s' % ( - test, session.dev_name) + cmds = ["lava-android-test", 'install', + test, + '-s', session.dev_name] if option is not None: - cmd += ' -o ' + option - logging.info("Execute command on host: %s" % cmd) - rc = os.system(cmd) + cmds.extend(['-o', option]) + logging.info("Execute command on host: %s" % (' '.join(cmds))) + rc = subprocess.call(cmds) if rc != 0: raise OperationFailed( "Failed to install test case(%s) on device(%s) with " === modified file 'lava_dispatcher/actions/lava-test.py' --- lava_dispatcher/actions/lava-test.py 2012-03-09 01:00:42 +0000 +++ lava_dispatcher/actions/lava-test.py 2012-03-19 01:56:32 +0000 @@ -1,6 +1,6 @@ #!/usr/bin/python -# Copyright (C) 2011 Linaro Limited +# Copyright (C) 2011-2012 Linaro Limited # # Author: Paul Larson # @@ -25,6 +25,7 @@ from lava_dispatcher.actions import BaseAction from lava_dispatcher.client.base import OperationFailed, CriticalError +from lava_dispatcher.utils import generate_bundle_file_name def _install_lava_test(client, session): @@ -32,7 +33,8 @@ session.run('apt-get update') #Install necessary packages for build lava-test cmd = ('apt-get -y --force-yes install ' - 'bzr usbutils python-apt python-setuptools python-simplejson lsb-release') + 'bzr usbutils python-apt python-setuptools ' + 'python-simplejson lsb-release') session.run(cmd, timeout=2400) session.run("apt-get -y --force-yes install python-pip") @@ -47,6 +49,7 @@ # cleanup the lava-test - old results, cached files... session.run('lava-test reset', timeout=60) + class cmd_lava_test_run(BaseAction): parameters_schema = { @@ -59,21 +62,21 @@ 'additionalProperties': False, } - def test_name(self, test_name, test_options = "", timeout=-1): + def test_name(self, test_name, test_options="", timeout=-1): return super(cmd_lava_test_run, self).test_name() + ' (%s)' % test_name - def run(self, test_name, test_options = "", timeout=-1): + def run(self, test_name, test_options="", timeout=-1): logging.info("Executing lava_test_run %s command" % test_name) with self.client.tester_session() as session: session.run('mkdir -p %s' % self.context.lava_result_dir) session.export_display() - bundle_name = test_name + "-" + datetime.now().strftime("%H%M%S") - + bundle_name = generate_bundle_file_name(test_name) if test_options != "": test_options = "-t '%s'" % test_options cmd = ('lava-test run %s %s -o %s/%s.bundle' % ( - test_name, test_options, self.context.lava_result_dir, bundle_name)) + test_name, test_options, self.context.lava_result_dir, + bundle_name)) try: rc = session.run(cmd, timeout=timeout) except: @@ -89,7 +92,9 @@ if rc is None: raise OperationFailed("test case getting return value failed") elif rc != 0: - raise OperationFailed("test case failed with return value: %s" % rc) + raise OperationFailed( + "test case failed with return value: %s" % rc) + class cmd_lava_test_install(BaseAction): """ @@ -111,8 +116,9 @@ 'additionalProperties': False, } - def run(self, tests, install_python = None, register = None, timeout=2400): - logging.info("Executing lava_test_install (%s) command" % ",".join(tests)) + def run(self, tests, install_python=None, register=None, timeout=2400): + logging.info( + "Executing lava_test_install (%s) command" % ",".join(tests)) with self.client.reliable_session() as session: @@ -136,7 +142,8 @@ class cmd_add_apt_repository(BaseAction): """ add apt repository to test image rootfs by chroot - arg could be 'deb uri distribution [component1] [component2][...]' or ppa: + arg could be 'deb uri distribution [component1] [component2][...]' + or ppa: """ parameters_schema = { === modified file 'lava_dispatcher/utils.py' --- lava_dispatcher/utils.py 2012-02-29 02:44:17 +0000 +++ lava_dispatcher/utils.py 2012-03-19 06:52:06 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2011 Linaro Limited +# Copyright (C) 2011-2012 Linaro Limited # # Author: Paul Larson # @@ -18,6 +18,7 @@ # along # with this program; if not, see . +import datetime import errno import logging import os @@ -33,12 +34,12 @@ urlpath = urlparse.urlsplit(url).path filename = os.path.basename(urlpath) if path: - filename = os.path.join(path,filename) + filename = os.path.join(path, filename) fd = open(filename, "w") try: response = urllib2.urlopen(urllib2.quote(url, safe=":/"), timeout=30) fd = open(filename, 'wb') - shutil.copyfileobj(response,fd,0x10000) + shutil.copyfileobj(response, fd, 0x10000) fd.close() response.close() except: @@ -47,6 +48,7 @@ raise RuntimeError("Could not retrieve %s" % url) return filename + def download_with_cache(url, path="", cachedir=""): cache_loc = url_to_cache(url, cachedir) if os.path.exists(cache_loc): @@ -60,7 +62,8 @@ if err.errno == errno.EEXIST: logging.debug("Cached copy of %s already exists" % url) else: - logging.exception("os.link '%s' with '%s' failed"%(cache_loc,file_location)) + logging.exception("os.link '%s' with '%s' failed" % (cache_loc, + file_location)) else: file_location = download(url, path) try: @@ -78,12 +81,14 @@ logging.exception("os.link failed") return file_location + def url_to_cache(url, cachedir): url_parts = urlparse.urlsplit(url) path = os.path.join(cachedir, url_parts.netloc, url_parts.path.lstrip(os.sep)) return path + def string_to_list(string): splitter = shlex(string, posix=True) splitter.whitespace = "," @@ -92,35 +97,43 @@ strip_newlines = lambda x: newlines_to_spaces(x).strip(' ') return map(strip_newlines, list(splitter)) + def logging_system(cmd): - logging.debug("Executing on host : '%r'"%cmd) + logging.debug("Executing on host : '%r'" % cmd) return os.system(cmd) class logging_spawn(pexpect.spawn): def sendline(self, *args, **kw): - logging.debug("sendline : %s" %args[0]) + logging.debug("sendline : %s" % args[0]) return super(logging_spawn, self).sendline(*args, **kw) def send(self, *args, **kw): - logging.debug("send : %s" %args[0]) + logging.debug("send : %s" % args[0]) return super(logging_spawn, self).send(*args, **kw) def expect(self, *args, **kw): # some expect should not be logged because it is so much noise. - if kw.has_key('lava_no_logging'): + if 'lava_no_logging' in kw: del kw['lava_no_logging'] return self.expect(*args, **kw) - if (kw.has_key('timeout')): + if 'timeout' in kw: timeout = kw['timeout'] else: timeout = self.timeout if len(args) == 1: - logging.debug("expect (%d): '%s'" %(timeout, args[0])) + logging.debug("expect (%d): '%s'" % (timeout, args[0])) else: - logging.debug("expect (%d): '%s'" %(timeout, str(args))) + logging.debug("expect (%d): '%s'" % (timeout, str(args))) return super(logging_spawn, self).expect(*args, **kw) + + +def generate_bundle_file_name(test_name): + return ("{test_id}.{time.tm_year:04}-{time.tm_mon:02}-{time.tm_mday:02}T" + "{time.tm_hour:02}:{time.tm_min:02}:{time.tm_sec:02}Z").format( + test_id=test_name, + time=datetime.datetime.utcnow().timetuple())