From patchwork Tue Jun 18 15:53:15 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Antonio Terceiro X-Patchwork-Id: 17964 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-ye0-f200.google.com (mail-ye0-f200.google.com [209.85.213.200]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id C2FB825C6D for ; Tue, 18 Jun 2013 15:58:29 +0000 (UTC) Received: by mail-ye0-f200.google.com with SMTP id r9sf4978195yen.7 for ; Tue, 18 Jun 2013 08:58:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-beenthere:x-forwarded-to:x-forwarded-for:delivered-to :mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state:x-original-sender :x-original-authentication-results:mailing-list:list-id :x-google-group-id:list-post:list-help:list-archive:list-unsubscribe :content-type; bh=+eL2s1HBMgpi9gbGs0QslXbIOYr5dIfEuU5ZSAIATCE=; b=dHFc8R7Orki+Ye2NySE+UXkZ+5Po4F7ba1klzOD30dQnNAHVMlbqpl6mmtchry8uRr MhENks6T9IixsU1dPzTfHp36XmxTRW0vnyZezMgILze3oiRbKbCXLOzePAKz1Re+BTSt 4vfTaseLKUoV40CXmAKRbSvXQetJhHvcKNdZDtFr+wiYwXUwYYz94CY7f7hI0ygv32Eh F8BotCjCwYwW4kud5j2/jMtLfAFjY0Xm/HHaHOhJ8HwW0hJSPIr6tsvCFC+F0zg6RVlv 6MZb9goVpIzh2SBAFYxrwERLunHqz/5qJk6mecCK71nUtSTIz+edNGem+76sV/aY3Ns/ rURQ== X-Received: by 10.236.127.108 with SMTP id c72mr11674087yhi.16.1371571109331; Tue, 18 Jun 2013 08:58:29 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.49.86.7 with SMTP id l7ls1397177qez.51.gmail; Tue, 18 Jun 2013 08:58:29 -0700 (PDT) X-Received: by 10.52.120.7 with SMTP id ky7mr1712620vdb.12.1371571109056; Tue, 18 Jun 2013 08:58:29 -0700 (PDT) Received: from mail-vc0-f174.google.com (mail-vc0-f174.google.com [209.85.220.174]) by mx.google.com with ESMTPS id og3si5474295vcb.6.2013.06.18.08.58.29 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 18 Jun 2013 08:58:29 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.174 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.220.174; Received: by mail-vc0-f174.google.com with SMTP id kw10so3012229vcb.5 for ; Tue, 18 Jun 2013 08:58:29 -0700 (PDT) X-Received: by 10.52.120.77 with SMTP id la13mr1729033vdb.23.1371571108948; Tue, 18 Jun 2013 08:58:28 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patches@linaro.org Received: by 10.58.165.8 with SMTP id yu8csp78767veb; Tue, 18 Jun 2013 08:58:26 -0700 (PDT) X-Received: by 10.14.6.198 with SMTP id 46mr2868720een.121.1371570796212; Tue, 18 Jun 2013 08:53:16 -0700 (PDT) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id j47si16353183eeo.110.2013.06.18.08.53.15 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 18 Jun 2013 08:53:16 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1UoyDb-0007Zv-6b for ; Tue, 18 Jun 2013 15:53:15 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id 0C1DAEA9C1 for ; Tue, 18 Jun 2013 15:53:15 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-tool X-Launchpad-Branch: ~linaro-validation/lava-tool/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 186 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-tool/trunk] Rev 186: Base for test suite helper functionality Message-Id: <20130618155315.25397.8977.launchpad@ackee.canonical.com> Date: Tue, 18 Jun 2013 15:53:15 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: list X-Generated-By: Launchpad (canonical.com); Revision="16667"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: c49989e3693efeca59684fe45f47db1a1fb7b261 X-Gm-Message-State: ALoCoQkTbv5BG8rDRbiXGITtSN6G7gr4TdcC0DU7xOkSKsjVVsC8mASWZvB8G/vvj3m/4Q3+VQD7 X-Original-Sender: noreply@launchpad.net X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.220.174 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org List-ID: X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , Merge authors: Antonio Terceiro (terceiro) Related merge proposals: https://code.launchpad.net/~terceiro/lava-tool/bash-completion/+merge/164495 proposed by: Antonio Terceiro (terceiro) review: Approve - Senthil Kumaran S (stylesen) https://code.launchpad.net/~terceiro/lava-tool/split-entry-points/+merge/164493 proposed by: Antonio Terceiro (terceiro) ------------------------------------------------------------ revno: 186 [merge] committer: Antonio Terceiro branch nick: trunk timestamp: Tue 2013-06-18 12:51:43 -0300 message: Base for test suite helper functionality added: ci-build entry_points.ini integration-tests integration-tests.d/ integration-tests.d/lava-job-new-existing.sh integration-tests.d/lava-job-new-with-config.sh integration-tests.d/lava-job-submit.sh integration-tests.d/lib/ integration-tests.d/lib/fixed_response integration-tests.d/lib/lava_config integration-tests.d/lib/server.py integration-tests.d/sample/ integration-tests.d/sample/nexus.json lava/config.py lava/job/ lava/job/__init__.py lava/job/commands.py lava/job/templates.py lava/job/tests/ lava/job/tests/__init__.py lava/job/tests/test_commands.py lava/job/tests/test_job.py modified: .bzrignore README lava/tool/dispatcher.py lava_dashboard_tool/tests/__init__.py lava_tool/authtoken.py lava_tool/tests/__init__.py lava_tool/tests/test_authtoken.py lava_tool/tests/test_commands.py setup.py --- lp:lava-tool https://code.launchpad.net/~linaro-validation/lava-tool/trunk You are subscribed to branch lp:lava-tool. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-tool/trunk/+edit-subscription === modified file '.bzrignore' --- .bzrignore 2011-05-04 02:16:55 +0000 +++ .bzrignore 2013-05-24 17:37:52 +0000 @@ -2,3 +2,4 @@ *.egg-info ./build ./dist +/tags === modified file 'README' --- README 2011-06-17 13:36:44 +0000 +++ README 2013-06-03 18:06:49 +0000 @@ -14,6 +14,39 @@ See INSTALL +Usage +===== + +Dealing with jobs + + $ lava job new file.json # creates file.json from a template + $ lava job submit file.json # submits file.json to a remote LAVA server + $ lava job run file.json # runs file.json on a local LAVA device + +Dealing with LAVA Test Shell Test Definitions + + $ lava testdef new file.yml # creates file.yml from a template + $ lava testdef submit file.yml # submits file.yml to a remote LAVA server + $ lava testdef run file.yml # runs file.yml on a local LAVA device + +Dealing with LAVA Test Shell Scripts + + $ lava script submit SCRIPT # submits SCRIPT to a remote LAVA server + $ lava script run SCRIPT # runs SCRIPT on a local LAVA device + +Bash completion +=============== + +Once lava-tool is installed, you can turn bash completion on for the `lava` and +`lava-tool` programs with the following commands (which you can also paste in +your ~/.bashrc): + + eval "$(register-python-argcomplete lava)" + eval "$(register-python-argcomplete lava-tool)" + +Then if you type for example "lava-tool su", it will complete that "su" +with "submit-job" for you. + Reporting Bugs ============== === added file 'ci-build' --- ci-build 1970-01-01 00:00:00 +0000 +++ ci-build 2013-06-03 20:56:10 +0000 @@ -0,0 +1,42 @@ +#!/bin/sh + +set -e + +if test -z "$VIRTUAL_ENV"; then + set -x + virtualenv ci-build-venv + . ci-build-venv/bin/activate + python setup.py develop +fi + +# requirement for integration tests +if ! pip show Flask | grep -q Flask; then + pip install 'Flask==0.9' +fi +if ! pip show PyYAML | grep -q PyYAML; then + pip install PyYAML +fi +# requirement for unit tests +if ! pip show mocker | grep -q mocker; then + pip install mocker +fi + +export LAVACONFIG=/dev/null + +if test -z "$DISPLAY"; then + # actual CI + + # will install tests dependencies automatically. The output is also more + # verbose + python setup.py test < /dev/null + + # integration-tests will pick this up and provide detailed output + export VERBOSE=1 +else + # in a development workstation, this will produce shorter/nicer output, but + # requires the test dependencies to be installed manually (or by running + # `python setup.py test` before). + python -m unittest lava_tool.tests.test_suite < /dev/null +fi + +./integration-tests === added file 'entry_points.ini' --- entry_points.ini 1970-01-01 00:00:00 +0000 +++ entry_points.ini 2013-05-27 20:51:39 +0000 @@ -0,0 +1,72 @@ +[console_scripts] +lava-tool = lava_tool.dispatcher:main +lava = lava.tool.main:LavaDispatcher.run +lava-dashboard-tool=lava_dashboard_tool.main:main + +[lava.commands] +help = lava.tool.commands.help:help +scheduler = lava_scheduler_tool.commands:scheduler +dashboard = lava_dashboard_tool.commands:dashboard +job = lava.job.commands:job + +[lava_tool.commands] +help = lava.tool.commands.help:help +auth-add = lava_tool.commands.auth:auth_add +submit-job = lava_scheduler_tool.commands:submit_job +resubmit-job = lava_scheduler_tool.commands:resubmit_job +cancel-job = lava_scheduler_tool.commands:cancel_job +job-output = lava_scheduler_tool.commands:job_output +backup=lava_dashboard_tool.commands:backup +bundles=lava_dashboard_tool.commands:bundles +data_views=lava_dashboard_tool.commands:data_views +deserialize=lava_dashboard_tool.commands:deserialize +get=lava_dashboard_tool.commands:get +make_stream=lava_dashboard_tool.commands:make_stream +pull=lava_dashboard_tool.commands:pull +put=lava_dashboard_tool.commands:put +query_data_view=lava_dashboard_tool.commands:query_data_view +restore=lava_dashboard_tool.commands:restore +server_version=lava_dashboard_tool.commands:server_version +streams=lava_dashboard_tool.commands:streams +version=lava_dashboard_tool.commands:version + +[lava.scheduler.commands] +submit-job = lava_scheduler_tool.commands:submit_job +resubmit-job = lava_scheduler_tool.commands:resubmit_job +cancel-job = lava_scheduler_tool.commands:cancel_job +job-output = lava_scheduler_tool.commands:job_output + +[lava.dashboard.commands] +backup=lava_dashboard_tool.commands:backup +bundles=lava_dashboard_tool.commands:bundles +data_views=lava_dashboard_tool.commands:data_views +deserialize=lava_dashboard_tool.commands:deserialize +get=lava_dashboard_tool.commands:get +make_stream=lava_dashboard_tool.commands:make_stream +pull=lava_dashboard_tool.commands:pull +put=lava_dashboard_tool.commands:put +query_data_view=lava_dashboard_tool.commands:query_data_view +restore=lava_dashboard_tool.commands:restore +server_version=lava_dashboard_tool.commands:server_version +streams=lava_dashboard_tool.commands:streams +version=lava_dashboard_tool.commands:version + +[lava_dashboard_tool.commands] +backup=lava_dashboard_tool.commands:backup +bundles=lava_dashboard_tool.commands:bundles +data_views=lava_dashboard_tool.commands:data_views +deserialize=lava_dashboard_tool.commands:deserialize +get=lava_dashboard_tool.commands:get +make_stream=lava_dashboard_tool.commands:make_stream +pull=lava_dashboard_tool.commands:pull +put=lava_dashboard_tool.commands:put +query_data_view=lava_dashboard_tool.commands:query_data_view +restore=lava_dashboard_tool.commands:restore +server_version=lava_dashboard_tool.commands:server_version +streams=lava_dashboard_tool.commands:streams +version=lava_dashboard_tool.commands:version + +[lava.job.commands] +new = lava.job.commands:new +submit = lava.job.commands:submit +run = lava.job.commands:run === added file 'integration-tests' --- integration-tests 1970-01-01 00:00:00 +0000 +++ integration-tests 2013-06-03 19:54:23 +0000 @@ -0,0 +1,80 @@ +#!/bin/sh + +set -e + +green() { + test -t 1 && printf "\033[0;32;40m$@\033[m\n" || echo "$@" +} + +red() { + test -t 2 && printf "\033[0;31;40m$@\033[m\n" >&2 || echo "$2" >&2 +} + +start_server() { + server_dir="${base_tmpdir}/_server" + mkdir -p "${server_dir}" + server_log="${server_dir}/log" + python integration-tests.d/lib/server.py > "${server_log}" 2>&1 & + server_pid=$? +} + +stop_server() { + curl -q http://localhost:5000/exit +} + +run_test() { + local testfile="$1" + local logfile="$2" + rc=0 + if test -n "$VERBOSE"; then + sh -x "$testfile" < /dev/null || rc=$? + else + sh -x "$testfile" > "${logfile}" 2>&1 < /dev/null || rc=$? + fi + if test $rc -eq 0; then + green "$testname: PASS" + passed=$(($passed + 1)) + else + failed=$(($failed + 1)) + red "$testname: FAIL" + if test -f "$logfile"; then + cat "$logfile" + fi + fi +} + +passed=0 +failed=0 +base_tmpdir=$(mktemp -d) +logs="${base_tmpdir}/logs" +mkdir "$logs" + +export PATH="$(dirname $0)"/integration-tests.d/lib:$PATH + +start_server + +tests="$@" +if test -z "$tests"; then + tests=$(echo integration-tests.d/*.sh) +fi + +for testfile in $tests; do + testname=$(basename "$testfile") + logfile="${logs}/${testname}.log" + export tmpdir="${base_tmpdir}/${testname}" + export LAVACONFIG="${tmpdir}/config" + mkdir "${tmpdir}" + run_test "$testfile" "$logfile" +done + +stop_server + +rm -rf "${base_tmpdir}" + +echo +if [ "$failed" -eq 0 ]; then + green "$passed tests passed, $failed tests failed." +else + red "$passed tests passed, $failed tests failed." + exit 1 +fi === added directory 'integration-tests.d' === added file 'integration-tests.d/lava-job-new-existing.sh' --- integration-tests.d/lava-job-new-existing.sh 1970-01-01 00:00:00 +0000 +++ integration-tests.d/lava-job-new-existing.sh 2013-05-27 21:24:06 +0000 @@ -0,0 +1,5 @@ +touch "${tmpdir}/foo.json" +lava job new "${tmpdir}/foo.json" +rc="$?" +test "$rc" -gt 0 + === added file 'integration-tests.d/lava-job-new-with-config.sh' --- integration-tests.d/lava-job-new-with-config.sh 1970-01-01 00:00:00 +0000 +++ integration-tests.d/lava-job-new-with-config.sh 2013-05-28 22:08:12 +0000 @@ -0,0 +1,10 @@ +cat > "${tmpdir}/config" < $tmpdir/output +grep "Job submitted with job ID 999" $tmpdir/output === added directory 'integration-tests.d/lib' === added file 'integration-tests.d/lib/fixed_response' --- integration-tests.d/lib/fixed_response 1970-01-01 00:00:00 +0000 +++ integration-tests.d/lib/fixed_response 2013-06-03 18:06:49 +0000 @@ -0,0 +1,11 @@ +#!/usr/bin/env python + +import os +import sys +import xmlrpclib + +data = eval(sys.argv[1]) + +output = os.path.join(os.path.dirname(__file__), 'fixed_response.txt') +with open(output, 'w') as f: + f.write(xmlrpclib.dumps((data,), methodresponse=True)) === added file 'integration-tests.d/lib/lava_config' --- integration-tests.d/lib/lava_config 1970-01-01 00:00:00 +0000 +++ integration-tests.d/lib/lava_config 2013-06-03 18:06:49 +0000 @@ -0,0 +1,5 @@ +#!/bin/sh + +set -e + +cat > "$LAVACONFIG" === added file 'integration-tests.d/lib/server.py' --- integration-tests.d/lib/server.py 1970-01-01 00:00:00 +0000 +++ integration-tests.d/lib/server.py 2013-06-03 18:06:49 +0000 @@ -0,0 +1,59 @@ +import os +import sys +import yaml + +from flask import ( + Flask, + request, +) + +app = Flask(__name__) + +aliases = { + 'ok': 200, + 'forbidden': 403, + 'notfound': 404, +} + +fixed_response = None + +@app.route('/exit') +def exit(): + # http://werkzeug.pocoo.org/docs/serving/#shutting-down-the-server + if not 'werkzeug.server.shutdown' in request.environ: + raise RuntimeError('Not running the development server') + request.environ['werkzeug.server.shutdown']() + return "" + +@app.route('/', methods=['GET', 'POST']) +def reply(status_code): + + status = int(aliases.get(status_code, status_code)) + + headers = {} + for k,v in request.headers: + headers[k] = v + + response_file = os.path.join(os.path.dirname(__file__), 'fixed_response.txt') + if os.path.exists(response_file): + response = open(response_file).read() + os.remove(response_file) + else: + response = yaml.dump( + { + 'status': status, + 'headers': headers, + 'body': request.form.keys(), + 'query': dict(request.args), + }, + encoding='utf-8', + default_flow_style=False, + ) + return response, status, { 'Content-Type': 'text/plain' } + +@app.route('/', methods=['GET','POST']) +def root(): + return reply(200) + +if __name__ == '__main__': + app.run(debug=('DEBUG' in os.environ)) === added directory 'integration-tests.d/sample' === added file 'integration-tests.d/sample/nexus.json' --- integration-tests.d/sample/nexus.json 1970-01-01 00:00:00 +0000 +++ integration-tests.d/sample/nexus.json 2013-06-03 18:06:49 +0000 @@ -0,0 +1,15 @@ +{ + "device_type": "nexus", + "job_name": "Boot test", + "actions": [ + { + "command": "deploy_linaro_image", + "parameters": { + "image": "http:///url/to/nexus.img" + } + }, + { + "command": "boot_linaro_image" + } + ] +} \ No newline at end of file === added file 'lava/config.py' --- lava/config.py 1970-01-01 00:00:00 +0000 +++ lava/config.py 2013-05-28 22:08:12 +0000 @@ -0,0 +1,95 @@ +# Copyright (C) 2013 Linaro Limited +# +# Author: Antonio Terceiro +# +# This file is part of lava-tool. +# +# lava-tool is free software: you can redistribute it and/or modify +# it under the terms of the GNU Lesser General Public License version 3 +# as published by the Free Software Foundation +# +# lava-tool is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with lava-tool. If not, see . + +import atexit +from ConfigParser import ConfigParser, NoOptionError, NoSectionError +import os +import readline + +__all__ = ['InteractiveConfig', 'NonInteractiveConfig'] + +history = os.path.join(os.path.expanduser("~"), ".lava_history") +try: + readline.read_history_file(history) +except IOError: + pass +atexit.register(readline.write_history_file, history) + +config_file = os.environ.get('LAVACONFIG') or os.path.join(os.path.expanduser('~'), '.lavaconfig') +config_backend = ConfigParser() +config_backend.read([config_file]) +def save_config(): + with open(config_file, 'w') as f: + config_backend.write(f) +atexit.register(save_config) + +class InteractiveConfig(object): + + def __init__(self, force_interactive=False): + self._force_interactive = force_interactive + self._cache = {} + + def get(self, parameter): + key = parameter.id + value = None + if parameter.depends: + pass + config_section = parameter.depends.id + '=' + self.get(parameter.depends) + else: + config_section = "DEFAULT" + + if config_section in self._cache: + if key in self._cache[config_section]: + return self._cache[config_section][key] + + prompt = '%s: ' % key + + try: + value = config_backend.get(config_section, key) + except (NoOptionError, NoSectionError): + pass + if value: + if self._force_interactive: + prompt = "%s[%s]: " % (key, value) + else: + return value + try: + user_input = raw_input(prompt).strip() + except EOFError: + user_input = None + if user_input: + value = user_input + if not config_backend.has_section(config_section) and config_section != 'DEFAULT': + config_backend.add_section(config_section) + config_backend.set(config_section, key, value) + + if value: + if config_section not in self._cache: + self._cache[config_section] = {} + self._cache[config_section][key] = value + return value + else: + raise KeyError(key) + +class NonInteractiveConfig(object): + + def __init__(self, data): + self.data = data + + def get(self, parameter): + return self.data[parameter.id] === added directory 'lava/job' === added file 'lava/job/__init__.py' --- lava/job/__init__.py 1970-01-01 00:00:00 +0000 +++ lava/job/__init__.py 2013-05-28 22:08:12 +0000 @@ -0,0 +1,47 @@ +# Copyright (C) 2013 Linaro Limited +# +# Author: Antonio Terceiro +# +# This file is part of lava-tool. +# +# lava-tool is free software: you can redistribute it and/or modify +# it under the terms of the GNU Lesser General Public License version 3 +# as published by the Free Software Foundation +# +# lava-tool is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with lava-tool. If not, see . + +from copy import deepcopy +import json + +from lava.job.templates import Parameter + +class Job: + + def __init__(self, template): + self.data = deepcopy(template) + + def fill_in(self, config): + def insert_data(data): + if isinstance(data, dict): + keys = data.keys() + elif isinstance(data, list): + keys = range(len(data)) + else: + return + for key in keys: + entry = data[key] + if isinstance(entry, Parameter): + data[key] = config.get(entry) + else: + insert_data(entry) + insert_data(self.data) + + def write(self, stream): + stream.write(json.dumps(self.data, indent=4)) + === added file 'lava/job/commands.py' --- lava/job/commands.py 1970-01-01 00:00:00 +0000 +++ lava/job/commands.py 2013-06-03 18:06:49 +0000 @@ -0,0 +1,94 @@ +# Copyright (C) 2013 Linaro Limited +# +# Author: Antonio Terceiro +# +# This file is part of lava-tool. +# +# lava-tool is free software: you can redistribute it and/or modify +# it under the terms of the GNU Lesser General Public License version 3 +# as published by the Free Software Foundation +# +# lava-tool is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with lava-tool. If not, see . + +from os.path import exists + +from lava.config import InteractiveConfig +from lava.job import Job +from lava.job.templates import * +from lava.tool.command import Command, CommandGroup +from lava.tool.errors import CommandError + +from lava_tool.authtoken import AuthenticatingServerProxy, KeyringAuthBackend +import xmlrpclib + +class job(CommandGroup): + """ + LAVA job file handling + """ + + namespace = 'lava.job.commands' + +class BaseCommand(Command): + + def __init__(self, parser, args): + super(BaseCommand, self).__init__(parser, args) + self.config = InteractiveConfig(force_interactive=self.args.interactive) + + @classmethod + def register_arguments(cls, parser): + super(BaseCommand, cls).register_arguments(parser) + parser.add_argument( + "-i", "--interactive", + action='store_true', + help=("Forces asking for input parameters even if we already " + "have them cached.")) + +class new(BaseCommand): + + @classmethod + def register_arguments(cls, parser): + super(new, cls).register_arguments(parser) + parser.add_argument("FILE", help=("Job file to be created.")) + + def invoke(self): + if exists(self.args.FILE): + raise CommandError('%s already exists' % self.args.FILE) + + with open(self.args.FILE, 'w') as f: + job = Job(BOOT_TEST) + job.fill_in(self.config) + job.write(f) + + +class submit(BaseCommand): + @classmethod + def register_arguments(cls, parser): + super(submit, cls).register_arguments(parser) + parser.add_argument("FILE", help=("The job file to submit")) + + def invoke(self): + jobfile = self.args.FILE + jobdata = open(jobfile, 'rb').read() + + server_name = Parameter('server') + rpc_endpoint = Parameter('rpc_endpoint', depends=server_name) + self.config.get(server_name) + endpoint = self.config.get(rpc_endpoint) + + server = AuthenticatingServerProxy(endpoint, + auth_backend=KeyringAuthBackend()) + try: + job_id = server.scheduler.submit_job(jobdata) + print "Job submitted with job ID %d" % job_id + except xmlrpclib.Fault, e: + raise CommandError(str(e)) + +class run(BaseCommand): + def invoke(self): + print("hello world") === added file 'lava/job/templates.py' --- lava/job/templates.py 1970-01-01 00:00:00 +0000 +++ lava/job/templates.py 2013-05-28 22:08:12 +0000 @@ -0,0 +1,63 @@ +# Copyright (C) 2013 Linaro Limited +# +# Author: Antonio Terceiro +# +# This file is part of lava-tool. +# +# lava-tool is free software: you can redistribute it and/or modify +# it under the terms of the GNU Lesser General Public License version 3 +# as published by the Free Software Foundation +# +# lava-tool is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with lava-tool. If not, see . + +class Parameter(object): + + def __init__(self, id, depends=None): + self.id = id + self.depends = depends + +device_type = Parameter("device_type") +prebuilt_image = Parameter("prebuilt_image", depends=device_type) + +BOOT_TEST = { + "job_name": "Boot test", + "device_type": device_type, + "actions": [ + { + "command": "deploy_linaro_image", + "parameters": { + "image": prebuilt_image + } + }, + { + "command": "boot_linaro_image" + } + ] +} + +LAVA_TEST_SHELL = { + "job_name": "LAVA Test Shell", + "device_type": device_type, + "actions": [ + { + "command": "deploy_linaro_image", + "parameters": { + "image": prebuilt_image, + } + }, + { + "command": "lava_test_shell", + "parameters": { + "testdef_urls": [ + Parameter("testdef_url") + ] + } + } + ] +} === added directory 'lava/job/tests' === added file 'lava/job/tests/__init__.py' === added file 'lava/job/tests/test_commands.py' --- lava/job/tests/test_commands.py 1970-01-01 00:00:00 +0000 +++ lava/job/tests/test_commands.py 2013-06-03 18:06:49 +0000 @@ -0,0 +1,93 @@ +# Copyright (C) 2013 Linaro Limited +# +# Author: Antonio Terceiro +# +# This file is part of lava-tool. +# +# lava-tool is free software: you can redistribute it and/or modify +# it under the terms of the GNU Lesser General Public License version 3 +# as published by the Free Software Foundation +# +# lava-tool is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with lava-tool. If not, see . + +""" +Unit tests for the commands classes +""" + +from argparse import ArgumentParser +import json +from os import ( + makedirs, + removedirs, +) +from os.path import( + exists, + join, +) +from shutil import( + rmtree, +) +from tempfile import mkdtemp +from unittest import TestCase + +from lava.config import NonInteractiveConfig +from lava.job.commands import * +from lava.tool.errors import CommandError + +from mocker import Mocker + +def make_command(command, *args): + parser = ArgumentParser(description="fake argument parser") + command.register_arguments(parser) + the_args = parser.parse_args(*args) + cmd = command(parser, the_args) + cmd.config = NonInteractiveConfig({ 'device_type': 'foo', 'prebuilt_image': 'bar' }) + return cmd + +class CommandTest(TestCase): + + def setUp(self): + self.tmpdir = mkdtemp() + + def tearDown(self): + rmtree(self.tmpdir) + + def tmp(self, filename): + return join(self.tmpdir, filename) + +class JobNewTest(CommandTest): + + def test_create_new_file(self): + f = self.tmp('file.json') + command = make_command(new, [f]) + command.invoke() + self.assertTrue(exists(f)) + + def test_fills_in_template_parameters(self): + f = self.tmp('myjob.json') + command = make_command(new, [f]) + command.invoke() + + data = json.loads(open(f).read()) + self.assertEqual(data['device_type'], 'foo') + + def test_wont_overwriteexisting_file(self): + existing = self.tmp('existing.json') + with open(existing, 'w') as f: + f.write("CONTENTS") + command = make_command(new, [existing]) + with self.assertRaises(CommandError): + command.invoke() + self.assertEqual("CONTENTS", open(existing).read()) + +class JobSubmitTest(CommandTest): + + def test_receives_job_file_in_cmdline(self): + cmd = make_command(new, ['FOO.json']) + self.assertEqual('FOO.json', cmd.args.FILE) === added file 'lava/job/tests/test_job.py' --- lava/job/tests/test_job.py 1970-01-01 00:00:00 +0000 +++ lava/job/tests/test_job.py 2013-05-28 22:08:12 +0000 @@ -0,0 +1,68 @@ +# Copyright (C) 2013 Linaro Limited +# +# Author: Antonio Terceiro +# +# This file is part of lava-tool. +# +# lava-tool is free software: you can redistribute it and/or modify +# it under the terms of the GNU Lesser General Public License version 3 +# as published by the Free Software Foundation +# +# lava-tool is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with lava-tool. If not, see . + +""" +Unit tests for the Job class +""" + +import json +from unittest import TestCase +from StringIO import StringIO + +from lava.config import NonInteractiveConfig +from lava.job.templates import * +from lava.job import Job + +class JobTest(TestCase): + + def test_from_template(self): + template = {} + job = Job(template) + self.assertEqual(job.data, template) + self.assertIsNot(job.data, template) + + def test_fill_in_data(self): + job = Job(BOOT_TEST) + image = "/path/to/panda.img" + config = NonInteractiveConfig( + { + "device_type": "panda", + "prebuilt_image": image, + } + ) + job.fill_in(config) + + self.assertEqual(job.data['device_type'], "panda") + self.assertEqual(job.data['actions'][0]["parameters"]["image"], image) + + def test_write(self): + orig_data = { "foo": "bar" } + job = Job(orig_data) + output = StringIO() + job.write(output) + + data = json.loads(output.getvalue()) + self.assertEqual(data, orig_data) + + def test_writes_nicely_formatted_json(self): + orig_data = { "foo": "bar" } + job = Job(orig_data) + output = StringIO() + job.write(output) + + self.assertTrue(output.getvalue().startswith("{\n")) === modified file 'lava/tool/dispatcher.py' --- lava/tool/dispatcher.py 2012-03-22 18:03:03 +0000 +++ lava/tool/dispatcher.py 2013-05-17 19:21:51 +0000 @@ -21,6 +21,7 @@ """ import argparse +import argcomplete import logging import pkg_resources import sys @@ -121,6 +122,8 @@ If arguments are left out they are looked up in sys.argv automatically """ + # Before anything, hook into the bash completion + argcomplete.autocomplete(self.parser) # First parse whatever input arguments we've got args = self.parser.parse_args(raw_args) # Adjust logging level after seeing arguments === modified file 'lava_dashboard_tool/tests/__init__.py' --- lava_dashboard_tool/tests/__init__.py 2013-04-22 17:33:58 +0000 +++ lava_dashboard_tool/tests/__init__.py 2013-05-21 15:48:22 +0000 @@ -1,52 +0,0 @@ -# Copyright (C) 2010,2011 Linaro Limited -# -# Author: Zygmunt Krynicki -# -# This file is part of lava-dashboard-tool. -# -# lava-dashboard-tool is free software: you can redistribute it and/or modify -# it under the terms of the GNU Lesser General Public License version 3 -# as published by the Free Software Foundation -# -# lava-dashboard-tool is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public License -# along with lava-dashboard-tool. If not, see . - -""" -Package with unit tests for lava_dashboard_tool -""" - -import doctest -import unittest - - -def app_modules(): - return [ - 'lava_dashboard_tool.commands', - ] - - -def test_modules(): - return [ - 'lava_dashboard_tool.tests.test_commands', - ] - - -def test_suite(): - """ - Build an unittest.TestSuite() object with all the tests in _modules. - Each module is harvested for both regular unittests and doctests - """ - modules = app_modules() + test_modules() - suite = unittest.TestSuite() - loader = unittest.TestLoader() - for name in modules: - unit_suite = loader.loadTestsFromName(name) - suite.addTests(unit_suite) - doc_suite = doctest.DocTestSuite(name) - suite.addTests(doc_suite) - return suite === modified file 'lava_tool/authtoken.py' --- lava_tool/authtoken.py 2012-10-23 22:02:14 +0000 +++ lava_tool/authtoken.py 2013-05-22 13:45:44 +0000 @@ -72,6 +72,15 @@ def request(self, host, handler, request_body, verbose=0): self.verbose = verbose + request = self.build_http_request(host, handler, request_body) + try: + response = self._opener.open(request) + except urllib2.HTTPError as e: + raise xmlrpclib.ProtocolError( + host + handler, e.code, e.msg, e.info()) + return self.parse_response(response) + + def build_http_request(self, host, handler, request_body): token = None user = None auth, host = urllib.splituser(host) @@ -88,12 +97,8 @@ if token: auth = base64.b64encode(urllib.unquote(user + ':' + token)) request.add_header("Authorization", "Basic " + auth) - try: - response = self._opener.open(request) - except urllib2.HTTPError as e: - raise xmlrpclib.ProtocolError( - host + handler, e.code, e.msg, e.info()) - return self.parse_response(response) + + return request class AuthenticatingServerProxy(xmlrpclib.ServerProxy): === modified file 'lava_tool/tests/__init__.py' --- lava_tool/tests/__init__.py 2011-06-08 01:47:44 +0000 +++ lava_tool/tests/__init__.py 2013-05-27 20:51:39 +0000 @@ -27,9 +27,9 @@ def app_modules(): return [ 'lava_tool.commands', - 'lava_tool.commands.misc', 'lava_tool.dispatcher', 'lava_tool.interface', + 'lava_dashboard_tool.commands', ] @@ -38,6 +38,9 @@ 'lava_tool.tests.test_authtoken', 'lava_tool.tests.test_auth_commands', 'lava_tool.tests.test_commands', + 'lava_dashboard_tool.tests.test_commands', + 'lava.job.tests.test_job', + 'lava.job.tests.test_commands', ] === modified file 'lava_tool/tests/test_authtoken.py' --- lava_tool/tests/test_authtoken.py 2011-07-10 23:43:30 +0000 +++ lava_tool/tests/test_authtoken.py 2013-05-22 13:45:44 +0000 @@ -31,74 +31,39 @@ from lava_tool.authtoken import ( AuthenticatingServerProxy, + XMLRPCTransport, MemoryAuthBackend, ) from lava_tool.interface import LavaCommandError -if sys.version_info[:2] <= (2, 6): - TWO_SIX = True -else: - TWO_SIX = False - class TestAuthenticatingServerProxy(TestCase): def auth_headers_for_method_call_on(self, url, auth_backend): parsed = urlparse.urlparse(url) - expected_host = parsed.hostname - if parsed.port: - expected_host += ':' + str(parsed.port) - server_proxy = AuthenticatingServerProxy( - url, auth_backend=auth_backend) + mocker = Mocker() - if url.startswith('https'): - cls_name = 'httplib.HTTPS' - expected_constructor_args = (expected_host, ARGS) - else: - cls_name = 'httplib.HTTP' - expected_constructor_args = (expected_host, ARGS) - if not TWO_SIX: - cls_name += 'Connection' - mocked_HTTPConnection = mocker.replace(cls_name, passthrough=False) - mocked_connection = mocked_HTTPConnection(*expected_constructor_args) - # nospec() is required because of - # https://bugs.launchpad.net/mocker/+bug/794351 - mocker.nospec() + transport = mocker.mock() + auth_data = [] - mocked_connection.putrequest(ARGS, KWARGS) - if TWO_SIX: - mocked_connection.send(ARGS, KWARGS) - - def match_header(header, *values): - if header.lower() == 'authorization': - if len(values) != 1: - self.fail( - 'more than one value for ' - 'putheader("Authorization", ...)') - auth_data.append(values[0]) - mocked_connection.putheader(ARGS) - mocker.call(match_header) - mocker.count(1, None) - - mocked_connection.endheaders(ARGS, KWARGS) - - if TWO_SIX: - mocked_connection.getreply(ARGS, KWARGS) - mocker.result((200, None, None)) - s = StringIO.StringIO(xmlrpclib.dumps((1,), methodresponse=True)) - mocked_connection.getfile() - mocker.result(s) - mocked_connection._conn - mocker.result(None) - else: - mocked_connection.getresponse(ARGS, KWARGS) - s = StringIO.StringIO(xmlrpclib.dumps((1,), methodresponse=True)) - s.status = 200 - mocker.result(s) - - mocked_connection.close() - mocker.count(0, 1) + + def intercept_request(host, handler, request_body, verbose=0): + actual_transport = XMLRPCTransport(parsed.scheme, auth_backend) + request = actual_transport.build_http_request(host, handler, request_body) + if (request.has_header('Authorization')): + auth_data.append(request.get_header('Authorization')) + + response_body = xmlrpclib.dumps((1,), methodresponse=True) + response = StringIO.StringIO(response_body) + response.status = 200 + response.__len__ = lambda: len(response_body) + + transport.request(ARGS, KWARGS) + mocker.call(intercept_request) + mocker.result(response) with mocker: + server_proxy = AuthenticatingServerProxy( + url, auth_backend=auth_backend, transport=transport) server_proxy.method() return auth_data === modified file 'lava_tool/tests/test_commands.py' --- lava_tool/tests/test_commands.py 2011-06-17 13:33:45 +0000 +++ lava_tool/tests/test_commands.py 2013-05-22 13:45:13 +0000 @@ -20,7 +20,7 @@ Unit tests for the launch_control.commands package """ -from mocker import MockerTestCase +from mocker import MockerTestCase, ARGS from lava_tool.interface import ( Command, @@ -101,10 +101,10 @@ class DispatcherTestCase(MockerTestCase): def test_main(self): - mock_LavaDispatcher = self.mocker.replace( - 'lava_tool.dispatcher.LavaDispatcher') - mock_LavaDispatcher().dispatch() + dispatcher = self.mocker.patch(LavaDispatcher) + dispatcher.dispatch(ARGS) self.mocker.replay() + self.assertRaises(SystemExit, main) def test_add_command_cls(self): === modified file 'setup.py' --- setup.py 2013-05-02 09:22:04 +0000 +++ setup.py 2013-05-17 19:21:51 +0000 @@ -19,7 +19,9 @@ # along with lava-tool. If not, see . from setuptools import setup, find_packages +from os.path import dirname, join +entry_points = open(join(dirname(__file__), 'entry_points.ini')).read() setup( name='lava-tool', @@ -32,69 +34,7 @@ url='https://launchpad.net/lava-tool', test_suite='lava_tool.tests.test_suite', license="LGPLv3", - entry_points=""" - [console_scripts] - lava-tool = lava_tool.dispatcher:main - lava = lava.tool.main:LavaDispatcher.run - lava-dashboard-tool=lava_dashboard_tool.main:main - [lava.commands] - help = lava.tool.commands.help:help - scheduler = lava_scheduler_tool.commands:scheduler - dashboard = lava_dashboard_tool.commands:dashboard - [lava_tool.commands] - help = lava.tool.commands.help:help - auth-add = lava_tool.commands.auth:auth_add - submit-job = lava_scheduler_tool.commands:submit_job - resubmit-job = lava_scheduler_tool.commands:resubmit_job - cancel-job = lava_scheduler_tool.commands:cancel_job - job-output = lava_scheduler_tool.commands:job_output - backup=lava_dashboard_tool.commands:backup - bundles=lava_dashboard_tool.commands:bundles - data_views=lava_dashboard_tool.commands:data_views - deserialize=lava_dashboard_tool.commands:deserialize - get=lava_dashboard_tool.commands:get - make_stream=lava_dashboard_tool.commands:make_stream - pull=lava_dashboard_tool.commands:pull - put=lava_dashboard_tool.commands:put - query_data_view=lava_dashboard_tool.commands:query_data_view - restore=lava_dashboard_tool.commands:restore - server_version=lava_dashboard_tool.commands:server_version - streams=lava_dashboard_tool.commands:streams - version=lava_dashboard_tool.commands:version - [lava.scheduler.commands] - submit-job = lava_scheduler_tool.commands:submit_job - resubmit-job = lava_scheduler_tool.commands:resubmit_job - cancel-job = lava_scheduler_tool.commands:cancel_job - job-output = lava_scheduler_tool.commands:job_output - [lava.dashboard.commands] - backup=lava_dashboard_tool.commands:backup - bundles=lava_dashboard_tool.commands:bundles - data_views=lava_dashboard_tool.commands:data_views - deserialize=lava_dashboard_tool.commands:deserialize - get=lava_dashboard_tool.commands:get - make_stream=lava_dashboard_tool.commands:make_stream - pull=lava_dashboard_tool.commands:pull - put=lava_dashboard_tool.commands:put - query_data_view=lava_dashboard_tool.commands:query_data_view - restore=lava_dashboard_tool.commands:restore - server_version=lava_dashboard_tool.commands:server_version - streams=lava_dashboard_tool.commands:streams - version=lava_dashboard_tool.commands:version - [lava_dashboard_tool.commands] - backup=lava_dashboard_tool.commands:backup - bundles=lava_dashboard_tool.commands:bundles - data_views=lava_dashboard_tool.commands:data_views - deserialize=lava_dashboard_tool.commands:deserialize - get=lava_dashboard_tool.commands:get - make_stream=lava_dashboard_tool.commands:make_stream - pull=lava_dashboard_tool.commands:pull - put=lava_dashboard_tool.commands:put - query_data_view=lava_dashboard_tool.commands:query_data_view - restore=lava_dashboard_tool.commands:restore - server_version=lava_dashboard_tool.commands:server_version - streams=lava_dashboard_tool.commands:streams - version=lava_dashboard_tool.commands:version - """, + entry_points=entry_points, classifiers=[ "Development Status :: 4 - Beta", "Intended Audience :: Developers", @@ -107,6 +47,7 @@ ], install_requires=[ 'argparse >= 1.1', + 'argcomplete >= 0.3', 'keyring', 'json-schema-validator >= 2.0', 'versiontools >= 1.3.1'