From patchwork Tue Apr 16 19:37:14 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Antonio Terceiro X-Patchwork-Id: 16162 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-bk0-f70.google.com (mail-bk0-f70.google.com [209.85.214.70]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id CB79623A3E for ; Tue, 16 Apr 2013 19:37:51 +0000 (UTC) Received: by mail-bk0-f70.google.com with SMTP id y8sf1119749bkt.1 for ; Tue, 16 Apr 2013 12:37:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:x-beenthere:x-received:received-spf:x-received :x-forwarded-to:x-forwarded-for:delivered-to:x-received:received-spf :mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state:x-original-sender :x-original-authentication-results:mailing-list:list-id :x-google-group-id:list-post:list-help:list-archive:list-unsubscribe :content-type; bh=OQFvuRjkXNMoH8b/BD92vcmf17K/Kcdu3dy5WdgwX/k=; b=oZ3O8vjI5zLf49nGain0476pDRBwqaWgksXCFPM5pxJ73ckLxYq1+0faFQN/tm9GDe ch1euZRtDByheELuNtyzkdA87pvyNo1w5T6NA4GlEWQfcJc6+GELJAWCkm/s1iwXJo40 HfXr1VO1aWYVUokIRHvi5UAGFb8lEqMfC4bi4i8aS2qPja1T9Mq3R0Ftp0AizfFROg1N 5lECc+GrN08VvfAT6G15xSUCW8gokGdEaH3KUkuBdSedK671rfPywzRFzwWVCk7uBcey FzS/yplrB9YDu88swlXQ1FAPrGwb9UChCaZoCx5x8NS1xBovR254oInVnbyIP9sYkeyK QSpQ== X-Received: by 10.180.96.225 with SMTP id dv1mr1327205wib.6.1366141037738; Tue, 16 Apr 2013 12:37:17 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.180.93.201 with SMTP id cw9ls338047wib.14.gmail; Tue, 16 Apr 2013 12:37:17 -0700 (PDT) X-Received: by 10.180.83.199 with SMTP id s7mr4011922wiy.19.1366141037668; Tue, 16 Apr 2013 12:37:17 -0700 (PDT) Received: from mail-ve0-f170.google.com (mail-ve0-f170.google.com [209.85.128.170]) by mx.google.com with ESMTPS id fu5si1164281wib.7.2013.04.16.12.37.17 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 16 Apr 2013 12:37:17 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.128.170 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.128.170; Received: by mail-ve0-f170.google.com with SMTP id 14so798712vea.29 for ; Tue, 16 Apr 2013 12:37:16 -0700 (PDT) X-Received: by 10.220.39.69 with SMTP id f5mr2640364vce.45.1366141036395; Tue, 16 Apr 2013 12:37:16 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patches@linaro.org Received: by 10.58.127.98 with SMTP id nf2csp40337veb; Tue, 16 Apr 2013 12:37:15 -0700 (PDT) X-Received: by 10.180.97.233 with SMTP id ed9mr21313791wib.32.1366141035232; Tue, 16 Apr 2013 12:37:15 -0700 (PDT) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id gi5si1154045wic.77.2013.04.16.12.37.14 (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 16 Apr 2013 12:37:15 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1USBgo-0007z0-Iz for ; Tue, 16 Apr 2013 19:37:14 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id 81301E00F3 for ; Tue, 16 Apr 2013 19:37:14 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-dispatcher X-Launchpad-Branch: ~linaro-validation/lava-dispatcher/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 580 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-dispatcher/trunk] Rev 580: Improve modularization of distro support Message-Id: <20130416193714.30618.26841.launchpad@ackee.canonical.com> Date: Tue, 16 Apr 2013 19:37:14 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: list X-Generated-By: Launchpad (canonical.com); Revision="16567"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: 744a5347e294b4ece26bb3453e34f3ce9f9335d2 X-Gm-Message-State: ALoCoQkOML2pfB4wVwuTfz0ypQeqrQlO+B2EyTQjGxmMyrxK5Cu70r2ErHiGP7wrKHhCvsFgxAVO X-Original-Sender: noreply@launchpad.net X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.128.170 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org List-ID: X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , Merge authors: Antonio Terceiro (terceiro) Related merge proposals: https://code.launchpad.net/~terceiro/lava-dispatcher/modularize-distro-support/+merge/159007 proposed by: Antonio Terceiro (terceiro) review: Approve - Fathi Boudra (fboudra) ------------------------------------------------------------ revno: 580 [merge] committer: Antonio Terceiro branch nick: trunk timestamp: Tue 2013-04-16 16:36:32 -0300 message: Improve modularization of distro support added: lava_test_shell/README lava_test_shell/distro/ lava_test_shell/distro/android/ lava_test_shell/distro/ubuntu/ lava_test_shell/distro/ubuntu/lava-install-packages lava_test_shell/distro/ubuntu/lava-installed-packages lava_test_shell/distro/ubuntu/lava-os-build lava_test_shell/lava-installed-packages lava_test_shell/lava-os-build renamed: lava_test_shell/lava-test-runner-android => lava_test_shell/distro/android/lava-test-runner lava_test_shell/lava-test-runner-ubuntu => lava_test_shell/lava-test-runner modified: lava_dispatcher/actions/lava_test_shell.py lava_test_shell/lava-test-runner --- lp:lava-dispatcher https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk You are subscribed to branch lp:lava-dispatcher. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk/+edit-subscription === modified file 'lava_dispatcher/actions/lava_test_shell.py' --- lava_dispatcher/actions/lava_test_shell.py 2013-04-07 19:43:05 +0000 +++ lava_dispatcher/actions/lava_test_shell.py 2013-04-15 19:21:33 +0000 @@ -106,7 +106,7 @@ # to the host and turned into a bundle for submission to the dashboard. from datetime import datetime -import glob +from glob import glob import logging import os import pexpect @@ -132,38 +132,18 @@ from lava_dispatcher.downloader import download_image LAVA_TEST_DIR = '%s/../../lava_test_shell' % os.path.dirname(__file__) -LAVA_TEST_ANDROID = '%s/lava-test-runner-android' % LAVA_TEST_DIR -LAVA_TEST_UBUNTU = '%s/lava-test-runner-ubuntu' % LAVA_TEST_DIR -LAVA_TEST_UPSTART = '%s/lava-test-runner.conf' % LAVA_TEST_DIR -LAVA_TEST_INITD = '%s/lava-test-runner.init.d' % LAVA_TEST_DIR -LAVA_TEST_SHELL = '%s/lava-test-shell' % LAVA_TEST_DIR -LAVA_TEST_CASE = '%s/lava-test-case' % LAVA_TEST_DIR -LAVA_TEST_CASE_ATTACH = '%s/lava-test-case-attach' % LAVA_TEST_DIR -LAVA_TEST_RUN_ATTACH = '%s/lava-test-run-attach' % LAVA_TEST_DIR -Target.android_deployment_data['lava_test_runner'] = LAVA_TEST_ANDROID -Target.android_deployment_data['lava_test_shell'] = LAVA_TEST_SHELL -Target.android_deployment_data['lava_test_case'] = LAVA_TEST_CASE -Target.android_deployment_data['lava_test_case_attach'] = LAVA_TEST_CASE_ATTACH -Target.android_deployment_data['lava_test_run_attach'] = LAVA_TEST_RUN_ATTACH +Target.android_deployment_data['distro'] = 'android' Target.android_deployment_data['lava_test_sh_cmd'] = '/system/bin/mksh' Target.android_deployment_data['lava_test_dir'] = '/data/lava' Target.android_deployment_data['lava_test_results_part_attr'] = 'data_part_android_org' -Target.ubuntu_deployment_data['lava_test_runner'] = LAVA_TEST_UBUNTU -Target.ubuntu_deployment_data['lava_test_shell'] = LAVA_TEST_SHELL -Target.ubuntu_deployment_data['lava_test_case'] = LAVA_TEST_CASE -Target.ubuntu_deployment_data['lava_test_case_attach'] = LAVA_TEST_CASE_ATTACH -Target.ubuntu_deployment_data['lava_test_run_attach'] = LAVA_TEST_RUN_ATTACH +Target.ubuntu_deployment_data['distro'] = 'ubuntu' Target.ubuntu_deployment_data['lava_test_sh_cmd'] = '/bin/bash' Target.ubuntu_deployment_data['lava_test_dir'] = '/lava' Target.ubuntu_deployment_data['lava_test_results_part_attr'] = 'root_part' -Target.oe_deployment_data['lava_test_runner'] = LAVA_TEST_UBUNTU -Target.oe_deployment_data['lava_test_shell'] = LAVA_TEST_SHELL -Target.oe_deployment_data['lava_test_case'] = LAVA_TEST_CASE -Target.oe_deployment_data['lava_test_case_attach'] = LAVA_TEST_CASE_ATTACH -Target.oe_deployment_data['lava_test_run_attach'] = LAVA_TEST_RUN_ATTACH +Target.oe_deployment_data['distro'] = 'oe' Target.oe_deployment_data['lava_test_sh_cmd'] = '/bin/sh' Target.oe_deployment_data['lava_test_dir'] = '/lava' Target.oe_deployment_data['lava_test_results_part_attr'] = 'root_part' @@ -376,11 +356,17 @@ f.write('set -ex\n') f.write('cd %s\n' % targetdir) - # TODO how should we handle this for Android? + distro = self.context.client.target_device.deployment_data['distro'] + + # generic dependencies - must be named the same across all distros + # supported by the testdef deps = self.testdef['install'].get('deps', []) + + # distro-specific dependencies + deps = deps + self.testdef['install'].get('deps-' + distro, []) + if deps: - f.write('sudo apt-get update\n') - f.write('sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -q ') + f.write('lava-install-packages ') for dep in deps: f.write('%s ' % dep) f.write('\n') @@ -484,7 +470,6 @@ def run(self, testdef_urls=None, testdef_repos=None, timeout=-1): target = self.client.target_device - self._assert_target(target) testdefs_by_uuid = self._configure_target(target, testdef_urls, testdef_repos) @@ -533,26 +518,25 @@ return False def _copy_runner(self, mntdir, target): - runner = target.deployment_data['lava_test_runner'] - shutil.copy(runner, '%s/bin/lava-test-runner' % mntdir) - os.chmod('%s/bin/lava-test-runner' % mntdir, XMOD) - - shcmd = target.deployment_data['lava_test_sh_cmd'] - - for key in ['lava_test_shell', 'lava_test_case_attach', 'lava_test_run_attach']: - fname = target.deployment_data[key] + shell = target.deployment_data['lava_test_sh_cmd'] + + # Generic scripts + scripts_to_copy = glob(os.path.join(LAVA_TEST_DIR, 'lava-*')) + + # Distro-specific scripts override the generic ones + distro = target.deployment_data['distro'] + distro_support_dir = '%s/distro/%s' % (LAVA_TEST_DIR, distro) + for script in glob(os.path.join(distro_support_dir, 'lava-*')): + scripts_to_copy.append(script) + + for fname in scripts_to_copy: with open(fname, 'r') as fin: - with open('%s/bin/%s' % (mntdir, os.path.basename(fname)), 'w') as fout: - fout.write("#!%s\n\n" % shcmd) + foutname = os.path.basename(fname) + with open('%s/bin/%s' % (mntdir, foutname), 'w') as fout: + fout.write("#!%s\n\n" % shell) fout.write(fin.read()) os.fchmod(fout.fileno(), XMOD) - tc = target.deployment_data['lava_test_case'] - with open(tc, 'r') as fin: - with open('%s/bin/lava-test-case' % mntdir, 'w') as fout: - fout.write('#!%s\n\n' % shcmd) - fout.write(fin.read()) - os.fchmod(fout.fileno(), XMOD) def _mk_runner_dirs(self, mntdir): utils.ensure_directory('%s/bin' % mntdir) @@ -618,16 +602,3 @@ with os.fdopen(fd, 'w') as f: DocumentIO.dump(f, bundle) - def _assert_target(self, target): - """ Ensure the target has the proper deployment data required by this - action. This allows us to exit the action early rather than going 75% - through the steps before discovering something required is missing - """ - if not target.deployment_data: - raise RuntimeError('Target includes no deployment_data') - - keys = ['lava_test_runner', 'lava_test_shell', 'lava_test_dir', - 'lava_test_sh_cmd'] - for k in keys: - if k not in target.deployment_data: - raise RuntimeError('Target deployment_data missing %s' % k) === added file 'lava_test_shell/README' --- lava_test_shell/README 1970-01-01 00:00:00 +0000 +++ lava_test_shell/README 2013-04-11 19:10:11 +0000 @@ -0,0 +1,11 @@ +This directory contains support scripts for lava-test-shell. + +The scripts in this directory will be copied into the target device and will be +in $PATH during the lava-test-shell execution. + +Distribution-specific scripts can be placed in distro/$distroname, and will +override the ones in the top level directory. For example, +distro/android/lava-test-runner will be used on Android instead of the +lava-test-runner script present at the same directory as this README file. + +All scripts have to be named using a "lava-" suffix. === added directory 'lava_test_shell/distro' === added directory 'lava_test_shell/distro/android' === renamed file 'lava_test_shell/lava-test-runner-android' => 'lava_test_shell/distro/android/lava-test-runner' === added directory 'lava_test_shell/distro/ubuntu' === added file 'lava_test_shell/distro/ubuntu/lava-install-packages' --- lava_test_shell/distro/ubuntu/lava-install-packages 1970-01-01 00:00:00 +0000 +++ lava_test_shell/distro/ubuntu/lava-install-packages 2013-04-15 19:16:44 +0000 @@ -0,0 +1,4 @@ +#!/bin/sh + +sudo DEBIAN_FRONTEND=noninteractive apt-get update +sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -q "$@" === added file 'lava_test_shell/distro/ubuntu/lava-installed-packages' --- lava_test_shell/distro/ubuntu/lava-installed-packages 1970-01-01 00:00:00 +0000 +++ lava_test_shell/distro/ubuntu/lava-installed-packages 2013-04-11 19:50:48 +0000 @@ -0,0 +1,3 @@ +#!/bin/sh + +dpkg-query -W -f '${status} ${package} : ${version}\n' | sed -n 's/^install ok installed/package:/p' === added file 'lava_test_shell/distro/ubuntu/lava-os-build' --- lava_test_shell/distro/ubuntu/lava-os-build 1970-01-01 00:00:00 +0000 +++ lava_test_shell/distro/ubuntu/lava-os-build 2013-04-11 19:50:48 +0000 @@ -0,0 +1,3 @@ +#!/bin/sh + +cat /etc/lsb-release | grep DESCRIPTION | cut -d\" -f2 === added file 'lava_test_shell/lava-installed-packages' --- lava_test_shell/lava-installed-packages 1970-01-01 00:00:00 +0000 +++ lava_test_shell/lava-installed-packages 2013-04-11 19:50:48 +0000 @@ -0,0 +1,3 @@ +#!/bin/sh + +echo 'Unsupported distro: cannot obtain list of installed packages' === added file 'lava_test_shell/lava-os-build' --- lava_test_shell/lava-os-build 1970-01-01 00:00:00 +0000 +++ lava_test_shell/lava-os-build 2013-04-11 19:50:48 +0000 @@ -0,0 +1,3 @@ +#!/bin/sh + +echo 'Unsupported distro: cannot determine build version' === renamed file 'lava_test_shell/lava-test-runner-ubuntu' => 'lava_test_shell/lava-test-runner' --- lava_test_shell/lava-test-runner-ubuntu 2013-03-15 16:50:30 +0000 +++ lava_test_shell/lava-test-runner 2013-04-11 19:50:48 +0000 @@ -5,12 +5,6 @@ RESULTSDIR="/lava/results" BINDIR="/lava/bin" -detect_distro() { - test -x /usr/bin/dpkg-query && echo "debian_based" && return - - echo "unknown" -} - hwcontext() { mkdir -p ${RESULTSDIR}/hwcontext @@ -21,31 +15,17 @@ [ -f ${meminfo} ] || cat /proc/meminfo > ${meminfo} } -unknown_swcontext() { - mkdir -p ${RESULTSDIR}/swcontext - build=${RESULTSDIR}/swcontext/build.txt - pkgs=${RESULTSDIR}/swcontext/pkgs.txt - - # we don't know about the software context - echo 'Unsupported distro: cannot determine build version' > "$build" - echo 'Unsupported distro: cannot obtain list of installed packages' > "$pkgs" -} - -debian_based_swcontext() +swcontext() { mkdir -p ${RESULTSDIR}/swcontext build=${RESULTSDIR}/swcontext/build.txt pkgs=${RESULTSDIR}/swcontext/pkgs.txt - [ -f ${build} ] || cat /etc/lsb-release | grep DESCRIPTION | cut -d\" -f2 > ${build} - # this does a query of installed packaged that will look similar to + lava-os-build > ${build} + + # this has to print a list of installed packages that will look similar to # what android's package list does - [ -f ${pkgs} ] || dpkg-query -W -f '${status} ${package} : ${version}\n' | sed -n 's/^install ok installed/package:/p' > ${pkgs} -} - -swcontext() { - distro=$(detect_distro) - ${distro}_swcontext + lava-installed-packages > ${pkgs} } cleanup()