From patchwork Thu Nov 15 04:55:12 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Doan X-Patchwork-Id: 12852 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id DE72823E01 for ; Thu, 15 Nov 2012 04:55:15 +0000 (UTC) Received: from mail-ia0-f180.google.com (mail-ia0-f180.google.com [209.85.210.180]) by fiordland.canonical.com (Postfix) with ESMTP id 70A9AA18C74 for ; Thu, 15 Nov 2012 04:55:15 +0000 (UTC) Received: by mail-ia0-f180.google.com with SMTP id f6so767478iag.11 for ; Wed, 14 Nov 2012 20:55:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf :content-type:mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state; bh=9VEr01P+slm5GllrReWON8ffGggA77bvWnp5jyPUj0Q=; b=nW5TADybOW7M/U77+xrNl2NY8jeAp53ml9ihBtNWb97zqZzjC16BLwmhRb6pwzTU/Z SSsPSngGMxHp8xnIQtzta7sZIbWYHRgXGI7wgtyrfNOneoeW8nY/iC8d0qIs6uDqRGLT TFzI2FZebaAkQXLtNZXqm5cH4wOh5/9ebBzlkKgzz+Md0qn6Dc1y8bWmQGP+EPY2kASg ovQ74P1VkwA/PNF0WcrQyT5JQWKKVAWY3PmncZjkE6JqJiGGS/acqo1quZNEKtPoTwxm YOctb+9i+d34AKxAv7MrpmRB0jeoseEFhpGlh67lSD2tf/npV/2uJddYxmwS3YmA40cl so+w== Received: by 10.50.42.168 with SMTP id p8mr1104073igl.57.1352955314845; Wed, 14 Nov 2012 20:55:14 -0800 (PST) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.50.67.148 with SMTP id n20csp771668igt; Wed, 14 Nov 2012 20:55:14 -0800 (PST) Received: by 10.216.144.170 with SMTP id n42mr2298404wej.94.1352955313469; Wed, 14 Nov 2012 20:55:13 -0800 (PST) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id y49si7737161wen.114.2012.11.14.20.55.12 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 14 Nov 2012 20:55:13 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) smtp.mail=bounces@canonical.com Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1TYrTs-0002tJ-M1 for ; Thu, 15 Nov 2012 04:55:12 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id 98152E0017 for ; Thu, 15 Nov 2012 04:55:12 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-dispatcher X-Launchpad-Branch: ~linaro-validation/lava-dispatcher/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 447 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-dispatcher/trunk] Rev 447: add new extract_tarball API to target Message-Id: <20121115045512.6597.1002.launchpad@ackee.canonical.com> Date: Thu, 15 Nov 2012 04:55:12 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="16265"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: 03d5f418575c1df0cfed7d72e4b8743ae5ecaa84 X-Gm-Message-State: ALoCoQlrHWHjHLgeMT1S2IQsnN/LYdRu5hsdX3D0T7lhNspgn5FwsNS/nllvzauSLX8RzuPup6S3 Merge authors: Andy Doan (doanac) Related merge proposals: https://code.launchpad.net/~doanac/lava-dispatcher/extract-tarball/+merge/133757 proposed by: Andy Doan (doanac) review: Approve - Michael Hudson-Doyle (mwhudson) ------------------------------------------------------------ revno: 447 [merge] committer: Andy Doan branch nick: lava-dispatcher timestamp: Wed 2012-11-14 22:54:00 -0600 message: add new extract_tarball API to target modified: lava_dispatcher/actions/android_install_binaries.py lava_dispatcher/actions/android_install_cts_medias.py lava_dispatcher/device/fastmodel.py lava_dispatcher/device/master.py lava_dispatcher/device/qemu.py lava_dispatcher/device/target.py --- lp:lava-dispatcher https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk You are subscribed to branch lp:lava-dispatcher. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk/+edit-subscription === modified file 'lava_dispatcher/actions/android_install_binaries.py' --- lava_dispatcher/actions/android_install_binaries.py 2012-09-30 17:01:44 +0000 +++ lava_dispatcher/actions/android_install_binaries.py 2012-11-09 20:26:45 +0000 @@ -18,6 +18,7 @@ # along with this program; if not, see . import logging + from lava_dispatcher.actions import BaseAction, null_or_empty_schema @@ -27,13 +28,10 @@ def run(self): driver_tarball = self.client.config.android_binary_drivers + partition = self.client.config.root_part + if driver_tarball is None: logging.error("android_binary_drivers not defined in any config") return - with self.client.target_device._as_master() as session: - session.run( - 'mount /dev/disk/by-label/testrootfs /mnt/lava/system') - self.client.target_device.target_extract( - session, driver_tarball, '/mnt/lava/system', timeout=600) - session.run('umount /mnt/lava/system') + self.client.target_device.extract_tarball(driver_tarball, partition) === modified file 'lava_dispatcher/actions/android_install_cts_medias.py' --- lava_dispatcher/actions/android_install_cts_medias.py 2012-10-25 06:33:00 +0000 +++ lava_dispatcher/actions/android_install_cts_medias.py 2012-11-09 20:26:45 +0000 @@ -39,12 +39,5 @@ logging.error("The url for the cts media files is not specified") return - with self.client.target_device._as_master() as session: - if not session.has_partition_with_label('sdcard'): - return - session.run('mkdir -p /mnt/lava/sdcard') - session.run( - 'mount /dev/disk/by-label/sdcard /mnt/lava/sdcard') - self.client.target_device.target_extract( - session, media_url, '/mnt/lava/sdcard', timeout=timeout) - session.run('umount /mnt/lava/sdcard') + partition = self.client.config.sdcard_part_android_org + self.client.target_device.extract_tarball(media_url, partition) === modified file 'lava_dispatcher/device/fastmodel.py' --- lava_dispatcher/device/fastmodel.py 2012-10-23 17:45:15 +0000 +++ lava_dispatcher/device/fastmodel.py 2012-11-09 20:25:51 +0000 @@ -45,6 +45,7 @@ ) from lava_dispatcher.utils import ( ensure_directory, + extract_targz, logging_spawn, logging_system, ) @@ -158,6 +159,13 @@ ensure_directory(path) yield path + def extract_tarball(self, tarball_url, partition, directory='/'): + logging.info('extracting %s to target' % tarball_url) + + with image_partition_mounted(self._sd_image, partition) as mntdir: + tb = download_image(tarball_url, self.context, decompress=False) + extract_targz(tb, '%s/%s' % (mntdir, directory)) + def _fix_perms(self): ''' The directory created for the image download/creation gets created with tempfile.mkdtemp which grants permission only to the creator of @@ -268,7 +276,7 @@ def get_device_version(self): cmd = '%s --version' % self._sim_binary try: - banner = subprocess.check_output(cmd, shell = True) + banner = subprocess.check_output(cmd, shell=True) return self._parse_fastmodel_version(banner) except subprocess.CalledProcessError: return "unknown" @@ -280,6 +288,7 @@ else: return "unknown" + class _pexpect_drain(threading.Thread): ''' The simulator process can dump a lot of information to its console. If don't actively read from it, the pipe will get full and the process will === modified file 'lava_dispatcher/device/master.py' --- lava_dispatcher/device/master.py 2012-11-13 09:59:11 +0000 +++ lava_dispatcher/device/master.py 2012-11-15 04:54:00 +0000 @@ -225,26 +225,30 @@ raise RuntimeError('extracting %s on target failed' % tar_url) - @contextlib.contextmanager - def file_system(self, partition, directory): - logging.info('attempting to access master filesystem %r:%s' % - (partition, directory)) - + def get_partition(self, runner, partition): if partition == self.config.boot_part: partition = '/dev/disk/by-label/testboot' elif partition == self.config.root_part: partition = '/dev/disk/by-label/testrootfs' - elif partition != self.config.data_part_android_org: + elif partition == self.config.sdcard_part_android_org: + partition = '/dev/disk/by-label/sdcard' + elif partition == self.config.data_part_android_org: + lbl = _android_data_label(runner) + partition = '/dev/disk/by-label/%s' % lbl + else: raise RuntimeError( 'unknown master image partition(%d)' % partition) + return partition + + @contextlib.contextmanager + def file_system(self, partition, directory): + logging.info('attempting to access master filesystem %r:%s' % + (partition, directory)) assert directory != '/', "cannot mount entire partition" with self._as_master() as runner: - if partition == self.config.data_part_android_org: - lbl = _android_data_label(runner) - partition = '/dev/disk/by-label/%s' % lbl - + partition = self.get_partition(runner, partition) runner.run('mount %s /mnt' % partition) try: targetdir = os.path.join('/mnt/%s' % directory) @@ -253,7 +257,8 @@ parent_dir, target_name = os.path.split(targetdir) - runner.run('tar -czf /tmp/fs.tgz -C %s %s' % (parent_dir, target_name)) + runner.run('tar -czf /tmp/fs.tgz -C %s %s' % + (parent_dir, target_name)) runner.run('cd /tmp') # need to be in same dir as fs.tgz self.proc.sendline('python -m SimpleHTTPServer 0 2>/dev/null') match_id = self.proc.expect([ @@ -292,6 +297,17 @@ self.proc.sendcontrol('c') # kill SimpleHTTPServer runner.run('umount /mnt') + def extract_tarball(self, tarball_url, partition, directory='/'): + logging.info('extracting %s to target' % tarball_url) + + with self._as_master() as runner: + partition = self.get_partition(runner, partition) + runner.run('mount %s /mnt' % partition) + try: + self.target_extract(runner, tarball_url, '/mnt/%s' % directory) + finally: + runner.run('umount /mnt') + def _connect_carefully(self, cmd): retry_count = 0 retry_limit = 3 @@ -418,7 +434,8 @@ # Looking for reboot messages or if they are missing, the U-Boot # message will also indicate the reboot is done. match_id = self.proc.expect( - [pexpect.TIMEOUT, 'Restarting system.', 'The system is going down for reboot NOW', + [pexpect.TIMEOUT, 'Restarting system.', + 'The system is going down for reboot NOW', 'Will now restart', 'U-Boot'], timeout=120) if match_id == 0: raise OperationFailed("Soft reboot failed") === modified file 'lava_dispatcher/device/qemu.py' --- lava_dispatcher/device/qemu.py 2012-10-22 17:56:18 +0000 +++ lava_dispatcher/device/qemu.py 2012-11-09 20:25:51 +0000 @@ -35,6 +35,7 @@ ) from lava_dispatcher.utils import ( ensure_directory, + extract_targz, logging_spawn, ) @@ -61,6 +62,13 @@ ensure_directory(path) yield path + def extract_tarball(self, tarball_url, partition, directory='/'): + logging.info('extracting %s to target' % tarball_url) + + with image_partition_mounted(self._sd_image, partition) as mntdir: + tb = download_image(tarball_url, self.context, decompress=False) + extract_targz(tb, '%s/%s' % (mntdir, directory)) + def _power_off(self, proc): if proc is not None: proc.close() @@ -80,7 +88,8 @@ def get_device_version(self): try: - output = subprocess.check_output([self.context.config.default_qemu_binary, '--version']) + output = subprocess.check_output( + [self.context.config.default_qemu_binary, '--version']) matches = re.findall('[0-9]+\.[0-9a-z.+\-:~]+', output) return matches[-1] except subprocess.CalledProcessError: === modified file 'lava_dispatcher/device/target.py' --- lava_dispatcher/device/target.py 2012-10-29 21:03:29 +0000 +++ lava_dispatcher/device/target.py 2012-11-09 20:25:51 +0000 @@ -63,7 +63,6 @@ 'TESTER_PS1_INCLUDES_RC': True, } - def __init__(self, context, device_config): self.context = context self.config = device_config @@ -77,7 +76,8 @@ @property def scratch_dir(self): if self._scratch_dir is None: - self._scratch_dir = utils.mkdtemp(self.context.config.lava_image_tmpdir) + self._scratch_dir = utils.mkdtemp( + self.context.config.lava_image_tmpdir) return self._scratch_dir def power_on(self): @@ -130,6 +130,14 @@ """ raise NotImplementedError('file_system') + def extract_tarball(self, tarball_url, partition, directory='/'): + """ This is similar to the file_system API but is optimized for the + scenario when you just need explode a potentially large tarball on + the target device. The file_system API isn't really suitable for this + when thinking about an implementation like master.py + """ + raise NotImplementedError('extract_tarball') + @contextlib.contextmanager def runner(self): """ Powers on the target, returning a CommandRunner object and will @@ -181,6 +189,7 @@ # just no upstart or dash assumptions self._customize_oe(mnt) + class SerialIO(file): def __init__(self, logfile): self.serialio = StringIO()