From patchwork Thu Oct 18 15:11:13 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Doan X-Patchwork-Id: 12345 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 8B21823F58 for ; Thu, 18 Oct 2012 15:11:19 +0000 (UTC) Received: from mail-ie0-f180.google.com (mail-ie0-f180.google.com [209.85.223.180]) by fiordland.canonical.com (Postfix) with ESMTP id BC860A188F6 for ; Thu, 18 Oct 2012 15:11:18 +0000 (UTC) Received: by mail-ie0-f180.google.com with SMTP id e10so13400135iej.11 for ; Thu, 18 Oct 2012 08:11:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf :content-type:mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state; bh=1fDs2CBfvCdXkno1Rq4nx+T92WhyvEoah29pqKsR2xE=; b=WcgETU3AXgG/0vcPpgUcwXMuZf4TKwB57RAaqT8/WtRUn6qBtsRepxpsNJ2QmC1pbu bfJUHWzOnwAigGDF5VJaoGrI//MP64QBdnOOhhoUQgwtoWm154HRucN6tkQK+312leEb 1Wv+rJY+jbuHd6jZK29TAH8PFYNC3eRFq4i7S8OLBavCJUnYiHD/HQZDoQNwk3rs7+fU k3mw0OFGSi+sbvTdapJAXLIaidAMO0bwYI2dFjtbfjbQ7gfF9DUyGxRdbF8Ezd71O7ra DpONjM1fkehuCewANvWykUEaAdLUIT6CVq8ZSesyWiH2MTDzoSc7okGwLMOUCUSeuI+t YuMA== Received: by 10.50.161.169 with SMTP id xt9mr5079343igb.62.1350573077185; Thu, 18 Oct 2012 08:11:17 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.50.67.148 with SMTP id n20csp1089757igt; Thu, 18 Oct 2012 08:11:16 -0700 (PDT) Received: by 10.180.93.33 with SMTP id cr1mr12029917wib.8.1350573075558; Thu, 18 Oct 2012 08:11:15 -0700 (PDT) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id g17si27749964weo.102.2012.10.18.08.11.15 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 18 Oct 2012 08:11:15 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) smtp.mail=bounces@canonical.com Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1TOrkf-00069V-VP for ; Thu, 18 Oct 2012 15:11:14 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id CD906E0514 for ; Thu, 18 Oct 2012 15:11:13 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-dispatcher X-Launchpad-Branch: ~linaro-validation/lava-dispatcher/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 408 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-dispatcher/trunk] Rev 408: fix health check regression for master images Message-Id: <20121018151113.22507.81131.launchpad@ackee.canonical.com> Date: Thu, 18 Oct 2012 15:11:13 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="16160"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: 11f2cacc86f847e086c6b538891fc2d9ebbdc537 X-Gm-Message-State: ALoCoQmVjzBfIrhHrhf0ndmx/ufMMZK2Ra2JXNxI0o7gK6pszuw6OKRIgSXOi9r82Fnji3FJTRn4 ------------------------------------------------------------ revno: 408 committer: Andy Doan branch nick: lava-dispatcher timestamp: Thu 2012-10-18 10:10:06 -0500 message: fix health check regression for master images Target implementations must now always set self.deployment_data. We were missing logic to set this when we find cached tarballs for health jobs. This commit does two things: 1. Add "data" to the tuple that gets cached. This data is flexible, but in practice is just "oe" or "ubuntu" to tell us which deployment_data is needed. 2. Make tarballcache more resilient. This changes how tarballcache is set up. So current installations not including this new "data" file will need to re-generate their tarballs. While I was at it, I also fixed logic so that accidental deletion of say "boot.tgz" can be managed properly (ie - it will clear the cache and regenerate) modified: lava_dispatcher/device/master.py lava_dispatcher/tarballcache.py --- lp:lava-dispatcher https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk You are subscribed to branch lp:lava-dispatcher. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk/+edit-subscription === modified file 'lava_dispatcher/device/master.py' --- lava_dispatcher/device/master.py 2012-10-17 16:52:05 +0000 +++ lava_dispatcher/device/master.py 2012-10-18 15:10:06 +0000 @@ -65,6 +65,14 @@ Target.android_deployment_data['boot_cmds'] = 'boot_cmds_android' Target.ubuntu_deployment_data['boot_cmds'] = 'boot_cmds' + # used for tarballcache logic to get proper boot_cmds + Target.ubuntu_deployment_data['data_type'] = 'ubuntu' + Target.oe_deployment_data['data_type'] = 'oe' + self.target_map = { + 'ubuntu': Target.ubuntu_deployment_data, + 'oe': Target.oe_deployment_data, + } + self.master_ip = None if config.pre_connect_command: @@ -124,8 +132,9 @@ self.boot_master_image() if self.context.job_data.get('health_check', False): - (boot_tgz, root_tgz) = tarballcache.get_tarballs( + (boot_tgz, root_tgz, data) = tarballcache.get_tarballs( self.context, image, self.scratch_dir, self._generate_tarballs) + self.deployment_data = self.target_map[data] else: image_file = download_image(image, self.context, self.scratch_dir) boot_tgz, root_tgz = self._generate_tarballs(image_file) @@ -171,7 +180,11 @@ tb = traceback.format_exc() self.sio.write(tb) raise - return boot_tgz, root_tgz + + # we need to associate the deployment data with these so that we + # can provide the proper boot_cmds later on in the job + data = self.deployment_data['data_type'] + return boot_tgz, root_tgz, data def target_extract(self, runner, tar_url, dest, timeout=-1, num_retry=5): decompression_char = '' === modified file 'lava_dispatcher/tarballcache.py' --- lava_dispatcher/tarballcache.py 2012-09-30 17:01:37 +0000 +++ lava_dispatcher/tarballcache.py 2012-10-18 15:10:06 +0000 @@ -47,18 +47,25 @@ with _cache_locked(image_url, context.config.lava_cachedir) as cachedir: boot_tgz = os.path.join(cachedir, 'boot.tgz') root_tgz = os.path.join(cachedir, 'root.tgz') + data_file = os.path.join(cachedir, 'data') if os.path.exists(boot_tgz) and os.path.exists(root_tgz): - logging.info('returning cached copies') - (boot_tgz, root_tgz) = _link(boot_tgz, root_tgz, scratch_dir) - return (boot_tgz, root_tgz) + data = _get_data(cachedir, data_file) + if data is not None: + logging.info('returning cached copies') + (boot_tgz, root_tgz) = _link(boot_tgz, root_tgz, scratch_dir) + return (boot_tgz, root_tgz, data) + else: + logging.info('no cache found for %s' % image_url) - logging.info('no cache found for %s' % image_url) + _clear_cache(boot_tgz, root_tgz, data_file) image = download_image(image_url, context, cachedir) - (boot_tgz, root_tgz) = generator(image) + (boot_tgz, root_tgz, data) = generator(image) + with open(data_file, 'w') as f: + f.write(data) _link(boot_tgz, root_tgz, cachedir) os.unlink(image) - return (boot_tgz, root_tgz) + return (boot_tgz, root_tgz, data) def _link(boot_tgz, root_tgz, destdir): @@ -69,6 +76,25 @@ return (dboot_tgz, droot_tgz) +def _clear_cache(boot_tgz, root_tgz, data): + logging.info('Clearing cache contents') + if os.path.exists(boot_tgz): + os.unlink(boot_tgz) + if os.path.exists(root_tgz): + os.unlink(root_tgz) + if os.path.exists(data): + os.unlink(data) + + +def _get_data(cachedir, data_file): + try: + with open(data_file, 'r') as f: + return f.read() + except IOError: + logging.warn('No data found for cached tarballs in %s' % cachedir) + return None + + @contextlib.contextmanager def _cache_locked(image_url, cachedir): cachedir = utils.url_to_cache(image_url, cachedir).replace('.', '-')