From patchwork Thu May 24 23:01:11 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Le Chi Thu X-Patchwork-Id: 8948 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 07C1C23F07 for ; Thu, 24 May 2012 23:01:16 +0000 (UTC) Received: from mail-gg0-f180.google.com (mail-gg0-f180.google.com [209.85.161.180]) by fiordland.canonical.com (Postfix) with ESMTP id AAD99A1882A for ; Thu, 24 May 2012 23:01:15 +0000 (UTC) Received: by ggnf1 with SMTP id f1so509845ggn.11 for ; Thu, 24 May 2012 16:01:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf :content-type:mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state; bh=1Bx+1rEdfVI/7+uGW/HskstEANFYJBC0cSrnx5EuEFk=; b=Yy5HHFp9nqopHsSOW2kGkACPaPEsCPyhBtf0IbzqgRX+woahGBLCWhWIG0SvqWug+v 8EfDmqAoskHmaQ+XzBWB8ugVM2vGO0pc4IBPWWZvTukbe3IZkoYpTfIs5r/abfCmGi5U Nq7B1rtwgHOwrvdnkOM7jXUKniMTr8veaknb3HFyOBq4oEp4GseX7cIJ6fXc4HTnLY1V FxoccIi6ooVx2d4FbtqFQPXWQyHxA4rZE3kNJag6xvOVPxW2ayFVxyehPjjgw9GEa8jp hKulF2AIv03YQaXw/Aqc7C9EEwv/uOnzsUtIXu4ghTzUPMp5F/Rm9g93sl63pgiE5rLb 83KQ== Received: by 10.42.89.72 with SMTP id f8mr749967icm.33.1337900474899; Thu, 24 May 2012 16:01:14 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.231.35.72 with SMTP id o8csp75431ibd; Thu, 24 May 2012 16:01:13 -0700 (PDT) Received: by 10.216.142.200 with SMTP id i50mr644944wej.47.1337900472395; Thu, 24 May 2012 16:01:12 -0700 (PDT) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id m15si991036wee.147.2012.05.24.16.01.11 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 24 May 2012 16:01:12 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) smtp.mail=bounces@canonical.com Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1SXh1r-0004dA-57 for ; Thu, 24 May 2012 23:01:11 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id 1E761E002E for ; Thu, 24 May 2012 23:01:11 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-dispatcher X-Launchpad-Branch: ~linaro-validation/lava-dispatcher/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 300 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-dispatcher/trunk] Rev 300: Added retry on image deployment Message-Id: <20120524230111.29963.2422.launchpad@ackee.canonical.com> Date: Thu, 24 May 2012 23:01:11 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="15288"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: 2fcc7127f89f33f5b0af207d70fe4a096422aa3e X-Gm-Message-State: ALoCoQnD4AL28LyVK7ALPA8M9cXhcfBDVEUp5rtmQvzFpRhUW4DZeLj8EzybKr2TjEv681DgnrzT Merge authors: Le Chi Thu le.chi.thu@linaro.org Related merge proposals: https://code.launchpad.net/~le-chi-thu/lava-dispatcher/add-deploy-retry/+merge/107293 proposed by: Le Chi Thu (le-chi-thu) review: Approve - Michael Hudson-Doyle (mwhudson) ------------------------------------------------------------ revno: 300 [merge] committer: Le Chi Thu le.chi.thu@linaro.org branch nick: trunk timestamp: Fri 2012-05-25 00:59:46 +0200 message: Added retry on image deployment modified: doc/changes.rst lava_dispatcher/client/master.py --- lp:lava-dispatcher https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk You are subscribed to branch lp:lava-dispatcher. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk/+edit-subscription === modified file 'doc/changes.rst' --- doc/changes.rst 2012-05-23 21:49:59 +0000 +++ doc/changes.rst 2012-05-24 17:32:08 +0000 @@ -3,6 +3,7 @@ * Fixed reboot issues * Skip raising exception on the home screen has not displayed for health check jobs +* Retry deployment if failed. .. _version_0_7_1: === modified file 'lava_dispatcher/client/master.py' --- lava_dispatcher/client/master.py 2012-05-22 13:03:31 +0000 +++ lava_dispatcher/client/master.py 2012-05-24 21:12:04 +0000 @@ -67,16 +67,43 @@ if rc: raise RuntimeError("Failed to create tarball: %s" % tarfile) -def _deploy_tarball_to_board(session, tarball_url, dest, timeout=-1): +def _deploy_tarball_to_board(session, tarball_url, dest, timeout=-1, num_retry=2): decompression_char = '' if tarball_url.endswith('.gz') or tarball_url.endswith('.tgz'): decompression_char = 'z' elif tarball_url.endswith('.bz2'): decompression_char = 'j' - session.run( - 'wget --no-proxy --connect-timeout=30 -S -O- %s --progress=dot -e dotbytes=2M | tar --numeric-owner -C %s -x%sf -' % ( - tarball_url, dest, decompression_char), - timeout=timeout) + + deploy_ok=False + + while (num_retry > 0 and not deploy_ok): + try: + session._empty_pexpect_buffer() + session._client.proc.sendline( + 'wget --no-proxy --connect-timeout=30 -S -O- %s --progress=dot -e dotbytes=2M |' + 'tar --numeric-owner -C %s -x%sf -' + % (tarball_url, dest, decompression_char)) + + match_id = session._client.proc.expect(["tar: Error", "unexpected end of file", + "tar: Unexpected EOF in archive",session._prompt_str], timeout=timeout) + if match_id != 3: + logging.warning("Deploy %s failed. %d retry left." %(tarball_url, num_retry-1)) + else: + deploy_ok=True + except: + logging.warning("Deploy %s failed. %d retry left." %(tarball_url, num_retry-1)) + # do some delay before retry + if (num_retry > 1 and not deploy_ok): + sleep_time=5*60 + logging.info("Wait %d second before retry" % sleep_time) + time.sleep(sleep_time) + # send CTRL C to exit last command. + session._client.proc.sendline("\003") + session._client.proc.sendline("echo 'retry left %s'" % num_retry-1) + num_retry = num_retry - 1 + + if not deploy_ok: + raise Exception("Deploy tarball (%s) to board failed" % tarball_url); def _deploy_linaro_rootfs(session, rootfs): logging.info("Deploying linaro image") @@ -754,6 +781,7 @@ self.proc.sendline("hardreset") # after hardreset empty the pexpect buffer self._empty_pexpect_buffer() + def _empty_pexpect_buffer(self): """Make sure there is nothing in the pexpect buffer.""" index = 0