From patchwork Tue May 29 02:23:10 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael-Doyle Hudson X-Patchwork-Id: 9014 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 9ECFE23E49 for ; Tue, 29 May 2012 02:23:13 +0000 (UTC) Received: from mail-gg0-f180.google.com (mail-gg0-f180.google.com [209.85.161.180]) by fiordland.canonical.com (Postfix) with ESMTP id 4F65AA1871C for ; Tue, 29 May 2012 02:23:13 +0000 (UTC) Received: by ggnf1 with SMTP id f1so2676880ggn.11 for ; Mon, 28 May 2012 19:23:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf :content-type:mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state; bh=oSGxvXzv3MJhpKXsP0b65mMsmDA8LSNsa2NFSSq/lPI=; b=ow2uKl92VLNyAQ6AtunLQbFLYn8gMM8ycEBqLRXXL4JwfJS/2tN8XNoit6rO8lZ84x 1UbrvUCeSoLz441SBVYV8CRSVRfX43KiWhvQe3YeuSVzrGT9SsH8VTXRF118FO8T16Uc tGpA26VIaNwrH1ZleQ6wp4PHwZN14AV+5V/XCIhKjK1GgzgVJ3Rr8ZnXk+oSCJv93Z0B eqji3XHs6DY3oGSxuPa9ERe4F20daFHCG1X8MnwYqNSWBqLixJXzWTlGgH5uclDa9goP WS6kYjN5C1hr8iyhWVCtNfaH8DP5T2I1rKE6AdT5cejARcxWlAlQTeNQGnpbRtiByYBO i1ug== Received: by 10.50.40.193 with SMTP id z1mr5814533igk.0.1338258192376; Mon, 28 May 2012 19:23:12 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.231.24.148 with SMTP id v20csp99021ibb; Mon, 28 May 2012 19:23:11 -0700 (PDT) Received: by 10.14.37.134 with SMTP id y6mr2948342eea.78.1338258190985; Mon, 28 May 2012 19:23:10 -0700 (PDT) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id di2si11832932wib.1.2012.05.28.19.23.10 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 28 May 2012 19:23:10 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) smtp.mail=bounces@canonical.com Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1SZC5W-00012E-CU for ; Tue, 29 May 2012 02:23:10 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id 51319E088E for ; Tue, 29 May 2012 02:23:10 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-dispatcher X-Launchpad-Branch: ~linaro-validation/lava-dispatcher/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 305 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-dispatcher/trunk] Rev 305: make the deployment retrying a little more robust and clearer Message-Id: <20120529022310.23730.62639.launchpad@ackee.canonical.com> Date: Tue, 29 May 2012 02:23:10 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="15316"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: f118ba03094b90dfdf81e5f42190d9a70f77efd6 X-Gm-Message-State: ALoCoQlbML+bA5H/iSPoUkj+MeR6YXDyv3SP4Aa6ynLs9tm//p2dVPbmENrM/jTRiJtqeAcTWv1P Merge authors: Michael Hudson-Doyle (mwhudson) Related merge proposals: https://code.launchpad.net/~mwhudson/lava-dispatcher/retry-fixes/+merge/107587 proposed by: Michael Hudson-Doyle (mwhudson) ------------------------------------------------------------ revno: 305 [merge] committer: Michael Hudson-Doyle branch nick: trunk timestamp: Tue 2012-05-29 14:21:39 +1200 message: make the deployment retrying a little more robust and clearer modified: lava_dispatcher/client/master.py --- lp:lava-dispatcher https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk You are subscribed to branch lp:lava-dispatcher. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk/+edit-subscription === modified file 'lava_dispatcher/client/master.py' --- lava_dispatcher/client/master.py 2012-05-28 06:46:23 +0000 +++ lava_dispatcher/client/master.py 2012-05-29 01:58:57 +0000 @@ -67,6 +67,8 @@ if rc: raise RuntimeError("Failed to create tarball: %s" % tarfile) +WGET_DEBUGGING_OPTIONS='-S --progress=dot -e dotbytes=2M' + def _deploy_tarball_to_board(session, tarball_url, dest, timeout=-1, num_retry=2): decompression_char = '' if tarball_url.endswith('.gz') or tarball_url.endswith('.tgz'): @@ -74,32 +76,29 @@ elif tarball_url.endswith('.bz2'): decompression_char = 'j' - deploy_ok=False + deploy_ok = False - while (num_retry > 0 and not deploy_ok): + while num_retry > 0: try: - session._empty_pexpect_buffer() - session._client.proc.sendline( - 'wget --no-proxy --connect-timeout=30 -S -O- %s --progress=dot -e dotbytes=2M |' + session.run( + 'wget --no-proxy --connect-timeout=30 %s -O- %s |' 'tar --numeric-owner -C %s -x%sf -' - % (tarball_url, dest, decompression_char)) - - match_id = session._client.proc.expect(["tar: Error", "unexpected end of file", - "tar: Unexpected EOF in archive",session._prompt_str], timeout=timeout) - if match_id != 3: - logging.warning("Deploy %s failed. %d retry left." %(tarball_url, num_retry-1)) - else: - deploy_ok=True - except: + % (WGET_DEBUGGING_OPTIONS, tarball_url, dest, decompression_char), + timeout=timeout) + except (OperationFailed, pexpect.TIMEOUT): logging.warning("Deploy %s failed. %d retry left." %(tarball_url, num_retry-1)) - # do some delay before retry - if (num_retry > 1 and not deploy_ok): + else: + deploy_ok = True + break + + if num_retry > 1: + # send CTRL C in case wget still hasn't exited. + session._client.proc.sendcontrol("c") + session._client.proc.sendline("echo 'retry left %s'" % (num_retry-1)) + # And wait a little while. sleep_time=5*60 logging.info("Wait %d second before retry" % sleep_time) time.sleep(sleep_time) - # send CTRL C to exit last command. - session._client.proc.sendline("\003") - session._client.proc.sendline("echo 'retry left %s'" % (num_retry-1)) num_retry = num_retry - 1 if not deploy_ok: