From patchwork Tue Jul 26 13:05:45 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Tunnicliffe X-Patchwork-Id: 3150 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 5AE4924142 for ; Tue, 26 Jul 2011 13:05:50 +0000 (UTC) Received: from mail-qw0-f52.google.com (mail-qw0-f52.google.com [209.85.216.52]) by fiordland.canonical.com (Postfix) with ESMTP id 1115FA18287 for ; Tue, 26 Jul 2011 13:05:49 +0000 (UTC) Received: by qwb8 with SMTP id 8so254910qwb.11 for ; Tue, 26 Jul 2011 06:05:49 -0700 (PDT) Received: by 10.229.236.10 with SMTP id ki10mr550947qcb.274.1311685549508; Tue, 26 Jul 2011 06:05:49 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.229.217.78 with SMTP id hl14cs111682qcb; Tue, 26 Jul 2011 06:05:49 -0700 (PDT) Received: by 10.216.198.146 with SMTP id v18mr4591526wen.94.1311685548287; Tue, 26 Jul 2011 06:05:48 -0700 (PDT) Received: from adelie.canonical.com (adelie.canonical.com [91.189.90.139]) by mx.google.com with ESMTP id l52si301605weq.1.2011.07.26.06.05.47; Tue, 26 Jul 2011 06:05:48 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.139 as permitted sender) client-ip=91.189.90.139; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.139 as permitted sender) smtp.mail=bounces@canonical.com Received: from loganberry.canonical.com ([91.189.90.37]) by adelie.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1QlhKV-0003mp-5i for ; Tue, 26 Jul 2011 13:05:47 +0000 Received: from loganberry.canonical.com (localhost [127.0.0.1]) by loganberry.canonical.com (Postfix) with ESMTP id 9E19A2EB331 for ; Tue, 26 Jul 2011 13:05:45 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: linaro-image-tools X-Launchpad-Branch: ~linaro-image-tools/linaro-image-tools/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 387 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-image-tools/linaro-image-tools/trunk] Rev 387: Merging in temporary workaround for bug #816015. Message-Id: <20110726130545.21230.54750.launchpad@loganberry.canonical.com> Date: Tue, 26 Jul 2011 13:05:45 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="13503"; Instance="initZopeless config overlay" X-Launchpad-Hash: 3f4e127e47a5c3b1a3ba19dddd322d79d430f170 Merge authors: James Tunnicliffe (dooferlad) Related merge proposals: https://code.launchpad.net/~dooferlad/linaro-image-tools/restrict_fetch_image_indexer/+merge/69252 proposed by: James Tunnicliffe (dooferlad) review: Approve - James Westby (james-w) ------------------------------------------------------------ revno: 387 [merge] committer: James Tunnicliffe branch nick: linaro-image-tools timestamp: Tue 2011-07-26 14:04:10 +0100 message: Merging in temporary workaround for bug #816015. Prevent the indexing of any snapshots that aren't in http://snapshots.linaro.org/11.05-daily/ so the current logic for hwpack and OS selection will not result in finding an OS binary without a hardware pack to go with it. When testing I noticed that I hadn't changed the name of the library FetchImage to fetch_image (without this change the indexer wouldn't work). I also moved the BeautifulSoup import in fetch_image.py to the single function where it is used, so BeautifulSoup is not needed to be installed on the server that is creating the index. modified: linaro-image-indexer linaro_image_tools/fetch_image.py --- lp:linaro-image-tools https://code.launchpad.net/~linaro-image-tools/linaro-image-tools/trunk You are subscribed to branch lp:linaro-image-tools. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-image-tools/linaro-image-tools/trunk/+edit-subscription === modified file 'linaro-image-indexer' --- linaro-image-indexer 2011-07-07 14:15:23 +0000 +++ linaro-image-indexer 2011-07-26 10:56:17 +0000 @@ -25,7 +25,7 @@ import urlparse import logging import bz2 -import linaro_image_tools.FetchImage +import linaro_image_tools.fetch_image RELEASES_WWW_DOCUMENT_ROOT = "/srv/releases.linaro.org/www/platform/" RELEASE_URL = "http://releases.linaro.org/platform/" @@ -41,7 +41,7 @@ def __init__(self): self.reset() self.db_file_name = "server_index" - self.db = linaro_image_tools.FetchImage.DB(self.db_file_name) + self.db = linaro_image_tools.fetch_image.DB(self.db_file_name) def crawl(self): self.db.set_url_parse_info(self.url_parse) @@ -55,17 +55,23 @@ def go(self, root_dir_, root_url_, table_): for root, subFolders, files in os.walk( root_dir_ ): - for file in files: - if(re.search('\.gz$', file)): - # Construct a URL to the file and save in the database - relative_location = re.sub(root_dir_, "", - os.path.join(root, file)) - url = urlparse.urljoin(root_url_, relative_location) - url = urlparse.urljoin(url, file) - - if not re.search('/leb-panda/', url): - logging.info(url) - self.db.record_url(url, table_) + + # --- Temporary hack to work around bug: + # https://bugs.launchpad.net/linaro-image-tools/+bug/816015 + # For the moment we just index platform == 11.05-daily when + # indexing the snapshots server. + if re.search("11.05-daily", root) or re.search("release", table_): + for file in files: + if(re.search('\.gz$', file)): + # Construct a URL to the file and save in the database + relative_location = re.sub(root_dir_, "", + os.path.join(root, file)) + url = urlparse.urljoin(root_url_, relative_location) + url = urlparse.urljoin(url, file) + + if not re.search('/leb-panda/', url): + logging.info(url) + self.db.record_url(url, table_) self.dump() === modified file 'linaro_image_tools/fetch_image.py' --- linaro_image_tools/fetch_image.py 2011-07-22 21:09:29 +0000 +++ linaro_image_tools/fetch_image.py 2011-07-26 10:56:17 +0000 @@ -35,7 +35,6 @@ import datetime import threading import subprocess -import BeautifulSoup import utils @@ -96,6 +95,8 @@ return a directory listing of the directory that url sits in """ + import BeautifulSoup + url = os.path.dirname(url) response = self.urllib2_open(url) page = response.read()