From patchwork Tue Jul 14 09:52:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 235518 Delivered-To: patch@linaro.org Received: by 2002:a92:d244:0:0:0:0:0 with SMTP id v4csp3583515ilg; Tue, 14 Jul 2020 02:54:54 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx/Q65rrLsv035RKBYZg3F0tZluw43QlZ+yEkBZAM9vEuntShrfDRUfoT/4/TGmN5klrZ3r X-Received: by 2002:a25:3b16:: with SMTP id i22mr6593125yba.66.1594720494569; Tue, 14 Jul 2020 02:54:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1594720494; cv=none; d=google.com; s=arc-20160816; b=txBCmceEfh00aqr9NJRd5dgb1D5xeasyFiNC89EDDEO7hy+mY+8y1Lt+93kPNH92jD ljKa6YC5bccT2j9sElCQn7HrX1CBhj4jvtcGACSTA+ex7dA5/dgn+15mUsQZtrJQo0d7 X3PjYeXsvt7DLnYAhhKwLJNffrFO+7Bsut/iJdmw9f4egBqtH/BQjT3NAL79ZfSqWOxq /vb58I+HAztefRPINpFYJ1Xyb18Vmaf5E1qFaklbe66iJgPv/X7OpsyDd/EoQW6BEMQO wFBCj7kClrRY9mi/N92w3t29o88MsTfueVTiA7UJ/fFaKGOzQHhZywcDaRVdmvQ9pr1m ypxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=nLGL97rROnaKpdqPQWyvMY+ljMO7k5wyI/22+OUtb64=; b=BlvxYHki7SwHUXNLpiS13cVaV2rATWVbtHDd0KRAt/nxTHiK3fW97uJBOhisXdjELd OAslLrNSEUTEMka8tdl+zQEnqIOnIUrYcF8Dr0aO+//VrKMYfFwiOeiD8GMt6b/tz7Gj tXVr5pJtK1HGf6EIO5RHNgrSYBqIwQz9rtCOK+DrDrjRyeBAvF8WvK47b9AqVezJYvR6 +0JedTwWQmmtV2eFg+d2CVYmy2WuKPx0goanP2oh5ZDxprbUCFUPaxRRRJS6O/7Njzrf 0oeXzwRnAe6lUD6jiaRHUjYtzD6RYv2s3VJNUpWw9FkxoCHV8iR7UPcPJ8wG7GVxiJdE gNdw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=GNX5au0T; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id t5si17397031ybp.39.2020.07.14.02.54.54 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 14 Jul 2020 02:54:54 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=GNX5au0T; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:60472 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jvHeI-0006pq-2P for patch@linaro.org; Tue, 14 Jul 2020 05:54:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58412) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jvHcP-0003QK-W6 for qemu-devel@nongnu.org; Tue, 14 Jul 2020 05:52:58 -0400 Received: from mail-wm1-x32d.google.com ([2a00:1450:4864:20::32d]:39891) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jvHcO-0003x7-CW for qemu-devel@nongnu.org; Tue, 14 Jul 2020 05:52:57 -0400 Received: by mail-wm1-x32d.google.com with SMTP id w3so4417325wmi.4 for ; Tue, 14 Jul 2020 02:52:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nLGL97rROnaKpdqPQWyvMY+ljMO7k5wyI/22+OUtb64=; b=GNX5au0Tkuw84VNUoSuRuStPXMs6wyxI01mUHNUAFw8dw8o+FM0pEx4ZYhjw6jktjW ICcnLfubwn0kIHZY90ONDs2v2smKsUMmZgv+bNKLaIjNESmIHNNNN++qQwLPTtFvAeBl 8jNhuuPkjOhNfxpSjwqZe7apRVSvQ47bVEQ8xccttKsG10hVL7RKmCDLZQw64V6gNhZK q8ghLehNHyUMK1UqYVGSeQ9WihpK65Fd8jVtPwdq87PO6Qwbkm3XsyfWchwmEaDccegR Eki28Bwi77GNWKHSnvGR7lydb9INM5LxiqzTJ4fLhtx+3eEzGgysoU+X81iSagVF9tI6 l5Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nLGL97rROnaKpdqPQWyvMY+ljMO7k5wyI/22+OUtb64=; b=dzz25PwkVSFhsYc+YR/n/8zl+k0cEb3RmDufCayuGtKVCnMIcIVWrwBxAbyuk6/zzv LFlANbUz9XyBLaheWz6a+wXAyAiILPRpm13XdNDrVj2bqFD5LUVtNJ5PJ2vkdA+WMjTm wwxbAHiM7LvgJ6LTHGKT85bIatVesT2WzdFWMVSc9U3RaR+BxtesvSliA/FVyWg3wORX xEquKQTy4YwILCn37rY9PyfiWtxYSouGGSs5hcbUqnaddTNq9K5ZBAVO7Qhmxmy0A6xl roBTeUIGkgT9sw9cbkuraEaQkCvw4PQBcVgnVnLj1wX2bidZCEEwUVgDdkQUm9RQyBzD cGgw== X-Gm-Message-State: AOAM531i8eM0t3kbIzAvso4+4tSeyLsmam/H7NvTTZMud6Mg4c0iYNhK YXi6wEDw5wpQ5Q14CxAuTcscVw== X-Received: by 2002:a7b:c09a:: with SMTP id r26mr3391769wmh.176.1594720374959; Tue, 14 Jul 2020 02:52:54 -0700 (PDT) Received: from zen.linaroharston ([51.148.130.216]) by smtp.gmail.com with ESMTPSA id x1sm27330282wrp.10.2020.07.14.02.52.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jul 2020 02:52:52 -0700 (PDT) Received: from zen.lan (localhost [127.0.0.1]) by zen.linaroharston (Postfix) with ESMTP id A53A41FF90; Tue, 14 Jul 2020 10:52:47 +0100 (BST) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: peter.maydell@linaro.org Subject: [PULL 4/9] docker.py: fix fetching of FROM layers Date: Tue, 14 Jul 2020 10:52:42 +0100 Message-Id: <20200714095247.19573-5-alex.bennee@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200714095247.19573-1-alex.bennee@linaro.org> References: <20200714095247.19573-1-alex.bennee@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::32d; envelope-from=alex.bennee@linaro.org; helo=mail-wm1-x32d.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , =?utf-8?q?Alex_Benn=C3=A9e?= , qemu-devel@nongnu.org, =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" This worked on a system that was already bootstrapped because the stage 2 images already existed even if they wouldn't be used. What we should have pulled down was the FROM line containers first because building on gitlab doesn't have the advantage of using our build system to build the pre-requisite bits. We still pull the image we want to build just in case we can use the cached data. Signed-off-by: Alex Bennée Reviewed-by: Daniel P. Berrangé Tested-by: Philippe Mathieu-Daudé Message-Id: <20200713200415.26214-4-alex.bennee@linaro.org> -- 2.20.1 diff --git a/tests/docker/docker.py b/tests/docker/docker.py index 2d67bbd15a..c9f20d8d09 100755 --- a/tests/docker/docker.py +++ b/tests/docker/docker.py @@ -306,14 +306,18 @@ class Docker(object): checksum = _text_checksum(_dockerfile_preprocess(dockerfile)) if registry is not None: - # see if we can fetch a cache copy, may fail... - pull_args = ["pull", "%s/%s" % (registry, tag)] - if self._do(pull_args, quiet=quiet) == 0: + sources = re.findall("FROM qemu\/(.*)", dockerfile) + # Fetch any cache layers we can, may fail + for s in sources: + pull_args = ["pull", "%s/qemu/%s" % (registry, s)] + if self._do(pull_args, quiet=quiet) != 0: + registry = None + break + # Make substitutions + if registry is not None: dockerfile = dockerfile.replace("FROM qemu/", "FROM %s/qemu/" % (registry)) - else: - registry = None tmp_df = tempfile.NamedTemporaryFile(mode="w+t", encoding='utf-8', @@ -339,6 +343,8 @@ class Docker(object): build_args += ["--build-arg", "BUILDKIT_INLINE_CACHE=1"] if registry is not None: + pull_args = ["pull", "%s/%s" % (registry, tag)] + self._do(pull_args, quiet=quiet) cache = "%s/%s" % (registry, tag) build_args += ["--cache-from", cache] build_args += argv