From patchwork Wed Jun 22 12:33:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 583853 Delivered-To: patch@linaro.org Received: by 2002:a5d:4bc2:0:0:0:0:0 with SMTP id l2csp1715694wrt; Wed, 22 Jun 2022 05:34:22 -0700 (PDT) X-Google-Smtp-Source: AGRyM1uE6urk6BQ+7lMTgEWP0ENfENKUQreXU9iTC0NLCnYIH8qZVkiEXccbV57ko7W/vsNSLqBz X-Received: by 2002:a05:622a:647:b0:306:6b30:bd0a with SMTP id a7-20020a05622a064700b003066b30bd0amr2626992qtb.327.1655901262600; Wed, 22 Jun 2022 05:34:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1655901262; cv=none; d=google.com; s=arc-20160816; b=kefXmCRUO1E7Q7tTs1qSM0foK5yVzIE12SfirhGs5+r+ifLHIc+28R6Eajb5+xc5HE yHUDHBe+29L/lDyZjbAM6fulNf3VaDg4a9nwd6/lRRmw+i1/DURMD7HS6cfN8B5aPzT/ xOeKoC6j8VP83cKxQfP7VDFFhZ8TYIdoIwfEHJDnKdoZdtAGO77iOOfh+1FHPadW12bO cVJe1uem5FHIDuOHdkFr4xltdCk5Qu0VIeiZZKRxXzm50ASUu0pQ0T2Cg1clvCOahtf1 bdJOCSxVydrSeY0kQT2oZWmyD7exRy+qht5h1UuORmWuT84Ev8gLNsOwABSjrJTpoiNR XJww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:message-id:date:subject:cc:to:from:dkim-signature; bh=To38TAhFPTyVe2LNjyAgXAUuxpNNahrHHcAaKJFoisQ=; b=ugzov45iuF8TMQ3ZZmlcvfmEAvsOOQ2kw5AAXdrmSnWBRVsjzSAPSYPc+HIf/o3ekV 5rYeZTdlHEpC75/htkd5XKsJFFZURshWdrOd2vWMR0LH7WqQqIFZ/pENreImDmNsODpx S2/jCd31gVpp0Zz9IqR63WdbF8dhVJAv7ftjk4+U7jX31whOMyzXqTS3TMJe/5AAePJy TBiR6SBRIRn+XkioxJJGy/6IKtRzO1OW3ss2UJ2GkJRqG7kEtlPER6Pz+ksDg0ESvVeg sJLhmwDcm/1/Cc0aInban+AgBYE4hW7cYFO+jEbFflixNNoTVMlj5NzDwfoeTVENzbrm tO+Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ieu1Nnxw; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id j4-20020a37c244000000b006a41413e2e7si9448813qkm.564.2022.06.22.05.34.22 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 22 Jun 2022 05:34:22 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ieu1Nnxw; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:59464 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o3zYs-0006Bc-3u for patch@linaro.org; Wed, 22 Jun 2022 08:34:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58018) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o3zY0-0006BF-1A for qemu-devel@nongnu.org; Wed, 22 Jun 2022 08:33:28 -0400 Received: from mail-ej1-x634.google.com ([2a00:1450:4864:20::634]:39723) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1o3zXv-0001aV-Bs for qemu-devel@nongnu.org; Wed, 22 Jun 2022 08:33:24 -0400 Received: by mail-ej1-x634.google.com with SMTP id ay16so14542651ejb.6 for ; Wed, 22 Jun 2022 05:33:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=To38TAhFPTyVe2LNjyAgXAUuxpNNahrHHcAaKJFoisQ=; b=ieu1NnxwxW7YEfutjkUvpqUR8NDRtLqZfGKU6dbjofRhRdamde4zQ47naygdSUJIO6 8AMnNHzRVBYPNgjmmki/lB0t64o+LtscxKy7tGOUJk5cYS+o9tfM0GJDVD/mB8l/fIeK h9pMoiCgKkfAn/rV2SG2JyqQVsO0PW6GwFyM1dS5CKaJ6MylpDVyb6WbqiB81O0Yg2z+ QDnTYCFPrEDIhJ2P+kbBRNVtbrU9Y0fcpi5RuKKqlEbOiPXWiJ5v11RKhzIxh8dUOjdC 5LIRfoAsG+zOyMBIPb5vB5sA6wdw4em6EMuvPA5Tug3Vj2xY6Ci7WpwHGEY+fUvRXHdh hnvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=To38TAhFPTyVe2LNjyAgXAUuxpNNahrHHcAaKJFoisQ=; b=ZgIX9eTaRLJRQVBrYrNzK80o+WLf9Bm/uy+V9DnGqoba/vXB1EHOoVVij9f4cDYHan IE4aKm7jaCcWXBm4JZ1g65glV3Oaywaa8NPg1i0JJj7wr1K2FkDwkjyXsyI+mdDl6kfG JwnOJpvg66P6m1FJSgxJ4Ct5QwfD9wSPU9BXrUz7iNFDai1XJZg4loXZJUVcZeW2VPnK OHzVBX2Wl886VcEmL4pSvBVWM7IkbmrNYutk6076PjeUFebPX+VXDO8tWn8UPLMDqlTZ cF/xap3tBIXcqHhetFMjtRvrDe1MDuLERJMn7VG/OUNC4dJLln1xWLcnCkbcIENKD7vp MCwQ== X-Gm-Message-State: AJIora+hbYLYUbCJmzTdof8DcT0av8eIqZpjB39oofN9ku9d6frGsRkq pcUFGy9a0RCwrhkAzPu0N7Fx0A== X-Received: by 2002:a17:906:b150:b0:711:c6a5:c5c1 with SMTP id bt16-20020a170906b15000b00711c6a5c5c1mr2977923ejb.177.1655901201689; Wed, 22 Jun 2022 05:33:21 -0700 (PDT) Received: from zen.linaroharston ([51.148.130.216]) by smtp.gmail.com with ESMTPSA id me26-20020a170906aeda00b00722e5b234basm2427756ejb.179.2022.06.22.05.33.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 05:33:20 -0700 (PDT) Received: from zen.lan (localhost [127.0.0.1]) by zen.linaroharston (Postfix) with ESMTP id 1FF021FFB7; Wed, 22 Jun 2022 13:33:20 +0100 (BST) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: qemu-devel@nongnu.org Cc: peter.maydell@linaro.org, =?utf-8?q?Alex_Benn=C3=A9e?= , Anders Roxell , Remi Duraffort , =?utf-8?q?Philippe_Mathieu-Daud?= =?utf-8?q?=C3=A9?= , Thomas Huth , Wainer dos Santos Moschetta , Beraldo Leal Subject: [RFC PATCH] gitlab: add a binary build to project registry Date: Wed, 22 Jun 2022 13:33:05 +0100 Message-Id: <20220622123305.3971169-1-alex.bennee@linaro.org> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::634; envelope-from=alex.bennee@linaro.org; helo=mail-ej1-x634.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" There have been requests from other projects such LKFT to have "official" docker images with pre-built QEMU binaries. These could then be consumed by downstream CI systems by pulling directly from the qemu-project container registry. The final image could then be run by doing: docker run --rm -it \ registry.gitlab.com/qemu-project/qemu/qemu/debian-amd64-binaries \ /opt/bin/qemu-system-aarch64 $ARGS To keep the build time down we skip user-mode, documents, plugins and a selection of the more esoteric hardware emulations. Even so it still takes over an hour to build and install everything. Signed-off-by: Alex Bennée Cc: Anders Roxell Cc: Remi Duraffort --- This is very much an RFC for now as we have traditionally not provided binaries for our users except for the slightly special case of Windows. As currently structured this will rebuild the binaries on every merge but we could change the generation rules to only trigger for tagged or stable branch pushes. We also wouldn't be testing these binaries so we are basically assuming they are OK by the nature of being built from master which in theory never breaks. The container layout is not ideal either as the QEMU dockerfiles are all based on needing to build things so by their nature include all the devel pacakges. We could get lcitool to spit out a non-dev version with only binary dependencies but that would increase the number of dockerfiles to manage. For now we just re-use the build images for the final installed product. --- .gitlab-ci.d/binary-builds.yml | 19 +++++++++++ .../debian-amd64-binaries.docker | 18 ++++++++++ .gitlab-ci.d/container-template.yml | 33 +++++++++++++++++++ .gitlab-ci.d/qemu-project.yml | 1 + 4 files changed, 71 insertions(+) create mode 100644 .gitlab-ci.d/binary-builds.yml create mode 100644 .gitlab-ci.d/binary-builds/debian-amd64-binaries.docker diff --git a/.gitlab-ci.d/binary-builds.yml b/.gitlab-ci.d/binary-builds.yml new file mode 100644 index 0000000000..bcb9d57f80 --- /dev/null +++ b/.gitlab-ci.d/binary-builds.yml @@ -0,0 +1,19 @@ +# +# Binary Builds +# +# These are docker images with built QEMU binaries which could form +# the basis of inputs for other projects that want recent QEMU builds +# for their own CI. +# + +include: + - local: '/.gitlab-ci.d/container-template.yml' + +build-binary-debian11: + extends: .container_native_template + needs: + - job: amd64-debian11-container + artifacts: true + timeout: 90m + variables: + NAME: debian-amd64-binaries diff --git a/.gitlab-ci.d/binary-builds/debian-amd64-binaries.docker b/.gitlab-ci.d/binary-builds/debian-amd64-binaries.docker new file mode 100644 index 0000000000..ea0c8fdf5b --- /dev/null +++ b/.gitlab-ci.d/binary-builds/debian-amd64-binaries.docker @@ -0,0 +1,18 @@ +# +# QEMU Binary Builds, in docker form +# +# + +# Build stage, assumes workdir +FROM registry.gitlab.com/qemu-project/qemu/qemu/debian-amd64 +ADD . /src +RUN mkdir -p /src/build +WORKDIR /src/build +RUN ../configure --disable-user --disable-plugins --disable-docs \ + --target-list-exclude=avr-softmmu,cris-softmmu,microblaze-softmmu,microblazeel-softmmu,or1k-softmmu,rx-softmmu,sh4-softmmu,sh4eb-softmmu,tricore-softmmu \ + --prefix=/opt/qemu +RUN ninja +RUN make install + +FROM registry.gitlab.com/qemu-project/qemu/qemu/debian11 +COPY --from=0 /opt/qemu /opt/qemu diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml index c434b9c8f3..be974d74ed 100644 --- a/.gitlab-ci.d/container-template.yml +++ b/.gitlab-ci.d/container-template.yml @@ -1,3 +1,8 @@ +# +# This container job uses QEMU's wrapper scripts to build a container +# which we can use for the basis of other tests. Generally these +# containers setup build environments for building QEMU or its tests. +# .container_job_template: extends: .base_job_template image: docker:stable @@ -20,3 +25,31 @@ - docker push "$TAG" after_script: - docker logout + +# +# This container job uses the native docker tools to build and run and +# container. Here the container may include the build itself which is +# used for things like preparing binary images. +# +.container_native_template: + extends: .base_job_template + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest" + - apk add python3 + - docker info + - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" + script: + - echo "TAG:$TAG" + - echo "COMMON_TAG:$COMMON_TAG" + - DOCKER_BUILDKIT=1 docker build + -t "qemu/$NAME" -f ".gitlab-ci.d/binary-builds/$NAME.docker" . + - docker tag "qemu/$NAME" "$TAG" + - docker push "$TAG" + after_script: + - docker logout + diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml index 691d9bf5dc..ed713f517a 100644 --- a/.gitlab-ci.d/qemu-project.yml +++ b/.gitlab-ci.d/qemu-project.yml @@ -13,3 +13,4 @@ include: - local: '/.gitlab-ci.d/custom-runners.yml' - local: '/.gitlab-ci.d/cirrus.yml' - local: '/.gitlab-ci.d/windows.yml' + - local: '/.gitlab-ci.d/binary-builds.yml'