| Message ID | 20220622123305.3971169-1-alex.bennee@linaro.org |
|---|---|
| State | New |
| Headers | show |
| Series | [RFC] gitlab: add a binary build to project registry | expand |
On Wed, 22 Jun 2022 at 13:33, Alex Bennée <alex.bennee@linaro.org> wrote: > > There have been requests from other projects such LKFT to have > "official" docker images with pre-built QEMU binaries. These could > then be consumed by downstream CI systems by pulling directly from the > qemu-project container registry. The final image could then be run by > doing: > > docker run --rm -it \ > registry.gitlab.com/qemu-project/qemu/qemu/debian-amd64-binaries \ > /opt/bin/qemu-system-aarch64 $ARGS > > To keep the build time down we skip user-mode, documents, plugins and > a selection of the more esoteric hardware emulations. Even so it still > takes over an hour to build and install everything. > This is very much an RFC for now as we have traditionally not provided > binaries for our users except for the slightly special case of > Windows. As currently structured this will rebuild the binaries on > every merge but we could change the generation rules to only trigger > for tagged or stable branch pushes. We also wouldn't be testing these > binaries so we are basically assuming they are OK by the nature of > being built from master which in theory never breaks. I'm a bit uncertain about providing "official" docker images or other pre-built binaries, because it might give the impression that these are OK to use with KVM, when in fact they won't necessarily get security fixes in a timely manner, and almost all users of QEMU for KVM purposes are better off with the distro QEMU. thanks -- PMM
Peter Maydell <peter.maydell@linaro.org> writes: > On Wed, 22 Jun 2022 at 13:33, Alex Bennée <alex.bennee@linaro.org> wrote: >> >> There have been requests from other projects such LKFT to have >> "official" docker images with pre-built QEMU binaries. These could >> then be consumed by downstream CI systems by pulling directly from the >> qemu-project container registry. The final image could then be run by >> doing: >> >> docker run --rm -it \ >> registry.gitlab.com/qemu-project/qemu/qemu/debian-amd64-binaries \ >> /opt/bin/qemu-system-aarch64 $ARGS >> >> To keep the build time down we skip user-mode, documents, plugins and >> a selection of the more esoteric hardware emulations. Even so it still >> takes over an hour to build and install everything. > >> This is very much an RFC for now as we have traditionally not provided >> binaries for our users except for the slightly special case of >> Windows. As currently structured this will rebuild the binaries on >> every merge but we could change the generation rules to only trigger >> for tagged or stable branch pushes. We also wouldn't be testing these >> binaries so we are basically assuming they are OK by the nature of >> being built from master which in theory never breaks. > > I'm a bit uncertain about providing "official" docker images > or other pre-built binaries, because it might give the impression > that these are OK to use with KVM, when in fact they won't necessarily > get security fixes in a timely manner, and almost all users of QEMU for > KVM purposes are better off with the distro QEMU. Do we have any sort of idea how long it takes from a CVE being fixed in a distro kernel to the eventual merge of a patch in master? Anyway the main use case for this is emulation where we want to get new features under -cpu max into the CI loops as soon as possible. I think the LKFT guys are testing KVM as well though and again want to see new KVM features as soon as possible. I'm not proposing these images are uploaded to docker.io so less likely to be hit by the default: docker run qemu:latest (they instead get 6 year old things packages by someone calling themselves qemu: https://hub.docker.com/u/qemu) > > thanks > -- PMM
On Wed, Jun 22, 2022 at 03:43:54PM +0100, Alex Bennée wrote: > > Peter Maydell <peter.maydell@linaro.org> writes: > > > On Wed, 22 Jun 2022 at 13:33, Alex Bennée <alex.bennee@linaro.org> wrote: > >> > >> There have been requests from other projects such LKFT to have > >> "official" docker images with pre-built QEMU binaries. These could > >> then be consumed by downstream CI systems by pulling directly from the > >> qemu-project container registry. The final image could then be run by > >> doing: > >> > >> docker run --rm -it \ > >> registry.gitlab.com/qemu-project/qemu/qemu/debian-amd64-binaries \ > >> /opt/bin/qemu-system-aarch64 $ARGS > >> > >> To keep the build time down we skip user-mode, documents, plugins and > >> a selection of the more esoteric hardware emulations. Even so it still > >> takes over an hour to build and install everything. > > > >> This is very much an RFC for now as we have traditionally not provided > >> binaries for our users except for the slightly special case of > >> Windows. As currently structured this will rebuild the binaries on > >> every merge but we could change the generation rules to only trigger > >> for tagged or stable branch pushes. We also wouldn't be testing these > >> binaries so we are basically assuming they are OK by the nature of > >> being built from master which in theory never breaks. > > > > I'm a bit uncertain about providing "official" docker images > > or other pre-built binaries, because it might give the impression > > that these are OK to use with KVM, when in fact they won't necessarily > > get security fixes in a timely manner, and almost all users of QEMU for > > KVM purposes are better off with the distro QEMU. > > Do we have any sort of idea how long it takes from a CVE being fixed in > a distro kernel to the eventual merge of a patch in master? > > Anyway the main use case for this is emulation where we want to get new > features under -cpu max into the CI loops as soon as possible. I think > the LKFT guys are testing KVM as well though and again want to see new > KVM features as soon as possible. > > I'm not proposing these images are uploaded to docker.io so less likely > to be hit by the default: > > docker run qemu:latest > > (they instead get 6 year old things packages by someone calling > themselves qemu: https://hub.docker.com/u/qemu) I guess I would question whether we really need to introduce ourselves as a dependancy of arbitrary downstream projects. We will have little insight into what other projects are doing with any containers we publish. This in turn means our scope for making changes to the containers over time is quite limited, unless we're willing to risk breaking those unknown projects' CI systems. Personally, I feel CI systems are already fragile enough that it is better to have a self contained system as much as possible. This is why, for example, libvirt's integration test suite builds QEMU from git itself. We get exact control over which QEMU features are built, and get to choose which distros we target too. It isn't hard for apps wanting to use QEMU to perform a build: https://gitlab.com/libvirt/libvirt/-/blob/master/ci/integration-template.yml#L5 Probably the hardest part here is understanding the build time package dependancies that are required to be installed, if you want to all QEMU features enabled. We could perhaps use lcitool to record the full set of build dependancies for each distro in git, so 3rd party CI can grab the package list and install it, and thus get the trivial build done. Even better if we publish the package list info in our docs too. I wonder why LKFT isn't building QEMU themselves in the way that they need already ? Would be nice to know what their problems are, rather than just assume that a container build is the solution. With regards, Daniel
On Wed, 22 Jun 2022 at 17:06, Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Wed, Jun 22, 2022 at 03:43:54PM +0100, Alex Bennée wrote: > > > > Peter Maydell <peter.maydell@linaro.org> writes: > > > > > On Wed, 22 Jun 2022 at 13:33, Alex Bennée <alex.bennee@linaro.org> wrote: > > >> > > >> There have been requests from other projects such LKFT to have > > >> "official" docker images with pre-built QEMU binaries. These could > > >> then be consumed by downstream CI systems by pulling directly from the > > >> qemu-project container registry. The final image could then be run by > > >> doing: > > >> > > >> docker run --rm -it \ > > >> registry.gitlab.com/qemu-project/qemu/qemu/debian-amd64-binaries \ > > >> /opt/bin/qemu-system-aarch64 $ARGS > > >> > > >> To keep the build time down we skip user-mode, documents, plugins and > > >> a selection of the more esoteric hardware emulations. Even so it still > > >> takes over an hour to build and install everything. > > > > > >> This is very much an RFC for now as we have traditionally not provided > > >> binaries for our users except for the slightly special case of > > >> Windows. As currently structured this will rebuild the binaries on > > >> every merge but we could change the generation rules to only trigger > > >> for tagged or stable branch pushes. We also wouldn't be testing these > > >> binaries so we are basically assuming they are OK by the nature of > > >> being built from master which in theory never breaks. > > > > > > I'm a bit uncertain about providing "official" docker images > > > or other pre-built binaries, because it might give the impression > > > that these are OK to use with KVM, when in fact they won't necessarily > > > get security fixes in a timely manner, and almost all users of QEMU for > > > KVM purposes are better off with the distro QEMU. > > > > Do we have any sort of idea how long it takes from a CVE being fixed in > > a distro kernel to the eventual merge of a patch in master? > > > > Anyway the main use case for this is emulation where we want to get new > > features under -cpu max into the CI loops as soon as possible. I think > > the LKFT guys are testing KVM as well though and again want to see new > > KVM features as soon as possible. > > > > I'm not proposing these images are uploaded to docker.io so less likely > > to be hit by the default: > > > > docker run qemu:latest > > > > (they instead get 6 year old things packages by someone calling > > themselves qemu: https://hub.docker.com/u/qemu) > > I guess I would question whether we really need to introduce ourselves > as a dependancy of arbitrary downstream projects. We will have little > insight into what other projects are doing with any containers we publish. > This in turn means our scope for making changes to the containers over > time is quite limited, unless we're willing to risk breaking those > unknown projects' CI systems. > > Personally, I feel CI systems are already fragile enough that it is > better to have a self contained system as much as possible. This is why, > for example, libvirt's integration test suite builds QEMU from git > itself. We get exact control over which QEMU features are built, and > get to choose which distros we target too. It isn't hard for apps wanting > to use QEMU to perform a build: > > https://gitlab.com/libvirt/libvirt/-/blob/master/ci/integration-template.yml#L5 > > Probably the hardest part here is understanding the build time package > dependancies that are required to be installed, if you want to all QEMU > features enabled. We could perhaps use lcitool to record the full set of > build dependancies for each distro in git, so 3rd party CI can grab the > package list and install it, and thus get the trivial build done. Even > better if we publish the package list info in our docs too. > > I wonder why LKFT isn't building QEMU themselves in the way that they > need already ? Would be nice to know what their problems are, rather > than just assume that a container build is the solution. Right now we use QEMU from Debian, and we miss new architectural feature's that gets into QEMU. Our plan is to do: 1. running the latest QEMU with all the new architectual feature's. 2. helping out finding regressions in QEMU and/or the kernel, when different kernels are booted with different QEMU versions. If the hub.docker.com/u/qemu namespace could be used to store all the different QEMU builds. With that we can get use of the latest architectural feature's, also tri to find if the kernel regressed or if its QEMU that regressed. Cheers, Anders
diff --git a/.gitlab-ci.d/binary-builds.yml b/.gitlab-ci.d/binary-builds.yml new file mode 100644 index 0000000000..bcb9d57f80 --- /dev/null +++ b/.gitlab-ci.d/binary-builds.yml @@ -0,0 +1,19 @@ +# +# Binary Builds +# +# These are docker images with built QEMU binaries which could form +# the basis of inputs for other projects that want recent QEMU builds +# for their own CI. +# + +include: + - local: '/.gitlab-ci.d/container-template.yml' + +build-binary-debian11: + extends: .container_native_template + needs: + - job: amd64-debian11-container + artifacts: true + timeout: 90m + variables: + NAME: debian-amd64-binaries diff --git a/.gitlab-ci.d/binary-builds/debian-amd64-binaries.docker b/.gitlab-ci.d/binary-builds/debian-amd64-binaries.docker new file mode 100644 index 0000000000..ea0c8fdf5b --- /dev/null +++ b/.gitlab-ci.d/binary-builds/debian-amd64-binaries.docker @@ -0,0 +1,18 @@ +# +# QEMU Binary Builds, in docker form +# +# + +# Build stage, assumes workdir +FROM registry.gitlab.com/qemu-project/qemu/qemu/debian-amd64 +ADD . /src +RUN mkdir -p /src/build +WORKDIR /src/build +RUN ../configure --disable-user --disable-plugins --disable-docs \ + --target-list-exclude=avr-softmmu,cris-softmmu,microblaze-softmmu,microblazeel-softmmu,or1k-softmmu,rx-softmmu,sh4-softmmu,sh4eb-softmmu,tricore-softmmu \ + --prefix=/opt/qemu +RUN ninja +RUN make install + +FROM registry.gitlab.com/qemu-project/qemu/qemu/debian11 +COPY --from=0 /opt/qemu /opt/qemu diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml index c434b9c8f3..be974d74ed 100644 --- a/.gitlab-ci.d/container-template.yml +++ b/.gitlab-ci.d/container-template.yml @@ -1,3 +1,8 @@ +# +# This container job uses QEMU's wrapper scripts to build a container +# which we can use for the basis of other tests. Generally these +# containers setup build environments for building QEMU or its tests. +# .container_job_template: extends: .base_job_template image: docker:stable @@ -20,3 +25,31 @@ - docker push "$TAG" after_script: - docker logout + +# +# This container job uses the native docker tools to build and run and +# container. Here the container may include the build itself which is +# used for things like preparing binary images. +# +.container_native_template: + extends: .base_job_template + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest" + - apk add python3 + - docker info + - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" + script: + - echo "TAG:$TAG" + - echo "COMMON_TAG:$COMMON_TAG" + - DOCKER_BUILDKIT=1 docker build + -t "qemu/$NAME" -f ".gitlab-ci.d/binary-builds/$NAME.docker" . + - docker tag "qemu/$NAME" "$TAG" + - docker push "$TAG" + after_script: + - docker logout + diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml index 691d9bf5dc..ed713f517a 100644 --- a/.gitlab-ci.d/qemu-project.yml +++ b/.gitlab-ci.d/qemu-project.yml @@ -13,3 +13,4 @@ include: - local: '/.gitlab-ci.d/custom-runners.yml' - local: '/.gitlab-ci.d/cirrus.yml' - local: '/.gitlab-ci.d/windows.yml' + - local: '/.gitlab-ci.d/binary-builds.yml'
There have been requests from other projects such LKFT to have "official" docker images with pre-built QEMU binaries. These could then be consumed by downstream CI systems by pulling directly from the qemu-project container registry. The final image could then be run by doing: docker run --rm -it \ registry.gitlab.com/qemu-project/qemu/qemu/debian-amd64-binaries \ /opt/bin/qemu-system-aarch64 $ARGS To keep the build time down we skip user-mode, documents, plugins and a selection of the more esoteric hardware emulations. Even so it still takes over an hour to build and install everything. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Cc: Anders Roxell <anders.roxell@linaro.org> Cc: Remi Duraffort <remi.duraffort@linaro.org> --- This is very much an RFC for now as we have traditionally not provided binaries for our users except for the slightly special case of Windows. As currently structured this will rebuild the binaries on every merge but we could change the generation rules to only trigger for tagged or stable branch pushes. We also wouldn't be testing these binaries so we are basically assuming they are OK by the nature of being built from master which in theory never breaks. The container layout is not ideal either as the QEMU dockerfiles are all based on needing to build things so by their nature include all the devel pacakges. We could get lcitool to spit out a non-dev version with only binary dependencies but that would increase the number of dockerfiles to manage. For now we just re-use the build images for the final installed product. --- .gitlab-ci.d/binary-builds.yml | 19 +++++++++++ .../debian-amd64-binaries.docker | 18 ++++++++++ .gitlab-ci.d/container-template.yml | 33 +++++++++++++++++++ .gitlab-ci.d/qemu-project.yml | 1 + 4 files changed, 71 insertions(+) create mode 100644 .gitlab-ci.d/binary-builds.yml create mode 100644 .gitlab-ci.d/binary-builds/debian-amd64-binaries.docker