| Message ID | 20200709024657.2500558-3-crosa@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | None | expand |
On Thu, Jul 09, 2020 at 12:07:32PM +0200, Philippe Mathieu-Daudé wrote: > On 7/9/20 4:46 AM, Cleber Rosa wrote: > > This is a mapping of Peter's "remake-merge-builds" and > > "pull-buildtest" scripts, gone through some updates, adding some build > > option and removing others. > > > > The jobs currently cover the machines that the QEMU project owns, and that > > are setup and ready to run jobs: > > > > - Ubuntu 18.04 on S390x > > - Ubuntu 20.04 on aarch64 > > > > During the development of this set of jobs, the GitLab CI was tested > > with many other architectures, including ppc64, s390x and aarch64, > > along with the other OSs (not included here): > > > > - Fedora 30 > > - FreeBSD 12.1 > > > > More information can be found in the documentation itself. > > > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > > --- > > .gitlab-ci.d/gating.yml | 146 +++++++++++++++++ > > .gitlab-ci.yml | 1 + > > docs/devel/testing.rst | 147 +++++++++++++++++ > > Time to consider moving the CI doc in a separate file... > > > scripts/ci/setup/build-environment.yml | 217 +++++++++++++++++++++++++ > > scripts/ci/setup/gitlab-runner.yml | 72 ++++++++ > > scripts/ci/setup/inventory | 2 + > > scripts/ci/setup/vars.yml | 13 ++ > > Should we name these last two as inventory.template > and vars.yml.template? > I get your point, and actually like your suggestion but I've not seen this done on any Ansible based projects I've come across. The inventory file, for instance, is tends to be this "localhost" version pretty much everywhere. The same goes for vars.yml, so I'm inclined to leave them as is... but if you have strong feelings against it, I won't fuss nor fight. > Maybe you can add them with gitlab-runner.yml and a part of > the documentation in a first patch, > > Then build-environment.yml and another part of the doc > in another patch, > > Finally gating.yml and the related missing doc as the > last patch? > Sounds good. This is indeed a very large patch as it is. Thanks! - Cleber.
On Thu, Jul 09, 2020 at 11:30:29AM +0100, Daniel P. Berrangé wrote: > On Wed, Jul 08, 2020 at 10:46:57PM -0400, Cleber Rosa wrote: > > This is a mapping of Peter's "remake-merge-builds" and > > "pull-buildtest" scripts, gone through some updates, adding some build > > option and removing others. > > > > The jobs currently cover the machines that the QEMU project owns, and that > > are setup and ready to run jobs: > > > > - Ubuntu 18.04 on S390x > > - Ubuntu 20.04 on aarch64 > > > > During the development of this set of jobs, the GitLab CI was tested > > with many other architectures, including ppc64, s390x and aarch64, > > along with the other OSs (not included here): > > > > - Fedora 30 > > - FreeBSD 12.1 > > > > More information can be found in the documentation itself. > > > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > > --- > > .gitlab-ci.d/gating.yml | 146 +++++++++++++++++ > > AFAIK, the jobs in this file just augment what is already defined > in the main .gitlab-ci.yml. Also since we're providing setup info > for other people to configure custom runners, these jobs are usable > for non-gating CI scenarios too. > If you mean that they introduced new jobs, you're right. > IOW, the jobs in this file happen to be usable for gating, but they > are not the only gating jobs, and can be used for non-gating reasons. > Right, I do not doubt these jobs may be useful to other people and on scenarios other than "before merging a patch series". > This is a complicated way of saying that gating.yml is not a desirable > filename, so I'd suggest splitting it in two and having these files > named based on what their contents is, rather than their use case: > > .gitlab-ci.d/runners-s390x.yml > .gitlab-ci.d/runners-aarch64.yml > > The existing jobs in .gitlab-ci.yml could possibly be moved into > a .gitlab-ci.d/runners-shared.yml file for consistency. > Do you imply that every gitlab CI job should be a gating job? And that the same jobs should be used when other people with their own forks? I find this problematic because: * It would trigger pipelines with jobs that, unless every user has the same runners configured, would have unfulfilled jobs that don't have a matching hardware. * It dilutes the idea that those jobs are inherently different with regards to the management of their infrastructure. * It destroys the notion of layered testing, for whatever people find that worth it, where a faster turnaround could/would be possible with fewer jobs for every push, and many more jobs before a merge. Finally, I find the split by runner architecture you suggested problematic because different organizations may have jobs for the same architecture. I believe that files for different organizations may be a better organization instead. Entries in the MAINTAINERS are one example where the grouping by architecture may not be optimal. > > .gitlab-ci.yml | 1 + > > docs/devel/testing.rst | 147 +++++++++++++++++ > > scripts/ci/setup/build-environment.yml | 217 +++++++++++++++++++++++++ > > scripts/ci/setup/gitlab-runner.yml | 72 ++++++++ > > scripts/ci/setup/inventory | 2 + > > scripts/ci/setup/vars.yml | 13 ++ > > 7 files changed, 598 insertions(+) > > create mode 100644 .gitlab-ci.d/gating.yml > > create mode 100644 scripts/ci/setup/build-environment.yml > > create mode 100644 scripts/ci/setup/gitlab-runner.yml > > create mode 100644 scripts/ci/setup/inventory > > create mode 100644 scripts/ci/setup/vars.yml > > > > diff --git a/.gitlab-ci.d/gating.yml b/.gitlab-ci.d/gating.yml > > new file mode 100644 > > index 0000000000..5562df5708 > > --- /dev/null > > +++ b/.gitlab-ci.d/gating.yml > > @@ -0,0 +1,146 @@ > > +variables: > > + GIT_SUBMODULE_STRATEGY: recursive > > + > > +# All ubuntu-18.04 jobs should run successfully in an environment > > +# setup by the scripts/ci/setup/build-environment.yml task > > +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04" > > +ubuntu-18.04-s390x-all-linux-static: > > + tags: > > + - ubuntu_18.04 > > + - s390x > > + rules: > > + - if: '$CI_COMMIT_REF_NAME == "staging"' > > I think I'd make it more flexible in particular to allow multiple > branches. For example I have multiple subsystems and have separate > branches for each. > > This could be as simple as allowing a regex prefix > > - if: '$CI_COMMIT_REF_NAME =~ /^staging/' > > > > > > diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml > > new file mode 100644 > > index 0000000000..89b35386c7 > > --- /dev/null > > +++ b/scripts/ci/setup/build-environment.yml > > @@ -0,0 +1,217 @@ > > +--- > > +- name: Installation of basic packages to build QEMU > > + hosts: all > > + vars_files: > > + - vars.yml > > + tasks: > > + - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04 > > + apt: > > + update_cache: yes > > + # This matches the packages on tests/docker/Dockerfiles/ubuntu1804.docker > > I'd be inclined to actually use docker on the custom runners. > > eg. instead of having separate physical machines or VMs for each > (distro, arch) pair, have a single host distro for the arch. Then > use docker to provide the build environment against each distro. > > IOW, a RHEL-8 aarch64 host, running docker for ubuntu18.04, fedora30 > etc. > I've come across so many caveats and corner cases that having the lowest common denominator proved to be the smart and sane thing to do. For instance, building on the example you gave, running a RHEL 8 aarch64 host is a NO GO Today because RHEL 8 doesn't ship with docker and the gitlab runner needs to be taught to talk to, say, Podman. gitlab-runner-helper images for different architectures, used to prepare the docker images before running jobs, also proved to be a challenge. Finally, it's going to be very important for some organizations to run tests outside of container environments. For instance, Red Hat needs to run QEMU+KVM tests on bare metal and on VMs, in addition to containers. > That way we don't end up duplicating all these packages, and instead > can use tests/docker/Dockerfiles/ubuntu1804.docker. This ensures > that if a user needs to reproduce a build failure on their own local > aarch64 machine, they can run docker and get the exact same build > architecture. > Like I explained before, containers-only won't cut it. So, we need tooling that is environment agnostic. So far, ansible playbooks seem to be a reasonable solution. But duplicating information bothers me as much as it seems to bother you, so we need to engage in common tooling that is capable of generating those container environments, but not be limited to it. One example of a tool that seems to be a good candidate is "Libvirt's" lcitool. > It also has the benefit that we don't need to worry about how to > setup gitlab runners for every distro we care about. We only need to > do gitlab runner for the standard host distro, which spawns a pristine > throwaway docker env. > > I appreciate this is a big change from what you've done in this patch > though, so don't consider this comment a blocker for initial merge. > I think we should do this as the long term strategy though. Essentially > for Linux builds, everything should always be container based. > Thanks for taking the time to give this feedback, and for making me check again the premises that led to the proposal of this design. Hopefully, this will be an improvement over the current state of "pre-merge" testing, and at the same time can be the first step towards a scenario that has more of the characteristics you pointed out. Best! - Cleber. > Regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Jul 29, 2020 at 11:16:29AM +0100, Stefan Hajnoczi wrote: > On Wed, Jul 08, 2020 at 10:46:57PM -0400, Cleber Rosa wrote: > > Awesome, thanks for creating this stuff! Minor suggestions: > > > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst > > index c1ff24370b..f8dab788ea 100644 > > --- a/docs/devel/testing.rst > > +++ b/docs/devel/testing.rst > > @@ -1003,3 +1003,150 @@ exercise as many corner cases as possible. It is a useful test suite > > to run to exercise QEMU's linux-user code:: > > > > https://linux-test-project.github.io/ > > + > > +CI > > +== > > + > > +QEMU has configurations enabled for a number of different CI services. > > +The most update information about them and their status can be found > > +at:: > > + > > + https://wiki.qemu.org/Testing/CI > > + > > +Gating CI > > +---------- > > + > > +A Pull Requests will only to be merged if they successfully go through > > +a different set of CI jobs. GitLab's CI is the service/framework used > > s/A Pull Requests/Pull Requests/ > s/will only to be merged/will only be merged/ > > I suggest simplifying the first sentence: > > Code is only merged after passing the "gating" set of CI jobs. > > Whether they are called Pull Requests or Merge Requests shouldn't matter > :). > Yep, makes sense. > > +for executing the gating jobs. > > + > > +The architecture of GitLab's CI service allows different machines to be > > +setup with GitLab's "agent", called gitlab-runner, which will take care > > s/setup/set up/ throughout this document > https://grammarist.com/spelling/set-up-vs-setup/ > Nice catch, thanks. > > +of running jobs created by events such as a push to a branch. > > + > > +Even though gitlab-runner can execute jobs on environments such as > > +containers, this initial implementation assumes the shell executor is > > +used, effectively running jobs on the same machine (be them physical > > s/them/they/ > > > +or virtual) the gitlab-runner agent is running. This means those > > s/the/where the/ > Right, thanks. > > +machines must be setup in advance, with the requirements matching the > > +jobs expected to be executed there. > > + > > +Machine configuration for gating jobs > > +------------------------------------- > > + > > +The GitLab's CI architecture allows different parties to provide > > +different machines that will run different jobs. At this point, QEMU > > +will deploy a limited set of machines and jobs. Documentation and/or > > +scripts to setup those machines is located under:: > > + > > + scripts/ci/setup > > + > > +Ansible playbooks have been provided to perform two different tasks > > +related to setting gitlab-runner and the build environment. > > s/setting/setting up/ > > > + > > +Other organizations involved in QEMU development may, in the near > > +future, contribute their own setup documentation/scripts under > > Comments about relative time lack context in a long-lived document like > this one: > s/in the near future// > You're right. That was already bothering but I couldn't tell why. > > diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml > > new file mode 100644 > > index 0000000000..89b35386c7 > > --- /dev/null > > +++ b/scripts/ci/setup/build-environment.yml > > @@ -0,0 +1,217 @@ > > +--- > > +- name: Installation of basic packages to build QEMU > > + hosts: all > > + vars_files: > > + - vars.yml > > + tasks: > > + - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04 > > + apt: > > + update_cache: yes > > + # This matches the packages on tests/docker/Dockerfiles/ubuntu1804.docker > > These comments will not age well :). If you really want to leave a note > then I suggest "Originally from > tests/docker/Dockerfiles/ubuntu1804.docker". > > > diff --git a/scripts/ci/setup/inventory b/scripts/ci/setup/inventory > > new file mode 100644 > > index 0000000000..8bb7ba6b33 > > --- /dev/null > > +++ b/scripts/ci/setup/inventory > > @@ -0,0 +1,2 @@ > > +[local] > > +localhost > > diff --git a/scripts/ci/setup/vars.yml b/scripts/ci/setup/vars.yml > > Perhaps this file can be called vars.yml.template and an entry for > vars.yml can be added to .gitignore. A file that needs local editing > should not be commited to git in-place. Otherwise it's easy to > accidentally commit the local changes to git (and expose the private > GitLab token!). Right... Philippe has already suggested this, and you've definitely increased its significance with the data leak example. So yes, let's do this rename. Thanks for the feedback and suggestions! - Cleber.
On Thu, Sep 03, 2020 at 08:11:39PM -0400, Cleber Rosa wrote: > On Thu, Jul 09, 2020 at 11:30:29AM +0100, Daniel P. Berrangé wrote: > > On Wed, Jul 08, 2020 at 10:46:57PM -0400, Cleber Rosa wrote: > > > This is a mapping of Peter's "remake-merge-builds" and > > > "pull-buildtest" scripts, gone through some updates, adding some build > > > option and removing others. > > > > > > The jobs currently cover the machines that the QEMU project owns, and that > > > are setup and ready to run jobs: > > > > > > - Ubuntu 18.04 on S390x > > > - Ubuntu 20.04 on aarch64 > > > > > > During the development of this set of jobs, the GitLab CI was tested > > > with many other architectures, including ppc64, s390x and aarch64, > > > along with the other OSs (not included here): > > > > > > - Fedora 30 > > > - FreeBSD 12.1 > > > > > > More information can be found in the documentation itself. > > > > > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > > > --- > > > .gitlab-ci.d/gating.yml | 146 +++++++++++++++++ > > > > AFAIK, the jobs in this file just augment what is already defined > > in the main .gitlab-ci.yml. Also since we're providing setup info > > for other people to configure custom runners, these jobs are usable > > for non-gating CI scenarios too. > > > > If you mean that they introduced new jobs, you're right. > > > IOW, the jobs in this file happen to be usable for gating, but they > > are not the only gating jobs, and can be used for non-gating reasons. > > > > Right, I do not doubt these jobs may be useful to other people and on > scenarios other than "before merging a patch series". > > > This is a complicated way of saying that gating.yml is not a desirable > > filename, so I'd suggest splitting it in two and having these files > > named based on what their contents is, rather than their use case: > > > > .gitlab-ci.d/runners-s390x.yml > > .gitlab-ci.d/runners-aarch64.yml > > > > The existing jobs in .gitlab-ci.yml could possibly be moved into > > a .gitlab-ci.d/runners-shared.yml file for consistency. > > > > Do you imply that every gitlab CI job should be a gating job? And > that the same jobs should be used when other people with their own > forks? I find this problematic because: > > * It would trigger pipelines with jobs that, unless every user has the > same runners configured, would have unfulfilled jobs that don't have > a matching hardware. Jobs that require a custom runner should not be set to run by default, but individual contributors must absolutely be able to opt-in to running those jobs simply by registering a runner on their account. > * It dilutes the idea that those jobs are inherently different with > regards to the management of their infrastructure. I don't really know what yiu mean here, but "Inherantly different" does not sound like a desirable property. > * It destroys the notion of layered testing, for whatever people find > that worth it, where a faster turnaround could/would be possible > with fewer jobs for every push, and many more jobs before a merge. The key goal of CI is to reduce the burden on maintainers. The biggest cost is if we merge code and failure is noticed after merge. IT is still a large cost, however, if Peter only finds a CI failure when he attempts the pre-merge test. He has to throw out the pull request putting more work on the subsystem maintainer. The subsystem maintainer may have to throw it back to the original author. The ideal scenario that we need to strive towards is that the original author has tested their code with 100% coverage of all the CI jobs QMEU has defined. Any time there is a job that is not run by authors, but only by the maintainers, we are putting increased burden on the maintainers, so must be minimize that. IOW, layered testing is not desirable as goal. Rather layered testing is just a default setup, but we'd encourage contributors to run the full set of CI jobs, especially if they are frequent contributors. The more they run themselves, the less burden on subsystem maintainers and Peter, and thus the better we all scale. > Finally, I find the split by runner architecture you suggested > problematic because different organizations may have jobs for the same > architecture. I believe that files for different organizations may be > a better organization instead. Entries in the MAINTAINERS are one > example where the grouping by architecture may not be optimal. I don't think we should be structuring jobs around organizations. We should be defining a set of desired jobs we wish to be able to run. Any organization can bring a runner that is capable of running the jobs and donate it to the QEMU project for our formal CI runner The organization is not defining the job though - QEMU is defining the jobs we expect to have used for testing. This is key because any contributor needs to be able to spin up an identical envrionment to replicate any build failures. We don't want runners for merge testing that are built as a blackbox by someone. That is the single biggest painpoint with Peter's current merge jobs - we can't easily replicate Peter's merge env even if we had the matching hardware available. Regards, Daniel
On Thu, 2020-09-03 at 17:12 -0400, Cleber Rosa wrote: > On Thu, Jul 09, 2020 at 10:55:07AM +0200, Erik Skultety wrote: > > On Wed, Jul 08, 2020 at 10:46:57PM -0400, Cleber Rosa wrote: > > > +.. note:: there are currently limitations to gitlab-runner itself when > > > + setting up a service under FreeBSD systems. You will need to > > > + perform steps 4 to 10 manually, as described at > > > + https://docs.gitlab.com/runner/install/freebsd.html > > > > What kinds of limitations? Is it architecture constrained maybe? I'm asking > > because we have all of the steps covered by an Ansible playbook, so I kinda got > > triggered by the word "manually". Also, the document only mentions 9 steps > > overall. > > FreeBSD's "service management" (systemd/sys-v like) is not covered by > the GO library[1] used on gitlab-runner. It's not ideal, and the > second best solution would be to script the equivalent handling within > the playbook, but I remember trying and finding some inconsistencies. > Then, I had to give it up and defer to whenever a FreeBSD job is > actually added. > > [1] - https://github.com/ayufan/golang-kardianos-service Note that this is a fork of https://github.com/kardianos/service where FreeBSD support was added recently with https://github.com/kardianos/service/commit/14b2cc59a290407a6f1cb3daba59069429d9665b I'm not sure why gitlab-runner would use a fork rather than the primary repository, but perhaps they can be convinced to switch and gain better FreeBSD support in the process.
On 7/9/20 4:46 AM, Cleber Rosa wrote: > This is a mapping of Peter's "remake-merge-builds" and > "pull-buildtest" scripts, gone through some updates, adding some build > option and removing others. > > The jobs currently cover the machines that the QEMU project owns, and that > are setup and ready to run jobs: > > - Ubuntu 18.04 on S390x > - Ubuntu 20.04 on aarch64 > > During the development of this set of jobs, the GitLab CI was tested > with many other architectures, including ppc64, s390x and aarch64, > along with the other OSs (not included here): > > - Fedora 30 > - FreeBSD 12.1 > > More information can be found in the documentation itself. [...] > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst > index c1ff24370b..f8dab788ea 100644 > --- a/docs/devel/testing.rst > +++ b/docs/devel/testing.rst > @@ -1003,3 +1003,150 @@ exercise as many corner cases as possible. It is a useful test suite > to run to exercise QEMU's linux-user code:: > > https://linux-test-project.github.io/ > + > +CI > +== > + > +QEMU has configurations enabled for a number of different CI services. > +The most update information about them and their status can be found > +at:: > + > + https://wiki.qemu.org/Testing/CI > + > +Gating CI > +---------- > + > +A Pull Requests will only to be merged if they successfully go through > +a different set of CI jobs. GitLab's CI is the service/framework used > +for executing the gating jobs. > + > +The architecture of GitLab's CI service allows different machines to be > +setup with GitLab's "agent", called gitlab-runner, which will take care > +of running jobs created by events such as a push to a branch. Now that GitLab announced the free tier time limit, it may be worth mentioning (if set up) the CI pipeline can be skipped using: $ git push --push-option=ci.skip [...]
Hi, > > IOW, a RHEL-8 aarch64 host, running docker for ubuntu18.04, fedora30 > > etc. > > I've come across so many caveats and corner cases that having the > lowest common denominator proved to be the smart and sane thing to do. > For instance, building on the example you gave, running a RHEL 8 > aarch64 host is a NO GO Today because RHEL 8 doesn't ship with docker > and the gitlab runner needs to be taught to talk to, say, Podman. podman got support for the docker daemon api (v2.0 I think), so this should be possible soon, without changing gitlab runner. Also there is a kubernetes runner so you can run the thing in pretty much any cluster. k3s.io seems to be the easiest way to boot up a small kubernetes cluster on pretty much any linux system. Works on aarch64 too. Didn't try (yet) to run the gitlab runner there. > Finally, it's going to be very important for some organizations to > run tests outside of container environments. For instance, Red Hat > needs to run QEMU+KVM tests on bare metal and on VMs, in addition > to containers. Also for *BSD testing. For setting up VM images we should use tests/vm. > Like I explained before, containers-only won't cut it. So, we need > tooling that is environment agnostic. So far, ansible playbooks seem > to be a reasonable solution. But duplicating information bothers me > as much as it seems to bother you, so we need to engage in common > tooling that is capable of generating those container environments, > but not be limited to it. One example of a tool that seems to be > a good candidate is "Libvirt's" lcitool. What is wrong with the docker files we have? It is a good idea anyway to use the same images for local test builds and CI. Makes it easier to reproduce CI failures locally. take care, Gerd
On Fri, Sep 04, 2020 at 11:11:25AM +0200, Andrea Bolognani wrote: > On Thu, 2020-09-03 at 17:12 -0400, Cleber Rosa wrote: > > On Thu, Jul 09, 2020 at 10:55:07AM +0200, Erik Skultety wrote: > > > On Wed, Jul 08, 2020 at 10:46:57PM -0400, Cleber Rosa wrote: > > > > +.. note:: there are currently limitations to gitlab-runner itself when > > > > + setting up a service under FreeBSD systems. You will need to > > > > + perform steps 4 to 10 manually, as described at > > > > + https://docs.gitlab.com/runner/install/freebsd.html > > > > > > What kinds of limitations? Is it architecture constrained maybe? I'm asking > > > because we have all of the steps covered by an Ansible playbook, so I kinda got > > > triggered by the word "manually". Also, the document only mentions 9 steps > > > overall. > > > > FreeBSD's "service management" (systemd/sys-v like) is not covered by > > the GO library[1] used on gitlab-runner. It's not ideal, and the > > second best solution would be to script the equivalent handling within > > the playbook, but I remember trying and finding some inconsistencies. > > Then, I had to give it up and defer to whenever a FreeBSD job is > > actually added. > > > > [1] - https://github.com/ayufan/golang-kardianos-service > > Note that this is a fork of > > https://github.com/kardianos/service > > where FreeBSD support was added recently with > > https://github.com/kardianos/service/commit/14b2cc59a290407a6f1cb3daba59069429d9665b > That's good news! > I'm not sure why gitlab-runner would use a fork rather than the > primary repository, but perhaps they can be convinced to switch and > gain better FreeBSD support in the process. > I can only imagine they were using the fork because the primary repository did not have the bits they needed there... and were not willing or were not successful and getting them there. We can ask/hope/wait for gitlab to switch, and then this will no longer be an issue indeed. Thanks, - Cleber. > -- > Andrea Bolognani / Red Hat / Virtualization >
diff --git a/.gitlab-ci.d/gating.yml b/.gitlab-ci.d/gating.yml new file mode 100644 index 0000000000..5562df5708 --- /dev/null +++ b/.gitlab-ci.d/gating.yml @@ -0,0 +1,146 @@ +variables: + GIT_SUBMODULE_STRATEGY: recursive + +# All ubuntu-18.04 jobs should run successfully in an environment +# setup by the scripts/ci/setup/build-environment.yml task +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04" +ubuntu-18.04-s390x-all-linux-static: + tags: + - ubuntu_18.04 + - s390x + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763 + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages + - ./configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + - make --output-sync -j`nproc` check-tcg V=1 + +ubuntu-18.04-s390x-all: + tags: + - ubuntu_18.04 + - s390x + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure --disable-libssh + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +ubuntu-18.04-s390x-alldbg: + tags: + - ubuntu_18.04 + - s390x + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure --enable-debug --disable-libssh + - make clean + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +ubuntu-18.04-s390x-clang: + tags: + - ubuntu_18.04 + - s390x + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +ubuntu-18.04-s390x-tci: + tags: + - ubuntu_18.04 + - s390x + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure --disable-libssh --enable-tcg-interpreter + - make --output-sync -j`nproc` + +ubuntu-18.04-s390x-notcg: + tags: + - ubuntu_18.04 + - s390x + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure --disable-libssh --disable-tcg + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +# All ubuntu-20.04 jobs should run successfully in an environment +# setup by the scripts/ci/setup/qemu/build-environment.yml task +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04" +ubuntu-20.04-aarch64-all-linux-static: + tags: + - ubuntu_20.04 + - aarch64 + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763 + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages + - ./configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + - make --output-sync -j`nproc` check-tcg V=1 + +ubuntu-20.04-aarch64-all: + tags: + - ubuntu_20.04 + - aarch64 + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure --disable-libssh + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +ubuntu-20.04-aarch64-alldbg: + tags: + - ubuntu_20.04 + - aarch64 + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure --enable-debug --disable-libssh + - make clean + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +ubuntu-20.04-aarch64-clang: + tags: + - ubuntu_20.04 + - aarch64 + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +ubuntu-20.04-aarch64-tci: + tags: + - ubuntu_20.04 + - aarch64 + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure --disable-libssh --enable-tcg-interpreter + - make --output-sync -j`nproc` + +ubuntu-20.04-aarch64-notcg: + tags: + - ubuntu_20.04 + - aarch64 + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure --disable-libssh --disable-tcg + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 349c77aa58..49efe38529 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,6 +1,7 @@ include: - local: '/.gitlab-ci.d/edk2.yml' - local: '/.gitlab-ci.d/opensbi.yml' + - local: '/.gitlab-ci.d/gating.yml' .update_apt_template: &before_script_apt before_script: diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index c1ff24370b..f8dab788ea 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -1003,3 +1003,150 @@ exercise as many corner cases as possible. It is a useful test suite to run to exercise QEMU's linux-user code:: https://linux-test-project.github.io/ + +CI +== + +QEMU has configurations enabled for a number of different CI services. +The most update information about them and their status can be found +at:: + + https://wiki.qemu.org/Testing/CI + +Gating CI +---------- + +A Pull Requests will only to be merged if they successfully go through +a different set of CI jobs. GitLab's CI is the service/framework used +for executing the gating jobs. + +The architecture of GitLab's CI service allows different machines to be +setup with GitLab's "agent", called gitlab-runner, which will take care +of running jobs created by events such as a push to a branch. + +Even though gitlab-runner can execute jobs on environments such as +containers, this initial implementation assumes the shell executor is +used, effectively running jobs on the same machine (be them physical +or virtual) the gitlab-runner agent is running. This means those +machines must be setup in advance, with the requirements matching the +jobs expected to be executed there. + +Machine configuration for gating jobs +------------------------------------- + +The GitLab's CI architecture allows different parties to provide +different machines that will run different jobs. At this point, QEMU +will deploy a limited set of machines and jobs. Documentation and/or +scripts to setup those machines is located under:: + + scripts/ci/setup + +Ansible playbooks have been provided to perform two different tasks +related to setting gitlab-runner and the build environment. + +Other organizations involved in QEMU development may, in the near +future, contribute their own setup documentation/scripts under +``scripts/ci/orgs/$ORG_NAME`` directory. + +The GitLab CI jobs definition for the gating jobs are located under:: + + .gitlab-ci.d/gating.yml + +Once the gating CI has proved mature with this set of initial jobs, +other members in the community may provide their own machine +configuration documentation/scripts, and accompanying job definitions. +Those contributed jobs should run as non-gating, until their +reliability is verified. + +Machine Setup Howto +------------------- + +For all Linux based systems, the setup can be mostly automated by the +execution of two Ansible playbooks. Start by adding your machines to +the ``inventory`` file under ``scripts/ci/setup``, such as this:: + + [local] + fully.qualified.domain + other.machine.hostname + +You may need to set some variables in the inventory file itself. One +very common need is to tell Ansible to use a Python 3 interpreter on +those hosts. This would look like:: + + [local] + fully.qualified.domain ansible_python_interpreter=/usr/bin/python3 + other.machine.hostname ansible_python_interpreter=/usr/bin/python3 + +Build environment +~~~~~~~~~~~~~~~~~ + +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook +will setup machines with the environment needed to perform builds of +QEMU and runs of the tests defined in the current gating jobs. It +covers a number of different Linux distributions and FreeBSD. + +To run the playbook, execute:: + + cd scripts/ci/setup + ansible-playbook -i inventory build-environment.yml + +gitlab-runner setup and registration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The gitlab-runner agent needs to be installed on each machine that +will run jobs. The association between a machine and a GitLab project +happens with a registration token. To find the registration token for +your repository/project, navigate on GitLab's web UI to: + + * Settings (the gears like icon), then + * CI/CD, then + * Runners, and click on the "Expand" button, then + * Under "Set up a specific Runner manually", look for the value under + "Use the following registration token during setup" + +Edit the ``scripts/ci/setup/vars.yml`` file, setting the +``gitlab_runner_registration_token`` variable to the value obtained +earlier. + +.. note:: gitlab-runner is not available from the standard location + for all OS and architectures combinations. For some systems, + a custom build may be necessary. Some builds are avaiable + at https://cleber.fedorapeople.org/gitlab-runner/ and this + URI may be used as a value on ``vars.yml`` + +To run the playbook, execute:: + + cd scripts/ci/setup + ansible-playbook -i inventory gitlab-runner.yml + +.. note:: there are currently limitations to gitlab-runner itself when + setting up a service under FreeBSD systems. You will need to + perform steps 4 to 10 manually, as described at + https://docs.gitlab.com/runner/install/freebsd.html + +Following the registration, it's necessary to configure the runner tags, +and optionally other configurations on the GitLab UI. Navigate to: + + * Settings (the gears like icon), then + * CI/CD, then + * Runners, and click on the "Expand" button, then + * "Runners activated for this project", then + * Click on the "Edit" icon (next to the "Lock" Icon) + +Under tags, add values matching the jobs a runner should run. For a +FreeBSD 12.1 x86_64 system, the tags should be set as:: + + freebsd12.1,x86_64 + +Because the job definition at ``.gitlab-ci.d/gating.yml`` would contain:: + + freebsd-12.1-x86_64-all: + tags: + - freebsd_12.1 + - x86_64 + +It's also recommended to: + + * increase the "Maximum job timeout" to something like ``2h`` + * uncheck the "Run untagged jobs" check box + * give it a better Description diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml new file mode 100644 index 0000000000..89b35386c7 --- /dev/null +++ b/scripts/ci/setup/build-environment.yml @@ -0,0 +1,217 @@ +--- +- name: Installation of basic packages to build QEMU + hosts: all + vars_files: + - vars.yml + tasks: + - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04 + apt: + update_cache: yes + # This matches the packages on tests/docker/Dockerfiles/ubuntu1804.docker + pkg: + - flex + - bison + - ccache + - clang + - gcc + - gettext + - git + - glusterfs-common + - libaio-dev + - libattr1-dev + - libbrlapi-dev + - libbz2-dev + - libcacard-dev + - libcap-ng-dev + - libcurl4-gnutls-dev + - libdrm-dev + - libepoxy-dev + - libfdt-dev + - libgbm-dev + - libgtk-3-dev + - libibverbs-dev + - libiscsi-dev + - libjemalloc-dev + - libjpeg-turbo8-dev + - liblzo2-dev + - libncurses5-dev + - libncursesw5-dev + - libnfs-dev + - libnss3-dev + - libnuma-dev + - libpixman-1-dev + - librados-dev + - librbd-dev + - librdmacm-dev + - libsasl2-dev + - libsdl2-dev + - libseccomp-dev + - libsnappy-dev + - libspice-protocol-dev + - libssh-dev + - libusb-1.0-0-dev + - libusbredirhost-dev + - libvdeplug-dev + - libvte-2.91-dev + - libzstd-dev + - make + - python3-yaml + - python3-sphinx + - sparse + - texinfo + - xfslibs-dev + state: present + when: "ansible_facts['distribution'] == 'Ubuntu'" + + - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x + apt: + update_cache: yes + pkg: + - libspice-server-dev + - libxen-dev + state: present + when: + - "ansible_facts['distribution'] == 'Ubuntu'" + - "ansible_facts['architecture'] != 's390x'" + + - name: Install basic packages to build QEMU on FreeBSD 12.x + pkgng: + # This matches the packages on .cirrus.yml under the freebsd_12_task + name: bash,bison,curl,cyrus-sasl,git,glib,gmake,gnutls,gsed,nettle,perl5,pixman,pkgconf,png,usbredir + state: present + when: "ansible_facts['os_family'] == 'FreeBSD'" + + - name: Install basic packages to build QEMU on Fedora 30 + dnf: + # This matches the packages on tests/docker/Dockerfiles/fedora.docker + name: + - bc + - bison + - brlapi-devel + - bzip2 + - bzip2-devel + - ccache + - clang + - cyrus-sasl-devel + - dbus-daemon + - device-mapper-multipath-devel + - findutils + - flex + - gcc + - gcc-c++ + - gettext + - git + - glib2-devel + - glusterfs-api-devel + - gnutls-devel + - gtk3-devel + - hostname + - libaio-devel + - libasan + - libattr-devel + - libblockdev-mpath-devel + - libcap-ng-devel + - libcurl-devel + - libfdt-devel + - libiscsi-devel + - libjpeg-devel + - libpmem-devel + - libpng-devel + - librbd-devel + - libseccomp-devel + - libssh-devel + - libubsan + - libusbx-devel + - libxml2-devel + - llvm + - lzo-devel + - make + - mingw32-bzip2 + - mingw32-curl + - mingw32-glib2 + - mingw32-gmp + - mingw32-gnutls + - mingw32-gtk3 + - mingw32-libjpeg-turbo + - mingw32-libpng + - mingw32-libtasn1 + - mingw32-nettle + - mingw32-nsis + - mingw32-pixman + - mingw32-pkg-config + - mingw32-SDL2 + - mingw64-bzip2 + - mingw64-curl + - mingw64-glib2 + - mingw64-gmp + - mingw64-gnutls + - mingw64-gtk3 + - mingw64-libjpeg-turbo + - mingw64-libpng + - mingw64-libtasn1 + - mingw64-nettle + - mingw64-pixman + - mingw64-pkg-config + - mingw64-SDL2 + - ncurses-devel + - nettle-devel + - nss-devel + - numactl-devel + - perl + - perl-Test-Harness + - pixman-devel + - python3 + - python3-sphinx + - PyYAML + - rdma-core-devel + - SDL2-devel + - snappy-devel + - sparse + - spice-server-devel + - systemd-devel + - systemtap-sdt-devel + - tar + - texinfo + - usbredir-devel + - virglrenderer-devel + - vte291-devel + - which + - xen-devel + - zlib-devel + - libzstd-devel + state: present + when: "ansible_facts['distribution'] == 'Fedora'" + + - name: Install basic packages to build QEMU on CentOS 8 + dnf: + # This matches the packages on tests/docker/Dockerfiles/centos8.docker + name: + - SDL-devel + - bison + - bzip2 + - bzip2-devel + - dbus-daemon + - flex + - gcc + - gcc-c++ + - gettext + - git + - glib2-devel + - libaio-devel + - libepoxy-devel + - lzo-devel + - make + - mesa-libEGL-devel + - nettle-devel + - perl-Test-Harness + - pixman-devel + - python36 + - rdma-core-devel + - spice-glib-devel + - spice-server + - tar + - zlib-devel + state: present + when: + - "ansible_facts['distribution'] == 'CentOS'" + - "ansible_facts['distribution_major_version'] == '8'" diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml new file mode 100644 index 0000000000..c2f52dad10 --- /dev/null +++ b/scripts/ci/setup/gitlab-runner.yml @@ -0,0 +1,72 @@ +--- +- name: Installation of gitlab-runner + hosts: all + vars_files: + - vars.yml + tasks: + - debug: + msg: 'Checking for a valid GitLab registration token' + failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'" + + - name: Checks the availability of official gitlab-runner builds in the archive + uri: + url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version }}/binaries/gitlab-runner-linux-386 + method: HEAD + status_code: + - 200 + - 403 + register: gitlab_runner_available_archive + + - name: Update base url + set_fact: + gitlab_runner_base_url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version }}/binaries/gitlab-runner- + when: gitlab_runner_available_archive.status == 200 + - debug: + msg: Base gitlab-runner url is {{ gitlab_runner_base_url }} + + - name: Set OS name (FreeBSD) + set_fact: + gitlab_runner_os: freebsd + when: "ansible_facts['system'] == 'FreeBSD'" + + - name: Create a group for the gitlab-runner service + group: + name: gitlab-runner + + - name: Create a user for the gitlab-runner service + user: + user: gitlab-runner + group: gitlab-runner + comment: GitLab Runner + home: /home/gitlab-runner + shell: /bin/bash + + - name: Remove the .bash_logout file when on Ubuntu systems + file: + path: /home/gitlab-runner/.bash_logout + state: absent + when: "ansible_facts['distribution'] == 'Ubuntu'" + + - name: Downloads the matching gitlab-runner + get_url: + dest: /usr/local/bin/gitlab-runner + url: "{{ gitlab_runner_base_url }}{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}" + owner: gitlab-runner + group: gitlab-runner + mode: u=rwx,g=rwx,o=rx + + - name: Register the gitlab-runner + command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'" + + - name: Install the gitlab-runner service using its own functionality + command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner + register: gitlab_runner_install_service_result + failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr" + when: "ansible_facts['os_family'] != 'FreeBSD'" + + - name: Enable the gitlab-runner service + service: + name: gitlab-runner + state: started + enabled: yes + when: "ansible_facts['os_family'] != 'FreeBSD'" diff --git a/scripts/ci/setup/inventory b/scripts/ci/setup/inventory new file mode 100644 index 0000000000..8bb7ba6b33 --- /dev/null +++ b/scripts/ci/setup/inventory @@ -0,0 +1,2 @@ +[local] +localhost diff --git a/scripts/ci/setup/vars.yml b/scripts/ci/setup/vars.yml new file mode 100644 index 0000000000..621435d030 --- /dev/null +++ b/scripts/ci/setup/vars.yml @@ -0,0 +1,13 @@ +# The version of the gitlab-runner to use +gitlab_runner_version: 13.1.1 +# The base location of gitlab-runner binaries, this will be suffixed by $OS-$ARCH +gitlab_runner_base_url: https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner- +# The URL of the gitlab server to use, usually https://gitlab.com unless you're +# using a private GitLab instance +gitlab_runner_server_url: https://gitlab.com +# Defaults to linux, checks can be used to change this +gitlab_runner_os: linux +# Defaults to amd64 (x86_64), checks can be used to change this +gitlab_runner_arch: amd64 +# A unique token made available by GitLab to your project for registering runners +gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
This is a mapping of Peter's "remake-merge-builds" and "pull-buildtest" scripts, gone through some updates, adding some build option and removing others. The jobs currently cover the machines that the QEMU project owns, and that are setup and ready to run jobs: - Ubuntu 18.04 on S390x - Ubuntu 20.04 on aarch64 During the development of this set of jobs, the GitLab CI was tested with many other architectures, including ppc64, s390x and aarch64, along with the other OSs (not included here): - Fedora 30 - FreeBSD 12.1 More information can be found in the documentation itself. Signed-off-by: Cleber Rosa <crosa@redhat.com> --- .gitlab-ci.d/gating.yml | 146 +++++++++++++++++ .gitlab-ci.yml | 1 + docs/devel/testing.rst | 147 +++++++++++++++++ scripts/ci/setup/build-environment.yml | 217 +++++++++++++++++++++++++ scripts/ci/setup/gitlab-runner.yml | 72 ++++++++ scripts/ci/setup/inventory | 2 + scripts/ci/setup/vars.yml | 13 ++ 7 files changed, 598 insertions(+) create mode 100644 .gitlab-ci.d/gating.yml create mode 100644 scripts/ci/setup/build-environment.yml create mode 100644 scripts/ci/setup/gitlab-runner.yml create mode 100644 scripts/ci/setup/inventory create mode 100644 scripts/ci/setup/vars.yml