| Message ID | 20201014052140.1146924-3-crosa@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | GitLab Custom Runners and Jobs (was: QEMU Gating CI) | expand |
On Wed, Oct 14, 2020 at 01:21:38AM -0400, Cleber Rosa wrote: > To run basic jobs on custom runners, the environment needs to be > properly set up. The most common requirement is having the right > packages installed. > > The playbook introduced here covers a number of different Linux > distributions and FreeBSD, and are intended to provide a reproducible > environment. > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > --- > docs/devel/ci.rst | 32 ++++ > scripts/ci/setup/build-environment.yml | 220 +++++++++++++++++++++++++ > scripts/ci/setup/inventory | 2 + > 3 files changed, 254 insertions(+) > create mode 100644 scripts/ci/setup/build-environment.yml > create mode 100644 scripts/ci/setup/inventory > > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst > index 41a4bbddad..208b5e399b 100644 > --- a/docs/devel/ci.rst > +++ b/docs/devel/ci.rst > @@ -52,3 +52,35 @@ As a general rule, those newly added contributed jobs should run as > The precise minimum requirements and exact rules for machine > configuration documentation/scripts, and the success rate of jobs are > still to be defined. > + > +Machine Setup Howto > +------------------- > + > +For all Linux based systems, the setup can be mostly automated by the > +execution of two Ansible playbooks. Start by adding your machines to > +the ``inventory`` file under ``scripts/ci/setup``, such as this:: > + > + [local] > + fully.qualified.domain > + other.machine.hostname > + > +You may need to set some variables in the inventory file itself. One > +very common need is to tell Ansible to use a Python 3 interpreter on > +those hosts. This would look like:: > + > + [local] > + fully.qualified.domain ansible_python_interpreter=/usr/bin/python3 > + other.machine.hostname ansible_python_interpreter=/usr/bin/python3 > + > +Build environment > +~~~~~~~~~~~~~~~~~ > + > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will > +set up machines with the environment needed to perform builds and run > +QEMU tests. It covers a number of different Linux distributions and > +FreeBSD. > + > +To run the playbook, execute:: > + > + cd scripts/ci/setup > + ansible-playbook -i inventory build-environment.yml > diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml > new file mode 100644 > index 0000000000..0e8894bca9 > --- /dev/null > +++ b/scripts/ci/setup/build-environment.yml > @@ -0,0 +1,220 @@ > +--- > +- name: Installation of basic packages to build QEMU > + hosts: all > + tasks: > + - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04 > + apt: > + update_cache: yes > + # Originally from tests/docker/dockerfiles/ubuntu1804.docker > + pkg: > + - ccache > + - clang > + - gcc > + - gettext > + - git > + - glusterfs-common > + - libaio-dev > + - libattr1-dev > + - libbrlapi-dev > + - libbz2-dev > + - libcacard-dev > + - libcap-ng-dev > + - libcurl4-gnutls-dev > + - libdrm-dev > + - libepoxy-dev > + - libfdt-dev > + - libgbm-dev > + - libgtk-3-dev > + - libibverbs-dev > + - libiscsi-dev > + - libjemalloc-dev > + - libjpeg-turbo8-dev > + - liblzo2-dev > + - libncurses5-dev > + - libncursesw5-dev > + - libnfs-dev > + - libnss3-dev > + - libnuma-dev > + - libpixman-1-dev > + - librados-dev > + - librbd-dev > + - librdmacm-dev > + - libsasl2-dev > + - libsdl2-dev > + - libseccomp-dev > + - libsnappy-dev > + - libspice-protocol-dev > + - libssh-dev > + - libusb-1.0-0-dev > + - libusbredirhost-dev > + - libvdeplug-dev > + - libvte-2.91-dev > + - libzstd-dev > + - make > + - python3-yaml > + - python3-sphinx > + - sparse > + - xfslibs-dev This needs updating to add meson, and with Paolo's series today you might as well go ahead and add ninja-build immediately too https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg04025.html Same for all the other distro package lists. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Oct 14, 2020 at 06:30:09PM +0100, Daniel P. Berrangé wrote: > On Wed, Oct 14, 2020 at 01:21:38AM -0400, Cleber Rosa wrote: > > To run basic jobs on custom runners, the environment needs to be > > properly set up. The most common requirement is having the right > > packages installed. > > > > The playbook introduced here covers a number of different Linux > > distributions and FreeBSD, and are intended to provide a reproducible > > environment. > > > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > > --- > > docs/devel/ci.rst | 32 ++++ > > scripts/ci/setup/build-environment.yml | 220 +++++++++++++++++++++++++ > > scripts/ci/setup/inventory | 2 + > > 3 files changed, 254 insertions(+) > > create mode 100644 scripts/ci/setup/build-environment.yml > > create mode 100644 scripts/ci/setup/inventory > > > > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst > > index 41a4bbddad..208b5e399b 100644 > > --- a/docs/devel/ci.rst > > +++ b/docs/devel/ci.rst > > @@ -52,3 +52,35 @@ As a general rule, those newly added contributed jobs should run as > > The precise minimum requirements and exact rules for machine > > configuration documentation/scripts, and the success rate of jobs are > > still to be defined. > > + > > +Machine Setup Howto > > +------------------- > > + > > +For all Linux based systems, the setup can be mostly automated by the > > +execution of two Ansible playbooks. Start by adding your machines to > > +the ``inventory`` file under ``scripts/ci/setup``, such as this:: > > + > > + [local] > > + fully.qualified.domain > > + other.machine.hostname > > + > > +You may need to set some variables in the inventory file itself. One > > +very common need is to tell Ansible to use a Python 3 interpreter on > > +those hosts. This would look like:: > > + > > + [local] > > + fully.qualified.domain ansible_python_interpreter=/usr/bin/python3 > > + other.machine.hostname ansible_python_interpreter=/usr/bin/python3 > > + > > +Build environment > > +~~~~~~~~~~~~~~~~~ > > + > > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will > > +set up machines with the environment needed to perform builds and run > > +QEMU tests. It covers a number of different Linux distributions and > > +FreeBSD. > > + > > +To run the playbook, execute:: > > + > > + cd scripts/ci/setup > > + ansible-playbook -i inventory build-environment.yml > > diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml > > new file mode 100644 > > index 0000000000..0e8894bca9 > > --- /dev/null > > +++ b/scripts/ci/setup/build-environment.yml > > @@ -0,0 +1,220 @@ > > +--- > > +- name: Installation of basic packages to build QEMU > > + hosts: all > > + tasks: > > + - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04 > > + apt: > > + update_cache: yes > > + # Originally from tests/docker/dockerfiles/ubuntu1804.docker > > + pkg: > > + - ccache > > + - clang > > + - gcc > > + - gettext > > + - git > > + - glusterfs-common > > + - libaio-dev > > + - libattr1-dev > > + - libbrlapi-dev > > + - libbz2-dev > > + - libcacard-dev > > + - libcap-ng-dev > > + - libcurl4-gnutls-dev > > + - libdrm-dev > > + - libepoxy-dev > > + - libfdt-dev > > + - libgbm-dev > > + - libgtk-3-dev > > + - libibverbs-dev > > + - libiscsi-dev > > + - libjemalloc-dev > > + - libjpeg-turbo8-dev > > + - liblzo2-dev > > + - libncurses5-dev > > + - libncursesw5-dev > > + - libnfs-dev > > + - libnss3-dev > > + - libnuma-dev > > + - libpixman-1-dev > > + - librados-dev > > + - librbd-dev > > + - librdmacm-dev > > + - libsasl2-dev > > + - libsdl2-dev > > + - libseccomp-dev > > + - libsnappy-dev > > + - libspice-protocol-dev > > + - libssh-dev > > + - libusb-1.0-0-dev > > + - libusbredirhost-dev > > + - libvdeplug-dev > > + - libvte-2.91-dev > > + - libzstd-dev > > + - make > > + - python3-yaml > > + - python3-sphinx > > + - sparse > > + - xfslibs-dev > > This needs updating to add meson, and with Paolo's series today you > might as well go ahead and add ninja-build immediately too > > https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg04025.html > > Same for all the other distro package lists. > > Good point, will do. Thanks! - Cleber. > Regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > >
diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst index 41a4bbddad..208b5e399b 100644 --- a/docs/devel/ci.rst +++ b/docs/devel/ci.rst @@ -52,3 +52,35 @@ As a general rule, those newly added contributed jobs should run as The precise minimum requirements and exact rules for machine configuration documentation/scripts, and the success rate of jobs are still to be defined. + +Machine Setup Howto +------------------- + +For all Linux based systems, the setup can be mostly automated by the +execution of two Ansible playbooks. Start by adding your machines to +the ``inventory`` file under ``scripts/ci/setup``, such as this:: + + [local] + fully.qualified.domain + other.machine.hostname + +You may need to set some variables in the inventory file itself. One +very common need is to tell Ansible to use a Python 3 interpreter on +those hosts. This would look like:: + + [local] + fully.qualified.domain ansible_python_interpreter=/usr/bin/python3 + other.machine.hostname ansible_python_interpreter=/usr/bin/python3 + +Build environment +~~~~~~~~~~~~~~~~~ + +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will +set up machines with the environment needed to perform builds and run +QEMU tests. It covers a number of different Linux distributions and +FreeBSD. + +To run the playbook, execute:: + + cd scripts/ci/setup + ansible-playbook -i inventory build-environment.yml diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml new file mode 100644 index 0000000000..0e8894bca9 --- /dev/null +++ b/scripts/ci/setup/build-environment.yml @@ -0,0 +1,220 @@ +--- +- name: Installation of basic packages to build QEMU + hosts: all + tasks: + - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04 + apt: + update_cache: yes + # Originally from tests/docker/dockerfiles/ubuntu1804.docker + pkg: + - ccache + - clang + - gcc + - gettext + - git + - glusterfs-common + - libaio-dev + - libattr1-dev + - libbrlapi-dev + - libbz2-dev + - libcacard-dev + - libcap-ng-dev + - libcurl4-gnutls-dev + - libdrm-dev + - libepoxy-dev + - libfdt-dev + - libgbm-dev + - libgtk-3-dev + - libibverbs-dev + - libiscsi-dev + - libjemalloc-dev + - libjpeg-turbo8-dev + - liblzo2-dev + - libncurses5-dev + - libncursesw5-dev + - libnfs-dev + - libnss3-dev + - libnuma-dev + - libpixman-1-dev + - librados-dev + - librbd-dev + - librdmacm-dev + - libsasl2-dev + - libsdl2-dev + - libseccomp-dev + - libsnappy-dev + - libspice-protocol-dev + - libssh-dev + - libusb-1.0-0-dev + - libusbredirhost-dev + - libvdeplug-dev + - libvte-2.91-dev + - libzstd-dev + - make + - python3-yaml + - python3-sphinx + - sparse + - xfslibs-dev + state: present + when: "ansible_facts['distribution'] == 'Ubuntu'" + + - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x + apt: + update_cache: yes + pkg: + - libspice-server-dev + - libxen-dev + state: present + when: + - "ansible_facts['distribution'] == 'Ubuntu'" + - "ansible_facts['architecture'] != 's390x'" + + - name: Install basic packages to build QEMU on FreeBSD 12.x + pkgng: + # This matches the packages on .cirrus.yml under the freebsd_12_task + name: bash,bison,curl,cyrus-sasl,git,glib,gmake,gnutls,gsed,nettle,perl5,pixman,pkgconf,png,usbredir + state: present + when: "ansible_facts['os_family'] == 'FreeBSD'" + + - name: Install basic packages to build QEMU on Fedora 30, 31 and 32 + dnf: + # Originally from tests/docker/dockerfiles/fedora.docker + name: + - SDL2-devel + - bc + - brlapi-devel + - bzip2 + - bzip2-devel + - ccache + - clang + - cyrus-sasl-devel + - dbus-daemon + - device-mapper-multipath-devel + - diffutils + - findutils + - gcc + - gcc-c++ + - genisoimage + - gettext + - git + - glib2-devel + - glusterfs-api-devel + - gnutls-devel + - gtk3-devel + - hostname + - libaio-devel + - libasan + - libattr-devel + - libblockdev-mpath-devel + - libcap-ng-devel + - libcurl-devel + - libepoxy-devel + - libfdt-devel + - libiscsi-devel + - libjpeg-devel + - libpmem-devel + - libpng-devel + - librbd-devel + - libseccomp-devel + - libssh-devel + - libubsan + - libudev-devel + - libusbx-devel + - libxml2-devel + - libzstd-devel + - llvm + - lzo-devel + - make + - mingw32-SDL2 + - mingw32-bzip2 + - mingw32-curl + - mingw32-glib2 + - mingw32-gmp + - mingw32-gnutls + - mingw32-gtk3 + - mingw32-libjpeg-turbo + - mingw32-libpng + - mingw32-libtasn1 + - mingw32-nettle + - mingw32-nsis + - mingw32-pixman + - mingw32-pkg-config + - mingw64-SDL2 + - mingw64-bzip2 + - mingw64-curl + - mingw64-glib2 + - mingw64-gmp + - mingw64-gnutls + - mingw64-gtk3 + - mingw64-libjpeg-turbo + - mingw64-libpng + - mingw64-libtasn1 + - mingw64-nettle + - mingw64-pixman + - mingw64-pkg-config + - ncurses-devel + - nettle-devel + - nss-devel + - numactl-devel + - perl + - perl-Test-Harness + - pixman-devel + - python3 + - python3-PyYAML + - python3-numpy + - python3-opencv + - python3-pillow + - python3-pip + - python3-sphinx + - python3-virtualenv + - rdma-core-devel + - snappy-devel + - sparse + - spice-server-devel + - systemd-devel + - systemtap-sdt-devel + - tar + - tesseract + - tesseract-langpack-eng + - usbredir-devel + - virglrenderer-devel + - vte291-devel + - which + - xen-devel + - zlib-devel + state: present + when: "ansible_facts['distribution'] == 'Fedora'" + + - name: Install basic packages to build QEMU on CentOS 8 + dnf: + # Originally from tests/docker/dockerfiles/centos8.docker + name: + - SDL-devel + - bzip2 + - bzip2-devel + - dbus-daemon + - gcc + - gcc-c++ + - genisoimage + - gettext + - git + - glib2-devel + - libaio-devel + - libepoxy-devel + - libgcrypt-devel + - lzo-devel + - make + - mesa-libEGL-devel + - nettle-devel + - perl-Test-Harness + - pixman-devel + - python36 + - rdma-core-devel + - spice-glib-devel + - spice-server + - tar + - zlib-devel + state: present + when: + - "ansible_facts['distribution'] == 'CentOS'" + - "ansible_facts['distribution_major_version'] == '8'" diff --git a/scripts/ci/setup/inventory b/scripts/ci/setup/inventory new file mode 100644 index 0000000000..8bb7ba6b33 --- /dev/null +++ b/scripts/ci/setup/inventory @@ -0,0 +1,2 @@ +[local] +localhost
To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed. The playbook introduced here covers a number of different Linux distributions and FreeBSD, and are intended to provide a reproducible environment. Signed-off-by: Cleber Rosa <crosa@redhat.com> --- docs/devel/ci.rst | 32 ++++ scripts/ci/setup/build-environment.yml | 220 +++++++++++++++++++++++++ scripts/ci/setup/inventory | 2 + 3 files changed, 254 insertions(+) create mode 100644 scripts/ci/setup/build-environment.yml create mode 100644 scripts/ci/setup/inventory