diff mbox series

[v1,11/33] tests/docker: update debian-amd64 with lcitool

Message ID 20220527153603.887929-12-alex.bennee@linaro.org
State Superseded
Headers show
Series testing/next (gitlab, junit, lcitool, x-compile) | expand

Commit Message

Alex Bennée May 27, 2022, 3:35 p.m. UTC
The one minor wrinkle we need to account for is the netmap support
still requires building from source. We also include cscope and GNU
global as they are used in one of the builds.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Philippe Mathieu-Daudé <f4bug@amsat.org>
Cc: Luigi Rizzo <rizzo@iet.unipi.it>
Cc: Giuseppe Lettieri <g.lettieri@iet.unipi.it>
Cc: Vincenzo Maffione <v.maffione@gmail.com>
---
 .gitlab-ci.d/containers.yml                  |   3 +-
 tests/docker/dockerfiles/debian-amd64.docker | 194 ++++++++++++++-----
 tests/lcitool/refresh                        |  19 ++
 3 files changed, 164 insertions(+), 52 deletions(-)

Comments

Daniel P. Berrangé May 27, 2022, 3:51 p.m. UTC | #1
On Fri, May 27, 2022 at 04:35:41PM +0100, Alex Bennée wrote:
> The one minor wrinkle we need to account for is the netmap support
> still requires building from source. We also include cscope and GNU
> global as they are used in one of the builds.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Cc: Luigi Rizzo <rizzo@iet.unipi.it>
> Cc: Giuseppe Lettieri <g.lettieri@iet.unipi.it>
> Cc: Vincenzo Maffione <v.maffione@gmail.com>
> ---
>  .gitlab-ci.d/containers.yml                  |   3 +-
>  tests/docker/dockerfiles/debian-amd64.docker | 194 ++++++++++++++-----
>  tests/lcitool/refresh                        |  19 ++
>  3 files changed, 164 insertions(+), 52 deletions(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


> +# Netmap still needs to be manually built as it is yet to be packaged
> +# into a distro. We also add cscope and gtags which are used in the CI
> +# test
> +debian11_extras = [
> +    "# netmap/cscope/global\n",
> +    "RUN DEBIAN_FRONTEND=noninteractive eatmydata \\\n",
> +    "  apt install -y --no-install-recommends \\\n",
> +    "  cscope\\\n",
> +    "  global\\\n",
> +    "  linux-headers-amd64\n",
> +    "RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap\n",
> +    "RUN cd /usr/src/netmap && git checkout v11.3\n",
> +    "RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install\n",

This will result in many layers bloating the image a little.

We could optimize it with approximately
[
  RUN export DEBIAN_FRONTEND=noninteractive && \
  eatmydata apt install -y --no-install-recommends \
    cscope \
    global \
    linux-headers-amd64 &&
  git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap && \
  cd /usr/src/netmap && \
  git checkout v11.3 && \0
  cd LINUX && \
  ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && \
  make install && \
  cd / && \
  rm -rf /usr/src/netmap &&
  eatmydata apt remove cscope global linux-headers-amd64

(untested)

Essentially this means we only add 1 layer to the docker image
and it only contains the final binary bits

Still, what you have is what already exists in tree, so on
that basis

  Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
  

With regards,
Daniel
diff mbox series

Patch

diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
index e9df90bbdd..be34cbc7ba 100644
--- a/.gitlab-ci.d/containers.yml
+++ b/.gitlab-ci.d/containers.yml
@@ -14,8 +14,7 @@  amd64-debian11-container:
 
 amd64-debian-container:
   extends: .container_job_template
-  stage: containers-layer2
-  needs: ['amd64-debian10-container']
+  stage: containers
   variables:
     NAME: debian-amd64
 
diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker
index ed546edcd6..503e282802 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian-amd64.docker
@@ -1,59 +1,153 @@ 
+# THIS FILE WAS AUTO-GENERATED
 #
-# Docker x86_64 target
+#  $ lcitool dockerfile --layers all debian-11 qemu
 #
-# This docker target builds on the Debian Buster base image. Further
-# libraries which are not widely available are installed by hand.
-#
-FROM qemu/debian10
-MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
-
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt build-dep -yy qemu
+# https://gitlab.com/libvirt/libvirt-ci
 
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        cscope \
-        genisoimage \
-        exuberant-ctags \
-        global \
-        libbz2-dev \
-        liblzo2-dev \
-        libgcrypt20-dev \
-        libfdt-dev \
-        librdmacm-dev \
-        libsasl2-dev \
-        libsnappy-dev \
-        libvte-dev \
-        netcat-openbsd \
-        openssh-client \
-        python3-numpy \
-        python3-opencv \
-        python3-venv
+FROM docker.io/library/debian:11-slim
 
-# virgl
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        libegl1-mesa-dev \
-        libepoxy-dev \
-        libgbm-dev
-RUN git clone https://gitlab.freedesktop.org/virgl/virglrenderer.git /usr/src/virglrenderer && \
-    cd /usr/src/virglrenderer && git checkout virglrenderer-0.8.0
-RUN cd /usr/src/virglrenderer && ./autogen.sh && ./configure --disable-tests && make install
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    apt-get update && \
+    apt-get install -y eatmydata && \
+    eatmydata apt-get dist-upgrade -y && \
+    eatmydata apt-get install --no-install-recommends -y \
+            bash \
+            bc \
+            bsdextrautils \
+            bzip2 \
+            ca-certificates \
+            ccache \
+            clang \
+            dbus \
+            debianutils \
+            diffutils \
+            exuberant-ctags \
+            findutils \
+            g++ \
+            gcc \
+            gcovr \
+            genisoimage \
+            gettext \
+            git \
+            hostname \
+            libaio-dev \
+            libasan5 \
+            libasound2-dev \
+            libattr1-dev \
+            libbpf-dev \
+            libbrlapi-dev \
+            libbz2-dev \
+            libc6-dev \
+            libcacard-dev \
+            libcap-ng-dev \
+            libcapstone-dev \
+            libcurl4-gnutls-dev \
+            libdaxctl-dev \
+            libdrm-dev \
+            libepoxy-dev \
+            libfdt-dev \
+            libffi-dev \
+            libfuse3-dev \
+            libgbm-dev \
+            libgcrypt20-dev \
+            libglib2.0-dev \
+            libglusterfs-dev \
+            libgnutls28-dev \
+            libgtk-3-dev \
+            libibumad-dev \
+            libibverbs-dev \
+            libiscsi-dev \
+            libjemalloc-dev \
+            libjpeg62-turbo-dev \
+            liblttng-ust-dev \
+            liblzo2-dev \
+            libncursesw5-dev \
+            libnfs-dev \
+            libnuma-dev \
+            libpam0g-dev \
+            libpcre2-dev \
+            libpixman-1-dev \
+            libpmem-dev \
+            libpng-dev \
+            libpulse-dev \
+            librbd-dev \
+            librdmacm-dev \
+            libsasl2-dev \
+            libsdl2-dev \
+            libsdl2-image-dev \
+            libseccomp-dev \
+            libselinux1-dev \
+            libslirp-dev \
+            libsnappy-dev \
+            libspice-protocol-dev \
+            libspice-server-dev \
+            libssh-gcrypt-dev \
+            libsystemd-dev \
+            libtasn1-6-dev \
+            libubsan1 \
+            libudev-dev \
+            liburing-dev \
+            libusb-1.0-0-dev \
+            libusbredirhost-dev \
+            libvdeplug-dev \
+            libvirglrenderer-dev \
+            libvte-2.91-dev \
+            libxen-dev \
+            libzstd-dev \
+            llvm \
+            locales \
+            make \
+            meson \
+            multipath-tools \
+            ncat \
+            nettle-dev \
+            ninja-build \
+            openssh-client \
+            perl-base \
+            pkgconf \
+            python3 \
+            python3-numpy \
+            python3-opencv \
+            python3-pillow \
+            python3-pip \
+            python3-sphinx \
+            python3-sphinx-rtd-theme \
+            python3-venv \
+            python3-yaml \
+            rpm2cpio \
+            sed \
+            sparse \
+            systemtap-sdt-dev \
+            tar \
+            tesseract-ocr \
+            tesseract-ocr-eng \
+            texinfo \
+            xfslibs-dev \
+            zlib1g-dev && \
+    eatmydata apt-get autoremove -y && \
+    eatmydata apt-get autoclean -y && \
+    sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \
+    dpkg-reconfigure locales && \
+    dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \
+    mkdir -p /usr/libexec/ccache-wrappers && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/c++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/clang && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
 
-# netmap
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        linux-headers-amd64
+ENV LANG "en_US.UTF-8"
+ENV MAKE "/usr/bin/make"
+ENV NINJA "/usr/bin/ninja"
+ENV PYTHON "/usr/bin/python3"
+ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
+# netmap/cscope/global
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+  apt install -y --no-install-recommends \
+  cscope\
+  global\
+  linux-headers-amd64
 RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap
 RUN cd /usr/src/netmap && git checkout v11.3
 RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install
 ENV QEMU_CONFIGURE_OPTS --enable-netmap
-
-RUN ldconfig
-
-# gcrypt
-ENV QEMU_CONFIGURE_OPTS $QEMU_CONFIGURE_OPTS --enable-gcrypt
diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index b947bb170d..5e260f8cd6 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -76,6 +76,23 @@  ubuntu2004_tsanhack = [
 ]
 
 
+# Netmap still needs to be manually built as it is yet to be packaged
+# into a distro. We also add cscope and gtags which are used in the CI
+# test
+debian11_extras = [
+    "# netmap/cscope/global\n",
+    "RUN DEBIAN_FRONTEND=noninteractive eatmydata \\\n",
+    "  apt install -y --no-install-recommends \\\n",
+    "  cscope\\\n",
+    "  global\\\n",
+    "  linux-headers-amd64\n",
+    "RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap\n",
+    "RUN cd /usr/src/netmap && git checkout v11.3\n",
+    "RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install\n",
+    "ENV QEMU_CONFIGURE_OPTS --enable-netmap\n"
+]
+
+
 def debian_cross_build(prefix, targets):
     conf = "ENV QEMU_CONFIGURE_OPTS --cross-prefix=%s\n" % (prefix)
     targets = "ENV DEF_TARGET_LIST %s\n" % (targets)
@@ -92,6 +109,8 @@  try:
     #
     generate_dockerfile("alpine", "alpine-edge")
     generate_dockerfile("centos8", "centos-stream-8")
+    generate_dockerfile("debian-amd64", "debian-11",
+                        trailer="".join(debian11_extras))
     generate_dockerfile("fedora", "fedora-35")
     generate_dockerfile("opensuse-leap", "opensuse-leap-152")
     generate_dockerfile("ubuntu2004", "ubuntu-2004",