From patchwork Wed May 12 10:20:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 435484 Delivered-To: patch@linaro.org Received: by 2002:a02:c901:0:0:0:0:0 with SMTP id t1csp4694999jao; Wed, 12 May 2021 03:33:22 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy/ZhFY7+lRD+aQU+Q5cSl+Vl2bkrT8mRKpS0kkPCdDa82eDr6EQwFPR8DY2pVp90uF97Oz X-Received: by 2002:a67:bc5:: with SMTP id 188mr29772928vsl.50.1620815602715; Wed, 12 May 2021 03:33:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1620815602; cv=none; d=google.com; s=arc-20160816; b=GtSRIJ6k3lb2VKU0+DSjdR5bQYdNf9oXDZHL7/z4wp9IV3HZDalvRXkNTAqU+Y2Efl izVGKZPUq54RWSWFb3ojb3La5YWXNw9YMH1Y6y3zBNfpTK30Y2MsqHM4gvnzVxs+JXZY xRiSsXy7Zu46uTzkkTJlVEpoj5TCN/sHvbXy7iCeekPpfKtWdeoI3UYF5G0YNPmiRMnD T4pPBk8gC1xdF2dNRX4zqiQhdaBFEhbw9tVb2bBcFRrFMaS5Xm29KxgGV1ysczgs8Ric PrxOLIlH96ONfxvYClG/PhNQ1lX7YNnjtTANJoxECooYmy/r+DDTiVvn3pHdHLopjGDD B6iQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=/MZdCQtxOS4WMjTXBL9ALIJU8/OPORnd/uYgF4h3xGc=; b=fCfgd9tS5obOvUZCW9+UiPTm/P6vTt996Cs0sI00Hk9u1uLaqGnPbAuwvqJXLKOb0q pHTIUpVhMO3FoZlWiH5HPLhrVZBlxhlvu3Vj6bl/qkyBYSuoMMlT39U9/cRpzqODdqyF J8SL31XgRIRT5W0l/HGtGx4Lnj7ByaSbayZIQ5R5Qmjcgp5FLhqCqGiebUj8xWmuwwR/ gYeuTybe8iou2vUcGWqTMM9HUyEQSJOLP5D2aOesdRwQsbvrf//2uNBEAJhBo/BT27Bk UWBCNztA71EmBrIFQ7bgZZ4ks91V2WPGE4qxjpIyDtJnjZSAs36bK1vO7PgYCuW8fT1P q+hA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b="X/6Sk1Sv"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id t4si2936358vsh.390.2021.05.12.03.33.22 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 12 May 2021 03:33:22 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b="X/6Sk1Sv"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:35076 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lgmB8-0003ml-11 for patch@linaro.org; Wed, 12 May 2021 06:33:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60198) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lglzI-0003Z1-Ge for qemu-devel@nongnu.org; Wed, 12 May 2021 06:21:08 -0400 Received: from mail-wr1-x42f.google.com ([2a00:1450:4864:20::42f]:41926) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lglzF-0003f8-Tg for qemu-devel@nongnu.org; Wed, 12 May 2021 06:21:08 -0400 Received: by mail-wr1-x42f.google.com with SMTP id d11so23035970wrw.8 for ; Wed, 12 May 2021 03:21:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/MZdCQtxOS4WMjTXBL9ALIJU8/OPORnd/uYgF4h3xGc=; b=X/6Sk1SvaMtkjQRD9QBWRy5AdbxYD/Ksjq1qLZOSJ/MAaKFf4jiuUWv4xiRdOFi6oJ 1i33aoNgTurr4lePD5oIN1oZjhfryKIWfhPLqDCJSymHcT9z0Yctu1tL0G0l+SSeDkhF N74iq3QFkOS/vD0d90TNO6hbuf7APt0MZkZPf6WjwI3aBJSHWh8o5SIuFLMTluVGdYrz NKFSGlJoMT+Ms6DIWIT6R3w868RFr7W9adfkYXeddgY3SMD8PMMG0RhomjP/IQvz1q91 8Ov/jn6kA/kkDjTVK5r27AFM3oR3XakWAlj9J5LX+TH/TReKWN3GI/PlamqWCJd3r60D okYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/MZdCQtxOS4WMjTXBL9ALIJU8/OPORnd/uYgF4h3xGc=; b=g9edZgJaCU9aGFZNRToTg6SSO20YG0lawEd5NQp2BKKWY38FYj22AFafBSFZ1aC6pD 8jIrUMDP8qR+dhLM+MBguHqvRdbdJRVu/8qAzdOCz0I/YFiDiqcxW9iwcRgmo6gDCcC1 I2vKMSwYt1cmuOBpvyN4X/5i71yjmrwoz5TuLKjQfpbU6oNYEPB/4mWfgRFWWOuRe1GX nKkcweUgeEoDNYUIWM9QMPCXhsCiqnztvJOp5LgwxgGU+5oDrzgJ1fM899kymk9QLN8c BD9l1J3y5ywaBlPogONK8QOvSedPpmGbs0HSJwwuvMdxwLk/JuvkA+pAsyLFGVAtuv4Q nONQ== X-Gm-Message-State: AOAM531J2w6C9/qIH+ZIbr3DLLRWRnFy5CxfvhbKMQV11vmRSh66xodc r9LXrvxgxOIIbh6+wHemVOO3Fg== X-Received: by 2002:a5d:4304:: with SMTP id h4mr44658971wrq.210.1620814864487; Wed, 12 May 2021 03:21:04 -0700 (PDT) Received: from zen.linaroharston ([51.148.130.216]) by smtp.gmail.com with ESMTPSA id c15sm14931122wrd.49.2021.05.12.03.20.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 May 2021 03:20:58 -0700 (PDT) Received: from zen.lan (localhost [127.0.0.1]) by zen.linaroharston (Postfix) with ESMTP id B23B71FF92; Wed, 12 May 2021 11:20:51 +0100 (BST) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: qemu-devel@nongnu.org Subject: [PATCH v3 06/31] docker: Add Hexagon image Date: Wed, 12 May 2021 11:20:26 +0100 Message-Id: <20210512102051.12134-7-alex.bennee@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210512102051.12134-1-alex.bennee@linaro.org> References: <20210512102051.12134-1-alex.bennee@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::42f; envelope-from=alex.bennee@linaro.org; helo=mail-wr1-x42f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fam@euphon.net, Alessandro Di Federico , Thomas Huth , berrange@redhat.com, Brian Cain , f4bug@amsat.org, Wainer dos Santos Moschetta , Willian Rampazzo , stefanha@redhat.com, crosa@redhat.com, pbonzini@redhat.com, =?utf-8?q?Alex_Benn=C3=A9e?= , aurelien@aurel32.net Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: Alessandro Di Federico This image is a little special because it takes a long time to build. As such most users don't want to be doing that and just pull random binaries from the ether as intended by the container gods. This involves someone with credentials and a beefy machine running: make docker-image-debian-hexagon-cross V=1 NOCACHE=1 J=30 docker tag qemu/debian-hexagon-cross registry.gitlab.com/qemu-project/qemu/qemu/debian-hexagon-cross docker push registry.gitlab.com/qemu-project/qemu/qemu/debian-hexagon-cross With a suitable binary in the "cloud" a normal user will run: make docker-image-debian-hexagon-cross or have it run for them through the dependency mechanism of our over-engineered makefiles and get the binary they wanted. There are a few wrinkles of course including needing to tweak the final image to have the credentials of the user so we can actually do our cross compiles. Tested-by: Philippe Mathieu-Daudé Reviewed-by: Willian Rampazzo Signed-off-by: Alex Bennée Cc: Alessandro Di Federico Cc: Philippe Mathieu-Daudé Cc: Brian Cain --- vAJB: - dropped all r-b/sob due to rewrite - split into a multi-stage dockerfile - make dockerfile standalone - play fugly make games v2 - update LLVM version per BN7PR02MB4194DF5752EF3BADE858018DB8799@BN7PR02MB4194.namprd02.prod.outlook.com v3 - add ninja to the spec (in stage 2) --- MAINTAINERS | 2 + tests/docker/Makefile.include | 22 +++ .../dockerfiles/debian-hexagon-cross.docker | 45 ++++++ .../build-toolchain.sh | 141 ++++++++++++++++++ 4 files changed, 210 insertions(+) create mode 100644 tests/docker/dockerfiles/debian-hexagon-cross.docker create mode 100755 tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh -- 2.20.1 diff --git a/MAINTAINERS b/MAINTAINERS index 06642d9799..30f1e7d4e4 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -197,6 +197,8 @@ F: linux-user/hexagon/ F: tests/tcg/hexagon/ F: disas/hexagon.c F: default-configs/targets/hexagon-linux-user.mak +F: docker/dockerfiles/debian-hexagon-cross.docker +F: docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh HPPA (PA-RISC) TCG CPUs M: Richard Henderson diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include index 820423d718..8967ecf118 100644 --- a/tests/docker/Makefile.include +++ b/tests/docker/Makefile.include @@ -148,6 +148,28 @@ docker-image-debian-s390x-cross: docker-image-debian10 docker-image-debian-sh4-cross: docker-image-debian10 docker-image-debian-sparc64-cross: docker-image-debian10 +# +# The build rule for hexagon-cross is special in so far for most of +# the time we don't want to build it. While dockers caching does avoid +# this most of the time sometimes we want to force the issue. +# +docker-image-debian-hexagon-cross: $(DOCKER_FILES_DIR)/debian-hexagon-cross.docker + $(if $(NOCACHE), \ + $(call quiet-command, \ + $(DOCKER_SCRIPT) build -t qemu/debian-hexagon-cross -f $< \ + $(if $V,,--quiet) --no-cache \ + --registry $(DOCKER_REGISTRY) --extra-files \ + $(DOCKER_FILES_DIR)/debian-hexagon-cross.docker.d/build-toolchain.sh, \ + "BUILD", "debian-hexagon-cross"), \ + $(call quiet-command, \ + $(DOCKER_SCRIPT) fetch $(if $V,,--quiet) \ + qemu/debian-hexagon-cross $(DOCKER_REGISTRY), \ + "FETCH", "debian-hexagon-cross") \ + $(call quiet-command, \ + $(DOCKER_SCRIPT) update $(if $V,,--quiet) \ + qemu/debian-hexagon-cross --add-current-user, \ + "PREPARE", "debian-hexagon-cross")) + # Specialist build images, sometimes very limited tools docker-image-debian-tricore-cross: docker-image-debian10 docker-image-debian-all-test-cross: docker-image-debian10 diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker b/tests/docker/dockerfiles/debian-hexagon-cross.docker new file mode 100644 index 0000000000..d5dc299dc1 --- /dev/null +++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker @@ -0,0 +1,45 @@ +# +# Docker Hexagon cross-compiler target +# +# This docker target is used for building hexagon tests. As it also +# needs to be able to build QEMU itself in CI we include it's +# build-deps. It is also a "stand-alone" image so as not to be +# triggered by re-builds on other base images given it takes a long +# time to build. +# +FROM qemu/debian10 + +# Install common build utilities +RUN apt update && \ + DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \ + DEBIAN_FRONTEND=noninteractive eatmydata \ + apt install -y --no-install-recommends \ + bison \ + cmake \ + flex \ + lld \ + rsync \ + wget + +ENV TOOLCHAIN_INSTALL /usr/local +ENV ROOTFS /usr/local + +ENV LLVM_URL https://github.com/llvm/llvm-project/archive/bfcd21876adc3498065e4da92799f613e730d475.tar.gz +ENV MUSL_URL https://github.com/quic/musl/archive/aff74b395fbf59cd7e93b3691905aa1af6c0778c.tar.gz +ENV LINUX_URL https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.6.18.tar.xz + +ADD build-toolchain.sh /root/hexagon-toolchain/build-toolchain.sh + +RUN cd /root/hexagon-toolchain && ./build-toolchain.sh + +FROM debian:buster-slim +# Duplicate deb line as deb-src +RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list +# Install QEMU build deps for use in CI +RUN apt update && \ + DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \ + DEBIAN_FRONTEND=noninteractive eatmydata apt install -yy git ninja-build && \ + DEBIAN_FRONTEND=noninteractive eatmydata \ + apt build-dep -yy --arch-only qemu +COPY --from=0 /usr/local /usr/local +ENV PATH $PATH:/usr/local/bin/ diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh b/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh new file mode 100755 index 0000000000..19b1c9f83e --- /dev/null +++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh @@ -0,0 +1,141 @@ +#!/bin/bash + +set -e + +BASE=$(readlink -f ${PWD}) + +TOOLCHAIN_INSTALL=$(readlink -f "$TOOLCHAIN_INSTALL") +ROOTFS=$(readlink -f "$ROOTFS") + +TOOLCHAIN_BIN=${TOOLCHAIN_INSTALL}/bin +HEX_SYSROOT=${TOOLCHAIN_INSTALL}/hexagon-unknown-linux-musl +HEX_TOOLS_TARGET_BASE=${HEX_SYSROOT}/usr + +function cdp() { + DIR="$1" + mkdir -p "$DIR" + cd "$DIR" +} + +function fetch() { + DIR="$1" + URL="$2" + TEMP="$(readlink -f "$PWD/tmp.tar.gz")" + wget --quiet "$URL" -O "$TEMP" + cdp "$DIR" + tar xaf "$TEMP" --strip-components=1 + rm "$TEMP" + cd - +} + +build_llvm_clang() { + fetch "$BASE/llvm-project" "$LLVM_URL" + cdp "$BASE/build-llvm" + + cmake -G Ninja \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_PREFIX=${TOOLCHAIN_INSTALL} \ + -DLLVM_ENABLE_LLD=ON \ + -DLLVM_TARGETS_TO_BUILD="Hexagon" \ + -DLLVM_ENABLE_PROJECTS="clang;lld" \ + "$BASE/llvm-project/llvm" + ninja all install + cd ${TOOLCHAIN_BIN} + ln -sf clang hexagon-unknown-linux-musl-clang + ln -sf clang++ hexagon-unknown-linux-musl-clang++ + ln -sf llvm-ar hexagon-unknown-linux-musl-ar + ln -sf llvm-objdump hexagon-unknown-linux-musl-objdump + ln -sf llvm-objcopy hexagon-unknown-linux-musl-objcopy + ln -sf llvm-readelf hexagon-unknown-linux-musl-readelf + ln -sf llvm-ranlib hexagon-unknown-linux-musl-ranlib + + # workaround for now: + cat < hexagon-unknown-linux-musl.cfg +-G0 --sysroot=${HEX_SYSROOT} +EOF +} + +build_clang_rt() { + cdp "$BASE/build-clang_rt" + cmake -G Ninja \ + -DCMAKE_BUILD_TYPE=Release \ + -DLLVM_CONFIG_PATH="$BASE/build-llvm/bin/llvm-config" \ + -DCMAKE_ASM_FLAGS="-G0 -mlong-calls -fno-pic --target=hexagon-unknown-linux-musl " \ + -DCMAKE_SYSTEM_NAME=Linux \ + -DCMAKE_C_COMPILER="${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang" \ + -DCMAKE_ASM_COMPILER="${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang" \ + -DCMAKE_INSTALL_PREFIX=${HEX_TOOLS_TARGET_BASE} \ + -DCMAKE_CROSSCOMPILING=ON \ + -DCMAKE_C_COMPILER_FORCED=ON \ + -DCMAKE_CXX_COMPILER_FORCED=ON \ + -DCOMPILER_RT_BUILD_BUILTINS=ON \ + -DCOMPILER_RT_BUILTINS_ENABLE_PIC=OFF \ + -DCMAKE_SIZEOF_VOID_P=4 \ + -DCOMPILER_RT_OS_DIR= \ + -DCAN_TARGET_hexagon=1 \ + -DCAN_TARGET_x86_64=0 \ + -DCOMPILER_RT_SUPPORTED_ARCH=hexagon \ + -DLLVM_ENABLE_PROJECTS="compiler-rt" \ + "$BASE/llvm-project/compiler-rt" + ninja install-compiler-rt +} + +build_musl_headers() { + fetch "$BASE/musl" "$MUSL_URL" + cd "$BASE/musl" + make clean + CC=${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang \ + CROSS_COMPILE=hexagon-unknown-linux-musl \ + LIBCC=${HEX_TOOLS_TARGET_BASE}/lib/libclang_rt.builtins-hexagon.a \ + CROSS_CFLAGS="-G0 -O0 -mv65 -fno-builtin -fno-rounding-math --target=hexagon-unknown-linux-musl" \ + ./configure --target=hexagon --prefix=${HEX_TOOLS_TARGET_BASE} + PATH=${TOOLCHAIN_BIN}:$PATH make CROSS_COMPILE= install-headers + + cd ${HEX_SYSROOT}/.. + ln -sf hexagon-unknown-linux-musl hexagon +} + +build_kernel_headers() { + fetch "$BASE/linux" "$LINUX_URL" + mkdir -p "$BASE/build-linux" + cd "$BASE/linux" + make O=../build-linux ARCH=hexagon \ + KBUILD_CFLAGS_KERNEL="-mlong-calls" \ + CC=${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang \ + LD=${TOOLCHAIN_BIN}/ld.lld \ + KBUILD_VERBOSE=1 comet_defconfig + make mrproper + + cd "$BASE/build-linux" + make \ + ARCH=hexagon \ + CC=${TOOLCHAIN_BIN}/clang \ + INSTALL_HDR_PATH=${HEX_TOOLS_TARGET_BASE} \ + V=1 \ + headers_install +} + +build_musl() { + cd "$BASE/musl" + make clean + CROSS_COMPILE=hexagon-unknown-linux-musl- \ + AR=llvm-ar \ + RANLIB=llvm-ranlib \ + STRIP=llvm-strip \ + CC=clang \ + LIBCC=${HEX_TOOLS_TARGET_BASE}/lib/libclang_rt.builtins-hexagon.a \ + CFLAGS="-G0 -O0 -mv65 -fno-builtin -fno-rounding-math --target=hexagon-unknown-linux-musl" \ + ./configure --target=hexagon --prefix=${HEX_TOOLS_TARGET_BASE} + PATH=${TOOLCHAIN_BIN}/:$PATH make CROSS_COMPILE= install + cd ${HEX_TOOLS_TARGET_BASE}/lib + ln -sf libc.so ld-musl-hexagon.so + ln -sf ld-musl-hexagon.so ld-musl-hexagon.so.1 + cdp ${HEX_TOOLS_TARGET_BASE}/../lib + ln -sf ../usr/lib/ld-musl-hexagon.so.1 +} + +build_llvm_clang +build_kernel_headers +build_musl_headers +build_clang_rt +build_musl