From patchwork Wed Mar 10 19:23:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 396777 Delivered-To: patch@linaro.org Received: by 2002:a17:906:a383:0:0:0:0 with SMTP id k3csp604202ejz; Wed, 10 Mar 2021 11:30:16 -0800 (PST) X-Google-Smtp-Source: ABdhPJxnt1aOyoU6F6ZmhJNjR/GoWh/vOxTepfDZBvNNxhB0F01j6qSm+i/S+ejd//Xpgr3ksSQy X-Received: by 2002:a37:6348:: with SMTP id x69mr4179309qkb.154.1615404615856; Wed, 10 Mar 2021 11:30:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1615404615; cv=none; d=google.com; s=arc-20160816; b=J4zlkW1jm4kXfTrX0puslvtSwkOy5rFubfXdQYODBYZpCz4nE/GLK8WdTNRcc1dRO7 BVunbHJsjAuWVkmXB+Kn4TgKHwS5+6Z/R4uOUSxtVKy6s3oopQgVyvP6Z3+mCwyAdBmu Kuh5LvEfrWawbUIAmZ8H78A73nQC2ZjUAMPXFGj0WNtwdAbvdSOo6ocVFd1UuhdYaKQC CQnuwS8SJquqRHJ+190pOK/quATB3YSbT23TYXdWWG/lXL6mLx2eRbohx2CYSUq8Rjeq uev85GpgPLmILCIFrIV1FAd1h7makH54rJRsLgfICOP81r9o0mBo94TuPOGAce/5ANuW TJ4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=kMSl2nUioQRSggV946FzlETeW29Qa6scCLbuYk6OMBk=; b=TeuBEjqALdN+DTNHssP/pdzJbDwKlQkf7mTfYdnQb1xGFhTYQOHOJm0q73DEKys2BO Eh0e/ywMaUUZebgg2UK6hf0JrHkTJCT5I4dB4VafIAjZJgaebFVCJo/2711wD0qXvAOM gUycgL/IXgzLMHlPWYZz3EoXFYjRyWPJjdz3tdhg3FhOakqYstHIDJUZJHcD9MrvgAgf CRPjGkPN73BFbojypINMquHQs47tEjmk9fcANHP2rz2MUn1SOqfzlO9+gTYV8OsjEgnl sUoNgZMvTJwR0zcq2KHapPzZ+xxe8kFX0ZgJX3nqwDDcE//tal7YBblk4XUzCuCdm+V4 Soxw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=hj+BkBKm; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id t20si204803qtw.378.2021.03.10.11.30.15 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 10 Mar 2021 11:30:15 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=hj+BkBKm; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:54604 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lK4X9-0002eY-23 for patch@linaro.org; Wed, 10 Mar 2021 14:30:15 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:46256) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lK4R4-00068O-CG for qemu-devel@nongnu.org; Wed, 10 Mar 2021 14:24:05 -0500 Received: from mail-ej1-x62d.google.com ([2a00:1450:4864:20::62d]:44759) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lK4Qq-0002l8-1a for qemu-devel@nongnu.org; Wed, 10 Mar 2021 14:23:56 -0500 Received: by mail-ej1-x62d.google.com with SMTP id ox4so25392977ejb.11 for ; Wed, 10 Mar 2021 11:23:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kMSl2nUioQRSggV946FzlETeW29Qa6scCLbuYk6OMBk=; b=hj+BkBKmL4JhqNKG/Rz/JuFlWzQP14mYwqG0p9wUQXuTus4tTNk3t3MO5eN3rqAInX quv9SMCYIDfbEuCT/2ZuE7AwEWf7ZkoeRGCzK8mJOqyInXLwJq6NoO/QH0uhoYvelGkA Awe3wVBsT2rVIBZQP4GK6wcZdjlFZc1q+cmFPn6+aN9vtP6q77zAByIxH0lgqM65iZMB pp8q7LdX0KXEaa5t7J1D0T1EYe0hhJD9/mwtr9M0TRgZh8ZTelBi/WATeMBQTWgNHrBc 4wfNdgaeem9vVBIaE5Sr4jwOrBkCFimbgeFdgKrv2w3dlGD/IYPCfe76YHI32oWkBA72 8zow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kMSl2nUioQRSggV946FzlETeW29Qa6scCLbuYk6OMBk=; b=ELcHT3Hv9/trt4v/MAGd2h1hDDAiIecdPQPTXYyVGlQ/DK7PSOIaOC79q+IQP9fcJF gmpJ95eguEAoFmUuH6sAdXybsfdMJcreFAZjE65VBW0qClf8czv0tumhQRz3+BTS0Fza Sb2mm/wIq157bgDqyok4MPwVrNJKtFxD7CRv33C/0jOXQjwPAH/deTQHG4Dn5n+mW+/K sQwKe3ZhKgOEBdps0rTRrBfKCC4UU/7Pim8nzrnqfX3PJz+auXHSsv8qafAxw3suxsod W9dS/o7inWHATDZobe5DKxSXrF3T5IV//602iY5R2II3NLORu+b9+F72Gi/KoAyzLOUN 7JJQ== X-Gm-Message-State: AOAM531UrwOmtuvwnNMGtxNr/wFU9BoPPmywZuwnTCtcDHOEWQ06kzRd dTxGXgXc3Li4hDjy2qpijiAs9Q== X-Received: by 2002:a17:906:32da:: with SMTP id k26mr5226475ejk.483.1615404222674; Wed, 10 Mar 2021 11:23:42 -0800 (PST) Received: from zen.linaroharston ([51.148.130.216]) by smtp.gmail.com with ESMTPSA id q12sm183577ejy.91.2021.03.10.11.23.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Mar 2021 11:23:38 -0800 (PST) Received: from zen.lan (localhost [127.0.0.1]) by zen.linaroharston (Postfix) with ESMTP id 622C91FF91; Wed, 10 Mar 2021 19:23:32 +0000 (GMT) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: qemu-devel@nongnu.org Subject: [PATCH v1 5/7] docker: Add Hexagon image Date: Wed, 10 Mar 2021 19:23:29 +0000 Message-Id: <20210310192331.29284-6-alex.bennee@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210310192331.29284-1-alex.bennee@linaro.org> References: <20210310192331.29284-1-alex.bennee@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::62d; envelope-from=alex.bennee@linaro.org; helo=mail-ej1-x62d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fam@euphon.net, Alessandro Di Federico , berrange@redhat.com, Brian Cain , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , f4bug@amsat.org, stefanha@redhat.com, crosa@redhat.com, pbonzini@redhat.com, =?utf-8?q?Alex_Benn=C3=A9e?= , aurelien@aurel32.net Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: Alessandro Di Federico This image is a little special because it takes a long time to build. As such most users don't want to be doing that and just pull random binaries from the ether as intended by the container gods. This involves someone with credentials and a beefy machine running: make docker-image-debian-hexagon-cross V=1 NOCACHE=1 J=30 docker tag qemu/debian-hexagon-cross registry.gitlab.com/qemu-project/qemu/qemu/debian-hexagon-cross docker push registry.gitlab.com/qemu-project/qemu/qemu/debian-hexagon-cross With a suitable binary in the "cloud" a normal user will run: make docker-image-debian-hexagon-cross or have it run for them through the dependency mechanism of our over-engineered makefiles and get the binary they wanted. There are a few wrinkles of course including needing to tweak the final image to have the credentials of the user so we can actually do our cross compiles. Signed-off-by: Alex Bennée Cc: Alessandro Di Federico Cc: Philippe Mathieu-Daudé Cc: Brian Cain --- vAJB: - dropped all r-b/sob due to rewrite - split into a multi-stage dockerfile - make dockerfile standalone - play fugly make games --- MAINTAINERS | 2 + tests/docker/Makefile.include | 22 +++ .../dockerfiles/debian-hexagon-cross.docker | 44 ++++++ .../build-toolchain.sh | 141 ++++++++++++++++++ 4 files changed, 209 insertions(+) create mode 100644 tests/docker/dockerfiles/debian-hexagon-cross.docker create mode 100755 tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh -- 2.20.1 diff --git a/MAINTAINERS b/MAINTAINERS index 3456993062..712c25b077 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -196,6 +196,8 @@ F: linux-user/hexagon/ F: tests/tcg/hexagon/ F: disas/hexagon.c F: default-configs/targets/hexagon-linux-user.mak +F: docker/dockerfiles/debian-hexagon-cross.docker +F: docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh HPPA (PA-RISC) TCG CPUs M: Richard Henderson diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include index 8f66049dcc..b075dfe9ce 100644 --- a/tests/docker/Makefile.include +++ b/tests/docker/Makefile.include @@ -145,6 +145,28 @@ docker-image-debian-s390x-cross: docker-image-debian10 docker-image-debian-sh4-cross: docker-image-debian10 docker-image-debian-sparc64-cross: docker-image-debian10 +# +# The build rule for hexagon-cross is special in so far for most of +# the time we don't want to build it. While dockers caching does avoid +# this most of the time sometimes we want to force the issue. +# +docker-image-debian-hexagon-cross: $(DOCKER_FILES_DIR)/debian-hexagon-cross.docker + $(if $(NOCACHE), \ + $(call quiet-command, \ + $(DOCKER_SCRIPT) build -t qemu/debian-hexagon-cross -f $< \ + $(if $V,,--quiet) --no-cache \ + --registry $(DOCKER_REGISTRY) --extra-files \ + $(DOCKER_FILES_DIR)/debian-hexagon-cross.docker.d/build-toolchain.sh, \ + "BUILD", "debian-hexagon-cross"), \ + $(call quiet-command, \ + $(DOCKER_SCRIPT) fetch $(if $V,,--quiet) \ + qemu/debian-hexagon-cross $(DOCKER_REGISTRY), \ + "FETCH", "debian-hexagon-cross") \ + $(call quiet-command, \ + $(DOCKER_SCRIPT) update $(if $V,,--quiet) \ + qemu/debian-hexagon-cross --add-current-user, \ + "PREPARE", "debian-hexagon-cross")) + # Specialist build images, sometimes very limited tools docker-image-debian-tricore-cross: docker-image-debian10 docker-image-debian-all-test-cross: docker-image-debian10 diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker b/tests/docker/dockerfiles/debian-hexagon-cross.docker new file mode 100644 index 0000000000..b6fb651450 --- /dev/null +++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker @@ -0,0 +1,44 @@ +# +# Docker Hexagon cross-compiler target +# +# This docker target is used for building hexagon tests. As it also +# needs to be able to build QEMU itself in CI we include it's +# build-deps. It is also a "stand-alone" image so as not to be +# triggered by re-builds on other base images given it takes a long +# time to build. +# +FROM qemu/debian10 + +# Install common build utilities +RUN apt update && \ + DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \ + DEBIAN_FRONTEND=noninteractive eatmydata \ + apt install -y --no-install-recommends \ + bison \ + cmake \ + flex \ + lld \ + rsync \ + wget + +ENV TOOLCHAIN_INSTALL /usr/local +ENV ROOTFS /usr/local + +ENV LLVM_URL https://github.com/llvm/llvm-project/archive/3d8149c2a1228609fd7d7c91a04681304a2f0ca9.tar.gz +ENV MUSL_URL https://github.com/quic/musl/archive/aff74b395fbf59cd7e93b3691905aa1af6c0778c.tar.gz +ENV LINUX_URL https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.6.18.tar.xz + +ADD build-toolchain.sh /root/hexagon-toolchain/build-toolchain.sh + +RUN cd /root/hexagon-toolchain && ./build-toolchain.sh + +FROM debian:buster-slim +# Duplicate deb line as deb-src +RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list +# Install QEMU build deps for use in CI +RUN apt update && \ + DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \ + DEBIAN_FRONTEND=noninteractive eatmydata \ + apt build-dep -yy --arch-only qemu +COPY --from=0 /usr/local /usr/local +ENV PATH $PATH:/usr/local/bin/ diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh b/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh new file mode 100755 index 0000000000..19b1c9f83e --- /dev/null +++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh @@ -0,0 +1,141 @@ +#!/bin/bash + +set -e + +BASE=$(readlink -f ${PWD}) + +TOOLCHAIN_INSTALL=$(readlink -f "$TOOLCHAIN_INSTALL") +ROOTFS=$(readlink -f "$ROOTFS") + +TOOLCHAIN_BIN=${TOOLCHAIN_INSTALL}/bin +HEX_SYSROOT=${TOOLCHAIN_INSTALL}/hexagon-unknown-linux-musl +HEX_TOOLS_TARGET_BASE=${HEX_SYSROOT}/usr + +function cdp() { + DIR="$1" + mkdir -p "$DIR" + cd "$DIR" +} + +function fetch() { + DIR="$1" + URL="$2" + TEMP="$(readlink -f "$PWD/tmp.tar.gz")" + wget --quiet "$URL" -O "$TEMP" + cdp "$DIR" + tar xaf "$TEMP" --strip-components=1 + rm "$TEMP" + cd - +} + +build_llvm_clang() { + fetch "$BASE/llvm-project" "$LLVM_URL" + cdp "$BASE/build-llvm" + + cmake -G Ninja \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_PREFIX=${TOOLCHAIN_INSTALL} \ + -DLLVM_ENABLE_LLD=ON \ + -DLLVM_TARGETS_TO_BUILD="Hexagon" \ + -DLLVM_ENABLE_PROJECTS="clang;lld" \ + "$BASE/llvm-project/llvm" + ninja all install + cd ${TOOLCHAIN_BIN} + ln -sf clang hexagon-unknown-linux-musl-clang + ln -sf clang++ hexagon-unknown-linux-musl-clang++ + ln -sf llvm-ar hexagon-unknown-linux-musl-ar + ln -sf llvm-objdump hexagon-unknown-linux-musl-objdump + ln -sf llvm-objcopy hexagon-unknown-linux-musl-objcopy + ln -sf llvm-readelf hexagon-unknown-linux-musl-readelf + ln -sf llvm-ranlib hexagon-unknown-linux-musl-ranlib + + # workaround for now: + cat < hexagon-unknown-linux-musl.cfg +-G0 --sysroot=${HEX_SYSROOT} +EOF +} + +build_clang_rt() { + cdp "$BASE/build-clang_rt" + cmake -G Ninja \ + -DCMAKE_BUILD_TYPE=Release \ + -DLLVM_CONFIG_PATH="$BASE/build-llvm/bin/llvm-config" \ + -DCMAKE_ASM_FLAGS="-G0 -mlong-calls -fno-pic --target=hexagon-unknown-linux-musl " \ + -DCMAKE_SYSTEM_NAME=Linux \ + -DCMAKE_C_COMPILER="${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang" \ + -DCMAKE_ASM_COMPILER="${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang" \ + -DCMAKE_INSTALL_PREFIX=${HEX_TOOLS_TARGET_BASE} \ + -DCMAKE_CROSSCOMPILING=ON \ + -DCMAKE_C_COMPILER_FORCED=ON \ + -DCMAKE_CXX_COMPILER_FORCED=ON \ + -DCOMPILER_RT_BUILD_BUILTINS=ON \ + -DCOMPILER_RT_BUILTINS_ENABLE_PIC=OFF \ + -DCMAKE_SIZEOF_VOID_P=4 \ + -DCOMPILER_RT_OS_DIR= \ + -DCAN_TARGET_hexagon=1 \ + -DCAN_TARGET_x86_64=0 \ + -DCOMPILER_RT_SUPPORTED_ARCH=hexagon \ + -DLLVM_ENABLE_PROJECTS="compiler-rt" \ + "$BASE/llvm-project/compiler-rt" + ninja install-compiler-rt +} + +build_musl_headers() { + fetch "$BASE/musl" "$MUSL_URL" + cd "$BASE/musl" + make clean + CC=${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang \ + CROSS_COMPILE=hexagon-unknown-linux-musl \ + LIBCC=${HEX_TOOLS_TARGET_BASE}/lib/libclang_rt.builtins-hexagon.a \ + CROSS_CFLAGS="-G0 -O0 -mv65 -fno-builtin -fno-rounding-math --target=hexagon-unknown-linux-musl" \ + ./configure --target=hexagon --prefix=${HEX_TOOLS_TARGET_BASE} + PATH=${TOOLCHAIN_BIN}:$PATH make CROSS_COMPILE= install-headers + + cd ${HEX_SYSROOT}/.. + ln -sf hexagon-unknown-linux-musl hexagon +} + +build_kernel_headers() { + fetch "$BASE/linux" "$LINUX_URL" + mkdir -p "$BASE/build-linux" + cd "$BASE/linux" + make O=../build-linux ARCH=hexagon \ + KBUILD_CFLAGS_KERNEL="-mlong-calls" \ + CC=${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang \ + LD=${TOOLCHAIN_BIN}/ld.lld \ + KBUILD_VERBOSE=1 comet_defconfig + make mrproper + + cd "$BASE/build-linux" + make \ + ARCH=hexagon \ + CC=${TOOLCHAIN_BIN}/clang \ + INSTALL_HDR_PATH=${HEX_TOOLS_TARGET_BASE} \ + V=1 \ + headers_install +} + +build_musl() { + cd "$BASE/musl" + make clean + CROSS_COMPILE=hexagon-unknown-linux-musl- \ + AR=llvm-ar \ + RANLIB=llvm-ranlib \ + STRIP=llvm-strip \ + CC=clang \ + LIBCC=${HEX_TOOLS_TARGET_BASE}/lib/libclang_rt.builtins-hexagon.a \ + CFLAGS="-G0 -O0 -mv65 -fno-builtin -fno-rounding-math --target=hexagon-unknown-linux-musl" \ + ./configure --target=hexagon --prefix=${HEX_TOOLS_TARGET_BASE} + PATH=${TOOLCHAIN_BIN}/:$PATH make CROSS_COMPILE= install + cd ${HEX_TOOLS_TARGET_BASE}/lib + ln -sf libc.so ld-musl-hexagon.so + ln -sf ld-musl-hexagon.so ld-musl-hexagon.so.1 + cdp ${HEX_TOOLS_TARGET_BASE}/../lib + ln -sf ../usr/lib/ld-musl-hexagon.so.1 +} + +build_llvm_clang +build_kernel_headers +build_musl_headers +build_clang_rt +build_musl