From patchwork Wed Sep 18 12:54:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Maydell X-Patchwork-Id: 829414 Delivered-To: patch@linaro.org Received: by 2002:a5d:44c6:0:b0:367:895a:4699 with SMTP id z6csp764396wrr; Wed, 18 Sep 2024 05:55:56 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCX9yqj5Dd3DOzyiQ1geOXbU6UE9pn05mekzGlVRbf/ZjAxCyJJlpNUea1yrb96dDUFqAnnwWw==@linaro.org X-Google-Smtp-Source: AGHT+IG53FlWGBvDJQG3UERs0uZya6R/OOFDhgOnp+812OhYz9wLVMQoPHTfl7Qw5dPIuddfDii9 X-Received: by 2002:a05:622a:6:b0:458:401e:c1ed with SMTP id d75a77b69052e-4599d2238dfmr303900081cf.12.1726664155962; Wed, 18 Sep 2024 05:55:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1726664155; cv=none; d=google.com; s=arc-20240605; b=D5bCsjw002pPB+uFwNykU8N9UoyYwZqHptZgLYkIDRIAl5Lnzq2TwyFDjvbKV7ydYd pZCDT5r7iRW9OAF6w+IJh0T1YFX6/MyIhyxITUnzRTRcR/BdoShR+kjd+kLX/0u2WkzJ Jq3cDDfGTJ2+Ij+mT4YYTJs9/tQW+fu3ceqbXSM0+cP/PJO/rssDfkohx+Wqsl0baZ92 qx+y8Vs0LBiw1Qm9jdOrGpjIeLBvrYudTJC0quIWZuS20Hi9cDqI3U2kM36YfwALFpLW dbSOYt4lq5aGCU6tUoSoWl3Pb91apafr8bMWEzePPMqrjmio9oj+CfKWPCg7FL3O7qPo EMFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=Z3xuY/dn/VE1/VAQJG02uSRnCOARSaZWLd5XFaTF7XQ=; fh=Vf8r+wNKcA5qEgoqm5j6VviSy9oOrKG0V9LUKtvLdlI=; b=RMftFMDi7lQ3sWJ2RZ6+pLUJa3tGzzxfcrY4LIUQw5SDEfKFBnX4PaHIFz7QWw7v+H i3lpp2tL9OPGgGsmAxzqd+16xbb28Mb6EInpQQdC+va8JIgBU8IsbRZ6wqmlRw2Ut298 mgyDum457j69WD7WgKPSyMuESXemL08VYan6cbjV5uEQ5ZHCgvlsn6UK1PrHYCDMOzZP VshAXI0hpvIfDCM20WUkYoWrr5jXEUXx7wY2A6U1x4fwVGwAMUpILvTb5eS2rrygPfVw LGhz4DdkbtVinvSO50+U9zj42t8rN/A5759TtnmrdqTe75sLzipn5m6WetyFupOuUChU U8RQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="r1iI/UB4"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org; dara=neutral header.i=@linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id d75a77b69052e-459aaf4b749si101089181cf.735.2024.09.18.05.55.55 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 18 Sep 2024 05:55:55 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="r1iI/UB4"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org; dara=neutral header.i=@linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1squCx-0003Ye-CB; Wed, 18 Sep 2024 08:54:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1squCu-0003UX-5A for qemu-devel@nongnu.org; Wed, 18 Sep 2024 08:54:56 -0400 Received: from mail-wm1-x32e.google.com ([2a00:1450:4864:20::32e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1squCs-0002pJ-2F for qemu-devel@nongnu.org; Wed, 18 Sep 2024 08:54:55 -0400 Received: by mail-wm1-x32e.google.com with SMTP id 5b1f17b1804b1-42e5e758093so30598235e9.1 for ; Wed, 18 Sep 2024 05:54:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1726664091; x=1727268891; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Z3xuY/dn/VE1/VAQJG02uSRnCOARSaZWLd5XFaTF7XQ=; b=r1iI/UB4xmJac71a0WPt7pWuX2HqDb/MzMBnkH3zDy4Cj/QXzvMHQTAROp/UZpQVOw StqPbra589wrJTl87OkKpmmzC8AnNxtvbwmPwKIkYFiGOA07BZ6nxPpJlDm/UkRjsW1C V2jD0/RmybTqtmRsdvpEYesVhIe8IMKM3Lic21xyCqhEc6sPOXnfiLuKC1G3YGXp4CQS j9PDfb2ltvL+Bx2XNILKmD38i647f9XFh0byECm/RxH0oVXhvpIjisbzcgeGcLlVt6zi 9HIFvKOBUtODEUXoFOhH4TSTB5YNbl40GjGRSsM+NKM0shxtq8GEnWWeY034nj22WDks XxGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726664091; x=1727268891; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Z3xuY/dn/VE1/VAQJG02uSRnCOARSaZWLd5XFaTF7XQ=; b=Yhn8SA5NGce94eKou6tnSG3CV4eSfFvduE0YrOb7qUCWzEr2xGWRn+N77C/2V5kLvr 0RvMnjmCANeViirZ106NqgsZRiexnjRfIDq8pSBaZzg2OAfje0rxbRsVFyN98Z4jX9AY CKho/tascxIgjSj0YSWF5KPo2qV+k9c6sQJEMxPBlSEtY4fslaJGqXibTrke2IFIRcfN kDEu0X3YCWGrgwz2WqJublYlGagWoGj/bRBa6BYk6nE9fntX5tw94pu8DHCM32yzRGn2 gmmoaWCSEELRAdyfX71WCvPGKAk+ZTNV3EiH2MslIBfqlxuMuJcG8FXyxzv5UWwDuOlS tdww== X-Gm-Message-State: AOJu0YyGCZah9xj4ttlAm661jozI9I82qhb2bT0+fB7id+1W7+uy6gQO PXpRFAkU0gDdQfvdEELOmAVITc2iTGfTsBiNYM/sFBuPTfcyr33XN+Hl+S1ZV+C1R+0IamIHT+V X X-Received: by 2002:a05:600c:3b1d:b0:42c:c401:6d8b with SMTP id 5b1f17b1804b1-42d9070af3emr139345795e9.7.1726664091456; Wed, 18 Sep 2024 05:54:51 -0700 (PDT) Received: from orth.archaic.org.uk (orth.archaic.org.uk. [2001:8b0:1d0::2]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42e7051335fsm16410645e9.30.2024.09.18.05.54.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Sep 2024 05:54:51 -0700 (PDT) From: Peter Maydell To: qemu-devel@nongnu.org Cc: =?utf-8?q?Alex_Benn=C3=A9e?= , =?utf-8?q?Philipp?= =?utf-8?q?e_Mathieu-Daud=C3=A9?= , Thomas Huth , Wainer dos Santos Moschetta Subject: [PATCH 1/2] .gitlab-ci.d: Split build and test in cross build job templates Date: Wed, 18 Sep 2024 13:54:48 +0100 Message-Id: <20240918125449.3125571-2-peter.maydell@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240918125449.3125571-1-peter.maydell@linaro.org> References: <20240918125449.3125571-1-peter.maydell@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::32e; envelope-from=peter.maydell@linaro.org; helo=mail-wm1-x32e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org In the native_build_job_template we have separate steps in the script for the build and the test steps. This is helpful because then gitlab will give separate timestamps in the log view for each, and you can see how long it took to compile vs how long to test. In the templates in crossbuild-template.yml, however, we do both the build and test in a single 'make' invocation, and so we don't get the separate timing information. Split the build and test, in the same way we do in the native build template. This will also give us a place to separate out how parallel we want to do the build by default from how parallel we want to do the tests by default, which might be helpful in future. Signed-off-by: Peter Maydell Reviewed-by: Thomas Huth --- Gitlab seems to be inconsistent about whether you get the separate time-taken bubbles for each line of the script section or not; but certainly if we don't split into separate 'make' invocations we definitely can't get the timing info... --- .gitlab-ci.d/crossbuild-template.yml | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml index 2ce0432eb77..d1cb7a35dbf 100644 --- a/.gitlab-ci.d/crossbuild-template.yml +++ b/.gitlab-ci.d/crossbuild-template.yml @@ -9,6 +9,7 @@ when: always timeout: 80m before_script: + - JOBS=$(expr $(nproc) + 1) - cat /packages.txt script: - export CCACHE_BASEDIR="$(pwd)" @@ -24,7 +25,11 @@ i386-softmmu microblaze-softmmu mips-softmmu mipsel-softmmu mips64-softmmu ppc-softmmu riscv32-softmmu sh4-softmmu sparc-softmmu xtensa-softmmu $CROSS_SKIP_TARGETS" - - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS + - make -j"$JOBS" all check-build + - if test -n "$MAKE_CHECK_ARGS"; + then + $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ; + fi - if grep -q "EXESUF=.exe" config-host.mak; then make installer; version="$(git describe --match v[0-9]* 2>/dev/null || git rev-parse --short HEAD)"; @@ -46,6 +51,8 @@ paths: - ccache/ key: "$CI_JOB_NAME" + before_script: + - JOBS=$(expr $(nproc) + 1) script: - export CCACHE_BASEDIR="$(pwd)" - export CCACHE_DIR="$CCACHE_BASEDIR/ccache" @@ -55,7 +62,11 @@ - cd build - ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS --disable-tools --enable-${ACCEL:-kvm} $EXTRA_CONFIGURE_OPTS - - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS + - make -j"$JOBS" all check-build + - if test -n "$MAKE_CHECK_ARGS"; + then + $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ; + fi .cross_user_build_job: extends: .base_job_template @@ -65,6 +76,8 @@ paths: - ccache/ key: "$CI_JOB_NAME" + before_script: + - JOBS=$(expr $(nproc) + 1) script: - export CCACHE_BASEDIR="$(pwd)" - export CCACHE_DIR="$CCACHE_BASEDIR/ccache" @@ -76,7 +89,11 @@ alpha-linux-user m68k-linux-user microblazeel-linux-user or1k-linux-user ppc-linux-user sparc-linux-user xtensa-linux-user $CROSS_SKIP_TARGETS" - - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS + - make -j"$JOBS" all check-build + - if test -n "$MAKE_CHECK_ARGS"; + then + $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ; + fi # We can still run some tests on some of our cross build jobs. They can add this # template to their extends to save the build logs and test results From patchwork Wed Sep 18 12:54:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Maydell X-Patchwork-Id: 829412 Delivered-To: patch@linaro.org Received: by 2002:a5d:44c6:0:b0:367:895a:4699 with SMTP id z6csp764261wrr; Wed, 18 Sep 2024 05:55:34 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXdPYhwlmH5W9uPn6J1xymF9Vdn1a9nUf8anHdxKHYEZAy7ZlQotK/OFM1eyCSJL9UiCLxJhA==@linaro.org X-Google-Smtp-Source: AGHT+IFpJGS/t2D6CZ5SKUjMvVYdrlXzjeQzkH98VDIk7T8TM1Ff1McyvXsinScTO3quWWY4cZ4b X-Received: by 2002:a05:622a:298b:b0:458:20df:d65e with SMTP id d75a77b69052e-4599d250bfbmr292468521cf.29.1726664134240; Wed, 18 Sep 2024 05:55:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1726664134; cv=none; d=google.com; s=arc-20240605; b=RcPTUoI5mt55PVzq48knC08hGbv/VdYbIspE23UCv4cyum4GRKGD/jcSUnecpUEIft gx2Zde5eIPWpmrP2bU2rfwccyEOK0oxTHLnon/u8IYvK53dc3BlNwCwio0awqRK0cGef eMMVxdOv24hj0OvycSnbCYY8xz+iEo29+VkVr2/mYt5vcCFnHCeDapOd3Fj3FhNv5H8P wzDKAFI966WA4qCfqnM/gztN6VDVJIY2vQrw+bWoEXIl3J12zKNXjC/tm2pLn9GaSTEL nEKncNu5inv9WeVW+0OsGFAqceOOuGMlTDeMkp78JuyOpAK+wBAYRftvQv/wzh7+lEYI E9lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=lG+Q8S7Fct5ef/VuA95TQ7RdVMOBfi9LTalM/pVY76g=; fh=Vf8r+wNKcA5qEgoqm5j6VviSy9oOrKG0V9LUKtvLdlI=; b=ZW0fFtNf8rpYnN5E0AdildGQk26D2Vp92F6qFdPYaswOQpvsGfjEfM8nasgzYc1cAk yseimqZXXrfTaGJNUW4fzwvU2vd0KnRECHUfy24a9dMYp6oRbVides9MstYnqpXL4d6z XIzCwD8ufI6AjsAhlhj4UF2HApThdVGG89mpn0lIUlxAIoA1pOh0W/z+zfl0FeJ5IKVS QF2HIzGeXxS+94FgR+MKtvgi+3fmO6bD+9Nu2j68ANA10Y6Nfj8dn/O6diSWeYaNDNZy WTQw1lRZS53DPcXYQ/awPQZzyHym+FLTmcg/dpMV+oQYr7sQDZPgDKjj/R5mB2yZAl5P uPWg==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=nZ4uQSzi; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org; dara=neutral header.i=@linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id d75a77b69052e-459aaf4c1f4si101466071cf.762.2024.09.18.05.55.34 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 18 Sep 2024 05:55:34 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=nZ4uQSzi; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org; dara=neutral header.i=@linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1squCz-0003gi-PC; Wed, 18 Sep 2024 08:55:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1squCv-0003Yd-OW for qemu-devel@nongnu.org; Wed, 18 Sep 2024 08:54:59 -0400 Received: from mail-wr1-x42a.google.com ([2a00:1450:4864:20::42a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1squCs-0002pO-2F for qemu-devel@nongnu.org; Wed, 18 Sep 2024 08:54:57 -0400 Received: by mail-wr1-x42a.google.com with SMTP id ffacd0b85a97d-374d29ad8a7so4756576f8f.2 for ; Wed, 18 Sep 2024 05:54:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1726664092; x=1727268892; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lG+Q8S7Fct5ef/VuA95TQ7RdVMOBfi9LTalM/pVY76g=; b=nZ4uQSziFqvC6DmS1XVLpOFijMpRNxsA1stNjDT+LL1z3bQbuQwZZdYtEnP0IiS1Yn pIUbTMn7sK7a6xOg92PdNmj160kPtlewD/kM0+djNNHDjfZX9B48D70OEZWTEZTC8Vy8 yIL7K9MeMajcXZIqjwwu3biM5mHsZr23so0q/lrO0h4FA3BEinNozNTztXyf/SnrmYlL L9L6ri9zjJxlRcuW7Cgxi5UWf9SLq8CVZv+/rBteiFR0C7G9amOUo4k9AINSnigkfxed QYu2pcG/yGJBv8pWc4hTUqEZZP4Ls38kaGa7moqwt3neV6BY6aA/f54RtoJan0Mgombq fv5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726664092; x=1727268892; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lG+Q8S7Fct5ef/VuA95TQ7RdVMOBfi9LTalM/pVY76g=; b=eVbLnWGpk1Ws7aKtq0adq940phwUn5tG3XLixgtKG1VIyG7s534gFy2A8DM11SxDM4 stbz+hdsvbqylqSdiqFTwaXmKbKhjo0h8Rsw6cgGKuW4CG7Tf6mZLSg63P4Eiu7C4OxE EXOiAQ013gR4koZXmPW59Y45stqUTH7SgDqyCvyfF7zIrU4MJgf8R5tRtkCYRXL8E6to 0d1S6CjHhhypOVxVDJV9RhTxAf4fipuAwObV3JmgjInjvCbTEvNoW4vNDzfBd+tZQUmp wldVdgNBRKAIXE9XFyJKkSKb3bfNMZGRBTlmmsCQZp6GYcrGPUbs9A42aF3rf74MsQSP dDdw== X-Gm-Message-State: AOJu0Yxo5J6B4b45QcXNwSCA6KBkESs6UVjakoJIKCdDkLtd2k4X1unq T1N73bhL+quSVoinGYet2l01aqL65CMjWkP7TDmuDes1GR/6yg+7CG5olDfpS4x1aYatAidB5SF h X-Received: by 2002:a5d:6789:0:b0:374:b960:f847 with SMTP id ffacd0b85a97d-378c2d51750mr11808484f8f.41.1726664092065; Wed, 18 Sep 2024 05:54:52 -0700 (PDT) Received: from orth.archaic.org.uk (orth.archaic.org.uk. [2001:8b0:1d0::2]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42e7051335fsm16410645e9.30.2024.09.18.05.54.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Sep 2024 05:54:51 -0700 (PDT) From: Peter Maydell To: qemu-devel@nongnu.org Cc: =?utf-8?q?Alex_Benn=C3=A9e?= , =?utf-8?q?Philipp?= =?utf-8?q?e_Mathieu-Daud=C3=A9?= , Thomas Huth , Wainer dos Santos Moschetta Subject: [PATCH 2/2] .gitlab-ci.d: Make separate collapsible log sections for build and test Date: Wed, 18 Sep 2024 13:54:49 +0100 Message-Id: <20240918125449.3125571-3-peter.maydell@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240918125449.3125571-1-peter.maydell@linaro.org> References: <20240918125449.3125571-1-peter.maydell@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::42a; envelope-from=peter.maydell@linaro.org; helo=mail-wr1-x42a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell Reviewed-by: Thomas Huth --- .gitlab-ci.d/buildtest-template.yml | 14 ++++++++++++++ .gitlab-ci.d/buildtest.yml | 1 + .gitlab-ci.d/crossbuild-template.yml | 25 ++++++++++++++++++++++++ scripts/ci/gitlab-ci-section | 29 ++++++++++++++++++++++++++++ 4 files changed, 69 insertions(+) create mode 100644 scripts/ci/gitlab-ci-section diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml index 5f2fc7e6f49..8c69c60d215 100644 --- a/.gitlab-ci.d/buildtest-template.yml +++ b/.gitlab-ci.d/buildtest-template.yml @@ -8,8 +8,11 @@ key: "$CI_JOB_NAME" when: always before_script: + - source scripts/ci/gitlab-ci-section + - section_start setup "Pre-script setup" - JOBS=$(expr $(nproc) + 1) - cat /packages.txt + - section_end setup script: - export CCACHE_BASEDIR="$(pwd)" - export CCACHE_DIR="$CCACHE_BASEDIR/ccache" @@ -19,6 +22,7 @@ - mkdir build - cd build - ccache --zero-stats + - section_start configure "Running configure" - ../configure --enable-werror --disable-docs --enable-fdt=system ${TARGETS:+--target-list="$TARGETS"} $CONFIGURE_ARGS || @@ -27,11 +31,16 @@ then pyvenv/bin/meson configure . -Dbackend_max_links="$LD_JOBS" ; fi || exit 1; + - section_end configure + - section_start build "Building QEMU" - $MAKE -j"$JOBS" + - section_end build + - section_start test "Running tests" - if test -n "$MAKE_CHECK_ARGS"; then $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ; fi + - section_end test - ccache --show-stats # We jump some hoops in common_test_job_template to avoid @@ -54,6 +63,8 @@ stage: test image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG script: + - source scripts/ci/gitlab-ci-section + - section_start buildenv "Setting up to run tests" - scripts/git-submodule.sh update roms/SLOF - meson subprojects download $(cd build/subprojects && echo *) - cd build @@ -63,7 +74,10 @@ - if [ "x${QEMU_TEST_CACHE_DIR}" != "x" ]; then $MAKE precache-functional ; fi + - section_end buildenv + - section_start test "Running tests" - $MAKE NINJA=":" $MAKE_CHECK_ARGS + - section_end test .native_test_job_template: extends: .common_test_job_template diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml index 2ab8c4806e4..87848c2ffe8 100644 --- a/.gitlab-ci.d/buildtest.yml +++ b/.gitlab-ci.d/buildtest.yml @@ -188,6 +188,7 @@ build-previous-qemu: # Override the default flags as we need more to grab the old version GIT_FETCH_EXTRA_FLAGS: --prune --quiet before_script: + - source scripts/ci/gitlab-ci-section - export QEMU_PREV_VERSION="$(sed 's/\([0-9.]*\)\.[0-9]*/v\1.0/' VERSION)" - git remote add upstream https://gitlab.com/qemu-project/qemu - git fetch upstream refs/tags/$QEMU_PREV_VERSION:refs/tags/$QEMU_PREV_VERSION diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml index d1cb7a35dbf..45a98103554 100644 --- a/.gitlab-ci.d/crossbuild-template.yml +++ b/.gitlab-ci.d/crossbuild-template.yml @@ -9,8 +9,11 @@ when: always timeout: 80m before_script: + - source scripts/ci/gitlab-ci-section + - section_start setup "Pre-script setup" - JOBS=$(expr $(nproc) + 1) - cat /packages.txt + - section_end setup script: - export CCACHE_BASEDIR="$(pwd)" - export CCACHE_DIR="$CCACHE_BASEDIR/ccache" @@ -19,22 +22,30 @@ - mkdir build - cd build - ccache --zero-stats + - section_start configure "Running configure" - ../configure --enable-werror --disable-docs --enable-fdt=system --disable-user $QEMU_CONFIGURE_OPTS $EXTRA_CONFIGURE_OPTS --target-list-exclude="arm-softmmu i386-softmmu microblaze-softmmu mips-softmmu mipsel-softmmu mips64-softmmu ppc-softmmu riscv32-softmmu sh4-softmmu sparc-softmmu xtensa-softmmu $CROSS_SKIP_TARGETS" + - section_end configure + - section_start build "Building QEMU" - make -j"$JOBS" all check-build + - section_end build + - section_start test "Running tests" - if test -n "$MAKE_CHECK_ARGS"; then $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ; fi + - section_end test + - section_start installer "Building the installer" - if grep -q "EXESUF=.exe" config-host.mak; then make installer; version="$(git describe --match v[0-9]* 2>/dev/null || git rev-parse --short HEAD)"; mv -v qemu-setup*.exe qemu-setup-${version}.exe; fi + - section_end installer - ccache --show-stats # Job to cross-build specific accelerators. @@ -52,6 +63,7 @@ - ccache/ key: "$CI_JOB_NAME" before_script: + - source scripts/ci/gitlab-ci-section - JOBS=$(expr $(nproc) + 1) script: - export CCACHE_BASEDIR="$(pwd)" @@ -60,13 +72,19 @@ - export PATH="$CCACHE_WRAPPERSDIR:$PATH" - mkdir build - cd build + - section_start configure "Running configure" - ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS --disable-tools --enable-${ACCEL:-kvm} $EXTRA_CONFIGURE_OPTS + - section_end configure + - section_start build "Building QEMU" - make -j"$JOBS" all check-build + - section_end build + - section_start test "Running tests" - if test -n "$MAKE_CHECK_ARGS"; then $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ; fi + - section_end test .cross_user_build_job: extends: .base_job_template @@ -77,6 +95,7 @@ - ccache/ key: "$CI_JOB_NAME" before_script: + - source scripts/ci/gitlab-ci-section - JOBS=$(expr $(nproc) + 1) script: - export CCACHE_BASEDIR="$(pwd)" @@ -84,16 +103,22 @@ - export CCACHE_MAXSIZE="500M" - mkdir build - cd build + - section_start configure "Running configure" - ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS --disable-system --target-list-exclude="aarch64_be-linux-user alpha-linux-user m68k-linux-user microblazeel-linux-user or1k-linux-user ppc-linux-user sparc-linux-user xtensa-linux-user $CROSS_SKIP_TARGETS" + - section_end configure + - section_start build "Building QEMU" - make -j"$JOBS" all check-build + - section_end build + - section_start test "Running tests" - if test -n "$MAKE_CHECK_ARGS"; then $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ; fi + - section_end test # We can still run some tests on some of our cross build jobs. They can add this # template to their extends to save the build logs and test results diff --git a/scripts/ci/gitlab-ci-section b/scripts/ci/gitlab-ci-section new file mode 100644 index 00000000000..9bbe80420d6 --- /dev/null +++ b/scripts/ci/gitlab-ci-section @@ -0,0 +1,29 @@ +# Copyright (c) 2024 Linaro Ltd +# SPDX-License-Identifier: GPL-2.0-or-later + +# gitlab-ci-section: This is a shell script fragment which defines +# functions section_start and section_end which will emit marker lines +# that GitLab will interpret as the beginning or end of a "collapsible +# section" in a CI job log. See +# https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections +# +# This is intended to be sourced in the before_script section of +# a CI config; the section_start and section_end functions will +# then be available for use in the before_script and script sections. + +# Section names are [-_.A-Za-z0-9] and the section_start pairs with +# a section_end with the same section name. +# The description can be any printable text without newlines; this is +# what will appear in the log. + +# Usage: +# section_start section_name "Description of the section" +section_start () { + printf "section_start:%s:%s\r\e[0K%s\n" "$(date +%s)" "$1" "$2" +} + +# Usage: +# section_end section_name +section_end () { + printf "section_end:%s:%s\r\e[0K\n" "$(date +%s)" "$1" +}