From patchwork Fri Dec 1 09:36:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 748984 Delivered-To: patch@linaro.org Received: by 2002:adf:f9cb:0:b0:32d:baff:b0ca with SMTP id w11csp168407wrr; Fri, 1 Dec 2023 01:38:24 -0800 (PST) X-Google-Smtp-Source: AGHT+IFB83B/DO0HJ8BKXV487bcMcd7lVGXrB0VsnPjEXU/0l54FvoC+t2M+zV2sGf8GJBgajvVo X-Received: by 2002:a05:6830:280f:b0:6d8:52e0:180a with SMTP id w15-20020a056830280f00b006d852e0180amr2523741otu.17.1701423504070; Fri, 01 Dec 2023 01:38:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701423504; cv=none; d=google.com; s=arc-20160816; b=MEg5N6GftPu2pMY/3UMgKSzl8wIBW+GPXWk5ZlzEYvFub+4eb0jhNCR2qcB5RWESlo 9QFbQPCXaH12nvvKSwRVXQmag91Z35GTxPvRUc54j9Xb08nqqM/YqF8y2NpXvepdiD9c NbbCURROdq9CXgCUsA/X0aMRd5ZLyaSt54PE8rMAhAlWkTDEHhDPCU/SyWcc2xuapyyd yPH0//n4vgWp7MWxaL27iYzQ8Wdi6b5orAO1zz/BVitg5EHcEJv8FqLJORR1SY+Xqgb1 W7JcO3Y7OYKbsdpaw2AkPXRbgFRUC6TIhGsOgwSQC6T+fh/b7U/SjokpZa7knul4WdPF t9HA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=1fLA9HqwzG3bToXvOE1aHD3zBjzOSiUc0V1bowUocRU=; fh=O7ROGzWhTwE5UjSDIfDBUgs09C/dTUir/POGCKKHRT0=; b=HPjP3e9pjgz+MHFHsEBldzaqVHn+N/kl65hDMbzfRokYjn4PT6aonpUsfhoAUBqb9+ VfDiWJSviahdMTVCiKfvOJPXVRom9zFFIKHY51kyDs2NCyH2fmIeUwpMiR0RQ4ILMEaa tiPIdVMlSQobSu98BRBhdU1d9iVt9yKAWyxgqeKDu2/fdKhE9SwccvTx14LH6OVM3T0G AvLF7I8LXdRjf2DJhLZtmp6t6G7qMy0bpjASNEfs+uAvD7CSSsS4J3KbhgwliCTBDXS+ IUqOOEykMHKjIEURs2yhYetjD8tn+p2m/fdqJHPaGzj/dH89Uxt7RQJDZjypC7ht9gs5 +Vqw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="aIWA7/cY"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id ci11-20020a05622a260b00b004239cefcde8si3199489qtb.224.2023.12.01.01.38.23 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 01 Dec 2023 01:38:24 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="aIWA7/cY"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r8zx1-0003rt-H1; Fri, 01 Dec 2023 04:36:47 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r8zwy-0003qx-P4 for qemu-devel@nongnu.org; Fri, 01 Dec 2023 04:36:45 -0500 Received: from mail-wm1-x32a.google.com ([2a00:1450:4864:20::32a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1r8zws-00045c-O2 for qemu-devel@nongnu.org; Fri, 01 Dec 2023 04:36:44 -0500 Received: by mail-wm1-x32a.google.com with SMTP id 5b1f17b1804b1-40b4c2ef58aso19234505e9.0 for ; Fri, 01 Dec 2023 01:36:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1701423397; x=1702028197; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1fLA9HqwzG3bToXvOE1aHD3zBjzOSiUc0V1bowUocRU=; b=aIWA7/cYfJdL6iWTk07nzxd+n2vebthfCP7Xc5qRnEtK4GWw1Rw8ahfK8UzUJ7vl85 HwZc02DXZQ3cQFS9kfCEqtKU5aI1rgbAwBPAgdab33qkhJ4n0O/IJgzrvlvFJDZVfwNO fFPbtWPxRH9/4d2aQ0tU6J6E9XXf+QkD0seq4roIBQ60KBfJCs9CUALQBdZ7QgHWw1n0 eaOPrhVXZunuEaTIEE2NClFGe1CE7ZZAHq8GmI73aXCCZPkcfDfWPxolrriGxDMN4Gx/ uf54tEZBCgyyTuwLP544IsGTeltLS7m0SSh9+x1UM6QPTMvAzN7i/aGbK4PV27+A3dRH dDIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701423397; x=1702028197; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1fLA9HqwzG3bToXvOE1aHD3zBjzOSiUc0V1bowUocRU=; b=dStGZ70FFLekqJbl1bQs0gz+VZ1Xe4NmDPtAUBJ1lpBBbtp9ZriXiX3jALcdPFTZm7 U012q9q0BJafCAaTcJE16r3z+WEcisFyyV2I//0rMRlNK1b3U975MhhnCum7s4POK415 cHHjjJ/PCBHJY5v+kN37WfaVeNS0dzZfowPdS4Rsz00KR39dS0tr95DLYJHNTgp5vYiz QhgO2X4lG3QEen0z7yAza6VZfyhgGdlfEhKWu8Gb2FmEiqVtgNP+a6rdd0QMO+Jx0e06 AYm9/Ysh+kXneFIwydxtJBAGc+q3yNwnlKD+/dYc2Gxbw8n+gP0VeznISrtSdPebebv3 8dLg== X-Gm-Message-State: AOJu0YzBepEYZOu6zomskMBEBb1YwKnUl+/0bAFaC13SmRGe1E502ZL7 GBtq5ZLJBy0Vj7GWPFZ8E/R2sA== X-Received: by 2002:a05:600c:1989:b0:40b:5e1c:2f9d with SMTP id t9-20020a05600c198900b0040b5e1c2f9dmr220565wmq.45.1701423397082; Fri, 01 Dec 2023 01:36:37 -0800 (PST) Received: from draig.lan ([85.9.250.243]) by smtp.gmail.com with ESMTPSA id f15-20020a05600c4e8f00b0040b3632e993sm8465107wmq.46.2023.12.01.01.36.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:36:34 -0800 (PST) Received: from draig.lan (localhost [IPv6:::1]) by draig.lan (Postfix) with ESMTP id 24DD35FB63; Fri, 1 Dec 2023 09:36:34 +0000 (GMT) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: qemu-devel@nongnu.org Cc: Eric Farman , Peter Maydell , Leif Lindholm , Beraldo Leal , Andrew Jeffery , Paolo Bonzini , Marcin Juszkiewicz , Pavel Dovgalyuk , Thomas Huth , qemu-arm@nongnu.org, Radoslaw Biernacki , =?utf-8?q?Alex_?= =?utf-8?q?Benn=C3=A9e?= , Yoshinori Sato , Eric Auger , Halil Pasic , qemu-s390x@nongnu.org, Wainer dos Santos Moschetta , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Christian Borntraeger , Aurelien Jarno , Cleber Rosa , Joel Stanley , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Richard Henderson Subject: [PATCH v2 2/7] docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS Date: Fri, 1 Dec 2023 09:36:28 +0000 Message-Id: <20231201093633.2551497-3-alex.bennee@linaro.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231201093633.2551497-1-alex.bennee@linaro.org> References: <20231201093633.2551497-1-alex.bennee@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::32a; envelope-from=alex.bennee@linaro.org; helo=mail-wm1-x32a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org It doesn't make sense to have two classes of flaky tests. While it may take the constrained environment of CI to trigger failures easily it doesn't mean they don't occasionally happen on developer machines. As CI is the gating factor to passing there is no point developers running the tests locally anyway unless they are trying to fix things. While we are at it update the language in the docs to discourage the QEMU_TEST_FLAKY_TESTS becoming a permanent solution. Message-Id: <20231130153333.2424775-3-alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Richard Henderson Signed-off-by: Alex Bennée --- v2 - be more explicit about raising bugs - add some bug urls --- docs/devel/testing.rst | 31 +++++++++++++++--------- tests/avocado/boot_linux.py | 8 +++--- tests/avocado/boot_linux_console.py | 5 ++-- tests/avocado/intel_iommu.py | 5 ++-- tests/avocado/linux_initrd.py | 5 ++-- tests/avocado/machine_aspeed.py | 8 +++--- tests/avocado/machine_mips_malta.py | 8 +++--- tests/avocado/machine_rx_gdbsim.py | 8 +++--- tests/avocado/machine_s390_ccw_virtio.py | 2 +- tests/avocado/replay_kernel.py | 6 +++-- tests/avocado/reverse_debugging.py | 14 +++++++---- tests/avocado/smmu.py | 5 ++-- tests/avocado/tuxrun_baselines.py | 4 +-- 13 files changed, 68 insertions(+), 41 deletions(-) diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 22218dbedb..76465b8f3d 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -1371,23 +1371,32 @@ conditions. For example, tests that take longer to execute when QEMU is compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable has been used to determine whether those tests should run or not. -GITLAB_CI -^^^^^^^^^ -A number of tests are flagged to not run on the GitLab CI. Usually because -they proved to the flaky or there are constraints on the CI environment which -would make them fail. If you encounter a similar situation then use that -variable as shown on the code snippet below to skip the test: +QEMU_TEST_FLAKY_TESTS +^^^^^^^^^^^^^^^^^^^^^ +Some tests are not working reliably and thus are disabled by default. +This includes tests that don't run reliably on GitLab's CI which +usually expose real issues that are rarely seen on developer machines +due to the constraints of the CI environment. If you encounter a +similar situation then raise a bug and then mark the test as shown on +the code snippet below: .. code:: - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + # See https://gitlab.com/qemu-project/qemu/-/issues/nnnn + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') def test(self): do_something() -QEMU_TEST_FLAKY_TESTS -^^^^^^^^^^^^^^^^^^^^^ -Some tests are not working reliably and thus are disabled by default. -Set this environment variable to enable them. +Tests should not live in this state forever and should either be fixed +or eventually removed. + +To run such tests locally you will need to set the environment +variable. For example: + +.. code:: + + env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado run \ + tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg Uninstalling Avocado ~~~~~~~~~~~~~~~~~~~~ diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py index be30dcbd58..9e9773e6e1 100644 --- a/tests/avocado/boot_linux.py +++ b/tests/avocado/boot_linux.py @@ -12,7 +12,7 @@ from avocado_qemu import LinuxTest, BUILD_DIR -from avocado import skipIf +from avocado import skipUnless class BootLinuxX8664(LinuxTest): @@ -93,7 +93,8 @@ class BootLinuxPPC64(LinuxTest): timeout = 360 - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_pseries_tcg(self): """ :avocado: tags=machine:pseries @@ -111,7 +112,8 @@ class BootLinuxS390X(LinuxTest): timeout = 240 - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_s390_ccw_virtio_tcg(self): """ :avocado: tags=machine:s390-ccw-virtio diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py index 6eab515718..231b4f68e5 100644 --- a/tests/avocado/boot_linux_console.py +++ b/tests/avocado/boot_linux_console.py @@ -15,7 +15,7 @@ from avocado import skip from avocado import skipUnless -from avocado import skipIf +from avocado import skipUnless from avocado_qemu import QemuSystemTest from avocado_qemu import exec_command from avocado_qemu import exec_command_and_wait_for_pattern @@ -1419,7 +1419,8 @@ def test_ppc_mac99(self): # This test has a 6-10% failure rate on various hosts that look # like issues with a buggy kernel. As a result we don't want it # gating releases on Gitlab. - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_sh4_r2d(self): """ :avocado: tags=arch:sh4 diff --git a/tests/avocado/intel_iommu.py b/tests/avocado/intel_iommu.py index 77635ab56c..2dd11a6346 100644 --- a/tests/avocado/intel_iommu.py +++ b/tests/avocado/intel_iommu.py @@ -9,10 +9,11 @@ # later. See the COPYING file in the top-level directory. import os -from avocado import skipIf +from avocado import skipUnless from avocado_qemu import LinuxTest -@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') +@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + class IntelIOMMU(LinuxTest): """ :avocado: tags=arch:x86_64 diff --git a/tests/avocado/linux_initrd.py b/tests/avocado/linux_initrd.py index ba02e5a563..c40a987bd1 100644 --- a/tests/avocado/linux_initrd.py +++ b/tests/avocado/linux_initrd.py @@ -13,7 +13,7 @@ import tempfile from avocado_qemu import QemuSystemTest -from avocado import skipIf +from avocado import skipUnless class LinuxInitrd(QemuSystemTest): @@ -53,7 +53,8 @@ def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self): max_size + 1) self.assertRegex(self.vm.get_log(), expected_msg) - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_with_2gib_file_should_work_with_linux_v4_16(self): """ QEMU has supported up to 4 GiB initrd for recent kernel diff --git a/tests/avocado/machine_aspeed.py b/tests/avocado/machine_aspeed.py index df31b2a8a8..258fb50c47 100644 --- a/tests/avocado/machine_aspeed.py +++ b/tests/avocado/machine_aspeed.py @@ -18,7 +18,7 @@ from avocado_qemu import interrupt_interactive_console_until_pattern from avocado_qemu import has_cmd from avocado.utils import archive -from avocado import skipIf +from avocado import skipUnless from avocado import skipUnless @@ -311,7 +311,8 @@ def do_test_arm_aspeed_sdk_start(self, image): self, 'boot', '## Loading kernel from FIT Image') self.wait_for_console_pattern('Starting kernel ...') - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_arm_ast2500_evb_sdk(self): """ :avocado: tags=arch:arm @@ -329,7 +330,8 @@ def test_arm_ast2500_evb_sdk(self): self.workdir + '/ast2500-default/image-bmc') self.wait_for_console_pattern('nodistro.0 ast2500-default ttyS4') - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_arm_ast2600_evb_sdk(self): """ :avocado: tags=arch:arm diff --git a/tests/avocado/machine_mips_malta.py b/tests/avocado/machine_mips_malta.py index 9bd54518bf..959dcf5602 100644 --- a/tests/avocado/machine_mips_malta.py +++ b/tests/avocado/machine_mips_malta.py @@ -11,7 +11,7 @@ import gzip import logging -from avocado import skipIf +from avocado import skipUnless from avocado import skipUnless from avocado.utils import archive from avocado_qemu import QemuSystemTest @@ -101,7 +101,8 @@ def test_mips_malta_i6400_framebuffer_logo_1core(self): """ self.do_test_i6400_framebuffer_logo(1) - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_mips_malta_i6400_framebuffer_logo_7cores(self): """ :avocado: tags=arch:mips64el @@ -111,7 +112,8 @@ def test_mips_malta_i6400_framebuffer_logo_7cores(self): """ self.do_test_i6400_framebuffer_logo(7) - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_mips_malta_i6400_framebuffer_logo_8cores(self): """ :avocado: tags=arch:mips64el diff --git a/tests/avocado/machine_rx_gdbsim.py b/tests/avocado/machine_rx_gdbsim.py index 6cd8704b01..350a73fbbb 100644 --- a/tests/avocado/machine_rx_gdbsim.py +++ b/tests/avocado/machine_rx_gdbsim.py @@ -10,7 +10,7 @@ import os -from avocado import skipIf +from avocado import skipUnless from avocado_qemu import QemuSystemTest from avocado_qemu import exec_command_and_wait_for_pattern from avocado_qemu import wait_for_console_pattern @@ -22,7 +22,8 @@ class RxGdbSimMachine(QemuSystemTest): timeout = 30 KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 ' - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_uboot(self): """ U-Boot and checks that the console is operational. @@ -46,7 +47,8 @@ def test_uboot(self): # FIXME limit baudrate on chardev, else we type too fast #exec_command_and_wait_for_pattern(self, 'version', gcc_version) - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_linux_sash(self): """ Boots a Linux kernel and checks that the console is operational. diff --git a/tests/avocado/machine_s390_ccw_virtio.py b/tests/avocado/machine_s390_ccw_virtio.py index ffd914ded9..61e75d8f9e 100644 --- a/tests/avocado/machine_s390_ccw_virtio.py +++ b/tests/avocado/machine_s390_ccw_virtio.py @@ -12,7 +12,7 @@ import os import tempfile -from avocado import skipIf +from avocado import skipUnless from avocado_qemu import QemuSystemTest from avocado_qemu import exec_command_and_wait_for_pattern from avocado_qemu import wait_for_console_pattern diff --git a/tests/avocado/replay_kernel.py b/tests/avocado/replay_kernel.py index 53cb7e5091..0d32cc280e 100644 --- a/tests/avocado/replay_kernel.py +++ b/tests/avocado/replay_kernel.py @@ -15,7 +15,7 @@ import time from avocado import skip -from avocado import skipIf +from avocado import skipUnless from avocado import skipUnless from avocado_qemu import wait_for_console_pattern from avocado.utils import archive @@ -82,6 +82,7 @@ def run_rr(self, kernel_path, kernel_command_line, console_pattern, class ReplayKernelNormal(ReplayKernelBase): + # See https://gitlab.com/qemu-project/qemu/-/issues/2010 @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test sometimes gets stuck') def test_x86_64_pc(self): """ @@ -179,7 +180,8 @@ def test_arm_virt(self): self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=1) - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_arm_cubieboard_initrd(self): """ :avocado: tags=arch:arm diff --git a/tests/avocado/reverse_debugging.py b/tests/avocado/reverse_debugging.py index ed04e92bb4..9a468321e5 100644 --- a/tests/avocado/reverse_debugging.py +++ b/tests/avocado/reverse_debugging.py @@ -10,7 +10,7 @@ import os import logging -from avocado import skipIf +from avocado import skipUnless from avocado_qemu import BUILD_DIR from avocado.utils import datadrainer from avocado.utils import gdb @@ -206,7 +206,8 @@ def get_pc(self, g): + self.get_reg_le(g, self.REG_CS) * 0x10 # unidentified gitlab timeout problem - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_x86_64_pc(self): """ :avocado: tags=arch:x86_64 @@ -223,7 +224,8 @@ class ReverseDebugging_AArch64(ReverseDebugging): REG_PC = 32 # unidentified gitlab timeout problem - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_aarch64_virt(self): """ :avocado: tags=arch:aarch64 @@ -247,7 +249,8 @@ class ReverseDebugging_ppc64(ReverseDebugging): REG_PC = 0x40 # unidentified gitlab timeout problem - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_ppc64_pseries(self): """ :avocado: tags=arch:ppc64 @@ -260,7 +263,8 @@ def test_ppc64_pseries(self): self.reverse_debugging() # See https://gitlab.com/qemu-project/qemu/-/issues/1992 - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_ppc64_powernv(self): """ :avocado: tags=arch:ppc64 diff --git a/tests/avocado/smmu.py b/tests/avocado/smmu.py index b3c4de6bf4..05b34418a5 100644 --- a/tests/avocado/smmu.py +++ b/tests/avocado/smmu.py @@ -9,10 +9,11 @@ # later. See the COPYING file in the top-level directory. import os -from avocado import skipIf +from avocado import skipUnless from avocado_qemu import LinuxTest, BUILD_DIR -@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') +@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + class SMMU(LinuxTest): """ :avocado: tags=accel:kvm diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py index c99bea6c0b..5f859f4e6f 100644 --- a/tests/avocado/tuxrun_baselines.py +++ b/tests/avocado/tuxrun_baselines.py @@ -13,7 +13,7 @@ import time import tempfile -from avocado import skip, skipIf +from avocado import skip, skipUnless from avocado_qemu import QemuSystemTest from avocado_qemu import exec_command, exec_command_and_wait_for_pattern from avocado_qemu import wait_for_console_pattern @@ -551,7 +551,7 @@ def test_s390(self): haltmsg="Requesting system halt") # Note: some segfaults caused by unaligned userspace access - @skipIf(os.getenv('GITLAB_CI'), 'Skipping unstable test on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') def test_sh4(self): """ :avocado: tags=arch:sh4