diff mbox series

[PULL,1/6] tests/avocado: push default timeout to QemuBaseTest

Message ID 20220823152558.301624-2-alex.bennee@linaro.org
State Accepted
Commit 48acf68cfe08496284fb12bf598721dea82d74a9
Headers show
Series [PULL,1/6] tests/avocado: push default timeout to QemuBaseTest | expand

Commit Message

Alex Bennée Aug. 23, 2022, 3:25 p.m. UTC 
All of the QEMU tests eventually end up derrived from this class. Move
the default timeout from LinuxTest to ensure we catch them all. As 15
minutes is fairly excessive we drop the default down to 2 minutes
which is a more reasonable target for tests to aim for.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220822165608.2980552-2-alex.bennee@linaro.org>

Comments

Richard Henderson Aug. 23, 2022, 9:33 p.m. UTC | #1 
On 8/23/22 08:25, Alex Bennée wrote:
> All of the QEMU tests eventually end up derrived from this class. Move
> the default timeout from LinuxTest to ensure we catch them all. As 15
> minutes is fairly excessive we drop the default down to 2 minutes
> which is a more reasonable target for tests to aim for.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> Message-Id: <20220822165608.2980552-2-alex.bennee@linaro.org>
> 
> diff --git a/tests/avocado/avocado_qemu/__init__.py b/tests/avocado/avocado_qemu/__init__.py
> index ed4853c805..0efd2bd212 100644
> --- a/tests/avocado/avocado_qemu/__init__.py
> +++ b/tests/avocado/avocado_qemu/__init__.py
> @@ -227,6 +227,10 @@ def exec_command_and_wait_for_pattern(test, command,
>       _console_interaction(test, success_message, failure_message, command + '\r')
>   
>   class QemuBaseTest(avocado.Test):
> +
> +    # default timeout for all tests, can be overridden
> +    timeout = 120
> +
>       def _get_unique_tag_val(self, tag_name):
>           """
>           Gets a tag value, if unique for a key
> @@ -512,7 +516,6 @@ class LinuxTest(LinuxSSHMixIn, QemuSystemTest):
>       to start with than the more vanilla `QemuSystemTest` class.
>       """
>   
> -    timeout = 900
>       distro = None
>       username = 'root'
>       password = 'password'

Bah.

https://gitlab.com/qemu-project/qemu/-/jobs/2923804714

  (001/192) tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg:  INTERRUPTED: 
Test interrupted by SIGTERM\nRunner error occurred: Timeout reached\nOriginal status: 
ERROR\n{'name': '001-tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg', 
'logdir': '/builds/qemu-project/qemu/build/tests/results/job-2022-08-23T21.03-6d06db2/t... 
(120.85 s)
  (003/192) tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_q35_tcg:  INTERRUPTED: Test 
interrupted by SIGTERM\nRunner error occurred: Timeout reached\nOriginal status: 
ERROR\n{'name': '003-tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_q35_tcg', 
'logdir': 
'/builds/qemu-project/qemu/build/tests/results/job-2022-08-23T21.03-6d06db2/test... (120.81 s)

The previous successful run had

  (001/192) tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg:  PASS (257.00 s)
  (003/192) tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_q35_tcg:  PASS (238.67 s)


r~
Alex Bennée Aug. 24, 2022, 9:19 a.m. UTC | #2 
Richard Henderson <richard.henderson@linaro.org> writes:

> On 8/23/22 08:25, Alex Bennée wrote:
>> All of the QEMU tests eventually end up derrived from this class. Move
>> the default timeout from LinuxTest to ensure we catch them all. As 15
>> minutes is fairly excessive we drop the default down to 2 minutes
>> which is a more reasonable target for tests to aim for.
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>> Message-Id: <20220822165608.2980552-2-alex.bennee@linaro.org>
>> diff --git a/tests/avocado/avocado_qemu/__init__.py
>> b/tests/avocado/avocado_qemu/__init__.py
>> index ed4853c805..0efd2bd212 100644
>> --- a/tests/avocado/avocado_qemu/__init__.py
>> +++ b/tests/avocado/avocado_qemu/__init__.py
>> @@ -227,6 +227,10 @@ def exec_command_and_wait_for_pattern(test, command,
>>       _console_interaction(test, success_message, failure_message, command + '\r')
>>     class QemuBaseTest(avocado.Test):
>> +
>> +    # default timeout for all tests, can be overridden
>> +    timeout = 120
>> +
>>       def _get_unique_tag_val(self, tag_name):
>>           """
>>           Gets a tag value, if unique for a key
>> @@ -512,7 +516,6 @@ class LinuxTest(LinuxSSHMixIn, QemuSystemTest):
>>       to start with than the more vanilla `QemuSystemTest` class.
>>       """
>>   -    timeout = 900
>>       distro = None
>>       username = 'root'
>>       password = 'password'
>
> Bah.
>
> https://gitlab.com/qemu-project/qemu/-/jobs/2923804714


Hmm weird - the avocado CFI job doesn't even appear on my CI list (even
with push-ci-now). Anyway I've reverted the timeout to 900s and sent a
v2 of the PR. I'll drop it back down to 120s next cycle and explicitly
increase the timeouts for the known slow tests.

>
>  (001/192)
>  tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg:
>  INTERRUPTED: Test interrupted by SIGTERM\nRunner error occurred:
>  Timeout reached\nOriginal status: ERROR\n{'name':
>  '001-tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg',
>  'logdir':
>  '/builds/qemu-project/qemu/build/tests/results/job-2022-08-23T21.03-6d06db2/t...
>  (120.85 s)
>  (003/192) tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_q35_tcg:
>  INTERRUPTED: Test interrupted by SIGTERM\nRunner error occurred:
>  Timeout reached\nOriginal status: ERROR\n{'name':
>  '003-tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_q35_tcg',
>  'logdir':
>  '/builds/qemu-project/qemu/build/tests/results/job-2022-08-23T21.03-6d06db2/test...
> (120.81 s)
>
> The previous successful run had
>
>  (001/192) tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg:  PASS (257.00 s)
>  (003/192) tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_q35_tcg:  PASS (238.67 s)
>
>
> r~
Thomas Huth Aug. 24, 2022, 9:42 a.m. UTC | #3 
On 24/08/2022 11.19, Alex Bennée wrote:
> 
> Richard Henderson <richard.henderson@linaro.org> writes:
> 
>> On 8/23/22 08:25, Alex Bennée wrote:
>>> All of the QEMU tests eventually end up derrived from this class. Move
>>> the default timeout from LinuxTest to ensure we catch them all. As 15
>>> minutes is fairly excessive we drop the default down to 2 minutes
>>> which is a more reasonable target for tests to aim for.
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>>> Message-Id: <20220822165608.2980552-2-alex.bennee@linaro.org>
>>> diff --git a/tests/avocado/avocado_qemu/__init__.py
>>> b/tests/avocado/avocado_qemu/__init__.py
>>> index ed4853c805..0efd2bd212 100644
>>> --- a/tests/avocado/avocado_qemu/__init__.py
>>> +++ b/tests/avocado/avocado_qemu/__init__.py
>>> @@ -227,6 +227,10 @@ def exec_command_and_wait_for_pattern(test, command,
>>>        _console_interaction(test, success_message, failure_message, command + '\r')
>>>      class QemuBaseTest(avocado.Test):
>>> +
>>> +    # default timeout for all tests, can be overridden
>>> +    timeout = 120
>>> +
>>>        def _get_unique_tag_val(self, tag_name):
>>>            """
>>>            Gets a tag value, if unique for a key
>>> @@ -512,7 +516,6 @@ class LinuxTest(LinuxSSHMixIn, QemuSystemTest):
>>>        to start with than the more vanilla `QemuSystemTest` class.
>>>        """
>>>    -    timeout = 900
>>>        distro = None
>>>        username = 'root'
>>>        password = 'password'
>>
>> Bah.
>>
>> https://gitlab.com/qemu-project/qemu/-/jobs/2923804714
> 
> 
> Hmm weird - the avocado CFI job doesn't even appear on my CI list (even
> with push-ci-now).

You likely have to set QEMU_CI_AVOCADO_TESTING in your gitlab settings, see 
docs/devel/ci-jobs.rst.inc.

I think we really have to rework the way we run (or rather not run) the 
avocado tests - since with the current default behavior, they'll be ignored 
by most people by default.

  Thomas
diff mbox series

Patch

diff --git a/tests/avocado/avocado_qemu/__init__.py b/tests/avocado/avocado_qemu/__init__.py
index ed4853c805..0efd2bd212 100644
--- a/tests/avocado/avocado_qemu/__init__.py
+++ b/tests/avocado/avocado_qemu/__init__.py
@@ -227,6 +227,10 @@  def exec_command_and_wait_for_pattern(test, command,
     _console_interaction(test, success_message, failure_message, command + '\r')
 
 class QemuBaseTest(avocado.Test):
+
+    # default timeout for all tests, can be overridden
+    timeout = 120
+
     def _get_unique_tag_val(self, tag_name):
         """
         Gets a tag value, if unique for a key
@@ -512,7 +516,6 @@  class LinuxTest(LinuxSSHMixIn, QemuSystemTest):
     to start with than the more vanilla `QemuSystemTest` class.
     """
 
-    timeout = 900
     distro = None
     username = 'root'
     password = 'password'