diff mbox series

test_sleep.py: make sleep time and margin configurable

Message ID 20200604152401.3086020-1-hs@denx.de
State Accepted
Commit fa914675d2c8699765f9b7644cff761ff04732cb
Headers show
Series test_sleep.py: make sleep time and margin configurable | expand

Commit Message

Heiko Schocher June 4, 2020, 3:24 p.m. UTC
make the sleep time and the margin configurable.

Signed-off-by: Heiko Schocher <hs at denx.de>
---

travis build:
https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225

This patch is needed as I start test/py now within tbot [1]. On
some configurations U-Boot is compiled on a build machine for
example in munich, while the board under test is in my lab in
hungary.

So the 0.25 seconds default margin is often to low because
of latencies on the net.

See as an example configuration (within tbot):

https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29

[1] http://tbot.tools/modules/tc.html#u-boot-test-py

 test/py/tests/test_sleep.py | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

Comments

Simon Glass June 7, 2020, 1:45 p.m. UTC | #1
On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs at denx.de> wrote:
>
> make the sleep time and the margin configurable.
>
> Signed-off-by: Heiko Schocher <hs at denx.de>
> ---
>
> travis build:
> https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225
>
> This patch is needed as I start test/py now within tbot [1]. On
> some configurations U-Boot is compiled on a build machine for
> example in munich, while the board under test is in my lab in
> hungary.
>
> So the 0.25 seconds default margin is often to low because
> of latencies on the net.
>
> See as an example configuration (within tbot):
>
> https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29
>
> [1] http://tbot.tools/modules/tc.html#u-boot-test-py
>
>  test/py/tests/test_sleep.py | 14 +++++++++++---
>  1 file changed, 11 insertions(+), 3 deletions(-)

Reviewed-by: Simon Glass <sjg at chromium.org>

Related, at some point we should change sandbox to fake the time
movement since this test currently waits for three seconds even on
sandbox.
Stephen Warren June 8, 2020, 4:43 p.m. UTC | #2
On 6/4/20 9:24 AM, Heiko Schocher wrote:
> make the sleep time and the margin configurable.

Reviewed-by: Stephen Warren <swarren at nvidia.com>
Stephen Warren June 8, 2020, 4:43 p.m. UTC | #3
On 6/7/20 7:45 AM, Simon Glass wrote:
> On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs at denx.de> wrote:
>>
>> make the sleep time and the margin configurable.
>>
>> Signed-off-by: Heiko Schocher <hs at denx.de>
>> ---
>>
>> travis build:
>> https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225
>>
>> This patch is needed as I start test/py now within tbot [1]. On
>> some configurations U-Boot is compiled on a build machine for
>> example in munich, while the board under test is in my lab in
>> hungary.
>>
>> So the 0.25 seconds default margin is often to low because
>> of latencies on the net.
>>
>> See as an example configuration (within tbot):
>>
>> https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29
>>
>> [1] http://tbot.tools/modules/tc.html#u-boot-test-py
>>
>>  test/py/tests/test_sleep.py | 14 +++++++++++---
>>  1 file changed, 11 insertions(+), 3 deletions(-)
> 
> Reviewed-by: Simon Glass <sjg at chromium.org>
> 
> Related, at some point we should change sandbox to fake the time
> movement since this test currently waits for three seconds even on
> sandbox.

We definitely shouldn't do that; that's the exact kind of failure this
test is intended to detect.
Simon Glass June 8, 2020, 5:12 p.m. UTC | #4
Hi Stephen,

On Mon, 8 Jun 2020 at 10:43, Stephen Warren <swarren at wwwdotorg.org> wrote:
>
> On 6/7/20 7:45 AM, Simon Glass wrote:
> > On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs at denx.de> wrote:
> >>
> >> make the sleep time and the margin configurable.
> >>
> >> Signed-off-by: Heiko Schocher <hs at denx.de>
> >> ---
> >>
> >> travis build:
> >> https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225
> >>
> >> This patch is needed as I start test/py now within tbot [1]. On
> >> some configurations U-Boot is compiled on a build machine for
> >> example in munich, while the board under test is in my lab in
> >> hungary.
> >>
> >> So the 0.25 seconds default margin is often to low because
> >> of latencies on the net.
> >>
> >> See as an example configuration (within tbot):
> >>
> >> https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29
> >>
> >> [1] http://tbot.tools/modules/tc.html#u-boot-test-py
> >>
> >>  test/py/tests/test_sleep.py | 14 +++++++++++---
> >>  1 file changed, 11 insertions(+), 3 deletions(-)
> >
> > Reviewed-by: Simon Glass <sjg at chromium.org>
> >
> > Related, at some point we should change sandbox to fake the time
> > movement since this test currently waits for three seconds even on
> > sandbox.
>
> We definitely shouldn't do that; that's the exact kind of failure this
> test is intended to detect.

No, we're not looking for bugs in sandbox's time handling. We are just
testing the plumbing associated with delaying (timer driver, etc.).

Regards,
Simon
Stephen Warren June 8, 2020, 5:25 p.m. UTC | #5
On 6/8/20 11:12 AM, Simon Glass wrote:
> Hi Stephen,
> 
> On Mon, 8 Jun 2020 at 10:43, Stephen Warren <swarren at wwwdotorg.org> wrote:
>>
>> On 6/7/20 7:45 AM, Simon Glass wrote:
>>> On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs at denx.de> wrote:
>>>>
>>>> make the sleep time and the margin configurable.
>>>>
>>>> Signed-off-by: Heiko Schocher <hs at denx.de>
>>>> ---
>>>>
>>>> travis build:
>>>> https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225
>>>>
>>>> This patch is needed as I start test/py now within tbot [1]. On
>>>> some configurations U-Boot is compiled on a build machine for
>>>> example in munich, while the board under test is in my lab in
>>>> hungary.
>>>>
>>>> So the 0.25 seconds default margin is often to low because
>>>> of latencies on the net.
>>>>
>>>> See as an example configuration (within tbot):
>>>>
>>>> https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29
>>>>
>>>> [1] http://tbot.tools/modules/tc.html#u-boot-test-py
>>>>
>>>>  test/py/tests/test_sleep.py | 14 +++++++++++---
>>>>  1 file changed, 11 insertions(+), 3 deletions(-)
>>>
>>> Reviewed-by: Simon Glass <sjg at chromium.org>
>>>
>>> Related, at some point we should change sandbox to fake the time
>>> movement since this test currently waits for three seconds even on
>>> sandbox.
>>
>> We definitely shouldn't do that; that's the exact kind of failure this
>> test is intended to detect.
> 
> No, we're not looking for bugs in sandbox's time handling. We are just
> testing the plumbing associated with delaying (timer driver, etc.).

The entire purpose of the test is to look for bugs in the backend
implementation of the time handling. I should know; I wrote the test!
Simon Glass June 17, 2020, 3:11 a.m. UTC | #6
Hi Stephen,

On Mon, 8 Jun 2020 at 11:25, Stephen Warren <swarren at wwwdotorg.org> wrote:
>
> On 6/8/20 11:12 AM, Simon Glass wrote:
> > Hi Stephen,
> >
> > On Mon, 8 Jun 2020 at 10:43, Stephen Warren <swarren at wwwdotorg.org> wrote:
> >>
> >> On 6/7/20 7:45 AM, Simon Glass wrote:
> >>> On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs at denx.de> wrote:
> >>>>
> >>>> make the sleep time and the margin configurable.
> >>>>
> >>>> Signed-off-by: Heiko Schocher <hs at denx.de>
> >>>> ---
> >>>>
> >>>> travis build:
> >>>> https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225
> >>>>
> >>>> This patch is needed as I start test/py now within tbot [1]. On
> >>>> some configurations U-Boot is compiled on a build machine for
> >>>> example in munich, while the board under test is in my lab in
> >>>> hungary.
> >>>>
> >>>> So the 0.25 seconds default margin is often to low because
> >>>> of latencies on the net.
> >>>>
> >>>> See as an example configuration (within tbot):
> >>>>
> >>>> https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29
> >>>>
> >>>> [1] http://tbot.tools/modules/tc.html#u-boot-test-py
> >>>>
> >>>>  test/py/tests/test_sleep.py | 14 +++++++++++---
> >>>>  1 file changed, 11 insertions(+), 3 deletions(-)
> >>>
> >>> Reviewed-by: Simon Glass <sjg at chromium.org>
> >>>
> >>> Related, at some point we should change sandbox to fake the time
> >>> movement since this test currently waits for three seconds even on
> >>> sandbox.
> >>
> >> We definitely shouldn't do that; that's the exact kind of failure this
> >> test is intended to detect.
> >
> > No, we're not looking for bugs in sandbox's time handling. We are just
> > testing the plumbing associated with delaying (timer driver, etc.).
>
> The entire purpose of the test is to look for bugs in the backend
> implementation of the time handling. I should know; I wrote the test!

OK and we have time_ut.c as well.

Perhaps we should disable this test on sandbox then? It really doesn't
make sense to be testing timeouts on sandbox IMO and it costs us 3
seconds on each test run.

Regards,
Simon
Stephen Warren June 17, 2020, 10:34 p.m. UTC | #7
On 6/16/20 9:11 PM, Simon Glass wrote:
> Hi Stephen,
> 
> On Mon, 8 Jun 2020 at 11:25, Stephen Warren <swarren at wwwdotorg.org> wrote:
>>
>> On 6/8/20 11:12 AM, Simon Glass wrote:
>>> Hi Stephen,
>>>
>>> On Mon, 8 Jun 2020 at 10:43, Stephen Warren <swarren at wwwdotorg.org> wrote:
>>>>
>>>> On 6/7/20 7:45 AM, Simon Glass wrote:
>>>>> On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs at denx.de> wrote:
>>>>>>
>>>>>> make the sleep time and the margin configurable.
>>>>>>
>>>>>> Signed-off-by: Heiko Schocher <hs at denx.de>
>>>>>> ---
>>>>>>
>>>>>> travis build:
>>>>>> https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225
>>>>>>
>>>>>> This patch is needed as I start test/py now within tbot [1]. On
>>>>>> some configurations U-Boot is compiled on a build machine for
>>>>>> example in munich, while the board under test is in my lab in
>>>>>> hungary.
>>>>>>
>>>>>> So the 0.25 seconds default margin is often to low because
>>>>>> of latencies on the net.
>>>>>>
>>>>>> See as an example configuration (within tbot):
>>>>>>
>>>>>> https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29
>>>>>>
>>>>>> [1] http://tbot.tools/modules/tc.html#u-boot-test-py
>>>>>>
>>>>>>  test/py/tests/test_sleep.py | 14 +++++++++++---
>>>>>>  1 file changed, 11 insertions(+), 3 deletions(-)
>>>>>
>>>>> Reviewed-by: Simon Glass <sjg at chromium.org>
>>>>>
>>>>> Related, at some point we should change sandbox to fake the time
>>>>> movement since this test currently waits for three seconds even on
>>>>> sandbox.
>>>>
>>>> We definitely shouldn't do that; that's the exact kind of failure this
>>>> test is intended to detect.
>>>
>>> No, we're not looking for bugs in sandbox's time handling. We are just
>>> testing the plumbing associated with delaying (timer driver, etc.).
>>
>> The entire purpose of the test is to look for bugs in the backend
>> implementation of the time handling. I should know; I wrote the test!
> 
> OK and we have time_ut.c as well.
> 
> Perhaps we should disable this test on sandbox then? It really doesn't
> make sense to be testing timeouts on sandbox IMO and it costs us 3
> seconds on each test run.

I'd rather keep such policy out of the test framework. If you personally
don't want to run the test, you can always specify '-k not sleep' (if I
got the syntax right...) when running test.py. I think e.g. the Travis
test configuration might already do this, as an example.
diff mbox series

Patch

diff --git a/test/py/tests/test_sleep.py b/test/py/tests/test_sleep.py
index b69edf26ef..392af29db2 100644
--- a/test/py/tests/test_sleep.py
+++ b/test/py/tests/test_sleep.py
@@ -11,6 +11,12 @@  change test behavior.
 # Setup env__sleep_accurate to False if time is not accurate on your platform
 env__sleep_accurate = False
 
+# Setup env__sleep_time time in seconds board is set to sleep
+env__sleep_time = 3
+
+# Setup env__sleep_margin set a margin for any system overhead
+env__sleep_margin = 0.25
+
 """
 
 def test_sleep(u_boot_console):
@@ -23,13 +29,15 @@  def test_sleep(u_boot_console):
 
     if u_boot_console.config.buildconfig.get('config_cmd_misc', 'n') != 'y':
         pytest.skip('sleep command not supported')
+
     # 3s isn't too long, but is enough to cross a few second boundaries.
-    sleep_time = 3
+    sleep_time = u_boot_console.config.env.get('env__sleep_time', 3)
+    sleep_margin = u_boot_console.config.env.get('env__sleep_margin', 0.25)
     tstart = time.time()
     u_boot_console.run_command('sleep %d' % sleep_time)
     tend = time.time()
     elapsed = tend - tstart
     assert elapsed >= (sleep_time - 0.01)
     if not u_boot_console.config.gdbserver:
-        # 0.25s margin is hopefully enough to account for any system overhead.
-        assert elapsed < (sleep_time + 0.25)
+        # margin is hopefully enough to account for any system overhead.
+        assert elapsed < (sleep_time + sleep_margin)