| Message ID | 20200604152401.3086020-1-hs@denx.de |
|---|---|
| State | Accepted |
| Commit | fa914675d2c8699765f9b7644cff761ff04732cb |
| Headers | show |
| Series | test_sleep.py: make sleep time and margin configurable | expand |
On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs at denx.de> wrote: > > make the sleep time and the margin configurable. > > Signed-off-by: Heiko Schocher <hs at denx.de> > --- > > travis build: > https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225 > > This patch is needed as I start test/py now within tbot [1]. On > some configurations U-Boot is compiled on a build machine for > example in munich, while the board under test is in my lab in > hungary. > > So the 0.25 seconds default margin is often to low because > of latencies on the net. > > See as an example configuration (within tbot): > > https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29 > > [1] http://tbot.tools/modules/tc.html#u-boot-test-py > > test/py/tests/test_sleep.py | 14 +++++++++++--- > 1 file changed, 11 insertions(+), 3 deletions(-) Reviewed-by: Simon Glass <sjg at chromium.org> Related, at some point we should change sandbox to fake the time movement since this test currently waits for three seconds even on sandbox.
On 6/4/20 9:24 AM, Heiko Schocher wrote: > make the sleep time and the margin configurable. Reviewed-by: Stephen Warren <swarren at nvidia.com>
On 6/7/20 7:45 AM, Simon Glass wrote: > On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs at denx.de> wrote: >> >> make the sleep time and the margin configurable. >> >> Signed-off-by: Heiko Schocher <hs at denx.de> >> --- >> >> travis build: >> https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225 >> >> This patch is needed as I start test/py now within tbot [1]. On >> some configurations U-Boot is compiled on a build machine for >> example in munich, while the board under test is in my lab in >> hungary. >> >> So the 0.25 seconds default margin is often to low because >> of latencies on the net. >> >> See as an example configuration (within tbot): >> >> https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29 >> >> [1] http://tbot.tools/modules/tc.html#u-boot-test-py >> >> test/py/tests/test_sleep.py | 14 +++++++++++--- >> 1 file changed, 11 insertions(+), 3 deletions(-) > > Reviewed-by: Simon Glass <sjg at chromium.org> > > Related, at some point we should change sandbox to fake the time > movement since this test currently waits for three seconds even on > sandbox. We definitely shouldn't do that; that's the exact kind of failure this test is intended to detect.
Hi Stephen, On Mon, 8 Jun 2020 at 10:43, Stephen Warren <swarren at wwwdotorg.org> wrote: > > On 6/7/20 7:45 AM, Simon Glass wrote: > > On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs at denx.de> wrote: > >> > >> make the sleep time and the margin configurable. > >> > >> Signed-off-by: Heiko Schocher <hs at denx.de> > >> --- > >> > >> travis build: > >> https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225 > >> > >> This patch is needed as I start test/py now within tbot [1]. On > >> some configurations U-Boot is compiled on a build machine for > >> example in munich, while the board under test is in my lab in > >> hungary. > >> > >> So the 0.25 seconds default margin is often to low because > >> of latencies on the net. > >> > >> See as an example configuration (within tbot): > >> > >> https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29 > >> > >> [1] http://tbot.tools/modules/tc.html#u-boot-test-py > >> > >> test/py/tests/test_sleep.py | 14 +++++++++++--- > >> 1 file changed, 11 insertions(+), 3 deletions(-) > > > > Reviewed-by: Simon Glass <sjg at chromium.org> > > > > Related, at some point we should change sandbox to fake the time > > movement since this test currently waits for three seconds even on > > sandbox. > > We definitely shouldn't do that; that's the exact kind of failure this > test is intended to detect. No, we're not looking for bugs in sandbox's time handling. We are just testing the plumbing associated with delaying (timer driver, etc.). Regards, Simon
On 6/8/20 11:12 AM, Simon Glass wrote: > Hi Stephen, > > On Mon, 8 Jun 2020 at 10:43, Stephen Warren <swarren at wwwdotorg.org> wrote: >> >> On 6/7/20 7:45 AM, Simon Glass wrote: >>> On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs at denx.de> wrote: >>>> >>>> make the sleep time and the margin configurable. >>>> >>>> Signed-off-by: Heiko Schocher <hs at denx.de> >>>> --- >>>> >>>> travis build: >>>> https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225 >>>> >>>> This patch is needed as I start test/py now within tbot [1]. On >>>> some configurations U-Boot is compiled on a build machine for >>>> example in munich, while the board under test is in my lab in >>>> hungary. >>>> >>>> So the 0.25 seconds default margin is often to low because >>>> of latencies on the net. >>>> >>>> See as an example configuration (within tbot): >>>> >>>> https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29 >>>> >>>> [1] http://tbot.tools/modules/tc.html#u-boot-test-py >>>> >>>> test/py/tests/test_sleep.py | 14 +++++++++++--- >>>> 1 file changed, 11 insertions(+), 3 deletions(-) >>> >>> Reviewed-by: Simon Glass <sjg at chromium.org> >>> >>> Related, at some point we should change sandbox to fake the time >>> movement since this test currently waits for three seconds even on >>> sandbox. >> >> We definitely shouldn't do that; that's the exact kind of failure this >> test is intended to detect. > > No, we're not looking for bugs in sandbox's time handling. We are just > testing the plumbing associated with delaying (timer driver, etc.). The entire purpose of the test is to look for bugs in the backend implementation of the time handling. I should know; I wrote the test!
Hi Stephen, On Mon, 8 Jun 2020 at 11:25, Stephen Warren <swarren at wwwdotorg.org> wrote: > > On 6/8/20 11:12 AM, Simon Glass wrote: > > Hi Stephen, > > > > On Mon, 8 Jun 2020 at 10:43, Stephen Warren <swarren at wwwdotorg.org> wrote: > >> > >> On 6/7/20 7:45 AM, Simon Glass wrote: > >>> On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs at denx.de> wrote: > >>>> > >>>> make the sleep time and the margin configurable. > >>>> > >>>> Signed-off-by: Heiko Schocher <hs at denx.de> > >>>> --- > >>>> > >>>> travis build: > >>>> https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225 > >>>> > >>>> This patch is needed as I start test/py now within tbot [1]. On > >>>> some configurations U-Boot is compiled on a build machine for > >>>> example in munich, while the board under test is in my lab in > >>>> hungary. > >>>> > >>>> So the 0.25 seconds default margin is often to low because > >>>> of latencies on the net. > >>>> > >>>> See as an example configuration (within tbot): > >>>> > >>>> https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29 > >>>> > >>>> [1] http://tbot.tools/modules/tc.html#u-boot-test-py > >>>> > >>>> test/py/tests/test_sleep.py | 14 +++++++++++--- > >>>> 1 file changed, 11 insertions(+), 3 deletions(-) > >>> > >>> Reviewed-by: Simon Glass <sjg at chromium.org> > >>> > >>> Related, at some point we should change sandbox to fake the time > >>> movement since this test currently waits for three seconds even on > >>> sandbox. > >> > >> We definitely shouldn't do that; that's the exact kind of failure this > >> test is intended to detect. > > > > No, we're not looking for bugs in sandbox's time handling. We are just > > testing the plumbing associated with delaying (timer driver, etc.). > > The entire purpose of the test is to look for bugs in the backend > implementation of the time handling. I should know; I wrote the test! OK and we have time_ut.c as well. Perhaps we should disable this test on sandbox then? It really doesn't make sense to be testing timeouts on sandbox IMO and it costs us 3 seconds on each test run. Regards, Simon
On 6/16/20 9:11 PM, Simon Glass wrote: > Hi Stephen, > > On Mon, 8 Jun 2020 at 11:25, Stephen Warren <swarren at wwwdotorg.org> wrote: >> >> On 6/8/20 11:12 AM, Simon Glass wrote: >>> Hi Stephen, >>> >>> On Mon, 8 Jun 2020 at 10:43, Stephen Warren <swarren at wwwdotorg.org> wrote: >>>> >>>> On 6/7/20 7:45 AM, Simon Glass wrote: >>>>> On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs at denx.de> wrote: >>>>>> >>>>>> make the sleep time and the margin configurable. >>>>>> >>>>>> Signed-off-by: Heiko Schocher <hs at denx.de> >>>>>> --- >>>>>> >>>>>> travis build: >>>>>> https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225 >>>>>> >>>>>> This patch is needed as I start test/py now within tbot [1]. On >>>>>> some configurations U-Boot is compiled on a build machine for >>>>>> example in munich, while the board under test is in my lab in >>>>>> hungary. >>>>>> >>>>>> So the 0.25 seconds default margin is often to low because >>>>>> of latencies on the net. >>>>>> >>>>>> See as an example configuration (within tbot): >>>>>> >>>>>> https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29 >>>>>> >>>>>> [1] http://tbot.tools/modules/tc.html#u-boot-test-py >>>>>> >>>>>> test/py/tests/test_sleep.py | 14 +++++++++++--- >>>>>> 1 file changed, 11 insertions(+), 3 deletions(-) >>>>> >>>>> Reviewed-by: Simon Glass <sjg at chromium.org> >>>>> >>>>> Related, at some point we should change sandbox to fake the time >>>>> movement since this test currently waits for three seconds even on >>>>> sandbox. >>>> >>>> We definitely shouldn't do that; that's the exact kind of failure this >>>> test is intended to detect. >>> >>> No, we're not looking for bugs in sandbox's time handling. We are just >>> testing the plumbing associated with delaying (timer driver, etc.). >> >> The entire purpose of the test is to look for bugs in the backend >> implementation of the time handling. I should know; I wrote the test! > > OK and we have time_ut.c as well. > > Perhaps we should disable this test on sandbox then? It really doesn't > make sense to be testing timeouts on sandbox IMO and it costs us 3 > seconds on each test run. I'd rather keep such policy out of the test framework. If you personally don't want to run the test, you can always specify '-k not sleep' (if I got the syntax right...) when running test.py. I think e.g. the Travis test configuration might already do this, as an example.
diff --git a/test/py/tests/test_sleep.py b/test/py/tests/test_sleep.py index b69edf26ef..392af29db2 100644 --- a/test/py/tests/test_sleep.py +++ b/test/py/tests/test_sleep.py @@ -11,6 +11,12 @@ change test behavior. # Setup env__sleep_accurate to False if time is not accurate on your platform env__sleep_accurate = False +# Setup env__sleep_time time in seconds board is set to sleep +env__sleep_time = 3 + +# Setup env__sleep_margin set a margin for any system overhead +env__sleep_margin = 0.25 + """ def test_sleep(u_boot_console): @@ -23,13 +29,15 @@ def test_sleep(u_boot_console): if u_boot_console.config.buildconfig.get('config_cmd_misc', 'n') != 'y': pytest.skip('sleep command not supported') + # 3s isn't too long, but is enough to cross a few second boundaries. - sleep_time = 3 + sleep_time = u_boot_console.config.env.get('env__sleep_time', 3) + sleep_margin = u_boot_console.config.env.get('env__sleep_margin', 0.25) tstart = time.time() u_boot_console.run_command('sleep %d' % sleep_time) tend = time.time() elapsed = tend - tstart assert elapsed >= (sleep_time - 0.01) if not u_boot_console.config.gdbserver: - # 0.25s margin is hopefully enough to account for any system overhead. - assert elapsed < (sleep_time + 0.25) + # margin is hopefully enough to account for any system overhead. + assert elapsed < (sleep_time + sleep_margin)
make the sleep time and the margin configurable. Signed-off-by: Heiko Schocher <hs at denx.de> --- travis build: https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225 This patch is needed as I start test/py now within tbot [1]. On some configurations U-Boot is compiled on a build machine for example in munich, while the board under test is in my lab in hungary. So the 0.25 seconds default margin is often to low because of latencies on the net. See as an example configuration (within tbot): https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29 [1] http://tbot.tools/modules/tc.html#u-boot-test-py test/py/tests/test_sleep.py | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)