| Message ID | 20240906180723.503855-3-peter.maydell@linaro.org |
|---|---|
| State | Superseded |
| Headers | show |
| Series | gitlab-ci: Force 'make check' single-threaded for cross-i686-tci | expand |
On 06/09/2024 20.07, Peter Maydell wrote: > The cross-i686-tci CI job is persistently flaky with various tests > hitting timeouts. One theory for why this is happening is that we're > running too many tests in parallel and so sometimes a test gets > starved of CPU and isn't able to complete within the timeout. > > Set the MESON_TESTTHREADS environment variable to 1 for this job; > this will cause 'meson test' to run only one test at a time. > > (Note that this relies on the change to meson2make that makes it > honour MESON_TESTTHREADS; otherwise it will have no effect.) > > Signed-off-by: Peter Maydell <peter.maydell@linaro.org> > --- > Seems worth a try -- if this doesn't have an effect then > we can revert it, but we'll at least have determined what > the problem isn't... > --- > .gitlab-ci.d/crossbuilds.yml | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml > index cb499e4ee0d..ca1db011b11 100644 > --- a/.gitlab-ci.d/crossbuilds.yml > +++ b/.gitlab-ci.d/crossbuilds.yml > @@ -70,6 +70,9 @@ cross-i686-tci: > ACCEL: tcg-interpreter > EXTRA_CONFIGURE_OPTS: --target-list=i386-softmmu,i386-linux-user,aarch64-softmmu,aarch64-linux-user,ppc-softmmu,ppc-linux-user --disable-plugins --disable-kvm > MAKE_CHECK_ARGS: check check-tcg > + # Force 'meson test' to run only one test at once, to > + # see whether this reduces the flakiness of this CI job. > + MESON_TESTTHREADS: 1 Can't we simply add "-j1" to the MAKE_CHECK_ARGS line? According to the man-page of "make": "If there is more than one -j option, the last one is effective." So adding a -j1 should override the previous setting, I think. Thomas
On Tue, 10 Sept 2024 at 13:13, Thomas Huth <thuth@redhat.com> wrote: > > On 06/09/2024 20.07, Peter Maydell wrote: > > The cross-i686-tci CI job is persistently flaky with various tests > > hitting timeouts. One theory for why this is happening is that we're > > running too many tests in parallel and so sometimes a test gets > > starved of CPU and isn't able to complete within the timeout. > > > > Set the MESON_TESTTHREADS environment variable to 1 for this job; > > this will cause 'meson test' to run only one test at a time. > > > > (Note that this relies on the change to meson2make that makes it > > honour MESON_TESTTHREADS; otherwise it will have no effect.) > > > > Signed-off-by: Peter Maydell <peter.maydell@linaro.org> > > --- > > Seems worth a try -- if this doesn't have an effect then > > we can revert it, but we'll at least have determined what > > the problem isn't... > > --- > > .gitlab-ci.d/crossbuilds.yml | 3 +++ > > 1 file changed, 3 insertions(+) > > > > diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml > > index cb499e4ee0d..ca1db011b11 100644 > > --- a/.gitlab-ci.d/crossbuilds.yml > > +++ b/.gitlab-ci.d/crossbuilds.yml > > @@ -70,6 +70,9 @@ cross-i686-tci: > > ACCEL: tcg-interpreter > > EXTRA_CONFIGURE_OPTS: --target-list=i386-softmmu,i386-linux-user,aarch64-softmmu,aarch64-linux-user,ppc-softmmu,ppc-linux-user --disable-plugins --disable-kvm > > MAKE_CHECK_ARGS: check check-tcg > > + # Force 'meson test' to run only one test at once, to > > + # see whether this reduces the flakiness of this CI job. > > + MESON_TESTTHREADS: 1 > > Can't we simply add "-j1" to the MAKE_CHECK_ARGS line? > > According to the man-page of "make": > > "If there is more than one -j option, the last one is effective." > > So adding a -j1 should override the previous setting, I think. You would also be relying on the rune in mtest2make to pick the last -j option in the list, but luckily I think it already does that: .mtestargs += $(subst -j,--num-processes , $(filter-out -j, $(lastword -j1 $(filter -j%, $(MAKEFLAGS))))) thanks -- PMM
diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml index cb499e4ee0d..ca1db011b11 100644 --- a/.gitlab-ci.d/crossbuilds.yml +++ b/.gitlab-ci.d/crossbuilds.yml @@ -70,6 +70,9 @@ cross-i686-tci: ACCEL: tcg-interpreter EXTRA_CONFIGURE_OPTS: --target-list=i386-softmmu,i386-linux-user,aarch64-softmmu,aarch64-linux-user,ppc-softmmu,ppc-linux-user --disable-plugins --disable-kvm MAKE_CHECK_ARGS: check check-tcg + # Force 'meson test' to run only one test at once, to + # see whether this reduces the flakiness of this CI job. + MESON_TESTTHREADS: 1 cross-mipsel-system: extends: .cross_system_build_job
The cross-i686-tci CI job is persistently flaky with various tests hitting timeouts. One theory for why this is happening is that we're running too many tests in parallel and so sometimes a test gets starved of CPU and isn't able to complete within the timeout. Set the MESON_TESTTHREADS environment variable to 1 for this job; this will cause 'meson test' to run only one test at a time. (Note that this relies on the change to meson2make that makes it honour MESON_TESTTHREADS; otherwise it will have no effect.) Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- Seems worth a try -- if this doesn't have an effect then we can revert it, but we'll at least have determined what the problem isn't... --- .gitlab-ci.d/crossbuilds.yml | 3 +++ 1 file changed, 3 insertions(+)