Message ID | 20230417134321.3627231-2-alex.bennee@linaro.org |
---|---|
State | Superseded |
Headers | show |
Series | testing/next: avocado, docs, gitlab | expand |
On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote: > > From: Kautuk Consul <kconsul@linux.vnet.ibm.com> > > Avocado version 101.0 has a fix to re-compute the checksum > of an asset file if the algorithm used in the *-CHECKSUM > file isn't the same as the one being passed to it by the > avocado user (i.e. the avocado_qemu python module). > In the earlier avocado versions this fix wasn't there due > to which if the checksum wouldn't match the earlier > checksum (calculated by a different algorithm), the avocado > code would start downloading a fresh image from the internet > URL thus making the test-cases take longer to execute. > > Bump up the avocado-framework version to 101.0. > > Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> > Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> > Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> > Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org> > > --- > v2 > - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%) > --- > tests/Makefile.include | 18 +++++++++++------- > tests/requirements.txt | 2 +- > 2 files changed, 12 insertions(+), 8 deletions(-) > > diff --git a/tests/Makefile.include b/tests/Makefile.include > index 9422ddaece..a4de0ad5a2 100644 > --- a/tests/Makefile.include > +++ b/tests/Makefile.include > @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv > # download all vm images, according to defined targets > get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD)) > > +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS)))) > + > check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images > - $(call quiet-command, \ > - $(TESTS_PYTHON) -m avocado \ > - --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ > - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ > - --filter-by-tags-include-empty-key) \ > - $(AVOCADO_CMDLINE_TAGS) \ > - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ > + $(call quiet-command, \ > + $(TESTS_PYTHON) -m avocado \ > + --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ > + $(if $(AVOCADO_TAGS),, \ > + --filter-by-tags-include-empty \ > + --filter-by-tags-include-empty-key) \ > + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \ > + $(AVOCADO_CMDLINE_TAGS) \ > + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ > "AVOCADO", "tests/avocado") > > check-acceptance-deprecated-warning: > diff --git a/tests/requirements.txt b/tests/requirements.txt > index 0ba561b6bd..a6f73da681 100644 > --- a/tests/requirements.txt > +++ b/tests/requirements.txt > @@ -2,5 +2,5 @@ > # in the tests/venv Python virtual environment. For more info, > # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 > # Note that qemu.git/python/ is always implicitly installed. > -avocado-framework==88.1 > +avocado-framework==101.0 > pycdlib==1.11.0 > -- > 2.39.2 > I thought there were test failures that prohibited us from bumping the Avocado-Framework version. Did those get rectified recently?
John Snow <jsnow@redhat.com> writes: > On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote: >> >> From: Kautuk Consul <kconsul@linux.vnet.ibm.com> >> >> Avocado version 101.0 has a fix to re-compute the checksum >> of an asset file if the algorithm used in the *-CHECKSUM >> file isn't the same as the one being passed to it by the >> avocado user (i.e. the avocado_qemu python module). >> In the earlier avocado versions this fix wasn't there due >> to which if the checksum wouldn't match the earlier >> checksum (calculated by a different algorithm), the avocado >> code would start downloading a fresh image from the internet >> URL thus making the test-cases take longer to execute. >> >> Bump up the avocado-framework version to 101.0. >> >> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> >> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> >> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> >> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org> >> >> --- >> v2 >> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%) >> --- >> tests/Makefile.include | 18 +++++++++++------- >> tests/requirements.txt | 2 +- >> 2 files changed, 12 insertions(+), 8 deletions(-) >> >> diff --git a/tests/Makefile.include b/tests/Makefile.include >> index 9422ddaece..a4de0ad5a2 100644 >> --- a/tests/Makefile.include >> +++ b/tests/Makefile.include >> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv >> # download all vm images, according to defined targets >> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD)) >> >> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS)))) >> + >> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images >> - $(call quiet-command, \ >> - $(TESTS_PYTHON) -m avocado \ >> - --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ >> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ >> - --filter-by-tags-include-empty-key) \ >> - $(AVOCADO_CMDLINE_TAGS) \ >> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ >> + $(call quiet-command, \ >> + $(TESTS_PYTHON) -m avocado \ >> + --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ >> + $(if $(AVOCADO_TAGS),, \ >> + --filter-by-tags-include-empty \ >> + --filter-by-tags-include-empty-key) \ >> + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \ >> + $(AVOCADO_CMDLINE_TAGS) \ >> + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ >> "AVOCADO", "tests/avocado") >> >> check-acceptance-deprecated-warning: >> diff --git a/tests/requirements.txt b/tests/requirements.txt >> index 0ba561b6bd..a6f73da681 100644 >> --- a/tests/requirements.txt >> +++ b/tests/requirements.txt >> @@ -2,5 +2,5 @@ >> # in the tests/venv Python virtual environment. For more info, >> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 >> # Note that qemu.git/python/ is always implicitly installed. >> -avocado-framework==88.1 >> +avocado-framework==101.0 >> pycdlib==1.11.0 >> -- >> 2.39.2 >> > > I thought there were test failures that prohibited us from bumping the > Avocado-Framework version. Did those get rectified recently? I'm not seeing any with your patch applied.
On 17/04/2023 15.43, Alex Bennée wrote: > From: Kautuk Consul <kconsul@linux.vnet.ibm.com> > > Avocado version 101.0 has a fix to re-compute the checksum > of an asset file if the algorithm used in the *-CHECKSUM > file isn't the same as the one being passed to it by the > avocado user (i.e. the avocado_qemu python module). > In the earlier avocado versions this fix wasn't there due > to which if the checksum wouldn't match the earlier > checksum (calculated by a different algorithm), the avocado > code would start downloading a fresh image from the internet > URL thus making the test-cases take longer to execute. > > Bump up the avocado-framework version to 101.0. > > Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> > Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> > Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> > Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org> > > --- > v2 > - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%) > --- > tests/Makefile.include | 18 +++++++++++------- > tests/requirements.txt | 2 +- > 2 files changed, 12 insertions(+), 8 deletions(-) > > diff --git a/tests/Makefile.include b/tests/Makefile.include > index 9422ddaece..a4de0ad5a2 100644 > --- a/tests/Makefile.include > +++ b/tests/Makefile.include > @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv > # download all vm images, according to defined targets > get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD)) > > +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS)))) > + > check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images > - $(call quiet-command, \ > - $(TESTS_PYTHON) -m avocado \ > - --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ > - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ > - --filter-by-tags-include-empty-key) \ > - $(AVOCADO_CMDLINE_TAGS) \ > - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ > + $(call quiet-command, \ > + $(TESTS_PYTHON) -m avocado \ > + --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ > + $(if $(AVOCADO_TAGS),, \ > + --filter-by-tags-include-empty \ > + --filter-by-tags-include-empty-key) \ > + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \ > + $(AVOCADO_CMDLINE_TAGS) \ > + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ We might need to revisit this --failfast logic, too. If I've got that right, failfast is now the default with the new system? So we might want to disable it by default again if GITLAB_CI is not set? Thomas
On 17/4/23 19:44, Alex Bennée wrote: > > John Snow <jsnow@redhat.com> writes: > >> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote: >>> >>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com> >>> >>> Avocado version 101.0 has a fix to re-compute the checksum >>> of an asset file if the algorithm used in the *-CHECKSUM >>> file isn't the same as the one being passed to it by the >>> avocado user (i.e. the avocado_qemu python module). >>> In the earlier avocado versions this fix wasn't there due >>> to which if the checksum wouldn't match the earlier >>> checksum (calculated by a different algorithm), the avocado >>> code would start downloading a fresh image from the internet >>> URL thus making the test-cases take longer to execute. >>> >>> Bump up the avocado-framework version to 101.0. >>> >>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> >>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> >>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> >>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org> >>> >>> --- >>> v2 >>> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%) >>> --- >>> tests/Makefile.include | 18 +++++++++++------- >>> tests/requirements.txt | 2 +- >>> 2 files changed, 12 insertions(+), 8 deletions(-) >>> >>> diff --git a/tests/Makefile.include b/tests/Makefile.include >>> index 9422ddaece..a4de0ad5a2 100644 >>> --- a/tests/Makefile.include >>> +++ b/tests/Makefile.include >>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv >>> # download all vm images, according to defined targets >>> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD)) >>> >>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS)))) >>> + >>> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images >>> - $(call quiet-command, \ >>> - $(TESTS_PYTHON) -m avocado \ >>> - --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ >>> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ >>> - --filter-by-tags-include-empty-key) \ >>> - $(AVOCADO_CMDLINE_TAGS) \ >>> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ >>> + $(call quiet-command, \ >>> + $(TESTS_PYTHON) -m avocado \ >>> + --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ >>> + $(if $(AVOCADO_TAGS),, \ >>> + --filter-by-tags-include-empty \ >>> + --filter-by-tags-include-empty-key) \ >>> + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \ >>> + $(AVOCADO_CMDLINE_TAGS) \ >>> + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ >>> "AVOCADO", "tests/avocado") >>> >>> check-acceptance-deprecated-warning: >>> diff --git a/tests/requirements.txt b/tests/requirements.txt >>> index 0ba561b6bd..a6f73da681 100644 >>> --- a/tests/requirements.txt >>> +++ b/tests/requirements.txt >>> @@ -2,5 +2,5 @@ >>> # in the tests/venv Python virtual environment. For more info, >>> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 >>> # Note that qemu.git/python/ is always implicitly installed. >>> -avocado-framework==88.1 >>> +avocado-framework==101.0 >>> pycdlib==1.11.0 >>> -- >>> 2.39.2 >>> >> >> I thought there were test failures that prohibited us from bumping the >> Avocado-Framework version. Did those get rectified recently? No, still not working on Darwin. At this point I'm thinking at sending a patch clarifying testing QEMU with Avocado is not supported on Darwin. > I'm not seeing any with your patch applied. Are you using running the Cirrus-CI jobs?
On Thu, Apr 20, 2023, 5:19 AM Philippe Mathieu-Daudé <philmd@linaro.org> wrote: > On 17/4/23 19:44, Alex Bennée wrote: > > > > John Snow <jsnow@redhat.com> writes: > > > >> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> > wrote: > >>> > >>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com> > >>> > >>> Avocado version 101.0 has a fix to re-compute the checksum > >>> of an asset file if the algorithm used in the *-CHECKSUM > >>> file isn't the same as the one being passed to it by the > >>> avocado user (i.e. the avocado_qemu python module). > >>> In the earlier avocado versions this fix wasn't there due > >>> to which if the checksum wouldn't match the earlier > >>> checksum (calculated by a different algorithm), the avocado > >>> code would start downloading a fresh image from the internet > >>> URL thus making the test-cases take longer to execute. > >>> > >>> Bump up the avocado-framework version to 101.0. > >>> > >>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> > >>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> > >>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> > >>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org> > >>> > >>> --- > >>> v2 > >>> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%) > >>> --- > >>> tests/Makefile.include | 18 +++++++++++------- > >>> tests/requirements.txt | 2 +- > >>> 2 files changed, 12 insertions(+), 8 deletions(-) > >>> > >>> diff --git a/tests/Makefile.include b/tests/Makefile.include > >>> index 9422ddaece..a4de0ad5a2 100644 > >>> --- a/tests/Makefile.include > >>> +++ b/tests/Makefile.include > >>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv > >>> # download all vm images, according to defined targets > >>> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, > $(FEDORA_31_DOWNLOAD)) > >>> > >>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter > -j%,$(MAKEFLAGS)))) > >>> + > >>> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images > >>> - $(call quiet-command, \ > >>> - $(TESTS_PYTHON) -m avocado \ > >>> - --show=$(AVOCADO_SHOW) run > --job-results-dir=$(TESTS_RESULTS_DIR) \ > >>> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ > >>> - --filter-by-tags-include-empty-key) \ > >>> - $(AVOCADO_CMDLINE_TAGS) \ > >>> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ > >>> + $(call quiet-command, > \ > >>> + $(TESTS_PYTHON) -m avocado > \ > >>> + --show=$(AVOCADO_SHOW) run > --job-results-dir=$(TESTS_RESULTS_DIR) \ > >>> + $(if $(AVOCADO_TAGS),, > \ > >>> + --filter-by-tags-include-empty > \ > >>> + --filter-by-tags-include-empty-key) > \ > >>> + --max-parallel-tasks $(JOBS_OPTION:-j%=%) > \ > >>> + $(AVOCADO_CMDLINE_TAGS) > \ > >>> + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), > \ > >>> "AVOCADO", "tests/avocado") > >>> > >>> check-acceptance-deprecated-warning: > >>> diff --git a/tests/requirements.txt b/tests/requirements.txt > >>> index 0ba561b6bd..a6f73da681 100644 > >>> --- a/tests/requirements.txt > >>> +++ b/tests/requirements.txt > >>> @@ -2,5 +2,5 @@ > >>> # in the tests/venv Python virtual environment. For more info, > >>> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 > >>> # Note that qemu.git/python/ is always implicitly installed. > >>> -avocado-framework==88.1 > >>> +avocado-framework==101.0 > >>> pycdlib==1.11.0 > >>> -- > >>> 2.39.2 > >>> > >> > >> I thought there were test failures that prohibited us from bumping the > >> Avocado-Framework version. Did those get rectified recently? > > No, still not working on Darwin. At this point I'm thinking at sending > a patch clarifying testing QEMU with Avocado is not supported on Darwin. > What's broken on Darwin? Is it broken with avocado-framework==88.1 too? (Is this a regression?) We might need to consider entirely separate issues: (1) We need to upgrade avocado-framework to a supported LTS version >= v90 for dependency harmony with qemu.git/python testing (2) We need to upgrade avocado-framework to >= v101.0 to fix the stated checksum issue in this patch (3) We would like avocado tests to work on Darwin. (Have they ever worked? When did they break? etc.) My understanding is that this patch helps with 1 and 2 but not 3, but if Darwin support isn't a regression, this is still a step forward. > > I'm not seeing any with your patch applied. > > Are you using running the Cirrus-CI jobs? > >
On 4/17/23 09:43, Alex Bennée wrote: > From: Kautuk Consul <kconsul@linux.vnet.ibm.com> > > Avocado version 101.0 has a fix to re-compute the checksum > of an asset file if the algorithm used in the *-CHECKSUM > file isn't the same as the one being passed to it by the > avocado user (i.e. the avocado_qemu python module). > In the earlier avocado versions this fix wasn't there due > to which if the checksum wouldn't match the earlier > checksum (calculated by a different algorithm), the avocado > code would start downloading a fresh image from the internet > URL thus making the test-cases take longer to execute. > > Bump up the avocado-framework version to 101.0. > > Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> > Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> > Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> > Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org> > > --- > v2 > - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%) > --- > tests/Makefile.include | 18 +++++++++++------- > tests/requirements.txt | 2 +- > 2 files changed, 12 insertions(+), 8 deletions(-) Hi everyone, Looks like I've mistakenly replied to the earlier thread, so here's pretty much the same message on a better location. First of all, thanks to Kautuk for sending this and thanks to Hariharan for further testing it. I'd like to give some context which not everyone may be aware of. Avocado 101.0 is a very different when compared with 88.1. Everything related to the execution of tests is brand new. To be more precise, on version 91.0[1], this new runner[2] became the default. On version 97.0, the old runner implementation (currently in use in QEMU) was finally removed. On most releases since then, I've been running the QEMU tests with the latest Avocado, and finding issues that are (as resources allow) addressed in later versions. As you probably noticed, Avocado 101.0 runs the QEMU tests without much (or any) visible issues for most people. But, I'm aware of two pending issues that may or may not be a big deal to users: I) The logging behavior is a bit different since Avocado 88.1. At a given point it was considered that Avocado should not mess around inadvertently with Python's root logger, and should be more picky about it includes in logs. For most cases, a simple workaround[4] does the trick. But, for some other use cases (say for 3rd party libraries' logs you want logged alongside Avocado's logs) there's a pending PR[5] that will take care of all known limitations. II) The support for killing tests (internally in Avocado represented as more generic "tasks") and all its children is a bit lacking. This is an issue I'm actively working on[6]. This may leave some processes (such as "qemu-system-*") running even after a test was interrupted. Fixes for both of these issues are due to be in version 102.0. The ETA for version 102.0 is 1-2 weeks. With that being said, I'm more than OK with this patch (alongside PATCH 2, without which havoc ensues :) provided people understand the two pending issues above. If this patch is taken before Avocado 102.0 is released, the delta from 101.0 would be much smaller, so it should be an easier change to test. Cheers, - Cleber. [1] - https://avocado-framework.readthedocs.io/en/101.0/releases/91_0.html [2] - The new runner is called "nrunner" and I am to be blamed for the naming lacking any originality [3] - https://avocado-framework.readthedocs.io/en/101.0/releases/97_0.html#users-test-writers [4] - https://gitlab.com/cleber.gnu/qemu/-/commit/a9f39c4f6671b756196a185c7275eb7ebd13e588 [5] - https://github.com/avocado-framework/avocado/pull/5645 [6] - https://github.com/avocado-framework/avocado/issues/4994
On 4/17/23 12:50, John Snow wrote: > On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote: >> From: Kautuk Consul <kconsul@linux.vnet.ibm.com> >> >> Avocado version 101.0 has a fix to re-compute the checksum >> of an asset file if the algorithm used in the *-CHECKSUM >> file isn't the same as the one being passed to it by the >> avocado user (i.e. the avocado_qemu python module). >> In the earlier avocado versions this fix wasn't there due >> to which if the checksum wouldn't match the earlier >> checksum (calculated by a different algorithm), the avocado >> code would start downloading a fresh image from the internet >> URL thus making the test-cases take longer to execute. >> >> Bump up the avocado-framework version to 101.0. >> >> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> >> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> >> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> >> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org> >> >> --- >> v2 >> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%) >> --- >> tests/Makefile.include | 18 +++++++++++------- >> tests/requirements.txt | 2 +- >> 2 files changed, 12 insertions(+), 8 deletions(-) >> >> diff --git a/tests/Makefile.include b/tests/Makefile.include >> index 9422ddaece..a4de0ad5a2 100644 >> --- a/tests/Makefile.include >> +++ b/tests/Makefile.include >> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv >> # download all vm images, according to defined targets >> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD)) >> >> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS)))) >> + >> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images >> - $(call quiet-command, \ >> - $(TESTS_PYTHON) -m avocado \ >> - --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ >> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ >> - --filter-by-tags-include-empty-key) \ >> - $(AVOCADO_CMDLINE_TAGS) \ >> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ >> + $(call quiet-command, \ >> + $(TESTS_PYTHON) -m avocado \ >> + --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ >> + $(if $(AVOCADO_TAGS),, \ >> + --filter-by-tags-include-empty \ >> + --filter-by-tags-include-empty-key) \ >> + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \ >> + $(AVOCADO_CMDLINE_TAGS) \ >> + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ >> "AVOCADO", "tests/avocado") >> >> check-acceptance-deprecated-warning: >> diff --git a/tests/requirements.txt b/tests/requirements.txt >> index 0ba561b6bd..a6f73da681 100644 >> --- a/tests/requirements.txt >> +++ b/tests/requirements.txt >> @@ -2,5 +2,5 @@ >> # in the tests/venv Python virtual environment. For more info, >> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 >> # Note that qemu.git/python/ is always implicitly installed. >> -avocado-framework==88.1 >> +avocado-framework==101.0 >> pycdlib==1.11.0 >> -- >> 2.39.2 >> > I thought there were test failures that prohibited us from bumping the > Avocado-Framework version. Did those get rectified recently? > Hi John, While testing before and after behavior of pretty much every test in tests/avocado/, I've found that some tests may get affected by the extra isolation (and overhead) of the new runner in recent Avocado and may get interrupted more often than on the previous runner. But this is highly dependent on the amount of resources the machine you run the tests on has, and how tight the current timeout is. Running all the tests under old and new Avocado also revealed some tests that are equally broken under both versions. Thomas has bisected some, and Cedric has come up with some patches too. To risk not being too verbose by default, let me know if you want the full details. Also, see my previous reply about the two issues I'm aware that would make this bump as seamless as possible. Thanks, - Cleber.
On 4/20/23 05:18, Philippe Mathieu-Daudé wrote: > On 17/4/23 19:44, Alex Bennée wrote: >> >> John Snow <jsnow@redhat.com> writes: >> >>> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> >>> wrote: >>>> >>>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com> >>>> >>>> Avocado version 101.0 has a fix to re-compute the checksum >>>> of an asset file if the algorithm used in the *-CHECKSUM >>>> file isn't the same as the one being passed to it by the >>>> avocado user (i.e. the avocado_qemu python module). >>>> In the earlier avocado versions this fix wasn't there due >>>> to which if the checksum wouldn't match the earlier >>>> checksum (calculated by a different algorithm), the avocado >>>> code would start downloading a fresh image from the internet >>>> URL thus making the test-cases take longer to execute. >>>> >>>> Bump up the avocado-framework version to 101.0. >>>> >>>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> >>>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> >>>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> >>>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org> >>>> >>>> --- >>>> v2 >>>> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%) >>>> --- >>>> tests/Makefile.include | 18 +++++++++++------- >>>> tests/requirements.txt | 2 +- >>>> 2 files changed, 12 insertions(+), 8 deletions(-) >>>> >>>> diff --git a/tests/Makefile.include b/tests/Makefile.include >>>> index 9422ddaece..a4de0ad5a2 100644 >>>> --- a/tests/Makefile.include >>>> +++ b/tests/Makefile.include >>>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv >>>> # download all vm images, according to defined targets >>>> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, >>>> $(FEDORA_31_DOWNLOAD)) >>>> >>>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter >>>> -j%,$(MAKEFLAGS)))) >>>> + >>>> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images >>>> - $(call quiet-command, \ >>>> - $(TESTS_PYTHON) -m avocado \ >>>> - --show=$(AVOCADO_SHOW) run >>>> --job-results-dir=$(TESTS_RESULTS_DIR) \ >>>> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ >>>> - --filter-by-tags-include-empty-key) \ >>>> - $(AVOCADO_CMDLINE_TAGS) \ >>>> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ >>>> + $(call quiet-command, \ >>>> + $(TESTS_PYTHON) -m >>>> avocado \ >>>> + --show=$(AVOCADO_SHOW) run >>>> --job-results-dir=$(TESTS_RESULTS_DIR) \ >>>> + $(if $(AVOCADO_TAGS),, \ >>>> + --filter-by-tags-include-empty \ >>>> + --filter-by-tags-include-empty-key) \ >>>> + --max-parallel-tasks >>>> $(JOBS_OPTION:-j%=%) \ >>>> + $(AVOCADO_CMDLINE_TAGS) \ >>>> + $(if $(GITLAB_CI),,--failfast) >>>> $(AVOCADO_TESTS), \ >>>> "AVOCADO", "tests/avocado") >>>> >>>> check-acceptance-deprecated-warning: >>>> diff --git a/tests/requirements.txt b/tests/requirements.txt >>>> index 0ba561b6bd..a6f73da681 100644 >>>> --- a/tests/requirements.txt >>>> +++ b/tests/requirements.txt >>>> @@ -2,5 +2,5 @@ >>>> # in the tests/venv Python virtual environment. For more info, >>>> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 >>>> # Note that qemu.git/python/ is always implicitly installed. >>>> -avocado-framework==88.1 >>>> +avocado-framework==101.0 >>>> pycdlib==1.11.0 >>>> -- >>>> 2.39.2 >>>> >>> >>> I thought there were test failures that prohibited us from bumping the >>> Avocado-Framework version. Did those get rectified recently? > > No, still not working on Darwin. At this point I'm thinking at sending > a patch clarifying testing QEMU with Avocado is not supported on Darwin. > Hi Phil, Do you mean with or without this version bump? Even though it was somewhat recently[1] that another level of Darwin compatibility was added to Avocado, where a good part of Avocado's selftests[2] were enabled in OS X, Avocado 101.0 should not be that far behind. I honestly believe people testing and reporting successful execution of the Avocado tests under Darwin with latest Avocado. Hopefully my memory is not playing tricks on me, but I'll attempt to verify that. Cheers, - Cleber. [1] - https://github.com/avocado-framework/avocado/pull/5622 [2] - https://github.com/avocado-framework/avocado/actions/runs/4417285317/jobs/7742772932?pr=5622#step:10:11 >> I'm not seeing any with your patch applied. > > Are you using running the Cirrus-CI jobs? >
On 4/20/23 12:54, John Snow wrote: > > > On Thu, Apr 20, 2023, 5:19 AM Philippe Mathieu-Daudé > <philmd@linaro.org> wrote: > > On 17/4/23 19:44, Alex Bennée wrote: > > > > John Snow <jsnow@redhat.com> writes: > > > >> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée > <alex.bennee@linaro.org> wrote: > >>> > >>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com> > >>> > >>> Avocado version 101.0 has a fix to re-compute the checksum > >>> of an asset file if the algorithm used in the *-CHECKSUM > >>> file isn't the same as the one being passed to it by the > >>> avocado user (i.e. the avocado_qemu python module). > >>> In the earlier avocado versions this fix wasn't there due > >>> to which if the checksum wouldn't match the earlier > >>> checksum (calculated by a different algorithm), the avocado > >>> code would start downloading a fresh image from the internet > >>> URL thus making the test-cases take longer to execute. > >>> > >>> Bump up the avocado-framework version to 101.0. > >>> > >>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> > >>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> > >>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> > >>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org> > >>> > >>> --- > >>> v2 > >>> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%) > >>> --- > >>> tests/Makefile.include | 18 +++++++++++------- > >>> tests/requirements.txt | 2 +- > >>> 2 files changed, 12 insertions(+), 8 deletions(-) > >>> > >>> diff --git a/tests/Makefile.include b/tests/Makefile.include > >>> index 9422ddaece..a4de0ad5a2 100644 > >>> --- a/tests/Makefile.include > >>> +++ b/tests/Makefile.include > >>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv > >>> # download all vm images, according to defined targets > >>> get-vm-images: check-venv $(patsubst > %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD)) > >>> > >>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter > -j%,$(MAKEFLAGS)))) > >>> + > >>> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images > >>> - $(call quiet-command, \ > >>> - $(TESTS_PYTHON) -m avocado \ > >>> - --show=$(AVOCADO_SHOW) run > --job-results-dir=$(TESTS_RESULTS_DIR) \ > >>> - $(if $(AVOCADO_TAGS),, > --filter-by-tags-include-empty \ > >>> - --filter-by-tags-include-empty-key) \ > >>> - $(AVOCADO_CMDLINE_TAGS) \ > >>> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ > >>> + $(call quiet-command, \ > >>> + $(TESTS_PYTHON) -m avocado > \ > >>> + --show=$(AVOCADO_SHOW) run > --job-results-dir=$(TESTS_RESULTS_DIR) \ > >>> + $(if $(AVOCADO_TAGS),, > \ > >>> + --filter-by-tags-include-empty \ > >>> + --filter-by-tags-include-empty-key) \ > >>> + --max-parallel-tasks $(JOBS_OPTION:-j%=%) > \ > >>> + $(AVOCADO_CMDLINE_TAGS) > \ > >>> + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), > \ > >>> "AVOCADO", "tests/avocado") > >>> > >>> check-acceptance-deprecated-warning: > >>> diff --git a/tests/requirements.txt b/tests/requirements.txt > >>> index 0ba561b6bd..a6f73da681 100644 > >>> --- a/tests/requirements.txt > >>> +++ b/tests/requirements.txt > >>> @@ -2,5 +2,5 @@ > >>> # in the tests/venv Python virtual environment. For more info, > >>> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 > >>> # Note that qemu.git/python/ is always implicitly installed. > >>> -avocado-framework==88.1 > >>> +avocado-framework==101.0 > >>> pycdlib==1.11.0 > >>> -- > >>> 2.39.2 > >>> > >> > >> I thought there were test failures that prohibited us from > bumping the > >> Avocado-Framework version. Did those get rectified recently? > > No, still not working on Darwin. At this point I'm thinking at sending > a patch clarifying testing QEMU with Avocado is not supported on > Darwin. > > > What's broken on Darwin? > Hi John, IIRC, the main issue is that the legacy runner (the one active in 88.1) pickles some stuff that do not play nice under Darwin. It has never (to the best of my knowledge) worked under Darwin. > Is it broken with avocado-framework==88.1 too? (Is this a regression?) > It was already broken with avocado-framework==88.1, but it should mostly work with 101.0 (see previous reply). > We might need to consider entirely separate issues: > > (1) We need to upgrade avocado-framework to a supported LTS version >= > v90 for dependency harmony with qemu.git/python testing > > (2) We need to upgrade avocado-framework to >= v101.0 to fix the > stated checksum issue in this patch > > (3) We would like avocado tests to work on Darwin. (Have they ever > worked? When did they break? etc.) > > My understanding is that this patch helps with 1 and 2 but not 3, but > if Darwin support isn't a regression, this is still a step forward. > To the best of my knowledge, this patch would help with 1, 2 and even 3. Cheers, - Cleber. > > > I'm not seeing any with your patch applied. > > Are you using running the Cirrus-CI jobs? >
On 4/18/23 03:10, Thomas Huth wrote: > On 17/04/2023 15.43, Alex Bennée wrote: >> From: Kautuk Consul <kconsul@linux.vnet.ibm.com> >> >> Avocado version 101.0 has a fix to re-compute the checksum >> of an asset file if the algorithm used in the *-CHECKSUM >> file isn't the same as the one being passed to it by the >> avocado user (i.e. the avocado_qemu python module). >> In the earlier avocado versions this fix wasn't there due >> to which if the checksum wouldn't match the earlier >> checksum (calculated by a different algorithm), the avocado >> code would start downloading a fresh image from the internet >> URL thus making the test-cases take longer to execute. >> >> Bump up the avocado-framework version to 101.0. >> >> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> >> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> >> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> >> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org> >> >> --- >> v2 >> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%) >> --- >> tests/Makefile.include | 18 +++++++++++------- >> tests/requirements.txt | 2 +- >> 2 files changed, 12 insertions(+), 8 deletions(-) >> >> diff --git a/tests/Makefile.include b/tests/Makefile.include >> index 9422ddaece..a4de0ad5a2 100644 >> --- a/tests/Makefile.include >> +++ b/tests/Makefile.include >> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv >> # download all vm images, according to defined targets >> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, >> $(FEDORA_31_DOWNLOAD)) >> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter >> -j%,$(MAKEFLAGS)))) >> + >> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images >> - $(call quiet-command, \ >> - $(TESTS_PYTHON) -m avocado \ >> - --show=$(AVOCADO_SHOW) run >> --job-results-dir=$(TESTS_RESULTS_DIR) \ >> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ >> - --filter-by-tags-include-empty-key) \ >> - $(AVOCADO_CMDLINE_TAGS) \ >> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ >> + $(call quiet-command, \ >> + $(TESTS_PYTHON) -m avocado \ >> + --show=$(AVOCADO_SHOW) run >> --job-results-dir=$(TESTS_RESULTS_DIR) \ >> + $(if $(AVOCADO_TAGS),, \ >> + --filter-by-tags-include-empty \ >> + --filter-by-tags-include-empty-key) \ >> + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \ >> + $(AVOCADO_CMDLINE_TAGS) \ >> + $(if $(GITLAB_CI),,--failfast) >> $(AVOCADO_TESTS), \ > > We might need to revisit this --failfast logic, too. If I've got that > right, failfast is now the default with the new system? So we might > want to disable it by default again if GITLAB_CI is not set? > > Thomas > Hi Thomas, I must be missing something, because under Avocado 101.0, I'm getting the following behavior without the --failfast flag: $ avocado run --max-parallel-tasks=1 -- /bin/true /bin/false /bin/true JOB ID : 646f476f01b8d5599a57530606de543f2d9a5366 JOB LOG : /root/avocado/job-results/job-2023-04-21T04.15-646f476/job.log (1/3) /bin/true: STARTED (1/3) /bin/true: PASS (0.01 s) (2/3) /bin/false: STARTED (2/3) /bin/false: FAIL (0.01 s) (3/3) /bin/true: STARTED (3/3) /bin/true: PASS (0.01 s) RESULTS : PASS 2 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 1.01 s Test summary: /bin/false: FAIL And this with --failfast: $ avocado run --failfast --max-parallel-tasks=1 -- /bin/true /bin/false /bin/true JOB ID : ae4894607a42194a7382efa545eccaccf7495fa3 JOB LOG : /root/avocado/job-results/job-2023-04-21T04.17-ae48946/job.log (1/3) /bin/true: STARTED (1/3) /bin/true: PASS (0.01 s) (2/3) /bin/false: STARTED (2/3) /bin/false: FAIL (0.01 s) Interrupting job (failfast). RESULTS : PASS 1 | ERROR 0 | FAIL 1 | SKIP 1 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 0.75 s Test summary: /bin/false: FAIL Maybe it's something in the Makefile I'm missing, or something specific to these tests, but that would be a long shot. Thanks, - Cleber.
On 21/04/2023 06.19, Cleber Rosa wrote: > > On 4/18/23 03:10, Thomas Huth wrote: >> On 17/04/2023 15.43, Alex Bennée wrote: >>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com> >>> >>> Avocado version 101.0 has a fix to re-compute the checksum >>> of an asset file if the algorithm used in the *-CHECKSUM >>> file isn't the same as the one being passed to it by the >>> avocado user (i.e. the avocado_qemu python module). >>> In the earlier avocado versions this fix wasn't there due >>> to which if the checksum wouldn't match the earlier >>> checksum (calculated by a different algorithm), the avocado >>> code would start downloading a fresh image from the internet >>> URL thus making the test-cases take longer to execute. >>> >>> Bump up the avocado-framework version to 101.0. >>> >>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> >>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> >>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> >>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org> >>> >>> --- >>> v2 >>> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%) >>> --- >>> tests/Makefile.include | 18 +++++++++++------- >>> tests/requirements.txt | 2 +- >>> 2 files changed, 12 insertions(+), 8 deletions(-) >>> >>> diff --git a/tests/Makefile.include b/tests/Makefile.include >>> index 9422ddaece..a4de0ad5a2 100644 >>> --- a/tests/Makefile.include >>> +++ b/tests/Makefile.include >>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv >>> # download all vm images, according to defined targets >>> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, >>> $(FEDORA_31_DOWNLOAD)) >>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS)))) >>> + >>> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images >>> - $(call quiet-command, \ >>> - $(TESTS_PYTHON) -m avocado \ >>> - --show=$(AVOCADO_SHOW) run >>> --job-results-dir=$(TESTS_RESULTS_DIR) \ >>> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ >>> - --filter-by-tags-include-empty-key) \ >>> - $(AVOCADO_CMDLINE_TAGS) \ >>> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ >>> + $(call quiet-command, \ >>> + $(TESTS_PYTHON) -m avocado \ >>> + --show=$(AVOCADO_SHOW) run >>> --job-results-dir=$(TESTS_RESULTS_DIR) \ >>> + $(if $(AVOCADO_TAGS),, \ >>> + --filter-by-tags-include-empty \ >>> + --filter-by-tags-include-empty-key) \ >>> + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \ >>> + $(AVOCADO_CMDLINE_TAGS) \ >>> + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ >> >> We might need to revisit this --failfast logic, too. If I've got that >> right, failfast is now the default with the new system? So we might want >> to disable it by default again if GITLAB_CI is not set? >> >> Thomas >> > Hi Thomas, > > I must be missing something, because under Avocado 101.0, I'm getting the > following behavior without the --failfast flag: > > $ avocado run --max-parallel-tasks=1 -- /bin/true /bin/false /bin/true > JOB ID : 646f476f01b8d5599a57530606de543f2d9a5366 > JOB LOG : /root/avocado/job-results/job-2023-04-21T04.15-646f476/job.log > (1/3) /bin/true: STARTED > (1/3) /bin/true: PASS (0.01 s) > (2/3) /bin/false: STARTED > (2/3) /bin/false: FAIL (0.01 s) > (3/3) /bin/true: STARTED > (3/3) /bin/true: PASS (0.01 s) > RESULTS : PASS 2 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | > CANCEL 0 > JOB TIME : 1.01 s > > Test summary: > /bin/false: FAIL > > And this with --failfast: > > $ avocado run --failfast --max-parallel-tasks=1 -- /bin/true /bin/false > /bin/true > JOB ID : ae4894607a42194a7382efa545eccaccf7495fa3 > JOB LOG : /root/avocado/job-results/job-2023-04-21T04.17-ae48946/job.log > (1/3) /bin/true: STARTED > (1/3) /bin/true: PASS (0.01 s) > (2/3) /bin/false: STARTED > (2/3) /bin/false: FAIL (0.01 s) > Interrupting job (failfast). > RESULTS : PASS 1 | ERROR 0 | FAIL 1 | SKIP 1 | WARN 0 | INTERRUPT 0 | > CANCEL 0 > JOB TIME : 0.75 s > > Test summary: > /bin/false: FAIL > > Maybe it's something in the Makefile I'm missing, or something specific to > these tests, but that would be a long shot. Ah, thanks, I think I simply got it mixed up while trying to read the above Makefile logic - never mind my question, it should be fine. Thomas
On 17/04/2023 15.43, Alex Bennée wrote: > From: Kautuk Consul <kconsul@linux.vnet.ibm.com> > > Avocado version 101.0 has a fix to re-compute the checksum > of an asset file if the algorithm used in the *-CHECKSUM > file isn't the same as the one being passed to it by the > avocado user (i.e. the avocado_qemu python module). > In the earlier avocado versions this fix wasn't there due > to which if the checksum wouldn't match the earlier > checksum (calculated by a different algorithm), the avocado > code would start downloading a fresh image from the internet > URL thus making the test-cases take longer to execute. > > Bump up the avocado-framework version to 101.0. > > Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> > Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> > Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> > Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org> > > --- > v2 > - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%) > --- > tests/Makefile.include | 18 +++++++++++------- > tests/requirements.txt | 2 +- > 2 files changed, 12 insertions(+), 8 deletions(-) > > diff --git a/tests/Makefile.include b/tests/Makefile.include > index 9422ddaece..a4de0ad5a2 100644 > --- a/tests/Makefile.include > +++ b/tests/Makefile.include > @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv > # download all vm images, according to defined targets > get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD)) > > +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS)))) > + > check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images > - $(call quiet-command, \ > - $(TESTS_PYTHON) -m avocado \ > - --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ > - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ > - --filter-by-tags-include-empty-key) \ > - $(AVOCADO_CMDLINE_TAGS) \ > - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ > + $(call quiet-command, \ > + $(TESTS_PYTHON) -m avocado \ > + --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ > + $(if $(AVOCADO_TAGS),, \ > + --filter-by-tags-include-empty \ > + --filter-by-tags-include-empty-key) \ > + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \ > + $(AVOCADO_CMDLINE_TAGS) \ > + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ > "AVOCADO", "tests/avocado") > > check-acceptance-deprecated-warning: > diff --git a/tests/requirements.txt b/tests/requirements.txt > index 0ba561b6bd..a6f73da681 100644 > --- a/tests/requirements.txt > +++ b/tests/requirements.txt > @@ -2,5 +2,5 @@ > # in the tests/venv Python virtual environment. For more info, > # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 > # Note that qemu.git/python/ is always implicitly installed. > -avocado-framework==88.1 > +avocado-framework==101.0 > pycdlib==1.11.0 Reviewed-by: Thomas Huth <thuth@redhat.com>
On 21/4/23 06:09, Cleber Rosa wrote: > > On 4/20/23 12:54, John Snow wrote: >> >> >> On Thu, Apr 20, 2023, 5:19 AM Philippe Mathieu-Daudé >> <philmd@linaro.org> wrote: >> >> On 17/4/23 19:44, Alex Bennée wrote: >> > >> > John Snow <jsnow@redhat.com> writes: >> > >> >> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée >> <alex.bennee@linaro.org> wrote: >> >>> diff --git a/tests/requirements.txt b/tests/requirements.txt >> >>> index 0ba561b6bd..a6f73da681 100644 >> >>> --- a/tests/requirements.txt >> >>> +++ b/tests/requirements.txt >> >>> @@ -2,5 +2,5 @@ >> >>> # in the tests/venv Python virtual environment. For more info, >> >>> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 >> >>> # Note that qemu.git/python/ is always implicitly installed. >> >>> -avocado-framework==88.1 >> >>> +avocado-framework==101.0 >> >>> pycdlib==1.11.0 >> >>> -- >> >>> 2.39.2 >> >>> >> >> >> >> I thought there were test failures that prohibited us from >> bumping the >> >> Avocado-Framework version. Did those get rectified recently? >> >> No, still not working on Darwin. At this point I'm thinking at >> sending >> a patch clarifying testing QEMU with Avocado is not supported on >> Darwin. >> >> >> What's broken on Darwin? >> > Hi John, > > IIRC, the main issue is that the legacy runner (the one active in 88.1) > pickles some stuff that do not play nice under Darwin. It has never (to > the best of my knowledge) worked under Darwin. Correct. >> Is it broken with avocado-framework==88.1 too? (Is this a regression?) >> > It was already broken with avocado-framework==88.1, Correct. > but it should mostly > work with 101.0 (see previous reply). > >> We might need to consider entirely separate issues: >> >> (1) We need to upgrade avocado-framework to a supported LTS version >= >> v90 for dependency harmony with qemu.git/python testing >> >> (2) We need to upgrade avocado-framework to >= v101.0 to fix the >> stated checksum issue in this patch >> >> (3) We would like avocado tests to work on Darwin. (Have they ever >> worked? When did they break? etc.) >> >> My understanding is that this patch helps with 1 and 2 but not 3, but >> if Darwin support isn't a regression, this is still a step forward. >> > To the best of my knowledge, this patch would help with 1, 2 and even 3. As of today using v8.0.0: $ make check-avocado GIT ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc MKDIR /Users/philmd/source/qemu/build/tests/results AVOCADO Downloading avocado tests VM image for aarch64 The image was downloaded: Provider Version Architecture File fedora 31 aarch64 /Users/philmd/avocado/data/cache/by_location/4f156e531446a679cbfe13caef8b7c9f9f79aafa/Fedora-Cloud-Base-31-1.9.aarch64.qcow2 AVOCADO tests/avocado Fetching asset from tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf Fetching asset from tests/avocado/boot_xen.py:BootXen.test_arm64_xen_411_and_dom0 Fetching asset from tests/avocado/boot_xen.py:BootXen.test_arm64_xen_414_and_dom0 Fetching asset from tests/avocado/boot_xen.py:BootXen.test_arm64_xen_415_and_dom0 Fetching asset from tests/avocado/machine_aarch64_virt.py:Aarch64VirtMachine.test_alpine_virt_tcg_gic_max Fetching asset from tests/avocado/replay_kernel.py:ReplayKernelNormal.test_aarch64_virt Fetching asset from tests/avocado/reverse_debugging.py:ReverseDebugging_AArch64.test_aarch64_virt JOB ID : 142a0b2d3960c7daccd657781d52f814e1617392 JOB LOG : /Users/philmd/source/qemu/build/tests/results/job-2023-04-21T11.35-142a0b2/job.log Avocado crashed: TypeError: cannot pickle '_thread.RLock' object Traceback (most recent call last): File "/Users/philmd/source/qemu/build/tests/venv/lib/python3.10/site-packages/avocado/core/job.py", line 632, in run_tests summary |= suite.run(self) File "/Users/philmd/source/qemu/build/tests/venv/lib/python3.10/site-packages/avocado/core/suite.py", line 311, in run return self.runner.run_suite(job, self) File "/Users/philmd/source/qemu/build/tests/venv/lib/python3.10/site-packages/avocado/plugins/runner.py", line 390, in run_suite if not self.run_test(job, test_factory, queue, summary, File "/Users/philmd/source/qemu/build/tests/venv/lib/python3.10/site-packages/avocado/plugins/runner.py", line 170, in run_test proc.start() File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 121, in start self._popen = self._Popen(self) File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 224, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 288, in _Popen return Popen(process_obj) File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__ super().__init__(process_obj) File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__ self._launch(process_obj) File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch reduction.dump(process_obj, fp) File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) TypeError: cannot pickle '_thread.RLock' object Please include the traceback info and command line used on your bug report Report bugs visiting https://github.com/avocado-framework/avocado/issues/new make: *** [check-avocado] Error 4 This is https://github.com/avocado-framework/avocado/issues/4888 which is still opened. Manually using "avocado run --test-runner=nrunner" helps running the tests but the console output is never displayed (see [*]) and once finished Avocado lets various QEMU processes running in detached mode, wasting resources. [*] https://lore.kernel.org/qemu-devel/1d0baf8b-c757-265c-b206-07ca3f218b2a@linaro.org/
On 21/4/23 12:07, Philippe Mathieu-Daudé wrote: > On 21/4/23 06:09, Cleber Rosa wrote: > Manually using "avocado run --test-runner=nrunner" helps running the > tests but the console output is never displayed (see [*]) and once > finished Avocado lets various QEMU processes running in detached > mode, wasting resources.= Using this patch: $ ./tests/venv/bin/avocado --show=console -V run -t arch:aarch64 tests/avocado/boot_* tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf: ERROR tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_xlnx_versal_virt: ERROR Not asking for the console output at least show the test being processed (similar to old runner): $ ./tests/venv/bin/avocado run -t machine:raspi3b tests/avocado/boot_* Fetching asset from tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf JOB ID : 1c963c12f7f07e109ef8480ddf204f02e61f8b8d JOB LOG : /Users/philmd/avocado/job-results/job-2023-04-21T12.24-1c963c1/job.log (1/1) tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf: STARTED (1/1) tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf: ERROR: ConnectError: Failed to establish session: EOFError\n Exit code: 1\n Command: ./qemu-system-aarch64 -display none -vga none -chardev socket,id=mon,fd=11 -mon chardev=mon,mode=control -machine raspi3b -serial null -chardev socket,id=console,path=/var/folders... (0.31 s) RESULTS : PASS 0 | ERROR 1 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 2.20 s Test summary: tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf: ERROR
diff --git a/tests/Makefile.include b/tests/Makefile.include index 9422ddaece..a4de0ad5a2 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv # download all vm images, according to defined targets get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD)) +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS)))) + check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images - $(call quiet-command, \ - $(TESTS_PYTHON) -m avocado \ - --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ - --filter-by-tags-include-empty-key) \ - $(AVOCADO_CMDLINE_TAGS) \ - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ + $(call quiet-command, \ + $(TESTS_PYTHON) -m avocado \ + --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ + $(if $(AVOCADO_TAGS),, \ + --filter-by-tags-include-empty \ + --filter-by-tags-include-empty-key) \ + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \ + $(AVOCADO_CMDLINE_TAGS) \ + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ "AVOCADO", "tests/avocado") check-acceptance-deprecated-warning: diff --git a/tests/requirements.txt b/tests/requirements.txt index 0ba561b6bd..a6f73da681 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -2,5 +2,5 @@ # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 # Note that qemu.git/python/ is always implicitly installed. -avocado-framework==88.1 +avocado-framework==101.0 pycdlib==1.11.0