| Message ID | 20240729152714.10225-1-philmd@linaro.org |
|---|---|
| State | New |
| Headers | show |
On Mon, Jul 29, 2024 at 05:27:00PM +0200, Philippe Mathieu-Daudé wrote: > The following changes since commit 93b799fafd9170da3a79a533ea6f73a18de82e22: > > Merge tag 'pull-ppc-for-9.1-2-20240726-1' of https://gitlab.com/npiggin/qemu into staging (2024-07-26 15:10:45 +1000) > > are available in the Git repository at: > > https://github.com/philmd/qemu.git tags/docs-testing-20240729 > > for you to fetch changes up to 8e2275592299b637e87cc905d20bfdeffa586a83: > > tests/avocado: test_arm_emcraft_sf2: handle RW requirements for asset (2024-07-29 16:33:24 +0200) > > Ignored warnings: > > WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? > WARNING: line over 80 characters > > ---------------------------------------------------------------- > Docs & testing patch queue > > - Test QAPI firmware.json schema (Thomas) > - Handle new env.doc2path() return value (Peter) > - Convert some Avocado tests to simpler python-based functional testing (Thomas) > - Improve how assets are used by some Avocado tests (Cleber) > > ---------------------------------------------------------------- > > Cleber Rosa (3): > tests/avocado: mips: fallback to HTTP given certificate expiration > tests/avocado: mips: add hint for fetchasset plugin > tests/avocado: test_arm_emcraft_sf2: handle RW requirements for asset > > Peter Maydell (1): > docs/sphinx/depfile.py: Handle env.doc2path() returning a Path not a > str > > Thomas Huth (9): > python: Install pycotap in our venv if necessary > tests/functional: Add base classes for the upcoming pytest-based tests > tests/Makefile.include: Increase the level of indentation in the help > text > tests/functional: Prepare the meson build system for the functional > tests > tests/functional: Convert simple avocado tests into standalone python > tests > tests/functional: Convert avocado tests that just need a small > adjustment > tests/functional: Convert the x86_cpu_model_versions test > tests/functional: Convert the riscv_opensbi avocado test into a > standalone test > gitlab-ci: Add "check-functional" to the build tests On the avocado review I mentioned that I didn't think we should be rebasing avocado during freeze. By the same token, I'm not convinced we should be introducing a new test framework during the freeze period. This is alot simpler than avocado, but at the same time this small subset of Thomas' patches isn't really fixing any problem on its own, as avocado still exists in parallel. With regards, Daniel
On 29/7/24 17:36, Daniel P. Berrangé wrote: > On Mon, Jul 29, 2024 at 05:27:00PM +0200, Philippe Mathieu-Daudé wrote: >> Thomas Huth (9): >> python: Install pycotap in our venv if necessary >> tests/functional: Add base classes for the upcoming pytest-based tests >> tests/Makefile.include: Increase the level of indentation in the help >> text >> tests/functional: Prepare the meson build system for the functional >> tests >> tests/functional: Convert simple avocado tests into standalone python >> tests >> tests/functional: Convert avocado tests that just need a small >> adjustment >> tests/functional: Convert the x86_cpu_model_versions test >> tests/functional: Convert the riscv_opensbi avocado test into a >> standalone test >> gitlab-ci: Add "check-functional" to the build tests > > On the avocado review I mentioned that I didn't think we should > be rebasing avocado during freeze. By the same token, I'm not > convinced we should be introducing a new test framework during > the freeze period. This is alot simpler than avocado, but at > the same time this small subset of Thomas' patches isn't really > fixing any problem on its own, as avocado still exists in > parallel. - As you noticed, Avocado isn't rebased here - The python file added is almost a copy of the avocado base class, already well tested - Test are moved from Avocado, so no waste of time running them twice - macOS developers can run 178 functional tests. This is the big win and time saver for me. Do you really see merging these few patches now as being counter productive for the community? Regards, Phil.
On Tue, 30 Jul 2024 at 11:05, Philippe Mathieu-Daudé <philmd@linaro.org> wrote: > > On 29/7/24 17:36, Daniel P. Berrangé wrote: > > On Mon, Jul 29, 2024 at 05:27:00PM +0200, Philippe Mathieu-Daudé wrote: > > >> Thomas Huth (9): > >> python: Install pycotap in our venv if necessary > >> tests/functional: Add base classes for the upcoming pytest-based tests > >> tests/Makefile.include: Increase the level of indentation in the help > >> text > >> tests/functional: Prepare the meson build system for the functional > >> tests > >> tests/functional: Convert simple avocado tests into standalone python > >> tests > >> tests/functional: Convert avocado tests that just need a small > >> adjustment > >> tests/functional: Convert the x86_cpu_model_versions test > >> tests/functional: Convert the riscv_opensbi avocado test into a > >> standalone test > >> gitlab-ci: Add "check-functional" to the build tests > > > > On the avocado review I mentioned that I didn't think we should > > be rebasing avocado during freeze. By the same token, I'm not > > convinced we should be introducing a new test framework during > > the freeze period. This is alot simpler than avocado, but at > > the same time this small subset of Thomas' patches isn't really > > fixing any problem on its own, as avocado still exists in > > parallel. > > - As you noticed, Avocado isn't rebased here > - The python file added is almost a copy of the avocado base class, > already well tested > - Test are moved from Avocado, so no waste of time running them > twice > - macOS developers can run 178 functional tests. This is the big > win and time saver for me. > > Do you really see merging these few patches now as being > counter productive for the community? I do feel from the discussion in other threads that "what is the way forward for our currently-in-avocado test cases?" is currently still an open question, so I tend to agree with Daniel that it would be better to resolve that question first, and we don't need to be in a hurry to change things in our test framework during the freeze period. thanks -- PMM
On 30/7/24 12:14, Peter Maydell wrote: > On Tue, 30 Jul 2024 at 11:05, Philippe Mathieu-Daudé <philmd@linaro.org> wrote: >> >> On 29/7/24 17:36, Daniel P. Berrangé wrote: >>> On Mon, Jul 29, 2024 at 05:27:00PM +0200, Philippe Mathieu-Daudé wrote: >> >>>> Thomas Huth (9): >>>> python: Install pycotap in our venv if necessary >>>> tests/functional: Add base classes for the upcoming pytest-based tests >>>> tests/Makefile.include: Increase the level of indentation in the help >>>> text >>>> tests/functional: Prepare the meson build system for the functional >>>> tests >>>> tests/functional: Convert simple avocado tests into standalone python >>>> tests >>>> tests/functional: Convert avocado tests that just need a small >>>> adjustment >>>> tests/functional: Convert the x86_cpu_model_versions test >>>> tests/functional: Convert the riscv_opensbi avocado test into a >>>> standalone test >>>> gitlab-ci: Add "check-functional" to the build tests >>> >>> On the avocado review I mentioned that I didn't think we should >>> be rebasing avocado during freeze. By the same token, I'm not >>> convinced we should be introducing a new test framework during >>> the freeze period. This is alot simpler than avocado, but at >>> the same time this small subset of Thomas' patches isn't really >>> fixing any problem on its own, as avocado still exists in >>> parallel. >> >> - As you noticed, Avocado isn't rebased here >> - The python file added is almost a copy of the avocado base class, >> already well tested >> - Test are moved from Avocado, so no waste of time running them >> twice >> - macOS developers can run 178 functional tests. This is the big >> win and time saver for me. >> >> Do you really see merging these few patches now as being >> counter productive for the community? > > I do feel from the discussion in other threads that "what is > the way forward for our currently-in-avocado test cases?" is > currently still an open question, Thomas demonstrated with few LoC we can leverage some Avocado libraries (util, archive, ...) without having to use Avocado framework as our entry point to run acceptance tests - which is what gave us headaches the last 2 years -. Richard only requirement is a "fetch assets" equivalent. We'll look at. I don't see anything else blocking this test conversion. > so I tend to agree with Daniel > that it would be better to resolve that question first, > and we don't need to be in a hurry to change things in > our test framework during the freeze period. OK. I'll repost without these patches. Regards, Phil.
The following changes since commit 93b799fafd9170da3a79a533ea6f73a18de82e22: Merge tag 'pull-ppc-for-9.1-2-20240726-1' of https://gitlab.com/npiggin/qemu into staging (2024-07-26 15:10:45 +1000) are available in the Git repository at: https://github.com/philmd/qemu.git tags/docs-testing-20240729 for you to fetch changes up to 8e2275592299b637e87cc905d20bfdeffa586a83: tests/avocado: test_arm_emcraft_sf2: handle RW requirements for asset (2024-07-29 16:33:24 +0200) Ignored warnings: WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? WARNING: line over 80 characters ---------------------------------------------------------------- Docs & testing patch queue - Test QAPI firmware.json schema (Thomas) - Handle new env.doc2path() return value (Peter) - Convert some Avocado tests to simpler python-based functional testing (Thomas) - Improve how assets are used by some Avocado tests (Cleber) ---------------------------------------------------------------- Cleber Rosa (3): tests/avocado: mips: fallback to HTTP given certificate expiration tests/avocado: mips: add hint for fetchasset plugin tests/avocado: test_arm_emcraft_sf2: handle RW requirements for asset Peter Maydell (1): docs/sphinx/depfile.py: Handle env.doc2path() returning a Path not a str Thomas Huth (9): python: Install pycotap in our venv if necessary tests/functional: Add base classes for the upcoming pytest-based tests tests/Makefile.include: Increase the level of indentation in the help text tests/functional: Prepare the meson build system for the functional tests tests/functional: Convert simple avocado tests into standalone python tests tests/functional: Convert avocado tests that just need a small adjustment tests/functional: Convert the x86_cpu_model_versions test tests/functional: Convert the riscv_opensbi avocado test into a standalone test gitlab-ci: Add "check-functional" to the build tests Thomas Weißschuh (1): docs: add test for firmware.json QAPI MAINTAINERS | 11 + docs/meson.build | 5 + docs/sphinx/depfile.py | 2 +- .gitlab-ci.d/buildtest-template.yml | 3 +- .gitlab-ci.d/buildtest.yml | 60 ++-- python/wheels/pycotap-1.3.1-py3-none-any.whl | Bin 0 -> 5119 bytes pythondeps.toml | 1 + tests/Makefile.include | 41 ++- tests/avocado/boot_linux_console.py | 24 +- tests/avocado/riscv_opensbi.py | 63 ---- tests/functional/meson.build | 86 +++++ tests/functional/qemu_test/__init__.py | 328 ++++++++++++++++++ .../test_cpu_queries.py} | 7 +- .../test_empty_cpu_model.py} | 7 +- .../test_info_usernet.py} | 11 +- .../test_mem_addr_space.py} | 52 +-- .../test_pc_cpu_hotplug_props.py} | 11 +- .../test_ppc_74xx.py} | 74 ++-- tests/functional/test_riscv_opensbi.py | 36 ++ .../version.py => functional/test_version.py} | 13 +- .../test_virtio_version.py} | 8 +- .../test_x86_cpu_model_versions.py} | 63 +--- tests/meson.build | 1 + 23 files changed, 636 insertions(+), 271 deletions(-) create mode 100644 python/wheels/pycotap-1.3.1-py3-none-any.whl delete mode 100644 tests/avocado/riscv_opensbi.py create mode 100644 tests/functional/meson.build create mode 100644 tests/functional/qemu_test/__init__.py rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%) mode change 100644 => 100755 rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%) mode change 100644 => 100755 rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%) mode change 100644 => 100755 rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (93%) mode change 100644 => 100755 rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%) mode change 100644 => 100755 rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%) mode change 100644 => 100755 create mode 100755 tests/functional/test_riscv_opensbi.py rename tests/{avocado/version.py => functional/test_version.py} (78%) mode change 100644 => 100755 rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%) mode change 100644 => 100755 rename tests/{avocado/x86_cpu_model_versions.py => functional/test_x86_cpu_model_versions.py} (92%) mode change 100644 => 100755