[v3,0/3] kunit: support running subsets of test suites from kunit.py

When using `kunit.py run` to run tests, users must populate a
`kunitconfig` file to select the options the tests are hidden behind and
all their dependencies.

The patch [1] to allow specifying a path to kunitconfig promises to make
this nicer as we can have checked in files corresponding to different
sets of tests.

But it's still annoying 
1) when trying to run a subet of tests
2) when you want to run tests that don't have such a pre-existing
kunitconfig and selecting all the necessary options is tricky.

This patch series aims to alleviate both:
1) `kunit.py run 'my-suite-*'`
I.e. use my current kunitconfig, but just run suites that match this glob
2) `kunit.py run --alltests 'my-suite-*'`
I.e. use allyesconfig so I don't have to worry about writing a
kunitconfig at all.

See the first commit message for more details and discussion about
future work.

This patch series also includes a bugfix for a latent bug that can't be
triggered right now but has worse consequences as a result of the
changes needed to plumb in this suite name glob.

[1] https://lore.kernel.org/linux-kselftest/20210201205514.3943096-1-dlatypov@google.com/

---
v1 -> v2:
  Fix free of `suites` subarray in suite_set.
  Found by Dan Carpenter and kernel test robot.
v2 -> v3:
  Add MODULE_PARM_DESC() for kunit.filter_glob.

Daniel Latypov (3):
  kunit: add kunit.filter_glob cmdline option to filter suites
  kunit: tool: add support for filtering suites by glob
  kunit: tool: fix unintentional statefulness in run_kernel()

 lib/kunit/Kconfig                   |  1 +
 lib/kunit/executor.c                | 93 ++++++++++++++++++++++++++---
 tools/testing/kunit/kunit.py        | 21 +++++--
 tools/testing/kunit/kunit_kernel.py |  6 +-
 4 files changed, 106 insertions(+), 15 deletions(-)


base-commit: 88bb507a74ea7d75fa49edd421eaa710a7d80598

On Thu, Feb 4, 2021 at 2:54 PM Daniel Latypov <dlatypov@google.com> wrote:
>

> When using `kunit.py run` to run tests, users must populate a

> `kunitconfig` file to select the options the tests are hidden behind and

> all their dependencies.

>

> The patch [1] to allow specifying a path to kunitconfig promises to make

> this nicer as we can have checked in files corresponding to different

> sets of tests.

>

> But it's still annoying

> 1) when trying to run a subet of tests

> 2) when you want to run tests that don't have such a pre-existing

> kunitconfig and selecting all the necessary options is tricky.

>

> This patch series aims to alleviate both:

> 1) `kunit.py run 'my-suite-*'`

> I.e. use my current kunitconfig, but just run suites that match this glob

> 2) `kunit.py run --alltests 'my-suite-*'`

> I.e. use allyesconfig so I don't have to worry about writing a

> kunitconfig at all.

>

> See the first commit message for more details and discussion about

> future work.

>

> This patch series also includes a bugfix for a latent bug that can't be

> triggered right now but has worse consequences as a result of the

> changes needed to plumb in this suite name glob.

>

> [1] https://lore.kernel.org/linux-kselftest/20210201205514.3943096-1-dlatypov@google.com/


I tried applying this series on top of several of your other patches
over the past cycle and running the kunit_tool unit tests
(kunit_tool_test.py) and I got several failures:

======================================================================
FAIL: test_exec_builddir (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tools/testing/kunit/kunit_tool_test.py", line 417, in test_exec_builddir
    self.linux_source_mock.run_kernel.assert_called_once_with(build_dir=build_dir,
timeout=300)
  File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=300)
Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit')

======================================================================
FAIL: test_exec_passes_args_pass (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tools/testing/kunit/kunit_tool_test.py", line 337, in
test_exec_passes_args_pass
    self.linux_source_mock.run_kernel.assert_called_once_with(build_dir='.kunit',
timeout=300)
  File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=300)
Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit')

======================================================================
FAIL: test_exec_timeout (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tools/testing/kunit/kunit_tool_test.py", line 385, in test_exec_timeout
    self.linux_source_mock.run_kernel.assert_called_once_with(build_dir='.kunit',
timeout=timeout)
  File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=3453)
Actual: run_kernel(timeout=3453, filter_glob='', build_dir='.kunit')

======================================================================
FAIL: test_run_builddir (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tools/testing/kunit/kunit_tool_test.py", line 400, in test_run_builddir
    self.linux_source_mock.run_kernel.assert_called_once_with(
  File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=300)
Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit')

======================================================================
FAIL: test_run_passes_args_pass (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tools/testing/kunit/kunit_tool_test.py", line 344, in
test_run_passes_args_pass
    self.linux_source_mock.run_kernel.assert_called_once_with(
  File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=300)
Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit')

======================================================================
FAIL: test_run_timeout (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tools/testing/kunit/kunit_tool_test.py", line 392, in test_run_timeout
    self.linux_source_mock.run_kernel.assert_called_once_with(
  File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=3453)
Actual: run_kernel(timeout=3453, filter_glob='', build_dir='.kunit')

----------------------------------------------------------------------
Ran 37 tests in 0.082s

FAILED (failures=6)