[RFC,v2,0/2] Add test to verify probe of devices from discoverable busses

Hi,

v1 [1] was discussed during Plumbers [2], where a lot of feedback was given. I
hope to justify the changes in v2 and address the feedback here.

One feedback from Shuah was that keeping per-platform files with the USB/PCI
devices to test as part of the kselftest tree wasn't maintainable. One proposed
alternative was to generate a list of probed devices on a known-good kernel and
use that as a reference. However you need someone to look at that generated
reference to be able to say it is a good one, and you need to save it to ensure
it will be reproducible later anyway, so that wouldn't actually solve the
problem. It is a matter of hand-crafting vs generating the test definitions, but
they will need to be vouched by someone and stored somewhere in both cases.

So for this v2, in patch 2 I just have a sample test definition, and the
per-platform test definitions would be added to a separate repository.

The other feedback received was that the BIOS might reconfigure the PCI
topology (at least on x86), meaning that relying on a sequence of device and
function numbers (eg 1d.0/02.0/0.0) as a stable description of a device on the
platform is not possible. I couldn't verify whether this is really the case (if
you have any more insight into this, please let me know), but with that in mind,
here in v2 I have taken a different approach. Here I'm using the device's
properties which are used for driver matching (the same that show on modalias)
to identify a device in a stable way.

This approach has some drawbacks compared to the one on v1. For one it doesn't
uniquely identify a device, so if there are multiple of the same device on a
platform they have to be checked as a group. Also the test definition isn't as
human-readable.

I'm adding in CC the people I recognized at the Plumbers session that were
interested in this work. Feel free to add anyone missing.

Thanks,
Nícolas

[1] https://lore.kernel.org/all/20231024211818.365844-1-nfraprado@collabora.com
[2] https://www.youtube.com/watch?v=oE73eVSyFXQ&t=9377s

Original cover letter:

This is part of an effort to improve detection of regressions impacting
device probe on all platforms. The recently merged DT kselftest [3]
detects probe issues for all devices described statically in the DT.
That leaves out devices discovered at run-time from discoverable busses.

This is where this test comes in. All of the devices that are connected
through discoverable busses (ie USB and PCI), and which are internal and
therefore always present, can be described in a per-platform file so
they can be checked for. The test will check that the device has been
instantiated and bound to a driver.

Patch 1 introduces the test. Patch 2 adds the test definitions for the
google,spherion machine (Acer Chromebook 514) as an example.

This is the sample output from the test running on Spherion:

TAP version 13
 Using board file:  boards/google,spherion
1..3
ok 1 usb.camera
ok 2 usb.bluetooth
ok 3 pci.wifi
 Totals: pass:3 fail:0 xfail:0 xpass:0 skip:0 error:0

[3] https://lore.kernel.org/all/20230828211424.2964562-1-nfraprado@collabora.com/

Changes in v2:
- Changed approach of encoding stable device reference in test file from
  HW topology to device match fields (the ones from modalias)
- Better documented test format

Nícolas F. R. A. Prado (2):
  kselftest: Add test to verify probe of devices from discoverable
    busses
  kselftest: devices: Add sample board file for google,spherion

 tools/testing/selftests/Makefile              |   1 +
 tools/testing/selftests/devices/.gitignore    |   1 +
 tools/testing/selftests/devices/Makefile      |   8 +
 .../selftests/devices/boards/google,spherion  |  12 ++
 .../devices/test_discoverable_devices.sh      | 160 ++++++++++++++++++
 5 files changed, 182 insertions(+)
 create mode 100644 tools/testing/selftests/devices/.gitignore
 create mode 100644 tools/testing/selftests/devices/Makefile
 create mode 100644 tools/testing/selftests/devices/boards/google,spherion
 create mode 100755 tools/testing/selftests/devices/test_discoverable_devices.sh