diff mbox series

[net-next,RFC] selftests/net: integrate packetdrill with ksft

Message ID 20240827193417.2792223-1-willemdebruijn.kernel@gmail.com
State New
Headers show
Series [net-next,RFC] selftests/net: integrate packetdrill with ksft | expand

Commit Message

Willem de Bruijn Aug. 27, 2024, 7:32 p.m. UTC
From: Willem de Bruijn <willemb@google.com>

Lay the groundwork to import into kselftests the over 150 packetdrill
TCP/IP conformance tests on github.com/google/packetdrill.

Florian recently added support for packetdrill tests in nf_conntrack,
in commit a8a388c2aae49 ("selftests: netfilter: add packetdrill based
conntrack tests").

This patch takes a slightly different implementation and reuses the
ksft python library for its KTAP, ksft, NetNS and other such tooling.

It also anticipates the large number of testcases, by creating a
separate kselftest for each feature (directory). It does this by
copying the template script packetdrill_ksft.py for each directory,
and putting those in TEST_CUSTOM_PROGS so that kselftests runs each.

To demonstrate the code with minimal patch size, initially import only
two features/directories from github. One with a single script, and
one with two. This was the only reason to pick tcp/inq and tcp/md5.

Any future imports of packetdrill tests should require no additional
coding. Just add the tcp/$FEATURE directory with *.pkt files.

Implementation notes:
- restore alphabetical order when adding the new directory to
  tools/testing/selftests/Makefile
- copied *.pkt files and support verbatim from the github project,
  except for
    - update common/defaults.sh path (there are two paths on github)
    - add SPDX headers
    - remove one author statement
    - Acknowledgment: drop an e (checkpatch)

Tested:
	make -C tools/testing/selftests/ \
	  TARGETS=net/packetdrill \
	  install INSTALL_PATH=$KSFT_INSTALL_PATH

	# in virtme-ng
	sudo ./run_kselftest.sh -c net/packetdrill
	sudo ./run_kselftest.sh -t net/packetdrill:tcp_inq.py

Result:
	kselftest: Running tests in net/packetdrill
	TAP version 13
	1..2
	# timeout set to 45
	# selftests: net/packetdrill: tcp_inq.py
	# KTAP version 1
	# 1..4
	# ok 1 tcp_inq.client-v4
	# ok 2 tcp_inq.client-v6
	# ok 3 tcp_inq.server-v4
	# ok 4 tcp_inq.server-v6
	# # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0
	ok 1 selftests: net/packetdrill: tcp_inq.py
	# timeout set to 45
	# selftests: net/packetdrill: tcp_md5.py
	# KTAP version 1
	# 1..2
	# ok 1 tcp_md5.md5-only-on-client-ack-v4
	# ok 2 tcp_md5.md5-only-on-client-ack-v6
	# # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0
	ok 2 selftests: net/packetdrill: tcp_md5.py

Signed-off-by: Willem de Bruijn <willemb@google.com>

---

RFC points for discussion

ksft: the choice for this python framework introduces a dependency on
the YNL scripts, and some non-obvious code:
- to include the net/lib dep in tools/testing/selftests/Makefile
- a boilerplate lib/py/__init__.py that each user of ksft will need
It seems preferable to me to use ksft.py over reinventing the wheel,
e.g., to print KTAP output. But perhaps we can make it more obvious
for future ksft users, and make the dependency on YNL optional.

kselftest-per-directory: copying packetdrill_ksft.py to create a
separate script per dir is a bit of a hack. A single script is much
simpler, optionally with nested KTAP (not supported yet by ksft). But,
I'm afraid that running time without intermediate output will be very
long when we integrate all packetdrill scripts.

nf_conntrack: we can dedup the common.sh.

*pkt files: which of the 150+ scripts on github are candidates for
kselftests, all or a subset? To avoid change detector tests. And what
is the best way to eventually send up to 150 files, 7K LoC.
---
 tools/testing/selftests/Makefile              |  7 +-
 .../selftests/net/packetdrill/.gitignore      |  1 +
 .../selftests/net/packetdrill/Makefile        | 28 ++++++
 .../net/packetdrill/lib/py/__init__.py        | 15 ++++
 .../net/packetdrill/packetdrill_ksft.py       | 90 +++++++++++++++++++
 .../net/packetdrill/tcp/common/defaults.sh    | 63 +++++++++++++
 .../net/packetdrill/tcp/common/set_sysctls.py | 38 ++++++++
 .../net/packetdrill/tcp/inq/client.pkt        | 51 +++++++++++
 .../net/packetdrill/tcp/inq/server.pkt        | 51 +++++++++++
 .../tcp/md5/md5-only-on-client-ack.pkt        | 28 ++++++
 10 files changed, 369 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/net/packetdrill/.gitignore
 create mode 100644 tools/testing/selftests/net/packetdrill/Makefile
 create mode 100644 tools/testing/selftests/net/packetdrill/lib/py/__init__.py
 create mode 100755 tools/testing/selftests/net/packetdrill/packetdrill_ksft.py
 create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh
 create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py
 create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/client.pkt
 create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/server.pkt
 create mode 100644 tools/testing/selftests/net/packetdrill/tcp/md5/md5-only-on-client-ack.pkt

Comments

Jakub Kicinski Aug. 28, 2024, 6:23 p.m. UTC | #1
On Wed, 28 Aug 2024 10:20:47 -0700 Stanislav Fomichev wrote:
> > As Jakub responded.
> > 
> > Indeed importing the whole application into the kernel sources is what
> > stopped me from attempting this before. Florian's nf_conntrack charted
> > the path here.  
> 
> Thanks both. I was expecting to find some build rules :-) But as long
> as we have this story sorted out, we're good.

FWIW I lied to you, the "script" for buildings things wasn't committed
in the repo, I pushed it now:
https://github.com/linux-netdev/nipa/commit/dde0f093774a4a4c85e595b5cf44bd3e4bd434d1
Stanislav Fomichev Aug. 28, 2024, 6:36 p.m. UTC | #2
On 08/28, Jakub Kicinski wrote:
> On Wed, 28 Aug 2024 10:20:47 -0700 Stanislav Fomichev wrote:
> > > As Jakub responded.
> > > 
> > > Indeed importing the whole application into the kernel sources is what
> > > stopped me from attempting this before. Florian's nf_conntrack charted
> > > the path here.  
> > 
> > Thanks both. I was expecting to find some build rules :-) But as long
> > as we have this story sorted out, we're good.
> 
> FWIW I lied to you, the "script" for buildings things wasn't committed
> in the repo, I pushed it now:
> https://github.com/linux-netdev/nipa/commit/dde0f093774a4a4c85e595b5cf44bd3e4bd434d1

Perfect, thanks!
Jakub Kicinski Aug. 28, 2024, 6:39 p.m. UTC | #3
On Wed, 28 Aug 2024 11:26:28 -0700 Mina Almasry wrote:
> I did not know about run_kselftest.sh. From a quick look, it seems to
> require selftests to be installed to run them. I think nipa relies on
> something slightly different, it does something like this, AFAIU this
> runs an individual kselftest without requiring it to be installed
> first (which is nice):
> 
> make -C tools/testing/selftests TARGETS="net"
> TEST_PROGS="rxtimestamp.sh" TEST_GEN_PROGS="" run_tests
> 
> It may be worth it to check if these added tests can be ran in a similar way.

They should, the ksft infra converges into the same bash script
(runner.sh) whether we enter via make or via the "installed" runner.

> This may also solve your issue with 'intermediate output' being
> printed after every test. AFAIU if the tests can be ran individually
> using a command like this, then you don't need intermediate output,
> maybe.

selftest infra tries to prefix raw output with # to make sure that
random prints don't get mixed with KTAP. IIRC there are two ways
to do that, one is a perl script and that doesn't buffer, but if
there's no perl in PATH it runs the output thru sed, and sed buffers.
Jakub Kicinski Aug. 28, 2024, 9 p.m. UTC | #4
On Wed, 28 Aug 2024 15:33:33 -0400 Willem de Bruijn wrote:
> That could work.
> 
> Is reporting one KTAP and exitcode per directory vs per packetdrill
> invocation good/bad/neither?

To me "neither", hopefully and unhelpfully I won't tell you anything
that's not common sense :) The balance is between:

 - having test cases which don't take too long (assuming debug kernel):
     <15min is good, 
     >1h I will start complaining,
     >1h30m is bad because we can't retry and still make the 3h
            deadline that NIPA has,
     >3h the test can't run in NIPA at all.

vs

 - flip side is having so many cases it's hard to keep track and render
   in the UI. JSON is relatively slow to process. If you have 150 cases,
   that's 300 per branch (debug and non-debug kernels), times 8
   branches a day => 2.4k results / day
   I think that's still fine-ish, but on the larger side for sure. For
   reference net and forwarding have ~100 tests each. FWIW we do have
   the ability to collect and display nested KTAP so the information is
   not lost (but it makes queries slower so we don't fetch it by default).

> Three other issues if this is calling packetdrill directly is
> - passing the non-trivial IP specific flags
> - running twice, for IPv4 and IPv6
> - chdir into the directory of the pkt file
> 
> That can be addressed by instead calling a small wrapper shell script.
> 
> That would do the test_func_builder part of packetdrill_ksft.py.
> But without the need to handle netns, popen/cmd, etc, and thus the
> ksft dependencies.

Right!
Jakub Kicinski Aug. 30, 2024, 5:33 p.m. UTC | #5
On Fri, 30 Aug 2024 11:20:05 -0400 Willem de Bruijn wrote:
> Kselftest install does not preserve directories.
> 
> So all .pkt files are copied into net/packetdrill root. This is messy.
> More fundamentally it breaks the includes in the files (e..g, `source
> ../common/defaults.sh`).

Can you show an example of exact commands and what happens?

We have directories in net/lib, and it's a target, and it works, no?
Willem de Bruijn Aug. 30, 2024, 9:46 p.m. UTC | #6
Willem de Bruijn wrote:
> Jakub Kicinski wrote:
> > On Fri, 30 Aug 2024 11:20:05 -0400 Willem de Bruijn wrote:
> > > Kselftest install does not preserve directories.
> > > 
> > > So all .pkt files are copied into net/packetdrill root. This is messy.
> > > More fundamentally it breaks the includes in the files (e..g, `source
> > > ../common/defaults.sh`).
> > 
> > Can you show an example of exact commands and what happens?
> 
> Running directly works fine:
> 
>     $ KSELFTEST_PKT_INTERP=packetdrill_ksft.sh
>     $ make -C tools/testing/selftests \
>             TARGETS=net/packetdrill O=/tmp run_tests
> 
>     TAP version 13
>     1..3
>     # timeout set to 45
>     # selftests: net/packetdrill: client.pkt
>     # TAP version 13
>     # 1..2
>     # ok 1 ipv4
>     # ok 2 ipv6
>     # # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0
>     ok 1 selftests: net/packetdrill: client.pkt
>     [..etc..]
> 
> Installing does not:
> 
>     $ make -C tools/testing/selftests/ \
>           TARGETS=net/packetdrill \
>           install INSTALL_PATH=$INSTALL_DIR
>     $ cd $INSTALL_DIR
>     $ export KSELFTEST_PKT_INTERP=packetdrill_ksft.sh
>     $ ./run_kselftest.sh -c net/packetdrill
> 
>     TAP version 13
>     1..3
>     # timeout set to 45
>     # selftests: net/packetdrill: client.pkt
>     # TAP version 13
>     # 1..2
>     # sh: line 1: ../common/defaults.sh: No such file or directory
>     # ./client.pkt: error executing init command: non-zero status 127
>     # not ok 1 ipv4
>     # sh: line 1: ../common/defaults.sh: No such file or directory
>     # ./client.pkt: error executing init command: non-zero status 127
>     # not ok 2 ipv6
>     # # Totals: pass:0 fail:2 xfail:0 xpass:0 skip:0 error:0
>     not ok 1 selftests: net/packetdrill: client.pkt # exit=1
> 
> Due to that relative path to defaults.sh inside the scripts.
> 
> It is arguably a bit weird that the relative path of the TEST_PROGS
> differs before and after install.

I guess what's weird is more what I'm trying to do. There has been
no need so far for progs in subdirectories from the TARGET.

One option is to just create a bunch of targets with a wildcard.
Something like below (entirely untested).

TARGETS += net/packetdrill/tcp/$(wildcard *)

> > We have directories in net/lib, and it's a target, and it works, no?
> 
> net/lib is not a TARGET in tools/testing/selftests/Makefile. Its
> Makefile only generates dependencies for other targets: TEST_FILES,
> TEST_GEN_FILES and TEST_INCLUDES.
> 
> This issue with preserving paths until recently also existed for
> helper files (TEST_FILES). TEST_INCLUDES was added expressly to
> preserve those paths (commit 2a0683be5b4c).

Another option is to dive yet deeper into kselftest infra and do the
same for TEST_PROGS, either TEST_PROGS itself or as a new
TEST_PROGS_FULLPATH or so.

It needs a change to INSTALL_RULE (maybe just passing -R to rsync),
to emit_tests to skip basename when writing kselftest-list.txt, and
probably to run_one() to break up the path and chdir. I'll see if I
can make this work without too much churn.
Willem de Bruijn Sept. 1, 2024, 9:15 p.m. UTC | #7
Willem de Bruijn wrote:
> Jakub Kicinski wrote:
> > On Fri, 30 Aug 2024 14:47:43 -0400 Willem de Bruijn wrote:
> > > > We have directories in net/lib, and it's a target, and it works, no?  
> > > 
> > > net/lib is not a TARGET in tools/testing/selftests/Makefile. Its
> > > Makefile only generates dependencies for other targets: TEST_FILES,
> > > TEST_GEN_FILES and TEST_INCLUDES.
> > 
> > Oh right, TEST_FILES vs TEST_INCLUDES :(
> > 
> > Looks like only x86 does some weird stuff and prepends $(OUTPUT) to all
> > test names. Otherwise the only TEST_NAME with a / in it is
> > 
> > x86_64/nx_huge_pages_test.sh
> 
> Oh interesting precedent. Let me take a look.
> 
> > But then again maybe we should give up on the idea of using directories?
> > Use some separator like --, I mean:
> > 
> > mv packetdrill/tcp/inq/client.pkt packetdrill/tcp--inq--client.pkt
> > 
> > Assuming we're moving forward with the interpreter idea we don't need
> > directories for multi-threading, just for organization. Which perhaps
> > isn't worth the time investment? Given that we'd mostly interact with
> > these tests via UI which will flatten it all back?
> 
> That's definitely simpler :)
> 
> I'd like to keep diffs between packetdrill scripts on github (and
> Google internal, we have more) and selftests to a minimum. This is
> invertible, as is rewriting source statements inside the pkt files.
> But that might be more work and more maintenance in the end.

Thanks again for the pointer and suggestion.

Changing kselftests to preserve directories turns out to be trivial.
Patch inline below.

But, existing TARGETS of course then start failing. Because they
depend on existing rsync without -R. In (at least) two ways:

amd-pstate fails because its TEST_FILES has includes from other
directories and it expects those files to land in the directory
with tests.

x86 prefixes all its output with $(OUTPUT) to form absolute paths,
which also creates absolute paths in kselftest-list.txt.

These two are examples, not necessarily the one instances of those
patterns. So switching to preserving directories for existing targets
like TEST_FILES seems intractable.

Plan B is to add a new TEST_PROGS_RECURSE, analogous to how
TEST_INCLUDES extended TEST_FILES with optional path preservation.
That is not much more complex.

---

+++ b/tools/testing/selftests/kselftest/runner.sh
@@ -101,7 +112,7 @@ run_one()
                echo "# timeout set to $kselftest_timeout" >> "$logfile"
        fi
 
-       TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST"
+       TEST_HDR_MSG="selftests: $DIR: $TEST"

+++ b/tools/testing/selftests/lib.mk
@@ -150,7 +150,7 @@ clean_mods_dir:

 define INSTALL_SINGLE_RULE
        $(if $(INSTALL_LIST),@mkdir -p $(INSTALL_PATH))
-       $(if $(INSTALL_LIST),rsync -a --copy-unsafe-links $(INSTALL_LIST) $(INSTALL_PATH)/)
+       $(if $(INSTALL_LIST),rsync -aR --copy-unsafe-links $(INSTALL_LIST) $(INSTALL_PATH)/)
 endef

 define INSTALL_MODS_RULE
@@ -180,8 +180,7 @@ endif
 
 emit_tests:
        for TEST in $(TEST_GEN_PROGS) $(TEST_CUSTOM_PROGS) $(TEST_PROGS); do \
-               BASENAME_TEST=`basename $$TEST`;        \
-               echo "$(COLLECTION):$$BASENAME_TEST";   \
+               echo "$(COLLECTION):$$TEST";    \
        done
Jakub Kicinski Sept. 2, 2024, 4:46 p.m. UTC | #8
On Sun, 01 Sep 2024 17:15:38 -0400 Willem de Bruijn wrote:
> Changing kselftests to preserve directories turns out to be trivial.
> Patch inline below.
> 
> But, existing TARGETS of course then start failing. Because they
> depend on existing rsync without -R. In (at least) two ways:
> 
> amd-pstate fails because its TEST_FILES has includes from other
> directories and it expects those files to land in the directory
> with tests.
> 
> x86 prefixes all its output with $(OUTPUT) to form absolute paths,
> which also creates absolute paths in kselftest-list.txt.
> 
> These two are examples, not necessarily the one instances of those
> patterns. So switching to preserving directories for existing targets
> like TEST_FILES seems intractable.

I wonder how many of the targets actually need this behavior (it's
intentionally useful to them) vs they grew to depend on it accidentally.

> Plan B is to add a new TEST_PROGS_RECURSE, analogous to how
> TEST_INCLUDES extended TEST_FILES with optional path preservation.
> That is not much more complex.

Alternative would be to allow opt-in (diff at the end), I'm personally
biased against yet another TEST_* group because it's hard enough already
to make sense of the existing ones. Maybe it's better for the test to
"declare":

PRESERVE_TEST_DIRS


---
 tools/testing/selftests/lib.mk | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk
index d6edcfcb5be8..7be9bd583642 100644
--- a/tools/testing/selftests/lib.mk
+++ b/tools/testing/selftests/lib.mk
@@ -51,6 +51,12 @@ else
 CC := $(CROSS_COMPILE)gcc
 endif # LLVM
 
+ifeq ($(PRESERVE_TEST_DIRS),)
+RSYNC_INSTALL_FLAGS=-a --copy-unsafe-links
+else
+RSYNC_INSTALL_FLAGS=-aR --copy-unsafe-links
+endif
+
 ifeq (0,$(MAKELEVEL))
     ifeq ($(OUTPUT),)
 	OUTPUT := $(shell pwd)
@@ -150,12 +156,12 @@ endif
 
 define INSTALL_SINGLE_RULE
 	$(if $(INSTALL_LIST),@mkdir -p $(INSTALL_PATH))
-	$(if $(INSTALL_LIST),rsync -a --copy-unsafe-links $(INSTALL_LIST) $(INSTALL_PATH)/)
+	$(if $(INSTALL_LIST),rsync $(RSYNC_INSTALL_FLAGS) $(INSTALL_LIST) $(INSTALL_PATH)/)
 endef
 
 define INSTALL_MODS_RULE
 	$(if $(INSTALL_LIST),@mkdir -p $(INSTALL_PATH)/$(INSTALL_LIST))
-	$(if $(INSTALL_LIST),rsync -a --copy-unsafe-links $(INSTALL_LIST)/*.ko $(INSTALL_PATH)/$(INSTALL_LIST))
+	$(if $(INSTALL_LIST),rsync $(RSYNC_INSTALL_FLAGS) $(INSTALL_LIST)/*.ko $(INSTALL_PATH)/$(INSTALL_LIST))
 endef
 
 define INSTALL_RULE
@@ -179,10 +185,17 @@ else
 endif
 
 emit_tests:
+ifeq ($(PRESERVE_TEST_DIRS),)
 	for TEST in $(TEST_GEN_PROGS) $(TEST_CUSTOM_PROGS) $(TEST_PROGS); do \
 		BASENAME_TEST=`basename $$TEST`;	\
 		echo "$(COLLECTION):$$BASENAME_TEST";	\
 	done
+else
+	for TEST in $(TEST_GEN_PROGS) $(TEST_CUSTOM_PROGS) $(TEST_PROGS); do \
+		BASENAME_TEST=$$TEST;	\
+		echo "$(COLLECTION):$$TEST";	\
+	done
+endif
 
 # define if isn't already. It is undefined in make O= case.
 ifeq ($(RM),)
Jakub Kicinski Sept. 2, 2024, 4:56 p.m. UTC | #9
On Mon, 2 Sep 2024 09:46:12 -0700 Jakub Kicinski wrote:
> PRESERVE_TEST_DIRS

This was meant to be:

PRESERVE_TEST_DIRS := 1

Testing this more, looks like rsync -aR breaks networking tests, too.
The net/lib target, specifically, is no longer able to copy out 
the files outside of tools/testing/selftests (the YAML specs which
live in Documentation/).

So unless we can pass some magic flag to rsync to skip leading ../
we'll be stuck in supporting both behaviors forever. In which case
maybe TEST_PROGS_RECURSE is indeed better. I was hoping the
PRESERVE_TEST_DIRS flag can spread and once it reaches all targets
we can make delete the old behavior. If it can't happen its no better
than TEST_PROGS_RECURSE... sorry for the noise.
Willem de Bruijn Sept. 2, 2024, 8:50 p.m. UTC | #10
Jakub Kicinski wrote:
> On Mon, 2 Sep 2024 09:46:12 -0700 Jakub Kicinski wrote:
> > PRESERVE_TEST_DIRS
> 
> This was meant to be:
> 
> PRESERVE_TEST_DIRS := 1

Oh I like this.

> Testing this more, looks like rsync -aR breaks networking tests, too.
> The net/lib target, specifically, is no longer able to copy out 
> the files outside of tools/testing/selftests (the YAML specs which
> live in Documentation/).
> 
> So unless we can pass some magic flag to rsync to skip leading ../
> we'll be stuck in supporting both behaviors forever. In which case
> maybe TEST_PROGS_RECURSE is indeed better. I was hoping the
> PRESERVE_TEST_DIRS flag can spread and once it reaches all targets
> we can make delete the old behavior. If it can't happen its no better
> than TEST_PROGS_RECURSE... sorry for the noise.

Oh right :/ 

A lot of TEST(_GEN|_CUSTOM)_PROGS(_EXTENDED) could probably be cleaned
up.

One part that I haven't looked at closely yet is x86's Makefile
prefixing everything with $(OUTPUT). That seems something that should
be consistent across target Makefiles if it exists at all, but
clearly isn't. Maybe it was when it was introduced in commit
a8ba798bc8ec.

So, we could opt out only TEST_FILES. But that still leaves a
branch there indefinitely and makes the option harder to understand.
Sounds like TEST_PROGS_RECURSE is simpler then, indeed.


Separate from the above, runner.sh also needs a few changes. This
part is mostly cosmetic, but there will be duplicate basenames,
for common terms such as client.pkt
 
-       TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST"
+       TEST_HDR_MSG="selftests: $DIR: $TEST"

run_in_netns and run_many need similar fixes to avoid name collisions.


Btw, for KSELFTEST_PKT_INTERP it probably makes sense to only try this
if the script is non-executable, similar to the existing opt-out that
looks at the shebang.
Willem de Bruijn Sept. 5, 2024, 3:27 a.m. UTC | #11
Jakub Kicinski wrote:
> On Fri, 30 Aug 2024 14:47:43 -0400 Willem de Bruijn wrote:
> > > We have directories in net/lib, and it's a target, and it works, no?  
> > 
> > net/lib is not a TARGET in tools/testing/selftests/Makefile. Its
> > Makefile only generates dependencies for other targets: TEST_FILES,
> > TEST_GEN_FILES and TEST_INCLUDES.
> 
> Oh right, TEST_FILES vs TEST_INCLUDES :(
> 
> Looks like only x86 does some weird stuff and prepends $(OUTPUT) to all
> test names. Otherwise the only TEST_NAME with a / in it is
> 
> x86_64/nx_huge_pages_test.sh
> 
> But then again maybe we should give up on the idea of using directories?
> Use some separator like --, I mean:
> 
> mv packetdrill/tcp/inq/client.pkt packetdrill/tcp--inq--client.pkt
> 
> Assuming we're moving forward with the interpreter idea we don't need
> directories for multi-threading, just for organization. Which perhaps
> isn't worth the time investment? Given that we'd mostly interact with
> these tests via UI which will flatten it all back?

Okay, went with the simplest approach for v1: indeed just flattening
the space like this.

Only after implementing TEST_PROGS_RECURSE and PRESERVE_TEST_DIRS.
Both make kselftest more complex and in subtle ways that could break
existing tests.

Also simplified the interpreter mechanism. Instead of requiring an
environment variable, look for a specific executable in the TARGET
dir. This will make the test just work when someone does a manual run.

All is tentative and up for revision, of course.
diff mbox series

Patch

diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index a5f1c0c27dff9..f03d6fee7ac54 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -65,10 +65,11 @@  TARGETS += net/af_unix
 TARGETS += net/forwarding
 TARGETS += net/hsr
 TARGETS += net/mptcp
-TARGETS += net/openvswitch
-TARGETS += net/tcp_ao
 TARGETS += net/netfilter
+TARGETS += net/openvswitch
+TARGETS += net/packetdrill
 TARGETS += net/rds
+TARGETS += net/tcp_ao
 TARGETS += nsfs
 TARGETS += perf_events
 TARGETS += pidfd
@@ -122,7 +123,7 @@  TARGETS_HOTPLUG = cpu-hotplug
 TARGETS_HOTPLUG += memory-hotplug
 
 # Networking tests want the net/lib target, include it automatically
-ifneq ($(filter net drivers/net drivers/net/hw,$(TARGETS)),)
+ifneq ($(filter net net/packetdrill drivers/net drivers/net/hw,$(TARGETS)),)
 ifeq ($(filter net/lib,$(TARGETS)),)
 	INSTALL_DEP_TARGETS := net/lib
 endif
diff --git a/tools/testing/selftests/net/packetdrill/.gitignore b/tools/testing/selftests/net/packetdrill/.gitignore
new file mode 100644
index 0000000000000..a40f1a600eb94
--- /dev/null
+++ b/tools/testing/selftests/net/packetdrill/.gitignore
@@ -0,0 +1 @@ 
+tcp*sh
diff --git a/tools/testing/selftests/net/packetdrill/Makefile b/tools/testing/selftests/net/packetdrill/Makefile
new file mode 100644
index 0000000000000..d94c51098d1f0
--- /dev/null
+++ b/tools/testing/selftests/net/packetdrill/Makefile
@@ -0,0 +1,28 @@ 
+# SPDX-License-Identifier: GPL-2.0
+
+# KSFT includes
+TEST_INCLUDES := $(wildcard lib/py/*.py ../lib/py/*.py)
+
+# Packetdrill support file(s)
+TEST_INCLUDES += tcp/common/defaults.sh
+TEST_INCLUDES += tcp/common/set_sysctls.py
+
+# Packetdrill scripts: all .pkt in subdirectories
+TEST_INCLUDES += $(wildcard tcp/**/*.pkt)
+
+# Create a separate ksft test for each subdirectory
+# Running all packetdrill tests in one go will take too long
+#
+# For each tcp/$subdir, create a test script tcp_$subdir.py
+# Exclude tcp/common, which is a helper directory
+TEST_DIRS := $(wildcard tcp/*)
+TEST_DIRS := $(filter-out tcp/common, $(TEST_DIRS))
+TEST_CUSTOM_PROGS := $(foreach dir,$(TEST_DIRS),$(subst /,_,$(dir)).py)
+
+$(TEST_CUSTOM_PROGS) : packetdrill_ksft.py
+	cp $< $@
+
+# Needed to generate all TEST_CUSTOM_PROGS
+all: $(TEST_CUSTOM_PROGS)
+
+include ../../lib.mk
diff --git a/tools/testing/selftests/net/packetdrill/lib/py/__init__.py b/tools/testing/selftests/net/packetdrill/lib/py/__init__.py
new file mode 100644
index 0000000000000..51bb6dda43d65
--- /dev/null
+++ b/tools/testing/selftests/net/packetdrill/lib/py/__init__.py
@@ -0,0 +1,15 @@ 
+# SPDX-License-Identifier: GPL-2.0
+
+import pathlib
+import sys
+
+KSFT_DIR = (pathlib.Path(__file__).parent / "../../../..").resolve()
+
+try:
+    sys.path.append(KSFT_DIR.as_posix())
+    from net.lib.py import *
+except ModuleNotFoundError as e:
+    ksft_pr("Failed importing `net` library from kernel sources")
+    ksft_pr(str(e))
+    ktap_result(True, comment="SKIP")
+    sys.exit(4)
diff --git a/tools/testing/selftests/net/packetdrill/packetdrill_ksft.py b/tools/testing/selftests/net/packetdrill/packetdrill_ksft.py
new file mode 100755
index 0000000000000..62572a5b8331c
--- /dev/null
+++ b/tools/testing/selftests/net/packetdrill/packetdrill_ksft.py
@@ -0,0 +1,90 @@ 
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""Run packetdrill tests in the ksft harness.
+
+   Run all packetdrill tests in a subdirectory.
+   Detect the relevant subdirectory from this script name.
+   (Because the script cannot be given arguments.)
+
+   Run each test, for both IPv4 and IPv6.
+   Return a separate ksft result for each test case.
+"""
+
+import glob
+import os
+import pathlib
+import shutil
+
+from lib.py import cmd, ksft_exit, ksft_run, KsftSkipEx, NetNS
+
+
+def test_func_builder(pktfile_path, ipv4):
+    """Create a function that can be passed to ksft_run."""
+
+    def f():
+        if ipv4:
+            args = ("--ip_version=ipv4 "
+                    "--local_ip=192.168.0.1 "
+                    "--gateway_ip=192.168.0.1 "
+                    "--netmask_ip=255.255.0.0 "
+                    "--remote_ip=192.0.2.1 "
+                    "-D CMSG_LEVEL_IP=SOL_IP "
+                    "-D CMSG_TYPE_RECVERR=IP_RECVERR "
+                    )
+        else:
+            args = ("--ip_version=ipv6 --mtu=1520 "
+                    "--local_ip=fd3d:0a0b:17d6::1 "
+                    "--gateway_ip=fd3d:0a0b:17d6:8888::1 "
+                    "--remote_ip=fd3d:fa7b:d17d::1 "
+                    "-D CMSG_LEVEL_IP=SOL_IPV6 "
+                    "-D CMSG_TYPE_RECVERR=IPV6_RECVERR"
+                    )
+
+        if not shutil.which("packetdrill"):
+            raise KsftSkipEx("Cannot find packetdrill")
+
+        netns = NetNS()
+
+        # Call packetdrill from the directory hosting the .pkt script,
+        # because scripts can have relative includes.
+        savedir = os.getcwd()
+        os.chdir(os.path.dirname(pktfile_path))
+        basename = os.path.basename(pktfile_path)
+        cmd(f"packetdrill {args} {basename}", ns=netns)
+        os.chdir(savedir)
+
+    if ipv4:
+        f.__name__ = pathlib.Path(pktfile_path).stem + "-v4"
+    else:
+        f.__name__ = pathlib.Path(pktfile_path).stem + "-v6"
+
+    return f
+
+
+def scriptname_to_testdir(filepath):
+    """Extract the directory to run from this filename."""
+
+    suffix = ".sh"
+
+    subdir = os.path.basename(filepath)
+    subdir = subdir[:-len(suffix)]
+    subdir = subdir.replace("_", "/")
+    return subdir
+
+
+def main() -> None:
+    subdir = scriptname_to_testdir(__file__)
+    files = glob.glob(f"{subdir}/**/*.pkt", recursive=True)
+
+    cases = []
+    for file in files:
+        for ipv4 in [True, False]:
+            cases.append(test_func_builder(file, ipv4=ipv4))
+
+    ksft_run(cases=cases)
+    ksft_exit()
+
+
+if __name__ == "__main__":
+    main()
diff --git a/tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh b/tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh
new file mode 100755
index 0000000000000..1095a7b22f44d
--- /dev/null
+++ b/tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh
@@ -0,0 +1,63 @@ 
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# Set standard production config values that relate to TCP behavior.
+
+# Flush old cached data (fastopen cookies).
+ip tcp_metrics flush all > /dev/null 2>&1
+
+# TCP min, default, and max receive and send buffer sizes.
+sysctl -q net.ipv4.tcp_rmem="4096 540000 $((15*1024*1024))"
+sysctl -q net.ipv4.tcp_wmem="4096 $((256*1024)) 4194304"
+
+# TCP timestamps.
+sysctl -q net.ipv4.tcp_timestamps=1
+
+# TCP SYN(ACK) retry thresholds
+sysctl -q net.ipv4.tcp_syn_retries=5
+sysctl -q net.ipv4.tcp_synack_retries=5
+
+# TCP Forward RTO-Recovery, RFC 5682.
+sysctl -q net.ipv4.tcp_frto=2
+
+# TCP Selective Acknowledgements (SACK)
+sysctl -q net.ipv4.tcp_sack=1
+
+# TCP Duplicate Selective Acknowledgements (DSACK)
+sysctl -q net.ipv4.tcp_dsack=1
+
+# TCP FACK (Forward Acknowldgement)
+sysctl -q net.ipv4.tcp_fack=0
+
+# TCP reordering degree ("dupthresh" threshold for entering Fast Recovery).
+sysctl -q net.ipv4.tcp_reordering=3
+
+# TCP congestion control.
+sysctl -q net.ipv4.tcp_congestion_control=cubic
+
+# TCP slow start after idle.
+sysctl -q net.ipv4.tcp_slow_start_after_idle=0
+
+# TCP RACK and TLP.
+sysctl -q net.ipv4.tcp_early_retrans=4 net.ipv4.tcp_recovery=1
+
+# TCP method for deciding when to defer sending to accumulate big TSO packets.
+sysctl -q net.ipv4.tcp_tso_win_divisor=3
+
+# TCP Explicit Congestion Notification (ECN)
+sysctl -q net.ipv4.tcp_ecn=0
+
+sysctl -q net.ipv4.tcp_pacing_ss_ratio=200
+sysctl -q net.ipv4.tcp_pacing_ca_ratio=120
+sysctl -q net.ipv4.tcp_notsent_lowat=4294967295 > /dev/null 2>&1
+
+sysctl -q net.ipv4.tcp_fastopen=0x70403
+sysctl -q net.ipv4.tcp_fastopen_key=a1a1a1a1-b2b2b2b2-c3c3c3c3-d4d4d4d4
+
+sysctl -q net.ipv4.tcp_syncookies=1
+
+# Override the default qdisc on the tun device.
+# Many tests fail with timing errors if the default
+# is FQ and that paces their flows.
+tc qdisc add dev tun0 root pfifo
+
diff --git a/tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py b/tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py
new file mode 100755
index 0000000000000..5ddf456ae973a
--- /dev/null
+++ b/tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py
@@ -0,0 +1,38 @@ 
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""Sets sysctl values and writes a file that restores them.
+
+The arguments are of the form "<proc-file>=<val>" separated by spaces.
+The program first reads the current value of the proc-file and creates
+a shell script named "/tmp/sysctl_restore_${PACKETDRILL_PID}.sh" which
+restores the values when executed. It then sets the new values.
+
+PACKETDRILL_PID is set by packetdrill to the pid of itself, so a .pkt
+file could restore sysctls by running `/tmp/sysctl_restore_${PPID}.sh`
+at the end.
+"""
+
+import os
+import subprocess
+import sys
+
+filename = '/tmp/sysctl_restore_%s.sh' % os.environ['PACKETDRILL_PID']
+
+# Open file for restoring sysctl values
+restore_file = open(filename, 'w')
+print('#!/bin/bash', file=restore_file)
+
+for a in sys.argv[1:]:
+  sysctl = a.split('=')
+  # sysctl[0] contains the proc-file name, sysctl[1] the new value
+
+  # read current value and add restore command to file
+  cur_val = subprocess.check_output(['cat', sysctl[0]], universal_newlines=True)
+  print('echo "%s" > %s' % (cur_val.strip(), sysctl[0]), file=restore_file)
+
+  # set new value
+  cmd = 'echo "%s" > %s' % (sysctl[1], sysctl[0])
+  os.system(cmd)
+
+os.system('chmod u+x %s' % filename)
diff --git a/tools/testing/selftests/net/packetdrill/tcp/inq/client.pkt b/tools/testing/selftests/net/packetdrill/tcp/inq/client.pkt
new file mode 100644
index 0000000000000..8cc7798c7808f
--- /dev/null
+++ b/tools/testing/selftests/net/packetdrill/tcp/inq/client.pkt
@@ -0,0 +1,51 @@ 
+// SPDX-License-Identifier: GPL-2.0
+// Test TCP_INQ and TCP_CM_INQ on the client side.
+`../common/defaults.sh
+`
+
+// Create a socket and set it to non-blocking.
+    0	socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+   +0	fcntl(3, F_GETFL) = 0x2 (flags O_RDWR)
+   +0	fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
+
+// Connect to the server and enable TCP_INQ.
+   +0	connect(3, ..., ...) = -1 EINPROGRESS (Operation now in progress)
+   +0	setsockopt(3, SOL_TCP, TCP_INQ, [1], 4) = 0
+
+   +0	> S 0:0(0) <mss 1460,sackOK,TS val 100 ecr 0,nop,wscale 8>
+ +.01	< S. 0:0(0) ack 1 win 5792 <mss 1460,sackOK,TS val 700 ecr 100,nop,wscale 7>
+   +0	> . 1:1(0) ack 1 <nop,nop,TS val 200 ecr 700>
+
+// Now we have 10K of data ready on the socket.
+   +0	< . 1:10001(10000) ack 1 win 514
+   +0	> . 1:1(0) ack 10001 <nop,nop,TS val 200 ecr 700>
+
+// We read 1K and we should have 9K ready to read.
+   +0	recvmsg(3, {msg_name(...)=...,
+		    msg_iov(1)=[{..., 1000}],
+		    msg_flags=0,
+		    msg_control=[{cmsg_level=SOL_TCP,
+				  cmsg_type=TCP_CM_INQ,
+				  cmsg_data=9000}]}, 0) = 1000
+// We read 9K and we should have no further data ready to read.
+   +0	recvmsg(3, {msg_name(...)=...,
+		    msg_iov(1)=[{..., 9000}],
+		    msg_flags=0,
+		    msg_control=[{cmsg_level=SOL_TCP,
+				  cmsg_type=TCP_CM_INQ,
+				  cmsg_data=0}]}, 0) = 9000
+
+// Server sends more data and closes the connections.
+   +0	< F. 10001:20001(10000) ack 1 win 514
+   +0	> . 1:1(0) ack 20002 <nop,nop,TS val 200 ecr 700>
+
+// We read 10K and we should have one "fake" byte because the connection is
+// closed.
+   +0	recvmsg(3, {msg_name(...)=...,
+		    msg_iov(1)=[{..., 10000}],
+		    msg_flags=0,
+		    msg_control=[{cmsg_level=SOL_TCP,
+				  cmsg_type=TCP_CM_INQ,
+				  cmsg_data=1}]}, 0) = 10000
+// Now, receive EOF.
+   +0	read(3, ..., 2000) = 0
diff --git a/tools/testing/selftests/net/packetdrill/tcp/inq/server.pkt b/tools/testing/selftests/net/packetdrill/tcp/inq/server.pkt
new file mode 100644
index 0000000000000..fd78609087b91
--- /dev/null
+++ b/tools/testing/selftests/net/packetdrill/tcp/inq/server.pkt
@@ -0,0 +1,51 @@ 
+// SPDX-License-Identifier: GPL-2.0
+// Test TCP_INQ and TCP_CM_INQ on the server side.
+`../common/defaults.sh
+`
+
+// Initialize connection
+    0	socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+   +0	setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+   +0	bind(3, ..., ...) = 0
+   +0	listen(3, 1) = 0
+
+   +0	< S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 10>
+   +0	> S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 8>
+ +.01	< . 1:1(0) ack 1 win 514
+
+// Accept the connection and enable TCP_INQ.
+   +0	accept(3, ..., ...) = 4
+   +0	setsockopt(4, SOL_TCP, TCP_INQ, [1], 4) = 0
+
+// Now we have 10K of data ready on the socket.
+   +0	< . 1:10001(10000) ack 1 win 514
+   +0	> . 1:1(0) ack 10001
+
+// We read 2K and we should have 8K ready to read.
+   +0	recvmsg(4, {msg_name(...)=...,
+		    msg_iov(1)=[{..., 2000}],
+		    msg_flags=0,
+		    msg_control=[{cmsg_level=SOL_TCP,
+				  cmsg_type=TCP_CM_INQ,
+				  cmsg_data=8000}]}, 0) = 2000
+// We read 8K and we should have no further data ready to read.
+   +0	recvmsg(4, {msg_name(...)=...,
+		    msg_iov(1)=[{..., 8000}],
+		    msg_flags=0,
+		    msg_control=[{cmsg_level=SOL_TCP,
+				  cmsg_type=TCP_CM_INQ,
+				  cmsg_data=0}]}, 0) = 8000
+// Client sends more data and closes the connections.
+   +0	< F. 10001:20001(10000) ack 1 win 514
+   +0	> . 1:1(0) ack 20002
+
+// We read 10K and we should have one "fake" byte because the connection is
+// closed.
+   +0	recvmsg(4, {msg_name(...)=...,
+		    msg_iov(1)=[{..., 10000}],
+		    msg_flags=0,
+		    msg_control=[{cmsg_level=SOL_TCP,
+				  cmsg_type=TCP_CM_INQ,
+				  cmsg_data=1}]}, 0) = 10000
+// Now, receive error.
+   +0	read(3, ..., 2000) = -1 ENOTCONN (Transport endpoint is not connected)
diff --git a/tools/testing/selftests/net/packetdrill/tcp/md5/md5-only-on-client-ack.pkt b/tools/testing/selftests/net/packetdrill/tcp/md5/md5-only-on-client-ack.pkt
new file mode 100644
index 0000000000000..42b712e14e562
--- /dev/null
+++ b/tools/testing/selftests/net/packetdrill/tcp/md5/md5-only-on-client-ack.pkt
@@ -0,0 +1,28 @@ 
+// SPDX-License-Identifier: GPL-2.0
+// Test what happens when client does not provide MD5 on SYN,
+// but then does on the ACK that completes the three-way handshake.
+
+`../common/defaults.sh`
+
+// Establish a connection.
+    0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+   +0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+   +0 bind(3, ..., ...) = 0
+   +0 listen(3, 1) = 0
+
+   +0 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 10>
+   +0 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 8>
+// Ooh, weird: client provides MD5 option on the ACK:
+ +.01 < . 1:1(0) ack 1 win 514 <md5 000102030405060708090a0b0c0d0e0f,nop,nop>
+ +.01 < . 1:1(0) ack 1 win 514 <md5 000102030405060708090a0b0c0d0e0f,nop,nop>
+
+// The TCP listener refcount should be 2, but on buggy kernels it can be 0:
+   +0 `grep " 0A " /proc/net/tcp /proc/net/tcp6 | grep ":1F90"`
+
+// Now here comes the legit ACK:
+ +.01 < . 1:1(0) ack 1 win 514
+
+// Make sure the connection is OK:
+   +0 accept(3, ..., ...) = 4
+
+ +.01 write(4, ..., 1000) = 1000