diff mbox series

[KTAP,V2] ktap_v2: add recognized test name line

Message ID 20230316225915.494688-1-rmoar@google.com
State New
Headers show
Series [KTAP,V2] ktap_v2: add recognized test name line | expand

Commit Message

Rae Moar March 16, 2023, 10:59 p.m. UTC
Add recognition of the test name line ("# Subtest: <name>") to the KTAP v2
spec.

The purpose of this line is to declare the name of a test before its
results. This functionality is especially useful when trying to parse test
results incrementally and when interpretting results after a crash.

This line is already compliant with KTAP v1 as it is interpretted as a
diagnostic line by parsers. Additionally, the line is currently used by
KUnit tests and was derived from the TAP 14 spec:
https://testanything.org/tap-version-14-specification.html.

Recognition of this line would create an accepted way for different test
frameworks to declare the name of a test before its results.

The proposed location for this line is between the version line and the
test plan line. This location ensures that the line would not be
accidentally parsed as a subtest's diagnostic lines. Note this proposed
location would be a slight differentiation from KTAP v1.

Example of test name line:

 KTAP version 2
 # Subtest: main_test
 1..1
   KTAP version 2
   # Subtest: sub_test
   1..2
   ok 1 test_1
   ok 2 test_2
 ok 1 sub_test

Here is a link to a version of the KUnit parser that is able to parse the
test name line for KTAP version 2. Note this includes a test name line for
the main level of KTAP.

Link: https://kunit-review.googlesource.com/c/linux/+/5709

Signed-off-by: Rae Moar <rmoar@google.com>
---

This is a RFC. I would like to know what people think and use this as a
platform for discussion on KTAP v2.

Note: this patch is based on Frank's ktap_spec_version_2 branch.

 Documentation/dev-tools/ktap.rst | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)


base-commit: 906f02e42adfbd5ae70d328ee71656ecb602aaf5

Comments

Daniel Latypov March 20, 2023, 7:21 p.m. UTC | #1
On Thu, Mar 16, 2023 at 3:59 PM Rae Moar <rmoar@google.com> wrote:
>
> Add recognition of the test name line ("# Subtest: <name>") to the KTAP v2
> spec.
>
> The purpose of this line is to declare the name of a test before its
> results. This functionality is especially useful when trying to parse test
> results incrementally and when interpretting results after a crash.
>
> This line is already compliant with KTAP v1 as it is interpretted as a

minor nit for if there's a v2, s/interprett/interpret (here and above)

Also, I want to elaborate on the previous paragraph a bit more, in
case the motivation isn't clear.
The problem with TAP and KTAP as-is is that the name of a test case is
only known *after* it completes.

So the scenario being referred to is

KTAP version 1
1..1
<lots of output>
<kernel crash, no more output>

It would be nice if parsers could report "test FOO caused a crash" as
opposed to "the first test case caused a crash, good luck figuring out
which one that was"

Daniel
Frank Rowand March 27, 2023, 2:41 a.m. UTC | #2
On 3/16/23 17:59, Rae Moar wrote:
> Add recognition of the test name line ("# Subtest: <name>") to the KTAP v2
> spec.
> 
> The purpose of this line is to declare the name of a test before its
> results. This functionality is especially useful when trying to parse test
> results incrementally and when interpretting results after a crash.
> 
> This line is already compliant with KTAP v1 as it is interpretted as a
> diagnostic line by parsers. Additionally, the line is currently used by
> KUnit tests and was derived from the TAP 14 spec:
> https://testanything.org/tap-version-14-specification.html.

It is convenient that "# Subtest: <name>" is compatible with v1, but I think
that there is a negative that overrides the convenience.

The "# Subtest: <name>" syntax means that we need to restrict the format of
diagnostic lines, such that "#Subtest:" is an illegal diagnostic, at least
for the line immediately following the Version line.

I think it would be cleaner to modify the Version line syntax to be:

  KTAP version 2 [# <subtest_name>]

I notice that the KTAP Specification version 1 fails to specify the
Version line syntax.  So the Specification would be updated from:

  All KTAP-formatted results begin with a "version line" which specifies which
  version of the (K)TAP standard the result is compliant with.

  For example:
  - "KTAP version 1"
  - "TAP version 13"
  - "TAP version 14"

to:

  The Version line is required and must have the format:

  .. code-block:: none

        KTAP version 2 [# <subtest_name>]

  All KTAP-formatted results begin with a "version line" which specifies which
  version of the (K)TAP standard the result is compliant with.

  For example:
  - "KTAP version 2"
  - "TAP version 13"
  - "TAP version 14"

> 
> Recognition of this line would create an accepted way for different test
> frameworks to declare the name of a test before its results.
> 
> The proposed location for this line is between the version line and the
> test plan line. This location ensures that the line would not be
> accidentally parsed as a subtest's diagnostic lines. Note this proposed
> location would be a slight differentiation from KTAP v1.
> 
> Example of test name line:
> 
>  KTAP version 2
>  # Subtest: main_test
>  1..1
>    KTAP version 2
>    # Subtest: sub_test
>    1..2
>    ok 1 test_1
>    ok 2 test_2
>  ok 1 sub_test
> 
> Here is a link to a version of the KUnit parser that is able to parse the
> test name line for KTAP version 2. Note this includes a test name line for
> the main level of KTAP.
> 
> Link: https://kunit-review.googlesource.com/c/linux/+/5709
> 
> Signed-off-by: Rae Moar <rmoar@google.com>
> ---
> 
> This is a RFC. I would like to know what people think and use this as a
> platform for discussion on KTAP v2.
> 
> Note: this patch is based on Frank's ktap_spec_version_2 branch.
> 
>  Documentation/dev-tools/ktap.rst | 19 ++++++++++++++-----
>  1 file changed, 14 insertions(+), 5 deletions(-)
> 
> diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
> index ff77f4aaa6ef..9c7ed66d9f77 100644
> --- a/Documentation/dev-tools/ktap.rst
> +++ b/Documentation/dev-tools/ktap.rst
> @@ -28,8 +28,7 @@ KTAP output is built from four different types of lines:
>  In general, valid KTAP output should also form valid TAP output, but some
>  information, in particular nested test results, may be lost. Also note that
>  there is a stagnant draft specification for TAP14, KTAP diverges from this in
> -a couple of places (notably the "Subtest" header), which are described where
> -relevant later in this document.
> +a couple of places, which are described where relevant later in this document.
>  
>  Version lines
>  -------------
> @@ -44,8 +43,8 @@ For example:
>  - "TAP version 14"
>  
>  Note that, in KTAP, subtests also begin with a version line, which denotes the

> -start of the nested test results. This differs from TAP14, which uses a
> -separate "Subtest" line.

^^^^ This is an error in the KTAP Specification version 1.  TAP14 allows the case
of "Bare Subtests", which would be the equivalent of the KTAP v1 method.

> +start of the nested test results. This differs from TAP14, which uses only a
> +"Subtest" line.
>  
>  While, going forward, "KTAP version 2" should be used by compliant tests, it
>  is expected that most parsers and other tooling will accept the other versions
> @@ -166,6 +165,12 @@ even if they do not start with a "#": this is to capture any other useful
>  kernel output which may help debug the test. It is nevertheless recommended
>  that tests always prefix any diagnostic output they have with a "#" character.
>  
> +One recognized diagnostic line is the "# Subtest: <name>" line. This line
> +is used to declare the name of a test before subtest results are printed. This
> +is helpful for parsing and for providing context during crashes. As a rule,
> +this line is placed after the version line and before the plan line. Note
> +this line can be used for the main test, as well as subtests.
> +
>  Unknown lines
>  -------------
>  
> @@ -206,6 +211,7 @@ An example of a test with two nested subtests:
>  	KTAP version 2
>  	1..1
>  	  KTAP version 2
> +	  # Subtest: example
>  	  1..2
>  	  ok 1 test_1
>  	  not ok 2 test_2
> @@ -219,6 +225,7 @@ An example format with multiple levels of nested testing:
>  	KTAP version 2
>  	1..2
>  	  KTAP version 2
> +	  # Subtest: example_test_1
>  	  1..2
>  	    KTAP version 2
>  	    1..2
> @@ -245,7 +252,7 @@ allows an arbitrary number of tests to be nested     no         yes
>  
>  The TAP14 specification does permit nested tests, but instead of using another
>  nested version line, uses a line of the form
> -"Subtest: <name>" where <name> is the name of the parent test.
> +"Subtest: <name>" where <name> is the name of the parent test as discussed above.
>  
>  Example KTAP output
>  --------------------
> @@ -254,6 +261,7 @@ Example KTAP output
>  	KTAP version 2
>  	1..1
>  	  KTAP version 2
> +	  # Subtest: main_test
>  	  1..3
>  	    KTAP version 2
>  	    1..1
> @@ -266,6 +274,7 @@ Example KTAP output
>  	    ok 2 test_2
>  	  ok 2 example_test_2
>  	    KTAP version 2
> +		# Subtest: example_test_3
>  	    1..3
>  	    ok 1 test_1
>  	    # test_2: FAIL
> 
> base-commit: 906f02e42adfbd5ae70d328ee71656ecb602aaf5
Frank Rowand March 27, 2023, 2:44 a.m. UTC | #3
On 3/20/23 14:21, Daniel Latypov wrote:
> On Thu, Mar 16, 2023 at 3:59 PM Rae Moar <rmoar@google.com> wrote:
>>
>> Add recognition of the test name line ("# Subtest: <name>") to the KTAP v2
>> spec.
>>
>> The purpose of this line is to declare the name of a test before its
>> results. This functionality is especially useful when trying to parse test
>> results incrementally and when interpretting results after a crash.
>>
>> This line is already compliant with KTAP v1 as it is interpretted as a
> 
> minor nit for if there's a v2, s/interprett/interpret (here and above)
> 
> Also, I want to elaborate on the previous paragraph a bit more, in
> case the motivation isn't clear.
> The problem with TAP and KTAP as-is is that the name of a test case is
> only known *after* it completes.
> 
> So the scenario being referred to is
> 
> KTAP version 1
> 1..1
> <lots of output>
> <kernel crash, no more output>
> 
> It would be nice if parsers could report "test FOO caused a crash" as
> opposed to "the first test case caused a crash, good luck figuring out
> which one that was"
> 
> Daniel

It would be useful to take some of the motivation explanation from TAP14,
which says (everything to the end of this email):

Commented Subtests are encouraged, as they provide the following benefits:

Easier for humans to read. For example:

  TAP version 14
              1..1
              ok 1 - hmm, what level is this?
vs:

  TAP version 14
  # Subtest: level 1
      # Subtest: level 2
          # Subtest: level 3
              1..1
              ok 1 - clearly level 3

Additional strictness around matching the Test Point description to Subtest Name can catch errors and detect accidentally interleaved output.
Rae Moar March 28, 2023, 9:08 p.m. UTC | #4
On Mon, Mar 20, 2023 at 3:21 PM 'Daniel Latypov' via KUnit Development
<kunit-dev@googlegroups.com> wrote:
>
> On Thu, Mar 16, 2023 at 3:59 PM Rae Moar <rmoar@google.com> wrote:
> >
> > Add recognition of the test name line ("# Subtest: <name>") to the KTAP v2
> > spec.
> >
> > The purpose of this line is to declare the name of a test before its
> > results. This functionality is especially useful when trying to parse test
> > results incrementally and when interpretting results after a crash.
> >
> > This line is already compliant with KTAP v1 as it is interpretted as a
>
> minor nit for if there's a v2, s/interprett/interpret (here and above)
>

Oops. Thanks for catching this. I will change this for a v2. Unsure
why I repeated the typo twice.

> Also, I want to elaborate on the previous paragraph a bit more, in
> case the motivation isn't clear.
> The problem with TAP and KTAP as-is is that the name of a test case is
> only known *after* it completes.
>
> So the scenario being referred to is
>
> KTAP version 1
> 1..1
> <lots of output>
> <kernel crash, no more output>
>
> It would be nice if parsers could report "test FOO caused a crash" as
> opposed to "the first test case caused a crash, good luck figuring out
> which one that was"
>

Yes, this is exactly what I think is the largest benefit of a line to
define the test name pre-results. Thanks for expanding. I will likely
add in some of this explanation into the commit message for v2.

Thanks,

Rae

> Daniel
>
> --
> You received this message because you are subscribed to the Google Groups "KUnit Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/CAGS_qxp2u3F4abmYgN%2BXwCG8CJN37NSqWSC%3D%3D%3DSWLCjZG8HYBw%40mail.gmail.com.
Rae Moar March 29, 2023, 6:34 p.m. UTC | #5
On Sun, Mar 26, 2023 at 10:41 PM Frank Rowand <frowand.list@gmail.com> wrote:
>
> On 3/16/23 17:59, Rae Moar wrote:
> > Add recognition of the test name line ("# Subtest: <name>") to the KTAP v2
> > spec.
> >
> > The purpose of this line is to declare the name of a test before its
> > results. This functionality is especially useful when trying to parse test
> > results incrementally and when interpretting results after a crash.
> >
> > This line is already compliant with KTAP v1 as it is interpretted as a
> > diagnostic line by parsers. Additionally, the line is currently used by
> > KUnit tests and was derived from the TAP 14 spec:
> > https://testanything.org/tap-version-14-specification.html.
>
> It is convenient that "# Subtest: <name>" is compatible with v1, but I think
> that there is a negative that overrides the convenience.
>
> The "# Subtest: <name>" syntax means that we need to restrict the format of
> diagnostic lines, such that "#Subtest:" is an illegal diagnostic, at least
> for the line immediately following the Version line.
>

Hi Frank,

Yes, I see what you are saying here. It would be inconvenient for
parsers to make an exception to the method of parsing diagnostic
lines.

> I think it would be cleaner to modify the Version line syntax to be:
>
>   KTAP version 2 [# <subtest_name>]
>

I like that this idea wouldn't introduce a new line, which is
invaluable. However, I would suspect this alternative may break more
parsers than the first proposal, as current parsers may search for the
full version line to find KTAP results (I know at least KUnit does
this). Therefore I slightly prefer the original proposal. Curious what
others prefer?

Overall, I advocate that KTAP should allow a way to define the name of
the test prior to the results based on the reasons discussed above and
by Daniel and Frank. So if this is the preferred method I would
understand.

> I notice that the KTAP Specification version 1 fails to specify the
> Version line syntax.  So the Specification would be updated from:
>
>   All KTAP-formatted results begin with a "version line" which specifies which
>   version of the (K)TAP standard the result is compliant with.
>
>   For example:
>   - "KTAP version 1"
>   - "TAP version 13"
>   - "TAP version 14"
>
> to:
>
>   The Version line is required and must have the format:
>
>   .. code-block:: none
>
>         KTAP version 2 [# <subtest_name>]
>

I like this added specificity. Would be happy to see specific version
line syntax added to the spec.

Thanks!

Rae

>   All KTAP-formatted results begin with a "version line" which specifies which
>   version of the (K)TAP standard the result is compliant with.
>
>   For example:
>   - "KTAP version 2"
>   - "TAP version 13"
>   - "TAP version 14"
>
> >
> > Recognition of this line would create an accepted way for different test
> > frameworks to declare the name of a test before its results.
> >
> > The proposed location for this line is between the version line and the
> > test plan line. This location ensures that the line would not be
> > accidentally parsed as a subtest's diagnostic lines. Note this proposed
> > location would be a slight differentiation from KTAP v1.
> >
> > Example of test name line:
> >
> >  KTAP version 2
> >  # Subtest: main_test
> >  1..1
> >    KTAP version 2
> >    # Subtest: sub_test
> >    1..2
> >    ok 1 test_1
> >    ok 2 test_2
> >  ok 1 sub_test
> >
> > Here is a link to a version of the KUnit parser that is able to parse the
> > test name line for KTAP version 2. Note this includes a test name line for
> > the main level of KTAP.
> >
> > Link: https://kunit-review.googlesource.com/c/linux/+/5709
> >
> > Signed-off-by: Rae Moar <rmoar@google.com>
> > ---
> >
> > This is a RFC. I would like to know what people think and use this as a
> > platform for discussion on KTAP v2.
> >
> > Note: this patch is based on Frank's ktap_spec_version_2 branch.
> >
> >  Documentation/dev-tools/ktap.rst | 19 ++++++++++++++-----
> >  1 file changed, 14 insertions(+), 5 deletions(-)
> >
> > diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
> > index ff77f4aaa6ef..9c7ed66d9f77 100644
> > --- a/Documentation/dev-tools/ktap.rst
> > +++ b/Documentation/dev-tools/ktap.rst
> > @@ -28,8 +28,7 @@ KTAP output is built from four different types of lines:
> >  In general, valid KTAP output should also form valid TAP output, but some
> >  information, in particular nested test results, may be lost. Also note that
> >  there is a stagnant draft specification for TAP14, KTAP diverges from this in
> > -a couple of places (notably the "Subtest" header), which are described where
> > -relevant later in this document.
> > +a couple of places, which are described where relevant later in this document.
> >
> >  Version lines
> >  -------------
> > @@ -44,8 +43,8 @@ For example:
> >  - "TAP version 14"
> >
> >  Note that, in KTAP, subtests also begin with a version line, which denotes the
>
> > -start of the nested test results. This differs from TAP14, which uses a
> > -separate "Subtest" line.
>
> ^^^^ This is an error in the KTAP Specification version 1.  TAP14 allows the case
> of "Bare Subtests", which would be the equivalent of the KTAP v1 method.
>
> > +start of the nested test results. This differs from TAP14, which uses only a
> > +"Subtest" line.
> >
> >  While, going forward, "KTAP version 2" should be used by compliant tests, it
> >  is expected that most parsers and other tooling will accept the other versions
> > @@ -166,6 +165,12 @@ even if they do not start with a "#": this is to capture any other useful
> >  kernel output which may help debug the test. It is nevertheless recommended
> >  that tests always prefix any diagnostic output they have with a "#" character.
> >
> > +One recognized diagnostic line is the "# Subtest: <name>" line. This line
> > +is used to declare the name of a test before subtest results are printed. This
> > +is helpful for parsing and for providing context during crashes. As a rule,
> > +this line is placed after the version line and before the plan line. Note
> > +this line can be used for the main test, as well as subtests.
> > +
> >  Unknown lines
> >  -------------
> >
> > @@ -206,6 +211,7 @@ An example of a test with two nested subtests:
> >       KTAP version 2
> >       1..1
> >         KTAP version 2
> > +       # Subtest: example
> >         1..2
> >         ok 1 test_1
> >         not ok 2 test_2
> > @@ -219,6 +225,7 @@ An example format with multiple levels of nested testing:
> >       KTAP version 2
> >       1..2
> >         KTAP version 2
> > +       # Subtest: example_test_1
> >         1..2
> >           KTAP version 2
> >           1..2
> > @@ -245,7 +252,7 @@ allows an arbitrary number of tests to be nested     no         yes
> >
> >  The TAP14 specification does permit nested tests, but instead of using another
> >  nested version line, uses a line of the form
> > -"Subtest: <name>" where <name> is the name of the parent test.
> > +"Subtest: <name>" where <name> is the name of the parent test as discussed above.
> >
> >  Example KTAP output
> >  --------------------
> > @@ -254,6 +261,7 @@ Example KTAP output
> >       KTAP version 2
> >       1..1
> >         KTAP version 2
> > +       # Subtest: main_test
> >         1..3
> >           KTAP version 2
> >           1..1
> > @@ -266,6 +274,7 @@ Example KTAP output
> >           ok 2 test_2
> >         ok 2 example_test_2
> >           KTAP version 2
> > +             # Subtest: example_test_3
> >           1..3
> >           ok 1 test_1
> >           # test_2: FAIL
> >
> > base-commit: 906f02e42adfbd5ae70d328ee71656ecb602aaf5
>
> --
> You received this message because you are subscribed to the Google Groups "KUnit Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/197889b6-5773-094c-8699-26843c6519fd%40gmail.com.
diff mbox series

Patch

diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
index ff77f4aaa6ef..9c7ed66d9f77 100644
--- a/Documentation/dev-tools/ktap.rst
+++ b/Documentation/dev-tools/ktap.rst
@@ -28,8 +28,7 @@  KTAP output is built from four different types of lines:
 In general, valid KTAP output should also form valid TAP output, but some
 information, in particular nested test results, may be lost. Also note that
 there is a stagnant draft specification for TAP14, KTAP diverges from this in
-a couple of places (notably the "Subtest" header), which are described where
-relevant later in this document.
+a couple of places, which are described where relevant later in this document.
 
 Version lines
 -------------
@@ -44,8 +43,8 @@  For example:
 - "TAP version 14"
 
 Note that, in KTAP, subtests also begin with a version line, which denotes the
-start of the nested test results. This differs from TAP14, which uses a
-separate "Subtest" line.
+start of the nested test results. This differs from TAP14, which uses only a
+"Subtest" line.
 
 While, going forward, "KTAP version 2" should be used by compliant tests, it
 is expected that most parsers and other tooling will accept the other versions
@@ -166,6 +165,12 @@  even if they do not start with a "#": this is to capture any other useful
 kernel output which may help debug the test. It is nevertheless recommended
 that tests always prefix any diagnostic output they have with a "#" character.
 
+One recognized diagnostic line is the "# Subtest: <name>" line. This line
+is used to declare the name of a test before subtest results are printed. This
+is helpful for parsing and for providing context during crashes. As a rule,
+this line is placed after the version line and before the plan line. Note
+this line can be used for the main test, as well as subtests.
+
 Unknown lines
 -------------
 
@@ -206,6 +211,7 @@  An example of a test with two nested subtests:
 	KTAP version 2
 	1..1
 	  KTAP version 2
+	  # Subtest: example
 	  1..2
 	  ok 1 test_1
 	  not ok 2 test_2
@@ -219,6 +225,7 @@  An example format with multiple levels of nested testing:
 	KTAP version 2
 	1..2
 	  KTAP version 2
+	  # Subtest: example_test_1
 	  1..2
 	    KTAP version 2
 	    1..2
@@ -245,7 +252,7 @@  allows an arbitrary number of tests to be nested     no         yes
 
 The TAP14 specification does permit nested tests, but instead of using another
 nested version line, uses a line of the form
-"Subtest: <name>" where <name> is the name of the parent test.
+"Subtest: <name>" where <name> is the name of the parent test as discussed above.
 
 Example KTAP output
 --------------------
@@ -254,6 +261,7 @@  Example KTAP output
 	KTAP version 2
 	1..1
 	  KTAP version 2
+	  # Subtest: main_test
 	  1..3
 	    KTAP version 2
 	    1..1
@@ -266,6 +274,7 @@  Example KTAP output
 	    ok 2 test_2
 	  ok 2 example_test_2
 	    KTAP version 2
+		# Subtest: example_test_3
 	    1..3
 	    ok 1 test_1
 	    # test_2: FAIL