In the middle of the thread about a patch to add the skip test result, I suggested documenting the process of deprecating the KTAP v1 Specification method of marking a skipped test:
https://lore.kernel.org/all/490271eb-1429-2217-6e38-837c6e5e328b@gmail.com/T...
In a reply to that email I suggested that we ought to have a process to transition the KTAP Specification from v1 to v2, and possibly v3 and future.
This email is meant to be the root of that discussion.
My initial thinking is that there are at least three different types of project and/or community that may have different needs in this area.
Type 1 - project controls both the test output generation and the test output parsing tool. Both generation and parsing code are in the same repository and/or synchronized versions are distributed together.
Devicetree unittests are an example of Type 1. I plan to maintain changes of test output to KTAP v2 format in coordination with updating the parser to process KTAP v2 data.
Type 2 - project controls both the test output generation and the test output parsing tool. The test output generation and a parser modifications may be controlled by the project BUT there are one or more external testing projects that (1) may have their own parsers, and (2) may have a single framework that tests multiple versions of the tests.
I think that kselftest and kunit tests are probably examples of Type 2. I also think that DT unittests will become a Type 2 project as a result of converting to KTAP v2 data.
Type 3 - project may create and maintain some tests, but is primarily a consumer of tests created by other projects. Type 3 projects typically have a single framework that is able to execute and process multiple versions of the tests.
The Fuego test project is an example of Type 3.
Maybe adding all of this complexity of different Types in my initial thinking was silly -- maybe everything in this topic is governed by the more complex Type 3.
My thinking was that the three different Types of project would be impacted in different ways by transition plans. Type 3 would be the most impacted, so I wanted to be sure that any transition plan especially considered their needs.
There is an important aspect of the KTAP format that might ease the transition from one version to another: All KTAP formatted results begin with a "version line", so as soon as a parser has processed the first line of a test, it can apply the appropriate KTAP Specification version to all subsequent lines of test output. A parser implementation could choose to process all versions, could choose to invoke a version specific parser, or some other approach all together.
In the "add skip test results" thread, I suggested deprecating the v1 method of marking a skipped test in v2, with a scheduled removal of the v1 method in v3. But since the KTAP format version is available in the very first line of test output, is it necessary to do a slow deprecation and removal over two versions?
One argument to doing a two version deprecation/removal process is that a parser that is one version older the the test output _might_ be able to process the test output without error, but would not be able to take advantage of features added in the newer version of the Specification.
My opinion is that a two version deprecation/removal process will slow the Specification update process and lead to more versions of the Specification over a given time interval.
A one version deprecation/removal process puts more of a burden on Type 3 projects and external parsers for Type 2 projects to implement parsers that can process the newer Specification more quickly and puts a burden on test maintainers to delay a move to the newer Specification, or possibly pressure to support selection of more than one Specification version format for output data.
One additional item... On the KTAP Specification version 2 process wiki page, I suggested that it is "desirable for test result parsers that understand the KTAP Specification version 2 data also be able to parse version 1 data." With the implication "Converting version 1 compliant data to version 2 compliant data should not require a "flag day" switch of test result parsers." If this thread discussion results in a different decision, I will update the wiki.
Thoughts?
-Frank
Hello everyone,
It has been a few months since there has been activity regarding the second version of KTAP. I wanted to bring this topic back up to the surface.
Currently, Frank has compiled a list of KTAPv2 patches here at this link: https://elinux.org/Test_Results_Format_Notes#KTAP_version_2
I am interested in starting the process of accepting these patches as changes to the KTAP documentation as KTAPv2.
First, a decision that needs to be made is which branch should be used to accept these changes. Frank has a git repository for KTAPv2. However, it is my understanding he has retired. As a reviewer of KUnit, I am happy to take the patches in through the KUnit branch. Would this work for everyone?
Second, we need to finalize the changes. A current list of proposed KTAPv2 patches is as follows:
[PATCH v3 0/2] begin KTAP spec v2 process [PATCH v3 1/2] ktap_v2: change version to 2-rc in KTAP specification [PATCH v3 2/2] ktap_v2: change "version 1" to "version 2" in examples [KTAP V2 PATCH] ktap_v2: add skip test result [KTAP V2 PATCH v4] ktap_v2: add test metadata
Note the patch on adding a skip test result has not yet been reviewed. So please take a look at this change if interested.
As a rule, for any feature of KTAPv1 that is replaced in KTAPv2, it will stay in the documentation as allowed but deprecated to allow a smooth transition.
If this process sounds good to people, I will try to get discussions going on current patches and ask for reviews. My goal is by the end of the summer, I can apply the approved patches and send them through a chosen branch (potentially KUnit, as discussed above).
Let me know what you think. Thanks! -Rae
On Sun, Mar 26, 2023 at 7:25 PM Frank Rowand frowand.list@gmail.com wrote:
In the middle of the thread about a patch to add the skip test result, I suggested documenting the process of deprecating the KTAP v1 Specification method of marking a skipped test:
https://lore.kernel.org/all/490271eb-1429-2217-6e38-837c6e5e328b@gmail.com/T...
In a reply to that email I suggested that we ought to have a process to transition the KTAP Specification from v1 to v2, and possibly v3 and future.
This email is meant to be the root of that discussion.
My initial thinking is that there are at least three different types of project and/or community that may have different needs in this area.
Type 1 - project controls both the test output generation and the test output parsing tool. Both generation and parsing code are in the same repository and/or synchronized versions are distributed together.
Devicetree unittests are an example of Type 1. I plan to maintain changes of test output to KTAP v2 format in coordination with updating the parser to process KTAP v2 data.
Type 2 - project controls both the test output generation and the test output parsing tool. The test output generation and a parser modifications may be controlled by the project BUT there are one or more external testing projects that (1) may have their own parsers, and (2) may have a single framework that tests multiple versions of the tests.
I think that kselftest and kunit tests are probably examples of Type 2. I also think that DT unittests will become a Type 2 project as a result of converting to KTAP v2 data.
Type 3 - project may create and maintain some tests, but is primarily a consumer of tests created by other projects. Type 3 projects typically have a single framework that is able to execute and process multiple versions of the tests.
The Fuego test project is an example of Type 3.
Maybe adding all of this complexity of different Types in my initial thinking was silly -- maybe everything in this topic is governed by the more complex Type 3.
My thinking was that the three different Types of project would be impacted in different ways by transition plans. Type 3 would be the most impacted, so I wanted to be sure that any transition plan especially considered their needs.
There is an important aspect of the KTAP format that might ease the transition from one version to another: All KTAP formatted results begin with a "version line", so as soon as a parser has processed the first line of a test, it can apply the appropriate KTAP Specification version to all subsequent lines of test output. A parser implementation could choose to process all versions, could choose to invoke a version specific parser, or some other approach all together.
In the "add skip test results" thread, I suggested deprecating the v1 method of marking a skipped test in v2, with a scheduled removal of the v1 method in v3. But since the KTAP format version is available in the very first line of test output, is it necessary to do a slow deprecation and removal over two versions?
One argument to doing a two version deprecation/removal process is that a parser that is one version older the the test output _might_ be able to process the test output without error, but would not be able to take advantage of features added in the newer version of the Specification.
My opinion is that a two version deprecation/removal process will slow the Specification update process and lead to more versions of the Specification over a given time interval.
A one version deprecation/removal process puts more of a burden on Type 3 projects and external parsers for Type 2 projects to implement parsers that can process the newer Specification more quickly and puts a burden on test maintainers to delay a move to the newer Specification, or possibly pressure to support selection of more than one Specification version format for output data.
One additional item... On the KTAP Specification version 2 process wiki page, I suggested that it is "desirable for test result parsers that understand the KTAP Specification version 2 data also be able to parse version 1 data." With the implication "Converting version 1 compliant data to version 2 compliant data should not require a "flag day" switch of test result parsers." If this thread discussion results in a different decision, I will update the wiki.
Thoughts?
-Frank
-- You received this message because you are subscribed to the Google Groups "KUnit Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/6d4afb49-3cb9-f176-61a2-5bbaab69....
linux-kselftest-mirror@lists.linaro.org
-
Frank Rowand -
Rae Moar -
Selena Thomas