On Fri, May 16, 2025 at 10:02:16AM +0200, David Hildenbrand wrote:
On 15.05.25 10:57, Mark Brown wrote:
The kselftest framework uses the string logged when a test result is reported as the unique identifier for a test, using it to track test results between runs. The gup_longterm test completely fails to follow this pattern, it runs a single test function repeatedly with various parameters but each result report is a string logging an error message which is fixed between runs.
As the person who wrote that test (that you apparently didn't CC for some
I just CCed whoever get_maintainers told me to CC for the patch.
reason), what exactly is the problem with that?
We run tests. If all pass, we're happy, if one fails, we investigate.
None of the tooling is able to either distinguish between the multiple tests that are being run in gup_longterm, nor compare the results of multiple runs effectively. If all the tests run they report themselves as being duplicates of the same test name, if one of them starts failing the effect is that one of the duplicates disappears and we get an entirely new test that's never passed reported. If multiple tests fail it's even worse. This means that UIs displaying test results end up reporting things unclearly (Was there a regression or was a new test that never worked added? What was the test?). Since it's difficult to track the tests between runs tooling that does reporting of things like "This last worked in X" in the UI doesn't work properly, and tool driven bisection of issues similarly struggles since it can't tell what's going on with any of the tests between runs.
Basically, the output is garbled and vastly less useful for people running this as a matter of routine or as part of a broader kselftest run. For example with my own automation I probably won't notice that a previously working test failed unless every single test fails, and newly added tests that never worked are a much lower priority to the point where I may never look at them depending on where they are.
If a selftest is reporting multiple tests it should report them with names that are stable and unique.