On Thu, Oct 29, 2020 at 7:34 PM David Gow davidgow@google.com wrote:

On Wed, Oct 21, 2020 at 7:32 AM Daniel Latypov dlatypov@google.com wrote:

...

For simplcity, strip all trailing whitespace from parsed output. I imagine no one is printing out meaningful trailing whitespace via KUNIT_FAIL() or similar, and that if they are, they really shouldn't.

At some point, the lines from `isolate_kunit_output()` started having trailing \n, which results in artifacty output like this:

$ ./tools/testing/kunit/kunit.py run [16:16:46] [FAILED] example_simple_test [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29

[16:16:46] Expected 1 + 1 == 3, but

[16:16:46] 1 + 1 == 2

[16:16:46] 3 == 3

[16:16:46] not ok 1 - example_simple_test

[16:16:46]

After this change: [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29 [16:16:46] Expected 1 + 1 == 3, but [16:16:46] 1 + 1 == 2 [16:16:46] 3 == 3 [16:16:46] not ok 1 - example_simple_test [16:16:46]

Signed-off-by: Daniel Latypov dlatypov@google.com

Thanks! This is a long-overdue fix, and it worked well for me.

Tested-by: David Gow davidgow@google.com

One comment below:

...

tools/testing/kunit/kunit_parser.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 8019e3dd4c32..e68b1c66a73f 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -342,7 +342,8 @@ def parse_run_tests(kernel_output) -> TestResult: total_tests = 0 failed_tests = 0 crashed_tests = 0

•   test_result = parse_test_result(list(isolate_kunit_output(kernel_output)))
  

•   test_result = parse_test_result(list(
  

•        l.rstrip() for l in isolate_kunit_output(kernel_output)))
  

Could we do this inside isolate_kunit_output() instead? That seems like it'd be a more logical place for it (removing the newline is a sort of isolating the output), and it'd avoid making this line quite as horrifyingly nested.

Good point. We could either do it on each yield (messy), or before, i.e.

diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 8019e3dd4c32..14d35deb96cd 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -54,6 +54,7 @@ kunit_end_re = re.compile('(List of all partitions:|' def isolate_kunit_output(kernel_output): started = False for line in kernel_output: + line = line.rstrip() # line always has a trailing \n if kunit_start_re.search(line): prefix_len = len(line.split('TAP version')[0]) started = True

I had some vague concerns about this as kunit_start_re = re.compile(r'TAP version [0-9]+$') has that anchor at the end.

This could ostensibly make it match more things than before. Since I'm using rstrip() out of laziness, that means strings like '<prefix we allow for some reason>TAP version 42\t\n' will now also match.

I don't really think that's an issue, but I'd sent this as a more conservative change initially. I can send the diff above as a replacement for this patch.

...

    if test_result.status == TestStatus.NO_TESTS:
            print(red('[ERROR] ') + yellow('no tests run!'))
    elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS:

base-commit: c4d6fe7311762f2e03b3c27ad38df7c40c80cc93

2.29.0.rc1.297.gfa9743e501-goog

On Fri, Oct 30, 2020 at 12:22 AM David Gow davidgow@google.com wrote:

On Fri, Oct 30, 2020 at 1:41 PM Daniel Latypov dlatypov@google.com wrote:

...

On Thu, Oct 29, 2020 at 7:34 PM David Gow davidgow@google.com wrote:

...

On Wed, Oct 21, 2020 at 7:32 AM Daniel Latypov dlatypov@google.com wrote:

...

For simplcity, strip all trailing whitespace from parsed output. I imagine no one is printing out meaningful trailing whitespace via KUNIT_FAIL() or similar, and that if they are, they really shouldn't.

At some point, the lines from `isolate_kunit_output()` started having trailing \n, which results in artifacty output like this:

$ ./tools/testing/kunit/kunit.py run [16:16:46] [FAILED] example_simple_test [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29

[16:16:46] Expected 1 + 1 == 3, but

[16:16:46] 1 + 1 == 2

[16:16:46] 3 == 3

[16:16:46] not ok 1 - example_simple_test

[16:16:46]

After this change: [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29 [16:16:46] Expected 1 + 1 == 3, but [16:16:46] 1 + 1 == 2 [16:16:46] 3 == 3 [16:16:46] not ok 1 - example_simple_test [16:16:46]

Signed-off-by: Daniel Latypov dlatypov@google.com

Thanks! This is a long-overdue fix, and it worked well for me.

Tested-by: David Gow davidgow@google.com

One comment below:

...

tools/testing/kunit/kunit_parser.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 8019e3dd4c32..e68b1c66a73f 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -342,7 +342,8 @@ def parse_run_tests(kernel_output) -> TestResult: total_tests = 0 failed_tests = 0 crashed_tests = 0

•   test_result = parse_test_result(list(isolate_kunit_output(kernel_output)))
  

•   test_result = parse_test_result(list(
  

•        l.rstrip() for l in isolate_kunit_output(kernel_output)))
  

Could we do this inside isolate_kunit_output() instead? That seems like it'd be a more logical place for it (removing the newline is a sort of isolating the output), and it'd avoid making this line quite as horrifyingly nested.

Good point. We could either do it on each yield (messy), or before, i.e.

diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 8019e3dd4c32..14d35deb96cd 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -54,6 +54,7 @@ kunit_end_re = re.compile('(List of all partitions:|' def isolate_kunit_output(kernel_output): started = False for line in kernel_output:

•           line = line.rstrip()  # line always has a trailing \n
            if kunit_start_re.search(line):
                    prefix_len = len(line.split('TAP version')[0])
                    started = True
  
  

I had some vague concerns about this as kunit_start_re = re.compile(r'TAP version [0-9]+$') has that anchor at the end.

This could ostensibly make it match more things than before. Since I'm using rstrip() out of laziness, that means strings like '<prefix we allow for some reason>TAP version 42\t\n' will now also match.

I don't really think that's an issue, but I'd sent this as a more conservative change initially. I can send the diff above as a replacement for this patch.

I prefer this if it works. From my cursory testing, it does (though the kunt_tool_tests.py tests will need updating). At the very least, I'm able to get it to work with --alltests / allyesconfig (with a few options tactically disabled), which was the main reason we needed isolate_kunit_output() in the first place.

So, unless anyone can find a real-world case where this breaks something, let's go with this.

Sounds good. Sent https://lore.kernel.org/linux-kselftest/20201030223853.554597-1-dlatypov@goo...

Cheers, -- David

...

...

...

    if test_result.status == TestStatus.NO_TESTS:
            print(red('[ERROR] ') + yellow('no tests run!'))
    elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS:

base-commit: c4d6fe7311762f2e03b3c27ad38df7c40c80cc93

2.29.0.rc1.297.gfa9743e501-goog