Jakub Kicinski wrote:
On Wed, 31 Jan 2024 15:27:34 -0500 Willem de Bruijn wrote:
Jakub Kicinski wrote:
+1 I also think we should run and ignore failure. I was wondering if we can swap FAIL for XFAIL in those cases:
tools/testing/selftests/kselftest.h #define KSFT_XFAIL 2
Documentation/dev-tools/ktap.rst
- "XFAIL", which indicates that a test is expected to fail. This is similar to "TODO", above, and is used by some kselftest tests.
IDK if that's a stretch or not. Or we can just return PASS with a comment?
Flaky tests will then report both pass and expected fail. That might add noise to https://netdev.bots.linux.dev/flakes.html?
I initially considered returning skipped on timing failure. But that has the same issue.
So perhaps just return pass?
Especially for flaky tests sometimes returning pass and sometimes returning expected to fa red/green dash such as
Right, we only have pass / fail / skip. (I put the "warn" result in for tests migrated from patchwork so ignore its existence for tests.)
We already treat XFAIL in KTAP as "pass". TCP-AO's key-managemeent_ipv6 test for example already reports XFAIL:
Ok perfect. Then I'll do the same.
# ok 15 # XFAIL listen() after current/rnext keys set: the socket has current/rn ext keys: 100:200
Skips look somewhat similar in KTAP, "ok $number # SKIP" but we fish those out specifically to catch skips. Any other "ok .... # comment" KTAP result is treated as a "pass" right now.