From: Willem de Bruijn willemb@google.com
Some packetdrill tests are flaky in debug mode. As discussed, increase tolerance.
We have been doing this for debug builds outside ksft too.
Previous setting was 10000. A manual 50 runs in virtme-ng showed two failures that needed 12000. To be on the safe side, Increase to 14000.
Link: https://lore.kernel.org/netdev/Zuhhe4-MQHd3EkfN@mini-arch/ Fixes: 1e42f73fd3c2 ("selftests/net: packetdrill: import tcp/zerocopy") Reported-by: Stanislav Fomichev sdf@fomichev.me Signed-off-by: Willem de Bruijn willemb@google.com --- tools/testing/selftests/net/packetdrill/ksft_runner.sh | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/tools/testing/selftests/net/packetdrill/ksft_runner.sh b/tools/testing/selftests/net/packetdrill/ksft_runner.sh index 7478c0c0c9aa..4071c133f29e 100755 --- a/tools/testing/selftests/net/packetdrill/ksft_runner.sh +++ b/tools/testing/selftests/net/packetdrill/ksft_runner.sh @@ -30,12 +30,17 @@ if [ -z "$(which packetdrill)" ]; then exit "$KSFT_SKIP" fi
+declare -a optargs +if [[ -n "${KSFT_MACHINE_SLOW}" ]]; then + optargs+=('--tolerance_usecs=14000') +fi + ktap_print_header ktap_set_plan 2
-unshare -n packetdrill ${ipv4_args[@]} $(basename $script) > /dev/null \ +unshare -n packetdrill ${ipv4_args[@]} ${optargs[@]} $(basename $script) > /dev/null \ && ktap_test_pass "ipv4" || ktap_test_fail "ipv4" -unshare -n packetdrill ${ipv6_args[@]} $(basename $script) > /dev/null \ +unshare -n packetdrill ${ipv6_args[@]} ${optargs[@]} $(basename $script) > /dev/null \ && ktap_test_pass "ipv6" || ktap_test_fail "ipv6"
ktap_finished
On Thu, Sep 19, 2024 at 08:43:42AM -0400, Willem de Bruijn wrote:
From: Willem de Bruijn willemb@google.com
Some packetdrill tests are flaky in debug mode. As discussed, increase tolerance.
We have been doing this for debug builds outside ksft too.
Previous setting was 10000. A manual 50 runs in virtme-ng showed two failures that needed 12000. To be on the safe side, Increase to 14000.
Link: https://lore.kernel.org/netdev/Zuhhe4-MQHd3EkfN@mini-arch/ Fixes: 1e42f73fd3c2 ("selftests/net: packetdrill: import tcp/zerocopy") Reported-by: Stanislav Fomichev sdf@fomichev.me Signed-off-by: Willem de Bruijn willemb@google.com
Reviewed-by: Simon Horman horms@kernel.org
On 09/19, Willem de Bruijn wrote:
From: Willem de Bruijn willemb@google.com
Some packetdrill tests are flaky in debug mode. As discussed, increase tolerance.
We have been doing this for debug builds outside ksft too.
Previous setting was 10000. A manual 50 runs in virtme-ng showed two failures that needed 12000. To be on the safe side, Increase to 14000.
Link: https://lore.kernel.org/netdev/Zuhhe4-MQHd3EkfN@mini-arch/ Fixes: 1e42f73fd3c2 ("selftests/net: packetdrill: import tcp/zerocopy") Reported-by: Stanislav Fomichev sdf@fomichev.me Signed-off-by: Willem de Bruijn willemb@google.com
Acked-by: Stanislav Fomichev sdf@fomichev.me
Thanks! Should probably go to net-next though? (Not sure what's the bar for selftests fixes for 'net')
On 9/19/24 23:04, Stanislav Fomichev wrote:
On 09/19, Willem de Bruijn wrote:
From: Willem de Bruijn willemb@google.com
Some packetdrill tests are flaky in debug mode. As discussed, increase tolerance.
We have been doing this for debug builds outside ksft too.
Previous setting was 10000. A manual 50 runs in virtme-ng showed two failures that needed 12000. To be on the safe side, Increase to 14000.
Link: https://lore.kernel.org/netdev/Zuhhe4-MQHd3EkfN@mini-arch/ Fixes: 1e42f73fd3c2 ("selftests/net: packetdrill: import tcp/zerocopy") Reported-by: Stanislav Fomichev sdf@fomichev.me Signed-off-by: Willem de Bruijn willemb@google.com
Acked-by: Stanislav Fomichev sdf@fomichev.me
Thanks! Should probably go to net-next though? (Not sure what's the bar for selftests fixes for 'net')
FTR, we want this kind of fixes in net, to reach self-test stability in both trees ASAP.
Cheers,
Paolo
Hi Willem,
On 19/09/2024 14:43, Willem de Bruijn wrote:
From: Willem de Bruijn willemb@google.com
Some packetdrill tests are flaky in debug mode. As discussed, increase tolerance.
Thank you for the patch!
We have been doing this for debug builds outside ksft too.
Previous setting was 10000. A manual 50 runs in virtme-ng showed two failures that needed 12000. To be on the safe side, Increase to 14000.
So far (in 3 runs), it looks like 14000 is enough. But I guess it is still a bit too early to conclude that.
https://netdev.bots.linux.dev/contest.html?executor=vmksft-packetdrill-dbg
(Your patch has been introduced in the net-next-2024-09-19--15-00 branch.)
Personally, I would not be chocked if the tolerance was even 10x higher to cope with this very slow environment where we care less about timing I think. But if less works, that's good:
Acked-by: Matthieu Baerts (NGI0) matttbe@kernel.org
Just one question for later: in the GitHub repo, some tests set the tolerance in the .pkt file, will it be OK for these tests? I guess yes, because the max they set is 10k, but I just want to double-check.
(Note that it is now easier to spot other errors :) e.g.)
https://netdev-3.bots.linux.dev/vmksft-packetdrill-dbg/results/779660/22-tcp...
Cheers, Matt
Hi Willem,
On 20/09/2024 00:03, Matthieu Baerts wrote:
On 19/09/2024 14:43, Willem de Bruijn wrote:
From: Willem de Bruijn willemb@google.com
(...)
We have been doing this for debug builds outside ksft too.
Previous setting was 10000. A manual 50 runs in virtme-ng showed two failures that needed 12000. To be on the safe side, Increase to 14000.
So far (in 3 runs), it looks like 14000 is enough. But I guess it is still a bit too early to conclude that.
https://netdev.bots.linux.dev/contest.html?executor=vmksft-packetdrill-dbg
(Your patch has been introduced in the net-next-2024-09-19--15-00 branch.)
One week after the introduction of this patch and >50 builds, it looks like the results are good, only one issue related to timing issues:
https://netdev-3.bots.linux.dev/vmksft-packetdrill-dbg/results/782181/1-tcp-...
And it passed after a retry.
https://netdev.bots.linux.dev/flakes.html?min-flip=0&tn-needle=packetdri...
Cheers, Matt
Matthieu Baerts wrote:
Hi Willem,
On 20/09/2024 00:03, Matthieu Baerts wrote:
On 19/09/2024 14:43, Willem de Bruijn wrote:
From: Willem de Bruijn willemb@google.com
(...)
We have been doing this for debug builds outside ksft too.
Previous setting was 10000. A manual 50 runs in virtme-ng showed two failures that needed 12000. To be on the safe side, Increase to 14000.
So far (in 3 runs), it looks like 14000 is enough. But I guess it is still a bit too early to conclude that.
https://netdev.bots.linux.dev/contest.html?executor=vmksft-packetdrill-dbg
(Your patch has been introduced in the net-next-2024-09-19--15-00 branch.)
One week after the introduction of this patch and >50 builds, it looks like the results are good, only one issue related to timing issues:
https://netdev-3.bots.linux.dev/vmksft-packetdrill-dbg/results/782181/1-tcp-...
And it passed after a retry.
https://netdev.bots.linux.dev/flakes.html?min-flip=0&tn-needle=packetdri...
Thanks Matthieu.
Hello:
This patch was applied to netdev/net.git (main) by Paolo Abeni pabeni@redhat.com:
On Thu, 19 Sep 2024 08:43:42 -0400 you wrote:
From: Willem de Bruijn willemb@google.com
Some packetdrill tests are flaky in debug mode. As discussed, increase tolerance.
We have been doing this for debug builds outside ksft too.
[...]
Here is the summary with links: - [net] selftests/net: packetdrill: increase timing tolerance in debug mode https://git.kernel.org/netdev/net/c/72ef07554c5d
You are awesome, thank you!
linux-kselftest-mirror@lists.linaro.org
-
Matthieu Baerts -
Paolo Abeni -
patchwork-bot+netdevbpf＠kernel.org -
Simon Horman -
Stanislav Fomichev -
Willem de Bruijn