The current selftests infrastructure formats the results in TAP 13. This version doesn't support subtests and only the end result of each selftest is taken into account. It means that a single issue in a subtest of a selftest containing multiple subtests forces the whole selftest to be marked as failed. It also means that subtests results are not tracked by CI executing selftests.
MPTCP selftests run hundreds of various subtests. It is then important to track each of them and not one result per selftest.
It is particularly interesting to do that when validating stable kernels with the last version of the test suite: tests might fail because a feature is not supported but the test didn't skip that part. In this case, if subtests are not tracked, the whole selftest will be marked as failed making the other subtests useless because their results are ignored.
Regarding this patch set:
- The two first patches modify connect and userspace_pm selftests to continue executing other tests if there is an error before the end. This is what is done in the other MPTCP selftests.
- Patches 3-5 are refactoring the code in userspace_pm selftest to reduce duplicated code, suppress some shellcheck warnings and prepare subtests' support by using new helpers.
- Patch 6 adds new helpers in mptcp_lib.sh to easily support printing the subtests results in the different MPTCP selftests.
- Patch 7-13 format subtests results in TAP 13 in the different MPTCP selftests.
Signed-off-by: Matthieu Baerts matthieu.baerts@tessares.net --- Matthieu Baerts (13): selftests: mptcp: connect: don't stop if error selftests: mptcp: userspace pm: don't stop if error selftests: mptcp: userspace_pm: fix shellcheck warnings selftests: mptcp: userspace_pm: uniform results printing selftests: mptcp: userspace_pm: reduce dup code around printf selftests: mptcp: lib: format subtests results in TAP selftests: mptcp: connect: format subtests results in TAP selftests: mptcp: pm_netlink: format subtests results in TAP selftests: mptcp: join: format subtests results in TAP selftests: mptcp: diag: format subtests results in TAP selftests: mptcp: simult flows: format subtests results in TAP selftests: mptcp: sockopt: format subtests results in TAP selftests: mptcp: userspace_pm: format subtests results in TAP
tools/testing/selftests/net/mptcp/diag.sh | 7 + tools/testing/selftests/net/mptcp/mptcp_connect.sh | 66 ++++++-- tools/testing/selftests/net/mptcp/mptcp_join.sh | 37 ++++- tools/testing/selftests/net/mptcp/mptcp_lib.sh | 66 ++++++++ tools/testing/selftests/net/mptcp/mptcp_sockopt.sh | 20 ++- tools/testing/selftests/net/mptcp/pm_netlink.sh | 6 +- tools/testing/selftests/net/mptcp/simult_flows.sh | 4 + tools/testing/selftests/net/mptcp/userspace_pm.sh | 181 +++++++++++++-------- 8 files changed, 298 insertions(+), 89 deletions(-) --- base-commit: 60cc1f7d0605598b47ee3c0c2b4b6fbd4da50a06 change-id: 20230712-upstream-net-next-20230712-selftests-mptcp-subtests-25d250d77886
Best regards,
No more tests were executed after a failure but it is still interesting to get results for all the tests to better understand what's still OK and what's not after a modification.
Now we only exit earlier if the basic tests are failing: no ping going through namespaces or unable to transfer data on the loopback interface.
Acked-by: Paolo Abeni pabeni@redhat.com Signed-off-by: Matthieu Baerts matthieu.baerts@tessares.net --- tools/testing/selftests/net/mptcp/mptcp_connect.sh | 27 ++++++++++++++++------ 1 file changed, 20 insertions(+), 7 deletions(-)
diff --git a/tools/testing/selftests/net/mptcp/mptcp_connect.sh b/tools/testing/selftests/net/mptcp/mptcp_connect.sh index bbae40882bfa..0f060af01a84 100755 --- a/tools/testing/selftests/net/mptcp/mptcp_connect.sh +++ b/tools/testing/selftests/net/mptcp/mptcp_connect.sh @@ -7,6 +7,7 @@ time_start=$(date +%s)
optstring="S:R:d:e:l:r:h4cm:f:tC" ret=0 +final_ret=0 sin="" sout="" cin_disconnect="" @@ -837,14 +838,25 @@ display_time() echo "Time: ${time_run} seconds" }
-stop_if_error() +log_if_error() { local msg="$1"
if [ ${ret} -ne 0 ]; then echo "FAIL: ${msg}" 1>&2 + + final_ret=${ret} + ret=0 + + return ${final_ret} + fi +} + +stop_if_error() +{ + if ! log_if_error "${@}"; then display_time - exit ${ret} + exit ${final_ret} fi }
@@ -934,23 +946,24 @@ for sender in $ns1 $ns2 $ns3 $ns4;do run_tests "$ns4" $sender 10.0.3.1 run_tests "$ns4" $sender dead:beef:3::1
- stop_if_error "Tests with $sender as a sender have failed" + log_if_error "Tests with $sender as a sender have failed" done
run_tests_peekmode "saveWithPeek" run_tests_peekmode "saveAfterPeek" -stop_if_error "Tests with peek mode have failed" +log_if_error "Tests with peek mode have failed"
# MPTFO (MultiPath TCP Fatopen tests) run_tests_mptfo -stop_if_error "Tests with MPTFO have failed" +log_if_error "Tests with MPTFO have failed"
# connect to ns4 ip address, ns2 should intercept/proxy run_test_transparent 10.0.3.1 "tproxy ipv4" run_test_transparent dead:beef:3::1 "tproxy ipv6" -stop_if_error "Tests with tproxy have failed" +log_if_error "Tests with tproxy have failed"
run_tests_disconnect +log_if_error "Tests of the full disconnection have failed"
display_time -exit $ret +exit ${final_ret}
No more tests were executed after a failure but it is still interesting to get results for all the tests to better understand what's still OK and what's not after a modification.
Now we only exit earlier if the two connections cannot be established.
Acked-by: Paolo Abeni pabeni@redhat.com Signed-off-by: Matthieu Baerts matthieu.baerts@tessares.net --- tools/testing/selftests/net/mptcp/userspace_pm.sh | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-)
diff --git a/tools/testing/selftests/net/mptcp/userspace_pm.sh b/tools/testing/selftests/net/mptcp/userspace_pm.sh index b180133a30af..c57ce4df4973 100755 --- a/tools/testing/selftests/net/mptcp/userspace_pm.sh +++ b/tools/testing/selftests/net/mptcp/userspace_pm.sh @@ -52,6 +52,7 @@ sec=$(date +%s) rndh=$(printf %x "$sec")-$(mktemp -u XXXXXX) ns1="ns1-$rndh" ns2="ns2-$rndh" +ret=0
print_title() { @@ -257,21 +258,21 @@ check_expected_one() # $@: all var names to check check_expected() { - local ret=0 + local rc=0 local var
for var in "${@}" do - check_expected_one "${var}" "${ret}" || ret=1 + check_expected_one "${var}" "${rc}" || rc=1 done
- if [ ${ret} -eq 0 ] + if [ ${rc} -eq 0 ] then stdbuf -o0 -e0 printf "[OK]\n" return 0 fi
- exit 1 + ret=1 }
verify_announce_event() @@ -323,7 +324,7 @@ test_announce() stdbuf -o0 -e0 printf "[OK]\n" else stdbuf -o0 -e0 printf "[FAIL]\n\ttype defined: %s\n" "${type}" - exit 1 + ret=1 fi
# ADD_ADDR from the client to server machine reusing the subflow port @@ -423,7 +424,7 @@ test_remove() stdbuf -o0 -e0 printf "[OK]\n" else stdbuf -o0 -e0 printf "[FAIL]\n" - exit 1 + ret=1 fi
# RM_ADDR using an invalid addr id should result in no action @@ -438,7 +439,7 @@ test_remove() stdbuf -o0 -e0 printf "[OK]\n" else stdbuf -o0 -e0 printf "[FAIL]\n" - exit 1 + ret=1 fi
# RM_ADDR from the client to server machine @@ -859,7 +860,7 @@ test_prio() [ -z "$count" ] && count=0 if [ $count != 1 ]; then stdbuf -o0 -e0 printf "[FAIL]\n\tCount != 1: %d\n" "${count}" - exit 1 + ret=1 else stdbuf -o0 -e0 printf "[OK]\n" fi @@ -870,7 +871,7 @@ test_prio() [ -z "$count" ] && count=0 if [ $count != 1 ]; then stdbuf -o0 -e0 printf "[FAIL]\n\tCount != 1: %d\n" "${count}" - exit 1 + ret=1 else stdbuf -o0 -e0 printf "[OK]\n" fi @@ -961,4 +962,4 @@ test_subflows_v4_v6_mix test_prio test_listener
-exit 0 +exit ${ret}
shellcheck recently helped to find an issue where a wrong variable name was used. It is then good to fix the other harmless issues in order to spot "real" ones later.
Here, three categories of warnings are ignored:
- SC2317: Command appears to be unreachable. The cleanup() function is invoke indirectly via the EXIT trap.
- SC2034: Variable appears unused. The check_expected_one() function takes the name of the variable in argument but it ends up reading the content: indirect usage.
- SC2086: Double quote to prevent globbing and word splitting. This is recommended but the current usage is correct and there is no need to do all these modifications to be compliant with this rule.
One error has been fixed with SC2181: Check exit code directly with e.g. 'if ! mycmd;', not indirectly with $?.
Acked-by: Paolo Abeni pabeni@redhat.com Signed-off-by: Matthieu Baerts matthieu.baerts@tessares.net --- tools/testing/selftests/net/mptcp/userspace_pm.sh | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/tools/testing/selftests/net/mptcp/userspace_pm.sh b/tools/testing/selftests/net/mptcp/userspace_pm.sh index c57ce4df4973..63d348693e98 100755 --- a/tools/testing/selftests/net/mptcp/userspace_pm.sh +++ b/tools/testing/selftests/net/mptcp/userspace_pm.sh @@ -1,6 +1,13 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0
+# Double quotes to prevent globbing and word splitting is recommended in new +# code but we accept it. +#shellcheck disable=SC2086 + +# Some variables are used below but indirectly, see check_expected_one() +#shellcheck disable=SC2034 + . "$(dirname "${0}")/mptcp_lib.sh"
mptcp_lib_check_mptcp @@ -11,8 +18,7 @@ if ! mptcp_lib_has_file '/proc/sys/net/mptcp/pm_type'; then exit ${KSFT_SKIP} fi
-ip -Version > /dev/null 2>&1 -if [ $? -ne 0 ];then +if ! ip -Version &> /dev/null; then echo "SKIP: Cannot not run test without ip tool" exit ${KSFT_SKIP} fi @@ -68,6 +74,8 @@ kill_wait() wait $1 2>/dev/null }
+# This function is used in the cleanup trap +#shellcheck disable=SC2317 cleanup() { print_title "Cleanup"
There are a few reasons to do that:
- When the tabs are not printed as 8 spaces, some results were not properly aligned
- Some lines printing the test name were very long due to the use of a lot of spaces/tabs at the end and stdbuf at the beginning.
- To reduce duplicated code, e.g. to print what has failed and set the status
But by centralising how the test results are printed, this also prepares future commits to avoid more duplicated code and ease the tracking of the different subtests.
Acked-by: Paolo Abeni pabeni@redhat.com Signed-off-by: Matthieu Baerts matthieu.baerts@tessares.net --- tools/testing/selftests/net/mptcp/userspace_pm.sh | 141 ++++++++++++---------- 1 file changed, 79 insertions(+), 62 deletions(-)
diff --git a/tools/testing/selftests/net/mptcp/userspace_pm.sh b/tools/testing/selftests/net/mptcp/userspace_pm.sh index 63d348693e98..6b733b97d358 100755 --- a/tools/testing/selftests/net/mptcp/userspace_pm.sh +++ b/tools/testing/selftests/net/mptcp/userspace_pm.sh @@ -65,6 +65,38 @@ print_title() stdbuf -o0 -e0 printf "INFO: %s\n" "${1}" }
+# $1: test name +print_test() +{ + stdbuf -o0 -e0 printf "%-63s" "${1}" +} + +print_results() +{ + stdbuf -o0 -e0 printf "[%s]\n" "${1}" +} + +test_pass() +{ + print_results " OK " +} + +test_skip() +{ + print_results "SKIP" +} + +# $1: msg +test_fail() +{ + print_results "FAIL" + ret=1 + + if [ -n "${1}" ]; then + stdbuf -o0 -e0 printf "\t%s\n" "${1}" + fi +} + kill_wait() { [ $1 -eq 0 ] && return 0 @@ -127,7 +159,8 @@ ip -net "$ns2" addr add dead:beef:2::2/64 dev ns2eth1 nodad ip -net "$ns2" link set ns2eth1 up
print_title "Init" -stdbuf -o0 -e0 printf "Created network namespaces ns1, ns2 \t\t\t[OK]\n" +print_test "Created network namespaces ns1, ns2" +test_pass
make_file() { @@ -212,16 +245,13 @@ make_connection() server_serverside=$(grep "type:1," "$server_evts" | sed --unbuffered -n 's/.*(server_side:)([[:digit:]]*).*$/\2/p;q')
- stdbuf -o0 -e0 printf "Established IP%s MPTCP Connection ns2 => ns1 \t\t" $is_v6 + print_test "Established IP${is_v6} MPTCP Connection ns2 => ns1" if [ "$client_token" != "" ] && [ "$server_token" != "" ] && [ "$client_serverside" = 0 ] && [ "$server_serverside" = 1 ] then - stdbuf -o0 -e0 printf "[OK]\n" + test_pass else - stdbuf -o0 -e0 printf "[FAIL]\n" - stdbuf -o0 -e0 printf "\tExpected tokens (c:%s - s:%s) and server (c:%d - s:%d)\n" \ - "${client_token}" "${server_token}" \ - "${client_serverside}" "${server_serverside}" + test_fail "Expected tokens (c:${client_token} - s:${server_token}) and server (c:${client_serverside} - s:${server_serverside})" exit 1 fi
@@ -255,7 +285,7 @@ check_expected_one()
if [ "${prev_ret}" = "0" ] then - stdbuf -o0 -e0 printf "[FAIL]\n" + test_fail fi
stdbuf -o0 -e0 printf "\tExpected value for '%s': '%s', got '%s'.\n" \ @@ -276,11 +306,11 @@ check_expected()
if [ ${rc} -eq 0 ] then - stdbuf -o0 -e0 printf "[OK]\n" + test_pass return 0 fi
- ret=1 + return 1 }
verify_announce_event() @@ -326,13 +356,12 @@ test_announce()
local type type=$(sed --unbuffered -n 's/.*(type:)([[:digit:]]*).*$/\2/p;q' "$server_evts") - stdbuf -o0 -e0 printf "ADD_ADDR 10.0.2.2 (ns2) => ns1, invalid token \t\t" + print_test "ADD_ADDR 10.0.2.2 (ns2) => ns1, invalid token" if [ "$type" = "" ] then - stdbuf -o0 -e0 printf "[OK]\n" + test_pass else - stdbuf -o0 -e0 printf "[FAIL]\n\ttype defined: %s\n" "${type}" - ret=1 + test_fail "type defined: ${type}" fi
# ADD_ADDR from the client to server machine reusing the subflow port @@ -340,7 +369,7 @@ test_announce() ip netns exec "$ns2"\ ./pm_nl_ctl ann 10.0.2.2 token "$client4_token" id $client_addr_id dev\ ns2eth1 > /dev/null 2>&1 - stdbuf -o0 -e0 printf "ADD_ADDR id:%d 10.0.2.2 (ns2) => ns1, reuse port \t\t" $client_addr_id + print_test "ADD_ADDR id:${client_addr_id} 10.0.2.2 (ns2) => ns1, reuse port" sleep 0.5 verify_announce_event $server_evts $ANNOUNCED $server4_token "10.0.2.2" $client_addr_id \ "$client4_port" @@ -349,7 +378,7 @@ test_announce() :>"$server_evts" ip netns exec "$ns2" ./pm_nl_ctl ann\ dead:beef:2::2 token "$client6_token" id $client_addr_id dev ns2eth1 > /dev/null 2>&1 - stdbuf -o0 -e0 printf "ADD_ADDR6 id:%d dead:beef:2::2 (ns2) => ns1, reuse port\t\t" $client_addr_id + print_test "ADD_ADDR6 id:${client_addr_id} dead:beef:2::2 (ns2) => ns1, reuse port" sleep 0.5 verify_announce_event "$server_evts" "$ANNOUNCED" "$server6_token" "dead:beef:2::2"\ "$client_addr_id" "$client6_port" "v6" @@ -359,7 +388,7 @@ test_announce() client_addr_id=$((client_addr_id+1)) ip netns exec "$ns2" ./pm_nl_ctl ann 10.0.2.2 token "$client4_token" id\ $client_addr_id dev ns2eth1 port $new4_port > /dev/null 2>&1 - stdbuf -o0 -e0 printf "ADD_ADDR id:%d 10.0.2.2 (ns2) => ns1, new port \t\t\t" $client_addr_id + print_test "ADD_ADDR id:${client_addr_id} 10.0.2.2 (ns2) => ns1, new port" sleep 0.5 verify_announce_event "$server_evts" "$ANNOUNCED" "$server4_token" "10.0.2.2"\ "$client_addr_id" "$new4_port" @@ -370,7 +399,7 @@ test_announce() # ADD_ADDR from the server to client machine reusing the subflow port ip netns exec "$ns1" ./pm_nl_ctl ann 10.0.2.1 token "$server4_token" id\ $server_addr_id dev ns1eth2 > /dev/null 2>&1 - stdbuf -o0 -e0 printf "ADD_ADDR id:%d 10.0.2.1 (ns1) => ns2, reuse port \t\t" $server_addr_id + print_test "ADD_ADDR id:${server_addr_id} 10.0.2.1 (ns1) => ns2, reuse port" sleep 0.5 verify_announce_event "$client_evts" "$ANNOUNCED" "$client4_token" "10.0.2.1"\ "$server_addr_id" "$app4_port" @@ -379,7 +408,7 @@ test_announce() :>"$client_evts" ip netns exec "$ns1" ./pm_nl_ctl ann dead:beef:2::1 token "$server6_token" id\ $server_addr_id dev ns1eth2 > /dev/null 2>&1 - stdbuf -o0 -e0 printf "ADD_ADDR6 id:%d dead:beef:2::1 (ns1) => ns2, reuse port\t\t" $server_addr_id + print_test "ADD_ADDR6 id:${server_addr_id} dead:beef:2::1 (ns1) => ns2, reuse port" sleep 0.5 verify_announce_event "$client_evts" "$ANNOUNCED" "$client6_token" "dead:beef:2::1"\ "$server_addr_id" "$app6_port" "v6" @@ -389,7 +418,7 @@ test_announce() server_addr_id=$((server_addr_id+1)) ip netns exec "$ns1" ./pm_nl_ctl ann 10.0.2.1 token "$server4_token" id\ $server_addr_id dev ns1eth2 port $new4_port > /dev/null 2>&1 - stdbuf -o0 -e0 printf "ADD_ADDR id:%d 10.0.2.1 (ns1) => ns2, new port \t\t\t" $server_addr_id + print_test "ADD_ADDR id:${server_addr_id} 10.0.2.1 (ns1) => ns2, new port" sleep 0.5 verify_announce_event "$client_evts" "$ANNOUNCED" "$client4_token" "10.0.2.1"\ "$server_addr_id" "$new4_port" @@ -423,39 +452,34 @@ test_remove() local invalid_token=$(( client4_token - 1 )) ip netns exec "$ns2" ./pm_nl_ctl rem token $invalid_token id\ $client_addr_id > /dev/null 2>&1 - stdbuf -o0 -e0 printf "RM_ADDR id:%d ns2 => ns1, invalid token \t"\ - $client_addr_id + print_test "RM_ADDR id:${client_addr_id} ns2 => ns1, invalid token" local type type=$(sed --unbuffered -n 's/.*(type:)([[:digit:]]*).*$/\2/p;q' "$server_evts") if [ "$type" = "" ] then - stdbuf -o0 -e0 printf "[OK]\n" + test_pass else - stdbuf -o0 -e0 printf "[FAIL]\n" - ret=1 + test_fail fi
# RM_ADDR using an invalid addr id should result in no action local invalid_id=$(( client_addr_id + 1 )) ip netns exec "$ns2" ./pm_nl_ctl rem token "$client4_token" id\ $invalid_id > /dev/null 2>&1 - stdbuf -o0 -e0 printf "RM_ADDR id:%d ns2 => ns1, invalid id \t"\ - $invalid_id + print_test "RM_ADDR id:${invalid_id} ns2 => ns1, invalid id" type=$(sed --unbuffered -n 's/.*(type:)([[:digit:]]*).*$/\2/p;q' "$server_evts") if [ "$type" = "" ] then - stdbuf -o0 -e0 printf "[OK]\n" + test_pass else - stdbuf -o0 -e0 printf "[FAIL]\n" - ret=1 + test_fail fi
# RM_ADDR from the client to server machine :>"$server_evts" ip netns exec "$ns2" ./pm_nl_ctl rem token "$client4_token" id\ $client_addr_id > /dev/null 2>&1 - stdbuf -o0 -e0 printf "RM_ADDR id:%d ns2 => ns1 \t"\ - $client_addr_id + print_test "RM_ADDR id:${client_addr_id} ns2 => ns1" sleep 0.5 verify_remove_event "$server_evts" "$REMOVED" "$server4_token" "$client_addr_id"
@@ -464,8 +488,7 @@ test_remove() client_addr_id=$(( client_addr_id - 1 )) ip netns exec "$ns2" ./pm_nl_ctl rem token "$client4_token" id\ $client_addr_id > /dev/null 2>&1 - stdbuf -o0 -e0 printf "RM_ADDR id:%d ns2 => ns1 \t"\ - $client_addr_id + print_test "RM_ADDR id:${client_addr_id} ns2 => ns1" sleep 0.5 verify_remove_event "$server_evts" "$REMOVED" "$server4_token" "$client_addr_id"
@@ -473,8 +496,7 @@ test_remove() :>"$server_evts" ip netns exec "$ns2" ./pm_nl_ctl rem token "$client6_token" id\ $client_addr_id > /dev/null 2>&1 - stdbuf -o0 -e0 printf "RM_ADDR6 id:%d ns2 => ns1 \t"\ - $client_addr_id + print_test "RM_ADDR6 id:${client_addr_id} ns2 => ns1" sleep 0.5 verify_remove_event "$server_evts" "$REMOVED" "$server6_token" "$client_addr_id"
@@ -484,8 +506,7 @@ test_remove() # RM_ADDR from the server to client machine ip netns exec "$ns1" ./pm_nl_ctl rem token "$server4_token" id\ $server_addr_id > /dev/null 2>&1 - stdbuf -o0 -e0 printf "RM_ADDR id:%d ns1 => ns2 \t"\ - $server_addr_id + print_test "RM_ADDR id:${server_addr_id} ns1 => ns2" sleep 0.5 verify_remove_event "$client_evts" "$REMOVED" "$client4_token" "$server_addr_id"
@@ -494,7 +515,7 @@ test_remove() server_addr_id=$(( server_addr_id - 1 )) ip netns exec "$ns1" ./pm_nl_ctl rem token "$server4_token" id\ $server_addr_id > /dev/null 2>&1 - stdbuf -o0 -e0 printf "RM_ADDR id:%d ns1 => ns2 \t" $server_addr_id + print_test "RM_ADDR id:${server_addr_id} ns1 => ns2" sleep 0.5 verify_remove_event "$client_evts" "$REMOVED" "$client4_token" "$server_addr_id"
@@ -502,7 +523,7 @@ test_remove() :>"$client_evts" ip netns exec "$ns1" ./pm_nl_ctl rem token "$server6_token" id\ $server_addr_id > /dev/null 2>&1 - stdbuf -o0 -e0 printf "RM_ADDR6 id:%d ns1 => ns2 \t" $server_addr_id + print_test "RM_ADDR6 id:${server_addr_id} ns1 => ns2" sleep 0.5 verify_remove_event "$client_evts" "$REMOVED" "$client6_token" "$server_addr_id" } @@ -529,25 +550,24 @@ verify_subflow_events() local dport local locid local remid + local info + + info="${e_saddr} (${e_from}) => ${e_daddr} (${e_to})"
if [ "$e_type" = "$SUB_ESTABLISHED" ] then if [ "$e_family" = "$AF_INET6" ] then - stdbuf -o0 -e0 printf "CREATE_SUBFLOW6 %s (%s) => %s (%s) "\ - "$e_saddr" "$e_from" "$e_daddr" "$e_to" + print_test "CREATE_SUBFLOW6 ${info}" else - stdbuf -o0 -e0 printf "CREATE_SUBFLOW %s (%s) => %s (%s) \t"\ - "$e_saddr" "$e_from" "$e_daddr" "$e_to" + print_test "CREATE_SUBFLOW ${info}" fi else if [ "$e_family" = "$AF_INET6" ] then - stdbuf -o0 -e0 printf "DESTROY_SUBFLOW6 %s (%s) => %s (%s) "\ - "$e_saddr" "$e_from" "$e_daddr" "$e_to" + print_test "DESTROY_SUBFLOW6 ${info}" else - stdbuf -o0 -e0 printf "DESTROY_SUBFLOW %s (%s) => %s (%s) \t"\ - "$e_saddr" "$e_from" "$e_daddr" "$e_to" + print_test "DESTROY_SUBFLOW ${info}" fi fi
@@ -818,7 +838,7 @@ test_subflows_v4_v6_mix() :>"$client_evts" ip netns exec "$ns1" ./pm_nl_ctl ann 10.0.2.1 token "$server6_token" id\ $server_addr_id dev ns1eth2 > /dev/null 2>&1 - stdbuf -o0 -e0 printf "ADD_ADDR4 id:%d 10.0.2.1 (ns1) => ns2, reuse port\t\t" $server_addr_id + print_test "ADD_ADDR4 id:${server_addr_id} 10.0.2.1 (ns1) => ns2, reuse port" sleep 0.5 verify_announce_event "$client_evts" "$ANNOUNCED" "$client6_token" "10.0.2.1"\ "$server_addr_id" "$app6_port" @@ -863,25 +883,23 @@ test_prio() sleep 0.5
# Check TX - stdbuf -o0 -e0 printf "MP_PRIO TX \t" + print_test "MP_PRIO TX" count=$(ip netns exec "$ns2" nstat -as | grep MPTcpExtMPPrioTx | awk '{print $2}') [ -z "$count" ] && count=0 if [ $count != 1 ]; then - stdbuf -o0 -e0 printf "[FAIL]\n\tCount != 1: %d\n" "${count}" - ret=1 + test_fail "Count != 1: ${count}" else - stdbuf -o0 -e0 printf "[OK]\n" + test_pass fi
# Check RX - stdbuf -o0 -e0 printf "MP_PRIO RX \t" + print_test "MP_PRIO RX" count=$(ip netns exec "$ns1" nstat -as | grep MPTcpExtMPPrioRx | awk '{print $2}') [ -z "$count" ] && count=0 if [ $count != 1 ]; then - stdbuf -o0 -e0 printf "[FAIL]\n\tCount != 1: %d\n" "${count}" - ret=1 + test_fail "Count != 1: ${count}" else - stdbuf -o0 -e0 printf "[OK]\n" + test_pass fi }
@@ -898,11 +916,9 @@ verify_listener_events() local sport
if [ $e_type = $LISTENER_CREATED ]; then - stdbuf -o0 -e0 printf "CREATE_LISTENER %s:%s\t\t\t\t\t"\ - $e_saddr $e_sport + print_test "CREATE_LISTENER $e_saddr:$e_sport" elif [ $e_type = $LISTENER_CLOSED ]; then - stdbuf -o0 -e0 printf "CLOSE_LISTENER %s:%s\t\t\t\t\t"\ - $e_saddr $e_sport + print_test "CLOSE_LISTENER $e_saddr:$e_sport" fi
type=$(grep "type:$e_type," $evt | @@ -927,7 +943,8 @@ test_listener() print_title "Listener tests"
if ! mptcp_lib_kallsyms_has "mptcp_event_pm_listener$"; then - stdbuf -o0 -e0 printf "LISTENER events \t[SKIP] Not supported\n" + print_test "LISTENER events" + test_skip return fi
In this selftest, "printf" is always used with "stdbuf".
With a new helper, it is possible to call "stdbuf" only from one place. This makes the code a bit clearer to read.
Acked-by: Paolo Abeni pabeni@redhat.com Signed-off-by: Matthieu Baerts matthieu.baerts@tessares.net --- tools/testing/selftests/net/mptcp/userspace_pm.sh | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
diff --git a/tools/testing/selftests/net/mptcp/userspace_pm.sh b/tools/testing/selftests/net/mptcp/userspace_pm.sh index 6b733b97d358..568ddee1d102 100755 --- a/tools/testing/selftests/net/mptcp/userspace_pm.sh +++ b/tools/testing/selftests/net/mptcp/userspace_pm.sh @@ -59,21 +59,24 @@ rndh=$(printf %x "$sec")-$(mktemp -u XXXXXX) ns1="ns1-$rndh" ns2="ns2-$rndh" ret=0 +_printf() { + stdbuf -o0 -e0 printf "${@}" +}
print_title() { - stdbuf -o0 -e0 printf "INFO: %s\n" "${1}" + _printf "INFO: %s\n" "${1}" }
# $1: test name print_test() { - stdbuf -o0 -e0 printf "%-63s" "${1}" + _printf "%-63s" "${1}" }
print_results() { - stdbuf -o0 -e0 printf "[%s]\n" "${1}" + _printf "[%s]\n" "${1}" }
test_pass() @@ -93,7 +96,7 @@ test_fail() ret=1
if [ -n "${1}" ]; then - stdbuf -o0 -e0 printf "\t%s\n" "${1}" + _printf "\t%s\n" "${1}" fi }
@@ -127,7 +130,7 @@ cleanup()
rm -rf $file $client_evts $server_evts
- stdbuf -o0 -e0 printf "Done\n" + _printf "Done\n" }
trap cleanup EXIT @@ -288,7 +291,7 @@ check_expected_one() test_fail fi
- stdbuf -o0 -e0 printf "\tExpected value for '%s': '%s', got '%s'.\n" \ + _printf "\tExpected value for '%s': '%s', got '%s'.\n" \ "${var}" "${!exp}" "${!var}" return 1 }
The current selftests infrastructure formats the results in TAP 13. This version doesn't support subtests and only the end result of each selftest is taken into account. It means that a single issue in a subtest of a selftest containing multiple subtests forces the whole selftest to be marked as failed. It also means that subtests results are not tracked by CIs executing selftests.
MPTCP selftests run hundreds of various subtests. It is then important to track each of them and not one result per selftest.
It is particularly interesting to do that when validating stable kernels with the last version of the test suite: tests might fail because a feature is not supported but the test didn't skip that part. In this case, if subtests are not tracked, the whole selftest will be marked as failed making the other subtests useless because their results are ignored.
This patch adds some helpers in mptcp_lib.sh to be able to easily format subtests results in TAP in the different MPTCP selftests.
Closes: https://github.com/multipath-tcp/mptcp_net-next/issues/368 Acked-by: Paolo Abeni pabeni@redhat.com Signed-off-by: Matthieu Baerts matthieu.baerts@tessares.net --- tools/testing/selftests/net/mptcp/mptcp_lib.sh | 66 ++++++++++++++++++++++++++ 1 file changed, 66 insertions(+)
diff --git a/tools/testing/selftests/net/mptcp/mptcp_lib.sh b/tools/testing/selftests/net/mptcp/mptcp_lib.sh index f32045b23b89..b1a0fdd0408b 100644 --- a/tools/testing/selftests/net/mptcp/mptcp_lib.sh +++ b/tools/testing/selftests/net/mptcp/mptcp_lib.sh @@ -1,8 +1,12 @@ #! /bin/bash # SPDX-License-Identifier: GPL-2.0
+readonly KSFT_PASS=0 readonly KSFT_FAIL=1 readonly KSFT_SKIP=4 +readonly KSFT_TEST=$(basename "${0}" | sed 's/.sh$//g') + +MPTCP_LIB_SUBTESTS=()
# SELFTESTS_MPTCP_LIB_EXPECT_ALL_FEATURES env var can be set when validating all # features using the last version of the kernel and the selftests to make sure @@ -102,3 +106,65 @@ mptcp_lib_kversion_ge() {
mptcp_lib_fail_if_expected_feature "kernel version ${1} lower than ${v}" } + +__mptcp_lib_result_add() { + local result="${1}" + shift + + local id=$((${#MPTCP_LIB_SUBTESTS[@]} + 1)) + + MPTCP_LIB_SUBTESTS+=("${result} ${id} - ${KSFT_TEST}: ${*}") +} + +# $1: test name +mptcp_lib_result_pass() { + __mptcp_lib_result_add "ok" "${1}" +} + +# $1: test name +mptcp_lib_result_fail() { + __mptcp_lib_result_add "not ok" "${1}" +} + +# $1: test name +mptcp_lib_result_skip() { + __mptcp_lib_result_add "ok" "${1} # SKIP" +} + +# $1: result code ; $2: test name +mptcp_lib_result_code() { + local ret="${1}" + local name="${2}" + + case "${ret}" in + "${KSFT_PASS}") + mptcp_lib_result_pass "${name}" + ;; + "${KSFT_FAIL}") + mptcp_lib_result_fail "${name}" + ;; + "${KSFT_SKIP}") + mptcp_lib_result_skip "${name}" + ;; + *) + echo "ERROR: wrong result code: ${ret}" + exit ${KSFT_FAIL} + ;; + esac +} + +mptcp_lib_result_print_all_tap() { + local subtest + + if [ ${#MPTCP_LIB_SUBTESTS[@]} -eq 0 ] || + [ "${SELFTESTS_MPTCP_LIB_NO_TAP:-}" = "1" ]; then + return + fi + + printf "\nTAP version 13\n" + printf "1..%d\n" "${#MPTCP_LIB_SUBTESTS[@]}" + + for subtest in "${MPTCP_LIB_SUBTESTS[@]}"; do + printf "%s\n" "${subtest}" + done +}
The current selftests infrastructure formats the results in TAP 13. This version doesn't support subtests and only the end result of each selftest is taken into account. It means that a single issue in a subtest of a selftest containing multiple subtests forces the whole selftest to be marked as failed. It also means that subtests results are not tracked by CIs executing selftests.
MPTCP selftests run hundreds of various subtests. It is then important to track each of them and not one result per selftest.
It is particularly interesting to do that when validating stable kernels with the last version of the test suite: tests might fail because a feature is not supported but the test didn't skip that part. In this case, if subtests are not tracked, the whole selftest will be marked as failed making the other subtests useless because their results are ignored.
This patch formats subtests results in TAP in mptcp_connect.sh selftest.
Link: https://github.com/multipath-tcp/mptcp_net-next/issues/368 Acked-by: Paolo Abeni pabeni@redhat.com Signed-off-by: Matthieu Baerts matthieu.baerts@tessares.net --- tools/testing/selftests/net/mptcp/mptcp_connect.sh | 39 ++++++++++++++++++++-- 1 file changed, 36 insertions(+), 3 deletions(-)
diff --git a/tools/testing/selftests/net/mptcp/mptcp_connect.sh b/tools/testing/selftests/net/mptcp/mptcp_connect.sh index 0f060af01a84..b1fc8afd072d 100755 --- a/tools/testing/selftests/net/mptcp/mptcp_connect.sh +++ b/tools/testing/selftests/net/mptcp/mptcp_connect.sh @@ -129,6 +129,7 @@ ns3="ns3-$rndh" ns4="ns4-$rndh"
TEST_COUNT=0 +TEST_GROUP=""
cleanup() { @@ -286,6 +287,7 @@ check_mptcp_disabled() # net.mptcp.enabled should be enabled by default if [ "$(ip netns exec ${disabled_ns} sysctl net.mptcp.enabled | awk '{ print $3 }')" -ne 1 ]; then echo -e "net.mptcp.enabled sysctl is not 1 by default\t\t[ FAIL ]" + mptcp_lib_result_fail "net.mptcp.enabled sysctl is not 1 by default" ret=1 return 1 fi @@ -298,11 +300,13 @@ check_mptcp_disabled()
if [ ${err} -eq 0 ]; then echo -e "New MPTCP socket cannot be blocked via sysctl\t\t[ FAIL ]" + mptcp_lib_result_fail "New MPTCP socket cannot be blocked via sysctl" ret=1 return 1 fi
echo -e "New MPTCP socket can be blocked via sysctl\t\t[ OK ]" + mptcp_lib_result_pass "New MPTCP socket can be blocked via sysctl" return 0 }
@@ -318,14 +322,16 @@ do_ping() local connector_ns="$2" local connect_addr="$3" local ping_args="-q -c 1" + local rc=0
if is_v6 "${connect_addr}"; then $ipv6 || return 0 ping_args="${ping_args} -6" fi
- ip netns exec ${connector_ns} ping ${ping_args} $connect_addr >/dev/null - if [ $? -ne 0 ] ; then + ip netns exec ${connector_ns} ping ${ping_args} $connect_addr >/dev/null || rc=1 + + if [ $rc -ne 0 ] ; then echo "$listener_ns -> $connect_addr connectivity [ FAIL ]" 1>&2 ret=1
@@ -404,7 +410,9 @@ do_transfer()
local addr_port addr_port=$(printf "%s:%d" ${connect_addr} ${port}) - printf "%.3s %-5s -> %.3s (%-20s) %-5s\t" ${connector_ns} ${cl_proto} ${listener_ns} ${addr_port} ${srv_proto} + local result_msg + result_msg="$(printf "%.3s %-5s -> %.3s (%-20s) %-5s" ${connector_ns} ${cl_proto} ${listener_ns} ${addr_port} ${srv_proto})" + printf "%s\t" "${result_msg}"
if $capture; then local capuser @@ -479,6 +487,7 @@ do_transfer()
local duration duration=$((stop-start)) + result_msg+=" # time=${duration}ms" printf "(duration %05sms) " "${duration}" if [ ${rets} -ne 0 ] || [ ${retc} -ne 0 ]; then echo "[ FAIL ] client exit code $retc, server $rets" 1>&2 @@ -491,6 +500,7 @@ do_transfer()
echo cat "$capout" + mptcp_lib_result_fail "${TEST_GROUP}: ${result_msg}" return 1 fi
@@ -550,6 +560,9 @@ do_transfer()
if [ $retc -eq 0 ] && [ $rets -eq 0 ]; then printf "[ OK ]" + mptcp_lib_result_pass "${TEST_GROUP}: ${result_msg}" + else + mptcp_lib_result_fail "${TEST_GROUP}: ${result_msg}" fi
if [ $cookies -eq 2 ];then @@ -692,6 +705,8 @@ run_test_transparent() local lret=0 local r6flag=""
+ TEST_GROUP="${msg}" + # skip if we don't want v6 if ! $ipv6 && is_v6 "${connect_addr}"; then return 0 @@ -703,6 +718,7 @@ run_test_transparent() # checking for a specific kernel version. if ! mptcp_lib_kallsyms_has "T __ip_sock_set_tos$"; then echo "INFO: ${msg} not supported by the kernel: SKIP" + mptcp_lib_result_skip "${TEST_GROUP}" return fi
@@ -720,6 +736,7 @@ EOF if [ $? -ne 0 ]; then echo "SKIP: $msg, could not load nft ruleset" mptcp_lib_fail_if_expected_feature "nft rules" + mptcp_lib_result_skip "${TEST_GROUP}" return fi
@@ -736,6 +753,7 @@ EOF ip netns exec "$listener_ns" nft flush ruleset echo "SKIP: $msg, ip $r6flag rule failed" mptcp_lib_fail_if_expected_feature "ip rule" + mptcp_lib_result_skip "${TEST_GROUP}" return fi
@@ -745,6 +763,7 @@ EOF ip -net "$listener_ns" $r6flag rule del fwmark 1 lookup 100 echo "SKIP: $msg, ip route add local $local_addr failed" mptcp_lib_fail_if_expected_feature "ip route" + mptcp_lib_result_skip "${TEST_GROUP}" return fi
@@ -774,6 +793,7 @@ run_tests_peekmode() { local peekmode="$1"
+ TEST_GROUP="peek mode: ${peekmode}" echo "INFO: with peek mode: ${peekmode}" run_tests_lo "$ns1" "$ns1" 10.0.1.1 1 "-P ${peekmode}" run_tests_lo "$ns1" "$ns1" dead:beef:1::1 1 "-P ${peekmode}" @@ -781,8 +801,11 @@ run_tests_peekmode()
run_tests_mptfo() { + TEST_GROUP="MPTFO" + if ! mptcp_lib_kallsyms_has "mptcp_fastopen_"; then echo "INFO: TFO not supported by the kernel: SKIP" + mptcp_lib_result_skip "${TEST_GROUP}" return fi
@@ -806,8 +829,11 @@ run_tests_disconnect() local old_cin=$cin local old_sin=$sin
+ TEST_GROUP="full disconnect" + if ! mptcp_lib_kallsyms_has "mptcp_pm_data_reset$"; then echo "INFO: Full disconnect not supported: SKIP" + mptcp_lib_result_skip "${TEST_GROUP}" return fi
@@ -856,6 +882,7 @@ stop_if_error() { if ! log_if_error "${@}"; then display_time + mptcp_lib_result_print_all_tap exit ${final_ret} fi } @@ -886,6 +913,8 @@ for sender in "$ns1" "$ns2" "$ns3" "$ns4";do do_ping "$ns4" $sender dead:beef:3::1 done
+mptcp_lib_result_code "${ret}" "ping tests" + stop_if_error "Could not even run ping tests"
[ -n "$tc_loss" ] && tc -net "$ns2" qdisc add dev ns2eth3 root netem loss random $tc_loss delay ${tc_delay}ms @@ -915,12 +944,15 @@ echo "on ns3eth4"
tc -net "$ns3" qdisc add dev ns3eth4 root netem delay ${reorder_delay}ms $tc_reorder
+TEST_GROUP="loopback v4" run_tests_lo "$ns1" "$ns1" 10.0.1.1 1 stop_if_error "Could not even run loopback test"
+TEST_GROUP="loopback v6" run_tests_lo "$ns1" "$ns1" dead:beef:1::1 1 stop_if_error "Could not even run loopback v6 test"
+TEST_GROUP="multihosts" for sender in $ns1 $ns2 $ns3 $ns4;do # ns1<->ns2 is not subject to reordering/tc delays. Use it to test # mptcp syncookie support. @@ -966,4 +998,5 @@ run_tests_disconnect log_if_error "Tests of the full disconnection have failed"
display_time +mptcp_lib_result_print_all_tap exit ${final_ret}
The current selftests infrastructure formats the results in TAP 13. This version doesn't support subtests and only the end result of each selftest is taken into account. It means that a single issue in a subtest of a selftest containing multiple subtests forces the whole selftest to be marked as failed. It also means that subtests results are not tracked by CIs executing selftests.
MPTCP selftests run hundreds of various subtests. It is then important to track each of them and not one result per selftest.
It is particularly interesting to do that when validating stable kernels with the last version of the test suite: tests might fail because a feature is not supported but the test didn't skip that part. In this case, if subtests are not tracked, the whole selftest will be marked as failed making the other subtests useless because their results are ignored.
This patch formats subtests results in TAP in pm_netlink.sh selftest.
Link: https://github.com/multipath-tcp/mptcp_net-next/issues/368 Acked-by: Paolo Abeni pabeni@redhat.com Signed-off-by: Matthieu Baerts matthieu.baerts@tessares.net --- tools/testing/selftests/net/mptcp/pm_netlink.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/tools/testing/selftests/net/mptcp/pm_netlink.sh b/tools/testing/selftests/net/mptcp/pm_netlink.sh index d02e0d63a8f9..f32038fe1ee5 100755 --- a/tools/testing/selftests/net/mptcp/pm_netlink.sh +++ b/tools/testing/selftests/net/mptcp/pm_netlink.sh @@ -58,16 +58,19 @@ check() local out=`$cmd 2>$err` local cmd_ret=$?
- printf "%-50s %s" "$msg" + printf "%-50s" "$msg" if [ $cmd_ret -ne 0 ]; then echo "[FAIL] command execution '$cmd' stderr " cat $err + mptcp_lib_result_fail "${msg} # error ${cmd_ret}" ret=1 elif [ "$out" = "$expected" ]; then echo "[ OK ]" + mptcp_lib_result_pass "${msg}" else echo -n "[FAIL] " echo "expected '$expected' got '$out'" + mptcp_lib_result_fail "${msg} # different output" ret=1 fi } @@ -193,4 +196,5 @@ subflow 10.0.1.1" " (nofullmesh)" subflow,backup,fullmesh 10.0.1.1" " (backup,fullmesh)" fi
+mptcp_lib_result_print_all_tap exit $ret
The current selftests infrastructure formats the results in TAP 13. This version doesn't support subtests and only the end result of each selftest is taken into account. It means that a single issue in a subtest of a selftest containing multiple subtests forces the whole selftest to be marked as failed. It also means that subtests results are not tracked by CIs executing selftests.
MPTCP selftests run hundreds of various subtests. It is then important to track each of them and not one result per selftest.
It is particularly interesting to do that when validating stable kernels with the last version of the test suite: tests might fail because a feature is not supported but the test didn't skip that part. In this case, if subtests are not tracked, the whole selftest will be marked as failed making the other subtests useless because their results are ignored.
This patch formats subtests results in TAP in mptcp_join.sh selftest.
In this selftest and before starting each subtest, the 'reset' function is called. We can then check if the previous test has passed, failed or has been skipped from there.
Link: https://github.com/multipath-tcp/mptcp_net-next/issues/368 Acked-by: Paolo Abeni pabeni@redhat.com Signed-off-by: Matthieu Baerts matthieu.baerts@tessares.net --- tools/testing/selftests/net/mptcp/mptcp_join.sh | 37 +++++++++++++++++++++---- 1 file changed, 32 insertions(+), 5 deletions(-)
diff --git a/tools/testing/selftests/net/mptcp/mptcp_join.sh b/tools/testing/selftests/net/mptcp/mptcp_join.sh index b5eeea4c3efa..f336f86d652e 100755 --- a/tools/testing/selftests/net/mptcp/mptcp_join.sh +++ b/tools/testing/selftests/net/mptcp/mptcp_join.sh @@ -39,7 +39,9 @@ evts_ns1="" evts_ns2="" evts_ns1_pid=0 evts_ns2_pid=0 -stats_dumped=0 +last_test_failed=0 +last_test_skipped=0 +last_test_ignored=1
declare -A all_tests declare -a only_tests_ids @@ -101,7 +103,6 @@ init_partial() fi done
- stats_dumped=0 check_invert=0 validate_checksum=$checksum
@@ -216,6 +217,8 @@ mark_as_skipped()
print_title "[ skip ] ${msg}" printf "\n" + + last_test_skipped=1 }
# $@: condition @@ -248,14 +251,32 @@ skip_test() return 0 }
+append_prev_results() +{ + if [ ${last_test_failed} -eq 1 ]; then + mptcp_lib_result_fail "${TEST_NAME}" + elif [ ${last_test_skipped} -eq 1 ]; then + mptcp_lib_result_skip "${TEST_NAME}" + elif [ ${last_test_ignored} -ne 1 ]; then + mptcp_lib_result_pass "${TEST_NAME}" + fi + + last_test_failed=0 + last_test_skipped=0 + last_test_ignored=0 +} + # $1: test name reset() { + append_prev_results + TEST_NAME="${1}"
TEST_COUNT=$((TEST_COUNT+1))
if skip_test; then + last_test_ignored=1 return 1 fi
@@ -442,10 +463,13 @@ reset_with_tcp_filter() fail_test() { ret=1 - failed_tests[${TEST_COUNT}]="${TEST_NAME}"
- [ "${stats_dumped}" = 0 ] && dump_stats - stats_dumped=1 + # just in case a test is marked twice as failed + if [ ${last_test_failed} -eq 0 ]; then + failed_tests[${TEST_COUNT}]="${TEST_NAME}" + dump_stats + last_test_failed=1 + fi }
get_failed_tests_ids() @@ -3599,4 +3623,7 @@ if [ ${ret} -ne 0 ]; then echo fi
+append_prev_results +mptcp_lib_result_print_all_tap + exit $ret
The current selftests infrastructure formats the results in TAP 13. This version doesn't support subtests and only the end result of each selftest is taken into account. It means that a single issue in a subtest of a selftest containing multiple subtests forces the whole selftest to be marked as failed. It also means that subtests results are not tracked by CIs executing selftests.
MPTCP selftests run hundreds of various subtests. It is then important to track each of them and not one result per selftest.
It is particularly interesting to do that when validating stable kernels with the last version of the test suite: tests might fail because a feature is not supported but the test didn't skip that part. In this case, if subtests are not tracked, the whole selftest will be marked as failed making the other subtests useless because their results are ignored.
This patch formats subtests results in TAP in diag.sh selftest.
Link: https://github.com/multipath-tcp/mptcp_net-next/issues/368 Acked-by: Paolo Abeni pabeni@redhat.com Signed-off-by: Matthieu Baerts matthieu.baerts@tessares.net --- tools/testing/selftests/net/mptcp/diag.sh | 7 +++++++ 1 file changed, 7 insertions(+)
diff --git a/tools/testing/selftests/net/mptcp/diag.sh b/tools/testing/selftests/net/mptcp/diag.sh index fa9e09ad97d9..85a8ee9395b3 100755 --- a/tools/testing/selftests/net/mptcp/diag.sh +++ b/tools/testing/selftests/net/mptcp/diag.sh @@ -65,12 +65,15 @@ __chk_nr() if [ $nr != $expected ]; then if [ $nr = "$skip" ] && ! mptcp_lib_expect_all_features; then echo "[ skip ] Feature probably not supported" + mptcp_lib_result_skip "${msg}" else echo "[ fail ] expected $expected found $nr" + mptcp_lib_result_fail "${msg}" ret=$test_cnt fi else echo "[ ok ]" + mptcp_lib_result_pass "${msg}" fi test_cnt=$((test_cnt+1)) } @@ -111,12 +114,15 @@ wait_msk_nr() printf "%-50s" "$msg" if [ $i -ge $timeout ]; then echo "[ fail ] timeout while expecting $expected max $max last $nr" + mptcp_lib_result_fail "${msg} # timeout" ret=$test_cnt elif [ $nr != $expected ]; then echo "[ fail ] expected $expected found $nr" + mptcp_lib_result_fail "${msg} # unexpected result" ret=$test_cnt else echo "[ ok ]" + mptcp_lib_result_pass "${msg}" fi test_cnt=$((test_cnt+1)) } @@ -276,4 +282,5 @@ flush_pids
chk_msk_inuse 0 "....chk 0 msk in use after flush"
+mptcp_lib_result_print_all_tap exit $ret
The current selftests infrastructure formats the results in TAP 13. This version doesn't support subtests and only the end result of each selftest is taken into account. It means that a single issue in a subtest of a selftest containing multiple subtests forces the whole selftest to be marked as failed. It also means that subtests results are not tracked by CIs executing selftests.
MPTCP selftests run hundreds of various subtests. It is then important to track each of them and not one result per selftest.
It is particularly interesting to do that when validating stable kernels with the last version of the test suite: tests might fail because a feature is not supported but the test didn't skip that part. In this case, if subtests are not tracked, the whole selftest will be marked as failed making the other subtests useless because their results are ignored.
This patch formats subtests results in TAP in simult_flows.sh selftest.
Link: https://github.com/multipath-tcp/mptcp_net-next/issues/368 Acked-by: Paolo Abeni pabeni@redhat.com Signed-off-by: Matthieu Baerts matthieu.baerts@tessares.net --- tools/testing/selftests/net/mptcp/simult_flows.sh | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/tools/testing/selftests/net/mptcp/simult_flows.sh b/tools/testing/selftests/net/mptcp/simult_flows.sh index 36a3c9d92e20..ce9203b817f8 100755 --- a/tools/testing/selftests/net/mptcp/simult_flows.sh +++ b/tools/testing/selftests/net/mptcp/simult_flows.sh @@ -261,6 +261,7 @@ run_test() printf "%-60s" "$msg" do_transfer $small $large $time lret=$? + mptcp_lib_result_code "${lret}" "${msg}" if [ $lret -ne 0 ]; then ret=$lret [ $bail -eq 0 ] || exit $ret @@ -269,6 +270,7 @@ run_test() printf "%-60s" "$msg - reverse direction" do_transfer $large $small $time lret=$? + mptcp_lib_result_code "${lret}" "${msg}" if [ $lret -ne 0 ]; then ret=$lret [ $bail -eq 0 ] || exit $ret @@ -305,4 +307,6 @@ run_test 10 10 1 50 "balanced bwidth with unbalanced delay" run_test 30 10 0 0 "unbalanced bwidth" run_test 30 10 1 50 "unbalanced bwidth with unbalanced delay" run_test 30 10 50 1 "unbalanced bwidth with opposed, unbalanced delay" + +mptcp_lib_result_print_all_tap exit $ret
The current selftests infrastructure formats the results in TAP 13. This version doesn't support subtests and only the end result of each selftest is taken into account. It means that a single issue in a subtest of a selftest containing multiple subtests forces the whole selftest to be marked as failed. It also means that subtests results are not tracked by CIs executing selftests.
MPTCP selftests run hundreds of various subtests. It is then important to track each of them and not one result per selftest.
It is particularly interesting to do that when validating stable kernels with the last version of the test suite: tests might fail because a feature is not supported but the test didn't skip that part. In this case, if subtests are not tracked, the whole selftest will be marked as failed making the other subtests useless because their results are ignored.
This patch formats subtests results in TAP in mptcp_sockopt.sh selftest.
Link: https://github.com/multipath-tcp/mptcp_net-next/issues/368 Acked-by: Paolo Abeni pabeni@redhat.com Signed-off-by: Matthieu Baerts matthieu.baerts@tessares.net --- tools/testing/selftests/net/mptcp/mptcp_sockopt.sh | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)
diff --git a/tools/testing/selftests/net/mptcp/mptcp_sockopt.sh b/tools/testing/selftests/net/mptcp/mptcp_sockopt.sh index dc8d473fc82c..8c8694f21e7d 100755 --- a/tools/testing/selftests/net/mptcp/mptcp_sockopt.sh +++ b/tools/testing/selftests/net/mptcp/mptcp_sockopt.sh @@ -183,11 +183,13 @@ do_transfer()
local mptcp_connect="./mptcp_connect -r 20"
- local local_addr + local local_addr ip if is_v6 "${connect_addr}"; then local_addr="::" + ip=ipv6 else local_addr="0.0.0.0" + ip=ipv4 fi
cmsg="TIMESTAMPNS" @@ -223,6 +225,8 @@ do_transfer() echo -e "\nnetns ${connector_ns} socket stat for ${port}:" 1>&2 ip netns exec ${connector_ns} ss -Menita 1>&2 -o "dport = :$port"
+ mptcp_lib_result_fail "transfer ${ip}" + ret=1 return 1 fi @@ -236,9 +240,11 @@ do_transfer() fi
check_transfer $cin $sout "file received by server" - rets=$?
+ mptcp_lib_result_code "${retc}" "mark ${ip}" + mptcp_lib_result_code "${rets}" "transfer ${ip}" + if [ $retc -eq 0 ] && [ $rets -eq 0 ];then return 0 fi @@ -264,6 +270,7 @@ do_mptcp_sockopt_tests()
if ! mptcp_lib_kallsyms_has "mptcp_diag_fill_info$"; then echo "INFO: MPTCP sockopt not supported: SKIP" + mptcp_lib_result_skip "sockopt" return fi
@@ -272,18 +279,22 @@ do_mptcp_sockopt_tests()
if [ $lret -ne 0 ]; then echo "FAIL: SOL_MPTCP getsockopt" 1>&2 + mptcp_lib_result_fail "sockopt v4" ret=$lret return fi + mptcp_lib_result_pass "sockopt v4"
ip netns exec "$ns_sbox" ./mptcp_sockopt -6 lret=$?
if [ $lret -ne 0 ]; then echo "FAIL: SOL_MPTCP getsockopt (ipv6)" 1>&2 + mptcp_lib_result_fail "sockopt v6" ret=$lret return fi + mptcp_lib_result_pass "sockopt v6" }
run_tests() @@ -310,10 +321,12 @@ do_tcpinq_test() if [ $lret -ne 0 ];then ret=$lret echo "FAIL: mptcp_inq $@" 1>&2 + mptcp_lib_result_fail "TCP_INQ: $*" return $lret fi
echo "PASS: TCP_INQ cmsg/ioctl $@" + mptcp_lib_result_pass "TCP_INQ: $*" return $lret }
@@ -323,6 +336,7 @@ do_tcpinq_tests()
if ! mptcp_lib_kallsyms_has "mptcp_ioctl$"; then echo "INFO: TCP_INQ not supported: SKIP" + mptcp_lib_result_skip "TCP_INQ" return fi
@@ -367,4 +381,6 @@ if [ $ret -eq 0 ];then fi
do_tcpinq_tests + +mptcp_lib_result_print_all_tap exit $ret
The current selftests infrastructure formats the results in TAP 13. This version doesn't support subtests and only the end result of each selftest is taken into account. It means that a single issue in a subtest of a selftest containing multiple subtests forces the whole selftest to be marked as failed. It also means that subtests results are not tracked by CIs executing selftests.
MPTCP selftests run hundreds of various subtests. It is then important to track each of them and not one result per selftest.
It is particularly interesting to do that when validating stable kernels with the last version of the test suite: tests might fail because a feature is not supported but the test didn't skip that part. In this case, if subtests are not tracked, the whole selftest will be marked as failed making the other subtests useless because their results are ignored.
This patch formats subtests results in TAP in userspace_pm.sh selftest.
Link: https://github.com/multipath-tcp/mptcp_net-next/issues/368 Acked-by: Paolo Abeni pabeni@redhat.com Signed-off-by: Matthieu Baerts matthieu.baerts@tessares.net --- tools/testing/selftests/net/mptcp/userspace_pm.sh | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/tools/testing/selftests/net/mptcp/userspace_pm.sh b/tools/testing/selftests/net/mptcp/userspace_pm.sh index 568ddee1d102..23f8959a8ea8 100755 --- a/tools/testing/selftests/net/mptcp/userspace_pm.sh +++ b/tools/testing/selftests/net/mptcp/userspace_pm.sh @@ -59,6 +59,8 @@ rndh=$(printf %x "$sec")-$(mktemp -u XXXXXX) ns1="ns1-$rndh" ns2="ns2-$rndh" ret=0 +test_name="" + _printf() { stdbuf -o0 -e0 printf "${@}" } @@ -71,7 +73,9 @@ print_title() # $1: test name print_test() { - _printf "%-63s" "${1}" + test_name="${1}" + + _printf "%-63s" "${test_name}" }
print_results() @@ -82,11 +86,13 @@ print_results() test_pass() { print_results " OK " + mptcp_lib_result_pass "${test_name}" }
test_skip() { print_results "SKIP" + mptcp_lib_result_skip "${test_name}" }
# $1: msg @@ -98,6 +104,8 @@ test_fail() if [ -n "${1}" ]; then _printf "\t%s\n" "${1}" fi + + mptcp_lib_result_fail "${test_name}" }
kill_wait() @@ -255,6 +263,7 @@ make_connection() test_pass else test_fail "Expected tokens (c:${client_token} - s:${server_token}) and server (c:${client_serverside} - s:${server_serverside})" + mptcp_lib_result_print_all_tap exit 1 fi
@@ -990,4 +999,5 @@ test_subflows_v4_v6_mix test_prio test_listener
+mptcp_lib_result_print_all_tap exit ${ret}
Hello:
This series was applied to netdev/net-next.git (main) by David S. Miller davem@davemloft.net:
On Mon, 17 Jul 2023 15:21:20 +0200 you wrote:
The current selftests infrastructure formats the results in TAP 13. This version doesn't support subtests and only the end result of each selftest is taken into account. It means that a single issue in a subtest of a selftest containing multiple subtests forces the whole selftest to be marked as failed. It also means that subtests results are not tracked by CI executing selftests.
[...]
Here is the summary with links: - [net-next,01/13] selftests: mptcp: connect: don't stop if error https://git.kernel.org/netdev/net-next/c/edbc16c43b27 - [net-next,02/13] selftests: mptcp: userspace pm: don't stop if error https://git.kernel.org/netdev/net-next/c/e141c1e8e4c1 - [net-next,03/13] selftests: mptcp: userspace_pm: fix shellcheck warnings https://git.kernel.org/netdev/net-next/c/8320b1387a15 - [net-next,04/13] selftests: mptcp: userspace_pm: uniform results printing https://git.kernel.org/netdev/net-next/c/e198ad759273 - [net-next,05/13] selftests: mptcp: userspace_pm: reduce dup code around printf https://git.kernel.org/netdev/net-next/c/d8463d81652d - [net-next,06/13] selftests: mptcp: lib: format subtests results in TAP https://git.kernel.org/netdev/net-next/c/c4192967e62f - [net-next,07/13] selftests: mptcp: connect: format subtests results in TAP https://git.kernel.org/netdev/net-next/c/dd350f46e35e - [net-next,08/13] selftests: mptcp: pm_netlink: format subtests results in TAP https://git.kernel.org/netdev/net-next/c/d85555ac11f9 - [net-next,09/13] selftests: mptcp: join: format subtests results in TAP https://git.kernel.org/netdev/net-next/c/7f117cd37c61 - [net-next,10/13] selftests: mptcp: diag: format subtests results in TAP https://git.kernel.org/netdev/net-next/c/ce9902573652 - [net-next,11/13] selftests: mptcp: simult flows: format subtests results in TAP https://git.kernel.org/netdev/net-next/c/675d99338e7a - [net-next,12/13] selftests: mptcp: sockopt: format subtests results in TAP https://git.kernel.org/netdev/net-next/c/9e86a297796b - [net-next,13/13] selftests: mptcp: userspace_pm: format subtests results in TAP https://git.kernel.org/netdev/net-next/c/f589234e1af0
You are awesome, thank you!
linux-kselftest-mirror@lists.linaro.org
-
Matthieu Baerts -
patchwork-bot+netdevbpf＠kernel.org