July 2024 - Linux-kselftest-mirror

[PATCH net-next] selftests: net: py: support verbose printing, display executed commands

by Mohsin Bashir

Add verbosity support to show the commands executed while running tests. Enable verbosity if either an environment variable 'VERBOSE' is set to a non-zero number or it is defined in a config file under driver tests as discussed here: https://github.com/linux-netdev/nipa/wiki/Running-driver-tests. Signed-off-by: Mohsin Bashir <mohsin.bashr(a)gmail.com> --- tools/testing/selftests/drivers/net/lib/py/env.py | 14 +++++++++++++- tools/testing/selftests/net/lib/py/__init__.py | 7 +++++++ tools/testing/selftests/net/lib/py/utils.py | 14 ++++++++++++++ 3 files changed, 34 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/drivers/net/lib/py/env.py b/tools/testing/selftests/drivers/net/lib/py/env.py index a5e800b8f103..ec53cf59e104 100644 --- a/tools/testing/selftests/drivers/net/lib/py/env.py +++ b/tools/testing/selftests/drivers/net/lib/py/env.py @@ -4,7 +4,7 @@ import os import time from pathlib import Path from lib.py import KsftSkipEx, KsftXfailEx -from lib.py import cmd, ethtool, ip +from lib.py import cmd, ethtool, ip, verbosity_ctl from lib.py import NetNS, NetdevSimDev from .remote import Remote @@ -42,6 +42,12 @@ class NetDrvEnv: self.env = _load_env_file(src_path) + try: + verbosity_ctl(level=int(self.env.get('VERBOSE', 0))) + except ValueError as e: + print(f'Ignoring \'VERBOSE\'. Unknown value \'{self.env.get("VERBOSE")}\'') + verbosity_ctl(level=0) + if 'NETIF' in self.env: self.dev = ip("link show dev " + self.env['NETIF'], json=True)[0] else: @@ -92,6 +98,12 @@ class NetDrvEpEnv: self._ns = None self._ns_peer = None + try: + verbosity_ctl(level=int(self.env.get('VERBOSE', 0))) + except ValueError as e: + print(f'Ignoring \'VERBOSE\'. Unknown value \'{self.env.get("VERBOSE")}\'') + verbosity_ctl(level=0) + if "NETIF" in self.env: if nsim_test is True: raise KsftXfailEx("Test only works on netdevsim") diff --git a/tools/testing/selftests/net/lib/py/__init__.py b/tools/testing/selftests/net/lib/py/__init__.py index b6d498d125fe..1541079fadce 100644 --- a/tools/testing/selftests/net/lib/py/__init__.py +++ b/tools/testing/selftests/net/lib/py/__init__.py @@ -1,8 +1,15 @@ # SPDX-License-Identifier: GPL-2.0 +import os from .consts import KSRC from .ksft import * from .netns import NetNS from .nsim import * from .utils import * from .ynl import NlError, YnlFamily, EthtoolFamily, NetdevFamily, RtnlFamily + +try: + verbosity_ctl(level=int(os.environ.get('VERBOSE', 0))) +except ValueError as e: + print(f'Ignoring \'VERBOSE\'. Unknown value \'{os.environ.get("VERBOSE")}\'') + verbosity_ctl(level=0) diff --git a/tools/testing/selftests/net/lib/py/utils.py b/tools/testing/selftests/net/lib/py/utils.py index 72590c3f90f1..4a59958649be 100644 --- a/tools/testing/selftests/net/lib/py/utils.py +++ b/tools/testing/selftests/net/lib/py/utils.py @@ -9,6 +9,18 @@ import subprocess import time +def verbosity_ctl(level=None): + global VERBOSITY_LEVEL + if level is not None: + VERBOSITY_LEVEL = level + return VERBOSITY_LEVEL + + +def verbose(*objs, **kwargs): + if verbosity_ctl() >= 1: + print(*objs, **kwargs) + + class CmdExitFailure(Exception): pass @@ -22,6 +34,8 @@ class cmd: self.stderr = None self.ret = None + verbose("#cmd|", comm) + self.comm = comm if host: self.proc = host.cmd(comm) -- 2.43.0

4 months, 3 weeks

2
3
0 0

[PATCH 00/12] tools/nolibc: improve LLVM/clang support

by Thomas Weißschuh

The current support for LLVM and clang in nolibc and its testsuite is very limited. * Various architectures plain do not compile * The user *has* to specify "-Os" otherwise the program crashes * Cross-compilation of the tests does not work * Using clang is not wired up in run-tests.sh This series extends this support. Signed-off-by: Thomas Weißschuh <linux(a)weissschuh.net> --- Thomas Weißschuh (12): tools/nolibc: use clang-compatible asm syntax in arch-arm.h tools/nolibc: limit powerpc stack-protector workaround to GCC tools/nolibc: move entrypoint specifics to compiler.h tools/nolibc: use attribute((naked)) if available selftests/nolibc: report failure if no testcase passed selftests/nolibc: avoid passing NULL to printf("%s") selftests/nolibc: determine $(srctree) first selftests/nolibc: setup objtree without Makefile.include selftests/nolibc: add support for LLVM= parameter selftests/nolibc: add cc-option compatible with clang cross builds selftests/nolibc: run-tests.sh: avoid overwriting CFLAGS_EXTRA selftests/nolibc: run-tests.sh: allow building through LLVM tools/include/nolibc/arch-aarch64.h | 4 ++-- tools/include/nolibc/arch-arm.h | 8 ++++---- tools/include/nolibc/arch-i386.h | 4 ++-- tools/include/nolibc/arch-loongarch.h | 4 ++-- tools/include/nolibc/arch-mips.h | 4 ++-- tools/include/nolibc/arch-powerpc.h | 6 +++--- tools/include/nolibc/arch-riscv.h | 4 ++-- tools/include/nolibc/arch-s390.h | 4 ++-- tools/include/nolibc/arch-x86_64.h | 4 ++-- tools/include/nolibc/compiler.h | 12 ++++++++++++ tools/testing/selftests/nolibc/Makefile | 27 ++++++++++++++++----------- tools/testing/selftests/nolibc/nolibc-test.c | 4 ++-- tools/testing/selftests/nolibc/run-tests.sh | 20 ++++++++++++++++---- 13 files changed, 67 insertions(+), 38 deletions(-) --- base-commit: 0db287736bc586fcd5a2925518ef09eec6924803 change-id: 20240727-nolibc-llvm-3fad68590d4c Best regards, -- Thomas Weißschuh <linux(a)weissschuh.net>

4 months, 3 weeks

4
28
0 0

[PATCH] tools/nolibc: pass argc, argv and envp to constructors

by Thomas Weißschuh

Mirror glibc behavior for compatibility. Signed-off-by: Thomas Weißschuh <linux(a)weissschuh.net> --- tools/include/nolibc/crt.h | 23 ++++++++++++----------- tools/testing/selftests/nolibc/nolibc-test.c | 5 +++-- 2 files changed, 15 insertions(+), 13 deletions(-) diff --git a/tools/include/nolibc/crt.h b/tools/include/nolibc/crt.h index 43b551468c2a..ac291574f6c0 100644 --- a/tools/include/nolibc/crt.h +++ b/tools/include/nolibc/crt.h @@ -13,11 +13,11 @@ const unsigned long *_auxv __attribute__((weak)); static void __stack_chk_init(void); static void exit(int); -extern void (*const __preinit_array_start[])(void) __attribute__((weak)); -extern void (*const __preinit_array_end[])(void) __attribute__((weak)); +extern void (*const __preinit_array_start[])(int, char **, char**) __attribute__((weak)); +extern void (*const __preinit_array_end[])(int, char **, char**) __attribute__((weak)); -extern void (*const __init_array_start[])(void) __attribute__((weak)); -extern void (*const __init_array_end[])(void) __attribute__((weak)); +extern void (*const __init_array_start[])(int, char **, char**) __attribute__((weak)); +extern void (*const __init_array_end[])(int, char **, char**) __attribute__((weak)); extern void (*const __fini_array_start[])(void) __attribute__((weak)); extern void (*const __fini_array_end[])(void) __attribute__((weak)); @@ -29,7 +29,8 @@ void _start_c(long *sp) char **argv; char **envp; int exitcode; - void (* const *func)(void); + void (* const *ctor_func)(int, char **, char **); + void (* const *dtor_func)(void); const unsigned long *auxv; /* silence potential warning: conflicting types for 'main' */ int _nolibc_main(int, char **, char **) __asm__ ("main"); @@ -66,16 +67,16 @@ void _start_c(long *sp) ; _auxv = auxv; - for (func = __preinit_array_start; func < __preinit_array_end; func++) - (*func)(); - for (func = __init_array_start; func < __init_array_end; func++) - (*func)(); + for (ctor_func = __preinit_array_start; ctor_func < __preinit_array_end; ctor_func++) + (*ctor_func)(argc, argv, envp); + for (ctor_func = __init_array_start; ctor_func < __init_array_end; ctor_func++) + (*ctor_func)(argc, argv, envp); /* go to application */ exitcode = _nolibc_main(argc, argv, envp); - for (func = __fini_array_end; func > __fini_array_start;) - (*--func)(); + for (dtor_func = __fini_array_end; dtor_func > __fini_array_start;) + (*--dtor_func)(); exit(exitcode); } diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c index 093d0512f4c5..0800b10fc3f7 100644 --- a/tools/testing/selftests/nolibc/nolibc-test.c +++ b/tools/testing/selftests/nolibc/nolibc-test.c @@ -686,9 +686,10 @@ static void constructor1(void) } __attribute__((constructor)) -static void constructor2(void) +static void constructor2(int argc, char **argv, char **envp) { - constructor_test_value *= 2; + if (argc && argv && envp) + constructor_test_value *= 2; } int run_startup(int min, int max) --- base-commit: 0db287736bc586fcd5a2925518ef09eec6924803 change-id: 20240728-nolibc-constructor-args-68bf56ff7438 Best regards, -- Thomas Weißschuh <linux(a)weissschuh.net>

4 months, 3 weeks

3
2
0 0

[PATCH 0/2] tools/nolibc: add support for [v]sscanf()

by Thomas Weißschuh

The implementation is limited and only supports numeric arguments. Signed-off-by: Thomas Weißschuh <linux(a)weissschuh.net> --- Thomas Weißschuh (2): tools/nolibc: add support for [v]sscanf() Revert "selftests: kselftest: Fix build failure with NOLIBC" tools/include/nolibc/stdio.h | 93 ++++++++++++++++++++++++++++ tools/testing/selftests/kselftest.h | 5 -- tools/testing/selftests/nolibc/nolibc-test.c | 59 ++++++++++++++++++ 3 files changed, 152 insertions(+), 5 deletions(-) --- base-commit: ae1f550efc11eaf1496c431d9c6e784cb49124c5 change-id: 20240414-nolibc-scanf-f1db6930d0c6 Best regards, -- Thomas Weißschuh <linux(a)weissschuh.net>

4 months, 3 weeks

3
7
0 0

[PATCH v2] kunit: add test duration attribute

by Rae Moar

Add a new test duration attribute to print the duration of a test run. Example: KTAP version 1 # Subtest: memcpy # module: memcpy_kunit 1..4 # memcpy_large_test.speed: slow # memcpy_large_test.duration: 1.134787584s ok 1 memcpy_large_test ... This attribute is printed for each test (excluding parameterized tests). Add documentation for this new attribute to KUnit docs. In order to save the timespec64 object, add the ability to save a memory allocated object to the attributes framework. Signed-off-by: Rae Moar <rmoar(a)google.com> --- Changes v1->v2: - Change sprintf to kasprintf .../dev-tools/kunit/running_tips.rst | 7 +++ include/kunit/attributes.h | 5 ++ include/kunit/test.h | 1 + lib/kunit/attributes.c | 54 ++++++++++++++++++- lib/kunit/test.c | 25 +++++++-- 5 files changed, 86 insertions(+), 6 deletions(-) diff --git a/Documentation/dev-tools/kunit/running_tips.rst b/Documentation/dev-tools/kunit/running_tips.rst index bd689db6fdd2..a528d92e5d8f 100644 --- a/Documentation/dev-tools/kunit/running_tips.rst +++ b/Documentation/dev-tools/kunit/running_tips.rst @@ -446,3 +446,10 @@ This attribute indicates whether the test uses init data or functions. This attribute is automatically saved as a boolean and tests can also be filtered using this attribute. + +``duration`` + +This attribute indicates the length of time in seconds of the test execution. + +This attribute is automatically saved as a timespec64 and printed for each test +(excluding parameterized tests). diff --git a/include/kunit/attributes.h b/include/kunit/attributes.h index bc76a0b786d2..89ca54ef380d 100644 --- a/include/kunit/attributes.h +++ b/include/kunit/attributes.h @@ -18,6 +18,11 @@ struct kunit_attr_filter { char *input; }; +/* + * Frees all of a test's allocated attributes. + */ +void kunit_free_attr(void *test_or_suite, bool is_test); + /* * Returns the name of the filter's attribute. */ diff --git a/include/kunit/test.h b/include/kunit/test.h index ec61cad6b71d..dca78d9bd3f6 100644 --- a/include/kunit/test.h +++ b/include/kunit/test.h @@ -82,6 +82,7 @@ enum kunit_speed { /* Holds attributes for each test case and suite */ struct kunit_attributes { enum kunit_speed speed; + struct timespec64 *duration; }; /** diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c index 2cf04cc09372..fd01d54e52d7 100644 --- a/lib/kunit/attributes.c +++ b/lib/kunit/attributes.c @@ -40,6 +40,7 @@ struct kunit_attr { int (*filter)(void *attr, const char *input, int *err); void *attr_default; enum print_ops print; + bool to_free; }; /* String Lists for enum Attributes */ @@ -79,8 +80,22 @@ static const char *attr_string_to_string(void *attr, bool *to_free) return (char *) attr; } +static const char *attr_duration_to_string(void *attr, bool *to_free) +{ + struct timespec64 *val = (struct timespec64 *)attr; + char *str = kasprintf(GFP_KERNEL, "%lld.%09lds", val->tv_sec, val->tv_nsec); + + *to_free = true; + return str; +} + /* Filter Methods */ +static int attr_default_filter(void *attr, const char *input, int *err) +{ + return false; +} + static const char op_list[] = "<>!="; /* @@ -246,8 +261,20 @@ static void *attr_is_init_get(void *test_or_suite, bool is_test) return ((void *) suite->is_init); } +static void *attr_duration_get(void *test_or_suite, bool is_test) +{ + struct kunit_case *test = is_test ? test_or_suite : NULL; + + if (test && !test->generate_params) + return ((void *) test->attr.duration); + else + return ((void *) NULL); +} + /* List of all Test Attributes */ +static struct timespec64 duration_default = {0, 0}; + static struct kunit_attr kunit_attr_list[] = { { .name = "speed", @@ -256,6 +283,7 @@ static struct kunit_attr kunit_attr_list[] = { .filter = attr_speed_filter, .attr_default = (void *)KUNIT_SPEED_NORMAL, .print = PRINT_ALWAYS, + .to_free = false, }, { .name = "module", @@ -264,6 +292,7 @@ static struct kunit_attr kunit_attr_list[] = { .filter = attr_string_filter, .attr_default = (void *)"", .print = PRINT_SUITE, + .to_free = false, }, { .name = "is_init", @@ -272,10 +301,33 @@ static struct kunit_attr kunit_attr_list[] = { .filter = attr_bool_filter, .attr_default = (void *)false, .print = PRINT_SUITE, + .to_free = false, + }, + { + .name = "duration", + .get_attr = attr_duration_get, + .to_string = attr_duration_to_string, + .filter = attr_default_filter, + .attr_default = (void *)(&duration_default), + .print = PRINT_ALWAYS, + .to_free = true, } }; -/* Helper Functions to Access Attributes */ +/* Helper Functions to Access/Free Attributes */ + +void kunit_free_attr(void *test_or_suite, bool is_test) +{ + int i; + void *attr; + + for (i = 0; i < ARRAY_SIZE(kunit_attr_list); i++) { + if (kunit_attr_list[i].to_free) { + attr = kunit_attr_list[i].get_attr(test_or_suite, is_test); + kfree(attr); + } + } +} const char *kunit_attr_filter_name(struct kunit_attr_filter filter) { diff --git a/lib/kunit/test.c b/lib/kunit/test.c index e8b1b52a19ab..0d18e4969015 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -376,11 +376,11 @@ static void kunit_run_case_check_speed(struct kunit *test, /* * Initializes and runs test case. Does not clean up or do post validations. */ -static void kunit_run_case_internal(struct kunit *test, +static struct timespec64 kunit_run_case_internal(struct kunit *test, struct kunit_suite *suite, struct kunit_case *test_case) { - struct timespec64 start, end; + struct timespec64 start, end, duration; if (suite->init) { int ret; @@ -389,7 +389,9 @@ static void kunit_run_case_internal(struct kunit *test, if (ret) { kunit_err(test, "failed to initialize: %d\n", ret); kunit_set_failure(test); - return; + duration.tv_sec = 0; + duration.tv_nsec = 0; + return duration; } } @@ -399,7 +401,11 @@ static void kunit_run_case_internal(struct kunit *test, ktime_get_ts64(&end); - kunit_run_case_check_speed(test, test_case, timespec64_sub(end, start)); + duration = timespec64_sub(end, start); + + kunit_run_case_check_speed(test, test_case, duration); + + return duration; } static void kunit_case_internal_cleanup(struct kunit *test) @@ -424,6 +430,7 @@ struct kunit_try_catch_context { struct kunit *test; struct kunit_suite *suite; struct kunit_case *test_case; + struct timespec64 duration; }; static void kunit_try_run_case(void *data) @@ -440,7 +447,7 @@ static void kunit_try_run_case(void *data) * abort will be called, this thread will exit, and finally the parent * thread will resume control and handle any necessary clean up. */ - kunit_run_case_internal(test, suite, test_case); + ctx->duration = kunit_run_case_internal(test, suite, test_case); } static void kunit_try_run_case_cleanup(void *data) @@ -521,6 +528,7 @@ static void kunit_run_case_catch_errors(struct kunit_suite *suite, { struct kunit_try_catch_context context; struct kunit_try_catch *try_catch; + struct timespec64 *duration = kmalloc(sizeof(struct timespec64), GFP_KERNEL); try_catch = &test->try_catch; @@ -533,6 +541,10 @@ static void kunit_run_case_catch_errors(struct kunit_suite *suite, context.test_case = test_case; kunit_try_catch_run(try_catch, &context); + duration->tv_sec = context.duration.tv_sec; + duration->tv_nsec = context.duration.tv_nsec; + test_case->attr.duration = duration; + /* Now run the cleanup */ kunit_try_catch_init(try_catch, test, @@ -670,6 +682,7 @@ int kunit_run_tests(struct kunit_suite *suite) } kunit_print_attr((void *)test_case, true, KUNIT_LEVEL_CASE); + kunit_free_attr((void *)test_case, true); kunit_print_test_stats(&test, param_stats); @@ -680,6 +693,7 @@ int kunit_run_tests(struct kunit_suite *suite) kunit_update_stats(&suite_stats, test_case->status); kunit_accumulate_stats(&total_stats, param_stats); + } if (suite->suite_exit) @@ -688,6 +702,7 @@ int kunit_run_tests(struct kunit_suite *suite) kunit_print_suite_stats(suite, suite_stats, total_stats); suite_end: kunit_print_suite_end(suite); + kunit_free_attr((void *)suite, false); return 0; } base-commit: 67c9971cd6d309ecbcb87b942e22ffc194d7a376 -- 2.46.0.rc2.264.g509ed76dc8-goog

4 months, 3 weeks

2
1
0 0

[PATCH v2] selftests: openat2: don't print total number of tests and then skip

by Muhammad Usama Anjum

Don't print that 88 sub-tests are going to be executed, but then skip. This is against TAP compliance. Instead check pre-requisites first before printing total number of tests. Old non-tap compliant output: TAP version 13 1..88 ok 2 # SKIP all tests require euid == 0 # Planned tests != run tests (88 != 1) # Totals: pass:0 fail:0 xfail:0 xpass:0 skip:1 error:0 New and correct output: TAP version 13 1..0 # SKIP all tests require euid == 0 Signed-off-by: Muhammad Usama Anjum <usama.anjum(a)collabora.com> --- Changes since v1: - Remove simplifying if condition lines - Update the patch message --- tools/testing/selftests/openat2/resolve_test.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/openat2/resolve_test.c b/tools/testing/selftests/openat2/resolve_test.c index bbafad440893c..85a4c64ee950d 100644 --- a/tools/testing/selftests/openat2/resolve_test.c +++ b/tools/testing/selftests/openat2/resolve_test.c @@ -508,12 +508,13 @@ void test_openat2_opath_tests(void) int main(int argc, char **argv) { ksft_print_header(); - ksft_set_plan(NUM_TESTS); /* NOTE: We should be checking for CAP_SYS_ADMIN here... */ if (geteuid() != 0) ksft_exit_skip("all tests require euid == 0\n"); + ksft_set_plan(NUM_TESTS); + test_openat2_opath_tests(); if (ksft_get_fail_cnt() + ksft_get_error_cnt() > 0) -- 2.39.2

4 months, 3 weeks

3
8
0 0

[PATCH 0/2] selftests/nolibc: run-user improvements

by Thomas Weißschuh

A small bugfix for "run-user XARCH=ppc64le" and run-user support for run-tests.sh. Signed-off-by: Thomas Weißschuh <linux(a)weissschuh.net> --- Thomas Weißschuh (2): selftests/nolibc: introduce QEMU_ARCH_USER selftests/nolibc: run-tests.sh: enable testing via qemu-user tools/testing/selftests/nolibc/Makefile | 5 ++++- tools/testing/selftests/nolibc/run-tests.sh | 22 +++++++++++++++++++--- 2 files changed, 23 insertions(+), 4 deletions(-) --- base-commit: ba335752620565c25c3028fff9496bb8ef373602 change-id: 20770915-nolibc-run-user-845375a3ec4f Best regards, -- Thomas Weißschuh <linux(a)weissschuh.net>

4 months, 3 weeks

5
8
0 0

[PATCH] selftests: tpm2: redirect python unittest logs to stdout

by Muhammad Usama Anjum

The python unittest module writes all its output to stderr, even when the run is clean. Redirect its output logs to stdout. Cc: Jarkko Sakkinen <jarkko(a)kernel.org> Signed-off-by: Muhammad Usama Anjum <usama.anjum(a)collabora.com> --- tools/testing/selftests/tpm2/test_async.sh | 2 +- tools/testing/selftests/tpm2/test_smoke.sh | 2 +- tools/testing/selftests/tpm2/test_space.sh | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/tpm2/test_async.sh b/tools/testing/selftests/tpm2/test_async.sh index 43bf5bd772fd4..cf5a9c826097b 100755 --- a/tools/testing/selftests/tpm2/test_async.sh +++ b/tools/testing/selftests/tpm2/test_async.sh @@ -7,4 +7,4 @@ ksft_skip=4 [ -e /dev/tpm0 ] || exit $ksft_skip [ -e /dev/tpmrm0 ] || exit $ksft_skip -python3 -m unittest -v tpm2_tests.AsyncTest +python3 -m unittest -v tpm2_tests.AsyncTest 2>&1 diff --git a/tools/testing/selftests/tpm2/test_smoke.sh b/tools/testing/selftests/tpm2/test_smoke.sh index 58af963e5b55a..20fa70f970a9a 100755 --- a/tools/testing/selftests/tpm2/test_smoke.sh +++ b/tools/testing/selftests/tpm2/test_smoke.sh @@ -6,4 +6,4 @@ ksft_skip=4 [ -e /dev/tpm0 ] || exit $ksft_skip -python3 -m unittest -v tpm2_tests.SmokeTest +python3 -m unittest -v tpm2_tests.SmokeTest 2>&1 diff --git a/tools/testing/selftests/tpm2/test_space.sh b/tools/testing/selftests/tpm2/test_space.sh index 04c47b13fe8ac..93894cbc89a80 100755 --- a/tools/testing/selftests/tpm2/test_space.sh +++ b/tools/testing/selftests/tpm2/test_space.sh @@ -6,4 +6,4 @@ ksft_skip=4 [ -e /dev/tpmrm0 ] || exit $ksft_skip -python3 -m unittest -v tpm2_tests.SpaceTest +python3 -m unittest -v tpm2_tests.SpaceTest 2>&1 -- 2.39.2

4 months, 3 weeks

4
6
0 0

[PATCH RFC] kselftest: devices: Add test to detect missing devices

by Nícolas F. R. A. Prado

Introduce a new test to identify regressions causing devices to go missing on the system. For each bus and class on the system the test checks the number of devices present against a reference file, which needs to have been generated by the program at a previous point on a known-good kernel, and if there are missing devices they are reported. Signed-off-by: Nícolas F. R. A. Prado <nfraprado(a)collabora.com> --- Hi, Key points about this test: * Goal: Identify regressions causing devices to go missing on the system * Focus: * Ease of maintenance: the reference file is generated programatically * Minimum of false-positives: the script makes as few assumptions as possible about the stability of device identifiers to ensure renames/refactors don't trigger false-positives * How it works: For each bus and class on the system the test checks the number of devices present against a reference file, which needs to have been generated by the program at a previous point on a known-good kernel, and if there are missing devices they are reported. * Comparison to other tests: It might be possible(*) to replace the discoverable devices test [1] with this. The benefits of this test is that it's easier to setup and maintain and has wider coverage of devices. Additional detail: * Having more devices on the running system than the reference does not cause a failure, but a warning is printed in that case to suggest that the reference be updated. * Missing devices are detected per bus/class based on the number of devices. When the test fails, the known metadata for each of the expected and detected devices is printed and some simple similitarity comparison is done to suggest the devices that are the most likely to be missing. * The proposed place to store the generated reference files is the 'platform-test-parameters' repository in KernelCI [2]. Example output: This is an example of a failing test case when one of the two devices in the nvmem bus went missing: # Missing devices for subsystem 'nvmem': 1 (Expected 2, found 1) # ================= # Devices expected: # # uevent: # OF_NAME=efuse # OF_FULLNAME=/soc/efuse@11c10000 # OF_COMPATIBLE_0=mediatek,mt8195-efuse # OF_COMPATIBLE_1=mediatek,efuse # OF_COMPATIBLE_N=2 # # uevent: # OF_NAME=flash # OF_FULLNAME=/soc/spi@1132c000/flash@0 # OF_COMPATIBLE_0=jedec,spi-nor # OF_COMPATIBLE_N=1 # # ----------------- # Devices found: # # uevent: # OF_NAME=efuse # OF_FULLNAME=/soc/efuse@11c10000 # OF_COMPATIBLE_0=mediatek,mt8195-efuse # OF_COMPATIBLE_1=mediatek,efuse # OF_COMPATIBLE_N=2 # # ----------------- # Devices missing (best guess): # # uevent: # OF_NAME=flash # OF_FULLNAME=/soc/spi@1132c000/flash@0 # OF_COMPATIBLE_0=jedec,spi-nor # OF_COMPATIBLE_N=1 # # ================= not ok 19 bus.nvmem Example of how the data for these devices is encoded in the reference file: bus: ... nvmem: count: 2 devices: - info: uevent: 'OF_NAME=efuse OF_FULLNAME=/soc/efuse@11c10000 OF_COMPATIBLE_0=mediatek,mt8195-efuse OF_COMPATIBLE_1=mediatek,efuse OF_COMPATIBLE_N=2 ' - info: uevent: 'OF_NAME=flash OF_FULLNAME=/soc/spi@1132c000/flash@0 OF_COMPATIBLE_0=jedec,spi-nor OF_COMPATIBLE_N=1 ' (Full reference file: http://0x0.st/Xp60.yaml;) Caveat: Relying only on the count of devices in a subsystem makes the test susceptible to false-negatives eg. if a device goes missing and another in the same subsystem is added the count will be the same so this regression won't be reported. In order to avoid this we may include properties that must match individual devices, but we must be very careful (and it's why I haven't done it) since matching against properties that aren't guaranteed to be stable will introduce false-positives (ie. detecting false regressions) due to eventual renames. Some things to improve in the near future / gather feedback on: * (*): Currently this test only checks for the existence of devices. We could extend it to also encode into the reference which devices are bound to drivers to be able to completely replace the discoverable devices probe kselftest [1]. * Expanding identifying properties: Currently the properties that are stored (when present) in the reference for each device to be used for identification in the result output are uevent, device/uevent, firmware_node/uevent and name. Suggestions of others properties to add are welcome. * Adding more filtering to reduce noise: * Ignoring buses/classes: Currently the devlink class is ignored by the test since it seems like a kernel internal detail that userspace doesn't actually care about. We should add others that are similar. * Ignoring non-devices: There can be entries in /sys/class/ that aren't devices. For now we're filtering down to only symlinks, but there might be a better way. * As mentioned in the caveat section above we may want to add actual matching of devices based on properties to avoid false-negatives if we identify suitable properties. * It would be nice to have an option in the program to compare a newer reference to an older one to make it easier for the user to see the differences and decide if the new reference is ok. * Since the reference file is not supposed to be manually edited, JSON might be a better choice than YAML since it is included in the python standard library. Let me know your thoughts. Thanks, Nícolas [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/too… [2] https://github.com/kernelci/platform-test-parameters --- tools/testing/selftests/Makefile | 1 + tools/testing/selftests/devices/exist/Makefile | 3 + tools/testing/selftests/devices/exist/exist.py | 268 +++++++++++++++++++++++++ 3 files changed, 272 insertions(+) diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index bc8fe9e8f7f2..9c49b5ec5bef 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -14,6 +14,7 @@ TARGETS += cpufreq TARGETS += cpu-hotplug TARGETS += damon TARGETS += devices/error_logs +TARGETS += devices/exist TARGETS += devices/probe TARGETS += dmabuf-heaps TARGETS += drivers/dma-buf diff --git a/tools/testing/selftests/devices/exist/Makefile b/tools/testing/selftests/devices/exist/Makefile new file mode 100644 index 000000000000..3075cac32092 --- /dev/null +++ b/tools/testing/selftests/devices/exist/Makefile @@ -0,0 +1,3 @@ +TEST_PROGS := exist.py + +include ../../lib.mk diff --git a/tools/testing/selftests/devices/exist/exist.py b/tools/testing/selftests/devices/exist/exist.py new file mode 100755 index 000000000000..8241b2fabc8e --- /dev/null +++ b/tools/testing/selftests/devices/exist/exist.py @@ -0,0 +1,268 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024 Collabora Ltd + +# * Goal: Identify regressions causing devices to go missing on the system +# * Focus: +# * Ease of maintenance: the reference file is generated programatically +# * Minimum of false-positives: the script makes as few assumptions as +# possible about the stability of device identifiers to ensure +# renames/refactors don't trigger false-positives +# * How it works: For each bus and class on the system the test checks the +# number of devices present against a reference file, which needs to have been +# generated by the program at a previous point on a known-good kernel, and if +# there are missing devices they are reported. + +import os +import sys +import argparse + +import yaml + +# Allow ksft module to be imported from different directory +this_dir = os.path.dirname(os.path.realpath(__file__)) +sys.path.append(os.path.join(this_dir, "../../kselftest/")) + +import ksft + + +def generate_devs_obj(): + obj = {} + + device_sources = [ + { + "base_dir": "/sys/class", + "add_path": "", + "key_name": "class", + "ignored": ["devlink"], + }, + { + "base_dir": "/sys/bus", + "add_path": "devices", + "key_name": "bus", + "ignored": [], + }, + ] + + properties = sorted(["uevent", "device/uevent", "firmware_node/uevent", "name"]) + + for source in device_sources: + source_subsystems = {} + for subsystem in sorted(os.listdir(source["base_dir"])): + if subsystem in source["ignored"]: + continue + + devs_path = os.path.join(source["base_dir"], subsystem, source["add_path"]) + dev_dirs = [dev for dev in os.scandir(devs_path) if dev.is_symlink()] + devs_data = [] + for dev_dir in dev_dirs: + dev_path = os.path.join(devs_path, dev_dir) + dev_data = {"info": {}} + for prop in properties: + if os.path.isfile(os.path.join(dev_path, prop)): + with open(os.path.join(dev_path, prop)) as f: + dev_data["info"][prop] = f.read() + devs_data.append(dev_data) + if len(dev_dirs): + source_subsystems[subsystem] = { + "count": len(dev_dirs), + "devices": devs_data, + } + obj[source["key_name"]] = source_subsystems + + return obj + + +def commented(s): + return s.replace("\n", "\n# ") + + +def indented(s, n): + return " " * n + s.replace("\n", "\n" + " " * n) + + +def stripped(s): + return s.strip("\n") + + +def devices_difference(dev1, dev2): + difference = 0 + + for prop in dev1["info"].keys(): + for l1, l2 in zip( + dev1["info"].get(prop, "").split("\n"), + dev2["info"].get(prop, "").split("\n"), + ): + if l1 != l2: + difference += 1 + return difference + + +def guess_missing_devices(cur_devs_subsystem, ref_devs_subsystem): + # Detect what devices on the current system are the most similar to devices + # on the reference one by one until the leftovers are the most dissimilar + # devices and therefore most likely the missing ones. + found_count = cur_devs_subsystem["count"] + expected_count = ref_devs_subsystem["count"] + missing_count = found_count - expected_count + + diffs = [] + for cur_d in cur_devs_subsystem["devices"]: + for ref_d in ref_devs_subsystem["devices"]: + diffs.append((devices_difference(cur_d, ref_d), cur_d, ref_d)) + + diffs.sort(key=lambda x: x[0]) + + assigned_ref_devs = [] + assigned_cur_devs = [] + for diff in diffs: + if len(assigned_ref_devs) >= expected_count - missing_count: + break + if diff[1] in assigned_cur_devs or diff[2] in assigned_ref_devs: + continue + assigned_cur_devs.append(diff[1]) + assigned_ref_devs.append(diff[2]) + + missing_devices = [] + for d in ref_devs_subsystem["devices"]: + if d not in assigned_ref_devs: + missing_devices.append(d) + + return missing_devices + + +def dump_devices_info(cur_devs_subsystem, ref_devs_subsystem): + def dump_device_info(dev): + for name, val in dev["info"].items(): + ksft.print_msg(indented(name + ":", 2)) + val = stripped(val) + if val: + ksft.print_msg(commented(indented(val, 4))) + ksft.print_msg("") + + ksft.print_msg("=================") + ksft.print_msg("Devices expected:") + ksft.print_msg("") + for d in ref_devs_subsystem["devices"]: + dump_device_info(d) + ksft.print_msg("-----------------") + ksft.print_msg("Devices found:") + ksft.print_msg("") + for d in cur_devs_subsystem["devices"]: + dump_device_info(d) + ksft.print_msg("-----------------") + ksft.print_msg("Devices missing (best guess):") + ksft.print_msg("") + missing_devices = guess_missing_devices(cur_devs_subsystem, ref_devs_subsystem) + for d in missing_devices: + dump_device_info(d) + ksft.print_msg("=================") + + +def run_test(ref_filename): + ksft.print_msg(f"Using reference file: {ref_filename}") + + with open(ref_filename) as f: + ref_devs_obj = yaml.safe_load(f) + + num_tests = 0 + for dev_source in ref_devs_obj.values(): + num_tests += len(dev_source) + ksft.set_plan(num_tests) + + cur_devs_obj = generate_devs_obj() + + reference_outdated = False + + for source, ref_devs_source_obj in ref_devs_obj.items(): + for subsystem, ref_devs_subsystem_obj in ref_devs_source_obj.items(): + test_name = f"{source}.{subsystem}" + if not ( + cur_devs_obj.get(source) and cur_devs_obj.get(source).get(subsystem) + ): + ksft.print_msg(f"Device subsystem '{subsystem}' missing") + ksft.test_result_fail(test_name) + continue + cur_devs_subsystem_obj = cur_devs_obj[source][subsystem] + + found_count = cur_devs_subsystem_obj["count"] + expected_count = ref_devs_subsystem_obj["count"] + if found_count < expected_count: + ksft.print_msg( + f"Missing devices for subsystem '{subsystem}': {expected_count - found_count} (Expected {expected_count}, found {found_count})" + ) + dump_devices_info(cur_devs_subsystem_obj, ref_devs_subsystem_obj) + ksft.test_result_fail(test_name) + else: + ksft.test_result_pass(test_name) + if found_count > expected_count: + reference_outdated = True + + if len(cur_devs_obj[source]) > len(ref_devs_source_obj): + reference_outdated = True + + if reference_outdated: + ksft.print_msg( + "Warning: The current system contains more devices and/or subsystems than the reference. Updating the reference is recommended." + ) + + +def get_possible_ref_filenames(): + filenames = [] + + dt_board_compatible_file = "/proc/device-tree/compatible" + if os.path.exists(dt_board_compatible_file): + with open(dt_board_compatible_file) as f: + for line in f: + compatibles = [compat for compat in line.split("\0") if compat] + filenames.extend(compatibles) + else: + dmi_id_dir = "/sys/devices/virtual/dmi/id" + vendor_dmi_file = os.path.join(dmi_id_dir, "sys_vendor") + product_dmi_file = os.path.join(dmi_id_dir, "product_name") + + with open(vendor_dmi_file) as f: + vendor = f.read().replace("\n", "") + with open(product_dmi_file) as f: + product = f.read().replace("\n", "") + + filenames = [vendor + "," + product] + + return filenames + + +def get_ref_filename(ref_dir): + chosen_ref_filename = "" + full_ref_paths = [os.path.join(ref_dir, f + ".yaml") for f in get_possible_ref_filenames()] + for path in full_ref_paths: + if os.path.exists(path): + chosen_ref_filename = path + break + + if not chosen_ref_filename: + tried_paths = ",".join(["'" + p + "'" for p in full_ref_paths]) + ksft.print_msg(f"No matching reference file found (tried {tried_paths})") + ksft.exit_fail() + + return chosen_ref_filename + + +parser = argparse.ArgumentParser() +parser.add_argument( + "--reference-dir", default=".", help="Directory containing the reference files" +) +parser.add_argument("--generate-reference", action="store_true", help="Generate a reference file with the devices on the running system") +args = parser.parse_args() + +if args.generate_reference: + print(f"# Kernel version: {os.uname().release}") + print(yaml.dump(generate_devs_obj())) + sys.exit(0) + +ksft.print_header() + +ref_filename = get_ref_filename(args.reference_dir) + +run_test(ref_filename) + +ksft.finished() --- base-commit: 73399b58e5e5a1b28a04baf42e321cfcfc663c2f change-id: 20240724-kselftest-dev-exist-bb1bcf884654 Best regards, -- Nícolas F. R. A. Prado <nfraprado(a)collabora.com>

4 months, 3 weeks

3
7
0 0

[PATCH net 0/7] mptcp: fix endpoints with 'signal' and 'subflow' flags

by Matthieu Baerts (NGI0)

When looking at improving the user experience around the MPTCP endpoints setup, I noticed that setting an endpoint with both the 'signal' and the 'subflow' flags -- as it has been done in the past by users according to bug reports we got -- was resulting on only announcing the endpoint, but not using it to create subflows: the 'subflow' flag was then ignored. My initial thought was to modify IPRoute2 to warn the user when the two flags were set, but it doesn't sound normal to ignore one of them. I then looked at modifying the kernel not to allow having the two flags set, but when discussing about that with Mat, we thought it was maybe not ideal to do that, as there might be use-cases, we might break some configs. Then I saw it was working before v5.17. So instead, I fixed the support on the kernel side (patch 5) using Paolo's suggestion. This also includes a fix on the options side (patch 1: for v5.11+), an explicit deny of some options combinations (patch 2: for v5.18+), and some refactoring (patches 3 and 4) to ease the inclusion of the patch 5. While at it, I added a new selftest (patch 7) to validate this case -- including a modification of the chk_add_nr helper to inverse the sides were the counters are checked (patch 6) -- and allowed ADD_ADDR echo just after the MP_JOIN 3WHS. The selftests modification have the same Fixes tag as the previous commit, but no 'Cc: Stable': if the backport can work, that's good -- but it still need to be verified by running the selftests -- if not, no need to worry, many CIs will use the selftests from the last stable version to validate previous stable releases. Signed-off-by: Matthieu Baerts (NGI0) <matttbe(a)kernel.org> --- Matthieu Baerts (NGI0) (7): mptcp: fully established after ADD_ADDR echo on MPJ mptcp: pm: deny endp with signal + subflow + port mptcp: pm: reduce indentation blocks mptcp: pm: don't try to create sf if alloc failed mptcp: pm: do not ignore 'subflow' if 'signal' flag is also set selftests: mptcp: join: ability to invert ADD_ADDR check selftests: mptcp: join: test both signal & subflow net/mptcp/options.c | 3 +- net/mptcp/pm_netlink.c | 47 +++++++++++++-------- tools/testing/selftests/net/mptcp/mptcp_join.sh | 55 ++++++++++++++++++------- 3 files changed, 73 insertions(+), 32 deletions(-) --- base-commit: 0bf50cead4c4710d9f704778c32ab8af47ddf070 change-id: 20240731-upstream-net-20240731-mptcp-endp-subflow-signal-181d640cf5e8 Best regards, -- Matthieu Baerts (NGI0) <matttbe(a)kernel.org>

4 months, 3 weeks

2
8
0 0

2024

2023

2022

2021

2020

2019

2018

2017

Linux-kselftest-mirror July 2024