December 2021 - Linux-kselftest-mirror

[REGRESSION] lkft kselftest for v5.16-rc4-6579-gea922272cbe5

by lkft＠linaro.org

## Build * kernel: 5.16.0-rc4 * git: ['https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git', 'https://gitlab.com/Linaro/lkft/mirrors/next/linux-next'] * git branch: master * git commit: ea922272cbe547bdf58da2aaf240d59782c6a009 * git describe: v5.16-rc4-6579-gea922272cbe5 * test details: https://qa-reports.linaro.org/lkft/linux-next-master/build/v5.16-rc4-6579-g… ## Test Regressions (compared to next-20211117) * dragonboard-410c, kselftest-capabilities - capabilities.test_execve * dragonboard-410c, kselftest-proc - proc.proc-fsconfig-hidepid * hi6220-hikey, kselftest-timers - timers.rtcpie - timers.set-timer-lat * qemu_arm, kselftest-zram - zram.zram.sh * x15, kselftest-capabilities - capabilities.test_execve * x15, kselftest-cgroup - cgroup.test_freezer - cgroup.test_kill - cgroup.test_kill.test_cgkill_simple * x15, kselftest-rtc - rtc.rtctest.rtc.alarm_alm_set - rtc.rtctest.rtc.alarm_alm_set_minute - rtc.rtctest.rtc.alarm_wkalm_set - rtc.rtctest.rtc.date_read * x86, kselftest-kvm - kvm.vmx_pmu_msrs_test ## Metric Regressions (compared to next-20211117) No metric regressions found. Reported-by: Linux Kernel Functional Testing <lkft(a)linaro.org> ## Test Fixes (compared to next-20211117) * dragonboard-410c, kselftest-sync - sync.sync_test * qemu_arm, kselftest-timers - timers.rtcpie * qemu_x86_64, kselftest-kvm - kvm.set_memory_region_test * qemu_x86_64, kselftest-rtc - rtc.rtctest * qemu_x86_64, kselftest-timers - timers.rtcpie * x15, kselftest-core - core.close_range_test * x15, kselftest-sync - sync.sync_test * x86, kselftest-kvm - kvm.memslot_perf_test * x86, kselftest-rtc - rtc.rtctest ## Metric Fixes (compared to next-20211117) No metric fixes found. ## Test result summary total: 3939, pass: 2058, fail: 419, skip: 1462, xfail: 0 ## Build Summary ## Test suites summary * kselftest-android * kselftest-arm64 * kselftest-arm64/arm64.btitest.bti_c_func * kselftest-arm64/arm64.btitest.bti_j_func * kselftest-arm64/arm64.btitest.bti_jc_func * kselftest-arm64/arm64.btitest.bti_none_func * kselftest-arm64/arm64.btitest.nohint_func * kselftest-arm64/arm64.btitest.paciasp_func * kselftest-arm64/arm64.nobtitest.bti_c_func * kselftest-arm64/arm64.nobtitest.bti_j_func * kselftest-arm64/arm64.nobtitest.bti_jc_func * kselftest-arm64/arm64.nobtitest.bti_none_func * kselftest-arm64/arm64.nobtitest.nohint_func * kselftest-arm64/arm64.nobtitest.paciasp_func * kselftest-bpf * kselftest-breakpoints * kselftest-capabilities * kselftest-cgroup * kselftest-clone3 * kselftest-core * kselftest-cpu-hotplug * kselftest-cpufreq * kselftest-drivers * kselftest-efivarfs * kselftest-filesystems * kselftest-firmware * kselftest-fpu * kselftest-futex * kselftest-gpio * kselftest-intel_pstate * kselftest-ipc * kselftest-ir * kselftest-kcmp * kselftest-kexec * kselftest-kvm * kselftest-lib * kselftest-livepatch * kselftest-lkdtm * kselftest-membarrier * kselftest-memfd * kselftest-memory-hotplug * kselftest-mincore * kselftest-mount * kselftest-mqueue * kselftest-openat2 * kselftest-pid_namespace * kselftest-pidfd * kselftest-proc * kselftest-pstore * kselftest-ptrace * kselftest-rseq * kselftest-rtc * kselftest-seccomp * kselftest-sigaltstack * kselftest-size * kselftest-splice * kselftest-static_keys * kselftest-sync * kselftest-sysctl * kselftest-timens * kselftest-timers * kselftest-tmpfs * kselftest-tpm2 * kselftest-user * kselftest-vm * kselftest-x86 * kselftest-zram -- Linaro LKFT https://lkft.linaro.org

4 years

1
0
0 0

[REGRESSION] lkft kselftest for next-20211203

by lkft＠linaro.org

## Build * kernel: 5.16.0-rc3 * git: ['https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git', 'https://gitlab.com/Linaro/lkft/mirrors/next/linux-next'] * git branch: master * git commit: 7afeac307a9561e3a93682c1e7eb22f918aa1187 * git describe: next-20211203 * test details: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20211203 ## Test Regressions (compared to next-20211117) * dragonboard-410c, kselftest-capabilities - capabilities.test_execve * dragonboard-410c, kselftest-pidfd - pidfd.pidfd_poll_test * dragonboard-410c, kselftest-proc - proc.proc-fsconfig-hidepid * dragonboard-410c, kselftest-seccomp - seccomp.seccomp_bpf - seccomp.seccomp_bpf.global.user_notification_filter_empty - seccomp.seccomp_bpf.global.user_notification_filter_empty_threaded * dragonboard-410c, kselftest-timers - timers.rtcpie * qemu_i386, kselftest-rtc - rtc.rtctest * x15, kselftest-capabilities - capabilities.test_execve * x15, kselftest-cgroup - cgroup.test_freezer - cgroup.test_kill - cgroup.test_kill.test_cgkill_simple * x15, kselftest-rtc - rtc.rtctest.rtc.alarm_alm_set - rtc.rtctest.rtc.alarm_alm_set_minute - rtc.rtctest.rtc.alarm_wkalm_set - rtc.rtctest.rtc.date_read * x86, kselftest-kvm - kvm.vmx_pmu_msrs_test ## Metric Regressions (compared to next-20211117) No metric regressions found. Reported-by: Linux Kernel Functional Testing <lkft(a)linaro.org> ## Test Fixes (compared to next-20211117) * qemu_arm, kselftest-timers - timers.rtcpie * qemu_x86_64, kselftest-rtc - rtc.rtctest * qemu_x86_64, kselftest-timers - timers.rtcpie * x15, kselftest-core - core.close_range_test * x15, kselftest-rtc - rtc.rtctest * x15, kselftest-sync - sync.sync_test ## Metric Fixes (compared to next-20211117) No metric fixes found. ## Test result summary total: 5129, pass: 2706, fail: 546, skip: 1877, xfail: 0 ## Build Summary ## Test suites summary * kselftest-android * kselftest-arm64 * kselftest-arm64/arm64.btitest.bti_c_func * kselftest-arm64/arm64.btitest.bti_j_func * kselftest-arm64/arm64.btitest.bti_jc_func * kselftest-arm64/arm64.btitest.bti_none_func * kselftest-arm64/arm64.btitest.nohint_func * kselftest-arm64/arm64.btitest.paciasp_func * kselftest-arm64/arm64.nobtitest.bti_c_func * kselftest-arm64/arm64.nobtitest.bti_j_func * kselftest-arm64/arm64.nobtitest.bti_jc_func * kselftest-arm64/arm64.nobtitest.bti_none_func * kselftest-arm64/arm64.nobtitest.nohint_func * kselftest-arm64/arm64.nobtitest.paciasp_func * kselftest-bpf * kselftest-breakpoints * kselftest-capabilities * kselftest-cgroup * kselftest-clone3 * kselftest-core * kselftest-cpu-hotplug * kselftest-cpufreq * kselftest-drivers * kselftest-efivarfs * kselftest-filesystems * kselftest-firmware * kselftest-fpu * kselftest-futex * kselftest-gpio * kselftest-intel_pstate * kselftest-ipc * kselftest-ir * kselftest-kcmp * kselftest-kexec * kselftest-kvm * kselftest-lib * kselftest-livepatch * kselftest-lkdtm * kselftest-membarrier * kselftest-memfd * kselftest-memory-hotplug * kselftest-mincore * kselftest-mount * kselftest-mqueue * kselftest-net * kselftest-openat2 * kselftest-pid_namespace * kselftest-pidfd * kselftest-proc * kselftest-pstore * kselftest-ptrace * kselftest-rseq * kselftest-rtc * kselftest-seccomp * kselftest-sigaltstack * kselftest-size * kselftest-splice * kselftest-static_keys * kselftest-sync * kselftest-sysctl * kselftest-timens * kselftest-timers * kselftest-tmpfs * kselftest-tpm2 * kselftest-user * kselftest-vm * kselftest-x86 * kselftest-zram -- Linaro LKFT https://lkft.linaro.org

4 years

1
0
0 0

[REGRESSION] lkft kselftest for v5.16-rc5-6644-gbcd5ddb85fad

by lkft＠linaro.org

## Build * kernel: 5.16.0-rc5 * git: https://gitlab.com/Linaro/lkft/mirrors/next/linux-next * git branch: master * git commit: bcd5ddb85fad453da26afd5b3fca8ad931b5253d * git describe: v5.16-rc5-6644-gbcd5ddb85fad * test details: https://qa-reports.linaro.org/lkft/linux-next-master/build/v5.16-rc5-6644-g… ## Test Regressions (compared to next-20211117) * dragonboard-410c, kselftest-capabilities - capabilities.test_execve * dragonboard-410c, kselftest-cgroup - cgroup.test_freezer - cgroup.test_freezer.test_cgfreezer_ptrace * dragonboard-410c, kselftest-kvm - kvm.get-reg-list * dragonboard-410c, kselftest-proc - proc.proc-fsconfig-hidepid * dragonboard-410c, kselftest-seccomp - seccomp.seccomp_bpf - seccomp.seccomp_bpf.global.user_notification_filter_empty - seccomp.seccomp_bpf.global.user_notification_filter_empty_threaded * qemu_arm, kselftest-zram - zram.zram.sh * qemu_i386, kselftest-rtc - rtc.rtctest * x15, kselftest-rtc - rtc.rtctest.rtc.alarm_alm_set - rtc.rtctest.rtc.alarm_alm_set_minute - rtc.rtctest.rtc.alarm_wkalm_set - rtc.rtctest.rtc.date_read * x86, kselftest-kvm - kvm.vmx_pmu_msrs_test ## Metric Regressions (compared to next-20211117) No metric regressions found. Reported-by: Linux Kernel Functional Testing <lkft(a)linaro.org> ## Test Fixes (compared to next-20211117) * dragonboard-410c, kselftest-sync - sync.sync_test * hi6220-hikey, kselftest-timers - timers.nsleep-lat * qemu_x86_64, kselftest-kvm - kvm.set_memory_region_test * qemu_x86_64, kselftest-timers - timers.rtcpie * x15, kselftest-sync - sync.sync_test * x86, kselftest-kvm - kvm.memslot_perf_test * x86, kselftest-rtc - rtc.rtctest ## Metric Fixes (compared to next-20211117) No metric fixes found. ## Test result summary total: 3495, pass: 1960, fail: 324, skip: 1211, xfail: 0 ## Build Summary ## Test suites summary * kselftest-android * kselftest-arm64 * kselftest-bpf * kselftest-breakpoints * kselftest-capabilities * kselftest-cgroup * kselftest-clone3 * kselftest-core * kselftest-cpu-hotplug * kselftest-cpufreq * kselftest-drivers * kselftest-efivarfs * kselftest-filesystems * kselftest-firmware * kselftest-fpu * kselftest-futex * kselftest-gpio * kselftest-intel_pstate * kselftest-ipc * kselftest-ir * kselftest-kcmp * kselftest-kexec * kselftest-kvm * kselftest-lib * kselftest-livepatch * kselftest-lkdtm * kselftest-membarrier * kselftest-memfd * kselftest-memory-hotplug * kselftest-mincore * kselftest-mount * kselftest-mqueue * kselftest-openat2 * kselftest-pid_namespace * kselftest-pidfd * kselftest-proc * kselftest-pstore * kselftest-ptrace * kselftest-rseq * kselftest-rtc * kselftest-seccomp * kselftest-sigaltstack * kselftest-size * kselftest-splice * kselftest-static_keys * kselftest-sync * kselftest-sysctl * kselftest-timens * kselftest-timers * kselftest-tmpfs * kselftest-tpm2 * kselftest-user * kselftest-vm * kselftest-x86 * kselftest-zram -- Linaro LKFT https://lkft.linaro.org

4 years

1
0
0 0

[REGRESSION] lkft kselftest for next-20211207

by lkft＠linaro.org

## Build * kernel: 5.16.0-rc4 * git: ['https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git', 'https://gitlab.com/Linaro/lkft/mirrors/next/linux-next'] * git branch: master * git commit: 04fe99a8d936d46a310ca61b8b63dc270962bf01 * git describe: next-20211207 * test details: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20211207 ## Test Regressions (compared to next-20211117) * dragonboard-410c, kselftest-capabilities - capabilities.test_execve * dragonboard-410c, kselftest-pidfd - pidfd.pidfd_poll_test * dragonboard-410c, kselftest-proc - proc.proc-fsconfig-hidepid * hi6220-hikey, kselftest-cgroup - cgroup.test_freezer * hi6220-hikey, kselftest-timers - timers.set-timer-lat * qemu_arm, kselftest-zram - zram.zram.sh * qemu_i386, kselftest-rtc - rtc.rtctest * x15, kselftest-capabilities - capabilities.test_execve * x15, kselftest-cgroup - cgroup.test_freezer - cgroup.test_kill - cgroup.test_kill.test_cgkill_simple * x15, kselftest-rtc - rtc.rtctest.rtc.alarm_alm_set - rtc.rtctest.rtc.alarm_alm_set_minute - rtc.rtctest.rtc.alarm_wkalm_set - rtc.rtctest.rtc.date_read * x86, kselftest-kvm - kvm.vmx_pmu_msrs_test ## Metric Regressions (compared to next-20211117) No metric regressions found. Reported-by: Linux Kernel Functional Testing <lkft(a)linaro.org> ## Test Fixes (compared to next-20211117) * hi6220-hikey, kselftest-timers - timers.nsleep-lat * i386, kselftest-rtc - rtc.rtctest * qemu_arm, kselftest-rtc - rtc.rtctest * qemu_arm, kselftest-timers - timers.rtcpie * qemu_x86_64, kselftest-timers - timers.rtcpie * x15, kselftest-core - core.close_range_test * x15, kselftest-sync - sync.sync_test * x86, kselftest-rtc - rtc.rtctest ## Metric Fixes (compared to next-20211117) No metric fixes found. ## Test result summary total: 3578, pass: 1828, fail: 395, skip: 1355, xfail: 0 ## Build Summary ## Test suites summary * kselftest-android * kselftest-arm64 * kselftest-arm64/arm64.btitest.bti_c_func * kselftest-arm64/arm64.btitest.bti_j_func * kselftest-arm64/arm64.btitest.bti_jc_func * kselftest-arm64/arm64.btitest.bti_none_func * kselftest-arm64/arm64.btitest.nohint_func * kselftest-arm64/arm64.btitest.paciasp_func * kselftest-arm64/arm64.nobtitest.bti_c_func * kselftest-arm64/arm64.nobtitest.bti_j_func * kselftest-arm64/arm64.nobtitest.bti_jc_func * kselftest-arm64/arm64.nobtitest.bti_none_func * kselftest-arm64/arm64.nobtitest.nohint_func * kselftest-arm64/arm64.nobtitest.paciasp_func * kselftest-bpf * kselftest-breakpoints * kselftest-capabilities * kselftest-cgroup * kselftest-clone3 * kselftest-core * kselftest-cpu-hotplug * kselftest-cpufreq * kselftest-drivers * kselftest-efivarfs * kselftest-filesystems * kselftest-firmware * kselftest-fpu * kselftest-futex * kselftest-gpio * kselftest-intel_pstate * kselftest-ipc * kselftest-ir * kselftest-kcmp * kselftest-kexec * kselftest-kvm * kselftest-lib * kselftest-livepatch * kselftest-lkdtm * kselftest-membarrier * kselftest-memfd * kselftest-memory-hotplug * kselftest-mincore * kselftest-mount * kselftest-mqueue * kselftest-openat2 * kselftest-pid_namespace * kselftest-pidfd * kselftest-proc * kselftest-pstore * kselftest-ptrace * kselftest-rseq * kselftest-rtc * kselftest-seccomp * kselftest-sigaltstack * kselftest-size * kselftest-splice * kselftest-static_keys * kselftest-sync * kselftest-sysctl * kselftest-timens * kselftest-timers * kselftest-tmpfs * kselftest-tpm2 * kselftest-user * kselftest-vm * kselftest-x86 * kselftest-zram -- Linaro LKFT https://lkft.linaro.org

4 years

1
0
0 0

[REGRESSION] lkft kselftest for next-20211206

by lkft＠linaro.org

## Build * kernel: 5.16.0-rc4 * git: https://gitlab.com/Linaro/lkft/mirrors/next/linux-next * git branch: master * git commit: 5d02ef4b57f6e7d4dcba14d40cf05373a146a605 * git describe: next-20211206 * test details: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20211206 ## Test Regressions (compared to next-20211117) * dragonboard-410c, kselftest-capabilities - capabilities.test_execve * dragonboard-410c, kselftest-seccomp - seccomp.seccomp_bpf - seccomp.seccomp_bpf.global.user_notification_filter_empty * dragonboard-410c, kselftest-timers - timers.set-timer-lat * qemu_arm, kselftest-zram - zram.zram.sh * qemu_i386, kselftest-cgroup - cgroup.test_freezer - cgroup.test_freezer.test_cgfreezer_ptrace * qemu_i386, kselftest-rtc - rtc.rtctest * qemu_x86_64, kselftest-kvm - kvm.rseq_test * x15, kselftest-capabilities - capabilities.test_execve * x15, kselftest-cgroup - cgroup.test_freezer - cgroup.test_kill - cgroup.test_kill.test_cgkill_simple * x15, kselftest-rtc - rtc.rtctest.rtc.alarm_alm_set - rtc.rtctest.rtc.alarm_alm_set_minute - rtc.rtctest.rtc.alarm_wkalm_set - rtc.rtctest.rtc.date_read * x15, kselftest-timers - timers.nsleep-lat - timers.rtcpie * x86, kselftest-kvm - kvm.vmx_pmu_msrs_test ## Metric Regressions (compared to next-20211117) No metric regressions found. Reported-by: Linux Kernel Functional Testing <lkft(a)linaro.org> ## Test Fixes (compared to next-20211117) * i386, kselftest-rtc - rtc.rtctest * qemu_x86_64, kselftest-timers - timers.rtcpie * x15, kselftest-core - core.close_range_test * x15, kselftest-rtc - rtc.rtctest * x15, kselftest-sync - sync.sync_test ## Metric Fixes (compared to next-20211117) No metric fixes found. ## Test result summary total: 3574, pass: 1837, fail: 405, skip: 1332, xfail: 0 ## Build Summary ## Test suites summary * kselftest-android * kselftest-arm64 * kselftest-arm64/arm64.btitest.bti_c_func * kselftest-arm64/arm64.btitest.bti_j_func * kselftest-arm64/arm64.btitest.bti_jc_func * kselftest-arm64/arm64.btitest.bti_none_func * kselftest-arm64/arm64.btitest.nohint_func * kselftest-arm64/arm64.btitest.paciasp_func * kselftest-arm64/arm64.nobtitest.bti_c_func * kselftest-arm64/arm64.nobtitest.bti_j_func * kselftest-arm64/arm64.nobtitest.bti_jc_func * kselftest-arm64/arm64.nobtitest.bti_none_func * kselftest-arm64/arm64.nobtitest.nohint_func * kselftest-arm64/arm64.nobtitest.paciasp_func * kselftest-bpf * kselftest-breakpoints * kselftest-capabilities * kselftest-cgroup * kselftest-clone3 * kselftest-core * kselftest-cpu-hotplug * kselftest-cpufreq * kselftest-drivers * kselftest-efivarfs * kselftest-filesystems * kselftest-firmware * kselftest-fpu * kselftest-futex * kselftest-gpio * kselftest-intel_pstate * kselftest-ipc * kselftest-ir * kselftest-kcmp * kselftest-kexec * kselftest-kvm * kselftest-lib * kselftest-livepatch * kselftest-lkdtm * kselftest-membarrier * kselftest-memfd * kselftest-memory-hotplug * kselftest-mincore * kselftest-mount * kselftest-mqueue * kselftest-openat2 * kselftest-pid_namespace * kselftest-pidfd * kselftest-proc * kselftest-pstore * kselftest-ptrace * kselftest-rseq * kselftest-rtc * kselftest-seccomp * kselftest-sigaltstack * kselftest-size * kselftest-splice * kselftest-static_keys * kselftest-sync * kselftest-sysctl * kselftest-timens * kselftest-timers * kselftest-tmpfs * kselftest-tpm2 * kselftest-user * kselftest-vm * kselftest-x86 * kselftest-zram -- Linaro LKFT https://lkft.linaro.org

4 years

1
0
0 0

./kselftest/prefix.pl read from pipe will block until the front command and its child processes all exit

by lizhijian＠fujitsu.com

Hi folks, Kees This issue confuses the LKP/0Day robot for a long time. Take below script as an example: lizj@FNSTPC:~/workspace/colo/linux/tools/testing/selftests$ cat a.sh #!/bin/bash sleep 10 & echo 1000000000000000 lizj@FNSTPC:~/workspace/colo/linux/tools/testing/selftests$ time ./a.sh | ./kselftest/prefix.pl # 1000000000000000 real 0m10.004s user 0m0.012s sys 0m0.001s ------------------------------------ Although the first script already exited, ./kselftest/prefix.pl will block until "sleep 10 &" exit. That means once some of the child process cannot finish/exit itself, for example, one test becomes *zombie* for some reasons, the whole ksefltest framework will hang forever. In addition, currently ksefltest timeout scheme[1][2] will not signal/kil the child processes, that make the blocking often happens. Since i'm not familiar with perl, not sure whether it can finish itself *directly* when first/front command(excluding child processes) exits. [1]: https://lkml.org/lkml/2021/12/17/140 [2]: http://lkml.iu.edu/hypermail/linux/kernel/2004.1/02379.html $ man timeout --foreground when not running timeout directly from a shell prompt, allow COMMAND to read from the TTY and get TTY signals; in this mode, children of COMMAND will not be timed out Thanks Zhijian

4 years

1
0
0 0

[RFC PATCH v2] Documentation: dev-tools: Add KTAP specification

by David Gow

From: Rae Moar <rmoar(a)google.com> It does not make any significant additions or changes other than those already in use in the kernel: additional features can be added as they become necessary and used. [1]: https://testanything.org/tap-version-13-specification.html Signed-off-by: Rae Moar <rmoar(a)google.com> Co-developed-by: David Gow <davidgow(a)google.com> Signed-off-by: David Gow <davidgow(a)google.com> --- Changes since RFC v1: https://lore.kernel.org/linux-kselftest/20211203064840.2871751-1-davidgow@g… - Add a "see also" section with some useful links. - Remove the XPASS directive, which isn't used anywhere. - Clear up / reorganise the discussion around differences between KTAP and TAP14. - Improve the wording around some directives. - Fix a bunch of typos. See prior discussion in the following RFC: https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqa…. --- Documentation/dev-tools/index.rst | 1 + Documentation/dev-tools/ktap.rst | 298 ++++++++++++++++++++++++++++++ 2 files changed, 299 insertions(+) create mode 100644 Documentation/dev-tools/ktap.rst diff --git a/Documentation/dev-tools/index.rst b/Documentation/dev-tools/index.rst index 010a2af1e7d9..4621eac290f4 100644 --- a/Documentation/dev-tools/index.rst +++ b/Documentation/dev-tools/index.rst @@ -32,6 +32,7 @@ Documentation/dev-tools/testing-overview.rst kgdb kselftest kunit/index + ktap .. only:: subproject and html diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst new file mode 100644 index 000000000000..878530cb9c27 --- /dev/null +++ b/Documentation/dev-tools/ktap.rst @@ -0,0 +1,298 @@ +.. SPDX-License-Identifier: GPL-2.0 + +======================================== +The Kernel Test Anything Protocol (KTAP) +======================================== + +TAP, or the Test Anything Protocol is a format for specifying test results used +by a number of projects. It's website and specification are found at this `link +<https://testanything.org/>`_. The Linux Kernel largely uses TAP output for test +results. However, Kernel testing frameworks have special needs for test results +which don't align with the original TAP specification. Thus, a "Kernel TAP" +(KTAP) format is specified to extend and alter TAP to support these use-cases. +This specification describes the generally accepted format of KTAP as it is +currently used in the kernel. + +KTAP test results describe a series of tests (which may be nested: i.e., test +can have subtests), each of which can contain both diagnostic data -- e.g., log +lines -- and a final result. The test structure and results are +machine-readable, whereas the diagnostic data is unstructured and is there to +aid human debugging. + +KTAP output is built from four different types of lines: +- Version lines +- Plan lines +- Test case result lines +- Diagnostic lines + +In general, valid KTAP output should also form valid TAP output, but some +information, in particular nested test results, may be lost. Also note that +there is a stagnant draft specification for TAP14, KTAP diverges from this in +a couple of places (notably the "Subtest" header), which are described where +relevant later in this document. + +Version lines +------------- + +All KTAP-formatted results begin with a "version line" which specifies which +version of the (K)TAP standard the result is compliant with. + +For example: +- "KTAP version 1" +- "TAP version 13" +- "TAP version 14" + +Note that, in KTAP, subtests also begin with a version line, which denotes the +start of the nested test results. This differs from TAP14, which uses a +separate "Subtest" line. + +While, going forward, "KTAP version 1" should be used by compliant tests, it +is expected that most parsers and other tooling will accept the other versions +listed here for compatibility with existing tests and frameworks. + +Plan lines +---------- + +A test plan provides the number of tests (or subtests) in the KTAP output. + +Plan lines must follow the format of "1..N" where N is the number of tests or subtests. +Plan lines follow version lines to indicate the number of nested tests. + +While there are cases where the number of tests is not known in advance -- in +which case the test plan may be omitted -- it is strongly recommended one is +present where possible. + +Test case result lines +---------------------- + +Test case result lines indicate the final status of a test. +They are required and must have the format: + +.. code-block:: + + <result> <number> [<description>][ # [<directive>] [<diagnostic data>]] + +The result can be either "ok", which indicates the test case passed, +or "not ok", which indicates that the test case failed. + +<number> represents the number of the test being performed. The first test must +have the number 1 and the number then must increase by 1 for each additional +subtest within the same test at the same nesting level. + +The description is a description of the test, generally the name of +the test, and can be any string of words (can't include #). The +description is optional, but recommended. + +The directive and any diagnostic data is optional. If either are present, they +must follow a hash sign, "#". + +A directive is a keyword that indicates a different outcome for a test other +than passed and failed. The directive is optional, and consists of a single +keyword preceding the diagnostic data. In the event that a parser encounters +a directive it doesn't support, it should fall back to the "ok" / "not ok" +result. + +Currently accepted directives are: + +- "SKIP", which indicates a test was skipped (note the result of the test case + result line can be either "ok" or "not ok" if the SKIP directive is used) +- "TODO", which indicates that a test is not expected to pass at the moment, + e.g. because the feature it is testing is known to be broken. While this + directive is inherited from TAP, its use in the kernel is discouraged. +- "XFAIL", which indicates that a test is expected to fail. This is similar + to "TODO", above, and is used by some kselftest tests. +- “TIMEOUT”, which indicates a test has timed out (note the result of the test + case result line should be “not ok” if the TIMEOUT directive is used) +- “ERROR”, which indicates that the execution of a test has failed due to a + specific error that is included in the diagnostic data. (note the result of + the test case result line should be “not ok” if the ERROR directive is used) + +The diagnostic data is a plain-text field which contains any additional details +about why this result was produced. This is typically an error message for ERROR +or failed tests, or a description of missing dependencies for a SKIP result. + +The diagnostic data field is optional, and results which have neither a +directive nor any diagnostic data do not need to include the "#" field +separator. + +Example result lines include: + +.. code-block:: + + ok 1 test_case_name + +The test "test_case_name" passed. + +.. code-block:: + + not ok 1 test_case_name + +The test "test_case_name" failed. + +.. code-block:: + + ok 1 test # SKIP necessary dependency unavailable + +The test "test" was SKIPPED with the diagnostic message "necessary dependency +unavailable". + +.. code-block:: + + not ok 1 test # TIMEOUT 30 seconds + +The test "test" timed out, with diagnostic data "30 seconds". + +.. code-block:: + + ok 5 check return code # rcode=0 + +The test "check return code" passed, with additional diagnostic data “rcode=0” + + +Diagnostic lines +---------------- + +If tests wish to output any further information, they should do so using +"diagnostic lines". Diagnostic lines are optional, freeform text, and are +often used to describe what is being tested and any intermediate results in +more detail than the final result and diagnostic data line provides. + +Diagnostic lines are formatted as "# <diagnostic_description>", where the +description can be any string. Diagnostic lines can be anywhere in the test +output. As a rule, diagnostic lines regarding a test are directly before the +test result line for that test. + +Note that most tools will treat unknown lines (see below) as diagnostic lines, +even if they do not start with a "#": this is to capture any other useful +kernel output which may help debug the test. It is nevertheless recommended +that tests always prefix any diagnostic output they have with a "#" character. + +Unknown lines +------------- + +There may be lines within KTAP output that do not follow the format of one of +the four formats for lines described above. This is allowed, however, they will +not influence the status of the tests. + +Nested tests +------------ + +In KTAP, tests can be nested. This is done by having a test include within its +output an entire set of KTAP-formatted results. This can be used to categorize +and group related tests, or to split out different results from the same test. + +The "parent" test's result should consist of all of its subtests' results, +starting with another KTAP version line and test plan, and end with the overall +result. If one of the subtests fail, for example, the parent test should also +fail. + +Additionally, all result lines in a subtest should be indented. One level of +indentation is two spaces: " ". The indentation should begin at the version +line and should end before the parent test's result line. + +An example of a test with two nested subtests: + +.. code-block:: + + KTAP version 1 + 1..1 + KTAP version 1 + 1..2 + ok 1 test_1 + not ok 2 test_2 + # example failed + not ok 1 example + +An example format with multiple levels of nested testing: + +.. code-block:: + + KTAP version 1 + 1..2 + KTAP version 1 + 1..2 + KTAP version 1 + 1..2 + not ok 1 test_1 + ok 2 test_2 + not ok 1 test_3 + ok 2 test_4 # SKIP + not ok 1 example_test_1 + ok 2 example_test_2 + + +Major differences between TAP and KTAP +-------------------------------------- + +Note the major differences between the TAP and KTAP specification: +- yaml and json are not recommended in diagnostic messages +- TODO directive not recognized +- KTAP allows for an arbitrary number of tests to be nested + +The TAP14 specification does permit nested tests, but instead of using another +nested version line, uses a line of the form +"Subtest: <name>" where <name> is the name of the parent test. + +Example KTAP output +-------------------- +.. code-block:: + + KTAP version 1 + 1..1 + KTAP version 1 + 1..3 + KTAP version 1 + 1..1 + # test_1: initializing test_1 + ok 1 test_1 + ok 1 example_test_1 + KTAP version 1 + 1..2 + ok 1 test_1 # SKIP test_1 skipped + ok 2 test_2 + ok 2 example_test_2 + KTAP version 1 + 1..3 + ok 1 test_1 + # test_2: FAIL + not ok 2 test_2 + ok 3 test_3 # SKIP test_3 skipped + not ok 3 example_test_3 + not ok 1 main_test + +This output defines the following hierarchy: + +A single test called "main_test", which fails, and has three subtests: +- "example_test_1", which passes, and has one subtest: + + - "test_1", which passes, and outputs the diagnostic message "test_1: initializing test_1" + +- "example_test_2", which passes, and has two subtests: + + - "test_1", which is skipped, with the explanation "test_1 skipped" + - "test_2", which passes + +- "example_test_3", which fails, and has three subtests + + - "test_1", which passes + - "test_2", which outputs the diagnostic line "test_2: FAIL", and fails. + - "test_3", which is skipped with the explanation "test_3 skipped" + +Note that the individual subtests with the same names do not conflict, as they +are found in different parent tests. This output also exhibits some sensible +rules for "bubbling up" test results: a test fails if any of its subtests fail. +Skipped tests do not affect the result of the parent test (though it often +makes sense for a test to be marked skipped if _all_ of its subtests have been +skipped). + +See also: +--------- + +- The TAP specification: + https://testanything.org/tap-version-13-specification.html +- The (stagnant) TAP version 14 specification: + https://github.com/TestAnything/Specification/blob/tap-14-specification/spe… +- The kselftest documentation: + Documentation/dev-tools/kselftest.rst +- The KUnit documentation: + Documentation/dev-tools/kunit/index.rst -- 2.34.1.400.ga245620fadb-goog

4 years

5
7
0 0

[PATCH][next] bpf, selftests: Fix spelling mistake "tained" -> "tainted"

by Colin Ian King

There appears to be a spelling mistake in a bpf test message. Fix it. Signed-off-by: Colin Ian King <colin.i.king(a)gmail.com> --- tools/testing/selftests/bpf/verifier/value_ptr_arith.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/verifier/value_ptr_arith.c b/tools/testing/selftests/bpf/verifier/value_ptr_arith.c index 4d347bc53aa2..359f3e8f8b60 100644 --- a/tools/testing/selftests/bpf/verifier/value_ptr_arith.c +++ b/tools/testing/selftests/bpf/verifier/value_ptr_arith.c @@ -1078,7 +1078,7 @@ .errstr_unpriv = "R0 pointer -= pointer prohibited", }, { - "map access: trying to leak tained dst reg", + "map access: trying to leak tainted dst reg", .insns = { BPF_MOV64_IMM(BPF_REG_0, 0), BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), -- 2.33.1

4 years

2
1
0 0

[PATCH 1/2] kselftest: signal all child processes

by Li Zhijian

We have some many cases that will create child process as well, such as pidfd_wait. Previously, we will signal/kill the parent process when it is time out, but this signal will not be sent to its child process. In such case, if child process doesn't terminate itself, ksefltest framework will hang forever. below ps tree show the situation when ksefltest is blocking: root 1172 0.0 0.0 5996 2500 ? S 07:03 0:00 \_ /bin/bash /lkp/lkp/src/tests/kernel-selftests root 1216 0.0 0.0 4392 1976 ? S 07:03 0:00 \_ make run_tests -C pidfd root 1218 0.0 0.0 2396 1652 ? S 07:03 0:00 \_ /bin/sh -c BASE_DIR="/usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests"; . /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/kselftest/runner.sh; if [ "X" != "X" ]; then per_test_logging=1; fi; run_many /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_test /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_fdinfo_test /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_open_test /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_poll_test /usr/src/perf_selftests-x86_64-rhel- 8. root 12491 0.0 0.0 2396 132 ? S 07:03 0:00 \_ /bin/sh -c BASE_DIR="/usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests"; . /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/kselftest/runner.sh; if [ "X" != "X" ]; then per_test_logging=1; fi; run_many /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_test /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_fdinfo_test /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_open_test /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_poll_test /usr/src/perf_selftests-x86_64-r he root 12492 0.0 0.0 2396 132 ? S 07:03 0:00 \_ /bin/sh -c BASE_DIR="/usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests"; . /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/kselftest/runner.sh; if [ "X" != "X" ]; then per_test_logging=1; fi; run_many /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_test /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_fdinfo_test /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_open_test /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_poll_test /usr/src/perf_selftests-x86_ 64 root 12493 0.0 0.0 2396 132 ? S 07:03 0:00 \_ /bin/sh -c BASE_DIR="/usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests"; . /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/kselftest/runner.sh; if [ "X" != "X" ]; then per_test_logging=1; fi; run_many /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_test /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_fdinfo_test /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_open_test /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_poll_test /usr/src/perf_selftests- x8 root 12496 0.0 0.0 2396 132 ? S 07:03 0:00 \_ /bin/sh -c BASE_DIR="/usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests"; . /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/kselftest/runner.sh; if [ "X" != "X" ]; then per_test_logging=1; fi; run_many /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_test /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_fdinfo_test /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_open_test /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/pidfd/pidfd_poll_test /usr/src/perf_selfte st root 12498 0.0 0.0 10564 6116 ? S 07:03 0:00 \_ perl /usr/src/perf_selftests-x86_64-rhel-8.3-kselftests-519d81956ee277b4419c723adfb154603c2565ba/tools/testing/selftests/kselftest/prefix.pl root 12503 0.0 0.0 2452 112 ? T 07:03 0:00 ./pidfd_wait root 12621 0.0 0.0 2372 1600 ? SLs 07:04 0:00 /usr/sbin/watchdog root 19438 0.0 0.0 992 60 ? Ss 07:39 0:00 /lkp/lkp/src/bin/event/wakeup activate-monitor Here we group all its child processes so that kill() can signal all of them in timeout. CC: Kees Cook <keescook(a)chromium.org> CC: Andy Lutomirski <luto(a)amacapital.net> CC: Will Drewry <wad(a)chromium.org> CC: Shuah Khan <shuah(a)kernel.org> CC: Christian Brauner <christian(a)brauner.io> CC: Philip Li <philip.li(a)intel.com> Suggested-by: yang xu <xuyang2018.jy(a)cn.fujitsu.com> Signed-off-by: Li Zhijian <lizhijian(a)cn.fujitsu.com> --- tools/testing/selftests/kselftest_harness.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/kselftest_harness.h b/tools/testing/selftests/kselftest_harness.h index ae0f0f33b2a6..c7251396e7ee 100644 --- a/tools/testing/selftests/kselftest_harness.h +++ b/tools/testing/selftests/kselftest_harness.h @@ -875,7 +875,8 @@ static void __timeout_handler(int sig, siginfo_t *info, void *ucontext) } t->timed_out = true; - kill(t->pid, SIGKILL); + // signal process group + kill(-(t->pid), SIGKILL); } void __wait_for_test(struct __test_metadata *t) @@ -985,6 +986,7 @@ void __run_test(struct __fixture_metadata *f, ksft_print_msg("ERROR SPAWNING TEST CHILD\n"); t->passed = 0; } else if (t->pid == 0) { + setpgrp(); t->fn(t, variant); if (t->skip) _exit(255); -- 2.33.0

4 years

4
9
0 0

[PATCH v4 0/7] Documentation: KUnit: Rework KUnit documentation

by Harinder Singh

The KUnit documentation was not very organized. There was little information related to KUnit architecture and the importance of unit testing. Add some new pages, expand and reorganize the existing documentation. Reword pages to make information and style more consistent. Changes since v3: https://lore.kernel.org/linux-kselftest/20211210052812.1998578-1-sharinder@… --Reworded sentences as per comments --Replaced Elixir links with kernel.org links or kernel-doc references Changes since v2: https://lore.kernel.org/linux-kselftest/20211207054019.1455054-1-sharinder@… --Reworded sentences as per comments --Expanded the explaination in usage.rst for accessing the current test example --Standardized on US english in style.rst Changes since v1: https://lore.kernel.org/linux-kselftest/20211203042437.740255-1-sharinder@g… --Fixed spelling mistakes --Restored paragraph about kunit_tool introduction --Added note about CONFIG_KUNIT_ALL_TESTS (Thanks Tim Bird for review comments) -- Miscellaneous changes Harinder Singh (7): Documentation: KUnit: Rewrite main page Documentation: KUnit: Rewrite getting started Documentation: KUnit: Added KUnit Architecture Documentation: kunit: Reorganize documentation related to running tests Documentation: KUnit: Rework writing page to focus on writing tests Documentation: KUnit: Restyle Test Style and Nomenclature page Documentation: KUnit: Restyled Frequently Asked Questions .../dev-tools/kunit/architecture.rst | 204 +++++++ Documentation/dev-tools/kunit/faq.rst | 73 ++- Documentation/dev-tools/kunit/index.rst | 172 +++--- .../kunit/kunit_suitememorydiagram.png | Bin 0 -> 24174 bytes Documentation/dev-tools/kunit/run_manual.rst | 57 ++ Documentation/dev-tools/kunit/run_wrapper.rst | 247 ++++++++ Documentation/dev-tools/kunit/start.rst | 198 +++--- Documentation/dev-tools/kunit/style.rst | 105 ++-- Documentation/dev-tools/kunit/usage.rst | 578 ++++++++---------- 9 files changed, 1047 insertions(+), 587 deletions(-) create mode 100644 Documentation/dev-tools/kunit/architecture.rst create mode 100644 Documentation/dev-tools/kunit/kunit_suitememorydiagram.png create mode 100644 Documentation/dev-tools/kunit/run_manual.rst create mode 100644 Documentation/dev-tools/kunit/run_wrapper.rst base-commit: 4c388a8e740d3235a194f330c8ef327deef710f6 -- 2.34.1.173.g76aa8bc2d0-goog

4 years

4
12
0 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

Linux-kselftest-mirror December 2021