Linux-kselftest-mirror

linux-kselftest-mirror@lists.linaro.org

15 participants
14189 discussions

[PATCH v8 0/6] cgroup/cpuset: Add new cpuset partition type & empty effecitve cpus

by Waiman Long

v8: - Reorganize the patch series and rationalize the features and constraints of a partition. - Update patch descriptions and documentation accordingly. v7: - Simplify the documentation patch (patch 5) as suggested by Tejun. - Fix a typo in patch 2 and improper commit log in patch 3. v6: - Remove duplicated tmpmask from update_prstate() which should fix the frame size too large problem reported by kernel test robot. This patchset makes four enhancements to the cpuset v2 code. Patch 1: Enable partition with no task to have empty cpuset.cpus.effective. Patch 2: Refining the features and constraints of a cpuset partition clarifying what changes are allowed. Patch 3: Add a new partition state "isolated" to create a partition root without load balancing. This is for handling intermitten workloads that have a strict low latency requirement. Patch 4: Enable the "cpuset.cpus.partition" file to show the reason that causes invalid partition like "root invalid (No cpu available due to hotplug)". Patch 5 updates the cgroup-v2.rst file accordingly. Patch 6 adds a new cpuset test to test the new cpuset partition code. Waiman Long (6): cgroup/cpuset: Allow no-task partition to have empty cpuset.cpus.effective cgroup/cpuset: Refining features and constraints of a partition cgroup/cpuset: Add a new isolated cpus.partition type cgroup/cpuset: Show invalid partition reason string cgroup/cpuset: Update description of cpuset.cpus.partition in cgroup-v2.rst kselftest/cgroup: Add cpuset v2 partition root state test Documentation/admin-guide/cgroup-v2.rst | 153 ++-- kernel/cgroup/cpuset.c | 393 +++++++---- tools/testing/selftests/cgroup/Makefile | 5 +- .../selftests/cgroup/test_cpuset_prs.sh | 664 ++++++++++++++++++ tools/testing/selftests/cgroup/wait_inotify.c | 87 +++ 5 files changed, 1115 insertions(+), 187 deletions(-) create mode 100755 tools/testing/selftests/cgroup/test_cpuset_prs.sh create mode 100644 tools/testing/selftests/cgroup/wait_inotify.c -- 2.27.0

4 years, 1 month

[PATCH] selftests/kselftest/runner.sh: Add optional command parameters in settings

by Cristian Marussi

Some testcases allow for optional commandline parameters but as of now there is now way to provide such arguments to the runner script. Add support to the per-test-directory "settings" file to provide such optional arguments; two new optional fields can now be defined in "settings": - args="<options>": general arguments common to all testcase commands in the test directory - <BASENAME_TEST>_args="<options>": custom arguments specific to only one specific testcase command Signed-off-by: Cristian Marussi <cristian.marussi(a)arm.com> --- Used to configure the use of a specific rtc device on CI systems with: tools/testing/selftests/rtc/settings: timeout=90 rtctest_args="/dev/rtc1" --- tools/testing/selftests/kselftest/runner.sh | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh index a9ba782d8ca0..f877a8571927 100644 --- a/tools/testing/selftests/kselftest/runner.sh +++ b/tools/testing/selftests/kselftest/runner.sh @@ -49,6 +49,15 @@ run_one() # Reset any "settings"-file variables. export kselftest_timeout="$kselftest_default_timeout" + + # Optional arguments for any command, possibly defined in settings + # as args="<options>" + kselftest_args="" + + # Optional arguments for this command, possibly defined in settings + # as <$BASENAME_TEST>_args="<options>" + kselftest_cmd_args_ref="kselftest_${BASENAME_TEST}_args" + # Load per-test-directory kselftest "settings" file. settings="$BASE_DIR/$DIR/settings" if [ -r "$settings" ] ; then @@ -69,7 +78,8 @@ run_one() echo "# Warning: file $TEST is missing!" echo "not ok $test_num $TEST_HDR_MSG" else - cmd="./$BASENAME_TEST" + eval kselftest_cmd_args="\$$kselftest_cmd_args_ref" + cmd="./$BASENAME_TEST $kselftest_cmd_args $kselftest_args" if [ ! -x "$TEST" ]; then echo "# Warning: file $TEST is not executable" -- 2.17.1

4 years, 1 month

[PATCH] tests: remove unneeded conversion to bool

by davidcomponentone＠gmail.com

From: Yang Guang <yang.guang5(a)zte.com.cn> The coccinelle report ./tools/testing/selftests/move_mount_set_group/move_mount_set_group_test.c:225:18-23: WARNING: conversion to bool not needed here Relational and logical operators evaluate to bool, explicit conversion is overly verbose and unneeded. Reported-by: Zeal Robot <zealci(a)zte.com.cn> Signed-off-by: Yang Guang <yang.guang5(a)zte.com.cn> --- .../selftests/move_mount_set_group/move_mount_set_group_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/move_mount_set_group/move_mount_set_group_test.c b/tools/testing/selftests/move_mount_set_group/move_mount_set_group_test.c index 860198f83a53..80ff0b692486 100644 --- a/tools/testing/selftests/move_mount_set_group/move_mount_set_group_test.c +++ b/tools/testing/selftests/move_mount_set_group/move_mount_set_group_test.c @@ -222,7 +222,7 @@ static int move_mount_set_group_supported(void) AT_FDCWD, SET_GROUP_TO, MOVE_MOUNT_SET_GROUP); umount2("/tmp", MNT_DETACH); - return ret < 0 ? false : true; + return ret >= 0; } FIXTURE(move_mount_set_group) { -- 2.30.2

4 years, 1 month

[PATCH selftests] selftests: harness: avoid false negatives if test has no ASSERTs

by Jakub Kicinski

Test can fail either immediately when ASSERT() failed or at the end if one or more EXPECT() was not met. The exact return code is decided based on the number of successful ASSERT()s. If test has no ASSERT()s, however, the return code will be 0, as if the test did not fail. Start counting ASSERT()s from 1. Fixes: 369130b63178 ("selftests: Enhance kselftest_harness.h to print which assert failed") Signed-off-by: Jakub Kicinski <kuba(a)kernel.org> --- tools/testing/selftests/kselftest_harness.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/kselftest_harness.h b/tools/testing/selftests/kselftest_harness.h index ae0f0f33b2a6..79a182cfa43a 100644 --- a/tools/testing/selftests/kselftest_harness.h +++ b/tools/testing/selftests/kselftest_harness.h @@ -969,7 +969,7 @@ void __run_test(struct __fixture_metadata *f, t->passed = 1; t->skip = 0; t->trigger = 0; - t->step = 0; + t->step = 1; t->no_print = 0; memset(t->results->reason, 0, sizeof(t->results->reason)); -- 2.31.1

4 years, 1 month

[PATCH 1/2] selftests: cgroup: build error multiple outpt files

by Anders Roxell

When building selftests/cgroup: with clang the following error are seen: clang -Wall -pthread test_memcontrol.c cgroup_util.c ../clone3/clone3_selftests.h -o /home/anders/.cache/tuxmake/builds/current/kselftest/cgroup/test_memcontrol clang: error: cannot specify -o when generating multiple output files make[3]: *** [../lib.mk:146: /home/anders/.cache/tuxmake/builds/current/kselftest/cgroup/test_memcontrol] Error 1 Rework to add the header files to LOCAL_HDRS before including ../lib.mk, since the dependency is evaluated in '$(OUTPUT)/%:%.c $(LOCAL_HDRS)' in file lib.mk. Suggested-by: Arnd Bergmann <arnd(a)arndb.de> Signed-off-by: Anders Roxell <anders.roxell(a)linaro.org> --- tools/testing/selftests/cgroup/Makefile | 12 +++++++----- tools/testing/selftests/lib.mk | 2 +- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/tools/testing/selftests/cgroup/Makefile b/tools/testing/selftests/cgroup/Makefile index 59e222460581..745fe25fa0b9 100644 --- a/tools/testing/selftests/cgroup/Makefile +++ b/tools/testing/selftests/cgroup/Makefile @@ -11,10 +11,12 @@ TEST_GEN_PROGS += test_core TEST_GEN_PROGS += test_freezer TEST_GEN_PROGS += test_kill +LOCAL_HDRS += $(selfdir)/clone3/clone3_selftests.h $(selfdir)/pidfd/pidfd.h + include ../lib.mk -$(OUTPUT)/test_memcontrol: cgroup_util.c ../clone3/clone3_selftests.h -$(OUTPUT)/test_kmem: cgroup_util.c ../clone3/clone3_selftests.h -$(OUTPUT)/test_core: cgroup_util.c ../clone3/clone3_selftests.h -$(OUTPUT)/test_freezer: cgroup_util.c ../clone3/clone3_selftests.h -$(OUTPUT)/test_kill: cgroup_util.c ../clone3/clone3_selftests.h ../pidfd/pidfd.h +$(OUTPUT)/test_memcontrol: cgroup_util.c +$(OUTPUT)/test_kmem: cgroup_util.c +$(OUTPUT)/test_core: cgroup_util.c +$(OUTPUT)/test_freezer: cgroup_util.c +$(OUTPUT)/test_kill: cgroup_util.c diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk index fe7ee2b0f29c..a40add31a2e3 100644 --- a/tools/testing/selftests/lib.mk +++ b/tools/testing/selftests/lib.mk @@ -141,7 +141,7 @@ endif # Selftest makefiles can override those targets by setting # OVERRIDE_TARGETS = 1. ifeq ($(OVERRIDE_TARGETS),) -LOCAL_HDRS := $(selfdir)/kselftest_harness.h $(selfdir)/kselftest.h +LOCAL_HDRS += $(selfdir)/kselftest_harness.h $(selfdir)/kselftest.h $(OUTPUT)/%:%.c $(LOCAL_HDRS) $(LINK.c) $(filter-out $(LOCAL_HDRS),$^) $(LDLIBS) -o $@ -- 2.33.0

4 years, 1 month

[PATCH 0/3] livepatch: Allow user to specify functions to search for on a stack

by Miroslav Benes

livepatch's consistency model requires that no live patched function must be found on any task's stack during a transition process after a live patch is applied. It is achieved by walking through stacks of all blocked tasks. The user might also want to define more functions to search for without them being patched at all. It may either help with preparing a live patch, which would otherwise require additional touches to achieve the consistency, or it can be used to overcome deficiencies the stack checking inherently has. For example, GCC may optimize a function so that a part of it is moved to a different section and the function would jump to it. This child function would not be found on a stack in this case, but it may be important to search for it so that, again, the consistency is achieved. The patch set adds a new API which allows the user to specify such functions. The first patch is only preparatory. The main work is in the second one. The third patch adds a test. Originally, I wanted to add it all to klp_patch struct, but it makes more sense to do it on klp_object level. It is also easier to reuse as much of the existing code as possible in that case. I am not good with naming so bike-shedding is welcome. Reviews even more. Miroslav Benes (3): livepatch: Move the initialization of old_func to a new function livepatch: Allow user to specify functions to search for on a stack selftests/livepatch: Test of the API for specifying functions to search for on a stack include/linux/livepatch.h | 11 +++ kernel/livepatch/core.c | 50 ++++++++--- kernel/livepatch/transition.c | 21 +++-- lib/Kconfig.debug | 1 + lib/livepatch/Makefile | 4 +- lib/livepatch/test_klp_funcstack_demo.c | 61 +++++++++++++ lib/livepatch/test_klp_funcstack_mod.c | 72 +++++++++++++++ tools/testing/selftests/livepatch/Makefile | 3 +- .../selftests/livepatch/test-func-stack.sh | 88 +++++++++++++++++++ 9 files changed, 293 insertions(+), 18 deletions(-) create mode 100644 lib/livepatch/test_klp_funcstack_demo.c create mode 100644 lib/livepatch/test_klp_funcstack_mod.c create mode 100755 tools/testing/selftests/livepatch/test-func-stack.sh -- 2.33.1

4 years, 1 month

[PATCH] selftests: net/fcnal-test.sh: add exit code

by Li Zhijian

Previously, the selftest framework always treats it as *ok* even though some of them are failed actually. That's because the script always returns 0. It supports PASS/FAIL/SKIP exit code now. CC: Philip Li <philip.li(a)intel.com> Reported-by: kernel test robot <lkp(a)intel.com> Signed-off-by: Li Zhijian <zhijianx.li(a)intel.com> --- tools/testing/selftests/net/fcnal-test.sh | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/tools/testing/selftests/net/fcnal-test.sh b/tools/testing/selftests/net/fcnal-test.sh index 3313566ce906..46297ceaa6d1 100755 --- a/tools/testing/selftests/net/fcnal-test.sh +++ b/tools/testing/selftests/net/fcnal-test.sh @@ -4077,3 +4077,11 @@ cleanup 2>/dev/null printf "\nTests passed: %3d\n" ${nsuccess} printf "Tests failed: %3d\n" ${nfail} + +if [ $nfail -ne 0 ]; then + exit 1 # KSFT_FAIL +elif [ $nsuccess -eq 0 ]; then + exit $ksft_skip +fi + +exit 0 # KSFT_PASS -- 2.32.0

4 years, 1 month

[PATCH v3 1/3] selftests/tc-testing: add exit code

by Li Zhijian

Mark the summary result as FAIL to prevent from confusing the selftest framework if some of them are failed. Previously, the selftest framework always treats it as *ok* even though some of them are failed actually. That's because the script tdc.sh always return 0. # All test results: # # 1..97 # ok 1 83be - Create FQ-PIE with invalid number of flows # ok 2 8b6e - Create RED with no flags [...snip] # ok 6 5f15 - Create RED with flags ECN, harddrop # ok 7 53e8 - Create RED with flags ECN, nodrop # ok 8 d091 - Fail to create RED with only nodrop flag # ok 9 af8e - Create RED with flags ECN, nodrop, harddrop # not ok 10 ce7d - Add mq Qdisc to multi-queue device (4 queues) # Could not match regex pattern. Verify command output: # qdisc mq 1: root # qdisc fq_codel 0: parent 1:4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 # qdisc fq_codel 0: parent 1:3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 [...snip] # ok 96 6979 - Change quantum of a strict ETS band # ok 97 9a7d - Change ETS strict band without quantum # # # # ok 1 selftests: tc-testing: tdc.sh <<< summary result CC: Philip Li <philip.li(a)intel.com> Reported-by: kernel test robot <lkp(a)intel.com> Signed-off-by: Li Zhijian <zhijianx.li(a)intel.com> Acked-by: Davide Caratti <dcaratti(a)redhat.com> Acked-by: Jamal Hadi Salim <jhs(a)mojatatu.com> --- V3: repost to netdev V2: Fix missing ':' --- tools/testing/selftests/tc-testing/tdc.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/tc-testing/tdc.py b/tools/testing/selftests/tc-testing/tdc.py index a3e43189d940..ee22e3447ec7 100755 --- a/tools/testing/selftests/tc-testing/tdc.py +++ b/tools/testing/selftests/tc-testing/tdc.py @@ -716,6 +716,7 @@ def set_operation_mode(pm, parser, args, remaining): list_test_cases(alltests) exit(0) + exit_code = 0 # KSFT_PASS if len(alltests): req_plugins = pm.get_required_plugins(alltests) try: @@ -724,6 +725,8 @@ def set_operation_mode(pm, parser, args, remaining): print('The following plugins were not found:') print('{}'.format(pde.missing_pg)) catresults = test_runner(pm, args, alltests) + if catresults.count_failures() != 0: + exit_code = 1 # KSFT_FAIL if args.format == 'none': print('Test results output suppression requested\n') else: @@ -748,6 +751,8 @@ def set_operation_mode(pm, parser, args, remaining): gid=int(os.getenv('SUDO_GID'))) else: print('No tests found\n') + exit_code = 4 # KSFT_SKIP + exit(exit_code) def main(): """ @@ -767,8 +772,5 @@ def main(): set_operation_mode(pm, parser, args, remaining) - exit(0) - - if __name__ == "__main__": main() -- 2.32.0

4 years, 1 month

[RFC PATCH] Documentation: dev-tools: Add KTAP specification

by David Gow

From: Rae Moar <rmoar(a)google.com> Document the kernel-specific variant of the Test Anything Protocol (TAP)[1] used by the kernel and kernel test frameworks. This version is called KTAP, and this version of the specification should describe how results produced by kselftest and KUnit (with some trivial changes to bring the two into alignment). It does not make any significant additions or changes other than those already in use in the kernel: additional features can be added as they become necessary and used. See prior discussion in the following RFC: https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqa…. [1]: https://testanything.org/tap-version-13-specification.html Signed-off-by: Rae Moar <rmoar(a)google.com> Co-developed-by: David Gow <davidgow(a)google.com> Signed-off-by: David Gow <davidgow(a)google.com> --- Hi all, This is a version of the KTAP RFC from [1], with a greater focus on being compatible with what's currently used, rather than prescribing a new format. The format here should be a superset of what KUnit and kselftest output, with a couple of notable exceptions: - KUnit currently uses the TAP14-style "Subtest" header instead of nested TAP version headers. This will be fixed in KUnit once this goes in. - The (unused) XPASS option is not mentioned (it'll devolve into a harmless comment) Cheers, -- David [1]https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJx… --- Documentation/dev-tools/index.rst | 1 + Documentation/dev-tools/ktap.rst | 287 ++++++++++++++++++++++++++++++ 2 files changed, 288 insertions(+) create mode 100644 Documentation/dev-tools/ktap.rst diff --git a/Documentation/dev-tools/index.rst b/Documentation/dev-tools/index.rst index 010a2af1e7d9..4621eac290f4 100644 --- a/Documentation/dev-tools/index.rst +++ b/Documentation/dev-tools/index.rst @@ -32,6 +32,7 @@ Documentation/dev-tools/testing-overview.rst kgdb kselftest kunit/index + ktap .. only:: subproject and html diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst new file mode 100644 index 000000000000..7c17956b998a --- /dev/null +++ b/Documentation/dev-tools/ktap.rst @@ -0,0 +1,287 @@ +.. SPDX-License-Identifier: GPL-2.0 + +======================================== +The Kernel Test Anything Protocol (KTAP) +======================================== + +TAP, or the Test Anything Protocol is a format for specifying test results used +by a number of projects. It's website and specification are found at this `link +<https://testanything.org/>`_. The Linux Kernel largely uses TAP output for test +results. However, Kernel testing frameworks have special needs for test results +which don't align with the original TAP specification. Thus, a "Kernel TAP" +(KTAP) format is specified to extend and alter TAP to support these use-cases. +This specification describes the generally accepted format of KTAP as it is +currently used in the kernel + +KTAP test results describe a series of tests (which may be nested: i.e., test +can have subtests), each of which can contain both diagnostic data -- e.g., log +lines -- and a final result. The test structure and results are +machine-readable, whereas the diagnostic data is unstructured and is there to +aid human debugging. + +KTAP output is built from four different types of lines: +- Version lines +- Plan lines +- Test case result lines +- Diagnostic lines + +Note that, in particular, nested test results work differently in KTAP from +the proposed TAP14 specification: +https://github.com/TestAnything/Specification/blob/tap-14-specification/specification.md + +Version lines +------------- + +All KTAP-formatted results begin with a "version line" which specifies which +version of the (K)TAP standard the result is compliant with. + +For example: +- "KTAP version 1" +- "TAP version 13" +- "TAP version 14" + +Note that, in KTAP, subtests also begin with a version line, which denotes the +start of the nested test results. This differs from TAP14, which uses a +separate "Subtest" line. + +While, going forward, "KTAP version 1" should be used by compliant tests, it +is expected that most parsers and other tooling will accept the other versions +listed here for compatibility with existing tests and frameworks. + +Plan lines +---------- + +A test plan provides the number of tests (or subtests) in the KTAP ouput. + +Plan lines must follow the format of "1..N" where N is the number of tests or subtests. +Plan lines follow version lines to indicate the number of nested tests. + +While there are cases where the number of tests is not known in advance -- in +which case the test plan may be omitted -- it is strongly recommended one is +present where possible. + +Test case result lines +---------------------- + +Test case result lines indicate the final status of a test. +They are required and must have the format: + +.. code-block:: + + <result> <number> [<description>][ # [<directive>] [<diagnostic data>]] + +The result can be either "ok", which indicates the test case passed, +or "not ok", which indicates that the test case failed. + +<number> represents the number of the test being performed. The first test must +have the number 1 and the number then must increase by 1 for each additional +subtest within the same test at the same nesting level. + +The description is a description of the test, generally the name of +the test, and can be any string of words (can't include #). The +description is optional, but recommended. + +The directive and any diagnostic data is optional. If either are present, they +must follow a hash sign, "#". + +A directive is a keyword that indicates a different outcome for a test other +than passed and failed. The directive is optional, and consists of a single +keyword preceding the diagnostic data. In the event that a parser encounters +a directive it doesn't support, it should fall back to the "ok" / "not ok" +result. + +Currently accepted directives are: + +- "SKIP", which indicates a test was skipped (note the result of the test case + result line can be either "ok" or "not ok" if the SKIP directive is used) +- "TODO", which indicates that a test is not expected to pass at the moment, + e.g. because the feature it is testing is known to be broken. While this + directive is inherited from TAP, its use in the kernel is discouraged. +- "XFAIL", which indicates that a test is expected to fail. This is similar + to "TODO", above, and is used by some kselftest tests. +- "XPASS", which indicated that a test is expected to pass. There is some code + which tracks this result, but it does not appear to currently be used. +- “TIMEOUT”, which indicates a test has timed out (note the result of the test + case result line should be “not ok” if the TIMEOUT directive is used) +- “ERROR”, which indicates a test has failed with a specific error that is + included in the diagnostic data (note the result of the test case result like + should be “not ok” if the ERROR directive is used) + +The diagnostic data is a plain-text field which contains any additional details +about why this result was produced. This is typically an error code for ERROR +or failed tests, or a description of missing dependencies for a SKIP result. + +The diagnostic data field is optional, and results which have neither a +directive nor any diagnostic data do not need to include the "#" field +separator. + +Example result lines include: + +.. code-block:: + + ok 1 test_case_name + +The test "test_case_name" passed. + +.. code-block:: + + not ok 1 test_case_name + +The test "test_case_name" failed. + +.. code-block:: + + ok 1 test # SKIP necessary dependency unavailable + +The test "test" was SKIPPED with the diagnostic message "necessary dependency +unavailable". + +.. code-block:: + + not ok 1 test # TIMEOUT 30 seconds + +The test "test" timed out, with diagnostic data "30 seconds". + +.. code-block:: + + ok 5 check return code # rcode=0 + +The test "check return code" passed, with additional diagnostic data is +“rcode=0” + + +Diagnostic lines +---------------- + +If tests wish to output any further information, they should do so using +"diagnostic lines". Diagnostic lines are optional, freeform text, and are +often used to describe what is being tested and any intermediate results in +more detail than the final result and diagnostic data line provides. + +Diagnostic lines are formatted as "# <diagnostic_description>", where the +description can be any string. Diagnostic lines can be anywhere in the test +output. As a rule, diagnostic lines regarding a test are directly before the +test result line for that test. + +Note that most tools will treat unknown lines (see below) as diagnostic lines, +even if they do not start with a "#": this is to capture any other useful +kernel output which may help debug the test. It is nevertheless recommended +that tests always prefix any diagnostic output they have with a "#" character. + +Unknown lines +------------- + +There may be lines within KTAP output that do not follow the format of one of +the four formats for lines described above. This is allowed, however, they will +not influence the status of the tests. + +Nested tests +------------ + +In KTAP, tests can be nested. This is done by having a test include within its +output an entire set of KTAP-formatted results. This can be used to categorize +and group related tests, or to split out different results from the same test. + +The "parent" test's result should consist of all of its subtests' results, +starting with another KTAP version line and test plan, and end with the overall +result. If one of the subtests fail, for example, the parent test should also +fail. + +Additionally, all result lines in a subtest should be indented. One level of +indentation is two spaces: " ". The indentation should begin at the version +line and should end before the parent test's result line. + +An example of a test with two nested subtests: + +.. code-block:: + + KTAP version 1 + 1..1 + KTAP version 1 + 1..2 + ok 1 test_1 + not ok 2 test_2 + # example failed + not ok 1 example + +An example format with multiple levels of nested testing: + +.. code-block:: + + KTAP version 1 + 1..2 + KTAP version 1 + 1..2 + KTAP version 1 + 1..2 + not ok 1 test_1 + ok 2 test_2 + not ok 1 test_3 + ok 2 test_4 # SKIP + not ok 1 example_test_1 + ok 2 example_test_2 + + +Major differences between TAP and KTAP +-------------------------------------- + +Note the major differences between the TAP and KTAP specification: +- yaml and json are not recommended in diagnostic messages +- TODO directive not recognized +- KTAP allows for an arbitrary number of tests to be nested + +The TAP14 specification does permit nested tests, but instead of using another +nested version line, uses a line of the form +"Subtest: <name>" where <name> is the name of the parent test. + +Example KTAP output +-------------------- +.. code-block:: + + KTAP version 1 + 1..1 + KTAP version 1 + 1..3 + KTAP version 1 + 1..1 + # test_1: initializing test_1 + ok 1 test_1 + ok 1 example_test_1 + KTAP version 1 + 1..2 + ok 1 test_1 # SKIP test_1 skipped + ok 2 test_2 + ok 2 example_test_2 + KTAP version 1 + 1..3 + ok 1 test_1 + # test_2: FAIL + not ok 2 test_2 + ok 3 test_3 # SKIP test_3 skipped + not ok 3 example_test_3 + not ok 1 main_test + +This output defines the following hierarchy: + +A single test called "main_test", which fails, and has three subtests: +- "example_test_1", which passes, and has one subtest: + + - "test_1", which passes, and outputs the dianostic message "test_1: initializing test_1" + +- "example_test_2", which passes, and has two subtests: + + - "test_1", which is skipped, with the explanation "test_1 skipped" + - "test_2", which passes + +- "example_test_3", which fails, and has three subtests + + - "test_1", which passes + - "test_2", which outputs the diagnostic line "test_2: FAIL", and fails. + - "test_3", which is skipped with the explanation "test_3 skipped" + +Note that the individual subtests with the same names do not conflict, as they +are found in different parent tests. This output also exhibits some sensible +rules for "bubbling up" test results: a test fails if any of its subtests fail. +Skipped tests do not affect the result of the parent test (though it often +makes sense for a test to be marked skipped if _all_ of its subtests have been +skipped). -- 2.34.1.400.ga245620fadb-goog

4 years, 1 month

[PATCH v2 1/3] selftests/tc-testing: add exit code

by Li Zhijian

Mark the summary result as FAIL to prevent from confusing the selftest framework if some of them are failed. Previously, the selftest framework always treats it as *ok* even though some of them are failed actually. That's because the script tdc.sh always return 0. # All test results: # # 1..97 # ok 1 83be - Create FQ-PIE with invalid number of flows # ok 2 8b6e - Create RED with no flags [...snip] # ok 6 5f15 - Create RED with flags ECN, harddrop # ok 7 53e8 - Create RED with flags ECN, nodrop # ok 8 d091 - Fail to create RED with only nodrop flag # ok 9 af8e - Create RED with flags ECN, nodrop, harddrop # not ok 10 ce7d - Add mq Qdisc to multi-queue device (4 queues) # Could not match regex pattern. Verify command output: # qdisc mq 1: root # qdisc fq_codel 0: parent 1:4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 # qdisc fq_codel 0: parent 1:3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 [...snip] # ok 96 6979 - Change quantum of a strict ETS band # ok 97 9a7d - Change ETS strict band without quantum # # # # ok 1 selftests: tc-testing: tdc.sh <<< summary result CC: Philip Li <philip.li(a)intel.com> Reported-by: kernel test robot <lkp(a)intel.com> Signed-off-by: Li Zhijian <zhijianx.li(a)intel.com> --- V2: Fix missing ':' --- tools/testing/selftests/tc-testing/tdc.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/tc-testing/tdc.py b/tools/testing/selftests/tc-testing/tdc.py index a3e43189d940..ee22e3447ec7 100755 --- a/tools/testing/selftests/tc-testing/tdc.py +++ b/tools/testing/selftests/tc-testing/tdc.py @@ -716,6 +716,7 @@ def set_operation_mode(pm, parser, args, remaining): list_test_cases(alltests) exit(0) + exit_code = 0 # KSFT_PASS if len(alltests): req_plugins = pm.get_required_plugins(alltests) try: @@ -724,6 +725,8 @@ def set_operation_mode(pm, parser, args, remaining): print('The following plugins were not found:') print('{}'.format(pde.missing_pg)) catresults = test_runner(pm, args, alltests) + if catresults.count_failures() != 0: + exit_code = 1 # KSFT_FAIL if args.format == 'none': print('Test results output suppression requested\n') else: @@ -748,6 +751,8 @@ def set_operation_mode(pm, parser, args, remaining): gid=int(os.getenv('SUDO_GID'))) else: print('No tests found\n') + exit_code = 4 # KSFT_SKIP + exit(exit_code) def main(): """ @@ -767,8 +772,5 @@ def main(): set_operation_mode(pm, parser, args, remaining) - exit(0) - - if __name__ == "__main__": main() -- 2.32.0

4 years, 1 month

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

Linux-kselftest-mirror