From: Rae Moar <rmoar(a)google.com>
Document the kernel-specific variant of the Test Anything Protocol
(TAP)[1] used by the kernel and kernel test frameworks. This version
is called KTAP, and this version of the specification should describe
how results produced by kselftest and KUnit (with some trivial changes
to bring the two into alignment).
It does not make any significant additions or changes other than those
already in use in the kernel: additional features can be added as they
become necessary and used.
See prior discussion in the following RFC:
https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqa….
[1]: https://testanything.org/tap-version-13-specification.html
Signed-off-by: Rae Moar <rmoar(a)google.com>
Co-developed-by: David Gow <davidgow(a)google.com>
Signed-off-by: David Gow <davidgow(a)google.com>
---
Hi all,
This is a version of the KTAP RFC from [1], with a greater focus on
being compatible with what's currently used, rather than prescribing a
new format. The format here should be a superset of what KUnit and
kselftest output, with a couple of notable exceptions:
- KUnit currently uses the TAP14-style "Subtest" header instead of
nested TAP version headers. This will be fixed in KUnit once this goes
in.
- The (unused) XPASS option is not mentioned (it'll devolve into a
harmless comment)
Cheers,
-- David
[1]https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJx…
---
Documentation/dev-tools/index.rst | 1 +
Documentation/dev-tools/ktap.rst | 287 ++++++++++++++++++++++++++++++
2 files changed, 288 insertions(+)
create mode 100644 Documentation/dev-tools/ktap.rst
diff --git a/Documentation/dev-tools/index.rst b/Documentation/dev-tools/index.rst
index 010a2af1e7d9..4621eac290f4 100644
--- a/Documentation/dev-tools/index.rst
+++ b/Documentation/dev-tools/index.rst
@@ -32,6 +32,7 @@ Documentation/dev-tools/testing-overview.rst
kgdb
kselftest
kunit/index
+ ktap
.. only:: subproject and html
diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
new file mode 100644
index 000000000000..7c17956b998a
--- /dev/null
+++ b/Documentation/dev-tools/ktap.rst
@@ -0,0 +1,287 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+========================================
+The Kernel Test Anything Protocol (KTAP)
+========================================
+
+TAP, or the Test Anything Protocol is a format for specifying test results used
+by a number of projects. It's website and specification are found at this `link
+<https://testanything.org/>`_. The Linux Kernel largely uses TAP output for test
+results. However, Kernel testing frameworks have special needs for test results
+which don't align with the original TAP specification. Thus, a "Kernel TAP"
+(KTAP) format is specified to extend and alter TAP to support these use-cases.
+This specification describes the generally accepted format of KTAP as it is
+currently used in the kernel
+
+KTAP test results describe a series of tests (which may be nested: i.e., test
+can have subtests), each of which can contain both diagnostic data -- e.g., log
+lines -- and a final result. The test structure and results are
+machine-readable, whereas the diagnostic data is unstructured and is there to
+aid human debugging.
+
+KTAP output is built from four different types of lines:
+- Version lines
+- Plan lines
+- Test case result lines
+- Diagnostic lines
+
+Note that, in particular, nested test results work differently in KTAP from
+the proposed TAP14 specification:
+https://github.com/TestAnything/Specification/blob/tap-14-specification/specification.md
+
+Version lines
+-------------
+
+All KTAP-formatted results begin with a "version line" which specifies which
+version of the (K)TAP standard the result is compliant with.
+
+For example:
+- "KTAP version 1"
+- "TAP version 13"
+- "TAP version 14"
+
+Note that, in KTAP, subtests also begin with a version line, which denotes the
+start of the nested test results. This differs from TAP14, which uses a
+separate "Subtest" line.
+
+While, going forward, "KTAP version 1" should be used by compliant tests, it
+is expected that most parsers and other tooling will accept the other versions
+listed here for compatibility with existing tests and frameworks.
+
+Plan lines
+----------
+
+A test plan provides the number of tests (or subtests) in the KTAP ouput.
+
+Plan lines must follow the format of "1..N" where N is the number of tests or subtests.
+Plan lines follow version lines to indicate the number of nested tests.
+
+While there are cases where the number of tests is not known in advance -- in
+which case the test plan may be omitted -- it is strongly recommended one is
+present where possible.
+
+Test case result lines
+----------------------
+
+Test case result lines indicate the final status of a test.
+They are required and must have the format:
+
+.. code-block::
+
+ <result> <number> [<description>][ # [<directive>] [<diagnostic data>]]
+
+The result can be either "ok", which indicates the test case passed,
+or "not ok", which indicates that the test case failed.
+
+<number> represents the number of the test being performed. The first test must
+have the number 1 and the number then must increase by 1 for each additional
+subtest within the same test at the same nesting level.
+
+The description is a description of the test, generally the name of
+the test, and can be any string of words (can't include #). The
+description is optional, but recommended.
+
+The directive and any diagnostic data is optional. If either are present, they
+must follow a hash sign, "#".
+
+A directive is a keyword that indicates a different outcome for a test other
+than passed and failed. The directive is optional, and consists of a single
+keyword preceding the diagnostic data. In the event that a parser encounters
+a directive it doesn't support, it should fall back to the "ok" / "not ok"
+result.
+
+Currently accepted directives are:
+
+- "SKIP", which indicates a test was skipped (note the result of the test case
+ result line can be either "ok" or "not ok" if the SKIP directive is used)
+- "TODO", which indicates that a test is not expected to pass at the moment,
+ e.g. because the feature it is testing is known to be broken. While this
+ directive is inherited from TAP, its use in the kernel is discouraged.
+- "XFAIL", which indicates that a test is expected to fail. This is similar
+ to "TODO", above, and is used by some kselftest tests.
+- "XPASS", which indicated that a test is expected to pass. There is some code
+ which tracks this result, but it does not appear to currently be used.
+- “TIMEOUT”, which indicates a test has timed out (note the result of the test
+ case result line should be “not ok” if the TIMEOUT directive is used)
+- “ERROR”, which indicates a test has failed with a specific error that is
+ included in the diagnostic data (note the result of the test case result like
+ should be “not ok” if the ERROR directive is used)
+
+The diagnostic data is a plain-text field which contains any additional details
+about why this result was produced. This is typically an error code for ERROR
+or failed tests, or a description of missing dependencies for a SKIP result.
+
+The diagnostic data field is optional, and results which have neither a
+directive nor any diagnostic data do not need to include the "#" field
+separator.
+
+Example result lines include:
+
+.. code-block::
+
+ ok 1 test_case_name
+
+The test "test_case_name" passed.
+
+.. code-block::
+
+ not ok 1 test_case_name
+
+The test "test_case_name" failed.
+
+.. code-block::
+
+ ok 1 test # SKIP necessary dependency unavailable
+
+The test "test" was SKIPPED with the diagnostic message "necessary dependency
+unavailable".
+
+.. code-block::
+
+ not ok 1 test # TIMEOUT 30 seconds
+
+The test "test" timed out, with diagnostic data "30 seconds".
+
+.. code-block::
+
+ ok 5 check return code # rcode=0
+
+The test "check return code" passed, with additional diagnostic data is
+“rcode=0”
+
+
+Diagnostic lines
+----------------
+
+If tests wish to output any further information, they should do so using
+"diagnostic lines". Diagnostic lines are optional, freeform text, and are
+often used to describe what is being tested and any intermediate results in
+more detail than the final result and diagnostic data line provides.
+
+Diagnostic lines are formatted as "# <diagnostic_description>", where the
+description can be any string. Diagnostic lines can be anywhere in the test
+output. As a rule, diagnostic lines regarding a test are directly before the
+test result line for that test.
+
+Note that most tools will treat unknown lines (see below) as diagnostic lines,
+even if they do not start with a "#": this is to capture any other useful
+kernel output which may help debug the test. It is nevertheless recommended
+that tests always prefix any diagnostic output they have with a "#" character.
+
+Unknown lines
+-------------
+
+There may be lines within KTAP output that do not follow the format of one of
+the four formats for lines described above. This is allowed, however, they will
+not influence the status of the tests.
+
+Nested tests
+------------
+
+In KTAP, tests can be nested. This is done by having a test include within its
+output an entire set of KTAP-formatted results. This can be used to categorize
+and group related tests, or to split out different results from the same test.
+
+The "parent" test's result should consist of all of its subtests' results,
+starting with another KTAP version line and test plan, and end with the overall
+result. If one of the subtests fail, for example, the parent test should also
+fail.
+
+Additionally, all result lines in a subtest should be indented. One level of
+indentation is two spaces: " ". The indentation should begin at the version
+line and should end before the parent test's result line.
+
+An example of a test with two nested subtests:
+
+.. code-block::
+
+ KTAP version 1
+ 1..1
+ KTAP version 1
+ 1..2
+ ok 1 test_1
+ not ok 2 test_2
+ # example failed
+ not ok 1 example
+
+An example format with multiple levels of nested testing:
+
+.. code-block::
+
+ KTAP version 1
+ 1..2
+ KTAP version 1
+ 1..2
+ KTAP version 1
+ 1..2
+ not ok 1 test_1
+ ok 2 test_2
+ not ok 1 test_3
+ ok 2 test_4 # SKIP
+ not ok 1 example_test_1
+ ok 2 example_test_2
+
+
+Major differences between TAP and KTAP
+--------------------------------------
+
+Note the major differences between the TAP and KTAP specification:
+- yaml and json are not recommended in diagnostic messages
+- TODO directive not recognized
+- KTAP allows for an arbitrary number of tests to be nested
+
+The TAP14 specification does permit nested tests, but instead of using another
+nested version line, uses a line of the form
+"Subtest: <name>" where <name> is the name of the parent test.
+
+Example KTAP output
+--------------------
+.. code-block::
+
+ KTAP version 1
+ 1..1
+ KTAP version 1
+ 1..3
+ KTAP version 1
+ 1..1
+ # test_1: initializing test_1
+ ok 1 test_1
+ ok 1 example_test_1
+ KTAP version 1
+ 1..2
+ ok 1 test_1 # SKIP test_1 skipped
+ ok 2 test_2
+ ok 2 example_test_2
+ KTAP version 1
+ 1..3
+ ok 1 test_1
+ # test_2: FAIL
+ not ok 2 test_2
+ ok 3 test_3 # SKIP test_3 skipped
+ not ok 3 example_test_3
+ not ok 1 main_test
+
+This output defines the following hierarchy:
+
+A single test called "main_test", which fails, and has three subtests:
+- "example_test_1", which passes, and has one subtest:
+
+ - "test_1", which passes, and outputs the dianostic message "test_1: initializing test_1"
+
+- "example_test_2", which passes, and has two subtests:
+
+ - "test_1", which is skipped, with the explanation "test_1 skipped"
+ - "test_2", which passes
+
+- "example_test_3", which fails, and has three subtests
+
+ - "test_1", which passes
+ - "test_2", which outputs the diagnostic line "test_2: FAIL", and fails.
+ - "test_3", which is skipped with the explanation "test_3 skipped"
+
+Note that the individual subtests with the same names do not conflict, as they
+are found in different parent tests. This output also exhibits some sensible
+rules for "bubbling up" test results: a test fails if any of its subtests fail.
+Skipped tests do not affect the result of the parent test (though it often
+makes sense for a test to be marked skipped if _all_ of its subtests have been
+skipped).
--
2.34.1.400.ga245620fadb-goog
Add <asm/types.h> include which eventually includes __u64 typedef.
The __u64 typedef is sometimes missing depending on the system headers
installed, leading to compile errors. Adding this include makes this
test more likely to compile on a wider variety of systems.
Signed-off-by: Luke Nowakowski-Krijger <luke.nowakowskikrijger(a)canonical.com>
---
tools/testing/selftests/memfd/fuse_test.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/tools/testing/selftests/memfd/fuse_test.c b/tools/testing/selftests/memfd/fuse_test.c
index be675002f918..725561ac5a38 100644
--- a/tools/testing/selftests/memfd/fuse_test.c
+++ b/tools/testing/selftests/memfd/fuse_test.c
@@ -32,6 +32,7 @@
#include <sys/syscall.h>
#include <sys/wait.h>
#include <unistd.h>
+#include <asm/types.h>
#include "common.h"
--
2.32.0
From: Peilin Ye <peilin.ye(a)bytedance.com>
Currently rp_filter tests in fib_tests.sh:fib_rp_filter_test() are
failing. ping sockets are bound to dummy1 using the "-I" option
(SO_BINDTODEVICE), but socket lookup is failing when receiving ping
replies, since the routing table thinks they belong to dummy0.
For example, suppose ping is using a SOCK_RAW socket for ICMP messages.
When receiving ping replies, in __raw_v4_lookup(), sk->sk_bound_dev_if
is 3 (dummy1), but dif (skb_rtable(skb)->rt_iif) says 2 (dummy0), so the
raw_sk_bound_dev_eq() check fails. Similar things happen in
ping_lookup() for SOCK_ICMP sockets.
Fix the tests by binding to dummy0 instead. Redirect ping requests to
dummy1 before redirecting them again to lo, so that sk->sk_bound_dev_if
agrees with our routing table.
These tests used to pass due to a bug [1] in iputils, where "ping -I"
actually did not bind ICMP message sockets to device. The bug has been
fixed by iputils commit f455fee41c07 ("ping: also bind the ICMP socket
to the specific device") in 2016, which is why our rp_filter tests
started to fail. See [2] .
Tested with ping from iputils 20210722-41-gf9fb573:
$ ./fib_tests.sh -t rp_filter
IPv4 rp_filter tests
TEST: rp_filter passes local packets [ OK ]
TEST: rp_filter passes loopback packets [ OK ]
[1] https://github.com/iputils/iputils/issues/55
[2] https://github.com/iputils/iputils/commit/f455fee41c077d4b700a473b2f5b3487b…
Reported-by: Hangbin Liu <liuhangbin(a)gmail.com>
Fixes: adb701d6cfa4 ("selftests: add a test case for rp_filter")
Reviewed-by: Cong Wang <cong.wang(a)bytedance.com>
Signed-off-by: Peilin Ye <peilin.ye(a)bytedance.com>
---
tools/testing/selftests/net/fib_tests.sh | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/tools/testing/selftests/net/fib_tests.sh b/tools/testing/selftests/net/fib_tests.sh
index 5abe92d55b69..b8bceae00f8e 100755
--- a/tools/testing/selftests/net/fib_tests.sh
+++ b/tools/testing/selftests/net/fib_tests.sh
@@ -453,15 +453,19 @@ fib_rp_filter_test()
$NS_EXEC sysctl -qw net.ipv4.conf.all.accept_local=1
$NS_EXEC sysctl -qw net.ipv4.conf.all.route_localnet=1
+ $NS_EXEC tc qd add dev dummy0 parent root handle 1: fq_codel
+ $NS_EXEC tc filter add dev dummy0 parent 1: protocol arp basic action mirred egress redirect dev dummy1
+ $NS_EXEC tc filter add dev dummy0 parent 1: protocol ip basic action mirred egress redirect dev dummy1
+
$NS_EXEC tc qd add dev dummy1 parent root handle 1: fq_codel
$NS_EXEC tc filter add dev dummy1 parent 1: protocol arp basic action mirred egress redirect dev lo
$NS_EXEC tc filter add dev dummy1 parent 1: protocol ip basic action mirred egress redirect dev lo
set +e
- run_cmd "ip netns exec ns1 ping -I dummy1 -w1 -c1 198.51.100.1"
+ run_cmd "ip netns exec ns1 ping -I dummy0 -w1 -c1 198.51.100.1"
log_test $? 0 "rp_filter passes local packets"
- run_cmd "ip netns exec ns1 ping -I dummy1 -w1 -c1 127.0.0.1"
+ run_cmd "ip netns exec ns1 ping -I dummy0 -w1 -c1 127.0.0.1"
log_test $? 0 "rp_filter passes loopback packets"
cleanup
--
2.20.1
Unconditonally define seccomp_metadata and remove the linux/ptrace.h
include.
There are conflicts between glibc system headers sys/ptrace.h and
linux/ptrace.h that can likely cause seccomp_metadata to not be
defined depending on what system header versions are installed,
leading to compile errors. This fix makes this test more likely to
compile on a wider variety of systems.
Signed-off-by: Luke Nowakowski-Krijger <luke.nowakowskikrijger(a)canonical.com>
---
v2 + resend:
Added comment to explain why there is a header definiton being
defined in the file and to suggest to future developers that they
might have to do the same for future fixes for definition issues like
this.
tools/testing/selftests/seccomp/seccomp_bpf.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/tools/testing/selftests/seccomp/seccomp_bpf.c b/tools/testing/selftests/seccomp/seccomp_bpf.c
index d425688cf59c..19d0b448511c 100644
--- a/tools/testing/selftests/seccomp/seccomp_bpf.c
+++ b/tools/testing/selftests/seccomp/seccomp_bpf.c
@@ -26,7 +26,6 @@
#include <sys/ptrace.h>
#include <sys/user.h>
#include <linux/prctl.h>
-#include <linux/ptrace.h>
#include <linux/seccomp.h>
#include <pthread.h>
#include <semaphore.h>
@@ -171,12 +170,17 @@ struct seccomp_data {
#ifndef PTRACE_SECCOMP_GET_METADATA
#define PTRACE_SECCOMP_GET_METADATA 0x420d
+#endif
+/*
+ * There are conflicting definitions in ptrace system headers that lead to
+ * struct seccomp_metadata to not be defined. So until those conflicts get
+ * sorted out, we should rely on some of our own in-tree ptrace definitions.
+ */
struct seccomp_metadata {
__u64 filter_off; /* Input: which filter */
__u64 flags; /* Output: filter's flags */
};
-#endif
#ifndef SECCOMP_FILTER_FLAG_NEW_LISTENER
#define SECCOMP_FILTER_FLAG_NEW_LISTENER (1UL << 3)
--
2.32.0
Here's the eleventh revision of the simulator.
As there was no reasoning with configfs maintainers for many months,
this time the whole concept of committable items has been dropped. Instead,
each configfs chip item (or rather a group - more on that later) exposes a new
attribute called 'live'. Writing 1 to it brings the chip on-line (registers
the platform device) and writing 0 tears it down.
There are some caveats to that approach - for example: we can't block
the user-space from deleting chip items when chips are live but is just
handled by silently destroying the chip device in the background.
In v11 the configfs structure has been deepened to allow creating
multiple banks per platform device. The sysfs interface has changed so
that the gpio_simX attributes are now under the bank's device node and
not the platform device's.
v1 -> v2:
- add selftests for gpio-sim
- add helper programs for selftests
- update the configfs rename callback to work with the new API introduced in
v5.11
- fix a missing quote in the documentation
- use !! whenever using bits operation that are required to return 0 or 1
- use provided bitmap API instead of reimplementing copy or fill operations
- fix a deadlock in gpio_sim_direction_output()
- add new read-only configfs attributes for mapping of configfs items to GPIO
device names
- and address other minor issues pointed out in reviews of v1
v2 -> v3:
- use devm_bitmap_alloc() instead of the zalloc variant if we're initializing
the bitmap with 1s
- drop the patch exporting device_is_bound()
- don't return -ENODEV from dev_nam and chip_name configfs attributes, return
a string indicating that the device is not available yet ('n/a')
- fix indentation where it makes sense
- don't protect IDA functions which use their own locking and where it's not
needed
- use kmemdup() instead of kzalloc() + memcpy()
- collected review tags
- minor coding style fixes
v3 -> v4:
- return 'none' instead of 'n/a' from dev_name and chip_name before the device
is registered
- use sysfs_emit() instead of s*printf()
- drop GPIO_SIM_MAX_PROP as it's only used in an array's definition where it's
fine to hardcode the value
v4 -> v5:
- drop lib patches that are already upstream
- use BIT() instead of (1UL << bit) for flags
- fix refcounting for the configfs_dirent in rename()
- drop d_move() from the rename() callback
- free memory allocated for the live and pending groups in configfs_d_iput()
and not in detach_groups()
- make sure that if a group of some name is in the live directory, a new group
with the same name cannot be created in the pending directory
v5 -> v6:
- go back to using (1UL << bit) instead of BIT()
- if the live group dentry doesn't exist for whatever reason at the time when
mkdir() in the pending group is called (would be a BUG()), return -ENOENT
instead of -EEXIST which should only be returned if given subsystem already
exists in either live or pending group
v6 -> v7:
- as detailed by Andy in commit 6fda593f3082 ("gpio: mockup: Convert to use
software nodes") removing device properties after the platform device is
removed but before the GPIO device gets dropped can lead to a use-after-free
bug - use software nodes to manually control the freeing of the properties
v7 -> v8:
- fixed some minor coding style issues as pointed out by Andy
v8 -> v9:
- dropped the patches implementing committable-items and reworked the
driver to not use them
- reworked the gpio-line-names property and configuring specific lines
in general
- many smaller tweaks here and there
v9 -> v10:
- make writing to 'live' wait for the probe to finish and report an
error to user-space if it failed
- add the ability to hog lines from the kernel-space
- rework locking (drop separate locks for line context objects)
- rework the sysfs interface (create a separate group for each line with
a constant number of attributes instead of going the other way around)
v10 -> v11:
- rework the configfs structure to represent a deeper hierarchy that
gpiolib supports, namely: multiple banks per platform device
Bartosz Golaszewski (6):
gpiolib: provide gpiod_remove_hogs()
gpiolib: allow to specify the firmware node in struct gpio_chip
gpio: sim: new testing module
selftests: gpio: provide a helper for reading chip info
selftests: gpio: add a helper for reading GPIO line names
selftests: gpio: add test cases for gpio-sim
Documentation/admin-guide/gpio/gpio-sim.rst | 93 +
drivers/gpio/Kconfig | 8 +
drivers/gpio/Makefile | 1 +
drivers/gpio/gpio-sim.c | 1592 +++++++++++++++++
drivers/gpio/gpiolib.c | 26 +-
include/linux/gpio/driver.h | 2 +
include/linux/gpio/machine.h | 2 +
tools/testing/selftests/gpio/.gitignore | 2 +
tools/testing/selftests/gpio/Makefile | 4 +-
tools/testing/selftests/gpio/config | 1 +
tools/testing/selftests/gpio/gpio-chip-info.c | 57 +
tools/testing/selftests/gpio/gpio-line-name.c | 55 +
tools/testing/selftests/gpio/gpio-sim.sh | 396 ++++
13 files changed, 2236 insertions(+), 3 deletions(-)
create mode 100644 Documentation/admin-guide/gpio/gpio-sim.rst
create mode 100644 drivers/gpio/gpio-sim.c
create mode 100644 tools/testing/selftests/gpio/gpio-chip-info.c
create mode 100644 tools/testing/selftests/gpio/gpio-line-name.c
create mode 100755 tools/testing/selftests/gpio/gpio-sim.sh
--
2.25.1