Hi, Willy, Thomas
This is not really for merge, but only let it work as a demo code to
test whether it is possible to restore the next test when there is a bad
pointer access in user-space [1].
Besides, a new 'run' command is added to 'NOLIBC_TEST' environment
variable or arguments to control the running iterations, this may be
used to test the reentrancy issues, but no failures found currently ;-)
With glibc, it works as following:
$ ./nolibc-test run:2,syscall:28-30,stdlib:1
Running iteration(s): 2
Current iteration: 1
Running test 'syscall', from 28 to 30
28 dup3_m1 = -1 EBADF [OK]
29 efault_handler ! 11 SIGSEGV [OK]
30 execve_root = -1 EACCES [OK]
Errors during this test: 0
Running test 'stdlib'
1 getenv_blah = <(null)> [OK]
Errors during this test: 0
Total number of errors in the 1 iteration(s): 0
Current iteration: 2
Running test 'syscall'
28 dup3_m1 = -1 EBADF [OK]
29 efault_handler ! 11 SIGSEGV [OK]
30 execve_root = -1 EACCES [OK]
Errors during this test: 0
Running test 'stdlib'
1 getenv_blah = <(null)> [OK]
Errors during this test: 0
Total number of errors in the 2 iteration(s): 0
With nolibc, it will be skipped (run:2,syscall:28-30,stdlib:10):
Running iteration(s): 2
Current iteration: 1
Running test 'syscall', from 28 to 30
28 dup3_m1 = -1 EBADF [OK]
29 efault_handler [SKIPPED]
30 execve_root = -1 EACCES [OK]
Errors during this test: 0
Running test 'stdlib', from 10 to 10
10 strrchr_foobar_o = <obar> [OK]
Errors during this test: 0
Total number of errors in the 1 iteration(s): 0
Current iteration: 2
Running test 'syscall', from 28 to 30
28 dup3_m1 = -1 EBADF [OK]
29 efault_handler [SKIPPED]
30 execve_root = -1 EACCES [OK]
Errors during this test: 0
Running test 'stdlib', from 10 to 10
10 strrchr_foobar_o = <obar> [OK]
Errors during this test: 0
Total number of errors in the 2 iteration(s): 0
Best regards,
Zhangjin
---
[1]: https://lore.kernel.org/linux-riscv/20230529113143.GB2762@1wt.eu/
Zhangjin Wu (4):
selftests/nolibc: allow rerun with the same settings
selftests/nolibc: add rerun support
selftests/nolibc: add user space efault handler
selftests/nolibc: add user-space efault restore test case
tools/testing/selftests/nolibc/nolibc-test.c | 247 +++++++++++++++++--
1 file changed, 221 insertions(+), 26 deletions(-)
--
2.25.1
On Tue, May 23, 2023 at 11:22:07PM +0000, Ziqi Zhao wrote:
> An output message:
>
> > # # waitpid WEXITSTATUS=0
>
> will be printed for 30,000+ times in the `pidfd_test` selftest, which
> does not seem ideal. This patch removes the print logic in the
> `wait_for_pid` function, so each call to this function does not output
> a line by default. Any existing call sites where the extra line might
> be beneficial have been modified to include extra print statements
> outside of the function calls.
>
> Signed-off-by: Ziqi Zhao <astrajoan(a)yahoo.com>
> ---
Fine by me,
Reviewed-by: Christian Brauner <brauner(a)kernel.org>
The default timeout for selftests tests is 45 seconds. Although
we already have 13 settings for tests of about 96 sefltests which
use a timeout greater than this, we want to try to avoid encouraging
more tests to forcing a higher test timeout as selftests strives to
run all tests quickly. Selftests also uses the timeout as a non-fatal
error. Only tests runners which have control over a system would know
if to treat a timeout as fatal or not.
To help with all this:
o Enhance documentation to avoid future increases of insane timeouts
o Add the option to allow overriding the default timeout with test
runners with a command line option
Suggested-by: Shuah Khan <skhan(a)linuxfoundation.org>
Signed-off-by: Luis Chamberlain <mcgrof(a)kernel.org>
---
Documentation/dev-tools/kselftest.rst | 22 +++++++++++++++++++++
tools/testing/selftests/kselftest/runner.sh | 11 ++++++++++-
tools/testing/selftests/run_kselftest.sh | 5 +++++
3 files changed, 37 insertions(+), 1 deletion(-)
diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
index 12b575b76b20..dd214af7b7ff 100644
--- a/Documentation/dev-tools/kselftest.rst
+++ b/Documentation/dev-tools/kselftest.rst
@@ -168,6 +168,28 @@ the `-t` option for specific single tests. Either can be used multiple times::
For other features see the script usage output, seen with the `-h` option.
+Timeout for selftests
+=====================
+
+Selftests are designed to be quick and so a default timeout is used of 45
+seconds for each test. Tests can override the default timeout by adding
+a settings file in their directory and set a timeout variable there to the
+configured a desired upper timeout for the test. Only a few tests override
+the timeout with a value higher than 45 seconds, selftests strives to keep
+it that way. Timeouts in selftests are not considered fatal because the
+system under which a test runs may change and this can also modify the
+expected time it takes to run a test. If you have control over the systems
+which will run the tests you can configure a test runner on those systems to
+use a greater or lower timeout on the command line as with the `-o` or
+the `--override-timeout` argument. For example to use 165 seconds instead
+one would use:
+
+ $ ./run_kselftest.sh --override-timeout 165
+
+You can look at the TAP output to see if you ran into the timeout. Test
+runners which know a test must run under a specific time can then optionally
+treat these timeouts then as fatal.
+
Packaging selftests
===================
diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh
index 294619ade49f..1c952d1401d4 100644
--- a/tools/testing/selftests/kselftest/runner.sh
+++ b/tools/testing/selftests/kselftest/runner.sh
@@ -8,7 +8,8 @@ export logfile=/dev/stdout
export per_test_logging=
# Defaults for "settings" file fields:
-# "timeout" how many seconds to let each test run before failing.
+# "timeout" how many seconds to let each test run before running
+# over our soft timeout limit.
export kselftest_default_timeout=45
# There isn't a shell-agnostic way to find the path of a sourced file,
@@ -90,6 +91,14 @@ run_one()
done < "$settings"
fi
+ # Command line timeout overrides the settings file
+ if [ -n "$kselftest_override_timeout" ]; then
+ kselftest_timeout="$kselftest_override_timeout"
+ echo "# overriding timeout to $kselftest_timeout" >> "$logfile"
+ else
+ echo "# timeout set to $kselftest_timeout" >> "$logfile"
+ fi
+
TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST"
echo "# $TEST_HDR_MSG"
if [ ! -e "$TEST" ]; then
diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh
index 97165a83df63..9a981b36bd7f 100755
--- a/tools/testing/selftests/run_kselftest.sh
+++ b/tools/testing/selftests/run_kselftest.sh
@@ -26,6 +26,7 @@ Usage: $0 [OPTIONS]
-l | --list List the available collection:test entries
-d | --dry-run Don't actually run any tests
-h | --help Show this usage info
+ -o | --override-timeout Number of seconds after which we timeout
EOF
exit $1
}
@@ -33,6 +34,7 @@ EOF
COLLECTIONS=""
TESTS=""
dryrun=""
+kselftest_override_timeout=""
while true; do
case "$1" in
-s | --summary)
@@ -51,6 +53,9 @@ while true; do
-d | --dry-run)
dryrun="echo"
shift ;;
+ -o | --override-timeout)
+ kselftest_override_timeout="$2"
+ shift 2 ;;
-h | --help)
usage 0 ;;
"")
--
2.39.2
running nolibc-test with glibc on x86_64 got such print issue:
29 execve_root = -1 EACCES [OK]
30 fork30 fork = 0 [OK]
31 getdents64_root = 712 [OK]
The fork test case has three printf calls:
(1) llen += printf("%d %s", test, #name);
(2) llen += printf(" = %d %s ", expr, errorname(errno));
(3) llen += pad_spc(llen, 64, "[FAIL]\n"); --> vfprintf()
In the following scene, the above issue happens:
(a) The parent calls (1)
(b) The parent calls fork()
(c) The child runs and shares the print buffer of (1)
(d) The child exits, flushs the print buffer and closes its own stdout/stderr
* "30 fork" is printed at the first time.
(e) The parent calls (2) and (3), with "\n" in (3), it flushs the whole buffer
* "30 fork = 0 ..." is printed
Therefore, there are two "30 fork" in the stdout.
Between (a) and (b), if flush the stdout (and the sterr), the child in
stage (c) will not be able to 'see' the print buffer.
Signed-off-by: Zhangjin Wu <falcon(a)tinylab.org>
---
tools/testing/selftests/nolibc/nolibc-test.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c
index 7de46305f419..88323a60aa4a 100644
--- a/tools/testing/selftests/nolibc/nolibc-test.c
+++ b/tools/testing/selftests/nolibc/nolibc-test.c
@@ -486,7 +486,13 @@ static int test_getpagesize(void)
static int test_fork(void)
{
int status;
- pid_t pid = fork();
+ pid_t pid;
+
+ /* flush the printf buffer to avoid child flush it */
+ fflush(stdout);
+ fflush(stderr);
+
+ pid = fork();
switch (pid) {
case -1:
--
2.25.1
Hi,
This follows the discussion here:
https://lore.kernel.org/linux-kselftest/20230324123157.bbwvfq4gsxnlnfwb@hou…
This shows a couple of inconsistencies with regard to how device-managed
resources are cleaned up. Basically, devm resources will only be cleaned up
if the device is attached to a bus and bound to a driver. Failing any of
these cases, a call to device_unregister will not end up in the devm
resources being released.
We had to work around it in DRM to provide helpers to create a device for
kunit tests, but the current discussion around creating similar, generic,
helpers for kunit resumed interest in fixing this.
This can be tested using the command:
./tools/testing/kunit/kunit.py run --kunitconfig=drivers/base/test/
Let me know what you think,
Maxime
Signed-off-by: Maxime Ripard <maxime(a)cerno.tech>
---
Maxime Ripard (2):
drivers: base: Add basic devm tests for root devices
drivers: base: Add basic devm tests for platform devices
drivers/base/test/.kunitconfig | 2 +
drivers/base/test/Kconfig | 4 +
drivers/base/test/Makefile | 3 +
drivers/base/test/platform-device-test.c | 278 +++++++++++++++++++++++++++++++
drivers/base/test/root-device-test.c | 120 +++++++++++++
5 files changed, 407 insertions(+)
---
base-commit: a6faf7ea9fcb7267d06116d4188947f26e00e57e
change-id: 20230329-kunit-devm-inconsistencies-test-5e5a7d01e60d
Best regards,
--
Maxime Ripard <maxime(a)cerno.tech>