Hi David,
On Fri, 28 Mar 2025 at 09:07, David Gow davidgow@google.com wrote:
Thanks for sending this out: I think this raises some good questions about exactly how to handle long running tests (particularly on older/slower hardware).
I've put a few notes below, but, tl;dr: I think these are all good changes, even if there's more we can do to better scale to slower hardware.
On Fri, 28 Mar 2025 at 00:07, Geert Uytterhoeven geert@linux-m68k.org wrote:
- Increase timeout by ten; ideally this should only be done for very slow tests, but I couldn't find how to access kunit_case.attr.case from kunit_try_catch_run(),
My feeling for tests generally is:
- Normal: effectively instant on modern hardware, O(seconds) on
ancient hardware.
- Slow: takes O(seconds) to run on modern hardware, O(minutes)..O(10s
of minutes) on ancient hardware.
- Very slow: O(minutes) or higher on modern hardware, infeasible on
ancient hardware.
Obviously the definition of "modern" and "ancient" hardware here is pretty arbitrary: I'm using "modern, high-end x86" ~4GHz as my "modern" example, and "66MHz 486" as my "ancient" one, but things like emulation or embedded systems fit in-between.
Ultimately, I think the timeout probably needs to be configurable on a per-machine basis more than a per-test one, but having a 10x multiplier (or even a 100x multiplier) for very slow tests would also work for me.
Yes, adapting automatically to the speed of the target maachine would be nice, but non-trivial.
I quickly tried hacking together something to pass through the attribute and implement this. Diff (probably mangled by gmail) below:
[...]
Thanks!
I'll get around to extending this to allow the "base timeout" to be configurable as a command-line option, too, if this seems like a good way to go.
- Mark rtc_time64_to_tm_test_date_range_1000 slow,
- Mark rtc_time64_to_tm_test_date_range_160000 very slow.
Hmm... these are definitely fast enough on my "modern" machine that they probably only warrant "slow", not "very slow". But given they're definitely causing problems on older machines, I'm happy to go with marking the large ones very slow. (I've been waiting for them for about 45 minutes so far on my 486.)
Do the time tests in kernel/time/time_test.c also need to be marked very slow, or does that run much faster on your setup?
Hmm, I did run time_test (insmod took (+7 minutes), but I don't seem to have pass/fail output. Will rerun...
Indeed:
# time64_to_tm_test_date_range.speed: slow
Another test that wanted to be marked as slow was:
# kunit_platform_device_add_twice_fails_test: Test should be marked slow (runtime: 30.788248702s)
I will rerun all, as it seems I have lost some logs...
Is this causing you enough strife that you want it in as-is, straight away, or would you be happy with it being split up and polished a bit first -- particularly around supporting the more configurable timeout, and shifting the test changes into separate patches? (I'm happy to do that for you if you don't want to dig around in the somewhat messy KUnit try-catch stuff any further.)
This is definitely not something urgent for me.
Thanks!
Gr{oetje,eeting}s,
Geert
-- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds