Hi Kees and All,
There are several tests in kselftest subsystem which load modules to tests the internals of the kernel. Most of these test modules are just loaded by the kselftest, their status isn't read and reported to the user logs. Hence they don't provide benefit of executing those tests.
I've found patches from Kees where he has been converting such kselftests to kunit tests [1]. The probable motivation is to move tests output of kselftest subsystem which only triggers tests without correctly reporting the results. On the other hand, kunit is there to test the kernel's internal functions which can't be done by userspace.
Kselftest: Test user facing APIs from userspace Kunit: Test kernel's internal functions from kernelspace
This brings me to conclusion that kselftest which are loading modules to test kernelspace should be converted to kunit tests. I've noted several such kselftests.
This is just my understanding. Please mention if I'm correct above or more reasons to support kselftest test modules transformation into kunit test.
[1] https://lore.kernel.org/all/20221018082824.never.845-kees@kernel.org/
On Mon, Jul 15, 2024 at 03:09:24PM +0500, Muhammad Usama Anjum wrote:
Hi Kees and All,
There are several tests in kselftest subsystem which load modules to tests the internals of the kernel. Most of these test modules are just loaded by the kselftest, their status isn't read and reported to the user logs. Hence they don't provide benefit of executing those tests.
I've found patches from Kees where he has been converting such kselftests to kunit tests [1]. The probable motivation is to move tests output of kselftest subsystem which only triggers tests without correctly reporting the results. On the other hand, kunit is there to test the kernel's internal functions which can't be done by userspace.
Kselftest: Test user facing APIs from userspace Kunit: Test kernel's internal functions from kernelspace
I would say this is a reasonable guide to how these things should be separated, yes. That said, much of what was kind of ad-hoc kernel internals testing that was triggered via kselftests is better done via KUnit these days, but not everything.
This brings me to conclusion that kselftest which are loading modules to test kernelspace should be converted to kunit tests. I've noted several such kselftests.
I would tend to agree, yes. Which stand out to you? I've mainly been doing the conversions when I find myself wanting to add new tests, etc.
On 7/15/24 9:40 PM, Kees Cook wrote:
On Mon, Jul 15, 2024 at 03:09:24PM +0500, Muhammad Usama Anjum wrote:
Hi Kees and All,
There are several tests in kselftest subsystem which load modules to tests the internals of the kernel. Most of these test modules are just loaded by the kselftest, their status isn't read and reported to the user logs. Hence they don't provide benefit of executing those tests.
I've found patches from Kees where he has been converting such kselftests to kunit tests [1]. The probable motivation is to move tests output of kselftest subsystem which only triggers tests without correctly reporting the results. On the other hand, kunit is there to test the kernel's internal functions which can't be done by userspace.
Kselftest: Test user facing APIs from userspace Kunit: Test kernel's internal functions from kernelspace
I would say this is a reasonable guide to how these things should be separated, yes. That said, much of what was kind of ad-hoc kernel internals testing that was triggered via kselftests is better done via KUnit these days, but not everything.
I started investigated when I found that kselftest doesn't parse the kernel logs to mark these tests pass/fail. (kselftest/lib is good example of it)
This brings me to conclusion that kselftest which are loading modules to test kernelspace should be converted to kunit tests. I've noted several such kselftests.
I would tend to agree, yes. Which stand out to you? I've mainly been doing the conversions when I find myself wanting to add new tests, etc.
lib test_bitmap prime_numbers test_printf test_scanf test_strscpy (already converted, need to remove this test) lock test-ww_mutex module net test_blackhole_dev user test_user_copy (probably already converted, need to remove this test) firmware test_firmware fpu test_fpu
Most of these modules are found in lib/*.
Would it be desired to move these to kunit?
On Tue, Jul 16, 2024 at 01:11:14PM +0500, Muhammad Usama Anjum wrote:
On 7/15/24 9:40 PM, Kees Cook wrote:
On Mon, Jul 15, 2024 at 03:09:24PM +0500, Muhammad Usama Anjum wrote:
Hi Kees and All,
There are several tests in kselftest subsystem which load modules to tests the internals of the kernel. Most of these test modules are just loaded by the kselftest, their status isn't read and reported to the user logs. Hence they don't provide benefit of executing those tests.
I've found patches from Kees where he has been converting such kselftests to kunit tests [1]. The probable motivation is to move tests output of kselftest subsystem which only triggers tests without correctly reporting the results. On the other hand, kunit is there to test the kernel's internal functions which can't be done by userspace.
Kselftest: Test user facing APIs from userspace Kunit: Test kernel's internal functions from kernelspace
I would say this is a reasonable guide to how these things should be separated, yes. That said, much of what was kind of ad-hoc kernel internals testing that was triggered via kselftests is better done via KUnit these days, but not everything.
I started investigated when I found that kselftest doesn't parse the kernel logs to mark these tests pass/fail. (kselftest/lib is good example of it)
This brings me to conclusion that kselftest which are loading modules to test kernelspace should be converted to kunit tests. I've noted several such kselftests.
I would tend to agree, yes. Which stand out to you? I've mainly been doing the conversions when I find myself wanting to add new tests, etc.
lib test_bitmap prime_numbers test_printf test_scanf
Yeah, these would be nice to convert.
test_strscpy (already converted, need to remove this test)
Yup, converted in bb8d9b742aa7 ("string: Merge strscpy KUnit tests into string_kunit.c")
lock test-ww_mutex module net test_blackhole_dev
I don't know these very well, but yeah worth looking into.
user test_user_copy (probably already converted, need to remove this test)
This is done in -next via cf6219ee889f ("usercopy: Convert test_user_copy to KUnit test")
firmware test_firmware
This might not work to convert: there's a userspace half for testing firmware loading (see the kselftest side...)
fpu test_fpu
Seems reasonable.
Most of these modules are found in lib/*.
Would it be desired to move these to kunit?
Checking with the authors/maintainer is probably the first thing to do; check the git history to see who has been working on them.
On 7/16/24 10:59 AM, Kees Cook wrote:
On Tue, Jul 16, 2024 at 01:11:14PM +0500, Muhammad Usama Anjum wrote:
On 7/15/24 9:40 PM, Kees Cook wrote:
On Mon, Jul 15, 2024 at 03:09:24PM +0500, Muhammad Usama Anjum wrote:
Hi Kees and All,
There are several tests in kselftest subsystem which load modules to tests the internals of the kernel. Most of these test modules are just loaded by the kselftest, their status isn't read and reported to the user logs. Hence they don't provide benefit of executing those tests.
I've found patches from Kees where he has been converting such kselftests to kunit tests [1]. The probable motivation is to move tests output of kselftest subsystem which only triggers tests without correctly reporting the results. On the other hand, kunit is there to test the kernel's internal functions which can't be done by userspace.
Kselftest: Test user facing APIs from userspace Kunit: Test kernel's internal functions from kernelspace
I would say this is a reasonable guide to how these things should be separated, yes. That said, much of what was kind of ad-hoc kernel internals testing that was triggered via kselftests is better done via KUnit these days, but not everything.
I started investigated when I found that kselftest doesn't parse the kernel logs to mark these tests pass/fail. (kselftest/lib is good example of it)
This brings me to conclusion that kselftest which are loading modules to test kernelspace should be converted to kunit tests. I've noted several such kselftests.
I would tend to agree, yes. Which stand out to you? I've mainly been doing the conversions when I find myself wanting to add new tests, etc.
lib test_bitmap prime_numbers test_printf test_scanf
Yeah, these would be nice to convert.
test_strscpy (already converted, need to remove this test)
Yup, converted in bb8d9b742aa7 ("string: Merge strscpy KUnit tests into string_kunit.c")
lock test-ww_mutex module net test_blackhole_dev
I don't know these very well, but yeah worth looking into.
user test_user_copy (probably already converted, need to remove this test)
This is done in -next via cf6219ee889f ("usercopy: Convert test_user_copy to KUnit test")
firmware test_firmware
This might not work to convert: there's a userspace half for testing firmware loading (see the kselftest side...)
fpu test_fpu
Seems reasonable.
Most of these modules are found in lib/*.
Would it be desired to move these to kunit?
Checking with the authors/maintainer is probably the first thing to do; check the git history to see who has been working on them.
Also maybe:
mm gup_test
This is a classic case of testing a kernel API via ioctl into the kernel, so I think it meets your criteria for moving to KUnit. Having said that, I must now go try out KUnit in a bit more depth before I'm sure that this will work out. But it should.
thanks,
On Tue, Jul 16, 2024 at 11:04:06AM -0700, John Hubbard wrote:
This is a classic case of testing a kernel API via ioctl into the kernel, so I think it meets your criteria for moving to KUnit. Having said that, I must now go try out KUnit in a bit more depth before I'm sure that this will work out. But it should.
I've found the docs really helpful: https://docs.kernel.org/dev-tools/kunit/
Searching for *_kunit.c will also give a lot of examples. And the kunit folks are really responsive to questions, helping guide finding solutions to novel testing needs, etc. :)
On 7/16/24 11:26 AM, Kees Cook wrote:
On Tue, Jul 16, 2024 at 11:04:06AM -0700, John Hubbard wrote:
This is a classic case of testing a kernel API via ioctl into the kernel, so I think it meets your criteria for moving to KUnit. Having said that, I must now go try out KUnit in a bit more depth before I'm sure that this will work out. But it should.
I've found the docs really helpful: https://docs.kernel.org/dev-tools/kunit/
Searching for *_kunit.c will also give a lot of examples. And the kunit folks are really responsive to questions, helping guide finding solutions to novel testing needs, etc. :)
OK, yes very helpful. I guess I've got some unique biases, because I found it extremely odd to put so much emphasis on the arch=UM, as opposed to loading a kunit kernel module on baremetal and running tests there. I mean, yes, VMs or user mode Linux have their place, but baremetal testing is really important (and in many cases, the only way).
Anyway, this confirms for me that kunit should work perfectly for converting mm/gup_test.
thanks,
On 7/16/24 10:59 PM, Kees Cook wrote:
On Tue, Jul 16, 2024 at 01:11:14PM +0500, Muhammad Usama Anjum wrote:
On 7/15/24 9:40 PM, Kees Cook wrote:
On Mon, Jul 15, 2024 at 03:09:24PM +0500, Muhammad Usama Anjum wrote:
Hi Kees and All,
There are several tests in kselftest subsystem which load modules to tests the internals of the kernel. Most of these test modules are just loaded by the kselftest, their status isn't read and reported to the user logs. Hence they don't provide benefit of executing those tests.
I've found patches from Kees where he has been converting such kselftests to kunit tests [1]. The probable motivation is to move tests output of kselftest subsystem which only triggers tests without correctly reporting the results. On the other hand, kunit is there to test the kernel's internal functions which can't be done by userspace.
Kselftest: Test user facing APIs from userspace Kunit: Test kernel's internal functions from kernelspace
I would say this is a reasonable guide to how these things should be separated, yes. That said, much of what was kind of ad-hoc kernel internals testing that was triggered via kselftests is better done via KUnit these days, but not everything.
I started investigated when I found that kselftest doesn't parse the kernel logs to mark these tests pass/fail. (kselftest/lib is good example of it)
This brings me to conclusion that kselftest which are loading modules to test kernelspace should be converted to kunit tests. I've noted several such kselftests.
I would tend to agree, yes. Which stand out to you? I've mainly been doing the conversions when I find myself wanting to add new tests, etc.
lib test_bitmap prime_numbers test_printf test_scanf
Yeah, these would be nice to convert.
I've created TODOs on my end. I'll start working on conversion soon.
test_strscpy (already converted, need to remove this test)
Yup, converted in bb8d9b742aa7 ("string: Merge strscpy KUnit tests into string_kunit.c")
lock test-ww_mutex module net test_blackhole_dev
I don't know these very well, but yeah worth looking into.
user test_user_copy (probably already converted, need to remove this test)
This is done in -next via cf6219ee889f ("usercopy: Convert test_user_copy to KUnit test")
firmware test_firmware
This might not work to convert: there's a userspace half for testing firmware loading (see the kselftest side...)
fpu test_fpu
Seems reasonable.
Most of these modules are found in lib/*.
Would it be desired to move these to kunit?
Checking with the authors/maintainer is probably the first thing to do; check the git history to see who has been working on them.
On Mon, 15 Jul 2024 at 18:09, Muhammad Usama Anjum usama.anjum@collabora.com wrote:
Hi Kees and All,
There are several tests in kselftest subsystem which load modules to tests the internals of the kernel. Most of these test modules are just loaded by the kselftest, their status isn't read and reported to the user logs. Hence they don't provide benefit of executing those tests.
I've found patches from Kees where he has been converting such kselftests to kunit tests [1]. The probable motivation is to move tests output of kselftest subsystem which only triggers tests without correctly reporting the results. On the other hand, kunit is there to test the kernel's internal functions which can't be done by userspace.
Kselftest: Test user facing APIs from userspace Kunit: Test kernel's internal functions from kernelspace
Yes: this is how we'd like to split things up. There are still a few cases where you might want to use kselftest to test something other than a user-facing API (if you needed to set up some complicated userspace structures, etc), or cases where KUnit might be used to test something other than individual pieces of functionality, but that categorisation is a good start.
The Documentation/dev-tools/testing-overview.rst page has a more detailed look at when to use which test framework (which basically just repeats those rules): https://docs.kernel.org/dev-tools/testing-overview.html
Cheers, -- David
Hi David,
On 7/16/24 12:33 PM, David Gow wrote:
On Mon, 15 Jul 2024 at 18:09, Muhammad Usama Anjum usama.anjum@collabora.com wrote:
Hi Kees and All,
There are several tests in kselftest subsystem which load modules to tests the internals of the kernel. Most of these test modules are just loaded by the kselftest, their status isn't read and reported to the user logs. Hence they don't provide benefit of executing those tests.
I've found patches from Kees where he has been converting such kselftests to kunit tests [1]. The probable motivation is to move tests output of kselftest subsystem which only triggers tests without correctly reporting the results. On the other hand, kunit is there to test the kernel's internal functions which can't be done by userspace.
Kselftest: Test user facing APIs from userspace Kunit: Test kernel's internal functions from kernelspace
Yes: this is how we'd like to split things up. There are still a few cases where you might want to use kselftest to test something other than a user-facing API (if you needed to set up some complicated userspace structures, etc), or cases where KUnit might be used to test something other than individual pieces of functionality, but that categorisation is a good start.
Yeah, makes sense. It is helpful to find out what others think. I'll be back with changes.
The Documentation/dev-tools/testing-overview.rst page has a more detailed look at when to use which test framework (which basically just repeats those rules): https://docs.kernel.org/dev-tools/testing-overview.html
Cheers, -- David
On 7/17/24 3:47 AM, Muhammad Usama Anjum wrote:
Hi David,
On 7/16/24 12:33 PM, David Gow wrote:
On Mon, 15 Jul 2024 at 18:09, Muhammad Usama Anjum usama.anjum@collabora.com wrote:
Hi Kees and All,
There are several tests in kselftest subsystem which load modules to tests the internals of the kernel. Most of these test modules are just loaded by the kselftest, their status isn't read and reported to the user logs. Hence they don't provide benefit of executing those tests.
I've found patches from Kees where he has been converting such kselftests to kunit tests [1]. The probable motivation is to move tests output of kselftest subsystem which only triggers tests without correctly reporting the results. On the other hand, kunit is there to test the kernel's internal functions which can't be done by userspace.
Kselftest: Test user facing APIs from userspace Kunit: Test kernel's internal functions from kernelspace
Yes: this is how we'd like to split things up. There are still a few
Me too. It works.
cases where you might want to use kselftest to test something other than a user-facing API (if you needed to set up some complicated userspace structures, etc), or cases where KUnit might be used to test something other than individual pieces of functionality, but that categorisation is a good start.
Yeah, makes sense. It is helpful to find out what others think. I'll be back with changes.
At some point we could connect up the two systems, without really changing any of the guidelines...much. One way to do that would be to add a tiny bit of kselftest support for easily launching a kunit baremetal testing, and then reading the results (which are right there in sysfs) back to user space.
So it would look like a kselftest, but it could run any kernel-based tests it needs via kunit.
And the two systems can still be used independently.
The Documentation/dev-tools/testing-overview.rst page has a more detailed look at when to use which test framework (which basically just repeats those rules): https://docs.kernel.org/dev-tools/testing-overview.html
Yes, a nice example of up-to-date documentation there. And it's very clear.
Cheers, -- David
thanks,
On 7/15/24 04:09, Muhammad Usama Anjum wrote:
Hi Kees and All,
There are several tests in kselftest subsystem which load modules to tests the internals of the kernel. Most of these test modules are just loaded by the kselftest, their status isn't read and reported to the user logs. Hence they don't provide benefit of executing those tests.
I've found patches from Kees where he has been converting such kselftests to kunit tests [1]. The probable motivation is to move tests output of kselftest subsystem which only triggers tests without correctly reporting the results. On the other hand, kunit is there to test the kernel's internal functions which can't be done by userspace.
Kselftest: Test user facing APIs from userspace Kunit: Test kernel's internal functions from kernelspace
This brings me to conclusion that kselftest which are loading modules to test kernelspace should be converted to kunit tests. I've noted several such kselftests.
This is just my understanding. Please mention if I'm correct above or more reasons to support kselftest test modules transformation into kunit test.
[1] https://lore.kernel.org/all/20221018082824.never.845-kees@kernel.org/
Please make sure you aren't taking away the ability to run these tests during boot. It doesn't make sense to convert every single test especially when it is intended to be run during boot without dependencies - not as a kunit test but a regression test during boot.
bitmap is one example - pay attention to the config help test - bitmap one clearly states it runs regression testing during boot. Any test that says that isn't a candidate for conversion.
I am going to nack any such conversions.
thanks, -- Shuah
On 7/27/24 12:35 AM, Shuah Khan wrote:
On 7/15/24 04:09, Muhammad Usama Anjum wrote:
Hi Kees and All,
There are several tests in kselftest subsystem which load modules to tests the internals of the kernel. Most of these test modules are just loaded by the kselftest, their status isn't read and reported to the user logs. Hence they don't provide benefit of executing those tests.
I've found patches from Kees where he has been converting such kselftests to kunit tests [1]. The probable motivation is to move tests output of kselftest subsystem which only triggers tests without correctly reporting the results. On the other hand, kunit is there to test the kernel's internal functions which can't be done by userspace.
Kselftest: Test user facing APIs from userspace Kunit: Test kernel's internal functions from kernelspace
This brings me to conclusion that kselftest which are loading modules to test kernelspace should be converted to kunit tests. I've noted several such kselftests.
This is just my understanding. Please mention if I'm correct above or more reasons to support kselftest test modules transformation into kunit test.
[1] https://lore.kernel.org/all/20221018082824.never.845-kees@kernel.org/
Please make sure you aren't taking away the ability to run these tests during boot.
The kunit tests are usually run at boot time. They can be run later as well. So I'm not trying to remove any functionality. Rather adding a way where failures would actually be detected.
It doesn't make sense to convert every single test especially when it is intended to be run during boot without dependencies - not as a kunit test but a regression test during boot.
I started investigating when these lib kselftests were just loading the test module without checking if test actually passed/failed (which proves that this type of kselftests can never detect regression as it doesn't process the results). It was strange. Hence I found out the conversion of such kselftests to kunit tests done by Kees and started this thread for discussion before doing actual work and sending patches.
bitmap is one example - pay attention to the config help test - bitmap one clearly states it runs regression testing during boot. Any test that says that isn't a candidate for conversion.
Please can you explain what do you mean by bitmap testing during boot? As far as I understand, the kselftests are usespace tests and are run after kernel has booted completely and userspace has started. It cannot be regarded as testing during boot.
I am going to nack any such conversions.
thanks, -- Shuah
On 7/29/24 01:55, Muhammad Usama Anjum wrote:
On 7/27/24 12:35 AM, Shuah Khan wrote:
On 7/15/24 04:09, Muhammad Usama Anjum wrote:
Hi Kees and All,
There are several tests in kselftest subsystem which load modules to tests the internals of the kernel. Most of these test modules are just loaded by the kselftest, their status isn't read and reported to the user logs. Hence they don't provide benefit of executing those tests.
I've found patches from Kees where he has been converting such kselftests to kunit tests [1]. The probable motivation is to move tests output of kselftest subsystem which only triggers tests without correctly reporting the results. On the other hand, kunit is there to test the kernel's internal functions which can't be done by userspace.
Kselftest: Test user facing APIs from userspace Kunit: Test kernel's internal functions from kernelspace
This brings me to conclusion that kselftest which are loading modules to test kernelspace should be converted to kunit tests. I've noted several such kselftests.
This is just my understanding. Please mention if I'm correct above or more reasons to support kselftest test modules transformation into kunit test.
[1] https://lore.kernel.org/all/20221018082824.never.845-kees@kernel.org/
Please make sure you aren't taking away the ability to run these tests during boot.
The kunit tests are usually run at boot time. They can be run later as well. So I'm not trying to remove any functionality. Rather adding a way where failures would actually be detected.
It doesn't make sense to convert every single test especially when it is intended to be run during boot without dependencies - not as a kunit test but a regression test during boot.
I started investigating when these lib kselftests were just loading the test module without checking if test actually passed/failed (which proves that this type of kselftests can never detect regression as it doesn't process the results). It was strange. Hence I found out the conversion of such kselftests to kunit tests done by Kees and started this thread for discussion before doing actual work and sending patches.
I explained this on two other threads. If you don't know how this test is used why are you making this change?
bitmap is one example - pay attention to the config help test - bitmap one clearly states it runs regression testing during boot. Any test that says that isn't a candidate for conversion.
Please can you explain what do you mean by bitmap testing during boot? As far as I understand, the kselftests are usespace tests and are run after kernel has booted completely and userspace has started. It cannot be regarded as testing during boot.
I responded to David - let me know you would like me to repeat it.
thanks, -- Shuah
On Sat, 27 Jul 2024 at 03:35, Shuah Khan skhan@linuxfoundation.org wrote:
On 7/15/24 04:09, Muhammad Usama Anjum wrote:
Hi Kees and All,
There are several tests in kselftest subsystem which load modules to tests the internals of the kernel. Most of these test modules are just loaded by the kselftest, their status isn't read and reported to the user logs. Hence they don't provide benefit of executing those tests.
I've found patches from Kees where he has been converting such kselftests to kunit tests [1]. The probable motivation is to move tests output of kselftest subsystem which only triggers tests without correctly reporting the results. On the other hand, kunit is there to test the kernel's internal functions which can't be done by userspace.
Kselftest: Test user facing APIs from userspace Kunit: Test kernel's internal functions from kernelspace
This brings me to conclusion that kselftest which are loading modules to test kernelspace should be converted to kunit tests. I've noted several such kselftests.
This is just my understanding. Please mention if I'm correct above or more reasons to support kselftest test modules transformation into kunit test.
[1] https://lore.kernel.org/all/20221018082824.never.845-kees@kernel.org/
Please make sure you aren't taking away the ability to run these tests during boot. It doesn't make sense to convert every single test especially when it is intended to be run during boot without dependencies - not as a kunit test but a regression test during boot.
Given KUnit tests can run at boot (and, indeed, do by default if enabled), I'd've assumed that this would be a good candidate for such a conversion. It does add the KUnit 'dependency', but I can't think of how that could be a problem. Is there a specific situation where enabling CONFIG_KUNIT would cause problems?
bitmap is one example - pay attention to the config help test - bitmap one clearly states it runs regression testing during boot. Any test that says that isn't a candidate for conversion.
Again, most KUnit tests are effectively regression tests at boot, so I don't really understand what makes bitmap different. If it's just a matter of there being a different interface to it, that's surely something that we'll either be able to adapt to, or to have some wrapper/shim to maintain compatibility. I agree that having needless churn in formats is bad, but KUnit does seem the proper place for these sorts of tests.
If this isn't the case, do we need to modify the testing guide to mention this, as it definitely suggests KUnit for tests of in-kernel functionality like this.
Cheers, -- David
On 7/29/24 23:23, David Gow wrote:
On Sat, 27 Jul 2024 at 03:35, Shuah Khan skhan@linuxfoundation.org wrote:
On 7/15/24 04:09, Muhammad Usama Anjum wrote:
Hi Kees and All,
There are several tests in kselftest subsystem which load modules to tests the internals of the kernel. Most of these test modules are just loaded by the kselftest, their status isn't read and reported to the user logs. Hence they don't provide benefit of executing those tests.
I've found patches from Kees where he has been converting such kselftests to kunit tests [1]. The probable motivation is to move tests output of kselftest subsystem which only triggers tests without correctly reporting the results. On the other hand, kunit is there to test the kernel's internal functions which can't be done by userspace.
Kselftest: Test user facing APIs from userspace Kunit: Test kernel's internal functions from kernelspace
This brings me to conclusion that kselftest which are loading modules to test kernelspace should be converted to kunit tests. I've noted several such kselftests.
This is just my understanding. Please mention if I'm correct above or more reasons to support kselftest test modules transformation into kunit test.
[1] https://lore.kernel.org/all/20221018082824.never.845-kees@kernel.org/
Please make sure you aren't taking away the ability to run these tests during boot. It doesn't make sense to convert every single test especially when it is intended to be run during boot without dependencies - not as a kunit test but a regression test during boot.
Given KUnit tests can run at boot (and, indeed, do by default if enabled), I'd've assumed that this would be a good candidate for such a conversion. It does add the KUnit 'dependency', but I can't think of how that could be a problem. Is there a specific situation where enabling CONFIG_KUNIT would cause problems?
bitmap is one example - pay attention to the config help test - bitmap one clearly states it runs regression testing during boot. Any test that says that isn't a candidate for conversion.
Again, most KUnit tests are effectively regression tests at boot, so I don't really understand what makes bitmap different. If it's just a matter of there being a different interface to it, that's surely something that we'll either be able to adapt to, or to have some wrapper/shim to maintain compatibility. I agree that having needless churn in formats is bad, but KUnit does seem the proper place for these sorts of tests.
The problem is whether not kunit can test at boot time. The issue is that this type of change removes a selftest which can be run without kunit dependency thereby removing the ability to run these tests on a running system without kunit.
This is absolutely necessary for testing on a running system to debug or sopt check.
If this isn't the case, do we need to modify the testing guide to mention this, as it definitely suggests KUnit for tests of in-kernel functionality like this.
Please do. It was never the intent to convert all existing tests toi kunit.
Repeating what I said on the bitmap removal thread:
This doesn't help people who want run a run bitmap test on a running system. This is a wrong direction to go to say all testing has to be done under kunit.
What happened to the effort to run selftests as is under KUnit? What is the motivation to convert all tests to kunit instead of trying to provide support to run kselftest under kunit environment?
We discussed this a few years ago as I recall. Let's work on that instead of removing existing selftests and regressing current use-cases?
Can we look into providing:
1. running kselftest under kunit environment without changes as user space applications?
2. Leave kselftests alone so we don't weaken kernel testing by removing existing ones and making them dependent on kunit thereby removing the existing ability to be able run them on non-kunit kernels.
It isn't about kunit vs. kselftest. It is about overall kernel validation and ability to test effectively by developers and users.
KUnit isn't ideal for cases where people would want to check a subsystem on a running kernel - KUnit covers some use-cases and kselftest covers others.
What happens if we are debugging a problem that requires us to debug on a running system? Please don't go converting kselftest into kunit without understanding how these frameworks are intended to be used.
Yes kselftest results need to be looked at. Write a parser which can be improved. What you are doing is reducing the coverage and talking away the ability to debug and test on running system.
Fix what needs to be fixed instead of deleting tests.
Think through the use-cases before advocating KUnit is suitable for all testing needs and talking about being able to force or not force people to use one or the other.
Reports aren't everything. The primary reason we have these tests is for developers to be able to test. Reports can be improved and shouldn't come at the expense of coverage and testing. Any patch that does that will be NACKed.
I already nacked several patches and will continue to nack. Please don't send me patches that remove existing kselftest tests and turn them into kunit tests.
thanks, -- Shuah
Hello @Muhammad Usama Anjum Your understanding is correct. Kselftest is primarily used for testing user facing APIs from userspace, while Kunit is designed for testing kernel internals from kernelspace; Converting kselftests that load modules to test kernelspace into Kunit tests would ensure more accurate reporting and effective testing of kernel internals.
Your understanding is correct; Kselftest is primarily used for testing user-facing APIs from userspace, while Kunit is designed for testing kernel internals from kernelspace & Thank you for highlighting this!
linux-kselftest-mirror@lists.linaro.org
-
David Gow -
iamolivasmith＠gmail.com -
John Hubbard -
Kees Cook -
Muhammad Usama Anjum -
Shuah Khan