Re: [PATCH] kunit: Reset test status on each param iteration

On Fri, 1 Sept 2023 at 05:49, Michal Wajdeczko michal.wajdeczko@intel.com wrote:

If we skip one parametrized test case then test status remains SKIP for all subsequent test params leading to wrong reports:

$ ./tools/testing/kunit/kunit.py run \ --kunitconfig ./lib/kunit/.kunitconfig *.example_params* --raw_output \

[ ] Starting KUnit Kernel (1/1)... KTAP version 1 1..1 # example: initializing suite KTAP version 1 # Subtest: example # module: kunit_example_test 1..1 KTAP version 1 # Subtest: example_params_test # example_params_test: initializing # example_params_test: cleaning up ok 1 example value 3 # SKIP unsupported param value 3 # example_params_test: initializing # example_params_test: cleaning up ok 2 example value 2 # SKIP unsupported param value 3 # example_params_test: initializing # example_params_test: cleaning up ok 3 example value 1 # SKIP unsupported param value 3 # example_params_test: initializing # example_params_test: cleaning up ok 4 example value 0 # SKIP unsupported param value 0 # example_params_test: pass:0 fail:0 skip:4 total:4 ok 1 example_params_test # SKIP unsupported param value 0 # example: exiting suite ok 1 example # SKIP

Reset test status and status comment after each param iteration to avoid using stale results.

Signed-off-by: Michal Wajdeczko michal.wajdeczko@intel.com Cc: David Gow davidgow@google.com Cc: Rae Moar rmoar@google.com

---

Nice catch, thanks!

This looks good to me.

Reviewed-by: David Gow davidgow@google.com

Cheers, -- David

lib/kunit/kunit-example-test.c | 5 +++-- lib/kunit/test.c | 6 ++++-- 2 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/lib/kunit/kunit-example-test.c b/lib/kunit/kunit-example-test.c index 01a769f35e1d..6bb5c2ef6696 100644 --- a/lib/kunit/kunit-example-test.c +++ b/lib/kunit/kunit-example-test.c @@ -190,6 +190,7 @@ static void example_static_stub_test(struct kunit *test) static const struct example_param { int value; } example_params_array[] = {

•   { .value = 3, },
    { .value = 2, },
    { .value = 1, },
    { .value = 0, },
  

@@ -213,8 +214,8 @@ static void example_params_test(struct kunit *test) KUNIT_ASSERT_NOT_NULL(test, param);

    /* Test can be skipped on unsupported param values */

•   if (!param->value)
  

•           kunit_skip(test, "unsupported param value");
  

•   if (!is_power_of_2(param->value))
  

•           kunit_skip(test, "unsupported param value %d", param->value);
  
    /* You can use param values for parameterized testing */
    KUNIT_EXPECT_EQ(test, param->value % param->value, 0);
  

I'm a little tempted to change this to something more power-of-two specific now, like param->value & (param->value - 1). Thoughts?

diff --git a/lib/kunit/test.c b/lib/kunit/test.c index 49698a168437..a53fd7e6d5bf 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -648,12 +648,14 @@ int kunit_run_tests(struct kunit_suite *suite) param_desc, test.status_comment);

•                           kunit_update_stats(&param_stats, test.status);
  

•                           /* Get next param. */
                            param_desc[0] = '\0';
                            test.param_value = test_case->generate_params(test.param_value, param_desc);
                            test.param_index++;
  

•                           kunit_update_stats(&param_stats, test.status);
  

•                           test.status = KUNIT_SUCCESS;
  

•                           test.status_comment[0] = '\0';
                    }
            }
  
  

-- 2.25.1