[PATCH 00/11] selftests: ublk: bug fixes & consolidation

List overview All Threads
Download

newer

older

[PATCH bpf-next v2 00/10]...

[RFC PATCH 0/5] KVM: guest_memfd:...

Ming Lei

3 Mar 2025 3 Mar '25

12:43 p.m.

Hello Jens and guys,

This patchset fixes several issues(1, 2, 4) and consolidate & improve the tests in the following ways:

- support shellcheck and fixes all warning

- misc cleanup

- improve cleanup code path(module load/unload, cleanup temp files)

- help to reuse the same test source code and scripts for other projects(liburing[1], blktest, ...)

- add two stress tests for covering IO workloads vs. removing device & killing ublk server, given buffer lifetime is one big thing for ublk-zc

[1] https://github.com/ming1/liburing/commits/ublk-zc

- just need one line change for overriding skip_code, libring uses 77 and kselftests takes 4

Ming Lei (11): selftests: ublk: make ublk_stop_io_daemon() more reliable selftests: ublk: fix build failure selftests: ublk: add --foreground command line selftests: ublk: fix parsing '-a' argument selftests: ublk: support shellcheck and fix all warning selftests: ublk: don't pass ${dev_id} to _cleanup_test() selftests: ublk: move zero copy feature check into _add_ublk_dev() selftests: ublk: load/unload ublk_drv when preparing & cleaning up tests selftests: ublk: add one stress test for covering IO vs. removing device selftests: ublk: add stress test for covering IO vs. killing ublk server selftests: ublk: improve test usability

tools/testing/selftests/ublk/Makefile | 6 + tools/testing/selftests/ublk/kublk.c | 43 +++-- tools/testing/selftests/ublk/kublk.h | 2 + tools/testing/selftests/ublk/test_common.sh | 167 ++++++++++++++---- tools/testing/selftests/ublk/test_loop_01.sh | 13 +- tools/testing/selftests/ublk/test_loop_02.sh | 14 +- tools/testing/selftests/ublk/test_loop_03.sh | 16 +- tools/testing/selftests/ublk/test_loop_04.sh | 14 +- tools/testing/selftests/ublk/test_null_01.sh | 9 +- .../testing/selftests/ublk/test_stress_01.sh | 47 +++++ .../testing/selftests/ublk/test_stress_02.sh | 47 +++++ 11 files changed, 300 insertions(+), 78 deletions(-) create mode 100755 tools/testing/selftests/ublk/test_stress_01.sh create mode 100755 tools/testing/selftests/ublk/test_stress_02.sh

-- 2.47.0

Show replies by date

Ming Lei

3 Mar 3 Mar

12:43 p.m.

New subject: [PATCH 01/11] selftests: ublk: make ublk_stop_io_daemon() more reliable

Improve ublk_stop_io_daemon() in the following ways:

- don't wait if ->ublksrv_pid becomes -1, which means that the disk has been stopped

- don't wait if ublk char device doesn't exist any more, so we can avoid to rely on inoitfy for wait until the char device is closed

And this way may reduce time of delete command a lot.

Signed-off-by: Ming Lei ming.lei@redhat.com --- tools/testing/selftests/ublk/kublk.c | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-)

diff --git a/tools/testing/selftests/ublk/kublk.c b/tools/testing/selftests/ublk/kublk.c index b65bdaf7e281..2072d880fdc4 100644 --- a/tools/testing/selftests/ublk/kublk.c +++ b/tools/testing/selftests/ublk/kublk.c @@ -691,13 +691,14 @@ static int ublk_start_daemon(const struct dev_ctx *ctx, struct ublk_dev *dev) return ret; }

-static int wait_ublk_dev(char *dev_name, int evt_mask, unsigned timeout) +static int wait_ublk_dev(const char *path, int evt_mask, unsigned timeout) { #define EV_SIZE (sizeof(struct inotify_event)) #define EV_BUF_LEN (128 * (EV_SIZE + 16)) struct pollfd pfd; int fd, wd; int ret = -EINVAL; + const char *dev_name = basename(path);

fd = inotify_init(); if (fd < 0) { @@ -761,18 +762,23 @@ static int ublk_stop_io_daemon(const struct ublk_dev *dev) char ublkc[64]; int ret = 0;

+ if (daemon_pid < 0) + return 0; + /* daemon may be dead already */ if (kill(daemon_pid, 0) < 0) goto wait;

- /* - * Wait until ublk char device is closed, when our daemon is shutdown - */ - snprintf(ublkc, sizeof(ublkc), "%s%d", "ublkc", dev_id); - ret = wait_ublk_dev(ublkc, IN_CLOSE_WRITE, 10); - /* double check and inotify may not be 100% reliable */ + snprintf(ublkc, sizeof(ublkc), "/dev/%s%d", "ublkc", dev_id); + + /* ublk char device may be gone already */ + if (access(ublkc, F_OK) != 0) + goto wait; + + /* Wait until ublk char device is closed, when the daemon is shutdown */ + ret = wait_ublk_dev(ublkc, IN_CLOSE, 10); + /* double check and since it may be closed before starting inotify */ if (ret == -ETIMEDOUT) - /* the daemon doesn't exist now if kill(0) fails */ ret = kill(daemon_pid, 0) < 0; wait: waitpid(daemon_pid, NULL, 0); @@ -910,8 +916,6 @@ static int __cmd_dev_del(struct dev_ctx *ctx) __func__, dev->dev_info.ublksrv_pid, number, ret); ublk_ctrl_del_dev(dev); fail: - if (ret >= 0) - ret = ublk_ctrl_get_info(dev); ublk_ctrl_deinit(dev);

return (ret >= 0) ? 0 : ret;

-- 2.47.0

Keith Busch

9:31 p.m.

New subject: [PATCH 01/11] selftests: ublk: make ublk_stop_io_daemon() more reliable

On Mon, Mar 03, 2025 at 08:43:11PM +0800, Ming Lei wrote:

...

Improve ublk_stop_io_daemon() in the following ways:

don't wait if ->ublksrv_pid becomes -1, which means that the disk

has been stopped

don't wait if ublk char device doesn't exist any more, so we can

avoid to rely on inoitfy for wait until the char device is closed

And this way may reduce time of delete command a lot.

Looks good,

Reviewed-by: Keith Busch kbusch@kernel.org

Ming Lei

12:43 p.m.

New subject: [PATCH 02/11] selftests: ublk: fix build failure

Fixes the following build failure:

ublk//file_backed.c: In function ‘backing_file_tgt_init’: ublk//file_backed.c:28:42: error: ‘O_DIRECT’ undeclared (first use in this function); did you mean ‘O_DIRECTORY’? 28 | fd = open(file, O_RDWR | O_DIRECT); | ^~~~~~~~ | O_DIRECTORY

when trying to reuse this same utility for liburing test.

Signed-off-by: Ming Lei ming.lei@redhat.com --- tools/testing/selftests/ublk/kublk.h | 1 + 1 file changed, 1 insertion(+)

diff --git a/tools/testing/selftests/ublk/kublk.h b/tools/testing/selftests/ublk/kublk.h index 8e43aebf7dfc..8f48eb8568ab 100644 --- a/tools/testing/selftests/ublk/kublk.h +++ b/tools/testing/selftests/ublk/kublk.h @@ -12,6 +12,7 @@ #include <getopt.h> #include <limits.h> #include <poll.h> +#include <fcntl.h> #include <sys/syscall.h> #include <sys/mman.h> #include <sys/ioctl.h>

-- 2.47.0

Keith Busch

9:31 p.m.

New subject: [PATCH 02/11] selftests: ublk: fix build failure

On Mon, Mar 03, 2025 at 08:43:12PM +0800, Ming Lei wrote:

...

Fixes the following build failure:

ublk//file_backed.c: In function `backing_file_tgt_init´: ublk//file_backed.c:28:42: error: `O_DIRECT´ undeclared (first use in this function); did you mean `O_DIRECTORY´? 28 | fd = open(file, O_RDWR | O_DIRECT); | ^~~~~~~~ | O_DIRECTORY

when trying to reuse this same utility for liburing test.

Looks good,

Reviewed-by: Keith Busch kbusch@kernel.org

Ming Lei

12:43 p.m.

New subject: [PATCH 03/11] selftests: ublk: add --foreground command line

Add --foreground command for helping to debug.

Signed-off-by: Ming Lei ming.lei@redhat.com --- tools/testing/selftests/ublk/kublk.c | 17 +++++++++++++---- tools/testing/selftests/ublk/kublk.h | 1 + 2 files changed, 14 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/ublk/kublk.c b/tools/testing/selftests/ublk/kublk.c index 2072d880fdc4..24557a3e5508 100644 --- a/tools/testing/selftests/ublk/kublk.c +++ b/tools/testing/selftests/ublk/kublk.c @@ -679,7 +679,10 @@ static int ublk_start_daemon(const struct dev_ctx *ctx, struct ublk_dev *dev) }

ublk_ctrl_get_info(dev); - ublk_send_dev_event(ctx, dev->dev_info.dev_id); + if (ctx->fg) + ublk_ctrl_dump(dev); + else + ublk_send_dev_event(ctx, dev->dev_info.dev_id);

/* wait until we are terminated */ for (i = 0; i < dinfo->nr_hw_queues; i++) @@ -867,6 +870,9 @@ static int cmd_dev_add(struct dev_ctx *ctx) { int res;

+ if (ctx->fg) + goto run; + ctx->_evtfd = eventfd(0, 0); if (ctx->_evtfd < 0) { ublk_err("%s: failed to create eventfd %s\n", __func__, strerror(errno)); @@ -876,8 +882,9 @@ static int cmd_dev_add(struct dev_ctx *ctx) setsid(); res = fork(); if (res == 0) { - __cmd_dev_add(ctx); - exit(EXIT_SUCCESS); +run: + res = __cmd_dev_add(ctx); + return res; } else if (res > 0) { uint64_t id;

@@ -1044,6 +1051,7 @@ int main(int argc, char *argv[]) { "debug_mask", 1, NULL, 0 }, { "quiet", 0, NULL, 0 }, { "zero_copy", 1, NULL, 'z' }, + { "foreground", 0, NULL, 0 }, { 0, 0, 0, 0 } }; int option_idx, opt; @@ -1087,7 +1095,8 @@ int main(int argc, char *argv[]) ublk_dbg_mask = strtol(optarg, NULL, 16); if (!strcmp(longopts[option_idx].name, "quiet")) ublk_dbg_mask = 0; - break; + if (!strcmp(longopts[option_idx].name, "foreground")) + ctx.fg = 1; } }

diff --git a/tools/testing/selftests/ublk/kublk.h b/tools/testing/selftests/ublk/kublk.h index 8f48eb8568ab..26d9aa9c5ca2 100644 --- a/tools/testing/selftests/ublk/kublk.h +++ b/tools/testing/selftests/ublk/kublk.h @@ -67,6 +67,7 @@ struct dev_ctx { char *files[MAX_BACK_FILES]; unsigned int logging:1; unsigned int all:1; + unsigned int fg:1;

int _evtfd; };

-- 2.47.0

Keith Busch

9:31 p.m.

New subject: [PATCH 03/11] selftests: ublk: add --foreground command line

On Mon, Mar 03, 2025 at 08:43:13PM +0800, Ming Lei wrote:

...

Add --foreground command for helping to debug.

Looks good,

Reviewed-by: Keith Busch kbusch@kernel.org

Ming Lei

12:43 p.m.

New subject: [PATCH 04/11] selftests: ublk: fix parsing '-a' argument

The argument of '-a' doesn't follow any value, so fix it by putting it with '-z' together.

Fixes: ed5820a7e918 ("selftests: ublk: add ublk zero copy test") Signed-off-by: Ming Lei ming.lei@redhat.com --- tools/testing/selftests/ublk/kublk.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/testing/selftests/ublk/kublk.c b/tools/testing/selftests/ublk/kublk.c index 24557a3e5508..148355717ee7 100644 --- a/tools/testing/selftests/ublk/kublk.c +++ b/tools/testing/selftests/ublk/kublk.c @@ -1068,7 +1068,7 @@ int main(int argc, char *argv[]) return ret;

optind = 2; - while ((opt = getopt_long(argc, argv, "t:n:d:q:a:z", + while ((opt = getopt_long(argc, argv, "t:n:d:q:az", longopts, &option_idx)) != -1) { switch (opt) { case 'a':

-- 2.47.0

Keith Busch

9:32 p.m.

New subject: [PATCH 04/11] selftests: ublk: fix parsing '-a' argument

On Mon, Mar 03, 2025 at 08:43:14PM +0800, Ming Lei wrote:

...

The argument of '-a' doesn't follow any value, so fix it by putting it with '-z' together.

Fixes: ed5820a7e918 ("selftests: ublk: add ublk zero copy test")

Looks good.

Reviewed-by: Keith Busch kbusch@kernel.org

Ming Lei

12:43 p.m.

New subject: [PATCH 05/11] selftests: ublk: support shellcheck and fix all warning

Add shellcheck, meantime fixes all warnings.

Signed-off-by: Ming Lei ming.lei@redhat.com --- tools/testing/selftests/ublk/Makefile | 3 ++ tools/testing/selftests/ublk/test_common.sh | 57 +++++++++++--------- tools/testing/selftests/ublk/test_loop_01.sh | 10 ++-- tools/testing/selftests/ublk/test_loop_02.sh | 10 ++-- tools/testing/selftests/ublk/test_loop_03.sh | 10 ++-- tools/testing/selftests/ublk/test_loop_04.sh | 10 ++-- tools/testing/selftests/ublk/test_null_01.sh | 6 +-- 7 files changed, 58 insertions(+), 48 deletions(-)

diff --git a/tools/testing/selftests/ublk/Makefile b/tools/testing/selftests/ublk/Makefile index 555a3ba5b481..9415f6f6df48 100644 --- a/tools/testing/selftests/ublk/Makefile +++ b/tools/testing/selftests/ublk/Makefile @@ -14,3 +14,6 @@ TEST_GEN_PROGS_EXTENDED = kublk include ../lib.mk

$(TEST_GEN_PROGS_EXTENDED): kublk.c null.c file_backed.c + +check: + shellcheck -x -f gcc *.sh diff --git a/tools/testing/selftests/ublk/test_common.sh b/tools/testing/selftests/ublk/test_common.sh index 304f22ffda58..61044cb58138 100755 --- a/tools/testing/selftests/ublk/test_common.sh +++ b/tools/testing/selftests/ublk/test_common.sh @@ -3,46 +3,49 @@

_create_backfile() { local my_size=$1 - local my_file=`mktemp ublk_bpf_${my_size}_XXXXX` + local my_file

- truncate -s ${my_size} ${my_file} - echo $my_file + my_file=$(mktemp ublk_file_"${my_size}"_XXXXX) + truncate -s "${my_size}" "${my_file}" + echo "$my_file" }

_remove_backfile() { local file=$1

- [ -f "$file" ] && rm -f $file + [ -f "$file" ] && rm -f "$file" }

_create_tmp_dir() { - local my_file=`mktemp -d ublk_bpf_dir_XXXXX` + local my_file;

- echo $my_file + my_file=$(mktemp -d ublk_dir_XXXXX) + echo "$my_file" }

_remove_tmp_dir() { local dir=$1

- [ -d "$dir" ] && rmdir $dir + [ -d "$dir" ] && rmdir "$dir" }

_mkfs_mount_test() { local dev=$1 local err_code=0 - local mnt_dir=`_create_tmp_dir` + local mnt_dir;

- mkfs.ext4 -F $dev > /dev/null 2>&1 + mnt_dir=$(_create_tmp_dir) + mkfs.ext4 -F "$dev" > /dev/null 2>&1 err_code=$? if [ $err_code -ne 0 ]; then return $err_code fi

- mount -t ext4 $dev $mnt_dir > /dev/null 2>&1 - umount $dev + mount -t ext4 "$dev" "$mnt_dir" > /dev/null 2>&1 + umount "$dev" err_code=$? - _remove_tmp_dir $mnt_dir + _remove_tmp_dir "$mnt_dir" if [ $err_code -ne 0 ]; then return $err_code fi @@ -73,12 +76,12 @@ _prep_test() { _check_root local type=$1 shift 1 - echo "ublk $type: $@" + echo "ublk $type: $*" }

_show_result() { - if [ $2 -ne 0 ]; then + if [ "$2" -ne 0 ]; then echo "$1 : [FAIL]" else echo "$1 : [PASS]" @@ -86,28 +89,32 @@ _show_result() }

_cleanup_test() { - ${UBLK_PROG} del -n $1 + "${UBLK_PROG}" del -n "$1" }

_add_ublk_dev() { - local kublk_temp=`mktemp /tmp/kublk-XXXXXX` - ${UBLK_PROG} add $@ > ${kublk_temp} 2>&1 - if [ $? -ne 0 ]; then - echo "fail to add ublk dev $@" - exit -1 + local kublk_temp; + local dev_id; + + kublk_temp=$(mktemp /tmp/kublk-XXXXXX) + if ! "${UBLK_PROG}" add "$@" > "${kublk_temp}" 2>&1; then + echo "fail to add ublk dev $*" + return 255 fi - local dev_id=`grep "dev id" ${kublk_temp} | awk -F '[ :]' '{print $3}'` + + dev_id=$(grep "dev id" "${kublk_temp}" | awk -F '[ :]' '{print $3}') udevadm settle - rm -f ${kublk_temp} - echo ${dev_id} + rm -f "${kublk_temp}" + echo "${dev_id}" }

_have_feature() { - if $UBLK_PROG "features" | grep $1 > /dev/null 2>&1; then + if "$UBLK_PROG" "features" | grep "$1" > /dev/null 2>&1; then return 0 fi return 1 }

-export UBLK_PROG=$(pwd)/kublk +UBLK_PROG=$(pwd)/kublk +export UBLK_PROG diff --git a/tools/testing/selftests/ublk/test_loop_01.sh b/tools/testing/selftests/ublk/test_loop_01.sh index 829e8df05942..1d3f934dca4c 100755 --- a/tools/testing/selftests/ublk/test_loop_01.sh +++ b/tools/testing/selftests/ublk/test_loop_01.sh @@ -8,13 +8,13 @@ ERR_CODE=0

_prep_test "loop" "write and verify test"

-backfile_0=`_create_backfile 256M` +backfile_0=$(_create_backfile 256M)

-dev_id=`_add_ublk_dev -t loop $backfile_0` +dev_id=$(_add_ublk_dev -t loop "$backfile_0")

# run fio over the ublk disk fio --name=write_and_verify \ - --filename=/dev/ublkb${dev_id} \ + --filename=/dev/ublkb"${dev_id}" \ --ioengine=libaio --iodepth=16 \ --rw=write \ --size=256M \ @@ -24,8 +24,8 @@ fio --name=write_and_verify \ --bs=4k > /dev/null 2>&1 ERR_CODE=$?

-_cleanup_test ${dev_id} "loop" +_cleanup_test "${dev_id}" "loop"

-_remove_backfile $backfile_0 +_remove_backfile "$backfile_0"

_show_result $TID $ERR_CODE diff --git a/tools/testing/selftests/ublk/test_loop_02.sh b/tools/testing/selftests/ublk/test_loop_02.sh index c71ae63059b8..df06b7804881 100755 --- a/tools/testing/selftests/ublk/test_loop_02.sh +++ b/tools/testing/selftests/ublk/test_loop_02.sh @@ -8,15 +8,15 @@ ERR_CODE=0

_prep_test "loop" "mkfs & mount & umount"

-backfile_0=`_create_backfile 256M` +backfile_0=$(_create_backfile 256M)

-dev_id=`_add_ublk_dev -t loop $backfile_0` +dev_id=$(_add_ublk_dev -t loop "$backfile_0")

-_mkfs_mount_test /dev/ublkb${dev_id} +_mkfs_mount_test /dev/ublkb"${dev_id}" ERR_CODE=$?

-_cleanup_test ${dev_id} "loop" +_cleanup_test "${dev_id}" "loop"

-_remove_backfile $backfile_0 +_remove_backfile "$backfile_0"

_show_result $TID $ERR_CODE diff --git a/tools/testing/selftests/ublk/test_loop_03.sh b/tools/testing/selftests/ublk/test_loop_03.sh index e781ac6db6b4..2255b4296590 100755 --- a/tools/testing/selftests/ublk/test_loop_03.sh +++ b/tools/testing/selftests/ublk/test_loop_03.sh @@ -10,13 +10,13 @@ _have_feature "ZERO_COPY" || exit 4

_prep_test "loop" "write and verify over zero copy"

-backfile_0=`_create_backfile 256M` +backfile_0=$(_create_backfile 256M)

-dev_id=`_add_ublk_dev -t loop $backfile_0 -z` +dev_id=$(_add_ublk_dev -t loop -z "$backfile_0")

# run fio over the ublk disk fio --name=write_and_verify \ - --filename=/dev/ublkb${dev_id} \ + --filename=/dev/ublkb"${dev_id}" \ --ioengine=libaio --iodepth=64 \ --rw=write \ --size=256M \ @@ -26,8 +26,8 @@ fio --name=write_and_verify \ --bs=4k > /dev/null 2>&1 ERR_CODE=$?

-_cleanup_test ${dev_id} "loop" +_cleanup_test "${dev_id}" "loop"

-_remove_backfile $backfile_0 +_remove_backfile "$backfile_0"

_show_result $TID $ERR_CODE diff --git a/tools/testing/selftests/ublk/test_loop_04.sh b/tools/testing/selftests/ublk/test_loop_04.sh index 6ab67247c809..a797b25213ec 100755 --- a/tools/testing/selftests/ublk/test_loop_04.sh +++ b/tools/testing/selftests/ublk/test_loop_04.sh @@ -8,15 +8,15 @@ ERR_CODE=0

_prep_test "loop" "mkfs & mount & umount with zero copy"

-backfile_0=`_create_backfile 256M` +backfile_0=$(_create_backfile 256M)

-dev_id=`_add_ublk_dev -t loop -z $backfile_0` +dev_id=$(_add_ublk_dev -t loop -z "$backfile_0")

-_mkfs_mount_test /dev/ublkb${dev_id} +_mkfs_mount_test /dev/ublkb"${dev_id}" ERR_CODE=$?

-_cleanup_test ${dev_id} "loop" +_cleanup_test "${dev_id}" "loop"

-_remove_backfile $backfile_0 +_remove_backfile "$backfile_0"

_show_result $TID $ERR_CODE diff --git a/tools/testing/selftests/ublk/test_null_01.sh b/tools/testing/selftests/ublk/test_null_01.sh index 04fc3ac7c716..b048ddc4ae6f 100755 --- a/tools/testing/selftests/ublk/test_null_01.sh +++ b/tools/testing/selftests/ublk/test_null_01.sh @@ -8,12 +8,12 @@ ERR_CODE=0

_prep_test "null" "basic IO test"

-dev_id=`_add_ublk_dev -t null` +dev_id=$(_add_ublk_dev -t null)

# run fio over the two disks -fio --name=job1 --filename=/dev/ublkb${dev_id} --ioengine=libaio --rw=readwrite --iodepth=32 --size=256M > /dev/null 2>&1 +fio --name=job1 --filename=/dev/ublkb"${dev_id}" --ioengine=libaio --rw=readwrite --iodepth=32 --size=256M > /dev/null 2>&1 ERR_CODE=$?

-_cleanup_test ${dev_id} "null" +_cleanup_test "${dev_id}" "null"

_show_result $TID $ERR_CODE

-- 2.47.0

Ming Lei

12:43 p.m.

New subject: [PATCH 06/11] selftests: ublk: don't pass ${dev_id} to _cleanup_test()

More devices can be created in single tests, so simply remove all ublk devices in _cleanup_test(), meantime remove the ${dev_id} argument of _cleanup_test().

Signed-off-by: Ming Lei ming.lei@redhat.com --- tools/testing/selftests/ublk/test_common.sh | 2 +- tools/testing/selftests/ublk/test_loop_01.sh | 2 +- tools/testing/selftests/ublk/test_loop_02.sh | 2 +- tools/testing/selftests/ublk/test_loop_03.sh | 2 +- tools/testing/selftests/ublk/test_loop_04.sh | 2 +- tools/testing/selftests/ublk/test_null_01.sh | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/tools/testing/selftests/ublk/test_common.sh b/tools/testing/selftests/ublk/test_common.sh index 61044cb58138..d70690281d14 100755 --- a/tools/testing/selftests/ublk/test_common.sh +++ b/tools/testing/selftests/ublk/test_common.sh @@ -89,7 +89,7 @@ _show_result() }

_cleanup_test() { - "${UBLK_PROG}" del -n "$1" + "${UBLK_PROG}" del -a }

_add_ublk_dev() { diff --git a/tools/testing/selftests/ublk/test_loop_01.sh b/tools/testing/selftests/ublk/test_loop_01.sh index 1d3f934dca4c..48a85796ca43 100755 --- a/tools/testing/selftests/ublk/test_loop_01.sh +++ b/tools/testing/selftests/ublk/test_loop_01.sh @@ -24,7 +24,7 @@ fio --name=write_and_verify \ --bs=4k > /dev/null 2>&1 ERR_CODE=$?

-_cleanup_test "${dev_id}" "loop" +_cleanup_test "loop"

_remove_backfile "$backfile_0"

diff --git a/tools/testing/selftests/ublk/test_loop_02.sh b/tools/testing/selftests/ublk/test_loop_02.sh index df06b7804881..0a4b5fadbc73 100755 --- a/tools/testing/selftests/ublk/test_loop_02.sh +++ b/tools/testing/selftests/ublk/test_loop_02.sh @@ -15,7 +15,7 @@ dev_id=$(_add_ublk_dev -t loop "$backfile_0") _mkfs_mount_test /dev/ublkb"${dev_id}" ERR_CODE=$?

-_cleanup_test "${dev_id}" "loop" +_cleanup_test "loop"

_remove_backfile "$backfile_0"

diff --git a/tools/testing/selftests/ublk/test_loop_03.sh b/tools/testing/selftests/ublk/test_loop_03.sh index 2255b4296590..5a11356e502c 100755 --- a/tools/testing/selftests/ublk/test_loop_03.sh +++ b/tools/testing/selftests/ublk/test_loop_03.sh @@ -26,7 +26,7 @@ fio --name=write_and_verify \ --bs=4k > /dev/null 2>&1 ERR_CODE=$?

-_cleanup_test "${dev_id}" "loop" +_cleanup_test "loop"

_remove_backfile "$backfile_0"

diff --git a/tools/testing/selftests/ublk/test_loop_04.sh b/tools/testing/selftests/ublk/test_loop_04.sh index a797b25213ec..7e0d4dd8127e 100755 --- a/tools/testing/selftests/ublk/test_loop_04.sh +++ b/tools/testing/selftests/ublk/test_loop_04.sh @@ -15,7 +15,7 @@ dev_id=$(_add_ublk_dev -t loop -z "$backfile_0") _mkfs_mount_test /dev/ublkb"${dev_id}" ERR_CODE=$?

-_cleanup_test "${dev_id}" "loop" +_cleanup_test "loop"

_remove_backfile "$backfile_0"

diff --git a/tools/testing/selftests/ublk/test_null_01.sh b/tools/testing/selftests/ublk/test_null_01.sh index b048ddc4ae6f..af11e73b7df6 100755 --- a/tools/testing/selftests/ublk/test_null_01.sh +++ b/tools/testing/selftests/ublk/test_null_01.sh @@ -14,6 +14,6 @@ dev_id=$(_add_ublk_dev -t null) fio --name=job1 --filename=/dev/ublkb"${dev_id}" --ioengine=libaio --rw=readwrite --iodepth=32 --size=256M > /dev/null 2>&1 ERR_CODE=$?

-_cleanup_test "${dev_id}" "null" +_cleanup_test "null"

_show_result $TID $ERR_CODE

-- 2.47.0

Ming Lei

12:43 p.m.

New subject: [PATCH 07/11] selftests: ublk: move zero copy feature check into _add_ublk_dev()

Move zero copy feature check into _add_ublk_dev() since we will have more tests which requires to cover zero copy.

Then one check function of _check_add_dev() has to be added for dealing with cleanup since '_add_ublk_dev()' is run in sub-shell, and we can't exit from it to terminal shell.

Meantime always return error code from _add_ublk_dev().

Signed-off-by: Ming Lei ming.lei@redhat.com --- tools/testing/selftests/ublk/test_common.sh | 56 ++++++++++++++++---- tools/testing/selftests/ublk/test_loop_01.sh | 1 + tools/testing/selftests/ublk/test_loop_02.sh | 2 +- tools/testing/selftests/ublk/test_loop_03.sh | 4 +- tools/testing/selftests/ublk/test_loop_04.sh | 2 +- tools/testing/selftests/ublk/test_null_01.sh | 1 + 6 files changed, 50 insertions(+), 16 deletions(-)

diff --git a/tools/testing/selftests/ublk/test_common.sh b/tools/testing/selftests/ublk/test_common.sh index d70690281d14..40bf42f1bed2 100755 --- a/tools/testing/selftests/ublk/test_common.sh +++ b/tools/testing/selftests/ublk/test_common.sh @@ -1,6 +1,8 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0

+UBLK_SKIP_CODE=4 + _create_backfile() { local my_size=$1 local my_file @@ -79,12 +81,37 @@ _prep_test() { echo "ublk $type: $*" }

+_remove_test_files() +{ + local files=$* + + for file in ${files}; do + [ -f "${file}" ] && rm -f "${file}" + done +} + _show_result() { - if [ "$2" -ne 0 ]; then - echo "$1 : [FAIL]" - else + if [ "$2" -eq 0 ]; then echo "$1 : [PASS]" + elif [ "$2" -eq 4 ]; then + echo "$1 : [SKIP]" + else + echo "$1 : [FAIL]" + fi + [ "$2" -ne 0 ] && exit "$2" + return 0 +} + +# don't call from sub-shell, otherwise can't exit +_check_add_dev() +{ + local tid=$1 + local code=$2 + shift 2 + if [ "${code}" -ne 0 ]; then + _remove_test_files "$@" + _show_result "${tid}" "${code}" fi }

@@ -92,13 +119,28 @@ _cleanup_test() { "${UBLK_PROG}" del -a }

+_have_feature() +{ + if $UBLK_PROG "features" | grep "$1" > /dev/null 2>&1; then + return 0 + fi + return 1 +} + _add_ublk_dev() { local kublk_temp; local dev_id;

+ if echo "$@" | grep -q "-z"; then + if ! _have_feature "ZERO_COPY"; then + return ${UBLK_SKIP_CODE} + fi + fi + kublk_temp=$(mktemp /tmp/kublk-XXXXXX) if ! "${UBLK_PROG}" add "$@" > "${kublk_temp}" 2>&1; then echo "fail to add ublk dev $*" + rm -f "${kublk_temp}" return 255 fi

@@ -108,13 +150,5 @@ _add_ublk_dev() { echo "${dev_id}" }

-_have_feature() -{ - if "$UBLK_PROG" "features" | grep "$1" > /dev/null 2>&1; then - return 0 - fi - return 1 -} - UBLK_PROG=$(pwd)/kublk export UBLK_PROG diff --git a/tools/testing/selftests/ublk/test_loop_01.sh b/tools/testing/selftests/ublk/test_loop_01.sh index 48a85796ca43..12bba9e5daa5 100755 --- a/tools/testing/selftests/ublk/test_loop_01.sh +++ b/tools/testing/selftests/ublk/test_loop_01.sh @@ -11,6 +11,7 @@ _prep_test "loop" "write and verify test" backfile_0=$(_create_backfile 256M)

dev_id=$(_add_ublk_dev -t loop "$backfile_0") +_check_add_dev $TID $? "${backfile_0}"

# run fio over the ublk disk fio --name=write_and_verify \ diff --git a/tools/testing/selftests/ublk/test_loop_02.sh b/tools/testing/selftests/ublk/test_loop_02.sh index 0a4b5fadbc73..9a163296ac83 100755 --- a/tools/testing/selftests/ublk/test_loop_02.sh +++ b/tools/testing/selftests/ublk/test_loop_02.sh @@ -9,8 +9,8 @@ ERR_CODE=0 _prep_test "loop" "mkfs & mount & umount"

backfile_0=$(_create_backfile 256M) - dev_id=$(_add_ublk_dev -t loop "$backfile_0") +_check_add_dev $TID $? "$backfile_0"

_mkfs_mount_test /dev/ublkb"${dev_id}" ERR_CODE=$? diff --git a/tools/testing/selftests/ublk/test_loop_03.sh b/tools/testing/selftests/ublk/test_loop_03.sh index 5a11356e502c..72a1d072cfbd 100755 --- a/tools/testing/selftests/ublk/test_loop_03.sh +++ b/tools/testing/selftests/ublk/test_loop_03.sh @@ -6,13 +6,11 @@ TID="loop_03" ERR_CODE=0

-_have_feature "ZERO_COPY" || exit 4 - _prep_test "loop" "write and verify over zero copy"

backfile_0=$(_create_backfile 256M) - dev_id=$(_add_ublk_dev -t loop -z "$backfile_0") +_check_add_dev $TID $? "$backfile_0"

# run fio over the ublk disk fio --name=write_and_verify \ diff --git a/tools/testing/selftests/ublk/test_loop_04.sh b/tools/testing/selftests/ublk/test_loop_04.sh index 7e0d4dd8127e..676c4652d758 100755 --- a/tools/testing/selftests/ublk/test_loop_04.sh +++ b/tools/testing/selftests/ublk/test_loop_04.sh @@ -9,8 +9,8 @@ ERR_CODE=0 _prep_test "loop" "mkfs & mount & umount with zero copy"

backfile_0=$(_create_backfile 256M) - dev_id=$(_add_ublk_dev -t loop -z "$backfile_0") +_check_add_dev $TID $? "$backfile_0"

_mkfs_mount_test /dev/ublkb"${dev_id}" ERR_CODE=$? diff --git a/tools/testing/selftests/ublk/test_null_01.sh b/tools/testing/selftests/ublk/test_null_01.sh index af11e73b7df6..e2847a50823a 100755 --- a/tools/testing/selftests/ublk/test_null_01.sh +++ b/tools/testing/selftests/ublk/test_null_01.sh @@ -9,6 +9,7 @@ ERR_CODE=0 _prep_test "null" "basic IO test"

dev_id=$(_add_ublk_dev -t null) +_check_add_dev $TID $?

# run fio over the two disks fio --name=job1 --filename=/dev/ublkb"${dev_id}" --ioengine=libaio --rw=readwrite --iodepth=32 --size=256M > /dev/null 2>&1

-- 2.47.0

Ming Lei

12:43 p.m.

New subject: [PATCH 08/11] selftests: ublk: load/unload ublk_drv when preparing & cleaning up tests

Load ublk_drv module in _prep_test(), and unload it in _cleanup_test(), so that test can always be done in consistent state.

Signed-off-by: Ming Lei ming.lei@redhat.com --- tools/testing/selftests/ublk/test_common.sh | 5 +++++ 1 file changed, 5 insertions(+)

diff --git a/tools/testing/selftests/ublk/test_common.sh b/tools/testing/selftests/ublk/test_common.sh index 40bf42f1bed2..bcb0c7aa3956 100755 --- a/tools/testing/selftests/ublk/test_common.sh +++ b/tools/testing/selftests/ublk/test_common.sh @@ -64,6 +64,7 @@ _check_root() {

_remove_ublk_devices() { ${UBLK_PROG} del -a + modprobe -r ublk_drv }

_get_ublk_dev_state() { @@ -78,6 +79,7 @@ _prep_test() { _check_root local type=$1 shift 1 + modprobe ublk_drv echo "ublk $type: $*" }

@@ -131,6 +133,9 @@ _add_ublk_dev() { local kublk_temp; local dev_id;

+ if [ ! -c /dev/ublk-control ]; then + return ${UBLK_SKIP_CODE} + fi if echo "$@" | grep -q "-z"; then if ! _have_feature "ZERO_COPY"; then return ${UBLK_SKIP_CODE}

-- 2.47.0

Ming Lei

12:43 p.m.

New subject: [PATCH 09/11] selftests: ublk: add one stress test for covering IO vs. removing device

Add stress_test_01 for running IO vs. removing device for verifying that ublk device removal can work as expected when heavy IO workloads are in progress.

null, loop and loop/zc are covered in this tests.

Signed-off-by: Ming Lei ming.lei@redhat.com --- tools/testing/selftests/ublk/Makefile | 2 + tools/testing/selftests/ublk/test_common.sh | 26 ++++++++++ .../testing/selftests/ublk/test_stress_01.sh | 47 +++++++++++++++++++ 3 files changed, 75 insertions(+) create mode 100755 tools/testing/selftests/ublk/test_stress_01.sh

diff --git a/tools/testing/selftests/ublk/Makefile b/tools/testing/selftests/ublk/Makefile index 9415f6f6df48..40af938cd277 100644 --- a/tools/testing/selftests/ublk/Makefile +++ b/tools/testing/selftests/ublk/Makefile @@ -9,6 +9,8 @@ TEST_PROGS += test_loop_02.sh TEST_PROGS += test_loop_03.sh TEST_PROGS += test_loop_04.sh

+TEST_PROGS += test_stress_01.sh + TEST_GEN_PROGS_EXTENDED = kublk

include ../lib.mk diff --git a/tools/testing/selftests/ublk/test_common.sh b/tools/testing/selftests/ublk/test_common.sh index bcb0c7aa3956..89244a7e275c 100755 --- a/tools/testing/selftests/ublk/test_common.sh +++ b/tools/testing/selftests/ublk/test_common.sh @@ -155,5 +155,31 @@ _add_ublk_dev() { echo "${dev_id}" }

+__remove_ublk_dev_return() { + local dev_id=$1 + + ${UBLK_PROG} del -n "${dev_id}" + local res=$? + udevadm settle + return ${res} +} + +__run_io_and_remove() +{ + local dev_id=$1 + local size=$2 + + fio --name=job1 --filename=/dev/ublkb"${dev_id}" --ioengine=libaio \ + --rw=readwrite --iodepth=64 --size="${size}" --numjobs=4 \ + --runtime=20 --time_based > /dev/null 2>&1 & + sleep 2 + if ! __remove_ublk_dev_return "${dev_id}"; then + echo "delete dev ${dev_id} failed" + return 255 + fi + wait +} + + UBLK_PROG=$(pwd)/kublk export UBLK_PROG diff --git a/tools/testing/selftests/ublk/test_stress_01.sh b/tools/testing/selftests/ublk/test_stress_01.sh new file mode 100755 index 000000000000..2dfd01cfd265 --- /dev/null +++ b/tools/testing/selftests/ublk/test_stress_01.sh @@ -0,0 +1,47 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +. test_common.sh +TID="stress_01" +ERR_CODE=0 +DEV_ID=-1 + +ublk_io_and_remove() +{ + local size=$1 + shift 1 + local backfile="" + if echo "$@" | grep -q "loop"; then + backfile=${*: -1} + fi + DEV_ID=$(_add_ublk_dev "$@") + _check_add_dev $TID $? "${backfile}" + + echo "run ublk IO vs. remove device(ublk add $*)" + if ! __run_io_and_remove "${DEV_ID}" "${size}"; then + echo "/dev/ublkc${DEV_ID} isn't removed" + _remove_backfile "${backfile}" + exit 255 + fi +} + +_prep_test "stress" "run IO and remove device" + +ublk_io_and_remove 8G -t null +ERR_CODE=$? +if [ ${ERR_CODE} -ne 0 ]; then + _show_result $TID $ERR_CODE +fi + +BACK_FILE=$(_create_backfile 256M) +ublk_io_and_remove 256M -t loop "${BACK_FILE}" +ERR_CODE=$? +if [ ${ERR_CODE} -ne 0 ]; then + _show_result $TID $ERR_CODE +fi + +ublk_io_and_remove 256M -t loop -z "${BACK_FILE}" +ERR_CODE=$? +_cleanup_test "stress" +_remove_backfile "${BACK_FILE}" +_show_result $TID $ERR_CODE

-- 2.47.0

Ming Lei

12:43 p.m.

New subject: [PATCH 10/11] selftests: ublk: add stress test for covering IO vs. killing ublk server

Add stress_test_01 for running IO vs. killing ublk server, so io_uring exit & cancel code path can be covered, same with ublk's cancel code path.

Especially IO buffer lifetime is one big thing for ublk zero copy, the added test can verify if this area works as expected.

Signed-off-by: Ming Lei ming.lei@redhat.com --- tools/testing/selftests/ublk/Makefile | 1 + tools/testing/selftests/ublk/test_common.sh | 29 ++++++++++++ .../testing/selftests/ublk/test_stress_01.sh | 2 +- .../testing/selftests/ublk/test_stress_02.sh | 47 +++++++++++++++++++ 4 files changed, 78 insertions(+), 1 deletion(-) create mode 100755 tools/testing/selftests/ublk/test_stress_02.sh

diff --git a/tools/testing/selftests/ublk/Makefile b/tools/testing/selftests/ublk/Makefile index 40af938cd277..5d8d5939f051 100644 --- a/tools/testing/selftests/ublk/Makefile +++ b/tools/testing/selftests/ublk/Makefile @@ -10,6 +10,7 @@ TEST_PROGS += test_loop_03.sh TEST_PROGS += test_loop_04.sh

TEST_PROGS += test_stress_01.sh +TEST_PROGS += test_stress_02.sh

TEST_GEN_PROGS_EXTENDED = kublk

diff --git a/tools/testing/selftests/ublk/test_common.sh b/tools/testing/selftests/ublk/test_common.sh index 89244a7e275c..92596d0d0013 100755 --- a/tools/testing/selftests/ublk/test_common.sh +++ b/tools/testing/selftests/ublk/test_common.sh @@ -155,6 +155,26 @@ _add_ublk_dev() { echo "${dev_id}" }

+# kill the ublk daemon and return ublk device state +__ublk_kill_daemon() +{ + local dev_id=$1 + local exp_state=$2 + local daemon_pid + local state + + daemon_pid=$(_get_ublk_daemon_pid "${dev_id}") + state=$(_get_ublk_dev_state "${dev_id}") + + for ((j=0;j<50;j++)); do + [ "$state" == "$exp_state" ] && break + kill -9 "$daemon_pid" > /dev/null 2>&1 + sleep 1 + state=$(_get_ublk_dev_state "${dev_id}") + done + echo "$state" +} + __remove_ublk_dev_return() { local dev_id=$1

@@ -168,11 +188,20 @@ __run_io_and_remove() { local dev_id=$1 local size=$2 + local kill_server=$3

fio --name=job1 --filename=/dev/ublkb"${dev_id}" --ioengine=libaio \ --rw=readwrite --iodepth=64 --size="${size}" --numjobs=4 \ --runtime=20 --time_based > /dev/null 2>&1 & sleep 2 + if [ "${kill_server}" = "yes" ]; then + local state + state=$(__ublk_kill_daemon "${dev_id}" "DEAD") + if [ "$state" != "DEAD" ]; then + echo "device isn't dead($state) after killing daemon" + return 255 + fi + fi if ! __remove_ublk_dev_return "${dev_id}"; then echo "delete dev ${dev_id} failed" return 255 diff --git a/tools/testing/selftests/ublk/test_stress_01.sh b/tools/testing/selftests/ublk/test_stress_01.sh index 2dfd01cfd265..c1cdde3e79f7 100755 --- a/tools/testing/selftests/ublk/test_stress_01.sh +++ b/tools/testing/selftests/ublk/test_stress_01.sh @@ -18,7 +18,7 @@ ublk_io_and_remove() _check_add_dev $TID $? "${backfile}"

echo "run ublk IO vs. remove device(ublk add $*)" - if ! __run_io_and_remove "${DEV_ID}" "${size}"; then + if ! __run_io_and_remove "${DEV_ID}" "${size}" "no"; then echo "/dev/ublkc${DEV_ID} isn't removed" _remove_backfile "${backfile}" exit 255 diff --git a/tools/testing/selftests/ublk/test_stress_02.sh b/tools/testing/selftests/ublk/test_stress_02.sh new file mode 100755 index 000000000000..ec758f649a97 --- /dev/null +++ b/tools/testing/selftests/ublk/test_stress_02.sh @@ -0,0 +1,47 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +. test_common.sh +TID="stress_02" +ERR_CODE=0 +DEV_ID=-1 + +ublk_io_and_kill_daemon() +{ + local size=$1 + shift 1 + local backfile="" + if echo "$@" | grep -q "loop"; then + backfile=${*: -1} + fi + DEV_ID=$(_add_ublk_dev "$@") + _check_add_dev $TID $? "${backfile}" + + echo "run ublk IO vs kill ublk server(ublk add $*)" + if ! __run_io_and_remove "${DEV_ID}" "${size}" "yes"; then + echo "/dev/ublkc${DEV_ID} isn't removed res ${res}" + _remove_backfile "${backfile}" + exit 255 + fi +} + +_prep_test "stress" "run IO and kill ublk server" + +ublk_io_and_kill_daemon 8G -t null +ERR_CODE=$? +if [ ${ERR_CODE} -ne 0 ]; then + _show_result $TID $ERR_CODE +fi + +BACK_FILE=$(_create_backfile 256M) +ublk_io_and_kill_daemon 256M -t loop "${BACK_FILE}" +ERR_CODE=$? +if [ ${ERR_CODE} -ne 0 ]; then + _show_result $TID $ERR_CODE +fi + +ublk_io_and_kill_daemon 256M -t loop -z "${BACK_FILE}" +ERR_CODE=$? +_cleanup_test "stress" +_remove_backfile "${BACK_FILE}" +_show_result $TID $ERR_CODE

-- 2.47.0

Ming Lei

12:43 p.m.

New subject: [PATCH 11/11] selftests: ublk: improve test usability

Add UBLK_TEST_QUIET, so we can print test result(PASS/SKIP/FAIL) only.

Also always run from test script's current directory, then the same test script can be started from other work directory.

This way helps a lot to reuse this test source code and scripts for other projects(liburing, blktests, ...)

Signed-off-by: Ming Lei ming.lei@redhat.com --- tools/testing/selftests/ublk/test_common.sh | 10 ++++++++-- tools/testing/selftests/ublk/test_loop_01.sh | 2 +- tools/testing/selftests/ublk/test_loop_02.sh | 2 +- tools/testing/selftests/ublk/test_loop_03.sh | 2 +- tools/testing/selftests/ublk/test_loop_04.sh | 2 +- tools/testing/selftests/ublk/test_null_01.sh | 2 +- tools/testing/selftests/ublk/test_stress_01.sh | 4 ++-- tools/testing/selftests/ublk/test_stress_02.sh | 4 ++-- 8 files changed, 17 insertions(+), 11 deletions(-)

diff --git a/tools/testing/selftests/ublk/test_common.sh b/tools/testing/selftests/ublk/test_common.sh index 92596d0d0013..350380facd9f 100755 --- a/tools/testing/selftests/ublk/test_common.sh +++ b/tools/testing/selftests/ublk/test_common.sh @@ -80,7 +80,7 @@ _prep_test() { local type=$1 shift 1 modprobe ublk_drv - echo "ublk $type: $*" + [ "$UBLK_TEST_QUIET" -eq 0 ] && echo "ublk $type: $*" }

_remove_test_files() @@ -209,6 +209,12 @@ __run_io_and_remove() wait }

+_ublk_test_top_dir() +{ + cd "$(dirname "$0")" && pwd +}

-UBLK_PROG=$(pwd)/kublk +UBLK_PROG=$(_ublk_test_top_dir)/kublk +UBLK_TEST_QUIET=1 export UBLK_PROG +export UBLK_TEST_QUIET diff --git a/tools/testing/selftests/ublk/test_loop_01.sh b/tools/testing/selftests/ublk/test_loop_01.sh index 12bba9e5daa5..c882d2a08e13 100755 --- a/tools/testing/selftests/ublk/test_loop_01.sh +++ b/tools/testing/selftests/ublk/test_loop_01.sh @@ -1,7 +1,7 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0