- Linux-kselftest-mirror - lists.linaro.org

[PATCH v2 bpf-next 0/3] bpftool: support loading flow dissector

by Stanislav Fomichev

v2 changes: * addressed comments/style issues from Jakub Kicinski & Quentin Monnet * removed logic that populates jump table * added cleanup for partial failure in bpf_object__pin This patch series adds support for loading and attaching flow dissector programs from the bpftool: * first patch fixes flow dissector section name in the selftests (so libbpf auto-detection works) * second patch adds proper cleanup to bpf_object__pin which is now being used to attach all flow dissector progs/maps * third patch adds actual support to the bpftool See third patch for the description/details. Stanislav Fomichev (3): selftests/bpf: rename flow dissector section to flow_dissector libbpf: cleanup after partial failure in bpf_object__pin bpftool: support loading flow dissector .../bpftool/Documentation/bpftool-prog.rst | 26 +++-- tools/bpf/bpftool/bash-completion/bpftool | 2 +- tools/bpf/bpftool/common.c | 30 +++--- tools/bpf/bpftool/main.h | 1 + tools/bpf/bpftool/prog.c | 94 ++++++++++++++----- tools/lib/bpf/libbpf.c | 58 ++++++++++-- tools/testing/selftests/bpf/bpf_flow.c | 2 +- .../selftests/bpf/test_flow_dissector.sh | 2 +- 8 files changed, 151 insertions(+), 64 deletions(-) -- 2.19.1.930.g4563a0d9d0-goog

6 years, 2 months

3
14
0 0

[PATCH bpf-next 0/2] bpftool: support loading flow dissector

by Stanislav Fomichev

This patch series adds support for loading and attaching flow dissector programs from the bpftool: * first patch fixes flow dissector section name in the selftests (so libbpf auto-detection works) * second patch adds actual support to the bpftool See second patch for the description/details. Stanislav Fomichev (2): selftests/bpf: rename flow dissector section to flow_dissector bpftool: support loading flow dissector .../bpftool/Documentation/bpftool-prog.rst | 16 ++- tools/bpf/bpftool/common.c | 32 +++-- tools/bpf/bpftool/main.h | 1 + tools/bpf/bpftool/prog.c | 135 +++++++++++++++--- tools/testing/selftests/bpf/bpf_flow.c | 2 +- .../selftests/bpf/test_flow_dissector.sh | 2 +- 6 files changed, 143 insertions(+), 45 deletions(-) -- 2.19.1.930.g4563a0d9d0-goog

6 years, 2 months

4
13
0 0

[PATCH resend] Documentation/dev-tools: clean up kselftest.rst

by Randy Dunlap

From: Randy Dunlap <rdunlap(a)infradead.org> This is a small cleanup to kselftest.rst: - Fix some language typos in the usage instructions. - Change one non-ASCII space to an ASCII space. Signed-off-by: Randy Dunlap <rdunlap(a)infradead.org> Cc: Shuah Khan <shuah(a)kernel.org> Cc: Jonathan Corbet <corbet(a)lwn.net> Cc: linux-kselftest(a)vger.kernel.org Cc: linux-doc(a)vger.kernel.org --- Documentation/dev-tools/kselftest.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) --- linux-next-20181101.orig/Documentation/dev-tools/kselftest.rst +++ linux-next-20181101/Documentation/dev-tools/kselftest.rst @@ -9,7 +9,7 @@ and booting a kernel. On some systems, hot-plug tests could hang forever waiting for cpu and memory to be ready to be offlined. A special hot-plug target is created -to run full range of hot-plug tests. In default mode, hot-plug tests run +to run the full range of hot-plug tests. In default mode, hot-plug tests run in safe mode with a limited scope. In limited mode, cpu-hotplug test is run on a single cpu as opposed to all hotplug capable cpus, and memory hotplug test is run on 2% of hotplug capable memory instead of 10%. @@ -89,9 +89,9 @@ Note that some tests will require root p Install selftests ================= -You can use kselftest_install.sh tool installs selftests in default -location which is tools/testing/selftests/kselftest or a user specified -location. +You can use the kselftest_install.sh tool to install selftests in the +default location, which is tools/testing/selftests/kselftest, or in a +user specified location. To install selftests in default location:: @@ -109,7 +109,7 @@ Running installed selftests Kselftest install as well as the Kselftest tarball provide a script named "run_kselftest.sh" to run the tests. -You can simply do the following to run the installed Kselftests. Please +You can simply do the following to run the installed Kselftests. Please note some tests will require root privileges:: $ cd kselftest @@ -139,7 +139,7 @@ Contributing new tests (details) default. TEST_CUSTOM_PROGS should be used by tests that require custom build - rule and prevent common build rule use. + rules and prevent common build rule use. TEST_PROGS are for test shell scripts. Please ensure shell script has its exec bit set. Otherwise, lib.mk run_tests will generate a warning.

6 years, 2 months

3
2
0 0

[PATCH] kselftests/bpf: use ping6 as the default ipv6 ping binary when it exists

by Li Zhijian

At commit deee2cae27 ("kselftests/bpf: use ping6 as the default ipv6 ping binary if it exists"), it fixed similar issues for shell script, but it missed a same issue in the C code. Fixes: 371e4fcc9d96 ("selftests/bpf: cgroup local storage-based network counters") CC: Philip Li <philip.li(a)intel.com> Reported-by: kernel test robot <rong.a.chen(a)intel.com> Signed-off-by: Li Zhijian <lizhijian(a)cn.fujitsu.com> --- tools/testing/selftests/bpf/test_netcnt.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/test_netcnt.c b/tools/testing/selftests/bpf/test_netcnt.c index 7887df6..44889bc 100644 --- a/tools/testing/selftests/bpf/test_netcnt.c +++ b/tools/testing/selftests/bpf/test_netcnt.c @@ -81,7 +81,11 @@ int main(int argc, char **argv) goto err; } - assert(system("ping localhost -6 -c 10000 -f -q > /dev/null") == 0); + if (system("which ping6 &>/dev/null") == 0) { + assert(system("ping6 localhost -c 10000 -f -q > /dev/null") == 0); + } else { + assert(system("ping -6 localhost -c 10000 -f -q > /dev/null") == 0); + } if (bpf_prog_query(cgroup_fd, BPF_CGROUP_INET_EGRESS, 0, NULL, NULL, &prog_cnt)) { -- 2.7.4

6 years, 2 months

1
0
0 0

[PATCH] media: rc: self test for IR encoders and decoders

by Sean Young

ir-loopback can transmit IR on one rc device and check the correct scancode and protocol is decoded on a different rc device. This can be used to check IR transmission between two rc devices. Using rc-loopback, we use it to check the IR encoders and decoders themselves. No hardware is required for this test. Signed-off-by: Sean Young <sean(a)mess.org> Cc: Shuah Khan <shuah(a)kernel.org> --- tools/testing/selftests/Makefile | 1 + tools/testing/selftests/ir/.gitignore | 1 + tools/testing/selftests/ir/Makefile | 5 + tools/testing/selftests/ir/ir_loopback.c | 199 ++++++++++++++++++++++ tools/testing/selftests/ir/ir_loopback.sh | 20 +++ 5 files changed, 226 insertions(+) create mode 100644 tools/testing/selftests/ir/.gitignore create mode 100644 tools/testing/selftests/ir/Makefile create mode 100644 tools/testing/selftests/ir/ir_loopback.c create mode 100755 tools/testing/selftests/ir/ir_loopback.sh diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index f1fe492c8e17..995034ea5546 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -15,6 +15,7 @@ TARGETS += futex TARGETS += gpio TARGETS += intel_pstate TARGETS += ipc +TARGETS += ir TARGETS += kcmp TARGETS += kvm TARGETS += lib diff --git a/tools/testing/selftests/ir/.gitignore b/tools/testing/selftests/ir/.gitignore new file mode 100644 index 000000000000..070ea0c75fb8 --- /dev/null +++ b/tools/testing/selftests/ir/.gitignore @@ -0,0 +1 @@ +ir_loopback diff --git a/tools/testing/selftests/ir/Makefile b/tools/testing/selftests/ir/Makefile new file mode 100644 index 000000000000..f4ba8eb84b95 --- /dev/null +++ b/tools/testing/selftests/ir/Makefile @@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0 +TEST_PROGS := ir_loopback.sh +TEST_GEN_PROGS_EXTENDED := ir_loopback + +include ../lib.mk diff --git a/tools/testing/selftests/ir/ir_loopback.c b/tools/testing/selftests/ir/ir_loopback.c new file mode 100644 index 000000000000..858c19caf224 --- /dev/null +++ b/tools/testing/selftests/ir/ir_loopback.c @@ -0,0 +1,199 @@ +// SPDX-License-Identifier: GPL-2.0 +// test ir decoder +// +// Copyright (C) 2018 Sean Young <sean(a)mess.org> + +// When sending LIRC_MODE_SCANCODE, the IR will be encoded. rc-loopback +// will send this IR to the receiver side, where we try to read the decoded +// IR. Decoding happens in a separate kernel thread, so we will need to +// wait until that is scheduled, hence we use poll to check for read +// readiness. + +#include <linux/lirc.h> +#include <errno.h> +#include <stdio.h> +#include <stdlib.h> +#include <stdbool.h> +#include <string.h> +#include <unistd.h> +#include <poll.h> +#include <time.h> +#include <sys/types.h> +#include <sys/ioctl.h> +#include <dirent.h> +#include <sys/stat.h> +#include <fcntl.h> +#include "../kselftest.h" + +#define TEST_SCANCODES 10 +#define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0])) + +static const struct { + enum rc_proto proto; + const char *name; + unsigned int mask; + const char *decoder; +} protocols[] = { + { RC_PROTO_RC5, "rc-5", 0x1f7f, "rc-5" }, + { RC_PROTO_RC5X_20, "rc-5x-20", 0x1f7f3f, "rc-5" }, + { RC_PROTO_RC5_SZ, "rc-5-sz", 0x2fff, "rc-5-sz" }, + { RC_PROTO_JVC, "jvc", 0xffff, "jvc" }, + { RC_PROTO_SONY12, "sony-12", 0x1f007f, "sony" }, + { RC_PROTO_SONY15, "sony-15", 0xff007f, "sony" }, + { RC_PROTO_SONY20, "sony-20", 0x1fff7f, "sony" }, + { RC_PROTO_NEC, "nec", 0xffff, "nec" }, + { RC_PROTO_NECX, "nec-x", 0xffffff, "nec" }, + { RC_PROTO_NEC32, "nec-32", 0xffffffff, "nec" }, + { RC_PROTO_SANYO, "sanyo", 0x1fffff, "sanyo" }, + { RC_PROTO_RC6_0, "rc-6-0", 0xffff, "rc-6" }, + { RC_PROTO_RC6_6A_20, "rc-6-6a-20", 0xfffff, "rc-6" }, + { RC_PROTO_RC6_6A_24, "rc-6-6a-24", 0xffffff, "rc-6" }, + { RC_PROTO_RC6_6A_32, "rc-6-6a-32", 0xffffffff, "rc-6" }, + { RC_PROTO_RC6_MCE, "rc-6-mce", 0x00007fff, "rc-6" }, + { RC_PROTO_SHARP, "sharp", 0x1fff, "sharp" }, +}; + +int lirc_open(const char *rc) +{ + struct dirent *dent; + char buf[100]; + DIR *d; + int fd; + + snprintf(buf, sizeof(buf), "/sys/class/rc/%s", rc); + + d = opendir(buf); + if (!d) + ksft_exit_fail_msg("cannot open %s: %m\n", buf); + + while ((dent = readdir(d)) != NULL) { + if (!strncmp(dent->d_name, "lirc", 4)) { + snprintf(buf, sizeof(buf), "/dev/%s", dent->d_name); + break; + } + } + + if (!dent) + ksft_exit_fail_msg("cannot find lirc device for %s\n", rc); + + closedir(d); + + fd = open(buf, O_RDWR | O_NONBLOCK); + if (fd == -1) + ksft_exit_fail_msg("cannot open: %s: %m\n", buf); + + return fd; +} + +int main(int argc, char **argv) +{ + unsigned int mode; + char buf[100]; + int rlircfd, wlircfd, protocolfd, i, n; + + srand(time(NULL)); + + if (argc != 3) + ksft_exit_fail_msg("Usage: %s <write rcN> <read rcN>\n", + argv[0]); + + rlircfd = lirc_open(argv[2]); + mode = LIRC_MODE_SCANCODE; + if (ioctl(rlircfd, LIRC_SET_REC_MODE, &mode)) + ksft_exit_fail_msg("failed to set scancode rec mode %s: %m\n", + argv[2]); + + wlircfd = lirc_open(argv[1]); + if (ioctl(wlircfd, LIRC_SET_SEND_MODE, &mode)) + ksft_exit_fail_msg("failed to set scancode send mode %s: %m\n", + argv[1]); + + snprintf(buf, sizeof(buf), "/sys/class/rc/%s/protocols", argv[2]); + protocolfd = open(buf, O_WRONLY); + if (protocolfd == -1) + ksft_exit_fail_msg("failed to open %s: %m\n", buf); + + printf("Sending IR on %s and receiving IR on %s.\n", argv[1], argv[2]); + + for (i = 0; i < ARRAY_SIZE(protocols); i++) { + if (write(protocolfd, protocols[i].decoder, + strlen(protocols[i].decoder)) == -1) + ksft_exit_fail_msg("failed to set write decoder\n"); + + printf("Testing protocol %s for decoder %s (%d/%d)...\n", + protocols[i].name, protocols[i].decoder, + i + 1, (int)ARRAY_SIZE(protocols)); + + for (n = 0; n < TEST_SCANCODES; n++) { + unsigned int scancode = rand() & protocols[i].mask; + unsigned int rc_proto = protocols[i].proto; + + if (rc_proto == RC_PROTO_RC6_MCE) + scancode |= 0x800f0000; + + if (rc_proto == RC_PROTO_NECX && + (((scancode >> 16) ^ ~(scancode >> 8)) & 0xff) == 0) + continue; + + if (rc_proto == RC_PROTO_NEC32 && + (((scancode >> 8) ^ ~scancode) & 0xff) == 0) + continue; + + struct lirc_scancode lsc = { + .rc_proto = rc_proto, + .scancode = scancode + }; + + printf("Testing scancode:%x\n", scancode); + + while (write(wlircfd, &lsc, sizeof(lsc)) < 0) { + if (errno == EINTR) + continue; + + ksft_exit_fail_msg("failed to send ir: %m\n"); + } + + struct pollfd pfd = { .fd = rlircfd, .events = POLLIN }; + struct lirc_scancode lsc2; + + poll(&pfd, 1, 1000); + + bool decoded = true; + + while (read(rlircfd, &lsc2, sizeof(lsc2)) < 0) { + if (errno == EINTR) + continue; + + ksft_test_result_error("no scancode decoded: %m\n"); + decoded = false; + break; + } + + if (!decoded) + continue; + + if (lsc.rc_proto != lsc2.rc_proto) + ksft_test_result_error("decoded protocol is different: %d\n", + lsc2.rc_proto); + + else if (lsc.scancode != lsc2.scancode) + ksft_test_result_error("decoded scancode is different: %llx\n", + lsc2.scancode); + else + ksft_inc_pass_cnt(); + } + + printf("OK\n"); + } + + close(rlircfd); + close(wlircfd); + close(protocolfd); + + if (ksft_get_fail_cnt() > 0) + ksft_exit_fail(); + else + ksft_exit_pass(); + + return 0; +} diff --git a/tools/testing/selftests/ir/ir_loopback.sh b/tools/testing/selftests/ir/ir_loopback.sh new file mode 100755 index 000000000000..0a0b8dfa39be --- /dev/null +++ b/tools/testing/selftests/ir/ir_loopback.sh @@ -0,0 +1,20 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +# Kselftest framework requirement - SKIP code is 4. +ksft_skip=4 + +if ! /sbin/modprobe -q -n rc-loopback; then + echo "ir_loopback: module rc-loopback is not found [SKIP]" + exit $ksft_skip +fi + +/sbin/modprobe rc-loopback +if [ $? -ne 0 ]; then + exit +fi + +RCDEV=$(grep -l DRV_NAME=rc-loopback /sys/class/rc/rc*/uevent | grep -o 'rc[0-9]\+') + +./ir_loopback $RCDEV $RCDEV +exit -- 2.17.2

6 years, 2 months

2
2
0 0

[RFC PATCH for 4.21 16/16] cpu-opv/selftests: Provide Makefile, scripts, gitignore

by Mathieu Desnoyers

A run_param_test.sh script runs many variants of the parametrizable tests. Wire up the cpu-opv Makefile, add directory entry into MAINTAINERS file. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers(a)efficios.com> Cc: Thomas Gleixner <tglx(a)linutronix.de> Cc: Joel Fernandes <joelaf(a)google.com> Cc: Peter Zijlstra <peterz(a)infradead.org> Cc: Catalin Marinas <catalin.marinas(a)arm.com> Cc: Dave Watson <davejwatson(a)fb.com> Cc: Will Deacon <will.deacon(a)arm.com> Cc: Shuah Khan <shuah(a)kernel.org> Cc: Andi Kleen <andi(a)firstfloor.org> Cc: linux-kselftest(a)vger.kernel.org Cc: "H . Peter Anvin" <hpa(a)zytor.com> Cc: Chris Lameter <cl(a)linux.com> Cc: Russell King <linux(a)arm.linux.org.uk> Cc: Michael Kerrisk <mtk.manpages(a)gmail.com> Cc: "Paul E . McKenney" <paulmck(a)linux.vnet.ibm.com> Cc: Paul Turner <pjt(a)google.com> Cc: Boqun Feng <boqun.feng(a)gmail.com> Cc: Josh Triplett <josh(a)joshtriplett.org> Cc: Steven Rostedt <rostedt(a)goodmis.org> Cc: Ben Maurer <bmaurer(a)fb.com> Cc: linux-api(a)vger.kernel.org Cc: Andy Lutomirski <luto(a)amacapital.net> Cc: Andrew Morton <akpm(a)linux-foundation.org> Cc: Linus Torvalds <torvalds(a)linux-foundation.org> --- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/cpu-opv/.gitignore | 6 + tools/testing/selftests/cpu-opv/Makefile | 39 +++++++ tools/testing/selftests/cpu-opv/run_param_test.sh | 134 ++++++++++++++++++++++ 5 files changed, 181 insertions(+) create mode 100644 tools/testing/selftests/cpu-opv/.gitignore create mode 100644 tools/testing/selftests/cpu-opv/Makefile create mode 100755 tools/testing/selftests/cpu-opv/run_param_test.sh diff --git a/MAINTAINERS b/MAINTAINERS index de59c7c12c8f..ad9addbc5510 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3878,6 +3878,7 @@ L: linux-kernel(a)vger.kernel.org S: Supported F: kernel/cpu_opv.c F: include/uapi/linux/cpu_opv.h +F: tools/testing/selftests/cpu-opv/ CRAMFS FILESYSTEM M: Nicolas Pitre <nico(a)linaro.org> diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index f1fe492c8e17..42bcc6e5fc5d 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -6,6 +6,7 @@ TARGETS += capabilities TARGETS += cgroup TARGETS += cpufreq TARGETS += cpu-hotplug +TARGETS += cpu-opv TARGETS += efivarfs TARGETS += exec TARGETS += filesystems diff --git a/tools/testing/selftests/cpu-opv/.gitignore b/tools/testing/selftests/cpu-opv/.gitignore new file mode 100644 index 000000000000..8c7bb1f8be79 --- /dev/null +++ b/tools/testing/selftests/cpu-opv/.gitignore @@ -0,0 +1,6 @@ +basic_cpu_opv_test +basic_percpu_ops_test +param_test +param_test_benchmark +param_test_compare_twice +param_test_skip_fastpath diff --git a/tools/testing/selftests/cpu-opv/Makefile b/tools/testing/selftests/cpu-opv/Makefile new file mode 100644 index 000000000000..46f49bf30bae --- /dev/null +++ b/tools/testing/selftests/cpu-opv/Makefile @@ -0,0 +1,39 @@ +# SPDX-License-Identifier: GPL-2.0+ OR MIT +CFLAGS += -O2 -Wall -g -I./ -I../rseq/ -I../../../../usr/include/ -L./ -Wl,-rpath=./ +LDLIBS += -lpthread + +# Own dependencies because we only want to build against 1st prerequisite, but +# still track changes to header files and depend on shared object. +OVERRIDE_TARGETS = 1 + +TEST_GEN_PROGS = basic_cpu_opv_test basic_percpu_ops_test \ + param_test param_test_skip_fastpath \ + param_test_benchmark param_test_compare_twice + +TEST_GEN_PROGS_EXTENDED = librseq.so libcpu-op.so + +TEST_PROGS = run_param_test.sh + +include ../lib.mk + +$(OUTPUT)/libcpu-op.so: cpu-op.c cpu-op.h + $(CC) $(CFLAGS) -shared -fPIC $< $(LDLIBS) -o $@ + +$(OUTPUT)/librseq.so: ../rseq/rseq.c ../rseq/rseq.h ../rseq/rseq-*.h + $(CC) $(CFLAGS) -shared -fPIC $< $(LDLIBS) -o $@ + +$(OUTPUT)/%: %.c $(TEST_GEN_PROGS_EXTENDED) ../rseq/rseq.h ../rseq/rseq-*.h cpu-op.h percpu-op.h + $(CC) $(CFLAGS) $< $(LDLIBS) -lrseq -lcpu-op -o $@ + +$(OUTPUT)/param_test_skip_fastpath: param_test.c $(TEST_GEN_PROGS_EXTENDED) \ + ../rseq/rseq.h ../rseq/rseq-*.h cpu-op.h percpu-op.h + $(CC) $(CFLAGS) -DRSEQ_SKIP_FASTPATH $< $(LDLIBS) -lrseq -lcpu-op -o $@ + +$(OUTPUT)/param_test_benchmark: param_test.c $(TEST_GEN_PROGS_EXTENDED) \ + ../rseq/rseq.h ../rseq/rseq-*.h cpu-op.h percpu-op.h + $(CC) $(CFLAGS) -DBENCHMARK $< $(LDLIBS) -lrseq -lcpu-op -o $@ + +$(OUTPUT)/param_test_compare_twice: param_test.c $(TEST_GEN_PROGS_EXTENDED) \ + ../rseq/rseq.h ../rseq/rseq-*.h cpu-op.h percpu-op.h + $(CC) $(CFLAGS) -DRSEQ_COMPARE_TWICE $< $(LDLIBS) -lrseq -lcpu-op -o $@ + diff --git a/tools/testing/selftests/cpu-opv/run_param_test.sh b/tools/testing/selftests/cpu-opv/run_param_test.sh new file mode 100755 index 000000000000..066d2479893c --- /dev/null +++ b/tools/testing/selftests/cpu-opv/run_param_test.sh @@ -0,0 +1,134 @@ +#!/bin/bash + +NR_CPUS=`grep '^processor' /proc/cpuinfo | wc -l` + +EXTRA_ARGS=${@} + +OLDIFS="$IFS" +IFS=$'\n' +TEST_LIST=( + "-T s" + "-T l" + "-T b" + "-T b -M" + "-T m" + "-T m -M" + "-T i" +) + +TEST_NAME=( + "spinlock" + "list" + "buffer" + "buffer with barrier" + "memcpy" + "memcpy with barrier" + "increment" +) +IFS="$OLDIFS" + +REPS=1000 +SLOW_REPS=100 +NR_THREADS=$((6*${NR_CPUS})) + +function do_tests() +{ + local i=0 + while [ "$i" -lt "${#TEST_LIST[@]}" ]; do + echo "Running test ${TEST_NAME[$i]}" + ./param_test ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS} || exit 1 + echo "Running skip fast-path test ${TEST_NAME[$i]}" + ./param_test_skip_fastpath ${TEST_LIST[$i]} -r ${SLOW_REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS} || exit 1 + echo "Running compare-twice test ${TEST_NAME[$i]}" + ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS} || exit 1 + let "i++" + done +} + +echo "Default parameters" +do_tests + +echo "Loop injection: 10000 loops" + +OLDIFS="$IFS" +IFS=$'\n' +INJECT_LIST=( + "1" + "2" + "3" + "4" + "5" + "6" + "7" + "8" + "9" +) +IFS="$OLDIFS" + +NR_LOOPS=10000 + +i=0 +while [ "$i" -lt "${#INJECT_LIST[@]}" ]; do + echo "Injecting at <${INJECT_LIST[$i]}>" + do_tests -${INJECT_LIST[i]} ${NR_LOOPS} + let "i++" +done +NR_LOOPS= + +function inject_blocking() +{ + OLDIFS="$IFS" + IFS=$'\n' + INJECT_LIST=( + "7" + "8" + "9" + ) + IFS="$OLDIFS" + + NR_LOOPS=-1 + + i=0 + while [ "$i" -lt "${#INJECT_LIST[@]}" ]; do + echo "Injecting at <${INJECT_LIST[$i]}>" + do_tests -${INJECT_LIST[i]} -1 ${@} + let "i++" + done + NR_LOOPS= +} + +echo "Yield injection (25%)" +inject_blocking -m 4 -y + +echo "Yield injection (50%)" +inject_blocking -m 2 -y + +echo "Yield injection (100%)" +inject_blocking -m 1 -y + +echo "Kill injection (25%)" +inject_blocking -m 4 -k + +echo "Kill injection (50%)" +inject_blocking -m 2 -k + +echo "Kill injection (100%)" +inject_blocking -m 1 -k + +echo "Sleep injection (1ms, 25%)" +inject_blocking -m 4 -s 1 + +echo "Sleep injection (1ms, 50%)" +inject_blocking -m 2 -s 1 + +echo "Sleep injection (1ms, 100%)" +inject_blocking -m 1 -s 1 + +echo "Disable rseq for 25% threads" +do_tests -D 4 + +echo "Disable rseq for 50% threads" +do_tests -D 2 + +echo "Disable rseq" +do_tests -d -- 2.11.0

6 years, 2 months

1
0
0 0

[RFC PATCH for 4.21 15/16] cpu-opv/selftests: Provide parametrized tests

by Mathieu Desnoyers

"param_test" is a parametrizable percpu operations test using both restartable sequences and cpu_opv. See the "--help" output for usage. "param_test_benchmark" is the same as "param_test", but it removes testing book-keeping code to allow accurate benchmarks. "param_test_compare_twice" is the same as "param_test", but it performs each comparison within rseq critical section twice, thus validating invariants. If any of the second comparisons fails, an error message is printed and the test aborts. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers(a)efficios.com> Cc: Thomas Gleixner <tglx(a)linutronix.de> Cc: Joel Fernandes <joelaf(a)google.com> Cc: Peter Zijlstra <peterz(a)infradead.org> Cc: Catalin Marinas <catalin.marinas(a)arm.com> Cc: Dave Watson <davejwatson(a)fb.com> Cc: Will Deacon <will.deacon(a)arm.com> Cc: Shuah Khan <shuah(a)kernel.org> Cc: Andi Kleen <andi(a)firstfloor.org> Cc: linux-kselftest(a)vger.kernel.org Cc: "H . Peter Anvin" <hpa(a)zytor.com> Cc: Chris Lameter <cl(a)linux.com> Cc: Russell King <linux(a)arm.linux.org.uk> Cc: Michael Kerrisk <mtk.manpages(a)gmail.com> Cc: "Paul E . McKenney" <paulmck(a)linux.vnet.ibm.com> Cc: Paul Turner <pjt(a)google.com> Cc: Boqun Feng <boqun.feng(a)gmail.com> Cc: Josh Triplett <josh(a)joshtriplett.org> Cc: Steven Rostedt <rostedt(a)goodmis.org> Cc: Ben Maurer <bmaurer(a)fb.com> Cc: linux-api(a)vger.kernel.org Cc: Andy Lutomirski <luto(a)amacapital.net> Cc: Andrew Morton <akpm(a)linux-foundation.org> Cc: Linus Torvalds <torvalds(a)linux-foundation.org> --- tools/testing/selftests/cpu-opv/param_test.c | 1187 ++++++++++++++++++++++++++ 1 file changed, 1187 insertions(+) create mode 100644 tools/testing/selftests/cpu-opv/param_test.c diff --git a/tools/testing/selftests/cpu-opv/param_test.c b/tools/testing/selftests/cpu-opv/param_test.c new file mode 100644 index 000000000000..c62e75f07385 --- /dev/null +++ b/tools/testing/selftests/cpu-opv/param_test.c @@ -0,0 +1,1187 @@ +// SPDX-License-Identifier: LGPL-2.1 +#define _GNU_SOURCE +#include <assert.h> +#include <pthread.h> +#include <sched.h> +#include <stdint.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <syscall.h> +#include <unistd.h> +#include <poll.h> +#include <sys/types.h> +#include <signal.h> +#include <errno.h> +#include <stddef.h> + +static inline pid_t gettid(void) +{ + return syscall(__NR_gettid); +} + +#define NR_INJECT 9 +static int loop_cnt[NR_INJECT + 1]; + +static int loop_cnt_1 asm("asm_loop_cnt_1") __attribute__((used)); +static int loop_cnt_2 asm("asm_loop_cnt_2") __attribute__((used)); +static int loop_cnt_3 asm("asm_loop_cnt_3") __attribute__((used)); +static int loop_cnt_4 asm("asm_loop_cnt_4") __attribute__((used)); +static int loop_cnt_5 asm("asm_loop_cnt_5") __attribute__((used)); +static int loop_cnt_6 asm("asm_loop_cnt_6") __attribute__((used)); + +static int opt_modulo, verbose; + +static int opt_yield, opt_signal, opt_sleep, + opt_disable_rseq, opt_threads = 200, + opt_disable_mod = 0, opt_test = 's', opt_mb = 0; + +#ifndef RSEQ_SKIP_FASTPATH +static long long opt_reps = 5000; +#else +static long long opt_reps = 100; +#endif + +static __thread __attribute__((tls_model("initial-exec"))) +unsigned int signals_delivered; + +#ifndef BENCHMARK + +static __thread __attribute__((tls_model("initial-exec"), unused)) +unsigned int yield_mod_cnt, nr_abort; + +#define printf_verbose(fmt, ...) \ + do { \ + if (verbose) \ + printf(fmt, ## __VA_ARGS__); \ + } while (0) + +#ifdef __i386__ + +#define INJECT_ASM_REG "eax" + +#define RSEQ_INJECT_CLOBBER \ + , INJECT_ASM_REG + +#define RSEQ_INJECT_ASM(n) \ + "mov asm_loop_cnt_" #n ", %%" INJECT_ASM_REG "\n\t" \ + "test %%" INJECT_ASM_REG ",%%" INJECT_ASM_REG "\n\t" \ + "jz 333f\n\t" \ + "222:\n\t" \ + "dec %%" INJECT_ASM_REG "\n\t" \ + "jnz 222b\n\t" \ + "333:\n\t" + +#elif defined(__x86_64__) + +#define INJECT_ASM_REG_P "rax" +#define INJECT_ASM_REG "eax" + +#define RSEQ_INJECT_CLOBBER \ + , INJECT_ASM_REG_P \ + , INJECT_ASM_REG + +#define RSEQ_INJECT_ASM(n) \ + "lea asm_loop_cnt_" #n "(%%rip), %%" INJECT_ASM_REG_P "\n\t" \ + "mov (%%" INJECT_ASM_REG_P "), %%" INJECT_ASM_REG "\n\t" \ + "test %%" INJECT_ASM_REG ",%%" INJECT_ASM_REG "\n\t" \ + "jz 333f\n\t" \ + "222:\n\t" \ + "dec %%" INJECT_ASM_REG "\n\t" \ + "jnz 222b\n\t" \ + "333:\n\t" + +#elif defined(__ARMEL__) + +#define RSEQ_INJECT_INPUT \ + , [loop_cnt_1]"m"(loop_cnt[1]) \ + , [loop_cnt_2]"m"(loop_cnt[2]) \ + , [loop_cnt_3]"m"(loop_cnt[3]) \ + , [loop_cnt_4]"m"(loop_cnt[4]) \ + , [loop_cnt_5]"m"(loop_cnt[5]) \ + , [loop_cnt_6]"m"(loop_cnt[6]) + +#define INJECT_ASM_REG "r4" + +#define RSEQ_INJECT_CLOBBER \ + , INJECT_ASM_REG + +#define RSEQ_INJECT_ASM(n) \ + "ldr " INJECT_ASM_REG ", %[loop_cnt_" #n "]\n\t" \ + "cmp " INJECT_ASM_REG ", #0\n\t" \ + "beq 333f\n\t" \ + "222:\n\t" \ + "subs " INJECT_ASM_REG ", #1\n\t" \ + "bne 222b\n\t" \ + "333:\n\t" + +#elif __PPC__ + +#define RSEQ_INJECT_INPUT \ + , [loop_cnt_1]"m"(loop_cnt[1]) \ + , [loop_cnt_2]"m"(loop_cnt[2]) \ + , [loop_cnt_3]"m"(loop_cnt[3]) \ + , [loop_cnt_4]"m"(loop_cnt[4]) \ + , [loop_cnt_5]"m"(loop_cnt[5]) \ + , [loop_cnt_6]"m"(loop_cnt[6]) + +#define INJECT_ASM_REG "r18" + +#define RSEQ_INJECT_CLOBBER \ + , INJECT_ASM_REG + +#define RSEQ_INJECT_ASM(n) \ + "lwz %%" INJECT_ASM_REG ", %[loop_cnt_" #n "]\n\t" \ + "cmpwi %%" INJECT_ASM_REG ", 0\n\t" \ + "beq 333f\n\t" \ + "222:\n\t" \ + "subic. %%" INJECT_ASM_REG ", %%" INJECT_ASM_REG ", 1\n\t" \ + "bne 222b\n\t" \ + "333:\n\t" +#else +#error unsupported target +#endif + +#define RSEQ_INJECT_FAILED \ + nr_abort++; + +#define RSEQ_INJECT_C(n) \ +{ \ + int loc_i, loc_nr_loops = loop_cnt[n]; \ + \ + for (loc_i = 0; loc_i < loc_nr_loops; loc_i++) { \ + rseq_barrier(); \ + } \ + if (loc_nr_loops == -1 && opt_modulo) { \ + if (yield_mod_cnt == opt_modulo - 1) { \ + if (opt_sleep > 0) \ + poll(NULL, 0, opt_sleep); \ + if (opt_yield) \ + sched_yield(); \ + if (opt_signal) \ + raise(SIGUSR1); \ + yield_mod_cnt = 0; \ + } else { \ + yield_mod_cnt++; \ + } \ + } \ +} + +#else + +#define printf_verbose(fmt, ...) + +#endif /* BENCHMARK */ + +#include "percpu-op.h" + +struct percpu_lock_entry { + intptr_t v; +} __attribute__((aligned(128))); + +struct percpu_lock { + struct percpu_lock_entry c[CPU_SETSIZE]; +}; + +struct test_data_entry { + intptr_t count; +} __attribute__((aligned(128))); + +struct spinlock_test_data { + struct percpu_lock lock; + struct test_data_entry c[CPU_SETSIZE]; +}; + +struct spinlock_thread_test_data { + struct spinlock_test_data *data; + long long reps; + int reg; +}; + +struct inc_test_data { + struct test_data_entry c[CPU_SETSIZE]; +}; + +struct inc_thread_test_data { + struct inc_test_data *data; + long long reps; + int reg; +}; + +struct percpu_list_node { + intptr_t data; + struct percpu_list_node *next; +}; + +struct percpu_list_entry { + struct percpu_list_node *head; +} __attribute__((aligned(128))); + +struct percpu_list { + struct percpu_list_entry c[CPU_SETSIZE]; +}; + +#define BUFFER_ITEM_PER_CPU 100 + +struct percpu_buffer_node { + intptr_t data; +}; + +struct percpu_buffer_entry { + intptr_t offset; + intptr_t buflen; + struct percpu_buffer_node **array; +} __attribute__((aligned(128))); + +struct percpu_buffer { + struct percpu_buffer_entry c[CPU_SETSIZE]; +}; + +#define MEMCPY_BUFFER_ITEM_PER_CPU 100 + +struct percpu_memcpy_buffer_node { + intptr_t data1; + uint64_t data2; +}; + +struct percpu_memcpy_buffer_entry { + intptr_t offset; + intptr_t buflen; + struct percpu_memcpy_buffer_node *array; +} __attribute__((aligned(128))); + +struct percpu_memcpy_buffer { + struct percpu_memcpy_buffer_entry c[CPU_SETSIZE]; +}; + +/* A simple percpu spinlock. */ +static void rseq_percpu_lock(struct percpu_lock *lock, int cpu) +{ + for (;;) { + int ret; + + ret = percpu_cmpeqv_storev(&lock->c[cpu].v, + 0, 1, cpu); + if (rseq_likely(!ret)) + break; + if (rseq_unlikely(ret < 0)) { + perror("cpu_opv"); + abort(); + } + /* Retry if comparison fails. */ + } + /* + * Acquire semantic when taking lock after control dependency. + * Matches rseq_smp_store_release(). + */ + rseq_smp_acquire__after_ctrl_dep(); +} + +static void rseq_percpu_unlock(struct percpu_lock *lock, int cpu) +{ + assert(lock->c[cpu].v == 1); + /* + * Release lock, with release semantic. Matches + * rseq_smp_acquire__after_ctrl_dep(). + */ + rseq_smp_store_release(&lock->c[cpu].v, 0); +} + +void *test_percpu_spinlock_thread(void *arg) +{ + struct spinlock_thread_test_data *thread_data = arg; + struct spinlock_test_data *data = thread_data->data; + long long i, reps; + + if (!opt_disable_rseq && thread_data->reg && + rseq_register_current_thread()) + abort(); + reps = thread_data->reps; + for (i = 0; i < reps; i++) { + int cpu = rseq_cpu_start(); + + rseq_percpu_lock(&data->lock, cpu); + data->c[cpu].count++; + rseq_percpu_unlock(&data->lock, cpu); +#ifndef BENCHMARK + if (i != 0 && !(i % (reps / 10))) + printf_verbose("tid %d: count %lld\n", (int) gettid(), i); +#endif + } + printf_verbose("tid %d: number of rseq abort: %d, signals delivered: %u\n", + (int) gettid(), nr_abort, signals_delivered); + if (!opt_disable_rseq && thread_data->reg && + rseq_unregister_current_thread()) + abort(); + return NULL; +} + +/* + * A simple test which implements a sharded counter using a per-cpu + * lock. Obviously real applications might prefer to simply use a + * per-cpu increment; however, this is reasonable for a test and the + * lock can be extended to synchronize more complicated operations. + */ +void test_percpu_spinlock(void) +{ + const int num_threads = opt_threads; + int i, ret; + uint64_t sum; + pthread_t test_threads[num_threads]; + struct spinlock_test_data data; + struct spinlock_thread_test_data thread_data[num_threads]; + + memset(&data, 0, sizeof(data)); + for (i = 0; i < num_threads; i++) { + thread_data[i].reps = opt_reps; + if (opt_disable_mod <= 0 || (i % opt_disable_mod)) + thread_data[i].reg = 1; + else + thread_data[i].reg = 0; + thread_data[i].data = &data; + ret = pthread_create(&test_threads[i], NULL, + test_percpu_spinlock_thread, + &thread_data[i]); + if (ret) { + errno = ret; + perror("pthread_create"); + abort(); + } + } + + for (i = 0; i < num_threads; i++) { + ret = pthread_join(test_threads[i], NULL); + if (ret) { + errno = ret; + perror("pthread_join"); + abort(); + } + } + + sum = 0; + for (i = 0; i < CPU_SETSIZE; i++) + sum += data.c[i].count; + + assert(sum == (uint64_t)opt_reps * num_threads); +} + +void *test_percpu_inc_thread(void *arg) +{ + struct inc_thread_test_data *thread_data = arg; + struct inc_test_data *data = thread_data->data; + long long i, reps; + + if (!opt_disable_rseq && thread_data->reg && + rseq_register_current_thread()) + abort(); + reps = thread_data->reps; + for (i = 0; i < reps; i++) { + int cpu, ret; + + cpu = rseq_cpu_start(); + ret = percpu_addv(&data->c[cpu].count, 1, cpu); + if (rseq_unlikely(ret)) { + perror("cpu_opv"); + abort(); + } +#ifndef BENCHMARK + if (i != 0 && !(i % (reps / 10))) + printf_verbose("tid %d: count %lld\n", (int) gettid(), i); +#endif + } + printf_verbose("tid %d: number of rseq abort: %d, signals delivered: %u\n", + (int) gettid(), nr_abort, signals_delivered); + if (!opt_disable_rseq && thread_data->reg && + rseq_unregister_current_thread()) + abort(); + return NULL; +} + +void test_percpu_inc(void) +{ + const int num_threads = opt_threads; + int i, ret; + uint64_t sum; + pthread_t test_threads[num_threads]; + struct inc_test_data data; + struct inc_thread_test_data thread_data[num_threads]; + + memset(&data, 0, sizeof(data)); + for (i = 0; i < num_threads; i++) { + thread_data[i].reps = opt_reps; + if (opt_disable_mod <= 0 || (i % opt_disable_mod)) + thread_data[i].reg = 1; + else + thread_data[i].reg = 0; + thread_data[i].data = &data; + ret = pthread_create(&test_threads[i], NULL, + test_percpu_inc_thread, + &thread_data[i]); + if (ret) { + errno = ret; + perror("pthread_create"); + abort(); + } + } + + for (i = 0; i < num_threads; i++) { + ret = pthread_join(test_threads[i], NULL); + if (ret) { + errno = ret; + perror("pthread_join"); + abort(); + } + } + + sum = 0; + for (i = 0; i < CPU_SETSIZE; i++) + sum += data.c[i].count; + + assert(sum == (uint64_t)opt_reps * num_threads); +} + +void percpu_list_push(struct percpu_list *list, + struct percpu_list_node *node, + int cpu) +{ + for (;;) { + intptr_t *targetptr, newval, expect; + int ret; + + /* Load list->c[cpu].head with single-copy atomicity. */ + expect = (intptr_t)RSEQ_READ_ONCE(list->c[cpu].head); + newval = (intptr_t)node; + targetptr = (intptr_t *)&list->c[cpu].head; + node->next = (struct percpu_list_node *)expect; + ret = percpu_cmpeqv_storev(targetptr, expect, newval, cpu); + if (rseq_likely(!ret)) + break; + if (rseq_unlikely(ret < 0)) { + perror("cpu_opv"); + abort(); + } + /* Retry if comparison fails. */ + } +} + +/* + * Unlike a traditional lock-less linked list; the availability of a + * rseq primitive allows us to implement pop without concerns over + * ABA-type races. + */ +struct percpu_list_node *percpu_list_pop(struct percpu_list *list, + int cpu) +{ + struct percpu_list_node *head; + intptr_t *targetptr, expectnot, *load; + off_t offset; + int ret; + + targetptr = (intptr_t *)&list->c[cpu].head; + expectnot = (intptr_t)NULL; + offset = offsetof(struct percpu_list_node, next); + load = (intptr_t *)&head; + ret = percpu_cmpnev_storeoffp_load(targetptr, expectnot, + offset, load, cpu); + if (rseq_unlikely(ret < 0)) { + perror("cpu_opv"); + abort(); + } + if (ret > 0) + return NULL; + return head; +} + +void *test_percpu_list_thread(void *arg) +{ + long long i, reps; + struct percpu_list *list = (struct percpu_list *)arg; + + if (!opt_disable_rseq && rseq_register_current_thread()) + abort(); + + reps = opt_reps; + for (i = 0; i < reps; i++) { + struct percpu_list_node *node; + + node = percpu_list_pop(list, rseq_cpu_start()); + if (opt_yield) + sched_yield(); /* encourage shuffling */ + if (node) + percpu_list_push(list, node, rseq_cpu_start()); + } + + printf_verbose("tid %d: number of rseq abort: %d, signals delivered: %u\n", + (int) gettid(), nr_abort, signals_delivered); + if (!opt_disable_rseq && rseq_unregister_current_thread()) + abort(); + + return NULL; +} + +/* Simultaneous modification to a per-cpu linked list from many threads. */ +void test_percpu_list(void) +{ + const int num_threads = opt_threads; + int i, j, ret; + uint64_t sum = 0, expected_sum = 0; + struct percpu_list list; + pthread_t test_threads[num_threads]; + cpu_set_t allowed_cpus; + + memset(&list, 0, sizeof(list)); + + /* Generate list entries for every usable cpu. */ + sched_getaffinity(0, sizeof(allowed_cpus), &allowed_cpus); + for (i = 0; i < CPU_SETSIZE; i++) { + if (!CPU_ISSET(i, &allowed_cpus)) + continue; + for (j = 1; j <= 100; j++) { + struct percpu_list_node *node; + + expected_sum += j; + + node = malloc(sizeof(*node)); + assert(node); + node->data = j; + node->next = list.c[i].head; + list.c[i].head = node; + } + } + + for (i = 0; i < num_threads; i++) { + ret = pthread_create(&test_threads[i], NULL, + test_percpu_list_thread, &list); + if (ret) { + errno = ret; + perror("pthread_create"); + abort(); + } + } + + for (i = 0; i < num_threads; i++) { + ret = pthread_join(test_threads[i], NULL); + if (ret) { + errno = ret; + perror("pthread_join"); + abort(); + } + } + + for (i = 0; i < CPU_SETSIZE; i++) { + struct percpu_list_node *node; + + if (!CPU_ISSET(i, &allowed_cpus)) + continue; + + while ((node = percpu_list_pop(&list, i))) { + sum += node->data; + free(node); + } + } + + /* + * All entries should now be accounted for (unless some external + * actor is interfering with our allowed affinity while this + * test is running). + */ + assert(sum == expected_sum); +} + +bool percpu_buffer_push(struct percpu_buffer *buffer, + struct percpu_buffer_node *node, + int cpu) +{ + for (;;) { + intptr_t *targetptr_spec, newval_spec; + intptr_t *targetptr_final, newval_final; + intptr_t offset; + int ret; + + offset = RSEQ_READ_ONCE(buffer->c[cpu].offset); + if (offset == buffer->c[cpu].buflen) + return false; + newval_spec = (intptr_t)node; + targetptr_spec = (intptr_t *)&buffer->c[cpu].array[offset]; + newval_final = offset + 1; + targetptr_final = &buffer->c[cpu].offset; + if (opt_mb) + ret = percpu_cmpeqv_storev_storev_release( + targetptr_final, offset, targetptr_spec, + newval_spec, newval_final, cpu); + else + ret = percpu_cmpeqv_storev_storev(targetptr_final, + offset, targetptr_spec, newval_spec, + newval_final, cpu); + if (rseq_likely(!ret)) + break; + if (rseq_unlikely(ret < 0)) { + perror("cpu_opv"); + abort(); + } + /* Retry if comparison fails. */ + } + return true; +} + +struct percpu_buffer_node *percpu_buffer_pop(struct percpu_buffer *buffer, + int cpu) +{ + struct percpu_buffer_node *head; + + for (;;) { + intptr_t *targetptr, newval; + intptr_t offset; + int ret; + + /* Load offset with single-copy atomicity. */ + offset = RSEQ_READ_ONCE(buffer->c[cpu].offset); + if (offset == 0) + return NULL; + head = RSEQ_READ_ONCE(buffer->c[cpu].array[offset - 1]); + newval = offset - 1; + targetptr = (intptr_t *)&buffer->c[cpu].offset; + ret = percpu_cmpeqv_cmpeqv_storev(targetptr, offset, + (intptr_t *)&buffer->c[cpu].array[offset - 1], + (intptr_t)head, newval, cpu); + if (rseq_likely(!ret)) + break; + if (rseq_unlikely(ret < 0)) { + perror("cpu_opv"); + abort(); + } + /* Retry if comparison fails. */ + } + return head; +} + +void *test_percpu_buffer_thread(void *arg) +{ + long long i, reps; + struct percpu_buffer *buffer = (struct percpu_buffer *)arg; + + if (!opt_disable_rseq && rseq_register_current_thread()) + abort(); + + reps = opt_reps; + for (i = 0; i < reps; i++) { + struct percpu_buffer_node *node; + + node = percpu_buffer_pop(buffer, rseq_cpu_start()); + if (opt_yield) + sched_yield(); /* encourage shuffling */ + if (node) { + if (!percpu_buffer_push(buffer, node, + rseq_cpu_start())) { + /* Should increase buffer size. */ + abort(); + } + } + } + + printf_verbose("tid %d: number of rseq abort: %d, signals delivered: %u\n", + (int) gettid(), nr_abort, signals_delivered); + if (!opt_disable_rseq && rseq_unregister_current_thread()) + abort(); + + return NULL; +} + +/* Simultaneous modification to a per-cpu buffer from many threads. */ +void test_percpu_buffer(void) +{ + const int num_threads = opt_threads; + int i, j, ret; + uint64_t sum = 0, expected_sum = 0; + struct percpu_buffer buffer; + pthread_t test_threads[num_threads]; + cpu_set_t allowed_cpus; + + memset(&buffer, 0, sizeof(buffer)); + + /* Generate list entries for every usable cpu. */ + sched_getaffinity(0, sizeof(allowed_cpus), &allowed_cpus); + for (i = 0; i < CPU_SETSIZE; i++) { + if (!CPU_ISSET(i, &allowed_cpus)) + continue; + /* Worse-case is every item in same CPU. */ + buffer.c[i].array = + malloc(sizeof(*buffer.c[i].array) * CPU_SETSIZE * + BUFFER_ITEM_PER_CPU); + assert(buffer.c[i].array); + buffer.c[i].buflen = CPU_SETSIZE * BUFFER_ITEM_PER_CPU; + for (j = 1; j <= BUFFER_ITEM_PER_CPU; j++) { + struct percpu_buffer_node *node; + + expected_sum += j; + + /* + * We could theoretically put the word-sized + * "data" directly in the buffer. However, we + * want to model objects that would not fit + * within a single word, so allocate an object + * for each node. + */ + node = malloc(sizeof(*node)); + assert(node); + node->data = j; + buffer.c[i].array[j - 1] = node; + buffer.c[i].offset++; + } + } + + for (i = 0; i < num_threads; i++) { + ret = pthread_create(&test_threads[i], NULL, + test_percpu_buffer_thread, &buffer); + if (ret) { + errno = ret; + perror("pthread_create"); + abort(); + } + } + + for (i = 0; i < num_threads; i++) { + ret = pthread_join(test_threads[i], NULL); + if (ret) { + errno = ret; + perror("pthread_join"); + abort(); + } + } + + for (i = 0; i < CPU_SETSIZE; i++) { + struct percpu_buffer_node *node; + + if (!CPU_ISSET(i, &allowed_cpus)) + continue; + + while ((node = percpu_buffer_pop(&buffer, i))) { + sum += node->data; + free(node); + } + free(buffer.c[i].array); + } + + /* + * All entries should now be accounted for (unless some external + * actor is interfering with our allowed affinity while this + * test is running). + */ + assert(sum == expected_sum); +} + +bool percpu_memcpy_buffer_push(struct percpu_memcpy_buffer *buffer, + struct percpu_memcpy_buffer_node item, int cpu) +{ + for (;;) { + intptr_t *targetptr_final, newval_final, offset; + char *destptr, *srcptr; + size_t copylen; + int ret; + + /* Load offset with single-copy atomicity. */ + offset = RSEQ_READ_ONCE(buffer->c[cpu].offset); + if (offset == buffer->c[cpu].buflen) + return false; + destptr = (char *)&buffer->c[cpu].array[offset]; + srcptr = (char *)&item; + /* copylen must be <= 4kB. */ + copylen = sizeof(item); + newval_final = offset + 1; + targetptr_final = &buffer->c[cpu].offset; + if (opt_mb) + ret = percpu_cmpeqv_memcpy_storev_release( + targetptr_final, offset, + destptr, srcptr, copylen, + newval_final, cpu); + else + ret = percpu_cmpeqv_memcpy_storev(targetptr_final, + offset, destptr, srcptr, copylen, + newval_final, cpu); + if (rseq_likely(!ret)) + break; + if (rseq_unlikely(ret < 0)) { + perror("cpu_opv"); + abort(); + } + /* Retry if comparison fails. */ + } + return true; +} + +bool percpu_memcpy_buffer_pop(struct percpu_memcpy_buffer *buffer, + struct percpu_memcpy_buffer_node *item, int cpu) +{ + for (;;) { + intptr_t *targetptr_final, newval_final, offset; + char *destptr, *srcptr; + size_t copylen; + int ret; + + /* Load offset with single-copy atomicity. */ + offset = RSEQ_READ_ONCE(buffer->c[cpu].offset); + if (offset == 0) + return false; + destptr = (char *)item; + srcptr = (char *)&buffer->c[cpu].array[offset - 1]; + /* copylen must be <= 4kB. */ + copylen = sizeof(*item); + newval_final = offset - 1; + targetptr_final = &buffer->c[cpu].offset; + ret = percpu_cmpeqv_memcpy_storev(targetptr_final, + offset, destptr, srcptr, copylen, + newval_final, cpu); + if (rseq_likely(!ret)) + break; + if (rseq_unlikely(ret < 0)) { + perror("cpu_opv"); + abort(); + } + /* Retry if comparison fails. */ + } + return true; +} + +void *test_percpu_memcpy_buffer_thread(void *arg) +{ + long long i, reps; + struct percpu_memcpy_buffer *buffer = (struct percpu_memcpy_buffer *)arg; + + if (!opt_disable_rseq && rseq_register_current_thread()) + abort(); + + reps = opt_reps; + for (i = 0; i < reps; i++) { + struct percpu_memcpy_buffer_node item; + bool result; + + result = percpu_memcpy_buffer_pop(buffer, &item, + rseq_cpu_start()); + if (opt_yield) + sched_yield(); /* encourage shuffling */ + if (result) { + if (!percpu_memcpy_buffer_push(buffer, item, + rseq_cpu_start())) { + /* Should increase buffer size. */ + abort(); + } + } + } + + printf_verbose("tid %d: number of rseq abort: %d, signals delivered: %u\n", + (int) gettid(), nr_abort, signals_delivered); + if (!opt_disable_rseq && rseq_unregister_current_thread()) + abort(); + + return NULL; +} + +/* Simultaneous modification to a per-cpu buffer from many threads. */ +void test_percpu_memcpy_buffer(void) +{ + const int num_threads = opt_threads; + int i, j, ret; + uint64_t sum = 0, expected_sum = 0; + struct percpu_memcpy_buffer buffer; + pthread_t test_threads[num_threads]; + cpu_set_t allowed_cpus; + + memset(&buffer, 0, sizeof(buffer)); + + /* Generate list entries for every usable cpu. */ + sched_getaffinity(0, sizeof(allowed_cpus), &allowed_cpus); + for (i = 0; i < CPU_SETSIZE; i++) { + if (!CPU_ISSET(i, &allowed_cpus)) + continue; + /* Worse-case is every item in same CPU. */ + buffer.c[i].array = + malloc(sizeof(*buffer.c[i].array) * CPU_SETSIZE * + MEMCPY_BUFFER_ITEM_PER_CPU); + assert(buffer.c[i].array); + buffer.c[i].buflen = CPU_SETSIZE * MEMCPY_BUFFER_ITEM_PER_CPU; + for (j = 1; j <= MEMCPY_BUFFER_ITEM_PER_CPU; j++) { + expected_sum += 2 * j + 1; + + /* + * We could theoretically put the word-sized + * "data" directly in the buffer. However, we + * want to model objects that would not fit + * within a single word, so allocate an object + * for each node. + */ + buffer.c[i].array[j - 1].data1 = j; + buffer.c[i].array[j - 1].data2 = j + 1; + buffer.c[i].offset++; + } + } + + for (i = 0; i < num_threads; i++) { + ret = pthread_create(&test_threads[i], NULL, + test_percpu_memcpy_buffer_thread, + &buffer); + if (ret) { + errno = ret; + perror("pthread_create"); + abort(); + } + } + + for (i = 0; i < num_threads; i++) { + ret = pthread_join(test_threads[i], NULL); + if (ret) { + errno = ret; + perror("pthread_join"); + abort(); + } + } + + for (i = 0; i < CPU_SETSIZE; i++) { + struct percpu_memcpy_buffer_node item; + + if (!CPU_ISSET(i, &allowed_cpus)) + continue; + + while (percpu_memcpy_buffer_pop(&buffer, &item, i)) { + sum += item.data1; + sum += item.data2; + } + free(buffer.c[i].array); + } + + /* + * All entries should now be accounted for (unless some external + * actor is interfering with our allowed affinity while this + * test is running). + */ + assert(sum == expected_sum); +} + +static void test_signal_interrupt_handler(int signo) +{ + signals_delivered++; +} + +static int set_signal_handler(void) +{ + int ret = 0; + struct sigaction sa; + sigset_t sigset; + + ret = sigemptyset(&sigset); + if (ret < 0) { + perror("sigemptyset"); + return ret; + } + + sa.sa_handler = test_signal_interrupt_handler; + sa.sa_mask = sigset; + sa.sa_flags = 0; + ret = sigaction(SIGUSR1, &sa, NULL); + if (ret < 0) { + perror("sigaction"); + return ret; + } + + printf_verbose("Signal handler set for SIGUSR1\n"); + + return ret; +} + +static void show_usage(int argc, char **argv) +{ + printf("Usage : %s <OPTIONS>\n", + argv[0]); + printf("OPTIONS:\n"); + printf(" [-1 loops] Number of loops for delay injection 1\n"); + printf(" [-2 loops] Number of loops for delay injection 2\n"); + printf(" [-3 loops] Number of loops for delay injection 3\n"); + printf(" [-4 loops] Number of loops for delay injection 4\n"); + printf(" [-5 loops] Number of loops for delay injection 5\n"); + printf(" [-6 loops] Number of loops for delay injection 6\n"); + printf(" [-7 loops] Number of loops for delay injection 7 (-1 to enable -m)\n"); + printf(" [-8 loops] Number of loops for delay injection 8 (-1 to enable -m)\n"); + printf(" [-9 loops] Number of loops for delay injection 9 (-1 to enable -m)\n"); + printf(" [-m N] Yield/sleep/kill every modulo N (default 0: disabled) (>= 0)\n"); + printf(" [-y] Yield\n"); + printf(" [-k] Kill thread with signal\n"); + printf(" [-s S] S: =0: disabled (default), >0: sleep time (ms)\n"); + printf(" [-t N] Number of threads (default 200)\n"); + printf(" [-r N] Number of repetitions per thread (default 5000)\n"); + printf(" [-d] Disable rseq system call (no initialization)\n"); + printf(" [-D M] Disable rseq for each M threads\n"); + printf(" [-T test] Choose test: (s)pinlock, (l)ist, (b)uffer, (m)emcpy, (i)ncrement\n"); + printf(" [-M] Push into buffer and memcpy buffer with memory barriers.\n"); + printf(" [-v] Verbose output.\n"); + printf(" [-h] Show this help.\n"); + printf("\n"); +} + +int main(int argc, char **argv) +{ + int i; + + for (i = 1; i < argc; i++) { + if (argv[i][0] != '-') + continue; + switch (argv[i][1]) { + case '1': + case '2': + case '3': + case '4': + case '5': + case '6': + case '7': + case '8': + case '9': + if (argc < i + 2) { + show_usage(argc, argv); + goto error; + } + loop_cnt[argv[i][1] - '0'] = atol(argv[i + 1]); + i++; + break; + case 'm': + if (argc < i + 2) { + show_usage(argc, argv); + goto error; + } + opt_modulo = atol(argv[i + 1]); + if (opt_modulo < 0) { + show_usage(argc, argv); + goto error; + } + i++; + break; + case 's': + if (argc < i + 2) { + show_usage(argc, argv); + goto error; + } + opt_sleep = atol(argv[i + 1]); + if (opt_sleep < 0) { + show_usage(argc, argv); + goto error; + } + i++; + break; + case 'y': + opt_yield = 1; + break; + case 'k': + opt_signal = 1; + break; + case 'd': + opt_disable_rseq = 1; + break; + case 'D': + if (argc < i + 2) { + show_usage(argc, argv); + goto error; + } + opt_disable_mod = atol(argv[i + 1]); + if (opt_disable_mod < 0) { + show_usage(argc, argv); + goto error; + } + i++; + break; + case 't': + if (argc < i + 2) { + show_usage(argc, argv); + goto error; + } + opt_threads = atol(argv[i + 1]); + if (opt_threads < 0) { + show_usage(argc, argv); + goto error; + } + i++; + break; + case 'r': + if (argc < i + 2) { + show_usage(argc, argv); + goto error; + } + opt_reps = atoll(argv[i + 1]); + if (opt_reps < 0) { + show_usage(argc, argv); + goto error; + } + i++; + break; + case 'h': + show_usage(argc, argv); + goto end; + case 'T': + if (argc < i + 2) { + show_usage(argc, argv); + goto error; + } + opt_test = *argv[i + 1]; + switch (opt_test) { + case 's': + case 'l': + case 'i': + case 'b': + case 'm': + break; + default: + show_usage(argc, argv); + goto error; + } + i++; + break; + case 'v': + verbose = 1; + break; + case 'M': + opt_mb = 1; + break; + default: + show_usage(argc, argv); + goto error; + } + } + + loop_cnt_1 = loop_cnt[1]; + loop_cnt_2 = loop_cnt[2]; + loop_cnt_3 = loop_cnt[3]; + loop_cnt_4 = loop_cnt[4]; + loop_cnt_5 = loop_cnt[5]; + loop_cnt_6 = loop_cnt[6]; + + if (set_signal_handler()) + goto error; + + if (!opt_disable_rseq && rseq_register_current_thread()) + goto error; + switch (opt_test) { + case 's': + printf_verbose("spinlock\n"); + test_percpu_spinlock(); + break; + case 'l': + printf_verbose("linked list\n"); + test_percpu_list(); + break; + case 'b': + printf_verbose("buffer\n"); + test_percpu_buffer(); + break; + case 'm': + printf_verbose("memcpy buffer\n"); + test_percpu_memcpy_buffer(); + break; + case 'i': + printf_verbose("counter increment\n"); + test_percpu_inc(); + break; + } + if (!opt_disable_rseq && rseq_unregister_current_thread()) + abort(); +end: + return 0; + +error: + return -1; +} -- 2.11.0

6 years, 2 months

1
0
0 0

[RFC PATCH for 4.21 14/16] cpu-opv/selftests: Provide basic percpu ops test

by Mathieu Desnoyers

"basic_percpu_ops_test" implements a few simple per-cpu operations and testing their correctness. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers(a)efficios.com> CC: Thomas Gleixner <tglx(a)linutronix.de> Cc: Joel Fernandes <joelaf(a)google.com> Cc: Peter Zijlstra <peterz(a)infradead.org> Cc: Catalin Marinas <catalin.marinas(a)arm.com> Cc: Dave Watson <davejwatson(a)fb.com> Cc: Will Deacon <will.deacon(a)arm.com> Cc: Shuah Khan <shuah(a)kernel.org> Cc: Andi Kleen <andi(a)firstfloor.org> Cc: linux-kselftest(a)vger.kernel.org Cc: "H . Peter Anvin" <hpa(a)zytor.com> Cc: Chris Lameter <cl(a)linux.com> Cc: Russell King <linux(a)arm.linux.org.uk> Cc: Michael Kerrisk <mtk.manpages(a)gmail.com> Cc: "Paul E . McKenney" <paulmck(a)linux.vnet.ibm.com> Cc: Paul Turner <pjt(a)google.com> Cc: Boqun Feng <boqun.feng(a)gmail.com> Cc: Josh Triplett <josh(a)joshtriplett.org> Cc: Steven Rostedt <rostedt(a)goodmis.org> Cc: Ben Maurer <bmaurer(a)fb.com> Cc: linux-api(a)vger.kernel.org Cc: Andy Lutomirski <luto(a)amacapital.net> Cc: Andrew Morton <akpm(a)linux-foundation.org> Cc: Linus Torvalds <torvalds(a)linux-foundation.org> --- .../selftests/cpu-opv/basic_percpu_ops_test.c | 295 +++++++++++++++++++++ 1 file changed, 295 insertions(+) create mode 100644 tools/testing/selftests/cpu-opv/basic_percpu_ops_test.c diff --git a/tools/testing/selftests/cpu-opv/basic_percpu_ops_test.c b/tools/testing/selftests/cpu-opv/basic_percpu_ops_test.c new file mode 100644 index 000000000000..2ce5202f25b2 --- /dev/null +++ b/tools/testing/selftests/cpu-opv/basic_percpu_ops_test.c @@ -0,0 +1,295 @@ +// SPDX-License-Identifier: LGPL-2.1 +#define _GNU_SOURCE +#include <assert.h> +#include <pthread.h> +#include <sched.h> +#include <stdint.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <stddef.h> + +#include "percpu-op.h" + +#define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0])) + +struct percpu_lock_entry { + intptr_t v; +} __attribute__((aligned(128))); + +struct percpu_lock { + struct percpu_lock_entry c[CPU_SETSIZE]; +}; + +struct test_data_entry { + intptr_t count; +} __attribute__((aligned(128))); + +struct spinlock_test_data { + struct percpu_lock lock; + struct test_data_entry c[CPU_SETSIZE]; + int reps; +}; + +struct percpu_list_node { + intptr_t data; + struct percpu_list_node *next; +}; + +struct percpu_list_entry { + struct percpu_list_node *head; +} __attribute__((aligned(128))); + +struct percpu_list { + struct percpu_list_entry c[CPU_SETSIZE]; +}; + +/* A simple percpu spinlock. */ +void rseq_percpu_lock(struct percpu_lock *lock, int cpu) +{ + for (;;) { + int ret; + + ret = percpu_cmpeqv_storev(&lock->c[cpu].v, + 0, 1, cpu); + if (rseq_likely(!ret)) + break; + if (rseq_unlikely(ret < 0)) { + perror("cpu_opv"); + abort(); + } + /* Retry if comparison fails. */ + } + /* + * Acquire semantic when taking lock after control dependency. + * Matches rseq_smp_store_release(). + */ + rseq_smp_acquire__after_ctrl_dep(); +} + +void rseq_percpu_unlock(struct percpu_lock *lock, int cpu) +{ + assert(lock->c[cpu].v == 1); + /* + * Release lock, with release semantic. Matches + * rseq_smp_acquire__after_ctrl_dep(). + */ + rseq_smp_store_release(&lock->c[cpu].v, 0); +} + +void *test_percpu_spinlock_thread(void *arg) +{ + struct spinlock_test_data *data = arg; + int i; + + if (rseq_register_current_thread()) { + fprintf(stderr, "Error: rseq_register_current_thread(...) failed(%d): %s\n", + errno, strerror(errno)); + abort(); + } + for (i = 0; i < data->reps; i++) { + int cpu = percpu_current_cpu(); + + rseq_percpu_lock(&data->lock, cpu); + data->c[cpu].count++; + rseq_percpu_unlock(&data->lock, cpu); + } + if (rseq_unregister_current_thread()) { + fprintf(stderr, "Error: rseq_unregister_current_thread(...) failed(%d): %s\n", + errno, strerror(errno)); + abort(); + } + + return NULL; +} + +/* + * A simple test which implements a sharded counter using a per-cpu + * lock. Obviously real applications might prefer to simply use a + * per-cpu increment; however, this is reasonable for a test and the + * lock can be extended to synchronize more complicated operations. + */ +void test_percpu_spinlock(void) +{ + const int num_threads = 200; + int i; + uint64_t sum; + pthread_t test_threads[num_threads]; + struct spinlock_test_data data; + + memset(&data, 0, sizeof(data)); + data.reps = 5000; + + for (i = 0; i < num_threads; i++) + pthread_create(&test_threads[i], NULL, + test_percpu_spinlock_thread, &data); + + for (i = 0; i < num_threads; i++) + pthread_join(test_threads[i], NULL); + + sum = 0; + for (i = 0; i < CPU_SETSIZE; i++) + sum += data.c[i].count; + + assert(sum == (uint64_t)data.reps * num_threads); +} + +int percpu_list_push(struct percpu_list *list, struct percpu_list_node *node, + int cpu) +{ + for (;;) { + intptr_t *targetptr, newval, expect; + int ret; + + /* Load list->c[cpu].head with single-copy atomicity. */ + expect = (intptr_t)RSEQ_READ_ONCE(list->c[cpu].head); + newval = (intptr_t)node; + targetptr = (intptr_t *)&list->c[cpu].head; + node->next = (struct percpu_list_node *)expect; + ret = percpu_cmpeqv_storev(targetptr, expect, newval, cpu); + if (rseq_likely(!ret)) + break; + if (rseq_unlikely(ret < 0)) { + perror("cpu_opv"); + abort(); + } + /* Retry if comparison fails. */ + } + return cpu; +} + +/* + * Unlike a traditional lock-less linked list; the availability of a + * rseq primitive allows us to implement pop without concerns over + * ABA-type races. + */ +struct percpu_list_node *percpu_list_pop(struct percpu_list *list, + int cpu) +{ + struct percpu_list_node *head; + intptr_t *targetptr, expectnot, *load; + off_t offset; + int ret; + + targetptr = (intptr_t *)&list->c[cpu].head; + expectnot = (intptr_t)NULL; + offset = offsetof(struct percpu_list_node, next); + load = (intptr_t *)&head; + ret = percpu_cmpnev_storeoffp_load(targetptr, expectnot, + offset, load, cpu); + if (rseq_unlikely(ret < 0)) { + perror("cpu_opv"); + abort(); + } + if (ret > 0) + return NULL; + return head; +} + +void *test_percpu_list_thread(void *arg) +{ + int i; + struct percpu_list *list = (struct percpu_list *)arg; + + if (rseq_register_current_thread()) { + fprintf(stderr, "Error: rseq_register_current_thread(...) failed(%d): %s\n", + errno, strerror(errno)); + abort(); + } + + for (i = 0; i < 100000; i++) { + struct percpu_list_node *node; + + node = percpu_list_pop(list, percpu_current_cpu()); + sched_yield(); /* encourage shuffling */ + if (node) + percpu_list_push(list, node, percpu_current_cpu()); + } + + if (rseq_unregister_current_thread()) { + fprintf(stderr, "Error: rseq_unregister_current_thread(...) failed(%d): %s\n", + errno, strerror(errno)); + abort(); + } + + return NULL; +} + +/* Simultaneous modification to a per-cpu linked list from many threads. */ +void test_percpu_list(void) +{ + int i, j; + uint64_t sum = 0, expected_sum = 0; + struct percpu_list list; + pthread_t test_threads[200]; + cpu_set_t allowed_cpus; + + memset(&list, 0, sizeof(list)); + + /* Generate list entries for every usable cpu. */ + sched_getaffinity(0, sizeof(allowed_cpus), &allowed_cpus); + for (i = 0; i < CPU_SETSIZE; i++) { + if (!CPU_ISSET(i, &allowed_cpus)) + continue; + for (j = 1; j <= 100; j++) { + struct percpu_list_node *node; + + expected_sum += j; + + node = malloc(sizeof(*node)); + assert(node); + node->data = j; + node->next = list.c[i].head; + list.c[i].head = node; + } + } + + for (i = 0; i < 200; i++) + pthread_create(&test_threads[i], NULL, + test_percpu_list_thread, &list); + + for (i = 0; i < 200; i++) + pthread_join(test_threads[i], NULL); + + for (i = 0; i < CPU_SETSIZE; i++) { + struct percpu_list_node *node; + + if (!CPU_ISSET(i, &allowed_cpus)) + continue; + + while ((node = percpu_list_pop(&list, i))) { + sum += node->data; + free(node); + } + } + + /* + * All entries should now be accounted for (unless some external + * actor is interfering with our allowed affinity while this + * test is running). + */ + assert(sum == expected_sum); +} + +int main(int argc, char **argv) +{ + if (rseq_register_current_thread()) { + fprintf(stderr, "Error: rseq_register_current_thread(...) failed(%d): %s\n", + errno, strerror(errno)); + goto error; + } + printf("spinlock\n"); + test_percpu_spinlock(); + printf("percpu_list\n"); + test_percpu_list(); + if (rseq_unregister_current_thread()) { + fprintf(stderr, "Error: rseq_unregister_current_thread(...) failed(%d): %s\n", + errno, strerror(errno)); + goto error; + } + return 0; + +error: + return -1; +} + -- 2.11.0

6 years, 2 months

1
0
0 0

[RFC PATCH for 4.21 13/16] cpu-opv/selftests: Provide percpu_op API

by Mathieu Desnoyers

Introduce percpu-op.h API. It uses rseq internally as fast-path if invoked from the right CPU, else cpu_opv as slow-path if called from the wrong CPU or if rseq fails. This allows acting on per-cpu data from various CPUs transparently from user-space: cpu_opv will take care of migrating the thread to the requested CPU. Use-cases such as rebalancing memory across per-cpu memory pools, or migrating tasks for a user-space scheduler, are thus facilitated. This also handles debugger single-stepping. The use from userspace is, e.g. for a counter increment: int cpu, ret; cpu = percpu_current_cpu(); ret = percpu_addv(&data->c[cpu].count, 1, cpu); if (unlikely(ret)) { perror("percpu_addv"); return -1; } return 0; Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers(a)efficios.com> CC: Shuah Khan <shuah(a)kernel.org> CC: Russell King <linux(a)arm.linux.org.uk> CC: Catalin Marinas <catalin.marinas(a)arm.com> CC: Will Deacon <will.deacon(a)arm.com> CC: Thomas Gleixner <tglx(a)linutronix.de> CC: Paul Turner <pjt(a)google.com> CC: Peter Zijlstra <peterz(a)infradead.org> CC: Andy Lutomirski <luto(a)amacapital.net> CC: Andi Kleen <andi(a)firstfloor.org> CC: Dave Watson <davejwatson(a)fb.com> CC: Chris Lameter <cl(a)linux.com> CC: Ingo Molnar <mingo(a)redhat.com> CC: "H. Peter Anvin" <hpa(a)zytor.com> CC: Ben Maurer <bmaurer(a)fb.com> CC: Steven Rostedt <rostedt(a)goodmis.org> CC: "Paul E. McKenney" <paulmck(a)linux.vnet.ibm.com> CC: Josh Triplett <josh(a)joshtriplett.org> CC: Linus Torvalds <torvalds(a)linux-foundation.org> CC: Andrew Morton <akpm(a)linux-foundation.org> CC: Boqun Feng <boqun.feng(a)gmail.com> CC: linux-kselftest(a)vger.kernel.org CC: linux-api(a)vger.kernel.org --- tools/testing/selftests/cpu-opv/percpu-op.h | 151 ++++++++++++++++++++++++++++ 1 file changed, 151 insertions(+) create mode 100644 tools/testing/selftests/cpu-opv/percpu-op.h diff --git a/tools/testing/selftests/cpu-opv/percpu-op.h b/tools/testing/selftests/cpu-opv/percpu-op.h new file mode 100644 index 000000000000..918171e585d7 --- /dev/null +++ b/tools/testing/selftests/cpu-opv/percpu-op.h @@ -0,0 +1,151 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ +/* + * percpu-op.h + * + * (C) Copyright 2017-2018 - Mathieu Desnoyers <mathieu.desnoyers(a)efficios.com> + */ + +#ifndef PERCPU_OP_H +#define PERCPU_OP_H + +#include <stdint.h> +#include <stdbool.h> +#include <errno.h> +#include <stdlib.h> +#include "rseq.h" +#include "cpu-op.h" + +static inline uint32_t percpu_current_cpu(void) +{ + return rseq_current_cpu(); +} + +static inline __attribute__((always_inline)) +int percpu_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, + int cpu) +{ + int ret; + + ret = rseq_cmpeqv_storev(v, expect, newv, cpu); + if (rseq_unlikely(ret)) { + if (ret > 0) + return ret; + return cpu_op_cmpeqv_storev(v, expect, newv, cpu); + } + return 0; +} + +static inline __attribute__((always_inline)) +int percpu_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, + off_t voffp, intptr_t *load, int cpu) +{ + int ret; + + ret = rseq_cmpnev_storeoffp_load(v, expectnot, voffp, load, cpu); + if (rseq_unlikely(ret)) { + if (ret > 0) + return ret; + return cpu_op_cmpnev_storeoffp_load(v, expectnot, voffp, + load, cpu); + } + return 0; +} + +static inline __attribute__((always_inline)) +int percpu_addv(intptr_t *v, intptr_t count, int cpu) +{ + if (rseq_unlikely(rseq_addv(v, count, cpu))) + return cpu_op_addv(v, count, cpu); + return 0; +} + +static inline __attribute__((always_inline)) +int percpu_cmpeqv_storev_storev(intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + int ret; + + ret = rseq_cmpeqv_trystorev_storev(v, expect, v2, newv2, + newv, cpu); + if (rseq_unlikely(ret)) { + if (ret > 0) + return ret; + return cpu_op_cmpeqv_storev_storev(v, expect, v2, newv2, + newv, cpu); + } + return 0; +} + +static inline __attribute__((always_inline)) +int percpu_cmpeqv_storev_storev_release(intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + int ret; + + ret = rseq_cmpeqv_trystorev_storev_release(v, expect, v2, newv2, + newv, cpu); + if (rseq_unlikely(ret)) { + if (ret > 0) + return ret; + return cpu_op_cmpeqv_storev_storev_release(v, expect, v2, newv2, + newv, cpu); + } + return 0; +} + +static inline __attribute__((always_inline)) +int percpu_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t expect2, + intptr_t newv, int cpu) +{ + int ret; + + ret = rseq_cmpeqv_cmpeqv_storev(v, expect, v2, expect2, newv, cpu); + if (rseq_unlikely(ret)) { + if (ret > 0) + return ret; + return cpu_op_cmpeqv_cmpeqv_storev(v, expect, v2, expect2, + newv, cpu); + } + return 0; +} + +static inline __attribute__((always_inline)) +int percpu_cmpeqv_memcpy_storev(intptr_t *v, intptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + int ret; + + ret = rseq_cmpeqv_trymemcpy_storev(v, expect, dst, src, len, + newv, cpu); + if (rseq_unlikely(ret)) { + if (ret > 0) + return ret; + return cpu_op_cmpeqv_memcpy_storev(v, expect, dst, src, len, + newv, cpu); + } + return 0; +} + +static inline __attribute__((always_inline)) +int percpu_cmpeqv_memcpy_storev_release(intptr_t *v, intptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + int ret; + + ret = rseq_cmpeqv_trymemcpy_storev_release(v, expect, dst, src, len, + newv, cpu); + if (rseq_unlikely(ret)) { + if (ret > 0) + return ret; + return cpu_op_cmpeqv_memcpy_storev_release(v, expect, dst, src, + len, newv, cpu); + } + return 0; +} + +#endif /* PERCPU_OP_H_ */ -- 2.11.0

6 years, 2 months

1
0
0 0

[RFC PATCH for 4.21 12/16] cpu-opv/selftests: Provide basic test

by Mathieu Desnoyers

"basic_cpu_opv_test" tests basic functionality of cpu_opv. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers(a)efficios.com> Cc: Thomas Gleixner <tglx(a)linutronix.de> Cc: Joel Fernandes <joelaf(a)google.com> Cc: Peter Zijlstra <peterz(a)infradead.org> Cc: Catalin Marinas <catalin.marinas(a)arm.com> Cc: Dave Watson <davejwatson(a)fb.com> Cc: Will Deacon <will.deacon(a)arm.com> Cc: Shuah Khan <shuah(a)kernel.org> Cc: Andi Kleen <andi(a)firstfloor.org> Cc: linux-kselftest(a)vger.kernel.org Cc: "H . Peter Anvin" <hpa(a)zytor.com> Cc: Chris Lameter <cl(a)linux.com> Cc: Russell King <linux(a)arm.linux.org.uk> Cc: Michael Kerrisk <mtk.manpages(a)gmail.com> Cc: "Paul E . McKenney" <paulmck(a)linux.vnet.ibm.com> Cc: Paul Turner <pjt(a)google.com> Cc: Boqun Feng <boqun.feng(a)gmail.com> Cc: Josh Triplett <josh(a)joshtriplett.org> Cc: Steven Rostedt <rostedt(a)goodmis.org> Cc: Ben Maurer <bmaurer(a)fb.com> Cc: linux-api(a)vger.kernel.org Cc: Andy Lutomirski <luto(a)amacapital.net> Cc: Andrew Morton <akpm(a)linux-foundation.org> Cc: Linus Torvalds <torvalds(a)linux-foundation.org> --- .../testing/selftests/cpu-opv/basic_cpu_opv_test.c | 1207 ++++++++++++++++++++ 1 file changed, 1207 insertions(+) create mode 100644 tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c diff --git a/tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c b/tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c new file mode 100644 index 000000000000..28221b0260fa --- /dev/null +++ b/tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c @@ -0,0 +1,1207 @@ +// SPDX-License-Identifier: LGPL-2.1 +/* + * Basic test coverage for cpu_opv system call. + */ + +#define _GNU_SOURCE +#include <assert.h> +#include <sched.h> +#include <signal.h> +#include <stdio.h> +#include <string.h> +#include <errno.h> +#include <stdlib.h> +#include <sys/time.h> +#include <sys/mman.h> +#include <sched.h> + +#include "../kselftest.h" + +#include "cpu-op.h" + +#define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0])) + +#define TESTBUFLEN 4096 +#define TESTBUFLEN_CMP 16 + +#define TESTBUFLEN_PAGE_MAX 65536 + +#define NR_PF_ARRAY 16384 +#define PF_ARRAY_LEN 4096 + +#define NR_HUGE_ARRAY 512 +#define HUGEMAPLEN (NR_HUGE_ARRAY * PF_ARRAY_LEN) + +/* 64 MB arrays for page fault testing. */ +char pf_array_dst[NR_PF_ARRAY][PF_ARRAY_LEN]; +char pf_array_src[NR_PF_ARRAY][PF_ARRAY_LEN]; + +static int test_ops_supported(void) +{ + const char *test_name = "test_ops_supported"; + int ret; + + ret = cpu_opv(NULL, 0, -1, CPU_OP_NR_FLAG); + if (ret < 0) { + ksft_test_result_fail("%s test: returned with %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (ret < NR_CPU_OPS) { + ksft_test_result_fail("%s test: only %d operations supported, expecting at least %d\n", + test_name, ret, NR_CPU_OPS); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int test_compare_eq_op(char *a, char *b, size_t len) +{ + struct cpu_op opvec[] = { + [0] = { + .op = CPU_COMPARE_EQ_OP, + .len = len, + .u.compare_op.a = (unsigned long)a, + .u.compare_op.b = (unsigned long)b, + .u.compare_op.expect_fault_a = 0, + .u.compare_op.expect_fault_b = 0, + }, + }; + int ret, cpu; + + do { + cpu = cpu_op_get_current_cpu(); + ret = cpu_opv(opvec, ARRAY_SIZE(opvec), cpu, 0); + } while (ret == -1 && errno == EAGAIN); + + return ret; +} + +static int test_compare_eq_same(void) +{ + int i, ret; + char buf1[TESTBUFLEN]; + char buf2[TESTBUFLEN]; + const char *test_name = "test_compare_eq same"; + + /* Test compare_eq */ + for (i = 0; i < TESTBUFLEN; i++) + buf1[i] = (char)i; + for (i = 0; i < TESTBUFLEN; i++) + buf2[i] = (char)i; + ret = test_compare_eq_op(buf2, buf1, TESTBUFLEN); + if (ret < 0) { + ksft_test_result_fail("%s test: returned with %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (ret > 0) { + ksft_test_result_fail("%s test: unexpected value %d. Should be %d.\n", + test_name, ret, 0); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int test_compare_eq_diff(void) +{ + int i, ret; + char buf1[TESTBUFLEN]; + char buf2[TESTBUFLEN]; + const char *test_name = "test_compare_eq different"; + + for (i = 0; i < TESTBUFLEN; i++) + buf1[i] = (char)i; + memset(buf2, 0, TESTBUFLEN); + ret = test_compare_eq_op(buf2, buf1, TESTBUFLEN); + if (ret < 0) { + ksft_test_result_fail("%s test: returned with %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (ret == 0) { + ksft_test_result_fail("%s test: unexpected value %d. Should be %d.\n", + test_name, ret, 1); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int test_compare_ne_op(char *a, char *b, size_t len) +{ + struct cpu_op opvec[] = { + [0] = { + .op = CPU_COMPARE_NE_OP, + .len = len, + .u.compare_op.a = (unsigned long)a, + .u.compare_op.b = (unsigned long)b, + .u.compare_op.expect_fault_a = 0, + .u.compare_op.expect_fault_b = 0, + }, + }; + int ret, cpu; + + do { + cpu = cpu_op_get_current_cpu(); + ret = cpu_opv(opvec, ARRAY_SIZE(opvec), cpu, 0); + } while (ret == -1 && errno == EAGAIN); + + return ret; +} + +static int test_compare_ne_same(void) +{ + int i, ret; + char buf1[TESTBUFLEN]; + char buf2[TESTBUFLEN]; + const char *test_name = "test_compare_ne same"; + + /* Test compare_ne */ + for (i = 0; i < TESTBUFLEN; i++) + buf1[i] = (char)i; + for (i = 0; i < TESTBUFLEN; i++) + buf2[i] = (char)i; + ret = test_compare_ne_op(buf2, buf1, TESTBUFLEN); + if (ret < 0) { + ksft_test_result_fail("%s test: returned with %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (ret == 0) { + ksft_test_result_fail("%s test: unexpected value %d. Should be %d.\n", + test_name, ret, 1); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int test_compare_ne_diff(void) +{ + int i, ret; + char buf1[TESTBUFLEN]; + char buf2[TESTBUFLEN]; + const char *test_name = "test_compare_ne different"; + + for (i = 0; i < TESTBUFLEN; i++) + buf1[i] = (char)i; + memset(buf2, 0, TESTBUFLEN); + ret = test_compare_ne_op(buf2, buf1, TESTBUFLEN); + if (ret < 0) { + ksft_test_result_fail("%s test: returned with %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (ret != 0) { + ksft_test_result_fail("%s test: unexpected value %d. Should be %d.\n", + test_name, ret, 0); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int test_2compare_eq_op(char *a, char *b, char *c, char *d, + size_t len) +{ + struct cpu_op opvec[] = { + [0] = { + .op = CPU_COMPARE_EQ_OP, + .len = len, + .u.compare_op.a = (unsigned long)a, + .u.compare_op.b = (unsigned long)b, + .u.compare_op.expect_fault_a = 0, + .u.compare_op.expect_fault_b = 0, + }, + [1] = { + .op = CPU_COMPARE_EQ_OP, + .len = len, + .u.compare_op.a = (unsigned long)c, + .u.compare_op.b = (unsigned long)d, + .u.compare_op.expect_fault_a = 0, + .u.compare_op.expect_fault_b = 0, + }, + }; + int ret, cpu; + + do { + cpu = cpu_op_get_current_cpu(); + ret = cpu_opv(opvec, ARRAY_SIZE(opvec), cpu, 0); + } while (ret == -1 && errno == EAGAIN); + + return ret; +} + +static int test_2compare_eq_index(void) +{ + int i, ret; + char buf1[TESTBUFLEN_CMP]; + char buf2[TESTBUFLEN_CMP]; + char buf3[TESTBUFLEN_CMP]; + char buf4[TESTBUFLEN_CMP]; + const char *test_name = "test_2compare_eq index"; + + for (i = 0; i < TESTBUFLEN_CMP; i++) + buf1[i] = (char)i; + memset(buf2, 0, TESTBUFLEN_CMP); + memset(buf3, 0, TESTBUFLEN_CMP); + memset(buf4, 0, TESTBUFLEN_CMP); + + /* First compare failure is op[0], expect 1. */ + ret = test_2compare_eq_op(buf2, buf1, buf4, buf3, TESTBUFLEN_CMP); + if (ret < 0) { + ksft_test_result_fail("%s test: returned with %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (ret != 1) { + ksft_test_result_fail("%s test: unexpected value %d. Should be %d.\n", + test_name, ret, 1); + return -1; + } + + /* All compares succeed. */ + for (i = 0; i < TESTBUFLEN_CMP; i++) + buf2[i] = (char)i; + ret = test_2compare_eq_op(buf2, buf1, buf4, buf3, TESTBUFLEN_CMP); + if (ret < 0) { + ksft_test_result_fail("%s test: returned with %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (ret != 0) { + ksft_test_result_fail("%s test: unexpected value %d. Should be %d.\n", + test_name, ret, 0); + return -1; + } + + /* First compare failure is op[1], expect 2. */ + for (i = 0; i < TESTBUFLEN_CMP; i++) + buf3[i] = (char)i; + ret = test_2compare_eq_op(buf2, buf1, buf4, buf3, TESTBUFLEN_CMP); + if (ret < 0) { + ksft_test_result_fail("%s test: returned with %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (ret != 2) { + ksft_test_result_fail("%s test: unexpected value %d. Should be %d.\n", + test_name, ret, 2); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int test_2compare_ne_op(char *a, char *b, char *c, char *d, + size_t len) +{ + struct cpu_op opvec[] = { + [0] = { + .op = CPU_COMPARE_NE_OP, + .len = len, + .u.compare_op.a = (unsigned long)a, + .u.compare_op.b = (unsigned long)b, + .u.compare_op.expect_fault_a = 0, + .u.compare_op.expect_fault_b = 0, + }, + [1] = { + .op = CPU_COMPARE_NE_OP, + .len = len, + .u.compare_op.a = (unsigned long)c, + .u.compare_op.b = (unsigned long)d, + .u.compare_op.expect_fault_a = 0, + .u.compare_op.expect_fault_b = 0, + }, + }; + int ret, cpu; + + do { + cpu = cpu_op_get_current_cpu(); + ret = cpu_opv(opvec, ARRAY_SIZE(opvec), cpu, 0); + } while (ret == -1 && errno == EAGAIN); + + return ret; +} + +static int test_2compare_ne_index(void) +{ + int i, ret; + char buf1[TESTBUFLEN_CMP]; + char buf2[TESTBUFLEN_CMP]; + char buf3[TESTBUFLEN_CMP]; + char buf4[TESTBUFLEN_CMP]; + const char *test_name = "test_2compare_ne index"; + + memset(buf1, 0, TESTBUFLEN_CMP); + memset(buf2, 0, TESTBUFLEN_CMP); + memset(buf3, 0, TESTBUFLEN_CMP); + memset(buf4, 0, TESTBUFLEN_CMP); + + /* First compare ne failure is op[0], expect 1. */ + ret = test_2compare_ne_op(buf2, buf1, buf4, buf3, TESTBUFLEN_CMP); + if (ret < 0) { + ksft_test_result_fail("%s test: returned with %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (ret != 1) { + ksft_test_result_fail("%s test: unexpected value %d. Should be %d.\n", + test_name, ret, 1); + return -1; + } + + /* All compare ne succeed. */ + for (i = 0; i < TESTBUFLEN_CMP; i++) + buf1[i] = (char)i; + for (i = 0; i < TESTBUFLEN_CMP; i++) + buf3[i] = (char)i; + ret = test_2compare_ne_op(buf2, buf1, buf4, buf3, TESTBUFLEN_CMP); + if (ret < 0) { + ksft_test_result_fail("%s test: returned with %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (ret != 0) { + ksft_test_result_fail("%s test: unexpected value %d. Should be %d.\n", + test_name, ret, 0); + return -1; + } + + /* First compare failure is op[1], expect 2. */ + for (i = 0; i < TESTBUFLEN_CMP; i++) + buf4[i] = (char)i; + ret = test_2compare_ne_op(buf2, buf1, buf4, buf3, TESTBUFLEN_CMP); + if (ret < 0) { + ksft_test_result_fail("%s test: returned with %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (ret != 2) { + ksft_test_result_fail("%s test: unexpected value %d. Should be %d.\n", + test_name, ret, 2); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int test_memcpy_op(void *dst, void *src, size_t len) +{ + struct cpu_op opvec[] = { + [0] = { + .op = CPU_MEMCPY_OP, + .len = len, + .u.memcpy_op.dst = (unsigned long)dst, + .u.memcpy_op.src = (unsigned long)src, + .u.memcpy_op.expect_fault_dst = 0, + .u.memcpy_op.expect_fault_src = 0, + }, + }; + int ret, cpu; + + do { + cpu = cpu_op_get_current_cpu(); + ret = cpu_opv(opvec, ARRAY_SIZE(opvec), cpu, 0); + } while (ret == -1 && errno == EAGAIN); + + return ret; +} + +static int test_memcpy(void) +{ + int i, ret; + char buf1[TESTBUFLEN]; + char buf2[TESTBUFLEN]; + const char *test_name = "test_memcpy"; + + /* Test memcpy */ + for (i = 0; i < TESTBUFLEN; i++) + buf1[i] = (char)i; + memset(buf2, 0, TESTBUFLEN); + ret = test_memcpy_op(buf2, buf1, TESTBUFLEN); + if (ret) { + ksft_test_result_fail("%s test: returned with %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + for (i = 0; i < TESTBUFLEN; i++) { + if (buf2[i] != (char)i) { + ksft_test_result_fail("%s test: unexpected value at offset %d. Found %d. Should be %d.\n", + test_name, i, buf2[i], (char)i); + return -1; + } + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int test_memcpy_u32(void) +{ + int ret; + uint32_t v1, v2; + const char *test_name = "test_memcpy_u32"; + + /* Test memcpy_u32 */ + v1 = 42; + v2 = 0; + ret = test_memcpy_op(&v2, &v1, sizeof(v1)); + if (ret) { + ksft_test_result_fail("%s test: returned with %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (v1 != v2) { + ksft_test_result_fail("%s test: unexpected value %d. Should be %d.\n", + test_name, v2, v1); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int test_memcpy_memcpy_release_op(void *dst1, void *src1, + void *dst2, void *src2, size_t len) +{ + struct cpu_op opvec[] = { + [0] = { + .op = CPU_MEMCPY_OP, + .len = len, + .u.memcpy_op.dst = (unsigned long)dst1, + .u.memcpy_op.src = (unsigned long)src1, + .u.memcpy_op.expect_fault_dst = 0, + .u.memcpy_op.expect_fault_src = 0, + }, + [1] = { + .op = CPU_MEMCPY_RELEASE_OP, + .len = len, + .u.memcpy_op.dst = (unsigned long)dst2, + .u.memcpy_op.src = (unsigned long)src2, + .u.memcpy_op.expect_fault_dst = 0, + .u.memcpy_op.expect_fault_src = 0, + }, + }; + int ret, cpu; + + do { + cpu = cpu_op_get_current_cpu(); + ret = cpu_opv(opvec, ARRAY_SIZE(opvec), cpu, 0); + } while (ret == -1 && errno == EAGAIN); + + return ret; +} + +static int test_memcpy_memcpy_release(void) +{ + int ret; + int v1, v2, v3; + const char *test_name = "test_memcpy_memcpy_release"; + + /* Test memcpy */ + v1 = 42; + v2 = v3 = 0; + ret = test_memcpy_memcpy_release_op(&v2, &v1, &v3, &v2, sizeof(int)); + if (ret) { + ksft_test_result_fail("%s test: returned with %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (v3 != v1) { + ksft_test_result_fail("%s test: unexpected value %d. Should be %d.\n", + test_name, v3, v1); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int test_add_op(int *v, int64_t increment) +{ + int ret, cpu; + + do { + cpu = cpu_op_get_current_cpu(); + ret = cpu_op_add(v, increment, sizeof(*v), cpu); + } while (ret == -1 && errno == EAGAIN); + + return ret; +} + +static int test_add(void) +{ + int orig_v = 42, v, ret; + int increment = 1; + const char *test_name = "test_add"; + + v = orig_v; + ret = test_add_op(&v, increment); + if (ret) { + ksft_test_result_fail("%s test: returned with %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (v != orig_v + increment) { + ksft_test_result_fail("%s test: unexpected value %d. Should be %d.\n", + test_name, v, + orig_v + increment); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int test_add_release_op(int *v, int64_t increment) +{ + int ret, cpu; + + do { + cpu = cpu_op_get_current_cpu(); + ret = cpu_op_add_release(v, increment, sizeof(*v), cpu); + } while (ret == -1 && errno == EAGAIN); + + return ret; +} + +static int test_add_release(void) +{ + int orig_v = 42, v, ret; + int increment = 1; + const char *test_name = "test_add_release"; + + v = orig_v; + ret = test_add_release_op(&v, increment); + if (ret) { + ksft_test_result_fail("%s test: returned with %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (v != orig_v + increment) { + ksft_test_result_fail("%s test: unexpected value %d. Should be %d.\n", + test_name, v, + orig_v + increment); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int test_two_add_op(int *v, int64_t *increments) +{ + struct cpu_op opvec[] = { + [0] = { + .op = CPU_ADD_OP, + .len = sizeof(*v), + .u.arithmetic_op.p = (unsigned long)v, + .u.arithmetic_op.count = increments[0], + .u.arithmetic_op.expect_fault_p = 0, + }, + [1] = { + .op = CPU_ADD_OP, + .len = sizeof(*v), + .u.arithmetic_op.p = (unsigned long)v, + .u.arithmetic_op.count = increments[1], + .u.arithmetic_op.expect_fault_p = 0, + }, + }; + int ret, cpu; + + do { + cpu = cpu_op_get_current_cpu(); + ret = cpu_opv(opvec, ARRAY_SIZE(opvec), cpu, 0); + } while (ret == -1 && errno == EAGAIN); + + return ret; +} + +static int test_two_add(void) +{ + int orig_v = 42, v, ret; + int64_t increments[2] = { 99, 123 }; + const char *test_name = "test_two_add"; + + v = orig_v; + ret = test_two_add_op(&v, increments); + if (ret) { + ksft_test_result_fail("%s test: returned with %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (v != orig_v + increments[0] + increments[1]) { + ksft_test_result_fail("%s test: unexpected value %d. Should be %d.\n", + test_name, v, + orig_v + increments[0] + increments[1]); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int test_cmpxchg_op(void *v, void *expect, void *old, void *n, + size_t len) +{ + int ret, cpu; + + do { + cpu = cpu_op_get_current_cpu(); + ret = cpu_op_cmpxchg(v, expect, old, n, len, cpu); + } while (ret == -1 && errno == EAGAIN); + + return ret; +} + +static int test_cmpxchg_success(void) +{ + int ret; + uint64_t orig_v = 1, v, expect = 1, old = 0, n = 3; + const char *test_name = "test_cmpxchg success"; + + v = orig_v; + ret = test_cmpxchg_op(&v, &expect, &old, &n, sizeof(uint64_t)); + if (ret < 0) { + ksft_test_result_fail("%s test: ret = %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (ret) { + ksft_test_result_fail("%s returned %d, expecting %d\n", + test_name, ret, 0); + return -1; + } + if (v != n) { + ksft_test_result_fail("%s v is %lld, expecting %lld\n", + test_name, (long long)v, (long long)n); + return -1; + } + if (old != orig_v) { + ksft_test_result_fail("%s old is %lld, expecting %lld\n", + test_name, (long long)old, + (long long)orig_v); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int test_cmpxchg_fail(void) +{ + int ret; + uint64_t orig_v = 1, v, expect = 123, old = 0, n = 3; + const char *test_name = "test_cmpxchg fail"; + + v = orig_v; + ret = test_cmpxchg_op(&v, &expect, &old, &n, sizeof(uint64_t)); + if (ret < 0) { + ksft_test_result_fail("%s test: ret = %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (ret == 0) { + ksft_test_result_fail("%s returned %d, expecting %d\n", + test_name, ret, 1); + return -1; + } + if (v == n) { + ksft_test_result_fail("%s returned %lld, expecting %lld\n", + test_name, (long long)v, + (long long)orig_v); + return -1; + } + if (old != orig_v) { + ksft_test_result_fail("%s old is %lld, expecting %lld\n", + test_name, (long long)old, + (long long)orig_v); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int test_memcpy_expect_fault_op(void *dst, void *src, size_t len) +{ + struct cpu_op opvec[] = { + [0] = { + .op = CPU_MEMCPY_OP, + .len = len, + .u.memcpy_op.dst = (unsigned long)dst, + .u.memcpy_op.src = (unsigned long)src, + .u.memcpy_op.expect_fault_dst = 0, + /* Return EAGAIN on fault. */ + .u.memcpy_op.expect_fault_src = 1, + }, + }; + int cpu; + + cpu = cpu_op_get_current_cpu(); + return cpu_opv(opvec, ARRAY_SIZE(opvec), cpu, 0); +} + +static int test_memcpy_fault(void) +{ + int ret; + char buf1[TESTBUFLEN]; + const char *test_name = "test_memcpy_fault"; + + /* Test memcpy */ + ret = test_memcpy_op(buf1, NULL, TESTBUFLEN); + if (!ret || (ret < 0 && errno != EFAULT)) { + ksft_test_result_fail("%s test: ret = %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + /* Test memcpy expect fault */ + ret = test_memcpy_expect_fault_op(buf1, NULL, TESTBUFLEN); + if (!ret || (ret < 0 && errno != EAGAIN)) { + ksft_test_result_fail("%s test: ret = %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int do_test_unknown_op(void) +{ + struct cpu_op opvec[] = { + [0] = { + .op = -1, /* Unknown */ + .len = 0, + }, + }; + int cpu; + + cpu = cpu_op_get_current_cpu(); + return cpu_opv(opvec, ARRAY_SIZE(opvec), cpu, 0); +} + +static int test_unknown_op(void) +{ + int ret; + const char *test_name = "test_unknown_op"; + + ret = do_test_unknown_op(); + if (!ret || (ret < 0 && errno != EINVAL)) { + ksft_test_result_fail("%s test: ret = %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int do_test_max_ops(void) +{ + int v = 0; + struct cpu_op opvec[] = { + [0] = { + .op = CPU_ADD_OP, + .len = sizeof(v), + .u.arithmetic_op.p = (unsigned long)&v, + .u.arithmetic_op.count = 0, + .u.arithmetic_op.expect_fault_p = 0, + }, + [1] = { + .op = CPU_ADD_OP, + .len = sizeof(v), + .u.arithmetic_op.p = (unsigned long)&v, + .u.arithmetic_op.count = 0, + .u.arithmetic_op.expect_fault_p = 0, + }, + [2] = { + .op = CPU_ADD_OP, + .len = sizeof(v), + .u.arithmetic_op.p = (unsigned long)&v, + .u.arithmetic_op.count = 0, + .u.arithmetic_op.expect_fault_p = 0, + }, + [3] = { + .op = CPU_ADD_OP, + .len = sizeof(v), + .u.arithmetic_op.p = (unsigned long)&v, + .u.arithmetic_op.count = 0, + .u.arithmetic_op.expect_fault_p = 0, + }, + }; + int cpu; + + cpu = cpu_op_get_current_cpu(); + return cpu_opv(opvec, ARRAY_SIZE(opvec), cpu, 0); +} + +static int test_max_ops(void) +{ + int ret; + const char *test_name = "test_max_ops"; + + ret = do_test_max_ops(); + if (ret < 0) { + ksft_test_result_fail("%s test: ret = %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int do_test_too_many_ops(void) +{ + int v = 0; + struct cpu_op opvec[] = { + [0] = { + .op = CPU_ADD_OP, + .len = sizeof(v), + .u.arithmetic_op.p = (unsigned long)&v, + .u.arithmetic_op.count = 0, + .u.arithmetic_op.expect_fault_p = 0, + }, + [1] = { + .op = CPU_ADD_OP, + .len = sizeof(v), + .u.arithmetic_op.p = (unsigned long)&v, + .u.arithmetic_op.count = 0, + .u.arithmetic_op.expect_fault_p = 0, + }, + [2] = { + .op = CPU_ADD_OP, + .len = sizeof(v), + .u.arithmetic_op.p = (unsigned long)&v, + .u.arithmetic_op.count = 0, + .u.arithmetic_op.expect_fault_p = 0, + }, + [3] = { + .op = CPU_ADD_OP, + .len = sizeof(v), + .u.arithmetic_op.p = (unsigned long)&v, + .u.arithmetic_op.count = 0, + .u.arithmetic_op.expect_fault_p = 0, + }, + [4] = { + .op = CPU_ADD_OP, + .len = sizeof(v), + .u.arithmetic_op.p = (unsigned long)&v, + .u.arithmetic_op.count = 0, + .u.arithmetic_op.expect_fault_p = 0, + }, + }; + int cpu; + + cpu = cpu_op_get_current_cpu(); + return cpu_opv(opvec, ARRAY_SIZE(opvec), cpu, 0); +} + +static int test_too_many_ops(void) +{ + int ret; + const char *test_name = "test_too_many_ops"; + + ret = do_test_too_many_ops(); + if (!ret || (ret < 0 && errno != EINVAL)) { + ksft_test_result_fail("%s test: ret = %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +/* Use 64kB len, largest page size known on Linux. */ +static int test_memcpy_single_too_large(void) +{ + int i, ret; + char buf1[TESTBUFLEN_PAGE_MAX + 1]; + char buf2[TESTBUFLEN_PAGE_MAX + 1]; + const char *test_name = "test_memcpy_single_too_large"; + + /* Test memcpy */ + for (i = 0; i < TESTBUFLEN_PAGE_MAX + 1; i++) + buf1[i] = (char)i; + memset(buf2, 0, TESTBUFLEN_PAGE_MAX + 1); + ret = test_memcpy_op(buf2, buf1, TESTBUFLEN_PAGE_MAX + 1); + if (!ret || (ret < 0 && errno != EINVAL)) { + ksft_test_result_fail("%s test: ret = %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +static int test_memcpy_single_ok_sum_too_large_op(void *dst, void *src, + size_t len) +{ + struct cpu_op opvec[] = { + [0] = { + .op = CPU_MEMCPY_OP, + .len = len, + .u.memcpy_op.dst = (unsigned long)dst, + .u.memcpy_op.src = (unsigned long)src, + .u.memcpy_op.expect_fault_dst = 0, + .u.memcpy_op.expect_fault_src = 0, + }, + [1] = { + .op = CPU_MEMCPY_OP, + .len = len, + .u.memcpy_op.dst = (unsigned long)dst, + .u.memcpy_op.src = (unsigned long)src, + .u.memcpy_op.expect_fault_dst = 0, + .u.memcpy_op.expect_fault_src = 0, + }, + }; + int ret, cpu; + + do { + cpu = cpu_op_get_current_cpu(); + ret = cpu_opv(opvec, ARRAY_SIZE(opvec), cpu, 0); + } while (ret == -1 && errno == EAGAIN); + + return ret; +} + +static int test_memcpy_single_ok_sum_too_large(void) +{ + int i, ret; + char buf1[TESTBUFLEN]; + char buf2[TESTBUFLEN]; + const char *test_name = "test_memcpy_single_ok_sum_too_large"; + + /* Test memcpy */ + for (i = 0; i < TESTBUFLEN; i++) + buf1[i] = (char)i; + memset(buf2, 0, TESTBUFLEN); + ret = test_memcpy_single_ok_sum_too_large_op(buf2, buf1, TESTBUFLEN); + if (!ret || (ret < 0 && errno != EINVAL)) { + ksft_test_result_fail("%s test: ret = %d, errno = %s\n", + test_name, ret, strerror(errno)); + return -1; + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +/* + * Iterate over large uninitialized arrays to trigger page faults. + * This includes reading from zero pages. + */ +int test_page_fault(void) +{ + int ret = 0; + uint64_t i; + const char *test_name = "test_page_fault"; + + for (i = 0; i < NR_PF_ARRAY; i++) { + ret = test_memcpy_op(pf_array_dst[i], + pf_array_src[i], + PF_ARRAY_LEN); + if (ret) { + ksft_test_result_fail("%s test: ret = %d, errno = %s\n", + test_name, ret, strerror(errno)); + return ret; + } + } + ksft_test_result_pass("%s test\n", test_name); + return 0; +} + +/* + * Try to use 2MB huge pages. + */ +int test_hugetlb(void) +{ + int ret = 0; + uint64_t i; + const char *test_name = "test_hugetlb"; + int *dst, *src; + + dst = mmap(NULL, HUGEMAPLEN, PROT_READ | PROT_WRITE, + MAP_HUGETLB | MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); + if (dst == MAP_FAILED) { + switch (errno) { + case ENOMEM: + case ENOENT: + case EINVAL: + ksft_test_result_skip("%s test.\n", test_name); + goto end; + default: + break; + } + perror("mmap"); + abort(); + } + src = mmap(NULL, HUGEMAPLEN, PROT_READ | PROT_WRITE, + MAP_HUGETLB | MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); + if (src == MAP_FAILED) { + if (errno == ENOMEM) { + ksft_test_result_skip("%s test.\n", test_name); + goto unmap_dst; + } + perror("mmap"); + abort(); + } + + /* Read/write from/to huge zero pages. */ + for (i = 0; i < NR_HUGE_ARRAY; i++) { + ret = test_memcpy_op(dst + (i * PF_ARRAY_LEN / sizeof(int)), + src + (i * PF_ARRAY_LEN / sizeof(int)), + PF_ARRAY_LEN); + if (ret) { + ksft_test_result_fail("%s test: ret = %d, errno = %s\n", + test_name, ret, strerror(errno)); + return ret; + } + } + for (i = 0; i < NR_HUGE_ARRAY * (PF_ARRAY_LEN / sizeof(int)); i++) + src[i] = i; + + for (i = 0; i < NR_HUGE_ARRAY; i++) { + ret = test_memcpy_op(dst + (i * PF_ARRAY_LEN / sizeof(int)), + src + (i * PF_ARRAY_LEN / sizeof(int)), + PF_ARRAY_LEN); + if (ret) { + ksft_test_result_fail("%s test: ret = %d, errno = %s\n", + test_name, ret, strerror(errno)); + return ret; + } + } + + for (i = 0; i < NR_HUGE_ARRAY * (PF_ARRAY_LEN / sizeof(int)); i++) { + if (dst[i] != i) { + ksft_test_result_fail("%s mismatch, expect %d, got %d\n", + test_name, i, dst[i]); + return ret; + } + } + + ksft_test_result_pass("%s test\n", test_name); + + if (munmap(src, HUGEMAPLEN)) { + perror("munmap"); + abort(); + } +unmap_dst: + if (munmap(dst, HUGEMAPLEN)) { + perror("munmap"); + abort(); + } +end: + return 0; +} + +static int test_cmpxchg_op_cpu(void *v, void *expect, void *old, void *n, + size_t len, int cpu) +{ + int ret; + + do { + ret = cpu_op_cmpxchg(v, expect, old, n, len, cpu); + } while (ret == -1 && errno == EAGAIN); + + return ret; +} + +static int test_over_possible_cpu(void) +{ + int ret; + uint64_t orig_v = 1, v, expect = 1, old = 0, n = 3; + const char *test_name = "test_over_possible_cpu"; + + v = orig_v; + ret = test_cmpxchg_op_cpu(&v, &expect, &old, &n, sizeof(uint64_t), + 0xFFFFFFFF); + if (ret == 0) { + ksft_test_result_fail("%s test: ret = %d\n", + test_name, ret); + return -1; + } + if (ret < 0 && errno == EINVAL) { + ksft_test_result_pass("%s test\n", test_name); + return 0; + } + ksft_test_result_fail("%s returned %d, errno %s, expecting %d, errno %s\n", + test_name, ret, strerror(errno), + 0, strerror(EINVAL)); + return -1; +} + +static int test_allowed_affinity(void) +{ + int ret; + uint64_t orig_v = 1, v, expect = 1, old = 0, n = 3; + const char *test_name = "test_allowed_affinity"; + cpu_set_t allowed_cpus, cpuset; + + ret = sched_getaffinity(0, sizeof(allowed_cpus), &allowed_cpus); + if (ret) { + ksft_test_result_fail("%s returned %d, errno %s\n", + test_name, ret, strerror(errno)); + return -1; + } + if (!(CPU_ISSET(0, &allowed_cpus) && CPU_ISSET(1, &allowed_cpus))) { + ksft_test_result_skip("%s test. Requiring allowed CPUs 0 and 1.\n", + test_name); + return 0; + } + CPU_ZERO(&cpuset); + CPU_SET(0, &cpuset); + if (sched_setaffinity(0, sizeof(cpuset), &cpuset) != 0) { + ksft_test_result_fail("%s test. Unable to set affinity. errno = %s\n", + test_name, strerror(errno)); + return -1; + } + v = orig_v; + ret = test_cmpxchg_op_cpu(&v, &expect, &old, &n, sizeof(uint64_t), + 1); + if (sched_setaffinity(0, sizeof(allowed_cpus), &allowed_cpus) != 0) { + ksft_test_result_fail("%s test. Unable to set affinity. errno = %s\n", + test_name, strerror(errno)); + return -1; + } + if (ret == 0) { + ksft_test_result_fail("%s test: ret = %d\n", + test_name, ret); + return -1; + } + + if (ret < 0 && errno == EINVAL) { + ksft_test_result_pass("%s test\n", test_name); + return 0; + } + ksft_test_result_fail("%s returned %d, errno %s, expecting %d, errno %s\n", + test_name, ret, strerror(errno), + 0, strerror(EINVAL)); + return -1; +} + +int main(int argc, char **argv) +{ + ksft_print_header(); + + test_ops_supported(); + test_compare_eq_same(); + test_compare_eq_diff(); + test_compare_ne_same(); + test_compare_ne_diff(); + test_2compare_eq_index(); + test_2compare_ne_index(); + test_memcpy(); + test_memcpy_u32(); + test_memcpy_memcpy_release(); + test_add(); + test_add_release(); + test_two_add(); + test_cmpxchg_success(); + test_cmpxchg_fail(); + test_memcpy_fault(); + test_unknown_op(); + test_max_ops(); + test_too_many_ops(); + test_memcpy_single_too_large(); + test_memcpy_single_ok_sum_too_large(); + test_page_fault(); + test_hugetlb(); + test_over_possible_cpu(); + test_allowed_affinity(); + + return ksft_exit_pass(); +} -- 2.11.0

6 years, 2 months

1
0
0 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

Linux-kselftest-mirror