From: "Paul E. McKenney" paulmck@kernel.org

This commit adds a test_boost_holdoff module parameter that tells the RCU priority-boosting tests to wait for the specified number of seconds past the start of the rcutorture test. This can be useful when rcutorture is built into the kernel (as opposed to being modprobed), especially on large systems where early start of RCU priority boosting can delay the boot sequence, which adds a full CPU's worth of load onto the system. This can in turn result in pointless stall warnings.

Signed-off-by: Paul E. McKenney paulmck@kernel.org Signed-off-by: Boqun Feng boqun.feng@gmail.com --- .../admin-guide/kernel-parameters.txt | 5 +++++ kernel/rcu/rcutorture.c | 19 ++++++++++++++++--- 2 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index fb8752b42ec8..ed1a0df03b18 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -5758,6 +5758,11 @@ rcutorture.test_boost_duration= [KNL] Duration (s) of each individual boost test.

+ rcutorture.test_boost_holdoff= [KNL] + Holdoff time (s) from start of test to the start + of RCU priority-boost testing. Defaults to zero, + that is, no holdoff. + rcutorture.test_boost_interval= [KNL] Interval (s) between each boost test.

diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index d26fb1d33ed9..fbf1d7fcf61d 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -135,6 +135,7 @@ torture_param(int, stat_interval, 60, "Number of seconds between stats printk()s torture_param(int, stutter, 5, "Number of seconds to run/halt test"); torture_param(int, test_boost, 1, "Test RCU prio boost: 0=no, 1=maybe, 2=yes."); torture_param(int, test_boost_duration, 4, "Duration of each boost test, seconds."); +torture_param(int, test_boost_holdoff, 0, "Holdoff time from rcutorture start, seconds."); torture_param(int, test_boost_interval, 7, "Interval between boost tests, seconds."); torture_param(int, test_nmis, 0, "End-test NMI tests, 0 to disable."); torture_param(bool, test_no_idle_hz, true, "Test support for tickless idle CPUs"); @@ -1148,8 +1149,19 @@ static int rcu_torture_boost(void *arg) unsigned long gp_state; unsigned long gp_state_time; unsigned long oldstarttime; + unsigned long booststarttime = get_torture_init_jiffies() + test_boost_holdoff * HZ;

- VERBOSE_TOROUT_STRING("rcu_torture_boost started"); + if (test_boost_holdoff <= 0 || time_after(jiffies, booststarttime)) { + VERBOSE_TOROUT_STRING("rcu_torture_boost started"); + } else { + VERBOSE_TOROUT_STRING("rcu_torture_boost started holdoff period"); + while (time_before(jiffies, booststarttime)) { + schedule_timeout_idle(HZ); + if (kthread_should_stop()) + goto cleanup; + } + VERBOSE_TOROUT_STRING("rcu_torture_boost finished holdoff period"); + }

/* Set real-time priority. */ sched_set_fifo_low(current); @@ -1225,6 +1237,7 @@ checkwait: if (stutter_wait("rcu_torture_boost")) sched_set_fifo_low(current); } while (!torture_must_stop());

+cleanup: /* Clean up and exit. */ while (!kthread_should_stop()) { torture_shutdown_absorb("rcu_torture_boost"); @@ -2512,7 +2525,7 @@ rcu_torture_print_module_parms(struct rcu_torture_ops *cur_ops, const char *tag) "shuffle_interval=%d stutter=%d irqreader=%d " "fqs_duration=%d fqs_holdoff=%d fqs_stutter=%d " "test_boost=%d/%d test_boost_interval=%d " - "test_boost_duration=%d shutdown_secs=%d " + "test_boost_duration=%d test_boost_holdoff=%d shutdown_secs=%d " "stall_cpu=%d stall_cpu_holdoff=%d stall_cpu_irqsoff=%d " "stall_cpu_block=%d stall_cpu_repeat=%d " "n_barrier_cbs=%d " @@ -2526,7 +2539,7 @@ rcu_torture_print_module_parms(struct rcu_torture_ops *cur_ops, const char *tag) stat_interval, verbose, test_no_idle_hz, shuffle_interval, stutter, irqreader, fqs_duration, fqs_holdoff, fqs_stutter, test_boost, cur_ops->can_boost, - test_boost_interval, test_boost_duration, shutdown_secs, + test_boost_interval, test_boost_duration, test_boost_holdoff, shutdown_secs, stall_cpu, stall_cpu_holdoff, stall_cpu_irqsoff, stall_cpu_block, stall_cpu_repeat, n_barrier_cbs,

From: "Paul E. McKenney" paulmck@kernel.org

With only eight bits per grace-period sequence number, wrap can happen in 64 grace periods. This commit therefore increases this to sixteen bits for normal grace-period sequence numbers and the combined short-form polling sequence numbers, thus deferring wrap for at least 16,384 grace periods. Because expedited grace periods go faster, expand these to 24 bits, deferring wrap for at least 4,194,304 expedited grace periods. These longer wrap times makes it easier to correlate these numbers to trace-event output.

Note that the low-order two bits are reserved for intra-grace-period state, hence the above wrap numbers being a factor of four smaller than you might expect.

Signed-off-by: Paul E. McKenney paulmck@kernel.org Signed-off-by: Boqun Feng boqun.feng@gmail.com --- kernel/rcu/rcu.h | 4 ++-- kernel/rcu/rcutorture.c | 12 ++++++------ kernel/rcu/tiny.c | 8 ++++---- kernel/rcu/tree.c | 18 +++++++++--------- 4 files changed, 21 insertions(+), 21 deletions(-)

diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index a6098997a14b..705fcbe6f500 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -590,8 +590,8 @@ void do_trace_rcu_torture_read(const char *rcutorturename, #endif static inline void rcu_gp_set_torture_wait(int duration) { } #endif -unsigned long rcutorture_gather_gp_seqs(void); -void rcutorture_format_gp_seqs(unsigned long seqs, char *cp); +unsigned long long rcutorture_gather_gp_seqs(void); +void rcutorture_format_gp_seqs(unsigned long long seqs, char *cp);

#ifdef CONFIG_TINY_SRCU

diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 2113583cae34..fb1b80498ae0 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -273,8 +273,8 @@ struct rt_read_seg { bool rt_preempted; int rt_cpu; int rt_end_cpu; - unsigned long rt_gp_seq; - unsigned long rt_gp_seq_end; + unsigned long long rt_gp_seq; + unsigned long long rt_gp_seq_end; }; static int err_segs_recorded; static struct rt_read_seg err_segs[RCUTORTURE_RDR_MAX_SEGS]; @@ -409,8 +409,8 @@ struct rcu_torture_ops { void (*gp_slow_register)(atomic_t *rgssp); void (*gp_slow_unregister)(atomic_t *rgssp); bool (*reader_blocked)(void); - unsigned long (*gather_gp_seqs)(void); - void (*format_gp_seqs)(unsigned long seqs, char *cp); + unsigned long long (*gather_gp_seqs)(void); + void (*format_gp_seqs)(unsigned long long seqs, char *cp); long cbflood_max; int irq_capable; int can_boost; @@ -3678,8 +3678,8 @@ rcu_torture_cleanup(void) } if (IS_ENABLED(CONFIG_RCU_TORTURE_TEST_LOG_GP) && cur_ops->gather_gp_seqs && cur_ops->format_gp_seqs) { - char buf1[16+1]; - char buf2[16+1]; + char buf1[20+1]; + char buf2[20+1]; char sepchar = '-';

cur_ops->format_gp_seqs(err_segs[i].rt_gp_seq, buf1); diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index f9c4a24dc59c..8cbec3401184 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -258,15 +258,15 @@ EXPORT_SYMBOL_GPL(kvfree_call_rcu); #endif

#if IS_ENABLED(CONFIG_RCU_TORTURE_TEST) -unsigned long rcutorture_gather_gp_seqs(void) +unsigned long long rcutorture_gather_gp_seqs(void) { - return READ_ONCE(rcu_ctrlblk.gp_seq) & 0xff; + return READ_ONCE(rcu_ctrlblk.gp_seq) & 0xffffULL; } EXPORT_SYMBOL_GPL(rcutorture_gather_gp_seqs);

-void rcutorture_format_gp_seqs(unsigned long seqs, char *cp) +void rcutorture_format_gp_seqs(unsigned long long seqs, char *cp) { - snprintf(cp, 8, "g%02lx", seqs & 0xff); + snprintf(cp, 8, "g%04llx", seqs & 0xffffULL); } EXPORT_SYMBOL_GPL(rcutorture_format_gp_seqs); #endif diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index e40c4b5c3267..83cba3d2cc48 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -539,22 +539,22 @@ void rcutorture_get_gp_data(int *flags, unsigned long *gp_seq) EXPORT_SYMBOL_GPL(rcutorture_get_gp_data);

/* Gather grace-period sequence numbers for rcutorture diagnostics. */ -unsigned long rcutorture_gather_gp_seqs(void) +unsigned long long rcutorture_gather_gp_seqs(void) { - return ((READ_ONCE(rcu_state.gp_seq) & 0xff) << 16) | - ((READ_ONCE(rcu_state.expedited_sequence) & 0xff) << 8) | - (READ_ONCE(rcu_state.gp_seq_polled) & 0xff); + return ((READ_ONCE(rcu_state.gp_seq) & 0xffffULL) << 40) | + ((READ_ONCE(rcu_state.expedited_sequence) & 0xffffffULL) << 16) | + (READ_ONCE(rcu_state.gp_seq_polled) & 0xffffULL); } EXPORT_SYMBOL_GPL(rcutorture_gather_gp_seqs);

/* Format grace-period sequence numbers for rcutorture diagnostics. */ -void rcutorture_format_gp_seqs(unsigned long seqs, char *cp) +void rcutorture_format_gp_seqs(unsigned long long seqs, char *cp) { - unsigned int egp = (seqs >> 8) & 0xff; - unsigned int ggp = (seqs >> 16) & 0xff; - unsigned int pgp = seqs & 0xff; + unsigned int egp = (seqs >> 16) & 0xffffffULL; + unsigned int ggp = (seqs >> 40) & 0xffffULL; + unsigned int pgp = seqs & 0xffffULL;

- snprintf(cp, 16, "g%02x:e%02x:p%02x", ggp, egp, pgp); + snprintf(cp, 20, "g%04x:e%06x:p%04x", ggp, egp, pgp); } EXPORT_SYMBOL_GPL(rcutorture_format_gp_seqs);

From: "Paul E. McKenney" paulmck@kernel.org

This commit adds an ftrace-compatible microsecond-scale timestamp to the failure/close-call output, but only in kernels built with CONFIG_RCU_TORTURE_TEST_LOG_GP=y.

Signed-off-by: Paul E. McKenney paulmck@kernel.org Signed-off-by: Boqun Feng boqun.feng@gmail.com --- kernel/rcu/rcutorture.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index fb1b80498ae0..1fdadc1df9ad 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -275,6 +275,7 @@ struct rt_read_seg { int rt_end_cpu; unsigned long long rt_gp_seq; unsigned long long rt_gp_seq_end; + u64 rt_ts; }; static int err_segs_recorded; static struct rt_read_seg err_segs[RCUTORTURE_RDR_MAX_SEGS]; @@ -1989,6 +1990,7 @@ static void rcutorture_one_extend(int *readstate, int newstate, bool insoftirq, // Sample grace-period sequence number, as good a place as any. if (IS_ENABLED(CONFIG_RCU_TORTURE_TEST_LOG_GP) && cur_ops->gather_gp_seqs) { rtrsp->rt_gp_seq = cur_ops->gather_gp_seqs(); + rtrsp->rt_ts = ktime_get_mono_fast_ns(); if (!first) rtrsp[-1].rt_gp_seq_end = rtrsp->rt_gp_seq; } @@ -3663,7 +3665,11 @@ rcu_torture_cleanup(void) pr_alert("\t: No segments recorded!!!\n"); firsttime = 1; for (i = 0; i < rt_read_nsegs; i++) { - pr_alert("\t%d: %#4x", i, err_segs[i].rt_readstate); + if (IS_ENABLED(CONFIG_RCU_TORTURE_TEST_LOG_GP)) + pr_alert("\t%lluus ", div64_u64(err_segs[i].rt_ts, 1000ULL)); + else + pr_alert("\t"); + pr_cont("%d: %#4x", i, err_segs[i].rt_readstate); if (err_segs[i].rt_delay_jiffies != 0) { pr_cont("%s%ldjiffies", firsttime ? "" : "+", err_segs[i].rt_delay_jiffies);

From: "Paul E. McKenney" paulmck@kernel.org

The RCU_TORTURE_TEST_CHK_RDR_STATE and RCU_TORTURE_TEST_LOG_CPU Kconfig options are pointlessly defined as tristate. This commit therefore converts them to bool.

Reported-by: kernel test robot oliver.sang@intel.com Closes: https://lore.kernel.org/oe-lkp/202412241458.150d082b-lkp@intel.com Signed-off-by: Paul E. McKenney paulmck@kernel.org Signed-off-by: Boqun Feng boqun.feng@gmail.com --- kernel/rcu/Kconfig.debug | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/kernel/rcu/Kconfig.debug b/kernel/rcu/Kconfig.debug index 25a9dc2be0dc..12e4c64ebae1 100644 --- a/kernel/rcu/Kconfig.debug +++ b/kernel/rcu/Kconfig.debug @@ -54,7 +54,7 @@ config RCU_TORTURE_TEST Say N if you are unsure.

config RCU_TORTURE_TEST_CHK_RDR_STATE - tristate "Check rcutorture reader state" + bool "Check rcutorture reader state" depends on RCU_TORTURE_TEST default n help @@ -70,7 +70,7 @@ config RCU_TORTURE_TEST_CHK_RDR_STATE Say N if you are unsure.

config RCU_TORTURE_TEST_LOG_CPU - tristate "Log CPU for rcutorture failures" + bool "Log CPU for rcutorture failures" depends on RCU_TORTURE_TEST default n help

From: "Paul E. McKenney" paulmck@kernel.org

The srcu_read_lock_nmisafe() and srcu_read_unlock_nmisafe() functions map to __srcu_read_lock() and __srcu_read_unlock() on systems like x86 that have NMI-safe this_cpu_inc() operations. This makes the underlying __srcu_read_lock_nmisafe() and __srcu_read_unlock_nmisafe() functions difficult to test on (for example) x86 systems, allowing bugs to creep in.

This commit therefore creates a FORCE_NEED_SRCU_NMI_SAFE Kconfig that forces those underlying functions to be used even on systems where they are not needed, thus providing better testing coverage.

Signed-off-by: Paul E. McKenney paulmck@kernel.org Signed-off-by: Boqun Feng boqun.feng@gmail.com --- kernel/rcu/Kconfig | 11 +++++++++++ 1 file changed, 11 insertions(+)

diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index b9b6bc55185d..c8e540af3a35 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -65,6 +65,17 @@ config TREE_SRCU help This option selects the full-fledged version of SRCU.

+config FORCE_NEED_SRCU_NMI_SAFE + bool "Force selection of NEED_SRCU_NMI_SAFE" + depends on !TINY_SRCU + select NEED_SRCU_NMI_SAFE + default n + help + This option forces selection of the NEED_SRCU_NMI_SAFE + Kconfig option, allowing testing of srcu_read_lock_nmisafe() + and srcu_read_unlock_nmisafe() on architectures (like x86) + that select the ARCH_HAS_NMI_SAFE_THIS_CPU_OPS Kconfig option. + config NEED_SRCU_NMI_SAFE def_bool HAVE_NMI && !ARCH_HAS_NMI_SAFE_THIS_CPU_OPS && !TINY_SRCU

From: "Paul E. McKenney" paulmck@kernel.org

Recent experience shows that the srcu_read_lock_nmisafe() and srcu_read_unlock_nmisafe() functions are not sufficiently tested. This commit therefore causes the torture.sh script's SRCU lockdep testing to use these two functions. This will cause these two functions to be regularly tested by several developers (myself included) who use torture.sh as an RCU acceptance test.

Signed-off-by: Paul E. McKenney paulmck@kernel.org Signed-off-by: Boqun Feng boqun.feng@gmail.com --- tools/testing/selftests/rcutorture/bin/srcu_lockdep.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/testing/selftests/rcutorture/bin/srcu_lockdep.sh b/tools/testing/selftests/rcutorture/bin/srcu_lockdep.sh index 2e63ef009d59..2db12c5cad9c 100755 --- a/tools/testing/selftests/rcutorture/bin/srcu_lockdep.sh +++ b/tools/testing/selftests/rcutorture/bin/srcu_lockdep.sh @@ -49,7 +49,7 @@ do do err= val=$((d*1000+t*10+c)) - tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 5s --configs "SRCU-P" --bootargs "rcutorture.test_srcu_lockdep=$val" --trust-make --datestamp "$ds/$val" > "$T/kvm.sh.out" 2>&1 + tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 5s --configs "SRCU-P" --kconfig "CONFIG_FORCE_NEED_SRCU_NMI_SAFE=y" --bootargs "rcutorture.test_srcu_lockdep=$val rcutorture.reader_flavor=0x2" --trust-make --datestamp "$ds/$val" > "$T/kvm.sh.out" 2>&1 ret=$? mv "$T/kvm.sh.out" "$RCUTORTURE/res/$ds/$val" if test "$d" -ne 0 && test "$ret" -eq 0