On Sun, Jul 23, 2023 at 10:19:27AM -0700, Paul E. McKenney wrote:
On Sun, Jul 23, 2023 at 10:50:26AM -0400, Joel Fernandes wrote:
On 7/22/23 13:27, Paul E. McKenney wrote: [..]
OK, if this kernel is non-preemptible, you are not running TREE03, correct?
Next plan of action is to get sched_waking stack traces since I have a very reliable repro of this now.
Too much fun! ;-)
For TREE07 issue, it is actually the schedule_timeout_interruptible(1) in stutter_wait() that is beating up the CPU0 for 4 seconds.
This is very similar to the issue I fixed in New year in d52d3a2bf408 ("torture: Fix hang during kthread shutdown phase")
Agreed, if there are enough kthreads, and all the kthreads are on a single CPU, this could consume that CPU.
Adding a cond_resched() there also did not help.
I think the issue is the stutter thread fails to move spt forward because it does not get CPU time. But spt == 1 should be very brief AFAIU. I was wondering if we could set that to RT.
Or just use a single hrtimer-based wait for each kthread?
[Joel] Yes this might be better, but there's still the issue that spt may not be set back to 0 in some future release where the thread gets starved.
But also maybe the following will cure it like it did for the shutdown issue, giving the stutter thread just enough CPU time to move spt forward.
Now I am trying the following and will let it run while I go do other family related things. ;)
Good point, if this avoids the problem, that gives a strong indication that your hypothesis on the root cause is correct.
[Joel] And the TREE07 issue is gone with that change! So I think I'll roll into a patch and send it to you. But I am also hoping that you are Ok with me setting the stutter thread to RT in addition to the longer schedule_timeout. That's just to make it more robust since I think it is crucial that it does not stutter threads indefinitely due to the scheduler (for any unforeseen reason in the future, such as scheduler issues). And maybe, as a part of that I could also tackle that other TODO item about cleaning up torture_create_kthead() as well to add support to it for setting things to RT and use it for that.
Let me know what you think, thanks!
- Joel