On Fri, Mar 22, 2024, Zide Chen wrote:
Currently, the migration worker delays 1-10 us, assuming that one KVM_RUN iteration only takes a few microseconds. But if C-state exit latencies are large enough, for example, hundreds or even thousands of microseconds on server CPUs, it may happen that it's not able to bring the target CPU out of C-state before the migration worker starts to migrate it to the next CPU.
If the system workload is light, most CPUs could be at a certain level of C-state, and the vCPU thread may waste milliseconds before it can actually migrate to a new CPU.
Well fudge. That's definitely not on my bingo sheet.
Thus, the tests may be inefficient in such systems, and in some cases it may fail the migration/KVM_RUN ratio sanity check.
Since we are not able to turn off the cpuidle sub-system in run time, this patch creates an idle thread on every CPU to prevent them from entering C-states.
First off, huge thanks for debugging this! That must have been quite the task (no pun intended).
While spinning up threads on every CPU is a clever way to ensure they don't go into a deep sleep state, I'm not exactly excited about the idea of putting every reachable CPU into a busy loop. And while this doesn't add _that_ much complexity, I'm not sure the benefit (preserving the assert for all systems) is worth it. I also don't want to arbitrarily prevent idle task (as in, the kernel's idle task) interactions. E.g. it's highly (highly) unlikely, but not impossible for there to be a bug that's unique to idle tasks, or C-states, or other edge case.
Are there any metrics/stats that can be (easily) checked to grant an exception to the sanity check? That's a very hand-wavy question, as I'm not even sure what type of stat we'd want to look at. Actual runtime of a task, maybe?
If that's not easy, what if we add an off-by-default command line option to skip the sanity check? I was resistant to simply deleting the assert in the past, but that was mainly because I didn't want to delete it without understanding what was causing problems. That would allow CI environments to opt-out as needed, while still keeping the sanity check alive for enough systems to make it useful.