On Wed 02-09-20 11:53:00, Vlastimil Babka wrote:
On 8/28/20 6:47 PM, Pavel Tatashin wrote:
There appears to be another problem that is related to the cgroup_mutex -> mem_hotplug_lock deadlock described above.
In the original deadlock that I described, the workaround is to replace crash dump from piping to Linux traditional save to files method. However, after trying this workaround, I still observed hardware watchdog resets during machine shutdown.
The new problem occurs for the following reason: upon shutdown systemd calls a service that hot-removes memory, and if hot-removing fails for
Why is that hotremove even needed if we're shutting down? Are there any (virtualization?) platforms where it makes some difference over plain shutdown/restart?
Yes this sounds quite dubious.
some reason systemd kills that service after timeout. However, systemd is never able to kill the service, and we get hardware reset caused by watchdog or a hang during shutdown:
Thread #1: memory hot-remove systemd service Loops indefinitely, because if there is something still to be migrated this loop never terminates. However, this loop can be terminated via signal from systemd after timeout. __offline_pages() do { pfn = scan_movable_pages(pfn, end_pfn); # Returns 0, meaning there is nothing available to # migrate, no page is PageLRU(page) ... ret = walk_system_ram_range(start_pfn, end_pfn - start_pfn, NULL, check_pages_isolated_cb); # Returns -EBUSY, meaning there is at least one PFN that # still has to be migrated. } while (ret);
This shouldn't really happen. What does prevent from this to proceed? Did you manage to catch the specific pfn and what is it used for? start_isolate_page_range and scan_movable_pages should fail if there is any memory that cannot be migrated permanently. This is something that we should focus on when debugging.