On Wed, Nov 20, 2024 at 10:29 AM Andi Kleen ak@linux.intel.com wrote:
Pasha Tatashin pasha.tatashin@soleen.com writes:
Page Detective is a new kernel debugging tool that provides detailed information about the usage and mapping of physical memory pages.
It is often known that a particular page is corrupted, but it is hard to extract more information about such a page from live system. Examples are:
- Checksum failure during live migration
- Filesystem journal failure
- dump_page warnings on the console log
- Unexcpected segfaults
Page Detective helps to extract more information from the kernel, so it can be used by developers to root cause the associated problem.
It operates through the Linux debugfs interface, with two files: "virt" and "phys".
The "virt" file takes a virtual address and PID and outputs information about the corresponding page.
The "phys" file takes a physical address and outputs information about that page.
The output is presented via kernel log messages (can be accessed with dmesg), and includes information such as the page's reference count, mapping, flags, and memory cgroup. It also shows whether the page is mapped in the kernel page table, and if so, how many times.
A lot of all that is already covered in /proc/kpage{flags,cgroup,count) Also we already have /proc/pid/pagemap to resolve virtual addresses.
At a minimum you need to discuss why these existing mechanisms are not suitable for you and how your new one is better.
Hi Andi,
Thanks for your feedback! I will extend the cover letter in the next version to address your comment about comparing with the existing methods.
We periodically receive rare reports of page corruptions detected through various methods (journaling, live migrations, crashes, etc.) from userland. To effectively root cause these corruptions, we need to automatically and quickly gather comprehensive data about the affected pages from the kernel.
This includes:
- Obtain all metadata associated with a page. - Quickly identify all user processes mapping a given page. - Determine if and where the kernel maps the page, which is also important given the opportunity to remove guest memory from the kernel direct map (as discussed at LPC'24).
We also plan to extend this functionality to include KVM and IOMMU page tables in the future. <pagemap> provides an interface to traversing through user page tables, but the other information cannot be extracted using the existing interfaces.
To ensure data integrity, even when dealing with potential memory corruptions, Page Detective minimizes reliance on kernel data structures. Instead, it leverages direct access to hardware structures like page tables, providing a more reliable view of page mappings.
If something particular is missing perhaps the existing mechanisms can be extended? Outputting in the dmesg seems rather clumpsy for a production mechanism.
I am going to change the output to a file in the next version.
I personally would just use live crash or live gdb on /proc/kcore to get extra information, although I can see that might have races.
For security reasons crash is currently not available on our production fleet machines as it potentially provides access to all kernel memory.
Thank you, Pasha