After a long delay I'm posting next iteration of lockless /proc/pid/maps reading patchset. Differences from v2 [1]: - Add a set of tests concurrently modifying address space and checking for correct reading results; - Use new mmap_lock_speculate_xxx APIs for concurrent change detection and retries; - Add lockless PROCMAP_QUERY execution support;
The new tests are designed to check for any unexpected data tearing while performing some common address space modifications (vma split, resize and remap). Even before these changes, reading /proc/pid/maps might have inconsistent data because the file is read page-by-page with mmap_lock being dropped between the pages. Such tearing is expected and userspace is supposed to deal with that possibility. An example of user-visible inconsistency can be that the same vma is printed twice: once before it was modified and then after the modifications. For example if vma was extended, it might be found and reported twice. Whan is not expected is to see a gap where there should have been a vma both before and after modification. This patchset increases the chances of such tearing, therefore it's event more important now to test for unexpected inconsistencies.
Thanks to Paul McKenney who developed a benchmark to test performance of concurrent reads and updates, we also have data on performance benefits:
The test has a pair of processes scanning /proc/PID/maps, and another process unmapping and remapping 4K pages from a 128MB range of anonymous memory. At the end of each 10-second run, the latency of each mmap() or munmap() operation is measured, and for each run the maximum and mean latency is printed. (Yes, the map/unmap process is started first, its PID is passed to the scanners, and then the map/unmap process waits until both scanners are running before starting its timed test. The scanners keep scanning until the specified /proc/PID/maps file disappears.) In summary, with stock mm, 78% of the runs had maximum latencies in excess of 0.5 milliseconds, and with more then half of the runs' latencies exceeding a full millisecond. In contrast, 98% of the runs with Suren's patch series applied had maximum latencies of less than 0.5 milliseconds. From a median-performance viewpoint, Suren's series also looks good, with stock mm weighing in at 13 microseconds and Suren's series at 10 microseconds, better than a 20% improvement.
[1] https://lore.kernel.org/all/20240123231014.3801041-1-surenb@google.com/
Suren Baghdasaryan (8): selftests/proc: add /proc/pid/maps tearing from vma split test selftests/proc: extend /proc/pid/maps tearing test to include vma resizing selftests/proc: extend /proc/pid/maps tearing test to include vma remapping selftests/proc: test PROCMAP_QUERY ioctl while vma is concurrently modified selftests/proc: add verbose more for tests to facilitate debugging mm: make vm_area_struct anon_name field RCU-safe mm/maps: read proc/pid/maps under RCU mm/maps: execute PROCMAP_QUERY ioctl under RCU
fs/proc/internal.h | 6 + fs/proc/task_mmu.c | 233 +++++- include/linux/mm_inline.h | 28 +- include/linux/mm_types.h | 3 +- mm/madvise.c | 30 +- tools/testing/selftests/proc/proc-pid-vm.c | 793 ++++++++++++++++++++- 6 files changed, 1061 insertions(+), 32 deletions(-)
base-commit: 79f35c4125a9a3fd98efeed4cce1cd7ce5311a44