Hi All,
Given my on-going work on large anon folios and contpte mappings, I decided it
would be a good idea to start running mm selftests to help guard against
regressions. However, it soon became clear that I couldn't get the suite to run
cleanly on arm64 with a vanilla v6.5-rc1 kernel (perhaps I'm just doing it
wrong??), so got stuck in a rabbit hole trying to debug and fix all the issues.
Some were down to misconfigurations, but I also found a number of issues with
the tests and even a couple of issues with the kernel.
This series aims to fix (most of) the test issues. It applies on top of
v6.5-rc1.
Reproducing
-----------
What follows is a write up of how I'm running the tests and the results I see
with this series applied. I don't yet have a concrete understanding of all of
the remaining failures. So if anyone has any comments on my setup or reasons for
the test failures it would be great to hear.
Source: v6.5-rc1 + this series + [1] + [2]. [1] is a patch from Florent Revest to
fix mdwe mmap_FIXED tests. [2] is a fix for a regression in the kernel that I
found by running `mlock-random-test` and `mlock2-tests`.
Compile the kernel (on arm64 system):
$ make defconfig
$ ./scripts/config --enable CONFIG_SQUASHFS_LZ4
$ ./scripts/config --enable CONFIG_SQUASHFS_LZO
$ ./scripts/config --enable CONFIG_SQUASHFS_XZ
$ ./scripts/config --enable CONFIG_SQUASHFS_ZSTD
$ ./scripts/config --enable CONFIG_XFS_FS
$ ./scripts/config --enable CONFIG_SYSVIPC
$ ./scripts/config --enable CONFIG_USERFAULTFD
$ ./scripts/config --enable CONFIG_TEST_VMALLOC
$ ./scripts/config --enable CONFIG_GUP_TEST
$ ./scripts/config --enable CONFIG_TRANSPARENT_HUGEPAGE
$ ./scripts/config --enable CONFIG_MEM_SOFT_DIRTY
$ make olddefconfig
$ make -s -j`nproc` Image
(In the above case, I'm building/testing a 4K kernel).
Note that it turns out that arm64 doesn't really support ZONE_DEVICE; Although
it defines ARCH_HAS_PTE_DEVMAP, it can't allocate `struct page`s for arbitrary
physical addresses. This means that the TEST_HMM module causes warnings to be
emitted when initializing because it tries to reserve arbitrary PA range then
requests struct page's for them. I haven't fully investigated this yet, but for
now, I'm just deliverately excluding ZONE_DEVICE, (which TEST_HMM depends upon).
This means that the `hmm-tests` selftest gets skipped at runtime.
Compile the tests:
$ make -j`nproc` headers_install
$ make -C tools/testing/selftests TARGETS=mm install INSTALL_PATH=<path/to/install>
Start a VM running the kernel we just compiled:
$ taskset -c 8-15 qemu-system-aarch64 \
-object memory-backend-file,id=mem0,size=6G,mem-path=/hugetlbfs,merge=off,prealloc=on,host-nodes=0,policy=bind,align=1G \
-object memory-backend-file,id=mem1,size=6G,mem-path=/hugetlbfs,merge=off,prealloc=on,host-nodes=0,policy=bind,align=1G \
-nographic -enable-kvm -machine virt,gic-version=3 -cpu max \
-smp 8 -m 12G \
-numa node,memdev=mem0,cpus=0-3,nodeid=0 \
-numa node,memdev=mem1,cpus=4-7,nodeid=1 \
-drive if=virtio,format=raw,file=ubuntu-22.04.xfs \
-object rng-random,filename=/dev/urandom,id=rng0 \
-device virtio-scsi-pci,id=scsi0 \
-netdev user,id=net0,hostfwd=tcp::8022-:22 \
-device virtio-rng-pci,rng=rng0 \
-device virtio-net-pci,netdev=net0 \
-kernel arch/arm64/boot/Image \
-append "earlycon root=/dev/vda2 secretmem.enable hugepagesz=1G hugepages=0:2,1:2 hugepagesz=32M hugepages=0:2,1:2 default_hugepagesz=2M hugepages=0:64,1:64 hugepagesz=64K hugepages=0:2,1:2"
This starts a VM with 2 numa nodes (needed by ksm and migration tests), with 6G
of memory and 4 CPUs on each node. The kernel command line enables secretmem
(needed for `memfd_secret` test), and preallocates a bunch of huge pages
(divined by reading the comments and source for a bunch of tests that require
huge pages). 128M of the default huge page size, and 4 pages of each of the
other sizes appear to be sufficient. I'm allocating half on each numa node.
Once booted, copy the selftests we just compiled onto it.
On the VM, run the tests:
$ cd path/to/selftests
$ sudo ./run_kselftest.sh
or alternatively:
$ cd path/to/selftests/mm
$ sudo ./run_vmtests.sh
Test Results
------------
TOP-LEVEL SUMMARY: PASS=42 SKIP=4 FAIL=2
Only showing nested tests if they are skipped or failed.
[PASS] hugepage-mmap
[PASS] hugepage-shm
[PASS] map_hugetlb
[PASS] hugepage-mremap
[PASS] hugepage-vmemmap
[PASS] hugetlb-madvise
[PASS] map_fixed_noreplace
[PASS] gup_test -u
[PASS] gup_test -a
[PASS] gup_test -ct -F 0x1 0 19 0x1000
[PASS] gup_longterm
[PASS] uffd-unit-tests
[PASS] uffd-stress anon 20 16
[PASS] uffd-stress hugetlb 128 32
[PASS] uffd-stress hugetlb-private 128 32
[PASS] uffd-stress shmem 20 16
[PASS] uffd-stress shmem-private 20 16
[PASS] compaction_test
[PASS] on-fault-limit
[PASS] map_populate
[PASS] mlock-random-test
[PASS] mlock2-tests
[PASS] mrelease_test
[PASS] mremap_test
[PASS] thuge-gen
[PASS] virtual_address_range
[SKIP] va_high_addr_switch.sh
# 4K kernel does not support big enough VA space for test
[SKIP] test_vmalloc.sh smoke
# Test requires test_vmalloc kernel module which isn't present
[PASS] mremap_dontunmap
[SKIP] test_hmm.sh smoke
# Test requires test_hmm kernel module - see ZONE_DEVICE issue above
[PASS] madv_populate
[PASS] test_softdirty
[SKIP] range is not softdirty
[SKIP] MADV_POPULATE_READ
[SKIP] range is not softdirty
[SKIP] MADV_POPULATE_WRITE
[SKIP] range is softdirty
# All skipped because arm64 does not support soft-dirty
[PASS] memfd_secret
[PASS] ksm_tests -M -p 10
[PASS] ksm_tests -U
[PASS] ksm_tests -Z -p 10 -z 0
[PASS] ksm_tests -Z -p 10 -z 1
[PASS] ksm_tests -N -m 1
[PASS] ksm_tests -N -m 0
[PASS] ksm_functional_tests
[SKIP] test_unmerge_uffd_wp
# UFFD_FEATURE_PAGEFAULT_FLAG_WP not available on arm64
[PASS] ksm_functional_tests
[SKIP] test_unmerge_uffd_wp
# UFFD_FEATURE_PAGEFAULT_FLAG_WP not available on arm64
[SKIP] soft-dirty
# Skipped because arm64 does not support soft-dirty
[FAIL] cow
[FAIL] vmsplice() + unmap in child ... with hugetlb
[FAIL] vmsplice() + unmap in child with mprotect() optimization ... with hugetlb
[FAIL] vmsplice() before fork(), unmap in parent after fork() ... with hugetlb
[FAIL] vmsplice() + unmap in parent after fork() ... with hugetlb
# Above are known issues for vmsplice + hugetlb
# Reproduces on x86
[SKIP] Basic COW after fork() ... with swapped-out, PTE-mapped THP
[SKIP] Basic COW after fork() with mprotect() optimization ... with swapped-out, PTE-mapped THP
[SKIP] vmsplice() + unmap in child ... with swapped-out, PTE-mapped THP
[SKIP] vmsplice() + unmap in child with mprotect() optimization ... with swapped-out, PTE-mapped THP
[SKIP] vmsplice() before fork(), unmap in parent after fork() ... with swapped-out, PTE-mapped THP
[SKIP] vmsplice() + unmap in parent after fork() ... with swapped-out, PTE-mapped THP
[SKIP] R/O-mapping a page registered as iouring fixed buffer ... with swapped-out, PTE-mapped THP
[SKIP] fork() with an iouring fixed buffer ... with swapped-out, PTE-mapped THP
[SKIP] R/O GUP pin on R/O-mapped shared page ... with swapped-out, PTE-mapped THP
[SKIP] R/O GUP-fast pin on R/O-mapped shared page ... with swapped-out, PTE-mapped THP
[SKIP] R/O GUP pin on R/O-mapped previously-shared page ... with swapped-out, PTE-mapped THP
[SKIP] R/O GUP-fast pin on R/O-mapped previously-shared page ... with swapped-out, PTE-mapped THP
[SKIP] R/O GUP pin on R/O-mapped exclusive page ... with swapped-out, PTE-mapped THP
[SKIP] R/O GUP-fast pin on R/O-mapped exclusive page ... with swapped-out, PTE-mapped THP
# Above all skipped due to "MADV_PAGEOUT did not work, is swap enabled?"
# swap is enabled though
# Reproduces on x86
[SKIP] Basic COW after fork() when collapsing after fork() (fully shared)
# MADV_COLLAPSE failed: Invalid argument
[PASS] khugepaged
[PASS] transhuge-stress -d 20
[PASS] split_huge_page_test
[FAIL] migration
[FAIL] migration.shared_anon
# move_pages() reports that the requested page was not migrated
# after a few iterations.
[PASS] mkdirty
[PASS] mdwe_test
[1] https://lore.kernel.org/lkml/20230704153630.1591122-3-revest@chromium.org/
[2] https://lore.kernel.org/linux-mm/20230711175020.4091336-1-Liam.Howlett@orac…
Thanks,
Ryan
Ryan Roberts (9):
selftests: Line buffer test program's stdout
selftests/mm: Give scripts execute permission
selftests/mm: Skip soft-dirty tests on arm64
selftests/mm: Enable mrelease_test for arm64
selftests/mm: Fix thuge-gen test bugs
selftests/mm: va_high_addr_switch should skip unsupported arm64
configs
selftests/mm: Make migration test robust to failure
selftests/mm: Optionally pass duration to transhuge-stress
selftests/mm: Run all tests from run_vmtests.sh
tools/testing/selftests/kselftest/runner.sh | 5 +-
tools/testing/selftests/mm/Makefile | 79 ++++++++++---------
.../selftests/mm/charge_reserved_hugetlb.sh | 0
tools/testing/selftests/mm/check_config.sh | 0
.../selftests/mm/hugetlb_reparenting_test.sh | 0
tools/testing/selftests/mm/madv_populate.c | 18 +++--
tools/testing/selftests/mm/migration.c | 14 +++-
tools/testing/selftests/mm/mrelease_test.c | 1 +
tools/testing/selftests/mm/run_vmtests.sh | 23 ++++++
tools/testing/selftests/mm/settings | 2 +-
tools/testing/selftests/mm/soft-dirty.c | 3 +
tools/testing/selftests/mm/test_hmm.sh | 0
tools/testing/selftests/mm/test_vmalloc.sh | 0
tools/testing/selftests/mm/thuge-gen.c | 4 +-
tools/testing/selftests/mm/transhuge-stress.c | 12 ++-
.../selftests/mm/va_high_addr_switch.c | 3 +-
.../selftests/mm/va_high_addr_switch.sh | 0
tools/testing/selftests/mm/vm_util.c | 17 ++++
tools/testing/selftests/mm/vm_util.h | 1 +
.../selftests/mm/write_hugetlb_memory.sh | 0
20 files changed, 127 insertions(+), 55 deletions(-)
mode change 100644 => 100755 tools/testing/selftests/mm/charge_reserved_hugetlb.sh
mode change 100644 => 100755 tools/testing/selftests/mm/check_config.sh
mode change 100644 => 100755 tools/testing/selftests/mm/hugetlb_reparenting_test.sh
mode change 100644 => 100755 tools/testing/selftests/mm/run_vmtests.sh
mode change 100644 => 100755 tools/testing/selftests/mm/test_hmm.sh
mode change 100644 => 100755 tools/testing/selftests/mm/test_vmalloc.sh
mode change 100644 => 100755 tools/testing/selftests/mm/va_high_addr_switch.sh
mode change 100644 => 100755 tools/testing/selftests/mm/write_hugetlb_memory.sh
--
2.25.1
Hi, Willy, Thomas
Thanks very much for your careful review and great suggestions, now, we
get v4 revision of the arch shrink series [1], it mainly include a new
fixup for -O0 under gcc < 11.1.0, the stackprotector support for
_start_c(), new testcases for startup code and two new test targets.
All of the tests passed or skipped (tinyconfig + few options +
qemu-system) for both -Os and -O0:
arch/board | result
------------|------------
arm/versatilepb | 165 test(s): 158 passed, 7 skipped, 0 failed => status: warning.
arm/vexpress-a9 | 165 test(s): 158 passed, 7 skipped, 0 failed => status: warning.
arm/virt | 165 test(s): 158 passed, 7 skipped, 0 failed => status: warning.
aarch64/virt | 165 test(s): 158 passed, 7 skipped, 0 failed => status: warning.
i386/pc | 165 test(s): 158 passed, 7 skipped, 0 failed => status: warning.
x86_64/pc | 165 test(s): 158 passed, 7 skipped, 0 failed => status: warning.
mipsel/malta | 165 test(s): 158 passed, 7 skipped, 0 failed => status: warning.
loongarch64/virt | 165 test(s): 158 passed, 7 skipped, 0 failed => status: warning.
riscv64/virt | 165 test(s): 158 passed, 7 skipped, 0 failed => status: warning.
s390x/s390-ccw-virtio | 165 test(s): 158 passed, 7 skipped, 0 failed => status: warning.
And more, for both -Os and -O0:
$ for r in run-user run-nolibc-test run-libc-test; do make clean > /dev/null; make $r | grep status; done
165 test(s): 157 passed, 8 skipped, 0 failed => status: warning
165 test(s): 157 passed, 8 skipped, 0 failed => status: warning
165 test(s): 153 passed, 12 skipped, 0 failed => status: warning
// for make run-user, the euid0 and 32bit limit related tests are
// skipped
$ make clean && make run-user
$ grep -i skip run.out
17 chroot_root [SKIPPED]
39 link_dir [SKIPPED]
62 limit_intptr_min_32 [SKIPPED]
63 limit_intptr_max_32 [SKIPPED]
64 limit_uintptr_max_32 [SKIPPED]
65 limit_ptrdiff_min_32 [SKIPPED]
66 limit_ptrdiff_max_32 [SKIPPED]
67 limit_size_max_32 [SKIPPED]
// for run-libc-test, the _auxv variables, euid0, 32bits limit and
// stackprotector related tests are skipped
$ make clean && make run-libc-test
$ grep -i skip run.out
9 environ_auxv [SKIPPED]
10 environ_total [SKIPPED]
12 auxv_addr [SKIPPED]
17 chroot_root [SKIPPED]
39 link_dir [SKIPPED]
62 limit_intptr_min_32 [SKIPPED]
63 limit_intptr_max_32 [SKIPPED]
64 limit_uintptr_max_32 [SKIPPED]
65 limit_ptrdiff_min_32 [SKIPPED]
66 limit_ptrdiff_max_32 [SKIPPED]
67 limit_size_max_32 [SKIPPED]
0 -fstackprotector not supported [SKIPPED]
$ make clean >/dev/null; make run-libc-test CC=/labs/linux-lab/src/examples/musl-install/bin/musl-gcc | grep status
165 test(s): 151 passed, 12 skipped, 2 failed => status: failure
// The failures are expected for musl has disabled both sbrk and brk
// but not the sbrk(0); the _auxv variables, euid0, 32bits limit and
// stackprotector related tests are skipped for musl too
$ grep FAIL -ur run.out
9 sbrk = 1 ENOMEM [FAIL]
10 brk = -1 ENOMEM [FAIL]
$ grep "SKIP" -ur run.out
9 environ_auxv [SKIPPED]
10 environ_total [SKIPPED]
12 auxv_addr [SKIPPED]
17 chroot_root [SKIPPED]
39 link_dir [SKIPPED]
62 limit_intptr_min_32 [SKIPPED]
63 limit_intptr_max_32 [SKIPPED]
64 limit_uintptr_max_32 [SKIPPED]
65 limit_ptrdiff_min_32 [SKIPPED]
66 limit_ptrdiff_max_32 [SKIPPED]
67 limit_size_max_32 [SKIPPED]
0 -fstackprotector not supported [SKIPPED]
For stackprotector, gcc 13.1.0 is used to test on x86_64 standalonely:
$ make run-user CROSS_COMPILE=x86_64-linux- | grep status
165 test(s): 157 passed, 8 skipped, 0 failed => status: warning
$ grep stack -ur run.out
0 -fstackprotector [OK]
$ make run-nolibc-test CROSS_COMPILE=x86_64-linux- | grep status
165 test(s): 157 passed, 8 skipped, 0 failed => status: warning
$ grep stack -ur run.out
0 -fstackprotector [OK]
Changes from v3 --> v4:
* tools/nolibc: arch-*.h: add missing space after ','
tools/nolibc: fix up startup failures for -O0 under gcc < 11.1.0
Both of the above changes are for _start, it is able to merge them
if necessary.
The first one is old for format errors reported by
scripts/checkpatch.pl
The second one is for -O0 failure under gcc < 11.1.0, applied the
optimize("-Os", "omit-frame-pointer") suggestion from Thomas.
* tools/nolibc: remove the old sys_stat support
As suggested by Willy, Document carefully about the statx supported
Linux version info.
* tools/nolibc: add new crt.h with _start_c
The code is polished carefully for smaller size and better
readability.
* tools/nolibc: stackprotector.h: add empty __stack_chk_init for !_NOLIBC_STACKPROTECTOR
tools/nolibc: crt.h: initialize stack protector
As suggested by Thomas, init stackprotector in _start_c() too.
* tools/nolibc: arm: shrink _start with _start_c
tools/nolibc: aarch64: shrink _start with _start_c
tools/nolibc: i386: shrink _start with _start_c
tools/nolibc: x86_64: shrink _start with _start_c
tools/nolibc: mips: shrink _start with _start_c
tools/nolibc: loongarch: shrink _start with _start_c
tools/nolibc: riscv: shrink _start with _start_c
tools/nolibc: s390: shrink _start with _start_c
Removed the stackprotector initialization from _start too, we
already have it in _start_c().
* selftests/nolibc: add EXPECT_PTRGE, EXPECT_PTRGT, EXPECT_PTRLE, EXPECT_PTRLT
selftests/nolibc: add testcases for startup code
Add a new startup test group to cover the testing of argc,
argv/argv0, envp/environ and _auxv.
Some testcases are enhanced, some are newly added from after the
discussion during v3 review.
* selftests/nolibc: allow run nolibc-test locally
selftests/nolibc: allow test -include /path/to/nolibc.h
Two new test targets are added to cover more scenes.
Hope you like this revisoin ;-)
Next patchset is powerpc & powerpc64 support, after that we will send
the v2 of tinyconfig support, at last the left rv32 patches (mainly
64bit time).
Best regards,
Zhangjin
---
[1]: https://lore.kernel.org/lkml/20230715100134.GD24086@1wt.eu/
Zhangjin Wu (18):
tools/nolibc: arch-*.h: add missing space after ','
tools/nolibc: fix up startup failures for -O0 under gcc < 11.1.0
tools/nolibc: remove the old sys_stat support
tools/nolibc: add new crt.h with _start_c
tools/nolibc: stackprotector.h: add empty __stack_chk_init for
!_NOLIBC_STACKPROTECTOR
tools/nolibc: crt.h: initialize stack protector
tools/nolibc: arm: shrink _start with _start_c
tools/nolibc: aarch64: shrink _start with _start_c
tools/nolibc: i386: shrink _start with _start_c
tools/nolibc: x86_64: shrink _start with _start_c
tools/nolibc: mips: shrink _start with _start_c
tools/nolibc: loongarch: shrink _start with _start_c
tools/nolibc: riscv: shrink _start with _start_c
tools/nolibc: s390: shrink _start with _start_c
selftests/nolibc: add EXPECT_PTRGE, EXPECT_PTRGT, EXPECT_PTRLE,
EXPECT_PTRLT
selftests/nolibc: add testcases for startup code
selftests/nolibc: allow run nolibc-test locally
selftests/nolibc: allow test -include /path/to/nolibc.h
tools/include/nolibc/Makefile | 1 +
tools/include/nolibc/arch-aarch64.h | 57 +---------
tools/include/nolibc/arch-arm.h | 83 ++-------------
tools/include/nolibc/arch-i386.h | 62 ++---------
tools/include/nolibc/arch-loongarch.h | 46 +-------
tools/include/nolibc/arch-mips.h | 76 ++-----------
tools/include/nolibc/arch-riscv.h | 69 ++----------
tools/include/nolibc/arch-s390.h | 63 ++---------
tools/include/nolibc/arch-x86_64.h | 58 ++--------
tools/include/nolibc/crt.h | 61 +++++++++++
tools/include/nolibc/stackprotector.h | 2 +
tools/include/nolibc/sys.h | 63 ++---------
tools/include/nolibc/types.h | 4 +-
tools/testing/selftests/nolibc/Makefile | 12 +++
tools/testing/selftests/nolibc/nolibc-test.c | 106 ++++++++++++++++++-
15 files changed, 246 insertions(+), 517 deletions(-)
create mode 100644 tools/include/nolibc/crt.h
--
2.25.1