- Linux-kselftest-mirror - lists.linaro.org

[PATCH v2 0/6] kunit: Add dynamically-extending log

by Richard Fitzgerald

Replace the original fixed-size log buffer with a dynamically- extending log. Patch 1 provides the basic implementation. The following patches add test cases, support for logging long strings, and an optimization to the string formatting that is now more thoroughly testable. Richard Fitzgerald (6): kunit: Replace fixed-size log with dynamically-extending buffer kunit: kunit-test: Add test cases for extending log buffer kunit: Handle logging of lines longer than the fragment buffer size kunit: kunit-test: Add test cases for logging very long lines kunit: kunit-test: Add test of logging only a newline kunit: Don't waste first attempt to format string in kunit_log_append() include/kunit/test.h | 25 +++- lib/kunit/debugfs.c | 65 +++++++-- lib/kunit/kunit-test.c | 321 ++++++++++++++++++++++++++++++++++++++++- lib/kunit/test.c | 127 +++++++++++++--- 4 files changed, 489 insertions(+), 49 deletions(-) -- 2.30.2

2 years, 1 month

3
12
0 0

[PATCH v6 00/13] RISCV: Add KVM_GET_REG_LIST API

by Haibo Xu

KVM_GET_REG_LIST will dump all register IDs that are available to KVM_GET/SET_ONE_REG and It's very useful to identify some platform regression issue during VM migration. Patch 1-7 re-structured the get-reg-list test in aarch64 to make some of the code as common test framework that can be shared by riscv. Patch 8 move reject_set check logic to a function so as to check for different errno for different registers. Patch 9 move finalize_vcpu back to run_test so that riscv can implement its specific operation. Patch 10 change to do the get/set operation only on present-blessed list. Patch 11 add the skip_set facilities so that riscv can skip set operation on some registers. Patch 12 enabled the KVM_GET_REG_LIST API in riscv. patch 13 added the corresponding kselftest for checking possible register regressions. The get-reg-list kvm selftest was ported from aarch64 and tested with Linux v6.5-rc3 on a Qemu riscv64 virt machine. --- Changed since v5: * Rebase to v6.5-rc3 * Minor fix for Andrew's comments Andrew Jones (7): KVM: arm64: selftests: Replace str_with_index with strdup_printf KVM: arm64: selftests: Drop SVE cap check in print_reg KVM: arm64: selftests: Remove print_reg's dependency on vcpu_config KVM: arm64: selftests: Rename vcpu_config and add to kvm_util.h KVM: arm64: selftests: Delete core_reg_fixup KVM: arm64: selftests: Split get-reg-list test code KVM: arm64: selftests: Finish generalizing get-reg-list Haibo Xu (6): KVM: arm64: selftests: Move reject_set check logic to a function KVM: arm64: selftests: Move finalize_vcpu back to run_test KVM: selftests: Only do get/set tests on present blessed list KVM: selftests: Add skip_set facility to get_reg_list test KVM: riscv: Add KVM_GET_REG_LIST API support KVM: riscv: selftests: Add get-reg-list test Documentation/virt/kvm/api.rst | 2 +- arch/riscv/kvm/vcpu.c | 375 +++++++++ tools/testing/selftests/kvm/Makefile | 13 +- .../selftests/kvm/aarch64/get-reg-list.c | 554 ++----------- tools/testing/selftests/kvm/get-reg-list.c | 401 +++++++++ .../selftests/kvm/include/kvm_util_base.h | 21 + .../selftests/kvm/include/riscv/processor.h | 3 + .../testing/selftests/kvm/include/test_util.h | 2 + tools/testing/selftests/kvm/lib/test_util.c | 15 + .../selftests/kvm/riscv/get-reg-list.c | 780 ++++++++++++++++++ 10 files changed, 1670 insertions(+), 496 deletions(-) create mode 100644 tools/testing/selftests/kvm/get-reg-list.c create mode 100644 tools/testing/selftests/kvm/riscv/get-reg-list.c -- 2.34.1

2 years, 1 month

5
22
0 0

[PATCH 0/2] x86/BPF: Add new BPF helper call bpf_rdtsc

by Tero Kristo

Hello, This patch series adds a new x86 arch specific BPF helper, bpf_rdtsc() which can be used for reading the hardware time stamp counter (TSC.) Currently the same counter is directly accessible from userspace (using RDTSC instruction), and kernel space using various rdtsc_*() APIs, however eBPF lacks the support. The main usage for the TSC counter is for various profiling and timing purposes, getting accurate cycle counter values. The counter can be currently read from BPF programs by using the existing perf subsystem services (bpf_perf_event_read()), however its usage is cumbersome at best. Additionally, the perf subsystem provides relative value only for the counter, but absolute values are desired by some use cases like Wult [1]. The absolute value of TSC can be read with BPF programs currently via some kprobe / bpf_core_read() magic (see [2], [3], [4] for example), but this relies on accessing kernel internals and is not stable API, and is pretty cumbersome. Thus, this patch proposes a new arch x86 specific BPF helper to avoid the above issues. -Tero [1] https://github.com/intel/wult [2] https://github.com/intel/wult/blob/c92237c95b898498faf41e6644983102d1fe5156… [3] https://github.com/intel/wult/blob/c92237c95b898498faf41e6644983102d1fe5156… [4] https://github.com/intel/wult/blob/c92237c95b898498faf41e6644983102d1fe5156…

2 years, 1 month

4
16
0 0

[PATCH 5.4 134/154] test_firmware: prevent race conditions by a correct implementation of locking

by Greg Kroah-Hartman

From: Mirsad Goran Todorovac <mirsad.todorovac(a)alu.unizg.hr> commit 4acfe3dfde685a5a9eaec5555351918e2d7266a1 upstream. Dan Carpenter spotted a race condition in a couple of situations like these in the test_firmware driver: static int test_dev_config_update_u8(const char *buf, size_t size, u8 *cfg) { u8 val; int ret; ret = kstrtou8(buf, 10, &val); if (ret) return ret; mutex_lock(&test_fw_mutex); *(u8 *)cfg = val; mutex_unlock(&test_fw_mutex); /* Always return full write size even if we didn't consume all */ return size; } static ssize_t config_num_requests_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t count) { int rc; mutex_lock(&test_fw_mutex); if (test_fw_config->reqs) { pr_err("Must call release_all_firmware prior to changing config\n"); rc = -EINVAL; mutex_unlock(&test_fw_mutex); goto out; } mutex_unlock(&test_fw_mutex); rc = test_dev_config_update_u8(buf, count, &test_fw_config->num_requests); out: return rc; } static ssize_t config_read_fw_idx_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t count) { return test_dev_config_update_u8(buf, count, &test_fw_config->read_fw_idx); } The function test_dev_config_update_u8() is called from both the locked and the unlocked context, function config_num_requests_store() and config_read_fw_idx_store() which can both be called asynchronously as they are driver's methods, while test_dev_config_update_u8() and siblings change their argument pointed to by u8 *cfg or similar pointer. To avoid deadlock on test_fw_mutex, the lock is dropped before calling test_dev_config_update_u8() and re-acquired within test_dev_config_update_u8() itself, but alas this creates a race condition. Having two locks wouldn't assure a race-proof mutual exclusion. This situation is best avoided by the introduction of a new, unlocked function __test_dev_config_update_u8() which can be called from the locked context and reducing test_dev_config_update_u8() to: static int test_dev_config_update_u8(const char *buf, size_t size, u8 *cfg) { int ret; mutex_lock(&test_fw_mutex); ret = __test_dev_config_update_u8(buf, size, cfg); mutex_unlock(&test_fw_mutex); return ret; } doing the locking and calling the unlocked primitive, which enables both locked and unlocked versions without duplication of code. The similar approach was applied to all functions called from the locked and the unlocked context, which safely mitigates both deadlocks and race conditions in the driver. __test_dev_config_update_bool(), __test_dev_config_update_u8() and __test_dev_config_update_size_t() unlocked versions of the functions were introduced to be called from the locked contexts as a workaround without releasing the main driver's lock and thereof causing a race condition. The test_dev_config_update_bool(), test_dev_config_update_u8() and test_dev_config_update_size_t() locked versions of the functions are being called from driver methods without the unnecessary multiplying of the locking and unlocking code for each method, and complicating the code with saving of the return value across lock. Fixes: 7feebfa487b92 ("test_firmware: add support for request_firmware_into_buf") Cc: Luis Chamberlain <mcgrof(a)kernel.org> Cc: Greg Kroah-Hartman <gregkh(a)linuxfoundation.org> Cc: Russ Weight <russell.h.weight(a)intel.com> Cc: Takashi Iwai <tiwai(a)suse.de> Cc: Tianfei Zhang <tianfei.zhang(a)intel.com> Cc: Shuah Khan <shuah(a)kernel.org> Cc: Colin Ian King <colin.i.king(a)gmail.com> Cc: Randy Dunlap <rdunlap(a)infradead.org> Cc: linux-kselftest(a)vger.kernel.org Cc: stable(a)vger.kernel.org # v5.4 Suggested-by: Dan Carpenter <error27(a)gmail.com> Signed-off-by: Mirsad Goran Todorovac <mirsad.todorovac(a)alu.unizg.hr> Link: https://lore.kernel.org/r/20230509084746.48259-1-mirsad.todorovac@alu.unizg… Signed-off-by: Greg Kroah-Hartman <gregkh(a)linuxfoundation.org> --- lib/test_firmware.c | 37 ++++++++++++++++++++++++++++--------- 1 file changed, 28 insertions(+), 9 deletions(-) --- a/lib/test_firmware.c +++ b/lib/test_firmware.c @@ -301,16 +301,26 @@ static ssize_t config_test_show_str(char return len; } -static int test_dev_config_update_bool(const char *buf, size_t size, - bool *cfg) +static inline int __test_dev_config_update_bool(const char *buf, size_t size, + bool *cfg) { int ret; - mutex_lock(&test_fw_mutex); if (strtobool(buf, cfg) < 0) ret = -EINVAL; else ret = size; + + return ret; +} + +static int test_dev_config_update_bool(const char *buf, size_t size, + bool *cfg) +{ + int ret; + + mutex_lock(&test_fw_mutex); + ret = __test_dev_config_update_bool(buf, size, cfg); mutex_unlock(&test_fw_mutex); return ret; @@ -340,7 +350,7 @@ static ssize_t test_dev_config_show_int( return snprintf(buf, PAGE_SIZE, "%d\n", val); } -static int test_dev_config_update_u8(const char *buf, size_t size, u8 *cfg) +static inline int __test_dev_config_update_u8(const char *buf, size_t size, u8 *cfg) { int ret; long new; @@ -352,14 +362,23 @@ static int test_dev_config_update_u8(con if (new > U8_MAX) return -EINVAL; - mutex_lock(&test_fw_mutex); *(u8 *)cfg = new; - mutex_unlock(&test_fw_mutex); /* Always return full write size even if we didn't consume all */ return size; } +static int test_dev_config_update_u8(const char *buf, size_t size, u8 *cfg) +{ + int ret; + + mutex_lock(&test_fw_mutex); + ret = __test_dev_config_update_u8(buf, size, cfg); + mutex_unlock(&test_fw_mutex); + + return ret; +} + static ssize_t test_dev_config_show_u8(char *buf, u8 cfg) { u8 val; @@ -392,10 +411,10 @@ static ssize_t config_num_requests_store mutex_unlock(&test_fw_mutex); goto out; } - mutex_unlock(&test_fw_mutex); - rc = test_dev_config_update_u8(buf, count, - &test_fw_config->num_requests); + rc = __test_dev_config_update_u8(buf, count, + &test_fw_config->num_requests); + mutex_unlock(&test_fw_mutex); out: return rc;

2 years, 1 month

1
0
0 0

[PATCH 4.19 312/323] test_firmware: fix a memory leak with reqs buffer

by Greg Kroah-Hartman

From: Mirsad Goran Todorovac <mirsad.todorovac(a)alu.unizg.hr> commit be37bed754ed90b2655382f93f9724b3c1aae847 upstream. Dan Carpenter spotted that test_fw_config->reqs will be leaked if trigger_batched_requests_store() is called two or more times. The same appears with trigger_batched_requests_async_store(). This bug wasn't trigger by the tests, but observed by Dan's visual inspection of the code. The recommended workaround was to return -EBUSY if test_fw_config->reqs is already allocated. Fixes: 7feebfa487b92 ("test_firmware: add support for request_firmware_into_buf") Cc: Luis Chamberlain <mcgrof(a)kernel.org> Cc: Greg Kroah-Hartman <gregkh(a)linuxfoundation.org> Cc: Russ Weight <russell.h.weight(a)intel.com> Cc: Tianfei Zhang <tianfei.zhang(a)intel.com> Cc: Shuah Khan <shuah(a)kernel.org> Cc: Colin Ian King <colin.i.king(a)gmail.com> Cc: Randy Dunlap <rdunlap(a)infradead.org> Cc: linux-kselftest(a)vger.kernel.org Cc: stable(a)vger.kernel.org # v5.4 Suggested-by: Dan Carpenter <error27(a)gmail.com> Suggested-by: Takashi Iwai <tiwai(a)suse.de> Signed-off-by: Mirsad Goran Todorovac <mirsad.todorovac(a)alu.unizg.hr> Reviewed-by: Dan Carpenter <dan.carpenter(a)linaro.org> Acked-by: Luis Chamberlain <mcgrof(a)kernel.org> Link: https://lore.kernel.org/r/20230509084746.48259-2-mirsad.todorovac@alu.unizg… Signed-off-by: Greg Kroah-Hartman <gregkh(a)linuxfoundation.org> Signed-off-by: Greg Kroah-Hartman <gregkh(a)linuxfoundation.org> --- lib/test_firmware.c | 10 ++++++++++ 1 file changed, 10 insertions(+) --- a/lib/test_firmware.c +++ b/lib/test_firmware.c @@ -618,6 +618,11 @@ static ssize_t trigger_batched_requests_ mutex_lock(&test_fw_mutex); + if (test_fw_config->reqs) { + rc = -EBUSY; + goto out_bail; + } + test_fw_config->reqs = vzalloc(array3_size(sizeof(struct test_batched_req), test_fw_config->num_requests, 2)); @@ -721,6 +726,11 @@ ssize_t trigger_batched_requests_async_s mutex_lock(&test_fw_mutex); + if (test_fw_config->reqs) { + rc = -EBUSY; + goto out_bail; + } + test_fw_config->reqs = vzalloc(array3_size(sizeof(struct test_batched_req), test_fw_config->num_requests, 2));

2 years, 1 month

1
0
0 0

[PATCH 4.14 203/204] test_firmware: fix a memory leak with reqs buffer

by Greg Kroah-Hartman

From: Mirsad Goran Todorovac <mirsad.todorovac(a)alu.unizg.hr> commit be37bed754ed90b2655382f93f9724b3c1aae847 upstream. Dan Carpenter spotted that test_fw_config->reqs will be leaked if trigger_batched_requests_store() is called two or more times. The same appears with trigger_batched_requests_async_store(). This bug wasn't trigger by the tests, but observed by Dan's visual inspection of the code. The recommended workaround was to return -EBUSY if test_fw_config->reqs is already allocated. Fixes: 7feebfa487b92 ("test_firmware: add support for request_firmware_into_buf") Cc: Luis Chamberlain <mcgrof(a)kernel.org> Cc: Greg Kroah-Hartman <gregkh(a)linuxfoundation.org> Cc: Russ Weight <russell.h.weight(a)intel.com> Cc: Tianfei Zhang <tianfei.zhang(a)intel.com> Cc: Shuah Khan <shuah(a)kernel.org> Cc: Colin Ian King <colin.i.king(a)gmail.com> Cc: Randy Dunlap <rdunlap(a)infradead.org> Cc: linux-kselftest(a)vger.kernel.org Cc: stable(a)vger.kernel.org # v5.4 Suggested-by: Dan Carpenter <error27(a)gmail.com> Suggested-by: Takashi Iwai <tiwai(a)suse.de> Signed-off-by: Mirsad Goran Todorovac <mirsad.todorovac(a)alu.unizg.hr> Reviewed-by: Dan Carpenter <dan.carpenter(a)linaro.org> Acked-by: Luis Chamberlain <mcgrof(a)kernel.org> Link: https://lore.kernel.org/r/20230509084746.48259-2-mirsad.todorovac@alu.unizg… Signed-off-by: Greg Kroah-Hartman <gregkh(a)linuxfoundation.org> --- lib/test_firmware.c | 10 ++++++++++ 1 file changed, 10 insertions(+) --- a/lib/test_firmware.c +++ b/lib/test_firmware.c @@ -621,6 +621,11 @@ static ssize_t trigger_batched_requests_ mutex_lock(&test_fw_mutex); + if (test_fw_config->reqs) { + rc = -EBUSY; + goto out_bail; + } + test_fw_config->reqs = vzalloc(sizeof(struct test_batched_req) * test_fw_config->num_requests * 2); if (!test_fw_config->reqs) { @@ -723,6 +728,11 @@ ssize_t trigger_batched_requests_async_s mutex_lock(&test_fw_mutex); + if (test_fw_config->reqs) { + rc = -EBUSY; + goto out_bail; + } + test_fw_config->reqs = vzalloc(sizeof(struct test_batched_req) * test_fw_config->num_requests * 2); if (!test_fw_config->reqs) {

2 years, 1 month

1
0
0 0

[PATCH 0/2] v2: F_OFD_GETLK extension to read lock info

by Stas Sergeev

This extension allows to use F_UNLCK on query, which currently returns EINVAL. Instead it can be used to query the locks on a particular fd - something that is not currently possible. The basic idea is that on F_OFD_GETLK, F_UNLCK would "conflict" with (or query) any types of the lock on the same fd, and ignore any locks on other fds. Use-cases: 1. CRIU-alike scenario when you want to read the locking info from an fd for the later reconstruction. This can now be done by setting l_start and l_len to 0 to cover entire file range, and do F_OFD_GETLK. In the loop you need to advance l_start past the returned lock ranges, to eventually collect all locked ranges. 2. Implementing the lock checking/enforcing policy. Say you want to implement an "auditor" module in your program, that checks that the I/O is done only after the proper locking is applied on a file region. In this case you need to know if the particular region is locked on that fd, and if so - with what type of the lock. If you would do that currently (without this extension) then you can only check for the write locks, and for that you need to probe the lock on your fd and then open the same file via another fd and probe there. That way you can identify the write lock on a particular fd, but such trick is non-atomic and complex. As for finding out the read lock on a particular fd - impossible. This extension allows to do such queries without any extra efforts. 3. Implementing the mandatory locking policy. Suppose you want to make a policy where the write lock inhibits any unlocked readers and writers. Currently you need to check if the write lock is present on some other fd, and if it is not there - allow the I/O operation. But because the write lock can appear at any moment, you need to do that under some global lock, which can be released only when the I/O operation is finished. With the proposed extension you can instead just check the write lock on your own fd first, and if it is there - allow the I/O operation on that fd without using any global lock. Only if there is no write lock on this fd, then you need to take global lock and check for a write lock on other fds. The second patch adds a test-case for OFD locks. It tests both the generic things and the proposed extension. The third patch is a proposed man page update for fcntl(2) (not for the linux source tree) Changes in v2: - Dropped the l_pid extension patch and updated test-case accordingly. Stas Sergeev (2): fs/locks: F_UNLCK extension for F_OFD_GETLK selftests: add OFD lock tests fs/locks.c | 23 +++- tools/testing/selftests/locking/Makefile | 2 + tools/testing/selftests/locking/ofdlocks.c | 132 +++++++++++++++++++++ 3 files changed, 154 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/locking/ofdlocks.c CC: Jeff Layton <jlayton(a)kernel.org> CC: Chuck Lever <chuck.lever(a)oracle.com> CC: Alexander Viro <viro(a)zeniv.linux.org.uk> CC: Christian Brauner <brauner(a)kernel.org> CC: linux-fsdevel(a)vger.kernel.org CC: linux-kernel(a)vger.kernel.org CC: Shuah Khan <shuah(a)kernel.org> CC: linux-kselftest(a)vger.kernel.org CC: linux-api(a)vger.kernel.org -- 2.39.2

2 years, 1 month

6
15
0 0

[PATCH] selftests/mm: FOLL_LONGTERM need to be updated to 0x100

by Ayush Jain

After commit 2c2241081f7d ("mm/gup: move private gup FOLL_ flags to internal.h") FOLL_LONGTERM flag value got updated from 0x10000 to 0x100 at include/linux/mm_types.h. As hmm.hmm_device_private.hmm_gup_test uses FOLL_LONGTERM Updating same here as well. Before this change test goes in an infinite assert loop in hmm.hmm_device_private.hmm_gup_test ========================================================== RUN hmm.hmm_device_private.hmm_gup_test ... hmm-tests.c:1962:hmm_gup_test:Expected HMM_DMIRROR_PROT_WRITE.. ..(2) == m[2] (34) hmm-tests.c:157:hmm_gup_test:Expected ret (-1) == 0 (0) hmm-tests.c:157:hmm_gup_test:Expected ret (-1) == 0 (0) ... ========================================================== Call Trace: <TASK> ? sched_clock+0xd/0x20 ? __lock_acquire.constprop.0+0x120/0x6c0 ? ktime_get+0x2c/0xd0 ? sched_clock+0xd/0x20 ? local_clock+0x12/0xd0 ? lock_release+0x26e/0x3b0 pin_user_pages_fast+0x4c/0x70 gup_test_ioctl+0x4ff/0xbb0 ? gup_test_ioctl+0x68c/0xbb0 __x64_sys_ioctl+0x99/0xd0 do_syscall_64+0x60/0x90 ? syscall_exit_to_user_mode+0x2a/0x50 ? do_syscall_64+0x6d/0x90 ? syscall_exit_to_user_mode+0x2a/0x50 ? do_syscall_64+0x6d/0x90 ? irqentry_exit_to_user_mode+0xd/0x20 ? irqentry_exit+0x3f/0x50 ? exc_page_fault+0x96/0x200 entry_SYSCALL_64_after_hwframe+0x72/0xdc RIP: 0033:0x7f6aaa31aaff After this change test is able to pass successfully. Signed-off-by: Ayush Jain <ayush.jain3(a)amd.com> Reviewed-by: Raghavendra K T <raghavendra.kt(a)amd.com> --- tools/testing/selftests/mm/hmm-tests.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/mm/hmm-tests.c b/tools/testing/selftests/mm/hmm-tests.c index 4adaad1b822f..20294553a5dd 100644 --- a/tools/testing/selftests/mm/hmm-tests.c +++ b/tools/testing/selftests/mm/hmm-tests.c @@ -57,9 +57,14 @@ enum { #define ALIGN(x, a) (((x) + (a - 1)) & (~((a) - 1))) /* Just the flags we need, copied from mm.h: */ + +#ifndef FOLL_WRITE #define FOLL_WRITE 0x01 /* check pte is writable */ -#define FOLL_LONGTERM 0x10000 /* mapping lifetime is indefinite */ +#endif +#ifndef FOLL_LONGTERM +#define FOLL_LONGTERM 0x100 /* mapping lifetime is indefinite */ +#endif FIXTURE(hmm) { int fd; -- 2.39.3

2 years, 1 month

3
2
0 0

[PATCH v27 0/6] Implement IOCTL to get and optionally clear info about PTEs

by Muhammad Usama Anjum

*Changes in v27:* - Handle review comments and minor improvements - Add performance improvement patch on top with test for easy review *Changes in v26:* - Code re-structurring and API changes in PAGEMAP_IOCTL *Changes in v25*: - Do proper filtering on hole as well (hole got missed earlier) *Changes in v24*: - Rebase on top of next-20230710 - Place WP markers in case of hole as well *Changes in v23*: - Set vec_buf_index in loop only when vec_buf_index is set - Return -EFAULT instead of -EINVAL if vec is NULL - Correctly return the walk ending address to the page granularity *Changes in v22*: - Interface change: - Replace [start start + len) with [start, end) - Return the ending address of the address walk in start *Changes in v21*: - Abort walk instead of returning error if WP is to be performed on partial hugetlb *Changes in v20* - Correct PAGE_IS_FILE and add PAGE_IS_PFNZERO *Changes in v19* - Minor changes and interface updates *Changes in v18* - Rebase on top of next-20230613 - Minor updates *Changes in v17* - Rebase on top of next-20230606 - Minor improvements in PAGEMAP_SCAN IOCTL patch *Changes in v16* - Fix a corner case - Add exclusive PM_SCAN_OP_WP back *Changes in v15* - Build fix (Add missed build fix in RESEND) *Changes in v14* - Fix build error caused by #ifdef added at last minute in some configs *Changes in v13* - Rebase on top of next-20230414 - Give-up on using uffd_wp_range() and write new helpers, flush tlb only once *Changes in v12* - Update and other memory types to UFFD_FEATURE_WP_ASYNC - Rebaase on top of next-20230406 - Review updates *Changes in v11* - Rebase on top of next-20230307 - Base patches on UFFD_FEATURE_WP_UNPOPULATED - Do a lot of cosmetic changes and review updates - Remove ENGAGE_WP + !GET operation as it can be performed with UFFDIO_WRITEPROTECT *Changes in v10* - Add specific condition to return error if hugetlb is used with wp async - Move changes in tools/include/uapi/linux/fs.h to separate patch - Add documentation *Changes in v9:* - Correct fault resolution for userfaultfd wp async - Fix build warnings and errors which were happening on some configs - Simplify pagemap ioctl's code *Changes in v8:* - Update uffd async wp implementation - Improve PAGEMAP_IOCTL implementation *Changes in v7:* - Add uffd wp async - Update the IOCTL to use uffd under the hood instead of soft-dirty flags *Motivation* The real motivation for adding PAGEMAP_SCAN IOCTL is to emulate Windows GetWriteWatch() and ResetWriteWatch() syscalls [1]. The GetWriteWatch() retrieves the addresses of the pages that are written to in a region of virtual memory. This syscall is used in Windows applications and games etc. This syscall is being emulated in pretty slow manner in userspace. Our purpose is to enhance the kernel such that we translate it efficiently in a better way. Currently some out of tree hack patches are being used to efficiently emulate it in some kernels. We intend to replace those with these patches. So the whole gaming on Linux can effectively get benefit from this. It means there would be tons of users of this code. CRIU use case [2] was mentioned by Andrei and Danylo: > Use cases for migrating sparse VMAs are binaries sanitized with ASAN, > MSAN or TSAN [3]. All of these sanitizers produce sparse mappings of > shadow memory [4]. Being able to migrate such binaries allows to highly > reduce the amount of work needed to identify and fix post-migration > crashes, which happen constantly. Andrei's defines the following uses of this code: * it is more granular and allows us to track changed pages more effectively. The current interface can clear dirty bits for the entire process only. In addition, reading info about pages is a separate operation. It means we must freeze the process to read information about all its pages, reset dirty bits, only then we can start dumping pages. The information about pages becomes more and more outdated, while we are processing pages. The new interface solves both these downsides. First, it allows us to read pte bits and clear the soft-dirty bit atomically. It means that CRIU will not need to freeze processes to pre-dump their memory. Second, it clears soft-dirty bits for a specified region of memory. It means CRIU will have actual info about pages to the moment of dumping them. * The new interface has to be much faster because basic page filtering is happening in the kernel. With the old interface, we have to read pagemap for each page. *Implementation Evolution (Short Summary)* From the definition of GetWriteWatch(), we feel like kernel's soft-dirty feature can be used under the hood with some additions like: * reset soft-dirty flag for only a specific region of memory instead of clearing the flag for the entire process * get and clear soft-dirty flag for a specific region atomically So we decided to use ioctl on pagemap file to read or/and reset soft-dirty flag. But using soft-dirty flag, sometimes we get extra pages which weren't even written. They had become soft-dirty because of VMA merging and VM_SOFTDIRTY flag. This breaks the definition of GetWriteWatch(). We were able to by-pass this short coming by ignoring VM_SOFTDIRTY until David reported that mprotect etc messes up the soft-dirty flag while ignoring VM_SOFTDIRTY [5]. This wasn't happening until [6] got introduced. We discussed if we can revert these patches. But we could not reach to any conclusion. So at this point, I made couple of tries to solve this whole VM_SOFTDIRTY issue by correcting the soft-dirty implementation: * [7] Correct the bug fixed wrongly back in 2014. It had potential to cause regression. We left it behind. * [8] Keep a list of soft-dirty part of a VMA across splits and merges. I got the reply don't increase the size of the VMA by 8 bytes. At this point, we left soft-dirty considering it is too much delicate and userfaultfd [9] seemed like the only way forward. From there onward, we have been basing soft-dirty emulation on userfaultfd wp feature where kernel resolves the faults itself when WP_ASYNC feature is used. It was straight forward to add WP_ASYNC feature in userfautlfd. Now we get only those pages dirty or written-to which are really written in reality. (PS There is another WP_UNPOPULATED userfautfd feature is required which is needed to avoid pre-faulting memory before write-protecting [9].) All the different masks were added on the request of CRIU devs to create interface more generic and better. [1] https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-… [2] https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@google.com [3] https://github.com/google/sanitizers [4] https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm#64-bit [5] https://lore.kernel.org/all/bfcae708-db21-04b4-0bbe-712badd03071@redhat.com [6] https://lore.kernel.org/all/20220725142048.30450-1-peterx@redhat.com/ [7] https://lore.kernel.org/all/20221122115007.2787017-1-usama.anjum@collabora.… [8] https://lore.kernel.org/all/20221220162606.1595355-1-usama.anjum@collabora.… [9] https://lore.kernel.org/all/20230306213925.617814-1-peterx@redhat.com [10] https://lore.kernel.org/all/20230125144529.1630917-1-mdanylo@google.com * Original Cover letter from v8* Hello, Note: Soft-dirty pages and pages which have been written-to are synonyms. As kernel already has soft-dirty feature inside which we have given up to use, we are using written-to terminology while using UFFD async WP under the hood. It is possible to find and clear soft-dirty pages entirely in userspace. But it isn't efficient: - The mprotect and SIGSEGV handler for bookkeeping - The userfaultfd wp (synchronous) with the handler for bookkeeping Some benchmarks can be seen here[1]. This series adds features that weren't present earlier: - There is no atomic get soft-dirty/Written-to status and clear present in the kernel. - The pages which have been written-to can not be found in accurate way. (Kernel's soft-dirty PTE bit + sof_dirty VMA bit shows more soft-dirty pages than there actually are.) Historically, soft-dirty PTE bit tracking has been used in the CRIU project. The procfs interface is enough for finding the soft-dirty bit status and clearing the soft-dirty bit of all the pages of a process. We have the use case where we need to track the soft-dirty PTE bit for only specific pages on-demand. We need this tracking and clear mechanism of a region of memory while the process is running to emulate the getWriteWatch() syscall of Windows. *(Moved to using UFFD instead of soft-dirty feature to find pages which have been written-to from v7 patch series)*: Stop using the soft-dirty flags for finding which pages have been written to. It is too delicate and wrong as it shows more soft-dirty pages than the actual soft-dirty pages. There is no interest in correcting it [2][3] as this is how the feature was written years ago. It shouldn't be updated to changed behaviour. Peter Xu has suggested using the async version of the UFFD WP [4] as it is based inherently on the PTEs. So in this patch series, I've added a new mode to the UFFD which is asynchronous version of the write protect. When this variant of the UFFD WP is used, the page faults are resolved automatically by the kernel. The pages which have been written-to can be found by reading pagemap file (!PM_UFFD_WP). This feature can be used successfully to find which pages have been written to from the time the pages were write protected. This works just like the soft-dirty flag without showing any extra pages which aren't soft-dirty in reality. The information related to pages if the page is file mapped, present and swapped is required for the CRIU project [5][6]. The addition of the required mask, any mask, excluded mask and return masks are also required for the CRIU project [5]. The IOCTL returns the addresses of the pages which match the specific masks. The page addresses are returned in struct page_region in a compact form. The max_pages is needed to support a use case where user only wants to get a specific number of pages. So there is no need to find all the pages of interest in the range when max_pages is specified. The IOCTL returns when the maximum number of the pages are found. The max_pages is optional. If max_pages is specified, it must be equal or greater than the vec_size. This restriction is needed to handle worse case when one page_region only contains info of one page and it cannot be compacted. This is needed to emulate the Windows getWriteWatch() syscall. The patch series include the detailed selftest which can be used as an example for the uffd async wp test and PAGEMAP_IOCTL. It shows the interface usages as well. [1] https://lore.kernel.org/lkml/54d4c322-cd6e-eefd-b161-2af2b56aae24@collabora… [2] https://lore.kernel.org/all/20221220162606.1595355-1-usama.anjum@collabora.… [3] https://lore.kernel.org/all/20221122115007.2787017-1-usama.anjum@collabora.… [4] https://lore.kernel.org/all/Y6Hc2d+7eTKs7AiH@x1n [5] https://lore.kernel.org/all/YyiDg79flhWoMDZB@gmail.com/ [6] https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@google.com/ Regards, Muhammad Usama Anjum Muhammad Usama Anjum (5): fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs fs/proc/task_mmu: Add fast paths to get/clear PAGE_IS_WRITTEN flag tools headers UAPI: Update linux/fs.h with the kernel sources mm/pagemap: add documentation of PAGEMAP_SCAN IOCTL selftests: mm: add pagemap ioctl tests Peter Xu (1): userfaultfd: UFFD_FEATURE_WP_ASYNC Documentation/admin-guide/mm/pagemap.rst | 64 + Documentation/admin-guide/mm/userfaultfd.rst | 35 + fs/proc/task_mmu.c | 715 +++++++++ fs/userfaultfd.c | 26 +- include/linux/hugetlb.h | 1 + include/linux/userfaultfd_k.h | 21 +- include/uapi/linux/fs.h | 59 + include/uapi/linux/userfaultfd.h | 9 +- mm/hugetlb.c | 34 +- mm/memory.c | 27 +- tools/include/uapi/linux/fs.h | 59 + tools/testing/selftests/mm/.gitignore | 2 + tools/testing/selftests/mm/Makefile | 3 +- tools/testing/selftests/mm/config | 1 + tools/testing/selftests/mm/pagemap_ioctl.c | 1491 ++++++++++++++++++ tools/testing/selftests/mm/run_vmtests.sh | 4 + 16 files changed, 2527 insertions(+), 24 deletions(-) create mode 100644 tools/testing/selftests/mm/pagemap_ioctl.c -- 2.39.2

2 years, 1 month

2
10
0 0

kselftest/next kselftest-seccomp: 2 runs, 1 regressions (v6.5-rc3-26-g2b2fe6052dd01)

by kernelci.org bot

kselftest/next kselftest-seccomp: 2 runs, 1 regressions (v6.5-rc3-26-g2b2fe6052dd01) Regressions Summary ------------------- platform | arch | lab | compiler | defconfig | regressions ----------------+-------+---------------+----------+------------------------------+------------ mt8173-elm-hana | arm64 | lab-collabora | gcc-10 | defconfig+kse...4-chromebook | 1 Details: https://kernelci.org/test/job/kselftest/branch/next/kernel/v6.5-rc3-26-g2b2… Test: kselftest-seccomp Tree: kselftest Branch: next Describe: v6.5-rc3-26-g2b2fe6052dd01 URL: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git SHA: 2b2fe6052dd01fdb4e9a31031c2c9d8f03cf7753 Test Regressions ---------------- platform | arch | lab | compiler | defconfig | regressions ----------------+-------+---------------+----------+------------------------------+------------ mt8173-elm-hana | arm64 | lab-collabora | gcc-10 | defconfig+kse...4-chromebook | 1 Details: https://kernelci.org/test/plan/id/64d2fa2ce56d88a33435b1f6 Results: 0 PASS, 1 FAIL, 0 SKIP Full config: defconfig+kselftest+arm64-chromebook Compiler: gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110) Plain log: https://storage.kernelci.org//kselftest/next/v6.5-rc3-26-g2b2fe6052dd01/arm… HTML log: https://storage.kernelci.org//kselftest/next/v6.5-rc3-26-g2b2fe6052dd01/arm… Rootfs: http://storage.kernelci.org/images/rootfs/debian/bullseye-kselftest/2023062… * kselftest-seccomp.login: https://kernelci.org/test/case/id/64d2fa2ce56d88a33435b1f7 failing since 295 days (last pass: linux-kselftest-next-6.0-rc2-11-g144eeb2fc761, first fail: v6.1-rc1)

2 years, 1 month

1
0
0 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

Linux-kselftest-mirror