- Linux-kselftest-mirror - lists.linaro.org

[RFC PATCH 00/21] VFIO live update support

by Vipin Sharma

Hello, This series adds the live update support in the VFIO PCI subsystem on top of Live Update Orchestrator (LUO) [1]. This series can also be found on GitHub: https://github.com/shvipin/linux vfio/liveupdate/rfc-v1 Goal of live update in VFIO subsystem is to preserve VFIO PCI devices while the host kernel is going through a live update. A preserved device means it can continue to work, perform DMA, not get reset while host under live update gets rebooted via kexec. This series registers VFIO with LUO, implements LUO callbacks, skip DMA clear, skip device reset, preserves and restores a device virtual config during live update. I have added a selftest towards the end of this series, vfio_pci_liveupdate_test, which sets certain properties of a VFIO PCI device, performs a live update, and then validates those properties are still same on the device. Overall flow for a VFIO device going through a live update will be something like: 1. Userspace passes a VFIO cdev FD along with a token to LUO for preservation. 2. LUO passes FD to VFIO subsystem to verify if FD can be preserved. If yes, it increases the refcount on the FD. 3. Eventually, userspace tells LUO to prepare for live update which results in LUO calling prepare() callback to each of its register filesystem handler with the passed FD it should be preparing. 4. VFIO subsystem saves certain properties which will be either lost or hard to recover from the device. 5. VFIO saves the needed data to KHO and provide LUO with the physical address of the data preserved by KHO. 6. Userspace sends FREEZE event to freeze the system. LUO forwards this to each of its registered subsystem. 7. VFIO disables interrupts configured on the device during freeze call. 8. Userspace performs kexec. 9. During kexec reboot, generally, all PCI devices gets their Bus Master Enable bit disabled. In live update case, preserved VFIO devices are skipped. 9. During boot, usual device enumeration happens and LUO also intializes itself. 10. Userspace uses the same token value (step 1), and ask LUO to return VFIO FD corresponding to token. 11. LUO ask VFIO to return VFIO cdev FD corresponding to the token. It gives it the physical address which VFIO returned it in step 5. 12. VFIO restore the KHO data and read the BDF value it saved. It iterates through all of the VFIO device it has in its VFIO cdev class and finds the BDF device. 13. VFIO creates an anonymous inode and file corresponding to the VFIO PCI device and returns it to LUO and LUO returns it to userspace. 14. Now FD returned to userspace works exactly same as if userspace has opened a VFIO device from /dev/vfio/device/* location. 15. It makes usual bind iommufd and attach page table calls. 16. During bind, when VFIO device is internally opened for the first time: - VFIO skips Bus Master Disable - VFIO skips device reset. - VFIO instead of initializing vconfig from the scratch uses the vconfig stored in KHO, and same for few other fields. This is what current series is implementing and validating through selftest. There are other things are which not implemented yet and some are also dependent on other subsystems. For example: 1. Once a device has been prepared, VFIO should not allow any changes to its state from userspace for example, changing PCI config values, resetting the device, etc. 2. Device IOVA is not preserved in this series. This work is done separately in IOMMMUFD live update preservation [2] 3. During PCI device enumeration, PCI subsystem writes to PCI config space, attach device to its original driver if present. This work is being done in PCI preservation [3]. 4. Enabling PCI device done in VFIO subsystem should be handled in PCI subsystem. Current, this patch series hasn't changed the behavior. 5. If live update gets canceled, interrupts which are disabled in freeze need to be reconfigured again. 6. In finish, if a device is not restored, how to know if KHO folio has been restored or not. 6. VFIO cdev is restored in anonymous file system. This should instead be done on devetmpfs For reviewers, following are the grouping of patches in this series: Patches 1-4 ----------- Feel free to ignore if you are only interested in VFIO. These are only for live update selftests. I had to make some changes on top LUO v4 series, to create a library out of them which can be used in other selftests (vfio), and fix some build issues. Patches 5-9 ----------- Adds basic live update support in VFIO. Registers to LUO, saves the device BDF in KHO during prepare, and returns VFIO cdev FD during restore. It doesn't save or skip anything else. Patches 10-17 ------------- Adds support for skipping certain opertions and preserving certain data needed to restore a device. Patches 18-21 ------------- - Integrate VFIO selftest with live update selftest library. - Adds a basic vfio_pci_liveupdate_test test which validates that Bus Master Enable bit is preserved, and virtual config is restored properly. Testing ------- I have done testing on QEMU with a test pci device and also on a bare metal with Intel DSA device. Make sure IDXD driver is not built in your kernel if testing with Intel DSA device. Basically, whichever device you use, it should not get auto-bind to any other driver. Important config options which should be enabled to test this series: - CONFIG_KEXEC_FILE - CONFIG_LIVEUPDATE - CONFIG_KEXEC_HANDOVER Besides this usual VFIO, VFIO_PCI, IOMMU and other dependencies are enabled. To build the test provide KHDR_INCLUDES to your make command if your headers are out-of-tree. KHDR_INCLUDES="-isystem ../../../../build/usr/include" make vfio_pci_liveupdate_test needs to be executed manually. This test needs to be executed two times; one before the live update and second after. ./run.sh -d 0000:00:04.0 vfio_pci_liveupdate_test Next Steps ---------- 1. Looking forward to feedback on this series. - What other things we should save? - Which things should not be saved? - Any locks or incorrect locking done in the series. - Any optimizations. 2. Integration with IOMMUFD and PCI series for complete workflow where a device continues a DMA while undergoing through live update. I will be going on a paternity leave soon, so, my responses gonna be intermittent. David Matlack (dmatlack(a)google.com) has graciously offered to work on this series and continue upstream engagement on this feature until I am back. Thank you, David! [1] LUO-v4: https://lore.kernel.org/linux-mm/20250929010321.3462457-1-pasha.tatashin@so… [2] IOMMUFD: https://lore.kernel.org/linux-iommu/20250928190624.3735830-1-skhawaja@googl… [3] PCI: https://lore.kernel.org/linux-pci/20250916-luo-pci-v2-0-c494053c3c08@kernel… Vipin Sharma (21): selftests/liveupdate: Build tests from the selftests/liveupdate directory selftests/liveupdate: Create library of core live update ioctls selftests/liveupdate: Move do_kexec.sh script to liveupdate/lib selftests/liveupdate: Move LUO ioctls calls to liveupdate library vfio/pci: Register VFIO live update file handler to Live Update Orchestrator vfio/pci: Accept live update preservation request for VFIO cdev vfio/pci: Store VFIO PCI device preservation data in KHO for live update vfio/pci: Retrieve preserved VFIO device for Live Update Orechestrator vfio/pci: Add Live Update finish callback implementation PCI: Add option to skip Bus Master Enable reset during kexec vfio/pci: Skip clearing bus master on live update device during kexec vfio/pci: Skip clearing bus master on live update restored device vfio/pci: Preserve VFIO PCI config space through live update vfio/pci: Skip device reset on live update restored device. PCI: Make PCI saved state and capability structs public vfio/pci: Save and restore the PCI state of the VFIO device vfio/pci: Disable interrupts before going live update kexec vfio: selftests: Build liveupdate library in VFIO selftests vfio: selftests: Initialize vfio_pci_device using a VFIO cdev FD vfio: selftests: Add VFIO live update test vfio: selftests: Validate vconfig preservation of VFIO PCI device during live update drivers/pci/pci-driver.c | 6 +- drivers/pci/pci.c | 5 - drivers/pci/pci.h | 7 - drivers/vfio/pci/Makefile | 1 + drivers/vfio/pci/vfio_pci_config.c | 17 + drivers/vfio/pci/vfio_pci_core.c | 31 +- drivers/vfio/pci/vfio_pci_liveupdate.c | 461 ++++++++++++++++++ drivers/vfio/pci/vfio_pci_priv.h | 17 + drivers/vfio/vfio_main.c | 20 +- include/linux/pci.h | 15 + include/linux/vfio.h | 8 + include/linux/vfio_pci_core.h | 1 + tools/testing/selftests/liveupdate/.gitignore | 7 +- tools/testing/selftests/liveupdate/Makefile | 31 +- .../liveupdate/{ => lib}/do_kexec.sh | 0 .../liveupdate/lib/include/liveupdate_util.h | 27 + .../selftests/liveupdate/lib/libliveupdate.mk | 18 + .../liveupdate/lib/liveupdate_util.c | 106 ++++ .../selftests/liveupdate/luo_multi_file.c | 2 - .../selftests/liveupdate/luo_multi_kexec.c | 2 - .../selftests/liveupdate/luo_multi_session.c | 2 - .../selftests/liveupdate/luo_test_utils.c | 73 +-- .../selftests/liveupdate/luo_test_utils.h | 10 +- .../selftests/liveupdate/luo_unreclaimed.c | 1 - tools/testing/selftests/vfio/Makefile | 15 +- .../selftests/vfio/lib/include/vfio_util.h | 1 + .../selftests/vfio/lib/vfio_pci_device.c | 33 +- .../selftests/vfio/vfio_pci_liveupdate_test.c | 116 +++++ 28 files changed, 900 insertions(+), 133 deletions(-) create mode 100644 drivers/vfio/pci/vfio_pci_liveupdate.c rename tools/testing/selftests/liveupdate/{ => lib}/do_kexec.sh (100%) create mode 100644 tools/testing/selftests/liveupdate/lib/include/liveupdate_util.h create mode 100644 tools/testing/selftests/liveupdate/lib/libliveupdate.mk create mode 100644 tools/testing/selftests/liveupdate/lib/liveupdate_util.c create mode 100644 tools/testing/selftests/vfio/vfio_pci_liveupdate_test.c base-commit: e48be01cadc981362646dc3a87d57316421590a5 -- 2.51.0.858.gf9c4a03a3a-goog

8 hours, 44 minutes

3
33
0 0

[RFC PATCH 0/4] selftests/damon: add a test for obsolete_target sysfs file

by SeongJae Park

An RFC patch series [1] that add a new DAMON sysfs file for arbitrary targets removal is under review. Add a selftest for the feature. The new test uses the feature using the python wrapper of DAMON sysfs interface, and confirm the expected internal data structure change is made using drgn. So this patch series may better to be a part of the other one [1] that introduces the obsolete_target file. But, because no significant change is requested on the series so far, I'm posting this as an individual RFC. In the next version, I may merge the two series into one, to add all related changes at one step. [1] https://lore.kernel.org/20251016214736.84286-1-sj@kernel.org SeongJae Park (4): selftests/damon/_damon_sysfs: support obsolete_target file drgn_dump_damon_status: dump damon_target->obsolete sysfs.py: extend assert_ctx_committed() for monitoring targets selftests/damon/sysfs: add obsolete_target test tools/testing/selftests/damon/_damon_sysfs.py | 11 ++++- .../selftests/damon/drgn_dump_damon_status.py | 1 + tools/testing/selftests/damon/sysfs.py | 48 +++++++++++++++++++ 3 files changed, 58 insertions(+), 2 deletions(-) base-commit: 1aba8bd57e6aaa1c9e699c8de66bcc931d4b1116 -- 2.47.3

11 hours, 11 minutes

1
4
0 0

[PATCH v6 00/10] liveupdate: Rework KHO for in-kernel users & Fix memory corruption

by Pasha Tatashin

This series addresses comments and combines into one the two series [1] and [2], and adds review-bys. This series refactors the KHO framework to better support in-kernel users like the upcoming LUO. The current design, which relies on a notifier chain and debugfs for control, is too restrictive for direct programmatic use. The core of this rework is the removal of the notifier chain in favor of a direct registration API. This decouples clients from the shutdown-time finalization sequence, allowing them to manage their preserved state more flexibly and at any time. Also, this series fixes a memory corruption bug in KHO that occurs when KFENCE is enabled. The root cause is that KHO metadata, allocated via kzalloc(), can be randomly serviced by kfence_alloc(). When a kernel boots via KHO, the early memblock allocator is restricted to a "scratch area". This forces the KFENCE pool to be allocated within this scratch area, creating a conflict. If KHO metadata is subsequently placed in this pool, it gets corrupted during the next kexec operation. [1] https://lore.kernel.org/all/20251007033100.836886-1-pasha.tatashin@soleen.c… [2] https://lore.kernel.org/all/20251015053121.3978358-1-pasha.tatashin@soleen.… Mike Rapoport (Microsoft) (1): kho: drop notifiers Pasha Tatashin (9): kho: allow to drive kho from within kernel kho: make debugfs interface optional kho: add interfaces to unpreserve folios and page ranes kho: don't unpreserve memory during abort liveupdate: kho: move to kernel/liveupdate kho: move kho debugfs directory to liveupdate liveupdate: kho: warn and fail on metadata or preserved memory in scratch area liveupdate: kho: Increase metadata bitmap size to PAGE_SIZE liveupdate: kho: allocate metadata directly from the buddy allocator Documentation/core-api/kho/concepts.rst | 2 +- MAINTAINERS | 3 +- include/linux/kexec_handover.h | 53 +- init/Kconfig | 2 + kernel/Kconfig.kexec | 15 - kernel/Makefile | 2 +- kernel/liveupdate/Kconfig | 38 ++ kernel/liveupdate/Makefile | 5 + kernel/{ => liveupdate}/kexec_handover.c | 588 +++++++++----------- kernel/liveupdate/kexec_handover_debug.c | 25 + kernel/liveupdate/kexec_handover_debugfs.c | 216 +++++++ kernel/liveupdate/kexec_handover_internal.h | 56 ++ lib/test_kho.c | 30 +- mm/memblock.c | 62 +-- tools/testing/selftests/kho/init.c | 2 +- tools/testing/selftests/kho/vmtest.sh | 1 + 16 files changed, 645 insertions(+), 455 deletions(-) create mode 100644 kernel/liveupdate/Kconfig create mode 100644 kernel/liveupdate/Makefile rename kernel/{ => liveupdate}/kexec_handover.c (78%) create mode 100644 kernel/liveupdate/kexec_handover_debug.c create mode 100644 kernel/liveupdate/kexec_handover_debugfs.c create mode 100644 kernel/liveupdate/kexec_handover_internal.h base-commit: f406055cb18c6e299c4a783fc1effeb16be41803 -- 2.51.0.915.g61a8936c21-goog

14 hours, 38 minutes

1
10
0 0

[PATCH v4 net-next 00/13] AccECN protocol case handling series

by chia-yu.chang＠nokia-bell-labs.com

From: Chia-Yu Chang <chia-yu.chang(a)nokia-bell-labs.com> Hello, Plesae find the v4 AccECN case handling patch series, which covers several excpetional case handling of Accurate ECN spec (RFC9768), adds new identifiers to be used by CC modules, adds ecn_delta into rate_sample, and keeps the ACE counter for computation, etc. This patch series is part of the full AccECN patch series, which is available at https://github.com/L4STeam/linux-net-next/commits/upstream_l4steam/ Best regards, Chia-Yu --- v4: - Add previous #13 in v2 back after dicussion with the RFC author. - Add TCP_ACCECN_OPTION_PERSIST to tcp_ecn_option sysctl to ignore AccECN fallback policy on sending AccECN option. v3: - Add additional min() check if pkts_acked_ewma is not initialized in #1. - Change TCP_CONG_WANTS_ECT_1 into individual flag add helper function INET_ECN_xmit_wants_ect_1() in #3. - Add empty line between variable declarations and code in #4. - Update commit message to fix old AccECN commits in #5. - Remove unnecessary brackets in #10. - Move patch #3 in v2 to a later Prague patch serise and remove patch #13 in v2. --- Chia-Yu Chang (11): tcp: L4S ECT(1) identifier and NEEDS_ACCECN for CC modules tcp: disable RFC3168 fallback identifier for CC modules tcp: accecn: handle unexpected AccECN negotiation feedback tcp: accecn: retransmit downgraded SYN in AccECN negotiation tcp: move increment of num_retrans tcp: accecn: retransmit SYN/ACK without AccECN option or non-AccECN SYN/ACK tcp: accecn: unset ECT if receive or send ACE=0 in AccECN negotiaion tcp: accecn: fallback outgoing half link to non-AccECN tcp: accecn: verify ACE counter in 1st ACK after AccECN negotiation tcp: accecn: detect loss ACK w/ AccECN option and add TCP_ACCECN_OPTION_PERSIST tcp: accecn: enable AccECN Ilpo Järvinen (2): tcp: try to avoid safer when ACKs are thinned gro: flushing when CWR is set negatively affects AccECN Documentation/networking/ip-sysctl.rst | 4 +- .../networking/net_cachelines/tcp_sock.rst | 1 + include/linux/tcp.h | 4 +- include/net/inet_ecn.h | 20 +++- include/net/tcp.h | 32 ++++++- include/net/tcp_ecn.h | 92 ++++++++++++++----- net/ipv4/sysctl_net_ipv4.c | 4 +- net/ipv4/tcp.c | 2 + net/ipv4/tcp_cong.c | 10 +- net/ipv4/tcp_input.c | 58 ++++++++++-- net/ipv4/tcp_minisocks.c | 40 +++++--- net/ipv4/tcp_offload.c | 3 +- net/ipv4/tcp_output.c | 42 ++++++--- 13 files changed, 241 insertions(+), 71 deletions(-) -- 2.34.1

15 hours, 49 minutes

5
20
0 0

[PATCH 0/2] KHO: Fix metadata allocation in scratch area

by Pasha Tatashin

This series fixes a memory corruption bug in KHO that occurs when KFENCE is enabled. The root cause is that KHO metadata, allocated via kzalloc(), can be randomly serviced by kfence_alloc(). When a kernel boots via KHO, the early memblock allocator is restricted to a "scratch area". This forces the KFENCE pool to be allocated within this scratch area, creating a conflict. If KHO metadata is subsequently placed in this pool, it gets corrupted during the next kexec operation. The series is structured in two parts: Patch 1/2 introduces a debug-only feature (CONFIG_KEXEC_HANDOVER_DEBUG) that adds checks to detect and fail any operation that attempts to place KHO metadata or preserved memory within the scratch area. This serves as a validation and diagnostic tool to confirm the problem without affecting production builds. Patch 2/2 provides the fix by modifying KHO to allocate its metadata directly from the buddy allocator instead of SLUB. This bypasses the KFENCE interception entirely. Pasha Tatashin (2): liveupdate: kho: warn and fail on metadata or preserved memory in scratch area liveupdate: kho: allocate metadata directly from the buddy allocator kernel/liveupdate/Kconfig | 15 ++++++ kernel/liveupdate/kexec_handover.c | 51 ++++++++++++++++----- kernel/liveupdate/kexec_handover_debug.c | 18 ++++++++ kernel/liveupdate/kexec_handover_internal.h | 9 ++++ 4 files changed, 81 insertions(+), 12 deletions(-) base-commit: 0b2f041c47acb45db82b4e847af6e17eb66cd32d -- 2.51.0.788.g6d19910ace-goog

16 hours, 24 minutes

4
16
0 0

[PATCH bpf-next v5 00/15] selftests/bpf: Integrate test_xsk.c to test_progs framework

by Bastien Curutchet (eBPF Foundation)

Hi all, Now that the merge window is over, here's a respin of the previous iteration rebased on the latest bpf-next_base. The bug triggering the XDP_ADJUST_TAIL_SHRINK_MULTI_BUFF failure when CONFIG_DEBUG_VM is enabled hasn't been fixed yet so I've moved the test to the flaky table. The test_xsk.sh script covers many AF_XDP use cases. The tests it runs are defined in xksxceiver.c. Since this script is used to test real hardware, the goal here is to leave it as it is, and only integrate the tests that run on veth peers into the test_progs framework. Some tests are flaky so they can't be integrated in the CI as they are. I think that fixing their flakyness would require a significant amount of work. So, as first step, I've excluded them from the list of tests migrated to the CI (cf PATCH 14). If these tests get fixed at some point, integrating them into the CI will be straightforward. PATCH 1 extracts test_xsk[.c/.h] from xskxceiver[.c/.h] to make the tests available to test_progs. PATCH 2 to 7 fix small issues in the current test PATCH 8 to 13 handle all errors to release resources instead of calling exit() when any error occurs. PATCH 14 isolates some flaky tests PATCH 15 integrate the non-flaky tests to the test_progs framework Signed-off-by: Bastien Curutchet (eBPF Foundation) <bastien.curutchet(a)bootlin.com> --- Changes in v5: - Rebase on latest bpf-next_base - Move XDP_ADJUST_TAIL_SHRINK_MULTI_BUFF to the flaky table - Add Maciej's reviewed-by - Link to v4: https://lore.kernel.org/r/20250924-xsk-v4-0-20e57537b876@bootlin.com Changes in v4: - Fix test_xsk.sh's summary report. - Merge PATCH 11 & 12 together, otherwise PATCH 11 fails to build. - Split old PATCH 3 in two patches. The first one fixes testapp_stats_rx_dropped(), the second one fixes testapp_xdp_shared_umem(). The unecessary frees (in testapp_stats_rx_full() and testapp_stats_fill_empty() are removed) - Link to v3: https://lore.kernel.org/r/20250904-xsk-v3-0-ce382e331485@bootlin.com Changes in v3: - Rebase on latest bpf-next_base to integrate commit c9110e6f7237 ("selftests/bpf: Fix count write in testapp_xdp_metadata_copy()"). - Move XDP_METADATA_COPY_* tests from flaky-tests to nominal tests - Link to v2: https://lore.kernel.org/r/20250902-xsk-v2-0-17c6345d5215@bootlin.com Changes in v2: - Rebase on the latest bpf-next_base and integrate the newly added tests to the work (adjust_tail* and tx_queue_consumer tests) - Re-order patches to split xkxceiver sooner. - Fix the bug reported by Maciej. - Fix verbose mode in test_xsk.sh by keeping kselftest (remove PATCH 1, 7 and 8) - Link to v1: https://lore.kernel.org/r/20250313-xsk-v1-0-7374729a93b9@bootlin.com --- Bastien Curutchet (eBPF Foundation) (15): selftests/bpf: test_xsk: Split xskxceiver selftests/bpf: test_xsk: Initialize bitmap before use selftests/bpf: test_xsk: Fix __testapp_validate_traffic()'s return value selftests/bpf: test_xsk: fix memory leak in testapp_stats_rx_dropped() selftests/bpf: test_xsk: fix memory leak in testapp_xdp_shared_umem() selftests/bpf: test_xsk: Wrap test clean-up in functions selftests/bpf: test_xsk: Release resources when swap fails selftests/bpf: test_xsk: Add return value to init_iface() selftests/bpf: test_xsk: Don't exit immediately when xsk_attach fails selftests/bpf: test_xsk: Don't exit immediately when gettimeofday fails selftests/bpf: test_xsk: Don't exit immediately when workers fail selftests/bpf: test_xsk: Don't exit immediately if validate_traffic fails selftests/bpf: test_xsk: Don't exit immediately on allocation failures selftests/bpf: test_xsk: Isolate flaky tests selftests/bpf: test_xsk: Integrate test_xsk.c to test_progs framework tools/testing/selftests/bpf/Makefile | 11 +- tools/testing/selftests/bpf/prog_tests/test_xsk.c | 2595 ++++++++++++++++++++ tools/testing/selftests/bpf/prog_tests/test_xsk.h | 294 +++ tools/testing/selftests/bpf/prog_tests/xsk.c | 146 ++ tools/testing/selftests/bpf/xskxceiver.c | 2696 +-------------------- tools/testing/selftests/bpf/xskxceiver.h | 156 -- 6 files changed, 3174 insertions(+), 2724 deletions(-) --- base-commit: bd61720310e0b11bfbb7c8e1f373bb87d98451d4 change-id: 20250218-xsk-0cf90e975d14 Best regards, -- Bastien Curutchet (eBPF Foundation) <tux(a)bootlin.com>

20 hours, 46 minutes

3
17
0 0

[PATCH net-next] bonding: show master index when dumping slave info

by Hangbin Liu

Currently, there is no straightforward way to obtain the master/slave relationship via netlink. Users have to retrieve all slaves through sysfs to determine these relationships. To address this, we can either list all slaves under the bond interface or display the master index in each slave. Since the number of slaves could be quite large (e.g., 100+), it is more efficient to show the master information in the slave entry. Signed-off-by: Hangbin Liu <liuhangbin(a)gmail.com> --- drivers/net/bonding/bond_netlink.c | 4 ++++ include/uapi/linux/if_link.h | 1 + 2 files changed, 5 insertions(+) diff --git a/drivers/net/bonding/bond_netlink.c b/drivers/net/bonding/bond_netlink.c index 286f11c517f7..ff3f11674a8b 100644 --- a/drivers/net/bonding/bond_netlink.c +++ b/drivers/net/bonding/bond_netlink.c @@ -29,6 +29,7 @@ static size_t bond_get_slave_size(const struct net_device *bond_dev, nla_total_size(sizeof(u16)) + /* IFLA_BOND_SLAVE_AD_PARTNER_OPER_PORT_STATE */ nla_total_size(sizeof(s32)) + /* IFLA_BOND_SLAVE_PRIO */ nla_total_size(sizeof(u16)) + /* IFLA_BOND_SLAVE_ACTOR_PORT_PRIO */ + nla_total_size(sizeof(u32)) + /* IFLA_BOND_SLAVE_MASTER */ 0; } @@ -38,6 +39,9 @@ static int bond_fill_slave_info(struct sk_buff *skb, { struct slave *slave = bond_slave_get_rtnl(slave_dev); + if (nla_put_u32(skb, IFLA_BOND_SLAVE_MASTER, bond_dev->ifindex)) + goto nla_put_failure; + if (nla_put_u8(skb, IFLA_BOND_SLAVE_STATE, bond_slave_state(slave))) goto nla_put_failure; diff --git a/include/uapi/linux/if_link.h b/include/uapi/linux/if_link.h index 3b491d96e52e..bad41a1807f7 100644 --- a/include/uapi/linux/if_link.h +++ b/include/uapi/linux/if_link.h @@ -1567,6 +1567,7 @@ enum { IFLA_BOND_SLAVE_AD_PARTNER_OPER_PORT_STATE, IFLA_BOND_SLAVE_PRIO, IFLA_BOND_SLAVE_ACTOR_PORT_PRIO, + IFLA_BOND_SLAVE_MASTER, __IFLA_BOND_SLAVE_MAX, }; -- 2.50.1

22 hours, 3 minutes

4
6
0 0

Re: [PATCH 1/1] Use new `--output-format=doctest` rustdoc command line flag to improve doctest handling

by Miguel Ojeda

On Tue, 17 Jun 2025 16:00:33 +0200 Guillaume Gomez <guillaume1.gomez(a)gmail.com> wrote: > > The goal of this patch is to remove the use of 2 unstable > rustdoc features (`--no-run` and `--test-builder`) and replace it with a > stable feature: `--output-format=doctest`, which was added in > https://github.com/rust-lang/rust/pull/134531. > > Before this patch, the code was using very hacky methods in order to retrieve > doctests, modify them as needed and then concatenate all of them in one file. > > Now, with this new flag, it instead asks rustdoc to provide the doctests > code with their associated information such as file path and line number. > > Signed-off-by: Guillaume Gomez <guillaume1.gomez(a)gmail.com> > --- (Procedural bit: normally we provide a changelog between versions after this `---` line so that reviewers now what changed so far.) I finally took a look at this again, so I rebased it and got: thread 'main' panicked at scripts/rustdoc_test_gen.rs:92:15: No path candidates found for `rust_kernel_alloc_allocator.rs`.This is likely a bug in the build system, or some files went away while compiling. which brings me to the bigger point: the main reason to have the new output format is to avoid all these hacks, including the "find the real path back to the original file" hack here. More generally, to avoid the 2 scripts approach. So now we can finally get rid of all that and simplify. That is, we can just merge it all in a single script that reads the JSON and builds the result directly, since now we have everything we need (originally I needed the 2 scripts approach since `rustdoc` executed the test builder once per test so I had to somehow collect the results). i.e. no more hundreds of generated files/processes, just a simple pipe. Anyway, just to check we had everything we needed, I did a quick try -- please see the draft patch below. I gave it a go -- please see the draft patch below. The diff w.r.t. your patch would be something like +217 -341, i.e. we get rid of quite a lot of lines. I added as well some more context in the commit message, and put the right docs in the unified script. This also improves the sorting of the tests (it now follows the line number better). We still have to preserve the support for the old compilers, so what I think I will do is just have the new script separately, keeping the old ones as-is until we can remove them when we upgrade the minimum for e.g. the next Debian Stable. Cc'ing David and KUnit, since this is closer to getting ready -- please let me know if this raises alarms for anyone. Thanks! Cheers, Miguel From 4aa4581e9004cb95534805f73fdae56c454b3d1d Mon Sep 17 00:00:00 2001 From: Guillaume Gomez <guillaume1.gomez(a)gmail.com> Date: Tue, 17 Jun 2025 16:00:33 +0200 Subject: [PATCH] [TODO] rust: use new `rustdoc`'s `--output-format=doctest` The goal of this patch is to remove the use of 2 unstable `rustdoc` features (`--no-run` and `--test-builder`) and replace it with a future stable feature: `--output-format=doctest` [1]. Before this patch, the KUnit Rust doctests generation needed to employ several hacks in order to retrieve doctests, modify them as needed and then concatenate all of them in one file. In particular, it required using two scripts: one that got run as a test builder by `rustdoc` in order to extract the data and another that collected the results of all those processes. We requested upstream `rustdoc` a feature to get `rustdoc` to generate the information directly -- one that would also be designed to eventually be made stable. This resulted in the `--output-format=doctest` flag, which makes all the information neatly available as a JSON output, including filenames, line numbers, doctest test bodies and so on. Thus take advantage of the new flag, which in turn allows to just use a single script that gets piped that JSON output from the compiler and uses it to directly build the generated files to be run by KUnit. Link: https://github.com/rust-lang/rust/issues/134529 [1] Signed-off-by: Guillaume Gomez <guillaume1.gomez(a)gmail.com> Co-developed-by: Miguel Ojeda <ojeda(a)kernel.org> Signed-off-by: Miguel Ojeda <ojeda(a)kernel.org> --- rust/Makefile | 12 +- scripts/.gitignore | 1 - scripts/Makefile | 2 - scripts/json.rs | 235 +++++++++++++++++++++++++ scripts/remove-stale-files | 2 + scripts/rustdoc_test_builder.rs | 300 ++++++++++++++++++++++++++------ scripts/rustdoc_test_gen.rs | 265 ---------------------------- 7 files changed, 485 insertions(+), 332 deletions(-) create mode 100644 scripts/json.rs delete mode 100644 scripts/rustdoc_test_gen.rs diff --git a/rust/Makefile b/rust/Makefile index 23c7ae905bd2..93bc456e3576 100644 --- a/rust/Makefile +++ b/rust/Makefile @@ -57,7 +57,6 @@ RUST_LIB_SRC ?= $(rustc_sysroot)/lib/rustlib/src/rust/library ifneq ($(quiet),) rust_test_quiet=-q rustdoc_test_quiet=--test-args -q -rustdoc_test_kernel_quiet=>/dev/null endif core-cfgs = \ @@ -224,21 +223,20 @@ quiet_cmd_rustdoc_test_kernel = RUSTDOC TK $< rm -rf $(objtree)/$(obj)/test/doctests/kernel; \ mkdir -p $(objtree)/$(obj)/test/doctests/kernel; \ OBJTREE=$(abspath $(objtree)) \ - $(RUSTDOC) --test $(filter-out --remap-path-prefix=%,$(rust_flags)) \ + $(RUSTDOC) $(filter-out --remap-path-prefix=%,$(rust_flags)) \ -L$(objtree)/$(obj) --extern ffi --extern pin_init \ --extern kernel --extern build_error --extern macros \ --extern bindings --extern uapi \ - --no-run --crate-name kernel -Zunstable-options \ + --crate-name kernel -Zunstable-options \ --sysroot=/dev/null \ + --output-format=doctest \ $(rustdoc_modifiers_workaround) \ - --test-builder $(objtree)/scripts/rustdoc_test_builder \ - $< $(rustdoc_test_kernel_quiet); \ - $(objtree)/scripts/rustdoc_test_gen + $< | $(objtree)/scripts/rustdoc_test_builder %/doctests_kernel_generated.rs %/doctests_kernel_generated_kunit.c: \ $(src)/kernel/lib.rs $(obj)/kernel.o \ $(objtree)/scripts/rustdoc_test_builder \ - $(objtree)/scripts/rustdoc_test_gen FORCE + FORCE +$(call if_changed,rustdoc_test_kernel) # We cannot use `-Zpanic-abort-tests` because some tests are dynamic, diff --git a/scripts/.gitignore b/scripts/.gitignore index c2ef68848da5..6e6ab7b8f496 100644 --- a/scripts/.gitignore +++ b/scripts/.gitignore @@ -7,7 +7,6 @@ /module.lds /recordmcount /rustdoc_test_builder -/rustdoc_test_gen /sign-file /sorttable /target.json diff --git a/scripts/Makefile b/scripts/Makefile index 46f860529df5..71c7d9dcd95b 100644 --- a/scripts/Makefile +++ b/scripts/Makefile @@ -10,7 +10,6 @@ hostprogs-always-$(CONFIG_ASN1) += asn1_compiler hostprogs-always-$(CONFIG_MODULE_SIG_FORMAT) += sign-file hostprogs-always-$(CONFIG_SYSTEM_EXTRA_CERTIFICATE) += insert-sys-cert hostprogs-always-$(CONFIG_RUST_KERNEL_DOCTESTS) += rustdoc_test_builder -hostprogs-always-$(CONFIG_RUST_KERNEL_DOCTESTS) += rustdoc_test_gen ifneq ($(or $(CONFIG_X86_64),$(CONFIG_X86_32)),) always-$(CONFIG_RUST) += target.json @@ -23,7 +22,6 @@ endif hostprogs += generate_rust_target generate_rust_target-rust := y rustdoc_test_builder-rust := y -rustdoc_test_gen-rust := y HOSTCFLAGS_sorttable.o = -I$(srctree)/tools/include HOSTLDLIBS_sorttable = -lpthread diff --git a/scripts/json.rs b/scripts/json.rs new file mode 100644 index 000000000000..aff24bfd9213 --- /dev/null +++ b/scripts/json.rs @@ -0,0 +1,235 @@ +// SPDX-License-Identifier: GPL-2.0 + +//! JSON parser used to parse rustdoc output when retrieving doctests. + +use std::collections::HashMap; +use std::iter::Peekable; +use std::str::FromStr; + +#[derive(Debug, PartialEq, Eq)] +pub(crate) enum JsonValue { + Object(HashMap<String, JsonValue>), + String(String), + Number(i32), + Bool(bool), + Array(Vec<JsonValue>), + Null, +} + +fn parse_ident<I: Iterator<Item = char>>( + iter: &mut I, + output: JsonValue, + ident: &str, +) -> Result<JsonValue, String> { + let mut ident_iter = ident.chars().skip(1); + + loop { + let i = ident_iter.next(); + if i.is_none() { + return Ok(output); + } + let c = iter.next(); + if i != c { + if let Some(c) = c { + return Err(format!("Unexpected character `{c}` when parsing `{ident}`")); + } + return Err(format!("Missing character when parsing `{ident}`")); + } + } +} + +fn parse_string<I: Iterator<Item = char>>(iter: &mut I) -> Result<JsonValue, String> { + let mut out = String::new(); + + while let Some(c) = iter.next() { + match c { + '\\' => { + let Some(c) = iter.next() else { break }; + match c { + '"' | '\\' | '/' => out.push(c), + 'b' => out.push(char::from(0x8u8)), + 'f' => out.push(char::from(0xCu8)), + 't' => out.push('\t'), + 'r' => out.push('\r'), + 'n' => out.push('\n'), + _ => { + // This code doesn't handle codepoints so we put the string content as is. + out.push('\\'); + out.push(c); + } + } + } + '"' => { + return Ok(JsonValue::String(out)); + } + _ => out.push(c), + } + } + Err(format!("Unclosed JSON string `{out}`")) +} + +fn parse_number<I: Iterator<Item = char>>( + iter: &mut Peekable<I>, + digit: char, +) -> Result<JsonValue, String> { + let mut nb = String::new(); + + nb.push(digit); + loop { + // We peek next character to prevent taking it from the iterator in case it's a comma. + if matches!(iter.peek(), Some(',' | '}' | ']')) { + break; + } + let Some(c) = iter.next() else { break }; + if c.is_whitespace() { + break; + } else if !c.is_ascii_digit() { + return Err(format!("Error when parsing number `{nb}`: found `{c}`")); + } + nb.push(c); + } + i32::from_str(&nb) + .map(|nb| JsonValue::Number(nb)) + .map_err(|error| format!("Invalid number: `{error}`")) +} + +fn parse_array<I: Iterator<Item = char>>(iter: &mut Peekable<I>) -> Result<JsonValue, String> { + let mut values = Vec::new(); + + 'main: loop { + let Some(c) = iter.next() else { + return Err("Unclosed array".to_string()); + }; + if c.is_whitespace() { + continue; + } else if c == ']' { + break; + } + values.push(parse(iter, c)?); + while let Some(c) = iter.next() { + if c.is_whitespace() { + continue; + } else if c == ',' { + break; + } else if c == ']' { + break 'main; + } else { + return Err(format!("Unexpected `{c}` when parsing array")); + } + } + } + Ok(JsonValue::Array(values)) +} + +fn parse_object<I: Iterator<Item = char>>(iter: &mut Peekable<I>) -> Result<JsonValue, String> { + let mut values = HashMap::new(); + + 'main: loop { + let Some(c) = iter.next() else { + return Err("Unclosed object".to_string()); + }; + let key; + if c.is_whitespace() { + continue; + } else if c == '"' { + let JsonValue::String(k) = parse_string(iter)? else { + unreachable!() + }; + key = k; + } else if c == '}' { + break; + } else { + return Err(format!("Expected `\"` when parsing Object, found `{c}`")); + } + + // We then get the `:` separator. + loop { + let Some(c) = iter.next() else { + return Err(format!("Missing value after key `{key}`")); + }; + if c.is_whitespace() { + continue; + } else if c == ':' { + break; + } else { + return Err(format!( + "Expected `:` after key, found `{c}` when parsing object" + )); + } + } + // Then the value. + let value = loop { + let Some(c) = iter.next() else { + return Err(format!("Missing value after key `{key}`")); + }; + if c.is_whitespace() { + continue; + } else { + break parse(iter, c)?; + } + }; + + if values.contains_key(&key) { + return Err(format!("Duplicated key `{key}`")); + } + values.insert(key, value); + + while let Some(c) = iter.next() { + if c.is_whitespace() { + continue; + } else if c == ',' { + break; + } else if c == '}' { + break 'main; + } else { + return Err(format!("Unexpected `{c}` when parsing array")); + } + } + } + Ok(JsonValue::Object(values)) +} + +fn parse<I: Iterator<Item = char>>(iter: &mut Peekable<I>, c: char) -> Result<JsonValue, String> { + match c { + '{' => parse_object(iter), + '"' => parse_string(iter), + '[' => parse_array(iter), + 't' => parse_ident(iter, JsonValue::Bool(true), "true"), + 'f' => parse_ident(iter, JsonValue::Bool(false), "false"), + 'n' => parse_ident(iter, JsonValue::Null, "null"), + c => { + if c.is_ascii_digit() || c == '-' { + parse_number(iter, c) + } else { + Err(format!("Unexpected `{c}` character")) + } + } + } +} + +impl JsonValue { + pub(crate) fn parse(input: &str) -> Result<Self, String> { + let mut iter = input.chars().peekable(); + let mut value = None; + + while let Some(c) = iter.next() { + if c.is_whitespace() { + continue; + } + value = Some(parse(&mut iter, c)?); + break; + } + while let Some(c) = iter.next() { + if c.is_whitespace() { + continue; + } else { + return Err(format!("Unexpected character `{c}` after content")); + } + } + if let Some(value) = value { + Ok(value) + } else { + Err("Empty content".to_string()) + } + } +} diff --git a/scripts/remove-stale-files b/scripts/remove-stale-files index 6e39fa8540df..190dee6b50e8 100755 --- a/scripts/remove-stale-files +++ b/scripts/remove-stale-files @@ -26,3 +26,5 @@ rm -f scripts/selinux/genheaders/genheaders rm -f *.spec rm -f lib/test_fortify.log + +rm -f scripts/rustdoc_test_gen diff --git a/scripts/rustdoc_test_builder.rs b/scripts/rustdoc_test_builder.rs index f7540bcf595a..dd65bb670d25 100644 --- a/scripts/rustdoc_test_builder.rs +++ b/scripts/rustdoc_test_builder.rs @@ -1,74 +1,260 @@ // SPDX-License-Identifier: GPL-2.0 -//! Test builder for `rustdoc`-generated tests. +//! Generates KUnit tests from `rustdoc`-generated doctests. //! -//! This script is a hack to extract the test from `rustdoc`'s output. Ideally, `rustdoc` would -//! have an option to generate this information instead, e.g. as JSON output. +//! KUnit passes a context (`struct kunit *`) to each test, which should be forwarded to the other +//! KUnit functions and macros. //! -//! The `rustdoc`-generated test names look like `{file}_{line}_{number}`, e.g. -//! `...path_rust_kernel_sync_arc_rs_42_0`. `number` is the "test number", needed in cases like -//! a macro that expands into items with doctests is invoked several times within the same line. +//! However, we want to keep this as an implementation detail because: //! -//! However, since these names are used for bisection in CI, the line number makes it not stable -//! at all. In the future, we would like `rustdoc` to give us the Rust item path associated with -//! the test, plus a "test number" (for cases with several examples per item) and generate a name -//! from that. For the moment, we generate ourselves a new name, `{file}_{number}` instead, in -//! the `gen` script (done there since we need to be aware of all the tests in a given file). +//! - Test code should not care about the implementation. +//! +//! - Documentation looks worse if it needs to carry extra details unrelated to the piece +//! being described. +//! +//! - Test code should be able to define functions and call them, without having to carry +//! the context. +//! +//! - Later on, we may want to be able to test non-kernel code (e.g. `core` or third-party +//! crates) which likely use the standard library `assert*!` macros. +//! +//! For this reason, instead of the passed context, `kunit_get_current_test()` is used instead +//! (i.e. `current->kunit_test`). +//! +//! Note that this means other threads/tasks potentially spawned by a given test, if failing, will +//! report the failure in the kernel log but will not fail the actual test. Saving the pointer in +//! e.g. a `static` per test does not fully solve the issue either, because currently KUnit does +//! not support assertions (only expectations) from other tasks. Thus leave that feature for +//! the future, which simplifies the code here too. We could also simply not allow `assert`s in +//! other tasks, but that seems overly constraining, and we do want to support them, eventually. -use std::io::Read; +use std::{ + fs::File, + io::{BufWriter, Read, Write}, +}; + +use json::JsonValue; + +mod json; fn main() { let mut stdin = std::io::stdin().lock(); - let mut body = String::new(); - stdin.read_to_string(&mut body).unwrap(); + let mut rustdoc_json = String::new(); + stdin.read_to_string(&mut rustdoc_json).unwrap(); - // Find the generated function name looking for the inner function inside `main()`. - // - // The line we are looking for looks like one of the following: - // - // ``` - // fn main() { #[allow(non_snake_case)] fn _doctest_main_rust_kernel_file_rs_28_0() { - // fn main() { #[allow(non_snake_case)] fn _doctest_main_rust_kernel_file_rs_37_0() -> Result<(), impl ::core::fmt::Debug> { - // ``` - // - // It should be unlikely that doctest code matches such lines (when code is formatted properly). - let rustdoc_function_name = body - .lines() - .find_map(|line| { - Some( - line.split_once("fn main() {")? - .1 - .split_once("fn ")? - .1 - .split_once("()")? - .0, - ) - .filter(|x| x.chars().all(|c| c.is_alphanumeric() || c == '_')) - }) - .expect("No test function found in `rustdoc`'s output."); - - // Qualify `Result` to avoid the collision with our own `Result` coming from the prelude. - let body = body.replace( - &format!("{rustdoc_function_name}() -> Result<(), impl ::core::fmt::Debug> {{"), - &format!( - "{rustdoc_function_name}() -> ::core::result::Result<(), impl ::core::fmt::Debug> {{" - ), + let JsonValue::Object(rustdoc) = JsonValue::parse(&rustdoc_json).unwrap() else { + panic!("Expected an object") + }; + + let Some(JsonValue::Number(format_version)) = rustdoc.get("format_version") else { + panic!("missing `format_version` field"); + }; + assert!( + *format_version == 2, + "unsupported rustdoc format version: {format_version}" ); - // For tests that get generated with `Result`, like above, `rustdoc` generates an `unwrap()` on - // the return value to check there were no returned errors. Instead, we use our assert macro - // since we want to just fail the test, not panic the kernel. + let Some(JsonValue::Array(doctests)) = rustdoc.get("doctests") else { + panic!("`doctests` field is missing or has the wrong type"); + }; + + let mut nb_generated = 0; + let mut number = 0; + let mut last_file = ""; + let mut rust_tests = String::new(); + let mut c_test_declarations = String::new(); + let mut c_test_cases = String::new(); + for doctest in doctests { + let JsonValue::Object(doctest) = doctest else { + unreachable!() + }; + + // We check if we need to skip this test by checking it's a rust code and it's not ignored. + if let Some(JsonValue::Object(attributes)) = doctest.get("doctest_attributes") { + if attributes.get("rust") != Some(&JsonValue::Bool(true)) { + continue; + } + if let Some(JsonValue::String(ignore)) = attributes.get("ignore") { + if ignore != "None" { + continue; + } + } + } + + let ( + Some(JsonValue::String(file)), + Some(JsonValue::Number(line)), + Some(JsonValue::String(name)), + Some(JsonValue::Object(doctest_code)), + ) = ( + doctest.get("file"), + doctest.get("line"), + doctest.get("name"), + doctest.get("doctest_code"), + ) + else { + continue; + }; + + let ( + Some(JsonValue::String(code)), + Some(JsonValue::String(crate_level_code)), + Some(JsonValue::Object(wrapper)), + ) = ( + doctest_code.get("code"), + doctest_code.get("crate_level"), + doctest_code.get("wrapper"), + ) + else { + continue; + }; + + let (Some(JsonValue::String(before)), Some(JsonValue::String(after))) = + (wrapper.get("before"), wrapper.get("after")) + else { + continue; + }; + + // For tests that get generated with `Result`, `rustdoc` generates an `unwrap()` on + // the return value to check there were no returned errors. Instead, we use our assert macro + // since we want to just fail the test, not panic the kernel. + // + // We save the result in a variable so that the failed assertion message looks nicer. + let after = if let Some(JsonValue::Bool(true)) = wrapper.get("returns_result") { + "\n} let test_return_value = _inner(); assert!(test_return_value.is_ok()); }" + } else { + after.as_str() + }; + + let body = format!("{crate_level_code}\n{before}\n{code}{after}\n"); + nb_generated += 1; + + // Generate an ID sequence ("test number") for each one in the file. + if file == last_file { + number += 1; + } else { + number = 0; + last_file = file; + } + + // Generate a KUnit name (i.e. test name and C symbol) for this test. + // + // We avoid the line number, like `rustdoc` does, to make things slightly more stable for + // bisection purposes. However, to aid developers in mapping back what test failed, we will + // print a diagnostics line in the KTAP report. + let kunit_name = format!( + "rust_doctest_{}_{number}", + file.replace('/', "_").replace('.', "_") + ); + + // Calculate how many lines before `main` function (including the `main` function line). + let body_offset = body + .lines() + .take_while(|line| !line.contains("fn main() {")) + .count() + + 1; + + use std::fmt::Write; + write!( + rust_tests, + r#"/// Generated `{name}` KUnit test case from a Rust documentation test. +#[no_mangle] +pub extern "C" fn {kunit_name}(__kunit_test: *mut ::kernel::bindings::kunit) {{ + /// Overrides the usual [`assert!`] macro with one that calls KUnit instead. + #[allow(unused)] + macro_rules! assert {{ + ($cond:expr $(,)?) => {{{{ + ::kernel::kunit_assert!( + "{kunit_name}", "{file}", __DOCTEST_ANCHOR - {line}, $cond + ); + }}}} + }} + + /// Overrides the usual [`assert_eq!`] macro with one that calls KUnit instead. + #[allow(unused)] + macro_rules! assert_eq {{ + ($left:expr, $right:expr $(,)?) => {{{{ + ::kernel::kunit_assert_eq!( + "{kunit_name}", "{file}", __DOCTEST_ANCHOR - {line}, $left, $right + ); + }}}} + }} + + // Many tests need the prelude, so provide it by default. + #[allow(unused)] + use ::kernel::prelude::*; + + // Unconditionally print the location of the original doctest (i.e. rather than the location in + // the generated file) so that developers can easily map the test back to the source code. // - // We save the result in a variable so that the failed assertion message looks nicer. - let body = body.replace( - &format!("}} {rustdoc_function_name}().unwrap() }}"), - &format!("}} let test_return_value = {rustdoc_function_name}(); assert!(test_return_value.is_ok()); }}"), - ); + // This information is also printed when assertions fail, but this helps in the successful cases + // when the user is running KUnit manually, or when passing `--raw_output` to `kunit.py`. + // + // This follows the syntax for declaring test metadata in the proposed KTAP v2 spec, which may + // be used for the proposed KUnit test attributes API. Thus hopefully this will make migration + // easier later on. + ::kernel::kunit::info(fmt!(" # {kunit_name}.location: {file}:{line}\n")); + + /// The anchor where the test code body starts. + #[allow(unused)] + static __DOCTEST_ANCHOR: i32 = ::core::line!() as i32 + {body_offset} + 1; + {{ + {body} + main(); + }} +}} + +"# + ) + .unwrap(); + + write!(c_test_declarations, "void {kunit_name}(struct kunit *);\n").unwrap(); + write!(c_test_cases, " KUNIT_CASE({kunit_name}),\n").unwrap(); + } + + if nb_generated == 0 { + panic!("No test function found in `rustdoc`'s output."); + } + + let rust_tests = rust_tests.trim(); + let c_test_declarations = c_test_declarations.trim(); + let c_test_cases = c_test_cases.trim(); + + write!( + BufWriter::new(File::create("rust/doctests_kernel_generated.rs").unwrap()), + r#"//! `kernel` crate documentation tests. + +const __LOG_PREFIX: &[u8] = b"rust_doctests_kernel\0"; + +{rust_tests} +"# + ) + .unwrap(); + + write!( + BufWriter::new(File::create("rust/doctests_kernel_generated_kunit.c").unwrap()), + r#"/* + * `kernel` crate documentation tests. + */ + +#include <kunit/test.h> + +{c_test_declarations} + +static struct kunit_case test_cases[] = {{ + {c_test_cases} + {{ }} +}}; - // Figure out a smaller test name based on the generated function name. - let name = rustdoc_function_name.split_once("_rust_kernel_").unwrap().1; +static struct kunit_suite test_suite = {{ + .name = "rust_doctests_kernel", + .test_cases = test_cases, +}}; - let path = format!("rust/test/doctests/kernel/{name}"); +kunit_test_suite(test_suite); - std::fs::write(path, body.as_bytes()).unwrap(); +MODULE_LICENSE("GPL"); +"# + ) + .unwrap(); } diff --git a/scripts/rustdoc_test_gen.rs b/scripts/rustdoc_test_gen.rs deleted file mode 100644 index c8f9dc2ab976..000000000000 --- a/scripts/rustdoc_test_gen.rs +++ /dev/null @@ -1,265 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 - -//! Generates KUnit tests from saved `rustdoc`-generated tests. -//! -//! KUnit passes a context (`struct kunit *`) to each test, which should be forwarded to the other -//! KUnit functions and macros. -//! -//! However, we want to keep this as an implementation detail because: -//! -//! - Test code should not care about the implementation. -//! -//! - Documentation looks worse if it needs to carry extra details unrelated to the piece -//! being described. -//! -//! - Test code should be able to define functions and call them, without having to carry -//! the context. -//! -//! - Later on, we may want to be able to test non-kernel code (e.g. `core` or third-party -//! crates) which likely use the standard library `assert*!` macros. -//! -//! For this reason, instead of the passed context, `kunit_get_current_test()` is used instead -//! (i.e. `current->kunit_test`). -//! -//! Note that this means other threads/tasks potentially spawned by a given test, if failing, will -//! report the failure in the kernel log but will not fail the actual test. Saving the pointer in -//! e.g. a `static` per test does not fully solve the issue either, because currently KUnit does -//! not support assertions (only expectations) from other tasks. Thus leave that feature for -//! the future, which simplifies the code here too. We could also simply not allow `assert`s in -//! other tasks, but that seems overly constraining, and we do want to support them, eventually. - -use std::{ - fs, - fs::File, - io::{BufWriter, Read, Write}, - path::{Path, PathBuf}, -}; - -/// Find the real path to the original file based on the `file` portion of the test name. -/// -/// `rustdoc` generated `file`s look like `sync_locked_by_rs`. Underscores (except the last one) -/// may represent an actual underscore in a directory/file, or a path separator. Thus the actual -/// file might be `sync_locked_by.rs`, `sync/locked_by.rs`, `sync_locked/by.rs` or -/// `sync/locked/by.rs`. This function walks the file system to determine which is the real one. -/// -/// This does require that ambiguities do not exist, but that seems fair, especially since this is -/// all supposed to be temporary until `rustdoc` gives us proper metadata to build this. If such -/// ambiguities are detected, they are diagnosed and the script panics. -fn find_real_path<'a>(srctree: &Path, valid_paths: &'a mut Vec<PathBuf>, file: &str) -> &'a str { - valid_paths.clear(); - - let potential_components: Vec<&str> = file.strip_suffix("_rs").unwrap().split('_').collect(); - - find_candidates(srctree, valid_paths, Path::new(""), &potential_components); - fn find_candidates( - srctree: &Path, - valid_paths: &mut Vec<PathBuf>, - prefix: &Path, - potential_components: &[&str], - ) { - // The base case: check whether all the potential components left, joined by underscores, - // is a file. - let joined_potential_components = potential_components.join("_") + ".rs"; - if srctree - .join("rust/kernel") - .join(prefix) - .join(&joined_potential_components) - .is_file() - { - // Avoid `srctree` here in order to keep paths relative to it in the KTAP output. - valid_paths.push( - Path::new("rust/kernel") - .join(prefix) - .join(joined_potential_components), - ); - } - - // In addition, check whether each component prefix, joined by underscores, is a directory. - // If not, there is no need to check for combinations with that prefix. - for i in 1..potential_components.len() { - let (components_prefix, components_rest) = potential_components.split_at(i); - let prefix = prefix.join(components_prefix.join("_")); - if srctree.join("rust/kernel").join(&prefix).is_dir() { - find_candidates(srctree, valid_paths, &prefix, components_rest); - } - } - } - - match valid_paths.as_slice() { - [] => panic!( - "No path candidates found for `{file}`. This is likely a bug in the build system, or \ - some files went away while compiling." - ), - [valid_path] => valid_path.to_str().unwrap(), - valid_paths => { - use std::fmt::Write; - - let mut candidates = String::new(); - for path in valid_paths { - writeln!(&mut candidates, " {path:?}").unwrap(); - } - panic!( - "Several path candidates found for `{file}`, please resolve the ambiguity by \ - renaming a file or folder. Candidates:\n{candidates}", - ); - } - } -} - -fn main() { - let srctree = std::env::var("srctree").unwrap(); - let srctree = Path::new(&srctree); - - let mut paths = fs::read_dir("rust/test/doctests/kernel") - .unwrap() - .map(|entry| entry.unwrap().path()) - .collect::<Vec<_>>(); - - // Sort paths. - paths.sort(); - - let mut rust_tests = String::new(); - let mut c_test_declarations = String::new(); - let mut c_test_cases = String::new(); - let mut body = String::new(); - let mut last_file = String::new(); - let mut number = 0; - let mut valid_paths: Vec<PathBuf> = Vec::new(); - let mut real_path: &str = ""; - for path in paths { - // The `name` follows the `{file}_{line}_{number}` pattern (see description in - // `scripts/rustdoc_test_builder.rs`). Discard the `number`. - let name = path.file_name().unwrap().to_str().unwrap().to_string(); - - // Extract the `file` and the `line`, discarding the `number`. - let (file, line) = name.rsplit_once('_').unwrap().0.rsplit_once('_').unwrap(); - - // Generate an ID sequence ("test number") for each one in the file. - if file == last_file { - number += 1; - } else { - number = 0; - last_file = file.to_string(); - - // Figure out the real path, only once per file. - real_path = find_real_path(srctree, &mut valid_paths, file); - } - - // Generate a KUnit name (i.e. test name and C symbol) for this test. - // - // We avoid the line number, like `rustdoc` does, to make things slightly more stable for - // bisection purposes. However, to aid developers in mapping back what test failed, we will - // print a diagnostics line in the KTAP report. - let kunit_name = format!("rust_doctest_kernel_{file}_{number}"); - - // Read the test's text contents to dump it below. - body.clear(); - File::open(path).unwrap().read_to_string(&mut body).unwrap(); - - // Calculate how many lines before `main` function (including the `main` function line). - let body_offset = body - .lines() - .take_while(|line| !line.contains("fn main() {")) - .count() - + 1; - - use std::fmt::Write; - write!( - rust_tests, - r#"/// Generated `{name}` KUnit test case from a Rust documentation test. -#[no_mangle] -pub extern "C" fn {kunit_name}(__kunit_test: *mut ::kernel::bindings::kunit) {{ - /// Overrides the usual [`assert!`] macro with one that calls KUnit instead. - #[allow(unused)] - macro_rules! assert {{ - ($cond:expr $(,)?) => {{{{ - ::kernel::kunit_assert!( - "{kunit_name}", "{real_path}", __DOCTEST_ANCHOR - {line}, $cond - ); - }}}} - }} - - /// Overrides the usual [`assert_eq!`] macro with one that calls KUnit instead. - #[allow(unused)] - macro_rules! assert_eq {{ - ($left:expr, $right:expr $(,)?) => {{{{ - ::kernel::kunit_assert_eq!( - "{kunit_name}", "{real_path}", __DOCTEST_ANCHOR - {line}, $left, $right - ); - }}}} - }} - - // Many tests need the prelude, so provide it by default. - #[allow(unused)] - use ::kernel::prelude::*; - - // Unconditionally print the location of the original doctest (i.e. rather than the location in - // the generated file) so that developers can easily map the test back to the source code. - // - // This information is also printed when assertions fail, but this helps in the successful cases - // when the user is running KUnit manually, or when passing `--raw_output` to `kunit.py`. - // - // This follows the syntax for declaring test metadata in the proposed KTAP v2 spec, which may - // be used for the proposed KUnit test attributes API. Thus hopefully this will make migration - // easier later on. - ::kernel::kunit::info(fmt!(" # {kunit_name}.location: {real_path}:{line}\n")); - - /// The anchor where the test code body starts. - #[allow(unused)] - static __DOCTEST_ANCHOR: i32 = ::core::line!() as i32 + {body_offset} + 1; - {{ - {body} - main(); - }} -}} - -"# - ) - .unwrap(); - - write!(c_test_declarations, "void {kunit_name}(struct kunit *);\n").unwrap(); - write!(c_test_cases, " KUNIT_CASE({kunit_name}),\n").unwrap(); - } - - let rust_tests = rust_tests.trim(); - let c_test_declarations = c_test_declarations.trim(); - let c_test_cases = c_test_cases.trim(); - - write!( - BufWriter::new(File::create("rust/doctests_kernel_generated.rs").unwrap()), - r#"//! `kernel` crate documentation tests. - -const __LOG_PREFIX: &[u8] = b"rust_doctests_kernel\0"; - -{rust_tests} -"# - ) - .unwrap(); - - write!( - BufWriter::new(File::create("rust/doctests_kernel_generated_kunit.c").unwrap()), - r#"/* - * `kernel` crate documentation tests. - */ - -#include <kunit/test.h> - -{c_test_declarations} - -static struct kunit_case test_cases[] = {{ - {c_test_cases} - {{ }} -}}; - -static struct kunit_suite test_suite = {{ - .name = "rust_doctests_kernel", - .test_cases = test_cases, -}}; - -kunit_test_suite(test_suite); - -MODULE_LICENSE("GPL"); -"# - ) - .unwrap(); -} base-commit: 0d97f2067c166eb495771fede9f7b73999c67f66 -- 2.51.0

1 day

2
1
0 0

[PATCH bpf-next v3 0/3] bpf: Add overwrite mode for BPF ring buffer

by Xu Kuohai

When the BPF ring buffer is full, a new event cannot be recorded until one or more old events are consumed to make enough space for it. In cases such as fault diagnostics, where recent events are more useful than older ones, this mechanism may lead to critical events being lost. So add overwrite mode for BPF ring buffer to address it. In this mode, the new event overwrites the oldest event when the buffer is full. v3: - remove half-round wakeup, drop unnecessary min in ringbuf_avail_data_sz(), switch to smp_load_acquire, update tests and fix typos, etc (Andrii) - rebase and re-collect performance data v2: https://lore.kernel.org/bpf/20250905150641.2078838-1-xukuohai@huaweicloud.c… - remove libbpf changes (Andrii) - update overwrite benchmark v1: https://lore.kernel.org/bpf/20250804022101.2171981-1-xukuohai@huaweicloud.c… Xu Kuohai (3): bpf: Add overwrite mode for BPF ring buffer selftests/bpf: Add overwrite mode test for BPF ring buffer selftests/bpf/benchs: Add overwrite mode benchmark for BPF ring buffer include/uapi/linux/bpf.h | 4 + kernel/bpf/ringbuf.c | 109 +++++++++++++++--- tools/include/uapi/linux/bpf.h | 4 + tools/testing/selftests/bpf/Makefile | 3 +- .../selftests/bpf/benchs/bench_ringbufs.c | 66 ++++++++++- .../bpf/benchs/run_bench_ringbufs.sh | 4 + .../selftests/bpf/prog_tests/ringbuf.c | 64 ++++++++++ .../selftests/bpf/progs/ringbuf_bench.c | 11 ++ .../bpf/progs/test_ringbuf_overwrite.c | 98 ++++++++++++++++ 9 files changed, 337 insertions(+), 26 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/test_ringbuf_overwrite.c -- 2.43.0

1 day, 3 hours

1
3
0 0

[PATCH bpf-next 0/5] selftests/bpf: convert test_tc_tunnel.sh to test_progs

by Alexis Lothoré (eBPF Foundation)

Hello, this series aims to convert another test to the test_progs framework to make sure that it is executed in CI for series sent on the mailing list. test_tc_tunnel.sh tests a variety of tunnels based on BPF: packets are encapsulated by a BPF program on the client egress. We then check that those packets can be decapsulated on server ingress side, either thanks to kernel-based or BPF-based decapsulation. Those tests are run thanks to two veths in two dedicated namespaces. - patches 1 to 3 are preparatory patches - patch 4 introduce tc_tunnel test into test_progs - patch 5 gets rid of the test_tc_tunnel.sh script The new test has been executed both in some x86 local qemu machine, as well as in CI: # ./test_progs -a tc_tunnel #454/1 tc_tunnel/ipip_none:OK #454/2 tc_tunnel/ipip6_none:OK #454/3 tc_tunnel/ip6tnl_none:OK #454/4 tc_tunnel/sit_none:OK #454/5 tc_tunnel/vxlan_eth:OK #454/6 tc_tunnel/ip6vxlan_eth:OK #454/7 tc_tunnel/gre_none:OK #454/8 tc_tunnel/gre_eth:OK #454/9 tc_tunnel/gre_mpls:OK #454/10 tc_tunnel/ip6gre_none:OK #454/11 tc_tunnel/ip6gre_eth:OK #454/12 tc_tunnel/ip6gre_mpls:OK #454/13 tc_tunnel/udp_none:OK #454/14 tc_tunnel/udp_eth:OK #454/15 tc_tunnel/udp_mpls:OK #454/16 tc_tunnel/ip6udp_none:OK #454/17 tc_tunnel/ip6udp_eth:OK #454/18 tc_tunnel/ip6udp_mpls:OK #454 tc_tunnel:OK Summary: 1/18 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Alexis Lothoré (eBPF Foundation) <alexis.lothore(a)bootlin.com> --- Alexis Lothoré (eBPF Foundation) (5): testing/selftests: rename tc_helpers.h to tcx_helpers.h selftests/bpf: add tc helpers selftests/bpf: make test_tc_tunnel.bpf.c compatible with big endian platforms selftests/bpf: integrate test_tc_tunnel.sh tests into test_progs selftests/bpf: remove test_tc_tunnel.sh tools/testing/selftests/bpf/Makefile | 2 +- tools/testing/selftests/bpf/prog_tests/tc_links.c | 46 +- tools/testing/selftests/bpf/prog_tests/tc_netkit.c | 22 +- tools/testing/selftests/bpf/prog_tests/tc_opts.c | 40 +- .../bpf/prog_tests/{tc_helpers.h => tcx_helpers.h} | 6 +- .../selftests/bpf/prog_tests/test_tc_tunnel.c | 684 +++++++++++++++++++++ .../testing/selftests/bpf/prog_tests/test_tunnel.c | 80 +-- tools/testing/selftests/bpf/progs/test_tc_tunnel.c | 99 ++- tools/testing/selftests/bpf/tc_helpers.c | 87 +++ tools/testing/selftests/bpf/tc_helpers.h | 9 + tools/testing/selftests/bpf/test_tc_tunnel.sh | 320 ---------- 11 files changed, 884 insertions(+), 511 deletions(-) --- base-commit: 22267893b8c7f2773896e814800bbe693f206e0c change-id: 20250811-tc_tunnel-c61342683f18 Best regards, -- Alexis Lothoré, Bootlin Embedded Linux and Kernel engineering https://bootlin.com

1 day, 7 hours

2
8
0 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

Linux-kselftest-mirror