The current implementation of netconsole sends all log messages in parallel, which can lead to an intermixed and interleaved output on the receiving side. This makes it challenging to demultiplex the messages and attribute them to their originating CPUs.
As a result, users and developers often struggle to effectively analyze and debug the parallel log output received through netconsole.
Example of a message got from produciton hosts:
------------[ cut here ]------------ ------------[ cut here ]------------ refcount_t: saturated; leaking memory. WARNING: CPU: 2 PID: 1613668 at lib/refcount.c:22 refcount_warn_saturate+0x5e/0xe0 refcount_t: addition on 0; use-after-free. WARNING: CPU: 26 PID: 4139916 at lib/refcount.c:25 refcount_warn_saturate+0x7d/0xe0 Modules linked in: bpf_preload(E) vhost_net(E) tun(E) vhost(E)
This series of patches introduces a new feature to the netconsole subsystem that allows the automatic population of the CPU number in the userdata field for each log message. This enhancement provides several benefits:
* Improved demultiplexing of parallel log output: When multiple CPUs are sending messages concurrently, the added CPU number in the userdata makes it easier to differentiate and attribute the messages to their originating CPUs.
* Better visibility into message sources: The CPU number information gives users and developers more insight into which specific CPU a particular log message came from, which can be valuable for debugging and analysis.
The changes in this series are as follows:
Patch 1: netconsole: Rename userdata to extradata ================================================= Create the a concept of extradata, which encompasses the concept of userdata and the upcoming sysdatao
Sysdata is a new concept being added, which is basically fields that are populated by the kernel. At this time only the CPU#, but, there is a desire to add current task name, kernel release version, etc.
Patch 2: netconsole: Helper to count number of used entries =========================================================== Create a simple helper to count number of entries in extradata. I am separating this in a function since it will need to count userdata and sysdata. For instance, when the user adds an extra userdata, we need to check if there is space, counting the previous data entries (from userdata and cpu data)
Patch 3: netconsole: add support for sysdata and CPU population =============================================================== This is the core patch. Basically add a new option to enable automatic CPU number population in the netconsole userdata Provides a new "cpu_nr" sysfs attribute to control this feature
Patch 4: "netconsole: selftest: test CPU number auto-population" ============================================================= Expands the existing netconsole selftest to verify the CPU number auto-population functionality Ensures the received netconsole messages contain the expected "cpu=<CPU>" entry in the message. Test different permutation with userdata
Patch 5: "netconsole: docs: Add documentation for CPU number auto-population" ============================================================================= Updates the netconsole documentation to explain the new CPU number auto-population feature Provides instructions on how to enable and use the feature
I believe these changes will be a valuable addition to the netconsole subsystem, enhancing its usefulness for kernel developers and users.
Signed-off-by: Breno Leitao leitao@debian.org --- Changes in v2: - Create the concept of extradata and sysdata. This will make the design easier to understand, and the code easier to read. * Basically extradata encompasses userdata and the new sysdata. Userdata originates from user, and sysdata originates in kernel. - Improved the test to send from a very specific CPU, which can be checked to be correct on the other side, as suggested by Jakub. - Fixed a bug where CPU # was populated at the wrong place - Link to v1: https://lore.kernel.org/r/20241113-netcon_cpu-v1-0-d187bf7c0321@debian.org
--- Breno Leitao (5): netconsole: Rename userdata to extradata netconsole: Helper to count number of used entries netconsole: add support for sysdata and CPU population netconsole: selftest: test for sysdata CPU netconsole: docs: Add documentation for CPU number auto-population
Documentation/networking/netconsole.rst | 45 +++++ drivers/net/netconsole.c | 223 ++++++++++++++++----- tools/testing/selftests/drivers/net/Makefile | 1 + .../selftests/drivers/net/lib/sh/lib_netcons.sh | 17 ++ .../selftests/drivers/net/netcons_sysdata.sh | 166 +++++++++++++++ 5 files changed, 407 insertions(+), 45 deletions(-) --- base-commit: 7b24f164cf005b9649138ef6de94aaac49c9f3d1 change-id: 20241108-netcon_cpu-ce3917e88f4b
Best regards,