Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git Commit: dfe283e9fdac - Linux 5.3.9-rc1
The results of these automated tests are provided below.
Overall result: FAILED (see details below) Merge: OK Compile: OK Tests: FAILED
All kernel binaries, config files, and logs are available for download here:
https://artifacts.cki-project.org/pipelines/262380
One or more kernel tests failed:
x86_64: ❌ LTP lite
We hope that these logs can help you find the problem quickly. For the full detail on our testing procedures, please scroll to the bottom of this message.
Please reply to this email if you have any questions about the tests that we ran or if you have any suggestions on how to make future tests more effective.
,-. ,-. ( C ) ( K ) Continuous `-',-.`-' Kernel ( I ) Integration `-' ______________________________________________________________________________
Compile testing ---------------
We compiled the kernel for 3 architectures:
aarch64: make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
ppc64le: make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
x86_64: make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
Hardware testing ---------------- We booted each kernel and ran the following tests:
aarch64: Host 1: ✅ Boot test ✅ Podman system integration test (as root) ✅ Podman system integration test (as user) ✅ LTP lite ✅ Loopdev Sanity ✅ jvm test suite ✅ Memory function: memfd_create ✅ Memory function: kaslr ✅ AMTU (Abstract Machine Test Utility) ✅ LTP: openposix test suite ✅ Ethernet drivers sanity ✅ Networking MACsec: sanity ✅ Networking socket: fuzz ✅ Networking sctp-auth: sockopts test ✅ Networking: igmp conformance test ✅ Networking route: pmtu ✅ Networking TCP: keepalive test ✅ Networking UDP: socket ✅ Networking tunnel: geneve basic test ✅ Networking tunnel: gre basic ✅ Networking tunnel: vxlan basic ✅ audit: audit testsuite test ✅ httpd: mod_ssl smoke sanity ✅ iotop: sanity ✅ tuned: tune-processes-through-perf ✅ ALSA PCM loopback test ✅ ALSA Control (mixer) Userspace Element test ✅ Usex - version 1.9-29 ✅ storage: SCSI VPD ✅ trace: ftrace/tracer 🚧 ✅ CIFS Connectathon 🚧 ✅ POSIX pjd-fstest suites 🚧 ✅ Networking bridge: sanity 🚧 ✅ Networking route_func: local ✅ Networking route_func: forward 🚧 ✅ L2TP basic test 🚧 ✅ Networking vnic: ipvlan/basic 🚧 ✅ storage: dm/common 🚧 ✅ Networking ipsec: basic netns transport 🚧 ✅ Networking ipsec: basic netns tunnel
Host 2: ✅ Boot test ✅ xfstests: ext4 ✅ xfstests: xfs ✅ selinux-policy: serge-testsuite ✅ lvm thinp sanity ✅ storage: software RAID testing 🚧 ✅ Storage blktests
ppc64le: Host 1: ✅ Boot test ✅ Podman system integration test (as root) ✅ Podman system integration test (as user) ✅ LTP lite ✅ Loopdev Sanity ✅ jvm test suite ✅ Memory function: memfd_create ✅ Memory function: kaslr ✅ AMTU (Abstract Machine Test Utility) ✅ LTP: openposix test suite ✅ Ethernet drivers sanity ✅ Networking MACsec: sanity ✅ Networking socket: fuzz ✅ Networking sctp-auth: sockopts test ✅ Networking route: pmtu ✅ Networking TCP: keepalive test ✅ Networking UDP: socket ✅ Networking tunnel: geneve basic test ✅ Networking tunnel: gre basic ✅ Networking tunnel: vxlan basic ✅ audit: audit testsuite test ✅ httpd: mod_ssl smoke sanity ✅ iotop: sanity ✅ tuned: tune-processes-through-perf ✅ ALSA PCM loopback test ✅ ALSA Control (mixer) Userspace Element test ✅ Usex - version 1.9-29 ✅ trace: ftrace/tracer 🚧 ✅ CIFS Connectathon 🚧 ✅ POSIX pjd-fstest suites 🚧 ✅ Networking bridge: sanity 🚧 ✅ Networking route_func: local ✅ Networking route_func: forward 🚧 ✅ L2TP basic test 🚧 ✅ Networking ipsec: basic netns tunnel 🚧 ✅ Networking vnic: ipvlan/basic 🚧 ✅ storage: dm/common
Host 2: ✅ Boot test ✅ xfstests: ext4 ✅ xfstests: xfs ✅ selinux-policy: serge-testsuite ✅ lvm thinp sanity ✅ storage: software RAID testing 🚧 ✅ Storage blktests
x86_64: Host 1: ✅ Boot test 🚧 ✅ IPMI driver test 🚧 ✅ IPMItool loop stress test
Host 2: ✅ Boot test ✅ Podman system integration test (as root) ✅ Podman system integration test (as user) ❌ LTP lite ✅ Loopdev Sanity ✅ jvm test suite ✅ Memory function: memfd_create ✅ Memory function: kaslr ✅ AMTU (Abstract Machine Test Utility) ✅ LTP: openposix test suite ✅ Ethernet drivers sanity ✅ Networking MACsec: sanity ✅ Networking socket: fuzz ✅ Networking sctp-auth: sockopts test ✅ Networking: igmp conformance test ✅ Networking route: pmtu ✅ Networking TCP: keepalive test ✅ Networking UDP: socket ✅ Networking tunnel: geneve basic test ✅ Networking tunnel: gre basic ✅ Networking tunnel: vxlan basic ✅ audit: audit testsuite test ✅ httpd: mod_ssl smoke sanity ✅ iotop: sanity ✅ tuned: tune-processes-through-perf ✅ pciutils: sanity smoke test ✅ ALSA PCM loopback test ✅ ALSA Control (mixer) Userspace Element test ✅ Usex - version 1.9-29 ✅ storage: SCSI VPD ✅ stress: stress-ng ✅ trace: ftrace/tracer 🚧 ✅ CIFS Connectathon 🚧 ✅ POSIX pjd-fstest suites 🚧 ✅ Networking bridge: sanity 🚧 ✅ Networking route_func: local ✅ Networking route_func: forward 🚧 ✅ L2TP basic test 🚧 ✅ Networking vnic: ipvlan/basic 🚧 ✅ storage: dm/common 🚧 ✅ Networking ipsec: basic netns transport 🚧 ✅ Networking ipsec: basic netns tunnel
Host 3: ✅ Boot test ✅ xfstests: ext4 ✅ xfstests: xfs ✅ selinux-policy: serge-testsuite ✅ lvm thinp sanity ✅ storage: software RAID testing 🚧 ✅ IOMMU boot test 🚧 ✅ Storage blktests
Test sources: https://github.com/CKI-project/tests-beaker 💚 Pull requests are welcome for new tests or improvements to existing tests!
Waived tests ------------ If the test run included waived tests, they are marked with 🚧. Such tests are executed but their results are not taken into account. Tests are waived when their results are not reliable enough, e.g. when they're just introduced or are being fixed.
Testing timeout --------------- We aim to provide a report within reasonable timeframe. Tests that haven't finished running are marked with ⏱. Reports for non-upstream kernels have a Beaker recipe linked to next to each host.
----- Original Message -----
Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git Commit: dfe283e9fdac - Linux 5.3.9-rc1The results of these automated tests are provided below.
Overall result: FAILED (see details below) Merge: OK Compile: OK Tests: FAILEDAll kernel binaries, config files, and logs are available for download here:
https://artifacts.cki-project.org/pipelines/262380
One or more kernel tests failed:
x86_64: ❌ LTP lite
Not a 5.3 -stable regression.
Failure comes from test that sanity checks all /proc files by doing 1k read from each. There are couple issues it hits wrt. snd_hda_*.
Example reproducer: dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access of=out.txt count=1 bs=1024 iflag=nonblock
It's slow and triggers soft lockups [1]. And it also requires lot of memory, triggering OOMs on smaller VMs: 0x0000000024f0437b-0x000000001a32b1c8 1073745920 seq_read+0x131/0x400 pages=262144 vmalloc vpages N0=262144
I'm leaning towards skipping all regmap entries in this test. Comments are welcomed.
Regards, Jan
[1] [15341.137116] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [dd:636573] [15341.144141] Modules linked in: nls_utf8 isofs dummy minix binfmt_misc nfsv3 nfs_acl nfs lockd grace fscache sctp rds brd vfat fat btrfs xor zstd_compress raid6_pq zstd_decompress loop tu n ip6table_nat ip6_tables xt_conntrack iptable_filter xt_MASQUERADE xt_comment iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 veth bridge stp llc overlay fuse snd_hda_codec_r ealtek snd_hda_codec_generic sunrpc ledtrig_audio snd_hda_codec_hdmi snd_hda_intel snd_hda_codec edac_mce_amd kvm_amd snd_hda_core ccp snd_hwdep snd_pcm kvm snd_timer irqbypass joydev snd c rct10dif_pclmul crc32_pclmul soundcore broadcom bcm_phy_lib ghash_clmulni_intel sp5100_tco fam15h_power k10temp tg3 wmi_bmof pcspkr i2c_piix4 acpi_cpufreq ip_tables xfs libcrc32c radeon ata _generic i2c_algo_bit pata_acpi drm_kms_helper firewire_ohci ttm crc32c_intel serio_raw pata_atiixp firewire_core drm crc_itu_t wmi [last unloaded: can] [15341.223635] CPU: 2 PID: 636573 Comm: dd Tainted: G L 5.3.9-rc1-dfe283e.cki #1 [15341.232073] Hardware name: AMD Pike/Pike, BIOS RPK1506A 09/03/2014 [15341.238467] RIP: 0010:regmap_readable+0x15/0x60 [15341.242996] Code: 25 28 00 00 00 75 07 48 83 c4 10 5b 5d c3 e8 92 08 a6 ff 66 90 0f 1f 44 00 00 48 83 bf b0 01 00 00 00 74 31 8b 97 48 01 00 00 <39> f2 0f 92 c0 85 d2 0f 95 c2 20 d0 75 1 d 48 83 7f 70 00 74 01 c3 [15341.261765] RSP: 0018:ffffb3b100697dc8 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13 [15341.269330] RAX: 0000000000000000 RBX: ffff8d1a63773400 RCX: 0000000000000b41 [15341.276723] RDX: 000000000fffffff RSI: 000000000f3b4139 RDI: ffff8d1a63773400 [15341.283858] RBP: 000000000f3b4139 R08: 0000000000000000 R09: 0000000000000000 [15341.290989] R10: 000000000000000f R11: ffff8d19901fffff R12: 0000000000000079 [15341.298114] R13: 000000000f3b4139 R14: 00000000ffffffff R15: 000000000000006e [15341.305501] FS: 00007f4e067a0580(0000) GS:ffff8d1a6aa80000(0000) knlGS:0000000000000000 [15341.313583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15341.319322] CR2: 00007f5083682dc0 CR3: 00000001287dc000 CR4: 00000000000406e0 [15341.326462] Call Trace: [15341.328912] regmap_volatile+0x4f/0xa0 [15341.332667] regmap_access_show+0x70/0x100 [15341.336997] seq_read+0xcd/0x400 [15341.340247] full_proxy_read+0x57/0x70 [15341.343997] vfs_read+0x9d/0x150 [15341.347231] ksys_read+0x5f/0xe0 [15341.350473] do_syscall_64+0x5f/0x1a0 [15341.354166] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [15341.359216] RIP: 0033:0x7f4e066823c2 [15341.362795] Code: c0 e9 c2 fe ff ff 50 48 8d 3d c2 0d 0a 00 e8 b5 f1 01 00 0f 1f 44 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 0 0 48 83 ec 28 48 89 54 24 [15341.381852] RSP: 002b:00007ffc6ad30e88 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [15341.389427] RAX: ffffffffffffffda RBX: 0000000000000400 RCX: 00007f4e066823c2 [15341.396568] RDX: 0000000000000400 RSI: 0000561bc69f0000 RDI: 0000000000000000 [15341.403979] RBP: 0000561bc69f0000 R08: 0000561bc69efb60 R09: 00000000000000c0 [15341.411111] R10: 0000561bc69f0000 R11: 0000000000000246 R12: ffffffffffffffff [15341.418244] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffffffffffff
On Mon, Nov 04, 2019 at 08:35:51AM -0500, Jan Stancek wrote:
----- Original Message -----
Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git Commit: dfe283e9fdac - Linux 5.3.9-rc1The results of these automated tests are provided below.
Overall result: FAILED (see details below) Merge: OK Compile: OK Tests: FAILEDAll kernel binaries, config files, and logs are available for download here:
https://artifacts.cki-project.org/pipelines/262380
One or more kernel tests failed:
x86_64: ❌ LTP liteNot a 5.3 -stable regression.
Failure comes from test that sanity checks all /proc files by doing 1k read from each. There are couple issues it hits wrt. snd_hda_*.
Example reproducer: dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access of=out.txt count=1 bs=1024 iflag=nonblock
That's not a proc file :)
It's slow and triggers soft lockups [1]. And it also requires lot of memory, triggering OOMs on smaller VMs: 0x0000000024f0437b-0x000000001a32b1c8 1073745920 seq_read+0x131/0x400 pages=262144 vmalloc vpages N0=262144
I'm leaning towards skipping all regmap entries in this test. Comments are welcomed.
Randomly poking around in debugfs is a sure way to cause crashes and major problems. Also, debugfs files are NOT stable and only for debugging and should never be enabled on "real" systems.
So what exactly is the test trying to do here?
thanks,
greg k-h
----- Original Message -----
On Mon, Nov 04, 2019 at 08:35:51AM -0500, Jan Stancek wrote:
----- Original Message -----
Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git Commit: dfe283e9fdac - Linux 5.3.9-rc1The results of these automated tests are provided below.
Overall result: FAILED (see details below) Merge: OK Compile: OK Tests: FAILEDAll kernel binaries, config files, and logs are available for download here:
https://artifacts.cki-project.org/pipelines/262380
One or more kernel tests failed:
x86_64: ❌ LTP liteNot a 5.3 -stable regression.
Failure comes from test that sanity checks all /proc files by doing 1k read from each. There are couple issues it hits wrt. snd_hda_*.
Example reproducer: dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access of=out.txt count=1 bs=1024 iflag=nonblock
That's not a proc file :)
Right. It's same test that's used for /proc too.
It's slow and triggers soft lockups [1]. And it also requires lot of memory, triggering OOMs on smaller VMs: 0x0000000024f0437b-0x000000001a32b1c8 1073745920 seq_read+0x131/0x400 pages=262144 vmalloc vpages N0=262144
I'm leaning towards skipping all regmap entries in this test. Comments are welcomed.
Randomly poking around in debugfs is a sure way to cause crashes and major problems. Also, debugfs files are NOT stable and only for debugging and should never be enabled on "real" systems.
So what exactly is the test trying to do here?
It's (unprivileged) user trying to open/read anything it can (/proc, /sys) to see if that triggers anything bad.
It can run as privileged user too, which was the case above.
[1] https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/fs/re...
On Mon, Nov 04, 2019 at 09:28:10AM -0500, Jan Stancek wrote:
----- Original Message -----
On Mon, Nov 04, 2019 at 08:35:51AM -0500, Jan Stancek wrote:
----- Original Message -----
Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git Commit: dfe283e9fdac - Linux 5.3.9-rc1The results of these automated tests are provided below.
Overall result: FAILED (see details below) Merge: OK Compile: OK Tests: FAILEDAll kernel binaries, config files, and logs are available for download here:
https://artifacts.cki-project.org/pipelines/262380
One or more kernel tests failed:
x86_64: ❌ LTP liteNot a 5.3 -stable regression.
Failure comes from test that sanity checks all /proc files by doing 1k read from each. There are couple issues it hits wrt. snd_hda_*.
Example reproducer: dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access of=out.txt count=1 bs=1024 iflag=nonblock
That's not a proc file :)
Right. It's same test that's used for /proc too.
It's slow and triggers soft lockups [1]. And it also requires lot of memory, triggering OOMs on smaller VMs: 0x0000000024f0437b-0x000000001a32b1c8 1073745920 seq_read+0x131/0x400 pages=262144 vmalloc vpages N0=262144
I'm leaning towards skipping all regmap entries in this test. Comments are welcomed.
Randomly poking around in debugfs is a sure way to cause crashes and major problems. Also, debugfs files are NOT stable and only for debugging and should never be enabled on "real" systems.
So what exactly is the test trying to do here?
It's (unprivileged) user trying to open/read anything it can (/proc, /sys) to see if that triggers anything bad.
It can run as privileged user too, which was the case above.
Sure, you can do tons of bad things as root poking around in sysfs, debugfs, and procfs. What exactly are you trying to do, break the system?
That sounds like a horrible test that is just setting itself up to lock the system up, you should revisit it...
thanks,
greg k-h
----- Original Message -----
On Mon, Nov 04, 2019 at 09:28:10AM -0500, Jan Stancek wrote:
----- Original Message -----
On Mon, Nov 04, 2019 at 08:35:51AM -0500, Jan Stancek wrote:
----- Original Message -----
Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git Commit: dfe283e9fdac - Linux 5.3.9-rc1The results of these automated tests are provided below.
Overall result: FAILED (see details below) Merge: OK Compile: OK Tests: FAILEDAll kernel binaries, config files, and logs are available for download here:
https://artifacts.cki-project.org/pipelines/262380
One or more kernel tests failed:
x86_64: ❌ LTP liteNot a 5.3 -stable regression.
Failure comes from test that sanity checks all /proc files by doing 1k read from each. There are couple issues it hits wrt. snd_hda_*.
Example reproducer: dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access of=out.txt count=1 bs=1024 iflag=nonblock
That's not a proc file :)
Right. It's same test that's used for /proc too.
It's slow and triggers soft lockups [1]. And it also requires lot of memory, triggering OOMs on smaller VMs: 0x0000000024f0437b-0x000000001a32b1c8 1073745920 seq_read+0x131/0x400 pages=262144 vmalloc vpages N0=262144
I'm leaning towards skipping all regmap entries in this test. Comments are welcomed.
Randomly poking around in debugfs is a sure way to cause crashes and major problems. Also, debugfs files are NOT stable and only for debugging and should never be enabled on "real" systems.
So what exactly is the test trying to do here?
It's (unprivileged) user trying to open/read anything it can (/proc, /sys) to see if that triggers anything bad.
It can run as privileged user too, which was the case above.
Sure, you can do tons of bad things as root poking around in sysfs, debugfs, and procfs. What exactly are you trying to do, break the system?
We are talking about read-only here. Is it unreasonable to expect that root can read all /proc entries without crashing the system?
Some entries are readable only by root. So an unprivileged user will skip considerable number of entries: -r--------. 1 root root 0 Nov 4 16:13 /proc/slabinfo
That sounds like a horrible test that is just setting itself up to lock the system up, you should revisit it...
On Mon, Nov 04, 2019 at 10:25:21AM -0500, Jan Stancek wrote:
----- Original Message -----
On Mon, Nov 04, 2019 at 09:28:10AM -0500, Jan Stancek wrote:
----- Original Message -----
On Mon, Nov 04, 2019 at 08:35:51AM -0500, Jan Stancek wrote:
----- Original Message -----
Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git Commit: dfe283e9fdac - Linux 5.3.9-rc1The results of these automated tests are provided below.
Overall result: FAILED (see details below) Merge: OK Compile: OK Tests: FAILEDAll kernel binaries, config files, and logs are available for download here:
https://artifacts.cki-project.org/pipelines/262380
One or more kernel tests failed:
x86_64: ❌ LTP liteNot a 5.3 -stable regression.
Failure comes from test that sanity checks all /proc files by doing 1k read from each. There are couple issues it hits wrt. snd_hda_*.
Example reproducer: dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access of=out.txt count=1 bs=1024 iflag=nonblock
That's not a proc file :)
Right. It's same test that's used for /proc too.
It's slow and triggers soft lockups [1]. And it also requires lot of memory, triggering OOMs on smaller VMs: 0x0000000024f0437b-0x000000001a32b1c8 1073745920 seq_read+0x131/0x400 pages=262144 vmalloc vpages N0=262144
I'm leaning towards skipping all regmap entries in this test. Comments are welcomed.
Randomly poking around in debugfs is a sure way to cause crashes and major problems. Also, debugfs files are NOT stable and only for debugging and should never be enabled on "real" systems.
So what exactly is the test trying to do here?
It's (unprivileged) user trying to open/read anything it can (/proc, /sys) to see if that triggers anything bad.
It can run as privileged user too, which was the case above.
Sure, you can do tons of bad things as root poking around in sysfs, debugfs, and procfs. What exactly are you trying to do, break the system?
We are talking about read-only here. Is it unreasonable to expect that root can read all /proc entries without crashing the system?
You are NOT reading /proc/ here. You are reading debugfs which you really have NOT idea what is in there. As you saw, you are reading from hardware that is slow and doing odd things when you read from it.
And yes, there are some /proc/ files that you should not read from as root and expect things to always work. PCI devices are notorious for this, and if you are reading those files as root, you "know" you know what you are doing and can accept the risk for when things go wrong.
It is fine to write tests to read specific /proc/ files that you know what is happening in them. To pick random ones is never a good idea.
thanks,
greg k-h
----- Original Message -----
On Mon, Nov 04, 2019 at 10:25:21AM -0500, Jan Stancek wrote:
----- Original Message -----
On Mon, Nov 04, 2019 at 09:28:10AM -0500, Jan Stancek wrote:
----- Original Message -----
On Mon, Nov 04, 2019 at 08:35:51AM -0500, Jan Stancek wrote:
----- Original Message ----- > > Hello, > > We ran automated tests on a recent commit from this kernel tree: > > Kernel repo: > git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git > Commit: dfe283e9fdac - Linux 5.3.9-rc1 > > The results of these automated tests are provided below. > > Overall result: FAILED (see details below) > Merge: OK > Compile: OK > Tests: FAILED > > All kernel binaries, config files, and logs are available for > download > here: > > https://artifacts.cki-project.org/pipelines/262380 > > One or more kernel tests failed: > > x86_64: > ❌ LTP lite >
Not a 5.3 -stable regression.
Failure comes from test that sanity checks all /proc files by doing 1k read from each. There are couple issues it hits wrt. snd_hda_*.
Example reproducer: dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access of=out.txt count=1 bs=1024 iflag=nonblock
That's not a proc file :)
Right. It's same test that's used for /proc too.
It's slow and triggers soft lockups [1]. And it also requires lot of memory, triggering OOMs on smaller VMs: 0x0000000024f0437b-0x000000001a32b1c8 1073745920 seq_read+0x131/0x400 pages=262144 vmalloc vpages N0=262144
I'm leaning towards skipping all regmap entries in this test. Comments are welcomed.
Randomly poking around in debugfs is a sure way to cause crashes and major problems. Also, debugfs files are NOT stable and only for debugging and should never be enabled on "real" systems.
So what exactly is the test trying to do here?
It's (unprivileged) user trying to open/read anything it can (/proc, /sys) to see if that triggers anything bad.
It can run as privileged user too, which was the case above.
Sure, you can do tons of bad things as root poking around in sysfs, debugfs, and procfs. What exactly are you trying to do, break the system?
We are talking about read-only here. Is it unreasonable to expect that root can read all /proc entries without crashing the system?
You are NOT reading /proc/ here.
No. That was a general question to usefulness of privileged read, using /proc as example where it commonly happens.
You are reading debugfs which you really have NOT idea what is in there. As you saw, you are reading from hardware that is slow and doing odd things when you read from it.
Agreed, I already sent a patch to LTP to blacklist it.
And yes, there are some /proc/ files that you should not read from as root and expect things to always work. PCI devices are notorious for this, and if you are reading those files as root, you "know" you know what you are doing and can accept the risk for when things go wrong.
It is fine to write tests to read specific /proc/ files that you know what is happening in them. To pick random ones is never a good idea.
Thanks for example.
On Mon, Nov 04, 2019 at 12:02:53PM -0500, Jan Stancek wrote:
----- Original Message -----
On Mon, Nov 04, 2019 at 10:25:21AM -0500, Jan Stancek wrote:
----- Original Message -----
On Mon, Nov 04, 2019 at 09:28:10AM -0500, Jan Stancek wrote:
----- Original Message -----
On Mon, Nov 04, 2019 at 08:35:51AM -0500, Jan Stancek wrote: > > > ----- Original Message ----- > > > > Hello, > > > > We ran automated tests on a recent commit from this kernel tree: > > > > Kernel repo: > > git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git > > Commit: dfe283e9fdac - Linux 5.3.9-rc1 > > > > The results of these automated tests are provided below. > > > > Overall result: FAILED (see details below) > > Merge: OK > > Compile: OK > > Tests: FAILED > > > > All kernel binaries, config files, and logs are available for > > download > > here: > > > > https://artifacts.cki-project.org/pipelines/262380 > > > > One or more kernel tests failed: > > > > x86_64: > > ❌ LTP lite > > > > Not a 5.3 -stable regression. > > Failure comes from test that sanity checks all /proc files by doing > 1k read from each. There are couple issues it hits wrt. snd_hda_*. > > Example reproducer: > dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access > of=out.txt > count=1 bs=1024 iflag=nonblock
That's not a proc file :)
Right. It's same test that's used for /proc too.
> It's slow and triggers soft lockups [1]. And it also requires lot > of memory, triggering OOMs on smaller VMs: > 0x0000000024f0437b-0x000000001a32b1c8 1073745920 > seq_read+0x131/0x400 > pages=262144 vmalloc vpages N0=262144 > > I'm leaning towards skipping all regmap entries in this test. > Comments are welcomed.
Randomly poking around in debugfs is a sure way to cause crashes and major problems. Also, debugfs files are NOT stable and only for debugging and should never be enabled on "real" systems.
So what exactly is the test trying to do here?
It's (unprivileged) user trying to open/read anything it can (/proc, /sys) to see if that triggers anything bad.
It can run as privileged user too, which was the case above.
Sure, you can do tons of bad things as root poking around in sysfs, debugfs, and procfs. What exactly are you trying to do, break the system?
We are talking about read-only here. Is it unreasonable to expect that root can read all /proc entries without crashing the system?
You are NOT reading /proc/ here.
No. That was a general question to usefulness of privileged read, using /proc as example where it commonly happens.
You are reading debugfs which you really have NOT idea what is in there. As you saw, you are reading from hardware that is slow and doing odd things when you read from it.
Agreed, I already sent a patch to LTP to blacklist it.
Hopefully you are blacklisting all of debugfs. and sysfs. And procfs. Adding specific files back is fine, as long as you "know" they are ok and you are actually testing something valid there.
thanks,
greg k-h
Greg KH gregkh@linuxfoundation.org writes:
On Mon, Nov 04, 2019 at 12:02:53PM -0500, Jan Stancek wrote:
----- Original Message -----
On Mon, Nov 04, 2019 at 10:25:21AM -0500, Jan Stancek wrote:
----- Original Message -----
On Mon, Nov 04, 2019 at 09:28:10AM -0500, Jan Stancek wrote:
----- Original Message ----- > On Mon, Nov 04, 2019 at 08:35:51AM -0500, Jan Stancek wrote: > > > > > > ----- Original Message ----- > > > > > > Hello, > > > > > > We ran automated tests on a recent commit from this kernel tree: > > > > > > Kernel repo: > > > git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git > > > Commit: dfe283e9fdac - Linux 5.3.9-rc1 > > > > > > The results of these automated tests are provided below. > > > > > > Overall result: FAILED (see details below) > > > Merge: OK > > > Compile: OK > > > Tests: FAILED > > > > > > All kernel binaries, config files, and logs are available for > > > download > > > here: > > > > > > https://artifacts.cki-project.org/pipelines/262380 > > > > > > One or more kernel tests failed: > > > > > > x86_64: > > > ❌ LTP lite > > > > > > > Not a 5.3 -stable regression. > > > > Failure comes from test that sanity checks all /proc files by doing > > 1k read from each. There are couple issues it hits wrt. snd_hda_*. > > > > Example reproducer: > > dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access > > of=out.txt > > count=1 bs=1024 iflag=nonblock > > That's not a proc file :)
Right. It's same test that's used for /proc too.
> > > It's slow and triggers soft lockups [1]. And it also requires lot > > of memory, triggering OOMs on smaller VMs: > > 0x0000000024f0437b-0x000000001a32b1c8 1073745920 > > seq_read+0x131/0x400 > > pages=262144 vmalloc vpages N0=262144 > > > > I'm leaning towards skipping all regmap entries in this test. > > Comments are welcomed. > > Randomly poking around in debugfs is a sure way to cause crashes and > major problems. Also, debugfs files are NOT stable and only for > debugging and should never be enabled on "real" systems. > > So what exactly is the test trying to do here?
It's (unprivileged) user trying to open/read anything it can (/proc, /sys) to see if that triggers anything bad.
It can run as privileged user too, which was the case above.
Sure, you can do tons of bad things as root poking around in sysfs, debugfs, and procfs. What exactly are you trying to do, break the system?
We are talking about read-only here. Is it unreasonable to expect that root can read all /proc entries without crashing the system?
You are NOT reading /proc/ here.
No. That was a general question to usefulness of privileged read, using /proc as example where it commonly happens.
You are reading debugfs which you really have NOT idea what is in there. As you saw, you are reading from hardware that is slow and doing odd things when you read from it.
Agreed, I already sent a patch to LTP to blacklist it.
Hopefully you are blacklisting all of debugfs. and sysfs. And procfs. Adding specific files back is fine, as long as you "know" they are ok and you are actually testing something valid there.
thanks,
greg k-h
TBH, most of the value of this test comes from being dumb and doing things nobody thought to do because it would be stupid. It's an accident that the test exists in the first place because I found some bugs in older kernels naively trying to dump the contents of /sys.
It is useful to occasionally run this test as root in a VM where it can't do any harm, even on debugfs and dev. We have detected missing patches doing things like that which would probably be stopped by a sane whitelist.
However I would agree that this creates too much noise for a CI under normal circumstances. We could maybe add some meta-data to the test saying it will create noise and freeze the SUT and in the future users can specify what level of noise they are prepared to accept when executing the LTP.
In the meantime, IMO, we can just force the test to switch to nobody by default which it already does when reading /dev.
-- Thank you, Richard.
On Mon, 04 Nov 2019 14:35:51 +0100, Jan Stancek wrote:
----- Original Message -----
Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git Commit: dfe283e9fdac - Linux 5.3.9-rc1The results of these automated tests are provided below.
Overall result: FAILED (see details below) Merge: OK Compile: OK Tests: FAILEDAll kernel binaries, config files, and logs are available for download here:
https://artifacts.cki-project.org/pipelines/262380
One or more kernel tests failed:
x86_64: ❌ LTP liteNot a 5.3 -stable regression.
Failure comes from test that sanity checks all /proc files by doing 1k read from each. There are couple issues it hits wrt. snd_hda_*.
Yes, the debugfs access for HD-audio may easily lead to the OOM or such because HD-audio provides a pretty sparse register maps and the regmap debugfs iterates all possible register values.
I'm going to push a patch to disable the regmap internal locking, which eventually disables the whole regmap debugfs exposure of the device.
But, as Greg already suggested, the whole debugfs access can't be treated as safe at all, no matter which device is handled, so it should be avoided. It's the interface allowing you to touch the hardware internals directly, so you can screw up the system very easily.
thanks,
Takashi
Example reproducer: dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access of=out.txt count=1 bs=1024 iflag=nonblock
It's slow and triggers soft lockups [1]. And it also requires lot of memory, triggering OOMs on smaller VMs: 0x0000000024f0437b-0x000000001a32b1c8 1073745920 seq_read+0x131/0x400 pages=262144 vmalloc vpages N0=262144
I'm leaning towards skipping all regmap entries in this test. Comments are welcomed.
Regards, Jan
[1] [15341.137116] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [dd:636573] [15341.144141] Modules linked in: nls_utf8 isofs dummy minix binfmt_misc nfsv3 nfs_acl nfs lockd grace fscache sctp rds brd vfat fat btrfs xor zstd_compress raid6_pq zstd_decompress loop tu n ip6table_nat ip6_tables xt_conntrack iptable_filter xt_MASQUERADE xt_comment iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 veth bridge stp llc overlay fuse snd_hda_codec_r ealtek snd_hda_codec_generic sunrpc ledtrig_audio snd_hda_codec_hdmi snd_hda_intel snd_hda_codec edac_mce_amd kvm_amd snd_hda_core ccp snd_hwdep snd_pcm kvm snd_timer irqbypass joydev snd c rct10dif_pclmul crc32_pclmul soundcore broadcom bcm_phy_lib ghash_clmulni_intel sp5100_tco fam15h_power k10temp tg3 wmi_bmof pcspkr i2c_piix4 acpi_cpufreq ip_tables xfs libcrc32c radeon ata _generic i2c_algo_bit pata_acpi drm_kms_helper firewire_ohci ttm crc32c_intel serio_raw pata_atiixp firewire_core drm crc_itu_t wmi [last unloaded: can] [15341.223635] CPU: 2 PID: 636573 Comm: dd Tainted: G L 5.3.9-rc1-dfe283e.cki #1 [15341.232073] Hardware name: AMD Pike/Pike, BIOS RPK1506A 09/03/2014 [15341.238467] RIP: 0010:regmap_readable+0x15/0x60 [15341.242996] Code: 25 28 00 00 00 75 07 48 83 c4 10 5b 5d c3 e8 92 08 a6 ff 66 90 0f 1f 44 00 00 48 83 bf b0 01 00 00 00 74 31 8b 97 48 01 00 00 <39> f2 0f 92 c0 85 d2 0f 95 c2 20 d0 75 1 d 48 83 7f 70 00 74 01 c3 [15341.261765] RSP: 0018:ffffb3b100697dc8 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13 [15341.269330] RAX: 0000000000000000 RBX: ffff8d1a63773400 RCX: 0000000000000b41 [15341.276723] RDX: 000000000fffffff RSI: 000000000f3b4139 RDI: ffff8d1a63773400 [15341.283858] RBP: 000000000f3b4139 R08: 0000000000000000 R09: 0000000000000000 [15341.290989] R10: 000000000000000f R11: ffff8d19901fffff R12: 0000000000000079 [15341.298114] R13: 000000000f3b4139 R14: 00000000ffffffff R15: 000000000000006e [15341.305501] FS: 00007f4e067a0580(0000) GS:ffff8d1a6aa80000(0000) knlGS:0000000000000000 [15341.313583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15341.319322] CR2: 00007f5083682dc0 CR3: 00000001287dc000 CR4: 00000000000406e0 [15341.326462] Call Trace: [15341.328912] regmap_volatile+0x4f/0xa0 [15341.332667] regmap_access_show+0x70/0x100 [15341.336997] seq_read+0xcd/0x400 [15341.340247] full_proxy_read+0x57/0x70 [15341.343997] vfs_read+0x9d/0x150 [15341.347231] ksys_read+0x5f/0xe0 [15341.350473] do_syscall_64+0x5f/0x1a0 [15341.354166] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [15341.359216] RIP: 0033:0x7f4e066823c2 [15341.362795] Code: c0 e9 c2 fe ff ff 50 48 8d 3d c2 0d 0a 00 e8 b5 f1 01 00 0f 1f 44 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 0 0 48 83 ec 28 48 89 54 24 [15341.381852] RSP: 002b:00007ffc6ad30e88 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [15341.389427] RAX: ffffffffffffffda RBX: 0000000000000400 RCX: 00007f4e066823c2 [15341.396568] RDX: 0000000000000400 RSI: 0000561bc69f0000 RDI: 0000000000000000 [15341.403979] RBP: 0000561bc69f0000 R08: 0000561bc69efb60 R09: 00000000000000c0 [15341.411111] R10: 0000561bc69f0000 R11: 0000000000000246 R12: ffffffffffffffff [15341.418244] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffffffffffff
Alsa-devel mailing list Alsa-devel@alsa-project.org https://mailman.alsa-project.org/mailman/listinfo/alsa-devel
linux-stable-mirror@lists.linaro.org
-
CKI Project -
Greg KH -
Jan Stancek -
Richard Palethorpe -
Takashi Iwai